Water Company Operations Breached

Wednesday, March 23, 2016 @ 01:03 PM gHale


By Gregory Hale
It is becoming a cliché throughout the industry, but even companies that don’t know they suffered a breach actually learn they are victims.

Take one water company as a perfect case scenario where attackers were able to modify application settings with little apparent knowledge of how the flow control system worked. In fact, they manipulated the system to alter the amount of chemicals that went into the water supply.

RELATED STORIES
IIoT: We Have to get Ahead of This
Attacking an ICS from ‘Inside Out’
Breach at IN Utility
Attacks on Rise, Incident Response Tougher
Security Framework Grows in Usage
Open Systems Needs Intrinsic Security

The Kemuri Water Company (KWC), as Verizon referred to it in its Data Breach Report 2016, was responsible for supplying and metering water usage over a number of neighboring counties. From the onset, KWC was adamant that no evidence of unauthorized access had been uncovered and the Verizon assessment was more of a proactive procedure. A part of an ongoing effort to maintain healthy operations of their systems and networks, according to the report. In scope they looked at all of KWC’s IT systems, which supported end users and corporate functions, as well as Operational Technology (OT) systems, which were behind the distribution, control and metering of the regional water supply.

Behind the scenes, KWC was a likely candidate for a data breach, according to the report. Its Internet facing perimeter showed several high-risk vulnerabilities often exploited.

The OT end of the water district relied heavily on antiquated computer systems running operating systems from ten-plus years ago. Even more concerning, critical IT and OT functions ran on a single AS400 system. KWC referred to this AS400 system as its “SCADA platform.” This system functioned as a router with direct connections into several networks, ran the water district’s valve and flow control application that was responsible for manipulating hundreds of Programmable Logic Controllers (PLCs), housed customer PII and associated billing information, as well as KWC’s financials.

First Point of Attack
Moreover, only a single employee was capable of administering it, according to the report. If a data breach occurred at KWC, this SCADA platform would be the first place attackers would look.

Interviews with the KWC IT network team revealed concerns surrounding recent suspicious cyber activity. It became clear that KWC management was aware of potential unauthorized access into the OT systems of the water district. More specifically, an unexplained pattern of valve and duct movements had occurred over the previous 60 days. These movements consisted of manipulating the PLCs that managed the amount of chemicals used to treat the water to make it safe to drink, as well as affecting the water flow rate, causing disruptions with water distribution.

KWC granted permission to cross-correlate KWC’s Internet traffic against the Verizon Cyber Intelligence Center’s cyber intelligence sources to identify evidence of communications with known attackers.

This test showed positive results.

First, adversary IP addresses from three recent investigations connected to KWC’s internet payment application. Second, these IP addresses ended up discovered while investigating hacktivist attacks. The third aspect was the suspect connections corroborated with the payment application’s web server logs, suggesting likely exploitation of Internet-facing vulnerabilities detected earlier in our assessment.

At this point, Verizon researchers identified likely evidence of a security breach with an avenue of intrusion and had compelling IOCs based on first-hand precedent. The next step was to prove or disprove the preliminary findings.

The Internet payment application enabled KWC’s customers to conveniently access their accounts from a laptop, a desktop system or even a mobile device. However, a quick look showed some serious security flaws. Access to customer water usage, PII and payment data required only a username and password.

Internet Facing
There was no second authentication factor employed. Researchers next found a direct cable connection between the application and the AS400 system. Making matters worse, the AS400 system had open access to the Internet and its internal IP address and administrative credentials were found on the payment application webserver in clear text within an initialization (.ini) file. In other words, we found a high probability any unauthorized access on the payment application would also expose sensitive information housed on the AS400 system.

Researchers said at this point they uncovered a “fire trail.”

As it turned out, the attackers exploited an easily identified vulnerability in the payment application, leading to the compromise of customer PII and payment information. The total population of unique records exfiltrated from the AS400 system exceeded 2.5 million. They could not find evidence of fraudulent activity on the stolen accounts. That was the good news.

The bad news was customer information was unfortunately not the full extent of the breach. The typical semantic footprint of a hacktivist attack shows greater interest in denying and disrupting the victim’s ability to conduct business than stealing information for financial gain. That was definitely the case here.

Endpoint forensic analysis revealed a linkage with the recent pattern of unauthorized crossover. Using the same credentials found on the payment app webserver, attackers were able to interface with the water district’s valve and flow control application, also running on the AS400 system. Verizon researchers also discovered four separate connections over a 60-day period, leading right up to our assessment.

During these connections, attackers modified application settings with little apparent knowledge of how the flow control system worked. In at least two instances, they managed to manipulate the system to alter the amount of chemicals that went into the water supply and thus handicap water treatment and production capabilities so the recovery time to replenish water supplies increased, the report said. Based on alert functionality, KWC was able to quickly identify and reverse the chemical and flow changes, largely minimizing the impact on customers. KWC could not find a motive for the attack.

Remediation and Recovery
With a clear indication of what occurred, working with KWC Verizon researchers turned toward remediation. Immediately IT administrators shut down access to and from the account management web front end and blocked outbound connectivity from the AS400 system. With the threat actor’s access presumably cut off, systems were rebuilt with baseline images and placed back online. Verizon recommended to KWC it replace its older systems with modern versions and apply up-to-date patching as necessary. Additionally, researchers highlighted issues in its continuity planning revolving around single points of failure — namely the lone AS400 system administrator. In addition to having no backup for emergencies such as this, operating alone and without oversight, configuration choices made for convenient management were unchecked by security considerations.

Lessons Learned
KWC’s breach was serious and could have easily been more critical.

If the attackers had a little more time, and with a little more knowledge of the ICS/SCADA system, KWC and the local community could have suffered serious consequences. Large organizations are often complex and KWC was no different.

On top of the complexities of actually managing and delivering water to homes, KWC also handled all of its own account and transaction information. In complex systems like these, centralizing assets can make management easier, but cannot be done without thought to security.

Having Internet facing servers, especially web servers, directly connected to SCADA management systems is not a best practice. Issues like outdated systems and missing patches contributed to the data breach — the lack of isolation of critical assets, weak authentication mechanisms and unsafe practices of protecting passwords also enabled the threat actors to gain far more access than should have been possible.

KWC’s alert functionality played a key role in detecting the changed amounts of chemicals and the flow rates. However, employing a layered defense-in-depth strategy could have detected the attack earlier, limiting its success or preventing it altogether.

Click here to download the entire report.