By Gregory Hale
One ransomware attack hit a major manufacturing company last year and in effect it shut down 30 plants and one incident response (IR) team had to get the plants back up and running in three days.
Three days.
“There is a long dwell time in the OT attacks,” said Bryan Singer, principal director of OT incident response global lead at Accenture during his presentation at the ICSJWG Spring Meeting in Salt Lake City, Utah, last week. “There are three steps in an attack, if you haven’t found it by one of the first steps, it is going to be a boom. When you have an IR, create some analytics, we conduct analytics, and we will start shutting things down. We are also going to leave those systems down so we can go about eradicating the threat actor.”
Most times when an incident response kicks off and the responders are starting to get active, the user will want to start plants back up while you are working on the incident, Singer said. “They will start crashing again, and they will be looking to you as a crisis manager to figure how in the world will you be maintaining operations while the IR is going on.”
Singer pointed to a case study where a company had 130 plants globally and it suffered a ransomware attack.
“We had 30 sites we had to do in three days,” he said. “They initially said they would shut down the plants for three days. These plants were located in North America, South America and Central America. I started work on the IR at 7:30 Monday morning and they tried to restart the plants that day. I said you were going to give me three days and they said we had no choice, the plants had to go back up.”
While the IR team was working on a timeline on what the threat actor did and when, they also had to deal with a new problem of trying to prove a live system against live threat actor behavior.
“We were also trying to separate a needle from a stack of needles because how many people have seen a plant with a single malware on it? This particular one had two APT actors at the same time,” Singer said.
Singer went on to say he tries to categorize the nature or types of incidents:
- IT compromise with the movement to OT where companies with IT assets end up compromised but no lateral movement or impact to OT. IT teams then can prove there is no OT impact. Proving that, though, takes a long time and Singer said it is a pain in the neck.
- IT compromise impacts OT, but not targeted to OT. Making sure no one has targeted OT and can be taken care of traditional IR teams.
- OT exfiltration/reconnaissance. OT systems probed and data exfiltrated but no direct attack against OT.
- Insider or insider equivalent attacks, insider intentionally compromises integrity of OT, or threat actors gain inside knowledge.
- Deep insert OT attacks. Heavily targeted OT malware or infrastructure level attacks down to I/O. Those involve full OT qualification of function, firmware and hardware analysis. You will need people with deep OT experience.
Singer said he builds his IR teams based off these categories.
Another aspect is incident responders often look for end points and then work their way back to stop the attack.
“If an OT incident occurs, I am not asking questions about end points,” Singer said. “This is where we start to fall down. A lot of incident responders still go on this idea if they find where the end points are I can eradicate stuff on the end points, and this ends up being incident response for OT. The question I have is when. When did the drive turn on? When did the drive start? When did the speed change? When did the pumps start/stop? When did the alarms stop?”
There are plenty of solid tools and techniques out there these days to help find an attack, but as Singer said, coming from an incident responder, they are often not deployed very well.
“The tools tend to look at ingress and egress North/South traffic, they do OK at that. It has nothing to do with the tool, but it has to do with the way it is deployed. You need to look at North/South traffic, but we need to start increasing our sensor count and look at East/West traffic. If there is not adequate sensor deployment in the environment and this environment ends up compromised, now I have to deploy more collectors, I need more data. Your job is to gain and maintain positive control over OT. “
Singer said the following are essential skills for OT incident response:
Essential documents for effective OT incident response, understanding the process:
- ICS protocol inspection
- Points for consideration
- Inspect for ICS protocol health, is it safe to return to service
- Layer in multiple sites and North/South, East/West traffic
- Continually reassess IoC
ICS communications analysis
- Are outbound and country of concern truly external traffic, or improper use of public IP space
- Are point usages properly identified
- Are offsite communications bot ingress and egress, or egress only?
Understand ICS/OT security alerts
Identity remote connections
- Are threat actors reusing credentials for external remote support connections
- Do remote support connections bypass IT AD authentication
Binary, SMB and executables
Understand industry safety
- Are any safety PLC/SIS bridged to other networks
- Are soft interlocks used where SIS should be
- Do C&E’s point to potential impending failures
No one incident response is going to be the same, but in this day and age of increased connectivity through digitalization, it means the attack surface will continue to grow. It also means there will be more incidents that threaten to shut down operations.
“The drive toward IT and OT convergence, and the cloud is a foregone solution and start preparing for it if you haven’t already,” Singer said.

