Unsupported ICS: Not an Easy Upgrade
Wednesday, October 7, 2015 @ 01:10 PM gHale
By Gregory Hale
Industrial control system (ICS) software moving into the unsupported realm is a true fear and problem for users. Just what can they do?
Yes, the quick answer is just to simply upgrade, but in the complex world of ICSes that is easier said than done. Then, which inevitably will happen, a vulnerability crops up in the unsupported system, which just happened. In that case, an upgrade will take time to test to make sure the user irons out all compatibility issues. Risks just don’t stop because you are aware of a vulnerability. Keeping a system running with a vulnerability is a risky endeavor. The user needs to take some type of action.
“It is important to realize that industrial control systems are installed and commissioned to perform very specific tasks,” said Joel Langill, operational security professional, ICS cyber security expert and founder of SCADAhacker.com. “If these systems are performing these tasks satisfactorily, many users may incur additional risk by deciding to upgrade from something that is ‘working’ to a product that is ‘unknown’. For this reason, it is difficult to balance the decision to upgrade versus a potential impact to operational integrity.”
Along those lines, oftentimes “ICS end-users are resistant to upgrading to the first release in a new major software revision, and decide to wait until the first minor release,” Langill said.
Upgrading to an existing brownfield site has its benefits and detriments. Additional functionality is definitely a bonus, that is, of course, if the manufacturer ends up using the added features. The downside is the potential danger of some type of incident the upgrade could cause.
Pros and Cons
“When it comes to upgrades, there are two mutually exclusive aspects one should consider,” Langill said. “On one side, each release will bring new functionality. For many of us, this new functionality may not be required and in fact, there is always a subtle fear that a feature upon which you depend is no longer available in a new release. In this case, when the new functionality is required, the decision to upgrade is rather straightforward.
“On the other side, new releases typically address security-related issues (both publicly and privately known). Many believe that the decision to upgrade is obvious. However, remember that in addition to the functionality issue just mentioned, there is also the potential the upgrade could result in an impact to operations therefore causing an outage, an environment incident, or any number of other undesirable events. For this reason, the decision to upgrade in these cases should be solely with the end-user and not ‘forced’ or ‘mandated’ by the vendor. Sure, they may not have fully supported software, but the cost of per-incident support could be a ‘drop in the bucket’ compared to a manufacturing outage. This is the reason why vendors should over construct mitigation strategies to assist end users when they cannot immediately perform an upgrade. These mitigations will obviously not solve the problem, but can provide credible compensating controls that could reduce risk to acceptable levels that may be significantly below those resulting from an operational incident.”
In the IT world, a system upgrade can occur on a regular basis and they can install the upgrade over the weekend when no one, or very few people, is at the office. In the manufacturing automation environment, availability 24×7 is vital. Systems just can’t come down.
“Users are often reluctant to upgrade their systems, or at least not every year or two as in the IT world,” said Graham Speake, vice president and chief product architect at NexDefense, Inc. “Systems are expensive and often complex, and upgrading part or all of the system may have to include a long and expensive test to ensure that everything has been configured correctly. Typically, an upgrade cycle of 5 years plus is the norm, but often this is stretched a lot more. Especially in the U.S., regulation may also play a part in the decision. Pharma for instance is often happy to run on more outdated equipment because it has been tested and approved and an upgrade often entails an expensive retest — coupled with a loss of production.
“Remote sites such as offshore platforms are also often overlooked in the upgrade regime. The isolation (or perceived isolation) often means that these are running on much older versions of software. Minor upgrades in particular are usually overlooked — as well as a lot of security upgrades.”
Speake also talked about the issues facing embedded systems.
“Another area to consider is the large pieces of equipment which have a computer and operating system in it, but sometimes not in an obvious manner,” he said. “Some IT equipment such as printers fall into this trap, but in industrial systems, items such as analyzers which have a high capital cost, and are essential to the business, are often hard to impossible to upgrade.”
Solutions are available, but, as Speake said, they may not be implemented due to cost, complexity or just a willingness to accept the risk assuming they even do a risk analysis.
“Defense in depth and deployment of network intrusion detection systems (NIDS), host intrusion detection systems (HIDS), whitelisting are methods of managing security,” said Marco Ayala, senior project manager at aeSolutions.
Speake added some suggested solutions are:
“1. Industrial firewalls. These work for some solutions, but not always ideal for larger or whole system issues. I have seen these work in front of analyzers for instance, but if the whole system needs upgrading, it can be an issue.
“2. Larger industrial firewalls. These work at the boundary and are having more industrial protocols, but really are just protecting the different layers. These are getting better and will improve over the next couple of years.
“3. Segmentation. This can work with a combination of the first two to ensure the older systems are separated out.
“Solutions 1, 2 and 3 may require some rework of the system architecture so may not be something the end user wants to do — he may as well upgrade. Usually these work if a large segment can be isolated. Also, 1 and 2 suffer from getting the rules right.
“4. Unidirectional gateways. These can work, but are usually high dollar price, and also need the right kind of system to be implemented with. The nuclear industry does use them a lot, and it is a good segmentation method. I also know that one oil and gas company is planning to deploy them on their drilling vessels.
“5. There are passive detection tools, such as an IDS or network anomaly detection tool. An IDS will need to be tuned and have rules put into it and this may be beyond the scope of a lot of OT people. Also, there are very few open source IDS rules for industrial protocols. There are a number of industrial behavior and anomaly detection tools coming onto the market. These have more and more awareness of industrial protocols often with deep packet inspection built into it.”