Robustness Testing: Saves Lives, Money

Tuesday, May 1, 2012 @ 03:05 PM gHale

By Nate Kube
Though it’s critical to accurately identify vulnerabilities in process control networks and devices, a chief executive or management team will likely question the additional investment in robustness testing.

At first glance, what they often don’t recognize is that additional investment will end up improving the bottom line.

Robustness testing provides insight into how environments perform under stress, it goes the extra mile beyond requirement specifications to ensure control systems exceed specifications in emergencies.

Siemens CERT Gains Achilles Status
Security First; Not in Smart Grid
Smart Meters Getting Smarter
Secure Smart Grid Moves Forward

National Aeronautics and Space Administration (NASA) and the U.S. Department of Defense (DoD) reports indicate over half of the bugs found in deployed devices directly relate to a lack of robustness testing. Now with resources in short supply, developers cannot be reverting to fixing flaws that should have never shipped.

While assuming there are extra resources to troubleshoot after the fact, the challenges of critical infrastructure testing are more significant. For instance, rebooting a PC can cause a minor disruption, rebooting a nuclear power plant has broader implications.

Defining Robustness Testing
The Institute of Electrical and Electronics Engineers (IEEE) defines robustness as “the degree to which a system or component can function correctly in the presence of invalid inputs or stressful environmental conditions.” Having a properly functioning system, despite the unpredictable, is essential in the industrial control systems world in order to “keep the lights on.”

Overlooking Robustness Testing
Quite a few DoD programs engage in what has been called “happy-path testing,” in other words only showing the system maintains functional requirements, according to a DoD assessment report. While this type of testing is essential, additional tests to ensure the system properly handles errors and failures appropriately are often neglected. Performing “happy-path testing” underscores that control system failure in the field is often due to a lack of robustness.

Although vendors’ solutions meet user requirements for particular installations, users may not be able to quantify the level of robustness required for specific installations. At present, most industrial control equipment manufacturers and software developers are limited in their ability to rigorously test new products for possible security flaws because of the lack of available tools.

As a result, new vulnerabilities are discovered each year, but only after the products are sold and installed by the end user. This is particularly true of control and SCADA systems used in critical infrastructures such as the oil and gas, water, and electrical generation/distribution industries. Standard information technology (IT) vulnerability testing does not typically address the unique resources and timing constraints of critical control systems or the specialized protocols used.

As business and technology continue to drive toward more open and connected networks, mission critical systems – including those used in the control of power generation, oil and gas production, water treatment and transportation – are becoming increasingly vulnerable to cyber attacks that penetrate or bypass perimeter defenses (e.g. firewalls).

Yet, how does one measure and assess something that doesn’t necessarily happen? Additional testing is a tough sell for management if the current testing regimes appear successful. How and why can one build a case for an expanded testing capability or continued diligence?

Dollars and Sense
NASA leads the industry in computer usage and complex systems. They advise robustness testing through usage of off-nominal cases. NASA believes a methodology able to test for off-nominal cases (i.e., hardware and software failures) during design, and the earlier test stages, could avoid over one-half of all failures and over two-thirds of the failures in the most severe classifications.

For a quick rubric:
1. Make an estimate of the additional costs you’ve incurred over the last year due to robustness failures of your systems in the field.
2. Multiply that figure by .50 to get the low end of the range and by .66 to get the high end of the range.

Although this can quickly determine the value of pursuing a course of action, this may not be enough to persuade management to make additional robustness testing investments.

Fortunately, several valuation models can aid in putting a dollar amount on security costs. Carnegie Mellon University for the U.S. Department of Homeland Security published a paper that makes a comparison between 13 different models for assessing the cost and value of software assurance. They found several features common to each model and categorized the models into four types: Cost-based, investment-based, quantitative estimation and environmental/contextual. A follow-up paper provided by the same company demonstrated organizing its approach focused specifically on the Balanced Scorecard model. The Balanced Scorecard is widely used; one major explanation to its success (and to the success of all quantitative methods) is data. Before embarking on any effort to quantify the cost of robustness testing, an organization must have metrics in place and data collected and validated.

Budgets are shrinking and threats are increasing. Times are difficult with economic hardships, but security cannot be compromised. Companies have the capability to increase the robustness of their systems to reduce the time to market and produce a quality product while decreasing overall costs.

Protecting critical infrastructure and “keeping the lights on,” is the singular aim of any robustness test. Robustness testing is not just important, it is essential. This expands from the plant floor to every point where an organization’s system is touched by the Internet.

As more devices become Ethernet-enabled in the control systems world, we can no longer depend on “security through obscurity.” Everyone needs to be confident that implemented security solutions function effectively under known, as well as, unexpected conditions.

Nate Kube founded Wurldtech Security Technologies in 2006 and as the company’s Chief Technical Officer is responsible for strategic alliances, technology and thought leadership.

Leave a Reply

You must be logged in to post a comment.