By Gregory Hale
There is no doubt cloud connectivity across the manufacturing sector is increasing, so when Microsoft on Wednesday said it had recovered all its cloud services after a networking outage took down its Azure cloud platform, questions arose on what a company can do to protect against a cloud outage.

Whether it is a big or small outage for Azure, which has 15 million corporate customers and over 500 million active users, it can impact multiple services and create a domino effect of issues as almost all the world’s largest companies use the platform. Azure’s status page showed services ended up impacted in the Americas, Europe, Asia Pacific, Middle East and Africa.

While the OT environment has not totally shifted to the cloud, businesses have become more dependent on online platforms after the pandemic caused a shift to more employees working from home.

“The answer for OT is really the answer for anything where it’s critical to have an on-premises component,” said Ron Fabela, CTO & co-founder of security provider SynSaber. “Many years ago, the conversation was about ‘Enterprise’ and plant managers had this same question and challenge: Can OT still function if there is a Wide Area Network (WAN) outage back to the enterprise networks? As more and more functions migrate from enterprise to the cloud, the challenge and solution is practically the same. Ensuring OT can continue to operate in a disconnected mode is critical to overall resilience so whether it is an enterprise or cloud outage, our industrial processes can persist.”

Schneider Bold

Implement Local Redundant Systems
Fabela said for companies to remain secure, they should identify, test, and implement local redundant systems.

“First, it’s important to understand what external dependencies the industrial process has to enterprise or cloud components, this may not be apparent as functions migrate. Once those dependencies have been identified operations can determine and test the impact of an outage. Finally, when impact is determined, redundant systems local or close to the industrial process can be put in place in order to minimize impact during an outage.”

At one point not too long ago, the idea of OT using the cloud was a point of contention, but today there are plenty of organizations implementing cloud solutions.

“In the past few years, the industry has witnessed edge cloud solutions that have allowed the industry to utilize, for example, Azure to pull/push data from ‘non-time-sensitive’ OT equipment,” said Dewan Chowdhury, chief executive and founder of security provider, malcrawler. “An example of this would be a gas utility company sensor that analyzes the air for any elements that indicate problems. The sensory information is processed by OT gear and sent to the cloud via cellular modem to a cloud solution. Due to logistic and financial difficulties, the industry needs more help connecting the corporate connectivity resource to field sites. Leveraging cloud solutions has become much more cost-effective.

“The biggest pro is the cost-effectiveness of companies utilizing cloud solutions to push/pull OT data versus the heavy investment of setting communication infrastructure from the site to their data centers. As more corporate IT resources get pushed to the cloud, this allows an avenue to leverage the same cloud infrastructure for OT data.

“Another advantage is cloud-based systems can provide improved data management and analysis capabilities. With cloud-based systems, companies can easily collect, store, and analyze large amounts of data from their OT systems, which can help them to identify trends, patterns, and opportunities for improvement.

Disaster Recovery Plan
“Innovative companies who leverage cloud technology for their OT devices have disaster recovery plans to ensure in the event the communication link fails, the data will be stored locally on the system for “X” amount of time specified by the company. When the communication gets restored, it will push that data back to the cloud. Some go as far as having alternative cloud companies or pushing it directly to the corporate side in the event of a failure,” Chowdhury said.

Microsoft said it had determined a network connectivity issue was occurring with devices across the Microsoft WAN. This impacts connectivity between clients on the Internet to Azure, as well as connectivity between services in data centers, it said.

Microsoft later said it had rolled back a network change it believed was causing the issue and was using “additional infrastructure to expedite the recovery process.”

When you talk cloud and manufacturing it all has to center around reliability.

“Reliability-wise, making cloud operations integral to industrial operations opens up reliability risk,” said Andrew Ginter, vice president industrial security at Waterfall Security Solutions. “For large organizations that standardize on a particular cloud/industrial cloud and come to rely on it for moment-to-moment operations, that provider (hacked or just malfunctioned) becomes a single point of failure for the entire enterprise.

“Security-wise, providing a communications path from your industrial processes into dozens of industrial cloud services from vendors large and small, well and poorly-secured, introduces cyber risk. An attacker – think modern sophisticated ransomware – breaks into a poorly defended cloud provider with hundreds of clients, both large, thoroughly-defended clients and smaller ones. The attacker uses the compromised cloud service to compromise hundreds of sites simultaneously, some of which are huge and otherwise well-defended sites.

Cloud That Makes Sense
“Cloud services that make sense are ‘open loop’ – you send the data to the cloud and the conclusions/benefits you get back are via the cloud’s website, or via email or text message to your cell phone. You, personally, filter those conclusions through your brain before deciding to accept them and take action in your industrial process. Not closed loop is where the cloud sends instructions directly to the industrial process for action.

“And the safest open loop connections have unidirectional gateway technology between the data source and the cloud service, so no compromised cloud, or compromised firmware, or compromised cryptosystem can leak attacks back into the industrial source,” Ginter said.

Microsoft did not disclose the number of users affected by the disruption, but data from outage tracking website Downdetector showed thousands of incidents across continents. The Downdetector site tracks outages by collating status reports from various sources including users.

“The reliability of cloud services has not reached the point where they should be considered sufficient to operate safety critical processes,” said Mark Carrigan, senior vice president of process safety & OT cybersecurity at Hexagon. “Owner operators should continue to use on-premises solutions for at least Level 1 where basic process control occurs, and likely for level 2 (which is the supervisory control area) that can be quickly disconnected from outside sources and operate independently.”

Pin It on Pinterest

Share This