By Gregory Hale
Trust is the cornerstone of any relationship when it comes to people or business. Once that trust ends up violated, whether on purpose or by accident, it can lead to the downfall of any relationship.
That is exactly the case the SolarWinds supply chain attack revealed this week where hackers inserted malware into a service that provided software updates for SolarWinds Orion platform, which is a suite of products used across a wide swath of organizations from the manufacturing industry to the government and other key sectors, to monitor the health of their IT networks. SolarWinds said 33,000 of its customers were Orion customers, and 18,000 of them may have the product that contained the malicious code.
“That is the beauty of a supply chain attack, sadly,” said Eric Byres, chief executive of aDolus, a SaaS-based system provider that integrates machine learning and security expertise to deliver a “trustworthiness” score for software. “You are exploiting the trust between the vendor and the customer. You assume as Mr. Customer what you are uploading is gold. That is where it falls back to the vendors. The vendors have to do a better job of maintaining that trust. Vendors have to take more responsibility for making sure the code that is actually getting shipped out to customers is more than just ‘we pushed it up to the server and now it is the customer’s problem.’ ”
Orion Platform
The vulnerability ended up inserted within the SolarWinds Orion Platform software builds for versions 2019.4 HF 5, 2020.2 with no hotfix, and 2020.2 HF 1, which, if present and activated, could potentially allow an attacker to compromise the server on which the Orion products run. This incident was likely the result of a highly sophisticated, targeted, and manual supply chain attack by an outside nation state.
“The attack’s sophistication was so incredible that even one of the world’s leading server security company FireEye had the hackers in their networks for months without them knowing,” said Dewan Chowdhury, chief executive and founder of security provider, malcrawler. “What made this even more devastating is the software was digitally signed to protect the software’s integrity. Digital signing of a file is a way to ensure peace of mind to the end-user, knowing that the software is free from manipulation.”
“This was signed code, and that is supposed to be bullet proof,” Byres said. “It didn’t’ work. Stuxnet was signed code. There is no shortage of signed code out there. I have been told, there is more signed malware out there than signed good code. The over reliance on ‘we will sign the code,’ is going to crumble when people see there is more they have got to do. It is even worse in our environment because code signing only works on a Windows box. There is no code signing in most of the industrial systems. People need to look at how can we build for really reliable whitelists. Stop just looking for malware and looking for vulnerabilities, but create a real-time whitelist on what should be running on our control system.”
Another way to help quell a supply chain attack is through network segmentation.
Network Segmentation
“In the industrial control system world, segmenting your crown jewels through network segmentation from the rest of the environment is the ultimate method to ensure only legitimate communication paths are established,” Chowdhury said. “No matter how sophisticated the attacker, if they attempt to access a segmented system, the alarm bells will start ringing. Industrial control systems, as complex as they are, their communication is minimal, so detecting anonymous traffic is relatively easy.”
“Risks associated with the exploitation of advanced management infrastructures is not new,” said Joel Langill, director of industrial cybersecurity services for mission engineering and resilience at Amentum. “Originally this was exposed by the exploitation of administrative accounts and the ability to have blanket access to entire architectures with a single compromise. The implementation of network management networks and tools like SolarWinds Orion are no different. This is why it is important that organizations should properly secure the security architectures they deploy. IEC/ISA 62443 offers the basis for this design through the creation of security zones and conduits, and by establishing strict least privilege policies for flows within and between these zones and conduits.
“In many cases, organizations deploy solutions for convenience (e.g., centralized management) rather than security. Actual field assessments have confirmed that organizations often violate basic security practices when deployment management and security solutions by allowing single points of access to enter architectures that span both enterprise and manufacturing domains. It has always been my approach that high-risk, high-value management should always be contained within a given security zone. In doing so, it helps to contain the extent of impact when a single asset or zone is compromised.”
Flows Between IT and OT
“We see SolarWinds everywhere,” said John Cusimano, vice president of industrial cybersecurity at aeSolutions. “It is definitely deployed into OT environments and, most concerning, is that SolarWinds tends to ‘live’ in IT which means SolarWinds traffic is permitted to flow between IT and OT through the firewalls. This embedded malware can get a free ride into the OT environment in companies that are setup this way, which is many.”
While this week has been a whirlwind of news regarding the attack, security experts feel this is just the beginning.
“I think we have only seen the tip of the iceberg in terms of the companies that will fall victim to this,” Cusimano said.
“In the next few weeks, we will see more government institutions, companies, and universities, acknowledging they have been victims of this attack,” Chowdhury said. “What concerns me is the hackers have been in some of the most prestigious government agencies, companies, universities, and for months operating, stealing intellectual property without anybody knowing. They have been doing this while cleaning their tracks, bypassing the tens of millions of dollars of cybersecurity tools these organizations have implemented in their environment.”
This isn’t the first time security professionals have suffered from this type of attack. They have occurred in the past, but they have remained fairly quiet or they just didn’t garner the big headlines.
“In 2014, BlackEnergy, a Russian state cyber actor, used the same hacking technique into one of the supply chains for an industrial control system manufacture,” Chowdhury said. “They embedded their malware into system updates/patches, then downloaded by industrial control system operators in their OT and IT network. This technique is not new, but what makes this attack so unique is they hit a company that practically every large and medium-size organization from private enterprise to public institutions utilize in their network management. I guarantee you most industrial control system operators from critical manufacturing to power utility have SolarWinds.”
Accountability
One of the issues coming forward in this scenario is accountability. It appears vendors are not keeping a complete eye on the product once it gains approval.
“The problem is once the software ships, it is like ‘oh, we won’t worry about it anymore. This stuff got posted up and it got through the checks.’ The issue is all the checks are for vulnerabilities but not malicious code,” Byres said.
The vendor just thinks it is posted, and it is just up there, “whoever downloads it, we don’t know, whatever gets downloaded, we don’t know, wherever it goes, we don’t know. It just goes into a black hole,” Byres said. “This got compiled and signed back in March and we are finding out about it now. This is part of the supply chain that is broken. It is just not good enough to put a package up there and forget about it. We have got to monitor it for the customers and the vendors on what is happening to your software, who is using it?
There needs to be more checkpoints when software releases, Byres said. It is not good enough that developers are doing the checks, there has to be other checks after compiling the software and putting it up on the server.
Vulnerability checker tools are fine, but they are looking for known vulnerabilities, not for potentially malicious code. “No one is looking for what is not normal here,” Byres said. “It is not looking for anomalies in our code right now that weren’t there this morning. We need code anomaly detection.”
There has to be a final check to say this is software we are willing to ship to our customers Byres said. Then the vendor has to monitor it continuously. Right now, this just doesn’t happen.
“Software is like a fine wine or bread, you just don’t make it and forget,” Byres said. “Once it is out there, you have got to monitor it and baby it, too. You have to look after it.”

