The OT Supply Chain Threat

OT security requires multiple layers of defense, in the book I am writing and some conference presentations later this year I explain 9 different protection layers of which only 4 are the typical digital defense layers. I named one of these layers the “Procurement Protection” layer, which explores the protection mechanisms to better control the external influences during the contracting and maintenance phases in the installation lifecycle. Procurement as a security control isn’t widely discussed in the security community, yet it’s crucial for any system’s security. As one of the tasks, the procurement protection layer is responsible for countering the threat posed by the supply chain. While being watchful of this threat beyond the procurement stage is necessary, the procurement step is of key importance in managing the supply chain threat effectively.

While asset owners of (national) critical infrastructure are increasingly focusing on the supply chain associated with Operational Technology (OT), surprisingly, the security market appears to be relatively quiet on this topic. I think several factors contribute to this silence:

🚫The Security Market has a Dominant Focus on Technical Defenses

The OT security market has traditionally focused on technical defense measures. This is largely driven by the scalability advantages that technical products offer in comparison to consultancy-focused topics like countering OT supply chain attacks. Consultancy services rely on skilled human resources, which results in increased expenses and elevated business risks for the OT security provider. Building a $100 million product-based business is considerably more lucrative and requires less effort compared to establishing an equally sized consultancy enterprise. Furthermore, individuals possessing expertise in OT systems combined with hands-on experience in securing these systems are scarce, and certificates are not a guarantee for good performance. Given these business considerations and the absence of technical solutions in the form of a product or service that defends against the OT supply chain attack, the attention of the OT security market is skewed toward discussing technical solutions for security products, solutions like sBOM and anomaly detection products.

Traditional consulting companies do try to talk about this topic, but they often don’t have enough knowledge about OT. So, their approach to the topic becomes similar to how they handle supply chain attacks for IT systems. However, there are distinctions due to the fact that disruptions in critical OT systems have a more significant effect and not all programmable/configurable OT components are in the form of the traditional computer systems. Some are very small, hardly noticeable in an equipment cabinet but sometimes with very critical functions.

Therefore these systems are less able to handle risks (they have a lower risk tolerance due to the impact on human life and environment), and their operational lifetime is notably longer than in the case of IT systems. An extended lifetime because OT systems often include vital industrial property locked into the system’s configuration, making them difficult to replace with system components of another vendor because of longer system downtime due to reengineering and testing costs. This extended lifetime results in more complex long-term vendor-asset owner relationships, potentially leading to substantial vendor support challenges and dependencies if changes are required.

🚫Inherent Trust

OT environments operate with an inherent level of trust among vendors and partners due to these longstanding relationships. This trust can potentially foster a sense of complacency and with that a mistaken belief that supply chain attacks are improbable. Nevertheless, the core systems of OT technology, such as for example DCS and SCADA systems, are supplied by a small community of companies that hold a significant share of the global OT system’s market. If we further group these companies based on geographical factors, it becomes evident that geopolitical tensions (take, for instance, the conflict between Russia and Ukraine / Europe / US) can render national critical infrastructure highly susceptible to supply chain attacks or depending on components no longer available. In principle, vendors of OT process automation usually adopt a global business strategy and tend to avoid discussing this aspect of their products. However, the geopolitical situation can sometimes compel them to address this aspect due to external factors.

🚫Lack of Clear Ownership

Mapping out a supply chain for an OT system and establishing unambiguous ownership of supply chain security presents difficulties. OT products/solutions/systems are constructed using a wide range of components, often developed by engineering teams across different countries and by different companies. Supply chain security responsibilities often span a variety of stakeholders located in different geopolitical areas. This complex sharing of responsibility leads to uncertainty and, as a result, shifts focused attention away from the matter due to its complex nature. This is one of the reasons why stakeholders often avoid taking responsibility, let alone accepting accountability.

🚫Regulatory Gaps

Regulatory frameworks for OT security currently do not explicitly address supply chain risks and accountability. Consequently, stakeholders may not give priority to addressing these risks due to the lack of well-defined regulatory directives. While regulators are becoming more aware of the significance of the supply chain threat, enforcing controls is challenging as it could potentially lead to protecting a market against foreign products, a practice in conflict with various trade agreements.

Based upon the questions I frequently get; I believe that asset owners and regulators are increasingly aware of the supply chain threat. However analyzing supply chains in the OT sector is very complex, involving numerous vendors, partners, and subcontractors. It often requires very detailed knowledge of the inner workings of an OT component to construct a supply chain map.

Addressing supply chain security requires a comprehensive understanding of these relationships, which seems to discourage market players and influencers from addressing the issue. But let’s investigate and identify first how deeply these supply chain attacks have happened. When we analyze previous attacks, it becomes evident that they occur at various (sometimes very granular) levels in the OT system. For instance, threat actors targeted:

• Microchips – In a paper for the IEEE (May 2008), Sally Adee talked about a situation where a Syrian radar system didn’t notice an Israeli airstrike against a nuclear facility being built. This happened because a company that makes computer chips in Europe included a secret “kill switch” in their chips. This switch could be used from remote to disable the radar function.
• Computer Manufacturers: Attacking the manufacturers of commercial-off-the-shelf (COTS) computer equipment used in OT systems can lead to compromised or malicious components being integrated into the final product. Computer motherboards made by Supermicro, and Levono have been altered to allow for backdoors, some of these computers were connected to networks of industrial control systems (ICS).
• Software Development: Supply chain attacks can target the software used in OT systems, such as the operating system, firmware, and application software. Malicious code can be inserted during development or distribution. Recent examples of this were MOVEit and SolarWinds. Apart from attacks at the application level, these attacks can also occur at the operating system level. For example, MS Windows or MacOS desktops. Software is being integrated into operator stations and used for panel operator-to-field operator communication by some of the modern wearable applications.
• Network and Communication: Supply chain attacks can also target the network infrastructure that connects OT system components. A well-known example of this was the attack against Cisco routers using low-cost counterfeit equipment.
• Counterfeit equipment market: There were cases detected in critical industries, like the nuclear industry. Maybe not always a deliberate attack on an installation, products that don’t meet specifications can lead to significant accidents and can also have vulnerabilities that cyber attacks can exploit. Counterfeit products are often cheaper options aiming for profit, but this profit comes at the expense of reduced safety and security.

So, we can conclude that the supply chain threat is real, and we need to be aware of this and prepare a defense. What can we do against the supply chain threat?

• Vendor Assessment: The first thing that comes to mind is a vendor assessment. OT vendors often include 3^rd party equipment in their solutions and project deliveries. Not always at the lowest cost because additional checks and responsibilities translate into a higher price. The danger here is that project teams of integrators, even if it is a project team of the vendor in the role of integrator, might try to reduce project costs by purchasing the equipment locally.
• Contractual Requirements: It is important to include security clauses in vendor contracts that outline security expectations, responsibilities, and consequences for non-compliance. Non-compliance should include a statement on the source of the delivered equipment.
• Secure Update Processes: Both for the operational phase as well as for the project phase establish secure procedures for hardware, software, and firmware updates to prevent unauthorized or compromised updates from being applied.
• Integrity Checks: Implement mechanisms during the factory acceptance test to verify the integrity of the test, and set up processes to check if the components, firmware, and software are intact. This helps spot any unauthorized changes during installation and use. Make sure the integrator confirms this by signing off.
• Vendor Diversity: Avoid overreliance on a single vendor to reduce the impact of a compromise in the supply chain. Having a single vendor offers many benefits but also increases the reliance on this single source. Diversity is a good security mechanism so apply it where it is more helpful than not.
• Supply Chain Monitoring: Establish supply chain monitoring mechanisms through the creation of a supply chain map. In many cases, components in an ICS come from third-party sources that serve multiple manufacturers. For instance, they might provide a standard TCP/IP stack or software for a specific protocol. When problems are found with Vendor A, it’s probable that the same issue could be present with Vendor B and C. Monitoring the supply chain map enhances response capabilities and shifts security processes to a proactive approach. A well-known example of this was the security vulnerability in the HART device type manager written by CodeWright and implemented by many ICS vendors. Initially, the security bulletins focused on a few vendors but in reality, many vendors were impacted, and it took months before all impacted were identified. Additionally, supply chain monitoring encourages vendors to adhere to security best practices and maintain transparency in their operations. Knowing that their activities are being closely watched can incentivize vendors to prioritize security and reduce the likelihood of negligent or malicious behavior.
• Secure Development: This is mainly a vendor task, but asset owners can enforce vendors by demanding insight into their development processes so that the software and components being integrated are developed securely, following best practices for coding and vulnerability management. One method of doing this can be demanding in the request for proposal that the vendor development process complies with the ISAsecure SDLA certification.

So, the supply chain threat is relevant, and we have several instruments to counter it. Maybe not the most popular security topic but certainly a topic we should not ignore. I have tried to stay high level in this blog so there might be gaps in my story.

Sinclair Koelemij

With 45 years of experience in process automation, Sinclair brings a wealth of expertise, with 25 years dedicated to process control and an additional 20 years specializing in networking, security, and risk management for process automation systems. During his extensive 43-year tenure at Honeywell, he played key roles in servicing, engineering, and securing diverse control and process safety solutions, spanning basic and advanced control systems. Sinclair also possesses hands-on experience in software development and the implementation of control and safety solutions for over 100 installations owned by various asset owners, varying in scale from smaller setups with fewer than 1,000 I/O points to large installations exceeding 100,000 I/O points. Furthermore, Sinclair holds several patents in the field of cyber-physical risk evaluation and mitigation..