Unraveling Security Engineering -Product vs. Systems in Critical Infrastructure

We’ve seen a lot of discussion lately on the topic of “Secure by Design” as well as Security Engineering as an emerging (but not new) domain of focus in securing critical infrastructure. But the process of security engineering is frequently misunderstood depending on whether we are talking about secure product development or systems integration and the implementation of complex systems of systems.

Product Security Engineering

Product security engineering focuses on building secure products from the ground up, integrating security measures throughout the lifecycle of a product. The goal is to create software or hardware solutions that can protect critical systems from both internal and external threats.

Product development is typically performed through a process driven by product vendors in coordination with their end users, or due to market research. In most cases, we are talking about an asset owner-operator such as a manufacturing plant, utility, or other critical infrastructure operator. This research includes market and feature viability analysis to ensure that the product will be profitable, including any necessary security features to be included. If these features significantly increase cost, will the product still be competitive? Will these same features create friction for the user, and ultimately impact buying decisions? Additionally, the use cases that go into development are narrowly focused on the product the vendor is bringing to market. Similarly, their threat modeling approach is likely based on their understanding of how that product will be used.

When considering automation solutions, much of the product’s logic is driven by automation engineering downstream from the product development process. In many cases, the product vendor may not have full visibility into how their products support operational processes or even their end customers’ critical functions.

Secure product development establishes the foundational building blocks for complex systems of systems through several activities.

Application Security

Application security establishes the primary line of defense in product development. It involves integrating security measures within the application software throughout its lifecycle. This process starts from the initial design phase and continues into the development, deployment, and maintenance stages. Product developers must identify potential vulnerabilities that could be exploited and implement protective measures such as input validation, session management, and error handling to mitigate them. This is primarily where initiatives such as the Secure Software Development Framework from NIST in the form of SP 800-218 and supporting Executive Orders have been focused, with additional insights from software transparency efforts such as software bill of materials.

Secure by Design Principles

Incorporating ‘Secure by Design principles in product development is a proactive approach toward securing software. It involves building security into the software from the beginning, rather than as an afterthought. This method can significantly reduce vulnerabilities and improve the software’s resilience to attacks. Developers adhere to certain design principles like least privilege, defense in depth, and fail-safe defaults, attempting to ensure that the product is inherently secure. Good practices from application security are part of this, but additional engineering concepts come into play such as those described in Cyber Informed Engineering such as design simplification, planned resilience, layered defense and as we will discuss next, cyber secure supply chains.

Software Supply Chain Security

The software supply chain has emerged as a critical area of focus for product developers, given its potential to become a conduit for cyber threats as a force multiplier for the adversary. Ensuring security within the supply chain involves scrutinizing each stage — from sourcing components, development and build pipelines to product delivery for potential security risks and capturing this information in the form of attestations that convey transparency to end users and other stakeholders. This process includes validating third-party components, monitoring for suspicious activity, and regularly updating software to patch vulnerabilities, and ultimately communicating these results downstream.

Threat Modeling

Threat modeling is another key activity in product development. It involves identifying potential threats to the hardware or software, and implicitly to the mission of the end-user, determining their severity, and formulating strategies to mitigate them. It’s a proactive approach to understanding the potential attack vectors and providing a clear roadmap for securing the product against these threats. Adam Shostack, a noted threat modeling expert, describes the simplified version of this process by asking 4 questions:

1. What are we working on? 
2. What can go wrong? 
3. What are we going to do about it?
4. Did we do a good job? 

At the end of this process, a product is ultimately delivered and a PSIRT or product security incident response team, takes over the ongoing operations and maintenance of the product, including incident response and vulnerability monitoring. There are several inputs into this process, both from a customer-facing standpoint, as well as ingesting threat and vulnerability information produced by the security community.

Systems Integration

Systems integration, on the other hand, is about bringing together disparate subsystems into a single, harmonized system. In terms of security engineering, it involves integrating various products and solutions into the existing infrastructure in a seamless and efficient manner. These products might be sourced from many different product vendors, and they may include both existing legacy systems, IT infrastructure and non-OT components such as ERP systems, as well as the new products to be integrated.

The process involves understanding the architecture of the existing system, planning the integration of new security measures, configuring, and implementing these measures, and finally, testing the integrated system. The goal is to ensure that all systems work together to support a reliable critical function, without causing any disruptions to the overall functionality of the infrastructure. In most cases, security remains an aspirational byproduct and not part of the core activities. More and more we are seeing an increased focus on security, but this is not typically the core objective for the integrator.

A critical aspect of systems integration is compatibility. Every piece of hardware or software has its own unique specifications and requirements. Therefore, integrating them into a cohesive unit can often present significant challenges, requiring a deep understanding of each system involved and how they interact with each other.

So, given all this complexity, why do we place so much focus on the product security lifecycle? Why do we sometimes rely on the services of very small engineering firms without the knowledge and staffing to deliver on highly complex projects requiring deep security subject matter expertise? Many of these firms are highly specialized in their area of focus and are valuable stakeholders for your engineering project but expecting them to properly manage the entire project using ad-hoc deployment processes, poorly defined requirements, and a lack of understanding of the security implications for all these moving parts is perhaps a tad unreasonable. This is further exacerbated by a lack of understanding of cybersecurity best practices and how the evolving threat environment is changing how we should be handling security challenges.

The Interplay: Balancing Development and Integration

While product development and systems integration are distinct processes, they must work in harmony within the realm of security engineering. The products developed need to be integrable with the existing infrastructure, and the systems integrator needs to understand the functionality of these products thoroughly.

In terms of security engineering in critical infrastructure, one of the biggest challenges is the integration of new security products into legacy systems. These systems, often designed and built before modern cyber threats emerged, weren’t designed with contemporary security considerations in mind. Hence, ensuring that new products are compatible with these systems, and can enhance their security without disrupting their functionality, is a significant task.

Moreover, the constantly evolving nature of cyber threats means that both product developers and systems integrators need to remain agile. They must continually update and improve their understanding to keep up with the latest threats and vulnerabilities. In fact, some asset owners are coming around to the idea that the lifespan of their systems may be much shorter than traditionally applied, as their risk posture changes significantly over time. This necessitates a continuous security optimization approach that is changing how engineering looks at system design.

Systems integration involves the harmonization of various subsystems into a single, cohesive system. This process is fundamental in ensuring that all systems work together, providing robust security without causing any disruptions to the overall functionality of the infrastructure. To understand this intricate process better, it is crucial to examine the roles of key stakeholders involved in systems integration, such as Engineering, Procurement, and Construction (EPC) companies, Main Automation Contractors (MAC), and Original Equipment Manufacturers (OEM). With so many stakeholders, and so few trained in cybersecurity, we have some big challenges facing us.

EPC companies play a crucial role in the systems integration process. They oversee the entire project, from design and procurement to construction and commissioning of the system. Given their comprehensive involvement, EPCs are responsible for ensuring that all the systems and subsystems are integrated securely and effectively. EPC companies coordinate with asset owners, product developers, OEMs, and integrators to ensure that the security products are compatible with the existing system. They ensure that the integration of these products doesn’t disrupt the system’s operation and that they enhance the overall security posture of the infrastructure. Sometimes design is performed by E&A (Engineering and Architecture) firms that specialize in this work, but do not oversee actual construction and commissioning. This too, creates another point of collaboration, and just like the Telephone game we learned as children, opportunities for miscommunication.

MACs are typically responsible for all automation-related aspects of a project. They have a deep understanding of the system’s operational requirements and are instrumental in ensuring that all automation components, including security products, are integrated seamlessly into the system. They work closely with OEMs and integrators to ensure that the security measures implemented align with the system’s operational requirements. They also collaborate with EPCs to ensure that the security products are integrated in a way that complements the design and construction of the infrastructure and also perform quality assurance of the functionality of the automation scope they were responsible for.

OEMs are the manufacturers of the products being integrated into the system. They provide the technical specifications and support necessary for the integration of their products. OEMs work closely with integrators, providing them with the information needed to effectively integrate their products into the system. They also collaborate with EPCs and MACs, ensuring that their products align with the project’s design, construction, and automation requirements. They continuously update their products to address emerging threats, working in sync with the other stakeholders to maintain the system’s security posture.

But unfortunately, many times this collaboration only occurs on paper, as the Integrators are ill-equipped to handle these conversations, and if the requirements were not specified by the end user, or are part of the integrator’s standard process, they do not happen. This is where working with security-conscious engineering firms and integrators, such as those that deliver Cyber Informed Engineering services or those leveraging software tools that help support a secure engineering workflow becomes incredibly important. In fact, this lack of security culture ingrained into many engineering processes is likely the root of much of the failure seen in the field today.

The Importance of Requirements and Specifications Management

Asset owners play a vital role in secure engineering. As the primary stakeholders responsible for the operation and maintenance of the critical infrastructure, they must ensure that the necessary security measures are in place and functioning correctly. The risk associated with the assets they own and operate is their risk. They are ultimately accountable for the downstream impacts to consumers of their service, regardless of whose fault it is.

Asset owners are responsible for defining the security requirements for the infrastructure based on its operational needs and the potential threats it faces. They should work closely with product developers, integrators, EPC companies, and MACs to ensure these requirements are met, but this is not always the case. They are also responsible for managing the lifecycle of the integrated systems and security measures implemented. This includes ensuring that products are updated to address new threats, that they continue to function as intended, and that any issues are resolved promptly.

Furthermore, asset owners often have the final say on engineering decisions and the approach to systems integration, as they are ultimately responsible for the infrastructure’s security and bear the risk if a security incident occurs. But far too often, they simply accept the guidance of third parties such as the stakeholders mentioned above, trusting that they know best. This is likely one of the greatest potential risk opportunities for the operator, with so many assumptions made that are never validated.

Cybersecurity consultants can also assist in this process to provide expert advice on securing critical infrastructure. They can assist asset owners in defining their security requirements, choosing the right security products, and planning the systems integration process. They also help in identifying potential threats and vulnerabilities and developing strategies to mitigate them.

It’s also incredibly vital to consider the actual users of the system. End-users are important stakeholders in secure engineering. They are the ones who interact with the system daily. Their feedback can provide valuable insights into the system’s security, and their adherence to security protocols can greatly affect the system’s overall security posture. Furthermore, an understanding of their process may lead to insights that can be used to engineer out failure. It is quite possible that a secure system operated insecurely may lead to catastrophic failure, and it is only through a holistic understanding of the people, process, and technology in play that we can execute on safe, secure, and reliable outcomes.

All of this is to say, that many of the design decisions that lead to security incidents are the result of a failure to properly define requirements and include relevant stakeholders in the process. Assumptions are made by project owners, whether it be an engineer or a line of business owner, or even a project manager, but without relevant inputs, these design decisions ultimately result in insecure systems being delivered. Secure products can help, but they are only part of the puzzle, and a shared responsibility model needs to be established between all relevant stakeholders to ensure that the asset owner operators’ objectives are met.