Need to step up proactive ICS incident response planning to build defenses against cyber attacks

Increasing cybersecurity threats and attacks drive organizations to structure and build their ICS incident response and develop appropriate plans that address adversarial tactics, techniques, and procedures (TTPs). Often, these hackers engage in extensive research and development (R&D), continuously refine their programs and campaigns, and invest significant effort in R&D. Organizations must also understand how adversaries steal information and gain access to a company’s ICS/OT network and systems, which ultimately progress towards threats that will evolve their disruptive and destructive capabilities in the future.

Asset owners must prioritize ICS-specific cyber incident response, ensure the ability to rapidly disconnect external connections, and consider recommendations for protecting their cyber-physical systems. Such environments are also increasingly prioritizing developing a focused cyber incident response capability for industrial control systems (ICS), focusing on prevention, preparation, planning, incident management, recovery, mitigation, and post-incident analysis.

In a report earlier this year, industrial cybersecurity firm Dragos identified that OT-specific incident response plans are essential for industrial asset owners. Organizations must develop a comprehensive incident response plan with appropriate contact points, skill levels, and next steps for specific scenarios, ensuring effective incident response.

The SANS Institute identified five critical controls for ICS cybersecurity, including an OT incident response plan. This plan establishes collection criteria for incident response, ensuring visibility and monitoring. SANS outlines three steps: scenario selection, consequence-based scenarios, and tabletop exercises to evaluate cybersecurity incident response plans, coordination, capability, resource employment, communication flow, and plan activation.

Industrial Cyber reached out to experts in the industrial cybersecurity sector to chalk out the first steps in setting up an industrial incident response plan. They also point out how this differs from generic enterprise incident response plans.

IR planning for OT does not look drastically different than it does on the enterprise side, Paul Shaver, Mandiant’s global practice leader for ICS/OT security consulting, told Industrial Cyber. 

Shaver identified three key differentiators. “While the computer security incident response team (CSIRT) team will include all the normal responders, there will be additional resources from internal operations and engineering teams, third-party engineering support teams and/or vendor representation, and potentially environmental, health and safety or local emergency response teams if a cyber event leads to a physical consequence,” he added.

“Understanding the interdependencies between IT and OT is critical. There is an increasing number of services that OT systems are dependent on in the IT environment and knowing when and how to sever the connection between the two is critical during an IR,” Shaver added. “Asset inventory in IT is much easier, there are plenty of tools available to identify what is connected to the IT network. This is not so straightforward in OT since many systems are using multiple communication protocols and in a lot of cases gateway devices converting from one communications method to another.

Dana-Megan Rossi, senior director of global incident response services at Dragos, said that an industrial cybersecurity incident response plan is an essential, and often overlooked, first step in any defensible program. “Organizations may begin by adopting an existing IT incident response plan and adding OT elements. This is a great start to create a framework that incorporates industrial cyber into existing processes that are familiar to the organization, follow the established incident management protocols, and account for events that may traverse IT/OT boundaries,” she added.

“Key areas of differentiation need to be captured, including physical, environmental, and safety considerations, identification and inclusion of systems and site champions, and regulatory compliance requirements for the sites when investigating a potential event,” Rossi told Industrial Cyber. “Additionally, having a communication-coordination framework for the interplay between IT and OT security teams and stakeholders will help facilitate a proper cadence for teams throughout the incident lifecycle. And no plan is complete without testing.” 

She also highlighted the need to conduct regular tactical and executive tabletop exercises is how good response frameworks become operational.

Setting up an industrial incident response should begin with understanding the specifics of the industrial system in question, Clint Bodungen, founder, president and CEO at ThreatGEN, told Industrial Cyber. “I know that sounds obvious, but I see far too many organizations treating their ICS IR plans the same as their enterprise IR plan without regard to the specific differences, considerations, and cautions that come with industrial systems.” 

Bodungen added that, unlike a generic enterprise incident response plan, industrial plans must take into account real-time processes and safety requirements of industrial environments, where downtime or disruption can have significant consequences. “Additionally, It’s essential to engage with HSE staff, because the IR Plan should be an extension of the existing production outage and safety protocols and emergency management plan.”

The executives also discussed the frequency at which organizations test their ICS incident response plan, as well as the preferred method for testing organizational response.

Shaver stated that it is encouraging to see more organizations regularly testing plans and playbooks, as well as a growing trend of continuous improvement. “While testing is very similar to conducting enterprise level tabletop exercises, or purple team exercises, we recommend layering in the added complexity of an environment where the procedure to investigate, remediate, and return operations to normal may have many additional factors,” he added. 

“The tools we have come to depend on in enterprise IR, such as EDR, are typically not available in OT IR so the process needs planning, testing, and refining,” Shaver highlighted. “Often when the OT process is down, so is the revenue stream. The pressure to get back operational is high – the decision-making process for how that happens needs to be clearly defined, with contingencies and practiced. We also see organizations exercising cyber security incident response as part of their existing OT disaster recovery and business continuity plans,” he added.

Tabletop exercises should include executive and technical level scenarios to ensure all roles and responsibilities are tested, Shaver said. “Carefully planned purple team exercises can test detection capabilities and help defenders build confidence in the process.”

Rossi stated that organizations have varying schedules for testing their plans. “At a minimum, an annual executive tabletop exercise should be part of an organization’s operational rhythm. These exercises are strategic and bring together senior executives on the key decision-making for business prioritization.” 

“Key areas to test for include the incident management structure and how that structure may integrate into existing incident and crisis management strategies,” according to Dragos’ Rossi. 

“Tactical exercises should be carried out on a regular basis in the form of drills, tabletops, and hybrid events,” Rossi said. “The tactical exercises help build muscle memory for the security teams and their colleagues across the organization to identify, escalate and prioritize industrial events. Gamifying these exercises can make them fun and generate KPIs to measure against as the organization matures.”

Organizations are typically told they should test their incident response plan annually, Bodungen said. “As a result, that’s what many organizations do. Many of them only do it because it’s a requirement.” 

“However, frequency is key if you want to get any real value out of it,” Bodungen pointed out. “How else are you going to track gaps, measure improvement, and actually work to improve? Conducting tabletop exercises provide a risk-free environment to evaluate procedures and train personnel on a more frequent basis. Then, functional exercises are where teams actually practice performing their duties in a simulated incident. This is the part that should be done annually.”

The executives also shed light on how ICS incident response and planning has evolved due to increased RaaS, APT, and nation-state sponsored attacks in the industrial cybersecurity sector over the last 18 months.

“The evolution that we have seen comes in the form of more organizations proactively building IR capabilities for their OT environments and testing these capabilities,” Mandiant’s Shaver said. “Understanding the threat landscape is crucial to updating/maintaining playbooks and testing response plans. We have seen an increase in the number of organizations leveraging cyber physical (ICS/OT) focused threat intel feeds and becoming proactive about performing architecture assessments and threat modeling for these environments.”

Rossi confirmed that there has been an uptick in organizations looking for OT-specific incident response plans, and ensuring the OT and IT security teams, executive decision-makers and their cross-functional colleagues are working in unison to address the risks we see today. “Many of these organizations have a dedicated ransomware playbook for industrial cybersecurity that goes into specific use cases and the nuance that is presented by these types of events. This includes understanding the business impact and third-party engagement necessary for responding to ransomware,” she added. 

“These third-party engagements include how, when and under what circumstances the organization will engage on potential insurance claims and working with outside counsel and forensic firms,” according to Rossi. “Considerations include thresholds for reserving notice to file an insurance claim, chain of custody procedures during a forensic review, communication and coordination with governments, and complying with all financial regulation laws and compliance requirements.”

Bodungen said that due to the increase in ransomware affecting industrial environments (mostly indirectly) RaaS (Ransomware as a Service) and APT (Advanced Persistent Threats), “I have seen more activity and attention from ICS organization towards incident response planning. They really are starting to think more proactively.”

He also added that due to these threats evolving so quickly, especially now with the potential for large language models (LLM) entering the picture with ‘WormGPT’ IR preparedness needs to become more threat-intelligence driven to focus IR plans and training on recent threats and TTPs. This is another reason why the frequency of training is so important. 

The executives analyze the role that third-party expertise plays in industrial organizations, specifically concerning incident response and planning for ICS environments. They also examine the typical services available to organizations after an incident, as well as the main lessons learned in terms of response, recovery, and learning.

“We still live in a world where 99% of the incidents we are responding to start with an enterprise-side compromise and the effect on OT is secondary. Strong IR capabilities on both the enterprise and OT sides are critical in supporting the forensic investigation process,” Shaver disclosed. “Third-party organizations, such as Mandiant, are responding to new incidents daily and get to see the threat landscape change in real-time. Having teams with that experience brings a wealth of knowledge to building effective cyber defense capability.” 

Shaver added that IR teams with hands-on experience in OT environments that understand the critical nature of these systems and the nuances of investigating and remediating improve the overall rate at which normal operations can be restored. “Post-incident the results of the investigation process should include key recommendations on shoring up attack vectors, improving defenses, and improving IR capability.” 

According to Shaver, “services that support optimizing cyber defenses, improving crisis communication processes, validating the remediation actions, and/or evaluating new tools/technologies are just a short list of projects we see customers undertaking after an incident.”

“Third-party expertise should be an extension of your team before, during, and after an incident response,” Rossi said. “We all know 95% of response is preparedness, and this holds true when working with industrial cybersecurity experts.”

“Having the ability to understand the organization, security architecture and systems, culture, preferred communication tempo, tooling and collection capabilities, and priority thresholds is paramount for great incident response,” according to Rossi. “Organizations that have built defensible programs will engage experts early and make them an extension of their team so there is a seamless integration during an event. Experts can shave valuable time off investigations when minutes mean millions of dollars by having proactive planning and engagement with organizations.”

She added that integrating the third party into the exercises, design planning, and incident response workshops proves an invaluable experience for the organization in maximizing the value they get from third parties.

“After an incident response, capturing lessons learned and acting on these key takeaways will be a true measure of maturity for the organization,” Rossi observed. “Success is guided by how we adapt and prepare by taking these invaluable lessons, sharing with tactical and strategic teams, and importantly, making them actionable to propel the organization forward with operations and exercises based on these improvements that can be measured.”

Bodungen said that third-party expertise is actually essential. “Sometimes it’s hard to see your own faults. It’s sort of like proofreading your own writing. It can be hard to catch your own typos when reading your own work. Outside expertise can watch you perform and review your plan from an objective point of view. They can also provide valuable insight from experience and lessons learned from other customers.” 

He added that third-party providers can offer services such as facilitating both tabletop and functional exercises, acting as an instructor as well, and offering guidance and feedback throughout the process. 

“The main lessons and outcomes should be the proper communication channels, how and when to take certain actions to maintain safety and security, understanding when to declare an event an incident, and probably most importantly, be so familiar with the process that when an incident does occur, staff can perform under pressure without panicking and reduce critical errors,” according to ThreatGEN’s Bodungen. “If you’re going by ‘the book,’ it’s to learn how to identify threats, contain the incident, eradicate the threat, and recover. Finally, every exercise should start with a list of objectives and end with a list of lessons learned, feedback, and areas for improvement.”

The executives assess the effect that regulations, directives, and executive orders had on the ICS incident response and planning of OT and critical infrastructure environments.

“For many organizations, the initial response to GRC requirements is some kind of assessment or analysis to determine if they are compliant and create a roadmap to compliance if necessary,” Shaver said. “Organizations with a mature, or maturing, security program typically follow up with a series of regular assessments and a trajectory of continual improvement in their ability to defend and respond. While less mature organizations are more likely to take a closer look at improving network segmentation, building asset inventories, and getting a better unstinting of their environments so they can build a roadmap for effective, yet manageable, improvement.” 

He added that Mandiant has also seen standard organizations such as the International Society of Automation (ISA) working to establish frameworks and standards focused on improving defense in depth for ICS/OT, Including ISA launching cybersecurity first responder credentialing program, Incident Command System for Industrial Control Systems (ICS4ICS) built on FEMA’s Incident Command System framework.

Rossi said that there have been a myriad of emerging laws and policies across the world aiming to reduce risk, but ultimately, they can add complexity without harmonization. “Working with organizations to have critical controls in place is the best planning and compliance tool to be applied to meet these requirements.”

“A meaningful step forward we see in industrial cybersecurity is the increasing frequency and importance given to cybersecurity at the board level,” according to Rossi. “There is an informal correlation between the introduction of the US Security and Exchange Commission’s rules for cybersecurity reporting and the increase in Board of Director cybersecurity exercises specifically aimed at OT cyber risk strategies and decision-making.”

Rossi added that industrial cybersecurity events are prioritized based on the safety of people first and foremost, environmental impact, and then business impact, including supply chain and contractual and regulatory risks. “This prioritization creates a familiar paradigm for decision-making when senior executives are exercising, as compared to more traditionally technology-driven enterprise IT-focused exercises.” 

She added that practicing and defining the thresholds for the business impact and external communications and investor relations implications for cybersecurity events can have a profoundly positive experience for boards and their understanding of cybersecurity at large.

Bodungen said that regulations, directives, and executive orders seem to have gotten people thinking about IR planning again, which is a good thing. “Unfortunately, that seems to be the main catalyst for any sort of movement in cybersecurity. That or actual incidents with global exposure. 

However, that pretty much only promotes the ‘annual tabletop’ fallacy, according to Bodungen. “In order to have a truly effective incident management program, it needs to be part of your organizational culture. And as I’ve said before, it needs to be an integrated part of your operations emergency management plan,” he concluded.