Cybersecurity Acceptance Testing for Industrial Control and Safety Systems

aeCyberSolutions vice president John Cusimano has more than 20 years of experience in industrial cybersecurity. As an industrial control system cybersecurity and functional safety expert, he’s performed many control system cybersecurity vulnerability and cyber risk assessments in the oil & gas, pipeline, chemical, water/wastewater, pharmaceutical and power industries. He’s also overseen and participated in the security testing and certification of several control and safety systems.

The case for cybersecurity acceptance testing

A number of years ago, Cusimano and the industrial cybersecurity team at aeCyberSolutions realized there was an issue with security testing at industrial organizations. Today’s industrial control systems (ICS) combine traditional automation technologies with digital information technologies that inadvertently expose them to cybersecurity risk. While organizations were increasingly performing cyber risk assessments on their existing control systems,  there was something missing when it came to new systems or major system upgrades.

Five years ago, aeCyberSolutions set out to change the status quo when working with a client who was installing a brand-new integrated control and safety system.

“The CISO for the client said, ‘I want to make sure new control systems are secure before we install them into our environment.’ He noted that while he was trying to secure the existing environment, at the same time, their engineering and capital projects teams were installing new systems without proper cybersecurity design and testing,” Cusimano says. “The way he put it to us was, ‘I want to stop the bleeding. I want to stop installing new systems that aren’t secure.’”

Testing Requirements

To address the CISO’s concerns, aeCyberSolutions developed cybersecurity specifications for the vendors, developed test plans and tested the client’s new system for cybersecurity while it was still in the factory being staged and commissioned, a practice that later became known as Cyber Acceptance Testing. This testing is performed to verify that a new or upgraded ICS is implemented and commissioned according to the security requirements set forth in the design.

“Companies are constantly upgrading and putting in new systems and if they’re not vigilant about cybersecurity in the design and implementation stage it gets even harder to secure the system once it’s installed,” says Cusimano. “Ultimately it’s going to save companies money if they build security in rather than try to do security as an afterthought.”

Cyber-attacks on plant environments are on the rise. However, unlike other environments in need of cybersecurity control measures, industrial settings don’t have the luxury of shutting down for such measures.

“Many of the cybersecurity control measures are fairly intrusive in the sense that some of them may require a plant interruption or shutdown,” says Krish Sridhar, director of industrial cybersecurity for aeCyberSolutions. “So, it’s almost impossible to implement some of these cybersecurity control measures once the plant is up and running. Once it’s commissioned, it’s extremely challenging because these are continuously operating plants.”

Cybersecurity acceptance testing can help address this issue when factored in as part of a project’s lifecycle. Building in cybersecurity is more effective and less costly than trying to retrofit it later because critical security issues are addressed prior to commissioning.

Benefits of Cybersecurity Acceptance Testing

“It’s more cost-effective to implement the controls while you’re in the design and implementation phase. It also ensures that the system as implemented meets the industry regulations and industry best practices,” Sridhar says. “Many of the cybersecurity controls require that you test and deploy them in a safe manner that doesn’t impact operations. Because of that it makes a lot of sense for asset owners to implement the measures before it goes into commissioning and production. From an operations perspective, any unscheduled shutdown or interruptions can be minimized significantly if these controls are implemented at the beginning of the project.”

Without cyber acceptance testing in the early stages of a project’s lifecycle, ICS operators can’t be sure their control and safety systems are cyber secure.

“If you didn’t perform this level of cybersecurity testing of a system prior to startup, it is very possible that you could install a brand-new system that has already been compromised. That has happened,” Cusimano says. “I’ve heard numerous stories from customers who say that they received a brand new control system from their vendor, plugged it into their network, and infected their plant/company because the system they received picked up a virus during fabrication and commissioning.  This is not uncommon because during commissioning people are making a lot of changes, plugging in USB sticks, connecting to the Internet, things you wouldn’t normally do in operations.”

Cyber acceptance testing ensures operations engineering staff have increased awareness of cybersecurity and are better trained prior to startup. The process involves identifying and documenting cybersecurity vulnerabilities found in the system and recommending ways to mitigate those vulnerabilities. This ensures staff are technically better prepared to manage, monitor, and respond to security incidents.

“There are many security features that might have been built into the system–antivirus, backup and recovery, whitelisting, intrusion detection monitoring,” Cusimano says. “During acceptance testing, we run them through their paces and by doing so, with operations personnel there and witnessing, they’re seeing how these controls actually work and how they’re configured. So, it’s definitely a learning exercise for the operations staff as the system is being turned over to them.”

Types of Testing

Today, aeCyberSolutions offers two phases of cyber acceptance testing: Cyber Factory Acceptance Testing (CyberFAT) and Cyber Site Acceptance Testing (CyberSAT). CFAT involves verifying that systems comply with ICS cybersecurity requirements and specifications, which includes checking that the required security settings were configured correctly and that necessary security components like firewalls were installed and properly configured. CSAT involves verifying all punch list items from CFAT are closed as well as testing “delta” modifications in configuration and network interfaces per site requirements.   In some instances, testing may also involve cybersecurity robustness testing or penetration testing designed to discover and identify the weaknesses or vulnerabilities in a system. This type of aggressive testing should not be performed on a production system, but it can be safely performed before the system is operational.

“When we perform these testing exercises we have to be very careful. We can’t get in the way of the people whose primary job is to get the system built and commissioned so we have to be very mindful and make sure that the testing we’re doing does not intrude on their objective of getting the system commissioned and installed,” Cusimano says. ”We make sure security features are operational, all the systems are up to date with the latest patches, system inventory is documented, and that there’s no known vulnerable software installed on the system. One way that saves money is it’s a whole lot easier to install patches and update software before you go live than trying to do that on a live system.”

Rebecca Addison

Industrial Cyber Editor. Rebecca Addison is a former newspaper editor with a decade of experience in the media industry. During her career, she has interviewed world leaders and corporate CEOs, and covered topics ranging from blockchain and CBD to healthcare and education.