OT security skills gap is a major challenge for industrial, manufacturing organizations

One of the underlying challenges faced by the industrial cybersecurity industry has been the OT security skills gap, which the industry has tried to overcome over the years. However, the needs of the OT sector for unique skills remain, while the industry copes with the growing challenges emerging from the evolving environment. 

Talent shortage tends to be even greater in the OT (operational technology) security sector, where the onus on cybersecurity professionals is not only to safeguard and understand cybersecurity but also to deal with the security challenges of the archaic and sometimes unsupported legacy process and control systems. Higher wage rates could help bridge the gap over time as people shift from other careers into cybersecurity, but counting on higher wages alone will not necessarily meet the skill and knowledge gaps that prevail. 

Recent industrial cybersecurity attacks, such as the recent SolarWinds supply chain and other cybersecurity incidents, have once again reminded all stakeholders of the OT security skills gap that exists in the sector. 

“The surge in demand for ICS and OT Security skills predates the SolarWinds supply chain and other cybersecurity incidents. This makes sense as the SolarWinds supply chain event had no ICS or OT security elements to it,” Robert Albach, senior product line manager, IoT Security at Cisco told Industrial Cyber. “Since at least 2017, the US Government has put both focus and funding on addressing the lack of cyber security workers with multiple workshops and program grants.”

“The demand for ICS/OT security skills seems to have increased more so due to the multiple ransomware attacks that effect OT environments rather than the SolarWinds incident,” Clint Bodungen, founder, president, and CEO at  ThreatGEN told Industrial Cyber. 

“The demand for OT/ICS security skills has been on the rise for a couple of years. As security regulations and IT-OT-security frameworks are becoming prevalent, organizations require more specialized staff and are making plans to invest in this area,” Daniel Trivellato, VP of OT Technology, Forescout said. “The recent attacks/incidents, alongside nationwide programs like the electric sub-sector 100-days plan and critical infrastructure Executive Order in the US, have accelerated the investment and search for people but also technology and processes.”

“We have definitely seen more focus and demand in the area of OT/ICS cybersecurity skillsets with the continued escalation and proliferation of threats.” Matt Morris, managing director at 1898 & Co., part of Burns & McDonnell, told Industrial Cyber.

“The demand for skills is already high for many years and the number of skilled people is limited.” Jules Vos, head of OT Cyber Security Services at Applied Risk told Industrial Cyber. “So recent incidents have definitely caught attention but not noticeably increased demand for OT security remediation.”

“I don’t think there’s any one particular driver behind the rise in demand for cybersecurity skills of any type—there’s really more of a global maturity across all fronts on the importance of cybersecurity risk management as cyberattacks become faster-moving and more complex,” Ramsey Hajj, Deloitte’s US Cyber Risk Services IoT & ICS (Industrial Controls Systems) leader and a Deloitte Risk & Financial Advisory principal told Industrial Cyber. 

“That said, the gap between supply and demand for talent with ICS and OT skills has been magnified as the need for sound security governance, practices, and operationalization of those practices remain critical to industrial organizations,” he added.

It is clear that a large number of industrial and OT security skills professionals are required by the sector. 

“According to the 2017 / 2018 research conducted on behalf of the White House, the expectation was that the global shortage next year (2022) would be approximately 1.8 million workers needed,” Albach said. “While we do not have specific numbers to report, we have direct experience with customers who have expressed their concern about the ability to find properly skilled personnel. Further retention of existing personnel has become a challenge as high demand and accompanying wage growth is driving more departures,” he added.

Morris explained that there is not a magic number when it comes to OT/ICS cybersecurity personnel. “It truly depends based on the size and threat level of a particular organization, its overall size (# of sites, etc.), and more. I am unsure as to how the market will play out in terms of new hires at the organizational level,” he added. 

“I can share that I expect our team to double in size over the next year,” Morris said.

“Every industrial operator that is at risk of cyber attack to their ICS operations (which is all of them that use TCP/IP network connectivity) should have at least one dedicated, full-time staff member with ICS/OT cybersecurity skills,” Bodungen said. 

“For smaller organizations that staff member should be able to handle the bulk of the responsibilities except for threat monitoring, which should either have another dedicated staff member or be outsourced (this includes cybersecurity program, governance, risk assessment, risk mitigation, etc.),” he added. “For larger organizations (with multiple plants/sites spread across several regions), there should be an ICS/OT skilled staff member dedicated for each region.” 

Bodungen also recommended that a SOC (security operations center) with 24/7 coverage would be a must, whether they are in-house or outsourced to an MSSP. “In-house staff should be trained in incident response but it is also recommended to retain a 3rd party firm as well. Finally, penetration testing and red team exercises from a 3rd party firm is also highly recommended. Assessments, penetration testing, and red team exercises usually require 2 to 3 personnel to staff the assessment/testing/red team on average,” he added.

“In summary, an average of 5 ICS/OT skilled staff are typically required to service the needs for each and every industrial operator at risk,” said ThreatGEN’s Bodungen.

Trivellato told Industrial Cyber that the overall manpower requirement varies a lot by vertical and organization approach to OT security. “Most organizations will require their IT hires to bring or pick up OT security skills, while larger organizations will allocate headcounts for dedicated OT security staff. An additional reason why new hire numbers may be lower than expected is because of the shortage of OT/ICS-focused security experts,” he added.

One of the general perceptions is that the demand for OT security skills can be met by in-house staff or using contractors, which is the trend that the IT services sector has adopted over the years. Both these models have their pros and cons.

There are varying views on the role out-sourced staff will play in the current scenario. “Most of the staff should be in-house with operators to run projects and to manage run-and-maintain phase,” said Applied Risk’s Vos. “Next to that operators/companies shall have design staff to continuously monitor performance of solutions and update solutions as needed. Consultancy companies will have to in-house train people. The number of really skilled contractor staff still is limited. Universities and colleges shall and will focus on delivering qualified OT cyber security engineers,” he added.

“There are not enough in-house staff ready to address the demand,” Albach said. “While many positions simply go without fulfillment, there is an increase in managed services providers stepping in.”  

“There is a current skills shortage in the industry to it will have to be a combination of in-house staff and outsourced contractors,” said ThreatGEN’s Bodungen. “I don’t see that trend changing anytime soon. Even currently, even the large super majors still outsource a portion of ICS/OT work. Especially threat monitoring and incident response.”

The trend that Forescout is seeing is that organizations are starting to promote in-house staff with an OT background to security officers and architects. “These employees already know the OT sites and the organization’s way-of-working and will often collaborate with the IT security teams to implement larger projects,” Trivellato of Forescout said. 

“Other organizations rely more on external contractors to advise and build the ICS/OT security frameworks or run incident response/security operations, but this is often not effective and has proven difficult to maintain. Due to a shortage of in-house skills we still see a lot of outsourcing as it takes time to build OT Security skillsets and resources,” he added.

Because of the shortage of industrial sector ICS/OT cybersecurity professionals, Deloitte has seen leading organizations taking a multi-pronged approach to boost their own teams’ skillsets by building and training teams from inception, such as training all new hires on ICS/OT as part of onboarding, and upskilling or reskilling existing talent, according to Hajj. 

“I can state confidently that there will likely be an uptick, overall, in OT/ICS cybersecurity practitioners as a result of the awareness of risks posed to organizations, but whether that uptick translates to new hire at each respective company, or whether that support comes in the form of services & solutions companies (like 1898 & Co. Security) or managed services providers like 1898 & Co.’s Managed Threat Detection & Response, is not immediately clear,” Morris said.

Given the depth of the ICS and OT domain, the OT security skills set and qualifications that companies look for in their ICS and OT security departments will be varied and diverse. 

Anyone looking to get into a career in ICS/OT should gain a solid cybersecurity base and then take process automation engineering training courses. An ICS/OT cybersecurity specialist is essentially a dual specialist, part cybersecurity professional, and part process automation engineer, according to Bodungen. “For any ICS/OT position, baseline the security skills requirements with the skills required for the equivalent IT security position. Then look for experience working in an ICS/OT environment, even if that experience is volunteer or college lab work,” he added.

“They should have a baseline understanding of how industrial processes work, basic understanding of PLC programming, and understand the differences between ICS and IT in terms of protocols, weaknesses, and the special caveats that go along with the sensitivities of ICS/OT processes and equipment,” Bodungen said. “They should also understand that the main consequences and impact to industrial systems is not informational, it’s kinetic,” he added.

A recent review of engineering curricula at the University of Texas showed that both chemical and mechanical engineering undergraduate programs offered less computer training than government studies, according to Albach. 

“ICS/OT security people preferably should have an OT background to understand OT criticality and the OT landscape diversity,” Vos said. “On top of this OT foundation, cyber security knowledge (NIST-800/IEC62443) and networking/IT systems/solutions knowledge and skills will be added. IT people can also move into OT cyber security but in general, for them, it is more difficult to gain the required OT experience and knowledge,” he added. 

Another fracture that exists in the industrial cybersecurity sector is that currently available manpower has not been trained for meeting the requirements of OT security skills. “Almost every cybersecurity diploma in post-secondary schools (colleges & universities) is focused on IT cybersecurity,” Morris said. “In fact, I’ve only heard of a handful (max) of OT cybersecurity degree programs offered, at least within the US.  I would honestly put the breakdown around 98-99% IT cybersecurity with 1-2% OT cybersecurity globally,” he added.

“Those with an OT background are not familiar with networking and security tools and can be reluctant to big changes in the OT infrastructure or are too busy with non-security tasks,” Trivellato said. “Those with an IT background are often too eager to change and implement new tooling without enough review/testing and acceptance of the local OT staff.”

“As with any technological advancement or digital transformation, continued education for talent needs to keep pace. With reeducation and training, I’m confident forward-thinking IT professionals can help close the skills gap in industrial cybersecurity ICS/OT,” Hajj said.

The OT security skills requirements in OT infrastructures are particularly different from those in IT. “For OT cybersecurity, practitioners generally need a much broader understanding of cybersecurity standards, and depending on the sector they work within, they may also need to be knowledgeable in regulatory standards as well,” Morris said. 

“OT Cybersecurity practitioners also need to have a solid understanding of OT control systems, proprietary vendors and protocols, the differences between OT and IT.  At the end of the day, the OT cyber practitioner has to understand that the goal is to maintain the safety of the people and the reliability/resiliency of the assets versus an end goal of data protection and confidentiality, which is the gold standard for IT environments,” he added.

Deloitte’s Hajj recommends combining all experience in IT security, learning OT security, and then bring the two together. 

Vos recognizes the production challenges that exist in OT environments 24/7. “The diversity of the OT landscape (multi-vendor, multi-generation, multi revision, connection vs isolated, etc), the many different systems used for different OT production purposes (process control, electrical, rotating, etc), and the fact the OT operator/engineers have one single objective: keep the plant running. Any change requires proper preparation. No trial and error,” he added.

Industrial industry leaders that Deloitte works with are seeing ICS and OT security as a top priority, as most expect the rise and frequency of cyberattacks and disruptions to only increase. “The industrials sector is working hard to bolster programs that help defend our critical infrastructure from cyber threats,” Hajj said.

“It really depends on the organization and the CISO or the c-suite and board,” Morris said. “I can definitely say that more of them are becoming aware, but given the level of overall awareness historically, there is still a long way to go!”

“OT Security investment is rising with both IT departments and business groups with OT responsibility making investments,” Albach said. “These investments manifest themselves in architectural foci, security tools expenditures, personnel training, and acquisition. Some of these are for general security practices as well as a growing amount of OT unique security tools.”

Vos agrees that ICS and OT operators are more accepting of the fact that their environments are at greater risk. “However there still is a knowledge gap that causes operators/company management to not fully understand the dynamics and speed of change of the current threat landscape. International regulations (like NIS directives) do help to push company management to speed up taking OT cyber security on board,” he added.

“However, with the recent events in the news regarding cyberattacks and ransomware affecting ICS/OT environments, that trend is slowly starting to shift in favor of more operators becoming more accepting and starting to increase budgets accordingly,” Bodungen said. “However, one trend I have seen over the past 25 years in this industry is that this industry does seem to have a short memory. Complacency has a tendency to set in fast,” he added.

“I wouldn’t say anyone is ‘more accepting’ of cyber risk, as ICS/OT professionals have always had to assume the risk exists,” Hajj said. “However, operators are definitely becoming more attuned to the unavoidable cyber risk management need and are working toward honing their programs accordingly,” he added.

Forescout sees more organizations allocating budgets specifically to IT-OT convergence or OT security projects, as seen in many annual reports. “High fines have to be paid when voiding prevailing security regulation so allocated budgets have risen,” Trivellato said. “Levels of awareness about the risk facing OT Operators are still low however and it is more accepted than OT sites will never be at the newest security and patch levels, instead ICS/OT operators invest more in indirect measures, e.g. standardizing architectures and vendors, segmentation, more frequent auditing, incident response plans,” he concluded.