Water sector set to ramp up PWS cybersecurity using sanitary surveys, yet gaps are expected to persist

The U.S. Environmental Protection Agency (EPA) rolled out last month a memorandum that calls for the evaluation of the cybersecurity of OT (operational technology) systems used by public water systems (PWSs), when conducting sanitary surveys or through other state programs. The memorandum explains various approaches to include cybersecurity in PWS sanitary surveys or other state programs to identify cybersecurity deficiencies as part of periodic sanitary surveys. 

These sanitary surveys must work on identifying significant deficiencies and PWSs must then correct those significant deficiencies, including cybersecurity-related significant deficiencies. For cybersecurity, significant deficiencies should include the absence of practice or control, or the presence of a vulnerability that has a high risk of being exploited, either directly or indirectly, to compromise an OT used in the treatment or distribution of drinking water. 

The EPA is also offering significant technical assistance and support to states in this effort as well as to PWSs in helping to close cybersecurity gaps. 

The sanitary surveys must include an evaluation of systems and technologies including source, treatment, distribution system, finished water storage, pumps, pump facilities, and controls, monitoring, reporting, and data verification, system management and operation, and operator compliance with state requirements. 

The sanitary survey of a PWS must assess if the PWS uses an ICS (industrial control system) or other OT as part of the equipment or operation of any required component of the sanitary survey. Then, the state must evaluate the adequacy of the cybersecurity of that OT for producing and distributing safe drinking water. Additionally, if the state determines that a cybersecurity deficiency identified during a sanitary survey is significant, then the state must use its authority to require the PWS to address the significant deficiency.

Water associations have, however, requested the revocation of the agency’s interpretive memorandum which added cybersecurity reviews to PWS sanitary surveys. In a letter addressed to the EPA, the American Water Works Association (AWWA), the Association of Metropolitan Water Agencies (AMWA), the National Association of Water Companies, and the US Conference of Mayors called upon the EPA to withdraw the Rule – that requires states to examine cybersecurity practices and controls at PWS while conducting sanitary surveys and collect data from PWS regarding the same – until such time that it has fulfilled the statutory obligations defined by the Paperwork Reduction Act (PRA) and regulations controlling paperwork burdens on the public. 

The agencies wrote that they “do not believe that the expiration of the current ICR satisfies any of the criteria that may justify emergency processing by OMB. There is no reasonable explanation for why the agency would be unable to comply with the normal clearance procedures for the existing ICR. The addition of new information collection obligations created by the Rule do not justify emergency processing either since this action has been under consideration by the agency since 2021.” 

The letter also points out that there has been no consultation or collaboration with the organizations representing PWSs regarding considerations that might ‘minimize the burden.’ Furthermore, given the absence of an OMB-approved ICR for the information required under the new Rule, state primacy agencies are not authorized to collect this information and PWSs are not required to respond to requests for information. 

The agencies call for the EPA to withdraw the Rule until such time that it has fulfilled the statutory obligations defined by the PRA and regulations controlling paperwork burdens on the public. “In addition, OMB should provide a statement clarifying that any information collected to support compliance with the Rule would constitute a violation of the PRA and regulations controlling paperwork burdens on the public,” the letter added. “We have further concerns with the lack of legally-required procedures followed before issuing the Rule which we look forward to addressing with both the agency and OMB.” 

Industrial Cyber asked cybersecurity experts how big of a game changer the EPA Memorandum will be when it comes to tackling cybersecurity at PWSs. They also examine the role that the guidance is likely to have in tackling cybersecurity at PWSs.

“AMWA agrees that all water and wastewater systems should be held to a high standard on cybersecurity. That’s why AMWA is working to promote expanded participation in WaterISAC, which provides water systems with critical information about cyber threats and best practices to protect their systems,” Tom Dobbins, CEO of the AMWA, told Industrial Cyber. “We are also eager to work with lawmakers on Capitol Hill to develop a comprehensive program of water system cybersecurity.” 

That being said, “we do not anticipate EPA’s March 3 guidance will do the job. Adding cybersecurity reviews to existing Public Water System Sanitary Surveys will charge state inspectors without cybersecurity expertise with evaluating systems’ preparedness, and could put sensitive information at risk,” Dobbins added. 

Jennifer Loudon, founder/CEO at Intelligent Water Services thinks that the EPA Memo lays a solid foundation for utilities moving into the future, and in the water industry, a federally created foundation can indeed be a game changer, pointing to the importance of water quality regulations dating back to the Clean Water Act. “The guidance gives a structure (the surveys) in which to address the topic of cybersecurity, something that can be an overwhelming ‘black box’ for utilities who have never considered the importance of cybersecurity before. It’s a detailed Step 1 in an industry used to working in terms of Standard Operating Procedures,” she added.

The Environmental Protection Agency’s memorandum on ‘Addressing PWS Cybersecurity in Sanitary Surveys or an Alternate Process’ is a ‘guide’ for PWS to evaluate and improve their cybersecurity posture, Richard Robinson, chief executive officer of Cynalytica, told Industrial Cyber. “The memorandum does provide some useful guidance and recommendations for PWS to identify, assess, and manage cybersecurity risks in their operations, including conducting vulnerability assessments, developing incident response plans, and implementing cybersecurity controls.”

One of the strengths of this memorandum is that it acknowledges the unique challenges and limitations that PWS is confronted with in implementing cybersecurity measures, such as limited financial resources or technical expertise, according to Robinson. The memorandum does provide practical recommendations that are tailored to the specific needs and resources of PWS, such as prioritizing critical assets and systems, and leveraging existing resources and partnerships, he added.

“However, there are significant limitations and challenges associated with the memorandum. For example, while the guidance provides detailed instructions for conducting vulnerability assessments and implementing cybersecurity controls, it does not provide sufficient guidance for PWS on how to prioritize and allocate resources to address the most critical risks,” Robinson points out. “Additionally, the guidance does not sufficiently address the rapidly evolving cybersecurity threat landscape and may not be comprehensive enough to address all potential risks and vulnerabilities.”

There are a few deficiencies in the memorandum, Robinson said. These include lack of specificity, insufficient guidance for small PWS, and lack of clarity on compliance requirements. “While the memorandum provides general guidance on how public water systems (PWS) can improve their cybersecurity posture, it does not provide enough detail on how to implement specific controls or address specific vulnerabilities,” he added.

“The memorandum does not provide enough guidance for small PWS that may have severely limited resources or technical expertise to implement cybersecurity measures,” according to Robinson. “While the memorandum provides recommendations for PWS to improve their cybersecurity posture, it is not entirely clear whether compliance with these recommendations is mandatory or voluntary, I will guess mostly voluntary.”

The memorandum mainly focuses on conducting vulnerability assessments, developing incident response plans, and implementing cybersecurity controls, Robinson added. “However, there are other substantive and more impactful areas of cybersecurity, such as ICS/SCADA physical communications monitoring (IP and non-IP), asset and inventory management, threat intelligence, or network segmentation, that should also be of critical importance for PWS to consider.”

The experts also investigate the potential problems that operators at these facilities may face as they conduct necessary assessments of the cybersecurity of OT systems at PWSs using sanitary surveys or alternative state initiatives.

“The guidance makes it difficult to tell whether a particular item viewed as a cybersecurity shortcoming will lead to a finding of a ‘significant deficiency’ under the Safe Drinking Water Act,” Dobbins highlights. “For example, it may be a best practice for all water systems to ‘offer regular opportunities to strengthen communication and coordinate between OT and IT personnel,’ but the lack of such regular activities at a particular utility does not necessarily mean that the system should fail its inspection. Much discretion is left to states, so it is hard for utilities to determine where they should prioritize resources.”

Loudon said that assessments can be intimidating, especially if one is uncertain of how their utility will score. “Just making the leap, being honest with yourself about the state of things, and accepting the results can be challenging. Then, there’s the daunting task of knowing what’s wrong but not knowing what to do about it.”  

Pointing out that water workers are problem solvers by nature, Loudon added that some of the issues brought to light in the assessment have easy/cheap fixes, but many do not. “To compensate for these challenges, utility operators and administrators need to look at these assessments as a way to ultimately reduce risk. Accepting that every utility has some degree of risk and focusing on doing what’s needed to reduce that over time, are going to be key in implementing the assessments,” she added.

Robinson said that going forward there are several challenges that PWSs will have, and ways the EPA’s approach could be improved in addressing cybersecurity concerns for PWSs. These include clearer guidance on cybersecurity requirements, mandatory cybersecurity assessments, increased technical expertise, more robust incident response planning, increased funding, and resources, requiring the monitoring of ICS/SCADA (serial, analog, and IP) communications, and requiring asset and inventory management of ICS/SCADA systems. He also points out that the memorandum does not provide a specific definition of what it means for a PWS to be ‘compliant’ with the guidance.

“The memorandum could provide more explicit guidance on the specific cybersecurity requirements for Public Water Systems. This could include details on which systems should be protected, the types of cybersecurity controls that should be in place, and how to effectively implement these controls,” according to Robinson. “The memorandum could make cybersecurity assessments mandatory for all Public Water Systems, rather than just recommended. This would ensure that all systems are regularly assessed for cybersecurity risks and vulnerabilities, and appropriate action is taken to mitigate them.”

He added that the memorandum could encourage PWSs to hire or contract with cybersecurity experts to help identify and address cybersecurity risks. This would help ensure that PWSs have the necessary technical expertise to effectively implement cybersecurity controls and respond to incidents.

“The memorandum could provide guidance on developing and implementing incident response plans that are tailored to the unique needs of Public Water Systems. This could include details on incident detection and reporting, incident response roles and responsibilities, and steps to be taken to minimize damage and restore normal operations following an incident,” Robinson said. “The memorandum could encourage increased funding and resources to support Public Water Systems in their efforts to improve cybersecurity resilience. This could include federal grants and other funding opportunities, as well as access to technical expertise and training resources.”

Requiring the monitoring of ICS/SCADA (serial, analog, and IP) communications could improve the EPA’s memorandum in addressing cybersecurity concerns for PWSs in several ways, Robinson added. “Requiring asset and inventory management of ICS/SCADA systems can significantly improve addressing cyber security concerns for Public Water Systems. By keeping a record of all the assets and inventory related to the ICS/SCADA systems, Public Water Systems can better understand their network and system topology, including potential vulnerabilities and risks. This information can help in the identification of critical assets and systems that require more protection and monitoring.”

Lastly, the memorandum does not provide a specific definition of what it means for a PWS to be compliant with the guidance, according to Robinson. “Instead, the memorandum provides guidance on how Public Water Systems can evaluate their cybersecurity posture and identify areas for improvement. It is ultimately up to each Public Water System to determine what actions they need to take to improve their cybersecurity resilience. In the long run, this will not be adequate to address the challenges that PWS currently faces or will continue to face,” he added.

The experts assess whether the EPA’s measures to provide guidelines and training sessions in 2023 are sufficient to assess cybersecurity in sanitary surveys. They also evaluate the value of such initiatives and how much of a difference they will make in improving the cybersecurity posture at these PWSs. 

“State inspectors tasked with completing Public Water System Sanitary Surveys are trained in the operation of public water systems,” Dobbins pointed out. “They are not necessarily experts at cybersecurity and preparedness, particularly as it relates to industrial control systems. AMWA is skeptical that the training opportunities offered by EPA this year will be sufficient to equip these state inspectors to take on this new role.”

Robinson underscores that PWSs need increased technical expertise and increased funding and resources. “And without Clearer Guidance and Mandatory requirements such as monitoring and asset and inventory management this will be less than optimal effort,” he added.

“Any continuing education initiative from the EPA is much appreciated,” Loudon said. “In a perfect world, financial and human capital wouldn’t be limiting factors, but utilities are all too familiar with this not being a perfect world. We have to continually do more with less, and guidance and training from the EPA can make the difference between initial success and failure for some utilities. Again, having that defined federal standard as at least a starting point is critical,” she added.

The memorandum could encourage PWSs to hire or contract with cybersecurity experts to help identify and address cybersecurity risks, Robinson said. “This would help ensure that Public Water Systems have the necessary technical expertise to effectively implement cybersecurity controls and respond to incidents.”

He added that the memorandum could encourage increased funding and resources to support PWSs in their efforts to improve cybersecurity resilience. “This could include federal grants and other funding opportunities, as well as access to technical expertise and training resources.”

The experts analyze how equipped are the smaller PWSs when it comes to assessing the cybersecurity of the OT systems being employed in their setting. They determine the inherent challenges that these utilities face when it comes to strengthening their cybersecurity posture.

Loudon said that the vast majority of utilities, both water and wastewater, around the country, are considered small or midsize. “These facilities are subject to the same problems that large utilities face, but with a traditionally much smaller pool of resources to draw from when addressing the problem- including OT cybersecurity. Staffing, in terms of both sheer numbers as well as in-house expertise, is going to be lower at these small and medium facilities,” she added.  

“Larger utilities usually have access to more financial resources and connections to talent trains to boost human capital. Smaller and medium utilities need to work their connections more and get creative wherever possible to make up for the smaller resource availability,” according to Loudon. “Collaboration between utilities, contracts with outside vendors, and public-private partnerships can go a long way to helping even the playing field.  Many small facilities don’t even have an in-house IT department, nevermind cybersecurity experts on staff. Once these creative strategies to acquire resources are implemented at the utility, communication between the groups is going to be the key to their ultimate success.” 

She also added that it’s common to think that small to medium sized utilities have the least to be concerned about because they are not likely to be on an attacker’s radar, “but I would argue that they have the most to be concerned about because they are the most susceptible to upset.  A larger utility can potentially absorb/counter a cyberattack through redundant means, but even small problems at a small utility can be detrimental.”

Pointing out that her biggest fear, regardless of the size of the utility, is that the recent release of new information regarding the Oldsmar incident in 2021 will send utility personnel back into a false sense of security. “Make no mistake, water and wastewater utilities of all sizes are still very much on the radar of those who wish to cause disruptions through cyberattacks. We cannot go back to burying our heads in the sand,” Loudon added.

“This isn’t a one size fits all answer. But what we have seen is most small PWSs are ill-equipped to assess the cybersecurity of the operational technology being employed in their setting,” Robinson said. “For many of the reasons I have outlined above; lack of technical expertise, lack of resources and funding, lack of clarity on compliance requirements (which would provide better resource justification from PWS owner-operators), and one of the biggest issues we have seen is that many operators do not necessarily have a good handle on their OT assets and inventories or how their networks are connected to their IT networks,” he added. 

He concluded that “all exercises, assessments, controls, and incident response planning will be useless unless you make and enforce the basics to be completed; complete asset and inventory and the monitoring of all OT communications (IP and non-IP).”