US PHMSA penalizes Colonial Pipeline nearly $1 million for control room management failures
A year after DarkSide ransomware hackers breached Colonial Pipeline, the U.S. Department of Transportation’s Pipeline and Hazardous Materials Safety Administration (PHMSA) proposed a civil penalty of close to US$1 million. In addition, the Alpharetta, Georgia-based company has been served a Notice of Probable Violation (NOPV) and Proposed Compliance Order, which also includes multiple probable violations of federal pipeline safety regulations (PSRs).
“The 2021 Colonial Pipeline incident reminds us all that meeting regulatory standards designed to mitigate risk to the public is an imperative,” Tristan Brown, PHMSA deputy administrator, said in a media statement. “PHMSA holds companies accountable for violations and aims to prevent any instances of non-compliance.”
From January through November 2020, PHMSA conducted an inspection of Colonial Pipeline’s procedures and records for Control Room Management (CRM) in Linden, New Jersey, Hebert, Los Angeles, Greensboro, North Caroline, and Alpharetta, Georgia. “PHMSA made preliminary determinations that Colonial Pipeline Company was in probable violation of several PSRs, including a probable failure to adequately plan and prepare for manual shutdown and restart of its pipeline system,” the statement added.
PHMSA informed Colonial Pipeline of the alleged non-compliance items shortly after the 2020 inspections concluded, according to the statement. “The NOPV alleges that failures to adequately plan and prepare for a manual restart and shutdown operation contributed to the national impacts when the pipeline remained out of service after the May 2021 cyber-attack,” it added.
When addressing control room management, Colonial “failed to follow its procedure, ADM-CPC-008 Rev.2 7/1/2019 Point-To-Point Verification, when documenting a point-to-point verification between SCADA displays and related field equipment, per § 195.446(c)(2), for 87 safety-related pressure transmitter alarms for the Linden Station in calendar year 2019,” the notice said. SCADA (supervisory control and data acquisition) systems typically run proprietary control protocols using specialized hardware and software and function solely in the operational technology (OT) environment.
The PHMSA added that a review of the Linden 2019 SRA point-to-point records identified 87 records for pressure transmitters where no documentation was entered for as-found/as-left field. “This included 19 records for both addition of new equipment and modification of existing equipment which failed to meet the requirement of procedure ADM-CPC-008 Rev.2 7/1/2019 Point-to-Point Verification,” it added.
Commenting on this key aspect of the notice, Padraic O’Reilly, chief product officer and co-founder at CyberSaint, wrote in an emailed statement that this kind of violation is central to the practice of cyber. “You cannot protect what you do not know you have. So while a lot of this sounds procedural, it also involves infrastructure and alarm accuracy. Vulnerabilities are only mitigated via policy and process,” he added.
“The service was disrupted to some extent because Colonial did not know the possible extent of a ransomware jump across into OT,” O’Reilly added.
Colonial also failed to follow procedures when conducting and documenting point-to-point verifications in SLM for Safety-Related Alarms (SRA) to ensure alarms are accurate and support safe pipeline operations, per § 195.446(e)(1) and § 195.428(d). “A review of 2019 SRA completed tests for Hebert, Linden, Alpharetta, and Greensboro identified either no documentation for as-found as-left conditions (cells were blank), or it was filled in with N/A or NA. These responses were inclusive for the SLM Reasons for the Point-to-Point related to Addition of New Equipment, Preventative Maintenance, and Modification of Existing Equipment,” according to the notice.
The PHMSA notice also added that the records provided by Colonial represent SRA point to point completed in 2019 for Alpharetta, Greensboro, Linden, and Hebert. “Dates with time stamps are included in the document of record. The procedure required point-to-point verifications with documentation of the values as-found and as-left. There was no consideration for no entry (blank spaces) or use of NA or N/A. Colonial failed to follow their procedures when conducting and documenting point-to-point verifications in SLM for Safety-Related Alarms,” it added.
Further, the notice pointed out that Colonial failed to complete and document verifications of alarm set-point and alarm descriptions in compliance with its procedures when associated field instruments were calibrated or changed for five safety-related points at the Greensboro facility. “They also were not able to verify, for the years 2017, 2018, and 2019, all safety-related alarm set-point values and alarm descriptions were correct,” it added.
Additionally, Colonial failed to provide records to demonstrate they verified correct safety-related alarm set-point values and alarm descriptions for all safety-related alarms at least once each calendar year not to exceed 15 months. “Colonial has safety-related alarms that relate to tank levels and reliefs requiring annual calibration, as discussed above. However, there are other safety-related alarms that the system manages outside of these alarms that must also be reviewed,” the notice added.
Colonial also failed to complete and document verifications of alarm set-point and alarm descriptions when associated field instruments were calibrated or changed for five safety-related points. The company also did not provide a procedure that requires verification of the correct safety-related alarm set-point values and alarm descriptions when associated field instruments are calibrated or changed and at least once each calendar year not to exceed 15 months.
The company also “failed to test and verify its internal communication plan to provide adequate means for manual operation of the pipeline at Linden and Hebert in 2017, 2018, and 2019, at Greensboro in 2018 and 2019, Alpharetta in 2017, Baton Rouge, Collins and Charlotte in 2018, 2019, 2020,” the PHMSA notice said.
The PHMSA notice also determined that Colonial Pipeline failed to test the SCADA backup servers at the Linden, Hebert, and Greensboro field operations control rooms at least once each calendar year, but at intervals not to exceed 15 months, for the years 2017, 2018, and 2019, in compliance with its operating procedures.
The notice also identified that “Colonial for Greensboro, Hebert, and Linden failed to identify and record, at least monthly, all points affecting safety that had been taken off scan in the SCADA host; all points that have had alarms inhibited; or that have had forced or manual values for periods of time exceeding that required for associated maintenance or operating activities, per § 195.446(e)(2), for the years 2017, 2018 and 2019.”
Commenting on the PHMSA notice, O’Reilly said that most violations and penalties levied tend to start with a physical violation, a leak, spill, etc. “The laws are written around this type of event and there is not a lot of cyber in the existing regs, which are in place around the physical security of pipelines and the associated processes and documentation,” he added.
“My hunch is that this penalty is unique in that the event is the downtime Colonial experienced, and the regulators went after flawed process and documentation,” according to O’Reilly. “CISA and PHMSA were much more delicate with the December 2019 attack on a natural gas compression facility that resulted in a two-day shutdown,” he added.
Last May, Colonial Pipelines, one of the largest fuel pipelines in the U.S., was forced to take its systems offline temporarily after DarkSide ransomware hackers struck. The halt in operations led to disruptions in gas supply and prompted a regional emergency declaration across 17 states. Further, the company paid $4.4 million in ransom to the cybercrime syndicate to regain access to its computer network. However, federal agencies could recover a significant chunk of the digital funds paid.
A complimentary guide to the who`s who in industrial cybersecurity tech & solutions
Free DownloadRelated
