CSRB report on Log4j open source vulnerabilities demands better software ecosystem, future investments

With the availability of the initial report from the Cyber Safety Review Board (CSRB) on the Log4j open source software library vulnerabilities, the needle of focus has shifted towards driving better security in software products, while enhancing public and private sector organizations’ ability to respond to severe vulnerabilities. Organizations must work with industry stakeholders to identify and address vulnerabilities in the ecosystem driving existing best practices to bring about greater cybersecurity hygiene.

The report found that organizations that responded most effectively to the Log4j event understood their use of Log4j and had technical resources and mature processes to manage assets, assess risk, and mobilize their organization and critical partners to action. However, while most modern security frameworks call these capabilities best practices, few organizations could execute this kind of response or the speed required during this incident, causing delays in their assessment of the risk and their management of it. 

Open source projects tend to be diverse and numerous, using a range of operating models. With rising cybersecurity threats and vulnerabilities, organizations should enforce formal baseline software supply chain security controls regardless of where and how code is developed, the risks of using open source or community-developed software are unique. 

The CSRB report contains 19 specific recommendations for government and industry. Internet giant Google announced this week that they would work together to increase open source security by driving the adoption of best practices, building a better software ecosystem, and making long-term investments in digital security. Google will also partner closely with industry stakeholders to identify and address vulnerabilities in the ecosystem and share best practices on how to address the latest security threats. It hopes that sharing the information will spark industry-wide discussion and progress on the security and sustainability of the open source ecosystem.

The board is unaware of any significant Log4j-based attacks on critical infrastructure systems. However, exploitation of Log4j occurred at lower levels than many experts predict, given the severity of the vulnerability. As the impacts are largely unknown, the industry should be cautious and alert. OT networks typically suffer worse visibility into the application infrastructure than IT, and even IT is not in a great place. 

Industrial Cyber got in touch with industry experts to find out whether the CSRB report advances the overall security and resiliency of the critical infrastructure sector.

“The Log4j vulnerability generated a strong response from the cybersecurity community, and the CSRB’s review serves both as an authoritative source of information as well as providing recommendations for both the private and public sectors,” Matt Gorham, senior managing director with PwC’s Cyber & Privacy Innovation Institute, told Industrial Cyber. “The composition of the CSRB ensures the insights of the government and the private sector are fused into a comprehensive view. The report advances the overall security and resilience of critical infrastructure both through the information in the report but also as the report itself is a validation of the concept of the CSRB,” he added.

Tony Turner, vice president for Fortress Labs (R&D) team at Fortress Information Security, told Industrial Cyber that the CSRB report is a good summary of what came before and what we can do about it going forwards. “It’s not immediately clear to the reader, though, how much of this is generic guidance vs applicable to this specific vulnerability event. The log4j issue was a ‘zero day’ in the sense that many organizations were using this library without the industry at large understanding it was vulnerable,” he added. 

“The guidance for SBOM, in this case, would have been a preparatory effort that would not have resulted in value until we all learned about the issue,” Turner said. “What’s really important to highlight here, is that this can dramatically shorten the time to respond when the vulnerability does get disclosed. As a follow-up to this, we have seen many instances where the ‘vulnerable’ library was found in software, but it was not exploitable. Similarly, we have seen situations, where vulnerability scanners used, could not detect more than 50% of the actually vulnerable installations,” he added. 

Turner also said that this is where the combination of SBOM and VEX, the ability to reduce the false positives identified in an SBOM to laser in on what is truly exploitable, produces a huge amount of value. “We were disappointed that the report does not mention VEX at all. In fact, even though CISA is actively championing VEX, NIST, by comparison, seems to be ignoring it. This is creating a disconnect in the industry today that is further exacerbated by the CSRB report,” he added.

With the CSRB report highlighting that the Log4j vulnerabilities did not significantly impact the critical infrastructure, Turner thinks it may lead some readers to make the wrong conclusion. “That critical infrastructure is not affected by these types of flaws. In our software analysis of hundreds of thousands of artifacts for critical infrastructure, we have seen a far lower presence of java-based software than we see in the enterprise. Typically speaking, 33% of IT software uses java, so a Java-based vulnerability like this creates a massive impact on IT. In critical infrastructure and highly regulated environments like CIP, we are seeing less than 5% of the applications we assess and produce SBOMs for using Java,” he added.

Turner also estimates that “it will be more than a year until the impacts are understood, and likely any actual compromises by determined adversaries may lie in wait for a future event before making their compromise known. Perhaps even in a combined kinetic and cyber event to maximize impacts,” he added.

“In May 2021, President Biden’s Executive Order 14028 was issued. The EO was meant to help mitigate cyber threats by requiring companies that provide products and services to the federal government to report any potential vulnerabilities or cyber incidents within their own systems,” Roya Gordon, security research evangelist at Nozomi Networks, told Industrial Cyber. “The EO also required the government to proactively implement basic safety features and processes within their own networks. One of the main initiatives birthed from this was the establishment of the Cybersecurity Safety Review Board (CSRB), which consists of both government and private sector experts.” 

The CSRB is to conduct in-depth analysis following significant cyber incidents, in which the lessons learned could help strengthen security in the future, Gordon said. “The review of Log4j is the CSRB’s first review since it was established, so it’s a bit too soon to tell how relevant or even how beneficial they are in advancing overall security and resiliency. The Log4j review seems to be pretty thorough, we just need to give additional time for the industry to respond and react to their findings,” she added.

The CSRB said in its report that “At this time, Log4j has become an ‘endemic vulnerability’ that will be exploited for years to come. The impact to organizations over the long term will be difficult to assess without better tools for discerning real exploitation and centralized reporting of successful compromises.” 

Gorham reacted by saying it’s true that the Log4j vulnerability will have a lasting impact, but it’s a known entity and one that organizations can protect themselves against, critical infrastructure included. “Conversely, threat actors are also now well versed on how to exploit the Log4j vulnerability, so it truly is a race for organizations to patch this vulnerability (and future ones) as fast as possible,” he added.

“President Biden’s July 2021 memorandum to modernize defenses in industrial control systems included the requirement of setting cyber performance goals, defined as ‘baseline security practices that critical infrastructure owners and operators should follow to protect national and economic security, as well as public health and safety,’ for both government and the private sector,” Gorham said. “As these goals are developed, organizations will face increased pressure to invest in technology and their people to improve protection against and response to threats. These investments will be imperative to manage cyber risk for an organization, as well as national security,” he added.

Turner once again mentioned that the concepts of SBOM and VEX seem to be inconsistently applied, and as these concepts are still unfolding, so is the tooling ecosystem. “But aside from the source data to be reported, the need for centralized data sharing is key to inform asset owners of the real risks they should be concerned about.”

Identifying that in the absence of this information, Turner said that one of two things typically occurs: Either everything is urgent and becomes a distraction to other real risks that get overshadowed in the face of Log4j hype. So real incidents might not get worked while operators focus on triaging all the possible Log4j issues. Or, nothing is urgent, and real Log4j exploitation opportunities persist, making the attacker’s job very easy.

“The Log4j exploit is a real threat to critical infrastructure companies, and it’s one that needs to be taken seriously. Without having network visibility, it’s difficult to tell if this vulnerability is being exploited or not,” Nozomi’s Gordon said. “The real problem is, most critical infrastructure companies don’t even know what devices they have or which devices should be communicating with each other. This creates a huge problem when trying to identify what devices are susceptible to the Log4j exploit,” she added.

According to Gordon, companies need tools that give them this visibility, not just for Log4j, but for identifying other types of anomalous behavior that could be associated with threat actor activity. “This incident is definitely the push that critical infrastructure industries need to beef up security, and it all starts with visibility. You can’t address a problem you can’t see, and I think that is the message that CISA is trying to get across to companies,” she added.

Determining the steps that critical infrastructure organizations must take to drive transformation in the software ecosystem to bring about a more proactive model of vulnerability management, Gorham listed four key areas cyber leaders should consider when evaluating their organization’s software ecosystem. 

“A multi-layered protection model is necessary to defend against real-world attack trends and threat vectors,” according to Gorham. “Defense-in-depth is a powerful capability against many threats, including software supply chain attacks. On high-risk applications, deploy network security strategies such as network zoning, network deception, micro-segmentation, anomalous behavior detection, and firewalls. Furthermore, many organizations are still not doing the basics, such as having risk management and assessment procedures for third-party providers, such as those who provide critical software.”

Gorham also includes reliance on open-source software and components, including cataloging and assessing open-source libraries in the system. “Determine what you would ‘allow’ and ‘deny’ for use by developers in your organization. Put controls around these libraries and monitor their use,” he added.

He also said that effective threat modeling shouldn’t happen just once, and it shouldn’t focus only on known methods of attack. “Part art, part science, your threat modeling needs creativity and imagination. You and your teams should expect the unexpected and plan — and act — accordingly. Run hypothetical scenarios with imagined zero-day vulnerabilities to assess whether you have the capabilities to respond,” according to Gorham.

Gorham also pointed to the scope of resilience plans. “Forty percent of the 3,249 executives in PwC’s 2021 Global Digital Trust Insights Survey told us they plan to increase resilience testing to help confirm that their critical business functions will stay up and running should a cyber event occur,” he added.

“Most importantly, we all need to demand more of our product suppliers. There is a litany of things we should be asking for, from IEC 62443 certification to more fidelity around their software development process, and yes, asking for SBOMs,” Turner said. “But it has to start with sending the message that the status quo is not good enough, and security is a differentiator to drive supplier revenue streams. We have to make it matter to the supplier to drive change. Clearly identifying requirements in contracts and in the RFP process is a great start. But so is publicly demonstrating how good security is a core operational requirement, not a nice to have,” he added.

“But legacy software remains a problem for decades to come. Contracting will not fix that, and we need to develop processes that prioritize supplier response but can mitigate the risks when suppliers can’t or won’t cooperate,” according to Turner. “The advent of reverse engineering firmware is a great example. New solutions have sprung up to deal with the legacy software problem. We need to have a plan, but we also need a backup plan when that does not work. For instance, 62443 is a role-based framework where integrators, software suppliers, and asset owners all have a role to play. But if the integrator does not do their part, we need to identify mitigating controls to fill in the gaps. Anything less is too fragile to succeed,” he added.

Gordon said that she does not think “that the next phase of securing the technology ecosystem is to focus on the open source software security. I mean, the point of it being open source is for it to be decentralized hence why it’s maintained by a community of engineers, but this type of decentralization has its disadvantages.” 

She spoke about running into issues when reporting vulnerabilities of widely known language libraries at her company. “It’s not as easy as a vendor pushing out a patch or firmware update. In these instances, the maintainer of the library can’t guarantee the fix because it is based on the community of coders to implement and update, which makes it difficult to report on. Additionally, if the library does get updated, the vulnerability is still widespread until asset owners update the code on the individual devices,” Gordon added.

“A lot of organizations don’t understand this process and I think that is this type of granularity we need in order to come up with better security solutions,” according to Gordon. “We need to start taking a deeper dive into who maintains these libraries, how often are they updated, what is the process for them to be secure, and then create a step-by-step for asset owners to follow in the event of another Log4j incident. That is what’s missing right now, and that is why Log4j and maybe the next type of log4j incident will be what CISA is calling an ‘endemic vulnerability’,” she added.

The Board notes that the community should implement improvements to forestall the next Log4j-type event while taking actions to intervene and remediate the present risks. 

Analyzing if the proposed measures are sufficient to deal with rising vulnerabilities and growing cybersecurity incidents amid geopolitical cyber threats, Gorham said that no single report is sufficient to deal with the growing cybersecurity risk, but this report is a necessary step in the right direction, as it provides an authoritative voice and strong recommendations around the incident and the use of open source software. Specifically, on the use of open source software, companies must ensure they understand what open source software they utilize and make risk-based decisions on where and when they use it, he added.

“Visibility, transparency, and detection are key in limiting the impacts of a cyber attack,” according to Gorham. The proposed measures will be effective, but there are three critical areas cyber leaders need to emphasize activity around critical assets, including data and business services; communication with and regular assessment of suppliers, business partners, and third parties; and detection, he added. 

Turner said that there is a huge gap in this paper regarding the evaluation of exploitability. “But the plan laid out for the establishment of a Cyber Safety Reporting System has a lot of potential; unfortunately, current industry security metrics are not in a good state. Adopting CVSS, for instance, or even enriching it with EPSS, may miss the mark entirely. The biggest issue we see with these methodologies for the critical infrastructure operator is the complete lack of context to drive actual consequences understanding,” he added. 

“Generic approaches are attractive and scalable, but not effective when the same product can be implemented in two different ways to manage two completely different processes,” Turner said. “We think there is an opportunity for collaboration with the private sector to establish an information sharing repository for critical infrastructure because it is important that asset owners are part of the conversation in identifying what is important to them. A purely government-run system may be ill-advisable.”

While the geopolitical climate is certainly a factor, at the response level, adversary attribution injects bias into assumptions made, Turner observed. “Is it China? They must be doing this. Is it Russia? They must be doing this. But if we want to be effective in dealing with the threats, we need more focus on the actual observable events, not on the assumptions we make because we expect ransomware from Russia and espionage from China,” he added.

Gordon said that this isn’t just a problem with Log4j—it’s a problem with every organization’s asset inventory and vulnerability response process. “Organizations need to know what devices are susceptible to exploitation and prioritize fixing them right away. This is true whether or not there’s ever been a Log4j vulnerability. So the recommendations that CISA is recommending are spot on,” she added.

“As far as geopolitical cyber threats, the lines are pretty blurred,” according to Gordon. “Ransomware threat actors and other cyber criminals, hacktivists, and nation-state threat actors can all utilize vulnerability exploits to carry out targeted attacks when conflict intensifies—and that’s precisely why it’s so critical for us to protect ourselves from those threats now. The point is, that securing Log4j is increasing overall network security and network resiliency,” she concluded.