Illustration: Si Weon Kim
In its first-ever report for the Department of Homeland Security, a group of top government and industry cyber experts said the Log4j vulnerability triggered “one of the most intensive cybersecurity community responses in history” last December — and it’s far from over.
The Cyber Safety Review Board has good news and bad news for network defenders in its highly anticipated review of the Log4j vulnerability, which affects tens of thousands of pieces of software running on a countless number of systems worldwide.
First, the good: The DHS-backed panel, loosely modeled on the National Transportation Safety Board, said it’s “not aware of any significant Log4j-based attacks on critical infrastructure systems.” Given the severity of the vulnerability, “generally speaking, exploitation of Log4j occurred at lower levels than many experts predicted,” the 15-member CSRB added.
As for the bad news: the CSRB said “significant risk remains” from the open source software flaw.
“Log4j is an ‘endemic vulnerability’ and that vulnerable instances of Log4j will remain in systems for many years to come, perhaps a decade or longer,” the report concluded.
The report underscores the broader problem of critical flaws in open source software having outsized impacts. The CSRB likened snippets of open source code to building blocks in global technology infrastructure: “These reusable building blocks, while useful for creating software at scale, also create dependencies and risks that are often not understood until they manifest as a security issue,” the group noted. When organizations don’t know where faulty building blocks are built into their systems, it sets the stage for a “once-in-a-generation security event” like Log4j.
Costly fallout
README covered similar open source security concerns last November — before Log4j was publicly disclosed — in response to reports of developers being targeted by malicious software packages. Log4j, a Java-based logging tool whose name has become synonymous with its massive vulnerability, is far from the only open source project to pose problems. Vulnerable or malicious packages have cropped up over and over and over again in recent months. The Log4j exploit, also known as Log4Shell, simply demonstrated the extent of these risks — and made it clear organizations could no longer afford to ignore them.
“Organizations spent significant resources as they struggled with [responding to the Log4Shell vulnerability],” the CSRB said. “ For example, one federal cabinet department reported dedicating 33,000 hours to Log4j vulnerability response to protect the department’s own networks. These costs, often sustained over many weeks and months, delayed other mission-critical work, including the response to other vulnerabilities.”
The Log4j review posed several challenges to the CSRB, which was formed in response to President Biden’s May 2021 executive order on cybersecurity, according to the group’s chair Robert Silvers and deputy chair Heather Adkins.
“First, unlike comparable studies of incidents in other sectors (such as transportation), we had no crash site or damaged vehicle to inspect, no stress tests to perform on failed equipment, and no wiring diagrams to review,” said Silvers, who is under secretary for policy at DHS, and Adkins, senior director for security engineering at Google. “Instead, we reviewed practices used to create and adopt technology, ecosystems, and processes.”
The report also noted cyber incident reporting is still largely voluntary and “no authoritative source exists to understand exploitation trends across geographies, industries, or ecosystems,” hindering the ability to draw many conclusions about Log4j’s reach.
The CSRB turned to “nearly 80 organizations and individuals representing software developers, end users, security professionals, and companies” and a mix of “classified and unclassified government information and industry data.” The resulting report is split into three sections: a summary of events, the CSRB’s findings, and its recommendations for avoiding similar issues.
Open source security risks
Organizations don’t just need to respond to Log4j; they must also address the underlying problem of relying on open source software without mitigating the risks of using other people’s code, the CSRB found.
Of the group’s 19 recommendations, four are specific to vulnerabilities in Log4j, while many of the remaining 15 focus on securely using open source software.
“[Log4j] called attention to security risks unique to the thinly-resourced, volunteer-based open source community,” the CSRB said. “To reduce recurrence of the introduction of vulnerabilities like Log4j, it is essential that public and private sector stakeholders create centralized resourcing and security assistance structures that can support the open source community going forward.”
Donald Fischer, CEO of open-source technology startup Tidelift, said he agrees with that CSRB finding.
“Honestly, I’ve been shocked that in the wake of Log4Shell it has not become patently obvious to everyone in the industry that improving open source software supply chain health and security begins with paying maintainers to ensure their software meets standards like those highlighted in the CSRB report,” Fischer told README.
“It is time we quit assuming that open source maintainers don’t know what to do,” he added, “and instead give them the financial incentive to start doing it” — an issue that Tidelift is working to address.
Tackling the next Log4j-scale vulnerability
Fischer pushed back against the CSRB’s insinuation that the Log4j vulnerability resulted from a lack of knowledge on the part of the Java tool’s developers, rather than constraints on their attention, finances, and time.
The CSRB report said “the Log4j developers might not have introduced the vulnerability in 2013 if they had had access at the time to training in secure coding practices consistent with established secure development lifecycle tools and techniques.”
The report added that a “a technology services company seconded that position and suggested that security-oriented design reviews, threat models, and security audits may have prevented events such as Log4j.”
Fischer told README that “this observation seems to pre-suppose that open source developers are less sophisticated in their secure coding practices than developers in private industry.” He said that open source developers are quite skilled but are held to a different standard. Because their contributions to the software ecosystem are used in so many contexts — not tucked behind the walls of a single company or product — “the potential blast radius is huge” if they make mistakes, he said.
“Why aren’t we as an industry and a society showing up to support our most critical developers, the ones behind the most widely-used open source projects, with the types of support that would really accelerate their work?” Fischer said.
Many of CSRB’s recommendations focus on providing that support and making it easier for organizations to respond when a vulnerability like Log4j all-but-inevitably appears in some other popular open source project.
Lawmakers have taken note: Rep. Jim Langevin (D-R.I.) pointed out that he passed an amendment to critical defense legislation to create Critical Technology Security Centers, which are aimed in part at testing and fixing open source software flaws.
“Widespread open source vulnerabilities, like Log4j, demonstrate the need to invest in open source software security,” Langevin said on Twitter. “Great recs from CSRB — let’s get to work.”
