When security updates go wrong: Patch problems plague tech giants
Keeping software up to date is a mainstay of good cybersecurity hygiene. But in rare cases when patches backfire — either by introducing new, more severe software flaws or failing to fix the old ones — it can set off a scramble to set things right.
Microsoft’s “Patch Tuesday” follows a familiar routine: On the second Tuesday of each month, hundreds of millions of Windows users are prompted to update their machines to add new features or fix security vulnerabilities.
But on May 10, a patch to a critical vulnerability in a fundamental Windows security control called Local Security Authority went awry, causing authentication problems in a number of Windows services. The hiccup prompted the U.S. Cybersecurity and Infrastructure Security Agency to take the rare step of removing the patch from its Known Exploited Vulnerabilities Catalog, the federal government’s blueprint for prioritizing bug fixes. Another severe Windows vulnerability affecting Remote Desktop Services had to be patched twice — once in January and again in April — as researchers at CyberArk detailed in a blog post last week.
Microsoft is hardly alone when it comes to dealing with unexpected effects from security patches. It’s a persistent challenge for every software vendor. And despite recent research into breaking the relentless cycle of patching, developers will be grappling with unintended consequences from certain patches for years to come.
A Log4Shell game
Software is more complicated than ever these days, with lots of libraries, APIs, drivers, and networking services that need to be implemented. That sophistication makes computers very versatile and powerful, but it also poses a challenge to the cybersecurity professionals whose jobs involve discovering vulnerabilities and correcting them.
Take the case of Log4j, the Apache Software Foundation’s widely used logging component that infamously “set the internet on fire” late last year.
When the Log4Shell remote code execution (RCE) vulnerability was disclosed to the public in December 2021, Apache had already been working on fixing it.
Apache released its first Log4Shell patch on December 10, Log4j version 2.15.0. But researchers quickly discovered a new RCE vulnerability in the updated framework, so Apache released version 2.16.0 just three days later. But New Log4Shell bugs were discovered, so 2.17.0 debuted on December 17. To put all that activity in context: Log4j 2.0 was released in 2012, so December 2021 had a lot of unusually substantial changes in a brief period.
On December 27, Apache addressed even more Log4Shell vulnerabilities. But the changes in versions 2.17.1 and the latest 2.17.2 were more subtle. Still, each fix hinged on the contributions of the open-source software community. While the Log4Shell nightmare is largely in Apache’s rear-view mirror, the string of Log4j patches illustrates how developing security updates can feel like a game of whack-a-mole to developers.
Finding the right tools
Jeremy Woertink, a developer for Joystick.tv, an online streaming platform targeted to adults, told README developers with little knowledge of security holes may pull the ingredients for their patches from popular web forums like StackOverflow and hope for the best — “implementing them blindly without understanding repercussions.”
“The other tricky thing is there’s lots of tools out there that can help you, but if you don’t know where to start, then it’s really not much help,” he said. “As a developer that’s not too familiar with the security side of things, you sort of have to rely on either landing a job that employs someone in this role in order to teach you, or wait until you’ve been attacked and you end up spending an entire night” rushing to find out how you deployed something with a severe vulnerability.
When developing applications, scripts, and software patches of all kinds, it’s standard practice to test new code on the devices it’s designed to execute on, or at the very least on virtual machines which simulate those devices and operating systems. Professional developers, including those who work for the biggest tech companies in the world, thoroughly test new code on devices and through virtualization before releasing anything to end users.
Unfortunately, even proper testing can’t always prevent software patches from having unintended effects. It’s not feasible to execute new code throughout the internet or on millions of machines before official release. So surprises sometimes happen, as with Microsoft’s first crack at fixing the Local Security Authority flaw last month. (Microsoft, for its part, told README it addressed the issue with a security update in May and is working with CISA concerning the agency’s report.)
Jim Manico, founder of the Manicode Security online training resource for programmers to improve the security of their code, told README security patch development can pose unique challenges. But he emphasized how important it is for network defenders to strive to keep everything up to date.
“The lack of patching security vulnerabilities in third party libraries and frameworks in software development is one of the most extreme risks and challenges in software development today, especially for teams that already use hundreds of libraries in their work, which is sadly very common,” he said.
