Rapid7 vs JetBrains: A vulnerability disclosure process gone bad

Johann Walter Bantz / Unsplash

A recent conflict between Rapid7 and JetBrains over how to disclose vulnerabilities was marred by blame, confusion and conflicting philosophies over how best to minimize attacks when software flaws become public.

A rare public dispute erupted earlier this month between cybersecurity research firm Rapid7 and Czech software vendor JetBrains over how to report security vulnerabilities. After Rapid7 discovered flaws in a JetBrains product and reported it to the vendor, the relationship between the two companies rapidly deteriorated over how much information to release and when to release it.

This incident exposed how security researchers and software developers collaborate to identify and publicly report fixes for software vulnerabilities. It also highlighted how these collaborations can go wrong, causing recrimination between the parties and confusion for security defenders.

Moreover, the conflict between the two firms, each of which says its approach minimized security risks, reiterated the value of a mutually beneficial coordinated disclosure process.

How the vulnerabilities disclosure process fell apart

On March 4, Czech software provider Jet Brains announced a new bug-fix version for its TeamCity On-Premises continuous integration and build management product. About 30,000 organizations use the product, including big blue-chip companies such as Citibank, Nike and Ferrari.

The new version fixed two serious JetBrains TeamCity CI/CD server vulnerabilities. The first, CVE-2024-27198, is a highly critical authentication bypass vulnerability in TeamCity's web component arising from an alternative path issue (CWE-288) with a CVSS base score of 9.8. It allows a remote, unauthenticated attacker to completely compromise a vulnerable TeamCity server.

The second flaw, CVE-2024-27199, is an authentication bypass vulnerability in the web component of TeamCity that arises from a path traversal issue (CWE-22) and has a CVSS base score of 7.3, still high but not as critical as the first flaw. This flaw permitted a limited amount of information disclosure and a limited amount of system modification, including the ability for an unauthenticated attacker to replace the HTTPS certificate in a vulnerable TeamCity server with a certificate of the attacker's choosing.

After JetBrains issued the bug-fix version, it started emailing all TeamCity On-Premises customers and Security Bulletin subscribers to warn them of the vulnerabilities and advise them to upgrade or install the security patch as soon as possible. JetBrains then published a blog post with limited details about the security flaws.

Hours after JetBrains issued its announcement on March 4, cybersecurity company Rapid7 disclosed more details about those flaws. Rapid7 said JetBrains released a fixed version of TeamCity without notifying Rapid7 that fixes were generally available and had been implemented. Stephen Fewer, Principal Security Researcher at Rapid7, discovered both flaws in February.

Rapid7 said it was disclosing the flaws in accordance with Rapid7's vulnerability disclosure policy, which includes a provision barring "silent patching," or non-public disclosure of security patches. Rapid7 said JetBrains suggested issuing the patches privately before public disclosure and twice asked for clarifications on Rapid7's policies against silent patching.

In its disclosure, Rapid7 presented proofs-of-concept for the flaws, demonstrating how attackers could exploit them and offering remediation steps. Rapid7 also presented a timeline for how it handled the discovery of the flaws, including communications with JetBrains.

On March 5, JetBrains released its own timeline for how the vulnerability disclosure went down, saying it never intended to release a fix silently without making the full details public. The company emphasized that it was following its practice of allowing for a time delay between releasing a fix and making full disclosure so that its customers could upgrade their TeamCity instances.

JetBrains also said it had decided not to make a coordinated disclosure with Rapid7 because it strongly believes that publishing all technical details at the same time as releasing a fix allows anyone to immediately exploit the issue before all customers have had a chance to patch their servers.

Six days later, on March 11, JetBrains issued another post, defending its own Coordinated Disclosure Policy. JetBrains blamed Rapid7 for "broadening the window of opportunity for attackers" so soon after its bug-fix version announcement. It also wrote in boldface, "We firmly believe that releasing the full technical details of a vulnerability and the exploit steps simultaneously with its fix is entirely unethical and harmful to our customers."

JetBrains pointed to three anonymous customers it said had been exploited due to Rapid7's full disclosure and publication of exploits. These attacks, the company said, were "due to the immediate availability of publicly documented exploit examples published by Rapid7, which meant attackers of any skill level had all the resources they needed to quickly exploit the vulnerabilities in the wild."

A silently spoken patch

Much of this dispute centers on silent patching, although JetBrains didn't issue a silent patch. It was more of a "silently spoken" patch with fewer details than Rapid7 released.

Silent patching, where a vendor fixes a vulnerability without publishing an advisory or registering a CVE, is "way more common than anyone knows," Roger Grimes, data-driven defense evangelist at KnowBe4, told README. "Many vendors find and fix dozens to thousands of bugs found internally without ever opening a proper CVE. A large percentage of organizations don't create a CVE unless it is found by an external party."

Staying mum, or relatively mum in JetBrains' case, about vulnerabilities and patches eliminates public embarrassment, but, as JetBrains argued, it might also minimize the number of instances in which attackers exploit a patched vulnerability. "Part of the problem is that when a vulnerability is publicly announced and known, it kicks off the race between the vendor trying to get everyone up to date and patched before the attackers take advantage of the unpatched instances," Grimes said. "And every patch has some non-minor percentage of instances that will not be patched in a timely manner or never patched at all."

However, withholding information about vulnerabilities might also limit the visibility of security defenders. "I think oftentimes vendors are a little bit too cautious when putting information out there, which is somewhat understandable," Daniel Dos Santos, head of security research at Forescout Technologies, told README. "They want to protect their customers, but in some cases, this might increase risks for others."

This increased risk for other defenders is at the heart of Rapid7's policy against silent patching. "When you silently patch, you are intending to limit knowledge of the patched vulnerability to skilled exploit devs," Rapid 7 says. While silent patching might exclude casual attackers, it also excludes "a huge population of IT protectors: penetration testers who are paid to write and run exploits to test defenses leap to mind, in addition to the folks who write and deploy defensive technologies like vulnerability management, intrusion detection and prevention, incident detection, and all the rest."

"I think I can understand a little bit of both sides," Dos Santos said. "It is very frustrating for a security researcher when the vendor doesn't seem to give enough attention to a case and then moves on to independently do something without working with the researcher to understand what's going on. We have seen instances where vendors think they have patched something. But they have patched only parts of the problem because they haven't discussed it enough with the researchers."

But, he added, "I think there is no right or wrong in every case. It's a bit nuanced, depending on the vulnerability. I am generally in favor of providing more information because, in many cases, you need, as a defender, that extra information to understand, create a detection rule, know how to use a detection rule, or apply the right kind of mitigation on your network to understand if something that you are seeing is a real attack or is a false positive and so on."

What did attackers know, and when did they know it

Although JetBrains pins its customer attacks on Rapid7's detailed disclosure, many experts believe threat actors and at least some security researchers likely already knew about the vulnerabilities. "I think it's naive for [JetBrains] to think that somebody wasn't already exploiting this vulnerability prior to any disclosure whatsoever," Bob Huber, CSO and head of research at Tenable, told README. Huber said he thinks JetBrains' naivety is all the more surprising given that it has a depth of experience in dealing with vulnerabilities, having reported hundreds of CVEs dating back to 2014.

Huber said it's commonly understood among cybersecurity defenders that attackers will begin to reverse engineer vulnerabilities when announced, with or without publicly available exploit models.

"Once we do disclose a vulnerability, and whether it's in coordinated disclosure or not, there is the expectation that attackers, adversaries, hackers will pick up whatever you've posted with whatever information you provided that could be a blueprint and add it to their repertoire, to their toolkits that they use," Huber said. "You certainly could see an uptick based upon a release of the vulnerability information. A lot of organizations will try to figure out how to reverse engineer the vulnerability and create an exploit."

Dos Santos pointed out that "there are many examples of public proofs of concept that are created by other researchers or attackers using the patch reverse engineering that JetBrains mentioned. Rapid7's release of details, while maybe accelerating the timeline for attackers, also provides an opportunity for defenders who, for whatever reason, cannot patch immediately but can better understand, detect, and mitigate risk from the vulnerability."

Coordinated disclosure rejection is 'not a good look'

JetBrains' refusal to participate in a coordinated disclosure might also have triggered the problem it now blames on Rapid7. It possibly pushed Rapid7 into a quandary about how to proceed, leaving the cybersecurity firm with little choice but to move forward with a disclosure that it believed was best for its customers and other defenders. (Neither JetBrains nor Rapid7 responded to requests for comment.)

"That might be the cause of the frustration for the Rapid7 team because they probably had some sort of response or something that could go out, but they were caught off guard by Jet Brains then releasing the patches and announcing things without doing coordination," Dos Santos said. "I am convinced that it was a difference in approach towards vulnerability disclosure where there's not necessarily a right or wrong party, but each organization sees a different way of protecting their customers. I am usually in favor of releasing as much detail as possible about vulnerabilities, and while I understand JetBrains' concern that Rapid7's actions may have led to the compromise of their customers, that's not necessarily true."

Tenable's Huber said that JetBrains' rejection of a coordinated disclosure is not "a good look for them because it makes it seem like they don't place importance on security. They could have coordinated everything, had the patch out there well in advance, and could have coordinated the window for Rapid7 to disclose as well," he said. "If you can't handle responsible disclosure, that's a bad look for your organization."