A trove of emails from top Homeland Security officials expose how the U.S. government scrambled to ensure the defenses of American utilities after Russia brought down parts of Ukraine’s power grid in 2015.
The Ukrainian electricity workers watched with dread as they lost control of their computers. Hackers had clawed their way into parts of western Ukraine’s power grid and cut power for thousands of people as they clicked open circuit breakers.
On that brisk December evening just over six years ago, Prykarpattyaoblenergo and two other Ukrainian power providers fell victim to a coordinated, unprecedented hack that knocked out electricity to nearly a quarter million customers.
Now, documents obtained under the Freedom of Information Act shed light on the U.S. response to that hack — later linked to Russia — and reveal how Department of Homeland Security officials wrestled with whether the U.S. grid could fall victim to a similar attack.
That question has taken on renewed urgency as the specter of a Russian invasion of Ukraine stokes fears of disruptive cyberattacks on Kyiv or its allies. Top Biden administration cybersecurity officials, Russia experts and members of Congress have all raised concerns over potential cyber retaliation if the U.S. applies more aggressive sanctions against Moscow or other measures in response to Vladimir Putin’s aggression toward Ukraine.
“My sense is that we need to be prepared — much better prepared — with the reality that we’re going to be subject to more and more cyberattacks and perhaps an intense effort at doing that from Russia,” said Sen. Rob Portman (R-Ohio), ranking member of the Senate Homeland Security and Governmental Affairs Committee, at a hearing Tuesday.
Indeed, the lessons learned from the Russian cyberattack on Ukraine in 2015, largely seen as a signal to the rest of the world that Moscow is capable and willing to carry out cyber-kinetic assaults, are more relevant than ever.
In early February 2016, Suzanne Spaulding, then undersecretary of the Department of Homeland Security’s main cybersecurity branch, raised concerns about Russian capabilities in an email to other government officials. She cited a CNN report breaking the news that the U.S. had uncovered proof that the hours-long Dec. 23, 2015, power outages in western Ukraine were the result of a cyberattack.
“This sentence is likely to get attention: ‘U.S. systems aren’t any more protected than those breached in Ukraine, the U.S. official said,’” Spaulding said in an email to six of her colleagues. “Do we agree?”
The immediate response in the email thread is redacted. But Spaulding, now a senior adviser for homeland security at the Center for Strategic and International Studies, told README that Ukrainians might have been particularly vulnerable because the country was once part of the Soviet Union.
“The Russians understood their systems in a way that the Russians hopefully don’t understand our systems: They had owned them,” she said.
Coupled with physical redundancies built into the U.S. grid, and a push toward making networked industrial control systems more secure, “we were farther along than the Ukrainians in building in some cybersecurity protections” at the time, she said. “So from that standpoint, the answer to the question might have been, ‘no.’”
Regardless of whether Ukraine is better or worse off today, it’s since grown impossible for U.S. utilities to ignore the cyberthreat posed by Moscow if they’re serious about ensuring reliability.
“This is the concept of resilience that we have to keep pressing for today: You’ve got to do all the things you can do to reduce the threat and reduce your vulnerabilities, but at the end of the day, in your planning, you have to assume that a determined adversary is going to find their way in anyway,” Spaulding said.
A calculated attack
The hackers who targeted Ukraine’s grid in 2015 were nothing if not determined.
Attackers later linked to Russia’s GRU military intelligence agency checked the boxes of a worst-case cyberthreat. That meant methodical planning, including replicating parts of Ukraine’s distribution-level control systems. The hackers drew on a sophisticated set of malware tools such as the BlackEnergy “Swiss Army knife” remote access Trojan and KillDisk, a nasty module designed to wipe victims’ hard drives by overwriting files with random data. They used tailor-made phishing emails to gain a foothold in their target networks months ahead of the actual grid takedown. They bombarded one utility with bogus phone calls in a bid to hamper restoration of electricity service. And they showed no regard for the fact that they were targeting civilian critical infrastructure.
“As a community the power industry is dedicated to keeping the lights on,” noted the late grid cybersecurity pioneer Michael Assante in the first official blog post confirming the hack for U.S. audiences. “What is now true is that a coordinated cyber attack consisting of multiple elements is one of the expected hazards they may face.”
Ukrainian utilities were able to quickly restore power in the wake of the attack, in part by deploying workers into the field to flip back circuit breakers by hand.
In the U.S., officials raised concerns that many utilities — particularly larger companies that serve millions of customers — had gone to lengths to embrace automation and that reverting to manual operations would be impossible in a cyber emergency.
There was also the discomforting fact that variants of the BlackEnergy malware used to break into the Ukrainian utilities had already been spotted in U.S. electricity companies. BlackEnergy featured some components designed to target industrial control systems, if not grid networks specifically.
Still, DHS privately assessed that the risk of a damaging cyberattack on U.S. energy infrastructure was low, and the latest FOIA documents show a coordinated effort to project confidence to the American public.
“DHS has received no reports of similar activity in the United States,” read a set of talking points circulated at the agency on Feb. 3, 2016. “However, we are constantly working with our private sector critical infrastructure asset owners/operators to improve their cybersecurity defenses.”
A trip to Kyiv
The draft talking points also cited a fact-finding mission to Ukraine that drew together officials from U.S. intelligence agencies including DHS and the Department of Energy, as well as representatives from the nonprofit North American Electric Reliability Corp. and the SANS Institute, where Assante worked.
The goal of the visit was to bring back actionable intelligence for U.S. utilities so they could avoid the same fate as the Ukrainian power providers. DHS would go on to lead a series of briefings for critical infrastructure owners and operators across the U.S., issuing alerts detailing the grid hackers’ tactics.
“We were very concerned that the way that Russia had gotten into those systems could be used against U.S. companies,” Spaulding said in a recent interview. “We did not feel confident that our electric companies or other companies using industrial control systems had done everything they should do to prevent them from being vulnerable to a similar attack.”
There are several thousand electric utilities scattered across the U.S., ranging from rural power providers without so much as a system administrator on staff to electricity giants with redundant control centers and double-blast-door-protected cybersecurity operations centers.
The 2015 Ukraine grid hack raised a challenge for all of them that persists to this day: How can U.S. officials help protect smaller utilities that lack the resources to defend against GRU hackers? And how can they communicate that cyberattacks are worth treating as real risks, when the power industry is already grappling with an aging workforce, pandemic-related disruptions and severe weather exacerbated by climate change?
“The threat can seem at that 10,000-foot level, where some small and medium-sized businesses are trying to survive the pandemic and ensure they have the talent on hand to complete their operations,” said Monty McGee, associate director of the Cyber Readiness Institute, which offers cybersecurity training and resources to business leaders. “What the U.S. government is doing through sector-specific agency support and webinars that communicate these things more clearly is helpful, but the distribution of that content and getting it in front of those who need it is an enduring challenge.”
McGee added in an interview this month that the 2015 Ukraine cyberattack was a “watershed moment” for cybersecurity, demonstrating that aggressive hacks could have real-world, physical impacts for hundreds of thousands of people.
“When you fast forward to today, we’ve had constant reminders that this is an issue that we cannot allow to go on cruise control or pause,” McGee said, citing the SolarWinds breach, last year’s Microsoft Exchange hack, and the Kaseya ransomware attack. “It requires constant attention, and increased focus and resources.”
Leaks and missteps
The FOIA documents highlighted the chaos and uncertainty that come with any never-before-seen hack, be it the Stuxnet worm that infected Iranian nuclear centrifuges or the DarkSide ransomware variant that forced last year’s shutdown of the Colonial fuel pipeline.
For instance, a DOE member of the U.S. delegation to Ukraine was sent home early after inadvertently exposing private documents shared by the victim utility companies. Perhaps worried about falling prey to a BlackEnergy infection — the modular malware was known to spread via Microsoft Office macros — the DOE worker uploaded sensitive files to VirusTotal to check if they were malicious. Researchers at cybersecurity firm iSight Partners picked up on the files from VirusTotal before they were taken down, using the information in their own analysis of the headline-grabbing hack.
“Fortunately, the team had built a great relationship with the Ukraine, and we made it through that situation,” noted Andy Ozment, then DHS assistant secretary for cybersecurity and communications, in a Feb. 2, 2016, email. He added that Brig. Gen. Greg Touhill — deputy assistant secretary in the same office — persuaded iSight to delay sharing information from the documents with clients.
“Fortunately, big kudos to Greg,” said Ozment, now chief technology risk officer and executive vice president at Capital One. “He reached out to iSight and because of his great (evening) work, they will hold on releasing the document.”
But word soon got out anyway of the trip and many of its key takeaways. News of a CEO-level call among utility companies and government representatives about the “Ukraine situation” leaked to Energy Daily, drawing apologies from grid executives and vexing DHS officials.
“Hence why we should have stopped discussing whether or not we should have published last week and actually published ahead of this in order to control the message,” said Marty Edwards, director of DHS’s Industrial Control Systems Cyber Emergency Response Team, referring to the agency’s initial findings about the first-of-its-kind grid cyberattack.
“Always frustrating to play catch up instead of leading like our stakeholders expect us to,” he added in an email thread about the leak.
Back for more
One key takeaway from the January 2016 fact-finding visit was that the BlackEnergy malware helped enable hackers to take down parts of Ukraine’s power grid, but did not directly cause the outages.
The finding was important because variants of the same malware had already been discovered in some U.S. utility networks over a year earlier. Were the latest “BlackEnergy3” tool equipped to cause blackouts, American electricity providers could have been in grave danger.
The distinction became even more significant in December 2016, when Russian hackers struck Ukraine’s grid yet again — this time with the custom “CrashOverride” malware tailor-made to disrupt electricity. The attack cut the lights out to a few hundred thousand people for hours, this time by targeting a transmission-level substation near Kyiv.
“CRASHOVERRIDE represents alarming tradecraft and the ability to disrupt operations, but the public must understand that the outages could be in hours or days not in weeks or months,” researchers at industrial cybersecurity firm Dragos, Inc., said in a June 2017 deep dive into the malware’s capabilities.
Unlike the catchall capabilities of BlackEnergy 3, CrashOverride was built specifically to open circuit breakers and keep them open, cutting off the flow of power.
“While the known capabilities do not appear to be U.S.-focused, it is important to recognize that the general [tactics, techniques and procedures] used in CrashOverride could be leveraged with modified technical implementations to affect U.S.-based critical infrastructure,” the Cybersecurity and Infrastructure Security Agency said in a rare alert about the malware.
Last year the U.S. government formally attributed CrashOverride to the Russian government, along with the original 2015 grid hack.
“I always felt that the attack on the Ukrainian grid in December 2015 in addition to being a warning to the Ukrainian government, was a warning to us,” Spaulding said. “I always suspected that it was in part a, ‘look what we can do, we could do this to you’ sort of shot across the bow.”
She called the 2015 attack a “wakeup call” for U.S. grid operators.
“The wakeup calls fade in the rearview mirror all too quickly, but for a while, we had the attention of our critical infrastructure owners and operators… People were taking it seriously.”
