Big Russian hack used a technique experts had warned about for years. Why wasn’t the U.S. government ready? Hackers got long-term, wide-ranging access to government and private networks by manipulating the software that vouches for those allowed inside.

(Washington Post Illustration/iStock) By Craig Timberg Feb. 9, 2021 at 10:21 p.m. GMT+1 The disastrous Russian hack of federal government networks last

year relied on a powerful new trick: Digital spies penetrated so

deeply that they were able to impersonate any user they wanted. It

was the computer network equivalent of sneaking into the State

Department and printing perfectly forged U.S. passports.

Cybersecurity researchers had warned for years that such an attack

was possible. Those from one firm, FireEye, even released hacking

tools in 2019 showing exactly how to do it — in hopes that the

revelation would spur the widespread deployment of better

defenses.

It didn’t.

Now there is urgent debate within cybersecurity circles about how

best to respond to the hack, which was so extensive that experts

describe it as historic.

Some are calling for stronger walls to keep out would-be

intruders or better burglar alarms to alert network administrators

that a hack had begun. Others, arguing that there’s no practical way

to keep the most sophisticated hackers from breaking into

important networks, say the smarter investment would be in

building better tools for hunting and ejecting intruders once they

inevitably get past security perimeters. AD

Meanwhile, questions remain about why this surge of corrective

action didn’t happen earlier for a type of hack that had been

discussed for years within cybersecurity circles and whether, even

now, the potential solutions are being deployed widely enough to

prevent future catastrophes.

Two months after the hack was discovered in December,

cybersecurity researchers say spies are probably still active in some

of the hundreds of breached networks. Victims included the

departments of State, Treasury, Homeland Security, Energy and

Commerce, and the National Institutes of Health and the National

Nuclear Security Administration. Also penetrated were private

companies in the consulting, technology, telecom, and oil and gas

companies in North America, Europe, Asia and the Middle East, as

well as FireEye itself, which first reported the attack on Dec. 8.

Russian hack was ‘classic espionage’ with stealthy, targeted tactics

The Russians used a variety of sophisticated tricks to penetrate the

networks in last year’s attack. But once inside, they often

manipulated a piece of Microsoft software, Active Directory

Federation Services, that vouches for the identities of authorized

users by issuing digital identity documents called “SAML tokens.”

An Israeli researcher had first described this technique, dubbed

a “Golden SAML Attack,” in 2017, but it had not been seen in a

major network intrusion until now, experts say. AD

Such systems for authenticating users are essential to securing the

cloud services used widely by government agencies, corporations,

hospitals, universities and most other places where people

collaborate across long distances. And the ability to forge SAML

tokens lets hackers roam widely among these cloud-based services,

while also minimizing the chances of getting quickly caught.

“All of this outward security doesn’t mean squat if you don’t have

this one thing locked up,” said Matthew D. Green, a Johns Hopkins

cybersecurity and cryptology expert. “This is crazy."

Authenticated SAML (rhymes with “camel”) tokens let intruders

move easily among the computer systems affiliated with an

organization, even if the individual elements are run by different

companies, such as Microsoft, Amazon Web Services or Dropbox.

Hackers can present these tokens as they seek access to different

troves of valuable data — email, document repositories, payroll

systems — while sidestepping common defensive measures, such as

strong passwords and two-factor authentication. AD

There are possible protections against a Golden SAML Attack,

including securing the encryption keys that create the tokens in

their own, well-defended piece of hardware, or sharply limiting who

has high-level access to the computers authorized to issue tokens.

Alerts warning of unusual behavior might help defenders act more

quickly, and more extensive logging could help the detective work

after signs of trouble are detected.

Former National Security Agency hacker Jake Williams said his

security consultancy has been helping clients respond to the recent

Russian attack. But even now, it’s not entirely clear to him which

defenses are best suited to prevent a repeat, given the sophistication

of the attackers, which U.S. officials have said were from the SVR,

Russian’s foreign-intelligence service. He favors bolstering systems

for detecting intruders once they’re inside.

“We are not going to keep a nation-state attacker who has targeted

you out,” said Williams, president of Rendition Infosec. “They are

going to outfox you.” AD

The question then becomes: How best to keep a network intrusion

from becoming a catastrophe?

Why didn’t anyone do something sooner? As this debate plays out within the cybersecurity community, Sen.

Ron Wyden (D-Ore.), a member of the Senate Intelligence

Committee, asked Microsoft and FireEye last month to explain how

a security weakness publicized years ago was not addressed before

the Russians took advantage of it. Microsoft released a tool to help

detect such attacks less than two weeks after the Russian hack was

publicly revealed.

“The American people deserve to know why hackers were able to

steal encryption keys from the U.S. government without anyone

noticing,” Wyden said in a statement to The Washington Post. “I

want to know why Microsoft didn’t provide its customers with tools

to better protect and detect the theft of encryption keys, and why

government agencies failed to deploy their own defenses. I’m also

interested in what steps FireEye took to warn Microsoft, its

customers and the U.S. government about a vulnerability it knew

about nearly two years ago.”

The U.S. government spent billions on a system for detecting

hacks. The Russians outsmarted it.

Both companies defended their handling of the Golden SAML

Attack in replies to Wyden — FireEye by letter, Microsoft by video

call — according to a Wyden aide who spoke on the condition of

anonymity to discuss communications not yet made public. AD

In comments to The Post, the companies noted the multiple

weaknesses the Russians exploited in their intrusions and also the

difficulty creating effective defenses against hackers who already

have penetrated networks so deeply that they can issue their own

SAML tokens. Both companies said better overall security practices

are key to defending against this and other attacks, ideally before

the initial intrusions succeed.

John Lambert, the head of Microsoft’s Threat Intelligence Center,

said in an interview that the company long has recommended

security measures that might have thwarted the Russians, such as

stand-alone hardware to guard encryption keys, and that handling

the issuance of SAML tokens through a cloud service, such as

Microsoft’s Azure, would offer increased protection and potentially

make hacks easier to detect.

He also said that some of the measures now under discussion

among independent cybersecurity experts — such as installing the

hardware modules Microsoft recommends for protecting encryption

keys — would make a Golden SAML Attack harder to execute in the

future. AD

“Defending identity has always been foundational,” Lambert said. “I

think if you go back to any set of attacks at any point of time in the

past, compromise of identities, abuse of identities, has always been

a common element. … Securing identities and the secrets that

underpin them have always been important."

FireEye’s role in publicizing the Golden SAML Attack was

highlighted in a Microsoft post that specifically cited one of the

hacking tools FireEye released in 2019, ADFSDump. The post said

that Microsoft’s Defender software could, as of Dec. 20, send alerts

when it detected ADFSDump and called it “the initial tool used” in

the Russian hacks.

Microsoft later revised this characterization after The Post

questioned FireEye about it, saying that the Russians used a

hacking tool resembling ADFSDump but that it was unclear

whether ADFSDump itself was the one. The company’s updated

version of the post removed the reference to the FireEye tool, saying

instead that Microsoft’s Defender software now had an alert to

“detect techniques used to obtain the information needed in order

to generate security tokens," as happened in last year’s Russian

attack. AD

FireEye acknowledged that its engineers had raised alarm about

Golden SAML Attacks and released a pair of hacking tools to exploit

it during a security conference in Germany in March 2019. But the

company said it found no evidence that these tools were used by the

Russians, though it couldn’t rule out the possibility. The goal of

releasing such tools is to help “red teams” of cybersecurity

researchers probe networks for flaws that can be corrected before

malicious hackers exploit them.

“FireEye develops red team tools to help improve enterprise

cybersecurity by demonstrating the impacts of successful attacks

and by showing the defenders … how to counter them in an

operational environment,” said Dan Wire, vice president of global

communications for FireEye. “Like many security companies, we

have an internal process for responsibly releasing tools, and we

review each release on a case-by-case basis.”

Russian government hackers are behind a broad espionage

campaign that has compromised U.S. agencies, including Treasury

and Commerce

The U.S. government response to the Russian hack, meanwhile,

came under fire Tuesday when the heads of the Senate Intelligence

Committee, Chairman Mark R. Warner (D-Va.) and Vice Chairman

Marco Rubio (R-Fla.), sent a letter to the heads of the FBI, National

Security Agency and other federal agencies demanding the

appointment of “a clear leader” to coordinate the response. AD

“The federal government’s response so far has lacked the leadership

and coordination warranted by a significant cyber event, and we

have little confidence that we are on the shortest path to recovery,”

they wrote.

Russian spies began their attack by hacking SolarWinds, a Texas-

based maker of network-monitoring software, and slipping what

security experts call a “Trojan horse” into the networks of the

company’s many clients during routine software updates. Once

inside, the hackers roamed unchecked for months and might have

stayed even longer had FireEye not found them within their own

systems in December. That discovery triggered detection of the

much wider, more troubling federal hack days later.

Many experienced network defenders point to the introduction of

the Solar Winds trojan — in what’s called a “supply-chain attack” —

as the problem most urgently demanding attention because federal

government systems rely on software produced by many private

companies, each of which offers targets for malicious hackers. Once

they get inside, experts say, there are numerous options, beyond

just a Golden SAML Attack, to exploit a network’s systems for

verifying user identities.

“There are literally dozens upon dozens of ways," said Dmitri

Alperovitch, who co-founded cybersecurity firm CrowdStrike and

now is executive chairman of Silverado Policy Accelerator, a think

tank. “No one can possibly defend against all of them. … The idea

that we should be chasing every single attack vector is a

wrongheaded approach.”

Early alarm brought no response Shaked Reiner, an Israeli cybersecurity expert who described the

Golden SAML Attack in a 2017 blog post, said the method offers

important advantages to hackers — namely its potential to enable

unusually wide-ranging, long-lasting and hard-to-detect intrusions

that may merit more robust defenses.

The initial blog post, made on the site of his employer, CyberArk

Labs, initially generated only modest attention. News of the Russian

hack, three years later, changed that. The National Security Agency

cited Reiner’s post in its advisory on how to detect such

intrusions on Dec. 17.

“Right away, we understood. This is what we were talking about,”

Reiner said.

He added that hackers deploying the Golden SAML Attack “can

pretty much impersonate any user in a network. … Detecting this

type of attack can be extremely difficult.”

Some experts, including Green at Johns Hopkins, argue that

sensitive government networks should invest in computer

equipment called “hardware secure modules” that would house the

encryption keys used to issue SAML tokens, making them almost

impossible to steal. This equipment is expensive, ranging from tens

to hundreds of thousands of dollars, and can add significant

complexity to the operation of cloud-computing networks — factors

that have been barriers to their widespread adoption.

Federal investigators find evidence of previously unknown tactics

used to penetrate government networks

Another approach would be to specify a small number of computers

— perhaps ones at the physical desks of system administrators —

that can gain high-level access to the identity-management software

itself. Even a skilled hacker, for example, would find it much harder

to execute a Golden SAML Attack from Moscow if only a handful of

computers were vulnerable to such manipulation from afar. Even

then, key computers could be left disconnected from the Internet,

adding more barriers to hackers operating remotely.

Some other experts, however, say that even without actually stealing

the encryption keys for issuing SAML tokens, hackers can still find

ways to manipulate network identities in ways that allow them to

expand and prolong intrusions.

Williams, of Rendition Infosec, said, “I agree that Microsoft could

have done a better job of detecting any number of active-directory

weaknesses or the exploitation of those weaknesses.”

But he added that more aggressive action by Microsoft, FireEye or

others would have been unlikely to thwart the Russians, given their

skills and resources.

“I’m confident that wouldn’t have changed the outcome here,”

Williams said.

The most viable solution for the future, some experts say, may be in

better alarms to rapidly alert defenders to suspicious behavior,

along with more extensive network logging of network activities —

preferably activated by default — to assist the detective work after

hacks inevitably occur.

CORRECTION: A previous version of this story said incorrectly

that Sen. Ron Wyden sent letters to FireEye and Microsoft last

month asking for answers related to the Russian attack. But in fact

only FireEye received a letter. The communication with Microsoft

was oral. 111 Comments

Craig Timberg Craig Timberg is a national technology reporter for The Washington Post. Since joining The Post in 1998, he has been a reporter, editor and foreign correspondent, and he contributed to The Post’s Pulitzer Prize-winning coverage of the National Security Agency.Follow

More from The Post

• Capitol riot defendants facing jail have regrets. Judges aren’t buying it.

•

Intermittent fasting and why when you eat makes a difference

• As Texans went without heat, light or water, some companies scored a big payday

• Perspective Carolyn Hax: A father relocating for work wonders how he can show up for his young children

• ‘It’s Donald Trump’s party’: How the former president is building a political operation to cement his hold on the GOP