PowerPoint Presentation

Learning to Live with an Advanced Persistent ThreatJohn DenuneIT Security DirectorUniversity of California, San Diegojdenune@ucsd.edu

ACT Infrastructure services

E-mailActive DirectoryNetworkingID ManagementSecurityTelecomData CenterDatabase AdministrationUNIX and Windows SupportDecentralized 100 OUs 800 OU Admins2ACT Security

9 StaffFirewallIntrusion DetectionVulnerability AssessmentForensicsAnti-virus and FDEPatch ManagementSSL CertsIncident ResponsePolicy and ComplianceVPNDecentralized 100 OUs 800 OU Admins3What is an APT?

Its not OpportunisticLow level criminal activitySpam, phishing, WAREZOff the shelf attacks, higher likelihood of AV detection4APTTargetedPatientSkilledTechnicalSocial EngineeringVaried AttacksPhysical threatsEspionageCorporateState-SponsoredTheftHacktivismYou have something they want and they will spend a lot of time trying to get itMonths or yearsOff the shelf, but also custom malware including zero-day Extremely low detection ratesTechnical, phishing, phone callsDropping infected USB drives in parking lot or keystroke loggers on lab keyboardsOften tied so some sort of espionage, either corporate to get insider information or state sponsored to get military infoHacktivism to expose information or long term DDOS to make a pointGood old fashioned theft5APT LifecycleComplete MissionInitial DetectionJune 2012Got luckyAV alert on ACT server.OU Admins compromised through unrelated staff account on VPNOnly 1 of 4 pieces of malware detectedChanged password, rebuild serversHappened again the following night with another unrelated VPN accountAlso found several other computers in unrelated departments, also OU admins compromisedThird time password changes and re-used

7

Initial DetectionJune 2012Got luckyAV alert on ACT server.OU Admins compromised through unrelated staff account on VPNOnly 1 of 4 pieces of malware detectedChanged password, rebuild serversHappened again the following night with another unrelated VPN accountAlso found several other computers in unrelated departments, also OU admins compromisedThird time password changes and re-used

11Lesson #1

Pay attention to anti-virus alertsToo many sysadmins view a detection as AV doing its job IF they even monitor at allModern malware loves company and almost always brings friends

12Lesson #2

Dont (completely) rely on your anti-virus productLow detection rates, especially for custom malware13Lesson #3

Where possible, track IPs instead of blocking themOnly had IP blocks 14Initial ReconFebruary 2012Initial CompromiseApril 2012

Going through org charts, reading about projects

15Gh0st RAT

Lesson #4

Make your local FBI agent your new best friendInsight into goalsAny others being attacked from same groupAssistance analyzing malwareHelp with managementThis attack is different. Not a patch, rebuild and youre doneThere are those who are hacked and know it, and those who are hacked and dontIRPS and Dali Lama17Lesson #5

Have a secure communications plan in placeSecurity staff had PGP keys but most sysadmins did notVoice mail unreliable due to unified messaging

Attackers were definitely reading e-mail18Lesson #6Log everything, especially authentication,netflow and DNS

AD logs are ugly and chatty HUGEInformation spread out over several lines using different infor (IP, system name, etc) so context is difficult

Netflow to understand where they are going within the network. VPN netflow added

DNS is HUGE but can provide a lot of insight, especially when connected through VPN.

Tremendous amount of data

19Dynamic DNS Beaconing$ nslookup host.somehackedsite.com** server can't find host.somehackedsite.com: NXDOMAIN

$ nslookup host.somehackedsite.comhost.somehackedsite.com has address 10.2.3.4

Attack timing

All attacks took place Sunday Thursday between the hours of 6pm and 3am PacificThis was somebodys jobInsight on when we could make system changes when the attackers werent active21Attack Path

Malware ObservationsYou dont need to rely on a lot of malware when youve already got a long list of credentials

You dont need to crack passwords when you can just pass a hashNTLM Authentication

User provides username and password. Client computes hash, stores it in memory and throws away the plaintext password.Client sends username to server.Server sends a challenge to the client.Client encrypts the challenge with the user hash and sends it back to the server. Server sends the username, challenge and encrypted response to the DC.DC retrieves user hash, encrypts the challenge and compares to the client encrypted response. If they match, authentication is successful.LSASS Local Security Authority Subsystem Service24Interactive Authentication

Client computes LM and NTLM hash and stores them in memory. Plaintext password is reversibly encrypted and stored in memory.Password hash is salted with username and stored in registry.LSASS Local Security Authority Subsystem Service25NTLM Authentication

Client sends username to server.LSASS Local Security Authority Subsystem Service26NTLM Authentication

Server sends a challenge to the client.LSASS Local Security Authority Subsystem Service27NTLM Authentication

Client encrypts the challenge with the user hash and sends it back to the server. LSASS Local Security Authority Subsystem Service28NTLM Authentication

Server sends the username, challenge and encrypted response to the DC.LSASS Local Security Authority Subsystem Service29NTLM Authentication

DC retrieves user hash, encrypts the challenge and compares to the client encrypted response. If they match, authentication is successful.LSASS Local Security Authority Subsystem Service30Administrator Hash

So, lets say the domain administrator RDPs to the clientDomain Admin NTLM hash now stored in client memory.Pass the Hash

Attacker compromises clientSteals hashes from memoryAccesses both server and domain controller

Pass the Hash

Attacker compromises clientSteals hashes from memoryAccesses both server and domain controller

GAME OVERMitigationsChange passwords multiple times per dayFast track two factor authenticationCompartmentalized passwordsSeparate user and admin credentialsMinimize lateral trustScan entire domain for scheduled tasksRebuild Domain ControlersAuthentications that used a hash but didnt have a corresponding interactive login34Lesson #7

Reconsider traditional password best practicesHow often do you change your password?A lot of best practice is based on outdated informationKeystroke loggers and phishing have invalidated most of that thinking

How long do you want the attackers to have access to your systems before kick them a=out and force them to reacquire creds?35Good passwords?*tecno9654postgresA Matt Hale Tribute CD would be cool..Access-Control-Allow-OriginAbundance4me2dayBulletformyvalentine123ElementarymydearwatsonPutin is nothing but commie scum.Video killed the radio star?antcolonyoptimizationEmergency ActionSeptember 2012Tried to capture e-mail of upper org chart of one of the targeted departmentsWebmail to check cred, POP to download e-mail

Swatting flies So many compromised credentials Reset the playing field37Lesson #8

Effectively and securely communicating a password change is hardMet with campus sysadmins to spread the message internallyhelpdeskCampus announcementsProminent notices on official campus web pages

Just before quarter startedFac, staff, priv role accountsAvoid Sept 11.35000 accounts, Many disabled outright as not been used5 day rolling disable

Huge phishing in the following weeksTry a few, and back off38We are not alone

Not just a windows problemSome backlash on whether AD could be trustedAs we starting protecting more and more AD creds, attackers tried local accounts in an attempt to hide their activity39ReengagementJuly 2013

ACTParting ThoughtsDetection can be subtle and an artHave a good AD TeamLogging visibility is essentialRegular password changes are a MUSTBe prepared to re-image any systemFirewalls to prevent lateral movementSeparation of user and admin credentialsRequire two-factor for OU Admins

FBI has now confirmed other activity from this particular group

42A New Hope

A New HopeStrengthened LSASS to prevent credential dumpsMany processes no longer store credentials in memoryBetter ways to restrict local account use over the networkRDP use without putting the credentials on the remote computerAddition of a new Protected Users group, whose members' credentials cannot be used in remote PtH attacksFurther ReadingKnow Your Digital Enemy Anatomy of a Gh0st RAThttp://www.mcafee.com/us/resources/white-papers/foundstone/wp-know-your-digital-enemy.pdf

Mitigating Pass-the-Hash (PtH) Attacks and Other Credential Theft Techniqueshttp://www.microsoft.com/en-us/download/details.aspx?id=36036

APT1: Exposing One of China's Cyber Espionage Unitshttp://intelreport.mandiant.com/Mandiant_APT1_Report.pdf

45If ignorant both of your enemy and yourself, you are certain to be in peril. Sun Tzu, The Art of War1462.8567