Agenda
• ThreatHutingBasics
• ThreatHuntingDataSources
• Sysmon EndpointData
• CyberKillChain
• WalkthroughofAttackScenarioUsingCoreSplunk (handson)
• EnterpriseSecurityWalkthrough
• ApplyingMachineLearningandDataSciencetoSecurity
LogInCredentials
January,February&March https://od-norcal-2.splunkoxygen.comApril,May&June https://od-norcal-3.splunkoxygen.comJuly,August&September https://od-norcal-4.splunkoxygen.comOctober,November&December https://od-norcal-5.splunkoxygen.com
User:hunterPass:pr3dator
BirthMonth
AmIintherightplace?
Somefamiliaritywith…
● CSIRT/SOCOperations
● GeneralunderstandingofThreatIntelligence
● GeneralunderstandingofDNS,Proxy,andEndpointtypesofdata
5
Thisisahands-onsession.
Theoverviewslidesareimportantforbuildingyour“hunt”methodology
10minutes- Seriously.
SANSThreatHuntingMaturity
8
AdHocSearch
StatisticalAnalysis
VisualizationTechniques
Aggregation Machine Learning/DataScience
85%55%50%48%32%
Source:SANSIR&ThreatHuntingSummit2016
HuntingTools:InternalData
9
• IPAddresses:threatintelligence,blacklist,whitelist, reputationmonitoringTools:Firewalls, proxies, Splunk Stream,Bro,IDS
• NetworkArtifactsandPatterns:networkflow,packetcapture,activenetworkconnections, historicnetworkconnections, portsandservicesTools:Splunk Stream,BroIDS,FPC,Netflow
• DNS:activity,queries andresponses, zonetransferactivityTools:Splunk Stream,BroIDS,OpenDNS
• Endpoint– HostArtifactsandPatterns:users,processes, services, drivers, files, registry,hardware,memory, disk activity,filemonitoring: hashvalues, integritycheckingandalerts,creationordeletionTools:Windows/Linux, CarbonBlack,Tanium, Tripwire,ActiveDirectory
• VulnerabilityManagementDataTools:TripwireIP360,Qualys, Nessus
• UserBehaviorAnalytics:TTPs,usermonitoring, timeofdaylocation,HRwatchlistSplunk UBA,(All oftheabove)
LogInCredentialsJanuary,February&March https://od-norcal-2.splunkoxygen.com
April,May&June https://od-norcal-3.splunkoxygen.com
July,August&September https://od-norcal-4.splunkoxygen.com
October,November&December https://od-norcal-5.splunkoxygen.com
User:hunterPass:pr3dator
Endpoint:MicrosoftSysmonPrimer
11
● TAAvailable ontheAppStore
● GreatBlogPosttogetyoustarted
● Increases thefidelity ofMicrosoftLogging
BlogPost:http://blogs.splunk.com/2014/11/24/monitoring-network-traffic-with-sysmon-and-splunk/
January,February&March https://od-norcal-2.splunkoxygen.comApril,May&June https://od-norcal-3.splunkoxygen.comJuly,August&September https://od-norcal-4.splunkoxygen.comOctober,November&December https://od-norcal-5.splunkoxygen.com
User:hunterPass:pr3dator
DemoStory- KillChainFrameworkSuccessfulbruteforce– downloadsensitivepdfdocument
Weaponize thepdffilewithZeusMalware
Convincingemailsentwithweaponizedpdf
Vulnerablepdfreaderexploitedbymalware.Dropper createdonmachine
Dropper retrievesandinstallsthemalware
Persistence viaregularoutboundcomm
DataExfiltration
Source:LockheedMartin
Servers
Storage
DesktopsEmail Web
TransactionRecords
NetworkFlows
DHCP/DNS
HypervisorCustomApps
T
PhysicalAccess
Badges
ThreatIntelligence
Mobile
CMDB
IntrusionDetection
Firewall
DataLossPrevention
Anti-Malware
VulnerabilityScans
Traditional
Authentication
StreamInvestigations– chooseyourdatawisely
19
APTTransactionFlowAcrossDataSources
21
http(proxy)sessiontocommand&controlserver
RemotecontrolStealdataPersistincompanyRentasbotnet
Proxy
ConductBusiness
Createadditionalenvironment
GainAccesstosystemTransaction
ThreatIntelligence
Endpoint
NetworkEmail,Proxy,DNS,andWeb
DataSources
.pdf executes&unpacksmalwareoverwritingandrunning“allowed”programs
Svchost.exe(malware)
Calc.exe(dropper)
AttackerhackswebsiteSteals.pdf files
WebPortal
Attackercreatesmalware,embed in.pdf,
emailstothetarget
Reademail,openattachment
OurInvestigationbeginsbydetectinghighriskcommunicationsthroughtheproxy,attheendpoint, andevenaDNScall.
Tobeginourinvestigation,wewillstartwithaquicksearchtofamiliarizeourselveswiththedatasources.
Inthisdemoenvironment,wehaveavarietyofsecurityrelevantdataincluding…
WebDNSProxyFirewallEndpointEmail
Takealookattheendpointdatasource. WeareusingtheMicrosoftSysmon TA.
Wehaveendpointvisibilityintoallnetworkcommunicationandcanmapeachconnectionbacktoaprocess.
}Wealsohavedetailedinfooneachprocessandcanmapitbacktotheuserandparentprocess.}
Letsgetourdaystartedbylookingusingthreatintel toprioritizeoureffortsandfocusoncommunicationwithknown highriskentities.
WehavemultiplesourceIPscommunicatingtohighriskentitiesidentifiedbythese2threatsources.
Weareseeinghighriskcommunicationfrommultipledatasources.
Weseemultiplethreatintel relatedeventsacrossmultiplesourcetypesassociatedwiththeIPAddressofChrisGilbert.Let’stakecloserlookattheIPAddress.
Wecannowseetheownerofthesystem(ChrisGilbert)andthatitisn’taPIIorPCIrelatedasset,sotherearenoimmediatebusinessimplicationsthatwouldrequireinformingagencies orexternal customerswithinacertaintimeframe.
Thisdashboardisbasedoneventdatathatcontainsathreatintelbasedindicatormatch(IPAddress,domain,etc.).ThedataisfurtherenrichedwithCMDBbasedAsset/identityinformation.
Wearenowlookingatonlythreatintel related activityfortheIPAddressassociatedwithChrisGilbertandseeactivityspanningendpoint,proxy,andDNSdatasources.
These trendlinestellaveryinterestingvisualstory.ItappearsthattheassetmakesaDNSqueryinvolvingathreatintel relateddomainorIPAddress.
ScrollDo
wn
Scrolldownthedashboardtoexamine thesethreatintel eventsassociatedwiththeIPAddress.
Wethenseethreatintel relatedendpointandproxyeventsoccurringperiodicallyandlikelycommunicatingwithaknownZeusbotnetbasedonthethreatintelsource(zeus_c2s).
It’sworthmentioningthatatthispointyoucouldcreateatickettohavesomeonere-image themachinetopreventfurtherdamageaswecontinueourinvestigationwithinSplunk.
Withinthesamedashboard,wehaveaccesstoveryhighfidelityendpointdatathatallowsananalysttocontinuetheinvestigationinaveryefficientmanner.Itisimportanttonotethatnearreal-timeaccesstothistypeofendpointdataisnotnotcommonwithinthetraditionalSOC.
Theinitialgoaloftheinvestigationistodeterminewhetherthiscommunicationismaliciousorapotentialfalsepositive.Expandtheendpointeventtocontinuetheinvestigation.
Proxyrelated threatintel matchesareimportantforhelpingustoprioritizeoureffortstowardinitiatinganinvestigation.Furtherinvestigationintotheendpointisoftenverytimeconsumingandofteninvolvesmultipleinternalhand-offstootherteamsorneeding toaccess additionalsystems.Thisencryptedproxytrafficisconcerningbecauseofthelargeamountofdata(~1.5MB)beingtransferredwhichiscommonwhendataisbeingexfiltrated.
Exfiltrationofdataisaseriousconcern andoutboundcommunicationtoexternal entitythathasaknownthreatintelindicator,especiallywhenitisencrypted asinthiscase.
Letscontinuetheinvestigation.
Anotherclue.Wealsoseethatsvchost.exe shouldbelocatedinaWindowssystemdirectorybutthisisbeingrunintheuserspace.Notgood.
Weimmediatelyseetheoutboundcommunicationwith115.29.46.99viahttpsisassociatedwiththesvchost.exeprocessonthewindowsendpoint. Theprocessidis4768.There isagreatdealmoreinformationfromtheendpointasyouscrolldownsuchastheuserIDthatstartedtheprocessandtheassociatedCMDBenrichment information.
WehaveaworkflowactionthatwilllinkustoaProcessExplorerdashboardandpopulateitwiththeprocessidextracted fromtheevent(4768).
ThisisastandardWindowsapp,butnotinitsusualdirectory, tellingusthatthemalwarehasagainspoofedacommonfilename.
Wealsocanseethattheparentprocessthatcreated thissuspicuous svchost.exe processiscalledcalc.exe.
ThishasbroughtustotheProcessExplorer dashboardwhichletsusviewWindowsSysmon endpointdata.
SuspectedMalware
Letscontinuetheinvestigationbyexaminingtheparentprocessasthisisalmostcertainlyagenuine threatandwearenowworkingtowardarootcause.
ThisisveryconsistentwithZeusbehavior.Theinitialexploitationgenerally createsadownloaderordropperthatwillthendownloadtheZeusmalware.Itseemslikecalc.exemaybethatdownloader/dropper.
SuspectedDownloader/Dropper
Thisprocesscallsitself“svchost.exe,”acommonWindowsprocess, butthepathisnotthenormalpathforsvchost.exe.
…whichisacommontraitofmalwareattemptingtoevadedetection.WealsoseeitmakingaDNSquery(port53)thencommunicatingviaport443.
TheParentProcessofoursuspecteddownloader/dropper isthelegitimatePDFReaderprogram.Thiswilllikelyturnouttobethevulnerable appthatwasexploitedinthisattack.
SuspectedDownloader/Dropper
SuspectedVulnerableAppWehaveveryquicklymovedfromthreatintel relatednetwork andendpointactivitytothelikelyexploitationofavulnerableapp.Clickontheparentprocesstokeepinvestigating.
WecanseethatthePDFReaderprocesshasnoidentifiedparentandistherootoftheinfection.
ScrollDo
wn
Scrolldownthedashboardtoexamine activityrelated tothePDFreaderprocess.
Chrisopened2nd_qtr_2014_report.pdfwhichwasanattachmenttoanemail!
Wehaveourrootcause!Chrisopened aweaponized .pdf filewhichcontained theZeusmalware. Itappearstohavebeendelivered viaemailandwehaveaccesstoouremaillogsasoneofourimportantdatasources.Letscopythefilename2nd_qtr_2014_report.pdf andsearchabitfurthertodeterminethescopeofthiscompromise.
Letssearchthoughmultipledatasourcestoquicklygetasenseforwhoelsemayhavehavebeenexposed tothisfile.
Wewillcomebacktothewebactivitythatcontainsreference tothepdf filebutletsfirstlookattheemaileventtodetermine thescopeofthisapparentphishingattack.
Wehaveaccesstotheemailbodyandcanseewhythiswassuchaconvincingattack.Thesenderapparentlyhadaccesstosensitiveinsiderknowledge andhintedatquarterlyresults.
There isourattachment.
HoldOn!That’snotourDomainName!Thespellingisclosebutit’smissinga“t”.Theattackerlikelyregistered adomainnamethatisveryclosetothecompanydomainhopingChriswouldnotnotice.
Thislookstobeaverytargetedspearphishingattackasitwassenttoonlyoneemployee (Chris).
RootCauseRecap
36
DataSources
.pdf executes&unpacksmalwareoverwritingandrunning“allowed”programs
http(proxy)sessiontocommand&controlserver
RemotecontrolStealdataPersistincompanyRentasbotnet
Proxy
ConductBusiness
Createadditionalenvironment
GainAccesstosystemTransaction
ThreatIntelligence
Endpoint
NetworkEmail,Proxy,DNS,andWeb
.pdfSvchost.exe(malware)
Calc.exe(dropper)
AttackerhackswebsiteSteals.pdf files
WebPortal
Attackercreatesmalware,embed in.pdf,
emailstothetarget
Reademail,openattachment
Weutilizedthreatintel todetectcommunicationwithknownhighriskindicatorsandkickoffourinvestigationthenworkedbackwardthroughthekillchaintowardarootcause.
Keytothisinvestigativeprocessistheabilitytoassociatenetworkcommunicationswithendpointprocessdata.
Thishighvalueandveryrelevantabilitytoworkamalwarerelated investigationthroughtorootcausetranslatesintoaverystreamlinedinvestigativeprocesscomparedtothelegacy SIEMbasedapproach.
37
Letsrevisitthesearchforadditionalinformationonthe2nd_qtr_2014-_report.pdf file.
Weunderstandthatthefilewasdeliveredviaemailandopenedattheendpoint.Whydoweseeareference tothefileintheaccess_combined (webserver) logs?
Selecttheaccess_combinedsourcetype toinvestigatefurther.
38
Theresultsshow54.211.114.134hasaccessed thisfilefromthewebportalofbuttergames.com.
There isalsoaknownthreatintelassociationwiththesourceIPAddressdownloading (HTTPGET)thefile.
39
SelecttheIPAddress,left-click,thenselect“New search”.WewouldliketounderstandwhatelsethisIPAddresshasaccessed intheenvironment.
40
That’sanabnormallylargenumberofrequestssourcedfromasingleIPAddressina~90minutewindow.
Thislookslikeascriptedactiongiventheconstanthighrateofrequestsoverthebelowwindow.
ScrollDo
wn
Scrolldownthedashboardtoexamineotherinterestingfieldstofurtherinvestigate.
NoticetheGooglebotuseragent string whichisanotherattempttoavoidraisingattention..
41
Therequestsfrom52.211.114.134aredominatedbyrequeststotheloginpage(wp-login.php). It’sclearlynotpossibletoattemptaloginthismanytimesinashortperiodoftime– thisisclearlyascriptedbruteforceattack.
Aftersuccessfullygainingaccesstoourwebsite, theattackerdownloaded thepdf file,weaponized itwiththezeusmalware, thendelivered ittoChrisGilbertasaphishingemail.
Theattackerisalsoaccessingadminpageswhichmaybeanattempttoestablishpersistence viaabackdoorintotheweb site.
KillChainAnalysisAcrossDataSources
42
http(proxy)sessiontocommand&controlserver
RemotecontrolStealdataPersistincompanyRentasbotnet
Proxy
ConductBusiness
Createadditionalenvironment
GainAccesstosystemTransaction
ThreatIntelligence
Endpoint
NetworkEmail,Proxy,DNS,andWeb
DataSources
.pdf executes&unpacksmalwareoverwritingandrunning“allowed”programs
Svchost.exe(malware)
Calc.exe(dropper)
AttackerhackswebsiteSteals.pdf files
WebPortal
Attackercreatesmalware,embed in.pdf,
emailstothetarget
Reademail,openattachment
Wecontinuedtheinvestigationbypivotingintotheendpointdatasourceandusedaworkflowactiontodeterminewhichprocessontheendpointwasresponsible fortheoutboundcommunication.
WeBeganbyreviewingthreatintel relatedeventsforaparticularIPaddressandobservedDNS,Proxy,andEndpointeventsforauserinSales.
Investigationcomplete!LetsgetthisturnedovertoIncident Reponse team.
Wetracedthesvchost.exeZeusmalwarebacktoit’sparentprocessIDwhichwasthecalc.exedownloader/dropper.
Onceourrootcauseanalysiswascomplete,weshiftedoutfocusintotheweblogstodetermine thatthesensitivepdffilewasobtainedviaabruteforceattackagainstthecompanywebsite.
Wewere abletoseewhichfilewasopenedbythevulnerable appanddetermined thatthemaliciousfilewasdeliveredtotheuserviaemail.
Aquicksearchintothemaillogsrevealed thedetailsbehindthephishingattackandrevealed thatthescopeofthecompromisewaslimitedtojusttheoneuser.
Wetracedcalc.exe backtothevulnerable applicationPDFReader.
WanttoFollowAlong?
● DownloadSplunk6.4.2http://www.splunk.com/en_us/download-21.html
● Download&InstalltheMachineLearningToolkithttp://tiny.cc/splunkmlapp
Filterable
KSIsspecifictoRisk
Riskassignedtosystem,userorother
UnderAdvancedThreat,selectRiskAnalysis
Filterable, downtoIoC
KSIsspecifictoThreat
Mostactivethreatsource
Scrolldown… Scroll
UnderAdvancedThreat,selectThreatActivity
ArtifactCategories –clickdifferent tabs…
STIXfeed
Customfeed
UnderAdvancedThreat,selectThreatArtifacts
DatafromassetframeworkConfigurable Swimlanes
Darker=more events
AllhappenedaroundsametimeChangeto“Today”ifneeded
AssetInvestigator,enter“192.168.56.102”
Supervised MachineLearning
61
DomainName TotalCnt RiskFactor AGD SessionTime RefEntropy NullUa Outcome
yyfaimjmocdu.com 144 6.05 1 1 0 0 Maliciousjjeyd2u37an30.com 6192 5.05 0 1 0 0 Maliciouscdn4s.steelhousemedia.com 107 3 0 0 0 0 Benignlog.tagcade.com 111 2 0 1 0 0 Benigngo.vidprocess.com 170 2 0 0 0 0 Benignstatse.webtrendslive.com 310 2 0 1 0 0 Benigncdn4s.steelhousemedia.com 107 1 0 0 0 0 Benignlog.tagcade.com 111 1 0 1 0 0 Benign
Unsupervised MachineLearning
• Notuning
• Programmaticallyfindstrends
• UBAisprimarilyunsupervised
• Rigorouslytestedforfit
63
AlgorithmRawSecurityData AutomatedClustering
MLToolkit&Showcase• SplunkSupportedframeworkforbuildingMLApps
– Getitforfree:http://tiny.cc/splunkmlapp
• LeveragesPythonforScientificComputing (PSC)add-on:– Open-sourcePythondatascienceecosystem– NumPy,SciPy,scitkit-learn,pandas,statsmodels
• Showcaseusecases:PredictHardDriveFailure,ServerPowerConsumption,ApplicationUsage,CustomerChurn&more
• Standardalgorithmsoutofthebox:– Supervised:LogisticRegression,SVM,LinearRegression,RandomForest,etc.– Unsupervised: KMeans,DBSCAN,SpectralClustering,PCA,KernelPCA,etc.
• Implementoneof300+algorithmsbyeditingPythonscripts
Splunk UBAUseCases
ACCOUNTTAKEOVER• Privilegedaccountcompromise• Dataexfiltration
LATERALMOVEMENT• Pass-the-hashkillchain• Privilegeescalation
SUSPICIOUS ACTIVITY• Misuseofcredentials• Geo-locationanomalies
MALWARE ATTACKS• Hiddenmalwareactivity
BOTNET,COMMAND&CONTROL• Malwarebeaconing• Dataleakage
USER&ENTITY BEHAVIORANALYTICS• Suspiciousbehaviorbyaccountsor
devices
EXTERNAL THREATSINSIDERTHREATS
SplunkUserBehaviorAnalytics(UBA)• ~100%ofbreachesinvolvevalidcredentials(Mandiant Report)• Needtounderstandnormal&anomalousbehaviorsforALLusers• UBAdetectsAdvancedCyberattacks andMaliciousInsiderThreats• LotsofMLunderthehood:
– BehaviorBaselining&Modeling– AnomalyDetection(30+models)– AdvancedThreatDetection
• E.g.,DataExfil Threat:– “Sawthisstrangelogin&datatransferfor userkwestin
at3aminChina…”– SurfacethreattoSOCAnalysts
Workflow
Raw Events
1
Statistical methods
Security semantics
2 Threat ModelsLateralmovement
ML
Patterns
Sequences
Beaconing
Land-speedviolation
Threats
Kill chain sequence
5
Supporting evidence
Threat scoring
Graph Mining
4
Con
tinuo
us s
elf-l
earn
ing
Anomalies graph
Entity relationship graph
3
Anomalies
SecurityWorkshops
● SecurityReadinessAssessments● Splunk UBADataScienceWorkshop● EnterpriseSecurityBenchmarkAssessment