Upload
gabrielle-knowles
View
27
Download
1
Embed Size (px)
Citation preview
Copyright©2016SplunkInc.
MachineLearningAndrewPhillipsSr.SalesEngineer
3
DisclaimerDuringthecourseofthispresentation,wemaymakeforwardlookingstatementsregardingfuture
eventsortheexpectedperformanceofthecompany.Wecautionyouthatsuchstatementsreflectourcurrentexpectationsandestimatesbasedonfactorscurrentlyknowntousandthatactualeventsorresultscoulddiffermaterially.Forimportantfactorsthatmaycauseactualresultstodifferfromthose
containedinourforward-lookingstatements,pleasereviewourfilingswiththeSEC.Theforward-lookingstatementsmadeinthethispresentationarebeingmadeasofthetimeanddateofitslivepresentation.Ifreviewedafteritslivepresentation,thispresentationmaynotcontaincurrentoraccurateinformation.
Wedonotassumeanyobligationtoupdateanyforwardlookingstatementswemaymake.
Inaddition,anyinformationaboutourroadmapoutlinesourgeneralproductdirectionandissubjecttochangeatanytimewithoutnotice. Itisforinformationalpurposesonlyandshallnot,beincorporatedintoanycontractorothercommitment.Splunkundertakesnoobligationeithertodevelopthefeatures
orfunctionalitydescribedortoincludeanysuchfeatureorfunctionality inafuturerelease.
Copyright©2016SplunkInc.
WhydoweneedML?
5
MLinEverydaylife
Copyright©2016SplunkInc.
HistoricalData Real-timeData StatisticalModels
DB,Hadoop/S3/NoSQL, Splunk MachineLearning
T– afewdays T+afewdays
Whyisthissochallengingusingtraditionalmethods?
• DATAISSTILLINMOTION,stillinaBUSINESS PROCESS.• Enrichreal-timeMACHINEDATAwithstructuredHISTORICALDATA• Make decisionsINREALTIME usingALLTHEDATA• CombineLEADINGandLAGGINGINDICATORS (KPIs)
Splunk
SecurityOperationsCenter
NetworkOperationsCenter
BusinessOperationsCenter
Copyright©2016SplunkInc.
WhatisML?
8
ML101:Whatisit?• MachineLearning(ML)isaprocessforgeneralizingfromexamples
– Examples=exampleor“training”data– Generalizing=building“statisticalmodels”tocapturecorrelations– Process=MLisneverdone,youmustkeepvalidating&refittingmodels
• SimpleMLworkflow:– Exploredata– FITmodelsbasedondata– APPLYmodelsinproduction– Keepvalidatingmodels
“Allmodelsarewrong,butsomeareuseful.”- GeorgeBox
9
TypesofMachineLearning1.Supervised Learning: generalizingfromlabeled data
10
TypesofMachineLearning2.Unsupervised Learning: generalizingfromunlabeled data
11
TypesofMachineLearning3.ReinforcementLearning:generalizingfromrewards intime
Leitner System Recommendersystems
Copyright©2016SplunkInc.
MLUseCases
13
ITOps:PredictiveMaintenance
1. Getresourceusagedata(CPU,latency,outagereports)
2. Exploredata,andfitpredictivemodelsonpast/real-timedata
3. Apply&validatemodelsuntilpredictionsareaccurate
4. Forecastresourcesaturation,demand&usage
5. SurfaceincidentstoITOps,whoINVESTIGATES&ACTS
Problem:Networkoutagesandtruckrollscausebigtime&moneyexpenseSolution: Buildpredictivemodeltoforecastoutagescenarios,actpre-emptively&learn
14
Security:FindInsiderThreatsProblem:Securitybreachescausebigtime&moneyexpenseSolution: Buildpredictivemodeltoforecastthreatscenarios,actpre-emptively&learn
1. Getsecuritydata(datatransfers,authentication,incidents)
2. Exploredata,andfitpredictivemodelsonpast/real-timedata
3. Apply&validatemodelsuntilpredictionsareaccurate
4. Forecastabnormalbehavior,riskscores¬ableevents
5. SurfaceincidentstoSecurityOps,whoINVESTIGATES&ACTS
15
BusinessAnalytics:PredictCustomerChurnProblem:Customerchurncausesbigtime&moneyexpenseSolution: Buildpredictivemodeltoforecastpossiblechurn,actpre-emptively&learn
1. Getcustomerdata(set-topboxes,weblogs,transactionhistory)
2. Exploredata,andfitpredictivemodelsonpast/real-timedata
3. Apply&validatemodelsuntilpredictionsareaccurate
4. Forecastchurnrate&identifycustomerslikelytochurn
5. SurfaceresultstoBusinessOps,whoINVESTIGATES&ACTS
16
Summary:TheMLProcessProblem:<Stuffintheworld>causesbigtime&moneyexpenseSolution: Buildpredictivemodeltoforecast<possibleincidents>,actpre-emptively&learn
1. Getallrelevantdatatoproblem
2. Exploredata,andfitpredictivemodelsonpast/real-timedata
3. Apply&validatemodelsuntilpredictionsareaccurate
4. ForecastKPIs¬ableeventsassociatedtousecase
5. SurfaceincidentstoXOps,whoINVESTIGATES&ACTS
Operatio
nalize
Copyright©2016SplunkInc.
MLwithSplunk
18
Splunkbuilt-inMLcapabilities
kmeans cluster
outliers/anomalies /anomalydetection
predict
19
MachineLearninginSplunkITSIAdaptiveThresholding:• Learnbaselines&dynamicthresholds• Alert&actondeviations• Managefor1000sofKPIs&entities• Stdev/Avg,Quartile/Median,Range
AnomalyDetection:• Find“hiccups”inexpectedpatterns• Catchesdeviationsbeyondthresholds• UsesHolt-Wintersalgorithm
20
SplunkUserBehaviorAnalytics(UBA)• ~100%ofbreachesinvolvevalidcredentials(MandiantReport)• Needtounderstandnormal&anomalousbehaviorsforALLusers• UBAdetectsAdvancedCyberattacks andMaliciousInsiderThreats• LotsofMLunderthehood:
– BehaviorBaselining&Modeling– AnomalyDetection(30+models)– AdvancedThreatDetection
• E.g.,DataExfil Threat:– “Sawthisstrangelogin&datatransfer
forusermpittman at3aminChina…”– SurfacethreattoSOCAnalysts
21
MLToolkit&Showcase– DIYML
• SplunkSupportedframeworkforbuildingMLApps– Getitforfree:https://splunkbase.splunk.com/app/2890/
• LeveragesPythonforScientificComputing (PSC)add-on:– Getitforfree:refertoSplunkbase foryourOSversion
ê https://splunkbase.splunk.com/app/2881/ to/2884/– Open-sourcePythondatascienceecosystem– NumPy,SciPy,scitkit-learn,pandas,statsmodels
• Showcaseusecases:PredictHardDriveFailure,ServerPowerConsumption,ApplicationUsage,CustomerChurn&more
22
Standardalgorithms outofthebox:
Clustering:DBSCAN,KMeans,Birch,SpectralClusteringRegression: LinearRegression,RandomForestRegressor,ElasticNet,Ridge,LassoClassification: LogisticRegression,RandomForestClassifier,SVM,NaïveBayes(GaussianNB,BernoulliNB)Transformation: PCA,KernelPCA,TFIDFVectorizer,StandardScalerTextAnalytics: TF-IDFFeatureExtraction: FieldSelector (e.g.Univariate,ANOVA,K-best,etc.)
Implementoneof300+algorithmsbyeditingPythonscripts
23
Copyright©2016SplunkInc.
BuildingMLApps
25
Analysts BusinessUsers
1.GetData&FindDecision-Makers
25
ITUsers
ODBCSDKAPI
DBConnectLook-Ups
AdHocSearch
MonitorandAlert
Reports/Analyze
CustomDashboards
GPS/Cellular
Devices Networks Hadoop
Servers Applications OnlineShoppingCarts
Analysts BusinessUsers
StructuredDataSources
CRM ERP HR Billing Product Finance
DataWarehouse
Clickstreams
26
2.ExploreData,BuildSearches&Dashboards• StartwiththeExploratoryDataAnalysisphase
– “80%ofdatascience issourcing,cleaning,andpreparingthedata”– Tip:leverageITSIKPIs– lotsofdomainknowledge
• Foreachdatasource,build“datadiagnostic”dashboard– What’sinteresting?Throwupsomebasiccharts.– What’srelevantforthisusecase?– Anyanomalies?Arethresholdsuseful?
• Mixdatastreams&computeaggregates– ComputeKPIs&statisticsw/stats,eventstats,etc.– Enrichdatastreamswithusefulstructureddata– statscountbyXY– whereX,Yfromdifferentsources– BuildnewKPIsfromwhatyoufind
27
3.Fit,Apply&ValidateModels• MLSPL – NewgrammarfordoingMLinSplunk• fit – fitmodelsbasedontrainingdata– [training data] | fit LinearRegression costly_KPI
from feature1 feature2 feature3 into my_model
• apply – applymodelsontestingandproductiondata– [testing/production data] | apply my_model
• ValidateYourModel (TheHardPart)– Whyhard?Becausestatistics ishard!Also:modelerror≠realworldrisk.– Analyzeresiduals,mean-squareerror,goodnessoffit,cross-validate,etc.– TakeSplunk’sAnalytics&DataScienceEducationcourse
28
4.Predict&Act• ForecastKPIs&predictnotableevents
– Whenwillmysystemhaveacriticalerror?– Inwhichserviceorprocess?– What’stheprobablerootcause?
• Howwillpeopleactonpredictions?– IsthisaSev 1/2/3event?Whoresponds?– DeliverviaNotableEventsordashboard?– Humanresponseorautomatedresponse?
• Howdoyouimprovethemodels?– Iterate,addmoredata,extractmorefeatures– Keeptrackoftrue/falsepositives
Copyright©2016SplunkInc.
Demo
Copyright©2016SplunkInc.
NextSteps
31
Gettingstarted• Pre-requisite: youmustberunning Splunk6.4.x
• DownloadandinstallthefreeMLToolkit&Showcase!– https://splunkbase.splunk.com/app/2890/– https://splunkbase.splunk.com/app/2881/ to/2884/
• SpeaktoyourlocalSE todiscusswaysyoucoulduseML
• JoinourlocalUserGroup– we’llberunningMLworkshops!– http://www.meetup.com/splunk-melbourne/
• Contactme!([email protected])
Copyright©2016SplunkInc.
Q&A
Copyright©2016SplunkInc.
ThankYou
34
ExampleSplunkSPL– ChurnUseCase|inputlookup churn.csv|samplepartitions=2seed=1234|searchpartition_number=0|fitLogisticRegression Churn?from "CustServ Calls""DayMins""EveMins"intoexample_churn_model|table*Churn*|`confusionmatrix("Churn?","predicted(Churn?)" )̀
|listmodels
|summaryexample_churn_model
|deletemodel example_churn_model
|inputlookup churn.csv|samplepartitions=2seed=1234|searchpartition_number=1|apply"example_churn_model"
|inputlookup churn.csv|samplepartitions=2seed=1234|searchpartition_number=1|apply"example_churn_model"|`confusionmatrix("Churn?","predicted(Churn?)" )̀
|inputlookup churn.csv|samplepartitions=2seed=1234|searchpartition_number=1|apply"example_churn_model"|`classificationstatistics("Churn?", "predicted(Churn?)")̀
#####exampletrainingusinglogisticregressionandrandomforestclassifierincombination
|inputlookup churn.csv|samplepartitions=2seed=1234|searchpartition_number=0|fitLogisticRegression "Churn?"from "CustServ Calls""DayMins""EveMins""Int'lPlan""IntlCalls""IntlCharge""IntlMins""NightCharge""NightMins""VMail Plan"into"LogReg_churn"|table*Churn*
|inputlookup churn.csv|samplepartitions=2seed=1234|searchpartition_number=0|fitRandomForestClassifier "Churn?" from"CustServ Calls""DayMins""EveMins""Int'lPlan""IntlCalls""IntlCharge""IntlMins""NightCharge""NightMins""VMail Plan"into"RF_churn"|table*Churn*
#####exampletestingusinglogisticregressionandrandomforestclassifierincombination
|inputlookup churn.csv|samplepartitions=2seed=1234|searchpartition_number=1|applyLogReg_churn asLogReg(Churn?)|applyRF_churn asRF(Churn?)|eval priorityscore(Churn?)=if('LogReg(Churn?)'="True.",10,0) +if('RF(Churn?)'="True.",100,0)+.1*'DayCharge'|sort- priorityscore(Churn?)|fieldspriorityscore(Churn?)*Churn?* "CustServ Calls""DayCalls""DayCharge"PhoneState|eval whattodo =if('priorityscore(Churn?)'>15,"Callthem!",null())|fieldformat "DayCharge"="$".round('DayCharge')|search"Churn?"="False."
35
ExampleSplunkSPL– MalwareUseCase|inputlookup firewall_traffic.csv
|inputlookup firewall_traffic.csv|fitLogisticRegression used_by_malware frombytes_received bytes_sent dest_port dst_iphas_known_vulnerability packets_received packets_sent receive_time serial_numbersession_id src_ip src_port intoexample_firewall_traffic_model|table*used_by_malware*|`confusionmatrix("used_by_malware","predicted(used_by_malware)")̀
|listmodels
|summaryexample_firewall_traffic_model
|deletemodel example_firewall_traffic_model
|inputlookup firewall_traffic.csv|samplepartitions=2seed=1234|searchpartition_number=1|apply"example_firewall_traffic_model”
|inputlookup firewall_traffic.csv|samplepartitions=2seed=1234|searchpartition_number=1|apply"example_firewall_traffic_model"|`confusionmatrix("used_by_malware","predicted(used_by_malware)")̀
|inputlookup firewall_traffic.csv|samplepartitions=2seed=1234|searchpartition_number=1|apply"example_firewall_traffic_model"|`classificationstatistics("used_by_malware", "predicted(used_by_malware)")̀
#####exampletrainingusinglogisticregressionandrandomforestclassifierincombination
|inputlookup firewall_traffic.csv|samplepartitions=2seed=1234|searchpartition_number=0|fitLogisticRegression used_by_malware frombytes_received bytes_sent dest_port dst_iphas_known_vulnerability packets_received packets_sent receive_time serial_numbersession_id src_ip src_port intoLogReg_used_by_malware|table*used_by_malware*|`confusionmatrix("used_by_malware","predicted(used_by_malware)")̀
|inputlookup firewall_traffic.csv|samplepartitions=2seed=1234|searchpartition_number=0|fitRandomForestClassifier used_by_malware frombytes_received bytes_sent dest_portdst_ip has_known_vulnerability packets_received packets_sent receive_time serial_numbersession_id src_ip src_port intoRF_used_by_malware|table*used_by_malware*|`confusionmatrix("used_by_malware","predicted(used_by_malware)")̀
#####exampletestingusinglogisticregressionandrandomforestclassifierincombination
|inputlookup firewall_traffic.csv|samplepartitions=2seed=1234|searchpartition_number=1|applyLogReg_used_by_malware asLogReg(used_by_malware)|applyRF_used_by_malware asRF(used_by_malware)|eval priorityscore(used_by_malware)=if('LogReg(used_by_malware)'="yes",10,0)+if('RF(used_by_malware)'="yes",100,0) +if(has_known_vulnerability="yes",50,0)|eval whattodo =if('priorityscore(used_by_malware)'>50,"Investigate!",null())|fieldswhattodo priorityscore(used_by_malware)*used_by_malware*receive_time src_ipserial_number session_id has_known_vulnerability|sortwhattodo