HowDukeUniversity* UsesSplunkto* ImproveSecurityand ... · AboutDuke! 14,600Students* 3,340Faculty 35,998Staﬀ Totalof68,000+AcLveUsers UniversityandMedicalCenter

Copyright © 2014 Splunk Inc.

Duke University Jeremy Hopkins, Phillip BaDon, & Eric Hope

How Duke University Uses Splunk to Improve Security and Reduce Fraud

Disclaimer

2

During the course of this presentaLon, we may make forward-‐looking statements regarding future events or the expected performance of the company. We cauLon you that such statements reflect our current expectaLons and

esLmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements,

please review our filings with the SEC. The forward-‐looking statements made in the this presentaLon are being made as of the Lme and date of its live presentaLon. If reviewed aUer its live presentaLon, this presentaLon may not contain current or accurate informaLon. We do not assume any obligaLon to update any forward-‐looking statements we may make. In addiLon, any informaLon about our roadmap outlines our general product direcLon and is subject to change at any Lme without noLce. It is for informaLonal purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligaLon either to develop the features or funcLonality described or to

include any such feature or funcLonality in a future release.

About the Presenters

!   Jeremy Hopkins – Sr. IT Analyst, Enterprise Internet Services !   Phillip BaDon – Sr. Security Analyst, University IT Security Office !   Eric Hope – Security Analyst, University IT Security Office

3

About Duke !   14,600 Students !   3,340 Faculty !   35,998 Staff !   Total of 68,000+ AcLve Users

!   University and Medical Center !   Worldwide Presence

4

Splunk @ Duke University 250GB License

200+ Indices & Sourcetypes •  Syslog •  OS (Win & *nix) •  Web •  Network •  IPS/IDS/Firewall •  Shibboleth •  Mail •  LDAP •  VPN •  Many More

Infrastructure: •  2 Central search heads •  4 Central indexers •  2 Central deployment servers

–  (1 Win & 1 Linux) •  9 Departmental pairs

–  (1 Search Head + 1 Indexer) •  Over 2500 hosts

Departments & Uses •  IT Security Office •  Systems Admin •  Messaging •  Network •  Database •  Emergency noLficaLon

Tracking •  Departmental IT groups •  Many uses by many

groups

5

Agenda

!   IniLal IncepLon of Splunk @ Duke –  OperaLon Find a Phish

!   Using Splunk to Bridge FuncLonal Teams –  Phishing ADacks Lead to Paycheck TheU/Fraud

! Splunk and Security –  CorrelaLons, EvaluaLons, and Risk Scoring

6

So it Began… Phishing to Fraud •  Phish targeLng 600+ faculty/staff •  Typical style emails (nothing overly

sophisLcated) –  Pay raise, login verificaLon, etc. –  Cloned login page

•  Compromised accounts used to access HR/Payroll sites

•  AUer successful login, bank rouLng numbers for direct deposit changed

•  Reports of monthly salaries not received

Source: The News & Observer

7

Before We Were Friends…

“Can you tell me who has logged in the most in the last hour?” – Manager !   Your quesLons are my distracLons…

‒  Where are the users logging in from? ‒  Where is SPAM coming from? ‒  Where is legit mail coming from?

!   Log analysis from a shell prompt using the ancient sysadmin combo of: grep | awk | sort | uniq grep sasl_username logfile | awk '{print $9}' | sort | uniq -‐c | sort -‐n | tail -‐5

7 [email protected] 7 [email protected] 8 [email protected] 10 [email protected] 58 [email protected]

8

hDps://xkcd.com/208/

9

Take NULL locaLons and puts a custom label on them

Use of event type makes this applicaLon independent

Creates a LocaLon label such as “Las Vegas, NV” instead of columns for each value.

Top SMTP Logins

evenDype=smtpauth | iplocaLon client_ip | eval LocaLon= if(CountryCode == "US",if(City=="",if(Region=="","Unknown LocaLon, "+Country,Region),if(Region=="",City+", ?? "+Country,City+", "+Region)),if(City=="",Country,City+", "+Country)) | eval LocaLon=

if(isnotnull(LocaLon), LocaLon, if(cidrmatch(”10.0.0.0/8",client_ip), "Duke -‐ Private”, client_ip ) )

| stats values(LocaLon) count(neLd) by neLd | rename … | table neLd Count LocaLon | sort -‐Count

Finds SMTP logins and builds a table with username, login count, and locaLon

10

Empowering Others Top 10 SMTP Logins using previous search example

11

Advanced XML to Map Mail Sources Example based on Sophos PureMessage scan engine

The hidden search to find inbound and delivered messages that are NOT originaLng from our IP space.

Title of the panel on the dashboard

Module to build and display the map

<module name="HiddenSearch" layoutPanel="panel_row4_col1" group="Source of Accepted Inbound Mail (last 5 minutes)" autoRun="True"> <param name="search"> index=mail host=mail* inbound ac@on=deliver (fur!=10.0.0.0/8 AND fur!=152.3.0.0/16) | geoip fur </param> <param name="earliest">-‐5m@m</param>

<module name="GoogleMaps"> <param name="mapType">splunk</param> <param name="scrollwheel">off</param> <param name="zoomLevel">3</param>

<param name="overlay">clusters</param> <param name="drilldown">true</param> <param name="drilldown_field">client_ip</param>

</module> </module>

12

Fun With Maps Source of Accepted Mail using Advanced XML in previous slide

13

Guess Where our Spam Originates…

14

evenDypes Log example from pos�ix mail server

hostname process

Client IP

Login String

!   Event types allow us to search common events with ease !   What does a login “look” like?

–  Raw log data: Sep 2 13:03:19 mail12.oit.duke.edu pos�ix/smtpd[74209]: 272863911E8_405C017F: client=unknown[152.16.52.172], sasl_method=LOGIN, [email protected]

–  All SMTP authenLcaLons have a few things in common: ê  Hostname of mail*.oit.duke.edu ê  Process of pos�ix/smtpd ê  Client IP ê  User login string of sasl_username=*@*duke.edu

15

evenDype=smtpauth Based on Pos�ix mail server and Cisco AnyConnect VPN

!   SMTP Auth Converted to a Splunk evenDype –  index=mail host=mail* process=pos�ix/smtpd sasl_username=*

!   Duke created event types for various login events from various sources: –  VPN – evenDype=vpnlogin

index=network sourcetype=vpn vpn_user=* vpn_inner_ipv4=* vpn_source_ip=*

!   Shibboleth (single sign on) -‐ evenDype=shiblogin –  index=idms_shib sourcetype=idp-‐process (shib_success=“[password]” OR

SSO=“true”)

16

Share With Others !   In theory, we can join the various login types together to find all login events –  evenDype=dukelogin is defined with the following search

ê  evenDype=vpnlogin OR evenDype=smtpauth OR evenDype=shiblogin

!   Event types allow others to use your logs without knowing the specifics of your applicaLon

17

Cool, But So What !   Plo�ng mail sources provided needed proof to management to allow implementaLon of geographically isolated mail flow and acceptance rules

!   Dashboards and evenDypes allowed the ITSO to quickly see account abuse and provided the groundwork for collaboraLon with other funcLonal teams

!   Prior to Splunk it would take the Messaging team several hours, at Lmes, to pull a list of recipients of a phishing email. Now it’s a maDer of minutes

18

Using Splunk to Bridge FuncLonal Teams Phishing ADacks Lead to Paycheck TheU/Fraud

Example of Phishing Email Received

Clicking here leads to URL on next slide

A pay rise… interesLng.

20

Link from Phish in Previous Slide

Believe it or not, Duke does not own

nl-‐tour.ru

21

Gone Phishin’ Example of Dashboard to Record Email into Phish Tracking Lookup Table

Your DUKE Pay Increase

Pay Increase -‐ 20141006

Search DuraLon

Actual Subject of Email

How we idenLfy a parLcular campaign

Adds to PhishList lookup table

22

User Lookup in the PhishList | inputlookup PhishList.csv | search [email protected]

23

Duke PhishList Lookup Table Custom lookup table to record historical informaLon on phishing emails

Macro to find message by Subject

Arbitrary name of Phish Map email to user

Pull user info (name, department, etc)

Append the output to our PhishList lookup table

`message_by_subject($subject$,"inbound acLon=deliver")` | rename … | eval PhishName=$phishname$ | lookup mailmap email as To | lookup NeLdLookup neLd OUTPUT eppa as affiliaLon | table PhishTime PhishName To From Subject SendingIP affiliaLon neLd | outputlookup append=true PhishList.csv

24

Phishing Dashboard Aggregate of PhishList Data with Search Panel

25

1.  What data points do we have? a.  Compromised accounts (Names & NetIDs) b.  Target sites (Duke HR/Payroll)

2.  What log sources have that data? a.  Shibboleth IdP Process

3.  Any correlaLon between accounts? a.  Phishing aDacks?

(Direct Deposit ModificaLon NoLficaLon)

26

And So the Security InvesLgaLon Begins

Analysis (grep) 2013-‐11-‐24:15:09:29.492 IP_address=BadActorIP1, User=NetID#1, enLtyId=hDps://duke.edu(site1), 2013-‐11-‐24:15:09:46.143 IP_address=BadActorIP1, User=NetID#1, enLtyId=hDps://duke.edu(site2),

2013-‐11-‐24:15:39:32.285 IP_address=BadActorIP1, User=NetID#2, enLtyId=hDps://duke.edu(site1),















27

Shibboleth (Single Sign On) Logs

!   Shibboleth = open source middleware for idenLty management based on SAML

! IdP Process Logs –  Example: Lmestamp – INFO [IdP AuthN Provider] – IP Address, User,

EnLtyId, Success/Failure Message

1.  IP Address = Source IP 2.  User = NetID 3.  EnLtyId = Target Website 4.  Success = [password] OR [SSO]

28

Props.conf (Shib)

29

Let the Splunking Begin

evenDype=shiblogin User=$User$ | iplocaLon shib_src | ldapfilter domain=Duke search=“(uid=$User$)” aDrs=“Name,Affilia*on,Dept,Title,Phone” | table _Lme, IP, City, Region, Country, Site, NetID, Name, Dept, Title, AffiliaLon, Phone | sort NetID, -‐_Lme

30

(Form – Enter UserID, Seach Shib for AuthN Logs)

And the IP Address…

evenDype=shiblogin IP=“IP address” | iplocaLon IP | ldapfilter domain=Duke search=“(uid=$User$)” aDrs=“Name,Affilia*on,Dept,Title,Phone” | table _Lme, IP, City, Region, Country, Site, NetID, Name, Dept, Title, AffiliaLon, Phone | sort NetID, -‐_Lme

EssenLally the same search as the previous slide, modified to query for IP rather than UserID

31

Once the Bleeding Stops…

!   How do we uLlize Splunk to become more proacLve (Shibboleth)? 1.  Look for “non-‐Duke” IPs with mulLple user logins to HR/Payroll 2.  Non-‐Duke IPs with mulLple user logins regardless of desLnaLon 3.  Query the number of ciLes an account has logged in from (24hrs) 4.  Query the number of countries an account has logged in from

(24hrs)

32

HR/Payroll Logins – MulLple Users, Single IP evenDype=shiblogin (site=“DukeHR” OR site=“DukePayroll”) NOT (IP=“DukePublic” OR IP=“DukePrivate”) | iplocaLon IP

| where state!=NC 0R where country!=US (opLonal) | stats dc(NetID) AS “User_Count”, values(NetID) by IP | where User_Count > 1 | table IP User_Count NetID | sort –User_Count

!   Easy to take out HR/Payroll specific search to look for trends regardless of desLnaLon

33

Single User, MulLple CiLes (24hrs) evenDype=shiblogin | geoip IP | stats dc(geo_info) AS “Number_of_CiLes”, values(geo_info) AS CiLes by NetID | where Number_of_CiLes > 2 | table NetID Number_of_CiLes CiLes | sort –Number_of_CiLes

!   Similarly, tweak the search to look for countries rather than ciLes

34

What Else?

!   Phishing recipients from Messaging –  | inputlookup DukePhish.csv

!   Direct deposit modificaLons –  | inputlookup DDtransacLons.csv

!   How do we incorporate these two and begin reporLng?

35

High Risk Report (Incorporate Phish & DD lists to look for access from outside the

US OR IPs with mulLple UserIDs authenLcaLng)

! evenDype=shiblogin [| inputlookup DirectDeposits.csv | fields NetID]

| iplocaLon IP | eval foreign=if(Country!="United States","true","false")

| fields …

| join NetID [| inputlookup PhishList.csv]

|where (PhishTime <= DDChanged) AND Site=DukeSite.edu | stats dc(NetID) AS User_Count values(ShibLoginTime) by NetID values(Site) values(IP) values(City) values(Region) values(Country) values(PhishReceived) values(DepositChanged)

| eval mulL=if(User_Count>1,"true","false")

| where (foreign="true" OR mulL="true")

36

The Fight ConLnues… !   MulLple IteraLons of DD Phish !   Team moves from Shib-‐only invesLgaLon > Web Hits !   Web log info to see IP info of “aDackers” + vicLms !   Chum Accounts

This type of visibility into logs just did not exist for the ITSO prior to Splunk It becomes organizaLonal, and not just the responsibility of security

!   App developments !   MFA

37

Because They Won’t Stop

38

Splunk and Security CorrelaLons, EvaluaLons, and Risk Scoring

When to Catch Direct Deposit Phishing?

!   Before the users are Phished !   AUer the users submit their credenLals but before the aDackers try and use them

!   AUer the aDackers change the direct deposit accounts but before payday

!   AUer payday when users contact us about missing paychecks

40

List of ADributes of Direct Deposit Changes !   Which countries !   How many countries !   Which regions (States) !   How many regions !   Coming from campus IP address !   How many IP addresses !   How many users came from the same IP address

!   How many direct deposit changes

!   Were the users phished !   How many Lmes were they phished

!   Were the users phished before any of the direct deposit changes

!   Does the user have mulL-‐factor AuthenLcaLon enabled

!   Number of web hits

41

No single aDribute/data point indicates the direct deposit informaLon has been modified

Except when the user calls to say they did not get paid

42

Assigning numeric values to aDributes

WeighLng the numeric values

Filtering based on weight

Example of WeighLng

<Initial search> | iplocation clientip | stats dc(Country) as countrycnt dc(Region) as regioncnt dc(clientip) as ipcnt by user | eval risk=0 | eval risk=risk + ((countrycnt - 1 ) * 10) | eval risk=risk + ((regioncount - 1 ) * 5 ) | eval risk=risk + ((ipcnt - 1 ) * 3 ) | where risk > 15 | sort -risk

43

User Name DD change

DD count

Phish Dates

Phish Count

Countries Country Risk

IP Count Max Users /

IP

Total Risk

juser1 Jane User

09/05 09/20 09/21

3 07/24 08/15 09/21

3 Nigeria US

Russia

21 14 12 25

juser2 Joseph User

09/12 09/21

2 09/20 1 US Germany

13 8 5 17

bprof5 Robert Prof

09/15 1 0 Cuba 17 12 1 11

Weighted Dashboard

44

User AcLvity Data Sources

!   Single sign on !   AuthenLcated web hits !   VPN access !   SSH logins !   Phishing emails received

!   Outbound email !   Windows authenLcaLon !   Account locks and unlocks !   LDAP informaLon !   Direct deposit changes

45

( index=sso username=$user$ ) OR ( index=web user=$user$ (status=200 OR status=3*) ) OR ( index=linux process=sshd user=$user$ (action=”Accepted” OR action=”Failed”) ) OR ( index=mail sasl=$user$ ) OR ( etc )

First ADempt at a Combined Login Form

46

Example Using Radio BuDons

47

Example Using Radio BuDons <fieldset> <input type="text" token="search_netid"> <label>NetID</label> <suffix/> </input> <input type=”radio” token=”mail_button”> <label>Mail Logins</label> <default>YES</default> <choice value=”index=mail”>YES</choice> <choice value=”index=NONESUCH”>NO</choice> </input> <input type="time"> <default>Last 7 days</default> <label>Time</label> </input> <fieldset> ~ <searchString>$mail_button$ host=”mail-gw-*” user=$search_netid$</searchString>

48

User InvesLgaLon Form

49

Example of Combining Details Using Case

| eval combined_event_type=case( isnotnull( vpn_login ), “VPN Login”, isnotnull( sasl_username ), “EMAIL”, match( event_action, “lock” ), “Account Lock”, match( event_action, “unlock” ), “Account UNLock”, isnotnull( clientip ) AND isnotnull( user ), “WEB HIT”, 1=1, “-”

)

50

Further Case Example

| eval combined_notes1=case( isnotnull( vpn_login ), “username = “.vpn_username, isnotnull( sasl_username ), “mail client = “.client match( event_action, “lock” ), “locked by = “.done_by.”, reason = “.lckrsn, isnotnull( clientip ) AND isnotnull( user ), “host = “.host 1=1, “-”

)

51

Complete User Login Form

52

Weighted Dashboard

User Name DD change

DD count

Phish Dates

Phish Count

Countries Country Risk

IP Count

Max Users /

IP

Total Risk

juser1 Jane User

09/05 09/20 09/21

3 07/24 08/15 09/21

3 Nigeria US

Russia

21 14 12 25

juser2 Joseph User

09/12 09/21

2 09/20 1 US Germany

13 8 5 17

bprof5 Robert Prof

09/15 1 0 Cuba 17 12 1 11

53

InvesLgaLng juser1's Logins

54

Jane User's Logins

_time event_type user src_ip host location notes1 notes2 2014-09-21 15:23:44

SSO Login juser1 152.3.14.90 Dhcp-152.3.14.90

Durham-NC-United States

EntityID=https://hr.duke.edu/

success=mfa

2014-09-15 06:14:00

Mail Login juser1 91.236.24.90 Host90.msu.ru Moscow—Russia - -

2014-09-14 05:56:14

VPN Login juser1 91.236.24.95 Host95.msu.ru Moscow--Russia

inner_vpn_ip = 192.168.140.8 -

2014-09-05 10:21:21

SSO Login juser1 152.3.14.88 Dhcp-152.3.14.88

Durham-NC-United States

EntityID=https://hr.duke.edu/

success=mfa

55

Joe User's Logins

_time event_type user src_ip host location notes1 notes2 2014-09-21 13:44:00

SSO Login juser2 192.168.14.50 Internal1450.duke.edu -- EntityID=traini

ng.duke.edu success=password

2014-09-21 13:21:15

SSO Login juser2 85.17.10.131 - Munich--Germany

EntityID=hr.duke.edu

success=password

2014-09-21 13:17:22

SSH Login juser2 85.17.10.131 - Munich--Germany

Server=login-01.duke.edu -

2014-09-21 13:11:55

Mail Login juser2 192.168.14.50 Internal1450.duke.edu -- - -

56

Benefits of Splunk to Duke !   Splunk allowed Duke to begin leveraging mulLple data sources almost immediately with very liDle ramp-‐up Lme

!   Detailed informaLon on recipients of phishing messages is now available to Security in minutes instead of hours

!   Splunk has allowed Duke to more than double the number of compromised accounts we detect and lock each month

!   Splunk provided the ability to create a custom SIEM like soluLon tailored to Duke’s needs

57

Key Takeaways 1.  Use Splunk to bridge the gap between teams and log knowledge 2.  Use event types, macros, and saved searches to make your long

crazy searches usable by others 3.  Decide what is suspicious for your use case 4.  Use the eval command to create custom weighLng algorithms 5.  ConLnue to educate your users about Phishing

58

Q&A

60

Security office hours: 11:00 AM – 2:00 PM @Room 103 Everyday Geek out, share ideas with Enterprise Security developers

Red Team / Blue Team -‐ Challenge your skills and learn new tricks Mon-‐Wed: 3:00 PM – 6:00 PM @Splunk Community Lounge Thurs: 11:00 AM – 2:00 PM

Learn, share and hack

Birds of a feather-‐ Collaborate and brainstorm with security ninjas Thurs: 12:00 PM – 1:00 PM @Meal Room

THANK YOU

Macro: message_by_subject(2) index=mail S="*$subject$" $vars$ | makemv delim="> t=<" pmx_to | rename … | mvexpand to | rex field=f "\<(?<from>.+)\>" | rex field=S "\??q?\??(?<subject>.+)" | eval subject=replace(subject,"_"," ")

62

Documents

How*Duke*University* Uses*Splunk*to* Improve*Security*and ... · AboutDuke*! 14,600*Students* 3,340*Faculty* 35,998*Staﬀ* Total*of*68,000+AcLve*Users* University*and*Medical*Center*

HowDukeUniversity* UsesSplunkto* ImproveSecurityand ... · AboutDuke! 14,600Students* 3,340Faculty 35,998Staﬀ Totalof68,000+AcLveUsers UniversityandMedicalCenter