User and entity behavior analytics: building an effective solution

User and Entity Behaviour AnalysisBuilding an Effective SolutionYolanta Beresna

Research Manager, Threat Detection and Remediation, Software Defined Cloud Group

10 November 2016

Outline

Overview of UEBA space Key components of an Effective Solution

– Threat Use cases– Data Sources– Analytics– Pluggable Analytics Modules

UEBA: Overview

4

User and Entity Behaviour AnalyticsThe Definition

User and entity behavior analytics is bringing profiling and anomaly detection based on machine learning to security, to detect malicious and abusive activity that otherwise goes unnoticed. Profile and baseline the activity of users, peer groups and

other entities such as endpoints, applications and networks.

Form peer groups based upon common user activities, using directory groupings and human resources information only as a starting point.

Correlate user and other entity activities and behaviors.

Detect anomalies using statistical models, machine learning and/or rules that compare activity to profiles.

Source: Gartner (September 2015)

UEBA across IT systemsUsers-accounts

• Mapping: user-account-hostname• Behaviour: account usage across

applications and domains• Suspicious behaviour:

Changes in behaviour for highly privileged users and core systems

Changes in access and account usage behaviour

Peer group comparison• Data: active directory, LDAP, system

and application account usage

Users-entities

• Mapping: user-hostname-ipaddress

• Behaviour: network traffic patterns

• Suspicious behaviour Historical changes in behaviour Outliers based on peer group

comparison Specific threat patterns: malware

infections, tunnelling traffic, beaconing• Data: DNS, HTTP, Netflow, VPN

Entities-servers

• Behaviour: network traffic patterns• Data: DNS, HTTP, Netflow, system logs

Connections

Linked information between: user-accounts user-entities entities-servers

Features of UEBA Solution

An effective UEBA the solution has at least the following properties:

Effective data collection and data representation layer

Correlation of entities identifiers to users and user accounts to users

Abnormal behaviour detectionSpecific threat detectionDiscovery of core systems and privileged users as

well as peer groups or communitiesLinking together of multiple detection results into a

coherent threat view across enterprise

Suspicious Entity and User Detection Analytics

In addition it is essential to have capabilities to add new analytics and reconfigure existing ones: play (by developing new analytics) and plug (for automated results) framework

7

Creating an Effective Solution

8

Core Components

The effectiveness of an UEBA greatly depends on these core components:

1. Focused threat scenarios and use cases

2. Availability of relevant data sources and variables

3. Appropriate analytics algorithms

9

Anatomy of Attacks

Threat Use Cases

 Threat Actor

External Internal

Goal

Theft

Attack Story 1: A hacker organisation gains

access to the system over the Internet and

steals user credentials and business data.

Attack Story 2: An employee uses their

access to the system to steal business data.

Sabotage

Attack Story 3: Ransomware attack:

Business data shared on the internal network is

encrypted by ransomware running on

a client machine.

Attack Story 4: An employee reconfigures

the machines in the network to render their services unavailable to

legitimate users.

• Attack stories describe concrete attacks• What is happening?• In which order?• When?• Where?

• Goal/Actor Matrix to develop stories:• Goal: What do the attackers want to achieve?• Actor: Who are the attackers?

• Attack Story Steps:1. Gain access2. Get means to achieve goal3. Reconnaissance and lateral movement4. Achieve goal

11

Attack Story 1: Data Exfiltration by External Actor

Stage Analytics Features Data Outcomes + Context

Gain Access/Initial Infection 1. Detect malicious web

communication from hosts to external web sites involving blacklisted/TI sites

2. Detect unusual/DGA DNS traffic with resolving domains

3. Identify user(s) with privileged access to those hosts and/or roles (e.g. AD administrator)

4. Analytic 1 AND/OR 2 triggers on at least an entity AND Analytic 3 identified a misused privilege user/account

- ENTITY: Requests of DGA Domains

- ENTITY: Access to blacklisted/TI domains

- ENTITY: DNS/HTTP traffic volume

- ENTITY: DNS NXDOMAIN rate and Resolving traffic rate

- USER: at least 1 user with privileged rights accessing that resource (phished/stolen credential)

- …

- Web proxy data- User-IP mapping

data- DNS data

- List of Privileged/Admin Users

- List of Critical Resources/Servers

- Timestamp- Suspicious entity- Suspicious user- Context:

INITIAL_INFECTION

12

Attack Story 4: Revenge by Disgruntled Employee

Stage Analytics Features Data Outcomes + Context

Reconnaissance and lateral movements 1. Detect abnormal sequence of

privileged & system commands on a system by local user/account (sudo, system file changes, etc.)

2. Detect changes of cron tables listing new, unrecognised programs. Detect command to install these programs.

3. Detect unusual traffic towards other networked systems with unusual success/failure rates

4. User belongs to a list of admin users

4. Analytic 1,2,3,4 triggers on at least a user and a device

- USER: use of privileged command activities

- USER: installation of new programs

- USER: modification of critical system files, such as crons

- ENTITY: number of netflow connections towards different systems

- …

- User commands- System commands- Netflow data

- List of Privileged/Admin Users

- Timestamp- Suspicious entity- Suspicious user- Context:

RECONNAISSANCE LATERAL MOVEMENTS

13

Data Sets

14

Data Sets for Analytics

Core Data– Netflow

– HTTP traffic or Web proxy Logs

– DNS traffic or DNS Logs

– AD Logs

System Data– Windows system logs from critical servers

– Linux audit and system logs

– Other server/app logs: DB, git, web server

User-Hostname-IP Mapping– DHCP

– VPN

– AD Logs

– Aruba Clearpass

Data Enrichment– GeoIP

– ASN

– Threat Intel

15

Scale of Core Data Sets Volume and Size within HPE worldwide network

Data Type # Events/day(after filtering)

TB/day Avg Event Size

Netflow 34 Billion(3 collection points)

3.40 TB 100 B

DNS 150 Million(4 collection points)

0.15 TB 1 KB

HTTP 65 Million(central collection)

0.13 TB 2 KB

AD 153 Million

TOTAL ~ 35 Billion/day ~ 3.7 TB/day

Analytics

17

Combination of Analytics

Abnormal Behaviour Detection

1. Inconsistent/abnormal behaviour Comparing to OthersOutliers by comparing to assumed “normal” behaviour across others or in peer community

2. Historical Changes in User-Entity Behaviour PatternsTemporal changes in an individual entity network patternsAbnormal user activity and account usage

Empirical Rules and Patterns

1. Specific malware infectionsDGA domains, malicious web traffic

2. Command & Control communicationsBeaconing + threat intelligence

3. Data ExfiltrationHigh volumes of data sent via DNS or HTTP

Graph Analytics

1. Using graph features to profile entities and detect abnormal behaviour

2. Enabling graph based queries on the already collected data sets: e.g. network activity

Anomaly Detection

Entity Profiling

Domain-name Server (DNS)

Web-Proxy Server (HTTP)

Internal Traffic (Netflow)

Threat Intelligence

Package analysis

Anti-virus logs

…

Events Sources

Users

Host machines

Domain Names

IP addresses

Port Numbers

Sites

…

Entities Profiles

𝑡 0 𝑡1 𝑡 2

𝑡 0 𝑡1 𝑡 2

Peer and Temporal Comparison

Entity type

Profiles

𝑡 0 𝑡1 𝑡 2

Peer comparison

analysis

Temporal analysis

Most anomalous entities returned as an outcome

Pattern-Based Analytics

Empirical Rules: Pattern-based Anomaly Detection

Initial Infection / Gain Access

Command & Control / Means to

Achieve Attack

Lateral Movement

Exfiltration / Damages

Analytics based on deep knowledge of security attack patterns and infiltration processes

Could be applied across all attack phases:

• Devices with DGA infections • Abnormal device communications to external sites • Detection of privilege escalation• Abnormal execution of privileged/admin commands• Abnormal creation/usage of admin accounts or AD domains at unusual times and locations• Abnormal number and types of accesses to a device from remote locations

• Beaconing traffic to suspicious external sites • New device communication and traffic patterns based on historical data and threat intelligence• Unusual number of failed connections from a device to external sites

• Port scanning detection• Abnormal volume of traffic or types of connections from a device towards critical servers (e.g. AD, …) or the way around • Unusually large number of clients• successfully connecting to other clients• Abnormal number of connection failures from devices to network services or specific service ports (e.g. SSH)

• Abnormal volume of traffic from a device towards unknown/suspicious external sites• Abnormal content in queries issued to a set of unknown domains• Abnormal external download of content from organisation’s external facing servers (e.g. web site)• Abnormal activities/patterns on specific servers (e.g. file encryption on file servers)• Abnormal traffic/uploading towards an external web site/Dropbox/etc.

User AccountCompromise

• Abnormal Login Failure/Success Rate• Abnormal set of privileged commands • Abnormal command sequences• Creation of privileged account coupled with one or more above anomalies• Abnormal time of logins and activities

23

Graph Analytics

24

Graphs for Security

Graph Visualisation– Assist security experts by flexibly visualizing linked data

(topology + features)

Graph Database– Allow to query the data more naturally when thought of as a

graph

Graph Analytics– Data representation and tools to support compute on the

entire data– Centrality– Graph Clustering– Similar pattern recognition

1

2

centrality

pattern matchingsub-graph search

25

Pluggable Analytics

Security Analytics Marketplace

Browse Analytics:- Threat Scenario- Use Case- Attack Stage- Analytics Type

End-User

DownloadAnalyticsModule(s)

AnalyticsModule(s)

Analytics Engine(s)

AnalyticsOrchestration

Visualization Configuration

Threat Findings

New Alert Types

Threat Links

Visual Widgets

AnalyticsResults

New LinkCorrelations

NewWidget

Analytics StoreLegal/Privacy

Audit

Software Deployment

27

Thank youYolanta Beresna

yolanta.beres@hpe.com