1

MAD: A Middleware Framework for Multi-Step Attack Detection

Panagiotis Papadopoulos, Thanasis Petsas, Giorgos Christou

and Giorgos Vasiliadis

2

Network Attacks• Automated attacks (e.g. worms and viruses) easy to detect by a

signature-based NIDS. • But there are also more sophisticated targeted attacks out there:

e.g. Advanced Persistent Threats (APT)Traditional Attacks APT Attacks

Attacker Mostly single person Highly organized, sophisticated, determined and well-resourced group

Target Unspecified, mostly individual Systems

Specific organizations, governmental institutions, commercial enterprises

Purpose Financial benefits, demonstrating abilities

Competitive advantages, strategic benefits

Approach Single-run, “smash and grab", short period

Repeated attempts, stays low and slow, adapts to resist defenses, long term

3

Multi-step attacksdesigned for political or economic espionage or sabotage and are fired

against governments, organizations, highly competitive companies, political activists etc.

require coordinated human involvement rather than an automated malicious script.

follow long-term steps of actions consist of multiple correlated steps to reach a specific target combine several attack methodologies:

• e.g. drive-by downloads, SQL injections, malware, spyware, phishing, spam emails etc. and tools:

• including zero-day vulnerability exploits, viruses, worms, and rootkits).

4

Attacks’ main characteristics (1/2)1. Persistence

• location of data, security controls deployed, existed vulnerabilities etc. not known a-priori. • attacker must overcome various security measures to finally gain access to

privileged hosts

2. Evasiveness• designed to evade common security mechanisms,• deliver threats through commonly used protocols (HTTP, SMTP, POP etc.)• To be stealthy custom malwares need to be used• Encryption techniques may be used to avoid firewalls while exfiltrating data out of

the target network.

5

Attacks’ main characteristics (2/2)3. Complexity

combination of attack vectors targeting as many vulnerabilities as possible (e.g. social engineering, Remote Access Trojans (RATs), or other custom malicious software)

very difficult to provide defense on all these different attack vectors

6

The attack phases1. Host Reconnaissance

collect useful information by scanning and studying the victim.2. Persistent Incursion

take advantage of possible host’s vulnerabilities and launch “low-and-slow” attack to avoid detection.

3. Control, Discover, Update, Spread map the network topology and the organization defenses from the inside, update tool chest, spread the infection to other nodes of the target network

4. Capture and Exfiltrationtotal control of a number of hosts and extraction of valuable data off the target network to be analyzed.

7

Multi-step attack example

8

Multi-step attack example

9

Multi-step attack example

10

Multi-step attack example

11

State-of-the-art countermeasureNetwork Intrusion Detection Systems (NIDS)

• The presence of a NIDS is a cornerstone in any modern security architecture e.g. Suricata, Snort, Bro.• captures the network traffic at ingress and egress points in the network• performs the required analysis and processing. • detects and stops malicious attacks or unwanted actions.

• Signature-based: a set of pre-defined signatures is matched against the live captured traffic.

12

NIDS Vs Multi-step attacks• When NIDS relies only on live network traffic accuracy decreases

significantly.

Solution:• archive the raw contents of the network traffic stream to disk• enable later inspection of activity

13

Data, data, data…• increasing network traffic and capacity make the collection and

archiving very challenging.

• E.g. 10 GbE network packet arrivals can be as short as 1.25 μs for a 1.5KB MTU storing full packet traces even for 2 hours can result to thousands of GB of data.

14

Challenges• Storage: wholesale recording and retention of entire data streams is

infeasible• A Gigabit network several TB per day• network trace with full packet content can provide much information for

investigating security incidents

• Data selection: only a very small subset of the traffic is relevant for later analysis• How to decide beforehand what data will be crucial?

• Analysis: data retrieval is like finding needle in a haystack

15

Our approach? Get MAD…

16

MAD: A middleware framework for Multi-step Attack Detection1. coupled with a network monitoring application enhances its functionalities

2. enables IDS to analyze and correlate multiple security incidents that may belong to the same attack pattern.

3. significantly reduces the rate of NIDS’s false alarms

4. post-mortem incident analysis in terms of forensic analysis asses the given damage

5. Includes different mechanisms to store the captured network traffic

17

Coupling MAD middleware framework with NIDS• Broadening the analysis context

Analyses traffic from past

• NIDS recovers from Packet DropsNIDS may incur measurement drops under heavy loadcan query MAD for connections that are missing packets and reprocess them

• PrioritizationNIDS can assign priorities to flows letting the rest of the traffic be processed

during idle times.

18

High Level Overview(1/2)• Packet Capturer

responsible for tapping the network link, monitor the traffic and filter the received packets.

• Query Engineresponsible for the responding to the IDS’s GET-requests.

19

High Level Overview(2/2)• Correlation Engine

component to correlate the attack steps by linking NIDS alerts.

• Size Controllerstorage capacity is not inexhaustible, appropriate actions are applied to achieve the highest possible storage size reduction

Snort

20

The Storage component

Storage

1. receives the raw packets, separates headers from payload, responds to the Query Engine

2. stores the fields of their header in a RDBS.3. packets payloads are stored serialized and

grouped by flow.

21

Handling the ever increasing size of datathe more “knowledge” MAD maintains, the more accurate the attack detection will become.

• ”knowledge”= the information residing in the archived historical traffic.

• traffic includes several events steps to a sequence, able to end up to a multi-step targeted attack

• maintaining such network history knowledge is a point of paramount significance

results in storing large amounts of network traces.

22

Controlling the size of the archived traffic• In Size Controller component we adopt several mechanisms to both:

reduce the size of data and at the same time protect the important information that might later be needed.

• These mechanisms include:- Compression - Deduplication - The Cutoff

Heuristic- Classification - Aggregation & Sampling

23

Mechanisms to control the stored data volume (1/3)• Classification

• classify the traffic according to the content of the packets• if we are interested in detecting multi-step attacks that target a specific

application, we can discard the rest non-suspicious traffic.

• Compression• the most efficient and fast method to reduce the required size of a large volume

of data. • flow-based algorithms for trace compression can result in 25% reduction of the

required storage size.• frequent IDS queries, may face increased response latency due to decompression.

24

Mechanisms to control the stored data volume (2/3)• Deduplication

• Reducing duplicates and redundant data better storage utilization.• Packet-level elimination techniques can reduce resource utilization by 10-50%• Inline: the data is processed immediately as its ingested (takes time).

Post-process: the data are deduplicated after they hit the disk (needs capacity).

• Aggregation & Sampling• Both significantly reduce the needed size.• Aggregation requires the traffic features of interest to be known in advance. Not

useful for signature-based checks since much of the payload’s information gets discarded• Random packet sampling decreases the detection accuracy

25

Mechanisms to control the stored data volume (3/3)• The Cutoff Heuristic• selective packet discarding technique. i.e., by discarding the less important

packets of a trace or a flow.

• most of the attacks are detected in the first few packets of a flow. (97% of the alerts are triggered within the first 100 packets of the flows)

• an attacker can evade detection, by transmitting data until the cutoff value has passed.

• a single network connection may exchange large amounts of data passing the cutoff value.

26

In summary…Compression Deduplication Classification Aggregation & Sampling Cutoff Heuristic

Up to 25% size reduction

10%-50% reduced resource utilization

Only specific flows will get archived

Significantly reduces the required storage size.

97% of the alerts in the first 100 packets

Decompression may increase response time in case of frequent queries

Inline takes time,

Post-process needs capacity

Specification of sensitive channels is needed.

Aggregation: the traffic features of interest has to be known in advance.

Random packet sampling may decrease the detection accuracy

Attacker can evade detection (transmit data until the cutoff value has passed.)

27

Future Work• This work is currently in progress

• Our further work would include: extensive evaluation:• in terms of performance, in comparison to other existing solutions • in terms of effectiveness by measuring the produced false positive alert

ratio. a case study to measure the detection percentage against several

known multi-step attacks.

28

Conclusions• Network attacks become more sophisticated and diversified. Several actions

may individually look harmless but when combined can constitute a serious threat.

• NIDS are inadequate countermeasures when rely only on live network traffic.

• We propose MAD to improve the accuracy of NIDS by archiving historical traffic providing knowledge regarding the previous steps.

• We examine several mechanisms to reduce the storage size needs and archive only important information.