DNS Monitoring, looking out for anomalies using the time ... · Lautaro Dolberg, Born in Buenos Aires - Argentina. 29 Years old I PhD Studies at Universite de Luxembourg. I Started

[email protected] 09/10/14

DNS Monitoring, looking out foranomalies using the time frame of

Name - IP association

Lautaro Dolberg

Introduction Background MAM Analysis Evaluation Conclusions

Outline

1 Introduction

2 Background

3 MAM

4 Analysis

5 Evaluation

6 Conclusions

2 / 37


Outline

1 Introduction

2 Background

3 MAM

4 Analysis

5 Evaluation

6 Conclusions

3 / 37


Personal InformationLautaro Dolberg, Born in Buenos Aires - Argentina. 29Years old

I PhD Studies at Universite de Luxembourg.I Started 15/1/2012 - Expected Defense 2X/3/2015I Secan Lab - Vehicular Lab. J Francois, R State, R.

Frank & T. Engel

I APR 14 - Prof Raouf Boutaba research team atWaterloo University, Ontario, Canada.

I ”Licenciatura en Ciencias de la Computacion” at DC -FCEN UBA (2011)

I Research Keywords: Security, Networks, VANET,Monitoring, Big Data, Mobile Devices, SDN

4 / 37




Frank & T. EngelI APR 14 - Prof Raouf Boutaba research team at

Waterloo University, Ontario, Canada.

I ”Licenciatura en Ciencias de la Computacion” at DC -FCEN UBA (2011)


4 / 37





Waterloo University, Ontario, Canada.I ”Licenciatura en Ciencias de la Computacion” at DC -

FCEN UBA (2011)


4 / 37





Waterloo University, Ontario, Canada.I ”Licenciatura en Ciencias de la Computacion” at DC -

FCEN UBA (2011)I Research Keywords: Security, Networks, VANET,

Monitoring, Big Data, Mobile Devices, SDN4 / 37


Research Context

I Unrestricted access networksI Internet Applications Mobile (Crowd-sourcing)I Traffic Sensing Applications (LuxTraffic)I Internet Name Resolution (DNS)

I Large Data Collections

I Monitoring for detecting misbehavior

I Mobile Security (Android + IOS)

I Software Defined Networks

I Network Awareness

Security & Management of distributed applications

5 / 37


Outline

1 Introduction

2 Background

3 MAM

4 Analysis

5 Evaluation

6 Conclusions

6 / 37


Motivation

DNS traffic reflects Internet activities and behaviors

I Internet Threats Growing: Phishing, Malware,Spoofed Domains.

I Identify malware behavior by assessing associationtime between names and networks.

I Helps for contextualizing other data such as Netflow,etc.

I Available resources such as Passive DNS.

I As both DNS and IP space follow hierarchicalorganization, MAM can be used.

7 / 37


Motivation cont’d

I Can we use DNS records and its changes over time totrace Internet activities?

I Do malicious domains behave different from others interms of name - ip association?

Example DNS Records

Type Name IPV4 TTLCNAME www.lip6.fr ww.lip6.fr XA ww.lip6.fr 132.227.104.15 X

8 / 37


Motivation cont’d



Example DNS Records


8 / 37


Motivation cont’d



Example DNS Records


8 / 37


DNS BackgroundDNS is an essential service for Internet

I Emerged in 1987 (RFC 1035, 1123, 2181)I Domain names are labels separated with dots.I Strictly hierarchical patternI TLD (eg: .com, .fr, .etc)

.fr

lip6 Inria

ROOT

(TLD)

Max depth 127. Limited to 253 characters, label limit 63characters.

9 / 37


DNS Structure & Procedure

Resolving www.lip6.fr


Relevant DNS Registers

I A: Association IPV4 or/and IPV6.

I CNAME: Redirect.

I PTR: Reverse DNS search.

10 / 37



Resolving www.lip6.fr Resolving www.lip6.fr

Root ServerWhere is www.lip6.fr ?

Try TLD

lip6.fr NS

TLD Server

Yes! I have it

Relevant DNS RegistersI A: Association IPV4 or/and IPV6.I CNAME: Redirect.I PTR: Reverse DNS search.

10 / 37





Root ServerWhere is www.lip6.fr ?

Try TLD

lip6.fr NS

TLD Server

Yes! I have it

Relevant DNS Registers

I A: Association IPV4 or/and IPV6.

I CNAME: Redirect.

I PTR: Reverse DNS search.10 / 37


Passive DNSA Passive DNS DB contains:

RecursiveDNS Server

Passive DNS

DNSQuery/response

DNS ServerDNS Server

DNS Server

Recursive Queries /response

I Where did this domain name point to in the past?

I What domain names are hosted by a givennameserver?

I What domain names point into a given IP network?I What subdomains exist below a certain domain name?

11 / 37



RecursiveDNS Server

Passive DNS

DNSQuery/response


DNS Server


I Where did this domain name point to in the past?I What domain names are hosted by a given

nameserver?

I What domain names point into a given IP network?I What subdomains exist below a certain domain name?

11 / 37



RecursiveDNS Server

Passive DNS

DNSQuery/response


DNS Server



nameserver?I What domain names point into a given IP network?

I What subdomains exist below a certain domain name?

11 / 37



RecursiveDNS Server

Passive DNS

DNSQuery/response


DNS Server



nameserver?I What domain names point into a given IP network?I What subdomains exist below a certain domain name?

11 / 37


State of the art

Previous Work

I Exposure: Finding malicious domains using passivedns analysis, in Network and Distributed SystemSecurity Symposium - NDSS, 2011.

I DNSSM: A large-scale Passive DNS SecurityMonitoring Framework, in IEEE/IFIP NetworkOperations and Management Symposium, 2012.

I SDBF: Smart DNS Brute-Forcer, in IEEE/IFIPNetwork Operations and Management Symposium -NOMS, 2012.

12 / 37


Outline

1 Introduction

2 Background

3 MAM

4 Analysis

5 Evaluation

6 Conclusions

13 / 37


MAM

How do we organize all the DNS data that we have?

I Preserve the hierarchy of Data (DNS & IP).

I Reduce the scale

I Minimize information loss due to aggregation.

I Fine control granularity for data analysis.

MAM is an enabler for aggregation and data retrieval.

14 / 37


MAM



I Reduce the scale




14 / 37


MAM



I Reduce the scale




14 / 37


MAM



I Reduce the scale




14 / 37


Multidimensional Aggregation Monitoring+ ApplicationsAggregation

I Scalable way to represent informationI Outline relevant correlated factsI Flexible granularity

I Features:I Custom Units (e.g. traffic packets, vehicle, traffic units)I choose criteria for aggregation

I Temporal and Spatial aggregationI Temporal: time windows split (β)I Spatial: keep nodes with activity > α e.g. traffic volume,

aggregate the others into their parents → needshierarchical relationships

Published in LISA’12.

15 / 37







Published in LISA’12.

15 / 37







Published in LISA’12.15 / 37


Space Partitioning

Example: IPV4 space partitioning, static vs dynamic

16 / 37


MAM for DNS

With MAM is possible to generate time aggregatedcombining multiple data.

I Two dimensions

I Hierarchically derived from data model (IPV4 & DNSData Space)

I Example

17 / 37


MAM for DNS


I Two dimensions

I Hierarchically derived from data model (IPV4 & DNSData Space)

I Example

17 / 37


MAM for DNS


I Two dimensionsI Hierarchically derived from data model (IPV4 & DNS

Data Space)I Example

dns: .ROOT

ip: 0.0.0.0/0

0.00% 100.00%

dns: $.betterblock.org..ROOT

ip: 173.201.208.1/32

33.33% 33.33%

dns: $.blockbuster.org..ROOT

ip: 72.233.2.58/32

33.33% 33.33%

dns: $.balconytv.com..ROOT

ip: 75.101.145.87/32

33.33% 33.33%

DNSIP

Accumulated activity Node activity

17 / 37


Example II

Multidimensional tree for source and destination names

18 / 37


Little Outline

So far...

I DNS Interest and Background

I Multidimensional Aggregation for hierarchical data

I Coming next ← Leverage DNS activity analysis withMAM

19 / 37


Little Outline

So far...




19 / 37


Little Outline

So far...




19 / 37


Outline

1 Introduction

2 Background

3 MAM

4 Analysis

5 Evaluation

6 Conclusions

20 / 37


Motivation

Asses the duration of the association (IP-NAME) over time

We want to keep the notion of subnet and subdomain.

This should be stable over timelets asume bmp.fr is fishing for bnp

21 / 37


Motivation

Asses the duration of the association (IP-NAME) over timeWe want to keep the notion of subnet and subdomain.This should be stable over time

lets asume bmp.fr is fishing for bnp

www.lip6.fr

132.227.104.15/32

21 / 37


Motivation

Asses the duration of the association (IP-NAME) over timeWe want to keep the notion of subnet and subdomain.

This should be stable over time

lets asume bmp.fr is fishing for bnp

www.bmp.fr

192.168.1.2/32

21 / 37


Formal Definition

Assuming a sequence of K Trees

I S = {T1 . . .TK} representing DNS recordassociation over time split in K

I n ∈ Nodes(Ti )I n.dns represents the DNS name (using ’dollar’ as terminal

symbol)I n.ip represents the IPv4 address as a tuple (address,

prefix length)I n.accum is the accumulated value representing the number

of A Records.

I The trees from S are aggregated according to a givenalpha

22 / 37


Formal Definition



I n ∈ Nodes(Ti )

I n.dns represents the DNS name (using ’dollar’ as terminalsymbol)

I n.ip represents the IPv4 address as a tuple (address,prefix length)

I n.accum is the accumulated value representing the numberof A Records.


22 / 37


Formal Definition




symbol)

I n.ip represents the IPv4 address as a tuple (address,prefix length)



22 / 37


Formal Definition





prefix length)



22 / 37


Formal Definition






of A Records.


22 / 37


Formal Definition






of A Records.


22 / 37


Steadiness Metric I (Similarity)

The similarity of nodes is positive if:

I n1, n2 Tree nodes are from the same domain (Totaloverlap)

I n1.dns ⊂ n2.dns OR n1.ip ⊂ n2.ip (Partial overlap)


Otherwise similarity is 0.Given n from T , we look for m ∈ Nodes(Tx) where∀ mx ∈ Nodes(Tx) : sim(n,m) ≥ sim(n,mx)

23 / 37








23 / 37







Otherwise similarity is 0.

Given n from T , we look for m ∈ Nodes(Tx) where∀ mx ∈ Nodes(Tx) : sim(n,m) ≥ sim(n,mx)

23 / 37








23 / 37


Steadiness Metric II (Similarity)

In other words:

I It’s almost as inserting the node into the tree, andlooking for a possible parent

24 / 37


Steadiness Metric II (Similarity)

In other words:

I It’s almost as inserting the node into the tree, andlooking for a possible parent

n

m0

m1 m2

24 / 37


Steadiness Metrics III

Tree comparison, how to establish a similarity criteria?

sim(n1, n2) = α× IP sim(n1, n2) + β ×DNS sim(n1, n2) + γ × vol sim(n1, n2)

IP sim(n1, n2) = 1− |n1prefix len − n2prefix len|32

DNS sim(n1, n2) =|n1dns ∩ n2.dns||n1dns ∪ n2dns |

vol sim(n1, n2) = 1− 0.01× |n1acc vol − n2acc vol |

25 / 37








25 / 37








25 / 37








25 / 37








25 / 37


Smoothing

Smoothing helps considering early time windows

I n1 ∈ Nodes(T1), n2 ∈ Nodes(T2)

I We compute n2 = most sim(n1) is

I stead(n1) = sim(n1, n2) + µ× stead(n2). With T0 asbase case and µ ∈ <.

So we compute the global steadiness of a Tree T by:

Persistence(T ) =

∑n∈Nodes(T )

stead(n)

|n∈Nodes(T )|

26 / 37


Smoothing

Smoothing helps considering early time windows

I n1 ∈ Nodes(T1), n2 ∈ Nodes(T2)

I We compute n2 = most sim(n1) is

I stead(n1) = sim(n1, n2) + µ× stead(n2). With T0 asbase case and µ ∈ <.

So we compute the global steadiness of a Tree T by:

Persistence(T ) =

∑n∈Nodes(T )

stead(n)

|n∈Nodes(T )|

26 / 37


Outline

1 Introduction

2 Background

3 MAM

4 Analysis

5 Evaluation

6 Conclusions

27 / 37


Data Set

Aggregation Window: 1 Week Time Length

I Macro: Up to 52 weeks from 2011-04-23 to2012-06-30 (662 K)

I Micro: 10 weeks maximum

Malicious data

I Time: Periodically, Steady

I Proportion: 0.1%, 1% and 10%

I Source: Blacklists (Exposure, WOT) 175K

Aggregation Granularity: 2%

28 / 37


Data Set




Malicious data





28 / 37


Data Set




Malicious data





28 / 37


Data Set




Malicious data





28 / 37


Steadiness Distribution

Distribution of Local Steadiness (Leafs)

I More than 50% of malicious nodes have less than 0.7of stability

I Less than 20% of malicious nodes have more than0.85 of stability

I Less than 40% of normal data have a steadiness of0.8% or less.

I Only 10% of normal data have a steadiness of lessthan 0.5

29 / 37








29 / 37








29 / 37








29 / 37


Microscopic observationMalicious domains causes a drop on average steadiness

30 / 37


Macroscopic observationMalicious domains causes a drop on average steadiness:Macro

31 / 37


Results IIAccuracy: Steadiness as metric for filtering maliciousdomains

32 / 37


Outline

1 Introduction

2 Background

3 MAM

4 Analysis

5 Evaluation

6 Conclusions

33 / 37


Contributions

I A methodology for assessing DNS - IP associationtime frame was proposed.

I Reduced the scale of data, helpful in the context ofnetwork security. (From 80K nodes to 2K withα = 2%)

I Definition of steadiness metrics for a local and globalscope was introduced.

I Evaluation using real data and during several timeframes. Validation of the metrics

I Scalability: It can be implemented using dynamicprogramming / distributed computing.

34 / 37


Contributions






34 / 37


Contributions






34 / 37


Contributions






34 / 37


Contributions






34 / 37


Contributions cont’d

Relevant Publications

TCP 2012: L. Dolberg, J. Francois, T. Engel.”Multidimensional Aggregation Monitoring”,Usenix LISA 2012

DNS 2013: L. Dolberg, J. Francois, T. Engel. ”DNSMalware Detection using Stability Metrics”,IEEE LCN 2013

35 / 37


Thanks!

36 / 37


Questions

37 / 37

Documents

DNS Monitoring, looking out for anomalies using the time ... · Lautaro Dolberg, Born in Buenos Aires - Argentina. 29 Years old I PhD Studies at Universite de Luxembourg. I Started