Can Drop but You Can’t Hide: persistent Estimation in High...

You Can Drop but You Can’t Hide:  ‐persistent Spread Estimation in High‐speed Networks

Presenter: Prof. Shigang Chen

He Huang1, Yu-E Sun2, Shigang Chen3, Shaojie Tang4,

Kai Han5, Jing Yuan6, Wenjian Yang1

1School of Computer Science and Technology, Soochow University, China2School of Rail Transportation, Soochow University, China

3Department of Computer and Information of Science and Engineering, University of Florida, US4Naveen Jindal School of Management, University of Texas at Dallas, US

5School of Computer Science and Technology, University of Science and Technology of China, China6Department of Computer Science, University of Texas at Dallas, US

19th April, 2018IEEE INFOCOM 2018

Traffic Measurement in High Speed Networks

2

Generalized Flow Size Measurement

Number of packets, number of bytes

Netflow

Generalized Flow Spread Measurement

Number of distinct elements in each flow, i.e. flow cardinality.

Scan detection, worm monitoring, proxy caching and content

access profiling, etc

Flow size v.s. Flow spread

3

1000000 packets

Size = 1000000, Spread = 1

……

1 packet

1 packet

1 packet

Size = 100Spread = 100

Persistent Spread

4

Stealthy DDoS attack, , , , ,

, , , , , Persistent element (source IP)

Limitation of Prior Art

5

Stealthy DDoS attack, , , , ,

, , , , , Persistent element (source IP)

Limitation 1: Only count persistent elements that appear in all periods

Limitation 2: Assume transient elements appear in one period

Problem Definition

6

We study a new problem called -persistent spreadestimation, which measures persist traffic elements ineach flow that appear during at least out of periods.

Other applications Identifying popular web files that are persistently accessed by

users over at least out of periods.

Profiling Internet access patterns

Monitoring scan activities

Online Persistent Traffic Measurement

7

Extremely high line speed

On-chip memory shared by Routing

Packet scheduling

Access control

Quality of service

Packet inspection and classification

Intrusion detection

Traffic measurement

How to fit in an extremely tight memory space!

Online Recoding

8

a bitmap for each flow f

0 0 0 0 0 0 0 0

0 1 2 3 4 5 6 71 11 1

Offline Operation: Bitwise SUM

9

1 0 0 1 0 0 0 1

0 1 2 3 4 5 6 7

,

,

,

, ,

,

0 0 0 1 0 1 0 1,

1 0 0 0 0 0 0 1,

Bitwise SUM0 0 2 0 1 0 32

Basic Idea

Known: , , fraction of counters whose valuesare

Unknown: , , number of elements that appearin out of measurement period.

Perdistent spread ∑ .

We derive the functional relationship between knownand unknown. , , provides T+1 equations to

solve for , .

0 0 2 0 1 0 32 V2 = 2 / 8

Per DestinationFlow

Recording Many Flows with Virtual Bitmaps

11

One physical bitmap for all

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

1

1

1 11 11 111 1

1 0 1 1 0 1 0 1 0 1 0 1 1 0 1 1 1 1 0 1 1 0 0 1

One virtual bitmap for each flow

Virtual Bitmaps

12

Space saving

Implicit indexing

Noise in virtual bitmap

12

0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0

1

1 10 11 1 1

1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 1 1 0 0 1

Experiment Results

Experiment setup: Dataset

One hour of data downloaded from CAIDA

38963 distinct flows, and 7179130 distinct elements

General setWe set 5 minutes as one measurement period.

Each study incoudes 8 measurement periods, i.e. T .

Memory ranges from 0.25MB ∼ 1MB.

13

Experiment Results (cont.)

14

Experiment Results (cont.)

15

Base Station

Conclusion  

A new traffic measurement problem that measuresnumber of persistent elements appearing in at leastout of predefined measurement periods.

A space-efficient solution for the problem

16

Q & AThank you!

Contact E-mail: sgchen@cise.ufl.edu