New Directions in TrafficMeasurement and Accounting

Cristian Estan

(joint work with George Varghese)

The Problem

• Measurement and monitoring of network traffic required for Internet backbones.

• Useful for short-term monitoring (e.g., DOS attacks), traffic engineering (e.g., rerouting), and accounting (e.g., usage based pricing)

• How can we do so without “tracking millions of ants to track a few elephants”?

State of the art – Cisco NetFlow

• Sample packets at high speeds;

• Per flow information based on samples;

• Aggregate (based on ASes, prefixes, ports) at the router;

• Problems: inaccurate (due to sampling and loss), memory-intensive, slow (needs DRAM).

Towards a NetFlow Alternative

• Small Percentage of flows (elephants) account for large percentage of traffic.

• Top 9% of flows account for 90% of AS pair traffic in backbones (Fang-Peterson).

• Can we directly track flows that send say over 1% of link bandwidth without keeping track of all flows?

How to identify large flows?

• Sample-Counting: uses sampling only to decide which flows to watch exhaustively.

• Multistage filter: uses multiple hash tables allowing large flows to be identified while only allowing a small number of small flows (false positives) to pass through filter.

We introduce two new methods for this purpose:

Identify large flows by sampling

Multistage filters

Operation of Sampled NetFlowHow accurate is Sampled NetFlow?

1 gigabyte/100 megabytes of data

Sampling one in 100 packets

Error 1GB 100MB

1% 39.24% 79.03%

3% 1.07% 41.48%

Operation of our algorithmsError 1GB 100MB

1% 5.6E-32 0.09%

3% 1.8E-94 6.96E-10

Error 1GB 100MB

1% 0.08% 49.69%

3% 4.66E-10 12.25%

Sampling 1/1000

Filter error: 0%

Comparison

Sampled NetFlow

Identify by sampling

Multistage filters

Accuracy Medium Good Good

False neg. Few (high var.) Few (low var.) Never

False pos. Few Few Few

Memory Big Small Very small

Complexity Low Low Medium

High Speed Implementation?

• John Huber of MMC Networks did a design of a chip doing filter counting.

• 450,000 transistors, under 1 watt of power, runs at OC-192 rates, 32 nsec per packet

• Seems easily feasible to implement sample counting with similar complexity.

Potential Application: scalable threshold accounting

• Measure flows sending over x% of link bandwidth using sample/filter counting.

• Bill using flat fee + per byte charge for flows over x%

• Track aggregates directly to avoid evasion using several flows, each < x%

• Generalizes usage based (x = 0) and duration based (x = 100) pricing.

Conclusions

• Paradigm shift for measurement by concentrating only on “heavy flows”

• Two new techniques (sample and filter counting) with small memory footprints and provable performance.

• Techniques make “threshold accounting” feasible, generalizing usage and duration based pricing.