Usenix Security 2004

AutographToward Automated, Distributed Worm Signature Detection

Hyang-Ah Kim Brad KarpCarnegie Mellon University Intel Research &

Carnegie Mellon University

Usenix Security 2004 2

Internet Worm Internet Worm

Large costs due to lost productivity Code Red: $2.6 billion, Slammer: $1 billion

Vulnerabilities still plentiful Smarter, faster, and more malicious worms easily possib

le [Staniford et al., 2002]

Internet Worm Quarantine Techniques Destination port blocking Infected source host IP blocking Content-based blocking Content-based blocking [Moore et al., 2003]

Usenix Security 2004 3

Content-based Blocking05:45:31.912454 90.196.22.196.1716 > 209.78.235.128.80: . 0:1460(1460) ack 1 win 8760 (DF)0x0000 4500 05dc 84af 4000 6f06 5315 5ac4 16c4 E.....@.o.S.Z...0x0010 d14e eb80 06b4 0050 5e86 fe57 440b 7c3b .N.....P^..WD.|;0x0020 5010 2238 6c8f 0000 4745 5420 2f64 6566 P."8l...GET./def0x0030 6175 6c74 2e69 6461 3f58 5858 5858 5858 ault.ida?XXXXXXX0x0040 5858 5858 5858 5858 5858 5858 5858 5858 XXXXXXXXXXXXXXXX . . . . .0x00e0 5858 5858 5858 5858 5858 5858 5858 5858 XXXXXXXXXXXXXXXX0x00f0 5858 5858 5858 5858 5858 5858 5858 5858 XXXXXXXXXXXXXXXX0x0100 5858 5858 5858 5858 5858 5858 5858 5858 XXXXXXXXXXXXXXXX0x0110 5858 5858 5858 5858 5825 7539 3039 3025 XXXXXXXXX%u9090%0x0120 7536 3835 3825 7563 6264 3325 7537 3830 u6858%ucbd3%u7800x0130 3125 7539 3039 3025 7536 3835 3825 7563 1%u9090%u6858%uc0x0140 6264 3325 7537 3830 3125 7539 3039 3025 bd3%u7801%u9090%0x0150 7536 3835 3825 7563 6264 3325 7537 3830 u6858%ucbd3%u7800x0160 3125 7539 3039 3025 7539 3039 3025 7538 1%u9090%u9090%u80x0170 3139 3025 7530 3063 3325 7530 3030 3325 190%u00c3%u0003%0x0180 7538 6230 3025 7535 3331 6225 7535 3366 u8b00%u531b%u53f0x0190 6625 7530 3037 3825 7530 3030 3025 7530 f%u0078%u0000%u00x01a0 303d 6120 4854 5450 2f31 2e30 0d0a 436f 0=a.HTTP/1.0..Co

. . . . .Signature: A Payload Content String Specific To A Worm

Signature for CodeRed II

Usenix Security 2004 4

Content-based Blocking

Our networkX

Traffic FilteringInternet

Signature for CodeRed II

Can be used by Bro, Snort, Cisco’s NBAR, ...

Usenix Security 2004 5

Signature derivation is too slow Current Signature Derivation Process

New worm outbreak Report of anomalies from people via phone/email/newsg

roup Worm trace is captured Manual analysis by security experts Signature generation

Labor-intensive, Human-mediated

Usenix Security 2004 6

Goal

Automatically generate signatures of previou

sly unknown Internet worms

as quickly as possible as accurately as possible

Usenix Security 2004 7

Our Work We focus on TCP worms that propagate

via scanning

Actually, any transport in which spoofed sources cannot communicate

successfully in which transport framing is known to monitor

Worm’s payloads share a common substring Vulnerability exploit part is not easily mutable

Usenix Security 2004 8

Outline Problem and Motivation Automated Signature Detection

Desiderata Technique Evaluation

Distributed Signature Detection Tattler Evaluation

Related Work Conclusion

Usenix Security 2004 9

Desiderata Automation: Minimal manual intervention

Signature quality: Sensitive & specific Sensitive: match all worms low false negative rate Specific: match only worms low false positive rate

Timeliness: Early detection

Application neutrality Broad applicability

Usenix Security 2004 10

Automated Signature Generation

Step 1: Select suspicious flows using heuristics Step 2: Generate signature using content-

prevalence analysis

Our network

Traffic Filtering

Internet Autograph Monitor

Signature

X

Usenix Security 2004 11

Heuristic: Flows from scanners are suspicious Focus on the successful flows from IPs who made unsuccessful con

nections to more than s destinations for last 24hours Suitable heuristic for TCP worm that scans network

Suspicious Flow Pool Holds reassembled, suspicious flows captured during the last time p

eriod t Triggers signature generation if there are more than flows

S1: Suspicious Flow SelectionReduce the work by filtering out vast amount of innocuous flows

Autograph (s=2)

Non-existent

Non-existentThis flow will be

selected

Usenix Security 2004 12

S1: Suspicious Flow Selection

Heuristic: Flows from scanners are suspicious Focus on the successful flows from IPs who made unsuccessful con

nections to more than s destinations for last 24hours Suitable heuristic for TCP worm that scans network

Suspicious Flow Pool Holds reassembled, suspicious flows captured during the last time p

eriod t Triggers signature generation if there are more than flows

Reduce the work by filtering out vast amount of innocuous flows

Usenix Security 2004 13

S2: Signature Generation

All instances of a worm have a common byte pattern specific to the worm Rationales Worms propagate by duplicating themselves Worms propagate using vulnerability of a service

Use the most frequent byte sequences across suspicious flows as signatures

How to find the most frequent byte sequences?

Usenix Security 2004 14

Worm-specific Pattern Detection Use the entire payloads

Brittle to byte insertion, deletion, reordering

XXXXXXXX9025753930399025YYYYFlow 1

Flow 2 XXXXXXX9025753930399025YYYYY

Usenix Security 2004 15

Worm-specific Pattern DetectionPartition flows into non-overlapping small blocks and c

ount the number of occurrences

Fixed-length Partition Still brittle to byte insertion, deletion, reordering

XXXXXXXX9025753930399025YYYYFlow 1

Flow 2 XXXXXXX9025753930399025YYYYY

Usenix Security 2004 16

Worm-specific Pattern Detection Content-based Payload Partitioning (COPP)

Determine boundaries of block using LBFS style Partition if Rabin fingerprint of a sliding window matches Breakmark Configurable parameters: content block size (minimum, average, ma

ximum), breakmark, sliding window Content Blocks

Breakmark = last 8bits of fingerprint (9025)

XXXXXXXX9025753930399025YYYYFlow 1

Flow 2 XXXXXXX9025753930399025YYYYY

Usenix Security 2004 17

Why Prevalence?

Worm flows dominate in the suspicious flow pool Content-blocks from worms are highly ranked

Nimda

CodeRed2

Nimda (16 different payloads)

WebDAV exploit

Innocuous, misclassified

< Prevalence Distribution in Suspicious Flow Pool>

Usenix Security 2004 18

Select Most Frequent Content Block

A B DA B EA C EA D

C FC D G

B

f0f1f2f3f4

f5H I Jf6I H Jf7G I Jf8

Usenix Security 2004 19

AAA

EE

A

FC

CC

DDDB

BB H

HGG

III

JJJ

Select Most Frequent Content Block

D

CEE

AAAA D

FCC D G

BB

B

HH

G

II

I

JJJ

f0f1f2f3f4

f5f6f7f8

f0 C Ff1 C D Gf2 A B Df3 A C Ef4 A B Ef5 A B Df6 H I Jf7 I H Jf8 G I J

Usenix Security 2004 20

Select Most Frequent Content Block

A

B

DA

B E

A

CE

A

DC

FC

DGB H

I JI

HJ

GI J

f0 C Ff1 C D Gf2 A B Df3 A C Ef4 A B Ef5 A B Df6 H I Jf7 I H Jf8 G I Jp≥3

W≥90%Signature:

Usenix Security 2004 21

Signature: A

Select Most Frequent Content Block

A

B

DA

B E

A

CE

A

DC

FC

DGB H

I JI

HJ

GI J

f0 C Ff1 C D Gf2 A B Df3 A C Ef4 A B Ef5 A B Df6 H I Jf7 I H Jf8 G I Jp≥3

W≥90%

Usenix Security 2004 22

Select Most Frequent Content Block

B

DBAAA

C EE

A

DF

CC

DGB H

I JI

HJ

GI J

p≥3

W≥90%Signature: Af0 C Ff1 C D Gf2 A B Df3 A C Ef4 A B Ef5 A B Df6 H I Jf7 I H Jf8 G I J

Usenix Security 2004 23

Select Most Frequent Content Block

FCC D

G HI JI

HJ

GI J

p≥3

W≥90%Signature: Af0 C Ff1 C D Gf2 A B Df3 A C Ef4 A B Ef5 A B Df6 H I Jf7 I H Jf8 G I J

I

Usenix Security 2004 24

Select Most Frequent Content Block

FCC DG

p≥3

W≥90%Signature: Af0 C Ff1 C D Gf2 A B Df3 A C Ef4 A B Ef5 A B Df6 H I Jf7 I H Jf8 G I J

IUSignature:

Usenix Security 2004 25

Outline Problem and Motivation Automated Signature Detection

Desiderata Technique Evaluation

Distributed Signature Detection Tattler Evaluation

Related Work Conclusion

Usenix Security 2004 26

Behavior of Signature Generation

Objectives Effect of COPP parameters on signature quality

Metrics Sensitivity = # of true alarms / total # of worm

flows false negatives Efficiency = # of true alarms / # of alarms

false positives Trace

Contains 24-hour http traffic Includes 17 different types of worm payloads

Usenix Security 2004 27

Signature Quality

Larger block sizes generate more specific signatures A range of w (90-95%, workload dependent)

produces a good signature

Usenix Security 2004 28

Outline Problem and Motivation Automated Signature Detection

Desiderata Technique Evaluation

Distributed Signature Detection Tattler Evaluation

Related Work Conclusion

Usenix Security 2004 29

Problem: Slow Payload Accumulation Before signature generation,

Detect scanners (possibly infected hosts) Aggressiveness (s ) of flow selection heuristics Accumulate payloads enough for content analysis Earliness ( ) of signature generation trigger

Infected vulnerable hosts ( =5, 63 monitors)

Info Sharing

Autograph Monitor

Aggressiveness of flow selection

s = 1 s = 4

None Luckiest 2% 60%Average 25% --

Scanners, Signatures Average <1% 15%

Usenix Security 2004 30

Faster Signature Detection Share the scanner information with others

Our network

Traffic FilteringInternet

Autograph Monitor

Network A

Network C

Netw

ork

B

tattler

Usenix Security 2004 31

Benefit from tattler

Objective Measure the detection speed and the signature quality

Methodology Trace generation

Background noise flows from the real 24hr trace Captured worm flows from simulation (63 monitors)

Signature generation with COPP varying suspect flow pool size Metrics

Percentage of infected hosts Number of unspecific signatures (that causes false positives)

Usenix Security 2004 32

Tradeoff: Speed vs. Quality

Decreasing s and parameters faster signature generation more false positives

x

x

= 15, s = 2,< 2% infected

Usenix Security 2004 33

Attacks Overload due to flow reassembly

Multiple instances of Autograph on separate HW (port-disjoint) Suspicious flow sampling under heavy load

Abuse Autograph for DoS: pollute suspicious flow pool Port scan and then send innocuous traffic Distributed verification of signatures at many monitors Source-address-spoofed port scan Reply with SYN/ACK on behalf of non-existent hosts/services

Usenix Security 2004 34

Related Work EarlyBird [Singh et al. 2003]

Pure content-based approach first, then address dispersion heuristic Targets a single high speed link; no flow reassembly

HoneyComb [Kreibich et al. 2003] Signature detection by Honeypot & LCS algorithm Targets host-based deployment; small number of flows

Honeyd [Provos 2003] Use distributed honeypots to gather worm payloads & infected IP ad

dresses quickly Focus on harvesting malicious traffic

DOMINO [Yegneswaran et al. 2004] Scanner IP information sharing among distributed nodes Distributed monitoring assures earlier & more accurate detection

Usenix Security 2004 35

Future Work Online evaluation with diverse traces Deployment on distributed sites Broader set of suspicious flow selection

heuristics Non-scanning worms (ex. hit-list worms,

topological worms, email worms) UDP worms

Distributed agreement for signature quality testing

Usenix Security 2004 36

Conclusion Stopping spread of novel worms requires

early generation of signatures Autograph: automated signature detection

system Suspicious flow selection→ Content prevalence

analysis COPP: robustness against payload variability Distributed monitoring: faster signature

generation Autograph finds sensitive & specific

signatures early in real network traces