Upload
zeal
View
87
Download
4
Tags:
Embed Size (px)
DESCRIPTION
Malware Detection. Slides courtesy of Mihai Christodorescu. The Rising Malware Tide. Malware is software with unwanted functionality. Viruses, trojans, backdoors, bots, adware, spyware, browser hijackers, downloaders, droppers, keyloggers, password stealers, ... “Blended” threats - PowerPoint PPT Presentation
Citation preview
Malware DetectionSlides courtesy of Mihai Christodorescu
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 2
The Rising Malware Tide• Malware is software with unwanted
functionality.Viruses, trojans, backdoors, bots, adware, spyware, browser hijackers, downloaders, droppers, keyloggers, password stealers, ...
• “Blended” threats
100,000,000 machines are infected.[Vint Cerf, World Economic Forum 2007]
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 3
Organized Cyber-Crime• Boom in online fraud:
– Spamming– Trade in stolen data– Financial fraud– ID theft
Malware is the tool of the trade.
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 4
The Changing Threat Landscape
1995: Hobby malware, for fun• Show programming prowess• Single author
2007: Professional malware, for profit• Collaborative development• Bug-fix releases, code reuse
Botnets: distributed computing has finally arrived.
Creator of the Melissa
worm
?
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 5
Failure of Signature Detectors
Malware detectors still use signatures.
Malware is obfuscated/transformed easily.Software diversity used successfully by malware.
Internet
ac028c0e86009d8edfac0ac075fbe81cfd72ef50b91000f7f15052b90:*:504b03040a0001000800*...*:188420:181779:*:8ad6900f5088cab9356678e43c...3:*:3e3c623e6c696e6b3c2f6...
Virus Scanner
Known Malware
New Malware 1New Malware 2
Paradigm shift in malware creation,
yet no change in malware detection!
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 6
Focus On Behavior
[Kaspersky Labs, Symantec]
2001 2002 2003 2004 2005 2006
10
1,000
100,000
10,000
1
8,82111,136 20,731
31,726 53,95086,876
New malware & malware families
Time
100
325 335 274 202 (est.)
A family is a collection of behaviors.A behavior can be shared by many families.
Family = malware with a common code base.
Number of families
stays constant.
Number of variants
grows exponentiall
y.
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 7
Main thesis
Detection of obfuscated malware requires a semantic analysis of program behavior.
Program verification provides the techniques necessary to perform malware
detection effectively and efficiently.
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 8
Specifying Behavior
Byte signatures allow for fast detection.– But not resilient to obfuscation.
High-level descriptions require expensive detection.– Resilient to obfuscation.
Syntactic Semantic
Execution of program M causes the system to reach a state where a copy of M has been sent by email.
“”
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 9
Connect
Send
Malspec: Self-Propagation by Email
Netsky.B
push 10hpush eaxpush edicall connectpush esipush eaxpush [ebp+hMem]call wsprintfAadd esp, 0Chpush [ebp+hMem]call lstrlenApush 0push eaxpush [ebp+hMem]push ebxpush eaxpush ecxpush edicall send
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 10
Connect
Send
push 10hpush eaxpush edicall connectpush esipush eaxpush [ebp+hMem]call wsprintfAadd esp, 0Chpush [ebp+hMem]call lstrlenApush 0push eaxpush [ebp+hMem]push ebxpush eaxpush ecxpush edicall send
Netsky.B
X := Arg1
Arg1 = X &Arg2 = “EHLO.*”
= +Semantic component
describesdependency constraints.
Syntactic component describestemporal
constraints.
Malspec: Self-Propagation by Email
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 11
“Read Own Exe. Image”“Send Email”
Building a Real Malspec
send(X,“DATA”)
X:=socket()
connect(X)
send(X,“EHLO”)
send(X,T)
Y:=read(Z)
Z:=open(S)
S:=process_name()
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 12
“Read Own Exe. Image”“Send Email”
send(X,“DATA”)
Building a Real MalspecX:=socket()
connect(X)
send(X,“EHLO”)
Y:=read(Z)
send(X,T)
Z:=open(S)
S:=process_name()
send(X,T))),Base64(l(StringEqua YT
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 13
send(X,“DATA”)
Malspec ConstraintsX:=socket()
connect(X)
send(X,“EHLO”)
Y:=read(Z)
send(X,T))),Base64(l(StringEqua YT
Z:=open(S)
S:=process_name()Local constraint
Dependence constraint:X after socket = X before connect
Dependence constraint
AutomatingMalspec Creation:Malspec Mining
MalwareSample
BenignProgramBenign
ProgramBenignProgramBenign
Program
—
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 14
Malspecs Benefits
X:=socket()
connect(X)
send(X,“EHLO”)
send(X,“DATA”)Y:=read(Z)
send(X,T))),Base64(l(StringEqua YT
Z:=open(S)
S:=process_name()
Choice of security-sensitive operations
Constraint-based execution order
Dependences free of obfuscation artifacts
Expressive to describe even obfuscated behavior.
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 15
Malspec Detection Strategies• Static analysis
• Dynamic analysis
• Host-based IDS
• Inline Reference Monitors
X:=socket()
connect(X)
send(X,“EHLO”)
send(X,“DATA”)Y:=read(Z)
send(X,T))),Base64(l(StringEqua YT
Z:=open(S)
S:=process_name()
Malspecs are independent of detection method.
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 16
Detection of Malicious Behavior
BinaryFile
MalwareDetector
X:=socket()
connect(X)
send(X,“EHLO”)
send(X,“DATA”) Y:=read(Z)send(X,T)
)),Base64(l(StringEqua YT
Z:=open(S)
S:=process_name()
Goal: Find a program path that matches the malspec.
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 17
Find A Malicious Program Path
X:=socket()
connect(X)
send(X,“EHLO”)
send(X,“DATA”) Y:=read(Z)send(X,T)
)),Base64(l(StringEqua YT
Z:=open(S)
S:=process_name()
Interprocedural Control-Flow Graph
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 18
1) Match Malspec OperationsX:=socket()
connect(X)
send(X,“EHLO”)
send(X,“DATA”)Y:=read(Z)
send(X,T))),Base64(l(StringEqua YT
Z:=open(S)
S:=process_name()
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 19
2) Match Malspec Constraints
X:=socket()
connect(X)
send(X,“EHLO”)
send(X,“DATA”)Y:=read(Z)
send(X,T))),Base64(l(StringEqua YT
Z:=open(S)
S:=process_name()
Malspec Constraint:Z after open = Z before
read
Program Constraint:The program fragment preserves the program expression bound to Z.
Like a semantic def-use
constraint.
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 20
2) Match Malspec Constraints
Program Constraint:The program fragment preserves the program expression bound to Z.
Semantic nop wrt E = program fragment preserving an expression E.
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 21
2) Match Malspec Constraints
Program Constraint:The program fragment preserves the program expression bound to Z.
Need an Oracle...
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 22
Advances in Decision Procedures
Dramatic improvements in SAT solvers:– SATO [Zhang, CADE 1997]– GRASP [Marques-Silva & Sakallah, 1999]– zChaff [Moskewicz et al., DAC 2001]– BerkMin [Goldberg & Novikov, DATE 2002]
SAT-based Bounded Model Checking:[Clarke et al., FMSD 2001]
– SAT-specific speedups [Strichman, CHARME 2001]– Richer logics [Seshia et al., DAC 2003]
A decision procedure can approx. an Oracle.
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 23
Using Decision Procedures
Program Constraint:The program fragment preserves the program expression bound to Z.
Decisionprocedure
P True/False
P
add esp, 0Chpush[ebp+hMem]
P
P
4
12
12
01
01
espesphMemebpmemoryespmemory
espesp
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 24
Semantics-AwareMalware Detector
Semantics-Aware Detector
Disassembler CFGconstructor
BinaryFile
CFG
Graphmatching
Malspec
Malspec operations
Malspec constraints
Yes / No
IDA Pro
[Detlefs et al., “Simplify,” 2004][Lahiri & Seshia, CAV 2004]
Constraintsatisfaction
Simplify
UCLID
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 25
Effective DetectionWith hard-coded semantic-nop
patterns:
With decision procedures:
Commercial AV
SAFE
Known malware 100% 100%Obfuscated variants
0% 100%
Malspec source
Variants detected
# of AV signatures
# of SAMD malspecs
Netsky.B C,D,O,P,T,W 7 1Bagle.I J,N,O,P,R,Y 7 1
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 26
Semantic-Nop Detection BenefitsSemantic-Nop features:• Flow sensitivity• Binding procedure• Decision procedures• Rich constraints
Obfuscation resilience:
• Code reordering• Register renaming• Junk code• Code substitution
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 27
Detection Performance
300–800 s
Powerful decision procedures are expensive.
1–9 s
Simplify theorem proverUCLID
bounded model
checker
SAFE pattern matching
Idea:Use expensive decision
procedures only if cheap decision procedures do not provide a
definitive answer.
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 28
Stack of Decision Procedures
Simplify theorem proverUCLID
bounded model
checker
SAFE pattern matching
Random execution
Average cost, same decision power.
Yes
No
Yes
Yes/No
“No, code does not
satisfy constraint!”
Constraint
Program fragment ?
?
?
University of Wisconsin, Madison Mihai Christodorescu – “Behavior-based Malware Detection” 29
Performance Results
Malware Minimum Average MaximumNetsky
(B,C,D,O,P,T,W)
Bagle(I,J,N,O,P,R,Y)
Bagle(obfuscated
variants)
Detection times in seconds
60.56 99.57 140.08
36.00 56.41 97.13
74.81 140.14 186.50
Test setup: 1 GHz CPU, 1 GB RAMComparison:
Commercial signature-based detector: <1sDecision procedure-based detector: >300s