Identification of Bot Commands By Run-time Execution Monitoring

Younghee Park, Douglas S. ReevesNorth Carolina State University

ACSAC 2009

1

OUTLINE

1. INTRODUCTION2. THE PROPOSED METHOD3. EXPERIMENTAL EVALUATION4. DISCUSSION5. CONCLUSION

2

OUTLINE

1. INTRODUCTION2. THE PROPOSED METHOD3. EXPERIMENTAL EVALUATION4. DISCUSSION5. CONCLUSION

3

About Botnets

• A major source of network threats– DDoS, spam, identity theft, click frauds

• A variety of protocols– IRC, HTTP, peer-to-peer

• Botnets is estimated to be in the millions of hosts

4

BotTee

• Monitoring and analyzing bot execution to identify the bot commands that are being executed.

• Bot commands with the same purpose that is highly correlated, across all types of bots.

• Bot commands can be accurately identified during execution.

5

OUTLINE

1. INTRODUCTION2. THE PROPOSED METHOD3. EXPERIMENTAL EVALUATION4. DISCUSSION5. CONCLUSION

6

System architecture for BotTee

7

Bot behavior classification through bot commands

8

Hooking API calls• These bots invoke Windows functions through

the API provided to applications.• When each API call is intercepted, the time is also

recorded.• To hook only a limited set of Windows API calls.• Approximately 300 commonly-used API functions

from 50 real bot instances.• 153 APIs were in file kernel32.dll; the rest

were found in user32.dll, advapi32.dll, ws2_32.dll (Wsock32.dll), etc.

9

Bot Command Identifier

• What sequence of system calls may correspond to a bot command? recv and send

• Repeated consecutive occurrences of the same API call in a trace are eliminated.

• γ = 2– AAABCCAAAADDDA → AABCCAADDA

• Semantic unit ‘synflood’– socket, TLSGetValue, InterlockedDecrement, ioctlsocket, connect, WaitForSingleObject, etc.

10

Correlation Engine

• This engine is used to create command templates, and to match captured system call traces to these templates.– Longest common subsequence algorithm (LCS) ,

and statistical correlation

• Define θ1 as P(ρi,j > δ) | H1)

11

Common API Call Trace• The CACTs for each command include important

APIs for identifying the execution of the bot command.

• These are termed the featured APIs.• CACT of ‘dns’ with the length 30.– recv, TlsGetValue, GetLocalTime, GetUserDefaultLCID, WideCharToMultiByte, GetTimeFormatA, GetConsoleMode, WriteConsoleA, WriteFile, inet_addr, ..., GetTickCount, InterlockedExchange, CloseHandle, gethostbynam, inet_ntoa, send,

12

A Real-time Semantic Behavior Matcher

• Semantic unit is compared to all of the templates of bot commands.

• A candidate template must be identified.• Computing the correlation of Semantic unit’s

timing vector with each timing vector in the template.

• Additional information can be recorded about the arguments of API calls that are hooked.

13

OUTLINE

1. INTRODUCTION2. THE PROPOSED METHOD3. EXPERIMENTAL EVALUATION4. DISCUSSION5. CONCLUSION

14

Implementation and Experiments

• Prototype of BotTee– Used the Deviare API for intercepting Windows

API calls on the fly.

• A botnet in a private network was deployed.• Among 167 available bot source codes, there

were 103 variants– Agobot, Spybot, Sdbot, and Jrbot

15

Performance Overhead of Hooking

16

Correlation Results

17

Identification of Specific Bot Commands

18

False Identification

• If CACTs are not distinctive enough to differentiate bots from non-bot programs.

19

Detection Rate with API Call Injection Attack

• Injection for obfuscation purposes may be intended to obfuscate timing analysis and correlation as well.

20

OUTLINE

1. INTRODUCTION2. THE PROPOSED METHOD3. EXPERIMENTAL EVALUATION4. DISCUSSION5. CONCLUSION

21

DISCUSSION

• The more accurately that botnet-driven network threats can be identified.

• BotTee can specify victims targeted by active botnets and infer the overall behaviors of the active botnets.

• The hooking technique allows potentially malicious bot commands to be replaced by more benign actions, or to be thwarted.

22

OUTLINE

1. INTRODUCTION2. THE PROPOSED METHOD3. EXPERIMENTAL EVALUATION4. DISCUSSION5. CONCLUSION

23

CONCLUSION

• A method for identifying the high-level commands being executed by a bot, in real time.

• Comparison of the resulting traces with a previously-captured set of bot command templates.

• This held true even for commands executed by bots from other bot families.

24