A Survey of Obfuscations in Prevalent Packer Tools Project Paradyn / Dyninst Week Madison, Wisconsin March 26, 2012 A Survey of Obfuscations in Prevalent Packer Tools Kevin Roundy

Paradyn Project

Paradyn / Dyninst Week

Madison, Wisconsin

March 26, 2012

A Survey of Obfuscations in

Prevalent Packer Tools

Kevin Roundy

1 A Survey of Prevalent Obfuscations

Types of program analysis


Friendly binary

Source code

Uncooperative binary

Hostile binary

CFG

mov eax, *[ebp+8]

leave

ret

Function

Basic block

loop

Analysis building blocks

Tools

o Interactive debuggers

o Automated testing

o Combinatorial testing

o Code-coverage test generation

o Fault localization

o Backwards slicing

o Correlation of statement

executions and test failures

o Vulnerability analysis

o Taint analysis

o Symbolic evaluation


Analysis steps 1. Extract code bytes

2. Disassemble

3. Identify functions

4. Build comprehension tools

5. Patch/modify the code

6. Trace code’s execution

Toolkit:

Defensive-mode Dyninst


o Automated testing




o Backwards slicing




o Taint analysis


Analysis building blocks

Toolkit:

Defensive-mode Dyninst


o Automated testing




o Backwards slicing




o Taint analysis




2. Disassemble



5. Patch/modify the code


Binary packing tools

5 A Survey of Prevalent Obfuscations * Packer (r)evolution. Panda Research, 2008. Two-month average Feb-March 2008.

Packer Malware market share*

OVERALL 75%-80%

UPX 9.45%

PolyEnE 6.21%

PECompact 2.59%

Upack 2.08%

nPack 1.74%

ASPack 1.29%

FSG 1.26%

Nspack 0.89%

ASProtect 0.43%

Armadillo 0.37%

Yoda’s Prot. 0.33%

WinUpack 0.17%

MEW 0.13%

Open source cross-

platform

Open source cross-

platform

Anti-reverse

engineer-ing

Fast, Small, Good

Anti-reverse

engineer-ing

Outline

a. Code packing

b. Code overwriting



2. Disassemble



5. Patch/modify the binary


Code packing


Storm Worm Entry Point

7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e

80 89 08 27 c0 73 1c 88 48 6a d8 6a d0 56 4b

fe 92 57 af 40 0c b6 f2 64 32 f5 07 b6 66 21

0c 85 a5 94 2b 20 fd 5b 95 e7 c2 16 90 14 8a

14 26 60 d9 83 a1 37 1b 2f b9 51 84 02 1c 22

8e 63 01

7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e

80 89 08 27 c0 73 1c 88 48 6a d8 6a d0 56 4b

fe 92 57 af 40 0c b6 f2 64 32 f5 07 b6 66 21

80 89 08 27 c0 73 1c 88 48 6a d8 6a d0 56 4b

0c 85 a5 94 2b 20 fd 5b 95 e7 c2 16 90 14 8a

14 26 60 d9 83 a1 37 1b 2f b9 51 84 02 1c 22

8e 63 01 c0 73 1c 88 48 c0 73 1c 88 48 77 0e

Code packing

Aspack

Code overwriting


Storm Worm Entry Point

7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e

80 89 08 27 c0 73 1c 88 48 6a d8 6a d0 56 4b

fe 92 57 af 40 0c b6 f2 64 32 f5 07 b6 66 21

0c 85 a5 94 2b 20 fd 5b 95 e7 c2 16 90 14 8a

14 26 60 d9 83 a1 37 1b 2f b9 51 84 02 1c 22

8e 63 01

7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e

80 89 08 27 c0 73 1c 88 48 6a d8 6a d0 56 4b

fe 92 57 af 40 0c b6 f2 64 32 f5 07 b6 66 21

80 89 08 27 c0 73 1c 88 48 6a d8 6a d0 56 4b

0c 85 a5 94 2b 20 fd 5b 95 e7 c2 16 90 14 8a

14 26 60 d9 83 a1 37 1b 2f b9 51 84 02 1c 22

8e 63 01 c0 73 1c 88 48 c0 73 1c 88 48 77 0e

Entry Point

Code packing Code overwriting

7a 77 0e 20 e9 3d e0 09 e8 68 c0 45 be 79 5e

80 89 08 27 c0 73 1c 88 48 6a d8 6a d0 56 4b

fe 92 57 af 40 0c b6 f2 64 32 f5 07 b6 66 21

0c 85 a5 94 2b 20 fd 79 5e 80 89 08 27 c0 73

1c 88 48 6a d8 5b 95 e7 c2 16 90 14 8a 14 26

60 d9 83 a1 37 1b 2f b9 51 84 02 1c 22 8e 63

60 d9 83 a1 37 1b 2f b9 51 84 02 1c 22 8e 63

malware Aspack

1B - 8KB

Upack

Outline

a. Unresolvable control flow

b. Call-stack tampering

c. Signals and exceptions

d. Ambiguous code & data

e. Disassembler fuzzing



2. Disassemble





invalid target non-standard indirect non-standard return

Unresolvable control flow


push eax ret

call 401000

Invalid Region

call ptr[eax]

?

jmp eax

? ?

Call-stack tampering


03 04 05 06 07 08 09 0a 0b 0c 0d

e8 03 00 00 00 e9 eb 04 5d 45 55 c3

CALL JMP 40d00a 459dd4f7

JMP POP INC PUSH RET 40d00e ebp ebp ebp

Address 40d002

Bytes

Storm Worm

Exception-based control flow

Exception State

eip 401002

...


eip 402d8a

xor eax,eax mov ecx,*[eax] push eax ...

Operating System

access violation handler

… mov *[ebp],eax mov 402d8a,edx mov edx,*[eax+b8]

Popov, Debray, Andrews. Usenix 2007. Danekhar. http://www.codeproject.com/KB/system/inject2exe.aspx 2005.

http://www.codeproject.com/KB/system/inject2exe.aspx

Ambiguous code and data

o Bytes after call instructions

o Junk after exception-raising instruction

o In-place decryption of unpacked code


Yoda’s Protector

Outline

a. Missing call/ret instructions

b. Extra call/ret instructions

c. Overlapping functions

d. Overlapping basic blocks



2. Disassemble





Outline

a. Missing call/ret instructions

b. Extra call/ret instructions

c. Overlapping functions

d. Overlapping basic blocks



2. Disassemble





Extra call/ret instructions


push <targ> ret

mov edi,esi pop ebp

jmp <targ>

call <targ>

jmp <targ2>

call <targ>

pop esi

pop ebp

call <targ>

Overlapping functions


Function Function Function

Overlapping functions


Function Function Function

Optional preamble

Shared teardown

code

0x454017

0x45401b

0x45401e

b8 eb 07 b9 eb 0f 90 eb 08 fd eb 0b

mov eax, ebb907eb

jmp 45402c

seto bl or ch,bh jmp 45402e

jmp 454028

Address

Bytes

Block 1

Block 2

Block 3

Overlapping blocks

20

Overlapping blocks


18 19 1a 1b 1c 1d 1e 1f 20 21 22

e8 03 00 00 00 e9 eb 04 5d 45 55 c3

mov eax, ebb907eb seto bl or ch,bh jmp 45402e

jmp 45402c

jmp 454028

Address 454017

Bytes

Block 1

Block 2

Block 3

Outline

a. Obfuscated constants

b. ABI violations

c. Do-nothing code



2. Disassemble





Outline

a. Self-checksumming

b. Stolen bytes

c. Anti-unpacking



2. Disassemble





Self-checksumming


checksum routine xor eax, eax

cmp eax, .chksum jne .fail

add eax, ptr[ebx] add 4, ebx cmp ebx, 0x41000 jne .loop

pass fail fail

process

fail

Payload code

Binary file

Bootstrap code

25

Stolen bytes

malware.exe Import Address Table

kernel32.dll

.loadlibrary

…

call ptr [IAT-entry]

kernel32.dll

loadlibrary

…

mov edi, edi

push ebp

mov ebp, esp

cmp ptr[ebp+8],0

…

malware.asprotect.exe Import Address Table

buffer

kernel32.dll

.loadlibrary mov edi, edi

push ebp

mov ebp, esp

cmp ptr[ebp+8],0

…

mov edi, edi

push ebp

mov ebp, esp

cmp ptr[ebp+8],0

.stolen

…

call buffer.stolen

kernel32.dll

loadlibrary

…

Outline

a. Stolen bytes

b. Non-standard API calls

c. Anti-debugging



2. Disassemble





Outline



2. Disassemble





Analysis tool

Dyninst

Adapting Dyninst for Malware

31 Malware Analysis and Instrumentation

Mutator

program binary

7a 77 0e 20 e9

3d e0 09 e8 68

c0 45 be 79 5e

80 89 08 27 c0

73 1c 88 48 6a

d8 6a d0 56 4b

fe 92 57 af 40

0c b6 f2 64 32

f5 07 b6 66 21

Control flow

analyzer

Instrumenter Data flow

analyzer

CFG CFG

Adapting Dyninst for Malware


Analysis tool

Dyninst

Control flow

analyzer

Instrument-

er

Data flow

analyzer

printf(…)

counter++

if (pred)

callback(…)

getTarget(insn)

Code snippets Mutator

SR- Dyninst

Control flow

analyzer

Data flow

analyzer

program binary

7a 77 0e 20 e9

3d e0 09 e8 68

c0 45 be 79 5e

80 89 08 27 c0

73 1c 88 48 6a

d8 6a d0 56 4b

fe 92 57 af 40

0c b6 f2 64 32

f5 07 b6 66 21

CFG

Instrumenter

static-dynamic analysis Sensitivity

Resistant

Instrumenter

Documents

A Survey of Obfuscations in Prevalent Packer Tools Project Paradyn / Dyninst Week Madison, Wisconsin March 26, 2012 A Survey of Obfuscations in Prevalent Packer Tools Kevin Roundy