2emGrimoire: Synthesizing Structure while Fuzzing - Usenix ... · Grimoire:SynthesizingStructurewhileFuzzing UsenixSecurity2019,SantaClara August16,2019 TimBlazytko,CorneliusAschermann,MoritzSchlögel,AliAbbasi,

Grimoire: Synthesizing Structure while Fuzzing

Usenix Security 2019, Santa ClaraAugust 16, 2019

Tim Blazytko, Cornelius Aschermann, Moritz Schlögel, Ali Abbasi,Sergej Schumilo, Simon Wörner, and Thorsten Holz

Chair for Systems SecurityRuhr-Universität Bochum

Goal: Finding bugs in programs expecting structured input

libxml2

Tiny C Compiler

JavaScriptCore

Boolector

1

Let’s fuzz!

First attempt: Blind fuzzing

3

First attempt: Blind fuzzing

State space

Interesting area

Uninteresting area

Can we do better?

4

Coverage-guided fuzzing

Program instrumentation

5


5


5


0

...

1

...

0

...

1

...

5


0

...

1

...

0

...

1

...

5

Small-scale mutations

• Bitflips

• Simple arithmetic• Force specific, “interesting” values• Havoc: “random” mutations• Repetition• Splicing

6


• Bitflips

• Simple arithmetic• Force specific, “interesting” values• Havoc: “random” mutations• Repetition• Splicing

A B C D

1010 1011 1100 1101

1011 1010 1101 1100

B A D C

6


• Bitflips• Simple arithmetic

• Force specific, “interesting” values• Havoc: “random” mutations• Repetition• Splicing

B A D C

1011 1010 1101 1100

1011 1010 1110 0100

B A E 4

+8

6


• Bitflips• Simple arithmetic• Force specific, “interesting” values

• Havoc: “random” mutations• Repetition• Splicing

B A E 4

1011 1010 1110 1100

1011 1010 1110 0000

B A E 0

6


• Bitflips• Simple arithmetic• Force specific, “interesting” values• Havoc: “random” mutations

• Repetition• Splicing

A B C D

1010 1011 1100 1101

1011 1010 1101 0000

B A D 0

6


• Bitflips• Simple arithmetic• Force specific, “interesting” values• Havoc: “random” mutations• Repetition

• Splicing

A B

1010 1011

1010 1011 1010 1011

A B A B

6


• Bitflips• Simple arithmetic• Force specific, “interesting” values• Havoc: “random” mutations• Repetition• Splicing

A B C D

1010 1011 1100 1100

1010 1011 1100 1101

B A D 0

6


Observation: Mutations modify the input only slightly

7

While this input works well ...

000000 050045 043104 030455 032456 022412152714 152301 154305 000010 142320 005306030061 030040 067440 065142 036012 020074000020 052057 070171 020145 054057 061117062552 072143 027440 000030 072523 072142070171 020145 043057 071157 020155 043057000040 071157 052155 070171 020145 020061041057 067502 020170 000050 020133 020060020060 030061 020060 030061 020060 020135000060 046457 072141 064562 020170 020133020061 020060 020060 000070 020061 020060020060 020135 051057 071545 072557 061562

8

While this input works well ...

000001 050045 043104 030455 032456 022412152714 152301 154305 000011 142320 005306030061 030040 067440 065142 036092 020074000021 052057 070171 020145 054057 061117062552 072143 027440 000031 072523 072142000000 020145 043057 071157 020155 043057000041 071157 052155 070171 020145 020061041057 067502 020170 000051 020133 020060020060 030061 020060 000000 020060 020135000061 046457 072141 064562 020170 020133020061 020060 020060 000071 020061 020060020060 020135 05105F 071545 072557 061562

8


State space

Interesting area

Uninteresting area

Mutations (cov.-guided)

9


Observation: Mutations modify the input only slightly

Caveat: Not all programs are equal

10

... this one is problematic

def some_function(self):s = "hi mom! "if self.famous:

return s + "I'm famous!"else:

self.confidence = 0return s + "*crying*"

Insight: Mutation requires input’s structure

11


deb1some_functioasdflf):s = "hi mom! "if ? ?``famous:

reABCDEFGH "I'm famous!"else:

self.confidence = 0return s + 0000ying*"


11


deb1some_functioasdflf):s = "hi mom! "if ? ?``famous:

reABCDEFGH "I'm famous!"else:

self.confidence = 0return s + 0000ying*"


11


State space

Interesting area

Uninteresting area


How to cross large gaps?

Solution: Grammar-based fuzzing

12


State space

Interesting area

Uninteresting area


How to cross large gaps?

Solution: Grammar-based fuzzing

12

Large-scale mutations

State space

Interesting area

Uninteresting area

Mutations (grammar)

Now crossing large gaps!

Problem: Creating a grammar requires human-effort

13

Large-scale mutations

State space

Interesting area

Uninteresting area

Mutations (grammar)

Now crossing large gaps!

Problem: Creating a grammar requires human-effort

13

Our approach

Grimoire: Best of both worlds

• Learn structure of inputs via fuzz testing

• Apply large-scale mutations on learned structures

• Profit!

15

Input generalization

pprint 'aaaa'

|pp|ri|nt|_'|aa|aa|'|

split

16


pprint 'aaaa'

|pp|ri|nt|_'|aa|aa|'|

split

16


pp|ri|nt|_'|aa|aa|'

pprint ' '

16


pp|ri|nt|_'|aa|aa|'

rint 'aaaa'

pprint ' '

16


pp|ri|nt|_'|aa|aa|'

rint 'aaaa'

pprint ' '

16


pp|ri|nt|_'|aa|aa|'

rint 'aaaa'

pp

rint ' '

16


pp|ri|nt|_'|aa|aa|'

ppnt 'aaaa'

ppri

nt ' '

16


pp|ri|nt|_'|aa|aa|'

ppri 'aaaa'

pprint

' '

16


pp|ri|nt|_'|aa|aa|'

pprintaaaa'

pprint '

'

16


pp|ri|nt|_'|aa|aa|'

pprint 'aa'

pprint '

'

16


pp|ri|nt|_'|aa|aa|'

pprint 'aa'

pprint '

'

16


pp|ri|nt|_'|aa|aa|'

pprint 'aa'

pprint '

'

16


pp|ri|nt|_'|aa|aa|'

pprint 'aaaa

pprint ' '

16


pp|ri|nt|_'|aa|aa|'

pprint 'aaaa

pprint ' '

pprint ' '

16


if(x>1) then x=3 end

if(x>1)|then|x=3|end

if(x>1) then end

17




if(x>1) then end

split

17




if(x>1) then end

generalize

17

Why do we generalize inputs?

if(x>1) then endpprint ' '

Structure-dependent mutations

18



if(x>1) then end

pick input


18



if(x>1) then end

select gap


18



if(x>1) then pprint ' ' end

replace gap


18



if(x>1) then pprint ' ' end

replace gap


18

Input extension

concat

pprint ' ' x= y+

19

Input extension

concat

pprint ' ' x= y+

19

Input extension

concat

pprint ' ' x= y+

pprint ''x=y+

19

Input extension

concat

pprint ' ' x= y+

pprint ''x=y+ x=y+pprint ''

19

Recursive replacement

pprint ' ' if(x>1) then end x= y+

if(x>1)

if(x>1) pprint ' '

if(x>1) pprint ' x= y+ '

20



if(x>1)

if(x>1) pprint ' '


20



if(x>1)

if(x>1) pprint ' '


20



if(x>1)

if(x>1) pprint ' '


20



if(x>1)

if(x>1) pprint ' '


20



if(x>1)

if(x>1) pprint ' '


20



if(x>1)

if(x>1) pprint ' '


20



if(x>1)

if(x>1) pprint ' '


if(x>1) pprint 'x=y+'

20

String replacement

replace

pprint 'aaaa' eval

eval 'aaaa'

21

String replacement

replace

pprint 'aaaa' eval

eval 'aaaa'

21

String replacement

replace

pprint 'aaaa' eval

eval 'aaaa'

21

Evaluation

Common fuzzers vs. Grimoire

We outperform AFL, QSYM, Angora, ... on almost all targets

3 3

3

3 7libxml2 3

Tiny C Compiler 3

Boolector 3

23

Evaluation

Grammar-based fuzzer vs. Grimoire

Comparison to a grammar-based fuzzer

Time

#BBs

Legend

— Grammar fuzzer— Grimoire

Grammar fuzzer >> Grimoire

25

Comparison to a grammar-based fuzzer

Time

#BBs

Legend

— Grammar fuzzer— Grimoire

Grammar fuzzer >> Grimoire

25

Using a grammar-based fuzzer as seed

Time

#BBs

Seed

Legend

— Grammar fuzzer— Grimoire + Seed

Grammar fuzzer < Grimoire + Seed

26

Using a grammar-based fuzzer as seed

Time

#BBs

Seed

Legend

— Grammar fuzzer— Grimoire + Seed

Grammar fuzzer < Grimoire + Seed

26

Conclusion

Take-aways

• Fuzzing structured inputs

• Common fuzzers: Small-scale mutations

• Grammar-based: Large-scale mutations

• Grimoire:

• Inference of input structure

• Large-scale mutations (extension, recursive replacement, string replacement)

• Real-world impact: 11 CVEs assigned

Thank you!

28

Take-aways




• Grimoire:




Thank you!

28

Take-aways




• Grimoire:




Thank you!

28

�@m_u00d8

� [email protected]

Documents

2emGrimoire: Synthesizing Structure while Fuzzing - Usenix ... · Grimoire:SynthesizingStructurewhileFuzzing UsenixSecurity2019,SantaClara August16,2019 TimBlazytko,CorneliusAschermann,MoritzSchlögel,AliAbbasi,