Fuzzing101 - webinar on Fuzzing Performance

WWW.CODENOMICON.COM

Fuzzing 101:Fuzzing Performance

Ari Takanen, CTO of Codenomicon

Nov 10th, 2010

WWW.CODENOMICON.COM2

Fuzzing 101:

• The webcast series for fuzzing industry

• Vendor neutral presentations on fuzzing technologies and use-cases

• Includes invited speakers from the industry

Codenomicon:

• Fuzzing research since 1996

• 2001, Spinoff from University of Oulu

• 50-100% annual growth in number of customers and revenues in fuzzing industry

About Fuzzing 101 and Codenomicon


About Ari Takanen

The Past: Researcher and Lecturer• 1998-2002• University of Oulu• OUSPG/PROTOS research group• Software Quality related lectures

The Present: Entrepreneur and Evangelist• 2001-today• CTO of Codenomicon• Evangelist: 10 conference talks every year • Author of two books:

• VoIP Security• Fuzzing


Agenda

Intro: • Fuzzing Overview• Fuzzing Performance

Demo in Theory• Multi-threaded execution• Test execution optimization• Command-line use

And if time allows: Demo in Practice for the interested

WWW.CODENOMICON.COM

Fuzzing Overview

Positive results from negative testing!


What is Testing - Three Flavors

Feature/Conformance tests validate positive requirements with use cases and test cases

Performance/Load testing repeats one of the feature tests to validate performance limits

Robustness/Fuzz testing uses misuse cases to verify software reliability with unexpected inputs


Security Vulnerability = Just A Bug


Codenomicon Labs Test Results

http://www.codenomicon.com/labs/results


Fuzz Test Effectiveness against WiFi


Some Helpful Definitions

Vulnerability – a weakness in software, a bug

Threat/Attack – exploit/worm/virus against a specific vulnerability

Protocol Modeling – Technique for explaining interface message sequences and message structures

Fuzzing – process and technique for security testing

Anomaly – abnormal or unexpected input

Failure – crash, busy-loop, memory corruption, or other indication of a bug in software


Fuzzing In Short

Fuzzing means crash-testing Also called:

• Negative testing• Robustness testing• Grammar testing

Based on sending systematically broken (rarely random) inputs to a software, in order to crash it

We will ignore random mutator fuzzers for now Two techniques of smart model-based fuzzers:

• Template-based• Specification-based


Model Based Fuzzing Techniques

Template Based Fuzzing• Quality of tests is based on the used seed and

modeling technique• Very quick to develop, but slow to run• Editing requires deep protocol know-how• Good for testing around known vulnerabilities

Specification Based Fuzzing• Full test coverage• Always repeatable• Short test cycle, more optimized tests• Easy to edit and add tests


Precision is about attack surface/protocol coverage

All interfaces/protocols tested?

All message sequences tested?

All message structures tested?

All data definitions tested?

All “tags” (values) tested?

Accuracy is about anomaly coverage

Anomaly categories? SQL? Buffer overflow?

All values: 0..65k, a..z, 0x00..0x255 ?

Combinations of anomalies?

Coverage

WWW.CODENOMICON.COM

Anomaly Coverage Selection


Traffic Capture Fuzzing


Traffic Capture Fuzzing Results

Test against samba seems to find zero-day


Key Questions in regards to Fuzzing

Test Coverage:• Q: Which interfaces and protocols are tested?• A: Understanding of threat models and attack surface

are the most critical starting point to fuzzing

• Q: How well something is tested?• A1: Specification coverage explains how deep

knowledge the fuzzer has on a tested interface, and is typically represented by protocol models or protocol templates

• A2: Anomaly coverage explains how well each protocol element is tested, and is the hardest metric to measure

WWW.CODENOMICON.COM

Fuzzing Performance

Faster, Better, More!

WWW.CODENOMICON.COM

Fuzzing Scalability

Software-based Fuzzers scale for all testing needs• The throughput depends only on the available hardware• The entire Network Under Test (NUT) can be virtualized• Software fuzzers can attack every interface, and against all

protocol layers• XML• HTTP• TLS• TCP/IP


Why Fast Speed Fuzzing?

“One of the most important aspects of fuzzing is how fast you can execute test cases. The faster you can execute test cases, the more test cases you can run and the more vulnerabilities you will find.” - Dr. Charlie Miller from Independent Security Evaluators. "

WWW.CODENOMICON.COM

Generating Load with Defensics

Full model-based message sequences

Options for monitoring, instrumentation, fuzzing, …

Amount of available CPUs and Logging Level impact performance

Some speed records (test cases per second):

• TLS: 2.400 tc/s• IPv6: 4.500 tc/s• HTTP: 16.000 tc/s• DNS: 41.000 tc/s (with capture replay)

21


Motivation for our Performance Study

Test our test generation efficiency for different type of protocol fuzzers using high-end servers

Analyze benefits (test efficiency, ROI) of software-based fuzzers to appliances


Raw Fuzzing Performance Metrics

Packets or bytes per second does not really work as each fuzz test case can consist of several packets back and forth

Test cases, or full protocol transactions, generated by each suite provides more insight

Performance is generated by having fuzzers running in parallel

These result in “test cases per second” or tc/s


Defensics Benefits

No hardware constraints

No operating system constraints

Parallel processes and threads


Performance Bottlenecks and Metrics

Network bandwidth: • Can be measured with number of full protocol

transactions, packets, and data volume CPU usage by different components of

Defensics• Test generator(s)• Test driver(s)

Memory usage of Defensics components

Most interesting metric: • New type of bugs related to performance?


Fuzz Test Can Generate 50x Load

WWW.CODENOMICON.COM

Generating Load with Dell R910

27

WWW.CODENOMICON.COM

Quick-Start to Fast-Speed Fuzzing

Demo


Test Plans Using Command-Line

With some fuzzers, you can run the tools faster on command-line

With Defensics, the GUI is sometimes even faster

E.g. with a laptop with dual-core Intel T7500 processor (with default tool settings):

• 10.000 x valid-case in loop• using GUI: 30 seconds• using command-line: 35 seconds


Running Fuzzers in Parallel

Defensics GUI has options to run several suites in parallel

• 2 x 5.000 x valid-case in loop• using GUI: 22 seconds (about 50-100% faster)


Using Multi-threaded Mode

When CPU resources are available, multi-treaded mode is often best way to speed up testing

Uses less memory

Note that all parallel execution creates more false positives

Let’s look at that and other speed optimization hints now in live demo...

WWW.CODENOMICON.COM

Conclusions

Not so fast!


Performance Value

Users can now…• Increase test coverage through parallel tests• Speed up testing• Find performance bottle-necks• Integrate Defensics also in other functional tests

Remember that Fuzzing ROI is measured in: • TCO vs. Value of Security Findings / Cost of Security

Compromise• Reduced maintenance costs, failure reproduction costs• Better customer confidence, higher retention rate of

customers• In performance: software-based tools can be significantly

cheaper (and faster) than appliances


More News from Codenomicon

Facebook:• Become fan of Codenomicon and Fuzzing

Twitter:• CodenomiconLTD

Codenomicon Website:• Newsletter every second month

Customer portal• The Backstage! Contact us for access!

34


Our Book On Fuzzing!

http://www.fuzz-test.com/book/

Takanen, DeMott and Miller: “Fuzzing for Software Security Testing and Quality Assurance”

Aimed at the general public, you do not need to be a security specialist to read this book

Purpose of the book is to teach next-gen testing approaches to:

• Software practitioners• Security engineers• Academics

http://www.fuzz-test.com/book/

WWW.CODENOMICON.COM

PROACTIVE SECURITY AND ROBUSTNESS SOLUTIONS

THANK YOU – QUESTIONS?

“Thrill to the excitement of the chase! Stalk bugs with care, methodology, and reason. Build traps for them.

....Testers!

Break that software (as you must) anddrive it to the ultimate

- but don’t enjoy the programmer’s pain.”

[from Boris Beizer]

Documents

Fuzzing101 - webinar on Fuzzing Performance