Bug Injection in SATE VI
Aurelien DelaitreLead, SATE VI Classic TrackPrometheus Computing LLC
SATE VI Workshop - September 19, 2019 - MITRE, McLean VA
https://samate.nist.gov/SATE.html
Why Bug Injection?
Relevance
GroundTruth
StatisticalSignificance
2
Why Bug Injection?
Relevance
GroundTruth
StatisticalSignificance
ProductionSoftware
Common Vulnerabilities and Exposures (CVE)
SyntheticTest Suites
3
Why Bug Injection?
Relevance
GroundTruth
StatisticalSignificance
BugInjection
4
Ways to “Get” Bugs
● Bug Injectors● Manual & Semi-Automated Injection● Specifically Developed Test Suites● Existing Bugs
○ Discovered○ Undiscovered
5
Bug Types in SATE VI
C: Undefined Behavior
● Pointers● Buffers● Initialization
Java: Code Injection
● Cross-Site Scripting (XSS)● SQL Injection
▶ High-Impact▶ Easy to Prove
6
Proof of Vulnerability (PoV)
Why?
● Proves Bug Matters● Retrieve Bug Trace
How?
● Fuzzing● Bug Tracker● Manual
7
Bug Traces
● Based on PoVs○ C: GDB / Valgrind / ASAN○ Java: Flow
● Manual Analysis○ Doc Review○ Code Review
8
What Went Wrong?
9
Cheap but Hard Bugs
10
packet-arp.c
▶ Almost Never Found by Tools
Asymmetrical Bug/Fix Pairs
11
SimplePageBean.java
Buggy
Fixed
Buggy Bugs
▶ Implementation-dependent▶ Unknown Sink
▶ Tainted Data Questionable▶ Unintended Bug Type
fts3_write.c
global.c
12
fts3_write.c
Buggy Fixes
▶ Tainted Data Questionable▶ Condition Always False
global.c
13
pragma.c
Buggy Bugs
▶ Tainted Data Questionable▶ Condition Always True
global.c
14
pragma.c
Buggy Fixes
▶ Tainted Data Questionable▶ Condition Always False
global.c
15
fts3_tokenize_vtab.c
Buggy Fixes
▶ Tainted Data Questionable▶ Condition Always False Due to Programming Error
global.c
16
Sink Separationdate.c insert.c
17
Shadowing
18
Shadowing
19
Shadowing
20
Take Away
21
AutomatedBug Injection Curation Test Suites
StrongerBetterFaster