MIT Adam Kiezunpeople.csail.mit.edu/akiezun/KiezunDefenseFinal.pdf · Adam Kiezun MIT Software...

E ective Software Testing with a String-Constraint Solver

Adam KiezunMIT

Software Testing Aims To Find Errors Before Users (Or Hackers) Do

Goals of software testing• improve quality• protect from adversaries

Reported Severe Vulnerabilitiessource: US-CERT

Software Testing Aims To Find Errors Before Users (Or Hackers) Do

Goal: help find errors by improving testing tools

Goals of software testing• improve quality• protect from adversaries

Reported Severe Vulnerabilitiessource: US-CERT

Research HackersTech transfer

Concolic Testing Is An E ective Software Testing Methodology

Implementation-based: exploit knowledge of program code

Dynamic: observe running program using

combined concrete and symbolic execution

Constraint solver systematically enumerate execution paths

Key idea: improve e ectiveness, applicability of concolic testing with a string-constraint solver

Tools: DART, CUTE, CREST, SAGE, EXE, Klee, Apollo, jFuzz

E ective Software Testing With A String-Constraint Solver

Hampi: String-Constraint Solver [ISSTA’09]

Concolic Security Testing [ICSE’09]

Grammar-based Concolic Testing [PLDI’08]

Concolic Testing

Results Summary: String-Constraint Solver

Novel solver for string constraints

Supports context-free grammars, regular constraints

E ective in concolic testing, program analysis

E cient: ~7x faster than a comparable solver

Results Summary: Concolic Security Testing

Novel technique for creating SQL injection and XSS attacks on Web applications

Uses Hampi for grammar constraints to construct attack inputs

First to create damaging second-order cross-site scripting (XSS) attacks

60 attacks (23 SQL injection, 37 XSS) on 5 PHP applications, 0 false positives

Results Summary: Grammar-based Concolic Testing

Novel technique for testing programs with structured inputs

Uses Hampi for input-format grammar constraints

Improves coverage by 30-100%

3 new infinite-loop errors

Concolic Testing

void main(char[] in){

int count=0;

if (in[0] == ’b’)

count++;

if (in[1] == ’a’)

count++;

if (in[2] == ’d’)

count++;

if (count == 3)

ERROR;

Concolic Testing Combines Dynamic Symbolic Execution, Path Enumeration

int count=0;

if (in[0] == ’b’)

count++;

if (in[1] == ’a’)

count++;

if (in[2] == ’d’)

count++;

if (count == 3)

ERROR;

(in[0] ’b’) Path constraint:

int count=0;

if (in[0] == ’b’)

count++;

if (in[1] == ’a’)

count++;

if (in[2] == ’d’)

count++;

if (count == 3)

ERROR;

Path constraint: (in[0] ’b’) (in[1] ’a’)

int count=0;

if (in[0] == ’b’)

count++;

if (in[1] == ’a’)

count++;

if (in[2] == ’d’)

count++;

if (count == 3)

ERROR;

Path constraint: (in[0] ’b’) (in[1] ’a’) (in[2] ’d’)

int count=0;

if (in[0] == ’b’)

count++;

if (in[1] == ’a’)

count++;

if (in[2] == ’d’)

count++;

if (count == 3)

ERROR;

Path constraint: (in[0] ’b’) (in[1] ’a’) (in[2]=’d’) xyd

int count=0;

if (in[0] == ’b’)

count++;

if (in[1] == ’a’)

count++;

if (in[2] == ’d’)

count++;

if (count == 3)

ERROR;

Path constraint: (in[0] ’b’) (in[1]=’a’) xaz

int count=0;

if (in[0] == ’b’)

count++;

if (in[1] == ’a’)

count++;

if (in[2] == ’d’)

count++;

if (count == 3)

ERROR;

Path constraint: (in[0]=’b’) byz

int count=0;

if (in[0] == ’b’)

count++;

if (in[1] == ’a’)

count++;

if (in[2] == ’d’)

count++;

if (count == 3)

ERROR;

Seed input

Concolic Testing Systematically Enumerates All Paths In The Program

int count=0;

if (in[0] == ’b’)

count++;

if (in[1] == ’a’)

count++;

if (in[2] == ’d’)

count++;

if (count == 3)

ERROR;

xyz xyd xaz byz

Generated inputs(each covers a new path)

int count=0;

if (in[0] == ’b’)

count++;

if (in[1] == ’a’)

count++;

if (in[2] == ’d’)

count++;

if (count == 3)

ERROR;

xyz xyd xaz byz xad byd baz

int count=0;

if (in[0] == ’b’)

count++;

if (in[1] == ’a’)

count++;

if (in[2] == ’d’)

count++;

if (count == 3)

ERROR;

xyz xyd xaz byz xad byd baz bad

Concolic testing creates inputs for all program paths.

Concolic Testing

Many Program Analyses Reduce To Constraint Generation And Solving

Benefits+ declarative formulation+ better modularity + e ciency improvements

Downsides- limited by solver’s theory

Hampi: constraint solver for a theory of strings

String-Constraint Solver Finds Assignments For String Variables

Finite alphabet (e.g., ASCII characters)

String variables over *

String constraints – language membership:

assert v L

String operations

concat(“foo”, v, “bar”)

Hampi Uses Length Bounding To Support Context-Free Constraints

bounded regularbound(any language)

Hampi Uses Length Bounding To Support Context-Free Constraints

Key Hampi idea: bound length of strings for high expressiveness, e ciency

bounded regularbound(any language)

Hampi Can Solve Context-Free and Regular Constraints

“Find a 4-character string v, such that:• (v) has balanced parentheses, and • (v) contains substring ()()”

var v:4;

cfg E := "()" | E E | "(" E ")";

reg Ebounded := bound(E, 6);

val q := concat( ”(" , v , “)" );

assert q in Ebounded;

assert q contains “()()";

var v:4;

cfg E := "()" | E E | "(" E ")";

val q := concat( ”(" , v , “)" );

var v:4;

cfg E := "()" | E E | "(" E ")";

val q := concat( ”(" , v , “)" );

var v:4;

cfg E := "()" | E E | "(" E ")";

val q := concat( ”(" , v , “)" );

var v:4;

cfg E := "()" | E E | "(" E ")";

val q := concat( ”(" , v , “)" );

Hampi finds satisfying assignment v = )()(

Hampi Supports Rich String Constraints

Wasserm

Hooijm

allero

context-free grammars

regular expressions

string concatenation

stand-alone tool

unbounded length

Hampi Encodes String Constraints In Bit-Vector Logic

string constraints

core string constraints

Normalizer

Encoder

Decoderbit-vector solution

bit-vector constraints

string solution

Bit-vector Solver

Hampi Normalizer Converts String Constraints To Core Form

string constraints

Normalizer

Encoder

string solution

Bit-vector Solver

Core string constraint have only regular expressions

Expand grammars to regexps• expand nonterminals• eliminate inconsistencies• enumerate choices exhaustively

bound(E, 6)

cfg E := "(" E ")” | E E | "()”;

string constraints

Normalizer

Encoder

string solution

Bit-vector Solver

Expand grammars to regexps expand nonterminals

• eliminate inconsistencies• enumerate choices exhaustively

cfg E := "(" E ")” | E E | "()”;

string constraints

Normalizer

Encoder

string solution

Bit-vector Solver

Expand grammars to regexps• expand nonterminals

eliminate inconsistencies• enumerate choices exhaustively

( E ) + E E + ()

cfg E := "(" E ")” | E E | "()”;

string constraints

Normalizer

Encoder

string solution

Bit-vector Solver

( E ) + E E +

cfg E := "(" E ")” | E E | "()”;

string constraints

Normalizer

Encoder

string solution

Bit-vector Solver

Expand grammars to regexps• expand nonterminals• eliminate inconsistencies

enumerate choices exhaustively

cfg E := "(" E ")” | E E | "()”;

( E ) + E E

string constraints

Normalizer

Encoder

string solution

Bit-vector Solver

cfg E := "(" E ")” | E E | "()”;

( E ) + E E + E E + E E + E E + E E + E E + E E

string constraints

Normalizer

Encoder

string solution

Bit-vector Solver

cfg E := "(" E ")” | E E | "()”;

( E ) + E E + E E + E E + E E + E E + E E + E E

string constraints

Normalizer

Encoder

string solution

Bit-vector Solver

cfg E := "(" E ")” | E E | "()”;

( E ) + E E + E E

string constraints

Normalizer

Encoder

string solution

Bit-vector Solver

([()() + (())]) +

()[()() + (())] +

[()() + (())]() bound(E, 6)

cfg E := "(" E ")” | E E | "()”;

Hampi Normalizer Uses Compact Representations Of Expressions

()[()() + (())] +

[()() + (())]() +

([()() + (())])

() shared graph nodes forcommon subexpressions

Bit vector B (length 6 bits)

(B[0:4] = B[2:4]) (B[1:3] = 101)

o set:length

Bit Vectors Are Ordered, Fixed-Size, Sets Of Bits

0 1 0 1 0 1

Bit vector B (length 6 bits)

Bit-vector solver finds the solution B = 010101

(B[0:4] = B[2:4]) (B[1:3] = 101)

o set:length

Bit Vectors Are Ordered, Fixed-Size, Sets Of Bits

Hampi Encodes Core Constraints As Bit-Vector Constraints

string constraints

Normalizer

Encoder

string solution

Bit-vector Solver

Map alphabet to bit-vector constants: (

Compute size of bit-vector B: (1+4+1) * 1 bit = 6 bits

( v ) ()[()() + (())] + [()() + (())]() + ([()() + (())])

Hampi Encodes Regular Expressions Recursively

string constraints

Normalizer

Encoder

string solution

Bit-vector Solver

( v ) ()[()() + (())] + [()() + (())]() + ([()() + (())])

Formula 1 Formula 2 Formula 3

Encode regular expressions recursively• union + disjunction • concatenation conjunction • Kleene star * conjunction • constant bit-vector constant

Hampi Encoder Exploits Shift-Symmetry In Constraints

string constraints

Normalizer

Encoder

string solution

Bit-vector Solver

( v ) ()[()() + (())] + [()() + (())]() + ([()() + (())])

B[0:2] = 01 B[5:2] = 01

Shift-symmetric constraints

Shift-symmetric constraints = identical modulo o set in bit vector

Hampi reuses encoding templates for symmetric constraints

Hampi Encoder Exploits Shift-Symmetry In Constraints

string constraints

Normalizer

Encoder

string solution

Bit-vector Solver

( v ) ()[()() + (())] + [()() + (())]() + ([()() + (())])

Shift-symmetric constraints

Shift-symmetric constraints = identical modulo o set in bit vector

Hampi reuses encoding templates for symmetric constraints

Hampi Uses Bit-Vector Solver And Decodes Solution

string constraints

Normalizer

Encoder

string solution

Bit-vector Solver

B = 010101

B = ()()()

v = )()(

Result 1: Hampi Is E ective In Static SQL Injection Analysis

1367 string constraints from [Wassermann PLDI’07]

Hampi solved 99.7% of constraints in < 1 sec per constraint

All solvable constraints had short solutions N 4

Hampi scales to large grammars

Result 2: Hampi Is Faster Than The CFGAnalyzer Solver

CFGAnalyzer encodes bounded grammar problems in SAT [Axelsson et al ICALP’08]

0 10 20 30 40 50 0

CFGAnalyzer

string size (characters)

For size 50, Hampi is 6.8x faster on average (up to 3000x faster)

Concolic Testing

Ardilla Mutates Generated Inputs To Construct Attacks

SQL Injection Attacks Modify Structure Of Database Queries

SELECT msg FROM messages WHERE topicid=‘ ’

Innocuous input: v 1

concat(SELECT msg FROM messages WHERE topicid=‘ v ’)

Symbolic expression for SQL query

concat(SELECT msg FROM messages WHERE topicid=‘ v ’)

Attack input: v

Attacker gets access to all messages

Symbolic expression for SQL query

var v : 12;

cfg SqlSmall := "SELECT ” [a-z]+ " FROM ” [a-z]+ " WHERE " Cond;

cfg Cond := Val "=" Val | Cond " OR " Cond;

cfg Val := [a-z]+ | "'” [a-z0-9]* "'" | [0-9]+;

reg SqlSmallBounded := bound(SqlSmall, 53);

val q := concat("SELECT msg FROM messages WHERE topicid='", v, "'");

assert q in SqlSmallBounded;

assert q contains "OR ‘0'=‘0'";

Example: Hampi Constraints That Create SQL Injection Attacks

var v : 12;

cfg SqlSmall := "SELECT ” [a-z]+ " FROM ” [a-z]+ " WHERE " Cond;

cfg Cond := Val "=" Val | Cond " OR " Cond;

cfg Val := [a-z]+ | "'” [a-z0-9]* "'" | [0-9]+;

reg SqlSmallBounded := bound(SqlSmall, 53);

val q := concat("SELECT msg FROM messages WHERE topicid='", v, "'");

assert q in SqlSmallBounded;

assert q contains "OR ‘0'=‘0'";

Hampi finds an attack input: v 1’ OR ‘0’=‘0

Example: Hampi Constraints That Create SQL Injection Attacks

Result: Ardilla Finds New Attacks

60 attacks on 5 PHP applications

23 SQL injection

29 XSS first order

8 XSS second order

0 false positives

216 Hampi constraints solved

• 46% of constraints in < 1 second per constraint

• 100% of constraints in < 10 seconds per constraint

Concolic Testing

Sometimes Concolic Testing Is Not Much Better Than Random Fuzzing

Random Fuzz Testing Concolic Testing

Program under test: JavaScript interpreter

Random Fuzz Testing Concolic Testing

Concolic TestingRandom Fuzz Testing

17.6%inputs reach

16.5%inputs reach

Concolic TestingRandom Fuzz Testing

Most Generated Inputs Get Rejected Quickly

Key idea: generate only valid inputs

Input-Format Grammar Guides Creation Of E ective Inputs

solve(PC’) new input

Input-Format Grammar Guides Creation Of E ective Inputs

solve(PC’ Grammar) new valid input

Hampi string solver

String-Constraint Solver Helps Create Valid Inputs

Seed input (for JavaScript interpreter): function f(){ var v = 3; }

Constraints on tokens

(created during execution) token0 = function

token1 = id

token2 = (

token3 = )

token4 = {

token5 = var

token1 = id

token2 = (

token3 = )

token4 = {

token5 var ffunction f(){ try v = 3; }

token1 = id

token2 = (

token3 = )

token4 = {

token5 var ffunction f(){ try v = 3; }

function f(){ try { } catch ( id ) { } finally { }; }

String-Constraint Solver Helps Avoid Dead-End Inputs

token1 = id

token2 = (

token3 = )

token4 {

function f() var var v = 3; }

String-Constraint Solver Helps Avoid Dead-End Inputs

token1 = id

token2 = (

token3 = )

token4 {

function f() var var v = 3; }

Results: Grammar-Based Concolic Testing Improves Deep Reachability

Up to 20x deep reachability improvement: more generated inputs reach beyond the parser

Results: Grammar-Based Concolic Testing Improves Coverage

Up to 2x coverage improvement

Results: Grammar-Based Concolic Testing Improves Coverage

Up to 2x coverage improvement

3 infinite-

loop bugs

and finds

new bugs

Summary: E ective Software Testing With A String-Constraint Solver

Hampi String-Constraint Solver

expressive: supports context-free grammars

e cient: solver real-world constraint quickly

Concolic Security Testing

creates attacks on Web applications by input generation

and mutation with Hampi string-constraint solver

Grammar-Based Concolic Testing

e ectively tests programs with structured inputs by using Hampi string-constraint solver and input grammars

MIT Adam Kiezunpeople.csail.mit.edu/akiezun/KiezunDefenseFinal.pdf · Adam Kiezun MIT Software...

Documents

1 Generating Strong Keys from Biometric Data Yevgeniy Dodis, New York U Leonid Reyzin, Boston U Adam Smith, MIT

MIT January Operational Internship Experience 2011 · 2019. 8. 30. · Internship Experience 2011 Danielle DeLatte Adam Furhmann Manal Habib Cecily Joujon-Roche Nnaemeka Opara Sabrina

1 STAR Double Longitudinal Spin Asymmetries of Inclusive Charged Pion Production in Polarized p+p Collisions at 200 GeV Adam Kocoloski, MIT For the STAR

ember 2013 - opel-infos.de · Opel ADAM 2 Modell-/Motorenübersicht ADAM ADAM ADAM JAM ADAM GLAM ADAM SLAM ADAM White Link ADAM Black Link Motor CO 2-Emission in g/km kombiniert Getriebe

SCIENTIFIC ABSTRACT ADAM, G. - ADAM,M. · Title: SCIENTIFIC ABSTRACT ADAM, G. - ADAM,M. Subject: SCIENTIFIC ABSTRACT ADAM, G. - ADAM,M. Keywords

Base-driven leveling in Yiddish verb paradigmsweb.mit.edu/albright/www/papers/Albright-LevelingInYiddishVerbs.pdf · Base-driven leveling in Yiddish verb paradigms Adam Albright MIT

Michael Bernstein, Adam Marcus, David Karger, Rob Miller MIT CSAIL MIT HUMAN - COMPUTER INTERACTION Enhancing Directed Content Sharing on the Web

MIT 101 - ISO | MIT

ADAM-8000 Series - Construmática.com...ADAM-5000 15 ADAM-6000 16 ADAM-8000 17 BAS PLC Control The ADAM-8000 series is a typical PLC control system. You can conﬁgure the ADAM- 8000

Adam Blum, adam@rhomobile

Automatic Creation of SQL Injection and Cross-Site Scripting Attacks 2nd-order XSS attacks 1st-order XSS attacks SQLI attacks Adam Kiezun, Philip J. Guo,

Profile of Adam 4 Adam Users

Finding Bugs in Dynamic Web Applications Shay Artzi, Adam Kiezun, Julian Dolby, Frank Tip, Danny Dig, Amit Paradkar, Michael D. Earnst Proceeding: ISSTA

Traveling Salesman Problems Motivated by Robot Navigation Maria Minkoff MIT With Avrim Blum, Shuchi Chawla, David Karger, Terran Lane, Adam Meyerson

HAMPI A Solver for String Constraints Vijay Ganesh MIT (With Adam Kiezun, Philip Guo, Pieter Hooimeijer and Mike Ernst)

Proficiency-Based Grading Adam Williams adam@progradebook.com

Exploring generative atomic models in cryo-EM reconstruction...Exploring generative atomic models in cryo-EM reconstruction Ellen D. Zhong MIT zhonge@mit.edu Adam Lerer Facebook AI

ADAM SMITH: THE WEALTH OF NATIONS - MIT · PDF file(6) Adam Smith, The Wealth of Nations. a. Text. Public domain, excerpted by A. C. Kibel ADAM SMITH: THE WEALTH OF NATIONS BOOK ONE

ESTHER: ADAM: ESTHER: ADAM: ESTHER: ADAM: ESTHER

Refactoring for Parameterizing Java Classesmernst/pubs/parameteri...Refactoring for Parameterizing Java Classes Adam Kiezun (MIT), Michael D. Ernst (MIT), Frank Tip (IBM), Robert M