Embed Size (px)
DESCRIPTION
Hybrid Concolic Testing. Rupak Majumdar Koushik Sen UC Los Angeles UC Berkeley. Automated Test Generation. Studied since 70’s King 76, Myers 79 30 years have passed, and yet no effective solution What Happened???. Automated Test Generation. Studied since 70’s - PowerPoint PPT Presentation
Citation preview
Hybrid Concolic Testing
Rupak Majumdar Koushik SenUC Los Angeles UC Berkeley
Automated Test Generation Studied since 70’s
King 76, Myers 79 30 years have passed, and yet no
effective solution What Happened???
Automated Test Generation Studied since 70’s
King 76, Myers 79 30 years have passed, and yet no
effective solution What Happened???
Program-analysis techniques were expensive
Automated theorem proving and constraint solving techniques were not efficient
Automated Test Generation Studied since 70’s
King 76, Myers 79 30 years have passed, and yet no
effective solution What Happened???
Program-analysis techniques were expensive
Automated theorem proving and constraint solving techniques were not efficient
In the recent years we have seen remarkable progress in static program-analysis and constraint solving SLAM, BLAST, ESP, Bandera, Saturn, MAGIC
Automated Test Generation Studied since 70’s
King 76, Myers 79 30 years have passed, and yet no
effective solution What Happened???
Program-analysis techniques were expensive
Automated theorem proving and constraint solving techniques were not efficient
In the recent years we have seen remarkable progress in static program-analysis and constraint solving SLAM, BLAST, ESP, Bandera, Saturn, MAGIC
Question: Can we combine program analysis with
classical testing techniques to Scale Automated Test
Generation?
Our Approach
Concolic Testing:
1.Combines Dynamic and Static Program Analysis
2.Exhaustive
3.Fails to scale
Random Testing:
1.Fast
2.Non-exhaustive
3.Redundant Executions and poor coverage
+
=Hybrid Concolic Testing
Goals of Test Generation (Simplified) Generate test inputs Execute program on generated test
inputs Catch assertion violations Problem: how to ensure that all reachable
statements are executed Solution:
Explore all feasible execution paths
Execution of Programs All Possible
Execution Paths Binary tree
Computation tree Internal node !
conditional statement execution
Edge ! execution of a sequence of non-conditional statements
Each path in the tree represents an equivalence class of inputs
F T
F F
F
F
T
T
T
T
T
T
Conditional Statements
Non-Conditional Statements
Fuzz (Random) Testing Random testing
Random Testing [Bird and Munoz 83]
Fuzz testing Windows NT [Forrester and Miller 00]
QuickCheck [Claessen & Hughes 01]
JCrasher [Csallner and Smaragdakis 04]
RUTE-J [Andrews et al. 06] Randoop [Pacheco et al.
07]
Very low probability of reaching an error
Problematic for complex data structures
Fuzz (Random) Testing Random testing
Random Testing [Bird and Munoz 83]
Fuzz testing Windows NT [Forrester and Miller 00]
QuickCheck [Claessen & Hughes 01]
JCrasher [Csallner and Smaragdakis 04]
RUTE-J [Andrews et al. 06] Randoop [Pacheco et al.
07]
Very low probability of reaching an error
Problematic for complex data structures
Example ( ) {
s = readString();
if (s[0]==‘I’ && s[1]==‘C’ &&
s[2]==‘S’ && s[3]==‘E’ &&
s[4]==‘2’ && s[5]==‘0’ &&
s[6]==‘0’ && s[7]==‘7’) {
printf(“Am I here?”);
}
}
Example ( ) {
s = readString();
if (s[0]==‘I’ && s[1]==‘C’ &&
s[2]==‘S’ && s[3]==‘E’ &&
s[4]==‘2’ && s[5]==‘0’ &&
s[6]==‘0’ && s[7]==‘7’) {
printf(“Am I here?”);
}
}
Input domain = {‘0’, ‘2’, ‘7’, ‘C’, ‘E’, ‘I’, ‘S’}
Probability of reaching printf = 7-8 » 10-7
Fuzz (Random) Testing Random testing
Random Testing [Bird and Munoz 83]
Fuzz testing Windows NT [Forrester and Miller 00]
QuickCheck [Claessen & Hughes 01]
JCrasher [Csallner and Smaragdakis 04]
RUTE-J [Andrews et al. 06] Randoop [Pacheco et al.
07]
Very low probability of reaching an error
Problematic for complex data structures
Example ( ) {
s = readString();
if (s[0]==‘I’ && s[1]==‘C’ &&
s[2]==‘S’ && s[3]==‘E’ &&
s[4]==‘2’ && s[5]==‘0’ &&
s[6]==‘0’ && s[7]==‘7’) {
printf(“Am I here?”);
}
}
Example ( ) {
s = readString();
if (s[0]==‘I’ && s[1]==‘C’ &&
s[2]==‘S’ && s[3]==‘E’ &&
s[4]==‘2’ && s[5]==‘0’ &&
s[6]==‘0’ && s[7]==‘7’) {
printf(“Am I here?”);
}
}
Input domain = {‘0’, ‘2’, ‘7’, ‘C’, ‘E’, ‘I’, ‘S’}
Probability of reaching printf = 7-8 » 10-7 Fast and
Inexpensive
Concolic Testing Combine concrete testing (concrete
execution) and symbolic testing (symbolic execution) [PLDI’05, FSE’05, FASE’06, CAV’06,
HVC’06]
Concrete + Symbolic = Concolic
Example
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Example
ERROR
2*y == x
x > y+10
Y
Y
N
N
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concolic Testing Approach
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concrete Execution
Symbolic Execution
concrete state
symbolic state
path condition
x = 22, y = 7 x = x0, y = y0
Concolic Testing Approach
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concrete Execution
Symbolic Execution
concrete state
symbolic state
path condition
x = 22, y = 7, z = 14
x = x0, y = y0, z = 2*y0
Concolic Testing Approach
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concrete Execution
Symbolic Execution
concrete state
symbolic state
path condition
x = 22, y = 7, z = 14
x = x0, y = y0, z = 2*y0
2*y0 != x0
Concolic Testing Approach
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concrete Execution
Symbolic Execution
concrete state
symbolic state
path condition
2*y0 != x0
Solve: 2*y0 == x0
Solution: x0 = 2, y0 = 1
x = 22, y = 7, z = 14
x = x0, y = y0, z = 2*y0
Concolic Testing Approach
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concrete Execution
Symbolic Execution
concrete state
symbolic state
path condition
x = 2, y = 1 x = x0, y = y0
Concolic Testing Approach
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concrete Execution
Symbolic Execution
concrete state
symbolic state
path condition
x = 2, y = 1, z = 2
x = x0, y = y0, z = 2*y0
Concolic Testing Approach
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concrete Execution
Symbolic Execution
concrete state
symbolic state
path condition
x = 2, y = 1, z = 2
x = x0, y = y0, z = 2*y0
2*y0 == x0
Concolic Testing Approach
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concrete Execution
Symbolic Execution
concrete state
symbolic state
path condition
x = 2, y = 1, z = 2
x = x0, y = y0, z = 2*y0
2*y0 == x0
x0 · y0+10
Concolic Testing Approach
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concrete Execution
Symbolic Execution
concrete state
symbolic state
path condition
x = 2, y = 1, z = 2
x = x0, y = y0, z = 2*y0
Solve: (2*y0 == x0) Æ (x0 > y0 + 10)
Solution: x0 = 30, y0 = 15
2*y0 == x0
x0 · y0+10
Concolic Testing Approach
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concrete Execution
Symbolic Execution
concrete state
symbolic state
path condition
x = 30, y = 15 x = x0, y = y0
Concolic Testing Approach
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concrete Execution
Symbolic Execution
concrete state
symbolic state
path condition
x = 30, y = 15 x = x0, y = y0
2*y0 == x0
x0 > y0+10
Program Error
Explicit Path (not State) Model Checking Traverse all
execution paths one by one to detect errors assertion violations program crash uncaught exceptions
combine with valgrind to discover memory errors
F T
F F
F
F
T
T
T
T
T
T
Explicit Path (not State) Model Checking Traverse all
execution paths one by one to detect errors assertion violations program crash uncaught exceptions
combine with valgrind to discover memory errors
F T
F F
F
F
T
T
T
T
T
T
Explicit Path (not State) Model Checking Traverse all
execution paths one by one to detect errors assertion violations program crash uncaught exceptions
combine with valgrind to discover memory errors
F T
F F
F
F
T
T
T
T
T
T
Explicit Path (not State) Model Checking Traverse all
execution paths one by one to detect errors assertion violations program crash uncaught exceptions
combine with valgrind to discover memory errors
F T
F F
F
F
T
T
T
T
T
T
Explicit Path (not State) Model Checking Traverse all
execution paths one by one to detect errors assertion violations program crash uncaught exceptions
combine with valgrind to discover memory errors
F T
F F
F
F
T
T
T
T
T
T
Explicit Path (not State) Model Checking Traverse all
execution paths one by one to detect errors assertion violations program crash uncaught exceptions
combine with valgrind to discover memory errors
F T
F F
F
F
T
T
T
T
T
T
Limitations Path Space of a Large Program is Huge
Path Explosion Problem
Entire Computation Tree
Limitations Path Space of a Large Program is Huge
Path Explosion Problem
Explored by Concolic Testing
Entire Computation Tree
Limitations: A Comparative View
Concolic: Broad, shallow
Random: Narrow, deep
Limitations: ExampleExample ( ) {1: state = 0;2: while(1) {3: s = input();4: c = input();5: if(c==‘:’ && state==0) state=1;6: else if(c==‘\n’ && state==1) state=2;7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) {
COVER_ME:; } }}
Example ( ) {1: state = 0;2: while(1) {3: s = input();4: c = input();5: if(c==‘:’ && state==0) state=1;6: else if(c==‘\n’ && state==1) state=2;7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) {
COVER_ME:; } }}
Similar code in
•Text editors (vi)
•Parsers (lexer)
•Event-driven programs (GUI)
•Want to hit COVER_ME
•input() denotes external input
•Can be hit on an input sequence
s = “ICSE”
c : ‘:’ ‘\n’
Limitations: ExampleExample ( ) {1: state = 0;2: while(1) {3: s = input();4: c = input();5: if(c==‘:’ && state==0) state=1;6: else if(c==‘\n’ && state==1) state=2;7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) {
COVER_ME:; } }}
Example ( ) {1: state = 0;2: while(1) {3: s = input();4: c = input();5: if(c==‘:’ && state==0) state=1;6: else if(c==‘\n’ && state==1) state=2;7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) {
COVER_ME:; } }}
•Pure random testing can get to
state = 2
But difficult to get ‘ICSE’ as a
Sequence
Probability 1/(28)6 » 3X10-15
•Conversely, concolic testing
can generate ‘ICSE’ but explores
many paths to get to state = 2
Hybrid Concolic Testing Interleave Random Testing and Concolic Testing
to increase coverage
Motivated by similar idea used in VLSI design validation:
Ganai et al. 1999, Ho et al. 2000
Hybrid Concolic Testing Interleave Random Testing and Concolic Testing
to increase coverage
while (not required coverage) {
while (not saturation)
perform random testing;
Checkpoint;
while (not increase in coverage)
perform concolic testing;
Restore;
}
Hybrid Concolic Testing Interleave Random Testing and Concolic Testing
to increase coverage
while (not required coverage) {
while (not saturation)
perform random testing;
Checkpoint;
while (not increase in coverage)
perform concolic testing;
Restore;
}
Deep, broad search
Hybrid Search
Hybrid Concolic Testing Random Phase
‘$’, ‘&’, ‘-’, ‘6’, ‘:’, ‘%’, ‘^’, ‘\n’, ‘x’, ‘~’ … Saturates after many
(~10000) iterations In less than 1 second COVER_ME is not
reached
Example ( ) {1: state = 0;2: while(1) {3: s = input();4: c = input();5: if(c==‘:’ && state==0) state=1;6: else if(c==‘\n’ && state==1) state=2;7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) {
COVER_ME:; } }}
Example ( ) {1: state = 0;2: while(1) {3: s = input();4: c = input();5: if(c==‘:’ && state==0) state=1;6: else if(c==‘\n’ && state==1) state=2;7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) {
COVER_ME:; } }}
Hybrid Concolic Testing Random Phase
‘$’, ‘&’, ‘-’, ‘6’, ‘:’, ‘%’, ‘^’, ‘\n’, ‘x’, ‘~’ … Saturates after many
(~10000) iterations In less than 1 second COVER_ME is not
reached
Concolic Phase s[0]=‘I’, s[1]=‘C’,
s[2]=‘S’, s[3]=‘E’ Reaches COVER_ME
Example ( ) {1: state = 0;2: while(1) {3: s = input();4: c = input();5: if(c==‘:’ && state==0) state=1;6: else if(c==‘\n’ && state==1) state=2;7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) {
COVER_ME:; } }}
Example ( ) {1: state = 0;2: while(1) {3: s = input();4: c = input();5: if(c==‘:’ && state==0) state=1;6: else if(c==‘\n’ && state==1) state=2;7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) {
COVER_ME:; } }}
Hybrid Concolic Testing 4x more coverage than random testing 2x more coverage than concolic testing
Results
Results: Red Black Treetest_driver()
RedBlackTree rb = new RedBlackTree();while(1) {
choice = input();data = input();switch(choice) {
case 1: rb.add(data); break;case 2: rb.remove(data); break;case 3: rb.find(data); break;default: rb.add_if_not_member(data); break;
}}
}
Results
Summary
Concolic Testing
Random Testing
Summary
Concolic Testing
Random Testing
Hybrid Concolic Testing
Thank You!
