DART Directed Automated Random Testing Patrice Godefroid, Nils Klarlund, and Koushik Sen Syed Nabeel

DARTDirected Automated Random Testing

Patrice Godefroid, Nils Klarlund,and Koushik Sen

Syed NabeelSyed Nabeel

MotivationMotivation

►Software Testing emerges as one of the most important aspects in Software Engineering

►Unit Testing ►Limitations in Unit Testing►DART a novel approach to counter

limitations in Unit Testing

Unit TestingUnit Testing

Testing small portions of programs, such as methods, groups of methods, or classes.

► a sequence of calls to the units (for e.g. methods)

► BEFORE the calls some code to generate parameters for the methods

► AFTER the calls some code checking whether the method has performed correctly.

Goals of Unit TestingGoals of Unit Testing

►Detect Errors in the components logic►Check ALL corner cases►Provide 100% code coverage

Question Question

What are some limitations of Unit Testing?

Limitations of Unit TestingLimitations of Unit Testing

►Requires test driver and harness code Requires test driver and harness code which is representative of ALL external which is representative of ALL external environmentenvironment

►Expensive and hard to perform Expensive and hard to perform manually (rarely done properly)manually (rarely done properly)

DART To The RescueDART To The Rescue

Automates unit testing removing need for Automates unit testing removing need for writing test driverswriting test drivers

► Automated extraction of the program Automated extraction of the program interface by static code parsinginterface by static code parsing

► Automatic generation of test driver for this Automatic generation of test driver for this interface that performs random testinginterface that performs random testing

► Dynamic Analysis of the program behavior Dynamic Analysis of the program behavior to generate specific test inputs for the to generate specific test inputs for the direction of program execution through direction of program execution through alternate program pathsalternate program paths

Execution Model For DARTExecution Model For DART

► Concrete Execution: based on a memory Concrete Execution: based on a memory model M that maps addresses to valuesmodel M that maps addresses to values

► Symbolic Execution: based on symbolic Symbolic Execution: based on symbolic memory S that maps addresses to memory S that maps addresses to expressionsexpressions

Directed Testing by Constraint Directed Testing by Constraint BuildingBuilding ► Progressively constraints are built as Progressively constraints are built as

branches are encounteredbranches are encountered

► Linear constraints are solved to explore Linear constraints are solved to explore various execution pathsvarious execution paths

► In case of getting stuck or non linear In case of getting stuck or non linear constraints symbolic execution falls back to constraints symbolic execution falls back to concrete executionconcrete execution

Consider An Example ….Consider An Example ….int double(int x) { int double(int x) {

return 2 * x; return 2 * x;

}}

void void test_metest_me(int x, int y) {(int x, int y) {

int z = double(x);int z = double(x);

if(z==y){if(z==y){

if(x != y+10){if(x != y+10){

printf(“printf(“I am fine hereI am fine here”);”);

} else {} else {

printf(“printf(“I should not reach I should not reach herehere”);”);

abort();abort();

}}

}}

Random TestingRandom Testingint double(int x) { int double(int x) {


}}



if(z==y){if(z==y){

if(x != y+10){if(x != y+10){


} else {} else {


abort();abort();

}}

}}

main(){main(){int tmp1 = randomInt();int tmp1 = randomInt();int tmp2 = randomInt();int tmp2 = randomInt();test_me(tmp1,tmp2);test_me(tmp1,tmp2);

}}

Random TestingRandom Testingint double(int x) { int double(int x) {


}}



if(z==y){if(z==y){

if(x != y+10){if(x != y+10){


} else {} else {


abort();abort();

}}

}}

main(){main(){int tmp1 = randomInt();int tmp1 = randomInt();int tmp2 = randomInt();int tmp2 = randomInt();test_me(tmp1,tmp2);test_me(tmp1,tmp2);

}}

Evident Problem: Probability of Reaching Abort is very lowEvident Problem: Probability of Reaching Abort is very low

DART ApproachDART Approachmain(){main(){

int t1 = randomInt();int t1 = randomInt();


test_me(t1,t2);test_me(t1,t2);

}}

int double(int x) { int double(int x) {


}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {

printf(“printf(“I should not reach hereI should not reach here”);”);

abort();abort();

}}

}}Full example taken from Presentation by Koushik

at http://osl.cs.uiuc.edu/~ksen/slides/dart-fm.ppt

DART ApproachDART Approach Concrete Execution

concrete state

symbolic state constraintsmain(){main(){




}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

t1=36 t1=m


concrete state

symbolic state constraints

t1=36, t2=-7 t1=m, t2=n

main(){main(){




}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

t1=36, t2=-7 t1=m, t2=n


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

x=36, y=-7 x=m, y=n


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

x=36, y=-7, z=72 x=m, y=n, z=2m


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

x=36, y=-7, z=72 x=m, y=n, z=2m 2m != n


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

2m != n

x=36, y=-7, z=72 x=m, y=n, z=2m


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

solve: 2m = nm=1, n=2


2m != n

x=36, y=-7, z=72 x=m, y=n, z=2m

Question Question

Which portion of code identifies a new constraint ?

The Example Again The Example Again main(){main(){




}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

The Example AgainThe Example Againmain(){main(){




}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

t1=1 t1=m


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

t1=1, t2=2 t1=m, t2=n


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

t1=1, t2=2 t1=m, t2=n


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

x=1, y=2 x=m, y=n


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

x=1, y=2 x=m, y=n


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

x=1, y=2 x=m, y=n


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

x=1, y=2 x=m, y=n2m = n

m != n+10


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

2m = n

m != n+10x=1, y=2, z=2 x=m, y=n, z=2m


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

2m = n

m != n+10

x=1, y=2, z=2 x=m, y=n, z=2m


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution


solve: 2m = n and m=n+10

m= -10, n= -20

2m = n

x=36, y=-7, z=72 x=m, y=n, z=2m

m != n+10


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

t1=-10 t1=m


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

t1=-10, t2=-20 t1=m, t2=n


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

t1=-10, t2=-20 t1=m, t2=n


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

x=-10, y=-20 x=m, y=n


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

x=-10, y=-20, z=-20 x=m, y=n, z=2m


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

x=-10, y=-20, z=-20 x=m, y=n, z=2m 2m = n


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

x=-10, y=-20, z=-20 x=m, y=n, z=2m

2m = n

m = n+10


concrete state





}}



}}



if(z==y){if(z==y){

if(x != y+10) { if(x != y+10) {


} }

else {else {


abort();abort();

}}

}}

Symbolic Execution

Program Error

x=-10, y=-20, z=-20 x=m, y=n, z=2m

2m = n

m = n+10

Properties of DARTProperties of DART

►Sound w.r.t errors found (no false Sound w.r.t errors found (no false positives)positives)

►Complete (in a limited sense) Complete (in a limited sense)

If DART terminates without reporting a If DART terminates without reporting a bug, no bug existsbug, no bug exists

Advantages:Advantages:

►Dynamic Data AnalysisDynamic Data Analysis

►Sound ErrorsSound Errors

Dynamic Data AnalysisDynamic Data Analysis

struct foo { int i; char c; }bar (struct foo *a) {

if (a->c == 0) {*((char *)a +

sizeof(int)) = 1;if (a->c != 0)abort();}

}

Dynamic Data AnalysisDynamic Data Analysis

► Static Analyzers would not be able to detect the Static Analyzers would not be able to detect the change in the value of a->c and declare the change in the value of a->c and declare the program to be safeprogram to be safe

► DART locates the error by means of satisfying the DART locates the error by means of satisfying the constraint a->c==0constraint a->c==0

Sound ErrorsSound Errors

foobar(int x, int y){if (x*x*x > 0){

if (x>0 && y==10) abort();

} else { if (x>0 && y==20) abort();

} }

Sound ErrorsSound Errors

► Static analysis using predicate abstraction Static analysis using predicate abstraction state that both aborts may be reachablestate that both aborts may be reachable

► Test-generation using symbolic execution Test-generation using symbolic execution get stuck at first conditionalget stuck at first conditional

► DART detects first abort with high DART detects first abort with high probabilityprobability

DART For CDART For C

► Interface Extraction done by light weight Interface Extraction done by light weight static parsing of source codestatic parsing of source code

►Test Driver Generation involves Test Driver Generation involves initialization of external variables with initialization of external variables with random values, stubs for external random values, stubs for external functionsfunctions

►Directed Search comprises of Directed Search comprises of implementation of constraint satisfaction, implementation of constraint satisfaction, parsing and analysis of C codeparsing and analysis of C code

DiscussionDiscussion

Documents

DART Directed Automated Random Testing Patrice Godefroid, Nils Klarlund, and Koushik Sen Syed Nabeel