Guided Test Visualization : Making Sense of Errors in Concurrent Programs

Guided Test Visualization: Making Sense of Errors in

Concurrent Programs Saint Wesonga & Neha Rungta & Eric Mercer Brigham Young University & NASA AMES, USA

State of the Art Stress Testing Dynamic Analysis Runtime monitoring Static Analysis Model Checking Symbolic Execution Software Model Checking

Guided-test [SPIN 2009]Slice And Dice [ICSE-NIER

2010]

Belief

Sea of choices

Guided by belief

Guided-test and Slice-and-dice

Guided-test wants to find real errors

Find an error or you do not have much to say

Not intended to be exhaustive

Thus not complete

Yet sound

Our SolutionPossible errorslocations from

imprecise analysis tools

Generate backward

slicesfrom

possibleerror

locationsRuntime

Environment

Sound error

detection

Add inter-thread dependenceinformation

Heuristics

Guided Symbolic Execution

InputPossible errorslocations from


Generate backward

slicesfrom

possibleerror

locationsRuntime

Environment

Sound error

detection


Heuristics


Small exampleThread A {

void run (Element elem) { lock(elem)

check(elem)unlock(elem)

}

void check(Element elem) {if(elem.e > 9)

Throw exception

}}

Thread B {void run (Element elem) {

int x; // input variable

if( x > 18) {

lock(elem)

elem.reset();

unlock(elem)}

}}Object Element {

int e;Element() { e = 1;

}

void reset() {e = 11;

}}

Small ExampleThread A {



}


Throw exception

}}



if( x > 18) {

lock(elem)

elem.reset();

unlock(elem)}

}}Object Element {


}


}}




}


Throw exception

}}



if( x > 18) {

lock(elem)

elem.reset();

unlock(elem)}

}}Object Element {


}


}}




}


Throw exception

}}



if( x > 18) {

lock(elem)

elem.reset();

unlock(elem)}

}}Object Element {


}


}}




}


Throw exception

}}



if( x > 18) {

lock(elem)

elem.reset();

unlock(elem)}

}}Object Element {


}


}}




}


Throw exception

}}



if( x > 18) {

lock(elem)

elem.reset();

unlock(elem)}

}}Object Element {


}


}}




}


Throw exception

}}



if( x > 18) {

lock(elem)

elem.reset();

unlock(elem)}

}}Object Element {


}


}}

Could it really be?Thread A {



}


Throw exception

}}



if( x > 18) {

lock(elem)

elem.reset();

unlock(elem)}

}}Object Element {


}


}}

Is it possible to arrive at this location on some

execution?

Initial Abstraction to Follow (not explore)

Possible errorslocations from


Generate backward

slicesto possible

error locations

Runtime Environment

Sound error

detection


Heuristics


Slice along sequential executionThread A {



}


Throw exception

}} Object Element {


}


}}

Target Locations & Start of ProgramsA.run lstart

Exception

Thread A {void run (Element elem) {

lock(elem)


}


Throw exception

}} Object Element {


}


}}

Call Sites and Start Locs of the calleeA.run lstart

check(elem)

A.check lstart

Exception

Object Element { int e;

Element() { e = 1;

}


}}


lock(elem)


}


Throw exception

}}

Conditional StatementsA.run lstart

check(elem)

A.check lstart

if(elem.e > 9)

Exception


lock(elem)


}


Throw exception

}} Object Element {


}


}}

Data DefinitionsA.run lstart

check(elem)

A.check lstart

if(elem.e > 9)

Exception


lock(elem)


}


Throw exception

}} Object Element {


}


}}

Synchronization OperationsA.run lstart

lock(elem)

check(elem)

unlock(elem) A.check lstart

if(elem.e > 9)

Exception


lock(elem)


}


Throw exception

}} Object Element {


}


}}

Abstract SystemA.run lstart

lock(elem)

check(elem)


if(elem.e > 9)

Exception


lock(elem)


}


Throw exception

}} Object Element {


}


}}

Abstract TraceA.run lstart

lock(elem)

check(elem)


if(elem.e > 9)

Exception


lock(elem)


}


Throw exception

}} Object Element {


}


}}


lock(elem)

check(elem)

A.check lstart

if(elem.e > 9)

Exception


lock(elem)


}


Throw exception

}} Object Element {


}


}}

One trace of many we could have selected: fair coin or

farm in parallel



Generate backward

slicesto possible

error locations

Runtime Environment

Sound error

detection


Heuristics


Extract current program location of the

most recently executed thread

Map Concrete state to a location

Thread-0

StackLocalVars

Thread-1

StackLocalVars

Thread-n

StackLocalVars

HeapGlobalVars

Data Input VariablesSymbolic Representations

ϕpath Constraint

s1

l1

Goal is to match a concrete state to the next program

location in our trace

Follow the trace!

Guidance Strategys0

s2

s3

s1

l1

l2

ln

Guidance Strategys0

s2

s3

s1

l1

l2

ln

Flip a fair coin and move forward

Guidance Strategys0

s2

s3

s1

s5 s6s4

l1

l2

ln

Distance heuristic to choose nearest to next location on the current trace



Generate backward

slicesto possible

error locations

Runtime Environment

Sound error

detection

Add inter-thread dependenceinformation (refine)

Heuristics


Followed trace to predicate…A.run lstart

lock(elem)

check(elem)

A.check lstart

if(elem.e > 9)

Exception

False in all successors

(zut!)

Where else is elem.e defined?Thread A {



}


Throw exception

}}



if( x > 18) {

lock(elem)

elem.reset();

unlock(elem)}

}}Object Element {


}


}}

Redefine a global variableB.run lstart

e := 11

A.run lstart

lock(elem)

check(elem)


if(elem.e > 9)

Exception

Generate a new backward sliceB.run lstart

e := 11

if(x > 18)

lock(elem)

unlock(elem) Element.reset lstart

elem.reset()



if( x > 18) {

lock(elem)

elem.reset();

unlock(elem)}

}} Object Element {


}


}}

Updated Abstract SystemB.run lstart

e := 11

if(x > 18)

lock(elem)


elem.reset()

A.run lstart

lock(elem)

check(elem)


if(elem.e > 9)

Exception

Update Abstract TraceB.run lstart

e := 11

if(x > 18)

lock(elem)

Element.reset lstart

elem.reset()

A.run lstart

lock(elem)

check(elem)

A.check lstart

if(elem.e > 9)

Exception

Check Lock DependenciesB.run lstart

e := 11

if(x > 18)

lock(elem)


elem.reset()

A.run lstart

lock(elem)

check(elem)

A.check lstart

if(elem.e > 9)

Exception

Update Abstract TraceB.run lstart

e := 11

if(x > 18)

lock(elem)


elem.reset()

unlock(elem)

A.run lstart

lock(elem)

check(elem)

A.check lstart

if(elem.e > 9)

Exception

Thread B {void run (Element

elem) { int x; //

input variableif( x > 18)

{

lock(elem)

elem.reset();

unlock(elem)}

}}

Update Abstract StateB.run lstart

e := 11

if(x > 18)

lock(elem)


elem.reset()unlock(elem)

A.run lstart

lock(elem)

check(elem)

A.check lstart

if(elem.e > 9)

Exception

Restart with this new trace from the initial

state

Slice and dice wants to find real errors and prove correct…

Don’t find an error yet have much to say

Goal is still to find errors

so optimize search

yet useful in no error


lock(elem)

check(elem)


if(elem.e > 9)

Exception


lock(elem)


}


Throw exception

}} Object Element {


}


}}

Slice and Dice Keep entire

abstraction Schedule only on

node in the abstraction

A.run lstart

lock(elem)

check(elem)


if(elem.e > 9)

Exception

When it is time to refine…B.run lstart

e := 11

if(x > 18)

lock(elem)


elem.reset()unlock(elem)

A.run lstart

lock(elem)

check(elem)

A.check lstart

if(elem.e > 9)

Exception…rather than

create a single trace…

Keep the entire abstractionB.run lstart

e := 11

if(x > 18)

lock(elem)


elem.reset()

A.run lstart

lock(elem)

check(elem)


if(elem.e > 9)

Exception

Visualization Demonstration

Belief

Sea of choices

Guided by belief

Guided-test and Slice-and-dice

jpf-guided-test

50

Verification & Validation LabBrigham Young University

Provo, UT 84606, USA

http://vv.cs.byu.edu/

VV Lab, Brigham Young University, USA



This slide left intentionally blank

Experimental Results A new GuidedSymbolic extension in Java

Pathfinder Uses the engine from the symbolic extension

(Pasareanu et. al. ISSTA 2008) Dynamic partial order reduction turned on Abstraction, refinement and heuristic

computation all performed on Java bytecode Libraries are part of the multi-threaded system

Multi-threaded Programs Reorder Benchmark

SLOC: 44, Reachability

Airline BenchmarkSLOC: 31, Reachability

VecDeadlock Real JDK 1.4 concurrent librarySLOC: 7267, Deadlock in Vector class

VecDeadlock Real JDK 1.4 concurrent librarySLOC: 7169, Deadlock in Vector class

VecRace Real JDK 1.4 concurrent librarySLOC: 7151, Race in ArrayList class

Error DiscoveryTi

me

in m

inut

es

60

30

20

10

Out of PatienceGuidedDFS

VecDeadlock0

VecDeadlock1

VecRace

ComparisonTi

me

in se

cond

s

60

30

20

10

GuidedDFS

VecDeadlock0

VecDeadlock1

VecRace

Error Discovery

56

Subject Guided Search

Thread # States Time (secs) Memory

Reorder(9,1) 11 205 1.67 7MB

Reorder(10,1) 12 239 1.67 7MB

Airline(15,3) 16 1210 3.23 5MB

Airline(20,2) 21 3279 7.46 5MB

Airline(20,1) 21 3609 7.46 5MB

VecDeadlock0 2 1370 4.56 66MB

VecDeadlock1 2 2948 6.89 66MB

VecRace 2 3120 7.98 65MB

Error Discovery

57

Subject Abstraction-Refinement

Total tracelength

Refinements

Reorder(9,1) 13 1

Reorder(10,1) 13 1

Airline(15,3) 3 13

Airline(20,2) 3 19

Airline(20,1) 3 20

VecDeadlock0 14 1

VecDeadlock1 15 2

VecRace 12 1

Documents

Guided Test Visualization : Making Sense of Errors in Concurrent Programs