Guided Test Visualization: Making Sense of Errors in Concurrent Programs Saint Wesonga & Neha Rungta & Eric Mercer Brigham Young University & NASA AMES,

Guided Test Visualization: Making Sense of Errors in

Concurrent Programs

Saint Wesonga & Neha Rungta & Eric Mercer

Brigham Young University & NASA AMES, USA

State of the Art Stress Testing Dynamic Analysis Runtime monitoring Static Analysis Model Checking Symbolic Execution Software Model Checking

Guided-test [SPIN 2009]Slice And Dice [ICSE-NIER

2010]

Belief

Sea of choices

Guided by belief

Guided-test and Slice-and-dice

Guided-test wants to find real errors

Find an error or you do not have much to say

Not intended to be exhaustive

Thus not complete

Yet sound

Our Solution

Possible errorslocations from

imprecise analysis tools

Generate backward

slicesfrom

possibleerror

locations

Runtime Environment

Sound error

detection

Add inter-thread dependenceinformation

Heuristics

Guided Symbolic Execution

Input



Generate backward

slicesfrom

possibleerror

locations

Runtime Environment

Sound error

detection


Heuristics


Small exampleThread A {

void run (Element elem) { lock(elem)

check(elem)unlock(elem)

}

void check(Element elem) {if(elem.e > 9)

Throw exception

}}

Thread B {void run (Element elem) {

int x; // input variable

if( x > 18) {

lock(elem)

elem.reset();

unlock(elem)}

}}Object Element {

int e;Element() { e = 1;

}

void reset() {e = 11;

}}

Small ExampleThread A {



}


Throw exception

}}



if( x > 18) {

lock(elem)

elem.reset();

unlock(elem)}

}}Object Element {


}


}}




}


Throw exception

}}



if( x > 18) {

lock(elem)

elem.reset();

unlock(elem)}

}}Object Element {


}


}}




}


Throw exception

}}



if( x > 18) {

lock(elem)

elem.reset();

unlock(elem)}

}}Object Element {


}


}}




}


Throw exception

}}



if( x > 18) {

lock(elem)

elem.reset();

unlock(elem)}

}}Object Element {


}


}}




}


Throw exception

}}



if( x > 18) {

lock(elem)

elem.reset();

unlock(elem)}

}}Object Element {


}


}}




}


Throw exception

}}



if( x > 18) {

lock(elem)

elem.reset();

unlock(elem)}

}}Object Element {


}


}}

Could it really be?Thread A {



}


Throw exception

}}



if( x > 18) {

lock(elem)

elem.reset();

unlock(elem)}

}}Object Element {


}


}}

Is it possible to arrive at this location on some

execution?

Initial Abstraction to Follow (not explore)



Generate backward

slicesto possible

error locations

Runtime Environment

Sound error

detection


Heuristics


Slice along sequential executionThread A {



}


Throw exception

}} Object Element {


}


}}

Target Locations & Start of Programs

A.run lstart

Exception

Thread A {void run (Element elem) {

lock(elem)


}


Throw exception

}} Object Element {


}


}}

Call Sites and Start Locs of the callee

A.run lstart

check(elem)

A.check lstart

Exception

Object Element { int e;

Element() { e = 1;

}


}}


lock(elem)


}


Throw exception

}}

Conditional Statements

A.run lstart

check(elem)

A.check lstart

if(elem.e > 9)

Exception


lock(elem)


}


Throw exception

}} Object Element {


}


}}

Data Definitions

A.run lstart

check(elem)

A.check lstart

if(elem.e > 9)

Exception


lock(elem)


}


Throw exception

}} Object Element {


}


}}

Synchronization Operations

A.run lstart

lock(elem)

check(elem)

unlock(elem) A.check lstart

if(elem.e > 9)

Exception


lock(elem)


}


Throw exception

}} Object Element {


}


}}

Abstract System

A.run lstart

lock(elem)

check(elem)


if(elem.e > 9)

Exception


lock(elem)


}


Throw exception

}} Object Element {


}


}}

Abstract Trace

A.run lstart

lock(elem)

check(elem)


if(elem.e > 9)

Exception


lock(elem)


}


Throw exception

}} Object Element {


}


}}

Abstract Trace

A.run lstart

lock(elem)

check(elem)

A.check lstart

if(elem.e > 9)

Exception


lock(elem)


}


Throw exception

}} Object Element {


}


}}

One trace of many we could have selected: fair coin or

farm in parallel

Our Solution



Generate backward

slicesto possible

error locations

Runtime Environment

Sound error

detection


Heuristics


Extract current program location of the

most recently executed thread

Map Concrete state to a location

Thread-0

StackLocalVars

Thread-1

StackLocalVars

Thread-n

StackLocalVars

HeapGlobalVars

Data Input VariablesSymbolic Representations

ϕpath Constraint

s1

l1

Goal is to match a concrete state to the next program

location in our trace

Follow the trace!

Guidance Strategys0

s2

s3

s1

l1

l2

ln

Guidance Strategys0

s2

s3

s1

l1

l2

ln

Flip a fair coin and move forward

Guidance Strategys0

s2

s3

s1

s5 s6s4

l1

l2

ln

Distance heuristic to choose nearest to next location on the current trace

Our Solution



Generate backward

slicesto possible

error locations

Runtime Environment

Sound error

detection

Add inter-thread dependenceinformation (refine)

Heuristics


Followed trace to predicate…A.run lstart

lock(elem)

check(elem)

A.check lstart

if(elem.e > 9)

Exception

False in all successors

(zut!)

Where else is elem.e defined?Thread A {



}


Throw exception

}}



if( x > 18) {

lock(elem)

elem.reset();

unlock(elem)}

}}Object Element {


}


}}

Redefine a global variableB.run lstart

e := 11

A.run lstart

lock(elem)

check(elem)


if(elem.e > 9)

Exception

Generate a new backward sliceB.run lstart

e := 11

if(x > 18)

lock(elem)

unlock(elem) Element.reset lstart

elem.reset()



if( x > 18) {

lock(elem)

elem.reset();

unlock(elem)}

}} Object Element {


}


}}

Updated Abstract SystemB.run lstart

e := 11

if(x > 18)

lock(elem)


elem.reset()

A.run lstart

lock(elem)

check(elem)


if(elem.e > 9)

Exception

Update Abstract Trace

B.run lstart

e := 11

if(x > 18)

lock(elem)

Element.reset lstart

elem.reset()

A.run lstart

lock(elem)

check(elem)

A.check lstart

if(elem.e > 9)

Exception

Check Lock Dependencies

B.run lstart

e := 11

if(x > 18)

lock(elem)


elem.reset()

A.run lstart

lock(elem)

check(elem)

A.check lstart

if(elem.e > 9)

Exception

Update Abstract Trace

B.run lstart

e := 11

if(x > 18)

lock(elem)


elem.reset()

unlock(elem)

A.run lstart

lock(elem)

check(elem)

A.check lstart

if(elem.e > 9)

Exception

Thread B {void run (Element

elem) { int x; //

input variableif( x > 18)

{

lock(elem)

elem.reset();

unlock(elem)}

}}

Update Abstract State

B.run lstart

e := 11

if(x > 18)

lock(elem)


elem.reset()unlock(elem)

A.run lstart

lock(elem)

check(elem)

A.check lstart

if(elem.e > 9)

Exception

Restart with this new trace from the initial

state

Slice and dice wants to find real errors and prove correct…

Don’t find an error yet have much to say

Goal is still to find errors

so optimize search

yet useful in no error

Abstract Trace

A.run lstart

lock(elem)

check(elem)


if(elem.e > 9)

Exception


lock(elem)


}


Throw exception

}} Object Element {


}


}}

Slice and Dice Keep entire

abstraction Schedule only on

node in the abstraction

A.run lstart

lock(elem)

check(elem)


if(elem.e > 9)

Exception

When it is time to refine…

B.run lstart

e := 11

if(x > 18)

lock(elem)


elem.reset()unlock(elem)

A.run lstart

lock(elem)

check(elem)

A.check lstart

if(elem.e > 9)

Exception…rather than

create a single trace…

Keep the entire abstractionB.run lstart

e := 11

if(x > 18)

lock(elem)


elem.reset()

A.run lstart

lock(elem)

check(elem)


if(elem.e > 9)

Exception

Visualization Demonstration

Belief

Sea of choices

Guided by belief

Guided-test and Slice-and-dice

jpf-guided-test

50

Verification & Validation LabBrigham Young University

Provo, UT 84606, USA

http://vv.cs.byu.edu/

VV Lab, Brigham Young University, USA



This slide left intentionally blank

Experimental Results A new GuidedSymbolic extension in Java

Pathfinder Uses the engine from the symbolic extension

(Pasareanu et. al. ISSTA 2008) Dynamic partial order reduction turned on Abstraction, refinement and heuristic

computation all performed on Java bytecode Libraries are part of the multi-threaded system

Multi-threaded Programs Reorder Benchmark

SLOC: 44, Reachability Airline Benchmark

SLOC: 31, Reachability VecDeadlock Real JDK 1.4 concurrent library

SLOC: 7267, Deadlock in Vector class VecDeadlock Real JDK 1.4 concurrent library

SLOC: 7169, Deadlock in Vector class VecRace Real JDK 1.4 concurrent library

SLOC: 7151, Race in ArrayList class

Error DiscoveryTi

me in m

inute

s

60

30

20

10

Out of PatienceGuided

DFS

VecDeadlock0

VecDeadlock1

VecRace

ComparisonTi

me in s

eco

nd

s

60

30

20

10

Guided

DFS

VecDeadlock0

VecDeadlock1

VecRace

Error Discovery

56

Subject Guided Search

Thread # States Time (secs) Memory

Reorder(9,1) 11 205 1.67 7MB

Reorder(10,1) 12 239 1.67 7MB

Airline(15,3) 16 1210 3.23 5MB

Airline(20,2) 21 3279 7.46 5MB

Airline(20,1) 21 3609 7.46 5MB

VecDeadlock0 2 1370 4.56 66MB

VecDeadlock1 2 2948 6.89 66MB

VecRace 2 3120 7.98 65MB

Error Discovery

57

Subject Abstraction-Refinement

Total tracelength

Refinements

Reorder(9,1) 13 1

Reorder(10,1) 13 1

Airline(15,3) 3 13

Airline(20,2) 3 19

Airline(20,1) 3 20

VecDeadlock0 14 1

VecDeadlock1 15 2

VecRace 12 1

Documents

Guided Test Visualization: Making Sense of Errors in Concurrent Programs Saint Wesonga & Neha Rungta & Eric Mercer Brigham Young University & NASA AMES,