Model Checking Java Programs using

Structural Heuristics

Alex GroceCarnegie Mellon University

Willem VisserNASA Ames Research Center

Model Checking

• Explores graph of reachable system states– Checking for local assertions, invariants and

general temporal (logic) properties

• Symbolic model checking

• Explicit-state model checking

Java PathFinder

void add(Object o) { buffer[head] = o; head = (head+1)%size;}

Object take() { … tail=(tail+1)%size; return buffer[tail];}

Java Code

JAVAC JVM

0: iconst_01: istore_22: goto #395: getstatic 8: aload_09: iload_210: aaload

Bytecode

Special JVM

Model Checker

Depth-first Search

push initial state on Stackwhile (Stack not empty)

s = top(Stack)if s has no more successors

pop the Stackelse

s’ = next successor of sif s’ not already visited

mark s’ visitedif s’ is a goal state

then terminatepush s’ on Stack

Problems with DFS

• Produces lengthy counterexamples

• If state-space is too large to fully explore– May expend all resources on a single path when

shallow counterexamples exist– Failed runs give little information because

states explored may be very “similar”

Directed Model Checking

• Model checking as a search in a state space

• Why not use heuristics to guide the search?– Need to know what we’re looking for

• Can we find good heuristics for model checking?

• Bug-finding rather than verification

Best-first Search

priority queue Q = {initial state}while (Q not empty)

s = state in Q with lowest fremove s from Qfor each successor state s’ of s

if s’ not already visitedmark s’ visitedif s’ is a goal state

then terminatef = h(s’)store (s’, f) in Q

Two Kinds of Heuristics

• Property-specific heuristics

– Directed at a specific error• Number of unblocked threads as a measure of

distance to deadlock

• Static analysis for distance to an assertion check

– Focus of most previous work in field

Two Kinds of Heuristics

• Structural heuristics

– Designed to explore the structure of a program in a systematic fashion

– But what do we mean by structure?

Structural Heuristics

• One obvious kind of structure in a program:

– Control flow

• Reachable control flow rather than just CFG

• Motivation for branch coverage metrics used in software testing

Branch Coverage

• Instrument model checker to calculate branch coverage

• Using a simple coverage measure as a heuristic doesn’t work well

– Easily falls into local minima (once any branches are taken, every state on that path has “better” coverage)

– Doesn’t distinguish between branches explored once and branches explored many times

The Branch Counting Heuristic

• Count the number of times each branch has been taken

• Heuristic value is then:– Branches never before taken get lowest value– Non-branching transitions are next lowest– Otherwise, score is equal to the count

(lower values are explored first)

Three Searches

CFGBranch Counting

DFS

BFS

Each CFG state is a basic block that incrementssome variable x.

ERROR

Three Searches

CFGBranch Counting

DFS

BFS

Three Searches

CFGBranch Counting

DFS

BFS

Three Searches

CFGBranch Counting

DFS

BFS

Three Searches

CFGBranch Counting

DFS

BFS

Heuristic avoids taking

Three Searches

CFGBranch Counting

DFS

BFS

Three Searches

CFGBranch Counting

DFS

BFS

Expands 15 states

Expands 25 states

Terminatesonly with

depth limit

Experimental Results

• DEOS real-time operating system example

• This version uses an integer valued counter, without abstraction

Results for DEOS

All experiments performed on a 1.4GHz Athlon, limiting Java heap size to 512MB, all times are in seconds

Search Strategy States Time Memory Length Max DepthBranch-count 2,701 60 91MB 136 139%-coverage 20,215 FAIL FAIL FAIL 334Random heuristic 8,057 162 240MB 334 360BFS 18,054 FAIL FAIL FAIL 135DFS 14,678 FAIL FAIL FAIL 14,678DFS depth 500 392,470 6,782 383MB 455 500DFS depth 1000 146,949 2,222 196MB 987 1,000DFS depth 4000 8,481 171 270MB 3,997 4,000

The Interleaving Heuristic

• An important (and very hard to find) class of errors in Java is concurrency errors

• What kind of structure could we explore to catch these?

– Thread-interdependency

The Interleaving Heuristic

• Not clear how to heuristically define actual thread-interdependence

• So we use an approximation:– Executions in which context is switched more

often are given better heuristic values– Explores executions unlikely to appear in

testing (JVM/JITs schedule quite differently)

The Interleaving Heuristic

• Keep track on each path of which threads are executed at each transition

• Give lower (better) heuristic score to paths in which the most recently executed thread has been run less frequently

• Slightly more complicated in practice, counting live threads

Limiting the Queue

• With heuristics we are more interested in finding bugs than in verification

• So, we apply a technique from heuristic search literature:– Limit the size of the priority queue!

• When queue has more than k states in it, remove all but k states with best heuristic values

Experimental Results

• Dining Philosophers– Comparison to other results:

• Godefroid and Khurshid in TACAS ’02 paper apply genetic algorithms to dining philosophers

– Best result reported is 17 philosophers, 177 seconds, 50% success rate (on a slower machine)

• HSF-SPIN– Not clear how to compare (times not given)

– Best result they show is 16 philosophers, and SPIN (using partial order reduction) itself fails with 14 philosophers

Experimental ResultsSearch Strategy Threads States Time Memory Length Max DepthRandom heuristic 8 218,500 FAIL FAIL FAIL 86BFS 8 436,068 FAIL FAIL FAIL 13DFS 8 398,906 FAIL FAIL FAIL 384,286DFS depth 500 8 1,354,747 FAIL FAIL FAIL 500DFS depth 1000 8 1,345,289 FAIL FAIL FAIL 1,000DFS depth 4000 8 1,348,398 FAIL FAIL FAIL 4,000Interleaving 8 487,942 FAIL FAIL FAIL 16Most-blocked 8 310,317 FAIL FAIL FAIL 285Interleaving (k = 1000) 8 354,552 60 137MB 67 67Most-blocked (k = 5) 8 891,177 17,259 378MB 78,353 78,353Most-blocked (k = 160) 8 25,023 10 12MB 172 172Most-blocked (k = 1000) 8 123,640 46 59MB 254 278Interleaving (k = 40) 16 69,987 16 45MB 131 131Interleaving (k = 160) 16 290,637 60 207MB 131 132Most-blocked (k = 40) 16 101,576 38 69MB 1,008 1,008Interleaving (k = 5) 64 101,196 59 206MB 514 514

One Last Heuristic

• The choose-free heuristic:– Works only for abstracted Java programs– Rewards transitions that do not involve

nondeterminism introduced by the abstraction– Prefers counterexamples that do not result from

loss of precision introduced by the abstraction

• Structure of abstraction, not program

Previous Work

• Edelkamp, Lafuente, and Leue– HSF-SPIN: SPIN + heuristic search framework

• Bloem, Ravi, and Somenzi– Symbolic Guided Search: BDDs + heuristics– With BDDs heuristics can aid verification

• Cobleigh, Clarke, and Osterweil– FLAVERS verification work

Conclusions

• Structural heuristics: a useful class of heuristics– When model checking is used for debugging, we may

not know what kinds of bugs we are hunting

• Property-specific heuristics are also useful; approach is complementary, not replacement– Most-blocked can perform as well or better than

interleaving in the Remote Agent example, depending on the k limit and search method

Future Work

• Experiment with other, larger examples• Static analysis for property-specific heuristics• Language for properties/search/heuristics• Discover how heuristics work when symbolic

execution is introduced into JPF• Counterexample analysis for “bug causality”• What other kinds of structure can be exploited

with heuristics?– Counting occurrences of data values, perhaps