1 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

Comparing Metaheuristic Algorithms for Error Detection in Java Programs

Francisco Chicano, Marco Ferreira and Enrique Alba

Introduction Background Algorithms Experiments Conclusions

& Future Work

2 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

•  Concurrent software is difficult to test ...

•  ... and it is in the heart of a lot of critical systems

•  Techniques for proving the correctness of concurrent software are required •  Model checking → fully automatic •  Traditional techniques for this purpose have problems with large models •  We compare several metaheuristics and classical algorithms for model checking

Motivation Motivation

Introduction Background Algorithms Experiments Conclusions

& Future Work

3 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

Intersection Büchi automaton

•  Objective: Prove that model M satisfies the property :

•  In the general case, f is a temporal logic formula (LTL, CTL, etc.)

s0

s4 s7

s6

s2

s1

s8 s9

s5

s3

s0

s1 s2

s3

s4

s5

Model M

s1

s0 s2

q p∧!q

!p∨q

LTL formula ¬ f (never claim)

∩

Explicit State Model Checking Explicit State MC Properties State Explosion Heuristic MC

Introduction Background Algorithms Experiments Conclusions

& Future Work

4 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

Intersection Büchi automaton

•  Objective: Prove that model M satisfies the property :

•  In the general case, f is a temporal logic formula (LTL, CTL, etc.)

s0

s4 s7

s6

s2

s1

s8 s9

s5

s3

s0

s1 s2

s3

s4

s5

Model M

s1

s0 s2

q p∧!q

!p∨q

LTL formula ¬ f (never claim)

∩ =

Explicit State Model Checking Explicit State MC Properties State Explosion Heuristic MC

Introduction Background Algorithms Experiments Conclusions

& Future Work

5 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

Intersection Büchi automaton

•  Objective: Prove that model M satisfies the property :

•  In the general case, f is a temporal logic formula (LTL, CTL, etc.)

s0

s4 s7

s6

s2

s1

s8 s9

s5

s3

s0

s1 s2

s3

s4

s5

Model M

s1

s0 s2

q p∧!q

!p∨q

LTL formula ¬ f (never claim)

∩

Using Nested-DFS

=

Explicit State Model Checking Explicit State MC Properties State Explosion Heuristic MC

Introduction Background Algorithms Experiments Conclusions

& Future Work

6 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

•  An error trail is an execution path ending in an error state •  The search for errors is transformed in a graph exploration problem (DFS, BFS)

s0

s4 s7

s6

s2

s1

s8 s9

s5

s3

Safety properties

Properties in JPF

•  Exceptions

•  Deadlocks

Introduction Background Algorithms Experiments Conclusions

& Future Work

Explicit State MC Properties State Explosion Heuristic MC

7 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

s0

s4 s7

s6

s2

s1

s8 s9

s5

s3

•  Number of states very large even for small models

•  Example: Dining philosophers with n philosophers → 3n states

•  For each state we need to store the heap and the stacks of the different threads

•  Solutions: collapse compression, minimized automaton representation, bitstate

hashing, partial order reduction, symmetry reduction

•  Large models cannot be verified but errors can be found

Memory

State Explosion Problem

Introduction Background Algorithms Experiments Conclusions

& Future Work

Explicit State MC Properties State Explosion Heuristic MC

8 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

•  The search for errors can be directed by using heuristic information

•  Different kinds of heuristic functions have been proposed in the past: •  Formula-based heuristics •  Structural heuristics

s0

s4 s7

s6

s2

s1

s8 s9

s5

s3 2

0

3

5

1

2

4 0

7 6

Heuristic value

Heuristic Model Checking

•  Deadlock-detection heuristics

•  State-dependent heuristics

Introduction Background Algorithms Experiments Conclusions

& Future Work

Explicit State MC Properties State Explosion Heuristic MC

9 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

Classification of Algorithms

Introduction Background Algorithms Experiments Conclusions

& Future Work

Algorithms GA GAMO PSO ACO SA

Guided

Non-guided

Complete Non-complete

Deterministic

Stochastic

Completeness

Det

erm

inis

m

DFS, BFS

A*

GA, GAMO, PSO, SA, ACO

RS

BS

RDFS

EDA

?

(working on)

10 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

Genetic Algorithm

0.5 0.1 0.9 0.3 0.5 0.9

Solution encoding

(floating point values)

Crossover

0.5 0.1 0.9 0.3 0.5 0.9 → 0.5 0.1 0.6 0.3 0.5 0.9 Mutation

Algorithms GA GAMO PSO ACO SA

Introduction Background Algorithms Experiments Conclusions

& Future Work

11 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

Genetic Algorithm with Memory Operator

0.5 0.1 0.9 0.3 0.5 0.9

Solution encoding

(floating point values)

0 1 2 3 … …

Index in a table of states

Introduction Background Algorithms Experiments Conclusions

& Future Work

Algorithms GA GAMO PSO ACO SA

12 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

Inertia Personal best

Global best

→ Position (solution)

→ Velocity

0.2 -1.4 -3.5

1.0 10.3 7.2

Particles

Introduction Background Algorithms Experiments Conclusions

& Future Work

Algorithms GA GAMO PSO ACO SA

Particle Swarm Optimization

13 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

•  The ant selects stochastically its next node

•  The probability of selecting one node depends on the pheromone trail and the heuristic value (optional) of the edge/node •  The ant stops when a complete solution is built

i

j l

m

k

Ni

τij

ηij

k Trail

Heuristic

Ant Colony Optimization

Introduction Background Algorithms Experiments Conclusions

& Future Work

Algorithms GA GAMO PSO ACO SA

14 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

Simulated Annealing

0.5 0.1 0.9 0.3 0.5 0.9 → 0.5 0.1 0.6 0.3 0.5 0.9 Neighbor

Introduction Background Algorithms Experiments Conclusions

& Future Work

Algorithms GA GAMO PSO ACO SA

15 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

•  We used 3 scalable and 2 non-scalable models for the experiments

•  Maximum number of expanded states: 200 000 •  Fitness function:

•  100 independent executions of stochastic algorithms

Program LoC Classes Processes dinj 63 1 j+1 phij 176 3 j+1 marj 186 4 j+1 giop 746 13 7 garp 458 7 7

Parameterization Parameterization Hit Rate Length of Error Trails

Introduction Background Algorithms Experiments Conclusions

& Future Work

16 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

•  We used 3 scalable and 2 non-scalable models for the experiments

•  Maximum number of expanded states: 200 000 •  Fitness function:

•  100 independent executions of stochastic algorithms

Program LoC Classes Processes dinj 63 1 j+1 phij 176 3 j+1 marj 186 4 j+1 giop 746 13 7 garp 458 7 7

Parameterization

j=4 to 20

j=4 to 36

j=2 to 10

Parameterization Hit Rate Length of Error Trails

Introduction Background Algorithms Experiments Conclusions

& Future Work

17 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

Hit rate

Introduction Background Algorithms Experiments Conclusions

& Future Work

Parameterization Hit Rate Length of Error Trails

18 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

Hit rate

Introduction Background Algorithms Experiments Conclusions

& Future Work

phi

Parameterization Hit Rate Length of Error Trails

19 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

Hit rate

Introduction Background Algorithms Experiments Conclusions

& Future Work

din

Parameterization Hit Rate Length of Error Trails

20 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

Hit rate

Introduction Background Algorithms Experiments Conclusions

& Future Work

Parameterization Hit Rate Length of Error Trails

21 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

Length of Error Trails

Introduction Background Algorithms Experiments Conclusions

& Future Work

BS: 753

Parameterization Hit Rate Length of Error Trails

22 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

Introduction Background Algorithms Experiments Conclusions

& Future Work

BS: 172

BS: 260 DFS: 245

DFS: 378

Parameterization Hit Rate Length of Error Trails

Length of Error Trails

23 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

Introduction Background Algorithms Experiments Conclusions

& Future Work

Parameterization Hit Rate Length of Error Trails

Length of Error Trails

24 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

•  Metaheuristics are more effective than classical algorithms in finding errors

•  Beam Search has advantages over complete search algorithms

•  An even distribution of the search in depth levels tends to raise hit rate

•  Stochastic algorithms obtain short error trails

Conclusions

Future Work

•  Design a stochastic complete guided algorithm to find errors and verify

•  Design of hybrid algorithms to more efficiently explore the search space

•  Explore the design of parallel metaheuristics for this problem

Conclusions & Future Work

Introduction Background Algorithms Experiments Conclusions

& Future Work

Conclusions & Future Work

25 / 25 SSBSE 2011, Szeged, Hungary, September 10-12

Thanks for your attention !!!

Comparing Metaheuristic Algorithms for Error Detection in Java Programs