LIFC – Université de Franche-Comté jeudi 15 octobre 2015 Model Based Testing using Symbolic Animation and Machine Learning CSTVA’2010 workshop Pierre-Christophe

LIFC – Université de Franche-Comté

jeudi 20 avril 2023

Model Based Testing using Symbolic Model Based Testing using Symbolic Animation and Machine LearningAnimation and Machine Learning

CSTVA’2010 workshop

Pierre-Christophe Bué, Frédéric Dadeau, Pierre-Cyrille Héam

LIFC – Université de Franche-Comté / INRIA CASSIS

{pierre-christophe.bue,frederic.dadeau,pierre-cyrille.heam}@lifc.univ-fcomte.fr

2LIFC – Université de Franche-Comté Thursday, April 20, 2023

MotivationsMotivations

Goal: generate model-based tests from B models use the B model to generate test cases (sequences of operations)

Existing work is unsatisfactory• LTG: Leirios Test Generator (commercial tool – Smartesting)

Structural model coverage criterion (control-flow graph of operations) On-the-fly exploration of the model states ( short test cases)

• ProTest: based on the ProB animator (Univ. Düsseldorf/Southampton) Finite state machine transition coverage Enumeration of all the existing states ( combinatorial explosion)

In this work: take the best of both worlds Use of symbolic techniques to build an abstraction of the system No complete enumeration of the system states Add diversity in the LTG test cases

Motivations > Goal



Technique: build a transition system from the B model

The B model: is seen as a white-box (know the model operation and their code) can be animated (i.e. one can check the executability of traces) Use of the CLPS-B solver (developed at the LIFC – Besançon)

Use of symbolic states: Symbolic state comparison (to detect state inclusion) is difficult Use of a higher-level state characterization criterion

Compute the abstraction incrementally Using examples or counter-examples In order to improve the « correctness » of the abstraction

Motivations > Technique



Proposal of this work:

• Compute the abstraction incrementally, by using Machine Learning algorithms

• Explore the abstraction so as to build the test cases

Use of symbolic animation, based on constraint solving

Motivations > Proposal


OutlineOutline

Introduction/motivations

Symbolic Animation of B Machines

Machine Learning

Experimentations

Conclusion and Future Work

Outline


Symbolic Animation of B Machines Symbolic Animation of B Machines

B is a formal development method Starting with a B abstract machine Refined until reaching an implementation Proof obligations ensure consistency of models

In this work, we focus on B abstract machines

B machines are made of: A set-theoretical data model (sets, relations, functions, integers) Generalized Substitutions describing operations

Symbolic Animation of B Machines > B Machines



Example : consider the example of a robot

Symbolic Animation of B Machines > B Machine example

Robot: Bender

Features: - intended to bend metal bars

But instead:- smokes - drinks (preferably alcohol)- steals (when possible) - is friend with some humans- wants to kill all humans

© Matt Groening - Futurama



MACHINE BenderBendingRodriguez

SETS ITEMS = {sunglasses, gun};

DRINK = {water, alcohol};

HUMANS;

BEHAVIORS = {be_nice, be_nasty, kill_human}

VARIABLES inventory, alcohol

INVARIANT

inventory ⊆ ITEMS ∧ alcohol ∈ -100..100

INITIALIZATION

inventory := ∅ || alcohol := 20

OPERATIONS

steal(obj) =

PRE obj ∈ ITEMS obj inventory THEN

inventory := inventory {obj}

END;

Symbolic Animation of B Machines > B Machine example

drink(kind,amount) = PRE kind ∈ DRINK ∧ amount ∈ {10,20,30,40,50} THEN IF kind = alcohol THEN IF (alcohol + amount > 100) THEN alcohol := 100

ELSE alcohol := alcohol + amount END

ELSE IF (alcohol - amount < -100) THEN alcohol := -100 ELSE alcohol := alcohol - amount END

END END;

b ← meet(human) = PRE human ∈ HUMANS THEN IF (alcohol < 0 ∧ {gun,sunglasses} = inventory) THEN b := kill_human

ELSE IF alcohol > 0 THEN b := be_nice ELSE b := be_nasty END END

ENDEND



Symbolic animation …

… is used to simulate the execution of the model by invoking operations without instantiating the parameters parameters are replaced with symbolic values related state variables also become symbolic

… involves the use Constraint Logic Programming Constraint solvers manage symbolic values

The activation of operation behaviors are seen as a CSP

Backtracking is used to iterate over the possible behaviors

Symbolic Animation of B Machines > Principles of symbolic animation



A behavior is a before-after predicate that describes one possible way to execute the operation

drink(kind,amount) =

PRE kind ∈ DRINK ∧ amount ∈ {10,20,30,40,50} THEN

IF kind = alcohol THEN

IF (alcohol + amount > 100)

THEN alcohol := 100

ELSE alcohol := alcohol + amount

END

ELSE

IF (alcohol - amount < -100)

THEN alcohol := -100

ELSE alcohol := alcohol - amount

END

END

END;


2

3 6

1kind ∈ DRINK ∧amount ∈{10,20,30,40,50}

kind = alcohol kind ≠ alcohol

0

alcohol + amount > 100

alcohol + amount ≤ 100

alcohol - amount ≥ -100

4

alcohol - amount < -100

5 7 8

alcohol' = alcohol + amount

alcohol' = alcohol - amount

alcohol' = -100

alcohol' = 100



Example: activation of behaviors

State:

inventory ={sunglasses},

alcohol = 70

Activation of drink(_K,_Am)

→ Path [1,2,3,4,0] activable

State':

inventory = {sunglasses}

alcohol = _A

With constraints:

_K = alcohol,

_Am ∈{10,20,30},

_A = 70+_Am


2

3 6



0




4


5 7 8


alcohol' = alcohol - amountalcohol' = 100 alcohol' =

-100




State:


alcohol = 70



State':

inventory' = {sunglasses}

alcohol' = 100

With constraints:

_K = alcohol,

_Am ∈{40,50}


2

3 6



0




4


5 7 8



-100




State:


alcohol = 70



State':

inventory = {sunglasses}

alcohol = _A

With constraints:

_K = water,

_Am ∈{10,20,30,40,50},

_A = 70 - _Am


2

3 6



0




4


5 7 8



-100




State:


alcohol = 70


→ Path [1,2,6,8,0] not activable


2

3 6



0




4


5 7 8



-100



In the end, constraint systems are instantiated to produce tests

State: inventory ={sunglasses}, alcohol = 70

Activation of drink(_K1,_Am1) → Path [1,2,3,4,0]

State':

inventory' = {sunglasses}, alcohol' = _A1

With constraints:

_K1 = alcohol, _Am1 {10,20,30}, _A1 = 70+_Am1

Then, activation of drink(_K2,_Am2) → Path [1,2,3,4,0]

State'':

inventory'' = {sunglasses}, alcohol'' = _A2 With constraints:

_A2 = _A1 + _Am2, _Am1 ∈{10,20}, _Am2 ∈{10,20},_Am1 + _Am2 ≤ 30, _A1 = 70+_Am1


2

3


kind = alcohol

0


4

alcohol' = alcohol + amountThree solutions:

_Am1 = 10 _Am1 = 10 _Am1 = 20

_Am2 = 10 _Am2 = 20 _Am2 = 10

_A1 = 80 _A1 = 90 _A1 = 90

_A2 = 90 _A2 = 100 _A2 = 100


OutlineOutline



Machine Learning

Experimentations


Outline


Machine LearningMachine Learning

Aims at inferring a DFA from the observation of a system, seen as a black box

Based on the knowledge of an alphabet L (input actions of the system)• Infers an DFA recognizing the language admitted by the system

• Based on examples/counter-examples (words) that refine the inferred DFA

Most famous L* introduced by Dana Angluin in 1987• Stores observations into tables

• Variants (proposed by Keans and Vaziran) store

observations into trees

Machine Learning > Principles


Machine LearningMachine Learning

Machine Learning > Example

The learnerThe system

The teacher

The Oracle

DFA

infers

Membership queries

Equivalence queries

Fry: Oh, almighty oracle, is this DFA correct?

DrZoidberg: Yes/No + counter-example

knowsExamples/counter-examples


Machine Learning ApplicationMachine Learning Application

Use of Machine Learning coupled with symbolic animation to execute membership queries each letter of the alphabet is a behavior of an operation

State merging criterion Executability of the same set of behaviors from a given state Similar to ready-simulation At most 2|L| states … … but in practice, numerous behaviors are mutually exclusive

Oracle issue: use technique from Vasilevski and Chow Compute sequences from the exploration of the abstraction If not replayable counter-examples used to refine the abstraction Otherwise keep sequences as test cases

Machine Learning > Application




1

- s: steal(_O)- d1: drink(alcohol, _Am) alcohol’ ≤ 100 - d2: drink(alcohol, _Am) alcohol’ =100- d3: drink(water, _Am) alcohol’ ≥ -100- d4: drink(water, _Am) alcohol’ = -100- m1: meet(_H) be_nice- m2: meet(_H) be_nasty- m3: meet(_H) kill_human

State 1 (initial state): s, d1, d3, m1

2

s

3

d1

4

d3

5

m1

State 2: s, d1, d3, m1 merged with State 1

State 3: s, d1, d2, d3, m1 State 4: s, d1, d3, m1, m2 State 5: s, d1, d3, m1

merged with State 1

6 7 8 9

s

d1 d2

m1

10

d3

State 6, 7, 10: s, d1, d2, d3, m1 merged with State 3State 8: s, d2, d3, m1

State 9: s, d1, d3, m1, m2 merged with State 4

11 12 13 14

d2 d3m1

s

States 11-14: s, d2, d3, m1 merged with State 8

15 16 17 18

sd1 d3

m2

19

m1

States 15-17: s, d1, d3, m1, m2

merged with State 4State 18: s, d1, d3, m1

merged with State 1State 19: s, d1, d3, m2

20 21 22 23

d1 d3m2

s

States 20, 22, 23 : s, d1, d3, m2

merged with State 19State 21: s, d1, d3, m1, m2

merged with State 4




1

(1) 3 4 (1)

s

d1 d3

m1

(3) (3) 8 (4)

s

d1 d2

m1

(3)

d3

(8) (8) (8) (8)

d2 d3m1

s

(4) (4) (4) (1)

sd1 d3

m2

19

m1

(19) 4 (19) (19)

d1 d3m2

s


1

3 4

8

s, m1

s, d1, m1

s, d1, d3

d3d1

d2

d3

s, d2, d3, m1

19

m2d1

s, d3, m2

m1



• Exploration of the inferred DFA to produce:• Test cases, if the sequence can be replayed on the B model

• Counter-examples, if the sequence can not be replayed on the B model

• Use of a dedicated exploration algorithm• Random walks

• Greedy algorithms

• Chinese Postman

• Etc.

• Examples of sequences • Seq1 = [d1,d3,m2,d1,m1,d1,d2] - ok• Seq2 = [s, d1, s, d2, s] - ko


s can be called at most twice!

Refine the abstraction

1

3 4

8

s, m1

s, d1, m1

s, d1, d3

d3d1

d2

d3

s, d2, d3, m1

19

m2d1

s, d3, m2

m1




1

23 4 (1)

s

d1 d3

m1

(3) (3) 8 (4)

s

d1 d2

m1

(3)

d3

(8) (8) (8) (8)

d2 d3m1

s

(4) (4) (4) (1)

sd1 d3

m2

19

m1

(19) 4 (19) (19)

d1 d3m2

s

24

25

26

d1

s

d2

Counter-example [s, d1, s, d2]• Unmerge first merged state• Add the counter-example (None of the node can be merged)• Recompute successors


d3…


OutlineOutline



Machine Learning

Experimentations


Outline


Experiments done on two case studies

- Toy example: process scheduler (Scheduler)

- Realistic case study: electronic purse (Demoney)

Experimentation

ExperimentationExperimentation

Specification Scheduler Demoney

#concrete states

#behaviors

124

6

> 1020

93

Abstraction computation

Computation time

#states/transitions

#behaviors covered

#tests (Chinese Postman)

9 sec

9/24

6/6 (100%)

1 (unfeasible)

Failed at step 15

4 min 28 sec

18/496

42/93 (45%)

5 (4 feasibles)

Failure at step 47 kept

Refinement

Computation time

#states/transitions

#tests Chinese Postman

51 sec

19/51

1 (unfeasible)

Failed at step 39

-

-

-


Results are not so good … what to do?

1. Change the state merging criterion (too coarse)

check activability of the pairs of successive behaviors from a given state

- increases computation time (obviously)

- but, increases the quality of the abstraction! no refinement needed

- moreover, tests are good (beat LTG tests on a mutational analysis)

- unfortunately, all behaviors from the original model are not necessarily covered

2. Change the DFA exploration algorithm (too likely to produce failing sequences)

[currently under study]

Experimentation

ExperimentationExperimentation

Specification Scheduler Demoney

Abstraction computation

Computation time

#states/transitions

#behaviors covered

#tests (Chinese Postman)

90 sec

15/40

6/6 (100%)

1 (40 steps)

1h 56 min 28 sec

33/914

48/93 (52%)

11 (avg. length: 104 steps,

max. length: 435 steps)


OutlineOutline



Machine Learning

Experimentations


Outline


Conclusion and Future WorkConclusion and Future Work

We have seen: A way to couple symbolic animation and machine learning

Address the problem of building an abstraction of a system

Can be used to produce model-based tests

Work in progress … many more things to do:

• Achieve a 100% coverage of the behaviors of the original model

Use existing tests as « examples » that refine the abstraction

• Comparative study on the influence of:

the state merging criterion,

the abstraction exploration/test generation algorithm,

the treatment of examples/counter-examples

… on the quality of the abstraction and the quality of the resulting tests.



To be continued…To be continued…

Thanks for your attention!

Questions?

Documents

LIFC – Université de Franche-Comté jeudi 15 octobre 2015 Model Based Testing using Symbolic Animation and Machine Learning CSTVA’2010 workshop Pierre-Christophe