Synthesis of Test Purpose Directed Reactive Planning Tester for Nondeterministic Systems 1

Teooriapäevad, Vanaõue, Sept. 2007

Synthesis of Test Purpose Directed Reactive Planning Tester for Nondeterministic Systems 1

Jüri Vain, Kullo Raiend, Andres Kull, Juhan P. Ernits

1 Accepted in ASE 2007, Atlanta, Georgia, November 5-9, 2007


On-the-fly testing: what? and why? The most appropriate technique for model-based

testing of IUT modeled using nondeterministic models Denotes test generation and execution algorithms

that compute successive stimuli at runtime, directed by the test purpose and the observed outputs of the IUT

Advantages: The state-space explosion problem experienced by

many offline test generation methods is reduced because only a limited part of the state-space needs to be kept track of at any point in time.

Drawbacks: Exhaustive planning is diffcult due to the limitations

of the available computational resources at the time of test execution.


On-the-fly testing: scale of methods Random walk (the simplest): select test stimuli in random

inefficient - based on random exploration of the state space leads to test cases that are unreasonably long may leave the test purpose unachieved

RW with additional heuristics: E.g., anti-ant algorithms are applied for guiding the exploration

of the state space [Li, Veanes]

........?

State space exploration with exhaustive planning - solving constraint systems at each step witness trace generated by model checking provides possibly

optimal selection of the next test stimulus. in case of explicit state MC the size of the model is critical leads to the explosion of the state space specially in models

that include "combination lock" or deep loops [J. Rushby].


On-the-fly testing: missing link -Reactive Planning Tester

Reactive planning [C. Williams and P. P. Nayak]: can cope with highly dynamic and unpredictable environments. instead of a complete plan with branches, a set of decision

rules is produced directing towards the planning goal. just one subsequent input is computed at every step, based on

the current context.

Reactive planning tester [J.Vain et al] Model “in-the-loop”: testing is guided by an EFSM model stimulae to IUT: outputs generated by model execution responses from IUT: inputs to the model decision rules of reactive planning are encoded in the guards

of model transitions the rules are constructed by offline analysis based on the given

IUT model and the test purpose.


The model

Key assumption: the IUT model is presented as an output observable nondeterministic state machine: FSM EFSM in which all transition paths are

feasible (see Duale,2004 for algorithms of making EFSM feasible)


I/O EFSM (I) Definition 1: (EFSM) M = (S; V; I; O; E), where

S - finite set of states, s0 S - initial state, V - finite set of variables with finite value domains, I - finite set of inputs, O - finite set of outputs, E - set of transitions.

Configuration of M is a pair (s; ) where s S mapping from V to values.


I/O EFSM (II) Transition e =(s; p; a; o; u; q),where

s - the source state of the transition, q - target state of the transition (s; q) p - transition guard - a logic formula over V, a - input of M (a I), o – output of M (o O), u - update function over V.

Nondeterministic EFSM: more than one outgoing transitions enabled simultaneously


Example: nondeterministic EFSM


Test purpose (TP): TP is a specific property of the IUT that the tester is set out

to test.

In MBT the TPs are defined in terms of the IUT model elements

Examples: "test a state change from state A to state B in a model", "test whether some selected states of a model are visited", "test whether all transitions are visited at least once in a

model", etc.

All TPs of examples can be specified in terms of structural elements of the model that should be traversed during the test execution


Encoding Test Purpose in the IUT Model

Trap - a boolean variable assignment attached to the transitions of the IUT model

A trap variable is initially set to false. The trap update functions attached to

the transitions are executed (set trap variables to true) when the transition is visited.


Example: nondeterministic EFSM extended with trap variables


Model of the tester EFSM generated from the IUT model attributed with test

purpose I/O conformance of IUT and tester models is required transition guards encode the rules of online planning 2 types of tester states :

active – tester controls the next move passive – IUT controls the next move

2 types of transitions: Observable – source state is passive state, guard: true, Controllable – source state is active state, guard: pS pT,

where pS – guard of the dual transition of IUT,

pT – gain guard

The gain guard (defined on trap variables) must ensure that only those outgoing are enabled having maximum gain


Tester construction algorithm: a sketch Construct a control structure structure of the tester

model where the inputs and outputs are inverted. Split the transitions to pairs of controllable and

observable transitions by i/o operation attached.

Extend the tester automaton controllable transitions guards with “place holders” for gain guards

The gain guards are construct with reachability analysis by taking into account the reachability of all trap-equipped transitions from

a given state the length of the paths to them the current value (visited or not) of each trap.


Constructing the control structure of the tester


Constructing the gain guards (GG) (I): intuition

GG must guarantee that each transition enabled by GG is a prefix of

some locally optimal (w.r.t. test purpose) path

tester should terminate after the test goal is reached or all unvisited traps are unreachable from the current stat

to have a quantitative measure of the gain of executing any transition e we define a gain function ge:e +


Constructing the gain guards (II): the gain function ge = 0, if it is useless to fire the transition e from the

current state with the current variable bindings

ge > 0, if fireing transition e from the current state with the current variable bindings visits or leads closer to at least one unvisited trap

gei > gej for transitions ei and ej with the same source state, if taking the transition ei leads to unvisited traps with smaller cost than taking the transition ej.

Having gain function ge with given properties define GG:

pT ge = maxek gek ge > 0


Constructing the gain guards (III): shortest path tree

Reachability problem of trap labeled transitions can be reduced to single- source shortest path problem.

Arguments of the gain function ge are Shortest path tree TRe with root node e VT – vector of trap variables

To construct TRe we create a graph G = (VD,ED) where the vertices VD of G correspond to the transitions of

the MT, the edges ED of G represent the pairs of subsequent

transitions sharing a state in MT.


Constructing the gain guards (IV): shortest path tree (example)

Figure 3: The dual graph of the tester model in Figure 2.

Figure 4: The shortest-paths tree (left) and the reduced shortest-paths tree (right) from the transition ec

01 of the graph shown in Figure 3.


Constructing the gain guards (IV): the gain function Represent the reduced tree TR(ei, G) as a set of elementary sub-

trees each specified by the production i j{1,..n}j

Rewrite the right-hand sides of the productions as arithmetic terms:

(3) ti - trap variable ti lifted to type , c - constant for the scaling of the numerical value of the gain

function, d(0, i) the distance between vertices 0 and i, where

l - the number of hyper-edges on the path between 0 and i

w j – weight of j-th hyperedge


Constructing the gain guards (IV): the gain function (continuation)

For each symbol i denoting a leaf vertex in TR(e,G) define a production rule

(4)

Apply the production rules (3) and (4) starting from the root symbol 0 of TR(e,G) until all non-terminal symbols i are substituted with the terms that include only terminal symbols ti and d(0, i)


Constructing the gain guards (IV): the gain function (example)


Constructing the gain guards (IV): the gain function (example continues)


Complexity of constructing and running the tester The complexity of the synthesis of the reactive planning

tester is determined by the complexity of constructing gain functions.

For each gain function the cost of finding the TRE by breadth-first-search is O(|VD| + |ED|) [Cormen], where |VD| = |ET| - number of transitions of MT |ED| - number of transition pairs of MT (is bounded by |ES|2)

For all controllable transitions of the MT the upper bound of the complexity of the computations of the gain functions is O(|ES|3).

At runtime each choice by the tester takes O(|ES|2) arithmetic operations to evaluate the gain functions


Experimental results


Conclusions Efficiency of planning:

number of rules that need to be evaluated at each step is relatively small (number of outgoing transition of current state)

the execution of decision rules is significantly faster than finding the test path by MC but

nevertheless leads to the test sequence that is lengthwise close to optimal.

Documents

Synthesis of Test Purpose Directed Reactive Planning Tester for Nondeterministic Systems 1