View
215
Download
0
Embed Size (px)
Citation preview
04/18/23 Marius Mikucionis, AAU SSE 1/22
Principles and Methods of Testing Finite State
Machines – A Survey
David Lee, Senior Member, IEEE
and
Mihalis Yannakakis
invited paper
Part about Conformance Testing
04/18/23 Marius Mikucionis, AAU SSE 2/22
Overview• Main concepts: FSM, sequences, equivalence• Conformance testing problem and assumptions• Separating family of sequences• Status messages and reset• Distinguishing sequences• Identifying sequences• Polynomial time randomised algorithm• Heuristic procedures and optimisations• Extensions: one machine, more states, partially
specified
04/18/23 Marius Mikucionis, AAU SSE 3/22
Concepts: FSMFinite State Machine (Mealy machine):• Is a tuple: M=(I, O, S, , ):
– I, O – input and output symbol sets (alphabets),
– S – state set, : SI S – transition function, : SI O – output function.
• Example:– When the machine is in state sS and
receives input symbol aI it moves to state (s, a) and produces output symbol (s, a).
04/18/23 Marius Mikucionis, AAU SSE 4/22
Concepts: sequences and state equivalence
• We extend transition and output functions:– Let x be an input string: x=a1,…,ak, aiI, for i=1..k
– then (s1, x)=sk+1 where si+1=(si, ai), for i=1..k
– and (s1, x)=b1…bn where bi=(si, x), for i=1..k
• x – is a sequence and b1…bn is a response to it.
• Equivalent si, sj S xI*: (si, x) =(sj, x)
• If si and sj are not equivalent then: separating sequence xI*: (si, x) (sj, x)
04/18/23 Marius Mikucionis, AAU SSE 5/22
Machine equivalence
• Equivalent A, B FSM sASA sB SB: sB equivalent to sA
sBSB sA SA: sA equivalent to sB
• There are many equivalent machines• Each equivalent machine class
contains minimized machine M with minimum number of states (all states are unique).
• Minimized machine is unique.
04/18/23 Marius Mikucionis, AAU SSE 6/22
Conformance testing problem• Given:
– Complete information of specification machine A (states, transition and output function)
– Implementation machine B, black box, only I/O is observable
• Goal:– Determine whether B is correct implementation
of (conforms to, is equivalent to) A by applying a test sequence to B and observing the output.
• Checking sequence for machine A with n states is an input sequence x that distinguishes A from any non-equivalent machine B with n states: (sB, x) (sA, x) B is not equivalent to A
04/18/23 Marius Mikucionis, AAU SSE 7/22
Conformance testing assumption
• Specification A is strongly connected– It must be possible to reach all states
• A is reduced (minimized)– We can determine equivalence only to
minimized machine, since equivalent states are not distinguishable.
• B does not change during experiment and has the same input alphabet as A
• B has no more states than A– Assume, faults do not increase number of
states, only:• Wrong output on transition• Wrong state in transition destination
04/18/23 Marius Mikucionis, AAU SSE 8/22
Conformance test structure• Algorithm structure:
– Initialization: move to some known state s1:• If s1 is given verify it (not always possible) • Else apply homing sequence that takes to some known
state s1 (possible for minimised machines)– Verify similarity of B to A– Verify each transition (si, a)=sj :
• Apply sequence that moves machine to si
• Apply a• Verify that machine is in sj
• Methods:– Status and Reset messages– Distinguishing sequences– Identifying sequences– Randomised sequences
04/18/23 Marius Mikucionis, AAU SSE 9/22
Separating family of sequences
• Separating family of sequences for A is a collection of n sets i (one set for each state):– For every pair of states si, sj (sisj) there is
sequence :1) (si, ) (sj, )
2) is a prefix in some xii and a prefix in some xjj
i is called a separating set of state si
• Elements of i are separating sequences of state si
• Examples:– A has a preset distinguishing sequence x then all
sets i may be equal to {x}.– UIO sequences fulfil 1) but may violate 2).
04/18/23 Marius Mikucionis, AAU SSE 10/22
Separating family properties
• General construction algorithm for reduced FSMs:– If A is reduced si,sj x – separating sequence– Partition states into blocks based on (sk, x):
• For every sk put x into Zk
– Repeat procedure for each block until all blocks become singletons.
• Every pair of states has a separating sequence with common prefix.
• Zi contains <n-1 sequences of length n.• Key property: only one state at most gives
the same response to all elements of particular i
04/18/23 Marius Mikucionis, AAU SSE 11/22
State and machine similarity
• State similarity: – qiSB is similar to siSA (qi, x)=(si, x) xi
• Because of key property: qi can be similar to at most one state of A
• Machine similarity:– B is similar to A siSA qiSB qi is similar to si
• All qi are distinct and B has at most n states one-to-one correspondence between A and B states.
04/18/23 Marius Mikucionis, AAU SSE 12/22
Status messages and Reset
• FSM has a reset capability if special input r takes the machine from any state to initial s1.
• If r input is defined for all states of B then reset is reliable.
• Status message tells the current state of machine without changing it.
• Reliable status message guarantees that state will stay the same as before message.
• Unreliable status messages must be applied twice when the state is expected to change.
s
status/[state]
s1r/
r/r/
r/
04/18/23 Marius Mikucionis, AAU SSE 13/22
Test with reliable reset• Let i be a family of separating sets• Build a spanning tree with states in nodes
from diagram of machine A• B similarity to A check:
– For every siSA • for every xZi:
– Reset B to state s1 by applying r– Move to state si according to tree path from s1 to si
– Apply x we are sure that B moved to a state similar to s i
we are sure that B has states similar to A ones, B has no more states than A has, B is similar to
A
04/18/23 Marius Mikucionis, AAU SSE 14/22
Test with reliable reset (2)
• Check all other transitions (si, a)=sj :– For every xZj:
• Reset to s1 by r• Move to state si according to tree path from
s1 to si
• Apply a• Apply x
we are sure that transition (si, a)=sj is OK
B is isomorphic to A
04/18/23 Marius Mikucionis, AAU SSE 15/22
Test with distinguishing sequences
• Let A have an adaptive distinguishing sequence, then Zi={xi}, where xi is a path in decision tree from root to state si
(si, sj) is a pre-computed transition sequence that takes machine from state si to state sj
• ti= (si, xi) – some state after separation• Similarity test:
– x1 (t1, s2)x2 (t2, s3)x3 …xn (tn, s1)x1
we have visited all n states and observed all distinguishing responses to distinguishing sequences
• Transition (si, a)=sj test when in state tk: (tk, si-1)xi-1(ti-1, si) a xj
04/18/23 Marius Mikucionis, AAU SSE 16/22
Test with identifying sequences
• Problem: to verify state si similarity we need to apply separating sequence |Zi| times precisely on the same state.
• Example:– Separating family: Zi={a, b}
– Machine is in state s1
– Apply “aaa”, observe “000”
– B was in q0, moved to q1, q2, q3
– However, at least two of states are the same q3=qi for some i=0..2 we have already applied a
on q3 let’s apply b on it! we have successfully applied a and b on the same
state, B has state q3 similar to state s1 in A.
1
2 3
a/0
a/0a/1
b/1
b/1
b/0
04/18/23 Marius Mikucionis, AAU SSE 17/22
Test with identifying sequences (2)
• A has n states• Similarity to si with separating set Zi={z1, z2}:
– Let qr=(si, z1(ti, si))r ), where ti=(si, z1)– Apply qn
– Then r<n : qr= qn, i.e. we have already applied z1 successfully on qn
– Apply z2
sequence (z1(ti, si))nz2 identifies a state in B which is similar to si
• Transition check: reuse reliable reset idea, because identifying sequences actually reset the state.
04/18/23 Marius Mikucionis, AAU SSE 18/22
Polynomial time randomized test
• Similarity:– For i=1,…,n:
• Repeat ki times:
– Apply sequence that takes to si from current state
– Choose a separating sequence zZi uniformly at random
– Apply z
• Let x be random input formed sequence from similarity test with– ki=O(n |Zi| min(p, |Zi|) log n)
• Then it can be shown that B is similar to A with high probability
04/18/23 Marius Mikucionis, AAU SSE 19/22
Heuristic procedures and optimizations
• Checking sequences guarantee complete fault coverage, but sometimes they are too long.
• Success example: circuit testing is based on faults model significantly limit possible faults.
• Covering paths:– Transition checking with UIO sequences leads to
Postman Tour Problem NP-hard.– Random walk may be “trapped” if system has “narrow
passages” or has just few faults exponential.– Guided random walks records partial history and
makes random choice based on priorities from history.– Test sequences from combinations of sub-sequences:
overlap in sub-sequences.
04/18/23 Marius Mikucionis, AAU SSE 20/22
Summary of algorithmsComplexity is very sensitive to preliminary conditions and requirements to the system:
p=|I| – number of inputs, n=|S| - number of states
Sequences Length Time
Reliable reset pn3 pn3
Distinguishing pn3 pn3
Identifying Exponential Exponential
Randomized Polynomial Polynomial
04/18/23 Marius Mikucionis, AAU SSE 21/22
Conformance test extensions
• One black-box only:– No difference for deterministic test algorithms– Randomized algorithms are more effective on a
single fixed machine: fault probability is squared while test length is doubled
• More states than in specification:– Unknown states and transitions have “combination
lock” property that require exponentially long sequences to cover.
• Partially specified machines:– Special treatment for undefined transitions may
transform machine to fully specified machines.– One fault - exponentially many machines– Only randomised checking has polynomial
complexity
04/18/23 Marius Mikucionis, AAU SSE 22/22
Evaluation and critics• Valuable digest of completely solved
problems.• Good introduction to partially solved ones.• Easy to read: concepts are presented
following the pattern: motivation, definition and simple example.
• Most of algorithms are “encoded” in complex sentences instead of nested blocks – gives motivation, but looses clarity.
04/18/23 Marius Mikucionis, AAU SSE 23/22
Thank you for your attention!
Questions?
04/18/23 Marius Mikucionis, AAU SSE 24/22
Interesting proposition
Let A and B satisfy the assumptions, then the following are equivalent:
1) A and B are isomorphic (there exists a bijection between equivalent A and B states)
2) A and B are equivalent3) At least one state of A has an
equivalent state in B