Upload
meryl-thompson
View
213
Download
0
Tags:
Embed Size (px)
Citation preview
Doron Peled,
Bar Ilan University
Testing of black box finite state machine
Know:
Transition relation
Size or bound on size
Wants to know:
In what state we started?
In what state we are?
Transition relation
Conformance
Satisfaction of a temporal property
Finite automata (Mealy machines)
S - finite set of states. (size n)– set of inputs. (size d)O – set of outputs, for each transition.(s0 S - initial state). S S - transition relation. S O – output on edge.
Why deterministic machines?
Otherwise no amount of experiments would guarantee anything.
If dependent on some parameter (e.g., temperature), we can determinize, by taking parameter as additional input.
We still can model concurrent system. It means just that the transitions are deterministic.
All kinds of equivalences are unified into language equivalence.
Also: connected machine (otherwise we may never get to the completely separate parts).
Determinism
When the black box is nondeterministic, we might never test some choices.
b/1a/1
a/1
Preliminaries: separating sequences
s1
s3
s2
a/0b/1 b/0
b/1
a/0
a/0
Start with one block containing all states {s1, s2, s3}.
A: separate to blocks of states with different output.
s1
s3
s2
a/0b/1 b/0
b/1
a/0
a/0
Two sets, separated using the string b {s1, s3}, {s2}.
Repeat B: Separate blocks based on moving to different blocks.
s1
s3
s2
a/0b/1 b/0
b/1
a/0
a/0
Separate first block using b to three singleton blocks.Separating sequences: b, bb.Max rounds: n-1, sequences: n-1, length: n-1.For each pair of states there is a separating sequence.
State identification: Want to know in which state the
system has started (was reset). Can be a preset distinguishing
sequence (fixed), or a tree (adaptive).
May not exist (PSPACE complete to check if preset exists, polynomial for adaptive).
Best known algorithm: exponential length for preset,polynomial for adaptive [LY].
Sometimes cannot identify initial state… thus need reliable reset
b/1a/1 s1
s3
s2
a/1
b/0
b/1
a/1
Start with a:in case of being in s1 or s3 we’ll move to s1 and cannot distinguish.Start with b:In case of being in s1 or s2 we’ll move to s2 and cannot distinguish.
The kind of experiment we do affects what we can distinguish. Much like the Heisenberg principle in Physics.
Conformance testing Unknown deterministic finite state system B. Known: n states and alphabet . An abstract model C of B. C satisfies all the
properties we want from B. C has m states. Check conformance of B and C. Another version: only a bound n on the number
of states l is known.
Check conformance with a given state machine
Black box machine has no more states than specification machine (errors are mistakes in outputs, mistargeted edges).
Specification machine is reduced, connected, deterministic. Machine resets reliably to a single initial state (or use homing
sequence).
s1
s3
s2
a/1
b/0
b/1
a/1
?=
a/1
b/1
Conformance testing [Ch,V]
a/1
b/1
Cannot distinguish if reduced or not.
a/1
b/1
a/1
b/1
a/1
b/1a/1
b/1
Conformance testing (cont.)
ab b
a
a
a
a b
b
b
a
Need: bound on number of states of B.
a
Preparation:Construct a spanning tree
b/1a/1 s1
s3
s2
a/1
b/0
b/1
a/1
s1
s2s3
b/1a/1
How the algorithm works?
According to the spanning tree, force a sequence of inputs to go to each state.
1. From each state, perform the distinguishing sequences.
2. From each state, make a single transition, check output, and use distinguishing sequences to check that in correct target state.
s1
s2s3
b/1a/1
Rese
t
Rese
t
Distinguishing sequences
Comments
1. Checking the different distinguishing sequences (n-1 of them) means each time resetting and returning to the state under experiment.
2. We assume a reliable reset.3. Since, by assumption, there are no
more than n states, and according to the experiment, no less than n states, there are n states exactly.
4. Isomorphism between the transition relation is found.
Combination lock automaton
Assume accepting states.Accepts only words with a specific suffix
(cdab in the example).
s1 s2 s3 s4 s5
bdc a
Any other input
When only a bound on size of black box is known…
Black box can “pretend” to behave as a specification automaton for n-l states, then upon using the right combination of size l, make a mistake.
b/1a/1s1
s3
s2
a/1
b/0
b/1
a/1
b/1
Pretends to be S3
Pretends to be S1
for n -l states.
a/1
Conformance testing algorithm [VC] The worst that can happen is a
combination lock branch that mimics usual behavior until the last state. The length of it is the difference between the bound n on the size of the black box and its actual size l.
Reach every state on the spanning tree and check every word of length n-l+1 or less. Check that after the combination we are at the state we are supposed to be, using the distinguishing sequences.
No need to check transitions: already included in above check.
Complexity: m2 n dn-l+1
Probabilistic complexity: Polynomial.
Distinguishing sequences
s1
s2s3
b/1a/1
Words of length n-l+1
Rese
t
Rese
t
Model Checking
Finite state description of a system B. LTL formula . Translate into an automaton P. Check whether L(B) L(P)=. If so, S satisfies . Otherwise, the intersection
includes a counterexample. Repeat for different properties.
Buchi automata P (-automata)
S - finite set of states. (Black box B has l n states)
S0 S - initial states. (P has m states) - finite alphabet. (contains p letters) S S - transition relation. F S - accepting states.Accepting run: passes a state in F infinitely
often.System automata: F=S, deterministic, one initial state.
Property automaton: not necessarily deterministic.
Example: check a
a, aa
a<>a
Example: check <>a
a
a
a
a
<>a
Example: check <>a
Use automatic translation algorithms, e.g., [Gerth,Peled,Vardi,Wolper 95]
a
a
a, a<>a
System
c b
a
Every element in the product is a counter example for the checked property.
c b
a
a
a
a
a
s1 s2
s3 q2
q1
s1,q1
s1,q2 s3,q2
s2,q1a
b
c
aAcceptance isdetermined byautomaton P.
<>a
Model Checking / Testing
Given Finite state system B.
Transition relation of B known.
Property represent by automaton P.
Check if L(B) L(P)=. Graph theory or BDD
techniques. Complexity:
polynomial.
Unknown Finite state system B.
Alphabet and number of states of B or upper bound known.
Specification given as an abstract system C.
Check if B C. Complexity: polynomial
if number states known. Exponential otherwise.
Black box checking [PVY]
Property represent by automaton P.
Check if L(B) L(P)=.
Graph theory techniques.
Unknown Finite state system B.
Alphabet and Upper bound on Number of states of B known.
Complexity: exponential.
Experiments
aa
bb cc
reset
a
a
b
b
c
c
try ba
a
b
b
c
c
try c
fail
Simpler problem: deadlock?
Nondeterministic algorithm:guess a path of length n from the initial state to a deadlock state.Linear time, logarithmic space.
Deterministic algorithm:systematically try paths of length n, one after the other (and use reset), until deadlock is reached.Exponential time, linear space.
Deadlock complexity
Nondeterministic algorithm:Linear time, logarithmic space.
Deterministic algorithm:Exponential (p n-1) time, linear space.
Lower bound: Exponential time (usecombination lock automata).
How does this conform with what we know about complexity theory?
Modeling black box checking
Cannot model this problem using Turing machines: not all the information about B is given. Only certain experiments are allowed.
We learn the model as we make the experiments.
Can use the model of games of incomplete information.
Games of incomplete information
Two players: player, player (here, deterministic). Finitely many configurations C. Including:
Initial Ci , Winning : W. An equivalence relation on C (the player cannot
distinguish between equivalent states). Labels L on moves (try a, reset, success, fail). The player has the same labels on moves from
configurations that are equivalent (it does not distinguishes between them).
Deterministic strategy for the player: will lead to a configuration in W. Cannot distinguish between equivalent configurations.
Nondeterministic strategy: Can distinguish between equivalent configurations.
Modeling BBC as games
Each configuration contains an automaton and its current state (and more).
Moves of the player are labeled withtry a, reset... Moves of the -player withsuccess, fail.
c1 c2 when the automata in c1 and c2 would respond in the same way to the experiments so far.
A naive strategy for BBC
Learn first the structure of the black box. Then apply the intersection. Enumerate automata with n states
(without repeating isomorphic automata). For a current automata and new
automata, construct a distinguishing sequence. Only one of them survives.
Complexity: O((n+1)p (n+1)/n!)
On-the-fly strategy Systematically (as in the deadlock
case), find two sequences v1 and v2 of length <=m n.
Applying v1 to P brings us to a state t that is accepting.
Applying v2 to P brings us back to t. Apply v1 v2
n-1 to B. If this succeeds,there is a cycle in the intersection labeled with v2, with t as the P (accepting) component.
Complexity: O(n2p2mnm).
v1
v2
Learning an automaton
Use Angluin’s algorithm for learning an automaton.
The learning algorithm queries whether some strings are in the automaton B.
It can also conjecture an automaton Mi and asks for a counterexample.
It then generates an automaton with more states Mi+1 and so forth.
Angluin’s algorithm in a nutshell
T 0 1 0 0 0 1 0 1 01 0 0 00 0 0 10 0 0 11 0 0 010 0 1 011 1 0
Access strings Z(prefix closed)
Z.
Distinguishing sequences(suffix closed)
Data structures of table are related to spanning tree + distinguishing sequences in [VC]!!
Angluin’s algorithm in a nutshell
T 0 1 0 0 0 1 0 1 01 0 0 00 0 0 10 0 0 11 0 0 010 0 1 011 1 0
Access strings Z(prefix closed)
Z.
Distinguishing sequences(suffix closed)
A state: an equivalence class of rows with same values.
Angluin’s algorithm in a nutshell
T 0 1 0 0 0 1 0 1 01 0 0 00 0 0 10 0 0 11 0 0 010 0 1 011 1 0
Access strings Z(prefix closed)
Z.
Distinguishing sequences(suffix closed)
Closed: each row for Z. exists already for Z.
In this example, this is not closed: no row with results like 011.
Add row from Z.into Z. Update rows.
Angluin’s algorithm in a nutshell
T 0 1 0 0 0 1 0 1 01 0 0 00 0 0 10 0 0 11 0 0 010 0 1 011 1 0
Access strings Z(prefix closed)
Z.
Distinguishing sequences(suffix closed)
Consistent: successors of equivalent rows are equivalent.
Since 001 we also expect 00010
This does not hold; add distinguishing sequence 01 to separate. Update columns.
A strategy based on learning
Start the learning algorithm.Queries are just experiments to B.For a conjectured automaton Mi ,
check if Mi P =
If so, we check conformance of Mi with B ([VC] algorithm).
If nonempty, it contains some v1 v2 .
We test B with v1 v2n. If this succeeds: error,
otherwise, this is a counterexample for Mi .
Complexity
m - size of the Buchi automaton. l - actual size of B. n - an upper bound of size of B. d - size of alphabet. Lower bound: reachability is similar to
deadlock. O(l 3 d l + l 2mn) if there is an error. O(l 3 d l + l 2 n dn-l+1+ l 2mn) if there is no
error.If n is not known, check while time allows. Probabilistic complexity: polynomial.
Some experiments
Basic system written in SML (by Alex Groce, CMU).
Experiment with black box using Unix I/O.
Allows model-free model checking of C code with inter-process communication.
Compiling tested code in SML with BBC program as one process.
“Adaptive Model Checking”[Groce, P, Yannakakis, 2002]
What happens if we need to check a black box that is a variant of an old design?
Some changes are made, some parts remain.
Should we use “black box checking” from scratch?
Or can we use some elements that were already computed and tested?
“Adaptive model checking”
Instead of starting BBC from scratch, we can use from the old table:
Rows: these are the access strings of a spanning tree.
Columns: these are the separating sequences.
So: make the experiments first based on the rows and columns of the previous design, assuming that not so much was changed.
Rows don’t help when the system was prefixed by a password (i.e., a combination lock…).
“Grey box checking”[Elkind, Genest, P, Qu, 2006]
What happens when the system is a product A B, where A is known and B is black box?
Case 1: we can still experiment with B separately. Then it makes sense to do the learning separately on B, then do model checking. The model checking of the counterexample of the combined system, if exists, is projected on B for testing it.
“Grey Box Checking”Case 2. We cannot test B separately.Then consider a case where using VC
algorithm will be exponential in the multiplication m n of sizes of A and B.
In this case, the naïve strategy (running all automata, and comparing pairs, each time removing one of them) has a lower complexity (exponential in p n log(n)).
Conclusions
Black Box Checking: automatic verification of unspecified system.
A hard problem, exponential in number of states, but polynomial on average.
Implemented, tested. Another use: when the model is given, but
is not exact. (This has also been applied for
compositional model checking.)