Lecture 7: Testing Derivation

Software Quality Assurance (INSE 6260/4-UU)

Winter 2016

2

INSE 6260/4-UU

Software Quality

Assurance

Software Quality

Factors and Models

Metrics

Quality Assurance

Inspection Testing

Techniques Reachability

Analysis

INSE 6260

3

Test Derivation Methods

INSE 6260

4

Transition tour [Nait]

For a given FSM S, a transition tour is a sequence, which takes the FSM S from the initial state, traverses every transition at least once, and returns to the initial state .

• Detects all output errors, • There is no guarantee that all transfer errors can be detected.

Fault detection power

INSE 6260

5

a/x

b/x

b/y

a/x

a/y

b/y

1

2

3

a/x

b/x

b/y

a/x

a/y

b/x

1

2

3

a/x

b/x

b/y

a/x

a/y

b/y

1

2

3

The specification S

A transition tour is : a.a.a.b.b.b

The implementation I1 contains an output error. Our transition tour will

detect it.

The implementation I2 contains a transfer error. Our transition tour will not detect it.

S I1

I2

Transition Tour Example

INSE 6260

6

An input sequence is a distinguishing sequence (DS) for a FSM S, if the output produced by the FSM S is different when the input sequence is applied to each different state. A DS is used as a state identification sequence.

• Detects all output errors, • Detects all transfer errors, • A DS may not be found for a given FSM.

DS-method [Gonenc]

INSE 6260

7

DS method Example

a/x

b/x

b/y

a/x

a/y

b/y

1

2

3

The specification S

S

A distinguishing sequence is : b.b

If we apply it from : • state 1 we obtain y.y • state 2 we obtain y.x • state 3 we obtain x.y

a/x

b/x

b/y

a/x

a/y

b/y

1

2

3

I2

A test case, which allows the detection of the transfer error is :

a.b.b

If we apply it from the initial state of: • the specification we obtain x.x.y • the implementation we obtain x.x.x

Impl.

INSE 6260

8

DS method: Technique 1

a/x

b/x

b/y

a/x

a/y

b/y

1

2

3

Phase 1: Identification of all states/ State cover

From state 1, we can reach state 2 with b/y

and state 3 with a/x

The Q set:

Q = { , a, b}

DS = b.b

Test suite = {r.b.b, r.a.b.b, r.b.b.b}

Phase 2: to cover all transitions for output faults

and transfer faults

The P set:

P = { , a, b, a.b, a.a, b.b, b.a}

Test suite:{r.b.b, r.a.b.b, r.b.b.b, r.a.b.b.b, r.a.a.b.b,

r.b.b.b.b, r.b.a.b.b}

INSE 6260

9

General methodology for state identification based methods

A) Test generation based on Specification

A-1) Find the Q set or the State cover: minimal inputs that reach a state from the initial

one

A-2) Find the P set or Transition cover: that will cover all remaining transitions

Generate Test Suites using Q and P sets

B) Fault detection

B-1) Apply the generated test suites to the specification to obtain Expected Outputs

B-2) Apply the generated test suites to the implementation to obtain Observed Outputs

Compare the expected and observed outputs (test results)

If they are different then the verdict is fail; otherwise it is a pass for the applied test suites.

INSE 6260

10

The test cases are: state 1: Reaching state 3: Reaching state 2: Test case structure: preamble.tested transition.state identification

a.b.b b.b.b a.a.b.b a.b.b.b b.a.b.b b.b.b.b

DS method: Technique 2

INSE 6260

11

The UIO-method can be applied if for each state of the specification, there is an input sequence such that the output produced by the machine, when it is initially in the given state, is different than that of all other states.

The UIOv-method is a variant of the UIO-method. It checks the uniqueness of the applied identification sequences on the implementation, meaning that each identification sequence must be applied on each state of the implementation and the outputs are compared with those expected from the specification.

UIO-Method [Sabnani]

and UIOv-Method [Vuong]

INSE 6260

12

a/x

b/x

b/y

a/x

a/y

b/y

1

2

3

The specification S

S UIO sequences are: • state 1: a.b • state 2: a.a • state 3: a

We assume the existence of a reset transition with no output (r/-) leading to the initial state for every state of S

A transition cover set is: P={e, a, a.b, a.a, b, b.a, b.b}

The test sequences generated by the UIO-method are: r.a.b, r.a.a, r.a.b.a.b, r.a.a.a.a, r.b.a.a, r.b.a.a.b, r.b.b.a

UIO Example

INSE 6260

13

The W-method involves two sets of input sequences: • W-set is a characteristic set of the minimal FSM, and consists of input sequences that can distinguish between the behaviors of every pair of states • P-set is a set of input sequences such that for each transition from state A to state B on input x, there are input sequences p and p.x in P such that p takes the FSM from the initial state into state A.

Method W [Chow]

INSE 6260

14

a/e

a/f

b/f

b/e

c/eb/f

c/e

c/f

a/f

1

3

2

The specification S

We assume the existence of a reset transition with no output (r/-) leading to the initial state for every state of S

A characterization set is W={a, b} •W1 state 1 : a/e, •W2 state 2 : a/f, b/f • W3 state 3 : b/e

W = Union of all Wi

A transition cover set for the specification S is : P={e, a, b, c, b.a, b.b, b.c, c.a, c.b, c.c}

The W-method generates the following test sequences: (P.W) = r.a, r.b, r.a.a, r.a.b, r.b.a, r.b.b, r.c.a, r.c.b, r.b.a.a, r.b.a.b, r.b.b.a, r.b.b.b, r.b.c.a, r.b.c.b, r.c.a.a, r.c.a.b, r.c.b.a, r.c.b.b, r.c.c.a, r.c.c.b

W method Example

INSE 6260

15

This method is a generalization of the UIO method, which is always applicable. It is as the same time an optimization of the W-method. The main advantage of the Wp-method, over the W-method, is to reduce the length of the test suite. Instead of using the set W to check each reached state si, only a subset of W is used in certain cases. This subset Wi depends on the reached state si, and is called an identification set for the state si.

Wp method [Fujiwara]

INSE 6260

16

The specification S

A characterization set is W={a, b} for W method • for state 1 : a/e • for state 2 : a/f, b/f • for state 3 : b/e

We assume the existence of a reset transition with no output (r/-) leading to the initial state for every state of S

The identification sets are : • W1={a}, distinguishes the state 1 from all other states • W2={a, b}, distinguishes the state 2 from all other states • W3={b}, distinguishes the state 3 from all other states

a/e

a/f

b/f

b/e

c/eb/f

c/e

c/f

a/f

1

3

2

state 1 2 3

a e f f

b f f e

Derivation of W

Example of Wp method (1/3)

INSE 6260

17

A state cover set for the specification S is : Q={, b, c}

A transition cover set for the specification S is : P={, a, b, b.c, b.a, b.b, c, c.a, c.c, c.b}

P-Q={a, b.c, b.a, b.b, c.a, c.c, c.b} Based on these sets, the Wp-method yields the following test

sequences :

• Phase 1: Q.Wi = {r.a1, r.b.a2, r.b.b2, r.c.b3} The ending state Wi is given in subscript

• Phase 2 : (P-Q).Wi ={r.a.a2, r.a.b2, r.b.c.a2, r.b.c.b2, r.b.a.a1, r.b.b.b3, r.c.a.b3, r.c.c.a2, r.c.c.b2, r.c.b.a1}

W 1 : { a/e } , W 2 : { a/f, b/f } , W 3 : { b/e }

Example of Wp method (2/3)

INSE 6260

18

a/e

a/f

b/f

b/e

c/eb/f

c/e

c/f

a/f

1 2

3

A faulty implementation I

I contains a transfer error 3-a/f->2 (fat arrow) instead of 3-a/f->3 as defined in the specification S

• The application of the test sequences obtained in Phase 2 leads to the following sequences of outputs :

e.f, e.f, f.f.f, f.f.f, f.f.e, f.f.e, e.f.f, e.e.f,

e.e.f, e.e.e • The output printed in bigger size is different from the one expected according to the specification. Therefore, the transfer error in the implementation is detected by this test sequence.

Example of Wp method (3/3)

INSE 6260

19

Test derivation based on FSM

• Transition tour

– guarantees coverage only for output faults

• Methods using state identification

– with coverage guaranteed for output and transfer faults.

• number of states is the same for implementation I and specification S

• number of states for I is possibly larger than for S, but bounded

• Methods without coverage guarantee

– Hand made test suite without test derivation procedure

INSE 6260

20

Transition tour example

Transition tour

TT: t1, t4, t3, t9, t2, t3, t6, t7, t8

TT (input/expected output): a/1.b/2.a/1.a/2.b/2.a/1.b/2.a/2.b/2

S 1 S 2

S 4 S 3

t1: a/1

t2: b/2 t4: b/2

t3:a/1

t6: b/2

t7: a/2

t8: b/2 t9: a/2

Test hypothesis: Initially connected machine

INSE 6260

21

State identification methods

Distinguishing Sequence, UIO, W, Wp

Test hypothesis

H1) Strongly connected machine

H2) Contain no equivalent states

H3) deterministic

H4) Completely specified machine

H5) the failure which increases the number of states doesn’t occur

The method is applied in two phases from the initial state

phase 1) -sequence to check that each state defined by the specification also

exists in the implementation

phase 2) -sequence to check all the individual transitions in the specification

for correct output and transfer in the implementation

INSE 6260

22

DS Method

S 0 S 1

S 4 S 2

a/0

b/1

a/1

a/0

b/1

b/0

Assume that Reset transition r/- does exist

Q1) Verify if a.a is a DS for S and explain why?

Q2) Find a DS different from “a.a” with length 2 for S

b/0

a/0

INSE 6260

23

W method

Assume that the reset exists and it brings the machine from any state to the initial state.

a) Find characterization set W and generate the set of test cases for the specification S

using the W method.

b) Does S have a DS sequence? If not explain why?

a/0

b/0 b/1

a/1 a/0

S0

S2

S1

b/0

INSE 6260

24

W method

S0 : b/1

S1 : a/1

S2 : a/0, b/0

W= U Wi

W={a, b}

Q = { ,a,b} State Cover

P = { ,a, b, a.b, a.a, b.a, b.b} transition Cover

P-Q = { a.b, a.a, b.a, b.b}

Phase 1 , Q.W = {r.a, r.b, r.a.a, r.a.b, r.b.a, r.b.b}

Phase 2, (P-Q).W = {r.a.b.a, r.a.b.b, r.a.a.a, r.a.a.b, r.b.a.a, r.b.a.b, r.b.b.a, r.b.b.b}

a/0

b/0 b/1

a/1 a/0

S0

S2

S1

b/0

INSE 6260

25

S1

S0

S2

a/0

b/1

a/0 b/0

a/1

b/0

S0 State

Input

Output

a

0

S1 S2

a a

1 0

S0 S1 S2

b b b

1 0 0

S0 S1 S2

a.b a.b a.b

0.1 1.0 0.0

Derive a DS of length up to 2 for S

a.b is a DS for S

Specification S

Comment: “a” as input at each state will loop on the state, sequence of “a.a.” cannot be a DS, the output will

be 0.0.. or 1.1…

Transition tour:

Input

Output

a.b.a.b.a.b

0.1.0.0.1.0

Examples Suite

INSE 6260

26

Q set: permits to reach each state

from the initial state

Q = { , b,b.b}

The first “b” is to reach the state S2

“b.b” is to reach the state S1.

P set is transition cover, permits to execute

each transition at least once starting from the

initial state

S1

S0

S2

a/0

b/1

a/0 b/0

a/1

b/0

S0

a

b b

a

b

b

b

b

b

b

b

a

How to derive P set: find all

Paths starting from the size 1 and up and each transition

should be traversed at least once

P = { , a, b, b.a, b.b, b.b.a, b.b.b}

more than one p set may exist, this depends on the alternative

paths that the automata may have.

INSE 6260

27

The goal of the Phase 1 is identification of

the states in the implementation

DS = a.b, Q = { , b,b.b}, P = {, a, b, b.a, b.b, b.b.a, b.b.b}

Phase 1

Q.DS = {r.a.b, r.b.a.b, r.b.b.a.b} Expected output of phase 1 is:

{-.0.1, -.1.0.0, -.1.0.1.0}

Phase 2 ( DS in bold)

P.DS= {r.a.b, r.a.a.b, r.b.a.b, r.b.a.a.b, r.b.b.a.b, r.b.b.a.a.b, r.b.b.b.a.b}

{-,0.1, -.0.0.1, -.1.0.0.0, -.1.0.1.0, -.1.0.1.1.0, -.1.0.0.0.1}

S1

S0

S2

a/0

b/1

a/0 b/0

a/1

b/0

Note that, the test suites for phase 1 and 2 should be

Derived from the specification and applied to the

implementation to check it for output and

transfer faults

INSE 6260

28

S1

S0

S2

a/0

b/1

a/0 b/0

a/1

b/0

S1

S0

S2

a/0

b/1

a/0

b/0

a/1

b/0

Specification S Implementation I

Apply the transition tour to the implementation I and comment

Transition tour applied to S

Input

Output of S

Output of I

a.b.a.b.a.b

0.1.0.0.1.0

0.1.0.0.1.0

The implementation I has a transfer fault,

the fault is not detected by

Transition tour.

Transition tour detects all output faults but

Doesn’t guarantee the detection of transfer faults

INSE 6260

29

S1

S0

S2

a/0

b/1

a/0

b/0

a/1

b/0

The goal of the Phase 1 is identification of

the states in the implementation

DS = a.b, Q = { , b,b.b}, P = {, a, b, b.a, b.b, b.b.a, b.b.b}

Phase 1

Q.DS = {r.a.b, r.b.a.b, r.b.b.a.b} Expected output of phase 1is:

{-.0.1, -.1.0.0, -.1.0.1.0}

{-.0.1, -.1.0.0, -.1.0.1.0} is the observed outputs from I

Phase 2 ( DS in bold)

P.DS= {r.a.b, r.a.a.b, r.b.a.a.b, r.b.b.a.b, r.b.b.a.a.b, r.b.b.b.a.b}

{-.0.1, -.0.0.1, -.1.0.0.0, -.1.0.1.0, -.1.0.1.1.0, -.1.0.0.0.1} expected output

{-.0.1, -.0.0.1, -.1.0.0.0, -.1.0.1.0, -.1.0.1.1.0, -.1.0.0.0.0} observed output from I, transfer

fault detected

Implementation I

INSE 6260

30

S2

S0

S1

a/0

b/0

a/0 b/0

c/0

a/1

C/0

b/0

c/1

Specification S

State

Input

Output

S0 S1 S2 S0 S1 S2 S0 S1 S2 S0 S1 S2

a a a b b b c c c a.c a.c a.c

0 0 1 0 0 0 1 0 0 0.1 0.0 1.1

Derive a UIO sequence for S

UIO state S0 = c/1

UIO state S2 = a/1

UIO state S1 = a/0.c/0

Transition tour for S

a.b.a.b.c.a.c.b.c

0.0.0.0.0.1.1.0.0