19
MDSE – Model Driven Software Engineering Model-based Testing 1 Model Driiven Software Engineering MAP-I, Ana Paiva, 11/11/2008 Model-based Testing Ana Paiva [email protected] www.fe.up.pt/~ apaiva Agenda Introduction Model-based testing • Model your system • Test coverage criteria and Test case generation Test case execution 2 Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008 Test case execution A model-based testing tool Research Project Testing Testing is the activity performed for evaluating a product quality, and for improving it, by identifying defects and problems [SWEBOK 2004] …consists of the dynamic verification of the behaviour of 3 Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008 …consists of the dynamic verification of the behaviour of a program on a finite set of test cases, suitably selected from the usually infinite executions domain, against the expected behaviour Testing executes software in order to detect failures and then tries to identify and fix the faults that cause them Fault vs Failure By definition, a fault is a structural imperfection in a software system that may lead to the system's eventually failing A fault is the cause of a failure. It is the execution of the faults in software that causes failures 4 Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008 faults in software that causes failures Tester look for failures and then try to identify the faults which cause those failures.

Model-based Testing

Embed Size (px)

Citation preview

MDSE – Model Driven Software Engineering

Model-based Testing

1Model Driiven Software Engineering MAP-I, Ana Paiva, 11/11/2008

Model-based Testing

Ana Paiva

[email protected] www.fe.up.pt/~apaiva

Agenda

� Introduction

� Model-based testing

• Model your system

• Test coverage criteria and Test case generation

• Test case execution

2Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

• Test case execution

� A model-based testing tool

� Research Project

Testing� Testing is the activity performed for evaluating a

product quality, and for improving it, by identifying

defects and problems [SWEBOK 2004]

� …consists of the dynamic verification of the behaviour of

3Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� …consists of the dynamic verification of the behaviour of

a program on a finite set of test cases, suitably selected

from the usually infinite executions domain, against the

expected behaviour

� Testing executes software in order to detect failures and

then tries to identify and fix the faults that cause them

Fault vs Failure

� By definition, a fault is a structural imperfection in a software

system that may lead to the system's eventually failing

� A fault is the cause of a failure. It is the execution of the

faults in software that causes failures

4Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

faults in software that causes failures

� Tester look for failures and then try to identify the faults

which cause those failures.

Software quality properties

� External Properties: The user’s perspective:

• Satisfaction: subjective view (pleasant, comfortable, intuitive, consistent)

• Reliable: refers to the errors a user can do when using the system

• Learnability: time taken to learn how to use the system

• Efficiency: how efficient a user can be when using the system

5Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� Internal Properties: The software engineering’s perspective:

• Code: readable, style,…

• Architecture: influences the degree of manageability and scalability

• Run time efficiency: related to the complexity of the algorithms

• Correctness: Verification (meets specification) and validation (meets users’ requirements)

[GC96]

General test (V&V) strategies

� Manual vs automatic

� Static vs dynamic

� White vs Black-box

� Error/Fault based vs specification-based

6Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� Error/Fault based vs specification-based

� API vs UI testing

� Functional vs non-functional (Usability, Performance,

Robustness)

Different kinds of tests: V-Model

7Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

Model-based testing

Agenda

� Introduction

� Model-based testing

• Model your system

• Test coverage criteria and Test case generation

• Test case execution

8Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

• Test case execution

� A model-based testing tool

� Research Project

Model-based testing

� Model-based testing is software testing in which test cases are derived in whole or in part from a model that describes some (usually functional) aspects of the system under test (SUT)

[UL07]

9Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� Model-based testing uses a model program to generate test cases or act as an oracle. Model-based testing includes both

• offline testing and

• on-the-fly (online) testing

� An oracle is the authority which provides the correct result used to make a judgment about the outcome of a test case - whether the test passed or failed

[A-site]

Why do we need MBT?� Unit testing is not enough

� We need meaningful sequences of actions calls that provide systematic behavioral coverage

10Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� In software industry model-based testing is being adapted as an integrated part of the testing process

� In academia several model-based testing conferences have been started

� Several commercial tools have been developed for model-based testing

Model-based testing� Advantages

• Higher degree of automation (test case generation)

• Allows more exhaustive testing

• Good for correctness/functional testing

• Model can be easily adapted to changes

Disadvantages

11Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� Disadvantages

• Requires a formal specification/model

• Test case explosion problem

• Test case generation has to be controlled appropriately to

generate a test case of manageable size

• Small changes to the model can result in a totally different test

suite

• Time to analyse failed tests (model, SUT, adaptor code)

Model-based testing process

system under test (SUT)

sw model1.Modelling

2.Test case generation

Abstract Test suite

Map

Concrete 3.Test case

Input data;

Outputs expected (oracle)

Requirements traceability matrix; model coverage

SUT coverage

12Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

4.Test case execution

Bugs report

Concrete Test suite

3.Test case concretization

SUT coverage

5. Analysis

Model-based testing challenges

system under test (SUT)

sw model1.Modelling

2.Test case generation

Abstract Test suite

Map

Concrete 3.Test case

Effort & guidelines

Test case explosion

Test adequacy

13Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

4.Test case execution

Bugs report

Concrete Test suite

3.Test case concretization

5. Analysis

Model to implementation gap

Model your system

� Design the model to meet your testing goals

� Choose the right level of abstraction (which aspects of SUT

you want to test)

� You can have a many-to-many relationship among the

operations of your model and the operations of the SUT

14Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

operations of your model and the operations of the SUT

� Choose a notation for modelling

� Once the model is written, ensure that it is accurate (validate

and verify your model)

[UL07]

Models: Built or Borrow?

� Reuse the development models – 100% reuse

• Too much detail

• Usually do not describe the dynamic behaviour of the SUT

• It is neither abstract enough nor precise

� Models to generate code or reverse engineered

15Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� Models to generate code or reverse engineered

• Lack of independence (implementation and test cases are derived from

the same source code)

� Develop test models from scratch – 0% reuse

• Maximum level of independence

[UL07]

Notations

� Pre/Post (or Model-based). Ex.: VDM, Z, Spec#.

� Transition-based. Ex.: FSM, Petri nets.

� Behavior-based (or history-based). Ex.: CSP, MSC.

16Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� Property-based (or functional-based). Ex.: OBJ.

� Hybrid approaches. Ex.: RAISE.

� …

� UML with OCL?

• pre/post, Set, OrderedSet, Bag, Sequence, and associations among

classes

Pre/Post: VDM++class Stack instance variablesstack : seq of int := []; -- inv

operationsStack : () ==> () Stack () == stack := []post stack = []; Push : int ==> ()Push(i) == stack := [i] ^ stack

17Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

Push(i) == stack := [i] ^ stackpost stack = [i] ^ ~stack; Pop : () ==> ()Pop() == stack := tl stackpre stack <> [] post stack = tl ~stack; Top : () ==> intTop() == return (hd stack)pre stack <> [] post RESULT = hd stack

and stack = ~stack; end Stack

Transition-based: FSM

� Choosing the set of states is a critical step

Len s = 0 Len s != 0

Push(k)

Pop

Pop

18Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� Statecharts are another option

Behaviour-based: CSP

Stack(< >) = push?x:E -> Stack(<x>)

Stack(<y>^s) = push?x:E->Stack(<x>^<y>^s)

| pop!y -> Stack(s)

19Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

OR

Stack() = push?x;Stack(x)

Stack(x) = push?y; Stack(y);Stack(x)

| pop!x;Stack()

Property-based: OBJSpec: Stack;

Extend Nat by

Sorts: Stack;

Operations:

newstack: → Stack

push: Stack × Nat → Stack

pop: Stack → Stack

20Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

pop: Stack → Stack

top: Stack → Nat

Variables:

s: Stack; n: Nat

Axioms:

pop(newstack) = newstack;

top(newstack) = zero;

pop(push(s,n)) = s;

top(push(s,n)) = n;

Choosing a notation

� Pre/Post (for modelling complex data) and transition-based

(for modelling control) are the most common notations used in

model-based testing processes

Whatever notation you choose, it has to be a formal language

21Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� Whatever notation you choose, it has to be a formal language

with precise semantics in order to write accurate models so as

to be used as test oracles

Agenda

� State of the art

� Model-based testing

• Model your system

• Test coverage criteria and Test case generation

• Test case execution

22Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

• Test case execution

� A model-based testing tool

� Research Project

Test suite

� Key Point – In this context, a test suite contains the sequence

of actions to perform, the input data and the results expected

The best test suit is the smallest one which is capable of

23Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� The best test suit is the smallest one which is capable of

finding the maximum number of bugs.

� A good test suite should combine a good code coverage with a

good requirements (or specification) coverage

Coverage criteria and analysis

� Coverage criteria help you generate a good test suit and determine when to stop testing but, above all, your expert knowledge of the SUT is the key factor for success

� Coverage analysis aims to measure the extent to which a

24Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� Coverage analysis aims to measure the extent to which a given verification activity has achieved its objectives and can be used to evaluate the quality of the test suite used and also determine when to stop the verification process. It is usually expressed as a percentage referring to the accomplished part of an activity

[UL07]

Coverage criteria

� 1.Structural coverage criteria – aims to exercise code or model concerning some coverage goal

� 2.Data coverage criteria – aims to cover the input data space of an operation or transition

� 3.Fault-based criteria – aims to generate test suites

25Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

3.Fault-based criteria – aims to generate test suites appropriate for detecting specific kinds of faults

� 4.Requirements coverage criteria – aims to ensure that each requirement will be tested

� 5.Others

[UL07]

1.Structural coverage criteria

� Statement coverage (SD) – every executable statement is invoked at least once;

� Decision coverage (DC) – expression outcomes are tested for true and false (e.g., (A or B) tested for TF and FF);

� Condition coverage (CC) – each condition within expression takes all possible outcomes (e.g., (A or B) tested for TF and FT);

26Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� Decision/condition coverage (D/CC) – combines the two previous criteria (e.g., (A or B) tested for TT and FF);

� Modified condition/decision coverage (MC/DC) – each condition affects independently the outcome of the decision (e.g., (A or B) tested for TF, FT, and FF);

� Multiple condition coverage (MCC) – test each possible combination of inputs. Test 2n for a decision with n inputs (almost of the times is unfeasible);

� … [HVCKR01]

1.Structural coverage criteria

Multiple Condition Coverage (MCC)

Modified Condition/Decision Coverage (MC/DC)

Full Predicate Coverage (FPC)

strongest+

27Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

Full Predicate Coverage (FPC)

Decision/Condition coverage (D/CC)

Decision converage (DC) Condition Coverage(CC)

Statement Coverage (SC) [UL07] -

1.Structural coverage criteria: FSM

All-Paths

All-One-Loop-Paths All-Transition-Pairs

strongest+

28Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

All-Round-Trips All-Configurations

All-Loop-Free-Paths All-Transitions

All-States

[UL07]

-

2.Data coverage criteria

� Choose variables’ values from their domains

•One value (almost of the times not enough)

• Smart choice of values

- Equivalence class partitioning (e.g., type-based)

- Boundary value analysis

Does not generate a test sequence of actions

29Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

- Boundary value analysis

- Randomly value generation

- Goal oriented methods (e.g., AI Planning)

- Path oriented methods

• All values (almost of the times unfeasible)

[UL07]

actions

2.Equivalence class partitioning

� A partition of some set, S, is a set of non-empty subsets

SS1, ..., SSn, such that each SSi and SSj are disjoint, and

the union of all SSi's equals S

30Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� If a defect is detected by one member of a class, it is

expected that the same defect would be detected by any

other element of the same class

all inputs

i1

i4i2 i3

2.Boundary value analysis

� Boundary value analysis tests boundary conditions of equivalence classes choosing input boundary values. This technique is based on the knowledge that input values at the boundaries or just beyond the boundaries of the input domain tend to cause errors in the system

31Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

• Test cases :

- class x < 0, arbitrary value: x = -10

- class x >= 0, arbitrary value x = 100

- classes x < 0, x >= 0, on boundary : x = 0

- classes x < 0, x >= 0, below and above: x = -1, x = 1

“Bugs lurk in corners and congregate at boundaries.”

[Boris Beizer, "Software testing techniques"]

2.Goal oriented

� Goal-oriented methods try to drive the system into a

given goal by two different methods: the chaining

approach and assertion-oriented approach

• The first one tries to find a path to the execution of a given goal

node based on data dependence analysis

32Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

• The second tries to find any path to an assertion that does not

hold

2.Path oriented

� E.g., Symbolic testing. It replaces program variables by symbols and

calculates constraints that represent possible symbolic execution paths.

When a program variable is changed during execution, the new value is

expressed as a constraint over the symbolic variables. A constraint solver

system can be used to find, when possible, concrete values that cause the

execution of the path described by each constraint

int x, y;

33Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

int x, y;

if (x > y) {

x = x + y;

y = x – y;

x = x – y;

If (x - y > 0)

assert false

}

x = X, y= Y

X > ? Y

[X<=Y] END [X>Y] x = X + Y

[X>Y]y = X+Y-Y =X

[X>Y] x = X+Y-X = Y

[X>Y] Y-X >? 0

[X>Y, Y-X <= 0] END [X>Y, Y-X>0] END

3.Fault-based criteria

� Mutation testing - Mutation techniques introduce small changes (faults) by applying mutation operators into the original specification (e.g., arithmetic operators

� The changed specifications are called mutants

� The goal is to construct test cases that distinguish each mutant from the original by producing different results. If that happens, it is said that the test case has killed the mutant

34Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� A good test case should be capable of killing the mutants because if it is able to detect the small differences generated by the mutation operators it is expectable that it will be good at finding real faults

� The rate of killed mutants (after removing mutants that are equivalent to the original code) gives an indication of the rate of undetected defects that may exist in the original code

� One of the problems of mutation testing is the incapacity of the technique to generate test data

[W03]

4.Requirements coverage criteria

� Requirements coverage is usually less systematic and usually those not contain the full specification of the behaviour of the system. However, there are at least two approaches that try to systematize it more:

• Record the requirements inside the behaviour model (as annotations on several parts of the model) so that the test generation process can ensure that all requirements have been tested

35Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

ensure that all requirements have been tested

• Formalize each requirement and then use that formal expression as a test selection criterion to drive the automated generation of one or more tests from the behaviour model

� Ex.: Coverage criterion measuring the degree in which use cases or scenarios were tested. Scenarios describe how the system and the user should interact to achieve a specific goal. They usually refer to common usages of the system and may not be a full description of the behaviour of the system

5.Model-checking� Whenever a property, expressed in temporal logic, does not hold in

a system described as a FSM, model-checking tries to generate a counter-example

� When a counter-example is produced, it can be used as a test case –sequence of transitions in the FSM with inputs and expected outputs

� To be effective as a test-case generation technique, the properties about the system should be described in such a way that counter-

36Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

about the system should be described in such a way that counter-examples produced by them can be used as test cases

ModelChecker

Property: G(x -> F y)

yesno

x = T,T,F,F,...y = F,F,F,T,...

System model (FSM)

[ABM98]

5.Test case generation from property-based models

� Common techniques used to generate test cases from

these specifications are rewriting and constraint solving

� Given a set of expressions (logical assertions or

37Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

Given a set of expressions (logical assertions or

equivalence relations) and the set of variables within

those expressions, constraint solving techniques try to

find an instantiation of the variables which reduce the

expressions to true - E.g., [X>Y, Y-X>0] = impossible!

5.Test case generation from behavioural-based models

� Analyses of the execution traces to generate test cases.

A trace in CSP is a finite sequence of events

• push?x;push?y;pop!x;push?x

Another example of test case generation from CSP

38Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� Another example of test case generation from CSP

specifications is illustrated in [BS02]. The goal is to test

Universal Mobile Telecommunications Systems (UMTS).

They start by constructing a transition graph with all

possible interleaving and parallel tasks. Then, a test

driver computes all paths through this graph that are

used as test sequences

Test case generation overview

� Depends on the specification characteristics:

•Pre/Post. Ex.: VDM, Z, Spec#- Partition testing- Generate FSM from the model. Ex.: Spec Explorer.

•Transition based. Ex.: FSM, EFSM, Statecharts.- Traversal algorithms (state and transition coverage)

39Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

- Traversal algorithms (state and transition coverage)- Model checking (algorithms based on state exploration; mutation analysis)

•Property-based. Ex.: Obj- Rewriting rules- Constraint solving

•Behaviour-based. Ex.: CSP- Trace analysis

Agenda

� State of the art

� Model-based testing

• Model your system

• Test coverage criteria and Test case generation

• Test case execution

40Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

• Test case execution

� A model-based testing tool

� Research Project

Test suite “concretization”

Test Cases

Adapter

abstract

Test Cases

Test Cases

Level of abstractio

41Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

Adapter

SUTconcrete

SUT

Adapter

SUT

Test Scripts

Test Scripts

a) Adaptation b) Transformation c) Mixed

abstraction

[UL07]

Test suite execution � “Lock-step” mode in which results are compared after each step

� “Batch-oriented” way, in which case the test suite is run as a whole in the specification level, and expected results are kept in memory for later comparison with the results obtained from the execution of the implementation (which is performed in a different execution time instant)

• One advantage of the batch-oriented way is the need to execute

42Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

• One advantage of the batch-oriented way is the need to execute the model only once and not every time test cases are executed. The main drawback is the additional need of memory to keep the results expected

� "On-the fly testing" combines in a single algorithm the test case generation and execution and executes each operation as a “lock-step” in each level comparing results after each of those execution steps

Agenda

� State of the art

� Model-based testing

• Model your system

• Test coverage criteria and Test case generation

• Test case execution

43Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

• Test case execution

� A model-based testing tool: Spec Explorer

� Research Project

System Testing with model programs

System Design

System Test

Spec Explorer

44Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

Microsoft Research (FSE) : Colin Campbell, Wolfgang Grieskamp, Microsoft Research (FSE) : Colin Campbell, Wolfgang Grieskamp, Yuri Gurevich, Lev Nachmanson, Wolfram Schulte,Yuri Gurevich, Lev Nachmanson, Wolfram Schulte,Nikolai Tillmann, Margus VeanesNikolai Tillmann, Margus Veanes

Model-Based Testing

Model

Pass

Provides expected results forGenerates

User

45Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

Implementation

Test OracleTest Cases

Are run by

Providesactual results for

PassFail

UserInfo

System testing� Traditional unit testing is not enough

• Single unit may work properly in isolation

• Incorrect interaction between units may cause serious

security/reliability failures

� System-level testing

46Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

• Requires model of system behavior

• Behavior is often reactive/nondeterministic

- Implementation is multi-threaded or distributed

• State space is typically infinite

- Objects, unbounded values

• Traditional FSM-based testing does not work

Microsoft approach

� Behavior is described by model programs

• Written in Spec#

• Describe reactive system behavior as interface automata

• Use alternating refinement for conformance

47Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� Model exploration, validation and model-based testing are provided by the Spec Explorer tool developed at MSR

• Supports scenario control

• Provides offline as well as online testing

• Views test cases as game strategies

• Checks violations of alternating refinement

[CGNSTV05]

How it works

� Modeling

• Define (infinite) transition system TP through program P

� Exploration

• Reduce TP to a finite test graph G

Test generation

48Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� Test generation

• Generate test cases from G

� Test execution

• Run the test cases using the model as the oracle

To avoid state space explosion, exploration, test generation and execution can be combined into a single on-the-fly algorithm

Modelling

� In Spec#/Spec Explorer:

• States are mappings of variables to values - ASM states or first-order structures

• Initial state is given by initial assignment to model variables

• Actions are defined by method invocations

• Preconditions of methods and model invariants define action enabling conditions

49Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

conditions

• Transition function is defined by method execution

Exploration

� Exploration is the process of unfolding a rule-based model

program into a transition system

• Actions move the system from state to state

• State comes from variables containing values

• Compound data types are possible (sets, maps, sequences, etc…)

• Objects are identities

50Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

• Objects are identities

State space of a stack model

� Stack model

var Seq<int> content = Seq{};var Seq<int> content = Seq{};var Seq<int> content = Seq{};var Seq<int> content = Seq{};

[Action][Action][Action][Action]public void Push(int x) {public void Push(int x) {public void Push(int x) {public void Push(int x) {content = Seq{x} + content;content = Seq{x} + content;content = Seq{x} + content;content = Seq{x} + content;

}}}}

[Action][Action][Action][Action]

[]

[0] [k]

Push(0) Push(k)Pop PopTop/0

Top/k

51Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

[Action][Action][Action][Action]public void Pop() {public void Pop() {public void Pop() {public void Pop() {requires !content.IsEmpty;requires !content.IsEmpty;requires !content.IsEmpty;requires !content.IsEmpty;content = content.Tail;content = content.Tail;content = content.Tail;content = content.Tail;

}}}}

[Action] [Action] [Action] [Action] public int Top() {public int Top() {public int Top() {public int Top() {requires !content.IsEmpty;requires !content.IsEmpty;requires !content.IsEmpty;requires !content.IsEmpty;return content.Head;return content.Head;return content.Head;return content.Head;

}}}}

[0] [k]

[0,k] [l,k]

PopPush(0) Push(l)

Pop

Top/0Top/l

Controlling exploration - 1

� In general the transition system is infinite, but we can impose limits

� Goal

• Create a state space of manageable size that satisfies a given testing goal

52Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� Two main tasks

• Restrict action parameter domains to interesting values

• Restrict the state space to interesting states

� Note: The two tasks are not necessarily independent!

Controlling exploration - 2

� The following techniques are used in Spec Explorer

• State filters

• Stopping conditions

• State groupings

• Search order

• Enabling conditions on actions

53Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

• Enabling conditions on actions

� Usually a combination of all (or some) of the

techniques is needed to achieve desired effect

State Groupings

� One or more state groupings may be defined

� A grouping G is a sequence of state based expressions g1,…,gk

� Two states s and t are in the same group G or G-equivalent if

54Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

G-equivalent if

• gis = gi

t for 1 ≤ i ≤ k

� A G-group is the set of all G-equivalent states

� Similar to predicate abstraction in model-checking (if grouping expressions are Boolean)

� Also used in viewing

Main purpose of groupings

� View all states in a G-group as being the same

� A way to define “what is an interesting state” from some

testing point of view

55Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� Example: content.Size in the stack model

Len s = 0 Len s != 0

Push(k)

Pop

Pop

Test case generation

_

FSM generation by

bounded exploration

Bounds

FSM

SUT (Spec# or AsmL)

56Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

Test suite

Coveragecriteria

Test casegeneration

Spec Explorer

Set bounds: state filters; additional pre-conditions; restriction of the domains; equivalence classes; stop conditions; scenario actions

Choose coverage criteria: full transition coverage; shortest path; random walk

Test generation

� The result of exploration is a finite transition system

� This connects us to the world of model-based testing

• Most formal approaches to testing in the literature are based on

transition systems

� Traversal underlies most techniques for generating tests from

57Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� Traversal underlies most techniques for generating tests from

automata

Test execution

� Generated test cases are executed in the implementation

under test

� The model acts as the oracle

58Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

Demo: stack

[]

[0] [k]

Push(0) Push(k)

Push(0) Push(l)

Pop PopTop/0Top/k

Len s = 0 Len s != 0

Pop

Pop

59Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

[0,k] [l,k]

PopPush(0) Push(l)

Pop

Top/0Top/l

Len s = 0 Len s != 0

Push(k)

Online Testing of Reactive systems

Constrained model

IUT wrapper

IUT

Refinement checker

Controllable actions

Controllable actions

Observableactions

Observable actionsd1

……c1

Object mapping

Component View

60Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

Model

Pass/Fail

d1……

c1

Example: Chat service

IUT wrapper

IUT

Chat

Controllable actions:

Login …SendRefinement checker

Constrained chat model

b.Send(“hi”)

m.Send(“hello”)

b.Send(“bye”Object mapping

61Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

at server

Observable actions: Receive

b.Send(“bye”)

m.Rcv(“hi”)

m.Rcv(“bye”)

b.Rcv(“hello”)b.Rcv(“hello”

)

m.Rcv(“bye”)

m.Rcv(“hi”)

b.Rcv(“hello”)

BobMarym

b

mapping

Refinement relation

� Conformance between the model and the system under test is

alternating refinement

b.Send(“hi”)

Spec System

b.Send(“hi”)

62Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

m.Send(“hello”) m.Send(“hello”)

m.Receive(b,“hi”)

b.Receive(m,“hello”)b.Receive(m,“hello”)

m.Receive(b,“hi”) b.Receive(m,“hello”)

Example: Valid chat scenario

Bob Server Mary

Service should guarantee local consistency of message delivery (FIFO wrt sender)Send is a controllable action

Receive is an observable action

63Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

Bob Server Mary

Time

Model program : Chat service

State:

Actions:

class Client { bool entered = false;Map<Client, Seq<string>> unreceivedMsgs = Map{};}

class Client {void Send(string message)requires entered; { foreach (Client c in enumof(Client), c != this, c.entered)

Controllable:Controllable:

64Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

foreach (Client c in enumof(Client), c != this, c.entered)c.unreceivedMsgs[this] += Seq{message}; }

void Receive(Client sender, string message)requires sender != this &&

unreceivedMsgs[sender].Length > 0 &&unreceivedMsgs[sender].Head == message;

{ unreceivedMsgs[sender] = unreceivedMsgs[sender].Tail;}

…}

Observable:Observable:

Online testing

� For large reactive systems deriving an exhaustive test-suite is

infeasible

� Merge test generation and test execution into one process

� Online testing is a form of model-based stress testing

65Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� Purely random exploration does not work

• Use scenario control to direct online testing

Example of scenario controlChat initialization

� Use a parameterized scenario action Start.

• Invoking Start(n) produces a sequence of n Create() invocations followed by n Enter() invocations

[Action(Kind=Scenario)]void Start(int nrOfMembers)requires enumof(Client) == Set{};

66Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

requires enumof(Client) == Set{};{Seq<Client> clients = Seq{};

// 1-- The given number of // clients are createdfor (int i = 0; i < nrOfMembers; i++)clients += Seq{Create()};

// 2-- all clients enter the sessionfor (int i = 0; i < nrOfMembers; i++)clients[i].Enter();

}

Start(2)

Create()/c0;Create()/c1;c0.Enter();c1.Enter();

Demo: chat

Bob Server Mary

67Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

Time

Experiences

� Most tool features driven by demands of internal

users at MS (mostly testers)

� Models help discover more bugs during modeling

(design bugs) than testing

68Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

(design bugs) than testing

•Testers do not get credit for finding those bugs !!!

� During testing models help discover deep

system-level bugs where manual test scenarios

don’t

•Such bugs are hard to understand and fix

Experiences (cont.)

� Bugs appear in both models and implementations (the ratio is roughly 50-50)

� Code coverage is a poor measure for testing concurrent software, often a single execution thread provides the same coverage

69Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

thread provides the same coverage

� New versions of implementation usually require only local changes to models, whereas manual tests often need to be rewritten completely

� The tool is built on .NET but has been used to test distributed C++ applications. Most test harnesses at MS are built in managed code

NModel – http://www.codeplex.com/NModel

� NModel is a model-based analysis and testing framework for

model programs written in C#.

� It provides software support for the book "Model-based

Software Testing and Analysis with C#", Cambridge University

Press, 2007

70Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

Press, 2007

� See http://staff.washington.edu/jon/modeling-book/ for

more information regarding the book

References Main references

� [P07] Ana C. R. Paiva, PhD Thesis untitled “Automated Model-Based Testing of Graphical User Interfaces”: www.fe.up.pt/~apaiva/PhD/PhDGUITesting.pdf

� [UL07] Mark Utting and Bruno Legeard, “Practical Model-Based Testing: A Tools Approach”, Morgan Kaufmann, Elsevier, 2007.

Other references

� [N00] N. Nyman, "Using Monkey Test Tools," in STQE - Software Testing and Quality Engineering Magazine, 2000.

� [UTF-site] For Unit testing frameworks visit sites – www.nunit.org; www.junit.org

71Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

[UTF-site] For Unit testing frameworks visit sites – www.nunit.org; www.junit.org

� [JMWU91] R. Jeffries, J. R. Miller, C. Wharton, and K. M. Uyeda, "User Interface Evaluation in the Real World: A Comparison of Four Techniques," 1991

� [GC96] C. Gram and G. Cockton, Design Principles for Interactive Software, Chapman & Hall, isbn:0412724707, 1996

� [HVCKR01] J. Hayhurst, D. S. Veerhusen, J. J. Chilenski, and L. K. Rierson, "A Practical Tutorial on Modified Condition / Decision Coverage," NASA/TM-2001-210876, 2001.

� [BS02] J. Bredereke and B.-H. Schlingloff, "An Automated, Flexible Testing Environment for UMTS", in Proceedings of the IFIP 14th International Conference on Testing Communicating Systems XIV, 2002.

� [ABM98] P. E. Ammann, P. E. Black, and W. Majurski, "Using Model Checking to Generate Tests from Specifications", in Proceedings of the 2nd IEEE International Conference on Formal Engineering Methods (ICFEM'98), M. G. H. John Staples, and Shaoying Liu(Eds.), Brisbane, Australia, 1998.

References

SpecExplorer and Spec# references

� [BLS04] M. Barnett, K. R. M. Leino, and W. Schulte, "The Spec# Programming System: An Overview," presented at CASSIS'04 - International workshop on Construction and Analysis of Safe, Secure and Interoperable Smart devices, Marseille, 2004.

� [CGNSTV05] C. Compbell, W. Grieskamp, L. Nachmanson, W. Schulte, N. Tillmann, and M. Veans, "Testing Concurrent Object-Oriented Systems with Spec Explorer," presented at FM, 2005.

� [FSE-site] Visit FSE web site for more information: research.microsoft.com/foundations

72Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

� [SE-site] Visit Spec Explorer site for download: research.microsoft.com/SpecExplorer/

Additional reading

� [W03] James A. Whittaker, “How to Break Software: A Practical Guide to Testing”, ISBN: 0201796198

� [A-site] Alan Hartman –http://www.agedis.de/documents/ModelBasedTestGenerationTools_cs.pdf

� Model-Based Testing papers –www.geocities.com/model_based_testing/online_papers.htm

END

Questions?

73Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008

Questions?