Esin Karaman

SWE550

Boğaziçi University

1

Agenda Software testing process evalution

What is model-based testing

The process of model-based testing

MBT (Model-based testing) Tools

Prerequisites for MBT

Benefits of MBT

Limitations of MBT

2

«Life is too short for manual testing.»

Harry Robinson, test architect for Microsoft’s Engineering Excellence Group and a driving force behind Microsoft’s model-based testing initiative, 1999-2004

3

Software testing process evalution

4

Manual Testing

Capture/Replay

Script Base Testing

Keyword-Driven Automated Testing

Model-Based

What is model-based testing? Model-based testing is a variant of testing for

automatic generation of test cases using models describing some aspects of the software under test (SUT).

It can automate the complete test design process, given a suitable model, and produces complete test sequences that can be transformed into executable tests cripts.

Model-based testing is the automation of the design of black-box tests

5

What is model-based testing? The difference from the usual black-box testing is that

rather than manually writing tests based on the requirements documentation, we instead create a model of the expected SUT behaviour, which captures some of the requirements. Then the model based testing tools are used to automatically generate tests from that model.

6

The process of model-based testing

Create a model detailed enough to describe the characteristics that we want to test

Automatically generate abstract test cases from the model by using a model-based testing tool

Transform abstract test cases into into executable test scripts

Execute test scripts by a test execution tool

7

8 Model-Based Testing Process

Modelling

A model of the SUT is built on the grounds of requirements

Includes expected behaviour

Must be concise and precise

9

Modelling (2)

Model-based testing uses models of the SUT and its environment.

Abstraction

- Function

- Data

M1-M3 is suitable

10

Modelling (3)

Modelling notations

State based (or Pre/Post) Notations

Transition-based Notations

Functional Notations

Operational Notations

Stochastic Notations

Data-Flow Notations

11

Test requirements selection

Used to control the generation of tests Steps:

Test selection criteria are defined

Test selection criteria are then transformed into test case specifications

12

Test selection criteria

Defining «best» test selection criteria is not possible. Test engineer chooses adequate test selection criteria. Criteria types:

Structural model coverage criteria

Data coverage criteria

Requirements-based coverage criteria

Ad-Hoc test case specifications

13

Test generation

Once the model and the test case specification are defined, an abstract test suite is generated.

Technology:

Random

Graph-search algorithms

Model-checking

Symbolic execution

Theorem proving

14

Test concretization(transformation)

Concretize abstract test suite to executable level (test script).

This can be done by a MBT Tool using some translation tables supplied by the test engineer.

The resulting executable tests may be Junit in Java, or in a dynamic language such as Tcl or Python, or in a dedicated test scripting language.

15

Test execution

Test scripts are executed and outcomes are compared to expected ones.

Tests are marked as pass, fail or inconclusive

Two approaches:

Online

Offline

16

Test execution(2)

Online Testing

Tests are executed as they are generated

MBT tool is tightly coupled to the SUT

Slow

Offline Testing

Tests are generated first, executed afterwards

Decouples the generation and execution phases

It can be used repeatedly for regression purposes

17

MBT Tools TorX

LTG (LEIRIOS Test Generator)

Matlab Simulink V&V

JUMBL

AETG

18

Prerequisites for MBT Someone enthusiastic

Test execution maturity

Modelling skills

Access to training, tools, and experts

The right kind of application

19

Benefits of MBT

Fault detection : MBT founds roughly the same number of errors as manually designed test suits

Reduced testing cost and time

Improved test quality

Requirements defect detection

Requirements evolution

20

Limitations of MBT

The model designers must be expert in the application area. This may require some training costs.

Usually used only for functional testing, little experience for other kinds of testing.

Sophisticated approach to testing, so it requires a reosonable level of test maturity. It is unwise to start using MBT without some experience with automated test execution.

21

References Mark Utting, Alexander Pretschner and Bruno Legeard

, A Taxonomy of Model-Based Testing, 2006

Bruno Legeard, Mark Utting, Practical Model-Based Testing A Tools Approach, 2007

A. Pretschner, W. Prenninger, S. Wagner, C. Kuhnel, M. Baumgartner, B. Sostawa, One Evaluation of Model-Based Testing and its Automation, 2005

22