Model-Based Testing – From theory to practice

Frédéric Dadeau – Elizabeta Fourneret – Bruno Legeard

Ecole des jeunes chercheurs en programmation – Nancy – 24 june 2015

2

Presentation of the workshop

Morning - Presentations • Part I – Introduction to MBT • Part II – MBT with Smartesting CertifyIt • Part III – PKCS#11 case study

Afternoon – Practical session

• Discovering ATM case study • Practicing CertifyIt

Model-Based Testing – From theory to practice

Part I – Introduction to MBT

Bruno Legeard

Ecole des jeunes chercheurs en programmation – Nancy – 24 june 2015

4

400 000 certified testers

5

Various faces of MBT

Tests

6

Various faces of MBT

Model-Based Testing : « Testing based on or involving models. » ISTQB Glossary - 2015

7

Outline

1. Introduction to MBT

a. Motivations and usage

b. Process and roles

2. A taxonomy of MBT approaches

3. Application domains

4. Conclusion

8

Motivations

Source : Model-Based Testing, Where do we stand? – CACM – 2/15, Binder, Legeard, Kramer

Source: MBT User Survey 2014

9

Motivations

“Test Models provide us more flexibility in the creation of automated test scripts by using different test case generation strategies according to the current needs (different robustness test, tests for changed or critical requirements, etc.)” “Executable model paradigm should enable us to create self-testing mechanisms.” “MBT helps us to improve change request process” “MBT supports test automation” “MBT is capable to also validate the test basis” “Massive increase of reuse” “Improved coverage of requirement readability of coverage” “Increased productivity and improved product reliability (new type of bugs, Increased test coverage)”

10

Test levels

Source: MBT User Survey 2014

11

Test types

Source: MBT User Survey 2014

12

Modeling practices

Do you use models in other development phases, for example for requirement elicitation or system design?

Yes No

58,9%

41,1%

Source: MBT User Survey 2014

13

Modeling practices - 2

0,0%

10,0%

20,0%

30,0%

40,0%

50,0%

60,0%

70,0%

80,0%

Use Casediagrams

Activitydiagrams

Statemachines

Sequencecharts

BusinessProcessmodels

DomainSpecific

Languagemodels

What type of models do you use in those other development phases?

Source: MBT User Survey 2014

14

Outline

1. Introduction to MBT

a. Motivations and usage

b. Process and roles

2. A taxonomy of MBT approaches

3. Application domains

4. Conclusion

15

MBT process - 1

Source : Mark Utting, Alexander Pretschner, and Bruno Legeard. 2012. A taxonomy of model-based testing approaches. Softw. Test. Verif. Reliab. 22, 5 (August 2012), 297-312.

16

MBT Process - 2

17

MBT Process - 3

18

MBT roles

19

MBT modeling

MBT modeling is strongly related to test objectives. MBT models should be:

– Abstract enough • Covering what is intended to be tested

– Detailled and precise enough • Enables to compute the expected results (Oracle)

Test verdict assignement

– Validated and verified • If the test « fails », what is wrong? The MBT model or the System

under Test (SUT)?

MBT is back-to-back testing – MBT model vs SUT

20

Outline

1. Introduction to MBT

a. Motivations and usage

b. Process and roles

2. A taxonomy of MBT approaches

3. Application domains

4. Conclusion

21

A taxonomy of MBT approaches

Source : Mark Utting, Alexander Pretschner, and Bruno Legeard. 2012. A taxonomy of model-based testing approaches. Softw. Test. Verif. Reliab. 22, 5 (August 2012), 297-312.

22

Subjet and focus of the MBT model

Subjet • System • Environment • Tests

Focus

• Structure • Behavior

An MBT model generally combines these various subject and focus aspects

23

MBT model - Example

24

MBT model - Example

25

Subject of the MBT model

Source : Mark Utting, Alexander Pretschner, and Bruno Legeard. 2012. A taxonomy of model-based testing approaches. Softw. Test. Verif. Reliab. 22, 5 (August 2012), 297-312.

26

Subject of the MBT model

05

101520253035404550

The modeldescribes how the

system works.

The modeldescribes the

system'senvironment,

concentrating oninterfaces and

stimuli (e.g. from auser's point of view).

The modeldescribes the testprocedure ratherthan the system

itself.

What is the subject of your model?

The one and onlysubject of the model

Important subject of themodel

Not the major subject ofthe model

Not at all a subject ofthe model

Source: MBT User Survey 2014

27

Focus of the MBT model

What aspect does your MBT model focus on?

Structural aspect

Behavioral aspect

Both

59%

35%

6%

Source: MBT User Survey 2014

28

Modeling paradigms for MBT • Many modeling paradigms used for MBT:

– Behavioral modeling (not exhaustive) • Flow Charts / BPMN / UML activity diagram

• Data Flow Diagrams

• Finite state machine – Statecharts – UML state machines

• Pre-post conditions / OCL

• Markov chain

• Event Sequence Graph

• Domain specific languages

– Structural modeling (Point of control & observation) • UML calss diagram

• Block diagram

• Representation : – Graphical (more « intuitive » )

– Textual / formal (more precise)

29

MBT modeling - BPMN

30

MBT modeling – OCL-based

31

Event Sequence Graph

32

Reuse of existing design models

Can we re-use design models for MBT?

In principle Yes, but: • The reused model is it at the right abstraction level to adress my test

objectives? • Is it complete enougch to adress my test objectives? • Does my MBT tool manage such model?

33

Reuse of existing design model

0,0%

5,0%

10,0%

15,0%

20,0%

25,0%

30,0%

35,0%

Completelyidentical

Slightly modified Largely modified Completelydifferent

Source: MBT User Survey 2014

If you use models both for analysis or design and for testing activities, how different are these models?

34

Reuse of models - Example

Business Process

Business

scenario

Test analyst

Business analyst

35

Taxonomy of MBT approaches

36

Generation of the test repository

Business

Requirements

Tests

Fonctionnels

Test

repository

manual auto

Tests ready for execution

and, or

Modelling for test generation

37

MBT generated artifacts

0,0%

10,0%

20,0%

30,0%

40,0%

50,0%

60,0%

70,0%

80,0%

90,0%

Test cases (formanual testexecution)

Test scripts (forautomated test

execution)

Test data Other artifacts(documentation,

test suites,...)

What artifacts do you generate from the model?

Source: MBT User Survey 2014

38

Traceability matrix

39

Test selection criteria

40

Test selection criteria

Source : ISTQB – Certification Model-Based Testing

41

Test selection criteria

0,0%

10,0%

20,0%

30,0%

40,0%

50,0%

60,0%

70,0%

80,0%R

eq

uir

em

en

tsco

ve

rag

e

Str

uctu

ral

mo

de

lco

ve

rag

e i)

Da

ta c

ove

rag

eii)

Ra

nd

om

&sto

ch

astic

Sce

na

rio

- /

pa

tte

rn-b

ase

diii

)

Pro

ject-

dri

ve

niv

)

What selection criteria do you apply to limit the number of generated tests?

Source: MBT User Survey 2014

42

Coverage monitoring

0,0%

10,0%

20,0%

30,0%

40,0%

50,0%

60,0%

70,0%

What kind of coverage is measured for your test runs?

Source: MBT User Survey 2014

43

Taxonomy of MBT approaches

44

MBT test execution

• Manual test execution • Offline MBT • Online MBT

Offline MBT “Model-based testing approach whereby test cases are generated into a repository for future execution.”

Online MBT “Model-based testing approach whereby test cases are generated and executed simultaneously.”

ISTQB Glossary - 2015

45

MBT test execution

0,0%

10,0%

20,0%

30,0%

40,0%

50,0%

60,0%

70,0%

80,0%

manually automatically off-line (firstgenerated, then executed)

automatically on-the-fly(generated and executed in

one step)

How are the tests executed?

Source: MBT User Survey 2014

46

Outline

1. Introduction to MBT

a. Motivations and usage

b. Process and roles

2. A taxonomy of MBT approaches

3. Application domains

4. Conclusion

47

Application domains

Enterprise IT (including packaged applications); 30,0%

Embedded controller (real-time); 27,1%

Embedded software (not real-time);

11,4%

Software Infrastructure;

5,7%

Web application; 18,6%

Gaming; 2,9%

Communications; 4,3%

What is the general application domain of the system under test?

48

Satisfaction level of MBT users

0,0% 10,0% 20,0% 30,0% 40,0% 50,0%

extremely uneffective

moderately uneffective

slightly uneffective

no effect

slightly effective

moderately effective

extremely effective

Overall, how effective do you think MBT has been?

Source: MBT User Survey 2014

49

Outline

1. Introduction to MBT

a. Motivations and usage

b. Process and roles

2. A taxonomy of MBT approaches

3. Application domains

4. Conclusion of Part I

50

An emerging technology in industry

Source : Harald Altinger, Franz Wotawa, and Markus Schurius. 2014. Testing methods used in the automotive industry: results from a survey. In Proceedings of the 2014 Workshop on Joining AcadeMiA and Industry Contributions to Test Automation and Model-Based Testing (JAMAICA 2014). ACM, New York, NY, USA, 1-6.

Re – Research PD – Pre Development SD – Serie Development TE - TEsting related department

51

UCAAT 2015

Source - http://model-based-testing.info

52

Thanks for your attention

Questions on Part I?