Practical use of Model-Based Testing in the Agile ...Not only Finite State Machine Models are great...

Practical use of Model-Based Practical use of Model-Based Testing in the Agile Development Testing in the Agile Development

CycleCycle

Kristian Karl

Copyright Prolore AB. All rights reserved.

Kristian Karl

Test consultant at Prolore, StockholmTest automation since 1994Model-based testing since 2004Founder of mbt.tigris.org

2

Agenda

3

What's different with mbt.tigris.org

How to model?

Metrics, and how to present results

Model-based testing and Agile Development

Showcases and key success factors

What's different with mbt.tigris.org

4

What's different with mbt.tigris.org

5

No UMLNo [graphical] exit/stop pointsFocus on OnlineConnects to any tool that understands Java or SOAP

No UML

6

A shallower learning curve• mbt.tigris.org's simple rule-set, in conjunction

with GraphML, makes it easier for tester to get started.

• As testers, we do not need all functionality that UML has to offer.

Typically, a tester can start designing after only a few hours.

No [graphical] exit/stop points

7

The way to tell mbt.tigris.org to stop generating a test sequence, are done by means of Stop Criteria.

Stop Criteria are passed as arguments to the tool, and they can be combined.

Creates different types of tests from the same model.

• Functional tests• Smoke tests• Bug hunting tests

Stop Criteria

8

REACHED_EDGE

REACHED_VERTEX

EDGE_COVERAGE

VERTEX_COVERAGE

TEST_LENGTH

TEST_DURATION

REACHED_REQUIREMENT

REQUIEMENT_COVERAGE

NEVER

Online

9

Supports online test sequence generation.Bug hunting! Generate long and unpredictable test sequences.

• We do not want to walk the same path every time we execute an automated test.

• We want variation, spiced with randomness. This will actually find bugs, not only verify requirements!

Enables non-deterministic tests.

Running Online

10

Instantiated in a Java test:org.tigris.mbt.ModelBasedTesting.getInstance()

SOAP – Running mbt as a web service● SetCurrentVertex● GetDataValue● ExecAction● GetNextStep● GetStatistics

● HasNextStep● Load● Reload● PassRequirement

The Walks

11

RANDOMA_STARSHORTEST_NON_OPTIMIZED

RANDOM

12

Navigates through the model in a completely random manor. Also called "The Drunkard’s walk". The algorithm selects an out-edge from current vertex by random, and repeats the process in the next vertex. Example:

java -jar mbt.jar offline -g RANDOM -s EDGE_COVERAGE:100 -f UC01.graphml

A_STAR

13

Will try to generate the shortest possible test sequence through a model with given stop condition. The algorithm only works well on smaller models.

Example:

java -jar mbt.jar offline -g A_STAR -s EDGE_COVERAGE:100 -f UC01.graphml

SHORTEST_NON_OPTIMIZED

14

This is a compromise between A_STAR and RANDOM. The algorithm works as follows:

1. Choose an edge not yet visited by random.

2. Select the shortest path to that edge using Dijkstra's algorithm.

3. Walk that path, and mark all walked edges as visited.

4. When reaching the chosen edge in step 1, start all over, repeating steps 1 → 4.

Example:

java -jar mbt.jar offline -g SHORTEST_NON_OPTIMIZED -s EDGE_COVERAGE:100 -f UC01.graphml

Command options in xml

15

<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE MBTINIT SYSTEM "mbt_setup.dtd" ><MBTINIT EXTENDED="true" EXECUTOR="java:myPackage.myTestClass" LOG-COVERAGE="10"> <MODEL PATH="model/Main.graphml"/> <MODEL PATH="model/basicSubModels"/> <SCRIPT PATH="model/basicFunctions.bsh"/> <SCRIPT> x = 0; </SCRIPT> <GENERATOR TYPE="A_STAR"> <CONDITION TYPE="REACHED_VERTEX" VALUE="v_MainWindow_DB_Loaded"/> </GENERATOR> <GENERATOR TYPE="RANDOM"> <OR> <AND> <CONDITION TYPE="EDGE_COVERAGE" VALUE="100"/> <CONDITION TYPE="VERTEX_COVERAGE" VALUE="100"/> </AND> <CONDITION TYPE="TEST_DURATION" VALUE="900"/> </OR> </GENERATOR></MBTINIT>

How to model?

16

The editor yEd from yWorks

17

mbt.tigris.org - Syntax

18

Edges

19

An edge is a transition. Could be a click on a button, a timeout, a server API call.

An edge can have a label, but it is not mandatory.• The label of an edge will map against a method, function

or sub routine during test execution.

As a rule of thumb, the best practice is to start the name with 'e_'. The reason for this, is that it is easier to recognize the function in the test execution tool, as an edge. For example: e_StartApplication e_EnterBasicView

The same name of an edge, can be re-used in the model.

Vertices

20

A vertex is some kind of state that is possible to verify using some kind of oracle.

A vertex must always have a label.• The label of a vertex is mapped against a method,

function or sub routine during test execution.

As a rule of thumb, the best practice is to start the name with 'v_'. The reason for this, is that it is easier to recognize the function in the test execution tool, as a vertex. For example: v_ApplicationStarted v_BasicView

The same name of a vertex, can be re-used in the model.

Labels

21

Label [cond.expr.] / Statement1;Statement2;

White spaces not allowed. Follow the naming convention for the implementing programming language.

Example:e_StartApplication

Example:e_EnterBasicView

Guards

22

Label [cond.expr.] / Statement1;Statement2;

Conditional expression that returns a Boolean value.The guard indicates if the edge is accessible in the current state of the machine. The language is either Java or JavaScript.

Example:e_Evaluate [x>0 && y<=10]

Example:e_Checkout [shoppingBasket.contains("Cerials")]

Statements

23

Label [cond.expr.] / Statement1;Statement2;

Any applicable Java or JavaScript statements

Example:e_Calculate / x=5;y=10*x;

Example:e_AddCerials / shoppingBasket.add("Cerials");

Keyword - Start

24

Used by Vertices only.

Each graph must have a start vertex, and that vertex holds this keyword. There is only one vertex in a graph, holding this keyword.The Start vertex can only have one out-edge.If the graph is the main-graph, that edge must have a label.If the graph is a sub-graph, that edge cannot have a label.

Keyword - REQTAG

25

Used by Vertices.Used by MBT for keeping track of which requirements are fulfilled during testing.

In the example above, if v_BookInfomation is passed, it would mean the the requirement with tag UC01 2.2.3 has passed.

Keyword - weight

26

Used by Edges only.

Used by MBT during random walks during test sequence generation. It holds a real value between 0 and 1, and represents the probability that a specific edge should be chosen. A value of 0.05, would mean a 5% chance of that edge to be selected during a run.

Note: It only works with the RANDOM generator, and must be enabled using the command-line switch -w

Keyword - BLOCKED

27

Used by both Vertices and Edges.

A vertex or an edge with the key word BLOCKED, will be excluded from the model when walked.

Keywords for Sub-graphs

28

The Stop vertex denotes an exit point in a sub-graph. Only sub-graphs can have a stop vertex.

MERGE is used when merging graphs. It means that the vertex containing this key word, should be merged with the first occurrence of a vertex with the same label.

NO_MERGE is used when merging graphs. It means that the vertex containing this key word, should not be merge (or expand) this vertex with correspondent sub graph.

Test Data

29

Test Data

30

Test Data

31

Test data should be provided by a tool (written in Java), which acts as an abstraction layer between the test and the data sources:

• tool.GetRandomLoginCredentials()• tool.GetRandomPerson()• tool.GetRandomBook()

Example - EFSM

32

Example – Requirements

33

Example - Sub-graph

34

Main graph

Sub-graph C

Result whenmerged

Metrics, and how to present Metrics, and how to present resultsresults

35

Metrics

36

Standard test metrics does not apply so well.Test cases are not meaningful when running online tests.MBT tends to drive the requirement work.

Metrics

37

Focuses on requirements (or verification points)

• ”How many requirements should have been verified?”or:

• ”How many verification points should have been verified?”

Reporting

38

Requirement Database• Ties requirements to Sprints• Ties requirements to Teams

Test Result Database.• Stores data from every test run• Knows what requirements was verified

Dashboard• Displays what requirements are verified• Allows drill-downs

Reporting - Example

39

Model-based testing and Model-based testing and Agile DevelopmentAgile Development

40

Model-based testing and Agile Development

41

Side-by-side from day one• The testers starts modeling immediately.• It is not unusual that test automation is

ready before developers.

Test is more exposed than earlier• Visualization• More people understands what test is

actually doing.• Test is getting more feedback.

Model-based testing and Agile Development

42

Not only Finite State Machine Models are great in Agile test, also consider Classification Trees Models.

Model-based testing and Agile Development

43

The use of mockups is very important.

• In a sprint, the team may not have all the functionality needed to run the tests. This is solved by using mockups.

The tests must be able to run on different environments.

• Test Data Tool

Version handling in the same branch as Dev.

Model-based testing and Agile Development

44

The Dev teams delivers to the Integration team after the demo.The Integration team has maintenance responsability.The Integration team runs continuous nightly builds, deploys and run all delivered tests.

Showcases and key success Showcases and key success factorsfactors

45

Swedish Pensions Agency

46

3-tier Client / Server• Mysql, Oracle, JBoss, Java, Perl

Mixed new functionality and maintenance of legacy codeTesting interfaces:

• Thin GUI clients• Fat GUI clients• Web Services• Batch jobs

4 dev teams, 1 integration team

Swedish Pensions Agency

47

One team test automated every test.

Test data tool – GTAD [Generate Test Automation Data]

10 testers wrote 69 trouble reports in 8 months.• What if we would have written 1.000 trouble

reports, and spent 2 hours on every report?• Many issues very exposed early because of the

models.

Every night we executed all our automated tests:• Unit tests• Functional API-tests• GUI tests• Performance tests

Neonode - N2

48

Any test should be able to run on any device.Automated GUI testsCall setupsSMS/MMS sending and receiving

Business Values

49

Proven technique for test automation in Agile Dev. Teams.

• Enables QA people with business domain knowledge to design automated tests.

• Models can be re-used for different purposes.

Will find bugs.Long life & Durability

Key Success Factors

50

Start modeling early.

A high abstraction level of the models.

Happy paths.

Show the models!

Use the tests for bug hunting, not just verification of requirements.

• This means online tests.

Use mockups.

Test Data Tool

Links

51

The tools:http://mbt.tigris.orghttp://www.yworks.comhttp://www.berner-mattner.com/en/berner-mattner-home/products/cte-xl/index.html

Email:kristian.karl@prolore.sekristian.hermann.karl@gmail.com