fMBT free Model-Based Testing tool Lesson 2: Test …Lesson 2: estT models and test generation In model-based testing tests are automatically generated from models . A model de nes

fMBT � free Model-Based Testing tool

Lesson 2: Test models and test generation

Antti Kervinen

Intel

[email protected]

This course contains seven lessons:

1 Model-based testing

2 Test models and test generation

3 Adapters and test execution

4 Modelling in GT

5 Modelling in AAL/C++, AAL/Pythonand AAL/Java

6 Remote and distributed adapters

7 Modelling concurrent systems, testinginterleavings of test steps

Antti Kervinen (Intel) fMBT � free Model-Based Testing tool 1 / 38


In this lesson you will learn:

Q1 What is a model?

Q2 Generating the �rst test from a GT model

Q3 Generating the �rst test from an AAL model

Q4 What happens in test generation?

Later lessons will teach modelling languages GT and AAL in moredetail. This lesson will introduce them in the test generation per-spective.

If you want to try fMBT commands in this lesson, please install thefMBT tool.



In model-based testing tests are automatically generated from models. Amodel de�nes two things:

1 actions, that is names of test steps.

2 preconditions for each test step. Test step can be run only if itsprecondition is true.






In the �rst lesson you already saw a model:

Test steps. . .

preview

captureImage

startVideocapt

stopVideocapt

and when they're enabled

always

if not capturing

if not capturing

always






The fMBT tool o�ers two alternative modelling languages:

GT (Graph Transformation language) de�nes and modi�esstate machine-like models.

AAL (Adapter Action language) de�nes models using pre- andpostconditions.



Q1 What is a model?




Now you know what is a model. Let's see how to create one usingthe GT language. We will generate a test from the model, too.



Test steps. . .

preview

captureImage

startVideocapt

stopVideocapt


always

if not capturing

if not capturing

always

Next we show how to express the above model using the GT language.



P(not-capturing, "gt:istate") ->

T(not-capturing, "iPreview", not-capturing)

T(not-capturing, "iCaptureImage", not-capturing)

T(not-capturing, "iStartVideocapt", capturing)

T(not-capturing, "iStopVideocapt", not-capturing)

T(capturing, "iPreview", capturing)

T(capturing, "iStopVideocapt", not-capturing)

Above GT rule (�le capture.gt) de�nes the model on Slide 7.










Above GT rule (�le capture.gt) de�nes the model on Slide 7.The model can be visualised as follows:fmbt-gt -f capture.gt | fmbt-view -

[initial state]

iCaptureImage

iPreview

iStopVideocapt

iPreview

iStartVideocapt iStopVideocapt










GT rules contain two parts separated by �->�. The �search� part onthe left and the �replace� on the right.

State tag is a tag (a string) associated with a state. Transition is anaction (a string) associated with two states: source and destinationstates of the transition. Elements in the �search� part match to statetags and transitions that exist in the state machine. Elements in the�replace� part add states, state tags and transitions to it.










P(not-capturing, "gt:istate") in the search part of the above rulematches to states with a special tag �gt:istate� on them. Therecan be only one such a state: the initial state of the state machine.

As this is the �rst appearance of �not-capturing� in the rule, it is�xed to refer to the initial state in later elements of the rule.

The fmbt-gt command on Slide 9 did not specify state machine tobe edited by this GT rule. Therefore fmbt-gt started editing a newstate machine which has only one state: its initial state.










T(not-capturing, "iStartVideocapt", capturing) in the replace part of therule means that test step �iStartVideocapt� can be tested in the�not-capturing� state, and test generation will continue from the�capturing� state after doing so.

As this is the �rst appearance of the �capturing� state in the rule,and we are already in the replace part, a new state is created. Thatstate will be called �capturing� in the remaining elements of therule.










Exercise: How would you change this model if capturing an imageshould be tested also when video is being captured? Capturing animage during video capture does not a�ect video capturing.










Exercise: How would you change this model if capturing an imageshould be tested also when video is being captured? Capturing animage during video capture does not a�ect video capturing.

Adding element T(capturing, "iCaptureImage", capturing) anywhere inthe replace part allows running test step �iCaptureImage� when themodel is in the �capturing� state. Executing this transition keepsthe model in the same state. That is, after the execution the test cancontinue with �iStopVideocapt� but not with �iStartVideocapt�.










Actions like iPreview, are pre�xed with �i�. Therefore fMBT willinterprete these actions as �input� whose execution is triggered bytest execution framework. The other option would be pre�x �o� for�output�. Execution of output actions would be triggered by thesystem under test.

Now, let's generate a test from this model.



iStopVideocapt

iStartVideocapt

iPreview

iStopVideocapt

iCaptureImage

iPreview

iStartVideocapt

iStopVideocapt

iPreview

iStopVideocapt

pass

We generated the test by running fmbt capture.conf | fmbt-log

Parameters for test generation are given in a test con�guration �le.Our capture.conf looked like this:

model = "lsts_remote(fmbt-gt -f capture.gt)"

pass = "steps(10)"

We will return to this con�guration when explaining test generation.



Q1 What is a model?




Now you have seen how to create a state machine model using theGT language, and how a test was generated from the model. Nextwe will create a model using AAL.



Test steps. . .

preview

captureImage

startVideocapt

stopVideocapt


always

if not capturing

if not capturing

always

Again, we will create a model similar to the model above.



aal "capture" {

language: C++ {};

variables { bool capturing; }

initial_state { capturing = false; }

action "iPreview" {

guard() { return true; }

}

action "iCaptureImage" {

guard() { return !capturing; }

}

action "iStartVideocapt" {


model() { capturing = true; }

}

action "iStopVideocapt" {


model() { capturing = false; }

}

}

capture.aal on the left con-tains an AAL model corre-sponding to the previous slide.

guard() function returnstrue if the action is enabled.If guard() is not de�ned,the action is always enabled.

model() function is calledwhen the test step has beenrun. If not de�ned, executingthe test step does not a�ectthe state of the model.



aal "capture" {

language: C++ {};



action "iPreview" {


}



}




}




}

}

Exercise: How would youchange this model if captur-ing an image should be testedalso when video is being cap-tured? Capturing an imageduring video capture does nota�ect video capturing.



aal "capture" {

language: C++ {};



action "iPreview" {


}



}




}




}

}

Exercise: How would youchange this model if captur-ing an image should be testedalso when video is being cap-tured? Capturing an imageduring video capture does nota�ect video capturing.

Removing guard() functionfrom action iCaptureImage

makes it always enabled.Changing guard() implemen-tation to return true; hasthe same a�ect.



AAL/C++ models are compiled to C++ and then built as a plugin (sharedlibrary) that will be loaded by fMBT. If g++ is available, fMBT is installedin standard location and AAL code does not need any unusual includes orlibraries, this can be done automatically with command

fmbt-aalc capture.aal -c -o capture.so

Otherwise you need to �rst compile AAL into C++ and then give the buildcommand with whatever include paths and libraries you need. For instance:

fmbt-aalc capture.aal -o capture.cpp

g++ -shared -fPIC -I/my/include/path capture.cpp -lmylib -o capture.so



Test generation from AAL models is similar to GT models. The onlydi�erence is how fMBT is con�gured to load the model. This time,however, we will generate a shorter test using con�guration �lecapture.aal.conf:

model = "lib(capture)"

heuristic = "lookahead"

coverage = "perm(1)"

and generate a test:

fmbt capture.aal.conf | fmbt-log

iPreview

iCaptureImage

iStartVideocapt

iStopVideocapt

pass



Q1 What is a model?




Now you have got the �rst impression on two di�erent ways to createmodels: GT and AAL. It's time to explain how fMBT generated thetests from the models in both cases.



start

choose anenabled input

execute theaction on SUT

ok by testmodel?

verdict:fail

updatetest model

endconditions?

verdict:pass/fail/inconc

suggested action

reported action

yes

no

no

yes

This �owchart presents fMBT test generationand execution logic.

At start the tool has read the test con�gura-tion �le. First it has loaded a model.

model = "lsts_remote(fmbt-gt -f capture.gt)"

on Slide 16 says �load model by executingcommand fmbt-gt -f capture.gt and interpretits output as an lsts�. GT rules only searchand replace elements of state machines. Ac-tual state machines are stored in lsts (labelledstate-transition system) format.

model = "lib(capture)"

on Slide 23 causes fMBT to look for a modelcalled �capture� in capture.so.



start



ok by testmodel?

verdict:fail

updatetest model

endconditions?


suggested action

reported action

yes

no

no

yes

The tool has loaded and initialized anadapter, too. Adapters are responsible forexecuting generated test steps.

As our �rst test con�guration examples do notinclude adapter speci�cation, fMBT will usethe default, �no-operation� adapter. There-fore test steps will only be generated, not re-ally executed.

Test generation is ready to start.



start



ok by testmodel?

verdict:fail

updatetest model

endconditions?


suggested action

reported action

yes

no

no

yes

The �rst task in test generation loop is de-ciding which input action should be executednext.

The choice is made among enabled input ac-tions. In the GT example this means the setof actions that can be executed in the currentstate of the state machine. In the �rst roundthis is the initial state (the not-capturing

state on Slide 15).

In the AAL example this means evaluatingguard() functions of all actions in AAL modeland choosing between those that returnedtrue.



start



ok by testmodel?

verdict:fail

updatetest model

endconditions?


suggested action

reported action

yes

no

no

yes

Exercise: Which test steps are possible in theinitial state of the GT model on Slide 15? Howabout the AAL model on Slide 21?



start



ok by testmodel?

verdict:fail

updatetest model

endconditions?


suggested action

reported action

yes

no

no

yes

Exercise: Which test steps are possible in theinitial state of the GT model on Slide 15? Howabout the AAL model on Slide 21?

All test steps, iPreview, iCaptureImage,iStartVideocapt and iStopVideocapt, arepossible. That's because. . .

The GT rule adds four transitions thatcan be executed in the initial state: onefor each action.

guard() functions of every actionevaluate to true in the initial state ofthe AAL model.

Let's see how the choice between actions ismade.



start



ok by testmodel?

verdict:fail

updatetest model

endconditions?


suggested action

reported action

yes

no

no

yes

Two optional parameters in test con�gurationsa�ect the choice of the next action:

heuristic de�nes which test generationalgorithm will be used.

The default is random: all enabled

input actions have equal chance to

become selected. Used on Slide 16.

lookahead on Slide 23 is a search

algorithm that looks for the shortest

path to the greatest improvement on

coverage. It chooses the �rst action

on such a path.

coverage de�nes how to measure whathas been tested. perm:1 on Slide 23measures the percentage of executedsingle actions. perm:2 would measurethe percentage of executed action pairs.



start



ok by testmodel?

verdict:fail

updatetest model

endconditions?


suggested action

reported action

yes

no

no

yes

Example: on Slide 23, after generating

iPreview

iCaptureImage

iStartVideocapt

capturing is true in the AAL model. Themodel is on Slide 21. Possible next actionsare iPreview and iStopVideocapt.

Lookahead compares:

iPreview does not increase perm:1

coverage, the percentage of executedactions, because it's already executed.

iStopVideocapt increases thepercentage to 100 %.

Lookahead chooses iStopVideocapt.



start



ok by testmodel?

verdict:fail

updatetest model

endconditions?


suggested action

reported action

yes

no

no

yes

Test generator sends the chosen input actionto the adapter for execution.

Adapters interact with the system under test,and report back which action they actuallyexecuted. For instance, suggesting an action�status() == 'idle'� could result in report-ing action �status() == 'online'�, or a spe-cial action 0 if the result cannot be reportedusing the set of actions available.

fMBT adapters allow implementing test stepsusing C/C++, Python, Java, shell commands,and Javascript for standard browsers.

If an adapter is not de�ned in the test con�g-uration �le, the default adapter only reportsthe execution of the suggested action.



start



ok by testmodel?

verdict:fail

updatetest model

endconditions?


suggested action

reported action

yes

no

no

yes

Test generator validates the action reported bythe adapter: does the model allow executionthis action?

If the adapter reports 0, or if reported actioncannot be executed in the current state of theGT model, or if the guard() of the action re-turns false in the AAL model, test has foundan error. Test will exit with verdict fail.

Otherwise, test step has been executed suc-cessfully.



start



ok by testmodel?

verdict:fail

updatetest model

endconditions?


suggested action

reported action

yes

no

no

yes

Test generator updates the model correspond-ing to the reported action.

In GT the update means changing the cur-rent state of the model to the destinationstate of the executed transition. For instance,if iStartVideocapt is executed in the initialstate of the GT model on Slide 15, the cur-rent state is updated from not-capturing tocapturing.

In AAL the update means executing model()

code block of the reported action. In case ofexecuting iStartVideocapt in AAL model onSlide 21, C++ code �capturing = true;� isexecuted.



start



ok by testmodel?

verdict:fail

updatetest model

endconditions?


suggested action

reported action

yes

no

no

yes

The last step in the loop checks should thetest generation continue or not.

Test con�guration �les de�ne end conditionsfor each test separately. If any of the condi-tions is true, fMBT exits with the given testverdict.

The �rst test con�guration on Slide 16 de�nesend condition pass = "steps(10)". That is,test ends with verdict �pass� after 10 steps.

The second test con�guration does not con-tain end conditions on Slide 23. It relies on theonly default, pass = "coverage(1.0)". Thatis, test passes when coverage is 100 %. Cov-erage �perm(1)� equals 1.0 when every actionin the model has been executed at least once.



Commands we used for test generation, such as

fmbt capture.conf | fmbt-log

launch fMBT with test con�guration �le capture.conf. fMBT writes logto the standard output because log �le is not given. Then fmbt-log parsesthe log, prints reported test steps and �nally the verdict. Therefore there is�pass� as the last line on Slides 16 and 23.



Q1 What is a model?




Now you have learned the idea how fMBT generates tests from bothGT and AAL models.



As a summary,

Q1 What is a model?

Model de�nes actions, that is names of test steps, and orders in which

they can be run.


GT de�nes and modi�es state machine-like models.


AAL de�nes models with pre- and postconditions.


Test generation and execution is a loop that runs until any of end

conditions is met or an error is found.

The loop is: choose an action, execute it in adapter, validate action

reported by the adapter, update the model accordingly.

This ends Lession 2.


Documents

fMBT free Model-Based Testing tool Lesson 2: Test …Lesson 2: estT models and test generation In model-based testing tests are automatically generated from models . A model de nes