View
37
Download
0
Tags:
Embed Size (px)
Citation preview
Atos, Atos and fish symbol, Atos Origin and fish symbol, Atos Consulting, and the fish itself are registered trademarks of Atos Origin SA. August 2006© 2006 Atos Origin. Confidential information owned by Atos Origin, to be used by the recipient only. This document or any part of it, may not be reproduced, copied, circulated and/or distributed nor quoted without prior written approval from Atos Origin.
Tester needed? No thanks, we use MBT!
EuroSTAR, The Hague, 12 November 2008
By Elise Greveraars
Atos Origin Nederland BV High Tech Campus 525656 AG Eindhoven - The Netherlands www.atosorigin.com
2/34
Agenda
» Introduction Model Based Testing
» Case studies & Lessons learned
» Future work
» Recommendations & conclusion
Scope: Administrative systems
3/34
Good morning, let’s wake up first!
» Have you ever..
» Created models before?
» Used models to design testcases?
» Used models to automatically generate testcases?
Many testers test model based
4/34
Introduction
» Informal Model Based Testing
» Formal Model Based Testing
5/34
Informal MBT: Introduction
Modelling tool
Risk Analysis
1. Create models
Requirements
2. Evaluaterequirement
s and models
Reviewresults
3. Design test cases
Test management tool
Manual test cases
4. Execution test cases
Test results
Test results
6. Execution Test cases
5. Administer test cases
Automated test cases
Test automation
tool
4. Adapts script for automation +
add physical test data
6/34
Advantages/disadvantage informal MBT
» Advantages» Easy to create» Early detection of specifications defects» Early insight into the application » Test model is guideline for design test cases» No costs for test generation tool
» Disadvantages» High initial costs on modelling» Manual work on test case design» No standardized test case design» Maintenance on models and test cases
7/34
Formal MBT: Introduction
(UML) Modelling tool
Risk Analysis
1. Create formal models
Requirements
3. Evaluate models
MBT solution tool
7. Administer test cases
Automated test cases
Test automation tool
6. Adapts script for automation +
map physical test data
Testmanagement toolManual testcases +
frame automated test cases
4. (Re) generate logical test cases
5. Import testcases
2. Evaluate requirements
Reviewresults
6. Map physical test data
Test results
8. Execute Test cases7. Execute Test cases
Test data
8/34
Advantages/disadvantages formal MBT
» Advantages» Early detection of specifications defects» Early insight into the application» Time saving on test case design» Standardized test case design» High test coverage
» Disadvantages» Extra costs for MBT tooling » High initial costs on modelling» Additional skills are needed» Mainly for functional/robustness testing
9/34
Position of formal MBT in V-Model
User requirements
System requirements
Acceptance test
System integration test
Global design Component
integration test
Component test
System test
Detailed design
Implementation
Functional domain
Technical domain
Problem domain
Solution domain
System requirements
System test
System integration test
Acceptance test
10/34
Model Driven Engineering (MDE) versus MBT
MDE:
MDE uses models for engineering purposes
» MDE» Purpose is to construct code» Technical models» Includes implementation information
» MBT» Purpose is to construct testcases» Functional models» No implementation information
MBT:
MBT uses models for testing purposes
11/34
MDE/MBT: To reuse or not to reuse models
» Test models from scratch» No (formal) models are available» Full independence» Fully customized models
» Partial reuse of engineering models » Strip irrelevant information» Partial independence » Model should be customized
» Reuse of reversed models » Good for regression testing » Basic set of models to be adjusted» No independence» Model should be customized
» Combination of creating and reusing
12/34
Formal MBT: Guidelines modelling
» Choose a model notation (for example UML) + MBT tool
» Choose the risks you want to model
» Model only functional information
» Model the expected behaviour, models should be:» Accurate» Reasonable sized » Easy to maintain
» Use parameters to represent data
13/34
Formal MBT: Test generation techniques
» Selection criteria to generate test case
» Selection criteria depends on type of model
» Example of selection criteria for state diagram» State coverage» Transition coverage» All transition pairs
» Selection criteria determine:» Depth of testing» Size of test suite
State 3 State 5
State 1
S1
S2
S5
S4S3
AB
C
D
E
G
F
State 3 State 5
State 1
S1
S2
S5S5
S4S4S3
AB
C
D
E
G
F
14/34
Current market
» Tools
» Market adoption
15/34
Current market development formal MBT: Tools
» Support generation of test cases
» Some support modelling
» Some support test execution
» Support of various:» Model notations» Test generation techniques» Test case output
» Suppliers MBT tooling:SmarTesting, Verisoft Technology, All4tec, etc
16/34
Current market development formal MBT: Adoption
17/34
Agenda
» Introduction Model Based Testing
» Case studies & Lessons learned
» Future work
» Recommendations & conclusion
18/34
Case studies
» Case study on Informal Model Based Reviewing
» Formal MBT
19/34
Case study on informal Model Based Reviewing
Case study settings:» 2 versions of a newly built application» Informally reviewed» Same functional designer
Version 1: 104 function points, no models usedVersion 2: 113 function points, model used
Version 1 Version 2
20/34
Lessons learned Model Based Reviewing
» Model Based Reviewing very useful
» Missing and unclear specifications found
» Testers gained much knowledge
» Create models during reviewing sessions
» Less commitment by designers in later phase
21/34
Case study on formal model based testing
» Team was trained in 3 weeks on formal modelling and tools
» Incremental & iterative approach
» Order intake application
22/34
Case study: Used models
» Class diagram
» Object diagram
23/34
» State diagram & Object Constraint Language (OCL):
Case study: Used models
24/34
Case study: generated testcases
» Coverage criteria used:» Each transition » Conditions in the decisions (D/CC criteria) » One data value per equivalence class
25/34
Lessons learned formal MBT
Pro’s
» Models provide a better overview then text
» Traceability possible
» Maintenance shifts to modelling
» Combine MBT & test automation
» Modelling is a very creative task
» New exiting challenge and fun for testers!!
Cons
» Models not business user friendly
» No risk based test case generation yet
» Room for improvement on readability
26/34
Lessons learned: Competence
New role needed to additional roles: Testconstructor
Needs knowledge on:» Transforming written specifications into models » Creating models » Testing» Domain » Programming
27/34
Lessons learned: Competence
Traditional roles:
» Gain knowledge on» MBT and tooling» Modelling and tooling
» Test manager/coordinator» Adapt test strategy
» Test automate» Explore MBT tooling» Explore generated test cases
» Test designer» Design test cases not covered by MBT
» Test executer » Execute manual test cases
Hop on board and get enthusiastic!!
28/34
Agenda
» Introduction Model Based Testing
» Case studies & Lessons learned
» Future work
» Recommendations & conclusion
29/34
Future work
Risk based test generation
» Test manager in control on test case generation» High level risks -> Higher coverage criteria» Low level risks -> Lower coverage criteria
» Test generation based on probability
Path A-B: 15%
Path A-E-F-G: 25%
Path C-D-G: 60%
State 3 State 5
State 1
S1
S2
S5
S4S3
AB
C
D
E
G
F
State 3 State 5
State 1
S1
S2
S5S5
S4S4S3
AB
C
D
E
G
F
30/34
Future work
» Business model based testing» Business friendly models + test cases» Simulation possible
User requirements
System requirements
Acceptance test
System integration test
Global design Component
integration test
Component test
System test
Detailed design
Implementation
Functional domain
Technical domain
Problem domain
Solution domain
Business requirements Business
requirements
User requirements
Acceptance test
31/34
Agenda
» Introduction Model Based Testing
» Case studies & Lessons learned
» Future work
» Recommendations & conclusion
32/34
Recommendations
» Use informal MBT
» Use formal MBT when regression tests are expected
» Educate a test constructor
» Start small to gain experience
» Create models based on risks
» Start to model as early as possible
» Keep the models simple
33/34
Conclusion
Tester still needed? Yes for sure!!
34/34
Questions?
More information on this presentation:
- Eurostar CD
- Mailto: [email protected]
- Contact:Elise GreveraarsMobile: +31 (0) 6 [email protected]