Upload
techwellpresentations
View
215
Download
2
Embed Size (px)
DESCRIPTION
For decades, software development tools and methods have evolved with an emphasis on modeling. Standards like UML and SysML are now used to develop some of the most complex systems in the world. However, test design remains a largely manual, intuitive process. Now, a significant opportunity exists for testing organizations to realize the benefits of modeling. Adam Richards describes how to leverage model-based testing to dramatically improve both test coverage and efficiency—and lower the overall cost of quality. Adam provides an overview of the basic concepts and process implications of model-based testing, including its role in agile. A survey of model types and techniques shows different model-based solutions for different kinds of testing problems. Explore tool integrations and weigh the pros and cons of model-based test development against a variety of system and project-level factors. Gain a working knowledge of the concepts, tools, and techniques needed to introduce model-based testing to your organization.
Citation preview
W14 Concurrent Class
10/2/2013 3:00:00 PM
"Model-Based Testing:
Concepts, Tools, and
Techniques"
Presented by:
Adam Richards
Critical Logic
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Adam Richards
Critical Logic
Adam Richards is the director of consulting services at Critical Logic, a consulting firm
specializing in model-based testing tools and services. With nearly two decades in the industry,
Adam has experience as a developer, business analyst, requirements manager, test analyst,
and QA manager. He brings first-hand experience implementing model-based testing
approaches on systems in a wide array of industries, across the entire gamut of development
methodologies.
9/20/2013
1
2013 Critical Logic
Model-Based Testing
Concepts, Tools, and Techniques
Presented by Adam Richards
Starwest, 2013
2013 Critical Logic
2
If the shoe fits…
Model-Based Testing offers something for everyone.
9/20/2013
2
2013 Critical Logic
3
Who uses MBT, how, and for what
Survey says…
2013 Critical Logic
4
Domains
Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012
http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf
9/20/2013
3
2013 Critical Logic
5
Interfaces
Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012
http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf
2013 Critical Logic
6
Methodologies
Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012
http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf
9/20/2013
4
2013 Critical Logic
7
Test Level At what scope do you use MBT?
Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012
http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf
2013 Critical Logic
8
Test Aspects Which aspects of the SUT do you test with MBT?
Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012
http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf
9/20/2013
5
2013 Critical Logic
9
Benefits
On average, respondents report MBT:
• Reduced escaped bugs by 59%
• Reduced testing costs by 17%
• Reduced testing duration by 25%
Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012
http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf
2013 Critical Logic
10
Okay…
But what is it???
9/20/2013
6
2013 Critical Logic
11
What we Mean by “Model”
Model Soup
2013 Critical Logic
12
Structural Models
• Static
• ERD, Class, Component, Data Model…
• Can define test environment (e.g. UTP)
• Foundation for Behavioral models
• Used for both Rqmts & System Models
9/20/2013
7
2013 Critical Logic
13
Behavioral Models
• Describe flow
• BPM, Activity, State, Cause-Effect…
• Can specify abstract test scenarios
(no data, test environment)
• Used for both Rqmts & System Models
2013 Critical Logic
14
Test Models
• Behavioral model + inputs + environment
• Specify tests at executable level
• Automatable
• May or may not include test generator
9/20/2013
8
2013 Critical Logic
15
Hanging the model types together
*Substitutions welcome
MBT Process Menu*
2013 Critical Logic
16
Rqmts
Model
Flavor 1: Model-Centric
• Rare today (yet)
• Generate all
• Big investment
• Big returns
• Maint. costs
• Human error
• Defect escapes
• Defect
introduction
Rqmts
Model
9/20/2013
9
2013 Critical Logic
17
Flavor 2a: Model-Driven
• Less uncommon
• No rqmts model
• Generate
executable tests
from test model
2013 Critical Logic
18
Flavor 2b: Model-Driven (Agile)
• Test model only
• Generate
acceptance criteria
• Maintain model,
not tests
• Supports CIT
9/20/2013
10
2013 Critical Logic
19
Flavor 3: Model-Based
• No test model
• Walk rqmts model
• Validation later
• Add data, env.
2013 Critical Logic
20
MBT Flavor Rainbow
Model “Centric”
• Model-generated tests & code
Model “Driven”
• Model-generated tests
Model “Based”
• Tests based on rqmts models, built by hand
9/20/2013
11
2013 Critical Logic
21
What Test Modeling Tools Can Do
Tool Ingredients
2013 Critical Logic
22
Test Model Tool Functions
Test Data Generators
Test Case Editors
Test Case Generators >> Test Coverage Criteria
9/20/2013
12
2013 Critical Logic
23
Test Coverage Criteria Sys/Rqmt Model Coverage
• Control Flow, Data Flow
• Transition-Based (FSM, statecharts)
• UML-Based (AD, Class, Seq, etc.)
Data Coverage • Boundary Value Analysis
• Classification Tree
• Statistical, Pairwise
Fault-Based Coverage • Mutation, Stuck-At Fault Model, MC/DC
Requirements-Based Coverage • All Requirements Coverage
Utting, M.; Legeard, B.: "Practical Model-Based Testing - A Tools Approach". Morgan Kaufmann Publ, Amsterdam, 2007
2013 Critical Logic
24
Test Modeling Tool Examples
Commercial
• RBT/DTT (Bender RBT, Critical Logic)
• Rhapsody ATG (IBM)
• Others (Conformiq, AETG, Leirios…)
Open Source
• ParTeG (Eclipse Plugin)
• Spec Explorer (Visual Studio)
• Others (JUMBL, Torx, Uppaal TRON…)
9/20/2013
13
2013 Critical Logic
25
More Survey
What kind of MBT tools were used? • 17% Open Source
• 30% Custom
• 53% Commercial
Testing modes used on project (avg)? • 33.6% MBT
• 36.7% Traditional
• 29.7% Manual
Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012
http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf
2013 Critical Logic
26
Coverage What kind of coverage is measured for your MBT test runs?
Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012
http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf
9/20/2013
14
2013 Critical Logic
27
More Survey (last one)
How do you connect MBT output, SUT control interfaces, & SUT observation interfaces?
• 06% Manual input & observation
• 15% Manual scripting required
• 15% Tool generates code/data + manual processing
• 26% Tool generates code/data for the SUT interface
• 38% Tool generates code/data for an adapter that abstracts the SUT interface
Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012
http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf
2013 Critical Logic
28
Tool Challenges Opportunities
Tool Integration • Automation, RMS, TMS, SDLC, TestIF
Coverage limited to source model • Can it synthesize sources to generate tests?
Constraints & infeasibles • Good coverage <> executable tests
Environment Independence • What if the scripting language changes?
9/20/2013
15
2013 Critical Logic
29
Model-Centric Example - CEM
1.Multi-Source
Synthesis
2.RMS
Traceability
3.Manual &
Auto
4.TMS
integration
5.MCDC tests
that run
6.Language
independence
“Test
Mo
del”
2013 Critical Logic
30
Advantages, Challenges, & Benefits
Yeah but…
9/20/2013
16
2013 Critical Logic
31
MBT Advantages 1. Flexibility (env, interface, model basis)
2. Low barrier to entry (80-100 hrs)
[Binder]
3. Measurable, repeatable test coverage
4. Basis for effort estimation
5. Review models, not tests
6. Move validation forward in SDLC
7. Store test definitions as data
8. Integrated test automation
9. Maintain models, not tests
2013 Critical Logic
32
MBT Challenges
• Education of test modelers
• MBT strategy is an intellectual task
• Expectations: models not “automatic”
• Selecting and integrating tool suite
• Integration with existing SDLC • People
• Tools
• Processes
9/20/2013
17
2013 Critical Logic
33
MBT Benefits
1. Requirements improvement
2. Efficient test generation, automation
3. Reduce maintenance costs
4. Models as basis for effort estimation
5. Reduce technical debt
6. Lower cost of quality from rework
7. Reduce defect escapes
8. Reduce defect introduction
9. Data-based result analysis and reporting
2013 Critical Logic
34
Tips to Get Started
• Start small • Pilot low-risk app/change
• Rev. eng. an old complex function
• Parallel = ideal
• Use a vendor • Planning, objectives, strategy
• Delivery, training & mentoring
• Identify demonstration objectives • Test Coverage improvement
• Requirements improvement
• Reduced defect introduction/escape rates
• Scalability, strategy
9/20/2013
18
2013 Critical Logic
35
Discussion
2013 Critical Logic
36
Thank You!
Adam Richards
Critical Logic
Tel: 415-814-9556
9/20/2013
19
2013 Critical Logic
37
References & Further Reading
Ambler, Scott W. Agile Modeling: Effective Practices for Extreme Programming and the Unified Process.
2002
Utting, M.; Legeard, B.: "Practical Model-Based Testing - A Tools Approach". Morgan Kaufmann Publ,
Amsterdam, 2007
Web site of Robert V. Binder, software engineer, expert on testing and automation, source of important
research information in this presentation: http://www.robertvbinder.com/
http://www.robertvbinder.com/open-source-tools-for-model-based-testing/
http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf
Web site of Stephan Weissleder, MBT theoretician, expert, and innovator: http://model-based-testing.de/
Please feel free to contact me directly if you have any questions or simply to further the discussion. Model-
Based Testing is the future!