Embed Size (px)
Citation preview
Yaxiong Lin
TestOptimal, LLC
Model-based TestingEffective Test Case Design
And Test Automation
Twin Cities Quality Assurance Association (TCQAA)
September 12, 2013
Agenda
What is Model-Based Testing (MBT) MBT in Action (demo) Benefits of MBT MBT Process MBT Use Cases
Testptimal LLC
Agenda What is Model-Based Testing (MBT) MBT in Action (demo) Benefits of MBT MBT Process MBT Use Cases
Testptimal LLC
Business Impacts of Software Defects
Source: Crosstalk, the Journal of Defense Software Engineering
• U.S. alone: $60 billion per year (NIST 2002)
Testptimal LLC
Where Defects are Found
Testptimal LLC
Source Unknown
Costs of Fixing Software Defects
Testptimal LLC
The Real Cost of Software Defects, LKP Consulting Grouphttp://www.lkpgroup.com/Cost%20of%20Software%20Defects.pdf
Software Testing Challenges Requirement/design defect impact: 64% Non-developer testing catches 69% of
defects Costs 100x to fix defect in production Gain higher confidence
better test coveragerequirement traceability
Testptimal LLC
What is Model-Based Testing?“Model-Based Testing is the Automation of Test Design of Black-box Tests”
Testptimal LLC
Practical Model-base Testing”, Mark Utting and Bruno Legeard
What is Model-Based Testing?“Model-Based Testing is the Automation of Test Design of Black-box Tests”
Testptimal LLC
Practical Model-base Testing”, Mark Utting and Bruno Legeard
Model-based testing is application of Model based design for designing and optionally also executing artifacts to perform software testing. Models can be used to represent the desired behavior of a System Under Test (SUT), or to represent testing strategies and a test environment. wikipedia
Why Model?
Visual representation of complex systems Ask What-If to understand cause-effect Important role in the technology
advancement and software development Software testing is a complex task
Testptimal LLC
MBT Modeling Notations Finite State Machine (FSM)
State Diagram (UML) Extended Finite State Machine (EFSM) Message Sequence Chart (MSC)
Sequence Diagram (UML) Control Flow Graph (CFG)
Activity Diagram (UML) Event Flow Diagram UML Testing Profile – UTP …
Testptimal LLC
Testing Vending Machine
Drink for $1 Accept Quarter and HalfDollar coins Cancel any time for refund
Testptimal LLC
Vending Machine Test Cases
Quarter + Quarter + Quarter + Quarter => Vend Quarter + Quarter + HalfDollar => Vend Quarter + HalfDollar + Quarter => Vend Quarter + HalfDollar + HalfDollar => Vend HalfDollar + Quarter + Quarter => Vend HalfDollar + HalfDollar => Vend HalfDollar + Quarter + HalfDollar => Vend Quarter + Cancel => Refund …
Testptimal LLC
Example Model
Testptimal LLC
Generated Test Sequence
Testptimal LLC
Agenda
What is Model-Based Testing (MBT) MBT in Action (demo) Benefits of MBT MBT Process MBT Use Cases
Testptimal LLC
Types of MBT
Offline Testing
Online Testing
Testptimal LLC
Demo:
Test Case Generation for Manual Testing or Offline Testing
Testptimal LLC
Demo:
Login Test Automation – Online Testing
Testptimal LLC
Demo:
WebStore Test Automation – Online Testing
Testptimal LLC
Testing Web App
Testptimal LLC
WebStore MBT Model
© 2012 Testptimal LLC
Generated Test Sequence
Testptimal LLC
Agenda
What is Model-Based Testing (MBT) MBT in Action (demo) Benefits of MBT MBT Process MBT Use Cases
Testptimal LLC
Recent MBT User Surveyby Robert Binder, System Verification Associates,2012
MBT reduced escaped bugs by 59%. MBT reduced testing costs by 17%. MBT reduced testing duration by 25%.
Testptimal LLC
Benefits of MBT
Catch requirement / design defects Catch defects earlier Reduce defects in fields Improve efficiency (Agility)
handling changesbetter communication on defects
Gain higher confidencebetter test coveragerequirement traceability
Testptimal LLC
Agenda
What is Model-Based Testing (MBT) MBT in Action (demo) Benefits of MBT MBT Process MBT Use Cases
Testptimal LLC
MBT Iterative Development Process
Testptimal LLC
Modeling
Test Generation
AutomationExecution
AnalysisMBT
Models
Requirements
User Stories
MBT Process Input/Output
Testptimal LLC
Input
Requirements User Stories
Test Objectives
AUT/Spec
Environment Configuration
Execution Result
MBT Step
Modeling
Test Generation
Automation
Execution
Analysis
Output
Behavior Models
Test Strategies Test Coverage
Automation Scripts
Test Coverage Defects, Perform. Stat
Model Modification
Model-Based Testing Process
Testptimal LLC
http://journal.thedacs.com/issue/52/145Department of Defense (DoD) Information Analysis Center (IAC)
Agenda
What is Model-Based Testing (MBT) MBT in Action (demo) Benefits of MBT MBT Process MBT Use Cases (more demo)
Testptimal LLC
Applications of Model-Based Testing
Conformance Testing Functional Testing System Integration Testing Security Vulnerability Testing
Testptimal LLC
MBT Use Cases (demo)
Finding Temporal Defects Realistic Load Simulation
PerformanceLoad/Stress Testing
Session /Scenario Based Testing Risk Based Testing Quick bug fix verification
Testptimal LLC
Questions?
MBT Papers: http://testoptimal.com/MbtReferences.html
Email: [email protected]
Testptimal LLC
