Upload
rvbinder
View
58
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Acceptance Test Driven Development (ATDD) and Behavior Driven Development (BDD) are well-established Agile practices that rely on developer intuition to identify and then translate statements into test suites. But the resulting test suites typically cover only a minuscule slice of happy-path behavior. And, as the test code base grows over time, work to maintain it either crowds out new development/testing or, typically, is simply ignored. Either is high-risk. That’s how Agile teams get eaten by the testing BackBlob. Using the Chat Server case study, this presentation provides an overview of a typical BDD tool (SPecFlow). Using a demo of SpecExplorer, a free model-based testing tool, suggests how model-based testing can both achieve better testing and greatly reduce the testing backblob.
Citation preview
Model-based Testing: Taking BDD/ATDD to the Next Level
Robert V. Binder System Verification Associates
#MoreModelsLessTests
Chicago Quality Assurance Association February 25, 2014
Overview
Model Driven Development: Taking BDD/ATDD to the Next Level 1
The Incredible
Vastness of
Testing
Eaten Alive by the
Testing Backblob
Tell Me
A Story
More Models,
Less Tests
Model-
Based
Testing
THE INCREDIBLE VASTNESS OF TESTING
The Incredible
Vastness of Testing
Saved by Revealing Sub-domains
Grains of sand to fill the Grand Canyon
~ 1.68 1025
~ 10 septillion
Trivial Program
Data space: 21024
Sequences: 21024
Execution space: 22048
1.34 10616
Large subsets of any programs execution space are revealing: if a bug exists in any member of that space, the space is reached it will trigger an observable failure. But, we rarely know exactly which sub-domains are revealing until we test.
Model Driven Development: Taking BDD/ATDD to the Next Level 4
Its release time. Do you know where your bugs are?
Average bugs per thousand equivalent source code lines, post-release
Its release time. Do you know where your bugs are?
Mobile
IT
Medical
Aerospace
6.0
4.4
1.8
0.8
6.0
5.0
1.9
0.7 Other Agile
Best In Class 4.7
Capers Jones statistics: 5 defects/FP, 95% Removal. Assumes 53 Java LOC/FP .
Model Driven Development: Taking BDD/ATDD to the Next Level 5
Predicted total latent bugs, 500 KLOC Java
IT
Medical
Aerospace
2202
875
375
2511
925
325 Other Agile
Its release time. Do you know where your bugs are?
Its release time. Do you know where your bugs are?
Best In Class 2350
Capers Jones statistics: 5 defects/FP, 95% Removal. Assumes 53 Java LOC/FP .
Model Driven Development: Taking BDD/ATDD to the Next Level 6
Driven to Distraction
What is it?
How does it work?
What can it do for me?
Test Driven Development TDD
Behavior Driven Development BDD
Acceptance Test Driven Development ATDD
Model-based Testing MBT
Model Driven Development: Taking BDD/ATDD to the Next Level 7
EATEN ALIVE BY THE TESTING BACKBLOB
Eaten Alive
by the
Testing
Backblob
Where does the time go? Total
Number of Test Cases
Sprint 1 Sprint 2 Sprint 3 Sprint 4
Available Dev/TestTime
Manual Test Cases not executed
Automated Test Cases not
maintained
Model Driven Development: Taking BDD/ATDD to the Next Level 9
The Attack of the BackBlob
TELL ME A STORY
Tell Me a Story
The Chat Server
Logon
List active users
Post a message
Broadcast message
Logoff
Client
Endpoint
Client
Endpoint
Chat
Server
!!!
Model Driven Development: Taking BDD/ATDD to the Next Level 13
TDD BDD ATDD
Programming strategies
Test suites are scaffolding
Test each method/feature at least once
Rerun after any add/change
Not testing strategies
Test Driven Development
Behavior Driven Development
Acceptance Test Driven Development
Model Driven Development: Taking BDD/ATDD to the Next Level 14
The TDD Red-Green Cycle
Logon
GetStatus
Logoff
Logon
GetStatus
Logoff
Imagine a class and its responsibilities
Code its API only
Model Driven Development: Taking BDD/ATDD to the Next Level 15
The Red-Green Cycle
TestLogon Logon
GetStatus
Logoff
Imagine a class and its responsibilities
Code its API only
Write and run a simple test it should fail
Model Driven Development: Taking BDD/ATDD to the Next Level 16
The Red-Green Cycle
TestLogon
GetStatus
Logoff
Logon
Imagine a class and its responsibilities
Code its API only
Write and run a simple test it should fail
Code implementation
Rerun all tests expect pass
Model Driven Development: Taking BDD/ATDD to the Next Level 17
The Red-Green Cycle
TestLogon
TestGetStatus
Logon
GetStatus
Logoff
Imagine a class and its responsibilities
Code its API only
Write and run a simple test it should fail
Code implementation
Rerun all tests expect pass
Repeat until feature complete
Model Driven Development: Taking BDD/ATDD to the Next Level 18
The Red-Green Cycle
TestLogon
TestGetStatus
Logon
GetStatus
Imagine a class and its responsibilities
Code its API only
Write and run a simple test it should fail
Code implementation
Rerun all tests expect pass
Repeat until feature complete
TestLogoff Logoff
Model Driven Development: Taking BDD/ATDD to the Next Level 19
TDD Limitations
No explicit linkage for user/customer needs, requirements, or stories
Gaps
UI automation
Feature testing
Components, Test Objects
Process, Activities
Model Driven Development: Taking BDD/ATDD to the Next Level 20
Components
*Unit
Test Objects
*DD Tool
Behavior Driven Development
Feature Files Scenario A
Scenario B
Scenario C
Features, Scenarios
Components, Test Objects Process,
Activities
Model Driven Development: Taking BDD/ATDD to the Next Level 21
System Under Test
*Unit
Test Objects
Test Tooling
Test Scripts
*DD Tool
Acceptance Test Driven Development
Feature Files Scenario A
Scenario B
Scenario C
Features, Scenarios
UI Test Cases
Process, Activities
Model Driven Development: Taking BDD/ATDD to the Next Level 22
Features and Scenarios
Feature: story
Scenario: instance of a story
Steps:
Given: pre-condition
When: trigger
Then: verification condition
Feature: Logon In order to use the chat server, As a validated chat client, I want to submit my credentials and establish a session
Scenario: Submit Good Credentials Given A connection to the server is open When I send the Login message Then I should see the "Hi Chatter" message
Model Driven Development: Taking BDD/ATDD to the Next Level 23
Overview Enter features, scenarios
Generates skeleton test code for each step Extracts step names
Can run immediately
Developer adds step code
Test run reports
CI integration
Model Driven Development: Taking BDD/ATDD to the Next Level 24
Components
Nunit Visual Studio
Test Objects
SpecFlow
Feature Files Scenario A
Scenario B
Scenario C
http://www.specflow.org
1. Create a Feature
As a
I want to
So that
Model Driven Development: Taking BDD/ATDD to the Next Level 25
2. Add Scenarios and Steps
Given Precondition
Data State
When Trigger
API call, GUI action
Then Verification
conditions (expected results)
Model Driven Development: Taking BDD/ATDD to the Next Level 26
3. Add code for steps
Model Driven Development: Taking BDD/ATDD to the Next Level 27
THE VERY MODEL OF A MODERN
Model-
based
Testing
What is a Model?
Focus
Smaller, simpler
In IT systems
Behavior
Structure
Content
Model Driven Development: Taking BDD/ATDD to the Next Level 29
IT Models
Behavior, Structure, Content
Technology Process, Activities
Model Driven Development: Taking BDD/ATDD to the Next Level 30
IT Models
Notional (design)
As built (documentation)
Test Models Focus on aspects that
support testing goals
Behavior
Sequence and time
Transformation
Input, output function
Verification (oracle)
Coverage
Risk Management
Model Driven Development: Taking BDD/ATDD to the Next Level 32
Chat Server Logon
Must be first Cant be repeated
GetStatus Only after logon Returns number of calls since
logon
Logoff Only after logon
Typical RFC status codes
Chat Server
Logon
GetStatus
Logoff
Model Driven Development: Taking BDD/ATDD to the Next Level 33
Simple Test Model nc = 0
GetStatus
Logoff
Logoff
Logoff
assert (401) assert (401) assert (200)
assert (200)
assert (200)
nc++ GetStatus
assert (200) assert (n == nc)
Logon Logon
assert (401)
System Under Test (SUT)
Logon
GetStatus
Logoff
Model Driven Development: Taking BDD/ATDD to the Next Level 34
Chat Server Message Sequence
Login a 1234
Hi Chatter a
Login b secret
Hi Chatter b
List Users
Users: a, b
Client Endpoint Chat Server
!!!
Model Driven Development: Taking BDD/ATDD to the Next Level 35 Client Endpoint
Chat Server Message Sequence
Ack a1 Pumpkin Latte #mypics
Post Pumpkin Latte #mypics
List Users
Users: a, b
Client Endpoint Client Endpoint Chat Server
Bcast a1 Pumpkin Latte #mypics
Logoff b
Ack Logoff b !!!
Model Driven Development: Taking BDD/ATDD to the Next Level 36
Spec Explorer Demo Model-based testing tool
Microsoft Research
Used to test all MSFT APIs
Robust and stable
Visual Studio power tool
C# code, not cartoons
Generates standalone executable test suite
Model Driven Development: Taking BDD/ATDD to the Next Level 37
1. Create Test Model Actions
[Rule] tags a method for exploration
Requires defines a exploration precondition must be true to explore
Model Driven Development: Taking BDD/ATDD to the Next Level 38
2. Define Accepting State(s)
Tag condition(s) that terminate an exploration
Model Driven Development: Taking BDD/ATDD to the Next Level 39
3. Create Cord File
Set options
Define machines
Limits Actions used in an exploration
Similar to regular expression
Composable
Very powerful!
Model Driven Development: Taking BDD/ATDD to the Next Level 40
4. Explore the Model
Select any explorable machine Explore
Generate test suite
Run on-the-fly test
Model Driven Development: Taking BDD/ATDD to the Next Level 41
5. Review Exploration Graph
Shows action/event sequence
Ellipse: model state computed by model program machine
Diamond: observation event get something from SUT (input to the model program)
Right click to inspect any item
Model Driven Development: Taking BDD/ATDD to the Next Level 42
6. Define machine(s) to generate test suites
Set options to generate test code
Use any machine or composite machine
Model Driven Development: Taking BDD/ATDD to the Next Level 43
7. Explore the Test Suite machine
Each root to leaf path becomes a test suite
Model Driven Development: Taking BDD/ATDD to the Next Level 44
8. Generate Test Code
Explores the test machine
Writes out test code using VS testing framework
Model Driven Development: Taking BDD/ATDD to the Next Level 45
9. Generate Test Code
Not intended to be maintained
Re-generate any time
Add helper code in
the model to extend/customize Model Driven Development: Taking BDD/ATDD to the Next Level 46
10. Run the Test Suite
Uses built-in Visual Studio test runner
Adapters
start/stop chat clients and chat server
Other SUT setup/cleanup
Test code sends messages, checks replies
Model Driven Development: Taking BDD/ATDD to the Next Level 47
11. Review Test Results
Detailed run log VS test runner
States
Events
Requirements
Add any other programmable function
Model State (test case inputs)
Event Handler (pass/fail)
Requirement Trace
Model Driven Development: Taking BDD/ATDD to the Next Level 48
Exploration Interactive model development Generate test suites
Spec Explorer Machines define composable
Action sequences
Options control exploration and data generation
Actions are explored
Rules define exploration conditions
Events define expected SUT response
Data structures support exploration
Adapters abstract, control, and observe any interface, any platform
Model Program
Cord Declarations
Machines
Options
Actions Rules
Events
Model Data Structures
Adapters
Model Driven Development: Taking BDD/ATDD to the Next Level 49
MBT Driven Development
SUT
Requirements, Stories, Scenarios...
Bug
Ambiguous, missing, contradictory, incorrect, redundant, incomplete
Missing, incorrect
Expected Outputs (Test Oracle)
Inputs (Test Sequences)
Run
Control Observe
Evaluate
Generate
Test Model
Test Suite
Coverage Requirements Model
Model error, omission
Explore
Model Driven Development: Taking BDD/ATDD to the Next Level 50
MORE MODELS, LESS TESTS
More Models,
Less Tests
Model-based Testing in the Real World
Model Driven Development: Taking BDD/ATDD to the Next Level 52
Requirements Study Modeling Adapter Coding Test Coding Test Execution
42% Less Time Per Requirement
Model-based Testing 1.4 Days/Requirement
Traditional Testing 2.4 Days/Requirement
Windows protocol compliance (Web services, RPC, DCOM)
Total effort: 250 person years (mostly junior SDETs)
Saved 50 person years with model-based testing
Scope of Test Asset Maintenance
Model
Test Code
Adapters
Model-based Testing Behavior Driven Development
Model Driven Development: Taking BDD/ATDD to the Next Level 53
More Models, Less Tests
More better testing-MBT
Rule-based
Systematic and deep exploration
Generate 1000s of scenarios in seconds
Icing the BackBlob
Maintain model
Regenerate test suites
Models grow much more slowly than test code
Model Driven Development: Taking BDD/ATDD to the Next Level 54
Icing the Test BackBlob
Q & A
#MoreModelsLessTests
Model Driven Development: Taking BDD/ATDD to the Next Level 56
Learn More - MBT UCCAT (User Conference on Advanced Automated Testing) Presentations
http://ucaat.etsi.org/2013/program_conf.html
2012 Model-based Testing User Survey
http://robertvbinder.com/wp-content/uploads/rvb-pdf/arts/MBT-User-Survey.pdf
Open Source MBT tools
http://www.robertvbinder.com/open-source-tools-for-model-based-testing/
Overview of MBT, list of commercial MBT tools
http://www.slideshare.net/robertvbinder/model-basedtestingignite
Spec Explorer
http://visualstudiogallery.msdn.microsoft.com/271d0904-f178-4ce9-956b-d9bfa4902745
http://msdn.microsoft.com/library/ee620411.aspx
Microsoft Open Protocol Project
http://cacm.acm.org/magazines/2011/7/109908-microsofts-protocol-documentation-program/fulltext#!
Model Driven Development: Taking BDD/ATDD to the Next Level 57
Learn More: BDD, etc. SpecFlow
http://www.specflow.org/
Using SpecFlow with WatiN
http://msdn.microsoft.com/en-us/magazine/gg490346.aspx
SpecFlow Quick Start
http://www.youtube.com/watch?v=SVwBz-ZTliw
Writing Cucumber Feature Files
https://github.com/cucumber/cucumber/wiki
Model Driven Development: Taking BDD/ATDD to the Next Level 58
Content and Image Sources Grand Canyon Time Lapse. Ezra Kamer http://www.youtube.com/watch?v=NF6O6HuNJ0k The Blob, Theatre Scene. Fairview Productions, 1958. http://www.youtube.com/watch?v=GODDLgM1gKo http://www.imdb.com/title/tt0051418/ Defect densities: Capers Jones, private communication. Robert E. Park. Software Size Measurement: A Framework for
Counting Source Statements. Technical Report CMU/SEI-92-TR-020. Software Engineering Institute, Carnegie Mellon University, September 1992.
SpecFlow http://www.specflow.org/
Contrast (Order and Chaos). M. C. Escher, 1950. National Gallery of Canada http://www.gallery.ca/en/see/collections/artwork.php?mkey=29270 Order and Chaos II. M. C. Escher, 1955. National Gallery of Canada http://www.gallery.ca/en/see/collections/artwork.php?mkey=41980 Spec Explorer. Microsoft http://visualstudiogallery.msdn.microsoft.com/271d0904-f178-4ce9-956b-d9bfa4902745 Spec Explorer Chat Model. Microsoft. The Blob, Final Scene. Fairview Productions, 1958. http://www.youtube.com/watch?v=0GEJgR-bX0w
All other content Copyright Robert V. Binder, 2014.
Model Driven Development: Taking BDD/ATDD to the Next Level 59