Upload
wolfgang-grieskamp
View
28
Download
0
Embed Size (px)
Citation preview
Microsoft’s Protocol Documentation Microsoft’s Protocol Documentation Program: Program:
A Success Story for Model-Based TestingA Success Story for Model-Based TestingWolfgang Grieskamp, Software Architect Windows Interoperability Engineering Team, Microsoft Corporation
Keynote given at A-MOST 2011, March 2011, Berlin, Germany.
Part 1: BackgroundPart 1: Background
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 2
Microsoft Microsoft Open Open ProtocolsProtocols
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
http://www.microsoft.com/protocols
9/4/2010 3
Technical Document Testing Program Technical Document Testing Program of Windows (as of 03/09)of Windows (as of 03/09)
222 protocols/technical documents tested 22,847 pages studied and converted into
requirements 36,875 testable requirements identified and
converted into test assertions 69% tested using MBT 31% tested using traditional test automation
66,962 person days (250+ years) Hyderabad: 250 test engineers Beijing: 100 test engineers
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 4
Formalization
Technical DocumentTest Suite
Test Suite
EnglishStatements(Requirements)
AssertionsChecked against Windows(Accuracy)
• Validate consistency
• Simulate implementation process
Test-Driven Document Quality Test-Driven Document Quality AssuranceAssurance
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 5
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Process: Managing a Project with 350 Test Engineers
Review after every phase•High-profile reviewer team consisting of industry experts and Microsoft senior engineers, build along the model of a program committee•Review based on process report and test suite artifacts•Verdict accept, conditional accept, reject
9/4/2010 6
Requirements TrackingRequirements Tracking
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Technical Document
Requirements Spec
Model or Test Design
Test Suite
Logs
Network Captures
Statements identified in document and tracked with requirement ID
IDs referenced from model or test assertion
IDs logged at test execution time
IDs send to network as beacon packages for network monitor capturing
manual
automated
9/4/2010 7
Typical Requirements Specification
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 8
Part 2: Model-Based Test Suite Part 2: Model-Based Test Suite DevelopmentDevelopment
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 9
Model-Based Document AnalysisModel-Based Document Analysis
ModelModel
ImplementationImplementation
Expected Outputs (Test Oracle)
Expected Outputs (Test Oracle)Inputs
(Test Sequences)
Inputs(Test Sequences)
Control
DocumentDocument
Generate
Observe
Feedback
Feedback
Author
Issue
Feedback
Verdict
Feedback
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 10
Process DrilldownProcess Drilldown
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Gather Requirements
Define Actions Define Adapters
Model & Explore
Generate Tests Implement Adapters
Run Tests
9/4/2010 11
Gathering RequirementsGathering Requirements
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Gather Requirements
Define Actions Define Adapters
Model & Explore
Generate Tests Implement Adapters
Run Tests
9/4/2010 12
Developing and Implementing AdaptersDeveloping and Implementing Adapters
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Gather Requirements
Define Actions Define Adapters
Model & Explore
Generate Tests Implement Adapters
Run Tests
9/4/2010 13
Adapter InterfaceAdapter Interface
TCTCTest Cases
(traditional or MBT)TC
TCTC
TC
TCTC
TCTC
TCTC
Adapter ImplementationAdapter Implementation
SUT(System under test)
• Abstracts SUT functionality• Contract between teams
• Test case team• Adapter team
• Abstracts SUT functionality• Contract between teams
• Test case team• Adapter team
• Pluggable• Different server setups• Different transports
• Pluggable• Different server setups• Different transports
Test Adapters
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 14
Develop Model and Generate TestsDevelop Model and Generate Tests
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
Gather Requirements
Define Actions Define Adapters
Model & Explore
Generate Tests Implement Adapters
Run Tests
Potential dependencyon data types
9/4/2010 15
Spec Explorer 2010 Technology Spec Explorer 2010 Technology BreakdownBreakdown Model programs
Guarded state update rules Rich object-oriented model state (collections, object graphs) Language agnostic (Based on .Net intermediate language
interpretation)
Trace patterns Regular style language to represent scenarios Slicing of model program by composition
Symbolic state exploration and test generation Expands parameters using combinatorial interaction testing Extracts a finite interface automaton (IA) from composed model Traverses IA to generate standalone test code –or- Runs on-the-fly tests from IA
Integrated into Visual Studio 2010
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 16
Spec Explorer 2010 Look & FeelSpec Explorer 2010 Look & Feel
Explore
Analyze
Generate
Execute
C# Model C# Model (or other .Net (or other .Net Language)Language)
Model GraphModel Graph
Test SuiteTest Suite
VSTT ResultVSTT Result
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 17
Spec Explorer Walkthrough: Chat ExampleSpec Explorer Walkthrough: Chat Example
Users can Enter the session Exit the session List all session users Broadcast a message
Received by all session users
18
Chat Room Server
User1
SUT
User2 User3
What happens if two users broadcast a message at virtually the same time?
Chat RequirementsChat Requirements
19
R1: User MUST receive response for logon request
R2: User MUST receive response for logoff request
R3: User MUST receive response for list request
R4: List response MUST contain the list of logged-on users if successful
R5: All logged-on users MUST receive broadcast message
R6: Messages from one sender MUST be received in order
Chat tracesChat traces
Is the trace correct?
(Assume two users (user1, user2) are logged on)
20
T1Broadcast(user1,”1”)Broadcast(user2,”2”)BroadcastAck(user2,”1”)BroadcastAck(user1,”2”) BroadcastAck(user1,”1”)BroadcastAck(user2,”2”)
T2Broadcast(user1,”1”)Broadcast(user2,”2”)BroadcastAck(user1,”2”)BroadcastAck(user1,”1”) BroadcastAck(user2,”1”)BroadcastAck(user2,”2”)
T3Broadcast(user1,”1”)Broadcast(user2,”2”)BroadcastAck(user1,”2”)BroadcastAck(user2,”2”) BroadcastAck(user1,”1”)BroadcastAck(user2,”1”)
If each user sends one message, any receiving order is correct!
Chat tracesChat traces
Is the trace correct?
(Assume two users (user1, user2) are logged on)
21
T4Broadcast(user1,”1a”)Broadcast(user1,”1b”)BroadcastAck(user2,”1a”)BroadcastAck(user1,”1a”) BroadcastAck(user1,”1b”)BroadcastAck(user2,”1b”)
T5Broadcast(user1,”1a”)Broadcast(user1,”1b”)BroadcastAck(user1,”1a”)BroadcastAck(user2,”1b”) BroadcastAck(user1,”1b”)BroadcastAck(user2,”1a”)
Local order consistency: messages sent by one user must be received in order
22
23
Spec Explorer’s Conformance NotionSpec Explorer’s Conformance Notion
Alternating simulation: SUT must “simulate” all stimuli of model Model must “simulate” all responses of SUT (responses are buffered)
24
Broadcast(1,”1”)
Broadcast(1,”2”)
Model SUT
Broadcast(1,”1”)
Broadcast(1,”2”)
BroadcastAck(1,”1”)
BroadcastAck(2,”1”)
BroadcastAck(1,”1”)
BroadcastAck(2,”1”)
BroadcastAck(2,”1”)
BroadcastAck(1,”2”)
Spec Explorer’s Approach to State ExplosionSpec Explorer’s Approach to State Explosion
Slicing techniques:
Parameter selection (Constraints, Pairwise, etc.) State filtering Trace patterns Requirement coverage
Slicing requires human intervention Smart slicing is an art
Spec Explorer tailored for slicing design25
Where a Trace can end: Accepting State Where a Trace can end: Accepting State ConditionCondition
Is the following a valid word?Micr
Does the following trace represents a useful test?Broadcast(1,”1”); Broadcast(1,”1”)
An Accepting state condition characterizes those states in which a trace can end Used to ensure that a trace does not stop at arbitrary points Used to ensure that a test leaves the system in a good state
Accepting state condition for Chat: All messages have been delivered to recipients
26
Demo: Chat ModelDemo: Chat Model
What the demo shows OO model state Using helper methods in models Modeling with events Slicing
Demo: Server Message Block Protocol Demo: Server Message Block Protocol v2v2What the demo shows:Modeling a key Microsoft productUsing C# to model state transition systemsUsing trace patterns to slice test purposes from modelsGenerating and executing tests
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 28
Part 3: EvaluationPart 3: Evaluation
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 33
Comparison MBT vs TraditionalComparison MBT vs Traditional
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation
• In % of total effort per requirement, normalizing individual vendor performance
• Vendor 2 modeled 85% of all test suites, performing relatively much better than Vendor 1
9/4/2010 35
Remarks on ComparisonRemarks on Comparison
Numbers backed up by correlation analysis conducted by empirics expert
Engineers applying MBT were lay folks, most of them freshly hired from college
Few power modelers achieve much higher efficiency rates
Though model authoring takes only a smaller part of overall effort per requirement, modeling approach seems to determine efficiency
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 36
Factors for SuccessFactors for Success Tool support
Modeling in mainstream languages and environments
Visualization of exploration and actual generated test suite (no magic as in online testing)
Dealing with state explosion by scenario slicing
Training and Guidance One week modeling classes for every new hire Size of project generated critical mass for
community help Reviewing process added additional guidance
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 37
ConclusionsConclusions
Biggest application of behavioral model-based testing known so far
Probe big enough for relevant empiric results, proving efficiency gain of 42%
Will lead to more momentum on modeling in general inside of Microsoft
Model-based testing rocks!
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 38
Advertisement…Advertisement…
Microsoft offers Spec Explorer 2010 a Visual Studio Powertool via
Licensing allows for commercial use Academic institutions can join MSDN
Academic Alliance for free access to Visual Studio 2010
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 39
© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS,
IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
Microsoft's Protocol Documentation Program: A Success Story for Model-Based Testing Copyright © 2010, Microsoft Corporation9/4/2010 40