Upload
bob-binder
View
786
Download
2
Embed Size (px)
DESCRIPTION
Keynote, ETSI Model-Based Testing User Conference. Tallinn, Estonia September 27, 2012. High-level discussion of model-based testing and trends driving software/system reliability. Explains how emergent behavior in complex systems ("dragon kings") causes catastrophic failures. My Multi-dimensional testing strategy can reveal this hard to find bugs/failure modes, but this requires a better approach to model-based testing. Overview: Is software eating the world? Bugs, Black Swans, Dragon Kings. Multi-dimensional Testing. Challenges.
Citation preview
MODEL-BASED TESTING:
TODAY AND TOMORROW
Robert V. Binder
ETSI Model-Based Testing User Conference
Tallinn, Estonia
September 27, 2012
© 2012, Robert V. Binder. All Rights Reserved
Overview
• Is software eating the world?
•Bugs
•Black Swans
•Dragon Kings
•Multi-dimensional Testing
•Challenges
2
“Software is Eating the World”
-Marc Andreessen
3
Wall Street Journal, Aug 20 2011
4
Internet Trends. Mary Meeker, D10 Conference. May 2012
5
Internet Trends. Mary Meeker, D10 Conference. May 2012
6
Internet Trends. Mary Meeker, D10 Conference. May 2012
Embedded and Connected
2015 Connected Device Landscape: The Expanding Computing Continuum.
David Perlmutter, Intel. 2009
7
IT Disruption
Gartner Top Predictions for 2012: Control Slips Away, Daryl Plummer. March 2012
8
9
Internet Trends. Mary Meeker, D10 Conference. May 2012
Is Software really eating the world? Methods/Tools since Y2K
• 2nd gen IDE
• MBT, MDD
• Aspect OP
• Open Source
• UML 2.0
• Cont. Integration
• Web 2.0
• Agile
• ISO 12027
• LAMP Stack
• Many protocols
• HTML 5
• iOS, Android
• Big Data
• Software Def Netwks http://langpop.com/
Relative rank from ~2M data points
10
Engineering Change to Accelerate Product and Systems Development.
Christopher Rommel, VDC Research. August 2012
Is Software really eating the world?
11
Hardware is eating the world
Everything Computes & Connects
• 10x to 100x
• ten years
• 100x to 1000x
• twenty years
Software Methods and Tools
• ?X Productivity
• ?X Quality
• Last ten years?
• Last twenty years?
• Cheaper, better, faster
• Disrupt or die
• Cheaper, better, faster
• “Choose two”
12
Old bugs, new bugs
• No fundamental or
transformative
improvements in
software
• End of the uni-stack
• Multi-stack, mobile,
connected
• Inherently unstable
• Bugs still as likely, more code
• New failure modes
What do you think will happen?
13
Old Bugs
• Incorrect output
• Missing feature
• Abnormal end
• Missed deadline
• Bohr Bugs Predictable, repeatable
• Mandel Bugs Obscure, complex trigger
• Heisen Bugs Disappears if you look
14
The Jargon File
http://www.catb.org/jargon
Bohr Bugs
15
Mandel Bugs
16
Heisen Bugs
17
New Bugs
•Power Law Models
•Black Swans
•Dragon Kings
18
Power Law Models
• Many natural
phenomena
• “Fat Tails”
Dragon-Kings, Black Swans and the Prediction of Crises. Didier Sornette
http://www.youtube.com/watch?v=FlTSbzOvKZI
19
Reliability Growth
The Lognormal Distribution of Software Failure Rates: Application to
Software Reliability Growth Modeling. Robert E. Mullen
20
Black Swans • Very rare
• Surprise
• Cannot predict
• Power law outliers
21
Software Black Swans
• Total eclipse
• Once every 360 years
• Multi-cycle race
conditions in hard
realtime systems
22
Dragon Kings
• Extreme Outlier
Beyond fat tail
• Latent Positive Feedback
• “Phase Shift”
Earthquakes
Material Fractures
Epileptic Seizures
Market Crashes
23
Sornett calls these exceptional events dragon
kings “to stress that we deal with a completely
different kind of animal, beyond the normal,
whose extraordinary characteristics [have]
profound significance.”
Dragon Kings
24
Latent Positive Feedback = Dragon King
25
New Failure Modes:
Amazon Cloud Outage
• Amazon web service Elastic Compute Cloud crashes (power failure)
• The Elastic Load Balancing (ELB) system frantically tries to assign workloads to available servers.
• As Amazon’s cloud rebooted, “a large number of ELBs came up in a state which triggered a bug we hadn’t seen before”
• ELB reaction: try to allocate more larger servers.
• Backlog in the “control plane” results.
• Demand from customers in unaffected availability zones continues
• System swamped and crashes again.
26
What kind of failure was this?
•Why didn’t test find it?
•What would we have to do to find
these kind of bugs?
27
Hic Sunt
Dracones
28
Here be Dragons
Latent positive feedback
External disruptors
Malicious attack target
Partial degradation
Provoked panic
Low testability
Multi-stack
Field-infeasible
Uncontrollable inputs
29
Looking under the Model-based Lampost?
• 9 in 10 MBT testers do
functionality only
• Climbs a tree, not the twigs
• Typically no load test
Logic errors and incorrect
assumptions are inversely
proportional to a path’s
execution probability.
Boris Beizer
30
Multi-Dimensional Testing
Behavior
• Realistic variation
• Markov event/data
• Include
• Fat finger
• Abuse cases
• Sneak paths
• Panics
Interfaces
• Drive all interfaces
• Disrupt resources
Load
• Vary input rate wrt time
Not possible without full automation
31
Test suite interleaves three kinds of variation
Interleave Behavior, Load, Interfaces
1
10
100
1000
10000
100000
1000000
10000000
X
-500.000
0.000
500.000
1000.000
1500.000
2000.000
2500.000
3000.000
3500.000
-5000 0 5000 10000 15000 20000 25000
Ev
en
ts P
er
Se
con
d
Time (seconds)
X
SUT
Test Verdict
Run Reports
Adapter
Adapter
Adapter
Adapter
Oracle
Comparator
Splainer
Simulator
=
32
Use Case A B C D E F G H I J K L
Hunting the Dragon King
• 0 You must be joking
• 1 Write if you get work
• 2 Even a broken clock is right…
• 3 One Sigma
• 4 Michael Jordan free throw (82%)
• 5 Sargent Preston
Common
Bugs
Black
Swans
Dragon
Kings
Fragile 1 0 0
Massively Manual 2 0 0
Test Driven Dev 3 1 0
Exploratory 3 1 0
Crowdsourced 2 0 0
Functional MBT 5 3 1
Multi-dimensional MBT 5 4 3
33
What have we learned?
• Information Technology is eating the world
• Common bug rates will not decline
• More new systems will have dragon kings
• Functional/model coverage MBT will not find
dragon kings
• Multi-dimensional MBT can find dragon kings
• Opportunity for test engineering to lead
34
May you live in interesting times …
Challenges for the MBT Community
Tool Providers
• Start shipping multi-
dimensional
solutions
MBT Test Engineers
• Learn to lead
• Start hunting your
dragon kings
35
Thank you !
36
www.robertvbinder.com
Content and Image Sources
37
First Generation of Re-Imagination
Android Phone Adoption has ramped even faster
Despite Tremendous Ramp so Far, Smartphone User Adoption…
Magnitude of Upcoming Change Will be Stunning…
Internet Trends. Mary Meeker, D10 Conference May 2012
http://kpcb.com/insights/2012-internet-trends
Embedded and Connected
2015 Connected Device Landscape: The Expanding
Computing Continuum. David Perlmutter, Intel. 2009
http://www.intel.com/pressroom/kits/events/idffall_2009/video.htm
IT Disruption
Gartner Top Predictions for 2012: Control Slips Away, Daryl
Plummer. March 2012
Is Software really eating the world?
Engineering Change to Accelerate Product and Systems
Development. Christopher Rommel, VDC Research. August
2012
Mandel Bug
http://www.incamouflage.com/2009/05/bug-camouflage/
Heisen Bug
http://www.adriennehedger.com/2009/07/june-bug-situation.html
http://openclipart.org/detail/145735/cartoon-fly-by-printerkiller
Power Law Models
Dragon-Kings, Black Swans and the Prediction of Crises.
Didier Sornette. International Journal of Terraspace Science and
Engineering 2009
http://www.youtube.com/watch?v=FlTSbzOvKZI
Reliability Growth
R. E. Mullen, “The Lognormal Distribution of Software Failure Rates:
Application to Software Reliability Growth Modeling,” International
Symposium on Software Reliability Engineering, 1998, p 134.
Black Swans
The Black Swan: the Impact of the Highly Improbable. Nassim
Taleb.
http://www.publicdomainpictures.net/view-
image.php?image=8248&picture=black-swan&large=1
Software Black Swans
http://apod.nasa.gov/apod/ap060404.html
Dragon Kings
http://wallpaper4me.com/wallpaper/Dragon-King-of-The-Land/
Latent Positive Feedback = Dragon King
http://www.everythingselectric.com/forum/index.php?topic=244.0
Amazon Outage
Hidden bugs that made Amazon Web Service outage worse. Cnet
News, July 3, 2012.
http://news.cnet.com/8301-1009_3-57465761-83/hidden-bugs-that-
made-amazon-web-service-outage-worse/
Hic Sunt Dracones
http://tomlytle.com/
Looking under the Model-based Lampost?
http://stconsultant.blogspot.com/2010/11/old-saying.html