Embed Size (px)
Citation preview
@ GMU
A Mutation CarolPast, Present and Future
Jeff OffuttSoftware Engineering
George Mason University
Fairfax, VA USA
www.cs.gmu.edu/~offutt/
Mutation Workshop 2009
Denver, CO, USA
4-April-2009
@ GMU Mutation Past – 1970s
Mutation 2009 © Offutt 2
Ghost
of
Mutati
on Past
Originally proposed by Dick Lipton in 1971 as a class project to Dave Parnas
@ GMU Mutation Past – 1970s
• Mutation operators for Fortran and Cobol• Unit testing• Several published papers• Working systems
– PIMS (Fortran), CPMS (Cobol), EXPER (Cobol)• Two PhD theses laid out original theoretical concepts
– Alan Acree (1980-Georgia Tech)– Tim Budd (1980-Yale)
Mutation 2009 © Offutt 3
DeMillo (Georgia Tech), Lipton (Princeton) and Sayward (Yale) started mutation research
@ GMU Mutation Past – 1980s
• Coupling effect (DeMillo, Lipton, Sayward 1978)• Competent programmer hypothesis (DeMillo,
Lipton, Sayward 1978)• Program neighborhoods (Budd’s dissertation, 1980)• Acceptors and generators (Budd’s dissertation,
1980)
Mutation 2009 © Offutt 4
This work developed the fundamental concepts and theory behind program-based mutation analysis
@ GMU Mutation Past – 1980s
• Limited• Not distributed• Only effectively used by developers
Mutation 2009 © Offutt 5
Working systems established that tools could be built to support mutation
@ GMU Mutation Past – 1980s
• Goal was to solve the many engineering problems associated with using mutation in practice
• Mothra was the first widely used working mutation system– Installed at hundreds of universities
and research labs– Source was provided (>100 KLOC)
in C—an early precursor to modern
open source software
Mutation 2009 © Offutt 6
The Mothra project was established in the 1980s to demonstrate practical feasibility
@ GMU Mutation Past – 1980s
• PhD theses : Offutt 1988, Agarwal 1990, Krauser 1991, Wong 1993
• Follow-on PhD theses : Kim 1989, Choi 1991, Untch 1995• Follow-on MS theses : Craft 1989, Seaman 1989, Lee 1991,
Pressley 1992, Zapf 1993, Pan 1994
Mutation 2009 © Offutt 7
Dozens of papers during the project, more after
@ GMU Mutation Past – 1980s
• Most effective mutation operators• Creation and storage of mutants• Definition and storage of test cases• Process and user interface• Test data generation• Parallelization of mutation execution• Comparison with other test technique
Mutation 2009 © Offutt 8
Engineering problems addressed by Mothra
@ GMU Mutation Past – 1980s
Mutation 2009 © Offutt 9
Architecturally, Mothra was designed as a collection of separate programs, integrating around a common
set of data stores with standardized APIs …
@ GMU Mutation Past – 1980s
Mutation 2009 © Offutt 10
The Mothra Tool Set
MOTHRA INTERMEDIA
TE LANGUAGE
MUTANTS
TEST CASES
parser
test case
manager
interpreter
mutant
maker
decoder
EXPECTED
OUTPUT
User
User
killer
P
several UIs
ATDG
@ GMU Mutation Past – 1990s
• Do fewer (fewer mutants)– Selective mutation, mutant sampling
• Do smarter– Weak mutation, distributed execution, different process
• Do faster– Schema-based analysis, separate compilation
• Eliminating manual labor– Automatic test data generation, equivalent mutant detection
Mutation 2009 © Offutt 11
Two problems identified during the Mothra project1. Mutation was too slow2. Mutation was too hard to use
@ GMU Mutation Past – 1990s
• Mutation operators– Modified individual statements– One at a time
• Traditional programming languages– Fortran– Ada– C– Lisp
Mutation 2009 © Offutt 12
Most of this work focused on program unit testing
@ GMU Mutation Past – 1990s
• Interface mutation– mutating function calls, moving beyond the unit level
• Specification mutation– mutating formal specifications, moving beyond the
program
Mutation 2009 © Offutt 13
First “out of the box” ideas for mutation
@ GMU Mutation Past – 1990s
• Time for industry adoption
• But I missed the significance of these two ideas …
Mutation 2009 © Offutt 14
In 2000, I thought mutation research was finished
@ GMU Mutation Present – 2000s
Mutation 2009 © Offutt 15
Ghost of
Mutation
Present
Lots of new tools• muJava• Proteum• Csaw• Certitude• Mu Dynamics• Jumble• PlexTest• Heckle• …
@ GMU Mutation Present – 2000s
• Interface mutation
• Classes• Object-oriented• Real-time• Concurrency
• …
Mutation 2009 © Offutt 16
Mutation has been applied at a variety of program levels, issues and languages
• Java• Ruby
• …
@ GMU Mutation Present – 2000s
• SMV• Z• Object-Z• Algebraic specs
• …
Mutation 2009 © Offutt 17
Several formal specification languages
@ GMU Mutation Present – 2000s
• XML• Statecharts• Activity diagrams• Input languages• SQL• HTML• Spreadsheet formulas• …
Mutation 2009 © Offutt 18
Eventually to other software artifacts and models
@ GMU Mutation Present – 2000s
• Security• Reliability• Complexity measurement• …
Mutation 2009 © Offutt 19
Not to mention problems other than testing
@ GMU Mutation Present – 2000s
Mutation 2009 © Offutt 20
Aggregating Realization
We are performing mutation analysis whenever we• use well defined rules• defined on syntactic descriptions• to make systematic changes• to the syntax or to objects developed from the
syntax
Diversity ViewMutation is one version of syntax directed
testing, which finds tests that cover a space defined by a grammar
@ GMU Mutation Present – 2000s
• Successful commercialization– Certitude by Certess tests integrated circuit designs in
VHDL or Verilog– PlexTest by ITRegister tests C++
• Use by non-researchers• And in Wikipedia
– http://en.wikipedia.org/wiki/Mutation_testing
Mutation 2009 © Offutt 21
Mutation has entered the mainstream !
@ GMU Mutation Present – 2000s
Mutation 2009 © Offutt 22
@ GMU Mutation Future – 2010++
Mutation 2009 © Offutt 23
Ghost of
Mutation
Future
@ GMU Mutation Present – 2010s
Mutation 2009 © Offutt 24
Prediction is difficult –
especially about the future— Niels Bohr
But damn fun !!
— Jeff Offutt
@ GMU Mutation Future – 2010s
• Developers do not want to understand mutation– They just want good tests
• Developers do not want to understand testing– They just want to find problems with their software
• Tool must ignore theoretical problems of completeness and infeasibility– Developers do not care about equivalent mutants– Developers do not care about the mutation score
Mutation 2009 © Offutt 25
(1) Mutation must be integrated with development
Developers just want to know when software fails
@ GMU Mutation Future – 2010s
Mutation 2009 © Offutt 26
compiler
syntax
errors
P
editor
mutationseman
ticfailure
s
Why do theyhave to here?
Users do not have to get
involved with this process
@ GMU Mutation Future – 2010s
• What kinds of faults is mutation good at finding ?– … particularly bad at finding ?
• What kinds of faults do we really care about ?– … do we not care about ?
• Java example : If you override equals(), you must override hashcode() ?– Can a mutant catch that fault ?
• Many OO mutants cannot be killed unless the tester subclasses or otherwise uses the mutated class
Mutation 2009 © Offutt 27
(2) What kinds of faults can we detect with mutation?
@ GMU Mutation Future – 2010s
Mutation 2009 © Offutt 28
Quantitative measures• Maintainability• Performance• Size of the resulting
program• …
(3) Mutate for improvement, not correctness
model of software
mutate
mutated model 1
mutated model N
mutated model 2 assess tradeoffs
@ GMU Mutation Future – 2010s
• What can we model and what can’t we model ?
• How do we define mutation operators ?– What theory for how to define mutation operators ?
• Some data indicate that our mutation operators are inefficient—what is a minimalist approach ?
Mutation 2009 © Offutt 29
(4) Deeper theory of mutation operators
@ GMU Mutation Future – 2010s
• What else can we model with grammars ?
• What additional problems can be solved with mutation ?
• How else can we use mutants ?
Mutation 2009 © Offutt 30
(5) What else can we mutate ?
@ GMU
© Offutt 31
A Contrarian View
Jeff Offutt
http://cs.gmu.edu/~offutt/Mutation 2009
If we have mutation, do we need other test criteria ?
