CS527Topics in Software Engineering(Software Testing and Analysis)
Darko Marinov
September 15, 2011
Schedule• First few lectures to help you select projects
– Shared memory: CHESS, IMUnit, CAPP– Message passing: Setac– Comment analysis: iComment– Regression testing: survey paper– Today: Model-based testing– Next week two papers
• 3 votes: generation/evaluation of test assertions• 2 votes: mock-based testing, GUI testing• 1 vote: genetic algorithms, concolic testing, web testing
• Your project can be something we don’t cover
Paper Today• Model-based
quality assurance of protocol documentation: tools and methodology Wolfgang Grieskamp, Nicolas Kicillof, Keith Stobie and Victor Braberman (STVR 2010)
• Journal paper– Not too long– Otherwise, we prefer conference papers
• Follows: personal slides on models for data, slides on MBT, questions
• Thanks to Wolfgang Grieskamp for MBT slides
Developed Two Kinds of Test Abstractions
• Declarative: describe what inputs look like– Two kinds of languages used
• Declarative language for properties of desired inputs[SOFTMC’01, ASE’01, FME’02, OOPSLA’02, SAT’03, MIT’03, J-ASE’04, SAT’05, LDTA’06, ALLOY’06]
– Properties written in the Alloy modeling language– Uses Alloy Analyzer for generation of test inputs
• Imperative language for properties of desired inputs[ISSTA’02, TR’03, MIT’04, ICSE-DEMO’07, STEP’07, FSE’07]
– Properties written in implementation language (Java, C#…)
– Developed Korat for generation of test inputs
• Imperative: describe how to generate inputs– More recent work on ASTGen [FSE’07]
Korat at Microsoft Research• Korat implemented in the AsmLT test tool
in Foundations of Software Engineering group– Predicates in Abstract state machine
Language (AsmL), not in Java or C#– GUI for setting finitization and manipulating
tests– Korat can be used stand-alone or to
generate inputs for method sequences– Some extensions
• (Controlled) non-exhaustive generation• Generation of complete tests from partial tests• Library for faster generation for common
datatypes
AsmLT/Korat at Microsoft• Used by testers in several product groups• Enabled finding numerous errors
– XML tools• XPath compiler (10 code errors, test-suite augmentation)• Serialization (3 code errors, changing spec)
– Web-service protocols• WS-Policy (13 code errors, 6 problems in informal spec)• WS-Routing (1 code error, 20 problems in informal spec)
– Others• SSLStream• MSN Authentication• …
• Errors found in– Important real-world applications– Code already well tested using best testing
practices
Some Comments from Microsoft• Positive comments on AsmLT and Korat
– “So far our stateless AsmL models are pretty successful.”
– “AsmL parameter generation tool is quite convenient and powerful.”
• Negative comments on AsmLT not Korat– “Most of our testers prefer to write as much
C# as possible.”– “Very difficult to debug AsmL.”
• Result: new tool for C# (.NET) SpecExplorer
Korat is
Korat
Questions for Discussion (1)• Are there any open source tools equivalent to
Spec Explorer? [KN]• How would this methodology change if
applied to something but protocols? [JT]• In the case where the number of internal unit-
tests that need to be tested, is it worth it at all to do MBT? [AY]– What's the right time to apply model-based
testing? [XW]• What other selection strategies exist but
transition coverage? [AK]
Questions for Discussion (2)• … possible to scale up MBT to make it work
on large-sized systems? [AD]• Does model based testing scale down to
small projects? [DeG]• NLP … to aid in … converting technical
documentation into testing reqs? [CM]– Is it possible to reverse engineer a spec from a
program or test suite? [SB]• test models generated automatically? [YL]• How to totally/partially automate slicing? [FS]
– How does the slicing affect the success? [SO]
Questions for Discussion (3)• integrate with regression testing? [KB]• generate a test that checks to see if an action
disallowed by the model program is in fact disallowed by the System Under Test? [DwG]
• Can Spec be used to generate proprieties for Chess? [SB]
• How are bugs in the model treated? [AK]• Is the modeling language enough to model
various types of applications? [SO]