Upload
softwarecentral
View
1.776
Download
0
Embed Size (px)
Citation preview
MDSE – Model Driven Software Engineering
Model-based Testing
1Model Driiven Software Engineering MAP-I, Ana Paiva, 11/11/2008
Model-based Testing
Ana Paiva
[email protected] www.fe.up.pt/~apaiva
Agenda
� Introduction
� Model-based testing
• Model your system
• Test coverage criteria and Test case generation
• Test case execution
2Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
• Test case execution
� A model-based testing tool
� Research Project
Testing� Testing is the activity performed for evaluating a
product quality, and for improving it, by identifying
defects and problems [SWEBOK 2004]
� …consists of the dynamic verification of the behaviour of
3Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� …consists of the dynamic verification of the behaviour of
a program on a finite set of test cases, suitably selected
from the usually infinite executions domain, against the
expected behaviour
� Testing executes software in order to detect failures and
then tries to identify and fix the faults that cause them
Fault vs Failure
� By definition, a fault is a structural imperfection in a software
system that may lead to the system's eventually failing
� A fault is the cause of a failure. It is the execution of the
faults in software that causes failures
4Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
faults in software that causes failures
� Tester look for failures and then try to identify the faults
which cause those failures.
Software quality properties
� External Properties: The user’s perspective:
• Satisfaction: subjective view (pleasant, comfortable, intuitive, consistent)
• Reliable: refers to the errors a user can do when using the system
• Learnability: time taken to learn how to use the system
• Efficiency: how efficient a user can be when using the system
5Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� Internal Properties: The software engineering’s perspective:
• Code: readable, style,…
• Architecture: influences the degree of manageability and scalability
• Run time efficiency: related to the complexity of the algorithms
• Correctness: Verification (meets specification) and validation (meets users’ requirements)
[GC96]
General test (V&V) strategies
� Manual vs automatic
� Static vs dynamic
� White vs Black-box
� Error/Fault based vs specification-based
6Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� Error/Fault based vs specification-based
� API vs UI testing
� Functional vs non-functional (Usability, Performance,
Robustness)
Different kinds of tests: V-Model
7Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
Model-based testing
Agenda
� Introduction
� Model-based testing
• Model your system
• Test coverage criteria and Test case generation
• Test case execution
8Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
• Test case execution
� A model-based testing tool
� Research Project
Model-based testing
� Model-based testing is software testing in which test cases are derived in whole or in part from a model that describes some (usually functional) aspects of the system under test (SUT)
[UL07]
9Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� Model-based testing uses a model program to generate test cases or act as an oracle. Model-based testing includes both
• offline testing and
• on-the-fly (online) testing
� An oracle is the authority which provides the correct result used to make a judgment about the outcome of a test case - whether the test passed or failed
[A-site]
Why do we need MBT?� Unit testing is not enough
� We need meaningful sequences of actions calls that provide systematic behavioral coverage
10Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� In software industry model-based testing is being adapted as an integrated part of the testing process
� In academia several model-based testing conferences have been started
� Several commercial tools have been developed for model-based testing
Model-based testing� Advantages
• Higher degree of automation (test case generation)
• Allows more exhaustive testing
• Good for correctness/functional testing
• Model can be easily adapted to changes
Disadvantages
11Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� Disadvantages
• Requires a formal specification/model
• Test case explosion problem
• Test case generation has to be controlled appropriately to
generate a test case of manageable size
• Small changes to the model can result in a totally different test
suite
• Time to analyse failed tests (model, SUT, adaptor code)
Model-based testing process
system under test (SUT)
sw model1.Modelling
2.Test case generation
Abstract Test suite
Map
Concrete 3.Test case
Input data;
Outputs expected (oracle)
Requirements traceability matrix; model coverage
SUT coverage
12Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
4.Test case execution
Bugs report
Concrete Test suite
3.Test case concretization
SUT coverage
5. Analysis
Model-based testing challenges
system under test (SUT)
sw model1.Modelling
2.Test case generation
Abstract Test suite
Map
Concrete 3.Test case
Effort & guidelines
Test case explosion
Test adequacy
13Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
4.Test case execution
Bugs report
Concrete Test suite
3.Test case concretization
5. Analysis
Model to implementation gap
Model your system
� Design the model to meet your testing goals
� Choose the right level of abstraction (which aspects of SUT
you want to test)
� You can have a many-to-many relationship among the
operations of your model and the operations of the SUT
14Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
operations of your model and the operations of the SUT
� Choose a notation for modelling
� Once the model is written, ensure that it is accurate (validate
and verify your model)
[UL07]
Models: Built or Borrow?
� Reuse the development models – 100% reuse
• Too much detail
• Usually do not describe the dynamic behaviour of the SUT
• It is neither abstract enough nor precise
� Models to generate code or reverse engineered
15Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� Models to generate code or reverse engineered
• Lack of independence (implementation and test cases are derived from
the same source code)
� Develop test models from scratch – 0% reuse
• Maximum level of independence
[UL07]
Notations
� Pre/Post (or Model-based). Ex.: VDM, Z, Spec#.
� Transition-based. Ex.: FSM, Petri nets.
� Behavior-based (or history-based). Ex.: CSP, MSC.
16Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� Property-based (or functional-based). Ex.: OBJ.
� Hybrid approaches. Ex.: RAISE.
� …
� UML with OCL?
• pre/post, Set, OrderedSet, Bag, Sequence, and associations among
classes
Pre/Post: VDM++class Stack instance variablesstack : seq of int := []; -- inv
operationsStack : () ==> () Stack () == stack := []post stack = []; Push : int ==> ()Push(i) == stack := [i] ^ stack
17Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
Push(i) == stack := [i] ^ stackpost stack = [i] ^ ~stack; Pop : () ==> ()Pop() == stack := tl stackpre stack <> [] post stack = tl ~stack; Top : () ==> intTop() == return (hd stack)pre stack <> [] post RESULT = hd stack
and stack = ~stack; end Stack
Transition-based: FSM
� Choosing the set of states is a critical step
Len s = 0 Len s != 0
Push(k)
Pop
Pop
18Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� Statecharts are another option
Behaviour-based: CSP
Stack(< >) = push?x:E -> Stack(<x>)
Stack(<y>^s) = push?x:E->Stack(<x>^<y>^s)
| pop!y -> Stack(s)
19Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
OR
Stack() = push?x;Stack(x)
Stack(x) = push?y; Stack(y);Stack(x)
| pop!x;Stack()
Property-based: OBJSpec: Stack;
Extend Nat by
Sorts: Stack;
Operations:
newstack: → Stack
push: Stack × Nat → Stack
pop: Stack → Stack
20Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
pop: Stack → Stack
top: Stack → Nat
Variables:
s: Stack; n: Nat
Axioms:
pop(newstack) = newstack;
top(newstack) = zero;
pop(push(s,n)) = s;
top(push(s,n)) = n;
Choosing a notation
� Pre/Post (for modelling complex data) and transition-based
(for modelling control) are the most common notations used in
model-based testing processes
Whatever notation you choose, it has to be a formal language
21Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� Whatever notation you choose, it has to be a formal language
with precise semantics in order to write accurate models so as
to be used as test oracles
Agenda
� State of the art
� Model-based testing
• Model your system
• Test coverage criteria and Test case generation
• Test case execution
22Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
• Test case execution
� A model-based testing tool
� Research Project
Test suite
� Key Point – In this context, a test suite contains the sequence
of actions to perform, the input data and the results expected
The best test suit is the smallest one which is capable of
23Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� The best test suit is the smallest one which is capable of
finding the maximum number of bugs.
� A good test suite should combine a good code coverage with a
good requirements (or specification) coverage
Coverage criteria and analysis
� Coverage criteria help you generate a good test suit and determine when to stop testing but, above all, your expert knowledge of the SUT is the key factor for success
� Coverage analysis aims to measure the extent to which a
24Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� Coverage analysis aims to measure the extent to which a given verification activity has achieved its objectives and can be used to evaluate the quality of the test suite used and also determine when to stop the verification process. It is usually expressed as a percentage referring to the accomplished part of an activity
[UL07]
Coverage criteria
� 1.Structural coverage criteria – aims to exercise code or model concerning some coverage goal
� 2.Data coverage criteria – aims to cover the input data space of an operation or transition
� 3.Fault-based criteria – aims to generate test suites
25Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
3.Fault-based criteria – aims to generate test suites appropriate for detecting specific kinds of faults
� 4.Requirements coverage criteria – aims to ensure that each requirement will be tested
� 5.Others
[UL07]
1.Structural coverage criteria
� Statement coverage (SD) – every executable statement is invoked at least once;
� Decision coverage (DC) – expression outcomes are tested for true and false (e.g., (A or B) tested for TF and FF);
� Condition coverage (CC) – each condition within expression takes all possible outcomes (e.g., (A or B) tested for TF and FT);
26Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� Decision/condition coverage (D/CC) – combines the two previous criteria (e.g., (A or B) tested for TT and FF);
� Modified condition/decision coverage (MC/DC) – each condition affects independently the outcome of the decision (e.g., (A or B) tested for TF, FT, and FF);
� Multiple condition coverage (MCC) – test each possible combination of inputs. Test 2n for a decision with n inputs (almost of the times is unfeasible);
� … [HVCKR01]
1.Structural coverage criteria
Multiple Condition Coverage (MCC)
Modified Condition/Decision Coverage (MC/DC)
Full Predicate Coverage (FPC)
strongest+
27Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
Full Predicate Coverage (FPC)
Decision/Condition coverage (D/CC)
Decision converage (DC) Condition Coverage(CC)
Statement Coverage (SC) [UL07] -
1.Structural coverage criteria: FSM
All-Paths
All-One-Loop-Paths All-Transition-Pairs
strongest+
28Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
All-Round-Trips All-Configurations
All-Loop-Free-Paths All-Transitions
All-States
[UL07]
-
2.Data coverage criteria
� Choose variables’ values from their domains
•One value (almost of the times not enough)
• Smart choice of values
- Equivalence class partitioning (e.g., type-based)
- Boundary value analysis
Does not generate a test sequence of actions
29Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
- Boundary value analysis
- Randomly value generation
- Goal oriented methods (e.g., AI Planning)
- Path oriented methods
• All values (almost of the times unfeasible)
[UL07]
actions
2.Equivalence class partitioning
� A partition of some set, S, is a set of non-empty subsets
SS1, ..., SSn, such that each SSi and SSj are disjoint, and
the union of all SSi's equals S
30Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� If a defect is detected by one member of a class, it is
expected that the same defect would be detected by any
other element of the same class
all inputs
i1
i4i2 i3
2.Boundary value analysis
� Boundary value analysis tests boundary conditions of equivalence classes choosing input boundary values. This technique is based on the knowledge that input values at the boundaries or just beyond the boundaries of the input domain tend to cause errors in the system
31Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
• Test cases :
- class x < 0, arbitrary value: x = -10
- class x >= 0, arbitrary value x = 100
- classes x < 0, x >= 0, on boundary : x = 0
- classes x < 0, x >= 0, below and above: x = -1, x = 1
“Bugs lurk in corners and congregate at boundaries.”
[Boris Beizer, "Software testing techniques"]
2.Goal oriented
� Goal-oriented methods try to drive the system into a
given goal by two different methods: the chaining
approach and assertion-oriented approach
• The first one tries to find a path to the execution of a given goal
node based on data dependence analysis
32Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
• The second tries to find any path to an assertion that does not
hold
2.Path oriented
� E.g., Symbolic testing. It replaces program variables by symbols and
calculates constraints that represent possible symbolic execution paths.
When a program variable is changed during execution, the new value is
expressed as a constraint over the symbolic variables. A constraint solver
system can be used to find, when possible, concrete values that cause the
execution of the path described by each constraint
int x, y;
33Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
int x, y;
if (x > y) {
x = x + y;
y = x – y;
x = x – y;
If (x - y > 0)
assert false
}
x = X, y= Y
X > ? Y
[X<=Y] END [X>Y] x = X + Y
[X>Y]y = X+Y-Y =X
[X>Y] x = X+Y-X = Y
[X>Y] Y-X >? 0
[X>Y, Y-X <= 0] END [X>Y, Y-X>0] END
3.Fault-based criteria
� Mutation testing - Mutation techniques introduce small changes (faults) by applying mutation operators into the original specification (e.g., arithmetic operators
� The changed specifications are called mutants
� The goal is to construct test cases that distinguish each mutant from the original by producing different results. If that happens, it is said that the test case has killed the mutant
34Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� A good test case should be capable of killing the mutants because if it is able to detect the small differences generated by the mutation operators it is expectable that it will be good at finding real faults
� The rate of killed mutants (after removing mutants that are equivalent to the original code) gives an indication of the rate of undetected defects that may exist in the original code
� One of the problems of mutation testing is the incapacity of the technique to generate test data
[W03]
4.Requirements coverage criteria
� Requirements coverage is usually less systematic and usually those not contain the full specification of the behaviour of the system. However, there are at least two approaches that try to systematize it more:
• Record the requirements inside the behaviour model (as annotations on several parts of the model) so that the test generation process can ensure that all requirements have been tested
35Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
ensure that all requirements have been tested
• Formalize each requirement and then use that formal expression as a test selection criterion to drive the automated generation of one or more tests from the behaviour model
� Ex.: Coverage criterion measuring the degree in which use cases or scenarios were tested. Scenarios describe how the system and the user should interact to achieve a specific goal. They usually refer to common usages of the system and may not be a full description of the behaviour of the system
5.Model-checking� Whenever a property, expressed in temporal logic, does not hold in
a system described as a FSM, model-checking tries to generate a counter-example
� When a counter-example is produced, it can be used as a test case –sequence of transitions in the FSM with inputs and expected outputs
� To be effective as a test-case generation technique, the properties about the system should be described in such a way that counter-
36Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
about the system should be described in such a way that counter-examples produced by them can be used as test cases
ModelChecker
Property: G(x -> F y)
yesno
x = T,T,F,F,...y = F,F,F,T,...
System model (FSM)
[ABM98]
5.Test case generation from property-based models
� Common techniques used to generate test cases from
these specifications are rewriting and constraint solving
� Given a set of expressions (logical assertions or
37Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
Given a set of expressions (logical assertions or
equivalence relations) and the set of variables within
those expressions, constraint solving techniques try to
find an instantiation of the variables which reduce the
expressions to true - E.g., [X>Y, Y-X>0] = impossible!
5.Test case generation from behavioural-based models
� Analyses of the execution traces to generate test cases.
A trace in CSP is a finite sequence of events
• push?x;push?y;pop!x;push?x
Another example of test case generation from CSP
38Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� Another example of test case generation from CSP
specifications is illustrated in [BS02]. The goal is to test
Universal Mobile Telecommunications Systems (UMTS).
They start by constructing a transition graph with all
possible interleaving and parallel tasks. Then, a test
driver computes all paths through this graph that are
used as test sequences
Test case generation overview
� Depends on the specification characteristics:
•Pre/Post. Ex.: VDM, Z, Spec#- Partition testing- Generate FSM from the model. Ex.: Spec Explorer.
•Transition based. Ex.: FSM, EFSM, Statecharts.- Traversal algorithms (state and transition coverage)
39Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
- Traversal algorithms (state and transition coverage)- Model checking (algorithms based on state exploration; mutation analysis)
•Property-based. Ex.: Obj- Rewriting rules- Constraint solving
•Behaviour-based. Ex.: CSP- Trace analysis
Agenda
� State of the art
� Model-based testing
• Model your system
• Test coverage criteria and Test case generation
• Test case execution
40Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
• Test case execution
� A model-based testing tool
� Research Project
Test suite “concretization”
Test Cases
Adapter
abstract
Test Cases
Test Cases
Level of abstractio
41Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
Adapter
SUTconcrete
SUT
Adapter
SUT
Test Scripts
Test Scripts
a) Adaptation b) Transformation c) Mixed
abstraction
[UL07]
Test suite execution � “Lock-step” mode in which results are compared after each step
� “Batch-oriented” way, in which case the test suite is run as a whole in the specification level, and expected results are kept in memory for later comparison with the results obtained from the execution of the implementation (which is performed in a different execution time instant)
• One advantage of the batch-oriented way is the need to execute
42Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
• One advantage of the batch-oriented way is the need to execute the model only once and not every time test cases are executed. The main drawback is the additional need of memory to keep the results expected
� "On-the fly testing" combines in a single algorithm the test case generation and execution and executes each operation as a “lock-step” in each level comparing results after each of those execution steps
Agenda
� State of the art
� Model-based testing
• Model your system
• Test coverage criteria and Test case generation
• Test case execution
43Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
• Test case execution
� A model-based testing tool: Spec Explorer
� Research Project
System Testing with model programs
System Design
System Test
Spec Explorer
44Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
Microsoft Research (FSE) : Colin Campbell, Wolfgang Grieskamp, Microsoft Research (FSE) : Colin Campbell, Wolfgang Grieskamp, Yuri Gurevich, Lev Nachmanson, Wolfram Schulte,Yuri Gurevich, Lev Nachmanson, Wolfram Schulte,Nikolai Tillmann, Margus VeanesNikolai Tillmann, Margus Veanes
Model-Based Testing
Model
Pass
Provides expected results forGenerates
User
45Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
Implementation
Test OracleTest Cases
Are run by
Providesactual results for
PassFail
UserInfo
System testing� Traditional unit testing is not enough
• Single unit may work properly in isolation
• Incorrect interaction between units may cause serious
security/reliability failures
� System-level testing
46Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
• Requires model of system behavior
• Behavior is often reactive/nondeterministic
- Implementation is multi-threaded or distributed
• State space is typically infinite
- Objects, unbounded values
• Traditional FSM-based testing does not work
Microsoft approach
� Behavior is described by model programs
• Written in Spec#
• Describe reactive system behavior as interface automata
• Use alternating refinement for conformance
47Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� Model exploration, validation and model-based testing are provided by the Spec Explorer tool developed at MSR
• Supports scenario control
• Provides offline as well as online testing
• Views test cases as game strategies
• Checks violations of alternating refinement
[CGNSTV05]
How it works
� Modeling
• Define (infinite) transition system TP through program P
� Exploration
• Reduce TP to a finite test graph G
Test generation
48Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� Test generation
• Generate test cases from G
� Test execution
• Run the test cases using the model as the oracle
To avoid state space explosion, exploration, test generation and execution can be combined into a single on-the-fly algorithm
Modelling
� In Spec#/Spec Explorer:
• States are mappings of variables to values - ASM states or first-order structures
• Initial state is given by initial assignment to model variables
• Actions are defined by method invocations
• Preconditions of methods and model invariants define action enabling conditions
49Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
conditions
• Transition function is defined by method execution
Exploration
� Exploration is the process of unfolding a rule-based model
program into a transition system
• Actions move the system from state to state
• State comes from variables containing values
• Compound data types are possible (sets, maps, sequences, etc…)
• Objects are identities
50Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
• Objects are identities
State space of a stack model
� Stack model
var Seq<int> content = Seq{};var Seq<int> content = Seq{};var Seq<int> content = Seq{};var Seq<int> content = Seq{};
[Action][Action][Action][Action]public void Push(int x) {public void Push(int x) {public void Push(int x) {public void Push(int x) {content = Seq{x} + content;content = Seq{x} + content;content = Seq{x} + content;content = Seq{x} + content;
}}}}
[Action][Action][Action][Action]
[]
[0] [k]
Push(0) Push(k)Pop PopTop/0
Top/k
51Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
[Action][Action][Action][Action]public void Pop() {public void Pop() {public void Pop() {public void Pop() {requires !content.IsEmpty;requires !content.IsEmpty;requires !content.IsEmpty;requires !content.IsEmpty;content = content.Tail;content = content.Tail;content = content.Tail;content = content.Tail;
}}}}
[Action] [Action] [Action] [Action] public int Top() {public int Top() {public int Top() {public int Top() {requires !content.IsEmpty;requires !content.IsEmpty;requires !content.IsEmpty;requires !content.IsEmpty;return content.Head;return content.Head;return content.Head;return content.Head;
}}}}
[0] [k]
[0,k] [l,k]
PopPush(0) Push(l)
Pop
Top/0Top/l
Controlling exploration - 1
� In general the transition system is infinite, but we can impose limits
� Goal
• Create a state space of manageable size that satisfies a given testing goal
52Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� Two main tasks
• Restrict action parameter domains to interesting values
• Restrict the state space to interesting states
� Note: The two tasks are not necessarily independent!
Controlling exploration - 2
� The following techniques are used in Spec Explorer
• State filters
• Stopping conditions
• State groupings
• Search order
• Enabling conditions on actions
53Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
• Enabling conditions on actions
� Usually a combination of all (or some) of the
techniques is needed to achieve desired effect
State Groupings
� One or more state groupings may be defined
� A grouping G is a sequence of state based expressions g1,…,gk
� Two states s and t are in the same group G or G-equivalent if
54Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
G-equivalent if
• gis = gi
t for 1 ≤ i ≤ k
� A G-group is the set of all G-equivalent states
� Similar to predicate abstraction in model-checking (if grouping expressions are Boolean)
� Also used in viewing
Main purpose of groupings
� View all states in a G-group as being the same
� A way to define “what is an interesting state” from some
testing point of view
55Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� Example: content.Size in the stack model
Len s = 0 Len s != 0
Push(k)
Pop
Pop
Test case generation
_
FSM generation by
bounded exploration
Bounds
FSM
SUT (Spec# or AsmL)
56Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
Test suite
Coveragecriteria
Test casegeneration
Spec Explorer
Set bounds: state filters; additional pre-conditions; restriction of the domains; equivalence classes; stop conditions; scenario actions
Choose coverage criteria: full transition coverage; shortest path; random walk
Test generation
� The result of exploration is a finite transition system
� This connects us to the world of model-based testing
• Most formal approaches to testing in the literature are based on
transition systems
� Traversal underlies most techniques for generating tests from
57Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� Traversal underlies most techniques for generating tests from
automata
Test execution
� Generated test cases are executed in the implementation
under test
� The model acts as the oracle
58Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
Demo: stack
[]
[0] [k]
Push(0) Push(k)
Push(0) Push(l)
Pop PopTop/0Top/k
Len s = 0 Len s != 0
Pop
Pop
59Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
[0,k] [l,k]
PopPush(0) Push(l)
Pop
Top/0Top/l
Len s = 0 Len s != 0
Push(k)
Online Testing of Reactive systems
Constrained model
IUT wrapper
IUT
Refinement checker
Controllable actions
Controllable actions
Observableactions
Observable actionsd1
……c1
Object mapping
Component View
60Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
Model
Pass/Fail
d1……
c1
Example: Chat service
IUT wrapper
IUT
Chat
Controllable actions:
Login …SendRefinement checker
Constrained chat model
b.Send(“hi”)
m.Send(“hello”)
b.Send(“bye”Object mapping
61Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
at server
Observable actions: Receive
b.Send(“bye”)
m.Rcv(“hi”)
m.Rcv(“bye”)
b.Rcv(“hello”)b.Rcv(“hello”
)
m.Rcv(“bye”)
m.Rcv(“hi”)
b.Rcv(“hello”)
BobMarym
b
mapping
Refinement relation
� Conformance between the model and the system under test is
alternating refinement
b.Send(“hi”)
Spec System
b.Send(“hi”)
62Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
m.Send(“hello”) m.Send(“hello”)
m.Receive(b,“hi”)
b.Receive(m,“hello”)b.Receive(m,“hello”)
m.Receive(b,“hi”) b.Receive(m,“hello”)
Example: Valid chat scenario
Bob Server Mary
Service should guarantee local consistency of message delivery (FIFO wrt sender)Send is a controllable action
Receive is an observable action
63Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
Bob Server Mary
Time
Model program : Chat service
State:
Actions:
class Client { bool entered = false;Map<Client, Seq<string>> unreceivedMsgs = Map{};}
class Client {void Send(string message)requires entered; { foreach (Client c in enumof(Client), c != this, c.entered)
Controllable:Controllable:
64Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
foreach (Client c in enumof(Client), c != this, c.entered)c.unreceivedMsgs[this] += Seq{message}; }
void Receive(Client sender, string message)requires sender != this &&
unreceivedMsgs[sender].Length > 0 &&unreceivedMsgs[sender].Head == message;
{ unreceivedMsgs[sender] = unreceivedMsgs[sender].Tail;}
…}
Observable:Observable:
Online testing
� For large reactive systems deriving an exhaustive test-suite is
infeasible
� Merge test generation and test execution into one process
� Online testing is a form of model-based stress testing
65Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� Purely random exploration does not work
• Use scenario control to direct online testing
Example of scenario controlChat initialization
� Use a parameterized scenario action Start.
• Invoking Start(n) produces a sequence of n Create() invocations followed by n Enter() invocations
[Action(Kind=Scenario)]void Start(int nrOfMembers)requires enumof(Client) == Set{};
66Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
requires enumof(Client) == Set{};{Seq<Client> clients = Seq{};
// 1-- The given number of // clients are createdfor (int i = 0; i < nrOfMembers; i++)clients += Seq{Create()};
// 2-- all clients enter the sessionfor (int i = 0; i < nrOfMembers; i++)clients[i].Enter();
}
Start(2)
Create()/c0;Create()/c1;c0.Enter();c1.Enter();
Demo: chat
Bob Server Mary
67Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
Time
Experiences
� Most tool features driven by demands of internal
users at MS (mostly testers)
� Models help discover more bugs during modeling
(design bugs) than testing
68Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
(design bugs) than testing
•Testers do not get credit for finding those bugs !!!
� During testing models help discover deep
system-level bugs where manual test scenarios
don’t
•Such bugs are hard to understand and fix
Experiences (cont.)
� Bugs appear in both models and implementations (the ratio is roughly 50-50)
� Code coverage is a poor measure for testing concurrent software, often a single execution thread provides the same coverage
69Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
thread provides the same coverage
� New versions of implementation usually require only local changes to models, whereas manual tests often need to be rewritten completely
� The tool is built on .NET but has been used to test distributed C++ applications. Most test harnesses at MS are built in managed code
NModel – http://www.codeplex.com/NModel
� NModel is a model-based analysis and testing framework for
model programs written in C#.
� It provides software support for the book "Model-based
Software Testing and Analysis with C#", Cambridge University
Press, 2007
70Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
Press, 2007
� See http://staff.washington.edu/jon/modeling-book/ for
more information regarding the book
References Main references
� [P07] Ana C. R. Paiva, PhD Thesis untitled “Automated Model-Based Testing of Graphical User Interfaces”: www.fe.up.pt/~apaiva/PhD/PhDGUITesting.pdf
� [UL07] Mark Utting and Bruno Legeard, “Practical Model-Based Testing: A Tools Approach”, Morgan Kaufmann, Elsevier, 2007.
Other references
� [N00] N. Nyman, "Using Monkey Test Tools," in STQE - Software Testing and Quality Engineering Magazine, 2000.
� [UTF-site] For Unit testing frameworks visit sites – www.nunit.org; www.junit.org
71Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
[UTF-site] For Unit testing frameworks visit sites – www.nunit.org; www.junit.org
� [JMWU91] R. Jeffries, J. R. Miller, C. Wharton, and K. M. Uyeda, "User Interface Evaluation in the Real World: A Comparison of Four Techniques," 1991
� [GC96] C. Gram and G. Cockton, Design Principles for Interactive Software, Chapman & Hall, isbn:0412724707, 1996
� [HVCKR01] J. Hayhurst, D. S. Veerhusen, J. J. Chilenski, and L. K. Rierson, "A Practical Tutorial on Modified Condition / Decision Coverage," NASA/TM-2001-210876, 2001.
� [BS02] J. Bredereke and B.-H. Schlingloff, "An Automated, Flexible Testing Environment for UMTS", in Proceedings of the IFIP 14th International Conference on Testing Communicating Systems XIV, 2002.
� [ABM98] P. E. Ammann, P. E. Black, and W. Majurski, "Using Model Checking to Generate Tests from Specifications", in Proceedings of the 2nd IEEE International Conference on Formal Engineering Methods (ICFEM'98), M. G. H. John Staples, and Shaoying Liu(Eds.), Brisbane, Australia, 1998.
References
SpecExplorer and Spec# references
� [BLS04] M. Barnett, K. R. M. Leino, and W. Schulte, "The Spec# Programming System: An Overview," presented at CASSIS'04 - International workshop on Construction and Analysis of Safe, Secure and Interoperable Smart devices, Marseille, 2004.
� [CGNSTV05] C. Compbell, W. Grieskamp, L. Nachmanson, W. Schulte, N. Tillmann, and M. Veans, "Testing Concurrent Object-Oriented Systems with Spec Explorer," presented at FM, 2005.
� [FSE-site] Visit FSE web site for more information: research.microsoft.com/foundations
72Model Driiven Software Engineering, MAP-I, Ana Paiva, 11/11/2008
� [SE-site] Visit Spec Explorer site for download: research.microsoft.com/SpecExplorer/
Additional reading
� [W03] James A. Whittaker, “How to Break Software: A Practical Guide to Testing”, ISBN: 0201796198
� [A-site] Alan Hartman –http://www.agedis.de/documents/ModelBasedTestGenerationTools_cs.pdf
� Model-Based Testing papers –www.geocities.com/model_based_testing/online_papers.htm