7/26/2019 Slides SEAA
1/19
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
A systematic mapping study on test
generation from Input/Output TransitionSystems
Sofia Costa PaivaAdenilso Simao
Instituto de Ciencias Matematicas e de ComputacaoSao Paulo University, Brazil
August 27, 2015
1 / 1 9
http://find/7/26/2019 Slides SEAA
2/19
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
1 IntroductionModel-Based Testing
Problem Statement2 Mapping Planning and Conduction
Goal and Research QuestionsSearch Strategy and Extraction Process
3 Analysis and Discussion of ResultsGeneral Information about StudiesIOTS Model CharacteristicsTest Selection CriteriaTechnology
Test ExecutionMap of selected studiesThreats to validity
4 Conclusion
2 / 1 9
http://find/7/26/2019 Slides SEAA
3/19
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
Context
Software testing:Demonstrate the actual and expected behaviors of a systemare at least similarManual and laborious process
Model-Based Testing (MBT)Strategy to minimize time and resource limitations
Automate the generation of high-quality test suites frombehavioural models.A model of desired implementation behaviour: the startingpoint of MBT.
3 / 1 9
http://find/7/26/2019 Slides SEAA
4/19
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
Techniques used in MBT
Finite State Machines -FSM
Test generation methodsfrom FSM guaranteefault coverage using a
fault model
Input/Output TransitionSystems - IOTS
Existing works merelyindicates that test suitemust be generated in a
nondeterministic way
4 / 1 9
I d i M i Pl i d C d i A l i d Di i f R l C l i
http://find/7/26/2019 Slides SEAA
5/19
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
Goal
Several recent studies have addressed issues related to testgeneration from IOTSs
IOTS is a suitable formalism to model complex systems, suchas communication protocols and distributed systems
Systematic mapping study
Important step to identify relevant studies and shortcomingsRigorous process of assessment and interpretation of allavailable studiesReported systematic mapping studies regarding MBT notincluding IOTS formalism.
5 / 1 9
I t d ti M i Pl i d C d ti A l i d Di i f R lt C l i
http://find/7/26/2019 Slides SEAA
6/19
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
Goal and Research Questions
GoalIdentify methods for test case generation from IOTSs througha systematic mapping.
Research questions:
RQ1. What are the main characteristics of IOTS modelsapplied in the studies?RQ2. What are the test selection criteria used in the studies?RQ3. What are the technologies used in the test generationfrom IOTSs?
RQ4. What are the features of the test executionimplemented in the studies?
6 / 1 9
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
http://find/7/26/2019 Slides SEAA
7/19
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
Mapping Planning and Conduction
Search strategy: using control studies
Keywords: test generation and IOTS
Engines: IEEE Xplore, Springer Link, Scopus, Web of
Knowledge, ACM Digital Library, ScienceDirect, Compendexand Oxford Journals
Figure : Mapping Process
7 / 1 9
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
http://find/http://goback/7/26/2019 Slides SEAA
8/19
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
Taxonomy of MBT approaches
8 / 1 9
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
http://find/7/26/2019 Slides SEAA
9/19
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
Taxonomy of MBT approaches
9 / 1 9
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
http://find/7/26/2019 Slides SEAA
10/19
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
Analysis and Discussion of Results
General information about studies
Figure : Distribution of studies overthe years
Figure : Distribution ofstudies into forums
10/19
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
http://find/7/26/2019 Slides SEAA
11/19
pp g g y
Analysis and Discussion of Results
General information about studiesComputational support
TGV (6 studies)
TorX (1 study)UPPAAL (1 study)
Tipe of evidence:
68% provide only examples13% report a case study8% report experimental results10% only discuss the application of the approach
11/19
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
http://find/http://goback/7/26/2019 Slides SEAA
12/19
pp g g y
Analysis and Discussion of Results
RQ1. IOTS Model Characteristics
Untimed (92%) andTimed (8%)
Deterministic (88%) andnon-deterministic (12%)
Discrete (98%), hybrid(1%) and continuous(1%)
12/19
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
http://find/7/26/2019 Slides SEAA
13/19
Analysis and Discussion of Results
RQ2. Test Selection Criteria
Structural model coverage (14 studies)Requirements coverage: test purpose (26 studies)Requirements coverage: properties (5 studies)Test case specification (7 studies)Test case specification: iocotheory (31 studies)Fault-based (12 studies)
13/19
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
http://find/7/26/2019 Slides SEAA
14/19
Analysis and Discussion of Results
RQ3. Technology
Random/non-deterministic (26 studies)Search-based algorithms (26 studies)Model-checking (8 studies)Simbolic execution (10 studies)
Constraint solving (6 studies)Inference rules (5 studies)Data abstraction (1 study)
14/19
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
http://find/7/26/2019 Slides SEAA
15/19
Analysis and Discussion of Results
RQ4. Test ExecutionOnline (34 studies)Offline (25 studies)Communication:
Synchronous (26 studies)
Asynchronous (10 studies)Distributed (12 studies)
Online and synchronous test generation have been widely used
adopted by the classical method of Tretmans.
Online testing use randomize choice techniques
provided only a probabilistic guarantee of coverage for(unrealistic) a long time execution.
15/19
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
http://find/7/26/2019 Slides SEAA
16/19
Analysis and Discussion of Results
Map of selected studies regarding Test Generation criteria andTechnology
16/19
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
http://find/7/26/2019 Slides SEAA
17/19
Threats to validity
Selection bias was treated by defining our inclusion criteria forgathering the largest possible amount of papers that would fitthe test generation from the IOTSs domain
A specialist validated the results so that inaccuracy in dataextraction and misclassification could be alleviated.
17/19
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
http://find/7/26/2019 Slides SEAA
18/19
Conclusion
Lack of a standard of IOTS model characteristics may restrictthe adoption of each method.Widely used coverage criteria, such as state and transitioncoverage, have been applied to IOTSs in only few recentstudies.Most of test generation technologies are non-deterministic
There exist no standard fault models, leading to most testgeneration algorithms being nondeterministic.
Completeness is guaranteed in theory, by repeating theprocess an unbounded number of times.
Recent studies on IOTS-testing have adopted concepts of
FSM-testingHowever, fault models (which guarantee the completeness oftest suites) has played a relatively small role in IOTS testing.
Future work:Replication of study and include more evidence
Quality evaluation18/19
Introduction Mapping Planning and Conduction Analysis and Discussion of Results Conclusion
http://find/7/26/2019 Slides SEAA
19/19
A systematic mapping study on test
generation from Input/Output TransitionSystems
Sofia Costa [email protected]
Instituto de Ciencias Matematicas e de ComputacaoSao Paulo University, Brazil
August 27, 2015
19/19
http://find/