Rachid Kherrazi 8-10-2014 Amsterdam
TESTING ASD INTERFACE COMPLIANCE WITH ASDSPEC “A TOOL COMBINING THE BEST OF TWO TECHNIQUES”
PEOPLE INVOLVED
• Rachid Kherrazi
– Sr. Consultant-Project Leader @ Nspyre
– Domain
• Process and Product Improvement
• Model Based Testing and Model Driven Engineering
• iSQI Accredited Certified Model Based Tester Trainer
• Also : Arjan van Der Meer & Marc Hamilton
2
ABOUT NSPYRE
MARKET SEGMENTS
• High Tech
• Traffic & Infra
• Industry
AREA’S OF EXCELLENCE
(Model Based) Systems Engineering /Model Driven
Engineering / Model Based Testing /Industrial Automation /
Simulation / Big Data / Mobile Solutions
3
CONTENTS
1) Introduction to Model Based Testing
2) Generation of test cases form ASD interface model using
ASDSpec tool
3) Case Study and Results
4) ASDSpec Technology
4
V-MODEL AND MODEL DRIVEN SYSTEM DEVELOPMENT
Requirements
Analysis
System Functional
Analysis
Design Synthesis
SW Analysis &
Design
SW Implementation & Unit
Test
Module Integration
& Test
(Sub-)System
Integration &Test
Model/
Requir
em
ents
Reposi
tory
Model Based System
Engineering
System Acceptance
Some of related technology
5
V-MODEL AND MODEL DRIVEN SYSTEM DEVELOPMENT
Requirements
Analysis
System Functional
Analysis
Design Synthesis
SW Analysis &
Design
SW Implementation & Unit
Test
Module Integration
& Test
(Sub-)System
Integration &Test
Model/
Requir
em
ents
Reposi
tory
Model Driven
Engineering
System Acceptance
Some of related technology
6
V-MODEL AND MODEL DRIVEN SYSTEM DEVELOPMENT
Requirements
Analysis
System Functional
Analysis
Design Synthesis
SW Analysis &
Design
SW Implementation & Unit
Test
Module Integration
& Test
(Sub-)System
Integration &Test
Model/
Requir
em
ents
Reposi
tory
System Acceptance
Model Based Testing
Some of related technology
7
Integration Design
Coding
Requirements System test
Test case specification
(design of logical test case)
(reduction of number of test cases by application of test techniques
Test execution
(reporting)
Test case generation
(design of physical test case)
(selection of input values and calculation of expected results)
PAGE 8
TEST PROCESS
3 main steps in test process
Module
Subsystem
System
8
MBT IS THE AUTOMATION OF TEST CASE GENERATION
Manual Automatic
Test specification
Test execution
Test generation
Model
base
d t
est
ing
Tra
dit
ional
manual
test
ing
Tra
dit
ional
auto
mati
c t
est
ing
Manual
Scripting
Testing skills + Scripting skills + Modeling skills
Manual
Modeling
9
MBT PROCESS IN A NUTSHELL
Requirements
SUT
Expected outputs
(Test Oracle)
Inputs
(Test sequences)
Verdict Failed Actual result # expected
result
Test suite
generate
author
feedback
feedback
feedback
Feedback
(via review, model
exploration, Model
checking)
control
Test Model
Actual outputs
Implementation Bug
Modeling error
Specification error
Design error
Conformance failures
10
EXAMPLE: MODELING WITH SPEC EXPLORER LOOK AND FEEL
Explore &
Analyze
Remodel
Generate
C# Model (or other .Net
Language)
Model Graph
Test Suite
11
Execute
Nunit
Note: this is not implementation
code, this is the textual Test model
(describing the interface behavior)
System Under Test
WHY ARE MODELS NEEDED FOR TESTING?
12
Component A
Component C
Component B
Unit A1
Unit A2
Unit B1 Unit B2
Unit C1 Unit C2
Unit testing is insufficient
- Single unit may work properly
in isolation
- Incorrect interaction between
units may cause serious
security/reliability failures
System-level testing:
- Requires model of externally visible system behavior
- Traditional FSM-based testing does not scale
- Behavior is often reactive/nondeterministic
- Implementation is multi-threaded or distributed
- Thread scheduling hard to control
- State space is typically infinite
SOME BENEFITS OF MODEL BASED
• Increased productivity (increased automation)
• Better test script maintenance
• Improved product reliability (early defect detection)
• Agility ( Easily react to new feature changes, Reusability of test semantics, Early test engagement, Drive quality upstream)
• Increased employee satisfaction (challenging, new horizon, fun)
13
MBT: CHALLENGES AND OPPORTUNITIES
Challenges: MBT is not easy: Different mind shift from traditional testing (Modeling skills)
and steep learning curve and high ramp up cost (tooling)
Opportunities: Modeling skills specialized training and certification
e.g. iSQI Certified MBT training (CMBT)
Nspyre is a iSQI training provider
Tooling More (Open Source) supporting integration of modeling, M2M
transformation,…
e.g. Nspyre ASDspec
14
SUPPORTING TOOLING
Manual Automatic
Test specification
Test execution
Test generation
Model
base
d t
est
ing
Tra
dit
ional
manual
test
ing
15
Testing skills
Manual
Modeling
MBT n
ext
Semi-automatic
Modeling
Manual
Scripting
+ Scripting skills Modeling skills
Tra
dit
ional
auto
mati
c t
est
ing
VERUM ASD:SUITE
17
Design model
Interface model
Requirements
Features
Use cases
Prototype
Decomposition
Code generation
Simulation & verification
NSPYRE ASDSPEC
18
Design model
Interface model
Requirements
Features
Use cases
Prototype
Decomposition
Test model
Test cases
ASDspec
Code generation
Simulation & verification
Testing
Note: reuse only the
Interface model! Design
model is not used
WHAT IS VERIFIED BY ASD
19
System Under Test
Component A
(ASD genereted)
Component C
(ASD genereted)
Component B
(hand written)
Unit A1
Unit A2
Unit B1 Unit B2
Unit C1 Unit C2
ASD verification doesn’t cover:
• Handwritten /legacy code
• Interaction between generated/handwritten component
• Data interaction
With MBT (Spec Explorer) we can cover all the above aspects!
ASDSPEC: WORKFLOW
Spec
Explorer
Model
Cord
file
Spec
Explorer
Test
Cases
20
tester /modeler
Cord file:
• Test Selection /Slicing /Scenario Selection
• Data Combination
• Model Composition
Model:
• Interface behavior
Spec Explorer (MBT tool):
• Generate automatically test cases
MBT
ASDSPEC: WORKFLOW
ASDspec
ASD
Interface
Model
Spec
Explorer
Model
Cord
Script
Spec
Explorer
Test
Cases
Reuse existing ASD interface models to generate Spec
Explorer MBT models automatically
21
MBT
ASDSPEC: DATA AND MODEL COMPOSITION
ASDspec
ASD
Interface
Model
Spec
Explorer
Model
Cord
file
Spec
Explorer
Test
Cases
Generated model can be extended with any Spec Explorer feature
in cord file (config file)
22
Cord file:
• Test Selection /Slicing /Scenario Selection
• Data Combination
• Model Composition
MBT
USE CASE: CONTAINER TERMINAL
• Multiple components
• Components need to interact to function
• Controller needed to coordinate interaction
23
RESULTS: ASD-CONTAINER TERMINAL
24
Higher overall productivity compared to traditional development.
Main remaining problems:
- Testing interaction and complete system (not fully supported)
- Data interaction not fully supported
- Debugging of third party library (some bugs found in manual written
code, legacy code) Testing still necessary
Productivity
Method
OVERVIEW OF PROJECT STAGES
25
Use case
/SUT
Goal
Technique
Tool
Results
Stage 1
Container Terminal (CT)
Development of control
software for the CT
Model Driven Engineering +
hand written sw
ASD:Suite
Create design Models +
interface models in ASD
Generate code
Productivity (code
generation) c.t. trad. dev
-
Testing complete system,
Interaction, data, external
code
Stage 2
Container Terminal (CT)
Verification of the
developed control software
of the CT (generated +
Hand written)
Model based Testing
Spec Explorer
Create Test Model in Spec
Explorer, generate test
suite
Test Productivity
Modeling skills,
complexity costs
Stage 3
Container Terminal (CT)
Verification of the developed control
software of the CT (generated + Hand
written)
Model based Testing
ASDspec & Spec Explorer
Generate Test Model from existing ASD
interface Model , complete Test Model ,
generate test suite
+ Productivity (code generation, partial test
generation)
Testing complete system, Interaction, data,
external code
(benefits only in case of existing of ASD
Models)
RESULTS: ASDSPEC- CONTAINER TERMINAL
26
Spec Explorer ASD + ASDSpec
Approach Generate test suite Generate test model
Techniques MBT MBT/MDE (M2M)
Effort/complexity Medium Low
Test cases 89 93
Perceived effectiveness Medium High
Bugs Some bugs in hand written code
/HAL
All known bus found
• Low effort model creation
- Easy reuse of existing work
• Testing of complete system
- Legacy code, data interaction
• High Quality, Cost Reduction
GENERAL MDSD ARCHITECTURE
Model
Reverse
Engineering
Model
Repository
Model Code,
Doc, etc.
Model
Editor(s)
Model
Checker
Model Transformation
M2M M2T
Metamodel(s)
27
ASDSPEC ARCHITECTURE
ASDSpec
makes use of
existing MDSD
architecture
Model
Reverse
Engineering
Model
Repository
Spec
Explorer
Model
SpecExplorer
Cord
Model
Editor(s)
Model
Checker
Model Transformation
M2M M2T
Metamodel(s)
ASD
Interface
models
28
ASDSPEC: TECHNOLOGIES
QVTO ASD
XML
SE
model
Cord
script
Acceleo
Acceleo
ASD
EMF
SE
EMF
Cord
EMF
QVTO
QVTO
ASDSpec
Model transformations bridge gap between ASD
and Spec Explorer
29
TOOL STATUS
• Tool as an standalone application.
• (Pending) tool will be available as an open source Eclipse plugin for the Eclipse community.
• For now available on request
• Future steps
– Support data handling/configuration aspects in ASDSpec, instead of relying on manual additions in Spec Explorer,
– Supporting other MBT tools.
30
ASDSPEC: USING BEST OF BOTH WORLDS
• Automatic generation of Spec Explorer test model from existing ASD interface model
Benefits
• Less effort for model creation
– reuse of existing work
• Testing of complete system including
– Legacy code
– External components
– Data combination testing
– Interaction testing
• Results: High Quality, Reduced cost
Statistical Dynamic
ASDspec
31
CONCLUSIONS
• MDE and MBT technologies have matured a lot in the
latest years
• It is a matter of time….. Evolution…..
32
Using Formal Specifications to
Support Model Based Testing
ASDSpec: A Tool Combining the
Best of Two Techniques
ASD
Static Dynamic
ASDspec
Static Dynamic
ASDspec
34
Source: On the Use of Verification and
Quantitative
Analysis in Industry
Jozef Hooman
Radboud University Nijmegen
&
Embedded Systems Institute (ESI),
Eindhoven
26 October 2012
TESTING STILL NECESSARY
TESTING STILL NECESSARY
35
Source: MDD and its impact on
testing...
A nanotech case study
ASD case
Bryan Bakker
EuroSTAR
May 2012