Software Architecture-based Testing and Model-checking -

1

Lecturer:Henry Muccini

Assistant Professor, Computer Science Department University of L'Aquila - Italy

[email protected][www.di.univaq.it/muccini] – [www.HenryMuccini.com]

Software Architecture-based Testing and Model-checking- ECI 2005, University of Buenos Aires -

Course Web-site: [www.di.univaq.it/muccini/ECI05/]

Lecture 5: Model-Checking driven Testing

2SEA Group

© 2005 by H. Muccini / ECI 2005 CourseSEA GroupSEA Group

Copyright Notice

» The material in these slides may be freely reproduced and distributed, partially or totally, as far as an explicit referenceor acknowledge to the material author is preserved.

Henry Muccini

3SEA Group


Acknowledgment

» This work is joined with Patrizio Pelliccione (University of L’Aquila), Pierluigi Pierini (Siemens CNX), and Antonio Bucchiarone (ISTI – CNR)

» Published in ITM 2004, Lecture Notes in Computer Science, LNCS, vol. 3236, pp. 351 - 365 (2004).

mailto:[email protected]

http://www.di.univaq.it/muccini

http://www.HenryMuccini.com

http://www.di.univaq.it/muccini/ECI05

2

4SEA Group


Agenda

» Introduction and Motivations:

- ModTest and TeStor

» ModTest in Siemens C.N.X.

» Challenges and Future Work

5SEA Group


Considerations» Different Analysis require different notations

- testing, model-checking, performance and reliability analysis requirespecific formalisms and annotations on UML-based models[UML&SA04]

- There is a huge relationship between what we specify and what kindof analysis we may perform

> Modeling for documenting vs. Modeling for analysis

…But…

6SEA Group


Considerations

» different Analysis techniques are usually related

- supposing an industry is interested in deadlock analysis and performance analysis, a complete result is obtained only using two different ADLs.

» Modeling is really expensive

- we want to reuse the same model for many analysistechniques

3

7SEA Group


Integrating Analysis Techniques…

provide confidence on the implementation fulfillment to its

architectural specSA-based Testing

Charmy Project

validate the SA model conformance

with respect to selected functional properties

[www.di.univaq.it/charmy]

8SEA Group


ModTest: Model-Checking driven Testing[ITM04, CBSE05,TR_March05]

9SEA Group


Software Model Checking and Software Testing

» Model Checking:

- It checks whether a certain property is valid for a certain modelof a system [Ruys_PhDThesis]

> Model checking is a model-based, automatic technique that, given a finite-state model M of a system and a property P, checks the validity of P in M

» Testing:

- “Software testing consists of the dynamic verification of the behavior of a program on a finite set of test cases, suitablyselected from the usually infinite executions domain, against the specified expected behavior” [Bertolino_SWEBOK]

http://www.di.univaq.it/charmy

4

10SEA Group


Advantages

skills on formal methodsgenerally not required

skills on formal methods

test case identification problemstate explosion problem

code-based, model-based, specification-based

only model-based

clever selection of limited and relevant test cases

- usually left to the testerexperience

exhaustive approach tocompletely check the system

- completely automated

TestingModel Checking

11SEA Group


Goals and Motivations > 1/2» General Goal:

-- integrationintegration of of modelmodel--checkingchecking and and testingtesting toprovide an useful tool to test modern complexsoftware systems

» In related approaches:

- By using model-checking features, counter-examples are produced, successively used to derive test cases

- Main Limitations:

> P1 : due to models complexity, the model checker techniques become inapplicable, thus not allowing to identify test cases;

> P2 : even on little examples, the number of generated test cases causes the intractability

12SEA Group


Goals and Motivations > 2/2

» Our Goal:

To apply Model-checking and Testing in a SoftwareArchitecture-based (SA) process, where:

> Model-checking techniques are used to validate the SA model conformance with respect to selected functionalproperties+ avoiding state explosion problem

> while testing techniques are used to provide confidence on the implementation fulfillment to its architecturalspecification+ Test case selection driven by model-checking

5

13SEA Group


Our Proposal

Charmy

validate the SA model conformance with respect to selected functional properties

provide confidence on the implementation fulfillment to its architectural spec

SA-based Testing

14SEA Group


SA-based model-checking

Charmy Project[www.di.univaq.it/charmy]

» Model-checking:

> It checks whether a certain property is valid for a certain FSM modelof a system [Ruys_PhDThesis]

FMS Model of the SA

Propertyexpressed through

scenariosvalidate the SA model

conformance to selected functional

properties

15SEA Group


Software ArchitectureProperty

M.C.

Requirements

Charmy: Informally

SPINTrue or FalseTranslate them

into Promela

Translate theminto Buchi Automata


6

16SEA Group


SA-based Testing

» Goal:

- Provide confidence on the implementation fulfillment to its architectural specification

SA-based Testing[ICSE01, FASE04,TSE04]

17SEA Group


SA-based Testing in ModTest

18SEA Group


Question» How can we integrate both techniques?

- Rephrasing: how can we use model-checking results for generating test specifications?

» Usually:

- By focussing on model-checking driven testing, counter-examples are produced and successively used to derive test cases

» Our idea:

- To use the properties to derive test specifications, by recoveringmissing information from state machines

> In this way, the SA is validated with respect to requirements and the implementation conformance to the SA is tested

- SA-based analysis process

7

19SEA Group


The TeStor Algorithm» TeStor:

- Inputs:

> sequence diagram (inSD) representing properties- test generation directives

> components’ behavioral models (in terms of components' state machines),

- the model of the software under test- Outputs:

> a set of test specifications, still in the form of sequence diagrams (outSD)

20SEA Group


TeStor objective Model of the system

Test directives

m9m3

m8m1

Test specification

21SEA Group


TeStor objective

m9m3

m8m1

Model of the system

Test directives

Test specification

8

22SEA Group


How TeStor can achieve this goal

» How TeStor could work:

- Parallel composition of the state-machines models, and theirtraversal (like model-checkers), or

- Challenge: to avoid parallel composition

> To limit state explosion problems…

23SEA Group


The Algorithm

» The TeStor algorithm can be split into two macro-steps:

- State machines (SM) Linearization

> Decomposes SM in a set of linear traces

- Test Sequence Generation

> Looks at each linearized trace in order to identify the sup-trace of the inSD.

> This macro-step is composed by a Validation part, which checkswhen and how sup-traces need to be combined to produce theoutSD.

> The Merge algorithm glues together the validated traces

24SEA Group


Linearization» Starting from the initial state in the components’ state diagrams;

- It creates a trace at any time a state with a branch is reached or analready visited state is reached

» The algorithm is iterated, starting from the previously reached state, until unvisited states still exist.

S0 à S0m8

S0 à S1à S2à S3m9 m3 m10

S3 à S3m9

S3 à S4à S5à S0m8 m1 m7

S1

S2S3S4

S5

9

25SEA Group


From State Machines to Traces

26SEA Group


Test Sequence Generation

» Starting from the traces generated from each componentstate machine

- For each message “mi” in the inSD, TeStor identifies those traceswhich contain it

> Validate algorithm

Property

e) Property to be verified

C3 C2 C1 C4

m9m3

m10

m8

m1

m7

C2’s i.s. = S0 C3’s i.s. = S0

S0 –m9> S1S0 --m9-> S1 -- m3->S2 --m10->S3 --m9->S3

S0 –m9> S0

27SEA Group


Test Sequence Generation

» Starting from the traces generated from each componentstate machine

- For each message “mi” in the inSD, TeStor identifies those traceswhich contain it

> Validate algorithm

> Merge algoritmProperty

e) Property to be verified

C3 C2 C1 C4

m9m3

m10

m8

m1

m7

10

28SEA Group


Supporting ToolsThe TeStor algorithm has been implemented has a plug-incomponent for Charmy, our validation framework forarchitectural analysis.

29SEA Group


ModTest: Model-Checking driven Testing

Charmy

validate the SA model conformance with respect to selected functional properties

SA-based Testing

provide confidence on the implementation fulfillment to its architectural spec

TeStor

[www.di.univaq.it/charmy]

Model of the system

Test directives

30SEA Group


ModTest in Siemens C.N.X.


11

31SEA Group


Siemens CNX : main research areas» Siemens CNX S.p.a. is a Siemens R&D lab; its mission is the

design and development of SDH(1) TLC equipments

» relevant research areas:

- Formal design methodologies

- System and software performance analysis

- Test design methodologies

- Intelligent agent application

- Network Processors

- Ethernet first mile

- Optics and cristal properties

- Electromagnetic compatibility

1) SDH Synchronous Digital Hierarchy

32SEA Group


System ArchitectureDesign

System TestsDesign

feedbacks

ImplementationEquipment Develop

Test ImplelemtationDesign

Review Review

TestExec

NOK

OKOK

SystemRequirements

NOKNOK

SystemRelease

OK

ModificationRequests

a

System ArchitectureDesign

SystemRequirements

ModelCheck

Implementation/Equipment Develop

Test GenerationEngine

TestExec

NOK

OK

NOK

OK

SystemRelease

b

Test Design Methodology > objective

»Improve the tets design process

33SEA Group


Case Study > some definitions» A SDH Network Element (NE, i.e. equipment) is modeled using

the functional model standardized by ETSI and ITU-T.

» The functional model is built around two specific concepts:

- “network layering", with a client/server relationship between adjacent layers;

- “atomic functions“ (connection, termination and adaptation), to specify the behavior of each layer.

» applicative functions should reside on top of a layer providing specific processing on transmitted information

» A “virtual network connection” can be established between mate network layers (or atomic/applicative functions) belonging to different NEs by means of transport services offered by the underlying layers

12

34SEA Group


Case Study > some definitions

35SEA Group


Case Study > EOW architecture

» The EOW supports a telephone link between NEs using dedicated voice channels defined on the SDH frame (i.e. the “EOW SubNetwork” [eowSN]);

» An EOW node consists of:

- A “handset” (HS) that manage the physical phone device;

- a “conference manager” (CM) that control the handset connection to the EOW subnetwork;

eowSNHS

CM

HS

CM

HS

CM

EOWNode

36SEA Group


Case Study > EOW components

HS1

CM1

eowSN

localNumSign1

call1callRequest

callRequest

callRequest

eowKeyDigit

eowKeyDigit

eowKeyDigit

HS2

CM2localNumSign2

call2

HS3

CM3localNumSign3

call3

Init Configconfig

IdleonHook

Congestion

offH

o o k onHook

Ringing

InConference

Dialing

Check

Busy

?call

time o

ut

timeout

offHook

onH ook

[cbu

sy=

=tru

e]

[cbusy==false]

time ou t

o nHook

?calloffHook

onHook

!lo ca lNumS ign

?call

eowKeyDig it (digit,numbe r)

!loc alNumS ign

Handset (HS)

S1?eowKeyDigit(digit,number)

S2

!callRequest(digit,number)

eowSubnetwork

S4

S3

S2S1?callRequest(digit,number)

? lo ca l N umS ig n /c b u sy = f a ls e

[(digit==x)&&(number!=1)]/!callRequest(digit,number)

[ (di g i t= = x )& & (n u m b e r!=

1 )] / !c a l l/ c

b u s y= t rue

! callRequest(dig it,number)

[digi

t==0]

/!call

Conference Manager (CM)

13

37SEA Group


Case Study > Functional Requirements

» EOW Functional Requirements/Properties:

A) when an operator makes a call dialling a selectivenumber, the target operator must receive the call.

B) it must be possible to enter a busy conference (with the special number-sign key) when a call is already in progress.

C) It must be always possible to exit to the conference(cleanly terminate a call).

38SEA Group


Case Study > Functional Requirements

39SEA Group


Case Study ResultsInteractive simulation & Test generation

» Simulation without constraint will result in an intractable number of traces;

» Simulation conditioned by the given properties;

» Up to 36 test traces was extracted;

- Most of them are eligible to become test cases;

» Test selection focus on some optimization criteria like:

- Maximization of system coverage,

- Minimization of global number of tests

- Minimization of test lenght (i.e. number of steps)

14

40SEA Group


Some ConsiderationsAdvantages:

» Model complexity and the state explosion reduction obtained by: SA-level model chekcing, iterative approach and abstraction ;

» Charmy → easy to use, practical, approach to model-checking, hidingthe modeling complexity;

» interactive simulation → we may identify traces of interest for testingthe whole system or just a relevant subsystem.

» test specifications are identified from the architectural model (notfrom requirements)

- Easiest alignment between SA and Test specifications;

- Easiest control of the design steps and evolution

41SEA Group


Some Considerations

Limitations:

» The Test Generator Engine can be automated; itsimplementation is in progress.

» The executable tests implementation from the generatedtest specifications is not automated yet. We approach thispoint with the aim to automate also this step.

» Models dimension and complexity still remain an issue, even if the iterative approach reduces the state explosionproblem.

42SEA Group


Challenges

15

43SEA Group


Challenge: SA-centric Analysis Process

44SEA Group


Charmy and TeStor [TR_March05]

45SEA Group


QuARS, ModTest, CowTest and UIT [QoSA05]

16

46SEA Group


Future Work

47SEA Group


Lessons Learned

» From this experience we learned integration is possible:

- Analysis integration: ModTest

> Future work will integrate other analysis techniques

» But we also need:

- Notation extension: It is possible to extend the same UML-based notation

- Tool extension: we can add a plugin implementing the new analysis technique

48SEA Group


Dually: Putting in Synergy UML 2.0 and ADLs

» A framework which

i) identifies a core set of architectural elements always required (A0),

ii) creates an UML profile for A0,

iii) provides extensibility mechanismsto add modeling concepts needed for specific analysis.

iv) describes how semantic linksmechanisms can be kept between different notations.

It is impossible to identify a unique language for representing SAs

Different Analysis requiredifferent notations

17

49SEA Group


Notation Extension: General Idea

+

=

=

=

=

Model Transformation

50SEA Group


Identify a core set of architectural elements always required

Create an UML profile able to model the core architectural elements previously identified

Provide extensibility mechanisms to add modeling concepts needed for specific analysis

Describe how semantic links mechanisms can be kept between different notations

Notation Extension: DUALLY [TR_May05]

51SEA Group


The roles of ADL, UML and XML

xArch

DUALLYProfile for

A0

ADL XMLUML

ADLs

DUALLYmodeling

extensions

XMLextensions

DUALLYXML

xArch &ACME

Experience

18

52SEA Group


Tool Extension: Charmy Studio

53SEA Group


Wish List: Tool·one: integration of analysis techniques

» Integration of many analysistechniques

» Integration of many UML-based notations

» Integration of differentanalysis tools

UML/SA profilefor Model-Checking

Semantic Relations

UML/SAprofile forTesting

Other

Testing toolOther

Input filters

Model-Checking tool

DUALLYXML

Model-checkingnotation

Testingnotation

UML/SA profilefor Performance

DUALLYProfile for A0

DUALLYstandard

editing andanalysis tools

Performancetool

Performancenotation

XMLextensions

Modelingextensions

Pluggedinto

DUALLY

Analysistools

Editingtools

feedback feedbackfeedback

Documents

Software Architecture-based Testing and Model-checking -