Making TTCN-3 work!

© NOKIA Originator: Martin Botteck / April 12, 2005 / Page:1

Nokia Research Center

CA

R/M

EM

/VTT

Making TTCN-3 work

Making TTCN-3 work!

Issues and strategies for its use in product development

Martin Botteck, Thomas Deiß, Colin Willcock


Bochum, Germany



CA

R/M

EM

/VTT

Making TTCN-3 work

TTCN-3 adoption today:

• Generally, uptake is slower than anticipated. Speed-up could be possible through :

• Handling variations of the testing task

• Integrating TTCN-3 into existing test environments

• Using TTCN-3 in the hardware development process



CA

R/M

EM

/VTT

Making TTCN-3 work

Overview

TTCN-3 for testing hardware

Conclusion/Summary

TTCN-3 in existing test environments

Variations in Test Systems



CA

R/M

EM

/VTT

Making TTCN-3 work

Variations in Testing

• Within a set of test cases there often are several commonalities

• Example: 3G “Call”

– several types of calls (voice, data,…) and calling routes– behaviour will in many cases be similar, but not the

same– “intuitive” solution: copy/paste code from one test case

to another “similar” one• -> maintenance problem when the 3G specification is

updated– alternative solution: add a parameter in the test case

invocation• -> code will become hard to read



CA

R/M

EM

/VTT

Making TTCN-3 work

Types of variations

• Value Variants– Variations that may be expressed by altering a

parameter inside a test case or message sent to the SUT

– seems trivial but soon becomes difficult due to • vast amount of parameters and combinations• typically not all combinations make sense• dependancies between the values is typically not

documented and implicit– a language or coding style solution needs to be

found for this



CA

R/M

EM

/VTT

Making TTCN-3 work

Types of variations (ctd)

• Behavioural Variants– Variations that may be expressed by a different

sequence of messages between the SUT and the tester• several messages or sets of sequences may remain the

same• some are different

– decomposition of the SUT may reveal common abstract test case structures with respect to the SUT components

– but what about complex composition variants of such components?

– -> Abstract Modelling of the tester? (UML testing profile, Test Purpose Language)



CA

R/M

EM

/VTT

Making TTCN-3 work

Types of variations (ctd)

• Variants of the System under test– product customisation– evolution of conformance standards

• Might be expressed by altering a parameter (e.g. “feature_x=OFF”, “call_type=‘rich_call’”)

• in combination with reference to a set of message sequences

– Typically, a vast amount of such combinations will be needed

• How to solve this without copy/paste of code but still avoiding combinatorial logic to the user?



CA

R/M

EM

/VTT

Making TTCN-3 work

Overview


Conclusion/Summary





CA

R/M

EM

/VTT

Making TTCN-3 work

The dilemma of choice

• Nowadays most R&D departments already perform testing:

• Conformance testing, hardware testing, product testing, module testing, unit testing…

• Proprietary test systems (“home grown” or “tailor made”)

• Multitude of vendors with closed environments

• Trying out a system is a lot of effort. So, shortcomings of any choice being made show up too late for reversing the choice



CA

R/M

EM

/VTT

Making TTCN-3 work

The testing tool chain

• for each stage components from different vendors need to interoperate

• clearly defined interfaces are needed

Testware

Test cases

Test System

Codec

Router

Test Development Tools

Editor

Compiler

Codec Generator

…

Test ExecutionControl

Log

Analysis

Thomas Deiß:

Usually, in the TTCN-3 world, the platform and the SUT adapter are

considered to work side by side, not the PA on top of the SA.

MBo: Better this way?

It does not become clear to me, what are the stages: Tools, Testware (development), Test sytem

(development?), Test execution. But 'tools' is not a stage.

MBo: Indeed. Let´s talk

Editorial comment: Test system <-> Test Execution, note different capitalization of

the second word

Thomas Deiß:

Usually, in the TTCN-3 world, the platform and the SUT adapter are

considered to work side by side, not the PA on top of the SA.

MBo: Better this way?

It does not become clear to me, what are the stages: Tools, Testware (development), Test sytem

(development?), Test execution. But 'tools' is not a stage.

MBo: Indeed. Let´s talk

Editorial comment: Test system <-> Test Execution, note different capitalization of

the second word

Run Time System

Executable Test System (ETS)

PlatformAdapter

SUTAdapter

SUT

Codec



CA

R/M

EM

/VTT

Making TTCN-3 work

Run Time System

Executable Test System (ETS)

PlatformAdapter

SUTAdapter

SUT

Codec

The testing tool chain interfaces

Testware

Test cases

Test System

Codec

Router

Test Development Tools

Editor

Compiler

Codec Generator

…

Test ExecutionControl

Log

Analysis

• TTCN-3 core notation language for exchange of test cases

• Run Time IF for Adaptors and Codecs

• Test Execution Control and logging through TCI

TCI/LogTRI



CA

R/M

EM

/VTT

Making TTCN-3 work

Further work on interfaces

• Tool vendors have a notion to “close shop” and not support public interfaces

• Codec interface part in TCI leads to non-interchangeable codecs

• realisation of a given specification in IDL depends on the target language.

• Java based tools and C based tools require different codecs• performance problems when separating the codec to a

separate executable• Standardisation sometimes lags behind (e.g. TCI)

• It is also desirable to generate as many test cases and parts of the environment as possible directly from the behavioural description of the IUT

• -> modelling representations (e.g. XMI) need to be supported in the tools



CA

R/M

EM

/VTT

Making TTCN-3 work

Overview


Conclusion/Summary





CA

R/M

EM

/VTT

Making TTCN-3 work

Hw design flowAlgorithmic description

informal model, e.g. C source code

Highest Level behavioural model

(e.g. SystemC)

Decomposition

(Block diagram)

Notion of “Time”

Bit accurate Hardware model

(e.g. VHDL)Notion of “Clock Cycles”

Physical Model

(e.g. RTL, Spice,…)

Introduction of physics

(layout, transistor, …)

• Description consistent through all stages/levels

• Automatic generation of tests (“Test vectors”)

• Back annotation integral part of the process



CA

R/M

EM

/VTT

Making TTCN-3 work

Sw design flowuse case description

formal model, e.g. UML UC


(e.g. UML sequence diagram, state chart)

Decomposition

(package diagram)

Refinement,

Introduction of “Signals”

executable

(e.g. binary, script,…)

Introduction of programming language syntax

(C, C++, Java,…)

more detailed behavioural model


• Rather “novel” approach• Automatic generation of tests (“Test vectors”) still in experimental phase• Back annotation is not formally specified



CA

R/M

EM

/VTT

Making TTCN-3 work

“Traditional” HW-SW integration

Algorithmic description

informal model, e.g. C source code


(e.g. SystemC)

Decomposition

(Block diagram)

Notion of “Time”

Bit accurate Hardware model

(e.g. VHDL)Notion of “Clock Cycles”

Physical Model

(e.g. RTL, Spice,…)

Introduction of physics

(layout, transistor, …)

use case description

formal model, e.g. UML UC



Decomposition

(package diagram)

Refinement,

Introduction of “Signals”

executable

(e.g. binary, script,…)

Introduction of programming language syntax

(C, C++, Java,…)

more detailed behavioural model


SW release HW Prototype

SW release HW prototype

Integration Integration Tests



CA

R/M

EM

/VTT

Making TTCN-3 work

Common nodes in the HW and SW design flow

• Integration tests are not determined by formal derivation from HW/SW high level models

• Integration typically is the 1st stage where HW and SW meet

• Integration results feed back “somewhere” in the process in SW

• HW and SW tests only derive from HW and SW models respectively

• How about

• utilisation of SW models (e.g. High level behavioural) for generation of HW tests?

• including behavioural HW models (SystemC) in High level behavioural SW models?

• generation of tests directly from the model representation?



CA

R/M

EM

/VTT

Making TTCN-3 work

Overview


Conclusion/Summary





CA

R/M

EM

/VTT

Making TTCN-3 work

Summary/Conclusions

• Further deployment of TTCN-3 needs convincing advantages:– integration into existing test environments

• interfaces, replaceable components from heterogenous test systems

– handling of variants through specific coding constructs

– automated generation of test cases from abstract models

– cover HW as well as SW through adequate representation in SW models



CA

R/M

EM

/VTT

Making TTCN-3 work

Thank you! Questions?

Acknowledgement: The authors would like to thank Mr. Risto Teittinen from Nokia Networks for his contributions to this topic

Documents

Making TTCN-3 work!