Test Suite Users Guide - IBM Research the user wants to enforce an invocation order, AGEDIS-1999-20218 Page 2 of 51 AGEDIS Test Suite Users Guide he should place the stimulate or observe

Users Guide

AGEDIS

Test Suite

Date: 14/04/03

AGEDIS-1999-20218 Page 1 of 51 AGEDIS Test Suite Users Guide

Change History Date Author Summary of changes 18/03/2002 Laurent

Mounier ATS 1st release

09/05/2002 Kenneth Nagin

Integrated Suite Event Trace into ATS

24/05/2002 Alan Hartman

Changed title page and footer only


Minor changes to Suite Event Trace: • host element’s clone attribute removed • process and sutObj elements’ clone attribute renamed to clones • sutObj element’s id attribute renamed to name - sutObj

element’s tedObj attibute removed • logEvent’s sutObj attribute renamed to object

17/07/2002 Kennneth Nagin

We address the following issues: 1. Concurrent vs Sequential processing semantics - It is

currently not clear when stimuli and observation are processed concurrently or sequentially (Verimag).

2. Timer- It is currently not clear how to express timers and timeout conditions (Verimag).

3. Object referencing and variables - The ATS definition does not contain an object referencing syntax for variables (France Telecom R&D).

4. Manual ATS creation- There exists test behavior that is hard to model but is commonly used when creating tests manually, e.g. do while loops. It is agreed that modeling limitation should not necessarily restrict ATS and that we should support manual ATS creations. A consequence of manual test creation is the requirement to extend ATS to use plug-in utilities from the test environment (France Telecom R&D, Imbus).

5. Do while loops - The ATS definition does not support do while loops, but we can show how this can be accomplished using proxy utility classes plus some minor changes to the ATS syntax (France Telecom R&D, Imbus).

6. Alternative stimulation – The current ATS definition only supports alternative observation support but there was a request for similar alternative stimulations (IBM).

The follow changes are introduced to the ATS: • Stimuli and observations within simulation or observation

blocks are treated as a set of elements and are invoked concurrently. If the user wants to enforce an invocation order,


he should place the stimulate or observe elements in separate stimulation or observation blocks (response to <1>).

• We remove the parallel elements since the default behavior makes it redundant.

• We allow stimulation and observation elements to be empty so they can be used as timer directives (see below). An empty observation has the special meaning of waiting for all previous synchronous stimulations to complete, i.e. return. If one wants to wait for the completion of a specific stimulation, then one adds an observation for that stimulus. In the case of void return an empty return element is included, e.g. <observe …><return/></observe> (response to <1> and <2>).

• The stimulation and observation elements may be thought of as timer directives. The timer directive supports two attributes: time delay and timeout. Time delay means the time to delay after performing the element’s operations, e.g. observations. Timeout means the maximum time to wait for the completion of all the element’s operation before timing out. The delay time and timeout are assigned in the test execution directive’s options element. In addition, specific time related values can be assigned to control and observable signature in the test execution directive’s sutSignature element. In addition, the timeout element is already defined in ATS (response to <2>).

• We introduce the attribute ref in the value element. The ref attribute references another object’s observable signature (it is of the form object.signature). This means that parameters and responses may reference another objects’ variables (response to <3>, <4> and <5>).

• Classes that are not defined in an ATS reference element can be reference by the object element’s class element. It is up to the execution engine to map the signatures used in the ATS test cases dynamically without help from the model classes. A consequence of this is to change the XREF type for object’s class attribute to string, since the class id may not be in the ATS file. (response to <4>).

• Proxy utility classes can be created and described in the model element or referenced in the object’s class attribute. Their objects, controls and observables can be used and referenced within the test cases (response to <4>).

• An empty alt is allowed within a set of other alt elements. Note: an observation element may contain a set of 2 or more alt elements that imply choice. The empty alt is interpreted as “otherwise” and is used to point to the next element (stimulation, observation or state) when none of the other alternative observations occur during test execution. The use of this empty alt plus a proxy utility class can be used to create a


“do while loop”. (response to <5> for an example see addElement).

• We allow sets of 2 or more alt elements within a stimulation element. The execution engine may choose any alternative stimulation (response to <6>).

• We have a problem with the current definition of observation that does not allow for getter methods to derive state information. In the current definition, an observation with the signature getState(String component_name):int would require a stimulation before the observation. I think the observe clause describes an act of observation that does not change the state of the SUT. If the same observation signature is used in the previous stimulation, then it describes the stimulation’s return value.

We illustrate most of the above changes in addElementTest.ats. 31/10/02 Kenneth

Nagin Title changed to Test Suite User’s Guide. Added many examples and samples. After the change the TS file will contain 2 top-level elements (under testSuite):

1.abstractTestSuite (1) 2.testSuiteTrace (0..∞)

old: model → class → controls → control (observables → observable) new: model → class → members → member: <members> <member signature=”signature”/> </members> object now has a boolean environment attribute to mark whether the object represents the SUT or evironment (tester). old: testCase → stimulation/observation new: testCase → step:

<step id=”id”> (interaction* | tcStage | timeout | join) </step>

old: testCase → stimulation/observation → stimulate/observe new: testCase → step → interaction:

<interaction initiator=”from” object=”obj” type=” call|return|call_return|send|receive|dend_receive|get|set|get_async|set_async” signature=”signature”> (value|structure)* (exception)?


</interaction> old: testCase → tcStage new: testCase → step → tcStage:

<step id=”id”> <tcStage type="end_tc" verdict="pass"/> </step>

old: testCase → stimulation/observation → alt new: testCase → altStep:

<testCase> ... <altStep id="S1" type=”all|any”> <step id=”CASE1” next=”S2”>...</step> <step id=”CASE2” next=”S3”>...</step> </altStep> ... </testCase>

“All” means do all steps and go according to the successful one. Any means choose any (just one) of the steps.

old: testCase → (stimulation | observation | tcStage)+ new: testCase → (step | altStep)+:

<testCase> <step id="S1">...</step> <altStep id="S2"> <step id="S2.1">...</step> <step id="S2.2">...</step> ….. </altStep> <step id="S3">...</step> ... </testCase>

old: testCase → stimulation/observation → stimulate/observe → params/return new: testCase → step → interaction → (value|struct)*


(exception)?: <interaction signature=”foo(int,recPerson):bool” ...> <value>123</value> <structure> <value>Harry</value> <value>Potter</value> </structure> <exception name=”java.lang.NullPointerException”/> </interaction> structure := (value)*

Timeout now has a integer duration attribute and a ref attribute to describe the timeout or reference a constant or field that contains the timeout period. We think that the changes result in a simpler ats that is easier to understand and create manually. In addition, it allows the same test suite to be used for both component and system tests.


Name changes Old name New name structure complexValue record structure Interaction type enumerations: get check set change get_async waitFor set_async (removed) step’s attribute next nextPass altStep’s attribute next nextFail const constant

New structures


enumeration destroy interaction type

enumeration Destroy an object

exit_ts tcStage (state) type enumeration

Exit test suite

suffix interaction attribute Signature suffix to support array elements, array intervals, and structure fields

topLevel testCase attribute Indicates whether to process test case automatically or only within subStep.

dataLists model element dataList collection dataList dataLists element A list of test case input maxRepetitions TestCase attribute Maximum repetitions

to limit dataList input coverage.

param testCase or subStep element

test case parameter, may reference dataList

inputCoverage TestCase element Test case datList input coverage strategy

forbiddenSet testCase element Forbidden combinations

forbid forbiddenSet element

Forbidden data

global class name attribute value

Reserved class name. Its elements can be referenced without the object clause.

sample altStep attribute Sampling strategy altValue interaction element alternative return value

or input value byte, short, long, double

Basic types Added these basic types for manual test writing

Comments The enum elements have an implicit order based on the definition order. The elements can be used as range's interval or an array's enumIndex


Sited what elements that are not supported by current release of Spider Test Execution Engine. Moved tour of samples to Model Driven Testing Overview document.


Added <value ref=”NULL”>


Contents 1 Introduction......................................................................................................... 11

1.1 Overview.................................................................................................................... 11 1.2 This Document........................................................................................................... 11 1.3 Related Documents .................................................................................................... 11 1.4 Objectives and design choices ................................................................................... 11

2 testSuite............................................................................................................... 13

3 abstractTestSuite................................................................................................ 13 3.1 directive...................................................................................................................... 13 3.2 model.......................................................................................................................... 14

3.2.1 class ................................................................................................................ 14 3.2.1.1 constants & constant ........................................................................ 15 3.2.1.2 types & type..................................................................................... 15 3.2.1.3 dataLists & dataList......................................................................... 20 3.2.1.4 members & member ........................................................................ 21

3.2.2 object .............................................................................................................. 22 3.3 statistics...................................................................................................................... 23 3.4 testCase ...................................................................................................................... 23

3.4.1 param .............................................................................................................. 25 3.4.2 inputCoverage................................................................................................. 25 3.4.3 forbiddenSet & forbid..................................................................................... 27 3.4.4 step.................................................................................................................. 28

3.4.4.1 interaction ........................................................................................ 29

1 32

1 32 3.4.4.2 join ................................................................................................... 39 3.4.4.3 timeout ............................................................................................. 39 3.4.4.4 tcStage ............................................................................................. 39

3.4.5 altStep ............................................................................................................. 40 3.4.6 subStep............................................................................................................ 42

4 testSuiteTrace ..................................................................................................... 44 4.1 host ............................................................................................................................. 44


4.1.1 process ............................................................................................................ 45 4.1.1.1 sutObj............................................................................................... 45

4.2 logEvent ..................................................................................................................... 46 4.3 testCaseTrace ............................................................................................................. 47

4.3.1 stepTrace......................................................................................................... 47 4.3.1.1 interactionTrace ............................................................................... 48 4.3.1.2 cloneTrace........................................................................................ 49

4.4 results ......................................................................................................................... 49 4.4.1 outcome .......................................................................................................... 50

5 Tour of Samples.................................................................................................. 50

6 References .......................................................................................................... 50 Tables Table 1 interaction values list translation ..................................................................................... 30 Table 2 Interaction relationship .................................................................................................... 32 Figures Figure 1 Interaction relationship................................................................................................... 31


1 Introduction

1.1 Overview The AGEDIS project is a three-year research and development effort on the part of a consortium of seven industrial and academic bodies, partially funded by the European Commission under the Fifth Framework Agreement. The aim of the project is to increase the efficiency and competitiveness of the European software industry by automating software testing, and improving the quality of software while reducing the expense of the testing phase. AGEDIS will achieve this by developing a methodology and tools for the automation of software testing in general, with emphasis on distributed component-based software systems.

1.2 This Document This document is the AGEDIS Test Suite Users Guide. It provides syntax, semantics and examples to describe the test suite. However, this document is only an informal description of the syntax and semantics of the test suite. A more formal description of the syntax is proposed in the test suite schema testSuite.xsd [ 5] . The AGEDIS Test Suite is composed of two primary components: the Abstract Test Suite and the Test Suite Traces. The Abstract Test Suite describes the expected test behavior. The Test Suite Traces describe the actual execution of the abstract test suite. The Abstract Test Suite is output by the AGEDIS test generator [ 4], IBM’s GOTCHA 4.0 test generator [ 6] [ 9] [ 11], and IBM’s UCBT 3.0 test generator [ 12]. You can edit it manually with the AGEDIS C.ATS graphical editor [ 1] [ 9]or any XML editing tool. Then you can execute the test suite with the Spider test execution engine[ 9] [ 10]. The Test Suite Trace is output by the Spider test execution engine [ 10].

1.3 Related Documents • AGEDIS C.ATS graphical editor [ 1] • AGEDIS test generator [ 4] • AGEDIS Intermediate Language Specification (IFE Public Specification) [ 3] • AGEDIS Language Specification [ 2] • AGEDIS Test Suite Schema [ 5] • Spider test execution engine [ 10] • Model Driven Test Overview [ 9]

1.4 Objectives and design choices The AGEDIS Test Suite is composed of two primary components, the Abstract Test Suite and the Test Suite Traces. The design of the Abstract Test Suite (ATS) is directed by several objectives:


• Expressiveness: the format we propose must be expressive enough to describe complex test scenarios, and to provide a rich set of interactions between a test engine and a system under testing. This includes concurrent and sequential processing semantics.

• Simplicity: the formalism and syntax chosen to express the test cases must be simple. It practice, abstract test suites may be produced by automated tools or manually. But this means the ATS syntax must be simple enough to be read, written, or modified by a human being.

• Flexibility: several versions of the tools will be developed within the project and some of the functionalities to be included in the next edition are not yet defined. Therefore, the ATS format should flexible enough to support new features or new primitives

• Compatibility: several formats already exist to describe abstract test suites. AGEDIS defined a new ATS format, since the existing formats were tool dependent or too specific to a particular application domain. However, the AGEDIS ATS format is influenced by three formalisms:

o The Gotcha-TCBeans ATS format [ 8], o The TGV ATS format [ 13], o The TTCN –3 (Tree and Tabular Combined Notation) language [ 14], standardized

by the ITU-T. In the future, some translation tools may be provided between these different formats (or at least between parts of them).

• Manual ATS creation - an important goal of ATS is to describe a common output format for different test generation tools; however, there exists test behavior that is hard to model but is commonly used when creating tests manually, e.g. do while loop. Modeling limitation should not necessarily restrict ATS and so it should support manual test creation. A consequence of manual test creation is the requirement to extend ATS to use plug-in utilities from the test environment.

The design of the Test Suite Trace is directed by several objectives:

The suite execution trace is integrated into the abstract test suite definition. Its design is directed by the following objectives:

• Expressiveness: the format must be expressive enough to capture the actual test suite execution. Most importantly, it must contain enough information to allow replay of a fault finding test scenario.

• Traceability: the format must allow the user to easily identify which abstract test suite paths are covered by the trace.

• Packaging Flexibility: the suite execution trace and abstract test suite may reside in the same or different files to simplify file handling and archiving.

• Repeatability: the trace may be used as input to repeat its execution. • Feedback: the trace may be used as feedback in order to generate a new abstract test

suite.

To satisfy these various requirements, we chose XML as an underlying language for our ATS format. The main motivation behind this choice is that this language is now a popular data exchange standard, with a simple (and easy to understand) definition, and for which several public parsers, browsers and editors are available.


2 testSuite The testSuite element is the root element of the abstract test suite. <testSuite> indicates the start of a test suite. </testSuite> indicates the end of a test suite. A test suite contains the following elements:

• 1 abstractTestSuite

• 0..∞ testSuiteTrace This is a XML testSuite example: <testSuite>

<abstractTestSuite> … </abstractTestSuite> <testSuiteTrace id="Trace1"/> … </ testSuiteTrace> <testSuiteTrace id="Trace2"/> … </ testSuiteTrace>

</testSuite> 3 abstractTestSuite The abstractTestSuite element describes the classes, objects, and logical scenarios (test cases) to test a system under test (SUT). <testSuite generator="name" dateTime="creation date and time" ref=”external location”> indicates the start of a test suite. </testSuite> indicates the end of a test suite. It contains the following elements:

• 0..1 directive • 0..1 model • 0..1 statistics • 0..∞ testCase

It contains the following attributes:

• generator - the test generation application, e.g. AGEDIS_TG. (optional)

• dateTime - the creation date and time of the test suite. It is expressed in the XML schema dateTime format [ 15] (optional).

• ref - points to an external location where the abstractTestSuite resides. It is expressed in the xml schema in anyURI format [ 17] (optional).

These are two XML abstractTestSuite examples: <abstractTestSuite generator="TGV" dateTime="2002-08-28T17:02:00"/> <abstractTestSuite ref="http://www.agedis.de/samples/helloworld.ats" "/>

3.1 directive The directive element describes additional information to document the connection between the ATS, the test generation (e.g., the command line that generated the test suite) and the test


execution directives. <directive format=”directive format”> indicates the start of the directive and </directive> indicates its end. The content may differ according to the test generator used and it is free form. The format attribute describes the text format (optional). This is an XML directive example: <directive>alternatingBitProtocol.aml</directive>

3.2 model The model element describes general information about the model and the test architecture used to build the test suite. <model> indicates the start of a model. </model> indicates the end of a model. It contains the following elements. They may be intermixed but at least one element must be chosen:

• 1..∞ class • 1..∞ object

This is an XML model example:

<model> <class name="Transmitter"/> … </class> <class name=" Receiver"/> … </class> <object name="transmitter_1" class="Transmitter"/> <object name="receiver_1" class="Receiver"/>

</model>

3.2.1 class The class element describes the test interface for a class. A class is a descriptor for a set of objects with similar structure, behavior, and relationships. <class name="name"> indicates the start of a class, </class> indicates the end of a class. It contains the following elements:

• 0..1 constants • 0..1 types • 0..1 dataLists • 0..1 members

The name attribute describes the name of the class, e.g. "Transmitter" (required). global is a reserved class name. The global class’s constants, types and dataLists may be references by other elements without an object prefix. We see no use in defining global members but there is no syntactic restriction. This is an XML class example: <class name="Transmitter"/> <constants> … < constants> <types> … <types> <members> … <members>


</class>

3.2.1.1 constants & constant This element is not supported in the current release of Spider [ 10]. The constants element defines the set of constants necessary to describe the testing interface of the class. <constants> indicates the start of the section and </constants> indicates its end. It contains the following elements:

• 1..∞ constant This is an XML constants example:

<constants> <constant name="numberOfMessages"> … </constant> <constant name="filename" type="string"/>

</constants> The constant element defines a constant used in the testing interface of the class. <const> indicates the start of the constant section, </constant> indicates its end. A constant definition starts with <constant name="name" type=” string|char|int|float|bool| byte|short|long|double|defined” ref=”type reference”> and ends with </const>. It contains the following elements that may be intermixed:

• 0..∞ value • 0..∞ complexValue

The value or complexValue elements describe the constant’s actual value. A missing value will have to be supplied in the test execution directives. It contains the following attributes:

• name - the name of the constant (required). • type – valid values are string|char|int|float|bool|byte|short|long|double|defined. When type

is “defined”, the ref attribute references another type element (optional but defaults to defined).

• ref – reference to another type name (optional, only valid when type is defined). The referenced type may be a type defined in this class or in the global class. Scoping rules apply, so when the same name is used in this class and the global class it refers to this class’s type.

These are two XML constant examples:

<constant name="numberOfMessages" type="int"> <value>255</value>

</constant> <constant name="filename" type="string" />

3.2.1.2 types & type The types element describes named types use in the classes testing interface. It starts with <types> and ends with </types>.


The test execution may imply some data exchanges between the tester and the application under test. To correctly interpret these exchanges as they appear in the ATS, the test execution engine has to know the type of the data that are used in the test sequences. In particular, some of these data may be of basic type (like integer, float, boolean, etc.), but some others may be of more structured types defined in the model description (such as user defined enumerated types, arrays, or structures). The purpose of this section is to provide a name for each of these (basic or non basic) types used in the testing interface of the classes. The test execution engine uses the name to map this abstract type to a real data structure (according to the test execution directives). It contains the following elements:

• 1..∞ type The type element describes a named type used in the abstract test suite. It starts with <type name=”type name”> and ends with </type>. It contains one of the following elements:

• basic • enum • range • array • structure

The name attribute is name of the type that may be referenced by other elements (required). These are some XML type examples:

<type name="ip"> <basic type="string" />

</type> <type name="color">

<enum> <element name="red" /> <element name="green" /> <element name="blue" />

</enum> </type>

3.2.1.2.1 basic The basic element defines a simple type used in the testing interface of the class. It has the following form <basic type=”string|char|int|float|bool| byte|short|long|double”/>. The type attribute defines a basic type. Valid values are string|char|int|float|bool byte|short|long|double (optional but defaults to string. This is an XML basic example:



</type>

3.2.1.2.2 enum & element The enum element defines an enumeration used in the testing interface of the class. It starts with <enum> and ends with </enum>. An enumeration contains the following elements:

• 1..∞ element An element defines an enumeration element. It has the following form <element name=”element name”/>. The name attribute is name of the element (required). The enumeration elements have an implicit order based on the definition order. The elements can be used as range’s interval or an array’s enumIndex. This is an XML enum example:


</enum>

3.2.1.2.3 range & interval The range element defines a range of integers, real numbers, characters or enumerations used in the testing interface of the class. It starts with <range type=”int|float|char|defined” ref=”type reference”> and ends with </range>. It has the following attributes:

• type – valid values are int|float|char|defined. When type is “defined”, the ref attribute references another type element (optional but defaults to defined).

• ref – reference to another type name. It is limited to referencing another basic type, enumeration type or range (optional, only valid when type is defined). The referenced type may be a type defined in this class or in the global class. Scoping rules apply, so when the same name is used in this class and the global class it refers to this class’s type.

The range element contains the following elements:

• 1..∞ interval The interval element defines a subrange of values. It has the following form: <interval from=”value” to=”value”/>. It contains the following attributes:

• from – beginning of interval (required). • to – end of interval (required).


This is an XML range example:

<range type="int"> <interval from="1024" to="65535" />

</range> This is an XML enumeration range example:

<range ref="color" type="defined"> <interval from="red" to="blue" />

</range>

3.2.1.2.4 array The array element defines an array of some type used in the testing interface of the class. It has the following form <array type=”string|char|int|float|bool| byte|short|long|double|defined” ref=”type reference” size=”number of elements in array” enumIndex=”enumeration Index”/>. It contains one interval element. It contains the following attributes:

• type – valid values are string|char|int|float|bool|byte|short|long|double|defined. When type is “defined”, the ref attribute references another type element (optional but defaults to defined).


• size – array size (optional). • enumIndex – reference to enumeration type that is used to index the array (optional).

The array is indexed by an integer starting with zero when no enumIndex is specified. This is an XML array of type string:

<array type="string" size="3" /> This is an XML example that uses the type=”defined” and the ref attribute to build an array of type int:

<type name="port"> <range type="int"> <interval from="1024" to="65535" />

</range> </type> <type name="portList"> <array type="defined" ref="port" size="2" />

</type> This is an XML example that uses an enumeration as an array index:

<type name="color">



</enum> </type> <type name="colorList"> <array type="string" enumIndex="color"/>

</type>

3.2.1.2.5 structure & field The structure element defines a complex data structure used in the testing interface of the class. It starts with <structure name=”name”>and ends with </structure >. It contains the following elements:

• 1..∞ field Its name attribute describes the structures names (required). The field element defines a structure field. It starts and ends with <field name=”name” type=”string|char|int|float|bool|byte|short|long|double|defined ref=”type reference”/>. It contains the following attributes:

• name – field name (required). • type – valid values are string|char|int|float|bool|byte|short|long|double|defined. When type

is “defined”, the ref attribute references another type element (optional but defaults to defined).


This is an XML example that uses the type=”defined” and the ref attribute to build a structure with two fields, with types string and int:


</type> <type name="port">

<range type="int"> <interval from="1024" to="65535" />

</range> </type> <type name="address">

<structure> <field name="host" type="defined" ref="ip" /> <field name="port" type="defined" ref="port"/>

</structure> </type>


3.2.1.3 dataLists & dataList These elements are not supported in the current release of Spider [ 10]. The dataLists element describes a collection of dataList elements. It starts with <dataLists> and ends with </dataLists>. It contains the following elements:

• 1..∞ dataList The dataList element represents a domain of data used as test case input. Its starts with <dataList name=”dataList name” type=”string|char|int|float|bool| byte|short|long|double|defined” ref=”type reference” externalRef=”external reference”> and ends with </dataList>. It contains 0..∞ value or 0..∞ complexValue elements. It is not allowed to intermix the two kinds of elements. The contained elements represent the list of data that may be supplied to as input to a testCase’s param element. A value may reference a class’s constants, members or dataList. A reference to another dataList is treated as a union operation, and the referenced dataList’s data is included inline into the referencing dataList. Of course, it is invalid to reference a dataList with a different type. It contains the following attributes:

• name – data list name (required) • type – string|char|int|float|bool|byte|short|long|double|defined (optional but defaults to

defined). When type is “defined”, the ref attribute references another type element. When the type is bool and and the dataList contains no value or complexValue elements then it is assumed that the data domain is true and false.

• ref – reference to another type name (optional, only valid when type is defined). The referenced type may be a type defined in this class or in the global class. Scoping rules apply, so when the same name is used in this class and the global class it refers to this class’s type. When it references a type range of discrete values (int, char, bool, enum) and the dataList contains no value or complexValue elements then the data domain is assumed to contain all the discrete values in the referenced type.

• externalRef – external reference to a list of data values (optional). The external reference points to file. Each line in the files represents a data value. When the externalRef is specified then the dataList must not contains any value or complexValue elements

This is an example an XML dataList contains a data domain with 3 values: <dataList name="operatingSystem" type="string"> <value>WINDOWS</value> <value>LINUX</value> <value>AIX</value>

</dataList>


This is an example an XML dataLists example with a few dataList elements: <types>

<type name="version"> <range type="int"> <interval from="0" to="2" />

</range> </type>

</types> <dataLists name="config"> <dataList name="bool" type="bool"/>

<dataList name="operatingSystem" type="string"> <value>WINDOWS</value> <value>LINUX</value> <value>AIX</value>

</dataList> <dataList name="browser" type="string"> <value>NETSCAPE</value> <value>EXPLORER</value>

</dataList> <dataList name="version" type="defined"/>

</dataLists>

3.2.1.4 members & member The members element defines the set of class members necessary to describe the testing interface of the class. It starts with <members>and ends with </members>. It contains the following elements:

• 1..∞ member The member element defines the testing interface to interact with a class's objects. A member may describe a synchronous operation, asynchronous signal, synchronous attribute and asynchronous attribute. A synchronous operation is a blocking operation with a call and return. A signal is a non-blocking operation, i.e. event. A synchronous attribute is name slot in a class that has a value that may be updated atomically. An asynchronous attribute may also be updated, but the tester may have to wait for the update to complete, e.g. buffer. It starts with <member signature=”unique signature”> and ends with </member>. The signature attribute describes the name, parameter types and return types of the member. Each signature is unique in the class and fully identifies the control. The signature is referenced by test case interaction elements (required). The signature has the following form: <name>

[“(” [<parameter type> (<parameter type> “,”)*] “)”] [ “:” <member type>]


The <name> part is required. The parameter () part describes the parameters in synchronous operations, signals, or attribute access methods. It may contain a list of 0..∞ parameters separated by “,”. The <member type> part describes the resulting type of synchronous operations or an attributes type. Signals do not have a <member type> part. Parameter and member types may be a basic type or a reference to a type element. The referenced type may be a type defined in this class or in the global class. Scoping rules apply, so when the same name is used in this class and the global class it refers to this class’s type. This is an XML members example with two members: state and log:

<members> <member signature="state:string" /> <member signature="log(string):bool" />

</members>

An alternate form allows descriptive names to follow the parameter types. For instance, the above example could be written as follows:

<member signature="log(string Text):bool" /> <member signature="send(string Message)" />

3.2.2 object The object element describes an object that is accessed within the test cases. <object name="object name" class="object class" environment=”false|true”/> describes an object definition. It contains the following attributes:

• name - the object name. The object is referenced by test case interaction elements (required).

• class - the name of the object’s class element or it references a class name that is resolved

at execution time. (optional)

• environment - describes the object’s relationship to the system under test (SUT). When it is false, then the object describes an SUT object or proxy object that communicates with the SUT. When its environment attribute is true, then the object describes the environment at the border of the SUT, e.g. UML actor. A true setting may also be used in a component test to indicate that the tester (test execution engine) should impersonate the SUT object. It is optional but defaults to false.

This is an XML example of four objects. Object 1, trans_1, is derived from the SUT Transmitter class. Object 2, receiver_1, is derived from the SUT Receiver class. Object 3, integer, is an external reference to the Java class Integer. Object 4, tester, is not within the boundaries of the SUT and is implemented by the environment. <object name="trans_1" class="Transmitter" environment="false" /> <object name="receiver_1" class="Receiver" environment="false" /> <object name="integer" class="java.lang.Integer" environment="false" />


<object name="tester" environment="true" />

Note that object names must be unique: they have to fully identify a single object in its class. Therefore, when a given class may contain several object instances each object name must be unique.

3.3 statistics The statistics element gives some statistical information about the model and the test suite associated with the ATS. Its content may differ, depending on the test generator used to produce the test suite. Typical information will concern the number of states and transition of the model, the number of states and transitions of each test cases, the time required to produce them, the test directives used, etc.

3.4 testCase The testCase element describes the operations that form a test case. It starts with <testCase id=”test case identifier” topLevel=”true|false” maxRepetitions=”0..∞”> and ends with </testCase>. It contains the following elements:

• 0..∞ param • 0..∞ inputCoverage • 0..∞ forbiddenSet • 1..∞ step • 1..∞ altStep • 1..∞ subStep

The step, altStep or subStep elements may be intermixed, but at least one of these elements must appear in an interaction. A testCase’s param element may contain a value or complexValue element that references a dataList element that represent a list of input data. A testCase with its topLevel attribute set to true is repeated to cover the required input combinations between the parameters. The testCase’s inputCoverage elements describe the required input combinations. If no inputCoverage element is specified then the testCase is repeated to cover each parameter’s data list at least once. The testCase’s forbiddenSet elements constrain which input combinations are valid. It contains the following attributes:

• id – test case identifier (required). • topLevel – true | false (optional, but defaults to true): when true the test case is processed

automatically by the test execution. When set to false the test case is only processed when referenced by a subStep element.


• maxRepetitions – maximum test case repetitions allowed (optional). The maxRepetitions attributes limits the number of times that a test case is repeated to fulfill the test case’s input coverage requirements (see above).

Formally, a test case is a labeled a transition system, or, in other words, a directed graph with a label associated with each transition. This label describes an atomic sequence of interactions to be performed by the test engine when executing the transition. We distinguish between two kinds of transitions:

• stimulation, corresponding to the execution of a sequence of controllable actions by the test environment (tester); the tester’s stimulation may change the system under test (SUT) state.

• observation, corresponding to the execution of a sequence of observable actions by the tester. The tester’s observation does not change the SUT state.

To be executable by a deterministic test engine, the test case has to remain controllable, which means that when one of the outgoing transitions from a given state corresponds to a stimulation, then this transition must be unique. On the other hand, to handle non-deterministic applications, several observable interactions can be allowed from a given state. From a syntactic point of view, each test case is expressed by a set of transition steps. A step may be a set of test interactions or some tcStage that ends the test case. Usually, a test interaction is initiated by the tester on a SUT object. However, SUT objects may initiate actions on the test environment or the tester may be required to observe interactions between SUT objects. To summarize, the communication schemes between the tester and the SUT can be classified as follows:

• asynchronous stimulation specifies that the test environment sends an asynchronous message to an SUT object. It indicates a non-blocking operation in which the response to the message occurs sometime after the message is sent. It has no return value. It corresponds to the TTCN send operation.

• synchronous stimulation specifies that the test environment calls a SUT object method. It indicates a blocking operation in which the test environment waits until the method returns. It corresponds to the TTCN call operation. A synchronous stimulation may also mean that the test environment is responding to a previously received message from the SUT. This 2nd case corresponds to the TTCN reply operation.

• asynchronous observation specifies that the test environment examines the SUT object for some asynchronous state change or message response. When the destination object is an environment class instance, an asynchronous observation specifies that the test environment receives an asynchronous message from the SUT. This 2nd case corresponds to the TTCN receive operation.

• synchronous observation specifies that the test environment examines the SUT for some synchronous change, return value or output. However, when the object is belongs to the environment (object’s environment attribute is true), a synchronous observation specifies that the test environment is called by the SUT. This 2nd case corresponds to the TTCN getcall or getreply operation.

This an XML testCase example:


<testCase id="T1"> <step id="S1">…</step> <altStep id="S2" nextPass="OK" nextFail="TIMEOUT">

<step id="S2.1">…</step> <step id="S2.2">…</step>

</altStep> <step id="TIMEOUT ">…</step> <step id="OK">…</step>

</testCase>

3.4.1 param This element is not supported in the current release of Spider [ 10]. The param element defines a test case parameter. Its starts with <param name=”parameter name” type=”string|char|int|float|bool| byte|short|long|double|defined” ref=”type reference”> and ends with </param>. It may contain one or none of the following elements:

• value • complexValue

Its value or complexValue represents the parameter’s default settings, but they can be changed when referenced from a subStep element. The current value is supplied when referenced by an interaction’s value element. It contains the following attributes:

• name – parameter name (required) • type – valid values are string|char|int|float|bool| byte|short|long|double|defined. When

type is “defined”, the ref attribute references another type element (optional but defaults to defined).

• ref – reference to a global class’s type name (optional, only valid when type is defined). This a param XML example:

<param name="param5" type="string"> <value>test.html</value>

</param>

3.4.2 inputCoverage This element is not supported in the current release of Spider [ 10]. The inputCoverage element defines how to cover the test case’s input parameters. Its starts with < inputCoverage paramList=”list of parameter names” coverageLevel=”1..∞”> and ends with </ inputCoverage>. It contains no other elements.



• paramList – list of parameter names (required). • coverageLevel – input coverage level (optional but defaults to 1). A testCase’s param

elements parameterizes a testCase’s input. The testCase is repeated to cover the required input combinations between the above parameters list. These param elements may reference dataLists so a parameter may represent a list of different values. The coverageLevel describes the domain of input combinations to supply to the testCase. A coverageLevel of 1 means cover each parameter’s dataList value at least once. A coverageLevel of 2 means cover all pairs of values between parameter dataLists. The higher coverage levels are defined analogously. The number of test cases is limited to the testCase’s maxRepetition attribute. The testCase’s forbiddenSets described those input combinations that will not be generated. Parameters that do not reference a dataList may be included in the parameter list, but they are treated as a dataList with a single value.

This is an XML example that illustrates how to use the inputCoverage element: <model>

<class name="global"> <types>

<type name="version_number"> <range type="int"> <interval from="0" to="2" />

</range> </type>

</types> <dataLists> <dataList name="default" type="bool" />

<dataList name="os" type="string"> <value>WINDOWS</value> <value>LINUX</value> <value>AIX</value>

</dataList> <dataList name="browser" type="string"> <value>NETSCAPE</value> <value>EXPLORER</value>

</dataList> <dataList name="version" type="defined"

type="version_number" /> </dataLists>

</class> …

</model> <testCase id="TC1">

<param name="param1"> <value ref="default" />

</param> <param name="param2"> <value ref="os" />


</param> <param name="param3"> <value ref="browser" />

</param> <param name="param4"> <value ref="version" />

</param> <inputCoverage paramList="param1 param2 param3 param4"

covageLevel="2" /> …

</testCase>

The above example produces nine test case repetitions with the following input combinations: test case param1 param2 param3 param4 repetition 1 true Windows Explorer 0 2 false Linux Netscape 0 3 true AIX Explorer 0 4 false Windows Netscape 1 5 true Linux Explorer 1 6 false AIX Netscape 1 7 true Windows Netscape 2 8 false Linux Explorer 2 9 true AIX Netscape 2

3.4.3 forbiddenSet & forbid This element is not supported in the current release of Spider [ 10]. The forbiddenSet element defines forbidden input combinations. Its starts with < forbiddenSet> and ends with </forbiddenSet>. It contains the following elements:

• 1..∞ forbid The forbid element defines some data in a forbidden combination. Its starts with <forbid ref=”param reference”> and ends with </forbid>. It contains 1 value or 1 complexValue elements. It is not allowed to intermix the two kinds of elements. The contained elements form a tuple that is forbidden to produce as test case input. A value and complexValue element that references a dataList represents a list of data that is forbidden in combination with the other forbid elements. Its ref attributes references the param that owns the value or complexValue element in the forbidden combination (required).


All of the testCase’s parameters do not have to be referenced in a forbidden combination. Those parameters not referenced do not play a role in constraining the permitted input combinations. This is an forbiddenSet XML example: <forbiddenSet>

<forbid ref="param1"> <value>LINUX</value>

</forbid> <forbid ref=" param2"> <value>EXPLORER</value>

</forbid> </forbiddenSet> This forbidden set would eliminate test repetitions 5 and 8 from the above list, since they contain the forbidden combination of LINUX with EXPLORER.

3.4.4 step The step element represents one unit of work in a test case. It starts with <step id=”test step identifier” nextPass=”next successful step id” nextFail=”next unsuccessful step id”> and ends with </step>. A step element may choose between the following sets of elements:

• 1..∞ interaction • 1 join • 1 timeout • 1 tcStage


• id - identifies the test step and may be pointed to by another step or altStep’s nextPass or nextFail attribute (required).

• nextPass – links to the next step, altStep, or subStep where the test execution should continue at the end of the current step that is judged to be successful. It may point to itself. If the nextPass attribute is missing itself, then successful test execution just continues in sequence in the test case (optional).

• nextFail – links to the next step, altStep, or subStep where the test execution should continue at the end of the current step that is judged to be unsuccessful. It may point to itself. If the nextFail attribute is missing itself, then unsuccessful test execution fails the test case (optional).

This is an XML example of a few step elements:

<step id="S1"> <interaction object="obj1" signature="start():int" type="call">

</step> <step id="S2" nextPass="OK">

<interaction object="obj1" signature="start():int" type="return"> <value>0</value>


</interaction> </step> <step id="FAIL"> <tcStage type="end_tc" verdict="fail" />

</step> <step id="OK"> <tcStage type="end_tc" verdict="pass" />

</step> The following XML segment illustrates using a step’s nextPass and nextFail to create a loop test element. It exercises the addElement(Object) method by calling it many times concurrently:

• S1 invokes addElement(Object) with the input the object obj_1 • S2 invokes add(int):int to add 1 to a counter. It has two alternatives. If the counter is

greater than integer_1.MAX_VALUE the step is judged successful so it exits the loop by going to S3. Otherwise it continues looping by going to S1.

• S3 uses a join to wait for all the addElement(Object) to complete. <step id="S1">

<interaction signature="addElement(Object):void" object="vector_1" type="call">

-  <value ref="obj_1" />

</interaction> </step>  <step id="S2" nextPass="S3" nextFail="S1">

<interaction signature="add(int):int" object="count_1" type="call_return">

<value>1</value> <value ref="integer_1.MAX_VALUE" />

</interaction> </step> -  <step id="S3">

-  <join />

</step>

3.4.4.1 interaction The interaction element describes a test action or observation. It starts with <interaction initiater=”initiating object” object=”owning object” signature=”interaction foot print” suffix=”[i] | [j..k] | field name” type=”call|return|call_return|send|receive|check|change|waitFor|create|destroy”> and ends with </interaction>. An interaction contains the following elements:



• 0..∞ altValue • 0.. ∞ exception

The value, complexValue and altValue elements can be intermixed. They describe the interaction input, response and attribute value setting. The interaction’s signature and type (see below) are used to interpret the values; the translation is illustrated in Table 1. Table 1 interaction values list translation

Type values translation call input parameters return response call_return input parameters followed by response send input parameters receive input parameters send_receive input parameters check input parameters followed by value setting change values to set attribute waitFor input parameters followed by value setting create input parameters destroy input parameters

The exception element is used when an interaction responds by throwing an exception. If more than one exception is present then any one of the exceptions is an expected responses. An interaction element contains the following attributes:

• object – owning object or test case (required). It references a specific object element to which this interaction belongs. Or it references the current testCase id when the signature is the testCase’s parameter.

• signature – owning member signature or test case parameter (required). It references a specific member in the object's class associated with this interaction. The member may link to an explicit abstract test suite member element or it may be resolved at execution time. Or it references a param name of the current testCase.

• suffix – signature suffix (optional). The suffix is useful for further qualifying the signature to describe array element(s) or a field name, using the following forms:

o array element – It is expressed in the form [index]. The index references an array element. The index may a positive integer or enumeration element name.

o array interval – It is expressed in the form [start_index..end_index] . The interval references the array elements from start_index through end_index. The indexes may be positive integers or enumeration element names.

o field name – The field name references a structure’s field. • type - describes the interaction type (required). The possible type enumerations are listed

below: • call – invoke a synchronous operation. • return – return from a previously issued synchronous operation call. • call_return – call followed by the return. • send – send an asynchronous signal. • receive - receive an asynchronous signal.


• send_receive – send and receive a signal. • check – check an attribute value. • change – set an attribute value. • waitFor – wait to check an asynchronous attribute value. • create – create the object • destroy – destroy or de-reference the object

• initiator – initiating object. It describes who is initiating the interaction. If no initiator is provided, then the initiator is the environment, otherwise the attribute points to an object element (optional).

The interaction element describes an interaction between the environment (or tester) and the SUT or between SUT objects. It describes a particular synchronous operation, asynchronous signal, synchronous attribute or asynchronous attribute. A synchronous operation is a blocking operation with a call and return. A signal is a non-blocking operation, i.e. an event. A synchronous attribute is a name slot in a class that has a value that may be accessed atomically. An asynchronous attribute may also be updated, but the tester may have to wait for the update to complete, e.g. buffer. Objects interact with each other. They call operations, send signals, get attributes or set attributes. But an interaction also has direction describing its initiator (from) and owner object (to). In addition, the abstract test suite supports two types of objects, environment objects and SUT objects. The initiator object and owner object may be either type. An object’s type is defined in the object element’s environment attribute. The different possible relationships are illustrated in Figure 1: Environment SUT

Execution Engine

SUT Objects

A

B

C

obj1

obj2

tester

Figure 1 Interaction relationship

• [A] – From the environment to the SUT. • [B] – From the SUT to the environment. • [C] – From one SUT object to another SUT object. The interaction is internal to the

system, but it can be visible to the environment.


The interaction relationship between the different object types in terms of the interaction types is illustrated in Table 2. Table 2 Interaction relationship

Direction Type

A (ENV→SUT) B (SUT→ENV) C (SUT → SUT)

call return call_return

Tester calls method of a SUT object.

SUT calls method of an environment.

SUT object calls method of another SUT object.

send receive send_receive

Tester sends signal to a SUT object.

SUT sends signal to an environment.

SUT object sends signal to another SUT object.

check Tester checks value of a SUT object attribute.

SUT gets value of an environment attribute (*).

change Tester sets value of a SUT object attribute (*).

SUT sets value of an environment attribute (*).

waitFor Tester waits for specific value of a SUT object attribute (*).

create Tester creates a SUT object

SUT object tells the tester to create a SUT object.

SUT object creates another SUT object.

destroy Tester destroys a SUT object

SUT object tells the tester to destroy a SUT object.

SUT object destroys another SUT object.

(*) – It is useful in manual ATS creation, but we do not know of any current test generator that can generate the interaction. This xml interaction segment illustrates a synchronous call operation being sent from the tester to the obj1:

<interaction initiator="tester" object="obj1" signature="log(string):bool" type="call">

<value>System is ready</value> </interaction>

This xml interaction segment illustrates a response from a synchronous operation:

<interaction initiator="tester" object="obj1" signature="log(string):bool" type="return">

<value>true</value> </interaction>

This xml interaction segment combines the two synchronous operations above into one call and return interaction:

<interaction initiator="tester" object="obj1" signature="log(string):bool" type="call_return">


<value>System is ready</value> <value>true</value>

</interaction> This xml interaction segment illustrates an asynchronous signal being sent from the tester to transmitter_1:

<interaction initiator="tester" object="transmitter_1" signature="put(int)" type="send">

<value>0</value> </interaction>

This xml interaction segment illustrates a signal being sent by the receiver_1 object

and then being received by the tester: <interaction initiator="receiver_1" object="tester"

signature="ack(bool)" type="send_receive"> <value>false</value>

</interaction>

This xml interaction segment illustrates an attribute being examined by the tester: <interaction initiator="tester" object="obj1"

signature="state:string" type="check"> <value>ok</value>

</interaction> This xml interaction segment illustrates an array interval being examined by the tester:

<interaction initiator="tester" object="obj1" signature="threeStooges:string" suffix="[1..2]" type="check">

<value>Larry</value> <value>Curly</value>

</interaction>

This xml interaction segment illustrates an asynchronous attribute being examined by the tester. The tester waits until the ready attribute becomes true:

<interaction initiator="tester" object="obj1" signature="ready:bool" type="waitFor">

<value>true</value> </interaction>

This xml interaction segment illustrates a interaction checking the test case input:

<interaction object="TC2" signature="param2" type="check"> <value>R</value>

</interaction> This XML segment invokes the o_db.state() – valid return values are not empty and empty or the exception recovery_in-progress <interaction object="o_db" signature="state():string"/> <altValue> <value>not empty</value> <value>empty</value>


</altValue> <exception name=" recovery_in-progress "/> </interaction>

3.4.4.1.1 Executing a step’s interactions A step may contain many interactions. All of the interactions are executed in parallel. A step is judged to be successful when the environment (tester) successfully executes all of its nested interactions and/or observes that the interaction responses match what the abstract test suite predicts. However, the interaction type and its relation to the environment affect how to evaluate its success. When a step is judged to be successful, the test proceeds to the step indicated by its nextPass attribute. When no nextPass attribute is provided, it proceeds to the next sequential step. When an interaction’s initiator is the environment, then the initiator types call, send, change, create or destroy are judged to be successful if the tester can execute the interaction. However, interaction types returning call_return, receive, send_receive, check, and waitFor require that the responses match the predicted results. When the interaction’s owner object is the environment, then an interaction is judged to be successful if the interaction occurs on the environment. When the interaction’s initiator and owner object are not the environment, then an interaction is judged to be successful when the tester observes that the interaction occurs on the system under test. This XML example illustrates a test that invokes the go() method on two different objects concurrently and then checks that both their return codes are false: <step id="S1"> <interaction object="user1" signature="go():boolean" type="call" /> <interaction object="user2" signature="go():boolean" type="call" />

</step> <step id="S2">

<interaction object="user1" signature="go():boolean" type="return">

<value>false</value> </interaction> <interaction object="user2" signature="go():boolean"

type="return"> <value>false</value>

</interaction> </step>


3.4.4.1.2 value The value element describes input for parameters, return responses, and constants. It has the two forms: <value ref=”object.constant|member|dataList|param|NULL”> or <value> input </value>. The ref attribute references a constant, member, dataList, param, or “NULL”. The object clause is optional. When the object clause is included it reference’s the object’s constant, member or dataList. When the object is not included it references element belongs to the global class or in the case of a param element it belongs to the current test case. Scoping rules apply so a test case parameter name takes precedence over a name used in a global class. Reference to the key word “NULL” means that the value is null. A dataList reference is only valid within a param, forbid, altValue or another dataList element. A param reference is only valid within another param element or interaction element. It is invalid to include the object clause. The param references the testCase’s param element in which the value element appears. A test case may call another test case with the subStep with their own param elements. These subStep param elements contain value elements that are applied to the called test case’s parameters. When the subStep’s value references a parameter the param belongs to the calling test case. The <value> input </value> form is best understood in the context of the type that it is describing, as illustrated by the following table: Type Value example <basic type=”int”/> <value>5</value> Type Null string value example <basic type=”string”/> <value ref=”NULL”/> Type Empty String value example <basic type=”string”/> <value><value> Type Null char value example <basic type=”char”/> <value>\0<value> Type Value example <enum name=”color”> <element name=”red”/> <element name=”green”/> <element name=”blue”/> </enum>

<value>blue</value>

Type Value example


<range name=”colorIndex” type=”int” from=”0” to=”2”/>

<value>2</value>

Type Value example <structure name=”person”> <field name=”fName” type=”string”/> <field name=”lName” type=”string”/> </structure>

<value>Harry</value> <value>Potter</value>

Type Value example <structure name=”person”> <field name=”fName” type=”string”/> <field name=”lName” type=”string”/> </structure> <interaction object="obj1" signature="name:person" suffix="fName" type="get">

<value>Harry</value>

Type Value example <array name=”intArray3” type=”int” size=”3”/>

<value>34</value> <value>124</value> <value>2</value>

Type Value example <array name=”intArray3” type=”int” size=”3”/> <interaction object="obj1" signature="A:intArray3" suffix="[1..2]" type="get">

<value>124</value> <value>2</value>

Writing XML markup characters in a value element: The special characters used for XML markup like <>&,” can not be directly used in a value. Rather a CDATA section or excape markup sequences are used. An XML CDATA section may occur anywhere that character data may occur. It is used to escape blocks of text containing characters that would otherwise be interpreted as part of the XML markup, such as ampersand (&) and less than (<). CDATA sections begin with the string '<![CDATA[' and end with the string ']]>'. CDATA Section Syntax:

CDSect CDStart CData CDEnd CDStart '<![CDATA[' CData (Char* - (Char* ']]>'

Char*)) CDEnd ']]>'

Within a CDATA section, only the CDEnd string is recognized as markup, so that left angle brackets and ampersands may occur in their literal form.


The other approach is to us escape markups: • & is & • ; is ' • > is > • < is < • ” is "

This is an XML value example that uses a CDATA section to input unparsed data:

<value> <![CDATA[ Unparsed Data: <step 1> asdf <step 2> zxcv -->>> OK <<<--

]]> </value>

This is an XML value example that inputs the text without CDATA:

<value> Data Unparsed < 1step >asdf <2step >zxcv --<<< OK <<<--

</value>

3.4.4.1.3 complexValue The complexValue element is a collection of values and other complexValues. It starts with <complexValue ref=”object.constant|member|dataList|param> and ends with </complexValue>. It contains the following intermixed elements. At least one element must be present:


The ref attribute has an equivalent meaning to the value element’s ref attribute. This is an example of an interaction that uses XML complexValue to pass an array of addresses. Each a address has two fields:

<interaction object="test" signature="printAddresses(addresses):int" type="call_return">

<complexValue> <complexValue> <value>127.0.0.1</value> <value>8080</value>


</complexValue> <complexValue> <value>127.0.0.1</value> <value>8081</value>

</complexValue> <complexValue> <value>127.0.0.1</value> <value>8082</value>

</complexValue> </complexValue>

<value>3</value> </interaction>

3.4.4.1.4 altValue This element is not supported in the current release of Spider [ 10]. The altValue is a collection of values and other complexValues. It starts with <altValue> and ends with </altValue>. It contains 1..∞ value or 1..∞ complexValue elements. It is not allowed to intermix the two kinds of elements. The contained elements represent the list of alternative data choices for an interaction’s response or input parameters. When it maps to a response each value or complexValue represents an expected response for the interaction. When it maps to an input parameter the test may choice any one of value or complexValues. A value or complexValue may reference the global class’s constants, members or dataList. A reference to a dataList is treated as additional list of possible data values. This XML segment invokes the o_db.state() – valid return values are not empty and empty. <interaction object="o_db" signature="state():string"/> <altValue> <value>not empty</value> <value>empty</value> </altValue> </interaction>

3.4.4.1.5 exception This element is not supported in the current release of Spider [ 10]. The exception element describes an exception thrown by an operation or an attribute access method. It starts with <exception name=”exception class”> and ends with </exception>. The exception contents may contain additional text describing the exception. The name attribute describes the exception name, i.e. class. <exception name="java.io.EOFException"/>


3.4.4.2 join This element is not supported in the current release of Spider [ 10]. The join element blocks the execution until all call interactions are completed. It starts and ends as <join/>. Join is useful when many call interactions are executed in a loop, causing a flurry of synchronous activity. However, the test must wait until all interactions have returned before checking their composite response.

3.4.4.3 timeout This element is not supported in the current release of Spider [ 10]. The timeout element describes a passage of time. It starts and ends with <timeout duration=”duration time” ref=” member|constant|param”/>. It contains no content. It contains the following attributes:

• duration - duration time. It is expressed in the xml schema type unsignedInt [ 18]. The duration is interpreted in milliseconds, but if there is a need, we can extend the definition to support other time units (optional).

• ref - references a object’s member or constant that contains the duration. It is expressed

in the form: <object>.<member|constant|param> . The referenced value must be an unsigned integer. The object clause is optional. When the object clause is included it reference’s the object’s constant or member. When the object is not included it references the current test case or the global class. Scoping rules apply so a test case parameter name takes precedence over a name used in a global class..

These are three XML timeout examples:

<timeout duration="12" /> <timeout ref="obj1.timeout"/> <timeout ref="param1"/>

3.4.4.4 tcStage The tcStage element is used in test cases both to identify particular control points (start and end of the test case, of the postamble, of the preamble, verdicts, etc.) and for documentation purposes. Each finite sequence of the test case should contain at least one tcStage to express its associated verdict. It starts and ends as <tcStage type=”kind of tcStage” verdict=”pass|fail|inconclusive”/>. It contains the following attributes:

• verdict – test case ending verdict (optional). Possible values are: o pass o fail


o inconclusive. • type - characterizes the current test case stage (optional). Possible values are:

o start_tc - initial stage of the test case o end_tc - final test case stage (possibly after a postamble) o end_preamble - the end of the preamble. o exit_ts – exit the test suite. Typically this is associated with a test failure so bad

that it is not worth continuing running the rest of the test suite. Each execution sequence of a test case (or test sequence) can be divided into three distinct parts: an optional preamble, a test body, and an optional postamble. In this ATS format, these parts will be distinguished by particular test case stages (see below). The preamble is a prefix of the test sequence whose purpose is to lead the SUT from its initial stage to the actual test starting point. It is useful, for instance, when a set of initializations has to be performed by the tester on the SUT to drive it into a particular state before starting the test execution itself. It can end with a fail or inconclusive verdict (when the SUT cannot be driven in this particular state). The test body is the effective part of the test case. It contains the sequence of stimulations and observations to be performed by the tester to decide whether or not the test is successful. It ends with a test verdict (pass, fail or inconclusive). The postamble is a suffix of the test sequence whose purpose is to lead the SUT back from the terminating state of the test sequence to a particular state (e.g., its initial state, or a “stable” state from which no more spontaneous actions can be performed by the SUT, etc.). It is useful, for instance, when one wants to link several test executions in sequence without completely resetting the SUT (but starting from a “clean” state). It can end with a fail or inconclusive verdict (when the SUT cannot be driven in this particular state). From the test execution point of view, this distinction is almost arbitrary: its behavior is the same in any part of the test execution sequence. However, from the user point of view, when the test does not succeed (i.e., a fail or inconclusive verdict is issued), it may be useful to know whether it is in the preamble, the test body or the postamble. Therefore, this distinction appears in the ATS format. Note the distinction between verdicts issued at the test’s preamble, test body and postamble exists in TTCN-3 [ 14]. This is an example of an xml tcStage element:

<tcStage type="end_tc" verdict="pass" />

3.4.5 altStep

The altStep elements represent a set of alternative steps. It is inspired by the TTCN-3 alt construct [ 14]. It starts with <altStep id=”test step identifier” sample=”all|random|inOrder”


nextPass=”next successful step id” nextFail=”next unsuccessful step id” > and ends with </ altStep>. It contains the following elements:

• 2..∞ step. It contains the following attributes:

• id - identifies the test step and may be pointed to by another step or altStep’s nextPass or nextFail attribute (required).

• sample – all|random|inOrder (optional but defaults to all). It describes which alternative steps to sample. The different chooses are described below:

o all - all of the alternative steps are sampled in parallel and a successful step’s nextPass step is followed. If more than one step is judged successful then one of the successful steps is chosen at random. The altStep’s nextPass step id is followed when the chosen successful step does not have a nextPass attribute (see nextPass below). All of the alternative steps’ nextFail attributes are ignored when sampleAll is chosen, but their altStep’s nextFail step may be used when all the alternative steps are judged unsuccessful (see nextFail below).

o random - one of the alternative steps is chosen to sample at random. The step’s nextPass step id is followed when judged successful. The step’s nextFail step id is followed when judged unsuccessful. The altStep’s nextPass or nextFail step id is followed when the chosen step does not have it own (see nextPass and nextFail below).

o inOrder - one of the alternative steps is chosen to sample cyclically in order upon each altStep execution. Its starts with the first alternative step and cycles back to the beginning after reaching the end of the list. The step’s nextPass step id is followed when judged successful. The step’s nextFail step id is followed when judged unsuccessful. The altStep’s nextPass or nextFail step id is followed when the chosen step does not have it own (see nextPass and nextFail below).

• nextPass – links to the next step or altStep when the chosen alternative step is judged to have executed successfully (optional. The nextPass attribute is only used when the chosen alternative step does not provide its own nextPass step id. It may point to itself. If it is missing, then successful alternative step execution continues in sequence to the next step or altStep.

• nextFail – links to the next step or altStep when all of the sampled alternative steps are judged to have executed unsuccessfully (optional). The nextFail attribute is only used when the chosen alternative step does not provide its own nextFail step id. It may point to itself. If the nextFail attribute is missing itself, then unsuccessful test execution fails the test case.

The following XML segment illustrates an altStep element. The tester sends a put() signal to the transmitter and waits for ack(bool) from the receiver. It receives ack(true) then it continues with the test to step s3. If it receive ack(false) then the test ends inconclusively. If it receives nothing for 100 milliseconds, it fails the test case.

<step id="s1">


<interaction initiator="tester" object="transmitter" signature="put()" type="send">

</interaction> </step> <altStep id="s2">

<step id="s2.a" nextPass="s3"> <interaction initiator="receiver" object="tester"

signature="ack(bool)" type="send_receive"> <value>true</value>

</interaction> </step> <step id="s2.b" nextPass="end_inconclusive">

<interaction initiator="receiver_1" object="tester" signature="ack(bool)" type="send_receive">

<value>false</value> </interaction>

</step> <step id="s2.c" nextPass="end_fail"> <timeout duration="100" /> </step>

</altStep> <step id="s3"> …. </step> …. <step id="end_inconclusive"> <tcStage type="end_tc" verdict="inconlusive" />

</step> <step id="end_fail"> <tcStage type="end_tc" verdict="fail" />

</step>

3.4.6 subStep

This element is not supported in the current release of Spider [ 10]. The subStep element is a test case step that executes another test case. It starts and ends with <subStep ref=”test case reference”/>. The referenced test case is executed inline. In other words its steps are executed as if they were a subroutine invoked by the current test case. It contains the following elements:

• 0..∞ param A param’s name attribute references the called test cases param. The param value or complexValue overrides its counterparts in the referenced test case. The param value or complexValue may reference the calling test case’s parameters. It contains the following attributes:


• ref - references a testCase id to call. • nextPass – links to the next step, altStep, or subStep where the test execution should

continue at the end of the current step that is judged to be successful. Success means that the reference test case completed with a tcStage pass verdict, or its last completed step is judged successful. It may point to itself. If the nextPass attribute is missing itself, then successful test execution just continues in sequence in the test case (optional).

• nextFail – links to the next step, altStep, or subStep where the test execution should continue at the end of the current step that is judged to be unsuccessful. Unsuccessful means that the referenced test case completed with a tcStage fail verdict, or its last completed step is judged unsuccessful. It may point to itself. If the nextFail attribute is missing itself, then unsuccessful test execution fails the test case (optional).

• nextInconclusive – links to the next step, altStep, or subStep where the test execution should continue at the end of the current step that is judged to be inconclusive. Inconclusive means that the referenced test case completed with a tcStage inconclusive verdict. It may point to itself. If the nextInconclusive attribute is missing itself, then inconclusive test execution completes the test case with an inconclusive verdict (optional).

This is a subStep XML example. Test case initialize is not run automatically, rather is called from test case T1 as a subStep. Likewise, initialize calls the close as a subStep. Each subStep replaces the test case’s parameters: <testCase id="initialize" topLevel="false">



<param name="p1" type="string" /> <step id="S1" nextFail="fail">

<interaction object="test" signature="remove(string):int" type="call_return">

<value>nofile</value> </interaction>

</step> <step id="pass"> <tcStage type="end_tc" verdict="pass" />

</step> 

<step id="fail"> <tcStage type="exit_ts" verdict="fail" />

</step> </testCase> <testCase id="TC1">

 <param name="param1"> <value ref="fileset[file]" />

</param> <subStep id="initialize" ref="initialize">


<param name="p1"> <value ref="param1" />

</param> </subStep>

</testCase> 4 testSuiteTrace The testSuiteTrace element describes an execution of the abstract test suite. The testSuiteTrace element and its abstract test suite may reside in the same or separate files. It starts with <testSuiteTrace id=”test suite identifier” timeDate=”time and date” executionEngine=”engine name”> and ends with </testSuiteTrace>. It contains the following elements:

• 0..1 directive • 0..∞ host • 0..∞ testCaseTrace • 0..1 results


• id - the trace’s execution id (required and unique). • dateTime - the trace’s creation date and time. It is expressed in the XML scheme

dataTime format [ 15] (optional). • executionEngine - identifies the execution engine (optional).

This an XML segment of a testSuitTrace element: <testSuiteTrace id="HWTrace1" dateTime="2002-04-08T11:34:00.000" executionEngine="Spider"/>

4.1 host The host element describes a SUT host used during test execution. It starts with <host name=”host name” address=”host address”> and ends with </host>. It contains the following elements:

• 0..1 directive • 0..∞ process

It contains the following attributes: • name – object name. Expressed in the XML schema format anyURI [ 17] (required). • address – ip address in which the object runs. Expressed in the XML schema format

anyURI [ 17] (optional). These are examples of two XML host elements:

<host name="TestDriver" address="9.148.32.113" /> <host name="host[1]" address="9.148.32.112">


4.1.1 process The process element describes a SUT process used during test execution. It is defined within a host element. It starts with <process name=”process name” clones=”number of clones”> and ends with </process>. It contains the following elements:

• 0..1 directive • 0..∞ sutObj

It contains the following attributes: • name – process name (required) • clones – the number of identical clones associated with the process (optional but defaults

to 1). This is an XML segment that describes one host:

<process name="proc[1]" name="2"> <sutObject name="hw[1]" atsObject="helloworld_1" />

</process>

4.1.1.1 sutObj The sutObj element describes a SUT object used during test execution. It starts with <sutObject name=”SUT object name” atsObj=”ATS object name”clones=”number of clones”> and ends with </sutObject>. It contains the following elements:

• 0..1 directive • 0..∞ init


• name – unique identification of SUT object (required). The id is referenced by the logEvent, stimulate and observe element. Since a sutObj may represent multiple clones the referencing elements, use indexing to indicate which clone is being referenced. For instance hw[2] references the 2nd hw[] sutObj clone (required).

• atsObj – reference to abstract test object (optional). • clones – the number of identical clones associated with the SUT object (optional but

defaults to 1). This is an XML segment that describes one SUT object with 5 clones of itself:

<sutObject name="hw" atsObject="helloworld_1" clones="5"/> This is an XML segment that describes one SUT object with two data initializations:

<sutObject name="hw" atsObject="helloworld_1"> <init name="case" type="string"> <value>upper</value>


</init> <init name="length" type="int"> <value>11</value>

</init> </sutObject>

4.1.1.1.1 init The init element describes the actual values used to initialize SUT object members. It starts with <init name=”member name” type=”member type”> and ends with </init>. It contains the following element:

• 1 value It contains the following attributes:

• name – member name (required) • type – the value type (optional)

This is an XML segment that describes one init value pair:

<init name="length" type="int"> <value>11</value>

</init>

4.2 logEvent The logEvent element describes a message originating the test execution environment or from a SUT object. It starts with < logEvent name=”member name” type=”message type” tester=”false|true” object=”SUT object” dateTime=”date and time”> and ends with </ logEvent>. It contains no additional elements but its content text describes the message. It contains the following attributes:

• type – describes the message type (optional). By convention, we include the following types, but any type can be used:

o error o ignored o exception o info o syserror o sysout o warning o verdict o observation

• tester – describes whether the event originated from the test execution environment or a SUT object (false | true). False means that the event originated from a SUT object. True


means that it originated from the test execution environment (optional, but defaults to false).

• object – references a sutObject element. When the tester attribute is false, then it describes the SUT object that sent the message. When the tester attribute is true, then it describes the subject SUT object for which the message is about (optional).

• dateTime - the trace’s creation date and time. It is expressed in the XML scheme dataTime format[ 15] (optional).

This an XML segment of a logEvent element: <logEvent tester="true" type="stepTrace" object="host[1].proc[1].hw[2]" dateTime="2002-04-08T11:34:01.100"> step id="T1.2" signature="next():char" actual: E expected: H </logEvent>

4.3 testCaseTrace The testCaseTrace element describes the execution trace of a specific abstract test case. It starts with <testCaseTrace id=”test case identifier” timeDate=”time and date”> and ends with </testCaseTrace>. It contains a mixer of the following elements:

• 0..∞ param • 1..∞ stepTrace • 1..∞ logEvent


• id – references the abstract test case for which this trace is associated (required) • dateTime - the traces creation date and time. It is expressed in the XML scheme

dataTime format [ 15] (optional). This is an example of an XML testCaseTrace element: <testCaseTrace id="TC1" dateTime="2002-04-08T11:34:00.000">

4.3.1 stepTrace The stepTrace element describes the execution trace of a specific abstract test case step. It starts with <stepTrace id=”test case step identifier”> and ends with </stepTrace>. Each stepTrace may choose between the following sets of elements:

• 1..∞ interactionTrace intermixed with 0..∞ logEvent • 1..∞ interactionTraceWithClones intermixed with 0..∞ logEvent • 1..∞ logEvent • 1 tcStage • 1 join • 1 timeout



• id – references the abstract test case step, altStep, or subStep. (required) This an XML segment containing a stepTrace with two interaction elements:

<stepTrace id="S0"> <interactionTrace object="host[1].proc[1].obj[1]"

interactionIndex="2"/> <interactionTrace object="host[1].proc[1].obj[1]"

interactionIndex="1"/> </stepTrace>

4.3.1.1 interactionTrace The interactionTrace element describes the actual trace of an abstract test case interaction. It starts with <interaction object=”sut object reference” signature=”interaction foot print” type=”interaction type” dateTime=”interaction date and time” interactionIndex=”index to the interaction”> and ends with </interaction>. It contains the following elements:

• 0..∞ value • 0..∞ complexValue • 0..1 exception • 0..∞ logEvent

The value and complexValue elements can be intermixed. They describe the interaction input, response and attribute value setting. The interaction’s signature and type (see below) are used to interpret the values; the translation is illustrated in the interaction element’s Table 1. The exception element is used when an interaction responds by throwing an exception. It contains the following attributes:

• The object attribute describes a specific SUT object to which this interaction belongs (required).

• interactionIndex - index to the abstract interaction. Typically, the index is used when there is more than interaction in a step, so there might be ambiguity as to which interaction this interacionTrace refers (optional).

• signature – interaction footprint. Typically, the signature is used when the actual member signature is different from the abstract member signature, otherwise it is omitted (optional).

• type - the interaction type. Typically, the type is used when the actual interaction type is different from the abstract type, otherwise it is omitted (optional). Valid type values are the same as those define for the interaction type attribute.

• dateTime - the traces creation date and time. It is expressed in the XML scheme dataTime format [ 15] (optional).

This an XML segment containing an interactionTrace with some input or output:


<interactionTrace object="host[1].proc[1].hw[1]"> <value>H</value>

</interactionTrace>

4.3.1.2 cloneTrace The cloneTrace element describes a test whose objects are cloned. Each testCase interaction is multiplied to test the multiple clones. It starts with < cloneTrace signature=”interaction foot print” type=”interaction type” dateTime=”interaction date and time” interactionIndex=”index to the interaction”> and ends with </cloneTrace>. It contains the following element:

• 1..∞ interactionTrace It contains the following attributes:

• interactionIndex - index to the abstract interaction. Typically, the index is used when there is more than one interaction in a step so there might be ambiguity as to which interaction this interacionTrace refers. (optional)

• signature – interaction footprint. Typically, the signature is used when the actual member signature is different from the abstract member signature, otherwise it is omitted (optional).

• type - the interaction type. Typically, the type is used when the interaction is added to the abstract test suite in the test execution directives [ 10], otherwise it is omitted (optional). Valid type values are the same as those define for the interaction type attribute.

This an XML segment containing a cloneTrace with two clone traces:

<cloneTrace> <interactionTrace object="host[1].proc[1].hw[1]"/> <interactionTrace object="host[1].proc[1].hw[2]"/>

</cloneTrace>

4.4 results The results element describes the results of a test suite execution. <results duration=”test suite execution duration time”> and ends with </results >. It contains the following element:

• 0..∞ outcome It contains the following attributes:

• duration - test suite execution duration time. Expressed as a XML schema duration type [ 16] (optional).

This is an XML results segment:

<results duration="P0Y0DT0H0M04.003S"> <outcome type="#traces">1</outcome>


<outcome type="#passes">0</outcome> <outcome type="#failures">1</outcome> <outcome type="#inconclusives">0</outcome> <outcome type="#errors">0</outcome>

</results>

4.4.1 outcome The outcome element describes the accumulated results of a test suite execution. It is defined within the results element. It starts with <outcome type=”outcome type”> and ends with </outcome>. Its content text describes the accumulated results as an integer and its type attribute qualifies the number. By convention, we include the following types, but any type can be used:

• #traces – the number of traces contained in the suite execution. • #passes – the number of pass verdicts contained in the suite execution. • #failures – the number of fail verdicts contained in the suite execution. • #inconclusives – the number of inconclusive verdicts contained in the suite execution. • #errors – the number of error verdicts contained in the suite execution.

This is an XML outcome segment:

<outcome type="#passes">0</outcome>

5 Tour of Samples See tour of samples in Model Driven Testing Overview [ 9]. 6 References

1. AGEDIS C.ATS Tool for graphically Creating and Editing Abstract Test Suites and Test Execution Directive, Raessler, H., 2002

2. AGEDIS Language Specification Document, Cavarra, A., Davies, J., 2002

3. AGEDIS Intermediate Format 2.0 Language with Test Directives, Bozga, M., Olvovsky, S., 2002

4. AGEDIS Test Generator 0.8, Verimag, Irisa, 2002

5. AGEDIS Test Suite Schema, testSuite.xsd, 2002

6. GOTCHA 4.0 Users Guide, IBM, Hartman, A., 2003



7. GOTCHA-TCBeans Reference Manual, Release 3.0.1, IBM, 2001.

8. GOTCHA Abstract Test Suite Specification Document, IBM, 2001.

9. Model Driven Testing Overview, IBM, 2003

10. Spider Test Execution Engine Users Guide, IBM, Kirshin, A., Nagin, K., 2003

11. Using model-based test generator to test for standard conformance (GOTCHA), Hartman, A., Farchi, E., and Pinter, S., IBM Systems Journal Vol. 41, No. 1, 2002

12. Toward a Test-Ready Meta-Model For Use Cases (UCBT), Williams, C., Practical UML-Based Rigorous Development Methods -- Countering or Integrating the eXtremists Workshop (co-located with UML 2001), July 27, 2001.

13. Test Generation Derived from Model Checking (TGV), Thierry Jeron and Pierre Morel, Computer Aided Verification, 1996.

14. TTCN-3 Draft new Recommendation ITU-T Z.140: The Tree and Tabular Combined Notation version 3 (Core Language), 2001

15. XML Schema dateTime CCYY-MM-DDThh:mm:ss (http://www.w3.org/TR/2001/REC-xmlschema-2-20010502/ - dateTime)

16. XML Schema duration PnYn MnDTnH nMnS (http://www.w3.org/TR/2001/REC-xmlschema-2-20010502/ - duration)

17. XML Schema anyURI (see http://www.w3.org/TR/2001/REC-xmlschema-2-20010502/ - anyURI)

18. XML Schema unsignedInt (http://www.w3.org/TR/2001/REC-xmlschema-2-20010502/ - unsignedInt)

http://www.w3.org/TR/2001/REC-xmlschema-2-20010502/