Achieving Grid Interoperability The ETSI Approach

World Class Standards

Achieving Grid InteroperabilityThe ETSI Approach

Stephan Schulz, Ph.D.

ETSI Centre for Testing and Interoperability

2


Presentation Outline

On achieving interoperability Conformance versus interoperability testing

ETSI test specification development Requirement extraction Test purposes Conformance Tests Interoperability Tests

Creation of test specifications at ETSI Centre for Testing and Interoperability Specialist Task Forces

On grid testing Grid STF Review of Byte IO and Grid RPC test specifications

3


Interoperability is …

The ultimate aim of standardisation in Information and Communication Technology

The red thread running through the entire standards development process, it’s not an isolated issue Not something to be somehow fixed at the end

ETSI approach ETSI produces with the required degree of parallelism Base

Standards and Test Specifications Base standards should be designed with interoperability in mind

• Profiles can help to reduce potential non-interoperability

Two complementary forms of testing• Conformance testing• Interoperability testing (more formal IOT”)

Testing provides vital feedback into the standardization work

4


About Conformance Testing

Is Black-Box testing Stimulation and Response

TestSystem

Device

Point of Control and Observation (PCO)

Reqs.

5


Conformance is Usually Tested Layer-by-Layer

lat

ii

1 2 3

4 5 67 8 9

* 8 #

latigid

TestSystem

SUTAPI

MMI

L3

L2

PHY

SUT = System Under Test

IUT = Implementation Under Test

L3

Upper Tester

Lower Tester

Lower Adapter

Upper Adapter

L3 Test

6


Characteristics of Conformance Testing

Tests IUT against requirements specified either in a base specification or profile (standard) Gives a high-level of confidence that key components of a device or

system are working as they were specified and designed to do Usually limited to one requirement per test

High IUT control and observability Can explicitly test error behaviour Can provoke and test non-normal (but legitimate) scenarios Can be extended to include robustness tests

Test execution can be automated and repeated Requires a test system development (and executable test cases) Can be expensive (e.g., radio-based test system) Tests under ideal conditions

Tests are thorough and accurate but limited in scope At level of detailed protocol messages, service primitives, or

procedure calls

7


Limitations of Conformance Testing

Does not prove end-to-end functionality (interoperability) between communicating systems Conformance tested implementations may still not interoperate

This is often a specification problem rather than a testing problem! Need for minimum requirements coverage or profiles

Tests individual system components But a system is often greater than the sum of its parts! Does not test the user’s ‘perception’ of the system

Standardised conformance tests do not include proprietary ‘aspects’ Difficult to test via proprietary interfaces, e.g., user APIs Can however be done by a manufacturer with own conformance

tests for proprietary requirements

8


About Interoperability Testing

End-to-End Interoperability of devices What is being tested? Assignment of verdicts?

Need to distinguish between More formal interoperability testing (= test specifications) versus Informal interoperability events used to validate/develop

technologies (e.g., ETSI Plugtests like events)

Device1 Devicen

Device2

9


SUT

API

MMI

L3

L2

PHY

API

MMI

L3

L2

PHY

QE = Qualified

Equipment

EUT = Equipment Under Test

ETSI Methodology for Interoperability Testing

Interoperability Testing(of terminals)

1 2 3

4 5 67 8 9

* 8 #

QE EUT

1 2 3

4 5 67 8 9

* 8 #

Means of Communication (MoC)

10


Characteristics of Interoperability Testing

It is system testing Tests a complete device or a collection of devices

Shows that (two) devices interoperate But within a limited scenario

Tests at a ‘high’ level (perception of users) Tests the ‘whole’, not the parts or layers

• E.g., protocol stacks integrated with applications Tests functionality

• Shows function is accomplished (but not how)

Does not usually involve test systems Uses existing interfaces (standard/proprietary)

Interoperability testing looks at end-to end functionality Less thorough but wide in scope Gives a high-level of confidence that devices (or components in a

system) will interoperate with other devices (components)

11


Limitations of Interoperability Testing

Does not prove that a device is conformant Interoperable devices may still interoperate even though they are

non-conformant to a base specification or standard

Cannot explicitly test error behaviour or unusual scenarios Invalid conditions may need to be forced

(lack of controllability) Has limited coverage (does not fully exercise the device)

Are executed manually (usually) Difficult to automate, tests use more time to be prepared than

executed.

Does not prove 100% interoperability with other implementations with which no testing has been done ‘A’ may interoperate with ‘B‘ and ‘B’ may interoperate with ‘C’. But

it doesn’t necessarily follow that ‘A’ will interoperate with ‘C’.

12


Conformance and Interoperability Testing are Complementary

ETSI experience As you move up a system stack the emphasis should change

from conformance to interoperability testing

Lower layer protocols, infrastructure Emphasis on conformance

Middleware, enablers Combination of conformance and

interoperability testing

Services, applications, systems Emphasis on interoperability testing

Conformance testing can be viewed as a pre-requisite to interoperability testing

Interop TestingConforman

ce Testing

Interoperability

13


Typical ETSI Standards Development Process

(Unit) Conformance Testing

Interoperability Testing

Products mature from prototypes to commercial products

ETSI: Development of base standards

Certification

Ind

ustr

y

time

Conformance Test Specifications

Interoperability Test SpecificationsIterative feedback

Iterative feedback

Sta

nd

ard

s

Bod

y

Testing provides vital feedback, e.g., holes, errors, or inconsistencies, in base standards

14


Stages in ETSI Test Specification Development

Base Standard or Profile Specification

Requirement Catalogue

Implementation Check List (ICS/IFS)

Identification of Test Group Structure (TSS)

Specification of Test Purposes (TP)

Specification of Test Cases (TC)

Validation of Test Cases

Specification of Test Descriptions (TD)

15


Requirements Cataloguing

Each requirement is identified, extracted, and catalogued as follows (example from IPv6): Requirement type:

• Mandatory (MUST, MUST NOT, SHALL, SHALL NOT)• Recommended (SHOULD, SHOULD NOT)• Optional (MAY, MAY NOT, COULD)

Requirement role (tested entity):• E.g., Host, Router, Node (Host or Router)

Requirement context Requirement text Base document citation and reference Functional groupings

• E.g., Process Fragmented packet, Generate ICMPv6 Error Type etc.

16


Test Purposes

Precise descriptions of the purpose of a test in relation to a particular (base standard) requirement Meaningful to a larger audience than technical experts

Define the functionality being tested, i.e., WHAT? Should be formulated using a similar level of information content

and wording as the corresponding requirement description Should only mention directly relevant aspects of interactions but

not describe them in detail Reference a test configuration (i.e., architecture) It is possible to have several test purposes per requirement

Specified at ETSI in Natural language, or ETSI’s Test Purpose Language (TPLan) (www.tplan.info) But definitely not coded!

17


TPLan Example from Digital Public Mobile Radio

TP id : TP_PMR_0406_01summary : 'Header frame acknowledges connect request'TP type : conformanceRQ ref : RQ_001_0406 –- Catalogue Identifier IUT Role : CSF -- Configured Service Function (CSF)config ref: CF_dPMR_CSF_01 -- CSF Implementation Under -- Test (IUT) and TESTER

with { IUT in standby }

ensure that { when { IUT receives a Connection_Request } then { IUT sends an Acknowledgement_Frame }

18


Interoperability Test Description

Specifies detailed steps to be followed to achieve stated test purpose, i.e., HOW?

State steps clearly and unambiguously without unreasonable restrictions on actual method: Example:

• Answer incoming call

NOT• Pick up telephone handset

Written in a structured and tabulated natural language so tests can be performed manually

In theory it is also possible to increase automation in with test scripts, e.g., with TTCN-3 Proprietary nature of user APIs makes this in practice difficult Requires additional adapter development from equipment vendors

19


Example Interoperability Test Description

Response to ‘ping’:

20


Conformance Test Suite A collection of detailed test cases or scripts that implement test

purposes Can be compiled and executed, e.g., when using TTCN-3

Specifies HOW to test Implements also handling of, e.g., unexpected or background behaviour, or

returning the SUT to the initial state after a test Composition of test cases

Each individual test has a preamble, test body (i.e., implementation of the Test Purpose), and postamble

Test components may be used clearly separate interactions with different logical IUT interfaces during a test

Assigns test verdicts Configurable at run-time, e.g., SUT address A test suite is not a test systems, but

ETSI ensures all tests can be compiled And validated by executing the tests against one or more real

implementations on at least one test system

21


What is TTCN-3?

Testing and Test Control Notation Version 3 Internationally standardized language developed specifically

for executable test specification Specified by ETSI MTS Technical Committee Is independent of a specific IUT or IUT interfaces Is independent of a test execution environment Standard available at portal.etsi.org via ETSI programme

Allows unambiguous implementation of tests Look and feel of a regular programming language Good tool support (today 6 commercial tools available) Successfully deployed at ETSI and industry in a variety of

application domains e.g., telecom, automotive, software, etc.

Good text book available http://www.wiley.com/legacy/wileychi/ttcn-3/

www.ttcn-3.org

22


Example TTCN-3 Test Case

testcase TC_COR_0047_01() runs on Ipv6Node system EtherNetAdapter { f_cf02Up(); // Configure test system for HS->RT // No preamble required in this case f_TP_HopsSetToOne(); // Perform test // No postamble required in this case f_cf02Down(); // Return test system to initial state}

function f_TP_HopsSetToOne() runs on Ipv6Node { var Ipv6Packet v_ipPkt; var FncRetCode v_ret := f_echoTimeExceeded( 1, v_ipPkt ); if ( v_ret == e_success and v_ipPkt.icmpCode == 0 ) { setverdict(pass);} else { setverdict(fail); }}

function f_echoTimeExceeded(in UInt8 p_hops, out Ipv6Packet p_ípPkt ) runs on Ipv6Node return FncRetCode { var Ipv6Packet v_ipPacket; var FncRetCode v_ret; ipPort.send( m_echoReqWithHops(p_hops) ); alt { [] ipPort.receive( mw_anyTimeExceeded ) -> value p_ipPkt { return e_success } [] ipPort.receive { return e_error } }}

23


Validation of (Conformance) Tests

Typical test suite validation requires That TTCN-3 tests compile on several tools (5 at ETSI) Tests executed on one or more platforms against various

implementations of the standard (usually by ETSI members)

Requires very close co-operation with test tool suppliers, test platform providers, equipment manufacturers etc. Strict and well-defined processes between

• Test lab (who execute tests[, certify] and provide feedback)• And ETSI (who updates the tests)

Timely availability of stable base specifications, test platforms (tools) and implementations is essential

ETSI does not certify products!

24


RequirementsCatalogue

Test PurposesTest Description

Test Cases(TTCN3)

Base StandardDatabase

Web interface, browsing by function, user-defined search and filter, traceability, document generation, …!

CTI Test Specification Data Base

25


Example: ETSI IP Testing Library

http://www.ipt.etsi.org/STF295-ph1/

26


Test Specification Development at ETSI

ETSI uses a variety of resources for test specification development ETSI Technical Committee on Methods for Testing and Specifications

(TC MTS)• Composed of testing experts from major telecom companies, tool vendors

research institutes and academia• Works on guidelines and languages for ETSI specification development,

e.g., TTCN-3, TPLan, IP Testing framework

ETSI Plugtests Service• Organizes interoperability events

ETSI Centre For Testing and Interoperability (CTI) ETSI Specialist Task Forces (STF)

27


In-house team of experts providing direct support as a service to ETSI Technical Bodies (e.g., ETSI GRID TC) on the use of formal techniques in standards

Supports ETSI test specification development - conformance and interoperability (60%)

Assistance in development of protocol standards and profiles, e.g., application of modern specification techniques (35%) ‘Standards Engineering’

Validation of standards (5%) E.g., testbed activities (IPv6, SIP/H.323)

ETSI Plugtests™ Service Validation of standards through interoperability events

About ETSI CTI

28


Areas ETSI CTI Involvement

Cellular: GSM and 3G (UMTS) terminals

WiFi: HiperMAN, HiperACCESS, WiMax

VoIP: H.323, SIP, SIGTRAN

Service Creation: OSA/Parlay (API, IDL)

IPv6: Core, Security, Mobility, v4-v6

Cordless phones: DECT

Radio communications: TETRA, DMR, dPMR

Access terminals: FSK, SMS

Broadband: ISDN, DSL

Smartcards: Readers, cards, security modules

Intelligent Transport Systems (ITS): DSRC

Early NGN/IMS

Future: More Security, GRID ...

29


About ETSI STFs

Team of experts seconded mostly from the ETSI membership 25-30 experts in 2006 Supported and often lead by CTI experts

Funded either by ETSI STF budget, European Union, or by commercial agreement Can range from small (2 pm) to large (58pm over 2-3 years) Approx. 2,7M€ of expert resource per year

Produce deliverables which are turned into freely and publicly available standards after approval from ETSI members Requirement catalogue, test purposes, test descriptions, test

cases Frameworks, specification infrastructure, guidelines Some validation etc.

30


Upcoming GRID STF

ETSI GRID Technical Committee has created an STF Completely funded by EC DG ENTR

Purpose is to identify gaps in grid specifications and develop GRID testing framework At this point focus is on analysis of available grid specifications Review of other GRID testing frameworks, e.g., ETICS project Actual tests will come are the next step

• Does not make sense to specify tests without clear scope

Call for experts is out (CL07_2532) Application deadline May 27th !! Interview of candidates on June 11th Start of work expected in Q3 Any ETSI or OGF member can apply – do so today

31


Results of GRID Test Specification Analysis

ETSI CTI has been asked by ETSI TC GRID and OGF members to review Byte IO and Grid RPC test specifications Dec 2006

Summary of observations Lack of references to the specification, e.g., document clause or

quotation of specification text Test cases mix discussion of mandatory from optional features Inconsistent use of terminology, e.g., “test case”

• Grid RPC corresponds to ETSI test purpose(Grid RPC “test code” corresponds to ETSI test case)

• Byte IO corresponds more to ETSI test description Missing isolation of test purpose in Byte IO test specification

makes understanding of test objective and verdict criteria difficult Both documents from ETSI point of view specify conformance

tests but claim to be interoperability tests (Byte IO specifies byte strings!)

Test cases should be more implementation language agnostic

32


Thank You!

Feel free to contact us:

e-mail [email protected]

WWW www.etsi.org/ptcc

Documents

Achieving Grid Interoperability The ETSI Approach