Upload
buitram
View
214
Download
0
Embed Size (px)
Citation preview
Agenda
Why standardized conformance test
specifications? The UMTS example
Introduction to Centre for Testing and
Interoperability (CTI)Interoperability (CTI)
The phases of the first TTCN-3 test
specifications
Cooperative ITS interoperability event
2
UMTS example
Standards Test System Test Lab CertificationTest Specifications
TC ITS
Standardized test specifications
• Are part of the product development process
• Are the basis for certification schemes
• Define clear and unambiguous tests
• Guarantee same test results independent of test systems and test labs3
Standards and
Test Specifications Prototype Test System
TC ITS
CTI
Centre for Testing and Interoperability (CTI)
SPECIFICATION
•Support ETSI Technical Committees on the
application of best practice protocol
specification methods, techniques and tools.
•e.g., ASN.1, UML, MBT (Model Based Testing)
•3GPP
•AERO
•ATTM
•BRAN
•CLOUD
•DECT
•eHEALTH
4
VALIDATION
TESTING
•Support ETSI Technical Committees on the
validation of standards.
•Mainly Plugtests events (organisation and
provision of testing expertise)
•Support ETSI Technical Committees on ALL
testing aspects including the development of
test frameworks, methodologies, test
specifications. Mostly through
participation/leadership of STFs
•e.g., STF 160, ITS, TTCN-3
•eHEALTH
•ERM
ESI
•HF
•INT
•ITS
•LI
•MTS
•PLT
•SCP
•STQ
•TETRA
•TISPAN
The initial phase: study the ITS specific features and provide appropriate guidance• ITS Testing Framework
The second phase: provide formal conformance test specifications• For CAM / DENM / GN (Geonetworking)
The third phase: develop a platform for conformance testing• To validate the conformance test specifications
Different Phases (projects) for testing
• To validate the conformance test specifications
• To give feedback on the conformity of products
Further phases:• Maintenance of test specifications, following updates of the base
specifications
• Development of further test specifications for other base standards,
• Development of test specifications for interoperability events (Plugtests)
• …
Guidance for the application of the two types of test methods commonly used in ETSI:• Conformance testing
• Interoperability testing
The ETSI guide consists of two main parts:• The first part is suitable for non-testing experts to understand how the testing
strategy commonly used at ETSI can be applied to ITS (clauses 4 to 7)
The initial phase (Feb – Oct 2010)
strategy commonly used at ETSI can be applied to ITS (clauses 4 to 7)
• The second part is more dedicated to testing experts, for understanding specific implementation and design principle to be applied for testing ITS systems (clause 8 and annexes A and B)
One deliverable was produced:
• EG 202 798: Intelligent Transport System (ITS); Testing Framework for conformance and interoperability testing
ACHIEVED
Implementation of CAM/DENM/GN test specifications according to the ITS Test Framework• ensuring a harmonized global approach towards testable ITS standards
and related test standards based on a common test framework
For example• The test purpose table proforma, which is providing:
•
The second phase (May – Dec 2010)
• The test purpose table proforma, which is providing:• all relevant fields to describe the expected test behaviour and the test
environment,
• A style guide for writing test behaviour in a clear and unambiguous manner
• Test architecture enabling to apply the standardized formal test methodologies
• ISO 9646 (testing concept, test descriptions, test language)
• ES 201 873 (improved test language)
ACHIEVED
PICS (Protocol Implementation Conformance Statement)• A PICS proforma is a questionnaire or check list that covers all requirements,
optional and conditional functions, procedures, messages, timers and other capabilities identified in the ITS standards.
• PICS proforma are to be completed by the suppliers of the ITS station to be tested
• See ISO 9646-7
Test purpose
Different parts of test specifications
Test purpose• A test purpose is a prose description of a well-defined objective of testing
• It focuses on a single requirement, which shall be followed by ITS equipment to be compliant with the standard
Abstract Test Suite / Abstract Test Case• An abstract test suite is a set of abstract test cases
• A test case is a complete formal specification of the actions and protocol messages required to achieve a given test purpose.
• Uses dedicated “test language”, enabling a direct compilation and then execution of the test cases (e.g. TTCN-3)
• See ISO 9646-1 to -3 and ES 201 873 (specifying the test language TTCN-3)
• See also http://www.ttcn-3.org/StandardSuite.htm
TTCN-3 details of CAM/DENM/GN testing
See ITS presentation in T3UC 2011 in Bled, Slovenia for all
details http://www.ttcn3uc.eu/
Test Purposes
• Use of Framework
• Systematic Approach• Systematic Approach
• Project specific TP Rules
Test Suite Design Decisions
• Port or PTC
• Upper Tester Design
• PCO Placement
• Naming Conventions
• Structure and modularization of the TTCN-3 code
9
Further Phases - Test Platform
Project duration from March 2011 – November 2012
Design, build and validate a TTCN-3 test platform
Validate the CAM/DENM/GN test specification by running test against real implementations
Allow for other protocol tests to be included
Allow for other hardware controller components to be Allow for other hardware controller components to be included
See http://www.etsi.org/WebSite/Technologies/IntelligentTransportSystemsTesting.aspx
Further Phases
Cooperative ITS Interop Event
Idea was approved during the TC ITS meeting in Mainz (Oct.
2010)
To organize an interoperability event with the support of CTI
• Offer the capability to proceed with face to face test sessions, where
vendors can assess the level of interoperability of their vendors can assess the level of interoperability of their
implementations
• Verify their correct understanding of the base standards requirements
• Rather Q1 2012 due to the availability of implementations
• Subscribe to [email protected] at
http://www.etsi.org/WebSite/document/Technologies/ITS%20Call%20
for%20Interest%20ITS%20Interoperability%20Event.pdf
ETSI is willing to cooperate with other ITS players (ERTICO,
C2C CC)
11
Cooperative ITS Interop Event - Scope
Interoperability testing
• V2V and V2I
• Involves OBE, RSE and infrastructure
• Lab conditions or field trial
• The first event could aim to verify communication stacks
• access, networking/transport, facilities
• But application are required to trigger the communication stacks
• Application scenarios
• Synchronization and synergy with ongoing FP7 projects
Pre-conformance testing:
• Provides pre-conformance test services as an output from Test
Platform project:
• Enable providing a compliance check before IOP tests
• A good opportunity to assess the usefulness of conformance testing
• Can also be a part of the STF VJ validation task.12
Making it Happen
Industry (e.g.
Fora)Participants
(Vendors)
CTI
Coordination, Organisation,
Technical Expertise etc.
Technical
Partner
13
Standards
(ETSI, IEEE,
ISO, ...)
How do we Organise Plugtests?
Event Co-ordination
• Event Promotion
• Legal Aspects
• Finance
• Logistics
• Dedicated Event Supervisor
14
Technical co-ordination
• Test plan
• Test Infrastructure
• Test Scheduling
• Test Session Reporting
• Final technical report
Typical Plugtests Events
In operation since 1999 (in CTI since 2007)
Nearly 150 events, involving more than 4000 engineers
For many diverse technologies
2010
EUROCAE#4 (Air Traffic Management)
15
EUROCAE#4 (Air Traffic Management)
Femtocell#1
DECT CAT-IQ 2.0#1,#2
GPON#5 (Gigabit Optical Networks)
Electronic Signature XaDES, CaDES
SIPit#27 (Taiwan)
Planned in 2011
Femtocell#2,#3, IMS#4, Eurocae#5, SIPit#28, IPv6, DSC Marine Radios
Conformance testing:
• Process for testing that an implementation is compliant with a protocol standard, which is realized by test systems simulating the protocol with test scripts executed against the implementation under test.
Interoperability testing:
Few definitions
Interoperability testing:
• Process for testing that devices can inter-operate, which is realized by connecting devices from different vendors and operating them, either manually or automatically, according to scenarios based on a protocol standard.
The ETSI Guide EG 202 798 is a Testing framework:
• a document providing guidance and examples necessary for the development and implementation of a test specification.
The functional TTCN-3 test architecture
The "Test Control" enables the management of the TTCN-3 test execution
The "TTCN-3 test components" are processes providing the test behaviour
The "Codec" is a functional part of the test system to encode and decode messages (ASN.1)
The "Test adapter" (TA) realizes the interface between the TTCN-3 ports using TTCN-3 messages, and the physical interfaces provided by the IUT
The “Upper Tester Transport” (Bus controller) enables sending or receiving service primitives from protocol layers above the IUT or from the management / security entity
23