Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
VII. Conformance Testing Methodology
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-1 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-1 Dr. Junzhao Sun
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Outline
General conceptsTesting documentsAbstract test methodsAbstract test suitesTest realizationConformance assessment process
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-2 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-2 Dr. Junzhao Sun
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Outline
General conceptsTesting documentsAbstract test methodsAbstract test suitesTest realizationConformance assessment process
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-3 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-3 Dr. Junzhao Sun
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
What is Conformance Testing?
The process of testing the extent to which implementations of protocol entities adhere to the requirements stated in the relevant standard or specificationstandard or specification
Concerned with external behavior (black box)Prerequisites for interoperability
The primary objective of conformance testing is to increase the probability that different product implementations actually interoperate.
Testing of implementationsBut NOT with performance, reliability, fault tolerance, efficiency, etc.
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-4 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-4 Dr. Junzhao Sun
y y
ISO 9646:A conforming implementation is one which satisfies both static and dynamic conformance requirements, consistent with the capabilities stated in the Protocol Implementation Conformance Statement (PICS)
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Standard
ISO/IEC 9646 (ITU-T X.290)Conformance Testing Methodology and Framework (CTMF) for the Implementations of OSI and ITU ProtocolsImplementations of OSI and ITU Protocols
7 part standardISO 9646-1. General ConceptsISO 9646-2. Abstract Test Suite SpecificationISO 9646-3. Tree and Tabular Combined Notation (TTCN)ISO 9646-4. Test RealizationISO 9646 5 Conformance Assessment Process
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-5 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-5 Dr. Junzhao Sun
ISO 9646-5. Conformance Assessment ProcessISO 9646-6. Protocol Profile Test SpecificationISO 9646-7. Implementation Conformance Statement
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Conformance Testing Limitations
Aspects of protocol testing not considered in ISO 9646Interoperability testing, performance testing and robustness testingTh li ti f i i itiThe realization of service primitivesInternal aspects of the system implementationThe system environment and testing toolsProcurement proceduresTest certificates, and accreditation of test labs
Conformance testing will always be limited, for technical and theoretical as well as for economic reasons
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-6 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-6 Dr. Junzhao Sun
theoretical as well as for economic reasonsNo guarantee can be given that an implementation absolutely conforms to the relevant protocol standardsConformance testing with positive test results will increase the degree of confidence in a protocol implementation
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Basic Standard vs. Profile
Base standard is a standard relative to a single protocol in a given OSI layerA profile is a set of one or more base standards necessary forA profile is a set of one or more base standards necessary for accomplishing a particular function
a multilayer stack of several adjacent base standardsFunction might be a type of application, or a transport service based on specific telecommunications techniquesA protocol can be a base standard by itself, or can belong to a profileThe purpose of defining a profile is to facilitate interoperability
P fil i t f ld
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-7 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-7 Dr. Junzhao Sun
Profile is twofold Multiplicity: a profile may retain options in a set of base specificationsRefinement: a profile restricts the choices of options in the base specifications. It may also specify additional conformance requirements.
We will mainly concentrate on the conformance testing of protocol
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Conformance Testing Process (1/2)
Test generationTo generate the abstract test suites (ATSs) manually or automaticallyDevelopment of proformas for implementation conformance statementsDevelopment of proformas for implementation conformance statements (ICS) and implementation extra information for testing (IXIT)
Test realizationThe realization of test tools and means of testing, capable of executing particular test suites To generate the executable test suites (ETSs) with ATSs
Including both selection and parameterization process
Test execution
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-8 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-8 Dr. Junzhao Sun
Test executionPerform test to an implementation with executable test case selected
Conformance assessment processPerformed at a test laboratory for a client, with a protocol conformance test report (PCTR) as the ultimate result
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Conformance Testing Process (2/2)
Abstract test suiteStandard protocol
specification
Test generationTTCN
System Under
Executable test suite
specification
Protocol implementation
Test realization
Test execution
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-9 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-9 Dr. Junzhao Sun
Under Test
IUT Tester Verdict: (result)• accept• failure• inconclusive• error (in test HW)
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Test types in testing processBasic interconnection tests (BIT)
To determine whether interconnection is possible at all as an initial stage
Test Types
To determine whether interconnection is possible at all as an initial stageTo establish main features of the protocol are implementedBasic interconnection tests are standardized but may be omitted
Capability testsTo validate that all the the observable external static capabilities of the IUT are valid with respect to the static conformance requirements expressed in the PICSCapabilities tests are standardized and mandatory
Behavior tests
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-10 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-10 Dr. Junzhao Sun
To check dynamic conformance of an IUT with respect to the dynamic conformance requirements specified in the protocol standardBehaviour tests are standardized and mandatory
Conformance resolution testsSpecial tests for purposes beyond ordinary behavior testing e.g. exceptionConformance resolution tests are not standardized and not mandatory
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Test Types
Behavior tests
Capability tests
Basic i i
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-11 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-11 Dr. Junzhao Sun
interaction tests Conformance
resolution tests
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Test Results
Test resultsPass
Observed test outcome shows that the test purpose has been fulfilled and theObserved test outcome shows that the test purpose has been fulfilled and the IUT has displayed only valid behaviors as specified in protocol standards
FailTest outcome proves that some conformance requirement has been violated
InconclusiveNeither pass nor fail can be assigned to the test outcomeHappens when underlying provider signals problems e.g. disconnect or reset
Documentation of test results
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-12 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-12 Dr. Junzhao Sun
SCTR = system conformance test reportPCTR = protocol conformance test reports
Requirements on test resultsRepeatable, comparable and auditable to a reasonable degree
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Outline
General conceptsTesting documentsAbstract test methodsAbstract test suitesTest realizationConformance assessment process
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-13 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-13 Dr. Junzhao Sun
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Test Generation
TPsSt nd rd ATS
Standardized (Protocol) Implementation Conformance Statement (tables)
Standardized Test Purposes (in English)
TPsStandard
Standardized Abstract Test Suite (in TTCN)
ATS
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-14 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-14 Dr. Junzhao Sun
ICSStandardized (Protocol) Implementation eXtra Information for Testing (tables)
IXIT
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Test Specification to Be Produced
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-15 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-15 Dr. Junzhao Sun
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Conformance Requirements
Systems are called conforming if their externally visible behavior fulfils conformance requirements stated in protocol specification
M d t i t l h t b b dMandatory requirements, always have to be observedConditional requirements, have to be observed under conditions defined
Dependence on previous choices of parameter values and optionsOptional requirements, can be freely selected
Static conformance requirementsSpecify permitted combinations of capabilities implemented in real systemE g value ranges for size parameters or timers as in ICS
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-16 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-16 Dr. Junzhao Sun
E.g. value ranges for size parameters or timers, as in ICS
Dynamic conformance requirementsSpecify the observable behavior of implementation permitted by standardDefine actual protocol e.g. PDU exchanges, timing, format of PDUs.
Profile is based on base specifications with additional constraints
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
ICS (1/2)
ICS = Implementation Conformance StatementA checklist of the capabilities supported by the IUT
A ti i l t d b li t t t f i tA questionnaires completed by client to present conformance requirementsProduced by filling out an ICS proforma with static requirementsProvides an overview of the features and options that are implementedUsed to select and parameterise test cases and as an indicator for basicinteroperability between different products
PICS = Protocol Implementation Conformance StatementIUT realizes a single protocol
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-17 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-17 Dr. Junzhao Sun
Profile ICSIUT contains a set of interrelated protocols, e.g. a stack of protocols
ICS is not just for testingUsed by specification writer to clarify the intent of standardsUsed by product implementer for selection of features
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
ICS (2/2) RL = Requirements List
Standardization organizations Client
Profile ICSBase
specificationBase specificationBase
specificationBase specification
PICS-proformaPICS-
proformaPICS-proformaPICS-
proforma
PICSPICSPICSPICS
Profile ICS
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-18 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-18 Dr. Junzhao Sun
Profile specification
Profile RL
Profile specific ICS proforma
Profile RL
Profile specific ICS
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
ICS Proforma – Protocol Capability
Status of capability in protocol specificationMandatory, optional , conditional
hibit d l d dprohibited, excludedout-of scope, not applicable
Capabilities can be grouped into categoriesMajor capabilitiesSupported PDUs and fields within PDUsSupported timersE di i ti
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-19 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-19 Dr. Junzhao Sun
Encoding variationsNegotiation mechanismsError procedures
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
ICS Proforma – Structure
ICS proforma corrigenda (version information)ICS proforma instructions, abbreviations and terms
Identification of the implementation, supplier and client
Major capabilities table
PICS proforma tables
Identification of the implementation, supplier and clientIdentification of the protocol, information object or profile
Protocol, information object or profile corrigenda
Supported negotiation Supported Supported
PDUs TableSupported Supported
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-20 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-20 Dr. Junzhao Sun
Multi-specification dependencies table
Supported protocol error handling table
capabilities tabletimer tables
Supported parameters table in PDUSupported parameters
table in PDUSupported parameters table in PDU
PDUs TablePDUs TableSupported
PDUs Table
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
ICS Proforma – ISDN ExampleIndex Protocol capability
Does the implementation support…Conditionsfor status
Status Reference Support
M1 3.1kHz audio calls O.1 6.1.2 [ ] Yes[ ][ ] No
M2 Data calls O.1 6.1.1 [ ] Yes[ ] No
M3 Packet mode calls O 6.3 [ ] Yes[ ] No
M4 Interworking with non-ISDN networks M 8.4 [ ] Yes[ ] No
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-21 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-21 Dr. Junzhao Sun
M41 Calling line identification presentation M 8.4 [ ] Yes[ ] No
M421 Calling line identification restriction M1not M1
MN/A
8.4 [ ] Yes[ ] No
M422 Advice of charge M1not M1
MN/A
8.4 [ ] Yes[ ] No
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
IXIT, SCS
IXIT = Implementation eXtra Information for TestingContains additional information beyond specification as ICS
A IXIT f fill d t b t t l b t d li tAn IXIT proforma filled out by test laboratory and clientWith both protocol IXIT and profile IXIT, as in ICS typesSpecify additional testing information related to an specification
Physical setup, connection of test, and testing environmentSUT hardware, socket, telephone numbers, and tester SAP addressesTest-specific timer or parameter values
SCS = System Conformance StatementLi t f t ifi ti f th IUT l t d ICS l t d t t
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-22 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-22 Dr. Junzhao Sun
List references to specifications of the IUT, related ICSs, related testreports, together with configuration information of SUT, I.e. an index
Conformance of an IUT: An IUT is said to be conforming if itsatisfied the static and dynamic requirements of the relevantstandards referenced in SCS and capabilities in ICSs.
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
IXIT Proforma – ISDN Example
#PC_BASIC TRUE
#PC POINT TO POINT FALSE#PC_POINT_TO_POINT FALSE
#PC_POINT_TO_MULTIPOINT TRUE
#PC_EN_BLOC_RECEIVING TRUE
#PC_OVERLAP_RECEIVING TRUE
#PX_CDPN_OCTET3 ’81’O
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-23 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-23 Dr. Junzhao Sun
#PX_CPN_PTC1 ‘323830313531’O
#PX_LCPN_PTC2 ‘07’O
#PX_CPN_PTC2 ‘323830313532’O
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
TSS & TP
TSS = Test Suite StructureTP = Test PurposesTSS & TP are derived from the relevant base standards.
Provide an informal, easy-to-read description of each test caseConcentrate on the meaning of the test rather than detailing how it may beachievedTP is a textual description of a single requirement or a set of relatedrequirements which should be testedTest Purposes are grouped into a logical Test Suite Structure according to
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-24 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-24 Dr. Junzhao Sun
Test Purposes are grouped into a logical Test Suite Structure according tosuitable criteria, e.g., basic interconnection, error handling, functionality etc.
Will be introduced later in ATS section
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
ATS & ETS
ATS = Abstract Test SuiteThe entire collection of Test Cases (ABSTRACT)E h T t C ifi th d t il d di f th T t P ittEach Test Case specifies the detailed coding of the Test Purposes, writtenin the standardized test specification language TTCNCan be generated manually from protocol and service standardCan be generated automatically from formal specification of protocol
CATG = Computer-Aided Test Case GenerationSome tools are commercially available to generate test suite from e.g. SDL
Will be studied later in more detail
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-25 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-25 Dr. Junzhao Sun
ETS = Executable Test SuiteTest cases for truly test executionCan be quickly and easily implemented from the ATS using the TTCNcompilers available on most modern test tool platforms
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Test Suites & Specification
Tests apply to a single system under test (SUT), and one protocol standardsExhaustive test of standards specificationsExhaustive test of standards specifications
Must cover all features – mandatory & optionalMust cover valid, invalid, & inopportune protocol exchanges
Use of standardized test specification mayImprove transparency and increase of the objectiveness for the test processProvide comparability of test resultsAs the criteria for accreditation bodies to assess test laboratories
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-26 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-26 Dr. Junzhao Sun
As the criteria for accreditation bodies to assess test laboratoriesCommon test specifications are the basis for harmonized test servicesProvide a common starting point to suppliers, users, test laboratories and certification bodies
Test specification language – TTCN
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Test Specification Development in ETSI
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-27 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-27 Dr. Junzhao Sun
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Typical Test Suites
GSM, GPRS, 3G (UMTS in 3GPP)Broadband Radio Access Networks (HiperLAN)DECT(Digital Enhanced Cordless Telecommunications)BluetoothLocal Loop, Digital Subscriber Line (DSL)IN, INAP (Intelligent Network Application Protocol)ISDN, B-ISDNSMS i PSTN/ISDN
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-28 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-28 Dr. Junzhao Sun
SMS in PSTN/ISDNSIP (Session Initiation Protocol), IPv6IP CablecomXML (Extensible Markup Language)
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Outline
General conceptsTesting documentsAbstract test methodsAbstract test suitesTest realizationConformance assessment process
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-29 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-29 Dr. Junzhao Sun
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
ATM
ATM = Abstract Test MethodsNOT Automatic Teller MachineNOT Asynchronous Transfer ModeNOT Asynchronous Transfer Mode
A set of ATMs are defined by OSI testing methodology, describing Abstract test configurationPossible types of observation and controlProviding for test coordination
ATMs are determined byConfiguration of IUT with respect to SUT
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-30 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-30 Dr. Junzhao Sun
Entire testing configuration, I.e. test system + SUT + service providerWhere tester can control and observe the IUTDifference degrees of precision of the test coordination procedures
Common test architecture and 4 test methods
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
OSI System & IUT
Complete system Partial systemApplication relay
systemNetwork relay
system
Transport
Application
PresentationSession
Data link
Network
Transport
Application
PresentationSession
Data link
Network
Transport
Application
PresentationSession
Data link
Network
Transport
Application
PresentationSession
Data link
Network
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-31 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-31 Dr. Junzhao Sun
IUT implement one or a combination of OSI protocols from one layer or several adjacent layers
Data link
Physical
Data link
Physical
Data link
Physical
Data link
Physical
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
OSI Terminology
Layer S Layer SSP
Session layer
Layer T Layer T
Service provided
Service used
PDU
SP
SPSAP
Transport layer
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-32 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-32 Dr. Junzhao Sun
Layer N Layer N
SP : Service PrimitiveSAP : Service Access PointPDU : Protocol Data Unit
Network layer
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Conformance Testing Terminology
Upper Tester
S
IUT
ASP
ASP
ASP
ASP
PCO
PCO
ASP: Abstract Service Primitive
PCO: Point of Control and Observation
IUT: Implementation Under Test
PDU: Protocol Data Unit
PDU
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-33 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-33 Dr. Junzhao Sun
Lower Tester
The PCO has two FIFO queues:• Send (from tester to IUT)• Receive (by tester from IUT )
A PCO maps to a SAP (Service Access Point) in the OSI reference model
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Conceptual Test Architecture (1/3)
System Under Test (SUT)Test System (TS)Test operator interface Product operator interface
IUT
ASPs
PCOLower Tester
Test Co-ordinationProcedures (TCP)
PDU
Lower Tester Control Function
Lower TesterLower Tester
TCP
Upper TesterUpper TesterUpper Tester
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-34 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-34 Dr. Junzhao Sun
ASPs
PCO
Lower level service provider
There can be several LTs and UTs being simultaneously used
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Conceptual Test Architecture (2/3)
Testing contextsSingle-party testing is for testing an IUT which communicates with exactly one real open system represented by a single lower testerone real open system, represented by a single lower testerMulti-party testing is for testing an IUT communicating with multiple real open systems, represented by more than one lower tester
Configuration for IUT components can be homogeneous or heterogeneous
Lower tester (LT) controls and observes the IUT’s lower service boundary, indirectly, via the underlying service provider
In single-party testing, behaves as the peer entity to IUT
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-35 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-35 Dr. Junzhao Sun
g p y g, p yIn multi-party testing, act as peer entities working in parallel
Lower tester control function (LTCF) coordinating all LTsAssign the test case verdictsMandatory in multi-party context, inapplicable in single-party context
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Conceptual Test Architecture (3/3)
Upper tester (UT) controls and observes IUT’s upper service boundary, by operator access, API, or hardware interface
I i l t t t UT b h f IUTIn single-party context, UT behaves as a user of IUTIn multi-party context, UTs working in parallel act as users of IUT
Test coordination procedures (TCPs) are used to ensure cooperation between the UTs and LTs
How tester shall respond Passing (preliminary) results S h i ti
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-36 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-36 Dr. Junzhao Sun
SynchronisationTCP is NOT Transport Control Protocol, as in TCP/IP
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
ATM Classification
ATMs for multi-party testingSeveral parallel upper and lower testersI l it ti t t t l f ti (UTCF) i d dIn complex situation a upper tester control function (UTCF) is neededSpecial cases include only one upper tester, or even no upper tester at all
ATMs for single-party testingLocal Test Method (L)
Upper Tester and Lower Tester in Test SystemDistributed Test Method (D)
Upper Tester in SUT Lower Tester in Test System
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-37 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-37 Dr. Junzhao Sun
Upper Tester in SUT, Lower Tester in Test SystemCo-ordinated Test Method (C)
As above, uses Test Management ProtocolRemote Test Method (R)
Lower Tester in Test System, no Upper Tester
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Test Case
Lower Tester IUT
ConnectRequest “CR”
Upper Tester
ConnectIndication “CONind”ConnectConfirm “CC” ConnectResponse “CONresp”
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-38 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-38 Dr. Junzhao Sun
Recall service primitivesRequestIndicationResponseConfirm
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Local Test Method
UT
LT
N-layer entityunder test
TCPcoordination
ASPs
ASPs
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-39 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-39 Dr. Junzhao Sun
• Upper Tester is located in Test System• Requires an upper interface on IUT• IUT is “built in” the tester• No ATSs for this method• Good for the testing of a hardware component• Example: Ethernet driver
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Local Test Method
System Under Test (SUT) Test System (TS)
Upper Tester
IUT
ASPs
PCO
System Under Test (SUT) Test System (TS)
Lower Tester
ASPs
TCPs
2) CONind
3) CONresp
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-40 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-40 Dr. Junzhao Sun
ASPs
PCO1) CR4) CC
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Distributed Test Method
UTTCP
Service provider
IUT
UT
LT ASPs
ASPs
PDUs
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-41 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-41 Dr. Junzhao Sun
Service provider
• UT in SUT, LT remote• Requires synchronisation• Suitable for upper layer protocols / protocols offering API• Example: socket communication
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Distributed Test Method
System Under Test (SUT) Test System (TS)
Upper Tester
IUT
PCO
System Under Test (SUT) Test System (TS)
Lower Tester
ASPs
Test Co-ordinationProcedures (TCP)
2) CONind 3) CONresp
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-42 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-42 Dr. Junzhao Sun
PCO1) CR4) CC
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Co-ordinated Test Method
UTTM-PDUs
Service provider
IUT
UTLT
ASPs
PDUs
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-43 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-43 Dr. Junzhao Sun
• UT in SUT but no access, LT remote• No assumption of upper interface to the IUT• Use only one PCO below the LT• Uses Test Management Protocol (TMP) embedded in ASPs• Suitable for mid-layer protocols
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Co-ordinated Test Method
Upper Tester
IUT
System Under Test (SUT) Test System (TS)
Lower Tester
ASPs
Test ManagementProtocol
2) CONind 3) CONresp
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-44 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-44 Dr. Junzhao Sun
PCO1) CR4) CC
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Remote Test Method
TCP
Service provider
IUT
UTLT
ASPs
PDUs
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-45 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-45 Dr. Junzhao Sun
Service provider
• No Upper Tester• Upper Tester can be native application or a user accessible interface• Manual co-ordination• Limited, but easy to use
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Remote Test Method
System Under Test (SUT) Test System (TS)
IUT Lower Tester
ASPs
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-46 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-46 Dr. Junzhao Sun
PCO1) CR2) CC
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
ATMs Put Together
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-47 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-47 Dr. Junzhao Sun
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Testing Configuration Examples – MHS
MHS = Message Handling Systems Multi-user in single-party contextRTS R li bl T f S t
UT-2
RTS = Reliable Transfer System
UT-1LT MHS-ASPs
TCP
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-48 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-48 Dr. Junzhao Sun
RT service provider
IUTRTS-ASPs
MHS-PDUs
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Testing Configuration Examples – Multiplexing
Multiplexing happens in transport or network layer protocols where more than one connection is multiplexed over a single lower connection e g transport connections over network one
UT-2
lower connection, e.g. transport connections over network one
UT-1LT Transport-
ASPsNW
TCP
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-49 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-49 Dr. Junzhao Sun
Service provider
IUTNetwork-ASPs
NW-PDUs
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Testing Configuration Examples – Relay
Relay systems’ testing should be in multi-party context, without using UTs
LT –1PDUs
TCP
LT –2PDUs
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-50 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-50 Dr. Junzhao Sun
Service provider
IUTASPs
Service provider
ASPs
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
A Example Test System Architecture
Protocol Tester SUT
IUT
Interface
TCEditor
PICS,PIXITEditor
Result Analysis&report
TCTranslator
TestSelect
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-51 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-51 Dr. Junzhao Sun
Service Provider
TCLibrary
TestExecute
Transmit Receive
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Outline
General conceptsTesting documentsAbstract test methodsAbstract test suitesTest realizationConformance assessment process
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-52 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-52 Dr. Junzhao Sun
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
ATS Development Process
ATS = Abstract Test Suites1. Determine the testable conformance requirements from the relevant specifications ICS and related documentsspecifications, ICS and related documents2. Determine the test groups & objectives that cover these requirements3. Develop test purposes covering requirements and test group objectives4. Choose testing context (single-/multi-party) and an ATM (configuration)
A test suite is always dedicated to a single test method5. Specify ATS, including test steps, using standardized notation like TTCN6. Specify the selection and parameterisation criteria with respect to the ICS
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-53 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-53 Dr. Junzhao Sun
p y p pand IXIT proformas
ATS should be used to derive ETS by compilation and adaptation to the means of testing (MOT)
MOT is the combination of test devices, equipments and procedures
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Recall the Test Generation Process
TPsSt nd rd ATS
Standardized (Protocol) Implementation Conformance Statement (tables)
Standardized Test Purposes (in English)
TPsStandard
Standardized Abstract Test Suite (in TTCN)
ATS
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-54 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-54 Dr. Junzhao Sun
ICSStandardized (Protocol) Implementation eXtra Information for Testing (tables)
IXIT
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
ATS Structure (1/2)
T t Test group
Test suite
Test group
Test caseTest caseTest case Test caseTest caseTest caseTest caseTest caseTest caseTest caseTest case
Test group
Test groupTest group
Test caseTest caseTest caseTest case
T t t
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-55 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-55 Dr. Junzhao Sun
Test step
Test caseTest caseTest event Test caseTest caseTest caseTest eventTest caseTest caseTest caseTest event
Test step
Test stepTest step
Test caseTest caseTest caseTest event
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
ATS Structure (2/2)
A test suite consists of test cases with no execution orderTest cases are logically grouped into test group according to test purposes
E h t t h i di id l t t t d t t t i b h iEach test case has an individual test purpose to demonstrate a certain behaviorA test group objective may be provided for each test groupThe nested hierarchy should be created in a top-down fashion
Ensure systematic derivation of test purposes Achieve good coverage of all protocol aspects and conformance requirements
A test case may be modularized into nesting test stepsModularization facilitates test case development and maintenance
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-56 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-56 Dr. Junzhao Sun
Modularization facilitates test case development and maintenanceGroup commonly used test steps together into test step libraries
Economical that they need not be specified repeatedly
Test event is the elementary unit within a test caseUsually the exchange of a PDU or an ASP at a PCO
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
TSS = Test Suite Structure
“which tests are performed on which aspects of the reference specification”Adopt a top down approach deriving TSS from base specificationAdopt a top down approach deriving TSS from base specificationwhen no technical constraint forbids it, structure TSS as a tree1. The name of base specification should be placed at the top of the TSS2. Test groups related to the "major functions", or "roles" of the base
specification3. Pre-defined groups of tests according to the "nature" of the tests:
Basic interconnection tests; capability tests; V lid b h i t t i lid b h i t t i t b h i t t
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-57 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-57 Dr. Junzhao Sun
Valid behaviour tests; invalid behaviour tests; inopportune behaviour tests;4. Pre-defined groups of tests according to the "functional aspect" tested:
State event transitions; parameter variations; parameter combinations; timers5. Groupings relevant to the base specification
E.g. name of functional units, then of PDUs
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
TSS FTAM Example
FTAM = File Transfer Access and Management
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-58 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-58 Dr. Junzhao Sun
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Test Purposes (1/2)
Each test case with one test purpose for meaning descriptionTest purposes focus on one or more related conformance requirements, derived from OSI specificationTest purposes and their hierarchical grouping are specified in a TSS & TP (Test suite structure and test purposes) standard
Structure chosen depends largely on personal experience and style
Test purposes for a test suite have to cover aspects of protocolCapabilities, in particular mandatory and optional ones
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-59 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-59 Dr. Junzhao Sun
Behavior tests, with the tester displaying valid/invalid behaviorPDUs sent to/received from IUT, and relationship between the PDUsProtocol phases and protocol machine statesTiming, encoding, individual parameters and parameter combinations
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Test Purposes (2/2)
The expression of a TP shall not be limited to checking a state or a state transition of a protocol state machine
E l TP "Ch k th t th IUT d COMMON CONTROL ACK ftExample TP: "Check that the IUT sends a COMMON CONTROL ACK after receipt of a COMMON CONTROL message containing ... , as a result of state transition S1 → S2.".
A clear explanation of the relation between TPs and the conformance requirements in the reference specification shall be providedThe TPs shall be designated by the same numbering and
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-60 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-60 Dr. Junzhao Sun
The TPs shall be designated by the same numbering and referencing scheme as the TSS
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Combined TPs (CTPs)
The final cost of a test campaign directly depends on the size of the ATS
The objective of CTP is to reduce the number of test cases, but to keep a good coverage
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-61 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-61 Dr. Junzhao Sun
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
TP Combination Principle
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-62 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-62 Dr. Junzhao Sun
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
TSS for CTP
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-63 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-63 Dr. Junzhao Sun
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Test Purpose Component
Initial condition should define precisely the expected state of the IUT at the beginning of the test
Example: Ensure that the IUT is in state connection established (reception of a connect confirm on the LT on response to a connect request).
Check(s) to be performed
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-64 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-64 Dr. Junzhao Sun
Example: Ensure that the IUT, on reception of a valid SETUP message with the Sending complete information element sends any of a CALL PROCEEDING, ALERTING or CONNECT message.
Verdict criteria should explicitly describe the observable events which will be used to assess the result of the test.
Example: Check that the IUT in state S1, on receipt of a valid connect request, enters state S2 (sends a valid connect confirm).
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Testing States
Testing state is a combination of the states of SUT (IUT), test system, and underlying provider which can occur during testing
St bl t t b i t i d b t t t ith t t t ti itiStable state can be maintained between test case without tester activitiesIdle state is that no connection is established, independent of previous tests
Test state is the one that test case is truly performed to check test purposeInitial state the start point of the execution of a test case
End state is the stop state of a testOne test case may stop at several end states
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-65 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-65 Dr. Junzhao Sun
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Test Case Scheme
A test case is conceptually composed of 3 partsTest body, the combination of test events for achieving test purpose
St t i i iti l t ti t tStarts in a initial testing stateTest preamble, takes the test from stable states to initial stateTest postamble, takes the test from end states to one stable state
If end state is not unique, it has to be checked by a verification step
Postamble
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-66 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-66 Dr. Junzhao Sun
The outcome for each test case should be assigned a verdictCalculated from verdicts assigned to test events in the outcome
End State (Verification)
End State (Test Body)
Stable State
Test State
Preamble Test Body Verification
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Outline
General conceptsTesting documentsAbstract test methodsAbstract test suitesTest realizationConformance assessment process
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-67 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-67 Dr. Junzhao Sun
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Test Realization
Test realization is the process of producing a means of testing(MOT) for conformance to protocol standards, by reference to an ATS standardATS standard.
MOT is a real test system used to execute a test suite on IUTsCombination of equipment and procedures that derive, select, parameterize and execute the test cases of a given ATS, together with produce conformance log
Does not mean the actual execution of test cases
ETS derivation is to convert the ATS into final ETSSelection of applicable test cases (abstract or executable)
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-68 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-68 Dr. Junzhao Sun
Selection of applicable test cases (abstract or executable)Performed mainly on the basis of ICS (Imple. Conformance Statement)
Parameterization of selected test cases (abstract or executable)Performed mainly on the basis of IXIT (Imple. eXtra Info for Testing)
Derivation process, see next
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Derivation Process
One-to-one correspondence between selected abstract &
ATS Derivation ETS
executable test casesCorrespondence in all respects e.g. behavior, test purpose, verdicts, relation to ICS/IXIT entries, selectability, independency, etc.May include some MOT
SATS
Selection
PATS
Parameterization
Selection
Parameterization
PETS
SETSDerivation
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-69 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-69 Dr. Junzhao Sun
yspecific selectionIndependent from and compatible with the selection & parameterization processes Derivation
& execution
PATS
Execution
PETSDerivation
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
MOT Functions
Realization of configured ATM (abstract test method)Realize the LTs (lower testers) and LTCF (LT control function)
LT’ d /d d li ti i l b t t ffi i tLT’s encoder/decoder realization: simple but not efficientLT’s reference protocol implementation: efficient but difficult
Specify or realize UTs (upper testers)UT can be dedicated software, human operator, existing protocol layers, etc.
Specify or realize TCP (test coordination procedures)
ATS to ETSContain derived executable test cases
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-70 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-70 Dr. Junzhao Sun
Contain derived executable test casesProvide means for selection and parameterizationProvide means for executing PETS
Conformance log generationMust be human readable and permit verification of test verdicts assigned
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Outline
General conceptsTesting documentsAbstract test methodsAbstract test suitesTest realizationConformance assessment process
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-71 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-71 Dr. Junzhao Sun
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Conformance Assessment Process (1/2)
Conformance assessment process is the entire activities that are necessary to validate and document the protocol conformance
T t i i th t l ti f th t t itTest campaign is the actual execution of the test suite
3 phases for conformance assessment process1. Preparation for testing
Production of SCS, ICS & IXITChoice of ATM & ATSPreparation of SUT & MOT
2 Test operations
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-72 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-72 Dr. Junzhao Sun
2. Test operationsStatic conformance review of ICSTest realization (selection & parameterization)Test campaigns
3. Test report productionProvides clients with documentation
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Conformance Assessment Process (2/2)
OSI specifications,proformas,
ifi i
Preparation for testing•administrative activities
l i f dtest specifications •completion of documents•choice of test method and ATS•choice of test specifications•SUT and MOT preparation
SCS, IXIT, ICS, PTS, ATS
static conformance
Test Operation•static conformance review•test selection & parameterizationt t i
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-73 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-73 Dr. Junzhao Sun
static conformance review results,
test results, test log
PCTRSCTR
•test campaigns
Test Report Production
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Test Laboratories and Test Clients
Test Laboratory is responsible for conducting the conformance assessment of IUTs and can be classified according to their role
Fi t t i t d b th li f IUT I i l t dFirst-party is operated by the supplier of IUT, I.e. implementer, vendorSecond-party is affiliated to the user of IUT who is willing to test itThird-party is independent from either of the two whose business is the testing of products
Clients will frequently be implementers or vendors of protocol implementation who want to have their own products tested and
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-74 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-74 Dr. Junzhao Sun
implementation who want to have their own products tested and certified, but can also be important user of IUT
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Preparations for Testing (1/2)
Production of SCS, ICSBy client filling proformas
Production of IXITBy lab & client filling proformas
Choice of ATM & ATSAgreement between client & lab
Preparation of SUTBy client
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-75 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-75 Dr. Junzhao Sun
Preparation of MOTBy lab
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Preparations for Testing (2/2)
SUT decs
SUT config.
ICSIXIT MOT
config.
Extra decs
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-76 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-76 Dr. Junzhao Sun
ATSICS pro. IXIT
pro.
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Testing Operations (1/3)
Static conformance reviewChecks ICS for consistency with static conformance requirements
Test selectionSelect a suitable subset of test cases from ATS resulting SATS
Test parameterizationParameters are assigned values as specified in ICS & IXITs resulting PATSTransform PATS into PETS (parameterized executable test suite)
Verification of TCPs
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-77 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-77 Dr. Junzhao Sun
To verity MOT & SUT are able to use TCP (Test Coordination Procedure)
Test campaign, see next slideExecution of PETS and production of conformance log
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Testing Operations (2/3)
Test campaignAssignment of verdicts = [pass, fail, inconclusive]
T t l b h t i tl t t di t t d t tTest lab has to assign exactly one test verdict per executed test caseAbnormal test termination or error leads to “not run” in report“Inconclusive” verdict have to be repeated at least once
Conformance logOutcome of all executed test cases to document for client
Negotiated exitTime when client and test lab jointly decide to stop the test campaign
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-78 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-78 Dr. Junzhao Sun
No formal test report will be produced
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
ETS
Testing Operations (3/3)
Selection
Derivation
Parameterization
SUT config.
ICSIXIT MOT
config.
SUT decs
Extra decs
SUT config.SETS
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-79 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-79 Dr. Junzhao Sun
ATSICS pro. IXIT
pro. PETS Execution
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Test Report Production (1/2)
ETS Selection
SUT config.
ICSIXIT MOT
config.Derivation
SUT decs
Extra decs
SUT config.SETS
Parameterization
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-80 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-80 Dr. Junzhao Sun
ATSICS pro. IXIT
pro. PETS ExecutionComp. Report
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Test Report Production (2/2)
SCRT = System Conformance Test ReportGives an overall summary of the conformance status to the SUTC t i f hi d t t lt & d i i t ti i f tiContains a summary of achieved test results & administrative information
PCTR = Protocol Conformance Test ReportOne for each protocol under testAdministrative and technical informationA summary of all achieved test resultsIndividual detailed presentation of each test case
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-81 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-81 Dr. Junzhao Sun
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
SCTR Proforma
1. Identification summaryAdministrative information for identification of SCTRI f ti b t t t l b t t t li t SUT fil tInformation about test laboratory, test client, SUT, profile, etc.
2. System report summaryOne section for each tested protocol is requiredDocument reference
IUT definition, protocol specification, ICSs, IXITs, PCTR, ATS, ATM, etc.Conformance status
Whether static and/or dynamic conformance error
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-82 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-82 Dr. Junzhao Sun
Whether static and/or dynamic conformance errorTest cases run
Number of passed, failed, and inconclusive test cases
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
PCTR Proforma
1. Identification summaryInformation about PCTR, IUT, and testing environment
2 Protocol conformance summary2. Protocol conformance summaryIUT conformance status
“This IUT has/has not been shown by conformance assessment to be non-conforming to the referenced base specification(s)”
Static conformance summary“The ICSs for this IUT is/is not consistent with the static conformance requirements in the referenced base specification”
Dynamic conformance summary
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-83 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-83 Dr. Junzhao Sun
y yThe test campaign did/did not reveal errors in the IUT
3. Protocol conformance reportStatic conformance review reportTest campaign report, for each test case in ATS
ATS reference, selected or not, run or not, verdict, and observations
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Test Conducting
Formal tests Done by test laboratories that will issue official test certificate
Informal testsIn-house or private lab product testTests at trade shows, bake-off meetsRemote test – access to tester through Internet (e.g. on Web server)
Quality of TestsConsistency and repeatability
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-84 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-84 Dr. Junzhao Sun
Standard ATSs & approved ETSsStandard of test laboratory operations
Test Report for procurement and regulatory bodiesNational and International recognition of test results
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Detailed Testing Process
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-85 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-85 Dr. Junzhao Sun
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Whole Information Flow
ETS Selection
SUT config.
ICSIXIT
SETSMOT
config.Derivation
SUT decs
Extra decs Parameterization
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-86 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-86 Dr. Junzhao Sun
ATSICS pro. IXIT
pro. PETSComp. ReportExecution
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
Summary
Conformance testingBlack-box approach of testing protocol implementations against protocol specificationspecification
Three phases for conformance assessment process1. Preparation for testing
Test generation: from ICS+IXIT & TPs, get ATS• Document proforma
Configuration of ATM architecture• 4 methods
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-87 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-87 Dr. Junzhao Sun
2. Test operationsTest realization: from ATS, through selection & parameterization, get PETSTest execution
3. Test report productionSCTR & PCTRs
521265A Telecommunication Software Ch7 – Conformance Testing Methodology521265A Telecommunication Software Ch7 – Conformance Testing Methodology
References of This Chapter
Bernd Baumgarten and Alfred Giessler, OSI Conformance Testing Methodology and TTCN, Elsevier Science, 1994
Dept. Electrical & Information Engineering, Computer Engineering Lab 7-88 Dr. Junzhao SunDept. Electrical & Information Engineering, Computer Engineering Lab 7-88 Dr. Junzhao Sun