Upload
elma
View
72
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Protocol Testing. Lesson 5. Outline for today. Guidelines for testing protocols TTCN – A Standard Language For Testing Protocols Motivation The Language The usage. Guidelines for protocol testing. Guidelines. Unit Testing tests if individual units of source code are fit for use. - PowerPoint PPT Presentation
Citation preview
IHA præsentation 1
Protocol Testing
Lesson 5
IHA præsentation 2
Outline for today
• Guidelines for testing protocols
• TTCN – A Standard Language For Testing Protocols• Motivation
• The Language
• The usage
IHA præsentation 3
Guidelines for protocol testing
IHA præsentation 4
Guidelines
• Unit Testing• tests if individual units of source code are fit for use.• A unit is the smallest testable part of an application
• Integration Testing• individual software modules are combined and tested
as a group • Occurs after unit testing
• Coverage Testing• describes the degree to which the source code has
been tested
IHA præsentation 5
Guidelines
• Smoke Testing• Collection of written tests that are performed on a
system prior to being accepted for further testing
• Regression Testing• seeks to uncover software regressions• previously working software functionality stops working
as intended
• Conformance Testing• to determine whether a system meets some specified
standard
IHA præsentation 6
Conformance Testing Framework
• Conformance testing is the process of testing the extent to which implementations adhere to the requirements stated in relevant standard or specification
• Conformance Testing is functional black-box testing
• Functional refers to the correct functional behavior of an Implementation Under Test (IUT)
• Black-box means that the internal structure of the IUT remains hidden
IHA præsentation 7
Conformance Testing Framework
IUT – Implementation Under Test
UT – Upper Tester function
LT – Lower Tester function
PCO – Point of Control & Observation
TCP – Test Coordination Procedures
ASP – Abstract Service Primitive
IHA præsentation 8
Conformance Testing Framework
UT plays the role of a user that makes use of the service provided by the IUT
LT plays the role of the peer entity of the IUT, i.e. the LT and the IUT communicate in order to provide the service to the IUT
IHA præsentation 9
Conformance Testing Framework
PCO is a standardized interface.
Typically the lower interface of an IUT is accessible only from the remote
Communication is always meant to be asynchronous => PCO is modeled as two FIFO queues
IHA præsentation 10
Conformance Testing Framework
TCP can be used to coordinate the actions of LT and UT.
LT and UT can be on separate testers.
IUT and LT communicates by means of ASP.
PDUs are encoded in ASPs of the underlying service.
IHA præsentation 11
Conformance Testing Framework
• Test case development starts with:
• Identification of a test purpose
• Description of a single requirement or set of related requirements
• A test case is an implementation of a test purpose
IHA præsentation 12
Conformance Testing Framework
Test Case SchemeA test case starts and ends in stable testing states.
Preamble to take the IUT from stable testing state to test state.
Test body is performed in order to check the test purpose.
Verification to check test body
Postamble to drive the IUT into a stable testing state again
IHA præsentation 13
Protocol Testing
• Until now we have:• Developed our protocol layer including:
– Timers– State - event model– PDUs to be exchanged
• Interfaces to upper and lower layers
HOW DO WE TEST OUR PROTOCOL LAYER?
IHA præsentation 14
Protocol Testing
• Protocol Testing is mainly about:• Testing the state – event model.
PDU (message)Our protocol layerTest System
??
Observe response
IHA præsentation 15
Protocol Testing
• Testing the state – event model
• Difficult part is when we send an unexpected message into our layer
Unexpected PDU (message)
Our protocol layerTest System
??
•Ignore – take no action
•Reject – return an explicit denial to the sender
•Allow – take action to process the event
Standard testing languages and tools available
IHA præsentation 16
Test & Testing Control Notation Edition 3(TTCN-3)
IHA præsentation 17
• Introduction• What is TTCN? • Why use TTCN?
• The TTCN-3 Language• Background• Example
• TTCN-3 Usage• Where is it used?• Future Outlook
IHA præsentation 18
What is TTCN?
• Internationally standardized testing language for formally defining test scenarios and their implementation.
• Designed purely for testing
• Tool independent
• Flexible and powerful language• Combined use with other languages like ASN.1• Well-defined syntax, interchange format• Different presentation formats (e.g. tabular and graphical)• Support text-based protocols• Support for IP protocols
IHA præsentation 19
What is TTCN?
• In TTCN we can….
• Define Test Suites
• Define Test Cases
• Define Test steps
• Declare variables (test suite & test case variables)
• Declare timers
• Create PDUs
• Etc..
IHA præsentation 20
What is TTCN?
• TTCN Edition 3 is documented in a serie of ETSI Specification
• ES 201 873-1 (TTCN-3 Core Language)
• ES 201 873-2 (Tabular presentation format)
• ES 201 873-3 (Graphical presentation format)
• etc.
• etc.
IHA præsentation 21
Why Use TTCN?
• Quality• a well-tested product increases confidence in the product’s quality
• Time/Cost Improvement• TTCN is a language designed for protocol conformance testing• TTCN is standardized• TTCN is fairly simple
• Formal• strict rules, complete BNF and operational semantics
• Abstract• TTCN test suites are independent of an implementation
IHA præsentation 22
Application Areas
• New application areas• Software Testing
• Text-based protocols…
• Additional communication paradigm• Message-based communication
• Procedure-based communication
• Different kinds of testing
• Functional testing
• Conformance testing (behavioral testing)
• Scalability testing…
IHA præsentation 23
SUT
TTCN Basics
PASS, FAIL orINCONCLUSIVE
TTCNTest Suite
Result
IUTImplementation
Under TestObserve Responses
Send Stimuli
• Behavioral testing
• send stimulus, observe response
• was response received within specified time?
Test System
IHA præsentation 24
TTCN – Example
testcase MyTestCase() runs on DNSTester {
P.send(query);
P.receive(answer);
setverdict(pass);
stop;
}
Tester
MTC
Verdict = pass
P P SUT
query = (www.iha.dk, A)
answer = (www.iha.dk, 193.65.199.110,A)
Minimal Test Configuration
IHA præsentation 25
TTCN-3 test systems in a nutshell
• TTCN-3 specifies a test but a test system is needed for test execution
• TRI and TCI standards define test system architecture
• TTCN- 3 tools are required to support internal interfaces
• Allows reuse of test platforms with different tools but also for different SUTs
• A test system requires• A TTCN-3 tool = TTCN-3
compiler and execution environment
• A test platform for a specific device under test
Note: Tools come with default Test Control & Logging
Test System ExecutorTest System Executor
SUT Adapter Platform Adapter
Test Control
System Under Test (SUT)
[compiled]TTCN-3 Executable
CodecsLogging
TTCN-3 Test System
TCI
TRI
( s + s)
( )
TCI = TTCN-3 Control InterfaceTRI = TTCN-3 Runtime Interface
IHA præsentation 26
An example adaptation: A IPv6 test system
Test System ExecutorTest System Executor
Ethernet Adapter Real-time Adapter
ParameterFile
Open Source FreeBSD Router
[compiled]
IPv6 ATS TTCN-3 Executable
IPv6 Codecs
ToolLogging
TTCN-3 Test System
TCI
TRI
IHA præsentation 27
User’s view of TTCN-3
TTCN-3 CoreLanguage
ASN.1 Types & Values
IIDL Types
XML Types
C/C++ Types
Other Types & Valuesn
Tabularformat
Graphicalformat
Presentationformatn
TTCN-3 User
Text format
IHA præsentation 28
The Core Language and Other Presentation Formats
TTCN-3 Core Language
Text format
PresentationFormat3
PresentationFormatn
Graphical Format
Tabular Format
• Core format is text based (most popular)
• TTCN-3 can be edited or viewed in other formats
• Tabular format (for TTCN-2 people)
• Graphical format (good for visual overview)
• Other standardized formats in the future?
• Proprietary formats possible
IHA præsentation 29
Example Core (Text) Format
testcase TC_resolveEtsiWww() runs on DnsClient {
timer t_ack;serverPort.send(m_dnsQuestion("www.etsi.org"));t_ack.start(1.0);alt {
[] serverPort.receive(mw_dnsAnswer("172.26.1.17")) {setverdict (pass);
}
[] serverPort.receive { // any other message setverdict(fail); }[] t_ack.timeout { setverdict(inconc);
}
} t_ack.stop;}
IHA præsentation 30
testcase TC_resolveEtsiWww() runs on DnsClient
Example Graphical Format
DnsClientmtc
DnsPortserverPort
m_dnsQuestion("www.etsi.org")
timer t_ack
t_ack
altmw_dnsAnswer("172.26.1.17")
?
t_ack
inconc
fail
pass
t_ack
IHA præsentation 31
Example Tabular FormatTestcase
Name TC_resolveEtsiWww()
Group
Purpose
System Interface
MTC Type DnsClient
Comments
Local Def Name Type Initial value Comments
t_ack timer
Behavior
serverPort.send(m_dnsQuestion("www.etsi.org"));
t_ack.start(1.0);
alt {
[] serverPort.receive(mw_dnsAnswer("172.26.1.17")) { setverdict (pass); }
[] serverPort.receive // any other message { setverdict(fail); }
[] t_ack.timeout { setverdict(inconc); }
}
t_ack.stop;
Detailed Comments:
IHA præsentation 32
Use of TTCN-3 With Other Languages
• TTCN can be integrated with types systems of other languages
• Fully harmonized with ASN.1 (1997)
• Harmonized with other languages
• IDL, XML, C/C++
TTCN-3 Core Language
IDL Types & Values
Other types & Valuesn
XML Types & Values
ASN.1 Types & Values
IHA præsentation 33
ASN.1
• Abstract Syntax Notation One• language for describing structured information• standardized internationally (ISO/IEC 8824, ITU-T X.680)
• TTCN includes ASN.1• used for creating data descriptions and constraints• can be part of TTCN script or imported as a module
• SDL also includes ASN.1• ITU-T Z.105
• ASN.1 can be re-used across design and test!
ASN.1
SDLTTCN
IHA præsentation 34
TTCN – Example
testcase MyTestCase() runs on DNSTester {P.send(query);P.receive(answer); setverdict(pass);
stop;}
Tester
MTC
Verdict = ???
P P SUT
query = (www.iha.dk, A)
???
• P.receive(answer) blocks until it receives a message that matches answer
• Any other message does not unblock the tester, which then blocks forever
• If no message is received, the tester will also block forever
IHA præsentation 35
TTCN Test Case Example
testcase MyTestCase2() runs on DNSTester {
timer t := 5.0; P.send(query);
t.start;
alt { [] P.receive(answer); { setverdict(pass); } [] P.receive { // any message setverdict(fail); } [] t.timeout { setverdict(inconc); } } stop;}
testcase MyTestCase2() runs on DnsTester
Testermtc
SUTserverPort
query("www.iha.dk")
timer t_ack
t_ack
altanswer("172.26.1.17")
?
t_ack
inconc
fail
pass
t_ack
IHA præsentation 36
Test Suite
Building blocks of a TTCN-3 Test Suite
Test Data Types
Data types which specify• Structure of messages or calls and their information elements (fields, parameters)• Internal data structures (e.g., for computation)• Possibly encoding or display informationBuilt-in basic types integer, boolean, float, bitstring, hexstring, octetstring, charstring, universal charstring... and structured types record, record of, set, set of union, enumerated
... and special types such ascomponent, port, verdicttype, default, etc
IHA præsentation 37
Test Suite
Building blocks of a TTCN-3 Test Suite
Actual Test Data
Test Data Types
Actual test data (values) used during testing • Constants or Templates for specific message or call parameter values• Matching expressions for allowing multiple message or call parameter values
• value range, value list, wildcards, presence, length, size, permutation• regular expressions
• Using also template decomposition, parameterization and modification
IHA præsentation 38
Test Suite
Building blocks of a TTCN-3 Test Suite
Test Configuration
Actual Test Data
Test Data Types
Static aspects
• Test component and port types
Dynamic aspects
• Dynamic instantiation and management of test components
• Mappings of test components to abstract test system interfaces
• Connections between test component interfaces
• Management of test components
IHA præsentation 39
Test Suite
Building blocks of a TTCN-3 Test Suite
Test Behaviour
Test Configuration
Actual Test Data
Test Data Types
test cases
• specify sending/receiving messages, computation (e.g., checksums), and verdict assignment
• can be decomposed with functions and altsteps
• can (re)use default behaviour
• can use timers and timeouts
test execution control (optional)
• order, repetition, conditions, etc
IHA præsentation 40
TTCN-3 Module
module EtsiDnsTests{
// Test definition part
control{
// Test execution part// (optional)
}}
Module (…)
Module Control
Module Definitions
IHA præsentation 41
Definitions Part
module EtsiDnsTests{
// Message structure// Actual test data// Test configuration// Test Case definitions
}
IHA præsentation 42
module EtsiDnsTests{
group MessageStructure{
// Defintions of message types }group TestData{
// Templates for messages instances} group TestSystemConfiguration{
// Port and component types and mappings}group TestCases{
// Test case definitions}
}
Structured Definitions Part
IHA præsentation 43
Message Structure and Test Datatype record DnsMsg // simplified message structure!{ DnsMsgKind kind, charstring question, charstring answer optional}type enumerated DnsMsgKind {e_query, e_response}
template DnsMsg m_dnsQuestion( charstring p_question ){ kind := e_query, question := p_question, answer := omit // no answer}
template DnsMsg mw_dnsAnswer( charstring p_answer ){ kind := e_answer, question := ?, // any question ok answer := p_answer }
IHA præsentation 44
Test Configuration
type port DnsPort message { inout DnsMsg} // Note: port types may also allow multiple different // message types or restrict the direction
type component DnsClient { port DnsPort serverPort} // Note: component types can also define multiple port // instances of the same or different port type and // declare timers, constants or variables
IHA præsentation 45
Test Behaviourquery
responseserverPort
testcase TC_resolveEtsiWww() runs on DnsClient {
timer t_ack;serverPort.send(m_dnsQuestion("www.etsi.org"));t_ack.start(1.0);alt {
[] serverPort.receive(mw_dnsAnswer("172.26.1.17")) {setverdict(pass);
}[] serverPort.receive { // any other message setverdict(fail); }[] t_ack.timeout { setverdict(inconc); }
} t_ack.stop;}
mtc
IHA præsentation 46
The Control Part
Module (…)
Module Control
Module Definitions
module EtsiDnsTests{
// Test definition partmodulepar boolean mp_example;
testcase TC_resolveEtsiWww() runs on DnsClient {
// .. as in previous slide }
// Test execution partcontrol { if (mp_example) {
execute(TC_resolveEtsiWww()); }
}}
IHA præsentation 47
TTCN-3 Usage
IHA præsentation 48
TTCN in 3GPP
• Lessons from GSM• TTCN ATSs contribute to interoperability• powerful competitive tool for entry to the global GSM mobile market• TTCN ATSs widely used within members’ development processes
• 3GPP• TTCN in Protocol Conformance Specs
• 34.123-1 Prose description of tests• 34.123-2 Implementation Conformance Statement (ICS)• 34.123-3 TTCN Abstract Test Suite
• TTCN will be the normative specification
IHA præsentation 49
TTCN and Bluetooth
• Bluetooth SIG selected TTCN for...
• definition of Protocol Requirements tests
• interoperability
• definition of Profile Requirements tests
• application interoperability
• end-user expectations
RF
Baseband
AudioLink Manager
L2CAP
TCP/IP HID RFCOMM
Applications
Data
Con
trol