1

Master Thesis

In

Basic testing strategy for block and multi-blocks developed in SDL

By

David Johansson, LTH in Helsingborg Vasilis Fotopoulos, LTH in Helsingborg

2001

Abstract

2

A B S T R A C T Choosing the right testing strategy in order to validate and verify telecommunication software systems is a crucial and difficult task. The objective is to gain confidence that the software is reliable. Since testing process influence the software product’s cost and quality, it is necessary to choose most appropriate testing methods that are suitable for the right phase during the development process. Our paper has two goals. First, it identifies the current testing process within Ericsson Radio Systems’ development process for SDL block and give a simplified picture for readers to understand the different testing methods and tools that are used during the various phases. In addition, we mention the problems of using the current testing tools and we suggest proposals for improving for each one of them. Second, it suggests an optimal testing process for Validation and verification of an SDL single- and Multiblock system. This testing strategy should minimized the testing time and hence testing cost and provide the most optimal testing methods for each phase in SDL development process.

However, the main goal with this paper is to improve the quality of software testing process as well as the tools that are involved in it. Additionally we investigate which features are possible to test on SDL level.

Acknowledgements

3

A C K N O W L E D G E M E N T S We would like to take the opportunity to thank our tutors, Magnus Nilsson at the department of the telecommunication system at Lund University, and Tommy Nordgren and Jörgen Palm at Ericsson Radio Systems AB for their great help and support throughout this thesis. Last but not least, we would like to thank all people that helped us through this thesis at the Ericsson Radio Systems AB in Hässleholm for their guidance during the first steps of our work and the various questions throughout our work. Hässleholm August 2001 Vasilis Fotopoulos David Johansson

4

Overview

5

O V E R V I E W Below is a summary of what each chapter contains. We also give proposals on how to read our thesis depending on the reader’s interest and experience.

Chapter 1 Introduction Contains a short introduction to work environment and includes general description of tasks for the current master thesis.

Chapter 2 Problem Description Includes the purpose and problem description that this thesis is involved with and problem statement. Delimitations that the current thesis treats are also described in this chapter. Chapter 3 Related Works The purpose of this chapter is to give an overview of related theories about system development. It starts with a description of the general system development process model following by general testing methods. Chapter 4 Ericsson Development Methods The purpose of this chapter is to give an overview of system development methods applied in Ericsson today. It starts with a description of the AXE 10 Software Design System Model and the SDL design method used in software development in BSC node that is a part of GSM system. Chapter 5 Ericsson Test Methods This chapter describes the current testing processes in Ericsson applied on SDL block design. The testing activities for each development phase are described with emphasis to basic test. Chapter 6 Investigation of Testing Tools Here we investigate the testing tools available during SDL block design at Ericsson. Those tools are described and problems and improvements are identified. Chapter 7 Different solutions This chapter describes the different solutions for improving the current test method for function blocks developed in SDL, which is the main objective of this thesis. It is based on our study and the results that we obtained through investigating and evaluating different testing methods and tools.

6

Chapter 8 Optimal Testing Strategy This chapter is containing description of our Optimal Testing Strategy as an example to apply on SDL block design. Chapter 9 Summary and conclusions In this section, we give a short outline of the entire thesis and summarize our paper with conclusions. Appendices The appendices include some details about SDL language Appendix A SDL, MCS diagrams Appendix B Message Sequence Charts (MSC), PLEX- C language Appendix C, Validator tool monitor commands, that can be issued during validation Appendix D Validator. In Appendix E we include a description of the most common mediator commands and define the requirements needed. Description and implementation details of Start and Restart is described in Appendix F. The problem that we encountered during Multiblock simulation and solutions are represented in Appendix G Multiblock simulation. In the last Appendix H Global System for Mobile communication (GSM) we describe in short about GSM. Read instructions Depending on the reader’s interest and knowledge on the different subjects in our thesis, some parts of this paper can be more or less interesting. Possible readers could be System Designers, Project co-ordinators and Testers, Students, Graduate Students or various Academicians. SDL designer. If the interest is to learn a specific testing method read the appropriate chapter and eventually appendices of specific interest. We recommend the chapter 6 section 6.3 Validation and 6.5 Multiblock Simulation since these methods are not applied in the current testing process. Thirst for knowledge academic or university graduate. Read everything that you do not already know by choosing the chapter(s) of your interest. Limited-time reader. Read the abstract, the Summary and Conclusions, in chapter 9. These ought to give a good overview of the different subjects. Later the reader can choose the chapter(s) of interest.

Thesis Overview

7

T H E S I S O V E R V I E W

Activity: Mediator Duration: 3 weeks Input: Simulation and general knowledge Output: Work instructions on how to use Mediator and requirements on the tool

Activity: Simulation Duration: 3 weeks Input: General knowledge Output: Simulation process knowledge and system design

Activity: Literature study Duration: 2 weeks Input: Plex-C1, Plex-C2, Introduction to design using SDL, SDL10 Design Project and Telelogic online help manuals. Output: General knowledge in domain area.

Activity: Validation Duration: 4 weeks Input: General knowledge Output: Work instructions on how to use Validator and a new Validation process

Activity: Multiblock Simulation Duration: 4 weeks Input: General and simulation knowledge. Output: Work instructions on how to set up Multiblock system and conclusions

Optimal testing

strategy

Figure Activities workflow

8

9

C O N T E N T S

ABSTRACT.................................................................................................................................................... 2

ACKNOWLEDGEMENTS............................................................................................................................ 3

OVERVIEW................................................................................................................................................... 5

THESIS OVERVIEW .................................................................................................................................... 7

CONTENTS.................................................................................................................................................... 9

1 INTRODUCTION ............................................................................................................................... 11

2 PROBLEM DESCRIPTION............................................................................................................... 13

2.1 PROBLEM STATEMENT .................................................................................................................. 13 2.2 DELIMITATIONS ............................................................................................................................ 14

3 RELATED WORKS............................................................................................................................ 15

3.1 OVERVIEW OF SYSTEM DEVELOPMENT PROCESS ............................................................................. 15 3.1.1 Requirements specification techniques...................................................................................... 16 3.1.2 System Design and techniques.................................................................................................. 16

3.2 SOFTWARE TESTING ...................................................................................................................... 16 3.3 TESTING PROCESS ......................................................................................................................... 17

3.3.1 Planning the Testing Effort ...................................................................................................... 17 3.3.2 Staffing Phase.......................................................................................................................... 18 3.3.3 Designing the test cases ........................................................................................................... 18

4 ERICSSON DEVELOPMENT METHODS ....................................................................................... 21

4.1 THE AXE 10 SOFTWARE DESIGN SYSTEM MODEL ......................................................................... 21 4.1.1 System Study............................................................................................................................ 24 4.1.2 System Analysis........................................................................................................................ 24 4.1.3 Source System Design .............................................................................................................. 24 4.1.4 Subsystem Design .................................................................................................................... 24 4.1.5 Function Specification ............................................................................................................. 24 4.1.6 Function Design ...................................................................................................................... 25 4.1.7 Block Design............................................................................................................................ 25 4.1.8 Software Unit Design and Basic Test........................................................................................ 25 4.1.9 Basic Function Test.................................................................................................................. 25 4.1.10 Function Test Planning........................................................................................................ 25 4.1.11 Function Test Specification.................................................................................................. 26 4.1.12 Function Test Design........................................................................................................... 26 4.1.13 Function Test ...................................................................................................................... 26

4.2 DESIGNING WITH SDL................................................................................................................... 27 4.3 PROJECT PROCESS ......................................................................................................................... 31

4.3.1 PROPS .................................................................................................................................... 31

5 ERICSSON TEST METHODS ........................................................................................................... 32

6 INVESTIGATION OF TESTING TOOLS......................................................................................... 36

6.1 ERICSSON TAU .............................................................................................................................. 36 6.2 SIMULATION ................................................................................................................................. 38

6.2.1 Simulator tool .......................................................................................................................... 38 6.2.2 Singleblock simulation ............................................................................................................. 39 6.2.3 Problems with simulation......................................................................................................... 39 6.2.4 RestartSimulation .................................................................................................................... 40

6.3 VALIDATION ................................................................................................................................. 42 6.3.1 Validation of a system.............................................................................................................. 42

10

6.3.2 Exploration Algorithms............................................................................................................ 43 6.3.3 Validator tool .......................................................................................................................... 44 6.3.4 Application areas..................................................................................................................... 44 6.3.5 Verification of MSCs................................................................................................................ 45 6.3.6 Validator Rules........................................................................................................................ 46 6.3.7 Results..................................................................................................................................... 48 6.3.8 Conclusions ............................................................................................................................. 50 6.3.9 Problems ................................................................................................................................. 51 6.3.10 Proposals............................................................................................................................ 51

6.4 EMULATION METHOD .................................................................................................................... 52 6.5 MULTIBLOCK SIMULATION ............................................................................................................ 54

6.5.1 Different approaches for Multiblock simulation........................................................................ 54 6.5.2 Results..................................................................................................................................... 55 6.5.3 Conclusions ............................................................................................................................. 55 6.5.4 Proposals for improvements..................................................................................................... 56 6.5.5 Problems and Solutions............................................................................................................ 57

7 DIFFERENT SOLUTIONS................................................................................................................. 61

7.1 DIFFERENT TESTING STRATEGIES ................................................................................................... 61 7.1.1 Comparison of the different strategies ...................................................................................... 65

7.2 GENERAL PROCESSES .................................................................................................................... 66 7.2.1 Singleblock simulation process................................................................................................. 66 7.2.2 Multiblock simulation process.................................................................................................. 69 7.2.3 Validation process ................................................................................................................... 76

8 APZ EMULATOR AND MEDIATOR ............................................................................................... 81

8.1.1 Problems and solutions ............................................................................................................ 86 8.1.2 Conclusions ............................................................................................................................. 87 8.1.3 Proposals ................................................................................................................................ 87

9 OPTIMAL TESTING STRATEGY.................................................................................................... 89

9.1 OVERVIEW.................................................................................................................................... 89 9.2 OPTIMAL TESTING PROCESS .......................................................................................................... 91

9.2.1 Test Planning........................................................................................................................... 92 9.2.2 Checklist.................................................................................................................................. 93 9.2.3 Simulation Script Files............................................................................................................. 93 9.2.4 Validation................................................................................................................................ 94 9.2.5 Singleblock simulation ............................................................................................................. 95 9.2.6 Multiblock simulation .............................................................................................................. 96 9.2.7 Test Metrics............................................................................................................................. 97 9.2.8 Evaluation of the testing results................................................................................................ 98 9.2.9 Conclusions ............................................................................................................................. 98

10 SUMMARY AND CONCLUSIONS ................................................................................................. 101

10.1 CONCLUSIONS............................................................................................................................. 102

11 APPENDICES ................................................................................................................................... 103

A SDL ..................................................................................................................................................... 103 B MESSAGE SEQUENCE CHARTS (MSC) .................................................................................................... 105 C PLEX PROGRAMMING LANGUAGE FOR EXCHANGES .............................................................................. 109 D VALIDATOR .......................................................................................................................................... 111 D MEDIATOR............................................................................................................................................ 113 F SIMULATION OF START RESTART............................................................................................................ 117 G MULTIBLOCK SIMULATION .................................................................................................................... 121 H GLOBAL SYSTEM FOR MOBILE COMMUNICATION (GSM) ........................................................................ 123

REFERENCES ........................................................................................................................................... 126

INTERNET LINKS ....................................................................................................................................... 127

Introduction

11

1 I N T R O D U C T I O N Testing activities is the main subject in this paper. This is to validate and verify the software produced during different phases in development process. This activity is a very important and crucial in order to achieve high product quality and/or reduce development time and cost. Since SDL language is rather new to the current development process, not much work has been done to explore new tools and testing techniques that could be applied during testing process. This exploration is a part of the current thesis there we examine existing and new SDL tools and investigate the way to put them in place. This thesis firstly investigates the current development process to understand the relationships between the different parts of the processes. This has been done in the study phase, which includes examining the activities and tools that are used in the development process in order to produce an abstract model of the process. We have studied the theory for different development methods especially testing methods applied during the general software development process. Beside that we have also studied the PLEX-C and SDL languages, and the development process and tools related to those languages at Ericsson. Next step was to analyze the abstract development model to discover process problems and weaknesses. First we analyzed the use of the Emulator tool during PLEX-C development and especially during testing process. We proceeded by analyzing the Simulator tool that is used during the testing process of SDL block design. Then we continued to investigate other methods and tools that could be applied as complements to the current testing process. At that point we found out that Validator and Mediator tools could be applied and appropriate methods to apply them were developed. The results from process analysis above and investigation of new tools and methods are used to identify the problems and weaknesses of the current development process with focus on testing process activities. The task in this step was to discover improvements in order to loose the bottlenecks in testing process and find new procedures, methods and tools to address the problems. We introduced processes for those new methods found during analysis and a description in details was developed for each activity. One of those methods found was the Multiblock simulation that we applied by testing a Multiblock system containing three SDL blocks. We introduced the improvements by putting in place those new methods and tools and integrating them with existed process activities. The improvements focus mainly on finding an optimal process regarding testing process. Our evaluation of this process is included in as a strategy we call the Optimal Testing Strategy i.e. processes on how to apply these tools during testing. This Optimal Testing Strategy is also compared to the current testing process applied on SDL blocks.

Introduction

12

Problem Description

13

2 P R O B L E M D E S C R I P T I O N The Telefonaktiebolaget LM Ericsson is one of the worlds biggest Telecommunication Company. Ericsson has reached the success through their AXE exchange (Digital Switching System), which is Ericsson’s main income source. The AXE exchange was introduced into the market in 1977 and since that time, it has evolved with new hardware and new functions. Base Station Controllers (BSC) manages all the radio-related functions of a GSM network. It is a high capacity switch that provides functions such as mobile subscriber handover, radio channel assignment and paging (find the mobile). This is the node in the GSM network, which is of interest for our study. The majority of source code is in PLEX-C language but almost nothing new-planed development is in PLEX-C. Ericsson Radio Systems has decided to use SDL 10 language for every new software project for developing of new function blocks or function units. Therefore it is very important for coming projects to investigate the development process phases and in our case the testing process methods. BSC node is a very complex node to add new features and needs large software development effort involving several months work. Ericsson is looking for solutions that are effective to cope with the problem when it is time to implement a new feature in the system. A solution is to use SDL in development and automatically generate code. Testing of those SDL developed function blocks is done by designers with testing tools such as Simulator and Emulator tools before they deliver the code for function test and system test, which are done in a real target machine (AXE Exchange) or on a simulated environment. 2.1 PROBLEM STATEMENT There are four overall questions that need to find an answer in our thesis. 1. How the current testing strategy is applied? An investigation has to be done in the

current testing process to find out which testing methods are applied in the development process. Weaknesses and limitations of the current testing tools have to be uncovered.

2. Are there any other testing strategies? Finding the alternative testing methods that

exists, but are not applied in today’s testing process it is also another important assignment.

3. Which testing method is the ‘optimal’ one? Comparing the existing and the new

founded methods by our investigation shall bring out the optimal testing strategy. 4. How this Optimal Testing Strategy is applied in the current development process?

The Optimal Testing Strategy is aimed to improve and reveal the drawbacks of the current testing process. We have to argue about why those testing methods shall be applied in a specific development phase in the current development process followed by the Ericsson Radio Systems.

14

Additionally, some specifically issues have to be addressed to obtain a clearer picture of the testing process. 1. Investigate and acquire the simulation strategy through evaluating the current simulation

process used for Singleblock simulation. 2. Investigate and acquire a Validation strategy in order to be applied in the development

process. Validator tool is not a part of the current development process. 3. Investigate and acquire a simulation strategy for Multiblock simulation. Multiblock

simulation is a complex method that is not applied in the current testing process. 2.2 DELIMITATIONS Delimitations in our thesis represent the areas that we have excluded for some particularly reasons. The following are the delimitations in our work: 1. The delimitation of time. The time to spend on the current thesis is delimited to 20 weeks.

This includes time for the investigation period, the implementation period and the evaluation period for each testing method. Then the time to write the investigation report.

2. Delimitation of tools. The tools do not support some functions required to accomplish a

particular test methodology (e.g. Multiblock Validation).

3. Delimitation of documentation. The existing documentation is not sufficient and not adequate. Additionally, some parts of documentation were unclear and abstract e.g. Telelogic online help.

Related Works

15

3 R E L A T E D W O R K S The purpose of this chapter is to give an overview of related theories about system development. It starts with a description of the general system development process model following by general testing methods. 3.1 OVERVIEW OF SYSTEM DE VELOPMENT PROCESS The development of telecommunications software systems is a highly complex process. In order to manage this complexity various software-engineering methods have been developed, ranging from requirements and design specification techniques to verification, validation, testing and implementation methods. A typical system engineering process can be seen in Figure 3.1. All of these phases can be put under the term systems engineering. The technical complexity of creating a product is usually handled by different development models. Various organizations adapt a development process that suits better their structure and products.

Requirements Definition

System Decommissioning

System Evolution

System Design

Subsystem Development System

Installation

System Integration

Figure 3.1 System engineer process

16

3.1.1 Requirements specification techniques A requirement in telecommunication systems depends on user’s perspective. The application programmer, the end user, and the owner of the system have different requirements on the system. However, important requirements for telecommunication systems are: • Functionality, the services provided by the system. • Reliability, the probability of failures of the system. • The performance of the system, the response times for different services. • Maintainability of the system, its flexibility with respect to changes, or quality of

documentation. • Cost effectiveness of the system, how performance reliability is provided at what cost? The requirement specifications can tell a lot about what the requirement authors wants from the system and not how the system will be implemented. It can for example tell which services a system should contain but not how they are going to be implemented.

3.1.2 System Design and techniques System design process involves deciding which system capabilities are to be implemented in software and which in hardware. It may involve the design of special-purpose hardware and software. The design of real-time system such as telecommunication system e.g. AXE is considered of system response times and events rather than objects and functions. In telecommunications systems design, a lot of work has been devoted to using formal descriptions for the specification of functional properties of the designed systems. SDL Specification and Description Language [Z.100] and Message Sequence Charts (MSCs) [Z.120] are frequently used for this purpose, and they will therefore play a central role here. 3.2 SOFTWARE TESTING Software companies encounter often problems when customers are finding errors in a program that was tested in numberless hours. Was that piece of software tested as should? Has the testing staffs used the right methods for the particular code? Where all of the input values tested? An answer to be given is to look more closely at the testing process. There are many definitions about testing. IEEE gives the official definition: “the process of analyzing a software item to detect the difference between existing and required conditions and to evaluate the features of the software items” [IEEE829]. Another approach from the tester’s point of view can be that “testing is the process of trying to discover every conceivable fault or weakness in a work product” [Kit]. Testers have an important task to verify that the system works as specified in requirements. In this way, testers add value to the product.

Related Works

17

In addition to the functional aspects, the real-time mechanisms are used in the design of the system in order to ensure progress in the system and avoid deadlocks. Real-time mechanism takes into consideration to detect errors like, message losses or unavailability of resources. 3.3 TESTING PROCESS Successful testing requires a methodical approach involving discipline, structure, analysis, and measurement. Testing process is consisted of a number of test phases. 1. Planning the testing effort. Here the test team organizes its work. 2. Staffing phase. Who should test what? Are all units covered with test experts? 3. Designing the test cases. Which test strategy should be used? 4. Writing the test cases. And maintaining individual tests. 5. Measuring testing process by collecting data. Testers cannot directly improve quality.

They can only measure it. 6. Evaluate the software product. Is that product ready for release? A general model of the testing process is shown in Figure 3.2.

3.3.1 Planning the Testing Effort The first step is to plan all the activities to be performed and the methods to be used. IEEE 1012-1986 standards contain information that assists the author in preparing the plan. It serves as a checklist for contains of the testing plan. The most important considerations for a testing plan are [IEEE86]: • Purpose: To prescribe the scope, approach, resources, and schedule of the testing

activities. • Referenced documents. • Definitions. • Verification and Validation (V&V) overview: organization, master schedule, resources

summary, responsibilities, tools, techniques, and methodologies. • Life cycle for V&V: tasks, inputs, and outputs per life-cycle phase.

Design test cases

Test cases

Prepare test data

Test data

Test results

Test reports

Run program with test data

Compare results to the test cases

Figure 3.2 Testing proces

18

• Software V&V reporting: describe content, format, and timing for all V&V reports. • V&V administrative procedures: policies, procedures, standards, practices, and

conventions.

3.3.2 Staffing Phase Testing must be done independently and a test group is usually dedicated to perform testing activities. There are different approaches to staff software testing in organizations [Kit]: Approach 1. Testing is person’s responsibility It occurs in small companies and the group of product developers has the responsibility to build the product and test their own code. The problem with this approach is that programmers are biased by the creative work they do. This blinds them to their own errors. Approach 2. Testing is each unit’s responsibility The developers test each other’s code within the group, which fixes the problem encountered in approach 1. Approach 3. Testing are performing by dedicated resource Testing managers select some product developers and give them a new job, that of test expert. The tricky part here is to pick the right person for testing. This person has to have an overall understanding for the system and testing process.

3.3.3 Designing the test cases Test cases are specifications of the inputs to the test and the expected output from the system plus a statement of what is being tested. There are several test design methods and rules of thumb from experienced test designers, and as such vary somewhat from author to author. Nevertheless, there are two fundamental testing strategies:

1. Black-box methods for function-based testing. 2. White-box methods for internal-based testing.

Black-box methods Black-box tests are derived from functional design specifications without respect to the internal program structure. Those tests are done without any internal knowledge of the product. The essential problem for the tester is to select inputs that have a high probability to uncover defects. Common black-box methods are:

• Equivalence partitioning (EP) is an input/output method where input tests are divided in classes (equivalence classes, EC) represent an individual test case.

Related Works

19

• Boundary-value analysis (BVA) is similar to EP method but here classes are chosen in the edge of ECs. BVA is chosen for both inputs and outputs.

• Error guessing is an ad hoc approach. It identifies test cases that are likely to expose

errors. Here the experienced tester knows or guesses the common error situations from the past.

• State transition testing is an analytical method, using finite-state machines, to design tests for programs that have similar, but slightly different, control functions. It is primarily a function-testing tool and also has a high pay-off in functional design verification.

• Graph matrix is a simple representation of a graph to organize data. A graph matrix is used for proving things about graphs and for developing algorithms.

An example of black-box test is Function Test at Ericsson. Black-box methods that includes in Function Test are Equivalence partitioning and Boundary-value analysis. The tester does not know internal program structure. The programs are tested by sending signals and commands with expected values and the outcome is evaluated. In our suggested Optimal Testing Strategy, some part of Function Test will be simulated by Simulator tool. State transition testing is used when the SDL blocks are tested with Validator. The MSC diagrams are evaluated against SDL block design.

White-box methods

White-box methods require knowledge of the internal program structure. The execution of a given test case against program P will “cover” certain parts of P’s internal logic. Common white-box testing methods are: • Statement coverage: Each statement is executed at least once. • Decision coverage: Each decision is executed at least once. • Condition coverage: Each condition in a decision takes on all possible outcomes. • Path coverage is a structural testing method whose objective is to cover every

independent execution path through a component or program. Those paths can be discovered by computing the Cyclomatic Complexity (McCabe, 1976) of the program flow graph. However, the minimum number of test cases required to test all program paths is equal to the cyclomatic complexity.

As an example for white-box method is Basic test and SDL Block design simulation and validation, when the SDL designer tests his own design. The white-box methods that are used are Statement coverage, Decision coverage, Condition coverage, and Path coverage.

Other Methods Software Reliability-Engineered Testing SRET [Musa] combines the use of quantitative reliability objectives and operational profiles (profiles of system use). The operational profiles guide the developers to test more realistically and track the reliability actually being achieved.

20

Ericsson Development Methods

21

4 E R I C S S O N D E V E L O P M E N T M E T H O D S The purpose of this chapter is to give an overview of system development methods applied in Ericsson today. It starts with a description of the AXE 10 Software Design System Model and the SDL design method used in software development in BSC node that is a part of GSM system. 4.1 THE AXE 10 SOFTWARE DESIGN SYSTEM MODEL The AXE system can be used for a number of network applications, for example: • Local Exchange. • National Transit Exchange. • International Exchange. • Operator Exchange. • Mobile Telephony Exchange. • Combined Exchange. The philosophy behind AXE is Modularity. Modularity means to divide a system in various subsystems to achieve easy handling, which of course means reduced costs and the flexibility to adapt to the changing world of telecommunications. AXE modularity also means adaptability in all market applications. The entire AXE system is a set of specified functions, implemented in function blocks. As shown in Figure 4.1 below, these blocks are combined to form subsystems. There are system level 1, 2, subsystem level, Function Block Level and Function Unit level. AXE System Level 1 AXE is the highest system level. System Level 2 APT Switching System is the switching part responsible for traffic handling. APZ Processing System is responsible for processing power, operating system functions, I/O functions, service functions, etc. Subsystem Level Similar functions (e.g. charging functions) are grouped together in one subsystem. Function Block Level The functions allocated to a certain subsystem are further divided into individual function blocks. Each function block constitutes a well-defined entity with its own data and standardized signal interwork

22

Function Unit Level A function block may consist of: • A hardware unit. • A software unit that deals with routine work, such as the scanning of hardware devices. • A support software unit. • A central software unit

All the AXE software is divided into a number of function blocks (independent modules, "black boxes"), which can be programmed, compiled, tested and loaded as separate units. In the beginning of AXE, all subsystems were implemented in C and PLEX-C (Programming Language for EXchange, C-version), a language developed by Ericsson, especially for the needs of a real-time programming. For not that long ago Ericsson started using another programming language SDL (Specification and Description Language). SDL is graphical high-level language that is much easier to work with than PLEX-C. The system designer has a better overview of system structure and can use SDL in many phases in system development process. The dialect of SDL language that is used for AXE system development is known as SDL 10. (An overview of SDL language can be seen in Appendices A SDL).

Much work has been done to adapt SDL language to suite the needs of telecommunication development process. Some software project activities where developers use SDL 10 are the following.

AXE

APT APZ

System level 1

System level 2

Subsystem level

Function block Level (software or hardware)

Function unit Level (Software or hardware)

Figure 4.1 AXE system hierarchy

Ericsson Development Methods

23

System Design

Function Design

Block Design

Unit Design

Basic Test

Basic Function Test

Function Test

System Test

Figure 4.2 Software Design

1. Developing of a new feature. 2. Changing an old feature. 3. Reverse engineering PLEX-C source code to SDL 10. Ericsson’s AXE 10 Software Design Model used for developing AXE 10 System software is described below. This Design Model uses a top-down design plan, which means that the system is developed in levels first System Design then System Analysis etc. The testing of AXE 10 System Model is a bottom-up approach that first performs a basic test then Basic Function Test etc. see Figure 4.2. The processes that are of interest for our study are Function Design, Block Design and Software Unit Design, Basic Test and Basic Function Test. AXE 10 Software Design Model consists of 13 processes: 1. System Study 2. System Analysis 3. Source System Design 4. Subsystem Design 5. Function Specification 6. Function Design 7. Block Design 8. Software Unit Design and Basic Test 9. Basic Function Test 10. Function Test Planning 11. Function Test Specification 12. Function Test Design 13. Function Test

24

4.1.1 System Study This is the first phase in development process. The requirements from the customer are analyzed and clarified. The requirement engineers approve the customer requirements and a list of documents called Design Base Proposal is produced as a design base. A System Study Report is produced that includes the implementation of the requirements and design base proposals.

4.1.2 System Analysis The implementation requirements from System Study Report are analyzed and approved at this phase. An analysis is made of how the new or changed functions should be introduced into the system and detailed requirements for each function are listed in System Analysis Report. Functions exist to facilitate modular design in AXE 10 system. An important document in this phase is Implementation Proposals, IP document that contains implementation ideas and alternative solutions for the new or changed functions for new or existing AXE blocks.

4.1.3 Source System Design The Source Study phase is an administrative phase where the Source System details are registered in the PRIM (PRoduct Information Management) database, which is a central document register. It is also here that one can fetch new version number for a document.

4.1.4 Subsystem Design The System has been divided in modules called subsystems during analysis. Subsystems have to be described in details. The details about subsystems have to be registered in PRIM (PRoduct Information Management) database

4.1.5 Function Specification Under this phase the functionality and the requirements for a particular subsystem has to be analyzed. The software designer has to ensure that the customer requirements are fulfilled. A description of Operation and Maintenance for the function is done and a review board approves the resulting documentation.

Ericsson Development Methods

25

4.1.6 Function Design Under Function Design, the external interfaces for the function are described. The function is broken into subfunctions. Subfunctions are allocated to existing AXE blocks or to new blocks. Definitions of signal interfaces and new signals are done, if it is necessary. The outcome of this phase is Function Description document, which describes the interactions between the current function and other functions. Regarding SDL design this includes MSC diagrams. The changes that required to the existing system are described in Function Framework (FF) document. Function Description document for SDL block design contains MSC diagrams for the visualization of selected system runs (traces), which are a subject for our investigation.

4.1.7 Block Design Block Design is concerned with the question of how a function is to be implemented with SDL 10. During the design, implementation and test the incremental model is used (see Figure 5.1). This is an iterative process consisting of both implementation and testing activities. Components of the function, the SDL blocks, and their interfaces are designed and tested. The output of this phase is PLEX-C code generated from SDL.

4.1.8 Software Unit Design and Basic Test The units design and basic test phase involves both programming and testing. The block units are implemented in SDL 10 with Block Description document in mind. During the Basic Test phase the logical correctness of code is tested. Basic Test for a PLEX-C program is based on Basic Test Instruction, BTI and is executed in EMU tool, which is a tool that emulates the APZ processor in the AXE. The tests are executed against the BTI test instructions. Concerning the SDL design method this phase is joined together with Block Design. The present testing process is to test SDL block with Simulator tool. We describe the tool in chapter 6 section 6.2 Simulation.

4.1.9 Basic Function Test The main activity of Basic Function Test is to test several PLEX-C blocks together using Emulator tool.

4.1.10 Function Test Planning This is a planning phase for execution of Function Test. The resources, cost, schedule, and risks are planed and listed in a Function Test Plan document.

26

4.1.11 Function Test Specification All function tests are specified in this phase. The output is a Function Test Specification. Function Test Specification informs the function tester what to test and not how it will be done.

4.1.12 Function Test Design The tests are specified in details. The Function Test Instructions is the output of this phase, which describes how to carry out the function test.

4.1.13 Function Test This test is performed on target machine e.g. an AXE exchange or in simulated environment. All tests described in Function Test Specification are executed and test results are documented in Test Report. Eventually corrections are done by the designer. Those corrections are done in ASA1 and a new test has to be done. However, our objective is to find out if those errors can be revealed before the software is sent to function test.

1 AXE 10 processor assembly language

Ericsson Development Methods

27

4.2 DESIGNING WITH SDL The design language, which is applied today for block design and implementation at Ericsson Radio Systems in Hässleholm, is SDL. Designing with SDL is involving several development phases. To learn more about SDL see Appendix A SDL. The basic theoretical model of an SDL system consists of a set of extended finite state machines (FSMs) that run in parallel. These machines are independent of each other and communicate with distinct signals. An SDL system consists of the following components: • Structure: system level, block level, process, and procedures level. • Communication: signals with either signal parameters or without and channels. • Behavior: processes. • Data: abstract data types (ADT). • Inheritance: describing relations and specialization. • Packages: contains for example signal packages and other such as Plextypes and

GeneralRoutines

The following subsections introduce the basic concepts of SDL. Structure SDL-system contains four main hierarchical levels: • System. • Blocks. • Processes. • Procedures. For more graphical presentation of an SDL system, see Figure 4.3. Dividing a system into a system, block, and process hierarchy is called partitioning a system. The partitioning of a system, has the following advantages for design perspective: • Hiding information (move details not important to lower levels). • Following natural functional subdivisions. • Creating modules of manageable sizes. • Creating a correspondence with actual software or hardware. • Reusing already-existing specifications in order to design new SDL blocks. Each SDL process type is defined as a nested hierarchical state machine. Each substate machine is implemented in a procedure. Procedures can be recursive; they are local to a process or they can be globally available depending on their scope. SDL also supports the remote procedures, which allow making a procedure call that executes in the context of another process. SDL processes have separate memory spaces (i.e., data is local to a process or procedure). A set of processes can be logically grouped into an SDL block. These breakdowns into subsystems are important because the telecommunication systems are often very large and involve many small teams to develop them. However, SDL simplifies the co-operation between the team members by providing a clear and precise interface between different subsystems

28

idle

t

s

idle

idle

1(1) Process P

signal s, t;

t s B

1(1) System Convert

c_in c_out

c_in

sc_out

tP

1(1)blo c

Figure 4.3 Structure of a system developed in SDL

Procedure

Ericsson Development Methods

29

Static and Dynamic Structure The static structure of a system is defined in terms of blocks and channels. The dynamic structure is defined with the help of the process and the signal route concepts. A process is an independent entity that reacts on the signals. Communication SDL does not use any global data. SDL has two basic communication mechanisms: asynchronous signals (and optional signal parameters) and synchronous remote procedure calls. Both mechanisms can carry parameters to interchange and synchronize information between SDL processes and with an SDL system and its environment (e.g., non–SDL applications or other SDL systems). SDL defines clear interfaces between blocks and processes by means of a combined channel and signal route architecture. Time is an important aspect in all real-time systems but also in most distributed systems. An SDL process can set timers that expire within certain time frames to implement time-outs, when exceptions occur, but also to measure and control response times from other processes and systems. When an SDL timer expires, the process that started the timer receives a notification (signal) in the same way as it receives any other signal. Actually, an expired timer is treated in exactly the same way as a signal. SDL time is abstract in the sense that it can be efficiently mapped to the time of the target system, be it an operating system timer or hardware timer. This makes it possible to simulate time in SDL models before the target system is available. Signal and process priorities are not within the scope of SDL. These issues are left instead to the implementation phase where the user with special directives can assign signal and process priorities. An SDL signal can only be sent to one specific process instance at a time. Behavior of an SDL system The dynamic behavior in an SDL system is described in the processes. The system/block hierarchy is only a static description of the system structure. Processes in SDL can be created at system start or created and terminated at run time. More than one instance of a process can exist. Each instance has a unique process identifier (PId). This makes it possible to send signals to individual instances of a process. The concept of processes and process instances that work autonomously and concurrently makes SDL a true real-time language. SDL design using MSC diagrams Regarding the SDL design, functional requirements can be described in a formal way, using MSC diagrams. MSC diagrams focus on the communication behavior of system components and their environment by means of message exchange. The MSC language is used most frequently in connection with SDL. In contrast to SDL, the set of specified MSCs usually covers a partial system behavior. Candidates for MSCs are primarily the standard cases. These standard cases may be supplemented by MSCs describing exceptional behavior altogether providing a use case like representation. To learn more about MSC language read Appendix B MSC.

30

Ericsson Development Methods

31

4.3 PROJECT P ROCESS Almost all development in Ericsson operates in projects. A project can involve designers from different cities and countries. Therefore, it is important to assure that the project has an adequate process. Most of the processes within Ericsson follow the PROPS process

4.3.1 PROPS PROPS states in Swedish Projektet Projekt Styrning, which means, "Project for project guidance". It was created and evolved by the Ericsson Project Management Institute. The purpose of the project flow in PROPS is to integrate and co-ordinate all individual efforts in the project in order to run the project successfully and have it reach its goal. PROPS include a general model for the flow of the project see Figure 4.4, which describes what, is to be done and when this is to be done.

Prestudy Phase

Feasibility Study Phase

Execution Phase

Conclusion Phase

Figure 4.4 PROPS Project Flow

Ericsson Test Methods

32

5 E R I C S S O N T E S T M E T H O D S In this chapter we are going to describe Ericsson’s development process with focus on Block/Unit Design. As we mentioned in the previous chapter this model is a ‘bottom-up’ model. See Figure 4.2 Software Design. The development process, which is applied at Ericsson, is an iterative process that swings between design and test. The implemented software is tested from the single unit up to more complex system level.

The development process is depicted in Figure 5.1. An increment means a system release. As it shows in figure, increment X consists of several iterations (Iter 1-2, Iter 2,..). It starts with Function design followed by Block/Unit Design, Function Test and at the end the whole system are tested before the release. The structure of the development work is divided up in smaller work breakdowns for the design and testing activities in order to perform them in parallel. These work breakdowns are dedicated to respective iterations, which includes several blocks see Figure 5.2. Each block is assigned to a development group that usually consists of two SDL designers and one tester. The testing personnel and SDL designers are working in parallel within each development group.

FD Iter 1-2

FD Iter 4

FD Iter 5

FD Iter 6

Increment X

Iter 1 Iter 2 Iter 3 Iter 4 Iter 5 Iter 6

BD/UD Iter 1

BD/UD Iter 2

BD/UD Iter 3

BD/UD Iter 4

BD/UD Iter 5

BD/UD Iter 6

FD Iter 3

FT Iter 1

FT Iter 2

FT Iter 3

FT Iter 4

FT Iter 5

FT Iter 6

System Test

Function Design

Block/Unit Design

Function Test

Figure 5.1 Development process

Ericsson Test Methods

33

The Figure 5.2 shows two iterations, iteration 1 and iteration 2 (similar to the iteration 1) which are a part of an increment. Iteration 1 consists of three blocks A, B, and C that are developed and tested in parallel before they start with the next iteration Our investigation is focused on the present testing process, which is dedicated to block design and unit design (BD/UD) phase. During the BD/UD test process at Ericsson, the designed items are tested against the requirements specified in Function Description document. While the SDL designers are implementing the block A drop 1.1, the tester is preparing the simulation script files and structure, which is going to be used later on during simulation process. When the SDL designers are delivering the drop 1.1, the tester starts the simulation 1.1. The errors that are found during simulation 1.1 are sent back to the SDL designers and

Block A

Block B

Block C

Time

Sim Preparation Struct/Skel

Sim 1.1

Block 1.1

Design

Sim 1.2+

reg1.1

Corr

Sim 1.n+ reg

1.1-1.2

Sim+ Emu

Corr+ Reg

Block 1.2

Design+Corr

Block 1.n

Design+Corr

Sim Preparation Struct/Skel

Sim 1.1

Block 1.1

Design

Sim 1.2+

reg1.1

Corr

Sim 1.n+ reg

1.1-1.2

Sim+ Emu

Corr+ Reg

Block 1.2

Design+Corr

Block 1.n

Design+Corr

Sim Preparation Struct/Skel

Sim 1.1

Block 1.1

Design

Sim 1.2+

Reg1.1

Corr

Sim 1.n+ reg

1.1-1.2

Sim+ Emu

Corr+ Reg

Block 1.2

Design+Corr

Block 1.n

Design+Corr

Iteration 1 Iteration 2

Figure 5.2 Incremental SDL Design

34

the corrections are included in the next drop 1.2. This iteration is going on until the final corrections are implemented in Corr drop and regression test is applied on all drops (1.1-1.n). This final test includes the Emulation process, which tests some functions that are not possible to test during the simulation. When all blocks within a particular iteration are developed and tested, they are delivered for Function Test. The development groups are assigned to new blocks in iteration 2 and initiate this iteration, while the function test personnel are sending back eventually errors found. Function Test. This test includes several blocks that are a part of a function. The Function Test is performed either on target machine or on an APZ Emulator tool. SDL blocks designs are transformed automatically into PLEX-C code and an Emulator dump file is created to use during function test. System Test. This test is testing the entire system before the increment release.

35

Investigation of Testing Tools

36

6 I N V E S T I G A T I O N O F T E S T I N G T O O L S Using testing tools, tests become easier to perform and manage and testing process turn out to be more productive and effective. Recent years have seen the rapid emergence of tools that test software automatically. Automatic testing tools can catch errors earlier in the development process, where they are easier and cheaper to fix and have less impact on the customer. In our paper we investigate the testing tools available during SDL block design at Ericsson. Those tools are described in the followed section. 6.1 ERICSSON TAU The Ericsson Tau has been developed by Telelogic in cooperation with Ericsson. It is a version of the wide used Telelogic TAU suite for primary developing software. Both Ericsson Tau and Telelogic TAU consist of a number of tools to use for different phases in software development process. Ericsson Tau has a wide field of application that extends from Functional Design phase to Block Design. During Function Design it is used to describe the system requirements by using MSC diagrams. With MSC diagrams the system’s signal exchange is represented by message interchange between system processes. During Block Design phase Ericsson Tau’s SDL editor is used to implement SDL blocks and then simulate them using Simulator tool. Some other features are to plot state charts, and generate index of all entities included in SDL system. A state chart shows different system states and transitions. However designing telecommunication systems using SDL is quite new to development process. Since many blocks have been developed in PLEX-C and became a part of very complex systems, are very hard to modify and maintain. Today some of these blocks need modifications in form of adding new functionality. Ericsson Tau tool supporting the reverse engineering work to transform the old PLEX-C code to SDL, which is easier to modify and maintain. Ericsson Tau tools are depicted in Figure 6.1 Development Tools. Ericsson Tau tools that is in use for telecommunication software product development at Ericsson: • PLEX2SDL, which is a reverse engineering tool that transform PLEX-C into SDL. • SDL2PLEX, which is a PLEX-C code generator tool, where SDL designed programs, can

be transformed into PLEX-C. • Mediator tool that is used for integration of the APZ-Emulator and PlexView with

Simulator in order to facilitate APZ Emulation adding a graphical interface. • SDLPACK, which is a utility for compressing SDL design files. • SDLSIGTOOL is a tool, which enables modification, creation and converting PLEX-C

Signal Descriptions to SDL10 Signal Descriptions.

Investigation of Testing Tools

37

• UML-Suite. It is a tool for system specification and design using Unified Modeling

Language and is connected to SDL design. • SDL-Suite. Contains a number of SDL and MSC editors. • TTCN-Suite. It is a tool to generate test cases and execute them. • Simulator. It is a tool to test SDL systems design. • Validator. It is a tool for validating the behavior of an SDL system. • TAU Organizer. It is the wraparound tool to organize the SDL system design and testing

process activities. The testing tools that are concerned with test activities are: • Simulator. • Validator. • Mediator. • APZ Emulator. • TTCN-suite.

We are going to investigate the testing tools except TTCN-suite, which is out of the scope of our thesis. In the following chapters, we are going to analyze, test, and suggest improvements for the testing tools mentioned above.

Figure 6.1 Development Tools

38

6.2 SIMULATION A simulation is the execution of a model, represented by a computer program that gives information about the system being investigated. The simulation approach of analyzing a model is opposed to the analytical approach, where the method of analyzing the system is purely theoretical. As this approach is more reliable, the simulation approach gives more flexibility and convenience. System simulation substitutes the operation of a real system, such as a day-to-day operation of an AXE digital exchange. Simulation is a good way to test the behavior of the system as well as the interaction between blocks that are part of this system. Using simulation during system development offers several advantages: • It allows designers to create a functional prototype of the system. Simulation of this

prototype will increase the understanding of the system and make it possible to decide on the changes of the system earlier in the development.

• It helps to discover weaknesses and errors in the system early in the development. The

earlier an error is found the easier it is to correct it and cost much less then finding the same error during system operation.

• It provides the designers with the ability to test an executable model much earlier in

development process. • It facilitates debugging of the system. • The design process will be continuously tested and improved (if needed).

6.2.1 Simulator tool The tool to use for simulation is the Simulator. The Simulator is used during Block Design phase. The simulator tool is a window-based tool consists of two main parts: the application itself (the simulated SDL system in form of *.sdt file), and an interactive monitor system. The monitor system is the interface between the user and the simulated system. It contains graphical components, such as buttons, menus, and scrollable lists, which make it easier to run the simulator in an intuitive way. It is designed to minimize the required amount of user interaction. The user has the option to take advantage of the graphical features, which are provided, or to enter commands in a command line fashion. The test personnel at Ericsson use a third approach to enter simulation script files that actually contains commands and macro definitions for different kinds of parameters for signal interaction. Simulator can be used during Singleblock simulation and Multiblock simulation. Those two methods are described in more detail later in this chapter in the respectively sections.

Investigation of Testing Tools

39

6.2.2 Singleblock simulation During the Singleblock simulation the SDL system contains only one block and all other blocks are represented as environment. The test personnel can: • Check the process instances creation. • Check that the signals that were sending from one block to environment have been

correctly received. • Check that the values of signal parameters in the receiving process are correctly received. • Check the values of timers. • Set breakpoints in the SDL flow to control the execution. • Follow the execution trace either in textual or graphical mode • Control the logging facilities included in Simulator tool.

During SDL Block design phase simulation, the testing activities start with creating simulation script files framework based on Function Description Document and with details from Block Design. These files are created by the testing personnel. The objective is to cover as much as possible in the SDL flow. These simulation script files usually are designed to reach 90 percent system coverage. When the SDL designers deliver SDL block design the simulation, as this test is called, can begin. During the simulation the simulation script files created as we mention above are used. The simulator script files are executed and all errors found are delivered to SDL designers for correction. When the errors are corrected by the responsible system designer(s) the designed SDL block is tested again with exactly the same tests scripts. This is called for regression test. The Singleblock simulation process is described more in details in chapter 7 section 7.2.1. The current testing process during Block Design phase works sufficient with some exceptions as we mention in the next section.

6.2.3 Problems with simulation There are some restrictions in SDL 10 design regarding simulation. There is some AXE related procedures that are impossible to simulate on SDL level. Those procedures are: • Start – Restart procedure (Later in this work we will show how to simulate this

procedure). • File Size Alteration. • Forlopp release handling.

40

Start-Restart procedure Start-Restart is an AXE–specific procedure and it is performed to restore the system to a predefined state. There are different restart cases INITIATE, SMALL, LARGE, and RELOAD. In some cases, it is appropriate to delay the system restart to a time when traffic is low. That is Selective Restart. All SDL blocks have to implement Start Restart procedures. File Size Alteration It is implemented in SDL by using special directives per process. File Size Alteration properties are controlled with two directives, #DSA and #SAE. DSA (Dynamic Size Alteration) is placed at process level and SAE is placed in each size alterable entity in the system to specify the properties for the individual entity. File Size Alteration cannot be simulated on SDL 10 level. The reason is that SDL individuals are created and terminated dynamically. The idea is to maintain good memory economy. Forlopp release handling Forlopp release handling is used to avoid restart of the entire system in case of failure in an individual. It just has to restart the failed individual. It is implemented by using an AXE-specific directive. We have experimented with a package included in Ericsson Tau called RestartSimulation in order to check if it is possible to implement a Start – Restart procedure simulation. More about this experiment in the following section.

6.2.4 RestartSimulation Ericsson Tau contains a package called RestartSimulation, which makes it possible to simulate the restart procedures implemented on SDL level. This package has been studied and implemented in our study phase in order to evaluate it. In the next section we will describe how to apply RestartSimulation package on an existed SDL block system. The RestartSimulation package adds a Restart block to the system and prepares the system’s BlockProcesses to receive Restart signals making possible to simulate the Restart behavior on SDL level. Prerequisites 1. Include the RestartSimulation package into the target SDL block 2. The package is included in the Organizer and a “Use “ clause has to be added on system

level. 3. Simulation script file, which is used to simulate different restart cases.

Investigation of Testing Tools

41

Restart simulation process This is a general Restart simulation process; see Figure 6.2 Restart simulation Process. For implementation details see Appendix F Simulation of Start Restart. 1. Adding an instance of Restart block to the system (for example MyRestart). The new

instance must contain a process called Restart that sends and receives all signals to and from environment and BlockProcess.

2. Connect the block instance (MyRestart) with the BlockProcess in SDL system. 3. All Restart procedures in an SDL system, which are not in the BlockProcess, have to be

declared as EXPORTED. Those procedures have to be IMPORTED in the BlockProcess. 4. Execute the simulation script file.

Add aninstance of

Restart block

Connect theRestart block

with SDL block

Export allrestart

procedures

Import restartprocedures inBlockProcess

Execute simulationof restart

procedures

Figure 6.2 Restart Simulation Process

Running a Restart Simulation The User activates Restart block by sending the signal SYREI that is actually an APZ command. Then Restart block sends STTOR signal to BlockProcess. The parameters of this signal determine what kind of restart case to execute and what phase. The phase contains information about which restart procedure to be executed next. When a restart case procedure is executed, the BlockProcess sends a response signal STTORRY with the following parameter: restart case and phase, to the Restart block as an acknowledgment, that STTOR has been received. After the STTORRY signal is received in Restart block, the requested phases are stored. By using RestartSimulation package the simulation of Start-Restart procedures is possible on SDL level. It is desirable to find a way to simulate Forlopp Release Handling and File Size Alteration on SDL level and consequently minimizes the use of Emulator. For more detail about Emulator see section 6.4.

42

6.3 VALIDATION Validation is an automated fault detection mechanism that checks the robustness of the application and finds inconsistencies and problems in an early stage of the development. It is also used to check that the formal specification itself is logical consistent. Systems can fail in some standard ways. The most important ones [Holzman]:

• Deadlocks: States in which no further execution is possible, for instance because all processes are waiting for conditions that can never be fulfilled.

• Livelocks: Execution sequences that can be repeated indefinitely often without making

effective progress.

• Improper termination: The completion of an execution without satisfying the proper termination condition.

The observance of these criteria cannot be verified by a manual inspection of specifications. More powerful tools are needed to prevent or detect them. A tool that supports validation is Validator in Ericsson Tau.

6.3.1 Validation of a system The Validation of a system is based on a technique called state space exploration, which is used for automatic analysis of distributed systems. System state space contains all reachable states of the system and the possible ways that they can be reached. State space structure can be described with behavior trees or reachabillity graphs as in Figure 6.3. Those structures represent the behavior of a system. The nodes of the tree represent system states. The sum of all system states is the state space.

The system states represent all possible states in a system. The edges between the nodes in behavior tree represent the systems transitions from one state to another. The edges are also called behavior tree transitions. Transitions represent SDL events like assignments, input and output of signals and tasks. The system states include: • Instances of processes that are active. • The value of all variables. • The SDL control flow state of all active process instances. • All procedure calls.

System states Transitions

Figure 6.3 Behavior tree

Investigation of Testing Tools

43

• All signals transmissions (with their parameters) that are present in the queues of the system (time queues and job queues).

• All active timers. There are a number of various state space exploration algorithms that could be applied to validate a system. Those algorithms are described below.

6.3.2 Exploration Algorithms By traversing the tree, the behavior of the system can be explored and the system states reached can be examined. Most automatic Validation systems are based on exhaustive reachabillity methods. This can be performed either manually or automatically. Automatic exploration can be achieved by using different algorithms to traverse the behavior tree. The particular exploration algorithm depends among other things on the size of the development. The algorithms to traverse are: Exhaustive exploration Covers the complete system state-space starting from the current system state and up to defined depth. Every system state stored in primary memory to avoid exploring system states several times. The exhaustive method is suitable for relative small systems, up to approximately 105 reachable system states ([Holzman] Chapter 11). Bit-State Exploration Covers the complete system state-space from current state to defined depth. The system states are stored using a hash value called Bit-State, in order to mark already visited nodes in Behavior Tree. This ensures that every system state is only explored once. The method is suitable for systems with reasonable complexity, up to approximately 10 8 reachable system states ([Holzman] Chapter 11). Random Walk The Random Walk method performs a depth-first search through the system state-space by selecting transitions to execute at random. This method try to explore as many system states as possible, trying those that can violate the system invariants i.e. system rules. When the maximum search depth is reached during such a random path, the search is restarted from the original state again and a new Random Walk is performed. The method is suitable for exceptional large systems. Power-Walk Power-Walk is a combination of the Exhaustive Exploration and the Random-Walk method, and is optimized to generate a set of paths with a high SDL symbol coverage. The main usage is to generate MSC paths for automatic test generation. The method is suitable for small and large systems and is the fastest way to give a good coverage of the SDL system.

44

Tree-Walk In compare to Random-Walk and Power-Walk, Tree Walk is based on a strongly deterministic algorithm that performs a sequence of tree searches with increasing depth starting at different states in the exploration space. Whenever Tree Walk makes a transition that increases the coverage, the next state is stored in a list of testing states that are used for further tree searches. The method is suitable for all kinds of systems. Tree-Search Performs a tree search of the state space from the current system state. A tree search is an exploration where all possible combinations of actions are executed. The tree that is explored is exactly the same tree that can manually be inspected using the Navigator tool. The method is not suitable for large systems. All of the algorithms perform an automatic exploration of the state space starting from the current system state.

6.3.3 Validator tool The Validator provides the possibility to interactively walk around in the behavior tree of an SDL block system. It is a powerful tool to Validate SDL block systems. In our study phase we implement and test a validation file for a Singleblock.

6.3.4 Application areas Using the Validator tool the designer/tester can validate the behavior of the entire system. Validation deals with the question Did I build a valid system. With valid system we mean that it is designed with respect to specific correctness criteria. We have mentioned three criteria earlier in this chapter the absence of deadlocks, livelocks and improper termination. Validation using Validator tool means validation and verification of an SDL system. Validation is used to check that the correct system is build with respect to the requirements. It validates that the created SDL system has the functionality and the behavior as defined with MSC diagrams in Function Description Document. Validator also supports a manually and automatically state space exploration to check the system for errors. We mention about the automatically exploration methods and algorithms that they use in the previous section. The manually exploration is also known as manually navigating in the state space. A dedicated graphical tool, the Navigator, is available in the Validator to facilitate manual navigating. The Navigator tool should be used on three different occasions: • When learning how states space exploration tool works in Validator.

Investigation of Testing Tools

45

• Can be used to walk to a suitable system start state of the SDL block system, from which the automatic exploration can be started.

• Supports investigating a report generated during automatic exploration, to check the alternative behaviors that are possible on the path to the reported situation.

Validator tool is not used during the Function Design and Block Design phase. During Block Design phase it can be used to checks the following: • Process creation and destruction. • That the system is not in deadlock. • That there are no output errors. • That there are no infinitive loops. • That there is no implicit signals consumption. • Verification of MSC diagrams against SDL system. Validator can also use MSC to generate TTCN test cases. TTCN (Tree and Tabular Combined Notation) is a formal language for the specification of test cases. This can be done with Autolink. Autolink is a tool for automatic test generation. It allows generating TTCN test suites based on a given SDL specification and MSC requirements.

6.3.5 Verification of MSCs MSC verification is one of the major application areas for the Validator. The MSC diagrams to verify are the formal description included in the Function Description document and illustrate a number of scenarios of a system run. The task of verifying an MSC diagram is to check if the SDL system corresponds to required function behavior. Validator applies Bit-State Exploration algorithm in order to verify an MSC diagram. There are some prerequisites to run MSC verification: • The MSC diagram to verify has to be compliant with the structural concepts that are

recommended by ITU-T recommendation Z.120. [Z120] (See details in Appendices B Message Sequence Charts (MSC))

• The MSC files have to describe interactions between the current block and to/from

environment. Those MSCs in Function Description Document are descriptive actions of the system, which might be changed in order to be accepted by Validator.

• Convert all instances that are not part of the system. Instance conversion will convert the

name of instance. If you have more than one instance representing the environment, the environment instances must be separated using channel names.

• It must be possible to map each MSC instance to either the environment, a channel

to/from the environment, the entire SDL system, a block, or a process. This mapping is done by matching the name of the MSC instance with the names of the corresponding SDL entities.

46

A typical MSC diagram that is accepted by Validator is illustrated in Figure 6.4. MSC Verification starts with the Validator Command “Verify-MSC <MSC diagram file>”. This command will perform an automatic exploration from the current system state to search for all execution traces. The file can either be an basic MSC file (.msc), a textual MSC file

(.mpr) or a high-level MSC file (.mrm).

6.3.6 Validator Rules The Validator checks the system against a number of rules executed for each transition. Whenever a rule is violated, the Validator saves a report that includes information about what rule was violated and the path in the state space to the violation. Table 6.1 shows all rules that are checked during the Validation. It is also compared with the rules checked during the simulation.

Opt Signal1(,,,,,)

1

Env_block1 Env_block2 Our_block environment

START_STATE

Task

FINISH_STATE

Signal sent from our block to env.

Figure 6.4 MSC diagram

AnsSignal1()

Signal2(1,2) AnsSignal2(6)

Investigation of Testing Tools

47

Table 6.1 Validation Rules

Rule Description Checked during the simulation

Deadlock A permanent condition in which a system cannot continue to function unless some corrective action is taken.

No

ImplSigCons Implicit Signal Consumption of signal: a signal was sent to a process that was not able to handle (or save) the signal in the current state, so it was implicitly consumed.

Yes

MaxQueueLength The value of the length defined by monitor command: Define-Max-Input-Port-Length was exceeded in the receiving process.

No

ChannelOutput

This rule checks the following: • Tried to send a signal to an invalid PId. • Tried to send a signal to a PId, but there was no

path of channels and signal routes between the sender and the receiver.

• Tried to send a signal to a PId that referred to a process instance, which is inactive.

• Tried to send a signal to a PId that was null.

Yes

Operator Operator Errors: errors that can be found in operators defined in the predefined data types. Yes

Subrange A variable of a restricted syntype is assigned a value out of its range. Yes

Index An array index is out of range. Yes Decision Error in SDL Decision: The value of the expression

in the SDL decision did not match any of the answers.

Yes

Create Error in SDL Create: An attempt has been made to create more static instances than maximum number of instances.

Yes

Import Import Errors: Error during execution of an import statement. E.g. File missing, wrong path, etc.

No

View View Errors: attempt to view from the environment or from NULL.

Yes

MaxTransLen Transition Length Error: The maximum number of SDL symbols executed in one behavior tree transition is more than the value defined by the monitor command Define-Max-Transition-Length.

No

UserRule User Defined Rule: A predefined signal, variable value or condition by the user is evaluated to true. E.g. state(CellProcess:4)=CP_Waiting. Means that the Validator should check if state of CellProcess: 4 is reaching CP_Waiting state.

Yes

MSCViolation MSC Verification Errors. No MSCVerification The MSC verification report is given. No Assertion Assertion Errors: A user-defined action in the SDL

system has called the function No

48

‘xAssertError’ which is a C inline expression, inside a TASK box. E.g. /*#CODE #ifdef XASSERT operationx(#(A) / #(B)) xAssertionError(“divided by zero”) #endif

In this case a report is generated when an error is encountered.

Loop Non-Progress Loop Error: loop detected. No RefError REF Errors: No reference to dynamically allocated

memory of a sort. This indicates a memory leak in the system.

No

When the automatic exploration finishes, a report are presented in a clickable tree overview, giving easy access to the system states that require investigation.

6.3.7 Results The results obtained during the validation present below by method. 1.A Validator file (*.val) has been created for a particular block. 2.A*.com file that should include accepted values assigned to: signal parameters, state size

(1000000), transition length (1000000) and process instances (10) This file can also contain User defined rules, in our case no such rules were defined. (Those rules can be found in the Table 6.1). *.com files can also be used as batch files for automation of validation process.

3.It must be possible to map each MSC instance to the environment, a channel to or from the

environment, the entire SDL system, a block, or a process. Optionally: The command “Define-Instances-Convention” before verification can be used to change the name of an MSC instances e.g. Define-Instance-Convention RNLCT env_1. This tries to map block/process RNLCT to environment env_12.

4.PId values are not allowed as parameters for MSC messages from the environment of the SDL system. PId values are allowed for internal messages and messages to the environment, but the values are not checked during the exploration.

2 If you have more than one instance representing the environment, the environment instances must be separated using channel names.

Investigation of Testing Tools

49

Bit-State Exploration: took fourteen hours, number of reports produced were eight, states generated 10711858 and symbol coverage 29.87%. The following violation of rules and the number of rules violated were found: ImplSigCons 6,View 1 and Assertion 1. Random-Walk Exploration: took 15 minutes, number of reports produced were seven, states generated 208505 and symbol coverage 19.08%. The following violation of rules and the number of rules violated were found: ImplSigCons 5,View 1 and Assertion 1. Tree-Walk Exploration: the time and symbol coverage are set by the user, we used the following setting time 20 minutes and coverage 20%, number of reports produced were five, states generated 22438 and symbol coverage 22.40%. The following violation of rules and the number of rules violated were found: ImplSigCons 3,View 1 and Assertion 1 Exhaustive Exploration: the validation could not be performed because of the size of the system. It is impossible to perform exhaustive exploration on large SDL block designed for use in BCS node. However, if the size allows the method can be used. Power-Walk Exploration: the validation could not be performed completely due to the predefined number of power walks. This value cannot be adjusted. The time of exploration until the number of predefined power walks is eight minutes, number of reports produced were six, states generated 94000 and symbol coverage 35.94%. The following violation of rules and the number of rules violated were found: ImplSigCons 4,View 1 and Assertion 1. Tree-Search Exploration: took more than 24 hours, number of reports produced were eight, states generated 22330970 and symbol coverage 18.20%. The following violation of rules and the number of rules violated were found: ImplSigCons 6,View 1 and Assertion 1. MSC Verification could be done for some MSCs that had the graphical syntax according to Figure 6.4 MSC diagram above. Some MSC had been verified after some signals and block names conversions had been altered. Several MSC violations have been reported. Most common violations were parsing errors because of unknown block names or signals. This is because Validator file contains information just for the current SDL block and its environment and does not parse other blocks or signal exchanges between other blocks.

50

6.3.8 Conclusions The conclusions are based on Singleblock validation process and we argue about each validation method separately. Bit-State Exploration: The method should not be used because it takes to long time to validate large SDL blocks. However, the method can be improved by lower the number of states and the number of instances. It is possible to specify user rules and constraints that reduce the system space. Random-Walk Exploration: The Random-Walk method is the most effective comparing to other methods. The method can be used before the simulation, to check the system for e.g. deadlocks. Algorithm is a very fast and adopted for reasonable large systems. Tree-Walk Exploration: The algorithm is effective for large systems and can be used before the simulation. The advantage of this algorithm is that the user can specify both time and symbol coverage. To specify symbol coverage is easy but to specify how long time the validation process needs to reach the specified coverage is hard. If the time expires and the specified coverage is not reached the Validator aborts validation without generating any reports. The Validator has to be restarted and validation process repeated. Exhaustive Exploration: This algorithm can only be used efficiently on small systems since states are stored in the primary memory the amount of memory needed to validate large systems should be enormous. Power-Walk Exploration: The advantages of this algorithm are its efficiency and high symbol coverage. The disadvantage is that the predefined number of power walks could not be changed. The algorithm should be used before the simulation if the number of power walks could be changed. Tree-Search Exploration: The algorithm should not be used because it takes to long time and finds same type of errors as Random-Walk. MSC Verification: To be able to conduct MSC verification some extra work needs to be done during the function design phase. If the MSCs are written without following the ITU-T’s Z.120 standards (see prerequisite in section Results above), it is not possible to verify them. The drawback with this is that those “executables” MSCs will become hard to understand, which is the point with the current function descriptive MSCs. Multiblock: We could not create Validator file from Multiblock system. The problem was with GNU C compilator, which was not able to handle large systems. Implicit signal consumption errors found during the validation should be ignored, because the Validator sends signals in random order.

Investigation of Testing Tools

51

6.3.9 Problems The presented methods are part of our experience with the validation process. The problems and the difficulties that occurred merely often were: • Removing a user-defined rule from a combined rule definition, all other rules will be

removed. • The absence of instances to interact with are not included in the system in the current

Validator file. • Signals that are sent by other blocks but the current are not parsed in MSC diagrams. This

is because we have tested descriptive MSCs from Function Description Document. • The Validator assigns boundary values by default to undefined parameters, which are

bigger than allowed for some signals. • Some parameter values of type R, are not defined on SDL level. • PId values are not checked if they are sent from the environment.

6.3.10 Proposals Some workarounds could be made to improve the validation process. This could be done by adding some functionality in the Validator tool. Proposals and solutions in our evaluation are: • Be sure that the prerequisites for MSCs mentioned in section Results, are satisfied during

Function Design phase. This could be ensured by adding a new rule in Design Rules Document. An improvement for Validator tool should be to ignore instances from other blocks then the current. Alternative solution could be to modify MSCs just before verification. This is because the designer needs those descriptive MSCs from Function Description Document to support the Block Design process. It is hard for a designer to understand Validator adaptable MSC diagrams.

• Be able to change predefined number of power-walks during the execution of the power-

walk algorithm. If this number could be changed, then the entire system could be validated.

• The reports should be generated even if the specified time or coverage is not reached

during the tree-walk algorithm. • The best way to use the Validator is to use it before the simulation. The method to use is

Random-Walk. This method gives the desirable coverage in reasonable time for large systems.

52

6.4 EMULATION METHOD Emulation is an automated mechanism that emulates program execution on Central Processor unit (CP) of the APZ processor in AXE 10 System. It is used to test the consistency of PLEX-C developed blocks. This is done by APZ Emulator tool that is situated in APStool family. Since the advent of the new SDL design development has arisen the demand for a tool to test AXE specific procedures (Start/Restart, File Size Alteration, etc) that are impossible to test in SDL level with Simulator tool. APZ Emulator tool is use to test AXE specific procedures in the current testing process, such as Start/Restart3, Forlopp Release Handling, File Size Alteration, Command blocks, and AXE specific signals properties that are impossible to simulate on SDL level.

The APZ Emulator APZ Emulator is a tool developed as a platform for test and debugging CP software written for AXE 10. APZ Emulator emulates a real processor. It can be used in different phases in the development process. The main uses are: a) Unit Design & Basic Test. b) Function Test (when a simulated environment is used).

BTI Driver The BTI Driver together with the APZ Emulator is the tool to manage Basic Tests. The BTI Driver extracts test cases from the BTI-script file, which are then runs on the APZ Emulator. Executions of selected test cases are presented in BTI Driver’s text area. The execution can also be shown in as well the Mediator as in the APZ Emulator. When the execution of selected test cases are complete the more detailed presentation of errors (if any) can be obtained in the BTI Driver’s upper text area.

3 Start/Restart procedure is possible to simulate on SDL level as we describe in the section 6.2.4.

Investigation of Testing Tools

53

The Mediator tool

The Mediator is part of Ericsson Tau suite. It is a communication bridge between APS System and Ericsson Tau (see Figure 6.5.). The Mediator can operate on both SDL-level and PLEX-C level with debugging features like: a) Setting a breakpoint in PLEX-C code execution. b) Single stepping through PLEX-C code. c) Executing to the next breakpoint. d) Viewing a variable. e) MSC trace of AXE Emulation. The main application area for Mediator tool is to transform the BTI script files into simulation script files. This is to gain time when reverse engineering process (PLEX-C to SDL transform) is used.

APStool Ericsson Tau

Figure 6.5 Mediator

APZ Emulator PlexView

BTI Driver

Simulator UI MSC Editor SDL Editor

Mediator UI

54

6.5 MULTIBLOCK SIMULATION To conduct a Multiblock simulation one needs at least two blocks. Theoretically, there is no limitation on maximum number of blocks for a Multiblock simulation. The limits are processor capacity and the tools that are to be used. To conduct a Multiblock simulation with many blocks requires a very fast machine and an efficient analyzing tool.

The question is why to conduct a Multiblock simulation? The answer is that Multiblock simulation is used to check signal sending and receptions between SDL blocks, which are a part of a function, and the correctness of parameter values. Those checks are not possible with Singleblock simulation Is it not enough with Singleblock simulation to test an SDL block? During the Singleblock simulation, one can never see that signals that were sending from one block to another have been correctly received. A Singleblock system contains only one block and all other blocks are represented as environment. The correctness of values of signal parameters in the receiving block is not checked during Singleblock simulation. Another advantage of applying Multiblock simulation is to test a function that extends to more then one block (which is often the case). Multiblock simulation makes it possible to see the interchange of signals between the sending and the receiving process instances in the right order and with right parameter values, which are part of the function.

The great disadvantage of Multiblock simulation is the overhead of building a Multiblock system.

6.5.1 Different approaches for Multiblock simulation To be able to conduct a Multiblock simulation one can choose from two different approaches: 1. Merging blocks into one system. 2. Co-Simulation. Merging blocks into one system was evaluated during our work. It is a straightforward approach. That is to create a system that contains arbitrary number of blocks. Include the necessary blocks to set up a Multiblock system and establish the connection between the blocks. Then connect the Multiblock system with environment.

• Apply Singleblock simulation if you want to test internal and external (i.e. environment) signal communication and signal parameter values of an SDL block.

• Apply Multiblock simulation if you want to test external (i.e. with other SDL blocks)

signal communication, signal parameter values, or some function.

Investigation of Testing Tools

55

During Co-Simulation, at least two or more SDT Simulator UI runs in parallel. The simulator files are generated from different systems in order to exchange any arbitrary signal between blocks. We have to mention that the first approach i.e. Merging blocks into one system requires more work then Co-Simulation. We explain later about those processes.

6.5.2 Results In our work, we used simulation strategy Interaction simulation and Co-simulation. We excluded Function simulation since the process is identical to Interaction Simulation. The only difference between Interaction simulation and Function simulation is test cases. In our investigation, we used already developed SDL blocks that had been successfully tested with Singleblock simulation. We used three blocks to setup a Multiblock system that is a part of GSM traffic control. The Multiblock system was compiled without any errors and simulation was performed without any errors found. The results that we obtained was expected with a few exceptions concerning some operators used in SDL design which did not behave as supposed to, which we mention in the Appendix G Multiblock simulations. The signal interactions that have been expected could be seen in MSC traces and the correctness of parameter values could be verified.

6.5.3 Conclusions In our conclusions we are going to comment the results obtained from Multiblock simulation and then we compare the methods with Singleblock simulation method and in the end of this section we compare three different Multiblock methods with each other and mention advantages/disadvantages about every method. The Multiblock simulation considered being the sufficient method to be used to test signal exchange between different blocks that are included in a Multiblock system and eventually test their functionality. On Basic Test level, it is only Multiblock simulation that reveals if those signal interchanges are executed correct and dispensable if the goal is to obtain an error free code. The differences between simulating a Singleblock system and Multiblock system are not that great i.e. they both use the same simulation process and tool. The main difference between them is that Multiblock system is much bigger and harder to overview in Ericsson Tau Organizer. Another difference is that Multiblock systems take considerable long time to prepare for simulation then Singleblock system. The Multiblock simulation can never replace Singleblock simulation. With Singleblock simulation the systems are easily managed and errors supposed to be found is on block level. Singleblock simulation can be used during a SDL block development and is aimed to find

56

errors inside the actual block and signal interface to the environment is checked against signal descriptions. Nevertheless, using Multiblock simulation to find those errors would be difficult because of the system complexity and size. The Multiblock simulation should be applied after Singleblock simulation (and after the correction in SDL block design is done) to discover interface errors that are not visible in Singleblock simulation. Beside that, we can check that all blocks in the Multiblock system function together. Comparing the Interaction and Function simulation with Co-Simulation the main different is that more work effort is required to run the first two simulation strategies. Advantages and disadvantages for individual Multiblock simulation present below. Advantages/Disadvantages with Interaction simulation Advantage with this method is interface errors are revealed, signals are correctly received and if signals have the right parameter values. Disadvantage with Interaction simulation is that it takes long time setting up the system as well as compiling with the existing tools. It is also a disadvantage that some specific SDL 10 operators e.g. Partner, FindNext and probably FindFirst are not supported in Multiblock simulation. Advantages/Disadvantages with Functional simulation Advantage with this method is that it can be verified that a function behaves, as it should, according to its Functional Description. This strategy facilitates to find errors early in development process. The disadvantages are the same as for the first strategy. Advantages/Disadvantages with Co-simulation The advantage of this approach is that it does not require so much time as the other two simulation strategies. A great disadvantage the tester has to send signals manually which is not accepted in today’s testing process. Another disadvantage is that all of Simulators runs their own MSC traces that make it hard to examine signal communication between different blocks. One last disadvantage is that one may find using different Simulators in parallel as difficult and hard to overview. The Multiblock simulation must be a part of today’s simulation process because of all advantages we referred to above. We presumed that those problems mentioned about operators in section Problems and Solutions below have been settled before it can be applicable in simulation process. However, if the problem with the operators cannot be managed, large part of SDL cannot be tested.

6.5.4 Proposals for improvements As we stated above there were some problems to conduct Multiblock simulation. In Conclusions, section 6.5.3 above we confirmed that the current implementation of some operators does not support Multiblock systems simulation. In this section, we are going to

Investigation of Testing Tools

57

assign some proposals to improve and facilitate Multiblock process. The areas that need improvements are mostly the following: • Operators used in SDL block design. Operators like Partner are not Multiblock compatible

because sending a signal from a block to their ‘partner’ individual process instances located in another block, without to know the actually process Id (PId).

• Adding new options for Ericsson Tau tool. The most desirable option is to automate setup of a Multiblock system by selecting the SDL blocks.

• Improve efficiency of Ericsson Tau tool. At least compile time of a Multiblock system shall be improved.

To develop or modify operators that are inevitable for Multiblock system is the main concern otherwise Multiblock simulation process will fail.

6.5.5 Problems and Solutions There is a need for adding new options in order to facilitate testing engineer for setting up a Multiblock system: 1. An essential option could be to provide a feature to build a Multiblock system

automatically by choosing the blocks that shall be added to the system. 2. Provide possibility to use same names on processes and procedures within different blocks

instead of renaming them. 3. Provide an option to automate merging all signal definitions packages into one package

and remove all redundant signal definitions. Another alternative is to generate signal packages from block list. Otherwise, it takes ages to remove them.

4. Provide an option to automate renaming process of all redundant data type definitions. It

is also necessary to introduce those changes in SDL system design. Another option is to declare data types specific for the block, in the block, and not in the package.

5. A desirable option is to manage the merging process of Multiblock synonym files into one

synonym file from different blocks. 6. A desirable option is to manage the merging process of Multiblock macro files into one

macro file from different blocks. The Ericsson Tau’s efficiency concerning Multiblock simulation needs to be improved in order to function with adequate manner. The following have to be improved: 1. Analyzing time. 2. The time to generate a make file. 3. Compiling time. 4. Time to generate index viewer file.

58

When the previously mentioned proposals are implemented, the Multiblock process will become a very powerful technique to deal with.

Investigation of Testing Tools

59

Different solutions

61

7 D I F F E R E N T S O L U T I O N S This chapter describes the different solutions for improving the current test method for function blocks developed in SDL, which is the main objective of this thesis. It is based on our study and the results that we obtained through investigating and evaluating different testing methods and tools. We present general testing processes for the testing methods used by different tools, which was described in chapter 6. 7.1 DIFFERENT TESTING STRATEGIES Using testing to locate and correct software defects can be an endless process. We can not test until all the defects are discovered and removed, it is simply impossible. Defects cannot be completely ruled out. Just as the complexity barrier indicates: chances are testing and fixing problems may not necessarily improve the quality and reliability of the software. Sometimes fixing a problem may introduce much more severe problems into the system. At some point, we have to stop testing and deliver the software. The question is when to stop testing. Testing is a trade-off between budget, time and quality. There are two approaches. The pessimistic approach is to stop testing whenever some, or any of the allocated resources time, budget, or test cases are exhausted. The optimistic approach is to stop testing when reliability meets the requirements. A model for tracking test execution status [Kit] can count the number of test cases in each of

the following categories: • Planned: Test cases that planned to be developed. • Available: Planned test cases that are available for execution. • Executed: Available test cases that have been executed. • Passed: Executed test cases whose most recent executions have detected no errors. If we plot these four numbers against the period of time (see figure 7.1) and analyze the trends we can decide to deliver the product when the passed to plan ratio is over 95 %.

Time - weeks

Num

ber

of te

st c

ases

Planned Available Executed Passed

10 5 15 20

50

100

150

200

Figure 7.1 Test Execution Graph

62

The issue is to improve the testing process either by lowering the testing time without affecting the quality or by increasing the quality with more effective testing methods. There are different testing strategies that could be applied: 1. Perform validation then complete4 simulation, and finally emulation. 2. Perform validation then Singleblock simulation and then Multiblock simulation 3. Perform Singleblock simulation and emulation. 4. Perform Multiblock simulation only. The testing activities proposed in the first testing strategy are shown in the Figure 7.2 above.

The process includes a number of blocks named from A to N (A, B, C,..,N). Each block is implemented by two SDL designers and is tested by a tester who takes care of validation and Singleblock simulation processes. The Multiblock process is conducted by one experienced

4 By the complete simulation we mean performing both Singleblock simulation and Multiblock simulation.

Block A

Block N

Multiblock process

Time

Sim Preparation Struct/Skel

Block 1.1 Design

Sim 1.1a

Block 1.n Design+Corr

Iteration 1

Figure 7.2 Incremental SDL Design Testing Strategy 1

Vali-dation

1.1

Block 1.1a Design+Corr

Sim 1.na+

reg 1.1

Vali-dation 1.n+ reg1.1

Block 1.1na Design+Corr

Corr

Multi-blockSim

1.1a Block A - N

Sim+ Emu

Corr+ reg

Sim Preparation Struct/Skel

Block 1.1

Design

Sim 1.1a

Block 1.n

Design+Corr

Vali-dation

1.1

Block 1.1a

Design +Corr

Sim 1.na+

reg 1.1

Vali-dation 1.n+ reg1.1

Block 1.1na Design+Corr

Corr

Sim+ Emu

Corr+ reg

Multiblock System A - N script

Preparation Struct/Skel

Multi-blockSim

1.na+reg1.1 Block A - N

Different solutions

63

tester. In the figure the time perspective is taken into account when activities are accomplished. The first testing activity, validation, is conducted when the first drop (1.1 in figure 7.2) is ready. This test is going to disclose any possible deadlock and livelock. The time to validate a drop is slightly short since no script preparation is required. On the other hand the quality is enhanced by detecting errors that are not possible to find using i.e. only simulation. The errors found during validation are corrected during the second drop (1.1a in figure7.2) by SDL designers. Next the Singleblock simulation is conducted and then regression test is follows to ensure that the software is error free. In the end Multiblock simulation is conducted on all blocks that are ready. The Multiblock tester has prepared the script files and starts to test blocks after the drop (1.1a) is corrected. When the block is ready, it’s transformed into PLEX-C block by the SDL designers. At PLEX-C level the blocks are then tested further. Emulation tests some procedures that are not possible to simulate or validate. The second strategy is suggested a process in the same manner as the first strategy without the emulation process. The process during second strategy is depicted in Figure 7.3.

Multiblock process

Block A

Block N

Time

Sim Preparation Struct/Skel

Block 1.1

Design

Sim 1.1a

Block 1.n

Design+Corr

Iteration 1

Figure 7.3 Incremental SDL Design Testing Strategy 2

Vali-dation

1.1

Block 1.1a

Design+Corr

Sim 1.na+

reg 1.1

Vali-dation 1.n+ reg1.1

Block 1.1na Design+Corr

Corr

Multi-blockSim

1.1a Block A - N

Sim Corr+

reg 1.1-1.na

Sim Preparation Struct/Skel

Block 1.1

Design

Sim 1.1a

Block 1.n

Design+Corr

Vali-dation

1.1

Block 1.1a

Design +Corr

Sim 1.na+

reg 1.1

Vali-dation 1.n+ reg1.1

Block 1.1na

Design +Corr

Corr

Sim Corr+

reg 1.1-1.na

Multiblock System A - N script

Preparation Struct/Skel

Multi-blockSim

1.na+reg1.1 Block A - N

64

The staffing under this process is two SDL designers for implementation, one tester for the validation and simulation processes, and one experienced tester for the Multiblock simulation. The third strategy (see Figure 7.4) lacks validation and Multiblock simulation processes. The SDL blocks are implemented by teams containing two SDL designers and one tester. The drawbacks of this strategy are that the livelocks and deadlocks are not revealed since validation is not performed and signal interchange errors between different SDL blocks are not discovered as well, because Multiblock simulation is left out.

The forth strategy can be seen in Figure 7.5. The staffing consists of two SDL designers for the implementation and one Multiblock tester. This strategy suggests solely Multiblock simulation process to test SDL block implementation. This approach is seemingly better regarding testing time then all other approaches but this can affect the quality by not finding all errors that are normally found by validation and Singleblock simulation.

Block A

Block N

Time

Sim Preparation Struct/Skel

Sim 1.1

Block 1.1

Design

Sim 1.2+

reg1.1

Corr

Sim 1.n+ reg

1.1-1.2

Sim Emu

Corr+ reg

Block 1.2

Design+Corr

Block 1.n

Design+Corr

Sim Preparation Struct/Skel

Sim 1.1

Block 1.1

Design

Sim 1.2+

reg1.1

Corr

Sim 1.n+ reg

1.1-1.2

Sim Emu

Corr+ reg

Block 1.2

Design+Corr

Block 1.n

Design+Corr

Iteration 1

Figure 7.4 Incremental SDL Design Testing Strategy 3

Different solutions

65

Block N

Block A

Multiblock process

Time

Block 1.1 Design

Block 1.n Design +Corr

Iteration 1

Figure 7.5 Incremental SDL Design Testing Strategy 4

Corr

Multi-blockSim

1.1 Block A - N

Block 1.1 Design

Block 1.n Design +Corr

Corr

Multiblock System A - N script

Preparation Struct/Skel

Multi-blockSim

1.n+reg1.1 Block A - N

7.1.1 Comparison of the different strategies The first strategy is not desirable because it will take long time to test meanwhile increase the quality by detecting most errors. The second strategy is more time effective comparing with the first approach since emulation process is left out. The meaning with this exclusion is that the implementations of the procedures like Forlopp Release and Size Alteration are standardized and seldom contain errors. The second approach is also more cost efficient in form of discovering errors as early as possible. The third testing strategy is the present testing strategy, is time effective but the quality level is lower comparing with the other two above by omitting validation and Multiblock simulation.

66

The forth testing strategy is the most time effective but contains the most drawbacks as well. The absence of Singleblock simulation, validation, and emulation processes is causing error leakage to the next development phase. According to our experience with testing methods we propose the using of the second testing strategy and we are going to unfold our arguments in chapter 9. 7.2 GENERAL P ROCESSES In this section we are going to present different testing processes for Singleblock simulation, Multiblock simulation and Validation. Each phase of the process will be described in detail.

7.2.1 Singleblock simulation process The simulation should start by simulating the normal branches (the main flow) and then continue by simulating less frequently used branches (error and exception branches). Each individual simulation covers one branch. To reach desirable system coverage the tester has to specify different branches to execute. Before any simulation can take place the system designer or tester has to identify all signals sending from/to the testing object. With testing object, we mean that it could be a feature, one block or many blocks that should be tested. In the current development process simulation is applied to a single block system. According to our Optimal Testing Strategy (see Chapter 9), the simulation strategy determines the method to follow depending on the testing object. The Figure 7.6 illustrates the simulation process used during Singleblock simulation. The phases are described separately in sections below

Different solutions

67

Initialization Once the SDL block system have been designed, analyzed and compiled, it is possible to simulate the system, i.e. to interactively inspect and check its actual behavior. During Initialization phase the Simulator executable file (*.sct) has to be prepared. Simulation Script Files are created simultaneously with SDL Block Design by system designers dedicated for simulation. Script files are also based on the following documents: • MSC Diagrams from Function Description Document. • SDL signal packages. • PLEX-C signal descriptions. • SDL type packages for signals. • SDL signal list packages. • SDL variables declarations for the different processes. The tester should understand which signal parameter values (if any) are acceptable and information is obtained from signal descriptions using SDLSIGTOOL.

Figure 7.6 Simulation process

STOP

Simulation Strategy

Verification

Coverage Viewer

Simulation

Initialization

Simulation Script Files

68

Simulation Simulations may be performed manually or automatically by using simulation script files. Typically, actions performed during simulation are stepping, setting breakpoints, examining the system, processes and variables, sending signals and tracing the execution, as you would do with a debugger, but applied on the SDL level. • Eventually define the external synonym file, which defines external parameters. • Executes simulator script files containing some test cases. A synonym file (“.syn”) has to be created before starting simulation in order to assign all External Values declared in SDL. This file includes all the Source Parameter List values of all the AXE blocks included in the SDL Model see an example of synonym file in Figure 7.7. Several synonym files containing different parameter values can be defined in order to have several possible configurations.

When a simulator is started, the static process instances in the system are created, but their initial transitions are not executed. The Simulator UI is now ready to accept commands to the monitor system. Whenever it is possible to enter a command, the prompt Command: is issued in the text area. In our case we first specify the location of Macro file with all signal parameter definitions. simulation script files can be executed one at a time. It is possible to invoke other simulation script files by using the monitor command Include-File. The simulation execution can be logged to a log file (simulation Log file) or in MSC trace. The next phase, Verification phase is using those log files as input to evaluate the test results. Verification In this phase, the outcome of simulation is evaluated and decision has to be taken based on: • Simulation Log file. • Interactive MSC trace file.

Figure 7.7 Synonym file

Different solutions

69

Those files have to be investigated if errors occur. This is used to be indicated by key word passed or failed in the end of each script execution in the simulation log file. A more graphical way to find out where in the SDL design the error was occurred is the MSC trace. From this phase, different decisions could be taken as shown in Figure 7.6. If the results were as the expected, which means that the results satisfy the requirement the decision is OK and the next phase is Stop. If the outcome of the simulation is not what the tester has been expected, it may happen that: • The SDL design error(s) should be corrected and simulation process has to run from Start

phase. • Simulation script files are incorrect i.e. wrong numbers of parameter or type etc. Coverage Viewer The Coverage Viewer is a graphical tool that shows how much of a system has been covered during a simulation in the terms of executed transitions or symbols. By checking the system coverage, you can see what parts of the system that have not been executed in the simulation so far. Saving a coverage file for each test case execution can check this. This file can tell if: • The desirable coverage for simulation is not reached. Some new simulation script files has

to be written or existing scripts have to be modified in order to increase symbol or transition coverage. simulation process has to run from Start phase.

• The desirable coverage is reached. Next phase is Stop. Stop The simulation is finished on the following criteria: • The wished coverage is reached. • No errors are found. • Time offset for simulation is expired.

7.2.2 Multiblock simulation process The following sections are a description of the process during Multiblock system simulation. This process is developed as a reflection of our work during setup and simulation of a Multiblock system. All phases illustrated in Figure 7.8 Multiblock process have been obtained by testing different methods until we found the most functional one, with respect to order between phases. The order between phases must be followed. There are other processes to conduct Multiblock simulation. Those processes have not been applied in our work but we mention them briefly in our paper. The following alternative processes could be applied during Multiblock simulation: • Incremental Multiblock system simulation. • Spiral Multiblock system simulation.

70

Incremental Multiblock system simulation In this process the activities (by the activities we mean those depicted in Figure 7.8 Multiblock process) during Multiblock simulation are divided into three increments. In the first increment, the Multiblock system is prepared for simulation. In the second increment, the chosen simulation strategy is applied and script modification is conducted. In the third increment the execution of script files is completed, the results are reviewed and corrective actions are taken if needed. The first and second increment should be executed in parallel in order to save time. The advantage of this process is that some work could be done in parallel. Spiral Multiblock system simulation In this process, the activities are the same as in the process depicted in Figure 7.8 Multiblock Process. The accomplishment of these activities occurs in a looping order i.e. when some parts of the system are tested directly after the implementation. The advantage of this process is that errors can be found very early and quickly in development of the system. Correcting errors early in development will not propagate errors to the next phases. This saves money and time budgets. The disadvantage of this process is that it is hard to test small parts in telecommunications systems due to great dependencies to other subsystems. Spiral process requires a careful planning but is anyway better than incremental process when an accurate plan is in place. If SDL design should be changed in any way, the regression test should be performed to ensure that the system functionality is retained.

Different solutions

71

The input to Multiblock simulation process, simulation strategy, is determined in our optimal testing strategy. It depends on what kind of test it needs to be performed and in what extend. For more details see chapter 9 section 9.2 on how simulation strategy is determined. We assume that at this point it is determined what strategy has been chosen. In the sections below we dealing with three different Multiblock simulation strategies that can be applied on a Multiblock system which are: 1. Interaction simulation 2. Function simulation 3. Co-simulation

Simulation Strategy

Simulation Script Files

Setup

Simulation

Verification

Stop

OK

Modify script files Correct the SDL block design Regression Test

Input

Figure 7.8 Multiblock Process

72

1. Interaction Simulation During the first simulation strategy “Interaction simulation”, tests the interaction of signals between the internal blocks of a Multiblock system is tested. Prerequisite To be able to perform this simulation strategy it needs: •• Setup phase completed as described in Setup. •• Simulations file (gcc-Simulation compiler to be preferred). •• Simulation script files.

What to test? With this strategy, we investigate that signals between different blocks are correct interchanged. This strategy ensures to test that signals are received in the desirable block and not just send to environment, as it is the case during Singleblock simulation. It also checks that signals have the right parameter values in the receiving block. When to use? This strategy should be used after completed Singleblock simulation and validation and before the system is delivered for function test. The motivation on why to apply this strategy after completed Singleblock simulation and validation is the following: with this strategy we are searching for errors in interfaces between different blocks. We have to be sure that everything else in the block functions correctly and conducting a Singleblock simulation and validation before Multiblock simulation can ensure this. Which errors are found? The errors that are revealed when this strategy is applied are typical interface errors. A typical example is that a signal that has been received by another block is either incorrectly received i.e. has been lost or sends to another process in the block. These kinds of errors are impossible to discover during Singleblock simulation. This means that these errors will leak to the next development phase if only Singleblock simulation was used. It is desirable to find errors as early as possible in development process and with this strategy, we achieve this objective. 2. Function Simulation The second simulation strategy “Function simulation” is applied to test if a function that extends over several blocks is corresponded with its requirements. Prerequisite •• Setup phase must be completed as described in Setup. •• If function consists of blocks that are not yet implemented in SDL, then functionality can

be replaced by script with the needed information. •• The simulation file for Multiblock system. •• Simulation script files.

Different solutions

73

What to test? With this strategy, we investigate if any arbitrary function behaves as specified in Function Description document. When to use? This strategy should be used after validation and before the system is delivered for function test. The motivation on why to apply this strategy after completed validation is that: we want to know that the system does not contain deadlock or Livelock and all signals are consumed correctly. Which errors are found? With this strategy, we found if a function’s behavior deviates from its Function Description. We try to prevent errors to leak into function test. The errors that we are searching for in this strategy are all signal sending and receiving errors i.e. if signals are sent in the right order and to the right process correspondent to requirements in function description. That all signal parameter values are correct correspondent to function description. Consequently, by finding these errors unnecessary corrections in PLEX or ASA code is minimized. 3. Co-Simulation The third simulation strategy “Co-Simulation” is applied to test individual signal reception between blocks. It is a manually simulation without using simulation script files. Prerequisite To perform this method, simulator files (i.e. name.sct) for different blocks are required as input. What to test? With this strategy, we investigate any arbitrary signal communication between blocks in particular signal consumption in the blocks. When to use? This strategy should be used after Singleblock simulation and validation, instead of Interaction Simulation if not so much time is available to set up a Multiblock system and before the system is delivered for function test. The motivation to why apply Co-Simulation after Singleblock simulation and validation is the same as for Interaction Simulation. Which errors are found? This simulation strategy aims on finding the same errors as in the first simulation strategy Interaction simulation. Simulation Script files Common requisite for both Function and Interaction Simulation is the creation of simulation Script files. Those files are created from MSC diagrams that are included in Functional Description assumed that this document is approved. Function Description should be studied by the testing personnel to find any appropriate MSC diagram. Those MSC diagrams should be related to the blocks that are part of the current Multiblock system to be tested. Since those MSC diagrams are descriptive modifications is needed to obtain executable simulation script files. The signal parameters have to be added, since those are never included in any FD. The block(s) that are not included in the Multiblock system have to be replaced with environment.

74

Additionally, except MSC diagrams, script files are also based on the following documents: • SDL signal descriptions. • PLEX-C signal descriptions. • SDL type packages for signals. • SDL signal list packages. • SDL variables declarations for the different processes. Modifications to Simulator Script files are only necessary for Interaction simulation and Functional simulation. For Co-simulation this phase is omitted. The modifications and additions to simulation script files in order to be used during the Multiblock simulation are the following: • Add name qualifiers to the Processes and instance numbers. • Add name qualifiers to Macro definitions used in signal parameters. • Name of the type definitions. • Add significant variables to test their values. • State altering has to be added. • Replace the environment instance with respective Block/or Processes instance. When the simulator script files are modified, the simulation phase can take place. Setup The setup process is essential for the entire Multiblock simulation. To prepare a Multiblock system requires a lot of work, especially if blocks are designed without thinking that they could be a part of a Multiblock system to simulate. The time on the preparation can vary depending on: • How many blocks are to be included? • If the system is designed for purpose of being used during Multiblock simulation. • Experience of the designer on functionality of the blocks that are involved. The following description assumes that Ericsson Tau tool is used. A general setup process consists of the following steps: 1. Importing blocks. Create a new system file in Ericsson Tau organizer (i.e.

multiblocksystem.sdt) and then include the necessary blocks. The setup should be performed manually by adding the blocks to be included in the Multiblock system’s Block Group.

2. Connecting blocks. Connect the blocks, which was included in step 1 with the

environment and establish internal block connections between them. New channels have to be defined internally between blocks.

3. Renaming processes and procedures. Rename all processes and procedures that appear

more than one time with unique names. A good rule is to add name qualifier with the block name before the actual name of process or procedure name. The renaming of process and procedure has to be done for all affected SDL blocks throughout SDL system.

Different solutions

75

4. Renaming signals and routes. Rename all signal routes and signal list, which have the

same name with a name qualifier as we describe in step 3 above. 5. Creating new signals routes. Create new signal routes, signal list and multidimensional

signal list (i.e. a multidimensional signal list includes other signal list) between the internal blocks of a Multiblock system.

6. Remove redundant packages. Remove all standard packages (e.g. Plextypes.sun) that

appear more then once in the organizer. 7. Create and import Multiblock signal package. Create a new package for signal

definitions for the entire Multiblock system. The old signal packages imported from different blocks have to be removed from respective SDL blocks. This new package has to be imported on the highest level i.e. on system level (e.g. by using Use packagename.sun clause).

8. Temporary signal definitions. Create a new package for temporary signal definitions (if

any) for the entire Multiblock system and remove all other packages with temporary signal definitions. Then import the new package as described in step 7.

9. Type definitions. The packages that contain type definitions should not contain duplicate

type definitions. 10. Synonym file. The synonym file has to be modified with definitions from all blocks that

are joined in the Multiblock system. If steps from 1 to 10 are followed then the Multiblock system will be analyzed and compiled successfully without any errors. Simulator file will be created and the Multiblock system will be ready for simulation.

Simulation The work effort during simulation phase is automated by the Simulator. A good simulation process is: • The tester runs the Macro file with all macro definitions of the Multiblock system. • Executes simulator script files that contain either interactions between different blocks

(Interaction simulation) or functionality testing (Function simulation). The work effort in Co-Simulation differs in that the tester performs a manually simulation. There is no need of simulation script files because all signals and values control is performed manually. The only reason to use this method is to test few arbitrary stray signals without to setup a Multiblock system and prepare simulation script files belonging to it. The output of simulation phase is MSC traces, coverage, and log files.

Note that we compiled with GNU’s gcc-Simulator compiler on our SUN UltraSPARC workstations!

76

Verification In Verification phase, the outcome of simulation is evaluated and decision has to be taken based on the outcome. The evaluation is based on the following items: • Simulation log file. • Interactive MSC trace file. • Coverage files. Those files are generated by the Simulator tool i.e. Simulator as the result of simulation phase. For example the results of expected-output-begin or if the test case is passed or failed are recorded in simulation log file. Those files help the test expert to locate where in the SDL flow the error has been occurred. There are different decisions that can be taken as shown in Figure 7.8 Multiblock Process. If the results were as the expected, which means that the results satisfy the requirement the decision is OK and the next phase is Stop. If the outcome of the simulation is not what the tester has been expected, it may happen that: • The SDL design contains some error(s) that should be corrected. • The desirable coverage for simulation is not reached. Some new simulation script files has

to be written or existing scripts have to be modified in order to increase symbol or transition coverage.

• Simulation script files are incorrect.

Errors that are found after running simulator script files have to be removed from SDL design. After all errors have been removed, a regression test is performed. Regarding the code coverage, this is up to the actual policy for the Multiblock that is applied and is decided by software project coordinators. Stop The Multiblock simulation is finished on the following criteria: • The wished coverage is reached. • No errors are found. • Time offset for simulation is expired.

7.2.3 Validation process The following is a description of the process during validation of an SDL system as it shows in Figure 7.9 below. The Validation Process should be applied incrementally for each block and then for the whole system. If SDL design should be changed in any way, the regression test should be performed to ensure that the system functionality is retained. The Validation Process depends from what is to be validated. The choice of the right methods depends on the size of the project. When selecting the appropriate method for Validation of

Different solutions

77

the system, the following should be considered: • How large is the system? • What kind of faults to detect? • How large is the amount of time and money available? In the Optimal Testing Strategy the Validation Strategy is described in chapter 9 section 9.2.4. It is necessary to define methods what to Validate, when, and how.

The methods, which are provided by the tool for use during the Validation, are: • SDL Block system validation using an arbitrary algorithm for state space exploration

mentioned in section above. • MSC Verification is used to verify requirements against the SDL system design. The output of Optimal Testing Strategy is the choice of the appropriate Validation strategy to be applied.

START

Initialization

Execution

Report Viewer

Coverage Viewer

STOP

Validation Strategy

Figure 7.9 Validation Process

Metrics

Correct faults

78

Start To conduct validation a Validator file is required. This is obtained by analyze and compile the system. The product of the compilation is an executable file in format: system_name.val. The output of this phase is a Validator file and a ready Validator waiting for further commands. Initialization Some modifications have to be made on the system space in this phase depends on the chosen validation method: • Define the values of the signal parameters in the file: name.com. Due to the large size of

the telecommunication systems some of the default options has to be modified, those are state size and transition length, how to do that see section: Appendices D Validator section Monitor Commands.

• Define user rules to be check. The Validator checks a number of predefined rules (shown

in Table 6.1.) for each system state traversed by default. Further rules can be defined by using “Define-Rule-<name>”. The Validator checks if a rule has been violated and a report is generated accordingly. Several rules can be combined with each other using logical operators.

• The constraint has to be defined by the tester during this phase in case of automatic test

case generation using Autolink tool. The constraints are used to delimit signal data. • Define the Start State if the Validation is not desirable to start from the predefined System

State. This is done by the Navigator tool. This tool makes it possible to navigate to any state of the system.

As mentioned above the input to this phase could be also a block that has been corrected from errors. This block has to be validated again. The output of this phase is a ready to run Validator. Execution The Validator is now configured to execute the chosen validation method with the desired constrains. A configuration can be saved in a file (.valinit in UNIX) for later usage. To execute the Validation, a command depending on the Validation strategy has to be entered in command line or click the respective button in Explore module in the Validator tool: • Bit-State. • Random-Walk. • Exhaustive-Exploration. • Power-Walk. • Tree-Walk. • Tree-Search. • Verify-MSC. The output of this phase is generation of reports and eventually an MSC trace.

Different solutions

79

Report Viewer The Report Viewer is a tool where generated reports are shown after the validation of the system. When the Validator is finished validating the system, the following data are displayed: • Number of reports • Generated states • Minimum depth • Symbol Coverage

Reports can be inspected in the Report Viewer. The viewer starts automatically after a completed validation, for all algorithms except Tree-Walk. In the Report Viewer, the rules (see Table 6.1.), which has been violated are presented with description of error type. One can get more information about specific violation, by clicking inside the box containing the description. Reporter Viewer then provides the MSC-trace and textual information. Coverage Viewer The Coverage Viewer is a tool that checks what symbol or transition coverage has been reached by the Validator. This tool is used to identify paths taken during the validation through the system states and the symbols executed. The coverage is determined by the number of hash collisions. The number of hash collisions, however, depends critically on the ratio of the size of the hash-table,

The number of bits in the state space, and the number of states that is stored. We call this factor the hash factor. It is calculated by the Validator after each run as the size of the hash-table divided by the number of states stored. A high number (more than 100) correlates with good coverage. Low numbers (near 1) imply poor coverage ( [Holzman], Chapter 13.9).

There are two options in Coverage Viewer that can be chosen whether to present the coverage of transitions or symbols in a transition tree. If there are any states that are not covered, then manually navigation can be used to go to these states and validation can continue from there. The output of this tool is knowledge about the current coverage. Metrics The input to this phase is the reports generated after validation by Report Viewer. In the metric phase the following data can be examined: • Number of errors • Type of errors • Coverage • Time data The output of this phase is a decision based on the data collected by the Report Viewer and the Coverage Viewer. The results can help to decide if another validating strategy should be applied or stop testing.

80

Stop The Validator exits when the following conditions are meet: • Specified time exceeds (specified in minutes). • No errors are found. • The symbol coverage (specified in percent) is achieved. • The desired depth is reached. • The desirable reliability has been reached. The specified number of repetitions is reached. Evaluation of the validation process The objective we pursue with our work is to evaluate the validation process. This was tested on two SDL blocks. Our evaluation process includes: • Try to validate an SDL block applying methods provided by the Validator tool. • Present the obtained results. • Show the conclusions based on results. • Present some proposals to improve the Validator tool. The main questions when we evaluated Validator was: • What can be won in terms of testing time? • Is it more effective to detect faults in the early phases of development? • Where in the development process can Validator be used? • How Validator and Simulator can be combined to achieve higher product quality? • What kinds of defects can Validator detect that Simulator cannot? • Can the Validator replace the Simulator? • Is Validator tool easy to use?

APZ Emulator and Mediator

81

8 A P Z E M U L A T O R A N D M E D I A T O R As we mentioned in the chapter 6 section 6.4 the main purpose of using the Mediator tool is to facilitate the transformation of BTI script files into the simulation script files. This is a part of a reverse engineering process when an old PLEX-C coded block is transformed to SDL. In the Figure 8.1 below the reverse engineering process and simulation process are shown.

The point is to win development time when the transformed SDL block is to be simulated by using Mediator tool to automatically generate simulation script files. When Mediator tool generates MSC diagrams it has to satisfy the necessary conditions in order to produce correct simulation script files. “Correct” simulation scripts files are when: • The simulation script commands should be generated syntactically correct. • The values of the variables and signal’s parameters should correspond to those referred in

BTI –script. • Ensure that the signals are “correctly” received and match those from BTI-script. • Ensure that the signals to/from the environment and to/from the current block are included

in the simulation script files. All other instances have to be replaced by the environment.

Each simulation script file has to correspond to each test case in BTI script. The macro definitions have also to be transformed to simulation Script file macro appropriated for the Simulator. To run the APZ Emulator a dump file is necessary for emulation. It can be produced with EmuDumpGen tool. This tool is used to create emulator dumps and PlexView files (*.pv). A Basic Test Instruction script file is needed for emulation. BTI script use to be written by Unit testers under unit design and basic test phase (UDBT). Using Emulator and Mediator This is a method not used in the current process. It is a combined emulation method with the Mediator’s graphical interface connected to APZ Emulator tool. The emudump file and eventually BTI-script are needed.

Block A PLEX-C

Reverse PLEX2SDL

Block A SDL

BTI + EMULATOR

Mediator MSCs Sim.

Simulation

Figure 8.1 Reverse engineering and simulation

82

This method can be implemented in two different approaches depending on when they are applied: • The first one is applied when the designer wants to test the functionality of the system

without BTI-script. In this case, the Mediator tool acts as graphical interface to the APZ Emulator, which is more convenient to work with. Commands can be issued in Mediator by pressing buttons instead of writing commands in textual mode.

• The second one is applied when the designer wants to test the functionality of the system

with BTI-script file and trace the execution in an MSC diagram. The benefit to have an MSC diagram is to generate simulation script files or just a convenient graphic trace of the execution.

The comparison and evaluation of the above methods is described in next section. Results The method was initialized by filtering away those block instances that are irrelevant to our evaluation. This is necessary in order to generate an MSC that is syntactical correct for generating simulation script files. This could be done with command button “MSC Filter”. In the dialog window the following options can be chosen: • Filter instances by exclusion. Ignores the specified blocks in MSC trace. • Filter instances by inclusion. Includes the specified blocks in MSC trace. • Remove instance filter. Remove the earlier specified filter(s). • Use other instances. Uses other, instances not included in the current emudump. In our case we selected “Filter instances by exclusion” to exclude the blocks that was not needed in MSC trace. “Filter instances by inclusion” were selected the following blocks: ENVIRONMENT and our block

After completed preparations the actual results obtained during the evaluation were: • Mediator could initiate, connect, and record activities from APZ Emulator. • A semantic almost “correct” MSC-diagram could be produced. See Listing 8.1 (e.g. set-

source #SDTREF(MSC,/home/edjohan/EmulatorTrace3.msc,129) is not needed) • The simulator script file produced from those MSCs had not always the right values

corresponding to BTI script i. e. transforming hexadecimal values were not correct. • The macro declarations in BTI scripts were not transformed into simulator macro script

file and not even “add-macro” command was included. • The simulator script file generated from MSC trace was just a big bundle containing all

test cases in the BTI-script (i.e. many BTI test case ⇔ one large simulator script file).

APZ Emulator and Mediator

83

• The instance name “Environment” is not displayed in MSC diagram when using “Filter instances by inclusion”. Instead “OtherInstances” is displayed which results a wrong MSC diagram and consequently empty simulation script files.

• The BTI test instruction Restore and Save are not transformed into simulator script file

(the simulator does not support those functions). • The BTI test instruction Expected IO is not transformed into simulator script file. • The BTI test instruction Expected Sig is not transformed into simulator script file if those

signals do not have any block that receives them. • The concept of breakpoint in BTI-script are missing, they are not needed in the same way

as for simulator. • All signals with no receiving block (i.e. “not yet consumed”) are excluded from the

simulation script file.

To create simulator script files manually takes 2 hours on each test case. Our testing block contains 383 test cases in BTI script. Theoretically it would take 2*383=766 hours. To see how a Mediator generated MSC diagram looks like, we chose an arbitrary test case shown in the Figure 8.2. Then from this MSC diagram we generated a simulator script file and which can be shown in Listing 8.1.

84

OtherInstances RNLC

GIVEFS

(298,0)GIVEFSEND

(1, 1, 1, 1, 1, 512, 65535, 0) GIVEFS

(298,0)GIVEFSEND

(1, 1, 1, 1, 1, 512, 65535, 0) GIVEFS

(298,0)GIVEFSEND

(1, 1, 1, 1, 1, 512, 65535, 0)

CONTFS

(298, 0, 10) CONTFSEND

GIVEFS

(298,0)GIVEFSEND

(1, 1, 1, 1, 1, 512, 65535, 0) SETFS

SETFSEND

(298, 0, 10)

GIVEFS

(298,0)GIVEFSEND

(1, 1, 1, 1, 1, 512, 65535, 0)

Figure 8.2 MSC generated by Mediator

APZ Emulator and Mediator

85

Listing 8.1 Simulator Script file

set-source #SDTREF(MSC,/home/edjohan/EmulatorTrace3.msc(1),106(2,2)) set-trace - 1 /* Textual trace for signals to and from environment */ break-output - env - - - - - - /* break when signals are sent to the environment */ set-source #SDTREF(MSC,/home/edjohan/EmulatorTrace3.msc,111) output-to GIVEFS (298 0) RNLC set-source #SDTREF(MSC,/home/edjohan/EmulatorTrace3.msc,114) check go expected-output-begin * OUTPUT of GIVEFSEND to env:1 from *RNLC* * Parameter(s) : 1 1 1 1 1 512 65535 0 expected-output-end set-source #SDTREF(MSC,/home/edjohan/EmulatorTrace3.msc,117) output-to GIVEFS (298 0) RNLC set-source #SDTREF(MSC,/home/edjohan/EmulatorTrace3.msc,120) check go expected-output-begin * OUTPUT of GIVEFSEND to env:1 from *RNLC* * Parameter(s) : 1 1 1 1 1 512 65535 0 expected-output-end set-source #SDTREF(MSC,/home/edjohan/EmulatorTrace3.msc,123) output-to GIVEFS (298 0) RNLC set-source #SDTREF(MSC,/home/edjohan/EmulatorTrace3.msc,126) check go expected-output-begin * OUTPUT of GIVEFSEND to env:1 from *RNLC* * Parameter(s) : 1 1 1 1 1 512 65535 0 expected-output-end set-source #SDTREF(MSC,/home/edjohan/EmulatorTrace3.msc,129) output-to CONTFS (298 0 10) RNLC set-source #SDTREF(MSC,/home/edjohan/EmulatorTrace3.msc,132) check go expected-output-begin * OUTPUT of CONTFSEND to env:1 from *RNLC* * Parameter(s) : 0 expected-output-end set-source #SDTREF(MSC,/home/edjohan/EmulatorTrace3.msc,135) output-to GIVEFS (298 0) RNLC set-source #SDTREF(MSC,/home/edjohan/EmulatorTrace3.msc,138) check go expected-output-begin * OUTPUT of GIVEFSEND to env:1 from *RNLC* * Parameter(s) : 1 1 1 1 1 512 65535 0 expected-output-end set-source #SDTREF(MSC,/home/edjohan/EmulatorTrace3.msc,141) output-to SETFS (298 0 10) RNLC set-source #SDTREF(MSC,/home/edjohan/EmulatorTrace3.msc,144) check go expected-output-begin * OUTPUT of SETFSEND to env:1 from *RNLC* expected-output-end set-source #SDTREF(MSC,/home/edjohan/EmulatorTrace3.msc,147) output-to GIVEFS (298 0) RNLC set-source #SDTREF(MSC,/home/edjohan/EmulatorTrace3.msc,150) check go expected-output-begin * OUTPUT of GIVEFSEND to env:1 from *RNLC* * Parameter(s) : 1 1 1 1 1 512 65535 0 expected-output-end

86

8.1.1 Problems and solutions 1. The menu to activate Mediator is not intuitive for the user. The invitation to start Mediator

with “Connect Emulator” in the “Local tools” in Organizer is not coherent. Solution: Change the menu “Connect Emulator” to “Mediator UI”.

2. During the execution of any test cases in BTI Driver, both Mediator and APZ Emulator are suspended.

Solution: To avoid this problem the following action must be taken in the right order: • From BTI Driver start APZ Emulator. • From Ericsson Tau start Mediator, by choosing “Connect emulator” in “Local tools”

menu from Organizer. Mediator then connects to APZ Emulator. • When connection is done, the APZ Emulator must be terminated. • From BTI Driver start over the APZ Emulator. • From the Mediator click on the “Connect EMU” button. • Then choose arbitrary number of test cases.

3. The test instruction “Expected Sig” in BTI Driver is not interpreted correctly by Mediator. This results in that MSC diagram displays those signals as “not yet consumed” which is not desirable behavior. Hypothesis: Mediator has no direct contact with BTI Driver. It is BTI Driver that decides what action to be taking after that expected signal has been received. Those decisions are not reachable by Mediator and consequently Mediator interprets those signals as having no receiver.

4. The instance name “Environment” is not displayed in MSC diagram when using “Filter instances by inclusion” where ‘Environment’ is a desirable name for all other instances. Instead, “OtherInstances” is displayed. This results in an empty simulator script file. Solution: When MSC trace file is generated, rename “OtherInstances” to “Environment”. Other wise the generation of simulation script files will fail.

5. Mediator does not register the variable values that are assigned in the beginning of each

test case in BTI Driver. This results on incomplete simulator script files. 6. Mediator does not register the checks of the variable values. This results on incomplete

simulator script files. 7. During the execution of test cases through BTI Driver, Mediator can not separate different

test cases. This results in one big script file, which is not the desirable result. One test case has to correspond to one simulator script file. Solution1: Choosing manually one test case from BTI script at the time gives exactly one simulator script file. However, this solution is inefficient and time consuming for the test engineer. A more effective solution to this problem could be found by studying both the BTI script and simulation files. Each test case in BTI script starts with “:h3” tag following by test

APZ Emulator and Mediator

87

case name. The only exception on this is the “start” test case which starts with “:h2” tag. Solution2: Could be to break parsing BTI script file when one “:h3” tag line appears and proceed with saving a separate simulation file or MSC trace. In this way proceed parsing until the desirable number of tests are parsed or to the end of the BTI script file.

8. The macro definitions in BTI script are not transformed into macro definitions for

simulator script files. Solution: • All lines in macro definition simulator file have to start with ”Add-Macro” following

by Macros name and its value. Eventually contain some comments originated from BTI script file.

• Parsing should search for lines starting with “$” following by the macros name and its value.

• Parsing should look for lines starting with “!”, which means a comment line, and append to corresponding simulator file.

• All other lines should be ignored.

9. The concept of processes and process instances are not supported in PLEX-C, therefore the addition to simulation script file are needed with process names and instances.

10. Restore and Save BTI instructions are not correspond to any Simulator test instructions.

Solution: To achieve such functionality in Simulator, script files depending on each other have to be included with simulator instruction: Execute-input-script, and executed in the beginning of each script file (impractical solution).

8.1.2 Conclusions Mediator was considered to automate the production of simulator script files from the existing BTI scripts. The results obtained from the transformation were quite different from the desirable outcome. From the results above, we can make some conclusions: • Mediator tool is not able to produce correct simulator script files. • Mediator tool needs to be modified in order to handle the transformation of BTI scripts

into the simulation script file. • Mediator tool needs to be connected with BTI Driver in order to monitor BTI executions. General conclusion: If Mediator tool should work as it supposed to, large amount of time could be saved during the production of simulator script files.

8.1.3 Proposals The problems occurred during our evaluation of Mediator tool encountered many problems as described above. In Conclusions above we figured out that the current version of Mediator tool needs additional modification in order to produce complete MSC-trace and consequently

88

correct simulator script files. More specifically, requirements can be seen at Appendices E Mediator section. Simulator script files produced from MSC trace should need some additional modifications to meet the requirements accepted by Simulator. Such modifications are not so hard to implement and do not require so much time to develop. For example on such addition is: <process name: process instance, block name/ block name>. It is also desirable from the user point of view to decide if the simulator script file should be generated directly from BTI script with or without MSC-trace. If the user wish to generate MSC trace it can always be generated through Simulator UI. Another proposal is to change the start procedure of Mediator by moving from “Local tools” menu to “Tools|SDL|Mediator UI”.

Optimal Testing Strategy

89

9 O P T I M A L T E S T I N G S T R A T E G Y This chapter presents an example of a testing process the Optimal Testing Strategy for function blocks developed in SDL. It is based on our study and the results that we obtained through testing and evaluating different testing methods and tools. We describe the identified improvement areas in details followed by the description of our optimal testing process. 9.1 OVERVIEW The tools and methods that we have investigated in this paper are the following:

• Simulator, for simulation purposes of Singleblock systems, in use today. • Simulator, for simulation purposes of Multiblock systems, not in use today.

• Validator, for verification purposes of Singleblock systems, not in use today.

• Validator, for verification purposes of Multiblock systems, not in use today.

The involved designers have been interviewed in order to get their opinions about the tools in use and the methods applied. Some of the testing tools above are in use today by development teams in the testing process, such as Simulator. For the other testing tools that are not applied until today, we had to find out an appropriate process to use them, and put them into the right application areas. Concerning the methods, such as testing a Singleblock or a Multiblock system, we had to decide which was the most applicable for a particular development phase. Our goal is to include those testing methods that suit in the current development process model. Additionally, we argue about why those testing methods shall be applied in a specific development phase in the current development process followed by the Ericsson Radio Systems. The phases that are affected by our Optimal Testing Strategy (from now on abbreviated to OTS) process for the blocks designed in SDL are the following: 1. Function Design. 2. Block Design. 3. Software Unit Design and Basic Test. 4. Function Test. During the SDL 10 design is assumed that Block Design and Software Unit Design are joined to as one phase. A short description of affected phases by OTS follows. Function Design The Function Design is concerned with the question on what is to be implemented and not how. The output of this phase is Function Description document, which contains all information about the decomposition to subfunctions. Each subfunction’s functionality is

90

allocated to blocks. Information about interfaces between blocks is also described in this document. The objectives of this phase are: • To describe how the function works in a system context. • To divide the function into subfunctions. • Allocate blocks to subfunctions. Block Design, Unit Design and Basic Test Block Design is concerned with the question of how a function is to be implemented with SDL 10. This phase is the main implementation phase where the major components of the function, the SDL blocks are designed. The design of those SDL blocks is then tested separately and finally delivered it to next phase, Function Test. The outcome of this phase during the current process is hopefully, fault free PLEX-C code, which is ready to be tested on the target machine. Function Test The Function Test is usually performed on target machine or simulated environment after that the SDL code is transformed to PLEX-C. However, it is concerned that SDL code is as correct as possible before performing Function Test. The goal is to improve the quality of SDL code prior to Function Test. Questions answered by OTS We suggest some necessary additions or modifications concerning testing methods and tools. The additions that need to be included to respective development phase by OTS process will be described in details later on in this chapter. The order that the methods shall be applied will be identified. There are several questions that are going to be answered in this chapter: • Which testing method is the best for a particular development phase and why? • Do we need to plan the testing effort and why? • Who are going to design test cases, designer or tester and when? • What tools to use and what errors they are going to find? • In which order the testing tools should be applied and why? • What strategy has to be applied for Multiblock simulation? • When to design test cases for Singleblock and why? • When to design test cases for Multiblock and why? • When to prepare for Validation? • Does Optimal testing strategy improve the current testing process and how? We are going to describe our OTS based on the hypothesis that the testing tools used are improved according to our suggestions as we described in proposal sections for Validator and Simulator. In the sections below 9.2, we are going to describe about how our OTS process fits to the current development process phases.

Optimal Testing Strategy

91

9.2 OPTIMAL TESTING P ROCESS Our Optimal Testing Strategy for SDL block design is concentrated to describe the appropriate testing process that checks the correctness of an SDL block design and ensures that SDL code that is delivered to Functions Test contains considerable less errors. The main task is validation and verification of an SDL block design. Validation is to see if the product contains the specified features. Verification of the product is to see that the features works as defined in the specification. The time aspect is also important to argue about. Our testing strategy should be time saving when it is applied and more cost efficient in form of discovering errors as early as possible.

Start

Test Planning

Checklist

Sim. Script files Validation

Singleblock Simulation

Multiblock Simulation

Sim. Script files

Evaluation

OkSingle BlockSimulation

0

1 0

2 0

3 0

4 0

5 0

6 0

7 0

8 0

9 0

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

East

West

North

noyes

MultiblockSimulation

Evaluation

CoverageOk

no

Stop

CoverageOk

Evaluation

no

yes

yestest metrics

Figure 9.1 Optimal Testing Process The phase that the Optimal Testing Strategy is focusing on is the Block Design. It should be applied as a testing process for every SDL block design. We suggest the following testing processes as shown in Figure 9.1 Optimal Testing Process. This picture shows the process flow that we suggest to apply. Testing objects are SDL blocks. Each SDL block is assigned to a development group that consists of two SDL designers and one tester. The OTS process activities are as following:

92

1. Test Planning. 2. Checklist. 3. Simulation script files. 4. Validation. 5. Singleblock simulation. 6. Multiblock Simulation. 7. Test metrics. 8. Evaluation of the testing results. The workflow is in parallel within the actual iteration which is usually consists of many SDL blocks (see Figure 5.1 in chapter 5).

9.2.1 Test Planning To argue about the first activity, the Test Planning, it is considering that planning the testing effort is very important in order to obtain a better organizational structure i.e. who does what and when, having in mind budget and time limitations. The necessary information needed for Test Planning is: • Time and money budget for testing. • The Function Specification (FS) Document, which is approved. • The Function Description (FD) Document, which is approved.

The main task in this testing process is to create the Testing Plan Document (TPD). The objective of this document is to thoroughly understand the nature of the system to be tested and, through a systematic process, to provide a clear, comprehensive road map for efficient SDL block and function testing. Testing Plan Document should be written by the project coordinator(s) who have overall responsibility for the development project. The Testing Plan Document should cover the following issues: • Determine the functionality to be tested with a textual description. • Define what approach to apply during testing process. If only one SDL block is to be

tested, a Singleblock simulation should be performed. If there are more then one SDL block then first Singleblock simulation followed by Multiblock simulation should be used.

• Define the testing methods and tools to be used in the testing process, i.e. first Validator

then Simulator and then Emulator. • Who is testing? Staffing the planned tests with test personnel and delegate testing tasks. • Outline a schedule with testing activities that suits the resources available for testing.

With testing activities, we mean Singleblock simulation, Multiblock simulation, etc. • Anticipate risks that might affect the project schedule or the quality of software product

being developed and take action to avoid these risks.

Optimal Testing Strategy

93

The output is a complete Testing Plan Document (TPD).

9.2.2 Checklist The second activity, is to create a Checklist, to ensure that: • The simulation script files created are corresponded to MSC diagrams in Function

Description Document. • Guarantee that all items, which are generated during transformation from MSC diagrams

into simulation script files, would result in correct simulation script file. The purpose of Checklist is to give guidelines for the testing person who attempts to modify and add data to simulation script files. To ensure that the simulation script files are corresponded to MSC diagram this checklist should be used. Checklists focus the tester’s attention on areas that have caused problems in the past and assure that they should be avoided in the future. Checklist contains information on what should be added or modified into simulation script files. It should also facilitate the review of simulation script files. The review activity should be conducted by testing personnel with respect to FD. Checklist should include the items that should be checked when simulation script files are created i.e. the correctness of system states, signal names, the parameter names, the number of signal parameters, number of instances. As testing personnel are gaining experience the checklist are improved by adding new items.

9.2.3 Simulation Script Files To design and compile correct simulation script files is an important activity. It is easy to generate simulation script file skeleton from respective MSC diagrams that are included in Function Description (FD) Document. An assumption to be made here is that MSC diagrams in FD Document follows the standards described in chapter 6 section 6.3.5 Verification of MSCs and Appendix B MSC. This is very important in order to generate any correct simulation script file. To complete the simulation script file skeletons the tester has to check different documents mentioned in chapter 7 section 7.2.1 Initialization. Simulation script files includes monitor commands to Simulator tool to check signal sending and receiving, parameter values, timers, number of instances, states, etc. The Checklist document mentioned above should be used to facilitate the creation of simulation script Files.

94

9.2.4 Validation The input to this test is a drop of an SDL Block. This activity is performed by a tester. Validation process answers the question ‘Did we build the right system?’ which means to verify the system against the specified requirements and to validate the system’s consistency. The choice of the right methods depends on the size of the system to be tested. When selecting the appropriate method for validation of the system, the following should be considered: • How large is the system? • What kind of faults to detect? • How large is the amount of time and money available? The methods, which are provided by the tool for use during the validation, are: 1. Bit-State Exploration: the method is suitable for systems with reasonable complexity. 2. Random-Walk Exploration: the method is suitable for very large systems. 3. Exhaustive Exploration: the exhaustive method is suitable for very small systems since it

is a very time and memory consuming method. 4. Power-Walk Exploration: the method is suitable for small and large systems and gives a

good coverage of the SDL system. 5. Tree-Walk Exploration: the method is suitable for all kinds of systems. 6. Tree-Search Exploration: the method is not suitable for large systems. 7. MSC Verification is used to verify requirements against the SDL system design. Regarding the SDL block development in telecommunication sector the size of an SDL block is considered large. Consequently, the methods to avoid for validation are definitely the Exhaustive Exploration method and Tree-Search method due to system size and inefficiency of the algorithms. Another method to avoid is Power-Walk Exploration because it is not possible to define number of reports to generate and the number of Power-Walks. See chapter 6 section 6.3.8 Conclusions. The methods to try with caution are Tree-Walk Exploration because the time to reach a given coverage has to be specified which is hard to estimate. Bit-State Exploration is efficient to reach high symbol coverage but it is time ineffective algorithm. Bit-State Exploration could be used if the algorithm to explore the states is improved. See in chapter 6 section 6.3.8 Conclusions. Finally, the method that remains to use is Random-Walk Exploration. This method gives the desirable results in a reasonable time because of the effectiveness of exploration algorithm and that it is suitable for large systems. MSC verification Another method to use with Validator tool is MSC verification, which is to validate the SDL block against the Function Description. MSC verification verifies that the SDL block is corresponded to Function Description. The default method that the tool uses in order to verify an MSC diagram is Bit-State Exploration. The method should be applied as early as possible in the Validation strategy phase.

Optimal Testing Strategy

95

The fault that can be revealed during MSC verification is if the current SDL Block design is incomplete or contains errors i.e. signals are missing or sends to wrong process, incorrect number of parameter, and wrong parameter types are specified. Prerequisite To be able to perform this validation strategy it needs: •• Start phase completed as described in Chapter 7 section. •• Validation file.

What errors are found by Validation? The fault that a validation can reveal is mostly deadlocks, Livelocks, and signals exchange errors. By deadlock we mean the state of the system in which no further execution is possible due to all processes are waiting for a condition that can never be fulfilled. Livelock is when the system is such state that executing sequences are repeated infinitely without any progress. Those kinds of errors are impossible to find during simulation. Are there any disadvantages with Validation? The first disadvantage with Validation is when applying Random-Walk Exploration algorithm, the signal exchange occurs in the random order and this is not the desirable in telecommunication systems. Because all signal exchange must occur in a specific order, otherwise the signal will not be correct consumed by the receiving process. The type of error is Implicit Signal Consumption. Which means that a signal was sent to a process that was not able to handle (or save) the signal in the current state, so it was implicitly consumed. The second drawback with Validation is that the parameter values in signals are set by default to minimum or maximum values. This is not desirable behavior because some SDL flows which contains an if statement will never be executed. In order to reach good coverage the designer has to specify the signal parameter values manually.

9.2.5 Singleblock simulation Simulation allows designers to create a functional prototype of the system and emulate the target machine in order to test its execution path. The input is a verified SDL block, after performed validation, and simulation script files created earlier in BD/UD phase. Singleblock simulation is used to reveal internal errors in an SDL block. It should be used to test the design of an SDL block. It is essential to run Singleblock simulation first before running a Multiblock simulation. The advantages of running Singleblock simulation before Multiblock simulation are: • Singleblock simulation tests the internal execution paths and the internal logic of an SDL

block, meanwhile Multiblock simulation tests interfaces between blocks. • The complexity of the system to be tested is not that big comparing with Multiblock. • Error correction is easier and can be done by the designer himself.

96

• Errors are found in an early stage of testing process and can be corrected to avoid the propagation into the Multiblock system.

• The time to compile the system in order to rerun Singleblock simulation is considerable small comparing with Multiblock.

Singleblock simulation should be performed by a designer, whom has an adequate knowledge of the system to be tested. It is essential that the tester of an SDL block is not the same person who designs that block. That person might find errors that the block designer himself could never found on his own. The other reason is that designers are more subjective for their own design.

9.2.6 Multiblock simulation Meanwhile Multiblock simulation is able to reveal internal errors in an SDL block/blocks, interface errors between SDL blocks, and uncover errors in a function that extends over several SDL blocks. It is important to address Multiblock simulation to test personnel that have an overall knowledge of the entire system to be tested. Since the Multiblock simulation is a complex and laborious task, it is advisable to dedicate it to experienced test personnel. What kind of Multiblock process to apply? As we mention in chapter 6 section 6.5 Multiblock simulation there exist two different processes to apply. The following alternative processes can be applied during Multiblock simulation: • Incremental Multiblock system simulation. • Spiral Multiblock system simulation. Incremental Multiblock system simulation’s main benefit is that some development and testing activities can be run in parallel and this saves development time. However, the development activities within each increment have to be accurate planed, which is a drawback. During Spiral Multiblock system simulation, the system errors can be found very early and quickly in development. Advantage with the Spiral is that a function can be tested as small software units within the function. Development activities in Spiral model are not possible to performed in parallel and is difficult to delegate small piece of work to several teams, as the size of telecom development systems are very large. Consequently, the process that should be used is the Incremental Multiblock system simulation. The input at this stage is several validated and (Singleblock)simulated SDL blocks. There are three strategies to apply on the Multiblock system simulation, which are: 1. Interaction simulation 2. Function simulation 3. Co-simulation As we describe in chapter 6 section 6.5.1Co-simulation is not suitable for the current testing process. Interaction simulation is to be preferred for common interface test between several SDL blocks. Function simulation is used when all block that a function consists of are to be simulated together in order to test the function.

Optimal Testing Strategy

97

General proposal to create Multiblock kernel system A general proposal to facilitate the setting up of a Multiblock system is to create a Multiblock kernel system containing SDL blocks. Those SDL blocks are a part of a function. When a new SDL block is developed it can be included into this Multiblock kernel system. The advantage of this Multiblock kernel approach is to avoid setting up a new Multiblock system each time a new SDL block has been developed. The only work remaining is to connect the new developed SDL block with the existing Multiblock kernel and then analyze, compile and simulate.

9.2.7 Test Metrics Test Metrics are essential in order to evaluate the testing results, testing processes, and in most cases determine when to stop testing. Metrics are also very useful when planning the testing activities. It is possible to estimate testing efforts and resources by using previous project’s test and defect records. Estimates of anticipated defects along with actual defect counts, measures of structural complexity, measures of test coverage, measures of release readiness etc. are essential to the testing process. Collecting testing metrics is not always easy, but is a necessary aspect of software testing. The unfortunate part is that it is also the most frequently omitted testing practice. Test Metrics, is partially used in the present testing process already in form of coverage metrics. The metrics that we suggest to be collected are: • Defect metrics. • Test coverage metrics.

98

• Time vs. error found. The defect metrics determine the type of errors, severity, cost and time estimation on fixing detecting error, and the number of errors found during validation and simulation of a particular block/Multiblock system. Defects metrics are collected by test personnel and recorded in defect database. It is important for designers, testing personnel, and project co-ordinators to know where defects occur more frequently and prevent to repeat them again. Test coverage metrics is at best indicators of how much you have tested not how well you have tested. It is important to have one or more measure of how completely you have tested during SDL Block testing. The introduction of defect metrics and time vs. error found metrics should be used in order to record the type of the defects and where in the code they appeared in Defect Report document. Metrics are also very useful when test managers are planning the testing activities. Metrics can clarify and focus the understanding for testing. Moreover, when done properly, sequential measurements of quality attributes of products and processes can provide an effective foundation for initiating and managing process improvement activities [Florac]. In order to manage and control the testing results, we must measure it. In order to improve the testing results, we must measure it.

9.2.8 Evaluation of the testing results Test Metrics introduced in all testing activities, is the key factor for evaluation. A decision has to be taken, if the test should continue or it should be terminated. It is usually up to the project co-ordinators to determine the percentage of coverage. Another task should be to evaluate the entire SDL design process itself. By analyzing the Metrics, can project co-ordinators determine if it was the right process according to planed and if not, what are the deviated values? As automated testing tools mature many of the essential metrics will be collected unobtrusively while testing is conducted. In addition, the tools will make the analysis of the measures easier with custom analysis and reporting capabilities.

9.2.9 Conclusions Optimal Test Strategy has to brought up SDL block as error free as possible to Function Test. This can be done by running Multiblock simulation and particularly function simulation. As we describe in Multiblock strategy section earlier, this method assures that the entire function works as specified in Function Description Document.

Optimal Testing Strategy

99

Until the simulation supports testing Size Alteration and Forlopp Release Handling procedures, the EMUtool has to be used for testing those procedures. BTI scripts should be created just to test those procedures. Two important additions that we suggested above, Test planning, and Checklist are aimed to improve the understanding for test process and provide a structural pattern for the testing activities for testing designers. We are going to argue about why the proposed additions are significant for the current development process. Testing Plan Document is the main structured addition to this phase. The question is why to make a test plan? The main reason to make a Testing Plan Document is to provide an overview of the entire testing effort for project coordinators, and for the testing staff. It will be easier to develop test cases if approach, schedule and testing staff are defined. The Testing Plan Document will work as assurance that everything that was planed was tested and nothing was left out. The effort of creation a Testing Plan Document is a laborious task, but it will be easier for the project co-ordinators to create such plans later, as they gain experience and become aware about the benefits of plan. The plan is also ensure that the project’s budget and time delimitation are guarantied since the plan was made with respect to those parameters. In order to facilitate creation of Testing Plan Documents efficiently, automated planning tool support is desirable. Automated planning tools can help the test personnel by administrating testing activities i.e. test design, implementation of test cases and, quality metrics, etc. Testing personnel can also be able to evaluate the progress in testing process by measuring the code coverage for a given set of test. The Checklist Document improves the quality of simulation script files framework created at this phase. The testing personnel have extract all his experience from earlier failures and defect detection to the points contained in the checklist in order to improve the test case design. The main task that checklist should support is to assure the highest MSC flow coverage for the SDL block. Checklist is a good companion for simulation script files inspection with its structured contains which is easier to trace and read than the MSC diagrams or textual descriptions from Function Description.

Summary and conclusions

101

1 0 S U M M A R Y A N D C O N C L U S I O N S In this section, we give a short outline of the mentioned Optimal Testing strategy. We assumed that the major number of the development projects is dealt with new SDL development. To evaluate the proposed Optimal Testing Strategy a pilot study project should be run in order to verify and validate its reliability. The testing strategy that we suggested can be summarized in the following points below. Testing activities and document produced • Create a Testing Plan Document. • Generate the simulation script files • Design Checklist • Conduct inspection on simulation script files and Checklist.

Testing activities for Block Design • Modify simulation script files for Singleblock simulation. • Verify MSC diagrams and collect metrics. • Run Validation with Random-Walk Exploration algorithm and collect metrics. • Choose simulation strategy (Start with Singleblock simulation). • Run Singleblock simulation and collect metrics. • Choose a Multiblock simulation strategy. • Modify/reuse simulation script files for Multiblock simulation • Run Multiblock simulation and collect metrics. • Evaluate the testing activities performed. • Evaluate the entire testing results.

Function Design

Block Design Multiblock Simulation

Function Test Testing Plan Document Checklist Document Simulation Script Files

Metrics Document Defect Report Evaluation Report

Figure 10.1 Test related documents created throughout SDL development

Metrics Document Defect Report Evaluation Report

SDL Block Test

102

We are also going to illustrate the responsibility matrix of the testing activities performed during Optimal Testing Strategy and who is responsible for each one. 10.1 CONCLUSIONS The Optimal Testing Strategy that we introduced in the previous is aimed to improve the current testing process. The most obvious weaknesses in the present testing process have been found and improvement areas have been identified. We describe in details as separate testing activities in each SDL development phase the additions that improve the testing process and argue about which areas that are concerned. We have identified the weaknesses regarding tools that are used in testing process and we proposed the needed modifications in order to be able to apply in our Optimal Testing Strategy. The objective to save time in testing does not seem to be reached directly, if the time to run all the added testing activities is counted. However, we saved time by adding the following improvements: • The creation of the simulation script files as early as possible in the development process. • The conduction of Multiblock simulation, saved time for correction if the faults where

found during Function Test. • The collection of metrics in order to evaluate and improve the testing process continually. We assumed that tools used in the Optimal Testing Strategy have gone through the improvements that we have suggested.

Appendices

103

1 1 AP P E N D I C E S A SDL Specification and Description Language SDL is a standard language for the specification and description of systems. It is defined by ITU - T Recommendation Z.100. History of SDL The SDL language has its origins in telecommunications. Development of SDL started in 1968 by ITU (International Telecommunication Union) they investigated the way to handle stored program control switching systems. This investigation leads to that: the language was needed as specification, programming and human machine interaction. The first known standard of SDL was produced in 1976 since then the language has been updated every four years. The most important events/milestones in the SDL’s, history: 1968: ITU study program control system. 1972: Specification, programming and HMI studies started. 1976: Basic graphical language. 1980: Process semantics defined. 1984: Structure data added. Start using tools. Users guide.

1988: Effective tools. Syntax well defined- formal definition. The language it self is the same as in the 1984. 1992:Object oriented SDL. Types for blocks, processes, service with inheritance and parameterization. Methodology guidelines.

1995: SDL with ASN.1 (Z.105)(Abstract Syntax Notation). 1996: SDL-92, MSC-96 (Message sequence Chart). 1999: SDL-2000, MSC-2000. Object modeling support. Improved implementation support. In the 1980, much input to SDL comes from Swedish engineers: Göran Hemdal, Nils

Lennmarker and Ivar Jacobson. Much other input comes from ITU members around the world.

Application area SDL can be used in an abstract way with informal text, so that SDL is a broad-spectrum language that can be used from requirement capture to implementation of a system. The SDL language becoming increasingly popular due to its graphical language presentation, even persons without any deep technical knowledge should understand how the system functions. Typical application areas where SDL is used today are: • High and low, level telecommunication systems. • Aerospace systems. • Distributed or highly complex mission critical systems. During 1999, the SDL was used in many other applications then in telecommunications. Example on other application areas is from medical equipment to the European Space Agency.

104

The SDL language The most important characteristic of SDL is its formality. The semantics behind each symbol and concept are precisely defined. This can be used to provide an unambiguous interface for the communication systems. SDL and other languages SDL can be interfaced with other languages, such as UML Unified Modeling Language, MSC Message Sequence Charts, and ASN.1 Abstract Syntax Notation One or CORBA/IDL (Common Objects Request Broker Architecture) data type definitions. There are tools that can generate executable code for some language such as C, C++ or PLEX-C directly from SDL design.

Appendices

105

B MESSAGE SEQUENCE CHARTS (MSC) Message Sequence charts (MSC) is a graphical language for the specification and description of the communication systems. MSC language is an enhanced and formalized version of Jacobson interaction diagrams. MSCs are applied in the telecommunication sector for the specification of system requirements and for testing in combination with SDL. MSCs express of interactions with the environment, and describe how these interactions are processed by an exchange of message between system components The MSC Language The ITU maintains recommendation Z.120 [ITU120], which contains a definition of MSC. Recommendation Z.120 contains a textual and graphical syntax and an informal description of the meaning of an MSC (Figure 10.1). The MSC recommendation Z.120 includes two syntactical forms, MSC/PR as a pure textual and MSC/GR as a graphical representation. An MSC in MSC/GR representation can be transformed automatically into a corresponding MSC/PR textual representation.

The formalization of MSC is described in [Mauw1], and algebraic semantics of Basic MSCs in [Mauw2].

Textual syntax abstract grammar concrete grammar static grammar

Graphical syntax abstract grammar concrete grammar drawing rules

Semantics Denotational operational

Figure 10.1 Parts of the MSC definition

MSC

106

Graphical syntax

The graphical notation is event oriented. A graphical language rules in Z.120 is accompanied by a series of drawing rules. There are two basic concepts in MSC:

• Basic MSCs. • High-level MSCs or HMSCs.

At this point of abstraction, the visual interpretation is that basic MSC (bMSC) in Figure 10.2 is executed iteratively We assume that the processes specified by an MSC can rely on the fact that the environment will feed them with input messages whenever they wish to receive a message from the environment. Processes can also rely on the environment being ready to accept a packet at any time so that a process may send a message to the environment whenever it wishes to do so. The justification for this assumption is that we want to specify as few assumptions concerning the environment as possible, and that enforcing any constraints on its communications behavior should be avoided. As a consequence, we do not see the need to represent a state of the environment in our global system state, and no need to represent the communication of a process with the environment in the global state. Instead, we introduce distinct environment send and environment receives events for MFGs. They distinguish themselves from normal send and receive events in the conditions on their enabling and the semantics of the state transitions. Also, we will not include incoming and outgoing sign edges related to environment communication events in the MFG. We will explain the necessary extensions to the formal definitions only informally here. Textual syntax MSC is basically a graphical language and focus mainly to graphical notation. The reason to define a textual syntax is for manipulation of MSCs, such as exchange MSCs between users or tools. An MSC consists of the abstract grammar and the concrete grammar [Hanene2]. The

Figure 10.2 Basic MSC

i1 i2 msc example1

i3 i4

a

m0 m1 m2

m3

m4

Figure 10.3 High-level MSC

HMSC

M

M1 M2

Appendices

107

abstract grammar describes the information contents of a textual MSC. It can also be used for transformation and manipulation of an MSC. The concrete grammar describes the appearance of an MSC to a user. It defines the actual tokens (lexical syntax), which make up an MSC text and the order in which they are allowed to appear. The textual representation of an MSC is instance oriented. This means that a Message Sequence Chart (MSC) is defined by specifying the behavior of all instances. A message output is denoted by “out m1 to i2” A message input is denoted by “in m1 from i2” Msc example1 instance i1; out m0 to env; out m1 to i2; in m4 from i2; endinstance; instance i2; in m1 from i2; out m2 to i3; action a; out m4 to i1; endinstance; instance i3; in m2 from i2; out m3 to i4; endinstance; instance i4; in m3 from i3; endinstance; endmsc;

Listing 10 Example 1

The above Listing 10 is the textual representation of Example1 MSC diagram. The most MSC Editor tools support the exchange from MSC/GR (graphic reference) to MSC/PR (phrase reference). MSCs in the software lifecycle The use of MSCs in the software lifecycle is a subject of debate in literature. However, some authors argue that MSC can be used to express early life-cycle requirements An MSC is not a description of the completely behavior of the system, it just expresses an execution trace. In contrast to SDL, the set of specified MSCs usually covers partial system behavior only, since each MSC represents exactly one scenario. The main uses of MSC in system development [Formal] are: • Requirements capture. Use MSC to identify Use Cases before beginning of design the SDL

system. It can be used Basic MSCs and High-level MSCs.

108

• Specifications of use case scenarios. • Behavior verification of SDL specifications. (As it is described in Validation Chapter). • Interface specification where a system or blocks interactions occur. • Overview of a service. • Basis for simulation and validation. • For object oriented design and analysis (object interaction).

Appendices

109

C PLEX PROGRAMMING LANGUAGE FOR EXCHANGES Programming language developed by Ericsson in the 1970’s. There are two versions of PLEX: • PLEX-C the version that is used in central processor in AXE 10 system. • PLEX-M the version to use in EMRP PLEX Features Modularity: The PLEX programs consist of modular structures. The modules cooperate with signals. One module may be modified without affecting the other. High level: PLEX-C commands are independent of the processor type and is translated into different instructions Single-Purpose: PLEX-C is designed for telephony systems. Real-Time: Programs have to meet certain time limits and predict execution times.

Block interactions As described above the language is modularized. The interaction between blocks of code is done with signals. The signal concept • A signal is usually a jump from one function unit to another. • A signal may or may not carry data. Subprograms PLEX programs is never executed from the beginning to the end, but usually from a signal-receiving statement, normally ENTER, to a signal-sending statement, SEND, followed by EXIT. The program sequence from ENTER to EXIT is called a subprogram. PROGRAM; PLEX; ENTER SELFIND1; SEND SELFIND1 REFERENCE COWNREF WITH DEVPTR, DEVPTR:BLSTATE, DELAY 300 MS; EXIT; Signals can be delayed by using time queues. Signal with the highest priority executes. All temporary variables and pointers are lost after the EXIT statement. Call Routines Call routines are used for frequently recurring of program sequences. There are two types of call routines in PLEX: • Subroutines. • Procedures. The main difference is that subroutines are written by the block designer during unit design, while procedures are already written.

110

Procedures Procedures are written in ASA code (Assembly language), allowing users to access specific registers in the CPU. The designer just calls the procedure e.g. calling procedure LOADREF

Appendices

111

D VALIDATOR Monitor Commands Monitor commands are issued in the same way as in the Simulator, since the same monitor system is used in both tools. Commands can be issued to the interactive monitor system when it becomes active. The Validator's monitor system becomes active: • When the Validator is started. • When the last command was Next or Random-Down and the transitions initiated by this

command have completed. • When the last command was Bit-State-Exploration, Verify-MSC or Exhaustive-

Exploration and the behavior tree down to the defined search depth has been fully explored.

• When the last command was Random-Walk and the defined number of repetitions have been executed.

• Immediately after a report with the report action Abort has been generated during automatic state space exploration.

• In the Validator UI, when the Break button is clicked in the Explore button module; in textual mode, when <Return> is pressed during an automatic state space exploration.

The syntax of commands and data type literals in the Validator monitor is the same as for the simulator monitor. All input to the monitor is case insensitive. The most important commands for validation are: • Define-Max-State-Size: The default value is too small (100,000 bytes) for validating

large systems. • Define-Max-Transition-length: The default value is too small. If default value is not

changed then the Validator aborts when the transition length value is reached. • Define-Parameter-Test-Value: The desirable test values can be assigned to parameters

in the signals and in the variables. • Define-Max-Instance: The maximum number of instances can be set by this command. • Clear-Test-Value: This command helps to adjust values for some type sorts that are not

used in decisions by the Validator. Validator detects that during the validation and generates a report on the problem, which is not the case.

• Define-Rule: User can define his own rules that should be checked. If the rule is satisfied, the report is generated.

• Clear-Rule: Deletes a user-defined rule. • Bit-State-Exploration: Starts the verification according to Bit-State algorithm.

Parameters: (None). Continue until defined Search depth: the maximum search depth of the exploration an integer value. This value can be changed by the command: Define-Bit-State-Depth. Hash table size: The size of the hash table used during bit state exploration an integer value. This value can be changed by the command: Define-Bit-State-Hash-Table-Size. Iteration step: The value of iteration step during the bit-state-exploration. This value can be changed by the command: Define-Bit-State-Iteration-Step.

• Random-Walk: Starts the verification according to Random-Walk algorithm. Parameters: (None). Continue until defined Search depth is reached or a number of Repetitions.

• Tree-Walk: Starts the verification according to Tree-Walk algorithm. Parameters: (Time in minutes and Coverage in percent). Continues until either time or coverage is reached.

112

• Power-Walk: Starts the verification according to Power-Walk algorithm. Parameters: (None). Continues until the predefined number of reports is reached. Continue until defined Repetition, timeout, or coverage is reached.

• Exhaustive: Starts the verification according to Exhaustive algorithm. Parameters: (None). Continue until defined Search depth is reached.

• Tree-Search: Starts the verification according to Tree-Search algorithm. Parameters: (None). Continue until defined Search depth is reached.

• Verify-MSC: Starts the verification of either HMSC or MSC. Parameters: (MSC name.msc files or MPR name.mpr files). Continues until the MSC is verified.

• Define-Report: Options available by this command are: 1. Continue- the Validator continues the state space exploration regardless error. 2. Prune- the Validator will not continue past the current state where the rule was

violated but will continue on the parent or sibling system state. 3. Abort- the Validator will stop the state space exploration. The exploration will

continue until the following action occurs:

Appendices

113

D MEDIATOR In this Appendices section, we will describe the most common mediator commands. The Table 10.1 describes all important mediator commands. Mediator Commands The Mediator commands are associated to action taken place in APZ Emulator. The Table 10.1 Mediator commands below are showing the correlation between the Mediator and APZ Emulator commands. The Table 10.1 describes also the must important Mediator commands to be used during the emulation process. Mediator commands can be invoked in two ways. Either by pressing the associated button or by typing the command in the command line. The first mentioned is to be preferred.

Table 10.1 Mediator commands

Mediator command Command description Emulator command “Load dump” button in Emulator Setup area or in Files | Load Dump menu

Load the emudump file LOAD <file>

“Connect EMU” button∗ in Emulator Setup area

Connect Mediator with Emulator Not Available

“Default” button in Emulator Setup area

Default button sets “SIGHALT OFF”, “CONFIRMCMD ON”, Signal trace “+S” and Instruction trace “+I”

Same commands as in Mediator

“LogWin On” button∗ in Emulator Setup area

The execution can be traced in textual mode. Choice “ON” is the default for both tools.

LOGWIN

“CmdWin On” button* in Emulator Setup area

Stores the invocation of commands CMDWIN

“Alteration” button in Emulator Setup area

The Seize Alteration event may be set. Takes tree parameters

SAII; SAE=xxx ,NI=i

“Variable” button in Emulator Trace area

Shows values of variables/variable +V,V,-V

“Set Block” button in Block, Signal & Execution area

Shows all active blocks B

“View Block” button in Block, Signal & Execution area

Activates Plex View to show the active block

B

“Send Sig” button in Block, Signal & Execution area

Sends the signal, takes signal name and signal parameters as input

S

“View Sig” button in Block, Signal & Execution area

Dialog appears asking for signal to view then Plex View activates and shows the signal parameters

S

“View Signals” button in Shows all of the signal available S

∗ Even used during approach with BTI Driver

114

Block, Signal & Execution area “Continue” button in Block, Signal & Execution area

When signal is “sent” this button must be pressed to really send the signal

CONT

“Variables” button in View area

Shows all available variables in the monitored block

V

“Registers” button in View area

Shows the current contain of registers R

“Blocks” button in View area

Shows all active block B

“Variable” button in View area

Dialog appears asking for variable to view

V <variable name>

“MSC Trace” button in SDL & MSC area

Dialog appears asking what kind of trace presentation to choose either ON(HEX) or ON(DEC) the third option is Off

Not Available

“MSC Filter” button in SDL & MSC area

Dialog appears asking of what kind of filter on instances to choose either by exclusion or inclusion

Not Available

“Break Plex” button in Breakpoints area

Dialog appears asking to specify on which block to set breakpoint

STOP

“Break Sig” button in Breakpoints area

Dialog appears asking to specify on which signal to set breakpoint

STOP IN

“Unbreak” button in Breakpoints area

Dialog appears asking to specify where to remove breakpoints

UNSTOP

Requirements to improve Mediator tool In this section, we will define the requirements needed to obtain correct MSC-trace and simulator script files. Each requirement will get a prioritization value from 1 to 3, where 1 means very important, 2 means important and 3 means not so important. For each requirement, the prioritization value follows in parenthesis. The question is why to prioritize the requirements? Here are some answers to that question: • A focus on what really matters. • Identification of high- and low priority requirements. • Implementation of the right requirements in the right order. R 8:1. All macro definitions in BTI script shall be transformed into separate simulation script

file with correct syntax according to Simulator (e.g. Add-Macro Macro name Macro value). Even comments shall be transformed into simulation script file. (1).

R 8:2. Test cases written in BTI Driver should be transformed in separate simulator script files. One test case ⇒ one simulation script file. The simulator command to use is “Breakpoint-Output”. (1).

R 8:3. The BTI test instruction Expected IO must be transformed into simulator script

command starting with “ Expected-output-begin” and ending with “Expected-output-end”. (2).

Appendices

115

R 8:4. Variable values assigned with BTI instruction must be transformed into simulator

script files. (1). Example on the syntax in the BTI script and the corresponded syntax in the simulator script file: BTI instruction: “Initiate: v CRCSLCBLOCK = #FFFF”. Simulator script instruction: “Assign-Value CRCSLCBLOCK < block name/ block name> #FFFF <process name: process instance>”

R 8:5. Variable values, which are checked in BTI-script, must be transformed into simulator

script files. (1). Example on the syntax in the BTI script and the corresponded syntax in the simulator script file: BTI instruction: “ Check var: $CBCHind4$: STATE = $CBCHBLOC$” Simulator script instruction: “Examine-Variable STATE < block name/ block

name> $CBCHBLOC$ <process name: process instance>”

R 8:6. The BTI test instruction Expected sig must be transformed into simulator script files (1).

R 8:7. Create an instance in MSC trace “Norec” which should receive all signals denoted

with “not yet consumed”. Otherwise, those signals will be excluded from the simulator script file. (1).

R 8:8. The BTI test instruction Restore and Save must be transformed into corresponded

simulator command “Execute-Input-Script” and followed by path specification of the input script. (1).

R 8:9. The instance head name “Environment” must be displayed at least once in order to

produce a valid simulator script files. Otherwise, the simulator script file will be empty. (1).

R 8:10. The instance head name “OtherInstances” has to be excluded. In addition, other

instances have to be filtered away as we mentioned above. (2). R 8:11. As a comprehensive requirement must Mediator transform all BTI instructions into

meaningful correspondent simulator commands. (2). R 8:12. As a general requirement must all simulator script files created by Mediator, must be

accepted by Simulator. (2). R 8:13. Depending on the BTI script size and complexity the transformation into simulator

script files should require reasonable amount of time. (3). R 8:14. It should be easy to add new functionality to Mediator. (3).

116

Appendices

117

F SIMULATION OF START RESTART Description and Implementation of start and restart of RNSDCCH-block To implement simulation of start and restart procedures on SDL-level the system needs some adjustments. In this paragraph, those adjustments will be described in chronological order. 1. The system must contain an instance of the block type Restart that is included in the

RestartSimulation package. We choose to call it SimRestart. Since SimRestart inherits from Restart using the “Use package NAME” clause The SimRestart block has to contain a process, also called Restart, which handles all the signals to and from Restart process to the BlockProcess in RNSDCCH-block. Since the concept of block types is not supported by the Plex-C Code Generator, the block SimRestart will be ignored during PLEX code generation, and will be used only for simulation purposes. After completing this adjustment, the system will look like Figure 10.4. The new block that is added to the System SS_RCS.

2. Next step is to connect block SimRestart with system SS_RCS block RNSDCCH and

environment. Connection is realized through channels: one channel that connects the block SimRestart with signal route to and from APZ and the other connection is to and from the block RNSDCCH. The signal, which comes from APZs block SR (Start/Restart), is called SYREI. This signals it occurs when operators of APZ request a restart of the system by typing the command SYREI (see above in background). The signal has following parameter: RestartType of type literals containing information about restart case (INITIATE, SMALL, LARGE, RELOAD). When SYREI signal is received in the Restart block of RestartSimulation package, then Restart block sends a signal called STTORsim to the process BlockProcess in the block RNSDCCH with the following parameters: RestartCase, phase, ID • RestartCase is of type RestartType • Phase is of type Integer and contains information about which restart procedure to

restart. There are 255 start phases, but in our system we only use the following phases: 20, 23, 29, 31 and 44

• ID is of type Integer and contains information about signal key

Figure 10.4 System SS_RCS

118

When all work is done in the BlockProcess, it sends a signal called STTORRYsim back to the SimRestart block. We describe in more details about this signal later. The signal STTORsim is included in the signal list SIM_to_BP and the signal STTORRYsim in the signal list BP_to_SIM. These signals are sending via a channel called Csim and the signal SYREI is sending via channel CsimInit. After completing those adjustments, the system shall have behavior according to Figure 10.5. In our system, we only have one block but if system contains more than one block then all blocks must be able to receive and send STTORsim and STTORRYsim.

Figure 10.5 Restart package connected to one block

Appendices

119

3. All restart procedures can only be called by the BlockProcess for the entire block group, thus procedures in CellProcess and SDCCHProcess must be declared as EXPORTED. The BlockProcess must IMPORT these procedures. This is done by adding in the RNSDCCH block group level the clause Remote Procedure and the names of the procedures to be imported, i.e. CPRestart29. Then we have to add in the BlockProcess the Import Procedure clause and the name of the procedures. Each BlockProcess must be able to receive STTORsim in any state. In our case there was only one state BP_Waiting.

4. When the STTORsim signal is received by the BlockProcess, then STTORRYsim is sent

back to acknowledge the receive of STTORsim signal. STTORRYsim’s Parameters are ID as first parameter, the same as in the STTORsim, following by 17 parameters that are the requested phases to store one for each restart phase. These phases are of type integer. The meaning with different restart phases is that it has to be a certain order to process the restart in APZ. The phases are processed in increasing order.

• Phases 20-22 mean that the LOADREF procedure is called, which is loading

individual block reference. • Phases 23-25 mean that interaction information between blocks is loaded. • Phases 26-28 mean that call tracing is loaded, which is trace calls that are in speech

position. • Phases 29-31 mean updating of states and blocking states, generation of IDLE lists

in device blocks, setting of start values in variables, and updating of EM data. • Phases 32-34 mean updating of block directed to other blocks. • Phases 35-37 mean updating the remaining data. All blocks have updated all data

after phase 37. • Phases 38-49 mean that the blocks activate themselves by storing their periodic

signals in job tables and job queues. Traffic handling starts after phase 44 at which time signaling links are activated.

Our system contains four processes, but only three contains restart procedures:

• BlockProcess with restart procedures:

BPRestart20 that has phase 20 and RestartCases: INITIATE, SMALL, LARGE, RELOAD. BPRestart23 that has phase 23 and RestartCases: INITIATE, SMALL, LARGE, RELOAD. BPRestart29 that has phase 29 and RestartCases: INITIATE, SMALL, LARGE, RELOAD. BPRestart44 that has phase 44 and RestartCases: SMALL, LARGE, RELOAD.

• CellProcess with restart procedures:

CPRestart29 that has phase 29 and RestartCases: INITIATE, SMALL, LARGE, RELOAD.

• SDCCHProcess with restart procedures:

SDCCHPRestart31Small that has phase 31and RestartCase SMALL SDCCHPRestart31Large that has phase 31and RestartCase LARGE

120

SDCCHPRestart31Other that has phase 31and RestartCase INITIATE, RELOAD When procedures have been processed, STTORRYsim is send back again by the BlockProcess. We have to design an initiation of these restart procedures in the BlockProcess and this can look like Figure 10.6 DetermineSimRestart. Not all declaration is included in the figure.

5. In our system, we just have one instance of BlockProcess but CellProcess and

SDCCHProcess have more then one instance, which must be handled by Restart-Simulation package. These should be called once for each active instance of the process. To find all these instances, the existing procedures, GetFirstInd and GetNextInd can be used to loop through all active instances. To see how this is done look at fig DetermineSimRestartCP, red box.

Figure 10.6 DetermineSimRestart

Appendices

121

G MULTIBLOCK SIMULATION Problem and solutions The problem that we encountered during Multiblock simulation was mostly throughout the execution of merging process when blocks were added to Multiblock system. The following errors was emerged: 1. Process names that are common for some SDL blocks (e.g. BlockProcess, CellProcess

etc) are reported as defined multiple times. This was not accepted by Analyzer tool. Solution: Rename those process names that are reported with unique name qualifiers.

2. Names of signal lists are reported as multiple defined. Solution: Rename signal lists. As a good practice the name of the block, have to be added before process name. I.e. BlockName_SignalListName.sun

3. The packages that are common for the blocks to be merged are reported as defined multiple times (package as Plextypes, RestartSimulation). Solution: It has to be only one package in the Multiblock system that contains package definitions for all blocks. Remove the redundant packages.

4. Some signals and temporary signals common for some blocks included in Multiblock system are reported as defined multiple times by analyzer tool. Solution: Merge all unique signals into one signal package for all blocks in Multiblock system.

5. The packages that contain type definitions for different blocks are reported as defined

multiple times by analyzer tool. Solution: Rename all types that are defined multiple times through all SDL design where these types are used.

6. Procedure names that are common for some SDL processes are reported as multiple

defined and are not accepted by Analyzer tool. Solution: Rename those procedure names that are reported with unique names. A good design style is to begin with the block name followed by the procedure name.

7. Operator FindNext caused a crash for Simulator tool. Operator FindNext caused segmentation fault. Hypothesis: FindNext operator cannot support Multiblock simulation due to compiled presentation of C code. Solution: Rewrite FindNext operator adopted for Multiblock systems.

8. Operator Partner gets an invalid PId value, which caused signal sends to null.

Hypothesis: Same as in 7. Solution: Rewrite Partner operator adopted for Multiblock systems

122

Appendices

123

H GLOBAL SYSTEM FOR MOBILE COMMUNICATION (GSM) Background Mobile telecommunications is one of the fastest growing and most demanding of all telecommunications technologies. Ericsson has been designing cellular radio systems since the 1970’s. It offers network products for all major standards, both analogue and digital. GSM is a global standard for mobile communications. Because GSM provides a common standard, cellular subscribers can use their telephones over the entire GSM service area, which includes all the countries around the world where the GSM system is used. In addition, GSM provides user services such as high-speed data communication, facsimile and a Short Message Service (SMS). GSM Network Components The GSM network is divided into two systems. Each of these systems is comprised of a number of functional units, which are individual components of the mobile network. The two systems are: •• Switching System (SS) •• Base Station System (BSS)

Abbreviations: AUC AUthentication Center BSC Base Station Controller BTS Base Transceiver Station EIR Equipment Identity Register HLR Home Location Register MS Mobile Station MSC Mobile services Switching Center NMC Network Management Center OMC Operation and Maintenance Center VLR Visitor Location Register The SS, Switching System is responsible for performing call processing and subscriber related functions. It includes the following functional units: • Mobile services Switching Center (MSC). Performs the telephony switching functions for

the mobile network. • Home Location Register (HLR). Is a centralized network database that stores and manages

all mobile subscriptions belonging to a specific operator. • Visitor Location Register (VLR). The VLR database contains information about all the

mobile subscribers currently located in an MSC service area. • AUthentication Center (AUC). Authenticate the subscribers attempting to use a network.

H Global System for Mobile communication (GSM)

124

• Equipment Identity Register (EIR). The EIR is a database containing mobile equipment

identity information, which helps to block calls from stolen, unauthorized, or defective MSs.

The BSS, Base Station System performs all the radio-related functions. The BSS is comprised of the following functional units: • Base Station Controller (BSC) • Base Transceiver Station (BTS) The OMC, Operation and Maintenance Center performs all the operation and maintenance tasks for the network such as monitoring network traffic and network alarms. The OMC has access to both the SS and the BSS. Basic Station Controllers (BSC) manages all the radio-related functions of a GSM network. It is a high capacity switch that provides functions such as MS handover, radio channel assignment and the collection of cell configuration data. A number of BSCs may be controlled by each MSC. This is the node in the GSM network, which is of interest for our study. Base Transceiver Station (BTS) controls the radio interface to the MS. The BTS comprises the radio equipment such as transceivers and antennas, which are needed to serve each cell in the network. A group of BTSs is controlled by a BSC. Cell. A cell is the basic unit of a cellular system and is defined as the area of radio coverage given by one BS antenna system. Each cell is assigned a unique number called Cell Global Identity (CGI). In a complete network covering an entire country, the number of cells can be quite high.

A Location Area (LA) is defined as a group of cells. Within the network, a subscriber’s location is known by the LA, which they are in. The identity of the LA in which an MS is currently located is stored in the VLR. When an MS crosses the boundary between two cells belonging to different LA’s, it must report its new Location Area to the network. If it crosses a cell boundary within a LA, it does not report its new cell location to the network. When there is a call for an MS, a paging message is broadcast within all the cells belonging to the relevant LA.

125

References

126

R E F E R E N C E S [Florac] William A. Florac, Robert E. Park, Anita D. Carleton, “Practical Software Measurement: Measuring for Process Management and Improvement”, Software Engineering Institute, April 1997. [IEEE829] IEEE Standard for Software Test Documentation, IEEE Std 829-1998. [IEEE86] IEEE/ANSI Standard 1012-1986, IEEE Standard for Software Verification and Validation Plan. [Z.120B] ITU-T. Recommendation Z.120, Annex B: Algebraic Semantics of Message Sequence Charts, ITU - Telecommunication Standardization Sector, Geneva, Switzerland, 1995. [Z.120] ITU-T. Recommendation Z.120. ITU - Telecommunication Standardization Sector, Geneva, Switzerland, May 1996. Review Draft Version. [Kit] Edward Kit, (1995), “Software Testing in Real World. Improving the process”, ACM Press, Addison-Wesley, ISBN 0-201-87756-2. [Holzman] Gerard J. Holzman, “Design and Validation of Computer Protocols”, Prentice Hall 1991, ISBN 0-13-539925-4 [Hanene1] Hanna Ben-Abdullah and Stefan Leue, (1997), “Syntactic Detection of Process Divergence and non-Local Choice in Message Sequence Charts” [Hanene2] H. Ben-Abdallah and S. Leue, “Syntactic Analysis of Message Sequence Chart Specifications”, Technical Report 96-12, Dept. of Electrical and Computer Engineering, University of Waterloo, November1996. [Formal] Michael Andersson and Johan Bergstrand “Formalizing Use Cases with Message Sequence Charts”, Appendices A, Master thesis, Department of Communication Systems at Lund Institute of technology, November 1995 [Leue] S. Leue. “Methods and Semantics for Telecommunications Systems Engineering”, PhD thesis, Department of Computer Science and Applied Mathematics, University of Berne, Switzerland, December 1994. [Ladkin] P. B. Ladkin and S. Leue. “Four Issues Concerning the Semantics of Message Flow Graphs”, in: D. Hogrefe and S. Leue (eds.) Formal Description Techniques VII, Proc. of the Seventh IFIP International Conference on Formal Description Techniques FORTE'94, Oct. 4-7, 1994, Chapman & Hall, 1995. [Mauw1] S. Mauw, “The Formalization of Message Sequence Charts”, In SDL 1995 with MSC in CASE, Tutorials 95.09.25, editor O. Haugen [Mauw2] S. Mauw & M. A. Reniers, “An algebraic semantics of Basic Message Sequence Charts”, The Computer Journal 37(4), pp. 269-277, 1994.

References

127

INTERNET LINKS “MSC Tutorial” http://www.win.tue.nl/~sjouke/tutorial.html “MSC Standards” http://www.itu.org/ “What is MSC” http://www.sdl-forum.org/MSC/index.htm “Research on Message Sequence Charts and Message Flow Graphs” - http://tele.informatik.uni-freiburg.de/~leue/msc.html