CODEN:LUTEDX ( TETS - 5465 ) / 1 - 90 / ( 2002 ) & local 21

Introducing a software test process into a small-scale software organisation

Bachelor thesis: Anders Bengtsson Magnus Björnfot

Copyright Anders Bengtsson, Magnus Björnfot LTH School of Engineering at Campus Helsingborg Lund University Box 882 SE-251 08 Helsingborg Sweden LTH Ingenjörshögskolan vid Campus Helsingborg Lunds Universitet Box 882 251 08 Helsingborg Printed in Sweden Media-Tryck Biblioteksdirektionen Lund University Lund 2002

ABSTRACT Introducing a software test process into a small-scale software organisation The purpose of the thesis is to present a strategy for the implementation of a software test process in the company in focus. The result is strategy that contains some milestones to be achieved at each step. The result is presented as a four-step strategy dealing with the main focus on testing and documentation of the system. The strategy is the outcome of the theory in the thesis as well as interviews, observations and gained experiences from having spent time at the office. The strategy takes under consideration that the company does not provide all the necessary elements for a test process. Instead the strategy provides a basis for the current tests being conducted and a future improved test process. A conclusion of the thesis is that introducing a new process into an organisation is a complex matter and affects all part of an organisation. Therefore the work with the thesis has included analyse of the company’s current work methods to find weaknesses and strengths in the organisation. The additional values gained in the thesis is that the “real thing” is much more difficult and diffuse then the textbook example. Keywords: Software testing, test strategy, test process, CMM, AHP, development process. SAMMANFATTNING Införandet av en testprocess på ett småskaligt programvaruföretag. Syftet med examensarbetet är att presentera en strategi för införandet av en test process på ett företag. Resultatet av examensarbetet är en strategi som bygger på fyra steg. Varje steg innehåller i sin tur milstolpar vars slutliga syfte är att presentera en fungerande testprocess och dokumentation av systemet. Strategin bygger på teori, intervjuer, observationer samt egna erfarenhet från tiden tillbringad på kontoret. De problem som företaget idag har tas under beaktning i strategin, då företaget idag inte har alla de nödvändiga element som krävs för en fungerade

testprocess. Istället utgör strategin en grund för nuvarande test behov samt framtida förbättringar av testprocessen. En slutsats som tagits är att introduktionen av en ny process i ett företag är både svårt och att den berör alla delar i en organisation. Därför har arbete med framtagandet av strategin inkluderat analys av företagets nuvarande arbetsmetoder för att hitta negativa och positiva saker i organisationen. En viktig erfarenhet av examensarbetet är att verkligheten är mer diffus och att allt inte är enligt läroböckerna.

FOREWORD This bachelor thesis is a result of a need to implement a test strategy in a company’s current work process. The work has been done at the company office with close cooperation with the future test manager. The cooperation with the personnel have for us been very important and useful when writing the bachelor thesis. We hope that the outcome of this thesis will gain surplus values for the company in their future work. Anders Bengtsson Magnus Björnfot

ACKNOWLEDGEMENTS Thanks to: Thomas Thelin at the Department of Communication Systems, Lund Institute of Technology, Lund University Peter at the company in focus

LIST OF ABBREVIATIONS AHP Analytic Hierarchy Process CASE Computer Aides Software Engineering CMM - SW Capability Maturity Model – Software COTS Commercial Of The Shelf system CRC Class Responsibility Collaborator GQM Goal Question Metric paradigm ISO International Organisation for Standardisation KPA Key Process Area PM Product Manager RUP Rational Unified Process SDP Software Development Plan SEI Software Engineering Institute SRS Software Requirement Specification SVVI Software Verification and Validation Instruction SVVP Software Verification and Validation Plan SVVS Software Verification and Validation Specification UML Unified Modelling Language XP Extreme Programming

LIST OF CONTENTS: 1.1 General description of the problem ........................................................................1

1.1.1 Goal ................................................................................................................ 2 1.1.2 Purpose ........................................................................................................... 2 1.1.3 Constraints ...................................................................................................... 2

1.2 Workflow - methodology .........................................................................................2 1.2.1 Information gathering and documentation ..................................................... 2 1.2.2 A survey at the company ................................................................................ 3 1.2.3 Formulation of appropriate test methodology ................................................ 3

1.3 Outline .......................................................................................................................3 2 RELATED THEORY ......................................................................................................4

2.1 Software Development Process ...............................................................................4 2.2 Fundamental Phases in Software Engineering ......................................................5

2.2.1 Specification documents................................................................................. 5 2.2.2 Requirement elicitation................................................................................... 6 2.2.3 Design............................................................................................................. 7 2.2.4 Implementation............................................................................................... 7 2.2.5 Testing ............................................................................................................ 8 2.2.6 Evolution ........................................................................................................ 8

2.3 Application of the Phases – Software Development Models.................................9 2.3.1 The waterfall model........................................................................................ 9 2.3.2 Evolutionary development - prototyping...................................................... 10 2.3.3 Incremental development ............................................................................. 11 2.3.4 Spiral development ....................................................................................... 12

2.4 Introduction to Quality ..........................................................................................13 2.4.1 Quality - what is it all about? ....................................................................... 13 2.4.2 Development of quality related work ........................................................... 13 2.4.3 Quality in software engineering ................................................................... 14 2.4.4 Quality assurance in software development ................................................. 15

2.5 Software Process Improvement.............................................................................16 2.6 CMM – The Capability Maturity Model..............................................................18

2.6.1 Introduction to CMM ................................................................................... 18 2.6.2 Operational definition of CMM.................................................................... 18 2.6.3 The five maturity levels of CMM................................................................. 19

2.7 RUP - Rational Unified Process ............................................................................24 2.7.1 The “6 best practices” of RUP...................................................................... 24 2.7.2 The Dynamic Behaviour of RUP ................................................................. 25 2.7.3 The static view of RUP................................................................................. 27

2.8 Design Using Unified Modelling Language (UML).............................................29 2.9 The Software Test process .....................................................................................32

2.9.1 Why is software testing important? .............................................................. 32 2.9.2 Strategies in software testing........................................................................ 32 2.9.3 Additional test types ..................................................................................... 37 2.9.4 When to test what? ....................................................................................... 38

2.10 Reviews ....................................................................................................................39 3 METHODOLOGY AND TECHNIQUES....................................................................42

3.1 GQM – the Goal Question Metric paradigm .......................................................42 3.1.1 Goals – the conceptual level......................................................................... 43 3.1.2 Questions – the operational level ................................................................. 43 3.1.3 Metrics – quantitative level .......................................................................... 44

3.2 Interviews................................................................................................................ 44 3.2.1 Different types of interviews........................................................................ 45 3.2.2 Different kinds of questions ......................................................................... 46 3.2.3 Formulating the questions ............................................................................ 46 3.2.4 Preparation for the interview........................................................................ 47 3.2.5 Carrying out the interview............................................................................ 48

3.3 The Analytic Hierarchy Process (AHP) ............................................................... 49 3.4 Survey methods applied......................................................................................... 50

3.4.1 Using interviews to collect information from the employees ...................... 50 3.4.2 Using AHP to prioritise quality aspects on the company’s system.............. 50

4 DESCRIPTION OF THE COMPANY ........................................................................ 53 4.1 Organisation ........................................................................................................... 53 4.2 Current work methods........................................................................................... 54

4.2.1 General workflow......................................................................................... 54 4.2.2 Requirement Elicitation................................................................................ 54 4.2.3 Design........................................................................................................... 55 4.2.4 Implementation............................................................................................. 55 4.2.5 Testing.......................................................................................................... 55 4.2.6 Evolution ...................................................................................................... 56

4.3 Goals with the testing............................................................................................. 56 4.4 Organisational changes.......................................................................................... 56 4.5 Current process improvement strategies ............................................................. 57

5 ANALYSIS...................................................................................................................... 58 5.1 Analysis of the interviews ...................................................................................... 58

5.1.1 Support ......................................................................................................... 58 5.1.2 Developers.................................................................................................... 59 5.1.3 Testers .......................................................................................................... 59

5.2 Analysis of the prioritising of quality aspects using AHP .................................. 60 5.2.1 Analysis of the prioritising per group .......................................................... 61

6 IMPLEMENTATION STRATEGY FOR THE TEST PROCESS ........................... 62 6.1 Description of the strategy..................................................................................... 62 6.2 Strategy for implementation of a test process ..................................................... 63

6.2.1 First step – SRS and SVVP.......................................................................... 63 6.2.2 Second step - System test ............................................................................. 65 6.2.3 Third step – Module/Unit testing ................................................................. 65 6.2.4 Fourth step – Integration testing................................................................... 66

7 CONCLUSIONS............................................................................................................. 67 8 FURTHER WORK ........................................................................................................ 68

8.1 Further work for the company ............................................................................. 68 9 REFERENCES............................................................................................................... 69 APPENDIX A ......................................................................................................................... 72 APPENDIX B.......................................................................................................................... 74

Summary of an interview with a developer at the company .......................................... 74 APPENDIX C ......................................................................................................................... 75

Summary of interviews with support personnel and support management................. 75 APPENDIX D ......................................................................................................................... 76

Summary of an interview with test personnel at the company ...................................... 76 APPENDIX E.......................................................................................................................... 77

The AHP Excel-sheet ......................................................................................................... 77 APPENDIX F.......................................................................................................................... 78

Template for software verification and validation specification....................................78 APPENDIX G .........................................................................................................................79

Template for software requirements specification..........................................................79

1 INTRODUCTION In this section, a description of the problem is presented together with the main purpose of the work. There is also a description of the method that is used to analyse and come up with a conclusion.

1.1 General description of the problem The main purpose of this bachelor thesis is to introduce a software development process in a small-scale software organisation. The focus of the process is on testing and the work is mainly concerned with the testing part of the development process. Software development is complex and there are four main phases to manage:

• Requirements and specification - what the system is supposed to do? • Design - how shall it be done? • Implementation – how to make the system do it? • Testing - does the system fulfil the requirements?

The software development process includes these four phases and it is how they are adapted by the organisation that defines the specific process. A more thorough discussion of the software development process is done in section 2.1. The organisation, denoted the company in this document, that is in focus of this thesis is a small-scale organisation, about 40 employees. The company develops and delivers complete solutions for retail dealers. The company has over the past two years expanded their organisation. This combined with their business strategy that implies that they should be a leading company at the market has resulted in the need for a defined process in the software development area and the need of improved quality focus on their systems. There is no possibility for the company to develop an independent test department, meaning that no person will be working full time with testing. Because of this there must be a bespoken outline on how the process improvement may be implemented. One issue that needs to be handled is what to be considered when implementing a new process in an organisation without increasing the workload of the personal. This is because there will not be any employment to cover the extra workload that will arise from implementing and adapting a software testing process. Instead the testing process is supposed to be integrated in the company’s future development process. This is a problem that sets great demands on the future process and test methodology.

1

The bachelor thesis is mainly concerned with the part of the organisation that develops the software for a teller machine.

1.1.1 Goal

”By mapping and analysing the software process at the company an outline for process improvement will be developed. The outline will steer the software towards the quality goals defined by the company.”

1.1.2 Purpose

The main task is to analyse the current work method at the company and come up with an outline on process improvement with focus on test methodology and how testing should be performed so that high quality is achieved in the system development.

1.1.3 Constraints

The work is limited to come up with an outline how the process improvement can be performed; the actual implementation of the process is out of the scope of this work.

1.2 Workflow - methodology This section provides an overview of the projected work.

1.2.1 Information gathering and documentation

The literatures used in the work are both scientific articles and books in the following areas.

• Literature on how to best make studies and analyses or evaluate working environment, for example literature about questionnaires (how to formulate, layout and analysing questions), interviews (how to conduct interviews), GQM- Goal Question Metric and so forth.

• Literature of the software testing area including test methodology and theory, with the purpose to give the reader an insight of the grade of knowledge that must be achieved before coming to conclusion.

• Literature on how to best make an implementation of a software testing process in an organisation: The purpose of this literature is to get a hint of what problems that may occur and what practical things need to be done.

• Literature on the software development area in general to give an overview of the whole software development process.

2

• Literature on Software Processes, with the purpose of see if there are any standard operations, documents etc. in the area.

1.2.2 A survey at the company

The survey is carried out trough interviews. These are developed using the Goal Question Metric (GQM) paradigm. The goals are formulated in co-operation with the company, and out from the formulation of questions according to the methodology described in section 3.2 are carried out. The knowledge gained from section 3.2 gives guidelines to the work concerning formulating the questions to the interviews.

1.2.3 Formulation of appropriate test methodology

The outcome from the interviews, the analysis of the system and the company’s needs give an opinion of what the company needs to direct their testing towards. This results is an outline of a structure of a defined test strategy and which parts of the company that are affected. This is presented in section 6.

1.3 Outline In this section a description of what the different chapters in this thesis contains. Chapter 2: This chapter describes related theories to this thesis. First an introduction of what software process engineering is and then a description of the different phases in software engineering. Software development models are then described and compared. The chapter also contains the fundamentals of quality aspects, software process improvement and finally the software test process. Chapter 3: The chapter contains a description of the different methodologies and techniques that where applied during the work. Chapter 4: Chapter 4 includes a description of the company in focus. With the organisational structure, products, work process and goals. Chapter 5: In this section analysis of interviews and the prioritising of quality aspects are carried out. Chapter 6: Chapter 6 contains a proposed strategy for implementing a test process.

3

Chapter 7: This chapter includes conclusions and a summary of the work and possible further work suggestions. 2 RELATED THEORY In this section, the basic theory that relates to the bachelor thesis is described. It starts with an overview of the software development process and its phases. First the specification phase with a glance at requirements engineering is discussed followed by the design phase and its related techniques. Because the company in focus for this thesis has decided to use UML (Unified Modelling Language), in their design process, an overview of the UML and its methods are presented. The software testing is discussed and analysed later on in the chapter. Later on in the section the concept of quality is discussed also process improvement, covering the Capability Maturity Model (CMM) and Rational Unified Process (RUP), is introduced in this chapter. Finally the software test process is discussed and presented.

2.1 Software Development Process The word process is derived from the Latin word “processus” which stands for progress. Bergman [15] defines a process like: “a process is a network of activities that are repeated over time and their purpose is to gain value to any extern or intern interest”. A process is about coordination between persons e.g. agreements between individuals that cooperate and their competence. It is more about teamwork than serial production [15]. Software development is a complex process. In this section an overview and a brief orientation of the different parts and aspects of the software development process are presented. In the software development domain there are a number of key issues that the reader must be familiar with to understand the principles of software development such as software process and software engineering. Software process is the defined activities and phases and the outcome from these that results in a software system [7]. The process model describes the steps and documents that will be developed during the development cycle. The process contains different phases (see section 2.2), which are performed in a

4

specific order. The outcomes from the phases are different well-defined products, like system components or system documentation [22]. Software engineering is the specification, development management and evolution of software systems. Software engineering also includes project management and in a cost and time effective way plans and perform the development of software. This set demands on the recourses in software projects. Not only a technical competence is of matter but also a process, management and economic knowledge are important inside the software engineering domain. There are according to Sommerville [7] two major kinds of software present. The first kind is a system that is generic and anyone who is in use of its service may buy it. These systems sometimes refer to COTS (Commercial Of-The-Shelf) [7]. The second kind is a system that is strictly developed for one costumer and is totally designed to fit their needs, so called customised systems.

2.2 Fundamental Phases in Software Engineering There is no ideal process for software development. All organisations got their own processes that are appropriate to their specific need. According to Sommerville [7] many organisations still relay on an ad-hoc1 process without a defined software developing process. These organisations may although develop and deliver fully functional systems. In the first phase the requirements are elicitated2, tests are specified and project and system goals are defined in detail. It is important to have a proper handling of the requirements. If the requirements are not fully understood the whole development cycle is affected negatively. The requirements, tests and project goals are documented in specification documents. There are according to Sommerville [7] some fundamental and important phases that are recommended to occur in the organisation’s development cycle. These are described below.

2.2.1 Specification documents

This is not a phase and is mentioned because of the importance of having a proper documented system. Generally every phase generates one or more documents.

1 Something is done for the purpose in an informal way. 2 Collect or bring forward , for example when gathering requirements.

5

Documents that are produced during this first phase serve as descriptions and guidelines to the further work in the development cycle. The required documents are a Software Development Plan (SDP), Software Requirements Specification (SRS) and Software Verification and Validation Plan (SVVP) [22]. It is important to have a properly SDP with specific deadlines for each phase or iteration. The SDP also defines responsibilities inside the project organisation. A detailed description of the planned work with a detailed time plan is also recommended. The plan includes what to do and when to do it and who is responsible for the activity.

2.2.2 Requirement elicitation

The first phase is called requirement elicitation and as mentioned earlier it is of great importance as it serves as descriptions to further work in the development cycle.

Software requirement engineering includes the elicitation and evaluation of requirements. There are according to Sommerville [7] three different kinds of requirements on a system, functional, non-functional and domain requirements. The requirements are documented in a Software Requirement Specification. It defines the requirements of the system to be developed.

Functional requirements are requirements that relate to the services the system shall provide. Functional requirements may also state what the system should not do, if this is important. A rule when formulating functional requirements is to give a description on what the system shall be able to perform not how the system shall do it.

Non-functional requirements are requirements that do not have a direct connection to the functionality of the system. It can be aspects as usability, performance or other quality related factors. It may also concern factors like standards to be used, requirements that relates to business contracts or even ethical aspects of the system.

Domain requirements concern aspects that are necessary for the system to work properly in its environment. For example, requirements that state a specific standard must be used when communicating with other systems.

The requirements are to be documented and formulated in the SRS so that they are [22]:

Correct – are the requirements in agreement with the costumers’ desires?

6

Complete – are all the costumers’ requirements present in the document? Verifiable – are the requirements formulated in a way so that they may be tested? Unambiguous – is there only one interpretation of the requirement? Contradiction – is there one or more requirements that are in conflict to another requirement? Traceability – is it easy to refer to each specific requirement? Organised – is it easy to find a specific requirement? The above aspects may serve as a guideline when reviewing the SRS in the end of the specification phase. After the requirement elicitation another phase called the design phase begins. This phase shows how the system performs the requirements.

2.2.3 Design

The design phase contains activities like high-level design of the system followed by a more detailed designed and reviews on the documents made. The high-level design is an architectural specification where the different parts (sub-systems) are identified. The specification also shows how they relate to each other. The architectural design is followed by a more abstract specification of each subsystem and describes each sub-system properties. It is crucial to have reviews on the design before the implementation is done. The reviews are done to track down defects in the design according to the requirements. A short presentation of reviews is carried out in section 2.10. When working with the high-level design, design models are a good way to realise the design. A model is an abstract view of the system [7]. They serve as a link between the requirements and the implementation. The purpose of a design model is to show the different parts of a system and the relations between them. A modelling technique that is widely used in object-oriented development is Unified Modelling Language (UML) [26]. The company in focus of this bachelor thesis has decided to use UML techniques in their design. Because of this a more detailed description of UML is presented in section 2.8.

2.2.4 Implementation

In the implementation phase the code is physically developed and reviewed. The detailed and well-defined design in the previous phase is realised in code. Often there is some kind of coding standard to use e.g. how to structure the code, how to make comments in the code [29].

7

As it is rare with defect free systems there must be a well functional version and configuration process. It should be possible to trace the origin of the defect so that it can be corrected. In this phase unit testing is performed to verify that each unit fulfils its stated requirements. The implementation phase is not over just because the testing of the system is started. A system will continuously evolve and mature according to the presence of defects, adjustments or updates to be made and so forth.

2.2.5 Testing

If the previous phases have not been successful it will show in the testing phase. The aim is to test whether the whole system fulfils the requirements that were set up for it. In large-scale software project it is crucial to have a well-defined testing process to ensure that the system fulfils its requirements [7]. The software testing area is described in detail in section 2.9.

The work with the system is not over because of the testing phase is completed. The evolution phase described below point out this.

2.2.6 Evolution

The evolution phase includes all activities that occur after the program has been delivered such as technical support, how to handle new releases etc. This phase also includes software maintenance. By this means that the software is changed after it has been delivered. The changes can for example concern platform changes or defect correction According to Sommerville [7] there are three main reasons to way maintenance is done on a system: to repair software defects, to adapt the system to a new operating environment and to add or modify the system’s functionality.

8

2.3 Application of the Phases – Software Development Models The phases described above provides a basis in software engineering. There are a number of different approaches to combine these phases. There are according to Sommerville [7] many different processes that are used today. The approaches is an abstract representation of a software process, some of these are:

• The waterfall model. • Evolutionary development. • Formal systems development. • Reuse-based development. • Spiral model • Extreme programming • Incremental development

Some of the approaches are more general and others are more specified than others. Below follows a descriptions of some of them.

2.3.1 The waterfall model

The most static (in theory) method is the waterfall model. The characteristics for this model are that all phases are independent. Used in a strict way very little going back and forward between the phases occur, meaning that one phase should be completed before the next phase could be initiated. Figure 1 shows a schematic view of the waterfall model. This approach is used as a guideline in many software engineering projects. However, this is not how this model is used in reality. Usually there is some form of iteration3 between the phases. Iteration of a phase is done when for example something in the development cycle is changed. An example: suppose a person discovers a defect in a specific requirement in the design phase, there must be some form of iteration back to the specification phase to update or delete the requirements and thereby making a new version of the document, see figure 1.

Figure 1:The waterfall model, where every iteration generates a new version

New version New version New version New version

Iteration Iteration Iteration Iteration

Evolution Testing

ImplementationDesign

Specifikation

9

3 Repetition, rework.

The value of the waterfall approach is that it is simple to manage and often results in a stabile and changeable system. There are some problems with the waterfall model. Because of its static structure it is important to understand the requirements in an early stage. It is much more expensive both in time and economical aspects to respond to a change in the requirements in a later phase than the first [7]. A requirement related defect that is detected in the test phase might be up to a hundred times more expensive to handle than in the specification phase. The waterfall model is very logical and is used in large-scale system development.

2.3.2 Evolutionary development - prototyping

This approach can be used when the requirements are not well defined. There are according to Sommerville [7] two main approaches to evolutionary development, evolutionary prototyping and throwaway prototyping (see Figure 2).

The aim of evolutionary prototyping is to develop an initial model of the system from the outline of requirements and in an evolutionary approach refine the system as the requirements become more defined. This is done by providing the users with an incomplete system and from their opinions evaluate and refine the system model so that it in the end live up to the expectations of the customer.

The second approach, throwaway prototyping, clarifies the requirements through the initial prototype. The implicit requirements and specification on the system are made explicit by evaluation and modification of the initial prototype. When this is done the initial model is thrown away and from the specification made from the initial prototype, a formal system specification and an executable prototype are implemented. Figure 2: Evolutionary- and throwaway prototyping Sommerville [7]

Evolutionary prototyping

Throw-away prototyping

Executable prototype + System specification

Delivered system

Outline requirements

10

2.3.3 Incremental development

There are some additional methods that are applied in software development. One of them is called incremental development. It combines the simplicity of the waterfall model and an evolutionary approach.

The essence of this model is that the development is done through a series of increments or iterations. The customers give outlines of what services the system shall provide, the most important service first and so on. The highest priority requirements are included in early increments of the system implementation.

Each of these increments is a whole software development lifecycle including analysis, design, implementation and testing (Figure 3). The purpose is to implement an initial version of the system and receive user remarks (the analysis) and take this into consideration in the next iteration/version of the system. The advantage of this approach is, according to Sommerville [7], that costumers can gain value from the system before it is completed. Because of the early delivery the users can, from their gained experience from using the incremental releases of the system, come up with new requirements or change existing requirements. The new requirements will be implemented in future versions of the system. Due to that the high priority services are implemented early they are likely to get tested most. The later increments are integrated and adjusted to these and the system will tend to work correctly in the most important parts. The difficulty in this approach lies in the size of the iterations. The increments shall be fairly small, and they will still deliver some kind of functionality, Sommerville [7].

Final version

Figure 3: Incremental development

Testing

Implemen-tation Analysis

Design

Gain user feedback

Iterate development cycle due to user remarks

New version

This model is very customer-friendly as the model relies on the user opinions and takes this into account when developing the system. In this way end users

11

get a system that they are satisfied with. Extreme programming that is discussed in section 2.3.5, is an evolution of this approach.

2.3.4 Spiral development

The spiral development is a software process model that was proposed by Boehm [19]. The model can be described as a spiral movement where each phase represents one loop in the spiral (Figure 4). The characteristic of the spiral model is that it takes the risks, which occur in a software development project, into explicit consideration. There are no fixed phases like in the waterfall model, loops in the spiral are chosen depending on what is required. Each loop is divided into four different sub-phases; Objective setting, Risk assessment and reduction, Development and validation and finally Planning.

In the objective setting part of a loop, the goals (performance, functionality and so on) and project risks are identified for that specific phase. There is a possibility to make up alternative strategies depending on the risks identified. The Risk assessment and reduction is a detailed analysis of the risks and action to reduce the risk is carried out. In consideration of the risk analysis a decision on which development method to use when developing the system is made. The final sub-phase contains review of the project and if a decision of that the project will continue plans are drawn out for the next phase in the project.

1

Figure 4: The Spiral development model Sommerville [7]

2

2.4 Introduction to Quality The purpose of this thesis is to design a suitable test process for the company. To improve a test process and even come up with an opinion of how it is supposed to be constructed there is a need to introduce the concept of quality.

2.4.1 Quality - what is it all about?

The word quality is derived from Latin’s ”qualitas” which stands for nature, character or characteristic. In the international standard ISO 9000:2000 quality is defined as [15]: “The grade of underlying characteristics that fulfils the requirement, e.g. given need or expectation, that in general is implicit or obligatory”. A more current opinion of what quality is according to Bergman and Klevsjö [15] - Ability of a product to satisfy needs and expectation. The definition of quality is not general for all products. The products’ specific characteristics e.g. where, why and by whom it will be consumed, affects the quality criteria for a specific product. According to Bergman and Klevsjö [15] the ISO definition is not complete and suggest the following definition of the term quality: “The quality of a product is its capability to meet and preferably exceed, the customers needs and expectations”.

The mentioned product in this sentence can be any output of any process like goods, software or a service [15].

When handling quality based questions there must be an understanding that it is not always enough to just satisfy the primal needs and expectations of the costumer. There must be an aim to exceed the expectations and make the costumer pleased [15].

2.4.2 Development of quality related work

In the beginning of quality work, the work was controlled oriented, meaning that a system was tested to control that it fulfilled its specification. But since quality related work have over the past decades gained more and more focus the quality work has become more assurance oriented. Instead of measuring the quality aspects afterwards they were taken under consideration before being implemented. The development can according to Bergman and Klevsjö [15] be described as Figure 6:

13

… After production

…. Under production

… Before production

…. Continuous improvements before, during and after

Quality control

Quality management

Quality assurance

Quality evolution Figure 6: The improvement in quality related work

At the lowest level of quality assurance (quality control), the work concerns guarantee that the system fulfils its requirements and functionality. In the next step, quality management, there is an aim to check the system during the development and in an early stage locate defects in the system. In quality assurance a quality plan is developed and risks are eliminated before production. At the highest level, quality evolution, there is a continuous improvement strategy. Evaluation and analysis are made on the quality process.

2.4.3 Quality in software engineering

The quality aspects of software are complex. This is because the degrees of quality rely on the purpose of the final system and how well the system meets these purposes. There are some dimensions to quality to consider e.g. what aspects contribute to the products quality level? In [15] this dimensions are divided into service and product issues. Because the final system of software is a mix of both a physical product and a provided service for the end user, a set of aspects from both areas will be presented and discussed. Some quality aspects that contributes to software is presented here [15]:

• Dependability – this concerns how often the system fails to do what it is supposed to. In software, dependability defects can be catastrophic. Example, a software is designed to steer the automated breaking system in a car and in some way the software produces a defect that sets the breaking system out of use. This is a very important quality aspect of software.

14

• Performance – The performance aspect is about effectiveness, lifetime, size or speed. In software this type can be crucial. For instance a complicated formula will be calculated and it takes long time before the software delivers an answer. This can be a cause of poor implementation or poor structure of the code. Maybe the answer is presented with too much exactness, too many decimals, than what is needed and thereby fails in performance.

• Safety – The safety aspect concerns that no injury or damage is done on

humans or physical objects. The outcome from the lack of this aspects lies close to dependability. An example can be a medical device that automatically injects medical substance; if the software somehow contributes to that the wrong amount of substance is injected then the system is not safe to use.

• Maintenance – Maintenance is the ability to change or update the

system. In software this can be to add new functionality, to correct defects found or to add new functionality. For example, when software has to be updated due to changes in operating systems.

• Correctness – The correctness aspect concerns that everything works

the way it supposes to. For software this can be that all files that are needed are delivered (installed).

• Robustness – this is about how much of load, pressure or strain the

system can handle before it crashes or how much input a system can manage. This can be crucial to software that have network dependencies and have heavy traffic and user load.

• Usability – Usability concerns how easy it is to manage the functions of

the system. An example of a usability attribute is learnability (how easy the system is to learn). Another example of a usability attribute is understandability and concerns how well a system is understood.

2.4.4 Quality assurance in software development

Achieving high quality is about establishing a framework of procedures and standards that are used in the development process and assure that they are followed. When referring to standards there are two types to adopt when a quality assurance process is established namely product standards and process standards.

The product standard defines the outcome of elements from the process standard. It includes document standards, structure of documents produced

15

(what the document shall include), coding standard (how a specific programming language shall be used) and documentation standards (how comments shall be written, layout).

Process standard defines the process used when developing the product. It shall be clear what to do and when to do it. This includes definitions of what to do in the specification, design and validation phases and what the outcome in terms of documents should be.

Quality assurance in software development is about having a quality assurance plan for the whole organisation. The plan should specify what quality attributes that the organisation is adopting and then adopt this to specific software projects. Quality evolution has to do with process improvement and the aim to adjust the quality plan according to gained experience.

2.5 Software Process Improvement A software process can be described according to Paulk et. al [20] as: “A set of activities, methods, practises, and transformations that people use to develop and maintain software and associated products”. There are several different ways to improve a software process. In this document the focus is on the Capability Maturity Model (CMM) and the Rational Unified Process (RUP), as the company in focus of this thesis has the aim to adapt parts from these two concepts. There is an extensive work in the implementation CMM at the company. A document that describes CMM and which parts from it to adapt to the organisation has been developed within the company. The testing improvement work is a part of the whole process improvement at the company. When an organisation matures the software process becomes more defined and it becomes better implemented in the company. The software process is a guideline of how to develop software. In the process it is specified how the work shall be carried out and it defines what activities that occur in the different phases of the development cycle. There are some fundamental software processes that are explained here [20].

• Software process capability is the capability to predict the outcome from the next software project, if the software process defined in the organisation is followed.

16

• Software process performance is the actual result that comes out from applying the defined software process to a project.

• Software process maturity is the potential growth in capability and it

implies the richness of an organisations software process and in which way it is applied in project throughout the organisation. Companies have a mature or an immature software process:

o Immaturity – The characteristics for an immature organisation are

no processes are followed during project. If there is a software process defined, it is often the case that the developers do not follow it. This leads to that product quality is difficult to predict and deadlines and product quality are often relented to meet schedule and cost estimates.

o Maturity – The characteristics for a mature organisation are that

there is a defined process and that the personnel are aware of the importance of having it and follow it in their daily work. The schedules and cost estimates are based on historical data and are most often realistic. Mature organisations have a little deviation in the expected results and the actual achieved result. When organisation gain maturity it assures that policies, standards and organisational structures are maintained and that they endure after those who originally defined them have left the company.

17

2.6 CMM – The Capability Maturity Model This section introduces CMM and gives understandability over the concepts of immature and mature organisations in the software development domain.

2.6.1 Introduction to CMM

CMM serves as a guide for software developing companies in their aims to improve their software process. The purpose of the CMM is to make the development process more efficient and increase the quality of the systems. The model may also be used to find weakness and strength in an organisation and handle potential risks. CMM guides software companies in selecting process improvement strategies by determining current process maturity and identifying issues that are most critical for software quality and process improvement. CMM turns an immature organisation into a mature organisation with help of its five levels of process maturity. CMM helps the organisation to decide at which level they find themselves at the moment and what has to be done to reach up to the next level. Each level is well defined and described in the CMM. The sections below handle the CMM and its content.

2.6.2 Operational definition of CMM

CMM can be used in different ways, such as determine a company’s strengths and weakness, help of choosing subcontractors and monitoring the contracts, platform for the upper management to understand the activities in a process improvement program and finally as a template to guide a company through a software process improvement program. The common features of CMM can be organized into the following parts [20]:

• Commitment to perform – What actions have to be done to ensure that the process is established?

• Ability to perform – The resources, organisational structures and training that must exist to make it possible to start the work on the CMM.

• Activities performed – The different roles and procedures that is necessary to implement the Key Process Areas (KPA).

• Measurement and Analysis – The need to measure and analyse the metrics collected before, during and after the process improvement.

• Verifying implementation – The verification that the different activities are performed according to the process description.

18

2.6.3 The five maturity levels of CMM

According to Paulk et. al [20] a maturity level is defined as: “A well defined evolutionary plateau toward achieving a mature software process”.

The levels define process goals to be achieved and implemented in the organisations development cycle. Most of the companies that adapt CMM are at the lower levels of CMM. The fact that there are very few organisations at higher levels is that it takes several years and a lot of effort and understanding of process maturity to achieve high maturity in an organisation [7]. CMM plays an important role in defining software process improvement. The process goals mentioned earlier are the different keys in the Key Process Areas (KPA). KPA means that every maturity level has a set of goals that must be achieved for a project to be classified in a certain maturity level [20]. Further more all maturity levels except for initial level got a KPA that is a cluster of different issues that must be attained in the different maturity levels. Every issue must be attained otherwise a certain maturity level cannot be reached. The issues are divided into different goals that should be attained for each KPA. There are no fixed ways for achieving the goals; the differences in the projects applications and environments are the main issues of how to reach the goal. It is important to point out that there are also other key areas in a process, but CMM only points out the key areas that involve process areas that affect process capability. A description of each level and its key areas is presented below. 1.Initial (Worship the hero) - This is the first level in the CMM and the companies that are defined on this level are many. The organisation does not typically provide a stable environment for maintaining and developing software. Few processes and successful projects rely on a few individual heroic efforts. The software process can sometimes be characterized as chaotic because the software process is constantly changed or modified as the work progresses (ad hoc or “fire fighting”) [20]. Keys in level 1 As mentioned before this level does not contain any key process. 2. Repeatable (Plan the work) - Experience from prior similar successful projects is taken into consideration within the current project. At this level the policies for managing software process and procedures to implement those policies are established. An organisation at this level is considered to be

19

disciplined because the organisation is planning and keeping track of cost and schedules so that earlier successes can be repeated [20]. The organisation has a capability to have an effective control of the software process and should be able to set up and follow realistic plans based on previous projects. Keys in level 2 The keys at this level primarily focus on project management control [20].

• Requirements management - The purpose of this key is to create a common understanding between the projects and the customer’s requirements. The understanding with the customer is the basis for planning and managing the project.

• Software project planning - The purpose is to establish a reasonable

plan for manage and performing the software projects.

• Software project tracking and oversight - The purpose is to gain visibility over actual progresses so that management can take proper actions if the project is diverging from the plan.

• Software subcontract management - The purpose is to contract the

right subcontractors.

• Software quality assurance - The purpose is to give the management visibility over the process being used in the project and of the systems that is being built.

• Software configuration management - The purpose is to establish

and maintain the integrity of the system throughout the software project lifecycle.

3. Defined (Work the plan) - A standard for the process is documented and used in both management processes as well as in development processes. The software process is standardized and integrated into the whole software process. The aims for the processes that are defined and established at this level are to increase efficiency in the technicians and the software manager’s performance. At this level a group that are responsible for the software process activities is appointed in the organisation. The outcome of the initiation at this level is that management has good insight into the technical progress on all projects because the software process is now well defined. The process at a level three organisation can be described as standard and consistent because the software development and the management activities are stable and repeatable [20].

20

Keys in level 3 The keys at this level mainly focus on the project and the organisational issues, as the company stables an infrastructure that strive for effective software engineering and management processes in all projects.

• Organisation process focus -The purpose is to establish organisational responsibility for the activities that improves the process capability.

• Organisation process definition -The purpose is to develop and maintain

a usable set of process assets that improves process performance across the project and provide a basis for defining meaningful data for process management.

• Training program – The purpose is to give engineers the necessary

skills and knowledge so that they can perform their tasks as affective as possible.

• Integrated software management - The purpose is to integrate the

software engineering and the management activities into a coherent, defined software process that is tailored from the organisation’s standard software process and related process assets.

• Software product engineering - The purpose is to consistently perform a

well-defined engineering process. The process should integrate all the software engineering activities. The outcome is a correct and consistent software product. Software engineers describe the requirements, designs, codes and tests.

• Intergroup coordination - The purpose is to establish activities that

make the projects more adapted to the customers needs.

• Peer reviews - The purpose is to remove defects as soon as possible in a project. It is also important to develop a better understanding of the software work products and of the defects that can be prevented.

21

4. Managed (Measure the work) - At this level the organisation collects and analyse data available from the projects defined software process. There is an organised way to collect and maintain data from different software projects and careful measurements are used to analyse and provide a tool for the evaluation of the software process. The capability of the organisation is to be able to track exceptional circumstances and the special cause can be identified and addressed. The process at this level can be described as predictable because the process is measured and operates within measurable limits [20]. There is quality thinking and a distinct and well-defined quality assurance throughout the entire project. Keys in level 4 The keys at level four focus on establishing a quantitative understanding of both software process and the software work products being built.

• Quantitative process management - The purpose is to control the process performance and the software project quantitatively. Software process performance represents the actual results achieved from following a software process. The focus is on finding special causes of variations and keep logs of them.

• Software quality management - The purpose is to develop a

quantitatively understanding of the quality of the projects software products and achieve specific quality goals.

5. Optimising (Work the measure) - There is a continuous process improvement in the entire organisation. This is done by analysing defects and their causes and then uses the results to improve the software process. The software process capability can be characterised as continuously improving because the organisation is continuously striving to improve the range of their process capability [20]. Keys in level 5 The keys at level 5 cover the issues that both the organisations and the projects must address to implement continuous and measurable software process improvement.

• Defect prevention - The purpose is to find the causes of defects and prevent them from happening again.

• Technology change management - The purpose is to find new

technical equipment that helps in the quality work, it could for example be tools, methods or new processes.

22

• Process change management - The purpose is to continuously improve

the software processes used in the organisation. The improvements could be increased productivity, increased software quality and decreased project cycle time.

23

2.7 RUP - Rational Unified Process The Rational Unified Process is a software engineering process that is developed and maintained by Rational® Software. It provides a disciplined approach to assigning tasks and responsibilities within a development organization. Its goal is to ensure the production of high-quality software that meets the needs of its end users, within a predictable schedule and budget [25]. RUP enhances team productivity because it proposes that every team member have the same knowledge and view of how software development and its process can be managed. RUP proposes that this is done by letting team members have easy access to guidelines, templates and tool mentors for critical development issues in their work [25]. RUP is a guide for how to use the Unified Modelling Language (UML) that was originally developed by Rational® Software but is now maintained by the standards organisation OMG (Objects Managements Group). UML is a tool for eliciting requirements, clarifying design issues and architecture.

2.7.1 The “6 best practices” of RUP

RUP proposes 6 practices to consider when developing software. The practises are derived from the main practises used in successful software development projects. The 6 practices are:

• Develop software iteratively • Manage requirements • Use component-based architectures • Visually model software • Verify software quality • Control changes to software

Develop software iteratively because very few of the (often complex) software systems that are being developed can result in a complete and correct system in one iteration, meaning that in one sequence first defining the entire problem, designing the entire solution, implement the software and then test the system at the end [25]. An iterative approach to software development is proposed by RUP. See section 2.3 Application of the phases – software development models to get the differences between iterative and non-iterative approaches. RUP assigns the iterations four phases, namely Inception, Elaboration, Construction and Transition phase. These four phases makes a whole iteration that results in a new generation of the system [25].

24

Manage requirements through eliciting, document and organise required functionality and constraints [25]. RUP describes how this shall be done. By the use of use cases and scenarios requirements may be captured and clarified. These methods drive the rest of the process so that the final system is more likely to fulfil the end user needs [25]. Use component-based architecture: RUP focuses on early development and baselines for architecture. The process describes how to achieve a flexible and understandable design and supports component-based software development. A component is a non-trivial4 module that provides a specific functionality. Visually Model Software to capture the structure and behaviour of the system [25]. The UML is used to make visual abstractions to help clarify how different elements of the system fit together, control the consistency of the design and implementation. Verify Software Quality by reviewing the quality in aspects of reliability, functionality, performance and system performance [25]. In RUP is quality assessment part of all activities and all members of the project are involved. Quality assessment is a part of the development process and not performed after development or by a separate group. Control Changes to Software because that changes in software is almost unavoidable it is important to manage change. RUP describes how to control, track and monitor changes to enable iterative development [25].

2.7.2 The Dynamic Behaviour of RUP

In [25] RUP is described as two dimensional, dynamic and static. The dynamic dimension is the time dependent parts like phases, iterations, cycles and milestones. The static dimension of RUP is how the activities, artefacts, workers and workflows are described in section 2.7.3 One cycle is divided into four consecutive phases, namely Inception, Elaboration, Construction and Transition. Each phase has internal milestones that define key goals that should be achieved in the specific phase. Inception phase: This is a specification phase where outlines for the project and business goals are established for the system. This is done through identification of actors of the system and how they will interact with it. All use cases shall be identified and a few significant ones shall be described (10-20% of them fully developed). A business case that includes success criteria, risk 4 trivial = useless, unimportant

25

assessment, resource estimation for the project and a phase plan including major milestones. Development of one or several prototypes of the system is also included in this first phase [25]. Elaboration phase: in this phase the architecture of the system is considered. According to [25] this is the most critical of the four phases. At the end of this phase a decision whether to go on to the construction and transition phases is made. The architectural decisions made shall be with a broad understanding of the whole system and its scope, major functionality and non-functional requirements. An executable architecture prototype is developed in one or more iterations [25] where (at least) the fully developed ones from the previous phase are implemented. Additional outcomes from this phase is a more defined use case model (80% complete minimum), more defined requirements including non functional and requirements that do not correspond to any of the use cases defined. Construction phase: In this phase all components and features are implemented in the system. The outcome from the construction phase is a software system integrated in the adequate platforms, user manuals and a description of the current release. Transition phase: This phase concerns the activities that relates to the handover of the system to the costumer. Once this is done there will arise some issues that must be handled; correct problems, updates or even develop new releases.

26

2.7.3 The static view of RUP

According to Rational Software Corporation [25] a process describes who is doing what, how and when. In RUP there is four static modelling elements (Figure 7), workers, activities, artefacts and workflows. Workers: this element can be seen as a “hat” an individual can wear in a project. One individual may wear many hats. In RUP a worker is not the individual itself but rather the role defining how the individuals should carry out the work. Activities: There is a strong connection between the worker and the activities that are related to the defined worker role. Every activity is a unit of work assigned to a specific individual in a specific worker role to perform. Activities are an element that is a combination of planning and progress. Artifacts: This may be input to or outcome from activities performed by workers. It is the concrete things a project produces or uses while working towards the final system. Examples of artifacts may be a design model, source code etc. Figure 7: Workers, activities and artifacts in RUP [25]

Responsible for

Artifact

Use-case designDesigner Use-case analysis

Actvities Worker

Use-case Workflows: a workflow is a sequence of activities performed by different induviduals and can be described in UML terms as a sequence diagram, colloboration diagram or an activity diagram [25].

27

In RUP there are nine core process workflows. These are divided into supporting and engineering workflows [25]. There are three supporting workflows:

• Project management workflow • Configuration and change management workflow • Environment workflow

The project management workflow presents an approach to managing the project and it specific focuses on iterative development process [25]. The configuration and change management workflows provide guidelines for managing multiple variants of software systems. The environment workflow shall provide the software development environment such as processes and tools that are needed to support the development team [25]. There are six engineering oriented workflows:

• Business modelling workflow • Requirements workflow • Analysis and design workflow • Implementation workflow • Test workflow • Deployment workflow

The difference between these workflows and traditional phases in a waterfall approach is that these workflows are revisited over and over again throughout the whole lifecycle due to iterations. The business modelling workflow uses business use cases to gain a common understanding among stakeholders of what business process that need to be supported in the organisation. The requirements workflow works with use cases to manage and identify the requirements and the actors of the system. This workflow is about what the system shall do while the analysis and design workflow deals with how the system shall do it. The design and analysis are realised in models that captures the major structural design decisions. The implementation workflow describes how to reuse components, or implement new components to make the system maintainable and reusable. The test workflow describes how to manage test from different quality aspects, reliability, functionality, application performance and system performance.

28

The deployment workflow contains activities related to releases. This workflow is less detailed than other workflows [25].

2.8 Design Using Unified Modelling Language (UML) UML (Unified Modelling Language) is a modelling technique that is widely used in object-oriented development. UML is a graphical modelling language with its own syntax to describe different relations and dependencies between objects or classes. UML provides both static and dynamic (behavioural) system representation. A dynamic or behavioural model is one that shows how a specific object responds and acts (behave) to a specific event. A static model is one where the system object or classes with their role and relationship to other objects/classes are described. Today many companies use UML in CASE-tools (Computer Aided Software Engineering tools) which makes the work with the different types of diagrams much more efficient [7]. UML provides different modelling techniques, a few of them is described below. Class diagrams – Class diagrams are used to document the static structure of the system. Which classes there are, how they are related to each other but not how they interact. A class is represented as a rectangle with its name, attributes and operations written inside, see Figure 8. The classes have associations that state the relationship between them. When a class diagram is complete the next step is to make CRC cards (Class Responsibility Collaborator) to get a better overview of the different classes, see Figure 9. A CRC card shows the class name, the class responsibility and which classes it collaborates with.

Figure 9: CRC card Figure 8 : Class modell

class

class

class class association

Name

Responsibilities Collaborators

29

Use Case models – Use cases shows the behaviour of the system from the users point of view. A user could be all from a human, hardware or an external machine that interacts with the system being developed. Use case modelling helps with capturing requirements, planning iterations of development and validating systems. According to Pooley [26] these three aspects are the most difficult. The use case diagram describes the interaction (task) between an actor and a use case. For example, a “book borrower” (actor) relates to “borrow a copy of a book” (use case), see Figure 9. An actor represents a role that someone/something plays. The use case points out a task that the actor might do or might be involved in.

Figure 9: Use case diagram

Use case – borrow a copy of a book Use case – return a copy of a book

Actor - book borrower Interaction diagrams – Interaction diagrams give the possibility to record how objects interact in detail. The main purpose is to realise particular scenarios in a use case. Also the use of CRC cards can help explore the interaction between objects and how they interact. There are two types of interaction diagrams: sequence (Figure 10), and collaboration diagrams. They are quite similar. A collaboration diagram point out the links between objects and sequence diagrams point out the messages that passes between objects. m

3

Figure 10: Sequence diagra

Command

class class

Actor

0

State and activity diagrams – State and activity diagrams describes how an object should respond when it receives a message. This makes sense as a class object is affected of the values of the incoming attribute. In order to implement, maintain or test the class it is important to understand the state of an object and its reaction to message or event. A state diagram records these dependencies. The activity diagram on the other hand describes how activities are co-ordinated. They are often used when showing that an operation has to achieve a number of different things, and shows this much better than an interaction diagram. Activity diagrams are also useful when describing how individual use cases unfold and depend on other use cases. Figure 11 describes a state diagram where the action is a reaction from an event taking by the system.

Figure 11: State diagram

State

Action

Action

State

Implementation diagrams – There are according to Pooley [26] two types of implementation diagrams namely component diagrams and deployment diagrams. Component diagrams express the structure of the implemented system. It helps keeping track of dependencies to ease maintenance, and to record the reuse of components. Deployment diagrams show how the system is deployed on a particular hardware configuration. In general the implementation diagrams are automatically generated by CASE-tools that generates the outline of the code, for example class definitions where you have to fill out the details in the methods. Package and subsystem models – A package is a collection of model elements. A specialised package is a subsystem model that describes a component. A model includes the model elements, which shows a particular view of a system at a certain level of abstraction. One or more diagrams represent a model.

31

2.9 The Software Test process The main purpose of verification and validation is to secure that the developed system meets its specification and if the developed system is what the customer wanted [7]. Verification and validation involve not only testing, it also concerns reviews (chapter 2.4.3) according to Sommerville [7]. Two testing strategies are black-box and white-box testing [21]. In this section, a discussion of why software testing is important is provided, together with a description of the main testing strategies.

2.9.1 Why is software testing important?

Software is today a major part of the new technology. Weinberg et al. compare software engineering with other engineering disciplines [11]. The software engineering discipline is in contrast to other engineering disciplines a comparatively young discipline (about 40 years) [11]. Compare this to construction engineering that goes back 5000 years and have approximately 125 times longer evaluation time than software engineering. The extreme progress in the software domain has increased the applications of software. This on the other hand has set demands on the process. The need for defined software development process is crucial to achieve large reliable software systems. Today the defect rate in software is high compared to other engineering disciplines. Gerald Weinberg says, “If builders built buildings the way software people build software, the first woodpecker that came along would destroy civilization” (Weinberg’s law) [11]. But this does not happen, why? This is because they have a functional and defined development and testing approach. Even if construction engineering does not physically test their finished system, they have during the specification made calculations on the system (construction) to certify that it meets its requirements and calculated risks, (quality assurance before production, section 2.4.3) like strength (reliability) and architecture (design). This has to be the aim for the software industry as well if the quality and reliability of the systems shall increase. It is important to understand, manage and perform the software testing to certify that the final system meets its requirements.

2.9.2 Strategies in software testing

As mentioned in the introduction to software testing by Kit [21], there are two fundamental testing strategies. The strategies are black-box testing and white-

32

box testing. Kit [21] also points out that these are strategies and not technical or analytical methods. Different methods in the two strategies are used in different activities; there are two major activities, called low-level and high-level activities [21]. Generally, low-level activities concern the white-box testing types and methods. High-level activities concern black-box testing types. Black-box testing Black-box testing is derived from the functional requirement specification and is done without any internal knowledge of the system [21]. The aim of method is to test the system according to the end users’ requirements. By studying the inputs and outputs through a system, it is possible to check if the system meets its specified functionality. This is also why black-box testing does not reveal any hidden functions [21]. Black-box testing is therefore only concerned with the functionality and not the implementation of the software [7]. Black-box testing is most often a high-level activity and is executed late in the software development process. However, this is dependent on the development process used (see chapter 2.1). The types of tests performed in black-box testing are [21]:

1. Usability testing – Kit [21] points out that the goal is to adapt a program to the user and not vice versa. The tests should be conducted as early in the development as possible and its purpose is to check how a user responds to the program, even if it only mean checking non-functional interfaces.

2. Function testing – Function testing is when defects are actively

searched in a program; the main issue is to find defects. The tests are derived exclusively from the functional specification and begin as soon as there is something to test, often after unit and integration testing.

3. System testing – System testing is according to Kit [21] the most

misunderstood and most difficult testing form. Its purpose is to demonstrate that the program does not meet its requirements and objectivities. The tests are for example conducted from the users point of view, how the user uses the system or from the requirement specification. The difficulty lays in the lack of design methodology for creating test cases and therefore requires a whole lot creative thinking by the tester. Instead of the use of a methodology, system tests are executed from categories and different type of tests. Some of the tests

33

that Kit [21] points out are performance testing, usability testing and security testing.

4. Acceptance testing - Acceptance testing is carried out after the

integration of all the components in a program. The purpose of acceptance testing is to check whether the final system meets the customer’s requirements [23]. It is often divided into two parts, Alpha and Beta testing [21]. The company developing the software does the Alpha test, and the users do the Beta test. The acceptance testing is not over until the customer and the company have come to an agreement about the outcome of the tests [7].

The test methods most frequently used in black-box testing are [21]:

• Equivalence partitioning – The basic idea is to identify the classes with similar outcome depending on the inputs that are tested. Equivalence partitioning got two distinct steps, to identify equivalence classes and to identify test cases.

• Boundary-value analysis – The basic idea is the same as with

equivalence partitioning but with two major differences. The first is to check the boundary values in each class and the second is that rather than focusing exclusively on inputs, outputs are also explored by defining equivalence classes for outputs.

• Error guessing – The basic idea is to make a list out of possible errors

and error situations made out from experience and intuition. The list is then translated into test cases.

White-box testing In opposite to black-box testing, white-box testing requires knowledge of the internal structure of the software system. As black-box testing does not test hidden functions, white-box testing does not test missing function [21]. The tests conducted in white-box testing are derived from the design specification or the code [21]. According to Sommerville [7], white-box tests are structural tests and are derived from the knowledge of the system and implementation. It is also called glass-box testing or clear-box testing [23]. In White-box testing there are different types of tests. The test types explained by Kit [21] are:

34

Unit (module) testing – Unit (module) testing is when tests are conducted on separate subprograms or procedures in a program. The aim is to find discrepancies between the actual behaviour of a module’s interface and the specification of the interface. Integration testing – Integration testing is performed after have had tested individual components and parts of programs (see unit testing above). To avoid the difficulties in finding where the defects are located, an incremental approach may be used, which adds more and more program modules. There are two incremental ways of doing this [7]:

• Top-down testing – The basic idea in this approach is that you test the high level components before the design and implementation are completed. In Figure 13 below, the boxes marked, as Level is components. They get tested with the stubs and then become Level 2 components. This goes on until you reach the bottom level, and the whole system has then been tested. Stubs have the same interfaces as the component but have very limited functionality.

Figure 13: Top-down integration testing Testing

sequence Level 1 Level 1 Level

2 Level 2 Level 2

Level 3 stubs

• Bottom-up testing – This type of testing does not require any architectural design, so the testing could start early in a project [7]. When doing the tests, you start at the bottom levels and work your way up through the different levels until you reach the top-level. The test drivers are simulators that simulate the modules environment at a higher level. This goes on until the final module is tested (Figure 14).

]

Testing sequence

Test drivers

NNN

1

Test drivers

Level

Level Level

Level N-

Level N-1

Figure 14: Bottom-up integration testing [7

35

There are different methods to ensure that the testing cover the whole system, some of these are presented below. Statement coverage Statement coverage is defined as: “Each statement is executed at least once” [21]. To execute each statement at lest once in the Figure 12 below there are two possible paths (1 and 2) and the path number are: <2,10,12,14>, <1,3,5,6,7,8,9>.

Figure 12: statements, branches and path in a program

3

1

11 13 12

4

5

9

7

8

6

2

10

14

Statements Branch Decision (branch) coverage Decision (branch) coverage is defined as: “Each statement is visited at least once; each decision takes on all possible outcomes at least once” [21]. To put this in numbers and writing consider Figure 12 again. Then there would be four possible paths (two possibilities for path one and two possible paths for number two). The path number are: <2,11>, <2,10,12,13,12,12,14>, <1,3,5,6,9> and <1,4,6,7,8,9> in figure 12 above. Condition coverage Condition coverage is defined as: “Each statement is covered at least once; each condition in a decision takes on all possible outcomes at least once” [21]. The path covered are: <2,11>, <2,10,12,14>, <2,10,12,13,12,14>, <1,3,5,6,9>, <1,3,5,6,7,8,9>, <1,4,6,9> and <1,4,6,7,8,9> in Figure 12 above. Path coverage Path coverage is according to Kit [21] generally impractical because of that they are exhaustively, which means that they are too thorough and time consuming. Kit [21] also says that the same efficiency results could be gained from using combinations of other white-box testing methods mentioned above.The path coverage is: All paths (infinite number): <2,11>, <2,10,12,14>, <2,10,12, (13, 12)n, 14>, <1,3,5,6,9>, <1,4,6,9>, <1,3,5,6, (7, 8)n, 9>, <1,4,6, (7,8)n, 9> (any n>0)

36

Simple paths (6 possible paths) numbered: <2,11>, <2,10,12,14>, <2,10,12,13,12,14>, <1,3,5,6,9>, <1,3,5,6,7,8,9>, <1,4,6,9>, <1,4,6,7,8,9>. Linearly independent paths (seven opportunities) numbered as: <2,11>, <2,10,12,14>, <2,10,12,13,12,14>, <1,3,5,6,9>, <1,3,5,6,7,8,9>, <1,4,6,9> and <1,4,6,7,8,9>. The methods mentioned above are most often used during low-level testing.

2.9.3 Additional test types

There are many types of tests in a test process, but Kit [21] points out that there are mainly two test strategies (see Black-box and white-box testing above). But there are also some other test types that are important to mention. Two of these are stress and regression testing, which are described below. Stress testing Stress testing is carried out to measure the performance in a system. Some performance tests have to be done so it is guarantied that the system can handle its intended loads. Stress testing is designed to control how the system handles failures when it is heavily used; the system my act strangely when it is heavily used. Depending on the kind of system that is developed, these tests are of different importance.The stress tests could be everything from handling a certain amount of transactions every minute in a bank to handling the amount incoming calls to an SOS alarm central during a crisis. Sommerville [7] also puts stress testing under the integration test phase, because that it is possible to conduct the test as soon as the system is integrated Regression testing Regression testing is not yet another test type of test [21]. Instead, when modifications are made within a software program, the test team should perform a regression test Regression testing is performed by rerunning existing test cases to determine whether the changes have affected anything that worked before. According to Somerville [7] all tests should be repeated, but in practice this is too expensive. Instead, as part of the test plans, the dependence between different test cases and parts of the system should be stated.

37

2.9.4 When to test what?

There are different phases in a testing process, and therefore should different tests be done throughout all phases in the software development process (see chapter 2.1).

Oshthca

38

Figure 15. The testing process

Unit Module

Sub-systemSystem

Acceptance

Component testing Integration testing Usage testing

ne of the difficult matters of software testing is what and which tests that ould be conducted e.g. which test cases that shall be developed to assure that e program is thoroughly tested. According to Kit [23] a well functional test se shall have the following characteristics:

• Reasonable probability of catching a defect, test to find defects.

• It shall not be redundant, if two tests locate the same defects, why run both?

• It is the best of its breed, if similar tests occur the one that is most likely to find the error is preferable. For example boundary values are more preferable is input than non-boundary values because they are more likely to demonstrate a defect [23].

• The tests are not supposed to be neither too simple nor too complex, rather use simple tests than complicated combined tests with the aim to test several things at once. These tests can be hard to execute and understand.

It makes program failures obvious, the tester must be aware how the test will present its result e.g. how does the tester know if the outcome reveals a defect or not. To make this easier, write down the expected output of each test, make the output from the test as short as possible to avoid that failures is hidden in a mass of boring print [23]. The testing of a system is not the only activity to find defects in a system, reviews is another activity.

2.10 Reviews Reviews are something that every organisation that is concerned with quality assurance should manage. The purpose with reviews is to manually read and look for defects and weaknesses in both implemented code as in specifications for the system. In this section reviews in the software development area is discussed. Reviews help to discover defects and to ensure system compliance to specification, standards or regulations. The main purposes of these reviews are according to Jarvis [17] the following:

• Identify required improvements in a system • Assure that the deliverable is complete • Assure that the deliverable is technically correct • Measure the progress of the project • Identify any defects early, thus resulting in cost and time savings

The objective for review meetings is to find the problems not to solve them, this is important to have in mind when actually performing the meeting. Stay focused on the existence of possible defects found and not on the solutions on these. In [17] there are some general guidelines for reviews.

• Preparation: It is important that the participants of the review get the related document(s) in advance and that time for preparation for the review is set of for the reviewers.

• Discussion: The discussions that arise during the review shall be

limited. The issues of the discussions shall be recorded so that they can be brought up on another meeting.

• Respect: The viewpoints that are conducted shall be objective and

criticism of the author shall be avoided. Focus on the substance and not the style.

39

• Agenda: Have an agenda that is strictly followed during the meeting. If

discussions occur remind the attendees gently about the purpose and time limit of the review meeting.

• Review records: It is important to take precise notes on the

issues/viewpoints raised on the meeting. Make sure that every attendant is aware of what is being recorded. Also set a resolution date of every issue.

• Resources: Make sure that the reviewers have gained formal training in

review and are aware of how the reviews shall be preformed (Guidelines and standards).

• Attendees: Keep the attendees at a suitable level, if the number

attendants are to many there is a risk that the focus is lost. Make sure that every attendant is prepared for the review.

The reviewing shall be a continuous process throughout the entire development cycle. In between the different phases of the development cycle review should be present e.g. in the beginning and end of specification (requirements), design, implementation (coding) and test phases. When for example reviewing a specification document, such as Software Requirements Specification (SRS) or Software Validation and Verification Plan (SVVP). There are some things to consider, of course there is more to consider in detail, this is just a brief glimpse of what should be checked.

• Clarity: Are the requirements unambiguous? Are the requirements understandable?

• Completeness: Is there a definition of terms? Are all requirements

present? Is there any requirement that is not fully explained?

• Consistency: Are there any requirements that are a contradiction to another requirement?

• Verifiability: Are there any requirements that are impossible to

implement?

40

In general, when reviewing any document the idea is the same as in the example given above, to check for clarity, completeness, consistency and verifiability.

41

3 METHODOLOGY AND TECHNIQUES In this section the methodology and techniques that were applied and studied during the work are presented. The methods for analyse and information gathering are the GQM paradigm and interviews. In this section an introduction to these methods is presented.

3.1 GQM – the Goal Question Metric paradigm GQM is a method that was proposed by Basil and Rombach in 1988 [2]. The purpose was to help decide what measurements should be taken and how they should be used. The method relies on three phases or levels, conceptual, operational and quantitative that interact with each other, namely Goals, Questions and Metrics [2]. GQM is a hierarchical structure (Figure 17) and refines an organisation’s goals and objectives and use them in a way so that a measurable result comes out from a set of questions that relates to these goals and objectives. The first step in the top-down refinement of the objectives is to set out the goals that should be the purpose of the questions. These goals clearly show what the purpose of the questions that relates to that specific goal is. The goals could serve as a tool to improve the working process. They may also help to refine and clarify those implicit models that occur inside the organisation. The outcome of the questions is the bottom-up analyses and interpretation of the metrics so that the result could help the organisation to understand, monitor, evaluate, predict, control and improve their software development process [1]. Figure 17: The Goal Question Metric

Bottom – Up

Analysis and interpretation Questionnaire

Top – Down refinement

metrics z2 metrics z1 metrics y2metrics y1metrics x2 metrics x1

Question z Question yQuestion x

GOAL

42

3.1.1 Goals – the conceptual level

The goals clarify what the organisation wants to achieve. There are five issues that should be considered when formulating the goals. These are according to Latum et al. [1]:

1. Object to measure 2. Purpose of measure 3. Quality focus 4. Viewpoint 5. Environment

An example (made-up) of such a goal could be “Analyse the X-system with the aim to minimize the faults done by the user so that the usability can be improved by the software development team at the company development department”. In this goal the Object to measure (1) is the ‘X-system’, the Purpose of measure (2) is ‘the aim to minimize the faults done by the user’, the Quality focus (3) is ‘usability (can be) improved’, the Viewpoint (4) is ‘the software development team’ and finally the Environment (5) is ‘the company development department’.

3.1.2 Questions – the operational level

At this level the aim is to refine the goals that were set up at the previous level and develop questions that will serve as a basis for the metrics. There are two main parts of this level, knowledge acquisition and measurement planning [1]. In the first part, knowledge acquisition, the aim is to capture the current knowledge inside the project or the organisation and make representative models of it. The knowledge could be gained through interviews where questions with both quality and quantitative focus are present. The aim is to elicit the implicit (hidden) models that occur inside the organisation or project and make them explicit (visible). It is important to understand the project members’ assumptions and definitions to be able to explain the models used and relate them to the goals. During the second part, measurement planning, a plan that describes the metrics and procedures that should be followed during the metric-phase is developed. The plan contains name and definitions for each unique metric that is to be measured. Description on how and when the data is to be collected, classification for each metric, definition of the data collection forms,

43

procedures for reporting, collection and validation of data and finally references to the metrics in the GQM plan.

3.1.3 Metrics – quantitative level

The metrics are closely related to the questions. The data that comes out from the questions is analysed and interpreted in order to answer the question that the data is associated with. There are two kinds of data, objective and subjective [2]. Objective data is data that only concerns and are affected by the object that being measured and by no other influence. This can be data like “ the number lines of code in module X” or “number of defects discovered in module Y”. Subjective data is data that is affected by more than one factor. This can be data like “usability of module X” or “Understand ability of document xxxx”. These two examples are, besides the data itself, influenced by the users opinion about what usability or Understand ability are. GQM is one method to collect information, but it is rather complex and could be difficult to handle. Another method is interviews, which also has its problems. The following section presents interviews in a broad manner and what to think of when conducting interviews.

3.2 Interviews Interviews are a good way to collect information from people involved in the area of investigation. There are different kinds of interviews and different kinds of questions. Some different interview techniques and kinds of questions are presented in this section. A difference between, for example, questionnaires and interviews is that in an interview the interview subject’s feelings can be observed due to their face expression [6] [3]. Also the sound of his/hers voice and the pauses that he/she takes may be hints of what the person is feeling. Another difference is that follow up questions to get a clearer view of what the person being interviewed means could be asked. A thing to consider is that the interview approach take up a lot of the interviewers’ own time. Interviews are often explained as a regular conversation between two people, as if they where exchanging information [4].

44

3.2.1 Different types of interviews

According to McNamara [24] there are four main types of interviews, informal conversational interview, general interview guide or semi-structured approach, standardised open-ended interview and closed fixed-response interview. The informal conversational interview approach is one where no predetermined questions are asked. The aim is to keep the conversation as open and adaptable as possible to the interviewees’ nature and priorities. The interviewer has a “go-with-the-flow” [24] approach during the interview and has the possibility to pursue questioning in whatever direction appears to be appropriate. The strength of this approach is that the interviewer is flexible and highly responsive to individual differences, situational changes and emerging new information. The weakness is that it may generate less systematic data that is difficult to classify and analyse. The general interview guide approach or semi-structured interviews are somewhat more structural than the informal conversational approach. Its intentions are to ensure that the same general information is collected from each interviewee. A pre-determined set of questions or issues is being explored during the interview. The advantage of the interview guide approach is that it makes collecting of data from different interviewees more systematic and comprehensive from having pre-determined issues to be taken up in the interview. There is still the possibility to ask additional questions as long as the pre-determined set of questions have been asked. When there are several interviewers the standardised open-ended interview approach is well suited. The same open-ended questions are asked with the same words and in the same sequence to all interviewees. This facilitates the evaluation of the interview because it allows the evaluator to collect detailed data systematically and facilitate comparability among all interviewees. An open-ended question is where respondents are free to choose how to answer the question e.g. no given answer alternatives is given, like “yes” or “no” [24]. The closed fixed-response interview is an approach where all the interviewees are asked the same questions and has a set of answering alternatives to choose from. This approach can be good for first time interviewers.

45

3.2.2 Different kinds of questions

A question should be structured and asked in an appropriate way. Depending of what kind of answering wanted there are different types of questions to write. According to McNamara [24] six types of questions are used. Questions may be asked in different tempus, past, present or future. The questions may concern different topics, such as:

• Behaviours: what a person has done or is doing. • Opinion/values: thoughts about a specific topic. • Feelings: what does a person feel about a specific topic • Knowledge: to get facts • Sensory: seen, touched, heard, tasted or smelled. • Background/demographics: age, education etc

3.2.3 Formulating the questions

It is important not to get carried away in the excitement of writing questions and get more and more eager to find more and more clever questions. When formulating a question there are a lot to considerate, for example [24]:

• Different people interpret questions differently, how should a proper question be asked? What kind of answers do I want? Which language should I use? To whom am I asking the questions?

• To ask questions in a way so that they all get answered, therefore it is

very important to be consistent and not cause any confusion.

• The relevance of a question. Are the questions relevant for the ones that are being asked?

• Leading questions are not recommended, since they may deliver a

mislead answer which the interviewer does not have any use of when analysing the survey. The same argument is used when it comes to questions where you have to weight something against something else.

• Offensive questions (questions that force a specific answer) are

something that the interviewer should handle with biggest caution, when using this kinds of questions make sure to get the answers primary needed, in other words ask the offensive questions late in a questionnaire or use category answers.

46

3.2.4 Preparation for the interview

When preparing for an interview, recall the base rules for formulating the questions. A bit more preparation is needed to ask the questions in person. In other words it is important that the questions work in a proper way. A difficult part when conducting an interview may be to get all the answers on paper, often the spontaneous answer is wanted and not the edited spontaneous answer. Before starting to design the interview questions, be sure to understand the problem or need that is addressed in the interview. The information gathered from the interviews should contribute to the solution of the problem [24]. Because of the face-to-face situation there are some things to consider. According to M Patton [24] there are some guidelines to have in mind before starting the interview:

• Choose a setting with little distraction to make sure that the interviewee is comfortable (ask them if they are). Think of that loud noise or strong light might be disturbing.

• Explain the purpose of the interview by informing which problems to be

solved?

• Address terms of confidentiality by explaining how their answers will be used and analysed. Also tell them who will get access to their answers.

• Explain the format and nature of the interview and its procedure.

• Indicate how long the interview usually takes.

• Tell the interviewee how to get in touch with you later if they want to.

• Ask the interviewee if they have any questions before starting the

interview. Do not count on the memory to recall their answers - ask for permission to either record or take notes of the information from the interview.

47

3.2.5 Carrying out the interview

There are some things to consider during the interview [24].

• If a tape recorder is used occasionally verify that it works properly. • Ask one question at a time. • Attempt to remain as neutral as possible (do not show strong reactions

to the answers). Encourage responses, like nodding or with “uh huh” [24].

• Do not take notes to obvious; it might give signals to the interviewee that may influence answers to upcoming questions.

• Make smooth transitions when changing topic like “ we have been talking about… I would like to move on to…”

• Do not lose control over the interview, have control over the time and keep focused on the topics of the interview, stay in charge.

After the interview it may be good to make sure that the notes taken are understandable. Also fill out missing information when necessary. As it said before do not count on your memory to remember what has been said during the interview. In the following section the Analytic Hierarchic Process (AHP) is introduced. AHP was used to prioritise quality aspects during this thesis.

48

3.3 The Analytic Hierarchy Process (AHP) The Analytic Hierarchy Process is a widely used prioritising method that was developed by Thomas L. Saaty [28]. AHP is pair-wise comparison method and allows individuals to use their own psychometric scale to making comparisons between criteria. There are some steps in the AHP:

• Set up a criterion for the prioritisation.

• Make a criterion question that makes every pair-wise comparison focused on the criterion.

• Perform the prioritisations by letting appropriate people weight

elements in focus for the prioritisation against each other.

• Analyse the result and its consistency. AHP can for example be used as a tool when prioritising requirements for software systems. A more complete description is given in section 3.4.2, where an AHP prioritisation is conducted.

49

3.4 Survey methods applied In this section a description is provided of how the methods and techniques in this thesis were used.

3.4.1 Using interviews to collect information from the employees

The interviews were carried out on employees of the company in focus. Interviews were conducted of developers, testers, support and management. The purpose with the interviews is to map the work and to be able to come up with proposals on process improvement. A semi-structured interview approach has been used during the interviews. The questions have been developed by defining goals and purpose of each interview and out from this formulating the questions. Goals and purpose of interview with a developer at the company:

• Map the current development process • The opinion and knowledge of testing • How testing is performed now

Goals and purpose for interview with support personnel at the company:

• Map the work for the support personnel • The support personnel’s opinion on testing

The questions were developed with the aim to gain understanding of how the current development process and testing are performed. The questions were divided into phase related questions such as Requirement, Design, Implementation and Testing. The questions were asked with an open discussion as a result. A summary of the interviews is in Appendix. Notes were taken on the answers and later a summary was made (see Appendix).

3.4.2 Using AHP to prioritise quality aspects on the company’s system

AHP is in this thesis used to prioritise quality aspects on the company’s current system. The aim is to see which quality aspect that contribute to the company’s current system the most and out from this identify the most important aspect with respect to testing. Seven quality aspects is evaluated, dependability, safety, maintainability, performance, correctness, usability and robustness. This is done through a form that was distributed by e-mail to 25

50

employees at the company. 14 of them returned and answered. Because of the somewhat low answering rate the result analysed in section 5.2 may not be complete. Developers, product management and support personnel made the weighting of the quality aspects. The weighting was made through an Excel-worksheet (see Appendix) and was analysed in another worksheet (see below). The prioritising was done as follows [28]: n(n-1)/2 (n = number of attributes to be prioritised) comparisons that compares every attribute to another, in this case 21, was performed by the personnel at the company. An example of how a pair wise comparison between Dependability and Performance was performed is shown in the sheets below. The example below relies on the data in the form in Appendix E.

Attribute A moreimportant

Attribute B more important

Attribute A 9 8 7 6 5 4 3 2 1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 1/9 Attribute B Dependa-bility X Performance

Then an n x n prioritising matrix was set up to evaluate the result [28].

Priority matrix

Dep

enda

bilit

y

Saf

ety

Mai

nten

ance

Per

form

ance

Cor

rect

ness

Usa

bilit

y

Rob

ustn

ess

Dependability 1,00 6,00 5,00 3,00 6,00 0,25 5,00 Safety 0,17 1,00 0,25 0,20 0,17 0,20 1,00 Maintenance 0,20 4,00 1,00 0,14 6,00 0,14 6,00 Performance 0,33 5,00 7,00 1,00 3,00 3,00 7,00 Correctness 0,17 6,00 0,17 0,33 1,00 0,25 4,00 Usability 4,00 5,00 7,00 0,33 4,00 1,00 5,00 Robustness 0,20 1,00 0,17 0,14 0,25 0,20 1,00

Above: Priority matrix where the values from the prioritising are inserted and the inverted relation is 1/prioritision-value. In this example 3 and 0,33 (bold values) comes from the comparison above the matrix.

51

A normalised matrix is then calculated where all the elements in the prioritising matrix are divided with their corresponding column sum.

Then a priority vector is calculated from the normalised matrix as: (row sum/n)* 100 (results in a percent ratio of the n attributes).

Normalised matrix

Dep

enda

bilit

y

Saf

ety

Mai

nten

ance

Per

form

ance

Cor

rect

ness

Usa

bilit

y

Rob

ustn

ess

Dependability 0,15 0,21 0,24 0,32 0,29 0,05 0,17 Safety 0,02 0,04 0,01 0,06 0,01 0,04 0,03 Maintenance 0,03 0,14 0,05 0,05 0,29 0,03 0,21 Performance 0,15 0,18 0,34 0,32 0,15 0,59 0,24 Correctness 0,02 0,21 0,01 0,11 0,05 0,05 0,14 Usability 0,59 0,18 0,34 0,11 0,20 0,20 0,17 Robustness 0,03 0,04 0,01 0,05 0,01 0,04 0,03

Priority vector To be able to decide if the prioritising is valid or consistent a consistency ratio (C.R.) must be calculated and it is done as follows: First a matrix multiplication is done, the prioritising matrix * the priority vector. The resulting vector is then divided element by element with the priority vector. The

vector is then summarised and an average is calculated over the n elements, now we have λ-max (which shall be as close as possible to n, in this case λ-max = 7,61) [28].

Dependability 20,6% Safety 3,1% Maintenance 11,4% Performance 28,1% Correctness 8,4% Usability 25,5% Robustness 2,9%

A consistency index, C.I., is then calculated as (λmax – n/n-1). The C.R is then calculated as C.I./ R.I. (where R.I. is the Random Index) [28]. If C.R. is < 0.10 the result is good. In this case the C.R is 0,077 so it is inside the limit. If the C.R. is not < 0,10 there must be a decision whether having a new prioritising, keep the result or discard that specific judgement.

52

4 DESCRIPTION OF THE COMPANY In this section the company in focus is presented with its organisational structure, products, work process, problems and goals.

4.1 Organisation As mentioned in section 1 of this document the company is a small-scale software developing company that provides systems for chain of retail store organisations. The company is mainly divided into administration, product, development, costumer support and technical sectors, see Figure 16. The development sector is divided into product divisions and each product division has its own developers. The task areas (integration, analysis, testing, programming and database are common to all products.

Chief executive

WEB & Marketing

Development SupportProduct group Sales & Projects

Product CProduct B Task areas

Integration

Analysis

Testing

Programming

Developer

Developer

Developer

Product A

Database

Administration

Other parts of the company

Figure 16: Organisation at the company

Technical

The technical, development and administrative sectors serve as an underlying support for the product service sectors. The technical sector is responsible for installing and maintaining the network and its servers. The support department provides helpdesk on all delivered and installed products.

53

The development department has the responsibility to develop, maintain and test the products that the company provides. The management sector is divided into product responsible, financial and top management. The company provides a set of products. The business is divided into two main divisions located in two different cities. The division in focus for this thesis has mainly one product to develop and maintain. The support department located at this division provides support for all the company’s products.

4.2 Current work methods This section is the result from the interviews carried out at the company. Interviews and experiences from having spent time at the office, listening to discussions and attending meetings are considered in the description. The system referred in this section is the software for retail solutions.

4.2.1 General workflow

The company does not have any defined software development process, basically there is an ad-hoc approach used during the whole development. The system is in its fullness developed by the company, meaning that no COTS- application is integrated in the system. The system is from the beginning mainly developed by one person. The only documentation of the system is the user manual derived from the expectations of what the system should manage. The system is developed in an object oriented programming language (C++).

4.2.2 Requirement Elicitation

There are no formal requirements management. Often a project or a work task starts with an idea from either the developer or the product manager. The ideas are briefly discussed between them before the implementation starts. An idea could be one of the following things:

• A new function that the Product Manager (PM) comes up with. • A new function that the customer wants. • Defects that are discovered in the system and need to be fixed.

The idea is taken under consideration by PM and he evaluates the possibilities for the next release. Then PM informs the developers by having a discussion about the idea. The idea is freely translated into arbitrary requirements with no consideration of the nearby functionality that may be affected. The implemented arbitrary requirements generate the next release. A release could be one of the following things:

• Fixed defects. • New functions. • Structural changes in the code, for example, better performance.

54

In the future, as UML are introduced as a part of the software development process, the requirements are elicited from use cases. As mentioned before, to be able to maintain a functional and structured test environment there must be a functional and structured requirement management. The tests are strictly related to the requirements as the tests verify that the system fulfil its requirements.

4.2.3 Design

Most of the design is made during the implementation, but sometimes drawings on papers are made to see that it is possible to integrate all the different parts of the system. The aim is to use UML in the design work (see 2.3.3.4) in the future.

4.2.4 Implementation

The company has a code standard to follow. The standard itself gives much room for interpretations and is therefore not always used. According to the developers they feel that a code standard is useless, as mainly one person have developed the system. They feel that it is easier for them to adapt his “code standard” when and if they are making any changes. Code reuse is however something that is preferred. Especially as many parts of the system are similar. As a contradiction to code reuse, the developers claim that when developing an algorithm the aspects taken under consideration are simplicity and performance. Whether it is reusable or not makes a little difference according to the developers.

4.2.5 Testing It is also important to have a standard in how to deal with and define new requirements and features that should be implemented in the system. The tester must have some form of documentation on the system and its structure to be able to track the origin of the defects found. The developers make the tests that they consider necessary. The tests made are primarily unit tests and decision and statement coverage; the focus is on their own code and on making the new release functional as soon as possible. The tests are not documented, as the developers feel that they have a good overview over what to test. Approximately 50% of development time is spent on testing. Note that this figure is only an estimation made by the developers. Today it is the customers that make the most important tests, integration tests and acceptance tests as they use the systems.

55

There are however an extensive work in test improvement at the company. There is one person that has the responsibility to develop a functional test process at the company. The aim is to develop a test environment that is close to the environment that the final system operates in. A tool that shall provide support in documenting test cases and follow up on defects is under development. The overall opinion at the company is that testing is something that is necessary to reduce problems that arise.

4.2.6 Evolution

There is continuously work with the system. The reported defects are corrected continually as they are reported to the help desk at the company. New features are also continuously developed as they are wished for. There are no fixed releases of the system. Rather when new functionality is wished for from the costumers or defects have been corrected. A major problem is that releases have occurred to often.

4.3 Goals with the testing Because of the problems that arise when delivering the system the company feels that testing is something that is very important and has to be adopted in the developing cycle. Not only to reduce the amount of defects in their system but also because of their awareness of that they in the future need a working and structured test environment. The company has come up with a vision of what the company will attain with the introduction of a test process.

• The aim is to increase the quality of the company’s products so that they will be the leading company on the market.

• The testing shall be performed so that no bottlenecks occur in the

development process.

• Test shall be a natural part of the company’s daily work.

4.4 Organisational changes Because there is not any possibility to have an independent test department the current developers have to perform the testing with current resources. This means that the proposed test personnel will have widely spread knowledge in the testing domain.

56

4.5 Current process improvement strategies The company has put a great deal of time thinking of how to work with process improvement. The company has considered different ISO standards, but found out that they were not suitable for the organisation, as standards in general have no continuously process improvement plan. The company felt that if adapting to a standard and has gained certification of this standard they would have felt satisfied over what they have accomplished and no further improvement work would have been done. The things the company was looking for:

• A guide for how processes should be formulated. • A model that supplies the necessary knowledge. • A model that got an overview with detailed depth. • A model that makes cooperation easier with other similar organisations. • Last but not least, something that gives a good impression among the

companies customer. The company found out that CMM-SW is the proper model for the organisation. The main reasons for choosing CMM-SW:

• That it is a model where process improvements are made continuously step-by-step.

• The model addresses the specific needs of software development.

• The model also states the infrastructure that is needed to gain a

permanent work method, as well as it addresses single details in the workflow.

• The model allows gradual introduction of the process improvement

work.

• Is widely used by other organisations, which makes it easier to integrate.

• The model has well built education and consultation network.

• The model addresses many of the problems that the organisation has.

57

5 ANALYSIS In this section the analysis from the interviews and the prioritising of quality aspects is carried out.

5.1 Analysis of the interviews This conclusion of the interviews includes both real facts given from the interviewees and observations gathered by the authors. In this section the problems in the different phases in the software development process used at the company are derived and concluded. A summary of the interviews is in Appendix.

5.1.1 Support

The problems that can be derived from the interviews made with the support are as follows:

• Actions are already taken to make the logging of different errands easier and more efficient.

• The content of the problem-reporting document today only tells if there

is problem, not any classification of the problem.

• There is today no stated chain of control when it comes to making decisions of different problem-errands. Who takes the decision of the appropriate actions?

• The knowledge about testing differs between management and

personnel. How much education is needed for the support in the matter of testing?

The matter of who should do what. The support personnel for example feels that a working test process would give them more time for education among their customer. As the test management sees it the support personnel should be a part of the testing when not helping the customer.

58

5.1.2 Developers

The problems that can be derived from the interviews made with the support are as follows:

• The biggest problem according to developers today is that they spend a lot of time in correcting defects that would have been found with a proper testing structure.

• The implementation has taken place as soon as an idea has been

presented. Ideas have become vague undocumented requirements that then have been freely implemented. The developers have as today relied on an ad-hoc process.

• All design is done when the developers implement.

• No reviews are conducted.

The test conducted by the developers is what they have thought been necessary. The conducted tests have mainly been decision coverage and statement coverage. According to the developers they put approximately 50-60 % (note that this is only a estimate made by a developer) of their development time in testing.

5.1.3 Testers

Because of the lack of a structured test process it is difficult to draw any conclusions of the falsifications that can be derived from the interviews and discussions with the test personnel. Instead a list of what need to be done according to the test responsible person is presented:

• Introduce a functional documentation of the system • A structured test environment with all its element • Education of test personnel in the software testing area.

There are tests performed but not in a structured way (see 5.2.2). There is also a support and understanding about the testing from the management.

59

5.2 Analysis of the prioritising of quality aspects using AHP Below is the result from the prioritising of quality aspects at the company. The two tables represent the opinion of the personal at the company. It is chosen to evaluate the result both with the judgements that not fulfil the C.R. < 0,10 and those judgements that fulfils it. In the table with the excluded NOT OK C.R. is four of the collected priority result left out due to too high consistency ratio. Dependability is the highest prioritised attribute both when the NOT OK C.R. is excluded and included. Total priority (included NOT OK C.R.) Total priority (excluded NOT OK C.R.) 1 Dependability (18%) 2 Safety (17%) 3 Usability (14%) 4 Correctness, Performance (13%) 6 Maintainability (9%) 7 Robustness (8%)

1 Dependability (21%) 2 Safety, Performance, Usability (17%) 5 Correctness (13%) 6 Maintainability (9%) 7 Robustness (7%)

The conclusion from the AHP is that Dependability has the highest priority. This is probably the outcome from that the system has a history of installation problems and the fact that the system provides solutions to the retail. If there is low dependability this will affect the users of the system (the retail personnel) as well as third part consumers (the costumers in the stores). A system breakdown may lead to severe financial loss for both the company and the costumer of the system. What comes next is unclear. Maintainability and Robustness have low priority in both tables so the conclusion from this is that they are slightly less important than the others according to the personnel. This may be the result from that the system is fully developed and maintainability is something that companies in general not are interested in because it is a expensive and time consuming activity. Is it the wish for less maintainability and the visualisation of the perfect (correct) system that is reflected in the result? The wish for a perfect system also goes hand in hand with a robust system. The four attributes Safety, Usability, Performance and Correctness are prioritised somewhat equally. A ranking between these four is difficult to do. Table 2 shows that Safety, Performance and Usability are equally weighted and that the attribute Correctness was weighted below these three.

60

5.2.1 Analysis of the prioritising per group

The data from table 3 shows that there is big difference between Developers and Support personnel when looking at Safety. Developers rank Safety as number one while Support ranks it as seven (last). This can be the result from the developers underlying responsibility over the system. Table 3 below shows how the different groups of employees prioritised the seven attributes. Table 3: Prioritising per group: Man = Management, Dev = developers

Prioritising per group Support Rank Man Rank Test Rank Dev Rank Dependability 20% 2 23% 1 16% 2 18% 2 Safety 7% 7 14% 4 11% 6 27% 1 Maintainability 8% 6 9% 5 16% 2 10% 5 Performance 13% 4 23% 1 13% 5 10% 5 Correctness 25% 1 9% 5 16% 2 11% 4 Usability 17% 3 17% 3 22% 1 15% 3 Robustness 10% 5 5% 7 5% 7 10% 5 Another big difference is that the management and the other groups rank the performance aspect differently. Management rank the performance aspect as number one while the other groups rank it considerable lower. This can be the outcome from that the management does not have a daily contact with the system and thereby have higher trust in the system. Support rank Correctness as the highest, this is probably because of this is the attribute that the most contributes to their workload. If there is a high defect rate in the system, their workload will probably increase

61

6 IMPLEMENTATION STRATEGY FOR THE TEST PROCESS

In this section an implementation strategy for the test process is presented. It is presented in a step-by-step manner. Each step contains milestones that are necessary to attain in the struggle for the implementation of the test process.

6.1 Description of the strategy The implementation takes some of the activities in CMM level 2 into consideration but the focus is on the strategy itself. A complete adaptation to CMM is not attained with the strategy suggested. The aim in the suggested test strategy is that the testing will refine the system from the top. The testing strategy implies that the company should start their actual tests at the system test level. This because of that the company already provides a functional system with continually releases. If the company should start from the bottom (unit testing) they would not be able to apply the new strategy on their system. There is also no proper documentation of the system. Because of this the first step in the strategy is to document the system in its current appearance. Figure 17 states the different steps in the strategy where every square represents one step, SRS and SVVP is step one etc. Figure 17. The testing strategy

SRS and SVVP System testing

Module/unit testing

Integration testing

Update

62

6.2 Strategy for implementation of a test process In this section a strategy for implementing a test process for the company in focus is presented in four steps. The main focus in this thesis is on the parts concerning the test process, for us the requirement and test phases. The requirement area must be handled to be able to develop a functional test strategy, as the tests should be derived from the requirements. The strategy relies on two fundamental documents SRS and SVVP that are being continually updated during each step. In each step in the strategy new types of tests are continually derived and specified from additions in form of requirements. The result from the formal interviews and informal talking to the personnel has given a knowledge of what the company need to consider for a functional test process and when presenting the strategy. The documentation of the system is a very important issue. The strategy helps the company to perform documentation on their current system as well as develop test cases.

6.2.1 First step – SRS and SVVP

In the first step the goal is to perform system testing. Clearly the company can do this today in some way. The idea is that if a defect is found it should be possible to trace it down to its requirement and to get a template over what to test. To reach the goal there are some milestones to achieve. Milestone one – Software Requirement Specification The first step is to implement a functional requirement specification so that tests cases and instructions can be derived. To do this the company needs to use some kind of template for an SRS. An example of an SRS template is presented in Appendix G. The aim with the template is to help the documentation of the requirements. The template is a document that contains headings of the required parts of the SRS. With each heading there is a short description of what the specific section will contain and how it could be structured. As mentioned in the description of the company there is a user manual and out from this it should be possible to elicit most of the functional requirements. Other sources could be the support department. Regarding the non-functional and domain requirements the developers are a good source. The aim is to document requirements with help of UML use-cases and that each use case will represent a set of requirements. The strategy implies that the requirements also are documented in the natural language so called feature

63

style [29], in addition to UML use-cases. The benefit of documenting requirements in feature style is that they are easy to understand and the risk for misinterpretations is less than in UML use-cases. Another benefit is that it makes the implementation progress more visual as it is possible to compare the implemented parts with the requirements. It is also important that the customer understands the requirements and UML use-cases require a lot of “technical” knowledge. The strategy implies to manage the requirement documentation as follows:

• Use UML use-case at an early stage in the requirement elicitation.

• Transform the information gained from the UML use-cases into feature style documented requirements.

• Document the requirement as above and UML use-case diagram as

complement.

• Use template in Appendix G Milestone two – Software Validation and Verification Plan When having an SRS and the requirements described in feature style, it is possible to begin develop an SVVP for the project, see Appendix F. This plan will define what, when, whom and why to test. The test plan should consists of two main parts:

1. The first one is a test specification document SVVS that thorough describes which tests that should be performed and a distinct description on what each test case purpose is and the predicted result from it.

2. The second part of the test plan is a test instruction document SVVI

that in detail describes which adjustments that have to be done to the system e.g. which state the system will be in for each test case. The SVVI also includes step-by-step instructions on how two perform each test case. Each test case will be able to be identified in an appropriate way. It should also be clear which parts of the systems that the test case concerns e.g. which requirement(s) that is tested by the specific test case.

64

6.2.2 Second step - System test

When having a complete SVVP it is possible to conduct the system test. There is a need for a proper working environment for the system e.g. a simulated environment. A simulated environment is the set of hardware/software configurations that the system is designed for. Every test should be executed at least once and its outcome should be documented in a test protocol. The test protocol includes:

• Date of the test. • Test environment configuration (every environment configuration

generates a new test protocol). • Number of the test case. • The name of the tester. • Outcome, success or failure. If failure (this is a judgement done by

the manager), a proper description of the defect is written (chapter 8, further work).

Step two continues until no more defects are found. This means that the defects found can be corrected as the testing continues. The question is how much regression testing that has to be done after the correction of the defects, as corrections could lead to other defects.

6.2.3 Third step – Module/Unit testing

The third step in the strategy puts effort in finding different modules and units in the system and performing tests on them. When the modules are identified the requirements are listed and the documents are updated. Milestone one – Identifying the modules Identifying the modules means that the requirements for every different module should be located. A module can for example be a class or a set of classes (units). When identifying the modules and their requirements the strategy implies that UML-class diagrams are used. The result from identifying the different modules is an updated SRS and SVVP. Each module represents a set of requirements and is different from other modules. Milestone two – Perform the module test It is now possible to conduct white-box tests on each module. Which method that is preferred does not really matter for the strategy itself. In addition to the test protocol described in step two there should be a description of the module being tested, with its functionality.

65

6.2.4 Fourth step – Integration testing

The fourth step takes the testing to a higher level. Integration testing means that different modules are integrated. Integration testing puts the focus in the interaction and the dependability of the different modules in the system. The strategy implies to separate the integration test cases from the other test cases so that they clearly could be differed in the SVVP. In other words, the integration test should have its own chapter in the SVVP. The integration tests are often crucial for the system because of many thoughts (different developers) is put together. Therefore this kind of testing requires more tests. Milestone one – Identify the integration tests To identify the integration tests different combinations of functions in the different modules must be derived. Is there any interactions between module x and module y? If there is this specific interaction it will result in an integration requirement and test case that should be described in both the SRS and SVVP. The UML-sequence diagrams should clearly state the different interaction between modules. The strategy implies that UML sequence diagrams are in use when finding integration test cases. Milestone two – Perform the integration testing Integration testing is done by executing the test cases that now are described in the SVVP. The sequence diagrams are a good source when identifying the outcome of the tests. In addition to the test protocol described in step two there should be a description of the combined modules being tested and which method being used, for example top-down, section 2.9.2. By the forth step the strategy ends and the company should have a complete documentation over the system which provides a basis for the further work in the area. As and if the strategy is implemented some refinements can be done during the different steps.

66

7 CONCLUSIONS The conclusion directly related to the problem description is that it is a complex matter to come up with an outline of a test strategy. There are many factors that affect the implementation of a new process in an organisation. One may have many ideas, but with restrictions they are not just possible to implement. One of the largest restrictions as shown in this thesis is that the company in focus has limited resources to implement a new process. The restriction lies in who should do what and when? Other obstacles are knowledge and time. Companies in general want fast results and sometimes they find it hard to understand that some things take time. The result presented as a strategy has taken all of the obstacles and restrictions under consideration. The result is presented as a four-step strategy dealing with the main focus on testing and documentation of the system. The theory in the thesis has worked as a basis as well as interviews, observations and gained experiences from having spent time at the office. The gained values from introducing a test process are many, but one of the most important things is that the company in the long perspective can earn money on less time for developers to correct fault detected late, probably after release. The outcome from the thesis is not only a strategy in how the company can implement their test process; it has also given knowledge of how a software organisation works in reality, which conflicts that can occur inside the organisation and how it is handled. The presented further work is closely related to the test process area and is also very important when reaching for a well functional software organisation. Among many things according to the prioritisation of quality aspects using AHP (section 5) the dependability in the system is one of the quality aspects that the testing should be focused on in the future. Things that may have been done different are for example the planning of what constraints of the thesis should be. Some parts have after completion been deleted or rewritten because of its irrelevance to the thesis problem description

67

8 FURTHER WORK This section provides description of the further work that may be conducted by the company in their test process improvement.

8.1 Further work for the company Many of the problems described in this thesis have already been taking into consideration by the company during the development of this document. As it is today the requirement and overall documentation of the system are the least developed areas. These two are the most important for a functional test process. Hopefully these also will be well functioning using the strategy suggested in this thesis. For further work there are some suggestions:

• Generally all requirements are to be implemented. The lack of time or resources may result in the need of prioritising the requirements. The prioritising shall be conducted in consultation between the product manager and the intended costumer.

• Another thing for the future is to conduct reviews of the different

documents that are being developed during the project.

• Quality assurance planning is an important part of extensive process improvement. A basis for quality related work is a functional test process as well as knowing the quality goals of a certain project or having defined quality goals for an organisation.

• A basis for quality related work in the current system is provided in the

AHP prioritisation of quality aspects.

• Another important test related aspect is the configuration management procedures. Configuration management is about how to handle defects, problem reports, updates of documents, standards etc.

The outcome for as well the strategy and the further work suggestion is of great interest for us.

68

9 REFERENCES [1] F v Latum, R.v Solingen, M Oivo, B Hoisl, D Rombach, G Ruhe, “Adopting GQM-based Measurements in an Industrial Environment”, IEEE Software, 1998 [2] Basili V.R, Caldiera.G, Rombach H.D: “The goal question approach “, Institute for Advanced Computer Studies Department of Computer Science University of Maryland, FB Informatik Universität Kaiserslautern Germany [3] Youngman M.B: “Designing and analysing Questionnaires” Bell.J, m.fl: Conducting small-scale investigations Educational management. London: Harper and Row, 1984 [4] Bell.J: “Doing your research project”, Open University Press, Buckingham, England, 1999 [5] Natt och Dag J. “ Do you know how to ask questions?”, Department of Communication Systems, Lund, Sweden, 2001 [6] Wärnerryd.B “Att Fråga”, Statistiska centralbyrån, Stockholm, Sweden, 1990 [7] Sommerville.I ”Software Engineering”, 6th Edition, USA, Addison-Wesley, 2001 [8]URL:http://www.awl.com/cseng/titles/0-201 89542/techniques/evolution.htm (2002-04-28) [9] K Beck, ”extreme Programming explained – Embrace change” USA, Addison Wesley, 2000 [10] URL: http://www.extremeprogramming.org/rules/userstories.html (2002-05-23) [11] Bender R, “SEI/CMM Proposed Software Evaluation and Test KPA”, Starbase Corporation [12] Rothman J: “Release Criteria: Is This Software Done? How to know if your software is ready to release” STQE magazine, volume 4, issue 2 March/April 2002

69

[14] El Elman. K, Madhavji N H: Elements of software process assessment &improvement” IEEE computer society, USA,1999 [15] Bergman. B, Klefsjö. B: “Kvalitet från behov till användning“ Studentlitteratur, Lund, Sverige 2001 [16] Barett Jr, J.P: ”ISO 9000 what is it? And how do I prepare for it?” Managing in a global environment pp. 140-143. Engineering Management Conference USA, 1992 [17] Jarvis A, Crandall V, “Inroads to software quality”, Prentice Hall PTR, 1997 [18] Davies E, Whyman M:“ISO 9000:2000 new ISO, new responsibilities for top management“, Engineering management journal, pp 244-248, October 2000 [19] Boehm, B. “A Spiral Model of Software Development and Enhancement”, IEEE Computer, 1988 [20] Paulk M, Curtis B, Chrissis M B, Weber C V, “Capability Maturity Model, Version 1.1” IEEE Software, February 1993 [21] Kit E, “Software testing in the real world”, ACM Press Books, Addison-Wesley, 1995. [22] Regnell B, Wohlin C, “Projekthandledning Programvaruutveckling för stora system, kompendium 1” version 1.6 - 990823, Institutionen för Telekommunikationssystem, Lunds Tekniska Högskola. [23] Kaner C et. al. ”Testing Computer Software, Second Edition”, John Wiley & Sons, Inc, 1999 [24] McNamara C, MBA, Phd, “General guidelines for Conducting Interviews” (1999) Free Management Library located at http://www.mapnp.org/library [25] Rational Software Corporation White Paper “Rational Unified Process – Best Practices for Software Development Teams”, Rational Software 1998 [26] Pooley.S: ”Using UML, Software engineering with objects and components”, USA, Addison Wesley, 2000

70

[27] Fenton N.E and Pflegger S.E “Measuring internal Product attributes: structure” (Software metrics, A rigorous and practical approach) pp.280-302, Thomson Computer Press, 1996 [28] B. McFeeley, “IDEAL: A User´s Guide to Software Process Improvement”, pp15 – 24, technical Report CMU/SEI-96-HB-001, Software Engineering Institute, 1996. [29] W S Humphrey, “A Discipline For Software Engineering” Addison-Wesley, USA 1995

71

APPENDIX A The Questions Requirements:

• How do you know what to implement? • Who decides what you should implement? • How is the information about what should be implemented handled to

you? • Do you think requirement process / handling is important, Why?

Design:

• How do you design those parts that you shall develop? • Do you use any specific method? • Do you design before or during the implementation? • Do you document your design, how do you relate the design to the

implementation? • Are you willing to follow some kind of model or method in the design?

Implementation:

• The bug tracing in the Do you document the implementation, comments in the code etc??

• Do you think documentation may facilitate code? • Do you know what a coding standard is?

o Do you use a coding standard o Why is it important according to you?

Testing:

• How do you perform test today? • Do you know why you test? • Are you familiar with the concept of testing? • How do you know what to test? • How much time do you spend on testing? • How do you verify that your program works properly? • How do you test when you correcting bugs? • Are there any difference between testing of new developed code and

testing to certify that a bug has been corrected? • What is a test process according to you? • Who does what? • What is your role in that process? • Is testing well?

72

• Is there any value for you to put more time in testing? • What are the benefits? • What are the benefits for you? • What do you think of the current working process, are there any

need for improvement? • What problems could according to you be avoided with a more

structured testing approach?

73

APPENDIX B

Summary of an interview with a developer at the company According to the interviewee there is no structured requirements elicitation. Often a project or a work task starts with an idea from either the developer or the product manager. The requirements are briefly discussed between the two of them before the developer starts with his implementation. There is with other words no documentation of the requirements. The company has no formal or structured design methodology today, but are in an early stage of implementing UML as a requirement and design tool within the organisation. Today the design is made during the implementation by the developer, which suits them fine as they can develop their own visionary ideas. Despite of the concept to be a visionary the developers are looking forward to be apart of something that is proved to be booth well developed and well presented. When implementing the developers got a predefined coding standard that they should follow. The coding standards contains directions of presentation of class names, how to structure the code and where and a comment should be presented. The developers also do documentations about the system and then later on they make the user manual. The developer executes sub-systems tests and uses primarily decision coverage and statement tests. The time spent with testing is approximately 50-60% of the time spent according to the developer. The developers does not make any documentation on the tests, they feel that they have a good overview over what to be tested and not. When it comes to testing after fixing defects only tests that are concerned directly with the defect are tested. The developer thinks that testing is a big part of reaching a well functional and high quality system.

74

APPENDIX C

Summary of interviews with support personnel and support management The support department’s basic work task is to provide as much help as possible for the company’s customer. To do this, the support personnel are experts on their assigned system. The work tasks also include education of customer personnel and the education of new customers. When supporting the customers a log is written by the support personnel, the log states the problem, time and date, which action that the support personnel has taken and the follow-up taken by the support personnel. If the support personnel come to the understanding of that it is a defect in the system and not an error made by the user, the report is sent to the development department that takes the necessary actions. Later on when the defect is fixed a message is sent to the support and the support makes the necessary follow-ups. The knowledge about the testing concept vary between the support management and the support personnel. The management has a good understanding about the concept testing, while the personnel is more aware of that testing are something that has to be done. What they have in common is the understanding of why there must be a functional and structured test process and they are very eager to take part of the outcome. The surplus value would according to them be more time for education, less support and a good first impression of the company and their systems. The differences lay in the understanding of how much work it takes for every individual before it becomes daily practice and the surplus values becomes measurable.

75

APPENDIX D

Summary of an interview with test personnel at the company There is one person that has the main responsibility to introduce the testing process in the company’s development. This person was from the beginning a developer at the company, which now have the responsibility to administrate testing. He has no former experience in the software testing area and this has resulted in that a lot of information gathering about software testing has been made. His work is mainly to develop a structure of how the tests will be performed. He also thinks that it is important to gain understanding about testing among the other employees. This is done through information meetings with the management and other personnel where the testing strategy and its underlying theory is presented. Because there is no documented requirements or other documentation on the system a somewhat ad-hoc testing has been used until now. To gain more structure in the testing a support tool that helps develop test plans, keep track and documenting defects. There was a discussion about either to use an already existing tool or develop a own. The decision was after evaluating the different alternatives to develop a own tool. A own tool could be tailor made for the company and could be adjusted as they see fit in the future.

76

APPENDIX E

The AHP Excel-sheet The picture below shows the prioritising excel sheet that was distributed by mail to the personnel (without the one signs).

77

APPENDIX F

Template for software verification and validation specification Software Verification and Validation Specification Date: Author: Document number: Version: 1. Introduction In this section a brief description of this document and its purpose is stated. It shall also state the approaches that is taken to ensure that the requirements in the SRS are tested 2. Reference document The documents that are related to this document should be clearly stated here. The software requirement specification is one for example. 3. Reviews The amount of reviews, what to review, when to review and the review approach should be stated here. 4. Tests This section should answer questions concerning:

• Which types of tests (functional tests, system tests and regression test) are conducted during the project?

• How the different tests are conducted. • What actions that should be taken when defects are found

5. Additional tests This section should show if there are any additional tests that are supposed to be done or if there is any part that have been more thoroughly tested for example different quality aspects that are of importance to a customer. 6. Appendix This section should contain all the specifications:

• Function test – includes a numbered list of short descriptions on all functional tests.

• System test – includes a numbered list of short descriptions on all system tests.

• Regression test – describes how it is intended to perform regression tests.

78

APPENDIX G

Template for software requirements specification Software Requirement Specification Date: Author: Document number: Version: The following is an outline of an SRS that the company use when documenting the requirements. 1 Introduction 1.1 Purpose of document In this section a brief description of this document and its purpose is stated. A description of the contents and the structure of the document like numbering of the requirements may be present. 1.2 Overview In this section an overview of the intended system is provided. Its main functionality and purposes is stated. 1.3 Terminology Here all specific words are explained. 1.4 References If there is any other document that helps clarifying the requirements it shall be stated here. This may be a document that states the costumer wishes/expectations on the system and its functionality, protocols from meetings, other requirement documents like UML diagram. 2 General Description of the system 2.1 Main Functions A list of the main functions of the system 2.2 System Context Describes the systems’ relation and dependencies to other systems.

79

3 Functional requirements In this section the different functions that the system will provide shall be stated e.g. what the system should do. The requirements shall be numbered so that each requirement may be identified easily because there is often a need for cross-references in a document like this. A suggestion of how the numbering of the requirements may be structured: SRS02-3.1.1 where the first section (SRS02) states the document name and the second (3.1.1) relates to where in the document the requirement is stated. 3.1 is the first functional requirement, 3.2 the second and so forth. The requirements shall be stated in a natural language short description where the most important matter of the specific requirement is stated. 4 Non-functional requirements In this section the non-functional constraints for the system is stated. Interface requirements, design constraints. 5 Additional requirements In this section the additional requirements like quality attributes is described (dependability, performance and usability).

80