RELIABILITY ASSESSMENT USINGBAYESIAN NETWORKSCase study on quantative reliability estimation ofa software-based motor protection relay

Atte Helminen, Urho Pulkkinen

VTT Industrial Systems

In STUK this study was supervised by Marja-Leerna Järvinen

STUK-YTO-TR 198 / JUNE 2 0 0 3

STUK • SÄTEILYTURVAKESKUSSTRÅLSÄKERHETSCENTRALEN

RADIATION AND NUCLEAR SAFETY AUTHORITY

Osoite/Address • Laippatie 4, 00880 HelsinkiPostiosoite / Postal address • PL / P.O.Box 14, FIN-00881 Helsinki, FINLANDPuh./Tel. (09) 759 881, +358 9 759 881 • Fax (09) 759 88 500, +358 9 759 88 500 • www.stuk.fi

ISBN 951-712 -690 -5 (p r in t )ISBN 951-712 -691 -3 (pdf )ISSN 0785-9325

Dark Oy, Vantaa /Fin land 2003

The conc lus ions presented in the STUK repor t ser ies a re those of the authorsand do not necessar i l y represent the o f f ic ia l pos i t ion o f STUK

S T U K - Y TO - T R 1 9 8

3

Abstract

In this report a quantitative reliability assessment of motor protection relay SPAM 150 Chas been carried out. The assessment focuses to the methodological analysis of the quanti-tative reliability assessment using the software-based motor protection relay as a casestudy. The assessment method is based on Bayesian networks and tries to take the fulladvantage of the previous work done in a project called Programmable AutomationSystem Safety Integrity assessment (PASSI).

From the results and experiences achieved during the work it is justified to claim that theassessment method presented in the work enables a flexible use of qualitative and quanti-tative elements of reliability related evidence in a single reliability assessment. At thesame time the assessment method is a concurrent way of reasoning one’s beliefs andreferences about the reliability of the system.

Full advantage of the assessment method is taken when using the method as a way tocultivate the information related to the reliability of software-based systems. The methodcan also be used as a communicational instrument in a licensing process of software-based systems.

HELMINEN Atte, PULKKINEN Urho (VTT Industrial Systems). Reliability assessment usingBayesian networks.. Case study on quantative reliability estimation of a software-based motorprotection relay. STUK-YTO-TR 198. Helsinki 2003. 28 pp. + Appendices 3 pp.

Keywords: reliability assessment, software-based system, Bayesian network

4

S T U K - Y TO - T R 1 9 8

This report is a description of a pilot case study in which the ideas developed earlier in aproject called Programmable Automation System Safety Integrity assessment (PASSI) aretested first time in practice. The practical implementation and further developed of thereliability assessment method has been laborious but also very interesting and encourag-ing. In the case study the theory and the assessment method are efficiently demonstrated.However, this has only been the first step and similar case studies need to be analysed inthe future research. Through case studies like the one carried out in the work it is possi-ble to obtain a maximum learning from the relation between software developmentprocess and software reliability. At the same time the case studies guide toward the bestways of implementing reliability estimation methods to software development processes.

The assessment was carried out in VTT Industrial Automation and the research teamconsists of Professor Urho Pulkkinen and research scientist Atte Helminen. The assess-ment was carried out in co-operation with ABB Substation Automation. The authorswould like to thank ABB Substation Automation and especially Tapio Hakola, GrelsLinqvist, Tapio Niemi and Henrik Sundell for their time and expertise in this reliabilityassessment process.

Foreword

S T U K - Y TO - T R 1 9 8

5

Contents

ABSTRACT 3

FOREWORD 4

1 INTRODUCTION 7

2 DESCRIPTION OF THE ASSESSMENT TARGET AND ASSUMPTIONS 92.1 Target of the case study 92.2 Assumptions in the assessment 10

3 DESCRIPTION OF ASSESSMENT PROCESS 113.1 Overview 113.2 Prior estimation 113.3 Operational experience 123.4 Version management 13

4 BAYESIAN NETWORK MODEL 144.1 Overview 144.2 Prior for first software version 144.3 Implementing operational experience of software version 154.4 Implementing influence of software changes 15

5 DESCRIPTION OF EVIDENCE 165.1 Overview 165.2 Expert judgements 16

5.2.1 Steps 1 to 3 165.2.2 Step 4 165.2.3 Step 5 185.2.4 Step 6 18

5.3 Operational experience 185.4 Changes between software versions 19

6 RESULTS 216.1 Failure frequency 216.2 Predictive estimation 216.3 Analysis of results 23

7 DISCUSSION 25

8 CONCLUSIONS 27

REFERENCES 28

APPENDIX 1 QUESTIONS OF THE INTERVIEW 29

APPENDIX 2 THE BAYESIAN NETWORK IN THE ASSESSMENT 30

APPENDIX 3 WINBUGS-CODE OF THE BAYESIAN NETWORK 31

S T U K - Y TO - T R 1 9 8

7

The need for shifting from analog instrumenta-tion and control (I&C) systems to correspondingdigital I&C systems is becoming current also withthe I&C systems of nuclear power plants. To havea control over this transition and to have basis forthe licensing of digital I&C systems there needs tobe solid methods for assessing the reliability ofthe new digital I&C systems. However, to give themaximum support for the licensing process of dig-ital I&C systems the reliability assessment meth-od should not only produce exact values related toreliability, but with the assessment method thereasons for the produced values should be clari-fied.

With the analog systems there is the nicefeature that systems can be tested thoroughly. Fordigital systems this is usually not the case. Digitalsystems mainly fail because of their inherentdesign faults, which are triggered at appropriateconditions with certain inputs. Since the numberof possible inputs for even relatively simple sys-tems becomes unreasonably large, the system canonly be tested to a certain extent. As the system isnot tested with all the possible inputs under alldifferent conditions i.e. under all operational pro-files, there is always some uncertainty left to thereliability of the system. The most convenient wayto handle such uncertainty is through the use ofprobabilistic calculus.

For digital systems that can’t be tested exhaus-tively there rises a question if such systems canever be used in applications for which high relia-bility is required. One way to tackle this dilemmais to compensate the lack of reliability relatedtesting evidence with evidence from other sources.Such sources are for example the system develop-ment process, design features and operationalexperience. In the reliability assessment of digitalsystems this compensation of evidence is not al-ways a straightforward operation. Different evi-

dence sources involve many qualitative character-istics that are difficult to translate to unambigu-ous quantitative measures. These characteristicscontain a lot of information relevant to the relia-bility of the system and should not be overlookedand thrown away. Therefore, the methods used forthe reliability estimation should have enough flex-ibility to overcome the problem.

The approach used in the work tries to take thepoints mentioned above into consideration. Theapproach is based on Bayesian statistics and inparticular to its technical solution called Bayesiannetworks. Bayesian statistics enables the imple-mentation of qualitative and quantitative infor-mation flexibly together as well as updating theestimation while new information is obtainedabout the system and introduced to the estima-tion. The reliability assessment carried out in thereport is a practical continuum to the work donepreviously in PASSI-project. The theory and ideason which the assessment is mainly based on canbe found more explicitly explained in report byHelminen [1]. The purpose of the report is to testthe developed assessment method with a practicalcase study. The emphasis has been on the method-ological analysis of the quantitative reliabilityassessment. The details related to the technicalfeatures of the case study system and to theevidence used have only been reviewed on a levelnecessary to test the functionality of the assess-ment method.

In Chapter 2 a general description of the sys-tem under case study and some initial assump-tions of the assessment are given. Chapter 3contains a representation of the assessment proc-ess and overview on the evidence involved in theassessment. Closer review on the mathematicalformalism and the Bayesian network model usedin the work for the reliability estimation is givenin Chapter 4. Chapter 4 includes more detailed

1 Introduction

8

S T U K - Y TO - T R 1 9 8

information, which is not needed for all readersbut gives valuable information for advanced read-ers interested on the mathematical viewpoint ofthe topic. In Chapter 5 a detailed explanation ofthe assessment process and the evidence is given.

The numerical results of the assessment are pre-sented in Chapter 6. The assessment process andthe results are discussed in Chapter 7 and the lastchapter, Chapter 8, is left for short conclusionsabout the work.

S T U K - Y TO - T R 1 9 8

9

2.1 Target of the case studyThe system under assessment is SPAM 150 C –motor protection relay produced by ABB Substa-tion Automation. A picture of the relay is shown inFigure 1. The numerical motor protection relaySPAM 150 C is an integrated design current meas-uring multifunction relay for the complete protec-tion of alternating current motors. The main areaof application covers large and medium-sizedthree-phase motors in all types of conventionalcontactor or circuit-breaker controlled motordrives.

The relay continuously measures the threephase currents and the residual current of theprotected object. When a fault occurs, the relaystarts and operates, if the fault persists longenough to exceed the set or calculated operatetime of the relay. Depending on the relay settingand on the fault occured the relay either gives analarm or a launch signal for the protection opera-tion. The multi-function relay module comprisesseven units: a three-phase overcurrent unit, a

2 Description of the assessment targetand assumptions

thermal overload unit, start-up supervision unit,a phase unbalance unit, an incorrect phase se-quence unit, an undercurrent unit and a non-directional earth-fault unit. The descriptions ofthe protection functions for different units arefollowing:• The overcurrent unit holds a low-set stage Is

and a high-set stage I>>. The high-set stageprovides short-circuit protection. The low-setstage can be used for start-up supervision orfor time overcurrent protection.

• The thermal unit supervises the thermal stressof the protected object during various loadconditions. The unit provides thermal prioralarm and thermal tripping and it prevents theprotected object from being re-energised, if theprotected object is too hot to be successfully re-energised.

• The start-up supervision can be realised ac-cording to several principles. It can be based onmeasuring the start time, measuring the ther-mal stress at start-up or the use of an externalspeed switch.

• The phase unbalance unit protects, e.g. motors,from the stress caused by an unbalanced net-work. The unit operates with inverse timecharacteristic. The operate time of the unit atfull unbalance, i.e. at loss of one phase is 1second.

• The incorrect phase sequence protection has afactory-set operate time of 0.6 seconds.

• The undercurrent unit is used for the protec-tion of motors at sudden loss of load in certainapplications, such as conveyors and submersi-ble pumps. The unit features definite timecharacteristic.

• The earth-fault unit provides a sensitive earth-fault protection with definite time characteris-tic.Figure 1. Motor protection relay SPAM 150 C. [2]

10

S T U K - Y TO - T R 1 9 8

The relay incorporates a sophisticated self-super-vision system with auto-diagnosis, which increas-es the availability of the relay and the reliabilityof the system. The self-supervision system contin-uously monitors the hardware and the software ofthe relay. The system also supervises the opera-tion of the auxiliary supply module and the volt-age generated by the module. [2]

2.2 Assumptions in the assessmentThe reliability assessment involves evidence ofboth quantitative and qualitative nature. To makethe assessment possible some assumptions on theevidence was made. For example the operationalexperience data is based directly on the informa-tion obtained from the target system manufactur-er, and no additional evaluation was carried out.The following evidence was accepted in the as-sessment as such:• classification of software faults• number of software faults for different soft-

ware versions• classification of software changes• number of software changes made between

different software versions

• estimated amount of operational experiencefor different software versions.

In principle, the evidence listed above could havebeen interpreted as uncertain, and therefore de-scribed as random variables in the assessmentmodel. In the assessment more evidence was col-lected and certain assumptions was made usingexpert judgements and the given evidence. To car-ry out the assessment following assumptions wasmade:• Prior estimation on the reliability of the first

software version given by the system develop-ers

• Prior estimation on the reliability changes be-tween software versions given by the assess-ment executives

• Different software versions function in a singleand similar operational profile

In addition, to enable the combination of dispa-rate evidence together and on the other hand tomaintain the workload in a reasonable scale cer-tain simplifications to the given evidence weremade. These simplifications are described in de-tail in the following chapters.

S T U K - Y TO - T R 1 9 8

11

3.1 OverviewThe idea in the assessment process is to combineall available evidence to form a reliability estima-tion of the target system. Characteristics of differ-ent kind of evidence and the main sources of relia-bility related evidence in the case of software-based safety critical system is explained more ex-plicitly in report by Helminen [1]. In the assess-ment the evidence was mainly obtained from thesystem development process and from the opera-tional experience of the system. The evidence ofsystem testing involved in the assessment wasintegrated as a part of the system developmentprocess evidence.

In the assessment process the whole life cycleof the motor protection relay was taken intoconsideration. A diagram representing the assess-ment process is depicted in Figure 2. First, a priorestimate for the reliability of system’s first soft-ware version was built. The prior estimate ismainly based on the quality evidence of systemdevelopment process. This estimate was then up-dated using the data, i.e. operational experience,obtained for the first software version. Later onwhen the software was modified the effect of themodifications to the system reliability was esti-

3 Description of assessment process

mated and operational experience for the secondversion was introduced to the estimation. Thisprocedure was repeated, as many times as therewere new software versions produced in the lifecycle of the system.

3.2 Prior estimationThe analysis began from building a prior estima-tion for the first software version of the system.The prior estimation was built using the expertjudgements given by the designers and the onesresponsible for the implementation of the system.The expert judgements were collected in an inter-view, which was mainly based on guidebooks, rec-ommendations and standards for developing highquality software-based systems. An important as-pect in the interview was to compare the validityof the given expert judgements to the documenta-tion produced from the system during the life cy-cle of the system. The questions asked in the in-terview is presented in Appendix 1. In the inter-view the questions concerning the software devel-opment process were divided in five separategroups: project control and quality, requirementspecification phase, design phase, implementationphase and testing phase.

Figure 2. Diagram representing the software (SW) reliability assessment process.

SW changes

between versions

A and B

SW changes

between versions

F and G

Prior estimate

for SW

version A

Reliability

estimate for SW

version A

Reliability

estimate for SW

version B

Operational

experience for SW

version A

Operational

experience for SW

version B

Reliability

estimate for SW

version G

Operational

experience for SW

version G

etc…

12

S T U K - Y TO - T R 1 9 8

Category Software fault criticality:

1 Causes no trouble to customer

2 Causes inconvenience to small fraction of customers

3 Causes inconvenience to over 80% of customers or critical fault to small fraction of customers

4 Critical fault, which requires announcement to all customers and updates if requested by the customer

5 Critical fault, which requires an update to all customers

Before the interview the experts received ashort period of training on the assessment processand how to give probability estimations based onexpert judgement overall. The training sessionwas given to the experts at the same time, but theinterviews were made individually. The questionsin the interview formed a basis for a discussionwith a expert on matters related to differentsoftware development phases. After the query ofeach software development phase an expert wasasked to quantify how well he thought the produc-tion team managed in the execution of the phasein a numerical scale of one to ten. The numericalvalue given by the expert was called a score value.The expert was also asked to give a weight on theimportance of the software development phase tothe system reliability. In the training session thenumerical scale used in the interview and themeaning of different score values were discussedand this way each expert was able to build animpression of his own about the meaning of thescaled values in the aspect of system reliability.

The last step of the interview, after giving thescore values and the weights, the expert wasasked to give a system failure frequency distribu-tion for two or three different score values be-tween one and ten. The values are used for thecalibration of the scale of score values. The valuesdidn’t need to be among the score values givenearlier in the interview, but the system failurefrequency distributions should represent expert’simpression of the score values as well as possible.By giving failure frequency distributions for arbi-trary score values it was possible to extrapolate afailure frequency distribution for other score val-ues in the scale.

The score values of each expert were combinedusing an additive value function, which then pro-vided a single score value representing the ex-pert’s reliability estimation for the first softwareversion of the system. In the final reliabilityassessment the reliability estimates or failurefrequency distributions of all the experts werecombined together to form the prior failure esti-mate for the first software version of the targetsystem.

3.3 Operational experienceThe operational experience data included to theestimation was statistical operational experiencedata estimated for each software version of thetarget system. The operational experience consid-ered in this work is the amount of working yearsestimated for each software version and theamount of software faults detected for the soft-ware version during these working years. The soft-ware faults encountered for each software versionwere reported either by the developer or the cus-tomers using the system. The faults were ana-lysed and depending on the nature of the faultsthey were corrected in the next software version.

In the assessment the software faults wereclassified into five categories depending on theirseverity from the customer’s point of view. Thecategories were numbered from one to five, onebeing the least critical and five being the mostcritical fault to the customer. The five categoriesand their descriptions are given in Table I. Thelabel software fault may be somewhat a harsh andmisleading description for all the categories de-scribed in Table I. Some of the faults encounteredfor the device from the customer’s point of viewmight be caused for example from using thedevice for a purpose it wasn’t even originallydesigned for. Among other things this aspect istaken into consideration in a new categorisationcarried out later in Chapter 5.

Table I. Categories of software faults.

S T U K - Y TO - T R 1 9 8

13

Category Software change criticality:

1 Limit value change, having no strain to CPU

2 Change to a module, which can be well tested using automatic tests scripts

3 Change to a protection function

4 Major change having an influence to the interruption vectors

5 Major change having an influence to many places in the software

3.4 Version managementIn case there are several software versions pro-duced during the life cycle of a software-basedsystem it is important to be able to use the evi-dence from previous software versions in the reli-ability estimation of the last software version.This is especially important if the operational ex-perience available from the latest software ver-sion is small.

If the reliability assessment is carried out for asoftware-based system involving two or more soft-ware versions, the effects of the modificationsbetween the software versions and their criticalityto the system reliability need to be evaluated. Inthe assessment the evaluation was carried out byclassifying the changes in software to five differ-ent categories depending on the criticality thechanges have to the software. The categories werenumbered from one to five. Making a change fromcategory one is expected to have the least criticalsignificance while making a change from categoryfive is expected to have the most critical signifi-

Table II. Categories of software changes.

cance to the software and therefore to the reliabil-ity of the system. The five different categories andtheir descriptions are given in Table II.

Based on the number of faults and changesmade for a software version and depending on thecategories the faults and changes fall into a suita-ble increment or reduction to the failure frequen-cy distribution is introduced in the assessment.More of this procedure is explained in Chapter 5.

14

S T U K - Y TO - T R 1 9 8

4.1 OverviewThe approach used in the assessment is based onBayesian statistics and in particular to its techni-cal solution called Bayesian networks. More de-tailed description of the Bayesian statistics andthe Bayesian network theory can be found for ex-ample in books by Gelman et al. [3] and Box &Tiao [4]. An application of a Bayesian frameworkin combining expert judgements is explained indetail for example in a report by Pulkkinen &Holmberg [5]. More explicit description on the the-ory of using Bayesian networks to the reliabilityassessment of software-based systems can befound for example in reports by Helminen [1] andKorhonen et al. [6]. The Bayesian network model-ling and simulations in this report are carried outusing WinBUGS program, and so all representa-tions of Bayesian networks shown below are inWinBUGS format. For a closer review about Win-BUGS program see Spiegelhalter et al. [7].

The basic principle of the Bayesian networkmodel used in the work follows the general de-scription of the assessment process given in Fig-ure 2. More detailed description of the evidenceused in the assessment and how to quantify it forthe use of the model will be given later in the text.In the model the prior distribution for the reliabil-ity of the first software version is determinedusing expert judgements on the software develop-ment process. The prior distribution is then up-dated by using the operation experience of thefirst software version. The corrections and chang-es made to develop the next software version weretaken into account by assuming that the reliabili-ty parameters are subjected to a random distur-bance. This leads to a prior distribution of thereliability of the second software version, which isupdated by using the operational experience ofthe second software version. The procedure isrepeated for all software versions.

A Bayesian network utilising the evidence re-lated to the assessment process is depicted inAppendix 2. The Bayesian network presented in

4 Bayesian network model

Appendix 2 is the network used to obtain a quan-titative reliability estimation of the system understudy. The WinBUGS-code giving an unambiguousdescription of the network is given in Appendix 3.In this chapter parts of the Bayesian networkshown in Appendix 2 are explained in more detail.

4.2 Prior for first software versionImplementation of the prior estimation for thefirst software version given by the four experts istaken care by the upper left part of the network.The part of network under interest is picturedalso in Figure 3. Figure 3 describes the determi-nation of prior distribution for the reliability pa-rameter of the first software version. Mathemati-cally, the reliability parameter, ThetaA, is as-sumed to be a mixture of normal distributions,determined by the judgements of the four experts.In the network a discrete value for parameter Z issampled in each sampling round. Z determinesthe parameter values of Mu and InvVar. The val-ues possible for Mu and InvVar are listed in pa-rameters OpMu[] and OpInvVar[]. In OpMu[] andOpInvVar[] are entered the prior parameters ofthe first software version given by the four ex-perts. Mu and InvVar are the parameters of thenormal distribution ThetaA. Therefore, the distri-

Figure 3. Network to build prior reliability estimationfor the first software version.

Class[1:4]

OpMu[] Z OpInvVar[]

Mu InvVar

ThetaA

S T U K - Y TO - T R 1 9 8

15

bution of ThetaA, correspond to mixture:

( )4

1

( ) ii

p ThetaA p ThetaA=

=∑4

1

( ) ii

p ThetaA p ThetaA=

=∑ ,,

where pi(ThetaA) = N(OpMu[i],1/OpInvVar[i]) arethe normal distributions based on the opinions ofthe four experts.

4.3 Implementing operationalexperience of software versionImplementation of operational experience is car-ried out separately for all software versions andcan be seen as the repetitive lower part of thenetwork shown in Appendix 2. The repetitive partfor the first software version is shown in Figure 4.Since the operational experience obtained in theassessment for each software version is in a formof failures found in a certain time interval it isreasonable to assume that failures X1 are Poissondistributed with mean parameter t1LambdaA asfollowing:

X1 ~ Poisson(t1LambdaA),

where parameter t1LambdaA is a product of timeinterval t1 and the failure frequency of the soft-ware version named as parameter LambdaA.

Normal distributed failure parameter ThetaAis integrated to the network to enable more flexi-ble combination of different prior estimations andsoftware versions using a single Bayesian net-work. The idea is analogous with the combinationof different operational environments in a singlenetwork as explained in report by Helminen [1].Normal distribution is defined in the interval

(-∞,∞), and since the failure frequency parameterLambdaA can only receive positive values a log-transformation is needed and, therefore, Lamb-daA is the log-transformation of parameterThetaA defined as following:

LambdaA = exp(ThetaA).

The underlying model in the network depicted inFigure 4 is so called lognormal-Poisson model.Closer review on using lognormal-Poisson modelin failure frequency rate estimation is found forexample in report by Pulkkinen & Simola [8].

4.4 Implementing influence ofsoftware changesImplementation of the influence of softwarechanges between software versions is carried outwith a repetitive upper part of the network shownin Appendix 2. The repetitive part for the first twosoftware versions is shown in Figure 5. Idea incombining the evidence from successive softwareversions is that the failure probability of the lat-ter software version, ThetaB, is the same as thefailure probability of the preceding software ver-sion, ThetaA, added with a normal distributed ran-dom accretion or reduction term named Omega-AB. The relation of the parameters is defined asfollowing:

ThetaB = ThetaA + OmegaAB.

Variables MuAB and InvVarAB determine the pri-or parameters for the normal distributed randomvariable OmegaAB as following:

( )1~ ,OmegaAB N MuAB InvVarAB1~ ,OmegaAB N MuAB InvVarAB ..

The values for parameters MuAB and InvVarABused in the assessment are given for two differentapproaches in the last two columns of Table IX.

Figure 4. Network to implement operationalexperience to software version.

LambdaA

t1LambdaA

ThetaA

X1

t1

Figure 5. Network to implement influence of softwarechanges.

ThetaA ThetaB

OmegaAB

MuAB InvVarAB

16

S T U K - Y TO - T R 1 9 8

5.1 OverviewThere were four experts taking part in the assess-ment process. The amount of experts was approxi-mately half the size of the whole production teamof motor protection relay SPAM 150 C. All expertsinvolved in the assessment process were currentemployees of ABB Substation Automation and hada major role in the development process and exe-cution of the relay. Two of the experts were mainlyinvolved with the tasks related to the system re-quirement specification and project management.The other two were mainly involved with the sys-tem design, implementation and testing.

The software in the motor protection relay wasstrongly based on the software made for its prede-cessors and the relay was a part of so calledSPACOM-product family. The legacy from the pre-vious products had probably a strong influence tothe reliability estimations given by the experts,but this matter was not explicitly considered inthe work.

The first version of SPAM 150 C was producedduring years 1989 and 1990 in which the approxi-mately amount of work was two man-years. Thetotal amount of man-years used for producing thepredecessors from year 1980 to 1989 was abouttwelve. The software includes approximately40 000 lines of code about half of the lines beingfor commentary.

5.2 Expert judgementsA diagram representing the actual steps of theexpert judgement process for building up the priorestimation for the first software version is pre-sented in Figure 6. Steps 1, 2 and 4 were carriedout during the interview described in general inChapter 3. The fourth step turned out to be verytroublesome for the experts. Reason for the diffi-culty lay in the approach used in the interview forthe elaboration of the failure frequency distribu-

5 Description of evidence

tions of the arbitrary score values. The questionsetting was such that the experts were supposedto give the failure frequency distribution of soft-ware faults causing the system to fail in a protec-tion function on a scale of system failures perprotection demands given to the system. Takinginto consideration the continuous protection char-acteristic of the relay a better approach wouldhave been to scale the failure frequency distribu-tions on a scale of system failures per time thesystem is operating. This impracticality was cor-rected by renewing the fourth step of the expertjudgement process in a written form within aweek from the original interview.

5.2.1 Steps 1 to 3In the first two steps of expert judgement processthe five different phases of the software develop-ment process were discussed in an interview.Based on the previous experience and the conclu-sions made in the interview the experts gave scorevalues and weights for the different software de-velopment phases. The score values and weightsfor the different experts are shown in Table III.

Using the additive value function presented inthe step 3 of Figure 6 total score values werecalculated from the score values and weightsgiven by the experts. The total score values for theexperts are listed in the last column of Table III.

5.2.2 Step 4To convert the total score value of the expert tocorresponding failure frequency distribution eachexpert was asked to give failure frequency distri-butions for two or three different score values. Tobe more specific, an expert was asked to give fail-ure frequency fractiles of given score values. Fail-ure frequency distributions for the score valueswere then solved by fitting lognormal distribu-tions for the failure frequency fractiles given by

S T U K - Y TO - T R 1 9 8

17

STEP 1:

Judgements of scores for different phases of

the SW development process by experts

Score for

phase i by

expert 1,i=1,…5

si1

Score for

phase i by

expert 4,i=1,…5

si4

…

STEP 2:

Judgements of weights for different phases of

the SW development process by experts

Weight for

phase i by

expert 1,i=1,…5

wi1

Weight for

phase i by

expert 4,i=1,…5

wi4

…

STEP 6:

Combination of experts distributions of failure

probability

Combined probability distributions of failure

probability

4

1iiiSpfpf

STEP 3:

Determination of total scores by experts

Total score by

expert 1

5

115

11

1

1i

i

li

i s

w

wS

Total score by

expert 4

5

145

14

4

4i

i

li

i s

w

wS

…

STEP 4:

Determination of distribution of failure

probability for any total score value by experts

Lognormal

distribution

of failureprobability

for total

score S byexpert 1

f1(p|S)

…Lognormal

distribution

of failureprobability

for total

score S byexpert 4

f4(p|S)

STEP 5:

Determination of distribution of failure

probability for total score value given by theexperts

Lognormal

distribution

of failureprobability

for total

score S1 byexpert 1

f1(p|S1)

…Lognormal

distribution

of failureprobability

for total

score S4 byexpert 4

f4(p|S4)

Phase: Project control and quality

Requirement specification

Design Implementation Testing Total

weight score weight score weight score weight score weight score score

Expert 1 5 5,5 8 8 7 7,5 6 7,5 7 8,75 7,58

Expert 2 7 7 8 8 9 6,5 7 8,5 10 9 7,83

Expert 3 9 8,5 10 9 8 8,5 5 8 10 9 8,68

Expert 4 8 9 10 9,5 7 8,5 9 9,25 8 9,5 9,18

Figure 6. Diagram presenting the six steps of the expert judgement process.

Table III. Score and weight values given by the experts.

18

S T U K - Y TO - T R 1 9 8

the expert. The failure frequency fractiles and themean and variance parameters of the most suita-ble lognormal distributions are shown in Table IV.

5.2.3 Step 5Based on the total score value in the last columnof Table III and the failure frequency distributionsof different score values in Table IV a failure fre-quency distribution of the first system softwareversion was determined for each expert by usingthe method of least square. The mean and vari-ance parameters of the lognormal distributionsobtained for the experts are shown in Table V.

5.2.4 Step 6The last step was to combine the four failure fre-quency distributions given by the experts to onejoint prior distribution reflecting the reliability ofthe first software version of the target system.The combination was carried out in the Bayesiannetwork model and by the calculation algorithmof WinBUGS-program.

5.3 Operational experienceSo far there had been seven software versions ofSPAM 150 C. The operational experience of differ-ent software versions available in the assessmentwas the approximated amount of working yearsand the amount and types of software faults en-

countered with the different software versions.The approximated number of working years foreach software version is given in Table VI. Basedon the numbers of devices sold on each year theamount of working years for each software ver-sions was estimated. The number of working yearswas calculated so that on the delivery year thedevices were in operation only half the year i.e.the devices were delivered uniformly during the

Table IV. Fractiles for different score values given by the experts and µ and σ2 parameter values of the mostsuitable lognormal distributions.

Score: 0 5 10

Expert p X(p) µ σ² p X(p) µ σ² p X(p) µ σ²

Expert 1 0,1 5·10e–5 –7,6 3,2 0,1 1,5·10e–5 –8,8 3,2 0,1 1·10e–6 –11,5 3,2

0,5 5·10e–5 0,5 1,5·10e–4 0,5 1·10e–5

0,9 5·10e–3 0,9 1,5·10e–3 0,9 1·10e–4

Expert 2 0,05 1·10e–5 –9,1 2,8 0,05 5·10e–6 –9,8 2,8 0,05 1·10e–6 –11,4 2,8

0,33 5·10e–5 0,33 2,5·10e–5 0,33 5·10e–6

0,5 1·10e–4 0,5 5·10e–5 0,5 1·10e–5

0,67 3·10e–4 0,67 1,5·10e–4 0,67 3·10e–5

0,95 1·10e–3 0,95 5·10e–4 0,95 1·10e–4

Scores: 0 8 10

Experts: p X(p) µ σ² p X(p) µ σ² p X(p) µ σ²

Expert 3 0,05 1·10e–3 –3,0 0,2 0,05 5·10e–5 –8,5 0,8 0,05 2·10e–5 –9,2 1,0

0,5 5·10e–2 0,5 2·10e–4 0,5 1·10e–4

0,95 1·10e–1 0,95 1·10e–3 0,95 5·10e–4

Expert 4 0,1 2·10e–1 –0,8 0,4 0,1 2·10e–4 –6,9 1,2 0,1 1·10e–4 –7,7 1,4

0,25 3·10e–1 0,25 5·10e–4 0,25 2·10e–4

0,5 5·10e–1 0,5 1·10e–3 0,5 5·10e–4

0,75 7·10e–1 0,75 2·10e–3 0,75 1·10e–3

0,9 1·10e0 0,9 4·10e–3 0,9 2·10e–3

Table V. Lognormal parameters of the failurefrequency distributions of the experts.

Expert:

Prior:

µ σ²

Expert 1 –10,3 3,2

Expert 2 –10,8 2,8

Expert 3 –8,6 0,8

Expert 4 –7,4 1,3

Table VI. The approximated number of working yearsfor software versions.

SW version: Working years per SW version:

A 68

B 345

C 115

D 369

E 9805

F 2386

G 36747

S T U K - Y TO - T R 1 9 8

19

year. On the following years the devices sold be-fore are in operation for the full year until a newsoftware version was introduced. It was assumedthat after commissioning the device was in opera-tion full time around the clock.

In Table I the categories for software faultcriticality were given. The number of softwarefaults found in different fault criticality categoriesfor different software versions is presented inTable VII. However, to give a better overall de-scription about the nature of software fault and tosimplify the calculation process of the assessmentthe categories of the faults encountered with dif-ferent software versions were reduced from theoriginal five categories to two groups. The soft-ware faults were put in one of the two groupsdepending on which of the five original categoriesthe faults found fall into. Two groups were namedas software inconveniences and software faults.Faults in categories 1 and 2 in Table I wereconsidered as software inconveniences whereasfaults in category 3, 4 and 5 were considered assoftware faults in the assessment. The quantity ofsoftware inconveniences and faults encounteredfor the different software versions are listed inTable VIII.

5.4 Changes between software versionsWhen considering the independent changes madebetween successive software versions and their in-fluence on the system reliability two different pol-icies were taken in the assessment. In the firstapproach neutral attitude towards the changeswas taken, which means that the prior mean val-ue of change in failure probability between suc-cessive software versions was assumed zero. Inthe second approach a rather conservative atti-tude was taken and it was assumed that as a priorassumption a change in software has always a

Table VII. The number of software defects found for different software versions.

SW version: Category 1 Category 2 Category 3 Category 4 Category 5

A 3 0 1 0 1

B 5 1 0 0 0

C 0 0 1 1 0

D 1 0 0 0 0

E 4 1 3 0 0

F 1 1 0 1 0

G — — — — —

negative influence to the reliability of the system.What this means is that as a fault is removedfrom a software new faults are always introducedto the software. The magnitude of reduction in thereliability depends on the amount and criticalityof the changes made to the software.

As in the previous section also the criticality ofsoftware changes was divided to two groups de-pending on which category in Table II the changesfall into. The groups were named as minor andmajor change made to the software. In the assess-ment a change from category 1 or 2 was consid-ered as a minor change whereas a change fromcategory 3, 4 or 5 was considered as a majorchange. The number of minor and major changesfor each software version is listed in Table IX.Depending on the amount of minor or majorchanges made for a new software version a suita-ble prior estimation was built to reflect the changeof system reliability the changes were believed tohave. Parameters of the normal distributed priorsused in the assessment to reflect the changebetween software versions are given in the lasttwo columns of Table IX. The prior distributionsare based on the expert judgement of the assess-ment executives.

Table VIII. Software faults and software inconveniencesencountered for different software versions.

SW version:

Number of SW inconveniences

(categories 1–2):

Number of SW faults

(categories 3–5):

A 3 2

B 6 0

C 0 2

D 1 0

E 5 3

F 2 1

G — —

20

S T U K - Y TO - T R 1 9 8

Table IX. Amount of software changes made to each software version and prior estimations on the influenceof software changes. µ and σ2 are the parameters of normal distributions.

SW version:

Number of minor changes

(categories 1–2):

Number of major changes

(categories 3–5):

Prior (neutral approach):

Prior (conservative approach):

µ σ² µ σ²

A — — — — — —

B 11 0 0 1 0,1 1

C 14 10 0 1 1 1

D 3 1 0 1 0,25 1

E 1 1 0 1 0,25 1

F 16 6 0 1 0,5 1

G 10 5 0 1 0,5 1

S T U K - Y TO - T R 1 9 8

21

6.1 Failure frequencyThe parameters we are most interested in the net-work presented in Appendix 2 are the values ofparameters Lambda, especially LambdaG, whichreflects the failure frequency distribution of thelast software version. Also the values of parame-ters Omega are some of interest, since they reflectthe change in the software reliability between suc-cessive software versions. The more negative thedistribution of parameter Omega is the greaterenhancement has been achieved with a new soft-ware version. However, the results considering pa-rameters Omega haven’t been contemplated inmore detail in the report.

Calculations with the Bayesian network werecarried out for four different scenarios using allthe evidence available at the time being. Thescenarios differ by the approach used on theinfluence of software changes and by the dataregarding the software faults encountered for thesoftware versions. Different data sets were usedfor the conservative and neutral approach on theinfluence of software changes. In the first data setonly the number of software faults in Table VIIIencountered for different software versions wereimplemented to the network as fault data. In thesecond data set both the software faults and thesoftware inconveniences in Table VIII were takeninto consideration.

Using all evidence in the assessment the poste-rior failure frequency distributions of differentsoftware versions for the conservative approach isillustrated in Figure 7. In the figure are the poste-rior failure frequency distributions ranging from2,5 percentile, the lower bar, to 97,5 percentile,the upper bar, and median marked as a dotsomewhere in between. Corresponding graph forthe neutral approach is given in Figure 8.

An approximation of the expectation value forthe number of devices encountering a software

6 Results

fault during a year of operation can be obtainedfrom the median values of distributions shown inFigures 7 and 8. The reader should, however, bereminded that median is not the same thing asexpectation value even though in symmetricaldistributions they concur. The posterior distribu-tions in this case are not symmetrical and there-fore median value gives only an approximation forthe expectation value. For example from Figure 7it can be estimated that for software version A itis expected that two devises out of thousand willencounter a software fault during a year of opera-tion. For software version G the expected numberis approximately four devices out of one hundredthousand during a year of operation. Here all thedevices are assumed to function in a similaroperational profile.

6.2 Predictive estimationAn interesting addition to the failure frequencyestimation using the full evidence is how well themodel predicts the failure frequency of a new soft-ware version when only part of the evidence isused in the assessment. Predictive estimation canbe seen as a chronological estimation. The failurefrequency distribution for a new software versionis estimated only using the evidence available atthat moment back in the history. Estimates belowwere calculated for all software versions using thesame operational experience times as in the previ-ous section.

Calculations for the first software version wascarried out using only the first part of the networkshown in Appendix 2, i.e. using the parts of net-work shown in Figures 3 and 4. The predictioncalculation was performed by marking the datarepresenting the amount of software failures ofthe first software version as missing. The realisedcalculation was carried out likewise but havingthe data representing the amount of software

22

S T U K - Y TO - T R 1 9 8

failures of the first software version implementedto the network. For the second software versiononly the first repetitive part of the network wasattached to the previous network and same proce-dure was carried out for the second softwareversion. Similar operations were repeated untilfor the last software version the network was thesame as shown in Appendix 2.

The estimates were carried out for the conserv-ative and neutral approach using the number ofsoftware faiults as data. The results of the calcula-

tions for the predictive failure frequencies areillustrated in Figures 9 and 10.

In Figures 9 and 10 the meaning of the ap-proaches used for the influence of software chang-es can be seen in more detail. For neutral ap-proach the median of the predictive failure fre-quency distribution of the new software version isthe same as the median of the failure frequencydistribution of the previous software version. Forconservative approach the median is shifted up-ward by the amount determined in Table VIII.

Figure 8. Posterior 0,025–0,5–0,975 percentiles for failure frequency distributions of different softwareversions for the neutral approach.

0,000001

0,00001

0,0001

0,001

0,01

0,1

1

A B C D E F G

Software version

Failu

re f

req

uenc

y [/

a]

software faults all inconveniences

Figure 7. Posterior 0,025–0,5–0,975 percentiles for failure frequency distributions of different softwareversions for the conservative approach.

0,000001

0,00001

0,0001

0,001

0,01

0,1

1

A B C D E F GSoftware version

Failu

re f

req

uenc

y [/

a]

software faults all inconveniences

S T U K - Y TO - T R 1 9 8

23

Figure 9. Predictive and calculated posterior 0,025–0,5–0,975 percentiles for failure frequency distributions ofdifferent software versions for the conservative approach.

0,000001

0,00001

0,0001

0,001

0,01

0,1

1

A B C D E F GSoftware version

Failu

re f

req

uenc

y [/

a]

prediction calculated

The differences in the variances of distributionsare caused by the different amounts of data fordifferent software versions.

6.3 Analysis of resultsSignificant differences between the posterior fail-ure frequency distributions of the two approachesused for the influence of software changes can’t bedetected. With the conservative approach the fail-ure frequency distributions of different softwareversions seems to be more monotonous, i.e. the

Figure 10. Predictive and calculated posterior 0,025–0,5–0,975 percentiles for failure frequency distributionsof different software versions for the neutral approach.

0,000001

0,00001

0,0001

0,001

0,01

0,1

1

A B C D E F G

Software version

Failu

re f

req

uenc

y [/

a]

prediction calculated

estimates for the early software versions are bet-ter than in the neutral approach and vice versafor the later software versions. However, the dif-ference between the two approaches for the fail-ure frequency of the last and crucial software ver-sion is negligible as can be verified in Figures 7and 8. Explanation to the small difference in thelast software version can most probably be foundfrom the amount, and therefore, the dominant rolethe data has reached in the assessment.

From the predictive estimates it can be seen

24

S T U K - Y TO - T R 1 9 8

that the prior estimates given by the experts arerather optimistic. Reasons for this can be foundfrom the facts that the system under assessmentis based on its predecessors and that the assess-ment was made long after the development of thesystem and information about the operationalexperience of the system was available to theexperts. It is evident that these facts have stronginfluence to the estimates given by the experts,even though it was emphasised before the inter-view process to the experts that these mattersshould be excluded from the expert judgements.

Interesting detail consolidating this belief is thatthe posterior failure frequency distribution of thelast software version matches pretty nicely withthe prior estimation of the first software version.

Another observation that can be made from thepredictive estimations is that in spite of the opti-mistic prior for the first software version theconservative approach on the influence of soft-ware changes seems over pessimistic. There isclear statistical evidence that improvement be-tween most of the software versions have beenachieved.

S T U K - Y TO - T R 1 9 8

25

Before the actual interview process several con-versations about the system under assessmentwere carried out between the assessment develop-ers and the experts giving judgements about thesystem. Purpose of these conversations was tomodify the assessment process suitable for theparticular target system. One of the main goals inthe interviews of the expert judgement processwas to ground the questions asked and discussedduring the interview to current guidebooks, rec-ommendations and standards of developing highquality software-based systems. This wasn’t aneasy task since there weren’t many practical ex-amples on how to implement the valid recommen-dations and standards to concrete questions. Aguidebook turning out to be useful in forming theinterview questions was a guidebook published bythe Finnish Society of Automation [9]. Especiallythe example documents at the end of the guide-book provided good basis for the questions in theinterview. There is no doubt that the questionsconsidered in the interview and presented in Ap-pendix 1 would cover all the aspects of developinga high quality software-based system. Instead, thequestions should be seen as a structured way ofrecalling the history and building an overviewabout the quality of the software-based systemunder estimation.

The comparison of the current standards forsafety-critical software-based systems and thesystem under assessment was rather difficultmatter. One main reason for this was that suitablestandards for safety-critical software-based sys-tems have mainly been drawn up after the devel-opment of the target system and therefore explicitconsiderations of such standards in the develop-ment phase of the system was not possible. Anoth-er main reason was that the motor protectionrelay SPAM 150 C wasn’t explicitly considered tobe a safety-critical application but only an appli-

7 Discussion

cation for which high reliability was required.Therefore, an important phase of explicit clarifica-tion of safety requirements for the system wasn’tcarried out in a sense recommended for examplein safety standard IEC 61508. To ensure thecorrect operation and high reliability of the relaydeveloper’s own quality assurance methods werefollowed in the development process. Also thedocumentation involved with the developmentprocess was carried out in a way specified by thedeveloper and not by a specific safety standard.All of the factors mentioned above had an influ-enced to the fact that the comparison of theanswers and the documentation of the systemdevelopment process to the requirements of cur-rent standards of safety-critical software-basedsystems was a rather troublesome matter.

The decision analytic method used in the as-sessment to build the experts prior estimation forthe software-based system was applied in a ratherinformal way. Extra effort wasn’t spent to theconsideration of things like heuristics, biases anddependability of preferences in the questions andgiven answers. More analytical interview processwould surely have been beneficial, but on theother hand, as mentioned already in the begin-ning the main emphasis of the work was on themethodological analysis of the reliability assess-ment and therefore an overview kind of interviewprocess was sufficient for the assessment.

As mentioned in section 5.2 there were sometrouble in converting the score values given by theexperts to failure distributions of the system.Main reason for the inconvenience was the im-proper approach taken by the developers of theassessment method, which was corrected later inthe assessment process. On the other hand itcould be noted that more practice on giving thereliability estimation of a system in a probabilis-tic way would be appropriate. One way of achiev-

26

S T U K - Y TO - T R 1 9 8

ing this goal would be by taking an assessmentmethod as shown here to the regular activity inthe software development process. In subsection6.3 it was suspected that the expert weren’t actu-ally considering the reliability of the first soft-ware version, as they should have, but merely thelatest software versions of the system. If this isreally the case the results indicate that the ex-perts have rather good intuition about the relia-bility of the system and therefore it is justified touse expert judgement in the reliability estimationof a software-based system. However, if the opera-tional experience evidence is as strong as it isassumed in the case study the evidence from theexpert judgements used for the building of theprior distribution has only a small influence to thefinal reliability estimation.

The most significant shortage in the assess-ment presented in the report is that the opera-tional experience data used hasn’t gone through aspecific analysis. The reliability of software-basedsystem is a property of the operational environ-ment as well as the system itself. Although theremay be faults in the software, these faults cancause a loss of safety function only when certaininputs occurring with very low probability areintroduced to the system. In other words, the

reliability of software-based system depends onthe operational profile, which as the probabilitydistribution of input sequences varies from oneenvironment to another. To make the assessmentmore complete the operational experience datashould be analysed in more specifics to be able tojustify the results for a certain operational envi-ronment.

Other area of improvement in the networkmodel presented is in the influence of softwarechanges. The results shown in the previous chap-ter indicate that there was clear statistical evi-dence on improvement between most of the soft-ware versions and therefore the prior approachestaken, especially in the conservative case, was toopessimistic. Through finding a balance betweenthe prior approaches and the actions made inchanging the software it is possible to find thebest practices for the software version manage-ment. The process for finding the best practices ofsoftware version management is a long learningprocess. The method described in the report canprovide a good statistical approach for supportingthe learning process of software version manage-ment and therefore supporting the development ofhigh quality software.

S T U K - Y TO - T R 1 9 8

27

The approach used in the report for the reliabilityestimation of software-based system seems ex-tremely promising. Despite the slight inconven-iences confronted in the assessment process withconverting the score values given by the expertsto corresponding failure probability distributionsthe experience on the assessment method is veryencouraging. The experience and results obtainedfrom the assessment process support our prior be-lief on the fact that one of the most suitable waysto estimate the reliability of software-based sys-tem involving diverse evidence from various kindsof sources is based on the use of Bayesian statis-tics. Especially for large systems where the sys-tem and the software contained by the systemextends to the limits where the system can’t bemodelled or tested presumably completely it isreasonable to rely on such assessment methods aspresented in this report. The example case studyon a quantitative reliability estimation of a soft-ware-based system described in the report utilisesmainly the evidence from the system developmentprocess and the operational experience of the sys-tem. Due to the limitation of the evidence sourcesthe assessment presented gives a reliability esti-mation of the target system from a certain angleonly. To obtain a full comprehension on the relia-bility of a software-based system a variety of dif-

8 Conclusions

ferent analyses like the one shown in the reportshould be practised.

The assessment procedure presented enable aflexible use of qualitative and quantitative ele-ments of reliability related evidence. At the sametime the assessment method is a concurrent wayof reasoning one’s beliefs and references about thereliability of the system. It is therefore justified toclaim that a reliability estimation method shownhere can provide a strong support for the licensingprocess of digital I&C systems. Most of all thereliability estimation method establishes a solidground for the communication between differentparticipants of a licensing process.

The assessment method shown in the report iseverything but complete. Further research needsto be carried out particularly in developing theinterview process of the assessment so that thequestions cover all the relevant areas of develop-ing a high quality software-based system. Thequestions and interview process should be formu-lated so that if necessary the grounds for theanswers given by an expert can be traced back tothe documentation of the system. The operationalexperience and the operational environmentswhere the operational experience have been col-lected from is also a significant area of additionalresearch effort.

28

S T U K - Y TO - T R 1 9 8

References

[1] Helminen A. Reliability estimation of safety-critical software-based systems using Bayesiannetworks. STUK-YTO-TR 178. Radiation andNuclear Safety Authority, Helsinki 2001: 1–23.

[2] Motor protection relay SPAM 150 C -productdescription. ABB Substation Automation Oy.http://fisub.abb.fi/products/bghtml/spam150.htm.

[3] Gelman A, Carlin JB, Stern HS, Rubin DB.Bayesian data analysis. Chapman & Hall, Lon-don 1995: 1–526.

[4] Box G, Tiao G. Bayesian inference in statisticalanalysis. Addison-Wesley Publishing Company,Reading 1972: 1–588.

[5] Pulkkinen U, Holmberg J. A method for usingexpert judgement in PSA. STUK-YTO-TR 129.Radiation and Nuclear Safety Authority, Hel-sinki 1997: 1–32.

[6] Korhonen J, Pulkkinen U, Haapanen P. Statis-tical reliability assessment of software-basedsystems. STUK-YTO-TR 119. Radiation andNuclear Safety Authority, Helsinki 1997: 1–31.

[7] Spiegelhalter D, Thomas A, Best N, Gilks W.BUGS 0.5 Bayesian inference using Gibbs sam-pling manual (version ii). MRC BiostatisticUnit, Cambridge 1996: 1–59.

[8] Pulkkinen U. & Simola K., Bayesian AgeingModels and Indicators for Repairable Compo-nents, Helsinki University of Technology, Es-poo 1999: 1–23.

[9] Ajo R, Hakonen S, Harju H, Järvi J, Kaskes K,Lenardic E, Niukkanen E, Nurminen T, Rita-la P, Tolppanen M, Tommila T. Quality in auto-mation—Best practices. Finnish Society of Au-tomation, Helsinki 2001: 1–245. (in Finnish)

S T U K - Y TO - T R 1 9 8

29

APPENDIX 1 QUESTIONS OF THE INTERVIEW

Project control and quality• Did project have a quality plan?• How would you describe the quality plan?

What were the good/bad things in it?• What kind of quality control methods did the

project group have during the project?• How well/badly were the quality control meth-

ods carried out in the project?• Did the project have a project plan?• How would you describe the project plan? Were

all the main elements included in the projectplan and how well did the project plan consid-ered the needs of the system under develop-ment?

• How well/badly was the project plan carriedout?

Requirement specification phase• Was all the functionality of the system de-

scribed in a functional specification?• How would you describe the functional specifi-

cation? What made it good/bad?• Was the criticality of the system functions

evaluated anyhow? How comprehensive wasthe evaluation? How were the results of theevaluation used? Was the possibility of thesystem functioning in safety critical applica-tions considered?

• How well/badly would you describe the func-tional specification covered all the require-ments set for the system in the user specifica-tions and system requirements?

Design phase• Was the system described in design specifica-

tions?• How would you describe the design specifica-

tions? What made them good/bad? How com-prehensive were the design specifications?

• What kinds of tools were used during thedesign phase? How would you describe thetools used? What made them good/bad?

• How well/badly would you describe the designspecifications cover all the requirements setfor the system in the functional specification?

Implementation phase• How well/badly was the system implementa-

tion carried out? What made the implementa-tion good/bad?

• What kinds of tools were used in the systemimplementation? How would you describe thetools used? What made the tool good/bad?

• How well/badly did the implementation followthe design specifications?

Testing phase• Did the system development process have a

test plan?• How well/badly would you describe the test

plan covered the system implementation?What made the test plan good/bad?

• How well/badly was the test plan followed inthe development process?

• When was the test plan generated? Were thetest plan and the test cases in accordance withquality plan, functional specifications and de-sign specifications?

30

S T U K - Y TO - T R 1 9 8

APPENDIX 2 THE BAYESIAN NETWORK IN THE ASSESSMENT

InvVarFG

MuFG

InvVarEF

MuEF

InvVarDE

MuDE

InvVarCD

MuCD

InvVarBC

MuBC

InvVarAB

MuAB

OmegaFG

ThetaG

X7

t7LambdaG

LambdaG

t7

X6

OmegaEF

ThetaF

LambdaF

t6LambdaF

t6

OmegaDE

ThetaE

X5

t5LambdaE

LambdaE

t5

OmegaCD

ThetaD

X4

t4LambdaD

LambdaD

t4

OmegaBC

ThetaC

X3

t3LambdaC

LambdaC

t3

X2

t2LambdaB

LambdaB

t2

OmegaAB

ThetaB

X1

t1LambdaA

LambdaA

t1

ThetaA

InvVar

Mu

OpInvVar[]

ZOpMu[]

Class[1:4]

S T U K - Y TO - T R 1 9 8

31

APPENDIX 3 WINBUGS-CODE OF THE BAYESIAN NETWORK

model;

{

Z ~ dcat(Class[1:4])

Mu <- OpMu[Z]

InvVar <- OpInvVar[Z]

ThetaA ~ dnorm(Mu,InvVar)

log(LambdaA) <- ThetaA

t1LambdaA <- t1 * LambdaA

X1 ~ dpois(t1LambdaA)

ThetaB <- ThetaA + OmegaAB

OmegaAB ~ dnorm(MuAB,InvVarAB)

log(LambdaB) <- ThetaB

t2LambdaB <- t2 * LambdaB

X2 ~ dpois(t2LambdaB)

log(LambdaC) <- ThetaC

t3LambdaC <- t3 * LambdaC

X3 ~ dpois(t3LambdaC)

ThetaC <- ThetaB + OmegaBC

OmegaBC ~ dnorm(MuBC,InvVarBC)

log(LambdaD) <- ThetaD

t4LambdaD <- t4 * LambdaD

X4 ~ dpois(t4LambdaD)

ThetaD <- ThetaC + OmegaCD

OmegaCD ~ dnorm(MuCD,InvVarCD)

log(LambdaE) <- ThetaE

t5LambdaE <- t5 * LambdaE

X5 ~ dpois(t5LambdaE)

ThetaE <- ThetaD + OmegaDE

OmegaDE ~ dnorm(MuDE,InvVarDE)

t6LambdaF <- t6 * LambdaF

log(LambdaF) <- ThetaF

ThetaF <- ThetaE + OmegaEF

OmegaEF ~ dnorm(MuEF,InvVarEF)

X6 ~ dpois(t6LambdaF)

log(LambdaG) <- ThetaG

t7LambdaG <- t7 * LambdaG

X7 ~ dpois(t7LambdaG)

ThetaG <- ThetaF + OmegaFG

OmegaFG ~ dnorm(MuFG,InvVarFG)

}