AD-A1297556 FITTNG AN EXPONENTIAL SOFTWARE RELIABIIT MODEL TOFIELDFAIURE DATA U POLYECHNC NSTOF NEWORKBROOKLY NDEPT OFELECRICAL ENGI. R WSCHMIDT MAY 82

UNCLASSIFIED PD YEE/CC-82 002 N00014-75-C 0858 FG 12/ N

* 83

Jll I . 21112 1.02.2

11L21116

MICROCOPY RESOLUTION TEST CHARTNATIONAL BUREAU OF STANDARDS- 963-A

* PolytechncInstitute

FITTING AN EXPONENTIAL SOFTWARE

RELIABILITY MODEL TO

OttFIELD FAILURE DATA

BY

R. W. SCHMIDT

Prepared for I

Office of Naval Research

Contract N00014-75-C-0858

Report No. POLY EE/CS 82-002

Polytechnic Institute of New York

Department of Electrical Engineering

and Computer Science

333 Jay Street

Brooklyn, New York 11201

1 FILE COPY 0j

SF.CURITV CL &A.ltS'I A r- ', ' I A It P-1 , I I I

REPORT DOCUMENTATION PAGE Il :;rir,I. REPORT NUMBER : ,OVT ACLSSION 0) 3 IFCIPIENT- CATALOG NUMBER

4 TITLE (i ae Suthtfte, 5 TOPE OF REPORT & PERIOD COVERFO

TechnicalFitting An Exponential Software ReliabilityModel to Field Failure Data 6. PERFORMING OG. REPORT NUMBER

POLY EE-82-0027. AUTMOR(s) 8. CONTRACT OR GRANT NUMBER(s)

R. W. Schmidt NOOOI4-75-C-0858

9. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT. PROJECT, TASKAREA 8 WORK UNIT NUMBERS

Dept. of Elec. Engineering & Computer Science

Polytechnic Institute of New York333 Jay Street, Brooklyn, New York 11201

I. CONTROLLING OFFICE NAME AND ADDRESS 12. REPORT DATE

Office of Naval Research May, 1982Arlington, Virginia 22217 13. NUMBER OF PAGES

5114. MONITORING AGENCY NAME & ADORESS(if dilferent from C., .trolir, d Office) IS. SECURITY CLASS. (of this report)

UnclassifiedtSa DECLASSIFICATION 'DOWNGRADING

SCHEDULE

16. DISTRIBUTION STATEMENT (of this Report)

Reproduction in whole or in part is permittedof the United States Government. i

'k oo f''- Pubtc "ra"5 1 Ubutjona uGUM.I

17. DISTRIBUTION STATEMENT (of the abstract enterd :.n lock 20. ,ri gi-e nt from Report)

18. SUPPLEMENTARY NOTES

19. KEY WORDS (Continue on reverse side if necessary and Identify by block number)

software reliability, model, model construction, field failure data

20. ABSTRACT (Continue on reverse side It necessary and identify hy block nttmher)

--The quantitative prediction and measurement of software reliability isof vital importance in the development of high quality cost effective soft-ware. Many software reliability models have been postulated in the literaturehowever few have been applied to field data. A model based upon the assump-tion that the failure rate of the software is proportional to the number ofresidual software errors leads to a constant failure rate and an exponentialreliability function. The model contains two constants: the proportionalityconstant K and the initial (total) number of errors T. Ic'

DD I JN7 1473 EGITION OF INOV SS IS OBSOLETESECURITY CLIIUnclassifiedi SECURITY CLASSIFICATION Of THIS PAGE (*Won Deis Entered)

SECUNITY CLASSIFICATION )F THIS PAGE(W7.q O.1. En-e.pI

'rhe constants K and LT can be estimated during early design by Comparisonri

of the present project with historical data. During the intergration test

phase, a more accurate determination of the model parameters can be obtained

by using simulator test data as if it were operational failure data. The

simulator data is collected at two different points in the integration testphase and the two parameters can be determined from moment estimator formulas.The more powerful maximum likelihood method can also be employed to obtainpoint and interval estimates. It is also possible to use least squaresmethods to obtain parameter estimates which is the simplest method and provides

insight into the analysis of the data.

This report utilizes a set of software development and field data takenby John D.- Mus* as a vehicle to study the ease of calculation and thecorrespondence of the three methods of parameter estimation. The sensitivity

of the reliability predictions to parameter changes are studied and comparedwith field results.

The results show if data is carefully collected, software reliabilitymodels are practical and yield useful results. These can serve as onemeasure to help in choosing among competitive designs and as a guage of whento terminate the integration test phase.

ror

pistribiAt"I -

Ave 11ty ode~s-Ariand/or

Dist

SE URITY CLASSIFICAlIOt. OF ?. AG Erm,an Fnwt. d)

FITTING AN EXPONENTIAL SOFTWARE

RELIABILITY MODEL TO

FIELD FAILURE DATA

BY

R. W. SCHMIDT

Prepared for

Office of Naval Research

Contract N00014-75-C-0858

Report No. POLY EE/CS 82-002

Polytechnic Institute of New York

Department of Electrical Engineering

and Computer Science

333 Jay Street

Brooklyn, New York 11201

L!

i

ABSTPACT

The qutitatie 7rediction and reasu--%:ert cf softue re-

liability is of vital Impor ance in the development of high quality cost

effective sofet-are. Yany software reliability models .avo been poatulaUt

in the literature (Ref. 11). however few have been applied to field data,

A model ba-,ad upon the assumption that the faiur3 rate of the seotwa-re 15

proportional to the number of resid=l scftware e-or- leda tt % constan.t.

failure at and an exponential relkab!_ iy f'anction, (I.ef. 1). T.. made'

contains two constants, the proportlor.iity constant K and Tha in! tial

(o #,a 3.) n rnbcr of e = v - 9

The constants K and ET can be estmatd du-ring e;.rly dsi n- Icy

ecpar-sac of hs preaent project with h-istoricl data. Dr.ng the in-

tegration tast phase, a more accurate determination of the model ara.meters

can be obtained by using simulator test data as if It were operational

failure data. The simulator data is collected at two different points

in the Integration test phase and the two parameters can be dete-Ained

from moment estimator formulas (Ref. 9). The more powerful maximum

likelihood method can also be employed to obtain point and interval

estimates (Ref. 3). It is also possible to use least squares methods

to obAin lzametar estimates which is the simplest method and provides

insight into the analysis of the data (Ref. 12).

: I.

iii

This thesis utilizes a set of software development and field

data taken by John D. Musa (Ref. 10) as & vehicle to study the ease of

alculatimn and the correspondence of the three nethods of yapaaeter

estimation. The sensitivity of the ra.±ability predictions to razete.r

changes a=e studied and ccmed with field results,

This thesis Is based in pat on & Joint paper Written by

the author and Professor Maztin L. Shaman, 1-esented at the OCk-TIS

Conference in Floida, Janu az 1981. (14)The 'rsults show that if data is caxefully collectod, soft-

lare reliability models are pmatical and yield useful esults. These

can serve as one zmeas to help In choosing among ocpetitive desg.;ns

and as a vage of when to tezinate the Inteation tcst phase.

lj- ' • .. .. .. + : +:; -+, .... - , ,, i ... , . .. . . . -. , ,

iv

TABLE OF CONTENTS

CHAPTER TITLE PAGE

Abstract ii

Table of Contents iv

Table of Figures vi

List of Tables vii

I Introduction 1

2 Development of Error and Reliability Model 3

2.1 An Error Removal Model 3

2.2 Development of the Exponential Reliability Model 7

2.3 Meantime to Software Failure 10

2.4 Musa's Model 14

3 Estimation of Model Parameters 16

3.1 Introduction 16

3.2 Moment Estimates 16

3.3 Least Square Estimates 17

3.4 Maximum Likelihood Estimates 18

4 Musa's Data 20

4.1 Introduction 20

4.2 Description of Raw Data 20

4.3 System Characteristics 23

TABLE OF CONTENTS

CHAPTER TITLE PACE

5 Estimation of Model Constants 25

5.1 Method of Moments 25

5.2 Least Squares Linear Regression 26

5.3 Maximum Likelihood Estimates 28

6 Model Parameter Accuracy and Sensitivity 35

6.1 Introduction 35

6.2 Model Accuracy 35

6.3 Model Sensitivity 36

7 Conclusion 41

References 43

_

_

_ _ _ __.o_..i

vl

TABLE OF FICUBE

FIGURE TITLE EG

I NORMALJIZED EILIOR RATE va DEBUGING TDO FOR TPUESU3RVISORY SYSTE'S1-

2 CUMUlATIVE ERROR CURVE FCR SYSTEM A (FIG. 1.) 5

2B CUMUATIVE ERROR CURVE FOR SYSMI~ B (F:G. 1) 6

20 CtUI~ATIVE ERROR CURVE FOM SYST.74% C (FIG. 1) 6

3 VARIATION OF RELIABILITY i"INCTICN, R(t) VIVI DEBT. G C9TDG

4 CCMWJISON OF MT'I7 vs DEBUCGflC TDIE FOR nT-- ERROR 9MODEL

44ERROR CORRECTION RATE I4ODEIS 11

4B. GROwTH CF &E(-C) AssccIlum WITiHP@) ,OF FZ-. IA 11

40 COMIARISCII OF j'w.F vrs'CFOR VARYTillG ERROR COl' LIECTiclf'A ZI

5GROliTH CURVE OF SOFTWARE XTITF (13) 12

6 ?zTF vs TEST TNE (8) 13

7 SYSTEM 3 CUZMUATriE ElRORS vs FAILURE RWE 29

8 ESTDNTD vs TOTAL EHRORS ETr - SYT" 3 33

9 TOTAL ERRORS vs TESTIG TflXE SYSTEMI 3 38

10 CHANGE I MTTF BY VARYING K39

11 ZPFECT ON HTTF BY VARYNG .40

ZTV

VII

LIST OF TIA3LM

1 YAM=U l-qTERVAI3-SYSTM#, #3 TMT MQSE 21

2 SYSEM #3 FAILTJE PATE AM 24

3 XVTHO OF ?1OWILNT ESTIYATION OF K,~2?

. LMAST SQMRES REGR-MION4 ELMATION OF X.,T 1 30

5 HAXDWU LIMMZ~OOD ESTDAION Q' 11, 4 34

6 COMBaRIS0OJ OF MCDEL ffiWICn.IOIlS WMI FPLD E~nl 42

!1

1.0 Introduction

Software presently represents the highest cost Item in the

development of ccmputer systems. There is a aucity of quantitative

measures to Judge the quality of the final software and use as a meacurs

of progress during the test and debugging phase. The reliability and

meantime to failure ( =rP) of the software is a most useful metric for

both the above purposes.

An important class of software reliability modelz (see

Refs. 1, 8) make the assumption that the operational software failure

rate is proportional to the remaining number of errors. Thus the failure

rate is dependent on development time T, but not on crerating time t.

This leads to a constant hazard and exponential reliability model, with

two unknown pa-ameters K and E,

A major focus of this thesis is to investi~ate and provIde

insight into a number of issues related to the estimation of these two

paameters :

1. The accuracy one can obtain by using historical data to

deteraine K and E.r (Musa's);

2. A comparison of the accuracy obtained rsing three dif-

ferent methods of parameter estimation: the maximum

likelihood, momenta, least squares;

3. A comarison of prtdicted values of MI.,? and observed

MT values from field failure data;

14e Model par'meter sensitivity; and,

5. Model accuracy/parameters related to the practical ap-

plication by the software manager.

• 1

2

The conclusions reached in this study clearly indicate that

pzameter estimation during system development is a highly practical tool

which can be used to successfully predict software M o and the 'debugging'

time required to achieve that goal@

3

2.0 Development of Error and Reliability Models

2.1 An Error Removal Model

The reliability model used in this paper has been described

in detail in a number of references (1, 2, 3). In brief, the model as-

sues that the pogram enters the integration test phase with ET total

ez-s remaining in the software. As integration testing prcceeds, all

detected errors are promptly corrected, and. at any point In the develop-

zent cycle (aftertmonths of development time)*, a total of E. (T) er-

ra-e have been corrected, and the remaining number of errors, Er is

E.(T) = ET - Ec(T) ()

In a zore advanced model (4) It is assumed in addition that new erors

are generated du-ing develo~ment. Ona can often norm-.iie the above

equaticr. th_-cugh division by the numter of object code instructiona IT

to yield

ErJ-ET -_. Ec-.c (2a)

Ir IT IT

CrtrJ= Er-c- (7-) (2b)/r

Wheje Cr=&d Cc - ECIT IT

Basically the error modal used in this pepr assumes that the

total number of errors in the progr m is fixed ard that if we record tho

in some cases the actual number of test hours Is estimated and is uned

as the development time variable rather t.an the cruder calendar days.

cumulative number of errors corrected during debugging, then the difference

represents the remaining errors. We can define error removal rtes as t

which can be normalized to yield

where

P(T-) - error-. removed/total number of instructions/test hours (5)

ep (T) - p (x)dx - cumulative errors/total number of instructions. (6)

In Reference 5 error data are reported for seven large super-

visory programs an.d applicat+ions programs * Zn Fig. 1 the nc.--alized err-r

zat*P(T) calculated from this data is plotted as a function of T, the

number of months of debugging after release for three of the seven systems.

Although several curve shapes might be fitted to this eata, one chr:cter-

istic is common for all curves. The normalized error rate decreases over

the entire curve or at least over the latter two-thi=dz or half of the

curves whereas Initial behavior cf P(r) differs from e ample to example.

A curve of the cumulative error data for the supervico-y system A of Fig. 1

is shown in Fig. 2. Similar curves of C (T) drawn for the other examples of

Fig. I all build up initially with a constant or iner-sing slope and then

exhibit a decreasing curvature appearing to bocome asymptotic. The smo-

othing nature of integration makes all the E (T) cumoe look more alike than

theP(T) curves do. (Figs. 2A & B). 3oth are needed for a detailed study.

0~0

0 3 45 6 7 0 4 5 6 7 8 0 1 2 34 5 6r Months

Supervisory A Supervisory 8 Supervisory CFIGURE I. INORMALIZED ERROR RATE vs DEBUGGING T.ME

FOR THREE SUPERVISORY PROGRAMS

0

5. r r

uDebugging time - T - Months43 FIGURE Z. Cumulative Error Curve for

Superisory System7 A Given in Figure 1.

. .. ... .. "7 .... ..G M 2.....

CUMUIATIVE EaROR CUR VE FOR SUPERVISORY SYSTEK B- .. :-.Z 7 ~ 8~ .-.-_. . . ... :-:... . .. ... .. ...- .... ... . - --

.-. ..................... . .... .

3 -- - -" -. .. ...- , .. . - __..--- . .... __-. -- __ __. _,---- -_

- 1 - 4 -- ---

• + , t+ . -- 0 .. :.- . . .. . --, - --- --- I- " . -;- - .- - -

S_DEBIUCTGIN TLME -T-MChTHS ___ .. _ ___ ,___ ::,_____-__....

_, ___ -- _ - ' ' -- FICURE, 2B - ..-

.._.CM____UITIVE ERROR CURVE FOR SUPERVISCRY' S" ST C-- -- I - I---.- -8 ,.. . '- - - . . ... " ..... . ... . --- " _'---- -

- - -.- - . ..S - I.-- ' • .._--' ---- -.+- .. . - .. ---- ... ............. ~

- -- ..-- -- ..I"7- ._ .------ -.- -_----- -.-- ..------. .......

--- _-_. .-... --- ' 7 --- - -I ..... . - --- -. - --::--.... -- - .. '-.. . .

-- - - . . .. _ . .+

. . . _ _ _ - . .. ,+--.. . .... ... . ..

--- .. . .- _ _

:-'-" -- ";- ' +--; -.......... .. ......... :........""......... 1...............

~ D~U~fl4CTIME-t MCNTES

.. .. ' " ........... ' '" " I-, ... .... ;----- :- -.. .... ...... Z:_'q _2 _' . . - ...... . . . . . . ._...... .... . ... .... ...... -

:-= -'-'" +-'"; '" - F " " ................ +:--r+ . - .. ' .. :............

* -__ -.-. _,.. I. .... o .,.. ..... ... . .6? . . . : .

If we assume that the total number -. of errors in the program

(ET) is constant and that the program contains ZT inztructions, then the

asymptote which the C(T) curves approa~ch Is ET/IT. (Figs. 2, 2A, 23).

The error model developed above will be used in the following

section to formulate a reliability model.

2.2 Development of the Exponential Reliability Model

Me assume that all operational software errors occur due to

the occasional traversing of a portion of the program in which a hidden

software error Is lurking. Ve begin by writing an expression for the

probability that an error is encountered in the time interval dt after t

successful hours of. operation. We make the assuption that this prob-

ability is proportional to +he number of errors remaining in the program.

(See Ref. 6 for data substantiating this assumption).

From a study of basic probability and reliability theory we

know that the probability of failure in time interval t to t +&t, given

that no failures have occurred up till time et, is proportional to the

failure rate (hazard function) z(t). Thus, we obtain

P(ttf4t +4t I t,>t) - z(t)4t - KEr(Z)4t (7)

where tf - time to failure, (occurrence of a software error)

P(tt.t +AtItf>t)- probability of failuro in interval a

given no previous failure.

K - an arbitrary constant

j

8

From reliability theory (7) we cart show that the probability

of no system failures in the Interal 0 to t is the rollability function

which is related to the hazard, s(t), bys

RHt (8)

If we substitute our ex-reszion for z(t) from Eq. ? into

Eq. 8 and assume K, and E (t), are independent of -or.ratinm time, we

obtain

R(t= xft) e - (9) '

Basically the above equation states that. the probability of

successful operation without scft .are errors Is an ex:nential function

of ope-ating time. When the system is fixst turned oa, t - 0 and H(O) - 1.

As operating time increases the reliability monotonicv.lly decreases as

shown in Fig. 3. We depict the reliability function -o= three values ef

debugging time, TC(j< r2 . From this curve we may m:. .:e various pre-

dictions about the system reliability. For exaple, looking along tho

vertical line t - 1/( we may states

1. If we spend T hours of debugging then R(Ii) - 0.35

2. If we spend T, hour.; of debuging then - -50-

3. f e ped hours of debugging then -~ 0.753. I we pend 2

0.8 ljmsdeugn

0.6- r,0 mlost debugging

ti o least debugginig

NORMALIZED OPERATING THMEtFIG.3 Variation of Reliability Function R(t )with

Debugging Time r

12-

RxMTTF~~. 8 I-arI

0 1/ 4 ar 1/ 3/4 1Normfalized 1Tim.3

o.Coristont Error Debug ;:*'.teFIGURE 4 Comnparison of 1.MT7TF vsDebug Time for The Error Muel.

10

2.3 Meantime to Software Failure

A simpler way to summarize the results of the reliability

model is to compute the mean time to (software) failure,MTrF, for

the system.

do

Xf''MJ R~t) dt, (10)

For prposes of illustration we letp(t) be modeled by

a constant rate of error correction Po. Solution of E&s. 2 and 10

then yields

' -" -a(-c)

where P2 X Sr/IT and Ct.% POIT/ZS:

This is depicted in Fig. 4 where3xMIM- is plotted vsC-T,

and we note that the greatest Improvement in ?ITTF occurs during the

last A of the debugging stage.

Reference 5 describes other error correction rate models,

as illustrated in Fig. 4A. In order to copare the relative effects on

system MTTF by assuming a varyingp(T), it is essential that we Integ-

rate the latter over the same range. This is readily verified by

noting that the total axea under each curve is identical. Intogration

results are plotted in Fig. 4B for each model to yield the cu.nulative

number of expected errors. The effect on systo. M.1F is depicted In

Fru. We.

J1 A

TI agum // .

do. .00/

... K

12

DAYS 30.530- DAYS

MTSSEI

20-

10-

6/28 1970 8/2 . 9/3 9/10

ra DEBUGGING TIME IN DAYS

F£gaz. 5. Crawtht Cuarve of Solttwza ~dA'tbijity MwlanzTime Bowman ScL,,,.. Errors. (Trom L

Wyam~~Aoil. 1.2).

13

20

18

6-

0 2 4 6 8 10 12 14 16 IS 20 2Z2.?4 ZG

TEST TIME - HOURS

Figure 6 MTBF vs. Test Time for Pr-oject 1.(Replotted from the data of Fig. 3,• .Musa, ufS

0122

The rapid rise in XTIT toward the end of t-he integration

I*ase is of considerable Importance in software -roject management.

Field data by others (see Figs. 5 & 6) confim=s this ba ic shape. If

a manager Is pressed to release a system to the field .t an early date,

he my accept the current reliability and deliver the software. if we

believe Figs. thru 6, however, then a few more weeks could yield a

big improvement in MT=F. The model predicts such behaviour, but if

one only had test data fora-, j then it would be difficult to pre-

diet the sharp rise near 1T-1. The model is thus of great use in

managing a proJect and setting its release date.

2.4 Mus's Model

Muss. has de.veloped a acdel similar to that .iven in See. 2.2.

Howeer, instead cf basing his model on developnent tira as the resource

measure during integration, he utilized actual CTU tim. Musa also used

regular test data rather than simuiatcr test data. His rodel is given

by (8),

ace) - (12)

T,, Toe - (C' / MOT (13 ).

15

Where

- hours of program opez-ation - t

T - mean time to failure i. operating hours -T

To - XTT? at the start of test (t 0)

C - ratio of equivalent operating time/test time

No - number of failures which must occur

to uncover all errors A.

- the CPJ time in hours during testing

Since we will be using Musa's data and some of his results

in See. 5 of this thesis, we must carefully account for.the different

_deinitions of time between Musa' s model and the exponcntimzl model

during the analysis of his data.

16

3.0 Estimation of Model Pam~ters

3.1 Introduction

The exponential reliability model given in Eq. 9 contains the

parameters K and Er(t) which must be estimated. In many cases one wishes

to use such a model to roughly predict MTTF during a proposal phase or

early design of the project. In such a case the only available technique

for determining values of the rarsAeters Is to use historical data,

Presently Rome Air Development Center is developing a handbook and database

on Software Reliability for just such a purpose.

3.2 Moment Estimates

In Ref. 9, a method is disc-ssed for measurement of the two

needed parameters based on simulation testing of the cftwa-re. A program

simulating the field environment is generally available for all real live

computer programs , It is necessazy that this program be run for a total

of H1 hours following T 1 months of development. It is assumed also that

the tes+ng will produce r1 failures, S iilarly aftcr T2 months of testing,

the sim lator is run and H2 hours and r2 failures aro obtained, The MT=F

for the data is given by the ratio H2/r 2 and is equatad to the MM ex-

pression obtained by substituting Eq. 9 into Eq., 10. The two equations

(for T, and T ) allow us to solve for the constants Kand

21 (14i)

H2. I-... K.ET - Ec-, .T2 (.()

17

Simultaneous solution of Eqs. 14 and 15 yields the desired values of

and i.

ETr (16)

K =A (17)Er -C ('C)

Note that in deriving the above results we have assu-:,i that the failure

zate is constant and have used the ccmmon notation .nd well known

results for constant failure rates s

s(t)m mi 4T

3.3 Least Square Estimates

Another method of estimating model p.ramc.' !rs is to rewrite

equation 11 in the fore

A

X,- ,[ ,- Ec(, (18)

for ) - failure rate

to yield E (i) - 1... (19)

This equation ropresents a straight line 1>::ose parameters

can be determined from the slope and intercept of a lcz st squares fit of

the data, where

18

I* -1 (20)

- intercept (21)

Naturally, the largerand the more accurate the data set, the

a e precisely can the model ;arameters be predicted,

3.41. Maximum Likelihood Estimates

Another method which can also be used to estimate the values

of X and Et is known as the Y.ax.mu= Likelihood Eztimatior. technique (ML).

The likelihood function, L is the Joint probability of occurrence for +he

observed set cf test values. If during simulaticn testing we observe r1

failure times (t, , ... t ) and n -r, successful runs testing (T,, T2 ,

... , ' . )then the likelihood function for a single test after E (r.)

errors have been removed is given by

L(KE) - f(t)f(t 2 ))....f(tr )R(t 1)R(T 2 )...(T l

where

f(t 1 ) - th density function K ( .) -

3(T 1 ) -the reliability function eK'r.)

To aximize the likelihood function we take partal de- itives with

respect to C and ET and set them equal to zero. To solve for the two

ope ations we need a second equation obtained frc anot'her likelihocd

equation based on a second set of tezt data at.timer 2. , Applying I'E

19

theory to two tests with r and r. fallures over H, and H2 total hours,

we obtain (Appendix A, Ref. 3)

r i 4.~ c"I r%2 ~ (22)

As is aften the case with MLE, the above equations requiire

numerical soluticn: however, most stmtistic!ans believe them to yield

superior results to moment estizates. An iterative computer solution

of Eqs. 22 and 23 is easily Implemented, yvt a ra~phical solution

using a slnpl calculator cuffices in mest cases. Tne fi--%t step is

to obtain starting vmlues for and K using some other method, such

as Method of Moments. Values cf k above and below tho sta-ting

value are substituted into Eqs. 22, 23 and the curves of K vs ET

plotted on the same axes* Their intersection detezzinos the value

Ofi and i.

20

4,0 Mus's Data

.1 TIntroduction

The software reliability data used La this paper was compiled

by John D. Musa (10) over a period of time on a vaziety of large software

systems, rnging from tens to hundreds of thousands o± object code in-

structions. * is objective was "to present In detail a substan.tial body

of data that has been gathered in the application cA" the execution time

theory of software reliability"; the end product is i-eally suited for

the pu.-pose of this .pmer, as it "esents a wealth of precise software

failure data obtained under carefully contro2-led circrmstances°

S4.2 Description of Raw Data

Software failure interval data was Dresei: td in tie following

format I

Failure number Failure teai Day of fi,

Failure interval was measured in seconds, and :-preeits either ruaning

clock time, (operating time on the computer), or, in one case, actual

CPi time. 'Day of Failure' is the workini day countc. from the s3.-a+

of project on which the failure cccurr=ed. (See Table I).

21

TABLE I

FUME INTERVALS - SYSTEM 3 TEST PHASE

Failure Failure Day of Failure Failure Day ofNumber Interval Failure NumberI interpal Failure

1 115 1 21 .85 262 0 1 22 390 26

178 3 24 1337 305 3 35 583 27

6 136 3 26 834 387 1077 3 27 3400 408 15 3 28 6 409 15 3 29 4561 42

10 92 3 30 3186 44

11 50 3 31 10571 4712 71 3 32 563 4713 606 6 33 2770 4714 1189 8 34 652 4815 40 8 35 5593 1 5016 788 18 36 11696 5417 I 222 18 37 6724 5418 j 72 18 38 2546 5519 615 18 39 10175" 5620 389 26

* Notes If the last interval is followed by an asteisk, there was

no failure at the end of the period a.d the time represents the

interval between the last failure and the end of tho period. (10).

A UN

22

For the purpose of this paper, failures occurTng on the same

working day were summed together to form one statistical data point, and

were tabulated under the following fomatt (See Table 2)

where*

S - sequential serial number assigned to each statistical

data point

WD - working day on which failure(s) occurred

Z - number of errors occurring that working day

Ec - cumulative errors to date

T - total operating time (failure interval tir.e) for that

working day

To - cumulative failure interval time to date

z - failure rate (E/T) for that working day

Presentation of Musa's data in this fcrmat had & twc-fold

purpose I

a) reduce the sheer bulk of the raw data without affecting

its statistical significance by group n.g the occurrence

of software failures by working day; a d,

b) tabulate the data in a format more suitable for subsequent

calculations.

23

4.3 System Characteristics

Systems j-E& studied by Musa are real-time ccr~nd and control

software packages consisting of 21,700 to 33,500 object instructions and

a failure sample size of 38 to 136.

System 5 is a realtime commercial application, consisting of

2,h45,000 object instructions and a failure sample size of 831- Musa

notes that for system 5, design changes involving 21% of the source

code were introduced after approximately 30% of total testing time had

elapsed.

24

'1ABIZ 2

SSTE't #3 FAZrUR=' AS DATA

SEz T Tc

1 2 2 .01739 115 1153 10 12 .00523 1911 2026

3 6 1 13 .ooi65 606 26324 8 2 15 .00163 1229 38615 18 4 i9 .00236 1697 5558

6 26 3 22 .00302 994 65527 27 1 23 .00054 1863 34158 30 1 24 .00075 1337 97529 36 2 25 °00022 45C8 14,260

10 38 1 26 .00119 834 15,094

11 40 2 28 .00059 Y"o6 16,50012 42 1 29 .0022 4561. 23,06113 44 1 30 .0C031 3186 26,24714 47 3 33 .00022 13,904 40,15115 48 1 34 .00153 652 40,803

16 50 1 35 .00018 5593 46,39617 5. 2 37 .0001i 18,420 64,si618 55 1 38 .00039 2546 67,362

!2C

5.0 Z.ztmation of Model Constants

In the following section, we ,ill be estimating the model

constant3 K and E Cf several systez, using system #3 as a working

example to demonstrate the calculations used for Least Squares, Method

of Moments, and Paximum Likelihood Estimates. These constants will

then be used to estimate system MTTF using equation 11.

As indicated in Sec. 4.2, Musa's ras. data was condensed to

form one statistical data point per working day during which one or

Ore e-Eors were uncovered.

In the following section, this condensed data was reduced

even further to allow 2, 3, 4, 6, 8... points to represent an entire

system as required. For example, I" it were desired to represent

system 3 (see Table 2) consisting of 18 entries, by two points (Fig. 7),

the first nine entries were averaged statistically to yield the first

point and the last 9 entries to yield the seuond point.

5.1 Method of Mcments

The data reported by Musa for system #3 haz been processed

and grouped by working day as described in Sec. 4.2 (tee Table 2). Using

data points fcr the 9th and 18th group of failures, (Ec 9, Tc 9),

(Ec 18, To 18), we obtain the average failure rate fcr the interval 0-91

! w 5 failures " 0.C0175 failures/sec. - 6.3 failures/hr.T9 14,260 sec.

and similarly the average failure rate for the Interal 10-18 is s

Eci - EC9L8 2X2-EeT"2S -Tc9 67,362 104,260

-2.448 X 10-4 failUres/sic. 0.881 fallurea/hr,

26

2/ 1 " .1396

-I2/\ Ea9) - Ecl9 (o 9 )-1- 1) (0.1396-1)

-40.110

and X -I . 04017i =1.i6o x io

T- E9 4D.110 -25

See Table 3 for a sugary of all systems.

5*2 Least Squares Linear Regression

In this method, we plot the failure rate z vs cumulative

e-rors Ec, Theoretically the best fit y-intercept yiolds 1 and the

negativo reciprocal ef the slore equals K.

The primary equations used in linear regression are t

slo a -N (4)

intercept y' ..ZY22 ME (25)

for the straight line given by :

y- ax + y' (26)

Applying the above to equation 19, and reforring oack to

TaUble 2,

27

METHOD CF MC.CTS ESMTL'ATIO! OF ET' K

System

1 4t 8

EC 136 1T .51.6 153.6 i26.4

K * 5.514 4.576 13.56

2 2 24 8

Ec - -.4 !. 61.6 52.6 38.7

K 3.909 5.688 7.731

3 #Pts 2 1 6

Ec w 38 ET 40.1 38.2 28.3

i £i.6o 21.40 19.61

4 #PtS 2 1 6

Ec 53 ET. 54.9 54.9 22.2

i * 19.88 13.40 26.20

6 #Pts 2 r.....

Ea- 74, L 111.7 117.7

. e 28.345 27.639

• N,B, K v:Lluas x tO-5

28

'29 .25 Zc18 -38

Tag 14,260 T018 = 67,362

We plot

17,53 ~ ~ x 0-4T -ZC - 25

X2 Z2 - 25 2.4 5 x 10-4. Y5 -x1Q8 38

to yield

- Jo.1 i - 1.16o3 lo "-

The above calculations are depoActed In Fig. 7 for system 3

and summa zed for all systems in Table 4.

5.3 farximum Like2hood Estimates (MLE)

The primary equations used (see Sec. 3.4) ar-

" 1r + r2 (.cm 1+2(22). H, (EEc )H2 (ET'E )

r

I L% + '2 j(23)xi~z EF'-o

. 29

FIGUPS 7

CUMIUATMV FMRS vs FAIMURE RATE35-

SYSTM 3

30

1 7-

rm~27 7) .

20

7 7-.

0

100 04 50

FA I WE RATE z x 1-

30

TABLE 4

LFASE SQiURr F=RM3I0N 5T3hA.T0MAIS OF.,

System

#Pts 2 4. 8 47*

1 Z 151.6 140.1 103.3 74.6

5.515 5.245 17.6C9 543.48

#'Ot3 2 8 12 2.

2 T 61.6 51.7 45.5 44.9 42.9

_ 3.909 6.253 8.333 9.395 10.279

J#Pt 8s 2 .6 39 28.4

T 40.1 1 34.9 32831 6 2*11.603 25.490 23.;2Z 27.942 56.593

#Pts 2 4 10 19*

4 54.9 46.8 49.8 42.4

K 19.885 14.C29 20.400 4o.6o1

5 E 2390 998.9 805.2 742.9 497.6

8.089 30.675 44.9;2 51.475 94.518

K.B. AU n values x 10"5

* Number of points used represents total sy:tem data wi.thout

interval grouping.

31

A numerical solution of the above equations can be readily

obtained, since we have two equations I'n two unlmown, K and Eo A

simpler approach assuming we only have a few estimates to make. is to

effect a graphical solutioni

Step Is obtain a starting value for ET by any method of

your choice: Method of Moments, Least Squares

Regression, Non-deter .niistic (eo... guess), *.l

Step 2: substitute a value E obtained in Step I into

equations 22 and 23. Try other values of F

above and below this value and repeat the cal-

culations, plotting them on the same axis of

vs 4 for the two equations; and,

Step 3t the intersection of these two curves yields the

required ;a_-ameters.

The method is illustrated using system 3 as an example:

Step Is ET - 40.1 using the stating value obtained by

the Method of Moments (MOM) Table 3.

Step 2s For ZT - 39, K 1 1504 x 10"- (from 22)

K2 - 2.195 x 10 (from 23)

For T - 41, K 1 8.355 x 1.o5 (from 22)

K - 7.008 x 10-5 (from 23)

Step 31 The above results are plotted on Fig,. 8 to yield

F?40*1t - ± .6 x !o"_

Once an approximate value is obtained fret. the curves' in-

.esect.on, itratlve methodf can be used to obtain the required degree

of &CCUz&cy.

See Table Sfo a sumaz7 c'f thie IM .st-tratOs f 0?r a11 'f 0

B-.MS

33

FIGURE 8

ETL'ALT IC vs TWtAL MRS Zr

Sy3TEM 3

..... . ..

- .. 4- 1'11.66 x 10...

. ... .. .. ..

5. .. : .7.

.340 41

TMTL ZIMORSE~ iT .

34

?IAXDMM LIIXLflHOOD ESTIVATS OF TK

System J ±

151.59 5.141 x 10-5

2 61.61 3.909 x 10-5

3 40.11 1.16o x 10

4 34.80 2.249 x 10

I

35

6.0 Model &rameter Accuracy and Sensitivity

6a1 Introduction

Several additional factors must be considered when using

this exponential model - or any other model for t.hat matter - in pre-

dieting software system peormance.

Two aspects ccnsidered in this thesis ares

1. Model Accu-- c. i The more data one has on a particular

system, the more confident one can be that the parti-

cular model predictions w4ill app:rimate re-ality;

2. Model Sensitlvit, When soft-are field failu-e data

is available to comp.re with model Lrediction estimates,

it seems initially that slight. cl-nZes in model pazz-

meters are not proportionally reflected in model ;re-

dictions for system XTTF;

6.2 Model Accuracy

Intuitively, one feels that the more data one has about a

pLticular system, the more accurate will be our predictions about

system performance* To a certain extent this is trae. This implies,

however, that model Drediction accuracy only apprcachas reAlity as we

near the final debugging stage, whereas the *software xmrnger' is .quite

interested in estimatLng software reliability during carly testing

stages to determine future trends. As noted in Section 2.3, this is

strongly implied by Fig. 4 which indicates that the maximum return for

36

onets efforts is obtained only during the latter 25% of the debugging

phase.

Hindsight will usually enable us to icok back at a parti-

cular software development project and to announce that 'x' time rather

than 'y' time should have been spent on development and testing.

The onset of. the release point can be determined in several

ways, two of which are tot

a) closely monitor the slop of our hTTF curve plotted

versus time spent on debugging and to watch for a

sharp increase as it nears a vertical asymptote

(Fig. 4, 4C) or,

b) ainice the system MTTF is directly related to the

number of software errors remaining (E - 1C), lcok

for the point in time when model predictions of

applied to the system in question approach a hori-

sontal asymptote. This aspect iS depicted in Fig. 9

by the shape of expected value andz2iccnfidence bands

around E as testing nears completion.

6.3 Model Sensitivity

The un-normalized equation used to calculate system XTq iz

W- (27)K(T1 I

37*!

and the derivative with respect to K is

W-- -(28)

This indicates that variations in K result in a quasi-linear effect on

MI'T. For small variations in K this variation can be linearized as

sham In Fig. 10 where the percentage change in X is plotted vs resulting

percentage change in ?M.

Slight vaxlation= of 4, however, result in drmatic changes

in MTT o, especially as the former approaches the value of Ea (C). Cal-

culating Z MTTF yields,

- -• (29)

Thus, as Zc(Z)->Z, the ser.nitivity becones quite large. it

can be seen frcm Figs 11 that a 10% change in kT results in #0%+ variation

of system ?XTrF'. 'nis factor alone em-hasizes the requirement of high-

qulity failure data if accurate model predictions are to ensue.

. .. . .: : . _r -,

*~~7 .: -7- -!-tT-~

- TOTAL RS '~aTMTIZ;C TflD 7T

rYS 3-L

.......... _ _

- - T

4-' _ETL ____ (h__)__

A 1 L _

1:m

~L...2. .:CANGE IN BpY VARYf4CZ X*.

. .. ... . ... . -. l .. .. ... . .. . .. .. . .....

S... ... . .0 . .. - ...

- ... . . .4 - .L

- . . .H I ~ ' ," /E'D G Z. . .. .. . .

,. ! .....~ ~- : L..... ' -

__ ______.. .. _ .... __ __ _-...... ., ..... . . . .. L . . . .. . . . .. _ .

_____71

. . ... -- :::_. -

- - .---- .. . ..- -- +-- " "- - . . . "" . . . .

________ -... e __ c.., .L .....

...... .. ... - --- + -.. --- .. L,- . . . . -"+

.. .. I -4+- - - - --

, ,, , _ . . .- -- / -. . ..... . -.-. --. __ _ __ __ + _ . __'- - - - - - --, ,__ . --- - - . _- "- -,_

.4 2 . -

--" '' -- • : '; -: -- "* .- -.- - .. +,. ......... - ... .. -

. .- ~ -. ..--..--. -... . . ,.- - - ' ' . , -- +- - -,

,+ . .- , .. '__ __ _ . ...... A..... ......

*" -- - " .A.A. - .. .- I.. .- 1I-- .;-.- - ;- ,

_ - _ .. . .. A. __ . ,_ _ . . - -._'- - -- -- " - - - . . _ ' L'F . - -+

, -I '--':. ... . T " 72 Z- ' L'_ 'z- .'.. -.... _-_

... ... .2UET ON M Y VARMIC 1'.

- -.-- at ...... . ~-- - o o

vtSY M# .- - - ---...

- -S.STM #

--70-.

-. . - .- . ~~- - - - .- . -

____ -4

tF . 2Zz.---- --- 4~~5

41

7.0 Conclusion

The primary aim of this thesis was to present a model of

software system parameter estimation, to subsequently compare predicted

system characteristics with actual field data, and to guide the software

manager in the practical application thereof to software systems under

his development.

Table 6 sumrmar-Ze3 model pa- -moters by system and method of

calculation. It can be seon that ?= prediction accuracy varied from

20% to 67%, depending on the system, with an overall average of 45%.

Surprisingly enough, there was less than 1% diffe-ence in

prediction accurcy between the three different methods of model para-

meter calculation Method of Moments, Least Squares, and Max-mum Likeli-

hood EstLxates. Although most statisticianr consider ?ITW to yield

superior results to other methods, the findings of tW.s thesis would

indicate that iarimeter estimation using Least Squares is preferable

for the software manager due to the less cumbersome calculations re-

quired.

Model parameter sensitivity was explored and indicated that

a variation in K resulted in a linear variation in estimated MTTF. This

was not the case with variations in total system esti.ated errors ( ),

however, .s minute varlatioe.s in the latter resulted in drastic changes

in predicted YsTTF.

Once model parameters and system M71F have been calculated,

it is of great interest to the software manager to use these calculations

to determine/predict the optimal release date for the system under develop-

ment. Sections 2.3 and 6.2 suggest several ways of doing no.

42

To= 6

C~aLt(~ (F i~~E. BICTI(.INS TM D ZXL MAX-CZ

$yet. '10. __________ ____ < I T O (cxm lMrjsAY 9

i1 .MCM 5514 152 136 .. 3 15.1 - 1.6 66.6 120.4. 39.7I 5.515 151 33.9 66.6

2 N 1 2 5 . 14 1 1.2 5 . 2 A4:5.2. .2

3.909 5 .935 13.6 12.7 31.. 59.5 I.53.909 62 .935 12.7 59.53.909 62 .93j 12.7 59.6

1.6o 40 38 1.13 13.2 15o 0.3 0.4 130.4 0.3311.603 40 1.14 15o 0o.311.60 o1.3 15-0 0.

19.88 55 53 :735 13.1 9.63 9-17 5 19.93 735 9.63 58.1

- Ardei'ted .MF .sed on te-Stng tiMe

.MIF - Predicted opertional (field) vrw

- Actujal operational (field) VW

%Idi - % difference between actual and predicted nW (exponential aodel)

%A2 - % dIfferor.ce between actual and predicted IT3 (SA'S model)

,MCM - Method of Momenta

is - Least Squares

HU - !axium Likelihood Estimate

143

1. Maxn L. 3hocman, "Probabilistic Models for Software Reliability

Pzediction", published in "Statistical Computer 1-erfczance

Evaluation", 4alter Freiberger editor, Academic Press, New York,

1972, pp. 485-497.

2. , "Software Reliability", dablished in

"Computing Systems Reliability", T. Anderson and 3. Randell editors,

Cambx-dgs University Press, New York, 1979.

3. , "Software Engineeringt Design, Reliability,

Management", Mc~raw-Hill Zook Co., New York, 1980.

4,. M.L. Shooman and S. Natazs-jan, "Effect of Manpowe- Deployment and

Bug Generation cn Software Error Models", Proced.ings of the 1976 MRI

Symposiu on Computer Soft-are Engineering, Folytcchnic Press, 1976.

5. J. Dickson, J. Hesse, A. Kientz, M. Shoomazn, "Quantitative Analysis

of Software Reliability". 1972 Annual Reliability Symposium Proceedings,

, Jan. 1972.

6. John D. Musa, "Valadity of Execution-Time Theory of Software Re-

1iability", IEEE Transactions on Reliability, S;3cil Issue on Soft-

ware Reliability, August 1979, vol. A-28, No. 3, p. 181-191.

4J4

7., Martin L. Shooman, "Probabilistic Reliabilitys An Engineering

Approach", McGaw-Hill Book Co., New York, 1968.

8. John D. Musa, "A Theory of Software Reliability and Its Applicaticn",

=E Transactions on Software Engineering, Vol. SE-1, No. 3, Sept. 75,

Pp. 312-327.

9. M.L. Shocman, "Operational Testing and Software Reliability Esti-

mation During Praogram Development", Record 1973, E= Symposium on

Cmputer Software Reliability, April 30 - May 2, 1973 IEEZ, New York,

pp. 51-57.

10. John D. Musa, "Software Reliability Data", Data and Analysis Center

for Software (DACS), Rome Air Development Center, Griffiss AFB,

N.!. 13441, 1979.

iie "Software Enineering Research Review-Quantitative Software Models",

Data and Analysis Center for Software (DACS), Rcmo Air Development

Center, Griffiss AFB, N.Y. 13441, Harch 1979.

12, M.L, Shooman, "Software Reliability Data Analysis and Model Fitting",

Pmeedings of Workshop on Quantitative Softwax-e Models, IEE, N.Y.,

Oct. 1979, pp. 182-188.

45

13. I. ?(iya~moto, "Software Reliability in Online Real Time Environment",

PAneedings, International Conference on Reliable Software, April 1975,

E= Cat. No. 75CH0940-7CSR.

14. M.L. Shooman and R.W. Schmidt, "Fitting of Software Error and

Reliability Models to Field Failure Data", January 1981, ORSA-TDlS

Conerence.

.......................