Imperfect Debugging in Software Reliability · Debugging Stage in Software Reliability The goal of software testing (i.e. debugging) is to detect/ﬁx software faults (bugs) inherent

Imperfect Debugging in Software Reliability

Tevfik Aktekin and Toros Caglar!

University of New HampshirePeter T. Paul College of Business and Economics

Department of Decision Sciencesand United Health Care!

July 9, 2013

1 / 26 Tevfik Aktekin and Toros Caglar! Imperfect Debugging in Software Reliability

Outline

Outline:

What is Meant by Software Reliability and Debugging?

Motivation and Uncertainties of Interest

Very Brief Overview of Software Reliability Models

The Proposed Model and Its Bayesian Inference

Numerical Applications of the Proposed Model

Investigating The Existence of Perfect vs. ImperfectDebugging in Software Data


What is Meant by Software Reliability?

Definition

Software Reliability: Can be defined as the probability ofsuccessful performance of software for a specified time intervalunder certain conditions. The fact that the software runs withoutany problems implies that the code is generating the intendedoutput.

A failure is said to occur on a piece of software, when a fault(or a so called bug) causes the software to fail.

Note: If a failure does not occur for a period of time, thisdoes not necessarily imply that the software is bug free.


Debugging Stage in Software Reliability

The goal of software testing (i.e. debugging) is to detect/fixsoftware faults (bugs) inherent in the software code and todecide when to release the software.

Software testing is expensive. Thus, there is a trade-o!between releasing a reasonably good piece of software andkeeping to debug.

There are many examples of buggy software being releasedand negatively a!ecting sales (first impression in the market ingaming software for instance).

The testing stage consists of several consecutive programexecutions. Whenever a failure occurs, the software engineerattempts to fix the problem.

As the faults reveal themselves and are eliminated by thesoftware engineer, the reliability of the software tends toincrease if no new faults are introduced to the software duringthe elimination stage.


Perfect vs. Imperfect Debugging

Definition

Perfect Debugging: If during the debugging stage the fault whichcaused the failure has been eliminated permanently and no newfaults are introduced, then a perfect debugging is said to haveoccurred (software reliability gets better).

Definition

Imperfect Debugging: Loosely speaking, if during the debuggingstage a fault is detected and is not eliminated permanently (ex: byintroducing new faults), then an imperfect debugging is said tohave occurred (software reliability stays the same or worsens).

Note that there are many possible definitions of imperfectdebugging (dealing with multiple faults vs. a single fault or thetype of software being dealt with as in gaming vs. wordprocessing).


What is Meant by Software Reliability?

P(T |!) is usually referred to as a software reliability model ,where T represents the time to failure of a piece of software.It represents the probability that a particular piece of softwarewill fail in a given interval.

P(T ! t|!) for some t ! 0 represents the reliability functionof T .

Note that here the notion of time is based on the missiontime t, in other words the time during which the software isexecuted.


Statistical Motivation and Uncertainties of Interest

Carry out inference on the unknown parameters such as thenumber of faults inherent in the code and fault detection rateafter each stage of testing.

Obtain the (predictive) reliability function after eachdebugging stage as a function of the parameters of interest.

P(Ti+1 ! t|!,D). (1)

Making a decision on whether to stop the testing and releasethe software based on the reliability assessment of thesoftware at time t.


Software Reliability Models in Literature

The following two can be considered to be the building blocks formost of the current software reliability models:

Jelinski and Moranda (JM) Model (1972)Each fault is permanently removed upon failure (perfectdebugging).Each fault contributes equally to the failure rate at any stageof the testing.

Littlewood and Verall (LV) Model (1973)Both assumptions from the JM Model have been relaxed bythe LV Model; perfect debugging and equally likelycontributions from faults at each stage of debugging.


Proposed Model

Consider modeling of a multiplicative failure rate model whosecomponents are evolving stochastically over testing stages (anNHPP type model).

Proposed model is based on the Jelinski Moranda (JM) modelas is the case for most subsequent work in the softwarereliability literature.

Two of the main assumptions of the JM model is that everyfault contributes equally to the failure rate at any stage oftesting and that each fault is removed permanently uponfailure.

Consider the case where each fault is removed permanentlyupon failure, along with the possibility of introducing newfaults during debugging.


Proposed Model and Definitions

ti , for i = 1, . . . ,N: time until failure during (or prior to) theith stage of testing.

"i , for i = 1, . . . ,N: fault detection rate per fault during (orprior to) the ith stage of testing.

#i , for i = 1, . . . ,N: the number of faults present on thesoftware code during (or prior to)the ith stage of testing.

#1: the number of faults present on the software code during(or prior to) the first stage of testing (i.e. prior o testing).Note that #i s will be functions of #1 and "i s.

"i#i : the failure rate of the software during (or prior to) theith stage of testing.


Proposed Model

Assume that the inter-failure times, ti , are exponentiallydistributed. The likelihood would be

L(",#;D) =N!

i=1

"i#iexp{"ti"i#i}, (2)

where " = {"1, . . . ,"N}, # = {#1, . . . ,#N} and D = {t1, . . . , tN}.Therefore the joint posterior of " and # would be given by

p(",#|D) #N!

i=1

"i#iexp{"ti"i#i}p(")p(#). (3)


Modeling the fault detection rate per fault, !i

"i s be given by the following power law relationship:

"i = "!i"1 $ $i , for i = 1, . . . ,N, (4)

where $i % LN(0, %2). We can obtain the following the linearmodel in logarithms:

log("i ) = &log("i"1) + 'i , for i = 1, . . . ,N, (5)

where 'i = log($i ). (5) is a first order autoregressive process of thelatent fault detection rates per fault in the log scale. Theconditional distributions of log("i )s can be written as

log("i )|log("i"1),& % N(&log("i"1),%2), for i = 1, . . . ,N, (6)

Note the scale of the inter-failure times (all above one in ournumerical example). "i s were all fractions and & was found to bebetween zero and one. To keep the interpretation of & consistent,it is always possible to rescale the data such that they are all aboveone.12 / 26 Tevfik Aktekin and Toros Caglar! Imperfect Debugging in Software Reliability

Modeling the fault detection rate per fault, !i

The relationship implied by (4) also dictates the type of debuggingthat occurs during the ith debugging stage.

If "i < "i"1, then perfect debugging is said to have occurred.

If "i ! "i"1, then imperfect debugging is said to haveoccurred.

In other words, when a failure is detected at the (i " 1)th failureepoch, a fault has been detected and repaired, however a new faultwas introduced during the same debugging stage. & determines onaverage how the fault detection rate per fault is changing fromstage to stage. For instance, when 0 < "i"1 < 1 and & > 1 thenperfect debugging tends to occur, conversely when 0 < & < 1 thenimperfect debugging tends to occur.


Modeling the total number of faults, "i

Conditional on whether perfect or imperfect debugging hasoccurred during the previous debugging stage, the total number offaults left in the software code,#i , is assumed to have the followingstructure

#i = #i"1 " (i , for i = 1, . . . ,N (7)

where

(i = 1, with probability p("i < "i"1)

= 0, with probability p("i ! "i"1) for i = 1, . . . ,N.

In (7), (i is a Bernoulli process whose probability of success is theprobability of perfect debugging, p("i < "i"1). When perfectdebugging occurs, #i goes down by one unit, since the fault thathas caused the failure has been found and fixed. When imperfectdebugging occurs, #i stays the same, since the fault that hascaused the failure has been found and fixed, however a new faulthas been introduced while fixing the previous one.14 / 26 Tevfik Aktekin and Toros Caglar! Imperfect Debugging in Software Reliability

Other Model Priors

For the model on "is,

%2 % Gamma(a, b) (8)

and& % U(c , d) (9)

and for the initial fault detection rate, "1, we assume the following

"1 % LN(e, f ) (10)

For the initial number of inherent faults, #1, we assume thefollowing

#1 % Poisson(!) (11)

with! % Gamma(g , h). (12)


Markov Chain Monte Carlo Estimation

Goal: To generate samples from

p(!, "1, #,!, $2|D) #

N!

i=1

%i Xi exp{"ti%i"i}p(!|!, $2)p("1|#)p(!)p($2)p(#). (13)

In (13), the conditional joint prior distribution for "i s usingthe chain rule and dropping terms that are independent canbe obtained as

p("|&,%2) = p("N |"N"1,&,%2) . . . p("2|"1,&,%

2)p("1). (14)



To generate the full conditionals

p("i | . . . ,D) for i = 1, . . . ,N: Use Metropolis-Hastings.

p(#1| . . . ,D): Discrete. In addition, one can compute #j as#j"1 " 1("j < "j"1) for j = 2, . . . ,N once we have therequired samples.

p(!| . . . ,D): Gamma

p(&| . . . ,D): Normal

p(%2| . . . ,D): Use Metropolis-Hastings.



To generate samples from p(!, "1, #,!, $2|DN)

1 Assume the starting points ("(0)1 ,%

(0)2 , #(0), ($2)(0), !(0)) and set l=1.

2 Generate %(l)1 using "

(l"1)1 , %

(l)2 , !(l"1) and ($2)(l"1) from (%1| . . . ,D).

3 Sequentially generate %(l)i for i = 2, . . . ,N using "

(l"1)1 , !(l"1), ($2)(l"1) and

%(l)i"1

from (%i | . . . ,D).

4 Generate !(l) using ($2)(l"1) and %(l)i for i = 1, . . . ,N from (!| . . . ,D).

5 Generate ($2)(l) using !(l) and %(l)i for i = 1, . . . ,N from ($2| . . . ,D).

6 Generate "(l)1 using %

(l)i for i = 1, . . . ,N and #(l"1) from (X1| . . . ,D.

7 Sequentially compute "(l)i for i = 2, . . . ,N using "

(l)i"1 and %

(l)i for i = 1, . . . ,N

via "i = "i"1 " 1(%i < %i"1).

8 Generate #(l) using "(l)1 from (#| . . . ,DN ).

9 Set l=l+1 and go back to step 1.


Numerical Example

Dataset 1: The numerical application of our model is carriedout on the well known dataset first reported in JM 1972. Thedataset consists of 31 software inter-failure times, 26 of whichwere obtained during the production stage of debugging andthe remaining 5 during the rest of the testing stage. In ourexample, all 31 inter-failure times were used (most inference isbased on this one).

Dataset 2: The military systems application data (data 17) ofJohn Musa of Bell Telephone Laboratories 1 with 38inter-failure times. (only used for comparison purposes).

Model Comparison: Use the harmonic mean estimator of themarginal likelihood and the DIC.

Models to Compare Against: MS88-M1, MS88-M2,KY96-GOS-W, KY96-RVS-P and a simple perfect debugging(PD) model.

1http://www.thedacs.com/databases/sled/swrel.php19 / 26 Tevfik Aktekin and Toros Caglar! Imperfect Debugging in Software Reliability

Numerical Example

ID PD MS88-M1 MS88-M2 KY96-GOS-W KY96-RVS-P

log{p(D)} -108.55 -191.06 -116.51 -113.81 -114.01 -111.77DIC 215.49 218.09 227.76 226.24 223.26 222.90

Table: log{p(D)} and DIC for the JM dataset

ID PD MS88-M1 MS88-M2 KY96-GOS-W KY96-RVS-P

log{p(D)} -50.41 -50.49 -57.87 -53.98.81 -53.36 -55.03DIC 101.95 100.58 114.52 107.89 105.45 109.59

Table: log{p(D)} and DIC for the MUSA dataset


Summary of Findings

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31

0.00

0.01

0.02

0.03

0.04

0.05

Debugging Stages

φ i

0 5 10 15 20 25 300.

10.

20.

30.

40.

50.

60.

70.

8

Debugging Stages

Prob

abilit

y of

Per

fect

Deb

uggi

ng

Figure: Boxplots of "i for i = 1, . . . , 31 (left) and the probability ofperfect debugging vs. debugging stages (right)


Summary of Findings

0.80 0.85 0.90 0.95 1.00 1.05

05

1015

β

Den

sity

0.000 0.005 0.010 0.015 0.020 0.025 0.030

020

4060

8010

012

0

φ1

Den

sity

10 20 30 40

0.00

0.02

0.04

0.06

0.08

θ

Den

sity

X1

Den

sity

10 12 14 16 18 20 22

0.00

0.05

0.10

0.15

0.20

Figure: Posterior distribution plots of &,"1, ! and #1


Summary of Findings

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31

510

1520

Debugging Stages

X i

Figure: Boxplots of #i for i = 1, . . . , 31


Summary of Findings

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31

0.00.1

0.20.3

0.40.5

0.6

Debugging Stages

Failu

re Ra

te

Figure: Boxplot of the failure rates, "i#i , for i = 1, . . . , 31


Predictive reliability function estimation

The predictive reliability function during the ith testing stage(given that the software has gone through (i " 1)th stages oftesting) can be computed via

R(ti |D(i"1)) = 1"

1

S

S"

j=1

F (ti |"(j)i ,#

(j)i ,D(i"1)). (15)

Once (15) is estimated it can easily be used as part of a softwarereliability optimal release scheme.

0 10 20 30 40 50

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Predictive Reliability Function After 28 Stages of Testing

t

R(t|

D)

0 10 20 30 40 50

0.4

0.5

0.6

0.7

0.8

0.9

1.0


t

R(t|

D)

0 10 20 30 40 50

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0


tR

(t|D

)

Figure: Predictive reliability functions for i = 29, . . . , 31


Concluding Remarks and Future Work

Introduce a Markov chain type of structure on the number ofbugs repaired or introduced during the debugging stageinstead of the current Bernoulli setup.

Investigate the possibility of state space evolution of the &

coe$cient in the power law relationship between theinter-failure times.

Note: Both extensions would be challenging from an MCMCestimation point of view.


Documents

Imperfect Debugging in Software Reliability · Debugging Stage in Software Reliability The goal of software testing (i.e. debugging) is to detect/ﬁx software faults (bugs) inherent