Extension of E(Θ) Extension of E(Θ) metric for Evaluation metric for Evaluation

of Reliabilityof Reliability

PresentationPresentation

►Background on Reliability and E(Θ) Background on Reliability and E(Θ) ►Testing MethodologiesTesting Methodologies►Present formulae for calculating E(Θ)Present formulae for calculating E(Θ)►Need of a new formulaNeed of a new formula►Formula DerivationFormula Derivation►Validation of Derivation TechniqueValidation of Derivation Technique►Summary and Future WorkSummary and Future Work

E(Θ)E(Θ) ? ?

►E(Θ)E(Θ) - Reliability of a software is - Reliability of a software is defined in terms of its E(Θ) value.defined in terms of its E(Θ) value.

►E(Θ) refers to the probability of the E(Θ) refers to the probability of the software encountering a failure software encountering a failure subsequent to testing subsequent to testing

Why measure reliabilityWhy measure reliability

► Cost of Designing a Faulty S/W in less Cost of Designing a Faulty S/W in less

time >> Cost of Designing a Reliable time >> Cost of Designing a Reliable S/WS/W

► Safety Critical ApplicationsSafety Critical Applications

Faults or BugsFaults or Bugs

► Failure – Unexpected O/PFailure – Unexpected O/P

► Faults – I/Ps causing FailuresFaults – I/Ps causing Failures

How to measure reliabilityHow to measure reliability

Testing:Testing:

►Means of Probing S/W for bugsMeans of Probing S/W for bugs

►Means of gaining confidence about its Means of gaining confidence about its reliabilityreliability

Source : Source : Evaluating Testing Methods by Delivered Reliability -Phyllis G. Frankl, Richard G. Hamlet, Member, IEEE, Bev Littlewood, Member, IEEE,and Lorenzo Strigini, Member, IEEE

Testing MethodsTesting Methods

►Operational – Operational –

Test Case = Statistical Test Case = Statistical Distribution or Distribution or Random (I/P domain)Random (I/P domain)

►Debug –Debug –

Test Case = Selective Sampling Test Case = Selective Sampling (I/P (I/P Domain)Domain)

Operational Vs. DebugOperational Vs. Debug

►Debug – Probes for bugs actively Debug – Probes for bugs actively (assumes S/W is faulty)(assumes S/W is faulty)

►Operational – Waits for Failures to Operational – Waits for Failures to occur.occur.

Sub-Domain TestingSub-Domain Testing

► A debug testing technique.A debug testing technique.

► Concentrates test-cases to specific sub-Concentrates test-cases to specific sub-domains with high failure probabilitydomains with high failure probability

► Based on Debugger’s IntuitionBased on Debugger’s Intuition

► Eg. Boundary Testing – Bugs lie in the Eg. Boundary Testing – Bugs lie in the cornerscorners

ExampleExample

0- ve x

+ ve x

- ve y

+ ve y

Failure Region

Input domain

Failure Region

Sub-Domain Testing So Sub-Domain Testing So Far…Far…

Terminologies usedTerminologies used

►q = failure probabilityq = failure probability►d = detection rated = detection rate►T = Number of test cases per sub-T = Number of test cases per sub-

domaindomain►n = n number of sub-domains/failure n = n number of sub-domains/failure

regions regions

Single Domain - Single Failure Single Domain - Single Failure RegionRegion

A, 20Failure Region

B, 30Domain nature:

Single Domain = 1 – 50 (50 points)

Failure Region A = 21 – 40 (20 points)

E(Θ) = q( 1 - d)E(Θ) = q( 1 - d)TT

Single Failure Region – Multiple Single Failure Region – Multiple Sub-DomainsSub-Domains

Ai,5 Aj, 15 Ak,10

B, 20 D, 25Domain nature:

Sub-Domain 1= 1 – 25 (25 pt.s) Sub-Domain 2= 25 – 50 (50 pt.s) Sub-Domain 3= 50 – 85 (35 pt.s) Failure Region A = 40-70 (30 pt.s)

E(Θ) = q E(Θ) = q nnΠΠi=1i=1 ( 1 - ( 1 - ddii))T T

C, 10

Multiple Failure Regions – Single Multiple Failure Regions – Single Sub-DomainSub-Domain

A, 10

B, 20

D, 55Domain nature:

Domain = 1 – 100 (100 pt.s)

Failure Region A = 20-30 (20 pt.s)Failure Region B = 50-70 (20 pt.s)Failure Region C = 80- 95(20 pt.s)E(Θ) = E(Θ) = nnΣΣi=1 i=1 q qii( 1 - d( 1 - dii))T T

C, 15

Multiple Failure Regions – Multiple Failure Regions – Multiple Sub-DomainsMultiple Sub-Domains

A, 20

B, 20

C, 30 D, 30Domain nature:

Sub-Domain 1= 1 – 50 (50 pt.s) Sub-Domain 2= 50 – 100 (50 pt.s) Failure Region A = 30-50 (10 pt.s)Failure Region A = 70-90 (20 pt.s)E(Θ) = ?E(Θ) = ?

No definitive Formula. Empirical Testing only SolutionNo definitive Formula. Empirical Testing only Solution

And Now……And Now……

New Set of TerminologiesNew Set of Terminologies

►q = Total failure probabilityq = Total failure probability►ddAA = Detection rate of a failure region = Detection rate of a failure region

in a in a sub-domain sub-domain►FFii = Number of points in a failure = Number of points in a failure

regionregion►M = Total number of failure points in M = Total number of failure points in

the the sub-domain sub-domain

Multiple Failure Regions – Multiple Failure Regions – Multiple Sub-DomainsMultiple Sub-Domains

A, 20

B, 20

C, 30 D, 30Domain nature:

Sub-Domain 1= 1 – 50 (50 pt.s) Sub-Domain 2= 50 – 100 (50 pt.s) Failure Region A = 20-40 (20 pt.s)Failure Region A = 70-90 (20 pt.s)

E(Θ) = q – ( dE(Θ) = q – ( dAAFF11/M+ d/M+ dBBFF22/M)/M)

Formula DerivationFormula Derivation

A, 20 B, 20

D, 30 Fault Case Regions Detected Faults Remaining

00 None A, B

01 B A

10 A B

11 A, B None

E (Θ) = (Probability that the particular case will occur) * (probability that the rest of the faults are undetected)

C, 30

Contd.Contd.Assuming 1 test-case per sub-domainAssuming 1 test-case per sub-domain

Test Case 1 Test Case 2 Fault A Fault B Case Number

Region D Region C undetected

Region D

Region A

Region A

Region B

Region C

Region B

undetected

undetected detected

detected undetected

detected detected

Seq. 1

Seq. 2

Seq. 3

Seq. 4

E (Θ) = E(Θ) (seq1) + E(Θ) (seq2) + E(Θ) seq3) + E(Θ) seq4) contd.

E (Θ) = E(Θ) (seq1) + E(Θ) (seq2) + E(Θ) seq3) + E(Θ) seq4)

= {(1- dA)* (1- dB)* (q)} + {(1-dA)* (dB)* (q –F1/M)} + {(dA)* (1- dB)* (q –F2/M)} + {(dA)* (dB)* (q –F1/M – F2/M)}

= {(30/50 * 30/50) * 40/100} + {(30/50 * 20/50) * 20/100} + {(20/50 * 30/50) * 20/100} + {(20/50 * 20/50) * 0/100}

= 0.144 + 0.048 + 0.048 = 0.24

Verification of the TechniqueVerification of the Technique

A, 20B, 20

C, 60Domain nature:

Domain = 1 – 100 (100 pt.s)

Failure Region A = 20-40 (20 pt.s)Failure Region A = 70-90 (20 pt.s)

Approach I:Approach I:E(Θ) = E(Θ) = nnΣΣi=1i=1 q qii( 1 - d( 1 - dii))TT

= 20/100(1- 20/100) + 20/100(1- 20/100)= 20/100(1- 20/100) + 20/100(1- 20/100) = 0.32

Approach II: E(Θ)E(Θ) = {(1- dA)* (1- dB)* (q)} + {(1-dA)* (dB)* (q –F1/M)} + {(dA)* (1- dB)* (q –F2/M)} + {(dA)* (dB)* (q –F1/M – F2/M)} contd.

= {(80/100 * 80/100)(40/100)} + {(80/100 * 20/100)(20/100)}+ {(20/100 * 80/100)(20/100)} + {(20/100 * 20/100)(0/100)}

= 0.32

Both approaches yield the same result

Similar tests done for different scenarios validate the 2nd technique of calculating E(Θ)E(Θ) with respect to Frankl’s Method

Getting back to Formula Getting back to Formula DerivationDerivation

Upon further solving the method just described we get ….

Method ReductionMethod Reduction

E (Θ) = { (1- dA)* (1- dB)* (q) } + { (1-dA)* (dB)* (q –F1/M) } + { (dA)*(1- dB)* (q –F2/M) } +

{ (dA)* (dB)* (q –F1/M – F2/M) }

= q [ {1–dB – dA + dAdB} + { (dB – dAdB) (1–F2/M*q) } + { (dA – dAdB) (1-F1/M*q) } + { dAdB (1–F1/M*q – F2/M*q) } ] Contd.

= q [ {1–dB –dA + dAdB} + { dB – dB*F2/M*q – dAdB + dAdBF2/M*q}+

{ dA – dAF1/M*q – dAdB + dAdBF1/M*q} + { dAdB–dAdB*F1/M*q – dAdBF2/M*q}

]

= q [ 1– dBF2/M*q – dAF1/M*q]

= q – ( dAF1/M + dBF2/M )

Three Failure Regions &Three Failure Regions & Three Three Sub-domainsSub-domains

E (Θ) = {(1- dA)* (1- dB)* (1- dC)*(q)} + {(1-dA)* (1-dB)* (dC)*(q –F3/M)} +

{(1-dA)* (dB)* (1-dC)*(q –F2/M)} + {(1-dA)* (dB)* (dC)*(q –F2/M–F3/M)} + {(dA)* (1-dB)* (1-dC)*(q –F1/M)} +

{(dA)* (1-dB)* (dC)*(q –F1/M–F3/M )} + {(dA)* (dB)* (1-dC)*(q –F1/M–F2/M)} + {(dA)* (dB)* (dC)*(q –F1/M–F2/M–F3/M)}

= q[ 1 – dCF3/ M*q – dBF2/M*q – dAF1/M*q]

= q – { dAF1/M + dBF2/M + dCF3/M}q – { dAF1/M + dBF2/M + dCF3/M}

SimulatorSimulator

►Based on the above mentioned & Based on the above mentioned & verified technique a simulator was verified technique a simulator was designed for empirical calculation of designed for empirical calculation of E (Θ)

►Coding Language : JavaCoding Language : Java

►Graphical User Interface : Java SwingsGraphical User Interface : Java Swings

Simulator screen shotsSimulator screen shots Step I:

Specify the total number of Sub-Domains & Failure Regions

Step II:Step II:

A, 20 B, 20

C, 30D, 30

Specify the number of points per Sub-Domain, Failure Specify the number of points per Sub-Domain, Failure RegionRegion

The result got from running the simulator is similar to the one calculated theoretically (above). This proves validity of the Simulator Design.

SummarySummary

►Value of E (Θ) for software with multiple failure regions spread across multiple sub-domains can be calculated using: E (Θ) = q – ( dAF1/M + dBF2/M)

E (Θ) = q – ( dAF1/M + dBF2/M + q – ( dAF1/M + dBF2/M +

dCF3/M)dCF3/M)

Future WorkFuture Work

►The number of test-cases per sub-The number of test-cases per sub-domain was limited to only 1domain was limited to only 1

►We would like to generalize the present We would like to generalize the present formula further by considering the formula further by considering the effect of ‘n’ (multiple) test cases per effect of ‘n’ (multiple) test cases per sub-domain.sub-domain.

►No generalization doesn’t merely No generalization doesn’t merely involve putting ‘T’ as a exponential involve putting ‘T’ as a exponential term in the equation ( have tried that )term in the equation ( have tried that )