capture-recapture Single Defect

Supplemental Data To Linked In Post On 6/29/2015 On Capture-

Recapture For Software Inspections With A Single Defect

James K. Orr6/30/2015

Motivation For This Presentation• I have been reviewing information that was still available from my time with the

Space Shuttle Primary Avionics Software (PASS) system.• In August / September, 2011, I emailed a number of files to colleagues continuing

to work at NASA. This material was on PASS quality data and discrepancy reports/analysis, with copies sent to my home email address.

• Recently, I was reviewing what information I still had access to, and moving data from email attachments to more permanent backup.

• In doing this, I realized that I did not have any of the data on the study that I did on using the Capture-Recapture concepts applied to inspections with a single defect.

• I created the post on 6/29/2015 by dumping what I remembers in a single composition activity.

• In hindsight, the absence on any data probably means that most readers will miss key insights that I gained from the original data.

• This presentation is an attempt to quantify the concepts, although by definition I am limited to artificial / simulated data only.

Warning On Potential For Misuse

• This presentation attempts to show significant advantages in use of data as collected on the Space Shuttle Primary Avionics Software (PASS) project for quality and process management.

• However, this data must never be used in personnel decisions such as appraisals.

• In the hands of software process professionals, it can provide accurate assessment of the state of the project and information for team training.

Reference, Background Information

• Reference:– http://www.sei.cmu.edu/library/assets/capture-recapture1.pdf– Getting More Out of Your Inspection Data: Using Capture-Recapture

Models for the Reinspection Decision; Julie Barnard, Khaled El Emam, and Dave Zubrow; European SEPG Conference – Amsterdam - April 12, 2002

• In the sample of data that I was using, there were 861 inspections, of which 553 had no major errors. Of the 308 with major errors, only 89 had meet the criteria for the above referenced study. My memory is that there were 71 inspections that had one and only one major error present.

http://www.sei.cmu.edu/library/assets/capture-recapture1.pdf

Source Of Data• I have spent about two hours recreating a simulation of software inspections

for the following narrow scope:– Inspection randomly (equally probably) have 3 to 6 inspectors– Inspection material contains one and only one major error– Probably that a single inspector will detect the one major error crudely

model on my memory of Space Shuttle Primary Avionics Software (PASS) project consistent with 2001 study in collaboration with Software Engineering Institute • Probability of a single inspector detecting on major error was

between 0 % and 40 % (randomly distributed).

Comment On Inspector Effectiveness

• For the entire set of 308 inspections, I analyzed each inspector for their individual effectiveness in finding major Errors– The data used in the SEI collaboration was provided to me with all indication of inspector’s

name converted to aliases. The maintainer of the inspection data could relate an alias to an individual inspector. I never desired nor never knew the identity of the individual inspectors; I only worked with the aliases.

– I identified data for all aliases (all inspectors) in the 308 inspections with major errors. For each alias, I computed the present of errors found by the individual alias (individual inspector) in all inspections by the total number of errors and later defects found in all inspections that the individual participated. This represented the probability that this individual would discover an error present in an individual inspection.

• Summary– A relatively large percentage of inspectors had never participated in a single inspection with

a major error present (553 of 861 inspections had no major errors).– Several inspectors had zero major error detected when in inspections with one or more

major errors.– For those inspectors who had been in inspections with a large number of major errors,

most had detected less than 40 % of the major errors present.

Results• Data is show for three separate simulations• Data presented is

– Probability that single major error is found by one or more inspectors– Probability that single major error is found by two or more inspectors

• Statistical simulation for each inspection (number of inspectors, fixed probability of finding error for each inspector) was done 1000 times

• “Actual” results was simulated by randomly selecting the outcome (results) on only one of the 1000 simulations

• This was done for 80 different inspections• 80 inspections were grouped into 8 groups of 10 each where the

lowest group contained the 10 inspections with the lowest mean probability of finding the error and the highest group contained the 10 inspection with the highest mean probability of finding the error.

Data For Simulation # 1

• Data will be shown for three simulations• Inputs for all three are identical• The only differences are the random statistical

differences (e.g., a different random number selected to simulate “actual”, different inspection team randomly assigned characteristics)

Probability Of Major Error Found

Probability Of Error Found By Two Or More Inspectors

Correlation Of Probability Of Two Inspectors Finding Error To Inspection Effectiveness


• The only differences are the random statistical differences (e.g., a different random number selected to simulate “actual”, different inspection team randomly assigned characteristics)





• The only differences are the random statistical differences (e.g., a different random number selected to simulate “actual”, different inspection team randomly assigned characteristics)




Documents

capture-recapture Single Defect