Upload
clare-morris
View
212
Download
0
Embed Size (px)
Citation preview
What Do We Know about What Do We Know about Defect Detection MethodsDefect Detection Methods
P. Runeson et al.; "P. Runeson et al.; "What Do We Know What Do We Know about Defect Detection Methods?about Defect Detection Methods?", IEEE ", IEEE Software, May/June. 2006 Page(s): 82-90. Software, May/June. 2006 Page(s): 82-90.
SummarySummary
A survey of defect detection studies comparing A survey of defect detection studies comparing inspection and testing techniquesinspection and testing techniques
Yields practical recommendations:Yields practical recommendations: Use inspections for requirements and design Use inspections for requirements and design
defectsdefects Use testing for codeUse testing for code
Article Vocab ReviewArticle Vocab Review
Evidence-based software engineeringEvidence-based software engineering Determines which methods to use and for what Determines which methods to use and for what
purposepurpose EBSE involves relevant questions, surveying and EBSE involves relevant questions, surveying and
appraising available evidenceappraising available evidence Integrates and evaluates new practices in a target Integrates and evaluates new practices in a target
environmentenvironment Used as basis of article to help determine most Used as basis of article to help determine most
efficient and effective methods for defect detectionefficient and effective methods for defect detection
Article Vocab Review (con’t)Article Vocab Review (con’t)
DefectDefect Relates to one or more underlying faults in an artifact such Relates to one or more underlying faults in an artifact such
as codeas code Maps to single faults (faults and defect are used Maps to single faults (faults and defect are used
interchangeably)interchangeably) ArtifactArtifact
RequirementsRequirements DesignDesign CodeCode Using industrial artifacts improves the studies Using industrial artifacts improves the studies
generalizabiltygeneralizabilty
Types of DefectsTypes of Defects
Grammatical defects are much different from missing Grammatical defects are much different from missing requirements in a requirements specificationrequirements in a requirements specification
Testing and inspection methods may deem better or Testing and inspection methods may deem better or worse depending on the type of defectworse depending on the type of defect
Omission (missing) vs. Commission (incorrect)Omission (missing) vs. Commission (incorrect) Unimportant, unimportant, crucialUnimportant, unimportant, crucial Classify defects on origins :Classify defects on origins :
RequirementsRequirements DesignDesign CodeCode
Actors & TechniqueActors & Technique
Who is the reviewer or testerWho is the reviewer or tester Freshman student in an experimentFreshman student in an experiment Experienced software engineerExperienced software engineer
TechniqueTechnique Relative to inspection and testingRelative to inspection and testing Inspection – families of verification techniquesInspection – families of verification techniques Testing – structural (white box) & functional Testing – structural (white box) & functional
(black box)(black box)
PurposePurpose
What's the purpose of the inspection and What's the purpose of the inspection and testing activity?testing activity? Validation – assuring the correct system is Validation – assuring the correct system is
developeddeveloped Verification – assuring the system meets its Verification – assuring the system meets its
specificationsspecifications Both?Both?
Detection (for later isolation) vs. Isolation Detection (for later isolation) vs. Isolation (isolate the underlying fault)(isolate the underlying fault)
Defect detection activitiesDefect detection activities
Consider defect origins:Consider defect origins: Primary defect detection (1)Primary defect detection (1) Secondary defect detection (2)Secondary defect detection (2)
Evaluation criteriaEvaluation criteria
Efficiency - # of defects found per time unit Efficiency - # of defects found per time unit spent on verificationspent on verification
Effectiveness – the share of the existing Effectiveness – the share of the existing defects founddefects found
The Survey (finally)!!!!!The Survey (finally)!!!!!
Data from the studiesData from the studies
The issues covered from Table 2&3The issues covered from Table 2&3
Requirements defectsRequirements defects Design defectsDesign defects Code defectsCode defects Different defect typesDifferent defect types Efficiency versus effectivenessEfficiency versus effectiveness
Requirements defects – Thank god Requirements defects – Thank god for the experts!for the experts!
Different requirements inspection methods; Different requirements inspection methods; same conclusion:same conclusion: Spending effort up front to establish a good set of Spending effort up front to establish a good set of
requirements is more efficient than developing a requirements is more efficient than developing a system on basis of incorrect requirements and then system on basis of incorrect requirements and then reworking itreworking it
Design defectsDesign defects
Is inspecting design documents or testing the Is inspecting design documents or testing the implemented function more efficient?implemented function more efficient? Inspections were significantly more effective and Inspections were significantly more effective and
efficient than testingefficient than testing More than half of the defects (53.5 %) found More than half of the defects (53.5 %) found
during inspectionduring inspection Less found during testing (41.8 %)Less found during testing (41.8 %)
Bottom line : design inspection is the winner!Bottom line : design inspection is the winner!
Code defectsCode defects
Which is better? Code inspection or testing Which is better? Code inspection or testing (functional or structural)?(functional or structural)?
Which is more effective?Which is more effective? Which is more efficient?Which is more efficient? No clear winner No clear winner
We’ll see why next slideWe’ll see why next slide
Average effectiveness for code defect Average effectiveness for code defect detectiondetection
Average efficiency for code defect Average efficiency for code defect detectiondetection
Survey issuesSurvey issues Different defect typesDifferent defect types
Absence or presence of different types of defects might Absence or presence of different types of defects might affect efficiency and effectivenessaffect efficiency and effectiveness
Studies report defects by differing types and cannot be used Studies report defects by differing types and cannot be used to investigate whether the differences depend on fault type, to investigate whether the differences depend on fault type, not only the techniquenot only the technique
Frequency of defects vary widelyFrequency of defects vary widely Only a fraction of defects were found among studies – Only a fraction of defects were found among studies –
affects defect type frequency affects defect type frequency Classification schemes involve subjective judgmentClassification schemes involve subjective judgment
Conclusion: jury is out as to the effect of defect types Conclusion: jury is out as to the effect of defect types on the performance of inspection versus testingon the performance of inspection versus testing
Survey issues (con’t)Survey issues (con’t)
Effectiveness and efficiencyEffectiveness and efficiency Only 25 to 50 percent of defects found during Only 25 to 50 percent of defects found during
inspectioninspection 30 to 60 percent found during testing30 to 60 percent found during testing More than HALF defects remain!!More than HALF defects remain!!
Conclusion: Primary defect detection methods’ Conclusion: Primary defect detection methods’ low effectiveness and efficiency values is that low effectiveness and efficiency values is that secondary detection methods play a larger role secondary detection methods play a larger role that the surveys concludethat the surveys conclude
The results are in!The results are in!
For requirements defects, no empirical evidence exists at all, but the fact that costs for requirements inspections are low compared to implementing incorrect requirements indicates that reviewers should look for requirements defects through inspection.
The results are in!The results are in!
For design specification defects, the case studies and one experiment indicate that inspections are both more efficient and more effective than functional testing.
The results are in!The results are in!
For code, functional or structural testing is ranked more effective or efficient than inspection in most studies. Some studies conclude that testing and inspection find different kinds of defects, so they’re complementary. Results differ when studying fault isolation and not just defect detection.
The results are in!The results are in!
Verification’s effectiveness is low; reviewers find only 25 to 50 percent of an artifact’s defects using inspection, and testers find 30 to 60 percent using testing. This makes secondary defect detection important. The efficiency is in the magnitude of 1 to 2.5 defects per hour spent on inspection or testing.
Results Summary TableResults Summary Table
Questions ???Questions ???