Upload
mimos-berhadopen-university-malaysiauniversiti-teknologi-malaysia
View
217
Download
0
Embed Size (px)
DESCRIPTION
Paper presented in International Symposium on IT 2010 (ITSim2010) as part of World Engineering, Science and Technology Congress 2010 (ESTCON2010)
Citation preview
[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence
Establishing A Defect Prediction Model Using A Combination of
Product Metrics As Predictors Via Six Sigma Methodology
Muhammad Dhiauddin Mohamed Suffian(M.Sc., B.Tech., Six Sigma Green Belt , CTAL-TM, CTFL)
Senior Engineer, MIMOS Test LabApril 8, 2023
[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence
( 2 )
Agenda
[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence
( 2 )
• Defect Prediction Model – An Overview• Project Background• Six Sigma Overview• Prior Studies on Defect Prediction• Findings & Discussion – Define• Findings & Discussion – Measure• Findings & Discussion – Analyze• Findings & Discussion – Design• Findings & Discussion – Verify• Conclusion & Recommendation
[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence
( 3 )
Defect Prediction Model – An Overview
[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence
( 3 )
“Build a functional defect prediction model for system testing phase in a form of mathematical equation by applying
Design for Six Sigma methodology”
Zero-KnownPost Release Defect
Defect Containment inTesting Phase
Wider Test Coverage &When to Stop Testing
Stability ofTesting Process
Test Defect Prediction
Model
[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence
( 4 )
Test Defect Prediction
Model
ISSUES/SCENARIOS• Resource planning for test
execution• Test coverage to discover
defects• Test execution time to meet
project deadline• Reliability of software
RESEARCH QUESTIONS• How to predict the total
number of defects to be found at the start of system testing phase using a model?• What are the key
contributors?• Factors contributing to
system test defect?• Relationship between
factors of defect with the total number of system defects?
• Defect type & total software defects?
• Prediction model & test process improvement?
• Type of data & how to get them?
WHAT HAS BEEN DEMONSTRATED?
• Establish a defect prediction model for software testing phase
• Demonstrate the approach in building a defect
• prediction model using Design for Six Sigma (DfSS) Methodology
• Identify the significant factors that contribute to a reliable defect prediction model
• Determine the importance of defect prediction model for improving testing process
Project Background
[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence
( 5 )
DEFINE
Identify opportunityDevelop team charter & teamProject schedulingFinancial estimationVOC analysis
MEASURE
Identify functional requirementsMeasurement system analysisPerform capability analysisGenerate concept model
ANALYZE
Quantify issues & determine significant factorsIdentify failure modes
DESIGN
Re-define significant factorsOptimize model
VERIFY
Assess reliability of selected designControl planClose project
Six Sigma Overview
A disciplined, data-driven approach and methodology for eliminating defects (driving toward six standard deviations between the mean and the nearest specification limit) in any process -- from manufacturing to transactional and from product to service
(www.isixsigma.com)
D-M-A-I-C D-M-A-D-V (DfSS)vs
[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence
( 6 )
Prior Studies on Defect Prediction
• D = 4.86 + 0.018L (Akiyama)
• D = V/3000 (Halstead)• D = 4.2 + 0.0015(L)4/3
(Gaffney)
Size/Complexity Metrics
TestingMetrics
Process Quality Data
Bayesian Belief Networks (BBN)
Product & Process Metrics
[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence
( 7 )
Findings & Discussion - Define
Requirements Design Developmen
t Testing Deploy
RequirementReview
Test Plan
DesignReview
Test CasesTest Scripts
Test ExecutionDefects Raising
Test Summary Report
Kick-Off
Software Development Life Cycle (SDLC) – Test Participation
[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence
( 8 )
Findings & Discussion – Define (cont.)
Zero-Known Post Release Defects
Defect Containment in Testing Phase
Potential # of
defects before test
Actual # of defects
after test
Customer Satisfaction
Level ofsatisfaction
Quality ofProcess
Peoplecapability
Processeffectivenes
s
ProjectManagement
Timelineallocation
Resourceallocation
High Level Schematic Diagram
[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence
( 9 )
Findings & Discussion – Define (cont.)Test Defect Prediction
Software Complexity
Requirement Pages
Design Pages
Programming
Language
Code Size
Knowledge
Developer Knowledg
e
Tester Knowledg
e
Test Process
Test Case Design
Coverage
Targeted Total Test
Cases
Test Automati
on
Test Case Execution Productivi
ty
Total Effort in
Test Design Phase
Total Effort in Phases Prior to System
Test
Errors
Requirement Error
Design Error
CUT Error
Test Plan Error
Test Cases Error
Fault
Requirement Fault
Design Fault
CUT Fault
Integration Fault
Test Case Fault
Historical Defect
DefectSeverity
Defect Type/
Category
DefectValidity
Total PR (Defects) Raised
Project
Project
Domain
Project
ThreadComponent
Application
Factors to consider
Detail Schematic Diagram – Y to X Tree
(Y)
[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence
( 10 )
Findings & Discussion – Measure
TC TC ID Actual TC Result
Tester 1
Tester 2
Tester 3
1 2 3 1 2 3 1 2 3
1 TC1 PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS
2 TC2 FAIL FAIL FAIL FAIL FAIL FAIL FAIL PASS PASS PASS
3 TC3 FAIL FAIL FAIL FAIL FAIL FAIL FAIL PASS PASS PASS
4 TC4 PASS FAIL FAIL FAIL FAIL FAIL FAIL PASS PASS PASS
5 TC5 PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS
6 TC6 PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS
7 TC7 PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS
8 TC8 FAIL FAIL FAIL FAIL FAIL FAIL FAIL FAIL FAIL FAIL
9 TC9 PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS
10 TC10 PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS
Data for Attribute Agreement Analysis
[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence
( 11 )
Findings & Discussion – Measure (cont.)
• Result is PASS for MSA Within Appraisers as the result shows 100% assessment agreement• Kappa = 1, which shows perfect agreement exist It demonstrates strong Repeatability of TestResult within tester himself/herself
• 3 Persons (Vivek, Sandeep and Kamala) PASS the MSA based on Kappa value of 0.7 or 70%• For the All Testers, the result is acceptable for accuracy of assessment against standard as the Kappa value is greater than 0.7 or 70%
• Overall Result for MSA is PASS since Kappa value is greater than 0.7 or 70%
Result of Attribute Agreement Analysis
[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence
( 12 )
Findings & Discussion – Analyze
ProjectName
Req.Error
Design Error
CUTerror KLOC
TotalTest
Cases
TestPlanError
TestCase Error
Automation%
TestEffort
Test Execution
ProductivityFunctional
Defects
PROJECT A 5 22 12 28.8 224 0 34 0 6.38 45.80 19
PROJECT B 0 0 1 6.8 17 0 6 0 9.36 17.00 1
PROJECT C 9 10 14 5.4 24 4 6 0 29.16 5.83 4
PROJECT D 7 12 2 1.1 25 4 9 0 13.17 7.00 0
PROJECT E 11 29 3 1.2 28 4 12 0 14.26 3.40 3
PROJECT F 0 2 7 6.8 66 1 7 0 32.64 31.00 16
PROJECT G 3 25 11 4.0 149 5 0 0 7.15 74.50 3
PROJECT H 4 9 2 0.2 24 4 0 0 18.78 7.67 0
PROJECT I 7 0 1 1.8 16 1 3 0 9.29 2.68 1
PROJECT J 1 7 2 2.1 20 1 4 0 6.73 1.95 0
PROJECT K 17 0 3 1.4 13 1 4 0 8.44 6.50 1
PROJECT L 3 0 0 1.3 20 1 7 0 14.18 9.75 1
PROJECT M 2 3 16 2.5 7 1 6 0 8.44 1.75 0
Data for Regression Analysis
[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence
( 13 )
Findings & Discussion – Analyze (cont.)
• P-value for selected individual predictors shows significant relationship with the Y (defects) which has value of less than 0.05
• R-sq and R-sq shows strong value which is more than 90%
Best Multiple Regression Result
[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence
( 14 )
Findings & Discussion – Design• Revise the predictors to generate prediction model that is logical and makes sense from the viewpoint of software practitioner• Filter the metrics to contain only valid data in order to generate the model• Reduced the model to have only coefficients that have logical correlation to the response (Defect)• Revisit the model to identify suitable response of the regression equation: Functional Defects or All Defects
PROJECT REQ .ERROR
DESIGNERROR
CUT ERROR
KLOC REQ. PAGE
DESIGN PAGE
TOTAL TEST
CASES
TEST CASES ERROR
TOTAL EFFORT
TEST DESIGN EFFORT
FUNCTIONAL DEFECTS
ALL DEFECTS
PROJECT A 5 22 12 28.8 81 121 224 34 16.79 15.20 19 19PROJECT B 0 0 1 6.8 171 14 17 6 45.69 40.91 1 1PROJECT C 9 10 14 5.4 23 42 24 6 13.44 13.44 4 4PROJECT D 7 12 2 1.1 23 42 25 9 4.90 4.90 0 0PROJECT E 11 29 3 1.2 23 54 28 12 4.72 4.59 3 3PROJECT F 0 2 7 6.8 20 70 66 7 32.69 16.00 16 27PROJECT G 3 25 11 4.0 38 131 149 0 64.00 53.50 3 3PROJECT H 4 9 2 0.2 26 26 24 0 5.63 5.63 0 0PROJECT I 17 0 3 1.4 15 28 13 4 9.13 7.88 1 1PROJECT J 61 34 24 36 57 156 306 16 89.42 76.16 25 28PROJECT K 32 16 19 12.3 162 384 142 0 7.00 7.00 12 12PROJECT L 0 2 3 3.8 35 33 40 3 8.86 8.86 6 6PROJECT M 15 18 10 26.1 88 211 151 22 30.99 28.61 39 57PROJECT N 0 4 0 24.2 102 11 157 0 41.13 28.13 20 33
Revised Data for Regression Analysis
[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence
( 15 )
Findings & Discussion – Design (cont.)
YFunctional Defects
XsRequirement Error,
CUT Error, KLOC, Requirement Pages,
Design Pages, Targeted Total Test Cases, Total Effort
Days
YAll Defects
XsRequirement Error,
CUT Error, KLOC, Requirement Pages,
Design Pages, Targeted Total Test Cases, Total Effort
Days
YFunctional Defects
XsRequirement Error,
CUT Error, KLOC, Requirement Pages,
Design Pages, Targeted Total Test Cases, Total Effort
Days in Test Design
YAll Defects
XsRequirement Error,
CUT Error, KLOC, Requirement Pages,
Design Pages, Targeted Total Test Cases, Total Effort
Days in Test Design
[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence
( 16 )
Findings & Discussion – Verify• Verification was done by applying the selected model on new/fresh project that have yet to go for System Test• The actual defects found after System Test completed are compared against the predicted defects• Verify that actual defects fall between 95% prediction interval of the model
Y Effort Predictors
Project Predicted Defects
Actual Defects 95% CI(min, max)
95% PI(min, max)
Functional Defect
All Tester Effort Prior to System Test
Agrimall Beta Release 182 187 (155, 210) (155, 210)
Grid Workflow 6 1 (0, 12) (0, 14)
TTS-English 1 1 (0, 3) (0, 6)
All DefectAll Tester
Effort Prior to System Test
Agrimall Beta Release 298 230 (242, 355) (241, 356)
Grid Workflow 9 9 (0, 21) (0, 24)
TTS-English 2 1 (0, 6) (0, 12)
Functional Defect
All Tester Effort in Test
Design Prior to System Test
Agrimall Beta Release 183 187 (202, 390) (201, 392)
Grid Workflow 8 1 (0, 17) (0, 19)
TTS-English 2 1 (0, 5) (0, 9)
All DefectAll Tester
Effort in Test Design Prior to
System Test
Agrimall Beta Release 296 230 (142, 224) (142, 225)
Grid Workflow 11 9 (0, 32) (0, 37)
TTS-English 3 1 (0, 10) (0, 19)
Revised Data for Regression Analysis
[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence
( 17 )
Findings & Discussion – Verify (cont.)
No Req. Error CUT Error KLOC Req. Page Des. Page Total TC Effort Days
1. 12 6 84.03 75 293 102 69.75
2. 49 15 8.69 64 38 65 91.9
3. 0 1 0.4 10 10 18 1.13
No Predicted Functional Defects Actual Functional Defects 95% CI (min, max) 95% PI (min, max)
1. 182 187 (155, 209) (154, 209)
2. 6 1 (0, 12) (0, 14)
3. 1 1 (0, 3) (0, 6)
Verify against new project:
Prediction result *:
* The actual functional defects were raised after both functional and ad-hoc testing completed
PROPOSED MODELFunctional Defects (Y) = 4.00 - 0.204 Req. Error - 0.631 CUT Error + 1.90 KLOC - 0.140 Req. Page + 0.125 Design Page - 0.169 Total Test Cases + 0.221 Effort Days
[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence
( 18 )
Findings & Discussion – Verify (cont.)
[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence
( 19 )
Conclusion & Recommendation
CONCLUSION•
Defect prediction model for testing phase can be constructed using identified factors via multiple regression
• Significant factors contributed to defects found in testing phase have been discovered
• Six Sigma can be used to develop test defect prediction model
• Test defect prediction model does contribute to zero-known post release defects and test process improvement
RECOMMENDATION• T
o consider other factors that contribute in predicting defects
• To improve model by being able to predict defect based on different severity
• To predict defects to be found against time
• To predict defects based on different type of software
THANK YOU
[email protected] © 2010 MIMOS Berhad. All Rights Reserved. Testing Excellence
Experience our Testing Excellence in
“End-to-End Testing Strategies” Workshop
See you again on 17th June 2010, 2.00 pm, Room 301