Upload
elsa-gillion
View
213
Download
0
Tags:
Embed Size (px)
Citation preview
A SIMPLE VIEW OF HOW TO REDUCE TOTAL COST OF QUALITY
PRESENTED BY CATHERINE OEHL
COQAA Monthly Meeting
December 5, 2012
AGENDA
12/5/2012 2
• What is Quality and Why is it important• What is Total Cost of Quality (TCoQ)• What is the purpose and value of
measuring TCoQ• How to measure and reduce TCoQ• Hypothetical Scenario applying TCoQ• Q&A
QUALITY DEFINED• Definition: A measure of excellence or a state of being free from
defects, deficiencies and significant variations. It is brought about by strict and consistent commitment to certain standards that achieve uniformity of a product in order to satisfy specific customer or user requirements.
• ISO 8402-1986 standard defines quality as "the totality of features and characteristics of a product or service that bears its ability to satisfy stated or implied needs."
• If an automobile company finds a defect in one of their cars and makes a product recall, customer reliability and therefore production will decrease because trust will be lost in the car's quality.
12/5/2012 3
Source: http://www.businessdictionary.com/definition/quality.html
CONSEQUENCES OF POOR QUALITY
12/5/2012 4
COMMON STATE OF QUALITYo Current projects spend 40-50% of the effort on avoidable reworko About 80% of avoidable rework comes from 20% of the defectso About 80% of defects comes from 20% of the modules and about half
of the modules are defects freeo 90% of the downtime comes from at most 10% of the defectso Peer Reviews catch 60% of the defectso Perspective-based reviews catch 35% more defects than non directed
reviewso Disciplined personal practices can reduce defect introduction rates by
up to 75%o While several organizations measure and track application total cost of
ownership (TCO), very few measure the total cost of quality
12/5/2012 5
WHAT IS TCOQ
• Total Cost of Quality (TCoQ) consists of the costs associated with preventing, assessing, finding and correcting defective work.
• Note that the TCoQ analysis looks only at our (producers) costs and not the customer’s cost due to poor quality.
12/5/2012 6
Cost of Quality
Cost of Good Quality
Prevention Costs Inspection Costs
Cost of Bad Quality
Internal Failure Costs
External Failure Costs
QUALITY COSTS OF TCOQPrevention cost + Inspection cost + Internal Failure cost + External Failure cost
Prevention costs arise from efforts to keep errors or defects from occurring at all
Inspection costs arise from detecting errors/defects via inspections, tests, audits
Internal failure costs arise from defects caught internally and dealt with by discarding or repairing the defective items
External failure costs arise from defects that actually reach customers. Often other costs are incurred in addition to correcting the failure.
12/5/2012 7
PURPOSE AND VALUE OF TCOQAddresses and measures the following questions:• What is the state of quality of your applications?• How does quality impact your business?• How is quality measured?• When are errors/defects introduced? Why?• When are errors/defects corrected? What are the costs?• What are the processes that impact quality?• How do you quantify the effectiveness of each process?• How much do failures cost?• What is the root cause of the failure?
12/5/2012 8
WHEN AND WHERE ARE DEFECTS INTRODUCED
Source: Writing Testable Requirements, Dick Bender
12/5/2012 9
Re-quire-ments56%
Design27%
Coding7%Unknown
10%
Source: NIST Software Quality Study
WHEN AND WHERE ARE DEFECTS DETECTED
12/5/2012 10
Source: NIST Software Quality Study
COST TO CORRECT DEFECTS
12/5/2012 11
Source: B. Boehm and V. Basili, “Software Defect Reduction Top 10 List”, IEEE Computer
Cost to remove defects is 100 times more
THE QUALITY LEVER
12/5/2012 12
Source: http://en.wikipedia.org/wiki/Quality_costs
Prevention / Reviews
Appraisal /Inspection
Failures / Reaction
Cost Savings
Greatest Leverage Good
Leverage LeastLeverage
PREVENTIVE COSTSGOAL: Prevent defects before they happen
(Work done right the first time)
Recommended Activities:– Formal Requirements Gathering Processes– Requirements Analysis and Management– Early Prototyping– Usability Analysis– Static Testing– Peer Reviews– Release and Configuration Management Processes– Standards for Test Environments– Quality Training– Accurate Internal Communication
12/5/2012 13
INSPECTION COSTSGOAL: Detect the Errors (Find them early)
Recommended Activities:– Code inspections– Design reviews– Glass box and black box testing– Test Automation– Usability Testing– Pre-release / User Acceptance Testing– Structured Personal reviews focused on
verification and validation
12/05/2012 14
INTERNAL FAILURE COSTSGOAL: Report and fix defects before production
release (Fix and Re-Test)
Incurred Activities:• Diagnose Defects• Defect Fixes• Defect Management• Re-Test and Regression Testing• Rework code and recompile• Wait states
12/5/2012 15
EXTERNAL FAILURE COSTSGOAL: Reduce and remediate impact to customers
(Apologize and recover)
Incurred Activities:• Technical Support Calls• Recalls• Customer Service• Scrap• Overtime• Regulatory / Legal Costs
12/5/2012 16
HYPOTHETICAL SCENARIO
• Suppose we have a software product that has one production release per quarter.
• On average each release contains 1000 must-fix defects that will be identified and repaired.
• We will assume developers catch 25% of defects during unit and integration testing at a cost of $10/defect
12/5/2012 17
1 - NO INSPECTION TESTING
12/5/2012 18
Inspection Testing CostsNon Formal Testing
Staff $0 Infrastructure $0 Tools $0 Total Investment $0
Development Must-Fix Defects Found 250Fix Cost (Internal Failure - $10/defect) $2,500
Testing (Pre-Release) Must-Fix Defects Found 0Fix Cost (Internal Failure - $100/defect) $0
Customer Support (Production) Must-Fix Defects Reported 750Fix Cost (External Failure - $1000/defect) $750,000
Cost of Quality Conformance (Inspection Costs) $0 NonConformance (Internal Failures + External Failures) $752,500
TCoQ $752,500 ROI N/A
2 - MANUAL INSPECTION TESTING
12/5/2012 19
Inspection Testing CostsNon Formal Testing
Manual Testing
Staff $0 $60,000 Infrastructure $0 $10,000 Tools $0 $0 Total Investment $0 $70,000 Development Must-Fix Defects Found 250 250Fix Cost (Internal Failure - $10/defect) $2,500 $2,500 Testing (Pre-Release) Must-Fix Defects Found 0 350Fix Cost (Internal Failure - $100/defect) $0 $35,000 Customer Support (Production) Must-Fix Defects Reported 750 400Fix Cost (External Failure - $1000/defect) $750,000 $400,000 Cost of Quality Conformance (Inspection Costs) $0 $70,000 NonConformance (Internal Failures + External Failures) $752,500 $437,500
TCoQ $752,500 $507,500 ROI N/A 350%
3 - MANUAL WITH AUTOMATED INSPECTION TESTING
12/5/2012 20
Inspection Testing CostsNon Formal Testing
Manual Testing
Automated Testing
Staff $0 $60,000 $60,000 Infrastructure $0 $10,000 $10,000 Tools $0 $0 $12,500 Total Investment $0 $70,000 $82,500 Development Must-Fix Defects Found 250 250 250Fix Cost (Internal Failure - $10/defect) $2,500 $2,500 $2,500 Testing (Pre-Release) Must-Fix Defects Found 0 350 500Fix Cost (Internal Failure - $100/defect) $0 $35,000 $50,000 Customer Support (Production) Must-Fix Defects Reported 750 400 250Fix Cost (External Failure - $1000/defect) $750,000 $400,000 $250,000 Cost of Quality Conformance (Inspection Costs) $0 $70,000 $82,500 NonConformance (Internal Failures + External Failures) $752,500 $437,500 $302,500
TCoQ $752,500 $507,500 $385,000 ROI N/A 350% 445%
What if we added some simple Prevention activities like Static Reviews, Peer Reviews, Change Management ?
4 - ADDING PREVENTION ACTIVITIES
12/5/2012 21
Prevention Costs Resource CostsStatic Reviews (10 people every 2 weeks reviewing 5 sets of documents x $75/hr) $97,500Peer Reviews (2 developers every 2 weeks reviewing 10 modules x $75) $39,000Change Management (10% Configuration Mgr & 10% Requirements Lead & 10% Test Lead) $46,800
Inspection Testing Costs Non Formal Testing Manual Testing Automated Testing Staff $0 $60,000 $60,000 Infrastructure $0 $10,000 $10,000 Tools $0 $0 $12,500 Total Investment (Prevention & Inspection costs) $0 $70,000 $82,500 $265,800 Development Must-Fix Defects Found 250 250 250 500Fix Cost (Internal Failure - $10/defect) $2,500 $2,500 $2,500 $5,000 Testing (Pre-Release) Must-Fix Defects Found 0 350 500 350Fix Cost (Internal Failure - $100/defect) $0 $35,000 $50,000 $35,000 Customer Support (Production) Must-Fix Defects Reported 750 400 250 50Fix Cost (External Failure - $1000/defect) $750,000 $400,000 $250,000 $50,000 Cost of Quality Conformance (Inspection Costs) $0 $70,000 $82,500 $265,800NonConformance (Internal Failures + External Failures) $752,500 $437,500 $302,500 $90,000
TCoQ $752,500 $507,500 $385,000 $355,800 ROI N/A 350% 445% 57%
SUMMARY• Quality creates value and reduces risk exposure• The “cost of quality” isn’t the price of creating a quality product or
service. It’s the cost of NOT creating a quality product or service.• Errors / defects are often introduced early and typically found later• Total Cost of Quality (TCoQ) consists of the costs associated with
preventing, assessing, finding and correcting defective work. • The central theme of quality improvement is that larger investments in
prevention drive even larger savings in quality-related failures and Inspection efforts.
• Any Quality improvement effort should use TCoQ to quantify the effectiveness of the process.
12/5/2012 22
CONTACT INFO AND Q&A
Catherine Oehl
614-440-6477
LinkedIn Profile:
http://lnkd.in/u8kcyy
12/5/2012 23
RESOURCES & REFERENCES Software Defect Reduction Top 10 List by Barry Boehm and Victor Basili Using Cost Benefit Analysis to Develop Software Process Improvement
Strategies, Data Analysis Center of Software Traditional P-A-F method by Juran (1951) and Feigenbaum (1956) –
Prevention Appraisal and Failure costs Phil Crosby’s Model from the book Quality is Free (1979) – conformance to
requirements – cost of conformance (done Right) and non-conformance (done Wrong)
Process Cost Model by Ross (1977) http://asq.org/learn-about-quality/cost-of-quality/overview/overview.html http://www.authorstream.com/Presentation/Hassanasif-493959-cost-of-quali
ty http://www.bexcellence.org/Cost-Of-Quality.html http://www.isixsigma.com/implementation/financial-analysis/cost-quality-not-
only-failure-costs/ http://www.projectconnections.com/articles/051005-koch.html
12/5/2012 24
SUGGESTED TCOQ BENCHMARK MEASUREMENTS
• Process (Productivity & Prevention)– Effectiveness of Test Processes
• Release Mgt, Configuration Mgt
– Efficiency of Test Processes• Defect Mgt, Test Environment Mgt, Test Data Mgt
– Consistency of Test Activities• Test Planning, Test Estimation, Test Execution
• Schedule– Testing Planning
• Duration of lifecycle phases• Duration of Test Phase/Stage
– Test Estimating/Projections• Planning Estimate• Component Based Estimates
12/5/2012 25
COMMON MEASUREMENTS OF FAILURES
• Failures– Volume of Defects
• Open, Closed, New, Test Phase/Stage Contained, Defect Aging (# of days open)
– Severity of Defects• Criticals, Highs, Mediums, Lows, Problem Areas,
Low to all reported ratio
12/5/2012 26
SIMPLE ACTUAL QUALITY COSTS
• Build Test Mgt Costs (P+I+IF)– Manual Test Hour Costs– Automated Test Hour Costs– Test Hour Costs per Test Phase/Stage– % of Total Build Costs
• Breakfix/Maintenance Costs (EF)
12/5/2012 27
VIEW OF TCOQ IN TERMS OF QA
12/5/2012 28