Spend Wisely,
Test WellJohn Fodeh,
HP, Denmark
Europe’s Premier Software Testing Event
Stockholmsmässan, Sweden
WWW.EUROSTARCONFERENCES.COM
“Testing For Real, Testing For Now”
© 2009 Hewlett-Packard Development Company, L.P.The information contained herein is subject to change without notice
SPEND WISELY, TEST WELLMaking a Financial Case for Testing
HP - Global Testing Practice
John Fodeh
Introductions
About You (and Me)
Briefly introduce yourself by telling us:
• Your name
• Your testing role
• How long you’ve been in the testing arena
• One thing you’d like to get out of this Workshop
About This Session
After you’ve completed this workshop, you should be able to:
• Address and even turn around the prevailing view of testing among business stakeholders
• Through the effective use of industry data:
− Establish and/or reinforce your credibility
− Substantiate your testing recommendations and strategies
• Demonstrate to business stakeholders – in terms they can understand and accept – the relative cost of testing –and not testing – for a specific project
• Credits: Ian Howles & Susan Herrick!
Administrivia
• The duration of this workshop is 1.5 hours
• There will be no breaks
• Ask questions whenever they occur to you
It’s All About Communication
• Remember, testing is a service organization.
• Our mission is to provide information about
quality and risk to decision makers within our
organization.
Testing
The Remainder of the Organization
Forgetting the Organizational Context
From The Nine Forgettings by Lee Copeland
• Testing does not create quality. All we do is look for it.
Forgetting the Organizational Context
• Testing should not be the Ship / Don’t Ship decision maker. That is not our stewardship.
The Business Spectrum
• Mass-market packaged software
• Market-dominating software
• Niche software
• High consumer tolerance
• Custom software
• Market-entry software
• Competitive software
• Low consumer tolerance
• Regulatory compliance
• Critical safety requirements
Defect Tolerance
High Low
Perfection is the enemy of the good
- Voltaire
Cost of Quality
Quality Perspective
C Quality = C Conformance + C Non-Conformance
WHERE
C Quality = Cost of Quality
C Conformance = Cost of Conformance
• Cost of prevention
• Cost of detection
• Cost of internal failure
C Non-Conformance = Cost of Non-Conformance
• Cost of external failure
• Other related costs
Cost of Quality
Business Perspective
PV – C Quality = NV
WHERE
PV = Potential Value, e.g., ALL sources of revenue associated with a software product, annualized cost reduction that results from productivity improvements
NV = Net Value, e.g., actual revenue for the software product, actual annualized cost reduction
AND
PV = PM or ROI (% + or -)
NV
WHERE
PM = Profit Margin
ROI = Return on Investment
It’s All Greek to Me!
The language of Testing focuses on:
• Quality
• Defects
• Test coverage
• Risk
The language of Business focuses on:
• Scope
• Time
• Money
• Risk
• (and, oh, yes, Quality)
Bridging the Communication Gap
• The decision-makers will tell you they care about quality and ―customer satisfaction‖ but not if they believe that achieving either will adversely affect their bottom line
• Talking about defect discovery alone isn’t enough
• Testing has to be able to:
− Demonstrate the financial consequences of testing involvement − or lack thereof − in a project
− Substantiate the financial consequences with reliable data
Show Me the Money
Testing requires financial investment in:
− Resources, tools, equipment
− Time to develop, test, manage
− ―Escaped‖ defects
− Rework and retesting
Business stakeholders legitimately expect a healthy return on that investment
Defining the Communication Challenge
Lets quickly discuss some views of testing that:
• You hear from decision-makers in your organization and/or on your projects
• You would like to change
Prevailing View vs. Desired View
• Testing is a back-end project phase
• Testing is easy and anyone cando it
• Significant rework is inevitable
• Testing provides advice that can be selectively ignored
• Testing creates bottlenecks and increases the risk of schedule overruns
• Testing costs too much
• Testing is an integral aspect of all phases in the product life cycle
• Effective and efficient testing requires skilled testing professionals
• Focusing on early defect discovery and prevention can significantly reduce rework
• Testing provides information that quantitatively specifies outstanding risk, ―application readiness‖ and potential cost — ignore it at your peril!
• Testing early and testing often can actually reduce the risk of schedule overruns and accelerate time-to-market
• Not testing costs way too much — in rework (pre- and post-release), failures, and production support
Exercise:
Turning a Problem into a Solution
• The following exercise is derived from a tutorial developed by Steve Bender of The Quality Connection (www.qualityconn.com):
Effective Methods to Develop Creative Approaches to Today’s Challenges
• The exercise requires you to:
− Define a problem
− Define a solution that is:
• Stated in positive terms
• Under your control
• Integrated
• Testable
Exercise
Step 1 - Defining the Problem
Take 5 minutes to answer these questions.
1. What’s the problem?
2. How long has this problem been a problem?
3. Why does this problem exist?
4. How are you and/or your team limited by this problem?
5. What is the worst thing this problem has caused?
Exercise
Step 2 - Defining the Solution
Take 5 minutes to answer these questions.
1. What do you want? Be as specific as possible.
2. How will you know when you get it?
3. Where, when and with whom do you want it?
4. When you get it, what else will improve?
5. What resources do you already have that will help you get it?
6. What is the first step to take to get it?
Defining the Problem
An Example
1. What’s the problem?Perception that testing costs more than it’s worth
2. How long has this problem been a problem?For as long as I can remember (15+ years)
3. Why does this problem exist?No real proof to the contraryTesting world does not speak the language of business
4. How are you and/or your team limited by this problem?Inadequate allocation of time, money, resourcesMarginal influence over allocation decisionsLack of senior management support
5. What is the worst thing this problem has caused?5-10% of project effort allocated to testingUnnecessary schedule and cost over-runsDefective software in production
Defining the Solution
An Example
1. What do you want? Be as specific as possible. ―Real‖, tangible proof that the decision-makers ―get it‖
2. How will you know when you get it?Commitment through actionProjects - Upfront testing involvement in planning, scheduling, impact analysisSenior management – Testing Policy and Strategy
3. Where, when and with whom do you want it?Senior management and project ―purse string holders‖
4. When you get it, what else will improve?Testing involvement and influenceProduct quality, cost, speed to market
5. What resources do you already have that will help you get it?Industry dataAccess to audience
6. What is the first step to take to get it?Pull together the dataFilter and home in on compelling ―proof‖
Communication Principle
The meaning and merit of the communication is in the response it elicits
The Proof is in the Data
I’m from Missouri … show me
In God we trust – all others bring data
- W. Edward Deming
A Common Modus Operandi
If you don’t know where you are going, any road will get you there
- Lewis Carroll
An Alternative Perspective
• If you don’t know where you’re going, how willyou know when you’ve arrived?
OR
• If you don’t know what testing – or not testing – is really going to cost you, how can you know if you’ve spent wisely?
Show Me the Value
Two axioms of management are:
− What gets measured gets done
− What gets valued gets funded
So, it's important to remember that testing delivers valuethat we can ― and should ― measure
- Rex Black
Addressing the Challenge
• Focus on the financial impact of defect resolution through testing activities throughout the product life cycle
• Use the following industry data to substantiate claims about the financial impact of:
− Defect Insertion Rate
− Amplification Factor
− Defect Removal Effectiveness
− Time to Resolve a Defect
− Cost of Defect Resolution
− Rework Rate
• Address the assumption that ―automation solves the problem‖
• Demonstrate the cost of schedule compression and inadequate testing staff (numbers and experience)
• Quantify the cost and cost avoidance associated with testing – or not testing
Some Definitions
• Defect insertion rate defines the defects introduced in a project phase as a % of the defects discovered throughout the product lifecycle
• Amplification factor is based on the premise that every defect introduced in a project phase that remains undiscovered migrates to the next project phase AND introduces additional defects
• Defect removal effectiveness is a quantitative measure of the effectiveness with which each discrete testing activity finds defects and supports their resolution
Some More Definitions
• Time to resolve a defect is based on the time to discover and remove one (1) defect by project phase in the product life cycle
• Cost of defect resolution is based on the cost of discovering and removing one (1) defect by project phase in the product lifecycle
• Rework rate is the effort associated with fixing and retesting defects introduced in a project phase as a % of overall effort associated with fixing the defects discovered throughout the product lifecycle
Defect Insertion Rate
Requirements56%
Design27%
Other10%Code
7%
“56% of defects are introduced in the requirements phase of a project.“
From James Martin
Amplification Factor
Requirements
1 defect 1.5 defects in Design
Design
1 defect 1.25 defects in Code
From What IT Managers Should Know About Testing ROI by Rex Black
Defect Removal Effectiveness
Each discrete testing activity can detect and support resolution of 30% of latent defects:
Defects that escaped from the previous testing activity
AND
Defects that were introduced during the project phase associated with the current testing activity
A latent defect is a defect that is present and capable of becoming visible, obvious, active, or symptomatic
From Capers Jones
50
100
200
500
Requirements Design Code System Testing
Acceptance
Testing
Operation &Maintenance
1
Cost of Defect ResolutionIn
cre
menta
l Costs
Product Life Cycle Stage
From Boehm & Baziuk, Standish Chaos Report
On large, complex projects, 50-70% of project effort is devoted to rework − an avoidable cost
Rework is the root cause of project bottlenecks, schedule delays and rising costs
Time to Resolve a Defect
Rework Rate
Requirements82%
Design13%
Other4%Code
1%
“82% of effort in fixing defects is caused by inadequate requirements.“
From James Martin
Time is Money
Design Review Code Inspection Testing
8.44 hours 1.38 hours 0.17 hours
Collofello and Woodfield (1989)
For every hour spent on…
Putting it all Together
Putting the Data to Work
• Each of these data sources provides thought-provoking insight, but not enough to make a compelling, quantitative case for testing ― or not testing
• Pulling the data together can help your business stakeholders understand:
− Whether testing will really cost more than not testing
− How much and what types of testing will reduce cost and maximize value through profit and/or return on investment (ROI)
Time, Cost and Value Calculator
• Considers both pre-release and post-release activities
• Classifies testing activities as:
− Static Testing
Testing of a component or system at the specification or implementation level without execution of that software, e.g. reviews or static code analysis
− Dynamic Testing
Testing that involves the execution of the software of a component or system.
From Standard Glossary of terms Used In Software Testing, v 2.0 - ISTQB
Time, Cost and Value Calculator
Static Testing
Testing Category
Potential # of Discrete Testing Activities
Typically Includes:
Requirements Verification
2 • Ambiguity and Risk Analysis
• Content Review
Design Verification
2 • High-Level Design Review
• Detailed Design / Prototype Review
Code Verification 2 • Code inspection (manual)
• Code complexity analysis (automated)
Time, Cost and Value Calculator
Dynamic Testing
Testing CategoryPotential # of Discrete Testing Activities
Typically Includes:
Component-Level Testing
2 • Unit Testing
• Component Integration Testing
System-Level Testing
5 • System Testing
• System Integration Testing
• Performance Testing
• Regression Testing
• Consolidated Integration Testing
Acceptance Testing 3 • User Acceptance Testing
• Operational Acceptance Testing
• Field Acceptance Testing
Time, Cost and Value Calculator
Inputs
Uses the following factors as inputs for the overall project:• Predicted number of latent defects
• Multiplier for the cost to resolve one (1) defect
• Projected value (e.g., revenue, annualized cost reduction from productivity improvement)
• Potential additional costs (e.g., contract penalties, training)
Uses the following factors as inputs for each testing category:
• Defect removal effectiveness based on # of discrete testing activities
• Experienced Team (yes, no, capable of process improvement)
• Number of personnel as % of required number of personnel (100%, 75%, 50%, 25%)
• Schedule compression (none, 25%, 50%, 75%)
• Test automation (used or not used)
• Amplification factor for defects introduced
Time, Cost and Value Calculator
Outputs
• Calculates the following outputs for each testing category:
− Number of escaped defects
− Escaped defects as % of predicted total
− Number of defects introduced
− Number of defects removed
− Time to resolve one (1) defect
− Time to resolve all defects
− Cost to resolve one (1) defect
− Actual cost of defect resolution
• Calculates the following outputs for the overall project:
− Time to resolve all defects
− Total defect resolution cost
− Total cost
− Net value (i.e., profit or loss)
− Profit margin / ROI (%)
Collaborative Exercise
Step 1 – Calculate Time, Cost and Value
Use the following information:• Custom application for a single customer
− New development
− Enough developers and acceptance testers
− Inexperienced developers and acceptance testers
• Based on a ―best guess‖, the predicted number of discovered defects is 500
• Multiplier for cost to resolve one (1) defect is 10 (VERY conservative)
Collaborative Exercise
Step 2 – Calculate Time, Cost and Value
Use the following information about the contract:
• Application price $4,000,000
• All defects to be resolved by end of 90-day warranty period
• Potential contract penalties from defects at release
− 10-150 defects 2.5% of application price
− 151-250 defects 5% of application price
− 251-500 defects 10% of application price
− 501-750 defects 25% of application price
− 751-1000+ defects 50% of application price
Collaborative Exercise
Step 3 – Calculate Time, Cost and ValueUse the following information about planned testing activity:
• Organization has ―decided‖ the following testing activities are sufficient:
− Component-Level Testing:Unit Testing (1 discrete testing activity)
− Acceptance Testing:User Acceptance Testing (1 discrete Testing Activity)
• Review the inserted Comments to determine the number of discrete testing activities you will perform in each category of Activities
• Organization is looking for a profit margin of at least 25%
Collaborative Exercise
Using the Time, Cost & Value Calculator
• We will now try and put it all together in the Time, Cost and Value Calculator
• Enter Projected Value
• Calculate the following results:
− Time (in effort hours) to resolve all defects
− Total Defect Resolution Cost
− Net Value
− Profit Margin (%)
• Let’s not forget to determine and enter the Additional Costs (based on penalty levels)
Collaborative Exercise
Step 1 – Initial Results
Time to Resolve All Defects 85,077 hours
Total Defect Resolution Cost $5,353,580
Additional Costs $1,000,000
Total Cost $6,353,580
Net Value -$2,353,380
Profit Margin / ROI -59%
Collaborative Exercise
Step 2 - Save Time and Make a Profit
Now, lets modify the strategy for this project to ―bring it into the black‖ – we are going to aim for:
− Significant time reduction
− A Profit Margin/ROI of at least 25%
• We will:
− Consider Static Testing activities, where the cost of defect resolution is lower and the impact of the amplification factor is higher
− Experiment with experience level, schedule compression, and test automation
Reduce the number of post-release defects enough to significantly reduce or even eliminate the penalty (don’t forget the defect ranges!)
Collaborative Exercise
A ―Good‖ Solution
• From the Format Menu, select Unhide T_C_V -Proposed
• Focuses on early defect detection and cross-functional collaboration by incorporating:
• Static Testing
− Review of requirements
− Review of Design
− Code Inspection
• Dynamic Testing
− System Testing
− Regression Testing
• Reduces time to resolve defects by >50%
• Achieves a profit margin of 36%
Refining the Data
• Industry data is sound, reliable and defensible
BUT
• YOUR data is equally, sound, reliable and defensible
… and also much more persuasive
A Final Message
If testing costs more than not testing, then don’t test
- Kent Beck
… but you’d better have the data to back up your decision
- John Fodeh
(Thanks to Susan Herrick &Ian Howles)
© 2009 Hewlett-Packard Development Company, L.P.The information contained herein is subject to change without notice
Contact:
John Fodeh