Upload
melissa-blackburn
View
214
Download
1
Tags:
Embed Size (px)
Citation preview
Using Metrics to Reduce Cost of Re-work
Dwight LamppertSenior Test Manager
Franklin Templeton
2
Opening Question
“What is the job of testing?”
3
Potential Response
“The job of testing is to find defects.”
4
Myth Exposed
Myth: The Job of Testing Is to Find Defects1
1The Seven Principles of Lean Software Development by Mary Poppendieck
5
“Quality is everyone’s responsibility”
Myth: The Job of Testing Is to Find Defects
“The job of tests, and the people that develop and run tests, is to prevent defects, not find them. A quality assurance organization should champion processes that build quality into the code from the start rather than test quality in later.”1
1The Seven Principles of Lean Software Development by Mary Poppendieck
6
At a recent software seminar
“The common attributes of successful software projects are
Heroes Chaos Re-work Untracked OT”
7
Today’s Discussion
Case Study – 4 projects2 DefinitionsTesting Metrics (what to collect &
analyze)Metrics from first 2 projects Process Improvement StepsSuccess Story – Projects 3 & 4Some trends & conclusions
8
Project Background
Legacy mature system (mainframe) Industry regulatory enhancements Time-boxed: 3 to 4 months Waterfall SDLC
First 2 Projects Project 1 – Dec09 – 1X Project 2 – June10 – 2.3X
9
Definition: Re-work
Re-work – AD time to fix and testing time to report and re-test System Testing and Production defects
10
Definition: Static Testing
Static Testing - Testing of a component or system at specification or implementation level without execution of that software, e.g., reviews or static code analysis.
11
Static vs. Dynamic Testing
From System Testing with an Attitude, p.14, by Nathan Petschenik
12
Testing Defect Analysis
When was the defect introduced (SDLC phase)
When was the defect discovered (testing phase)
13
Points of Origin for Defects2
1. Requirements2. Design3. Code4. User documentsUser documents5. Bad fixes (new defects due to repairs of
older defects)
2Software Engineering Best Practices: Lessons from Successful Projects in the Top Companies by Capers Jones
14
Defect Discovery Phases
1. Static Testing Document walk-throughs & reviews Pre-testing Test Plan review All before Build is complete
2. System Testing
3. Production
15
Tracking the Cost of Re-work
1. Defect Fix Time
2. Re-test Time
COR (Cost of Re-work) – calculated by multiplying time x rate – cost per defect
16
Testing Rework Workflow
Re-work time during Test Execution is tracked as 2 Components
1. Time to Fix a Defect
2. Time to Re-test Defect Fixes
17
Project 1 - Defect Fix Cost
Defect Fix Cost
$0 $21 $49
$406
$1,735
$0
$500
$1,000
$1,500
$2,000
Requirements Design Build System Test Transition
SDLC Phase of Defect Discovery
18
Issues from first 2 projects
1. High % of project = Cost of Re-work
19
COR = 23 to 29% Project
Cost of Rework - % of Total Project
29%
23%
0%5%
10%15%20%25%30%35%
1 2 3 4
Project
% o
f R
ewo
rk
COR - % of Project
20
Issues from first 2 projects
1. High % of project = Cost of Re-work
2. High % of Requirements & Design (R&D) defects found later
21
20-30% R&D Defects found later
% R & D Defects found later
29%
21%
0%
5%
10%
15%
20%
25%
30%
35%
1 2 3 4
Project
%
% R & D Defectsfound later
22
Issues from first 2 projects
1. High % of project = Cost of Re-work
2. High % of Requirements & Design (R&D) defects found later
3. DDP (Defect Discovery %) – slightly better than average
23
Defect Discovery Percentage (DDP)
DDP = % of Total Defects discovered before implementation
Provides a metric to evaluate the overall effectiveness of testing.
90% - Good95% - Very Good
24
DDP = average +
Defect Detection %
89.288.2
80
85
90
95
100
1 2 3 4
Project
% DDP%
25
Process Improvement Steps
1. Developed framework in Quality Center to track COR
2. Presentations to business partners & executives
3. Lively discussion during presentations4. AD appointed a new AD Lead5. AD improved their internal code review6. More emphasis on and participation
during reviews, walk-throughs – Static Testing
26
Process Improvement Steps
7. More collaboration between AD and testing teams
8. Increased emphasis on unit testing
9. Risk-based testing approach
10. Improved regression work
11. Staff increased in domain knowledge
27
Projects 3 & 4 – Process Improved
1. Project 1 – Dec09 – 1x
2. Project 2 – June10 – 2.3x
3. Project 3 – May11 – 3.4x
4. Project 4 – June11 – 2.3x
28
Static Testing emphasized
Defects Found During Static Testing
10
92
339
0
50
100
1 2 3 4
Project
# of
Def
ects
Static Testing Defects
29
System Testing Defects
Defects Found During System Testing
12
23
3
8
05
1015
2025
1 2 3 4
Project
# of
Def
ects
System TestingDefects
30
% R&D Defects found later
% R & D Defects found later
7%
29%
21%
2%
0%
5%
10%
15%
20%
25%
30%
35%
1 2 3 4
Project
%
% R & D Defectsfound later
31
DDP (Defect Discovery %)
Defect Detection %
89.2%88.2%
98.1%
95.5%
80%
85%
90%
95%
100%
1 2 3 4
Project
% DDP%
32
COR as % Project Cost
Cost of Rework - % of Total Project
12%
29%
23%
6%0%5%
10%15%20%25%30%35%
1 2 3 4
Project
% o
f R
ewo
rk
COR - % of Project
33
Defect Discovery Rate
DDR = % of Total Defects discovered during an SDLC or Test Phase
Static Testing System Testing TransitionGoal - Trying to find out if we are
finding defects earlier.
34
DDR Trends
17.6
70.5
11.7
27
62.1
10.8
89.3
7.8
1.9
88.6
6.84.5
0
10
20
30
40
50
60
70
80
90
100
Static System Production
----------------C
od
ing
-----------------
As DDR for Static Testing
increases, DDR for System Testing and Production
Defects decreases
35
When is Unit Testing complete?
“A necessary criterion for completion of developer testing is that the developers believe that the software is ready to be delivered to the users on the day it is delivered to the system-test team.”
From System Testing with an Attitude, p. 33, by Nathan Petschenik
36
Today’s Discussion
Defect metrics – powerful storyStatic Testing worksIt is a collaborative effortThe positive effect ripples into Unit
Testing, System Testing, UAT, and implementation
No shortcuts here – influence your project to do it right
37
Some Final Thoughts
Training needed for our staff (testing, AD, business, PM community) - how to be more effective in document reviews and walk-throughs.
Perspectives Tester – static testing > a paradise (a
new maturity level)
38
Questions & Answers
???