Upload
neil-thompson
View
753
Download
0
Embed Size (px)
Citation preview
www.qualtechconferences.com© Thompson information Systems Consulting Limited
1
ROI at the Bug Factory:Goldratt and Throughput
1st European Conference on Software Projects, Process & People01 - 03 December 2004 – Köln, GermanySoftware Projects Track session WP1
Neil ThompsonThompson information Systems Consulting Limited
www.TiSCL.com
www.qualtechconferences.com© Thompson information Systems Consulting Limited
2
1. Introduction• Primary objective of presentation – to explore some analogies between
processes for software development and implementation and those for manufacture (of physical things):– to see whether recent (and not-so-recent) revolutionary changes in
manufacturing have relevance for software– to relate such analogies to existing work in agile methods– to relate this thinking to the software development lifecycle whether agile
or not, and to verification, validation & testing (VV&T) – to help you in planning and managing projects & programmes
• Secondary objectives: – to use the concept of Return On Investment (ROI) as an introduction to
thinking about process effectiveness, efficiency and improvement– to use Goldratt’s Theory Of Constraints (TOC) as the basis of considering
manufacturing processes, and for its general thinking tools
www.qualtechconferences.com© Thompson information Systems Consulting Limited
3
What is ROI, why measure it?
• ROI: return / investment (usually expressed as a percentage):– “return” could be profit, (value – cost), (net benefit)…– “investment” may be expressed as net cost
• I’m a “crossover” from EuroSTAR (but wider background)• So I’ll start with the example of ROI for VV&T:
– to decide how many test environments to buy, and how life-like– to decide how many testers & QA people to hire– to estimate duration of testing– to justify budgets– to decide how to divide effort between testing & reviews/inspections– to focus training– to justify automation– to motivate staff?– to make managers look good?
www.qualtechconferences.com© Thompson information Systems Consulting Limited
4
Why ROI difficult for IT systems
• Although costs relatively easy to measure, not trivial• Benefits no longer about replacing manual systems; often are
enhancing market position, or opening new market, and those can be very difficult to forecast
• Misleading / unbalanced precision between costs & benefits• Misguided cost-cutting (eg through inadequate understanding of
benefits)• Excuses for not calculating ROI “properly”:
– “we’ve simply got to have it”– project / system too small to bother– it’s too difficult– wouldn’t be believed anyway
• Problem of “construct validity”: you may not really be measuring what you think you’re measuring Cem Kaner
www.qualtechconferences.com© Thompson information Systems Consulting Limited
5
ROI even more difficult for VV&T
• Partitioning difficulties:– how to distinguish as activities, when much is part of develop’t work– whether / how / why to distinguish by job title
• Biggest problem is quantifying benefits in terms of costs saved by preventing failures later:– Boehm’s exponential-type cost rise questioned for modern methods– isn’t as simple as “early=good”, because some bugs not (easily)
visible until later (though continuous integration helps)– how can we say fault A would have caused failure B at date X if it
had not been found by test level C?– even where live failure costs calculated, validity is questionable– how distinguish poor VV&T from “good VV&T where there are very
few bugs there to find”?
www.qualtechconferences.com© Thompson information Systems Consulting Limited
6
Also other complications, but let’s persist still
• Additional complications to using ROI for VV&T:– “Tyranny of numbers” two different books
– whether testing actually reduces risk or is merely (a misinterpretation of?) “Project Intelligence” Paul Gerrard
– is effectiveness / efficiency distinction feasible / useful?• But ROI thinking should still be helpful:
– still need to answer those questions, don’t we?• which VV&T activities to do, which not, which partially?• (coarse-tune) which people to use?• (fine-tune) how to maximise effectiveness & efficiency?
– so Agenda…
www.qualtechconferences.com© Thompson information Systems Consulting Limited
7
Agenda
• 2. Goldratt’s Theory Of Constraints• 3. The code factory and the bug factory• 4. Beyond the factory: the supply chain, and live systems• 5. The Systems Development LifeCycle as an
inventory network• 6. Conclusions & way forward
www.qualtechconferences.com© Thompson information Systems Consulting Limited
8
2. Goldratt’s Theory Of Constraints: what it is
• Now, “multifaceted management philosophy”, but originally emerged in early 1980s from new approach to materials scheduling in factories
• I already applied Goldratt’s thinking tools to process definition in software testing STAREast 2003
• Here looking wider, at whole SDLC, going back to Goldratt’s original arguments in manufacturing The Goal, The Race
• Evidence of TOC success in reducing inventory, lead times & cycle times; improving due-date performance & revenue
• Associated with Japanese improvements (eg Just In Time, Lean), but JIT came first, Goldratt was a response and a claimed improvement; Lean followed a few years later
www.qualtechconferences.com© Thompson information Systems Consulting Limited
9
Parts of TOC using here• Goal definition, eg make money:
– measures cash flow (survival), profit (absolute) & ROI (relative)• Throughput better for these measures than cost management• Adverse effect of inventory on competitive edge:
– it damages not only ROI & cash flow but also profit• Relationship of throughput to inventory & operating expense:
– throughput (T) = money in (should maximise)– inventory = money tied up (should minimise, but not damage T)– operating expense = money out (should minimise, but not damage T)
• Drum-Buffer-Rope as a technique to optimise inventory• Relationship to Synchronous Flow Manufacturing, Japanese
methods and other quality techniques
www.qualtechconferences.com© Thompson information Systems Consulting Limited
10
Applicability of TOC beyond manufacturing
• Military logistics• Marketing, sales & distribution• Project management• Measurements, human relationships, medicine etc• Using technology, eg assessing benefits of functionality• IT systems development:
– focussing of Goldratt’s Critical Chain on hi-tech projects Robert C Newbold
– methodology design Alistair Cockburn
– “Lean software development” Mary & Tom Poppendieck
– Agile management using Feature Driven Development David J Anderson
www.qualtechconferences.com© Thompson information Systems Consulting Limited
11
But software development isn’t like manufacturing?• Software isn’t like hardware• Intellect adds value less predictably than machines• The manufacturing part of software development is disk
duplication: “development” is really a design activity• People are more important than processes• Software development doesn’t repeat exactly, people are
always tinkering with the processes• Development involves discovery, production involves
reducing variation• But I say: does that make all the analogies worthless, or
do they just need interpreting? I suggest the latter…
www.qualtechconferences.com© Thompson information Systems Consulting Limited
12
3. The code factory and the bug factory
• No-waste factory
• Now here’s some waste: meeting/escalation (and loaded personal memories), or inventory?
Programming
Programming
a ab b’
c d
Documentedrequirements
Implicitrequirements
Meeting / escalation to agree
I I Acceptance tests
?
?
a b c Demonstrations &acceptance tests
Statedrequirements
www.qualtechconferences.com© Thompson information Systems Consulting Limited
13
Specs & unfinished software are inventory
• Specifications are generally not needed after go-live (I will come later to exceptions) so they are not end-product, they are work-in-progress (especially intermediate levels like functional & non-functional specs)
• Untested software, and even finished software not paid for, is also inventory
• Iterative lifecycles help if “adaptive” (product-based) rather than “transformational” (where specifications multiply!)
a b
ProgrammingProgramming
b’
a’
c
I
May includeredesign
Revised & new requirements
www.qualtechconferences.com© Thompson information Systems Consulting Limited
14
Full W-model bulges with inventory
Requirements Statement
Functional Spec.
Technical Design
Module Specs.
Code
Business objectives
V&V MS, + spec. UT
Unit test
Unit retest, fix & reg. test
Post-implement- ation review
Verify & Validate RS, + spec. AT
Acceptance test
Acc. retest, fix & reg. test
Sys. retest, fix & reg. test
Int. retest, fix & reg. test
System test
Integrat- ion test
Verify & Validate FS, + spec. AT
V&V TD, + spec. IT
Makeinto
Test against
Retest lowerlevelswhere
necessary
Static-check
Verify
Validate(incl. “QA”)
www.qualtechconferences.com© Thompson information Systems Consulting Limited
15
In a factory, small batches reduce inventory
Based on Goldratt, The race (North River Press 1986)
200Multi-batch(ie “iterative”)
(i) 1.3
(ii) 10.0
(iii) 1.0
(iv) 10.0
(v) 2.0
0 1 2 3 4 months
200 200 200 2001000
Single-batch(ie “waterfall”)
0 1 2 3 4 months
Inventory Inventory
Stages &units / hour
www.qualtechconferences.com© Thompson information Systems Consulting Limited
16
Why inventory damages competitiveness (i)
Goldratt’s objectives for competitiveness
My IT analogies
Product: 1. quality Quality & scope: 1. few defects now, reliable in future
2. engineering 2. amount & sophistication of funct- ionality, non-functional attributes
Price: 3. higher margins Low-cost: 3. economical development & testing so flexible pricing
4. lower investment per unit
4. no environments panic, lower break- even point & flexibility to compete
Responsiveness: 5. due-date performance
Timely delivery: 5. rarely / never late
6. shorter quoted lead- time
6. rapid application development!
www.qualtechconferences.com© Thompson information Systems Consulting Limited
17
Why inventory damages competitiveness (ii)
Objectives for competitiveness
Advantages of low inventory (and/or iterative lifecycles!)
Product: 1. quality Defect detection & fixing sooner after their introduction, fewer recurrences
2. engineering Quicker realisation of improvements
Price: 3. higher margins Lower probability of needing overtime
4. lower investment per unit
Lower probability of needing additional machines (test environments)
Responsiveness: 5. due-date performance
More reliable materials forecasting (smoother staff recruitment)
6. shorter quoted lead- time
Shorter overall time (already shown on slide 15)
www.qualtechconferences.com© Thompson information Systems Consulting Limited
18
Throughput relationships, why good
Competitive edge Extended from Goldratt, The race (North River Press 1986)
High quality Good scopeLow people costsLow machine costsRarely lateRapid
InventoryOperatingexpenseThroughput
ROINet profit Cash flow
www.qualtechconferences.com© Thompson information Systems Consulting Limited
19
Drum-buffer-rope• Optimise throughput by:
– (1) drumbeat based on constraining stage (a) & orders (b)– (2) buffer to protect constraining stage from upstream disruptions– (3) rope to prevent leader extending gap on constraining stage
• For subassemblies feeding in, have additional buffers
assembly constrainingstage (a) ”leader”
buffer
orders (b)
subassembly buffer
raw materialsin
“troops marching” materials flow
(1)
(2)
(3)
www.qualtechconferences.com© Thompson information Systems Consulting Limited
20
Buffer management
x days
y hours
healthy
toocautious
toolean
interruptionof inventory
in
prematurerelease
of inventoryin
www.qualtechconferences.com© Thompson information Systems Consulting Limited
21
Tracing buffer holes back to process steps to fix
3 5
4
2 1
www.qualtechconferences.com© Thompson information Systems Consulting Limited
22
Relationship to other quality approaches
• To recap: Goldratt argues why DBR is better than JIT• In addition to TOC, use lean techniques to minimise waste• TOC project management (critical chain, see Newbold):
– deprecates multi-tasking– translates buffer management to contingency placement– recommends also time management & Pareto techniques– uses risk management to calculate buffer sizes
• Goldratt is more process-oriented than product- (as was Deming)
www.qualtechconferences.com© Thompson information Systems Consulting Limited
23
Is VV&T a factory?• VV&T produces (depending on your viewpoint):
– “project intelligence”;– risk reduction;– a mountain of test plans, scripts, logs, reports etc etc;– bug reports
• Customer does not want to pay for bugs• Bugs are imperfections on inventory rather than inventory
themselves; may not be many in the first place• “Removal of bugs” is not an easy end product to grasp,
and the developers do it anyway • So the code factory & the bug factory are two aspects of
the same thing; VV&T can help identify constraints
www.qualtechconferences.com© Thompson information Systems Consulting Limited
24
4. Beyond the factory: supply chain &
live systems• TOC/JIT says that functionality should be pulled by
customers rather than pushed by analysts / vendors (but why does this often not happen?)
• Software passing testing but not yet paid for is still WIP• Software development has its value chain• Poppendieck principles include “decide as late as
possible”: contradiction with Newbold?• Biggest point here is that most systems go on for years,
and need some documentation or staff continuity (this point receives insufficient attention often)
www.qualtechconferences.com© Thompson information Systems Consulting Limited
25
5. SDLC as an inventory network
• TOC suggests choosing SDLC with a view to what else is wanted other than working software, eg training material, maintenance documentation, regulatory audit
• This affects value of inventory• Cockburn uses TOC principles, of which one consequence is effect on
method• Another contradiction? Advice on handling slack• TOC process improvement can monitor buffers, and is not just for
agile methods• Can still think of VV&T ROI as tangibles & intangibles, even if we
can’t (yet?) calculate• But via the throughput route to ROI we can still seek to optimise
www.qualtechconferences.com© Thompson information Systems Consulting Limited
26
Inventory moving through SDLC
Amount offunctionality
Date
Systemtesting
Live andpaid-for
Acceptancetesting
Programming &unit testing
Integrationtesting
Requirements
Design
Specification
If lines not approx parallel, inventory is growing
Inventory inprocessoverall
Inventory inthis stageof process
Lead time forthis stageof process
Within each stage of testing, can subdivide by
pass/fail, bug states etc
Based on Anderson, Agile management for software engineering (Prentice Hall 2004)
www.qualtechconferences.com© Thompson information Systems Consulting Limited
27
SDLC as an inventory network (cont’d)
order(s)Acceptancetesting
Systemtesting
Integrationtesting
Unittesting
Programming
Requirements
raw materialsin Where is/are the
constraining stage(s)?
Where should buffersbe / not be?
Design
Specification
assembly
sub-assemblies
www.qualtechconferences.com© Thompson information Systems Consulting Limited
28
6. Conclusions and way forward
• Although original Goldratt ideas don’t translate directly to software development, after so much success and so many analogies, can’t ignore (and thinking tools useful for “anything”)
• Several of the later interpretations have plausible messages • Although a few authors have already applied TOC to agile
methods, I hope that by going back to the original principles in some detail I have widened consideration to potentially any SDLC, and provided a platform for me and others to develop this thinking further…
www.qualtechconferences.com© Thompson information Systems Consulting Limited
29
Way forward
Constraint-ledprocess definition& improvement
Context → “Methodologyper project”
project-informing
thinkingtools
Maintenance regime
Circumstance-driven
Publishedmethodologies(“Tailorable”?)
www.qualtechconferences.com© Thompson information Systems Consulting Limited
30
Summary
“How clean is your factory?”
Heavy industry, waste, burnout Light industry, lean, cool fumes
www.qualtechconferences.com© Thompson information Systems Consulting Limited
31
References & acknowledgements
• Main references:– Goldratt, Eli: The race & Necessary but not sufficient– Newbold, Robert: Project management in the fast lane– Mellis, Werner: Process & product orientation– Cockburn, Alistair: Agile software development– Poppendieck, Mary & Tom: Lean software development – DeMarco & Lister: Waltzing with bears– Anderson, David: Agile management for s’ware engineering– Boehm & Turner: Balancing agility & discipline
• Acknowledgements:– Jens Pas EuroSTAR 1998, metrics– Greg Daich STAREast 2002, documentation superstitions
www.qualtechconferences.com© Thompson information Systems Consulting Limited
32
Contact details
Neil Thompson
www.TiSCL.comQuestions?
23 Oast House CrescentFarnham, Surrey, EnglandGU9 0NP, United Kingdom
phone +44 (0)7000 NeilTh (634584) or +44 (0)7710 305907