Upload
tathagat-varma
View
597
Download
1
Embed Size (px)
Citation preview
From Waterfall to Weekly Releases: A Case Study in using
Evo & Kanban (2004-‐05)
Tathagat Varma hHp://thoughtleadership.in
Our Product • Network Management domain • Windows-based specialized hardware
(“Appliances”) • Installed in data centers for traffic monitoring,
analysis and network troubleshooting – but not generally on production network
• Typical users are technical folks – CIO, Network Manager, Network Engineers
• Selling cycles typically align with quarterly or annual budget cycles
• Many sales require customer specials
Old Process, circa 2003 • Customer Bugs prioritized based on multiple business parameters,
including (partial list) - – Severity – Impact on Revenue, Volume, Competitive, etc. – Case age
• PMO would prepare Maintenance Release Plan of Record (MR POR) and get buy-in for various types of MRs - – Service Packs – bunch up ~50-60 bugs typically every quarter – Hot Fixes – 1-2 high-urgency bugs that can’t wait until next SP – Patches – workaround for customer-specific issues
• SPs would have – Above The Line (ATL) requirements – must fix – Below The Line (BTL) requirements – fix if time permits
Problems with Old Process • Dev team had no bandwidth to take on maintenance releases • Huge pile of customer escalations without “home” • Compounded by high incoming field rate • Low closure rate (largely due to no dedicated resources) • Large wait for customers to get bug fixes • Tech Support often tasked team directly and broke the process • Hot fixes not always available to all customers • Sometimes, a new bug fix might break a hot fix • If a hot fix failed in the field, rollbacks would be very difficult • Difficult to estimate time to resolve a bug and give an ETA • High-priority bugs could arrive at any time • Customer specials could arrive anytime with top priority • High internal rejection rate of bug fixes by Tech Support
New Process, 2004-05
Dedicated Customer Sustaining
Team
New “Cumulative
Hot Fix” process
Improved collaboration
with all stakeholders
EvoluLonary Project Management “Evo”
• E1: Decompose by performance results and stakeholders • E2: Do high-‐risk steps early, learn how ‘unknowns’ really perform • E3: Focus on improving your most valuable performance objecLves
first • E4: Base your early evoluLon on exisLng frameworks and
stakeholders • E5: Design to cost dynamically • E6: Design to performance dynamically • E7: Invest in an open-‐ended architecture early on • E8: MoLvate your team by rewarding results • E9: PrioriLze changes by value, not place in queue • E10: Learn fast, change fast, adapt to reality fast.
Process improvement…the beginnings…
Cumulative Hot Fix Process
Weekly Build Process
Product C
Product D
Protocols
Decide Drivers
Backend
Product B
GUI
Product A
Our Kanban Process in action…
PMO
CST Manager
QA Team
Dev Team = 15
WIP = 15
WIP = 2
WIP = 1wk
WIP = 1wk
Tech Support Que
ue = 0
Queue = 0
So, what is happening? • Though not an originally stated vision or goal, the “Work in
Progress” (WIP) is being limited to # of team members • At any time, one developer is assigned only one piece of work,
thereby achieving “One-Piece Flow” • New work is only assigned when current work is
completed (or cancelled/stalled), and a team member is available
• No wait state or switching costs at an individual level • Smaller lead time for bugs (in contrast to lead time for SP) • The process is allowing ‘weekly deployment’ of each of the
hot fixes • Finally, the flexibility gained is not a zero-sum game – there is
no penalty on performance in rest of the process
Did this move the needle? • Bugs addressed each quarter • Quality of bug fixes • “Homes” for bugs • Total bugs open • Open days open • People motivation
Shift from SPs to Cumulative Hot Fixes while maintaining High Quality
Maintenance Releases (Service Packs, Patches, Hot Fixes) Q3 2003 Through Q4 2006 (Fiscal Year)
0 2 3 4 4 4 4 2 2 2 1 2 3 15 2 1
5 3 1 00 1 0 0 0
10
28
11
16 18 20
62
28 2726 26 24
28
21
100
92
87
96
8892
85
80
97 9693
100
9491
7
1215
25 25 25
30 3028 27 26
32
22
66
0
10
20
30
40
50
60
70
Q3 03 Q4 03 Q1 04 Q2 04 Q3 04 Q4 04 Q1 05 Q2 05 Q3 05 Q4 05 Q1 05 Q2 06 Q3 06 Q4 060
10
20
30
40
50
60
70
80
90
100
Service Packs Patches Hot Fixes % Successful
Percentage of released Maintenance Releases (Service Packs, Patches, Hots Fixes) that addressed customer reported
Total Maintenance Releases (Service Packs, Patches, Hot Fixes) for this quarter.
ServicePacks
Patches
Hot Fixes
Increase in bugs with “homes”
Customer Direct and Indirect % w/Homes - HistoricalWeek Ending February 03, 2006
55
46 45 47 44 45 45 44
5552 54 56
35
4539 38 41
5146
31
40 40
5963 60 61 61
68 6974 72 69 69
55
73
61
53565548
54 5552 51 51 51 50 49
54 55
47 47
1115 18 16
4753 55
43 42 43
7582
76 76 7582 83 82 81
71 7265
8782
7580
55
4751 51 50 48 48 48 48 48
53 54 51 51
2127 27 25
44
52 51
43 41 42
6772
68 69 6875 76 78 76
70 71
60
80
7164
68
44 47
0
10
20
30
40
50
60
70
80
90
100
5/6/200
5
5/13/2
005
5/20/2
005
5/27/2
005
6/3/200
5
6/10/2
005
6/17/2
005
6/24/2
005
7/1/200
5
7/8/200
5
7/15/2
005
7/22/2
005
7/29/2
005
8/5/200
5
8/12/2
005
8/19/2
005
8/26/2
005
9/2/200
5
9/9/200
5
9/16/2
005
9/23/2
005
9/30/2
005
10/7/
2005
10/14
/2005
10/21
/2005
10/28
/2005
11/4/
2005
11/11
/2005
11/18
/2005
11/25
/2005
12/2/
2005
12/9/
2005
12/16
/2005
12/23
/2005
12/30
/2005
1/6/200
6
1/13/2
006
1/20/2
006
1/27/2
006
2/3/200
6
Direct Indirect Goal (80% w/Homes) Combined
Percent with Releases Identified: 56% (Driect), 80% Indirect, 68% (Combined)
Total Bugs Open Severity Across Products (Historical)
Week Ending March 31, 2006
222
191
243
213
224230
236
201199
210211
196197195
161
147146146146
132
122
131
121127
111
127125
148150
65605857
5357
626466676768
71747575
6866
4947465049495052
606369707069
747368
61605555
47495149
4435383233323435
36
0
10
20
30
40
50
60
70
80
90
100
110
120
130
31-Dec-03
01-Oct-04
29-Oct-04
19-Nov-04
10-Dec-04
31-Dec-04
21-Jan-05
11-Feb-05
04-Mar-05
25-Mar-05
15-Apr-05
06-May-05
27-May-05
17-Jun-05
08-Jul-05
29-Jul-05
19-Aug-05
09-Sep-05
30-Sep-05
21-Oct-05
11-Nov-05
02-Dec-05
23-Dec-05
13-Jan-06
03-Feb-06
24-Feb-06
17-Mar-06
0
20
40
60
80
100
120
140
160
180
200
220
240
260
Total Severity 1 Severity 2 Severity 3 Severity 4 Severity 5
Average Days Open Severity: Average Days By Severity (Historical)
Week Ending March 31, 2006
203
155
136
244242246246244248
234235226223
232
244
260268275275
299
232
201210207
179
165172
156162
180
168163156152
147142132135
139143148
140144147146144
155
135145
152148
157156156149
133
144140
144147154155157
163166
177179183182
164162165
154146
141149
123117
134134
123
0
25
50
75
100
125
150
175
200
225
250
275
300
325
350
375
400
425
450
475
500
31-Dec-03
01-Oct-04
29-Oct-04
19-Nov-04
10-Dec-04
31-Dec-04
21-Jan-05
11-Feb-05
04-Mar-05
25-Mar-05
15-Apr-05
06-May-05
27-May-05
17-Jun-05
08-Jul-05
29-Jul-05
19-Aug-05
09-Sep-05
30-Sep-05
21-Oct-05
11-Nov-05
02-Dec-05
23-Dec-05
13-Jan-06
03-Feb-06
24-Feb-06
17-Mar-06
0
25
50
75
100
125
150
175
200
225
250
275
300
Avg Days Open Severity 1 Severity 2 Severity 3 Severity 4 Severity 5
People motivation
12 12 12 12 12 10 10 10
4 4 4 4 4 6 6 6
16 16 16 16 16 16 16 16 16 16 16 16
0 2 4 6 8
10 12 14 16 18
Resource Staffing
Sustaining Mainline Approved Headcount
• Started with 16 people dev team • We had zero attrition in the team • Once the backlog started coming down, engineers were ramped
off the team to do new features • Eventually dismantled the team and rolled-up engineers into dev
teams when backlog came down to single digits
What is Kanban? • Kanban (literally signboard or billboard) is a scheduling system
for lean and just-in-time (JIT) production. According to its creator, Taiichi Ohno, kanban is one means through which JIT is achieved.
• Kanban is not an inventory control system; it is a scheduling system that helps determine what to produce, when to produce it, and how much to produce.
• The need to maintain a high rate of improvement led Toyota to devise the kanban system. Kanban became an effective tool to support the running of the production system as a whole.
• In addition, it proved to be an excellent way for promoting improvements because reducing the number of kanban in circulation highlighted problem areas.
hHps://en.wikipedia.org/wiki/Kanban
A Kanban System at Toyota dealership
hHps://twitpic.com/het3u
How does it work?
hHp://www.toyota-‐global.com/company/vision_philosophy/toyota_producLon_system/just-‐in-‐Lme.html
How Kanban helps achieve “Just-in-Time”
• For example, to efficiently produce a large number of automobiles, which can consist of around 30,000 parts, it is necessary to create a detailed producLon plan that includes parts procurement. Supplying "what is needed, when it is needed, and in the amount needed" according to this producLon plan can eliminate waste, inconsistencies, and unreasonable requirements, resulLng in improved producLvity.
hHp://www.toyota-‐global.com/company/vision_philosophy/toyota_producLon_system/just-‐in-‐Lme.html
Kanban for Software • Visualize the Workflow: Represent the work items and
the workflow on a card wall or electronic board • Limit Work-in-Progress (WIP): Set agreed upon limits
on how many work items are in progress at a time • Measure and Manage Flow: Track work items to see if
they are proceeding at a steady, even pace • Make Process Policies Explicit: Agree upon and post
policies about how work will be handled • Use Models to Evaluate Improvement Opportunities:
Adapt the process using ideas from Systems Thinking, Deming, etc.
Kanban: Successful Evolutionary Change for your Technology Business – David Anderson
Why Kanban in Software Engineering?
Don’t build features that nobody needs right now
Don’t write more specs than you can code
Don’t write more code than you can test
Don’t test more code than you can deploy
hHps://leanandkanban.files.wordpress.com/2009/04/kanban-‐for-‐sohware-‐engineering-‐apr-‐242.pdf
What did We learn?
• Process improvement should be driven by business needs – and NOT because some process looks sexy!
• Don’t let a process limit your potential – think beyond gurus!
• Don’t let absence of a process limit your potential – do whatever it takes to serve customer better!
Resources • Was that Kanban? - http://finance.groups.yahoo.com/group/kanbandev/message/4131 and
http://finance.dir.groups.yahoo.com/group/kanbandev/message/4166 • http://refcardz.dzone.com/refcardz/getting-started-kanban • Ship early and ship twice as often,
https://www.facebook.com/notes/facebook-engineering/ship-early-and-ship-twice-as-often/10150985860363920
• How we built Flickr, http://www.plasticbag.org/archives/2005/06/cal_henderson_on_how_we_built_flickr/
• Continuous Deployment at IMVU: Doing the impossible fifty times a day, http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/
• A New Version of Google Chrome now due every six weeks, http://techcrunch.com/2010/07/22/google-chrome-versions/
• In praise of continuous deployment: The WordPress.com story, http://toni.org/2010/05/19/in-praise-of-continuous-deployment-the-wordpress-com-story/
• CONWIP, https://en.wikipedia.org/wiki/CONWIP • Kanban applied to Software Development: from Agile to Lean,
http://www.infoq.com/articles/hiranabe-lean-agile-kanban
Photo by orcmid - Creative Commons Attribution License https://www.flickr.com/photos/91555706@N00 Created with Haiku Deck
Photo by Neil Melville-Kenney - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/27774670@N07 Created with Haiku Deck
Photo by Brett Whaley - Creative Commons Attribution-NonCommercial License https://www.flickr.com/photos/21920030@N02 Created with Haiku Deck
Photo by raneko - Creative Commons Attribution License https://www.flickr.com/photos/24926669@N07 Created with Haiku Deck
Photo by Photosightfaces - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/30595068@N06 Created with Haiku Deck
Photo by geezaweezer - Creative Commons Attribution License https://www.flickr.com/photos/33909206@N04 Created with Haiku Deck
Photo by familymwr - Creative Commons Attribution License https://www.flickr.com/photos/36196762@N04 Created with Haiku Deck
Photo by Mónica, M - Creative Commons Attribution-NonCommercial License https://www.flickr.com/photos/52329444@N00 Created with Haiku Deck
Photo by joiseyshowaa - Creative Commons Attribution-ShareAlike License https://www.flickr.com/photos/30201239@N00 Created with Haiku Deck
Photo by `James Wheeler - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/24128704@N08 Created with Haiku Deck
Photo by William Christiansen - Creative Commons Attribution License https://www.flickr.com/photos/56222080@N04 Created with Haiku Deck
Photo by merrycrafts - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/10314331@N08 Created with Haiku Deck
Photo by susanvg - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/28362388@N00 Created with Haiku Deck
Photo by monkeyc.net - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/73584213@N00 Created with Haiku Deck
Photo by Romain [ apictureourselves.org ] - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/72898690@N00 Created with Haiku Deck