Upload
phil-sarin
View
468
Download
1
Tags:
Embed Size (px)
Citation preview
Queues: An Invisible Money Drain
Phil Sarin VP Engineering, GameChanger Media
@philsarin http://216ways.net
“It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.
— Mark Twain
Things I once believedYou shouldn’t release often because releases have high overhead.
You need to track all of your bugs.
Slack is bad. Your team should be busy.
Specialists will always be more productive than generalists.
What we’ll cover tonightWhat a queue is
Identifying the queues in your organization
Tactics for handling queues
Some war stories
“Queues are the root cause of the majority of economic waste in product development.”
— Donald G. Reinertsen
Examples of queuesFeature QueuesWaiting to get designed Waiting to get developed Waiting for testing Waiting to get released Waiting to get validated
Bug QueuesWaiting for triage Waiting to get fixed Waiting to get verified Waiting for release
What’s wrong with big queues?Big queues increase cycle time and delay cost
Big queues increase variability
Big queues require management
Big queues delay feedback
Big queues hurt morale
Why small batches are (usually) better
Realize benefits sooner
Faster feedback
Higher motivation
Less schedule slippage
Old School: Releases are expensive so let’s batch up work for release.
New School: Let’s drive down the cost of releases so that we can release more often.
Queue takeaways
Queues introduce delay.
Queues have economic and psychological costs.
High capacity utilization drives up queue size.
Small batches can reduce the cost of queues.
How we’ve managed queues
1. Swarm
2. Generalize queue handling
3. Obliterate queues
4. Reduce testing batch sizes
The stages of swarming
1. Skepticism/suspension of disbelief
2. Guilt and busywork
3. Scope blows up
4. The “surge capacity effect”
Multiple queues, 1 server each
Single queue, Single server
Single queue, pool of servers
Sustained huge queuesare least likely
Pressures to specializeA specialty differentiates your business
“I don’t want to touch that code!”
Initial builder owns it forever.
Early adopters become de-facto specialists.
Some people want specialist careers.
3. Obliterate queues
http://provocateurs.ca/2014/11/16/delmar-dont-be-afraid-to-hit-delete/
2 months development 2 weeks testing
2 months development 2 weeks testing 4 months bug-driven development
2 weeks dev test 2 weeks
dev test 2 weeks dev test 2 weeks
dev test 2 weeks dev test
Can we reduce this batch size further?
Our first batch size reduction
2 weeks dev test 2 weeks
dev test 2 weeks dev test 2 weeks
dev test 2 weeks dev test
Can we reduce this batch size further?
2 weeks dev+test T 2 weeks
dev+test T 2 weeks dev+test T 2 weeks
dev+test T 2 weeks dev+test T 2 weeks
dev+test T
Continuous pre-release delivery?
How we fixed it
More automated test coverage
Instituted small batch manual testing process
Hired more testers
Release Days5.15 215.14 145.13 265.12 215.11 235.10 45.9 665.8 75.6 21
It worked, mostly, eventually
Things I once believedYou shouldn’t release often because releases have high overhead.
You need to track all of your bugs.
Slack is bad. Your team should be busy.
Specialists will always be more productive than generalists.
Takeaways
Queues are real and expensive
You should know where your queues are
There are a bunch of ways to reduce the cost of queues
Further Reading
The Principles of Product Development Flow, Donald G. Reinertsen
“Software Inventory” by Joel Spolsky: http://www.joelonsoftware.com/items/2012/07/09.html
“How we fixed more bugs by deleting our bug DB” and “Building around generalists” on my blog (216ways.net)