The Upside of Downtime (Velocity 2010)

http://gapingvoid.com/

Sunday, June 20, 2010

The Upside of DowntimeTurning disaster into opportunity

Who’s had a site go down?

Who’s hasn’t had a site go down?

There’s always that one guy!

Downtime sucks

Source: http://www.motivatedphotos.com/?id=8080

Why downtime sucks

Business

$1,500

$2,250

$3,000

0 2 4 6 8 10 12 14 16 18 20 22

Why downtime sucks

Business

Why downtime sucks

Business

Why downtime sucks

Business

Downtime = Bad! (Duh)

Approach #1Don’t fail

Source: http://kansansforlife.files.wordpress.com/2009/12/titanic.jpg

“Everything fails all the time”-- Werner Vogels (Amazon, CTO)

Your site will fail

Werner Vogels (Amazon, CTO)

Why?!?

Risk Homeostasis

Why Failure Happens

Source: http://joshuahind.files.wordpress.com/2009/09/bicycle-crash.jpg

Risk Homeostasis

Black Swan

Why Failure Happens

Source: Amazon.com

Risk Homeostasis

Black Swan

Unknown unknowns

Why Failure Happens

Source: http://www.apoliticus.com/wp-content/uploads/2009/01/6_21_080306_rumsfeld.jpg

Risk Homeostasis

Black Swan

Unknown unknowns

Change

Why Failure Happens

Source: http://bozark.net/wordpress/wp-content/uploads/2008/09/barack_obama_change_fairey.jpg

Risk Homeostasis

Black Swan

Unknown unknowns

Change

Many small failures

Why Failure Happens

Source: http://www.biojobblog.com/uploads/image/dominos.jpg

Risk Homeostasis

Black Swan

Unknown unknowns

Change

Many small failures

Humans

Why Failure Happens

Source: http://www.librarian.net/talks/clc/CLC.key/SJ_Shoulder_Shrug.jpg

Not unusual

Polisherblocked

Source: http://www.gladwell.com/1996/1996_01_22_a_blowup.htm

Not unusual Not expected

Polisherblocked

Moisture leaks into air system

Not unusual

Polisherblocked

Flow of cold water stopped

Not expected Not good

Not unusual

Polisherblocked

Not expectedBackup disabled

Not unusual

Polisherblocked

Indicator blockedDoh!

Not unusual

Polisherblocked

Indicator blocked

Relief valve broken

Dammit

Not unusual

Polisherblocked

Indicator blocked

Relief valve broken

Gauge broken

Dammit

Not unusual

Polisherblocked

Meltdown

Indicator blocked

Relief valve broken

Gauge broken

Dammit

Source: http://support.rightscale.com/09-Clouds/AWS/02-Amazon_EC2/Designing_Failover_Architectures_on_EC2/03-Advanced_Failover_Architecture

“accidental power failure”

Source: http://www.datacenterknowledge.com/archives/2010/06/16/power-failure-kos-intuit-sites-for-24-hours/

“traffic accident damaged a nearby utility transformer”

Source: http://www.datacenterknowledge.com/archives/2007/11/13/truck-crash-knocks-rackspace-offline/

“unfortunate code change”Source: http://www.datacenterknowledge.com/archives/2010/06/11/errant-code-change-crashes-10-million-blogs/

“Unhappy customers may get some attention, but unhappy networked customers can quickly impact your business”

-- Clay Shirky

Source: http://happenupon.files.wordpress.com/2009/02/technology-guru-clay-shir-001.jpg, http://scholarlykitchen.sspnet.org/2010/03/02/shirky-at-nfais-how-abundance-breaks-everything/

http://labs.webmetrics.com/crowdsourceduptimeSunday, June 20, 2010

Your site will fail

Your site will fail+Downtime is bad

Your site will fail+Downtime is bad+Everyone will find out

Your site will fail+Downtime is bad+Everyone will find out=Screw it, I’ll become a lumberjack

Source: http://sbadrinath.files.wordpress.com/2009/03/different26rqcu3.jpg

“Embrace fear of outages and degradation. Use it to guide your architecture, your code, your infrastructure. So lean into it.”

-- John Allspaw, VP Tech. Ops at Etsy

Approach #2Prepare for downtime

Disclaimer: Try hard to avoid downtime

Learning by example...

Case Study #1Facebook

“The larger issue here isn't just that a portion of Facebook's platform has gone down - numerous web services have issues from time to time, including everything from Gmail to Twitter. An outage of this length, however, with no official communication from the company itself is disturbing.”

-- N.Y. Times

Downtime Disturbing

Facebook

Case Study #2Google App Engine

Downtime Kudos

Google App Engine

Case Study #3Atlassian

Downtime

Atlassian

http://atlassian.com/

Downtime:Opportunity to Build Trust

Downtime:Opportunity to Destroy Trust

How To: Prepare for Downtime

Something > Nothing

Upside of Downtime Framework 1.0

Oh crapLife is good That sucked

CommunicatePrepare Explain

1. Communication channel

Something is wrong

Can’t tell if it’s me or you

I’ll assume it’s you

You suck

Something is wrong

Can’t tell if it’s me or you

I’ll assume it’s you

I know it’s youTell me when you’re back

You suck a lot less

1. Communication channel Easy to find

Hosted off-site

Real-time / automated

7 keys for public health dashboards

1. Must show current status for each “service”

2. Data must be accurate and timely

3. Must be easy to find

4. Must provide details for events in real time

5. Provide historical uptime and performance data

6. Provide a way to be notified of status changes

7. Provide details on the data is gathered

Source: http://www.transparentuptime.com/2008/11/rules-for-successful-public-health.html

Hosted off-site

2. Process

Hosted off-site

2. Process Authority

Hosted off-site

Mean-Time-To-Communicate (MTTC)

Hosted off-site

Mean-Time-To-Communicate (MTTC)

On-call/drills/escalations/etc.Sunday, June 20, 2010

Your servers

Prepare ExplainCommunicate

1. Communicate

1. Communicate Use communication channel

Who/what is affected

When the incident started

Update regularly

2. Fix it!Sunday, June 20, 2010

Phew, close one!

1. Postmortem

1. PostmortemAdmit failure

Source: http://en.blog.wordpress.com/2010/02/19/wp-com-downtime-summary/

Sound like a human

Source: http://www.bureauofcommunication.com/compose/apology

“We apologize for any inconvenience this may

have caused”

Sound like a human

Start time and end time

Source: https://groups.google.com/group/google-appengine/browse_thread/thread/a7640a2743922dcf

Sound like a human

Who/what was impacted

Source: http://techcrunch.com/2009/11/02/large-scale-downtime-at-rackspace-cloud/

Sound like a human

What went wrong

Source: http://www.zendesk.com/2010/03/tuesday-double-whammy.html

Sound like a human

What went wrong

Lessons learned

Source: http://graysky.org/2010/02/downtime-postmortem/

Sound like a human

What went wrong

Lessons learned

“I was completely overwhelmed by the amount of positive feedback and support I received.”

Sound like a human

What went wrong

Lessons learned

2. Improve for the futureSunday, June 20, 2010

“Google is not just saying sorry, they are actually implementing serious changes which probably represents millions of dollars of development to help make sure this doesn't happen again.”

Source: http://news.ycombinator.com/item?id=1168493

Source: https://groups.google.com/group/google-appengine/browse_thread/thread/a7640a2743922dcf

Be human

Be authentic

Be transparent

Accept responsibility

Learn and improve

CommunicatePrepare Explain1. Communication channel - Easy to find - Off-site - Real-time

2. Process - Give authority - M.T.T.C. - On-call/escalations

1. Communicate - Use channel - M.T.T.C. - Who/what affected - When started - ETA to resolution - Update regularly

2. Fix it!

1. Post-mortem - Admit failure - Sound like a human - Start time and end time - Who/what was impacted - What went wrong - Lessons learned

2. Learn and improve

Be HumanBe TransparentBe Prepared + +

2. Fix it!

Be HumanBe TransparentBe Prepared + +

Trust=

2. Fix it!

Disclaimer:Don’t screw up too often

Transparent Not Transparent

Caught

Not Caught

Downtime Prisoner’s Dilemma

Caught

Not Caught Win

Caught

Not Caught

Big Loss

Caught

Not Caught

Big Win Big Loss

Caught

Not Caught

Big Win Big Loss

Win Win

Caught

Not Caught

Big Win Big Loss

Win Win

BenefitsGain trust

Reduce churn, increase loyalty

Reduce support costs

Ability to control the message

Competitive advantage

More time to focus on the actual problem

Reduce stress

Change != Easy

Change != Impossible

Keys to Adoption

Getting past a culture of “hide the problem”

Keys to Adoption

Overriding commitment to want to improve

Keys to Adoption

Available resources to improve

Keys to Adoption

Buy-in

Product Management

Support

Sales/Marketing

Engineering/Operations

Product Management

Support

Default: Lets wait for complaints

Sales/Marketing

Product Management

Support

Reality: Proactiveness => Forgiveness

Sales/Marketing

Product Management

Support

Default: Too much work

Sales/Marketing

Product Management

Support

Reality: More upfront, less when it matters

Sales/Marketing

Product Management

Support

Default: Don’t want to look bad

Sales/Marketing

Product Management

Support

Reality: Opportunity to learn/improve

Sales/Marketing

Product Management

Support

Default: I don’t want my customers to knowSales/Marketing

Product Management

Support

Default: I don’t want my customers to know

Reality: They’ll find out, better from usSales/

Marketing

Product Management

Support

Default: I don’t want my customers to know

Reality: They’ll find out, better from usSales/

Marketing

Source: http://delicious.com/lennysan/healthdashboard

Simple as that!

Your site will still fail!

“The measure of a society is how well it transforms pain and suffering into something worthwhile.”

-- Fredrick Nietzsche

“The measure of a company is how well it transforms pain of downtime into something worthwhile.”

-- Lenny Rachitsky

Source: Original quote inspired by Fredrick Nietzsche

Bare minimum:Register a Twitter account

Lenny Rachitsky@lennysanhttp://www.transparentuptime.com/

Webmetrics/Neustar@webmetricshttp://www.webmetrics.com/

Slides: http://bit.ly/upside-of-downtime

Thank You

2. Fix it!

"Unlikely that an accidental surface or subsurface oil spill would occur from the proposed activities"

-- Exploration and environmental impact plan

Source: http://en.wikipedia.org/wiki/Deepwater_Horizon_drilling_rig_explosion

2. Fix it!

“Be not afraid of transparency; some are born transparent, some achieve transparency, and others have transparency thrust upon them.”

-- Burrowed from William Shakespeare

Making change1. Find the bright spots - (this presentation has a bunch)

2. Script the critical moves - (framework)

3. Point to the destination - (W.W.G.D.)

4. Find the feeling - (how would you feel?)

5. Shrink the change - (start small)

6. Grow your people - (everyone is learning as they go)

7. Tweak the environment - (create a simple process)

8. Build habits - (build process organically)

9. Rally the herd - (get buy in, rest will follow)

The Upside of Downtime (Velocity 2010)

Technology

Zero Downtime Migration

UPside No. 2

Learning Upside Down:

Upside Down - catalogue

Downtime Tech Mixer

Upside down thinking

Words Upside Down

Does Upgrade = Downtime?

Upside Down #02

UPside No 7

Upside down greatness

Upside of anger

Towards zero downtime

The upside of disruption - ValueWalk · The upside of disruption. Megatrends shaping 2016 and beyond 3. 4 . The upside of disruption Megatrends shaping 2016 and beyond The upside

Upside Down

Downtime Status (Autosaved)

Money Upside Down

UPside No. 5

Reducing Network Downtime

SES314. Availability %Downtime / Year Downtime / Month Downtime / Week 99%3.65 days7.20 hours1.68 hours 99.9%8.76 hours43.20 minutes10.10 minutes