Humans By The HundredDevOps Days Ohio 2015

$ whoami

SRE Manager at Yelp

CWRU Alum

Pittsburgh native

<3 web operations

Just a dude

Yelp’s Mission:Connecting people with great

local businesses.

Yelp Stats:As of Q3 2015

89M 3271%90M

Growth

Growth means embracing change.

Growth means embracing DevOps.

DevOps: someone having your back.

Dogma

In the Beginning...

Deployment: the early days

Get a few people together in slack/irc/etc.

Merge up the code

Run the tests

Poke at it in stage

Cross your fingers

Things get slower...

Tests take longer to run

More hosts = longer downloads, bounces

More developers = more eyeballs

More features = more code

The Problem:

The Problem: Humans Are Fallible

“…oh @$#&”

The Problem, With Math

Assume:Every change has a chance of success: 98%

That means no test failures, no reverts, etc.

Every deploy has a number of changes: n

Any failure in the pipeline invalidates the deploy

Let’s figure out the probability of a

successful deployment: p

The Problem, With Math

Only youp = 98%

You and a friendp = .98 * .98 = 96%

You and nine co-workersp = .98 * .98 * .98 * … * .98 = 82%

The Problem, With Math

p = (.98)n

The Problem, With Math

p = (.98)n

exponential decay!

This doesn’t scale!

More developers = more changes

More changes = longer deploys

Longer deploys = less time to develop

Less time to develop = slower to iterate

Mitigating Exponential Decay

p = (.98)n

Mitigating Exponential Decay

p = (.98)n

Making it harder to screw up

Write more tests

Write better tests

Get better code reviews

Get better infrastructure

Switch programming languages

Use better tools

Just write better software and stop making mistakes!

PROBLEM SOLVED

The Real World

Testing builds confidence in our changes

Testing does not protect you from failure

Better tools, tests, and infrastructure can

raise our success rates

Mitigating Exponential Decay

p = (.98)n

Mitigating Exponential Decay

p = (.98)n

Service-Oriented Architecture

Large monolith → smaller services

Services communicate over networkUsually HTTP, but you can do RPC, SOAP, etc.

Service = independent code base

Independent deployments

Service-Oriented Architecture

BenefitsSmaller code bases = upper bound to n

Failure domains become isolated

Technology independence

Federated responsibility

Service-Oriented Architecture

Drawbackseverything becomes decoupled

function calls start looking like HTTP requests

versioning can be a nightmare

tracking dependencies is hard

data consistency becomes challenging

end-to-end testing becomes hard(er), if not

impossible

SOA scales people, not code.

Conquering SOA

With the monolith, it’s easy to focus on

mean time between failures (MTBF)

Conquering SOA

In a SOA, focus on mean time to recovery

(MTTR)

Conquering SOA

Fail fast

Anticipate failure

Leverage iteration speed to recover fast

Conquering SOA

Treat everything as distributed

Pick a size that works for youmicro

macro

somewhere in between

Size doesn’t have to be uniform!

PROBLEM SOLVED

Spreading the Love

The Problem: Humans Specialize

Ops Deputies

Developers ‘deputized’ to do operations

Elevated privileges

Tackle infrastructure needs for their team

Contribute improvements to shared infra

Become first-hop for operations questions

DEVOPS

Reinventing the Shuttle

vs.

Build Buy

Build Buy

Composition

You

You

?

Your new thing

You

You fill in these bits

PaaSTA

=

Glue

Define interfaces between componentsMakes it easy to swap out later

Expect change

Minimize code, keep it simple

Think about EOL when you startYour company is changing, and so will your needs

Parting Words

Embrace DevOps

DevOps culture, not just the technology!

Do what’s right for you

Don’t let dogma rule! Tailor your approaches to the

talent around you.

Plan for change

Bit rot is real. Plan how you’re going to deal with it!

@YelpEngineering

YelpEngineers

engineeringblog.yelp.com

github.com/yelp

yelp.com/careers

THANK YOU