8 Rollouts a day

Booking.com, part of priceline.com (Nasdaq: PCLN), headquartered in Amsterdam, 50+ offices worldwide. www.booking.com/offices

8 Rollouts a dayKristian Köhntopp


Most interesting!


This is what we do


This is what we do


This is what we do

• Most hotels worldwide are not in a chain.

• Many hotels communicate by fax.

• Guests need help.

• Self-Service okay, Personal if needed.

• 110+ countries, 40+ languages.


Problem statement


Most likely your problem is different

Problem statement


• Rapid growth - constantly:

• Last years maximumreached early in the new year

• Year after year

Problem statement


• The growth pattern is predictible and stable.

• Due to growth, good practice is constantly obsolete.

• New bottlenecks appear all the time, the larger part of the year we operate in 'blue water'.

Problem statement


Problem statement

The Mission Statement:

Create the required

organisational

personal

technical

structure to match the expected size in time.


Problem statement

There are no constants. Embrace Change.


About booking

• 2006:

• Under 40 persons in IT.

• They know the code, because they wrote it. And the CPAN modules it uses.

• Conditional code in production. Collection of statistics.

• No Ops team.


About booking

• Long startup time for new devs.

• Unstable, slow rollouts.

• Code hard to test, too many variants.

• Code hard to test, changing too fast.

• Drowning in Ops work, too many variants.


Decision

• Velocity. Scalability. Performance.

• Prove business impact.

• Scale the method. Create more great old ones.


Decision





Decision





Decision





Decision





Velocity


Velocity

• Validate requirements: Do they actually help improving the biz?


Velocity

HIghest Paid Persons OpinionExperiments kill HIPPOs


Velocity


Velocity

• Data proves:

• HIPPO ideas are 0% better than improvements by random strangers.


Velocity

• Data proves:


• Conclusion:

• The only reliable way to improve is to run many experiments.


Velocity

• Data proves:


• Conclusion:

• The only reliable way to improve is to run many experiments.

• Velocity: Roll out code as fast as possible,

• and prove it with real users.


http://blog.ads.pof.com/2012/04/03/ throw-everything-you-know-about-ads-out-the-window-pics-inside/

http://blog.ads.pof.com/2012/04/03/throw-everything-you-know-about-ads-out-the-window-pics-inside/



Experiments




Experiments




Experiments



Experiments

• »Experiment« Framework

• Conditional Code,

• measure impact on conversion, performance, etc.


Velocity

• Focus is on capturing quantitative (business) performance data.

• and a low Time To Rollout.


Decision





Proven requirements

ExperimentalImplementation


Proven requirements

• Successful Experiment:

• Full On, even if it is painful! It is earning money!

• Optimize that code.

• Trailing performance team sees only business-positive code.


Proven requirements

• Better insight than tests: The best simulation of a real user is…

• There is a term for this: Testing In Production

http://www.thetestingplanet.com/2011/11/the-future-of-software-testing-part-one-testing-in-production/http://www.setheliot.com/blog/wp-content/uploads/2012/03/STP2012_503_A_to_Z_Testing_in_Production-v2.0.pdf

http://www.thetestingplanet.com/2011/11/the-future-of-software-testing-part-one-testing-in-production/


Testing in production

• Experiment framework

• decoupling deployment/activation

• ramped activation

• exposure control



• Load/Capacity test in production

• determine true capacity, true failure mode

• use a probe to determine latency

• concentrate real user load on few frontends, observe probe latency



• Telemetry

• modern browsers can report page load times back

• add additional system data

• Data mining

• real time monitors

• offline data mining


Decision





Scale the method

Wait.What method?


Shared Spaces


Shared Spaces

Risk compensation is an effect whereby individual people may tend to adjust their behaviour in response to perceived changes in risk. …

Another way of stating this is that individuals will behave less cautiously in situations where they feel "safer" or more protected.

http://en.wikipedia.org/wiki/Risk_compensation


Shared Spaces

Education

Visibility

Empowerment


Shared Spaces


Shared Spaces

Education: You should know your trade.

We're painting you blue.


Shared Spaces

Visibility: Effects of changes should be obvious.

Linear, continuous systems.


Shared Spaces

Empowerment:

Take away all mechanism that makes decisions = Give back responsibility


Booking Rule #1

If you break it,

will you even notice?


Booking Rule #2

If you break it,

can you fix it?


Craftsmanship

The ability to debug a user story front to back,

from Biz to deployment.


Culture of Failure

A good fail is fast and early.

There is a lesson to take away from a good fail.

Bad fails are extended suffering and pain.


So what exactly?

• Hire experienced people

• Real-world goal vs. SE academicscode earning money vs. beautiful code

• with a proven track record in OSS

• working in a cooperative environment, self-organizing, self-driven personality


So what exactly?

• We are not building on a green lawn in the middle of nowhere


So what exactly?


So what exactly?

• We are not building on a green lawn in the middle of nowhere,

• but we are also very aware of the fact that code has a finite lifetime.

• In fact, more than 90% of our code changes are thrown away.


Components

• Experimentation

• Testing in Production

• Shared Spaces

• Education, Visibility, Empowerment

• A culture of failure


Operations POV

So how does this relate to DevOps?


Operations POV


Operations POV

• Kill all sysadmins!

• BOFH is a poisonous term


Operations POV



• Infrastructure Developer

• automate infrastructure tasks


Operations POV



• Infrastructure Developer

• automate infrastructure tasks

• Inhouse Operations Expert

• support developers before and after failure


Operations POV


Operations POV

• Infrastructure Developer:


Operations POV


• Make sure the same features work at a 10x scale.


Operations POV



• Make sure things work with zero manual interactions.


Operations POV



• Make sure things work with zero manual interactions.

• Mindset differs from feature dev a lot.


Operations POV


Operations POV

• Methods differ


Operations POV

• Methods differ

• Scrum-ish vs. Kanban-ish vs. Firefighting


Operations POV

• Methods differ

• Scrum-ish vs. Kanban-ish vs. Firefighting

• All have prio lists

• Feature dev: release oriented, timeboxed

• Infrastructure dev: mission oriented, # of things in flight

• Translator, Agent: look at top10 things in flames right now, pick any


Operations POV


Operations POV


• Feature dev works on feature front to back, biz to deploy


Operations POV



• Infrastructure expert offer local special knowledge, possible solutions and their 'prices'

• Enable dev to make informed decision


Operations POV



• Infrastructure expert offer local special knowledge, possible solutions and their 'prices'

• Enable dev to make informed decision

• Dev then still may choose to break things


Operations POV


Operations POV

• Example:

• DBAs are not 'owning' their databases

• Dev can get advice on schema design, query help, etc at any timπe

• Dev can also elect to go forward against this advice


Operations POV

• Example:

• DBAs are not 'owning' their databases

• Dev can get advice on schema design, query help, etc at any timπe

• Dev can also elect to go forward against this advice

• The experts do not have control


Lock in

• We are in fact locked in…

• Code changes must be small to debug, so we MUST roll out often.

• "Book": No rollout for x months, took weeks to reintegrate.


Lock in

• Dependence on functional and fast monitoring:

• Eventlogging breaks: We cannot roll out.

• Monitors all over the place.

• Obsessed with data-based reasoning.


Lock in

• Individual changes must be small and rolled out fast:

• We cannot use GIT branches, they are tools to accumulate large diffs.

• Unified codebase everywhere.

• We need to roll out systems even if they do not change.


Lock in

• We are load testing weekly:

• Codebase changes quickly.

• A lot of really bad code is put life in experiments, has capacity impact.

• Load testing?

• In production, playing w/ LB weights, of course.


It works for us

"Have you broken the site already?""No.""What am I paying you for?" -- CEO to new dev over lunch


Craftsmanship

• Hire experts: "Fachwissen" They know all possible design choices and the price tags for each of them.

• Teach history: "Sachwissen"The choices we made.


Craftsmanship

• C-Level endorsement.

• Teach business to devs, dev to business people.


Craftsmanship

• No cop out: You can get counselling, but the choice to roll out is yours.

• You wrote it. You are the only expert on that code we have. Your call.

• Avoiding risk compensation.


End-to-end responsibility



• DevOps is magic:



• DevOps is magic:

If you treat adult expertsas responsible adult experts, they will handle themselves as responsible adult experts.


No biz vs. dev

• We are doing many small projects.

• We are creating tight feedback loops.

• We are working interdisciplinary. Business people sit with devs, as needed. Operations people sit with devs, as needed.


Operations as a service



• Primary goal: Velocity.

• Everything else we do as needed.





• We are deploying imperfect, badly performing or otherwise broken code, crap schema designs, or break internal rules. Tough luck. Deal.





• We are deploying imperfect, badly performing or otherwise broken code, crap schema designs, or break internal rules. Tough luck. Deal.

• Some people can't. They quit.


Should you do it?

• Should you do it?

• Website. We control the rollout.

• High growth environment.

• 8 months/year in blue water.

• »Slack is good.« Baby fat eaten by growth rapidly anyway.


Hell yes

• Treat your responsible adults as responsible adults, and they will handle themselves as responsible adults.

• Kill HIPPOs with data.

• Measure. Real world data. For that, you need to be live.


Live by the rules

If you break it, will you even notice?

If you break it, can you fix it?


http://booking.com/jobs

Join us!


http://booking.com/jobs

Join us!

Internet

8 Rollouts a day