Continuous Deployment

@rantav@outbrain

How much time does it take your code to meet the users?

How much time does it take your code to meet the users?• More than a year?

How much time does it take your code to meet the users?• More than a year?

• 6 - 12 Months?

How much time does it take your code to meet the users?• More than a year?

• 6 - 12 Months?

• 1- 6 months?

How much time does it take your code to meet the users?• More than a year?

• 6 - 12 Months?

• 1- 6 months?

• 2 - 4 weeks?

How much time does it take your code to meet the users?• More than a year?

• 6 - 12 Months?

• 1- 6 months?

• 2 - 4 weeks?

• 1 - 14 Days? 

How much time does it take your code to meet the users?• More than a year?

• 6 - 12 Months?

• 1- 6 months?

• 2 - 4 weeks?

• 1 - 14 Days?

• 1 - 24 Hours?

How much time does it take your code to meet the users?• More than a year?

• 6 - 12 Months?

• 1- 6 months?

• 2 - 4 weeks?

• 1 - 14 Days?

• 1 - 24 Hours?

•  < 12 Minutes? 

~ $ svn ci -m "Implement the super-sharp image scaler. #deloy:ImageServer #to:ny"

Outbrain enables readers to discover the most interesting, relevant and timely links to stories (paid and organic)

Multi Billion Page Views per month

Note: Outbrain is typically installed on *every* article/blog post on each of these sites, immediately under the content.

Who's in?

WHY

WHY

For the Business

For Fun

WHYWhat was so bad before ?

WHY

What is a startup?

• A startup is always on a quest to find Product-Market fit

• => You have to iterate Fast• ... before running out of money

WHY

• Feedback loop speed is important (see REPL in programming)

WHY

What's used to be the case before:

• Inefficient Waits (wait for QA, wait for other features etc)

• Inefficient Context Switch• Feature Delayed - b/c of other features• Big Changes – Big Problems.

HOW

HOW

HOWContinuous Deployment Themes

• Release is a marketing decision.

• Deployment is an engineering decision!

HOWContinuous Deployment Themes

• Small Changes – Reduce Risks

• Kent Beck: You can spill a bucket but you can't spill a Hose. 

HOWContinuous Deployment Themes

• Deploy fast, release often

• Fast turnarounds lead to happy customers

• ... and happy developers

HOWCulture

• Everyone have to care about everything!o buildo testso qualityo productiono monitoringo business

HOWCulture

• No Broken Windows!

Broken Windows Theory

HOWCulture

“What's the worse that could happen?”

WORKFLOW

WORKFLOWDev

WORKFLOWDev

The

 INGREDIENTS

INGREDIENTSTrunk Stable

•Everyone run tests before commit•No branches, really•Trunk may get released any

moment. If you commit now, users will see it really soon.

    => Test your code, really good•Use feature flags (but only if you

must)• Forward and backward compatibility

INGREDIENTSAutomated Tests

•We have about 2000 test cases•They run in < 4 minutes•We use TeamCity•We regularly test production

Desired state: No manual QA whatsoever.

INGREDIENTSInfrastructure Automation

"Infrastructure as code"

•Using kickstart script to install OS and Chef agent.

•All infrastructure is deployed by Chef.

•All apps are deployed by Glu•All scripts, and chef cookbooks, glu

configs are SVNed.•Very easy to deploy large number of

machines. 

INGREDIENTSDeployer

INGREDIENTSServicization

• At outbrain we have ~ 25 services• Each service is deployed at > 1 server• If there's damage, it's contained• It's easy to make small changes• If there's an error it's easy to find it• If rollback is needed it's easier• Everything is either proxies (HAProxy) or queued (AMQ)

• Challenge: It's sometimes hard to find the right balance:o number of services that need maintananceo complexityo performanceo api conformance

INGREDIENTSThe Immune System

• Codingo Reviewo Testing

• Monitoringo We use Nagios for the vitalso Regularly check servers via instrumentationo KeyNoteo Test Production (selenium,APT)

• Instrumentationo Self-Testo Performance 

• Monitoring KPIso PVs, o 3 different CTRs,o clicks, revenue, RPMo etc...

INGREDIENTSThe Immune System

• The line of defense will always be broken

=> Multiple lines of defense

INGREDIENTSVisibilty

• Everyone monitor the services • When there's a deployment, you see it on the graph

INGREDIENTSVisibilty

• Yammer

INGREDIENTSVisibilty

• svn changelog• glu changelog• glu audit log

INGREDIENTSLearn Fast, Adapt Fast

• 5 Ys

ARSENALOutbrain's arsenal

Architecture:

• Java in most parts, Spring for the most and Struts2• SOA (dah...) with REST• Message queues (ActiveMQ)• Service multiplicity with HAProxy• Data stored in:

o MySQL, Cassandra, HDFS, NFS•  Caching: Memcached

ARSENALOutbrain's arsenal

Testing:

• JUnit• TeamCity• Selenium• Staging environment (for high risk deployments)

ARSENALOutbrain's arsenal

Deployment:

• Kickstart• Chef• Glu• TeamCity• In-house Deployment Manager• RPM + YUM

ARSENALOutbrain's arsenal

Monitoring:

• Nagios• "Testing production" every 10 minutes. • Keynote

ARSENALOutbrain's arsenal

Communications:

• Yammer

Fun Numbers

• 5-50 production changes a day!!!• More then 2000 code tests running in

less then 4 minutes.• More then 600 production services tests

runs every 10 minutes.• It takes ~30 minutes from code

complete to ~100 machines deployed.

References

• Why Continuous Deployment / Eric Ries• Continuous Deployment at outbrain / Ran Tavory • Deployment Infrastructure for Continuous Deployment /

WealthFront• Continuous Deployment presentation / Eishay Smith• Quantum of Deployment / Etsy• Chrome Release Cycle / Anthony Laforge