Right - YOW! Conferences...Release Anti-Patterns Making Easy = Right 15 Extensive, detailed release documentation Reliance on manual testing to conﬁrm app is correct Explaining why

DevOps @ Wotif Making Easy = Right

Alexandra Spillane • @ajbw Matt Callanan • @mcallana

Right

Easy

Outline

•  What is “The Right Thing”? •  How do we Make it Easy? •  Lessons Learned

Making Easy = Right 2

What is “The Right Thing”? DevOps @ Wotif: Making Easy = Right


“The Right Thing”


Collaborate!

Automate!

But… The Dev<->Ops Chasm


What Were we Doing Wrong?


Downward Spiral of Manual Activity


Entrenched Silos


High Release Overheads


Huge Batch Sizes


Calendar of Doom


Unpredictable Prioritisation


Superstitious Gatekeeping


just in case

Stagnating Applications


Release Anti-Patterns


Extensive, detailed release documentation

Reliance on manual testing to confirm app is correct

Explaining why deployment is going wrong on release day

Frequent corrections to release process during release

Environments that differ in their configuration

Releases that take more than a few minutes to perform

Releases that are unpredictable in their outcome

Cycle Time measured in

weeks/months

Application Pipelines Ops

Days

Weeks!

Days

Staging Load Test Production

Days

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA !?

Prio

ritis

ed R

elea

se Q

ueue

Plus… Migrating to MSA

•  Explosion of manual effort


A B

C D

E F

A

B1 C1 G C2 H I

B2 D1 J

D2 K L1 M L2 N

E F1 O P

F2 Q R S T

HeavyWeight to LightWeight

Glassfish •  Feature overhead •  Encouraged manual config •  Complex/Slow deployment

DropWizard •  Trimmed down to basics •  Standard config files •  Simple/Fast deployment


But… Combinatorial Explosion


Unpredictability Slows You Down


Moving from wrong to right


Wrong

Right

Reduce Cycle Time

Making Easy = Right

Standardisation


Standardised

Freedom

•  Standardise on how services are deployed and how they communicate.

•  Be flexible about their contents.

Defining Standards

•  Discussions –  One-on-one –  IM –  Emails –  Meetings

•  Incentivise input •  Confluence

–  Application Deployment Standards 1.0


Example Standards


Logging •  Log file locations, filenames

Log rotate Directories/Files •  Ownership,

permissions

Puppet

Supervisord config.yaml •  Filename, location

Metrics JVM settings

Network •  Ports •  Firewall rules

Cron Endpoints •  Healthchecks, status,

metrics

SSL •  Certs, keystores

RPM •  Versioning •  Packaging

init.d scripts •  Sub-commands

Versioning Migration Notes

Example Standards


Future Considerations


Standards Lifecycle


Common Ground


How do we Make it Easy? DevOps @ Wotif: Making Easy = Right


Automated Verification

Making Easy = Right

Compliance Test Suite Bring Operations into Development Fast Operational Feedback Backwards Compatibility Test Driven Operational Compatibility

Ops Testing with Fabric


• Does rpm have correct metadata linking to git repo? Check existence/absence of files rpm • Is correct version actually installed? yum • Check existence/absence of files. Check file permissions/ownership ls • Is correct version actually running? lsof • Check correct ports exposed netstat • Is correct log format in use? tail • Check cron config grep • Check contents of standalone jar file – e.g. metadata, library versions unzip • Check standard endpoints for e.g. content, status code, response time curl • Check JVM options jps • Check 1 and only 1 process running ps

Compliance Test Example


Reference Implementation

•  “helloworld-service” •  Deployed to production


Rolling Upgrade

•  Safely automate cluster upgrades •  Orchestrated with Fabric


• Check Load Balancer

Exit Pool

• Orchestrate Puppet

Upgrade

• Compliance Test •  Smoke Test •  Feature Test

Test

• Check Load Balancer

Enter Pool

•  exitpool •  stop •  yum remove •  yum install • Wait for manual

testing! •  enterpool

Manual Release

SLIPway


•  Big review of old release processes

•  Simple Lightweight Independent Path Way

•  Simple rules for release process

•  Keep changes independent

•  Focus on reducing cycle time

Simple Rules for SLIPWay

•  Independent •  One application per release •  Backwards-compatible •  No DB/Network/OS changes

•  No manual testing •  Cannot book specific time

•  Feature switch for time •  Ops will service queue within 24 hours

•  Compliance with latest standards


Days

Weeks!

Days


Days

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA !?

Prio

ritis

ed R

elea

se Q

ueue

39


Prio

ritis

ed R

elea

se Q

ueue

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA










Hours Hours

Sim

ple

Rule

s


SLIPway Chat Room


SLIPway Example Release


Freeing up 3 full time employees

0

5

10

15

20

25

Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov

Man

-hou

rs p

er re

leas

e*

man

-hou

rs t

o sh

ip a

rel

ease

95%

reduction in man-hours

GRIPway

SLIPway

SLIPway introduction

-18.5hrs

Hours Spent Shipping Release

0

5

10

15

20

Cycl

e ti

me

in b

usin

ess

days

busi

ness

day

s -8

-2

86% cycle time reduction

GRI

Pway

SLIPway

GRIPway average 2013

GRIPway today

Jan Jan Jul Apr Oct Apr

2014 2013

Jul Oct

Time-to-market Reduction

Making Easy = Right

0

20

40

60

80

100

120

Num

ber

of r

elea

ses

per

mon

th

Thro

ughp

ut x

2.9

190% releases

GRIPway average 2013

SLIPway introduction

GRI

Pway

SL

IPw

ay SLIPway average

GRI

Pway

Jan Jan Jul Apr Oct Apr

2014 2013

Jul

2.9x Releases Per Month

Making Easy = Right

Release Cycle Time Comparison

Making Easy = Right

0

1

2

3

4

5

6

7

8

9

10

GRIPway Avg Cycle Time Days SLIPway Avg Cycle Time Days


Making Easy = Right

7x lower Avg. Cycle Time

0

1

2

3

4

5

6

7

8

9

10



Making Easy = Right


0

1

2

3

4

5

6

7

8

9

10


Commit stage

Acceptance stage

Capacity Testing

Production

Con%nuous Delivery

Automa%c Trigger Manual Trigger

Commit stage

Acceptance stage

Capacity Testing

Production

Con%nuous Delivery

Con%nuous Deployment

Automa%c Trigger Manual Trigger

Commit stage

Acceptance stage

Capacity Testing

Production


Making Easy = Right


21mins

0

1

2

3

4

5

6

7

8

9

10

GRIPway Avg Cycle Time Days SLIPway Avg Cycle Time Days Continuous DEPLOYMENT


Making Easy = Right

7x lower Avg. Cycle Time 30x lower vs SLIPway 200x lower vs GRIPway

21mins

0

1

2

3

4

5

6

7

8

9

10

GRIPway Avg Cycle Time Days SLIPway Avg Cycle Time Days Continuous DEPLOYMENT

Lessons Learned DevOps @ Wotif: Making Easy = Right

53

Lessons Learned


Get the Balance Right Chip Away At The Iceberg The Carrot, Not The Stick Strongly Type Your Discussions Use Semantic Versioning Embrace Legacy

Get the Balance Right

No Standards Too Many Checkpoints


Chip Away at the Iceberg


The Carrot, Not The Stick

•  We tried the stick approach for years –  You’re doing it wrong! –  (But we won’t tell you how

to do it right)

•  Blame games, anger •  No business impetus to

improve things

•  Now we’re trying the carrot approach –  Offer a great new

alternative –  It’s optional! –  You know you want it…

•  Business invested in the process


Strongly Type Your Discussions

•  Choose catchy names – Helps make decisions – Helps climb out of muck

•  Use State Transition tables – State -> Action -> New State – Visualise problems – Decide what to support


E.g. Glassfish Service States


E.g. Lightweight Service States


Use Semantic Versioning

•  Especially for shared libraries


1 . 2 . 3 Major Minor Patch

semver.org

Be Opinionated – Not Arrogant

•  Strong decisions are important •  But you’ll never get it 100% up front


I’m kind of a big

deal

Building Blocks SLIPway •  Simple Rules, Independence

Rolling Upgrade •  Automated Deploy & Test

Compliance Tests •  Test-driven Ops Compatibility

Standards •  The Agreed “Right Thing”


The Right Thing

Making It Easy


Right

Easy


Right

Easy

Thanks for listening! Any questions?

Alexandra Spillane • @ajbw Matt Callanan • @mcallana

slides, etc • mattcallanan.net

Making Easy = Right

Image Attribution •  Creative Commons:

–  Downward Spiral: http://flic.kr/p/67Giiz –  Chasm: https://flic.kr/p/abuWHt –  Wrong way: https://flic.kr/p/cRjc6q –  Silos: https://flic.kr/p/6WueVm –  Testing Ice-cream: http://watirmelon.files.wordpress.com/2012/01/softwaretestingicecreamconeantipattern.png –  Lifting boat: https://flic.kr/p/ddUf4S –  Batch Size: http://dev2ops.org/2012/03/devops-lessons-from-lean-small-batches-improve-flow/ –  Changed Priorities: https://flic.kr/p/debvm –  Stop: https://flic.kr/p/aUEW1D –  Stagnant: https://flic.kr/p/3MvvsH –  Thinking: https://flic.kr/p/6wdLat –  Rubber Bands: https://flic.kr/p/a2XZDB –  Climbing: https://flic.kr/p/4U2BB1 –  Cycling: http://commons.wikimedia.org/wiki/File:Alexander_Vinokourov_2,_London_2012_Time_Trial_-_Aug_2012.jpg –  Indian Traffic: http://frrl.files.wordpress.com/2012/07/logo_indiatraffic.png –  Red Tape: https://flic.kr/p/b7T3JD –  Slipway: https://flic.kr/p/9oQVh8 –  Iceberg: https://flic.kr/p/hV1vwH –  River: https://flic.kr/p/7EMs4T

•  Additional Stock photos: –  iPhotoStock –  Shutterstock

•  Standardisation Theory –  Diagram inspired by: http://www.slideshare.net/spnewman/practical-microservices-yow-2013/56


Documents

Right - YOW! Conferences...Release Anti-Patterns Making Easy = Right 15 Extensive, detailed release documentation Reliance on manual testing to conﬁrm app is correct Explaining why