Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually

DevOps @ Wotif

Making Easy = Right

Alexandra Spillane • @ajbwMatt Callanan • @mcallana

RightEasy

Outline

• What is “The Right Thing”?• How do we Make it Easy?• Lessons Learned

Making Easy = Right 3

What is “The Right Thing”?DevOps @ Wotif: Making Easy = Right


“The Right Thing”


Collaborate!

Automate!

But… The Dev<->Ops Chasm


What Were we Doing Wrong?


Downward Spiral of Manual Activity


Entrenched Silos


High Release Overheads


Huge Batch Sizes


Poor Visibility


Calendar of Doom


Unpredictable Prioritisation


Superstitious Gatekeeping


just in case

Stagnating Applications


Manual Deployment Anti-Patterns


Extensive, detailed release documentation

Reliance on manual testing to confirm app is correct

Explaining why deployment is going wrong on release day

Frequent corrections to release process during release

Environments that differ in their configuration

Releases that take more than a few minutes to perform

Releases that are unpredictable in their outcome

Cycle Time measured in

weeks/months

Application Pipelines Ops

Days

Weeks!

Days

Staging Load Test

Production

Days

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA !?

Prio

ritis

ed R

elea

se Q

ueue

Let’s Migrate to Microservices!

• Explosion of manual effort


AB

C

D

E F

A

B1C1 GC2 H I

B2D1 J

D2 KL1 ML2 N

EF1 O P

F2 Q RST

HeavyWeight to LightWeight

Glassfish• Feature overhead• Encouraged manual config• Complex/Slow deployment

DropWizard• Trimmed down to basics• Standard config files• Simple/Fast deployment


But… Combinatorial Explosion


Unpredictability Slows You Down


Moving from wrong to right


Wrong

Right

Reduce Cycle Time

Making Easy = Right

Flexibility vs Predictability

• Freedom is Great for Innovation• Consistency is Essential For Speed• How do we promote a culture of freedom

and responsibility but still provide predictability?

Flexible Predictable


Standardisation


Standardised

Freedom

• Standardise on how services are deployed and how they communicate.

• Be flexible about their contents.

Defining Standards

• Discussions– One-on-one– IM– Emails– Meetings

• Incentivise input• Confluence– Application Deployment

Standards 1.0


Example Standards


Logging• Log file locations,

filenames

Log rotate Directories/Files• Ownership,

permissions

Puppet

Supervisord config.yaml• Filename, location

Metrics JVM settings

Network• Ports• Firewall rules

Cron Endpoints• Healthchecks, status,

metrics

SSL• Certs, keystores

RPM• Versioning• Packaging

init.d scripts• Sub-commands

Versioning Migration Notes

Example Standards


Future Considerations


Standards Lifecycle


Common Ground


How do we Make it Easy?DevOps @ Wotif: Making Easy = Right


Automated Verification

Making Easy = Right

Compliance Test SuiteBring Operations into DevelopmentFast Operational FeedbackTest Driven Operational CompatibilityBackwards Compatibility

Ops Testing with Fabric


•Does rpm have correct metadata linking to git repo? Check existence/absence of filesrpm•Is correct version actually installed?yum•Check existence/absence of files. Check file permissions/ownershipls• Is correct version actually running?lsof•Check correct ports exposednetstat • Is correct log format in use?tail•Check cron configgrep•Check contents of standalone jar file – e.g. metadata, library versionsunzip•Check standard endpoints for e.g. content, status code, response timecurl•Check JVM optionsjps•Check 1 and only 1 process runningps

Compliance Test Example


Reference Implementation

• “helloworld-service”• Deployed to production


Rolling Upgrade

• Safely orchestrate cluster upgrades – automated deployment & testing

• Initially interactive


• Check Load Balancer

Exit Pool

• Orchestrate Puppet

Upgrade

• Compliance Test• Smoke Test• Feature Test

Test

• Check Load Balancer

Enter Pool

• exitpool• stop• yum remove• yum install• Wait for manual

testing!• enterpool

Manual Release

SLIPway


• Big review of old release processes– Focus on reducing cycle

time• Simple Lightweight

Independent Path Way• Simple rules for release

process• Keep changes

independent

Simple Rules for SLIPWay

• Independent• One application per release• Backwards-compatible • No DB/Network/OS changes

• No manual testing• Cannot book specific time

• Feature switch for time• Ops will service queue within 24 hours

• Compliance with latest standards


42Making Easy = Right


43Making Easy = Right


Days

Weeks!

Days

Staging Load Test

Production

Days

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA !?

Prio

ritis

ed R

elea

se Q

ueue

45


Prio

ritis

ed R

elea

se Q

ueue

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Dev QA

Staging Load Test Production









HoursHours

Sim

ple

Rule

s


SLIPway Chat Room


SLIPway Example Release


Freeing up 3 full time employees

0

5

10

15

20

25

Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov

Man

-hou

rs p

er re

leas

e*

man

-hou

rs t

o sh

ip a

rele

ase 95%

reduction in man-hours

GRIPway

SLIPway

SLIPway introduction

-18.5hrs

Hours Spent Shipping Release

0

5

10

15

20

Cycl

e ti

me

in b

usin

ess d

ays

busi

ness

day

s-8

-2

86%cycle time reduction

SLIPway

GRIPwayaverage 2013

GRIPway today

Jan JanJulApr Oct Apr

20142013

Jul Oct

Time-to-market Reduction

Making Easy = Right

2014

0

20

40

60

80

100

120

Num

ber o

f rel

ease

s pe

r m

onth

Thro

ughp

ut x

2.9190%

releases

GRIPway average 2013

SLIPway introduction

GRI

Pway

SLIP

way

SLIPway average

GRI

Pway

Jan JanJulApr Oct Apr

20142013

Jul

2.9x Releases Per Month

Making Easy = Right

Release Cycle Time Comparison

Making Easy = Right

7x lower Avg. Cycle Time

0

1

2

3

4

5

6

7

8

9

10

GRIPway Avg Cycle Time Days SLIPway Avg Cycle Time Days

Commit stage

Acceptance stage

Capacity Testing

Production

Continuous Delivery

Continuous Deployment

Automatic Trigger Manual Trigger

Commit stage

Acceptance stage

Capacity Testing

Production

Release Cycle Time Comparison

Making Easy = Right

7x lower Avg. Cycle Time 30x lower vs SLIPway200x lower vs GRIPway

21mins

0

1

2

3

4

5

6

7

8

9

10

GRIPway Avg Cycle Time Days SLIPway Avg Cycle Time Days Continuous DEPLOYMENT

Upward Spiral


Lessons LearnedDevOps @ Wotif: Making Easy = Right

57

Lessons Learned


Get the Balance RightChip Away At The IcebergMigration NotesThe Carrot, Not The StickStrongly Type Your DiscussionsUse Semantic VersioningBe Opinionated – Not ArrogantWarranty PeriodEmbrace Legacy

Get the Balance Right- Optimise for Speed

No Rules Too Many Rules


Chip Away at the Iceberg


Migration Notes

• Annotate changes since previous standards – E.g. “(since v2.3)”

• Each standard version has migration notes from previous version

• Can upgrade between several versions by following migration notes nav bar


1.0 2.0 2.1 2.2 2.3

The Carrot, Not The Stick

• We tried the stick approach for years– You’re doing it wrong!– (But we won’t tell you how

to do it right)

• Blame games, anger• No business impetus to

improve things

• Then we tried the carrot approach– Offer a great new

alternative– It’s optional!– You know you want it…

• Business invested in the process


Strongly Type Your Discussions

• Choose catchy names– Helps make decisions– Helps climb out of muck

• Use State Transition tables– State -> Action -> New State– Visualise problems– Decide what to support


State Transitions – Current


State Transitions – Future


Use Semantic Versioning

• Especially for shared libraries


1 . 2 . 3Major Minor Patch

semver.org

Be Opinionated – Not Arrogant

• Strong decisions are important• But you’ll never get it 100% up front


I’m kind of a big

deal

Warranty Period

• Can’t get standards 100% right

• But need to lock them down

• Two-week warranty period

• Trial in range of different applications


Small Batch Size for Standards

• Minor updates every 1-2 months

• Fast feedback• Enable innovation• Easier to update


Embrace Legacy

• You’re creating legacy• Make migration easy• Incentivise catch-up


Building BlocksSLIPway• Simple Rules, Independence

Rolling Upgrade• Automated Deploy & Test

Compliance Tests• Test-driven Ops Compatibility

Standards• The Agreed “Right Thing”


The Right Thing

Making It Easy

Key Takeaways• Make the interface between teams (dev/ops)

consistent and predictable• Cultural choices (agreement) > tool choices• Prioritise operational feedback to developers• Faster releases leads to smaller batch size leads

to lower risk leads to happier customers• Start with interactive automation• Independent releases with no dependencies• Incentivise legacy catch-up through operational

compliance to latest agreed standards


Consistent and predictable interface between teams (dev/ops)

Cultural choices (agreement) > tool choices

Prioritise fast operational feedback to developers

Faster releases à smaller batch size à lower risk à happier customersBuild trust with interactive automation

Decoupled releases – independent& dependency-‐freeà speed

Incentivise legacy catch-‐up


RightEasy

Thanks for listening!Any questions?

Alexandra Spillane • @ajbwMatt Callanan • @mcallana

slides, etc • mattcallanan.net

Making Easy = Right

Image AttributionImage

“Downward Spiral” (http://flic.kr/p/67Giiz) by Chad K is licensed under CC BY 2.0 (https://creativecommons.org/licenses/by/2.0/)

“Angels Landing from the Deertrap Mountain Trail” (Chasm -‐ https://flic.kr/p/abuWHt) by Zion National Park is licensed under CC BY 2.0 (https://creativecommons.org/licenses/by/2.0/)

“WRONG WAY” (https://flic.kr/p/cRjc6q) by David Goehring is licensed under CC BY 2.0 (https://creativecommons.org/licenses/by/2.0/)

“Old Gravel Silos, Buffalo Bayou, East of Jensen, Houston, Texas 0906091555BW” (https://flic.kr/p/6WueVm) by Patrick Feller is licensed under CC BY 2.0 (https://creativecommons.org/licenses/by/2.0/)

“Fisher men lifting a boat in Bangladesh. Photo by Finn Thilsted” (https://flic.kr/p/ddUf4S) by WorldFish is licensed under CC BY NC ND 2.0 (https://creativecommons.org/licenses/by-‐nc-‐nd/2.0)

“Poor visibility from Kelud eruption 14 February 2014, Yogyakarta” (http://bit.ly/1N3Lf7A) by Aldnonymous is licensed under BY SA 3.0 (https://creativecommons.org/licenses/by-‐sa/3.0/deed.en)

“Changed priorities ahead” (https://flic.kr/p/debvm) by Peter Reed is licensed under CC BY NC 2.0 (https://creativecommons.org/licenses/by-‐nc/2.0)

“Stop Sign” (https://flic.kr/p/aUEW1D) by DonkeyHotey is licensed under CC BY 2.0 (https://creativecommons.org/licenses/by/2.0/)

“green scum” (https://flic.kr/p/3MvvsH) by M&R Glasgow is licensed under CC BY ND 2.0 (https://creativecommons.org/licenses/by-‐nd/2.0)

“Thinking… please wait “ (https://flic.kr/p/6wdLat) by Karola Riegler is licensed under CC BY ND 2.0 (https://creativecommons.org/licenses/by-‐nd/2.0)

“Elastic“ (https://flic.kr/p/a2XZDB) by Chris Stevenson is licensed under CC BY NC 2.0 (https://creativecommons.org/licenses/by-‐nc/2.0)

“A Garden Of Climbs” (https://flic.kr/p/4U2BB1) by Jasen Miller is licensed under CC BY 2.0 (https://creativecommons.org/licenses/by/2.0/)

“Alexander Vinokourov competing in the London 2012 Men's Olympic Time Trial” (http://bit.ly/1QndiBd) by Diliff is licensed under BY SA 3.0 (https://creativecommons.org/licenses/by-‐sa/3.0/)

“Upward Spiral” (https://flic.kr/p/noFia) by Clint Vigil by is licensed under CC BY NC 2.0 (https://creativecommons.org/licenses/by-‐nc/2.0)

“Rivers of Humanity” (Indian Traffic -‐ https://flic.kr/p/3yMSMW) by pangalactic gargleblaster and the heart of gold is licensed under CC BY NC ND 2.0 (https://creativecommons.org/licenses/by-‐nc-‐nd/2.0)

“Red Tape“ (https://flic.kr/p/b7T3JD) by Free Press/ Free Press Action Fund is licensed under BY NC SA 2.0 (https://creativecommons.org/licenses/by-‐nc-‐sa/2.0)

“To the Right A Bit” (Slipway -‐ https://flic.kr/p/9oQVh8) by tiffany terry is licensed under CC BY NC ND 2.0 (https://creativecommons.org/licenses/by-‐nc-‐nd/2.0)

“Iconic iceberg sail-‐by” (https://flic.kr/p/hV1vwH) by Visit Greenland is licensed under CC BY NC ND 2.0 (https://creativecommons.org/licenses/by-‐nc-‐nd/2.0)

“Luckiamute Falls” (https://flic.kr/p/7EMs4T) by Ian Sane is licensed under CC BY 2.0 (https://creativecommons.org/licenses/by/2.0/)

“Three Keys” (https://openclipart.org/detail/219938/water) by [email protected] is licensed under unlimited-‐commercial-‐use (https://openclipart.org/unlimited-‐commercial-‐use-‐clipart)

Presentation includes stock images used under license from Shutterstock.com and iStockPhoto.com. Authors include: Stepanek Photography/Shutterstock.com, skvoor/Shutterstock.com, Michael D Brown/Shutterstock.com, argus/Shutterstock.com, Sergey Nivens/Shutterstock.com, Anze Mulec/Shutterstock.com, Jaz_czc/Shutterstock.com, DenisNata/Shutterstock.com, Arnaud Weisser/Shutterstock.com, iStock.com/gehringj

Standardisation Theory Diagram inspired by Sam Newman: http://www.slideshare.net/spnewman/practical-‐microservices-‐yow-‐2013/56

Documents

Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually