2012 Velocity London: DevOps Patterns Distilled

1

DevOps Patterns Distilled

Patrick Debois (@patrickdebois)Damon Edwards (@damonedwards)

Gene Kim (@realgenekim)John Willis (@botchagalupe)

Velocity Europe 2012

2Source: Allspaw/Hammond (2009)

3Source: Allspaw/Hammond (2009)

4Source: Allspaw/Hammond (2009)

5Source: Allspaw/Hammond (2009)

6Source: John Jenkins, Amazon.com

Every Company Is An IT Company…

95% of all capital projects have an IT component…

50% of all capital spending is technology-related

We are here…

Where we need to be…

IT is always in the way

(again…)

8

John Allspaw (@allspaw) Patrick Debois (@patrickdebois)

Damon Edwards (@damonedwards)Gene Kim (@realgenekim)

Mike Orzen (@mikeorzen_leanit)John Willis (@botchagalupe)

The DevOps Cookbook (Coming H1 2013)

The First Way:Systems Thinking

The First Way:Systems Thinking (Left To Right)

Understand the flow of work Always seek to increase flow Never unconsciously pass defects downstream Never allow local optimization to cause global

degradation Achieve profound understanding of the system

The Second Way:Amplify Feedback Loops

The Second Way:Amplify Feedback Loops (Right to Left)

Understand and respond to the needs of all customers, internal and external

Shorten and amplify all feedback loops: stop the line when necessary

Create quality at the source Create and embed knowledge where we need it

The Third Way:Culture Of Continual Experimentation & Learning

“Devops Areas”a way to ‘codify’ problems/solutions

OPSDEV

Area 1: Extend delivery to production“think Jez Humble”

Area 1

OPSDEV

Area 2: Extend operations feedback to projectthink “John Allspaw”

Area2

OPSDEV

Area 3: Embed Dev into Opsthink “Adrian Cockcroft”

Area 3

OPSDEV

Area 4: Embed Ops into Devthink “Chris Read”

Area 4

OPSDEV

Area 4: Embed Operations knowledge into Project

Area 2: Extend operations feedback to project

Area 1: Extend delivery to production

Area 3: Embed Projectknowledge into Operations

The Third Way:Culture Of Continual Experimentation & Learning

Foster a culture that rewards: Experimentation (taking risks) and learning from failure Repetition is the prerequisite to mastery

Why? You need a culture that keeps pushing into the danger

zone And have the habits that enable you to survive in the

danger zone

21

Area 1:

Extend Continuous Deliver Into ProductionPatrick Debois

@patrickdebois

22

GOALS

22

Refocus on BusinessView End-To-End

DEV

OPSQA

Business

Big Goal

Practical Goal

Get conversation startedBring the pain forward

23

Step #1 - Re-Establish Trust

Co-location TeamsFace to Face meetings

IRC, Chat, Group feeling

Align Management GoalsHR Policies

24

Step #2 - Understand your bottleneck

•Understand (a) ‘shared’ goal

•Value Stream Mapping

•Bug Reports, Backlog

•Where?

•Tools, Process, People

25

Step #3 - Think Continuous Integration + Infra as Code

•Version Control Everything

•Single Repository of Truth

•One step Dev, Test, Prod Environment build process

“Technology” Focused

26

Step #4 - Think Continuous Delivery

•Extend Release into Prod

•Reduce Technical Debt

•Definition of Done

•Visualize Tasks/Bugs

Mind Shift

27

Step #5 - Integrate other roles in the process

QA

CAB

Security

Management27

28

Devops Anti Patterns

29

Anti Pattern #1 - Config Mgmt = Devops

29

Tools

Process

Culture

30

Especially for John

30

CULTURE

31

Repeat after me

32

Anti-Pattern #2 - Guerrilla Devops

32

Integrate

into the process

Only local changes

don’t have much effect

33

Anti-Pattern #3- Start a separate devops group

33

(Yet Another Silo)

Act as a change agent

34

Anti-Pattern #4 - Silo X takes over

34

If devs take over, they have to (re)-learn production

Symmetry of Ignorance/Arroganc

e

35

Anti-Pattern #5 - Sell it as a buzzword

35

36

Anti-Pattern #6 - Organizational Inertia

Nash Equilibrium - Game Theory

Group experiment:

> 80% people invest: Investors 80$ , other 0$< 80% people invest: Investors -10$ , other 0$

Convergance to invest or not investdepends on

initial group decision

37

Outcomes

37

Robust Technology

Trust People

Business Goal(s)

Shared Process

38

Area 2:

Provide Feedback and VisibilityDamon Edwards

@damonedwards

39

GOAL

Provide feedback and visibility

40

GOAL

Provide feedback and visibility...but why?

41

GOAL

Provide feedback and visibilityto align your organization’s improvement efforts

42

HOW DO YOU ALIGN YOUR ORGANIZATION?

1. Clear goals and operating instructions

2. Shared situational awareness

43

HOW DO YOU CREATE SHARED SITUATIONAL AWARENESS?

44

FOUR TYPES OF DATA YOU NEED

SituationalAwareness

InfrastructureData

BusinessData

ApplicationData

People & Process

Data

45

Step 1: MAKE ALL INFRASTRUCTURE DATA VISIBLE

•Network, Disk I/O, Memory, Utilization, etc...

•Present data in context of the application

•Standardize and extend to all environments

•Create awareness of deviations from normInfrastructure

Data

BusinessData

People & Process

Data

SituationalAwareness

46

STEP 2: MAKE ALL APPLICATION DATA VISIBLE

SituationalAwareness

BusinessData

ApplicationData

People & Process

Data

•Performance, faults, availability, logs, etc...

•Dev takes ownership of instrumenting their applications, but anyone can view or extend

•Enable self-service metric creation (“one line of code”)

• Increase signal, decrease noise

47

SituationalAwareness

InfrastructureData

BusinessData

ApplicationData

STEP 3: BREAK BUSINESS DATA OUT OF IT’S SILO

•Sales, signups, churn, clickstream, etc...

•Make goals explicit (KPIs, one metric that matters)

•Link all other metrics to business metrics

•Empower improvement by showing cause and effect

Key Business Metric

Secondary Business Metric

Technical/Process Metric

My activity

48

People & Process

Data

STEP 4: COLLECT AND VISUALIZE ORGANIZATION & PROCESS DATA

48

•Change activity, quality, cycle time,effectiveness, etc...

•Focus on effectiveness, not efficiency

•Visualize the flow across the entire lifecycle

•Capture change data and enable overlays on any graph

49

USE TO DRIVE CONTINUOUS IMPROVEMENT

SituationalAwareness

InfrastructureData

BusinessData

ApplicationData

Organization & Process

Data

50

“PAINT THE WALLS”

51

Area 3:

Embed Dev into OpsGene Kim

@realgenekim

5252

53

Goals

Shorten and amplify feedback loops

Create knowledge and capabilities where we need it

Ensure that we’re optimizing for the entire system

53

5454

“We found that when we woke up developers at 2am, defects got fixed faster than ever”

Patrick LightbodyFounder/CEO, BrowserMob

55

IT Operations As The Developers’ Best Friend

Tom Limoncelli Patrick Debois Adrian Cockcroft

56

Require That Dev Initially Maintain Their Own Service

56

Source: Tom Limoncelli, Google (Usenix 2012)

57

Test Whether Developers Qualify For IT Operations Resources

Types/frequency of pager alerts

Maturity of monitoring

System architecture review

Release process

Defect counts and severity

Production hygiene

57

Source: Tom Limoncelli, Google (Usenix 2012)

58

Return Fragile Services Back To Dev

58

Source: Tom Limoncelli, Google (Usenix 2012)

59

Integrate Dev Into IT Operations

Integrate Dev into IT Operations escalation processes

Have Dev cross-train IT Operations staff

Have Dev improve the environment

59

60

Eric Passmore

61

Area 4:Embed Ops Into

DevJohn Willis

@botchagalupe

62

Devops Alignment

63

Embed Ops into Dev

64

Why

• Seeing End to End • Sharing the Pain• Operations Andon Cord• Create a Common Language• Educate Dev to Think Like Ops• Flattening Knowledge Chain • Create Patterns of Fault Tolerance • Manage Technical Debt

65

Engagement Models for Embedding

• One Off• Cross Functional Teams• Mercenaries • Specialized Teams• NoOps

66

Design for Operations

• Improve Application• Config files• Instrumentation• logging• Improve Environment• Configuration Management• Immune system (BDD)

67

@cread

• DevOps Facilitator at DRW• London, United Kingdom• www.chris-read.net

68

Institutionalize IT Operations Knowledge

• Building Reusable IT Operations• Embedded Operations• Design• Architecture• Controls• Monitoring• Deployment

69

Break Things Early And Often

“Do painful things more frequently, so you can make it less painful… We don’t get pushback from Dev, because they know it makes rollouts smoother.”

-- Adrian Cockcroft, Architect, Netflix

70

You Don’t Choose Chaos Monkey…Chaos Monkey Chooses You

71

Outcomes

• Improved Operational Readiness• Improved Deployment Success• Decreased Cycle Time

72

Anti Patterns

• Embedding is a Social Experiment• Understand Motivation• Previous Relationships

7373

74

Why Do I Think This Is An Important

Problem?

Help The Business Win…

With Support From Your Peers…

And Do More With Less Effort…

78

When IT Fails: A Business Novel and The DevOps Cookbook

Coming January 15, 2013 and Q1 2013

“The lessons in When IT Fails might just save your business if IT fails for you. Every IT executive should share this book with their business peers.” -James Turnbull, VP Operations, Puppet Labs and author of “Pro Puppet”

“The greatest IT management book of our generation.” –Branden Williams, CTO Marketing, RSA

“This book will have a profound effect on IT, just as The Goal did for manufacturing.’ - Jez Humble, co-author of the Jolt award-winning book Continuous Delivery, and Principal at ThoughtWorks Studios.

Our Mission: Positively Impact The Lives Of One Million IT Workers By 2017

For these slides, the “Top 10 Things You Need To Know About DevOps,” Rugged DevOps resources, and updates on the books:

Or text “[email_address] 75271” to +1 (858) 598-3980

Or signup at: http://www.instantcustomer.com/go/75271

Or email genek@realgenekim.me