Embed Size (px)
DESCRIPTION
2012 Velocity London, Presentation by Patrick Debois (@patrickdebois), Damon Edwards (@damonedwards), Gene Kim (@realgenekim), John Willis (@botchagalupe)
Citation preview
1
DevOps Patterns Distilled
Patrick Debois (@patrickdebois)Damon Edwards (@damonedwards)
Gene Kim (@realgenekim)John Willis (@botchagalupe)
Velocity Europe 2012
2Source: Allspaw/Hammond (2009)
3Source: Allspaw/Hammond (2009)
4Source: Allspaw/Hammond (2009)
5Source: Allspaw/Hammond (2009)
6Source: John Jenkins, Amazon.com
Every Company Is An IT Company…
95% of all capital projects have an IT component…
50% of all capital spending is technology-related
We are here…
Where we need to be…
IT is always in the way
(again…)
8
John Allspaw (@allspaw) Patrick Debois (@patrickdebois)
Damon Edwards (@damonedwards)Gene Kim (@realgenekim)
Mike Orzen (@mikeorzen_leanit)John Willis (@botchagalupe)
The DevOps Cookbook (Coming H1 2013)
The First Way:Systems Thinking
The First Way:Systems Thinking (Left To Right)
Understand the flow of work Always seek to increase flow Never unconsciously pass defects downstream Never allow local optimization to cause global
degradation Achieve profound understanding of the system
The Second Way:Amplify Feedback Loops
The Second Way:Amplify Feedback Loops (Right to Left)
Understand and respond to the needs of all customers, internal and external
Shorten and amplify all feedback loops: stop the line when necessary
Create quality at the source Create and embed knowledge where we need it
The Third Way:Culture Of Continual Experimentation & Learning
“Devops Areas”a way to ‘codify’ problems/solutions
OPSDEV
Area 1: Extend delivery to production“think Jez Humble”
Area 1
OPSDEV
Area 2: Extend operations feedback to projectthink “John Allspaw”
Area2
OPSDEV
Area 3: Embed Dev into Opsthink “Adrian Cockcroft”
Area 3
OPSDEV
Area 4: Embed Ops into Devthink “Chris Read”
Area 4
OPSDEV
Area 4: Embed Operations knowledge into Project
Area 2: Extend operations feedback to project
Area 1: Extend delivery to production
Area 3: Embed Projectknowledge into Operations
The Third Way:Culture Of Continual Experimentation & Learning
Foster a culture that rewards: Experimentation (taking risks) and learning from failure Repetition is the prerequisite to mastery
Why? You need a culture that keeps pushing into the danger
zone And have the habits that enable you to survive in the
danger zone
21
Area 1:
Extend Continuous Deliver Into ProductionPatrick Debois
@patrickdebois
22
GOALS
22
Refocus on BusinessView End-To-End
DEV
OPSQA
Business
Big Goal
Practical Goal
Get conversation startedBring the pain forward
23
Step #1 - Re-Establish Trust
Co-location TeamsFace to Face meetings
IRC, Chat, Group feeling
Align Management GoalsHR Policies
24
Step #2 - Understand your bottleneck
•Understand (a) ‘shared’ goal
•Value Stream Mapping
•Bug Reports, Backlog
•Where?
•Tools, Process, People
25
Step #3 - Think Continuous Integration + Infra as Code
•Version Control Everything
•Single Repository of Truth
•One step Dev, Test, Prod Environment build process
“Technology” Focused
26
Step #4 - Think Continuous Delivery
•Extend Release into Prod
•Reduce Technical Debt
•Definition of Done
•Visualize Tasks/Bugs
Mind Shift
27
Step #5 - Integrate other roles in the process
QA
CAB
Security
Management27
28
Devops Anti Patterns
29
Anti Pattern #1 - Config Mgmt = Devops
29
Tools
Process
Culture
30
Especially for John
30
CULTURE
31
Repeat after me
32
Anti-Pattern #2 - Guerrilla Devops
32
Integrate
into the process
Only local changes
don’t have much effect
33
Anti-Pattern #3- Start a separate devops group
33
(Yet Another Silo)
Act as a change agent
34
Anti-Pattern #4 - Silo X takes over
34
If devs take over, they have to (re)-learn production
Symmetry of Ignorance/Arroganc
e
35
Anti-Pattern #5 - Sell it as a buzzword
35
36
Anti-Pattern #6 - Organizational Inertia
Nash Equilibrium - Game Theory
Group experiment:
> 80% people invest: Investors 80$ , other 0$< 80% people invest: Investors -10$ , other 0$
Convergance to invest or not investdepends on
initial group decision
37
Outcomes
37
Robust Technology
Trust People
Business Goal(s)
Shared Process
38
Area 2:
Provide Feedback and VisibilityDamon Edwards
@damonedwards
39
GOAL
Provide feedback and visibility
40
GOAL
Provide feedback and visibility...but why?
41
GOAL
Provide feedback and visibilityto align your organization’s improvement efforts
42
HOW DO YOU ALIGN YOUR ORGANIZATION?
1. Clear goals and operating instructions
2. Shared situational awareness
43
HOW DO YOU CREATE SHARED SITUATIONAL AWARENESS?
44
FOUR TYPES OF DATA YOU NEED
SituationalAwareness
InfrastructureData
BusinessData
ApplicationData
People & Process
Data
45
Step 1: MAKE ALL INFRASTRUCTURE DATA VISIBLE
•Network, Disk I/O, Memory, Utilization, etc...
•Present data in context of the application
•Standardize and extend to all environments
•Create awareness of deviations from normInfrastructure
Data
BusinessData
People & Process
Data
SituationalAwareness
46
STEP 2: MAKE ALL APPLICATION DATA VISIBLE
SituationalAwareness
BusinessData
ApplicationData
People & Process
Data
•Performance, faults, availability, logs, etc...
•Dev takes ownership of instrumenting their applications, but anyone can view or extend
•Enable self-service metric creation (“one line of code”)
• Increase signal, decrease noise
47
SituationalAwareness
InfrastructureData
BusinessData
ApplicationData
STEP 3: BREAK BUSINESS DATA OUT OF IT’S SILO
•Sales, signups, churn, clickstream, etc...
•Make goals explicit (KPIs, one metric that matters)
•Link all other metrics to business metrics
•Empower improvement by showing cause and effect
Key Business Metric
Secondary Business Metric
Technical/Process Metric
My activity
48
People & Process
Data
STEP 4: COLLECT AND VISUALIZE ORGANIZATION & PROCESS DATA
48
•Change activity, quality, cycle time,effectiveness, etc...
•Focus on effectiveness, not efficiency
•Visualize the flow across the entire lifecycle
•Capture change data and enable overlays on any graph
49
USE TO DRIVE CONTINUOUS IMPROVEMENT
SituationalAwareness
InfrastructureData
BusinessData
ApplicationData
Organization & Process
Data
50
“PAINT THE WALLS”
51
Area 3:
Embed Dev into OpsGene Kim
@realgenekim
5252
53
Goals
Shorten and amplify feedback loops
Create knowledge and capabilities where we need it
Ensure that we’re optimizing for the entire system
53
5454
“We found that when we woke up developers at 2am, defects got fixed faster than ever”
Patrick LightbodyFounder/CEO, BrowserMob
55
IT Operations As The Developers’ Best Friend
Tom Limoncelli Patrick Debois Adrian Cockcroft
56
Require That Dev Initially Maintain Their Own Service
56
Source: Tom Limoncelli, Google (Usenix 2012)
57
Test Whether Developers Qualify For IT Operations Resources
Types/frequency of pager alerts
Maturity of monitoring
System architecture review
Release process
Defect counts and severity
Production hygiene
57
Source: Tom Limoncelli, Google (Usenix 2012)
58
Return Fragile Services Back To Dev
58
Source: Tom Limoncelli, Google (Usenix 2012)
59
Integrate Dev Into IT Operations
Integrate Dev into IT Operations escalation processes
Have Dev cross-train IT Operations staff
Have Dev improve the environment
59
60
Eric Passmore
61
Area 4:Embed Ops Into
DevJohn Willis
@botchagalupe
62
Devops Alignment
63
Embed Ops into Dev
64
Why
• Seeing End to End • Sharing the Pain• Operations Andon Cord• Create a Common Language• Educate Dev to Think Like Ops• Flattening Knowledge Chain • Create Patterns of Fault Tolerance • Manage Technical Debt
65
Engagement Models for Embedding
• One Off• Cross Functional Teams• Mercenaries • Specialized Teams• NoOps
66
Design for Operations
• Improve Application• Config files• Instrumentation• logging• Improve Environment• Configuration Management• Immune system (BDD)
67
@cread
• DevOps Facilitator at DRW• London, United Kingdom• www.chris-read.net
68
Institutionalize IT Operations Knowledge
• Building Reusable IT Operations• Embedded Operations• Design• Architecture• Controls• Monitoring• Deployment
69
Break Things Early And Often
“Do painful things more frequently, so you can make it less painful… We don’t get pushback from Dev, because they know it makes rollouts smoother.”
-- Adrian Cockcroft, Architect, Netflix
70
You Don’t Choose Chaos Monkey…Chaos Monkey Chooses You
71
Outcomes
• Improved Operational Readiness• Improved Deployment Success• Decreased Cycle Time
72
Anti Patterns
• Embedding is a Social Experiment• Understand Motivation• Previous Relationships
7373
74
Why Do I Think This Is An Important
Problem?
Help The Business Win…
With Support From Your Peers…
And Do More With Less Effort…
78
When IT Fails: A Business Novel and The DevOps Cookbook
Coming January 15, 2013 and Q1 2013
“The lessons in When IT Fails might just save your business if IT fails for you. Every IT executive should share this book with their business peers.” -James Turnbull, VP Operations, Puppet Labs and author of “Pro Puppet”
“The greatest IT management book of our generation.” –Branden Williams, CTO Marketing, RSA
“This book will have a profound effect on IT, just as The Goal did for manufacturing.’ - Jez Humble, co-author of the Jolt award-winning book Continuous Delivery, and Principal at ThoughtWorks Studios.
Our Mission: Positively Impact The Lives Of One Million IT Workers By 2017
For these slides, the “Top 10 Things You Need To Know About DevOps,” Rugged DevOps resources, and updates on the books:
Or text “[email_address] 75271” to +1 (858) 598-3980
Or signup at: http://www.instantcustomer.com/go/75271
Or email [email protected]
