Devops Metrics

DEVOPS METRICS

ANDREW SHAFER - PATRICK DEBOIS - ISRAEL GATThursday 16 June 2011

http://www.vectronicsappleworld.com/collection/articlepics/joysticks/snap23.jpg

Patrick Debois

Jedi

@patrickdebois

http://jedi.be/blog

Thursday 16 June 2011



http://jedi.be/blog

http://jedi.be/blog

Israel Gat

Cutter Consortium

@agileexec

http://theagileexecutive.com


http://agileexec.com

http://agileexec.com

Andrew Shafer - Rabble Rouser - Cloudscaling

@littleidea - http://stochasticresonance.wordpress.comThursday 16 June 2011

http://stochasticresonance.com

http://stochasticresonance.com

http://www.flickr.com/photos/quinnanya/3588989219




http://www.flickr.com/photos/zen/3233023066/


To Comparehttp://www.flickr.com/photos/thebusybrain/2492945625/


http://www.flickr.com/photos/thebusybrain/2492945625/

http://www.flickr.com/photos/thebusybrain/2492945625/

http://www.flickr.com/photos/topshampatti/219953012/




ToUnderstand


To Protect


To Improvehttp://www.flickr.com/photos/vizzzual-dot-com/2655969483/


http://www.flickr.com/photos/vizzzual-dot-com/2655969483/

http://www.flickr.com/photos/vizzzual-dot-com/2655969483/

Describing a Metric

http://www.flickr.com/photos/chandramarsono/4324373384/




Describing a Metric

•basis of measurement





Describing a Metric

•basis of measurement•assumptions





Describing a Metric

•basis of measurement•assumptions•level and usage





Describing a Metric

•basis of measurement•assumptions•level and usage•desired trend





Describing a Metric

•basis of measurement•assumptions•level and usage•desired trend•when to use





Describing a Metric

•basis of measurement•assumptions•level and usage•desired trend•when to use•when to stop





Describing a Metric

•basis of measurement•assumptions•level and usage•desired trend•when to use•when to stop•how to game





Describing a Metric

•basis of measurement•assumptions•level and usage•desired trend•when to use•when to stop•how to game•warnings





Beware of Vanity Metrics

UNACTIONABLE

INACCESSIBLE

UNAUDITABLEhttp://www.flickr.com/photos/centralasian/5829549813/


http://www.flickr.com/photos/centralasian/5829549813/

http://www.flickr.com/photos/centralasian/5829549813/

Red is good , Green is worthless

• http://www.reliableplant.com/Read/21305/lean-metrics-red-is-good,-green-is-worthless


http://www.reliableplant.com/Read/21305/lean-metrics-red-is-good,-green-is-worthless

http://www.reliableplant.com/Read/21305/lean-metrics-red-is-good,-green-is-worthless


“In truth, a good case could be made that if your knowledge is meagre and unsatisfactory, the last thing in the world you should do is make measurements; the chance is negligible that you will measure the right things accidentally.”

http://stochasticresonance.wordpress.com/2008/08/03/miles-per-gallon/

George Miller




http://livepage.apple.com/

http://livepage.apple.com/

What we can learn from monitoringhttp://www.experimentalistsanonymous.com/stuff/paralleluniverse/images/2edmoneyshot.jpg


http://www.experimentalistsanonymous.com/stuff/paralleluniverse/images/2edmoneyshot.jpg

http://www.experimentalistsanonymous.com/stuff/paralleluniverse/images/2edmoneyshot.jpg

Level 0 - Individual Resource Thursday 16 June 2011

Availability

Network, Memory, CPUProcesses, Port,SSH Login


Availability



Capacity



Capacity



Health

http://www.geocities.jp/nikonikocalendar/index_en.html

Thresholds

http://www.smiley-faces.org/wallpaper/smiley-face-wallpaper-001.jpg






Health


Thresholds







http://www.slideshare.net/krivitsky/agile-metrics-presentation-654259

Measurement Metric

Diagnostic

Pete Behrens, Trail Ridge Consulting

Does itcontribute to

thebottom line?

Metric versus Diagnostic

Yes

No




Level 1 - Services LevelThursday 16 June 2011

Performance

SSH, HTTP, NTP


Performance

SSH, HTTP, NTP

# Servers/Admin


Performance

# Servers/Admin

# Line of Code/Developer

SSH, HTTP, NTP


“Gameable”

http://kithicor.org/wp/wp-content/uploads/2010/01/gamerEvolution.jpg




Level 2 - Team LevelThursday 16 June 2011

Aggregation

WEB1

LB-WEB

WEB2

WEB3


Aggregation

DEV

DEV Project Team

DEV DEV


Burn up chart


Team Utilization


“Velocity also has a direction component”

http://www.randomstew.com/wp-content/uploads/mooneye.jpg




What about efficiency?http://www.flickr.com/photos/jdiggans/92981578/


http://www.flickr.com/photos/jdiggans/92981578/

http://www.flickr.com/photos/jdiggans/92981578/

http://sievertschreiber.files.wordpress.com/2009/12/good-code-is-measured-in-wtf-per-minute.jpg




Incentives & Behaviorshttp://www.flickr.com/photos/turtlemadness/2439050717/


http://www.flickr.com/photos/turtlemadness/2439050717/

http://www.flickr.com/photos/turtlemadness/2439050717/

http://www.xgn.nl/images/upload/20081020225036.jpg http://www.kawaiikakkoiisugoi.com/wp-content/uploads/2011/03/pacman-screen.jpg

OperationsDevelopment

Level 3 - Integration LevelThursday 16 June 2011

http://www.xgn.nl/images/upload/20081020225036.jpg

http://www.xgn.nl/images/upload/20081020225036.jpg

http://www.kawaiikakkoiisugoi.com/wp-content/uploads/2011/03/pacman-screen.jpg

http://www.kawaiikakkoiisugoi.com/wp-content/uploads/2011/03/pacman-screen.jpg

Integration

DBWEB APP


Collaboration

# of deploys/ day

OPSDEV

http://www.sage.org/lists/sage-members-archive/2001/msg01208.html http://www.datacenterknowledge.com/archives/2009/12/30/how-many-servers-can-one-admin-manage/ http://fc04.deviantart.net/fs70/f/2010/108/2/3/Pac_Man_Widescreen_by_molotov_arts.jpg

QA


http://www.sage.org/lists/sage-members-archive/2001/msg01208.html

http://www.sage.org/lists/sage-members-archive/2001/msg01208.html

http://www.datacenterknowledge.com/archives/2009/12/30/how-many-servers-can-one-admin-manage/

http://www.datacenterknowledge.com/archives/2009/12/30/how-many-servers-can-one-admin-manage/

http://fc04.deviantart.net/fs70/f/2010/108/2/3/Pac_Man_Widescreen_by_molotov_arts.jpg

http://fc04.deviantart.net/fs70/f/2010/108/2/3/Pac_Man_Widescreen_by_molotov_arts.jpg

Counting Collaboration

• http://www.nature.com/news/2010/101013/full/news.2010.538.html

More interaction doesn’t mean a better party


http://www.nature.com/news/2010/101013/full/news.2010.538.html

http://www.nature.com/news/2010/101013/full/news.2010.538.html

Measuring Collaboration?

http:///www.healthylarimer.org/documents/PlayNice.ppt“Play Nice Study”Thursday 16 June 2011

http://www.healthylarimer.org/documents/PlayNice.ppt

http://www.healthylarimer.org/documents/PlayNice.ppt

Levels Collaboration

Cooperation

Coordination

Coalition

Collaboration


Cooperation

provide information

somewhat defined roles

formal communication

decisions made independently


Coordination•share information & resources

•defined roles

•frequent communication

•some shared decision making


Coalition

•share ideas

•share resources

•frequent & prioritized communication

•all members have a vote on decisions


Collaboration

•members belong to one system

•frequent communication characterized by mutual trust

•consensus on all decisions


http://www.au.af.mil/au/awc/awcgate/navy/onr_metrics_on_collab.pdf

Knowledge sharing




Team Resilience

Truck Factor10th Floor Test

http://www.kitchensoap.com/

Resilience should be a 4th management objective,

alongside Better/Faster/Cheaper John Allspaw

The number of people on your team who have to be hit with a truck before the project is in serious trouble

to take a random machine in your infrastructure , drop it from the 10th floor, and be able to restore your infrastructure to working within 5-10 minute


Black Hole Theory

http://www.collaborate.com/blog/2010/01/28/musing-metrics-collaboration

Absorbs light, to see the effects you need to look at nearby

objects




http://communityandbusiness.groupsite.com/group/Collaboration20/post/measuring-the-value-of-collaboation

• Saving time or Money (tangible)

• Increase Quality (tangible... but less so)

• Innovating and/or providing decision support (tangible but less than quality)

• Easing access to and interactions with subject-matter experts (intangible)

Quest for proof




http://socialconsulting.wordpress.com/2009/02/24/measuring-collaboration/

Quality Improvement

Tune existing serversAdd more servers

Process Improvement

Easier More Difficult!!

More vs Better Widgets




Collaboration Challenge

“Maximize

Team Utility”


Level 4 - Business LevelThursday 16 June 2011

AHA KA-CHING!


http://www.flickr.com/photos/hanekomu/4481674104/

Return on Investment




Inventory (WIP)


Time to Market

http://stefanroock.wordpress.com/2010/03/02/kanban-definition-of-lead-time-and-cycle-time/




http://museumdosgames.blogspot.com/2010/04/game-over.html

Mean Time Between Failure




Mean Time Between Resolution


Technical Debt


Leaks


Debt Metaphornot all debt is created equal

http://www.flickr.com/photos/shoot-art/4536509717/




Customer Responsiveness

Jim HighsmithThursday 16 June 2011

Types of Tech debt

http://forums.construx.com/blogs/stevemcc/archive/2007/11/01/technical-debt-2.aspx

“oops” “yeah I know”

“we’ll fix it the next minor revision”

“we’ll fix it the next major revision”

unintentional intentional

short term (tactical) long term (strategical)




Gartner

• http://www.castsoftware.com/resources/materials/analyst/Gartner-How_to_Monetize_Application_Technical_Debt.pdf


http://www.castsoftware.com/resources/materials/analyst/Gartner-How_to_Monetize_Application_Technical_Debt.pdf

http://www.castsoftware.com/resources/materials/analyst/Gartner-How_to_Monetize_Application_Technical_Debt.pdf

The Vicious Cycle of Technical Debt

1

(More) Relentless Pressure

Take Technical

Debt

Fail to Pay Debt Back

Neglect Maintenance

Technical Debt

Accrues

Diminished Dev Velocity

Vicious Cycle of Technical Debt

Israel Gathttp://theagileexecutive.com/2010/09/20/how-to-break-the-vicious-cycle-of-technical-debt/


http://theagileexecutive.com/2010/09/20/how-to-break-the-vicious-cycle-of-technical-debt/

http://theagileexecutive.com/2010/09/20/how-to-break-the-vicious-cycle-of-technical-debt/

Symptoms - Measurable

• Poor Customer Responsiveness

• Long Delivery Times

• Late deliveries

• Lots of defects (Toxic Code)

• Rising Development costs

• Frustrated & Poor performing Teams

http://www.theagileengineer.com/public/Home/Home_files/TechnicalDebt_published.pdf




Secret Sauce

http://radar.oreilly.com/archives/2007/10/Ops-ROI-jesserobbins.html




The Team Knows

http://kingsinsight.com/2010/07/31/ask-the-team-about-technical-debt-the-creeping-doom-graph/




http://farm4.static.flickr.com/3167/2769773425_16c235dc28.jpg

Failure Demand

Value Demand%

As a TeamThursday 16 June 2011



Level 5 - End User LevelThursday 16 June 2011

End User Perspective

• END to END monitoring

• User Experience (browser testing)

• External parties (Email, ISP, Plugins)


Participatory Design

http://www.informaworld.com/smpp/content~db=all~content=a790786652




“Crowd” - MonitoringSudden

Change in UserActivity Pattern

Detection of problem?

A/B Testing



Spend more time creatively , early in

the processAutomation =>

Traditional Engineering




Design Strategies

http://www.psych.lancs.ac.uk/people/lindenball.html

http://www.amazon.com/Research-Design-Thinking-Nigel-Cross/dp/9062757960 p 4

To Design is Human

To Design does not require super powers

Design used to be a collective or shared ability




http://www.amazon.com/Research-Design-Thinking-Nigel-Cross/dp/9062757960


Open Innovation

http://en.wikipedia.org/wiki/Open_innovation

http://www.clickadvisor.com/downloads/Bughin_Next_Step_in_Open_Innovation.pdf

Wikipedia

SharedInfrastructures

SharedConfig Management

Open Source






Novice vs Expert

• depth first

• vertical oriented

• can not deal with uncertainty


http://www.amazon.com/Research-Design-Thinking-Nigel-Cross/dp/9062757960 p 147

Linden Ball

• breath first , depth dives

• horizontal oriented

• uncertainty = options






Continuous Evolution

“Problem space evolves with the

solution”








Cultural Taxonomy


Cultural Taxonomy


tl;dr

• avoid vanity metrics

• insight > data

• don`t hate the player

• hate the game

• business value good

• beware of technical debt


Questions?



Thank you!


Technology

Devops Metrics