Devops: The War is over - If you want it

http://jedi.be/blogPatrick Debois

OPSDEV

When I was a young developer

http://www.flickr.com/photos/ninahiironniemi/1118868590/sizes/l/

http://1000awesomethings.files.wordpress.com/2008/11/campfire.jpg

I loved to developsimple applications

http://www.rookgordijn.net/wp-content/uploads/2009/08/Texel_21.JPG

I kept my own little server going

http://www.foodgeekery.com/fullimg/ribfest09-cowboy2.jpg

Most of the time , the results were fine

http://www.flickr.com/photos/abbyladybug/2644163076/sizes/l/

Sometimes the applications would crash and I didn’t really know why

http://www.motoer75.nl/images/BBQ2009030.jpg

But overall my friends seemed to like my work

http://www.bfeedme.com/wp-content/uploads/2007/07/legends-of-texas-bbq-cookbook-of-the-day-7-30-07.jpg

I learned from the pro’s

http://www.freepatentsonline.com/6557545-0-large.jpg

Read about software

architecture

http://patiopalace.com/assets/images/gas-grills/webber/certifiedexperts3.jpg

I even got my Scrum certification

http://www.flickr.com/photos/tradica/3071394619/sizes/l/

Still I was experiencing the occasional problems

I blamed the operations guys

http://media.photobucket.com/image/bbq%20cowboy/NotleyQue/The%20BBQ%20Family/CowboyChili.jpg

I decided to leave the cowboys

http://www.flickr.com/photos/iimaj/3959147070/sizes/o/

And make plans of my own

http://www.flickr.com/photos/brachiator/107510145/sizes/l/

I really wanted to focus on the customer

http://www.dongeovanny.com/images/meatpage2kobe.jpg

I had the developers use the coolest programming languages

http://www.bbqsaucereviews.com/wp-content/uploads/2009/06/bbqsaucereview-001.jpg

Use the best frameworks available

http://jeffreysmeatmarketblog.files.wordpress.com/2009/06/picture-4-3-0412.jpg

We had the DBA’s take care of the database schemas

http://www.flickr.com/photos/jayneandd/4078134280/sizes/l/http://www.flickr.com/photos/bhamsandwich/3601018752/sizes/l/

Explained them that User Interface is important

http://lovefeasttable.com/blog/wp-content/uploads/2009/05/may09c067.jpg

Up until now we had done only some manual checks.

http://www.flickr.com/photos/cookthink/210948100/sizes/l/

So we moved to unit tests

http://blogs.suntimes.com/food/4-29_white_BBQ_1.jpg

Even Behavior Driven Development

We were still experiencing problems with our application

servers.

http://www.flickr.com/photos/80417459@N00/2056898218/

We brought in bigger storage boxes

http://euroross.blogspot.com/bbq.jpg

gave the sysadmins the biggest servers

http://propanesmoker.org/wp-content/uploads/2009/11/Propane_smoker.jpg

created virtual machines to

create different environments

http://www.bbqpits.com/people_of_bbq/greg_nj3.jpg

Used the cloud to solve some scalability problems

http://www.flickr.com/photos/theogeo/3317291208/sizes/l/in/photostream/

Still the customer wasn’t happy

http://www.funnypictures.net.au/images/firefighter-bbq1.jpg

And the IT department was firefighting all the time

http://www.flickr.com/photos/atesca/2457127172/sizes/l/

Then there was a complete power failure

http://www.flickr.com/photos/zanyflash/2278286656/sizes/o/

Nobody had checked the power installation

http://www.flickr.com/photos/cirox/363284194/sizes/l/

Leaving the application in an unknown state

http://badgerdiary.files.wordpress.com/2008/12/closed-sign-smer.jpg

Resulting again in a major outage

http://www.nonplusultra.nl/data/producten/12727-groot.jpg

Our small fixes would not help anymore

http://www.proexotics.com/faq_images/soaking_water_monitor.jpg

All backups were corrupt because we could only backup properly when the application was down

http://www.flickr.com/photos/jmv/16366498/sizes/o/

Luckily we had a cold Standby Backup

http://www.flickr.com/photos/tom-poes/60373623/

Time to take a

step back

http://allwecaneat.files.wordpress.com/2008/08/_igp7826_sm.jpg

We worked hard on minimizing technical debt in our code

http://www.eggcookers.net/images/pictures/progressive-stainless-steel-bbq-tongs-bbq505.jpg

http://ecx.images-amazon.com/images/I/41wSaS%2BMy8L.jpg

We started to improve our deployment tools

http://www.flickr.com/photos/wiphey/66525468/sizes/o/

Put monitoring in our development

and test environment

http://3.bp.blogspot.com/_ySMs_t20v08/Sw8m5JjVg1I/AAAAAAAAAFQ/jr2nXJYSfbI/s1600/DSC_4835.jpg

Reused it in the production environment

http://weblog.greenpeace.org/makingwaves/welldone430.jpg

Implemented scaling from the ground up

http://www.universalbarbeque.com/images/untitled-2_copy.jpg

We made deployment reproducible

http://www.flickr.com/photos/cjscott69/400160562/sizes/l/

Improved the way to collect metrics

http://www.flickr.com/photos/namealus/3114627540/

To offload some of the appserverswe installed a caching system

http://www.flickr.com/photos/benchilada/4231838748/sizes/l/

And loadbalancers solved the webserver scaling problem

http://www.izzysburgerspa.com/images/uploaded/image/Delivery%20Truck%201.jpg

Content Delivery Networks

http://strongdogz.com/wp-content/uploads/2008/09/beauceron.jpg

Made our application security proof

http://www.flickr.com/photos/loop_oh/3086747855/sizes/o/

Comply to regulations

http://www.mzmfarm.net/linked/img_3507.jpg

Integrated backups on the application level

http://carrotsncake.com/wp-content/uploads/2009/03/img_6705.jpg

Solved the multi-user problem problem

For some knowstates

We would automaticallyreset the system

Self healing on our database

http://assets.whirlpool.com/files/usr/970/FilterReset5.jpg

So you want to be a great agile chef?

http://www.damniwish.com/wp-content/uploads/2009/12/IMG_1265.jpg

Software Infrastructure

Presentation

Operations

Testing

Project

#1 care for the whole lifecycle#2 look at it from a business perspective

#3 work together as a team

It’s not about the software

It’s not about the hardware

I like my servers, software steak well done

http://www.flickr.com/photos/spenceke/2250116189/sizes/l/

IT Tooling is changing

Virtualization

• kvm

• xen

• solaris zones

• vmware

• virtualbox

• ...

Cloud showed us how to scale

Abstraction API

http://www.jclouds.org/

Adrian Cole@jclouds

http://libvirt.org/

Platform as a Service

Hudson as Service

http://www.cloudbees.com/

But who will manage

your IAASplatform?

First steps of Automation

https://fedorahosted.org/cobbler/

http://spacewalk.redhat.com/

http://fai-project.org/

Cloning

Virtual Sprawl

Configuration Management

Chef

Pallet

http://www.cfengine.org/

http://www.puppetlabs.com

http://www.opscode.com/chef

https://github.com/hugoduncan/pallet

# tomcat.pp

class tomcat {

! package {! ! "java-1.6.0-sun-compat":! ! ! ensure => "installed";! ! "tomcat6":! ! ! require => [Package["java-1.6.0-sun-compat"], Package["ntc-tomcat-log4j"]],! ! ! ensure => "installed";! ! "tomcat6-admin-webapps":! ! ! ensure => "installed";! ! "log4j":! ! ! ensure => "installed";! ! "tanukiwrapper":! ! ! ensure => "installed";! ! "mysql-connector-java":! ! ! ensure => "installed";! }

! file {! ! "/etc/init.d/tomcat6": mode => "0755", owner => "root", group => "root", source => "puppet:///tomcat/tomcat6.init"; }

! user {! ! "tomcat":! ! ! ensure => "present",! ! ! require => Package["tomcat6"];! }}!

tomcat",! ! ! mode => "0755",! ! ! require => Package['tomcat6'],! ! ! content => template("headers/header-xml.erb","tomcat/tomcat6-init.erb");! ! "/etc/init.d/build-jar-repo":! ! ! owner => "tomcat", group => "tomcat", mode => "0755",! ! ! require => Package['tomcat6'], source => "puppet:///tomcat/build-jar-repo";! ! "/var/log/tomcat6/$name":! ! ! owner => "tomcat",! ! ! group => "tomcat",! ! ! mode => "0755",! ! ! require => Package['tomcat6'],! ! ! ensure => "directory";! ! "/var/cache/tomcat6/temp/$name": owner => "root", group => "root", mode => "0755",! ! ! require => Package['tomcat6'], ensure => "directory";! ! "/var/cache/tomcat6/work/$name": owner => "root", group => "root", mode => "0755",! ! ! require => Package['tomcat6'], ensure => "directory";! ! "/usr/local/tomcat6-$name/temp":! ! ! ensure => "link", require => File["/var/cache/tomcat6/temp/$name"], target => "/var/cache/tomcat6/temp/$name";! ! "/usr/local/tomcat6-$name/work":! ! ! ensure => "link", require => File["/var/cache/tomcat6/work/$name"], target => "/var/cache/tomcat6/work/$name";! ! "/usr/local/tomcat6-$name/work/Catalina": owner => "tomcat", group => "tomcat", ensure => "directory", require => File["/usr/local/tomcat6-$name/work"]; "/usr/local/tomcat6-$name/work/Catalina/localhost": owner => "tomcat", group => "tomcat", ensure => "directory", require => File["/usr/local/tomcat6-$name/work/Catalina"]; "/usr/local/tomcat6-$name/work/Catalina/localhost/manager": owner => "tomcat", group => "tomcat", ensure => "directory", require => File["/usr/local/tomcat6-$name/work/Catalina/localhost"];! }

! service { "tomcat6-$name": enable => "false", require => [Package['tomcat6'], Package['tanukiwrapper']];! ! "build-jar-repo":! ! ! enable => "true",! ! ! start => "/etc/init.d/build-jar-repo", require => File["/etc/init.d/build-jar-repo"]];! }!

Infrastructure as code

Model Driven Infrastructure

https://github.com/lusis/vogeler

John E. Vincent@lusis

Definitionof your

infrastructure

Version Control

Reusable components

https://github.com/camptocamp

Code/Server SmellsPrivate!Playground

The sysadmin uses the system as his toy playground, doesn't clean up.

• /tmp & /var/tmp full of old install files• / full of files

Gready!Server

One server that does every function

• combined mail and web and dns and fileshares

• all users on the same systemRoot is the cause of all!evil

• last show login all root• no sudo is activated• no sshd keys for logins• nfs share/root?• Chmod 777• most processes run as root

Sharing Recipes/Code

http://forge.puppetlabs.com/

http://cookbooks.opscode.com/

Automation without tests is

Lindsay Holmwood @auxesis

Feature: google.com It should be up And I should be able to search for things

Scenario: Searching for things When I visit "http://www.google.com" And I fill in "q" with "wikipedia" And I press "Google Search" Then I should see "www.wikipedia.org"

$ cucumber-nagios google.featureCritical: 0, Warning: 0, 4 okay | passed=4, failed=0

cucumber-nagios

Lindsay Holmwood @auxesis

Scenario: Checking /etc/passwd When I ssh to "example.org" with the following credentials: | username | password | keyfile | | jacob | spifeofstrife | /home/jacob/.ssh/id_dsa | And I run "cat /etc/passwd" Then I should see "jacob" in the output

cucumber-ssh

Julian Simpson@builddoctor

apache.featureFeature: In order to deploy my application As a systems administrator I want to know that the config files are legal

Scenario: Valid Apache config Given a config file template apache2.conf in etc/apache2 When I generate it Then there should be a file called apache2.conf in etc/apache2 And it should be valid

cucumber-apache

http://pastie.org/693713

Feature: cucumber-puppet In order to run my puppet manifest's test suite As an admin I want the cucumber-puppet gem installed

Scenario: Install cucumber-puppet Given a node of class "cucumber-puppet" When I compile the catalog Then gem "cucumber-puppet" should be "installed"

https://github.com/nistude/cucumber-puppet

cucumber-puppet

Nikolay Sturm@nistude

Orchestration

http://controltier.com/ http://marionett-collective.org

@ripienaar@damonedwards

Vagrant

http://vagrantup.com/

Allows an easy way to use the production recipes in your own dev environment

Mitchell Hashimoto@mitchellh

IAAS and PAAS blurring

Pull in both your recipes and your code

http://blog.crisp.se/henrikkniberg/2008/07/14/1216002720000.html

Agile Infrastructure

&

Agile Sysadmin

Comparing Agile and ITIL

http://www.winstonind.com/images/brands/traditional_cooking_process.gif

Agile ITIL

Certifications

Process Fundamentalism

Manifesto’s for the world

Meat Cloud Manifesto

Agile Manifesto

== Rules ==On Infrastructure—————–There is one system, not a collection of systems.The desired state of the system should be a known quantity.The “known quantity” must be machine parseable.The actual state of the system must self-correct to the desired state.The only authoritative source for the actual state of the system is the system.The entire system must be deployable using source media and text files.

On Buying Software——————-

.....

James White Manifesto

Truck Factor

The number of people on your team

who have to be hit with a truck before

the project is in serious trouble

The ability to take a random machine in your infrastructure

, drop it from the 10th floor,

and be able to restore your infrastructure to working within 5-10 minutes

10th floor test

Backlog of items

• User Stories

• Features

• Tickets

• Incidents

• Problems

Requirements

• Login/Logout

• Print order

• CRUD order

• Security

• Capacity

• Stability

• Scalability

FunctionalNon-Functional

Business ValueOrdered by

the value gainedOrdered by the value

protected

Increase Change Avoid Change

Proxy Customers

ServiceManager

Understand what the business wants

Support the team

Scrum ManagerService Delivery

Manager

ProductOwner1

Ideally one product owner

ProductOwner2

ProductOwner3

Development Team

Project1 Project 2 Project 3

Operations ManagerScrum Master

Operations Team

Outsourcing

Call CentersOutsourced

Development

Specialist & GeneralistsRockstar Mentality

Testers UIDesigners

Frontend

Backend

AnalystsSecurity

Network

Storage

Sysadmins

Global Ownership

It’s the javascript

It’s the backend

It’s the database

It’s the network

It’s the server

It’s the storage

Sprint Review Meeting

Change AdvisoryBoard

Approve Changes

Iterations & Flow

Kanban

1day sprints Operations

SprintVelocity

Service Levels

Team Approved

Estimate

Improves over timeStoryPoints

Uptime 99%

Continuous ImprovementLean Thinking

Lean ITILLean

Software Development

Value Stream Mapping

Technical waste

What’s new?

I know all that

The map is not

the territoryAlfred Korzybski

Systems Thinking

Optimize the whole not just the individual parts

People make it happen

Culture & Mindset

Collaboration

TestersAnalysts

Developers

Security

ArchitectsUI Designers

Network

Sysadmins

not only when things go wrong

No rock star mentality

Mine are bigger then yours

Craftsmanship

Trust

It’s about customervalue

http://www.flickr.com/photos/spenceke/2250116189/sizes/l/

Interesting Books

Want to know more?

#devops on twitter & irchttp://devopsdays.orghttp://devopscafe.org

http://planetdevops.netdevops googlegroup

devops-toolchain googlegroup

The End