11 Tools for your devops stack
Kris Buytaert@krisbuytaert
Kris Buytaert
I used to be a Dev,
Then Became an Op
Chief Trolling Officer and Open Source Consultant @inuits.eu
Everything is an effing DNS Problem
What's this devops thing anyhow ?
World , 200X-2009Patrick Debois, Gildas Le Nadan, Andrew Clay
Shafer, Kris Buytaert, Jezz Humble, Lindsay Holmwood, John Willis,
Chris Read, Julian Simpson, Luke Kanies, John Allspaw and lots of
others ..Gent , October 2009Mountain View , June 2010Hamburg ,
October 2010Boston, March 2011Mountain View, June 2011Bangalore,
Melbourne, Goteborg , October 2011Rome, earlier this month!....
Devops is a growing professional and cultural movement
We don't have all the answers yet
We are reaching out to different communities
We will point out problems we see..
Only the name is new
While we are still working out the solutions
Devops, a definition:
Adopt the new philosophy. We are in a new economic age. Western
management must awaken to the challenge, must learn their
responsibilities, and take on leadership for change.
Cease dependence on inspection to achieve quality. Eliminate the
need for massive inspection by building quality into the product in
the first place.
Improve constantly and forever the system of production and
service, to improve quality and productivity, and thus constantly
decrease costs.
Institute training on the job.
Institute leadership The aim of supervision should be to help
people and machines and gadgets do a better job.
Drive out fear, so that everyone may work effectively for the
company.
Break down barriers between departments. People in research,
design, sales, and production must work as a team, in order to
foresee problems of production and usage that may be encountered
with the product or service.
Eliminate slogans, exhortations, and targets for the work force
asking for zero defects and new levels of productivity. Such
exhortations only create adversarial relationships, as the bulk of
the causes of low quality and low productivity belong to the system
and thus lie beyond the power of the work force.
Eliminate management by objective. Eliminate management by
numbers and numerical goals. Instead substitute with
leadership.
Remove barriers that rob the hourly worker of his right to pride
of workmanship. The responsibility of supervisors must be changed
from sheer numbers to quality.
Remove barriers that rob people in management and in engineering
of their right to pride of workmanship.
Institute a vigorous program of education and
self-improvement.
Put everybody in the company to work to accomplish the
transformation. The transformation is everybody's job.
William Edwards Deming 1986, Out of the Crisis.
http://en.wikipedia.org/wiki/W._Edwards_Deming
CAMS
Culture
Automation
Measurement
Sharing
Damon Edwards and John Willis
Culture,automation, Measturement, sharing
Devs vs Ops
Breaking the Silos
Getting Along
Ops
Devs
What tool?
Culture, Automation,Measurement,Sharing
Solved problems
Deployment automation :
Fai , kickstart, preseed, cobbler,
Configuration Management
Puppet, chef, cfengine,
....
challenges
Build & Release Management
Package & Repository management
Orchestration
Jenkins
Open Source Continuous Integration Server
A zillion plugins (400)
Have developers build stable and deployable code
Test Infra code
Jenkins
fpm
#packagingsucksMissing upstream
Ancient upstream
Unneeded dependencies
Broken upstream
Distro Policies
Maximum RPM ?
Packaging is needed
Dependencies, tooling, repositories
Anger driven development
fpm
fpm
fpm -t rpm -s dir -n hornetq -v 2.2.5 hornetq Executing(%prep):
/bin/sh -e /var/tmp/rpm-tmp.nNkVwh+ umask 022+ cd
/usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ exit
0Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.yUd4MV+ umask 022+
cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ cd
/usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ tar -zxf
/usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/data.tar.gz+ exit
0Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.jkpqeA+ umask
022+ cd /usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+
/usr/lib/rpm/brp-compress+ /usr/lib/rpm/brp-strip+
/usr/lib/rpm/brp-strip-static-archive+
/usr/lib/rpm/brp-strip-comment-noteProcessing files:
hornetq-2.2.5-1.x86_64Checking for unpackaged file(s):
/usr/lib/rpm/check-files
/usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILDWrote:
/usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/SRPMS/hornetq-2.2.5-1.src.rpmWrote:
/usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/RPMS/x86_64/hornetq-2.2.5-1.x86_64.rpmExecuting(%clean):
/bin/sh -e /var/tmp/rpm-tmp.z2UL3B+ umask 022+ cd
/usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ rm -rf
/usr/local/build-rpm-hornetq-2.2.5.x86_64.rpm/BUILD+ exit 0Created
/usr/local/hornetq-2.2.5.x86_64.rpm
fpm in action
https://github.com/Inuits/build-gems
Fork, pull
Jenkins pulls , builds , pushes to repo
(variants for Nagios Plugins / Jenkins Plugins available)
The Marionette Collective
Distributed ssh ++
What version of ssh do I have installed on my servers ?
On what servers is XYZ running ?
Clean al my ssl certs ?
Restart apache on all servers with fact X
mc-service
mc-service httpd status -W /uat/
* [
============================================================> ]
80 / 80
bleh.poc.com status=stopped xs03.ta.google.com status=stopped
elasticsearch02.uat.com status=stopped dbs01-.poc.com
status=stopped dino.uat.com status=stopped dsp01.uat.com
status=stopped webserver01.uat.com status=running dbs01-old.uat.com
status=stopped alexdb02-hdp.poc.com status=stopped postfix.uat.com
status=stopped ns01.uat.com status=stopped devtools03.uat.com
status=running
mc-package
mc-package -W /dev/ status jdk * [
============================================================> ]
33 / 33
servicesdb01.dev.com version = -absent services.dev.google.com
version = jdk-1.6.0_13-fcs drbdtest02.dev.google.com version =
-absent services3.dev.google.com version = jdk-1.6.0_20-fcs
um.dev.google.com version = jdk-1.5.0_19-fcs devtools03.uat.com
version = jdk-1.6.0_29-fcs alexandria02.dev.google.com version =
-absent weblink01.dev.com version = -absent wikitest.dev.google.com
version = jdk-1.6.0_24-fcs payment.dev.google.com version =
jdk-1.5.0_17-fcs tiff2pdf01.dev.com version = -absent
devdoos.dev.com version = jdk-1.6.0_30-fcs wiki.dev.google.com
version = jdk-1.6.0_24-fcs reporting01.dev.com version = -absent
devtools01-dev.uat.com version = jdk-1.6.0_23-fcs
devtools02.uat.com version = jdk-1.6.0_29-fcs
drbdtest01.dev.google.com version = -absent
---- package agent summary ---- Nodes: 33/33 Versions: 1 *
1.5.0_17-fcs, 1 * 1.5.0_19-fcs, 1 * 1.6.0_13-fcs, 1 * 1.6.0_20-fcs,
1 * 1.6.0_23-fcs, 2 * 1.6.0_24-fcs, 2 * 1.6.0_29-fcs, 1 *
1.6.0_30-fcs, 1 * 1.6.0_31-fcs, 22 * absent Elapsed Time: 1.73
s
Culture, Automation,Measurement,Sharing
#monitoringsucks
Monitoring is AWESOME. Metrics are AWESOME. I love it. Here's
what I don't love: Having my hands tied with the model of host and
service bindings.
Having to set up "fake" hosts just to group arbitrary metrics
together
Having to either collect metrics twice - once for alerting and
another for trending
Only being able to see my metrics in 5 minute intervals
Having to chose between shitty interface but great monitoring or
shitty monitoring but great interface
Dealing with a monitoring system that thinks IT is the system of
truth for my environment
Not actually having any real choicesJohn Vincent (@lusis) on his
blog
http://lusislog.blogspot.com/2011/06/why-monitoring-sucks.html
A sub movement
#monitoringsucks trending
https://github.com/monitoringsucks/
2008 Study :Nagios + Friends
2011 Conclusion : Nagios/Icinga are the only automatable
alternatives
Monitoring and trending at scale , new kids Graphite, Icinga,
flapjack, etc
Old Cool Kids, Ganglia
Logstash
Not your average centralized logging tool
Elasticsearch backed
Shipper
Indexer
Web
Logstash
Collect from anywhere
Filter
Send anywhere
Queuing support
Logstash in Action
Kibana in Action
Puppet and Logstash
https://github.com/KrisBuytaert/vagrant-puppet-logstash
Includes Logstash / Kibana / Grok / ES modules as submodules
!
git clone
git submodule init
git submodule update
vagrant up
Graphite
Graphing at Scale
Graphing at Ease
Any metric is a graph
echo "somestring $somevalue $timestamp" | nc 2003
Graphite Composer
x
Graphite API
Friends of Graphite
Collection :
Statsd,
Collectd + Carbonplugin
Jmxtrans
Logster
Dashboards
Tattle
Gdash
...
Gdash In action
Puppet and Graphite
https://github.com/KrisBuytaert/vagrant-graphite/
Includes Graphite / Gdash / Jmxtrans / Logster / Collectd /
Statsd / Tattle and more modules as submodules !
git clone
git submodule init
git submodule update
vagrant up
Culture, Automation, Measurement,Sharing
Open Source
Github
Talk about Experiences
Open Spaces
Sharing environments
Build identical environments
Share code
Shared ownership of content , code and configuration
Vagrant
Abstraction layer for VirtualBox
Integrates well with Puppet/Chef
Project =
Vagrantfile
Manifests / Cookbooks
Portable, Small , Versionable
Use veewee to build your boxen
Vagrantfile
Vagrant::Config.run do |config| # All Vagrant configuration is
done here. The most common configuration # options are documented
and commented below. For a complete reference, # please see the
online documentation at vagrantup.com.
config.vm.define :mongo1 do |mongo1_config|
mongo1_config.ssh.max_tries = 100 mongo1_config.vm.box =
"MyCentOS2" mongo1_config.vm.network("192.168.99.101")
mongo1_config.vm.host_name = "mongo1" mongo1_config.vm.provision
:puppet do |mongo1_puppet| mongo1_puppet.pp_path =
"/tmp/vagrant-puppet" mongo1_puppet.manifests_path = "manifests"
mongo1_puppet.module_path = "modules" mongo1_puppet.manifest_file =
"site.pp" end end config.vm.define :mongo2 do |mongo2_config|
mongo2_config.ssh.max_tries = 100 mongo2_config.vm.box =
"MyCentOS2" mongo2_config.vm.network("192.168.99.102")
mongo2_config.vm.host_name = "mongo2" mongo2_config.vm.provision
:puppet do |mongo2_puppet| mongo2_puppet.pp_path =
"/tmp/vagrant-puppet" mongo2_puppet.manifests_path = "manifests"
mongo2_puppet.module_path = "modules" mongo2_puppet.manifest_file =
"site.pp" end end
Vagrant Rocks
Vagrant init
Vagrant up
Vagrant provision
Vagrant down
Vagrant destroy
Sushi
Quiz Time : Which tool did I forget ?
You
Links
http://puppetlabs.com/mcollective/introduction/https://github.com/jordansissel/fpmhttp://logstash.net/https://github.com/rashidkpc/Kibanahttp://graphite.wikidot.com/https://github.com/wayfair/Graphite-Tattlehttp://vagrantup.com/https://github.com/KrisBuytaert
Contact
Kris Buytaert [email protected]
Further Reading@krisbuytaert
http://www.krisbuytaert.be/blog/http://www.inuits.be/
Inuits
Duboistraat 502060 AntwerpenBelgium891.514.231
+32 475 961221
