C H E For how to make computers do the work for us

Marcin Kulik, Lunar Logic Polska

KRUG 2011/11/08

Everyday we're dealing with mechanical, repetitive tasks... wecan automate.

What is Chef?

Automation tool

written in Ruby

DSL

Created by Opscode

"Chef is an open source systems integration framework builtto bring the benefits of configuration management to your

entire infrastructure."

"You write source code to describe how you want each part ofyour infrastructure to be built, then apply those descriptions

to your servers."

"The result is a fully automated infrastructure: when a newserver comes on line, the only thing you have to do is tell Chef

what role it should play in your architecture."

Why do you need it?

Economics + Efficiency + Scalability

Terms

Noderemote server, local machine...

Roleweb server, database server, ruby dev workstation...

Cookbookmysql, ssh-access, dotfiles...

Recipeinstall mysql-server, create database, add user...

Resourcefile, dir, user, package, service, gem, virtual host...

Run listlist of recipes to run in order

{ "run_list": [ "recipe[mysql]", "recipe[git]", "recipe[ruby19]" ]}

Cookbook structure |-- config | |-- node.json | `-- solo.rb |-- cookbooks | |-- book1 | | |-- attributes | | |-- files | | |-- metadata.rb | | |-- recipes | | | |-- default.rb | | | `-- source.rb | | `-- templates | |-- book2 | | |-- attributes | | | `-- default.rb | | |-- files | | |-- recipes | | | `-- default.rb | | `-- templates

| | `-- templates | | `-- default | | `-- authorized_keys.erb | |-- book3 | | |-- attributes | | |-- files | | | `-- default | | | `-- secret-key | | |-- recipes | | | `-- default.rb | | `-- templates |-- config | |-- node.json | `-- solo.rb |-- cookbooks | |-- book1 | | |-- attributes | | |-- files | | |-- metadata.rb | | |-- recipes | | | |-- default.rb | | | `-- libs.rb | | `-- templates

Installation

$ gem install chef

Modes of operation

Cookbooks stored

in central repository(free cookbooks hosting by Opscode:

https://manage.opscode.com/)

$ sudo chef-client

Cookbooks stored

on the node

$ sudo chef-solo -c /path/to/cfg.rb -j /path/to/node-data.json

Use cases

Configure new machine

(in the cloud with Knife)Amazon EC2, Engine Yard, Linode, BrightBox...

Manage config of existing

company serversClient demo apps (directory, vhost, god config), developers' ssh

keys...

Bootstrap workstation!rvm + ruby 1.9, git, mysql, vim/emacs...

Enough with theory!

Lunar Stationhttps://github.com/LunarLogicPolska/lunar-station

Lunar Station is a set of Chef cookbooks and a bash script (???)for bootstrapping developers machines at Lunar Logic Polska.

You need ruby to run Chef

(We assume) you use RVMNo need for system ruby for ruby devs nowadays

bootstrap.sh

detects platform (Ubuntu, Fedora, OSX)

installs compilers and other RVMdependencies

installs RVM & ruby 1.9 & chef gem

downloads latest Lunar Stationcookbooks

runs chef-solo

$ curl -skL http://bit.ly/lunar-station | bashInitializing Lunar Workstation...>> Fedora Linux detected.>> Checking for RVM...>> Fetching latest version of Lunar Station cookbooks...>> Starting chef-solo run...[Mon, 07 Nov 2011 22:19:54 +0100] INFO: *** Chef 0.10.4 ***[Mon, 07 Nov 2011 22:19:54 +0100] INFO: Setting the run_list to ...

Nodes

# linux-rubydev.json

{ "run_list": [ "role[rubydev]" ]}

# osx-rubydev.json

{ "run_list": [ "role[osx]", "role[rubydev]" ]}

Roles

# base.rb

run_list 'recipe[repos]', 'recipe[curl]', 'recipe[wget]', 'recipe[git]', 'recipe[libxml2]', 'recipe[ack]', 'recipe[vim]', 'recipe[ctags]', 'recipe[skype]', 'recipe[firefox]' , 'recipe[google-chrome]'

# rubydev.rb

run_list 'role[base]', 'recipe[mysql]'

# osx.rb

run_list "recipe[homebrew]"

Cookbooks

repos cookbook

# cookbooks/repos/recipes/default.rb

case node[:platform]when 'fedora' path = "/tmp/rpmfusion-free-release-stable.noarch.rpm"

bash "download rpmfusion free package" do code "wget http://download1.rpmfusion.org/.../" + "rpmfusion-free-release-stable.noarch.rpm -O #{path}"

not_if { File.exist?(path) } end

package "rpmfusion-free-release-stable" do source path options "--nogpgcheck" end

when 'ubuntu' ...end

end

# cookbooks/repos/recipes/default.rb

case node[:platform]when 'fedora' ...

when 'ubuntu' bash "enable multiverse repo" do code "head -n 1 /etc/apt/sources.list | " + "sed 's/main universe/multiverse/' " + ">> /etc/apt/sources.list"

not_if "egrep '^deb.+multiverse' /etc/apt/sources.list" endend

vim cookbook

# cookbooks/vim/recipes/default.rb

case node[:platform]when "ubuntu" package "vim" package "vim-gnome"

when "fedora" package "vim-enhanced" package "vim-X11"

when 'mac_os_x' package "macvim"end

skype cookbook

# cookbooks/skype/recipes/default.rb

case node[:platform]when 'ubuntu' include_recipe 'init::ubuntu' # for partner repo

package 'skype'

when 'mac_os_x' dmg_package "Skype" do source "http://www.skype.com/go/getskype-macosx.dmg" action :install end

when 'fedora' ...end

Lunar Kitchen

Source of LLP servers configuration data and a set of Chefcookbooks

chef-solo invoked onremote machines

no chef server

Each server we configure has its corresponding nodeconfiguration file in nodes/ directory of kitchen project that

specifies run_list and few other settings

# nodes/deneb.json

{ "run_list": [ "recipe[ssh_access]" ],

"ssh_access": [ "marcin.kulik", "anna.lesniak", ...],

"opened_ports": { "tcp": [80, 443, 22, 8080], "udp": [] }, ...

How do we run chef-soloon remote machine?

Capistrano!

# See the list of configured servers:

$ cap -T

# Make the changes happen on the server:

$ cap configure:deneb

How does Capfile look like?

set :user, 'chef'

NODE_LIST = Dir["nodes/*.json"].map do |nodefile| File.basename(nodefile, '.json')end

NODE_LIST.each do |node| role node.to_sym, nodeend

NODE_CONFIG = <<-EOS file_cache_path '/tmp/chef-solo' cookbook_path '/tmp/chef-solo/cookbooks' role_path '/tmp/chef-solo/roles'EOS...

...namespace :configure do NODE_LIST.each do |node| desc "Configure #{node}" task node.to_sym, :roles => node.to_sym do run "if [ ! -e /tmp/chef-solo ]; then mkdir /tmp/chef-solo; fi" upload("cookbooks", "/tmp/chef-solo/", :via => :scp, :recursive => true) upload("roles", "/tmp/chef-solo/", :via => :scp, :recursive => true) upload("nodes/#{node}.json", "/tmp/chef-solo/node.json", :via => :scp) put(NODE_CONFIG, "/tmp/chef-solo/solo.rb") run "rvmsudo chef-solo " + "-c /tmp/chef-solo/solo.rb " + "-j /tmp/chef-solo/node.json" end endend

SSH access

├── Capfile├── config├── cookbooks├── nodes├── README.md├── roles└── ssh_keys ├── anna.lesniak ├── artur.bilski ├── ... └── marcin.kulik

# cookbooks/access/recipes/default.rb

username = 'dev'

ssh_keys = node[:ssh_access].map do |f| File.read("/tmp/chef-solo/ssh_keys/#{f}")end

template "/home/#{username}/.ssh/authorized_keys" do source "authorized_keys.erb" owner username group 'users' mode "0600" variables :ssh_keys => ssh_keysend

# cookbooks/access/templates/authorized_keys.erb

# Generated by Chef, do not edit!

<%= @ssh_keys.join("\n") %>

Tips

Learn step by stepEC2 + Chef + Knife + Opscode... = Fuuuuuuuuuuuuuuuuuuuuu

Start with chef-solo

Run on local machineEasy to troubleshoot problems

Use Vagranthttp://vagrantup.com/

Great for testing cookbooks - doesn't pollute your system

Q?

Thanks!marcin.kulik@llp.pl | @sickill | https://github.com/sickill