Chef 11 Preview/Chef for OpenStack

Boston 1/22/2013

Matt Raymatt@opscode.com

@mattray

Tuesday, January 22, 13

What's New Since Chef 0.10?

• Windows support

• Librarian

• Spiceweasel

• Foodcritic

• Cookbooks in separate repositories

• Private Chef

• Food Fight podcast

• knife plugins for openstack, hp, azure, gce

• 0.10 renumbered to Chef 10

• full-stack client

• why-run

• output formatters

• Test Kitchen

• Solaris support

• Chef Developer Summit(2!)

• Berkshelf

• ChefConf

• Fauxhai

• docs.opscode.com

• Hangouts for reviews

• nyan-cat output formatter

Tuesday, January 22, 13

Vaguely chronological, so much good stuff in the Chef Community

What is NOT in Chef 11

• CouchDB

• Ruby-based chef-server-api

• depsolver, gecode, treetop

• Merb

• OpenID support in Web UI

• Migration tooling (coming soon)

http://trainweb.org/carl/Pullman150/IMG_3380.jpgTuesday, January 22, 13

What’s NEW in Chef 11

• PostgreSQL

• erchef

• nginx

• Rails

• bookshelf

• omnibus-chef server

• chef-apply

• partial search

• users with key pairs

• partials in templates

• knife-essentials

Tuesday, January 22, 13

nginx

erchef

solr(lucene)

bookshelf

PostgreSQLfilesystemcookbookstore

filesystemsearchindex

RabbitMQ

chef-expander

chef-webui

Chef Server Architecture

Tuesday, January 22, 13

Postgres has replaced Couch (who knew SQL was so good?)Erlang-based erchef has replaced Ruby-based chef-server-apichef-webui has been upgraded to Rails 3 from MerbBookshelf is our S3-compatibilish storage service

erchef

chef_wm

chef_index chef_dbchef_authnchef_objects

solrbookshelfPostgreSQL

RabbitMQ

erchef Architecture

Tuesday, January 22, 13

Erlang applications, each on GitHub

CPU Usage on Chef Server

Tuesday, January 22, 13

These are old graphs from last year. Code is actually much faster and more efficient now that we've had a year of tuning.This is with erchef and Ruby code on the same box3 CPU vm graph is aggregate

CouchDB Uptime

Tuesday, January 22, 13

the smaller spikes were when we chron'd restarts

Database CPU

CouchDB MySQL

Tuesday, January 22, 13

Postgres is about the same performance, but we've found we're able to focus optimizations on a single rdbms and put time that we would otherwise have to spend on ensuring both backend dbs work properly into other areas of the code for fixes, enhances, features.

Database Memory

CouchDB MySQL

Tuesday, January 22, 13

Nice and flat

Database Load Average

CouchDB MySQL

Tuesday, January 22, 13

Licensing is much better for us with Postgres

API Average Latency

Tuesday, January 22, 13

And things have gotten better from here with Private Chef and Open Source Chef. Not everything has been ported to Opscode Hosted Chef yet. Work has started to complete the migration of OHC to the OPC code base, things with OHC will get far better.

omnibus-chef server

• full-stack for the Chef server, everything you need

• installs to /opt/chef-server

• RPMs, DEBs for now

• build your own packages

• chef-server-ctl

• status

• start/stop

• tail

http://apod.nasa.gov/apod/astropix.html

Tuesday, January 22, 13

Much of the same tooling OPC

chef-apply

• run a single recipe file

• without modifying the node's run_list

• included with Chef gem, /usr/bin/chef-apply

• chef-apply /path/to/recipe_file

• chef-apply "content of a recipe file"

http://www.flickr.com/photos/albill/sets/72157628046395000/

Tuesday, January 22, 13

http://tickets.opscode.com/browse/CHEF-3571

partial search

http://www.flickr.com/photos/albill/sets/72157628046395000/

partial_search(:node,  'role:web',      :keys  =>  {  'name'  =>  [  'name'  ],                            'ip'      =>  [  'ipaddress'  ],                            'kernel_version'  =>  [  'kernel',  'version'  ]                        }).each  do  |result|    puts  result['name']    puts  result['ip']    puts  result['kernel_version']end

• instead of entire node, just the pieces you want

• massive reduction in bandwidth and memory

• http://community.opscode.com/cookbooks/partial_search

• to use with Chef 10 and Hosted Chef today

Tuesday, January 22, 13

users with keypairs

• users can have key pairs, just like clients

• knife actions as a user instead of a client

• post a public key when you create a user/client

http://www.flickr.com/photos/albill/sets/72157628046395000/

Tuesday, January 22, 13

For users in Chef 11, the key point is:Before: users were just a concept of the webui. If you wanted a knife setup, you needed a client.After: users can have key pairs just like clients. So you can do knife actions as your user.

<?xml version='1.0' encoding='UTF-8'?>

<server xmlns="urn:jboss:domain:1.2"> <extensions> <% if @infinispan -%> <extension module="org.jboss.as.clustering.infinispan"/> <% end -%> <extension module="org.jboss.as.web"/> <% if @webservices -%> <extension module="org.jboss.as.webservices"/> <% end -%> <extension module="org.jboss.as.weld"/> </extensions> <% if @infinispan include_template "infinispan.xml.erb" end -%> <% if @webservices include_template "webservices.xml.erb" end -%> <socket-binding-group name="standard-sockets" > ...... </socket-binding-group></server>

• Thanks to Andrea Campi!

partials in templates

http://www.flickr.com/photos/modern_fred/2095565021/

Tuesday, January 22, 13

knife-essentials

• Unified commands that work on everything

• https://github.com/jkeiser/knife-essentials

• knife download [pattern1 pattern2 ...]

• knife download roles data_bags cookbooks/emacs

• knife diff cookbooks/*apache*

• knife show *base*

• knife upload apache*

• knife list data_bags/users

• knife deps roles/base.json

http://photography.nationalgeographic.com/photography/photo-of-the-day/Tuesday, January 22, 13

knife downloadknife diffknife showknife listknife upload

Chef 11 Breaking Changes

• http://wiki.opscode.com/display/chef/Breaking+Changes+in+Chef+11

• chef-shell (formerly shef)

• no implicits node attributes (no more node['blah'] = 'foo')

• attribute files may access role and environment attributes

• delayed notifications run after failed converge

• encrypted data bag item format change

• chef-client lock so safe from simultaneous runs

http://www.flickr.com/photos/modern_fred/2096352938/Tuesday, January 22, 13

More changes on the wiki page

Chef 11 Server Preview

• http://wiki.opscode.com/display/chef/Chef+11+Server+Preview

• Nightly Builds

• Centos 5 & 6

• Ubuntu 10.04, 11.04 and 12.04

• Everything is on GitHub

• More testing means better releases!

• Client is in beta: http://lists.opscode.com/sympa/arc/chef-dev/2013-01/msg00028.html

http://www.brandonbird.com/lazysunday.html

Tuesday, January 22, 13

Coming soon!

Overview & Status of Chef for OpenStack

Tuesday, January 22, 13

Chef for OpenStack: Who

Tuesday, January 22, 13

These companies are currently involved to some extent

Chef for OpenStack: Why

• Community for the automated deployment and management of OpenStack

• Reduce fragmentation and encourage collaboration

• Deploying OpenStack is not "secret sauce"

• Project not a product

• Apache 2 license

Tuesday, January 22, 13

Chef for OpenStack: What

• Chef Repository for Deploying OpenStack

• Documentation for Chef for OpenStack

• Cookbooks• Keystone

• Glance

• Nova

• Horizon

• Swift

• Quantum

• Cinder

• knife-openstackTuesday, January 22, 13

Chef for OpenStack: Where

• groups.google.com/group/opscode-chef-openstack

• #openstack-chef on irc.freenode.net

• github.com/opscode/openstack-chef-repo

• github.com/mattray/openstack-chef-docs

• github.com/opscode-cookbooks/

• keystone, glance, nova, horizon, swift,quantum,cinder

• github.com/opscode/knife-openstack

• @chefopenstack

Tuesday, January 22, 13

• Chef repo for Essex (2012.1.1 tag)

• Operating Systems (Ubuntu 12.04)

• Hypervisors (KVM, LXC)

• Databases (MySQL)

• FlatDHCP & VLAN networking

• manual floating IPs documented

• Test Kitchen integration

Chef for OpenStack: When (Today)

Tuesday, January 22, 13

Chef for OpenStack: When (Tomorrow)

• Documentation (docs.opscode.com)

• Folsom is under active development

• AT&T, DreamHost and Rackspace have active branches

• Move to openstack-common

• Cinder (lvm, Netapp)

• Quantum (Nicira with Open vSwitch)

Tuesday, January 22, 13

• Documentation (docs.opscode.com)

• Continuous Integration testing

• Grizzly & trunk(!?)

• Cinder (Ceph)

• Quantum (Midokura)

• Hypervisors (Hyper-V, bare metal)

• Databases (PostgreSQL)

• Operating Systems (RHEL, Debian, SUSE)

• HA Configurations

Chef for OpenStack When: (Roadmap)

Tuesday, January 22, 13

github.com/mattray/openstack-chef-docsmoving to

github.com/opscode/chef-docs

Chef for OpenStack: How

Tuesday, January 22, 13

Working on a permanent URLHTML, PDF, Epub for your iPad

Questions?

Boston 1/22/2013

Matt Raymatt@opscode.com

@mattray

Tuesday, January 22, 13

ChefConf 2013 April 2013 "OPSCODE-MEETUP"Metarepo and CI work