OpenStack Deployments with Chef
OpenStack ConferenceSpring 2012
Matt [email protected]
IRC/Twitter/LP/GitHub: mattraywww.opscode.com
Deploying and Managing OpenStack is not simple.
Chef makes it easier.
See Node
Application Server
See Nodes
Application Server
Application Database
See Nodes Grow
Application Server
Application Databases
Application Servers
Application Databases
See Nodes Grow
Application Servers
Application Databases
Load Balancer
See Nodes Grow
See Nodes Grow
Application Servers
Application Databases
Load Balancers
See Nodes Grow
Application Servers
Application Database Cache
Load Balancers
Application Databases
Tied together with Config
Application Servers
Application Database Cache
Load Balancers
Application Databases
Infrastructure is a Snowflake
Application Servers
Application Database Cache
Load Balancers
Floating IP?
Application Databases
Evolving Complexity
Load Balancers
Application Servers
NoSQL
Database Slaves
ApplicationCache
Database Cache
Database
Complexity Grows Quickly
DC1
DC3
DC2
http://www.flickr.com/photos/16339684@N00/2681435235/
And it Continues to Evolve
Chef is Infrastructure as Code
http://www.flickr.com/photos/louisb/4555295187/
• Programmatically provision and configure
• Treat like any other code base
• Reconstruct business from code repository, data backup, and bare metal resources.
• Chef-Client generates configurations directly on nodes from their run list
• Reduce management complexity through abstraction
• Store the configuration of your programs in version control
http://www.flickr.com/photos/ssoosay/5126146763/
Nodes
Collections of Resources
• Networking
• Files
• Directories
• Symlinks
• Mounts
• Routes
• Users
• Groups
• Tasks
• Packages
• Software
• Services
• Configurations
• Other Stuffhttp://www.flickr.com/photos/stevekeys/3123167585/
Declarative Interface to Resources
• Define policy
• Say what, not how
• Pull not Push
http://www.flickr.com/photos/bixentro/2591838509/
Ruby!
extra_packages = case node['platform'] when "ubuntu","debian" %w{ ruby1.8 ruby1.8-dev rdoc1.8 ri1.8 libopenssl-ruby } endextra_packages.each do |pkg| package pkg do action :install endend
Recipes and Cookbooks
• Recipes are collections of Resources
• Cookbooks contain recipes, templates, files, custom resources, etc
• Code re-use and modularity
• Hundreds already on Community.opscode.com
http://www.flickr.com/photos/shutterhacks/4474421855/
http://www.flickr.com/photos/kathycsus/2686772625
• IP addresses
• Hostnames
• FQDNs
• Search for nodes with Roles
• Find configuration data
Search
pool_members = search("node","role:webserver”)
template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]"end
Pass Results to Templates
# Set up application listeners here.listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%><% if node["haproxy"]["enable_admin"] -%>listen admin 0.0.0.0:22002 mode http stats uri /<% end -%>
Pass Results to Templates
Jboss App
Memcache
Postgres Slaves
Postgres Master
So when this
NagiosGraphite
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
Becomes this
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
Updates can be automatic
NagiosGraphite
Count the resources
Jboss App
Memcache
Postgres Slaves
• Load balancer config
• Nagios host ping
• Nagios host ssh
• Nagios host HTTP
• Nagios host app health
• Graphite CPU
• Graphite Memory
• Graphite Disk
• Graphite SNMP
• Memcache firewall
• Postgres firewall
• Postgres authZ config
• 12+ resource changes for 1 node addition
Build anything
• Simple internal applications
• Complex external applications
• Workstations
• Hadoop clusters
• IaaS infrastructure
• PaaS infrastructure
• SaaS applications
• Storage systems
• You name it
http://www.flickr.com/photos/hyku/245010680/
And manage it simply
http://www.flickr.com/photos/helico/404640681/
• Automatically reconfigure everything
• Linux, Windows, Unixes, BSDs
• Load balancers
• Metrics collection systems
• Monitoring systems
• Cloud migrations become trivial
The Chef Community
• Apache License, Version 2.0
• 675+ Individual contributors
• 125+ Corporate contributors
• Dell, Rackspace, VMware, DreamHost, Joyent, Heroku, RightScale, Deutsche Telekom and many more
• 450+ cookbooks
• http://community.opscode.com
Resources
openstack-cookbooksopenstack-chefchef-openstack
crowbar
What's Out There?
Opscode
• Bexar/Cactus, now deprecated
• original source for many other efforts
• Crowbar
• Mercado Libre
• HP
• Apache 2 license
• http://wiki.opscode.com/display/chef/Deploying+OpenStack+with+Chef
• github.com/mattray/openstack-cookbooks
TryStack
• TryStack.org, NTT & others
• Smokestack testing
• Xen & PostgreSQL
• Apache 2 license
• #trystack on irc.freenode.org & Facebook
• github.com/openstack/openstack-chef
Rackspace Cloud Builders
• Rackspace, DreamHost, Opscode & others
• KVM & MySQL on Ubuntu 12.04 (others)
• Apache 2 license
• #openstack-chef on irc.freenode.net
• github.com/osops/
• monitoring, logging, operations tools
• github.com/rcbops/chef-cookbooks
Crowbar
• Datacenter installer from Dell
• Extension of the Chef server
• "Barclamps" contain cookbooks + more
• Dell released under the Apache 2 license
• Dell, enStratus, Opscode & many others
• Mailing list and Skype
• github.com/dellcloudedge/crowbar
Deploying OpenStack
• Chef ties it all together automatically
• Scaling changes how we deploy
• Interchangeable components
• Configurations shared, supported & documented
• Licensing makes it available to everyone
knife openstack
knife openstack
$ knife openstackAvailable openstack subcommands: (for details, knife SUB-COMMAND --help)
** OPENSTACK COMMANDS **knife openstack flavor list (options)knife openstack image list (options)knife openstack server create (options)knife openstack server delete SERVER [SERVER] (options)knife openstack server list (options)
knife openstack flavor list
$ knife openstack flavor listID Name Virtual CPUs RAM Disk 1 m1.tiny 1 512 MB 0 GB 2 m1.small 1 2048 MB 20 GB 3 m1.medium 2 4096 MB 40 GB 4 m1.large 4 8192 MB 80 GB 5 m1.xlarge 8 16384 MB 160 GB
knife openstack image list
$ knife openstack image listID Name 13 natty-server-cloudimg-amd64 12 natty-server-cloudimg-amd64-kernel 15 oneiric-server-cloudimg-amd64 14 oneiric-server-cloudimg-amd64-kernel
knife openstack server create --node-name ko1 --flavor 1 --image 13 -S trystack
knife openstack server create
$ ssh -i ~/.ssh/trystack.pem [email protected] authenticity of host '8.21.28.24 (8.21.28.24)' can't be established.RSA key fingerprint is 0c:d8:3e:34:d1:de:c4:ee:5f:bc:b5:89:11:0d:73:e0.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added '8.21.28.24' (RSA) to the list of known hosts.Welcome to Ubuntu 11.04 (GNU/Linux 2.6.38-13-virtual x86_64)
* Documentation: https://help.ubuntu.com/
System information as of Thu Feb 16 23:43:29 UTC 2012
System load: 0.08 Processes: 63 Usage of /: 40.8% of 1.35GB Users logged in: 0 Memory usage: 6% IP address for eth0: 8.21.28.24 Swap usage: 0%---------------------------------------------------------------------<snip>Get cloud support with Ubuntu Advantage Cloud Guest http://www.ubuntu.com/business/services/cloud
The programs included with the Ubuntu system are free software;the exact distribution terms for each program are described in theindividual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted byapplicable law.
To run a command as administrator (user "root"), use "sudo <command>".See "man sudo_root" for details.
ubuntu@ko1:~$
Chef for Infrastructure Portability
• knife ec2
• knife rackspace
• knife hp
• knife openstack
• ... and many others
What's Next?
http://www.flickr.com/photos/felixmorgner/4347750467/
knife-openstack Roadmap
• http://wiki.opscode.com/display/chef/OpenStack+Bootstrap+Fast+Start+Guide
• Merges & publish on rubygems.org
• Testing with Essex
• tickets.opscode.com/browse/KNIFE_OPENSTACK
Chef Roadmap
• Chef 0.10.10
• Full Stack installer
• Linux, BSDs, OSX, Solaris (Illumos), Windows
• Chef 11
• Reporting
• Dry Run
• Additional Hypervisors
• More Databases
• Operations Support
• HA Configuration
• Additional Operating Systems
• Folsom (Quantum, Melange, etc.)
OpenStack Cookbooks Roadmap
• Collaborate with the Community
• Reference cookbooks for Essex
• OpenStack Quick Start tutorial
• Refresh Deploying OpenStack with Chef
• Knife OpenStack enhancements
Opscode's OpenStack Roadmap
Recommended
