Considerations for Operating an OpenStack Cloud

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 1

Mark T. Voelker, Technical Leader @ Cisco

OpenStack ATC/StackForge Puppet Core/Foundation Member #54

All Things Open 2014


@marktvoelker

• Tech Lead at Cisco, StackForge Puppet core developer, OS Foundation Member #54

• Fact: can be bribed with doughnuts

• Currently works in Cisco’s Cloud & Virtualization Group

• In copious (hah!) spare time: OpenStack solutions, Big Data, Massively Scalable Data Centers, Devops, making sawdust with extreme prejudice

https://twitter.com/marktvoelker


• Tech lead, manager, software developer, architect

• Started in OpenStack in 2011 at the Diablo Design Summit


The great thing about my job is that I get to have fun exploring a lot of new things…


….and I get to help build a LOT of clouds.


Today’s talk won’t be overly formal….


…because I tend to get excited by this stuff.




……then you know how to get to Day 1.

Now let’s talk about getting to Day 30…


• Architecture

• Components

• High Availability

• Bare metal bring-up

• Config management

• CI/CD

• Packaging

• Automated test

• Monitoring

• Up/down alerting

• Trending data

• Logging and log search


High

Availability?

Sounds

great--I’ll

take two!


• Consider whether you want active/active or active/passive

• Setup and tooling differs a bit, but I generally like active/active

• Note that docs.openstack.org has an HA Guide

• A bit dated…patches welcome!

• Prioritize HA for the control plane

• That also means thinking about your database, network, and RPC bus

• Instance-level HA: there be dragons

• But yes, it’s being looked at

• Pets vs cattle

• Note: HA == more hardware

• Some components need at least 3 nodes

http://docs.openstack.org/high-availability-guide/content/index.html

https://wiki.openstack.org/wiki/How_To_Contribute#If_you.27re_into_doc.2C_we.27d_love_your_help

http://blog.russellbryant.net/2014/10/15/openstack-instance-ha-proposal/


• Stuff OpenStack needs to run: message brokers

• Check out RabbitMQ clustering and mirrored queues

• Check out Galera for MySQL/MariaDB

• I usually see Percona XtraDB

• Frontend with an HAProxy/Keepalived pair


• Don’t do rabbit clustering

over a WAN

• Be aware of the SELECT…

FOR UPDATE issue


• Long story short: Neutron and some parts of Nova invoke an SQL pattern known as “SELECT…FOR UPDATE” which Galeradoesn’t support due to issues with cross-node locking.

• Can cause deadlocks symptoms.

• Neutron/nova code being refactored to remove, but will likely not be done until at least Kilo.

• Meanwhile: use HAProxy to send writes to a single Galera node and you should be fine

• With the obvious scalability bottleneck

• More info here.

• Thank Jay Pipes & Peter Boros for

the find!

http://lists.openstack.org/pipermail/openstack-dev/2014-May/035264.html


• Use Swift, Ceph, or other highly available storage to back Glance

• Pick a highly available storage backend for Cinder too

• Use Keepalived/HAProxy to front-end multiple API servers

• Or another load balancer technology of your choice

• Can be deployed as dedicated nodes for scale, or cohabitate

• Network: DVR vs Provider Network Extensions

• Distributed Virtual Routers are a new experimental feature in Juno (not yet ready for production)

• Please go test it and report/fix bugs!

• Provider networks essentially punt the availability issue to your physical network

• Allows you to use standard tools like virtual port channels and VRRP

• Also highly performant

https://wiki.openstack.org/wiki/Neutron/DVR/HowTo


• Architecture

• Components




• CI/CD• Packaging

• Automated test

• Monitoring• Up/down alerting

• Trending data



We start with bare metal.


• For a cloud of any real size, you don’t want to be installing operating systems by hand

• Remember that baremetal bringup actually isn’t something that just happens once…often recurs for upgrades, capacity expansion, etc.

• Baremetal bringup tools can also have other uses, like inventory or bootstrapping configuration management agents.


• A simple (~15k lines of Python code) tool for managing baremetaldeployments

• Flexible usage (API, CLI, GUI)

• Allows you to define systems (actual machines) and profiles (what you want to do with them)

• Provides hooks for Puppet so you can then do further automation once the OS is up and running

• Provides control for power (via IPMI or other means), DHCP/PXE (for netbooting machines), and more.




• Razor• Developed by EMC, managed by Puppet Labs (occasionally used with Chef

too)

• Initial release in 2012

• Uses a “microkernel” loaded onto the machine to gather facts before provisioning

• Tag + Policy model

• Crowbar• Originally written by Dell, now a community project

• Originally designed to deploy OpenStack on all the way from baremetal

• Now deploys other stuff too (namely, Hadoop)

• Uses Chef to handle everything after the OS install

• Foreman• Used by Red Hat among others

• Does baremetal bringup and serves as a Puppet ENC

https://puppetlabs.com/solutions/next-generation-provisioning/

http://www.slideshare.net/mattray/bare-metal-to-openstack-with-razor-and-chef

http://www.dell.com/Learn/us/en/555/cloud-computing/crowbar-software-framework?c=us&l=en&s=biz

http://theforeman.org/


• Architecture

• Components





• Automated test


• Trending data




“Cloud isn’t just an infrastructure technology….it’s a new operations model. And with OpenStack in particular, it’s one that’s very well suited to a DevOps style of management. Many companies aren’t just adopting cloud, they’re changing how they operate.”

“Besides, logging into servers to mess with config files makes me sad.”

--That ranty guy in Raleigh again


• Remember, OpenStack is a set of interoperating distributed systems

• That means you’re going to have a lot of software to configure on a lot of machines

• You’re probably going to want to make changes over time

• You’re probably going to have more than one person touching your cloud

• CM tools help you treat configuration as code, so you can collaborate more easily


Pile of

Bash

Scripts



• An increasingly common pattern:

• Puppet or Chef for configuration management, PLUS

• Ansible or Salt for cross-node orchestration

• Recommendation: use the tools that work for you!

• But remember: you don’t have to do it alone.

• Several CM tools have thriving collaborators in the OpenStack community

• Links for later:

• Puppet for OpenStack

• Chef for OpenStack

• Ansible for OpenStack

• SaltStack for OpenStack

• Pile of bash scripts for OpenStack

https://wiki.openstack.org/wiki/Puppet-openstack

http://docs.getchef.com/openstack.html

https://github.com/openstack-ansible

http://docs.saltstack.com/en/latest/topics/cloud/openstack.html


• Unit tests for your deployment code are a good idea

• ServerSpec tests to make sure your config management system did what it was supposed to are great

http://serverspec.org/


• Architecture

• Components





• Automated test


• Trending data



…well, haven’t you always wanted a butler?


• DevOps: actually pretty handy

• OpenStack change velocity (community’s and yours)

• Anecdote: the majority of deployments I work with have some customizations or backports from future releases

• It’s not just OpenStack, it’s all the underpinning components and your CM code too!


• OpenStack itself uses CI/CD tools in it’s development process…you should consider using them in your cloud buildouttoo!

• The OpenStack Infra team has created some awesome tools: JJB, Zuul, etc

• They’re all open source and you can even see how OpenStack’s own CI is set up (check out Elizabeth Joseph’s slides from yesterday for more!).

• The basics:

• An integration server (Jenkins, Go, Travis, etc)

• A code review and repository tool (Gerrit, Cgit, GitHub, etc)

• A battery of automated tests (lint checks, rspec-puppet, Tempest, Rally, etc)

• Some form of packaging (rpmbuild/mock, sbuilder/pbuilder, etc)

• An artifact repository (Artifactory, yum/apt repos, etc)

• Optionally, some deployment jobs (usually powered by your CM tool)

http://ci.openstack.org/

http://allthingsopen.org/speakers/elizabeth-joseph/

https://wiki.openstack.org/wiki/Puppet-openstack#Rspec_puppet_tests

http://docs.openstack.org/developer/tempest/

https://wiki.openstack.org/wiki/Rally


• …you never intend to change the code yourself

• …building your own packages would violate a support contract with your distribution

• …you’ve never used a CI/CD pipeline before (but really: you should start learning)

• …you have a static environment that absolutely will not change, need to add capacity, etc.


• Architecture

• Components





• Automated test


• Trending data



• Now that you have a cloud, you’ll probably want to know that all it’s parts stay in good working order.




• I’ve worked on a lot of OpenStack clouds and almost everyone has their own preferred monitoring toolset.

• One possible exception: almost everybody seems to love Graphite.

• The golden rule is: use the tools that work for you!

• Very often this will be whatever you’re using in the rest of your infrastructure.

• Break it down into at least two buckets:

• Up/down and alerting (ex: Nagios or it’s derivatives…yes, there are OpenStack plugins out there on NagiosExchange)

• Trending data collection/plotting (ex: collectd/statsd feeding graphite)

• Also: use your peers!

• Check out Tong Li’s Monitoring as a Service talk later today!

• Operators often willing to share, so ask on the openstack-operators list.

http://graphite.wikidot.com/

http://allthingsopen.org/talks/monitoring-as-a-service/

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


• Architecture

• Components




• CI/CD

• Packaging

• Automated test

• Monitoring

• Up/down alerting

• Trending data




• Distributed systems generate logs…..all over the place.

• Finding the root of problems may mean correlating logs from different machines…but which?

• OpenStack in particular *can* be pretty verbose

• You may also be dealing with logs from other distributed tools in your cloud (RabbitMQ, databases, etc)

• Generally you want to get logs together, be able to search them, and be able to visualize them.


Unlike monitoring tools, there seems to be pretty broad consensus on good tools here in deployments I’ve worked with….


http://www.elasticsearch.org/blog/openstack-elastic-recheck-powered-elk-stack/

(visualization)

(collection)

(search/analytics)


Questions?@marktvoelker

http://openstack.org/

http://cisco.com/go/openstack/

(yes, we’re hiring!)

http://twitter.com/marktvoelker

http://openstack.org/

http://cisco.com/go/openstack/

Technology

Considerations for Operating an OpenStack Cloud