View
490
Download
2
Category
Preview:
Citation preview
Deploying OpenStack Using Docker in Production
Overview• The Pain of Operating Openstack• Possible Solutions• Why Docker Works• Why Docker Doesn’t Work• Docker @ TWC• Lessons Learned
• Docker in production in July 2015• First service was Designate• Added Heat, Nova and Keystone• Nova using Ceph and Solidfire Backends• Neutron in progress• Glance and Cinder later this year• Using Docker 1.10 and Docker Registry V2
Docker & OpenStack @ TWC
• Started with packages for deployments• Don’t like big-bang upgrades• Want to be able to carry local patches• Want to run mixed versions of services• Smaller upgrades, more often
How Did We End Up Here?
Why Not Packages?• Built packages for Keystone• Worked for local patches• Worked for updating stable branches• Doesn’t work for mixed releases• Limited by distro python packaging• Packaging workflow is a pain• Packages slow down your workflow• Package may not exist yet
Why Not Python Virtual Envs?• Deployed Designate with Virtual Envs• Mirrored Python packages internally• Built Virtual Envs on servers• Was slow to deploy• Still have to install/manage non-Python deps
Why Docker?
Everyone Else Is Doing It?
• Reproducible builds• Easy to distribute artifacts• Contains all dependencies• Easy to install multiple versions of an image
Why Docker?
• Restarting docker restarts containers • Intermittent bugginess• Complex services are hard to fit into Docker• Requires new tooling for build/deployment/etc
Why Not Docker?
Docker @ TWC: Images• Building base images using debootstrap• Build openstack-dev image based on that
–Contains all common deps• Image per OpenStack Service• Per service base requirements.txt and a frozen one• Frozen requirements.txt is used for image builds• Uses upper-constraints.txt for frozen requirements1
1. https://github.com/openstack/requirements/blob/master/upper-constraints.txt
Docker @ TWC: Image Tags• Tag should:
– Identify OpenStack service version– Identify tooling version–Be automatically generated–Be unique
Docker @ TWC: Image Tags
5.0.1-9-g0441ca8.16.dd35404
5.0.1-9-g0441ca8 16 dd35404
git-describe for Heat Tooling # commits Tooling commit hash
Docker @ TWC: Image Distribution• Using Docker Registry V2• Registry using file backend for local storage• Publish to master registry via Jenkins• Replicate to registry mirrors via rsync• Mirrors provide read-only access to images• No dependency on production environment
Docker @ TWC: Deployments• Images installed with puppet-docker• Managed with twc-openstack/os_docker• Worked with Puppet OpenStack project to add
hooks for software and service management• The os_docker module uses these to extend
OpenStack Puppet modules
Docker Registry Scaling• Docker recommends (almost requires) TLS for
registry• We deploy to 20 hypervisors in parallel• 8 vCPU Docker Registry• Supports concurrent 40 pulls * 500mb images• Size your registry for concurrent pulls * image size
Beware Docker Networking• We use --net host for all containers• Many services *require* --net host• Docker always creates bridge and NAT rules• NAT rules aren’t tied to a specific interface• Docker picks unused network range
–But can’t see VM IP addresses• Found this out on first Nova Compute deploy
OpenStack Upgrades With Docker• Allows upgrading single services!• Allows staging the upgrade images ahead of time• Not exciting
Why Not Kolla?• At the time didn’t meet our requirements:
–Didn’t support plugins, no source build–These things are resolved, or being resolved
• Great reference for running OpenStack with Docker• Recommended
Questions?Clayton O’Neill
– clayton.oneill@twcable.com– IRC: clayton– Twitter: @clayton_oneill
Eric Peterson–eric.peterson1@twcable.com– IRC: ducttape_–Twitter: @_ducttape
Recommended