A worthy goal or a fool’s errand?

Continuous Integration + OpenStack

#rackstackatl2

• Brian Lamar– Public Cloud Deployments– OpenStack contributor since Wed Feb 16 16:06:47 2011 – Ansible– Puppet– Python

Who Am I?

#rackstackatl

What is “Continuous Integration” and how does it apply to OpenStack?

#rackstackatl4

• Continuous Delivery• Continuous Deployment

What We’re Not Talking About (Right Now)

#rackstackatl5

• OpenStack always works• Allows for more frequent deployments

What are the benefits of Continuous Integration?

#rackstackatl6

• Maintain a Single Source Repository• Automate the Build• Make Your Build Self-Testing• Everyone Commits to the Mainline • Every Commit Should Build the Mainline on an Integration Machine• Keep the Build Fast• Test in a Clone of the Production Environment• Make it Easy for Anyone to Get the Latest Executable• Everyone Can See What’s Happening• Automate Deployment

Practices of Continuous Integration

Source: http://martinfowler.com/articles/continuousIntegration.html

#rackstackatl7

review.openstack.orgJenkinsZuul

XConstant interoperability between projects

Maintain a Single Source Repository

New Nova Commit Install dependencies from pip

pbr>=0.6,!=0.7,<1.0

SQLAlchemy>=0.7.8,<=0.9.99

amqplib>=0.6.1

...

python-keystoneclient>=0.7.0

Should be from git and not pip!

#rackstackatl8

DevStack for “basic” deploys3rd party voting bots for everything else

Automate The Build

#rackstackatl9

Unit / Integration tests includedTempestXNo hard requirements

Make Your Build Self-Testing

#rackstackatl10

XNo easy way to enforceXNo consensus on what a small patch is

Everyone Commits to the Mainline Every Day

#rackstackatl11

We do this

Every Commit Should Build the Mainline on an Integration Machine

#rackstackatl12

XProject test suites are still slowNot sure this really matters

Keep the Build Fast

#rackstackatl13

3rd party voting bots

Test in a Clone of the Production Environment

#rackstackatl14

XNo packages produced upstream on each commitXNo consistent release for PyPI packages

• python-*client

Make it Easy for Anyone to Get the Latest Executable

#rackstackatl15

Great gate + 3rd party gate3rd party bot output requirementsXInformation overload

Everyone Can See What’s Happening

#rackstackatl16

XNo official deployment methodXWhat is a deployment?

Left to 3rd party deploy bots

Automate Deployment

#rackstackatl

What is Rackspace doing now?

#rackstackatl18

• Daily pulls of upstream• Merge our custom code repositories (plypatch)• Build custom virtual environments• Run unit tests• Build tarball packages• Deploy tarball packages• Run tests• Promote from CI -> Preproduction• Run tests• Staggered production deployments

What is Rackspace doing now?

#rackstackatl19

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

It takes us ~21 days choose a release branch… then ~45 days before we look at the upstream code again.

#rackstackatl20

Meet Our Patches

#rackstackatl21

• CI• QE• Pre-Production• Network Dev• London• Dulles (VA)• Dallas• Chicago• Sydney• Hong Kong

Meet Our Environments

~ 40,000 things to deploy to

#rackstackatl22

• 90% of my time is identifying and fixing issues with…– Issues our code caused in deployment– Our Jenkins infrastructure– Our packaging process– Our deployment process

• Integration issues with– Billing– Auth– Others

So, what takes so long?

#rackstackatl

How do we improve?

#rackstackatl24

• Stop carrying our own patches• Stop carrying our own patches• Stop carrying our own patches• Packaging upstream• Deployment methods upstream (?)

How do we improve?

#rackstackatl25

• Speeds development time– But at what cost?

• Proprietary features– Please no

• Proprietary integration– This makes sense

Why do we have custom patches?

#rackstackatl26

• Plugins• Configuration options• Billing system story• Developers are going to hate me

Allowing for Proprietary Integration

#rackstackatl27

• For OpenStack– Talk about official deployments– Upstream packaging– Project interoperability– Strongly encourage smaller commits– Keep encouraging tests

Going Forward Into Juno

• For Rackspace– Stop carrying patches– Encourage plugable code– Comment on Gerrit with our deployment test results

#rackstackatl28

• Continuous Delivery by Jez Humble and David Farley• The Phoenix Project by Gene Kim, Kevin Behr, and George Spafford

Fun Reads

#rackstackatl

RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM

RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COMRACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM

RACKSPACE® HOSTING | 5 MILLINGTON ROAD | HAYES, UNITED KINGDOM UB3 4AZUK SALES: +44 (0)20 8712 6507 | UK SUPPORT: 0800 988 0300 | WWW.RACKSPACE.CO.UK