A CI/CD Pipeline to Deploy and Maintain OpenStack - cfgmgmtcamp2015

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

Simon McCartney, Site Reliability Engineering

cfgmgmtcamp 2015

CI/CD Pipeline to Deploy and Maintain an OpenStack IaaS Cloud


Who is this guy anyway?● Dev, then Ops, then Management, back to WebOps● Solaris & Makefiles● CentOS w/ kickstart & hairy bash/php → puppet● Chef ● Salt● Likes to take the best tools to the next job (kitchen-salt)


Background

● This project predates HP Helion OpenStack

● We’re using Ubuntu 12.04, OpenStack Grizzly & SaltStack 0.16.2


Why?


Why Continuous Integration & Delivery?

• Repeatable, Reliable & Incrementally Improving

• Constant flow of changes to benefit users



● Infrastructure as Code > Infrastructure as Art

● Frequent Small Batches > Infrequent Big Batches



● Time Vampires

● Manual Builds

● Manual Testing

● Improved consistency & quality

● Faster Cycle Time


Why Continuous Integration & Delivery

● Unit Tests - salt formulae● Integration Tests - do these salt formulae work together?● End-to-End Tests - does the OpenStack Cloud we built work?● Performance Test - did we break/de-tune something?● Deployment Tests - do we have everything?● Configuration Tests - can we test different environments before

hitting the real environment?


Our Pipeline


Our Tech Stack● Vendor packages for OpenStack (Ubuntu)

● Configuration management (SaltStack)

● Software Engineering● git, gerrit, gitshelf & jenkins● test-kitchen (kitchen-salt, serverspec)

● Infrastructure Engineering● Vagrant/VirtualBox & Public Cloud


Our Pipeline1. Individual Salt Formula development2. Personal multi-node, package based OpenStack environment on your

workstation for dev & validation3. Go public – push to gerrit for review & automated testing4. Pull in to the deploy-kit5. deploy-kit tooling kicks in & builds deploy artefacts6. Auto deploy to ephemeral public cloud test environment7. Deploy to physical staging environment8. Ready for production



Validating & Approving a change• gerrit - defacto web based open source code review system

• submit changes (a patch set) for review by your peers..• ..and for testing by Jenkins

• Jenkins, Jenkins Job Builder & Gerrit Trigger• Gerrit plugins allow certain jobs to be triggered on patch-set events

(create, merge etc)• results of these jobs are then recorded in gerrit

• test-kitchen• test harness to execute your configuration management code in isolation (i.e.

fresh VMs)• providers (vagrant, LXC, openstack, gce, aws etc)• provisioners (Chef, Salt, Puppet)• testing frameworks (Bats, RSpec, serverspec etc)• can be used interactively in your dev environment as well as in validation jobs


Jenkins Job Builder● Because nobody like XML● Generate Jenkins Jobs from re-useable & composable YAML

fragments




gitshelf● Accidentally re-invented the wheel ● BUT, it's still very useful● ~= Librarian Puppet or Berkshelf or dependencies.txt● Specify specific versions of the contents of a git repo at specific

location (tag, branch, SHA1 etc)● Also supports variable expansion & symlink management

● You should probably use repo from Google's AOSP



In-cloud validation

1. contractor creates instances & networks

2. bootstrap.sh used prepare vanilla image for use with salt

3. Jenkins uploads release artifacts and deploys salt states

4. Performa validation tests

"roles": {

...

"compute": {

"image": "261844b3-479c-...",

"flavor": "101",

"keypair": "jenkins",

"instances": {

"ae1": [{

"az": "az3",

"number": 1,

"nics": [{

"network": "public-net",

"fixed_ip": "172.16.0.101",

"floating_ip": "15.126.241.109"

},

{

"network": "private-net",

"fixed_ip": "10.0.0.101"

}]


Deployment Automation

• Production Deploys• Move slowly when required• Confirm host is still “good” using existing monitoring framework• Caution over nova-* service restarts

• Next• rundeck• chatops


Links ● http://devops.com/blogs/continuous-delivery-pipeline/● http://martinfowler.com/bliki/DeploymentPipeline.html● https://github.com/secure-pipeline/jenkins-example● https://speakerdeck.com/garethr/continuous-integration-for-infrastructure

● https://github.com/gitshelf/gitshelf● https://github.com/moniker-dns/contractor● https://github.com/macgreagoir/gobstack

https://github.com/secure-pipeline/jenkins-example





https://speakerdeck.com/garethr/continuous-integration-for-infrastructure

https://github.com/gitshelf/gitshelf

https://github.com/gitshelf/gitshelf

https://github.com/moniker-dns/contractor

https://github.com/moniker-dns/contractor


Questions?


Thank you!

Simon McCartney@simonmcc


Internet

A CI/CD Pipeline to Deploy and Maintain OpenStack - cfgmgmtcamp2015