A CI/CD Pipeline to Deploy and Maintain OpenStack - cfgmgmtcamp2015

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

Simon McCartney, Site Reliability Engineering

cfgmgmtcamp 2015

CI/CD Pipeline to Deploy and Maintain an OpenStack IaaS Cloud

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

Who is this guy anyway?● Dev, then Ops, then Management, back to WebOps● Solaris & Makefiles● CentOS w/ kickstart & hairy bash/php → puppet● Chef ● Salt● Likes to take the best tools to the next job (kitchen-salt)

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

Background

● This project predates HP Helion OpenStack

● We’re using Ubuntu 12.04, OpenStack Grizzly & SaltStack 0.16.2

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

Why?

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

Why Continuous Integration & Delivery?

• Repeatable, Reliable & Incrementally Improving

• Constant flow of changes to benefit users

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

Why Continuous Integration & Delivery?

● Infrastructure as Code > Infrastructure as Art

● Frequent Small Batches > Infrequent Big Batches

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

Why Continuous Integration & Delivery?

● Time Vampires

● Manual Builds

● Manual Testing

● Improved consistency & quality

● Faster Cycle Time

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

Why Continuous Integration & Delivery

● Unit Tests - salt formulae● Integration Tests - do these salt formulae work together?● End-to-End Tests - does the OpenStack Cloud we built work?● Performance Test - did we break/de-tune something?● Deployment Tests - do we have everything?● Configuration Tests - can we test different environments before

hitting the real environment?

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

Our Pipeline

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

Our Tech Stack● Vendor packages for OpenStack (Ubuntu)

● Configuration management (SaltStack)

● Software Engineering● git, gerrit, gitshelf & jenkins● test-kitchen (kitchen-salt, serverspec)

● Infrastructure Engineering● Vagrant/VirtualBox & Public Cloud

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

Our Pipeline1. Individual Salt Formula development2. Personal multi-node, package based OpenStack environment on your

workstation for dev & validation3. Go public – push to gerrit for review & automated testing4. Pull in to the deploy-kit5. deploy-kit tooling kicks in & builds deploy artefacts6. Auto deploy to ephemeral public cloud test environment7. Deploy to physical staging environment8. Ready for production

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

Validating & Approving a change• gerrit - defacto web based open source code review system

• submit changes (a patch set) for review by your peers..• ..and for testing by Jenkins

• Jenkins, Jenkins Job Builder & Gerrit Trigger• Gerrit plugins allow certain jobs to be triggered on patch-set events

(create, merge etc)• results of these jobs are then recorded in gerrit

• test-kitchen• test harness to execute your configuration management code in isolation (i.e.

fresh VMs)• providers (vagrant, LXC, openstack, gce, aws etc)• provisioners (Chef, Salt, Puppet)• testing frameworks (Bats, RSpec, serverspec etc)• can be used interactively in your dev environment as well as in validation jobs

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

Jenkins Job Builder● Because nobody like XML● Generate Jenkins Jobs from re-useable & composable YAML

fragments

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

gitshelf● Accidentally re-invented the wheel ● BUT, it's still very useful● ~= Librarian Puppet or Berkshelf or dependencies.txt● Specify specific versions of the contents of a git repo at specific

location (tag, branch, SHA1 etc)● Also supports variable expansion & symlink management

● You should probably use repo from Google's AOSP

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

In-cloud validation

1. contractor creates instances & networks

2. bootstrap.sh used prepare vanilla image for use with salt

3. Jenkins uploads release artifacts and deploys salt states

4. Performa validation tests

"roles": {

...

"compute": {

"image": "261844b3-479c-...",

"flavor": "101",

"keypair": "jenkins",

"instances": {

"ae1": [{

"az": "az3",

"number": 1,

"nics": [{

"network": "public-net",

"fixed_ip": "172.16.0.101",

"floating_ip": "15.126.241.109"

},

{

"network": "private-net",

"fixed_ip": "10.0.0.101"

}]

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

Deployment Automation

• Production Deploys• Move slowly when required• Confirm host is still “good” using existing monitoring framework• Caution over nova-* service restarts

• Next• rundeck• chatops

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

Links ● http://devops.com/blogs/continuous-delivery-pipeline/● http://martinfowler.com/bliki/DeploymentPipeline.html● https://github.com/secure-pipeline/jenkins-example● https://speakerdeck.com/garethr/continuous-integration-for-infrastructure

● https://github.com/gitshelf/gitshelf● https://github.com/moniker-dns/contractor● https://github.com/macgreagoir/gobstack

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

Questions?

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

Thank you!

Simon McCartney@simonmcc

© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.