Towards Continuous Deployment with Django

1

Towards Continuous Deployment with Django

Roger Barnes@mindsocket

[email protected]

PyCon Australia 2012

BTech ICS

Co-founder @ Arribaa

Python/Django

Web development

Frisbee

Photography

Adventure

Beer

1997 1999 2001 2003 2005 2007 2009 2011

Concepts from other talks

● The why– Lean startups and customer discovery– Do more with less

● The how– EC2 / provisioning– Architecture– Developing for cloud deployment– Monitoring live web applications– IaaS vs PaaS

What is continuous delivery*

"Rapid, incremental, low-risk delivery of high quality, valuable new functionality to users through automation of the build, testing and deployment process" - Jez Humble

Deploying every good version of your software...

… or at least being able to

"Minimising MTTBID (Mean Time to Bad Idea Detection)" – Etsy

* delivery == deployment for the purposes of this talk

More Than Technology

● Technology - For automation● People - Cross-functional team, organised around

product● Processes – Conventions and some glue that

technology can't automate

Why

● Fail fast, win fast - Build, Measure, Learn● Competitive advantage● Avoid YAGNI - do less● Less manual process, less risk● Real-time control

– Deploy on demand– Decoupled deployment and release– Self service environments

Continuous Delivery in Practice

● Single path to production● Optimise for resilience● Get comfortable with being uncomfortable● Automate as much as possible● If it hurts, do it more often● Becomes natural, return on investment is fast/high● Lots of ways to do it, here's what I've done...

Environments

● Make dev/test/prod as similar as possible– Full stack, no "./manage.py runserver"– Some exceptions make sense. eg for development:

● dummy email backend● dummy message broker● fewer threads/workers● less memory● less analytics instrumentation

Develop Commit Build Stage Deploy Measure

Environments

Dev/test virtualised using Vagrant

Create and configure lightweight, reproducible, and portable development environments

$ vagrant up

$ vagrant ssh


Environments

Vagrantfile configures base image, provisioning, shared folders, forwarded ports

Vagrant::Config.run do |config| config.vm.box = "precise64" config.vm.box_url = "http://files.vagrantup.com/precise64.box"

config.vm.provision :puppet do |puppet| puppet.manifests_path = "puppet/vagrant-manifests" puppet.manifest_file = "dev.pp" puppet.module_path = "puppet/modules" end

config.vm.forward_port 80, 8000 config.vm.forward_port 3306, 3306 config.vm.share_folder "arribaa", "/opt/django-projects/arribaa", ".."end


Environments

● Repeatable, versioned configuration– Snowflake bad, phoenix good– Puppet (masterless) for provisioning OS and services– Fabric for scripted tasks, eg:

● copy database from production● update requirements● migrate database● commit and push to repository

● Anti-pattern– Can't spin up new environment with one command


Development top level puppet config

include uwsgiinclude statsdinclude solrinclude memcachedinclude rabbitmqinclude nginxinclude testinginclude arribaainclude arribaa::db_nodeinclude arribaa::nginxinclude arribaa::celery


Test configuration is the same

Production adds:backup

monitoring

Using Fabric with Vagrantdef vagrant(): # get vagrant ssh setup vagrant_config = _get_vagrant_config() env.key_filename = vagrant_config['IdentityFile'] env.hosts = ['%s:%s' % (vagrant_config['HostName'], vagrant_config['Port'])] env.user = vagrant_config['User']

def _get_vagrant_config(): with lcd('../vagrant'): result = local('vagrant ssh-config', capture=True) conf = {} for line in iter(result.splitlines()): parts = line.split() conf[parts[0]] = ' '.join(parts[1:])

return conf

Based on https://gist.github.com/1099132


https://gist.github.com/1099132

Dependencies

● virtualenv – separate env for each app● pip – install dependencies into virtualenv

– use requirements file– use explicit versions

● for repository based dependencies, use commit id● or fork on github


Dependencies

● Using virtualenv and pip with fabricenv.site_dir = '/opt/django-projects/arribaa'env.app_dir = '/opt/django-projects/arribaa/arribaa'env.pip_file = 'requirements.txt'

def ve_run(command, func=run, base_dir=env.app_dir, *args, **kwargs): with cd(base_dir): with prefix("source /opt/virtualenvs/%s/bin/activate" % env.virtualenv): return func(command, *args, **kwargs)

def update_reqs(): ve_run('pip install -r %s' % env.pip_file, base_dir=env.site_dir)


Database migration

● South– Schema changes– Data migrations

● Deploy separate expand and contract operations, eg:– Expand

● add new column (update model, manage.py schemamigration …)● map from old column (manage.py datamigration …)

– Deploy– Contract (optional)

● remove old column ( update model, manage.py schemamigration …)


Database migration

● Using South with fabricdef syncdb(): ve_run("python manage.py syncdb --noinput --migrate")


Tying it all together

● Update pip requirements on vagrant VM

$ fab vagrant update_reqs● Run data migration on vagrant VM

$ fab vagrant syncdb


Source control

● Pick one, learn it● Use tags to keep track of build status● Avoid long-lived branches

– or integrate them – more overhead

● Tracks all code/documentation✔ application code✔ deployment code✔ provisioning code✔ configuration code✔ documentation✗ not build artifacts


arribaa Application code├── apps Django apps│ └── ...├── static Django static files│ └── ...├── templates Django templates│ └── ...├── fabfile.py Fabric scripts├── requirements.txt Pip└── ...bin└── jenkins.sh Jenkins jobdocs└── ...

vagrant├── Vagrantfile Dev VM config└── puppet ├── modules Puppet modules │ ├── arribaa │ ├── backup │ ├── graphite │ ├── ... │ ├── uwsgi │ └── wget └── vagrant-manifests ├── dev.pp Puppet dev ├── prod.pp Puppet prod* └── test.pp Puppet test

* Not used by vagrant, but convenient to store here

Automated Testing

● Django’s test framework● Continuous Integration and Testing

– Jenkins

● django-jenkins– tests– pylint– coverage– css/jslint

● factory_boy instead of fixtures


Test separation

● Split up tests by type● Can parallelise, keeping test run times low

– < 20 minutes from commit to production

● Unit tests first, faster feedback● Run tests on different triggers, keep some out of the main

build● Etsy - Divide and Concur

– http://codeascraft.etsy.com/2011/04/20/divide-and-concur/


http://codeascraft.etsy.com/2011/04/20/divide-and-concur/

Test separation

● Unit● Functional/UI

– Selenium et al

● Staging and production smoke tests– Crawl URLs– Load/performance testing

● Flaky tests, Slow tests– exclude from regular build, run on a separate schedule


Test separation

class TestBooking(TestCase):

@pipelineTag("unit") def test_when_lapsed(self): oldbooking = factories.BookingFactory.build(when=some_old_date) self.assertTrue(oldbooking.when_lapsed())

● How to do this with Django?

– Suites

– Custom test runner

– Other test systems (Nose has an attrib plugin)● My current solution

– Annotating tests, and adapted django-jenkins command


Test separation# Used to ignore testsignore = lambda test_item: None

def pipelineTag(*args): """ Let a testcase run if any of the tags are in sys.argv and none of the tags match not-* in sys.argv. """ # Abstain if we're not in a build pipeline (ie using django-jenkins) # or if no tag(s) specified if 'jenkins_pipeline' not in sys.argv or not any(['tagged-' in arg for arg in sys.argv]): return _id

tags = args assert set(tags) < set(['unit', 'functional', 'flaky', 'slow', 'staging']) #Silently ignore if no matching tag if not any(['tagged-'+tag in sys.argv for tag in tags]): return ignore # Skip if "not-blah" tagged if any(['not-'+tag in sys.argv for tag in tags]): return skip('tagged: ' + ','.join([tag for tag in tags if 'not-'+tag in sys.argv])) # This test made it through return _id


Build Pipeline

● One pipeline● All developers feed start of pipeline via VCS (master)

– Improve policy of "don't deploy broken code"...– ...with system where it's harder to deploy broken code

● Jenkins Build Pipeline Plugin– Series of dependant jobs, each runs different tests– Feeds into pre-deploy staging job– Successful staging job enables deploy job


Build PipelineDevelop Commit Build Stage Deploy Measure

Currently simple linear flow

Build Pipeline

● Jenkins job – unit test stage:

bash -ex ./bin/jenkins.sh tagged-unit not-flaky not-slow● jenkins.sh

# Setup virtualenv, pip requirements, settings.py etc

...snip...

cd $WORKSPACE/arribaa

python manage.py clean_pyc

python manage.py jenkins_pipeline "$@"


Staging/QA Job● fab staging-deploy – very similar process to production deploy● Test run of production deployment using Fabric

– Deploy code and config to staging environment (Vagrant VM)– Apply dependency updates

● puppet config● pip requirements

– Copy database from production and run data migration

● Should now have a deployed QA environment● Browser smoke tests – currently selenium

– Not using Django 1.4 LiveTestCase here, we have a full stack to play with

● Tag successful build as deployable


Deployment Strategies

● Optimistic– Deploy, reload services – some appservers handle this well

● Blue/Green– Parallel environments, 1 idle– Idle environment gets new version

– Smoke tests and monitoring before switching over

● Canary– Take subset of app servers out of pool– Deploy to subset

– Smoke tests and monitoring before deploying rest of cluster

● Deploy whole new (app) server instance– then rewire load-balancer


Deployment with Jenkins & Fabric

Using Git tags– Set in Jenkins after successful staging run– Used by deploy in Fabric

def pull(): with cd(env.site_dir): run('git fetch') latest_tag = run('git describe --tags --match "ci-passed-staging-*" origin/master', pty=False) run('git reset --hard refs/tags/%s' % latest_tag, pty=False)


Tags set by build pipeline



Jenkins script:

virtualenv -q /opt/virtualenvs/arribaa-jenkins

source /opt/virtualenvs/arribaa-jenkins/bin/activate

pip install -r requirements.txt

pip install -r requirements-testing.txt

cd $WORKSPACE/arribaa

fab deploy



def deploy():

dbbackup() # django-dbbackup – to dropbox

changes = pull() # git fetch, git describe, git log, git reset

apply_puppet() # puppet apply … prod.pp

update_requirements() # pip install -r requirements.txt

clean_pyc() # django-extensions command

syncdb() # and –migrate (South)

collectstatic()

reload_uwsgi() # reload app server

restart_celeryd() # bounce task server

run('git tag -f ci-deployed && git push –tags') # tag deployed version

send_email(changes) # send notification


Decoupling deployment and release

● Feature flags vs feature branches– Real time control with flags– Cleanup/maint needed for flags– Harder/slower to test different combinations in either case

● Good for testing ideas (on/off, A/B, power users, QA etc)● Gargoyle

– includes optional admin interface– selective criteria (%age of users, staff only, ...)

● Alternative: Django Waffle


Measurement

● Eyes wide open– Track system and user behaviour

● Post-deployment– Error, performance and behaviour heuristics– Trend monitoring– Goal conversion rates


MeasurementDevelop Commit Build Stage Deploy Measure


Measurement - Munin



Measurement

● django-statsd, statsd and graphite– page timing– counters

http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/

Rollback options

● Level of automation– Depends on risk aversion and complexity of environment– 12+ years of deployment, average 5 deployments per week,

very very few actual rollbacks

● Engineer around it– lots of small changes → more likely to "roll-forward"– staged deployment → minimise harm

● Engineer for it– avoid backwards incompatible data migrations– tag deployments, keep previous versions handy

Rollback options

"My deployment rollback strategy is

like a car with no reverse gear.

You can still go backwards,

but you have to get out and push.

Knowing that makes you drive carefully."

- Me

Where to from here

Have an existing app with some gaps?

Think about what's slowing you down the most● Suggested priorities

– Automated tests (Django built-in)– Continuous integration (jenkins, django-jenkins)– Measure all the things (sentry, statsd + graphite, new relic)– Scripted database (South)– Scripted deployment (fabric)– Configuration management (puppet)– Virtualised development (Vagrant)– Test separation (test decorator, customise test runner)– Feature flags (gargoyle)

Looking Forward

● Better version pinning● Fully automated provisioning● Better test separation and parallelisation● Combined coverage results● Reduce dependence on external services● Make deployment more atomic● Staged rollouts to cluster – canary deployment● Make the "big red deploy button" big and red

Resources● Slides – http://slideshare.net/mindsocket/ ● Continuous delivery/deployment

– Continuous Delivery Primer - http://www.informit.com/articles/article.aspx?p=1641923– Anatomy of the Deployment Pipeline - http://www.informit.com/articles/printerfriendly.aspx?p=1621865 – Code as Craft – Etsy http://codeascraft.etsy.com/– Safe deploying on the cutting edge – Urban Airship http://lanyrd.com/2011/djangocon-us/shbqz/

● Vagrant – http://vagrantup.com● Vagrant with Fabric - https://gist.github.com/1099132 ● Jenkins Build Pipeline Plugin -

http://www.morethanseven.net/2011/03/20/A-continuous-deployment-example-setup.html● Git/fabric based rollback options

– http://dan.bravender.us/2012/5/11/git-based_fabric_deploys_are_awesome.html– http://www.askthepony.com/blog/2011/07/setup-a-complete-django-server-deploy-rollback-%E2%80%93-all-in-one-

powerful-script/– http://lethain.com/deploying-django-with-fabric/

Thank You!

Questions?

http://slideshare.net/mindsocket

http://slideshare.net/mindsocket/

http://www.informit.com/articles/article.aspx?p=1641923

http://www.informit.com/articles/printerfriendly.aspx?p=1621865

http://codeascraft.etsy.com/

http://lanyrd.com/2011/djangocon-us/shbqz/

http://vagrantup.com/

https://gist.github.com/1099132

http://www.morethanseven.net/2011/03/20/A-continuous-deployment-example-setup.html

http://lethain.com/deploying-django-with-fabric/