1
Towards Continuous Deployment with Django
Roger Barnes@mindsocket
PyCon Australia 2012
BTech ICS
Co-founder @ Arribaa
Python/Django
Web development
Frisbee
Photography
Adventure
Beer
1997 1999 2001 2003 2005 2007 2009 2011
Concepts from other talks
● The why– Lean startups and customer discovery– Do more with less
● The how– EC2 / provisioning– Architecture– Developing for cloud deployment– Monitoring live web applications– IaaS vs PaaS
What is continuous delivery*
"Rapid, incremental, low-risk delivery of high quality, valuable new functionality to users through automation of the build, testing and deployment process" - Jez Humble
Deploying every good version of your software...
… or at least being able to
"Minimising MTTBID (Mean Time to Bad Idea Detection)" – Etsy
* delivery == deployment for the purposes of this talk
More Than Technology
● Technology - For automation● People - Cross-functional team, organised around
product● Processes – Conventions and some glue that
technology can't automate
Why
● Fail fast, win fast - Build, Measure, Learn● Competitive advantage● Avoid YAGNI - do less● Less manual process, less risk● Real-time control
– Deploy on demand– Decoupled deployment and release– Self service environments
Continuous Delivery in Practice
● Single path to production● Optimise for resilience● Get comfortable with being uncomfortable● Automate as much as possible● If it hurts, do it more often● Becomes natural, return on investment is fast/high● Lots of ways to do it, here's what I've done...
Environments
● Make dev/test/prod as similar as possible– Full stack, no "./manage.py runserver"– Some exceptions make sense. eg for development:
● dummy email backend● dummy message broker● fewer threads/workers● less memory● less analytics instrumentation
Develop Commit Build Stage Deploy Measure
Environments
Dev/test virtualised using Vagrant
Create and configure lightweight, reproducible, and portable development environments
$ vagrant up
$ vagrant ssh
Develop Commit Build Stage Deploy Measure
Environments
Vagrantfile configures base image, provisioning, shared folders, forwarded ports
Vagrant::Config.run do |config| config.vm.box = "precise64" config.vm.box_url = "http://files.vagrantup.com/precise64.box"
config.vm.provision :puppet do |puppet| puppet.manifests_path = "puppet/vagrant-manifests" puppet.manifest_file = "dev.pp" puppet.module_path = "puppet/modules" end
config.vm.forward_port 80, 8000 config.vm.forward_port 3306, 3306 config.vm.share_folder "arribaa", "/opt/django-projects/arribaa", ".."end
Develop Commit Build Stage Deploy Measure
Environments
● Repeatable, versioned configuration– Snowflake bad, phoenix good– Puppet (masterless) for provisioning OS and services– Fabric for scripted tasks, eg:
● copy database from production● update requirements● migrate database● commit and push to repository
● Anti-pattern– Can't spin up new environment with one command
Develop Commit Build Stage Deploy Measure
Development top level puppet config
include uwsgiinclude statsdinclude solrinclude memcachedinclude rabbitmqinclude nginxinclude testinginclude arribaainclude arribaa::db_nodeinclude arribaa::nginxinclude arribaa::celery
Develop Commit Build Stage Deploy Measure
Test configuration is the same
Production adds:backup
monitoring
Using Fabric with Vagrantdef vagrant(): # get vagrant ssh setup vagrant_config = _get_vagrant_config() env.key_filename = vagrant_config['IdentityFile'] env.hosts = ['%s:%s' % (vagrant_config['HostName'], vagrant_config['Port'])] env.user = vagrant_config['User']
def _get_vagrant_config(): with lcd('../vagrant'): result = local('vagrant ssh-config', capture=True) conf = {} for line in iter(result.splitlines()): parts = line.split() conf[parts[0]] = ' '.join(parts[1:])
return conf
Based on https://gist.github.com/1099132
Develop Commit Build Stage Deploy Measure
Dependencies
● virtualenv – separate env for each app● pip – install dependencies into virtualenv
– use requirements file– use explicit versions
● for repository based dependencies, use commit id● or fork on github
Develop Commit Build Stage Deploy Measure
Dependencies
● Using virtualenv and pip with fabricenv.site_dir = '/opt/django-projects/arribaa'env.app_dir = '/opt/django-projects/arribaa/arribaa'env.pip_file = 'requirements.txt'
def ve_run(command, func=run, base_dir=env.app_dir, *args, **kwargs): with cd(base_dir): with prefix("source /opt/virtualenvs/%s/bin/activate" % env.virtualenv): return func(command, *args, **kwargs)
def update_reqs(): ve_run('pip install -r %s' % env.pip_file, base_dir=env.site_dir)
Develop Commit Build Stage Deploy Measure
Database migration
● South– Schema changes– Data migrations
● Deploy separate expand and contract operations, eg:– Expand
● add new column (update model, manage.py schemamigration …)● map from old column (manage.py datamigration …)
– Deploy– Contract (optional)
● remove old column ( update model, manage.py schemamigration …)
Develop Commit Build Stage Deploy Measure
Database migration
● Using South with fabricdef syncdb(): ve_run("python manage.py syncdb --noinput --migrate")
Develop Commit Build Stage Deploy Measure
Tying it all together
● Update pip requirements on vagrant VM
$ fab vagrant update_reqs● Run data migration on vagrant VM
$ fab vagrant syncdb
Develop Commit Build Stage Deploy Measure
Source control
● Pick one, learn it● Use tags to keep track of build status● Avoid long-lived branches
– or integrate them – more overhead
● Tracks all code/documentation✔ application code✔ deployment code✔ provisioning code✔ configuration code✔ documentation✗ not build artifacts
Develop Commit Build Stage Deploy Measure
arribaa Application code├── apps Django apps│ └── ...├── static Django static files│ └── ...├── templates Django templates│ └── ...├── fabfile.py Fabric scripts├── requirements.txt Pip└── ...bin└── jenkins.sh Jenkins jobdocs└── ...
vagrant├── Vagrantfile Dev VM config└── puppet ├── modules Puppet modules │ ├── arribaa │ ├── backup │ ├── graphite │ ├── ... │ ├── uwsgi │ └── wget └── vagrant-manifests ├── dev.pp Puppet dev ├── prod.pp Puppet prod* └── test.pp Puppet test
* Not used by vagrant, but convenient to store here
Automated Testing
● Django’s test framework● Continuous Integration and Testing
– Jenkins
● django-jenkins– tests– pylint– coverage– css/jslint
● factory_boy instead of fixtures
Develop Commit Build Stage Deploy Measure
Test separation
● Split up tests by type● Can parallelise, keeping test run times low
– < 20 minutes from commit to production
● Unit tests first, faster feedback● Run tests on different triggers, keep some out of the main
build● Etsy - Divide and Concur
– http://codeascraft.etsy.com/2011/04/20/divide-and-concur/
Develop Commit Build Stage Deploy Measure
Test separation
● Unit● Functional/UI
– Selenium et al
● Staging and production smoke tests– Crawl URLs– Load/performance testing
● Flaky tests, Slow tests– exclude from regular build, run on a separate schedule
Develop Commit Build Stage Deploy Measure
Test separation
class TestBooking(TestCase):
@pipelineTag("unit") def test_when_lapsed(self): oldbooking = factories.BookingFactory.build(when=some_old_date) self.assertTrue(oldbooking.when_lapsed())
● How to do this with Django?
– Suites
– Custom test runner
– Other test systems (Nose has an attrib plugin)● My current solution
– Annotating tests, and adapted django-jenkins command
Develop Commit Build Stage Deploy Measure
Test separation# Used to ignore testsignore = lambda test_item: None
def pipelineTag(*args): """ Let a testcase run if any of the tags are in sys.argv and none of the tags match not-* in sys.argv. """ # Abstain if we're not in a build pipeline (ie using django-jenkins) # or if no tag(s) specified if 'jenkins_pipeline' not in sys.argv or not any(['tagged-' in arg for arg in sys.argv]): return _id
tags = args assert set(tags) < set(['unit', 'functional', 'flaky', 'slow', 'staging']) #Silently ignore if no matching tag if not any(['tagged-'+tag in sys.argv for tag in tags]): return ignore # Skip if "not-blah" tagged if any(['not-'+tag in sys.argv for tag in tags]): return skip('tagged: ' + ','.join([tag for tag in tags if 'not-'+tag in sys.argv])) # This test made it through return _id
Develop Commit Build Stage Deploy Measure
Build Pipeline
● One pipeline● All developers feed start of pipeline via VCS (master)
– Improve policy of "don't deploy broken code"...– ...with system where it's harder to deploy broken code
● Jenkins Build Pipeline Plugin– Series of dependant jobs, each runs different tests– Feeds into pre-deploy staging job– Successful staging job enables deploy job
Develop Commit Build Stage Deploy Measure
Build PipelineDevelop Commit Build Stage Deploy Measure
Currently simple linear flow
Build Pipeline
● Jenkins job – unit test stage:
bash -ex ./bin/jenkins.sh tagged-unit not-flaky not-slow● jenkins.sh
# Setup virtualenv, pip requirements, settings.py etc
...snip...
cd $WORKSPACE/arribaa
python manage.py clean_pyc
python manage.py jenkins_pipeline "$@"
Develop Commit Build Stage Deploy Measure
Staging/QA Job● fab staging-deploy – very similar process to production deploy● Test run of production deployment using Fabric
– Deploy code and config to staging environment (Vagrant VM)– Apply dependency updates
● puppet config● pip requirements
– Copy database from production and run data migration
● Should now have a deployed QA environment● Browser smoke tests – currently selenium
– Not using Django 1.4 LiveTestCase here, we have a full stack to play with
● Tag successful build as deployable
Develop Commit Build Stage Deploy Measure
Deployment Strategies
● Optimistic– Deploy, reload services – some appservers handle this well
● Blue/Green– Parallel environments, 1 idle– Idle environment gets new version
– Smoke tests and monitoring before switching over
● Canary– Take subset of app servers out of pool– Deploy to subset
– Smoke tests and monitoring before deploying rest of cluster
● Deploy whole new (app) server instance– then rewire load-balancer
Develop Commit Build Stage Deploy Measure
Deployment with Jenkins & Fabric
Using Git tags– Set in Jenkins after successful staging run– Used by deploy in Fabric
def pull(): with cd(env.site_dir): run('git fetch') latest_tag = run('git describe --tags --match "ci-passed-staging-*" origin/master', pty=False) run('git reset --hard refs/tags/%s' % latest_tag, pty=False)
Develop Commit Build Stage Deploy Measure
Tags set by build pipeline
Develop Commit Build Stage Deploy Measure
Deployment with Jenkins & Fabric
Jenkins script:
virtualenv -q /opt/virtualenvs/arribaa-jenkins
source /opt/virtualenvs/arribaa-jenkins/bin/activate
pip install -r requirements.txt
pip install -r requirements-testing.txt
cd $WORKSPACE/arribaa
fab deploy
Develop Commit Build Stage Deploy Measure
Deployment with Jenkins & Fabric
def deploy():
dbbackup() # django-dbbackup – to dropbox
changes = pull() # git fetch, git describe, git log, git reset
apply_puppet() # puppet apply … prod.pp
update_requirements() # pip install -r requirements.txt
clean_pyc() # django-extensions command
syncdb() # and –migrate (South)
collectstatic()
reload_uwsgi() # reload app server
restart_celeryd() # bounce task server
run('git tag -f ci-deployed && git push –tags') # tag deployed version
send_email(changes) # send notification
Develop Commit Build Stage Deploy Measure
Decoupling deployment and release
● Feature flags vs feature branches– Real time control with flags– Cleanup/maint needed for flags– Harder/slower to test different combinations in either case
● Good for testing ideas (on/off, A/B, power users, QA etc)● Gargoyle
– includes optional admin interface– selective criteria (%age of users, staff only, ...)
● Alternative: Django Waffle
Develop Commit Build Stage Deploy Measure
Measurement
● Eyes wide open– Track system and user behaviour
● Post-deployment– Error, performance and behaviour heuristics– Trend monitoring– Goal conversion rates
Develop Commit Build Stage Deploy Measure
MeasurementDevelop Commit Build Stage Deploy Measure
Develop Commit Build Stage Deploy Measure
Measurement - Munin
Develop Commit Build Stage Deploy Measure
Develop Commit Build Stage Deploy Measure
Measurement
● django-statsd, statsd and graphite– page timing– counters
http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/
Rollback options
● Level of automation– Depends on risk aversion and complexity of environment– 12+ years of deployment, average 5 deployments per week,
very very few actual rollbacks
● Engineer around it– lots of small changes → more likely to "roll-forward"– staged deployment → minimise harm
● Engineer for it– avoid backwards incompatible data migrations– tag deployments, keep previous versions handy
Rollback options
"My deployment rollback strategy is
like a car with no reverse gear.
You can still go backwards,
but you have to get out and push.
Knowing that makes you drive carefully."
- Me
Where to from here
Have an existing app with some gaps?
Think about what's slowing you down the most● Suggested priorities
– Automated tests (Django built-in)– Continuous integration (jenkins, django-jenkins)– Measure all the things (sentry, statsd + graphite, new relic)– Scripted database (South)– Scripted deployment (fabric)– Configuration management (puppet)– Virtualised development (Vagrant)– Test separation (test decorator, customise test runner)– Feature flags (gargoyle)
Looking Forward
● Better version pinning● Fully automated provisioning● Better test separation and parallelisation● Combined coverage results● Reduce dependence on external services● Make deployment more atomic● Staged rollouts to cluster – canary deployment● Make the "big red deploy button" big and red
Resources● Slides – http://slideshare.net/mindsocket/ ● Continuous delivery/deployment
– Continuous Delivery Primer - http://www.informit.com/articles/article.aspx?p=1641923– Anatomy of the Deployment Pipeline - http://www.informit.com/articles/printerfriendly.aspx?p=1621865 – Code as Craft – Etsy http://codeascraft.etsy.com/– Safe deploying on the cutting edge – Urban Airship http://lanyrd.com/2011/djangocon-us/shbqz/
● Vagrant – http://vagrantup.com● Vagrant with Fabric - https://gist.github.com/1099132 ● Jenkins Build Pipeline Plugin -
http://www.morethanseven.net/2011/03/20/A-continuous-deployment-example-setup.html● Git/fabric based rollback options
– http://dan.bravender.us/2012/5/11/git-based_fabric_deploys_are_awesome.html– http://www.askthepony.com/blog/2011/07/setup-a-complete-django-server-deploy-rollback-%E2%80%93-all-in-one-
powerful-script/– http://lethain.com/deploying-django-with-fabric/
Thank You!
Questions?
http://slideshare.net/mindsocket