Prodution Architecture and

Deploymentwith Fabric

- Andy McCurdy -@andymccurdy

Whiskey Media

Whiskey Sites

Your First Django App

Basic Config (web)•Apache•mod_wsgi

•use daemon mode•threads more efficient•processes if you're unsure of thread safety

WSGIDaemonProcess my-site python-path=/home/code/ processes=2 threads=150 maximum-requests=5000WSGIProcessGroup my-siteWSGIScriptAlias / /home/code/my-site/deploy/wsgi/my-site.wsgi

Basic Config (media)•Nginx

•Use Nginx to proxy traffic to Apache•Meanwhile Nginx serves media

upstream my-site { server 127.0.0.1:8000;}server { listen 80; location ~ ^/media/ { root /home/code/my-site; expires 30d; } location / { proxy_pass http://my-site; proxy_set_header X-Real-IP $remote_addr; }}

Basic Config (db)•Databases•PostgreSQL

•Frank Wiles @ www.revsys.com

•MySQL•Percona @ www.mysqlperformanceblog.com

Basic Config (cache)•Use Memcached! Even 16mb will significantly help against a Digg or being Slashdot'ed•It's incredibly easy...

# settings.pyMIDDLEWARE_CLASSES = ( 'django.middleware.cache.UpdateCacheMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.cache.FetchFromCacheMiddleware')

CACHE_BACKEND = 'memcached://127.0.0.1:11211/'CACHE_MIDDLEWARE_SECONDS = 60*5 # 5 minutesCACHE_MIDDLEWARE_ANONYMOUS_ONLY = True

Basic Deployment•Copy Code & Media (rsync or scp)•Run DB migrations / syncdb•Bounce wsgi daemons

Managing Growth (db)•Move DB to its own server

•As much RAM as possible•Write-heavy (>10%)? Get fast disks•Tune your config file

Managing Growth (web)•Add more web servers

•Use a resource monitoring tool like Munin to understand if your app is CPU or memory bound

Even More Growth•Replicated or sharded Databases•Multiple load balancers for redundancy•Message queues•Crons•Search daemons (Solr, Sphinx)•etc...

(not so) Basic Deployment

image credit:Brad

Fitzpatrick

Deployment Reqs•Copy media to CDN•Maintenance splash page•Run DB migrations•Install/Upgrade Python dependencies•Add a new web server to the cluster•Execute arbritrary commands for sysadmin maintenance tasks

Deployment OptionsCapistrano

+ Out of box support for common use cases+ Hooks to customize tasks+ Source control integration+ Threaded deployment to multiple hosts

- Ruby :(

Deployment OptionsFabric

+ Very simple, tasks are just Python functions+ Easy to chain together tasks to create complex scripts out of bite size pieces

- No source control integration- No out of box support- Some bugs, although fairly easy to work around, and new maintainer is working on fixes

Fabric Basics•sudo easy_install fabric•need a fabfile.py•from fabric.api import *

•be mindful of tasks that may fail•each remote command starts fresh•changing directories

Core Functionality•local() - Run a command locally

•run() - Run a command remotely

•sudo() - Run a command remotely as another user

•put() - Copy a file from local to remote

•get() - Copy a file from remote to local

•many more helper-ish commands

Authentication•Relies on SSH model•Use SSH keys•Control access to root user via sudoers•When you have to revoke access, you just turn off their SSH account

Configuration•Fabric environment (env) -- it's just a dictionary•Hosts and Roles•Code Repositories•Whatever you need

•~/fabricrc•Global settings or all Fabric deployments•SSH username

Example Config# fabfile.pyfrom fabric.api import *

env.roledefs = { 'web' : ['10.1.1.1', '10.1.1.2'], 'db' : ['10.1.1.3'], 'lb' : ['10.1.1.4'], }

env.repositories = {...}

Tasks# fabfile.pydef uptime(): run('uptime')

$> fab uptime -H 10.1.1.3[10.1.1.3] run: uptime[10.1.1.3] out: 05:20:39 up 88 days, 12:00, 0 users, load average: 0.03, 0.03, 0.00

Mapping Roles to Tasks# fabfile.py

@roles('web')def uptime(): run('uptime')

$> fab uptime[10.1.1.1] run: uptime[10.1.1.1] out: 05:20:39 up 88 days... [10.1.1.2] run: uptime[10.1.1.2] out: 05:20:39 up 88 days...

Decorator Problems•Some problems with Fabric's role management•Can't override decorated tasks at command line as docs suggest

def default_roles(*role_list): def selectively_attach(func): if not env.roles and not env.hosts: return roles(*role_list)(func) else: if env.hosts: func = hosts(*env.hosts)(func) if env.roles: func = roles(*env.roles)(func) return func return selectively_attach

All better now#fabfile.py

@default_roles('web', 'db')def uptime(): run('uptime')

$> fab uptime# runs on all hosts in the 'web' and 'db' roles

$> fab uptime --roles lb# runs only on hosts in the 'lb' role

Dealing with Failures•By default Fabric dies if a task fails•Use a context manager when failures are anticipated

# fabfile.pyfrom __future__ import with_statement # py2.5

def symlink_me(): with settings(warn_only=True): run('rm /path/to/symlink') run('ln -s /home/andy /path/to/symlink')

Easy sys-admin•Make an "invoke" command•Great for sys-admin and one-off tasks

# fabfile.py

@default_roles('all')def invoke(command): "Invoke an arbritrary command" sudo(command)

# install new packages on all hosts in one command$> fab invoke:"apt-get install git-core"

Real World Tasks$> fab --list

Available commands:

bounce_wsgi_procs Bounce the WSGI procs by touching the filesdeploy Full deploymentdeploy_media Push media to S3invoke Invoke an arbritrary commandmigrate Run any migrations via Southreload_nginx Update Nginx's running configsplash_off Configure Nginx to serve the sitesplash_on Configure Nginx to serve a downed-site pageupdate_repositories Push code to serversupdate_dependencies Update dependencies to third party libs

Whiskey's Deploymentdef deploy(splash='no'):

"Full deployment" deploy_media() update_cached_repositories() update_dependencies() generate_releases() if splash == 'yes': splash_on() _symlink_code() migrate() bounce_wsgi_procs() if splash == 'yes': splash_off()

$> fab deploy:splash=yes

Questions?