Retrofitting Continuous

DeliveryA DevOps Tale

Hi. I’m Alan.

betterment :

investing::

devops : engineering

Phase 1: Snowflakes

Betterment Core Applications

“brochure”

wordpressapache

phpmysql

batch(trading)

cronjava

springhibernate

mysql

webapp

apachetomcat

javaspring

hibernatemysql

Rackspace Loadbalancer

pre-prodapachetomcatmysqlhudsonjiraconfluence“warehouse”sendmailfile lockerbackup

Look at the Mess You’ve Made

● Overburdened, inflexible infrastructure from Rackspace● Only one pre-prod environment

○ And it runs five other things including Jira/Confluence/etc.● Building and deploying artifacts manually, ad-hoc, as needed

○ “Why is your trunk .war 23 bytes larger than mine?”● No Automated Functional Testing ● Deployments require downtime.● Deployments happen at 3am ET● Manual Provisioning on persistent servers

○ “This wordpress plugin needs php5.3 but we have 5.1” ● Branching (and merging) with Subversion requires significant

effort● …

yuck. so what’s the good

news?

we shipped a ton of functionality.

delivery speed faster than

innovation speed

Phase 2: Addicted to Automation

stop building features.fix what’s painful.

DevOps

fix onboarding.

boxen.

github // @jbarnette + @wfarr

mac osx+ homebrew

+ puppet+ facter

github.com/boxen/puppet-*

fix tooling.

>

$ time ant clean test deployment19m 22s

$ time gradle clean test intTest war zipJar2m 36s

(~builds/day · Δt · devs · man_days/mo)/min =

(6 · 17 · 19 · 20) / 60 = 646 hrs/mo

fix notifications & logging.

Whoa.

fix parallel workflows.

everybody loves jenkins

build open pull requests* build develop/stage/mastercobertura reports coverage

* https://wiki.jenkins-ci.org/display/JENKINS/GitHub+pull+request+builder+plugin

fix crowded infrastructure.

AWS + Rackspace(sitting in a tree, N-E-T-W-O-R-K-I-N-G)

<10ms !

“canary” ec2 webapps exposed via route53

weighting

fix provisioning.

Ansible (not manual) AMIs

● Playbooks are YML that execute over SSH● Playbooks define necessary machine config

o Results in AMI creation in AWSo Returns AMI_ID

● Commit playbooks and current AMI_ID● Jenkins reads AMI_ID from build artifact

o Spins-up necessary EC2 instanceso Runs deploy playbook, unrelated to provisioning playbook

● If you need to modify a playbooko Run playbook to create new AMI_IDo Commit new playbook and AMI_ID to repo.

no more snowflakes.

everybody loves jenkins again

deploys to rackspacedeploys to ec2

bash or ansible playbooks

fix database migrations.

Flyway Database Migrations

● Looks a lot like rake db:migrateo core/src/main/resources/db/migrationso core/src/main/resources/db/cleanups

● Kept track of in a schema_versions table.

● Cleanups are promoted to migrations when develop moves to stage.

● Run as part of weekly releases.● http://flywaydb.org/

Database Subsetting with Jailer

● Walks the constraint graph from the mysql schema.

● Requires some manual intervention to cut off circular or “eager” relations.

● Exports a referentially valid subset of data.o User, Account(s), Trade(s), Etc.

● http://jailer.sourceforge.net/

Users. Duh.

+ = freshdb

subsets live in s3. download and run flyway.instant db on any branch.

eos:better-core(develop)$ ./fresh -fForcing download...Downloading fresh data... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 21.0M 100 21.0M 0 0 1004k 0 0:00:21 0:00:21 --:--:-- 1365k:setupFreshDb:setupFreshDb:complete

__ _ / _| |_ ___ ____ _ _ _| |_| | | | \ \ /\ / / _` | | | || _| | |_| |\ V V / (_| | |_| ||_| |_|\__, | \_/\_/ \__,_|\__, | |___/ by fido |___/

[stderr] INFO AbstractCLI.doMain(287) | cli_started[stderr] INFO FlywayDbMigrationCLI.executeBeforeSpring(164) | jdbc:mysql://127.0.0.1:3306/bettermentdbfresh [stderr] INFO FlywayWrapper.migrateTo(154) | Running ALL pending migrations ...[stderr] INFO ApacheCommonsLog.info(43) | Validated 146 migrations (execution time 00:00.616s)[stderr] INFO ApacheCommonsLog.info(43) | Current version of schema `bettermentdbfresh`: 20140617123653[stderr] INFO ApacheCommonsLog.info(43) | Migrating schema `bettermentdbfresh` to version 20140620132438[stderr] INFO ApacheCommonsLog.info(43) | Migrating schema `bettermentdbfresh` to version 20140620132439[stderr] INFO ApacheCommonsLog.info(43) | Migrating schema `bettermentdbfresh` to version 20140620132440[stderr] INFO ApacheCommonsLog.info(43) | Migrating schema `bettermentdbfresh` to version 20140620132441[stderr] INFO ApacheCommonsLog.info(43) | Migrating schema `bettermentdbfresh` to version 20140625091709...

ah. that’s better.now what?

Phase 3: Innovation That Scales

innovating is easy.demoing is easy.

product teams move faster than

infrastructure teams.

jruby on railsfactory girl for Javamultiple schemas

webservices

delivery speed slower than

innovation speed

DevOps

DevOps

Solution?Undercover DevOps

every team builds product features with

delivery in mind

Questions?

first one: can you start at phase 3?

alan@betterment.comgithub/twitter: @nonrational