Embed Size (px)
DESCRIPTION
Continuous deployment has proven to be a successful and even addicting part of Etsy's engineering culture. See where it's applicable, some of the tools that make it easy, and the kind of architectural change that it makes possible.
Citation preview
Continuous Deploymentof Architectural Change
Matt GrahamCore Engineer @ EtsyContinuous Deployer
#iasanySeptember 27, 2011
A Brief History of Deployment
The Internet
Web Applications
Agility
Continuous
Database Deployment @ Etsy
Thursday
What it's all about
● Broadly Applicable● Tools Help● Enables the Unfeasible
MTTR vs MTTF
MTTR vs MTTF
● Cheaper for electrons
MTTR vs MTTF
● Cheaper for electrons
● Cheaper for protons & neutrons
MTTR vs MTTF
● Cheaper for electrons
● Cost prohibitive for humans
● Cheaper for protons & neutrons
MTTR vs MTTF
● Cheaper for electrons
● Cost prohibitive for humans
● Necessitates low MTTD
● Cheaper for protons & neutrons
The Case of Electrons
● Consider a project that has 6 bugs● continuous deployment testing misses 4& deploys fixes in 4 hours
● monthly release testing misses only 2& deploys fixes in 24 hours
● 16 hours of broken < 48 hours of broken
Good Excuses
● Infrequent Changes
Good Excuses
● Infrequent Changes● Infrequent Executions
Good Excuses
● Infrequent Changes● Infrequent Executions● Life and Death
Good Excuses
● Infrequent Changes● Infrequent Executions● Life and Death● Physical Investment
Deployment Tools @ Etsy
Culture Before Tools
● No planned deploys; only on demand
Culture Before Tools
● No planned deploys; only on demand
● A positive change is ready & tested
Culture Before Tools
● No planned deploys; only on demand
● A positive change is ready & tested
● Software is stable & supported
Jenkins
● Unit Tests
Jenkins
● Unit Tests● Functional Tests
Jenkins
● Unit Tests● Functional Tests● Manual Testing
tail -f | grep
IRC
Key Tools
Communication Tools
Graphs
Feature Flags
Feature Flags
Deploy != Product Launch
Dark Launch
def get_feature_a_link():
return ...
Dark Launch
def get_feature_a_link():
if enabled('NewFeatureA'):
return new_feature_a_link()
else:
return old_feature_a_link()
Dark Launch
application_config - NewFeatureA: admin - NewFeatureB: off - NewFeatureC: on
Ramp Up
application_config - NewFeatureA: 1% - NewFeatureB: off - NewFeatureC: on
Ramp Up
application_config - NewFeatureA: 5% - NewFeatureB: off - NewFeatureC: on
Ramp Up
application_config - NewFeatureA: 25% - NewFeatureB: off - NewFeatureC: on
Ramp Up
application_config - NewFeatureA: 100% - NewFeatureB: off - NewFeatureC: on
AB Testing
● Prove success of interface changes
● Prove interest in new features
Graphs!!!
Ganglia
Graphite
Event Overlay
StatsD
github.com/etsy/statsd
if ($success) { StatsD::timing('feature.runtime', $time);} else { StatsD::increment('feature.failure');}
Community Communication
Forums / Message Boards
etsystatus.com
twitter.com/etsystatus
twitter.com/etsystatus
Deployment is First Class
Deployment is aFirst Class Feature
Engineers are Users Too
Enable Architectural Change
Architecture is Relative
Organic Architecture
Premature Architecture
Premature Architecture
What now?
Passing Time => Change
● Scale● Product● Technology● Engineering Team
Passing Time => Change
● Scale● Product● Technology● Engineering Team● The Correct Architecture Changes
Examples
● Images From Twisted to PHP● From PostgreSQL to MySQL Shards
From Twisted to PHP
● Run Apache/PHP on a new port
From Twisted to PHP
● Run Apache/PHP on a new port● Implement one service in PHP
From Twisted to PHP
● Run Apache/PHP on a new port● Implement one service in PHP● Ramp up users on new service
From Twisted to PHP
● Run Apache/PHP on a new port● Implement one service in PHP● Ramp up users on new service● Repeat for remaining services
From Twisted to PHP
● Run Apache/PHP on a new port● Implement one service in PHP● Ramp up users on new service● Repeat for remaining services● Shut down Twisted version
PostgreSQL to MySQL Shards
● Migrate table by table
PostgreSQL to MySQL Shards
● Migrate table by table● Teed writes to MySQL
PostgreSQL to MySQL Shards
● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL
PostgreSQL to MySQL Shards
● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL● Verify data matches in both DBs
PostgreSQL to MySQL Shards
● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL● Verify data matches in both DBs● Ramp up users reading from MySQL
PostgreSQL to MySQL Shards
● Migrate table by table● Teed writes to MySQL● Backfill old data from PostgreSQL● Verify data matches in both DBs● Ramp up users reading from MySQL● Stop Postgres writes & drop tables
Continuous Deploy Pattern
● Change in small steps
Continuous Deploy Pattern
● Change in small steps● Dark launch placeholder via config
Continuous Deploy Pattern
● Change in small steps● Dark launch placeholder via config● Deploy to production while dark
Continuous Deploy Pattern
● Change in small steps● Dark launch placeholder via config● Deploy to production while dark● Maintain old & new in parallel
Continuous Deploy Pattern
● Change in small steps● Dark launch placeholder via config● Deploy to production while dark● Maintain old & new in parallel● Ramp up to new architecture
Continuous Deploy Pattern
● Change in small steps● Dark launch placeholder via config● Deploy to production while dark● Maintain old & new in parallel● Ramp up to new architecture● Remove old architecture
Review
● Broadly applicable● Key Tools: Feature Flags, Graphs, Communication
● Enables Architectural Change● It's really fun!!!
Continuous Deploymentof Architectural Change
Matt Grahamhttp://twitter.com/lapsu
http://lapsu.tv
Core Engineer @ EtsyContinuous Deployer
http://codeascraft.etsy.comWe're Hiring!
http://bit.ly/etsywantsawesome
