Sam Haskins - velocity.oreilly.com.cnvelocity.oreilly.com.cn/2012/ppts/shaskinsvelocitychina2012.pdf · Continuous Deployment Sam Haskins Velocity China 2012 Etsy.com [email protected]

Continuous Deployment

Sam Haskins

Velocity China 2012

[email protected]

Friday, December 21, 12

mailto:[email protected]

mailto:[email protected]


20,000,000 Members


800,000 Active Shops


17,000,000 items


0

150

300

450

600

2005 2006 2007 2008 2009 2010 2011

525.6

306.7

176.8

87.3

263.80.17

Millions of $


0

150

300

450

600

2005 2006 2007 2008 2009 2010 2011

525.6

306.7

176.8

87.3

263.80.17

Millions of $


(deploys/day)


0

150

300

450

600

2005 2006 2007 2008 2009 2010 2011

525.6

306.7

176.8

87.3

263.80.17

Millions of $


200 deployers


200 deployersEngineers?


200 deployersEngineers +Designers


200 deployersEngineers +Designers +Product


33 deploys/day


33 deploys/day(average that includes weekends)


33 deploys/day(average that includes weekends)

(just the web stack)


Continuous Deployment


Take the ceremonyout of releases


Smaller changes,more often


Fewer changes/deploy=

Fewer things thatcan go wrong


Fewer changes/deploy=

Easier to see effects of your change



How to implement


code

test

release

feedback


code

test

release

feedback


<?php// search engine

$results = Search::use_grep($_GET[‘query’]);print_r($results);


code

test

release

feedback


$ curl http://shaskins.dev.etsy.com/search.php?query=bowties


code

test

release

feedback


$ scp -r ~/code webserver.prod.etsy.com:/docroot


code

test

release

feedback


$ tail -f access_log[15/Nov/2012:17:14:41 +0000] "GET /search?query=bowties" 200 [15/Nov/2012:17:14:42 +0000] "GET /search?query=knit+hat" 200[15/Nov/2012:17:14:45 +0000] "GET /search?query=vintage+chair" 200



code

test

release

feedback


“Always Ship Trunk”Paul Hammondhttp://www.paulhammond.org/2010/06/trunk/


http://www.paulhammond.org/2010/06/trunk/

http://www.paulhammond.org/2010/06/trunk/

Centralized configuration file


<?php// Site configuration

$config[‘search’] = array( ‘enabled’ => ‘on’ ));





<?php// search engineif (!Feature::isEnabled(‘search’)) { header(‘HTTP/1.1 503 Service Unavailable’);}




$config[‘search_method’] = array( ‘grep’ => ‘100%’

));



$config[‘search_method’] = array( ‘grep’ => ‘100%’, ‘solr_service’ => ‘0%’ ));


<?php// search engineif (!Feature::isEnabled(‘search’)) { header(‘HTTP/1.1 503 Service Unavailable’); exit();}

$results = Search::use_grep($_GET[‘query’]);

print_r($results);Friday, December 21, 12

<?php// search engineif (!Feature::isEnabled(‘search’)) { header(‘HTTP/1.1 503 Service Unavailable’); exit();}

switch (Feature::variant(‘search_method’)) { case ‘grep’: $results = Search::use_grep($_GET[‘query’]); break;

case ‘solr_service’: $results = Search::use_solr($_GET[‘query’]); break;}

print_r($results);Friday, December 21, 12

Lots of types of options



$config[‘new_feature’] = array( ‘enabled’ => ‘0%’, ‘admin’ => ‘on’ ));



$config[‘new_feature’] = array( ‘enabled’ => ‘0%’, ‘users’ => array( ‘test_user’, ‘big_seller’, ‘best_friend’ ) ));



$config[‘new_feature’] = array( ‘enabled’ => ‘0%’, ‘groups’ => array( 1234, 6742 ) ));



$config[‘new_feature’] = array( ‘enabled’ => array( ‘variant_1’ => ’15%’, ‘variant_2’ => ’20%’, ‘control‘ => ’65%’ ) ));


code

test

release

feedback



Development Staging Production


Development


DevelopmentSeparate database


DevelopmentSeparate databaseSeparate network


Staging and Production


Staging and ProductionSame databases


Staging and ProductionSame databasesSame caches


Staging and ProductionSame databasesSame cachesSame services


Staging and ProductionSame databasesSame cachesSame servicesSame hardware


code

test

release

feedback


_________ ______ _____ ___________ /_____ ________ ___ /______ _____ _____(_)_______ ______ ___ /_______ _________ __ / _ _ \___ __ \__ / _ __ \__ / / /__ / __ __ \_ __ `/_ __/_ __ \__ ___// /_/ / / __/__ /_/ /_ / / /_/ /_ /_/ / _ / _ / / // /_/ / / /_ / /_/ /_ /\__,_/ \___/ _ .___/ /_/ \____/ _\__, / /_/ /_/ /_/ \__,_/ \__/ \____/ /_/ /_/ /____/



Configuration-onlydeploys


code

test

release

feedback


Metrics


MetricsHundreds of thousands of metrics


Business Metrics


Business MetricsNumber of sales


Business MetricsNumber of salesDollars of revenue


Business MetricsNumber of salesDollars of revenueSources of traffic


Application Metrics


Application MetricsApplication actions


Application MetricsApplication actionsUsage patterns


Application MetricsApplication actionsUsage patternsTimes events happen


System Metrics


System MetricsCPU usage


System MetricsCPU usageNetwork bandwidth


System MetricsCPU usageNetwork bandwidthDisk IO


Make metricseasy to create






$results = Search::use_grep($_GET[‘query’]);print_r($results);StatsD::increment(‘search.queries’);



Put them into dashboards


Logs



Make metrics from logs


Big-data analytics


Culture of trust


Tools solve technological friction


Culture solves organizational friction


Get rid of barriers


Get rid of barriers“Talk to ops to get access to the logs”


Get rid of barriers“Talk to ops to get access to the logs”“Get approval from CTO to push code”


Get rid of barriers“Talk to ops to get access to the logs”“Get approval from CTO to push code”“Release manager pushes the buttons”


Take responsibility


Something’s broken?


Something’s broken?Ops problem?


Something’s broken?Ops problem?Developer problem?


Something’s broken?Ops problem?Developer problem?Product problem?


Something’s broken?Ops problem?Developer problem?Product problem?


Something’s broken?Ops problem?Developer problem?Product problem?Problem.


Everyone is responsible


Dev/Ops/ProductCollaboration


Blameless Postmortems


Why Etsy Loves Continuous Deployment


Features getdeveloped faster


Iterative Development


Increased confidence


Data-backed decisions


Better communication


Mean time to recovery


Tools Mentioned:•Feature flag tool (not yet released, but soon!)•Jenkins (http://jenkins-ci.org)•Deployinator (https://github.com/etsy/deployinator)•StatsD (https://github.com/etsy/statsd)•Graphite (http://graphite.wikidot.com)•Ganglia (http://ganglia.sourceforge.net)•Logster (https://github.com/etsy/logster)•Supergrep (https://github.com/etsy/supergrep)•Much more at https://github.com/etsy


http://jenkins-ci.org

http://jenkins-ci.org

https://github.com/etsy/deployinator




https://github.com/etsy/statsd

https://github.com/etsy/statsd

http://graphite.wikidot.com

http://graphite.wikidot.com

http://ganglia.sourceforge.net

http://ganglia.sourceforge.net

https://github.com/etsy/logster

https://github.com/etsy/logster

https://github.com/etsy/supergrep

https://github.com/etsy/supergrep

https://github.com/etsy

https://github.com/etsy

Thanks!


Documents

Sam Haskins - velocity.oreilly.com.cnvelocity.oreilly.com.cn/2012/ppts/shaskinsvelocitychina2012.pdf · Continuous Deployment Sam Haskins Velocity China 2012 Etsy.com [email protected]