Scale
Deploymentcontinuous
at
@premshree
1.6 million active sellers
24 million active buyers
$2.39 billion GMS (2015)
819 employees
Premshree PillaiSr. Engineer, Etsy
principles
q & a
tooling + culture
enable innovation(you are creating products, not code)
optimize for purpose
optimize for purpose
optimize for autonomy
optimize for mastery
experimentationA/B testing
quick iteration(improving products/features)
quick iteration(fail fast > stagnant code)
continuous improvement
continuous deployment
continuous delivery/
commit build tests
commit build tests user tests
commit build tests user tests release
commit build tests user tests release
commit build tests user tests release
frequent check-ins
frequent check-ins(to master!)
frequent check-insbranching in code
Featuregithub.com/etsy/feature
$server_config['my_feature'] = [ 'enabled' => 'on'];
$server_config['my_feature'] = [ 'enabled' => 'off'];
$server_config['my_feature'] = [ 'enabled' => 1];
$server_config['my_feature'] = [ 'enabled' => 1, 'bucketing' => 'user'];
if (Feature::isEnabled('my_feature') {!}
if (Feature::isEnabledFor('my_feature', $user) {!}
ExperimentationA/B testing
$server_config['my_feature'] = [ 'enabled' => 'on'];
$server_config['my_feature'] = [ 'enabled' => [ 'layout1' => 1, 'layout2' => 3, 'layout3' => 3 ]];
if (Feature::isEnabled('my_feature')) { switch (Feature::variant('my_feature')) { case 'layout1': // layout1 code break; case 'layout2': // layout2 code break; }}
continuous integration
continuous integration(and automated tests)
keep the build green
ready to releaseanytime
commit build tests user tests release
try
Try(before you commit)
TryLibgithub.com/etsy/TryLib
commit build tests user tests release
try deployinator
Deployinatorgithub.com/etsy/deployinator
commit build tests user tests release
try deployinator ci/jenkins
commit build tests user tests release
try deployinator ci/jenkins manual tests
commit build tests user tests release
try deployinator ci/jenkins manual tests deploy
commit build tests user tests release
commit build tests user tests princess prod
http://www.flickr.com/photos/zsqr0000/5269002895
… but do not fear
template changes css tweaks unreferenced code
$server_config['my_feature'] = [ 'enabled' => 1, 'bucketing' => 'user'];
$server_config['my_feature'] = [ 'enabled' => 50, 'bucketing' => 'user'];
premshree> .join
john
john + premshree
premshree>
john + premshree
premshree> .good
john + premshree
premshree> .bueno
john + premshree*
premshree>
john + premshree*
sally> .join
john + premshree* | sally
sally>
push train(with PushBot)
push traingithub.com/etsy/pushbot
post-deploy(gaining confidence)
supergrepgithub.com/etsy/supergrep
dashboards(deploy)
StatsD + Graphite
StatsD + Graphitegithub.com/etsy/statsd
pre-flight check
ready to push
ready to deploy
testing/verification
confidence
pre-flight check
ready to push
ready to deploy
testing/verification
confidence
try/git hooks
push train/irc
deployinator
user testing
supergrep/dashboards
https://www.flickr.com/photos/saschaaa/152502539/
assume best intentions
cultivate empathy
failure is an option
failure is an option(but not our intention)
failure is an option(we strongly want NOT to fail)
failure is an opportunity
blameless post-mortems
morguegithub.com/etsy/morgue
mixergithub.com/etsy/mixer
culture + tooling
deploy
fail ok
post-mortem
remediationstimulii
30+ config deploys