Embed Size (px)
DESCRIPTION
Citation preview
Gilt in a nutshell ...
• 5, 500, 1000/200, 300, 1, 18, ...
Gilt in a nutshell ...
• 2 DCs + Amazon EC2
• CentOS Linux • CDN (Akamai) • Puppet, Cobbler, ... • Zenoss, Nagios, ... • Git & Gerrit • Jenkins
Frontend (HTML[5], CSS, JS, ...)
Load Balancing (Zeus)
Aggregation (Java, Scala, ...)
Load Balancing (Zeus)
Services (Java, Scala, Ruby, ...)
Database (PostgreSQL, MongoDB, ...)
Puppet at Gilt ...
• We've been using Puppet for almost 3 years • ~ 1000 puppet modules • Typical setup
o Apache httpd+passenger for Puppetmaster o Puppetcommander (MCollective plugin) to schedule
puppet runs o Puppet Dashboard (Reporting) o In-house External Node Classifier (aka 'Mothership')
Puppet - Incremental rollout
Puppet - Incremental rollout
• Deployment of changes o Commit to 'development' branch o Push changes for code review o Once changes have been reviewed and verified,
they get merged into 'development' o A 'canary' flag can be set for a number of nodes o Change is cherry-picked to 'master' branch o A 'tag' from master is created and deployed
See behavior in the canary environment Expand the canary env. Rinse & repeat Remove canary flag
http://mothership.sourceforge.net/
Puppet - External node Classifier
Puppet - External node Classifier
• Mothership o In-house assets management & provisioning tool. It
also acts as an External Node Classifier for Puppet Can provision bare-metal & virtual machines Integrated with Cobbler Also manages users/groups & sudoers
• Synchronized to LDAP • Puppet deploys sudoers file
DNS management Hosts can have one or more labels (tags) that
are mapped to Puppet modules
Lessons learned / Best practices ...
Puppet ... • Don't just run it! • Node mgmt can become difficult (+100) • Keep change in mind (e.g. OS upgrades) Mothership ... • ... is not enough. You need at least 2 views
o Logical (deployed/reserved), Physical
• You need to live with Physical != Logical o Physical will be generated. It is a feedback loop.
Next steps ...
Requirements ...
• Accelerate our ability to do incremental deployments (multiple times a day) o ... with easy rollback o ... accelerating our ability to innovate o ... while maintaining 100% uptime
• The "Happy Path" needs to be without human intervention o From Commit to A/B test
Architecture ...
Ion Cannon
Code - Gerrit
Build - Jenkins/Nexus
QA - Selenium / TestComplete
Performance
Deployment - Puppet
Analytics
Takeaways ...
• Keep your modules small, keep them simple • Keep change in mind
o Do incremental rollout of changes o Provide a 'default' case in your puppet modules and
use 'fail' (sdtlib) as a default case.
• If you are looking for a complete CD platform, then you need to embed Puppet into a larger solution
Questions ...
