Upload
docker-inc
View
6.885
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Since July 2014 Shopify's been serving thousands of requests per second of production web traffic from Docker containers. This was an 8 month effort, with multiple pivots of direction from the team—and we're only getting started. This talk covers the lessons learned through the trial and error of an in-flight architecture redesign, spanning hundreds of hosts, as well as the technical vision of the future of our platform.
Citation preview
Docker at ShopifyFrom This-Looks-Fun to Production
Shopiscale• 200-300K RPM during day, > 500K during large sales
• ~100 large metal servers running containers
• 100+ product developers, ~10 infrastructure, ~10 core ops
• 10 year old monolith
• Ruby on Rails, MySQL, Redis, memcached, ..
Make it easier to solve problems at
Shopify
Container-------Platform
Timeline to production• Mid 2013. Investigated Mesos and Docker
• Winter 2014 Legwork for containerization, failed attempt to deploy Mesos
• Spring 2014 Containerization work and no success with CoreOS either
• Summer 2014 Pivot to Chef, runit and Capistrano. !'ed it
Ship Docker as the only change
Successfully shipping Docker1. Play
2. Prepare
3. Containerize
4. Feature
1. Play stage• Develop feel for Docker
• Study namespaces, companies, cgroups, ..
• Prototyping
• Don't get stuck or starstruck
2. Preparation stage• Secrets (github.com/Shopify/ejson)
• Assets
• Logs
• Graceful shutdown
• Monitoring
• Roles/Procfile
• Containerization patterns
3. Containerization stage• Don't add new features to your infrastructure
• Building images/CI/dev
• Deployment
• Static orchestration
• Union filesystems
• init in container, Linux namespaces, cgroups, routing, monitoring, security updates, registry..
4. Feature stage !• Buttons
• Distributed orchestration
• Service discovery
• Routing
• CoreOS
• PaaS
• ..
Verschlimmbesserung
What we took to production
Image Builds
Image builder1. Commit <sha> lands in master
2. Build server builds image for new SHA (~2m)
3. Pushed to internal registry: registry.shopify.com/shopify:<sha>
We don't use Dockerfiles
Builder API• script/prepare System-level packages or other infrequent
changes on top of base image.
• script/build Frequently changed dependencies, run on every build.
• config/roles.json Roles for the application, our Procfile.
Final image for ShopifyUbuntu----------------Baseimage (750 MiB)================`script/prepare` (100MiB)----------------`script/build`: Start of week (800MiB)----------------`script/build`: Start of day (100MiB)-----------------`script/build`: Current commit (80MiB)
container "shopify" do role :web instances 5 health_check true http_health_check "/ping/ponies" revision "file" env({ RAILS_ENV: "production", UNICORN_WORKERS: node[:cores] / 5 })end
Turns into supervisor scripts:docker run -e CONTAINER=1 -e HOST_IP=172.16.1.227 \ -e ENV=production -e PORT=20003 \ -e APP_HEALTH_CHECK=/ping/ponies \ -e ENDPOINT=172.16.1.227:20003 -e UNICORN_WORKERS=5 \ -e NODENAME=borg-shopify-unicorn-4.omgweb.dc.shopify.com \ --volume=/tmp/borg-run/keys/11607:/key \ --cidfile=/tmp/borg-run/keys/11607/cidfile \ --name="borg-shopify-unicorn-4-$$" \ --net=host internal.registry.com/shopify:$(cat /etc/borg/shopify/current) web
Deploys1. Deploy two stacks at once.
2. Capistrano updates revision in /etc/shopify/revision.
3. ls -d /etc/sv/shopify-web* | xargs -L1 sudo sv restart
• Restart ~20% of containers at a time.
Containers are the future, but it's still hard in
production
Thank You.
Staircase Tutorial, Andreas Leversa (https://creativecommons.org/licenses/by/2.0/)Sandbox, Kate Mereand-Sinha (https://creativecommons.org/licenses/by/2.0/)Panama canal construction photo, World Bank Photo Collection (https://creativecommons.org/licenses/by-nc-nd/2.0/)Other pictures used by permissions of Florian WeingartenFlight of the Unicorns, Eric Wagner (https://creativecommons.org/licenses/by-nc-sa/2.0/)Container, Izabela Reimers (https://creativecommons.org/licenses/by-nc-sa/2.0/)Sam Satch, Chinese Ornamental Chilli Sprouting (https://creativecommons.org/licenses/by-nd/2.0/)Anvil & Hammer, Brad Knabel (https://creativecommons.org/licenses/by-nc-nd/2.0/)Matt Kowal, DIY Darkroom - Alternative Process (https://creativecommons.org/licenses/by-sa/2.0/)Over-stuffed Walrus picture kindly provided with permission by the Horniman Museum and Gardens, U.K.Video of me in walrus costume, Lily Rogers and Emma Craig