Running Docker in production (and staying alive to talk about that)

A Little bit about our Production

● High Traffic: 2.5 Billion requests per day

● Maintaining Lots of data

● Data integrity

● Fast growing user base

● Need to deploy new features fast

● Bottom line: We need tools that can help us grow fast!

What this lecture is about and what not

Yes:

✔ Sharing our experience with docker in production ✔ Production deployments and automation with dockers✔ Echo system tools that we integrate with docker✔ Tools we developed locally to ease containers usage✔ The reasons to use or not use docker for specific Application / Infrastructure✔ Where we failed and how to avoid that

No:

✗ Docker/containers basics✗ Testing environments✗ Tutorial✗ Kernel implementations behind the technology✗ Analogs between harbor and docker principles

External tools we use with Docker● Docker registry (internal server)

● Consul – Docker deployment configuration

● Mesos for docker clustering

● Marathon as Mesos API

● Chef for “parent host configuration”

● Custom aws “docker AMI”

Additional tools we use for the Deployment flow:

➔ Jenkins – builds / pull resources from Bitbucket

➔ Rabbitmq – message passing between central deployment and agents

➔ LDAP – Deployment Authentication

➔ Graphite – Containers metrics

Stuff we built internally

● Santa – Docker deployment tool

● Docker log tailer

● Docker metric collection

● Chef – docker protection

● Configuration bootstrap

● Marathon – Consul integration

Live Deployment Demo + Example usage of various

tools

Deployment Flow – From Bitbucket to Production with santa

● What we dockerize and what not, examples and reasons

● When we use Santa and when Mesos

● Best practices in the deployment flow (Auto image creation & store, version management, reverts, configuration set in Consul, Using Docker/Consul/Jenkins API, reviving from failures, feedback loop, etc)

● Viewing containers logs

● Monitoring the containers

Best Practices

Failures/Mistakes & Fixes

● Chef and dockers: Clash of the titans

● Forcing an elephant into a box

● Connection tracking table, mmm...

● Deployment failures: retries & revive

● Volumes hell

Provisioning & Deploy Flow with Mesos – Stage 1Provisioning & Deploy Flow with Mesos – Stage 1Provisioning & Deploy Flow with Mesos – Stage 1

I need x docker instances in AWS I can do that

Now we have Mesos cluster of X instances, each with Chef, Docker Daemon, Mesos slave and Consul agent which serves as discovery source

Provisioning & Deploy Flow with Mesos – Stage 2Please deploy an application to the cluster from the registry

That's fine by me

Provisioning & Deploy Flow with Mesos – Stage 2Please deploy an application to the cluster from the registry

That's fine by me Deploy, deploy, deploy

Also, I will need you to provide me the servers and ports that serve this application, I would like to update my buddy consul

Sure

Here's what Marathon knows, go gossip about it

I need x application jarI need x application jarI need x application jar

let me grab it from Bitbucket and build it for you

You know that's a legacy code, right? ;-)

Done, just take it from the target directory

Now, Docker Daemon, Please build, tag and push the image to the repository