Docker Tallinn @Pipedrive

Renno Reinurm2016

How application performance requirements impacted the (r)evolution of the Docker infrastructure at Pipedrive.

Visibility

Management

Interfacehttp://www.slideshare.net/rreinurm/living-with-microservices-at-pipedrive

Requirement 1 You shall not build/test/deploy Docker container over 5 minutes

Based on: xkcd.com

Optimized DockerfilesFirst iteration, 14 layers:FROM nodeENV SERVICE_NAME=company-statisticsENV SERVICE_DESC="Company statistics"ENV SERVICE_TAGS=statisticsENV NODE_ENV=liveENV SERVICE_CHECK_HTTP=/healthENV SERVICE_CHECK_INTERVAL=10sENV SERVICE_CHECK_TIMEOUT=5sENV DOCKER=1EXPOSE 8000WORKDIR /srcUSER nobodyCOPY . /src/CMD ["node", "."]

Optimized, 8 layers and ordered:FROM node:6-slimENV SERVICE_NAME=company-statistics \ SERVICE_DESC="Company statistics" \ SERVICE_TAGS=statistics \ NODE_ENV=live \ SERVICE_CHECK_HTTP=/v1/companyStatistics/health \ SERVICE_CHECK_INTERVAL=10s \ SERVICE_CHECK_TIMEOUT=5s \ DOCKER=1EXPOSE 8000WORKDIR /srcUSER nobodyCMD ["node", "."]COPY libraries/ /src/COPY src/ /src/

https://youtu.be/X_q2l8hotAc?t=365

Deployment process optimizations

NB! https://docs.docker.com/engine/userguide/storagedriver/selectadriver/

Replacement of Devicemapper to AUFS reduced deployment process time 10x.

There are still improvements possible:

● Handle Linux signals● Parallel rolling updates

Requirement 2

Consumers shall connect only to healthy services

Beware the service discovery corruption

● Always enable health checks

● Use unique health checks

SERVICE_CHECK_HTTP=/health

vs

SERVICE_CHECK_HTTP=/v1/companyStatistics/health

Requirement 3

Container shall handle 10 000 connections and constant high load.

https://youtu.be/PivpCKEiQOQ

Issues

● Linux kernel 3.13● Fluentd logging agent● Graylog logging driver● Kernel sysctl parameters● Swap usage● PEBKAC

○ "net.ipv4.ip_forward" => 0

● WARNING: No memory limit support● WARNING: No swap limit support● WARNING: No kernel memory limit support● WARNING: No oom kill disable support● WARNING: No cpu cfs quota support● WARNING: No cpu cfs period support

Docker hosts tuning

● Linux kernel 4.x● Removed Swap● Dropped fluentd driver● Docker Engine upgrade/patches● Kernel performance tuning

http://www.devconf.cz/files/slides2015/friday/Performance%20Tuning%20of%20Docker%20and%20RHEL%20Atomic.pdf

Service risk mitigation

● Number of nodes in cluster○ If in doubt increase the number

● Spreading policies● Multiple instances● Memory limitations● Healing policies

○ Autorestart○ Reschedule

Security mitigation

● TLS authentication● Minimal & official Docker images● Signed Docker images● User Namespace Remapping● PID limits● Capabilities● Seccomp profiles● AppArmor/SELinux profiles

https://github.com/riyazdf/dockercon-workshop

Recommendations for goingLive with Docker● You still need to take care of OS ● Read Github issues● Read from the source● Keep it up to date● (Performance) Test it

Thank you!

Give me your feedback @rreinurm