REAL WORLD DOCKER:

TEN THINGS WE’VE LEARNED

Speakers

• Thorsten von Eicken

• CTO

• Raphael Simon

• Senior Systems Architect

• Benefits of Docker

• Phases of Docker adoption

• Ten Things We’ve Learned

Agenda

2

Who is RightScale?

Self-Service Cloud Analytics

RightScale Cloud Portfolio Management

Cloud Management

Design

Virtualized

Environments

Public

Clouds

Other

Services

Private

Clouds

Automate

Multi-Cloud Orchestration

Operate Deploy Report Optimize

POLLS

4

The Route to Docker Exciting …

… and scary

Benefits of Docker at RightScale 1. Ability to run integration systems “all-in-one”

2. More robust deployments: same code,

language runtime and library dependencies in

dev, staging and production

3. Separation of concerns: devs produce Docker

images, ops runs them

4. Easier deployment thanks to isolation: no

need to worry about cross-dependencies

5. Smaller set of base VM images to maintain

6. Optimization of resource utilization

7. ...

It’s a spectrum ranging from quick wins to requiring

fundamental changes

● Dev - “more per laptops”

● QA - more consistent builds

● Prod - more decoupling from

dev

Adopting Docker in Phases

● Define steps with clear goals

● Start with quick wins

● Learn as we go

● Progress towards the “holy grail”:

o Generic cluster of VMs used to dynamically deploy

web services

● 4 identified phases - currently deploying phase 2, starting

dev on phase 3

Phase 1: Dev / Test

● Setup CI to build docker images

● Use containers to:

o Run unit tests

o Run “distributed” integration

tests

● Benefits:

o Guaranteed clean environment with every run

o Ability to run full integration environments on one box

o Only one Jenkins slave image / configuration

Phase 2: App Deploy

● Push docker images built in CI to production

● Minimal changes to VM images (need recent kernel +

docker daemon)

● Benefits:

o Code shipped runs exactly

as tested

o Streamlined deployments

Phase 3: “Generic” Virtual Machines

● Run “everything” in containers

e.g. use fig to start services

● Dynamic app configuration

● Benefits:

o One base image

(Docker + utilities)

o On-ramp to Phase 4

Phase 4: Automated “Pod” Deploys

● Automated container cluster management

● Scheduler to deploy pods of containers on the “right” host

o Swarm, Kubernetes, CoreOS Fleet, Mesos ... ?

● Benefits:

o Fast auto-scaling / auto-healing

o Resource utilization optimization

o Completely streamlined continuous delivery

10 Things We’ve Learned

How to cruise …

… and what to avoid

Docker captures stdout/stderr

docker logs command prints combined stdout/err

docker logs -tail 0 -f : running tail

1. Logging – how Docker does it

$ docker run --name hello -d busybox echo hello Santa Barbara a3c0caa675e106cc0cf208dade762afcc341ed5b9ea8f3d75b6e2092745a5faa $ docker logs hello hello Santa Barbara $

1: Logging – how not to do it

● Log to stdout/stderr and collect them in the VM o Not all apps log to stdout/err, many don’t add timestamps

o No log rotation (can use logrotate with copytruncate)

o Limited tailing to ship the logs

● Run syslog daemon inside the container o Containers ≠ VMs

o Configuration hell

1: Logging – solutions ● Bind-mount /tmp/dev -> /dev

o Can’t bind-mount /dev/log!

o Move /dev/log to /tmp/dev/log

o See http://jpetazzo.github.io/2014/08/24/syslog-docker/

● Fix docker daemon to forward to syslog o Fixing stdout/err is happening (#7195)

o Ready to add support for syslog source, but not active

container docker

syslog

file stdout/err

json

2: Monitoring – how not to do it

● Monitoring daemon inside

each container

o Container ≠ VM

o Monitoring daemons require privs

o Configuration/management hell

Monitoring – how to do it

● Collect stats in VM using container-aware monitoring o Stats are in /sys/fs/cgroup/…

See: Docker doc article on run-time metrics

o Docker support: cAdvisor, DataDog, … ?

● Just monitor at the process level

$ docker run --name hello -d busybox sleep 60 3a804b088b432035c5cee541f4baef3cc728d27dded3378fd253c6b4abeb077a $ cat /sys/fs/cgroup/cpuacct/docker/3a804b088b432035c5cee541f4ba ef3cc728d27dded3378fd253c6b4abeb077a/cpuacct.usage_percpu 630924 4774818 7494614 3622216

3. Secrets – How not do it

Add then remove creds using Dockerfile statements

FROM rightscale/ruby-215 ADD github.key /root/.ssh/id_rsa # OOPS RUN git clone git@github.com:rs/app ... RUN rm /root/.ssh/id_rsa # DOES NOT HELP

3. Secrets – Take away

● Each Dockerfile ADD and RUN command results in a

new committed layer

● All image layers (built or pulled) are readily

accessible

● For now: Make sure to remove any unnecessary

credential from the context prior to building (see #8)

● In the future: Take advantage of “nested builds”,

see Docker github issue #7115

4. Container access ● Launch image manually with

custom command to troubleshoot

● Inspect files inside running container

● Launch shell into running container

using docker exec (new in 1.3)

$ docker exec -it hopeful_shockley /bin/sh # ps -ax PID TTY STAT TIME COMMAND 1 ? Ss+ 0:00 /usr/bin/ruby ← Main container process 43 ? S 0:00 /bin/sh ← Current shell 49 ? R+ 0:00 ps -ax

5. Aufs vs. btrfs ● aufs corruption of container filesystems, issue #7229

o Ubuntu use kernel 3.13.0-37 or newer

o RHEL 6.x/CentOS 6/... use kernel 2.6.32-504 or newer

● btrfs seems to work better (default in CoreOS)

● btrfs “requires” separate partition

$ mkfs.btrfs /dev/xvdb $ mount /dev/xvdb /mnt $ mkdir -p /mnt/docker $ ln -sf /mnt/docker /var/lib/docker $ sed -i -e '/DOCKER_OPTS/s/.*/DOCKER_OPTS="-s=btrfs"/' /etc/default/docker $ restart docker

6. Got Infinite disk space?

● Container logs grow indefinitely o Use logrotate with copytruncate

● Containers accumulate indefinitely o Becomes an issue if containers are frequently

restarted due to upgrades or crashes

o Use docker run --rm

but then how do you troubleshoot?

o Write script to docker rm old unused containers?

7. Huge Containers – how not to do it

Overlays don’t go away

FROM ubuntu:14.04 RUN apt-get update RUN apt-get install -y libjpeg RUN apt-get install -y libjpeg-dev build-essential gcc 109 MB ADD source /build 5 MB? WORKDIR /build - RUN ./configure 0 MB RUN make 100 MB? RUN make install CMD /usr/local/bin/myexe

Use a tools container, share build results via volume

In the future: “nested builds” #7115, “squash” #4232 ?

FROM ubuntu:14.04 VOLUME /opt/app ADD src /build WORKDIR /build RUN apt-get update RUN apt-get install -y libjpeg-dev build-essential gcc RUN ./configure RUN make RUN make install RUN mkdir -p /opt/app RUN cp -r /build/out/* /opt/app/

7. Huge Containers – solutions

8. Building Images - The issues

Build typically requires access to private repos

➔Need to pull from them without leaving

passwords/keys around (see issue #3)

Build typically requires env-specific tools

➔Need to run the tools in a controlled environment ◆ (Example: import dependency analysis)

8. Building Images - Strategy #1

public

repos

private

repos

build server docker build

image

image

repository

$ cat Makefile build: Dockerfile git clone git@github.com:rs/app cd app && rm -rf ./.git bundle install docker build -t rightscale/app . cd .. && rm -rf app $ cat Dockerfile FROM rightscale/ruby-215 ADD app /app

➔ Pull code onto build box using creds; copy into container image from local FS

➔ But: Image build is at the mercy of the build box env

8. Building Images - Strategy #2

public

repos

private

repos

build server

image

repository

➔ Image build runs within appropriate environment

base

image

base image docker run docker commit

$ cat Makefile build: docker run \ --volume $SRC:"/src" \ --name "tmp_app" \ $BASE_IMAGE \ sh -c $BUILD_SCRIPT && docker commit tmp_app tmp $ cat Dockerfile FROM tmp RUN "rainbows -c config.rb"

9. Backups

Userguide: backup-restore-or-migrate-data-volumes ● Create DB container with /data volume

● Backup /data “anytime” from the VM

● Or launch 2nd backup container with --volumes-from

➣ Simple in a 1-off server, but how to automate in general?

… or don’t back containers up

10.Docker Clusters

● Does Docker Cluster software solve all these issues?

● Swarm, Kubernetes, Mesos, Fleet, … o apparently not (yet?)

● And, they require an overlay network…

VM 1

Container 1

Runs app 1

172.16.4.3

VM 2

Container 2

Runs app 1

172.16.4.6

10.0.0.1 10.0.0.2

Wrapping up

A phased approach:

● Start with highest impact / least disruption

● Typically around dev & continuous integration

Beware of issues and pitfalls

● Good news is that you’re not alone

● Most become real issues only at scale

● Use pragmatic work-arounds for the time being

Overall very promising and great to work with

Q&A

Read our blogs on Docker

www.rightscale.com/blog

Containers vs VMs? Combining Both for Cloud Portability Nirvana

eng.rightscale.com

Dockerizing RightScale

Containers vs. Virtual Machines

Differences:

● Size

● Boot time

● Performance

● Isolation

● Compatibility

Containers vs. Processes

ME

M

PG

M

regs

pro

c

textbook process

ME

M

regs

pro

c

/etc

/lib

/bin

real process

ME

M

regs

pro

c

/etc

/lib

/bin

container

ne

t

⇒Containers are processes with env, not mini-VMs

Host 1

VM 1

Tenant A

Containers in a VM

Container 1

Runs app 1

Tenant A

Container 2

Runs app 2

Tenant A

VM 1

Tenant B

Container 3

Runs app 3

Tenant B

Container 4

Runs app 4

Tenant B

● Containers are produced

by development

● VMs are produced and

managed by ops

● Hosts are managed by the

cloud provider

Do not trust containers to provide

a hard security boundary

THANK YOU.