© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Scott Windsor, Sr. Software Development Engineer, AWS

Mao Geng, Site Reliability Engineer, Pinterest

December 1, 2016

CON401

Amazon ECR Deep Dive

on Image Optimization

What to Expect from the Session

• Anatomy of Docker images

• Optimizing image builds

• Overview of Docker Registry V2 API

• CI/CD best practices

• ECR learnings from Pinterest

About me

Sr. Software Engineer on AWS,

EC2 Container Registry

Amazon EC2 Container Registry

Highly available SecureFully managed

Anatomy of Docker Images

Docker Images

Packaged

application code

Reproducible Immutable Portable

Docker Images

Docker Containers

Dockerfiles

Dockerfiles: Hello, world

FROM alpine

MAINTAINER "Scott Windsor"

CMD ["echo", "Hello, re:Invent!"]

Dockerfiles: Hello, world

$ docker build -t hello-reinvent -f Dockerfile.hello .

Sending build context to Docker daemon 699.1 MB

Step 1 : FROM alpine

---> baa5d63471ea

Step 2 : MAINTAINER "Scott Windsor"

---> Running in 06bdf87b81ea

---> 4f73f4a232f9

Removing intermediate container 06bdf87b81ea

Step 3 : CMD echo Hello, re:Invent!

---> Running in cf349feca920

---> fa069f014b6a

Removing intermediate container cf349feca920

Successfully built fa069f014b6a

Dockerfiles: Hello, world

$ docker history hello-reinvent

IMAGE CREATED CREATED BY SIZE

fa069f014b6a 3 minutes ago /bin/sh -c #(nop) CMD ["echo" "Hello, re:Inv 0 B

4f73f4a232f9 3 minutes ago /bin/sh -c #(nop) MAINTAINER "Scott Windsor" 0 B

baa5d63471ea 3 weeks ago /bin/sh -c #(nop) ADD file:7afbc23fda8b0b3872 4.803 MB

Dockerfiles: Hello, world

$ docker history alpine

IMAGE CREATED CREATED BY SIZE

baa5d63471ea 3 weeks ago /bin/sh -c #(nop) ADD file:7afbc23fda8b0b3872 4.803 MB

Dockerfiles: Hello, world

$ docker build -t hello-reinvent -f Dockerfile.hello .

Sending build context to Docker daemon 699.1 MB

Step 1 : FROM alpine

---> baa5d63471ea

Step 2 : MAINTAINER "Scott Windsor"

---> Using cache

---> 4f73f4a232f9

Step 3 : CMD echo Hello, re:Invent <3 <3 <3!

---> Running in 51080848d506

---> 19c7fb45a397

Removing intermediate container 51080848d506

Successfully built 19c7fb45a397

Optimizing Image Builds

Optimizing Image Builds: Hello, ruby

FROM ruby:2.3

ADD Gemfile Gemfile.lock /

RUN bundle install

ADD hello.rb .

CMD ["ruby", "hello.rb"]

Optimizing Image Builds: Hello, ruby

source 'https://rubygems.org'

gem "colorize"

Optimizing Image Builds: Hello, ruby

#!/bin/env ruby

require "colorize"

color_wheel = [:light_magenta, :light_blue,

:light_green, :light_red, :light_yellow].shuffle.cycle

greeting = "Hello, re:Invent!"

s = greeting.each_char.map do |c|

c.colorize color_wheel.next.to_sym

end.join ''

puts s

Optimizing Image Builds: Hello, ruby

Sending build context to Docker daemon 5.111 MB

Step 1 : FROM ruby:2.3

---> 45766fabe805

Step 2 : ADD Gemfile Gemfile.lock /

---> f30a7987497f

Removing intermediate container d9828a354cab

Step 3 : RUN bundle install

---> Running in 3e501ce02c9c

Fetching gem metadata from https://rubygems.org/.............

Fetching version metadata from https://rubygems.org/.

Installing colorize 0.8.1

Using bundler 1.13.5

Bundle complete! 1 Gemfile dependency, 2 gems now installed.

Bundled gems are installed into /usr/local/bundle.

---> 0e217ba0ced7

Removing intermediate container 3e501ce02c9c

Optimizing Image Builds: Hello, ruby

Step 4 : ADD hello.rb .

---> b41a47dceb98

Removing intermediate container 486b3b403224

Step 5 : CMD ruby hello.rb

---> Running in 8e1e1428d56c

---> 2cb069515f81

Removing intermediate container 8e1e1428d56c

Successfully built 2cb069515f81

Optimizing Image Builds: Hello, ruby

$ docker run -it hello-ruby

Hello, re:Invent!

Optimizing Image Builds: Hello, ruby

#!/bin/env ruby

require "colorize"

color_wheel = [:light_magenta, :light_blue, :light_green,

:light_red, :light_yellow].shuffle.cycle

greeting = "Hello, re:Invent "

s = greeting.each_char.map do |c|

c.colorize color_wheel.next.to_sym

end.join ''

puts s + "<3 <3 <3".colorize(:light_magenta) + "!"

Optimizing Image Builds: Hello, ruby

Sending build context to Docker daemon 5.124 MB

Step 1 : FROM ruby:2.2

---> 8fecf438974e

Step 2 : ADD Gemfile Gemfile.lock /

---> Using cache

---> 3b8bc7055d6f

Step 3 : RUN bundle install

---> Using cache

---> 9a8fe024a6c1

Step 4 : ADD hello.rb .

---> d1bd2fd4e2d2

Removing intermediate container b323bf7041a4

Step 5 : CMD ruby hello.rb

---> Running in 55b4ec70135f

---> 706d48ba1af3

Removing intermediate container 55b4ec70135f

Successfully built 706d48ba1af3

Optimizing Image Builds: Hello, ruby

$ docker run -it hello-ruby

Hello, re:Invent <3 <3 <3!

Tips

• Stick to specific versions of base layers

• Separate application code from dependencies

Optimizing Image Builds: Hello, golang

FROM golang:1.7

ADD main.go .

RUN go build -o hello

CMD ["./hello"]

Optimizing Image Builds: Hello, golang

package main

import "fmt"

func main() {

fmt.Println("你好, re:Invent!")

}

Optimizing Image Builds: Hello, golang

$ docker build -t hello-golang -f Dockerfile.golang-combined .

Sending build context to Docker daemon 5.226 MB

Step 1 : FROM golang:1.7

---> 47734a1408b7

Step 2 : ADD main.go .

---> 85d0f410b40d

Removing intermediate container ae79fe0c7846

Step 3 : RUN go build -o hello

---> Running in bc6e71b350ae

---> 5b59c94af88f

Removing intermediate container bc6e71b350ae

Step 4 : CMD ./hello

---> Running in 252c9b2ef359

---> 3f982ea19213

Removing intermediate container 252c9b2ef359

Successfully built 3f982ea19213

Optimizing Image Builds: Hello, golang

$ docker run -it hello-golang

你好, re:Invent!

Optimizing Image Builds: Hello, golang

$ docker images hello-golang

REPOSITORY TAG IMAGE ID CREATED SIZE

hello-golang latest 3f982ea19213 4 days ago 674 MB

Optimizing Image Builds: Hello, golang

$ docker history hello-golang

IMAGE CREATED CREATED BY SIZE

COMMENT

3f982ea19213 4 days ago /bin/sh -c #(nop) CMD ["./hello"] 0 B

5b59c94af88f 4 days ago /bin/sh -c go build -o hello 1.634 MB

85d0f410b40d 4 days ago /bin/sh -c #(nop) ADD file:22b3017a338f88f902 78 B

47734a1408b7 2 weeks ago /bin/sh -c #(nop) COPY file:f6191f2c86edc9343 2.478 kB

<missing> 2 weeks ago /bin/sh -c #(nop) WORKDIR /go 0 B

<missing> 2 weeks ago /bin/sh -c mkdir -p "$GOPATH/src" "$GOPATH/bi 0 B

<missing> 2 weeks ago /bin/sh -c #(nop) ENV PATH=/go/bin:/usr/loca 0 B

<missing> 2 weeks ago /bin/sh -c #(nop) ENV GOPATH=/go 0 B

<missing> 2 weeks ago /bin/sh -c curl -fsSL "$GOLANG_DOWNLOAD_URL" 243.6 MB

<missing> 2 weeks ago /bin/sh -c #(nop) ENV GOLANG_DOWNLOAD_SHA256 0 B

<missing> 2 weeks ago /bin/sh -c #(nop) ENV GOLANG_DOWNLOAD_URL=ht 0 B

<missing> 2 weeks ago /bin/sh -c #(nop) ENV GOLANG_VERSION=1.7.1 0 B

<missing> 2 weeks ago /bin/sh -c apt-get update && apt-get install 138.8 MB

<missing> 3 weeks ago /bin/sh -c apt-get update && apt-get install 122.6 MB

<missing> 3 weeks ago /bin/sh -c apt-get update && apt-get install 44.3 MB

<missing> 3 weeks ago /bin/sh -c #(nop) CMD ["/bin/bash"] 0 B

<missing> 3 weeks ago /bin/sh -c #(nop) ADD file:c6c23585ab140b0b32 123 MB

Optimizing Image Builds: Hello, golang

FROM golang:1.7

VOLUME /build # Add volume for copying out

ADD main.go .

RUN go build -o hello

CMD ["cp", "hello", "/build/hello"]

# Copy binary out of container into volume

Optimizing Image Builds: Hello, golang

FROM scratch # Use scratch (empty)

ADD hello . # Add binary

CMD ["hello"] # Run!

Optimizing Image Builds: Hello, golang$ docker build -t hello-golang-build -f Dockerfile.build .

Sending build context to Docker daemon 1.639 MB

Step 1 : FROM golang:1.7

---> 47734a1408b7

Step 2 : VOLUME /build

---> Running in 53ba8193299a

---> b86a8ebd4f66

Removing intermediate container 53ba8193299a

Step 3 : ADD main.go .

---> b197080c46cd

Removing intermediate container 8df45d03b69d

Step 4 : RUN go build -o hello

---> Running in af46ee1da8bb

---> 219aacd4c756

Removing intermediate container af46ee1da8bb

Step 5 : CMD cp hello /build/hello

---> Running in f43137a7aafe

---> c27e7cb97e2d

Removing intermediate container f43137a7aafe

Successfully built c27e7cb97e2d

Optimizing Image Builds: Hello, golang$ mkdir build

$ docker run -v ${PWD}/build:/build hello-golang-build

$ docker build -t hello-golang -f Dockerfile.run .

Sending build context to Docker daemon 1.644 MB

Step 1 : FROM scratch

--->

Step 2 : ADD ./build/hello .

---> 0b84a2ca1810

Removing intermediate container 21467dda26ab

Step 3 : CMD ./hello

---> Running in 6baa329f27b4

---> 316b3116d7ef

Removing intermediate container 6baa329f27b4

Successfully built 316b3116d7ef

Optimizing Image Builds: Hello, golang

$ docker run -it hello-golang

你好, re:Invent!

Optimizing Image Builds: Hello, golang$ docker images hello-golang

REPOSITORY TAG IMAGE ID CREATED SIZE

hello-golang latest 316b3116d7ef 50 seconds ago 1.634 MB

$ docker history hello-golang

IMAGE CREATED CREATED BY SIZE

316b3116d7ef About a minute ago /bin/sh -c #(nop) CMD ["./hello"] 0 B

0b84a2ca1810 About a minute ago /bin/sh -c #(nop) ADD file:82ff6b7578af24a8f0 1.634 MB

Tips

• Bring only what you need at runtime

• Separate build vs. runtime containers

• Reducing layers and sizes will improve pull times

Advanced Tips

• Building your own base images:

https://docs.docker.com/engine/userguide/eng-

image/baseimages/

• Squashing layers (in Docker1.13.0-rc2)

https://github.com/docker/docker/commit/362369b4bbea

38881402d281ee2015d16e8b10ce

Docker Registry V2 API

Pulling an Image

$ docker pull <registry-uri>/<image-name>:<tag>

Docker daemon:1. Fetches image manifest at tag

2. For each layer that it doesn’t have:

1. Fetch layer

Pulling an Image

$ docker pull <registry-uri>/<image-name>:<tag>

Docker daemon:1. GET /v2/<image-name>/manifests/<tag>

2. For each layer it doesn’t have:

1. GET /v2/<image-name>/blobs/<digest>

Pushing an Image

$ docker push <registry-uri>/<image-name>:<tag>

Docker daemon:1. For each layer:

1. Check if layer exists

2. Initiate layer upload

3. Send layer data

4. Completes the upload

2. Pushes manifest when all layers complete

Pushing an Image

$ docker push <registry-uri>/<image-name>:<tag>

Docker daemon:1. For each layer:

1. HEAD /v2/<image-name>/blobs/<digest>

2. POST /v2/<image-name>/blobs/uploads/

3. PATCH /v2/<image-name>/uploads/<uuid>

4. PUT /v2/<image-name>/blobs/<uuid>?digest=<digest>

2. PUT /v2/<image-name>/manifests/<tag>

Docker V2 Registry

Fetching auth token and url:

$ TOKEN_RESULT=`aws ecr get-authorization-token`

$ ECR_TOKEN=`echo $TOKEN_RESULT \

| jq -r ".authorizationData[].authorizationToken"`

$ ECR_URL=`echo $TOKEN_RESULT \

| jq -r ".authorizationData[].proxyEndpoint”`

Docker V2 Registry

Getting username / password:

$ ECR_USERNAME=`echo $ECR_TOKEN | base64 -D | cut -d: -f1`

$ ECR_PASSWORD=`echo $ECR_TOKEN | base64 -D | cut -d: -f2`

Logging in:

$ docker login –u $ECR_USERNAME –p $ECR_PASSWORD $ECR_URL

Docker V2 Registry

Create a repository:$ IMAGE_URI=`aws ecr create-repository --repository-name hello \

| jq -r ".repository.repositoryUri"`

Deleting (when we’re done):$ aws ecr delete-repository --repository-name hello --force

Docker V2 Registry

Pushing an image:

$ docker tag hello ${IMAGE_URI}

$ docker push ${IMAGE_URI}

Docker V2 Registry

Fetching an image manifest:$ curl –u ${ECR_USERNAME}:${ECR_PASSWORD} \

${ECR_URL}/v2/hello/manifests/latest -o hello-manifest.json

$ cat hello-manifest.json

{

"schemaVersion": 1,

"name": "hello",

"tag": "latest",

"architecture": "amd64",

...

Docker V2 Registry

...

"fsLayers": [

{

"blobSum":

"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"

},

{

"blobSum":

"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"

},

{

"blobSum":

"sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f"

}

],

...

Docker V2 Registry...

"history": [

{

"v1Compatibility": "{\"architecture\":\"amd64\",\"author\":\"\\\"Scott

Windsor\\\"\",\"config\":{\"Hostname\":\"1d811a9194c4\",\"Domainname\":\"\",\"User\":\"\",\"Att

achStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":fals

e,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/

bin\"],\"Cmd\":[\"echo\",\"Hello, re:Invent \\u003c3 \\u003c3

\\u003c3!\"],\"Image\":\"sha256:4f73f4a232f9809fe21ef1a8fe0e2ef3140c38341d6590ad4e97a4f973b6360

a\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":[],\"Labels\":{}},\"co

ntainer\":\"51080848d50609793dd2c9821873254c5f96d0b7f5b9642749d80a7029f6c90c\",\"container_conf

ig\":{\"Hostname\":\"1d811a9194c4\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"A

ttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":fals

e,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/b

in/sh\",\"-c\",\"#(nop) \",\"CMD [\\\"echo\\\" \\\"Hello, re:Invent \\u003c3 \\u003c3

\\u003c3!\\\"]\"],\"Image\":\"sha256:4f73f4a232f9809fe21ef1a8fe0e2ef3140c38341d6590ad4e97a4f973

b6360a\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":[],\"Labels\":{}}

,\"created\":\"2016-11-14T01:21:09.976223694Z\",\"docker_version\":\"1.12.0-

rc3\",\"id\":\"07c891b928a4818a46a4e9a6c5bbbdec4be0d201290056ae593ada8a2c8dbf50\",\"os\":\"linu

x\",\"parent\":\"db1d48cc60e1d888c9c57d75e0bc64f6a7f70d6e3bd8af91f106d60ac65cb37e\",\"throwaway

\":true}"

},

...

Docker V2 Registry

...

{

"v1Compatibility":

"{\"id\":\"db1d48cc60e1d888c9c57d75e0bc64f6a7f70d6e3bd8af91f106d60ac65cb37e\",\"parent\

":\"4b59778f82f9d17a484a278bd23d2d0b3c7ddcf022ab5250cf4f59308b3bc3f5\",\"created\":\"20

16-11-14T01:08:23.946015875Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c

#(nop) MAINTAINER \\\"Scott Windsor\\\"\"]},\"author\":\"\\\"Scott

Windsor\\\"\",\"throwaway\":true}"

},

{

"v1Compatibility":

"{\"id\":\"4b59778f82f9d17a484a278bd23d2d0b3c7ddcf022ab5250cf4f59308b3bc3f5\",\"created

\":\"2016-10-18T20:31:22.321427771Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c

#(nop) ADD file:7afbc23fda8b0b3872623c16af8e3490b2cee951aed14b3794389c2f946cc8c7 in /

\"]}}"

}

],

...

Docker V2 Registry

...

"signatures": [

{

"header": {

"jwk": {

"crv": "P-256",

"kid": "R3RU:PVFU:SPAV:GX42:JCTL:NTFT:W2BC:6S6N:IAU2:YGVJ:URDN:MS5G",

"kty": "EC",

"x": "HowD731zWxKgRhk5r2mywedUJGV6thJru8YqMSVOb9M",

"y": "G8Zi951QEjSLHcQeX1H9Bv9hAYFsvVG-mOK8rbRnrKg"

},

"alg": "ES256"

},

"signature":

"j0GQtDPSPMCrNUPBnZllOhltEaCkAfrHdpdLZGyuFLOE7qQvgf1wqeeHRZSY1mIoqSx3jiDPlEWPVlnrA2lACQ",

"protected":

"eyJmb3JtYXRMZW5ndGgiOjI3MzcsImZvcm1hdFRhaWwiOiJDbjAiLCJ0aW1lIjoiMjAxNi0xMS0yMlQxODoxNzowNVoifQ

"

}

]

}

Docker V2 Registry

Fetching layers

$ cat hello.manifest.json | \

jq ".fsLayers[].blobSum" -r \

> hello.layers.txt

$ <hello.layers.txt xargs -I {} \

curl -L –u ${ECR_USERNAME}:${ECR_PASSWORD} \

${ECR_URL}/v2/hello/blobs/{} \

-o {}.tar.gz

Docker V2 Registry

Inspecting layers

$ du -hs sha256:*.tar.gz

du -hs sha*

2.2M sha256:3690ec4760f95690944da86dc4496148a63d85c9e31

00669a318110092f6862f.tar.gz

4.0K sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16

422d00e8a7c22955b46d4.tar.gz

Docker V2 Registry

Inspecting layers

$tar tfvz

sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092

f6862f.tar.gz

drwxr-xr-x 0 0 0 0 Oct 18 11:58 bin/

lrwxrwxrwx 0 0 0 0 Oct 18 11:58 bin/ash ->

/bin/busybox

lrwxrwxrwx 0 0 0 0 Oct 18 11:58 bin/base64 ->

/bin/busybox

lrwxrwxrwx 0 0 0 0 Oct 18 11:58 bin/bbconfig ->

/bin/busybox

-rwxr-xr-x 0 0 0 805032 Aug 12 07:38 bin/busybox

...

Docker V2 Registry

Inspecting layers

$ tar tfvz

sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8

a7c22955b46d4.tar.gz

$

Empty layer!

Tips

• Reducing number of layers reduces upload/downloads

needed

• Reducing layer contents reduces uploads/download time

• Minimizing layer changes results in faster deployments

Continuous

Integration/Deployment

Best Practices

CI/CD Best Practices

• Always tag each build with any of these schemes:

• Semantic version (i.e. “1.3.2-9”)

• Git SHA (i.e. “aebcbca”)

• Build Number (i.e., “127”)

• Build Id (i.e. “511d5e51-b415-4cb2-b229-b3c8a46b7a2f”)

• Avoid ”latest” or ”stable”

• This allows for rolling deployments (and rollback)

CI/CD Tips

• Build caches can help improve build times

• Reduce build image sizes & number of layers

• Build and push in same AWS region if possible

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Mao Geng

Pinterest

About Me

● Enterprise Software Developer

● Performance Engineer

● Site Reliability Engineer

gengmao

Mao Geng

64

Catalog of Ideas

75 billion pins

1.5 billion boards

150 million monthly active users

100 million weekly active users

80% of users are via mobile app

> 50% of users are from outside the U.S.

Docker @ Pinterest

EC2

instances

Amazon

ECR

Usage

Issues

● #22648 Unstoppable Zombie

Processes - AUFS

● #12327 pip install fails -

overlay

● #12080 Overlayfs

does not work with

unix domain

sockets

Tips

● Use the storage driver works for you

● Use newer kernel• “WE DO NOT BREAK USERSPACE!” - Linus Torvalds

● Bake Docker Engine in AMI

● Less is more• --net=host

• --volume

● Use new version Docker Engine

Build Images of Mono Repos

● Python, Java, C++, Golang

● Build vs Runtime images

● One build image per mono repo could be very large and

time-consuming to build

● Separate dependencies into multiple build images• Organize a hierarchy of build images and use Makefile to

manage their build order

• Use shell script to calculate checksum of a build image’s

dependencies, and use the checksum to tag build image

● --cache-from probably is better solution

ECR login token

● Deal with TOOMANYREQUESTS: Rate exceeded

● Share docker config cross users

• $(aws ecr get-login) in cron

• DOCKER_CONFIG=/etc/.docker/config.json

docker-credential-ecr-login

● https://github.com/awslabs/amazon-

ecr-credential-helper

•/usr/bin/docker-credential-ecr-

login

•cache token for ½ refresh

window•Authenticating Amazon ECR

Repositories for Docker CLI with

Credential Helper

ECR with Mesos

● URIS: [ “file:///etc/docker.tar.gz” ]

• Marathon: Using a Private Docker

Registry

● --docker-

config=/etc/.docker/config.json

• Mesos >= 1.0

• Mesos Configuration

● #4278 Marathon Web UI not

support mesos containerizer

Thank you!

Remember to complete

your evaluations!