Intro to Docker and clustering with Rancher from scratch

john culvinergithub: github.com/johnculvinerblog: johnculviner.comtwitter: @johnculvineremail: john@johnculviner.com

intro to

with a side of

About Me Free range, sometimes organic

Full-stack Independent Consultant @ Veritas in Roseville

Backend DevOps (Docker, Ansible, Linux etc) NoSql (ElasticSearch, MongoDB) Distributed systems (RabbitMQ, Kafka etc.) Node.js Groovy/Spring/Java C#

Front End Angular.js, React.js, Knockout.js, Durandal.js, jQuery, CSS/SASS etc. SPA development

Open Source “Street Cred” AngularAgility jQuery File Download FluentKnockoutHelpers

OverviewDocker

How does it work Why would I use it

Rancher What does it give me

Building a Clustered Docker + Rancher environment from scratch Terraform (DigitalOcean) Ansible Node.js Microservice

Objective:To leave feeling confident about if Docker might make sense for your next project (or might not!) and how to get started easily if it looks like it is the right tool for the job for you.

What is ?It’s all about the containers!

ImagesInternal Docker

Registryhostname: MY_REG:5000myapp:1.0

myapp:1.1

yourapp:1.0

yourapp:1.1

…

Public Docker Registry

AKA: hub.docker.comelasticsearch:5.0.0

elasticsearch:5.0.1

rabbitmq:3.6.4

rabbitmq:3.6.5

…

Any machine running DockerMY_REG:5000/

myapp:1.1elasticsearch:5.0.1

may equalwhen :tag not specified defaults to

Confused? Container vs Image

A container is an “instance” of an “immutable” imageCould be running or stoppedMy machine running Docker for Mac

Loaded Imagesmongo:latest

Running Containers

Image Namemongo:latestContainer Namemyfirstmongo

Image Namemongo:latestContainer Name mysecondmongo

…

Moderate Mongo Mess

mongo:latest isn’t terribly useful to know what the version really is

There is no external/port level access to the containers There are no volume mounts for persistent data (very bad for

perf on with high I/O applications) If the container dies it’s not coming up again without me

restarting it Fortunately? there is:

docker run --name=myfirstmongo --detach --publish="27017:27017" --restart=always --volume="/some/local/path:/data/db" mongo

A better way: docker-compose

Tearse & readily source controlled YAML definition

docker-compose.yml

Idempotence(to an extent)

docker-compose for CI/CD!Run isolated integration testing CI/CD of your whole

app stack from anywhere! (local, Jenkins etc.)Builds a local Dockerfile

Define DNS aliases of references

only available from my_appstdout/err comes out to pass/fail Jenkins build

Test command: stdout/err comes out of container to pass/fail the buildMongo only addressable to my_app at DNS “mongodb”No stdout/err

Real live chrome/selenium server in a container using xvfb

Benefits of Images & Containers

Better Isolation & Consistency with ImagesDocker Repository vs. Artifactory, NPM, Nuget etc.Debug a production image on my local machine

EX: Run 10 different YOUR_FAV_LANG apps using 10 different versions the runtime all on port 8080 on same box**with a SDN (software defined network)

Security**When you don’t run as root, use SELinux,

sandbox volumes among other things

+Docker

Building images with layersDone with a Dockerfile, lets do it!See layers with “docker inspect IMAGE_NAME”What we did:

image layer: alpine:latest

image layer: first_file added

image layer: second_file added

container: second-container

container: first-container

Layer re-creation/sharing Docker will re-use existing layers when it can:

When a layer changes subsequent layers are invalidated otherwise they are re-used

This effects:

Proportion of Image Size

Changes every build(probably)

npm install only runsif package.json (a dependency/package manifest) changes!

pull/push HTTP trafficserver filesystem usagerepository storage space

BUILD TIMES!

Docker ObservationsSet up development environment quickly

with a docker-compose for a projectE2E Integration testing easily with a docker-

composeImage consistency to production

stdout/stderr aggregation

QA serversmyapp:1.2.

3

PROD servers

myapp:1.2.3

DEV serversmyapp:1.2.

3

- Commit- Build- Test

deploy

server-a

server-b

server-c

server-d

ElasticSearch+

Kibana

stdout/err from all containers

Well that was cool for DEV but…

How do I run containers on multiple machines and orchestrate them?

How do I ensure HA (high availability)How do I load balance HTTP/S applicationsHow do I schedule based on load

Does Docker actually make sense to run real applications in PROD?

*well I have at least with less work and less downtime than other approaches I’ve encountered… so far

Partial lay of the land*

*as I see it: grain of salt please

+

What is ? A really slick UI that illustrates what is going

on in a very clear mannerActually helps you learn real Docker (full API

surface almost!) visually and then helps you script things after you have “pointed and clicked your way to success”

Easily runs in Docker container(s)Container orchestration/clustering support for a

variety of different platforms:

What is Cattle?A relatively simple container orchestration framework

that is natively supported by RancherPros

Built in layer 5 (haproxy based) load balancer that supports scaling, rolling upgrades, rollback changes etc.

Slick SDN (Software Defined Network) does DNS based round-robin inter-container network resolution

Simpler & quicker to get going than anything else “3AM Googleability” is very high / vibrant community Works with Docker rather than against it Realistically free! I’ve battle tested it and has worked well so far

Cons Scheduler is rather simple / no automatic container

creation support

+ +Setup entire stack from scratch in a repeatable

(idempotent), clear & source controllable manner*Some of the Rancher stuff we will “point and click our

way to success” for brevity and to show you the UI but I’ve done it before with 100% Ansible + docker/rancher-compose files.

RequirementsPOSIX shellDigitalOcean account with API key env variableSSH

~/.ssh/id_rsa + ~/.ssh/id_rsa.pub setupAnsible (get with Python, PIP)TerraformWeb browser

The Goal

docker0 docker2docker1 docker3rancher/server

rancher/agent

rancher/agent rancher/agent rancher/agent

Idempotent Cloud VM creation tool

Cloud VM Provider

Ubuntu 16.04Cloud VMsw/

Containers

IdempotentServer

ProvisioningTool

johnculviner/nodejs-echo-hostname

johnculviner/nodejs-echo-hostname

johnculviner/nodejs-echo-hostnamejohnculviner/nodejs-echo-hostname

johnculviner/nodejs-echo-hostnamejohnculviner/nodejs-echo-hostname

…… …

rancher haproxy load balancer

HTTPTraffic

+ few SSH commands

The codehttps://github.com/johnculviner/docker-rancher-presentation

E2E IRL IdeasA Jenkins pipeline build

questions/comments?

john culvinergithub: github.com/johnculvinerblog: johnculviner.comtwitter: @johnculvineremail: john@johnculviner.com