Weave User Group Talk - DockerCon 2017 Recap

Patrick Chanezon, @chanezon

Docker for Devs and Ops

What’s new and What’s next

May 2017

French

Polyglot

Platforms

Software Plumber

San Francisco

Developer Relations

@chanezon

Docker

The world needs

tools of mass innovation

A programmable Internet would be the ultimate

tool of mass innovation

A commercial product,

built on

a development platform,

built on

infrastructure,

built on

standards.

Docker is building a stack to program the Internet

Docker is building a stack to program the Internet

CE

EE

Docker for Developers

The best tools…

1. Get out of the way

The best tools…

1. Get out of the way

2. Adapt to you

The best tools…

1. Get out of the way

2. Adapt to you3. Make the powerful

simple

The best tools…

enterprise edition

Ubuntu

Fedora

Mac

Azure

CentOS

Windows 10

AWS

Debian

community edition

Ubuntu

Windows Server

Azure

CentOS

Suse

Red Hat

AWS

Oracle Linux

Bettertools for developers

Docker removes frictionin the development cycle

How to remove developer friction in 3 easy steps

Step 1.

Developer

complains

about detail

Step 2.

Fix Detail

Step 3.

Repeat

FOREVER

In the developer experience, details add up...

Example #1

My container images are too big!“My container images are too big!”

Introducingmulti-stage builds

Example #1

“My container images are too big!”

Build smaller images withmulti-stage builds

First stage: complete build environment

Second stage: minimal runtime environment

One Dockerfile, one build

FROM big-buildbase……

FROM tiny-runbase

…COPY --from=0/artifact /run/app …

One Dockerfile,one build

Stage 0: large build environment

Stage 1: minimal run environment

Copy artifacts from one stage to the nextOnly copy what you need!

Build smaller images withmulti-stage builds

MAC AWS

“I wish it was easier to

take my app from desktop

to cloud”

Example #2

IntroducingDESKTOP-TO-CLOUD

“I wish it was easier to

take my app from desktop

to cloud”

Example #2

Desktopintegration

27

Built-in collaboration with Docker

Cloud & Docker ID

docker.com/getdocker

edge channel

Available in an Edge release near you

enterprise edition

Ubuntu

Fedora

Mac

Azure

CentOS

Windows 10

AWS

Debian

community edition

Ubuntu

Windows Server

Azure

CentOS

Suse

Red Hat

AWS

Oracle Linux

Docker for Ops

Going to Production is Hard

Going to Production SECURELY is EXTREMELY Hard

Challenges to a

secure production:

Distributed Systems1

Challenges to a

secure production:

Distributed Systems1

Solutions:

Distributed systems are just

more systems.

Use the same tools.

Solutions:

Challenges to a

secure production:

Distributed Systems1 Secure orchestration

Orchestration

Container Runtime

OS

Infrastructure Management

Let’s talk about secure orchestration

Application Services

Raft StoreNode

IdentitySecrets

Routing

MeshEncrypted Networking

Application Services

Core Orchestration Engine

Secure Orchestration with SwarmKit

Secure Node

Introduction

SwarmKit

SWMTKN-1-mx8suomaom825bet6-cm6zts22rl4hly2

Known

Prefix

Token

Version

Hash

of Root CA

Random

Secret

Cryptographic

Node Identity

SwarmKit

MTLS

Between All

Nodes

SwarmKit

Cluster

Segmentation

SwarmKit

Encrypted

Networks

SwarmKit

Secure

Secret

Distribution

SwarmKit

Moby

Orchestration

Container Runtime

OS

Infrastructure Management

Container Platform Layers

Application Services

Docker is a platform made of components

Raft StoreNode

IdentitySecrets

Routing

Mesh

Overlay

Networking

Swarm Orchestration

Engine

Application Services

12,000,000,000

11,000,000,000

10,000,000,000

9,000,000,000

8,000,000,000

7,000,000,000

6,000,000,000

5,000,000,000

4,000,000,000

3,000,000,000

2,000,000,000

1,000,000,000

Notary

runC

containerd

HyperKit , VPNKit, DataKit

SwarmKit

libcontainer

libnetwork

InfraKit

2013 2014 2015 2016 2017

1M2014

PULLS

1B2015

PULLS

6B2016

PULLS

12B2017

PULLS

linuxKit

LinuxKitA toolkit for building secure, portable and lean operating systems for containers

Taking Dockermulti-platform

“I want Docker for X”

Desktop Server Cloud

I want Docker for…

Not every platform provides a Linux subsystem

Not every platform provides a Linux subsystem

Orchestration

Container Runtime

Linux Subsystem

Infrastructure Management

Application Services

The container movement needs asecure, lean, portable subsystem

The container movement needs

a secure, lean, portable Linux subsystem.

introducing

Only works with

containers

- Smaller attack

surface

- Immutable

infrastructure

- Sandboxed system

services

- Specialized patches

and configuration

Incubator for

security innovations

- Wireguard,

Landlock, KSPP

- MirageOS type

safe system

daemons

Community-first

security process

- Linux is too big

for any one

company to

secure it

- Participate in

existing Linux

security efforts

1. LinuxKit: a SECURE Linux subsystem

- Minimal size, minimal boot time

- All system services are containers

- Everything can be removed or

replaced

2. LinuxKit: a LEAN Linux subsystem

- Desktop, server, IoT, mainframe

- Intel & ARM

- Bare metal & virtualized

3. LinuxKit: a PORTABLE Linux subsystem

Docker and Microsoft collaborate to bringLinux containers to Windows

+ +

https://github.com/linuxkit/linuxkit

Get Started with LinuxKit

MobyAn open framework to assemble specialized container systems without reinventing the wheel.

Pioneers 2013 - 2014

Production Model: open-source!

Use case: cloud native apps on Linux server

Early Adopters 2015 - 2016

Production Model: OPEN COMPONENTS

Mainstream 2017 - 2018Containers are spreading to every category of computing:

server, datacenter, cloud, IoT, desktop, mobile…

Case study:

Specializing Docker for the mainstream

Desktop Server Cloud

The open component model shows its limits…

The auto industry has solved this problem: COMMON ASSEMBLIES.

Scaling the Docker production model: share components AND

ASSEMBLIES.

It’s time to take our ecosystem to the next level…

By collaborating on components AND COMMON ASSEMBLIES.

– Library of 80+ components

– Package your own

components as containers

– Reference assemblies

deployed on millions of nodes

– Create your own assemblies

or start from an existing one

A framework to assemble

specialized container

systems without

reinventing the wheel.

Docker uses Moby for its

open-source

– Thousands of contributors,

hundreds of patches/week

– Component development

– Specialized assembly

development

– Integration tests

– Architecture design

– Integration with other projects

– Experimentation and bleeding

edge features

Docker uses Moby for its

open-source...

and so can you!

– Community-run

– Open governance inspired by

the Fedora project

– Plays well with existing

projects - no donation

necessary!

Moby and Docker

What it means for you

Moby helps you

innovate without tying

you to Docker

System BuildersDocker Users

Docker will better leverage

the ecosystem to innovate

faster for you

Moby transforms multi-month R&D projects into weekend projects.

locked-down Linux with remote attestation

Weekend project #1:

Notary

custom CI/CD stack

Weekend project #2:

Notary Registry Docker Builder

+

custom CI/CD stack + Debian+ Terraform

Weekend project #3:

Notary Docker Builder

+

Registry

“RedisOS”

Weekend project #4:

"RedisOS"for Windows

"RedisOS"for Mac

"RedisOS"for bare metal

HyperKit

bare metal

Etcd clustering on Google Cloud

Weekend project #5:

SSHD

Kubernetes on the Mac

Weekend project #6:

HyperKit

Getting Started

- Blog https://mobyproject.org/blog

- Twitter @moby

- Github moby/moby

Let’s take containers mainstream!

InfraKitA toolkit for building declarative, self-healing infrastructure.

What is it?

90

• Launched at LinuxCon, Berlin in October, 2016.

• Toolkit for building declarative, self-managing

distributed applications

• Active management with active controllers

• scaling groups, rolling updates

• monitoring / health checks

• connecting nodes to L4 / ingress

• Declarative infrastructure

Architecture

CLI

API

container orchestration

Where does it fit?

92

kubectl run nginx --image=nginx

gcloud container node-pools list --zone us-

central1-f --cluster MyWorkers

aws autoscaling update-auto-scaling-group

--auto-scaling-group-name MyWorkers

docker create service nginx …

infrakit group describe workers

az vmss create --resource-group vmss-

test-1 --name MyWorkers

container orchestration

infrastructure orchestrationinfrastructure orchestration

list, err :=

group.Controller.Describe(“workers”)

App Opscontainer orchestrationApp Ops

One console across environments

93

kubectl run nginx --image=nginx docker create service nginx …

infrakit group describe workers

container orchestration

infrastructure orchestration

list, err :=

group.Controller.Describe(“workers”)

AWS RackHDAZ GCP OneVIEWMAASKVM VMW

Cloud Ops Hardware OpsCluster Ops

Configuration

Example config file (zk.conf): Group configuration = Instance + Flavor

{"Properties": {

/* raw configuration */

}}

{"groups" : {

"my_zookeeper_nodes" : {"Properties" : {

"Instance" : {"Plugin": "instance-vagrant","Properties": {

"Box": "bento/ubuntu-16.04"}

},"Flavor" : {

"Plugin": "flavor-zookeeper","Properties": {

"type": "member","IPs": ["192.168.1.200", "192.168.1.201", "192.168.1.202"]

}}

}}

}}

Current Status

Support more platforms

96

• Compute:

• Bare-metal: HP OneView, MAAS, RackHD

• Public cloud: AWS, GCP

• MacOS X (HyperKit); Docker containers

• Coming soon: Azure, IBM, Digital Ocean,

Packet, libvirt

• Other resource types

• AWS - vpc, subnets, gateways, etc.

Improve usability

97

• Templates

• Complex scripts and configuration in any format;

no more escape quotes in JSON

• Fetch templates from remote repositories

• Playbooks

• CLI - flags, prompts — config driven and

dynamic

• Share “playbooks” from remote repositories

Improve core system

98

• High Availability — Swarm Mode or etcd

• New Plugin types — Metadata and Events

• Metadata: cluster-wide sysfs and reflection

• Events - publish / subscribe

• Remote client access: infrakit -H host:port to remote cluster

Road Map

Use Cases

100

• Support container orchestration

• bootstrapping + day N management

• API for cluster autoscaling

• k8s, Docker Swarm Mode

• Bare-metal + GPU provisioning

• IoT — LinuxKit integration / custom kernel

deployment

Improve usability

101

• Finalize API / Schema for 1.0

• Make it easy to consume

• Simplify setup - fewer daemons and binaries

• Embeddable / vendor API

• Sensible CLI for stable / experimental features

• Make it easy to extend / contribute

• metadata / instance plugins

• playbooks / reusable templates

• community CI / compatibility testing

• Documentation

Improve core system

102

• Provisioning of diverse resource types

• networks / proxies / load balancers

• GPU

• Stability / performance of core controllers

• Asynchronous messaging - mqtt, natsd, amqp

• Monitoring + Health check SPI

Support more platforms

103

• Direct libvirt / KVM / CUDA

• Better bare-metal / hardware ops integration

• Kernel image build pipeline — LinuxKit

Build, test, and deploy clusters from infrastructure

definitions to kernel images

Get involved

https://github.com/docker/infrakit

dockercommunity.slack.com: #infrakit

Learn More

- blog.docker.com

- mobyproject.org

THANK YOU