Open Source Docker John Willis

Director of Ecosystem Development

About Me

• One of the founding members of the “Devops” movement.

• Author of the “Devops Handbook”. • Author of the “Introduction to Devops” on

Linux Foundation edX. • Podcaster at devopscafe.org • Devops Enterprise Summit - Cofounder • Found of Socketplane (Acquired by Docker) • Formally Director of Devops at Dell • Formally Director at Chef • 10 Startups over 25 years

2

Github: botchagalupe/my-presentationsTwitter: botchagalupeWechat: botchagalupe

Devops Practices and Patterns• Continuous Delivery

• Everything in version control • Small batch principle • Trunk based deployments • Manage flow (WIP) • Automate everything

• Culture • Everyone is responsible • Done means released • Stop the line when it breaks • Remove silos

3itrevolution.com/devops-handbookhttp://www.bookdepository.com/

The world needs tools of mass innovation

A programmable Internet would be the ultimate tool of mass innovation

Let’s eliminate friction in the development cycle

A commercial product,

built on a development platform,

built on infrastructure,

built on standards.

Docker is building a stack to program the Internet

8

Docker Project Sponsor •Primary authors, contributor maintainer •6B+ Downloads, 3000+ Contributors, 500,000+ Applications •100’s of ecosystem partners •Millions of developers use Docker. Millions of servers run Docker Commercial Docker Solutions •Integrated solutions to build, ship, run Docker at scale

• Orchestration, registry, security, workflow, control plane • CaaS (containers as a service)

•Official providers of commercial technical support •10K’s cloud customers, 300+ F500 customers

About Docker, Inc.

Gerber, Anna. “The State of Containers and the Docker Ecosystem: 2015” O’Reilly, September 2015

Docker users already running in production

60%

China is part of Worldwide Docker Community

Meetups in Beijing, Changsha, Chengdu, Chongqing, Dalian, Fuzhou, Guangzhou, Hangzhou, Hong Kong, Nanjing, Qingdao, Shanghai, Shenzen, Suzhou, Tianjin, Wuhan, Xi’an, Xiamen, and Zhuhai

The Docker ecosystem

Dev Tools

Official Repositories

Operating Systems

Big Data

Service Discovery

Build / Continuous Integration

Configuration ManagementConsulting &Training

Management

Storage

Clustering & Scheduling

Networking

Infrastructure & Service Providers

Storage

Security

Monitoring & Logging

10

Docker Platform

COMPARING CONTAINERS AND VIRTUAL MACHINES

Isolation using Linux kernel featuresnamespaces ● pid ● mnt ● net ● uts ● ipc ● user

cgroups ● memory ● cpu ● blkio ● devices

Image layers

Docker Engine extensibility and plugins• Built in orchestration expands the opportunity for

the plugin to manage swarm wide vs a single Engine

• Updated architecture standardizes plugin process for ecosystem partners

– Benefits users and vendors – Standardized process of granting plugin permissions – Containerized plugins on roadmap

Docker Engine

Networking

Swarm Mode

Volumes

Plugins

Developer experience

1. Get out of the way

The best tools…

2. Adapt to you

3. Make the powerful simple

Docker for Mac Docker for Windows

Orchestration

Introducing the best way to orchestrate Docker: Docker.

Docker 1.12: now with orchestration built-in.

Swarm mode

Service API

Node identity

Built-in routing mesh

Docker 1.12: now with orchestration built-in.

Using the beta? You already have 1.12 installed.

> docker swarm init > docker service create

Docker CaaS

BUILD Development Environments

SHIP Registry: Secure Content &

Collaboration

RUN Control Plane: Deploy,

Orchestrate, Manage, Scale

Networking Volumes MonitoringLoggingConfig MgtCI/CD

IT OperationsDevelopers IT Operations

Docker CaaS Workflow

Docker Containers as a Service platform

25

BUILD Developer Workflows

SHIP Registry Services

RUN Management

Docker for Mac and Windows Docker Trusted Registry Docker Universal Control Plane

Docker Cloud

Docker Container Engine

Ecosystem Plugins and Integrations

Plumbing

2013

- 05

2013

- 06

2013

- 07

2013

- 08

2013

- 09

2013

- 10

2013

- 11

2013

- 12

2014

- 01

1,000,0000

2014

- 02

2014

- 03

2014

- 04

2014

- 05

2014

- 06

2014

- 07

2014

- 08

2014

- 09

2014

- 10

2014

- 11

2014

- 12

2015

- 01

2015

- 02

2015

- 03

2015

- 04

2015

- 05

2015

- 06

2015

- 07

2015

- 08

2015

- 09

2015

- 10

2015

- 11

2015

- 12

2016

- 0

1

1,000,000,000~

10,000,0009,000,0008,000,0007,000,0006,000,0005,000,0004,000,0003,000,0002,000,000

6,000,000,0005,750,000,0005,500,000,0005,250,000,0005,000,000,0004,750,000,0004.500,000,0004,250,000,0004,000,000,0003,750,000,0003,500,000,0003,250,000,0003,000,000,0002,750,000,0002,500,000,0002,250,000,0002,000,000,0001,750,000,0001,500,000,0001,250,000,000

2013

- 05

2013

- 06

2013

- 07

2013

- 08

2013

- 09

2013

- 10

2013

- 11

2013

- 12

2014

- 01

1,000,0000

2014

- 02

2014

- 03

2014

- 04

2014

- 05

2014

- 06

2014

- 07

2014

- 08

2014

- 09

2014

- 10

2014

- 11

2014

- 12

2015

- 01

2015

- 02

2015

- 03

2015

- 04

2015

- 05

2015

- 06

2015

- 07

2015

- 08

2015

- 09

2015

- 10

2015

- 11

2015

- 12

2016

- 0

1

~

2016

- 09

1,000,000,000~

10,000,0009,000,0008,000,0007,000,0006,000,0005,000,0004,000,0003,000,0002,000,000

6,000,000,0005,750,000,0005,500,000,0005,250,000,0005,000,000,0004,750,000,0004.500,000,0004,250,000,0004,000,000,0003,750,000,0003,500,000,0003,250,000,0003,000,000,0002,750,000,0002,500,000,0002,250,000,0002,000,000,0001,750,000,0001,500,000,0001,250,000,000

NotaryrunC •

containerd •

HyperKit , VPNKit, DataKit •

SwarmKit •

libcontainer •

libnetwork • • Docker 1.8 : Docker Content Trust

• Docker for Mac Docker for Windows

• Docker 1.12 with built-in orchestration

• Docker 0.9 : Pluggable execution

• Docker 1.7 : Multi-Host Networking

• Docker 1.11: OCI support

runC

RunCThe universal container runtime

https://runc.io

Open Container Initiative (OCI)

An open governance structure for creating

open industry standards: a common container runtime and

image format.

•A Linux Foundation Collaborative Project•Free from control by any particular vendor’s specific cloud stack or ecosystem

• Includes a specification, reference runtime* and now, a specified image format*seeded with runc + libcontainer by Docker

OCI Specs & Status> Announced June 20th,

2015> Charter signed on

December 8th, 2015> 49 current member

companies> Both specifications

nearing 1.0 release targets

https://opencontainers.orghttps://github.com/opencontainers

> Runtime specification: Release 1.0.0-rc2 / September 2016

https://github.com/opencontainers/runtime-spec/releases/tag/v1.0.0-rc2

1. Very close to an official 1.0 release of the runtime spec 2. Includes required core for Linux, Windows, and Solaris> Image format specification: Release 0.5.0 / September

2016

https://github.com/opencontainers/image-spec/releases/tag/v0.5.0

1. Seeded with Docker registry v2.2 specification 2. v1.0.0-rc1 release being voted/approved on mailing list

runc: An open innovation platform for containers

Implement low-level container featuresOperating system level features should be defined in the OCI runtime specificationNew capabilities (PID cgroup controls, checkpoint/restore, seccomp) implemented in

runC

INTEREST

OCI compliance/pluggable execution engineImplement a OS/environment for containers via an OCI spec compliant binaryExamples: runz (Solaris zones), runv (hypervisor-based), Intel Clear Containers

Iterative container configuration test/debugSimple variant of “Docker-like” containers with less friction for quick modificationsLow bar for dependencies: single binary + physical rootfs bundle + JSON config

INTEREST

INTEREST

containerdA daemon to control runC

built for performance and density

http://containerd.tools/

containerd

Docker 1.11

Docker for Mac

Docker for Mac architecture (simplified)

Hypervisor Framework

vmnet Framework

Docker Container Engine

Hypervisor

Linux

VPN

Data Service

Interface

Client Libraries

Admin GUI

CLI

Security Sandbox

Docker for Mac internals

Unikernels

http://unikernel.org/

Hypervisor Framework

vmnet Framework

Docker Container Engine

Hyperkit

Linux

VPNKit

DataKit

Client Libraries

Admin GUI

CLI

Security Sandbox

Improving Docker with unikernel tech

InfraKit

Problem:

Managing Docker on different infrastructure is difficult and not portable.

Consistent User Experience

44

How do we handle updates to a cluster??

Docker for AWS

EBS ELB

Container Engine

Storage plugin

Infrastructure Management

Network plugin Orchestration

IAM

CloudFormation

EC2VPC

Admin interface

Linux

User Applications / Services

Docker for AWS

EBS ELB

Container Engine

Storage plugin

InfraKit

Network plugin Orchestration

IAM

CloudFormation

EC2VPC

Admin interface

Linux

User Applications / Services

InfraKit

A toolkit for building declarative, self-healing infrastructure.

Declarative

• JSON configuration for desired infrastructure state: • Specification of instances — vm image, instance type, etc. • Group properties — size, logical identifiers, etc.

• Design patterns encourage • encapsulation • composition

• Config is input to all operations — system figures out what to do48

Self-healing

• Composed of a set of active components / processes that • monitor infrastructure state • detect state divergence • take actions

• Continuous monitoring and reconciliation — always on

• No downtime — rolling update49

Toolkit• Primitives for managing collections of resources

• create, scale, destroy • rolling update

• Abstractions & Developer SPI • Group - manages collection of resources • Instance - describes the physical resource • Flavor - extra semantics for handling instances

• A collection of executable, active components — plugins • Initially, Go daemons in the toolkit • Soon, easy management via Docker Plugins (runc)

Architecture

Instance Plugin

• Spec: specification / model of an instance (e.g. vagrant, EC2): • Logical ID, Init, Tags, and attachment • Platform-specific properties

• Methods: • /Instance.Validate • /Instance.Provision • /Instance.Destroy • /Instance.DescribeInstances

• Examples: instance plugins for EC2, Azure VM, Vagrant, … 52

Flavor Plugin

• Gives more context about the group members: • Size, or list of Logical ID’s (e.g. IP addresses for ‘pets’) • Application-specific notions of ‘health’

Is the node not only present but also joined a swarm? • Methods:

• /Flavor.Validate • /Flavor.Prepare • /Flavor.Healthy

• Examples: flavor for Zookeeper members, Docker swarm nodes53

Group Plugin

• Main entry point for user interaction: • Create, describe update, update, destroy • Config JSON is always the input

• Composed of Instance and Flavor — mix and match to manage cattle (fungible) or pets (special)

• Methods: • /Group.Watch • /Group.Unwatch • /Group.Inspect

54

• /Group.DescribeUpdate • /Group.Update • /Group.StopUpdate

• /Group.Destroy

ConfigurationExample config file (zk.conf): Group configuration = Instance + Flavor

{ "Properties": {

/* raw configuration */

} }

{ "groups" : { "my_zookeeper_nodes" : { "Properties" : { "Instance" : { "Plugin": "instance-vagrant", "Properties": { "Box": "bento/ubuntu-16.04" } }, "Flavor" : { "Plugin": "flavor-zookeeper", "Properties": { "type": "member", "IPs": ["192.168.1.200", "192.168.1.201", "192.168.1.202"] } } } } } }

Today

56

• InfraKit is just getting started… only primitives for working with groups like clusters of hosts

• But we have big plans • Improve group management strategies • More resource types — networking, load

balancers, storage…

• A cohesive framework for active management of infrastructure — physical, virtual, or containers

Get Involved

• Help define and implement new and interesting plugins • Instance plugins for different infrastructure providers • Flavor plugins for systems like etcd or mysql clusters • Group controller plugins — metrics-driven auto scaling

and more

• Help define interfaces and implement new infrastructure resource types — load balancers, networks and storage volume provisioners

57

More Info

• Github: https://github.com/docker/infrakit

• A quick tutorial: https://github.com/docker/infrakit/blob/master/docs/tutorial.md

58

Docker Open Source Projects Summary

Extra Slides

Docker on Windows