What's new in Kubernetes

Google Cloud Platform

What’s new in KubernetesDocker & Bay Area OpenSource meetup

February 16, 2016

Daniel Smith <dbsmith@google.com>Senior Software Engineer

Kubernetes

Greek for “Helmsman”; also the root of the words “governor” and “cybernetic”

• Runs and manages containers

• Inspired and informed by Google’s experiences and internal systems

• Supports multiple cloud and bare-metal environments

• Supports multiple container runtimes

• 100% Open source, written in Go

Manage applications, not machines

Google has been developing and using containers to manage applications forover 10 years.

Images by Connie Zhou

Review: What’s old in Kubernetes?

kubelet

kubelet CLI

users master nodes

The 10000 foot view

kubelet

scheduler

controllers

apiserver

Small group of containers & volumes

Tightly coupled

The atom of scheduling & placement

Shared namespace• share IP address & localhost• share IPC, etc.

Managed lifecycle• bound to a node, restart in place• can die, cannot be reborn with same ID

Example: data puller & web server

ConsumersContent Manager

File Puller

Web Server

Volume

Volumes

Very similar to Docker’s concept

Pod scoped storage

Share the pod’s lifetime & fate

Support many types of volume plugins• Empty dir (and tmpfs)• Host path• Git repository• GCE Persistent Disk• AWS Elastic Block Store• Azure File Storage• iSCSI• Flocker

• NFS• GlusterFS• Ceph File and RBD• Cinder• FibreChannel• Secret, ConfigMap, DownwardAPI• Flex (exec a binary)• ...

ReplicationControllers

A simple control loop

Runs out-of-process wrt API server

Has 1 job: ensure N copies of a pod• if too few, start some• if too many, kill some• grouped by a selector

Cleanly layered on top of the core• all access is by public APIs

Replicated pods are fungible• No implied order or identity

ReplicationController- name = “my-rc”- selector = {“App”: “MyApp”}- podTemplate = { ... }- replicas = 4

API Server

How many?

Start 1 more

How many?

Services

A group of pods that work together• grouped by a selector

Defines access policy• “load balanced” or “headless”

Gets a stable virtual IP and port• sometimes called the service portal• also a DNS name

VIP is managed by kube-proxy• watches all services• updates iptables when backends change

Hides complexity - ideal for non-native apps

Client

Virtual IP

External Services

Services IPs are only available inside the cluster

Need to receive traffic from “the outside world”

Builtin: Service “type”• NodePort: expose on a port on every node• LoadBalancer: provision a cloud load-balancer

DiY load-balancer solutions• socat (for nodePort remapping)• haproxy• nginx

What’s new in Kubernetes?

Ingress (L7)

Services are assumed L3/L4

Lots of apps want HTTP/HTTPS

Ingress maps incoming traffic to backend services

• by HTTP host headers• by HTTP URL paths

HAProxy, NGINX, AWS and GCE implementations in progress

Now with SSL!

Status: BETA in Kubernetes v1.2

URL Map

Client

Service-foo: 10.0.0.1 Service-bar 10.0.0.2

api.company.com24.7.8.9

http://api.company.com/foo http://api.company.com/bar

Ingress API Ingress (L7)

apiVersion: extensions/v1beta1kind: Ingressmetadata: name: testspec: rules: - host: k8s.io http: paths: - path: /foo backend: serviceName: fooSvc servicePort: 80 - path: /bar backend: serviceName: barSvc servicePort: 80

fooSvc barSvc

http://k8s.io/foo http://k8s.io/bar

Ingress (L7)

apiVersion: extensions/v1beta1kind: Ingressmetadata: name: testspec: rules: - host: asdf.io http: paths: - backend: serviceName: qwertySvc servicePort: 80 - host: aoeu.io http: paths: - backend: serviceName: dvorakSvc servicePort: 80

qwertySvc dvorakSvc

http://asdf.io/* http://aoeu.io/*

Ingress (L7)

Ingress Object Ingress Controller● GCE● HAProxy● ...

Ingress (L7)

kube-proxy

iptables kube-proxy

iptables

kube-proxy apiserverNode X

iptables

services & endpoints

iptables kube-proxy

iptables

kubectl run ...

iptables kube-proxy

iptables

schedule

iptables kube-proxy

iptables

kubectl expose ... iptables kube-proxy

iptables

new service!

update

iptables kube-proxy

iptables

configure

iptables kube-proxy

iptables

iptables kube-proxy

iptables

new endpoints!

update

iptables kube-proxy

iptables

configure

iptables kube-proxy

iptables

iptables kube-proxy

iptables

Client

iptables kube-proxy

iptables

Client

iptables kube-proxy

iptables

Client

iptables kube-proxy

iptables

Client

iptables kube-proxy

ConfigMaps (and Secrets)

ConfigMaps

Problem: how to manage app configuration• ...without making overly-brittle container images

12-factor says config comes from the environment

• Kubernetes is the environment

Manage config via the Kubernetes API

Inject config as a virtual volume into your Pods• late-binding, live-updated (atomic)• also available as env vars

Status: GA in Kubernetes v1.2

Pod ConfigMap

Secrets

Problem: how to grant a pod access to a secured something?

• don’t put secrets in the container image!

12-factor says config comes from the environment

• Kubernetes is the environment

Manage secrets via the Kubernetes API

Inject secrets as virtual volumes into your Pods• late-binding, tmpfs - never touches disk• also available as env vars

Pod Secret

Rolling updates

Rolling Updates

ReplicationController- replicas: 3- selector:

- app: MyApp- version: v1

Service- app: MyApp

Rolling Updates

Service- app: MyApp

# Update pods of frontend-v1 using new replication controller data in frontend-v2.json.$ kubectl rolling-update frontend-v1 -f frontend-v2.json

# Update pods of frontend-v1 using JSON data passed into stdin.$ cat frontend-v2.json | kubectl rolling-update frontend-v1 -f -

# Update the pods of frontend-v1 to frontend-v2 by just changing the image, and switching the# name of the replication controller.$ kubectl rolling-update frontend-v1 frontend-v2 --image=image:v2

# Update the pods of frontend by just changing the image, and keeping the old name$ kubectl rolling-update frontend --image=image:v2

Service- app: MyApp Rolling Updates

Deployments

Rolling update is too imperative

Deployment manages RC changes for you• stable object name• updates are done server-side rather than client• kubectl edit or kubectl apply is all you need

Aggregates stats

Can have multiple updates in flight

Status: BETA in Kubernetes v1.2 ...

Deployments

apiVersion: extensions/v1beta1kind: Deploymentmetadata: name: nginx-deploymentspec: replicas: 3 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80

Run-to-completion, as opposed to run-forever• Express parallelism vs. required completions• Workflow: restart on failure• Build/test: don’t restart on failure

Aggregates success/failure counts

Built for batch and big-data work

Start Finish

apiVersion: extensions/v1beta1kind: Jobmetadata: name: ffmpegspec: selector: matchLabels: app: ffmpeg template: metadata: labels: app: ffmpeg spec: containers: - name: ffmpeg image: ffmpeg restartPolicy: OnFailure

Start Finish

apiVersion: extensions/v1beta1kind: Jobmetadata: name: ffmpegspec: selector: matchLabels: app: ffmpeg # run 5 times before done completions: 5

Start Finish

apiVersion: extensions/v1beta1kind: Jobmetadata: name: ffmpegspec: selector: matchLabels: app: ffmpeg # run 5 times before done completions: 5 parallelism: 2...

DaemonSets

Problem: how to run a Pod on every node• or a subset of nodes

Similar to ReplicationController• principle: do one thing, don’t overload

“Which nodes?” is a selector

Use familiar tools and patterns

Status: BETA in Kubernetes v1.2

Graceful Termination

Give pods time to clean up• finish in-flight operations• log state• flush to disk• 30 seconds by default

Catch SIGTERM, cleanup, exit ASAP

Pod status “Terminating”

Declarative: ‘DELETE’ manifests as an object field in the API

HorizontalPodAutoscalers

HorizontalPodAutoScalers

Automatically scale ReplicationControllers to a target utilization

• CPU utilization for now• Probably more later

Operates within user-defined min/max bounds

Set it and forget it

Cluster Auto-Scaling

Cluster Scaling

Add nodes when needed• e.g. CPU usage too high• nodes self-register with API server

Remove nodes when not needed• e.g. CPU usage too low

Status: Works on GCE, need other implementations

New and coming soon

• Cron (scheduled jobs)• Custom metrics• “Apply” a config (even more declarative)• Interactive containers• Bandwidth shaping• Third-party API objects• Scalability: 1000 nodes, 100+ pods/node• Performance• Machine-generated Go clients (less deps!)• Volume usage stats• Multi-zone (AZ) support• Multi-scheduler support• Node affinity and anti-affinity

• Multi-cluster federation• API federation• More volume types• Private Docker registry• External DNS integration• Volume classes and auto-provisioning• Node fencing• DiY Cloud Provider plugins• More container runtimes (e.g. Hyper)• Better auth{n,z}• Network policy (micro-segmentation)• Big data integrations• Device scheduling (e.g. GPUs)

Kubernetes status & plans

Open sourced in June, 2014• v1.0 in July, 2015• v1.1 in November, 2015• v1.2 ... soon!

Google Container Engine (GKE)• hosted Kubernetes - don’t think about cluster setup

PaaSes:• RedHat OpenShift, Deis, Stratos

Distros:• CoreOS Tectonic, Mirantis Murano (OpenStack),RedHat

Atomic, Mesos

Hitting a ~3 month release cadence

The Goal: Read-write open source

Containers are a new way of working

Requires new concepts and new tools

Google has a lot of experience...

...but we are listening to users!

Your input does make a difference!

The Goal: Read-write open source

Kubernetes is Open- open community- open design- open source- open to ideas

http://kubernetes.iohttps://github.com/kubernetes/kubernetes

slack: kubernetestwitter: @kubernetesio

What's new in Kubernetes

Software

SharePoint 2010 - What's New, What's Possible and What's (still) Missing

What's NEW, What's NEXT, What's HOT!

Tech Trends: What's new, What's important in 2013

Kubernetes: What's it do? - Red Hateparis/kubernetes/kube.pdf · Kubernetes: What's it do? Presenter Eric Paris Red Hat. 2 Eric Paris Agenda What is Kubernetes Kubernetes Primitives

Building What's Next - Red Hat...web — with data centers and software designed to power ... Hosting Kubernetes PaaS hosting for Kubernetes Online, Enterprise or Dedicated ... Some

Ephesoft: What's New, What's Different, What's Coming

The New HIPAA Era: What's New, What's Different and What's Actually Important

Kubernetes: What's it do?people.redhat.com/~eparis/kubernetes/kube.pdf · 2014-12-17 · Kubernetes does NOT and will not expose all of the 'features' of the docker command line

Rotary's Websites: What's New and What's Next

Globalization: What's New? What's Not?

Angular 2 - What's new and what's different

Windows 10: What's New? What's Different?

Kubernetes 1.10 What’s New in - Cloud Native Computing ...€¦ · Kubernetes implementation of Container Storage Interface (CSI) beta in Kubernetes v1.10 Introduced as alpha in

What's Old is What's New

Early Warning Systems: What's New? What's Working?

PlantPAx® System What's New & What's Next

2010 777: The Social Business: What's Now, What's New and What's Next?

Crossing the Chasm - What's New, What's Not

Spring Tooling: What's new and what's coming

New Features of Kubernetes v1.2.0 beta