BRKSDN-2115

Intro to Containers & Container Networking

Rohit Agarwalla, Senior Technical Leader

Duane DeCapite, Director Product Management and StrategyBRKSDN-2115

BRKSDN-2115 2

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 3

Legal Disclaimer

Many of the products and features described herein remain in varying stages of development and will be offered on a when-and-if-available basis. This roadmap is subject to change at the sole discretion of Cisco, and Cisco will have no liability for delay in the delivery or failure to deliver any of the products or features set forth in this document.

BRKSDN-2115

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

• Container Overview • Container Ecosystems • Container Orchestration Systems• Container Networking • OpenStack and Containers with Demo• Cisco Microservice Platform (Mantl/Shipped) • Containers on Cisco Boxes• Summary /Q&A

Agenda

BRKSDN-2115 4


Container Overview

BRKSDN-2115 5


New Application Architectures

microservices

monolithic apps

Monolithic Apps Cloud Native Appsserver / hypervisor server clusters, containers

dependencies easy upgrade

stateful microservices

pets cattle

traditional dev and ops teams agile devops teams

BRKSDN-2115


Example: Stackanetes

[1] Alex Polvi, CoreOS. Microservices. https://www.youtube.com/watch?v=DPYJxYulxO4&feature=youtu.beBRKSDN-2115


Milestones in Container History2015 CNCF/OCI Linux Foundation Collaborative

Projects

2014 Rocket (rkt) CoreOS reference images/runtime

2013 Docker Tools to build/retrieve images

2011 Cloud Foundry Warden Manage collection of containers

2007 Control Groups Merged into Linux kernel

2004 Solaris Zones – similar to jails but snapshot/clone ZFS

2000 FreeBSD Introduced jails

1979 Unix Developed chroot

source: pivotal.ioBRKSDN-2115


Containers – The Building Block

A container is a sandbox environment layered on top of a host OS that provides:• Isolation – namespaces• Resource Limits – control groups (cgroups)

[1] Intel Containers 101 (van de Ven)source: intel

BRKSDN-2115


Linux Containers• A Linux container lets you run a Linux system

within another Linux system.

• A container is a group of processes on a Linux machine.

• Those processes form an isolated environment.

• Inside the container, it looks like a VM.

• Outside the container, it looks like normal processes running on the machine.

• It looks like a VM, but it is more efficient: Containers = Lightweight Virtualization

Zones

BRKSDN-2115


App Containers enable Microservices"The microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms” 1

5 Architectural Constraints of Microservices2

1. Elastic – be able to scale, up or down, independently of other services in the same application.

2. Resilient – fail without impacting other services in the same application.3. Composable – offer an interface that is uniform and is designed to support service composition4. Minimal, and – only contain highly cohesive entities5. Complete – be functionally complete

“Disruptor: Continuous Delivery with Containerized Microservices” – Adrian Cockcroft

[1] Martin Fowler. Microservices. http://martinfowler.com/articles/microservices.html[2] Jim Bugwadia. http://nirmata.com/2015/02/microservices-five-architectural-constraints/[3] Adrian Cockcroft,. On the State of Microservices, DockerCon Europe, 2014 http://thenewstack.io/dockercon-europe-adrian-cockcroft-on-the-state-of-microservices/

BRKSDN-2115

http://martinfowler.com/articles/microservices.html

http://martinfowler.com/articles/microservices.html

http://nirmata.com/2015/02/microservices-five-architectural-constraints/

http://nirmata.com/2015/02/microservices-five-architectural-constraints/

http://thenewstack.io/dockercon-europe-adrian-cockcroft-on-the-state-of-microservices/


Containers are the Buzz because..

• Load fast

• Cloud Native

• Build once run (almost) anywhere

• Portable (easy to move between clouds)

• Light (share kernel)

• Scalable

• Enable microservicessource: https://hub.docker.com

BRKSDN-2115


Containers are “almost” like little Virtual Machines• Containers have their own network interface (and IP address)

• Can be bridged, routed... just like with Xen, KVM etc.

• Containers have their own file system• For example a Debian host can run Fedora container (and vice-versa)

• Security: Containers are “isolated” from each other• Two containers can't see each other (separate namespaces)

• Resource Control: Containers can have dedicated resources• Soft & hard quotas for RAM, CPU, I/O... (cgroups)

BRKSDN-2115


Containers and Virtual Machines

App A

Bins/Libs

Hypervisor (Type 2)

Host OS

Server

Host OS

Server

App A’

Bins/Libs

Guest OS

App B

Bins/Libs

Guest OS

Bins/Libs Bins/Libs Cont

aine

rCo

ntro

lApp A

App A’

App B

App B’

App C’

App C’

VM

Container

Containers are isolated but share OS and whereappropriate bins/libraries

Guest OS

BRKSDN-2115


Isolation: Container NamespacesPID Namespace

• Processes in a PID namespace don't see processes of the whole system

• Each pid namespace has a PID #1

• pid namespaces are actually nested

• A given process can have multiple PIDs• One in each namespace it belongs to• So you can easily access processes of children

namespace

• Can't see/affect processes in parent/sibling namespace

Host

Container


Container Ecosystems

BRKSDN-2115 16


Container Ecosystem Overview

BRKSDN-2115


Docker Milestones

• Founded in 2010 by Solomon Hykes with contributions from others• Originally developed at PaaS provider dotCloud• Defined first container image standard • Docker was released as open source in March 2013• Moby Dock (mascot and logo) created by community in June 2013• Created first container image repository (Docker Hub)• Partnership with Red Hat for Fedora/RHEL and OpenShift in September 2013• Docker, Inc closes $15M Series B funding in January 2014

BRKSDN-2115


Docker Milestones

• 100M container downloads in December 2014• SocketPlane acquisition (libnetwork) in March 2015• $95M Series D funding in April 2015• 1B container downloads in November 2015• Docker, Inc acquires Unikernal Systems in January 2016• Docker Cloud launched in March 2016

BRKSDN-2115


Docker Enginedocker engine• daemon – directly manages the containers on the host• client – communicates with the docker daemon to control containers• container – LXC or libcontainer (default)

docker --daemon=true

docker

docker

BRKSDN-2115


Docker Images and Containers• Images layered via union file

system – enables multiple layered file systems images to be seen as one image.

kernel /bootfs

Ubuntu base image

add open-ssl

add apache

writeable

container read-only image layers

copy-on-write

https://docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/BRKSDN-2115


CoreOS Milestones• Founded in January 2013

• Brandon Philips, developer at SUSE and Rackspace• Alex Polvi, Mozilla, CloudKick, Rackspace

• Defined new container standard(rkt) in 2014• No daemon• Additional Security• App Container (appc)

• Released CoreOS (MicroOS for containers)

BRKSDN-2115


CoreOS Milestones

Released Tectonic (Kubernetes + CoreOS for Business)

BRKSDN-2115


App Container Spec“App Container (appc) is a well-specified and community developed specification that defines an image format, runtime environment and discovery mechanism for application containers.”

The App Container (appc) spec aims to have the following properties:• Composable - All tools for downloading, installing, and running containers should be well

integrated, but independent and composable.

• Secure - Isolation should be pluggable, and the cryptographic primitives for strong trust, image auditing and application identity should exist from day one.

• Decentralized - Discovery of container images should be simple and facilitate a federated namespace and distributed retrieval. This opens the possibility of alternative protocols, such as BitTorrent, and deployments to private environments without the requirement of a registry.

• Open - The format and runtime should be well-specified and developed by a community. We want independent implementations of tools to be able to run the same container consistently.

source: https://github.com/appc/spec/, coreos.comBRKSDN-2115

https://github.com/appc/spec/

https://github.com/appc/spec/


CoreOS/rkt can also run Docker

CoreOS Host

systemd

docker containers

$ sudo rkt –insecure-skip-verify fetch docker://redis... (docker2aci converts docker image to ACI)sha512-962bae14761e5e1ec121e4d49d010f29

$ sudo rkt run sha512-962bae14761e5e1ec121e4d49d010f29

$ sudo rkt –insecure-skip-verify fetch docker://ubuntu$ sudo rkt run –interactive=true <image ID>

BRKSDN-2115


Additional CoreOS Projects• etcd – sync cluster state, distributed key-value store, lock management, leader election

(Raft). Flannel stores routing in etcd. etcd is used by Kubernetes

• flannel – builds overlay network across machines. Used by Kubernetes.

• fleet – cross-cluster scheduler, combines systemd and etcd into a distributed init

• Tectonic – Kubernetes + CoreOS for business

• Enterprise Registry (powered by Quay.io) – private registry, public and private options

BRKSDN-2115


Open Container Initiative (OCI)

Source: http://www.opencontainers.org

Project to create open industry standards around container formats and runtime

BRKSDN-2115


Containers are started as a child process of runC and can be embedded into various other systems without having to run a Docker daemon

runC is built on libcontainer, the same container

Docker images can be run with runC

Embeddable

Battle Hardened

Docker Compatible

runC: OCI Container RuntimerunC: is a CLI tool for spawning and running containers according to OCP specification

BRKSDN-2115 28


Docker 1.11 based on OCI Runtime

source: https://blog.docker.com/2016/04/docker-engine-1-11-runc/ BRKSDN-2115


OCI Image Spec

source: http://thenewstack.io/open-container-initiative-launches-container-image-format-spec/

Allow Developers to package and sign application containers

Run them in a variety of container engines

Use build tools and execution schemes that best meet their needs

Containers run without modification on rkt, Docker, Kubernetes, Amazon ECS

The project will be based on Docker v2.2 and draws from Core OS’s appc spec

BRKSDN-2115 30


Cloud Native Computing Foundation

xxx

“Will create and drive the adoption of a new set of common container technologies informed by technical merit and end user value, and inspired by Internet-scale computing”

Platinum Sponsors

source: http://thenewstack.io/open-container-initiative-launches-container-image-format-spec/ BRKSDN-2115 31



Organization marks important milestone to advance essential infrastructure

“Cloud Native Computing Foundation Accepts Kubernetes as first hosted project; Technical Oversight Committee”

- KubeCon, March 10, 2016BRKSDN-2115 32



Organization marks important milestone to advance essential infrastructure

“Cloud Native Computing Foundation Accepts Prometheus as second hosted project; Technical Oversight Committee”-San Francisco, May 9, 2016

BRKSDN-2115 33


Intel Clear Containers• Intel Clear Containers use Intel Virtualization Technology (VT-x) instead of

Namespaces for isolation between containers

• Lightweight hypervisor

• Let go of the PC in the Virtual Machine

• Result: Startup time 30 msec

• Supported rkt Stage 1 (0.8)

• www.clearlinux.org

BRKSDN-2115


Windows Server and Hyper-V Containers (not Linux)

BRKSDN-2115


Mesosphere Milestones

• Founded in 2013 by Florian Leibert, Ben Hindman and Tobi Knaup, all with web scale engineering experience from the likes of Twitter and Airbnb.

• 2014 – Headquartered in San Francisco with international operations in Hamburg, Germany

• Released Data Center Operating System (DC/OS) built on top of Mesos, Marathon and Chronos. Docker & Linux container support.

• 2015 – announced Infinity Stack with Cisco for Big Data• 2016 - open sourced DC/OS • 2016 - support for Kubernetes.• 2016 – Microsoft to build Azure Container Service on DC/OS

BRKSDN-2115


Additional Container OSs• Red Hat RHEL 7 Project Atomic (March 2015) – fast transactional updates with rollback,

security (SELinux), Docker support, Kubernetes support, super-privileged containers• Snappy Ubuntu Core (Dec 2014) – fast transactional updates with rollback, security

(AppArmor), Docker support• VMware Photon (April 2015) – support for Docker, rkt and Garden

BRKSDN-2115


Container Orchestration Systems

BRKSDN-2115 38


Container Orchestration Platform

• Cloud native applications consist several (100s-1000s) of containers

• Requires a platform that can provide – • Clustering and Scheduling • Networking, Storage and

Security• Container Management• Image Registry

Infrastructure Provisioning Tool

Nodes (on-prem or cloud)

Logging

Monitoring

Operating System

Networking, Storage and Security

Container Service Deployment

Image RegistryScheduling

BRKSDN-2115


Container Orchestration FrameworksList of a few frameworks -

• Docker Swarm

• Google Kubernetes

• CoreOS Tectonic

• Apache Mesos

• Kontena

• Google Container Engine

• Azure Container Service

• Amazon ECS

Selecting a framework depends on -

• Container run time

• Cloud v/s On-Prem

• Image Registry

• Open Source

• Networking

BRKSDN-2115


Docker Swarm

BRKSDN-2115 41


Docker and Docker Compose• Inside Docker

• Docker Client• Docker Images• Docker Registry• Docker Containers• Docker Hosts

• Docker Compose• Tool for defining and running multi-

container Docker applications• Isolate environments on a single host• Focused on Development and Testing workflows

• Docker Machine• Automate Infrastructure Provisioning (driver

based EC2, Virtualbox etc)

docker run -it ubuntu /bin/bash

BRKSDN-2115


Common Docker commands• docker run -it ubuntu:14.04 /bin/bash

• docker run -d ubuntu /bin/sh -c "while true; do echo hello world; sleep 1; done”

• docker run -d -p 80:5000 training/webapp python app.py

• docker port nostalgic_morse 5000

• docker ps

• docker logs -f nostalgic_morse

• docker inspect nostalgic_morse (JSON document containing useful configuration and status information for the specified container)

• docker start/stop nostalgic_morse

• docker rm nostalgic_morse

• docker images

• docker pull ubuntu

• docker commit -m "Added json gem" -a ”Cisco Live!" 0b2616b0e5a8 ciscolive!/ubuntu:14.04.1

• FROM ubuntu:14.04 MAINTAINER Cisco Live! <[email protected]> RUN apt-get update && apt-get install -y ruby ruby-dev RUN gem install sinatra

• docker build -t ciscolive!/ubuntu:14.04.1 <location of Dockerfile>

• docker rmi ciscolive!/ubuntu:14.04.1

BRKSDN-2115


Docker Networking• Bridge network driver (--driver=bridge)

• None network driver (--driver=none)

• Host network driver (--driver=host)

• Overlay network driver (--driver=overlay) – Multi-Host using VXLAN + Libkv

• Remote drivers - means of supporting drivers over a remote transport

• Uses IPTables for container external access and port mappings

BRKSDN-2115


Docker Networking Internals

eth0iptables : NAT/port-mapping

Docker0

eth0

C1

eth0

C2

iptables : NAT/port-mapping

isolatedbridge

eth0

C3

eth0

C4


docker_gw

eth0C1

eth1

eth0C3

eth1veth pairs veth pairs

veth pairs

Distributed Store (Libkv - Consul, Etcd, Zookeeper)

SerfPopulate Neighbor Table entries

IP address and VXLAN ID allocation(Network, Subnet) – VXLAN ID

ov-net1 ov-net2


docker_gw

eth0C2

eth1

eth0C4

eth1

veth pairs

ov-net1 ov-net2

Bridge Networking

Overlay Networking

ToR

BRKSDN-2115


Docker Networking Commands

• Three default networks $ docker network ls NETWORK ID NAME DRIVER 7fca4eb8c647 bridge bridge 9f904ee27bf5 none null cf03ee007fb4 host host

• docker network ls

• docker network inspect bridge

• docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' web

• docker network connect/disconnect my-bridge-network web

• docker network create -d bridge my-bridge-network

• Docker Network Commands

BRKSDN-2115


What is Docker Swarm ?• Native clustering for Docker

that turns a pool of Docker hosts into a single, virtual host.

• Distributed Store - store metadata for service discovery to register machines and endpoints inside the cluster.

• Discovery Service – provides node discovery in Swarm

• Scheduler – enables container placement choices

Swarm Manager (HA)

Docker Host (s)

Distributed Store (Libkv - Consul, Etcd, Zookeeper)

Docker Host (s) Swarm Host (s)

Discovery Service

Scheduler

CLI

API

Compose

Container

docker run swarm manage docker run -d swarm join --advertise=172.30.0.69:2375 consul://172.30.0.161:8500

BRKSDN-2115


Scheduling - Filters• Container – container placement, or on the availability of images

on a host• Affinity• Dependency• Port

• Node - characteristics of the Docker host or on the configuration of the Docker daemon• Constraints• Health

BRKSDN-2115


Scheduling – Strategies (Rank)• Node (CPU, RAM, # of containers)

• Spread• Bin Pack • Random

BRKSDN-2115


Docker Universal Control Plane, Data Center

BRKSDN-2115


Google Kubernetes

BRKSDN-2115 51


What is Kubernetes (K8S) ?• Open Source system started by Google in 2014 for automating deployment,

operations, and scaling of containerized applications

• Written in Go (aka golang)

• Portable• Public, Private and Hybrid

• Extensible• Modular, Pluggable

• Self-Healing• Auto-scaling, Auto-replication

• Contributed to CNCF in July 2015

Source: http://redmonk.com/fryan/2016/03/14/the-further-evolution-of-kubernetes/

BRKSDN-2115


Kubernetes ComponentsMaster - • Etcd – Distributed Key-Value Watchable

Storage

• API server - Processes REST operations on the cluster, validates them, and updates the corresponding objects in etcd

• Scheduler – Pluggable service that schedules workloads onto nodes

• Controller Manager – Endpoint, Node and Replication Controllers

Node -

• Kubelet – Manages Pods and containers on the nodes

• Kube-Proxy – network proxy and a load balancer

BRKSDN-2115


Kubernetes Deployment

Master(s)

SchedulerController Manager

Etcd

Node(s)

Docker

Kubelet

API

Kube-Proxy

Pod

ContainerAPI

CLI

UI

Pod

Container

BRKSDN-2115


Kubernetes Concepts• Cluster - A cluster is a set of physical or virtual machines and other infrastructure

resources used by Kubernetes to run your applications

• Namespaces - partition resources created by users into a logically named group

• Context - context defines a cluster, user, namespace tuple (all three are optional)

• Node - A node is a physical or virtual machine running Kubernetes, onto which pods can be scheduled

• Pod - A pod is a co-located group of containers and volumes and is the smallest deployable unit

• Replication controller - manage the lifecycle of pods and ensure that a specified number of pods are running at any given time, by creating or killing pods as required

• Service – Defines IP address to access a set of pods and does basic load balancing

• Labels – key/value pair that is attached to a resource, such as a pod

BRKSDN-2115


YAML Config files

apiVersion: v1kind: Pod metadata: name: Front End v1 labels: app: webapp role: frontend version: v1spec: containers: - name: nginx image: nginx ports: - containerPort: 80

Pod

apiVersion: v1kind: ReplicationController metadata: name: Front End v1 spec: replicas: 2 selector: app: webapp role: frontend version: v1 template: metadata: name: Front End v1 labels: app: webapp role: frontend version: v1 spec: containers: - name: nginx image: nginx ports: - containerPort: 80

Replication ControllerapiVersion: v1kind: Servicemetadata: name: Front End spec: selector: app: webapp role: frontend version: v1ports: protocol: TCP port: 443 targetPort: 443

Service

BRKSDN-2115


Kubernetes Service, Pods, Replication Controllers, Labels

Front End ServiceS

app=webapp,role=frontend, version=v1

Front End v1 PodP

app=webapp, role=frontend, version=v1

Front End v1 PodP


Front End v2 PodP


Front End v1 ControllerR

Desired Count = 2

app=webapp,role=frontend,version=v1

Front End v2 ControllerR

Desired Count = 1

app=webapp,role=frontend,version=v2

BRKSDN-2115


Kubernetes Common Commands• kubectl config set-cluster e2e --server=https://1.2.3.4

• kubectl create -f ./file.yml (create a resource – namespace, pod, replication controller, service)

• kubectl run nginx --image=nginx --replicas=5

• kubectl label pods <pod-name> new-label=awesome

• kubectl get namespace, nodes, services, pods, rc

• kubectl describe namespace, nodes, services, pods, rc

• kubectl exec <pod-name> -c <container-name> -- ls

• kubectl logs -f -c ngnix frontend

BRKSDN-2115


Kubernetes NetworkingFundamental requirements -

• All containers can communicate with all other containers without NAT

• All nodes can communicate with all containers (and vice-versa) without NAT

• Enable applications to communicate directly without port forwarding from nodes to containers

Implementation Details -

• Containers within a pod share a common container network namespace

• IP’s are assigned per pod

• Nodes are assigned unique IP subnets

BRKSDN-2115


Kubernetes Networking Internals

Pod

C1--net=container:infra--ipc=container:infra

Communicate using localhost static portsShared namespace – IP address, IPC

Infra10.0.1.3

C2--net=container:infra--ipc=container:infra

Container to Container

10.0.1.0/24Node

10.0.1.4Pod

C C

10.0.1.3Pod

C C10.0.2.0/24

Node

10.0.2.4Pod

C C

10.0.3.0/24Node

10.0.3.3Pod

C C

BRKSDN-2115


Scheduling - Predicates • Mandatory rules to schedule a new pod on the cluster

Predicate Node’s requirement

PodFitsPorts Needs to be able to host the pod without any port conflicts

PodFitsResources Has enough resources to host the pod

NoDiskConflict Has enough space to fit the pod and the volumes linked

MatchNodeSelector Match the selector query parameter defined in the pod’s description

HostName Has the name of the host parameter defined in the pod’s description

BRKSDN-2115


Scheduling - Priorities • Used to find the most suitable node to run the pod out of machines shortlisted

based on predicates

Priorities Node(s) considered as the best(s)

LeastRequestedPriority Calculates the percentage of memory and CPU requested by the pods that are already on the node. The node with the minimum percentage is the best.

BalancedResourceAllocation Nodes that have a similar memory and CPU usage

ServiceSpreadingPriority Prefers the nodes that have different pods using them

EqualPriority Give an equal priority to all the nodes in the cluster

BRKSDN-2115


Mesos

BRKSDN-2115 63


What is Mesos ?• Originally developed at UC Berkley AMPLab as a research project

• An open-source Apache project that provides efficient resource allocation across a cluster

• Provides a layer of abstraction for CPU, Memory, and Storage resources. Essentially acts as the kernel for a datacenter.

• From a resource perspective, • Pools server resources to be centrally managed as a single unit

• From an application perspective,• Dispatches workloads to consume pooled resources

• Described as Data Center Operating System (DCOS)

BRKSDN-2115


How does Mesos work ?

Elastic Search

Mesos

Marathon Chronos Spark Hadoop

Node Node Node Node Node Node

Frameworks

Scheduler

Nodes

BRKSDN-2115


Mesos components• Master

• Manages Mesos Agents• Enables fine-grained sharing of resources

across frameworks in form of resource offers

• Agent (Slave)• Deployed on the nodes that run the tasks

• Distributed Store (zookeeper)• Maintains cluster state and current master

• Frameworks• Scheduler and Executor

Mesos Slave

Marathon Executor

TasksTasks

Mesos Master (HA)

Distributed Store (Zookeeper)

Marathon Scheduler

Mesos Slave

Marathon Executor

TasksTasks

Allocation Module

BRKSDN-2115


Container Networking

BRKSDN-2115 73


Container Networking Challenges• Scale

• Several containers per host/cluster, more IP endpoints

• Speed• Plumbing must be fast to match container lifecycle management

• Network Management• Simple network integration with bare-metal, VMs and containers

• Segmentation• Network tenant isolation

BRKSDN-2115


Container Networking SolutionsFlannel CoreOS

WeaveNet WeaveWorks

OVN VMWare

Contiv Cisco

Calico MetaSwitch Networks

Libnetwork Docker

OpenShift SDN RedHat

Nuage-SDN Nokia

OpenContrail Juniper

Contiv

BRKSDN-2115


Container Network Model (CNM)

Docker Container

Endpoint

Network Sandbox

Green Network

Docker Container

Endpoint

Network Sandbox

Blue Network

• Proposed by Docker to provide networking abstractions/API for container networking

• Sandbox contains configuration of a container's network stack (Linux network namespace)

• An endpoint is container's interface into a network (veth pair)

• A network is collection of endpoints that can communicate with each other (Linux Bridge, VLAN)

• A container can belong to multiple endpoints (and therefore multiple networks)

BRKSDN-2115


CNM - Details• CNM allows for co-existence of multiple drivers,

with a network managed by one driver

• IPAM Driver APIs - Request/Release Pool (local/global), Allocate/Release IP Address

• Network Driver APIs - Network Create/Delete, Endpoint Create/Delete/Join/Leave

• Capabilities - features the remote IPAM driver can express during registration with libnetwork (eg: MAC address)

• Used by Docker tools and other schedulers that runs standard Docker containers e.g. Mesos Docker Containerizer

Docker Daemon

Libnetwork (CNM)

Native Drivers Remote Drivers

BRKSDN-2115


Container Network Interface (CNI)• Proposed by CoreOS as part of appc

specification

• Common interface between container run time and network plugin

• Gives driver freedom to manipulate network namespace

• Network described by JSON config

• Plugins support two commands:- Add Container to Network- Remove Container from Network

Container

Network namespace

Driver plumbing

Kubernetes, Rocket…

Container Network Interface

Plugins

BRKSDN-2115


CNI (Container Network Interface) - Details

• Provides Container Create/Delete events

• Need to provide information of network namespace, container interface name to the driver to plumb networking

• Networking and IPAM (both executables) run using the network configuration file

• Used by Kubernetes, Cloud Foundry, Weave, Calico, Contiv

$ mkdir -p /etc/cni/net.d $ cat >/etc/cni/net.d/10-mynet.conf <<EOF { "name": "mynet", "type": "bridge", "bridge": "cni0", "isGateway": true, "ipMasq": true, "ipam": { "type": "host-local", "subnet": "10.22.0.0/16", "routes": [ { "dst": "0.0.0.0/0" } ] } } EOF

BRKSDN-2115


Contiv

BRKSDN-2115 80


Contiv – Introduction• Container Networking and Storage with Ops Policies

• Provides REST interfaces

• Open Source at https://github.com/contiv

Container Connectivity Policies for networking Variety of connectivity options Works with Kubernetes,

Docker, Mesos, Nomad

Contiv Networking

Policy for volume allocation Snapshots, IOPs rate-limiting,

Garbage Collection, etc. Works with Docker

Contiv Storage

Node Discovery, Inventory Node Life-Cycle Management Complete Stack, managed Works for cloud, optimized for

Bare-Metal

Contiv Cluster

Contiv UI

BRKSDN-2115

https://github.com/contiv

https://github.com/contiv


Contiv Network Components

- Distributed Cluster wide function- Stateless: useful in node failure/restart, upgrade- Implements cluster wide network and policy- Manage Global Resources: IPAM, VLAN/VXLAN Pools

- Container Networking for: . Kubernetes, Mesos, and Swam- Route Distribution using BGP/EVPN- Custom openflow pipeline for host networking

Contiv Master

Contiv Host Agent

- Tools to manipulate Contiv objects- Implements CRUD using REST interface

Contiv CLI/UI

Docker Host

vswitch

TasksC1

Contiv Master (HA)

Docker Host

vswitch

TasksC2

CLI (netctl)/UI

Contiv netplugi

n

Contiv netplugi

n

BRKSDN-2115


Contiv Commands• netctl net create contiv-net --subnet=10.1.1.0/24 --gateway=10.1.1.254 --pkt-

tag=100

• netctl policy create ciscolive!_policy

• netctl policy rule-add ciscolive!_policy 1 -direction=in -protocol=tcp -action=deny

• netctl policy rule-add ciscolive!_policy 2 -direction=in -protocol=tcp -port=80 -action=allow -priority=10

• netctl policy rule-add ciscolive!_policy 3 -direction=in -protocol=tcp -port=443 -action=allow -priority=10

• netctl group create contiv-net web -policy=ciscolive!_policy

• docker run -itd --net web.contiv-net ubuntu bash

BRKSDN-2115


FD.IO

BRKSDN-2115 85


VPP overview• User-space network packet processing stack for commodity hardware

• Uses Intel DPDK open source network I/O technology

• High performance • ~14 Mpps (Millions packets per second) from a single x86-64 CPU core• Processing multiple packets at a time to optimize the use of commodity hardware

resources. • Uses a completely different software architecture compared to the traditional scalar

approach of processing one packet at a time• Built on a packet processing graph

• Rich multi-layer networking functionality ― Layer 2, IPv4, and IPv6 forwarding with large tables,VRFs, multiple types of tunneling etc

• Extensible through the use of plugins for writing new features

BRKSDN-2115


VPP Operation• Grab all available packets (pointers)

from device RX ring buffer (using DPDK)

• Form a frame (vector) comprising packet indices in received order (similar packets sampled)

• Process frames using a directed graph of nodes

• No I-cache thrashing problem

• Mitigates the dependent read latency problem due to big MAC or IP tables

• Reduces stack depth and addresses D-cache misses on stack addresses Source: https://www.metaswitch.com/the-switch/fd.io-takes-over-vpp

BRKSDN-2115


Container Integration

• Netlink Server to ‘intercept’ netlink calls from existing container control planes

• Netlink Server then programs VPP as appropriate

• Container control planes can evolve to interfacing directly with VPP at their own pace

VPP

Cont

rol P

lane

Data

Pla

ne Netlink Server

libne

twor

k

cni

weav

e

calic

o…

flann

el

BRKSDN-2115


Containers and OpenStack

BRKSDN-2115 92


Kolla Mission Statement

Production-ready containers and deployment tools for operating OpenStack clouds

“Kolla provides Docker containers and Ansible playbooks to meet Kolla’s mission”

K LLA

BRKSDN-2115 93


• Services include ceph, mariadb, rabbitmq, memcached, glance, keystone, nova, neutron (ovs & linuxbridge), murano, heat, cinder, swift, ironic

• Ceph backed storage

• Distro choice of CentOS, Oracle Linux, RHEL, Ubuntu

• Deploy the big tent at 100 node scale

• Small runtime dependency footprint of docker-py and docker-engine

• Minimal operational dependencies

Kolla Liberty: Deploy OpenStack Clouds

BRKSDN-2115


• Security enhancements

• Added upgrade action

• Added reconfigure action

• Diagnostics

• Deployment time reduced 80%

• Named Volumes for persistence

• New services: ElasticSearch, Heka, Kibana, Manila, Mistral

Kolla Mitaka: Operate OpenStack Clouds

BRKSDN-2115


Magnum in OpenStackMaking Containers a First Class Resource

Magnum is First

HTTP/1.1 201 Created

Not Re-Implementing Orchestration

SSO Experience for Cloud Users

Multi-Tenant Control and Data Planes

Asynchronous API

Uses OpenStack Orchestration (Heat)

Uses OpenStack Identity (Keystone)

Source: Mid Cycle Meetup

BRKSDN-2115


Magnum: Multi-Tenant CaaS

Heat orchestrates OS image with Docker & k8sBRKSDN-2115


Kuryr: Container Networking with Neutron

Docker Libnetwork driver for OpenStack NeutronBRKSDN-2115


Calico

vRouter in each compute node (No Overlay)

CNI, libnetwork

BRKSDN-2115


Cisco Physical Infrastructure

Network VIM

Linux (RHEL OS), Hyper Visor (KVM), Host Packages, Software Defined Storage

NFVI Scope

NetworkCompute (UCS) Storage Ceph

Uni

fied

Man

agem

ent

with

ass

uran

ce.

UC

SD

API

GUI

Virtual Infrastructure Manager

RHEL OSP

Ass

uran

ce

Cisco NFVI SolutionLeading Industry Partnerships

Performance Acceleration,Enhanced Platform Awareness

Certified by Red HatJoint Engineering

Integrated platform Design and Validation

Legend

Simple Access to Support

Single Point of Contact

VTS VPP OVSor or or

BRKSDN-2115


Mantl/Shipped

BRKSDN-2115 101


What Is Mantl?

Cisco’s Open Source Containerization Platform

An end to end, cloud agnostic, highly extensible, integrated stack for running container workloads and big data.

Including deployment automation, security, and monitoring.

Designed to grow into a platform for applications and data services.

Mantl.io

BRKSDN-2115

http://mantl.io/

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 103BRKSDN-2115


ciscoshipped.io

Interface for cloud native developmentDeveloper/Operations friendlyEasy-to-adopt, easy-to-useBuilds a ‘Developer-Experience / PaaS layer’

Deployed on top of our Mantl Container stack

What Is SHIPPED?

BRKSDN-2115

http://ciscoshipped.io/


Project Shipped

ProjectManagement

ContinuousIntegration

SourceControl

ContinuousDeployment

ApplicationOrchestration

DeveloperTools

ServiceAssurance

JiraRallyGitHub

ConsulInfluxDBElasticsearchApache SparkZoomDataLogStash

IssueManagement

Mesos (Cloud Foundry)ConsulKubernetes (OS)Kafka

Terraform(UCS, CF, OS)

DroneGitLabShipped CICD

GitHubGitLab

Build PacksEclipseVagrantDocker ComposeCloud Foundry/Openshift

JiraRallyGitHub

Collaboration

Cisco Spark

BRKSDN-2115


Containers on Cisco Boxes

BRKSDN-2115 106


Containers on Cisco Boxes (BRKSDN-2116)

BRKSDN-2115


Summary/Q&A

BRKSDN-2115 108


• Container ecosystems and orchestrations are evolving• OCI and CNCF will create common standard• Containers can be deployed as part of OpenStack• Kolla and Magnum are key projects• Visit Cisco booth for more NFVI/Mantl/Contiv demos

Summary/Q&A

BRKSDN-2115


Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.

110BRKSDN-2115

http://ciscolive.com/Online

http://ciscolive.com/us


Continue Your Education• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

BRKSDN-2115


Please join us for the Service Provider Innovation Talk featuring:

Yvette Kanouff | Senior Vice President and General Manager, SP BusinessJoe Cozzolino | Senior Vice President, Cisco Services

Thursday, July 14th, 201611:30 am - 12:30pm, In the Oceanside A room

What to expect from this innovation talk• Insights on market trends and forecasts• Preview of key technologies and capabilities • Innovative demonstrations of the latest and greatest products• Better understanding of how Cisco can help you succeed

Register to attend the session live now or watch the broadcast on cisco.com

BRKSDN-2115 112

Thank you

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSDN-2115 113


Sources Include• BRKDEV-1002, Rosenbloom• BRKDEV-2116, Brockners• Pivotal.io• Intel.com• Redhat.com• Docker.com• Coreos.com• CNCF.io• opencontainers.org• Openstackfoundation.org

BRKSDN-2115