Kubernetes IntrodutionProduction-GradeContainerOrchestration

Jan 2017, Peng Xiao

Network Consulting Engineer

Cisco Systems

8

https://research.google.com/pubs/pub35290.html

In The Very Beginning…

Hardware

Application

Operating System

Scale & High Availability

Hardware

App

Operating System

Hardware

App

Operating System

Hardware

App

Operating System

Hardware Virtualization

Infratructure

App

Operating System

App

Operating System

App

Operating System

Machine Machine Machine

Containerized

Infratructure

App

Operating System

App

Operating System

App

Operating System

Machine Machine Machine

Container Runtime Container Runtime Container Runtime

ContainerOrchestration

Infratructure

App

Operating System Operating System Operating System

Machine Machine Machine

Container Runtime Container Runtime Container Runtime

Container Orchestration

App App App App

ContainerOrchestration

Infratructure

Operating System Operating System Operating System

Machine Machine Machine

Container Runtime Container Runtime Container Runtime

Resource Management

Scheduling

Service Management

AppsOrche

stratio

n

ContainerOrchestration

• Schedule containers to physical/virtual machines• Restart containers if they stop• Provide private container network• Scale up and down• Service discovery

ContainerOrchestration War?

Kubernetes

• Greek for “Helmsman”; also the root ofthe word “governor” and “cybernetic”• Orchestrator for containers• Builds on Docker containers

• Also supporting other container technologies• Multi-cloud and bare-metal environments• Inspired and informed by Google’sexperiences and internal systems• 100% Open Source, written in Go.• Release 1.0 21th July 2015

Large-scaleclustermanagementatGooglewithBorg https://research.google.com/pubs/pub43438.html

Velocity

1.0

1.1

1.2

1.3

Tot

al C

omm

its

1.5

Commits Since July 2014

1.4

Kubernetes Architecture

Quick Recap

• Docker• Docker Compose• Docker Swarm

KubernetesArchitecture

Setup Kubernetes Environment

• Minikube• Simplest way to get Kubernetes cluster up and running• Support Microsoft Windows and Mac OSX

• Kubernetes Multi-Node Cluster• Emulates production environment• Good for testing advanced scenarios

• Google Container Engine• Hosted and managed by Google• Powered by Google Compute Engine

Getting Started with Minikube

• Install Oracle VirtualBox for Mac• Install Docker Toolbox for Mac• Install Docker Version Manager• Install the latest version of Minikube for Mac OSX• Download the latest version of kubectl from this link• Run the following commands from the directory where kubectl is downloaded

• chmod +x ./kubectl• sudo mv kubectl /usr/local/bin

• Launch minikube with the following command:• minikube start –wm-driver=virtualbox

• Test minikube installation with the following commands• minikube status• kubectl get cs

kubeadm

master.myco.com# apt-get install -y kubelet kubeadm kubectl kubernetes-cnimaster.myco.com# kubeadm initKubernetes master initialized successfully!You can now join any number of nodes by running the following command:kubeadm join --token 48b69e.b61e2d0dd5c 10.140.0.3

node-01.myco.com# apt-get install -y kubelet kubeadm kubectl kubernetes-cninode-01.myco.com# kubeadm join --token 48b69e.b61e2d0dd5c 10.140.0.3Node join complete.

master.myco.com# kubectl apply -f https://git.io/weave-kubeNetwork setup complete.

KubernetesMaster

Kubernetes Node

Kubernetes Pod

• Groupofoneormorecontainersthatarealwaysco-located,co-scheduled,andruninasharedcontext• Containersinthesamepodhavethesamehostname• Eachpodisisolatedby• ○ ProcessID(PID)namespace• ○ Networknamespace• ○ Interprocess Communication(IPC)namespace• ○ UnixTimeSharing(UTS)namespace

• AlternativetoaVMwithmultipleprocesses

Kubernetes Pod

• ContainerswithinthesamepodcommunicatewitheachotherusingIPC• Containerscanfindeachothervialocalhost• Eachcontainerinheritsthenameofthepod• EachpodhasanIPaddressinaflatsharednetworkingspace• Volumesaresharedbycontainersinapod

Deploying a pod

Services

• AnabstractiontodefinealogicalsetofPodsboundbyapolicybytoaccessthem• Servicesareexposedthroughinternalandexternalendpoints• Servicescanalsopointtonon-KubernetesendpointsthroughaVirtual-IP-Bridge• SupportsTCPandUDP• Interfaceswithkube-proxytomanipulateiptables• Servicecanbeexposedinternalorexternaltothecluster

Service Types

• ClusterIP• Service is reachable only from inside of the cluster

• NodePort• Service is reachable through <NodeIP>:NodePort address.

• LoadBalancer• Service is reachable through an external load balancer mapped to<NodeIP>:NodePort address

Service Discovery -Environmentvariables

• KubernetescreatesDockerLinkcompatibleenvironmentvariablesinallPods• Containerscanusetheenvironmentvariabletotalktotheserviceendpoint

Service Discovery - DNS

• TheDNSserverwatchesKubernetesAPIfornewServices• TheDNSservercreatesasetofDNSrecordsforeachService• Servicescanberesolvedbythenamewithinthesamenamespace• PodsinothernamespacescanaccesstheServicebyaddingthenamespacetotheDNSpath• my-service.my-namespace

Replication Controller

• EnsuresthataPodorhomogeneoussetofPodsarealwaysupandavailable• AlwaysmaintainsdesirednumberofPods• IfthereareexcessPods,theygetkilled• Newpodsarelaunchedwhentheyfail,getdeleted,orterminated

• Creatingareplicationcontrollerwithacountof1ensuresthataPodisalwaysavailable• ReplicationControllerandPodsareassociatedthroughLabels

Scaling Pods with Replication Controller

Replica Set

• ReplicaSetsarethenextgenerationReplicationControllers• Ensuresspecifiednumberofpodsarealwaysrunning• PodsarereplacedbyReplicaSetswhenafailureoccurs• Newpodsareautomaticallyscheduled

• LabelsandSelectorsareusedforassociatingPodswithReplicaSets• UsuallycombinedwithPodswhendefiningthedeployment

Kubernetes Networking

Docker Networking

Kubernetes Networking

• Highly-coupledcontainer-to-container communications• Pod-to-Pod communications• Pod-to-Service communications• External-to-internal communications

Container to Container

• All containers within a pod can reach each other’s port on localhost

Pod to Pod

• Kubernetesimposesthefollowingfundamentalrequirementsonanynetworkingimplementation• allpods (containers) cancommunicatewithallothercontainerswithoutNAT

• allnodescancommunicatewithallcontainers(andvice-versa)withoutNAT

• theIPthatacontainerseesitselfasisthesameIPthatothersseeitas

• Network model• Can be L3 routed• Can be underlayed (cloud)• Can be overlayed (SDN)

Pod to Pod: How?

• OnGCE/GKE• GCEAdvancedRoutes(programthefabric)• “Everythingto10.1.1.0/24,sendtothisVM”

• Plentyofotherways• AWS:RouteTables• Weave• Calico• Flannel• OVS• OpenContrail• CiscoContiv• Others...

Pod to Service

Adoption

~4k Commitsin 1.5

+25% UniqueContributors

Top 0.01% of all Github Projects

3500+ ExternalProjects Based

on K8s

Companies Contributing

Companies Using

Thanks for your time