Embed Size (px)
Citation preview
DevOps with Kubernetes
Agenda
• Kubernetes overview
• Usage and demo
• Architecture
• Kubernetes on AWS with Cloud Formation
Kubernetes Overview
What is Kubernetes?
Quick facts
• System for managing and orchestrating containerized applications in clusters, a.k.a. cluster management software
• Open source, MIT licensed, developed by Google
• Used in GCE, OpenShift, other projects
Kubernetes is
• portable: public, private, hybrid, multi-cloud, written in Go
• extensible: modular, pluggable, hookable, composable
• self-healing: auto-placement, auto-restart, auto-replication, auto-scaling
• scalable and reliable: all components are scalable and clear setup path exists to setup scalable and reliable cluster
• documented: a lot of documentation, training materials, community support
• open source: MIT license, large and active community
With Kubernetes you can
• Orchestrate complex application deployments quickly and predictably
• Scale your applications on the fly
• Seamlessly roll out new features
• Easily setup complex operations scenarios, e.g. rolling update, canary deployments etc
• Optimize use of your hardware by using only the resources you need
• Manage persistent storage
• Automate
Kubernetes solves
• application composition: co-locating helper processes preserving the “one-application-per-container” model,
• mounting storage systems,
• distributing configuration and secrets,
• application health checking,
• replicating application instances,
• horizontal (auto-)scaling,
• naming and discovery,
• load balancing,
• rolling updates,
• resource monitoring,
• log access and ingestion,
• support for introspection and debugging, and
• identity and authorization.
Kubernetes Usage
Kubernetes management
• Kubectl CLI• Independent binaries for multiple platforms (Go)• put config file to $HOME/.kube or set $KUBECONFIG• Automation friendly with multiple output formats: text, json, yaml, jsonpath• Supports proxy into cluster network, container attachment and log retrieval
• REST API• Available at https://<master-ip>• Self-documented, swagger documentation• Supports proxy into cluster network
• Basic Web dashboard• Available at https://<master-ip>/ui• Only some objects are displayed
Kubernetes objects
Primitives
• Namespace
• Node
• Pod
• Service
• Config Map
• Secret
• Volume
• Persistent Volume
Controllers
• Replication Controller
• Deployment
• Job
• Daemon Set
• Ingress
• ...
Kubernetes objects: common
• All objects include metadata with
• Name – unique
• Labels – searchable, selectable
• Annotations – arbitrary additional information
• Spec – object specific description/specification of the object
• Status – object status within the cluster
• Object information may be received in different formats
Kubernetes objects: names and namespaces
• Namespaces are used to separate groups of objects, e.g. by user, team, project etc
• Namespaces are scopes for names; names are unique per type within namespace
• Namespaces may also be a basis for access control separation
• Resource quotas may be associated with namespaces
kubectl get namespaces [ <ns> ... ]
kubectl describe namespaces [ <ns> ... ]
Kubernetes objects: nodes
• Nodes represent a physical or virtual worker machine where kubelet, kube proxy, and docker run
• Kubelet registers a node on the master and maintains keep-alive check
• Nodes may be annotated and labeled to specify workload affinity and constraints
kubectl get nodes [ <nd> ... ]
kubectl describe nodes [ <nd> ... ]
Kubernetes objects: pods
• Pod is a group of containers
• Run on the same node – co-located and co-scheduled
• Shared storage
• Shared localhost network and port space
• Unique IP within a cluster
• Example: app server and log shipper
Kubernetes objects: nodes and pods
Node 1 Node 2
Pod A-1
10.0.0.3
Cnt1
Cnt2
Pod A-2
10.0.0.5
Cnt1
Cnt2
Pod B-1
10.0.0.8
Cnt3
Kubernetes objects: volumes and persistent volumes
• Used to manage persistent storage
• Multiple types supported:
• AWS EBS
• Azure block store
• Git
• NFS
• GlusterFS
• Ceph
• ...
Kubernetes objects: pods and volumes
Pod
Container 1 Container 2
Persistent
VolumeVolume
Volume
Claim
Volume
MountVolume
Mount
Kubernetes objects: config maps and secrets
• Config maps and secrets are used for distribution of configuration information including secrets like password, certificates, keys etc
• Kubelet registers a node on the master and maintains keep-alive check
• Nodes may be annotated and labeled to specify workload affinity and constraints
kubectl get configmaps [ <cm> ... ]
kubectl describe configmaps [ <cm> ... ]
kubectl get secrets [ <sc> ... ]
kubectl describe secrets [ <sc> ... ]
Kubernetes objects: services
• Service is an abstraction that defines a set of pods a policy to access them
• Service is a distributed L3 load balancer
• Single unique IP within a cluster
• Used to expose pods to the world:
• Default
• NodePort
• LoadBalancer
external
port
Kubernetes objects: pods and services abstraction
Cluster
Pod A-1
10.0.0.3
Pod A-2
10.0.0.5
Pod B-1
10.0.0.8
SrvA
10.7.0.1
SrvB
10.7.0.3
Kubernetes objects: pods and services
Node 1 Node 2
Pod A-1
10.0.0.3
Pod A-2
10.0.0.5
Pod B-1
10.0.0.8
SrvA
10.7.0.1
SrvB
10.7.0.3
SrvA
10.7.0.1
SrvB
10.7.0.3
external
port
external
port
iptables iptables
Service Discovery: DNS
DNS
• <service-name>.<namespace-name>.svc.cluster.local
• <service-name>.<namespace-name>
• <service-name> - in the same namespace
• DNS SRV _<port>._<proto>.<service-name> - for port numbere.g. “SRV _http._tcp.nginx”
Kubernetes Object: Controllers
• Deployment
• Daemon Set
• Job
• Ingress
• Replication Controller
• Replication Set
Kubernetes Controller: Job
• Create one or more pods and ensure that specified number of them successfully terminates
• Jobs may be used for operations automation
Kubernetes Controller: Deployment
• Deployment provides declarative updates for Pods and Replica Sets
• Orchestrate updates and rollbacks
• Scale up or down
Kubernetes Addons
• DNS
• UI
• Logging
• Monitoring
Kubernetes Architecture
Kubernetes architecture
Kubernetes architecture: node
• kubelet manages pods, their containers, images, volumes, network etc
• kube-proxy is a simple network proxy and load balancer responsible for reflecting services on the nodes
Kubernetes architecture: node
• kubelet manages pods, their containers, images, volumes, network etc
• kube-proxy is a simple network proxy and load balancer responsible for reflecting services on the nodes. Userspace (legacy) or iptables(modern) modes are supported.
Kubernetes architecture: control plane
• etcd is a reliable watchable storage for all persistent master state
• API Server is a CRUD-y REST server with most/all logic implemented in plugins that serves Kubernetes API.It mainly processes REST operations, validates them, and updates the corresponding objects in etcd.
Kubernetes architecture: control plane
• etcd is a reliable watchable storage for all persistent master state
• API Server is a CRUD-y REST server with most/all logic implemented in plugins that serves Kubernetes API.It mainly processes REST operations, validates them, and updates the corresponding objects in etcd.
Kubernetes architecture: control plane
• Scheduler binds unscheduled pods to nodes
• Control manager performs all other cluster-level functions, e.g. deployments rollout, job control, pod replication control etc
Kubernetes architecture: control plane
• Scheduler binds unscheduled pods to nodes
• Control manager performs all other cluster-level functions, e.g. deployments rollout, job control, pod replication control etc
Kubernetes architecture: control plane
• Kubectl client is CLI to manage K8S cluster
Kubernetes architecture: security
• Authentication and authorization are pluggable. By default – file based, but may be easily switched to external resources (OAuth, authorization service)
• Transport security is based on TLS, key distribution is deployment specific
Kubernetes architecture: security
• Authentication and authorization are pluggable: file based by default, but may be easily switched to external resources (OAuth, authorization service)
• Transport security is based on TLS, key distribution is deployment specific
Example orchestration scenario
1. User creates a new Deploymentobject via REST
2. Controller Manager sees a Deployment object with no Pods and creates Pod objects based on the Deployment object specification
3. Scheduler sees Pod objects not assigned to Nodes and allocates them according to the Nodes load and the Pods specifications
4. Kubelets running on Nodes see Podobjects allocated to their corresponding Nodes and start Pods’containers based on the Pods’specifications
1
23
4
4
Kubernetes Deployment on AWS
Deployment options
• kube-up.sh scriptAvailable in k8s distro and supported by the developers
• Other methods as described in kubernetes documentation
• Other projects and systems based on kubernetes, such as GCE
• EBT AWS CloudFormation template
AWS Cloud Formation K8S Cluster Improvements
• Master is in auto scaling group for auto recovery
• Nodes are in multi-zone auto scaling group for high availability
• Multiple auto scaling groups are supported for nodes
• Simple no-client cluster rollout and teardown
• Support for node EIP auto-assignment
Summary
The good, the bad, and the ugly
Pros
• Multi-platform
• Rich OTB abstractions and functionality
• Extensibility
Cons and problems
• Complex architecture and setup (AWS CF Template solves the problem for AWS)
• Manifest parameterization is outside K8S
Future work
• Simple deployment to other clouds (Azure) and on-prem
• Multi-zone master
• Single-node deployment (reusable master)
• Multi-region, multi-cloud and federated deployment
• Persistent volume management and backup in prod
• Monitoring and log collection in prod
• Integration with Jenkins
• ...
• Use Vault for key and secret storage
• Packaged components: HAProxyingress, glusterfs, elasticsearch, mongo DB, MySql Cluster(?), Galena Cluster(?), WildFly, ActiveMQ, RabbitMQ (?), HippoCMS, Keycloak, OpenAM, Hadoop (?), Rstudio Server, Jupyter, etc
• Web UI
• ...
