Kubernetes on aws

Kubernetes on awsYousun Jeong

jerryjung@apach.org

Agenda

• Docker

• Kubernetes

• Kubernetes on AWS

• Demo

Docker

Automate the deployment of any application as a lightweight, portable, self-sufficient container

that will run virtually anywherehttps://www.docker.com/what-docker#Why-Should-I-Care-(For-Developers)

Docker For Developers

Docker automates the repetitive tasks of setting up and configuring development environments so that developers

can focus on what matters: building great software.

Docker for Ops

• Develop and deploy bug fixes and new features without roadblocks and scale applications in real time.

Diff VM• Containers have similar resource isolation and

allocation benefits as virtual machines but a different architectural approach allows them to be much more portable and efficient.

Docker Engine

• It uses a shared kernel and independent user mode sessions to provide isolated run-times for multiple applications on the same host.

Docker Container

• It is a packaged application running in an isolated mode on a Docker host.

Docker Client

• It provides a command line interface to control the Docker engine.

Docker Image

• It is an application template ready to be deployed multiple times.

Docker image

http://blog.octo.com/en/docker-registry-first-steps/

Docker Architecture

http://southworks.com/blog/tag/docker/

Docker Compose

• Docker Compose allows you to define your multi-container application with all of its dependencies in a single file, then spin your application up in a single command

Docker Orchestration

https://www.linkedin.com/pulse/docker-redefining-cloud-computing-landscape-akash-goswami

Kubernetes• Open Source Automation Framework for

deploying, managing, and scaling applications.

Why Kubernetes

• Kubernetes provides a common API and self-healing framework which automatically handles machine failures and application deployments, logging, and monitoring.

Kubernetes Concept• Clusters - set of compute, storage, network

resource

• Pods - colocated group of application containers that share volumes and a networking stack

• Replication Controllers - ensure a specific number of pods, manage pods, status updates

• Services - cluster wide service discovery

Architecture

Components

etcd flannel

kubernetes master

kubernetes node

kubernetes node

kubectl• Pod

• Replication controller

• Deployment

• Service

• Volumes

https://kubernetes.io/docs/user-guide/deployments/

Kubernetes on AWS

https://kubernetes.io/docs/getting-started-guides/aws/

Kubernetes on AWSexport AWS_DEFAULT_PROFILE=default export KUBERNETES_PROVIDER=aws; wget -q -O - https://get.k8s.io | bash export AWS_ACCESS_KEY_ID=XXXX export AWS_SECRET_ACCESS_KEY=XXX export KUBERNETES_PROVIDER=aws export KUBE_AWS_ZONE=us-east-1c export NUM_NODES=3 export MASTER_SIZE=t2.medium export NODE_SIZE=m3.large export AWS_S3_REGION=us-east-1 export AWS_S3_BUCKET=XXX export INSTANCE_PREFIX=k8s export NODE_ROOT_DISK_SIZE=10 export MASTER_ROOT_DISK_SIZE=10 export AWS_SSH_KEY=/Users/dummy/.ssh/metatron

kube-up.sh

Amazon Container Service

• Amazon ECS makes it easy to deploy, manage, and scale Docker containers running applications, services, and batch processes.

Amazon Container Servie

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html

Amazon Container Servie

Screenshot

Screenshot

Episode

• Pod restart

• Failed Containers that are restarted by the kubelet are restarted with an exponential back-off delay (10s, 20s, 40s …) capped at five minutes, and is reset after ten minutes of successful execution

PersistentVolume• awsElasticBlockStore

kind: PersistentVolume apiVersion: v1 metadata: name: metatron-pv labels: type: amazonEBS spec: capacity: storage: 5Gi accessModes: - ReadWriteMany awsElasticBlockStore: volumeID: vol-XXXXXXXXXXXXXXXX fsType: ext4

PersistentVolumeClaimkind: PersistentVolumeClaim apiVersion: v1 metadata: name: metatron-pvc labels: type: amazonEBS spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi

Deployment spec: volumes: - name: ebs-volume awsElasticBlockStore: volumeID: vol-XXXXXXXXXXXXXXXX fsType: ext4 containers: - name: jupyter image: jerryjung/jupyter-notebook-gfs:latest volumeMounts: - mountPath: "/home/jupyter/work" name: ebs-volume

Kubernetes meets Hadoop• Persistent Volume Issue by permission

• Shared Nothing Architecture - hadoop

• DaemonSet or StatefulSet

for i in 0 1; do kubectl exec yarn-nm-$i -- sh -c 'hostname'; done yarn-nm-0 yarn-nm-1

DEMO

Q&A

Thank you