©2016 Apigee Corp. All Rights Reserved.

Kubernetes: Managing Microservices at Scale

APPRENDA.COM

Isaac “Ike” Arias

Sr. Director

Worldwide Solutions Engineering

Apprenda

ike@apprenda.com

@iketweets

APPRENDA.COM

A (short) story about containers...

APPRENDA.COM

chroot(1979)

BSD Jails(2000)

Solaris Zones(2004)

Linux CGroups(2006)

LXC(2008)

Docker(2013)

rkt(2014)

LXD(2014)

OS-level Virtualization Is Not New

1980 1990 2000 2010

APPRENDA.COM

Process

Process

Process

Process

Process

Process

Net I/O

CPUM

em

Mount

PIDNet

IPCUT

SUs

er

Process

OS Containers = Isolation + Resource Management

APPRENDA.COM

Process

Linux Container Technologies

Namespaces

CGroups

AppA

rmor SELinux

APPRENDA.COM

Why Containers?PERFORMANCE • SIZE • REPEATABILITY • ISOLATION • QUALITY OF SERVICE • ACCOUNTING • VISIBILITY • PORTABILITY

Most agile and efficient way of managing applications

Server 1

VM1OS

VM2OS

VM3OS

VirtualizationServer 2

OS

Containers

APPRENDA.COM

Need more support for microservices than containers alone

Security

Container Hosting

Code Quality

Peer Discover Configuration

Changes, Supervision and Monitoring

Rolling Deployment Lib NetworkOrchestration

DockerDEVELOPMENT

PRODUCTION

APPRENDA.COM

Google Has Very Extensive Experience With Containers

•Google starts over 2,000,000,000 containers per week•15+ years of container and orchestration

R&D (Borg, Omega)• Many, many lessons learned• Distributed systems DNA:

– cgroups– mapreduce– Bigtable

"Everything at Google runs in a container… Everything"

Joe Beda, Kubernetes Founder

APPRENDA.COM

Enter Kubernetes (GA in 2015) (kubernētēs)

• Greek for “Helmsman”; also the root of the word “Governor” and “Cybernetics”

• Container orchestrator (cgroups/rkt/docker)• Supports all cloud and bare-metal

environments• Inspired and informed by Google’s experiences

and internal systems (mostly Borg)• Open source, written in Go• Declarative vs. Procedural

Manage applications, not machines!

APPRENDA.COM

PodVolume Data

Loader

Front End

Container

PodsSet of Related Containers

Tightly Coupled

Share Namespace

Ephemeral/Stateless

Share Network

Unit of Management

APPRENDA.COM

Pod Networking

DC

BA

Routable(layer 3)

No NAT(internode)

APPRENDA.COM

Labels

DC

BAapp = portal

tier = frontend

version = v1

app = portal

tier = backend

version = v1

app = portal

tier = frontend

version = v2

app = portal

tier = backend

version = v2

APPRENDA.COM

Label Selectors

DC

BAapp = portal

tier = frontend

version = v1

app = portal

tier = backend

version = v1

app = portal

tier = frontend

version = v2

app = portal

tier = backend

version = v2

app == portalSelector

APPRENDA.COM

Label Selectors

DC

BAapp = portal

tier = frontend

version = v1

app = portal

tier = backend

version = v1

app = portal

tier = frontend

version = v2

app = portal

tier = backend

version = v2

app == portaltier == frontend

APPRENDA.COM

Label Selectors

DC

BAapp = portal

tier = frontend

version = v1

app = portal

tier = backend

version = v1

app = portal

tier = frontend

version = v2

app = portal

tier = backend

version = v2

app == portalversion == v1

APPRENDA.COM

Controllers

Desired State

Current State

ActCompare

APPRENDA.COM

Replica Set (Controller)

Node 1

A

Node 2

A

Node 3

A

Node 4

A

Replicas = 4

Replicas = 4

Desired

Current

APPRENDA.COM

Replica Set (Controller)

Node 1

A

Node 2

A

Node 3

A

Node 4

A

Replicas = 4

Replicas = 3

Desired

Current

APPRENDA.COM

Replica Set (Controller)

Node 1

A

Node 2

A

Node 3

A

Node 4

A

Replicas = 4

Replicas = 4

Desired

Current

A

APPRENDA.COM

(µ)Services

app == portaltier == frontend

version == v1

Selector

A A A

Clients

SvcA

Stable Cluster-wide IP

Cluster-wide DNS name

Layer 3 Load Balancer

Updates continuously

VIP

APPRENDA.COM

(µ)Services

A A A

Clients

SvcA

A

app == portaltier == frontend

version == v1

Selector

APPRENDA.COM

• Most active GitHub project out of

3.6M

• 7K professionals list Kubernetes on

their LinkedIn profiles

• Largest number of vendors and

providers, hedges against

vendor lock-in

Kubernetes has Unprecedented Community Stats

GITHUB

36,000+COMMITS

160+RELEASES

900+CONTRIBUTORS

Top 100FORKED GITHUB

PROJECT

Top 2STARRED

GO PROJECT

Top 0.01%STARRED GITHUB

PROJECT

APPRENDA.COM

APPRENDA + GOOGLE CLOUD Moving customers to Google Container Engine

APPRENDA.COM

Who is Apprenda?

• Apprenda is cloud application platform software

• 10 years of experience in distributed systems, cloud native computing and container based application platforms

• Have been part of Kubernetes community offering solutions, including support and services, since March ‘16

APPRENDA.COM

Looked into comparable cloud native solutions:

TechnologyGoogle knows scale & it is

evident in Kubernetes

CommunityTruly open and welcoming to

all members

MarketGrowing exponentially as

orgs build better SW

APPRENDA.COM

Why Our Customers Want GKE Multicloud?

APPRENDA.COM

Customer Case StudyKubernetes on AWS and Google Container Engine Multi-cloud

• AWS Primary Cloud

• Originally Running K8S on EC2 and S3

• Migrated to GKE multi-cloud

• 60% decrease in cluster management cost

• 70% increase in developer productivity

KEY HIGHLIGHTS

• Production Kubernetes• Increased Dependence

on Mobile• Disrupted Market

APPRENDA.COM

Several Operations & Workflow Challenges

● No DevOps available to manage the cluster, web agency doing guesswork

● Teams, CI/CD spun up K8S clusters daily for development

● AWS Specific Challenges:

○ Everything under single VPC (RDS, Elasticache)

○ K8s Scripts had unintended consequences

○ EC2 Volumes filling up because of ElasticSearch setup

○ PVC leak bug with v1.2 & AWS left orphan volumes

● Sub-optimal monitoring of these temporary environments was hampering rapid

development

APPRENDA.COM

Why GKE?

● Focus on code instead of managing K8S components (etcd, master, etc.)

● Supported ancillary cloud services for K8S (storage, network, etc.)

● Closely tracks latest version● High availability architecture not left up

to user● High cpu and net performance● Easier to manage volumes,

memory/CPU utilization and find orphaned resources

APPRENDA.COM

Kubernetes is the Kernel of Container Management

Kubernetes MarketplaceInfrastructure Automation

Logging

Configuration Management

External Services

App Telemetry

Notification Services

Installation, maintenance, and

management

Storage

Middleware

Databases

Container Lifecycle Management

Policy Control

APPRENDA.COM

Key Point: Kubernetes Makes Multiple Environments Easy

Database 1 Database 2 Database 3 Database 4

App 1 (.NET) App 1 (.NET)

App 1 (.NET)

App 2 (Java)

App 2 (Java)

App 3 (Docker)

App 4 (Docker)App 3 (Docker)

App 4 (Docker)

App 4 (Docker)

PUBLIC INFRASTRUCTURE

App 3 (Docker)App 4 (Docker)

App 4 (Docker)Database 3

Database 4

DATA CENTER

App 1 (.NET)

Database 1 Database 2

App 2 (Java)

APPRENDA.COM

Key Takeaways

Lessons learned:

● Managing production Kubernetes “by hand” can be hard, GKE = “Easy Button”

● GKE has easier learning curve (compare to AWS sub-systems)

● Multi-cloud environments are easier when using “pure” Kubernetes abstractions

● Leverage expert distros/providers, support and services

APPRENDA.COM