Kubernetes on EGO : Bringing enterprise resource management and scheduling to KubernetesDa Ma (madaxa@cn.ibm.com)Software Architect, IBMOwner of kube-incubator/kube-mesos-framework

Yong Feng (yongfeng@ca.ibm.com)Senior Software Architect, IBM

Why “Kubernetes on EGO”?

Computing, Storage and Network

Application Application

Container RuntimeThe container runtime packages and launches application instance in a sandbox with portable and flexible capability.Docker and rkt are container runtimes.

Workload ManagementThe workload management component manages the life cycle of an application as well as access to the application, including service compose, service discovery, load balanceKubernetes and Marathon are workload managers.

Resource ManagementThe resource management component provides an abstraction of resources (cpu, mem, …) for application and then allocates/provision resources among tenants and applications. Mesos is an open source resource manager. EGO is an IBM enterprise resource manager.

Why Kubernetes on EGO?

1992

PBS/SGE/LSF Res mgr & wld mgr

tightly coupledBatch wld only

Monolithic

2003 2016 future

??Shared state

between fws by Optimistic Offer

Shared state

Two Level Scheduling

Mesos/YARN/EGO

……

Architecture Overview

EGO Master

VEMKD

MLIM

BASE API

LIM PEM

EGO Agent

LIM PEM

EGO Agent

LIM PEM

EGO Agent

UDP SocketTCP Socket

PLUGIN

k8s-apiserver

k8s-controller-manager

k8se-scheduler

kubelet

kube-proxy

resreq alloc1. Get Pods

2. Send resource request to EGO

3. Get allocations from EGO

4. Bind Pods with Host

5. Run Pods by kubelet

1

2 3

4

5

EGO: Enterprise Resource Manager• Hierarchical consumer• Enterprise sharing policies• Smart preemption• Rich resource attributes

and resource requirement language

• Unified management console• Security• Monitor and alert• HA and multiple site• Resource usage analysis

EGO: Hierarchical Consumer

Dept-1

ORG

Dept-2

Dept-n

Team-1

Team-2

Proj-1

Proj-2

Hierarchical resource budget plan

Hierarchical role based access control

EGO: Enterprise Sharing PoliciesTime-windows based resource

plan per resource groupOwnership and one-to-one

lending/borrow policy

EGO: Enterprise Sharing PoliciesDynamic sharing from top

down to leaf consumerHybrid sharing polices

o At T0, A has a demand of 20

A = 20o At T1, B1 has a demand of 20 and reclaims its parent’s 16

A:B1 = 4:16 o At T2, A cancels all workload and becomes idle

B1 = 20o At T3, B2 has a demand of 20 thus reclaims its 12

B1:B2=8:12

/

B1

BS=1

S=3S=1

S=4

(A=4)

(B1=4) (B2=12)

20 slots in total

A

B2

Hybrid Ownership Share ratioSharing by default X xReserve slots from being shared

X X

Plan configured by absolute number

X X

Sibling first borrowing X xbalance checking X X Proportional borrowing X xProportional reclaiming X x

EGO: Enterprise Sharing Policies

Flexible framework of scheduling plugin for

customized sharing policies

EGO: Smart Preemption• Asynchronized resource negotiation protocol

Issue resource request via allocation which allows client to orchestrate multiple services from different tenants; update resource request on the fly; receive resource allocation by event;

• Grace period in resource planContract between resource lender and borrower used to decide how resources will be returned if required

• Candidate resource listAllows the borrower to optimize when making decisions on which resources to return within grace period

EGO: Rich Variety of Resource Attributes and Resource Requirement Language• Various types of resource attributes and ways to define and collect them

Static vs dynamic; integer vs Boolean vs string vs ip vs topology; user defined vs collected by script

• Resource requirement language

select(), order(), affinity(), antiaffinity(), rusage() …

Demo Video

We’re Contributorto the community !!!

Mesos: OpenSource Resource Management• Hierarchy Role (MESOS-6375)• Multi-Role Frameworks (MESOS-1763)• Scheduling (Pending)

oResource RevocationoOversubscription for reservationoQuota Chunkso…..

Kubernetes on Mesos• Sponsor: Tim Hockin (Google)• Champion: David Eads (Redhat)• Owner: Klaus Ma (IBM)• Github: kuberntes-incubator/kube-mesos-framework

Kubernetes on Mesos (kube-mesos-framework)

1. Get Pods

2. Match Pods and Offers

3. Bind Pods with Host

4. Update Pods status

5. Run Pods by kubelet

IBM Spectrum Conductor for Container

Spectrum Conductor with

Spark

Watson / Cognitive

Container Cloud

Session Scheduler

Workflow

Installer(Deploy,

Reconfigure, HA, Scale,

Rolling update)Mesos Agent

K8s executor

pod pod pod container container

containercontainer

Mesos Master

Kubernetes

GUI

Service Discovery

AuthenticationAuthorization

Distributed Key-value

Store

Image registry

Monitor

HPC

App Store

Persistent Volume

Service Load Balance

Trouble-shooting

Network Topology

Community Value IBM Value-add Customer Value

Docker Hub Registry holds a repository of 75000+ Docker images

Lots of application integrated with MesosKubernetes enable micro-service architecture

• Client unique registry available on premises• Security readiness guidance via the Vulnerability Advisor• Build-in applications of popular open source projects and IBM enterprise

products in App Store

Access to the images and application you require to deploy containers that meet your business needs and strategy

Open-source, standardized, lightweight, self sufficient container technology

• Balance workload between on-prem and off-prem• Deployment choice with openPOWER and x86_64

Flexibility to choose on-prem and off-prem or mix for your business

Build, ship, and run standardized containers• Integrated monitoring & logging• Elasticity to grow storage & container needs• Integrated CI/CD flow• Life-cycle management of containers and data volumes

Docker ease of use combined with enterprise-level integrity and confidence

Create a Container Cloud for developers supporting DevOps practices and cloud-native apps. Pre-built app catalog for fast deployment of OSS tools. Reduce developer friction, creating faster time to results

1

Improve Developer Productivity

Fine grain, dynamic allocation of resources maximizes efficiency of Spark instances sharing a common resource pool.

2Increase Resource Utilization

Proven architecture at extreme scale, with enterprise class workload management, monitoring, reporting, and security capabilities.

3Reduce Administration Costs

Mesos

Kubernetes(role = *, bigdata-daemons)

MyriadSlaves

(weight)

SparkSlaves(weight)

App Area (label: app) BigData Area (label: bigdata)role = bigdata-daemon: Reserve resources for HDFS and Yarn/Spark masterrole = bigdata-comute: Reserve resource for Yarn/Spark agents

Spark SessionSchedulerMyriad Masters

Dep 1 Dep 2 Dep 3

ns1+

quota1

ns2+

quota2

ns3+

quota3

Container service

role = *

BigData Service & Applications

(role = bigdata-comute)

ResourceSharing

Hierarchy Consumer

Smart preemption &

Sharing policies

NS/Quota

Network/DNS

Scheduling

Dream ???

Resource Requirement

Spark with kube-mesos

What’s next?• Support Sharing Policies & Smart Preemption:

Revocable resources support (#19529)Scheduling enhancement (# 31068)

• Support Hierarchical Consumer:Namespace/Quota support/integrate (#31069)Multiple roles support

• Kube-DNS integrate with external DNS (# 28453)

• …

Roadmap of kube-mesos-framework (DRAFT)

Nov, 2016 End of 2016 2017

v0.7 release new code base

v0.9 release new features

v0.8 releasek8sm refactor v1.0 release

Production Ready

Thank You !!