49
Extend and Build on Kubernetes Dr. Stefan Schimanski [email protected] @the1stein Kubernetes Meetup Frankfurt, Mar 10 2017

Extend and build on Kubernetes

Embed Size (px)

Citation preview

Page 1: Extend and build on Kubernetes

Extend and Build onKubernetes

[email protected]

@the1steinKubernetes Meetup Frankfurt,Mar102017

Page 2: Extend and build on Kubernetes

Nune Isabekyan,https://x-team.com/blog/introduction-kubernetes-architecture/

Page 3: Extend and build on Kubernetes

Disclaimer:Kubernetes happens to be able to launch pods.It‘s even quite good atit.

We willnotlaunch pods today.

Page 4: Extend and build on Kubernetes

RestfulhttpAPI

//version

/api/api/v1/pods/api/v1/pods/status

/apis/apis/batch/apis/batch/v2alpha1/apis/batch/v2alpha1/jobs/apis/batch/v2alpha1/cronjobs/apis/batch/v1/apis/batch/v1/jobs

kube-apiserver

kubeletproxy

Node:

$kubectl create -ffoo.yaml

User:

scheduler controllermanager

apiserveringresscontroller

Master:

cloud nativeapps

Pods:

Page 5: Extend and build on Kubernetes

$kube-apiserver--secure-port 0--etcd-servershttp://127.0.0.1:2379--service-cluster-ip-range10.0.0.0/16--storage-backendetcd2--storage-media-typeapplication/json

$ etcd

$ kubectl config set-clusterlocal --server=http://127.0.0.1:8080$kubectl config set-context local --cluster=local$kubectl config use-context local

$ kubectl get namespaces –v=7$ kubectl get namespace default -ojson

$ kubectl annotate namespace default meetup=hello

$curl http://127.0.0.1:8080/api/v1/namespaces/default

$ etcdctl get /--recursive$ etcdctl get /registry/namespaces/default$ etcdctl -oextended get /registry/namespaces/default

Page 6: Extend and build on Kubernetes

$httpGET http://127.0.0.1:8080/api/v1/namespaces/default{

"apiVersion":"v1","kind": "Namespace", "metadata": {

"annotations": { "meetup": “hallo"

}, "creationTimestamp":"2017-03-10T07:51:39Z","name": "default", "resourceVersion": “73",

}, "spec": { "finalizers": ["kubernetes“]},"status": { "phase": "Active“ }

}

$httpGET http://127.0.0.1:8080/api/v1/namespaces/default |jq ".metadata.annotations[\"meetup\"]= \"$(date)\""|httpPUT http://127.0.0.1:8080/api/v1/namespaces/default

HTTP/1.1200OKContent-Length:341Content-Type:application/jsonDate:Fri,10Mar201708:28:01GMT

Page 7: Extend and build on Kubernetes

$while true;dohttpGEThttp://127.0.0.1:8080/api/v1/namespaces/default |

jq ".metadata.annotations[\"meetup\"]=\"$(date)\""|http--check-statusPUThttp://127.0.0.1:8080/api/v1/namespaces/default ||break

done

HTTP/1.1409ConflictContent-Length:310Content-Type:application/jsonDate:Fri,10Mar201708:27:58GMT{

"apiVersion":"v1","code":409,"details":{

"kind":"namespaces","name":"default“

},"kind":"Status","message":"Operationcannot be fulfilled onnamespaces \"default\":the object has been

modified;please apply your changes to the latest version and try again","metadata":{},"reason":"Conflict","status":"Failure“

}

Page 8: Extend and build on Kubernetes

/apis/batch/v2alpha1/jobs

Apigroup VersionResource

HTTPpaths:

{“apiVersion“:“v2alpha1“,“kind“:“Job“,“metadata“:{

“name“:“nightly“},“spec“:{ ...}}

Page 9: Extend and build on Kubernetes

{"apiVersion":"v1","kind":"Status","metadata":{},"code":409,"message":“...","status":"Failure“}/apis/batch/v2alpha1/jobs

{“apiVersion“:“v2alpha1“,“kind“:“Job“,“metadata“:{

“name“:“nightly“},“spec“:{ ...}}

Resource vs.Kind

httppath vs.logical object

Page 10: Extend and build on Kubernetes

/apis/extensions/v1alpha1/jobs

/apis/batch/v2alpha1/jobs

/apis/batch/v1/jobs {“apiVersion“:“v1alpha1“,“kind“:“Job“,“metadata“:{

“name“:“nightly“},“spec“:{ ...}}

{“apiVersion“:“v2alpha1“,“kind“:“Job“,“metadata“:{

“name“:“nightly“},“spec“:{ ...}}

{“apiVersion“:“v1“,“kind“:“Job“,“metadata“:{

“name“:“nightly“},“spec“:{ ...}}

Page 11: Extend and build on Kubernetes

/apis/extensions/v1alpha1/jobs

/apis/batch/v2alpha1/jobs

/apis/batch/v1/jobs {“apiVersion“:“v1alpha1“,“kind“:“Job“,“metadata“:{

“name“:“nightly“},“spec“:{ ...}}

{“apiVersion“:“v2alpha1“,“kind“:“Job“,“metadata“:{

“name“:“nightly“},“spec“:{ ...}}

{“apiVersion“:“v1“,“kind“:“Job“,“metadata“:{

“name“:“nightly“},“spec“:{ ...}}

Page 12: Extend and build on Kubernetes

/apis/extensions/v1alpha1/jobs/nightly

/apis/batch/v2alpha1/jobs/nightly

/apis/batch/v1/jobs/nightly

{“apiVersion“:“v1alpha1“,“kind“:“Job“,“metadata“:{

“name“:“nightly“},“spec“:{...}}

{“apiVersion“:“v2alpha1“,“kind“:“Job“,“metadata“:{

“name“:“nightly“},“spec“:{ ...}}

{“apiVersion“:“v1beta1“,“kind“:“Job“,“metadata“:{

“name“:“nightly“},“spec“:{ ...}}

*Iomitted the namespace in/apis/batch/v1/jobs/namespaces/default/nightly

Page 13: Extend and build on Kubernetes

{“apiVersion“:“v1alpha1“,“kind“:“Job“,“metadata“:{

“name“:“nightly“},“spec“:{ ...}}

{“apiVersion“:“v2alpha1“,“kind“:“Job“,“metadata“:{

“name“:“nightly“},“spec“:{ ...}}

{“apiVersion“:“v1“,“kind“:“Job“,“metadata“:{

“name“:“nightly“},“spec“:{ ...}}

etcd

{“apiVersion“:“v1“,“kind“:“Job“,“metadata“:{

“name“:“nightly“},“spec“:{ ...}}

Protobufor

JSON

storageversion

encoding

Page 14: Extend and build on Kubernetes

Iwant my own kindsand store them inthe apiserver and use kubectl.

Page 15: Extend and build on Kubernetes

Discoveryhow kubectl knows which kinds/resources exist

Page 16: Extend and build on Kubernetes

$http127.0.0.1:8080/apis/{

"groups":[{"name": "batch","preferredVersion": {"groupVersion": "batch/v1", "version": "v1“},"versions": [{"groupVersion": "batch/v1","version":"v1"}]

},...]}

$http127.0.0.1:8080/apis/batch/v1{

"apiVersion":"v1","groupVersion":"batch/v1","kind":"APIResourceList","resources":[{

"kind":"Job","name":"jobs","namespaced":true,"verbs":["create","delete","deletecollection",

"get","list","patch","update","watch“]

},...]}

resource name⇒ /apis/batch/v1/jobs

Page 17: Extend and build on Kubernetes

ThirdPartyResources

Page 18: Extend and build on Kubernetes

ThirdPartyResources

*they are marked as beta inthe API,butbecame beta before we even had alpha.

Page 19: Extend and build on Kubernetes

apiVersion:extensions/v1beta1kind:ThirdPartyResourcemetadata:

name:databases.example.comdescription:"Aspecification of aSQLdatabase.“versions:

- name:v1

$kubectl create –fdatabases-tpr.yaml

Page 20: Extend and build on Kubernetes

apiVersion:extensions/v1beta1kind:ThirdPartyResourcemetadata:

name:databases.example.comdescription:"Aspecification of aSQLdatabase.“versions:

- name:v1

$kubectl create –fdatabases-tpr.yaml $kubectl create –fwordpress-databases.yaml

apiVersion:example.com/v1kind:Databasesmetadata:

name:wordpressspec:

user:wppassword:secretencoding:unicode

Page 21: Extend and build on Kubernetes

apiVersion:extensions/v1beta1kind:ThirdPartyResourcemetadata:

name:database.example.comdescription:"Aspecification of aSQLdatabase.“versions:

- name:v1

$kubectl create –fdatabases-tpr.yaml $kubectl create –fwordpress-database.yaml

apiVersion:example.com/v1kind:Databasemetadata:

name:wordpressspec:

user:wppassword:secretencoding:unicode

Page 22: Extend and build on Kubernetes

$ kubectl get databases –w --no-headerswordpress <none>{"apiVersion":"example.com/v1","kind":"Databases",...wordpress <none>{"apiVersion":"example.com/v1","kind":"Databases",...

Page 23: Extend and build on Kubernetes
Page 24: Extend and build on Kubernetes

https://github.com/kubernetes/features/issues/95

Page 25: Extend and build on Kubernetes

ThirdPartyResources are limited

• no version conversion• no defaulting• no validation• no subresources (scale,status)• no admission• alpha⇒ APImight change

• but: demand is high,expect improvements in1.7+

• Today‘s users of TPRs:https://gist.github.com/philips/a97a143546c87b86b870a82a753db14c

Page 26: Extend and build on Kubernetes

Controllerswhere is the business logic?

Page 27: Extend and build on Kubernetes

$while true;dohttpGEThttp://127.0.0.1:8080/api/v1/namespaces/default |

jq ".metadata.annotations[\"meetup\"]=\"$(date)\""|http--check-statusPUThttp://127.0.0.1:8080/api/v1/namespaces/default ||break

done⟲

Page 28: Extend and build on Kubernetes

$kubectl get namespaces -w --no-headers|

while read NSSTATUSTIME;do#dowhatever you likehere,e.g.change the namespaceecho"$NSchanged“

done⟲

$curl -f'http://127.0.0.1:8080/api/v1/namespaces?watch=true&resourceVersion=4711‘

{"type":"ADDED","object":{"kind":"Namespace","apiVersion":"v1","metadata":{"name ...{"type":“MODIFIED","object":{"kind":"Namespace","apiVersion":"v1","metadata":{"name ...{"type":“DELETED","object":{"kind":"Namespace","apiVersion":"v1","metadata":{"name ...

Page 29: Extend and build on Kubernetes

APIAggregation

Page 30: Extend and build on Kubernetes

Why

• ThirdPartyResources are limited• no version conversion• no defaulting• no validation• no subresources (scale,status)• no admission

• Some things need full powerof Go• Servicecatalog• OpenShift PaaS• other powerfulAPIs

≫ Goal:allow powerfulextensions without modifying Kubernetes itself

Page 31: Extend and build on Kubernetes

Alphainv1.6:k8s.io/apiserver

• generic apiserver library inGo• today used inside

• kube-apiserver• federation apiserver• service catalog

• allows creation of custom apiservers inacouple hundred lines of code

• each custom apiserver is its own process,communicating viaHTTPS• delegates authentication/authorization to kube-apiserver• uses etcd storage (possibly shared with kube)

Page 32: Extend and build on Kubernetes

Nune Isabekyan,https://x-team.com/blog/introduction-kubernetes-architecture/

Page 33: Extend and build on Kubernetes

kube-apiserver service catalog PaaS

kube-aggregator

Page 34: Extend and build on Kubernetes

kube-apiserver service catalog PaaS

kube-aggregator

API API API

API

Pods Jobs ... announcement ... build test project

Page 35: Extend and build on Kubernetes

kube-apiserver kube-apiserver kube-apiserver

federationapiserver

API API API

API

Pods replicasets Pods replicasets Pods replicasets

deployment service

Europe US Asia

Notthis:

federationcontrollers⟲

federated resources:

availability zones +regions:

Page 36: Extend and build on Kubernetes

kube-apiserver service catalog PaaS

kube-aggregator

discovery

discovery

Page 37: Extend and build on Kubernetes

kube-apiserver service catalog PaaS

kube-aggregator

GET

GET/apis/servicecatalog/subscription/database-prod-wordpress

Page 38: Extend and build on Kubernetes

kube-apiserver service catalog PaaS

kube-aggregator

GET/apis/servicecatalog/subscription/database-prod-wordpress

GET

RBAC Namespace

Page 39: Extend and build on Kubernetes

kube-apiserver service catalog PaaS

kube-aggregator

GET

Vision:$ helm install service-catalog$ kubectl create service-announcement ....

GET/apis/servicecatalog/subscription/database-prod-wordpress

Page 40: Extend and build on Kubernetes
Page 41: Extend and build on Kubernetes

Status

• willbe part of Kubernetes 1.6as analpha• https://github.com/kubernetes/sample-apiserver• potentially kube-aggregator integrated into kube-apiserver in1.7

Page 42: Extend and build on Kubernetes

Links

• https://docs.google.com/document/d/1y16jKL2hMjQO0trYBJJSczPAWj8vAgNFrdTZeCincmI/Two Ways to Extend the K8sAPI - Addresources to aKubernetes APIwith TPRor AA• https://github.com/kubernetes/community/blob/master/contributors/design-proposals/aggregated-api-servers.md• https://gist.github.com/philips/a97a143546c87b86b870a82a753db14c - Kubernetes Third-PartyResource Users

Page 43: Extend and build on Kubernetes

https://github.com/kubernetes/community/blob/master/contributors/devel/client-libraries.md

Page 44: Extend and build on Kubernetes

Backup

Page 45: Extend and build on Kubernetes

RestfulhttpAPI

//version

/api/api/v1/pods/api/v1/pods/status

/apis/apis/batch/apis/batch/v2alpha1/apis/batch/v2alpha1/jobs/apis/batch/v2alpha1/cronjobs/apis/batch/v1beta1/apis/batch/v1beta1/jobs

/apis/batch/v2alpha1/jobs

GroupVersionResourceHTTPpaths:

InGo: gvk :=schema.GroupVersionKind{Group:“batch“,Version:“v2alpha1“,Kind:“Job“}obj :=api.Scheme.New(gvk)codec :=api.Codecs.LegacyCodec(gvk.GroupVersion())codec.Decode(reqBody,gvk,obj)

typeJob struct {metav1.TypeMetametav1.ObjectMetaSpec JobSpecStatusJobStatus

}pkg/apis/batch/v2alpha1/types.go

typeTypeMeta struct {KindstringAPIVersion string

}

typeObjectMeta struct {Namestring...

}

Page 46: Extend and build on Kubernetes

RestfulhttpAPI

//version

/api/api/v1/pods/api/v1/pods/status

/apis/apis/batch/apis/batch/v2alpha1/apis/batch/v2alpha1/jobs/apis/batch/v2alpha1/cronjobs/apis/batch/v1beta1/apis/batch/v1beta1/jobs

MaxInFlightLimit

Timeo

utForNon

LongRu

nningReq

uests

PanicRe

covery

CORS

Authen

tication

Audit

Impe

rson

ation

Authoriza

tion

k8s.io/apiserver/pkg/server.DefaultBuildHandlerChain

„Filters“

k8s.io/apiserver/pkg/server/routes/index.go – /k8s.io/apiserver/pkg/server/routes/version.go – /versionk8s.io/apiserver/pkg/server/routes/swagger.go – /swaggerapik8s.io/apiserver/pkg/server/routes/openapi.go – /swagger.json

„Routes“

mux

k8s.io/apiserver/pkg/endpoints.APIGroupVersion.InstallREST

AddSupportedResourcesWebService – /apis/batch/v2alpha1

k8s.io/apiserver/pkg/endpoints.APIInstaller.Install

/apis/batch/v2alpha1/jobs/apis/batch/v2alpha1/cronjobs...

With

Requ

estIn

fo

ctx.RequestInfo

Page 47: Extend and build on Kubernetes

RestfulhttpAPI

mux

k8s.io/apiserver/pkg/endpoints.APIGroupVersion.InstallREST

AddSupportedResourcesWebService – /apis/batch/v2alpha1

k8s.io/apiserver/pkg/endpoints.APIInstaller.Install/apis/batch/v2alpha1/jobs/apis/batch/v2alpha1/cronjobs...

pkg/apis/batchtype Jobsstruct

pkg/apis/batch/v2alpha1type Jobsstruct

api.Schemek8s.io/apiserver

pkg/api api.Scheme.Convert(&internalJob,&v2alohaJob)

/apis/batch/v2alpha1/jobsGETPUTPOSTDELETE...

/status/scale/proxy...

subresources

Page 48: Extend and build on Kubernetes

RestfulhttpAPI

mux

pkg/apis/batchtype Jobsstruct

pkg/apis/batch/v2alpha1type Jobsstruct

api.Scheme api.Scheme.Convert(&job,&v1job)

POST/apis/batch/v2alpha1/jobs

k8s.io/apiserverpkg/endpoints/handlers.CreateNamedResource

binaryJSON

payload

Gostructv2alpha1.Job

HTTPRequest

Gostructinternal.Job

Storek8s.io/apiserver

pkg/registry/generic

Storagek8s.io/apiserverpkg/storage/etcd3

ProtoBufJob

Gostructv2alpha1.Job

etcd

Page 49: Extend and build on Kubernetes

type Scheme struct• AddKnownTypes(gv, obj Object) • Default(src Object)• Copy(src Object) Object• Convert(in, out interface{})• New(gvk) Object

ApiGrouppkg/apis/batchpkg/apis/batch/v1pkg/apis/batch/v2alpha1pkg/apis/batch/register.gopgk/apis/batch/install

GroupVersionKindResource

type Object interface• GetObjectKind() string

client-go/pkg/api.Schemeclient-go/pkg/api.Codecs

Discoverytype APIGroupList structtype APIVersions structtype APIResourceList struct

GroupVersionKind „gvk“GroupVersionResource

Unversioned typesUnstructuredList

Registry / Storagetype Storage interfacetype Lister interfacetype Updater interfacetype Getter interfacetype Deleter interface....

deepcopy-genconversion-gendefaulting-gen

Code Generation

type OwnerReference structtype ObjectReference structtype TypeMeta structtype ObjectMeta struct

Meta

api.Schemeapi.Codecsapi.Registryapi.GroupFactoryRegistry

Globals