Kubernetes HAMontreal Kubernetes Meetup

October 12

Hello, my name is Alexandre

@alex_gervais

alexgervais

AppDirect background

- Chef provisioning

- Centos 7

- Multiple deployments

- AWS

- On-premise

- Automation, automation, automation!

- Packer

- Terraform

- CI

- In production since k8s 1.1

- Rolling upgrade every minor release

- Experimenting since early beta

Although it is easy to deploy and make your applications and micro-services highly available within a Kubernetes cluster, Kubernetes masters are not HA in typical

setups.

It requires a little more work, but not that much…

Here’s the 3-step program.

0. Single master

1.etcd clustering

$ curl https://discovery.etcd.io/new?size=3

2. Master election

podmaster and hyperkube

On every master node:

/etc/kubernetes/manifests/podmaster.yamlgcr.io/google_containers/podmaster:1.1

/srv/kubernetes/kube-controller-manager.yamlgcr.io/google_containers/hyperkube:1.4.0

/srv/kubernetes/kube-scheduler.yamlgcr.io/google_containers/hyperkube:1.4.0

On the elected node:

The podmaster will copy kube-controller-manager.yaml and kube-scheduler.yaml to /etc/kubernetes/manifests and kubelet picks them up!

Disclaimer

Since kubernetes 1.2

--leader-elect

--apiserver-count=3

3. API load balancing

🎉

$ kubectl get po --namespace=kube-system -o wide

NAME READY STATUS RESTARTS AGE IP NODEkube-addon-manager-ip-172-31-29-97.ec2.internal 1/1 Running 1 40d 172.31.29.97 ip-172-31-29-97.ec2.internalkube-controller-manager-ip-172-31-29-97.ec2.internal 1/1 Running 1 40d 172.31.29.97 ip-172-31-29-97.ec2.internalkube-dns-v19-5ut0y 3/3 Running 3 40d 10.0.55.2 ip-172-31-51-130.ec2.internalkube-dns-v19-srphp 3/3 Running 0 13d 10.0.50.5 ip-172-31-46-232.ec2.internalkube-dns-v19-tf5u6 3/3 Running 1 33d 10.0.20.3 ip-172-31-29-97.ec2.internalkube-scheduler-ip-172-31-29-97.ec2.internal 1/1 Running 1 40d 172.31.29.97 ip-172-31-29-97.ec2.internalkubernetes-dashboard-v1.1.0-zta4y 1/1 Running 0 40d 10.0.55.5 ip-172-31-51-130.ec2.internalpodmaster-ip-172-31-29-97.ec2.internal 3/3 Running 3 40d 172.31.29.97 ip-172-31-29-97.ec2.internalpodmaster-ip-172-31-52-169.ec2.internal 3/3 Running 6 33d 172.31.52.169 ip-172-31-52-169.ec2.internalpodmaster-ip-172-31-7-176.ec2.internal 3/3 Running 3 40d 172.31.7.176 ip-172-31-7-176.ec2.internal

$ kubectl get ep

NAME ENDPOINTS AGEkubernetes 172.31.29.97:6443,172.31.52.169:6443,172.31.7.176:6443 40d

Cluster-wide upgrades

- Chef(ing)

- Rolling upgrades of existing nodes

- Terraform(ing)

- Replace nodes, one-by-one

- Datadog monitoring

References

- etcd clusteringhttps://coreos.com/etcd/docs/latest/clustering.html

- hyperkubehttps://github.com/kubernetes/kubernetes/tree/master/cluster/images/hyperkube

- Master node deploymentshttps://coreos.com/kubernetes/docs/latest/deploy-master.html

- Kubernetes HA recipehttp://kubernetes.io/docs/admin/high-availability/

AppDirect Shameless Plug