Copyright © 2016 Mirantis, Inc. All rights reserved
www.mirantis.com
Kubernetes SDN Performance and Architecture
Jakub PavlikMarek Celoud
Copyright © 2016 Mirantis, Inc. All rights reserved
Presentation Agenda
1. Overlay vs Non-Overlay2. Calico3. OpenContrail4. Connection/comparison5. Q&A
Copyright © 2016 Mirantis, Inc. All rights reserved
About us
Marek [email protected]@MCeloud
Jakub Pavlí[email protected]@JakubPav
Copyright © 2016 Mirantis, Inc. All rights reserved
Networking in Kubernetes
● Networking in containers used to be an issue● Kubernetes solved the biggest problems of port mapping● Different approaches for different use cases
● Overlay vs. Non-overlay● Multitenancy and security● Performance and scaling
● Multiple plugins similar like OpenStack Neutron
Copyright © 2016 Mirantis, Inc. All rights reserved
Network solutions in Kubernetes
SDNs:● Calico● OpenContrail● Romana● Weave● Contiv● OpenVSwitch● ...
Copyright © 2016 Mirantis, Inc. All rights reserved
Overlay vs. Non-overlay
Common Overlay concerns:● Loose benefit of simplicity● Loose performance● Difficult to maintain and
troubleshoot
Overlay benefits:● Multitenancy, Security,
Micro-segmentation● L2, L3, EVPN, L3VPN
capability● Analytics
From performance perspective not using an overlay, it is still necessary to use an internal bridge to demux the container virtual-ethernet interface pairs.
“The key aspect to consider is operational complexity!” Pedro Marques
Copyright © 2016 Mirantis, Inc. All rights reserved
Test environment
● Run various functional and performance tests● Calico bare metal● OpenContrail bare metal● OpenContrail running on Kubernetes with Calico● OpenContrail and Kubernetes next together● Calico in OpenStack with OpenContrail● OpenContrail Kubernetes in OpenStack with OpenContrail
● 100 nodes with 32GB RAM with 8 CPUs and 2x 10Gb links
Copyright © 2016 Mirantis, Inc. All rights reserved
Calico Overview
● CNI network plugin● BIRD routing daemon● Etcd● Confd● Felix● Pure L3
Copyright © 2016 Mirantis, Inc. All rights reserved
Calico
CalicoPros:
● No overhead● Reduce Complexity● Using standard
protocols
Cons:
● Underlay depended● No L2
Copyright © 2016 Mirantis, Inc. All rights reserved
Calico with k8s
● Using CNI● Calico 0.22.0 version with kubernetes 1.4● Kubernetes Policy for security
Copyright © 2016 Mirantis, Inc. All rights reserved
Production consideration for Calico
● Use separate etcd cluster for Calico● Use at least etcd v3● Disable BGP full mesh peering● Do not run Calico in k8s manifests, but as separated
systemd/docker
Copyright © 2016 Mirantis, Inc. All rights reserved
OpenContrail Overview
● Overlay SDN● Control, config, analytics, database, agent● Multiple encapsulations (MPLSoverGRE/UDP, VXLAN)● Uses (usually) physical gateways
Copyright © 2016 Mirantis, Inc. All rights reserved
OpenContrail overview
OpenContrailPros:
● Underlay agnostic● Advanced networking
features● Uses physical
gateways
Cons:
● Overhead● Complex
Copyright © 2016 Mirantis, Inc. All rights reserved
OpenContrail with s8s
● Network manager which provides bridge between Contrail and k8s
● Using ECMP instead of kube-proxy (iptables) balancing● Networks created based on labels in manifests● Security and Multi-tenancy done by policy● Contrail 3.0.3 supports Kubernetes 1.4
Copyright © 2016 Mirantis, Inc. All rights reserved
Production consideration for OpenContrail
● Separate Cassandra cluster for analytics● Use physical routers as gateways
Copyright © 2016 Mirantis, Inc. All rights reserved
Multi-cloud examples
● Connection Baremetal, VMs, container
● Run k8s on top of OpenStack with same Contrail (VM sub-interfaces)
Copyright © 2016 Mirantis, Inc. All rights reserved
Kubernetes production findings
● build own binaries (Mirantis Downstream) instead of reusing existing docker containers with unknown origin
● use single or high available cluster setup● run ETCD control services in systemd not only in
manifests and docker● cleanup from mixing bash, salt, and unrelated features for
production● manage native SSL cert by Salt or external cert entity● pull images from private docker registry with
authentication