Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
© 2020 Arm Limited (or its affiliates)
Trevor Tao, Song ZhuMar/2020
Empowering Edge Software Stack on Arm with High Performance Container
Networking Technology
2 © 2020 Arm Limited (or its affiliates)
Agenda
• Introduction
• Arm Edge Software Stack Architecture
• High Performance CNIs(Calico, Cilium…) for Arm Edge Software Stack
• SRIOV and SmarNIC support for Container Networking
• Performance Tests
• Future Work(Provisional)
© 2019 Arm Limited
Introduction
4 © 2020 Arm Limited (or its affiliates)
Introduction• The purpose of edge computing and MEC is to bring real-time, high-bandwidth, low-latency access to
latency-dependent applications, distributed at the edge of the network. Arm CPUs are cost effective, consume low power and are customizable; making them a preferred choice for edge cloud service vendors.
• On the other side, a high performance, flexible and easy deployable container networking of edge software stack is the key to the success of using Arm platform on edge cloud since the widely used VNFs are deployed by container-based orchestration engine, such as Kubernetes, OpenShift or others.
https://community.arm.com/developer/ip-products/processors/b/processors-ip-blog/posts/arm-and-telco-systems-partner-to-redefine-universal-cpe
5 © 2020 Arm Limited (or its affiliates)
Arm Edge Reference Stack
• Heterogeneous Architecture• VM, container, bare metal• Servers and customized Edge platforms• Virtualized NFs and Physical NFs• Accelerator interface
• Resource constraints• Kubernetes• SDN Controller for K8s
• HW Accelerations• Integrated accelerators• PCIe/CCIX attached accelerator (Smart NICs…)
5
Network Equipment Switch/GW
Edge Servers or Networking Edge Platform
Acceleration
Smart NICsFPGA/GPU
Integrated Accelerators
Controller
Flannel, Calico
Cilium
Lightweight App orchestration
Real Time Linux distribution
SR-IOV, DPDK
Infra Orchestration and Installer
Kubernetes
Containerized Compass
VPP,OVS
NetworkingSoftware
Apps Apps
Bare metal Container
Linux System Networking
OVN-K8s
Knative Kubeflow EdgeX
eBPF
CNCF Service Mesh
Docker
6 © 2020 Arm Limited (or its affiliates)
Arm Edge Stack Deployment Reference Cluster
Small deployment with Marvell Mcbin DBShot Boards
Arm server samples used in edge cloud
© 2019 Arm Limited
High Performance Container Networking for Arm Edge Software Stack
8 © 2020 Arm Limited (or its affiliates)
High Performance CNIs available for Arm Edge Stack
Things now available in Akraino IEC Arm edge stack as a ref:
8
IEC Arm Edge Stack
Calico Cilium Contiv-VPP OVN-K8s FlannelSRIOV
• pure IP networking fabric
• high-level network policy management by iptables
• Good scalability
• Support direct(non-overlay) and overlay(IPINIP, VxLAN) network connection
• Easy deployment
• Calico-VPP appears
• Linux-Native, API-Aware Networking and Security for Containers
• Linux eBPF based network policy, load balance and security which is believed to be with incredible performance
• L3 networking between hosts
• Good scalability too
• uses FD.io VPP to provide network connectivity between PODs
• Native DPDK interface support for phy NIC
• Native VPP ACL/NAT based network policy and access
• Good performance but with rather complex configuration
• Hard to debug
• OVS/OVN-controller based K8s networking solution
• Rather good performance with OVS inherited
• Use OVN logical switches/routers to connect Pods and for outside access
• No OVS-DPDK support now
• Direct physical interfaces(PF/VFs) support for Pods
• High performance with direct Linux kernel eth driver or DPDK PMD driver
• Usually co-work with other CNIs, such as Flannel, Calico by Multus or other glue CNI
• Need resource description or annotation when do the configuration for CNI and Pod setup
• Widely used and almost easiest deployment for a simple K8s networking
• Linux network bridge for pod connection and overlay based communication for inter-hosts access
• Easy to be integrated into other container networking solution, e.g., Cilium
• No good network policy support
Repo: https://gerrit.akraino.org/r/admin/repos/iec
9 © 2020 Arm Limited (or its affiliates)
Calico on Kubernetes
1. Pure IP networking
fabric
2. No encapsulation
needed when simple L2
connection available
for nodes
3. Easy deployment and
debug
4. Supporting Kubernetes
Network Policy by
iptables
5. Good scalability with
BGP based routing
K8s MASTER Node 10.0.0.2/24
Route Table
BIRD
Confd
Felix
Calico/node
Veth0
Pod A 192.168.1.1
Veth1
Route Item
K8s SLAVE Node 10.0.0.3/24
Route Table
BIRD
Confd
Felix
Calico/node
Kernel
Pod B 192.168.2.1
Veth1
Veth0
Kernel
Route Item
Route Table
192.168.1.1/32 Veth0
192.168.2.1/32 Eth0
10.0.0.0/24 Eth0
Route Table
192.168.2.1/32 Veth0
192.168.1.1/32 Eth0
10.0.0.0/24 Eth0
BGP route information
Src: 192.168.1.1Dest: 192.168.2.1
Src: 192.168.1.1Dest: 192.168.2.1
Src: 192.168.1.1Dest: 192.168.2.1
10 © 2020 Arm Limited (or its affiliates)
Cilium – API Aware Networking
Cilium: API Aware Netowrking & Network Security for Microservices using BPF & XDPCilium Architecture
BPF/XDP load balancing 10x performance over IPVS
API Aware(HTTP, Kafka, gRPC)
© 2019 Arm Limited
SRIOV and SmarNIC support for Container Networking
12 © 2020 Arm Limited (or its affiliates)
12
Kubernetes
CNI
Multus CNI Plugin
Flannel/Calico
Plugin
Flannel/Calico Linux Bridge
eth0
PF: enp8s0f0np0
VF0
SR-IOV CNI/Device Plugin
eth1
SR-IOV CNI/Device Plugin
eth1
VF1
SmartNIC
Pod
optional
Containers
enp9s0 enp9s1
enp9s0
eth1 eth0
Flannel/Calicobridge
eno2SmartNIC PS225
SR-IOVPlugin
Pod1
Pod2Kernel Driver
Kernel Driver
Uses only 1 of them
High Performance Networking with SmartNIC SRIOV Interfaces by SRIOV CNI data:
config.json: |{
"resourceList": [{"resourceName":
"ps225_sriov_netdevice","selectors": {
"vendors": ["14e4"],"devices": ["d800"],"drivers": ["bnxt_en"],"pfNames": ["enp8s0f0np0"]
}}, …
}]
}
Device Resource Description:
VFs need to be created
beforehand
For DPDK device driver(another Intel x710 nic as a sample):{
"resourceName": "intel_sriov_dpdk",
"selectors": {"vendors": ["8086"],"devices": ["154c"],"drivers": ["vfio-pci"],"pfNames": ["enp12s0f1"]
}}
apiVersion: v1kind: ConfigMapmetadata:name: sriovdp-confignamespace: kube-system
13 © 2020 Arm Limited (or its affiliates)
Podoptional
Containers
enp9s0
eth1 eth0
Flannel/Calicobridge
eno2enp9s1
Flannel/Calicobridge
eno2
VF0 VF1Host
Podoptional
Containers
eth1 eth0
enP8p1s0f2np0 VF0
SR-IOVPlugin
SR-IOVPlugin
IP:10.56.2.30/24Ro: 10.56.3.0/24GW: 10.56.2.1
10.56.3.30/24Ro: 10.56.2.0/24GW: 10.56.3.1
NIC
A72 CPUs
10.56.2.1/24 10.56.3.1/24
PF0 PF0
Sample: 2 Pods Connected byStingray PS225VFs in a single host
enP8p1s0f2np0 VF1
SmartNIC PS225
Networking Model
14 © 2020 Arm Limited (or its affiliates)
A Very Raw Thinking on the Possible Model for Container Networking with SmartNIC OVS offload support
14
Container
OVS-Bridge
veth_p
veth
Container
veth_p
veth
vSwitch Offload
PF/VF
NIC
NICNIC
OVS/OVN Control
Ovsdb-server
Ovs-vswitchd
Ovn-k8s-agent
KubernetesKubelet
Kube-ctrl-manager
OVN-KubeOvn-kube-controller
Ovn-k8s-agent-master
Container
vNIC
Container
vNIC
vSwitch Offload
NIC
NICNIC
OVS/OVN Control
Ovsdb-server
Ovs-vswitchd
Ovn-k8s-agent
OVN-KubeOvn-kube-controller
Ovn-k8s-agent-master
veth veth
User Space
Kernel
Model 1: Model 2:
User Space
Kernel
…
KubernetesKubelet
Kube-ctrl-manager
…
veth
OVS Flow
Offload
Partial OVS Flow
Offload
© 2019 Arm Limited
Performance Tests
16 © 2020 Arm Limited (or its affiliates)
13.3Gbps
15.8 Gbps
11.4 Gbps
0
2
4
6
8
10
12
14
16
18
Calico Pod-to-Pod Calico Node-to-Pod OVN-K8s Pod-to-Pod
Performance of Calico and OVN-Kubernetes(tested by iperf3)
In a single Arm server
Performance of Calico and OVN-Kubernetes
17 © 2020 Arm Limited (or its affiliates)
Pod
Containers
enp9s0
eth1 eth0
Flannel/Calicobridge
eno2
SmartNIC Stingray PS225
enp9s1
Flannel/Calicobridge
eno2VF0 VF1
Host
Pod
Containers
eth1 eth0
enP8p1s0f2np0 VF0
SR-IOVPlugin
SR-IOVPlugin
IP:10.56.217.13/24Ro: 10.56.217.0/24GW: 10.56.217.1
NIC
A72 CPUs
PF0 PF0
Perf Test1: 2 Pods Connection withStingray PS225 in the same host
enP8p1s0f2np0 VF1
IP:10.56.217.14/24Ro: 10.56.217.0/24GW: 10.56.217.1
IPerfclient
IPerfserver
Kernel Driver
Kernel Driver
Performance Test with SRIOV CNIContainer Networking by SRIOV Interface for Intra Host Communication
18 © 2020 Arm Limited (or its affiliates)
Pod
Containers
enp9s0
eth1 eth0
Flannel/Calicobridge
eno2
SmartNIC Stingray PS225
VF0Host1
Host2
enP8p1s0f2np0 VF0
SR-IOVPlugin
IP:10.56.217.13/24Ro: 10.56.217.0/24GW: 10.56.217.1
NIC
A72 CPUs
PF0
Perf Test2: 1 Pod Connected with another host via SmartNIC VF
IPerfclient
IPerfserver
Kernel Driver
Thunderx2-02
Thunderx2-04
enp137s0f0
PerformanceTest with SRIOV CNIContainer Networking by SRIOV Interface for Inter Host Communication
19 © 2020 Arm Limited (or its affiliates)
Performance Test Result
4.77
5.535.48 5.46
5.37
5.15
4.2
4.4
4.6
4.8
5
5.2
5.4
5.6
# of parallel threads
Initial Test Result(To be tuned)
1 2 3 4 5 6
Pod Communication Performance with SRIOV CNI in the same host via PS225 VFs
Gb
its/
sec
Note: Here we choose iperf instead of iperf3 due to its multi-threaded implementation
20 © 2020 Arm Limited (or its affiliates)
8.17
8.07
9.26
8.88
9.24 9.21
7.4
7.6
7.8
8
8.2
8.4
8.6
8.8
9
9.2
9.4
# of parallel threads
Axis Title
Perf Test 2 Result
1 2 3 4 5 6
Pod Communication Performance with SRIOV CNI in 2 hosts via PS225 VF(10GConnection)
Gb
its/
sec
Performance Test Result
© 2019 Arm Limited
Future Work(Provisional)
22 © 2020 Arm Limited (or its affiliates)
Future Work(Provisional)
• Performance evaluation and tuning for Calico, Cilium CNI with Pod-2-Pod, Node-2-Pod cases on arm platform, mainly for cross-node. The possible tuning parameters include:➢ RPS/RFS
➢ Linux system netdev parameters
➢ RX/TX queue
➢ MTU
➢ eBPF/XDP
➢ Overlay/N-Overlay
• Service mesh integration with high performance CNIs, such as Cilium/Proxy
• Further DPDK incorporated container networking usage model and performance evaluation
• Further CI for high performance container networking solutions on arm with up-to-date processes
• More thinking and possible actual work on SmartNIC support for container networking with OVS offload
22
© 2020 Arm Limited (or its affiliates)
Thank YouDankeMerci谢谢
ありがとうGracias
Kiitos감사합니다
धन्यवाद
شكًراধন্যবাদתודה