11
OpenStack Powered by Tungsten FabricSukhdev KapurDistinguished Engineer, Juniper Networks
Krzysztof KajkowskiDirector of Engineering, CodiLime
Open Infrastructure Summit, Shanghai, November 2019
2
Tungsten Fabric Architecture Overview
Physical IP Fabric(no changes)
TF CONTROLLER
Host O/SvRouter
Network / Storage Orchestration
(Config, Control, Analytics, CSN)
(Windows, Linux ….) on BMS
TOR
Compute Orchestration
Virtual Network Blue Virtual Network Red
FW
Logical View
BGP
BGP XMPPNETCONF
Host O/SvRouter
… ……
DC Computes CPE Devices Public Cloud VM
Dis
trib
ute
d P
olic
y E
nfo
rcem
ent
Cen
tral
ized
Po
licy
Defi
nit
ion
ORCHESTRATOR / APPS
Internet / WAN or Legacy Env.
Gateway
…
2
3
vRouter Architecture Overview
vRouter Agent● Exchanging control state such as routes with the Control nodes using
XMPP.● Receiving low-level configuration state such as routing instances and
forwarding policy from the Control nodes using XMPP● Reporting analytics state such as logs, statistics, and events to the
analytics nodes.● Installing forwarding state into the forwarding plane● Discovering the existence and attributes of VMs in cooperation with
the Nova agent.● Applying forwarding policy for the first packet of each new flow and
installing a flow entry in the flow table of the forwarding plane.● Proxying DHCP, ARP, DNS
vRouter Kernel/DPDK● Encapsulating packets sent from the overlay network and
de-capsulating packets received for the overlay network.● Packets received from the overlay network are assigned to a routing
instance based on the MPLS label or Virtual Network Identifier (VNI).● Doing a lookup of the destination address of the in the Forwarding
Information Base (FIB) and forwarding the packet to the correct destination. The routes may be layer-3 IP prefixes or layer-2 MAC addresses.
● Doing RPF check before sending Virtual machine traffic to destination. This is configurable.
Host Compute
User space
Kernel space
vRouter Kernel
Virtual Machine
(Tenant A)
Virtual Machine
(Tenant B)
XMPP
Control Node
pkt0 tap-abc tap-xyz
vRouter Agent
NETLINK
vhost0
Routing Instance
Routing Instance
Config VRFs Policy Table
ethX OR bondX
3
4
vRouter Deployment ModelsKERNEL vROUTER DPDK vROUTER
SRIOV/ vROUTER COEXISTENCE SMARTNIC vROUTER
4
● vRouter runs as a user space process and uses DPDK for fast path Packet I/O.
● Full set of SDN Capabilities Supported
● Requires the VMs to have DPDK enabled for performance benefits
● vRouter fwding plane runs within the NIC
● Workloads are SRIOV-connected to the NIC
● Some workloads can directly SR-IOV into the NIC, while others go through the vRouter
● Sometimes a VNF can have multiple interfaces some of which are SRIOV-ed to the NIC
● Interfaces that are SRIOV-ed into NIC don’t get the benefits / features of vRouter
● This the normal operation where fwding plane of vRouter runs in the kernel and are connected to VMs using TAP interface (or veth pair for containers)
● vRouter itself is enhanced using other performance related features:○ TSO / LRO○ Multi-Q Virtio
VM1vRouterAgent
VM 2...VM1vRouterAgent
VNF 2...
VM1 VM...vRouterAgent
... VM1 VM...vRouterAgent
...
5
Distributed Networking for VMs, PODs, & BMS
5
KubernetesCNI
Neutron/CNI/DM/FabricSDN Controller
Edge/MC-GW
OpenStackNeutron
Edge/POP Site
Edge/POP Site
Edge/POP Site
Basic Networking:L2/L3 or L2/L3 NetworkIPAM/DHCP, DNS, Multi-Tenancy
Advanced Networking:VLAN-ID, VRRP, VIP, LB, Routes Advertisement,GW Function, Service Chaining, Traffic Steering, Flow awareness,QoS, SR-IOV/DPDK, BGP-VPN,Inter Site Federation, Health Checks, FW, IPSec/TLS Support
BMS & FabricManager
On-Prem:● Core Site● Core Distributed Site● Edge Site
6
Policy Framework
Old B e h a v i o r New B e h a v i o r
Can we use one policy to be applied in all the different deployments?
…
Web App db
App1, Deployment = Prod
Network Policy = P3
1. Reduced Complexity2. Simplified Management3. Improved Scalability
Web App db
App1, Deployment = Staging
Network Policy = P2
Web App db
App1, Deployment = Dev
Network Policy = P1
…
Web App db
App1, Deployment = Dev
Web App db
App1, Deployment = Staging
Web App db
App1, Deployment = Prod
Policy = P
6
7
Policy Framework
Web App db
App1, Deployment = Dev-AWS
…
Web App db
App1, Deployment = Dev
Web App db
App1, Deployment = Staging
Web App db
App1, Deployment = Prod
Reuse of policies across multiple clouds and with multiple orchestrators
Web App db
App1, Deployment = Dev-K8s
Web App db
App1, Deployment = Dev-Mesos
Reuse policy
Web App db
App1, Deployment = Staging-BMS
B a r e M e t a l S e r v e r s
Reuse policy
Reuse policy
Reuse policy
Policy = P
Define/Review/Approve Once → Use Everywhere
7
8
Policy Framework – Use Case Example
site = US site = EMEA
Web App
App = Finance, Deployment = Dev
Web App
App = Finance, Deployment = Prod
Web App
App = Finance, Deployment = Dev
Web App
App = Finance, Deployment = Staging
match deploymentallow https-traffic tier=web > tier=app1
allow mysql-traffic tier=app > tier=db match site2
Dev
Pro
duct
ion
Sta
ging
Legacy Data (tier =
db)
&& siteE
n f
o r c
e m
e n
tD
e f
n
Legacy Data (tier = db)
8
9
Tungsten Fabric Deployment Models with Openstack
● Two Deployment models
○ Monolithic Plugin
○ ML2 based - this is used in the demo
Neutron
ML2 Plugin
MechanismDriver
VLAN GRE VxLAN Flat
Op
envS
wit
ch
TypeDriver
Op
enD
aylig
ht
Ari
sta
Cis
co N
exu
s
Net
wor
kin
g
Op
en C
ontr
ail
9
10
● Running Tungsten Fabric SDN along with other ML2 drivers
● This facilitates:○ Running OVS, SR-IOV and vRouter based works simultaneously
○ Running OVS and SR-IOV workloads and have Tungsten Fabric manage the fabric
○ Live migration of OVS based computes to vRouter based computes
https://opendev.org/x/networking-opencontrail
Tungsten Fabric and ML2 demo
10
11
Demo Setup Overview
b1s19 - node1 b1s19 - node3 b1s19 - node4eth0 eth0 eth0
VM VMVM VMVM VMSRIOV OVS TF
vMX (on b1s19 - node2)
ge-0/0
QFX
xe-0/1
vMX
xe-0/2 xe-0/3xe-0/0
11
12
Live Migration Scenario
12
b1s19 - node1 b1s19 - node3 b1s19 - node4
VM-SRIOV
eth0
VM-OVS
eth0 eth0
50.50.50.200 50.50.50.231
LAN50.50.50.0/24
VM-MIGRATE
50.50.50.183
VM-ROUTER
50.50.50.76
VM-MIGRATE
50.50.50.183
13
The Demo
14
Questions & Answers
15
Try Tungsten Fabric
Tungsten Fabric 15 minute deployment with k8s on AWS
THANK YOU.
16