NET1863BU NSX-T Advanced Architecture, …...François Tallet, NSBU NET1863BU #VMworld #NET1863BU NSX-T Advanced Architecture, Switching and Routing VMworld 2017 Content: Not for publication

François Tallet, NSBU

NET1863BU

#VMworld #NET1863BU

NSX-T AdvancedArchitecture, Switching and Routing

VMworld 2017 Content: Not fo

r publication or distri

bution

• This presentation may contain product features that are currently under development.

• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

• Technical feasibility and market demand will affect final delivery.

• Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Disclaimer

CONFIDENTIAL 2



bution

Agenda

1 Architecture and Switching

2 Routing

3 Distributed and Edge Firewall

#NET1863BU CONFIDENTIAL 3

Introduction to NSX-T Architecture [NET1510BU]

- Andrew Voltmer, Group Product Line Manager, VMware

- Dimitri Desmidt, Senior Technical Product Manager, VMware



bution

NSX-T Architecture & Switching



bution

NSX Architecture and Components

Data Plane• High Performance Data Plane

• Scale-out Distributed Forwarding Model

NSX Edge (VM or

Bare Metal)

Physical

Infrastructure

Cloud Consumption• Self Service Portal

• OpenStack, Custom

Transport Nodes

Layer 2

Bridge

8

ESXi HV KVM HV

vSwitch vSwitch

CCP Cluster

Central Control Plane (CCP) Nodes- VM form factor

• Control-Plane Protocol

• Dynamic state

• Separation of Control and Data Plane

Control Plane

VPN

• Concurrent configuration portal

• REST API entry-point

• UI

Management Plane (MP) Node – VM form factor

Management Plane



bution

Switching Demo: Logical Switch Creation

9

Web1 Web2

VIF1 VIF2

• Virtual Interface (VIF): Compute manager object representing the VM vnic

vCenter1 vCenter2

172.16.10.11 172.16.10.12

ESXi HV1

vSwitch

ESXi HV2

vSwitch



bution

On vCenter1 On vCenter2

Switching Demo: Logical Switch Creation

10

Web1 Web2

VIF1 VIF2


vCenter1 vCenter2

172.16.10.11 172.16.10.12

ESXi HV1

vSwitch

ESXi HV2

vSwitch



bution

Switching Demo: Ping

11

Web1 Web2

VIF1 VIF2

172.16.10.11 172.16.10.12


ESXi HV1

vSwitch

ESXi HV2

vSwitch

vCenter1 vCenter2



bution

• Tunnel End Point (TEP)


• Logical Interface (LIF): port on the logical switch

Central Control Plane Cluster

Management Plane Node

NSX Architecture in Action

12

Compute

Manager

(vCenter1)

Compute

Manager

(vCenter2)Web1

Web-LS

ESXi HV1

vSwitch

TEP1

ESXi HV2

vSwitch

TEP2

Web2

VIF1 VIF2

1. Create Web-LS 2. Configure Web-LS LS1

LIF1 VIF1

3. Advertise Web-LS to ESXi HVs

4. Attach Web1 to Web-LS

6. Configure LIF1 on Web-LS attached to VIF1

5. attach VIF1 to Web-LS



bution

• Tunnel End Point (TEP)


• Logical Interface (LIF): port on the logical switch



NSX Architecture in Action

13

Compute

Manager

(vCenter1)

Compute

Manager

(vCenter2)Web1

Web-LS

ESXi HV1

vSwitch

TEP1

ESXi HV2

vSwitch

TEP2

Web2

LIF1

VIF1 VIF2

7. Advertise Web-LS and LIF1 to CCP

9. LIF1 created on Web-LS

10. Mac1 associated to TEP1

Mac1 TEP1

MAC@ TEP IP

LS1

LIF1 VIF1

8. Web-LS created, I’m master



bution

Identify the VIF of a KVM Virtual Machine

15

Web1 Web2

VIF2VIF1

LIF1 LIF2

Web3

VIF3

VIF3 UUID: ?

172.16.10.11 172.16.10.13172.16.10.12

ESXi HV1

vSwitch

ESXi HV2

vSwitch

KVM HV3

vSwitch



bution

Identify the VIF of a KVM Virtual Machine

16

Web1 Web2

VIF2VIF1

LIF1 LIF2

Web3

VIF3

VIF3 UUID: 57601300-2e82-48c4-8c27-1e961ac70e79

172.16.10.11 172.16.10.13172.16.10.12

ESXi HV1

vSwitch

ESXi HV2

vSwitch

KVM HV3

vSwitch



bution

Attach KVM Virtual Machine to a Logical Switch with Logical Port

17

Web1 Web2

VIF2VIF1

LIF1 LIF2

Web3

VIF3

LIF3

VIF3 UUID: 57601300-2e82-48c4-8c27-1e961ac70e79

172.16.10.11 172.16.10.13172.16.10.12

ESXi HV1

vSwitch

ESXi HV2

vSwitch

KVM HV3

vSwitch



bution

Ping KVM/ESXi

18

Web1 Web2

VIF2VIF1

LIF1 LIF2

Web3

VIF3

LIF3

172.16.10.11 172.16.10.13172.16.10.12

ESXi HV1

vSwitch

ESXi HV2

vSwitch

KVM HV3

vSwitch



bution



Adding KVM Port

19

Compute

Manager

(vCenter1)

Compute

Manager

(vCenter2)Web1

Web-LS

ESXi HV1

vSwitch

TEP1

ESXi HV2

vSwitch

TEP2

Web2

LIF1

VIF1

LIF2

VIF2

KVM HV3

vSwitch

TEP3

Web3

LIF3

VIF3

2. Configure LIF3 attached to VIF3 on Web-LS

3. Advertise LIF3

4. LIF3 created

Mac1 TEP1

MAC@ TEP IP

Mac2 TEP2

Mac3 TEP3

Mac1 TEP1

MAC@ TEP IP

Mac2 TEP2

1. Attach VIF3 to Web-LS

LS1

LIF1 VIF1

LIF2 VIF2

LIF3 VIF3

5. Mac3 associated to TEP3

6. Mac TEP associations advertised to HV3VMworld 2017 Content: N

ot for publicatio

n or distribution

Unicast Packet Walk

• Web3 sends a unicast to Web1

• A lookup is made for Mac1

• If it’s a hit {

– Frame is encapsulated

– Frame is sent unicast to remote TEP

} else {

– Frame is flooded

}

20

HV3

Web1 Web3

LS

HV1

TEP1 TEP3

MAC@ TEP IP

Mac1 TEP1

Mac2 TEP2

Mac3 local

Mac1 ?

Central Control Plane Cluster Mac1 TEP1

Mac2 TEP2

Mac3 TEP3

MAC@ TEP IP

ma

c1

TE

P1

ma

c1

ma

c1 Overlay

encapsulated frame

Mac1 ?

Web2



bution

BUM Traffic Handling : Unicast (MTEP)

• Traffic flooded from Web1 on HV1 on a Logical Switch

• Frame replication is achieved at two tiers, based on the TEP subnets

21

HV1

HV2

HV3

HV4

HV5

HV6

HV7

HV8

HV9

TEP1

TEP2

TEP3

TEP4

TEP5

TEP6

TEP7

TEP8

TEP9

TEP1, TEP2, TEP3 have IP addresses in

subnet A


subnet B


subnet C

1. HV1 replicates the frame to all TEPs in its subnet A

2. HV1 forwards the frame to one TEP in each remote subnet B & C

3. Remote TEPs in subnet B & C replicate the frame to other interested TEPs in their respective subnet.

HV6 has no logical port in the logical switch

Web1



bution

HV5HV4HV1

TEP1 TEP4

Web1Mac1

MAC@ TEP IP

TEP5

Flood and Learn

• The controller distribute Mac TEP association, but NSX can also do data plane learning

• Example of data plane learning of Mac1 of VM Web 1 from a flooded frame:

22

Web1Mac1

Src IP:TEP1Dest IP:TEP2Src Mac1: Dest Mac FFL2 PayloadHV1

TEP1

HV2

TEP2

MAC@ TEP IP

Mac1 TEP1

Tunnel HeaderInner Mac @s

Mac1 Mac1

Mac1 TEP4

wrong

• Now, a more complex example (MTEP replication, as seen previous slide)



bution

Flood and Learn

• The controller distribute Mac TEP association, but NSX can also do data plane learning

• Example of data plane learning of Mac1 on HV5 from a frame flooded by VM Web1

23

Web1Mac1

Src IP:TEP1Dest IP:TEP4Src Mac1: Dest Mac FFL2 PayloadHV1

TEP1

HV2

TEP2

MAC@ TEP IP

Mac1 TEP1

Tunnel HeaderInner Mac @s

• Now, a more complex example (MTEP replication, as seen previous slide)

• Solution: Carry some metadata identifying the source TEP in the encapsulation

HV5HV4HV1

TEP1 TEP4

Web1

Mac1

Mac1 Mac1

Mac1 TEP1

MAC@ TEP IP

TEP5S:Tep1 S:Tep1



bution

Choice for NSX Overlay Encapsulation

• Metadata is critical to any distributed system,

• Encapsulations designed around hardware-based forwarding typically have fixed fields

• New features might require new metadata…

• NSX is currently leveraging GENEVE as a tunneling mechanism (https://datatracker.ietf.org/doc/draft-ietf-nvo3-geneve/)

– It maintains the traditional offload capabilities offered by NICs for best performance

– Provides complete flexibility for inserting Metadata as Type Length Value (TLV) fields

Note:

• Third party devices don’t need to understand NSX tunnels

• Tools for looking inside GENEVE tunnels are available (Wireshark dissector for ex.)

• NSX can handle different types of tunnels simultaneously.

24



bution

https://datatracker.ietf.org/doc/draft-ietf-nvo3-geneve/

NSX-T Routing



bution

Logical Routing Demo: Create Logical Router

29

App1 Web1

172.16.20.11 172.16.10.11



bution

Logical Routing Demo: Create Logical Router

30

App1 Web1

to App-LS

172.16.20.1

app-ls

to Web-Ls

172.16.10.1

web-ls

Tenant1 Router

172.16.20.11 172.16.10.11



bution

Logical Routing Demo: Traceroute

31

App1 Web1

to App-LS

172.16.20.1

app-ls

to Web-LS

172.16.10.1

web-ls

Tenant1 Router

172.16.20.11 172.16.10.11



bution

Distributed Routing

32

HV1 HV2

Web1App1 Web2

172.16.20.1 172.16.10.1 172.16.20.1 172.16.10.1



bution

Distributed Routing

33

HV1 HV2

Web1App1 Web2

172.16.20.1 172.16.10.1 172.16.20.1 172.16.10.1

In-kernel routing:

this is not a VM



bution

Traceflow Demo

34

App1 Web1

to App-LS

172.16.20.1

app-ls

to Web-LS

172.16.10.1

web-ls

Tenant1 Router



bution

Edge

Node

Distributed Routing with a Centralized Component

35

Introducing the Edge Node

HV1 HV2

Web1App1 Web2

172.16.20.1 172.16.10.1 172.16.20.1 172.16.10.1

172.16.20.1 172.16.10.1 Physical Port

Physical

Router



bution

Edge

Node

Distributed Routing with a Centralized Component

36

Introducing the Edge Node

HV1 HV2

Web1App1 Web2

172.16.20.1 172.16.10.1 172.16.20.1 172.16.10.1

Physical Port172.16.20.1 172.16.10.1

“Intra-Router” tunnel

Physical

Router



bution

• Edge Nodes are appliances with pool of capacity for handling services that cannot be distributed. Example of services:

– Peering with the physical infrastructure

– NAT

– DHCP Server, MetaData Proxy

– Edge Firewall

• Edges are available in 2 form factors – Bare Metal & VM

• Both leverage Intel’s DPDK (DataPlane Development ToolKit)

– High forwarding performance

– Linear performance increase by addition of cores.

Edge Nodes

37

Edge

Node1

Edge Cluster

Edge

Node2

Edge

Node3

DHCP

Those are services, not VMs

More on NSX-T Performance:

NSX Performance Deep Dive [NET1343BU]

- Samuel Kommu, Sr. Technical Product Manager, Vmware



bution

Two-Tier Routing

• Provider Logical Router – Tier0 LR

– Role – Attach to the physical routing infrastructure

– Manual management

• Tenant Logical Router – Tier1 LR

– Role – Per tenant first hop router

– Cloud Management Platform (CMP) driven management

• No dynamic routing between tiers: NSX distributes the appropriate routes

39

vmBvmA vmC vmD

Admin

Tenants/CMP



bution

HV2

2-Tier Routing is Distributed

• Tier0 and Tier1 routers are also instantiated on the hypervisors in order to prevent hair-pinning

• Fully distributed architecture : as much routing as possible is performed upfront at the source

HV1

vmD vmD

40

• Again: the forwarding tables of the distributed components are populated by NSX There is no routing protocol involved for communication within NSX

vmA



bution

Gateway Service to the Physical Infrastructure

• Tier0 logical router supports:

– Static routes towards physical

– eBGP towards physical

• ECMP supported using static routes and eBGP

• BFD towards physical to protect against link failures

Edge

Node

Edge Cluster

Edge

Node

Edge

Node

Edge

Node

Peering to a single Tier 0 router

from the perspective of the

physical infrastructure

eBGP

eBGP

eBGP

eBGP

41



bution

Edge

Node

HV1

ECMP with Physical Infrastructure

Edge

Node

Edge Cluster

vmA42

vmA

Physical viewLogical view



bution

BGP Demo: Setting Up BGP Neighbor

44

192.168.240.1

Tenant1 Router

Edge

TN1

T0 Router

Uplink-LS1

edgecluster1

Physical Router

Uplink1:192.168.240.3

172.16.20.1/24

App1Web2 Web2 Web2

172.16.10.1/24



bution

BGP Demo: Setting Up BGP Neighbor

45

192.168.240.1

Tenant1 Router

Edge

TN1

T0 Router

Uplink-LS1

edgecluster1

Physical Router

Uplink1:192.168.240.3

172.16.20.1/24

App1Web2 Web2 Web2

172.16.10.1/24

AS200

AS100

EBGP



bution

BGP Demo: Redistribute NSX Static Routes

46

192.168.240.1

Tenant1 Router

Edge

TN1

T0 Router

Uplink-LS1

edgecluster1

AS200

AS100

Physical Router

Uplink1:192.168.240.3

172.16.20.1/24

App1Web2 Web2 Web2

172.16.10.1/24

EBGP

BG

P A

dve

rtis

e

NS

X S

tatic



bution

BGP Demo: Attach Tier1 Router to Tier0 Router

47

192.168.240.1

Tenant1 Router

Edge

TN1

T0 Router

Uplink-LS1

edgecluster1

AS200

AS100

100.64.0.0/31

100.64.0.1/31

Physical Router

Uplink1:192.168.240.3

172.16.20.1/24

App1Web2 Web2 Web2

172.16.10.1/24

EBGP

BG

P A

dve

rtis

e

NS

X S

tatic



bution

BGP Demo: Show BPG Routes on Physical Infrastructure

48

192.168.240.1

Tenant1 Router

Edge

TN1

T0 Router

100.64.0.0/31

Uplink-LS1

edgecluster1

AS200

AS100

100.64.0.1/31

Physical Router

Uplink1:192.168.240.3

172.16.20.1/24

App1Web2 Web2 Web2

172.16.10.1/24

EBGP

BG

P A

dve

rtis

e

NS

X S

tatic



bution

BGP Demo: Advertise Connected Routes from Tier1

49

More on NSX-T Routing:

NSX Logical Routing [NET1416BU]- Pooja Patel, Senior Manager, Technical Product Management, VMware

- Jerome Catrouillet, Senior Product Line Manager, VMware

192.168.240.1

Tenant1 Router

Edge

TN1

T0 Router

100.64.0.0/31

Uplink-LS1

edgecluster1

AS200

AS100

100.64.0.1/31

Physical Router

Uplink1:192.168.240.3

172.16.20.1/24

App1Web2 Web2 Web2

172.16.10.1/24

EBGP

Ad

ve

rtis

e

co

nn

ecte

d

BG

P A

dve

rtis

e

NS

X S

tatic



bution

NSX Firewall



bution

Micro-Segmentation with Distributed Firewall (DFW)

• Each VM is its own perimeter

• Policies align with logical groups

• Prevents threats from spreading

• DFW available on ESXi and KVM

51

Web3Web2 NAT01Web1

App1 App2

DB1



bution

Micro-Segmentation Demo: Traceflow Web1 Web3

52



bution

Micro-Segmentation Demo: NSGroup

Tags can be dynamically applied to:

- Logical Switch

- Logical Ports

- VMs

NSGroups can be created by combining tags and VM names.

53



bution

Micro-Segmentation Demo: Preventing Web to Web Traffic

54



bution

Micro-Segmentation Demo: New Traceflow Web1Web3

55



bution



Adding Firewall Rule

56

vmA

LS 1

ESXi HV1

vSwitch

TEP1

ESXi HV2

vSwitch

TEP2

vmB

LIF A

VIF A

LIF B

VIF B

KVM HV3

vSwitch

TEP3

vmC

LIF C

VIF C

1. Drop Web to Web communication

Drop 172.16.10.11 172.16.10.12

Drop 172.16.10.12 172.16.10.13

Drop 172.16.10.11 172.16.10.13

5. Rule programmed in datapath of affected hosts

2. Rule saved in database

3. Push rule to CCP



bution

HV1

Edge Firewall

vmA

57

• Stateful Firewall we need to see both directions of the traffic

• Practically, the firewall has to be centralized

• For the DFW, the firewall is naturally centralized on the LIF where the VM vnics attach

• For a Firewall on the uplink of a router, we’ll use an Edge node (same as peering to physical)

Edge

Node

vmC

HV2

vmD

Context

Edge Firewall



bution

HV1

Edge

Node

HV2

Packet Walk with Edge Firewall Service

• The FW sees traffic both ways

vmA vmC vmD

58

Edge Firewall

More on Security:

The Future of Networking and Security with NSX-T [NET1821BU]- Bruce Davie - CTO VMware



bution

Wrapping Up



bution

NSX-T

• This presentation was on understanding how basic NSX-T networking capabilities worked

• In particular NSX-T

– Decouple networking from the hardware and from vCenter

– Is Multi-Hypervisor

– Uses high performance Edge Nodes for its centralized services

– Interconnects its components with minimal user intervention

SPL182601U VMware NSX-T – Getting Started

SPL182602U VMware NSX-T - NSX-T with Kubernetes

60



bution



bution



bution

Documents

NET1863BU NSX-T Advanced Architecture, …...François Tallet, NSBU NET1863BU #VMworld #NET1863BU NSX-T Advanced Architecture, Switching and Routing VMworld 2017 Content: Not for publication