AP231 EVPN & VXLAN · AP231 EVPN & VXLAN Daryl Wan Technical Marketing Engineering - Aruba Data Center Networking

AP231 EVPN & VXLAN

Daryl WanTechnical Marketing Engineering - Aruba Data Center Networking

2@ArubaAPAC | #ATM19APAC

Agenda• VXLAN Overview• VXLAN/EVPN Fundamentals• VXLAN/EVPN Wireshark Captures• VXLAN Traffic Load Sharing• AOS-CX 10.3 Data Center VXLAN/EVPN

Use Cases• VMware NSX-V/8325 Integration

VXLAN Overview


VXLAN and Overlay Networking Introduction•Virtual Extensible Local Area Network (VXLAN) is a network encapsulation mechanism that supports up to 16 million virtual network identifiers (VNIs)

•VXLAN overlay tunnels are typically created over an IP underlay network to provide L2/L3 network connectivity and multi-tenancy

•VXLAN allows traffic to be load shared across multiple equal cost paths

•VXLAN supports both intra-DC and inter-DC deployment scenarios

Virtual Overlay VXLAN tunnels

Physical Underlay Network

Data Center (DC) 1

IP

Physical Underlay Network

Data Center (DC) 2

IPIP WAN

Extended Over WAN

Intra-DC

Inter-DC

– VXLAN capable device = VXLAN Tunnel End Point (VTEP)


Benefits of Underlay and Overlay NetworkingOverlay Networks• Provide L2/L3 network connectivity for Virtual Machines (VMs)/containers/servers/routers/firewalls in different racks via

VXLAN tunnels• Cloud agility – Virtual networks can be created quickly/easily via automation between hardware/software VTEPs• Scale beyond 4K VLANs for multi-tenancy

Underlay Networks• Provide a distributed/high performance, scalable network fabric with all leaf switch network ports having equal latency for

East/West traffic• Failure of single link/spine/leaf should not impact the fabric• No Spanning Tree Protocol (STP)• Loop free, multi-pathing network L3 Fabric

VMs VMs

vSwitch vSwitch

…

…


Software VTEP to Software VTEP•Provides VM connectivity between physical hypervisors with VTEP functionality

•Not all hypervisors have VTEP functionality, additional software is typically required

•OpenStack, HPE Distributed Cloud Networking (HPE DCN), VMware NSX currently have this capability

Software VTEP A in rack 1 Software VTEP B in rack 100

Traffic between VMs via overlay VXLAN tunnel

VM2 10.0.0.6/24

172.16.10.0/24

VM1 10.0.0.2/24

Underlay Network Layer 3


Hardware VTEP to Hardware VTEP•Offloads VTEP processing from hypervisor to network device

•Provides connectivity between VMs/physical devices

•8325 currently has this capability

Hardware VTEP A in rack 1

Bare MetalServer 2

Hardware VTEP B in rack 100

Bare MetalServer3

Traffic between segments via overlay VXLAN tunnel

VM1 10.0.0.2/24

10.0.0.6/24

PhysicalRouter

Physical Firewalls

172.16.10.0/24



Software VTEP to Hardware VTEP•Provides VM connectivity to physical firewalls/WAN routers/bare metal servers

•HPE DCN, VMware NSX-V currently have this capability

VM traffic bridged to physical network via overlay VXLAN tunnel

Physical Routers

Physical Firewalls

Bare Metal Server10.0.0.6/24


VM1 10.0.0.2/24

Software VTEP A in rack 1


172.16.10.0/24


VXLAN Deployment With Centralized Control Plane •VXLAN with centralized control plane (e.g. HPE DCN, VMware NSX controller)

•Typically a VM and includes clustering capabilities for High Availability (HA)

•OVSDB / NETCONF are examples of protocols used between controller and VTEPs to setup/teardown VXLAN tunnels and share MAC addresses

Software VTEP A in rack 1

VM traffic bridged to physical network via overlay VXLAN tunnel

Physical Routers

Physical Firewalls

Bare Metal Server


Network Virtualization Controller Cluster

10.0.0.6/24

OVSDB / NETCONF

VM1 10.0.0.2/24

172.16.10.0/24



VXLAN Deployment Without Centralized Control Plane •VXLAN tunnels can be setup manually (CLI) or dynamically (MP-BGP EVPN)

Traffic between segments via overlay VXLAN tunnel

Hardware VTEP A in rack 1

Bare MetalServer 2


Bare MetalServer3

VM1 10.0.0.2/24

10.0.0.6/24

PhysicalRouter

Physical Firewalls

172.16.10.0/24


VXLAN/EVPN Fundamentals


L3 Fabric

MP-BGP EVPN Distributed VXLAN Control Plane

Resilient & Efficient

Secure

Scalable

Open Standards

Spine & Leaf L3 FabricVXLAN & EVPN

VMs VMs

vSwitch vSwitch

…

…

Multi-Protocol Label Switching (MPLS)

RFC7432

Provider Backbone Bridges (PBB+MPLS)

draft-ietf-l2vpn-pbb-evpn

Network Virtualization Overlay (VxLAN, NVGRE, MPLSoGRE)

Draft-ietf-bess-evpn-overlay

EVPN (MP-BGP) RFC7432

Data Plane

Control Plane


Before EVPN － Static VXLAN Tunnels / MAC Flood & Learn

VTEP1 1.1.1.1

VTEP2 2.2.2.2

VM1 – 101.1.0.1/24d07e-28cf-9980

VM2 – 101.1.0.2/24d07e-28cf-9900

VTEP3 3.3.3.3

VM3 – 101.1.0.3/24d07e-28cf-9940

VTEP3# sh mac-address-tableMAC age-time : 300 secondsNumber of MAC addresses : 4

MAC Address VLAN Type Port----------------------------------------------------------------------d07e-28cf-9900 10 dynamic vxlan1(2.2.2.2)d07e-28cf-9940 10 dynamic 1/1/14d07e-28cf-9980 10 dynamic vxlan1(1.1.1.1)

MAC addresses

of VMs

1. VM sends broadcast during ARP/GARP

2. MAC learned via data plane

3. MAC learned via data plane flooding

Static overlayVXLAN tunnelsbetween VTEPs

3. MAC learned via data plane flooding

vlan 10,20interface vxlan 1source ip 1.1.1.1no shutdownvni 10vlan 10vtep-peer 2.2.2.2vtep-peer 3.3.3.3vni 20vlan 20vtep-peer 2.2.2.2vtep-peer 3.3.3.3

Sample VTEP1 Configuration


With EVPN － Dynamic VXLAN Tunnels / Scalable Control Plane Learning

VTEP3# sh mac-address-tableMAC age-time : 300 secondsNumber of MAC addresses : 4

MAC Address VLAN Type Port----------------------------------------------------------------------d07e-28cf-9900 10 evpn vxlan1(2.2.2.2)d07e-28cf-9940 10 dynamic 1/1/14d07e-28cf-9980 10 evpn vxlan1(1.1.1.1)

dynamic tunnels

Local MACs shared to remote VTEPs via MP-BGP control plane

vlan 10,20evpnvlan 10rd autoroute-target export autoroute-target import auto

evpnvlan 20rd autoroute-target export autoroute-target import auto

interface vxlan 1source ip 1.1.1.1no shutdownvni 10vlan 10vni 20vlan 20

router bgp 65001bgp router-id 1.1.1.1neighbor 2.2.2.2 remote-as 65001neighbor 2.2.2.2 update-source lo 0neighbor 3.3.3.3 remote-as 65001neighbor 3.3.3.3 update-source lo 0address-family l2vpn evpnneighbor 2.2.2.2 activateneighbor 2.2.2.2 send-communityneighbor 3.3.3.3 activateneighbor 3.3.3.3 send-community

Sample VTEP1 Configuration

• Manual peering not required• Scales when there are more remote VTEPs



VTEP1 1.1.1.1

VTEP2 2.2.2.2

VTEP3 3.3.3.3

VM1 – 101.1.0.1/24d07e-28cf-9980

VM2 – 101.1.0.2/24d07e-28cf-9900

VM3 – 101.1.0.3/24d07e-28cf-9940


EVPN － MAC learning, neighbor discovery, tunnel establishment

Underlay

Rack1

VM1 –MAC 1

IP 1

VTEP1

Rack100

MAC IP VNI NH

MAC 1 IP 1 10 LOCAL

1) VTEP1 learns local VM1 MAC and IP address, triggers MP-BGP update

MAC IP VNI NH

MAC 1 IP 1 10 VTEP1

3) VTEP2 receives BGP advertisement, finds the same VNI A and learns VM1 MAC/IP info and establishes VXLAN tunnel with VTEP1

VM2 –MAC 2

IP 2

VTEP2

2) VTEP1 advertises VXLAN Network Identifier (VNI) and MAC/IP through MP BGP to BGP peer (via underlay network)

BGP UpdateRD VNI 10MAC 1 /IP 1Next hop: VTEP1RT 100:100

Overlay VXLAN Tunnel created via EVPN


EVPN － VM Migration

Underlay

VM1 –MAC 1

IP 1

VTEP1

MAC IP VNI NH

MAC 1 IP 1 10 VTEP2

4) VTEP1 receives BGP update and updates VM1 info

MAC IP VNI NH

MAC 1 IP 1 10 LOCAL

2) VTEP2 learns VM1 MAC/IP info in VNI 10, triggers BGP update

VTEP2

3) VTEP2 advertises BGP update to BGP peer

BGP UpdateRD VNI 10MAC 1 /IP 1Next hop: VTEP2RT 100:100

1) VM1 migrates from VTEP1 to VTEP2

Overlay VXLAN Tunnel created via EVPN

Rack1 Rack100


EVPN － VXLAN Unicast Traffic Forwarding

Underlay

VM1 –MAC1

IP1

VTEP1 VM2 –MAC2

IP2

VTEP2

IP MAC VNI NH

IP1 MAC1 10 LOCAL

IP2 MAC2 10 VTEP2

IP MAC VNI NH

IP1 MAC1 10 VTEP1

IP2 MAC2 10 LOCAL

S-MAC:MAC1D-MAC:MAC2S-IP:IP1D-IP:IP2



S-IP: VTEP1D-IP: VTEP2

S-MAC: VTEP1D-MAC: next hop MAC

VXLAN VNI: 10


S-IP:VTEP1D-IP:VTEP2

S-MAC: last hop MACD-MAC:VTEP2

VXLAN VNI: 10

1) VM1 sends traffic to VM2

2) VTEP1 checks EVPN table for MAC2 next hop

3) VTEP1 encapsulates VM1 traffic into VXLAN tunnel with next hop of VTEP2

4) VTEP2 decapsulates VM1 traffic in VXLAN tunnel and sends traffic via egress port to VM2

5) VM2 receives traffic from VM1Rack1 Rack100


L2 VXLAN BUM Traffic Forwarding

Underlay

VM1 VTEP1 VM2VTEP2

BUM: Broadcast, Unknown unicast and Multicast

VM3 VTEP3

VM4

1) VM1 sends out BUM traffic2) Broadcasts are sent to local VMs in the same VNI as VM1 within the same VTEP3) VTEP1 performs Head End Replication (HER) and encapsulates the frames in VXLAN tunnels, destination addresses are remote VTEPs with the same VNI4) Remote VTEPs decapsulate VXLAN header and broadcasts Ethernet frame to devices in the same VNI5) Split horizon prevents frames received from VXLAN tunnels from being forwarded back to other VTEPs

1)2)

3)

4)4)

NLRI:EVPN Type3: Inclusive multicast

RD=VTEP1VNI=10IP address=VTEP1

Ext community:Tunnel type=6Tunnel Identifier=VTEP1

VXLAN/EVPN Wireshark Captures


VXLAN Data Plane

VTEP IPs 50 Bytes encapsulation

overhead

InnerPacket

VM IPs

VNI

VXLAN = UDP Dst Port 4789


EVPN Control Plane

Multiple update

messages

Update message

info

EVPN type 2 route

VM MAC


EVPN Control Plane

Multiple update

messages

Update message

info

EVPN type 3 route

VTEP IP

VXLAN Traffic Load Sharing


VXLAN/EVPN Traffic Load Sharing

• VXLAN traffic between Leaf VTEPs should automatically load share across multiple physical underlay links via Equal Cost Multi Path (ECMP) routing

Overlay VXLAN Tunnels

Leaf VTEPs

Spine1 Spine2 Spine3 Spine4POD1

Spines

100G

40GL2

L3


Test#1: 2 way VXLAN/EVPN ECMP test with single traffic flow


Test#1: 2 way VXLAN/EVPN ECMP test with single traffic flow

• Done with back to back leafs, spines are aggregators, not required in ECMP tests between a pair of leafs• ECMP provided by OSPF• Do not expect equal load balancing, traffic is load shared across links on a per flow basis

Leaf1 Leaf2

IXIA

VXLAN Tunnel

1/1/1 – 192.168.1.0/31 1/1/1 – 192.168.1.1/31

1/1/2 – 192.168.1.2/31 1/1/2 – 192.168.1.3/31

BGP #65001Lo0 – 192.168.1.101/32 Lo0 – 192.168.1.102/32OSPF AREA 0

1/1/5 1/1/5


Test#1: Traffic flow Info

• Single traffic flow used

Leaf1 Leaf2

IXIA

VXLAN Tunnel

1/1/1 – 192.168.1.0/31 1/1/1 – 192.168.1.1/31

1/1/2 – 192.168.1.2/31 1/1/2 – 192.168.1.3/31


1/1/5 1/1/5

100.1.0.1 100.1.0.11


Test#1: Interface Bandwidth Utilization

Leaf1 Leaf2

IXIA

VXLAN Tunnel

1/1/1 – 192.168.1.0/31 1/1/1 – 192.168.1.1/31

1/1/2 – 192.168.1.2/31 1/1/2 – 192.168.1.3/31


1/1/5 1/1/5

100.1.0.1 100.1.0.11No traffic No traffic


Test#2: 2 way VXLAN/EVPN ECMP test with multiple traffic flows


Test#2: 2 way VXLAN/EVPN ECMP test with multiple flows


Leaf1 Leaf2

IXIA

VXLAN Tunnel

1/1/1 – 192.168.1.0/31 1/1/1 – 192.168.1.1/31

1/1/2 – 192.168.1.2/31 1/1/2 – 192.168.1.3/31


1/1/5 1/1/5


Test#2: Traffic Flow Info

• Multiple traffic flows used

Leaf1 Leaf2

IXIA

VXLAN Tunnel

1/1/1 – 192.168.1.0/31 1/1/1 – 192.168.1.1/31

1/1/2 – 192.168.1.2/31 1/1/2 – 192.168.1.3/31


1/1/5 1/1/5

100.1.0.1

100.1.0.2

100.1.0.3

…

100.1.0.11

100.1.0.12

100.1.0.13

…



Leaf1 Leaf2

IXIA

VXLAN Tunnel

1/1/1 – 192.168.1.0/31 1/1/1 – 192.168.1.1/31

1/1/2 – 192.168.1.2/31 1/1/2 – 192.168.1.3/31


1/1/5 1/1/5






Leaf1 Leaf2

IXIA

VXLAN Tunnel

1/1/1 – 192.168.1.0/31 1/1/1 – 192.168.1.1/31

1/1/2 – 192.168.1.2/31 1/1/2 – 192.168.1.3/31

BGP #65001

Lo0 – 192.168.1.101/32 Lo0 – 192.168.1.102/32

OSPF AREA 0

1/1/5 1/1/5

1/1/3 – 192.168.1.4/31 1/1/3 – 192.168.1.5/31

1/1/4 – 192.168.1.6/31 1/1/4 – 192.168.1.7/31


Test#3: Traffic Flow Info

Leaf1 Leaf2

IXIA

VXLAN Tunnel

1/1/1 – 192.168.1.0/31 1/1/1 – 192.168.1.1/31

1/1/2 – 192.168.1.2/31 1/1/2 – 192.168.1.3/31

BGP #65001

Lo0 – 192.168.1.101/32 Lo0 – 192.168.1.102/32

OSPF AREA 0

1/1/5 1/1/5

1/1/3 – 192.168.1.4/31 1/1/3 – 192.168.1.5/31

1/1/4 – 192.168.1.6/31 1/1/4 – 192.168.1.7/31

• Multiple traffic flows used

100.1.0.1

100.1.0.2

100.1.0.3

…

100.1.0.11

100.1.0.12

100.1.0.13

…



Leaf1 Leaf2

IXIA

VXLAN Tunnel

1/1/1 – 192.168.1.0/31 1/1/1 – 192.168.1.1/31

1/1/2 – 192.168.1.2/31 1/1/2 – 192.168.1.3/31

BGP #65001

Lo0 – 192.168.1.101/32 Lo0 – 192.168.1.102/32

OSPF AREA 0

1/1/5 1/1/5

1/1/3 – 192.168.1.4/31 1/1/3 – 192.168.1.5/31

1/1/4 – 192.168.1.6/31 1/1/4 – 192.168.1.7/31

AOS-CX 10.3 Data Center VXLAN/EVPN Use Cases


AOS-CX 10.3 DC Use Case - Centralized L2 Gateway with VXLAN/EVPN

• Centralized L2 gateway is typically used when centralized firewall functions as default gateway• Traffic on the same subnet between VTEPs does not need to traverse border leaf

Spine2

L2 VTEP1

Spine1

L2 VTEP2

POD1Overlay VXLAN

Tunnels

VMs

VLAN 11 10.1.1.10/24

VLAN 12 10.1.2.10/24

VMs

VLAN 11 10.1.1.11/24

VLAN 12 10.1.2.11/24

Default gateways 10.1.1.1/24

10.1.2.1/24External Network

802.1Q trunk (with VLANs

11/12)

10.200.200.0/24

L3 VTEP3

AS#65001

RR2RR1Leaf/spines = 8325

Leaf1 Leaf2Border Leaf3

Firewall


AOS-CX 10.3 DC Use Case - Centralized L3 Gateway with VXLAN/EVPN

• Centralized L3 gateway (border leaf) is typically used when Centralized Firewall inspection is required for traffic entering and leaving the POD• The Border leaf functions as a L2/L3 VTEP to route between subnets, provide VRF isolation and provide network connectivity between the

POD subnets and the core network• Traffic between VRFa/VRFb is not allowed (e.g. 10.3.1.10/24 cannot communicate with 10.3.2.11/24 as they are in different VRFs), traffic

within a VRF is allowed (e.g. 10.3.2.10/24 can communicate with 10.3.3.11/24 as they are both in VRFb)• Traffic between VRFa/VRFb and the external network is allowed

Spine2Spine1POD1Overlay VXLAN

Tunnels

VMs

VRFa 10.1.1.10/24

VRFb 10.1.2.10/24

VRFb 10.1.3.10/24

VMs

VRFa 10.1.1.11/24

VRFb 10.1.2.11/24

VRFb 10.1.3.11/24

Default gateways 10.1.1.1/24 (VRFa)

10.1.2.1/24 (VRFb)

10.1.3.1/24 (VRFb)

External Network

802.1Q trunk with VRFa and VRFb

OSPF/BGP

10.200.200.0/24

RR2RR1AS#65001 Leaf/spines = 8325

Leaf1 Leaf2 Border Leaf3

Firewall

L2 VTEP1 L2 VTEP2

L3 VTEP3


AOS-CX 10.3 supports• The 2 use cases mentioned in previous slides

• A dedicated border leaf is not mandatory, the border leaf can connect to servers too

• Only 8325 supports VXLAN, IBGP EVPN and IBGP Route Reflector functionality

• EVPN type 2 - MAC/IP advertisement route • Advertises MAC reachability information and host route information (host ARP or ND information)

• EVPN type 3 - Inclusive multicast Ethernet tag (IMET) route• Advertises VTEP and VXLAN mappings for automating VTEP discovery, VXLAN tunnel establishment, and VXLAN tunnel assignment

• IPv4 L3 unicast routing in the overlay network

• IPv4 L2 multicast BUM traffic in the overlay network

• IPv4 unicast VTEPs in the underlay network

• 1:1 VNI/VLAN mapping

• L3 VXLAN centralized gateway supports DHCP server and DHCP relay

VMware NSX-V/8325 Integration


• Used in environments with NSX, VMs and bare metal servers• Provides L2 network connectivity between VMs (on ESXi hosts) and Bare

Metal Servers connected to the Hardware VTEP switch (8325)• Standalone 8325/NSX-V integration currently works in 10.3 • Standalone 8325/NSX-V certification planned for 10.4 (suitable for production

deployment)

VMware NSX-V (VMware NSX Data Center for vSphere) and 8325 Integration

Underlay Network

VXLAN Overlay Tunnels

VM2101.1.0.13/24

NSX Controller 6.4

NSX Manager 6.4

vCenter 6.7

8325

VM1101.1.0.12/24

OVSDB

OOBM 10.10.10.163/24

Bare Metal101.1.0.11/24

Proprietary control plane

Intelligent Edge Related Sessions:• AP215 Modern switches with Built-in Network Analytics (Sep 24, 12.30pm)

• AP221 Designing The New Edge - Building a Foundation for Intelligent Edge Networking (Sep 24, 12.30pm/3.15pm, Sep 25, 10am)

• AP222 Limiting your Network Blast Radius with Aruba's HA Architecture (Sep 24, 3.15pm/5.15pm)

• AP231 EVPN & VXLAN (Sep 25, 2.45pm)

• AP218 Protect and Secure your IoT with Aruba's Dynamic Segmentation (Sep 25, 3.45pm/5:30pm) and (Sep 26, 11am)

• AP219 Intelligent Wired Edge Solutions and Innovations with End-User Panel (Sep 25, 5:30pm)

• AP220 Embrace Network Automation with Ansible (Sep 26, 9am)

• AP216 Orchestrating Network Configuration with Aruba's NetEdit (Sep 26, 10am)

Intelligent Edge Related Demos at Tech Playground

• SimpliVity/AOS-CX Integration

• NetEdit

• NAE

and more….

Rate this sessionAccess this survey via the mobile app and let us know what you think.

Locate this session: - Agenda - Select Date - Find this session - Click Survey

Download the Event AppGain access to the latest event information.

Scan. Play. Win.Play Now! Switch your thinking, say goodbye to the old ways and get ready for new innovations.

Visit the Tech Playground now!

Ask Aruba - Session 1: Tuesday, 24 Sep, 2:00pm – 2:45pm- Session 2: Wednesday, 25 Sep, 1:45pm – 2:30pm - Location: Town Hall at Tech Playground

Submit your Ask Aruba questions using the mobile app now!

Join the Airheads Community Scan the QR code to sign up now!


Thank You

Documents

AP231 EVPN & VXLAN · AP231 EVPN & VXLAN Daryl Wan Technical Marketing Engineering - Aruba Data Center Networking