VXLAN Nexus 9000 Module 5 – MP-BGP EVPN @onecloudinc.com

VXLAN Nexus 9000 Module 5 – MP-BGP EVPN

@onecloudinc.com

Agenda

MP-BGP EVPN Overview VXLAN with EVPN Control Plane Route Optimization using EVPN VXLAN EVPN Features Add Control Plane EVPN For

Ingress Replication VXLAN EVPN Configuration VXLAN EVPN Design Options Scalability Limits

MP-BGP EVPN Overview

Multiprotocol-BGP (MP-BGP): Carries multiple address-family reachability info (IPv4, IPv6, VPN, EVPN)

MP-iBGP: Internal BGP session between internal peers

MP-eBGP: External BGP session between external peers (in different BGP autonomous systems)

Route-reflector (RR): Propagate BGP prefixes to iBGP peers

Route Distinguisher (RD): 8-byte field, VRF parameters; unique value to make VPN IP routes unique

VPNv4 address: RD + VPN prefix

Route Target (RT): 8-byte field, unique value to define the import/export rules for VPNv4 routes

VRF Table: Hold customer routes at PE/VTEP

VNI: Virtual Network Interface

PE: Provider Edge Router

CE: Customer Router

MP-BGP and E-VPN Terminology

Extension to Border Gateway Protocol (BGP)

Multi-Protocol BGP (MP-BGP) carries multiple address family information IPv4 (Unicast and Multicast) IPv6 (Unicast and Multicast) VPNv4, MVPN, EVPN (Ethernet VPN)

Address families controls type of routed protocol information exchanged

Used to distribute VPN routing information between PE’s

Allows to distinguish overlapping customer routes using Route Distinguisher (RD)

Allows reachability among different virtual networks using Route Target (RT)

Customer routes held in different VPN tables

Multi-Protocol BGP (MP-BGP)

Exchange of VPN Policies Among PE Routers

MP-BGP with MPLS VPN Route Distribution

Full mesh of BGP sessions among all PE routers BGP Route Reflector

Multi-Protocol BGP extensions (MP-iBGP) to carry VPN policies

PE-CE routing options Static routes eBGP OSPF IS-IS

Label Switched Traffic

PE-CELink

Blue VPN Blue VPN

Red VPN Red VPN

BGP Route Reflector

VXLAN with EVPN Control Plane

VXLAN Evolution

Uses Multi-Protocol BGP (IBGP or EBGP) w EVPN Address Family for: Dynamic Tunnel Discovery Dynamic Host reachability information

VTEP VTEP VTEP VTEP VTEP

RouteReflector

IBGP Route Reflector*(on spine or different

VXLAN Overlay

BGP Peers on VTEPs

Scalable BGP EVPN Control Plane

Control Protocol for VXLAN

Protocol Learning

BGP propagates routes for the host to all other VTEPs

Overlay Forwarding TableHost1 <MAC,IP> , VTEP IP AHost2 <MAC,IP> , VTEP IP B

VTEPs advertise host routes (IP+MAC) to RR

IP A IP B

BGP Route Reflect

Overlay Forwarding TableHost1 <MAC,IP> , VTEP IP A

3VTEPs obtain host routes for remote hosts and install in RIB/FIB

BGP MPLS Based Ethernet VPN (draft-ietf-l2vpn-evpn-02) Network Virtualization Overlay Solution using EVPN (draft-sd-l2vpn-evpn-overlay-02)

Protocol Learning

BGP : The Advantages

Scale Policy Security

With EVPN Address Family

Carries Host L2-address Carries Host IP/L3 address

Bridge Route

Overlay

Optimally

Active/Active Multipathing

Resilient & Efficient

No Tromboning

Using MP-BGP to distribute overlay information Use MP-BGP with EVPN Address Family for Tunnel Endpoint Discovery and Host Reachability Distributed protocol – no single point of failure Proven BGP scalability Physical and virtual endpoints can become BGP peers to participate

Why not use a Central Controller? Single point of control could become a single point of failure Scalability Scope of the controller is limited to devices that understand the communication protocol

used by the controller

Why VXLAN EVPN Control Plane?

IGP for underlay network reachability

Node reachability for overlay

Quick reaction to fabric link/node failure

Enhanced for mesh topologies

Underlay Control Protocol doesn’t distribute

Host Routes

Host originated control traffic

Server subnet informationOSPF Adjacencies

VXLAN Underlay - Unicast

L3 Core

Step 1 – OSPF/IGP as Underlay Control Plane

OSPF Control Plane

Multicast for underlay network

VXLAN VTEP peer discovery

Forward BUM traffic (broadcast, unknown and multicast)

Multicast Enabled Fabric

PIM Adjacencies

VXLAN Underlay - MulticastStep 2 – Enable Multicast on Underlay network

L3 Core

Host Route Distribution decoupled from the Underlay IGP protocol

Use MP-BGP on the leaf nodes to distribute internal host/subnet routes and external reachability information

MP-BGP enhancements to carry up to 100s of thousands of routes and reduce convergence time

Step 3 – Host and Subnet Route Distribution

iBGP Adjacencies

VXLAN Overlay MP-BGP EVPN Control Plane

L3 Core

Host/Subnet Route Injection External

Subnet Route Injection

MP-BGP Control Plane

Route-Reflectors deployed for scaling purposes

Control-Plane EVPNPeer and Host discovery

VXLAN Peer and Host Learning Options

Host Learning

Data-Plane Control-Plane

Multicast

UnicastVlan 2 vn-segment 4098Interface nve 1 host-reachability protocol bgp member vni 4098 ingress-replication protocol bgp

Vlan 2 vn-segment 4098Interface nve 1member vni 4098 ingress-replication protocol static

Vlan 2 vn-segment 10000Interface nve 1 host-reachability protocol bgp member vni 4098 mcast-group 225.1.1.1

Vlan 2 vn-segment 4098Interface nve 1member vni 10000 mcast-group 225.1.1.1

Flood and LearnPeer Learning: DP

EVPN-MulticastPeer Learning: BGP

Static Ingress-ReplicationPeer Learning: CLI

EVPN Ingress-ReplicationPeer Learning: BGP

Dynamic VTEP Peer Discovery

VTEP-2

VTEP-1

VTEP-3

H-MAC-1H-IP-1VLAN-1 /VNI-1

BGP Update:H-MAC-1H-IP-1VTEP-1VNI-1

BGP Update

Route Reflector

Dynamic VTEP peer discovery:

VTEP1 peer via EVPN MP-BGP Update

Local learning of host info:

H-MAC-1 (MAC table)

H-IP-1 (VRF IP host table )

Dynamic VTEP peer discovery:

VTEP1 peer via EVPN MP-BGP Update

With EVPN Control Plane: Underlay multicast is NOT needed for VTEP peer discovery

MAC Host IP

VNI VTEP

H-MAC-1

H-IP-1 VNI-1

VTEP-1

Dynamic Host Reachability Advertisement

Install host info to RIB/FIB:

H-MAC-1 MAC table

H-IP-1 VRF IP host table

Install host info to RIB/FIB:

H-MAC-1 MAC table

H-IP-1 VRF IP host table

MAC Host IP

VNI VTEP

H-MAC-1

H-IP-1 VNI-1 VTEP-1

MAC Host IP

VNI VTEP

H-MAC-1

H-IP-1 VNI-1 VTEP-1

H-MAC-1

H-IP-1

VLAN-1 /VNI-1

Leaf must discover first locally connected devices

Local learning of host info:

H-MAC-1 (MAC table)

H-IP-1 (VRF IP host table )

Detection of local hosts using ARP/ND/DHCP (New mac learn)

Detection of remote hosts via Received MP-BGP updates

Route Reflector

VTEP-3

VTEP-1

VTEP-2

Use MP-BGP with EVPN Address Family on VTEPs to distribute:

Internal host MAC/IP addresses

Subnet routes

External reachability information

MP-BGP enhancements to carry up to 100s of thousands of routes with reduced convergence time

VTEP Functions are on leaf layer

Spine nodes don’t need to be VTEP

EVPN Control Plane -- Host and Subnet Route Distribution

BGP Update• Host-MAC• Host-IP• Internal IP Subnet• External Prefixes

MP-BGP for VXLAN EVPN Control PlaneEVPN Control Plane – Reachability Distribution

L2 L3 L4

S2 S3 S4 LeafVTEPVTEPVTEPVTEP

Any subnet anywhere => Any leaf can instantiate any subnet

All leafs share gateway IP and MAC for a subnet (No HSRP)

ARPs are terminated on leafs, no flooding beyond leaf

Facilitates VM Mobility, workload distribution, arbitrary clustering

Seamless L2 or L3 communication between physical hosts and virtual machines

GW IP: 11.11.11.1GW MAC: 0011:2222:3333

GW IP: 10.10.10.1GW MAC: 0011:2222:3333

Anycast Gateway

Anycast Gateway at the Leaf

VTEP-1

VTEP-2

VTEP-3

VTEP-4

VTEP-5

VTEP-6

Optimized Network

VTEP-1 detects Host1 and advertise an EVPN route for Host1 with seq# 0 Host1 Moves behind VTEP-3 VTEP-3 detects Host1 and advertises an EVPN route for Host1 with seq #1 VTEP-1 sees more recent route and withdraws its advertisement

MAC IP VNI Next-Hop Encap Seq

MAC-1 IP-1 5000 VTEP-1 VXLAN 0

VXLAN BGP Control Plane

Host 1MAC1IP 1

VNI 5000

EVPN Control Plane - Host Movement

VTEP-4VTEP-3VTEP-2VTEP-1

NLRI:• Host MAC1, IP1 • NVE IP 1• VNI 5000• Next-Hop: VTEP-3

Ext. Community:• Encapsulation:

VXLAN• Cost/Sequence: 1

Ext. Community:• Encapsulation:

VXLAN• Cost/Sequence: 0

VTEP-1 detects Host1 and advertise an EVPN route for Host1 with seq# 0 Host1 Moves behind VTEP-3 VTEP-3 detects Host1 and advertises an EVPN route for Host1 with seq #1 VTEP-1 sees more recent route and withdraws its advertisement

VXLAN BGP Control Plane

Host 1MAC1IP 1

VNI 5000

EVPN Control Plane - Host Movement

VTEP-4

VTEP-3

VTEP-2

VTEP-1

Ext. Community:• Encapsulation: VXLAN• Cost/Sequence: 1

Ext. Community:• Encapsulation: VXLAN• Cost/Sequence: 0

Summary Classic Vs EVPN Control Plane

Classic VXLAN EVPN Control Plane

Peer Discovery Data-driven flood-&-learn MP-BGP

Host Route Learning Local hosts: Data-driven flood-&-learnRemote hosts: Data-driven flood-&-learn

Local Host: Data-drivenRemote host: MP-BGP

BUM Traffic forwarding Multicast replicationUnicast/Ingress replication

Multicast replicationUnicast/Ingress replicationMulticast Dependency Reduced

Route Optimization Hair-pinning, Need extra VTEPs and Routers

Routing on source VTEP, Anycast Gateway

Control Plane EVPN Summary

• Head-end replication to allow unicast-mode only operation• Introduce a control plane to allow for dynamic VTEP

discoveryMulticast Dependency

• Workload MAC addresses are known once they are connected to the VXLAN capable devices

• Leverage the control plane also to exchange L2/L3 address-to-VTEP association information

Flood and Learn based Learning

• Introduce VXLAN GatewaysExternal Connectivity

Route Optimization using EVPN

VXLAN Routing

Different integrated Route/Bridge (IRB) Modes

Overlay Networks do follow two slightly different integrated Route/Bridge (IRB) semantics

Asymmetric

Ingress VTEP performs both Layer-2 bridging and Layer-3 routing lookup, whereas the egress VTEP performs only Layer-2 bridging lookup

Symmetric

Both the ingress and egress VTEPs perform Layer-2 and Layer-3 lookups

Cisco follows Symmetric IRB

Host 1H-MAC-1H-IP-1VNI-A

Host 2H-MAC-2H-IP-2VNI-B

IP Transport Network

Routing ?

Asymmetric IRB (Cont’d)

VNI A VNI B VNI BVNI A

S-MAC: H-MAC-1

D-MAC: H-MAC-2

S-IP: H-IP-1D-IP: H-IP-2

Ingress VTEP routes packets from source VNI to destination VNI. D-MAC in the inner header is the destination host MAC

1S-MAC: H-

MAC-1D-MAC: H-

MAC-2S-IP: H-IP-1D-IP: H-IP-2

S-IP: VTEP-1D-IP: VTEP-4

VNI: VNI-B

S-MAC: H-MAC-1

D-MAC: H-MAC-2

S-IP: H-IP-1D-IP: H-IP-2 Egress VTEP

bridges packets in the destination VNI

Asymmetric Routing and Bridging on the ingress VTEP Bridging on the egress VTEP Both source and destination VNIs need to reside on the ingress VTEP

Symmetric IRB

Routing on both ingress and egress VTEPs Layer-3 VNI

Tenant VPN indicator One per tenant VRF

VTEP Router MAC Ingress VTEP routes packets onto the Layer-3 VNI Egress VTEP routes packets to the destination Layer-2 VNI

Symmetric IRB (Cont’d)

VTEP-4Router MAC-4

VTEP-3VTEP-2VTEP-1Router MAC-1

L3 VNI

S-MAC: H-MAC-1

D-MAC: H-MAC-2

Ingress VTEP routes packets from source VNI to L3 VNI. D-MAC in the inner header is the egress VTEP router MAC

1S-MAC: Router-MAC-

1D-MAC: Router-MAC-

4S-IP: H-IP-1D-IP: H-IP-2

S-IP: VTEP-1D-IP: VTEP-4VNI: L3 VNI

S-MAC: H-MAC-1

D-MAC: H-MAC-2

Egress VTEP routes packets from L3 VNI to the destination VNI/VLAN2

Symmetric IRB - Benefits

VTEPs don’t need to learn and maintain MAC address information for the remote hosts attached to egress VNIs for which it doesn’t have local hosts

Better utilization of the MAC address table and ARP adjacencies on a VTEP.

Routing and bridging is more scalable than with asymmetric IRB.

Cisco NX-OS implements symmetric IRB to achieve optimal learning and scaling.

EVPN Multi-Tenancy and VNI Types (Cont’d)

Layer-3 VNI (VRF VNI)

Layer-2 VNI (Network

Layer-2 VNI (Network VNI)

vlan 200 vn-segment 20000vlan 201 vn-segment 20100

vlan 3900 name l3-vni-vlan-for-tenant-1 vn-segment 39000

interface Vlan3900 description l3-vni-for-tenant-1-routing no shutdown vrf member evpn-tenant-1 ip address 39.0.0.1/16 fabric forwarding mode anycast-gateway

vrf context evpn-tenant-1 vni 39000 rd auto address-family ipv4 unicast route-target import 39000:39000 route-target export 39000:39000 route-target both auto evpn

interface Vlan200 no shutdown vrf member evpn-tenant-1 ip address 20.0.0.1/24 fabric forwarding mode anycast-gateway

EVPN Multi-Tenancy and VNI Types

Tenant A (VRF A)

VLAN B

Layer-2 VNI B’

VLAN C

Layer-2 VNI C’

VLAN A

Layer-3 VNI A’

1 Layer-3 VNI per Tenant (VRF) for routing

VNI X’ is used for routed packets

1 Layer-2 VNI per Layer-2 segment

Multiple Layer-2 VNIs per tenant VNI A’ and B’ are used for bridged

packets

Multi-tenant Packet Forwarding in Symmetric IRB

VTEP-1 VTEP VTEP VTEP-2

Host 1H-MAC-1H-IP-1VNI-AL3-VNI-AVRF-A

Host 2H-MAC-2H-IP-2VNI-BL3-VNI-AVRF-A

IP Transport Network

S-MAC: H-MAC-1

D-MAC: H-MAC-2

S-MAC: Router-MAC-1

D-MAC: Router-MAC-2

VNI: L3-VNI-A

Use VTEP addresses in the outer header to route encapsulated packets to the egress VTEP

S-MAC: Router-MAC-1D-MAC: Router-MAC-4

VNI: L3 –VNI-A Use L3-VNI to identify the tenant VRF

Tenant AVRF-AL3-VNI-A

H-IP-2

Tenant BVRF-BL3-VNI-B

Tenant CVRF-CL3-VNI-C

Tenant AVRF-AL3-VNI-A

H-IP-2

S-MAC: H-MAC-1

D-MAC: H-MAC-2

S-IP: H-IP-1

D-IP: H-IP-2

Symmetric IRB has optimal utilization of ARP and MAC tables on a VTEP. Symmetric IRB scales better for end hosts. Symmetric IRB scales better in terms of the total number of VNIs a VXLAN overlay

network can support.

Multi-vendor interoperability: Some vendors implemented Asymmetric IRB. It’s been agreed upon among multiple vendors that Symmetric IRB is the ultimate

solution. Cisco implemented Symmetric IRB. Cisco will introduce backward compatibility with asymmetric IRB by adding the

support for it.

Symmetric IRB vs Asymmetric IRB

Remote host route learning through MP-BGP

Optimal VXLAN Routing with Symmetric IRB and Anycast Gateway

LeafVTEP-4

VTEP-3

VTEP-2

VTEP-1

VTEP-5

SVI 100

SVI 200

SVI 100

SVI 200

Host-based fabric routing and bridging with optimal and flexible VXLAN VNI placement Every VTEP is an anycast

gateway for its VXLAN subnets. Anycast gateway VTEPs share: The same virtual Gateway

IP The same virtual MAC

address

Distributed inter-vxlan host-based

routing on local VTEP

Host IP Port

IP-A Eth1/1

IP-B VTEP-4

Host IP VTEP

IP-A VTEP-2

IP-B Eth1/1

VLAN 100SVI VLAN 100VNI 5100Host IP-A

VLAN 200SVI VLAN 200VNI 5200Host IP-B

SVI 100

SVI 200

SVI300

EVPN VXLAN Fabric External Routing

VTEP-1

VTEP-2

VTEP-3

VTEP-4

Host-1 (VLAN 100) External Network or Internet

VTEP-6

VTEP-5

Routing Protoco

l of Choice

IP Routing

Global Default VRF

Or User Space VRFs

VXLAN Overlay

EVPN VRF/VRFs Space

Border Leaf

VXLAN Overlay

EVPN MP-BGP

Host 1MAC_1/ IP_1

EVPN VXLAN Fabric External Routing (Cont’d)

Tenant VRF or Default VRF

VRF OSPF Process

Overlay

EVPN VRF A

Overlay

EVPN VRF B

Overlay

EVPN VRF C

For Layer 3 interfaces, use one per overlay VRF instance. The routing protocol neighbor is in the EVPN VRF address family.Layer 3 interfaces on the external devices can be in either tenant VRF instances or the global default VRF instance.

External Router

VRFC Interface-Type Options:

Physical Routed Ports Subinterfaces VLAN SVIs over Trunk Ports

Border Leaf

VXLAN EVPN Features

ARP Suppression in MP-BGP EVPN

Host-1 in VLAN 10 sends an ARP request for Host-2’s IP-2 address.

Host 1MAC1IP 1

VLAN 10VXLAN 5000

Host 2MAC2IP 2

VLAN 10VXLAN 5000

S2 S3 S4

VTEP-1 intercepts the ARP request and checks in its ARP suppression cache. It finds a match for IP-2 in VLAN 10 in its ARP suppression cache.* 2

IP Address

MAC Address

VLAN Physical Interface Index (ifindex)

IP-1 MAC-1 10 E1/1 Local

IP-2 MAC-2 10 Null Remote

IP-3 MAC-3 10 Null Remote

VTEP-1 sends an ARP response back to Host-1 with MAC-2.*

Host-1 learns the IP-2 and MAC-2 mapping.

* If VTEP-1 doesn’t have a match for IP-2 in its ARP suppression cache table, it will flood the ARP request to all other VTEPs in this VNI

ARP suppression reduces network flooding due to host learning

interface nve1

no shutdown

source-interface loopback0

host-reachability protocol bgp

member vni 20000

suppress-arp

mcast-group 239.1.1.1

member vni 21000

suppress-arp

mcast-group 239.1.1.2

member vni 39000 associate-vrf

member vni 39010 associate-vrf

ARP Suppression in MP-BGP EVPN (Cont’d) ARP Suppression can be enabled on a per-VNI basis under the interface nve1

configuration.

S2 S3 S4

n9396-vtep-1.sakommu-lab.com# sh ip arp suppression topo-info

ARP L2RIB Topology informationTopo-id ARP-suppression mode100 L2 ARP Suppression200 L2/L3 ARP Suppression201 L2/L3 ARP Suppression

Head-end Replication

Host-1 sends BUM traffic into the VXLAN VNI

VTEP-1 receives the overlay BUM traffic, encapsulates the packets into unicast VXLAN packets, sends one copy to each remote VTEP peer in the same VXLAN VNI

Multicast-Free

Underlay

Head-end Replication (aka. Ingress replication):

Eliminate the need for underlay multicast to transport overlay BUM traffic

Local Scoping of VLANs –ToR Local

---- N-Way ----

VTEP-1 VTEP-2 VTEP-3 VTEP-n

Host-1 (VLAN 10) Host-2 (VLAN 60)

16 million possible VNIs global scope

Host -1MAC_1

/10.1.1.1

VLANS are Locally Scoped at Top of Rack/ Gateway

VNI 5000 maps to VLAN 10

Possible VLAN IDs 1-4K

Host -2MAC_2/10.1.

VLANS are Locally Scoped at Top of Rack/ Gateway

VNI 5000 maps to VLAN 60

Possible VLAN IDs 1-4K

Local Scoping of VLANs – Port Local

---- N-Way ----

VTEP-1 VTEP-2 VTEP-3 VTEP-n

Host-1 (VLAN 10) Host-2 (VLAN 60)

16 million possible VNIs global scope

Host -1MAC_1

/10.1.1.1

VNI 10000 maps to (E1/1, VLAN 10)

VLANS are Locally Scoped VLAN to VNI mapping is per-

port significantPossible VLAN IDs 1-4K

(Eth1/1, Vlan10) => VNI 10000 (Eth1/2, Vlan10) => VNI 10001 (Eth1/2, Vlan11) => VNI 10000

Host -2MAC_2/10.1.

VNI 10000 maps to (E1/2, VLAN 10)

VLANS are Locally Scoped VLAN to VNI mapping is per-

port significantPossible VLAN IDs 1-4K

EVPN Control Plane Advantages

A multi-tenant fabric solution with host-based forwarding

Industry standard protocol for multi-vendor interoperability

Build-in multi-tenancy support Leverage MP-BGP to deliver VXLAN with L3VPN characteristics

Truly scalable with protocol-driven learning Host MAC/IP address advertisement through EVPN MP-BGP

Fast convergence upon host movements or network failures MP-BGP protocol driven re-learning and convergence

Upon host movement, the new VTEP will send out a BGP update to advertise the new

location of the host

EVPN Control Plane Advantages (Cont’d)

Optimal traffic forwarding supporting host mobility Anycast IP gateway for optimal forwarding for host generated traffic No need for hair-pinning to to reach the IP gateway

ARP suppression Minimize ARP flooding in overlay

Head-end Replication with dynamically learned remote-VTEP list Head-end replication enables multicast-free underlay network Dynamically learned remote-VTEP list minimizes the operational overhead of head-end

replication

VTEP peer authentication via MP-BGP authentication Added security to prevent rogue VTEPs or VTEP spoofing

A multi-tenant fabric solution with host-based forwarding

Ingress Replication for Control Plane EVPN

VxLAN Overview

VxLAN provides a way to extend Layer2 extension across Layer 3 infrastructure using MAC-in-UDP encapsulation and tunneling.

VxLAN uses a 24-bit Virtual Network ID (VNID), allowing a maximum of 16 million VxLAN segments to coexist in the same administrative domain.

VxLAN packets are IP routed through the underlying network based on its its Layer 3 header.

VXLAN Peer and Host Learning Options

Host Learning

Data-Plane Control-Plane

Multicast

UnicastVlan 2 vn-segment 4098Interface nve 1 host-reachability protocol bgp member vni 4098 ingress-replication protocol bgp

Vlan 2 vn-segment 4098Interface nve 1member vni 4098 ingress-replication protocol static

Vlan 2 vn-segment 10000Interface nve 1 host-reachability protocol bgp member vni 4098 mcast-group 225.1.1.1

Vlan 2 vn-segment 4098Interface nve 1member vni 10000 mcast-group 225.1.1.1

Flood and LearnPeer Learning: DP

EVPN-MulticastPeer Learning: BGP-RnH

Static Ingress-ReplicationPeer Learning: CLI

EVPN Ingress-ReplicationPeer Learning: BGP-IMET

Flooding of Broadcast/Unknown-unicast/Multicast (BUM) packets across underlying core network: Option 1: Use multicast IP core

Each VNI is mapped to a multicast group. For packets coming on access side interfaces miss MAC lookup, they are encapsulated with VNI group address as DIP and sent out along the multicast tree to core

Option 2: Use Ingress Replication (IR) Some customers would want to avoid using multicast in their core Remote VTEPs are automatically learnt through Inclusive Multicast Ethernet Tag (IMET)

routes. When missing MAC lookup, a packet is replicated to all remote VTEPs in the VNI, each is

encapsulated with one VTEP IP as unicast Destination IP Support multiple VTEPs per VNI and a VTEP in multiple VNIs

VXLAN Flooding with BGP-EVPN

VTEP-2

VTEP-1

VTEP-3

Host-1

Host-3

Ethernet 1/1: 102.1.1.1Loopback0 : 202.1.1.1VxLAN L2 Gateway

IP Network

Router-2

Router-3

Hoist-4

Router-1

feature nv overlay feature vn-segment-vlan-based

interface nve1 no shutdown source-interface loopback0 member vni 4098 ingress-replication protocol bgp member vni 4099 ingress-replication protocol bgp

Host-2

Hoist-5

VLAN 2 VLAN 3 VLAN 2

VLAN 2

VLAN 3

feature nv overlay feature vn-segment-vlan-based

vlan 2 vn-segment 4098

interface nve1 no shutdown source-interface loopback0 member vni 4098 ingress-replication protocol bgp

Topology and configuration

VXLAN EVPN Configuration

VXLAN with MP-iBGP EVPN Configuration

LeafVTEPVTEPVTEPVTEP VTEP VTEP

SpineRRRR

Spine nodes are iBGP route reflector

feature nv overlay

feature vn-segment-vlan-based

feature bgp

nv overlay evpn

Enable VXLAN and MP-BGP EVPN Control Plane

Enable VXLAN

Enable VLAN-based VXLAN (the currently only mode)

Enable OSPF if it’s chosen to be the underlay IGP routing protocol

Enable VLAN SVI interfaces if the VTEP needs to be IP gateway and route for the VXLAN VLAN IP subnet.

Enable EVPN control plane for VXLAN

feature ospf

feature pim

feature interface-vlan

Other features may need to be enabled

Enable BGP

Enable IP PIM multicast routing in the underlay network

Initial configuration – Per Switch

Create VXLAN tenant VRFCreate a VXLAN Tenant VRF

Specify the Layer-3 VNI for VXLAN routing within the tenant VRF

Define VRF Route Target and import/export policies in address-family ipv4 unicast

Define VRF RD (route distinguisher)

Example to create a 2nd tenant VRF following the above steps

EVPN Tenant VRF

interface Vlan3900 description l3-vni-for-tenant-1-routing no shutdown vrf member evpn-tenant-1 vrf context evpn-tenant-1 vni 39000 rd auto address-family ipv4 unicast route-target import 39000:39000 route-target export 39000:39000 route-target both auto evpn

Configure Layer-3 VNI per EVPN Tenant VRF Routing Instant

Create the VLAN for the Layer-3 VNI. One Layer-3 VNI per tenant VRF routing instance

Create the SVI interface for the Layer-3 VNIPut this SVI interface into the tenant VRF context

Associate the Layer-3 VNI with the tenant VRF routing instance.

Layer-3 VNI Per Tenant for EVPN Routing

interface Vlan3901 description l3-vni-for-tenant-2-routing no shutdown vrf member evpn-tenant-2

Configure Layer-3 VNI per EVPN Tenant VRF Routing Instant

Define Layer-3 VNI for a 2nd tenant following the same steps in the previous slide

EVPN Layer-3 VNI Per Tenant for Routing Instance

Map VLANs to VXLAN VNIs and Configure their MP-BGP EVPN Parameters

Map VLAN to VXLAN VNI

evpn vni 20000 l2 rd auto route-target import auto route-target export auto vni 21000 l2 rd auto route-target import auto route-target export auto

Under EVPN configuration, define RD and RT import/export policies for each Layer-2 VNIs

EVPN Layer-2 Network VXLAN VNI

Create SVI interface for Layer-2 VNIs for VXLAN routing

Create SVI interface for a Layer-2 VNI.Associate it with the tenant VRF.

Enable distributed anycast gateway for this VLAN/VNI

All VTEPs for this VLAN/VNI should have the same SVI interface IP address as the distributed IP gateway.

EVPN Layer-2 Network VXLAN VLAN SVI Interface

fabric forwarding anycast-gateway-mac 0002.0002.0002

Configure virtual IP addressAll VTEPs for this VLAN should have the same virtual IP address

Configure distributed gateway virtual MAC addressOne virtual MAC per VTEPAll VTEPs should have the same virtual MAC address

Enable distributed gateway for this VLAN

EVPN Distributed Gateway

interface nve1 no shutdown source-interface loopback0 host-reachability protocol bgp member vni 20000 suppress-arp mcast-group 239.1.1.1 member vni 21000 suppress-arp mcast-group 239.1.1.2 member vni 39000 associate-vrf member vni 39010 associate-vrf

interface loopback 0 ip address 10.1.1.11/32 ip ospf network point-to-point ip router ospf 1 area 0.0.0.0 ip pim sparse-mode

Configure VXLAN tunnel interface nve1

Specify loopback0 as the source interface

Define BGP as the mechanism for host reachability advertisement

Add Layer-3 VNIs, one per tenant VRF

Associate tenant VNIs to the tunnel interface nve1Define the mcast group on a per-VNI basisEnable arp suppression on a per-VNI basis

The loopback interface to source VXLAN tunnels

VXLAN Tunnel Interface Configuration

router bgp 100 router-id 10.1.1.11 log-neighbor-changes address-family ipv4 unicast address-family l2vpn evpn neighbor 10.1.1.1 remote-as 100 update-source loopback0 address-family ipv4 unicast address-family l2vpn evpn send-community extended neighbor 10.1.1.2 remote-as 100 update-source loopback0 address-family ipv4 unicast address-family l2vpn evpn send-community extended

vrf evpn-tenant-1 address-family ipv4 unicast advertise l2vpn evpn vrf evpn-tenant-2 address-family ipv4 unicast advertise l2vpn evpn

Address-family ipv4 unicast for prefix-based routing

Define MP-BGP neighbors. Under each neighbor define address-family ipv4 unicast and l2vpn evpn

Under address-family ipv4 unicast of each tenant VRF instance, enable advertising EVPN routes

Send extended community in l2vpn evpn address-family to distribute EVPN route attributes

Address-family l2vpn evpn for evpn host routes

MP-BGP Configuration on VTEP

router bgp 100 router-id 10.1.1.1 log-neighbor-changes address-family ipv4 unicast address-family l2vpn evpn retain route-target all template peer vtep-peer remote-as 100 update-source loopback0 address-family ipv4 unicast send-community both route-reflector-client address-family l2vpn evpn send-community both route-reflector-client neighbor 10.1.1.11 inherit peer vtep-peer neighbor 10.1.1.12 inherit peer vtep-peer neighbor 10.1.1.13 inherit peer vtep-peer neighbor 10.1.1.14 inherit peer vtep-peer

Address-family ipv4 unicast for prefix-based routing

iBGP RR client peer template

Send both standard and extended community in address-family l2vpn evpn

Send both standard and extended community in address-family ipv4 unicast

Address-family l2vpn evpn for EVPN vxlan host routesRetain route-targets attributes

MP-BGP Configuration on iBGP Route Reflector

VXLAN Fabric with MP-eBGP EVPN

VTEPVTEPVTEPVTEP VTEP VTEP

AS 6500

AS 65000

[BGP configuration on a spine switch as in Figure

16 design]

route-map permit-all permit 10

set ip next-hop unchanged

router bgp 65000

router-id 10.1.1.1

address-family ipv4 unicast

redistribute direct route-map permitall

address-family l2vpn evpn

nexthop route-map permit-all

retain route-target all

neighbor 192.167.11.2 remote-as 65001

send-community extended

route-map permit-all out

Set next-hop policy to not change the next-hop attributes.

Retain routes with all route targets when advertising the EVPN BGP routes to eBGP peers.

Set outbound policy to advertise all routes to this eBGP neighbor.

VXLAN Fabric MP-eBGP EVPN (Cont’d)

[BGP configuration on a spine switch as in Figure 16 design]

route-map permit-all permit 10

set ip next-hop unchanged

router bgp 65000

router-id 10.1.1.1

redistribute direct route-map permitall

nexthop route-map permit-all

retain route-target all

Set next-hop policy to not change the next-hop attributes.

Retain routes with all route targets when advertising the EVPN BGP routes to eBGP peers.

Set outbound policy to advertise all routes to this eBGP neighbour.

EVPN VXLAN Fabric External Routing

LeafVTEPVTEPVTEPVTEP VTEP VTEP

SpineRRRR

Global Default VRF

Or User Space VRFs

Border Leaf

VXLAN Overlay

EVPN MP-BGP

IP Routing

Routing Protocol

of Choice

VXLAN Overlay

EVPN VRF/VRFs Space

EVPN VXLAN External Routing with BGP

IP Routing in the Default VRF

Instance

SpineRRRR

VXLAN Overlay

EVPN VRF Instance Space VTEP

Border Leaf interface Ethernet2/9.10 mtu 9216 encapsulation dot1q 10 vrf member evpn-tenant-1 ip address 30.10.1.1/30

interface Ethernet1/50.10 mtu 9216 encapsulation dot1q 10 ip address 30.10.1.2/30

Router bgp 100 vrf evpn-tenant-1 address-family ipv4 unicast network 20.0.0.0/24 neighbor 30.10.1.2 remote-as 200 address-family ipv4 unicast prefix-list outbound-no-hosts out

router bgp 200 address-family ipv4 unicast network 100.0.0.0/24 network 100.0.1.0/24 neighbor 30.10.1.1 remote-as 100 address-family ipv4 unicast

Sample Configuration

router bgp 100 router-id 10.1.1.16 log-neighbor-changes address-family ipv4 unicast address-family l2vpn evpn neighbor 10.1.1.1 remote-as 100 update-source loopback0 address-family ipv4 unicast address-family l2vpn evpn send-community extended neighbor 10.1.1.2 remote-as 100 update-source loopback0 address-family ipv4 unicast address-family l2vpn evpn send-community extended vrf evpn-tenant-1 address-family ipv4 unicast network 20.0.0.0/24 neighbor 30.10.1.2 remote-as 200 address-family ipv4 unicast prefix-list outbound-no-hosts out

ip prefix-list outbound-no-hosts seq 5 deny 0.0.0.0/0 eq 32 ip prefix-list outbound-no-hosts seq 10 permit 0.0.0.0/0 le 32

On the VXLAN Border Leaf

The eBGP neighbor is on the outside.It’s in address-family ipv4 unicast of the tenant VRF routing instance.

For better scalability, apply prefix-list to filter out /32 IP host routes. Advertise prefix routes only to the external eBGP neighbor.

EVPN VXLAN External Routing with BGPSample Configuration – On the Border Leaf

EVPN VXLAN External Routing with BGPn9396-vtep-1# sh ip bgp vrf evpn-tenant-1 100.0.0.0 BGP routing table information for VRF evpn-tenant-1, address family IPv4 UnicastBGP routing table entry for 100.0.0.0/24, version 70Paths: (1 available, best #1)Flags: (0x08041a) on xmit-list, is in urib, is best urib route vpn: version 75, (0x100002) on xmit-list

Advertised path-id 1, VPN AF advertised path-id 1 Path type: internal, path is valid, is best path, no labeled nexthop Imported from unknown dest AS-Path: NONE, path sourced internal to AS 10.1.1.16 (metric 3) from 10.1.1.1 (10.1.1.1) Origin IGP, MED not set, localpref 100, weight 0 Received label 39000 Extcommunity: RT:100:39000 ENCAP:8 Router MAC:6412.2574.6ae7 Originator: 10.1.1.16 Cluster list: 10.1.1.1

VRF advertise information: Path-id 1 not advertised to any peer

VPN AF advertise information: Path-id 1 not advertised to any peern9396-vtep-1# n9396-vtep-1# sh ip route vrf evpn-tenant-1 100.0.0.0/24IP Route Table for VRF "evpn-tenant-1"'*' denotes best ucast next-hop'**' denotes best mcast next-hop'[x/y]' denotes [preference/metric]'%<string>' in via output denotes VRF <string>

100.0.0.0/24, ubest/mbest: 1/0 *via 10.1.1.16%default, [200/0], 01:01:14, bgp-100, internal, tag 100 (evpn)segid: 0x9858 tunnelid: 0xa010110 encap: 1 n9396-vtep-1#

This is the external route.

The tenant is VRF L3 VNI.

The next hop is the VTEP address of the border leaf.

10.1.1.16 is the BGP router ID of the border leaf. 10.1.1.1 is the spine route reflector.

This is the iBGP route. The next hop is the VTEP address of the border leaf.

EVPN VXLAN External Routing with OSPFSample Configuration

IP Routing in the Default VRF

Instance

SpineRRRR

VXLAN Overlay

EVPN VRF and VRF Instance Space VTEP

Border Leaf

interface Ethernet2/9.10 mtu 9216 encapsulation dot1q 10 vrf member evpn-tenant-1 ip address 30.10.1.1/30 ip router ospf 1 area 0.0.0.0

interface Ethernet1/50.10 mtu 9216 encapsulation dot1q 10 ip address 30.10.1.2/30 ip router ospf 1 area 0.0.0.0

route-map permit-bgp-ospf permit 10route-map permit-ospf-bgp permit 10router ospf 1 router-id 10.1.1.16 vrf evpn-tenant-1 redistribute bgp 100 route-map permit-bgp-ospfrouter bgp 100 router-id 10.1.1.16 log-neighbor-changes address-family ipv4 unicast address-family l2vpn evpn retain route-target all vrf evpn-tenant-1 address-family ipv4 unicast advertise l2vpn evpn redistribute ospf 1 route-map permit-ospf-bgp

EVPN VXLAN External Routing with OSPF

ip prefix-list bgp-ospf-no-hosts seq 5 permit 0.0.0.0/0 eq 32 route-map permit-bgp-ospf deny 5 match ip address prefix-list bgp-ospf-no-hosts route-map permit-bgp-ospf permit 10route-map permit-ospf-bgp permit 10

router ospf 1 router-id 10.1.1.16 vrf evpn-tenant-1 redistribute bgp 100 route-map permit-bgp-ospf

router bgp 100 router-id 10.1.1.16 log-neighbor-changes address-family ipv4 unicast address-family l2vpn evpn retain route-target all neighbor 10.1.1.1 remote-as 100 update-source loopback0 address-family ipv4 unicast address-family l2vpn evpn send-community extended neighbor 10.1.1.2 remote-as 100 update-source loopback0 address-family ipv4 unicast address-family l2vpn evpn send-community extended vrf evpn-tenant-1 address-family ipv4 unicast advertise l2vpn evpn redistribute ospf 1 route-map permit-ospf-bgp

Redistribute BGP routes to OSPF. Filter out /32 IP host routes.

A BGP router will modify route targets in l2vpn evpn routes when it is an autonomous system boundary router. The original route target must be retained.

Redistribute OSPF to BGP. Advertise the redistributed routes to L2VPN EVPN.

Sample Configuration - The relevant configuration on the border leaf

EVPN VXLAN External Routing with OSPF

n9396-vtep-1# sh vrf evpn-tenant-1 detail VRF-Name: evpn-tenant-1, VRF-ID: 3, State: Up VPNID: unknown RD: 10.1.1.11:3 VNI: 39000 Max Routes: 0 Mid-Threshold: 0 Table-ID: 0x80000003, AF: IPv6, Fwd-ID: 0x80000003, State: Up Table-ID: 0x00000003, AF: IPv4, Fwd-ID: 0x00000003, State: Up

n9396-vtep-1# sh bgp l2vpn evpn rd 10.1.1.11:3 100.0.0.0BGP routing table information for VRF default, address family L2VPN EVPNRoute Distinguisher: 10.1.1.11:3 (L3VNI 39000)BGP routing table entry for [5]:[0]:[0]:[24]:[100.0.0.0]:[0.0.0.0]/224, version 396Paths: (1 available, best #1)Flags: (0x00001a) on xmit-list, is in l2rib/evpn

Advertised path-id 1 Path type: internal, path is valid, is best path, no labeled nexthop Imported from 10.1.1.16:3:[5]:[0]:[0]:[24]:[100.0.0.0]:[0.0.0.0]/120 AS-Path: NONE, path sourced internal to AS 10.1.1.16 (metric 3) from 10.1.1.1 (10.1.1.1) Origin IGP, MED not set, localpref 100, weight 0 Received label 39000 Extcommunity: RT:100:39000 ENCAP:8 Router MAC:6412.2574.6ae7 Originator: 10.1.1.16 Cluster list: 10.1.1.1

Path-id 1 not advertised to any peer

n9396-vtep-1#

The external route learned through MP-BGP EVPN is imported into the tenant VRF.

This is the Layer 3 VNI of the tenant VRF routing instance.

The next hop is the VTEP address of the border leaf.

The internal VTEPs learn the external routes through MP-BGP EVPN

Scalability Limits

Unicast Routing Verified Scalability Limits

Feature 9500 Series Verified Limit 9300 Series Verified Limit

BFD sessions (echo mode) 512 (IPv4 only)512 (IPv6 only)256 (IPv4) + 256 (IPv6)

256 (IPv4 only)256 (IPv6 only)128 (IPv4) + 128 (IPv6)

BGP neighbors 2000 (IPv4 only)2000 (IPv6 only)1000 (IPv4) + 1000 (IPv6)

EIGRP routes 20,000 20,000

EIGRP neighbors 360 (IPv4 only)360 (IPv6 only)180 (IPv4) + 180 (IPv6)

HSRP groups per interface or I/O module 490 250

IPv4 ARP 45,000 (default system routing mode)60,000 (max-host routing mode)

45,000

Unicast Routing

Unicast Routing Verified Scalability Limits (Cont’d)

IPv4 host routes 208,000 (default system routing mode)60,000 (max-host routing mode)

208,000 (default system routing mode)16,000 (ALPM routing mode)

IPv6 host routes 40,000 (default system routing mode)30,000 (max-host routing mode)

IPv6 ND 40,000 40,000

IPv4 unicast routes (LPM) 128,000 (default system routing mode)16,000 (max-host routing mode)128,000 with no IPv6 routes (64-bit ALPM routing mode)

IPv6 unicast routes (LPM) 20,000 (default system routing mode)4000 (max-host routing mode)80,000 with no IPv4 routes (64-bit ALPM routing mode)

7000 (6000 routes < /64, 1000 routes > /64) (default system routing mode)20,000 (ALPM routing mode)

IPv4 and IPv6 unicast routes (LPM) in 64-bit ALPM routing mode

x IPv6 routes and y IPv4 routes, where 2x +y <= 128,000

Not applicable

Unicast Routing

Unicast Routing Verified Scalability Limits (Cont’d)

MAC addresses 90,000 90,000

OSPFv2 neighbors 1000 256

OSPFv3 neighbors 300 256

VRFs 1000 1000

VRRP groups per interface or I/O module 250 250

Unicast Routing

Nexus 2000 Series Fabric Extenders (FEX) Verified Scalability Limits

Feature 9500 Series Verified Limit

9300 Series Verified Limit

Fabric Extenders and Fabric Extender server interfaces

Not applicable 16 and 768

VLANs per Fabric Extender Not applicable 2000 (across all Fabric Extenders)

VLANs per Fabric Extender server interface

Not applicable 75

Port channels Not applicable 500

Interfaces Verified Scalability Limits

Generic routing encapsulation (GRE) tunnels

Port channel links 32 8

SVIs 490 250

vPCs 275 100 (280 with Fabric Extenders)

Layer 2 Switching Verified Scalability Limits

MST instances 64 64

MST virtual ports 85,000 48,000

RPVST virtual ports 22,000 12,000

VLANs 4000 3900

VLANs in RPVST mode 500 500

Multicast Routing Verified Scalability Limits

IPv4 multicast routes 32,000 8000

Outgoing interfaces (OIFs) 40 (see CSCum58876) 40 (see CSCum58876)

Layer 2 Switching Verified Scalability Limits

MST instances 64 64

MST virtual ports 85,000 48,000

RPVST virtual ports 22,000 12,000

VLANs 4000 3900

VLANs in RPVST mode 500 500

Thank You

VXLAN Nexus 9000 Module 5 – MP-BGP EVPN @onecloudinc.com

Documents

VXLAN and BGP EVPN Configuration ... - topics-cdn.dell.com · 12.12.2012 · Notes, cautions, and warnings NOTE: A NOTE indicates important information that helps you make better

VCF on VxRail Multirack Deployment using BGP EVPN

SP Summit EVPN jichalou - Cisco IOS-XR EVPN · 2020. 7. 14. · EVPN Next generation network services ELINE ELAN ETREE DCI DC Fabric L3VPN VPWS VPLS P2MP VPLS VPLS / L3 VPN VXLAN

VXLAN Network with MP-BGP EVPN Control Plane...routing, hence affecting the choice of hardware platform. MP-BGP EVPN VXLAN Support on Cisco Nexus 9000 Series Switches The MP-BGP EVPN

Gluon Project with EVPN Solution - conference.cn · Project Proposal – Support BGP-EVPN SDN-X ONOS BGP-EVPN ... Vpn-Port-Binding VTN APP ... • Deployed E2E Gluon EVPN project

VXLAN BGP EVPN: TECHNOLOGY BUILDING BLOCKS....O -OSPF, IA -OSPF inter area, E1 - OSPF external type 1, E2 -OSPF external type 2, N1 -OSPF NSSA external type 1, N2 -OSPF NSSA external

Ansible EVPN/VXLAN Documentation - Read the Docs · PDF fileAnsible EVPN/VXLAN Documentation, ... ble projects to easily generate Overlay & Underlay conﬁguration. • Playbook to

Cisco VXLAN EVPN Multifabric Designyves-louis.com/DCI/wp-content/uploads/2020/04/C11-738503-00_Cis… · Virtual Extensible LAN (VXLAN) with Multiprotocol Border Gateway Protocol

Building Data Centre Networks with VXLAN BGP EVPN Data Centre Networks...EVPN –Ethernet VPN Control-Plane Data-Plane Multi-Protocol Label Switching (MPLS) draft-ietf-l2vpn-evpn EVPN

A Modern, Open, and Scalable Fabric: VXLAN EVPN

VXLAN - Amazon Simple Storage Service · Participants: Cisco, Juniper, Alcatel-Lucent, Ixia ... for VXLAN Layer-2 bridging • Nexus 9300 functioned as both EVPN iBGP route reflector

Day One: Deploying Optimized Multicast in EVPN/VXLAN€¦ · Day One: Deploying Optimized Multicast in EVPN/VXLAN by Vikram Nagarajan, Princy Elizabeth, and Himanshu Agarwal Part

Cisco VXLAN EVPN Multifabric Design White Paper · Cisco VXLAN EVPN Multifabric Design ... VXLAN Overview: Cisco Nexus 9000 Series Switches ... you can use a direct DWDM transport

Traffic Load Balancing in EVPN/VXLAN Networks · Load Balancing Scenarios ... This control-plane technology uses Multiprotocol BGP ... Traffic Load Balancing in EVPN/VXLAN Networks

Building and operating VXLAN BGP EVPN Fabrics with … MDS 9148S 9706 Scalable Capacity Virtualization & Cloud CapEx & OpEx Efficiencies Agility Security UNIFIED FABRIC Consistency

BGP-EVPN and SR DC Fabricd2zmdbbm9feqrf.cloudfront.net/2017/eur/pdf/BRKSPG-2509.pdf · BGP-EVPN and SR DC Fabric ... BGP EVPN All Active per-flow Load balancing Leaf ... • EVPN

with BGP Monitoring Protocol Monitoring a EVPN-VxLAN fabricdelaat/rp/2019... · A preﬁx, in EVPN, is exchanged as a type-5 (IP Prefix Route) route Carried along with the BGP UPDATE

ETHERNET VPN (EVPN) NEXT-GENERATION VPN FOR · PDF fileethernet vpn (evpn) next-generation vpn for ethernet services alastair johnson ... - alcatel-lucent: evpn, pbb-evpn, evpn-vxlan

AP231 EVPN & VXLAN · AP231 EVPN & VXLAN Daryl Wan Technical Marketing Engineering - Aruba Data Center Networking

VxLAN BGP-EVPN - archive.nanog.org · tagged traffic) comes from VLAN into VxLAN segment (encapsulation) or − The Ingress VxLAN packet egresses out an 802.1q tagged interface (de-