Brkdcn 2035 multi-x

VXLAN BGP EVPN based Multi-Pod, Multi-Fabric, Multi-Site

Max Ardica – Principal EngineerLukas Krattiger – Principal EngineerBRKDCN-2035

© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public

Cisco Spark

Questions? Use Cisco Spark to chat with the speaker after the session

1. Find this session in the Cisco Live Mobile App2. Click “Join the Discussion”3. Install Spark or go directly to the space4. Enter messages/questions in the space

How

cs.co/ciscolivebot#BRKDCN-2035Cisco Spark spaces will be available until July 3, 2017.


Who Are the Presenters?

Max ArdicaPrincipal Engineer - INSBU

Lukas KrattigerPrincipal Engineer – INSBU

@ccie21921

BRKDCN-2035 4


Session Objectives

At the end of the session, the participants should be able to:

� Articulate the different deployment options to interconnect VXLAN EVPN Networks (Multi-Pod vs. Multi-Fabric vs. Multi-Site)

� Understand the functionalities and specific design considerations associated to the new VXLAN Multi-Site architecture

Initial assumption:

� The audience already has a good knowledge of the VXLAN EVPN technology and its use to deploy modern Data Center Fabrics

BRKDCN-2035 5


Session Reference• BRKDCN-2304

• L4-L7 Service Integration in Multi-Tenant VXLAN EVPN Data Center Fabrics

• BRKDCN-3378• Building DataCenter Networks with VXLAN BGP-EVPN• Wednesday, Jun 28, 1:30 pm

• BRKDCN-2125• Overlay Management and Visibility with VXLAN• Thursday, Jun 29, 10:30 am

• BRKDCN-2342• Programmable Fabric Automation and Management with DCNM 10• Thursday, Jun 29, 1:00 p.m.

6BRKDCN-2035


Agenda� Introduction

� VXLAN EVPN Interconnect Evolution• Multi-Pod• Multi-Fabric• Multi-Site

� VXLAN EVPN Multi-Site Deep Dive• Walkthrough• Control- and Data-Plane• Deployment Considerations

� Conclusions and Q&A

Introduction


Layer 2 Domain ElasticityLocal LAN Fabric

Extended LAN fabric

VN-link notifications

IP MobilityOptimal Ingress and Egress

Routing

VM-awarenessVXLAN, DFA, ACI, VN-link

Storage ElasticitySAN Extensions

Network Service LocalizationAny service anywhere

OTV

OTV

OTV

OTV

Fabric ConsolidationUnified Fabric & I/ODevice Virtualization

Segmentation

Data Center Interconnect – DCI Model Connecting Virtualized Data Centers

Multi-tenancy/SegmentationSegment-IDs in VXLAN, LISP, FabricPath,

and OTV

Storage Solutions & Partners:FCIP, I/O Acceleration

EMC, NetApp

BRKDCN-2035 9


Back ThenYet Another Encapsulation

� Flood & Learn (Multicast-based)� Data-Plane only Yesterday

VXLAN for the Data Center – Intra-DC� Control-Plane� Active VTEP Discovery� Multicast and Unicast

Now!VXLAN for DCI – Inter-DC

� DCI Ready � ARP/ND caching/suppress� Multi-Homing� Failure Domain Isolation� Loop Protection

VXLAN Evolves as the Control Plane Evolves!

BRKDCN-2035 10


Back ThenVXLAN for Interconnecting Networks

BRKDCN-2035 11


Inter-X Connectivity

• Single Fabric with End-to-End Encapsulation

• Build Hierarchy in the Underlay – Flatten it in the Overlay

Multi-Pod

OverlayVTE

PVTE

PVTE

PVTE

P

Baremeta

l

Baremeta

l

Fabric #2

OverlayVTE

PVTE

P

Baremeta

l

Baremeta

l

VTEP

VTEP

Fabric #1EVPN Control-Plane Domain 1

EVPN Control-Plane Domain 2

Single Data-Plane – End-to-End

BGP EVPN

OverlayVTE

PVTE

PVTE

PVTE

P

Baremetal

Baremetal

Fabric #2

OverlayVTE

PVTE

P

Baremetal

Baremetal

VTEP

VTEP



Data-Plane Domain 1 Data-Plane Domain 2DCI Data-Plane

• Multiple Fabrics –Normalized through Ethernet

• Multiple Fabrics Interconnect using DCI (Layer 2 and Layer 3)

Multi-Fabric

Data-Plane Domain 1 Data-Plane Domain 2

DCI Data-Plane

OverlayVTE

PVTE

PVTE

PVTE

P

Baremetal

Baremetal

Fabric #2

OverlayVTE

PVTE

P

Baremetal

Baremetal

VTEP

VTEP



BGP EVPN

• Multiple Fabrics with Integrated DCI

• Integrated DCI –Scaling within and between Fabrics

• The Happy Place -

Multi-Site

BRKDCN-2035 12

VXLAN EVPN Interconnect Evolution

Multi-Pod


VXLAN EVPN – Single Pod / Single Fabric

SpineSpine Spine Spine

VTEP VTEPVTEP VTEPVTEP VTEP VTEP

Pod 1

VTEP VTEP

External Network

BRKDCN-2035 15


VXLAN EVPN – Multi-Pod



Pod 1

VTEP VTEP



Pod n

VTEP VTEP

Underlay Extension

BRKDCN-2035 16


� Single Overlay Domain – End-to-End Encapsulation

� Single Overlay Control-Plane Domain – End-to-End EVPN Updates

� Single Underlay Domain End-to-End

� Single Replication Domain for BUM

� Single VNI Administrative Domain

Multi-Pod Characteristics – ”The Single”

Building Underlay Hierarchies – Non Hierarchical Overlay

BRKDCN-2035 17


Multi-Pod – End-to-End Encapsulation



Pod 1

VTEP VTEP



Pod n

VTEP VTEP

Underlay Extension

Overlay

Baremetal Baremetal

Unicast

VTEP10.1.1.1

VTEP10.2.2.7

BRKDCN-2035 18


Multi-Pod – BUM Replication



Pod 1

VTEP VTEP



Pod 2

VTEP VTEP

Underlay Extension

Overlay

Baremetal

BUM

BRKDCN-2035 19


� Single Overlay Domain – End-to-End Encapsulation• Scaling the VXLAN EVPN Network

� Single Overlay Control-Plane Domain – End-to-End EVPN Updates• Overlay Control-Plane Update Propagation

� Single Underlay Domain End-to-End• Network must be extended in Underlay (VTEP to VTEP reachability)

� Single Replication Domain for BUM• One BUM flooding domain through out all connected Pods

Multi-Pod Challenges – ”The Single”

BRKDCN-2035 20

Multi-Fabric


VXLAN EVPN – Multi-Fabric



Fabric 1

VTEP VTEP



Fabric 2

VTEP VTEP

Underlay No ExtensionL2 DCI L2 DCI

L3 DCI L3 DCIL2 DCI L2 DCI

BRKDCN-2035 22


• Separate Overlay Domains –Independent L2 and L3 DCI (complexity)

• Separate Overlay Control-Plane Domains – Manual Configuration

• Separate Underlay Domains - Isolated

• Separate Replication Domains for BUM – Independent BUM transport/DCI

• Dedicated Border Leaf – no local End-Point Attachment

Multi-Fabric Characteristics – ”The Separate”

Underlay Isolation – Separate DC Interconnection

BRKDCN-2035 23




Fabric 1

VTEP VTEP



Fabric n

VTEP VTEP

Underlay No ExtensionL2 DCI L2 DCI

L3 DCI L3 DCIL2 DCI L2 DCI

Multi-Fabric – End-to-End Encapsulation

Overlay Site 1 Overlay Site n

L2 DCI

Baremetal Baremetal

Unicast

VLAN Hand-Off VRF-Lite Hand-Off

BRKDCN-2035 24

VXLAN EVPN Multi-Site

Multi-Site Walkthrough


VXLAN EVPN – Multi-Site



Site 1

VTEP VTEP



Site n

VTEP VTEP

No Underlay Extension

BGW BGW BGW BGW

BRKDCN-2035 27


� Multiple Overlay Domains – Interconnected & Controlled

� Multiple Overlay Control-Plane Domains – Interconnected & Controlled

� Multiple Underlay Domains - Isolated

� Multiple Replication Domains for BUM – Interconnected & Controlled

� Multiple VNI Administrative Domains – Phase 2

Multi-Site Characteristics – ”The Multiple”

Underlay Isolation – Overlay Hierarchies

BRKDCN-2035 28


Multi-Site – Hierarchical Overlay Domains



Site 1

VTEP VTEP



Site n

VTEP VTEP


Overlay Multi-Site

Baremetal Baremetal

Unicast

BGW BGW BGW BGW

BRKDCN-2035 29


Multi-Site – Underlay Isolation



Site 1

VTEP VTEP



Site n

VTEP VTEP


VTEP10.1.1.1

Border (VIP)10.1.1.111


Site 1 Underlay Routing Table

Leaf:10.1.1.110.1.1.210.1.1.310.1.1.410.1.1.510.1.1.610.1.1.7

Border:10.1.1.10110.1.1.10210.1.1.111

VTEP10.2.2.7

Site n Underlay Routing Table

Leaf:10.2.2.110.2.2.210.2.2.310.2.2.410.2.2.510.2.2.610.2.2.7

Border:10.2.2.10110.2.2.10210.2.2.222

BGW BGW BGW BGWBorder (PIP)10.1.1.101

Border (PIP)10.1.1.102



BRKDCN-2035 30


Inter Site Network

Multi-Site – Inter Site Network



Site 1

VTEP VTEP



Site n

VTEP VTEP

VTEP10.1.1.1



Inter-Site NetworkRouting TableBorder Site1:10.1.1.10110.1.1.10210.1.1.111

Border Site2:10.2.2.10110.2.2.10210.2.2.222

VTEP10.2.2.7

BGW BGW BGW BGWBorder (PIP)10.1.1.101




BRKDCN-2035 31

Border Gateways Deployment Considerations


Border Gateways Deployment Considerations

Site 1

VTEP

BGWVTEP

BGWVTEP

BGWVTEP

BGW

Site 1

VTEP

BGWVTEP

BGW

� Border Gateways used for two main functions:1. Interconnecting each site to the Inter-Site network (for

East-West traffic flows)2. Connecting each site to the external Layer 3 domain

(for North-South traffic flows)

� May also be used to connect End-Points and/or network service nodes (FWs, ADCs)

� Two deployment models supported:1. Anycast Border Gateways2. VPC Border Gateways

Anycast Border Gateways

VPC Border Gateways

BRKDCN-2035 33

Anycast Border Gateways


Site 1

Anycast Border Gateway (1)Anycast Border Gateway� Up to 4 Border Gateways� Border Gateway

• Deploying at Leaf – 7.0(3)I7(1)• Deploying at Spine – 7.0(3)I7(2)

VTEP

BGWVTEP

BGWVTEP

BGWVTEP

BGW

BRKDCN-2035 35


Site 1

Anycast Border Gateway (2)Anycast Border Gateway� Common Virtual IP (VIP) across BGW

• VIP is used for Intra- and Inter-Site Communication

• VIP for communication between the Border Gateways in different Sites

• VIP for communication between Border Gateway and Leaf within a Site

� Individual Primary IP (PIP) per BGW• Used for Broadcast, Unknown Unicast and

Multicast (BUM) replication• PIP for communication with Single-Homed

End-Points (routed only), intra- and inter-Site

VTEP

BGWVTEP

BGWVTEP

BGWVTEP

BGW

Border VIP10.1.1.111


PIP-BGW110.1.1.101

PIP-BGW210.1.1.102

PIP-BGW310.1.1.103

PIP-BGW410.1.1.104

BRKDCN-2035 36


Site 1

Anycast Border Gateway (3)Anycast Border Gateway� Per-VNI Designated Forwarder (DF) election

• Each BGW can serve as DF for a single or a set of Layer-2 VNI

• DF election and assignment is automatic� Using BGP EVPN Route Type 4 for DF election

• Operator Managed Assignment (Type: 00)• Six Octet Site Identifier (System MAC:

00:00:00:00:00:01)• Multi-Site Discriminator (Ethernet-Segment:

00:00:07)• Originators IP Address (PIP): 10.1.1.101• Layer-2 VNI: 30010

VTEP

BGWVTEP

BGWVTEP

BGWVTEP

BGW

Spine

RRSpine

RR

BGP EVPN

Type: 00System MAC: 00:00:00:00:00:01Ethernet Segment: 00:00:074 IP: 10.1.1.101

VNI: 30010

DF30010

DF30099

DF30012

DF30011

BRKDCN-2035 37

VPC Border Gateways


Site 1

VPC Border Gateway (1)VPC Border Gateway� 2 Border Gateways� Border Gateway

• Using a Leaf – 7.0(3)I7(2)VTEP

BGWVTEP

BGW

BRKDCN-2035 39


Site 1

VPC Border Gateway (2)VPC Border Gateway� Common Virtual IP (VIP) across BGW

• VIP is used for Intra- and Inter-Site Communication

• VIP for communication between the Border Gateways in different Sites

• VIP for communication between Border Gateway and Leaf within a Site

� Individual Primary IP (PIP) per BGW• Used for Broadcast, Unknown Unicast and

Multicast (BUM) replication• PIP for communication with Single-Homed

End-Points, intra- and inter-Site

VTEP

BGWVTEP

BGW



PIP-BGW110.1.1.101

PIP-BGW210.1.1.102

BRKDCN-2035 40


Site 1

VPC Border Gateway (3)VPC Border Gateway� VPC-based Designated Forwarder Election� Per-Site Designated Forwarder (DF) election

• Using same approach as in VPC• Best Path to Rendezvous-Point or VPC

Primary Node

VTEP

BGWVTEP

BGW

DF

BRKDCN-2035 41


Site 1

VPC Border Gateway (4)VPC Border Gateway� Single- or Dual-Homed End-Points

• Services Appliance (i.e. Firewall, ADC etc.)• Physical or Virtual Servers

� Advertised and Reachable through Virtual IP Address (VIP)• Intra-Site: Leaf nodes use VIP to reach End-

Points connected to Border Gateways• Inter-Site: Remote Border Gateways use VIP

to reach End-Points connected to Border • Traffic potentially traverses VPC Peer-Link

VTEP

BGWVTEP

BGW

VTEPType MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.

2 0000.3010.1101/48 30010, 65599:30010 192.168.10.101/32 50001, 65599:50001 10.1.1.111

2 0000.3010.1102/48 30010, 65599:30010 192.168.10.102/32 50001, 65599:50001 10.1.1.111



ADC0000.3010.1102192.168.10.102

ADC

ADC0000.3010.1101192.168.10.101

ADC

BRKDCN-2035 42

Multi-Site Control Plane


Control Plane Deployment Considerations

� Both MP-eBGP or MP-iBGP peering supported intra-Site between leaf nodes

� Only MP-eBGP EVPN sessions supported inter-Sites Æ mandates that each site is part of a separate AS

� Full mesh of MP-eBGP EVPN adjacencies only currently supported across sites • Recommended to deploy a couple of Route-Servers in the Inter-Site network when 3 or

more sites are deployed• Route-Servers only perform control plane functions (“eBGP Route-Reflectors”)• Need to ensure that Route-Servers offer support for Route Type 4 EVPN routes,

required for DF election

BRKDCN-2035 44


Fabric

DCI

Multi-Site – Overlay Control-Plane (L3Core)

Spine SpineVXLAN EVPN

Site1

VTEP VTEP VTEP VTEP

VTEP VTEP….Spine Spine

VXLAN EVPN

Site2

VTEP VTEP VTEP VTEP

VTEP VTEP….BGW BGW BGW BGW

RR RRiBGP-EVPN iBGP-EVPN

DC Core(Layer-3 Unicast)

BRKDCN-2035 45


Fabric

DCI



Site1

VTEP VTEP VTEP VTEP


VXLAN EVPN

Site2

VTEP VTEP VTEP VTEP




RS Route Server (eBGP ”Route Reflector”) RS

BRKDCN-2035 46


Fabric

DCI



RS


Site1

VTEP VTEP VTEP VTEP


VXLAN EVPN

Site2

VTEP VTEP VTEP VTEP



RS – Route Server (eBGP ”Route Reflector”) BRKDCN-2035 47


Fabric

DCI

Multi-Site – Overlay Control-Plane (L3Core, no RS)



Site1

VTEP VTEP VTEP VTEP


VXLAN EVPN

Site2

VTEP VTEP VTEP VTEP



RS – Route Server (eBGP ”Route Reflector”)

eBGP-EVPN

BRKDCN-2035 48


Fabric

DCI

Multi-Site – Overlay Control-Plane


RS


Site1

VTEP VTEP VTEP VTEP


VXLAN EVPN

Site2

VTEP VTEP VTEP VTEP


RR RRiBGP-EVPN

Host10000.3010.1101192.168.10.101

Host30000.3010.1102192.168.10.102

Host20000.3020.2101192.168.20.101

VRFTenant1

L3VNI: 50001Route-Target: 65501:50001

VRFTenant1


L2VNI: 30010 (VLAN 10)L3VNI: 50001 (Tenant1)



iBGP-EVPN

VIP110.1.1.111

VIP210.2.2.222



Fabric

DCI

Multi-Site – Overlay Control-Plane (Site1)


RS


Site1

VTEP VTEP VTEP VTEP


VXLAN EVPN

Site2

VTEP VTEP VTEP VTEP


RR RR

Host10000.3010.1101192.168.10.101

Host30000.3010.1102192.168.10.102

Host20000.3020.2101192.168.20.101

VRFTenant1





VRFTenant1


VIP110.1.1.111

VIP210.2.2.222

Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.

2 0000.3010.1101/48 30010, 65501:30010 192.168.10.101/32 50001, 65501:50001 10.1.1.1

2 0000.3020.2101/48 30020, 65501:30020 192.168.20.101/32 50001, 65501:50001 10.1.1.111

2 0000.3010.1102/48 30010, 65501:30010 192.168.10.102/32 50001, 65501:50001 10.1.1.111

BRKDCN-2035 50


Fabric

DCI

Multi-Site – Overlay Control-Plane (Site2)


RS


Site1

VTEP VTEP VTEP VTEP


VXLAN EVPN

Site2

VTEP VTEP VTEP VTEP


RR RR

Host10000.3010.1101192.168.10.101

Host30000.3010.1102192.168.10.102

Host20000.3020.2101192.168.20.101

VRFTenant1





VRFTenant1


VIP110.1.1.111

VIP210.2.2.222


2 0000.3010.1101/48 30010, 65502:30010 192.168.10.101/32 50001, 65502:50001 10.2.2.222

2 0000.3020.2101/48 30020, 65502:30020 192.168.20.101/32 50001, 65502:50001 10.2.2.1

2 0000.3010.1102/48 30010, 65502:30010 192.168.10.102/32 50001, 65502:50001 10.2.2.3

BRKDCN-2035 51


Fabric

DCI

Multi-Site – Overlay Control-Plane (DCI)


RS



Site1

VTEP VTEP VTEP VTEP


VXLAN EVPN

Site2

VTEP VTEP VTEP VTEP


RR RR

VRFTenant1


VRFTenant1


VIP110.1.1.111

VIP210.2.2.222




Host10000.3010.1101192.168.10.101

Host30000.3010.1102192.168.10.102

Host20000.3020.2101192.168.20.101


2 0000.3010.1101/48 30010, 65599:30010 192.168.10.101/32 50001, 65599:50001 10.1.1.111

2 0000.3020.2101/48 30020, 65599:30020 192.168.20.101/32 50001, 65599:50001 10.2.2.222

2 0000.3010.1102/48 30010, 65599:30010 192.168.10.102/32 50001, 65599:50001 10.2.2.222

BRKDCN-2035 52

Multi-Site –Selective Advertisements


Multi-Site – Selective Advertisements

� The Multi-Site architecture provides granular control on how Layer-2 and Layer-3 communication is extended across sites

� Layer-2 and/or Layer-3 VNIs configured on the Border Gateways (BGW) control the Control-Plane advertisement towards DCI

� Enhances the overall scalability of the solution• Scale up the total number of End-Points supported across sites

BRKDCN-2035 54


Fabric

DCI

Multi-Site – Selective Advertisements (DCI)


RS


Site1

VTEP VTEP VTEP VTEP


VXLAN EVPN

Site2

VTEP VTEP VTEP VTEP


RR RR


Only prefixes of VRF ”Tenant1” and L2VNI 30010 are advertised from Site1 towards DCI. In this example this is Host1.

All prefixes of VRF ”Tenant2” and L2VNI 30020 are notadvertised from Site2 towards DCI. These prefixes are not seen within the DCIVRF

Tenant1


VIP110.1.1.111

VIP210.2.2.222Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.

2 0000.3010.1101/48 30010, 65599:30010 192.168.10.101/32 50001, 65599:50001 10.1.1.111

Host10000.3010.1101192.168.10.101



Host20000.3020.2101192.168.20.101

BRKDCN-2035 55

Multi-Site Data Plane


Fabric

DCI

Multi-Site – Overlay Data Plane



Site1

VTEP VTEP VTEP VTEP


VXLAN EVPN

Site2

VTEP VTEP VTEP VTEP

VTEP VTEP….VIP1

10.1.1.111VIP2

10.2.2.222BGW BGW BGW BGW

Host10000.3010.1101192.168.10.101

Host30000.3010.1102192.168.10.102

Host20000.3020.2101192.168.20.101

Intra-site VXLAN Data Plane

Inter-site VXLAN Data Plane

De-capsulation and Re-encapsulation on BGW

De-capsulation and Re-encapsulation on BGW

BRKDCN-2035 57

Multi-Site Packet Walk (BUM)


Packet Walk – Layer-2 (BUM) – Site1

VXLAN EVPNSite2

VTEP

Leaf20

VTEP

BGW21

VTEP

BGW22

VXLAN EVPNSite1

VTEP

Leaf10

VTEP

BGW11

VTEP

BGW12

VXLAN EVPNDCI

Baremetal

Host 10000.3010.1101192.168.10.101

Baremetal

Host 20000.3010.1102192.168.10.102

SIP DIP VXLAN SMAC DMAC SIP DIPPayload

L10 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255

Bridge

DF30010

DF30010

Host 1 sends a L2 BUM frame

1

2

Leaf10 replicates traffic intra-Site

BRKDCN-2035 59


Packet Walk – Layer-2 (DF & Split Horizon) – Site1

VXLAN EVPNSite2

VTEP

Leaf20

VTEP

BGW21

VTEP

BGW22

VXLAN EVPNSite1

VTEP

Leaf10

VTEP

BGW11

VTEP

BGW12

VXLAN EVPNDCI

Baremetal

Host 10000.3010.1101192.168.10.101

Baremetal

Host 20000.3010.1102192.168.10.102


L10 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255

Bridge

DF30010

DF30010

BUM Forward

Drop due to Split-Horizon rule

Drop due to Designated Forwarder (DF) rule

BRKDCN-2035 60


Packet Walk – Layer-2 (BUM) – DCI

VXLAN EVPNSite2

VTEP

Leaf20

VTEP

BGW21

VTEP

BGW22

VXLAN EVPNSite1

VTEP

Leaf10

VTEP

BGW11

VTEP

BGW12

VXLAN EVPNDCI

Baremetal

Host 10000.3010.1101192.168.10.101

Baremetal

Host 20000.3010.1102192.168.10.102

Bridge

DF30010

DF30010

SIP DIP VXLAN SMAC DMAC SIP DIP

PayloadBGW-VIP1 BGW21 30010 H1-MAC ALL-F H1-IP ALL-255

BGW-VIP1 BGW22 30010 H1-MAC ALL-F H1-IP ALL-255


3

BGW11 replicates traffic inter-Sites toward BGW nodes

BUM Forward

BRKDCN-2035 61


Packet Walk – Layer-2 (DF & Split Horizon) – DCI

VXLAN EVPNSite2

VTEP

Leaf20

VTEP

BGW21

VTEP

BGW22

VXLAN EVPNSite1

VTEP

Leaf10

VTEP

BGW11

VTEP

BGW12

VXLAN EVPNDCI

Baremetal

Host 10000.3010.1101192.168.10.101

Baremetal

Host 20000.3010.1102192.168.10.102

Bridge

DF30010

DF30010

SIP DIP VXLAN SMAC DMAC SIP DIP

PayloadBGW-VIP1 BGW21 30010 H1-MAC ALL-F H1-IP ALL-255



BUM Forward

BUM Forward



BRKDCN-2035 62



VXLAN EVPNSite2

VTEP

Leaf20

VTEP

BGW21

VTEP

BGW22

VXLAN EVPNSite1

VTEP

Leaf10

VTEP

BGW11

VTEP

BGW12

VXLAN EVPNDCI

Baremetal

Host 10000.3010.1101192.168.10.101

Baremetal

Host 20000.3010.1102192.168.10.102

Bridge

DF30010

DF30010


BGW-VIP2 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255

4

BGW22 replicates traffic intra-Site

BUM Forward

BRKDCN-2035 63


Packet Walk – Layer-2 (DF & Split Horizon) – Site2

VXLAN EVPNSite2

VTEP

Leaf20

VTEP

BGW21

VTEP

BGW22

VXLAN EVPNSite1

VTEP

Leaf10

VTEP

BGW11

VTEP

BGW12

VXLAN EVPNDCI

Baremetal

Host 10000.3010.1101192.168.10.101

Baremetal

Host 20000.3010.1102192.168.10.102

Bridge

DF30010

DF30010


BGW-VIP2 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255

BUM Forward



BRKDCN-2035 64



VXLAN EVPNSite2

VTEP

Leaf20

VTEP

BGW21

VTEP

BGW22

VXLAN EVPNSite1

VTEP

Leaf10

VTEP

BGW11

VTEP

BGW12

VXLAN EVPNDCI

Baremetal

Host 10000.3010.1101192.168.10.101

Baremetal

Host 20000.3010.1102192.168.10.102

Bridge

DF30010

DF30010

Leaf20 sends traffic to local Host 2

5

BRKDCN-2035 65

Multi-Site Packet Walk (Bridging)


Packet Walk – Layer-2 (Host 1 to Host 2) – Site1

VXLAN EVPNSite2

VTEP

Leaf20

VTEP

BGW21

VTEP

BGW22

VXLAN EVPNSite1

VTEP

Leaf10

VTEP

BGW11

VTEP

BGW12

VXLAN EVPNDCI

Baremetal

Host 10000.3010.1101192.168.10.101

Baremetal

Host 20000.3010.1102192.168.10.102


L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP

Bridge

Host 1 sends traffic destined to remote Host 2

1

2

Leaf10 performs L2 lookup and encapsulates toward local BGW VIP1 address

VIP2VIP1

BRKDCN-2035 67


Packet Walk – Layer-2 (Host 1 to Host 2) – DCI

VXLAN EVPNSite2

VTEP

Leaf20

VTEP

BGW21

VTEP

BGW22

VXLAN EVPNSite1

VTEP

Leaf10

VTEP

BGW11

VTEP

BGW12

VXLAN EVPNDCI

Baremetal

Host 10000.3010.1101192.168.10.101

Baremetal

Host 20000.3010.1102192.168.10.102


BGW-VIP1 BGW-VIP2 30010 H1-MAC H2-MAC H1-IP H2-IP

Bridge

3

BGW11 performs L2 lookup and encapsulates toward

remote BGW VIP2 address

VIP2VIP1

BRKDCN-2035 68



VXLAN EVPNSite2

VTEP

Leaf20

VTEP

BGW21

VTEP

BGW22

VXLAN EVPNSite1

VTEP

Leaf10

VTEP

BGW11

VTEP

BGW12

VXLAN EVPNDCI

Baremetal

Host 10000.3010.1101192.168.10.101

Baremetal

Host 20000.3010.1102192.168.10.102

Bridge


BGW-VIP2 L20 30010 H1-MAC H2-MAC H1-IP H2-IP

4


destination L20 node

Leaf20 bridges traffic to local Host 2

5

VIP2VIP1

BRKDCN-2035 69



VXLAN EVPNSite2

VTEP

Leaf20

VTEP

BGW21

VTEP

BGW22

VXLAN EVPNSite1

VTEP

Leaf10

VTEP

BGW11

VTEP

BGW12

VXLAN EVPNDCI

Baremetal

Host 10000.3010.1101192.168.10.101

Baremetal

Host 20000.3010.1102192.168.10.102

Bridge



Host 2 replies to remote Host 1

6

7

Leaf20 performs L2 lookup and encapsulates toward local BGW VIP2 address

VIP2VIP1

BRKDCN-2035 70



VXLAN EVPNSite2

VTEP

Leaf20

VTEP

BGW21

VTEP

BGW22

VXLAN EVPNSite1

VTEP

Leaf10

VTEP

BGW11

VTEP

BGW12

VXLAN EVPNDCI

Baremetal

Host 10000.3010.1101192.168.10.101

Baremetal

Host 20000.3010.1102192.168.10.102

Bridge


BGW-VIP2 BGW-VIP1 30010 H2-MAC H1-MAC H2-IP H1-IP

8



VIP2VIP1

BRKDCN-2035 71



VXLAN EVPNSite2

VTEP

Leaf20

VTEP

BGW21

VTEP

BGW22

VXLAN EVPNSite1

VTEP

Leaf10

VTEP

BGW11

VTEP

BGW12

VXLAN EVPNDCI

Baremetal

Host 10000.3010.1101192.168.10.101

Baremetal

Host 20000.3010.1102192.168.10.102

Bridge


BGW-VIP1 L10 30010 H2-MAC H1-MAC H2-IP H1-IP

9


destination L10 node

Leaf10 bridges traffic toward Host 1

10

VIP2VIP1

BRKDCN-2035 72

Multi-Site Packet Walk (Routing)



VXLAN EVPNSite2

VTEP

Leaf20

VTEP

BGW21

VTEP

BGW22

VXLAN EVPNSite1

VTEP

Leaf10

VTEP

BGW11

VTEP

BGW12

VXLAN EVPNDCI

Baremetal

Host 10000.3010.1101192.168.10.101

Baremetal

Host 30000.3010.1102192.168.20.102


L10 BGW-VIP1 50001 L10-MAC BGW-VMAC1 H1-IP H3-IP

Route

Host 1 sends a data packet to the remote

Host 3

1

2

Leaf10 performs a L3 lookup and encapsulates toward local BGW VIP1 address

VIP2VMAC2

VIP1VMAC1

BRKDCN-2035 74



VXLAN EVPNSite2

VTEP

Leaf20

VTEP

BGW21

VTEP

BGW22

VXLAN EVPNSite1

VTEP

Leaf10

VTEP

BGW11

VTEP

BGW12

VXLAN EVPNDCI

Baremetal

Host 10000.3010.1101192.168.10.101

Baremetal

Host 30000.3010.1102192.168.20.102

Route


BGW-VIP1 BGW-VIP2 50001 BGW-VMAC1 BGW-VMAC2 H1-IP H3-IP

3

BGW11 performs a L3 lookup and encapsulates toward


VIP2VMAC2

VIP1VMAC1

BRKDCN-2035 75



VXLAN EVPNSite2

VTEP

Leaf20

VTEP

BGW21

VTEP

BGW22

VXLAN EVPNSite1

VTEP

Leaf10

VTEP

BGW11

VTEP

BGW12

VXLAN EVPNDCI

Baremetal

Host 10000.3010.1101192.168.10.101

Baremetal

Host 30000.3010.1102192.168.20.102

Route


BGW-VIP2 L20 50001 BGW-VMAC1 L20-MAC H1-IP H3-IP

4

BGW21 performs a L3 lookup and encapsulates

toward destination L20 node

Leaf20 routes traffic to local Host 3

5

VIP2VMAC2

VIP1VMAC1

BRKDCN-2035 76

Multi-Site and Failure Detection on BGW


Steady State Traffic – Site1

VXLAN EVPNSite2

VTEP

Leaf20

VTEP

BGW21

VTEP

BGW22

VXLAN EVPNSite1

VTEP

Leaf10

VTEP

BGW11

VTEP

BGW12

VXLAN EVPNDCI

Baremetal

Host 10000.3010.1101192.168.10.101

Baremetal

Host 20000.3010.1102192.168.10.102



VIP2VIP1

BRKDCN-2035 78


DCI Link Failure BGW12 – Site1

VXLAN EVPNSite2

VTEP

Leaf20

VTEP

BGW21

VTEP

BGW22

VXLAN EVPNSite1

VTEP

Leaf10

VTEP

BGW11

VTEP

BGW12

VXLAN EVPNDCI

Baremetal

Host 10000.3010.1101192.168.10.101

Baremetal

Host 20000.3010.1102192.168.10.102



� On DCI Link Failure (i.e. BGW12)

• Virtual IP (VIP) on BGW is disabled

• BGW will stop participating in DF election

• BGW acts like a Leaf (Layer-3 only)

• Traffic towards others Sites is served by remaining BGWs (i.e. BGW11)

BGW12 gets isolated from the DCI Core Network

Intra-site VXLAN traffic re-routing

VIP2VIP1

BRKDCN-2035 79


Fabric Link Failure BGW12 – Site1

VXLAN EVPNSite2

VTEP

Leaf20

VTEP

BGW21

VTEP

BGW22

VXLAN EVPNSite1

VTEP

Leaf10

VTEP

BGW11

VTEP

BGW12

VXLAN EVPNDCI

Baremetal

Host 10000.3010.1101192.168.10.101

Baremetal

Host 20000.3010.1102192.168.10.102



VIP2VIP1

BRKDCN-2035 80


Fabric Link Failure BGW12 – Site1

VXLAN EVPNSite2

VTEP

Leaf20

VTEP

BGW21

VTEP

BGW22

VXLAN EVPNSite1

VTEP

Leaf10

VTEP

BGW11

VTEP

BGW12

VXLAN EVPNDCI

Baremetal

Host 10000.3010.1101192.168.10.101

Baremetal

Host 20000.3010.1102192.168.10.102



VIP2VIP1

BGW12 gets isolated from the Spine nodes

Intra-site VXLAN traffic re-routing

� On Fabric Link Failure (i.e. BGW12)

• Virtual IP (VIP) on BGW is disabled

• Primary IP (PIP) on BGW is disabled

• BGW will stop participating in the Overlay

BRKDCN-2035 81

Multi-Site Setup Walkthrough


Fabric

Site 1 Setup – Enable Border Gateway

Spine Spine

VTEP VTEP VTEP VTEP

VTEP VTEP….BGW1 BGW2

� Multi-Site Commands are marked in red

� Various options do exist but the recommended design choices are:

• Fabric Internal

¾ IGP Underlay, iBGP Overlay

• DCI (primary choice)

¾ eBGP Underlay, eBGP Overlay

¾ Route Server for DCI Overlay peerings

¾ DC Core for reachability across n Sites

• DCI (alternative option)

¾ Any Routing Protocol Underlay, eBGP Overlay

¾ Full-Mesh for DCI Overlay peerings

¾ Back-to-Back Site Reachability (physical, full-mesh)

BRKDCN-2035 83


Fabric

Site 1 Setup – Enable Border Gateway

Spine Spine

VTEP VTEP VTEP VTEP


feature nv overlaynv overlay evpn

feature bgpfeature interface-vlanfeature vn-segment-vlan-based

evpn multisite border-gateway

BGW2BGW1

BRKDCN-2035 84


Fabric

Site 1 Setup – BGW 1 Loopback & VTEP

Spine Spine

VTEP VTEP VTEP VTEP

VTEP VTEP….BGW1

interface loopback1description PIP VTEPip address 10.1.1.101/32 tag 12345ip router ospf UNDERLAY area 0.0.0.0ip pim sparse-mode

interface loopback100description VIP Multi-Site 1ip address 10.1.1.111/32 tag 12345ip router ospf UNDERLAY area 0.0.0.0ip pim sparse-mode

interface loopback0description RIDip address 10.10.10.101/32 tag 12345ip router ospf UNDERLAY area 0.0.0.0ip pim sparse-mode

BGW1

BRKDCN-2035 85


Fabric

Site 1 Setup – BGW 2 Loopback & VTEP

Spine Spine

VTEP VTEP VTEP VTEP

VTEP VTEP…. BGW2

interface loopback1description PIP VTEPip address 10.1.1.102/32 tag 12345ip router ospf UNDERLAY area 0.0.0.0ip pim sparse-mode

interface loopback100description VIP Multi-Site 1ip address 10.1.1.111/32 tag 12345ip router ospf UNDERLAY area 0.0.0.0ip pim sparse-mode

interface loopback0description RIDip address 10.10.10.102/32 tag 12345ip router ospf UNDERLAY area 0.0.0.0ip pim sparse-mode

BGW2

BRKDCN-2035 86


Fabric

Site 1 Setup – Fabric Link Tracking BGW 1

Spine Spine

VTEP VTEP VTEP VTEP

VTEP VTEP….BGW1

interface Ethernet1/53description TO-SPINE1ip address 10.0.1.1/30ip router ospf UNDERLAY area 0.0.0.0ip pim sparse-modeevpn multisite fabric-tracking


BGW1

Allows to bring down the PIP/VIP loopback interfaces when the

BGW is isolated from the spines

BRKDCN-2035 87


Fabric

Site 1 Setup – Fabric Link Tracking BGW 2

Spine Spine

VTEP VTEP VTEP VTEP

VTEP VTEP…. BGW2



BGW2

BRKDCN-2035 88



Fabric

Site 1 Setup – Multi-Site Underlay Interface

Spine Spine

VTEP VTEP VTEP VTEP


interface Ethernet1/1description TO-DC-CORE1ip address 10.111.111.1/30 tag 12345evpn multisite dci-tracking


BGW1

DCI



BGW2

Allows to bring down the PIP/VIP loopback interfaces when the BGW is isolated from the DC core BRKDCN-2035 89



Fabric

Site 1 BGW 1 Setup – Multi-Site Overlay Peering

Spine Spine

VTEP VTEP VTEP VTEP

VTEP VTEP….BGW1

DCI

RS router bgp 65501router-id 10.10.10.101address-family ipv4 unicastredistribute direct route-map REDIST-LOCAL

neighbor 10.111.111.2remote-as 65599update-source ethernet1/1address-family ipv4 unicast

neighbor 10.111.222.2remote-as 65599update-source ethernet1/2address-family ipv4 unicastneighbor 10.99.99.201remote-as 65599update-source loopback0ebgp-multihop 5peer-type fabric-externaladdress-family l2vpn evpnrewrite-evpn-rt-asnsend-communitysend-community both

BGW1




Fabric

Site 1 BGW 2 Setup – Multi-Site Overlay Peering

Spine Spine

VTEP VTEP VTEP VTEP

VTEP VTEP….DCI

RS router bgp 65501router-id 10.10.10.102address-family ipv4 unicastredistribute direct route-map REDIST-LOCAL

neighbor 10.222.111.2remote-as 65599update-source ethernet1/1address-family ipv4 unicast

neighbor 10.222.222.2remote-as 65599update-source ethernet1/2address-family ipv4 unicastneighbor 10.99.99.201remote-as 65599update-source loopback0ebgp-multihop 5peer-type fabric-externaladdress-family l2vpn evpnrewrite-evpn-rt-asnsend-communitysend-community both

BGW1


BGW2

BRKDCN-2035 91



Fabric

Site 1 Setup – Multi-Site Overlay Peering

Spine Spine

VTEP VTEP VTEP VTEP

VTEP VTEP….BGW1

DCI

RS


BGW2

� peer-type fabric-external

• Enables Next-Hop Rewrite for Multi-Site

• Defines Site External BGP neighbors for EVPN exchange

� rewrite-evpn-rt-asn

• Rewrites Route-Target Auto information to simplify MAC-VRF and IP-VRF configuration

• Normalizes outgoing Route-Targets AS number to match remote AS number

• Uses BGP configured Neighbors Remote AS

BRKDCN-2035 92


Fabric

DCI




Site1

VTEP VTEP VTEP VTEP


VXLAN EVPN

Site2

VTEP VTEP VTEP VTEP

VTEP VTEP….VIP1

10.1.1.111VIP2


Host10000.3010.1101192.168.10.101

peer-type fabric-external peer-type fabric-external

BGP Update:MAC: 0000.3010.1101 (L2VNI 30001)IP: 192.168.20.101 (L3VNI 50001)NH: 10.2.2.222RMAC: BGW-VMAC2

BGP Update:MAC: 0000.3010.1101 (L2VNI 30001)IP: 192.168.20.101 (L3VNI 50001)NH: 10.1.1.111RMAC: BGW-VMAC1

Rewrite Next-Hop IP and Next-Hop MAC (RMAC) based on Neighbor Site BGW

BGP Update:MAC: 0000.3010.1101 (L2VNI 30001)IP: 192.168.20.101 (L3VNI 50001)NH: 10.1.1.1RMAC: Leaf1

Rewrite Next-Hop IP and Next-Hop MAC (RMAC) based on Neighbor Site BGW

BRKDCN-2035 93


Fabric

DCI




Site1

VTEP VTEP VTEP VTEP


VXLAN EVPN

Site2

VTEP VTEP VTEP VTEP

VTEP VTEP….VIP1

10.1.1.111VIP2


Host10000.3010.1101192.168.10.101

Host20000.3020.2101192.168.20.101

rewrite-evpn-rt-asn rewrite-evpn-rt-asn

BGP Update:Remote AS: 65502VNI: 50001Route-Target: 65502:50001

Rewrite Route-Target based on BGP Neighbors Remote ASN

BGP Update:Remote AS : 65501VNI: 50001Route-Target: 65501:50001

BGP Update:Remote AS: 65502VNI: 50001Route-Target: 65502:50001

BRKDCN-2035 94



Fabric

Site 1 Setup – Anycast BGW VTEP Configuration

Spine Spine

VTEP VTEP VTEP VTEP

VTEP VTEP….BGW1

DCI


interface nve1no shutdownhost-reachability protocol bgpmultisite ethernet-segment 7system-mac 0000.0000.0001source-interface loopback1multisite border-gateway interface loopback100member vni 30010multisite ingress-replicationmcast-group 239.1.1.1

member vni 30011-30020mcast-group 239.1.1.2member vni 50001 associate-vrf

BGW1

BGW2

BGW2

BRKDCN-2035 95



Fabric

Site 1 Setup – Anycast BGW VTEP Configuration

Spine Spine

VTEP VTEP VTEP VTEP

VTEP VTEP….BGW1

DCI

BGW2

� multisite ethernet-segment

• Defines the discriminator for Sites in a common Domain � system-mac

• Defines the Multi-Site Site-Id (6 octets hex)� multisite border-gateway interface loopback#

• Defines the Loopback Interface used for the Border Gateway Virtual IP Address (VIP)

� multisite ingress-replication

• Per-VNI knob for extending Layer-2 VNI

• Defines the Multi-Site BUM Replication methodBRKDCN-2035 96



Fabric

Site 1 Setup – Multi-Site Overlay Traffic Policy

Spine Spine

VTEP VTEP VTEP VTEP

VTEP VTEP….BGW1

DCI

BGW2

• BUM Traffic Policing

• Limits Broadcast, Unknown Unicast and Layer-2 Multicast Traffic across Multi-Site

• Level 0 = No B/U/M Forwarding

• Level 100 = All B/U/M Forwarding Forwarding• Enforced on Encapsulation towards remote Sites

evpn storm-control broadcast level 10evpn storm-control unicast level 10evpn storm-control multicast level 10

BGW1 BGW2

BRKDCN-2035 97





Site 1

VTEP VTEP



Site n

VTEP VTEP

Overlay Multi-Site


Baremetal

BUM

BGW BGW BGW BGW

BRKDCN-2035 98





Site 1

VTEP VTEP



Site n

VTEP VTEP

Overlay Multi-Site


Baremetal

BUM

Storm ControlBroadcast 0-100%Unknown Unicast 0-100%Multicast 0-100%

Storm ControlBroadcast 0-100%Unknown Unicast 0-100%Multicast 0-100%

BGW BGW BGW BGW

BRKDCN-2035 99


Site 1 Setup – Multi-Site BUM Replication Modes



Site 1

VTEP VTEP



Site n

VTEP VTEP

Overlay Multi-Site


Multicast Multicast

Ingress Replication

BGW BGW BGW BGW

BRKDCN-2035 100





Site 1

VTEP VTEP



Site n

VTEP VTEP

Overlay Multi-Site


Ingress Replication Ingress Replication

Ingress Replication

BGW BGW BGW BGW

BRKDCN-2035 101





Site 1

VTEP VTEP



Site n

VTEP VTEP

Overlay Multi-Site


Ingress Replication Multicast

Ingress Replication

BGW BGW BGW BGW

BRKDCN-2035 102

Connectivity to the External Layer 3 Domain


Connectivity to the External Layer 3 Domain

� The BGW nodes can be used to provide Layer-3 external connectivity to each site

� Different connectivity models are supported• VRF-Lite peering with an external pair of WAN Edge routers• MP-BGP EVPN peering with the external WAN Edge routers (GOLF)• Dedicated or shared pair of WAN Edge routers across sites

� External Layer-3 network may be different from the DCI network used for inter-site communication

BRKDCN-2035 104


Fabric

DCI

Multi-Site – Border Gateway and VRF-Lite


Site1

VTEP VTEP VTEP VTEP


VXLAN EVPN

Site2

VTEP VTEP VTEP VTEP

VTEP VTEP….VIP1

10.1.1.111VIP2


Host10000.3010.1101192.168.10.101

Host30000.3010.1102192.168.10.102

Host20000.3020.2101192.168.20.101

VRF-CVRF-BVRF-ASeparate routing peering for each VRF (IGP or eBGP)

Dedicated interface (logical or physical) for each VRF

BRKDCN-2035 105


Fabric

DCI

Multi-Site – Border Gateway and GOLF


Site1

VTEP VTEP VTEP VTEP


VXLAN EVPN

Site2

VTEP VTEP VTEP VTEP

VTEP VTEP….VIP1

10.1.1.111VIP2


Host10000.3010.1101192.168.10.101

Host30000.3010.1102192.168.10.102

Host20000.3020.2101192.168.20.101

VRF-CVRF-BVRF-ASingle MP-BGP EVPN instance to exchange routes for all VRFs

VXLAN Data Plane between BGW and WAN Edge Router

BRKDCN-2035 106



MPLSL3VPN

Fabric

DCI


Site1

VTEP VTEP VTEP VTEP


VXLAN EVPN

Site2

VTEP VTEP VTEP VTEP

VTEP VTEP….VIP1

10.1.1.111VIP2


Host10000.3010.1101192.168.10.101

Host20000.3020.2101192.168.20.101

Host30000.3010.1102192.168.10.102

Multi-Site – Shared Internet/WAN GatewaysInternet/WAN

BorderPE BorderPE

Inter-Site VXLAN Communication between Border Gateways

BRKDCN-2035 107


Multi-Site – Per Site Internet/WAN Gateway


MPLSL3VPN

Fabric

DCI


Site1

VTEP VTEP VTEP VTEP


VXLAN EVPN

Site2

VTEP VTEP VTEP VTEP

VTEP VTEP….VIP1

10.1.1.111VIP2


Host10000.3010.1101192.168.10.101

Host20000.3020.2101192.168.20.101

Host30000.3010.1102192.168.10.102

Internet/WANBorderPE BorderPE BorderPE BorderPE


BRKDCN-2035 108


MPLSL3VPN

Fabric

DCI


Site1

VTEP VTEP VTEP VTEP


VXLAN EVPN

Site2

VTEP VTEP VTEP VTEP

VTEP VTEP….VIP1

10.1.1.111VIP2


Host10000.3010.1101192.168.10.101

Host20000.3020.2101192.168.20.101

Host30000.3010.1102192.168.10.102

Internet/WAN BorderPE BorderPE BorderPE BorderPE

Multi-Site – Consolidated WAN and DCI NetworkPerform simple routing for inter-site flows, VXLAN (or VRF-Lite) to MPLS VPN hand-off for north-south communication


BRKDCN-2035 109

Ingress and Egress Traffic Optimization


Spine Spine

VXLAN EVPN

Site1

VTEP VTEP VTEP VTEP

VTEP VTEP

Spine Spine

VXLAN EVPN

Site2

VTEP VTEP VTEP VTEP

VTEP VTEP

BGW

WAN

Active FWActive FW

� The stretching of Layer-2 domains across separate sites may lead to the creation of asymmetric traffic paths

� Deploying independent stateful services (like FWs) across sites would result in traffic drops

� In this case it is required to ensure the symmetry of ingress and egress communication paths

Ingress and Egress Traffic OptimizationThe Issue of Extending Layer 2 Domains

BGW BGW BGW


BRKDCN-2035 111



� Guarantee routing symmetry with the outside of the Data Center• Egress Æ Always prefer the local BGW• Ingress Æ Steer traffic to the specific destination

End-Point’s location

� Maintain optimal routing over the dedicated DCI network (if existing) for Server-to-Server traffic• The DC fabric must discriminate between DC

and WAN destinations

� If required provide a fallback path via DCI for WAN isolation situations


Site1

VTEP VTEP VTEP VTEP

VTEP VTEP


Site2

VTEP VTEP VTEP VTEP

VTEP VTEP

BGW BGW BGW BGW

WAN

Ingress and Egress Traffic OptimizationMaintaining Traffic Symmetry over Optimal Paths

BRKDCN-2035 112



MPLSL3VPN

Spine Spine

VXLAN EVPN

Site1VTEP VTEP VTEP VTEP

VTEP VTEP

Spine Spine

VXLAN EVPN


VTEP VTEPVIP210.2.2.222

BGW BGW BGW BGW

Host10000.3010.1101192.168.10.101

Host30000.3010.1102192.168.10.102

BorderPE BorderPE BorderPE BorderPE

VIP110.1.1.111

Multi-Site – Egress Path Optimization172.16.1.10

172.16.1.0/24 Æ Border-PEs 1-2 172.16.1.0/24 Æ Border-PEs 3-4

172.16.1.0/24 Æ VIP1 172.16.1.0/24 Æ VIP2

eBGP-EVPN

Less preferred advertisement of 172.16.1.0 because of longer AS-Path

BRKDCN-2035 113



MPLSL3VPN

Spine Spine

VXLAN EVPN


VTEP VTEP

Spine Spine

VXLAN EVPN



BGW BGW BGW BGW

Host10000.3010.1101192.168.10.101

Host30000.3010.1102192.168.10.102


VIP110.1.1.111

Multi-Site – Egress Path Optimization172.16.1.10

eBGP-EVPNOptimized Egress Traffic Path

Optimized Egress Traffic Path

BRKDCN-2035 114



MPLSL3VPN

Spine Spine

VXLAN EVPN


VTEP VTEP

Spine Spine

VXLAN EVPN



BGW BGW BGW BGW

Host10000.3010.1101192.168.10.101

Host30000.3010.1102192.168.10.102


VIP110.1.1.111

Multi-Site – Egress Path OptimizationWAN Isolation Scenario

172.16.1.10

172.16.1.0/24 Æ Border-PEs 3-4

172.16.1.0/24 Æ VIP1 172.16.1.0/24 Æ VIP2

eBGP-EVPN

WAN Isolation Scenario

172.16.1.0/24 Æ VIP2

BRKDCN-2035 115



MPLSL3VPN

Spine Spine

VXLAN EVPN


VTEP VTEP

Spine Spine

VXLAN EVPN



BGW BGW BGW BGW

Host10000.3010.1101192.168.10.101

Host30000.3010.1102192.168.10.102


VIP110.1.1.111

Multi-Site – Ingress Path Optimization

192.168.10.0/24 Æ BGW 1-2192.168.10.101/32 Æ BGW-1-2

192.168.10.0/24 Æ BGW 3-4192.168.10.102/32 Æ BGW 3-4

192.168.10.101/32 Æ Leaf1 192.168.10.102/32 -> Leaf3

eBGP-EVPN

Host routes advertised across sites but NOT re-advertised toward the local Border-PEs

192.168.10.0/24 Æ Border-PE 1-4192.168.10.101/32 Æ Border-PE 1-2192.168.10.102/32 Æ Border-PE 3-4Host routes

advertisement in the WAN

Deploying LISP on the Border-PEs is a viable alternative to host routes advertisement

Filter out host routes received from remote sites. Only announce local host route information

BRKDCN-2035 116



MPLSL3VPN

Spine Spine

VXLAN EVPN


VTEP VTEP

Spine Spine

VXLAN EVPN



BGW BGW BGW BGW

Host10000.3010.1101192.168.10.101

Host30000.3010.1102192.168.10.102


VIP110.1.1.111

Multi-Site – Ingress Path Optimization

eBGP-EVPN

192.168.10.0/24 Æ Border-PE 1-4192.168.10.101/32 Æ Border-PE 1-2192.168.10.102/32 Æ Border-PE 3-4

Optimized Ingress Traffic Path

Optimized Ingress Traffic Path

BRKDCN-2035 117



MPLSL3VPN

Spine Spine

VXLAN EVPN


VTEP VTEP

Spine Spine

VXLAN EVPN



BGW BGW BGW BGW

Host10000.3010.1101192.168.10.101

Host30000.3010.1102192.168.10.102


VIP110.1.1.111

eBGP-EVPN

192.168.10.0/24 Æ Border-PE 3-4192.168.10.101/32 Æ Border-PE 1-2192.168.10.102/32 Æ Border-PE 3-4

Multi-Site – Ingress Path OptimizationWAN Isolation Scenario

WAN Isolation Scenario

BRKDCN-2035 118

Network Services Integration


Network Services Integration

� Couple of different options where to connect network services:1. Service Leaf nodes: recommended to connect devices used for east-

west communication2. Border Gateway Nodes: used to connect network services for north-

south traffic flows

� Depending on the specifics of the Multi-Site deployment, the following deployment models would be possible:• Active/Standby Service Nodes pair connected to different sites• Active/Active cluster of Service Nodes deployed across sites• Independent Active/Standby Service nodes pairs deployed in separate

sites

BRKDCN-2035 120


Network Services IntegrationActive/Standby Pair Deployed across Sites

Spine Spine

VXLAN EVPN

Site1

VTEP VTEP VTEP VTEP

VTEP VTEP

Spine Spine

VXLAN EVPN

Site2

VTEP VTEP VTEP VTEP

VTEP VTEP

BGW

WAN

Standby FWActive FW

BGW BGW BGW


� Requirement to extend Layer 2 communication between Active/Standby nodes for keep-alivesand state information exchange� Perimeter service nodes connected to VPC Border

Gateways

� Ingress and egress traffic always traversing the Active node in Site 1� No issues related to the creation of asymmetric

traffic paths

� East-West flows must be hair-pinned to the active FW connected to the Service leaf nodes in Site 1• Need to properly dimension bandwidth in the DC

Core to accommodate for this extra traffic

Active FW Standby FWBaremetal BaremetalBaremetal

North-South traffic flows


East-West traffic flows

BRKDCN-2035 121


Network Services IntegrationActive/Active Cluster of Service Nodes Deployed across Sites

Spine Spine

VXLAN EVPN

Site1

VTEP VTEP VTEP VTEP

VTEP VTEP

Spine Spine

VXLAN EVPN

Site2

VTEP VTEP VTEP VTEP

VTEP VTEP

BGW

WAN

Active/Active FW Cluster

BGW BGW BGW


� Requirement to extend Layer 2 communication between Active/Active nodes for intra-cluster communication and traffic redirection� Perimeter service nodes connected to VPC Border

Gateways

� Asymmetric traffic issues taken care by native intra-cluster traffic redirection

� Option to deploy ingress/egress optimization technique to avoid inter-site traffic hair-pinning

� Service node cluster integration not supported at FCS and planned for a future SW release

Active/Active FW Cluster

Logical Intra-Cluster Link (ICL)

Baremetal


BRKDCN-2035 122


Network Services IntegrationIndependent Active/Standby Pair Deployed in Separate Sites

Spine Spine

VXLAN EVPN

Site1

VTEP VTEP VTEP VTEP

VTEP VTEP

Spine Spine

VXLAN EVPN

Site2

VTEP VTEP VTEP VTEP

VTEP VTEP

BGW

WAN

Active/Standby FW

Active/Standby FW

BGW BGW BGW


� Mandates the deployment of Ingress/Egress traffic optimization to avoid creation of asymmetric traffic path for north-south communication

� Active/Standby nodes can use direct links to sync state� Perimeter service nodes can connected to Anycast

Border Gateways

� Active/Standby pair (or cluster) still required for service nodes used for east-west traffic flows

Baremetal Baremetal

Active FW Standby FW



BRKDCN-2035 123

Legacy Site Integration


Multi-Site and Legacy Site Integration

� Extend Layer-2 and Layer-3 connectivity between sites • Coexistence and/or application migration use cases

� Proposed approach is to deploy a pair of ‘remote’ VPC Border Gateways in the legacy site• Offers native Multi-Site functionalities (BUM containment, etc) to the legacy site



Greenfield Site

VTEP VTEP

Legacy Site

VTEP VTEP

BGW BGW BGW BGW

Pair of VPC Border Gateways

BRKDCN-2035 125


Multi-Site and Legacy Site IntegrationLayer-2 Connectivity with the ‘Remote’ BGW

Legacy Site

VTEP VTEP

BGW BGW

Legacy Aggregation Layer devices support MLAG

Single logical link to extend VLANs toward the Greenfield VXLAN EVPN site

VLANs mapped to L2VNIs on the Border Gateways

Legacy Site

VTEP VTEP

BGW BGW

Legacy Aggregation Layer devices do not support MLAG

Single port-channel from each aggregation layer device

VLANs mapped to L2VNIs on the Border Gateways

Recommended to move the STP root to the BGW devices

BRKDCN-2035 126


Multi-Site and Legacy Site IntegrationLayer-2 Control Plane Exchange across Sites

Baremetal

Host 10000.3010.1101192.168.10.101

Baremetal

Host 20000.3010.1102192.168.10.102



Greenfield Site

VTEP VTEP

Legacy Site

VTEP VTEP

BGW BGW BGW BGW

VIP110.1.1.111

VIP210.2.2.222

MAC NH

0000.3010.1101 Leaf1

0000.3010.1102 VIP2

MAC NH

0000.3010.1101 VIP1

0000.3010.1102 Po1

Po1

All End-Points in the legacy site are learned as directly connected to the BGW

eBGP-EVPN

BRKDCN-2035 127


� Integration between Anycast Gateway and legacy default gateway (HSRP, VRRP, etc.) not initially supported with VXLAN Multi-Site

� First option is to keep on the legacy network the active default gateway for the stretched IP subnets



Greenfield Site

VTEP VTEP

Legacy Site

VTEP VTEP

BGW BGW BGW BGW

Default Gateway deployed on the legacy aggregation devices

Multi-Site and Legacy Site IntegrationDefault Gateway Deployment – Option 1

L3

L2

Greenfield VXLAN EVPN Fabric only offers L2 services for the stretched IP subnets

BRKDCN-2035 128


� Recommended approach is to migrate the default gateway from the legacy aggregation devices to the Border Gateways (VXLAN EVPN Anycast Gateway)

� Optimize routing between End-Points deployed across sites



Greenfield Site

VTEP VTEP

Legacy Site

VTEP VTEP

BGW BGW BGW BGW

Default Gateway migrated to the Border Gateways (VXLAN EVPN Anycast Gateway)

Multi-Site and Legacy Site IntegrationDefault Gateway Deployment – Option 2

L3

L2

Greenfield VXLAN EVPN Fabric offers L2 and L3 services for the stretched IP subnets

Legacy infrastructure offers only L2 services

Distributed Anycast Gateway function

L3

L2

BRKDCN-2035 129


Multi-Site and Legacy Site IntegrationLayer-3 Control Plane Exchange across Sites

Baremetal

Host 10000.3010.1101192.168.10.101



Greenfield Site

VTEP VTEP

Legacy Site

VTEP VTEP

BGW BGW BGW BGW

VIP110.1.1.111

VIP210.2.2.222

Po1

All End-Points in the legacy site are learned as directly connected to the BGW

eBGP-EVPN

L3

L2

Baremetal

Host 30000.3010.1102192.168.20.101

IP NH

192.168.10.101 Leaf1

192.168.20.101 VIP1

IP L3VNI

192.168.10.101 VIP1

192.168.20.101 Po1

BRKDCN-2035 130

Migration to Multi-Site


Migration to Multi-SiteUse Cases

1. Site addition: need to connect a Greenfield VXLAN EVPN Fabric to an existing VXLAN EVPN Fabric built with 1st

generation Nexus 9000

2. Migrating a VXLAN Multi-Pod Fabric to Multi-Site

3. Migrating a VXLAN Multi-Fabric design to Multi-Site

BRKDCN-2035 132


Migration to Multi-SiteSite Addition



Existing VXLAN EVPN Fabric

VTEP VTEP

BGW BGW



Greenfield Fabric

VTEP VTEP

BGW BGW



Existing VXLAN EVPN Fabric

� Step 1: add a pair of Border Gateways to the existing VXLAN EVPN Fabric, running the proper SW release supporting Multi-Site

Note: no requirement to change the HW/SW version on existing leaf nodes

� Step 2: connect the BGW to the inter-site network and establish control plane peering with the BGW in the Greenfield Fabric

� Step 3: configure on the BGW the L2VNIs and L3VNIs to be extended

eBGP-EVPN

BRKDCN-2035 133


Migration to Multi-SiteMulti-Fabric to Multi-Site



Site 1

VTEP VTEP



Site 2

VTEP VTEP


BGW BGW BGW BGW

� Step 1: add a pair of Border Gateways to each Pod (if needed) and connect them to the spines and to the inter-site network

� Step 2: upgrade the SW on both Fabrics BGW to be able to support Multi-Site

� Step 3: establish control plane adjacencies across sites

� Step 4: disconnect the previously used DCI and extend Layer-2 and Layer-3 across Multi-Site



Fabric 1

VTEP VTEP



Fabric 2

VTEP VTEP

BRKDCN-2035 134

Conclusion


� Multiple Overlay Domains – Interconnected & Controlled• Scaling and Segregating VXLAN EVPN Networks

� Multiple Overlay Control-Plane Domains – Interconnected & Controlled• Limited Overlay Control-Plane Update Propagation

� Multiple Underlay Domains - Isolated• Isolated Underlay Domains – No need for Extension

� Multiple Replication Domains for BUM – Interconnected & Controlled• Individual BUM flooding domain with Traffic control

Multi-Site Advantages – ”The Multiple”

BRKDCN-2035 136


• New IETF Draft for Multi-Site Design

• Multi-site EVPN based VXLAN using Border Gateways

• https://tools.ietf.org/html/draft-sharma-multi-site-evpn

VXLAN EVPN – Multi-Site

BRKDCN-2035 137

https://tools.ietf.org/html/draft-sharma-multi-site-evpn-00


• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 gift card.

• Complete your session surveys through the Cisco Live mobile app or on www.CiscoLive.com/us.

Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on demand after the event at www.CiscoLive.com/Online.

http://www.ciscolive.com/us

http://ciscolive.com/Online

http://ciscolive.com/Online

http://www.ciscolive.com/Online


Continue Your Education• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

BRKDCN-2035 139

Thank you