Upload
mason-mei
View
33
Download
8
Embed Size (px)
Citation preview
VXLAN BGP EVPN based Multi-Pod, Multi-Fabric, Multi-Site
Max Ardica – Principal EngineerLukas Krattiger – Principal EngineerBRKDCN-2035
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco Spark
Questions? Use Cisco Spark to chat with the speaker after the session
1. Find this session in the Cisco Live Mobile App2. Click “Join the Discussion”3. Install Spark or go directly to the space4. Enter messages/questions in the space
How
cs.co/ciscolivebot#BRKDCN-2035Cisco Spark spaces will be available until July 3, 2017.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Who Are the Presenters?
Max ArdicaPrincipal Engineer - INSBU
Lukas KrattigerPrincipal Engineer – INSBU
@ccie21921
BRKDCN-2035 4
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Session Objectives
At the end of the session, the participants should be able to:
� Articulate the different deployment options to interconnect VXLAN EVPN Networks (Multi-Pod vs. Multi-Fabric vs. Multi-Site)
� Understand the functionalities and specific design considerations associated to the new VXLAN Multi-Site architecture
Initial assumption:
� The audience already has a good knowledge of the VXLAN EVPN technology and its use to deploy modern Data Center Fabrics
BRKDCN-2035 5
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Session Reference• BRKDCN-2304
• L4-L7 Service Integration in Multi-Tenant VXLAN EVPN Data Center Fabrics
• BRKDCN-3378• Building DataCenter Networks with VXLAN BGP-EVPN• Wednesday, Jun 28, 1:30 pm
• BRKDCN-2125• Overlay Management and Visibility with VXLAN• Thursday, Jun 29, 10:30 am
• BRKDCN-2342• Programmable Fabric Automation and Management with DCNM 10• Thursday, Jun 29, 1:00 p.m.
6BRKDCN-2035
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Agenda� Introduction
� VXLAN EVPN Interconnect Evolution• Multi-Pod• Multi-Fabric• Multi-Site
� VXLAN EVPN Multi-Site Deep Dive• Walkthrough• Control- and Data-Plane• Deployment Considerations
� Conclusions and Q&A
Introduction
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Layer 2 Domain ElasticityLocal LAN Fabric
Extended LAN fabric
VN-link notifications
IP MobilityOptimal Ingress and Egress
Routing
VM-awarenessVXLAN, DFA, ACI, VN-link
Storage ElasticitySAN Extensions
Network Service LocalizationAny service anywhere
OTV
OTV
OTV
OTV
Fabric ConsolidationUnified Fabric & I/ODevice Virtualization
Segmentation
Data Center Interconnect – DCI Model Connecting Virtualized Data Centers
Multi-tenancy/SegmentationSegment-IDs in VXLAN, LISP, FabricPath,
and OTV
Storage Solutions & Partners:FCIP, I/O Acceleration
EMC, NetApp
BRKDCN-2035 9
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Back ThenYet Another Encapsulation
� Flood & Learn (Multicast-based)� Data-Plane only Yesterday
VXLAN for the Data Center – Intra-DC� Control-Plane� Active VTEP Discovery� Multicast and Unicast
Now!VXLAN for DCI – Inter-DC
� DCI Ready � ARP/ND caching/suppress� Multi-Homing� Failure Domain Isolation� Loop Protection
VXLAN Evolves as the Control Plane Evolves!
BRKDCN-2035 10
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Back ThenVXLAN for Interconnecting Networks
BRKDCN-2035 11
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Inter-X Connectivity
• Single Fabric with End-to-End Encapsulation
• Build Hierarchy in the Underlay – Flatten it in the Overlay
Multi-Pod
OverlayVTE
PVTE
PVTE
PVTE
P
Baremeta
l
Baremeta
l
Fabric #2
OverlayVTE
PVTE
P
Baremeta
l
Baremeta
l
VTEP
VTEP
Fabric #1EVPN Control-Plane Domain 1
EVPN Control-Plane Domain 2
Single Data-Plane – End-to-End
BGP EVPN
OverlayVTE
PVTE
PVTE
PVTE
P
Baremetal
Baremetal
Fabric #2
OverlayVTE
PVTE
P
Baremetal
Baremetal
VTEP
VTEP
Fabric #1EVPN Control-Plane Domain 1
EVPN Control-Plane Domain 2
Data-Plane Domain 1 Data-Plane Domain 2DCI Data-Plane
• Multiple Fabrics –Normalized through Ethernet
• Multiple Fabrics Interconnect using DCI (Layer 2 and Layer 3)
Multi-Fabric
Data-Plane Domain 1 Data-Plane Domain 2
DCI Data-Plane
OverlayVTE
PVTE
PVTE
PVTE
P
Baremetal
Baremetal
Fabric #2
OverlayVTE
PVTE
P
Baremetal
Baremetal
VTEP
VTEP
Fabric #1EVPN Control-Plane Domain 1
EVPN Control-Plane Domain 2
BGP EVPN
• Multiple Fabrics with Integrated DCI
• Integrated DCI –Scaling within and between Fabrics
• The Happy Place -
Multi-Site
BRKDCN-2035 12
VXLAN EVPN Interconnect Evolution
Multi-Pod
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN EVPN – Single Pod / Single Fabric
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Pod 1
VTEP VTEP
External Network
BRKDCN-2035 15
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN EVPN – Multi-Pod
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Pod 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Pod n
VTEP VTEP
Underlay Extension
BRKDCN-2035 16
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
� Single Overlay Domain – End-to-End Encapsulation
� Single Overlay Control-Plane Domain – End-to-End EVPN Updates
� Single Underlay Domain End-to-End
� Single Replication Domain for BUM
� Single VNI Administrative Domain
Multi-Pod Characteristics – ”The Single”
Building Underlay Hierarchies – Non Hierarchical Overlay
BRKDCN-2035 17
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Pod – End-to-End Encapsulation
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Pod 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Pod n
VTEP VTEP
Underlay Extension
Overlay
Baremetal Baremetal
Unicast
VTEP10.1.1.1
VTEP10.2.2.7
BRKDCN-2035 18
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Pod – BUM Replication
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Pod 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Pod 2
VTEP VTEP
Underlay Extension
Overlay
Baremetal
BUM
BRKDCN-2035 19
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
� Single Overlay Domain – End-to-End Encapsulation• Scaling the VXLAN EVPN Network
� Single Overlay Control-Plane Domain – End-to-End EVPN Updates• Overlay Control-Plane Update Propagation
� Single Underlay Domain End-to-End• Network must be extended in Underlay (VTEP to VTEP reachability)
� Single Replication Domain for BUM• One BUM flooding domain through out all connected Pods
Multi-Pod Challenges – ”The Single”
BRKDCN-2035 20
Multi-Fabric
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN EVPN – Multi-Fabric
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Fabric 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Fabric 2
VTEP VTEP
Underlay No ExtensionL2 DCI L2 DCI
L3 DCI L3 DCIL2 DCI L2 DCI
BRKDCN-2035 22
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Separate Overlay Domains –Independent L2 and L3 DCI (complexity)
• Separate Overlay Control-Plane Domains – Manual Configuration
• Separate Underlay Domains - Isolated
• Separate Replication Domains for BUM – Independent BUM transport/DCI
• Dedicated Border Leaf – no local End-Point Attachment
Multi-Fabric Characteristics – ”The Separate”
Underlay Isolation – Separate DC Interconnection
BRKDCN-2035 23
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Fabric 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Fabric n
VTEP VTEP
Underlay No ExtensionL2 DCI L2 DCI
L3 DCI L3 DCIL2 DCI L2 DCI
Multi-Fabric – End-to-End Encapsulation
Overlay Site 1 Overlay Site n
L2 DCI
Baremetal Baremetal
Unicast
VLAN Hand-Off VRF-Lite Hand-Off
BRKDCN-2035 24
VXLAN EVPN Multi-Site
Multi-Site Walkthrough
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN EVPN – Multi-Site
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site n
VTEP VTEP
No Underlay Extension
BGW BGW BGW BGW
BRKDCN-2035 27
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
� Multiple Overlay Domains – Interconnected & Controlled
� Multiple Overlay Control-Plane Domains – Interconnected & Controlled
� Multiple Underlay Domains - Isolated
� Multiple Replication Domains for BUM – Interconnected & Controlled
� Multiple VNI Administrative Domains – Phase 2
Multi-Site Characteristics – ”The Multiple”
Underlay Isolation – Overlay Hierarchies
BRKDCN-2035 28
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Site – Hierarchical Overlay Domains
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site n
VTEP VTEP
Overlay Site 1 Overlay Site n
Overlay Multi-Site
Baremetal Baremetal
Unicast
BGW BGW BGW BGW
BRKDCN-2035 29
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Site – Underlay Isolation
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site n
VTEP VTEP
No Underlay Extension
VTEP10.1.1.1
Border (VIP)10.1.1.111
Border (VIP)10.2.2.222
Site 1 Underlay Routing Table
Leaf:10.1.1.110.1.1.210.1.1.310.1.1.410.1.1.510.1.1.610.1.1.7
Border:10.1.1.10110.1.1.10210.1.1.111
VTEP10.2.2.7
Site n Underlay Routing Table
Leaf:10.2.2.110.2.2.210.2.2.310.2.2.410.2.2.510.2.2.610.2.2.7
Border:10.2.2.10110.2.2.10210.2.2.222
BGW BGW BGW BGWBorder (PIP)10.1.1.101
Border (PIP)10.1.1.102
Border (PIP)10.2.2.101
Border (PIP)10.2.2.102
BRKDCN-2035 30
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Inter Site Network
Multi-Site – Inter Site Network
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site n
VTEP VTEP
VTEP10.1.1.1
Border (VIP)10.2.2.222
Border (VIP)10.1.1.111
Inter-Site NetworkRouting TableBorder Site1:10.1.1.10110.1.1.10210.1.1.111
Border Site2:10.2.2.10110.2.2.10210.2.2.222
VTEP10.2.2.7
BGW BGW BGW BGWBorder (PIP)10.1.1.101
Border (PIP)10.1.1.102
Border (PIP)10.2.2.101
Border (PIP)10.2.2.102
BRKDCN-2035 31
Border Gateways Deployment Considerations
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Border Gateways Deployment Considerations
Site 1
VTEP
BGWVTEP
BGWVTEP
BGWVTEP
BGW
Site 1
VTEP
BGWVTEP
BGW
� Border Gateways used for two main functions:1. Interconnecting each site to the Inter-Site network (for
East-West traffic flows)2. Connecting each site to the external Layer 3 domain
(for North-South traffic flows)
� May also be used to connect End-Points and/or network service nodes (FWs, ADCs)
� Two deployment models supported:1. Anycast Border Gateways2. VPC Border Gateways
Anycast Border Gateways
VPC Border Gateways
BRKDCN-2035 33
Anycast Border Gateways
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1
Anycast Border Gateway (1)Anycast Border Gateway� Up to 4 Border Gateways� Border Gateway
• Deploying at Leaf – 7.0(3)I7(1)• Deploying at Spine – 7.0(3)I7(2)
VTEP
BGWVTEP
BGWVTEP
BGWVTEP
BGW
BRKDCN-2035 35
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1
Anycast Border Gateway (2)Anycast Border Gateway� Common Virtual IP (VIP) across BGW
• VIP is used for Intra- and Inter-Site Communication
• VIP for communication between the Border Gateways in different Sites
• VIP for communication between Border Gateway and Leaf within a Site
� Individual Primary IP (PIP) per BGW• Used for Broadcast, Unknown Unicast and
Multicast (BUM) replication• PIP for communication with Single-Homed
End-Points (routed only), intra- and inter-Site
VTEP
BGWVTEP
BGWVTEP
BGWVTEP
BGW
Border VIP10.1.1.111
Border VIP10.1.1.111
PIP-BGW110.1.1.101
PIP-BGW210.1.1.102
PIP-BGW310.1.1.103
PIP-BGW410.1.1.104
BRKDCN-2035 36
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1
Anycast Border Gateway (3)Anycast Border Gateway� Per-VNI Designated Forwarder (DF) election
• Each BGW can serve as DF for a single or a set of Layer-2 VNI
• DF election and assignment is automatic� Using BGP EVPN Route Type 4 for DF election
• Operator Managed Assignment (Type: 00)• Six Octet Site Identifier (System MAC:
00:00:00:00:00:01)• Multi-Site Discriminator (Ethernet-Segment:
00:00:07)• Originators IP Address (PIP): 10.1.1.101• Layer-2 VNI: 30010
VTEP
BGWVTEP
BGWVTEP
BGWVTEP
BGW
Spine
RRSpine
RR
BGP EVPN
Type: 00System MAC: 00:00:00:00:00:01Ethernet Segment: 00:00:074 IP: 10.1.1.101
VNI: 30010
DF30010
DF30099
DF30012
DF30011
BRKDCN-2035 37
VPC Border Gateways
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1
VPC Border Gateway (1)VPC Border Gateway� 2 Border Gateways� Border Gateway
• Using a Leaf – 7.0(3)I7(2)VTEP
BGWVTEP
BGW
BRKDCN-2035 39
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1
VPC Border Gateway (2)VPC Border Gateway� Common Virtual IP (VIP) across BGW
• VIP is used for Intra- and Inter-Site Communication
• VIP for communication between the Border Gateways in different Sites
• VIP for communication between Border Gateway and Leaf within a Site
� Individual Primary IP (PIP) per BGW• Used for Broadcast, Unknown Unicast and
Multicast (BUM) replication• PIP for communication with Single-Homed
End-Points, intra- and inter-Site
VTEP
BGWVTEP
BGW
Border VIP10.1.1.111
Border VIP10.1.1.111
PIP-BGW110.1.1.101
PIP-BGW210.1.1.102
BRKDCN-2035 40
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1
VPC Border Gateway (3)VPC Border Gateway� VPC-based Designated Forwarder Election� Per-Site Designated Forwarder (DF) election
• Using same approach as in VPC• Best Path to Rendezvous-Point or VPC
Primary Node
VTEP
BGWVTEP
BGW
DF
BRKDCN-2035 41
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1
VPC Border Gateway (4)VPC Border Gateway� Single- or Dual-Homed End-Points
• Services Appliance (i.e. Firewall, ADC etc.)• Physical or Virtual Servers
� Advertised and Reachable through Virtual IP Address (VIP)• Intra-Site: Leaf nodes use VIP to reach End-
Points connected to Border Gateways• Inter-Site: Remote Border Gateways use VIP
to reach End-Points connected to Border • Traffic potentially traverses VPC Peer-Link
VTEP
BGWVTEP
BGW
VTEPType MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
2 0000.3010.1101/48 30010, 65599:30010 192.168.10.101/32 50001, 65599:50001 10.1.1.111
2 0000.3010.1102/48 30010, 65599:30010 192.168.10.102/32 50001, 65599:50001 10.1.1.111
Border VIP10.1.1.111
Border VIP10.1.1.111
ADC0000.3010.1102192.168.10.102
ADC
ADC0000.3010.1101192.168.10.101
ADC
BRKDCN-2035 42
Multi-Site Control Plane
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Control Plane Deployment Considerations
� Both MP-eBGP or MP-iBGP peering supported intra-Site between leaf nodes
� Only MP-eBGP EVPN sessions supported inter-Sites Æ mandates that each site is part of a separate AS
� Full mesh of MP-eBGP EVPN adjacencies only currently supported across sites • Recommended to deploy a couple of Route-Servers in the Inter-Site network when 3 or
more sites are deployed• Route-Servers only perform control plane functions (“eBGP Route-Reflectors”)• Need to ensure that Route-Servers offer support for Route Type 4 EVPN routes,
required for DF election
BRKDCN-2035 44
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Overlay Control-Plane (L3Core)
Spine SpineVXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP….Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP….BGW BGW BGW BGW
RR RRiBGP-EVPN iBGP-EVPN
DC Core(Layer-3 Unicast)
BRKDCN-2035 45
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Overlay Control-Plane (L3Core)
Spine SpineVXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP….Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP….BGW BGW BGW BGW
RR RRiBGP-EVPN iBGP-EVPN
DC Core(Layer-3 Unicast)
RS Route Server (eBGP ”Route Reflector”) RS
BRKDCN-2035 46
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Overlay Control-Plane (L3Core)
DC Core(Layer-3 Unicast)
RS
Spine SpineVXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP….Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP….BGW BGW BGW BGW
RR RRiBGP-EVPN iBGP-EVPN
RS – Route Server (eBGP ”Route Reflector”) BRKDCN-2035 47
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Overlay Control-Plane (L3Core, no RS)
DC Core(Layer-3 Unicast)
Spine SpineVXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP….Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP….BGW BGW BGW BGW
RR RRiBGP-EVPN iBGP-EVPN
RS – Route Server (eBGP ”Route Reflector”)
eBGP-EVPN
BRKDCN-2035 48
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Overlay Control-Plane
DC Core(Layer-3 Unicast)
RS
Spine SpineVXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP….Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP….BGW BGW BGW BGW
RR RRiBGP-EVPN
Host10000.3010.1101192.168.10.101
Host30000.3010.1102192.168.10.102
Host20000.3020.2101192.168.20.101
VRFTenant1
L3VNI: 50001Route-Target: 65501:50001
VRFTenant1
L3VNI: 50001Route-Target: 65502:50001
L2VNI: 30010 (VLAN 10)L3VNI: 50001 (Tenant1)
L2VNI: 30020 (VLAN 20)L3VNI: 50001 (Tenant1)
L2VNI: 30010 (VLAN 10)L3VNI: 50001 (Tenant1)
iBGP-EVPN
VIP110.1.1.111
VIP210.2.2.222
RS – Route Server (eBGP ”Route Reflector”) BRKDCN-2035 49
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Overlay Control-Plane (Site1)
DC Core(Layer-3 Unicast)
RS
Spine SpineVXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP….Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP….BGW BGW BGW BGW
RR RR
Host10000.3010.1101192.168.10.101
Host30000.3010.1102192.168.10.102
Host20000.3020.2101192.168.20.101
VRFTenant1
L3VNI: 50001Route-Target: 65502:50001
L2VNI: 30010 (VLAN 10)L3VNI: 50001 (Tenant1)
L2VNI: 30020 (VLAN 20)L3VNI: 50001 (Tenant1)
L2VNI: 30010 (VLAN 10)L3VNI: 50001 (Tenant1)
VRFTenant1
L3VNI: 50001Route-Target: 65501:50001
VIP110.1.1.111
VIP210.2.2.222
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
2 0000.3010.1101/48 30010, 65501:30010 192.168.10.101/32 50001, 65501:50001 10.1.1.1
2 0000.3020.2101/48 30020, 65501:30020 192.168.20.101/32 50001, 65501:50001 10.1.1.111
2 0000.3010.1102/48 30010, 65501:30010 192.168.10.102/32 50001, 65501:50001 10.1.1.111
BRKDCN-2035 50
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Overlay Control-Plane (Site2)
DC Core(Layer-3 Unicast)
RS
Spine SpineVXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP….Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP….BGW BGW BGW BGW
RR RR
Host10000.3010.1101192.168.10.101
Host30000.3010.1102192.168.10.102
Host20000.3020.2101192.168.20.101
VRFTenant1
L3VNI: 50001Route-Target: 65502:50001
L2VNI: 30010 (VLAN 10)L3VNI: 50001 (Tenant1)
L2VNI: 30020 (VLAN 20)L3VNI: 50001 (Tenant1)
L2VNI: 30010 (VLAN 10)L3VNI: 50001 (Tenant1)
VRFTenant1
L3VNI: 50001Route-Target: 65502:50001
VIP110.1.1.111
VIP210.2.2.222
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
2 0000.3010.1101/48 30010, 65502:30010 192.168.10.101/32 50001, 65502:50001 10.2.2.222
2 0000.3020.2101/48 30020, 65502:30020 192.168.20.101/32 50001, 65502:50001 10.2.2.1
2 0000.3010.1102/48 30010, 65502:30010 192.168.10.102/32 50001, 65502:50001 10.2.2.3
BRKDCN-2035 51
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Overlay Control-Plane (DCI)
DC Core(Layer-3 Unicast)
RS
RS – Route Server (eBGP ”Route Reflector”)
Spine SpineVXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP….Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP….BGW BGW BGW BGW
RR RR
VRFTenant1
L3VNI: 50001Route-Target: 65501:50001
VRFTenant1
L3VNI: 50001Route-Target: 65502:50001
VIP110.1.1.111
VIP210.2.2.222
L2VNI: 30010 (VLAN 10)L3VNI: 50001 (Tenant1)
L2VNI: 30020 (VLAN 20)L3VNI: 50001 (Tenant1)
L2VNI: 30010 (VLAN 10)L3VNI: 50001 (Tenant1)
Host10000.3010.1101192.168.10.101
Host30000.3010.1102192.168.10.102
Host20000.3020.2101192.168.20.101
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
2 0000.3010.1101/48 30010, 65599:30010 192.168.10.101/32 50001, 65599:50001 10.1.1.111
2 0000.3020.2101/48 30020, 65599:30020 192.168.20.101/32 50001, 65599:50001 10.2.2.222
2 0000.3010.1102/48 30010, 65599:30010 192.168.10.102/32 50001, 65599:50001 10.2.2.222
BRKDCN-2035 52
Multi-Site –Selective Advertisements
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Site – Selective Advertisements
� The Multi-Site architecture provides granular control on how Layer-2 and Layer-3 communication is extended across sites
� Layer-2 and/or Layer-3 VNIs configured on the Border Gateways (BGW) control the Control-Plane advertisement towards DCI
� Enhances the overall scalability of the solution• Scale up the total number of End-Points supported across sites
BRKDCN-2035 54
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Selective Advertisements (DCI)
DC Core(Layer-3 Unicast)
RS
Spine SpineVXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP….Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP….BGW BGW BGW BGW
RR RR
RS – Route Server (eBGP ”Route Reflector”)
Only prefixes of VRF ”Tenant1” and L2VNI 30010 are advertised from Site1 towards DCI. In this example this is Host1.
All prefixes of VRF ”Tenant2” and L2VNI 30020 are notadvertised from Site2 towards DCI. These prefixes are not seen within the DCIVRF
Tenant1
L3VNI: 50001Route-Target: 65501:50001
VIP110.1.1.111
VIP210.2.2.222Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.
2 0000.3010.1101/48 30010, 65599:30010 192.168.10.101/32 50001, 65599:50001 10.1.1.111
Host10000.3010.1101192.168.10.101
L2VNI: 30010 (VLAN 10)L3VNI: 50001 (Tenant1)
L2VNI: 30020 (VLAN 20)L3VNI: 50002 (Tenant2)
Host20000.3020.2101192.168.20.101
BRKDCN-2035 55
Multi-Site Data Plane
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Overlay Data Plane
DC Core(Layer-3 Unicast)
Spine SpineVXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP….Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP….VIP1
10.1.1.111VIP2
10.2.2.222BGW BGW BGW BGW
Host10000.3010.1101192.168.10.101
Host30000.3010.1102192.168.10.102
Host20000.3020.2101192.168.20.101
Intra-site VXLAN Data Plane
Inter-site VXLAN Data Plane
De-capsulation and Re-encapsulation on BGW
De-capsulation and Re-encapsulation on BGW
BRKDCN-2035 57
Multi-Site Packet Walk (BUM)
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (BUM) – Site1
VXLAN EVPNSite2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPNSite1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPNDCI
Baremetal
Host 10000.3010.1101192.168.10.101
Baremetal
Host 20000.3010.1102192.168.10.102
SIP DIP VXLAN SMAC DMAC SIP DIPPayload
L10 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
Bridge
DF30010
DF30010
Host 1 sends a L2 BUM frame
1
2
Leaf10 replicates traffic intra-Site
BRKDCN-2035 59
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (DF & Split Horizon) – Site1
VXLAN EVPNSite2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPNSite1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPNDCI
Baremetal
Host 10000.3010.1101192.168.10.101
Baremetal
Host 20000.3010.1102192.168.10.102
SIP DIP VXLAN SMAC DMAC SIP DIPPayload
L10 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
Bridge
DF30010
DF30010
BUM Forward
Drop due to Split-Horizon rule
Drop due to Designated Forwarder (DF) rule
BRKDCN-2035 60
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (BUM) – DCI
VXLAN EVPNSite2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPNSite1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPNDCI
Baremetal
Host 10000.3010.1101192.168.10.101
Baremetal
Host 20000.3010.1102192.168.10.102
Bridge
DF30010
DF30010
SIP DIP VXLAN SMAC DMAC SIP DIP
PayloadBGW-VIP1 BGW21 30010 H1-MAC ALL-F H1-IP ALL-255
BGW-VIP1 BGW22 30010 H1-MAC ALL-F H1-IP ALL-255
BGW-VIP1 BGW12 30010 H1-MAC ALL-F H1-IP ALL-255
3
BGW11 replicates traffic inter-Sites toward BGW nodes
BUM Forward
BRKDCN-2035 61
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (DF & Split Horizon) – DCI
VXLAN EVPNSite2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPNSite1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPNDCI
Baremetal
Host 10000.3010.1101192.168.10.101
Baremetal
Host 20000.3010.1102192.168.10.102
Bridge
DF30010
DF30010
SIP DIP VXLAN SMAC DMAC SIP DIP
PayloadBGW-VIP1 BGW21 30010 H1-MAC ALL-F H1-IP ALL-255
BGW-VIP1 BGW22 30010 H1-MAC ALL-F H1-IP ALL-255
BGW-VIP1 BGW12 30010 H1-MAC ALL-F H1-IP ALL-255
BUM Forward
BUM Forward
Drop due to Split-Horizon rule
Drop due to Designated Forwarder (DF) rule
BRKDCN-2035 62
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (BUM) – Site2
VXLAN EVPNSite2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPNSite1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPNDCI
Baremetal
Host 10000.3010.1101192.168.10.101
Baremetal
Host 20000.3010.1102192.168.10.102
Bridge
DF30010
DF30010
SIP DIP VXLAN SMAC DMAC SIP DIPPayload
BGW-VIP2 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
4
BGW22 replicates traffic intra-Site
BUM Forward
BRKDCN-2035 63
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (DF & Split Horizon) – Site2
VXLAN EVPNSite2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPNSite1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPNDCI
Baremetal
Host 10000.3010.1101192.168.10.101
Baremetal
Host 20000.3010.1102192.168.10.102
Bridge
DF30010
DF30010
SIP DIP VXLAN SMAC DMAC SIP DIPPayload
BGW-VIP2 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
BUM Forward
Drop due to Split-Horizon rule
Drop due to Designated Forwarder (DF) rule
BRKDCN-2035 64
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (BUM) – Site2
VXLAN EVPNSite2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPNSite1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPNDCI
Baremetal
Host 10000.3010.1101192.168.10.101
Baremetal
Host 20000.3010.1102192.168.10.102
Bridge
DF30010
DF30010
Leaf20 sends traffic to local Host 2
5
BRKDCN-2035 65
Multi-Site Packet Walk (Bridging)
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (Host 1 to Host 2) – Site1
VXLAN EVPNSite2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPNSite1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPNDCI
Baremetal
Host 10000.3010.1101192.168.10.101
Baremetal
Host 20000.3010.1102192.168.10.102
SIP DIP VXLAN SMAC DMAC SIP DIPPayload
L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP
Bridge
Host 1 sends traffic destined to remote Host 2
1
2
Leaf10 performs L2 lookup and encapsulates toward local BGW VIP1 address
VIP2VIP1
BRKDCN-2035 67
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (Host 1 to Host 2) – DCI
VXLAN EVPNSite2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPNSite1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPNDCI
Baremetal
Host 10000.3010.1101192.168.10.101
Baremetal
Host 20000.3010.1102192.168.10.102
SIP DIP VXLAN SMAC DMAC SIP DIPPayload
BGW-VIP1 BGW-VIP2 30010 H1-MAC H2-MAC H1-IP H2-IP
Bridge
3
BGW11 performs L2 lookup and encapsulates toward
remote BGW VIP2 address
VIP2VIP1
BRKDCN-2035 68
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (Host 1 to Host 2) – Site2
VXLAN EVPNSite2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPNSite1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPNDCI
Baremetal
Host 10000.3010.1101192.168.10.101
Baremetal
Host 20000.3010.1102192.168.10.102
Bridge
SIP DIP VXLAN SMAC DMAC SIP DIPPayload
BGW-VIP2 L20 30010 H1-MAC H2-MAC H1-IP H2-IP
4
BGW22 performs L2 lookup and encapsulates toward
destination L20 node
Leaf20 bridges traffic to local Host 2
5
VIP2VIP1
BRKDCN-2035 69
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (Host 2 to Host 1) – Site2
VXLAN EVPNSite2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPNSite1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPNDCI
Baremetal
Host 10000.3010.1101192.168.10.101
Baremetal
Host 20000.3010.1102192.168.10.102
Bridge
SIP DIP VXLAN SMAC DMAC SIP DIPPayload
L20 BGW-VIP2 30010 H2-MAC H1-MAC H2-IP H1-IP
Host 2 replies to remote Host 1
6
7
Leaf20 performs L2 lookup and encapsulates toward local BGW VIP2 address
VIP2VIP1
BRKDCN-2035 70
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (Host 2 to Host 1) – DCI
VXLAN EVPNSite2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPNSite1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPNDCI
Baremetal
Host 10000.3010.1101192.168.10.101
Baremetal
Host 20000.3010.1102192.168.10.102
Bridge
SIP DIP VXLAN SMAC DMAC SIP DIPPayload
BGW-VIP2 BGW-VIP1 30010 H2-MAC H1-MAC H2-IP H1-IP
8
BGW21 performs L2 lookup and encapsulates toward
remote BGW VIP1 address
VIP2VIP1
BRKDCN-2035 71
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-2 (Host 2 to Host 1) – Site1
VXLAN EVPNSite2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPNSite1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPNDCI
Baremetal
Host 10000.3010.1101192.168.10.101
Baremetal
Host 20000.3010.1102192.168.10.102
Bridge
SIP DIP VXLAN SMAC DMAC SIP DIPPayload
BGW-VIP1 L10 30010 H2-MAC H1-MAC H2-IP H1-IP
9
BGW12 performs L2 lookup and encapsulates toward
destination L10 node
Leaf10 bridges traffic toward Host 1
10
VIP2VIP1
BRKDCN-2035 72
Multi-Site Packet Walk (Routing)
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-3 (Host 1 to Host 3) – Site1
VXLAN EVPNSite2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPNSite1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPNDCI
Baremetal
Host 10000.3010.1101192.168.10.101
Baremetal
Host 30000.3010.1102192.168.20.102
SIP DIP VXLAN SMAC DMAC SIP DIPPayload
L10 BGW-VIP1 50001 L10-MAC BGW-VMAC1 H1-IP H3-IP
Route
Host 1 sends a data packet to the remote
Host 3
1
2
Leaf10 performs a L3 lookup and encapsulates toward local BGW VIP1 address
VIP2VMAC2
VIP1VMAC1
BRKDCN-2035 74
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-3 (Host 1 to Host 3) – DCI
VXLAN EVPNSite2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPNSite1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPNDCI
Baremetal
Host 10000.3010.1101192.168.10.101
Baremetal
Host 30000.3010.1102192.168.20.102
Route
SIP DIP VXLAN SMAC DMAC SIP DIPPayload
BGW-VIP1 BGW-VIP2 50001 BGW-VMAC1 BGW-VMAC2 H1-IP H3-IP
3
BGW11 performs a L3 lookup and encapsulates toward
remote BGW VIP2 address
VIP2VMAC2
VIP1VMAC1
BRKDCN-2035 75
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Packet Walk – Layer-3 (Host 1 to Host 3) – Site2
VXLAN EVPNSite2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPNSite1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPNDCI
Baremetal
Host 10000.3010.1101192.168.10.101
Baremetal
Host 30000.3010.1102192.168.20.102
Route
SIP DIP VXLAN SMAC DMAC SIP DIPPayload
BGW-VIP2 L20 50001 BGW-VMAC1 L20-MAC H1-IP H3-IP
4
BGW21 performs a L3 lookup and encapsulates
toward destination L20 node
Leaf20 routes traffic to local Host 3
5
VIP2VMAC2
VIP1VMAC1
BRKDCN-2035 76
Multi-Site and Failure Detection on BGW
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Steady State Traffic – Site1
VXLAN EVPNSite2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPNSite1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPNDCI
Baremetal
Host 10000.3010.1101192.168.10.101
Baremetal
Host 20000.3010.1102192.168.10.102
SIP DIP VXLAN SMAC DMAC SIP DIPPayload
L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP
VIP2VIP1
BRKDCN-2035 78
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DCI Link Failure BGW12 – Site1
VXLAN EVPNSite2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPNSite1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPNDCI
Baremetal
Host 10000.3010.1101192.168.10.101
Baremetal
Host 20000.3010.1102192.168.10.102
SIP DIP VXLAN SMAC DMAC SIP DIPPayload
L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP
� On DCI Link Failure (i.e. BGW12)
• Virtual IP (VIP) on BGW is disabled
• BGW will stop participating in DF election
• BGW acts like a Leaf (Layer-3 only)
• Traffic towards others Sites is served by remaining BGWs (i.e. BGW11)
BGW12 gets isolated from the DCI Core Network
Intra-site VXLAN traffic re-routing
VIP2VIP1
BRKDCN-2035 79
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric Link Failure BGW12 – Site1
VXLAN EVPNSite2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPNSite1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPNDCI
Baremetal
Host 10000.3010.1101192.168.10.101
Baremetal
Host 20000.3010.1102192.168.10.102
SIP DIP VXLAN SMAC DMAC SIP DIPPayload
L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP
VIP2VIP1
BRKDCN-2035 80
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric Link Failure BGW12 – Site1
VXLAN EVPNSite2
VTEP
Leaf20
VTEP
BGW21
VTEP
BGW22
VXLAN EVPNSite1
VTEP
Leaf10
VTEP
BGW11
VTEP
BGW12
VXLAN EVPNDCI
Baremetal
Host 10000.3010.1101192.168.10.101
Baremetal
Host 20000.3010.1102192.168.10.102
SIP DIP VXLAN SMAC DMAC SIP DIPPayload
L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP
VIP2VIP1
BGW12 gets isolated from the Spine nodes
Intra-site VXLAN traffic re-routing
� On Fabric Link Failure (i.e. BGW12)
• Virtual IP (VIP) on BGW is disabled
• Primary IP (PIP) on BGW is disabled
• BGW will stop participating in the Overlay
BRKDCN-2035 81
Multi-Site Setup Walkthrough
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
Site 1 Setup – Enable Border Gateway
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP….BGW1 BGW2
� Multi-Site Commands are marked in red
� Various options do exist but the recommended design choices are:
• Fabric Internal
¾ IGP Underlay, iBGP Overlay
• DCI (primary choice)
¾ eBGP Underlay, eBGP Overlay
¾ Route Server for DCI Overlay peerings
¾ DC Core for reachability across n Sites
• DCI (alternative option)
¾ Any Routing Protocol Underlay, eBGP Overlay
¾ Full-Mesh for DCI Overlay peerings
¾ Back-to-Back Site Reachability (physical, full-mesh)
BRKDCN-2035 83
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
Site 1 Setup – Enable Border Gateway
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP….BGW1 BGW2
feature nv overlaynv overlay evpn
feature bgpfeature interface-vlanfeature vn-segment-vlan-based
evpn multisite border-gateway
BGW2BGW1
BRKDCN-2035 84
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
Site 1 Setup – BGW 1 Loopback & VTEP
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP….BGW1
interface loopback1description PIP VTEPip address 10.1.1.101/32 tag 12345ip router ospf UNDERLAY area 0.0.0.0ip pim sparse-mode
interface loopback100description VIP Multi-Site 1ip address 10.1.1.111/32 tag 12345ip router ospf UNDERLAY area 0.0.0.0ip pim sparse-mode
interface loopback0description RIDip address 10.10.10.101/32 tag 12345ip router ospf UNDERLAY area 0.0.0.0ip pim sparse-mode
BGW1
BRKDCN-2035 85
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
Site 1 Setup – BGW 2 Loopback & VTEP
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP…. BGW2
interface loopback1description PIP VTEPip address 10.1.1.102/32 tag 12345ip router ospf UNDERLAY area 0.0.0.0ip pim sparse-mode
interface loopback100description VIP Multi-Site 1ip address 10.1.1.111/32 tag 12345ip router ospf UNDERLAY area 0.0.0.0ip pim sparse-mode
interface loopback0description RIDip address 10.10.10.102/32 tag 12345ip router ospf UNDERLAY area 0.0.0.0ip pim sparse-mode
BGW2
BRKDCN-2035 86
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
Site 1 Setup – Fabric Link Tracking BGW 1
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP….BGW1
interface Ethernet1/53description TO-SPINE1ip address 10.0.1.1/30ip router ospf UNDERLAY area 0.0.0.0ip pim sparse-modeevpn multisite fabric-tracking
interface Ethernet1/54description TO-SPINE2ip address 10.0.2.1/30ip router ospf UNDERLAY area 0.0.0.0ip pim sparse-modeevpn multisite fabric-tracking
BGW1
Allows to bring down the PIP/VIP loopback interfaces when the
BGW is isolated from the spines
BRKDCN-2035 87
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
Site 1 Setup – Fabric Link Tracking BGW 2
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP…. BGW2
interface Ethernet1/53description TO-SPINE1ip address 10.0.1.5/30ip router ospf UNDERLAY area 0.0.0.0ip pim sparse-modeevpn multisite fabric-tracking
interface Ethernet1/54description TO-SPINE2ip address 10.0.2.5/30ip router ospf UNDERLAY area 0.0.0.0ip pim sparse-modeevpn multisite fabric-tracking
BGW2
BRKDCN-2035 88
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core(Layer-3 Unicast)
Fabric
Site 1 Setup – Multi-Site Underlay Interface
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP….BGW1 BGW2
interface Ethernet1/1description TO-DC-CORE1ip address 10.111.111.1/30 tag 12345evpn multisite dci-tracking
interface Ethernet1/2description TO-DC-CORE2ip address 10.111.222.1/30 tag 12345evpn multisite dci-tracking
BGW1
DCI
interface Ethernet1/1description TO-DC-CORE1ip address 10.222.111.1/30 tag 12345evpn multisite dci-tracking
interface Ethernet1/2description TO-DC-CORE2ip address 10.222.222.1/30 tag 12345evpn multisite dci-tracking
BGW2
Allows to bring down the PIP/VIP loopback interfaces when the BGW is isolated from the DC core BRKDCN-2035 89
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core(Layer-3 Unicast)
Fabric
Site 1 BGW 1 Setup – Multi-Site Overlay Peering
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP….BGW1
DCI
RS router bgp 65501router-id 10.10.10.101address-family ipv4 unicastredistribute direct route-map REDIST-LOCAL
neighbor 10.111.111.2remote-as 65599update-source ethernet1/1address-family ipv4 unicast
neighbor 10.111.222.2remote-as 65599update-source ethernet1/2address-family ipv4 unicastneighbor 10.99.99.201remote-as 65599update-source loopback0ebgp-multihop 5peer-type fabric-externaladdress-family l2vpn evpnrewrite-evpn-rt-asnsend-communitysend-community both
BGW1
RS – Route Server (eBGP ”Route Reflector”) BRKDCN-2035 90
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core(Layer-3 Unicast)
Fabric
Site 1 BGW 2 Setup – Multi-Site Overlay Peering
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP….DCI
RS router bgp 65501router-id 10.10.10.102address-family ipv4 unicastredistribute direct route-map REDIST-LOCAL
neighbor 10.222.111.2remote-as 65599update-source ethernet1/1address-family ipv4 unicast
neighbor 10.222.222.2remote-as 65599update-source ethernet1/2address-family ipv4 unicastneighbor 10.99.99.201remote-as 65599update-source loopback0ebgp-multihop 5peer-type fabric-externaladdress-family l2vpn evpnrewrite-evpn-rt-asnsend-communitysend-community both
BGW1
RS – Route Server (eBGP ”Route Reflector”)
BGW2
BRKDCN-2035 91
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core(Layer-3 Unicast)
Fabric
Site 1 Setup – Multi-Site Overlay Peering
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP….BGW1
DCI
RS
RS – Route Server (eBGP ”Route Reflector”)
BGW2
� peer-type fabric-external
• Enables Next-Hop Rewrite for Multi-Site
• Defines Site External BGP neighbors for EVPN exchange
� rewrite-evpn-rt-asn
• Rewrites Route-Target Auto information to simplify MAC-VRF and IP-VRF configuration
• Normalizes outgoing Route-Targets AS number to match remote AS number
• Uses BGP configured Neighbors Remote AS
BRKDCN-2035 92
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Site 1 Setup – Multi-Site Overlay Peering
DC Core(Layer-3 Unicast)
Spine SpineVXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP….Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP….VIP1
10.1.1.111VIP2
10.2.2.222BGW BGW BGW BGW
Host10000.3010.1101192.168.10.101
peer-type fabric-external peer-type fabric-external
BGP Update:MAC: 0000.3010.1101 (L2VNI 30001)IP: 192.168.20.101 (L3VNI 50001)NH: 10.2.2.222RMAC: BGW-VMAC2
BGP Update:MAC: 0000.3010.1101 (L2VNI 30001)IP: 192.168.20.101 (L3VNI 50001)NH: 10.1.1.111RMAC: BGW-VMAC1
Rewrite Next-Hop IP and Next-Hop MAC (RMAC) based on Neighbor Site BGW
BGP Update:MAC: 0000.3010.1101 (L2VNI 30001)IP: 192.168.20.101 (L3VNI 50001)NH: 10.1.1.1RMAC: Leaf1
Rewrite Next-Hop IP and Next-Hop MAC (RMAC) based on Neighbor Site BGW
BRKDCN-2035 93
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Site 1 Setup – Multi-Site Overlay Peering
DC Core(Layer-3 Unicast)
Spine SpineVXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP….Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP….VIP1
10.1.1.111VIP2
10.2.2.222BGW BGW BGW BGW
Host10000.3010.1101192.168.10.101
Host20000.3020.2101192.168.20.101
rewrite-evpn-rt-asn rewrite-evpn-rt-asn
BGP Update:Remote AS: 65502VNI: 50001Route-Target: 65502:50001
Rewrite Route-Target based on BGP Neighbors Remote ASN
BGP Update:Remote AS : 65501VNI: 50001Route-Target: 65501:50001
BGP Update:Remote AS: 65502VNI: 50001Route-Target: 65502:50001
BRKDCN-2035 94
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core(Layer-3 Unicast)
Fabric
Site 1 Setup – Anycast BGW VTEP Configuration
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP….BGW1
DCI
RS – Route Server (eBGP ”Route Reflector”)
interface nve1no shutdownhost-reachability protocol bgpmultisite ethernet-segment 7system-mac 0000.0000.0001source-interface loopback1multisite border-gateway interface loopback100member vni 30010multisite ingress-replicationmcast-group 239.1.1.1
member vni 30011-30020mcast-group 239.1.1.2member vni 50001 associate-vrf
BGW1
BGW2
BGW2
BRKDCN-2035 95
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core(Layer-3 Unicast)
Fabric
Site 1 Setup – Anycast BGW VTEP Configuration
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP….BGW1
DCI
BGW2
� multisite ethernet-segment
• Defines the discriminator for Sites in a common Domain � system-mac
• Defines the Multi-Site Site-Id (6 octets hex)� multisite border-gateway interface loopback#
• Defines the Loopback Interface used for the Border Gateway Virtual IP Address (VIP)
� multisite ingress-replication
• Per-VNI knob for extending Layer-2 VNI
• Defines the Multi-Site BUM Replication methodBRKDCN-2035 96
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core(Layer-3 Unicast)
Fabric
Site 1 Setup – Multi-Site Overlay Traffic Policy
Spine Spine
VTEP VTEP VTEP VTEP
VTEP VTEP….BGW1
DCI
BGW2
• BUM Traffic Policing
• Limits Broadcast, Unknown Unicast and Layer-2 Multicast Traffic across Multi-Site
• Level 0 = No B/U/M Forwarding
• Level 100 = All B/U/M Forwarding Forwarding• Enforced on Encapsulation towards remote Sites
evpn storm-control broadcast level 10evpn storm-control unicast level 10evpn storm-control multicast level 10
BGW1 BGW2
BRKDCN-2035 97
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1 Setup – Multi-Site Overlay Traffic Policy
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site n
VTEP VTEP
Overlay Multi-Site
Overlay Site 1 Overlay Site n
Baremetal
BUM
BGW BGW BGW BGW
BRKDCN-2035 98
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1 Setup – Multi-Site Overlay Traffic Policy
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site n
VTEP VTEP
Overlay Multi-Site
Overlay Site 1 Overlay Site n
Baremetal
BUM
Storm ControlBroadcast 0-100%Unknown Unicast 0-100%Multicast 0-100%
Storm ControlBroadcast 0-100%Unknown Unicast 0-100%Multicast 0-100%
BGW BGW BGW BGW
BRKDCN-2035 99
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1 Setup – Multi-Site BUM Replication Modes
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site n
VTEP VTEP
Overlay Multi-Site
Overlay Site 1 Overlay Site n
Multicast Multicast
Ingress Replication
BGW BGW BGW BGW
BRKDCN-2035 100
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1 Setup – Multi-Site BUM Replication Modes
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site n
VTEP VTEP
Overlay Multi-Site
Overlay Site 1 Overlay Site n
Ingress Replication Ingress Replication
Ingress Replication
BGW BGW BGW BGW
BRKDCN-2035 101
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Site 1 Setup – Multi-Site BUM Replication Modes
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site n
VTEP VTEP
Overlay Multi-Site
Overlay Site 1 Overlay Site n
Ingress Replication Multicast
Ingress Replication
BGW BGW BGW BGW
BRKDCN-2035 102
Connectivity to the External Layer 3 Domain
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Connectivity to the External Layer 3 Domain
� The BGW nodes can be used to provide Layer-3 external connectivity to each site
� Different connectivity models are supported• VRF-Lite peering with an external pair of WAN Edge routers• MP-BGP EVPN peering with the external WAN Edge routers (GOLF)• Dedicated or shared pair of WAN Edge routers across sites
� External Layer-3 network may be different from the DCI network used for inter-site communication
BRKDCN-2035 104
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Border Gateway and VRF-Lite
Spine SpineVXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP….Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP….VIP1
10.1.1.111VIP2
10.2.2.222BGW BGW BGW BGW
Host10000.3010.1101192.168.10.101
Host30000.3010.1102192.168.10.102
Host20000.3020.2101192.168.20.101
VRF-CVRF-BVRF-ASeparate routing peering for each VRF (IGP or eBGP)
Dedicated interface (logical or physical) for each VRF
BRKDCN-2035 105
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric
DCI
Multi-Site – Border Gateway and GOLF
Spine SpineVXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP….Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP….VIP1
10.1.1.111VIP2
10.2.2.222BGW BGW BGW BGW
Host10000.3010.1101192.168.10.101
Host30000.3010.1102192.168.10.102
Host20000.3020.2101192.168.20.101
VRF-CVRF-BVRF-ASingle MP-BGP EVPN instance to exchange routes for all VRFs
VXLAN Data Plane between BGW and WAN Edge Router
BRKDCN-2035 106
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core(Layer-3 Unicast)
MPLSL3VPN
Fabric
DCI
Spine SpineVXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP….Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP….VIP1
10.1.1.111VIP2
10.2.2.222BGW BGW BGW BGW
Host10000.3010.1101192.168.10.101
Host20000.3020.2101192.168.20.101
Host30000.3010.1102192.168.10.102
Multi-Site – Shared Internet/WAN GatewaysInternet/WAN
BorderPE BorderPE
Inter-Site VXLAN Communication between Border Gateways
BRKDCN-2035 107
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Site – Per Site Internet/WAN Gateway
DC Core(Layer-3 Unicast)
MPLSL3VPN
Fabric
DCI
Spine SpineVXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP….Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP….VIP1
10.1.1.111VIP2
10.2.2.222BGW BGW BGW BGW
Host10000.3010.1101192.168.10.101
Host20000.3020.2101192.168.20.101
Host30000.3010.1102192.168.10.102
Internet/WANBorderPE BorderPE BorderPE BorderPE
Inter-Site VXLAN Communication between Border Gateways
BRKDCN-2035 108
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
MPLSL3VPN
Fabric
DCI
Spine SpineVXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP….Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP….VIP1
10.1.1.111VIP2
10.2.2.222BGW BGW BGW BGW
Host10000.3010.1101192.168.10.101
Host20000.3020.2101192.168.20.101
Host30000.3010.1102192.168.10.102
Internet/WAN BorderPE BorderPE BorderPE BorderPE
Multi-Site – Consolidated WAN and DCI NetworkPerform simple routing for inter-site flows, VXLAN (or VRF-Lite) to MPLS VPN hand-off for north-south communication
Inter-Site VXLAN Communication between Border Gateways
BRKDCN-2035 109
Ingress and Egress Traffic Optimization
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
BGW
WAN
Active FWActive FW
� The stretching of Layer-2 domains across separate sites may lead to the creation of asymmetric traffic paths
� Deploying independent stateful services (like FWs) across sites would result in traffic drops
� In this case it is required to ensure the symmetry of ingress and egress communication paths
Ingress and Egress Traffic OptimizationThe Issue of Extending Layer 2 Domains
BGW BGW BGW
DC Core(Layer-3 Unicast)
BRKDCN-2035 111
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core(Layer-3 Unicast)
� Guarantee routing symmetry with the outside of the Data Center• Egress Æ Always prefer the local BGW• Ingress Æ Steer traffic to the specific destination
End-Point’s location
� Maintain optimal routing over the dedicated DCI network (if existing) for Server-to-Server traffic• The DC fabric must discriminate between DC
and WAN destinations
� If required provide a fallback path via DCI for WAN isolation situations
Spine SpineVXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
Spine SpineVXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
BGW BGW BGW BGW
WAN
Ingress and Egress Traffic OptimizationMaintaining Traffic Symmetry over Optimal Paths
BRKDCN-2035 112
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core(Layer-3 Unicast)
MPLSL3VPN
Spine Spine
VXLAN EVPN
Site1VTEP VTEP VTEP VTEP
VTEP VTEP
Spine Spine
VXLAN EVPN
Site2VTEP VTEP VTEP VTEP
VTEP VTEPVIP210.2.2.222
BGW BGW BGW BGW
Host10000.3010.1101192.168.10.101
Host30000.3010.1102192.168.10.102
BorderPE BorderPE BorderPE BorderPE
VIP110.1.1.111
Multi-Site – Egress Path Optimization172.16.1.10
172.16.1.0/24 Æ Border-PEs 1-2 172.16.1.0/24 Æ Border-PEs 3-4
172.16.1.0/24 Æ VIP1 172.16.1.0/24 Æ VIP2
eBGP-EVPN
Less preferred advertisement of 172.16.1.0 because of longer AS-Path
BRKDCN-2035 113
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core(Layer-3 Unicast)
MPLSL3VPN
Spine Spine
VXLAN EVPN
Site1VTEP VTEP VTEP VTEP
VTEP VTEP
Spine Spine
VXLAN EVPN
Site2VTEP VTEP VTEP VTEP
VTEP VTEPVIP210.2.2.222
BGW BGW BGW BGW
Host10000.3010.1101192.168.10.101
Host30000.3010.1102192.168.10.102
BorderPE BorderPE BorderPE BorderPE
VIP110.1.1.111
Multi-Site – Egress Path Optimization172.16.1.10
eBGP-EVPNOptimized Egress Traffic Path
Optimized Egress Traffic Path
BRKDCN-2035 114
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core(Layer-3 Unicast)
MPLSL3VPN
Spine Spine
VXLAN EVPN
Site1VTEP VTEP VTEP VTEP
VTEP VTEP
Spine Spine
VXLAN EVPN
Site2VTEP VTEP VTEP VTEP
VTEP VTEPVIP210.2.2.222
BGW BGW BGW BGW
Host10000.3010.1101192.168.10.101
Host30000.3010.1102192.168.10.102
BorderPE BorderPE BorderPE BorderPE
VIP110.1.1.111
Multi-Site – Egress Path OptimizationWAN Isolation Scenario
172.16.1.10
172.16.1.0/24 Æ Border-PEs 3-4
172.16.1.0/24 Æ VIP1 172.16.1.0/24 Æ VIP2
eBGP-EVPN
WAN Isolation Scenario
172.16.1.0/24 Æ VIP2
BRKDCN-2035 115
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core(Layer-3 Unicast)
MPLSL3VPN
Spine Spine
VXLAN EVPN
Site1VTEP VTEP VTEP VTEP
VTEP VTEP
Spine Spine
VXLAN EVPN
Site2VTEP VTEP VTEP VTEP
VTEP VTEPVIP210.2.2.222
BGW BGW BGW BGW
Host10000.3010.1101192.168.10.101
Host30000.3010.1102192.168.10.102
BorderPE BorderPE BorderPE BorderPE
VIP110.1.1.111
Multi-Site – Ingress Path Optimization
192.168.10.0/24 Æ BGW 1-2192.168.10.101/32 Æ BGW-1-2
192.168.10.0/24 Æ BGW 3-4192.168.10.102/32 Æ BGW 3-4
192.168.10.101/32 Æ Leaf1 192.168.10.102/32 -> Leaf3
eBGP-EVPN
Host routes advertised across sites but NOT re-advertised toward the local Border-PEs
192.168.10.0/24 Æ Border-PE 1-4192.168.10.101/32 Æ Border-PE 1-2192.168.10.102/32 Æ Border-PE 3-4Host routes
advertisement in the WAN
Deploying LISP on the Border-PEs is a viable alternative to host routes advertisement
Filter out host routes received from remote sites. Only announce local host route information
BRKDCN-2035 116
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core(Layer-3 Unicast)
MPLSL3VPN
Spine Spine
VXLAN EVPN
Site1VTEP VTEP VTEP VTEP
VTEP VTEP
Spine Spine
VXLAN EVPN
Site2VTEP VTEP VTEP VTEP
VTEP VTEPVIP210.2.2.222
BGW BGW BGW BGW
Host10000.3010.1101192.168.10.101
Host30000.3010.1102192.168.10.102
BorderPE BorderPE BorderPE BorderPE
VIP110.1.1.111
Multi-Site – Ingress Path Optimization
eBGP-EVPN
192.168.10.0/24 Æ Border-PE 1-4192.168.10.101/32 Æ Border-PE 1-2192.168.10.102/32 Æ Border-PE 3-4
Optimized Ingress Traffic Path
Optimized Ingress Traffic Path
BRKDCN-2035 117
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
DC Core(Layer-3 Unicast)
MPLSL3VPN
Spine Spine
VXLAN EVPN
Site1VTEP VTEP VTEP VTEP
VTEP VTEP
Spine Spine
VXLAN EVPN
Site2VTEP VTEP VTEP VTEP
VTEP VTEPVIP210.2.2.222
BGW BGW BGW BGW
Host10000.3010.1101192.168.10.101
Host30000.3010.1102192.168.10.102
BorderPE BorderPE BorderPE BorderPE
VIP110.1.1.111
eBGP-EVPN
192.168.10.0/24 Æ Border-PE 3-4192.168.10.101/32 Æ Border-PE 1-2192.168.10.102/32 Æ Border-PE 3-4
Multi-Site – Ingress Path OptimizationWAN Isolation Scenario
WAN Isolation Scenario
BRKDCN-2035 118
Network Services Integration
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Network Services Integration
� Couple of different options where to connect network services:1. Service Leaf nodes: recommended to connect devices used for east-
west communication2. Border Gateway Nodes: used to connect network services for north-
south traffic flows
� Depending on the specifics of the Multi-Site deployment, the following deployment models would be possible:• Active/Standby Service Nodes pair connected to different sites• Active/Active cluster of Service Nodes deployed across sites• Independent Active/Standby Service nodes pairs deployed in separate
sites
BRKDCN-2035 120
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Network Services IntegrationActive/Standby Pair Deployed across Sites
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
BGW
WAN
Standby FWActive FW
BGW BGW BGW
DC Core(Layer-3 Unicast)
� Requirement to extend Layer 2 communication between Active/Standby nodes for keep-alivesand state information exchange� Perimeter service nodes connected to VPC Border
Gateways
� Ingress and egress traffic always traversing the Active node in Site 1� No issues related to the creation of asymmetric
traffic paths
� East-West flows must be hair-pinned to the active FW connected to the Service leaf nodes in Site 1• Need to properly dimension bandwidth in the DC
Core to accommodate for this extra traffic
Active FW Standby FWBaremetal BaremetalBaremetal
North-South traffic flows
North-South traffic flows
East-West traffic flows
BRKDCN-2035 121
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Network Services IntegrationActive/Active Cluster of Service Nodes Deployed across Sites
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
BGW
WAN
Active/Active FW Cluster
BGW BGW BGW
DC Core(Layer-3 Unicast)
� Requirement to extend Layer 2 communication between Active/Active nodes for intra-cluster communication and traffic redirection� Perimeter service nodes connected to VPC Border
Gateways
� Asymmetric traffic issues taken care by native intra-cluster traffic redirection
� Option to deploy ingress/egress optimization technique to avoid inter-site traffic hair-pinning
� Service node cluster integration not supported at FCS and planned for a future SW release
Active/Active FW Cluster
Logical Intra-Cluster Link (ICL)
Baremetal
North-South traffic flows
BRKDCN-2035 122
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Network Services IntegrationIndependent Active/Standby Pair Deployed in Separate Sites
Spine Spine
VXLAN EVPN
Site1
VTEP VTEP VTEP VTEP
VTEP VTEP
Spine Spine
VXLAN EVPN
Site2
VTEP VTEP VTEP VTEP
VTEP VTEP
BGW
WAN
Active/Standby FW
Active/Standby FW
BGW BGW BGW
DC Core(Layer-3 Unicast)
� Mandates the deployment of Ingress/Egress traffic optimization to avoid creation of asymmetric traffic path for north-south communication
� Active/Standby nodes can use direct links to sync state� Perimeter service nodes can connected to Anycast
Border Gateways
� Active/Standby pair (or cluster) still required for service nodes used for east-west traffic flows
Baremetal Baremetal
Active FW Standby FW
North-South traffic flows
North-South traffic flows
BRKDCN-2035 123
Legacy Site Integration
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Site and Legacy Site Integration
� Extend Layer-2 and Layer-3 connectivity between sites • Coexistence and/or application migration use cases
� Proposed approach is to deploy a pair of ‘remote’ VPC Border Gateways in the legacy site• Offers native Multi-Site functionalities (BUM containment, etc) to the legacy site
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Greenfield Site
VTEP VTEP
Legacy Site
VTEP VTEP
BGW BGW BGW BGW
Pair of VPC Border Gateways
BRKDCN-2035 125
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Site and Legacy Site IntegrationLayer-2 Connectivity with the ‘Remote’ BGW
Legacy Site
VTEP VTEP
BGW BGW
Legacy Aggregation Layer devices support MLAG
Single logical link to extend VLANs toward the Greenfield VXLAN EVPN site
VLANs mapped to L2VNIs on the Border Gateways
Legacy Site
VTEP VTEP
BGW BGW
Legacy Aggregation Layer devices do not support MLAG
Single port-channel from each aggregation layer device
VLANs mapped to L2VNIs on the Border Gateways
Recommended to move the STP root to the BGW devices
BRKDCN-2035 126
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Site and Legacy Site IntegrationLayer-2 Control Plane Exchange across Sites
Baremetal
Host 10000.3010.1101192.168.10.101
Baremetal
Host 20000.3010.1102192.168.10.102
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Greenfield Site
VTEP VTEP
Legacy Site
VTEP VTEP
BGW BGW BGW BGW
VIP110.1.1.111
VIP210.2.2.222
MAC NH
0000.3010.1101 Leaf1
0000.3010.1102 VIP2
MAC NH
0000.3010.1101 VIP1
0000.3010.1102 Po1
Po1
All End-Points in the legacy site are learned as directly connected to the BGW
eBGP-EVPN
BRKDCN-2035 127
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
� Integration between Anycast Gateway and legacy default gateway (HSRP, VRRP, etc.) not initially supported with VXLAN Multi-Site
� First option is to keep on the legacy network the active default gateway for the stretched IP subnets
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Greenfield Site
VTEP VTEP
Legacy Site
VTEP VTEP
BGW BGW BGW BGW
Default Gateway deployed on the legacy aggregation devices
Multi-Site and Legacy Site IntegrationDefault Gateway Deployment – Option 1
L3
L2
Greenfield VXLAN EVPN Fabric only offers L2 services for the stretched IP subnets
BRKDCN-2035 128
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
� Recommended approach is to migrate the default gateway from the legacy aggregation devices to the Border Gateways (VXLAN EVPN Anycast Gateway)
� Optimize routing between End-Points deployed across sites
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Greenfield Site
VTEP VTEP
Legacy Site
VTEP VTEP
BGW BGW BGW BGW
Default Gateway migrated to the Border Gateways (VXLAN EVPN Anycast Gateway)
Multi-Site and Legacy Site IntegrationDefault Gateway Deployment – Option 2
L3
L2
Greenfield VXLAN EVPN Fabric offers L2 and L3 services for the stretched IP subnets
Legacy infrastructure offers only L2 services
Distributed Anycast Gateway function
L3
L2
BRKDCN-2035 129
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Multi-Site and Legacy Site IntegrationLayer-3 Control Plane Exchange across Sites
Baremetal
Host 10000.3010.1101192.168.10.101
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Greenfield Site
VTEP VTEP
Legacy Site
VTEP VTEP
BGW BGW BGW BGW
VIP110.1.1.111
VIP210.2.2.222
Po1
All End-Points in the legacy site are learned as directly connected to the BGW
eBGP-EVPN
L3
L2
Baremetal
Host 30000.3010.1102192.168.20.101
IP NH
192.168.10.101 Leaf1
192.168.20.101 VIP1
IP L3VNI
192.168.10.101 VIP1
192.168.20.101 Po1
BRKDCN-2035 130
Migration to Multi-Site
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Migration to Multi-SiteUse Cases
1. Site addition: need to connect a Greenfield VXLAN EVPN Fabric to an existing VXLAN EVPN Fabric built with 1st
generation Nexus 9000
2. Migrating a VXLAN Multi-Pod Fabric to Multi-Site
3. Migrating a VXLAN Multi-Fabric design to Multi-Site
BRKDCN-2035 132
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Migration to Multi-SiteSite Addition
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Existing VXLAN EVPN Fabric
VTEP VTEP
BGW BGW
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Greenfield Fabric
VTEP VTEP
BGW BGW
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Existing VXLAN EVPN Fabric
� Step 1: add a pair of Border Gateways to the existing VXLAN EVPN Fabric, running the proper SW release supporting Multi-Site
Note: no requirement to change the HW/SW version on existing leaf nodes
� Step 2: connect the BGW to the inter-site network and establish control plane peering with the BGW in the Greenfield Fabric
� Step 3: configure on the BGW the L2VNIs and L3VNIs to be extended
eBGP-EVPN
BRKDCN-2035 133
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Migration to Multi-SiteMulti-Fabric to Multi-Site
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Site 2
VTEP VTEP
No Underlay Extension
BGW BGW BGW BGW
� Step 1: add a pair of Border Gateways to each Pod (if needed) and connect them to the spines and to the inter-site network
� Step 2: upgrade the SW on both Fabrics BGW to be able to support Multi-Site
� Step 3: establish control plane adjacencies across sites
� Step 4: disconnect the previously used DCI and extend Layer-2 and Layer-3 across Multi-Site
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Fabric 1
VTEP VTEP
SpineSpine Spine Spine
VTEP VTEPVTEP VTEPVTEP VTEP VTEP
Fabric 2
VTEP VTEP
BRKDCN-2035 134
Conclusion
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
� Multiple Overlay Domains – Interconnected & Controlled• Scaling and Segregating VXLAN EVPN Networks
� Multiple Overlay Control-Plane Domains – Interconnected & Controlled• Limited Overlay Control-Plane Update Propagation
� Multiple Underlay Domains - Isolated• Isolated Underlay Domains – No need for Extension
� Multiple Replication Domains for BUM – Interconnected & Controlled• Individual BUM flooding domain with Traffic control
Multi-Site Advantages – ”The Multiple”
BRKDCN-2035 136
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
• New IETF Draft for Multi-Site Design
• Multi-site EVPN based VXLAN using Border Gateways
• https://tools.ietf.org/html/draft-sharma-multi-site-evpn
VXLAN EVPN – Multi-Site
BRKDCN-2035 137
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 gift card.
• Complete your session surveys through the Cisco Live mobile app or on www.CiscoLive.com/us.
Complete Your Online Session Evaluation
Don’t forget: Cisco Live sessions will be available for viewing on demand after the event at www.CiscoLive.com/Online.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
BRKDCN-2035 139
Thank you