Real World Fabric Based Deploymentd2zmdbbm9feqrf.cloudfront.net/2016/anz/pdf/BRKDCT-2083.pdfInter-DC Core (Layer-3 IP/MPLS) BGP AS#65500 ... Building Your IP Network –Interface Principles

Real World Fabric Based Network Design and

DeploymentDavid Jansen, Distinguished Systems Engineer

[email protected]

• Requirements

• Fabric Roles and Definitions

• Fabricpath/DFA Customer Deployment

• VXLAN Customer Deployment

• ACI Customer deployment

• Conclusion

Agenda

Requirements

Business Drivers & Solutions for Network Segmentation

SOLUTIONS

VRF

L3VPN

Multicast VPN

Multi-tenancy

Sharedservices

Compliance

Mergers

Acquisitions

• Multi-tenancy

• Security and Separation

• Traffic Engineering

• Scalable

• Flexible topology

• Minimise oversubscription

• Scale out and scale up

• Scalable L4-7 Service Layer

• No spanning tree

• Incremental scale

• Virtual FW/LB per tenant

• Flexible placement

• Incremental capacity

MAN/WAN

FabricPath

/BGP

MAN/WAN

VXLAN

/EVPN

Data Centre “Fabric” Journey

STP

VPC

MAN/WA

N

FabricPath

VXLAN

ACI Fabric

Application Policy

Infrastructure

Controller

APIC

Application Centric Infrastructure

Fabric Roles and Definitions

Leaf and Spine Topology – Device Roles

spine spine

leaf leafleaf leaf leaf leaf leaf leaf Border leaf

MAN/WAN

• Spine

• Interconnecting Leafs and Border Leafs

• IP Forwarder (East / West)

• Route-Reflector (RR) for EVPN

• Rendezvous-Point (RP) for Underlay

• Does not require VTEP

• Leaf (VTEP)

• VXLAN Edge-Device

• Route/Bridges Classic Ethernet frames &

encapsulates them into VXLAN

• Requires VTEP

• Virtual Machines

• Physical Machines

• FEX

• 3rd-party Switches

• UCS FI

• Blade Switches

• Border Leaf (VTEP)

• External Connectivity

Border-Leaf Topology – Device Roles

VRF OSPF Process

External Router

Border-Leaf

EVPN

Overlay

Tenant

VRF A

EVPN

Overlay

Tenant

VRF B

EVPN

Overlay

Tenant

VRF C

VRFA VRFB VRFC

EVPN

Overlay

Tenant

VRF A

EVPN

Overlay

Tenant

VRF B

EVPN

Overlay

Tenant

VRF C

VRFBVRF

AVRFC

• Border Leaf (VTEP)

• VXLAN Edge-Device

• Route and Bridges Classic Ethernet frames from an outside network and

encapsulates them into VXLAN (North/South)

• Internetworking of LISP/MPLS traffic from an outside network and re-

encapsulates it into VXLAN (North/South)

• Speaks IGP/EGP routing protocols with the outside network (North/South)

• Requires VTEP

• IPv4/IPv6 routes are exchanged with the external neighbour through the

IPv4/IPv6 unicast address family within the VRF

• Interface options: Physical Routed Ports, sub-interfaces, VLAN SVIs over Trunk

Ports

Services Leaf – Device Role

border spine

leaf leafleaf leaf leaf leaf leaf leaf

border spine

• Services leaf (VTEP)

• Firewalls

• Load balancers

• Proxy services

• IPS services

leaf leaf

Note: the different leaf roles are logical

and not physical. The same leaf switch

could perform all three functions

(regular, services, and border leaf)

Border Spine Topology – Device Roles

border spine

leaf leafleaf leaf leaf leaf leaf leaf

MAN/WAN

border spine

• Border Spine (VTEP)• Interconnecting Leafs and Border Leafs

• External Connectivity

• VXLAN Edge Device

• Route and Bridges Classic Ethernet frames from an outside network and encapsulates them into VXLAN (North/South)

• Decapsulates MPLS/LISP traffic from an outside network and re-encapsulates it into VXLAN (North/South)

• Speaks IGP/EGP routing protocols with the outside network (North/South)

• Requires VTEP

• IP transport forwarder between Leaf (East/West)

• Potentially hosting Rendezvous-Point (RP) for Underlay

• Potentially hosting Route-Reflector (RR) for EVPN

leaf leaf

Minimum Maximum Transmission Unit (MTU) Guidance:

• OTV: 1542 Bytes

• OTV w/UDP: 1550 Bytes (7.2 with F3 modules)

• LISP

• IPv4 1536 Bytes

• IPv6 1556 bytes

• FabricPath: 1516 Bytes

• VXLAN: 1550 Bytes

12

• Fabricpath/DFA Customer Deployment

• VXLAN Customer Deployment

• ACI Customer deployment

• Conclusion

Agenda

… so, Please …

14

Fabricpath/DFA Customer Deployment

DC Fabric w/FabricPath

16

• Externally the Fabric looks like a single switch

• Internally, ISIS adds Fabric-wide intelligence and ties the elements together.

• Provides in a plug-and-play fashion:• Optimal, low latency connectivity any to any

• High bandwidth, high resiliency

• Open management and troubleshooting

• ISIS for multipathing and reachability

FabricPath FabricPath

FabricPath: Design

FabricPath

- Default-Gateway

- Nx7k FP Spine (F3)

- Anycast-HSRP

- Nx5k FP leaf

UCS-FI

- F3 mac-scale (ARP)

Routing at FabricPath Spine

Anycast HSRP L3

SVISVISVISVI

Anycast HSRP

GWY IP X

GWY MAC A

GWY IP X

GWY MAC A

GWY IP X

GWY MAC A

GWY IP X

GWY MAC A

GWY MAC A→L1,L2,L3,L4

FabricPath

L3

L2/L3 boundary

All Anycast HSRP forwarders

share same VIP and VMAC

Hosts resolve shared

VIP to shared VMACRouted traffic spread

over spines based on

ECMP

Anycast HSRP

between agg switches

Layer 3 LinkLayer 2 CELayer 2 FabricPath

FabricPath: Services

FabricPath

- Default-Gateway


- Anycast-HSRP

- Nx5k FP leaf

FabricPath: Traffic flows

FabricPath

- Default-Gateway


- Nx5k FP leaf

Intra-VRF Inter-VRF

FP (or) vPC

FabricPath: External / WAN Connectivity

FabricPath

- Default-Gateway


- MPLS PE Layer

• Spine/leaf architecture

• FabricPath for L2 multi-pathing

• MPLS Integration to WAN

• No spanning-tree

• Default gateway at spine layer

• ASA for firewall layer

• Nexus 5600 DC Access

- ASR9000

- MPLS / LISP

- ASR9000

- MPLS / LISP

MPLS, WAN

, Internet, Campus

Note:

- F3 simplifies the deploy with MPLS and FabricPath Support.

- Previously we leveraged F2 for FabricPath (VDC)

- M2 for MPLS Connectivity (VDC)

Fabric

Management

Stand Alone Fabric (FabricPath/DFA)

Workload

Automation

Virtual FabricsOptimised

Networking

Bundled' functions'are'Modular,'Flexible'and'follows'your'Choice'of'Integration'and'Speed'of'Adoption!

• DC Fabric with a FabricPath based data plane and MP-iBGP control plane.

• Use MP-iBGP on the leaf nodes to distribute internal host/subnet routes and external reachability

information.

• Introduced Segment ID to increase name space to 16M identifier in the fabric.

Standalone Fabric (FabricPath/DFA)Host and Subnet Route Distribution

MAN/WAN

N1KV/OVS

External Subnet

Route Injection

MP-iBGP AdjacenciesRR RR

Fabric Host/Subnet

Route Injection

MP-iBGP Control Plane

FabricPath DataPlane

Route-Reflectors deployed for scaling purposes

Optimised Networking

• Distributed Gateway exists on all Leafs where VLAN/Segment-ID is active

• No HSRP

• There are different Forwarding Modes for the Distributed Gateway:

• Proxy-Gateway (Enhanced Forwarding)• Leverages local proxy-ARP • Intra and Inter-Subnet forwarding based on Routing• Contain floods and failure domains to the Leaf

• Anycast-Gateway (Traditional Forwarding)• Intra-Subnet forwarding based on Bridging• Data-plane based conversational learning for

endpoints MAC addresses• ARP is flooded across the fabric

Distributed Gateway Mode

24

vlan 123vn-segment 30000

!interface vlan 123

vrf member OrgA:PartAfabric forwarding mode proxy-gatewayip address 10.10.10.1/24no shutdownno ip redirects

vlan 145vn-segment 31000

!interface vlan 145

vrf member OrgA:PartAfabric forwarding mode anycast-gatewayip address 20.20.20.1/24no shutdown

IP Forwarding Between Fabrics Across L3 Based DCI

Inter-DC Core(Layer-3 IP/MPLS)

BGP AS#65500Control-Plane peering (eBGP)

with local Edge-Router; no multi-

hop peering

FabricPath

BGP AS#100Border-leaf Border-leaf

Edge router Edge router

FabricPath

BGP AS#200

Control-Plane peering (eBGP)

with local Edge-Router; no multi-

hop peering

eBGP eBGP

DFA Border-Leaf – Control-Plane ConnectivityRouted connection to Core-Network (e.g. WAN)

• External-BGP Session to Edge-Router

• similar to the MPLS CE-PE Routing concept (VRF-lite)

• One dedicated eBGP per DFA Virtual-Fabric (VRF) including the Backbone-Network (default VRF)

FabricPath

BGP AS#100

Inter-DC Core(Layer-3 IP/MPLS)

BGP AS#65500

Border-leaf

Edge router

eBGP

router bgp 100

fabric-soo 100:1

[snip]

neighbor 10.254.254.2 remote-as 65500

description BACKBONE (DEFAULT VRF)

peer-type fabric-external

address-family ipv4 unicast


vrf Ciscolive



description VF:Ciscolive



send-community extended

DCNM Infrastructure Provisioning Platform

DCNM cluster

NXAPI

[Southbound]

Nexus

Platform

NXAPI for Southbound APIs for

reduced reliance on SNMP, Netconf

REST

[Northbound]Updated northbound REST APIs

1000+

Nexus

N5000Nexus

N9000

Nexus

N7000

Modular device packs/driver for more

rapid Platform [HW/SW] updates

Scale >1000+ switches. Higher potential

with clustering

Enterprise HA Database support using

internal DB

POAP Support with templates for

VXLAN-EVPN

Topology Views for Phy, L2, L3,

VXLAN & VPC Overlays.

New GUI using HTML5 for completely new user

experience

No Java LAN Client – Simplifies Client Operation

Multi-site support - single pane management view and

template sync across multiple sites/clusters

Config and delta config. management


VXLAN Customer Deployment

The Underlay

• MTU and Overlays

• Unicast Routing Protocol and IP Addressing

• Multicast for BUM Traffic Replication

Deployment Considerations: Underlay

MTU and VXLAN: Underlay

• VXLAN adds 50 Bytes to the Original Ethernet Frame

• Avoid Fragmentation by adjusting the IP Networks MTU

• Data Centres often require Jumbo MTU; most Server NICs

do support up to 9000 Bytes

• Using a MTU of 9216* Bytes accommodates VXLAN

Overhead plus server max. MTU

*Cisco Nexus 5600/6000 switches only support 9192 Byte for Layer-3 Traffic

Building Your IP Network – Interface Principles

• Know your IP addressing and IP scale requirements

• Best to use single Aggregate for all Underlay Links and Loopbacks

• IPv4 only

• For each Point-2-Point (P2P) connection, minimum /31 required

• Loopback requires /32

• Routed Ports/Interfaces

• Layer 3 Interfaces between Spine and Leaf (no switchport)

• VTEP uses Loopback as Source-Interface

L2L1

L3

S1 S2 S3 S4

Building Your IP Network – Interface Configuration

Interface Configuration Example for (L1)

L2L1

L3

# Loopback Interface Configuration (VTEP)

interface loopback 0

ip address 10.10.10.L1/32

mtu 9192

# Point-2-Point (P2P) Interface Configuration

interface Ethernet 2/1

no switchport

ip address 192.168.1.1/31

mtu 9192


no switchport

ip address 192.168.1.3/31

mtu 9192

.

.

S1 S2 S3 S4

IP Unnumbered – Simplifying The Principles

• IP Unnumbered – Single IP address for multiple

Interfaces

• Remember way-back when.. On serial interfaces

• Used for Layer 3 Interfaces between Spine and Leaf

(no switchport)

• For each switch in the fabric, single IP address is

sufficient

• Loopback for VTEP

• IP Unnumbered from Loopback for routed

Interfaces

L2L1

L3

S1 S2 S3 S4

Note: IP Unnumbered cross-platform support, Nexus 9000 added in 7.0(3)I3(1)

IP Unnumbered – Interface Configuration

Interface Configuration Example for (L1)

L2L1

L3

# Loopback Interface Configuration (VTEP & IP

Unnumbered)



mtu 9192



no switchport

ip unnnumbered loopback 0

mtu 9192


no switchport

ip unnnumbered loopback 0

mtu 9192

.

.

Check Platform & Rlease Support for Ethernet IP Unnumbered

S1 S2 S3 S4

IP Unnumbered– Simplifying The Math

Check Platform & Release Support for Ethernet IP Unnumbered

L2L1

L3

Example from topology:4 Spine + 3 Leaf = 7 Individual Devices

= 7 IP Addresses for Loopback Interface(Used for VTEP & Routed Interfaces; IP Unnumbered)

7 IP Addresses required == /29 Prefix

A More Realistic Scenario:4 Spine + 40 Leaf = 44 Individual Devices

= 44 IP Addresses for Loopback Interface(Used for VTEP & Routed Interfaces; IP Unnumbered)

44 IP Addresses required == /26 Prefix

S1 S2 S3 S4

Building Your IP Network – Routing Protocols: OSPF

• OSPF – watch your network type

• Network Type Point-2-Point (P2P)

• Preferred (only LSA type-1)

• No DR/BDR election

• Suits well for routed interfaces/ports (optimal from a LSA Database perspective)

• Full SPF calculation on Link Change

• Network Type Broadcast

• Suboptimal from a LSA Database perspective (LSA type-1 & 2)

• DR/BDR election

• Additional election and Database Overhead

L2L1

L3

S1 S2 S3 S4

Building Your IP Network – Routing Protocols: OSPF

Configuration Example for (L1)

L2L1

L3




mtu 9192

ip router ospf 1 area 0.0.0.0

ip ospf network point-to-point



no switchport

ip address 192.168.1.1/31

mtu 9192

ip router ospf 1 area 0.0.0.0

ip ospf network point-to-point

S1 S2 S3 S4

Underlay Deployment with Multicast Routing

• PIM-ASM or PIM-BiDir (Different hardware has different capabilities)

• Spine and Aggregation Switches make good Rendezvous-Point (RP); much like RR

• PIM-ASM (sparse-mode)

• Source-trees, build a couple of unidirectional trees from RP; (s,g)

• Every VTEP is Source and Destination

• PIM-Anycast RP vs MSDP for example

• PIM-BiDir

• No Sources tree use a bi-directional shared tree

• No (S,G), we have (*,G)

• Phanton RP (Leverages Unicast for convergence)

• Each VNI does not need the same Multicast Group; can be different.

Multicast-enabled Underlay

Nexus 1000v Nexus 3000 Nexus 5600 Nexus 7000/F3 Nexus 9000ASR 1000

CSR 1000ASR 9000

Multicast

Mode

IGMP v2/v3 PIM ASM PIM BiDir PIM ASM / PIM BiDir PIM ASM PIM BiDir PIM ASM / PIM BiDir

Multicast-enabled Underlay – PIM ASM

L2L1

L3

RP RP

Rendezvous-PointRP

# Anycast-RP Configuration

ip pim rp-address 10.10.10.anycast

ip pim anycast-rp 10.10.10.anycast 10.10.10.S1

ip pim anycast-rp 10.10.10.anycast 10.10.10.S2

# Loopback Interface Configuration (RP)


ip address 10.10.10.S1/32

mtu 9192

ip pim sparse-mode

# Loopback Interface Configuration (Anycast RP)


ip address 10.10.10.anycast/32

mtu 9192

ip pim sparse-mode

Configuration Example for (Spine)

Configuration Example for (L1)

# Using Anycast Rendezvous-Point

ip pim rp-address 10.10.10.anycast




mtu 9192

ip pim sparse-mode

# Point-2-Point (P2P) Interface Configration


no switchport

ip address 192.168.1.1/31

mtu 9192

ip pim sparse-mode

For YourReference

Multicast Replication for VXLAN EVPNHandling of VXLAN Overlay BUM Traffic

Multicast replication in the underlay network

• Each VNI is mapped to a multicast group. BUM traffic in the VNI will be encapsulated into multicast packets using this multicast group as the outer destination IP address and then sent to the remote VTEPs using the underlay network multicast replication and forwarding.

• Broadcast/Unknown-unicast/Multicast (BUM) traffic in a VXLAN overlay network can be transported through the underlay network.

Flood-&-Learn mode VXLAN:Vlan 2

vn-segment 4098

Interface nve 1

member vni 10000

mcast-group 225.1.1.1

VXLAN EVPN:Vlan 200

vn-segment 20000Interface nve 1host-reachability protocol

bgpmember vni 20000

mcast-group 225.1.1.1

Introducing VXLAN /EVPN Overlay

Overlay with Optimised Routing

SpineRR RR

V

V

V

VV

V

EVPN Control Plane -- Host and Subnet Route Distribution

BGP Update

• Host-MAC• Host-IP• Internal IP Subnet• External Prefixes

RRRoute-Reflectors deployed for scaling purposes (iBGP)

BGP Adjacencies

Border

Scalable Multi-Tenancy with Multiprotocol BGP

EVPN Address-Family: Host MAC+IP, internal/external IP Subnets

BGP enhanced for Fast Convergence at Large Scale

Extensions for Fast and Seamless Host Mobility

Distributed Gateway with Traffic Flow Symmetry

ARP Suppression

Distributed IP Anycast Gateway

SpineRR RR

V

V

V

VV

V

SVI 200

SVI 100

SVI 100

SVI 100, Gateway IP: 192.168.1.1

SVI 200, Gateway IP: 10.10.10.1

Host1MAC: AA:AA:AA:AA:AA:AA

IP: 192.168.1.11

VLAN 100

VXLAN VNI 30001

Host3MAC: CC:CC:CC:CC:CC:CC

IP: 192.168.1.33

VLAN 100

VXLAN VNI 30001

Host2MAC: BB:BB:BB:BB:BB:BB

IP: 10.10.10.22

VLAN 200

VXLAN VNI 30002

bridge

route

Any Subnet Routed Anywhere – Any VTEP can serve any Subnet

Integrated Route & Bridge (IRB) - Route whenever you can, Bridge when needed

No Hairpinning – Optimised East/West and North/South Routing

Seamless Mobility - All Leaf share same Gateway MAC

Reduced Failure Domain – Layer-2/Layer-3 Boundary at Leaf

Optimal Scalability – Route Distribution & closest to the Host

Local LAN

Segment

Physical

Host

Local LAN

Segment

Physical

Host

Virtual Hosts

Local LAN

Segment

Virtual Switch

Edge Device

Edge Device

Edge Device

IP Interface

IP Fabric Overlay Taxonomy (1)

Local LAN

Segment

Physical

Host

Local LAN

Segment

Physical

Host

Virtual Hosts

Local LAN

Segment

Virtual Switch

VTEP

VTEP

VTEP

VTEP – VXLAN Tunnel End-Point

VNI/VNID – VXLAN Network Identifier

VV

V

Encapsulation

IP Fabric Overlay Taxonomy (2)

• Route Type 2 provides End-Host reachability information

• The following fields are part of the EVPN prefix in the NLRI

• Ethernet Tag ID (zeroed out)

• MAC Address Length (/48), MAC Address

• IP Address Length (/32, /128), IP Address [Optional]

• Additional Route Attributes

• Ethernet Segment Identifier (ESI) (zeroed out)

• MPLS Label1 (L2VNI)

• MPLS Label2 (L3VNI)

RD (1 octet)

ESI (10 octets)

Ethernet Tag ID (4 octets)

MAC Address Length (1 octet)

MAC Address (6 octets)

IP Address Length (1 octet)

IP Address (0, 4, or 16 octets)

MPLS Label1 (3 octets)

MPLS Label2 (0 or 3 octets)

MP-BGP EVPN Route Type 2MP-BGP EVPN Route Type 2 - MAC/IP Advertisement Route

• Route Type 5 provides IP Prefix advertisement in EVPN

• RT-5 decouples IP prefix from MAC (RT-2) and provides

flexible advertisement of IPv4 and IPv6 Prefixes with variable

length

• The following fields are part of the EVPN prefix in the NLRI

• IP Prefix Length (0-32 bits for IPv4 or 0-128 bits for IPv6)

• IP Prefix (IPv4 or IPv6)

• GW IP Address

• MPLS Label (L3VNI)

RD (8 octet)

ESI (10 octets)

Ethernet Tag ID (4 octets)

IP Prefix Length (1 octet)

IP Prefix (4 or 16 octets)

GW IP Address (4 or 16 octets)

MPLS Label (3 octets)

MP-BGP EVPN Route Type 5MP-BGP EVPN Route Type 5 - IP Prefix Route

V2# show bgp l2vpn evpn 192.168.1.73

BGP routing table information for VRF default, address family L2VPN EVPN

Route Distinguisher: 10.0.0.1:32868

BGP routing table entry for

[2]:[0]:[0]:[48]:[0050.56a3.c2bb]:[32]:[192.168.1.73]/272,

version 4

Paths: (1 available, best #1)

Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked

Advertised path-id 1

Path type: internal, path is valid, is best path, no labeled nexthop

AS-Path: NONE, path sourced internal to AS

10.0.0.1 (metric 3) from 10.0.0.111 (10.0.0.111)

Origin IGP, MED not set, localpref 100, weight 0

Received label 30001 50001

Extcommunity: RT:65501:30001 RT:65501:50001 ENCAP:8 Router MAC:5087.89d4.5495

Originator: 10.0.0.1 Cluster list: 10.0.0.111

Ethernet Segment

Identifier

Ethernet Tag

Identifier

MAC Address

LengthMAC Address IP Address Length IP Address

Route Type:

2 - MAC/IP

L3VNI

Route Target:

L2VNI (VLAN)

Route Target:

L3VNI (VRF)

Router MAC of

Remote VTEP

Overlay Encapsulation:

8 - VXLAN

Remote VTEP

IP Address

L2VNI

Virtual Switch

RR RR

Host A

MAC_A / IP_A

Host C

MAC_C / IP_C

Host Y

MAC_Y / IP_Y

Host B

MAC_B / IP_B

V1

V3

V2

VTEPs advertise End-Host reachabilityinformation (MAC,IP) within MP-BGP1

1

11

MAC, IP L2VNI L3VNI NH

MAC_C, IP_C 30000 50000 local

MAC_Y, IP_Y 30001 50000 local


MAC_B, IP_B 30000 50000 local


MAC_A, IP_A 30000 50000 local

Protocol Learning & Distribution

Virtual Switch

RR RR

Host A

MAC_A / IP_A

Host C

MAC_C / IP_C

Host Y

MAC_Y / IP_Y

Host B

MAC_B / IP_B

V1

V3

V2

BGP Route-Reflector “reflects” Overlay relatedreachability information to other VTEPs 2

2

2








2


Virtual Switch

RR RR

Host A

MAC_A / IP_A

Host C

MAC_C / IP_C

Host Y

MAC_Y / IP_Y

Host B

MAC_B / IP_B

V1

V3

V2

VTEPs receive respective reachability informationand installs them related to route-policy into RIB/FIB




MAC_A, IP_A 30000 50000 IP_V1

MAC_B, IP_B 30000 50000 IP_V2



MAC_A, IP_A 30000 50000 IP_V1

MAC_C, IP_C 30000 50000 IP_V3

MAC_Y, IP_Y 30001 50000 IP_V3



MAC_B, IP_B 30000 50000 IP_V2

MAC_C, IP_C 30000 50000 IP_V3

MAC_Y, IP_Y 30001 50000 IP_V3

3 3

3

3


Multitenancy

• A mode of operation, where multiple independent instances (tenant) operate in a shared environment.

• Each instance (i.e. VRF/VLAN) is logically isolated, but physically integrated.

What is Multi-Tenancy

Multi-Tenancy at Layer-2

• Per-Switch VLAN-to-VNI mapping

• Per-Port VLAN Significance

Multi-Tenancy at Layer-3

• VRF-to-VNI mapping

• MP-BGP for scaling with VPNs

Where can we apply Multi-Tenancy

Layer-2 Multi-Tenancy

SpineRR RR

V

V

V

VV

VVLAN 100

VLAN 100


IP: 192.168.1.11

VLAN 100

VXLAN VNI 30001


IP: 192.168.1.33

VLAN 100

VXLAN VNI 30001

bridge

Layer-2 Multi-Tenancy – Bridge Domains


IP: 192.168.1.11

VLAN 100

VXLAN VNI 30001


IP: 192.168.1.33

VLAN 100

VXLAN VNI 30001

Leaf

VV

VLAN 100 VLAN 100

VXLAN Overlay

(VNI 30001)

Bridge Domain

Layer-2 Multi-Tenancy – Bridge Domains


IP: 192.168.1.11

VLAN 100

VXLAN VNI 30001


IP: 192.168.1.33

VLAN 100

VXLAN VNI 30001

Leaf

VV

VLAN 100 VLAN 100

VXLAN Overlay (VNI 30001)

Bridge Domain

The Bridge Domain is the Layer-2 Segment from Host to Host

In VXLAN, the Bridge Domain consists of three Components

1) The Ethernet Segment (VLAN), between Host and Switch

2) The Hardware Resources (Bridge Domain) within the Switch

3) The VXLAN Segment (VNI) between Switch and Switch

VLAN-to-VNI mapping


IP: 192.168.1.11

VLAN 100

VXLAN VNI 30001


IP: 192.168.1.33

VLAN 100

VXLAN VNI 30001

Leaf

VV

VLAN 100 VLAN 100

VXLAN Overlay

(VNI 30001)


IP: 192.168.1.22

VLAN 100

VXLAN VNI 30001

Leaf#1

vlan 100

vn-segment 30001

Leaf#2

vlan 100

vn-segment 30001

• VLAN to VNI configuration on a per-switch basis

• VLAN becomes “Switch Local Identifier”

• VNI becomes “Network Global Identifier”

CLI Modes - VLAN based (per-Switch)For YourReference

Per-Switch VLAN-to-VNI mapping


IP: 192.168.1.11

VLAN 100

VXLAN VNI 30001


IP: 192.168.1.33

VLAN 200

VXLAN VNI 30001

Leaf

VV

VLAN 100 VLAN 200

VXLAN Overlay

(VNI 30001)


IP: 192.168.1.22

VLAN 100

VXLAN VNI 30001

Leaf#1

vlan 100

vn-segment 30001

Leaf#2

vlan 200

vn-segment 30001

• VLAN to VNI configuration on a per-switch basis

• VLAN becomes “Switch Local Identifier”

• VNI becomes “Network Global Identifier”

• 4k VLAN limitation has been removed

CLI Modes - VLAN based (per-Switch)For YourReference

Per-Port VLAN-to-VNI mapping


IP: 192.168.1.11

VLAN 100

VXLAN VNI 30001


IP: 192.168.1.33

VLAN 300

VXLAN VNI 30001

Leaf

VV

VLAN 100 VLAN 300

VXLAN Overlay

(VNI 30001)


IP: 192.168.1.22

VLAN 200

VXLAN VNI 30001

VLAN 200

Leaf#1

vlan 2500

vn-segment 30001


switchport mode trunk

switchport vlan mapping enable

switchport vlan mapping 100 2500


switchport mode trunk

switchport vlan mapping enable

switchport vlan mapping 200 2500

CLI Modes - VLAN based (per-Port) For YourReference

Layer-3 Multi-Tenancy

SpineRR RR

V

V

V

VV

V

SVI 200

SVI 100

VRF-A (VNI 50001)

VRF-B (VNI 50002)

SVI 100, Gateway IP: 192.168.1.1 (VRF-A)

SVI 200, Gateway IP: 10.10.10.1 (VRF-B)

SVI 300, Gateway IP: 172.16.1.1 (VRF-B)

Host1IP: 192.168.1.11 (VRF-A)

VLAN 100

Host3IP: 172.16.1.33 (VRF-B)

VLAN 300

Host2IP: 10.10.10.22 (VRF-B)

VLAN 200

SVI 300

route

route

Layer-3 Multi-Tenancy – VRF-VNI or L3VNI

Host1

IP: 192.168.1.11 (VRF-A)

VLAN 100

Host3

IP: 172.16.1.33 (VRF-B)

VLAN 300

Leaf

VV

SVI 100

V

Host2

IP: 10.10.10.22 (VRF-B)

VLAN 200

SVI 200 SVI 300

VRF-A

(VNI 50001)

VRF-B

(VNI 50002)

Routing

Domain

VRF-B

Routing

Domain

VRF-A

Layer-3 Multi-Tenancy – VRF-VNI or L3VNI

Host1

IP: 192.168.1.11 (VRF-A)

VLAN 100

Host3

IP: 172.16.1.33 (VRF-B)

VLAN 300

Leaf

VV

VLAN 100

V

Host2

IP: 10.10.10.22 (VRF-B)

VLAN 200

SVI 200 SVI 300

VRF-A

(VNI 50001)

VRF-B

(VNI 50002)

Routing

Domain

VRF-B

Routing

Domain

VRF-A

The Routing Domain is the VRF owning multiple Subnets across multiple Switches

In VXLAN EVPN, the Routing Domain consists of three Components

1) The Routing Domains (VRF), local to the Switch

2) The Routing Domain (L3VNI) between the Switches

3) Multi-Protocol BGP with EVPN Address-Family

Layer-3 Multi-Tenancy – VXLAN EVPN

Leaf

VV

SVI 300SVI 200SVI 100 SVI 400

L3VNI 50002

L3VNI 50001VXLAN

Host1

MAC: AA:AA:AA:AA:AA:AA

IP: 192.168.1.11 (VRF-A)

VLAN 100

VXLAN VNI 30001

Host2

MAC: BB:BB:BB:BB:BB:BB

IP: 10.10.10.22 (VRF-B)

VLAN 200

VXLAN VNI 30002

Host3

MAC: CC:CC:CC:CC:CC:CC

IP: 172.16.1.33 (VRF-B)

VLAN 300

VXLAN VNI 30003

Host4

MAC: DD:DD:DD:DD:DD:DD

IP: 10.44.44.44 (VRF-A)

VLAN 400

VXLAN VNI 30004

vrf context VRF-A

vni 50001

rd auto


route-target both auto

route-target both auto evpn

vrf context VRF-B

vni 50002

rd auto




vrf context VRF-B

vni 50002

rd auto




vrf context VRF-A

vni 50001

rd auto




router bgp 65500



address-family l2vpn evpn


vrf VRF-A


advertise l2vpn evpn

vrf VRF-B



router bgp 65500



address-family l2vpn evpn


vrf VRF-A



vrf VRF-B



For YourReference

Integrated Route & Bridge + Multi-Tenancy

SpineRR RR

V

V

V

VV

V

SVI 200

SVI 100

SVI 100


IP: 192.168.1.11 (VRF-A)

VLAN 100

VXLAN VNI 30001


IP: 192.168.1.33 (VRF-A)

VLAN 100

VXLAN VNI 30001


IP: 10.10.10.22 (VRF-A)

VLAN 200

VXLAN VNI 30002

VRF-A (VNI 50001)



Layer 4-7 Services Integration

1) The load balancer is deployed in one-arm mode with source-NAT (SNAT).

2) Load balancer and Firewall Service chain

3) Firewall is the first device in the service chain to protect the load balancer

4) Servers are leveraging anycast gateway in both examples.

Service Chain: Firewall + Load Balancer

Client Router

Load-

balancer

VIP

w/SNAT

Servers

VLAN 40

VRF-A

Service Chain Load Balancer (SNAT)+ Firewall Services:

Logical FlowDefault-gateway =ToR

(anycast gateway)

VLAN 100

VRF-C

VLAN 40

(server)

VRF-A

VLAN 41

(transit)

VRF-A

VLAN 101

(transit)

VRF-C

VIP-> VLAN21 (VRF-A)

= Fabric= Distributed Anycast Gateway

VIP-> VLAN40 (VRF-A)

Firewall

VLAN 20

VRF-A

VLAN 21

(transit)

VRF-A

VIP-> VLAN101 (VRF-A)Client-> VIP VLAN40-> VLAN41 (VRF-A)

default route 0.0.0.0/0default route 0.0.0.0/0

Service Chain Load Balancer(SNAT) + Firewall Services• Firewall is the first device in the service chain

• Load balancer is the second device in the service chain

• Source NAT implemented on the load balancer

• Fabric providing anycast gateway

• Traffic is symmetric in both directions for the LB + FW

• Additional VIP(s) can be implemented in this model

VLAN 21: VRF-A

…………………..

VLAN 100: VRF-C10.10.10.100

VLAN 40

VRF-A

10.10.10.101

VLAN 40

VRF-A

Client

Anycast-gateway

192.168.40.1/24

VIP1

(192.168.40.110/32)

VLAN 41

Anycast-gateway

10.10.10.1/24

Anycast-gateway

10.10.10.1/24

External Connectivity

VXLAN and Interaction with Spanning-Tree

• Spanning-Tree and VXLAN

• VXLAN has no integration with

Spanning-Tree for Loop protection

• VXLAN does not forward BPDU

• Loop-free topologies required

southbound of VXLAN Edge-Devices

• Use VPC to provide Ethernet-based

Loop-free topologies

L3

L1

L2fwd

fwd

fwd

VXLAN and Interaction with Spanning-Tree

• Spanning-Tree and VXLAN

• Virtual Port-Channel (vPC) will allow safe

integration with Spanning-Tree

• No Loop-Protection required as per

logical Loop-free topology

• Note

• Follow best practices to protect the

Network Border as in Classic Ethernet

Networks

• BPDU Guard

• Root Guard

• Storm Control

• etc

L3

fwd

fwd

L1

L2

• The VXLAN vPC Domain follows the

configuration similar as for Classic Ethernet

• There are some VXLAN specifics for vPC

peer-link configuration

• With vPC, an additional common secondary

IP address is attached to the VTEP –

Anycast IP for VTEP

Host A

192.168.1.101

V1

V2

Virtual Port-Channel (VPC) Concept

10.10.10.1/32

10.10.10.254/32 secondary

10.10.10.2/32

10.10.10.254/32 secondary

RR RR

L3

L1

L2

Border-leaf with VRF-lite

• Layer 3 @ Border with VRF-lite

• aka Inter-AS Option “A “

• Provides connectivity for external

routing connectivity

• Interconnect using sub-interfaces for

Multitenant capable handoff

• Per-VRF routing adjacency based on

IEEE 802.1Q tagging

• Various routing protocols available

(eBGP, OSPF, EIGRP etc)

BL1

BL2

Layer-3

Sub-Interface

BGP AS# 65500

Interface-Type Options:

• Physical Routed Ports

• Sub-Interfaces

• VLAN SVIs over Trunk Ports

RR RR

L3

L1

L2

Border-leaf with VRF-lite (Inter-AS Option “A “)

BLA B C

Peering Interface can

be in Global or Tenant VRF

VRF for External Routing

needs to exist on Border Leaf

VTEP(s) Configured on Border-leaf

BGP AS# 65500

BGP AS# 65599

RR RR

L3

L1

L2

Border-leaf with eBGP VRF-lite Configuration

BL

A B C

# Sub-Interface Configuration

interface Ethernet1/1

no switchport

interface Ethernet1/1.10

mtu 9216

encapsulation dot1q 10

vrf member VRF-A

ip address 10.254.254.1/30

# eBGP Configuration

router bgp 65500

…

vrf VRF-A



aggregate-address 10.0.0.0/8 summary-only


update-source Ethernet1/1.10



send-community both

# Interface Configuration

interface Ethernet1/1.10

mtu 9216


vrf member VRF-A

ip address 10.254.254.2/30

# eBGP Configuration

router bgp 65599

…

vrf VRF-A



update-source Ethernet1/1.10



DCNM cluster

NXAPI

[Southbound]

Nexus

Platform

NXAPI for Southbound APIs for

reduced reliance on SNMP, Netconf

REST

[Northbound]Updated northbound REST APIs

1000+

Nexus

N5000Nexus

N9000

Nexus

N7000

Modular device packs/driver for more

rapid Platform [HW/SW] updates

Scale >1000+ switches. Higher potential

with clustering

Enterprise HA Database support using

internal DB

POAP Support with templates for

VXLAN-EVPN

Topology Views for Phy, L2, L3,

VXLAN & VPC Overlays.

Intelligent fabric lifecycle management

• Fabric-wide focus – auto-configuration and management of fabric

• Initial support for Cisco Nexus 9000 Familyrunning stand-alone NX-OS mode

• Automation based on knowledge of underlying fabric architecture

• Designed to simplify fabric management through its various lifecycle phases

• Delivered via VXLAN-based architecture

Cisco Nexus Fabric Manager (NFM)

Fabric Management Lifecycle

Creation Expansion

Fault MgmtReporting

Connection

NFM

Cisco VTS: Virtual Topology System Overlay Controller

VTS

vCenter

REST API

GUI

Nexus Portfolio

Nexus 2k – 9k

Programmable Fabric (VXLAN)

Scalable Multi-Tenancy

• MP-BGP EVPN control plane

• Physical and Virtual overlay support

• High performance virtual forwarding

Automated Provisioning

• Group Based Policy model

• Overlay Provisioning

• Service Chaining

Open, Standards Based

• Rest based Northbound APIs

• Multi-protocol support (EVPN, VXLAN)

• Multi-Hypervisor

Overlay Management

• Automatic Topology Discovery

• Resources Management

• Overlay monitoring and troubleshooting

ACI Customer Deployment

App-Based Automation

Automated L4-7 Stitching

Turnkey network automation

Application Centric Infrastructure (ACI)

APIC

ACI Fabric OverviewSpine and Leaf Architecture / Design

Spine

Leaf

ACI Fabric OverviewAttaching the ACI APIC(s)

APIC APIC

Out-of-band Management (OOB)

Defining Terms

• Tenant: Logical separator for: Customer, BU, group etc. separates traffic,

admin, visibility, etc.

• Context: Equivalent to a VRF, separates routing instances, can be used as an

admin separation

• Bridge Domain: Not a VLAN, simply a container for subnets. It can be used

to define a L2 boundary.

• End-Point Group (EPG) Container for objects requiring the same policy

treatment, i.e. app tiers, or services

Bridge

Domain

Logical Model Overview

root

Tenant A Tenant B

Context A Context B Context A

Bridge

Domain

Subnet A

Bridge

Domain

Subnet B

Subnet C

Bridge

Domain

Subnet A

Context and subnets are independent between tenants

EPG A EPG BEPG C EPG D EPG E

Design / Deployment Requirements• Greenfield Deployment

• Fabric Hardware:

• (3) APIC Controllers

• (3) Nexus 9508 Spines

• (many) Nexus 9300 Leaf switches (mix of 9396/9372/9332)

• Enterprise compute block:

• (3) vCentres / (4) vDS

• Services blocks: FW, LB, Infoblox, mainframe

• 9332 connecting to ASR9K belong to these blocks

• Compute UCS-B blades and UCS-FI

• The design is taking a network-centric approach:

• VLAN is mapped to EPG/BD

• Contract is permit-any for all the EPGs

• Each risk domain is mapped to context (VRF) in ACI:

• Communication within the same risk domain between different sites go through the WAN router within the

corresponding VRF.

• Inter-context communication with Firewall policy

Design / Deployment Requirements

• Default gateway is on ACI for BDs with one exception; the load-balanced deployed in 2-ARM mode.

• Layer-3 Routing:

• OSPF to ASR9K WAN router (vPC)

• OSPF to Infoblox/Mainframe (treat like OSPF Stub Areas)

• Static routes to FW/LB (except extranet FW, which use OSPF)

• Fabric provide network connection (L2/L3) for FW/LB

• No L4-7 device-package level integration

• L3 multicast design:

• ASR1K as external mrouter interfaces

• Exchange multicast source information with ASR9K via MP-BGP.

EP EP

EPGEPG

EP EP

Bridge

Domain

EP EP

EPGEPG

EP EPEP EP

EPGEPG

EP EP

Bridge

Domain

Tenant “X”

Context:

Risk Domain “A”

(VRF)

Context:

Risk Domain “C”

(VRF)

Tenant “Y”

Bridge Domain

L3-Out

(ASR9000)L3-Out:

(ASR9000)

(Mainframe)

(FW)

(Infoflox)

(Citrix-LB)

ACI Policy ModelHigh Level Overview

Static-path bindings

(ASR1000)

Bridge

DomainBridge

Domain

Context:

Risk Domain “B”

(VRF)

ACI FabricAttaching the Compute Resource to the Fabric

Spine

Leaf

(OOB)(OOB)

(OOB)

(OOB)

ACI FabricAttaching the Services to the Fabric

Spine

Leaf

InfobloxCitrix Load-balancer(s)

ExtranetLocal-Internet

LAN1HA

Checkpoint Firewall(s)

ACI FabricAttaching the VMM/Orchestration to the Fabric

Spine

Leaf

vCentre 5.5

vCentre 6UCS director

Out-of-band Management (OOB)

ACI FabricAttaching the External WAN/Enterprise to the Fabric

Spine

Leaf

Intranet/Internet

ASR9000ASR9000

ACI FabricAttaching the External IP Multicast Routers to the Fabric

Spine

Leaf

ASR1000

(mrouter)

ASR1000

(mrouter) Intranet/Internet

ASR9000

ASR9000

VLAN = EPG

EPG-A EPG-n

- Connect non-ACI networks to ACI leaf nodes

- Connect at L2 with VLAN trunks (802.1Q)

- Objective: Map VLANs to EPGs, extend policy model to non-ACI networks

EPG-B

End-

point(s)

End-

point(s)

End-

point(s)

ACI Policy Model: EPG To EPG Communication

EPG-A EPG-n

Zero Trust Security Model

- Need to define a Contract (Policy); - A contract is used to specify the interaction between two EPG(s), a provider/consumer pair.

- The goal is to provide a global policy view that focuses on improving automation and scalability.

Provides

policies

Consumes

policies

Allow HTTP

Allow ICMP

ACI and IP Multicast

ASR1000 IP Multicast EPG Deployment: Static-path Binding (EPG)

Bridge-Domain: “B”

EPG1

VLAN 311…+

Bridge-Domain: “A”

EPG3

VLAN 411…+

ASR1000 PIM Interface (mrouter)

• No L3 routing between ASR1000 and ACI fabric

• PIM routers attached to L2 Network

• IGMPv2 and IGMPv3 in the Fabric

• VLAN Encap provides L2/L3 (VRF) separation

EPG2

VLAN 511…+EPG4

VLAN 611…+

ACI Multicast Configuration

1) Create Layer-2 Bridge-domain

2) Create EPGs for BDs where multicast traffic are flowing

3) Deploy static path binding for the EPGs created for external PIM interfaces

4) 1:1 Static-path binding for each BD (which requires Multicast traffic)

5) ASR1000 Attach to the fabric like any other server for example (EPG

Configuration)

Note: LLDP and CDP must be turned off on ASR1000, since

ASR1000 shares the same MAC for all sub-interfaces, even with

different dot1q encapsulations.

For YourReference

1) Bridge-domain Configuration

1) Create Bridge-domain

2) Associate with proper Context/VRF

3) Enable Flooding

2) EPG Configuration

Bridge-domain

1) Create EPG

2) Associate with the BDs where multicast traffic is

required

3) Static-path Bindings Configuration

1) Configure static path bindings for the EPGs

2) These are the ASR1000 PIM interfaces

connected to the fabric.

3) 1:1 Static-path binding for each BD (which

require Multicast)

4) ASR1000 Attach to the fabric like any other

server for example (EPG Configuration)

VLAN Encap of 311

4) Verifying the two ASR1000(s) connected to EPG

ASR1K-2

ASR1K-1

VLAN Encap of 311

ASR 1000 IP Multicast Configuration (VLAN-311 + others)

interface Port-channel1.311

encapsulation dot1Q 311

vrf forwarding ”A”

ip address 172.18.54.253 255.255.255.0

ip pim dr-priority 10

ip pim sparse-mode

ip igmp version 3

Note: LLDP and CDP must be turned off on ASR1000, since ASR1000 shares the same MAC for all sub-interfaces, even with different

dot1q encapsulations.



vrf forwarding “B”

ip address 172.18.133.254 255.255.255.0

ip pim query-interval 15

ip pim sparse-mode

ip igmp version 3



vrf forwarding “C”

ip address 172.18.131.254 255.255.255.0

ip pim query-interval 15

ip pim sparse-mode

ip igmp version 3

Showing the two ASR1000(s) sub-interfaces (VLAN-311)

ASR1K-1#show int port-channel 1.311

Port-channel1.311 is up, line protocol is up

Hardware is 10GEChannel, address is 0023.5e49.20c0 (bia

0023.5e49.20c0)

Description: BD ENT_INTRA_LOGISTICS1 L2ext

Internet address is 172.18.54.254/24

MTU 1500 bytes, BW 20000000 Kbit/sec, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 1/255

Encapsulation 802.1Q Virtual LAN, Vlan ID 311

ARP type: ARPA, ARP Timeout 04:00:00

Keepalive set (10 sec)

Last clearing of "show interface" counters never

ASR1K-1#



Hardware is 10GEChannel, address is 0021.a00c.86c0 (bia

0021.a00c.86c0)





Encapsulation 802.1Q Virtual LAN, Vlan ID 311




ASR1K-2#

Verifying the two ASR1000(s) connected to EPG (VLAN-312)

Different bridge-

domain

ASR1K-2

ASR1K-1

VLAN Encap of 312

ASR 1000 IP Multicast Configuration (VLAN-312)

ASR1K-1#show runn int port-channel 1.312


description BD ENT_INTRA_LOGISTICS2 L2ext


vrf forwarding Intra

ip address 172.18.53.254 255.255.255.0

ip pim sparse-mode

ip igmp version 3

ASR1K-1#

ASR1K-2#show runn interface Port-channel1.312


description BD ENT_INTRA_LOGISTICS2 L2ext


vrf forwarding Intra

ip address 172.18.53.253 255.255.255.0

ip pim dr-priority 10

ip pim sparse-mode

ip igmp version 3

ASR1K-2#

Showing the two ASR1000(s) sub-interfaces (VLAN-312)



Hardware is 10GEChannel, address is 0023.5e49.20c0 (bia

0023.5e49.20c0)





Encapsulation 802.1Q Virtual LAN, Vlan ID 312.




ASR1K-1#



Hardware is 10GEChannel, address is 0021.a00c.86c0 (bia

0021.a00c.86c0)





Encapsulation 802.1Q Virtual LAN, Vlan ID 312.




ASR1K-2#

Infoblox DNS/DHCPIntegration

Infoblox Anycast (DNS/DHCP) L3-Out ACI Deployment

Context (VRF): “A”


L3Out-InfoBlox

Anycast GW Anycast GW Anycast GW

EPG Green EPG Orange EPG Black

APP Green APP OrangeAPP Black

OSPF AREA

0.0.0.2172.16.0.0/28

- Access Interface (Untagged)

- Leaf advertises default-route to the Infoblox. "External Network Instance Profile advertise 0.0.0.0/0 to Infoblox – like OSPF Stub no-summary.

- Infoblox OSPF Priority = 0

- OSPF Network Type: Broadcast

- HA Active / Standby Anycast Management VIP

- Physical: Infoblox1 LAN1/HA connects to Leaf1. Infoblox2 LAN1/HA connects to Leaf2. (2 OSPF peers)

- LAN and HA interfaces all have to be in the same EPG/BD/Subnet.

- Passive nodes listen to VRRP advertisements on the HA port while Active nodes listen on the LAN port.

- Peering is on leaf interface, the SVI for the default gateway

- Default route leak policy being used as an alternative to a pre-existing default-route. The VRF-Intra, it is being injected via the ASR9000 (OSPF) or configure a static-route via the FW (security policy on L3-Out)

Anycast DNS address 172.16.0.25/32

LAN1

HA (VRRP)

- Anycast DNS Address (OSPF)

- Grid Management Address (OSPF)

(Floats btw act/std)

Grid Management 172.16.0.8/32

Infoblox Grid Geographical Redundancy

Grid Management 172.16.0.8/32 OSPF AREA

0.0.0.2172.16.0.x/28

Floating IP .1 (SVI), does not have OSPF enabled. This is the default gateway for the Infoblox Grid management.

Anycast DNS address 172.16.0.25/32

.3

.4

ACI Fabric

LAN1

HA

LAN1

HA

.9

.11

.10

.12

Infoblox-1

Infoblox-2

leaf-1

(router-id)

leaf-2

(router-id)

Infoblox

Grid

Manager

.1

Different Network

L3-Outside Configuration: OSPF

VRF

OSPF Area

OSPF Area Type

1) Configure L3Out for OSPF

2) Select Context / VRF

3) Define OSPF Area, in this case OSPF Area 0.0.0.2

4) Define OSPF Area type, in this case regular OSPF Area

5) The external routed domain, policy for managing the

physical infrastructure, such as ports/VLANS, that can be used

by an L3 routed outside network.

For YourReference

Logical Node Profile: Leaf OSPF Router-id (Node) For YourReference

Logical Interface Profile

LAN1

HA

For YourReference

ACI Configuration: Logical Interface

Node-204; interface eth1/15 Node-204; interface eth1/16

For YourReference

Infoblox OSPF Area: Default-route• Today, Infoblox is deployed as TSSA OSPF Area

• The TSSA Areas do not have type 4 or 5 LSAs.

• Infoblox/Mainframe are configured as a full OSPF area, the ACI Leaf(s) are OSPF

ASBR; due to iBGP redistribution with Spines as Route Reflectors. Since the Area

is a full OSPF Area, the Infoblox/mainframe devices will see a default-route

advertised from the fabric as a Type-5 LSA.

• Verify OSPF database LSA; the routes appear as E2:0.0.0.0/0 appears as Type 5 LSA

AS External Link States

Link ID ADV Router Age Seq# CkSum Route

0.0.0.0 203.0.0.1 16 0x80000002 0xba49 E2 0.0.0.0/0 [0xffffffff]

0.0.0.0 204.1.1.1 16 0x80000002 0xa25e E2 0.0.0.0/0 [0xffffffff

Mainframe OSPF Integration

Mainframe: L3-Out ACI Deployment

Context (VRF): “Intra”


L3Out-

Mainframe

(SVI)

Anycast GW

Mainframe Green

Mainframe

OSPF AREA

0.0.0.1172.16.0.0/28

- Mainframe L3-out is a regular OSPF Area.

- Defined external network instance for Export Route

Control Subnet for 0.0.0.0/0 (make sure un-check

"Aggregate Export“).

- Trying to “treat” as OSPF Stub Area.

- Type 5 LSA(s); leaf(s) are OSPF ASBR(s)

ENCAP VLAN 751

Context (VRF): “risk-domain”


L3Out-

Mainframe

(SVI)

Anycast GW

Mainframe Blue

Mainframe

ENCAP VLAN 753

ACI Configuration: External Networks

ACI Configuration: default-route

- The default-route already exists in each VRF.

- Export control subnet, in this case, IP Address is

0.0.0.0; the subnets configured for IP Address 0.0.0.0;

that is what I want you to advertise.

- Aggregate Export, do not enable. We do not want all of

the fabric routes advertised.

Verify Mainframe Routing information VRF: “Intranet” has a default-route advertised by WAN router ASR9K via ospfmainframe# sh ip route ospf-1 vrf Intranet

IP Route Table for VRF "Intranet"

'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

0.0.0.0/0, ubest/mbest: 2/0

*via 172.18.3.67, Vlan751, [110/1], 1d13h, ospf-1, type-2, tag 4294967295

*via 172.18.3.68, Vlan751, [110/1], 1d13h, ospf-1, type-2, tag 4294967295

VRF: “Risk-domain” has a static default-route pointing to FW cluster via OSPF.

mainframe# sh ip route ospf-1 vrf risk-domain

IP Route Table for VRF "risk-domain"




0.0.0.0/0, ubest/mbest: 2/0

*via 172.18.15.66, Vlan753, [110/1], 00:00:13, ospf-1, type-2, tag 4294967295

*via 172.18.15.67, Vlan753, [110/1], 00:00:20, ospf-1, type-2, tag 4294967295

Citrix Load-balancersIntegration

Citrix 2-arm Load-balancer: Static-Bindings

Internal-arm (VLAN) is the Server default-gateway on the load-balancer

External-arm (VLAN) for the VIP / Client

L2 Bridge-Domain

(Server subnet)

L3Out Static route to Servers

Static route for LB servers pointing to VIPVLAN 10 SVI on L3out

VIP: 20.20.20.20/32

VLAN 400 (Bridge-domain same for Servers)

192.168.50.100

1) External-arm: VIP / Client

2) Internal-arm: server default-gateway is on the load-

balancer.

Server(s) default-gateway

ACI: Configuring the Server-side bridge-domainEnabled Flooding (ARP) as this L2 Only

ACI: Configuring the Server-side bridge-domainNo Unicast routing enabled, as we want the external LB to be the gateway; not BD.

External Connectivity

ACI Interaction with STP

• No STP running within ACI fabric

• BPDU frames are flooded within EPG. No Configuration required

• External switches break any potential loop upon receiving the flooded BPDU frame fabric

• BPDU filter and BPDU guard can be enabled with interface policy

STP Root Switch

Same EPG

APIC

ASR9000 External L3out OSPF via SVI and vPC

VRF: risk-domain

VLAN 902

172.18.159.64/29

OSPF Area 0

ASR9000:A ASR9000:B

VRF: risk-domain

VLAN 903

172.18.159.72/29

OSPF Area 0

VRF: Intra

VLAN 900

172.18.0.64/29

OSPF Area 0

VRF: Intra

VLAN 901

172.18.0.72/29

OSPF Area 0

VRF: risk-domain

VLAN 904

172.18.181.0/29

OSPF Area 0

VRF: risk-domain

VLAN 905

172.18.181.8/29

OSPF Area 0

Intranet/Internet

ACI Fabric

(SVI)

L3-out to ASR9000 VRF:Intra

VRF

external routed domain

OSPF Area

OSPF Area Type

1) Configure L3Out for OSPF

2) Select Context / VRF

3) Define OSPF Area, in this case OSPF Area

0.0.0.0

4) Define OSPF Area type, in this case regular

OSPF Area

5) The external routed domain, policy for

managing the physical infrastructure, such

as ports/VLANS, that can be used by an L3

routed outside network.

ACI Configuration: Logical Interface Profile vPC to ASR9000 VRF:Intra

1) Leaf231 and leaf232 are a

logical vPC pair

2) Configure SVI(s) on “leaf231”

and “leaf232”

3) Configuration for other 9332

border-leaf

4) Define SVI(s) for OSPF Area 0

to ASR9000

ACI Configuration: SVI interface vPC to ASR9000 VRF:Intra

Leaf 231 and 232 to ASR9k-1 Leaf 231 and 232 to ASR9k-2

ASR9000 OSPF Configuration: VRF-Intra

vrf Intra


address-family ipv4 multicast

interface Bundle-Ether1

!

interface Bundle-Ether1.900

vrf Intra

ipv4 address 172.18.0.69 255.255.255.248


interface Loopback0

vrf Intra

ipv4 address 9.9.9.1 255.255.255.255

router ospf 1

nsr

log adjacency changes detail

router-id 9.1.1.1

area 0

vrf Intra

router-id 33.33.33.1

default-information originate always

redistribute bgp 3000 metric 100 metric-type 1


area 0

dead-interval 20

retransmit-interval 3

hello-interval 5

transmit-delay 1

interface Bundle-Ether1.900

For YourReference

Verify OSPF Output: ACI border-leaf (VRF-Intra)

leaf231# show ip ospf neighbors vrf Active:DA_Intra

OSPF Process ID default VRF Active:DA_Intra

Total number of neighbors: 4

Neighbor ID Pri State Up Time Address Interface

32.1.9.1 1 FULL/DR 1w0d 172.18.0.68 Vlan7

33.33.33.1 1 FULL/DROTHER 1w0d 172.18.0.69 Vlan7

32.1.9.1 1 FULL/BDR 1w0d 172.18.0.76 Vlan8

33.33.33.2 1 FULL/DR 1w0d 172.18.0.77 Vlan8

leaf231# show ip route 0.0.0.0 vrf Active:DA_Intra

IP Route Table for VRF "Active:DA_Intra"




'%<string>' in via output denotes VRF <string>

0.0.0.0/0, ubest/mbest: 2/0

*via 172.18.0.77, vlan8, [110/1], 01w07d, ospf-default, type-2, tag 2

*via 172.18.0.69, vlan7, [110/1], 01w07d, ospf-default, type-2, tag 2

leaf231#

Verify OSPF Neighbors

Verify default-route from ASR9000 to 9332 vrf-intra

For YourReference

Checkpoint / ASA Firewall Integration

Extranet: Routing Between Contexts



Context (VRF): “B”


L3Out-A L3Out-B

L3Out OSPF Area 0.0.0.0 on each L3OutExtranet

Local-Internet: Logical view



L3Out-A

(Static to FW

per VRF)






L3Out-B

(Static to FW

per VRF)




Local-Internet

1) Intra-VRF default routes from ASR9k to Fabric to Internet Only

2) Other VRF(s) will have default-route point to Firewall and Firewall will route to Intranet; based on FW policy

Static Routes: Logical SVI Interface / VRF

Static Routes: Static Routes for Inter-Context Firewall Communication (VRF)

Other Context/VRF

HRD VRF Firewall InterfaceExternal Network

LRD Firewall Interface

Intra-VRF and Intera-VRF Traffic Flows



Extranet

OSPF AREA

0.0.0.0

L3Out






OSPF AREA

0.0.0.0

L3Out




Local-Internet

Inter-VRF Flow

Intra-VRF Flow

Logical

Static Routes to Intra

Static Routes to Intra

Intra

Risk-Doamin-A

Risk-Domain-B

OSPF

Area0

Extranet IntranetLocal-Internet

OSPF

Area0

OSPF

Area0

OSPF

Risk-Domain-A Risk-Domain-B Other(s)Intra

OSPF

Static

Static

Static

0.0.0.0/0 sent to fabric VRF intra from ASR9000

End to End IP Multicast

End to End Multicast: Configuration steps

1. Configure OSPF and MP-eBGP between the ASR1000(s) and ASR9000(s) per VRF

2. Enable the Multicast address-family only for MP-eBGP

3. ASR9000 originates default routes to ASR1000 via multicast address family

4. Configure Anycast RP and MSDP between ASR1000(s)

5. Configure Anycast RP and MSDP between ASR9000(s)

6. Configure inter-domain MSDP between ASR1000(s) and ASR9000(s)

7. Configure PIM on the path between sources and receivers

8. Send Mcast traffic, and verifiy the remote receiver can receive the mcast traffic without loss.

For YourReference

End to End MulticastASR1000 PIM Multicast

ACI Fabric

Multicast Sources

Multicast Receivers

WAN/MAN/Multicast

Multicast Sources

Multicast Receivers

ASR9000 PIM Multicast

- The ASR9000 interfaces connected to the ACI border-leaf(s) / fabric does NOT have Multicast (PIM) enabled.

- So, the ASR9000 WAN routers will not inject multicast from remote source into the fabric directly, it will flows

via the ASR1000(s).

- Also, the Multicast sources will not send Multicast traffic directly to ASR9000(s); it will also flow through the

ASR1000(s).

OSPF AREA

0.0.0.5

vB

GP

AS

# 3

00

1

vB

GP

AS

# 3

00

0

Multicast Domain #2

Multicast Domain #1

End to End Multicast

ACI Fabric

Multicast Sources

Multicast Receivers

WAN/MAN/Multicast

Multicast Sources

Multicast Receivers

- ASR1000 and ASR9000 are directly connected L3 sub-interfaces

- OSPF is enabled between the ASR1000 and ASR9000

- PIM is enabled on these interfaces for multicast RPF check

- Support for both PIM-ASM & SSM; IGMPv2 and v3 receivers

OSPF AREA

0.0.0.5

vB

GP

AS

# 3

00

1

vB

GP

AS

# 3

00

0

MP-eBGP Session

MP-eBGP Session



End to End Multicast

• Exchange multicast source information with ASR9000 via MP-BGP

• MP-eBGP will carry IP Multicast address-family.

• The ASR9000 will learn the inside multicast sources via ASR1000(s) and originate default route to

ASR1000(s) in the multicast address family.

• Inter-domain MSDP for exchanging sa-cache

• Anycast-RP and MSDP between two ASR1000(s) & between the ASR9000(s)

Multicast Domain #2

Multicast Domain #1

ACI Fabric

Multicast Sources

Multicast Receivers

WAN/MAN/Multicast

Multicast Sources

Multicast Receivers

OSPF AREA

0.0.0.5

vB

GP

AS

# 3

00

1

vB

GP

AS

# 3

00

0



Multicast Source/MSDP

Multicast Source/MSDP

End to End Multicast: Traffic flows

Multicast traffic flows were verified and monitored under different failure scenarios;

1) Intra VLAN:

L2 multicast with sources and receivers attached to different leafs within the fabric

2) Inter VLAN:

L3 multicast with routing via the ASR1K. Sources and receivers are attached to different leafs within

the fabric

3) External Multicast Source:

The ASR9K routes multicast traffic via the ASR1K towards receivers attached to the ACI fabric.

4) External Multicast Receiver:

The ASR1K routes multicast traffic from sources within the ACI fabric via the ASR9K towards

receivers in the corporate Intranet.

For YourReference

vCentre VMM Integration with ACI/APIC

ACI and VMM vCentre Integration

- Cisco APIC integrates with the VMware vCentre.

- Ability to transparently extend the Cisco ACI

policy framework to VMware vSphere

workloads.

- APIC uses Application Network Profiles (ANPs)

to represent the Cisco ACI policy.

- APIC creates a virtual distributed switch (VDS)

in VMware vCentre for virtual networking.

- APIC manages all application infrastructure

components. The network administrator creates

EPGs and pushes them to VMware vCentre as

port groups on the DVS.

- Server administrators can then associate the

virtual machines and provision them

accordingly.


- Show configured VMware VMM

vCentre

- Focusing on vCentre 6 instances

vCentre 6 instance integrated into APIC

ACI: EPG/ANP

- Create EPG


- Add VMM Domain to EPG

- This will create port-group to

vCentre

ACI: EPG(s) pushed to vCentre Port-groups

- port-groups on the

vDS


Failure Scenarios

Failure Scenarios and Outages

1) OSPF Failover: SVI - ASR9K Failure

ASR9K-1 Power supply down:

OSPF Dead timers:

Intra 20s

LRD 40s

HRD 80s

Traffic outage time:

Intra 18s

LRD 36s

2) OSPF Failover: Point-to-Point - ASR9K Failure

Traffic outage time:

LRD 2.5s

Intra 2.7s

3) Uni-Cast Traffic: Transit - Border Leaf Failure

Border Leaf-232 Failure with Unicast Traffic Flow 4

(Intranet-VRF) outage times:

Inbound - 1.4s

Outbound - 1.7s

UCS Director work-flows

UCS Director workflows- Provision new server

- Decommission server

- ACI - Create Context

- ACI - Create Bridge Domain

- ACI - Create EPG

- ACI - Create Application Profile

- ACI - Create Contract

- ACI - Assign EPG to PortChannel/Alias

- ACI - Unassign EPG from PortChannel/Alias

- ACI Combined Provisioning Workflow

- ACI Combined De-provisioning Workflow

- Create a data LUN (array based on 'class') for presentation via VPLEX

- Expand LUN and volume

- Remove LUN and volume

- Present virtual volume to a host

- Present virtual volume to a RP cluster

Q & A

Complete Your Online Session Evaluation

Learn online with Cisco Live!

Visit us online after the conference

for full access to session videos and

presentations.

www.CiscoLiveAPAC.com

Give us your feedback and receive a

Cisco 2016 T-Shirt by completing the

Overall Event Survey and 5 Session

Evaluations.– Directly from your mobile device on the Cisco Live

Mobile App

– By visiting the Cisco Live Mobile Site http://showcase.genie-connect.com/ciscolivemelbourne2016/

– Visit any Cisco Live Internet Station located

throughout the venue

T-Shirts can be collected Friday 11 March

at Registration

http://www.ciscoliveapac.com/

http://showcase.genie-connect.com/ciscolivemelbourne2016/

Thank you

Documents

Real World Fabric Based Deploymentd2zmdbbm9feqrf.cloudfront.net/2016/anz/pdf/BRKDCT-2083.pdfInter-DC Core (Layer-3 IP/MPLS) BGP AS#65500 ... Building Your IP Network –Interface Principles