1. Argentina, Chile, Mexico, Puerto Rico, BrasilMay-Ago/2012
2011 Cisco and/or its affiliates. All rights reserved. Cisco
Confidential 1
2. Cisco Unified Data CenterArchitecture & Evolution Carlos
Pereira Distinguished Systems Engineer II Data Center May/2012 2011
Cisco and/or its affiliates. All rights reserved. Cisco
Confidential 2
3. Need for better high availability (HA) and lower fate
sharingNeed to achieve higher scalability (L2 Multipath, L3, MAC,
VMs, etc.)Need to accommodate diverse workloads concurrentlyNeed to
further simplify operational modelsNeed better network
visibilityNeed to be prepared for:These require Cisco to address:
Intel next-gen CPUs Increase feature, function and scale without
increasing PCIe 3.0complexity 10G LOMsContinue to leverage/develop
standards protocols to 10G-T support open and interoperable
environments 40G Uplinks Increase visibility, instrumentation and
manageability 100G Interconnects QSFP+ optics Evolutionary Steps
prevent disruption to operational models 3
4. Denser Server Cabinets, Denser PODs & Denser
X-connectsLonger cable distances, diverse connector typesServer
Migration to 10Guplinks to higher density 10G or 40GEarly 40G
server adoption specific workloadsHigher 10G and 40G switch density
per RU & Denser 2nd switch tierFlexible L2/L3 Boundary
Placement & Redundancy from 1+1 to N+1Virtual Machine &
bare metal mobility: within and across DCsApplication logical
isolation at scaleApplication processing closer to the wire4
5. Cisco UnifiedData Center Fabric 5
6. client-to-server client-to-serverAggregation
AggregationL3ServiceServiceprocessingsrv-to-srv or vm-to-vm
processingAccessAccessL2 srv-to-srv srv-to-srv or vm-to-vm
srv-to-srvVirtual Access Virtual AccessL2VM-to-VM srv-to-srv or
vm-to-vmsrv-to-srvVM-to-VMsrv-to-srv srv-to-srv srv-to-srv
srv-to-srvSrv-to-clientSrv-to-clientsrv-to-srvTraffic Patterns
Changing: More server to server traffic, and more L2 server to
server traffic. Apps such as VM mobility, clustering, intra-Tier
and largersubnets Client to server traffic to same subnet instances
across DCs: increase of /32 from DC out Virtual Server Environments
could perform a fair degree of local switching Server Roll-outs and
workload movement require physical and network infrastructure
coordination 2011 Cisco and/or its affiliates. All rights
reserved.Cisco Confidential 6
7. Hypervisor based server virtualization and the Data Center
Row 1 associated capabilities (VM Mobility, ) are changing multiple
aspects of the Data Center design Where is the server now? Where is
the access port? Where does the VLAN exist? Any VLAN Anywhere? How
large do we need to scale Layer 2? Data Center Row 2 What are the
capacity planning requirements for flexible workloads? Where are
the policy boundaries with flexible workload (Security, QoS, WAN
acceleration,)? 2011 Cisco and/or its affiliates. All rights
reserved. Cisco Confidential 7
8. Server, Storage, Application and Facilities are driving
Layer 2 Scalability requirementsServer Virtualization and
Clusteringdriving the need for every / any VLANeverywhere based
designFacilities requirements defining thenetwork topology No watt
shall beleft behindVM requirements along with Data Storage growth
mandating a need for more efficient and pervasive network based
storage Technology changes will impact any cabling plant
designMigration to 10GE as the default LoMtechnology 2011 Cisco
and/or its affiliates. All rights reserved. Cisco Confidential
8
9. Ethernet is the network for Data Center! 2011 Cisco and/or
its affiliates. All rights reserved. Cisco Confidential 9
10. IEEE 802Evolution of Ethernet 10 GE, 40 GE, 100 GE, copper
and fiberEvolution of switching DCB: Data Center Bridging 802.1BR:
Bridge Port Extension INCITS/T11Evolution of Fibre ChannelFCoE
(Fibre Channel over Ethernet) IBTA (Infiniband Trade
Association)RoCE (RDMA over converged Ethernet), aka IBoE or RoE
IETFLayer 2 Multi-Path (L2MP)TRILL (Transparent Interconnection of
Lots of Links) 2011 Cisco and/or its affiliates. All rights
reserved. Cisco Confidential 10
11. 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential 11
12. Scaling UP the Network Pod and Scaling OUT the Fabric
Scaling Up of the building blocks (High Density 10G, Unified IO,
FEX, Adapter-FEX, vPC, FabricPath) Scaling Out of the Fabric
(FabricPath, OTV, OSPF/EIGRP/ISIS/BGP, MPLS)Scaling OUT the Fabric
Scaling UP the aggregation blockVM VM VM(POD)VM VM VM #2 #3 #4VM VM
VM #2 #3 #4#2 #3 #4 2011 Cisco and/or its affiliates. All rights
reserved. Cisco Confidential 12
13. TraditionalScalable PODs Scalable FabricTopological
Approach FEX and switch scaling Multipathing Strategic
InvestmentsL2 / L3 10/40 GbE Leadership Fabric Scale (L2 & L3)
Manageability and Programmability Enterprise, SMB Location
Independence Cloud providers, large DC Enterprise, SMB, HPC/
Mobility 100s 1,000s Control Plane Intelligence100s -
10,000s10,000s - 100,000sservers per POD Servers/VMs per
PODServers/VMs per POD Simplicity of Management 2011 Cisco and/or
its affiliates. All rights reserved.Cisco Confidential 13
14. 2010 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential 1414
15. Network Planes of Operation The business glue of the
network. Rules execution, decisionPolicy making, Service Manager
and all the other components to make aPlaneproductize service.
ServicesOverlay Layer 7 application flow built on the foundation of
thePlaneother layers. Dependent on the other layers.Management The
management plane is the logical path of all traffic related to
Plane the system management of the platform. Control Its the brain
of any networking platform and the technical glue of the network.
The control plane is where all routing, switching,Planeother
protocols and control information are exchanged The data plane
receives, processes, and transmits network data Data Planebetween
network elements, and represents the bulk of network traffic that
passes to and through the gear. 2011 Cisco and/or its affiliates.
All rights reserved. Cisco Confidential 15
16. Fully specifies a Port Extender (FEX Equivalent)Extended
Bridge Extends ports of a switch to lower entities in a network
Port Extenders are not individually managed Controllin g Their
ports become ports of the controlling switchPE Bridge Cascading
Port Extenders Allows one to choose the appropriate controlling
switch PEPE Frame replication supported for efficient multicast /
floodingBridgePE Traffic from each Extended Port is reliably
segregated to an E-channel and identified by a tag containing an
E-channel identifier (ECID) Does not require prior knowledge of MAC
addresses; switch performs standard learning functionsServer PE
Works with all devices including VEBs, VEPAs, individual VMs,ECID
physical services, and devices providing transparent servicesVMvF
1W Controlling Bridge + PE = Extended Bridge Single Point of
Management PE Port Extender 2011 Cisco and/or its affiliates. All
rights reserved.Cisco Confidential 16
17. Distributed Modular System to the ToR, Server, and Virtual
MachineOne NetworkParent Switch to Top of RackFEX
ArchitectureNetworkAdministrator IEEE 802.1 BR*Consolidates network
managementFEX managed as line card of parentMany
applicationsFEXswitchrequireUses Pre-standard IEEE 802.1Qbhmultiple
interfacesLegacy*IEEE 802.1QR Pre-Standard 2011 Cisco and/or its
affiliates. All rights reserved. Cisco Confidential 17
18. Distributed Modular System to the ToR, Server, and Virtual
MachineOne Network Parent Switch to AdapterNetworkAdministrator
IEEE 802.1 BR* Adapter FEX FEX Consolidates multiple 1Gb
interfaceMany applications into a single 10Gb
interfacerequiremultiple interfacesExtends network into server Uses
Pre-standard IEEE 802.1QbhIEEE 802.1 Qbh*Legacy Adapter FEX *IEEE
802.1QR Pre-Standard 2011 Cisco and/or its affiliates. All rights
reserved.Cisco Confidential 18
19. Fabric Extender Evolution Distributed Modular System to the
ToR, Server, and Virtual Machine One NetworkVirtual Same As
PhysicalNetworkAdministrator IEEE 802.1 BR* FEX VM-FEXConsolidates
virtual and physicalnetworkEach VM gets a dedicated port onIEEE
802.1 Qbh* IEEE 802.1 Qbh* switchUses Pre-standard IEEE
802.1QbhHypervisorVM networkmanaged byServeradministratorLegacy
Adapter FEXVM-FEX*IEEE 802.1QR Pre-Standard 2011 Cisco and/or its
affiliates. All rights reserved. Cisco Confidential 19
20. Distributed Modular System to the ToR, Server, and Virtual
Machine One Network Parent Switch to Application Single Point of
ManagementNetworkAdministratorFEX Architecture IEEE 802.1 BR*Manage
network all Consolidates network management FEXthe way to FEX
managed as line card of parentthe OS interface switchPhysical and
Adapter FEXVirtual Consolidates multiple 1Gb interfaceIEEE 802.1
Qbh* IEEE 802.1 Qbh* into a single 10Gb interface Extends network
into server VM-FEX Consolidates virtual and physicalHypervisor
network Each VM gets a dedicated port on switchLegacy Adapter FEXVM
FEX *IEEE 802.1QR Pre-Standard 2011 Cisco and/or its affiliates.
All rights reserved.Cisco Confidential 20
21. 1 2 3 4 5786 SwitchNexus 5500 EthEth 1 2 345Port
Extension802.1BR PE Tag 802.1BRPortExtender1Nexus 2200 (FEX)123 PE
TagServer 802.1BRAdapter Port 0Port 1 Port n HypervisorNIV Capable
Adapter VM VM VM VM VM VM vNICvNIC vNICvNIC vNIC1 23 45IEEE Bridge
Port Extender= Cisco FEX (Fabric Extender) 2011 Cisco and/or its
affiliates. All rights reserved. Cisco Confidential 21
22. Nexus 2000 Fabric Extender (FEX)Cisco Nexus 7000 Cisco
Nexus 5500+ +Distributed High DensityEdge Switching SystemCisco
Nexus 2000 FEX Cisco Nexus 2000 FEX 2011 Cisco and/or its
affiliates. All rights reserved. Cisco Confidential 22
23. MultiChassis EtherChannel (MCEC) vPC is a Port-channeling
conceptextending link aggregation to two separatephysical switches
Allows the creation of resilient L2topologies based on Link
Aggregation.Physical Topology Logical Topology Eliminates the need
for STP in theVirtual Port Channel access-distribution L2 Provides
increased bandwidth Si Si All links are actively forwarding vPC
maintains independent control planeNon-vPCvPCIncreased BW with vPC
2011 Cisco and/or its affiliates. All rights reserved. Cisco
Confidential 23
24. Co-existence of LAN and SAN LAN and SAN utilize different
High Availability Models SAN is dual fabric, LAN is fully meshed
fabric vPC enables both architectures at the edge (single device
models not acceptable to SAN customers) WAN Core FCCore
CoreL3AggregationL2 Access Edge 2011 Cisco and/or its affiliates.
All rights reserved.Cisco Confidential 24
25. 2 Rack 13 Rack 14 Rack 24 2011 Cisco and/or its affiliates.
All rights reserved. Cisco Confidential 25
26. Cisco Nexus 5x00 and 2200represent a virtual accessswitch
POD VPC pair Nexus 7000 at AggregationLayer NO Loop Nexus
5x00/2200VirtualizedAccess Switch PODs ...NO STP 2011 Cisco and/or
its affiliates. All rights reserved.Cisco Confidential 26
27. Nexus 7000 vPC+Unified ComputingSystem (UCS)Nexus 5000 /
5500 + 2200Virtual Access Switch PODNexus 7000 + 2200Virtual Blade
SwitchingVirtual Access (VBS) Switch POD 2011 Cisco and/or its
affiliates. All rights reserved. Cisco Confidential 27
28. 2010 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential 2828
29. Large Scale Web 2.0 Environments L31. Soft L3 on access -
per ToR VLANs L32. East-west traffic: 2 tiers large scale L3 3.
Limited VLAN extension overlays (ex.: OTV) L2 L3 Enterprise
Environments1.VPC / STP used for L2 restricted VLAN Scale L3 L2
2.Pod traffic: 2 tiers limited scale3.Cross-pod East-West traffic =
3 tiers L2 L3 L2 Large Scale SPDC: Hosting & Cloud1. L2 on
access and aggregation L22. VLAN Scale limited high fate sharing3.
East-west traffic: 2 or 3 tier L2 2011 Cisco and/or its affiliates.
All rights reserved. Cisco Confidential 29
30. Data Center Fabric Topologies & Attributes - Trending
L3 Cloud Large Scale Web 2.0 Environments1. L3 on access - per ToR
VLANs L3 2. Migration to 10G: 3 tiers very large scale3. Broad VLAN
extension through overlays L3 L2 L3 FabricsCommonalities between
Enterprise & SPDC1. L2MP (FabricPath) used for L2 increase VLAN
spread L3 L3 Cloud 2. 2-tier east to west traffic L3 3. N-way tier
2 (spine) L2Differences between Enterprise & SPDC L21. Host
Density L3/L22. VLAN Scale virtualization scaleFabrics3.
Public-cloud vs private cloud 2011 Cisco and/or its affiliates. All
rights reserved. Cisco Confidential 30
31. The protocol choices allow like topologies to be buildEqual
Cost Multi Path (ECMP) over L2 or L3Plug and Play Nature of L2
ProtocolsRedundancy, stability and scale of Layer 3 ProtocolsHigh
Availability models become similar for L2 and L3: N+1
redundancyDifferent Environments Have:A preferred placement for the
L2/L3 boundaryA High-Availability, Scale, and functional
targetsThese Converge given the flexibility offered by the
forthcoming protocolsChoice of L2 or L3 Protocols does not prevent
redundancy and multi-pathing requirementsLocation of L2/L3 boundary
does not prevent adjacency or redundancy optionsL2/L3 Boundary
becomes less relevantCLOS Topologies dominate new
implementationsHigh Availability models shiftServer Edge becomes
more intelligentData Center Fabric becomes more scalable 2011 Cisco
and/or its affiliates. All rights reserved.Cisco Confidential
31
32. L3 L3/L2 L3/L2L2L2 East-West traffic Fate Sharing Domain
Larger POD East-West Traffic Fate Sharing Domain STP has been the
protocol of choice N+1 redundancy 1+1 redundancy limited forwarding
paths IS-IS is the protocol of choice Replicated Stateful Services
per PodBroad forwarding paths Broader Adjacency Support East-West
across L3 boundaries Service Insertion is not replicated OSPF/EIGRP
are protocols of choice N+1 redundancy Broad forwarding Paths Same
number of physical boxes and links North-South traffic Protocol
behavior is L3-like OSPF/EIGRP are protocols of choiceMulti-pathing
over L2 and L3 N+1 redundancy Broad forwarding paths More flexible
L2 adjacency, better scale capacity Better latency consistency
within POD 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential 32
33. WHY should I andHOW can Ileverage this evolution on my
ownData Center infrastructure 2011 Cisco and/or its affiliates. All
rights reserved. Cisco Confidential 33
34. Focused on Stability FeaturesN Network port (Bridge
Assurance)E Edge port- Normal port type Data CenterB BPDUguard Core
R RootguardL LoopguardF Global BPDU filter HSRP HSRP ACTIVE STANDBY
Layer 3 AggregationN N Backup Root RootLayer 2 (STP + Bridge
Assurance)N N N -N N N -R R R R R R R RLayer 2 (STP + BA +
Rootguard)N N AccessNN N NL L E F E FE F E F E F B BB B BLayer 2
(STP + BPDUguard) 2011 Cisco and/or its affiliates. All rights
reserved.Cisco Confidential 34
35. Focused on Scalability Features, with a pair of Aggregation
boxes. N Network port (Bridge Assurance) E Edge port - Normal port
typeData CenterB BPDUguardCore R Rootguard L Loopguard F Global
BPDU filterVPCHSRP domain HSRPACTIVEACTIVELayer 3AggregationNN
RootRoot Layer 2 (STP + Bridge Assurance)- - - - -- - - R R R RR R
R RLayer 2 (STP + Rootguard)-Access-- L EEEEE FFFFF BBBBBLayer 2
(STP + BPDUguard) 2011 Cisco and/or its affiliates. All rights
reserved.Cisco Confidential 35
36. MAC addresses encode no location or network hierarchy
Default forwarding behavior in bridged network is flood MAC
filtering database limits scope of flooding Ultimately, may not
scale well as every switch learns every MAC MAC TableMAC
TableAALayer 2Domain MAC Table MAC TableMAC Table A MAC TableAA A
2011 Cisco and/or its affiliates. All rights reserved.Cisco
Confidential 36
37. MAC v.s. IP Network Address
10.0.0.0/240011.1111.111110.0.0.10 /24Non-hierarchical Host Address
Address 10.0.0.10 0011.1111.1111 0011.1111.111110.0.0.0/16
20.0.0.0/16 0011.1111.1111 10.0.0.0/24 20.0.0.0/24
0011.1111.11110011.1111.111110.0.0.10 20.0.0.20 L2 Forwarding
(Bridging)L3 Forwarding (Routing) Data-plane learning Control-plane
learning Flat address space and forwarding Hierarchical address
space andtable (MAC everywhere !!!)forwarding Flooding required for
unknown unicast Only forwarding to destinationdestination addresses
with matching routes in the Destination MACs need to be
knowntablefor all switches in the same network toFlooding is
isolated within subnetsavoid flooding No dependence on data-plane
formaintaining forwarding table 2011 Cisco and/or its affiliates.
All rights reserved.Cisco Confidential 37
38. What Can Be Improved? Network Address Scheme:
FlatHierarchical Additional header is required to allow L2 Routing
instead of Bridging. Switch ID comes to the picture. Provide
additional loop-prevention mechanism like TTL Address Learning:
Data Plane Control Plane Eliminate the needs to program all MACs on
every switches to avoid flooding Control Plane: Distance-Vector
Link-State Improve scalability, minimize convergence time, and
allow multipathing inherently The ultimate solution needs to take
both control and data plane intoconsideration this time!!! 2011
Cisco and/or its affiliates. All rights reserved. Cisco
Confidential 38
39. Cisco FabricPathData Plane InnovationControl Plane
InnovationFabricPath encapsulation Plug-n-Play Layer 2
IS-ISConversation LearningSupport unicast and multicastRouting, not
bridgingFast, efficient, and scalableBuilt-in loop-mitigationEqual
Cost Multipathing Time-to-Live (TTL) (ECMP) RPF Check VLAN and
Multicast Pruning Cisco NX-OS Cisco Nexus Platform 2011 Cisco
and/or its affiliates. All rights reserved. Cisco Confidential
39
40. Multi-Domain SilosFabricPath Any App, Anywhere!FabricWeb
ServersApp Servers New AppsWeb ServersApp ServersSilo 1 Silo 2Silo
3 New Apps Benefits server team by providing a network Fabric that
looks like a single switch Breaks down silos, permits workload
mobility, provides maximum flexibility Lowers OPEX by simplifying
server team operation Reduces dependency on/interaction with
network team 2011 Cisco and/or its affiliates. All rights
reserved.Cisco Confidential 40
41. Externally, a Fabric looks like a single switch Internally,
a protocol adds Fabric-wide intelligence and ties the elements
together. This protocol provides in a plug-and-play
fashion:Optimal, low latency connectivity any to anyHigh bandwidth,
high resiliencyOpen management and troubleshooting Cisco FabricPath
provides additional capabilities in term of scalability and L3
integrationFabricPath FabricPath 2011 Cisco and/or its affiliates.
All rights reserved.Cisco Confidential 41
42. Enabling Network Fabrics FabricPath Connect a group of
switches using an arbitrary topology With a simple CLI, aggregate
them into a Fabric: N7K(config)# interface ethernet 1/1
N7K(config-if)# switchport mode fabricpath An open protocol based
on L3 technology provides Fabric- wide intelligence and ties the
elements together 2011 Cisco and/or its affiliates. All rights
reserved.Cisco Confidential 42
43. Example 1: Classical POD Migration Q: Why migrate
traditional Access/ Aggregation building block to FabricPath? A: No
STP No STP sync, no topology changes, no blocked ports, no risk of
loops Simple configuration Total flexibility in design and cabling
Enables organic bandwidth growth Grow where and whenever needed
with minimal impact 2011 Cisco and/or its affiliates. All rights
reserved. Cisco Confidential 43
44. Q: How to interconnect DC PODs in order to have VLANs
anywhere? A: Provide server/host connection to any edge port in
network, regardless of physical location Physical/rack/distribution
pair location of host irrelevant with respect to IP subnet and
Layer 2 adjacency with other hosts Gateway placement options
include GLBP, MHSRP and leaf-attached gateways, so far.POD 1 POD 2
POD 3 PODS 1-3 VLANs 100-199VLANs 200-299VLANs 300-399 VLANs
100-399 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential 44
45. Cisco FabricPathSpanning-Tree vPC
FabricPath16SwitchesActive PathsSingleDual 16 WayPodUp to 10 TbpsUp
to 20 Tbps Up to 160 TbpsBandwidth 2011 Cisco and/or its
affiliates. All rights reserved.Cisco Confidential 45
46. PodSpineTier 2 Leaf Tier 1 2011 Cisco and/or its
affiliates. All rights reserved. Cisco Confidential 46
47. Terminology Interface connected to another FabricPath
deviceSends/receives traffic with FabricPath headerDoes not run
spanning treeDoes not perform MAC learning!Exchanges topology info
through L2 ISIS adjacencyForwarding based on Switch ID TableFP Core
Ports S10 S20 S30S40 Spine Switch FabricPath (FP)S100 S200 S300
Leaf Switch1/1 1/2 Classical Ethernet (CE)A B CE Edge
PortsInterface connected to traditional network
deviceSends/receives traffic in standard 802.3 Ethernet
frameformatParticipates in STP domainForwarding based on MAC table
2011 Cisco and/or its affiliates. All rights reserved.Cisco
Confidential 47
48. A New Data Plane The association MAC address/Switch ID is
maintained at the edge S10 S20S30S40Switch ID space:S300:
FabricPathRouting Routing Tabledecisions areA B S100 S300made based
on SwitchIFthe FabricPathrouting tableFabricPath (FP) S100 S200
S300 S100L1, L2, L3, L4MAC address1/11/2S300: CE MACspace: Address
TableSwitching based Classical Ethernet (CE) MACIFon MAC address
ABB1/2tables A S100 Traffic is encapsulated across the Fabric 2011
Cisco and/or its affiliates. All rights reserved. Cisco
Confidential 48
49. Conversational MAC Learning S10 S20 S30 S40A B S100M
FabricPath Lookup B: HitS100S200 S300 Learn source ALookup B:
MissFloodLookup B: Miss Dont learn S100: CE MAC1/1S200: CE MAC
1/2S300: CE MAC Address TableAddress Table Address TableMAC IFMAC
IF MACIF A B A 1/1B 1/2AS100 Classical Ethernet 2011 Cisco and/or
its affiliates. All rights reserved. Cisco Confidential 49
50. Conversational MAC Learning S10 S20S30 S40S300: FabricPath
Routing Table B A S300 S100Lookup A: HitLookup A: Hit SwitchIF
Learn source B FabricPath Send to S100S100 S200 S300 S100 L1, L2,
L3, L4 S100: CE MAC1/1 S200: CE MAC 1/2S300: CE MAC Address Table
Address Table Address TableMAC IF MACIF MAC IF AB A1/1 B1/2 B S300
A S100 Classical Ethernet Conversational Learning 2011 Cisco and/or
its affiliates. All rights reserved.Cisco Confidential 50
51. Its a Routed Network Describes shortest (best) paths to
each Switch IDbased on link metrics Equal-cost paths supported
between FabricPathswitchesS10 S20 S30S40FabricPathRouting Table
onS100 Switch IFOne best path S10L1to S10 (via L1)S20L2S30L3S40L4
S200 L1, L2, L3, L4 S100 S200FabricPathS300 Four equal-cost S300
L1, L2, L3, L4paths to S300 2011 Cisco and/or its affiliates. All
rights reserved. Cisco Confidential 51
52. (1) Broadcast ARP RequestRoot for Root forMulti-destination
Tree 1 Tree 2Trees on Switch 10S10S20 S30 S40 Tree4IFFtag 1
po100,po200,po300po3002 po100DAFFpo100
po200Ftag1SA100.0.12Multidestination DAFFDMACFFFtag1Trees on Switch
100po10 po20 po30 po40SMACA po10 po20 po30 po40 SA100.0.12Tree 3IF
S300Payload DMACFFS100 Broadcast 1 po10 S200 SMACA
2po10,po20,po30,po40Multidestination Payload Trees on Switch 300 5
6 FabricPath e1/13Tree IF e2/29 Payload MAC Table on S100DMACFF
SMACASMACAFtag 1po10,po20,po30,po40MAC IF/SID DMACFF2 po40Ae1/13
(local) 2PayloadMAC A1 MAC B FabricPath MAC Table on
S200MACIF/SIDDont learn MACs fromLearn MACs of directly-connected
framesflooddevices unconditionally 2011 Cisco and/or its
affiliates. All rights reserved. Cisco Confidential 52
53. MAC Address Table after the first ARP frame S100:S100# sh
mac address-table dynamicLegend:* - primary entry, G - Gateway MAC,
(R) - Routed MAC, O - Overlay MACage - seconds since last seen,+ -
primary entry using vPC Peer-LinkVLAN MAC AddressTypeage Secure
NTFY
Ports/SWID.SSID.LID---------+-----------------+--------+---------+------+----+------------------*
10 0000.0000.000a dynamic 0FF Eth1/13MAC A learned aslocal entry on
e1/13 S10 (and S20, S30, S40, S200):S10# sh mac address-table
dynamicLegend:* - primary entry, G - Gateway MAC, (R) - Routed MAC,
O - Overlay MACage - seconds since last seen,+ - primary entry
using vPC Peer-LinkVLAN MAC AddressTypeage Secure NTFY
Ports/SWID.SSID.LID---------+-----------------+--------+---------+------+----+------------------
S300: MAC A notS300# sh mac address-table dynamic learned on other
switchesLegend:* - primary entry, G - Gateway MAC, (R) - Routed
MAC, O - Overlay MACage - seconds since last seen,+ - primary entry
using vPC Peer-LinkVLAN MAC AddressTypeage Secure NTFY
Ports/SWID.SSID.LID---------+-----------------+--------+---------+------+----+------------------
2011 Cisco and/or its affiliates. All rights reserved. Cisco
Confidential 53 53
54. (2) Broadcast ARP Reply Root forRoot for
MultidestinationTree 1Tree 2 S10S20S30S40 Trees on Switch 1010
TreeIF Ftag 1po100,po200,po300 po3002po100 DAMC1 Ftag1po100 po200
SA300.0.64DAMC1Ftag1 DMACAMultidestinationSA300.0.64Trees on Switch
100SMACB po10 po20 po30po40DMACApo10 po20 po30 PayloadTree11IFpo40
S300 SMACBFtag 1 po10S200MultidestinationPayload 2
po10,po20,po30,po40Trees on Switch 3009 Tree IF7FabricPath e1/13MAC
Table on S100Payload Unknown 1 po10,po20,po30,po40 e2/29DMACA
2po40SMACB MAC IF/SIDSMACBPayload Ae1/13 (local) 12 DMACA MAC
AFabricPath B300.0.64 (remote) MAC Table on S300 MAC B 8 MAC MAC
IF/SIDIF/SIDA MISS Be2/29 (local) If DMAC is known, then learn
remote MAC *MC1 = 01:0f:ff:c1:01:c0 2011 Cisco and/or its
affiliates. All rights reserved.Cisco Confidential 54
55. MAC Address Table after the first ARP frame S100: S100# sh
mac address-table dynamic Legend:* - primary entry, G - Gateway
MAC, (R) - Routed MAC, O - Overlay MACage - seconds since last
seen,+ - primary entry using vPC Peer-LinkVLANMAC Address Typeage
Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 100000.0000.000a dynamic 90 F F Eth1/1310 0000.0000.000b dynamic
60 F F 300.0.64 S100 learns MAC Bas remote entryreached through
S100#S300 S300: S300# sh mac address-table dynamic Legend:* -
primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay
MACage - seconds since last seen,+ - primary entry using vPC
Peer-LinkVLANMAC Address Typeage Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 100000.0000.000b dynamic 0F F Eth2/29 MAC B learned as local
entry on S300# e2/29 2011 Cisco and/or its affiliates. All rights
reserved. Cisco Confidential 55 55
56. Unicast Data - RoutedFabricPath RoutingTable on S30
S10S20S30S40 SwitchIF S300 S300 po300 16po300 DA300.0.64DA300.0.64
FabricPath Routing Ftag1 Ftag1 Table on S100 SA100.0.12SA100.0.12
SwitchIF DMACBDMACB S10 po10 SMACASMACA po10 po20 po30 po40 S20
po20po10 po20 po30 PayloadPayloadHashpo40 S30 po30 S300 S40 po40
S200 FabricPath RoutingS100po10, po20,Table on S300S200po30,
po401715 SwitchIFpo10, po20,e1/13S300 PayloadS300 po30,
po40e2/29S300 S300Use LID (64) SMACA DMACB FabricPathDMACB MAC
Table on S100 SMACAFabricPath MAC AMACIF/SIDPayload MAC Table on
S300 MAC B Ae1/13 (local) 14 13MACIF/SID 18A S100.0.12 (remote) BB
300.0.64 (remote)If DMAC is known, thenlearn remote MACBe2/29
(local) 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential 56
57. Unicast forwarding S100:S100# sh mac address-table
dynamicLegend: * - primary entry, G - Gateway MAC, (R) - Routed
MAC, O - Overlay MAC age - seconds since last seen,+ - primary
entry using vPC Peer-Link VLAN MAC Address Typeage Secure NTFY
Ports/SWID.SSID.LID---------+-----------------+--------+---------+------+----+------------------*
10 0000.0000.000a dynamic 90 F F Eth1/1310 0000.0000.000b dynamic
60 F F 300.0.64S100# S300:S300# sh mac address-table dynamicLegend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay
MAC age - seconds since last seen,+ - primary entry using vPC
Peer-Link VLAN MAC Address Typeage Secure NTFY
Ports/SWID.SSID.LID---------+-----------------+--------+---------+------+----+------------------
S100 learns MAC A10 0000.0000.000a dynamic 30 F F 100.0.12 as
remote entryreached through S100* 10 0000.0000.000b dynamic 90 F F
Eth2/29S300# 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential 57 57
58. Unicast ForwardingS100# sh fabricpath routeFabricPath
Unicast Route Tablea/b/c denotes ftag/switch-id/subswitch-id[x/y]
denotes [admin distance/metric] Topology (ftag),ftag 0 is local
ftagSwitch ID, Sub-subswitch-id 0 is default subswitch-idSwitch ID
AdministrativeFabricPath Unicast Route Table for Topology-Default
distance, routing0/100/0, number of next-hops: 0metricvia ---- ,
[60/0], 0 day/s 04:43:51, local1/10/0, number of next-hops: 1via
Po10, [115/20], 0 day/s 02:24:02, isis_fabricpath-defaultRoute
age1/20/0, number of next-hops: 1via Po20, [115/20], 0 day/s
04:43:25, isis_fabricpath-default1/30/0, number of next-hops: 1via
Po30, [115/20], 0 day/s 04:43:25, isis_fabricpath-default1/40/0,
number of next-hops: 1 Client protocolvia Po40, [115/20], 0 day/s
04:43:25, isis_fabricpath-default1/200/0, number of next-hops: 4via
Po10, [115/40], 0 day/s 02:24:02, isis_fabricpath-defaultvia Po20,
[115/40], 0 day/s 04:43:06, isis_fabricpath-defaultNext-hopvia
Po30, [115/40], 0 day/s 04:43:06, isis_fabricpath-default
interface(s)FabricPathvia Po40, [115/40], 0 day/s 04:43:06,
isis_fabricpath-default1/300/0, number of next-hops: 4S10S20
S30S40via Po10, [115/40], 0 day/s 02:24:02,
isis_fabricpath-defaultvia Po20, [115/40], 0 day/s 04:43:25,
isis_fabricpath-defaultvia Po30, [115/40], 0 day/s 04:43:25,
isis_fabricpath-defaultvia Po40, [115/40], 0 day/s 04:43:25,
isis_fabricpath-defaultS100# po10po20po30 po40 S100 S200 S300 AB C
2011 Cisco and/or its affiliates. All rights reserved.Cisco
Confidential 5858
59. S3 FabricPath CE L1L2Introducing vPC+F1VPC+ F1S1F1F1S2
Allows dual-homed connections from edge ports intoFabricPath domain
with active/active forwardingF1F1 Can also provide active/active
HSRP Configuration virtually identical to standard VPCPhysicalHost
A Supported on both N7K (F1/F2) and 5500 VPC+ peer switches share a
virtual FabricPath switch ID LogicalS3 Host AS4L1,L2 MAC addresses
behind VPC+ port-channels appear asconnected to the virtual switch,
not the VPC+ physical L1L2peer switchesF1VPC+ F1 Allows
load-balancing within FabricPath domain toward theVPC+ virtual
switch S1 F1 F1S2 VPC+ requires F1/F2 modules on N7K with
FabricPath F1F1enabled in the VDC or FabricPath enabled on the
Nexus5500.S4Virtual Switch 4 becomes egress switch for Host A in
FabricPath domain Host A 2011 Cisco and/or its affiliates. All
rights reserved. Cisco Confidential 59
60. Driven by multiple vendors, including CiscoTRILL is now an
IETF proposed standardFabricPath will provide a TRILL mode with a
software upgrade (Cisco hardware is alreadyTRILL capable)Cisco is
pushing FabricPath specific enhancements to
TRILLFabricPathTRILLFrame routing YesYes(ECMP, TTL, RPFC etc
)vPC+YesNoFHRP active/activeYesNoMultiple topologies
YesNoConversational learning YesNoInter-switch links Point-to-point
only Point-to-point OR sharedhttp://datatracker.ietf.org/wg/trill/
2011 Cisco and/or its affiliates. All rights reserved.Cisco
Confidential 60
61. HSRPHSRPL3 B L3 B Active/Active Active/Standby FabricPathG1
G2 TRILLG1 G2 MultipathingMultipathinga1a2 a5 a6a1a2 a5a6VPC+
TRILLActive/ActiveActive/Standby ACA C End-to-end multipathing (L2
edge, Fabric, L3 edge) provides resiliency and fast convergence
2011 Cisco and/or its affiliates. All rights reserved.Cisco
Confidential 61
62. 2011 Cisco and/or its affiliates. All rights reserved.
Cisco Confidential 62