Nexus 5000 Deep Dive - Cisco · Nexus 5020. 56-Port L2 Switch • 40 Ports 10GE/FCoE, fixed • 2 Expansion Modules. Cisco Fabric Manager and Cisco Data Center Network Manager. Cisco

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 1

Nexus 5000 Deep DiveNexus 5000 Deep Dive


Agenda

System Hardware Overview

Internal Architecture

Fabric Data Path

Lossless data path

Forwarding and Policy Enforcement

NX-OS Overview

Select Switch Features

Network design


System Hardware Overview


Nuova Systems Product PortfolioIndustry’s First I/O Consolidation Virtualization Fabric for

Enterprise Data Center Industry’s First I/O Consolidation Virtualization Fabric for

Enterprise Data Center

OS

Nexus 502056-Port L2 Switch• 40 Ports 10GE/FCoE, fixed• 2 Expansion Modules

Cisco Fabric Manager and Cisco Data Center Network Manager

Cisco NX-OS

FC + Ethernet • 4 Ports 10GbE/FCoE • 4 Ports 1/2/4G FC

Mgmt

Expansion Modules

Ethernet • 6 Ports

10GE/FCoE

Eco System PartnersEco System Partners

http://www.panduit.com/default.asp


Rear PanelsNX5020NX5020

Expansion Module(s)

Cables connect in the rear for ease of server wiringCables connect in the rear for ease of server wiring

Power Entry

Base 10GE 10/100/1000

Out of Band Mgmt Console

All 10GE ports are FCoE capable!

Support for 1 GE Support for crypto


Front PanelsNX5020NX5020

N+1 redundant fans

Replaceable components on the front for easy accessReplaceable components on the front for easy access

Dual redundant power supplies


Power

NX5020Power Supply

NX5020Power Supply

Fully redundant, load sharing and hot swappableFully redundant, load sharing and hot swappable

Maximum Power – 750WTypical Operating Power – 480WAC Input - 208 VoltsEfficiency - 82-88%Protection - 110%-150% max load


Expansion Modules

Ethernet Expansion Module

Ethernet Expansion Module

CombinationExpansion Module

CombinationExpansion Module

Six 10G Ethernet

Four 10G Ethernet

Four 1/2/4GFibre Channel


Cooling

NX5020Cooling Module

NX5020Cooling Module

Max RPM - 12KFailover - N+1Op Temp - 0 to 40 CHumidity - 95% non-condensingElevation - 10K feet


SFP+ Transmission Media

CableTransceiver

Latency (link)Power

(each side)DistanceTechnology

Twinax ~0.25 μs~0.1W10mSFP+ CU Copper

MM OM2 MM OM3 ~0.1 μs1W82m

300mSFP+ SR short reach

MM OM2 MM OM3 ~0.1 μs1W10m

100mSFP+ USR

ultra short reach

Cat6 Cat6a/7 Cat6a/7

2.5μs 2.5μs 1.5μs

~8W ~8W ~4W

55m 100m 30m

10GBASE-T

•Low power consumption•Low cable cost•Low transceivers latency•Low error rate (10 exp-17)


Internal Architecture


Hardware Architecture


Supervisor Details

CPU 1.66 GHz Intel LV Xenon - LF80538KF0281M

IO Chip Set Intel 3100 South Bridge for embedded applications

DRAM 2 GBytes of DDR2 400 (PC2 3200) in two DIMM slots

Program Store 1 GBytes of USB based (NAND) Flash

Boot/BIOS 2 Mbytes of EEPROM with locked recovery image

On-board Fault Log 64 MBytes of Flash for failure analysis Kernel Stack traces, boot record and fault logs

NVRAM 2 Mbytes of SRAM – Syslog and licensing information

Secure Keystore Renesas AE46C1 – Credentials and secure RNG

Management Interfaces RS-232 console port – console0

10/100/1000BASE-T – mgmt0 partitioned from inband VLANs


Unified Crossbar Fabric58 port crossbar and scheduler

3 unicast and 1 multicast crosspoints

Central tightly coupled scheduler

Request, propose, accept, grant, acknowledge semanticsPacket enhanced iSLIP scheduler

Distinct unicast and multicast schedulers

Eight classes of serviceEgress buffer creditsDWRR class of serviceDWRR ingress interface

Total SRAM 24.6 MbitsGates 12.4 MillionTransistors ~200 MillionMetal Layers 7Signal Pins 1286

SerDes 232 @ 3.75Gbps


Unified Port Controller

Media Access Controllers1/10G Ethernet and 1/2/4G Fibre Channel

Packet Buffering and QueuingTotal of 1.875 MBytes used in four slices

Forwarding ControllerEthernet, Fibre ChannelLayered policy engine

Four data path slicesOne 1/10G Ethernet or two 1/2/4G Fibre Channel portsConnects to one Altos port

All switching done in Altos crossbar480 KBytes of buffering

Total SRAM 35 Mbits

Total TCAM 1 Mbit

Logic Gates 18 Million

Transistors ~300 Million

Metal Layers 7

Total Pins 900

SerDes 32 @ 3.75Gbps

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 16Unified Crossbar Fabric


Slice 4Slice 2 Slice 3

Switch ASIC Architecture

Slice 1

Forwarding

?

1/10G MAC

Transceiver

Virtual Queues

Egress Queues

Packet Buffer

Virtual Queues

Egress Queues

Packet Buffer


Slice 2 Slice 3Slice 1

Virtual Queues

Egress Queues

Packet Buffer

Slice 4

4 @ 3.75G – 12Gbps 4 @ 3.75G – 12Gbps 4 @ 3.75G – 12Gbps

XAUI – 10 Gbps4 @ 3.125G

Transceiver

XAUI – 10 Gbps4 @ 3.125G

1/10GE Attached Server

10GE LAN Uplink

58 source busses in total

Parsing &Editing

Forwarding

?Forwarding

Parsing &Editing

1/10G MACFC MAC

SAN B

FC MAC

Fibre Channel SAN Uplinks

1/2/4G Fibre Channel1 @ 1.0625/2.125/4.25G

Parsing &Editing

Fabric Buffer Fabric Buffer Fabric BufferUnicast and

Multicast Schedulers

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 17Unified Crossbar Fabric


Slice 4Slice 2 Slice 3

Packet Walkthrough

Slice 1

Forwarding

?

1/10G MAC

Transceiver

Virtual Queues

Egress Queues

Packet Buffer

Fabric Buffer

Virtual Queues

Egress Queues

Packet Buffer

Fabric Buffer


Slice 2 Slice 3Slice 1

Virtual Queues

Egress Queues

Packet Buffer

Slice 4

Forwarding

Fabric BufferUnicast and


4 @ 3.75G – 12Gbps 4 @ 3.75G – 12Gbps 4 @ 3.75G – 12Gbps

XAUI – 10 Gbps4 @ 3.125G

Transceiver

XAUI – 10 Gbps4 @ 3.125G

1/10GE Attached Server

10GE LAN Uplink

?Forwarding


Parsing &Editing

Parsing &Editing

1/10G MACFC MAC

SAN B

FC MAC

Fibre Channel SAN Uplinks

1/2/4G Fibre Channel1 @ 1.0625/2.125/4.25G

Parsing &Editing

1. Decode, align, synchronize bytes

Decrypt, verify, authenticate frames

5. Queue frames and manage crossbar service requests

4. Store frame content when waiting

6. Match requests, available outputs,

and fairness criteria

7. Landing place for frames in flight

9. Evaluate frame fields for, filtering,

and editing

3. Evaluate frame fields for forwarding, filtering, and editing

2. Extract frame fieldsAdd/remove headers

and edit frame contents

10. Encrypt frames and encode bytes

8. Extract frame fieldsAdd/remove headers and

edit frame contents


Switch Fabric Data Path


Unified Crossbar Fabric


Port 4

Data Path Deep Dive

Media Access Controllers

Crossbar operationUnicastMulticast

Latency

Port 1

Forwarding

?

1/10G MAC

Transceiver

Virtual Queues

Egress Queues

Packet Buffer

Fabric Buffer

Virtual Queues

Egress Queues

Packet Buffer



10GE Attached Servers


Parsing &Editing

1/10G MAC

Parsing &Editing

Transceiver


Media Access Controllers

Each Unified Port Controller slice has…One 1 Gigabit Ethernet MACOne 10 Gigabit Ethernet MACTwo 1/2/4 Gigabit Fibre Channel MACs

Two of the slices in each Gatos have an 802.1AE LinkSec encryption engine

Integrated Flow Control handlingEthernet – 802.3X “PAUSE” and Cisco Priority Flow ControlFibre Channel – BB_credits


Crossbar Overview

Tightly coupled scheduler and crosspoint20% link speedup

12 Gbps

Unicast SchedulerVirtual Output Queuing3x fabric speed up

3 crosspointsMultiple frames transferred per scheduling event

“Superframing”

Multicast SchedulerSystem Class queuingSeparate crosspoint


Unicast Virtual Output Queuing


Day in the Life of a Unicast Frame

1. Frame arrive into Packet buffer

2. Frame pointer posted to Virtual Output Queue

3. VOQ posts request to Scheduler

4. Scheduler arbitrates and grants access

5. Frame sent to Fabric Buffer

6. Fabric Buffer sends to egress queue

7. Egress port sends frame on wire

8. Egress buffer indicates freed buffer resources


Unicast scheduler overview

Each VOQ send a request to egress Scheduler associated with the corresponding portEgress Scheduler choose an ingress and make a proposal

A priority is selected (Fixed priority, or DWRR)An ingress is selected within that priority

Ingress Scheduler choose an egress and send acceptSelection is based on fixed Round Robin

Egress Scheduler send a grant to the VOQ


Unicast scheduler overview (continue)








Multicast


Multicast, fabric replication

CC

Ingress Fabric Egress

BMMcast

AUcast

BMcast

C

Mcast

AU-VOQU-VOQ

BBU-VOQU-VOQ

U-VOQU-VOQ

Use cases• Ethernet multicast

M-VOQM-VOQAA

Mcast

A

Mcast

A


SPAN, ingress replication

M-VOQM-VOQ

Ingress Port configured for SPAN

Fabric Egress

Ucast

A

BMUnicast

AUcast

BMcast

C

Ucast

AAA

U-VOQU-VOQ

Use cases•Ingress SPAN sessions•Egress SPAN sessions

AA

VOQ for SPAN packets


Multicast Class Queuing


Multicast Scheduling Algorithm


Altos

Gatos

Port 4

Latency3.2 µsec port-to-port

First-In-First-OutFull featured forwarding

6.7 µsec kernel to kernelStateless offloads

no DDP1.4 µsec host send2.1 µsec host receiveIncreases with OS, interrupt, and transport overheads

Socket layer app-to-appLinux 2.6Raw – 10.1 µsecUDP – 11.2 µsecTCP – 11.8 µsec

Port 1

Forwarding

?

1/10G MAC

Transceiver

Virtual Queues

Egress Queues

Packet Buffer

Fabric Buffer

Virtual Queues

Egress Queues

Packet Buffer





Parsing &Editing

1/10G MAC

Parsing &Editing

Transceiver


Lossless Data PathLossless Data Path


QoS Flow (animated)

EntersNuova Switch

DropPolicy

Queue 1

Queue 2

Queue 3

Queue 8

X-Bar

QoS Actions at Ingress Unified Port Controller

UnifiedCrossbar

Fabric

QoS Actions atEgress Unified Port

Controller

• No Drop• Drop

RXRXSystemClassmap

Ingresspolicer

Cos classification

Input portclassification

Egress Queuing (with Priority and Bandwidth)&

BCNTXTX Buffering Marking

VOQ&

Buffering


Class Based Data Path

Different classes of traffic require different treatment, e.g.FC class of traffic requires lossless or no drop treatmentMarket Data Ethernet class traffic may also require no dropRemaining Ethernet Data may only require best effort

Nexus 5000 data path resource and features are all per class based; for example,

Per class VOQs and egress queues, buffers, MTU, drop behavior

Per Class behavior should be consistently configured system wideand network wide


Nexus 5000 supports Modular QoS CLI (MQC) for all QoS configuration

System is a new target introduced at the global cfg levelSystem classes are instantiated within a system policy System policy is a service-policy attached to the ‘system’ target

At FCS, parameters configurable under system class:MTUDrop, no Drop

At ingress, packets are classified into a system classAt FCS, classification can be based on .1p or interface

Once classified, this class assignment travels with the packet through the entire system to select per class treatment at every step

At Egress, 802.1p rewrite is supported. 802.1p value can then be consistently used throughout the network to select the same system class treatment

System Class


switch(config)# systemswitch(config-system)# service-policy dc-policy

switch(config)# policy-map dc-policyclass eth-lo

mtu 9000class eth-hi

pause no-drop mtu 1500

class dc-controlmtu 4000

switch(config)#class-map eth-lomatch cos 2

class-map eth-himatch cos 4

class-map dc-controlmatch cos 5

System Policy Example


System Class Defaults

System classes defaults allow for plug N play IOC datapathBy default, there are 4 classes:

FC-default no drop, FC and FCoE traffic automatically belong to this class

Sup-hi – high priority control plane traffic such as BPDUs, FSPFSup-low – low priority control plane traffic such as CDPEth-default – all other traffic types classified into this class

Drop (best effort class)

User cannot remove Sup-HI, Sup-Low and FC-defaultUser can modify parameters in FC-default class but not Sup classes

User can define up to 5 new system classes including the Ethernet class-default


PFC and BB_Credits

IEEE 802.3x Pause provides no drop flow control similar to BB credits for FC

Priority Flow Control is a finer grained mechanism of flow control over standard pause or link level BB credits

Priority Flow Control uses .1p CoS value mapping to a system class to send appropriate pause to previous hop


Priority Flow ControlPriority based Flow ControlPriority based Flow Control

• Enables lossless behavior for each class of service

• PAUSE sent per priority when buffers limit exceeded


Priority based bandwidth managementPriority based

Bandwidth Management Priority based

Bandwidth Management

• Enables Intelligent sharing of bandwidth between traffic classes control of bandwidth

• 802.1Qaz Enhanced Transmission


Forwarding


Port 4Port 1


Data Path

Unified Crossbar Fabric

Forwarding

?

1/10G MAC

Transceiver

Virtual Queues

Egress Queues

Packet Buffer

Fabric Buffer

Virtual Queues

Egress Queues

Packet Buffer





Parsing &Editing

1/10G MAC

Parsing &Editing

Transceiver


Forwarding Pipeline

Wire rate “fixed” latency

Parsed frame fields, configuration, and control plane state are evaluated to determine destination(s)

Policy engine filters based on configuration, bindings, and layered ACLs

Layered equal cost multi path expansion

Fibre ChannelEtherChannel/ SAN PortChannel


Destination address

Source address

Ethertype = IP

FCS

Ethertype = .1Q VLANCoS d

TOS Total lenVer IHLIdentification Flg Frgm offset

TTL Proto Header cksumSource address

Destination address

IP options

Src port Dst portSeq numberAck number

Hdr len Flags Win sizeCksum Urgent ptr

TCP options and data

checksum check

FCS check

Parsing ethernet IP packets


Destination address

Source address

FCS


Ethertype = FCoE Ver

ReservedSOF

EOF Reserved

r_ctl d_id

seq_id df_ctl seq_cntox_id rx_id

Payload

Parameters

cs_ctl s_idtype f_ctl

CRC

CRC check FCS check

Fibre Channel frames are FCoE encapsulated prior to

forwarding

Parsing FCoE packets


Acquiring Interface StatePhysical Interface Table

Physical interface related configuration and state

Virtual Interface TableVirtual interface related configuration and state

VLAN State TableVLAN related configuration and state


Ethernet Forwarding16K Entry dLeft hash table (StationTable)

Searched by {VLAN, destination address}

SelectsLocal port/PortChannelMulticast index

Unknown addresses forwarded by VLAN multicast vectors

Unknown unicastUnregistered multicastBroadcast

IP Multicast forwarded by MAC addressIP multicast groups registered by IGMP snooping

Same mechanism forwards Fibre Channel in the local domain and N_port Virtualizer


Fibre Channel Forwarding

4K Entry dual index search tableSearched by {VSAN, domain_id}Misses are Fibre Channel exceptions

SelectsLocal port or PortChannel


Ethernet Address LearningIngress and Egress learning searches

Line rate on for all framesFacilitates distributed table population

Ingress notifies Supervisor to develop

database

Supervisor pushes new addresses to

all Unified Port ControllersAdds entries if missedRe-enforces existing entries

Supervisor queries tables to check for

consistencyMaintains aging state

CPU removes entries that are obsolete


Policy Enforcement Frames evaluated by multi-stage engineSearches occur in parallel

Results evaluated in pipelineDiagnostics and control plane “tap” pipeline at any point

Multipath Expansion

VLAN Membershipcheck

Interface, VLAN, and MAC Binding

MAC and L3 Binding(IP & Fibre Channel)

Fibre Channel Zone membership check

Port ACLs

VLAN ACLs (ingress)

QoS ACLs (ingress)

Role Based ACLs (egress)

Control Plane R

edirect/Snooping

Switch Port Analyzer (SPAN

) and Diagnostic Sam

pling

failpass

pass fail

pass

pass

permit

permit

permit

permit

fail

fail

deny

deny

deny

policer drop

to Supto SPANsession


ACL Search Engine

2048 Ternary match ACEsEach entry available to all functions

Labels allow sharing of Access Control Entries (ACEs)ACLs have a labelPolicy definition points select a label

Interfaces, VLANs, RolesLabels and frame fields form search keys

Flexible region assignmentTune ACL resource allocation to network policies

ACLs scopeVLAN and Control plane are Global scope – same on all Unified Port ControllersPort, QoS, RB, and SPAN are local scope – specific to each Unified Port Controller

TCAM 2Kx432

Priority

Search KeyPort ACLs

(576)QoS ACLs

(64 ingress) Role Based ACLs

(egress)

Vlan ACLs(1024)

SPAN and Diagnostic ACLs (64)

Control Plane ACLs (128)

Priority

Priority


Multipath Expansion

Two stage expansion processEach can lead to the nextSame mechanism for all expansionsConfiguration of expansion unique to each expansion

Fibre Channel SwitchingSelects a path to a target Fibre Channel switch

Fibre Channel Shortest Path First (FSPF)

Etherchannel/PortChannelSelects a path to a physically adjacent device

Fibre Channelswitching?

FC Multipath Expansion

EtherChannel/ PortChannel Expansion

List of Virtual Output Queues

Policy Enforcement


Expansion Algorithm

Relevant frame fieldsEthernet SA and DA always availableIP frames allows inclusion of IP v4/v6 SA and DA

TCP/UDP frames can include source and destination portsFibre Channel frames can include D_ID and S_ID

OX_ID can also be included per VSAN

Each field is divided by one of two CRC-8 polynomialsResult of field CRC division is combined via bitwise XORResult selected using modulo division by number of equal cost paths

256 possibilities are reduced to avoid biasWorst case imbalance is 6%

Ethernet DA

Ethernet SA

IP DA or FC D_ID

IP SA or FC S_ID

TCP DP

TCP SP or FC OX_ID

CRC-8 A

Field selectPolynomial select

XOR Modulo

Number of equal paths

256 possibilities Selected

Path

CRC-8 B


Editing FC packets for VOQ

FCS


Destination address

Source addressr_ctl d_id


Payload

Parameters


CRCEthertype = FCoE Ver

ReservedSOF

EOF Reserved

r_ctl d_id


Payload

Parameters


CRC


NX-OS Overview


Nexus 5000 Software Architecture Overview Consistent user experience across NX 7000, NX 5000 and MDS

Nuova Extensions to Support converged

interfaces

Layer-2 Protocols I/O ConsolidationFeatures & Protocols

(Nuova Addition)

Storage Protocols(SANOS3.1(3)+ NPV)

Interface Management

Chassis Management

Kernel

Lice

nsin

g IN

fras

truc

ture

AA

A, C

AllH

ome,

Sys

log,

OB

FL

Chip/Driver Infrastructure

VSANmgr

FSPF

F_portSvr

Zoning

NPV

WWNMgnt

DCBXQoS Mgt

Priority Flow ControlSPAN

CDP

802.1X

FC flow

CTS

……

Protocol Stack (IPv4, IPv6 , L2, fc2)

SNM

P, X

ML,

CLI

man

agem

ent

Softw

are

Upg

rade

Infr

astr

uctu

re

Hig

h A

vaila

bilit

y In

fras

truc

ture

VLANmgr

STP

IGMPsnp

LACP

Nuova HW architectureSpecific Components

DCOS4.0 code base


Ongoing Syncing and Committing


NX-OS Features and Benefits

Software compatibility Common software throughout the data center Modular software design Quick development of enhancements and problem fixes Troubleshooting and diagnostics

Smart Call Home Cisco GOLD

Programmatic XML interface Simple Network Management Protocol (SNMP)Role-based access control (RBAC)


Nexus 5000 NX-OS Image Layout and Upgrade

Field Upgradeable BIOS and Bootloader stored together in one flash

Field Upgradeable Kickstart and System images stored in a separate flash

BIOS and Bootloader image packaged in System Image

Image RecoveryIf system image is bad, kickstart can retrieve image from networkIf kickstart+system image flash is bad, bootloader can netboot image

NX-OS Install and Version Check Infrastructure

Future ISSU leveraging NX-OS stateful process restart and upgrade infrastructure


Nexus 5020 Licenses

NX-OS Licensing InfrastructureAll support time based license and grace periodsLicenses are enabled through the same workflow as Nexus 7000 and

MDS

Base software includes Ethernet featuresIncluded with every 5020, no license enablement required

Two feature licenses are available for 5020 N5020-SSK9 Nexus 5020 Storage Protocol

ServicesN5000FMS1K9 Nexus 5000 Fabric Manager Server

N5020-SSK9 includes protocol support for FC and FCoERequired for any fibre channel expansion module.


Nexus 5000 Feature Overview


I/O Consolidation Ethernet and Fibre Channel


IO Consolidation

virtual-ethernet interface (veth)Paired with host’s Ethernet deviceConfiguration point for all Ethernet features

virtual-fc interface (vfc)Paired with host’s HBA deviceConfiguration point for all Fibre Channel features

virtual-interface-group (vig)Logical representation of a switch port

Consists of one veth and one vfcConfigured online or offlineBound to physical switch port for deployment

Etherchannel post FCS

SCSI IP

eth0

SAN A SAN B

host0

vig

vethvfc

Ethernet ForwardingFC Forwarding

mux

mux

Ethernet

LAN

Connecting LAN and SAN on a single physical link

fc fc eth eth


IO consolidation: interface configurationCreate virtual-interface-group and bind to physical interface

switch(config)# interface vig 20

Bind virtual-interface-group to physical interfaceswitch(config)# interface vig 20switch(config-if)# bind Ethernet 1/1

Configure virtual-ethernet and virtual-fcswitch(config-if)# interface veth 20/1switch(config-if)# interface vfc 20/1

vfc30/1veth30/1vfc20/1veth20/1

vig20

Eth1/1

vig30

Eth1/33


Fibre Channel N_Port Virtualizer

Provides physical port level virtualization of multiple FC end nodes to one F_Port off a FC Switch

Nuova Switch operates in N_Port Proxy Mode (not in FC Switch mode)Simplifies multi-vendor interoperation

Eliminates the FC domain on Nuova switch

Simplifies management

Used in conjunction with NPIV

SAN

FC N_Port Virtualizer

A

Nuova Switch

B C

Fibre Channel Attachments

CA&B


N_Port Virtualizer terminology

Border Interface

N_ports

Server interface

F_Port

NP_Port


N_Port Virtualizer

Nuova Server Switch

F_Port F_Port F_Port

NP_Port NP_Port

N_Port Virtualizer in detail

NPIV Proxy captures all login associated packets from the HBA and Border Interfaces (BI)

Hosts pinned to Border Interfaces

Support NPIV over Server Interfaces (SI)

Relies on NPIV on Border interfaces

Retry failed login requests from one Border Interface on a different interface

Handle Events by generating proxy LOGOs

SISI SI

N_Port Virtualizer

Mode

BI BI

MDS #1 MDS #2

HBA #1 (NPIV)

HBA #2(NPIV)

HBA #3


LAN

Nuova Ethernet Host Virtualizer

Eliminates Need for Spanning Tree Protocol on Uplink Bridge Ports

Reduces CPU load on upstream switches

Allows Multiple Active Uplinks from Nuova Switch to Network

Doubles effective bandwidth vs STP

Prevents Loops by Pinning a MAC Address to Only One Port

Completely Transparent to Next Hop Switch

Ethernet Host Virtualizer

Nuova Switch

Active-Active

MAC B

MAC A

MAC B

MAC A


Pinning

Border interface

Server interface


Outgoing traffic: known unicast

Border

SIF

Traffic sourced by a station connected to a SIF goes out of its pinned border interface


Outgoing traffic: Multicast/broadcast

Border

SIF

Local replication to all SIFs is done by the End Host Virtualizer switch

One copy of the packet is sent out of the source SIF’s pinned border interface


Incoming traffic: Reverse Path Forwarding

Border

SIF

Packets destined to a station behind a SIF are accepted only by the SIF pinned border interface

A station can’t send anything out until her MAC has fully propagated to all Gatos (otherwise RPF can’t be honored)


Incoming traffic: Multicast/broadcast portal

Border

SIF

One border interface is elected to receive broadcast, multicast and unknown unicast traffic for all the SIFs


Incoming traffic: Deja-vu check

Border

SIF

If the source MAC belongs to a local station

The multicast/broadcast portal drops the packetThe pinned port accepts the packet, but no replication is done

This is regardless of the destination MAC (known/unknown unicast, multicast or broadcast)


Configuration Exchange and Validation


DCB Capability Exchange Protocol

Link level capability and configuration exchangeSimilar to FLOGI and PLOGI in Fibre ChannelAllows either full configuration or configuration checking

Based on LLDP (Link Level Discovery Protocol)Added reliable transportLink partners can choose supported features and willingness to accept configuration from peer

Feature TLVsPriority Groups (Link Scheduling)Priority-based Flow ControlCongestion Management (Backwards Congestion Notification)Application (frame priority usage)Logical Link Down


Host Redundancy

NIC Bonding (active-standby)Server uplink redundancyMultiple NICs connect to different switchesOne IP interface to host

One MAC address

SCSI Multi-Pathing Presents logical devices to the host applicationsLoad balances I/O requests across redundant linksactive/active or active/standby

Based on target capabilities

OS

SCSI multipathing

host0

sd0

file system

host1

sd1IP

bond0

SAN A SAN B

FCoE

Active for SAN

Active for LAN and SAN

mux

FCoE

mux


SPANSupport local SPAN and ERSPAN(After FCS)2 SPAN sessions per boxSPAN source can be a port, VLAN, or EtherChannelExpand the SPAN to support traffic monitoring between FC and Ethernetsource vlan/vsan list to filter spanned traffic for port and port-channel source

Souce SPAN Destination SPAN SPAN type

Ethernet Ethernet Local SPAN, ERSPAN

Fibre Channel Fibre Channel Local SPAN

Fibre Channel Ethernet Local SPAN ERSPAN(FC FCoE)

Virtual Ethernet Ethernet Local SPAN, ERSPAN

Virtual Fibre Channel Fibre Channel Local SPAN(FCoE FC)

Virtual Fibre Channel Ethernet Local SPAN, ERSPAN


Network Design


DISTRIBUTION

8

POD 1

Server Cabinet Pair 1 Server Cabinet Pair N

POD N

Server Cabinet Pair 1 Server Cabinet Pair N

MDS 9500

ACCESSNexus 5020

8

10GE/FCoE CNA

SAN-BLAN Core

SAN-A

STP BLK

EthernetFibre ChannelFCoE

I/O consolidation with TOR Design


POD 1 POD N

8

Server Cabinet Pair 1 Server Cabinet Pair N Server Cabinet Pair 1 Server Cabinet Pair N

ACCESSNexus 5020

8

10GE/FCoE CNA

SAN-BLAN Core

SAN-A

End Host ModeActive/Active


DISTRIBUTIONMDS 9500

I/O consolidation with TOR Design End Host Mode


POD 1 POD N

DISTRIBUTION

8

Server Cabinet Pair 1 Server Cabinet Pair N Server Cabinet Pair 1 Server Cabinet Pair N

MDS 9500

ACCESSNexus 5020

8

10GE/FCoE CNA

SAN-BLAN Core

SAN-A

VSS Supportat Aggregation


I/O consolidation with TOR Design VSS Support at Aggregation Layer


4 x 4G FC

10GE

ACCESSNexus 5020

Blade Switch = 10GE x 1GE

4

Row 1

Server Rack 1 Server Rack N

Row N

Server Rack 1 Server Rack N

MDS9134

2 x 10GE

MDS9134

SAN-B

LAN Core

SAN-A

DISTRIBUTIONMDS 9500

Nexus 5020 in Blade Center Environment 10G Aggregation

Documents

Nexus 5000 Deep Dive - Cisco · Nexus 5020. 56-Port L2 Switch • 40 Ports 10GE/FCoE, fixed • 2 Expansion Modules. Cisco Fabric Manager and Cisco Data Center Network Manager. Cisco