Cisco Nexus 6000 and 5600 with Fabric Extender 2000 Switchd2zmdbbm9feqrf.cloudfront.net/2014/usa/pdf/BRKARC-… · · 2014-05-21Cisco Nexus 6000 and 5600 with Fabric Extender 2000

Cisco Nexus 6000 and 5600 with Fabric Extender 2000 Switch Architecture

BRKARC-3452

Nicolas Delecroix

Technical Marketing Engineer

Data Center Group – Nexus Business Unit

[email protected]

mailto:[email protected]

© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public

Nexus 5000 Evolution

Customers

Nexus 5010 28-Port, 1RU




Nexus 5596T 10Gbase-T, 2RU

3


Customers






Nexus 6004

96 port 40G, 4RU

Introducing Cisco Nexus 5600/6000

4


Customers

Nexus 5672

48 port 10G, 6 port 40G, 1RU

Nexus 6004

96 port 40G, 4RU

Nexus 56128

96 port 10G, 8 port 40G, 2RU






5

Introducing Cisco Nexus 5600/6000

Cisco Nexus 6000 and 5600 with Fabric Extender 2000 Switch Architecture

BRKARC-3452

Nicolas Delecroix

Technical Marketing Engineer

Data Center Group – Nexus Business Unit

[email protected]

mailto:[email protected]


Nexus 5600/6000 Architecture - Agenda

7

Overview

Fabric Extender

Performance

Internal Architecture

Transceivers/Cables

Access Control List

Multicast

Quality of Service

Analytics

Applications


6 fixed 40G QSFP interfaces

Nexus 5672 Chassis

• All ports provide Ethernet and FCoE

• 16 Unified Ports (orange) also provide 2/4/8G FC

• L2 and L3 VXLAN

Port-Side View

8

48 fixed 1/10G SFP+ interfaces


Cisco Nexus 5672 Chassis Power Supply and Fans

9

Power Supply 1+1

Redundancy

Console Mgmt0 USB Fan Module 2+1

Redundancy


Nexus 56128 Chassis

• 48 Fixed 10G SFP+ Ports, 4 fixed 40G QSFP+ ports

• 2 Expansion Slots

• Expansion module provides 24 Unified ports 10G Ethernet/FCoE or 2/4/8G FC, and 2 ports 40G Ethernet/FCoE

Port-Side View

10


Cisco Nexus 56128 Chassis Power Supply and Fans

11

Power Supply 2+2

Redundancy

Console Mgmt0 USB Fan Module 3+1

Redundancy


Cisco Nexus 6004 Chassis Port-Side View

12

4RU

12 QSFP+ ports

Expansion Module

N6K-6004-M12Q


Cisco Nexus 6004 Chassis

• Six power supply slots; a minimum of three is required. Supports both 3 + 1 and 3 + 3 redundancy.

• Each power supply is rated 1100W, 90—240 VAC.

• The chassis has four fan trays. A minimum of three is required.

Power Supply and Fans

13

Power Supply 3 + 3

Grid Redundancy

or 3 + 1 Redundancy

Console Mgmt0 USB

Fan Module 3+1

Redundancy


Unified Port Line-Card Expansion Module - N6004X-M20UP

For Nexus 6004

14

• 20 * 10G SFP+ unified ports expansion module

• 2/4/8G FC port

• 1G/10G and FCoE

• Max number of native FC ports per chassis is 48 (as of NXOS 7.0(2)N1(1) release)

• SFP+ port allows support for a larger variety of optical transceivers

• Same transceivers supported as on 5600 10G ports – including 10G long distance, 1G copper and optical


Cisco Nexus 5600/6000 Airflow

• The units have a “port side” and a “fan side”. No front or back

• The units support both port side exhaust and port side intake – Port side exhaust (used for in server rack airflow alignment)

– Port side intake (used for network rack airflow alignment)

• Different power supply and fan modules are required for different airflow directions

15

© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public 16

Feature Nexus 5600/6000 Nexus 5500

Native 40GE Support Yes 4xQSFP module – 10G flows

1GE Support Yes Yes

Unified Ports Yes Yes

FEX Scaling 48 (6004 L2), 24 (6004 L3, 5600) 24 with L2, 16 with L3

MAC Table 256K (shared with ARP/ND) 32K

IPv4 Unicast Routes 32K 16K

IPv4 ARP/Host Route Entries 128K 16K (shared with IPv6 ND and

mroute)

IPv4 Multicast Routes 32K 8K (shared with ARP)

N5600/6000 and N5500 Comparison

Latency ~1us 1.8us

L3 Throughput Line rate 160Gbps

L2 Throughput Line rate Line rate

© 2014 Cisco and/or its affiliates. All rights reserved. BRKARC-3452 Cisco Public 17

Feature Nexus 5600/6000 Nexus 5500

VLAN 4K 4K

VRF 4K 1K

Segment ID Yes No

ACL 4K 4K

L4ops for ACL 24 total (16 for UDP/TCP, 8 for TCP flag) 8 for UDP and 8 for TCP

Packet Buffer 25MB per 3xQSFP (or 12xSFP) ports

2.1MB per 10G port

640KB per 10G port

vPC, FabricPath Yes Yes

N5600/6000 and N5500 Comparison (cont.)

IPv6 Multicast Routes 32K 8K (shared with ARP, ND. Limit is

with /64 routes)

IPv6 ND Entry/Host Route Entries 85K (shared with MAC and ARP) 8K (shared with mroute and ARP)

IPv6 Unicast Routes 8K (shared with IPv4) 8K (shared with IPv4)

Dynamic Fabric Automation Yes No

VXLAN Yes* No

*Software roadmap feature



18

Overview

Fabric Extender

Performance


Transceivers/Cables

Access Control List

Multicast

Quality of Service

Analytics

Applications


Nexus 2000 100M/1G Fabric Extender Portfolio

19

Nexus 2224TP Nexus 2248TP Nexus 2248TP-E

Uplink Ports 2 x 10G SFP+ 4 x 10G SFP+ 4 x 10G SFP+

Host Facing Ports 24 x 100/1000BASE-T RJ45

48 x 100/1000BASE-T RJ45

48 x 100/1000BASE-T RJ45

Oversubscription 1 : 1.2 1 : 1.2 1 : 1.2

FCoE N/A N/A N/A

Buffer 3.8MB 7.6MB 32MB

Shared Buffer No No Yes


Nexus 2000 1G/10G Fabric Extender Portfolio

20

Nexus 2232PP Nexus 2232TM Nexus 2232TM-E Nexus 2248PQ

Uplink Ports 8 x 10G SFP+ 8 x 10G SFP+ 8 x 10G SFP+ 4 x 40G QSFP+

Host Facing Ports 32 x SFP/SFP+ (1/10G)

32 x 1/10GBASE-T RJ45

32 x 1/10GBASE-T RJ45

48 x SFP/SFP+ (1/10G)

Oversubscription 1 : 4 1 : 4 1 : 4 1 : 3

FCoE Yes No Yes (30m / Cat6a/7) Yes

Buffer 10.2MB 10.2MB 10.2MB 32MB

Shared Buffer No No No Yes


Nexus 2000 Fabric Extender Scalability

• Nexus 5600 and 6000 support all types of FEX, including HP, Fujitsu, Dell, IBM FEX for blade servers. Adapter-FEX and VM-FEX are also supported.

21

+ 48

xF

EX

Cisco Nexus 5600/6000 Cisco Nexus 2000 FEX

Nexus

5000

Nexus 5500 Nexus 5600 Nexus

6004

Nexus

7000

Nexus 7700

L2 12 24 24 48 48 64

L3 x 16 24 24 48 64



22

Overview

Fabric Extender

Performance


Transceivers/Cables

Access Control List

Multicast

Quality of Service

Analytics

Applications


Nexus 5672 - 72 x 10G Unicast Throughput

64 80 96 128 256 512 1000 1024 1280 1518 2240 4096 9100

RFC 2544 Unicast L2 Port Pair 100 100 100 100 100 100 100 100 100 100 100 100 100


0

10

20

30

40

50

60

70

80

90

100

Throughput (% of line rate)

Packet Size (Bytes)

23


Nexus 5672 - 72 x 10G Multicast Throughput

64 80 96 128 256 512 1000 1024 1280 1518 2240 4096 9100

RFC 3918 Mcast L2 One to Many Accumulated 100 100 100 100 100 100 100 100 100 100 100 100 100


RFC 3918 Mcast L2 One to Many Distributed 100 100 100 100 100 100 100 100 100 100 100 100 100


0

10

20

30

40

50

60

70

80

90

100


Packet Size (Bytes)

24


Nexus 5672 – 72 x 10G Unicast 100% Load Latency

25

64 128 256 512 1024 1280 1518 2240 4096 9216

RFC 2544 unicast L2 port pair 1001 1021 1091 1111 1105 1104 1104 1103 1101 1098

RFC 2544 unicast L3 port pair 937 963 1029 1048 1047 1046 1045 1044 1043 1040

0

200

400

600

800

1000

1200

1400

1600

Average Latency (Nano Seconds)

Packet Size (Bytes)


Nexus 6004 Performance – Fully Loaded Switch

26


Nexus 6004 Performance – Spirent Traffic Generator

27


Nexus 6004 - 96 x 40G Unicast Throughput

64 80 96 128 256 512 1000 1024 1280 1518 2240 4096 9100



0

10

20

30

40

50

60

70

80

90

100


Packet Size (Bytes)

Spirent third party performance report for Nexus 6004

28

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps12806/white_paper_c11-728442.pdf


Nexus 6004 - 96 x 40G Multicast Throughput

64 80 96 128 256 512 1000 1024 1280 1518 2240 4096 9100





0

10

20

30

40

50

60

70

80

90

100


Packet Size (Bytes)

Spirent third party performance report for Nexus 6004

29

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps12806/white_paper_c11-728442.pdf


Nexus 6004 - 384 x 10G Unicast 100% Load Latency

64 80 96 128 256 512 1000 1024 1280 1518 2240 4096 9100

RFC 2544 Unicast L2Port Pair

1.028 0.99 1.005 1.023 1.121 1.118 1.111 1.116 1.11 1.107 1.11 1.108 1.101

RFC 2544 Unicast L3Port Pair

1.032 1.001 1.011 1.03 1.122 1.121 1.116 1.118 1.113 1.111 1.112 1.11 1.106

0

0.2

0.4

0.6

0.8

1

1.2

1.4

Average Latency (microseconds)

Packet Size (Bytes)

30


Nexus 6004 - 384 x 10G Multicast 100% Load Latency

64 80 96 128 256 512 1000 1024 1280 1518 2240 4096 9100

RFC 3918 Mcast L2 One-to-ManyAccumulated

1.004 0.989 1.029 1.017 1.095 1.086 1.079 1.084 1.083 1.082 1.082 1.08 1.077

RFC 3918 Mcast L3 One-to-ManyAccumulated

1.067 1.051 1.061 1.064 1.087 1.083 1.079 1.081 1.08 1.079 1.078 1.076 1.073

RFC 3918 Mcast L2 One-to-ManyDistributed

0.951 0.949 0.954 0.962 1.019 1.042 1.043 1.042 1.042 1.042 1.043 1.043 1.043

RFC 3918 Mcast L2 One-to-ManyDistributed

0.96 0.958 0.963 0.974 1.019 1.044 1.044 1.044 1.044 1.044 1.044 1.044 1.044

0

0.2

0.4

0.6

0.8

1

1.2

1.4

Average Latency (microseconds)

31


Nexus 6004 - 384 x 10G Unicast and Multicast 100% Load Jitter

64 80 96 128 256 512 1000 1024 1280 1518 2240 4096 9100

Unicast L2 Jitter (ns) 8 5 8 5 6 8 6 7 6 10 10 9 9

Unicast L3 Jitter (ns) 8 5 9 5 6 8 6 7 6 10 10 9 10

Multicast L2 Jitter (ns) 9 6 9 5 5 8 6 7 6 10 9 9 9

Multicast L3 Jitter (ns) 9 5 9 5 5 8 6 7 6 9 9 9 9

0

2

4

6

8

10

12

Average Jitter (nanoseconds)

Packet Size (Bytes)

32



33

Overview

Fabric Extender

Performance


Transceivers/Cables

Access Control List

Multicast

Quality of Service

Analytics

Applications


UPC 1

48xSFP+

Unified Fabric Controller (UFC)

Switch Fabric

6xQSFP+ CPU

Supervisor

UPC 0

Cisco Nexus 5672 Internal Architecture

34

UPC 2 UPC 3 UPC 4 UPC 5 UPC 6

FC PHY


UPC 1

48xSFP+

Unified Fabric Controller (UFC)

2x

QSFP+

CPU

Supervisor

UPC 2 UPC 3 UPC 4 UPC 5 UPC 6

2x

QSFP+

UPC 7 UPC 8 UPC 9 UPC 10 UPC 11 UPC 12

24xSFP+ 2x

QSFP+ 24xSFP+

2x

QSFP+

Slot 1 Slot 2


35

FC PHY FC PHY FC PHY FC PHY


UPC 1

UFC 1 UFC 2 UFC 3 UFC 4

Switch Fabric

CPU

Supervisor

UPC 0


36

UPC 2 UPC 3 UPC 4

12xQSFP+ 20xSFP+

FC PHY

UPC 1 UPC 2

12x40G LEM (M12Q) 20x10G LEM (M20UP)


Unified Port Controller - UPC ASIC • Multimode MAC; built-in PHY for 1, 10, and 40GE

• Packet parsing and rewriting

• Lookup engine and access control: L2, L3, FabricPath, ACL, FCoE, and policing

• Buffering and queuing: Buffer management, PFC for lossless traffic, queuing (Strict Priority Queuing and DWRR), and packet replication (SPAN and multicast)

37

UPC

UPC

224Gbps

224Gbps

448Gbps

448Gbps

Twelve 10 GE

Three 40 GE

Front Panel Ports Fabric Connections

• Extra fabric bandwidth for SPAN and multi-destination traffic


Cisco Nexus 5600/6000 Packet Processing Flow Ingress Pipeline

38

Un

ified

Cro

ss

ba

r

Fa

bric

MAC

Unicast VOQ

(8 per egress port)

8000 Multicast VOQs

Ingress UPC

1, 10, and 40 GE

Buffer Allocation

Buffer Accounting

PAUSE Signal

Buffer

Management

Buffer

Packet

Rewrite

Ingress

SPAN

Replication

Parser

Ingress ACL

Ingress Policing

Forwarding

Lookup

VLAN Table

MAC Table

ARP/ND Table

IPv4/IPv6 Routing Table

FabricPath

FC routing Table

Security ACL

QoS ACL

SPAN Filter

Policing/Stats

Storm Control


Nexus 5600/6000 Packet Processing Flow Egress Pipeline

39

Un

ifie

d C

ros

sb

ar

Fa

bri

c

MAC

Egress UPC

Unicast buffer

Multicast

congestion

management

Unicast Queue

Buffer

Editing

TX SPAN

Replication

To Ingress

Buffer Management

Multicast Queue

Multicast

Replication

Buffer

Management

*Egress policing is not supported with current software

Scheduling Egress ACL

Egress Policing*


Cisco Nexus 5600/6000 Key Forwarding Tables

40

Host table: 256,000 -entry hashing table;

actual capacity is slightly less than 256,000

Host table: Shared between MAC, ARP, and

ND and /32 host route

Host table default carving: 128,000 MAC,

128,000 IP hosts

LPM table: 32,000 entries. Also known as

summary routes

IP Host Region (ARP/ND/Host

route/(*,G))

MAC Region Summary Routes (S,G)

Host Table* LPM Table(32,000) Mroute Table(64,000)*

25

6,0

00

En

trie

s

* Hardware table size. Please check configuration limit for software scaling


Cisco Nexus 5600/6000 Host Table Carving

• In Layer 2, virtualized environments, it can be interesting to have more than 128k MAC addresses on the switch

• MAC/ARP Hardware Resource Carving allows to change the size of MAC and IP regions to allocate more space for the MAC region.

IP Host Region (ARP & ND & /32 Host route)

MAC Region

Host Table 2

56

,00

0 E

ntr

ies

41


Cisco Nexus 5600/6000 Carving Profiles

• Use of pre-defined carving profiles.

• HRT = Host Route Table = IP host region.

• STM = Station Table Management = MAC region.

Template Profile Description

hrt-128-stm-128 HRT size: 128k, STM size: 128k (default profile)

hrt-96-stm-160 HRT size: 96k, STM size: 160k



42


Cisco Nexus 5600/6000 MAC/ARP Resource Carving CLI

• Specify the resource template to use: • switch(config)# hardware profile route resource service-

template template-name

• Save the config and reload the switch

• Show commands: • show hardware profile route resource template

• show hardware profile route resource template default

• show running-config hardware profile route resource template

• show startup-config hardware profile route resource template

43



44

Overview

Fabric Extender

Performance


Transceivers/Cables

Access Control List

Multicast

Quality of Service

Analytics

Applications


QSFP 40G Bidirectional Transceiver

40G deployment with 10G cabling

45

• QSFP pluggable 40Gbit/s Bidirectional (BiDi) Transceiver that works with existing Multi-mode fiber infrastructure

– Using the existing 10Gbit/s Multi-mode infrastructure

– QSFP-40G-SR-BD has dual LC Connector

– 40Gbit/s BiDi transceiver has two 20Gbit/s channels each transmitted and received simultaneously on two wavelengths


QSFP-40G-SR-BD

46

40G-BD 40G-BD

40G-BD

10G-SR

10G-SR

10G-SR

10G-SR

Multimode

LC Duplex

Supports 100m with OM3 with MMF and 150m with OM4 (OM4+) MMF

LC duplex connectors

Wavelength: ~850nm and ~900nm

Pull-Tab Color: grey

Optical

DeMux

Multimode

LC Duplex


QSFP-40G-SR4

47

40G-SR4 40G-SR4

40G-SR4

10G-SR

10G-SR

10G-SR

10G-SR

MPO-12 MPO-12

MPO-12 LC Breakout

• Support 100M with OM3 MMF and 150M with OM4 MMF


QSFP-40G-CSR4

48

• Supports 300M with OM3 MMF and 400M with OM4 MMF

• Compatible with Cisco QSFP-40G-SR4

• Compatible with 10G SFP-SR

40G-CSR4 40G-SR4/CSR4

40G-CSR4

10G-SR

10G-SR

10G-SR

10G-SR

MPO-12 MPO-12

MPO-12 LC Breakout


Multimode Ribbon Fiber 40GE

49

For QSFP-40G-SR4 and

QSFP-40G-CSR4

12-Fiber MPO Connector

• MPO/MTP connector with 12

pins

• Use 4 fiber pairs

− 4 TX and 4 RX allows for 40GE

− 4 unused fibers in the center


QSFP-40GE-LR4

50

Host

Card

IEEE Standard 40GBASE-LR4

4x ROSA

4x TOSA

LC TIA

Multiplexer

LC

• QSFP-40GE-LR4 can only be connected to each other. 10KM with Single mode fiber.

Can’t connect to 10Gbase-LR


FET-40G

• Low-cost QSFP optical transceiver connecting FEX to Cisco Nexus 6004 • Supported on Cisco Nexus 5600/6004 and Nexus 2248PQ-10G • Interoperable with FET-10G • Support for 100m distance with OM3

51

Cisco Nexus 5600/6000 Cisco Nexus 5600/6000

Cisco Nexus 2232PP and 2232TM-E 2232TM

Cisco Nexus 2248TP-E 2248TP Cisco Nexus 2248PQ-10G

FET-40G

FET-40G

FET-10G

FET-40G



52

Overview

Fabric Extender

Performance


Transceivers/Cables

Access Control List

Multicast

Quality of Service

Analytics

Applications


ACL Types and Features

• Security ACL

− MAC, IPv4, and IPv6 ACLs

− PACL: ACL enabled under L2 interface

− VACL: ACL enabled for L2 VLAN traffic

− RACL: ACL enabled for routed traffic

− RBACL: Role-based ACL for CTS (roadmap feature)

• ACL for QoS classification and Policing

• Policy Based Routing (PBR)

– User-configured ACL to redirect traffic

• ACL for SPAN/ERSPAN

• ACL for control traffic

– To redirect control traffic to CPU, not user-configurable

– CoPP

53


ACL Scaling and TCAM Partition

• 4096 Access Control Entries per UPC are organized in blocks of 64 ACE entries.

• Each IPv6 ACL (without port range) requires two TCAM entries.

54

switch# sh platform afm info tcam 0 region ifacl

ifacl tcam TCAM configuration for asic id 0:

[ vacl tcam]: range 0 - 1023

[ifacl tcam]: range 1024 - 2175 *

[ qos tcam]: range 2176 - 2623

[rbacl tcam]: range 2624 - 3775

[ span tcam]: range 3776 - 3839

[ sup tcam]: range 3840 - 4095

TCAM [ifacl tcam]: [v:1, size:1152, start:1024

end:2175]

In use tcam entries: 24

1024-1039,2168-2175

Default TCAM Partition

VACL(1024)

IFACL(1152)

QoS(448)

RBACL(1024)

SPAN(64)

Control Traffic(256)

40

96

Entr

ies

TCAM Region

Usage


• TCAM Carving allows to change the size of each region

• Create a template:

switch(config)# hardware profile tcam resource template t1

switch(config-tcam-templ)#

• Set region sizes:

switch(config-tcam-templ)# ifacl ?

<320-3584> Enter size of ifacl region (in 64 entry

increments)

TCAM Carving

55


TCAM Carving

• Similarly, other region sizes can be changed: switch(config-tcam-templ)# ?

ifacl Configure size of ifacl region

no Change size to default value

qos Configure size of qos region

rbacl Configure size of rbacl region

span Configure size of span region

vacl Configure size of vacl region

• Display the configuration for a template along with the current usage: switch(config)# show hardware profile tcam resource template name

<my_template>

56


TCAM Carving

switch(config-tcam-templ)# show hardware profile tcam resource template name t1

-------------------------------------------------------------------

Template name: t1

Current state: Created

Region Size-allocated Current-size Current-usage Available/free

-------------------------------------------------------------------

Vacl 1152 1024 28 996

Ifacl 1024 1152 266 971

Rbacl 1152 1152 3 1149

Qos 448 448 30 418

Span 64 64 6 58

Sup 256 256 74 182

-------------------------------------------------------------------

57


TCAM Carving

• Commit a template: switch(config)# hardware profile tcam resource service-template t1

Details of the t1 template you are trying to commit are as follows:

-------------------------------------------------------------------------------

Template name: t1

Current state: Committed (Startup/Running)

Region Features Size-allocated Current-size Current-usage Available/free

-------------------------------------------------------------------------------

Vacl Vacl 1152 2048 11 2037

[…]

------------------------------------------------------------------------------

Template committed.

Please do the following for the template to be applied:

1> Save running config : "copy running-config startup-config"

2> Reboot the switch : "reload”

58



59

Overview

Fabric Extender

Performance


Transceivers/Cables

Access Control List

Multicast

Quality of Service

Analytics

Applications


Why Analytics? – A Data Center Today:

• 1G -> 10G -> 40G -> 100G

• Virtualized

• More and more applications

• Those applications are more and more complex (Hadoop…)

• More and more flows

• Latency is very important

• Ports speed mismatch

• Buffer discrepancy between the devices

• With spine/leaf, design is less complex, but there’s more speed and more applications

60


Switched Port Analyzer (SPAN)

61

Monitor Source

Sniffer Device

Monitor Destination

Host-A Host-B

Switch


Encapsulated Remote SPAN (ERSPAN)

62

Host-A

Host-B

Sniffer Device

Monitor Source Monitor Destination

IP Cloud

GRE Encapsulated

monitored traffic


Cisco Nexus 5600/6000 SPAN

63

*Software roadmap feature

SPAN Features Nexus 5600/6000 Nexus 5500

Total SPAN sessions 31 bi-directional (16 with current NXOS) 4 bi-directional

Local SPAN sessions 31 bi-directional (16 with current NXOS) 4 bi-directional

ERSPAN sessions 16 4

Prioritize data over SPAN Yes (through scheduling) Yes (SPAN policing)

Line-rate SPAN throughput Yes No

ERSPAN destination session Yes No

ERSPAN with 1588 PTP timestamp Yes* No

MTU Truncated SPAN and ERSPAN Yes Yes

ACL filter for SPAN and ERSPAN Yes Yes

SPAN-on-Drop Yes No

SPAN-on-Latency Yes No

SPAN with multiple destination ports Yes (each destination port uses one

SPAN session)

No


High-Performance SPAN

64

Unified

Crossbar

Fabric

Scheduler Prioritizes Production Data

Data

SPAN

Extra Fabric Bandwidth for SPAN

Separate Buffer Pool for SPAN


SPAN Buffer Monitoring

Station

Ingress

Data Buffer

SP

AN

-On

-Dro

p

Tail-

Dro

p

SPAN on Drop - NX-OS 7.0(1)N1(1), Q1/2014 SPAN-on-Drop

65


SPAN on Drop - NX-OS 7.0(1)N1(1), Q1/2014

• Copies the dropped packet to capture device

• Correlate the packet drop with application

• Works for unicast packets only

• Packet drops can be monitored on ingress only

• There is a dedicated buffer space for SPAN, so SPAN-on-Drop does not affect production traffic

• Supports both local SPAN and ERSPAN

• Configuration uses existing SPAN commands

• One SPAN-on-Drop session is supported

• Can have multiple source ports, and multiple destination ports

• Source cannot be FEX HIF port. But FEX fabric port is supported

• Source port(s) can be a part of a SPAN-on-Drop session, and a local SPAN session simultaneously

SPAN-on-Drop

66


SPAN on Drop - NX-OS 7.0(1)N1(1), Q1/2014

The source interface is the ingress port for which we want to monitor drops.

switch(config)# monitor session 1 type span-on-drop

switch(config-span-on-drop)# source interface e3/1

switch(config-span-on-drop)# destination interface e3/2

switch(config)# monitor session 2 type span-on-drop-erspan

switch(config-span-on-drop-erspan)# source interface e3/1

switch(config-span-on-drop-erspan)# destination ip 100.1.1.2

SPAN-on-Drop

67


Latency Monitoring

Latency Monitoring:

Min: 856ns

Max: 1208ns

Avg: 901ns

TimeStamp

Packet

T0

T1 – T0

T1 – T0

T0

Latency Histogram:

Latency Range: 800ns - 10usec

Stats:

Within the Range: x packets

Out of the Range: y packets

• FIFO Latency

• Per Port-Pair Data

68


Latency Monitoring - NX-OS 7.0(1)N1(1), Q1/2014

• Latency Monitoring provides {min, average, max} latency between a specified port pair and also maintains latency histogram (accuracy in few nanosecs)

• By default instantaneous Latency Monitoring is enabled between pair of ports

• Latency Histogram can be enabled for specific port-pair to provide histogram instead of instantaneous mode

• Measures switch latency for each packet, no sampling required

• Fully implemented in HW, no CPU impact, no traffic impact

Latency Monitoring

69


Instantaneous Mode

• Enabled by default on all pairs of ports. No configuration is required

switch# show hardware profile latency monitor interface e1/23 interface e1/22 -------------------------------------------------------------------------------- Latency Statistics in nano seconds -------------------------------------------------------------------------------- | Ingress Port | Egress Port | Minimum | Maximum | Average | -------------------------------------------------------------------------------- | Ethernet1/22| Ethernet1/23| 856| 1208| 901| --------------------------------------------------------------------------------

70


Custom Histogram

• Count the number of packets falling in a specific range of latency

• Example for port-pair e1/1 and e1/3:

switch# show hardware profile latency monitor histogram interface e1/3 interface e1/1 -------------------------------------------------------------------------------- | Egress Interface : Ethernet1/3 Ingress Interface : Ethernet1/1 | -------------------------------------------------------------------------------- | All Latency Values are in nano seconds | -------------------------------------------------------------------------------- | Range | 800 <= Latency < 10000 | Outside the first range | ------------------------------------------------------------------------------- | counter | 3542818572 | 16290988 |

switch(config)# interface e1/3

switch(config-if)# packet latency low-latency 800 high-latency 10000 mode custom

interface e1/1

71


SPAN-on-Latency

Latency Monitoring:

Min: 856ns

Max: 1208ns

Avg: 901ns

TimeStamp

Packet

T0

T1 – T0

T3 – T2

T2

Monitoring

Station

If Latency Threshold > 10 usec:

SPAN to 1/64

72


SPAN-on-Latency

• This feature replicates packets from the interface when latency exceeds a configured threshold

• Latency threshold is per-port

• This again helps to identify the victim of congestion

• Replicated traffic uses the SPAN buffer so it doesn't impact the production traffic

• One SPAN-on-Latency session is supported in hardware

• Uses similar SPAN CLI, with new session type “span-on-latency”

• ERSPAN-on-latency is also supported

73


SPAN-on-Latency Configuration

monitor session 1 type span-on-latency

source interface Ethernet1/3 tx

destination interface Ethernet1/4

• The SPAN-on-Latency source port is the egress port on which we monitor the latency

• The SPAN-on-Latency session makes a copy of all high-latency packets egressing on this port, coming from any ingress port

• Sample configuration to SPAN packets if latency goes beyond 10us when egressing on e1/3:

interface Ethernet1/3

hardware profile latency monitor threshold 10001

interface Ethernet1/4

switchport mode monitor

Always Tx: packets egressing on 1/3

(any source) with latency >10us will

be replicated to the SPAN dest 1/4

74


SPAN-on-Latency

• Source port can be an regular ethernet port, not a port-channel. Can be a port-channel member

• Source port cannot be FEX HIF port. But FEX fabric port is supported

• Multiple sources can be configured – latency threshold is per SPAN-on-drop TX source port

• A SPAN-on-Latency source port cannot be in another SPAN session

• Destination is only a single ethernet port, not port-channel

• Only one destination port can be configured

75



76

Overview

Fabric Extender

Performance


Transceivers/Cables

Access Control List

Multicast

Quality of Service

Analytics

Applications


Cisco Nexus 5600/6000 Multicast Features Summary

• IGMP snooping for IGMPv1, v2, and v3

• PIM-SM

• PIM-SSM without vPC and PIM-SSM with vPC+

• PIM-BiDir without vPC and PIM-BiDir with vPC+

• PIM Policy

• MSDP

• Anycast RP with PIM (RFC 4610)

• Anycast RP with MSDP

• 8000 IGMP snooping entries

• 8000 mroutes with vPC, 16000 mroutes without vPC supported by software

77


Efficient Multicast Replication

• Optimized multicast replication

• Fabric replication and egress replication; one copy is replicated to egress UPC, where there is a receiver—minimizing the traffic load on the switch fabric and eliminating the switch fabric congestion

• Line-rate multicast replication

78

Ingress UPC

Unified

Crossbar

Fabric

Egress

UPC

Egress

UPC


Multicast VOQ • 8000 multicast VOQs to eliminate HOLB and help ensure high throughput

• Tracks the fan-out of the egress UPC; packets with different egress UPC fanouts are assigned to different VOQs so that it can be scheduled and served independently

79

Unified

Crossbar

Fabric

Ingress UPC

Egress UPC 1 224.1.1.1

224.1.1.1

224.1.1.2

224.1.1.2

224.1.1.2

Egress UPC 2

Egress UPC 3

Un

ica

st

VO

Q

Mu

ltic

as

t V

OQ


Cisco Nexus 5600/6000 Key Forwarding Tables

80

Host table: 256,000 -entry hashing table;

actual capacity is slightly less than 256,000

Host table: Shared between MAC, ARP, and

ND and /32 host route

Host table default carving: 128,000 MAC,

128,000 IP hosts

LPM table: 32,000 entries. Also known as

summary routes

• Mroute table: 64,000 entries

IP Host Region (ARP/ND/Host

route/(*,G))

MAC Region Summary Routes (S,G)

Host Table* LPM Table(32,000) Mroute Table(64,000)*

25

6,0

00

En

trie

s

* Hardware table size. Please check configuration limit for software scaling


MET Table

• 64,000 entries in MET table

• Currently supported: 8,000 vPC, 16,000 non-vPC

• Today the MET table is the limiting factor for multicast

scale

Ingress UPC

Egress UPC 1

Un

ica

st

VO

Q

Mu

ltic

as

t V

OQ

Egress UPC 2

1

2

10

Switch Fabric

MET: 123

MET: 123

MET: 123

123 122

16384 16385

65536

1, 4, 5

123 122

16384 16385

65536

2, 10

4

5

81


Multicast Hashing over Port Channel • N5600/6000 implements flow based hashing for multi-destination traffic, and it supports multicast load

sharing over Port Channel with 5-tuple packet header.

• Traffic is replicated to all egress UPCs where Port Channel member resides

• Egress UPC runs hash calculation and one egress port is chosen to send out multicast packets. The UPC ASIC that is not supposed to send out packet will drop packet (Egress UPC 1 in the example)

82

Egress UPC 1

Egress UPC 2

Port 1

Port 2

Port 3

Port 4

Port 3

Selected

Port 3

Selected

Unified

Crossbar

Fabric

Ingress UPC

Un

ica

st

VO

Q

Mu

ltic

as

t V

OQ



83

Overview

Fabric Extender

Performance


Transceivers/Cables

Access Control List

Multicast

Quality of Service

Analytics

Applications


Cisco Nexus 5600/6000 QoS Features

• Eight classes of service;

– 2 reserved for control traffic, 6 for data traffic

• Traffic classification

– DSCP, CoS, and ACL

• Strict Priority Queuing and DWRR

– DCBX 802.1Qaz

• Packet marking

– DSCP, CoS, and ECN

• Ingress and egress policing

– 4096 policers per ASIC

• No drop system class

• Flexible buffer management 84


Cisco Nexus 5600/6000 QoS Processing Flow

85

Cro

ssb

ar

Fab

ric

MAC Traffic

Classification

Ingress

Cos/DSCP

Marking

MTU

Checking

VOQs for Unicast (8 per egress port)

Egress Queues PAUSE ON/OFF

Signal

Trust CoS/DSCP

L2/L3/L4 info with ACL

Truncate or Drop

Packets if MTU is Violated

If Buffer Usage Crosses Threshold:

• Tail drop for drop class

• Assert pause signal to MAC

for no-drop system class

Strict Priority +

DWRR Scheduling

Multicast Queues

MAC

Ingress UPC

Unicast

Multicast

Ingress

Policing

ECN

Marking Egress

Policing

Egress

Scheduling

Egress UPC

Per-class

Buffer Usage

Monitoring


Increased Packet Buffer

• 25MB packet buffer is shared by every three 40GE ports or twelve 10GE ports.

• Nexus 5672: 150MB of total buffer



86

Ingress UPC Egress UPC

Un

ica

st

VO

Q

Mu

ltic

as

t V

OQ

Unified

Crossbar

Fabric

16MB 9MB


Ingress Buffer - Flexible Buffer Management

• Shared buffer is good for burst absorption.

• Dedicated buffer is good for predictable performance for each port.

• N5600/6000 buffer management is flexible: dedicated plus shared.

• Long-distance FCoE, video editing (i.e., AVID), Big Data, and distributed storage

87

Port 1

Dedicated

Port 2

Dedicated

Port 3

Dedicated

Shared Packet Buffer

~14MB

SPAN Control

UPC Ingress Buffer (16MB)


Tune Shared vs Dedicated Buffer Allocation at Ingress • “queue-limit” under “network-qos” policy specifies the dedicated buffer for each port and each class.

• Without “queue-limit” each class of service will get 100 KB of dedicated buffer.

• The size of dedicated buffer can be different for different classes of service. The policy applies to all ports

in the chassis.

• Total ingress buffer minus the dedicated buffer and buffer for control and SPAN will be in the shared

buffer pool.

• The following example sets the dedicated buffer for “class-default” to be 400 KB for all ports.

88

switch(config)# policy-map type network-qos Policy-buffer

switch(config-pmap-nq)# class type network-qos class-default

switch(config-pmap-nq-c)# queue-limit 400000 bytes

switch(config-pmap-nq-c)# system qos

switch(config-sys-qos)# service-policy type network-qos Policy-buffer


Egress Buffer Design

• 9-MB buffer is reserved for egress traffic

• Unicast traffic can be buffered at egress and ingress.

• Multicast is buffered at egress in case of interface oversubscription.

89

Multicast Buffer Shared ~6MB

UPC Egress Buffer (9MB)

Unicast Buffer Dedicated per port


Unicast Flow Control and Buffering • Unicast is queued at egress buffer pool first and then ingress buffer after egress queue

is full.

• Take advantage of ingress buffer from multiple port or ASIC for unicast burst absorption.

• Ensure fairness among multiple ingress ports

90


Identify Packet Drops Due to Congestion

91

switch# show interface ethernet 1/7 Ethernet1/1 is down (SFP not inserted)

Dedicated Interface

<snip>

RX <snip>

0 input error 0 short frame 0 overrun 0 underrun 0

ignored

0 watchdog 0 bad etype drop 0 bad proto drop 0 if down

drop

0 input with dribble 235847488 input discard 0 Rx pause

TX

0 unicast packets 0 multicast packets 0 broadcast

packets

0 output packets 0 bytes

0 jumbo packets

0 output error 0 collision 0 deferred 0 late collision

0 lost carrier 0 no carrier 0 babble 0 output discard 0 Tx pause

switch# show queuing interface ethernet 1/7 Ethernet1/1 queuing information:

TX Queuing

qos-group sched-type oper-bandwidth

0 WRR 100

RX Queuing

qos-group 0

q-size: 100160, HW MTU: 1500 (1500 configured)

drop-type: drop, xon: 0, xoff: 0

Statistics:

Pkts received over the port : 710025080

Ucast pkts sent to the cross-bar : 474177592

Mcast pkts sent to the cross-bar : 0

Ucast pkts received from the cross-bar : 0

Pkts sent to the port : 0

Pkts discarded on ingress :

235847488 Per-priority-pause status : Rx

(Inactive), Tx (Inactive)

Broadcast/Multicast/Unknown unicast flooding drop at egress

Unicast drop at ingress

Ingress drop for each class

Dedicated buffer for this class


Identify Congested Egress Port with VOQ Counters

92

I

X

I

A

1/7

I

X

I

A

Ingress UPC 3

Egress UPC 3

Ingress UPC 4 I

X

I

A

1/10

1/8

Unified Crossbar

Fabric

Using “show interface” CLI, we noticed drops on ingress ports

How do we know which egress port is affected?


Step 1: Map Interface to UPC Number

93

N6004# show hardware internal bigsur all-ports

Bigsur Port Info:

Port |asic|inst|inst|

name |idx |slot|asic|eport|logi|flag|adm|opr|if_index|diag|ucVer

---------+----+----+----+-----+----+----+---+---+--------+----+-----

sup1 |0 |17 |0 | 0 - |0 |b3 |en |dn |15010000|pass| 0.00

sup0 |0 |17 |0 | 1 - |1 |b3 |en |dn |15020000|pass| 0.00

40gb1/1 |1 |0 |0 | 0 - |0 |b3 |en |dn |1a070000|pass| 0.00

40gb1/2 |1 |0 |0 | 1 - |1 |b3 |en |dn |1a071000|pass| 0.00

40gb1/3 |1 |0 |0 | 2 - |2 |b3 |en |dn |1a072000|pass| 0.00

40gb1/4 |2 |0 |1 | 0 - |3 |b3 |en |dn |1a073000|pass| 0.00

40gb1/5 |2 |0 |1 | 1 - |4 |b3 |dis|dn |1a074000|pass| 0.00

40gb1/6 |2 |0 |1 | 2 - |5 |b3 |dis|dn |1a075000|pass| 0.00

40gb1/7 |3 |0 |2 | 0 p |6 |b3 |en |up |1a076000|pass| 0.00

40gb1/8 |3 |0 |2 | 1 p |7 |b3 |en |up |1a077000|pass| 0.00

40gb1/9 |3 |0 |2 | 2 p |8 |b3 |en |up |1a078000|pass| 0.00

40gb1/10 |4 |0 |3 | 0 p |9 |b3 |en |up |1a079000|pass| 0.00

40gb1/11 |4 |0 |3 | 1 - |10 |b3 |dis|dn |1a07a000|pass| 0.00

40gb1/12 |4 |0 |3 | 2 p |11 |b3 |en |up |1a07b000|pass| 0.00

Done.


Step 2: Identify Congested Egress Port with VOQ Counters

94

N6004# show platform software qd info counters voq asic-num 4 +----------+------------------------------+------------------------+-----------+

| port| TRANSMIT| TAIL DROP| HEAD DROP|

+----------+------------------------------+------------------------+-----------+

Eth1/8

QUEUE-3 2574819434 2248180087 0

+----------+------------------------------+------------------------+-----------+

N6004# show platform software qd info counters voq asic-num 3 +----------+------------------------------+------------------------+-----------+

| port| TRANSMIT| TAIL DROP| HEAD DROP|

+----------+------------------------------+------------------------+-----------+

Eth1/8

QUEUE-3 3888954496 2248133231 0

+----------+------------------------------+------------------------+-----------+

Congested egress port and class of service

Ingress

UPC number



95

Overview

Fabric Extender

Performance


Transceivers/Cables

Access Control List

Multicast

Quality of Service

Analytics

Applications


Nexus 5600/6000 Applications Compact Aggregation

96

CORE

AGG.

ACCESS

Nexus 7000

Nexus 7000

Nexus 6004

Nexus 6004/

Nexus 5600

Nexus 2000 FEX

L3

vPC to Nexus 7000 or 6004


Nexus 5600/6000 Applications Large Scale Fabric (Layer 2 or Layer 3)

97

CORE

SPINE

LEAF

Nexus 7000

Nexus 6004

Nexus 6004/

Nexus 5600+2000

L3

FabricPath


Nexus 5600/6000 Applications High Performance Computing (HPC)

98

CORE/AGG.

ACCESS

Nexus 6004

Nexus 3548

Nexus 5600/6004

L3/L2

Fabric


Nexus 6004X Chassis

• Nexus 6004X has 8 slots (similar to the 6004EF chassis)

• Nexus 6004X chassis supports VXLAN functionality

• Provides Flexible Ordering Options – mix/match 10/40G LEMs

• Supports new 12 x 40G LEM VXLAN capable (backward compatible with Nexus 6004EF chassis)

• VXLAN capable LEM will work in non-VXLAN mode in 6004EF chassis

• Same software as on Nexus 6004

• Supports all the optics, FEX models as on Nexus 6004

Target Q3 CY14

99


Nexus 5600/6000 Key Takeaways

100

Performance

Rich Feature set

Line rate L2 and L3 with low latency

STP/vPC/FabricPath/DFA, FEX, L3, VXLAN,

Adapter-FEX/VM-FEX

Scalability 256K MAC / 128K ARP / 32K LPM routes,

16K bridge domains with segment ID

Innovations Intelligent SPAN, Buffer monitoring,

Latency monitoring


Nexus 2000/5600/6000 More Information

101

• http://cisco.com/go/nexus2000



http://cisco.com/go/nexus5000







Complete Your Online Session Evaluation

• Give us your feedback and you could win fabulous prizes. Winners announced daily.

• Complete your session evaluation through the Cisco Live mobile app or visit one of the interactive kiosks located throughout the convention center.

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

102

https://www.ciscolive.com/online


Continue Your Education

• Demos in the Cisco Campus

• Walk-in Self-Paced Labs

• Table Topics

• Meet the Engineer 1:1 meetings

103

Documents

Cisco Nexus 6000 and 5600 with Fabric Extender 2000 Switchd2zmdbbm9feqrf.cloudfront.net/2014/usa/pdf/BRKARC-… · · 2014-05-21Cisco Nexus 6000 and 5600 with Fabric Extender 2000