Nexus9000(Standalone) Architecture And Troubleshooting

Shridhar V. Dhodapkar –Technical Leader (Services)

CCIE 6367 (Routing & Switching)

BRKDCT-3101

Session Abstract

This session presents briefly the architecture of the latest generation

of Nexus 9000 Series Modular switches. Topics include supervisors,

fabrics, I/O modules, forwarding engines, and physical design elements, as

well as the Top of the Rack Nexus9300 Switches.

The session will also cover how to monitor the health of the system.

We will walk you through in depth troubleshooting Tools and Techniques.

Session Goal

• To provide an overall understanding of the Nexus 9000 switching architecture, supervisor, fabric, and I/O module design, packet flows, and key forwarding engine functions

• This session will introduce System Telemetry, Troubleshooting tool Kits and troubleshooting case scenarios

• This session will not examine NX-OS software architecture or other Nexus platform architectures

Related Sessions

BRKARC-2222 - Cisco Nexus 9000 architecture

BRKARC-3471 - Cisco NX-OS Software Architecture

BRKDCT-3144 - Advanced - Troubleshooting Cisco Nexus 7000 Series Switches

• Introduction

• Architecture

• System Health check Telemetry

• Troubleshooting Toolkit

• Nexus 9000 Troubleshooting • Common Link Layer Issues-L1

• Fabric Connectivity and

• In band

• L2/L3 Packet Forwarding

• vPC

• Nexus9000 Specific Limitation and Goodies

Agenda

Introduction

Introduction-What is Nexus9000 Family ?

Nexus 9500 Series Switches Nexus 9300 Series Switches

Nexus9504/Nexus9508/Nexus9516 N9K-C9332PQ N9K-C9372PX N9K-C9372TX N9K-C9396

Architecture

9500 Field Upgradeable Units (FRU)

• 9500 has the following modular components which can upgraded or

replaced in the field

• Supervisor

• Fabric Module

• Line Card

• System Controller

• Fan Tray

• Power Supply

• The Supervisor, System controller ,Fabric Module and LC have OBFL

(On-Board Failure Logging) for failure analysis

Nexus® 9508 Front View Nexus® 9508 Rear View

Nexus 9500 Platform FRU

Supervisor Module-What it is Role

• Redundant Half-width supervisor engine

• Common for 4-, 8-, and 16- slot chassis

• External Clock Input (PTP)

• Responsible for control-plane functions

System Controller-What it is Role

• Offload supervisor from internal device management tasks

• Central Point of Chassis Control

• EOBC Switch (Ethernet Out of Band Channel)

• EPC Switch (Ethernet Protocol Channel)

• Power Supplies via SMB (System Management Bus)

• Fan Trays

Nexus 9500 Platform Line Card

• I/O module with Merchant and Merchant+ ASIC

• Have Various Forwarding Tables

• L2 Mac Table And L3 Host Table

• ACL and Buffers for Queuing

F

P

1

F

P

2

F

P

3

F

P

4

F

P

5

F

P

6

F

P

7

F

P

8

F

P

9

F

P

1

0

F

P

1

1

F

P

1

2

F

P

1

3

F

P

1

4

F

P

1

5

F

P

1

6

F

P

1

7

F

P

1

8

F

P

1

9

F

P

2

0

F

P

2

1

F

P

2

2

F

P

2

3

F

P

2

4

F

P

2

5

F

P

2

6

F

P

2

7

F

P

2

8

F

P

2

9

F

P

3

0

F

P

3

1

F

P

3

2

F

P

3

3

F

P

3

4

F

P

3

5

F

P

3

6

F

P

3

7

F

P

3

8

F

P

3

9

F

P

4

0

F

P

4

1

F

P

4

2

F

P

4

3

F

P

4

4

F

P

4

5

F

P

4

6

F

P

4

7

F

P

4

8

Note: Internal ports are called as Hi-Gig/HG ports

10G SFP+ Ports

40G

QSFP

HG

MUX1

HG

MUX3

FP

49

FP

50

FP

51

FP

52

Northstar 1

Warpcor

e

MF Port

7

-

5

2

-

0

3

1-

2

9

2

6-

2

4 T2

7

-

5

2

6-

2

4

0

-

2

3

-

5

6-

8

9-

1

1

FM4 FM3FM5FM6 FM2 FM1

HG

MUX4

HG

MUX2

HG

MUX5

HG

MUX6

MN Port0 1

2 3

4 5

6 78 9

10

11 Northstar 2MF Port

0

-

2

9-

1

1

MN Port0 1

2 3

4 5

6 78 9

10

11

T2

7

-

5

2

-

0

3

1-

2

9

2

6-

2

4

HG

Line Cards

ASIC Name

NFE=Network Forwarding Engine-Trident 2(T2)

ALE=Application Leaf Engine-North Star(NS)

-Donner N9K-X9564PQ

Nexus 9500 Fabric Module

• Interconnect Line Card slots

• Installed at the rear of the chassis

• Leverages Broadcom Trident II ASICs

• Max 1.92 Tbps per line card slot (6 Fabric Cards)

• 960 Gbps per line card slot (3 Fabric Cards)

• All Fabric Cards are active and carry traffic

• Fan Tray requires Fabric Card to be present in even slot

Trident II

ASIC-NFE

Trident II

ASIC-NFE

32 x 40G

Hi-Gig2

32 x 40G

Hi-Gig2

Fabrics Modules

T2Fabric 1

T2

320 Gbps

(8x 40Gbps)

T2Fabric 2

T2

320 Gbps

(8x 40Gbps)

T2Fabric 3

T2

320 Gbps

(8x 40Gbps)

T2Fabric 4

T2

320 Gbps

(8x 40Gbps)

T2Fabric 5

T2

320 Gbps

(8x 40Gbps)

T2Fabric 6

T2

320 Gbps

(8x 40Gbps)

Line Card Slot

1.92 Tbps

• An 8-Slot chassis fabric module can provide up to 320Gbps to each Line Card slot

• With 6 fabric modules, each Line Card slot can have up to 1.92Tbps duplex

forwarding bandwidth

Data Plane Scaling for 8-Slot Chassis

1.60 Tbps1.28 Tbps

960 Gbps640 Gbps

320 Gbps

Nexus 9500 Fabric Module

NFE

ALE ALE

NFE

Fabric 1

Nx NFE

Fabric 2

Nx NFE

Fabric 3

Nx NFE

Fabric 4

Nx NFE

Fabric 5

Nx NFE

Fabric 6

Nx NFE

2 x 42 Gbps 2 x 42 Gbps

12 x 42 Gbps 12 x 42 Gbps

N = 1 for N9504

N = 2 for N9508

N = 4 for N9516

NFE

ALE ALE

NFE

12 x 42 Gbps 12 x 42 Gbps

Note: Internal ports are called as Hi-Gig/HG ports

Distributed Data Plane of Nexus 9500 Series Switches

Nexus9500 Series Line Card SummaryInformation X9600 Series Line

Cards

X9500 Series Line

Cards

X9400 Series

ASIC Technology Merchant only

N9K-X9636PQ

Merchant+

N9K-X9564PX

N9K-X9564TX

N9K-X9536PQ

Merchant only

• N9K-X9432PQ

• N9K-X9464PX

• N9K-X9464TX

Number of ASIC 3 T2 2 T2 + 2 NS

2 T2 + 2 NS

2 T2 + 2 NS

2 T2 40 gig 32 Ports

1 T2 48 1/10 gig , 4

QSFP

Non Blocking Non Blocking Line rate > 200 byte

packet

Buffer Size 36 MB 104 MB 12 MB with one T2

24 MB with two T2

High Level Block Diagram-N9500

All PSU, SC, SUP, FM, and

LC plug into the same

Power Supply Interface

N9K-C9300 Series

• Fixed Chassis

• Port QSFP+ Uplink Module

• 1 RU or 2RU or 3RU

• AC/DC Power Supply

• Front-to-Back & Back-to-Front Airflow

• Latency: 1-2 usec

• Wire-Speed L2/L3 Forwarding

• Switch will not boot up without GEM

Expansion Module

Nexus 9300 Series Switch Summary

N9396TX/PX N93128TX N9372TX N9372TX N9372PX

NFE (BCM T2) 1 1 2 1 1

ALE ( NorthStar)/GEM GEM-1 NS GEM-1 NS No GEM-1

Donner

No GEM -1

Donner

No GEM- 1

Donner

Oversubscribed No 1.5:1 No No No

Line Rate Yes Yes (packets >

194-Bytes)

Yes Yes

QoS Classes 8 4 8 4 4

Buffer (MB) 36 (12*3) 104 (12*2+40*2) 24 (12*2) 104 (12*2+40*2) 104 (12*2+40*2)

High Level Block Diagram-N9300

Northstar

Egress

(12+12)x12

Ingress

(12+12)x12

BRCM Trident2

48 10G x 12 40G

CPU

2C 1.5GHz

DD

R3

DIM

M2

16G

B T

ota

l

PCIe

Trident II

ASIC

NorthStar

ASIC 1

Network Interfaces

12 x 40G

Hi-Gig2

12 x 40G

Ethernet

Front Panel 48x 1GE/10GE Ports

GEM 4x 40GE QSFP+ Uplinks

1000BaseT

Mgmt Port

2 USB

Ports

eUSB

Boot Flash

12C

• The last 2/3 numbers stand for

total bandwidth in Gigabits

• 93128 – 128G (96 x 10G + 8 x 40G)

• 9396 – 96G (48 x 10G + 12 x 40G)

• 9372 – 72G ( 48 x 10G + 8 x 40 G)

T2-NFE

Parser

L2/L3

Lookup &

forwarding

I-ACL

Traffic

Classification

& Remarking

Ingress

Accounting &

Policing

Packet

Modification

E-ACL

Output Q

& Shaping

EoQALE-NS

Network Interface

Fabric Module

L3 LPM Lookup & Forwarding

T2-NFE

Parser

L2/L3

Lookup &

forwarding

I-ACL

Traffic

Classification

& Remarking

Ingress

Accounting &

Policing

Packet

Modification

E-ACL

Output Q

& Shaping

EoQALE-NS

Network Interface

Ingress Line Card Egress Line Card

Nexus9500 Unicast Packet Flow

Parse the first 128

Byte and extract

header info

L2/L3 Lookup in

MAC Table and IP

Host Table

Classify traffic

based on 802.1q

COS, IP Pres,

DSCP &ACL

Remark if needed

Egress Line card

sends packet to

egress port based

on DMOD/DPORT

Class-based output

queues. Support 6

classes including

control traffic class

Additional buffer is

available for

extended out put

Ques EoQ

Fabric Module

Performs L3 LPM

lookup and resolves

Egress port and

next-hop

OOBFC

SignalingOOBFC

Signaling

N9K-C9300 High Level Block Diagram

HiGiG2 Interface on T2

MACF ports on the GEM and to MACN ports

(16 x 10G) x 3 =

480G FP Bandwidth

(12 x 40G) = 480G

Bandwidth to GEM

Module

(12 x 40G) = 480G FP

Bandwidth Uplink Ports

MACN ports.

(16 x 10G) x 3 =

480G FP Bandwidth

(12 x 40G) = 480G

Bandwidth to GEM

Module

Main Features of Trident2 1280Gbps Switch ASIC

Packet

Buffer

Content aware Engine

DCB Engine

L2

MACL3

Route

L2/L3

Multicast

128 Integrated SerDes

Dynamic Memory

Manager

L2/L3

Processing

Host IF

Counters

128 SERDES@10Gbps

OR

32 SERDES@40Gbps

Features Information

Maximum IO and Core bandwidth 1280G

MAC(L2) Entries 32K min -288K max

L3 Hosts IPv4:16K min-112Kmax

IPv6:8K min-56 max

L3 Multicast Group 8K

Virtual Ports 16K

Maximum number of Physical ports 104

North Star

Features Information

Support Mixed Speed but in Fixed

configuration.

Network Interface:12 Ports Fabric

Interface: 12 40 Gig

Forwarding 720Mpps lookup rate on Ingress

Datapath

720Mpps lookup rate at Egress

Datapath

Shared Memory Subsystem

Ingress Path Buffer

Egress Path Buffer

10 Mbytes

30 Mbytes

Maximum number of Physical ports 24

Broadcom Unified Forwarding Table

SUPPORTED COMBINATIONS

T2 has the following Unified Forwarding Table:

Mode L2 L3 Host LPM

0 288K 16K 16K

1 224K 56K 16K

2 160K 90K 16K

3 98K 122K 16K

4 32K 16K 128K

Routing Mode for Nexus9300

LPM Routing Mode Broadcom T2

Mode

CLI Command

Default system routing mode 3

ALPM Routing mode 4 System routing max-mode l3

N93K#show system routing mode

Configured System Routing Mode: Hierarchical

Applied System Routing Mode: Hierarchical (Default)

N93K#show hardware internal forwarding table utilization module 1

Max Host Route Entries (shared v4/v6): 124928

Max LPM Table Entries : 16384

Routing Mode for Nexus9500

show hardware internal forwarding table utilization mod 1

Max Host Route Entries (shared v4/v6):16384

Max LPM Table Entries : 131072

show hardware internal forwarding table utilization mod 21

Max Host Route Entries (shared v4/v6): 0

Max LPM Table Entries :0

LPM Routing Mode Broadcom T2 Mode Cli Command

Default System routing mode 3 (For Line card)4 (For Fabric Module)

Max-host routing mode 2--Line Card- V6 in LPM3--For Fabric Module

System routing max-mode host

Nonhierarchical routing mode 3--For Line Card4--With max-l3-mode option For Line cardNo Routes on Fabric Module

System routing non-hierarchical Option [max-l3-mode]

64-bit ALPM routing mode Sub mode of mod 4 forFabric modules

System routing mode hierarchical 64b-alpm

Non hierarchical

routing mod

ACL TCAM TABLE

Characteristic

• Ingress ACL: 4K TCAM entries - 4x 512 banks + 8x 256 banks

• Egress ACL: 1K TCAM entries - 4x 256 banks

• Each ACL type needs its own dedicated bank/banks

• IPv4, IPv6 or MAC each needs dedicated bank/banks

• MAC-ACL IPv6 & any QOS needs double-width entries, which means needs at least 2 banks

• VACL is programmed symmetrically in both egress and ingress ACL

Interface

Type

Ingress ACL Egress ACL

SVI TCAM Shared TCAM Not shared

L3 TCAM Shared TCAM Shared

ACL Characteristics

• Atomic/hitless update of existing applied ACL while modified

• Temporary label swap (no use of default-result)

• Two acl copies in tcam, if there is no enough space, process fails

• ACL TCAM banks chaining not supported

• L4OPs/LOUs only used for expansion beyond 5 lines, configurable

• 10 L4op per acl limit

• Specific applications (dhcp, bfd) may install their own ACLs which must merge

with user configured racl, vacl, pacl

TCAM Carving for Nexus 9000TCAM Region-N9500 Size Per Region

IPV4 RACL 1536

IPv4 L3 QOS 256

Ingress System 256

SPAN 256

Ingress CoPP 256

Redirect 256

vPC Convergence 512

Egress IPv4

RACL

768

Egress System 256

256

Ingress

Egress

3X512

256

256

256

256

3X256

512

256

TCAM Region-N9300 Size Per Region

IPv4 PACL 512

IPV4 VACL 512

IPV4 RACL 512

IPv4 Port QOS 256

Ingress System 256

SPAN 256

Ingress CoPP 256

Redirect 256

vPC Convergence 512

Egress IPv4 RACL 256

Egress IPv4 VACL 512

Egress System 256

256

Ingress

Egress

512

256

512

256

256

512

256

512

512

256

256

ACL TCAM Default Region and Carving

• TCAM Banks will first get assigned to Feature which has largest region.

• Next TCAM Bank will get assigned to Feature which need double Width.

• TCAM Carving requires Line Card/TOR reload to take effect

• To read current TCAM allocation

N9K#Show system internal access-lists global

• To reconfigure TCAM Region

N9K(config)hardware access-list tcam <feature name> <size>

Buffer And Queuing-T2

Shared

Buffer

12 MB

Control

Default

OOBFC

• T2 has 12 Mbytes of Buffer shared by all ports for all Traffic

Shared

Buffer

12 MB

Control

Default

Module with T2 only Module with T2 And NS

OOBFC: Out of band flow control unicast service pool

• Shared buffer divided Into Control and default service pool if module is T2 only

• Shared buffer divided into Control, default and OOBFC service Pool if Module is T2 and NS based

Buffer And Queuing-North Star

Trident II

ASIC

NorthStar

ASIC 1

12 x 40G

Hi-Gig2

12 x 40G

Ethernet

Front Panel 48x 1GE/10GE Ports

GEM 4x 40GE QSFP+ Uplinks

Shared

Buffer

Control

Default

SPAN

• North Star has 40 Mbytes of Buffer

• Divided in to Three Pool

• Control , SPAN , Default

10 MB

Buffer

20 MB

Buffer

10 MB

Buffer

Buffer Boost Function with T2 and NS

• Buffer boost is function which allow T2 to use extra

buffer of NS

• When Buffer boost is enabled on a port , T2 Local switch

traffic is Sent to NS for extra buffer space-

• When Buffer boost is disabled on a port, T2 local traffic

to this port remains local on this NFE

• Buffer Boost is enabled by default and can be disabled

on a per port basis

1/10GE 1/10GE 1/10GE

ALE-NS

NFE

T2

Network Interface

10 MB

Buffer 10 MB

Buffer

20 MB

Buffer

12 MB Buffer Shared by all

ports

Fabric Module

1/10GE

System Health check Telemetry

Most Common System Health Check

• What is the Best Recommended NX-OS Release

• CPU & Memory usage

• Inter Process Messaging usage-MTS

• Traffic Stats/Drop To CPU

• CoPP/Hardware Rate Limiter Drops

• Ethernet Out of Band Drops/Error

• Instant Buffer usage Stats

• FATAL System Errors

• Interface Errors for STP/Error disable

• Inter ASIC Utilization

• Hardware Capacity Check

• Consistency Checkers –Various Tables

• GOLD Diagnostic Checks

• Sev1/2 Syslog

Platform Series Minimum Release Recommended Release

Cisco Nexus 9500 6.1(2)I2(2b) 6.1(2)I3(4a)

Cisco Nexus 9300 6.1(2)I2(2b) 6.1(2)I3(4a)

General Recommendation for New and Existing Deployments

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/recommended_releaseb_Minimum_and_Recommended_Cisco_NXOS_Releases_for_Cisco_Nexus_9000_Series_Switches.html

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6-x/scalability/guide_34/b_Cisco_Nexus_9000_Series_NXOS_Verified_Scalability_Guide_612I34/b_Cisco_Nexus_9000_Series_NXOS_Verified_Scalability_Guide_612I34_chapter_01.html

• Software Recommendation

• Verified Scale limits for different features and protocol for each release

CPU & Memory UsageN9K#show system resources Load average: 1 minute: 0.00 5 minutes: 0.03 15 minutes: 0.05

Processes : 432 total, 1 running

CPU states : 2.76% user, 0.75% kernel, 96.48% idle

CPU0 states : 0.00% user, 0.00% kernel, 100.00% idle

CPU1 states : 0.00% user, 1.01% kernel, 98.98% idle

CPU2 states : 0.00% user, 2.94% kernel, 97.05% idle

CPU3 states : 10.89% user, 1.98% kernel, 87.12% idle

Memory usage: 16402328K total,3443588K used, 12958740K free

Current memory status: OK

N9K#show system internal memory-usage-per-module in-KB

Slot 01:Used:1647420 Kbytes,Free:425680 Kbytes,Total:2073100 Kbytes

Slot 02:Used:1627524 Kbytes,Free:445576 Kbytes,Total:2073100 Kbytes

Slot 04:Used:1647560 Kbytes,Free:425540 Kbytes,Total:2073100 Kbytes

N9K#show system internal memory-alerts-log Make sure log is clean

CPU

D

R

A

M

D

R

A

M

CPU & Memory Usage

show processes cpu sort | head lines 12

PID Runtime(ms) Invoked uSecs 1Sec Process

----- ----------- -------- ----- ------ -----------

3357 220 3100 7099 45.50% adjmgr

5853 31655 10181 3109 0.50% ipqosmgr

5859 9489 52308 181 2.00% diag_port_lb

3477 672 3107 216 0.50% netstack

3478 268 175 1535 0.50% ospf

Possibly ARP Table Churn

Provides top process using CPU cycle

N9K#run bash

bash-4.2$ top

top - 11:13:32 up 9 days, 3:34, 4 users, load average: 0.11, 0.11, 0.08

Tasks: 226 total, 1 running, 220 sleeping, 0 stopped, 5 zombie

Cpu(s): 0.8%us, 0.2%sy, 0.0%ni, 98.5%id, 0.0%wa, 0.1%hi, 0.3%si, 0.0%st

Mem: 16402328k total, 3445044k used, 12957284k free, 72676k buffers

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

1 root 20 0 2152 620 556 S 0 0.0 0:08.05 init

2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd

3 root 20 0 0 0 0 S 0 0.0 0:00.58 ksoftirqd/0

Top Command-display top CPU processes

Auto update

“top” provides an ongoing look at processor activity in real time

N9K#sh system internal mts buffers sum | diff

node sapno recv_q pers_q npers_q log_q

sup 320 0 0 4592 0

sup 284 0 19 0 0

sup 250 2 0 0 0

N9K#sh sockets client detail | inc pim|drops|Errors

select drops: 10

Errors:

select drops: 0

Errors:

select drops: 0

Errors:

Inter Process Messaging Usage

For SAP 320 own

by “OSPF” npers_q

increasing

Make Sure Drops/Errors not incrementing

Message and transaction service-MTS

N9K#show hardware internal cpu-mac inband statseth2 stats:

RMON counters Rx Tx

total packets 601163425 318962431

Per Queue Stats

Queue Idx Packet Count Bytes Drops Csum Errors Allocation Failure

Queue 0 17677525 111822449180 0 0 0

- - - - - - - - - - - - - - -SNIP- - - - - - - - - - - - - - - - - - - - - - - -

Queue 7 17677525 111822449180 0 0 0

Interrupt Counters

Rx overrun 0

Error counters

Inband Driver Statistics-CPU Drops

Rate statistics

Rx packet rate (current/peak) 717 / 80695 pps

Tx packet rate (current/peak) 360 / 1338 pps

CRC errors/Collisions/late Collisions

Alignment errors

Symbol errors

Sequence errors/Rx jabbers

RX errors/Rx length errors

N9K# show system internal frame traffic | in drops

Global input drops: bad-interface 0, bad-encap 0, failed-decap 0,

Global output drops:

eth_output_err 0, gre_err 0 otv_err 0 span_drop_en: 0 span_drops: 0

Crossbar down drops : 0 Flood_to_core LTL: Hits: 0 Misses: 0

Traffic Stats/Drops to CPU— (Cont’d)

N9K# show system inband queuing statistics | in drop

bpdu: recv 68, drop 0, congested 0 rcvbuf 2097152, sndbuf 262142 no drop 0

(q0): recv 1249377, drop 0, congested 0 rcvbuf 2097152, sndbuf 262142 no drop

(q1): recv 4138154, drop 0, congested 0 rcvbuf 2097152, sndbuf 262142 no drop

Drops From PKTmgr

Instant Buffer Utilization For CPU Port

show hardware internal buffer info pkt-stats cpu

[Q00-07] 0 0 0 0 0 0 0 0

[Q08-15] 0 0 0 0 0 0 0 0

[Q16-23] 0 0 0 0 0 0 0 0

[Q24-31] 0 0 0 0 0 0 0 0

[Q32-39] 0 0 0 0 0 0 0 0

[Q40-47] 0 0 0 0 0 0 0 0

• Total 48 Queues

• Each Line Display Cell utilized for 8 queues

• One Cell represent approximately208 Bytes

Congestion encountered if Counters

keep incrementing

Ethernet Out Of Band Drops/Errors

N9K#show hardware internal eobc stats | inc dropped

RX packets:248308217 errors:0 dropped:0 overruns:0 frame:0

TX packets:71554006 errors:0 dropped:0 overruns:0 carrier:0

N9K# show system internal emon statsEMON MOD ONLINE BMP: 37f00067FSM ID: 0 EOBCMON=======================================HB tx_req 186396module 1:rx_req 176410rx_resp 176426

rx_miss 7

tx_resp 176410

Provides Stats for all Modules

including Fabric module

Heart bit miss

Instant Buffer Usage Stats

N9K#show hardware internal buffer info pkt-stats mod 1

INSTANCE: 0

----------------------------------------------------------

Output Shared Service Pool Buffer Utilization (in cells)

SP-0 SP-1 SP-2 SP-3

-----------------------------------------------------------

Total Instant Usage 4474 0 89 2939

Remaining Instant

Usage 25466 0 14255 3405

Peak/Max Cells Used 4821 0 327 3060

Switch Cell Count 29940 0 14344 6344

----------------------------------------------------------

show hardware internal ns buffer info pkt-stats

• Instant Buffer utilization per queue per port

• One cell represents 208 bytes

Show hardware internal buffer info pkt-stats input mod 1

• SP-3-Dedicted resource for Control Traffic

• SP-0-Resource for Locally Switched Unicast ,Multicast and SPAN

• SP-2 Extended Output queue for Unicast using buffers from North Star

Buffer polling interval for 7.0

Release is 500msecs

N9K#show hardware internal buffer info pkt-stats mod 1

INSTANCE: 0

Output Shared Service Pool Buffer Utilization (in cells)

SP-0 SP-1 SP-2 SP-3

-------------------------------------------------------------------------

Total Instant Usage 4474 0 89 2939

Remaining Instant Usage 25466 0 14255 3405

-------------------------------------------------------------------------

ASIC Port Q3 Q2 Q1 Q0 CPU SPAN

[13]

UC(OOBFC)->0 0 0 0

UC-> 0 0 0 1249 332 0

MC-> 0 0 0 3247 1996 0

Only printed if there is congestion

• SP-3 Started filling the Queue

• CPU buffer filling up

Port 13 onwards are Front Panel Port

Instant Buffer Usage Stats - With Buffer Usage

CoPP Drops

We recommend that you use the strict default CoPP policy initially and then later modify the CoPP

policies based on the data center and application requirements.

Parameters Default

Default policy Strict

Default Policy 9 policy entries

N9K# show policy-map interface control-plane mod 1 | in dropped

dropped 0 packets;

dropped 0 packets;

dropped 0 packets;

dropped 0 packets;

dropped 7800 packets; Drops Seen for Default-Class at minimal rate is normal

CoPP Drops-Exception drops

class-map copp-system-p-class-l3uc-data (match-any)

match exception glean

class-map copp-system-p-class-redirect (match-any)

match access-group name copp-system-p-acl-ptp

class-map copp-system-p-class-exception (match-any)

match exception ip option

match exception ip icmp unreachable

match exception ipv6 option

match exception ipv6 icmp unreachable

class-map copp-system-p-class-exception-diag (match-any)

match exception ttl-failure

match exception mtu-failure

Goal is to Classify all Traffic Using CoPP

Hardware Rate Limiter

N9K# show hardware rate-limiter mod 1

Units for Config: packets per second

Allowed, Dropped & Total: aggregated since last clear counters

Module: 1

R-L Class Config Allowed Dropped Total

+----------+-----+------------+------------+-------------+

L3 glean 100 0 0 0

L3 mcast loc-grp 3000 0 0 0

access-list-log 100 0 0 0

bfd 10000 1352890 0 1352890

fex 3000 0 0 0

span 50 0 0 0

FATAL System Errors

N9K#show logging onboard mod 1 exception-log | incl FATAL prev 15

------------------------------------------------------------------------

Date (mm/dd/yy)=01/15/15 Time (hs:mn:sec): 00:16:58

OBFL Exception log data for THIS SUP Module:0

********* Exception info for module 0 ********

exception information --- exception instance 1 ----

Device Name : System Manager

Device Errorcode : 0x0000023a

ErrNum (devInfo) : 58 (0x3a)

System Errorcode : 0x401e0089 Service in VDC has had a hap-reset

Error Type : FATAL error

Common Interface Error counters and Status

N9K# show interface counters errors mod 4

Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards

--------------------------------------------------------------------------

Eth4/1 0 100 0 581 0 0

N9K# show interface status err-disabled

Port Name Status Reason

--------------------------------------------------------------------------

Eth4/1 err-disable link-flap

Interface Queuing StatsN9K#show queuing interface 4/18

Egress Queuing for Ethernet4/18 [System]

QoS-Group# Bandwidth% PrioLevel Shape Qlimit

Min Max Units

3 1 - - - 6(D)

-------------------------SNIP--------------------------

0 100 - - 6(D)

----------------------------------------------------

QOS GROUP 0

Unicast | OOBFC Unicast | Multicast

Dropped Pkts | 0| 0| 0|

------------------------------------------------------------

QOS GROUP 7

Unicast | OOBFC Unicast | Multicast

Dropped Pkts | 0| 0| 0|

N9K#show system internal interface counters mod 1

Internal Port Counters (150 secs rate) for Slot: 1

====================================================

Interface ASIC ASIC BCM TxBitRate(BwUtil) TxPktRate RxBitRate(BwUtil) RxPktRate

Port Inst Port (bps) (pps) (bps) (pps)

-----------------------------------------------------------------------------------------

ii1/1/1 HG0 0 1 170512 (0.00) 0 0(0.00) 0

-------------------------------------------Snip------------------------------------------

ii1/1/14 HG1 1 2 0( 0.00) 0 1129882872(2.51) 960753

ii1/1/25 HG0 1 1 1790648 (0.00) 1043 22864(0.00) 20

Inter ASIC Utilization-HG PortsT2 #0 T2 #1

T2 #0 T2 #1 T2 #2

HG00HG00

Line Card

Fabric Module

Verify Consistency Between Software and Hardware Table

Table CLI

Physical Interface show consistency-checker link-state

Port-Channel Membership

show consistency-checker membership port-channels

Mac Address Table show consistency-checker l2

Vlan Membership show consistency-checker membership vlan

L3 interface-LIF programming

L3 interface-LIF programming –Logical Interface for Routing

For RIB and FIB show consistency-checker forwarding ipv4 unicast

Consistency Checkers-Link and STP state

N9K#show consistency-checker link-state mod 1

Link State Checks: Link state only

Consistency Check: PASSED

No inconsistencies found for:

Ethernet1/1

2015 Mar 24 03:23:27 N9508a-SJ %$ VDC-1 %$ vshd: CC_LINK_STATE: Consistency

Check: PASSED

N9K# show consistency-checker stp-state vlan 18

Checks: Spanning tree state

Consistency Check: PASSED

2015 Mar 24 03:25:21 N9508a-SJ %$ VDC-1 %$ vshd: CC_VLAN_STP_STATE:

Consistency Check: PASSED

Consistency Checkers-Port Channel-Vlan Membership

N9K# show consistency-checker membership vlan 18

Checks: Port membership of Vlan in vlan and egr_vlan table

Ports configured as "switchport monitor” will be skipped

Consistency Check: PASSED

Vlan:18, Hardware state consistent for:

Ethernet2/49

2015 Mar 24 03:28:31 N95a%$ VDC-1 %$ vshd: CC_VLAN_MEMBERSHIP: Consistency

Check: PASSED

N9K#show consistency-checker membership port-channels

Checks: Trunk group and trunk membership table.

Consistency Check: Failed

Inconsistency found for port-channel1:

Module:1, Unit: ['Ethernet3/49', 'Ethernet2/49']

Module:26, Unit: ['Ethernet3/49', 'Ethernet2/49’]

Consistency Checkers-Mac address Table

N9K# show consistency-checker l2 module 1

Consistency check: PASSED

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen, + - primary entry using vPC Peer-Link,

(T) - True, (F) - False

Missing entries in the HW MAC Table

VLAN MAC Address Type age Secure NTFY Ports

---------+-----------------+--------+---------+------+----+------------------

Extra and Discrepant entries in the HW MAC Table

VLAN MAC Address Type age Secure NTFY Ports

---------+-----------------+--------+---------+------+----+------------------

Consistency Checkers-L3 Interface

N9K# show consistency-checker l3 mod 1

L3 LIF Checks: L3 Vlan, CML Flags, IPv4 Enable

Consistency Check: PASSED

No inconsistencies found for:

Ethernet1/1

Ethernet1/2

Ethernet1/3

2015 Mar 24 04:07:27 N9508a-SJ %$ VDC-1 %$ vshd: CC_L3_LIF: Consistency Check:

PASSED

Consistency Checker –Unicast Forwarding

N9K#test consistency-checker forwarding

Consistency check started.

N9K# show consistency-checker forwarding ipv4 unicast module 1

IPV4 Consistency check (in progress): table_id(0x1) slot(1)

Elapsed time : 8257 ms

N9K# show consistency-checker forwarding ipv4 unicast module 1

IPV4 Consistency check : table_id(0x1) slot(1)

Execution time : 13244 ms ()

No inconsistent adjacencies.

No inconsistent routes.

Consistency-Checker: PASS for 1

Gold Diagnostic ChecksN9K# show diagnostic result mod 2

Module 2: 48x1/10G-T 4x40G Ethernet Module

Test results:(.=Pass, F=Fail,I=Incomplete,U=Untested,A=Abort,E=Error disabled)

1) ASICRegisterCheck------------> .

2) PrimaryBootROM---------------> .

3) SecondaryBootROM-------------> .

4) OBFL-------------------------> .

6) BootFlash--------------------> .

7) AsicMemory-------------------> .

8) FpgaRegTest---------------- -> .

9) PortLoopback:--------------- > .

Port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

-----------------------------------------------------

U U U . U U U U . . U U U . U U

RewriteEngineLoopback

On Demand Diagnostic can be executed

Sev1/2 Syslog

show logging logfile | incl -1-|-2-

2015 Feb 25 10:30:17 N9508a-SJ %PLATFORM-2-MOD_PWRUP: Module 26 powered up

(Serial number SAL1738D37W)

2015 Feb 25 10:32:37 N9508a-SJ %XBAR-2-XBAR_HGLINK_NOT_UP: fabric link 1 on

module 2 unit 0 connected to fabric module 26 unit:0 is not up during module

bring up

2015 Feb 25 10:32:39 N9508a-SJ %MODULE-2-MOD_FAIL: Initialization of module 26

(Serial number: SAL1738D37W) failed

2015 Feb 25 10:32:39 N9508a-SJ %PLATFORM-2-MOD_PWRDN: Module 26 powered down

(Serial number SAL1738D37W)

Troubleshooting Toolkit

Troubleshooting Toolkit

• Ethanalyzer

• TCP Dump

• ELAM

• Packet Tracer

• Flex Counter

• ERSPAN

• Consistency Checkers

Ethanalyzer-When To Use it

• To Analyze the traffic sent and received by CPU

• It uses wireshark’s code (an open source software)

• Troubleshooting High CPU

• Troubleshoot Control Plane issues Ex. OSPF , PIM , STP

Flap.

SUP

Netstack

NIC-ETH2

Pseudo Inband

Note: Ethanalyzer does not allow capturing of hardware switched traffic between data

ports of the switch

Ethanalyzer-CLI

N9K# ethanalyzer local interface inband capture-filter "pim” detail

Capturing on inband

Frame 1 (60 bytes on wire, 60 bytes captured)

Arrival Time: Mar 24, 2015 10:01:10.018889000

-------Snip------------------

[Protocols in frame: eth:ip:pim]

N9K#ethanalyzer local interface inband display-filter "ospf” detail

Capturing on inband

Frame 1 (82 bytes on wire, 82 bytes captured)

Arrival Time: Mar 24, 2015 10:04:11.425523000

-------------------Snip--------------------

[Frame is marked: False]

[Protocols in frame: eth:ip:ospf]

Some Available Options

autostop :Autostop

decode-internal :Internal

header decoding

limit-captured-frames :Maximum

number of

TCP Dump

• Tcpdump command works on most flavors of Linux operating system

• Helps to prints out a description of the contents of packets on a network interface

• Tcpdump will, if not run with the -c flag, continue capturing packets until it is

interrupted by a SIGINT signal –CTRL-C

• Tcpdump output can be saved to file for further reference

• More info at http://www.tcpdump.org/

Tcpdump -syntax

Syntax: tcpdump -h

tcpdump version 4.1.1

libpcap version 1.2.1

Usage: tcpdump [-aAbdDefIKlLnNOpqRStuUvxX] [ -B size ] [ -c count]

[ -C file_size ] [ -E algo:secret ] [ -F file ] [ -G seconds]

[ -i interface ] [ -M secret ] [ -r file ]

[ -s snaplen ] [ -T type ] [ -w file ] [ -W filecount ]

[ -y datalinktype ] [ -z command ] [ -Z user ]

[ expression ]

bash-4.2#

N9K# show feature | in bash

Feature Name Instance State

bash-shell 1 enabled

N9K# run bash

bash-4.2# sudo su

Password:******

bash-4.2# whoami

root

bash-4.2# tcpdump –c 10 –I ps-inb

Tcpdump-Examples-

bash-4.2# tcpdump -c 100 -w tcpdump.pcap -vvvv -i ps-inb

tcpdump: WARNING: ps-inb: no IPv4 address assigned

tcpdump: listening on ps-inb, link-type EN10MB (Ethernet), capture size 65535

bytes

100 packets captured

102 packets received by filter

bash-4.2#cd /bootflash

bash-4.2# tcpdump -tttt -r tcpdump.pcap | more

reading from file tcpdump.pcap, link-type EN10MB (Ethernet)

2015-04-26 03:21:31.309350 00:0e:ee:01:1b:01 (oui Unknown) > 00:00:00:ff:ff:01

(oui Ethernet), ethertype Unknown (0x8833), length 160:

0x0000: 0000 fc08 0b00 0000 0000 0800 0000 0ffd ...............

-------------------------------------more---------------------------------

Capturing 100 packets And

writing to file

Reading captured file

tshark

bash-4.2$ tshark -i ps-inb

Capturing on inband

0.000000 00:0e:ee:01:1b:01 -> 00:00:00:ff:ff:01 0x8833 Ethernet II

12.328377 00:0e:ee:01:1b:01 -> 00:00:00:ff:ff:01 0x8833 Ethernet II

^C2 packets captured

bash-4.2$

Elam-Embedded Logic Analyzer Module-NS

• Elam Allows to capture single packet based on Trigger

• Triggers are configured using Packet information

• Only Supported on North Star Based(ALE) Line Cards and GEMs

• Use with TAC Supervision

• Help to Answer following Questions

• Was the Packet indeed Received by device on given Line card?

• How did the Packet Look like?

• How was the packet rewritten based on forwarding Decision made by T2?

• Was the Packet correctly forwarded or Dropped?

ELAM Configuration

1. Init

2. Config

5. Reset

3. Arm

4. Read

Trigger

• Init – Initialize the ELAM – select the Asic instance, pipeline and

select lines

module-1# debug platform internal ns elam asic

module-1(NS-elam)# trigger init ingress in-select 3 out-select 5

• Config – Configure the trigger based on different fields in the packet

module-1(NS-elam-insel3)# set outer ipv4 src_ip 13.13.13.10

• Arm – Arm the trigger by setting the fields to match in hardware

module-1(NS-elam-insel3)# start

• Read – Once the trigger is triggered, read the report

module-1(NS-elam-insel3)# report

• Reset – Once the process is complete, reset the trigger to restart

the process

module-1(NS-elam-insel3)# reset

Elam Ingress & Egress Direction-TOR

Front Panel 48x 1GE/10GE Ports

GEM 4x 40GE QSFP+ Uplinks

Trident II

ASIC

NorthStar

ASIC 1

Network Interfaces

12 x 40G

Hi-Gig2

12 x 40G

Ethernet

IP.Add=13.13.13.10

• Traffic entering GEM ports which has NS and exiting T2 is Egress Pipeline

Ex. trigger init egress in-select 3 out-select 5

set outer ipv4 dst_ip 13.13.13.10

• Traffic Entering T2 and Exiting GEM ports is Ingress Pipeline

Ex. trigger init ingress in-select 3 out-select 5

set outer ipv4 src_ip 13.13.13.10

IngressEgress

Elam Ingress & Egress Direction-EOR

Front Panel 48x 1GE/10GE Ports

13.13.13.10

Trident II

ASIC

North Star ASIC

Network Interfaces

12 x 40G

Hi-Gig2

12 x 40G

Ethernet

N

FE

Fabric 1N

FE

Fabric 3

Line Card

• Traffic entering from Fabric Module in to NS of Line Card is Egress Pipeline

Ex. trigger init egress in-select 3 out-select 5

set outer ipv4 dst_ip 13.13.13.10

• Traffic Entering NS and exiting towards Fabric Module is Ingress Pipeline

Ex. trigger init ingress in-select 3 out-select 5

set outer ipv4 src_ip 13.13.13.10

IngressEgress

ELAM Sample Configuration & Key Info

N9K# attach mod 6

module-6# debug platform internal ns elam asic 1

module-6(NS-elam)# trigger init egress in-select 3 out-select 5

module-6(NS-elam-insel3)# set outer ipv4 dst_ip 13.13.13.10

module-6(NS-elam-insel3)# start

module-6(NS-elam-insel3)# status

module-6(NS-elam-insel3)# report

Eth5/1 Eth6/52

Nexus9508 with N9K-X9564TX

13.13.13.1/30

N9K-X9564TX 4 40Gig Port On NS 40 1/10 Gig On T2

13.13.13.10/30

If Packet Captured

Status: Triggered

Important ELAM Fields

GBL_C++: [MSG] - sideband is complete

GBL_C++: [INFO] ovector: 000FFF

GBL_C++ [INFO] hg2_srcmod: 0E

GBL_C++ [INFO] hg2_srcpid: 0D

GBL_C++ [INFO] hg2_dstmod: 11

GBL_C++ [INFO] hg2_dstpid: 0A

GBL_C++ [INFO] ip_da: 000000000000D0D0D0A

GBL_C++ [INFO] ip_sa: 000000000000D0D0D01

N9K# show interface hardware-mappings

-------------------------------------------

----------------------------

Name Ifindex Smod Unit HPort FPort NPort VPort

------------------------------------------

Eth5/2 1a280000 14 0 13 255 0 -1

Eth6/52 1a286600 17 1 10 255 51 -1

Information

is in Hex

Convert to

Dec.

Sideband is the result where

packet will be sprayed.

Should never be “0”

Packet Tracer-T2

• Helps to Trace the packet inside Switch.

• Only packets in the direction of the flow are traced

• Two Acls are installed for each filter on each Line card

• One ACL for Front Panel Port Group

• Second ACL for traffic exiting Fabric Module and ingressing Line

card

Trident II

ASIC

Network Interfaces

FM Mod

Packet Tracer Configuration

13.13.13.10/30

Configure Filter

Start Tracer

Clear/Remove-all

Stop Tracer

Check Counter

Filter

rt

test packet-tracer dst-ip 13.13.13.10 detail-fp

test packet-tracer dst-ip 13.13.13.10 detail-hg

test packet-tracer start

test packet-tracer stop

test packet-tracer show

test packet-tracer clear remove

Sample Configuration & Identify Front Port-LC

13.13.13.10/30

N9K#test packet-tracer dst-ip 13.13.13.10 src-ip 13.13.13.1 detail-fp

N9K#test packet-tracer show filter 1 non-zero Packet-tracer stats

Module 6:

Filter 1 installed: src-ip 13.13.13.1 dst-ip 13.13.13.10 detail-fp

Module 21:

Filter 1 installed: src-ip 13.13.13.1 dst-ip 13.13.13.10 detail-fp

Module 26:

Filter 1 installed: src-ip 13.13.13.1 dst-ip 13.13.13.10 detail-fp

Eth6/52Eth6/1

Nexus9508 with N9K-X9564TX

13.13.13.1/30 13.13.13.10/30

Packet Tracer Sample Configuration & Key Info

N9K# test packet-tracer start filter 1

N9K# test packet-tracer show filter 1 mod 6 non-zero

Packet-tracer stats

Module 6:

Filter1 installed: src-ip 13.13.13.1 dst-ip

13.13.13.10 detail-fp

ASIC instance 0:

Entry 1: id = 7426, count = 5, active, fp, port 13

N9K# show interface hardware-mappings | grep 6/1

Name Ifindex Smod Unit Hport FPort Nport VPort

Eth6/1 1a280000 16 0 13 255 0 -1

13.13.13.10/30

Eth6/52Eth6/1

Nexus9508 with N9K-X9564TX

13.13.13.1/30 13.13.13.10/30

Sample Configuration Identify Fabric Port LC From FM

N9K# test packet-tracer dst-ip 13.13.13.10 src-ip 13.13.13.1 detail-hg

N9K# test packet-tracer start filter 1

N9K# test packet-tracer show mod 6 non-zero

Module 6:

Filter 1 installed: src-ip 13.13.13.1 dst-ip 13.13.13.10 detail-hg

ASIC instance 0:

Entry 0: id = 7425, count = 68, stopped, fp,

ASIC instance 1:

Entry 1: id = 7426, count = 13, stopped, hg, port 1

Entry 2: id = 7427, count = 11, stopped, hg, port 2

13.13.13.10/30Eth6/52Eth8/1

Nexus9508 with N9K-X9564TX

13.13.13.1/30 13.13.13.10/30

Flex Counters –Adjacency Statistics

• Flex counters used to count Next hop Adjacency stats

• One can attach Stats to multiple Adjacency at same time

• One Stat Counter per adjacency

• Total Flex Counters are 16K per Switch

How To Configure Flex Counters

N9K# sh ip route 13.13.13.10

IP Route Table for VRF "default"

‘'%<string>' in via output denotes VRF <string>

13.13.13.8/30, ubest/mbest: 1/0

*via 13.13.13.6, Eth6/52, [110/41], 00:33:14, ospf-10, intra

N9K# test hardware internal adjacency statistics nexthop ipv4 13.13.13.6

interface ethernet 6/52 (enable |disable | show)

13.13.13.10/30Eth6/52Eth6/1

Nexus9508 with N9K-X9564TX

13.13.13.1/30 13.13.13.10/30

Sample Configuration

13.13.13.10/30

N9K# test hardware internal adjacency statistics nexthop ipv4 13.13.13.6

interface ethernet 6/52 show

Module:21 Unit:0

------------------

Adjacency counters for nhip 13.13.13.6 if Ethernet6/52:

Ucast: Packets 738 Bytes 90036

Mcast: Packets 0 Bytes 0

Module:22 Unit:1

------------------

Adjacency counters for nhip 13.13.13.6 if Ethernet6/52:

Ucast: Packets 946 Bytes 115412

Mcast: Packets 0 Bytes 0

Eth6/52Eth6/1

Nexus9508 with N9K-X9564TX

13.13.13.1/30 13.13.13.10/30

SPAN & ERSPAN

• Switch Port Analyzer”• Provides efficient, high-performance traffic monitoring service• Duplicates network traffic to one or more monitor interfaces • Types Of SPAN

• Local SPAN• Encapsulated Remote SPAN(ERSPAN)

• Applications:• Troubleshooting connectivity issues• Base lining network utilization/performance• Detecting anomalous traffic flows

• On Nexus9000 Span Traffic uses dedicated queue• Queue carrying SPAN traffic has low Priority over other queue’s

during congestion

SPAN QOS Queue

N9K# show queuing interface ethernet 4/18 | begin SPAN

| SPAN QOS GROUP |

+-----------------------------------------------------------------+

| | Unicast | OOBFC Unicast | Multicast |

+------------------------------------------------------------------+

| Tx Pkts | 0| 0| 0|

| Tx Byts | 0| 0| 0|

| Dropped Pkts | 0| 0| 0|

| Dropped Byts | 0| 0| 0|

| Q Depth Byts | 0| 0| 0|

SPAN Configuration

N9K(config)# monitor session 1

N9K(config-monitor)# source interface sup-eth 0 both

N9K(config-monitor)# source interface ethernet 6/1

N9K(config-monitor)# destination interface ethernet 6/2

N9K(config-monitor)# No Shut

N9K(config-monitor)# show monitor

Session State Reason Description

--- ----- ------------ --------------------

1 up The session is up Local SPAN Session

e6/1 e6/2

Local SPAN

LocalSup-eth

N9K(config)#int et 6/2

N9K(config-if)# switchport monitor

ERSPAN Configuration

N9K(config)# monitor erspan origin ip-address 13.13.13.2

global

N9K(config)# monitor session 1 type erspan-source

N9K(config-erspan-src)# header-type 3

N9K(config-erspan-src)# source interface ethernet 6/1

N9K(config-erspan-src)# erspan-id 1

N9K(config-erspan-src)# ip ttl 16

N9K(config-erspan-src)# vrf default

N9K(config-erspan-src)# destination ip 9.1.1.2

N9K(config-erspan-src)# marker-packet-2

N9K(config-erspan-src)# no shut

Layer 3

e6/1

ERSPAN

e6/2

L3

Only Supports Source ERSPAN

Type-3 Header 32-bit Timestamp

Supports on Nexus9300 only

Marker packet carry original UTC time

stamp to over come 32-bit wrapper

issue

Consistency Checkers-Summary

• Show consistency-checker stp-state vlan

• Show consistency-checker link-state

• Show consistency-checker membership vlan

• Show consistency-checker membership port-channels

• Show consistency-checker membership port-channels

• Show consistency-checker l2

• Show consistency-checker l3

• Show consistency-checker forwarding ipv4 unicast

Nexus 9000 Troubleshooting

Understanding T2 interfaces-Xe0/hg

N9K# bcm-shell mod 1 "show unit"Unit 0 chip BCM56852_A2 (current)Unit 1 chip BCM56852_A2

N9K#bcm-shell mod 1 “0:ps”ena/ speed/ link auto STP lrn inter max loop

port link duplex scan neg? state pause discrd ops face frame back

hg0 up 42G FD HW No Forward None FA XGMII 16360hg2 up 42G FD HW No Forward None FA XGMII 16360--------------------------------Snip----------------------------------Hg11 up 42G FD HW No Forward None FA XGMII 16360Xe0 !ena 40G FD HW No Disable None FA XGMII 1582xe1 up 40G FD HW No Disable None FA XGMII 1582--------------------------------Snip----------------------------------Xe11 !ena 40G FD HW No Disable None FA XGMII 1582

Hg=Internal Ports

Xe=Front Panel Port

QSPF

Ports

QSPF

PortsF

P

01

F

P

02

F

P

03

F

P

04

F

P

05

F

P

06

F

P

07

F

P

08

F

P

09

F

P

10

F

P

11

F

P

12

F

P

13

F

P

14

F

P

15

F

P

16

F

P

17

F

P

18

F

P

19

F

P

20

F

P

21

F

P

22

F

P

23

F

P

24

T2

Instance 0T2

Instance 1

Eth1/1 Eth1/24

Xe0 Xe0

hg0 hg11

Xe11

Eth1/12

Xe11

Eth1/13

hg0 hg11

Layer -1 Issues- Transceiver Not Recognized

N9K# show interface ethernet 4/18 transceiver details

Ethernet4/18

transceiver is not present

module-4# show hardware internal bcm-usd event-history xcvr 18

1) Event:E_STRING, length:135, at 220346 usecs after Thu Apr 16 20:50:17 2015

bcm_usd_xcvr_fcot_notify_default(941): [unit=0 nxosport=18 bcmport=30]

fcot_state:0x2 fcot_type:0 sent MTS_OPC_FCOT_EVENT_INFO, rc 0x0

2) Event:E_STRING, length:93, at 647132 usecs after Thu Apr 16 20:50:14 2015

bcm_usd_xcvr_fcot_scan_sfp(3003): [unit=0 nxosport=18 bcmport=30]

FCOT not supported err=-1

Interface MTU/Speed/Flow Control Verification

N9K# show interface Ethernet 4/18

Ethernet4/18 is up

admin state is up, Dedicated Interface Belongs to Po10

Hardware: 10000/40000 Ethernet, address: 7c69.f66e.d860 (bia 7c69.f66e.d860)

MTU 9216 bytes, BW 40000000 Kbit, DLY 10 usec

N9K# bcm-shell module 4 ” 1: ps Xe17"

ena/ speed/ link auto STP lrn inter max loop

port link duplex scan neg? state pause discrd ops face frame back

xe17 up 40G FD HW No Disable None FA SR4 9298

Interface Flow Control Check

N9K#Show interface ethernet 1/1 flowcontrol

Port Send FlowControl Receive FlowControl RxPause TxPause

admin oper admin oper

-----------------------------------------------------------------------------

Eth1/1 off off off off

0 0

N9K#bcm-shell module 1 "ps" Wrong programming

ena/ speed/ link auto TP lrn inter max loop

port link duplex scan neg? state pause discrd ops face frame back

xe0 up 10G FD HW No Disable TX RX None FA SFI 9298

Interface Input Drops

N9K#bcm-shell mod1 “ cstat xe29”+------------------Programmable Statistics Counters[Port xe29]------+

| Type | No. | Value | Enabled For |

+----------------------------------------------------------------- -+

| RX | 0(R)| 19163028| RIPD4 RIPD6 RDISC RPORTD |

| | | | PDISC VLANDR |

| | 1(R)| 28744286| IMBP |

| | 4 | 993820| RPORTD FcmPortClass3RxDiscards |

| | 6 | 19163407| RFILDR FcmPortClass2RxDiscards |

| | 7 | 19163048| RDROP |

| | 8 | 18169208| VLANDR |

+-------------------------------------------------------------------+

| | 3(R)| 14704| TPKTD |

| | 4(R)| 968303| TGIP4 TGIP6 FcmPortClass3TxFrames|

| | 6 | 968303| TGIP4 FcmPortClass3TxFrames |

+-------------------------------------------------------------------+

Ethernet1/30 is upHardware: 1000/10000 Ethernet, address: 7426.acea.ceb9 (bia 7426.acea.ceb9)

EtherType is 0x8100

0 input with dribble 1316 input discard

bcm-shell mod 6 "cstat info" | gre VLANDR

VLANDR Rx VLAN drops

Fabric Connectivity and Troubleshooting

• In an 4-slot chassis N9K-C9504-FM has 1 T2 per module

• In an 8-slot chassis N9K-C9508-FM has 2 T2 per module

• In an 16-slot chassis N9K-C9516-FM has 4 T2 per module

• FMs provides redundancy for internal data flow, the loss of FMs just increases

the oversubscription factor.

T2

T2 T2

T2T2

T2

N9K-C9508-FM-8 N9K-C9516-FM-16

T2

N9K-C9508-FM-4

Full-Rate Mode(FRM) V/S Oversubscribed Mode(OSM)

• Each T2 have 32 40Gigport with total capacity of 1.2Tbps with “2” switching

modeOSM(Default) - Uses all 32 40 Gig ports Line Rate achieved for packets > 200 Bytes

FRM - Uses only 24 40 Gig ports Line rate achieved for > 64 Bytes

Configuration Knob to Change the mode. N9K(config)# system fabric-mode full-rate

Configuration effective after RebootN9K#show system fabric-mode

Applied System Fabric Mode:Full rate mode

Use FRM mode to achieve line rate for 64 byte packets on 9636PQ , 9564PQ ,

9564TX cards

All other 94xx line cards will not be powered up in this mode

RTAG7 and DLB• Two Packet Hashing algorithm available from LC to FM

• RTAG7-To Select HG Port use Packet Header.

• For a flow same HG Link is used

• DLB-Dynamic Load Balancing- Default algorithm

• Initial Hash same as RTAG7

• Based on Link Quality pick up optimum HG Port

• Better utilization of all HG links

• N9K(config)# port-channel load-balance internal [dlb/rtg7]

• N9K# show port-channel load-balance internal algorithm

• HighGig port-channel load balance algorithm: dlb

LC1 LC2

FM-2FM1FM6

HG-Ports

HG-Ports

Higig Link Failures – Fabric Module Policy

• For any single Higig link failure between FM and LC

Bring down the FM, if there is more than one FM

Else bring down LC

• Multiple Higig links failures for a Single LC going to Multiple FM - Bring down

the LC module.

• Multiple Higig links failures on LC to one of the FM - Bring down the LC module

4/8 slot Chassis – Fabric Connectivity

N9K-X9536PQ

T

2

T

2T

2

T

2T

2

T

2T

2

T

2T

2

T

2

T2

T

2

T

2

T2

40 Gig Link

• 9500/9600 Series Line Card’s T2

have connectivity to all 6 Fabric

Module’s T2

• 9400 series Line cards connects to

all T2 but use only 4 Fabric Modules

-No Connection to Slot 21 & 25

• Traffic between 9500/9600 Series

Line Card and 9400 Line card will

use subset Hi Gig links .

16 slot Chassis – Fabric ConnectivityN9K-C9516-FM

T

2

T

2

T

2

T

2

T

2

T

2

T

2

T

2

T

2

T

2

T

2

T

2

T

2

T

2

T

2

T

2

T

2

T

2

T

2

T

2

N9K-X9536PQ

T

2

T

2

T2

T

2

T

2

T2

• 9500 Series Line Card’s T2 will have

connection to all 6 Fabric Module but to

only 2 T2’s from each Fabric Module

• 9500 series Line Card’s T2 will have

connection to all 4 T2’s of Fabric module if

there are only 3 Fabric module present

• 9400 series cards connects to all T2 but

use only 4 FM-No Connection to Slot 21 &

25

• Traffic between 9500 Series Line Card and

9400 will use subset Hi gig links.

• N9K-X9636PQ line card module is not

supported in 16 slot chassis

40 Gig Link

• With 3 FM configuration All 4 T2 units in

each FM are connected to 9500 series LC

modules' T2 units

• Each blue line represents one 40 Gig link

16 slot Chassis – Fabric Connectivity

T

2

T

2

T

2

T

2T

2

T

2

T

2

T

2T

2

T

2

T

2

T

2

N9K-X9536PQ

T2 T2

HG

MUX1

HG

MUX3

Northstar 1

Warpcore

MF Port

7-

5

2-

0

31-

29

26-

24

T2

7-

5

26-

24

0-

2

3-

5

6-8 9-

11

FM24 FM23FM25FM26 FM22 FM21

HG

MUX4

HG

MUX2

HG

MUX5

HG

MUX6

MN Port

0 1 2

3

4 5 6

78 9 10

11

Northstar 2MF Port

0-

2

9-

11

MN Port0 1 2

3

4 5 6

78 9 10

11

T2

7-

5

2-

0

31-

29

26-

24

• Line cards N9K-X9464PX/TX , N9K-X9564PQ/TX have Mux

• Mux used for connecting HiG Link from Line Cards to multiple Fabric Module

• Mux available only for Half of the HiG interface of LC

• By Default Mux Link Active to Odd number of Fabric Module

Line Card

Fabric Module’s

Active Mux Link

Standby Mux Link

Line Cards With Mux to FM

FM Connectivity For N9K-X9564PX –With MUX

show system internal fabric connectivity mod 5 | in B

HiGIG Link-info Linecard slot:5

LC-Slot LC-Unit LC-HGLink MUX FM-Slot FM-Unit FM-HGLink

5 0 HG02 1B 25 0 HG12

5 0 HG03 1B 25 1 HG12

show system internal fabric connectivity mod 5

HiGIG Link-info Fabriccard slot:5

LC-Slot LC-Unit LC-HGLink MUX FM-Slot FM-Unit FM-HGLink

5 0 HG02 1A 26 0 HG14

5 0 HG03 1A 26 1 HG14

With FM from Slot 25 Down FM-26

T2-0 T2-1

HG014

LC

T2-0 T2-1

FM-25

T2-0 T2-1

MUX

HG02 HG03

HG012HG012 HG014

AB

• Line cards N9K-X9464PX/TX , N9K-X9564PQ/TX have Mux

• Mux used for connecting HiG Link from Line Cards to multiple Fabric Module

• Mux available only for Half of the HiG interface of LC

• By Default Mux Link Active to Odd number of Fabric Module

FM Connectivity For N9K-X9564PX –With MUX

show system internal fabric connectivity mod 5 | in B

HiGIG Link-info Linecard slot:5

LC-Slot LC-Unit LC-HGLink MUX FM-Slot FM-Unit FM-HGLink

5 0 HG02 1B 25 0 HG12

5 0 HG03 1B 25 1 HG12

show system internal fabric connectivity mod 5

HiGIG Link-info Fabriccard slot:5

LC-Slot LC-Unit LC-HGLink MUX FM-Slot FM-Unit FM-HGLink

5 0 HG02 1A 26 0 HG14

5 0 HG03 1A 26 1 HG14

With FM from Slot 25 Down

LC

T2-0 T2-1

FM-25

T2-0 T2-1

FM-26

T2-0 T2-1

MUX

HG02 HG03

HG012HG012 HG014 HG014

LC

T2-0 T2-1

FM-25

T2-0 T2-1

FM-26

T2-0 T2-1

MUX

HG02 HG03

HG012HG012 HG014 HG014

A

A

B

B

Fabric Troubleshooting commands

show system internal fabric connectivity mod 1

HiGIG Link-info Linecard slot:1

LC-Slot LC-Unit LC-HGLink MUX FM-Slot FM-Unit FM-HGLink

1 0 HG00 - 21 0 HG00

1 0 HG01 - 21 1 HG00

show system internal fabric connectivity mod 21

HiGIG Link-info Fabriccard slot:21

FM-Slot FM-Unit FM-HGLink LC-Slot LC-Unit LC-HGLink MUX

21 0 HG00 1 0 HG00

21 1 HG00 1 0 HG01

T2

#0

T2

#1

T2

#0

T2

#1

T2

#2

Line Card Slot-1

Fabric Module Slot-21

HG00HG00

Fabric Port Drops and Link Status

N9K# bcm-shell mod 21 "ps” | inc hg0

ena/ speed/ link auto STP lrn inter max loop

port link duplex scan neg? state pause discrd ops face frame back

hg0 up 42 FD HW No Forward None FA XGMII 16360

N9K# show hardware internal fabric interface asic counters mod 21

Counters for Fabric Ports:

FabricInterface Forward Forward Error Pkt Error Pkt QOS Rx QOS Tx

RxDrops TxDrops RxDrops TxDrops Drops Drops

0 / 1 / HG0 0 0 0 0 0 0

1 / 1 / HG0 0 0 1 0 0 0 0

Fabric Port STP State HW point of View

N9K# sh vlan id 100

VLAN Name Status Ports

---- ------------------ --------- 100

VLAN0100 active Po1, Eth1/1

show sys internal xbar event-history {trace|errors|msgs|sw}

show sys internal xbar-client event-history {trace|errors|msgs|sw}

show tech-support xbar

N9K# bcm mod 21 " stg show”

STG 5: contains 1 VLAN (100)

Forward: hg

Path of the Packet -Inband

CPU

NIC-Eth2

Netstack

NIC-Eth3

System Controller-SC1

Fa

bric

Mo

du

le

Fabric

Module

Fa

bric

Mo

du

le

Line Card

Mod21Mod26

Mod29

Mod23

OSPF Hello

Eth6/1

• Traffic from all ingress Line Card

to Supervisor will hash to one

Fabric module

• Traffic from Supervisor Card to

Egress Line cad will hash on one

FM. May not be same

• CoPP is operational on all LC.

However aggregate CoPP is on

FM

Check for Drops/Errors-Line Card

N9K#show hardware internal interface ethernet 6/1 asic counters

Important Counters/Drops

--------------- --------- --------- --------- --------- --------- ---------

Interface Name Forward Forward Error Pkt Error Pkt QOS Rx QOS Tx

RxDrops TxDrops RxDrops TxDrops Drops Drops

--------------- --------- --------- --------- --------- --------- ---------

Ethernet6/1 870 0 100 0 0 0

--------------- --------- --------- --------- --------- --------- ---------

Forward Rx Drops = [ RDBGC0 RDBGC4 RDBGC6 RDBGC7 RDBGC8 ]

Forward Tx Drops = [ TDBGC1 TDBGC3 TDBGC5 (excludes expected Multicast drops)]

ErrorPkt Rx Drops= [ IUNHGI IUNKOPC RFCS RALN RFLR RERPKT RJBR RSCHCRC RUND RMTUE]

ErrorPkt Tx Drops= [ TJBR TFCS TRPKT RMTUE TUFL TPCE ]

QOS Rx Drops = [ RDISC DROP_PKT_ING DROP_PKT_IMTR DROP_PKT_YEL DROP_PKT_RED ]

QOS Tx Drops = [ MCQ_DROP_PKT(0) MCQ_DROP_PKT(1) MCQ_DROP_PKT(2)

Use slot <#> show hardware internal interface indiscard-stats instance <#>

N9K#bcm-shell mod 6 "listreg RALN"| grep Description

Description: Receive Alignment Error Frame Counter

Trident II

ASIC

North Star ASIC

Network Interfaces

Line Card

RDBGC0

Instant Buffer Usage Stats-With Buffer UsageN9K#show hardware internal buffer info pkt-stats mod 6

INSTANCE: 0

Output Shared Service Pool Buffer Utilization (in cells)

SP-0 SP-1 SP-2 SP-3

-------------------------------------------------------------------------

Total Instant Usage 4474 0 89 2939

Remaining Instant Usage 25466 0 14255 3405

------------------------------------------------------------------------

ASIC Port Q3 Q2 Q1 Q0 CPU SPAN

[13]

UC(OOBFC)-> 0 0 0 0

UC-> 0 0 0 1249 332 0

MC-> 0 0 0 3247 1996 0

Only printed if there is congestion

• SP-3 Started filling the Queue

• CPU buffer filling

up

CoPP Drops on Line Card

N9K# show policy-map interface control-plane mod 6 class copp-system-p-class-

critical | in ospf|trans|dropped

match access-group name copp-system-p-acl-ospf

transmitted 21898 packets;

dropped 0 packets;

Trident II

ASIC

North Star ASIC

Network Interfaces

Line Card

Identify FM -Check CoPP Drops

N9K# show hardware internal cpu-mac inband active-fm traffic-to-sup

Active FM Module for traffic to sup:

0x00000015 Fabric Module in Slot 21 carry all traffic to Sup

N9K# show policy-map interface control-plane mod 21 class copp-system-p-class-

critical | in ospf|trans|dropped

match access-group name copp-system-p-acl-ospf

match access-group name copp-system-p-acl-ospf6

transmitted 21898 packets;

dropped 0 packets;

Check for Drops/Errors-Fabric ModuleN9K# show system internal fabric connectivity mod 6 | grep 21 Identify HG Port on LC and FM

LC-Slot LC-Unit LC-HGLink MUX FM-Slot FM-Unit FM-HGLink

6 0 HG10 3B 21 0 HG15

N9K# sh hardware internal fabric interface asic counters module 6 instance 0 asic-port 11

Important Counters/Drops Verify Drops/Error on HG port on LC

FabricInterface Forward Forward Error Pkt Error Pkt QOS Rx QOS Tx

RxDrops TxDrops RxDrops TxDrops Drops Drops

0 / 11 / HG10 0 0 0 0 0 0

N9K# sh hardware internal fabric interface asic counters mod 21 in 0 asic-port 16

RxDrops TxDrops RxDrops TxDrops Drops Drops

0 / 11 / HG15 0 0 0 0 0 0

Verify Drops Between FM and SC

module-21# show mvdxn internal port-status

Switch type: Marvell 98DXN11 - 10 port switch Fabric Module in Slot 21

Port Descr Enable Status ANeg Speed Mode InByte OutByte InPkts OutPkts

3 SC1EPCswitch Yes UP No 2 6 109548011 117051401 274144 587285

module-29# show mvdxn internal port-status

Switch type: Marvell 98DXN11 - 10 port switch System Controller in Slot 29

Port Descr Enable Status ANeg Speed Mode InByte OutByte InPkts OutPkts

7 FM1EPCswitch Yes UP No 2 6 746159513 60543666 620863 269592

10 port switch on System

controller and Fabric

module connect SC to FM

FABRIC CARD

System Controller

MVDXN-SW

MVDXN-SW

Drops/Errors On Supervisor

N9K#show hardware internal cpu-mac inband counters in eth|ps-

inb|dro

eth2 Link encap:Ethernet HWaddr 00:00:00:01:1b:01

RX packets:2922013 errors:0 dropped:0 overruns:2 frame:0

TX packets:1652929 errors:0 dropped:0 overruns:0 carrier:0

eth3 Link encap:Ethernet HWaddr 00:00:00:01:1b:01

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

ps-inb Link encap:Ethernet HWaddr 00:00:00:01:1b:01

RX packets:54204 errors:0 dropped:3579 overruns:0 frame:0

TX packets:50626 errors:0 dropped:0 overruns:0 carrier:0

Netstack

NIC-Eth2

Pseudo Inband

NIC-Eth3

Supervisor Card

Drops/Errors On Supervisor-Cont.

N9K#show hardware internal cpu-mac inband stats | in errors|rate|Queue

Queue Idx Packet Count Bytes Drops Csum Errors Allocation Failure

Queue 0 65429 580195964 2 0 0

Queue 7 65429 580195964 0 0 0

CRC errors ...................... 0

Alignment errors ................ 0

Symbol errors ................... 0

Carrier extension errors .........0

Rx packet rate (current/peak) 812 / 1097 pps

Tx packet rate (current/peak) 454 / 741 pps

Related show tech(s)

Nexus9500# sh tech-support inband

counters

Nexus9500# show tech-support pktmgr

Nexus9500# show tech-support <service>

L2 Mac And Vlan Table Verification

N9K# sh mac address-table dynamic vlan 100

Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link, (T) -

True, (F) - False

VLAN MAC Address Type age Secure NTFY Ports

* 100 547f.ee1c.06fc dynamic 0 F F Eth6/1

N9K# bcm-shell mod 6 " l2 show" | in Hit

mac=54:7f:ee:1c:06:fc vlan=100 GPORT=0x800800d modid=16 port=13/xe0 Hit

N9K# bcm-shell mod 6 "vlan show 100”

vlan 100 ports xe0,hg ....... untagged xe0

interface Ethernet6/1

switchport

switchport access vlan 100

no shutdown

Eth6/1

Mac=547f:ee1c.06fc

Spanning Tree VerificationN9K# sh spanning-tree interface ethernet 6/1

Vlan Role Sts Cost Prio.Nbr Type

VLAN0100 Desg FWD 4 128.1537 P2p

N9K# bcm-shell mod 6 "dump vlan 100”

VLAN.ipipe0[100]: <VP_GROUP_BITMAP=0x00000……STG=0X67

FID_ID=0x64

N9K# bcm-shell mod 6 "stg stp 103”

STG 103:

Block: xe1-xe47

Forward: xe0,hg

interface Ethernet6/1

switchport

switchport access vlan 100

no shutdown

Eth6/1

Mac=547f:ee1c.06fc

N9K# Dec 0x67=103

STG= STP Group IDN9K# Dec 0x64=100

FID_ID=Vlan ID.

Unicast L3 Forwarding

• T2 has combination of dedicated TCAM table space and shared hash table

memory known as Unified Forwarding Table (UFT)

• The UFT is partitioned into three forwarding tables

• MAC Address Table

• IP Host Table

• Longest Prefix Match-LPM Table

• To maximize the system-wide forwarding scalability UFT tables on line

cards and fabric modules for different forwarding lookup functions

FM

LC

Feature Scale

L3 Host Table

And L2/L3

Multicast

120K

L2 Mac Table 96K

Feature Scale

L3 LPM

Table

128K

Unicast L3 Forwarding- Component Information

Hardware-T2

uFDM

Supervisor

AM

uRIB

OSPF ARP

FIB Manager

Forwarding Hardware

Theory of Operation

Software/Hardware Programming

• OSPF communicates with uRIB to build the

routing table

• AM builds the next-hop adjacency entry

• uFDM distributes the information to the line

cards

• IP FIB (running on the line cards) programs the

ASIC components with the forwarding and

adjacency information.

Remember: Software forwarding by the SUP is only

used for control and exception packets

L3 Unicast Troubleshooting Flow

HW Programming

On LC/FM

Use BCM commands

Next-Hop

Check the routing table

Checking Route on

RIB And FIB.

ARP/MAC

Check the ARP Table

Check Forwarding Route

Show ip route [ipv4] [<prefix>]

Show ip arp [ipv4]

show ip adjacency (Ipv4]

show forwarding adjacency platform [ipv4]

module <mod>

show forwarding [ipv4] route module <mod>

bcm-shell mod 22 "l3 defip show"

Unicast L3 Forwarding- Two Possible Scenarios

Case 1: If incoming packet hit /32 host route on LC, forwarding decision is made on LC

Case 2: If incoming packet miss /32 host route on LC. Now for Longest Prefix

match (LPM) the packet get forwarded to FM

• Install a default route 0.0.0.0/0 on Line Cards using the virtual MOD ID for Fabric Module

as the DMOD to force Line Cards to forward LPM packets to Fabric Modules

• Fabric Modules perform LPM lookup and forward packets to the resolved Destination

MOD/Destination PORT

Also will verify How to Check ECMP Route

Network Diagram-Problem Definition

13.13.13.0/30

13.13.13.12/30

.1

13.13.13.8/30

.2 .17 .9 .10

N9K#

Nexus3064Q-ESC#N9508d-SJ#

N9508c-SJ#

Nexus3064Q-ESC# ping 13.13.13.10

PING 13.13.13.10 (13.13.13.10): 56 data bytes

Request 0 timed out

Nexus3064Q-ESC# traceroute 13.13.13.10

traceroute to 13.13.13.10 (13.13.13.10), 30 hops max, 40 byte

packets

1 13.13.13.2 (13.13.13.2) 1.124 ms 0.911 ms 0.752 ms

2 * * *

.18

.13 .14

13.13.13.16/30

Router MAC Programming Check

• Router Mac address must be programmed in Hardware

N9K1#show interface ethernet 6/1 | grep address

Hardware: 100/1000/10000 Ethernet, address: 003a.99fc.dd7f

N9K1# bcm-shell mod 6 "0:d chg my_station_tcam" | grep dd7f

MY_STATION_TCAM.ipipe0[0]: <VALID=1,------snip----MAC_ADDR=0x003a99fcdd7f,

Verify /32 Host Route on Line card-Case 1

N9K1#show ip route 13.13.13.14

13.13.13.14/32, ubest/mbest: 1/0, attached

*via 13.13.13.14, Eth6/33, [250/0], 00:37:24, am

N9K1#bcm-shell mod 6 "0:l3 l3table show" | grep 13.13.13.14

Entry VRF IP address Mac Address INTF MOD PORT CLASS HIT

10 1 13.13.13.14 00:00:00:00:00:00 100010 0 0 0 y

N9K1#bcm-shell mod 6 "0:l3 egress show"| grep 100010

Entry Mac Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop

100010 88:f0:31:bf:ad:17 4095 4432 45 16 -1 no no

N9K1#show system internal ethpm info interface ethernet 6/33 | grep -i STATIC

IF_STATIC_INFO: port_name=Ethernet6/33,if_index:0x1a284000,ltl=40875,slot=5,

nxos_port=32,dmod=16,dpid=45,

/32 Host Entry

Next Hop Reached via L3-Port Channel

N9K1#show ip route 10.164.112.22

10.164.112.22/32, ubest/mbest: 1/0

*via 13.13.13.14, Po200, [110/3], 00:09:33, ospf-10, intra

N9K1#bcm-shell mod 6 "0:l3 l3table show" | grep 10.164.112.22

Entry VRF IP address Mac Address INTF MOD PORT CLASS HIT

175660 1 10.164.112.22 00:00:00:00:00:00 100012 0 0 0 y

N9K1#bcm-shell mod 6 "0:l3 egress show"| grep 100012

Entry Mac Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop

100010 88:f0:31:bf:ad:17 665 4761 3t 0 -1 no no

N9K1#show system internal ethpm info interface port-channel 200 |grep –I STATIC

IF_STATIC_INFO: port_name=port-channel200,if_index:0x160000c7,ltl=2597,slot=0,

nxos_port=02,dmod=0,dpid=3,

/32 Host Entry

Verify HW-Programming on LC or FM ? Case 2

N9K# show ip route 13.13.13.10

IP Route Table for VRF "default”

13.13.13.8/30, ubest/mbest: 1/0

*via 13.13.13.6, Eth6/52, [110/41],

00:22:29, ospf-10, intra

N9K# show forwarding route 13.13.13.10 module 21

IPv4 routes for table default/base

Prefix | Next-hop Interface | Labels

13.13.13.8/30 13.13.13.6 Ethernet6/52

This is not /32 host Route.

Packet forwarding decision responsibility is of the Fabric Module

ALL FM will be programmed

with this Route

Line Card Punting Packets to Fabric For LPM ?

N9K# show hardware internal forwarding adjacency statistics default-route mod 6

Module:6 Unit:0

Traffic matched adjacency for default route (destined to FM):

Unicast: Packets 148 Bytes 13382

N9K# bcm-shell mod 6 "0:l3 defip show"

Unit 0, Total Number of DEFIP entries: 12288

# VRF Net addr Next Hop Mac INTF MODID PORT PRIO CLASS HIT VLAN

3072Override 0.0.0.0/0 00:00:00:00:00:00 149149 0 0 0 0 y

N9K# bcm-shell mod 6 "l3 egress show" | inc 149149

Entry Mac Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop

149149 00:12:12:12:12:12 4095 8189 1 100 -1 no no

Mod 100 is assign to Fabric Module

Longest Prefix Match on Fabric Module

N9K# bcm-shell mod 22 "l3 defip show" | grep 13.13.13.8

# VRF Net addr Next Hop Mac INTF MODID PORT PRIO CLASS HIT VLAN

196620 1 13.13.13.8/30 00:00:00:00:00:00 100008 0 0 0 0 n

N9K# bcm-shell mod 22 "l3 egress show" | grep 100008

Entry Mac Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop

100008 88:f0:31:bf:ad:17 4095 4520 10 17 -1 no no

N9K# show system internal ethpm info interface eth 6/52 | grep dmod

IF_STATIC_INFO:

port_name=Ethernet6/52,if_index0x1a286600,ltl=40856,slot=5,nxos_port=51,

dmod=17,dpid=10,unit=1,

Mac add used for rewrite

ECMP Route ValidationN9K#show ip route 10.164.112.22

10.164.112.22/32, ubest/mbest: 2/0

*via 13.13.13.14, Eth6/33, [110/5], 01:11:55, ospf-10, intra

*via 13.13.13.18, Eth6/34, [110/5], 01:11:55, ospf-10, intra

N9K#sh routing hash 13.13.13.2 10.164.112.22 mod 6

Hashing to path *13.13.13.18

Out Interface: Eth6/34

N9K#bcm-shell mod 6 "0:l3 l3table show" | grep 10.164.112.22

Entry VRF IP address Mac Address INTF MOD PORT CLASS HIT

17 1 10.164.112.22 00:00:00:00:00:00 200256 0 0 0 n (ECMP)

N9K#bcm-shell mod 6 "l3

multipath show"

Multipath Egress Object 200256

Interfaces: 100008 100010

Follow same steps demonstrated for /32 Host entry to learn about Interface in multipath show cli

Multi-Path

Use Tools From Toolkit

• ELAM- IF Line Card has North Star

module-6# debug platform internal ns elam asic 1

module-6(NS-elam)# trigger init egress in-select 3 out-select 5

module-6(NS-elam-insel3)# set outer ipv4 dst_ip 13.13.13.10

• Packet Tracer- For All FM and LC having T2N9K# test packet-tracer dst-ip 13.13.13.10 src-ip 13.13.13.1 detail-fp

• Flex Counter- Check Adjacency hit counterN9K# test hardware internal adjacency statistics nexthop ipv4 13.13.13.6

interface ethernet 6/52 enable

• Consistency Checker

show consistency-checker forwarding ipv4 unicast

show tech-support forwarding l3 unicast

show tech-support adjmgr

show tech routing unicast

Virtual Port-Channel-vPC

• Allow a single device to use a port channel across two upstream switches

• Eliminate STP blocked ports

• Dual-homed server operate in active-active mode

• HSRP-Both active and standby peers forward packets-ARP response by Active

• Configuration steps Same as other Nexus Products

Logical Topology with vPC

Case:1 All vPC Leg UP

MCT-1/1, 4/1

N9k1 N9k2

vPC20vPC10

Eth4/18

Eth6/20

Keep Alive

Eth4/18

Eth6/20

SVI10

10.10.10.1/24

SVI-Mac 78da.6e71.9a3f

Standby 10.10.10.3

HSRP-Mac 0000.0c07.ac0a

SVI20

SVI-mac 78da.6e71.9a3f

10.10.20.1/24

Standby 10.10.20.3

HSRP-Mac 0000.0c07.ac14

Switch-A Switch-B

Vlan-10 Vlan-20

10.10.10.x/24 20.20.20.x/24

HOST-A HOST-B

SVI10

10.10.10.2/24

SVI-mac 003a.99fc.dd7f

Standby 10.10.10.3

HSRP-Mac 0000.0c07.ac0a

SVI20

SVI-mac 003a.99fc.dd7f

10.10.20.2/24

Standby 10.10.20.3

HSRP-Mac 0000.0c07.ac14

Scenario: Traffic of a Host in Vlan 10 connected to Switch-A hash to N9K1 to reach Host in Vlan 20

connected to Switch-B

PC1-PeerLink

vPC Peer Link =Eth1/1,4/1

vPC-Router MAC Programming Check

• Both Active and Standby Peer responsible for L3 switching

• Virtual Mac address must be programmed in Hardware on Both peers

Interface Grp Prio P State Active addr Standby addr Group addr

Vlan10 10 100 Active 10.10.10.2 local 10.10.10.3

N9K1# bcm-shell mod 4 "0:d chg my_station_tcam" | grep

VLAN_ID=0xa

VLAN_ID=0xa,VALID=1, MAC_ADDR=0xc07ac0a,

Interface Grp Prio P State Active addr Standby addr Group addr

Vlan10 10 100 Standby 10.10.10.2 local 10.10.10.3

N9K2# bcm-shell mod 4 "0:d chg my_station_tcam" | grep

VLAN_ID=0xa

VLAN_ID=0xa,VALID=1, MAC_ADDR=0xc07ac0a,

vPC Peer Gateway Programming Check

• Are N9K’s Configured with Peer-Gateway

N9K1-SJ# show mac address-table vlan 10 | in G

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

G 10 0000.0c07.ac0a static - F F vPC Peer-Link(R)

G 10 003a.99fc.dd7f static - F F sup-eth1(R) N9K2 SVI MAC

G 10 78da.6e71.9a3f static - F F vPC Peer-Link®

N9K# bcm-shell mod 4 "0:d chg my_station_tcam" | egrep 0x003a99fcdd7f

MY_STATION_TCAM.ipipe0[0]:

<VALID=1,MAC_ADDR_MASK=0xffffffffffff,MAC_ADDR=0x003a99fcdd7f,KEY=0x00000000003

a99fcdd7f,IPV6_TERMINATION_ALLOWED=1,IPV4_TERMINATION_ALLOWED=1,DATA=0x38,ARP_R

ARP_TERMINATION_ALLOWED=1>

vPC Check For Traffic Ingressing Peer LinkEgress Block Mask

• vPC Check-Traffic from Peer Link should Not L2/L3 Switch with local and remote Legs up

N9K1# show vpc brief | grep Po

id Port Status Active vlans

1 Po1 up 10-20

id Port Status Consistency Reason Activevlans

10 Po10 up success success 10-20

20 Po20 up success success 10-20

N9K2# show vpc brief | grep Po

id Port Status Active vlans

1 Po1 up 10-20

id Port Status Consistency Reason Activevlans

10 Po10 up success success 10-20

20 Po20 up success success 10-20

MCT-1/1, 4/1

N9k1 N9k2

vPC20vPC10

Eth4/18

Eth6/20

Keep Alive

Eth4/18

Eth6/20

Switch-A Switch-B

PC1-PeerLink

MCT-1/1, 4/1

N9k1 N9k2

vPC20vPC10

Eth4/18

Eth6/20

Keep Alive

Eth4/18

Eth6/20

Switch-A Switch-B

PC1-PeerLink

vPC Check for Traffic Ingressing Peer Link (Cont’d)

N9K1#show port-ch summary | in Po

Group Port- Type Protocol Member Ports

1 Po1(SU) Eth LACP Eth1/1(P)Eth4/1(P)

10 Po10(SU)Eth LACP Eth4/18(P)

20 Po20(SU)Eth LACP Eth6/20(P)

N9K1# show system internal vpcm info mask

module 6

Masked ports for Module 6, Unit 0:

[Src Port None]: Eth6/20

[Src Port Eth1/1]: Eth6/20

[Src Port Eth4/1]: Eth6/20

Masked ports for Module 6, Unit 1:

Traffic Ingressing on Eth1/1 and

Eth4/1 will not exit Eth 6/20

ACL redirect logic for routed packets-vPC Leg Down

• Redirect ACL installed to redirect routed packets for the vPC for which local interface goes down

• Mac address learned from vPC points virtual port

MCT-1/1, 4/1

N9k1 N9k2

vPC20vPC10

Eth4/18

Eth6/20

Keep Alive

Eth4/18

Eth6/20

Switch-A Switch-B

PC1-PeerLink

Link Down

N9K1# show hardware access-list tcam region | grep vpc

VPC Convergence [vpc-convergence] size = 512

N9K1# sh mac address-table address30f7.0d9b.d401

VLAN MAC Address Type age Secure NTFY Ports

20 30f7.0d9b.d401 dynamic 0 F F vPC Peer-Link

ACL redirect logic for routed packets-vPC Leg Down

• On N9K1 traffic entering Eth6/20 after L3 switch should egress Peer Link

• N9K2 Should not drop traffic entering Peer link and forward traffic out to Eth 4/8

MCT-1/1, 4/1

N9k1 N9k2

vPC20vPC10

Eth4/18

Eth6/20

Keep Alive

Eth4/18

Eth6/20

Switch-A Switch-B

PC1-PeerLink

Ln Down

N9K# bcm-shell module 6 "fp show group 57”

InPorts->L3Routable

DstTrunk

Offset: 213 Width: 16

DATA=0x00008003

action={act=RedirectTrunk, param0=1(0x1) Trunk-id of vPC Peerlink

Trunk-id of “3” Down vPC

ACL redirect logic for routed packets-Verify TrunkID

N9Ka# show system internal ethpm info int port-channel1 | grep dpid

IF_STATIC_INFO: port_name=port-channel1,if_index:0x16000000,ltl=2595,slot=95

dpid=1,unit=0,queue=0,xbar_unitbmp=0x0 ns_pid=0

N9508a-SJ# show system internal ethpm info int port-channel10 | grep dpid

IF_STATIC_INFO: port_name=port-channel10,if_index:0x16000000,ltl=2595,slot=95

dpid=3,unit=0,queue=0,xbar_unitbmp=0x0 ns_pid=0

show tech-support vPC

show tech-support cfs

show tech-support port-channel

Some important info to capture

ACL redirect logic for routed packets-Verify TrunkIDNX-OS -7.0(3)I1(2)

N9508a-SJ# show system internal access-list vpc-convergence mod 6

------------------------------------------------------------

VPC Convergence Entries

------------------------------------------------------------

Instance: 0

==========

Ingress:

----------

Entry-ID DstTrunk-GID RedirectTrunk-GID Packet-Count

------------------------------------------------------------------------

1539 3 1 6082015

Trunk-id of “3” Down vPC

Trunk-id of vPC Peerlink

Nexus9000 Specific Limitation and Goodies

Email from Nexus9000 To Cisco SR

• Commands output directly sent to email address

• Information from Nexus9000 Can be directly attached to Service Request.

• Information is sent as body to email- not as attachment

N9K(config)# email

N9K(config-email)# smtp

N9K(config-email)# smtp-host 173.37.37.37

N9K(config-email)# from N9508a-sj@cisco.com

N9K(config-email)# smtp-port 25

show run | email subject <SR-number> attach@cisco.com

Bash Support !!!!

• Goes beyond what standard CLI can provide

• Customers demand more capabilities/freedom Creativity

• Feature: bash-shell

• User Role: dev-ops or network-admin or vdc-admin*

• Strongly recommended: Some experience with shell/Linux-Use with

extreme care

Broadcom ASIC shell access on the Nexus 9000 !!!

• The Nexus 9000 is based largely on the Broadcom Trident II ASIC-Known as T2

• The modular unit Fabric Modules (FM) and Line Cards (LC) each contain multiple

instances of the T2 ASIC, as well as the TOR (top of rack) units

• Access is provided to each and every instance of the T2 ASIC

• No additional license is required to access the bcm-shell

• Permitted by default role network-admin

• Role based access control (RBAC) can be used to limit user access

• Accounting log available for BCM activity

BCM Access some Examples

N9K# bcm-shell mod 6 "show unit"

Unit 0 chip BCM56852_A2 (current)

Unit 1 chip BCM56852_A2

N9K# bcm-shell mod 6 "ps" | in 19

xe19 up 1G FD SW Yes Disable None FA XGMII 1582

N9K# show accounting log | last 2

Mon Apr 20 08:31:52 2015:type=update:id=console0:user=admin:cmd=bcm-shell

module 6 "show unit" (SUCCESS)

Mon Apr 20 08:32:14 2015:type=update:id=console0:user=admin:cmd=bcm-shell

module 6 "ps" | in 19 (SUCCESS)

QSPF

Ports

QSPF

PortsF

P

01

F

P

02

F

P

03

F

P

04

F

P

05

F

P

06

F

P

07

F

P

08

F

P

09

F

P

10

F

P

11

F

P

12

F

P

13

F

P

14

F

P

15

F

P

16

F

P

17

F

P

18

F

P

19

F

P

20

F

P

21

F

P

22

F

P

23

F

P

24

T2

Instance 0T2

Instance 1

Eth1/1 Eth1/24

Xe0 Xe0

hg0 hg11

Xe11

Eth1/12

Xe11

Eth1/13

hg0 hg11

BCM Access some Examples (Cont’d)

N9K# bcm-shell mod 21 "config show l3"

l3_alpm_enable=2

l3_max_ecmp_mode=1

l3_mem_entries=16384

N9K# bcm-shell mod 4 "config show l2 ”

l2xmsg_hostbuf_size=16384

l2_mem_entries=98304

Python !!!!

• Python is - Established, Modern and Powerful, Clean, lots of libraries, liberal

license

• Perl is available in gdb images only – not available in final images

• Tcl is there but no one uses it in NX-OS

• The license that Python has (GPL-Like with very few restrictions on modification,

distribution and commercial use) make it very attractive to embed and distribute

• On the box applications that can currently use Python scripts

• Embedded Event Manager

• Power On Auto Provisioning (POAP)

• Create your own scripts that are like “Super commands”

• Create your own command modifiers – the things that act on commands applied with a

pipe “|”

Python-Continued

• There are two Python environments on the N9000

• One executed from VSH

• One executed from Bash

• Both run in their own forked process

• The main differences comes from the environment that they get initialized into

• These differences between them should be minimal

• There is a sandbox that should primarily contain lower privileged users

• Network-admin users get basically a “pure” 2.7.5 python environment

• That sandbox mostly applies to lower privileged users, they may be prevented from doing certain things in python

• Also prevents file operations on files outside of bootflash

Python-Example

N9K# python

Python 2.7.5 (default, Oct 8 2013, 23:59:43)

[GCC 4.6.3] on linux2

Type "help", "copyright", "credits" or

"license" for more information.

>>>

N9K# run bash python

Python 2.7.5 (default, Oct 8 2013, 23:59:43)

[GCC 4.6.3] on linux2

Type "help", "copyright", "credits" or

"license" for more information.

>>>

switch between VSH and the

Interpreter (Bash 1)

switching between VSH and Python

Python Script Example

Why Patching?

Begin Code Test &

Qualification Cycle

Target Deployment

Bug Found, Diagnose, Root

Cause

Defect Resolved, integrated

into Maint.

Maint. Released

Restart Qual Cycle Actual Deployment

6 Months

10 Months

Many customers spend extensive time and effort to test and qualify software prior to deployment. In today’s

environments, if a defect is found, effectively root-caused, and integrated, since it is rolled out through a

maintenance release, customers would need to restart their qualification cycle, wasting time, and pushing out

deployment dates…

NX-OS Image Patching

Begin Code Test &

Qualification Cycle

Target Deployment

Bug Found, Diagnose, Root

Cause

Defect Resolved, Patch

Released

Continue Qual

With additional tests Actual Deployment

6 Months

7 Months

The Nexus9000 Standalone platforms introduces new patching capabilities that allows specific defects to be

rolled out in an independent package that can be applied to existing base software binaries. This will help

reduce customer code certification times, leading to greater customer satisfaction.

Patching Overview

• NXOS platforms release major versions when introducing new features and engineering special builds to provide bug fixes.

• The new goal will be to allow customers to deploy patches for specific fixes only without affecting the data plane of the device.

• The patching architecture comes from IOS XR (SMU – Software Maintenance Upgrade) used to deliver Quick, Effective and Focused patches for specific sections of code.

• Both binaries and libraries can be patched.

• Supervisors and Line Card services can be patched.

• Software patching will leverage process restart/reload or ISSU

Patch Uninstall Workflow - Detailed

• User invokes “install deactivate <patch_name>”

• System manager gracefully shuts down each impacted process

• Softlinks are changed from active SMU to one in backup folder (if present).

• Relevant SMU is removed from the /var/installer/activated/SMU directory.

• System Manager triggers restart of impacted processes

• (Optional) “install remove” deletes the patch from the local repository

CLI Commands – Patch Install

Command Syntax Function Notes

Install add install add <uri> [activate] Download patch from URI and add

patch to repository.

Only one patch can be added at

a time. Optionally can activate

patch in this step.

Install remove install remove [<package> |

inactive]

User can remove only non-

activated patches

Confirmation y/n will be prompted

Install activate install activate <package> [test] Installs a patch from the local

repository. If not present, an error

will be returned.

Only one patch can be activated

at a time. No show commands

permitted during operation.

Install deactivate install deactivate <package> Uninstall patch and move it to non-

activated repository

Only one patch can be

deactivated at a time. **Patches

must have no other patch

dependencies

Install commit install commit Preserves all activated patches

across reloads.

Activated patches are committed

to a list kept in the patch

repository

CLI Commands – Show Commands

Command Function Sample

show install request Shows current install operation along

with time stamp, package name,

initiating user and % complete.

Fri May 10 09:06:55.921 UTC

Install operation 13 '(admin) ‘install activate n9000-dk.6.0.2.U1.1.CSCuf08219.bin’

Started by user 'cisco' via CLI at 09:06:48 UTC Fri May 10 2013The operation is 10% complete

show install log [id | detail

| from | last | reverse]

Shows user information on previous

installation operations. Optional [detail]

command for verbose information.

Install operation 1 by user ‘admin’ at Tue Sep 28 01:37:02 2004:

install commit

Operation completed successfully

Install operation 2 by user ‘admin’ at Mon Oct 18 17:26:36 2004:

install add tftp://10.52.241.252/bcarter/n3000-uk9.6.0.2.U1.1.CSCuf08219.bin

Operation completed successfully

Install operation 7 by user ‘lab’ at Mon Oct 18 17:31:13 2004:

install activate n3000-uk9.6.0.2.U1.1.CSCuf08219’Operation failed because service failed to come up.

show install active [on-

reload]

Displays boot images and active or

committed patches

switch# show install active

Boot Images:

Kickstart Image: bootflash:/n9000-dk.6.1.234.gbin

System Image: package:/isanboot/bin/images/sys

Active Packages:

n9000-dk.6.1.1.CSCui56298.bin

CLI Commands – Show Commands (Cont’d)

Command Function Sample

show install inactive [on-

reload]

Shows patches in the repository not

yet activated

switch# show install inactive

Boot Images:

Image: bootflash:/inseor.6.1.1.234.gbin

System Image: package:/isanboot/bin/images/sys

Inactive Packages:switch#

show install pkg-info

<package>

Shows details of a specific patch.

Requires that patch has been added

using ‘install add’ first.

switch# show install pkg-info n9000-dk.6.1.1.CSCui56298.bin

Contents of Package file 'n9000-dk.6.1.1.CSCui56298.bin':

Expiry date : Jan 19, 2015 02:55:56 UTC

Uncompressed size : 17892613

Vendor : Cisco Systems

Desc : Bug Fix for CDET: CSCui56298

Build : Built on Wed May 10 08:04:58 UTC 2013

Source : By n9k-infra-bld

Platform: Nexus-9000.

Supersedes: n9000-uk9.6.1.1.U1.1.CSCuf09119, n9000-uk9.6.1.1.U1.1.CSCuf02229

Pre-requisite: n9000-uk9.6.1.1.U1.1.CSCuf09219Restart information: BGP process restart.

Sample Patch Install – Copy Patch to Switch

N9K# copy

scp://sdn@172.18.217.42/home/sdn/n9k/inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin

bootflash:

Enter vrf (If no input, current vrf 'default' is considered): management

sdn@172.18.217.42's password:

inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin 100% 233KB

232.7KB/s 00:01

Copy complete, now saving to disk (please wait)...

N9508#

N9508# dir | grep .gbin

238230 Jan 15 10:52:31 2014inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin

N9508#

Sample Patch Install – Add patch to repository & verify

N9K# install add bootflash:inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbinInstall operation 19 completed successfully at Wed Jan 15 10:55:14 2014N9508#

N9K# show install packages

-----------------------------------------------------------

inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin inactive-commit

Modules

Module #27: inactive-commit

Module #28: inactive-commit

-----------------------------------------------------------

N9K# show install inactive

Inactive Packages:

inseor_CSCuxP1fix.6.1.2.I1.2.CSCab00001.gbin

N9K#

Important Limitations

• For every Feature please review Guidelines and Limitations

• Cisco Nexus 9000 Series NX-OS Verified Scalability Guide

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6-x/scalability/guide_34/b_Cisco_Nexus_9000_Series_NX-OS_Verified_Scalability_Guide_612I34.html

• Only one software image (called nx-os) is required to load the Cisco NX-OS operating system.

• EPLD Upgrade are recommended but are not mandatory

• User Configured MAC address for SVI- Packets will not be flooded if Layer 2 Adjacency is missing

• Diagnostic-The Port Loop back and Boot up Port Loop back tests are not supported

• ASIC Memory-NS test is applicable only for the N9K-X9564PX and N9K-X9564TX line cards.

• Priority flow control (PFC) is supported on Cisco Nexus 9500 Series switches with the N9K-X9636PQ line card.

• FEX is supported only on the Cisco Nexus 9372PX and 9396PX switches.

• Cisco Nexus 9500 Series Switch can run in 8-queue mode only if all of its line cards are capable of running 8-queue mode.

Participate in the “My Favorite Speaker” Contest

• Promote your favorite speaker through Twitter and you could win $200 of Cisco Press products (@CiscoPress)

• Send a tweet and include

• Your favorite speaker’s Twitter handle <Speaker—enter your Twitter handle here>

• Two hashtags: #CLUS #MyFavoriteSpeaker

• You can submit an entry for more than one of your “favorite” speakers

• Don’t forget to follow @CiscoLive and @CiscoPress

• View the official rules at http://bit.ly/CLUSwin

Promote Your Favorite Speaker and You Could Be a Winner

Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys though the Cisco Live mobile app or your computer on Cisco Live Connect.

Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Table Topics

• Meet the Engineer 1:1 meetings

• Related sessions

Thank you

Backup Slides

Backup Slides !!!!

Fabric Module

NFE

Fabric Module for Nexus 9504

NFE NFE

Fabric Module for Nexus 9508

NFE NFE NFE NFE

Fabric Module for Nexus 9516

Chassis Type Nexus 9504 Nexus 9508 Nexus 9516

NFEs per Fabric Module 1 2 4

Nexus 9500 Platform FRU- Line CardConnect to Fabric Modules

Connect to Hosts or

Network

NFE 1

ALE 1

12 x 42 Gbps

Network

Interfaces

12 x 42

Gbps

18x 40

Gbps

Ethern

et

18x 40Gbps

NFE NFE

12 x 40 Gbps12 x 40 Gbps

NFE

12 x 40 Gbps

N

F

E

Fabric 1

N

F

E

N

F

E

Fabric 2

N

F

E

N

F

E

Fabric 3

N

F

E

N

F

E

Fabric 4

N

F

E

N

F

E

Fabric 5

N

F

E

N

F

E

Fabric 6

N

F

E

1 x 42

Gbps

1 x 42

Gbps

N9K-X9636PQ

HG Ports HG Ports HG Ports

QSPF PortsQSPF PortsQSPF Ports

FP

01

FP

02

FP

03

FP

04

FP

05

FP

06

FP

07

FP

08

FP

09

FP

10

FP

11

FP

12

FP

13

FP

14

FP

15

FP

16

FP

17

FP

18

FP

19

FP

20

FP

21

FP

22

FP

23

FP

24FP

25

FP

26

FP

27

FP

28

FP

29

FP

30

FP

31

FP

32

FP

33

FP

34

FP

35

FP

36

FM3 FM4FM2FM1 FM5 FM6

T2Instance 0

T2Instance 1

T2Instance 2

N9K-X9464PX

HG Ports HG Ports

10G SFP+ Ports 40G QSFP

FM3FM2 FM4 FM6

MUX1-2 MUX3-4

T2

FP

1

FP

2

FP

3

FP

4

FP

5

FP

6

FP

7

FP

8

FP

9

FP

10

FP

11

FP

12

FP

13

FP

14

FP

15

FP

16

FP

17

FP

18

FP

19

FP

20

FP

21

FP

22

FP

23

FP

24

FP

25

FP

26

FP

27

FP

28

FP

29

FP

30

FP

31

FP

32

FP

33

FP

34

FP

35

FP

36

FP

37

FP

38

FP

39

FP

40

FP

41

FP

42

FP

43

FP

44

FP

45

FP

46

FP

47

FP

48

FP

49

FP

50

FP

51

FP

52

N9K-X9464TX

HG Ports HG Ports

100/1000/10000 T Ports 40G QSFP

FM3FM2 FM4 FM6

MUX1-2 MUX3-4

T2

10G

PHY

10G

PHY

10G

PHY

10G

PHY

10G

PHY

10G

PHY

10G

PHY

10G

PHY

10G

PHY

10G

PHY

10G

PHY

10G

PHYFP

49

FP

50

FP

51

FP

52FP

1

FP

2

FP

3

FP

4

FP

5

FP

6

FP

7

FP

8

FP

9

FP

10

FP

11

FP

12

FP

13

FP

14

FP

15

FP

16

FP

17

FP

18

FP

19

FP

20

FP

21

FP

22

FP

23

FP

24

FP

25

FP

26

FP

27

FP

28

FP

29

FP

30

FP

31

FP

32

FP

33

FP

34

FP

35

FP

36

FP

37

FP

38

FP

39

FP

40

FP

41

FP

42

FP

43

FP

44

FP

45

FP

46

FP

47

FP

48

N9K-X9432PQ

QSPF PortsQSPF Ports

HG Ports HG Ports

FP

01

FP

02

FP

03

FP

04

FP

05

FP

06

FP

07

FP

08

FP

09

FP

10

FP

11

FP

12FP

21

FP

22

FP

23

FP

24

FP

25

FP

26

FP

27

FP

28

FP

29

FP

30

FP

31

FP

32

FM3 FM4FM2 FM6

T2Instance 0

T2Instance 2

FP

13

FP

14

FP

15

FP

16

FP

17

FP

18

FP

19

FP

20

N9K-X9564PQ

10G SFP+ Ports

40G QSFP

HG MUX1 HG MUX3

FP

49

FP

50

FP

51

FP

52FP

1

FP

2

FP

3

FP

4

FP

5

FP

6

FP

7

FP

8

FP

9

FP

10

FP

11

FP

12

FP

13

FP

14

FP

15

FP

16

FP

17

FP

18

FP

19

FP

20

FP

21

FP

22

FP

23

FP

24

FP

25

FP

26

FP

27

FP

28

FP

29

FP

30

FP

31

FP

32

FP

33

FP

34

FP

35

FP

36

FP

37

FP

38

FP

39

FP

40

FP

41

FP

42

FP

43

FP

44

FP

45

FP

46

FP

47

FP

48

Northstar 1

Warpcore

MF Port

7-5 2-0 31-29 26-24

T2

7-5 26-24

0-2 3-5 6-8 9-11

FM4 FM3FM5FM6 FM2 FM1

HG MUX4 HG MUX2 HG MUX5 HG MUX6

MN Port0 1 2 3 4 5 6 7 8 9 10 11

Northstar 2MF Port

0-2 9-11

MN Port0 1 2 3 4 5 6 7 8 9 10 11

T2

7-5 2-0 31-29 26-24

N9K-X9564TX

100/1000/10000 T Ports40G QSFP

HG MUX1 HG MUX3

T2

FP

49

FP

50

FP

51

FP

52

10G

PHY

10G

PHY

10G

PHY

10G

PHY

10G

PHY

10G

PHY

10G

PHY

10G

PHY

10G

PHY

10G

PHY

10G

PHY

10G

PHY

FP

1

FP

2

FP

3

FP

4

FP

5

FP

6

FP

7

FP

8

FP

9

FP

10

FP

11

FP

12

FP

13

FP

14

FP

15

FP

16

FP

17

FP

18

FP

19

FP

20

FP

21

FP

22

FP

23

FP

24

FP

25

FP

26

FP

27

FP

28

FP

29

FP

30

FP

31

FP

32

FP

33

FP

34

FP

35

FP

36

FP

37

FP

38

FP

39

FP

40

FP

41

FP

42

FP

43

FP

44

FP

45

FP

46

FP

47

FP

48

Northstar 1MF Port

7-5 2-0 31-29 26-24

T2

7-5 26-24

0-2 3-5 6-8 9-11

FM4 FM3FM5FM6 FM2 FM1

HG MUX4 HG MUX2 HG MUX5 HG MUX6

MN Port0 1 2 3 4 5 6 7 8 9 10 11

Northstar 2MF Port

0-2 9-11

MN Port0 1 2 3 4 5 6 7 8 9 10 11

Multicast L3 Forwarding

• Before hardware can forward any Multicast packets, forwarding information has to propagate from Sup to the LC

• Several layers are to be verified:

MRIB (control-plane is created here)

MFDM PI /PD (platform independent & forwarding information)

• MFIB-IPFIB

• IP FIB process programs hardware:FIB Table contains (*,G) and (S,G) forwarding entries and RPF informationGROUP table contains forwarding and pointers replication information (pointers to MC VLAN)MC VLAN tables contain replication information (~OIF lists)

Hardware (packets are forwarded here) & SDK

Supervisor

MRIB

MF DM

IP FIB

IGMPPIM MSDP

T2

FIB Table MC VLAN Table

IPMC_GR

Line Card

L2/L3 Multicast Packet WalkFabric Module

Trident II

Parser

Network Interfaces

L2/L3

Lookup &

pkt rewrite

10GE 40GE

EACL

Egress Q

Trident II

Parser

L2/L3 Lookup &

Pkt rewrite

EACL

Egress Q

Trident IIIACL

Traffic Classification& Remarking

IACL

Traffic

Classification&

Remarking

Network Interfaces

10GE 40GE

Lkup in Host Table

& L2 Table

Lookup to resolve egr.

modules;

Sends one copy to each

egr. module;

Examines ingress

packet. Get packet

headers for

processing.

Lookup for local

receiving ports;

replicate pkts onto

those ports.L2/L3 mcast lookup;

Replicate pckts to local

receiving ports;

Send 1 copy to fabric

module;

Multicast L3 Forwarding-MRIB

N9K# show ip mroute 239.10.10.10 shared-tree

IP Multicast Routing Table for VRF "default”

(*, 239.10.10.10/32), uptime: 00:23:32, ip pim

Incoming interface: Ethernet6/1, RPF nbr:

13.13.13.1

Outgoing interface list: (count: 1)

Ethernet6/52, uptime: 00:22:42, pim

Supervisor

MRIB

MF DM

IP FIB

PIM MSDPIGMP

Multicast L3 Forwarding-mFDM PI-Supervisor

N9K# show forwarding distribution multicast outgoing-

interface-list l3 1

Outgoing Interface List Index: 1

Reference Count: 4

Platform Index: 0xb00001

Number of Outgoing Interfaces: 1 t6/52

N9K# show forwarding distribution ip multicast route group 239.10.10.10 source

13.13.13.14 | in 13|Index

(13.13.13.14/32, 239.10.10.10/32), RPF Interface: Ethernet6/1, flags:

Outgoing Interface List Index: 1

Supervisor

MRIB

MF DM

IP FIB

PIM MSDPIGMP

Multicast L3 Forwarding IPFIB-Line card

N9K# show forwarding ip multicast route group 239.10.10.10 source

13.13.13.14 mod 6 | inc 239|Eth

(13.13.13.14/32, 239.10.10.10/32), RPF Interface: Ethernet6/1, flags:

Outgoing Interface List Index: 1

Outgoing Interface List Index: 0x1

Ethernet6/52

T2

FIB Table MC VLAN Table

IPMC_GR

Line Card

Mod 6 is N9K-X9564TXTo reach Ethernet 6/52 which is on NS from front port of T2,Packets need to cross Fabric module

Multicast L3 Forwarding Entries on LC –BCM Shell

N9K# bcm-shell mod 6 "ipmc table show"

SRC IP ADDRESS MC IP ADDRESS MC GROUP VID VRF COS HWIDX CLASS HIT

13.13.13.14 239.10.10.10 0x2000007 0 1 0 75680 1 no

0.0.0.0 239.10.10.10 0x2000007 0 1 0 86578 2 no

N9K#bcm-shell module 6 "mc show group=0x2000007"

Group 0x2000007 (L3)

port hg0, encap id -1

-------snip------------

port hg11, encap id -1

T2

FIB Table MC VLAN

Table IPMC_GR

Line Card

Traffic spared to Hig towards Fabric

Multicast L3 Forwarding Entries on LC –BCM Shell

N9K# bcm-shell mod 6 " search l3_entry_ipv4_multicast group_ip_addr=0xef0a0a0a

source_ip_addr=0x0d0d0d0e”

L3_ENTRY_IPV4_MULTICAST.ipipe0[75680]:

SOURCE_IP_ADDR=0xd0d0d0e,

GROUP_IP_ADDR=0xef0a0a0a,

L3MC_INDEX=7

N9K# bcm-shell mod 6 " dum chg l3_entry_ipv4_multicast 75680”

IPV4MC:EXPECTED_L3_IIF=0x112e,

N9K# show system internal eltm info interface ethernet 6/1 | in LIF

cr_flags = INTF LIF , LIF = 4398 (0x112e), LTL = 40959 (0x9fff) (S 0x0, P 0x0)

T2

FIB Table MC VLAN Table

IPMC_GR

Line Card

show tech-support multicast`

show tech-support forwarding multicast

IGM SnoopingForwarding programming in vPC Scenario

• IGMP Process Provides both Layer 3 IGMP Processing , and Layer 2 IGMP snooping functionality

• Receivers use IGMP (Internet Group Management Protocol) to report their multicast group

Membership to router

• Layer 2 IGMP Snooping functions of IGMP process include processing snooped multicast router

Packets Including IGMP reports and leaves sent by receiver

• Once the group membership is learned , the Supervisor Engine informs I/O modules , which

program Hardware

• This will Constrain data-plane multicast packets to only those ports with multicast routeror interested

receivers in HW

IGMP Snooping continued…

• BCM on FM are in Mode 4. This will have L2 Table size of 32K & L3 Host Table 16K

• L3 Host table will be used to program (*,G) /(S,G) entry. This will will accommodate

maximum of 8K entry.

• MFDM sends two OIF List information to MFIB. One for LC (S,G) OIF List and other for

FM ( Mac, Group) OIF List in PIM disable Vlan.

• MFIB will use (S,G) OIF list to program LC and Mac Group to Program FM in 32K L2 Table.

• If PIM is enable FM can accommodate 8K(VRF, S,G) and will program Hardware.

• Address aliasing is possible because on FM we use L2 table to program Mac Group information

IGMP Snooping (Cont’d)• With vPC IGMP will have knowledge of multi chassis Ether Channel trunk (MCT) interface.

• When one of the vPC peer receives IGMP join , it will sync up this with peer over MCT link

using cFS-Cisco Fabric Services over Ethernet .

• Duplication of traffic crossing MCT is avoided using Port block Mask

• VPC Support PIM-SM Only

• For source in VPC domain – dual Forwarders are used

• For Source in Layer 3 Cloud , Unicast best metric determines active forwarder

• VPC Operational Primary in case of tie. CFS used to negotiate active Forwarder role

Configuration-IGMP Snooping enable by default

Nexus9508-13# sh ip igm snooping vlan 103

IGMP Snooping information for vlan 103

IGMP snooping enabled

Lookup mode: IP

Optimised Multicast Flood (OMF) enabled

IGMP querier present, address: 10.10.103.5, version: 2, i/f Po30

Nexus9508-13# sh ip igm snooping vlan 100

IGMP Snooping information for vlan 100

IGMP snooping enabled

Lookup mode: IP

Optimised Multicast Flood (OMF) enabled

IGMP querier present, address: 192.168.100.2, version: 2, i/f Vlan100

Querier interval: 125 secs

Querier last member query interval: 1 secs

Reference Topology for Troubleshooting

N35KEth 1/17,Eth 1/19 , Eth 1/33-34

N9508-12 N9508-13

N93k

vPC 35vPC30

vPC Keep Alive

vPC Peer Link PO-10

Ixia 10/2-Source Ixia 10/1-Receiver

Eth1/3/1-4Eth 6/9/1-4

Eth 1/48Eth1/48

Eth 3/1-2 Eth 3/1-2

Eth 1/17-18 ,Eth 1/33-34

IGMP Snooping Troubleshooting• Stream will enter one of the VPC-Peer , Which will get forwarded across Peer link to other VPC Peer

• Both boxes will have (S ,G)

• Upon Creation of (S,G) , VPC Peers negotiate best metric

• Both realize source is VPC-Connected

• Install Entry as Win-Force

• If either peer gets a PIM/IGMP Join for the given source , they both add Interface to OIF

Nexus9508-12(config)# sh ip pim internal vpc rpf-source

PIM vPC RPF-Source Cache for Context "default" - Chassis Role Primary

Source: 192.168.100.10

Pref/Metric: 0/0

Source role: primary

Forwarding state: Win-force (forwarding)

MRIB Forwarding state: forwarding

Nexus9508-13# sh ip pim internal vpc rpf-source

PIM vPC RPF-Source Cache for Context "default" - Chassis Role Secondary

Source: 192.168.100.10

Pref/Metric: 0/0

Source role: secondary

Forwarding state: Win-force (forwarding)

MRIB Forwarding state: forwarding

• IGMP Join from one of the receiver enter one of the VPC Pee.

• This Peer encapsulates IGMP in CFS , sends to other Peer

• Both Peer have identical State

• Both Peer install OIF

• Data traffic flows down to Receiver, also forwarded to other Peer on Peer Link

• Other Peer drop the packet either by PORT BLOCK MASK blocking or no OIF

Nexus9508-ESC-12# sh ip mroute 239.10.10.10 192.168.100.10

IP Multicast Routing Table for VRF "default"

(192.168.100.10/32, 239.10.10.10/32), uptime: 01:00:09, ip pim mrib

Incoming interface: Vlan100, RPF nbr: 192.168.100.10, uptime: 01:00:09, internal

Outgoing interface list: (count: 1)

Vlan101, uptime: 00:59:40, mrib

Nexus9508-ESC-12#

Nexus9508-ESC-13# sh ip mroute 239.10.10.10 192.168.100.10

IP Multicast Routing Table for VRF "default"

(192.168.100.10/32, 239.10.10.10/32), uptime: 04:25:36, ip pim mrib

Incoming interface: Vlan100, RPF nbr: 192.168.100.10, uptime: 04:25:36

Outgoing interface list: (count: 1)

Vlan101, uptime: 02:04:41, mrib

Nexus9508-ESC-13#

vPC Peer receiving Join

Step to verify PI On Supervisor. Verify on Both PeersNexus9508-ESC-12# sh ip igmp groups 239.10.10.10

IGMP Connected Group Membership for VRF "default" - matching Group "239.10.10.10"

Type: S - Static, D - Dynamic, L - Local, T - SSM Translated

Group Address Type Interface Uptime Expires Last Reporter

239.10.10.10 D Vlan101 00:01:23 00:02:56 192.168.101.13

Nexus9508-ESC-12#

Nexus9508-ESC-13# sh ip igmp groups 239.10.10.10

IGMP Connected Group Membership for VRF "default" - matching Group "239.10.10.1

0"

Type: S - Static, D - Dynamic, L - Local, T - SSM Translated

Group Address Type Interface Uptime Expires Last Reporter

239.10.10.10 D Vlan101 00:01:18 00:03:01 192.168.101.13

Nexus9508-ESC-13#

CFS Provide info

Nexus9508-ESC-12# sh ip igmp snooping groups vlan 101 detail

IGMP Snooping group membership for vlan 101

Group addr: 239.10.10.10

Group ver: v2 [old-host-timer: not running]

Last reporter: 192.168.101.10

IGMPv2 member ports:

IGMPv1/v2 memb ports:

Po35 [1 GQ missed], cfs:false, native:true

vPC grp peer-link flag: exclude

M2RIB vPC grp peer-link flag: exclude

Nexus9508-ESC-12#

Nexus9508-ESC-13# sh ip igm snooping groups vlan 101 det

IGMP Snooping group membership for vlan 101

Group addr: 239.10.10.10

Group ver: v2 [old-host-timer: not running]

Last reporter: 192.168.101.10

IGMPv2 member ports:

IGMPv1/v2 memb ports:

Po35 [0 GQ missed], cfs:true, native:false

vPC grp peer-link flag: exclude

M2RIB vPC grp peer-link flag: exclude

Nexus9508-ESC-13#

Verifying Multicast forwarding Distribution Module

Platform Independent On SupervisorNexus9508-ESC-12# sh forwarding distribution multicast route group 239.10.10.10 source 192.168.100.10

(192.168.100.10/32, 239.10.10.10/32), RPF Interface: Vlan100, flags:

Received Packets: 1073 Bytes: 36977

Number of Outgoing Interfaces: 2

Outgoing Interface List Index: 10

Vlan100

( Mem L2 Ports: port-channel10 )

Vlan101

( Mem L2 Ports: port-channel35 )

Note: On shutting down local vpc only, igmp does not send update to mfdm/ipfib to update the mroute state.

That is why you did not see mfdm/ipfib removing local vpc. So if local leg of vPC is down we will still PC in the above output.

Not showing PC 10 for Vlan 101 because of

exclude flag seen while checking igmp

snooping stats.

Verifying Multicast forwarding Distribution Module Platform Independent On Supervisor-(Cont’d)Nexus9508-12# sh forwarding multicast route group 239.10.10.10 source 192.168.100.10 mod 1

(192.168.100.10/32, 239.10.10.10/32), RPF Interface: Vlan100, flags:

Received Packets: 1111 Bytes: 72215

Outgoing Interface List Index: 9

Number of next hops: 2

Outgoing Interface List Index: 9

Vlan: 101

port-channel35

bridged Vlan

port-channel10

Hardware Outgoing Interface List Index: 33554443

Verifying Multicast forwarding Distribution Module

Platform Independent On Supervisor-IGMP-SnoopingNexus9508-12# sh forwarding distribution ip igmp snooping vlan 101 group 239.10.10.10 det

Vlan: 101, Group: 239.10.10.10, Source: 0.0.0.0

Outgoing Interface List Index: 4

Reference Count: 1

Platform Index: 0xa00004

Vpc peer link exclude flag set

Number of Outgoing Interfaces: 2

port-channel10

port-channel35

Nexus9508-13# sh forwarding distribution ip igmp snooping vlan 101 group

239.10.10.10 det

Vlan: 101, Group: 239.10.10.10, Source: 0.0.0.0

Outgoing Interface List Index: 5

Reference Count: 1

Platform Index: 0xa00005

Vpc peer link exclude flag set

Number of Outgoing Interfaces: 2

port-channel10

port-channel35

Verifying Multicast Forwarding Distribution Module

Platform Independent On Supervisor-Snooping Group.Nexus9508-12# sh forwarding distribution l2 multicast mac-based vlan 101

Vlan: 101, Group: 0100.5e0a.0a0a, Source: 0000.0000.0000

Outgoing Interface List Index: 3

Reference Count: 1

Platform Index: 0xa00003

Vpc peer link exclude flag set

Number of Outgoing Interfaces: 2

port-channel10

port-channel35

Nexus9508-13# sh forwarding distribution l2 multicast mac-based vlan 101

Vlan: 101, Group: 0100.5e0a.0a0a, Source: 0000.0000.0000

Outgoing Interface List Index: 8

Reference Count: 1

Platform Index: 0xa00008

Vpc peer link exclude flag set

Number of Outgoing Interfaces: 2

port-channel10

port-channel35

IPFIB on LC for IGMP Snooping programming.Nexus9508--12# sh forwarding multicast route group 239.10.10.10 source 192.168.100.10 mod 1

(192.168.100.10/32, 239.10.10.10/32), RPF Interface: Vlan100, flags:

Received Packets: 5708 Bytes: 371020

Outgoing Interface List Index: 5

Number of next hops: 2

Outgoing Interface List Index: 5

port-channel30 (Vlan: 101)

port-channel10 (bridged)

Hardware Outgoing Interface List Index: 33554441

Nexus9508-13# sh forwarding multicast route group 239.10.10.10 source 192.168.100.10 mod 6

(192.168.100.10/32, 239.10.10.10/32), RPF Interface: Vlan100, flags:

Received Packets: 6798 Bytes: 441870

Outgoing Interface List Index: 19

Number of next hops: 2

Outgoing Interface List Index: 19

port-channel30 (Vlan: 101)

port-channel10 (bridged)

Hardware Outgoing Interface List Index: 33554437

Nexus9508--12# bcm-shell mod 1 "mc show group=33554441"

Executing mc show group=33554441 on bcm shell on module 1

Group 0x2000009 (L3)

port hg0, encap id 400005

port hg1, encap id 400005

port xe10, encap id 21

port xe11, encap id 21

Verifying Hardware Programming

Nexus9508-12# bcm-shell mod 3 "mc show group=33554441"

Executing mc show group=33554441 on bcm shell on module 3

Group 0x2000009 (L3)

port hg0, encap id 400005

port xe0, encap id -1

port xe1, encap id -1

Nexus9508-12# sh system internal eltm info interface vlan 101 | in LIF

cr_flags = INTF VLAN , LIF = 21 (0x15), LTL = -1 (0xffffffff) (S 0x0, P 0x0)

Nexus9508-ESC-12#

If we see encap id a positive #

then it is LIF

If we see encap id = -1 then it is

L2 bridge copy.

Nexus9508-12# bcm-shell module 1 "l2 show" | in MCast

mac=01:00:5e:0a:0a:0a vlan=101 GPORT=0x0 modid=0 port=0 Static Hit MCast=33554435

mac=01:00:5e:0a:0a:14 vlan=100 GPORT=0x0 modid=0 port=0 Static MCast=33554435

Nexus9508-12# sh ip igmp gr vlan 100

From BCM to check what is HW index for given Group

• Static entry of Mcast group

• Hit Bit indicate flow is present

• Mcast Index is where the traffic need to bridge

show tech-support ip igmp snooping

show tech-support ip multicast