VPC & VSS: Operation and Troubleshooting - Add …docshare01.docshare.tips/files/30199/301996596.pdf · VSS on bridging and routing Learn how to troubleshoot ... Cisco Public 7 VSS

BRKCRS-1930

VPC & VSS: Operation and Troubleshooting

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKCRS-1930 2

VSS and VPC

No blocked ports, More usable bandwidth, Load-sharing

Distribution or link failure != network reconvergence

…enable us to build EtherChannel to 2 separate

switches and transform network building block

to this from this …or, logically


Goals

Understand general concepts of VPC on Nexus 7000 and VSS on Catalyst 6500

Study the impact of VPC and VSS on bridging and routing

Learn how to troubleshoot VPC and VSS


Spirit of this session

Simple description on how things work

Special cases

Troubleshooting

More on the topic

Cisco Catalyst Virtual Switching System (BRKCRS-3468)

Advanced Enterprise Campus Design: Virtual Switching System (BRKCRS-3035)

Deploying Virtual Port Channel in NXOS(BRKDCT-2048)

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKCRS-1930

VSS


VSS Agenda

Initialization

Internal redundancy considerations

Spanning Tree

1st hop redundancy

Traffic forwarding

Multicast considerations


VSS

1 active redundant control plane

single config

single point of management

2 active data planes

Standby switch is essentially a

set of additional linecards

Control messages and Data

frames flow between active and

standby via VSL(can be seen as backplane

extension)

Special encapsulation on VSL

frames to carry additional

information

ActiveData Plane

ActiveControl Plane

ActiveData Plane

StandbyControl Plane

MEC

VSL

Dual-Active

detection link

Active Standby

VSS domain


VSS initializationBefore the Virtual Switch domain can become active, the Virtual Switch Link

(VSL) must be brought online to determine Active and Standby roles. The

initialization process essentially consists of 3 steps:

Role Resolution Protocol (RRP) used to determine compatible Hardware and

Software versions to form the VSL as well as determine which switch becomes

Active and Hot Standby from a control plane perspective

LMP LMP

RRPRRP

Link Management Protocol (LMP) used to track and reject Unidirectional Links,

Exchange Chassis ID and other information between the 2 switches

Link Bringup to establish connectivity with remote chassis1

2

3


Troubleshooting VSS: quick sanity check

vss# sh switch virtualSwitch mode : Virtual SwitchVirtual switch domain number : 111Local switch number : 1Local switch operational role: Virtual Switch ActivePeer switch number : 2

vss# sh switch virtual linkVSL Status : UPVSL Uptime : 18 hours, 38 minutesVSL SCP Ping : PassVSL ICC Ping : PassVSL Control Link : Te1/6/1

vss# sh switch virtual link portLMP summary

Link info: Configured: 2 Operational: 1Peer Peer Peer Peer Timer(s)running

Interface Flag State Flag MAC Switch Interface (Time remaining)--------------------------------------------------------------------------------Te1/5/4 v link_down - - - -Te1/6/1 vfs operational vfs 0007.0d72.4800 2 Te2/6/1 T4(960ms)

T5(29.98s)...vss# sh redundancy states

my state = 13 -ACTIVEpeer state = 4 -STANDBY COLD

Mode = Duplex...

In VSS mode?

Domain# unique for each VSS?

Role of this switch

Peer-switch visible?

VSL is up?

Link used to carry control plane

messages (ICC, IPC, SCP)

VSL member-links state

Redundancy mode SSO?


Troubleshooting VSL: counters

vss# sh switch virtual link counters

Port InOctets InUcastPkts InMcastPkts InBcastPktsPo10 3084500343 31059 7382085 1046088Te1/6/4 523470151 139662 1323349 1045940Te1/6/5 2814244020 11346 6883221 258

Port OutOctets OutUcastPkts OutMcastPkts OutBcastPktsPo10 1457635126 1467466 9890548 0Te1/6/4 363835687 264788 2732502 0Te1/6/5 1214900160 1202788 8103037 0...

Port Align-Err FCS-Err Xmit-Err ...Po10 0 0 0 ...Te1/6/4 0 0 0 ...Te1/6/5 0 0 0 ...Port Single-Col Multi-Col Late-Col ...Po10 0 0 0 ...Te1/6/4 0 0 0 ...Te1/6/5 0 0 0 ...Port SQETest-Err Deferred-Tx IntMacTx-Err ...Po10 0 0 0 ...Te1/6/4 0 0 0 ...Te1/6/5 0 0 0 ...

Aside from packet/bit rate this is

one-stop-shop command for VSL

packet and error counters

Always take 2-3 samples

All errors should be at or near zero

and most importantly not

incrementing (giants are ok)


Troubleshooting VSL: LMPvss# sh switch virtual link detail...LMP summary

...LMP neighbors

Peer Group info: # Groups: 1 (* => Preferred PG)

PG # MAC Switch Ctrl Interface Interfaces---------------------------------------------------------------*1 0004.9bbe.ac00 2 Te1/6/4 Te1/6/4, Te1/6/5...LMP hello timer

...LMP FSM info

sm(vslp_lmp 6/4), running yes, state operationalLast transition recorded: (hello)-> operational (t4_exp)-> operational (hello)-> operational (hello)-> operational (t4_exp)-> operational (hello)-> operational...LMP counters

Tx RxInterface OK Fail Bidir Uni Fail Bad--------------------------------------------------------------------Te1/6/4 805969 0 806270 7 0 0Te1/6/5 640674 0 640726 3 0 0

Rx error detailsInterface My info My info Bad MAC Bad switch Domain id Peer info

mismatch absent Address id mismatch mismatch-------------------------------------------------------------------------------Te1/6/4 0 7 0 0 0 0Te1/6/5 0 3 0 0 0 0

Complete information about LMP

layer of VSLP

At least 1 link should be operational

Should see a neighbor

Should not see any events except

t4_exp (hello tx timer expiry)

Non-zero (low number) error

counters are acceptable as long as

they do not increment (take 2-3

snapshots)


Troubleshooting VSL: LMPvss# sh switch virtual link portLMP summary

Link info: Configured: 2 Operational: 2

Peer Peer Peer Peer Timer(s)runningInterface Flag State Flag MAC Switch Interface (Time remaining)--------------------------------------------------------------------------------Te1/6/4 vfsp operational vfsp 0004.9bbe.ac00 2 Te2/6/4 T4(756ms)

T5(29.98s)Te1/6/5 vfsp operational vfsp 0004.9bbe.ac00 2 Te2/6/5 T4(756ms)

T5(29.92s)

Flags: v - Valid flag set f - Bi-directional flag sets - Negotiation flag set p - Peer detected flag set

Timers: T4 - Hello Tx Timer T5 - Hello Rx Timer

LMP Status

Last operational Current packet Last Diag Time sinceInterface Failure state State Result Last Diag-------------------------------------------------------------------------------Te1/6/4 Link down Hello bidir Never ran --Te1/6/5 Link down Hello bidir Never ran --

LMP hello timer

Hello Tx (T4) ms Hello Rx (T5*) msInterface State Cfg Cur Rem Cfg Cur Rem-------------------------------------------------------------------------Te1/6/4 operational - 1000 756 - 30000 29896Te1/6/5 operational - 1000 756 - 30000 29228

Compared to previous command

this one provides details of the

previous failure (if there was any) of

VSL links

Rest of the information is identical


Troubleshooting VSL: RRP

vss# sh switch virtual role detail

Switch Switch Status Preempt Priority Role Session IDNumber Oper(Conf) Oper(Conf) Local Remote

------------------------------------------------------------------LOCAL 1 UP FALSE(N ) 100(100) ACTIVE 0 0REMOTE 2 UP FALSE(N ) 100(100) STANDBY 6480 9910

RRP Counters:--------------------------------------------------------------------

Inst. Peer Direction Req Acc Est Rsugg Racc----------------------------------------------------------------------1 1 Tx 0 1 0 1 31 1 Rx 2 0 1 0 3

RRP FSM info:--------------------------------------------------------------------sm(vslp_rrp RRP SM information for Instance 1, Peer 1), running yes, state role_resLast transition recorded: (lmac)-> lstart (req)-> hold (srt_exp)-> hold (req)-> hold (est)-> role_neg (srt_exp)-> role_neg (racc)-> role_res (racc)-> role_res (srt_exp)-> role_res (racc)-> role_res (srt_exp)-> role_res (srt_exp)-> role_res

In dual-active recovery mode: No

One of the switches must be

standby. If both are active it means

VSS has recovered from dual-

active condition, but new standby

has not been reloaded, most likely

due to unsaved config

This only refers to local switch


Troubleshooting VSL

vss# sh switch virtual link port-channelFlags: D - down P - bundled in port-channel

I - stand-alone s - suspendedH - Hot-standby (LACP only)R - Layer3 S - Layer2U - in use N - not in use, no aggregationw - waiting to be aggregated

Group Port-channel Protocol Ports------+-------------+-----------+-------------------10 Po10(RU) - Te1/6/4(P) Te1/6/5(P)20 Po20(RU) - Te2/6/4(P) Te2/6/5(P)

vss# ping vslp output interface t1/6/4 count 100 size 1388

Type escape sequence to abort.Sending 100, 1388-byte VSLP ping to peer-sup via output port 1/6/4, timeout is 2 seconds:!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Success rate is 100 percent (100/100), round-trip min/avg/max = 12/12/28 ms

All ports on both sides of VSL

should be in bundled (P) state

Verify reliability of each individual

VSL link – output interface specifies

egress link (one of the VSL

interfaces). VSLP ping should work

when VSL is up, even if remote is in

RPR mode etc


Note: with VSS many commands use ‘switch <#> module <#>’ notation instead of just ‘module <#>’

In case of issues with VSL or VSS bring up, collect the following information

sh tech (if VSS is split, collect from both sides)

remote command switch sh monitor event vslp all detail(if VSS is split, collect from both sides)

Troubleshooting VSL:what information to collect


VSS Agenda

Initialization


Spanning Tree

1st hop redundancy

Traffic forwarding



High AvailabilityRedundancy Mechanisms

The default redundancy mechanism between the 2 VSS chassis and their associated

supervisors is NSF/SSO, allowing state information and configuration to be

synchronized. Additionally, only in NSF/SSO mode does the Standby supervisor PFC,

Switch Fabric, modules and their associated DFCs become active…

VSL

Should a mismatch of information occur between the Active and Standby Chassis, the

Standby Chassis will revert to RPR mode, where only configuration is synchronized, but

PFC, Switch Fabric and modules will not be brought up

Switch 1

Active

Switch 2

SSO Standby

VSL

Switch 1

12.2(33)SXI3

Active

Switch 2

12.2(33)SXH2

RPR Standby


In case of certain mismatches standby will only boot to RPR mode(fabric, PFC & modules will be down)

vss# show switch virtual redundancyMy Switch Id = 1

Peer Switch Id = 2Last switchover reason = none

Configured Redundancy Mode = ssoOperating Redundancy Mode = rpr

...vss# show switch virtual redundancy mismatch

Startup Config Mismatch:Mismatch in config file between local Switch 1 and peer Switch 2:ACTIVE : Interface TenGigabitEthernet1/6/5 shutdownSTANDBY : Interface TenGigabitEthernet1/6/5 not shut

Other possibilities

IOS version mismatch

Other VSL-related config mismatch

Non-SSO redundancy mode is configured

Forwarding engine (PFC) mismatch

Troubleshooting redundancy:why standby is not in SSO mode


VSS with 4 supervisors

Initially in-chassis redundant supervisors were kept in rommon not used

As of 12.2(33)SXI4 in-chassis redundant supervisors function as a linecard – ports are useable

Before switching to linecard mode supervisors will boot to RPR-warm mode meaning they will have their configuration synchronized

If active supervisor fails entire chassis is reloaded 2nd chassis takes over same model as with 2 sups

If supervisor fails completely (doesn’t boot) or removed, the in-chassis redundant supevisor will boot as active supervisor no need to follow procedure for supervisor replacement

VSL

SiSi SiSi

Active SSO

rommon> rommon>

VSL

SiSi SiSi

Active SSO

RPR-warm RPR-warm

Pre-12.2(33)SXI4

12.2(33)SXI4 and later


What is Dual-Active?

If VSL goes down standby needs to know if it was just VSL or the active switch that failed

For faster failovers assumption is that active switch fails Old standby becomes Active a.s.a.p.

If old Active is still there however we will have 2 devices with identical config on the network

IGP adjacencies will start to flap or will go down

L2 MEC will be error-disabled after ~1 minute by EtherChannelmisconfig guard (because of receiving 2 different BPDUs)

VSLSiSi SiSi

Active Standby

SiSi

Active

Dual-active, if not detected will cause severe network outage

Configure robust dual-active detection

Layer2-MEC

Layer3-MEC


Dual-Active Detection options

Enhanced PAGP

Hot StandbyActive

Switch 1 Switch 2

IP-BFD

Switch 1

VSLP VSLP BFD BFD

Switch 2

Hot StandbyActive

Switch 1 Switch 2

Hot StandbyActive

VSLP Fast Hello

L2 Heart Beat Link

Software-12.2(33)SXI

Enhanced subsecond detection in

12.2(33)SXI3

L3 Heart Beat Link

Software -12.2(33)SXH1

Requires PAGP+ capable neighbor with• 3750

12.2(46)SE• 4500

12.2(44)SE • 6500

12.2(33)SXH

Software -12.2(33)SXH1


Dual Active Recovery

Switch 1 detects that switch 2 is now also active triggering dual active condition thus switch 1 brings down all the local interfaces to avoid network instability. Until VSL link restoration occurs, switch 1 is isolated from the network;

Once the VSL link comes up, the role negotiation determines that switch 1 needs to come up in STAND_BY mode hence it reboots itself; finally, all interface on switch 1 are brought on line and switch 1 assumes STAND_BYrole

Switch 1 All

Interfaces Down

Dual Active Recovery

Switch 1 Reboot and

Comes Up in STAND_BY

Mode

VSS Restoration

Switch 2 inACTIVE Mode

OLDACTIVE

NewACTIVE


If configuration was changed but has not been saved the would-be-standby switch will not be reloaded following VSL recovery

Save the config & reload standby

19:54:59: %VSLP-SW2_SP-5-RRP_MSG: Role change from Active to Standby and hence need to reload

19:54:59: %VSLP-SW2_SP-5-RRP_UNSAVED_CONFIG: Ignoring system reload since there are unsaved configurations. Please save the relevant configurations

19:54:59: %VSLP-SW2_SP-5-RRP_MSG: Use 'redundancy reload shelf' to bring this switch to its preferred STANDBY role

Dual-active recovery, …

Reload from active switch will not correct this

After reloading it might happen that config between Active and Standby is not consistent Standby will come up in RPR modeSave the config once again and reload standby again (redundancy reload peer)


Virtual Switching System Which Dual Active Recovery Method Should I Use?

Since dual-active detection is important redundancy is highly recommended

Use Fast-hello + e-PAgP

In case of all-LACP deployment, use Fast-hello over port-channel

Only case where BFD had advantage was in pre-SXI3 release with routed ECMP uplinks and OSPF

SiSiSiSi

RedundantVSL Fiber

ePAgP

ePAgP

VSLP Fast-Helloor BFD


VSS Agenda

Initialization


Spanning Tree

1st hop redundancy

Traffic forwarding



Spanning Tree and VSS

STP process

Active Standby

VSS domain behaves as a single bridge

STP runs only on SP of active switch

VSL is not part on STP and will not be blocked

BPDUs will travel across single link of the MEC

STP will be blocking ports is there are redundant

links Keep STP enabled

Physical Logical

1

2

3

4

1 2

3

4


Troubleshooting STP

vss#sh spanning-tree interface po201 detail

Port 5767 (Port-channel201) of VLAN0001 is designated forwarding

Port path cost 3, Port priority 128, Port Identifier 128.5767.

Designated root has priority 0, address 001e.4963.7b94

Designated bridge has priority 32768, address 0008.e3ff.fdbd

Designated port id is 128.5767, designated path cost 16

Timers: message age 0, forward delay 0, hold 0

Number of transitions to forwarding state: 1

Link type is point-to-point by default

BPDU: sent 4447, received 12

...

vss# remote login switch

vss-sp# debug interface po201

Condition 1 set

vss-sp# debug spanning-tree switch tx

Spanning Tree Switch Shim transmit bpdu debugging is on

Dec 6 14:59:22.594: SW1_SP: STP SW: FAST TX: VLAN 555 Port-channel201: bpdu size 116, refcnt 1




vss-sp# debug spanning-tree switch tx decode

Spanning Tree Switch Shim decode transmitted packets debugging is on

Dec 6 14:59:43.510: SW1_SP: STP SW: FAST TX: 0180.c200.0000<-0015.6301.26f8 type/len 0026

Dec 6 14:59:43.510: SW1_SP: encap SAP linktype ieee-st vlan 1 len 112 on v1 Po201

Dec 6 14:59:43.510: SW1_SP: 42 42 03 SPAN

Dec 6 14:59:43.510: SW1_SP: CFG P:0000 V:00 T:00 F:00 R:0000 001e.4963.7b94 00000010

Dec 6 14:59:43.510: SW1_SP: B:8000 0008.e3ff.fdbd 96.87 A:0400 M:1400 H:0200 F:0F00

...

vss-sp# undebug all

All possible debugging has been turned off

STP state, role and BPDU counters

for given port

All debugging for STP is on active

SP

Limit debugs to port in question

Abbreviated BPDU debug

Detailed BPDU debug (when

enabled together with abbreviated

one)

Observe normal precautions

regarding debugs


Spanning Tree stability features recap

Feature Condition Works on Effect Note

UDLD

Detects if link becomes

unidirectional

I.e. link cannot carry BPDUs

both ways causes loops

Physical

port

Error-disables

unidirectional

links

Useful on port-channels to

take out broken links,

alternative fast-timers

PAGP/LACP

Bridge

Assurance

(BA)

Expects to receive a BPDU

every hello_time from the

peer.

I.e. cases of dead control

plane on the remote side,

also BPDU loss

Logical

port

Blocks port at

STP level

(BA-

inconsistent

state)

Main protection mechanism

where supported, alternative

is Loop Guard

Dispute

Checks the remote port role

in the received BPDU, role

should not be designated in

BPDU received on

designated port

Cases of unidirectional

communication

Logical

port

Blocks port at

STP level

(Disputed

state)

Complements BA, on by

default. Somewhat overlaps

with UDLD, but not as

effective on port-channels.

Only works with RSTP/MST

BPDUs

Loop

Guard

Doesn’t allow port to take

designated role if it stopped

receiving BPDUs

Unidirectional

communication, control plane

issues on remote

Logical

port

Blocks port at

STP level

(Loop-

inconsistent)

Superseded by BA + Dispute,

use with PVST+ or when BA

is not supported


Bridge assurance, Dispute & UDLD

Preferred combination is Bridge Assurance + UDLD normal mode + Dispute (on all interswitch links) when both sides support it

UDLD is needed to take out bad links from port-channels (otherwise BA or Dispute will keep whole port-channel blocked). PAgP/LACP will take out bad links, but will take longer (~105sec vs ~20sec for UDLD with 7 sec timer)

If preferred config is not supported use Loop Guard + UDLD(supported by all Cisco switches)

Defaults: BA/UDLD – disabled, Dispute - enabled


VSS Agenda

Initialization


Spanning Tree

1st hop redundancy

Traffic forwarding



Asymmetric Routing

Alternating HSRP Active between distribution switches can be used for upstream load balancing, however downstream traffic hits both distribution block switches

This can cause a problemwith unicast flooding

ARP entries age in 4 hours while L2 entries age in 5 minutes

ARP entry with no matching L2entry unicast flooding

In many cases when the HSRP standby needs to forward a frame it will have to unicast flood the frame since it’s CAM table is empty VLAN 2

SiSiSiSi

VLAN 3

Switch 1: Active

HSRP and Root

Bridge VLAN 3

VLAN 2VLAN 3

Switch 2: Active

HSRP and Root

Bridge VLAN 2

CAM Table

Empty for

VLAN 2

CAM Table

Empty for

VLAN 3

B

BB

B

B

With VSS there is single logical router thus no asymmetric routing


1st hop redundancy with VSS

MAC_A Router MAC

IP A IP B

Router MAC

0001.0002.0003

Router MAC

0001.0002.0003

MAC_B Router MAC

IP B IP A

PC A

PC BVSS acts as 1 router there is 1 router MAC

address, both switches will L3 switch packets

destined to that MAC address

Once either switch learns dynamic MAC address,

other switch will also learn no unicast floods

due to asymmetry of traffic between switches

In case of failover router MAC address does not

change Inherrent 1st hop redundancy


VSS mac-address

By default VSS will use Router mac-address from active switch backplane

Router mac-address is maintained across switchovers – no 1st hop redundancy protocol is needed

If entire VSS system is brought down and then up again and switch 2 ends up being active – router mac-address might change (this will only have impact on devices that ignore gratuitous ARPs)

To avoid such change, use ‘mac-address use-virtual’ – with this command VSS will use special mac-address reserved for VSS

vss(config)#switch virtual domain 111vss(config-vs-domain)#mac-address use-virtual

Configured Router mac address is different from operational value. Change will take effect after config is saved and the entire Virtual Switching System (Active and Standby) is reloaded.

Virtual mac is based on 0008.e3ff.fc00

Alternatively router-mac maybe statically configured with ‘mac-address <address>’ in the domain config context


Troubleshooting Router-MAC

vss# sh interface vlan 226

Vlan226 is up, line protocol is up

Hardware is EtherSVI, address is 0008.e3ff.fdbc (bia 0008.e3ff.fdbc)

Internet address is 192.168.222.18/30

...

vss# sh mac-address-table address 0008.e3ff.fdbc vlan 226 all

Legend: * - primary entry

age - seconds since last seen

n/a - not available

vlan mac address type learn age ports

------+----------------+--------+-----+----------+--------------------------

Supervisor switch 1 Module 6

* 226 0008.e3ff.fdbc static No - Router


* 226 0008.e3ff.fdbc static No - Router

vss# sh mac-address-table address 0008.e3ff.fdbc vlan 226 detail switch 2 module 6

MAC Table shown in details

========================================PI_E RM RMA Type Alw-Lrn Trap Modified Notify Capture Flood Mac Address Age Pvlan SWbits Index XTag

----+---+---+----+-------+----+--------+------+-------+------+--------------+----+------+------+------+----


Yes No No ST No No No No No No 0008.e3ff.fdbc 0xE8 226 0 0x380 1

What is router MAC for given

interface

It should be pointing to the ‘Router’

Actual hardware L2 entry must

have non-zero Xtag in order for

forwarding engine to consider such

packets for L3 switching

When VSS receives a packet destined to Router-MAC it will try to L3 switch (route in hardware) the packet, else the packet will be bridged


MAC address learning with VSS

A ↓ A ↓

PC A

PC B

MAC A is learned on lower MEC, triggering the

frame to be sent to every forwarding engine

(DFC/PFC) Flood to Fabric mechanism (HW)1

Internal frame header (carried over VSL) includes

source index which identifies source port and

hence the MAC is learned on lower MEC although

the frame is received on VSL

Depending on how traffic is flowing through VSS

some forwarding engines might not see the

packets from A after initial flood to fabric which

might lead to aging of address and flooding

MAC synchronization feature keeps address from

expiring as long as traffic from that address is

seen anywhere in the system

1

2

2


MAC address synchronization Initial new learns are syncronized between switch 1 and switch 2

However if only switch 1 or switch 2 ‘sees’ the traffic for given address L2 entry might age out in one of the switches (this behavior is per forwarding engine: PFC/DFC)

In order to reduce chance of unicast flooding we need to keep L2 entries consistent access both switches

‘mac-address-table synchronize’ feature will keep L2 tables synchronized

Enabled by default when WS-X6708 linecard is present in the chassis

Enabled by default in VSS as of 12.2(33)SXI4

Recommended in all cases

Make sure there is at least 2x aging intervals in synchonization interval(i.e. for sync interval 160, L2 aging is >320 seconds, 480 recommended)

vss(config)# mac-address-table synchronize

% Current OOB activity time is [160] seconds

% Recommended aging time for all vlans is atleast three times the activity interval

and global aging time will be changed automatically if required

When troubleshooting unicast flooding, 2 items are very important

What module traffic arrives to (use commands to check ether-channel load-balancing)

Whether the module in question has the mac-address learned (use ‘sh mac-address address <mac> all’)


VSS Agenda

Initialization


Spanning Tree

1st hop redundancy

Traffic forwarding



Ingress forwarding model

Distributed architecture. Ingress forwarding engine makes forwarding, ingress *and* egress ACL/QOS decisions

IMPORTANT: If the linecard where packet is received has DFC –entries on that linecard need to be looked at when troubleshooting. Otherwise look at active supervisor’s forwarding entries

i.e. ‘sh mls cef <prefix> module <mod#>’

or ‘sh mls cef <prefix>’

DFC DFC

Ingress EgressXFabric

Traffic flow


Traffic locality

Main concept for traffic forwarding is locality

– Only local ports are used to send traffic out

– … except when there are no local ports, this is when traffic will cross VSL/Peer-link


Traffic locality for ECMP routes

ECMP follows a similar behavior, locallinks are preferred and all traffic is forwarded out of a locally attached link

Hardware FIB inserts entries for ECMProutes using locally attached links

If all local links fail the FIB is programmed to forward across the VSL link

vss# sh ip route 10.121.0.0 255.255.128.0 longer-prefixes

D 10.121.0.0/17

[90/3328] via 10.122.0.33, 2d10h, TenGigabitEthernet2/2/1




vss# sh mls cef 10.121.0.0 17 switch 1

Codes: decap - Decapsulation, + - Push Label

Index Prefix Adjacency

102400 10.121.0.0/17 Te1/2/2 , 0012.da67.7e40 (Hash: 0001)

Te1/2/1 , 0018.b966.e988 (Hash: 0002)

Four ECMPEntries

Two FIB Entries

Te1/2/2

Te1/2/1

SW1

SiSi SiSi


Important:: Only use parameters

consistent with the configured

load-balancing algorithm.

Command uses all the specified

arguments to calculate the hash.

VSS L2/L3 Forwarding (Data Plane)

Identify the physical path for flow from host 2 host 1 (out of Port-channel 2)

vss# show etherchannel load-balance hash-result interface Port-channel 2 switch 1

ip 9.0.1.2 vlan 705 8.0.1.1

Computed RBH: 0x6

Would select Gi1/6/2 of Po2

vss# show etherchannel load-balance hash-result interface Port-channel 2 switch 2

ip 9.0.1.2 vlan 705 8.0.1.1

Computed RBH: 0x6

Would select Gi2/9/15 of Po2

Packet coming in on switch 1, needing to go

out on Po2 will select Gi1/6/2

Packet coming in on switch id 2, needing to

go out on Po2 will select Gi2/9/15

Verify the load-balance algorithm used

vss# show etherchannel load-balance switch 2 module 2

EtherChannel Load-Balancing Configuration:

src-dst-ip vlan included

mpls label-ip

EtherChannel Load-Balancing Addresses Used Per-Protocol:

Non-IP: Source XOR Destination MAC address

IPv4: Source XOR Destination IP address

IPv6: Source XOR Destination IP address

MPLS: Label or IP

VSS Data Plane Troubleshooting L2 MECVSS specific commands

augmented with switch id


VSS L2/L3 Forwarding (Data Plane)

Routing table shows two Equal Cost Paths to 9.0.0.0/8vss# show ip route 9.0.0.0 | i via

Known via "eigrp 101", distance 90, metric 3072, type internal

Redistributing via eigrp 101

7.7.1.2, from 7.7.1.2, 1d00h ago, via TenGigabitEthernet2/2/7

* 7.6.1.2, from 7.6.1.2, 1d00h ago, via TenGigabitEthernet1/3/2

Looking at the HW table shows next hop directly attached to local switch is preferred

vss# show mls cef lookup 9.0.1.0 switch 1 mod 3



108775 9.0.0.0/8 Te1/3/2 , 000f.35ed.7c00

vss# show mls cef lookup 9.0.1.0 switch 2 mod 2



108775 9.0.0.0/8 Te2/2/7 , 000f.35ed.7c00

DUT# show mls cef exact-route 8.0.1.1 0 9.0.1.2 0 switch 1 mod 3

Interface: Te1/3/2, Next Hop: 7.6.1.2, Vlan: 4064, Destination Mac: 000f.35ed.7c00

DUT# show mls cef exact-route 8.0.1.1 0 9.0.1.2 0 switch 2 mod 2

Interface: Te2/2/7, Next Hop: 7.7.1.2, Vlan: 4056, Destination Mac: 000f.35ed.7c00

Packet coming in on switch 1 module 3, for 9.0.0.0/8

prefers next hop attached to local switch id 1

Packet coming in on switch 2 module 2, for 9.0.0.0/8

prefers next hop attached to local switch id 2

VSS Data Plane Troubleshooting ECMP: Host 1 Host 2


vss# sh mac-address-table address 0005.9a3b.6c80 vlan 226

...

vlan mac address type learn age ports

------+----------------+--------+-----+----------+--------------------------


* 226 0005.9a3b.6c80 dynamic Yes 10 Po3


* 226 0005.9a3b.6c80 dynamic Yes 10 Po3

vss# sh etherchannel 3 summary

...

Group Port-channel Protocol Ports

------+-------------+-----------+-----------------------------------------------

3 Po3(SU) PAgP Gi1/1/15(D) Gi2/6/3(P)

VSS

Po4

What is the port for this mac

address

What are physical ports of port-

channel

All ports on switch1 side are

down

If packet will arrive to switch1 to

be switched to po3, packet will

cross VSL

Po3

1/1/33

2/4/33

1/1/15

2/6/30005.9a3b.6c80

Will the packet cross VSL link?


vss# sh mac-address-table address 0005.9a3b.6c80 vlan 226 detail switch 1 module 6MAC Table shown in details

========================================PI_E RM RMA Type Alw-Lrn Trap Modified Notify Flood Mac Address Age Pvlan Index XTag----+---+---+----+-------+----+--------+------+------+--------------+----+------+------+----Supervisor switch 1 Module 6Yes No No DY No No Yes No No 0005.9a3b.6c80 0x86 226 0xB40 0

vss# remote command switch test switch virtual ltl index 0xB40...

Unmapped index: 0xB40------+----------------------------------------SW viewIndex | Ports------+----------------------------------------0x0B40 Po3[Gi2/6/3],Po10[Te1/6/4]...------+----------------------------------------HW viewIndex | Ports------+----------------------------------------0x0B40 Te1/6/4,Gi2/6/3...

vss# sh switch virtual link port-channel | i PoGroup Port-channel Protocol Ports10 Po10(RU) - Te1/6/4(P)20 Po20(RU) - Te2/6/4(P)

VSS

Po4

Find the index for given mac

address on ingress forwarding

engine

Find what ports on the local

switch (1) this index includes

Index should include VSL ports

How to verify if the packet from

switch 1 will cross VSL in order to

reach next-hop mac-address?

Po3

1/1/33

2/4/33

1/1/15

2/6/30005.9a3b.6c80

Will the packet cross VSL link?


VSS forwarding troubleshooting summary

Unless the traffic is crossing VSL, troubleshooting VSS packet forwarding is exactly the same as troubleshooting standalone cat6500

When traffic crosses VSL, verify

– L3 entries on the ingress forwarding engine (PFC or DFC)

– L2 entries (for next hop destination mac) on forwarding engine servicing the VSL on the 2nd chassis (strictly speaking L2 entries need to be checked on all DFCs along the packet path)


Special case for flooding

MAC_A

MAC_B

MAC B is not known flood the frame11

Internal frame header (carried over VSL) includes

destination index which is remapped by egress

switch to another index that does not include any

MEC that has operational ports on ingress switch

2

Frame is flooded to devices that are single

connected to egress switch (on the right)

3

2

3


Each flow is assigned to 1 of 8 ‘buckets’

Each port in port-channel transmits traffic for some buckets (i.e. 4 for 2-port channel, 2 for 4-port etc)

When ports are joining/leaving channel the buckets are redistributed among operational ports in deterministic fashion

Flows that remain on operational ports might be disturbed while ASICs are being programmed

With adaptive hash option, only buckets that must move are reprogrammed

Member 1 Member 2

1 2

3 4

5 6

7 8

Member 1 Member 2 Member 3

1 2 3

4 5 6

7 8

New member

joins

EtherChannel Adaptive Hash

Member 1 Member 2

1 2

3 4

5 6

7 8

Member 1 Member 2 Member 3

1 2 3

5 4 6

7 8

New member

joins

buckets that must move

buckets moving between

operational ports

buckets that must move


Adaptive hash is enabled by default on VSL link

If there is 1 link / chassis / MEC – adaptive hash on MEC will not make any difference

If the network consists of several adjacent VSS systems, adaptive hash was enhanced to avoid traffic polarization (as of 12.2(33)SXI)

Configured per port-channel

With adaptive hash less flows should be impacted when ports join or leave port-channels

This is mostly evident when control-plane is busy (i.e. when many changes are happening at the same time – during failovers etc)

EtherChannel Adaptive Hash

vss(config)#int port-channel200

vss(config-if)#port-channel port hash-distribution adaptive


SPAN

When SPAN’ed traffic is crossing VSL it is transmitted over single link this might cause oversubscription of VSL link if amount of SPAN’ed traffic is significant

Use MEC as SPAN destination to prevent SPAN’edtraffic from crossing VSL

If one side of the MEC goes down – SPAN’ed traffic will cross VSL

Provision enough bandwidth on VSL

Use ‘port-channel min-links’ LACP feature on SPAN destination MEC to bring down MEC if link is down on one side

Use EEM script to shut down MEC or SPAN session when one side of SPAN destination MEC goes down


VSS Agenda

Initialization


Spanning Tree

1st hop redundancy

Traffic forwarding



Multicast forwarding

Layer 2 access has two multicast routers on the access subnet, RPFchecks and split roles between high and low IP address routers

VSS has a single multicast router which simplifies multicast topology

The multicast forwarder is selected based on which member of VSSlink receives multicast traffic

SiSi

Designated

Router

(High IP Address)

IGMP Querier

(Low IP address)

Non-DR Has to

Drop All

Non-RPF Traffic

SiSi

Single Logical Multicast Designated Router and IGMP Querier


MEC behavior upon VSS recovery after SSO switchover

vss(config)#port-channel load-defer 120vss(config)#int po200vss(config-if)#port-channel port load-deferThis will enable the load share deferral feature on this port-channel.The port-channel should connect to a Virtual Switch (VSS).Do you wish to proceed? [yes/no]: y

To prevent this issue, configure ‘port-channel load-defer’ feature on upstream switch

Upstream switch will delay sending traffic to newly bundled port for configured duration

Following SSO switchover left switch comes up

after reload

1

MEC link from left switch is brought up and joins

the bundle

2

Top switch starts sending a share of traffic to the left

switch, but the left switch might still be converging

(loading FIB tables, programming ASICs etc), so it

might not be fully ready to correctly forward the this

traffic

this might cause part of traffic to be lost for

some time after the switch recovery

3

1

2

3


Multicast fast-redirect

When a member of egress Layer2 port-channel (MEC or DEC) is unbundled/bundled On VSS replicating multicast traffic in egress mode it might take noticeable time to reprogram hardware to send traffic via remaining links (local or across VSL)

Fast-redirect feature shortens reprogramming time by preprogramming most of the needed changes

SiSi SiSi

MEC

MEC

Sources

Receivers

vss(config)#interface port-channel 40vss(config-if)#mls ip multicast egress fast-redirect


VSS: summary

1 active redundant control plane

single config

single point of management


Standby switch is essentially a

set of additional linecards

Control messages and Data

frames flow between active and

standby via VSL(can be seen as backplane

extension)

Special encapsulation on VSL

frames to carry additional

information

ActiveData Plane

ActiveControl Plane

ActiveData Plane

StandbyControl Plane

MEC

VSL

Dual-Active

detection link

Active Standby

VSS domain


VPC


Both VPC and VSS

• simplify logical Layer 2 topology

• use Traffic Locality for efficient shortest path

forwarding


VPC Agenda

Initialization

Redundancy considerations

Spanning Tree

Traffic forwarding

1st hop redundancy



VPC – Virtual Port channel 2 active control planes

2 configs

2 points of management


Primary-Secondary notion for some

aspects of operation

Control messages and Data frames

flow between active and standby via

Peer-Link

Peer-Link is L2 trunk with plain 802.1q

encapsulation

Control messages are carried by CFS

over Peer Link

Peer keepalive link to detect dual-

active condition

We call VPC the MCEC between VPC

domain and access switches

ActiveData Plane

ActiveControl Plane

ActiveData Plane

ActiveControl Plane

VPC

Peer-Link

Peer

Keepalive link

Primary Secondary

VPC domain


VPC initialization

VPC init is largely independent of NXOS boot eachswitch boots on its own

VPC feature starts

Keep-alive linkup / peer communication is established

Peer-link linkup / CFS communication is established

Primary/Secondary role is resolved

Consistency is checked via CFS and applications synced

Peer-Link brought UP for data

VPCs brought UP


Nexus# sh cfs application----------------------------------------------Application Enabled Scope----------------------------------------------arp Yes Physical-ethstp Yes Physical-ethvpc Yes Physical-ethigmp Yes Physical-ethl2fm Yes Physical-eth...

Cisco Fabric ServicesCFS

Uses

• Configuration validation

• MAC member port synchronization

• vPC member port status

• IGMP snooping synchronization

• vPC status

For VPC CFS messages are encapsulated in Ethernet frames delivered between peers on the peer-link

CFS messaging


VPC has distributed management plane. Configurations of both switches are managed separately

Some configurations inconsistencies could lead to undesirable forwarding implications (packet duplication, blackholing etc). VPCtakes different action depending on the type of inconsistency

Type 1: VPC will not come up

Type 2: VPC will come up, but undesirable forwarding implications might occur, syslog will be printed upon detected inconsistency

VPC Configuration consistency

Nexus# sh vpc consistency-parameters interface port-channel 1Name Type Local Value Peer Value------------- ---- ---------------------- -----------------------lag-id 1 [(7f9b, [(7f9b,...mode 1 active activeSTP Port Type 1 Default DefaultSTP Port Guard 1 None NoneSTP MST Simulate PVST 1 Default DefaultNative Vlan 1 1 1Port Mode 1 trunk trunkMTU 1 1500 1500Duplex 1 full fullSpeed 1 10 Gb/s 10 Gb/sAllowed VLANs - 101 101

Nexus# sh vpc consistency-parameters globalName Type Local Value Peer Value------------- ---- ---------------------- -----------------------STP Mode 1 Rapid-PVST Rapid-PVSTSTP Disabled 1 None NoneSTP MST Region Name 1 "" ""STP MST Region Revision 1 0 0STP MST Region Instance to 1VLAN MappingSTP Loopguard 1 Disabled DisabledSTP Bridge Assurance 1 Enabled EnabledSTP Port Type, Edge 1 Normal, Disabled, Normal, Disabled,BPDUFilter, Edge BPDUGuard Disabled DisabledSTP MST Simulate PVST 1 Enabled EnabledInterface-vlan admin up 2 101 101Interface-vlan routing 2 1,101 1,101


Troubleshooting VPC initialization Use sh vpc to check the feature status

vpc1# show feature | i vpc

vpc 1 enabled

vpc1# sh vpc

Legend:

(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 1

Peer status : peer adjacency formed ok

vPC keep-alive status : peer is alive

Configuration consistency status: success

Type-2 consistency reason : Consistency Check Not Performed

vPC role : primary

Number of vPCs configured : 1

Peer Gateway : Disabled

Dual-active excluded VLANs : -

vPC Peer-link status

---------------------------------------------------------------------

id Port Status Active vlans

-- ---- ------ --------------------------------------------------

1 Po100 up 1,101

vPC status

----------------------------------------------------------------------

id Port Status Consistency Reason Active vlans

-- ---- ------ ----------- ------ ------------

1 Po1 up success success 101

CFS can communicate with the

peer

We hear peer-alives

Configs are compatible

Master/Slave for certain apps

Peer-Link will come up after CFS +

Peer-Keepalive + Config check are

ok


Troubleshooting VPC initialization

Stable, not expecting issues here

Set VPC logging level to 5 (default) to see more verbose messaging during the VPC bringup

vpc1(config)# logging level vpc 5

08:18:47 %ETHPORT-5-SPEED: Interface port-channel100, operational speed changed to 10 Gbps Peer-Link comes up

08:18:51 %VPC-3-PEER_UNREACHABLE: Remote Switch Unreachable

08:18:51 %VPC-3-VPC_PEER_LINK_BRINGUP_FAILED: vPC peer-link bringup failed (vPC peer is not reachable over cfs)

08:18:51 %ETHPORT-3-IF_ERROR_VLANS_SUSPENDED: VLANs 1,100-101 on Interface port-channel100 are being suspended.(Reason: vPC peer is not reachable over cfs)

08:18:51 %ETHPORT-5-IF_UP: Interface port-channel100 is up in mode trunk

08:18:58 %VPC-4-VPC_ROLE_CHANGE: In domain 1, VPC role status has changed to primary

08:18:58 %ETHPORT-3-IF_ERROR_VLANS_REMOVED: VLANs 1,100-101 on Interface port-channel100 are removed fromsuspended state.

08:18:58 %VPC-5-VPC_DELAY_SVI_BUP_TIMER_START: vPC restore, delay interface-vlan bringup timer started

08:19:08 %VPC-5-VPC_DELAY_SVI_BUP_TIMER_EXPIRED: vPC restore, delay interface-vlan bringup timer expired, reiniting interface-vlans

08:19:08 %VPC-5-VPC_RESTORE_TIMER_START: vPC restore timer started to reinit vPCs

08:19:38 %VPC-5-VPC_RESTORE_TIMER_EXPIRED: vPC restore timer expired, reiniting vPCs

In case process does not go beyond certain stage, one should look at communication between the peers (CFS)


VPC config remarks

Check config consistency using ‘sh vpc consistency-parameters’

Complete list of parameters which should be consistent is quite extensive: physical port config, QOS, security, STP, routing protocols etc

check config guide for specific NXOS version

Domain id must be unique for each domain reachable adjacent at Layer 2

VPC domain 100

VPC domain 200

VPC

Domain id MUST be

different

(can’t be 100 on both

Pair)


VPC: CFS troubleshooting

Cisco Fabric Services Transport of control messages between VPC peers

Nexus# show cfs status

Distribution : Enabled

Distribution over IP : Disabled

IPv4 multicast address : 239.255.70.83

IPv6 multicast address : ff15::efff:4653

Distribution over Ethernet : Enabled

Nexus# show cfs peers

Physical Fabric

---------------------------------------------

Switch WWN IP Address

---------------------------------------------

20:00:00:1b:54:c2:42:41 10.48.73.222 [Local]

Nexus

20:00:00:1b:54:c2:42:44 0.0.0.0

Total number of entries = 2

Nexus# show cfs internal ethernet-peer statistics| i Trans|Rece

Number of Segments Transmitted : 218

Number of Acks Transmitted : 223

Maximum Segment Size Transmitted : 0

Number of Transmission Timeouts : 0

Number of segments in Transmit Queue : 0

Number of segments in Re-Transmit Queue : 0

Total Number of Segments Received : 441

Number of Acks Received : 217

Number of Duplicate Messages Received : 0

Number of Unexpected Segments Received : 0

Number of fragmented segments Received : 2

Number of duplicate fragments Received : 0

Number of unfragmented segments Received : 210

Number of Received Segments Dropped : 0

Number of Unreliable segments Transmitted : 1

Number of Unreliable segments Received : 1

Nexus# sh cfs internal notification log name vpc

Sun Nov 14 15:27:22 2010: Peer add 20:00:00:1b:54:c2:42:44

Sun Nov 14 19:05:25 2010: Peer gone 20:00:00:1b:54:c2:42:44

Sun Nov 14 19:08:03 2010: Peer add 20:00:00:1b:54:c2:42:44

TX/RX counters should move when

VPC is active or coming up

Remote peer should be seen

Shows timestamps for when CFS

communication for VPC was

interrupted (peer-reload, peer-link

issues etc)


More information

sh tech(collect for offline analysis, takes ~5 min when redirected to file)

sh tech vpc(collect when there is no time for ‘big’ sh tech)

debug vpc peer(peer events, useful for indepth vpc troubleshooting)

debug vpc peer-link(peer-link events, for indepth vpc bringup troubleshooting)

debug cfs event ethernet(cfs event – peer communication)


VPC Agenda

Initialization


Spanning Tree

Traffic forwarding

1st hop redundancy



Process restartability

Supervisor redundancy

VPC redundancy

Active

Standby(SSO)

Active

Standby(SSO)

Process 1

Process 2

Process X

…

Process 1

Process 2

Process X

…

Switch 1 Switch 2

VPC Domain

Processes checkpoint their runtime state

Crashing process is restarted statefully by

system manager

HA-policy will trigger

supervisor switchover

in response to

excessive process

crashing, software,

hardware or

diagnostic failure

VPC redundancy model

Devices dual-attached to VPC domain are protected against

single switch failure (power, hardware, maintenance etc)


Peer-link failure handling(similar to dual-active detection in VSS)

VPC peer-link failure

I am primary

Done

Receiving

Keepalives*

Bring down all VPC ports

Become primary

primary

2ndary

yes

no

Primary is alive

Primary is gone

VPC peers do not require reload following

peer-link failure or recovery


Keepalive link

Heartbeat between vPC peers to prevent dual-active scenario

Keepalives are sent every second by default on UDP port 3200

3 second hold timeout on peer-link loss (ignore keepalive to leave time for convergence before taking action)

5 seconds keepalive timeout (starts after hold timeout after peer-link down) – if no keepalive received during this timeout dual active detection seconday bring down VPC

Use dedicated link, though NXOS does not enforce this – just IP connectivity is verified

Mgmt interface can be used as keepalive link, but do not connect the managemet interfaces together directly (only active supervisor management interface is up)

vpc1# debug vpc peer-keepalive13:10:54.257099 vpc: received new OOB packet, version(0) flags(0) my_context(0) your_context(0) my_epoch(604049) your_epoch(604104) my_ip(1.1.1.2)13:10:54.257126 vpc: your_ip(1.1.1.1) domainId(1)13:10:55.257442 vpc: received new OOB packet, version(0) flags(0) my_context(0) your_context(0) my_epoch(604050) your_epoch(604105) my_ip(1.1.1.2)13:10:55.257469 vpc: your_ip(1.1.1.1) domainId(1)13:10:56.257324 vpc: received new OOB packet, version(0) flags(0) my_context(0) your_context(0) my_epoch(604051) your_epoch(604106) my_ip(1.1.1.2)13:10:56.257351 vpc: your_ip(1.1.1.1) domainId(1)

Peer Keepalives


Troubleshooting VPC peer-keepalives

Nexus# show vpc peer-keepalive


--Send status : Success

--Last send at : 2009.06.19 00:41:15 589 ms

--Sent on interface : Eth2/35

--Receive status : Success

--Last receive at : 2009.06.19 00:41:14 580 ms

--Received on interface : Eth2/35

--Last update from peer : (1) seconds, (9) msec

vPC Keep-alive parameters

--Destination : 7.7.7.77

--Keepalive interval : 1000 msec

--Keepalive timeout : 5 seconds

--Keepalive hold timeout : 3 seconds

--Keepalive vrf : v1

--Keepalive udp port : 3200

--Keepalive tos : 192

Nexus# show vpc statistics peer-keepalive


vPC keep-alive statistics

----------------------------------------------------

peer-keepalive tx count: 9773

peer-keepalive rx count: 8985

average interval for peer rx: 991

Count of peer state changes: 0

Peer-keepalive is only essential at

the time when peer-link goes down

At any other time peer-keepalive

failure will only trigger syslog

Peer-keepalives might be affected

by extreme control plane load

(check CPU utilization & COPP)

Number of keepalive state

transitions, closer to 0 - better


VPC behavior at initialization(default)

VPC needs to be able to talk to the peer (over peer-link) before bringing up VPC port-channels

Negotiate LACP/STP operating roles for the chassis

Wait for per-port peer parameters and handshake to bring up vPC ports

Performs peer parameters consistency check on each VPC bringup

Only after VPC port-channels are brought up.

What if after a full DC outage (both Nexus down), only one switch is coming up ?

Will not bring up VPCs if after a datacenter outage, only one VPC peer comes back up


VPC Reload Restore

Allows to bring up VPCs after timeout if peer is presumed dead

Default timeout 240 sec

Assumes primary role for STP and LACP

Nexus(config)# vpc domain 1Nexus(config-vpc-domain)# reload restore ?<CR>delay Duration to wait before assuming

peer dead and restoring vpcs

Nexus(config-vpc-domain)# reload restore delay ?<240-3600> Time-out for restoring vPC links

(in seconds)


ARP synchronization

PC A

PC B

ARP

Ip B Mac B

ARP

Ip B ???

Needs to be

Resolved ?

When traffic pattern changes (due to VPC links going up/down, due to failover etc) the peer that handles the traffic might need to resolve ARP before being able to forward packets

This might introduce additional delay to traffic recovery

ARP sync feature is supported as of 4.2(6), and allows VPC peers to synchronize their ARP tables over CFS

vpc(config)# vpc domain 1vpc(config-vpc-domain)# ip arp synchronize


More information

sh log last <x>(review sequence of events)

show file logflash://sup-standby/log/messages(in case other supervisor was active when everything started)

sh process log(which processes have crashed when)

sh redundancy status(status of supervisor redundancy & last switchover data)

sh system reset-reason(last reset/switchover reason per module)

sh logging onboard internal reset-reason(reset reason from different components point of view – useful for complex cases)

sh tech /from main VDC/(collects most of the above for offline analysis)


VPC Agenda

Initialization


Spanning Tree

Traffic forwarding

1st hop redundancy



Handling of Spanning Tree: VPC

STP process

Primary Secondary

STP process

STP runs on both switches (2 active control

planes) but only primary switch controls VPCs.

(even if root is secondary , then Primary will send

bpdu with root info being secondary)

VPC port states changes are communicated to

secondary via CFS messages.

For non-VPC ports domain appears as 2 bridges

1

Peer-link is part of STP. BPDU handling is

modified such that Peer-link will never be blocked

(similar to MST implementation of IST)

2

Non-VPC ports are managed independently by

local STP process on each switch

1 1

2


STP troubleshooting

Peer link is running STP

Left-Root# sh spanning vlan 35

VLAN0035

Spanning tree enabled protocol rstp

Root ID Priority 24611

Address 001b.54c2.4241

This bridge is the root

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 24611 (priority 24576 sys-id-ext 35)

Address 001b.54c2.4241

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------

Po1 Desg FWD 1 128.4096 (vPC) P2p

Po100 Desg FWD 2 128.4195 (vPC peer-link) Network P2p

Right# sh spanning-tree vl 35 detail | i "^ Port|BPDU"

Port 4096 (port-channel1, vPC) of VLAN0035 is designated forwarding


Port 4195 (port-channel100, vPC Peer-link) of VLAN0035 is root forwarding


On the other end of peer-link po1 is designated despite not sending or receiving single BPDU


STP troubleshooting Looking at BPDUs

Left-Root# debug spanning-tree bpdu_tx tree 101

14:20:37.556707 stp: RSTP(101): transmitting RSTP BPDU on port-channel100

14:20:37.556750 stp: vb_vlan_shim_send_bpdu(1933): VDC 4 Vlan 101 port port-channel100 enc_type 1 len 42


14:20:37.556863 stp: vb_vlan_shim_send_bpdu(1933): VDC 4 Vlan 101 port port-channel1enc_type 2 len 36

Left-Root# debug spanning-tree all


14:22:23.560169 stp: vb_vlan_shim_send_bpdu(1933): VDC 4 Vlan 1 port port-channel100enc_type 2 len 36

14:22:23.560219 stp: BPDU TX: vb 1 vlan 1 port port-channel100 len 36 ->0180c2000000CFG P:0000 V:02 T:02 F:78 R:80:01:00:1b:54:c2:42:43 00000002 B:80:01:00:1b:54:c2:42:44 9063 A:0000 M:0014 H:0002 F:000f

Left-Root# sh spanning-tree internal event-history tree 0 interface port-channel 50

VDC02 MST0000 <port-channel50>

0) Transition at 497772 usecs after Tue Oct 20 17:42:01 2009

State: FWD Role: Root Age: 5 Inc: no [STP_PORT_STATE_CHANGE]


State: FWD Role: Root Age: 4 Inc: no [STP_PORT_ROLE_CHANGE]


State: BLK Role: Root Age: 5 Inc: no [STP_PORT_STATE_CHANGE]

...

Looking at past events…

This output can be easily limited to

necessary Vlan/Interface, but it

doesn’t dump the BPDU

Very chatty – use ‘debug logfile

<file>’ to redirect output to a file


STP inconsistencies

%STP-2-VPC_PEER_LINK_INCONSIST_BLOCK: vPC peer-link detected BPDU receive timeoutblocking port-channel11 VLAN0121.

When STP detects certain abnormal situations it may mark ports as inconsistent and block them to prevent forwarding loops

Root – Root Guard feature detected inconsistency (unwanted bridge tries to become root)

Loop – Loop Guard feature detected inconsistency (port becomes designated because no BPDUs are being received)

Bridge Assurance (BA)

(no BPDUs are received from remote side)

VPC Peer-link(any of above inconsistencies happened on VPC peer-link)


Handling Peer-Link STP inconsistencies on Primary switch

Primary SecondaryWhen peer-link STP inconsistency is detected on

primary switch the link will be put in ‘inconsistent’

STP state (effectively blocking state)

1

BPDUs are not sent on peer-link when it is

inconsistent. This is to allow secondary switch to

detect inconsistency and react

1

inco

nsi

sten

cy


Handling Peer-Link STP inconsistencies on Secondary switch

Primary Secondary

When peer-link STP inconsistency is detected on

secondary switch the peer link will be put in

‘inconsistent’ STP state (effectively blocking

state)

1

Respective vlans or MST instances are also

blocked on all VPCs

22

2

1inco

nsi

sten

cy

inco

nsi

sten

cy


Bridge assurance, Dispute & UDLD

BA is default enabled on Peer-Link (and recommended to remain enable), not recommended for VPCs unless Peer-Switch feature is used

Dispute is default enabled (for both RSTP and MST on VPC)

UDLD [normal mode] is recommended to take out bad links from channels (otherwise LACP takes ~100sec vs ~20 with UDLD)

Recommendation

Preferred BA + UDLD + Dispute (on all interswitch links when using Peer-switch) when all switches support this (nexus7000/5000 and cat6500/VSS do support)

Without Peer-switch BA should be kept only on Peer-Link (no BA/Loop guard on VPCs)

If preferred config is not supported use Loop Guard + UDLD(supported by all Cisco switches)


STP behavior upon VPC primary failure

Primary SecondaryOP-Primary

ROOT ROOTBackup

ROOT

Depending on control plane load it might take few

seconds for Op-primary to start sending BPDUs.

This might cause STP reconvergence on

connected switches hence increasing hello time

or peer-switch feature might be considered in

large deployments

Primary switch (STP root) fails1

Secondary switch becomes operational primary

and STP root

2

STP root port doesn’t change for access switch

nor any STP port states for VPCs, forwarding

continues1

2


STP behavior upon VPC primary recovery

SecondaryOP-Primary

ROOT ROOT

OP-Secondary

SYNC Backup

ROOT

Left switch comes back up1

Peer-Link comes back up2

VPC role is resolved as Operational-secondary3

Left switch has better STP priority becomes

STP root

4

STP root port of right switch will change and that

will trigger SYNC: all non-edge STP ports will be

temporarily blocked

5

Once sync is complete ports will resume

forwarding

1

23

4 5


VPC Peer-Switch feature

Primary Secondary

Both VPC switches originate BPDUs with preconfigured information. This allows to keep the same BPDU when primary fails/recovers no extra SYNC required avoid short interruption in forwarding described on previous slide is avoided

Both left and right switches consider themselves root

Both left and right switches send BPDUs all the time no need to raise hello time

Available 4.2(6) – 5.x software

spanning-tree vlan 1-1000 priority 8192vpc domain 1peer-switch

spanning-tree vlan 1-1000 priority 8192vpc domain 1peer-switch

ROOT ROOT


VPC Peer-Switch featurePrimary Secondary

left# sh span vlan 101

VLAN0101Spanning tree enabled protocol rstpRoot ID Priority 8293

Address 0023.04ee.be01This bridge is the root

...

Bridge ID Priority 8293 (priority 8192)Address 0023.04ee.be01

...

Interface Role Sts Cost Prio.Nbr Type---------------- ---- --- --------- -------- ---------------Po1 Desg FWD 1 128.4096 (vPC) P2pPo100 Root FWD 2 128.4195 (vPC peer-link)

left# sh vpc role | i macvPC system-mac : 00:23:04:ee:be:01 vPC local system-mac : 00:1b:54:c2:42:43

right# sh span vlan 101

VLAN0101Spanning tree enabled protocol rstpRoot ID Priority 8293

Address 0023.04ee.be01This bridge is the root

...

Bridge ID Priority 8293 (priority 8192)Address 0023.04ee.be01

...Interface Role Sts Cost Prio.Nbr Type---------------- ---- --- --------- -------- ---------------Po1 Desg FWD 1 128.4096 (vPC) P2pPo100 Desg FWD 2 128.4195 (vPC peer-link)

In Peer-Switch mode bridge-ID comes from system-mac as opposed to local mac in normal mode

ROOT ROOT


More information

show spanning-tree internal event-history all(allows to look back at past STP events, not included in sh tech)

sh tech stp(from both sides of VPC)

sh tech(from both sides of VPC, this will include in it ‘sh tech stp’, in case VPC is is non-default VDC collect also sh tech from VDC 1)


VPC Agenda

Initialization


Spanning Tree

Traffic forwarding

1st hop redundancy



Special case for forwarding

xx

x

PC A ends a packet to PC B1

MAC B is not known by left switch flood2

MAC B is not known by right switch flood3

B receives duplicate frames4

MAC A will be learned on wrong port on the lower

access switch blackholing traffic to A

5

Frames received on Peer-Link may not be flooded

out of VPCs

PC A

PC B

A ←

1

2 3

4

5A ↑ x


Special case for forwarding: VPC implementation

MAC B is not known by left switch flood1

Frames received from Peer-Link are never sent

out of VPC (except those without operational

ports on ingress switch)

Egress port ASICs will drop the frame

Frame is still flooded to devices that are solely

connected to egress switch3

This rule (called ‘VPC check’) stands for all traffic

(L2, L3, unicast, multicast, broadcast, flooded etc)

1

3

2

2

2

PC A

PC B


Summary: VPC traffic forwarding

√ √ X √

x


vPC view Layer 2 topology Layer 3 topology

Port-channel looks like a single L2 pipe.

Hashing will decide which link to chose

Layer 3 will use ECMP for northbound traffic

7k1 7k2

R

7k1 7k2

R

7k vPC

R

R could be any router, L3 switch or VSS

building a port-channel

VPC forwarding and L3 implication

R can Decide to send to 7k1 at L3 (next-hop = 7k1 if Po) and

uses link to 7k2 at L2 level !!!

Path is R 7k2 7k1 DROPPED (per VPC check) as

incoming on peer-link if it must be routed to another VPC


Router

7k1 7k2

Switch

Po1

Po2

Use L3 links to hook up routers and peer with a vPC domain

Don’t use L2 port channel to attach routers to a vPC domain unless you statically route to HSRP address

If both, routed and bridged traffic is required, use individual L3 links for routed traffic and L2 port-channel for bridged traffic

Use of peer-gateway does NOT change above recommendations

Router

Switch

L3 ECMP

Po2

Layer 3 and vPC Design update

PP

P

Routing Protocol Peer

Dynamic Peering Relationship

P

P


Layer 3 and VPC – consideration

Best : use Routed links from VPC pair to routers

Alternative : VPC in a pure L2 VDC and routing in a separate VDC

Do not make L3 routing protocol peering between VPC pair of switches on a VPC vlan.

May lead to routing frame towards Peer-link leading to drop per VPC-Check

If peering between VPC devices is needed, must be done outside of the peer link

Keep SVI interface administrative status in sync (both up or both down) – This is a type 2 consistency check


Special case for L2 learning

A ↓ Ax

A ↓

MAC A is learned on lower VPC1

PC A

PC BMAC A is learned on Peer-Link2

Frame destined to A arriving to right switch will be

sent to Peer-Link

3

Traffic should prefer local links when available

(traffic locality rule)

1

2

3


L2 learning: VPC implementation

A ↓ A ↓

MAC A is learned on lower VPC1

PC A

PC B

MAC addresses are never learned from traffic on

Peer-Link

Frame destined to A arriving to right switch will be

sent out of lower VPC3

1

2

3

Left switch sends a CFS message to right switch

telling about MAC A learned on lower VPC. Right

switch updates MAC address table

2

CFS message


TroubleshootingLayer 2

20.1.2.391.0.0.10

0013.1908.e246

Po50

Vlan 50

Po22

Vlan 20

nexus# sh mac address-table address 0013.1908.e246 vlan 50

VLAN MAC Address Type age Secure NTFY Ports---------+-----------------+--------+---------+------+----+------------------* 50 0013.1908.e246 dynamic 0 F F Po50

nexus# sh spanning-tree vlan 50 interface port-channel 50Mst Instance Role Sts Cost Prio.Nbr Type---------------- ---- --- --------- -------- --------------------------------MST0002 Desg FWD 200 128.4145 (vPC) P2p

nexus# sh hardware mac address-table 2 address 0013.1908.e246 vlan 50

Valid| PI | BD | MAC | Index | Stat| SW | Modi| Age | Tmr || | | | | ic | | fied| Byte| Sel |

-----+----+-------+---------------+--------+-----+----+-----+-----+-----+1 1 161 0013.1908.e246 0x00a36 0 3 0 141 1

nexus# sh system internal pixm info ltl 0x00a36 | i Eth.*,0x0a36 Eth2/36,

nexus# sh mac address-table address 0021.55e0.66c2 vlan 20

VLAN MAC Address Type age Secure NTFY Ports---------+-----------------+--------+---------+------+----+------------------* 20 0021.55e0.66c2 dynamic 660 F F Po22

nexus# sh spanning-tree vlan 20 interface port-channel 22Mst Instance Role Sts Cost Prio.Nbr Type---------------- ---- --- --------- -------- --------------------------------MST0000 Desg FWD 200 128.4117 (vPC) Network P2p

nexus# sh hardware mac address-table 1 address 0021.55e0.66c2 vlan 20Valid| PI | BD | MAC | Index | Stat| SW | Modi| Age | Tmr |

| | | | | ic | | fied| Byte| Sel |-----+----+-------+---------------+--------+-----+----+-----+-----+-----+1 1 18 0021.55e0.66c2 0x00a32 0 2 0 103 1

nexus# sh system internal pixm info ltl 0x00a32 | i Eth.*,0x0a32 Eth1/13, Eth1/14,

MAC addresses should point

to expected ports in expected

vlans (path towards source)

The ports should be in STP

forwarding mode

Hardware MAC address

table should be consistent

with software table

Finding port# for given index

VPC


TroubleshootingLayer 3

nexus# sh routing ip 20.1.2.3...20.1.2.3/32, ubest/mbest: 1/0

*via 20.1.1.240, Vlan20, [1/0], 03:48:59, static

nexus# sh ip arp 20.1.1.240Address Age MAC Address Interface20.1.1.240 00:02:17 0021.55e0.66c2 Vlan20

nexus# sh forwarding ip route 20.1.2.3 module 2...------------------+------------------+---------------------Prefix | Next-hop | Interface------------------+------------------+---------------------20.1.2.3/32 20.1.1.240 Vlan20

nexus# sh forwarding adjacency 20.1.1.240 module 2

IPv4 adjacency information

next-hop rewrite info interface-------------- --------------- -------------20.1.1.240 0021.55e0.66c2 Vlan20

nexus# sh int vl 20 | i addressHardware is EtherSVI, address is 0023.ac66.1a42

nexus# sh mac address-table address 0023.ac66.1a42 vlan 20

VLAN MAC Address Type age Secure NTFY Ports---------+-----------------+--------+---------+------+----+------------------G 20 0023.ac66.1a42 static - F F sup-eth1(R)

Is there route to

destination

Is the next hop resolved

Looking at module 2

because this is where

packets in question

should be received

Is adjacency consistent

with ARP

Router MAC must have

Gateway flag in order for

packet to be L3 switched

20.1.2.391.0.0.10

0013.1908.e246

Po50

Vlan 50

Po22

Vlan 20

VPC


Where given packet will be load-balanced

For equal-cost routes

nexus# sh routing hash 91.0.0.10 20.1.2.3Load-share parameters used for software forwarding:load-share mode: address source-destination port source-destinationUniversal-id seed: 0xcdb5769fHash for VRF "default"Hashing to path *20.1.1.3 (hash: 0x2a), for route:

20.1.2.3/32, ubest/mbest: 2/0*via 20.1.1.3, Vlan20, [1/0], 00:01:37, static*via 20.1.1.240, Vlan20, [1/0], 16:32:42, static

For port-channels

nexus# sh port-channel load-balance forwarding-path interface port-channel 22 dst-ip20.1.2.3 src-ip 91.0.0.10 vlan 20 module 2

Missing params will be substituted by 0's.

Module 2: Load-balance Algorithm: source-dest-ip-vlan

RBH: 0 Outgoing port id: Ethernet1/14

Load-balancing is configurable

under ‘ip load-sharing address’ in

default VDC and affects all VDCs

Load-balancing is configurable

under ‘port-channel load-balance’

in default VDC and affects all VDCs

Use ‘sh port-channel rbh-distribution’ to see which link sends traffic for which of 8 available load-balancing ‘buckets’


Hardware path packet dropsnexus# sh hardware internal errors all----------------------------------------Hardware errors as reported in module 1----------------------------------------

|------------------------------------------------------------------------|| Device:R2D2 Role:MAC ||------------------------------------------------------------------------|Instance:7ID Name Value Ports-- ---- ----- -----28688 aric_no_port_select_error 0000000000000002 1,3,5,7 I2...|------------------------------------------------------------------------|| Device:Ashburton Role:MAC Mod: 1 ||------------------------------------------------------------------------|Instance:03629 Egress Port-1 VSL Dropped Packet Count 0000000853635833 5 -3630 Egress Port-2 VSL Dropped Packet Count 0000000857893046 3 -...|------------------------------------------------------------------------|| Device:Naxos Role:MAC SECURITY ||------------------------------------------------------------------------|Instance:0ID Name Value Ports-- ---- ----- -----106 m1_fab_p25_txq_tc0_drop_count 00000000000012af 2 -...|------------------------------------------------------------------------|| Device:Metropolis Role:REWR ||------------------------------------------------------------------------|Instance:1ID Name Value Ports-- ---- ----- -----70 Krypton input controller zero portsel cnt 0000000000000038 18,20,22,24,26,28,30,32|------------------------------------------------------------------------|| Device:Lamira Role:L3 ||------------------------------------------------------------------------|Instance:0ID Name Value Ports-- ---- ----- -----93 CL2 Invalid Pkt count 00000008759cb9cb 1-32 I1...

#1 command to look for hardware

packet drops

Not every drop listed here is actual

data packet drop

Run several times to see if any

counters increase at rate similar to

traffic loss

To clear counters, use

‘clear statistics module-all device all’


VPC Agenda

Initialization


Spanning Tree

Traffic forwarding

1st hop redundancy



1st hop redundancy with VPC

MAC_A vMAC

IP A IP B

Router MAC1

0001.0002.0003

Virtual MAC

0000.0c07.ac00

Router MAC2

0005.0006.0007

Virtual MAC

0000.0c07.ac00

MAC_B vMAC

IP B IP A

PC A

PC B

HSRP

Each of VPC peers will L3 forward packets

destined to its respective Router MAC address

HSRP/VRRP/GLBP used for 1st hop redundancy

Both switches will L3 switch packets to vMAC

address as long as one of them is HSRP active or

HSRP standby.

If both switches are HSRP listening, they will not

L3 switch packets to vMAC


Left# sh hsrp briefInterface Grp Prio P State Active addr Standby addr Group addrVlan1 1 100 Standby 1.1.1.253 local 1.1.1.254

Left# sh mac address-table address 0000.0c07.ac01VLAN MAC Address Type age Secure NTFY Ports

---------+-----------------+--------+-----+------+------+-----------G 1 0000.0c07.ac01 static - False False sup-eth1(R)

Right# sh hsrp briefInterface Grp Prio P State Active addr Standby addr Group addrVlan1 1 100 Active local 1.1.1.252 1.1.1.254

Right# sh mac address-table address 0000.0c07.ac01VLAN MAC Address Type age Secure NTFY Ports

---------+-----------------+--------+-----+------+------+-----------G 1 0000.0c07.ac01 static - False False sup-eth1(R)

First hop redundancy troubleshooting

HSRP

Interface Vlan1ip address 1.1.1.252/24hsrp 1ip 1.1.1.254

Interface Vlan1ip address 1.1.1.253/24hsrp 1ip 1.1.1.254

Both peers will L3forward packets destined to vMac address as long as either peer in VPC domain is in ‘active’ or ‘standby’ state for corresponding group

Virtual mac address (vMac) will be installed in both peers

‘G’ (gateway) flag must be present on any MAC address for which the nexus is expected to L3forward packets

Only active will respond to ARP for VIP

standby active


1st hop issue with some devices

MAC_A vMAC

IP A IP B

Router MAC1

0001.0002.0003

Virtual MAC

0000.0c07.ac00

Router MAC2

0005.0006.0007

Virtual MAC

0000.0c07.ac00

PC A

Server B

Router MAC1 MAC_B

IP A IP B

MAC_B Router MAC1

IP B IP A

MAC_B Router MAC1

IP B IP A

X

Left VPC switch will receive the packet and

forward it to Server B, note Source MAC of

outgoing packet will be that of Router1

2

PC A sends a packet to Server B1

Server B responding to PC A will populate

destination MAC from source MAC of received

frame (this is wrong, it should use ARP)

3

If frame from BA will be load-balanced to right

switch the MAC address of Router1 will point to

Peer-Link and this is where the frame will be sent

4

Left switch will receive the frame from Peer-Link

and drop it

5

Why? Frames received from Peer-Link are never

sent out of VPC except those without operational

ports on ingress switch

(egress port ASICs will drop the frame)

1

2

3

4

5


Peer-Gateway : the workaround

PC A

Server B

MAC_B Router MAC1

IP B IP A

MAC_B Router MAC1

IP B IP A

With peer-gateway both peers will install router

MACs of each other in L2 table which will allow

them to L3 forward traffic destined to either

Router MAC

Server B responding to PC A will populate

destination MAC from source MAC of received

frame (this is wrong, it should use ARP)

1

Right switch will forward packet towards

destination

2

1

2

Router MAC1

0001.0002.0003

Virtual MAC

0000.0c07.ac00

Router MAC2

0005.0006.0007

Virtual MAC

0000.0c07.ac00

Router MAC1

0001.0002.0003

Router MAC2

0005.0006.0007

Virtual MAC

0000.0c07.ac00

Router MAC2

0005.0006.0007

Router MAC1

0001.0002.0003

Virtual MAC

0000.0c07.ac00


Peer-Gateway : the implications

Router MAC1

0001.0002.0003

Router MAC2

0005.0006.0007

Virtual MAC

0000.0c07.ac00

Router MAC2

0005.0006.0007

Router MAC1

0001.0002.0003

Virtual MAC

0000.0c07.ac00

X

MAC_B Router MAC1

IP TOP IP LEFT, TTL 1

Top device attempts to establish OSPF adjacency

with the left switch

1

If peer-gateway is enabled in VPC domain and

OSPF unicast packet will be load-balanced to the

right switch, this packet will be dropped

2

Why? Right switch will try to L3-switch the

unicast packet (because RouterMAC1 is marked

as gateway MAC and destination IP is not local)

As packet has TTL==1 it will be dropped

Same applies to any other protocol that uses

unicast packets with TTL==1 entering right switch

but destined to left switch (or vise versa)

Routing protocol peering with devices attached to

VPC domain via SVI interface is not supported

Routed interface should be used in this case

1

2


More information

sh mac address-table <address>(L2 entry for given MAC )

sh hardware mac address-table <mod> address <address>(hardware L2 entry for given MAC should be consistent with above)

sh system internal l2fm l2dbg macdb address <addr>(history of changes for given mac address)

sh tech hsrp(from both sides of VPC)


VPC Agenda

Initialization


Spanning Tree

Traffic forwarding

1st hop redundancy



Once (S1,G) traffic starts arriving, VPC peers will

resolve which one will be forwarder for that (S,G):

peer with best metric to source or primary in a tie

(this mechanism is specific to PIM in VPC mode,

normally PIM would use assert)

IP Multicast with VPC

Receiver

Source S1

Receiver sends IGMP report (join)

DR (left peer) sends PIM Join to RP

Only forwarder will have OIFs populated in (S,G)

the non-forwarder won’t have VPC SVIs in OIF list

RP

Primary 2ndary

CFS:IGMP

IGMP join

IGMP is encapsulated in CFS and sent to left peer

(*,G)VPC (*,G)VPC

(S1,G)VPC (S1,G)null

Access switch sends join to right VPC peer

Right VPC peer creates (*,G) adds VPC to OIF (as

proxy-DR)

Left peer (DR) creates (*,G) adding VPC to OIF

DR

Forwarder will send a copy of frame to the peer-

link for receivers single-connected to other peer

Proxy-DR

Goal is to allow peer that 1st ‘sees’ source traffic to forward it to receivers behind VPC


IP Multicast with VPCPrebuilt-SPT

Source S1

With ‘ip pim pre-build-spt’ proxy-DR will also send

a PIM Join to source/RP to draw the traffic

RP

Primary 2ndary

(*,G)VPC (*,G)VPC

(S1,G)VPC (S1,G)null

In case of DR failure proxy-DR becomes DR and

posts OIF-list from (*,G) to (S,G), but it will also

need to pull traffic from RP/source which delays

recovery

DR Traffic pulled by proxy-DR will be dropped until it

becomes DR – provision uplink accordingly (if

pre-build-spt is used)

Receiver


IP Multicast with VPCsource behind VPC

Source S1

RP

Primary 2ndary

(*,G)VPC2 (*,G)VPC2

(S1,G)VPC2 (S1,G)VPC2

When Source is behind VPC both DR and Proxy-

DR will add OIFs for the group to (S,G)

This is because either peer can receive source

traffic and need to be able to send it to receivers

behind VPCs without crossing peer-link (to keep

traffic locality and to avoid dropping the traffic by

VPC check)

Receiver

VPC1 VPC2

Going to Left switch from Source

Or going to Right switch from Source


For sources behind VPC both peers will forward as they have no control on which one will get the traffic…

VPC1# sh ip pim internal vpc rpf

Source: 10.0.1.1Pref/Metric: 110/21Source role: primaryForwarding state: Win (forwarding)

VPC1# sh ip pim internal vpc rpf

Source: 1.1.1.1Pref/Metric: 0/0Source role: primaryForwarding state: Win-force (forwarding)

Peers do ‘metrics exchange’ over CFS for each new source

Peer that has better metric to source or primary will be forwarder

Which of VPC peers will be forwarder


Are packets being switched by this entry?

Nexus# show ip mroute 239.1.2.3

(*, 239.1.2.3/32), uptime: 06:46:05, igmp pim ip staticIncoming interface: Vlan36, RPF nbr: 36.0.0.3Outgoing interface list: (count: 2)Ethernet2/43, uptime: 03:01:36, staticVlan37, uptime: 06:46:05, igmp

(33.0.0.33/32, 239.1.2.3/32), uptime: 06:46:05, ip pim mribIncoming interface: Vlan36, RPF nbr: 36.0.0.3Outgoing interface list: (count: 2)Ethernet2/43, uptime: 03:01:36, mribVlan37, uptime: 06:46:04, mrib

control plane state for this group

where information came from

stable?

RPF interface

Nexus# show ip mroute 239.1.2.3 summary software-forwarded

Total number of routes: 3Total number of (*,G) routes: 1Total number of (S,G) routes: 1Total number of (*,G-prefix) routes: 1Group count: 1, rough average sources per group: 1.0

Group: 239.1.2.3/32, Source count: 1Source packets bytes aps pps bit-rate oifs(*,G) 0 0 0 0 0.000 bps 2

sw-pkts: 033.0.0.33 5046908 252345396 49 200 80.053 kbps 2

sw-pkts: 1

Is traffic being switched for this group?

counters updated once ~1 minute

packets forwarded in software

average packet size

VPC mcast: following packet flow

Nexus# show ip igmp snooping groups vlan 37Type: S - Static, D - Dynamic, R - Router port

Vlan Group Address Ver Type Port list37 */* - R Vlan3737 239.1.2.3 v2 D Eth2/8

where are receivers on this vlan?


Following the flow: forwarding information

Nexus# show forwarding multicast route group 239.1.2.3

slot 1=======

(*, 239.1.2.3/32), RPF Interface: Vlan36, flags: GReceived Packets: 0 Bytes: 0Number of Outgoing Interfaces: 2Outgoing Interface List Index: 4

Vlan37 Outgoing Packets:0 Bytes:0Ethernet2/43 Outgoing Packets:N/A Bytes:N/A

(33.0.0.33/32, 239.1.2.3/32), RPF Interface: Vlan36, flags:Received Packets: 5723369 Bytes: 366295616Number of Outgoing Interfaces: 2Outgoing Interface List Index: 4

Vlan37 Outgoing Packets:0 Bytes:0Ethernet2/43 Outgoing Packets:N/A Bytes:N/A

slot 2=======

(*, 239.1.2.3/32), RPF Interface: Vlan36, flags: GReceived Packets: 0 Bytes: 0Number of Outgoing Interfaces: 2Outgoing Interface List Index: 4

Vlan37 Outgoing Packets:5725816 Bytes:366452224Ethernet2/43 Outgoing Packets:3032294 Bytes:194066816

(33.0.0.33/32, 239.1.2.3/32), RPF Interface: Vlan36, flags:Received Packets: 0 Bytes: 0Number of Outgoing Interfaces: 2Outgoing Interface List Index: 4

Vlan37 Outgoing Packets:5725816 Bytes:366452224Ethernet2/43 Outgoing Packets:3032294 Bytes:194066816

This is platform independent forwarding

information

Ingress linecard entry

Egress linecard entry

Counters are updated once per ~1minute

Counters between ingress/egress do not have to

match, as information is collected not at the same

exact time, receiver might join after the entry was

created etc


When traffic arrives via VPC How to find which slot receives the

S,G flow when ingress interface is port-channel scattered across several modules?

show forwarding multicast route group <g> source <s>

Nexus# show forwarding multicast route group 239.1.1.1 source 1.0.1.2 | i Received|slotslot 1

Received Packets: 0 Bytes: 0slot 2

Received Packets: 727203 Bytes: 487290999

VPC domain 100

VPC


Following the flow: hardware entries

Nexus# show system internal forwarding ipv4 multicast route group 239.1.2.3 source 33.0.0.33 detail

slot 1

(33.0.0.33/32, 239.1.2.3/32), Flags: *SLamira: 1, HWIndex: 0x2200, VPN: 1RPF Interface: Vlan36, LIF: 0x45, PD oiflist index: 0x2ML3 Adj Idx: 0xa016, MD: 0x2003, MET0: 0x2004, MET1: 0x2004, MTU Idx: 0x1Metro Instance: 0Dev: 1 Index: 0xa019 Type: MDT elif: 0xc0002

dest idx: 0x7fe7 recirc-dti: 0xe20000Metro Instance: 1Dev: 1 Index: 0xa019 Type: MDT elif: 0xc0002



dest idx: 0x7fe7 recirc-dti: 0xe20000

slot 2

(33.0.0.33/32, 239.1.2.3/32), Flags: *SLamira: 1, HWIndex: 0x2200, VPN: 1RPF Interface: Vlan36, LIF: 0x45, PD oiflist index: 0x2ML3 Adj Idx: 0xa026, MD: 0x2003, MET0: 0x2004, MET1: 0x2004, MTU Idx: 0x1Metro Instance: 0Dev: 1 Index: 0xa029 Type: MDT elif: 0xc0002

dest idx: 0x7fe7 recirc-dti: 0xe20000Dev: 1 Index: 0x6046 Type: OIF elif: 0x80046 Vlan37

dest idx: 0x0 smac: 001b.54c2.4241Metro Instance: 1Dev: 1 Index: 0xa029 Type: MDT elif: 0xc0002

dest idx: 0x7fe7 recirc-dti: 0xe20000Dev: 1 Index: 0xa028 Type: OIF elif: 0x84029 Ethernet2/43

dest idx: 0x44c smac: 001b.54c2.4241

Ingress forwarding engine (FE)

replicates packets to receivers on that

linecard and creates ‘distribution copy’

of the packet for other linecards

MET pointers (MD + MET0)

RPF interface read from entry

TCAM Entry

Decoded MET chain (on ingress there

is only MD copy created)

Egress linecard will receive distribution

copy and replicate it to receivers (using

MET1 pointer) connected to the card

MET1 on egress linecard points to

receivers on vlan37 and e2/43


Are there drops in forwarding path?

Start looking from Ingress module

Nexus# show hardware internal errors module 1----------------------------------------Hardware errors as reported in module 1----------------------------------------...|------------------------------------------------------------------------|| Device:Lamira Role:L3 Mod: 1 || Last cleared @ Thu Apr 8 12:57:37 2010| Device Statistics Category :: ERROR|------------------------------------------------------------------------|Instance:0ID Name Value Ports-- ---- ----- -----259 L3 Fib Miss Pkt ctr 0000000000000007 1-32 I1262 L3 Non-Rpf Drop Pkt ctr 0000000000125617 1-32 I1319 NF2 V4 IPMAC Lkup Error 0000000000272277 1-32 I1455 Exception cause: DROP (Unicast) 0000000000025510 1-32 I1465 Exception cause: DROP (Multicast) 0000000000226148 1-32 I1

Always take several snapshots and look for drops that grow coherently with [suspected] multicast traffic drops

There are always some drops shown by above command – this doesn’t always mean the actual network packets are dropped. Some of these are diag packets, some are packets that are dropped on blocked ports, extra floods etc


Wrapping UP


VPC compared to VSS

VPC VSS

Control Plane Distributed Redundant Centralized

SSO InTRAchassis (w/2 sups) InTERchassis

HSRP/VRRP 2 routers, each forwards traffic

Inherent 1st hop redundnancy, no need for HSRP

Traffic locality Yes Yes

Failover time Subsecond Subsecond

Configuration synchronization

Separate configs, key parameters checked via CFS

Using IOS redundancy framework

Dual active detection

via the Peer-Keepalive link via L2 hellos and PAgP+


VPC/VSS: summary

Remember about the implications of 2 control planes and 2 data planes active at the same time

Pay special attention to configuration and operational consistency, not only to what is enforced, but also L3interfaces including their operational state, FHRPconfig, ACL config, queueing config

Troubleshoot like a standalone switch 1st, then dive into VPC/VSS specifics: main one being traffic locality

Both VPC and VSS

• simplify logical Layer 2 topology

• use Traffic Locality for efficient shortest path

forwarding


Also browse on-site Cisco Store for suitable reading

BRKCRS-1930Recommended Reading


We value your feedback - don't forget to complete your online session evaluations after each session. Complete 4 session evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Networkers 20th Anniversary t-shirt.

All surveys can be found on our onsite portal and mobile website: www.ciscoliveeurope.com/connect/mobi/login.ww

You can also access our mobile site and complete your evaluation from your mobile phone:

1. Scan the Access Code(See http://tinyurl.com/qrmelist for software,

alternatively type in the access URL)

2. Login

3. Complete and Submit the evaluation

Please complete your Session Survey

http://www.ciscoliveeurope.com/connect/mobi/login.ww

http://tinyurl.com/qrmelist

Documents

VPC & VSS: Operation and Troubleshooting - Add …docshare01.docshare.tips/files/30199/301996596.pdf · VSS on bridging and routing Learn how to troubleshoot ... Cisco Public 7 VSS