© 2007 Cisco Systems, Inc. All rights reserved.Cisco ConfidentialBRKRST-3468 1 Understanding Virtual Switching System on the Cisco Catalyst 6500 Cisco

© 2007 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialBRKRST-3468 1

Understanding Virtual Switching System on the Cisco Catalyst 6500

Cisco Users Group

Tommy Dodd ([email protected])

SE, Cisco Systems


Current Network ChallengesEnterprise Campus

Traditional Enterprise Campus deployments have been designed in such a way that allows for scalability, differentiated services and high availability. However they also face many challenges, some of which are listed in the below diagram…

Access

L2/L3 Distribution

L3 Core

FHRP, STP, Asymmetric routing,Policy Management

Extensive routing topology, Routing reconvergence

Single active uplink per VLAN (PVST), L2 reconvergence


Current Network ChallengesData Center

Traditional Data Center designs are requiring ever increasing Layer 2 adjacencies between Server nodes due to prevalence of Virtualization technology. However, they are pushing the limits of Layer 2 networks, placing more burden on loop-detection protocols such as Spanning Tree…

L2/L3 Core

L2 Distribution

L2 Access

Dual-Homed Servers to single switch, Single active uplink per VLAN (PVST), L2 reconvergence

Single active uplink per VLAN (PVST), L2 reconvergence, excessive BPDUs

FHRP, HSRP, VRRPSpanning TreePolicy Management


Virtual Switching SystemVirtual Switch System is a new technology break through for the Catalyst 6500 family…


Introduction to Virtual SwitchConcepts


Virtual Switch SystemEnterprise Campus

A Virtual Switch-enabled Enterprise Campus network takes on multiple benefits including simplified management & administration, facilitating greater high availability, while maintaining a flexible and scalable architecture…

Access

L2/L3 Distribution

L3 Core

No FHRPsNo Looped topologyPolicy Management

Reduced routing neighbors, Minimal L3 reconvergence

Multiple active uplinks per VLAN, No STP convergence


Virtual Switch ArchitectureVSL Initialization

Before the Virtual Switch domain can become active, the Virtual Switch Link (VSL) must be brought online to determine Active and Standby roles. The initialization process essentially consists of 3 steps:

Role Resolution Protocol (RRP) used to determine compatible Hardware and Software versions to form the VSL as well as determine which switch becomes Active and Hot Standby from a control plane perspective

Role Resolution Protocol (RRP) used to determine compatible Hardware and Software versions to form the VSL as well as determine which switch becomes Active and Hot Standby from a control plane perspective

LMPLMP LMPLMP

RRPRRPRRPRRP

Link Management Protocol (LMP) used to track and reject Unidirectional Links, Exchange Chassis ID and other information between the 2 switches Link Management Protocol (LMP) used to track and reject Unidirectional Links, Exchange Chassis ID and other information between the 2 switches

Link Bringup to determine which ports form the VSLLink Bringup to determine which ports form the VSL1

2

3


Virtual Switch ArchitectureLink Bringup

Pre-Parse ConfigSwitch 1




Each member of the Virtual Switch domain must determine which links are candidate for VSL very early on in the bootup cycle. The Switch Processor (SP) pre-parses the configuration to determine which links are configured for VSL…

System detected Virtual Switch configuration... Interface TenGigabitEthernet 1/5/4 is member of PortChannel 1 Interface TenGigabitEthernet 1/5/5 is member of PortChannel 1




The SP will then bring up the line card/s where the VSL is configured, download the required configuration and initiate Link Management Protocol (LMP)


Virtual Switch ArchitectureLink Management Protocol (LMP)

LMP runs on each individual link that is part of the VSL, and is used to program information such as member details, forwarding indices, as well as perform the following checks:

LMPLMP LMPLMP

LMPLMP LMPLMP

Verify neighbor is Bi-Directional

Ensure the member is connected to another Virtual Switch

Transmit and receive keepalives to maintain health of the member and the VSL

1

2

3

After successful LMP negotiation, a Peer Group (PG) is formed which is a collection of all VSL members that connects to the same VS. For each PG, a Peer Group Control Link (PGCL) is elected to carry further control information…


Virtual Switch ArchitectureRole Resolution Protocol (RRP)

RRP also runs on each individual link of the VSL. It will run on each Peer Group and will elect one of them to resolve the role between the 2 Virtual Switches. It is also part of VSLP and performs the following functions:

Determine whether hardware and software versions allow a Virtual Switch to form

Determine which chassis will become Active and Hot Standby from a control plane perspective

1

2

RRPRRP RRPRRP

RRPRRP RRPRRPVSL


Virtual Switch ArchitectureVSLP Ping

A new Ping mechanism has been implemented in VSS mode to allow the user to objectively verify the health of the VSL itself. This is implemented as a VSLP Ping…

VSL

Switch 1 Switch 2

VSLPVSLP VSLPVSLP

VSLPVSLP VSLPVSLP

vss#ping vslp output interface tenGigabitEthernet 1/5/4

Type escape sequence to abort.Sending 5, 100-byte VSLP ping to peer-sup via output port 1/5/4, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 12/12/16 msvss#

vss#ping vslp output interface tenGigabitEthernet 1/5/4

Type escape sequence to abort.Sending 5, 100-byte VSLP ping to peer-sup via output port 1/5/4, timeout is 2 seconds:!!!!!Success rate is 100 percent (5/5), round-trip min/avg/max = 12/12/16 msvss#

The VSLP Ping operates on a per-physical interface basis and parameters such as COUNT, DESTINATION, SIZE, TIMEOUT may also be specified…


Virtual Switch ArchitectureVSL Configuration Consistency Check

After the roles have been resolved through RRP, a Configuration Consistency Check is performed across the VSL switches to ensure proper VSL operation. The following items are checked for consistency:

Switch Virtual Domain IDSwitch Virtual Domain ID

Switch Virtual Node TypeSwitch Virtual Node Type

Switch PrioritySwitch Priority

Switch PreemptSwitch Preempt

VSL Port Channel Link IDVSL Port Channel Link ID

VSL Port state, interfaces…VSL Port state, interfaces…

Power Redundancy modePower Redundancy mode

Power Enable on VSL cardsPower Enable on VSL cards

Note that if configurations do not match, the standby switch will revert to RPR mode, disabling all non-VSL interfaces…

Note that if configurations do not match, the standby switch will revert to RPR mode, disabling all non-VSL interfaces…


Virtual Switch ArchitectureForwarding Operation

In Virtual Switch Mode, while only one Control plane is active, both Data Planes (Switch Fabric’s) are active, and as such, each can actively participate in the forwarding of data …

Virtual Switch Domain

Switch 1 - Control Plane Active Switch 2 - Control Plane Hot Standby


Switch 1 - Data Plane Active Switch 2 - Data Plane Active


Virtual Switch ArchitectureVirtual Switch Domain

A Virtual Switch Domain ID is allocated during the conversion process and represents the logical grouping the 2 physical chassis within a VSS. It is possible to have multiple VS Domains throughout the network…

The configurable values for the domain ID are 1-255. It is always recommended to use a unique VS Domain ID for each VS Domain throughout the network… The configurable values for the domain ID are 1-255. It is always recommended to use a unique VS Domain ID for each VS Domain throughout the network…

VS Domain 10

VS Domain 20 VS Domain 30


Virtual Switch ArchitectureRouter MAC Address

In a standalone Catalyst 6500 system, the router MAC address is derived from the Chassis MAC EEPROM and is unique to each Chassis. In a Virtual Switch System, since there is only a single routing entity now, there is also only ONE single router MAC address…

Router MAC = 000f.f8aa.9c00Router MAC = 000f.f8aa.9c00

The MAC address allocated to the Virtual Switch System is negotiated at system initialization. Regardless of either switch being brought down or up, the same MAC address will be retained such that neighboring network nodes and hosts do not need to re-ARP for a new address.


EtherChannel Concepts


Etherchannel ConceptsAn Etherchannel combines multiple physical links into a single logical link. Ideal for load sharing or link redundancy – can be used by both layer 2 and Layer 3 subsystems…

Physical ViewMultiple ports are defined as being

part of an Etherchannel

group

Logical ViewSubsystems running

on the switch only see one logical link

An Etherchannel can be defined on Ethernet, Fast Ethernet, Gigabit Ethernet or 10 Gigabit Ethernet Ports

An Etherchannel can be defined on Ethernet, Fast Ethernet, Gigabit Ethernet or 10 Gigabit Ethernet Ports


Etherchannel ConceptsMultichassis EtherChannel (MEC)

Prior to Virtual Switch, Etherchannels were restricted to reside within the same physical switch. In a Virtual Switch environment, the 2 physical switches form a single logical network entity - therefore Etherchannels can now also be extended across the 2 physical chassis…

Regular Etherchannel on single chassis Multichassis EtherChannel across 2 VSL-enabled Chassis

Virtual Switch Virtual Switch

Both LACP and PAGP Etherchannel protocols and Manual ON modes are

supported…

Both LACP and PAGP Etherchannel protocols and Manual ON modes are

supported…


Etherchannel ConceptsMultichassis EtherChannel

Support for Etherchannel management is performed by the Control plane on the Active Switch in the Virtual Switch Domain…

Standby Control Plane

Active Control Plane

MEC links on both the switches in the VS domain are managed by PAgP or LACP

running on the Master Switch via internal control messages. PAgP or LACP packets destined to a MEC link on the standby core

will be sent across VSL

MEC links on both the switches in the VS domain are managed by PAgP or LACP

running on the Master Switch via internal control messages. PAgP or LACP packets destined to a MEC link on the standby core

will be sent across VSL


Etherchannel ConceptsEtherchannel Hash for MEC

Deciding on which link of a Multi-chassis Etherchannel to use in a Virtual Switch is skewed in favor towards local links in the bundle - this is done to avoid overloading the Virtual Switch Link (VSL) with unnecessary traffic loads…

Link A1 Link B2

Blue Traffic destined for the Server will result in Link A1 in the MEC link bundle being chosen as the destination path…

Orange Traffic destined for the Server will result in Link B2 in the MEC link bundle being chosen as the destination path…

Server



Localizing the decision to use a link in the bundle that is resident on the local Switch (thus avoiding forwarding over the VSL) is done as follow…

Virtual Switch The BUNDLE_SELECT register in the port ASIC is programmed to see only the local links of the Etherchannel bundle even though links may exist in the same bundle are resident in the VS peer chassis…

This behavior is fixed and cannot be changed by any configuration option…

NOTE: If all links in the local bundle go down, then the BUNDLE_SELECT register is programmed to point packets to the VSL…



RBH values are reprogrammed for each core to reflect only the local links that are in the Etherchannel bundle…

Virtual SwitchSwitch 1 Switch 2

RBH (No MEC)8 Link Bundle Example

RBH (No MEC)8 Link Bundle Example

RBH (for MEC)8 Link Bundle Example

RBH (for MEC)8 Link Bundle Example

Bit 7Bit 7 Link 1Link 1Bit 6Bit 6 Link 2Link 2Bit 5Bit 5 Link 3Link 3Bit 4Bit 4 Link 4Link 4Bit 3Bit 3 Link 5Link 5Bit 2Bit 2 Link 6Link 6Bit 1Bit 1 Link 7Link 7Bit 0Bit 0 Link 8Link 8

Bit 7Bit 7 Link 1Link 1Bit 6Bit 6 Link 1Link 1Bit 5Bit 5 Link 2Link 2Bit 4Bit 4 Link 2Link 2Bit 3Bit 3 Link 3Link 3Bit 2Bit 2 Link 3Link 3Bit 1Bit 1 Link 4Link 4Bit 0Bit 0 Link 4Link 4

1 2 3 4 5 6 7 8

MEC


Hardware Requirements


Hardware RequirementsSupervisor

In order to enable the Virtual Switch feature and configure the Virtual Switch Links (VSL) between 2 Catalyst 6500 chassis, the new Catalyst 6500 Virtual Switching Supervisor 720 is required to be used. It is the only Supervisor that will support VSS as it supports both the new PFC3C/XL forwarding engine…

The PFC3C/XL contains new hardware to support the extra LTL indices and mappings required to forward traffic across multiple physical chassis, lookup enhancements as well as MAC address table handling enhancements…

VS-S720-10G-3C/XL


Hardware RequirementsVSL-Capable Interfaces

The VSL is a special link that requires extra headers to be imposed onto the frame. These require new port ASICs that exist only on the 10 GigabitEthernet interfaces on the following modules…

These interfaces are based off the new R2D4 ASIC, allowing for frames across the VSL to be encapsulated / de-encapsulated with the VSH…

Supervisor 720-10G

WS-X6708-10G-3C/XL

Note that these interfaces may also be used as standard network interfaces


Hardware RequirementsOther Supported Modules…

Other modules that may exist in the VSL domain include all CEF720 and dCEF720 cards (WS-X67xx-series), as well as SVC-NAM-1 and SVC-NAM-2. Classic, CEF256 and dCEF256 cards are not supported…

CEF720, dCEF720

and NAM only

CEF720, dCEF720

and NAM only


Hardware RequirementsDistributed Forwarding Cards

Distributed Forwarding Cards (DFCs) improve the performance of the Catalyst 6500 by offloading the lookup processing from the PFC to the ingress linecard. Only DFC3C or DFC3CXL is supported in a Virtual Switch domain. If DFCs are not used on CEF720 modules, a Centralized Forwarding Card (CFC) must be installed in its place…

Note that if a lower revision DFC (3A, 3B or 3BXL) is used in a VSL domain, the system will fall to a lowest common

denominator mode which will not allow support for VSL…

Note that if a lower revision DFC (3A, 3B or 3BXL) is used in a VSL domain, the system will fall to a lowest common

denominator mode which will not allow support for VSL…


Conversion Process


Conversion ProcessConversion to VSS

The conversion process requires configuration of both switches that will form part of the Virtual Switch Domain and requires a reboot on the part of both switches during the conversion…



For the purposes of this explanation - let’s assume the following setup is required…

Switch - 1 Switch - 2

VSL Link Bundle

T5/4

T5/5

T5/4

T5/5

Port-Channel 1 Port-Channel 2

Switch Virtual Domain #100



Configuration for the conversion takes the following path…


Router(config)#host VSSVSS(config)#switch virtual domain 100VSS(config-vs-domain)#switch 1VSS(config-vs-domain)#exit

VSS(config)#interface port-channel 1VSS(config-if)#switch virtual link 1

VSS(config-if)#interface tenG 5/4 VSS(config-if)#channel-group 1 mode onVSS(config-if)#interface tenG 5/5 VSS(config-if)#channel-group 1 mode onVSS(config-if)#^ZVSS #

Router(config)#host VSSVSS(config)#switch virtual domain 100VSS(config-vs-domain)#switch 2VSS(config-vs-domain)#exit

VSS(config-if)#interface port-channel 2VSS(config-if)#switch virtual link 2

VSS(config-if)#interface tenG 5/4 VSS(config-if)#channel-group 2 mode onVSS(config-if)#interface tenG 5/5 VSS(config-if)#channel-group 2 mode onVSS(config-if)#^ZVSS #





VSS#switch convert mode virtual

This command will convert all interfacenames to naming convention "interface-typeswitch-number/slot/port",save the running config to startup-config and reload the switch.Do you want proceed? [yes/no]: yesConverting interface namesBuilding configuration...

AT THIS POINT THE SWITCH WILL REBOOT

VSS#switch convert mode virtual

This command will convert all interfacenames to naming convention "interface-typeswitch-number/slot/port",save the running config to startup-config and reload the switch.Do you want proceed? [yes/no]: yesConverting interface namesBuilding configuration...

AT THIS POINT THE SWITCH WILL REBOOT





SWITCH CONSOLE OUTPUT

<…snip…>System detected Virtual Switch configuration... Interface TenGigabitEthernet 1/1/4 is member of PortChannel 1 Interface TenGigabitEthernet 1/1/5 is member of PortChannel 1

<…snip…>00:00:22: %PFREDUN-6-ACTIVE: Initializing as ACTIVE processor for this switch

<…snip…>00:00:28: %VSL_BRINGUP-6-MODULE_UP: VSL module in slot 1 switch 1 brought upInitializing as Virtual Switch active

SWITCH CONSOLE OUTPUT

<…snip…>System detected Virtual Switch configuration... Interface TenGigabitEthernet 2/1/4 is member of PortChannel 2 Interface TenGigabitEthernet 2/1/5 is member of PortChannel 2

<…snip…>00:00:23: %PFREDUN-6-ACTIVE: Initializing as ACTIVE processor for this switch

<…snip…>00:00:28: %VSL_BRINGUP-6-MODULE_UP: VSL module in slot 1 switch 2 brought upInitializing as Virtual Switch standby





VSS#sh switch virtual role

Switch Switch Status Preempt Priority Role Session ID Number Oper(Conf) Oper(Conf) Local Remote------------------------------------------------------------------LOCAL 1 UP FALSE(N) 110(110) ACTIVE 0 0 REMOTE 2 UP FALSE(N) 100(100) STANDBY 4217 6561

In dual-active recovery mode: No

VSS-sdby>enStandby console disabled

VSS-sdby>

Both switches are now converted with Switch 1 as the Master (Active) and Switch 2 as the Standby

Switch 2 console is now disabled for normal console activity…


Conversion ProcessBoot-up Priority

Normal operation is for first switch to boot to assume VS Active role - this behavior can be changed allowing a pre defined switch to assume Active role by specifying a priority (higher priority uses a higher number)…

Virtual Switch







Switch 1 Switch 2


Operational Management


Operational ManagementVirtual Switch CLI

Multiple console interfaces exist within a Virtual Switch Domain, but only the active RP/SP consoles are enabled for command interaction…


Operational ManagementReloading the VSS

Should there be a requirement to reload the entire Virtual Switch System (both chassis), the command “reload” can be used to accomplish this task…

vss#reloadProceed with reload? [confirm]

1d04h: %SYS-5-RELOAD: Reload requested by console. Reload Reason: Reload Command.

****** --- SHUTDOWN NOW ---***

1d04h: %SYS-SP-5-RELOAD: Reload requestedSystem Bootstrap, Version 8.5(1)Copyright (c) 1994-2006 by cisco Systems, Inc.Cat6k-Sup720/SP processor with 1048576 Kbytes of main memory<…snip…>

vss#reloadProceed with reload? [confirm]

1d04h: %SYS-5-RELOAD: Reload requested by console. Reload Reason: Reload Command.

****** --- SHUTDOWN NOW ---***

1d04h: %SYS-SP-5-RELOAD: Reload requestedSystem Bootstrap, Version 8.5(1)Copyright (c) 1994-2006 by cisco Systems, Inc.Cat6k-Sup720/SP processor with 1048576 Kbytes of main memory<…snip…>


Operational ManagementReloading a member of the VSS

It is also possible to reload each chassis individually by specifying the Switch ID assigned through the following command set…

vss#redundancy reload shelf ? <1-2> shelf id <cr>

vss#redundancy reload shelf 2Reload the entire remote shelf[confirm]Preparing to reload remote shelf

vss#

vss#redundancy reload shelf ? <1-2> shelf id <cr>

vss#redundancy reload shelf 2Reload the entire remote shelf[confirm]Preparing to reload remote shelf

vss#


Operational ManagementSetting the System-wide PFC Mode

Only PFC/DFC 3C/CXL are supported in a VSS. However, it is possible to mix modules in a 3C and 3CXL system, bearing in mind that the system will take the lowest common denominator as the system-wide PFC mode. A new CLI has been implemented to allow the user to pre-configure the system mode to prevent modules from not powering up…

vs-vsl#conf tEnter configuration commands, one per line. End with CNTL/Z.vs-vsl(config)#platform hardware vsl pfc mode pfc3c vs-vsl(config)#^Zvs-vsl#

vs-vsl#conf tEnter configuration commands, one per line. End with CNTL/Z.vs-vsl(config)#platform hardware vsl pfc mode pfc3c vs-vsl(config)#^Zvs-vsl#

vs-vsl#sh platform hardware pfc mode PFC operating mode : PFC3CConfigured PFC operating mode : PFC3Cvs-vsl#

vs-vsl#sh platform hardware pfc mode PFC operating mode : PFC3CConfigured PFC operating mode : PFC3Cvs-vsl#


Operational Management SNMP Support for VSS

The SNMP process for a VSS necessitates support for “Put’s” and “Get’s” across 2 physical chassis, changes to existing MIB’s and support for a new MIB…


Switch 1 - Active Switch 2 - Standby

SNMP Process Active SNMP Process Inactive

SNMP Server

SNMP Get’sSNMP Put’s

SNMP Modified MIB’s

SNMP Modified MIB’s

SNMP New MIB’sSNMP New MIB’s


Operational Management SNMP Modified MIB’s

The following MIB’s have been modified to allow the collection of data in a Virtual Switch configuration…

MIB Name Description of Change

CISCO-LAG-MIB Extended to support 6000 ports

CISCO-EXT-BRIDGE-MIB Supports extension of BRIDGE-MIB (which is a standard and cannot be changed. Extended to support up to 6000 ports.

CISCO-VLAN-MEMBERSHIP-MIB Extended to support 6000 ports

CISCO-ENVMON-MIB Virtual Switch Chassis number will be included in the “Description” field

CISCO-STACK-MIB No longer supported. Use ENTITY-MIB instead.

CISCO-OLD-CHASSIS-MIB No longer supported. Use ENTITY-MIB instead.

CISCO-CAT6K-CROSSBAR-MIB Support Standby Core with a different naming scope

Virtual Switch


Operational Management New Virtual Switch MIB

CISCO-VIRTUAL-SWITCH-MIB has been defined to support SNMP access to the Virtual Switch Configuration - the following MIB variables are accessible to an SNMP manager…

CISCO-VIRTUAL-SWITCH-MIBCISCO-VIRTUAL-SWITCH-MIB

cvsGlobalObjects - Domain #, Switch #, Switch Mode

cvsCoreSwitchConfig - Switch Priority and Preempt

cvsChassisTable - Chassis Role and Uptime

cvsVSLConnectionTable - VSL Port Count, Operational State

cvsVSLStatsTable - Total Packets, Total Error Packets

cvsVSLPortStatsTable - TX/RX Good, Bad, Bi-dir and Uni-dir Packets

This MIB will be the main vehicle though which Network Management stations access information relevant to the operation of the Virtual Switch…


Operational ManagementSlot/Port Numbering

After conversion, port definitions for switches within the Virtual Switch Domain inherit the Chassis ID as part of their naming convention…

Router#show ip interface briefInterface IP-Address OK? Method Status ProtocolVlan1 unassigned YES NVRAM up up Port-channel1 unassigned YES NVRAM up up Te1/1/1 10.1.1.1 YES unset up up Te1/1/2 192.168.1.2 YES unset up up Te1/1/3 unassigned YES unset up up Te1/1/4 unassigned YES unset up up GigabitEthernet1/2/1 10.10.10.1 YES unset up up GigabitEthernet1/2/2 10.10.11.1 YES unset up up <snip>

Router#show ip interface briefInterface IP-Address OK? Method Status ProtocolVlan1 unassigned YES NVRAM up up Port-channel1 unassigned YES NVRAM up up Te1/1/1 10.1.1.1 YES unset up up Te1/1/2 192.168.1.2 YES unset up up Te1/1/3 unassigned YES unset up up Te1/1/4 unassigned YES unset up up GigabitEthernet1/2/1 10.10.10.1 YES unset up up GigabitEthernet1/2/2 10.10.11.1 YES unset up up <snip>

PORT NUMBERING: <CHASSIS-ID><SLOT-NUMBER><PORT-NUMBER>PORT NUMBERING: <CHASSIS-ID><SLOT-NUMBER><PORT-NUMBER>

Chassis-ID WILL ALWAYS be either a “1” or a “2”


Operational ManagementFile System Naming

After the conversion to a Virtual Switch, some of the File System naming conventions have changed to accommodate the new setup - an example of the new setup is shown below…

Active Supervisor - Slot 5 Hot Standby Supervisor - Slot 5


e.g.OLD: disk0:NEW: sw1-slot5-disk0:

Switch 1 Switch 2

e.g.OLD: slavedisk0:NEW: sw2-slot5-disk0:

SW<NUMBER>SLOT<NUMBER>FILESYSTEMSW<NUMBER>SLOT<NUMBER>FILESYSTEM

AN EXAMPLE


Operational ManagementFile System Naming

Some filenames have remained the same - others have changed - some examples of file system names in a Virtual Switch include the following…

VIRTUAL SWITCHVIRTUAL SWITCH

sw<number>slot<number>disk0:sw<number>slot<number>disk0:

PREVIOUSPREVIOUS

disk0:disk0:

sw<number>slot<number>bootflash:sw<number>slot<number>bootflash:bootflash:bootflash:

sw<number>slot<number>sup-bootdisk:sw<number>slot<number>sup-bootdisk:sup-bootdisk:sup-bootdisk:

sw<number>slot<number>nvram:sw<number>slot<number>nvram:nvram:nvram:

slavedisk0:slavedisk0:slavedisk0:slavedisk0:

slavebootflash:slavebootflash:slavebootflash:slavebootflash:

slavesup-bootdisk:slavesup-bootdisk:slavesup-bootdisk:slavesup-bootdisk:

sw<number>slot<number>const_nvram:sw<number>slot<number>const_nvram:const_nvram:const_nvram:


Operational ManagementNVRAM Considerations

ConfigSwitch AConfig

Switch AConfig

Switch BConfig

Switch B

Combined ConfigSwitch A + B

Combined ConfigSwitch A + B

The Sup720 is equipped with 2Mb of NVRAM, but considerations must be made to evaluate the impact of combining the configurations of each switch into a combined configuration …

6506#dir sw1-slot5-nvram:Directory of nvram:/

<snip>

1964024 bytes total (1950786 bytes free)

6506#dir sw1-slot5-nvram:Directory of nvram:/

<snip>

1964024 bytes total (1950786 bytes free)

Note: Internal Testing estimates show combining two switch configurations into one with VS yields a saving of up to ~30% when compared to the size of the two original configurations …

Combining the configurations is done automatically during the VS conversion…


Operational ManagementNetflow

In a Virtual Switch, with both Data Planes active, Netflow data collection is performed on each Supervisor’s PFC - while Netflow export is only performed by the Control Plane on the VS Active …


VS State : ActiveControl Plane: ActiveData Plane: ActiveNetflow Collection: ActiveNetflow Export: Active

VS State : StandbyControl Plane: StandbyData Plane: ActiveNetflow Collection: ActiveNetflow Export: In-Active

VSL

Netflow operation in a Virtual Switch is similar to the way in which Netflow operates in a single chassis with Distributed Forwarding Card’s (DFC) present…

Switch 1 Supervisor Switch 2 Supervisor


Operational ManagementNetflow Export

The Virtual Switch Link will be used as the transit path to allow the standby Sup to forward Netflow data to the active Supervisor for Netflow export - the VS Link should be dimensioned to accommodate the expected Netflow export load…


VS State : ActiveNetflow Collection: ActiveNetflow Export: Active

VS State : StandbyNetflow Collection: ActiveNetflow Export: In-Active

VSL

NetflowData

NetflowData

NetflowExport

Netflow Collector


Operational ManagementSPAN

In a Virtual Switch Domain, the number of SPAN sessions is limited to what the VS Active Supervisor can provide. SPAN capacity on the VS Standby is not factored into available SPAN sessions…


VS State : ActiveControl Plane: ActiveData Plane: ActiveSPAN Management: ActiveReplication: Active

VS State : StandbyControl Plane: StandbyData Plane: ActiveSPAN Management: In-ActiveReplication: Active

VSL


Virtual Switch is supported in Whitney 1 which introduces the following SPAN capabilities per Virtual Switch Domain…

TX SPAN Sessions RX/Both SPAN Sessions Total SPAN Sessions

Virtual Switch Domain 14 2 16


Operational ManagementCiscoWorks LMS 3.0.1

CiscoWorks LAN Management Solution (LMS) 3.0.1 release will support an additional plugin that will allow for it to support various functions to manage a Virtual Switching System…

Conversion from Standalone to VSS

Inventory Management and Reporting

Configuration Collection and Archive Management

1

2

3

Image Upgrade - Base image and Patch4

CiscoView Management5

LMS will support the following functions:


Operational ManagementConversion

A new feature of LMS 3.0 will allow for the conversion from two Standalone chassis to a single VSS domain. This is achieved through a Wizard that will guide the user through the necessary steps…

Select the Standalone Switches to be converted - note that these devices must be already seen by RME…

1

Verify the appropriate Hardware and Software versions have been installed…

2


Operational ManagementConversion

After the conversion process has executed, the appropriate configuration changes will be applied and the switches will be reset…

Specify the interfaces to be used to set up the VSL…

3

Confirm the configuration changes and apply the configurations…

4


Operational ManagementInventory Management (RME)

Inventory Data can be collected and Reported through Resource Management Essentials (RME) from both the Active and Standby Chassis…

The Detailed Inventory Report can show the Entire Inventory along with both Hardware and Software information…


Operational ManagementConfiguration Collection (RME)

RME will also support the ability to fetch the running configuration from the VSS as well as provide Archive and configuration version management abilities…


Operational ManagementCiscoView

CiscoView is designed to show the view of both Active and Standby chassis INDIVIDUALLY. The user can switch between the views using Device-level menu options…

Each chassis is identified by a label indicating whether it is Active or Standby

Each chassis is identified by a label indicating whether it is Active or Standby


High Availability


High AvailabilityRedundancy Schemes

The default redundancy mechanism between the 2 VSS chassis and their associated supervisors is NSF/SSO, allowing state information and configuration to be synchronized. Additionally, only in NSF/SSO mode does the Standby supervisor PFC, Switch Fabric, modules and their associated DFCs become active…

VSL

Should a mismatch of information occur between the Active and Standby Chassis, the Standby Chassis will revert to RPR mode, where only configuration is synchronized, but PFC, Switch Fabric and modules will not be brought up

Switch 112.2(33)SXH1Active

Switch 212.2(33)SXH1NSF/SSO

VSL

Switch 112.2(33)SXH1Active

Switch 212.2(33)SXH2RPR


High AvailabilityNSF/SSO Requirements

After the roles have been resolved through RRP, a Configuration Consistency Check is performed across the VSL switches to ensure proper VSL operation. The following items are checked for consistency:

Switch Virtual Domain IDSwitch Virtual Domain ID

Switch Virtual Node TypeSwitch Virtual Node Type

Switch PrioritySwitch Priority

Switch PreemptSwitch Preempt

VSL Port Channel Link IDVSL Port Channel Link ID

VSL Port state, interfaces…VSL Port state, interfaces…

Power Redundancy modePower Redundancy mode

Power Enable on VSL cardsPower Enable on VSL cards

Additionally, software version, installed patches and PFC modes also need to be consistent for NSF/SSO mode to be entered…

Additionally, software version, installed patches and PFC modes also need to be consistent for NSF/SSO mode to be entered…


High AvailabilityDual-Active Detection

In a Virtual Switch Domain, one switch is elected as Active and the other is elected as Standby during bootup by VSLP. Since the VSL is always configured as a Port Channel, the possibility of the entire VSL bundle going down is remote, however it is a possibility…


VS State : ActiveControl Plane: ActiveData Plane: Active

VS State : StandbyControl Plane: StandbyData Plane: Active

VSL


It is always recommended to deploy the VSL with 2 or more links and distribute those interfaces across multiple modules to ensure the greatest redundancy

It is always recommended to deploy the VSL with 2 or more links and distribute those interfaces across multiple modules to ensure the greatest redundancy


High AvailabilityDual-Active Detection

If the entire VSL bundle should happen to go down, the Virtual Switch Domain will enter a Dual Active scenario where both switches transition to Active state and share the same network configuration (IP addresses, MAC address, Router IDs, etc…) potentially causing communication problems through the network…




VSL


2 mechanisms have been implemented in the initial release to detect and recover from a Dual Active scenario:

Enhanced Port Aggregation Protocol (PAgP)

Dual-Active Detection over IP-BFD

1

2


High AvailabilityDual-Active Detection - Enhanced PAgP

Enhanced PAgP allows for new TLVs to be relayed from the individual Virtual Switches to a remote device that is EtherChanneled to the Virtual Switch Domain. During normal operation the Virtual Switches will send the ID of the Active VS to the PAgP neighbor, and it will respond with the same Active ID…

Switch 1 Switch 2

Active: Switch 1 Active: Switch 1

Switch 1 Switch 2


Should the VSL go down, the Standby switch will transition immediately to Active state and start sending PAgP message with the new Active switch ID



The Enhnaced PAgP-capable neighbor will proceed to send the new Active Switch ID to all member ports of the port channel that it received the new Active Switch ID on, including the previous-active Virtual switch (Switch 1) …

Switch 1 Switch 2


Switch 1 Switch 2

Active: Switch 2

On Switch 1, Upon reception of PAgP messages with the Active ID of Switch 2, it will be aware that a Dual-Active scenario has occurred and will proceed to bring down all local interfaces*

Dual-Active!!Dual-Active!!



Enhanced PAgP Dual-Active recovery is enabled by default and may be disabled globally under the switch virtual configuration sub-mode…

vs-vsl#conf tEnter configuration commands, one per line. End with CNTL/Z.vs-vsl(config)#switch virtual domain 100vs-vsl(config-vs-domain)#no dual-active detection pagpvs-vsl(config-vs-domain)# ^Z vs-vsl#

vs-vsl#conf tEnter configuration commands, one per line. End with CNTL/Z.vs-vsl(config)#switch virtual domain 100vs-vsl(config-vs-domain)#no dual-active detection pagpvs-vsl(config-vs-domain)# ^Z vs-vsl#

vs-vsl#sh switch virtual dual-active pagp

Channel group 10 dual-active detect capability w/nbrsDual-Active version: 1.1Dual-Active configured: Yes Dual-Active Partner Partner PartnerPort Detect Capable Name Port VersionGi1/8/1 Yes vs-access-1 Gi5/1 1.1Gi2/8/1 Yes vs-access-1 Gi5/2 1.1

vs-vsl#sh switch virtual dual-active pagp

Channel group 10 dual-active detect capability w/nbrsDual-Active version: 1.1Dual-Active configured: Yes Dual-Active Partner Partner PartnerPort Detect Capable Name Port VersionGi1/8/1 Yes vs-access-1 Gi5/1 1.1Gi2/8/1 Yes vs-access-1 Gi5/2 1.1

Dual-Active Detection capabilities require that the neighboring device be Dual-Active Detection Aware. This can be verified with the following command…


High AvailabilityDual-Active Detection - IP-BFD

Dual-Active Detection with IP-BFD allows for the detection of a Dual-Active scenario subsequent to the Standby RP becoming Active. This mechanism requires that a direct heartbeat link be used to carry the IP-BFD frames from Switch 1 to Switch 2…

The IP-BFD Heartbeat link may exist on any interface but must have an IP address assigned to it on a different network

The IP-BFD Heartbeat link may exist on any interface but must have an IP address assigned to it on a different network

VSL

IP-BFD Heartbeat LinkSwitch 1 Switch 2

VSL

IP-BFD Heartbeat LinkSwitch 1 Switch 2

BFDBFD BFDBFD


High AvailabilityDual-Active Detection - IP-BFD

vss(config)#interface gigabitethernet 1/5/1vss(config-if)#no switchportvss(config-if)#ip address 200.230.230.231 255.255.255.0vss(config-if)#bfd interval 100 min_rx 100 multiplier 50vss(config-if)#no shutdownvss(config-if)#interface gigabitethernet 2/5/1vss(config-if)#no switchportvss(config-if)#ip address 201.230.230.231 255.255.255.0vss(config-if)#bfd interval 100 min_rx 100 multiplier 50vss(config-if)#no shutdownvss(config-if)#exitvss(config)#switch virtual domain 100vss(config-vs-domain)#dual-active detection ipbfdvss(config-vs-domain)#dual-active pair interface g 1/5/1 interface g 2/5/1 ipbfd

adding a static route 200.230.230.0 255.255.255.0 Gi2/5/1 for this dual-active pairadding a static route 201.230.230.0 255.255.255.0 Gi1/5/1 for this dual-active pair

vss(config-vs-domain)#

vss(config)#interface gigabitethernet 1/5/1vss(config-if)#no switchportvss(config-if)#ip address 200.230.230.231 255.255.255.0vss(config-if)#bfd interval 100 min_rx 100 multiplier 50vss(config-if)#no shutdownvss(config-if)#interface gigabitethernet 2/5/1vss(config-if)#no switchportvss(config-if)#ip address 201.230.230.231 255.255.255.0vss(config-if)#bfd interval 100 min_rx 100 multiplier 50vss(config-if)#no shutdownvss(config-if)#exitvss(config)#switch virtual domain 100vss(config-vs-domain)#dual-active detection ipbfdvss(config-vs-domain)#dual-active pair interface g 1/5/1 interface g 2/5/1 ipbfd

adding a static route 200.230.230.0 255.255.255.0 Gi2/5/1 for this dual-active pairadding a static route 201.230.230.0 255.255.255.0 Gi1/5/1 for this dual-active pair

vss(config-vs-domain)#

To enable IP BFD as the detection mechanism, two directly-connected interfaces must be configured as BFD message links…


High AvailabilityDual-Active Detection - Exclude Interfaces

Upon detection of a Dual Active scenario, all interfaces on the previous-Active switch will be brought down so as not to disrupt the functioning of the remainder of the network. The exception interfaces include VSL members as well as pre-determined interfaces which may be used for management purposes…

vs-vsl#conf tEnter configuration commands, one per line. End with CNTL/Z.vs-vsl(config)#switch virtual domain 100vs-vsl(config-vs-domain)#dual-active exclude interface Gig 1/5/1vs-vsl(config-vs-domain)#dual-active exclude interface Gig 2/5/1vs-vsl(config-vs-domain)# ^Z vs-vsl#

vs-vsl#conf tEnter configuration commands, one per line. End with CNTL/Z.vs-vsl(config)#switch virtual domain 100vs-vsl(config-vs-domain)#dual-active exclude interface Gig 1/5/1vs-vsl(config-vs-domain)#dual-active exclude interface Gig 2/5/1vs-vsl(config-vs-domain)# ^Z vs-vsl#


High AvailabilityDual-Active Recovery

Upon the restoration of one or more VSL interfaces, VSLP will detect this and will proceed to reload Switch 1 so that it may be able to re-negotiate Active/Standby role after bootup…

After role has been resolved and SSO Hot Standby mode is possible, interfaces will be brought up and traffic will resume back to 100% capacity…

VSL Up! Reload…VSL Up! Reload…

Switch 1 Switch 2

Switch 1 Switch 2VSLPVSLP VSLPVSLP


High AvailabilityGeneric OnLine Diagnostics (GOLD)

Some enhancements to the GOLD framework have been implemented in a VSS environment, which leverages a Distributed GOLD environment. In this case, each supervisor runs an instance of GOLD, but is centrally managed by the Active Supervisor in the Active chassis…

VS State : ActiveLocal GOLD: Active

VS State : StandbyLocal GOLD: Active

VSLSwitch 1 Switch 2

Distributed GOLD Manager

There are 4 new tests that are available in VSS mode:

TestVSLLocalLoopback

TestVSLBridgeLink

TestVSLStatus

1

2

3

TestVSActiveToStandbyLoopback4


High AvailabilityDHCP Snooping & Integrated Security

The 12.2(33)SXH release allows for the stateful synchronization of the binding and posture tables to be statefully synchronized across the members of the VSS. This will alleviate the requirement for the Standby VS to re-learn DHCP and posture information in the case of a switchover…

Virtual SwitchSwitch 1 Switch 2

DHCP SnoopingBinding Table

DHCP SnoopingBinding Table

IP AddIP Add MAC AddMAC Add10.10.10.1010.10.10.10 00:50:56:01:e1:0200:50:56:01:e1:02172.26.18.2172.26.18.2 00:02:b3:3f:3b:9900:02:b3:3f:3b:99172.26.19.34172.26.19.34 00:16:a1:c2:ee:3200:16:a1:c2:ee:3210.10.10.4310.10.10.43 00:16:cb:03:d3:4400:16:cb:03:d3:44

VLANVLAN1010181819191010

InterfaceInterfacePo10Po10Po10Po10Po20Po20Po20Po20


Quality of Service


Quality of ServiceClassification & Policing

Both Classification and Policing functions are handled by PFC QoS, and is executed by either the PFC on the Active and Hot Standby Supervisor, or the ingress linecard DFC. There are 2 important caveats which must be understood whilst implementing these functions…

Policies must either be applied on L3 interfaces (SVIs or Physical interfaces), or Port Channels. Policies on L2 interfaces are not supported in this release.

1

policy-map CLASSIFY class class-default set ip dscp 40

interface GigabitEthernet 2/3/48 switchport service-policy input CLASSIFY


interface GigabitEthernet 2/3/48 switchport service-policy input CLASSIFY


interface PortChannel 10 switchport service-policy input CLASSIFY


interface PortChannel 10 switchport service-policy input CLASSIFY


Quality of ServiceClassification & Policing

Aggregate policers that are applied on SVIs or Port Channels that have interfaces distributed across multiple forwarding engines are subject to Distributed Policing caveats…

2

policy-map POLICE class class-default police average 10000000

Interface GigabitEthernet 1/2/10 channel-group 20 mode desireable


interface PortChannel 20 service-policy input POLICE

policy-map POLICE class class-default police average 10000000



interface PortChannel 20 service-policy input POLICE


Quality of ServiceQoS on the VSL

The VSL itself has QoS provisioned by default and in the FCS release of the software, it is not configurable. A few important aspects relating to VSL QoS are as follows:

VSLP and other Control frames are always marked as Priority packets and are always queued and classified as such

1

VSL is always configured as “Trust CoS” and hence ingress queuing is enabled2

Service Policies are not supported on the VSL3

VSL

Switch 1 Switch 2

VSLPVSLPFTPFTPHTTPHTTP

CoS Maps, Thresholds and Queues are not configurable on the VSL4


Summary


Virtual Switching SystemDeployment Considerations

Virtual Switch will incorporate some deployment considerations as best practice…


Virtual Switching SystemBenefits


Virtual Switching SystemSummary


Q and A


Documents

© 2007 Cisco Systems, Inc. All rights reserved.Cisco ConfidentialBRKRST-3468 1 Understanding Virtual Switching System on the Cisco Catalyst 6500 Cisco