Upload
vukhanh
View
286
Download
11
Embed Size (px)
Citation preview
BRKDCT-2951
Deploying Nexus 7000 in Data Center Networks
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 2
Session Abstract
This session is targeted to network administrators and operators who have deployed or are considering the deployment of the Nexus 7000. The session starts with a brief introduction to the Nexus 7000 hardware components. Then it is followed by a brief design discussion. The focus of the presentation is on implementation and best practices. The implementation section will cover installation, layer-2 & layer-3 protocols, security features, and system management features. The session will cover NX-OS CLI but troubleshooting is not part of this presentation’s scope.
Attendee should have a basic knowledge of the Nexus 7000 hardware and software platform as well as solid knowledge of L2 and L3 protocols.
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 3
Associated Sessions/Labs
Cisco Nexus 7000 Switch Architecture - BRKARC-3470
Cisco NX-OS Software Architecture - BRKARC-3471
Deploying Virtual Port Channel in NXOS - BRKDCT-2048
Nexus 7000/NX-OS Hands On Lab - LRTDCT-2847
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 4
Agenda
Hardware Overview
Data Center Designs
Implementation and Best Practices
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 5
Hardware Overview7010 Chassis
8 I/O slots, 2 supervisor slots (5, 6)
Front-to-back air flow utilizing 2 system fan trays and 2 fabric fan trays
21 RU (2 per 42 RU rack)
Up to three power supplies
6kW AC, 7.5kW AC and 6kW DC PS
5 fabric module slots
46 Gbps per I/O module slot
fabric module is unique to chassis type
All components support Online Insertion and Removal (OIR)
Optional air filter satisfies NEBS requirements
Front View Rear View
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 6
Hardware Overview7018 Chassis
Front View Rear View
16 I/O slots, 2 supervisor slots (9, 10)
Side-to-side (right to left) air flow utilizing 2 system fan trays
25 RU
Up to four power supplies
6kW AC, 7.5kW AC and 6kW DC PS
5 fabric module slots
46 Gbps per I/O module slot
fabric module is unique to chassis type
All components support Online Insertion and Removal (OIR)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 7
Hardware OverviewI/O Modules (Non-XL)
32 port 10GE (80G) SFP+
4:1 port-level oversubscription
Default rate-mode is shared
48 port 10/100/1000 (46G) RJ45
48 ports 1GE (46G) SFP
N7K-M132XP-12
N7K-M148GT-11
N7K-M148GS-11
Capability Size
MAC entries 128K
IPv4 / IPv6 routes (128K / 64K)
Security / QoS ACL entries 64K
Netflow 512K
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 8
Hardware OverviewI/O Modules (XL)
8 port 10GE XL (80G) X2
2 forwarding engines (up to 120Mpps)
48 ports 1GE XL (46G) SFP
N7K-M108X2-12LN7K-M148GS-11L
CapabilitySize (w/o Scalable Feature License)
Size (w/ Scalable Feature License)
MAC entries 128K 128K
IPv4 / IPv6 routes (128K / 64K) Up to (1M / 350K)*
Security / QoS ACL entries 64K 128K
Netflow 512K 512K
NX-OS 5.0 NX-OS 5.0
* Actual limit depends on prefix distribution
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 9
Agenda
Hardware Overview
Data Center Designs
Implementation and Best Practices
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 10
Virtual Port-Channel (vPC)Design Motivations
Provides multi-chassis etherchannel capability (L2 port-channel only)
Eliminates STP blocked ports and reduce STP complexity
Does not depend on access switches for STP convergence
Uses all available uplink bandwidth
Enables dual-homed servers to operate in active-active mode
Provides fast convergence upon link/device failure
Software Version Number of vPC
Pre 4.2 release 196
4.2(1) and later 256
Double-sided vPC
vPC vPC
vPC vPC
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 11
Virtual Device Contexts (VDCs)Design Motivations
Consolidate and support multiple business units, departments, and networks
Web, App, Database
Production, OOB mgmt, Development, Test
Customer A, Customer B, Customer C
Provide network segmentation to meet security compliance requirements
Internet, Extranet, DMZ, Intranet
Non-Secured, Secured, PCI
Implement logical tier design
Core, Aggregation, Access
VDC2
Secure
VDC3Non-
Secure
VDC2Prod
VDC3Dev
VDC3Agg
VDC4Access
VDC2Core
VDC2BU1 /App 1
VDC3BU2 / App 2
VDCs provide logical separation of control-plane,
data-plane, management, resources, and system
processes within a physical switch
VDC4Test
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 12
Data Center Design Example 1
Large Data Center utilizing 3-Tier DC design
Nexus 7000 in core and aggregation
10GE/GE ToR and GE MoR access layer switches
Implement vPC / double-sided vPC to eliminate L2 loops and to support active/active server connections
VPC
L2
L3
L3
L2
L2 Channel
L3 link
L2 link
L3 Channel
Access
Core
Aggregation ……..
Core1 Core2
agg1bagg1a aggNa aggNb
......
VPC
active standbyactive active active standbyactive active
VPCVPC
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 13
Data Center Design Example 2
Large Data Center utilizing 3-Tier DC design
Nexus 7000 in core and aggregation, Nexus 5000 and Nexus 2000 in access layer
Implement vPC / double-sided vPC to eliminate L2 loops
Two different vPC redundancy models can be utilized to support active/active or active/standby server connections
L2L3
L3
L2
L2 Channel
L3 link
L2 link
L3 Channel
Access
Core
Aggregation
……..VPCVPC
VPC VPC VPC VPC
active standby active standby Active/Active Active/Active
Core2Core1
aggNa aggNbagg1a agg1b
vPC vPC vPC vPC
vPC vPC
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 14
Data Center Design Example 3
Large Data Center utilizing 3-Tier DC design
Nexus 7000s in Core and Aggregation
Utilize VDCs in aggregation layer to create a non-secured zone and a secured zone
10GE/GE ToR and GE MoR access layer switches
Implement vPC / double-sided vPC to eliminate L2 loops and to support active/active server connections
L2
L3
L3
L2
L2 Channel
L3 link
L2 link
L3 ChannelCore
Aggregation
Access
SW-2b
VDC3
SW-2a
VDC3SW-2a
VDC2SW-2b
VDC2
SW-1a
VDC3SW-1b
VDC3
SW-1a
VDC2
SW-1b
VDC2
Core2Core1
vPC vPC
active activeactive standbyactive activeactive standby
vPCvPC vPCvPC
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 15
Data Center Design Example 4
Small Data Center with a “virtualized” 3-Tier DC design
Utilize VDCs on a single device to create a core and aggregation layer
GE and 10GE ToR access layer switches
Implement vPC / double-sided vPC to eliminate L2 loops and to support active/active server connections
L2
L3
L3L2 Channel
L3 link
L2 link
L3 ChannelCore
Aggregation
Access
SW-1b
VDC2SW-1a
VDC2
active standby
SW-1b
VDC3SW-1a
VDC3
vPC
L2active active
vPC
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 16
Agenda
Hardware Overview
Data Center Designs
Implementation and Best Practices
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 17
Implementation and Best Practices
Installation and Maintenance
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 18
Chassis Installation
Use standard four-post, 19-inch EIA data center rack
Cabinet can be leveraged to convert 7018 to front-to-back air cooling
When installing 7018
Reserve 11” space on both sides of the rack to allow for side-to-side airflow
Route cables on front side of the rack to clear the rear side for airflow
Always perform chassis / system grounding
7010 chassis
7018 chassis
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 19
Hardware Installation
Two supervisors are recommended for high availability and ISSU
Configure redundant power redundancy-mode
Available power in redundant mode is the minimum of IS and PS redundancy
System default is PS redundant
Connect PS input sources to two different power grids
Setting maximum number of fabric modules per system allows the system to release some of the reserved power (supported in NX-OS 5.0)
By default system reserves enough power for five fabric modules
Fabric modules must be installed in the first N fabric module slots
220V
Grid 1 Grid 2
220V
Nexus7K(config)#
power redundancy-mode redundant
Nexus7K(config)#
hardware fabrics max 3
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 20
Virtual Device Contexts (VDCs)
VDC2Agg1
VDC4Test
VDC1Admin
VDC3Acc1
HA Policy =
Bring down
Production VDC
HA Policy =
Switchover
Nexus7K(config-vdc)# ha-policy dual-sup <policy> single-sup <policy>
Nexus7K(config-vdc)# limit-resource vlan minimum <#> maximum <#>
Test VDC
vlan = 50
Linecard 1 Linecard 2
64K 64K
128K 128K
FIB
TCAMACL
TCAM
VDC 2 VDC 3 VDC 4
Assign I/O modules to VDCs such that TCAM resources are shared effectively
Allocate entire I/O module per VDC if possible
All ports in the same port group on the 32 port 10GE I/O modules must be allocated to the same VDC
Customize VDC HA policy and resource configurations as necessary
Dual-sup default is switchover and single-sup default is restart
Only non-default VDCs can be suspended, resumed, reloaded, or restarted Nexus7K(config)# vdc <name> suspend Nexus7K# reload vdc <name>
Nexus7K(config-vdc)#allocate interface e2/1,e2/3,e2/5,e2/7
If 3 or less data forwarding VDCs are required, reserve the default VDC as the administrative VDC
On the default VDC, assign accounts with minimum privileges necessary to accomplish operational tasks
FIB
TCAMACL
TCAM
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 21
OOB (out-of-band) Management Network
Supports both mgmt0 and CMP ports
The IP address for default and non-default VDCs must be assigned to the same IP subnet
Assign different IP address’ for redundant CMP (same IP address for redundant mgmt0 interface)
If default VDC is reserved as the “admin” VDC, OOB mgmt network is necessary to provide access
Provides the option to assign all system management servers in “management” VRF and control access via ACL on mgmt interface
Consider the resiliency of the OOB mgmt network
Mgmt0 Mgmt0
Mgmt0 x2 Mgmt0 x2
Agg1a Agg1b
Acc1 Acc2
Core2Core1
OOB MgmtDist
L3
VDC2Agg1
VDC3Agg2
VDC1Admin
VDC2Agg1
VDC3Agg3
VDC1Admin
Mgmt0 Mgmt0
OOB MgmtNetwork
CMP x2 CMP x2
Mgmt0 x2Mgmt0 x2
Sys Mgmt server
It is recommended to implement an
OOB management network
mgmt1 mgmt2
Management VRF
Default VRF
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 22
Software Licensing
Feature License Features
Enterprise LAN OSPF, EIGRP, BGP, ISIS, PIM, MSDP, PBR, GRE
Advanced LAN CTS, VDCs
Scalable Feature XL TCAM
Transport Services OTV
Base NX-OS features do not require a license
Include basic Infrastructure, L2 switching, etc
All features are shipped with NX-OS image
Install individual licenses or enable the license grace period (120 days) to enable advanced features
License is tied to chassis serial number
License is stored in dual redundant NVRAM modules on chassis backplane
If chassis is replaced, work with Cisco TAC to re-key the license
If supervisor is replaced, license can be re-installed but not required (show license usage will indicate the license is installed but is missing)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 23
Software Licensing (cont.)License PAK
PAK +
chassis serial #
<xml...
licA ...>
license
file
Nexu7K# install license bootflash:<license_file.lic>
………
Nexu7K# copy bootflash:<license_file.lic> tftp:….
…………
Nexu7K# show license usage
Feature Ins Lic Status Expiry Date Comments
Count---------------------------------------------------------------------------------------------------------------
LAN_ADVANCED_SERVICES_PKG Yes - In use Never -
LAN_ENTERPRISE_SERVICES_PKG Yes - Unused Never -
The required licenses can be either factory installed or manually installed
License installation is non-disruptive to features already running under the grace period
Follow the steps to manually install the licenses
Identify chassis serial number and PAK (Product Activation Key)
Nexu7K# show license host-id
License hostid: VDH=TBM########
Obtain the license key file from http://www.cisco.com/go/license
Copy licenses to bootflash, install licenses and backup the licenses
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 24
Software UpgradeCold Start Upgrade
Utilize cold start upgrade procedure to minimize the upgrade window for non-production devices
It is recommended to synchronize the kickstart image and the system image
Nexu7K(config)#
boot system bootflash:<system-image> sup-1 sup-2
boot kickstart bootflash:<kickstart-image> sup-1 sup-2
Nexus7K# copy run startup-config
Nexus-3# sh boot
---deleted---
Boot Variables on next reload:
sup-1
kickstart variable = bootflash:/<kickstart-image>
system variable = bootflash:/<system-image>
sup-2
kickstart variable = bootflash:/<kickstart-image>
system variable = bootflash:/<system-image>
No module boot variable set
Nexus7K# reload
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 25
In-Service Software Update (ISSU)
Utilize ISSU to upgrade devices with zero disruption
Upgrade the system with a single “install all” command
Reload the CMP modules to complete the CMP upgrade
It is recommended to synchronize the kickstart image and the system image
Show commands can be used ahead of time to determine any potential impact prior to performing ISSU
Issue “show install all impact” to determine upgrade impact
When downgrading software, use “show incompatibility-all” to determine if any features need to be disabled
Nexus7K# show install all impact kickstart bootflash:<kickstart> system bootflash:<system>
Nexus7K# show incompatibility-all system bootflash:<system-image>
The following configurations on active are incompatible with the system image
1) Service : vpc , Capability : CAP_FEATURE_VPC_RELOAD_RESTORE
---deleted---
Nexus7K# install all kickstart bootflash:<kickstart-image> system bootflash:<system-image>
Nexus7K# show install all status
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 26
ISSU (cont.)
ISSU upgrade performs the following actions
Verify and validate the image, check image compatibility, provide descriptive upgrade information and option to cancel, sync images to standby sup, upgrade and switchover standby sup, upgrade previous active sup and I/O modules, load new image to CMP
Avoid disruption to the system during upgrade
STP topology change, module removal, power interruption, etc
Understand configuration conditions that cause ISSU failure
Active config sessions, suspended VDCs, disabling OSPF/ EIGRP / BGP / ISIS graceful restart, BGP hold timer tuned to less than switchover time (15 sec)
Pre-Upgrade check failed. Return code 0x80960002 (No such file or directory)
Nexus7K# show install all failure-reason
Service "session-mgr" returned error: Session-Manager active sessions present,
Nexus7K# show install all failure-reason
Service "stp" in vdc: 1 returned error: STP topology change in progress which can impact
ISSU. As a precaution ISSU is rejected. (x40DD0033)
Nexus7K# sh install all status
This is the log of last installation.
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 27
EPLD Upgrade
EPLDs (Electronic Programmable Logical Devices) upgrade is used to enhance hardware functionality or to resolve known issues
Performed on all the field replaceable modules (fan trays, fabric modules, I/O modules, and supervisor)
It is recommended to upgrade to the latest EPLD image for non-production devices
EPLD upgrade is a separate and independent process from ISSU and is typically not required
Check EPLD release notes or issue “show install all impact epld” to determine if EPLD upgrade is required
Nexus7K# show install all impact epld bootflash:<EPLD_image_name>
Nexus7K# install all epld bootflash:<EPLD_image_name>
Nexus7K# show version <type> <mod #> epld
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 28
EPLD Upgrade (cont.)
When perform EPLD upgrade for a dual supervisor system, upgrade the standby first, then switchover and upgrade previous active supervisor
In a redundant system, only EPLD upgrade for an I/O module can disrupt traffic since the module need to be power-cycled
When performing EPLD upgrade for mission critical systems, upgrade I/O module individually instead of all installed modules
Nexus7K# install module <module> epld bootflash:<EPLD_image_name>
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 29
Command Line Interface (CLI)
Leverage CLI alias to replace frequently used commands / actions
Nexu7K(config)#
cli alias name wri copy run start
cli alias name vpcpreempt conf t ; vpc dom 1 ; role pri 16384 ; int po 1 ; sh ; no sh
Nexus7K# show cli var
VSH Variable List
-----------------
SWITCHNAME="Nexus7K"
TIMESTAMP="2010-05-06-20.49.24"
Nexus7K# copy run bootflash:/$(TIMESTAMP)-$(SWITCHNAME)-cfg
Nexus7K# dir bootflash:
4265 May 06 20:22:24 2010 2010-05-06-20.50.24-Nexus7K-cfg
Nexus7K# show cli syntax | i spanning-tree
(788)[ no ] debug spanning-tree all
---deleted---
Nexu7K(config)# show cli syntax | i spanning-tree
(125) spanning-tree mode <stp-mode> | no spanning-tree mode [ <stp-mode> ]
Nexu7K(config-if)# show cli syntax | i spanning-tree
(58)[ no ] spanning-tree [ vlan <vlan-id> ] cost auto
Nexus7K# sh cli list arp
Reference CLI variables in scripts and CLI commands
Utilize CLI syntax / CLI list to identify available commands
Utilize CLI history to identify the command history
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 30
Nexu7K# checkpoint checkpt1
Processing the Request... Please Wait
Nexus7K# show diff rollback-patch running-config checkpoint checkpt1
Processing the Request... Please Wait
Nexus7K# config t
Enter configuration commands, one per line. End with CNTL/Z.
Nexus7K# rollback running-config checkpoint checkpt1……..
Configuration Rollback
The rollback feature allows users to take a configuration snapshot and reapply the configuration at any point
Create up to 10 checkpoints per VDC
Nexus7K(config)# no feature ospf
Nexus7K(config)# sh checkpoint all
----------------------------------------------
Name: system-fm-__inst_1__ospf
The rollback changes can be viewed before committing to the rollback operation
Auto-checkpoint protects against unintended loss of configuration (invoked with feature removal and license expiration)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 31
Other Installation Considerations
Configure complete boot-up diagnostic level (default)
Before bringing staged devices to production, power-cycle again to perform boot-up diagnostic
Utilize “show hardware capacity” to determine system capacity and capacity planning
Nexus7K(config)#
diagnostic bootup level complete
Nexus7K# show diagnostic result module all
Nexus7K# show hardware capacity
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 32
Implementation and Best Practices
Layer-2 Features
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 33
VLAN Trunking Protocol (VTP)
VTP “OFF” mode is recommended
Switches do not participate in VTP and all VTP advertisements are not forwarded
Utilize VTP transparent mode if VTP domain needs to extend across Nexus 7000 switches
Must ensure VLAN1 is allowed on trunks when operating in VTP transparent mode
VTP client / server mode is not supported
DCNM (Data Center Network Manager) can be leveraged to replicate VLAN database
Internal VLANs (3968 - 4047, 4094) are reserved and can’t be re-allocated
The reserved VLANs for Cat6Ks are 1002 –1005. Additionally, users can configure the internal VLAN allocation policy (ascending from 1006 or descending from 4094)
Nexus7K(config)# feature vtpNexus7K(config)# vtp domain <name>
Nexus7K(config)# no feature vtp
VTP server VTP client
Transparent
VTP
packets
Acc2
agg1bagg1a
Acc1
Transparent
Off
Acc2
agg1bagg1a
Acc1
Must allow
VLAN1
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 34
General Layer-2 Features
Enable UDLD feature to configure UDLD normal mode on all fiber interfaces
Enabling UDLD feature is equivalent to configuring UDLD normal mode globally
All fiber interfaces will inherit the global UDLD setting
Nexus7K(config)# feature udld
Nexus7K(config-if-range)# udld aggressive
Nexus7K(config-if)#
mtu <mtu>
Nexus7K(config)#
system jumbomtu <MTU>
It is recommended to configure UDLD aggressive on port-channel member ports
Interface configuration supersedes the global UDLD setting
The default CAM aging timer is 1800s and ARP timeout is 1500s
The default timers limits unicast flooding associated with asymmetric forwarding by synchronizing the CAM aging with ARP timeout
When implementing jumbo frame, L2 MTU must match the system jumbo MTU
The default system jumbo MTU is 9216
Match L3 MTU if L3 forwarding is required
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 35
General Layer-2 Features (cont.)
Implement one of the following methods to prevent double encapsulation 802.1Q attack
Assign unused VLAN as native VLAN (consistent across the same L2)
Clear native VLAN from the trunk
Configure to tag the native VLAN on all trunksNexus7K(config)# vlan dot1Q tag native
Nexus7K# show interface status err-disabled
Nexus7K(config)# errdisable recovery cause <cause>
Nexus7K(config)# errdisable recovery interval <time>
It is recommended to manually bring up error-disabled interface after the cause is identified
Errdisable recovery is disabled by default
Implement storm-control on L2 host ports and access layer to prevent disruptions caused by broadcast and multicast storm
Unsupported Layer-2 features
DTP, ISL Trunk, Flexlink, Link-State Tracking
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 36
EtherChannel
Active Utilize LACP to negotiate both L2 and L3 port-channels N7010-2
N7010-1 BPDUs
ONRoot
Active
N7010-2N7010-1
ON L3 Po is
down !
L3 Po
is up
BPDUs
Nexus7K(config)# feature lacp
Nexus7K(config)# int e<mod>/<port>
Nexus7K(config-if)# channel-group <#> mode activeDispute!
Mismatch Conditions
Nexus7K(config)#
Shut
no lacp graceful-convergence
no shut
Nexus7K(config)#
Shut
no lacp suspend-individual
no shut
Disable LACP suspend-individual only on “edge” port-channel
Nexus7K(config-if-range)# lacp rate normal
Disable LACP “graceful-convergence” on port-channel if “graceful-convergence” interoperability is an issue
If required, disable LACP “suspend-individual” on port-channel interface to allow the individual member ports to operate as “individual”
Implement port channels with 2, 4 or 8 members for optimal traffic distribution
Understand LACP compatibility enhancements
Implement normal LACP timer in a dual supervisor system (default)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 37
EtherChannel (Cont.)
Understand port-channel failure behaviors
BW and IGP cost for L3 channel are recalculated when physical member fails
STP cost for L2 channels does not recalculate when physical member fails
OSPF Cost
50 100
50
Nexus7K#sh port-channel load-balance forwarding-path interface port-channel 1 src-ip 1.1.1.1 dst-ip 2.2.2.2 vlan 2 mod 3
Missing params will be substituted by 0's.
Module 3: Load-balance Algorithm: source-dest-ip-vlan
RBH: 0x7 Outgoing port id: Ethernet3/3
Access
Aggr1a Aggr1b
Core1-1 Core-2
Statically configure IGP cost on L3 channel if the default behavior is not desired
Modify port-channel load-balancing to match needs
Configure on default VDC and the default is Source-Destination-IP-VLANNexus7K(config)# port-channel load-balance ethernet <lb-method>
Nexus7K(config)# port-channel load-balance ethernet <lb-method> module <mod>
Unsupported etherchannel features
PAgP, LACP min-link
Nexus7K# sh port-channel load-balance
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 38
Spanning-tree (STP)
Implement consistent STP mode in the same L2 domain
RPVST+ is the default and is backward compatible with PVST
Nexus7K# sh spanning-tree active | i PeerPo11 Desg FWD 12 128.4106 P2p Peer(STP)
Nexus7K# show spanning-tree summary total
----deleted----
Name Blocking Listening Learning Forwarding STP Active
---------- -------- --------- -------- --------- ---------
9 vlans 0 0 0 18 18
Total number of
logical ports
Nexus7K# sh spanning-tree active | i Bound
Po11 Desg FWD 100000 128.4106 P2p Bound(PVST)
Utilize MST to scales large L2 network
MST supports 75K logical ports (90K in NX-OS 5.0) and RPVST+ supports 16K logical ports
MST introduces some complexity and requires proper planning
MST interoperates with both RPVST+ and PVST+ by utilizing PVST+ simulation
3 msts 2 0 0 8 10 MST ports
Configure the allowed VLANs on trunk interfaces
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 39
Spanning-tree (Cont.)
Configure aggregation switches as the STP primary and secondary root
Enable Bridge Assurance (BA) if supported on both local and remote switches
BA is enabled globally (default) and active only on interfaces configured as port type “network”
BPDUs are sent on all active BA ports
BPDU-
guard
Bridge Assurance
Loop-Guard
Port Type Edge / Edge Trunk
Pri / Sec
root
agg1a agg1b
Access2
Access1
L3 FWL3 FW
Enable loopguard globally If BA is not supported on access switches
Configure host ports as port type “edge” or port type “edge trunk”
Enable STP BPDU-guard globally
Dispute mechanism is integrated by default
Nexus7K(config-if)#spanning-tree port type edge trunk
Nexus7K(config)#spanning-tree port type edge bpduguard default
Nexus7K(config-if)# spanning-tree port type network
Nexus7K(config)# spanning-tree vlan <vlan> pri <pri>
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 40
Spanning-tree (Cont.)
Implementing STP long path-cost method
RSTP default is short and MST default is long
Utilize port-profiles to enforce consistent configuration
Nexus7K(config)#port-profile type ethernet host-port
state enable
switchport
switchport mode access
spanning-tree port type edge
spanning-tree bpduguard enable
no shut
Nexus7K(config-if)#
switchportinherit port-profile host-port
switchport access vlan <vlan>
Nexus7K(config)#port-profile type ethernet trunk-port
state enable
switchport
switchport mode trunk
switchport trunk native vlan <vlan>
spanning-tree port type network
no shut
Nexus7K(config-if)#
switchportinherit port-profile trunk-port
switchport trunk allow vlan <vlans>
Note: Port-Profiles are live profiles (modify or
delete port-profiles will be reflected on the
assigned interfaces)
Nexus7K(config)#
spanning-tree pathcost method long
Unsupported STP features
PVST+
Nexus7K# sh run int e<mod/port> expand-port-profile
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 41
Multiple Spanning-tree (MST)
Determine the maximum number of MST instances
Develop the VLAN plan
Map the entire ranges of VLANs to pre-determined MST instances
agg1a agg1b
Acc1
VLANs Description
IST 0 3968-4047,
4094
Internal VLANs
IST 1 1-299 Production 1
IST 2 300 -599 Production 2
IST 3 600 -699 Service
State / Keepalive
IST 4 700-3967,
4048-4093
Reserved for
future
Primary
root
IST 0,1,3
Primary
root
IST 2,4
IST1
IST2
IST2
IST1
IST3
Acc2
Nexus7K#
spanning-tree mst configuration
instance 1 vlan 1-299
instance 2 vlan 300-599
instance 3 vlan 600-699
instance 4 vlan 700-3967,4048-4093
name <name>
revision <rev>
!
spanning-tree mode mst
VLAN numbers are provided as an example
Plan ahead to avoid future MST
configuration changes
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 42
MST (cont.)
Configure aggregation switches as STP MST primary root and STP MST secondary root
Disable PVST+ simulation for tighter administration control
PVST+ simulation can be disabled per interface or globally
Nexus7K(config)#
no spanning-tree mst simulate pvst global
Nexus7K(config-if)#
spanning-tree mst simulate pvst disable
Nexus7K# sh spanning-tree active | i Bound
Po1 Root FWD 1000 128.4096 P2p Bound(RSTP)
Nexus7K-1a(config)#
spanning-tree mst 0,1,3 priority 8192
spanning-tree mst 2,4 priority 16384
Nexus7K-1b(config)#
spanning-tree mst 2,4 priority 8192
spanning-tree mst 0,1,3 priority 16384
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 43
Implementation and Best Practices
Virtual Port-Channel (vPC)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 44
Virtual Port-Channel (vPC)vPC Terminology
vPC peer - a vPC switch, one of a pair
vPC member port - one of a set of ports (port channels) that form a vPC
vPC - the combined port channel between the vPC peers and the downstream device
vPC peer-link (vPC_PL) - synchronize state between vPC peer devices (must be 10GE port-channel)
vPC peer-keepalive link (vPC_PKL) - detect the status of vPC peer devices
CFS - Cisco Fabric Services protocol, used for state synchronization and configuration validation between vPC peer devices
vPC VLANs - VLANs carried over the peer-link
Non-vPC VLANs - VLANs not carried over the peer-link
vPC orphan-ports - non vPC ports that are mapped to the vPC VLANs
vPC_PL
vPC_PKL
vPC
orphan portvPC member
port
agg1a
Access1 Access2
agg1b
Core1Core2
CFS Protocol
agg1bagg1a
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 45
vPC (cont.) vPC failure convergence summary
No impact with vPC peer-keepalive link failure
No impact with supervisor failover or ISSU
When vPC channel member fails, traffic is re-hashed over existing member
When half of the vPC channel fails, traffic is re-hashed / re-routed to vPC peer
Some traffic will traverse across vPC peer-link
When vPC primary switch fails, traffic is re-hashed / re-routed to vPC secondary switch
vPC peer-switch can be implemented to eliminate STP convergence
When vPC secondary switch fails, traffic is re-hashed / re-routed to vPC primary switch
vPC_PL
vPC_PKL agg1bagg1a
Acc2Acc1
Core
vPC_PL
vPC_PKL agg1bagg1a
Acc2Acc1
Core
vPC SecondaryvPC Primary
Half or vPC channel fails
vPC primary device fails
New
root
rootOld
root
convergenceconvergence
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 46
vPC SecondaryvPC Primary
Operation
secondary
vPC (cont.) vPC failure convergence summary
If vPC peer-link fails, the secondary vPC peer suspends local vPCs and shuts down SVIs of vPC VLANs
Single-homed devices connected to vPC secondary devices will be isolated
Dual-active does not occur if peer-keepalive link fails after vPC peer-link failure
If vPC peer-keepalive link fails first and vPC peer-link fails later (or both fail together), both vPC peers become active
Need to bring up both vPC_PKL and vPC_PL to recover from dual active state
After recovery the “configured” vPC secondary peer will remain as the operational primary
A vPC role change requires bring down/up peer-link
Auto vPC preempt is not supported since role change is disruptive
agg1bagg1a
vPC_PL
vPC_PKL
vPC dual
active!
Isolated!!
Acc2
agg1bagg1a
Acc2
vPC Primary
Acc1
Acc1
Core
vPC SecondaryvPC Primary
vPC_PL
vPC_PKL
vPC peer-link fails
vPC Secondary
Operation
primary
Both vPC_PL and vPC_PKL fail
Shut SVIs
Route all traffic
to agg1a
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 47
Core2
vPC (cont.) Implementation Best Practices
Utilize diverse 10GE modules to form vPC peer-link
Support a mix of 8 and 32 port 10GE modules
Single 10GE module implementation is supported but not recommended
Implement physical vPC peer-link ports in dedicated rate-mode
Shared rate-mode is supported but not recommended
Use a dedicated link for vPC peer keepalive link and assign it to a separate VRF
If mgmt0 interface is used as vPC keepalive link, it should be connected to an OOB mgmt network
Back-to-back mgmt0 connection should only be used in single supervisor implementation
Do not use SVI interface between vPC peer-link as vPC keepalive link
routing peer agg1b
vPC Secondary
role pri 16384
domain 1
agg1a
vPC Primary
role pri 8192
domain 1
Acc1a
vPC domain 2
Acc1b
Core1
Acc2
vPC_PL
vPC_PKL
agg1bagg1avPC_PL
vPC_PKLMgmt0
X 2Mgmt0
X 2
Mgmt
Network
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 48
Core2
vPC (cont.) Implementation Best Practices
routing peer agg1b
vPC Secondary
role pri 16384
domain 1
agg1a
vPC Primary
role pri 8192
domain 1
Acc1a
vPC domain 2
Acc1b
Core1
Assign unique vPC domain-ID for each pair of vPC peer devices in the same “L2 domain”
Acc2
vPC_PL
vPC_PKL
Nexus7K# show vpc orphan-ports
Define vPC primary peer role with lower role priority
Do not configure HSRP tracking, implement IGP routing over vPC peer-link to re-route traffic in case of complete uplink failure
Enable vPC delay restore (supported and enabled by default in NX-OS 4.2)
Match vPC with port-channel number
Nexus7K# sh vpc role | i "vPC system-mac”
vPC system-mac : 00:23:04:ee:be:01
Dual home all devices to vPC domain using vPC
If required, connect single attached devices to vPC primary peer and leverage “dual-active exclude interface-vlan”
Nexus7K(config-vpc-domain)# dual-active exclude interface-vlan 11
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 49
Nexus7K-Agg1a#
feature vpc
feature lacp
feature ospf
feature interface-vlan!
vlan 98,99,<vPC vlans>!
vrf context vpc-keepalive!
int e3/48
vrf member vpc-keepalive
ip address 10.1.1.1/30
no shut!
vpc domain 1
role priority 8192
peer-keepalive destination 10.1.1.2 source
10.1.1.1 vrf vpc-keepalive !
int e1/1,e2/1
rate-mode dedicated
switchport
switchport mode trunk
channel-group 1 mode active
no shut!
int port-channel 1
switchport
switchport mode trunk
vpc peer-link
spanning-tree port type network!
int e3/1-2
switchport
switchport mode trunk
channel-group 11 mode active
no shut!
int port-channel 11
switchport
switchport mode trunk
switchport trunk allowed vlan remove 98-99
vpc 11!
router ospf 1!
interface vlan 98
ip address 10.1.98.1/30
ip router ospf 1 area 0
ip ospf network point-to-point
no shut!
interface vlan 99
ip address 10.1.99.1/30
ip router ospf 1 area 1
ip ospf network point-to-point
no shut
vPC (cont.) Sample vPC Configuration
routing peer
VPC 11
Po11 Po11
e1/1,e2/1
e3/1-2
e1/1,e2/1
e3/1-2
e3/48 e3/48
V98,V99
agg1bagg1a
Access1 Access2
vPC_PL
vPC_PKL
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 50
Nexus7K-Agg1b#
feature vpc
feature lacp
feature ospf
feature interface-vlan!
vlan 98,99,<vPC vlans>!
vrf context vpc-keepalive!
int e3/48
vrf member vpc-keepalive
ip address 10.1.1.2/30
no shut!
vpc domain 1
role priority 16384
peer-keepalive destination 10.1.1.1 source
10.1.1.2 vrf vpc-keepalive !
int e1/1,e2/1
rate-mode dedicated
Switchport
switchport mode trunk
channel-group 1 mode active
no shut!
int port-channel 1
switchport
switchport mode trunk
vpc peer-link
spanning-tree port type network!
int e3/1-2
Switchport
switchport mode trunk
channel-group 11 mode active
no shut!
int port-channel 11
switchport
switchport mode trunk
switchport trunk allowed vlan remove 98-99
vpc 11!
router ospf 1!
interface vlan 98
ip address 10.1.98.2/30
ip router ospf 1 area 0
ip ospf network point-to-point
no shut!
interface vlan 99
ip address 10.1.99.2/30
ip router ospf 1 area 1
ip ospf network point-to-point
no shut
vPC (cont.) Sample vPC Configuration
routing peer
VPC 11
Po11 Po11
e1/1,e2/1
e3/1-2
e1/1,e2/1
e3/1-2
e3/48 e3/48
V98,V99
agg1bagg1a
Access1 Access2
vPC_PL
vPC_PKL
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 51
vPC (cont.) STP Best Practices
Do not disable STP !!
Configure aggregation vPC peers as root and secondary root
If vPC peer-switch is implemented, both vPC peers will behave a single STP root
Align STP primary root, HSRP active router and PIM DR with vPC primary peer
BA is enabled by default on vPC peer-link
Do not enable Loopguard and BA on vPC (disabled by default)
Enable STP port type “edge” and port type “edge trunk” on host ports
Enable STP BPDU-guard globally
Disable STP channel-misconfig guard if supported by access switches
BPDU-guardBPDU-guard
Port Type
Edge / Edge Trunk
routing peeragg1bagg1a
Acc1 Acc2
Disable STP channel-
misconfig guard
vPC primary
VLAN 1-4094 root
MST 0 – 3 root
HSRP Active
PIM DR
vPC primary STP
VLAN 1-4094 sec root
MST 0 – 3 sec root
HSRP Standby
vPC_PL
vPC_PKL
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 52
vPC (cont.) STP Best Practices
BPDU-guardBPDU-guard
Port Type
Edge / Edge Trunk
routing peeragg1bagg1a
Acc1 Acc2
Implement consistent STP mode in the same L2 domain
Configure the allowed VLANs on trunk interfaces
Utilize MST to scale L2 domain
Logical port limitation is applicable with vPC implementation
Plan ahead to avoid future configuration changes that can trigger vPC type-1 consistency failure
Sample global type-1 parameters include MST region configuration, STP mode, STP global configuration, STP state, etc
Sample of the interface type-1 parameters include port-channel mode, trunk configuration on vPC channel, link speed, etc
Disable STP channel-
misconfig guard
vPC primary
VLAN 1-4094 root
MST 0 – 3 root
HSRP Active
PIM DR
vPC primary STP
VLAN 1-4094 sec root
MST 0 – 3 sec root
HSRP Standby
vPC_PL
vPC_PKL
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 53
vPC (cont.) Special Considerations
It is recommended to configure “peer-gateway” to enable vPC peer devices to act as the gateway for packets destined to the vPC peer device's MAC address (supported in NX-OS 4.2)
The feature is necessary to support NAS devices, load-balancers, and other devices which reply to sender’s mac-address instead of HSRP virtual mac-address
Disable IP redirects on all SVIs of the vPC VLANs to avoid generating IP redirect messages if “peer-gateway” is configuredNexus7K(config)# vpc domain <domain-id>
Nexus7K(config-vpc-domain)# peer-gateway
Note: Disable IP redirects on all interface-vlans of this vPC domain for correct operation of this feature!
interface vlan <vlan x>, vlan <vlan y>
no ip redirects
Upgrade to NX-OS 4.2 to provides interoperability support for appliances which use unicast ARP requests to monitor gateway reachability (enabled by default)
Unicast ARP requests received by the HSRP standby router will be forwarded via the vPC peer-link to HSRP active router
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 54
vPC (Cont.)Connect layer-3 routing device
When connect layer-3 routing devices to a vPC domain, do not form routing adjacency with vPC peer devices over the vPC peer-link (unsupported design)
routing
peer
routing
peervPC
vPC
agg1bagg1a
Acc1a
vPC
agg1bagg1a
Acc1arouting
peer
routing
peer
routing
peerrouting
peer
agg1bagg1aL3 FW L3 FW
VPC
VLANs
VPC
VLANs
IBM OSA
vPC_PL
vPC_PKL
vPC_PL
vPC_PKL
vPC_PKL
vPC_PL
L3 link
L2 link
L3 link
L2 link
L3 link
L2 link
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 55
vPC (Cont.)Connect layer-3 routing device
If dynamic routing is required to a vPC domain, L3 routed interfaces should be utilized
If L3 routed interfaces can not be used, connect L3 routing devices to a vPC domain using vPC and implement static routing to FHRP address
routing
peer
routing
peer
agg1bagg1a
Static route to FHRP
vPC
agg1bagg1a
routing
peer
routing
peer
IBM OSA
L3 FW
vPC
vPC_PKL
vPC_PL
vPC_PKL
vPC_PL
L3 link
L2 link
L3 link
L2 link
Dynamic routing
L3L3
L3 L3
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 56
vPC (Cont.)Service appliances
Dedicate a L2 port-channel for the service appliances’ state and keepalive VLANs
Connect service appliances to vPC domain via vPC and configure static routes to HSRP address
If port-channel is not supported, this can create orphan ports
Implementing a separate L2 port channel for non-vPC VLANs can be used to
Support single attached devices without creating orphan ports by mapping interfaces to non-vPC VLANs and assign them to different VRFs
Support both routed and bridged traffics
routing
peer
Static
FHRP
agg1bagg1aL3 FW L3 FW
VPC
VLANsVPC
VLANs State/Keepalive
Static
FHRP
agg1bagg1a
L3 FW L3 FWVPC
VLANs
VPC
VLANs
State/Keepalive
Static
VIP
Static
VIP
Static
FHRP
Static
FHRP
vPC orphan ports
vPC_PL
agg1bagg1a
L3 FWL3 FW
Non-vPC
VLANs
State/Keepalive
Non-VPC VLANs Non-vPC
VLANs
routing
peer
vPC_PKL
vPC_PL
vPC_PKL
vPC_PL
vPC_PKL
VRF1agg1bagg1a
L3 FW L3 FW
State/Keepalive
vPC_PKL
Non-VPC VLANs
VRF2 vPC_PL
VRF1
VRF2
Non-vPC
VLANs
Non-vPC
VLANs
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 57
vPC (cont.) Single 10GE LC Implementation
If uplinks to core switches and vPC peer-link are implemented with a single 10GE LC, enable object tracking to prevent traffic black-hole (supported in NX-OS 4.2)
Since either device can be operational primary, enable object tracking on both vPC switches
routing peer agg1bagg1a
Acc1 Acc2
Core
Nexus7K-1a(config)#
track 1 interface port-channel1 line-protocol
track 2 interface ethernet1/25 line-protocol
track 3 interface ethernet1/26 line-protocol!
track 10 list boolean or
object 1
object 2
object 3!
vpc domain 1
track 10
e1/25-26 e1/25-26
po1 po1
vPC 11 vPC 12
routing peer agg1bagg1a
Acc1 Acc2
Core
e1/25-26 e1/25-26
po1 po1
vPC 11 vPC 12
w/o object
tracking!
vPC SecondaryvPC Primary
Nexus7K-1a# show int po 11
port-channel11 is down (suspended by vpc)
Nexus7K-1a# show int vlan 11
Vlan11 is down, line protocol is down
Nexus7K-1a# show track 10
Track 10
List Boolean or
Boolean or is DOWN
6 changes, last change 00:11:12
Track List Members:
object 3 DOWN
object 2 DOWN
object 1 DOWN
Tracked by:
vPCM
vPC_PKL
vPC_PL
vPC_PKL
vPC_PL
Shut SVIs
No routes!
vPC PrimaryvPC Primary
Operation
Secondary
vPC SecondaryvPC Secondary
Operation
Primary
with object
tracking!
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 58
vPC (cont.) Multi-layer vPC
If utilizing a single HSRP group for the inter-DC VLANs, configure active/standby router in one DC and configure listen/listen router in the other DC (supported in NX-OS 4.2)
Implement BPDU-filter to segment the STP domain between data centers
L2
L3
L3
Core
Aggregation
AccessDCi1b
DCi1a
DCi2b
DCi2a
VPC VPC
DC1
Agg1a
DC1
Agg1bDC2
Agg1a
DC2
Agg1b
DC1
Core1a
DC1
Core1b
DC2
Core1a
DC2
Core1b
vPC
Data
Center 1
L2
L3
L3
Core
Aggregation
Access
BPDU-filter
HSRP
Active
HSRP
Standby
HSRP
Listen HSRP
Listen
Domain 1
Domain 2 Domain 3
vPC
Domain 4
Nexus7K-DCi(config)#
int po 100
vpc 100
spanning-tree bpdufilter enable
Po 100 Po 100
Po 100 Po 100
vPC
100
Data
Center 2
Nexus7K-Agg1a(config)#
int vlan <vlan>
hsrp <group>
priority 130
Nexus7K-Agg1b(config)#
int vlan <vlan>
hsrp <group>
priority 120
Nexus7K-Agg2a(config)#
int vlan <vlan>
hsrp <group>
priority 110
Nexus7K-Agg2b(config)#
int vlan <vlan>
hsrp <group>
priority 100
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 59
vPC (cont.) vPC peer-switch
vPC peer-switch feature allows a pair of vPC peer devices to behave as a single STP device and send BPDUs from both vPC devices
Improves vPC convergence during vPC primary switch failure
Simplifies STP configuration by configuring both vPC with the same STP priority
Supports a hybrid topology of vPC and non-vPC connections by using the spanning-tree pseudo-information
Nexus7K-1a(config-vpc-domain)#
peer-switch
Nexus7K-1a(config)#
Spanning-tree vlan 1-4094 pri 8192
vPC
agg1bagg1a
Acc1a
STP root pri 8192 Bridge ID = vPC system ID
STP root
VLAN 1 - 4094 Priority 8192
MST 0 – 4 Priority 8192
Nexus7K-1a# show spanning-tree summary | i peer
vPC peer-switch is enabled (operational)
Nexus7K-1b# show spanning-tree summary | i peer
vPC peer-switch is enabled (operational)
Nexus7K-1a# sh spanning vlan 1
---deleted---
Root ID Priority 8193
Address 0023.04ee.be01
This bridge is the root
---deleted---
Po1 Desg FWD 1 128.4096(vPC peer-link) Network P2p
Nexus7K-1a# sh spanning vlan 1
---deleted---
Root ID Priority 8193
Address 0023.04ee.be01
This bridge is the root
---deleted---
Po1 Root FWD 1 128.4096 (vPC peer-link) Network P2p
Nexus7K-1b(config-vpc-domain)#
peer-switch
Nexus7K-1b(config)#
Spanning-tree vlan 1-4094 pri 8192
vPC_PKL
vPC_PL
New in NX-OS 5.0 !!
BPDUs BPDUs
No convergence
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 60
vPC (cont.) vPC restore on reload
If both vPC switches reload, by default all vPCs are suspended until peer adjacency is reestablished between vPC devices
If only one vPC device becomes operational, the local vPC ports will remain suspended
Nexus7K(config-vpc-domain)#reload restore delay <delay>
Warning:
Enables restoring of vPCs in a peer-detached state after reload, will wait for 240 seconds (by default) to determine if peer is un-reachable
vPC
agg1bagg1a
Acc1a
vPC
agg1a
Acc1a
Both vPC switches reloaded !!
w/o vPC
Restore
on Reload
Agg1b
is
down
vPC_PKL
vPC_PL
vPC_PKL
vPC_PL
vPC
agg1a
Acc1a
with vPC
Restore
on
Reload
Agg1b
is
down
vPC_PKL
vPC_PL
vPC Primary
STP root
Wait until all LCs ups12 Start timer
3 Timer expires
Bring up local
vPC ports
vPC restore on reload allows the one vPC device to assume STP / vPC primary role and bring up all local vPCs after delay timer expiration
Both vPC switches needs to be configured
The default and minimum delay timer is 240s
4
5
Nexus7K# sh vpc 11
---deleted---
11 Po11 up success Type checks were bypassed 1,10-15
for the vPC
New in NX-OS 5.0 !!
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 61
Implementation and Best Practices
Layer-3 Features
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 62
Interior Gateway Protocol (IGP)
Enable NSF/Graceful Restart (default)
Configure IETF graceful OSPF restart on neighboring devices as Nexus 7000s only supports standard NSF
Use default IGP timers in a dual supervisor system to avoid unnecessary convergence w/ supervisor failover
Reduced IGP timers can be leverage over L2 cloud or in a single supervisor system
Bidirectional Forwarding Detection is supported in NX-OS 5.0
BFD is performed by the I/O modules
Nexus7K(config)# feature bfd
Please disable the ICMP redirects on all interfaces
running BFD sessions using the command below
'no ip redirects'
Nexus7K(config)# feature bfd
BFD Feature could not be enabled.
Please disable the address-identical IDS check for BFD Echo to be operational using the configuration command given below in the default VDC.
'no hardware ip verify address identical'
Nexus7K(config)# router eigrp 1
Nexus7K(config-router)# bfd
Nexus7K(config)# router ospf 1
Nexus7K(config-router)# bfd
Nexus7K# show bfd neighbors details
IOS(config-router)# nsf ietf
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 63
OSPFGeneral OSPF best practices
General OSPF Best Practices Notes
Enable NSF/Graceful Restart Default (IETF only)
Implement consistent auto-cost reference bandwidth default is 40G
Configure OSPF point-to-point network on point-to-point interfaces
Configure passive-interface on server VLANs
Implement routing protocol authentication
Implement OSPF route summarization
Configure deterministic router-id (loopback0)
Enable routing process on the router-id interface
Utilize OSPF stub/NSSA or totally stub/NSSA area for server VLANs
Utilize OSPF stub for IBM Open System Adaptor (OSA)
Configure intra-area transit link between the ABRs
Configure OSPF log adjacency changes disabled by default
Utilize route-map when redistributing routes default
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 64
Nexus7K(config)#
feature ospf
feature interface-vlan !
int loopback0
ip address <address>/32
ip router ospf <process> area 0
!
vlan <vlan#>!
ip prefix-list <name> seq 5 permit <net1>/<mask>
ip prefix-list <name> seq 10 permit <net2>/<mask>!
route-map <name>
match ip address prefix-list <name>
set metric <metric>!
int e<mod>/<port>
no shut
rate-mode dedicated
no ip redirects
ip address <ip address>/<mask>
ip ospf authentication message-digest
ip ospf message-digest-key <id> md5 <pw>
ip router ospf <process> area 0
ip ospf network point-to-point!
interface vlan <area 0 vlan>
ip address <ip address>/<mask>
ip router ospf <process> area 0
ip ospf network point-to-point
no shut!
interface vlan <area x vlan>
ip address <ip address>/<mask>
ip router ospf <process> area <area-x>
ip ospf network point-to-point
no shut!
interface vlan <server vlan>
no ip redirects
ip address <ip address>/<mask>
ip router ospf <process> area <area-x>
ip ospf passive-interface
no shut!
router ospf <process>
router-id <loopback>
log-adjacency-changes
redistribute static route-map <name>
auto-cost reference-bandwidth 100000
area <area-x> nssa no-summary
area <area-x> range <network/mask>
OSPF (cont.)Sample OSPF Configuration
agg1a agg1b
NSSA
Area X
Core1 Core2
agg2a agg2b
NSSA
Area Y
Backbone Area
Area 0
Summarize
routes
Summarize
routes
Summarize
routes
e1/25
Area 0 Area 0
Area xArea x
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 65
EIGRPGeneral EIGRP best practices
General EIGRP Best Practices Notes
Enable NSF/Graceful Restart default
Configure passive-interface on server VLANs
Implement routing protocol authentication
Implement EIGRP route summarization
Configure deterministic router-id (loopback0)
Enable routing process on the router-id interface
Configure EIGRP log adjacency changes default
Utilize route-map when redistributing routes default
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 66
EIGRP (cont.)Sample EIGRP Configuration
Nexus7K(config)#
feature eigrp
feature interface-vlan
!
int loopback0
ip address <address>/32
ip router eigrp 1
!
vlan <vlan#>
!
ip prefix-list <name> seq 5 permit <net1>/<mask>
ip prefix-list <name> seq 10 permit <net2>/<mask>
!
route-map <name>
match ip address prefix-list <name>
!
key chain <name>
key 1
key-string <string>
!
int e1/25
no shut
no ip redirects
ip address <ip address>/<mask>
ip router eigrp 1
ip authentication mode eigrp 1 md5
ip authentication key-chain eigrp 1 <name>
ip summary-address eigrp 1 <network>/<mask>
!
interface vlan <inter-switch vlan>
ip address <ip address>/<mask>
ip router eigrp 1
ip authentication mode eigrp 1 md5
ip authentication key-chain eigrp 1 <name>
ip summary-address eigrp 1 <network>/<mask>
no shut
!
interface vlan <server vlan>
no ip redirects
ip address <ip address>/<mask>
ip router eigrp 1
ip passive-interface eigrp 1
no shut
!
router eigrp 1
router-id <loopback>
redistribute static route-map <name>
agg1a agg1b
Core1 Core2
agg2a agg2b
Summarize
routes
Summarize
routes
Summarize
routes
e1/25
routing
peer
routing
peer
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 67
Border Gateway Protocol (BGP)
Enable NSF/Graceful Restart (default)
If full internet routing table is required, XL I/O modules should be utilized
If non-XL I/O modules are used, limit BGP table size by configuring maximum BGP AS paths and maximum BGP prefixes
Dynamic FIB TCAM allocation allows non-XL I/O modules to support up to 112k IPv4 unicast routes (supported in NX-OS 4.2 and enabled by default)
Without Dynamic FIB TCAM allocation, non-XL I/O modules support up to 56k IPv4 unicast routes
BFD is supported in NX-OS 5.0
Nexus7K(config-router)#
maxas-limit <# of AS paths>
Nexus7K(config-router-neighbor-af)#
maximum-prefix <# of prefix>
Nexus7K# hardware forwarding dynamic-allocation enable
Nexus7K# show hardware internal forwarding table utilization mod <mod>
Nexus7K# show hardware capacity forwarding | b Unicast
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 68
General Layer-3 Features
Configure extended hold timers for HSRP to support NSF during ISSU and supervisor switchovers
Not applied with sub-second timers
Configure on all HSRP routers with the same timer (default /minimum is 10s)
Sub-second FHRP timers are not recommended for a dual-sup system
Nexus7K(config)#
feature hsrp
feature interface-vlan
!
vlan <vlan>
!
hsrp timers extended-hold <time>
!
interface vlan <vlan>
description <description>
no shutdown
no ip redirects
ip address <address>/<mask>
hsrp <group>
authentication <text>
preempt delay minimum 180
priority 110
timers 1 3
ip <hsrp address>
General L3 Best Practices
Utilize per flow load-balancing (default)
Disable IP redirects
Configure HSRP preemption delay and authentication
Re-use HSRP group for all VLANs or enable HSRP v2
so HSRP group can match with the VLAN number
1s hello and 3s hold timer is recommended
If vPC is implemented, aggressive timers are not necessary
BFD is supported in NX-OS 5.0
Unsupported Layer-3 features
MPLS, NAT, Interface IP dampening, IP SLA
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 69
Implementation and Best Practices
Security Features
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 70
Access Control List (ACL)
Utilize config session manager with atomic ACL updates for non-disruptive ACL updates
Atomic Update is enabled by default
ACL management can be simplified by utilizing object groups
Nexus7K# config session test1ip access-list vlan11-acl
no 20 …….
32 permit ….
….
verify
commit
Nexus7K(config)# no hardware access-list update atomic
Nexus7K(config)# hardware access-list update default-result permit
Nexus7K(config)# hardware access-list resource pooling mod <mod>
Nexus7K(config)#
object-group ip address <name>
10.10.1.0/24
10.10.2.0/24
……
ip access-list acl1
deny ip addrgroup <name> any
XL I/O modules support 128K ACL entries per module
Enable TCAM bank-chaining on to support large ACL
XL I/O modules supports 32K entries in each TCAM bank
Non-XL I/O modules supports 16K entries in each TCAM bank
If I/O module lacks the TCAM resources, disable atomic ACL update and optionally permit all traffic during non-atomic update
Nexus7K#
int vlan 11ip access-group vlan11-acl in
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 71
Nexus7K(config)#
no feature telnet
!
vrf context management
ip route 0.0.0.0/0 <IP address>
!
ip access-list <ACL-name>
10 remark allow specific ssh
11 permit tcp <addr>/24 any eq 22
12 permit tcp any eq 22 <addr>/24
13 deny tcp any any eq 22
14 deny tcp any eq 22 any
20 remark allow specific snmp
21 permit udp <addr>/24 any eq snmp
………..
50 permit ip any any
!
interface mgmt0
ip address <ip address>/<mask>
ip access-group <ACL-name> in
!
line vty
exec-timeout <time>
session-limit <session#>
line console
exec-timeout <time>
!
int cmp-mgmt module <module>
ip address <addr>/<mask>
ip default-gateway <IP addr>
Network Access
Allow only SSH remote access (default)
If telnet access is required, “feature telnet” needs to be configured
If telnet access to CMP is required, “telnet server enable” need to be configured on the CMP
Secure interface mgmt0 with ACL
CoPP does not protect interface mgmt0
ACL with the logging option is supported in NX-OS 5.0
ACL is not supported on VTY
CoPP can be leveraged to secure VTYaccess
Configure exec-timeout for VTY and console access
Nexus7K-cmp10(config)#
telnet server enable
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 72
Control Plane Policing
Implement strict control plane policing (default)
If default policy is used, run “setup” command to reapply the default policy after software upgrade between major releases
Any non-default CoPP policies need to be reapplied after setup
Future software release will generate syslog on CoPP policy changes
Tune default CoPP policy according to needs
The configured setting is per line card and not per system. If high number of I/O modules are installed, the conform rate may need to be tuned down
Future enhancement to generate syslog messages if drops exceed user configured threshold
Critical
39600
kbps
Important
1060 kbps
Mgmt
10000
kbps
Normal
680
kbps
Redirect
280 kbps
Monitoring
130 kbps
Exception
360 kbps
Undesirable
32 kbps
conform drop
Default
100
kbps
Strict (bc ) 250 ms 1000 ms 250 ms 250 ms 250 ms 1000 ms 250 ms 250 ms 250 ms
Moderate (bc) 310 ms 1250 ms 310 ms 310 ms 310 ms 1250 ms 310 ms 310 ms 310 ms
Lenient (bc) 375 ms 1500 ms 375 ms 375 ms 375 ms 1500 ms 375 ms 375 ms 375 ms
Nexus7K# show policy-map interface control-plane | inc violated
violated 59 bytes; action: drop
Nexus7K# setup
----deleted----
Configure best practices CoPP profile (strict/moderate/lenient/none) [strict]:
per module statistics
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 73
Control Plane Policing (cont.)Tuning Example
Example: Customer utilizes ICMP to monitor the network. The ICMP packet rate exceeds the default setting for the monitoring class. Increase the CIR to allow the monitoring tools to function properly.
Nexus7K(config)# policy-map type control-plane copp-system-policy
Nexus7K(config-pmap)# class copp-system-class-monitoring
Nexus7K(config-pmap-c)# police cir 200 kbps bc 1000 ms conform transmit violate drop
Nexus7K# sh policy-map int control-plane | b monitor
class-map copp-system-class-monitoring (match-any)
match access-grp name copp-system-acl-icmp
match access-grp name copp-system-acl-icmp6
match access-grp name copp-system-acl-traceroute
police cir 200 kbps , bc 1000 ms
Nexus7K(config)# policy-map type control-plane copp-system-policy
Nexus7K(config-pmap)# class copp-system-class-normal
Nexus7K(config-pmap-c)# police cir 680 kbps bc 400 ms conform transmit violate drop
Nexus7K# sh policy-map interface control-plane | b normal
class-map copp-system-class-normal (match-any)
match access-grp name copp-system-acl-dhcp
match redirect dhcp-snoop
match protocol arp
police cir 680 kbps , bc 400 ms
Example: The newly active LB appliance sends out large amount of gratuitous ARPs after a failover and exceed the default setting for the normal class. Increase the burst interval allow to interoperate with the LB appliances
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 74
Control Plane Policing (cont.)Tuning Example
This is a sample CoPP configuration to limit SSH access to VTY. Only SSH traffic to and from the management network is allowed to access the Nexus 7000
Nexus7K(config)#
ip access-list copp-system-acl-allow
10 permit tcp <IP network>/24 any eq 22
20 permit tcp any eq 22 <IP network>/24!
ip access-list copp-system-acl-deny
1 remark ### catch-all for modified mgmt traffic ###
10 permit tcp any any eq 22
20 permit tcp any eq 22 any!
class-map type control-plane match-any copp-system-class-management
no match access-group name copp-system-acl-ssh!
class-map type control-plane match-any copp-system-class-management-allow
match access-group name copp-system-acl-allow
class-map type control-plane match-any copp-system-class-management-deny
match access-group name copp-system-acl-deny !
policy-map type control-plane copp-system-policy
class copp-system-class-management-allow insert-before copp-system-class-normal
police cir 3000 kbps bc 250 ms conform transmit violate drop
class copp-system-class-management-deny insert-before copp-system-class-normal
police cir 3000 kbps bc 250 ms conform drop violate drop
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 75
Hardware Rate-Limiter
Hardware-limiters complement CoPP to protect the CPU (enabled by default)
Rate limit supervisor-bound egress exception and egress redirected traffic
Configure on the default VDC and apply to all VDCs
The configured setting is per line card
Modify and enable hardware rate-limiters according to needs
Rate Limiter Class Default
(pps)
Layer-3 MTU 500
Layer-3 TTL 500
Layer-3 control 10,000
Layer-3 glean 100
Layer-3 multicast directly-
connected
3,000
Layer-3 multicast local-
groups
3,000
Layer-3 multicast rpf-leak 500
Layer-2 storm-control Disabled
Access-list-log 100
Copy 30,000
Receive 30,000
Layer-2 port-security Disabled
Layer-2 mcast-snooping 10,000
Layer-2 vpc-low 4,000
Nexus7K# sh hardware rate-limiter
Units for Config: packets per second
Allowed, Dropped & Total: aggregated since last clear counters
Rate Limiter Class Parameters
-------------------------------------------------
layer-3 mtu Config : 500
Allowed : 0
Dropped : 0
---deleted---
Nexus7K(config)#
hardware rate-limiter layer-2 <class> <packets/s>
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 76
Packet Sanity Checks
The IDS check performs sanity checks on the IP headers to protect the network and the system (enabled by default)
In NX-OS 5.0, the system generates syslogs on IDS drops (max is one every 30 min)
It is recommended to disable fragment IDS check since some applications sends IP fragment Packets with DNF bit
Fragment IDS check is disabled by default in NX-OS 5.0
Disable individual IDS checks as required
Ex. If BFD is configured, disable “address identical” IDS check
Nexus7K# show hardware forwarding ip verify
IPv4 and v6 IDS Checks Status Packets Failed
-----------------------------+---------+------------------
---deleted---
address identical Enabled 0
---deleted---
fragment Enabled 0
---deleted---
Nexus7K(config)# no hardware ip verify fragment
Nexus7K(config)# no hardware ip verify address identical
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 77
Nexus7K(config)#
feature ospf
feature interface-vlan!
vrf context inside!
interface vlan 10
ip address 10.1.1.1/24
ip router ospf 1 area 0!
interface vlan 20
mac-address <mac-address>
vrf member inside
ip address 10.1.1.2/24
ip router ospf 1 area 0
!
router ospf 1
vrf inside
Virtual Routing & Forwarding (VRFs)
VRFs can be utilized to provide network segmentation within VDC
VRF Import/Export is not supported
External devices or connections can be used to interconnect multiple VRFs
Policy based routing (PBR) provides the option to interconnect multiple VRFs without utilizing external connections
When forming routing peer between VRFs within the same VDC, static router mac-address must be configured to avoid address conflict
The mac-address conflict problem is masked on releases prior to 4.2(4)
Default
VRF
Inside
VRF
VLAN 10
VLAN 20
L2 FW
Default
VRF
Inside
VRF
VLAN 10
VLAN 20
L3 FW
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 78
Nexus7K(config)#
feature pbr
feature interface-vlan!
vlan 10,20!
route-map VRF-A_to_VRF-B permit 10
match ip address VRF-A_to_VRF-B
set vrf VRF-B
!
route-map VRF-B_to_VRF-A permit 10
match ip address VRF-B_to_VRF-A
set vrf VRF-A!
vrf context VRF-A
vrf context VRF-B!
interface vlan 10
vrf member VRF-A
ip address 10.1.1.0/24
ip policy route-map VRF-A_to_VRF-B
no shutdown!
interface vlan 20
vrf member VRF-B
ip address 10.1.2.0/24
ip policy route-map VRF-B_to_VRF-A
no shutdown!
ip access-list VRF-A_to_VRF-B
10 permit ip 10.1.1.0/24 10.1.2.0/24
!
ip access-list VRF-B_to_VRF-A
10 permit ip 10.1.2.0/24 10.1.1.0/24
VRFs (cont.)Sample PBR configuration
VRF-AVRF-B
VLAN 10
10.1.1.0/24VLAN 20
10.1.2.0/24
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 79
Implementation and Best Practices
Quality of Service (QoS)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 80
Quality of ServiceConfiguration Example 1
Applications COS 1P3Q4T (GE) 1P7Q4T (10GE)
OSPF, BGP, HSRP, 6, 7 Q3T3 Q7T3
Voice over IP 5 PQ PQ
HD Video Conference 4 PQ PQ
SD Video Conference 4 PQ PQ
Voice/Video Signaling 3 Q3T2 Q7T2
SSH, Telnet 3 Q3T2 Q7T2
DLSW, TACACs 2 Q2T3 Q2T3
Oracle, Citrix 2 Q2T3 Q2T3
TFTP, FTP 1 Q1T2 Q1T2
Default 0 Q1T3 Q1T3
By default, traffic with COS 5 – 7 are mapped to priority queue and traffic with
COS 0 – 4 are mapped to default queue. The default queue is assigned with
82% of the queue-limit and 25% of remaining bandwidth. A single tail drop
threshold will be used by the default queue.
This example modifies the default egress queuing based on the following table
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 81
Nexus7K(config)#
class-map type queuing match-any 1p3q4t-out-pq1
match cos 4, 5
class-map type queuing match-any 1p3q4t-out-q3
match cos 3,6-7
class-map type queuing match-any 1p3q4t-out-q2
match cos 2
class-map type queuing match-any 1p3q4t-out-q-default
match cos 0-1!----------------------------------------------------------------------------------------------------------------------------
policy-map type queuing GE-Outbound
class type queuing 1p3q4t-out-pq1
priority level 1
queue-limit percent 15
class type queuing 1p3q4t-out-q3
bandwidth remaining percent 25
queue-limit percent 15
queue-limit cos 7 percent 100
queue-limit cos 6 percent 100
queue-limit cos 3 percent 80
class type queuing 1p3q4t-out-q2
bandwidth remaining percent 50
queue-limit percent 30
class type queuing 1p3q4t-out-q-default
bandwidth remaining percent 25
queue-limit percent 40
random-detect cos-based
random-detect cos 1 minimum-threshold percent 60 maximum-threshold percent 100
random-detect cos 0 minimum-threshold percent 80 maximum-threshold percent 100!
interface e<mod>/1 – 48, e<mod>/1 – 48
service-policy type queuing output GE-Outbound
Quality of Service (cont.)Configuration Example 1
Configure on the default VDC
Configure on the specific VDC
Apply service-policy on all GE interfaces
System Queuing Class Maps
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 82
Quality of Service (cont.)Configuration Example 2
Nexus7K(config)#
policy-map type queuing 10ge-reset-cos
class type queuing 8q2t-in-q-default
set cos 0
bandwidth percent 100!
policy-map type queuing ge-reset-cos
class type queuing 2q4t-in-q-default
set cos 0
bandwidth percent 100!
ip access-list for-untrusted
10 permit ip any any
class-map type qos match-all reset-dscp
match access-group name for-untrusted
policy-map type qos reset-dscp
class reset-dscp
set dscp 0!
interface e<mod/port>
service-policy type qos input reset-dscp
service-policy type queuing input 10ge-reset-cos
By default, the port is set to trust DSCP and COS. This example configures a port as untrusted port by setting DSCPand COS to 0.
Configure on the specific VDC
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 83
Implementation and Best Practices
System Management Features
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 84
Simple Network Management Protocol
Apply ACL to control SNMP access
SNMP ACL is supported in NX-OS 4.2
Use CoPP to limit access prior to NX-OS 4.2
Configure SNMP traps notification
Enable SNMP traps and specify host receivers for SNMP traps
Specify the correct VRF (Management VRF is the default)
Nexus7K(config)#ip access-list <ACL-name>
1 remark <remark>
10 permit ip <network 1> <mask> any
20 permit ip <network 2> <mask> any!
snmp-server contact <contact>
snmp-server location <location>
snmp-server host <address> traps version <ver> <commu>
snmp-server source-interface traps <interface>
snmp-server host use-vrf <vrf-name>
snmp-server enable traps
snmp-server community <RO-string> rosnmp-server community <RO-string> group use-acl <ACL-name>
snmp-server community <RW-string> rwsnmp-server community <RW>-string group use-acl <ACL-name>
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 85
Nexus7K(config)#ntp server <address1> prefer use-vrf <vrf>
ntp server <address2> use-vrf management
ntp source-interface <interface>
clock timezone <zone> <hour offset> <min offset>
clock summer-time <zone> ….
Network Time Protocol (NTP)
NTP is only configured on the default VDC
Synchronize time with NTP servers
Prefer primary server with “prefer” key word
NTP server mode is not supported
Specify the source interface
Specify the correct VRF (default VRF is the default)
Nexus7K-1(config)#cfs ipv4 distribute
ntp distribute
ntp server ....
ntp commit
Nexus7K-2(config)#cfs ipv4 distribute
ntp distribute
CFS can be utilize to distribute NTP configuration
NTP enhancements in NX-OS 5.0
NTP Authentication, NTP ACL, NTP logging
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 86
Nexus7K(config)#
vlan 100
remote-span!
monitor session 1
description <description>
source interface <interface>
filter vlan <vlan range>
destination interface <int range>
no shut!
monitor session 2
source vlan 100
….!
monitor session 3
source interface sup-eth 0
destination interface <interface>
shut!
int <interface>
switchport
switchport monitor
no shut!
int <interface>
switchport
switchport monitor
Switchport mode trunk
Switchport trunk allowed vlan <vlan>
no shut
Switched Port Analyzer (SPAN)
Configure up to 18 SPAN session templates to simplify operation
2 active bidirectional SPAN sessions are supported per system
Virtual SPAN (VSPAN) can be used to scale the SPAN session limitation
SPAN source can be a combination of interfaces and VLANs
Configure SPAN destination port as a “monitor” port
Monitor supervisor inband port to troubleshoot control plane issues
RSPAN VLANs can be used as source VLANs or extended to another switch
Unsupported SPAN related features
VACL capture, ERSPAN, RSPAN
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 87
Nexus7K(config)#logging server <addr1> <lvl> prefer use-vrf <vrf>
Nexus7K(config)#logging server <addr2> <lvl> use-vrf <vrf>
Logging
Customize logging level for individual features as necessary
Nexus7K# show logging onboard status
Nexus7K# show logging onboard mod 1 ?
Nexus7K(config)#no hw-module logging onboard <parameter>
Parameter Default
Console Enabled, level 2
Monitor Enabled, level 5
Log file Enabled, level 5
Name=Messages
Size = 1G
Module Enabled, level 5
Time-stamp Seconds
Syslog Disabled
Nexus7K(config)#logging level spanning-tree <level>
Nexus7K(config)#logging timestamp milliseconds
Set the logging time-stamp units to millisecond
Debug logging is always time-stamped to microsecond
Enable logging to syslog servers
Standard level recommendation is notification
Support up to 3 syslog servers
Specify the correct VRF(default VRF is the default”)
Enable link-status and trunk-status logging globally (default)
Onboard Failure Logging (OBFL) is enabled by default
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 88
Nexus7K(config)#
feature tacacs+
!
username admin password <password> role network-admin
username netop password <password> role network-operator
username <name> password <password> role vdc-admin
username <name> password <password> role vdc-operator
!
tacacs-server key <unencrypted key>
ip tacacs source-interface loopback0
tacacs-server host <IP address1>
tacacs-server host <IP address2>
tacacs-server directed-request!
aaa group server tacacs+ <group-name>
server <IP address tacacs-1>
server <IP address tacacs-2>
use-vrf <vrf-name>
!
aaa authentication login default group <group-name>
aaa authentication login console local
aaa authorization config-commands default group <group-name> local
aaa authorization commands default group <group-name> local
aaa accounting default group <group-name>
aaa authentication login error-enable
AAASample Configuration
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 89
Other System Management Features
Ethanalyzer can be used to capture packets to or from the supervisor
Capture traffic on inband or management interface
Nexus7K# ethanalyzer local interface ?
inband Inband/Outband interface
mgmt Management interface
Leverage Netflow feature to provide statistics for accounting, network monitoring, and network planning
It is recommended to use Netflow in sampled mode on high BW interfaces to reduce CPU
Netflow supports version 5 and 9 export format
Version 9 to support variable field specification format, IPv6, Layer 2, MPLS field and more efficient network utilization
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 90
Nexus7K(config)#
feature netflow!flow record Netflow-Record-1description <description>match ipv4 source addressmatch ipv4 destination addressmatch transport destination-portcollect counter bytescollect counter packets!flow exporter Netflow-Exporter-1description <description>destination <destination IP>source <source interface>version 9!flow monitor Netflow-Monitor-1description <description> record Netflow-Record-1exporter Netflow-Exporter-1!interface ethernet <mod>/<port>ip flow monitor Netflow-Monitor-1 input
Other System Management Features (cont.)Sample Netflow Configuration
Nexus7K(config)#
sampler Netflow-Sampler-1
mode 1 out-of <number>
!
interface ethernet <mod>/<port>
no ip flow monitor Netflow-Monitor-1 input
ip flow monitor Netflow-Monitor-1 input sampler Netflow-Sampler-1
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 91
Q & A
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 92
Complete Your Online Session Evaluation
Give us your feedback and you could win fabulous prizes. Winners announced daily.
Receive 20 Cisco Preferred Access points for each session evaluation you complete.
Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.
Don’t forget to activate your
Cisco Live and Networkers Virtual
account for access to all session
materials, communities, and on-demand
and live activities throughout the year.
Activate your account at any internet
station or visit www.ciscolivevirtual.com.
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 93
Check the Recommended Reading brochure for
suggested products available at the Cisco Store
Enter to Win a 12-Book Libraryof Your Choice from Cisco Press
Visit the Cisco Store in the World of Solutions, where you will be asked to enter this Session ID code
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 94
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKDCT-2951 95
Hardware OverviewSupported Transceivers
As of NX-OS release 5.0
Product Number Min SW
SFP-10G-SR 4.0(1)
SFP-10G-LR 4.0(3)
SFP-10G-ER 5.0(2)
X2-10GB-LRM 5.0(2)
X2-10GB-SR 5.0(2)
X2-10GB-LR 5.0(2)
X2-10GB-ER 5.0(2)
DWDM-X2-<> 5.0(2)
Product Number Min SW
SFP-GE-S / GLC-SX-MM 4.1(2)
SFP-GE-L / GLC-LH-SM 4.1(2)
SFP-GE-Z / GLC-ZX-SM 4.1(2)
SFP-GE-T / GLC-T 4.2(1)
CWDM-SFP- 4.2(1)
DWDM-SFP- 4.2(1)