Upload
saud80
View
303
Download
9
Embed Size (px)
Citation preview
7/26/2019 Nexus Technology Labs
1/313
Nexus Technology Labs - Fabric Extenders(FEX)
FEX Active Standby
Task
Configure N5K1 to pair with the Fabric Extender N2K1 as follows:
Enable the Fabric Extender feature.
Configure N5K1's link connecting to N2K1 as a FEX port.
N2K1 should be module number 101.Configure N5K2 to pair with the Fabric Extender N2K2 as follows:
Enable the Fabric Extender feature.
Configure N5K2's link connecting to N2K2 as a FEX port.
N2K2 should be module number 102.
Configure the links between N5K1 & N5K2 as 802.1q trunk links.
Configure N5K1's links to Server 1 and the Emulex CNA Server in VLAN 10.
Configure N5K2's links to Server 2 and the Emulex CNA Server in VLAN 10.
Configure Server 1 with the IP address 10.0.0.1/24 on this link.Configure Server 2 with the IP address 10.0.0.2/24 on this link.
Configure the Emulex CNA Server to do Active Standby NIC teaming as follows:
Use the IP address 10.0.0.10/24 for the NIC Team.
Use the link to N2K1 as the primary active path and the link to N2K2 as the
secondary standby path.
Verify that both Server 1 and Server 2 have connectivity to the Emulex CNA Server,
and that traffic to the server is flowing only through N2K1.
Disable the FEX port from N5K1 to N2K1, and verify that connectivity to the CNA
Server is maintained by using the backup path through N2K2.
Configuration
N5K1:
feature fex
!
vlan 10
7/26/2019 Nexus Technology Labs
2/313
!
interface Ethernet1/1
switchport access vlan 10
spanning-tree port type edge
speed 1000
!
interface Ethernet1/3 - 5
switchport mode trunk
spanning-tree port type network
!
interface Ethernet1/10
switchport mode fex-fabric
fex associate 101
!
interface Ethernet101/1/1
switchport access vlan 10
N5K2:
feature fex
!
vlan 10
!
interface Ethernet1/2
switchport access vlan 10
spanning-tree port type edge
speed 1000
!
interface Ethernet1/3 - 5
switchport mode trunk
spanning-tree port type network
!
interface Ethernet1/11
switchport mode fex-fabric
fex associate 102
!
interface Ethernet102/1/1
switchport access vlan 10
Verification
In Active/Standby FEX topologies, hosts are physically attached to multiple FEXes,
but only actively forward on one path. Note that this topology is not related to vPC,
as vPC is used to achieve active/active forwarding, not active/standby. This
7/26/2019 Nexus Technology Labs
3/313
topology also requires the end hosts support of teaming through software. In this
particular case, the teaming is achieved through the Emulex OneCommand utility,
which manages the NIC Team/Port Channel config of the end adapter.
First, the end host is configured to team its links together, with the type defined as
failover in this case. Some other utilities call this active/standby or
primary/secondary, but they essentially mean the same thing. Note that the first
connection is listed as Primary, which is the link to N2K1 (hence N5K1), whereasthe second connection goes to N2K2 (hence N5K2).
IP addressing is configured on the logical Team adapter, similar to how IOS or NX-
OS puts logical configuration on a port channel interface.
To test the traffic flows, you can use the iPerf application to generate bulk TCP or
UDP traffic. In the output below, we see the CNA Server receiving two TCP streams
of approximately 1Gbps each, one from Server 1 and one from Server 2.
7/26/2019 Nexus Technology Labs
4/313
From the network side, the interface counters indicate that both of these flows are
going through the link to N2K1/N5K1, while the backup link through N2K2/N5K2 is
unused.
N5K1# show interface e101/1/1 | include rate
30 seconds input rate 38455104 bits/sec, 75106 packets/sec
30 seconds output rate 1819517120 bits/sec, 149849 packets/sec input rate 38.46 Mbps, 75.11 Kpps;
output rate 1.82 Gbps
, 149.85 Kpps
N5K2# show interface e102/1/1 | include rate
30 seconds input rate 1072 bits/sec, 2 packets/sec
30 seconds output rate 200 bits/sec, 0 packets/sec input rate 1.07 Kbps, 2 pps; output rate 200 bps
, 0 pps
A failure of the FEX port from N5K1 to N2K1 signals a link-down event to the end
host.
N5K1# config t
Enter configuration commands, one per line. End with CNTL/Z. N5K1(config)# int e1/10
N5K1(config-if)# shut
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/10 is down(Config change)
2013 Mar 2 19:19:46 N5K1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 1 of Fex 101 that is connected with Etherne
2013 Mar 2 19:19:46 N5K1 %NOHMS-2-NOHMS_ENV_FEX_OFFLINE: FEX-101 Off-line (Serial Number SSI16330GT8)
2013 Mar 2 19:19:46 N5K1 %PFMA-2-FEX_STATUS: Fex 101 is offline
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet101/1/1 is down (module remov
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_MODULE_REMOVED: Interface Ethernet101/1/2 is down (module remov
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/3 is down (Interface
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/4 is down (Interface
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/5 is down (Interface
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/6 is down (Interface
7/26/2019 Nexus Technology Labs
5/313
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/7 is down (Interface
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/8 is down (Interface
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/9 is down (Interface
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/10 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/11 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/12 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/13 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/14 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/15 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/16 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/17 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/18 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/19 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/20 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/21 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/22 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/23 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/24 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/25 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/26 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/27 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/28 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/29 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/30 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/31 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/32 is down (Interfac
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/1 is down (Interface
2013 Mar 2 19:19:46 N5K1 %ETHPORT-5-IF_DOWN_INTERFACE_REMOVED: Interface Ethernet101/1/2 is down (Interface
N5K1(config-if)# 2013 Mar 2 19:19:47 N5K1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 1 of Fex 101 that is conne
2013 Mar 2 19:19:47 N5K1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet1/10 is down (Administratively do
The end hosts NIC Teaming software detects the primary link failure and begins to
forward via the backup path.
7/26/2019 Nexus Technology Labs
6/313
From the network view, we see that traffic is now re-routed through the backup path
via N2K2/N5K2.
N5K2# show interface e102/1/1 | include rate
30 seconds input rate 36073720 bits/sec, 70450 packets/sec
30 seconds output rate 1706995208 bits/sec, 140583 packets/sec input rate 36.07 Mbps, 70.45 Kpps;
output rate 1.71 Gbps
, 140.58 Kpps
7/26/2019 Nexus Technology Labs
7/313
Nexus Technology Labs - Fabric Extenders(FEX)
FEX Active Active Host vPC
Task
Configure N5K1's link to Server 1 in VLAN 10.
Configure N5K2's link to Server 2 in VLAN 10.
Configure Server 1 with the IP address 10.0.0.1/24 on this link.
Configure Server 2 with the IP address 10.0.0.2/24 on this link.Configure FEX support as follows:
Configure N5K1 to pair with N2K1 using FEX number 101.
Configure N5K2 to pair with N2K2 using FEX number 102.
Configure a vPC between N5K1 and N5K2 as follows:
Configure vPC domain 1 on the vPC peers N5K1 and N5K2.
Use the mgmt0 port as the vPC Peer Keepalive link.
Use LACP for negotiation of all port channels.
Configure all links between the vPC peers as Port-Channel 1, and use thisas the vPC Peer Link.
Configure N5K1 and N5K2's links to the Emulex CNA Server as Port-
Channel 10 and vPC 10.
Port-Channel 10 should be an access port in VLAN 10.
Configure the Emulex CNA Server with LACP NIC Teaming, and use the IP address
10.0.0.10/24 for the NIC Team.
Verify that both Server 1 and Server 2 have connectivity to the Emulex CNA Server,
and that traffic to the server is being load balanced across both links through
N2K1/N5K1 and N2K2/N5K2.
Configuration
N5K1:
feature lacp
feature vpc
feature fex
7/26/2019 Nexus Technology Labs
8/313
!
vlan 10
!
vpc domain 1
peer-keepalive destination 192.168.0.52
!
interface port-channel1
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface port-channel10
switchport access vlan 10
vpc 10
!
interface Ethernet1/1
switchport access vlan 10
spanning-tree port type edge
speed 1000
!
interface Ethernet1/3 - 5
switchport mode trunk
spanning-tree port type network
channel-group 1 mode active
!
interface Ethernet1/10
switchport mode fex-fabric
fex associate 101
!
interface Ethernet101/1/1
switchport access vlan 10
channel-group 10 mode active
N5K2:
feature lacp
feature vpc
feature fex
!
vlan 10
!
vpc domain 1
peer-keepalive destination 192.168.0.51
!
interface port-channel1
switchport mode trunk
spanning-tree port type network
7/26/2019 Nexus Technology Labs
9/313
vpc peer-link
!
interface port-channel10
switchport access vlan 10
vpc 10
!
interface Ethernet1/2
switchport access vlan 10
spanning-tree port type edge
speed 1000
!
interface Ethernet1/3 - 5
switchport mode trunk
spanning-tree port type network
channel-group 1 mode active
!
interface Ethernet1/11
switchport mode fex-fabric
fex associate 102
!
interface Ethernet102/1/1
switchport access vlan 10
channel-group 10 mode active
Verification
Fabric Extender (FEX) and vPC topologies currently come in three forms. The first is
a vPC from the FEX southbound to the end server, sometimes called a Host vPC;
the second is a vPC from the FEX northbound to the parent switches, sometimes
called a Fabric vPC; and the third is both a southbound and northbound vPC from
the FEX, which is considered an Enhanced vPC or EvPC. Note that EvPC is only
supported on newer hardware platforms with corresponding newer software
releases. This particular configuration is considered the first variation, a Host vPC.
This example uses the same physical topology as before, except now the server
attached to the Fabric Extenders can do active/active forwarding. This isaccomplished by configuring a vPC between the parent switches of the FEXes.
Logically, this topology would be the same as if the CNA server were physically
wired to N5K1 with one link of its NIC, and then to N5K2 with the other link. This is
again because the FEX simply acts as a remote line card of the parent switch and
behaves just like a module of a modular switch. Because of the vPC configuration,
N5K1 and N5K2 appear to be the same upstream switch from the CNA servers
perspective; therefore, it can do active/active forwarding and load balancing just as
7/26/2019 Nexus Technology Labs
10/313
if it was dual attached to a single switch.
From the server side, the NIC Teaming software is configured to form an LACP-
based team. Note that although the terms LACPand 802.3adare normally
interchangeable, some variations of NIC Teaming software use one term to define a
channel as mode on and the other as mode active. In the case of the Emulex
OneCommand, if you choose 802.3ad teaming, you would need to configure the
channel-group 10 mode on on the NX-OS side, while LACP means that thechannel mode can be active. As shown below, the load balancing method can also
be chosen based on load, IP address, or MAC address.
Like before, the IP address goes on the logical team adapter, not the physical links.
From the network side, the first major verification is to ensure that the vPC peering
is up between the 5Ks. Only after the keepalive is confirmed and the vPC peer link
is formed can the vPC to the end host actually form. Note that the same vPC
consistency rules apply to FEX-based vPCs as to regular vPCs.
7/26/2019 Nexus Technology Labs
11/313
N5K1# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 1 Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status: success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 1
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------1 Po1 up
1,10
vPC status
----------------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------- -----------10 Po10 up
success success 10
The FEX configuration itself it technically unrelated to the vPC, as the FEX Fabric
Ports are configured the same as before.
N5K1# show interface fex-fabric
Fabric Fabric Fex FEX
Fex Port Port State Uplink Model Serial
--------------------------------------------------------------- 101 Eth1/10 Active
1 N2K-C2232PP-10GE SSI16330GT8
N5K2# show interface fex-fabric
Fabric Fabric Fex FEX
Fex Port Port State Uplink Model Serial
--------------------------------------------------------------- 102 Eth1/11 Active
2 N2K-C2232PP-10GE SSI15030C1R
For final verification, generate traffic flows between the servers and note the
interface statistics of the FEX host ports to the Emulex CNA server. In the below
output, iPerf is used to generate bulk TCP flows from Server 1 and Server 2 to the
7/26/2019 Nexus Technology Labs
12/313
CNA server.
These two flows are near line-rate for the 1GigE attached Server 1 and Server 2.
The difference between this example and the last one, however, is that these 2 x1Gbps flows are distributed between the vPC member ports to the CNA server. This
can be verified as seen through the interface counters below:
N5K1# show interface e101/1/1 | include rate
30 seconds input rate 19755792 bits/sec, 38585 packets/sec
30 seconds output rate 934557000 bits/sec, 76969 packets/sec input rate 19.75 Mbps, 38.58 Kpps;
output rate 934.56 Mbps
, 76.97 Kpps
N5K2# show interface e102/1/1 | include rate
30 seconds input rate 19651672 bits/sec, 38381 packets/sec
30 seconds output rate 934679720 bits/sec, 76978 packets/sec input rate 19.65 Mbps, 38.38 Kpps;
output rate 934.68 Mbps
, 76.98 Kpps
In the case of a link failure, traffic will automatically be rerouted to the other
available member links after LACP detects the fault. As shown below, when N5K2's
link to the downstream N2K2 FEX goes down, both 1Gbps traffic flows are rerouted
to the other FEX.
N5K2# config t
Enter configuration commands, one per line. End with CNTL/Z. N5K2(config)# int e1/11
N5K2(config-if)# shut
2013 Mar 2 22:10:07 N5K2 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/11 is down(Config change)
2013 Mar 2 22:10:07 N5K2 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 2 of Fex 102 that is connected with Etherne
2013 Mar 2 22:10:07 N5K2 %NOHMS-2-NOHMS_ENV_FEX_OFFLINE: FEX-102 Off-line (Serial Number SSI15030C1R)
2013 Mar 2 22:10:07 N5K2 %PFMA-2-FEX_STATUS: Fex 102 is offline
7/26/2019 Nexus Technology Labs
13/313
N5K1# show interface e101/1/1 | include rate
30 seconds input rate 38782896 bits/sec, 75746 packets/sec
30 seconds output rate 1838416440 bits/sec, 151406 packets/sec input rate 38.78 Mbps, 75.75 Kpps;
output rate 1.84 Gbps
, 151.41 Kpps
7/26/2019 Nexus Technology Labs
14/313
Nexus Technology Labs - Fabric Extenders(FEX)
Fabric Extenders (FEX)
Task
Configure N5K1 to pair with the Fabric Extender N2K1 as follows:
Enable the Fabric Extender feature.
Configure N5K1's link connecting to N2K1 as a FEX port
N2K1 should be module number 101.Configure N5K1's links to Server 1 and the Emulex CNA Server in VLAN 10.
These links should both be STP Edge Ports.
Configure Server 1 with the IP address 10.0.0.1/24 on this link.
Configure the Emulex CNA Server with the IP address 10.0.0.10/24 on this link.
When complete, Server 1 and the Emulex CNA Server should have IP reachability to
each other.
Configuration
7/26/2019 Nexus Technology Labs
15/313
N5K1:
feature fex
!
vlan 10
!
interface Ethernet1/1
switchport access vlan 10
spanning-tree port type edge
speed 1000
!
interface Ethernet1/10
switchport mode fex-fabric
fex associate 101
!
interface Ethernet101/1/1
switchport access vlan 10
Verification
Fabric Extenders (FEXes) are access switches that behave as remote line cards of
a parent switch. After the FEX is paired with the parent switch, such as a Nexus 5K
or 7K, all configuration occurs on the upstream parent. From the parent switchs
perspective, the FEX is simply another module or line card, and is configured as
such.
In the output below, we can see that a Nexus 2232PP FEX is paired with the parent
switch N5K1 as FEX number 101. This means that the FEX is simply treated as
module 101 from the 5Ks perspective.
N5K1# show fex
FEX FEX FEX FEX
Number Description State Model Serial
------------------------------------------------------------------------
101 FEX0101 Online N2K-C2232PP-10GE SSI16330GT8
The detailed output below shows the specifics of this FEX, such as the software
version downloaded from the parent and what the state is. When the state is online,
the most important portion of the output shown below is how the downstream FEX
ports are pinned to the upstream Fabric Ports. In this topology, there is only one
7/26/2019 Nexus Technology Labs
16/313
physical uplink from the FEX to N5K1, so all FEX ports are pinned to the Fabric Port
E1/10. In a case in which more physical links are used, the pinning of FEX ports can
be controlled with the pinning max-links command under the global FEX
configuration, or the Fabric Port can be configured as a port-channel, essentially
dynamically pinning all FEX ports to all ports in the channel at the same time. In the
latter case, traffic is then load balanced based on the Port-Channel load balancing
method.
N5K1# show fex detail
FEX: 101 Description: FEX0101 state: Online
FEX version: 5.1(3)N1(1a) [Switch version: 5.1(3)N1(1a)]
FEX Interim version: 5.1(3)N1(1a)
Switch Interim version: 5.1(3)N1(1a) Extender Serial: SSI16330GT8
Extender Model: N2K-C2232PP-10GE
, Part No: 73-12533-05
Card Id: 82, Mac Addr: 54:78:1a:30:3d:c2, Num Macs: 64
Module Sw Gen: 12594 [Switch Sw Gen: 21]
post level: complete pinning-mode: static Max-links: 1
Fabric port for control traffic: Eth1/10
FCoE Admin: false
FCoE Oper: true
FCoE FEX AA Configured: false
Fabric interface state: Eth1/10 - Interface Up. State: Active
Fex Port State Fabric Port Eth101/1/1 Up Eth1/10
Eth101/1/2 Up Eth1/10
Eth101/1/3 Down None
Eth101/1/4 Down None
Eth101/1/5 Down None
Eth101/1/6 Down None
Eth101/1/7 Down None
Eth101/1/8 Down None
Eth101/1/9 Down None
Eth101/1/10 Down None
Eth101/1/11 Down None
Eth101/1/12 Down None
Eth101/1/13 Down None
Eth101/1/14 Down None
Eth101/1/15 Down None
Eth101/1/16 Down None
Eth101/1/17 Down None
Eth101/1/18 Down None
Eth101/1/19 Down None
Eth101/1/20 Down None
Eth101/1/21 Down None
Eth101/1/22 Down None
7/26/2019 Nexus Technology Labs
17/313
Eth101/1/23 Down None
Eth101/1/24 Down None
Eth101/1/25 Down None
Eth101/1/26 Down None
Eth101/1/27 Down None
Eth101/1/28 Down None
Eth101/1/29 Down None
Eth101/1/30 Down None
Eth101/1/31 Down None
Eth101/1/32 Down None
Logs:
03/02/2013 18:17:32.337586: Module register received
03/02/2013 18:17:32.339470: Registration response sent
03/02/2013 18:17:32.465539: Module Online Sequence 03/02/2013 18:17:35.611664: Module Online
When pairing between the FEX and the parent switch is complete, furtherconfiguration of the FEX ports is the same as any other physical link. Note that there
are some behavioral differences between FEX host ports and other physical links;
for example, the FEX ports always run as STP Edge Ports with BPDU Filter and
Guard enabled. This can be seen below; although the spanning-tree port type edge
is not configured on the FEX port, it still operationally runs in that mode.
N5K1# show spanning-tree vlan 10
VLAN0010
Spanning tree enabled protocol rstp
Root ID Priority 32778
Address 000d.eca2.edbc
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)
Address 000d.eca2.edbc
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Eth1/1 Desg FWD 4 128.129 Edge P2p
Eth101/1/1 Desg FWD 2 128.1153 Edge P2p
N5K1# show spanning-tree interface e101/1/1 detail
Port 1153 (Ethernet101/1/1) of VLAN0010 is designated forwarding
Port path cost 2, Port priority 128, Port Identifier 128.1153
7/26/2019 Nexus Technology Labs
18/313
Designated root has priority 32778, address 000d.eca2.edbc
Designated bridge has priority 32778, address 000d.eca2.edbc
Designated port id is 128.1153, designated path cost 0
Timers: message age 0, forward delay 0, hold 0
Number of transitions to forwarding state: 1 The port type is edge
Link type is point-to-point by default Bpdu guard is enabled
Bpdu filter is enabled by default
BPDU: sent 11, received 0
7/26/2019 Nexus Technology Labs
19/313
Nexus Technology Labs - Fabric Extenders(FEX)
FEX Active Active Fabric vPC
Task
Configure N5K1's link to Server 1 in VLAN 10.
Configure N5K2's link to Server 2 in VLAN 10.
Configure Server 1 with the IP address 10.0.0.1/24 on this link.
Configure Server 2 with the IP address 10.0.0.2/24 on this link.Configure FEX support as follows:
Configure N5K1 and N5K2 to pair with N2K1 using FEX number 101.
Configure N5K1 and N5K2 to pair with N2K2 using FEX number 102.
Configure a vPC between N5K1 and N5K2 as follows:
Configure vPC domain 1 on the vPC peers N5K1 and N5K2.
Use the mgmt0 port as the vPC Peer Keepalive link.
Configure all links between the vPC peers as Port-Channel 1, and use this
as the vPC Peer Link.Configure the FEX Fabric ports from N5K1 and N5K2 to N2K1 as Port-
Channel 101, and as vPC 101.
Configure the FEX Fabric ports from N5K1 and N5K2 to N2K2 as Port-
Channel 102, and as vPC 102.
Configure the Emulex CNA Server to do Active Standby NIC teaming as follows:
Use the link to N2K2 as the primary active path and the link to N2K1 as the
secondary standby path.
Use the IP address 10.0.0.10/24 for the NIC Team, and assign its links to
VLAN 10.
Verify that both Server 1 and Server 2 have connectivity to the Emulex CNA Server,
and that traffic to the CNA Server is flowing only through N2K2.
Disable the link from the CNA Server to N2K2, and verify that connectivity is
maintained by using the backup path through N2K1.
7/26/2019 Nexus Technology Labs
20/313
Configuration
N5K1:
feature lacp
feature vpc
feature fex
!
vlan 10
!
vpc domain 1
peer-keepalive destination 192.168.0.52
!
interface port-channel1
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface port-channel101
switchport mode fex-fabric
fex associate 101
vpc 101
!
interface port-channel102
switchport mode fex-fabric
fex associate 102
vpc 102
!
interface Ethernet1/1
switchport access vlan 10
spanning-tree port type edge
speed 1000
!
interface Ethernet1/3 - 5
switchport mode trunk
spanning-tree port type network
channel-group 1 mode active
!
interface Ethernet1/10
switchport mode fex-fabric
fex associate 101
channel-group 101 mode on
!
interface Ethernet1/11
7/26/2019 Nexus Technology Labs
21/313
switchport mode fex-fabric
fex associate 102
channel-group 102 mode on
!
interface Ethernet101/1/1
switchport access vlan 10
!
interface Ethernet102/1/1
switchport access vlan 10
N5K2:
feature lacp
feature vpc
feature fex
!
vlan 10
!
vpc domain 1
peer-keepalive destination 192.168.0.51
!
interface port-channel1
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface port-channel101
switchport mode fex-fabric
fex associate 101
vpc 101
!
interface port-channel102
switchport mode fex-fabric
fex associate 102
vpc 102
!
interface Ethernet1/2
switchport access vlan 10
spanning-tree port type edge
speed 1000
!
interface Ethernet1/3 - 5
switchport mode trunk
spanning-tree port type network
channel-group 1 mode active
!
interface Ethernet1/10
7/26/2019 Nexus Technology Labs
22/313
switchport mode fex-fabric
fex associate 101
channel-group 101 mode on
!
interface Ethernet1/11
switchport mode fex-fabric
fex associate 102
channel-group 102 mode on
!
interface Ethernet101/1/1
switchport access vlan 10
!
interface Ethernet102/1/1
switchport access vlan 10
Verification
Fabric Extender (FEX) and vPC topologies currently come in three forms. The first is
a vPC from the FEX southbound to the end server, sometimes called a Host vPC,
the second is a vPC from the FEX northbound to the parent switches, sometimes
called a Fabric vPC, and the third is both a southbound and northbound vPC from
the FEX, which is considered an Enhanced vPC, or EvPC. Note that EvPC is only
supported on newer hardware platforms with corresponding newer software
releases. This particular configuration is considered the second variation, the Fabric
vPC.
In the Fabric vPC, the end host may be single or dual attached to one or more
FEXes, but the FEX does not perform port channeling southbound to the end host.
Instead, the FEX forms a vPC northbound to multiple parent switches. Although this
does not contribute to any load distribution between the end server and the FEX, it
does more evenly distribute the load from the FEXes northbound to their parents.
The potential danger with this design, however, is that multiple parent switches,
each with separate management and control planes, are referencing the same FEX
host ports. This means that if the configuration becomes out of sync between the
parent switches, there could be a problem in the data plane of the FEX host ports. A
possible resolution to this problem is to use the Configuration Synchronization
feature, which is demonstrated in a separate scenario.
The configuration of this scenario is similar to the other FEX pairings, with the
exception that the port channel southbound from the parent switch to the FEX does
not run LACP. This is because the FEX uplink ports do not support LACP: they only
support static channels. When complete, both parent switches must agree on
7/26/2019 Nexus Technology Labs
23/313
identical configurations down to the FEX Fabric Ports and to the FEX Host Ports.
The consistency of the parent switches configurations is protected against using the
vPC consistency check. Note that the show vpc output below indicates that each of
the FEX Host Ports now participates in the vPC, even though there is not
channeling configured on the host ports. Like other vPC configurations, the first
verification should be that the vPC Peer Keepalive is up and the vPC Peer Link
adjacency has been formed.
N5K1# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 1 Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status: success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary Number of vPCs configured : 66
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------1 Po1 up
1,10
vPC status
----------------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------- ----------- 101 Po101 up
success success - 102 Po102 up
success success -
102400 Eth101/1/1 up success success 10
102401 Eth101/1/2 up success success 1
102402 Eth101/1/3 down* Not Consistency Check Not -
Applicable Performed
102403 Eth101/1/4 down* Not Consistency Check Not -
Applicable Performed
102404 Eth101/1/5 down* Not Consistency Check Not -
Applicable Performed
7/26/2019 Nexus Technology Labs
24/313
102405 Eth101/1/6 down* Not Consistency Check Not -
Applicable Performed
102406 Eth101/1/7 down* Not Consistency Check Not -
Applicable Performed
102407 Eth101/1/8 down* Not Consistency Check Not -
Applicable Performed
102408 Eth101/1/9 down* Not Consistency Check Not -
Applicable Performed
102409 Eth101/1/10 down* Not Consistency Check Not -
Applicable Performed
102410 Eth101/1/11 down* Not Consistency Check Not -
Applicable Performed
102411 Eth101/1/12 down* Not Consistency Check Not -
Applicable Performed
102412 Eth101/1/13 down* Not Consistency Check Not -
Applicable Performed
102413 Eth101/1/14 down* Not Consistency Check Not -
Applicable Performed
102414 Eth101/1/15 down* Not Consistency Check Not -
Applicable Performed
102415 Eth101/1/16 down* Not Consistency Check Not -
Applicable Performed
102416 Eth101/1/17 down* Not Consistency Check Not -
Applicable Performed
102417 Eth101/1/18 down* Not Consistency Check Not -
Applicable Performed
102418 Eth101/1/19 down* Not Consistency Check Not -
Applicable Performed
102419 Eth101/1/20 down* Not Consistency Check Not -
Applicable Performed
102420 Eth101/1/21 down* Not Consistency Check Not -
Applicable Performed
102421 Eth101/1/22 down* Not Consistency Check Not -
Applicable Performed
102422 Eth101/1/23 down* Not Consistency Check Not -
Applicable Performed
102423 Eth101/1/24 down* Not Consistency Check Not -
Applicable Performed
102424 Eth101/1/25 down* Not Consistency Check Not -
Applicable Performed
102425 Eth101/1/26 down* Not Consistency Check Not -
Applicable Performed
102426 Eth101/1/27 down* Not Consistency Check Not -
Applicable Performed
102427 Eth101/1/28 down* Not Consistency Check Not -
Applicable Performed
7/26/2019 Nexus Technology Labs
25/313
102428 Eth101/1/29 down* Not Consistency Check Not -
Applicable Performed
102429 Eth101/1/30 down* Not Consistency Check Not -
Applicable Performed
102430 Eth101/1/31 down* Not Consistency Check Not -
Applicable Performed
102431 Eth101/1/32 down* Not Consistency Check Not -
Applicable Performed
103424 Eth102/1/1 up success success 10
103425 Eth102/1/2 up success success 1
103426 Eth102/1/3 down* Not Consistency Check Not -
Applicable Performed
103427 Eth102/1/4 down* Not Consistency Check Not -
Applicable Performed
103428 Eth102/1/5 down* Not Consistency Check Not -
Applicable Performed
103429 Eth102/1/6 down* Not Consistency Check Not -
Applicable Performed
103430 Eth102/1/7 down* Not Consistency Check Not -
Applicable Performed
103431 Eth102/1/8 down* Not Consistency Check Not -
Applicable Performed
103432 Eth102/1/9 down* Not Consistency Check Not -
Applicable Performed
103433 Eth102/1/10 down* Not Consistency Check Not -
Applicable Performed
103434 Eth102/1/11 down* Not Consistency Check Not -
Applicable Performed
103435 Eth102/1/12 down* Not Consistency Check Not -
Applicable Performed
103436 Eth102/1/13 down* Not Consistency Check Not -
Applicable Performed
103437 Eth102/1/14 down* Not Consistency Check Not -
Applicable Performed
103438 Eth102/1/15 down* Not Consistency Check Not -
Applicable Performed
103439 Eth102/1/16 down* Not Consistency Check Not -
Applicable Performed
103440 Eth102/1/17 down* Not Consistency Check Not -
Applicable Performed
103441 Eth102/1/18 down* Not Consistency Check Not -
Applicable Performed
103442 Eth102/1/19 down* Not Consistency Check Not -
7/26/2019 Nexus Technology Labs
26/313
Applicable Performed
103443 Eth102/1/20 down* Not Consistency Check Not -
Applicable Performed
103444 Eth102/1/21 down* Not Consistency Check Not -
Applicable Performed
103445 Eth102/1/22 down* Not Consistency Check Not -
Applicable Performed
103446 Eth102/1/23 down* Not Consistency Check Not -
Applicable Performed
103447 Eth102/1/24 down* Not Consistency Check Not -
Applicable Performed
103448 Eth102/1/25 down* Not Consistency Check Not -
Applicable Performed
103449 Eth102/1/26 down* Not Consistency Check Not -
Applicable Performed
103450 Eth102/1/27 down* Not Consistency Check Not -
Applicable Performed
103451 Eth102/1/28 down* Not Consistency Check Not -
Applicable Performed
103452 Eth102/1/29 down* Not Consistency Check Not -
Applicable Performed
103453 Eth102/1/30 down* Not Consistency Check Not -
Applicable Performed
103454 Eth102/1/31 down* Not Consistency Check Not -
Applicable Performed
103455 Eth102/1/32 down* Not Consistency Check Not -
Applicable Performed
N5K2# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 1 Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status: success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : secondary Number of vPCs configured : 66
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
7/26/2019 Nexus Technology Labs
27/313
1 Po1 up
1,10
vPC status
----------------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------- ----------- 101 Po101 up
success success - 102 Po102 up
success success -
102400 Eth101/1/1 up success success 10
102401 Eth101/1/2 up success success 1
102402 Eth101/1/3 down* Not Consistency Check Not -
Applicable Performed
102403 Eth101/1/4 down* Not Consistency Check Not -
Applicable Performed
102404 Eth101/1/5 down* Not Consistency Check Not -
Applicable Performed
102405 Eth101/1/6 down* Not Consistency Check Not -
Applicable Performed
102406 Eth101/1/7 down* Not Consistency Check Not -
Applicable Performed
102407 Eth101/1/8 down* Not Consistency Check Not -
Applicable Performed
102408 Eth101/1/9 down* Not Consistency Check Not -
Applicable Performed
102409 Eth101/1/10 down* Not Consistency Check Not -
Applicable Performed
102410 Eth101/1/11 down* Not Consistency Check Not -
Applicable Performed
102411 Eth101/1/12 down* Not Consistency Check Not -
Applicable Performed
102412 Eth101/1/13 down* Not Consistency Check Not -
Applicable Performed
102413 Eth101/1/14 down* Not Consistency Check Not -
Applicable Performed
102414 Eth101/1/15 down* Not Consistency Check Not -
Applicable Performed
102415 Eth101/1/16 down* Not Consistency Check Not -
Applicable Performed
102416 Eth101/1/17 down* Not Consistency Check Not -
Applicable Performed
102417 Eth101/1/18 down* Not Consistency Check Not -
Applicable Performed
7/26/2019 Nexus Technology Labs
28/313
102418 Eth101/1/19 down* Not Consistency Check Not -
Applicable Performed
102419 Eth101/1/20 down* Not Consistency Check Not -
Applicable Performed
102420 Eth101/1/21 down* Not Consistency Check Not -
Applicable Performed
102421 Eth101/1/22 down* Not Consistency Check Not -
Applicable Performed
102422 Eth101/1/23 down* Not Consistency Check Not -
Applicable Performed
102423 Eth101/1/24 down* Not Consistency Check Not -
Applicable Performed
102424 Eth101/1/25 down* Not Consistency Check Not -
Applicable Performed
102425 Eth101/1/26 down* Not Consistency Check Not -
Applicable Performed
102426 Eth101/1/27 down* Not Consistency Check Not -
Applicable Performed
102427 Eth101/1/28 down* Not Consistency Check Not -
Applicable Performed
102428 Eth101/1/29 down* Not Consistency Check Not -
Applicable Performed
102429 Eth101/1/30 down* Not Consistency Check Not -
Applicable Performed
102430 Eth101/1/31 down* Not Consistency Check Not -
Applicable Performed
102431 Eth101/1/32 down* Not Consistency Check Not -
Applicable Performed
103424 Eth102/1/1 up success success 10
103425 Eth102/1/2 up success success 1
103426 Eth102/1/3 down* Not Consistency Check Not -
Applicable Performed
103427 Eth102/1/4 down* Not Consistency Check Not -
Applicable Performed
103428 Eth102/1/5 down* Not Consistency Check Not -
Applicable Performed
103429 Eth102/1/6 down* Not Consistency Check Not -
Applicable Performed
103430 Eth102/1/7 down* Not Consistency Check Not -
Applicable Performed
103431 Eth102/1/8 down* Not Consistency Check Not -
Applicable Performed
103432 Eth102/1/9 down* Not Consistency Check Not -
Applicable Performed
7/26/2019 Nexus Technology Labs
29/313
103433 Eth102/1/10 down* Not Consistency Check Not -
Applicable Performed
103434 Eth102/1/11 down* Not Consistency Check Not -
Applicable Performed
103435 Eth102/1/12 down* Not Consistency Check Not -
Applicable Performed
103436 Eth102/1/13 down* Not Consistency Check Not -
Applicable Performed
103437 Eth102/1/14 down* Not Consistency Check Not -
Applicable Performed
103438 Eth102/1/15 down* Not Consistency Check Not -
Applicable Performed
103439 Eth102/1/16 down* Not Consistency Check Not -
Applicable Performed
103440 Eth102/1/17 down* Not Consistency Check Not -
Applicable Performed
103441 Eth102/1/18 down* Not Consistency Check Not -
Applicable Performed
103442 Eth102/1/19 down* Not Consistency Check Not -
Applicable Performed
103443 Eth102/1/20 down* Not Consistency Check Not -
Applicable Performed
103444 Eth102/1/21 down* Not Consistency Check Not -
Applicable Performed
103445 Eth102/1/22 down* Not Consistency Check Not -
Applicable Performed
103446 Eth102/1/23 down* Not Consistency Check Not -
Applicable Performed
103447 Eth102/1/24 down* Not Consistency Check Not -
Applicable Performed
103448 Eth102/1/25 down* Not Consistency Check Not -
Applicable Performed
103449 Eth102/1/26 down* Not Consistency Check Not -
Applicable Performed
103450 Eth102/1/27 down* Not Consistency Check Not -
Applicable Performed
103451 Eth102/1/28 down* Not Consistency Check Not -
Applicable Performed
103452 Eth102/1/29 down* Not Consistency Check Not -
Applicable Performed
103453 Eth102/1/30 down* Not Consistency Check Not -
Applicable Performed
103454 Eth102/1/31 down* Not Consistency Check Not -
Applicable Performed
103455 Eth102/1/32 down* Not Consistency Check Not -
7/26/2019 Nexus Technology Labs
30/313
Applicable Performed
Note that both N5K1 and N5K2 are pairing with the same downstream FEXes.
N5K1# show interface fex-fabric
Fabric Fabric Fex FEX
Fex Port Port State Uplink Model Serial
---------------------------------------------------------------
101 Eth1/10 Active 1 N2K-C2232PP-10GE SSI16330GT8
102 Eth1/11 Active 1 N2K-C2232PP-10GE SSI15030C1R
N5K2# show interface fex-fabric
Fabric Fabric Fex FEX
Fex Port Port State Uplink Model Serial
---------------------------------------------------------------
101 Eth1/10 Active 2 N2K-C2232PP-10GE SSI16330GT8
102 Eth1/11 Active 2 N2K-C2232PP-10GE SSI15030C1R
From the end servers perspective, their links have been configured in
active/standby failover teaming, with the primary path being to N2K2.
For final verification of traffic distribution, Server 1 and Server 2 generate TCP flows
toward the CNA server. The end result is multiple flows for a total nearing 2Gbps,
which is the combined line rates of Server 1 and 2.
7/26/2019 Nexus Technology Labs
31/313
From the network side, both N5K1 and N5K2 see that all flows towards the CNA
server exit via N2K2.
N5K1# show interface e101/1/1 | include rate
30 seconds input rate 1072 bits/sec, 2 packets/sec
30 seconds output rate 880 bits/sec, 1 packets/sec input rate 1.07 Kbps, 2 pps; output rate 912 bps
, 1 pps
N5K1# show interface e102/1/1 | include rate
30 seconds input rate 43354968 bits/sec, 84644 packets/sec
30 seconds output rate 1965663832 bits/sec, 161924 packets/sec input rate 43.14 Mbps, 84.21 Kpps;
output rate 1.95 Gbps
, 160.63 Kpps
N5K2# show interface e101/1/1 | include rate
30 seconds input rate 1072 bits/sec, 2 packets/sec
30 seconds output rate 704 bits/sec, 1 packets/sec input rate 1.07 Kbps, 2 pps; output rate 912 bps
, 1 pps
N5K2# show interface e102/1/1 | include rate
30 seconds input rate 43728760 bits/sec, 85367 packets/sec
30 seconds output rate 1967795096 bits/sec, 162099 packets/sec input rate 43.15 Mbps, 84.23 Kpps;
output rate 1.95 Gbps
, 160.63 Kpps
Note that these outputs are nearly identical on both parent switches, as they are
both referencing the same physical FEX host ports. The key difference with this
configuration design, however, is that traffic is load balanced from the parent
switches down to the N2K2 FEX. This can be verified by viewing the counters of the
FEX Fabric Ports of the parent switches, as seen below.
N5K1# show interface e1/10 - 11 | include rate|Ethernet1
Ethernet1/10 is up
30 seconds input rate 19152 bits/sec, 2 packets/sec
7/26/2019 Nexus Technology Labs
32/313
30 seconds output rate 3184 bits/sec, 2 packets/sec input rate 18.55 Kbps, 2 pps;
output rate 2.94 Kbps
, 2 pps
Ethernet1/11 is up
30 seconds input rate 17752 bits/sec, 2 packets/sec
30 seconds output rate 989910144 bits/sec, 81007 packets/sec input rate 23.91 Kbps, 2 pps;
output rate 985.18 Mbps
, 80.56 Kpps
N5K2# show interface e1/10 - 11 | include rate|Ethernet1
Ethernet1/10 is up
30 seconds input rate 9232 bits/sec, 1 packets/sec
30 seconds output rate 3288 bits/sec, 2 packets/sec input rate 16.00 Kbps, 2 pps;
output rate 2.84 Kbps
, 2 pps
Ethernet1/11 is up
30 seconds input rate 50142584 bits/sec, 84628 packets/sec
30 seconds output rate 990521328 bits/sec, 81056 packets/sec input rate 50.03 Mbps, 84.39 Kpps;
output rate 986.24 Mbps
, 80.64 Kpps
Note that neither parent switch is sending traffic to FEX Fabric Port E1/10 because
this is the link to N2K1, which the CNA server is using as the standby connection.
Although the total of flows from Server 1 and 2 are 2Gbps, they are nearly equally
split between the FEX Fabric Ports going from N5K1 and N5K2 southbound to the
N2K2 FEX.
In the case of a failure of the servers primary uplink, traffic is automatically re-routed to the backup link via FEX N2K1, as shown below.
N5K1# show interface e1/10 - 11 | include rate|Ethernet1
Ethernet1/10 is up
30 seconds input rate 20552 bits/sec, 2 packets/sec
7/26/2019 Nexus Technology Labs
33/313
30 seconds output rate 987383384 bits/sec
, 80800 packets/sec
input rate 17.62 Kbps, 2 pps; output rate 518.97 Mbps, 42.43 Kpps
Ethernet1/11 is up
30 seconds input rate 8624 bits/sec, 1 packets/sec 30 seconds output rate 3128 bits/sec
, 2 packets/sec
input rate 16.26 Kbps, 2 pps; output rate 463.30 Mbps, 37.85 Kpps
N5K2# show interface e1/10 - 11 | include rate|Ethernet1
Ethernet1/10 is up
30 seconds input rate 50370824 bits/sec, 85031 packets/sec 30 seconds output rate 990017536 bits/sec
, 81015 packets/sec
input rate 24.29 Mbps, 40.97 Kpps; output rate 478.05 Mbps, 39.09 Kpps
Ethernet1/11 is up
30 seconds input rate 24968 bits/sec, 2 packets/sec 30 seconds output rate 3040 bits/sec
, 2 packets/sec
input rate 25.69 Mbps, 43.28 Kpps; output rate 505.58 Mbps, 41.32 Kpps
7/26/2019 Nexus Technology Labs
34/313
Nexus Technology Labs - Fabric Extenders(FEX)
FEX and N5K Config Sync
Task
Enable the vPC, FEX, and LACP features on N5K1 and N5K2.
Enable Cisco Fabric Services over IP (CFSoIP) distribution between N5K1 and N5K2.
Configure vPC domain 1 between N5K1 and N5K2, and use the mgmt0 link for the
vPC Peer Keepalive.Create a Config Sync session on both N5K1 and N5K2, and use the switch profile
name N5K.
Use the mgmt0 IP addresses as the config sync peers destination.
Verify that N5K1 and N5K2 can reach each other over CFSoIP for the config sync
session.
Without making any additional changes on N5K2, use the switch profile on N5K1 to
replicate the following configuration to both switches:
Pre-provision FEX modules 101 and 102, both of type N2K-C2232P.Create VLAN 10.
All links to Server 1 and Server 2 should be access ports in VLAN 10.
Configure all links between the vPC peers as Port-Channel 1, and use this
as the vPC Peer Link.
Configure N5K1 and N5K2 to pair with N2K1 using FEX number 101.
Configure N5K1 and N5K2 to pair with N2K2 using FEX number 102.
Configure the FEX Fabric ports from N5K1 and N5K2 to N2K1 as Port-
Channel 101, and as vPC 101.
Configure the FEX Fabric ports from N5K1 and N5K2 to N2K2 as Port-
Channel 102, and as vPC 102.
Configure the links to the Emulex CNA Server in VLAN 10.
Commit the config and verify that both N5K1 and N5K2 identically accept it into their
running configuration.
7/26/2019 Nexus Technology Labs
35/313
Configuration
N5K2:
feature vpc
feature fex
feature lacp
cfs ipv4 distribute
!
vpc domain 1
peer-keepalive destination 192.168.0.51 vrf management
!
end
config sync
switch-profile N5K
sync-peers destination 192.168.0.51
verify
N5K1:
feature vpc
feature fex
feature lacp
cfs ipv4 distribute
!
vpc domain 1
peer-keepalive destination 192.168.0.52 vrf management
!
end
config sync
switch-profile N5K
sync-peers destination 192.168.0.52
verify
show switch-profile status
slot 101
provision model N2K-C2232P
!
slot 102
provision model N2K-C2232P
!
vlan 10
!
interface port-channel1
7/26/2019 Nexus Technology Labs
36/313
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface port-channel101
switchport mode fex-fabric
fex associate 101
vpc 101
!
interface port-channel102
switchport mode fex-fabric
fex associate 102
vpc 102
!
interface Ethernet1/1 - 2
switchport access vlan 10
spanning-tree port type edge
speed 1000
!
interface Ethernet1/3 - 5
switchport mode trunk
spanning-tree port type network
channel-group 1 mode active
!
interface Ethernet1/10
switchport mode fex-fabric
fex associate 101
channel-group 101 mode on
!
interface Ethernet1/11
switchport mode fex-fabric
fex associate 102
channel-group 102 mode on
!
interface Ethernet101/1/1
switchport access vlan 10
!
interface Ethernet102/1/1
switchport access vlan 10
!
commit
7/26/2019 Nexus Technology Labs
37/313
Verification
Configuration Synchronization, also known as Config Sync for short or Switch
Profiles, is a way to apply a template of configuration onto multiple Nexus switches
at the same time. This feature is especially useful between vPC peers, or when FEX
deployments are used in active/active, where a downstream FEX peers with morethan one upstream parent switch. This feature helps to ensure that configurations
stay consistent between the vPC peers or FEX parents (or both as in the case
shown below), and avoid problems such as vPC failure caused by a consistency
check error.
Note that of current releases, this feature is not supported on the Nexus 7K.
Additionally, not all commands are supported in the switch profile mode, and instead
must be configured in regular global config. In this particular case, the unsupported
commands are the feature enablement of vPC, FEX, and LACP, as well as the vPC
domain creation, as the configuration is different between the vPC peers(specifically the peer keepalive destination address).
To use config sync, the switches must first be configured to use Cisco Fabric
Services over IP (CFSoIP), as this is the control plane protocol that is actually used
to sync the config between the switches. This is enabled simply as follows:
N5K1# conf t
Enter configuration commands, one per line. End with CNTL/Z.N5K1(config)# cfs ipv4 distribute
N5K1(config)# show cfs peers
Physical Fabric
-------------------------------------------------------------------------
Switch WWN IP Address
------------------------------------------------------------------------- 20:00:00:0d:ec:a2:ed:80
192.168.0.51
[Local] N5K1
20:00:00:0d:ec:a4:74:00 192.168.0.52
Total number of entries = 2
Next, start a config sync session, create an identically named switch profile on each
switch, specify the IP address of the peer to sync with, and then verify their
connectivity.
N5K1# config sync
7/26/2019 Nexus Technology Labs
38/313
Enter configuration commands, one per line. End with CNTL/Z. N5K1(config-sync)#switch-profile N5K
Switch-Profile started, Profile ID is 1 N5K1(config-sync-sp)# sync-peers destination 192.168.0.52
N5K1(config-sync-sp)# verify
Verification Successful
N5K1(config-sync-sp)#
vN5K2# config sync
Enter configuration commands, one per line. End with CNTL/Z.N5K2(config-sync)#switch-profile N5K
Switch-Profile started, Profile ID is 1 N5K2(config-sync-sp)# sync-peers destination 192.168.0.51
N5K2(config-sync-sp)# verify
Verification Successful
N5K2(config-sync-sp)#
If the switch profile is in sync between the peers, they should both agree on the
profile revision number and show the sync status as in sync.
N5K1(config-sync-sp)# show switch-profile status
switch-profile : N5K
----------------------------------------------------------
Start-time: 382018 usecs after Sun Mar 3 15:44:48 2013
End-time: 441035 usecs after Sun Mar 3 15:44:50 2013
Profile-Revision: 1
Session-type: Initial-Exchange
Session-subtype: Init-Exchange-All
Peer-triggered: Yes
Profile-status: Sync Success
Local information:
---------------- Status: Commit Success
Error(s):
Peer information:
---------------- IP-address: 192.168.0.52
Sync-status: In sync
Status: Commit Success
Error(s):
N5K2(config-sync-sp)# show switch-profile status
switch-profile : N5K
----------------------------------------------------------
Start-time: 831674 usecs after Sun Mar 3 16:35:38 2013
End-time: 875222 usecs after Sun Mar 3 16:35:40 2013
7/26/2019 Nexus Technology Labs
39/313
Profile-Revision: 1
Session-type: Initial-Exchange
Session-subtype: Init-Exchange-All
Peer-triggered: No
Profile-status: Sync Success
Local information:
---------------- Status: Commit Success
Error(s):
Peer information:
---------------- IP-address: 192.168.0.51
Sync-status: In sync
Status: Commit Success
Error(s):
Now the switches are ready to accept the configuration changes to synchronize.
Commands are entered just like in global config, but they are not immediately
applied. Instead they are sent to the switch profile buffer, as shown below. Before
the buffer is committed, the buffer can be deleted or modified as desired. The line
numbers of the buffer show how the config will sequentially be applied. Therefore,
any configurations that are sensitive to order of operations must have the correct
line numbering in the buffer before a commit is executed.
N5K1(config-sync-sp)# slot 101
N5K1(config-sync-sp-slot)# provision model N2K-C2232P
N5K1(config-sync-sp-slot)# !
N5K1(config-sync-sp-slot)# slot 102
N5K1(config-sync-sp-slot)# provision model N2K-C2232P
N5K1(config-sync-sp-slot)# !
N5K1(config-sync-sp-slot)# vlan 10
N5K1(config-sync-sp-vlan)# !
N5K1(config-sync-sp-vlan)# interface port-channel1
N5K1(config-sync-sp-if)# switchport mode trunk
N5K1(config-sync-sp-if)# spanning-tree port type network
N5K1(config-sync-sp-if)# vpc peer-link
N5K1(config-sync-sp-if)# !
N5K1(config-sync-sp-if)# interface port-channel101
N5K1(config-sync-sp-if)# switchport mode fex-fabric
N5K1(config-sync-sp-if)# fex associate 101
N5K1(config-sync-sp-if)# vpc 101
N5K1(config-sync-sp-if)# !
N5K1(config-sync-sp-if)# interface port-channel102
7/26/2019 Nexus Technology Labs
40/313
N5K1(config-sync-sp-if)# switchport mode fex-fabric
N5K1(config-sync-sp-if)# fex associate 102
N5K1(config-sync-sp-if)# vpc 102
N5K1(config-sync-sp-if)# !
N5K1(config-sync-sp-if)# interface Ethernet1/1 - 2
N5K1(config-sync-sp-if-range)# switchport access vlan 10
N5K1(config-sync-sp-if-range)# spanning-tree port type edge
N5K1(config-sync-sp-if-range)# speed 1000
N5K1(config-sync-sp-if-range)# !
N5K1(config-sync-sp-if-range)# interface Ethernet1/3 - 5
N5K1(config-sync-sp-if-range)# switchport mode trunk
N5K1(config-sync-sp-if-range)# spanning-tree port type network
N5K1(config-sync-sp-if-range)# channel-group 1 mode active
N5K1(config-sync-sp-if-range)# !
N5K1(config-sync-sp-if-range)# interface Ethernet1/10
N5K1(config-sync-sp-if)# switchport mode fex-fabric
N5K1(config-sync-sp-if)# fex associate 101
N5K1(config-sync-sp-if)# channel-group 101 mode on
N5K1(config-sync-sp-if)# !
N5K1(config-sync-sp-if)# interface Ethernet1/11
N5K1(config-sync-sp-if)# switchport mode fex-fabric
N5K1(config-sync-sp-if)# fex associate 102
N5K1(config-sync-sp-if)# channel-group 102 mode on
N5K1(config-sync-sp-if)# !
N5K1(config-sync-sp-if)# interface Ethernet101/1/1
N5K1(config-sync-sp-if)# switchport access vlan 10
N5K1(config-sync-sp-if)# !
N5K1(config-sync-sp-if)# interface Ethernet102/1/1
N5K1(config-sync-sp-if)# switchport access vlan 10
N5K1(config-sync-sp-if)#
N5K1(config-sync-sp-if)# show switch-profile buffer
switch-profile : N5K
----------------------------------------------------------
Seq-no Command
----------------------------------------------------------
1 slot 101
1.1 provision model N2K-C2232P
2 slot 102
2.1 provision model N2K-C2232P
3 vlan 10
4 interface port-channel1
4.1 switchport mode trunk
4.2 spanning-tree port type network
4.3 vpc peer-link
7/26/2019 Nexus Technology Labs
41/313
5 interface port-channel101
5.1 switchport mode fex-fabric
5.2 fex associate 101
5.3 vpc 101
6 interface port-channel102
6.1 switchport mode fex-fabric
6.2 fex associate 102
6.3 vpc 102
7 interface Ethernet1/1-2
7.1 switchport access vlan 10
7.2 spanning-tree port type edge
7.3 speed 1000
8 interface Ethernet1/3-5
8.1 switchport mode trunk
8.2 spanning-tree port type network
8.3 channel-group 1 mode active
9 interface Ethernet1/10
9.1 switchport mode fex-fabric
9.2 fex associate 101
9.3 channel-group 101 mode on
10 interface Ethernet1/11
10.1 switchport mode fex-fabric
10.2 fex associate 102
10.3 channel-group 102 mode on
11 interface Ethernet101/1/1
11.1 switchport access vlan 10
12 interface Ethernet102/1/1
12.1 switchport access vlan 10
When the commands in the buffer are acceptable, the profile is committed. During
the commit procedure, the config is synchronized across to the other peer using
CFSoIP, and applied sequentially. If there is an error in applying the config, all
commands in the buffer are rolled back and the commit fails. In other words, either
the commit succeeds 100 percent, or no config is applied to either peer. In the
output below, we see that the commit was successful, and syslog messages begin
to appear as config changes, link up/down events, etc. occur just as if you hadapplied the commands manually on each switch individually.
7/26/2019 Nexus Technology Labs
42/313
N5K1(config-sync-sp-if)# commit
Verification successful...
Proceeding to apply configuration. This might take a while depending on amount of configuration in buffer.
Please avoid other configuration changes during this time.
2013 Mar 3 15:49:31 N5K1 %ETH_PORT_CHANNEL-5-CREATED: port-channel1 created
2013 Mar 3 15:49:31 N5K1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel1 is down (No
2013 Mar 3 15:49:33 N5K1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel1 is down (No
2013 Mar 3 15:49:33 N5K1 %ETH_PORT_CHANNEL-5-CREATED: port-channel101 created
2013 Mar 3 15:49:33 N5K1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel101 is down (N
2013 Mar 3 15:49:33 N5K1 last message repeated 2 times
2013 Mar 3 15:49:33 N5K1 %ETH_PORT_CHANNEL-5-CREATED: port-channel102 created
Commit Successful
N5K1(config-sync)#
N5K2(config-sync-sp)#
2013 Mar 3 16:40:22 N5K2 %ETH_PORT_CHANNEL-5-CREATED: port-channel1 created
2013 Mar 3 16:40:22 N5K2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel1 is down (No
2013 Mar 3 16:40:24 N5K2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel1 is down (No
2013 Mar 3 16:40:24 N5K2 %ETH_PORT_CHANNEL-5-CREATED: port-channel101 created
2013 Mar 3 16:40:24 N5K2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel101 is down (N
2013 Mar 3 16:40:24 N5K2 last message repeated 2 times
2013 Mar 3 16:40:24 N5K2 %ETH_PORT_CHANNEL-5-CREATED: port-channel102 created
When the commit is successful, N5K1 automatically exits out of the switch profile
configuration mode. If additional config changes are required, a new switch profile
session must be started, using the same session name as before. Note in the output
below that both switches agree on the switch profile revision number, and the
switches are in sync.
N5K1(config-sync)# show switch-profile status
switch-profile : N5K
----------------------------------------------------------
Start-time: 130047 usecs after Sun Mar 3 15:49:28 2013
End-time: 663864 usecs after Sun Mar 3 15:49:38 2013
Profile-Revision: 2
Session-type: Commit
Session-subtype: -
Peer-triggered: No Profile-status: Sync Success
7/26/2019 Nexus Technology Labs
43/313
Sync-status: In sync Status: Commit Success
Error(s):
N5K2(config-sync-sp)# show switch-profile status
switch-profile : N5K
----------------------------------------------------------
Start-time: 830375 usecs after Sun Mar 3 16:40:18 2013
End-time: 361267 usecs after Sun Mar 3 16:40:29 2013
Profile-Revision: 2
Session-type: Commit
Session-subtype: -
Peer-triggered: Yes Profile-status: Sync Success
Local information:
---------------- Status: Commit Success
Error(s):
Peer information:
----------------
IP-address: 192.168.0.51
Sync-status: In sync Status: Commit Success
Error(s):
From N5K2s perspective, the configuration commands appear in the running config
just as if they had been entered manually in global configuration.
N5K2# show run interface
!Command: show running-config interface
!Time: Sun Mar 3 17:12:09 2013
version 5.1(3)N1(1a)
interface port-channel1
switchport mode trunk
spanning-tree port type network
vpc peer-link
7/26/2019 Nexus Technology Labs
44/313
interface port-channel101
switchport mode fex-fabric
fex associate 101
vpc 101
interface port-channel102
switchport mode fex-fabric
fex associate 102
vpc 102
Further verification shows that the configured features such as the vPC, FEX Fabric
Ports, VLANs, etc. are functional.
N5K2# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 1
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status: success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : secondary
Number of vPCs configured : 66
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 Po1 up 1,10
vPC status
----------------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------- -----------
101 Po101 up success success -
102 Po102 up success success -
102400 Eth101/1/1 up success success 10
7/26/2019 Nexus Technology Labs
45/313
102401 Eth101/1/2 up success success 1
102402 Eth101/1/3 down* Not Consistency Check Not -
Applicable Performed
102403 Eth101/1/4 down* Not Consistency Check Not -
Applicable Performed
102404 Eth101/1/5 down* Not Consistency Check Not -
Applicable Performed
102405 Eth101/1/6 down* Not Consistency Check Not -
Applicable Performed
102406 Eth101/1/7 down* Not Consistency Check Not -
Applicable Performed
102407 Eth101/1/8 down* Not Consistency Check Not -
Applicable Performed
102408 Eth101/1/9 down* Not Consistency Check Not -
N5K2# show spanning-tree vlan 10
VLAN0010
Spanning tree enabled protocol rstp
Root ID Priority 32778
Address 000d.eca2.edbc
Cost 1
Port 4096 (port-channel1)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)
Address 000d.eca4.743c
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Root FWD 1 128.4096 (vPC peer-link) Network P2p
Eth1/1 Desg FWD 4 128.129 Edge P2p
Eth1/2 Desg FWD 4 128.130 Edge P2p
Eth101/1/1 Desg FWD 1 128.1153 (vPC) Edge P2p
Eth102/1/1 Desg FWD 1 128.1281 (vPC) Edge P2p
7/26/2019 Nexus Technology Labs
46/313
Nexus Technology Labs - Virtual PortChannels (vPC)
vPC and HSRP
Task
This task applies only to INE Bootcamp partipants. Load
balancing is not included in the self-paced training curriculum at
this time.
Configure vPC between N7K1 and N7K2 as follows:
N7K1 and N7K2 are the vPC Peers.
Configure all available F1 ports between the vPC peers as Port-Channel 1,
and use this as the vPC Peer Link.
Use the mgmt0 ports for the Peer Keepalive Link.
Configure all available links from N7K1 and N7K2 to N5K1 in Port-Channel
51, and as vPC 51.
All port- channels should be trunks, STP Network Ports, and use LACP fornegotiation.
Configure VLAN assignments and the servers as follows:
Configure the link from N5K1 to Server 1 as an access port in VLAN 10.
Configure the link from N5K1 to Server 2 as an access port in VLAN 20.
Server 1 should use the IP address 10.0.0.1/24, and a default gateway of
10.0.0.254.
Server 2 should use the IP address 20.0.0.2/24, and a default gateway of
20.0.0.254.
Configure Inter-VLAN Routing and HSRP on N7K1 and N7K2 as follows:
Create interfaces VLAN 10 and VLAN 20 on N7K1 and N7K2, using the IP
address 10.0.0.X/24, where X is the last octet of the IP address on their
mgmt0 interfaces.
Configure HSRP group 10 for VLAN 10 on N7K1 and N7K2 using the virtual
address 10.0.0.254/24.
7/26/2019 Nexus Technology Labs
47/313
Configure HSRP group 20 for VLAN 20 on N7K1 and N7K2 using the virtual
address 20.0.0.254/24.
Set the Port Channel load balancing method on the Nexus switches to include the
source and destination layer 4 port numbers.
When complete, Server 1 and Server 2 should have IP reachability to each other,
and traffic between them should be load distributed across all links in the vPC.
Configuration
N5K1:
feature lacp
!
vlan 10,20
!
port-channel load-balance ethernet source-dest-port
!
interface Ethernet1/1
switchport mode access
switchport access vlan 10
speed 1000
!
interface Ethernet1/2
switchport mode access
switchport access vlan 20
speed 1000
interface Ethernet1/6-9
switchport mode trunk
spanning-tree port type network
channel-group 51 mode active
no shutdown
!
interface port-channel51
switchport mode trunk
spanning-tree port type network
N7K1-1:
feature vpc
feature lacp
!
vlan 10,20
!
port-channel load-balance src-dst ip-l4port-vlan
!
7/26/2019 Nexus Technology Labs
48/313
vpc domain 1
peer-keepalive destination 192.168.0.75
!
interface Ethernet2/1-2
switchport mode trunk
spanning-tree port type network
channel-group 1 mode active
no shutdown
!
interface port-channel1
switchport
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface Ethernet2/3-4
switchport mode trunk
spanning-tree port type network
channel-group 51 mode active
no shutdown
!
interface port-channel51
switchport
switchport mode trunk
spanning-tree port type network
vpc 51
!
interface Vlan10
no shutdown
ip address 10.0.0.71/24
hsrp 10
ip 10.0.0.254
!
interface Vlan20
no shutdown
ip address 20.0.0.71/24
hsrp 20
ip 20.0.0.254
N7K2-1:
feature vpc
feature lacp
!
vlan 10,20
!
port-channel load-balance src-dst ip-l4port-vlan
7/26/2019 Nexus Technology Labs
49/313
!
vpc domain 1
peer-keepalive destination 192.168.0.71
!
interface Ethernet2/1-2
switchport mode trunk
spanning-tree port type network
channel-group 1 mode active
no shutdown
!
interface port-channel1
switchport
switchport mode trunk
spanning-tree port type network
vpc peer-link
!
interface Ethernet2/5-6
switchport mode trunk
spanning-tree port type network
channel-group 51 mode active
no shutdown
!
interface port-channel51
switchport
switchport mode trunk
spanning-tree port type network
vpc 51
!
interface Vlan10
no shutdown
ip address 10.0.0.75/24
hsrp 10
ip 10.0.0.254
!
interface Vlan20
no shutdown
ip address 20.0.0.75/24
hsrp 20
ip 20.0.0.254
Verification
This scenario demonstrates how the forwarding pattern of vPC and HSRP combined
differs from that of just HSRP on its own. The results of this scenario would be
7/26/2019 Nexus Technology Labs
50/313
similar if either VRRP or GLBP were used, because all of the First Hop Redundancy
Protocols (FHRPs) have special behavior that interacts with vPC.
First, the layer 2 only switch N5K1 has access ports in VLANs 10 and 20, and a
trunking port channel that carries both VLANs. From N5K1s perspective, this port
channel logically connects to just one upstream switch, but in reality it is the two
physical vPC Peers, N7K1 and N7K2.
N5K1# show spanning-tree vlan 10,20
VLAN0010
Spanning tree enabled protocol rstp
Root ID Priority 4106
Address 68bd.abd7.6041
Cost 2
Port 4146 (port-channel51)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)
Address 000d.eca2.edbc
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------- Po51
Root FWD 1 128.4146 Network P2p Eth1/1
Desg FWD 4 128.129 P2p
VLAN0020
Spanning tree enabled protocol rstp
Root ID Priority 4116
Address 68bd.abd7.6041
Cost 2
Port 4146 (port-channel51)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 32788 (priority 32768 sys-id-ext 20)
Address 000d.eca2.edbc
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------- Po51
Root FWD 1 128.4146 Network P2p Eth1/2
Desg FWD 4 128.130 P2p
According to normal layer 2 switching vs. layer 3 routing logic, any hosts in VLAN 10
7/26/2019 Nexus Technology Labs
51/313
that want to talk to hosts in VLAN 20 must have their traffic switched up to the
default gateway and have the layer 2 header re-written with a new source and
destination MAC address, then switched to the final destination. In this case, there
are two default gateways for each VLAN 10 and 20, both N7K1 and N7K2 that share
the HSRP virtual IP address. In the output below we can see that N7K1 is the active
HSRP router for both groups.
N7K1-1# show hsrp
Vlan10 - Group 10
(HSRP-V1) (IPv4) Local state is Active
, priority 100 (Cfged 100)
Forwarding threshold(for vPC), lower: 1 upper: 100
Hellotime 3 sec, holdtime 10 sec
Next hello sent in 1.595000 sec(s)
Virtual IP address is 10.0.0.254 (Cfged)
Active router is local
Standby router is 10.0.0.75 , priority 100 expires in 5.957000 sec(s)
Authentication text "cisco"
Virtual mac address is 0000.0c07.ac0a (Default MAC)
4 state changes, last state change 00:48:39
IP redundancy name is hsrp-Vlan10-10 (default)
Vlan20 - Group 20
(HSRP-V1) (IPv4) Local state is Active
, priority 100 (Cfged 100)
Forwarding threshold(for vPC), lower: 1 upper: 100
Hellotime 3 sec, holdtime 10 sec
Next hello sent in 1.594000 sec(s)
Virtual IP address is 20.0.0.254 (Cfged)
Active router is local
Standby router is 20.0.0.75 , priority 100 expires in 6.264000 sec(s)
Authentication text "cisco"
Virtual mac address is 0000.0c07.ac14 (Default MAC)
2 state changes, last state change 02:33:39
IP redundancy name is hsrp-Vlan20-20 (default)
N7K2-1# show hsrp
Vlan10 - Group 10
(HSRP-V1) (IPv4) Local state is Standby
, priority 100 (Cfged 100)
Forwarding threshold(for vPC), lower: 1 upper: 100
Hellotime 3 sec, holdtime 10 sec
Next hello sent in 2.399000 sec(s)
Virtual IP address is 10.0.0.254 (Cfged)
Active router is 10.0.0.71, priority 100 expires in 9.872000 sec(s)
Standby router is local
Authentication text "cisco"
Virtual mac address is 0000.0c07.ac0a (Default MAC)
7/26/2019 Nexus Technology Labs
52/313
7 state changes, last state change 00:23:01
IP redundancy name is hsrp-Vlan10-10 (default)
Vlan20 - Group 20
(HSRP-V1) (IPv4) Local state is Standby
, priority 100 (Cfged 100)
Forwarding threshold(for vPC), lower: 1 upper: 100
Hellotime 3 sec, holdtime 10 sec
Next hello sent in 0.602000 sec(s)
Virtual IP address is 20.0.0.254 (Cfged)
Active router is 20.0.0.71, priority 100 expires in 5.152000 sec(s)
Standby router is local
Authentication text "cisco"
Virtual mac address is 0000.0c07.ac14 (Default MAC)
6 state changes, last state change 00:23:01
IP redundancy name is hsrp-Vlan20-20 (default)
The potential problem with this design is that if traffic is switched to N7K2, thestandby HSRP router, because of the Port Channel load balancing method of N5K1,
it would have to be sent to the active HSRP router, N7K1, to be routed. This means
that traffic would have to transit the vPC Peer Link, which is undesirable because
the aggregate of flows from vPC Member Ports would quickly overwhelm the vPC
Peer Link. To prevent this from being necessary, vPC changes the behavior of the
FHRPs so that the standby router can forward the same as the active router. The
result of this can be seen below.
Server 2 generates bulk TCP flows to Server 1 using the iPerf app. The aggregate
of flows nears 1Gbps.
When the access switch, N5K1, receives these flows, they have the destination
MAC address of the virtual HSRP address. This MAC address is reachable via the
7/26/2019 Nexus Technology Labs
53/313
port channel to the upstream 7K, and is then load balanced based on the layer 4
port information of the flows as configured.
N5K1# show port-channel summary
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
51 Po51(SU) Eth LACP Eth1/6(P) Eth1/7(P) Eth1/8(P)
Eth1/9(P)
N5K1# show interface e1/6 - 9 | include rate|Ethernet
Ethernet1/6 is up
Hardware: 1000/10000 Ethernet, address: 000d.eca2.ed8d (bia 000d.eca2.ed8d)
30 seconds input rate 2930112 bits/sec, 5292 packets/sec
30 seconds output rate 117990496 bits/sec, 13644 packets/sec
input rate 2.96 Mbps, 5.28 Kpps; output rate 111.59 Mbps
, 13.04 Kpps
Ethernet1/7 is up
Hardware: 1000/10000 Ethernet, address: 000d.eca2.ed8e (bia 000d.eca2.ed8e)
30 seconds input rate 232864888 bits/sec, 21953 packets/sec
30 seconds output rate 117809336 bits/sec, 13602 packets/sec
input rate 222.07 Mbps, 20.96 Kpps; output rate 113.44 Mbps
, 13.19 Kpps
Ethernet1/8 is up
Hardware: 1000/10000 Ethernet, address: 000d.eca2.ed8f (bia 000d.eca2.ed8f)
30 seconds input rate 451188568 bits/sec, 37074 packets/sec
30 seconds output rate 342359448 bits/sec, 29597 packets/sec
input rate 433.19 Mbps, 35.54 Kpps; output rate 326.19 Mbps
, 28.20 Kpps
Ethernet1/9 is up
Hardware: 1000/10000 Ethernet, address: 000d.eca2.ed90 (bia 000d.eca2.ed90)
30 seconds input rate 229147160 bits/sec, 21784 packets/sec
30 seconds output rate 337983576 bits/sec, 29262 packets/sec
input rate 220.18 Mbps, 20.96 Kpps; output rate 327.17 Mbps
, 28.28 Kpps
In the output above, we see that some traffic goes from N5K1 to N7K1, and some
7/26/2019 Nexus Technology Labs
54/313
from N5K1 to N7K2. Without the vPC modification to HSRP, this traffic shouldhave
to be switched from N7K2 to N7K1 before it can be routed, because N7K2 isnt the
active HSRP router. However, the interface counters of the vPC Peer Link, as seen
below, indicate that the flows are not switched in that direction, and instead N7K2 is
routing them itself even though it is HSRP standby.
N7K2-1# show interface port-channel 1 | include rate
30 seconds input rate 1560 bits/sec, 1 packets/sec
30 seconds output rate 1544 bits/sec, 1 packets/sec input rate 1.56 Kbps, 1 pps; output rate 1.54 Kbps
, 1 pps
This behavior can be further verified by disabling the uplinks from N5K1 to N7K1, as
shown below.
N5K1# show cdp neighbors
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
S - Switch, H - Host, I - IGMP, r - Repeater,
V - VoIP-Phone, D - Remotely-Managed-Device,
s - Supports-STP-Dispute
Device-ID Local Intrfce Hldtme Capability Platform Port ID
Nexus-MGMT-SW mgmt0 175 S I WS-C3550-48 Fas0/31
N5K2(FLC12480280) Eth1/3 140 S I s N5K-C5020P-BF Eth1/3
N5K2(FLC12480280) Eth1/4 140 S I s N5K-C5020P-BF Eth1/4
N5K2(FLC12480280) Eth1/5 140 S I s N5K-C5020P-BF Eth1/5
N7K1-1(JAF1510CMLQ) Eth1/6
170 R S s N7K-C7010 Eth2/3 N7K1-1(JAF1510CMLQ) Eth1/7
169 R S s N7K-C7010 Eth2/4
N7K2-1(TBM14311481) Eth1/8 167 R S s N7K-C7010 Eth2/5
N7K2-1(TBM14311481) Eth1/9 167 R S s N7K-C7010 Eth2/6
N5K1# config t
Enter configuration commands, one per line. End with CNTL/Z.N5K1(config)# int e1/6 7
N5K1(config-if-range)# shutdown
2013 Mar 5 21:54:43 N5K1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel51: Ethernet1/7 is down
2013 Mar 5 21:54:43 N5K1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/7 is down(Config change)
2013 Mar 5 21:54:43 N5K1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel51: Ethernet1/6 is down
2013 Mar 5 21:54:43 N5K1 %ETHPORT-5-IF_DOWN_CFG_CHANGE: Interface Ethernet1/6 is down(Config change)
2013 Mar 5 21:54:43 N5K1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet1/7 is down (Administratively dow
2013 Mar 5 21:54:43 N5K1 %ETHPORT-5-IF_DOWN_ADMIN_DOWN: Interface Ethernet1/6 is down (Administratively dow
With the links from N5K1 to N7K1 disabled, the only way for them to be switched
7/26/2019 Nexus Technology Labs
55/313
northbound is via N7K2, as shown below.
vN5K1# show interface e1/6 - 9 | include Ethernet1|rate
Ethernet1/6 is down (Administratively down)
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
Ethernet1/7 is down (Administratively down)
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
Ethernet1/8 is up
30 seconds input rate 231436152 bits/sec, 21853 packets/sec
30 seconds output rate 455582888 bits/sec, 42864 packets/sec
input rate 231.44 Mbps, 21.85 Kpps; output rate 455.58 Mbps
, 42.86 Kpps
Ethernet1/9 is up
30 seconds input rate 682938608 bits/sec, 63848 packets/sec
30 seconds output rate 458806416 bits/sec, 42838 packets/sec
input rate 682.94 Mbps, 63.85 Kpps; output rate 458.81 Mbps
, 42.84 Kpps
Because N7K1 still has the vPC Peer Link forwarding VLANs 10 and 20, it is still the
HSRP Active router.
N7K1-1# show hsrp | include state|Group
Vlan10 - Group 10
(HSRP-V1) (IPv4) Local state is Active
, priority 100 (Cfged 100)
4 state changes, last state change 02:02:24 Vlan20 - Group 20
(HSRP-V1) (IPv4) Local state is Active
, priority 100 (Cfged 100)
2 state changes, last state change 03:47:25
N7K2-1# show hsrp | include state|Group
Vlan10 - Group 10
(HSRP-V1) (IPv4) Local state is Standby
, priority 100 (Cfged 100)
7 state changes, last state change 01:36:48 Vlan20 - Group 20
(HSRP-V1) (IPv4) Local state is Standby
, priority 100 (Cfged 100)
6 state changes, last state change 01:36:48
If N7K1 is the router that is doing the layer 2 header re-write, the vPC Peer Link
7/26/2019 Nexus Technology Labs
56/313
should show 1Gbps input and output, which it does not according to the output
below.
N7K2-1# show interface port-channel 1 | include rate
30 seconds input rate 1576 bits/sec, 1 packets/sec
30 seconds output rate 1504 bits/sec, 1 packets/sec input rate 1.58 Kbps, 1 pps; output rate 1.50 Kbps
, 1 pps
Note that this behavior, in which both the active and standby HSRP routers are able
to forward traffic, is the default. There is no additional configuration needed to
accomplish this. As long as HSRP/VRRP/GLBP is configured in conjunction with
vPC, this behavior will be seen.
7/26/2019 Nexus Technology Labs
57/313
Nexus Technology Labs - Virtual PortChannels (vPC)
Back-to-Back vPC
Task
C