HP-5820 Switch - FCoE/Enet Configuration Guideqgt.qlogic.com/Hidden/support/Current Answer Attachments/HP_5820... · HP-5820 Switch - FCoE/Enet Configuration Guide Modified ... rather

HP-5820 Switch - FCoE/Enet Configuration Guide

Modified - 8 April 2013 Page 1

Author: Susan Grandys

Creation Date: Feb 15, 2013

Last Modified: March 01, 2013

Contents

1 Device Overview ................................................................................................................................... 2

1.1 Device Capabilities ........................................................................................................................ 2

1.2 Usefulness in FCoE testbeds ......................................................................................................... 2

2 To Access the Switch ............................................................................................................................. 3

3 Navigating the User Interface ............................................................................................................... 3

3.1 Web GUI ........................................................................................................................................ 3

3.2 Ethernet CLI ................................................................................................................................... 3

3.3 Using 5820 to connect serially to a Condor module ..................................................................... 3

3.4 Interface numbering on Ethernet ................................................................................................. 4

4 HP5820 configuration - basic Ethernet configuration .......................................................................... 5

4.1 Basic switch setup ......................................................................................................................... 5

4.2 Creating and removing VLANs and adding/removing ports to a VLAN ........................................ 5

4.3 Setting Ethernet port type for Ethernet ....................................................................................... 5

4.4 PT ports on Condor module .......................................................................................................... 7

4.5 Setting Ethernet port type to carry FCoE tagged packets............................................................. 7

5 Configure Priority Flow Control (PFC) settings ..................................................................................... 8

5.1 Example PFC configuration ........................................................................................................... 9

5.1.1 Network requirement ........................................................................................................... 9

5.1.2 Configuration procedure ....................................................................................................... 9

5.2 Link Aggregation Groups (LAG, link bundling, trunking, etc) ...................................................... 10

6 Layer 2 static Link Aggregation Groups (LAG) ..................................................................................... 11

6.1 configuration example ................................................................................................................ 11

6.1.1 Network requirements ........................................................................................................ 11

6.1.2 Configuration procedure ..................................................................................................... 11

7 Configure Device as FCoE Gateway .................................................................................................... 12



This is quick document on how to configure an HP-5820 switch to function as an Ethernet switch or FCoE

gateway.

NOTE: There are also handbooks for Condor to refer to in:

\\epfile01\System_Test\Condor\SWD-testing\Docs\Condor_Handbook-Condor-Setup.docx

And

\\epfile01\System_Test\Condor\SWD-testing\Docs\General_Condor_Handbook.docx

1 Device Overview

1.1 Device Capabilities

The HP 5820 switches can be configured as a Layer2 /Layer 3 switch with CEE/DCB capabilities.

Alternatively, the device can be configured as an FCoE gateway, providingthe FCF gateway but is in transparent mode (NPIV) – so it uses nameserver of remote FC switch.

To use the FCF, it uses a module (made by QLogic with our usual FCoE gateway software and CLIsh) inserted in one or both of the module slots on the HP 5820 Ethernet switch. Our module is called an LSW1FC4P0 card. It has 4 SPF Flex ports. These can be FC Transparent TF ports, or reconfigured as Ethernet PT ports.

1.2 Usefulness in FCoE testbeds This switch can be used in two distinctly different ways in an FCoE testbed.

As a full-featured FCoE gateway. Ethernet server traffic comes in the Ethernet ports, where it logs in to an FCF. The Fibre Channel ports can be connected directly to storage devices or to a fabric.

As a Layer 2 switch configured to pass both untagged and tagged Ethernet packets (including FCoE frames) in one Ethernet port and out another.



2 To Access the Switch Use digi ports to reach the console. We have had issues setting up telnet through the management port.

(Traffic flooding to hub, setting gateway, etc.)

Currently on Digi 10.20.148.100, digi ports 9 and 11.

So from a Linux telnet session:

telnet 10.20.148.100 7009 (or 7011)

3 Navigating the User Interface

3.1 Web GUI There is a Web GUI, but because of telnet difficulties, we use the CLI through the digi port.

3.2 Ethernet CLI • To do almost any configuration, you have to be in the “system-view” system-view

• To show anything, use “display” display vlan 10 display interface brief • To delete something use “undo” rather than the Cisco “no” undo <command> • To back up a level in the menu tree: quit • To see current configuration: display current-configuration • To save configuration changes: save

• To backspace through the digi console connection: <cntl> h

3.3 Using 5820 to connect serially to a Condor module The Condor module does not have an external serial port for users to connect through. Instead, use the

5820 to connect serially to the condor module

Left Condor = SubSlot1

Right Condor = SubSlot2



<H3C>mcms connect slot 1 system SubSlot1

or

<H3C>mcms connect slot 1 system SubSlot2

Use <ctrl-k> to exit

3.4 Interface numbering on Ethernet

[H3C-vlan10]display interface br

The brief information of interface(s) under route mode:

Link: ADM - administratively down; Stby - standby

Protocol: (s) - spoofing

Interface Link Protocol Main IP Description

NULL0 UP UP(s) --

Vlan1 UP UP 10.0.0.1

The brief information of interface(s) under bridge mode:

Link: ADM - administratively down; Stby - standby

Speed or Duplex: (a)/A - auto; H - half; F - full

Type: A - access; T - trunk; H - hybrid

Interface Link Speed Duplex Type PVID Description

GE1/0/15 DOWN auto A A 1 <-14 Ethernet ports on 5820 switch itself

GE1/0/16 DOWN auto A A 1



XGE1/0/1 UP 10G(a) F(a) T 1

XGE1/0/2 UP 10G(a) F(a) T 1

XGE1/0/3 UP 10G(a) F(a) T 1

XGE1/0/4 UP 10G(a) F(a) T 1

XGE1/0/5 UP 10G(a) F(a) T 1

XGE1/0/6 DOWN auto A T 1

XGE1/0/7 UP 10G(a) F(a) T 1

XGE1/0/8 UP 10G(a) F(a) T 1

XGE1/0/9 UP 10G(a) F(a) T 1

XGE1/0/10 UP 10G(a) F(a) T 1

XGE1/0/11 UP 10G(a) F(a) A 2003

XGE1/0/12 UP 10G(a) F(a) A 2002

XGE1/0/13 DOWN auto A A 2001

XGE1/0/14 DOWN auto A A 2000

XGE1/1/1 DOWN auto A T 1 <-4 Ports to QLogic module in slot 1

XGE1/1/2 UP 10G(a) F(a) T 1



XGE1/1/3 UP 10G(a) F(a) T 1

XGE1/1/4 UP 10G(a) F(a) T 1

XGE1/2/1 UP 10G(a) F(a) T 1 <-4 Ports to QLogic module in slot 2

XGE1/2/2 UP 10G(a) F(a) T 1

XGE1/2/3 UP 10G(a) F(a) T 1

XGE1/2/4 UP 10G(a) F(a) T 1

4 HP5820 configuration - basic Ethernet configuration

4.1 Basic switch setup ** See configuration example in last section of this document, there are a lot of undocumented

things that H3C just said to set.

4.2 Creating and removing VLANs and adding/removing ports to a VLAN

• To create separate vlans vlan 10

• To remove a vlan and all that is in it: system-view

undo vlan 10

• To add ports to the VLAN: system-view

vlan 10 port Ten-GigabitEthernet1/0/1 (or for a range:) port Ten-GigabitEthernet1/0/1 to TenGigabitEthernet1/0/9

• To remove ports in the VLAN: system-view

vlan 10 undo port Ten-GigabitEthernet1/0/1

4.3 Setting Ethernet port type for Ethernet

• For regular Ethernet traffic, set port type to “access” mode and put in a VLAN.

system-view interface Ten-GigabitEthernet1/0/14

port link-mode bridge <- must be bridge, rather than router for L2 port access vlan 2000 <- access mode for pure Enet

priority-flow-control auto priority-flow-control no-drop dot1p 3

stp disable



qos wrr 7 group sp qos wrr 6 group sp



qos wrr 1 group 1 byte-count 15 qos wrr 0 group 1 byte-count 15

qos trust dot1p lldp tlv-enable dot1-tlv dcbx

• If you need to set the speed on the port: system-view interface Ten-GigabitEthernet1/0/1 speed 10000 (or 1000, or auto)

• For FCoE or other trunk traffic, it would look more like this:

interface Ten-GigabitEthernet1/0/1 port link-mode bridge port link-type trunk port trunk permit vlan all priority-flow-control auto priority-flow-control no-drop dot1p 3 dot1x stp disable qos apply policy fcoe outbound qos wrr 7 group sp qos wrr 6 group sp qos wrr 5 group sp qos wrr 4 group sp qos wrr 3 group sp qos wrr 2 group sp qos wrr 1 group 1 byte-count 15 qos wrr 0 group 1 byte-count 15 qos trust dot1p lldp tlv-enable dot1-tlv dcbx



4.4 PT ports on Condor module • Go to IP of Condor module and set with QLogic CLIsh

H3C #> admin start

H3C (admin) #> config edit

The config named interop_fcf is being edited.

H3C (admin-config) #> set config port 0

A list of attributes with formatting and current values will follow.

Enter a new value or simply press the ENTER key to accept the current value.

If you wish to terminate this process before reaching the end of the list

press 'q' or 'Q' and the ENTER key to do so.

Configuring Port Number: 0

------------------------

AdminState (1=Online, 2=Offline, 3=Diagnostics, 4=Down) [Online]

PortType (PT,TF) [TF ] PT -Set to PT port

LinkSpeed (1=1Gb/s, 2=2Gb/s, 4=4Gb/s, 8=8Gb/s, A=Auto) [Auto ]

SymPortName (string, max=32 chars) [Port0 ]

Finished configuring attributes.

This configuration must be saved (see config save command) and

activated (see config activate command) before it can take effect.

To discard this configuration use the config cancel command.

4.5 Setting Ethernet port type to carry FCoE tagged packets

You can set the PFC at the interface level, and it would have to be trunked to all VLANs’ as in the previous section.

system-view

interface Ten-GigabitEthernet1/0/1

priority-flow-control ?

auto Automatically negotiate to enable PFC function

enable Enable PFC function

no-drop Enable lossless transmission for packets with the specified 802.1p

priorities



priority-flow-control no-drop ?

dot1p 802.1p priority

priority-flow-control no-drop dot1p ?

STRING List of 802.1p priorities ranged from 0 to 7, example: 0,2-3,5,7

Set the qos trust - by default, Ethernet interfaces do not trust the priorities of incoming packets: qos trust dot1p

5 Configure Priority Flow Control (PFC) settings

PFC performs flow control based on 802.1p priorities. With PFC enabled, an interface requires its peer to suspend sending packets with certain 802.1p priorities when congestion occurs. By decreasing the transmission rate, PFC helps avoid packet loss. You can enable PFC for certain 802.1p priorities at the two ends of a link. When network congestion occurs, the local device checks the PFC status for the 802.1p priority carried in each arriving packet.

If PFC is enabled for the 802.1p priority, the local device accepts the packet and sends a PFC pause frame to the peer. The peer stops sending packets carrying this 802.1p priority for an interval as specified in the PFC pause frame. This process repeats until the congestion is removed.

If PFC is disabled for the 802.1p priority, the local port drops the packet. Each local precedence value corresponds to a queue. The 802.1p-to-local priority mapping is as shown in Table 1. Modify the 802.1p-to-local priority mapping table with the qos map-table dot1p-lp and import import-value-list export export-value commands. For more information about the two commands, see the ACL and QoS Command Reference.

Table 1 The default 802.1p-to-local priority mapping table 802.1p priority Local precedence value

0 2 1 0 2 1 3 3 4 4 5 5 6 6 7 7

You can set the PFC at the interface level:

system-view


priority-flow-control ?

auto Automatically negotiate to enable PFC function



enable Enable PFC function

no-drop Enable lossless transmission for packets with the specified 802.1p

priorities

priority-flow-control no-drop ?

dot1p 802.1p priority

priority-flow-control no-drop dot1p ?

STRING List of 802.1p priorities ranged from 0 to 7, example: 0,2-3,5,7

Set the qos trust - by default, Ethernet interfaces do not trust the priorities of incoming packets: qos trust dot1p

5.1 Example PFC configuration

5.1.1 Network requirement

XGE1/0/1 XGE1/0/2 XGE1/0/1

Switch B Switch A DC Server Switch A connects to the FCoE HBA of the DC server through Ten-GigabitEthernet 1/0/1, and connects to Switch B through Ten-GigabitEthernet 1/0/2. The 802.1p priority of FCoE packets is 3. Configure the switches to guarantee no packet loss for FCoE packets forwarded among Switch B, Switch A, and DC server. Suppose Switch A and Switch B support PFC, and Switch A and DC server support DCBX. You should also configure DCBX on the port that connects Switch A to the FCoE HBA. This example describes configuring DCBX to advertise the PFC data.

5.1.2 Configuration procedure

1. Configure Switch A

Configure DCBX to advertise the PFC data. # Enable LLDP globally. <SwitchA> system-view

[SwitchA] lldp enable

# Enable LLDP and DCBX TLV advertising on interface Ten-GigabitEthernet 1/0/1. [SwitchA] interface ten-gigabitethernet 1/0/1

[SwitchA-Ten-GigabitEthernet1/0/1] lldp enable



[SwitchA-Ten-GigabitEthernet1/0/1] lldp tlv-enable dot1-tlv dcbx

# Enable PFC in auto mode on interface Ten-GigabitEthernet 1/0/1, and enable PFC for 802.1p priority value 3 on the interface. In addition to enabling PFC on the local end, this configuration also enables synchronizing the PFC data on the local end to the FCoE HBA of the remote end. [SwitchA-Ten-GigabitEthernet1/0/1] priority-flow-control auto

[SwitchA-Ten-GigabitEthernet1/0/1] priority-flow-control no-drop dot1p 3

[SwitchA-Ten-GigabitEthernet1/0/1] qos trust dot1p

[SwitchA-Ten-GigabitEthernet1/0/1] quit

Enable PFC on port Ten-GigabitEthernet 1/0/2, which the FCoE packets pass through. # Enable PFC on interface Ten-GigabitEthernet 1/0/2, and enable PFC for packets carrying 802.1p priority value 3 on the interface. [SwitchA] interface ten-gigabitethernet 1/0/2

[SwitchA-Ten-GigabitEthernet1/0/2] priority-flow-control enable

[SwitchA-Ten-GigabitEthernet1/0/2] priority-flow-control no-drop dot1p 3

[SwitchA-Ten-GigabitEthernet1/0/2] qos trust dot1p

[SwitchA-Ten-GigabitEthernet1/0/2] quit

2. Configure Switch B # Enable PFC on interface Ten-GigabitEthernet 1/0/1, which the FCoE packets pass through, and enable PFC for packets carrying 802.1p priority value 3 on the interface. <SwitchB> system-view

[SwitchB] interface ten-gigabitethernet 1/0/1

[SwitchB-Ten-GigabitEthernet1/0/1] priority-flow-control enable

[SwitchB-Ten-GigabitEthernet1/0/1] priority-flow-control no-drop dot1p 3

[SwitchB-Ten-GigabitEthernet1/0/1] qos trust dot1p

[SwitchB-Ten-GigabitEthernet1/0/1] quit

5.2 Link Aggregation Groups (LAG, link bundling, trunking, etc) **if these are needed for anything

Assign interfaces to bridge-aggregation group

system-view

interface bridge-aggregation #

quit


port link-aggregation group #

(repeat last 2 steps to assign multiple L2 interfaces to aggregation group)

quit

Assign load sharing criteria (if needed) The default global aggregation load sharing criteria are ingress port, source MAC address and destination MAC address for Layer 2 packet types

system-view



link-aggregation load-sharing mode ?

destination-ip Destination IP address

destination-mac Destination MAC address

destination-port Socket destination port

ingress-port Ingress port

local-first Local forwarding first

source-ip Source IP address

source-mac Source MAC address

source-port Socket source port

6 Layer 2 static Link Aggregation Groups (LAG)

6.1 configuration example

6.1.1 Network requirements

Device A and Device B are connected through their respective Layer 2 Ethernet interfaces GigabitEthernet 1/0/1 through GigabitEthernet 1/0/3.

Configure a Layer 2 static link aggregation group on Device A and Device B. Then enable VLAN 10 at one end of the aggregate link to communicate with VLAN 10 at the other end, and VLAN 20 at one end to communicate with VLAN 20 at the other end.

Enable traffic to be load-shared across aggregation group member ports based on source and destination MAC addresses.

6.1.2 Configuration procedure 1. Configure Device A

# Create VLAN 10, and assign port GigabitEthernet 1/0/4 to VLAN 10. <DeviceA> system-view

[DeviceA] vlan 10

[DeviceA-vlan10] port gigabitethernet 1/0/4

[DeviceA-vlan10] quit

# Create VLAN 20, and assign port GigabitEthernet 1/0/5 to VLAN 20.

[DeviceA] vlan 20

[DeviceA-vlan20] port gigabitethernet 1/0/5

[DeviceA-vlan20] quit

# Create Layer 2 aggregate interface Bridge-Aggregation 1. [DeviceA] interface bridge-aggregation 1

[DeviceA-Bridge-Aggregation1] quit

# Assign ports GigabitEthernet 1/0/1 through GigabitEthernet 1/0/3 to link aggregation group 1. [DeviceA] interface gigabitethernet 1/0/1

[DeviceA-GigabitEthernet1/0/1] port link-aggregation group 1

[DeviceA-GigabitEthernet1/0/1] quit

[DeviceA] interface gigabitethernet 1/0/2





[DeviceA] interface gigabitethernet 1/0/3



# Configure Layer 2 aggregate interface Bridge-Aggregation 1 as a trunk port and assign it to VLANs 10 and 20. This configuration automatically propagates to all member ports in link aggregation group 1. [DeviceA] interface bridge-aggregation 1

[DeviceA-Bridge-Aggregation1] port link-type trunk

[DeviceA-Bridge-Aggregation1] port trunk permit vlan 10 20

Please wait... Done.

Configuring GigabitEthernet1/0/1... Done.



[DeviceA-Bridge-Aggregation1] quit

# Configure the device to use the source and destination MAC addresses of packets as the global linkaggregation load sharing criteria. [DeviceA] link-aggregation load-sharing mode source-mac destination-mac

2. Configure Device B Configure Device B as you configure Device A. 3. Verify the configurations

# Display the summary information about all aggregation groups on Device A. [DeviceA] display link-aggregation summary

Aggregation Interface Type:

BAGG -- Bridge-Aggregation, RAGG -- Route-Aggregation

Aggregation Mode: S -- Static, D -- Dynamic

Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing

Actor System ID: 0x8000, 000f-e2ff-0001

AGG AGG Partner ID Select Unselect Share

Interface Mode Ports Ports Type

-------------------------------------------------------------------------------

BAGG1 S none 3 0 Shar

The output shows that link aggregation group 1 is a load shared Layer 2 static aggregation group and it contains three selected ports. # Display the global link-aggregation load sharing criteria on Device A. [DeviceA] display link-aggregation load-sharing mode

Link-Aggregation Load-Sharing Mode:

destination-mac address, source-mac address

The output shows that all link aggregation groups created on the device perform load sharing based on source and destination MAC addresses.

7 Configure Device as FCoE Gateway On QLogic module, set ports to TF and put in VLAN’s and FCF’s as needed (default is all ports in VLAN

1002.)



Nowhere does the 5820 Ethernet documentation talk about FCoE setup. Best guess is to set it up as they

have instructed us in the past. Here is my best guess:

[H3C]display current-configuration

#

version 5.20, Release 1211P02

#

sysname H3C

#

irf mac-address persistent timer

irf auto-update enable

undo irf link-delay

#

domain default enable system

#

telnet server enable

#

acl number 4000

rule 0 permit type 8906 ffff

rule 1 permit type 8914 ffff

#

vlan 1

#

vlan 2 to 9

#

# vlan 10

name FCoE vlan

#

vlan 11 to 4094

#

radius scheme system

server-type extended

primary authentication 127.0.0.1 1645

primary accounting 127.0.0.1 1646

user-name-format without-domain

#

domain system

access-limit disable

state active

idle-cut disable



self-service-url disable

#

traffic classifier fcoe operator or

if-match acl 4000

traffic classifier emulex operator or

if-match acl 4000

#

traffic behavior fcoe

remark dot1p 3

traffic behavior emulex

remark dot1p 3

#

qos policy fcoe

classifier fcoe behavior fcoe mode dcbx

qos policy emulex

classifier emulex behavior emulex mode dcbx

#

qos map-table dot1p-lp

import 0 export 0

import 2 export 0

import 3 export 1

import 4 export 0

import 5 export 0

import 6 export 0

#

user-group system

#

interface NULL0

#

interface Vlan-interface1

ip address 10.0.0.1 255.0.0.0

#

interface Vlan-interface10

#

interface GigabitEthernet1/0/15

port link-mode bridge

#



#





#



#

interface Ten-GigabitEthernet1/0/1 < - example of ports for FCoE traffic, trunked and “vlan all”


port link-type trunk

port trunk permit vlan all

priority-flow-control auto

priority-flow-control no-drop dot1p 3

stp disable

qos apply policy fcoe outbound

qos wrr 7 group sp

qos wrr 6 group sp

qos wrr 5 group sp

qos wrr 4 group sp

qos wrr 3 group sp

qos wrr 2 group sp

qos wrr 1 group 1 byte-count 15


qos trust dot1p

lldp tlv-enable dot1-tlv dcbx

#







stp disable

qos apply policy fcoe inbound


qos wrr 7 group sp

qos wrr 6 group sp

qos wrr 5 group sp

qos wrr 4 group sp

qos wrr 3 group sp

qos wrr 2 group sp





qos trust dot1p


#

.

. <- all ports for FCoE to FC seem to be the same

.

interface Ten-GigabitEthernet1/0/13 <- These may be to enet or PT ports, type “access”, and one VLAN


port access vlan 2001



stp disable


qos wrr 7 group sp

qos wrr 6 group sp

qos wrr 5 group sp

qos wrr 4 group sp

qos wrr 3 group sp

qos wrr 2 group sp



qos trust dot1p


#



port access vlan 2000



stp disable


qos wrr 7 group sp

qos wrr 6 group sp

qos wrr 5 group sp

qos wrr 4 group sp

qos wrr 3 group sp

qos wrr 2 group sp



qos trust dot1p lldp tlv-enable dot1-tlv dcbx

#



#

interface Ten-GigabitEthernet1/1/1 <- these are the ports going to QLogic module 1






qos trust dot1p


#







qos trust dot1p


#







qos trust dot1p


#







qos trust dot1p


#

.

. <- Ditto for ports to QLogic module in slot 2

.

#



snmp-agent

snmp-agent local-engineid 800063A2030023893F860A

snmp-agent log all

snmp-agent sys-info version v3

#

load xml-configuration

#

load tr069-configuration

#

user-interface aux 0

user-interface vty 0

authentication-mode none

user privilege level 2

set authentication password simple password

history-command max-size 20

idle-timeout 30 0

screen-length 30

protocol inbound telnet

user-interface vty 1 4

authentication-mode none

idle-timeout 30 0

user-interface vty 5 15

idle-timeout 30 0

#

return

Documents

HP-5820 Switch - FCoE/Enet Configuration Guideqgt.qlogic.com/Hidden/support/Current Answer Attachments/HP_5820... · HP-5820 Switch - FCoE/Enet Configuration Guide Modified ... rather