PRIMERGY BX600 10 GbE Switch Blade 10/2 and 10 …manuals.ts.fujitsu.com/file/3632/bx600-10gbe-wp-en.pdfEdition January 2009 PRIMERGY BX600 10 GbE Switch Blade 10/2 and 10 GbE LAN

Edition January 2009

PRIMERGY BX600 10 GbE Switch Blade 10/2 and 10 GbE LAN I/O Module PCIe Pages 108

Contents

1 Introduction 3 2 Switch Connectivity 5

2.1 Interfaces 5 2.2 CLI Help Functions and Keyboard Shortcuts 6 2.3 Setting the IP Address of the Management Port 6 2.4 Resetting the Configuration 6 2.5 Interface Overview 7 2.6 Saving the Configuration 7 2.7 Assigning the IP Address for the 10 GbE Switch Blade 7 2.8 Enabling/Disabling HTTPS 7 2.9 Enabling/Disabling SSH 7 2.10 Upgrading the Firmware 7 2.11 Layer 1 Initialization 8

2.11.1 Auto Negotiation Up-Downlink 8 2.11.2 Auto Negotiation Management Port 8

2.12 Port Aggregation 9 2.12.1 Introduction 9 2.12.2 Recommended Solution LACP or Static 10 2.12.3 Recommended Solution Load Balancing 10 2.12.4 Configuration 10

2.13 VLANs and Trunks 13 2.13.1 Introduction 13 2.13.2 Recommended Solution 13 2.13.3 Configuration 14

2.14 Spanning Tree Protocol 17 2.14.1 Introduction 17 2.14.2 Recommended Solution 25 2.14.3 Configuration with VLAN Trunks 26 2.14.4 Configuration without VLAN Trunks 32

2.15 Redundant Configuration with Two PRIMERGY BX600 10 GbE Switch Blades 10/2 39 2.15.1 Introduction 39 2.15.2 STP and VLD 41 2.15.3 Recommended Solution 41

2.16 Configuring the Access Ports of the Switches 42 2.16.1 Overview 42

2.17 Recommended Solution 42 2.17.1 Configuration 42

3 Basic Multicast Services 43 3.1 Introduction 43 3.2 Recommended Solution 43 3.3 Configuration 44

4 Switch Management 47 4.1 Logging and Synchronization 47

4.1.1 Introduction 47 4.1.2 Recommended Solution 47 4.1.3 Configuring syslog and NTP 48

4.2 SNMP 49 4.2.1 Introduction 49

White Paper ⏐ Version: January 2009 ⏐"PRIMERGY BX600 10 GbE Switch Blade 10/2 and 10 GbE LAN I/O Module PCIe"

10GbE_wp_en.doc © Fujitsu Siemens Computers 2008 All rights reserved

Page: 2 of 108

4.2.2 Recommended Solution 49 4.2.3 Configuring SNMP 49

4.3 Remote Console Access 51 4.3.1 Introduction 51 4.3.2 Recommended Solution 51 4.3.3 Configuring SSH 51

4.4 Integration into Radius and TACACS+ 53 4.4.1 Recommended Solution 53 4.4.2 Configuring RADIUS 53 4.4.3 Configuring TACACS 58

4.5 Port Monitoring / Mirroring 64 4.5.1 Introduction 64 4.5.2 Configuring Port Monitoring/Mirroring 64

5 10 GbE I/O Module PCIe 66 5.1 Installing the Drivers 66

5.1.1 Installing the Drivers and Software under Microsoft Windows Server 2003 66 5.1.2 Installing the Drivers and Software under Microsoft Windows Server 2008 70 5.1.3 Installing under SuSE Linux 73 5.1.4 Installing under Red Hat Linux 76

5.2 Configuring iSCSI HBA 76 5.2.1 Overview of iSCSI Storage Networks 76 5.2.2 Integrating Additional Storage Media via iSCSI 77 5.2.3 iSCSI Booting 81

5.3 Installing the Operating System on an iSCSI Target 88 5.3.1 Installing MS Windows with ServerStart 88 5.3.2 Native MS Windows Installation 92 5.3.3 Installing Linux with ServerStart 92 5.3.4 Native Linux Installation 93

5.4 Environment-Specific Settings 99 5.4.1 Settings for Multi-Path Configurations under Microsoft Windows and Linux 99 5.4.2 Additional Settings for Microsoft Windows 100 5.4.3 Settings for Targets from Different Manufacturers 100

Appendix: Network Driver Performance Tuning 101 Appendix: Known Issues 107



Page: 3 of 108

1 Introduction Most data center networks today run with switches from a single vendor. Although most of the protocols used are standardized, there are a number of proprietary ones – especially redundancy and management protocols. Other features may be so individual that interoperability is possible but not simple.

This white paper will guide you through integrating the 10GbE Switch Blade in your network, especially in Cisco networks.

A number of major aspects that are common to most data center networks are covered and have been tested in Fujitsu Siemens Computers laboratories. All the features of Cisco switches mentioned in this paper have been tested with Catalyst 6500 series switches.

The following Cisco IOS software was used for the integration tests:

Catalyst 6500 IOS 12.2(17d)SEE1 Advanced IP Services.

6500-1#show version Cisco Internetwork Operating System Software IOS (tm) s72033_rp Software (s72033_rp-PK9SV-M), Version 12.2(17d)SXB11a, RELEASE SOFTWARE (fc1) ………… Compiled Thu 13-Apr-06 04:50 by kehsiao Image text-base: 0x40020FBC, data-base: 0x41F18000 ROM: System Bootstrap, Version 12.2(17r)S4, RELEASE SOFTWARE (fc1) BOOTLDR: s72033_rp Software (s72033_rp-PK9SV-M), Version 12.2(17d)SXB11a, RELEASE SOFTWARE (fc1) System image file is "sup-bootflash:s72033-pk9sv-mz.122-17d.SXB11a.bin" ……… cisco WS-C6506-E (R7000) processor (revision 1.0) with 458752K/65536K bytes of memory. Processor board ID SAL1020MXUM SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache Last reset from power-on X.25 software, Version 3.0.0. Bridging software. 8 Ethernet/IEEE 802.3 interface(s) 4 Virtual Ethernet/IEEE 802.3 interface(s) 50 Gigabit Ethernet/IEEE 802.3 interface(s) 8 Ten Gigabit Ethernet/IEEE 802.3 interface(s) 1917K bytes of non-volatile configuration memory. 8192K bytes of packet buffer memory. 65536K bytes of Flash internal SIMM (Sector size 512K). Configuration register is 0x2102

Catalyst 4948-10GE IOS 12.2(25)SG Advanced IP Services

C4948-10GE2#show version Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500-ENTSERVICESK9-M), Version 12.2(25)SG, RELEASE SOFTWARE (fc2) . Compiled Wed 17-Aug-05 17:15 by alnguyen Image text-base: 0x10000000, data-base: 0x11642900 ROM: 12.2(25r)EWA Pod Revision 0, Force Revision 31, Tie Revision 19 …… cisco WS-C4948-10GE (MPC8540) processor (revision 5) with 262144K bytes of memory. Processor board ID FOX100511UK MPC8540 CPU at 667Mhz, Fixed Module Last reset from Reload 3 Virtual Ethernet interfaces 48 Gigabit Ethernet interfaces 2 Ten Gigabit Ethernet interfaces 511K bytes of non-volatile configuration memory. Configuration register is 0x2101



Page: 4 of 108

Overview

Figure 1: 10 GbE Switch Blade • 2 x 10Gbit (10GBASE) via optional X2

transceivers: • Copper, X2-CX4 • FO, X2-fiber

Figure 2: 10 GbE I/O Module • 2 x 10Gbit/s Ethernet ports to the midplane • PCIe interface to the server blade system board

The 10 GbE Switch Blade 10/2 is an integrated 10-Gigabit Ethernet switch for use in the PRIMERGY BX 600 S3 Blade Server basic unit. You can install up to two 10 GbE Switch Blades in the BX600 S3 basic unit. It offers 2 x 10Gbit uplink ports (external connections using suitable X2 modules) and 10 x 10Gbit downlink ports to the midplane of the basic unit for connecting the server blades that have a 10GbE LAN I/O Module PCIe installed. This I/O module offers two channels, each with 10-Gigabit Ethernet. It is implemented as a daughter card which is plugged directly into the system board.

The 10 GbE Switch Blade and the 10GbE LAN I/O Module must be used in combination, with the first 10GbE LAN I/O Module requiring at least one 10 GbE Switch Blade. A 10GbE Switch Blade in turn supports ten daughter cards, each on their first channel.

The 10 GbE Switch Blade also has one internal V24 port and one external 1Gbit port for management purposes.

The 10 GbE Switch Blade is designed to be used in a redundant configuration within a BX600 Basic Unit.

Product names and abbreviations

The following table gives an overview of the product names and abbreviations used in this document.

Product name Short form (in continuous text) Abbreviation

PRIMERGY BX600 10 GbE Switch Blade 10/2 10 GbE Switch Blade BX (derived from BladeXchange)

PRIMERGY BX600 10 GbE I/O Module PCIe 10 GbE I/O Module BX (derived from BladeEngine)

PRIMERGY BX600 S3 Basic Unit BX600 Basic Unit

Notational conventions

Typewriter font Output and comments on the CLI

Typewriter font bold

Input on the CLI

!

CAUTION! This symbol indicates risks that may lead to data loss and device damage.



Page: 5 of 108

2 Switch Connectivity 2.1 Interfaces As its name implies, the 10 GbE Switch Blade 10/2 has ten 10Gbit/s downlink ports and two 10Gbit/s uplink ports. In addition, the 10 GbE Switch Blade offers an external 1 Gbit/s port, which is only used for management tasks, not for data transfer. A and B are the two ports of the 10GbE LAN I/O Module, also referred to as the BladeEngine (BE). This is a daughter card which is plugged directly into the server blade system board (more information later).

Ser

ver B

lade

1

BE0

BA

Ser

ver B

lade

2

BE0

BA

Ser

ver B

lade

3

BE0

BA

Ser

ver B

lade

4

BE0

BA

Ser

ver B

lade

5

BE0

BA

Ser

ver B

lade

6

BE0

BA

Ser

ver B

lade

7

BE0

BA

Ser

ver B

lade

8

BE0

BA

Ser

ver B

lade

9

BE0

BA

Ser

ver B

lade

10

BE0

BA

BXNET4

BXNET3

10 x 10 GbitDownlink= ap1 - ap10

10 x 10 GbitDownlink= ap1 - ap10

2 x 10 GbitUplink= ap11/ap12

2 x 10 GbitUplink= ap11/ap12

10/100/1000Management Port= eth0

10/100/1000Management Port= eth0

Figure 3: Port overview – bold font indicates the short form of the port name used internally by the 10 GbE Switch Blade 10/2



Page: 6 of 108

2.2 CLI Help Functions and Keyboard Shortcuts ! 10 GbE Switch Blade help This CLI provides advanced help features. When you need help, anytime at the command line please press ‘?’. If nothing matches, the help list will be empty and you must backup until entering a ‘?’ shows the available options. Two styles of help are provided: 1. Full help is available when you are ready to enter a Command argument (e.g. ‘show ?’) and describes each possible argument. 2. Partial help is provided when an abbreviated argument is entered and you want to know what arguments match the input (e.g. ‘show ve ?’.)

Keyboard shortcut

<Tab> – Complete command

Ctrl-B Cursor back one character

Ctrl-F Cursor forward one character

Ctrl-A Cursor to beginning of line

Ctrl-E Cursor to end of line

ESC B Back one word

ESC F Forward one word

Ctrl-P Repeat last command line

Ctrl-N Repeat next command line

Ctrl-U Delete from beginning of line up to cursor position

Ctrl-K Delete from cursor position up to end of line

Ctrl-D Delete one character

Ctrl-W Delete one word

Ctrl-l Overwrite one command line

Ctrl-R Overwrite one command line

Do Execute an exec mode command in configure mode

2.3 Setting the IP Address of the Management Port First, the management port of the 10 GbE Switch Blade requires an IP address, which you can configure via the management blade of the BX600 Basic Unit, see page 7.

Alternatively, you can configure the IP address directly via the switch console redirection of the management blade.

How to configure the management blade, the switch blade and the console redirection is described in the manual "PRIMERGY BX Blade Server Systems RemoteView Management Blade“.

2.4 Resetting the Configuration ! 10 GbE Switch Blade ! Reset / delete configuration reset config This command resets the switch to its default configuration. Continue? (y/n):y Please reboot the switch for the default configuration to take effect. reboot reboot system? (y/n): y Broadcast message from root (pts/1) Mon Nov 12 17:34:47 2007... The system is going down for reboot NOW !!



Page: 7 of 108

2.5 Interface Overview ! 10 GbE Switch Blade Show interface …. Interface ap11 is UP Hardware is Loopback index 1 metric 1 mtu ...

Interfaces are ap1-ap10 for the downlinks and ap11-ap12 for the uplinks.

2.6 Saving the Configuration write

2.7 Assigning the IP Address for the 10 GbE Switch Blade ! 10 GbE Switch Blade Enable Config interface eth0 ip address 192.168.1.4 255.255.255.0 ip default-gateway 10.20.1.1

Checking the IP settings ! 10 GbE Switch Blade show ip interface Interface IP-Address Status Protocol eth0 192.168.1.4 up up

2.8 Enabling/Disabling HTTPS Enabling secure access via HTTPS ! 10 GbE Switch Blade ! Setup for 10 GbE Switch Blade HTTPS Server ! Recommendation: Only activate HTTPS ! Standard port 443 does not have to be changed ip http secure-server ! If required, deactivate HTTP No ip http server

2.9 Enabling/Disabling SSH ! 10 GbE Switch Blade ip ssh enable No ip ssh enable

2.10 Upgrading the Firmware ! 10 GbE Switch Blade ! A TFTP server is required, e.g. on the SeverStart DVD ! DVD:\PROGRAMS\GENERAL\TFTP\PumpKIN.exe ! ! fwupgrade <tftp server ip address> <image name> fwupgrade 10.20.33.200 l2-f-1763.ufi reboot reboot system? (y/n): y



Page: 8 of 108

2.11 Layer 1 Initialization The 10 GbE Switch Blade is equipped with 2 x 10-Gigabit Ethernet ports for uplink, which are implemented as specified in the 10G-Base-T standard.

2.11.1 Auto Negotiation Up-Downlink

10G-Base-T only allows 10-Gigabit Ethernet and Full Duplex. Connections to 1000/100/10 ports are not possible.

10 GbE Switch Blade (uplink and downlink)

Fix

Hal

f Dup

lex

10

Fix

Hal

f Dup

lex

100

Fix

Full

Dup

lex

10

Fix

Full

Dup

lex

1000

Full

Dup

lex1

0 G

Fix Half Duplex 10 N/A N/A N/A N/A N/A

Fix Half Duplex 100 N/A N/A N/A N/A N/A

Fix Full Duplex 10 N/A N/A N/A N/A N/A



Cisco Switch

Full Duplex 10 G N/A N/A N/A N/A OK Table 1: Speed and duplex settings

2.11.2 Auto Negotiation Management Port

The management port can be run with different data rates 100/1000 Mbit/s and different duplex settings

Figure 4: Management port (arrow)

For the management port, you can use the Layer 1 default setting.



Page: 9 of 108

2.12 Port Aggregation 2.12.1 Introduction

Sometimes you need more than 1 x 10 Gbit when connecting a 10 GbE Switch Blade in a data center. In this case two links are set up to form a port-channel, also known as a Fast Ethernet Channel (FEC) or Gigabit Ethernet Channel (GEC) in Cisco networks. Figure 5 shows a typical uplink configuration for a 10 GbE Switch Blade: Each port-channel is formed of two links running with 10 Gbit in full duplex mode. The redundancy mechanisms between these links will be discussed later. In principle, port-channels can be configured statically or using a port aggregation protocol. Cisco supports LACP as specified in 802.3ad and their proprietary PagP, while the 10 GbE Switch Blade supports LACP as specified in 802.3ad. Using static or LACP dynamic configuration, you can form up to 2 x 10GE links between the 10 GbE Switch Blade and one other switch.

10 GbE Switch Blade 10/2

ap11 ap12 PRIMERGY BX600 S3

TenGi3/4 TenGi4/4

Cisco A

SA1

Po1

02

01

Figure 5: Uplink configuration for 10 GbE Switch Blade

Table 2 shows the possible combinations of port-channel settings between 10 GbE Switch Blade and Cisco switches. The combinations marked amber are very risky and would lead to network loops.

PRIMERGY BX600 10 GbE Switch Blade 10/2

No Channel

LACP Active

LACP Passive Static

No channel OK NOK NOK NOK

Active NOK OK OK NOK Passive NOK OK NOK NOK

Cisco

On NOK NOK NOK OK Table 2: Possible port-channel configurations

So-called “split channels”, where one channel from one switch is terminated at two other switches, are supported neither by the 10 GbE Switch Blade nor by Cisco switches.



Page: 10 of 108

2.12.2 Recommended Solution LACP or Static

Although Cisco switches and the 10 GbE Switch Blade both support LACP, and although this feature has been tested to be compatible between these devices, we recommend using static configured trunks. This is the best practice to minimize the risk of incompatibilities and misconfigurations.

2.12.3 Recommended Solution Load Balancing

EtherChannel load balancing can use MAC addresses, either source mode, destination mode, or both. The mode you select applies to all EtherChannels that you configure on the switch. Use the option that provides the greatest variety in your configuration. For example, if the traffic on a channel only goes to a single MAC address, use of the destination MAC address results in the choice of the same link in the channel each time. Use of source addresses can result in a better load balance. To configure the load balancing, issue the port-channel load-balance {src-mac | dst-mac | src-dst-mac |} global configuration command.

! CAUTION!

In order to avoid loops in the network, please ensure that the affected ports of a port-channel are shut down during the configuration process. Generating loops in a data center network may cause serious network problems!

2.12.4 Configuration

The setup in Figure 6 would be configured in the following steps: Step 1: Shut down the affected ports to avoid loops Step 2: Set up the port-channel Step 3: Bring up the affected ports Step 4: Verify the operation of the port-channels

Step 1: Shut down the affected ports to avoid loops ! 10 GbE Switch Blade interface ap11 shutdown exit interface ap12 shutdown exit

Checking the port status ! 10 GbE Switch Blade <Ctrl-Z> show interface ap11 Interface ap11 is DOWN ... show interface ap12 Interface ap12 is DOWN ...

! Cisco A interface range Gi 1/1 –2 shutdown end



Page: 11 of 108

Step 2: Set up the port-channel group ! 10 GbE Switch Blade Enable config interface ap11 static-channel-group exit interface ap12 static-channel-group exit

Verifying the settings ! 10 GbE Switch Blade show static-channel-group % Static Aggregator: sa1 % Member: ap11 - Link down ap12 - Link down exit interface sa1 port-channel load-balance src-dst-mac exit

! Cisco A interface Port-channel 1 ! interface range tenGi 0/1 -2 channel-group 1 mode on end

Step 3: Bring up the affected ports ! 10 GbE Switch Blade Enable config interface ap11 no shutdown exit interface ap12 no shutdown exit

! Cisco A interface Po 1 no shutdown end



Page: 12 of 108

Step 4: Verify the operation of the port-channels

! 10 GbE Switch Blade show static-channel-group % Static Aggregator: sa1 % Member: ap11 - Link up ap12 - Link up

! Cisco A show etherchannel summary Flags: D -down P -in port-channel I -stand-alone s -suspended H -Hot-standby (LACP only) R -Layer3 S -Layer2 U -in use f -failed to allocate aggregator u -unsuitable for bundling w -waiting to be aggregated d -default port Number of channel-groups in use: 1 Number of aggregators: 1 Group Port-channel Protocol Ports ------+-------------+-----------+---------------------------------------------- 1 Po1(SU) -Gi0/1(P) Gi0/2(P)



Page: 13 of 108

2.13 VLANs and Trunks 2.13.1 Introduction

Most network administrators want to partition their network into multiple broadcast domains to provide better network stability and better information security. This is implemented using virtual LAN technology (VLANs), which provides multiple virtual LAN segments in one switched network domain as specified in the standard 802.1Q. A number of protocols have been developed to simplify the management of such VLANs. While Cisco uses its own proprietary VLAN Trunking Protocol (VTP), the IEEE describes the GARP VLAN Registration Protocol (GVRP), which has been implemented in the 10 GbE Switch Blade.

10 GbE Switch Blade

ap11 ap12 PRIMERGY BX600 S3

TenGi3/4 TenGi4/4

Cisco A

Po1

Po1

02

01

VLAN 1,10,10,20,30

VLAN 1,10,10,20

Port-channel and VLAN trunktransporting 1, 10 and 20

Figure 6: VLAN trunk between 10 GbE Switch Blade and Cisco Switch

When multiple switches are interconnected, there is often a need to transport multiple VLANs over one line. This technique is called VLAN trunking and is described in the IEEE standard 802.1Q and implemented in the 10 GbE Switch Blade. Some older Cisco switches implement a proprietary and incompatible ISL, but all devices found in modern data centers will support 802.1Q trunks. Figure 2 shows a typical setup between a Cisco and a 10 GbE Switch Blade, whereby a port-channel is combined with a VLAN trunk.

It is important to know the role of the so-called native VLAN on an 802.1Q trunk. All the packets on the trunk are encapsulated in 802.1Q packets, which means that a header containing the VLAN number and certain other information is added to the packet before it is transported over the trunk. Only the packets of the native VLAN are untagged for a variety of reasons.

2.13.2 Recommended Solution

Cisco’s VTP and standard GVRP are not compatible. Since a VLAN registration protocol is only useful when applied to several switches within a switch domain, GVRP is not recommended in a Cisco environment.

A number of features of the current version 2.0 make it neither usual nor advisable to use VTP in data center networks.

The design of the VTP server and client concept is extremely delicate: If you bring in a VTP client switch with a higher configuration version number than the rest of the network, all the switches will copy the VLAN database from this switch. This will be a disaster if the new switch has been used in a laboratory and one or more VLANs had been deleted in the meantime.

Manual trunk configuration specifies exactly which VLAN is on which trunk. This will simplify troubleshooting.



Page: 14 of 108

Manual trunk configuration may help the administrator to set up simple load sharing. We therefore recommend using manual VLAN registration in a Cisco data center network.

Since the 10 GbE Switch Blade does not support ISL, the only solution for VLAN trunks to Cisco switches is IEEE 802.1Q. When STP is used, which is the case for most data centers, it is necessary to use a native VLAN because the standard defines that BPDUs have to be transported untagged. (See also Spanning Tree Protocol on page 17.)

Cisco recommends not using VLAN 1 for anything productive. It therefore makes sense to configure the management IP address of the 10 GbE Switch Blade into another VLAN, but it is nevertheless important to have one native VLAN defined on the trunk.


You set up a VLAN trunk as shown in Figure 6 and our recommendations by performing the following steps: • Step 1: Configure the port-channels • Step 2: Define the VLANs • Step 3: Configure the VLAN trunk • Step 4: Verify the VLAN trunk

Step 1: Configure the port-channels

Please see section 2.12.4.

Step 2: Define the VLANs ! 10 GbE Switch Blade ! Configure the VLANs (VLAN 1 is the default and can’t be configured) vlan database vlan 10 bridge 1 name VLAN-10 vlan 20 bridge 1 name VLAN-10 exit

! Cisco-A ! Configure the VLANs (VLAN 1 is the default and can’t be configured) vlan 10 name VLAN-10 exit vlan 20 name VLAN-20 exit vlan 30 name VLAN-30 exit



Page: 15 of 108

Step 3: Configure the VLAN trunk ! 10 GbE Switch Blade ! Configure the interfaces for VLAN trunking interface sa1 switchport mode trunk switchport trunk allowed vlan add 10,20

! Cisco-A interface Port-channel 1 ! Port-channel and ports must be in mode L2. Switch with switchport <cr> switchport trunk native vlan 1 switchport trunk encapsulation dot1q switchport mode trunk switchport trunk allowed vlan 1,10,20 ! interface range tenGi 0/1 -2 ! The native VLAN 1 is the default and not normally displayed in configuration switchport trunk native vlan 1 switchport trunk allowed vlan 1,10,20 switchport trunk encapsulation dot1q switchport mode trunk channel-group 1 mode on



Page: 16 of 108

Step 4: Verify the VLAN trunk ! 10 GbE Switch Blade show vlan all bridge 1 Bridge VLAN ID Name State Member ports (u)-Untagged, (t)-Tagged =============== ======= ================ ======= =============================== 1 1 default ACTIVE ap1(u) ap2(u) ap3(u) ap4(u) ap5(u) ap6(u) ap7(u) ap8(u) ap9(u) ap10(u) ap11(t) sa1(t) ap12(t) 1 10 VLAN0010 ACTIVE sa1(t) ap11(t) ap12(t) 1 20 VLAN0020 ACTIVE sa1(t) ap11(t) ap12(t) show interface switchport bridge 1 Interface name : ap11 Switchport mode : trunk Ingress filter : disable Acceptable frame types : vlan-tagged only Default Vlan : 1 Configured Vlans : 1 10 20 Interface name : ap12 Switchport mode : trunk Ingress filter : disable Acceptable frame types : vlan-tagged only Default Vlan : 1 Configured Vlans : 1 10 20 Interface name : sa1 Switchport mode : trunk Ingress filter : disable Acceptable frame types : vlan-tagged only Default Vlan : 1 Configured Vlans : 1 10 20

! Cisco-A Cisco-A# show interface trunk Port Mode Encapsulation Status Native vlan Po1 on 802.1q trunking 1 Port Vlans allowed on trunk Po1 1,10,20 Port Vlans allowed and active in management domain Po1 1,10,20 Port Vlans in spanning tree forwarding state and not pruned Po1 1,10,20



Page: 17 of 108

2.14 Spanning Tree Protocol 2.14.1 Introduction

When the only standard for spanning tree protocols in LANs was STP, as specified in 802.1D, Cisco developed a number of proprietary protocol enhancements. Some of these were adopted into the RSTP standard but others were not. Cisco therefore also modified their RSTP implementation to be compatible with their enhanced STP. Table 3 shows all current STP implementations.

STP 802.1D STP as specified in 802.1D. Slow convergence, does not support multiple instances for VLAN trunks.

10 GbE Switch Blade: conforms to the standard Cisco: supported only on access ports, not on trunks.

RSTP 802.1w Rapid STP as specified in 802.1w. Fast convergence, does not support multiple instances for VLAN trunks.

10 GbE Switch Blade: conforms to the standard Cisco: supported only on access ports, not on trunks.

MSTP 802.1s Multiple-instance STP as specified in 802.1s. Fast convergence, supports multiple instances for VLAN trunks.

10 GbE Switch Blade: conforms to the standard Cisco: conforms to the standard but not common in Cisco environments.

PVST+ STP as specified in 802.1D with the following enhancements:

Port-fast feature

Uplink-fast feature

Backbone-fast features

Spanning tree for each VLAN Fast convergence, compatible with 802.1D even on VLAN trunks.

Cisco: proprietary solution 10 GbE Switch Blade.

PVST Like PVST+ but supporting only ISL trunks. Cisco: proprietary solution.

RAPID-PVST+ RSTP as specified in 802.1w with the following enhancements:

Spanning tree for each VLAN Fast convergence, compatible with 802.1D even on VLAN trunks.

Cisco: proprietary solution 10 GbE Switch Blade.

Table 3: Spanning tree protocol implementations



Page: 18 of 108

If there are two 10 GbE Switch Blades 10/2, a proprietary L2 protocol (VLD Virtual Link Down) is used for the connection to the 10 GbE I/O module to reduce the failover time that is caused, for example, when a forwarding connection is removed. This is the best-practice configuration.

VLD in combination with 2x BX Switch Blades and the 10 GbE I/O Module is described in more detail in section 2.15 Redundant Configuration with Two PRIMERGY BX600 10 GbE Switch Blades 10/2. However, the configuration examples can easily be transferred to a 2-switch configuration.

Therefore, the following spanning tree examples are only shown for BX600 configurations with only one 10 GbE Switch Blade.

Best practice: We recommend that the two uplinks go to the next switch as EtherChannels, i.e. no spanning tree configurations are created. This allows the highest downlink bandwidth, and the spanning tree configuration is transferred to the next switch level.

On the Cisco switch side, VLAN trunks or access points are then simply configured, and the other side on the 10 GbE Switch Blade is configured accordingly.

ServerBlade

BE 0

PRIMERGY BX600 S3

BA

NET 3 NET 4

Cisco A Cisco B

No STP

Figure 7: 1x 10 GbE Switch Blade without STP (recommended)



Page: 19 of 108

ServerBlade

BE 0

PRIMERGY BX600 S3

BA

NET 3 NET 4

Cisco A Cisco B

STP

n/a

Figure 8: 1x 10 GbE Switch Blade without STP (recommended) and 2 x 10 GbE Switch Blades with STP (not recommended)

When connecting switches without VLAN trunks, PVST+ and STP are compatible with RSTP and RAPID-PVST respectively without any problems. Other combinations are discussed in the following section.



Page: 20 of 108

Running ST P 802.1D with PVST+ on VLAN Trunks

When running STP over VLAN trunks, MSTP is the only STP protocol implemented by Cisco that completely complies with the IEEE standard. Unfortunately it is not usually used in data center networks, where PVST+ and RAPID-PVST are more common. Unlike 802.1D, in which only one STP instance is used to control the STP state of the trunk, PVST+ runs one STP instance per VLAN, sends BPDUs and maintains one STP state per VLAN on a trunk. In addition to this major deviation from the standard, Cisco added a number of minor changes, such as the port-fast, uplink-fast and backbone-fast features, which have only local effects and do not limit their interoperability. PVST+ is also compatible with STP as specified in 802.1D when there is a native VLAN on the trunk. Figure 9 shows a scenario in which two Cisco switches are running PVST+ and a 10 GbE Switch Blade is running STP as specified in 802.1D

10 GbE Switch Bladepriority 32768

ap11 ap12

PRIMERGY BX600 S3

TenGi 3/4

Cisco Apriority 0 for all VLANs

Po3

On all trunks:VLAN 1 nativeVLAN 10 taggedVLAN 20 tagged

Cisco Bpriority 4096 for all VLANs

Po3

Designatedport forwarding


Root portforwarding


TenGi 4/4

Alternatediscarding

GI 1/1 and GI 1/3 GI 1/1 and GI 1/3

Figure 9: Combining PVST+ and 802.1D

Switch A is configured as the root bridge, while switch B will take over the root role when A fails. Since switch A sends untagged BPDUs from VLAN 1 to TenGi 1/50, the 10 GbE Switch Blade uses ap11 as the root port. Ap12 of the 10 GbE Switch Blade will take on the port role “alternate” and will be in the state “discarding” and will not send any BPDUs at this port. Switch B will therefore also set its port TenGi 1/50 to “designated” and “forwarding”. The 10 GbE Switch Blade takes all decisions as indicated by the BPDUs in VLAN 1, and all other BPDUs will be ignored. It is therefore important that one native VLAN is defined at both VLAN trunks. Cisco recommends that this native VLAN be the same for both trunks to the 10 GbE Switch Blade. If the ap11 link or switch A itself fails, the 10 GbE Switch Blade will change the role of ap12 to “designated” and its state to “forwarding”, after going through the state “learning”. According to the standard this will lead to a failover time of approximately twice the forward delay, which in normal cases will be about 30 seconds. Depending on the size of the network, this time can be reduced by tuning the STP timers, but this must be done very carefully in order to provide a stable network. Please refer to the standard 802.1D or Cisco’s recommendations for timer tuning. When the 10 GbE Switch Blade is running 802.1D, it supports features such as Cisco’s proprietary port-fast when the “spanning-tree edgeport” command is applied. This means that an access port will take on the state “forwarding” and will omit the states “listening” and “learning”. This is needed when PXE boot mechanisms are used.



Page: 21 of 108

Running PVST+ on VLAN Trunks while Disabling STP at the 10 GbE Switch Blade

When STP is disabled at the 10 GbE Switch Blade, it bridges the BDPUs without any modifications. Figure 10 shows this scenario.


ap11 ap12

PRIMERGY BX600 S3

TenGi 3/4


Po3



Po3



Root portforwarding


TenGi 4/4

Alternatediscarding


Figure 10: PVST + with disabled STP on the 10 GbE Switch Blade

Since switch B receives the BPDUs of switch A, its port ap12 will get the role “alternate” and it will take on the state “discarding”. The 10 GbE Switch Blade will not be involved in any decisions while the topology is changing. If the link ap11 fails, switch B will not receive any BPDUs at ap12. After three times the “hello” interval, ap12 will initiate its change to the role “designated” and will subsequently take on the “forwarding” state. Since no STP is enabled at the 10 GbE Switch Blade, all the switch’s ports will be enabled and forwarding as soon as they come up. Without STP timer tuning, worst-case failover times resulting from link or switch failures were found to be approximately 45 seconds.



Page: 22 of 108

Rapid Spanning Tree

The standard IEEE 802.1w (RSTP) defines only BPDUs in the native VLAN as implemented by the 10 GbE Switch Blade. Cisco also enhanced RSTP to RAPID-PVST, which is compatible with RSTP in a number of ways. Figure 5 shows this scenario.

Alternatediscarding



Po3



Po3



Root portforwarding


Server 1VLAN 10

MAC Address TableMAC_1 @ Port 0/1MAC_2 @ Port Po3

MAC Address TableMAC_1 @ Port Po2MAC_2 @ Port 01

MAC Address TableMAC_1 @ Port Po3MAC_2 @ Port Po1

ap11ap12

TenGI 4/4

TenGI 1/50

Server 2VLAN 10

PRIMERGY BX600 S3

GI 1/1 andGI 1/2

GI 1/1 andGI 1/2


Figure 11: Combining RAPID-PVST and 802.1w

All RSTP features are functioning for the native VLAN (in this example VLAN1). Since the 10 GbE Switch Blade implements the standard and does not know about tagged BPDUs, RAPID-PVST has the same restrictions as PVST+. There is an additional problem due to the fact that RSTP generates a Topology Change Notification (TCN) only when changing a port to the state “designated”. If the ap11 link in Figure 11: Combining RAPID-PVST and 802.1w fails, port TenGi 1/50 of switch A will go down and will not generate a TCN as specified in 802.1w. The 10 GbE Switch Blade will change the role of port ap12 to root port and its state to “forwarding” and will generate a TCN as specified in 802.1w on the native VLAN. This has the effect that the Cisco switches will flush their MAC address tables for VLAN1 but not for the other VLANs.



Page: 23 of 108

Root portforwarding

10 GbE Switch Blade 10/2RSTP 802.1wpriority 32768

Cisco ARAPID-PVSTpriority 0 for all VLANs

Po3


Cisco BRAPID-PVSTpriority 4096 for all VLANs

Po3


down

Server 1VLAN 10

MAC Address TableMAC_1 @ Port 0/1MAC_2 @ Port Po3

MAC Address TableMAC_1 @ Port Po2MAC_2 @ Port 01

Server 2VLAN 10

MAC Address TableMAC_1 @ Port Po3MAC_2 @ Port Po1

ap11ap12

TenGi 4/4

TenGi 1/50

down



Figure 12: Combining RAPID-PVST and 802.1w after failure of ap11

Figure 12 shows this scenario. When server 1 now wants to send data to server 2, switch B will send it to switch A via Po3 (as indicated by the MAC address table), which has no connection to the 10 GbE Switch Blade and will drop the packet. This will not change until either the MAC address table entry times out (after ~300 seconds) or the server 10 GbE Switch Blade sends a packet that has been seen by switch B, whichever happens first. This scenario shows that RSTP and RAPID-PVST are not compatible in this respect. A worst-case failover time of 300 sec will not be acceptable.



Page: 24 of 108

Running RAPID-PVST on VLAN Trunks while Disabling STP at the 10 GbE Switch Blade

When RAPID-PVST is running at the Cisco switches and STP is disabled at the 10 GbE Switch Blade, we have almost the same scenario as above, where the Cisco switches were running STP and STP was disabled at the 10 GbE Switch Blade. Figure 13 shows this scenario.



ap11 ap12

PRIMERGY BX600 S3

TenGI 4/4


Po3



Po3



Root portforwarding

TenGI 1/50

Alternatediscarding

GI 1/1 and GI 1/2GI 1/1 and GI 1/2

Root portforwarding

Figure 13: RAPID-PVST while STP is disabled at 10 GbE Switch Blade

When the ap11 link fails, the TenGi 1/50 of switch B will stop receiving BPDUs. After three times the “hello” interval, the switch will change the state of port TenGi 1/50 to “learning” and will then follow the normal state machine so that the convergence time is the same as with 802.1D. Since the RSTP cannot operate with the proposal/agreement mechanism on this link, root changes will also be relatively slow within all the VLANs that are running on the trunks to the 10 GbE Switch Blade.



Page: 25 of 108


As discussed earlier, there are a number of different combinations of STP protocols that can be selected when integrating 10 GbE Switch Blade switches into Cisco networks. Although using MSTP between the Cisco and the 10 GbE Switch Blade would be the best solution, it will not be discussed further in this paper because MSTP is so very unusual in Cisco networks. If you were to run MSTP (802.1s) on the 10 GbE Switch Blade switches while using STP or RSTP at the Cisco switches, MSTP would fall back to RSTP and STP respectively.

The resulting and possible solutions are shown in the table below.

10 GbE Switch Blade

802.1D 802.1w No STP

PVST+ Ok* Ok Ok Cisco Switch

RAPIDPVST with restrictions (problems with TCN)

with restrictions (problems with TCN)

Ok

Table 4 : Possible STP combinations when using VLAN trunks

The recommended solution when running STP over VLAN trunks between Cisco and PRIMERGY BX600 10 GbE Switch Blades 10/2 is to disable STP completely at the 10 GbE Switch Blade and run the STP or RSTP protocol at the Cisco switches (see 2.15 Redundant Configuration with Two PRIMERGY BX600 10 GbE Switch Blades 10/2).

When the 10 GbE Switch Blade is connected to Cisco switches without VLAN trunks, the preferred solution is RSTP, because this would lead to the shortest failover times.

CAUTION!

In order to avoid loops in the network, please ensure that the VLAN configuration on both uplinks is the same. Misconfiguration may lead to unidirectional links and network loops!

!



Page: 26 of 108

2.14.3 Configuration with VLAN Trunks

You set up the scenario shown in Figure 14 by performing the following steps: • Step 1: Configure the switches • Step 2: Verify the configuration


ap11 ap12

PRIMERGY BX600 S3

TenGI 4/4


Po3



Po3



Root portforwarding

TenGI 1/50

Alternatediscarding


Root portforwarding

Figure 14: Configuration example of RAPID-PVST while STP is disabled at 10 GbE Switch Blade



Page: 27 of 108

Step 1: Configure the switches ! 10 GbE Switch Blade configuration ! ! If required, remove static channel groups interface ap11 no static-channel-group exit interface ap12 no static-channel-group exit ! Disable STP for the whole switch ! (This command is not normally displayed) no bridge 1 spanning-tree enable ! Define the VLANs vlan database vlan 10 bridge 1 name VLAN-10 vlan 20 bridge 1 name VLAN-10 exit ! Configure VLAN trunks for the interfaces interface ap11 switchport mode trunk switchport trunk allowed vlan add 10,20 ! Forward bpdu packets for ap11 spanning-tree portfast exit interface ap12 switchport mode trunk switchport trunk allowed vlan add 10,20 ! Forward bpdu packets for ap12 spanning-tree portfast exit



Page: 28 of 108

! Cisco Switch A ! Enable and configure RSTP spanning-tree mode rapid-pvst spanning-tree vlan 1,10,20 priority 0 ! Timers are tuned. Please refer to Cisco documentation before ! using this part of the configuration spanning-tree vlan 1,10,20 hello-time 1 spanning-tree vlan 1,10,20 forward-time 8 spanning-tree vlan 1,10,20 max-age 11 vlan 10 name VLAN-10 vlan 20 name VLAN-20 ! Define the port-channels interface Port-channel3 !(If required, switch to L2 with switchport <cr>) switchport trunk native vlan 1 switchport trunk encapsulation dot1q switchport mode trunk switchport trunk allowed vlan 1,10,20 interface range GigabitEthernet 1/1 -2 !(If required, switch to L2 with switchport <cr>) switchport trunk native vlan 1 switchport trunk allowed vlan 1,10,20 switchport trunk encapsulation dot1q switchport mode trunk channel-group 3 mode on exit exit interface tenGigabitEthernet 4/4 !(If required, switch to L2 with switchport <cr>) switchport trunk native vlan 1 switchport trunk allowed vlan 1,10,20 switchport trunk encapsulation dot1q switchport mode trunk



Page: 29 of 108

! Cisco Switch B ! Enable and configure RSTP spanning-tree mode rapid-pvst spanning-tree vlan 1,10,20 priority 4096 ! Timers are tuned. Please refer to Cisco documentation before ! using this part of the configuration spanning-tree vlan 1,10,20 hello-time 1 spanning-tree vlan 1,10,20 forward-time 8 spanning-tree vlan 1,10,20 max-age 11 vlan 10 name VLAN-10 vlan 20 name VLAN-20 ! Define the port-channels interface Port-channel3 !(If required, switch to L2 with switchport <cr>) switchport trunk native vlan 1 switchport trunk encapsulation dot1q switchport mode trunk switchport trunk allowed vlan 1,10,20 interface range GigabitEthernet 1/1 -2 !(If required, switch to L2 with switchport <cr>) switchport trunk native vlan 1 switchport trunk allowed vlan 1,10,20 switchport trunk encapsulation dot1q switchport mode trunk channel-group 3 mode on interface tenGigabitEthernet 1/50 !(If required, switch to L2 with switchport <cr>) switchport trunk native vlan 1 switchport trunk allowed vlan 1,10,20 switchport trunk encapsulation dot1q switchport mode trunk



Page: 30 of 108

Step 2: Verify the configuration

! Check if STP is disabled at 10 GbE Switch Blade show spanning-tree !(No output means no running spanning-tree configuration ok)

! Check RSTP state at Cisco Switch A ! Cisco-A#show spanning-tree VLAN0001 Spanning tree enabled protocol rstp Root ID Priority 0 Address 0017.df07.3581 This bridge is the root Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Bridge ID Priority 0 Address 0017.df07.3581 Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Aging Time 300 Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Te4/4 Desg FWD 2 128.388 P2p Po3 Desg FWD 3 128.1665 P2p VLAN0010 Spanning tree enabled protocol rstp Root ID Priority 0 Address 0017.df07.358a This bridge is the root Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Bridge ID Priority 0 Address 0017.df07.358a Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Aging Time 300 Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Te4/4 Desg FWD 2 128.388 P2p Po3 Desg FWD 3 128.1665 P2p VLAN0020 Spanning tree enabled protocol rstp Root ID Priority 0 Address 0017.df07.3594 This bridge is the root Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Bridge ID Priority 0 Address 0017.df07.3594 Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Aging Time 300 Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Te4/4 Desg FWD 2 128.388 P2p Po3 Desg FWD 3 128.1665 P2p



Page: 31 of 108

! Check RSTP state at Cisco Switch B Cisco-B#show spanning-tree VLAN0001 Spanning tree enabled protocol rstp Root ID Priority 0 Address 0017.df07.3581 Cost 3 Port 643 (Port-channel3) Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Bridge ID Priority 4097 (priority 4096 sys-id-ext 1) Address 0015.fa80.9f00 Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Aging Time 300 Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Te1/50 Desg FWD 2 128.50 P2p Po3 Root FWD 3 128.643 P2p VLAN0010 Spanning tree enabled protocol rstp Root ID Priority 0 Address 0017.df07.358a Cost 2 Port 50 (TenGigabitEthernet1/50) Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Bridge ID Priority 4106 (priority 4096 sys-id-ext 10) Address 0015.fa80.9f00 Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Aging Time 300 Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Te1/50 Root FWD 2 128.50 P2p Po3 Altn BLK 3 128.643 P2p VLAN0020 Spanning tree enabled protocol rstp Root ID Priority 0 Address 0017.df07.3594 Cost 2 Port 50 (TenGigabitEthernet1/50) Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Bridge ID Priority 4116 (priority 4096 sys-id-ext 20) Address 0015.fa80.9f00 Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Aging Time 300 Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Te1/50 Root FWD 2 128.50 P2p Po3 Altn BLK 3 128.643 P2p



Page: 32 of 108

2.14.4 Configuration without VLAN Trunks

You set up the scenario shown in Figure 15 by performing the following steps: • Step 1: Configure the switches • Step 2: Verify the configuration


ap11 ap12

PRIMERGY BX600 S3

TenGI 4/4


Po3

No trunks


Po3



Root portforwarding

TenGI 1/50

Alternatediscarding


Root portforwarding


Figure 15: Configuration example of RSTP without VLAN trunks



Page: 33 of 108

Step 1: Configure the switches ! 10 GbE Switch Blade configuration ! ! If required, remove static channel groups interface ap11 no static-channel-group exit interface ap12 no static-channel-group exit

! Disable STP for the whole switch ! (This command is not normally displayed) no bridge 1 spanning-tree enable ! Configure VLAN trunks for the interfaces interface ap11 switchport mode trunk switchport trunk allowed vlan add 10,20 ! Forward bpdu packets for ap11 spanning-tree portfast exit interface ap12 switchport mode trunk switchport trunk allowed vlan add 10,20 ! Forward bpdu packets for ap12 spanning-tree portfast exit



Page: 34 of 108

! Cisco Switch A ! Enable and configure RSTP spanning-tree mode rapid-pvst spanning-tree vlan 1,10,20 priority 0 ! Timers are tuned. Please refer to Cisco documentation before ! using this part of the configuration spanning-tree vlan 1,10,20 hello-time 1 spanning-tree vlan 1,10,20 forward-time 8 spanning-tree vlan 1,10,20 max-age 11 ! vlan 10 name VLAN-10 ! vlan 20 name VLAN-20 ! Define the port-channels interface Port-channel3 !(If required, switch to L2 with switchport <cr>) switchport trunk native vlan 1 switchport trunk encapsulation dot1q switchport mode trunk switchport trunk allowed vlan 1,10,20 interface range GigabitEthernet 1/1 -2 !(If required, switch to L2 with switchport <cr>) switchport trunk native vlan 1 switchport trunk allowed vlan 1,10,20 switchport trunk encapsulation dot1q switchport mode trunk channel-group 3 mode on exit exit interface tenGigabitEthernet 4/4 !(If required, switch to L2 with switchport <cr>) switchport trunk native vlan 1 switchport trunk allowed vlan 1,10,20 switchport trunk encapsulation dot1q switchport mode trunk ! Cisco Switch B ! Enable and configure RSTP spanning-tree mode rapid-pvst spanning-tree vlan 1,10,20 priority 4096 ! Timers are tuned. Please refer to Cisco documentation before ! using this part of the configuration spanning-tree vlan 1,10,20 hello-time 1 spanning-tree vlan 1,10,20 forward-time 8 spanning-tree vlan 1,10,20 max-age 11 vlan 10 name VLAN-10 vlan 20 name VLAN-20 ! Define the port-channels interface Port-channel3 !(If required, switch to L2 with switchport <cr>) switchport trunk native vlan 1 switchport trunk encapsulation dot1q switchport mode trunk switchport trunk allowed vlan 1,10,20 interface range GigabitEthernet 1/1 -2



Page: 35 of 108

!(If required, switch to L2 with switchport <cr>) switchport trunk native vlan 1 switchport trunk allowed vlan 1,10,20 switchport trunk encapsulation dot1q switchport mode trunk channel-group 3 mode on interface tenGigabitEthernet 1/50 !(If required, switch to L2 with switchport <cr>) switchport trunk native vlan 1 switchport trunk allowed vlan 1,10,20 switchport trunk encapsulation dot1q switchport mode trunk

Step 2: Verify the configuration ! Check if STP is disabled @ 10 GbE Switch Blade show spanning-tree !(No output means no running spanning-tree configuration ok)

! Check RSTP state at Cisco Switch A ! Cisco-A#show spanning-tree VLAN0001 Spanning tree enabled protocol rstp Root ID Priority 0 Address 0017.df07.3581 This bridge is the root Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Bridge ID Priority 0 Address 0017.df07.3581 Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Aging Time 300 Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Te4/4 Desg FWD 2 128.388 P2p Po3 Desg FWD 3 128.1665 P2p VLAN0010 Spanning tree enabled protocol rstp Root ID Priority 0 Address 0017.df07.358a This bridge is the root Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Bridge ID Priority 0 Address 0017.df07.358a Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Aging Time 300 Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Te4/4 Desg FWD 2 128.388 P2p Po3 Desg FWD 3 128.1665 P2p VLAN0020 Spanning tree enabled protocol rstp Root ID Priority 0 Address 0017.df07.3594 This bridge is the root Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec



Page: 36 of 108

Bridge ID Priority 0 Address 0017.df07.3594 Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Aging Time 300 Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Te4/4 Desg FWD 2 128.388 P2p Po3 Desg FWD 3 128.1665 P2p

! Check RSTP state at Cisco Switch B Cisco-B#show spanning-tree VLAN0001 Spanning tree enabled protocol rstp Root ID Priority 0 Address 0017.df07.3581 Cost 3 Port 643 (Port-channel3) Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Bridge ID Priority 4097 (priority 4096 sys-id-ext 1) Address 0015.fa80.9f00 Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Aging Time 300 Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Te1/50 Root FWD 2 128.50 P2p Po3 Altn BLK 3 128.643 P2p VLAN0010 Spanning tree enabled protocol rstp Root ID Priority 0 Address 0017.df07.358a Cost 2 Port 50 (TenGigabitEthernet1/50) Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Bridge ID Priority 4106 (priority 4096 sys-id-ext 10) Address 0015.fa80.9f00 Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Aging Time 300 Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Te1/50 Root FWD 2 128.50 P2p Po3 Altn BLK 3 128.643 P2p VLAN0020 Spanning tree enabled protocol rstp Root ID Priority 0 Address 0017.df07.3594 Cost 2 Port 50 (TenGigabitEthernet1/50) Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Bridge ID Priority 4116 (priority 4096 sys-id-ext 20) Address 0015.fa80.9f00 Hello Time 1 sec Max Age 11 sec Forward Delay 8 sec Aging Time 300 Interface Role Sts Cost Prio.Nbr Type ---------------- ---- --- --------- -------- -------------------------------- Te1/50 Root FWD 2 128.50 P2p Po3 Altn BLK 3 128.643 P2p



Page: 37 of 108

! 10 GbE Switch Blade configuration ! Enable RSTP for the whole switch bridge 1 rapid-spanning-tree enable ! Configure the interfaces interface ap11 spanning-tree portfast exit interface ap12 spanning-tree portfast exit ! Cisco Switch A ! Enable and configure RSTP spanning-tree mode rapid-pvst spanning-tree vlan 1 priority 0 ! Timers are tuned. Please refer to Cisco documentation before ! using this part of the configuration spanning-tree vlan 1 hello-time 1 spanning-tree vlan 1 forward-time 8 spanning-tree vlan 1 max-age 11 ! Define the port-channels interface tenGigabitEthernet 4/4 ! These commands are defaults and not normally displayed switchport mode access switchport access vlan 1 interface Port-channel3 ! These commands are defaults and not normally displayed switchport mode access switchport access vlan 1 interface range GigabitEthernet1/1 -2 ! These commands are defaults and not normally displayed switchport mode access switchport access vlan 1 channel-group 3 mode on ! Cisco Switch B ! Enable and configure RSTP spanning-tree mode rapid-pvst spanning-tree vlan 1 priority 0 ! Timers are tuned. Please refer to Cisco documentation before ! using this part of the configuration spanning-tree vlan 1 hello-time 1 spanning-tree vlan 1 forward-time 8 spanning-tree vlan 1 max-age 11 ! Define the port-channels interface tenGigabitEthernet 1/50 ! These commands are defaults and not normally displayed switchport mode access switchport access vlan 1 interface Port-channel3 ! These commands are defaults and not normally displayed switchport mode access switchport access vlan 1



Page: 38 of 108

Interface range GigabitEthernet1/1 -2 ! These commands are defaults and not normally displayed switchport mode access switchport access vlan 1 channel-group 3 mode on

Step 2: Verify the configuration ! Check if RSTP is enabled @ 10 GbE Switch Blade show spanning-tree % 1: Spanning Tree enabled % 1: root path cost 0 - priority 32768 % 1: forward-time 15 - hello-time 2 - max-age 20 - root port 0 % 1: root id 8000001688040001 % 1: bridge id 8000001688040001 % 1: hello timer 1 - tcn timer 0 - topo change timer 0 % 1: 0 topology changes - last topology change Thu Jan 1 00:00:00 1970 % 1: portfast bpdu-filter disabled % 1: portfast bpdu-guard disabled % 1: portfast errdisable timeout disabled % 1: portfast errdisable timeout interval 300 sec % ap12: port 15 - id 800f - path cost 2000 - designated cost 0 % ap12: designated port id 800f - state Forwarding - priority 128 % ap12: designated root 8000001688040001 % ap12: designated bridge 8000001688040001 % ap12: forward-timer 0 - hold-timer 0 - msg age timer 0 % ap12: forward-transitions 2 % ap12: portfast disabled % ap12: portfast bpdu-guard default - Current portfast bpdu-guard off % ap12: portfast bpdu-filter default - Current portfast bpdu-filter off % ap12: no root guard configured - Current root guard off % % ap11: port 14 - id 800e - path cost 2000 - designated cost 0 % ap11: designated port id 800e - state Forwarding - priority 128 % ap11: designated root 8000001688040001 % ap11: designated bridge 8000001688040001 % ap11: forward-timer 0 - hold-timer 0 - msg age timer 0 % ap11: forward-transitions 3 % ap11: portfast disabled % ap11: portfast bpdu-guard default - Current portfast bpdu-guard off % ap11: portfast bpdu-filter default - Current portfast bpdu-filter off % ap11: no root guard configured - Current root guard off



Page: 39 of 108

2.15 Redundant Configuration with Two PRIMERGY BX600 10 GbE Switch Blades 10/2 2.15.1 Introduction

To ensure high availability of the servers, most BX600 Basic Units will be equipped with two PRIMERGY BX600 10 GbE Switch Blades 10/2. In this case, each server blade has one NIC port connected to the first 10 GbE Switch Blade 10/2 and another port connected to the second 10 GbE Switch Blade 10/2.

The switch blades in NET 3 and NET 4 are active at the same time, while the BE card (= 10 GbE I/O Module) only uses one port (A or B) at a time (active/standby). The operating system only shows one network card. Card failover is automatically ensured by the firmware and requires no configuration.

However, in the case of connection errors between the uplink ports of the 10GbE Switch Blade switch and the switches of the next higher level, the server blade cannot detect a connection or port error in time.

In such cases, in normal circumstances, there could be a downtime of 5-10 seconds. To speed up this switching process, the 10 GbE Switch Blade and the 10 GbE I/O Module use the VLD protocol to communicate with each other and decide which switch to use to send the data packets.

ServerBlade

BE 0

BA

BXNET4

BXNET3

FW Decides which port is activebased on BX port state (uplink)

active

BX600 S3 Blade Server

VLD VLD

Figure 16: VLD and BE communication

As mentioned in section 2.14 (Spanning Tree Protocol), a proprietary L2 protocol VLD (Virtual Link Down) is used to reduce the switching times of the 10 GbE I/O Module on the server blade from the active port to the passive port. As the 10 GbE I/O Modules have port switching integrated in the firmware and are always active, cross-over cabling for connecting to external switches is not required. Therefore, there is also no need for an STP implementation.

VLD not only controls the LINK and LACP status but also the STP status of the paths (blocking, forwarding, disabled).



Page: 40 of 108

Figure 17 shows the communication between a 10 GbE Switch Blade and the 10 GbE I/O Module. This figure and Table 5 illustrate the parameters used by the firmware of the 10 GbE I/O Module to decide which port is currently active. The rule referring to the most active data paths is also used for the automatic fall-back. This is required to ensure that, in configurations with more than one server blade, the active data paths of the individual blades are not distributed over NET3 and NET4. In the case of an equal number of active paths, NET3 is always given priority (which means that a fall-back is also initiated when at least one path of NET3 becomes active again and NET4 also has only one active port). ap11 and ap12 have the same priority, therefore not all the possible variants of the two ports are listed.

ServerBlade

BE 0

BA

BXNET4

BXNET3

FWDecides which portis activebased on BX portstate (uplink)

active

standbyup only when:- line up- STP forwarding- LACP syncand BX SwitchNET3 has one ormore ports down

ap11 ap12 ap11 ap12

Communicateport status

VLDLink state awareSTP awareLACP aware


Figure 17: 10 GbE Switch Blades and port assignment for I/O modules with VLD dependencies

Status BX Switch NET 3 ap11

NET 3 ap12 Status BX Switch NET 4 ap11

NET 4 ap12 NET 3 active

NET 4 active

LINK STATE: 2 active paths Not relevant X 1 active path Max. 1 active path X 1 active path 2 active paths X 0 active paths Min. 1 active path X

LACP: 2 active paths Not relevant X 1 active path Max. 1 active path X 1 active path 2 active paths X 0 active paths Min. 1 active path X

Table 5: VLD switching matrix



Page: 41 of 108

2.15.2 STP and VLD

Status BX Switch NET 3 ap11

NET 3 ap12 Status BX Switch NET 4 ap11

NET 4 ap12 NET 3 aktiv

NET 4 aktiv

STP: 1 port forwarding, 1 port blocking Max. 1 active path X

1 port blocked or link down, 1 port forwarding Max. 1 active path X 1 port blocked or link down, 1 port not forwarding Min. 1 active path X

Table 6: VLD switching matrix STP and VLD (not recommended)


For server blades equipped with 10GbE LAN I/O Modules there is only one failover mechanism available:

Active Standby Failover

ServerBlade

BE 0

BA

BXNET4

BXNET3

FWDecides which portis activebased on BX portstate (uplink)

active

ap11 ap12 ap11 ap12


sa1 sa1

Po1 Po1

Cisco A Cisco B

Figure 18: VLD with LACP

Figure 18 shows the two 10 GbE Switch Blades NET3 and NET4 used with a 10 GbE I/O Module. There are several criteria according to which the active path is chosen.

2.6.2 Configuration Thanks to the VLD, no link-state groups are required. VLD is automatically active, as is the failover function of the 10 GbE I/O Module. No configuration is required, and these mechanisms are independent of the operating system installed on the server blade. Therefore, the recommended configuration is active instantly and requires no installation steps for the 10 GbE Switch Blade and the 10 GbE I/O Module.



Page: 42 of 108

2.16 Configuring the Access Ports of the Switches 2.16.1 Overview To enable the server blades to access the VLAN, you must define the access ports.

2.17 Recommended Solution Due to the VLD capability of the 10 GbE Switch Blades and 10 GbE I/O Modules, link-state groups are not required.

ServerBlade

BE 0

BA

BXNET4

BXNET3



ap1 ap1

ap11 ap12 ap12ap11

Access portdefinitionwith VLAN 10


! Configuration of PRIMERGY BX600 10 GbE Switch Blade 10/2 -a interface ap11 ! This line is only needed if you are running STP on the switch spanning-tree portfast ! Forbid all VLANs except the access VLAN switchport mode trunk switchport trunk allowed vlan none ! Permit the access VLAN switchport trunk allowed vlan add 10 interface ap1 ! This line is only needed if you are running STP on the switch spanning-tree portfast switchport mode access ! Set the access VLAN as native VLAN switchport access vlan 10



Page: 43 of 108

3 Basic Multicast Services 3.1 Introduction IP Multicast applications are common to many data center networks. At least the deployment software for the blade server often uses multicast to deploy multiple servers using one data stream. In most Cisco networks, the 10 GbE Switch Blade will act as a Layer 2 switch, which has to perform IGMP snooping in order to avoid unnecessary multicast traffic at ports that are not interested in this traffic.

ServerBlade

BE 0

BA

BXNET4

BXNET3



ap1 ap1

ap11 ap12 ap12ap11

Access portdefinitionwith VLAN 10

Figure 19: Redundant configuration with redundant VLD and PortChannel switches

3.2 Recommended Solution It is advisable to enable IGMP snooping over the whole broadcast domain and therefore at all switches. To get IGMP snooping running you will need one IGMP querier per VLAN. In most cases there will be a Layer 3 switch in each VLAN which is also the unicast router for that VLAN. We recommended that you configure this router for multicast routing and enable an IGMP querier in this way, because the multicast router will need the IGMP information anyway. At the 10 GbE Switch Blade and at all other Layer 2 switches, you only need to enable IGMP snooping.



Page: 44 of 108

3.3 Configuration

The following steps are necessary to set up IGMP snooping:

1. Enable multicast routing and IGMP at the Layer 3 switch 2. Enable IGMP snooping at all Layer 2 switches 3. Verify the configuration

1. Enable multicast routing and IGMP at the Layer 3 switch ! Layer 3 Switch Configuration (CISCO A) ! In this example PIM dense mode is activated, as this is the ! simplest solution. In data center networks a more sophisticated solution ! should be used, but multicast routing is not within the scope of this document. ! ip multicast-routing interface Vlan1 ip address 192.168.1.1 255.255.255.0 ip pim dense-mode exit interface Vlan10 ip address 192.168.10.1 255.255.255.0 ip pim dense-mode exit interface Vlan20 ip address 192.168.20.1 255.255.255.0 ip pim dense-mode

2. Enable IGMP snooping at all Layer 2 switches

! Layer 2 Switch Configuration (Cisco B) ! All these commands are enabled by default and are not normally seen in the config ! Enable global IGMP snooping ip igmp snooping ! Enable IGMP snooping for VLANs ip igmp snooping vlan 1 ip igmp snooping vlan 10 ip igmp snooping vlan 20 ! 10 GbE Switch Blade Switch Configuration ! ! Enable global IGMP snooping ip igmp snooping ! Enable IGMP snooping for VLANs ip igmp snooping vlan 10 ip igmp snooping vlan 20 exit ! Enable IGMP snooping for ports ip igmp snooping mrouter interface ap11 exit



Page: 45 of 108

3. Verify the configuration

! 10 GbE Switch Blade #show ip igmp snooping IGMP Snooping is globally enabled IGMP Snooping Proxy is disabled Bridge 1: VLAN 1 IGMP snooping is enabled IGMP snooping query interval is 125000 ms IGMP snooping max query response time is 100 cs IGMP Snooping last member query interval is 1000 ms IGMP snooping other querier timeout interval is 120000 ms IGMP snooping group membership interval is 260000 ms IGMP snooping v1 router present timeout is 400000 ms IGMP snooping interface ap2 version 2 IGMP snooping interface ap3 version 2 IGMP snooping interface ap4 version 2 IGMP snooping interface ap5 version 2 IGMP snooping interface ap6 version 2 IGMP snooping interface ap7 version 2 IGMP snooping interface ap8 version 2 IGMP snooping interface ap9 version 2 IGMP snooping interface ap10 version 2 IGMP snooping interface ap11 version 2 Bridge 1: VLAN 10 IGMP snooping is enabled IGMP snooping query interval is 125000 ms IGMP snooping max query response time is 100 cs IGMP Snooping last member query interval is 1000 ms IGMP snooping other querier timeout interval is 120000 ms IGMP snooping group membership interval is 260000 ms IGMP snooping v1 router present timeout is 400000 ms IGMP snooping interface ap1 version 2 IGMP snooping interface ap11 version 2 Bridge 1: VLAN 20 IGMP snooping is enabled IGMP snooping query interval is 125000 ms IGMP snooping max query response time is 100 cs IGMP Snooping last member query interval is 1000 ms IGMP snooping other querier timeout interval is 120000 ms IGMP snooping group membership interval is 260000 ms IGMP snooping v1 router present timeout is 400000 ms IGMP snooping interface ap11 version 2 (bx6-10 GbE Switch Blade -b) #show ip igmp snooping mrouter Bridge 1: VLAN: 1 Igmp Snooping Enabled Mrouter -> ap11 (Configured) VLAN: 10 Igmp Snooping Enabled Mrouter -> ap11 (Configured) VLAN: 20 Igmp Snooping Enabled Mrouter -> ap11 (Configured)



Page: 46 of 108

WSend and WListen were used for the test:

WSend test with multicast utility on 224.0.0.1

WListen test with multicast utility on 224.0.0.1



Page: 47 of 108

4 Switch Management 4.1 Logging and Synchronization 4.1.1 Introduction

When there are problems in a network it is vital to log the events at all network devices. Since a data center network often consists of many network devices, a central logging server is used to collect the information from all components. Logging information is usually sent using the protocol syslog (RFC 3164), which is supported by both the 10 GbE Switch Blade and Cisco switches.

The server may be a UNIX system, in which a syslog daemon is usually distributed with the operating system, or a Windows system with a special syslog server installed.

A syslog message includes a time stamp to enable administrators to correlate events, and it is therefore necessary to synchronize the time bases used by all the devices.

The standards for this task are NTP and SNTP. NTP (Network Time Protocol) is a mechanism that ensures reliable synchronization between devices over IP networks, even where there is a high delay on the lines, such as when the synchronization is running over WAN links.

In a LAN environment, SNTP is also sometimes used, but the 10 GbE Switch Blade only supports NTP. In the test, a freeware NTP server was used (Zeitgeist NTP Server 1.0).

10 GbE Switch Blade 10/2NET3


ap1

ap11ap12

Management Console

SYSLOG Consoleand SNMP Management Console

Layer 3 switch Layer 2 switch

eth0

SendsSYSLOG, SNMP trapsand NTP packagesonly via eth0

Figure 20: BX Switch sends syslog and SNMP traps to the management console


Since syslog is an unreliable protocol, we recommend that you also enable logging to memory at the 10 GbE Switch Blade. The synchronization should be performed by configuring two NTP servers or using an NTP broadcast source, as specified in whichever standard is in use at the data center.



Page: 48 of 108

4.1.3 Configuring syslog and NTP

The following steps are necessary to enable logging and NTP: • Step 1: Configure the 10 GbE Switch Blade for NTP • Step 2: Configure the 10 GbE Switch Blade for logging and syslog • Step 3: Test the configuration

Step 1: Configure the 10 GbE Switch Blade for NTP

! 10 GbE Switch Blade NTP configuration ! Enable the NTP client ! NTP can only be retrieved via the management port eth0 ntp on ntp server 10.20.33.200 ntp timezone 1 0 before-utc GMT+1

NTP output is as follows::

! 10 GbE Switch Blade show ntp associations address ref clock st when poll reach delay offset disp ~10.20.33.200 172.25.96.31 5 - 64 077 0.0 4294967296.0 203.7 *~127.127.1.0 127.127.1.0 5 34 64 077 0.0 0.0 188.4 * master (synced), # master (unsynced), + selected, - candidate, ~ configured

Step 2: Configure the 10 GbE Switch Blade for logging and syslog

! 10 GbE Switch Blade logging configuration ! Send syslog messages to 10.20.33.200 port 514 (default) ! Include all messages up to debug severity logging syslog 10.20.33.200 7

Step 3: Test the configuration

! 10 GbE Switch Blade logging test with port ap1 shutdown and no shutdown Interface ap1 shutdown No shutdown

On the syslog server, entries may look like this:



Page: 49 of 108

4.2 SNMP 4.2.1 Introduction

In most enterprise networks, SNMP is used for monitoring network components. The most common protocol versions are SNMPv1 and SNMPv2c, which are fully supported by the 10 GbE Switch Blade. SNMPv3 is seldom used today.


We recommend that you enable SNMPv1 and SNMPv2c at the 10 GbE Switch Blade and (for security reasons) enable authentication for SNMPv3.

4.2.3 Configuring SNMP

The following steps are necessary to configure SNMP: • Step 1: Configure SNMP for SNMPv1 and SNMPv2c • Step 2: Configure SNMPv3 authentication • Step 3: Test the SNMP configuration using your preferred SNMP management tool

Step 1: Configure SNMP for SNMPv1 and SNMPv2c

! SNMP v1 and v2c setup for 10 GbE Switch Blade ! Configure the description, system name, contact and location ! System Description "FSC SwitchBlade" snmp-server sysname "bx6-10 GbE Switch Blade -a" snmp-server location "Team PCT" snmp-server contact "Test123" ! Configure two SNMP community strings (e.g. read and write) snmp-server community ro ip 10.20.33.200 ro snmp-server community rw ip 10.20.33.200 rw ! Remove the default community strings no snmp-server community public no snmp-server community private ! Configure the trap receiver ! … for SNMPv2c snmptrap community MySNMPv2 ip 10.20.33.200 v2c ! … for SNMPv1 snmptrap community MySNMPv1 ip 10.20.33.200 v1

Step 2: Configure SNMPv3 authentication

It is important to set the SNMPv3 authentication protocol to “MD5” for each configured user name, to ensure that nobody can access the switch using SNMPv3 without authentication.

! SNMP v1 and v2c setup for 10 GbE Switch Blade ! Use this command to create a user for BX login and with SNMP v3 configuration without ! authentication and encryption. User snmpV3user with password BXsnmpuserV3 ! The user’s password must be 8 characters in length user username snmpV3user passwd BXsnmpuserV3 ! Set the authentication protocol. MD5 and SHA are supported. user snmpv3authprotocol snmpV3user MD5 ! To provide privacy (encryption) for an existing authenticated SNMP v3 user, set snmpv3encrypt to DES user snmpv3encrypt snmpV3user DES myuserkey



Page: 50 of 108

Step 3: Test the SNMP configuration using your preferred SNMP management tool

The following tests have been done using NET-SNMP with SNMPv2:

! Windows command line ! The tools from http://net-snmp.sourceforge.net/ were used for the test C:\> snmpwalk -v 2c -c ro 10.20.22.200 system SNMPv2-MIB::sysDescr.0 = STRING: bx 2.4.20_mvl31-pq2fads Build 1741 #1 Wed Oct 17 04:43:32 PDT 2007 ppc ZebOS 7.2.13.e0 SNMPv2-MIB::sysObjectID.0 = OID: iso.2.3.4 DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (1144666) 3:10:46.66 SNMPv2-MIB::sysContact.0 = STRING: SNMP v2c Write Test SNMPv2-MIB::sysName.0 = STRING: "bx6-10 GbE Switch Blade -a" SNMPv2-MIB::sysLocation.0 = STRING: "Team PCT" SNMPv2-MIB::sysServices.0 = INTEGER: 10 SNMPv2-MIB::sysORLastChange.0 = Timeticks: (5) 0:00:00.05 SNMPv2-MIB::sysORID.1 = OID: SNMP-FRAMEWORK-MIB::snmpFrameworkMIBCompliance SNMPv2-MIB::sysORID.2 = OID: SNMP-MPD-MIB::snmpMPDCompliance SNMPv2-MIB::sysORID.3 = OID: SNMP-USER-BASED-SM-MIB::usmMIBCompliance SNMPv2-MIB::sysORID.4 = OID: SNMPv2-MIB::snmpMIB SNMPv2-MIB::sysORID.5 = OID: TCP-MIB::tcpMIB SNMPv2-MIB::sysORID.6 = OID: IP-MIB::ip SNMPv2-MIB::sysORID.7 = OID: UDP-MIB::udpMIB SNMPv2-MIB::sysORID.8 = OID: SNMP-VIEW-BASED-ACM-MIB::vacmBasicGroup SNMPv2-MIB::sysORDescr.1 = STRING: The SNMP Management Architecture MIB. SNMPv2-MIB::sysORDescr.2 = STRING: The MIB for Message Processing and Dispatching. SNMPv2-MIB::sysORDescr.3 = STRING: The management information definitions for the SNMP User-based Security Mod el. SNMPv2-MIB::sysORDescr.4 = STRING: The MIB module for SNMPv2 entities SNMPv2-MIB::sysORDescr.5 = STRING: The MIB module for managing TCP implementations SNMPv2-MIB::sysORDescr.6 = STRING: The MIB module for managing IP and ICMP implementations SNMPv2-MIB::sysORDescr.7 = STRING: The MIB module for managing UDP implementations SNMPv2-MIB::sysORDescr.8 = STRING: View-based Access Control Model for SNMP. SNMPv2-MIB::sysORUpTime.1 = Timeticks: (3) 0:00:00.03 SNMPv2-MIB::sysORUpTime.2 = Timeticks: (3) 0:00:00.03 SNMPv2-MIB::sysORUpTime.3 = Timeticks: (3) 0:00:00.03 SNMPv2-MIB::sysORUpTime.4 = Timeticks: (4) 0:00:00.04 SNMPv2-MIB::sysORUpTime.5 = Timeticks: (5) 0:00:00.05 SNMPv2-MIB::sysORUpTime.6 = Timeticks: (5) 0:00:00.05 SNMPv2-MIB::sysORUpTime.7 = Timeticks: (5) 0:00:00.05 SNMPv2-MIB::sysORUpTime.8 = Timeticks: (5) 0:00:00.05 C:\> snmpwalk -v 2c –c ro 10.20.22.200 syscontact SNMPv2-MIB::sysContact.0 = STRING: Test123 C:\>snmpset -v 2c -c rw 10.20.22.200 sysContact.0 s "SNMP v2c Write Test" SNMPv2-MIB::sysContact.0 = STRING: SNMP v2c Write Test C:\>snmpget -v 2c -c ro 10.20.22.200 sysContact.0 SNMPv2-MIB::sysContact.0 = STRING: SNMP v2c Write Test C:\>snmpset -v 2c -c private 10.20.23.200 sysContact.0 s "SNMP -Master" SNMPv2-MIB::sysContact.0 = STRING: SNMP-Master C:\>snmpget -v 2c -c public 10.20.23.200 sysContact.0 SNMPv2-MIB::sysContact.0 = STRING: SNMP-Master

http://net-snmp.sourceforge.net/



Page: 51 of 108

4.3 Remote Console Access 4.3.1 Introduction

In addition to the Web interface, the 10 GbE Switch Blade supports three methods of accessing the command line interface: • Console access using console redirection of the management blade • Telnet access • SSH access

During the initial setup, console redirection is the only possible way of accessing the switch. Access using telnet or SSH will subsequently be more convenient.


Telnet is an unencrypted protocol, which means that not only the data but also the password is sent unencrypted over IP. For this reason most corporate customers prefer not to use telnet. SSH encrypts not only the password but also the entire data traffic and is the preferred protocol for remote console access. We recommend that you enable SSH and disable telnet access to the switch.

4.3.3 Configuring SSH

The following steps are necessary to enable SSH and disable telnet: • Step 1: Configure the 10 GbE Switch Blade • Step 2: Test the login

Step 1: Configure the 10 GbE Switch Blade

! 10 GbE Switch Blade ssh configuration ! Enable ssh ip ssh enable

! BX600 10 GbE Switch Blade 10/2 telnet configuration ! Disable telnet no ip telnet enable

Step 2: Test the telnet and SSH login

C:\>telnet 10.20.22.200 PRIMERGY BX600 10GbE Switch Blade 10/2 v1.0.1763 Copyright 2006-2007 Fujitsu Siemens Computers. login: root Password: MontaVista(R) Linux(R) Professional Edition 3.1 % Connection is closed! Connection to host lost.



Page: 52 of 108

One of the popular SSH clients is “putty” which is distributed under license from MIT.

C:\>putty

Figure 21: PuTTY configuration menu



Page: 53 of 108

4.4 Integration into Radius and TACACS+ Introduction

Radius and TACACS are protocols that can be used for authentication, authorization and accounting. Businesses often use one of these protocols to authenticate administrative users of network components.

The 10 GbE Switch Blade supports RADIUS and TACACS+ for the authentication of users who want to access the switch using the Web interface, telnet or SSH.



ap1

ap11 ap12

TACACS andRADIUS loginonly via eth0

eth0

Server xy

BROADCOM 1 Gb NIC

Cisco ACS Server V4.010.10.100.224

eth0 ip=10.20.22.200

SB9NET2

PC ip=10.20.33.200

Figure 22: Authentication with TACACS and RADIUS


In most Cisco networks a Cisco Secure ACS is used as TACACS+ and RADIUS server. The protocol should be selected in compliance with company policy, so both configurations are described here

4.4.2 Configuring RADIUS

The following steps are necessary to integrate a PRIMERGY BX600 10 GbE Switch Blade 10/2 into RADIUS authentication: • Step 1. Prepare the ACS • Step 2. Configure the PRIMERGY BX600 10 GbE Switch Blade 10/2 • Step 3. Test the login



Page: 54 of 108

Step 1: Prepare the ACS

To prepare the ACS to be an authentication server for the PRIMERGY BX600 10 GbE Switch Blade 10/2, log on to the Web interface of the PRIMERGY BX600 10 GbE Switch Blade 10/2 and perform the following configuration:

Add device/entry.

Use RADIUS IETF and enter the shared key.



Page: 55 of 108

Add the users “test-ro” and “test”.

Add a password.



Page: 56 of 108

Step 2: Configure RADIUS on the 10 GbE Switch Blade

! 10 GbE Switch Blade 10/2 Configuration for RADIUS ! Enable RADIUS radius-server enable ! Add RADIUS Server 1 radius-server server-ip 1 10.20.100.224 ! Enable RADIUS server 1 radius-server server 1 enable ! Add key for RADIUS Server 1 radius-server key 1 fsc

! Test a login with correct user name but wrong password (bx6-PRIMERGY BX600 10 GbE Switch Blade 10/2 -a) > C:\> telnet 10.20.22.200 User:test-ro Password: WRONG

! Test a login with correct user name and password User:test-ro Password:test-ro (bx6-PRIMERGY BX600 10 GbE Switch Blade 10/2 -a) >



Page: 57 of 108

On the ACS you can see the failed and successful attempts:

Display of the failed attempt.

Display of the passed authentication.



Page: 58 of 108

4.4.3 Configuring TACACS

The following steps are necessary to integrate a PRIMERGY BX600 10 GbE Switch Blade 10/2 into TACACS authentication: • Step 1. Prepare the ACS • Step 2. Configure the PRIMERGY BX600 10 GbE Switch Blade 10/2 • Step 3. Test the login

Step 1: Prepare the ACS

To prepare the ACS to be an authentication server for the PRIMERGY BX600 10 GbE Switch Blade 10/2, log on to the Web interface of the PRIMERGY BX600 10 GbE Switch Blade 10/2 and perform the following configuration:

Add the device/entry.



Page: 59 of 108

Use TACACS+ and enter the shared key.

Add the users “test-ro” and “test”.



Page: 60 of 108

Add a password.



Page: 61 of 108

The imish service used by the BX must be made known to the ACS. First enter it under Interface Configuration ...

...then enable the new imish service in the group setup of the relevant user in the TACACS+ Setting section.



Page: 62 of 108

! PRIMERGY BX600 10 GbE Switch Blade 10/2 Configuration for TACACS+ ! Enable TACACS Tacacs enable ! Set the shared key for server 1 tacacs key 1 fsc ! Set the IP address of server 1 tacacs server-ip 1 10.20.100.224 ! Enable tacacs server 1 tacacs server 1 enable

Step 3: Test the login

! Test a login with correct username but wrong password (bx6-PRIMERGY BX600 10 GbE Switch Blade 10/2 -b) C:\>telnet 10.20.22.200 User:test-ro Password:WRONG

! Test a login with correct username and password User:test-ro Password:test-ro (bx6-PRIMERGY BX600 10 GbE Switch Blade 10/2 -b) >

On the ACS you can see the failed and successful attempts (the display of successful attempts is disabled by default):

Display of the failed attempt.



Page: 63 of 108

Display of the passed authentication.



Page: 64 of 108

4.5 Port Monitoring / Mirroring 4.5.1 Introduction

When a network analyzer is used in a switched network, a special switch port configuration is needed in order to copy frames from a specified port to the analyzer port. This feature is called the “port-monitor” for the PRIMERGY BX600 10 GbE Switch Blade 10/2 or the “port-mirror” for Cisco switches. The PRIMERGY BX600 10 GbE Switch Blade 10/2 supports one monitor session with multiple source interfaces, and one destination interface to which the network analyzer is connected. At present port-mirror is not supported on port-channel interfaces.

Due to the high speed of the 10 GB interface, only receive traffic or transmit traffic can be mirrored, i.e. not both kinds of traffic at the same time.



ap1

ap11 ap12

eth0

Server xy

ap2

Port mirror

ap1 traffic is mirrored to ap2

4.5.2 Configuring Port Monitoring/Mirroring

The following steps are necessary to configure a port monitor session: • Step 1: Configure the PRIMERGY BX600 10 GbE Switch Blade 10/2 • Step 2: Check the configuration

Step 1: Configure the PRIMERGY BX600 10 GbE Switch Blade 10/2

! PRIMERGY BX600 10 GbE Switch Blade 10/2 ! Check whether a mirror port is currently active show mirror Port mirror: Disabled ! Configure port mirror interface ap1 mirror interface ap2 direction receive



Page: 65 of 108

Step 2: Check the configuration ! PRIMERGY BX600 10 GbE Switch Blade 10/2 Configuration for TACACS+ show mirror Port mirror: Enabled Monitored port: ap2



Page: 66 of 108

5 10 GbE I/O Module PCIe The 10 GbE LAN I/O Module PCIe is implemented as a daughter card for BX6xx server blades with a PCIe bus. It is based on a BladeEngine™ chip and supports both data exchange via standard IP networks (LAN) and – using the iSCSI protocol – storage networks (SAN).

The 10 GbE LAN I/O Module has two 10 Gbit Ethernet LAN channels. To the operating system, the 10 GbE LAN I/O Module only appears to have one channel, the so-called active port, which is usually port 0. The second channel acts as a standby port. The 10 GbE LAN I/O Module exchanges connection status information with the 10 GbE Switch Blades 10/2 using a special protocol. The firmware of the I/O module uses this status information to ensure that the port with the best access data is always used as the active port. This failover mechanism does not have to be configured, see also section Redundant Configuration with Two PRIMERGY BX600 10 GbE Switch Blades 10/2 on page 39.

This chapter describes how to install the drivers and software for the 10 GbE LAN I/O Module, and how to configure its iSCSI functionality. For more information, see the following manuals:

Microsoft Windows Server 2003 • 10GbE LAN I/O MODUL (PCIe) – iSCSISelect Reference Guide Version 1.0.748.0 • iSCSI Initiator SM-CLP Command Reference Guide Version 1.0.748.0

Microsoft Windows Server 2008 • BladeEngine iSCSISelect Reference Guide Version 1.0.748.1255 • BladeEngine iSCSI Initiator SM-CLP Command Refernece Guide Version 1.0.748.1255

5.1 Installing the Drivers The following drivers and software are provided for installing and configuring the 10 GbE I/O Module: • iSCSI driver • NIC driver • Firmware • SMCLP Server (SMCLP = Server Management Command Line Protocol including Open SSH Server) • SNMP Extension Agent • SMCLP client for configuring the I/O Module from a management server

If you install the operating system from the ServerStart DVD, the drivers and the configuration tools for the 10 GbE I/O Module are also installed automatically.

If you install the operating system without ServerStart, you require a driver diskette for the 10 GbE I/O Module and you must install the configuration software manually. The required files are included in a driver package, which you can download from the Fujitsu Siemens software pool at http://support.fujitsu-siemens.com/com/support/downloads.html.

You also require the driver package for installing the 10 GbE I/O Module later and for updating drivers and the firmware.

i

For information on currently released drivers, check the central release document on the Fujitsu Siemens download server (http://support.fujitsu-siemens.com/Download/Showdescription_KMT.asp?Info=&DokuID=170334).

The following sections describe the manual installation of the drivers and software under MS Windows, SuSE Linux and Red Hat Linux. How to install the drivers and software together with the operating system is explained within the context of iSCSI booting in section 5.3 onwards.

5.1.1 Installing the Drivers and Software under Microsoft Windows Server 2003

The following describes a standard installation.

Insert the software CD in the CD drive and open the Installation Utility folder.

Click Windows to select your operating system.

Run the setupwin.exe file.

The welcome screen of the installation program is displayed.

i

The SNMP service must be active on the target system.

http://support.fujitsu-siemens.com/com/support/downloads.html

http://support.fujitsu-siemens.com/Download/Showdescription_KMT.asp?Info=&DokuID=170334



Page: 67 of 108

Click Next.

Select Custom and click Next.

Select the two options Windows BladeEngine Server and Windows Management Client Note: To manage a server remotely via the SMCLP, you must install the SMCLP client on a PC. In this case, only select Windows Management Client.

Click Next and follow the instructions to install the Windows Management Client.

Create an SMCLP user by entering an SMCLP user name and an SMCLP password. Note: This creates a local Windows user on the server.

Click Next.



Page: 68 of 108

Click Install to run the installation with the displayed options.

The installation takes a few seconds.

To complete the installation, you must restart Windows.

Select Yes, restart my computer and click Finish.

Checking the installation

Open the Windows Computer Management to check the driver and software installation.

The 10 GbE LAN I/O Module is listed in the Device Manager as ServerEngines BladeEngine 10Gb NIC/TOE Adapter under Network adapters and as ServerEngines BladeEngine 10Gb iSCSI Initiator under SCSI and RAID controllers.



Page: 69 of 108

The user smclp is listed under Local Users and Groups – Users.

The Open SSH Server is listed as Openssh SSHD under Services.



Page: 70 of 108

5.1.2 Installing the Drivers and Software under Microsoft Windows Server 2008

The following describes a standard installation.

Insert the software CD in the CD drive and open the Installer folder.

Click Windows to select your operating system.

Run the setup.exe file.

The welcome screen of the installation program is displayed.

Click Next.

Select Typical and click Next.

SNMP is not required. However, if SNMP is not installed, the system issues a warning. Click Next to coninue, or stop the installation and resume it once you have installed SNMP.

i

The SNMP service must be active on the target system.



Page: 71 of 108

Create an SMCLP user by entering an SMCLP user name and an SMCLP password.

iThis creates a local MS Windows user on the server.

Click Next.

Click Install to run the installation with the displayed options.

The installation takes a few seconds.



Page: 72 of 108

To complete the installation, you must restart MS Windows.

Select Restart now or I want to manually restart later and click Finish.

Checking the installation

Open theMS Windows Computer Management to check the driver and software installation.

The 10 GbE LAN I/O Module is listed in the Device Manager as ServerEngines BladeEngine 10Gb NIC/TOE Adapter under Network adapters and as ServerEngines BladeEngine 10Gb iSCSI Initiator under Storage controllers.

The user smclp is listed under Local Users and Groups – Users.



Page: 73 of 108

The Open SSH Server is listed as Openssh SSHD under Services.

5.1.3 Installing under SuSE Linux

The following examples show the installation of the driver and the firmware update for the 10 GbE I/O Module under SuSE Linux Enterprise Server 10 (SLES 10).

For information on 10 GbE I/O Module driver installation and firmware update under the installation or update of SLES 10 please refer the PRIMERGY Novell PLDP README provided by http://www.fujitsu-siemens.com/products/standard_servers/linux_readmes_popup.html.

! CAUTION! The driver and firmware versions of the 10 GbE I/O Module must always be the same.

Installing the driver

Select your server blade and your operating system in the Fujitsu Siemens download portal http://support.fujitsu-siemens.com/com/support/downloads.html.

Download the latest Linux Driver Package and save it on a memory stick.

Copy the ZIP archive file to the /tmp/SuSE_10_x86-64 directory on the target system.

bx620s4pvt5:/tmp/SuSE_10_x86-64 # cp /media/disk/Treiber_Pakete_Linux/SuSE_10_x86- 64/FSC_LINUX_SUSE_SLES_10_X86_64_DRIVERS_071207_071207__1018747.ZIP .

Unzip the ZIP archive file.

bx620s4pvt5:/tmp/SuSE_10_x86-64 # unzip FSC_LINUX_SUSE_SLES_10_X86_64_DRIVERS_071207_071207__1018747.ZIP

Change to the directory in which you have unzipped the archive.

bx620s4pvt5:/tmp/SuSE_10_x86-64 # cd SUSE_SLES_10_X86_64 bx620s4pvt5:/tmp/SuSE_10_x86-64/SUSE_SLES_10_X86_64

The primesetup tool is installed with the fsc-utils-n.n-nn.noarch.rpm package, which is included in the Linux driver package.

Check whether the fsc-utils-n.n-nn.noarch.rpm package is already installed on the server blade and, if so, which version it is.

bx620s4pvt5:/tmp/SuSE_10_x86-64/SUSE_SLES_10_X86_64 # rpm -qa | grep fsc fsc-utils-0.6-13

i If the server blade has Internet access, you can also download the Linux Driver Package directly into the /tmp/SuSE_10_x86-64 directory on the target system.

i The actual driver installation is done with the primesetup tool. For detailed information on primesetup, see the document fsc-linux.pdf or fsc-linux.html, which comes with the driver package.

http://www.fujitsu-siemens.com/products/standard_servers/linux_readmes_popup.html




Page: 74 of 108

If the fsc-utils-n.n-nn.noarch.rpm package is not installed, install it with the following command:

bx620s4pvt5:/tmp/SuSE_10_x86-64/SUSE_SLES_10_X86_64 # rpm -i fsc-utils-0.6-13.noarch.rpm

If the package is already installed and older than the version in the downloaded Linux driver package, install it with the following command:

bx620s4pvt5:/tmp/SuSE_10_x86-64/SUSE_SLES_10_X86_64 # rpm -U fsc-utils-0.6-13.noarch.rpm

It the installed package is newer or the same version as the one in the downloaded Linux driver package, leave it as it is.

Unzip the ...drivers.zip archive file.

bx620s4pvt5:/tmp/SuSE_10_x86-64/SUSE_SLES_10_X86_64 # unzip SuSE-SLES_10-X86_64-drivers.zip

Start primesetup and select the drivers benet and beiscsi for installation.

bx620s4pvt5:/tmp/SuSE_10_x86-64/SUSE_SLES_10_X86_64 # primesetup FSC Driver Installation Utility 0.6.13 Copyright (c) Fujitsu Siemens Computers 2002 - 2007 This software comes with NO WARRANTY. See documentation for details. Hit Ctrl-C at any time to quit the utility. Checking for FSC KMPs... Please select kernels to install drivers for: 1: [*] 2.6.16.21-0.8-smp (SuSE SLES_10, 10.1) 2: [*] 2.6.16.21-0.8-xen (SuSE SLES_10, 10.1) Select kernels (? for help): -2 1: [*] 2.6.16.21-0.8-smp (SuSE SLES_10, 10.1) 2: [ ] 2.6.16.21-0.8-xen (SuSE SLES_10, 10.1) Select kernels (? for help): ok Looking for drivers, please wait ... Detected hardware with FSC-provided drivers: ... 1: [ ] megaraid_sas 2: [*] benet 3: [ ] megasr 4: [ ] mptsas 5: [*] beiscsi 6: [ ] e1000 Select drivers (? for help): ok The following drivers will be installed: benet 1.0.707.0 for 2.6.16.46-0.12-smp (SuSE SLES_10, 10.1) beiscsi 1.0.707.0 for 2.6.16.46-0.12-smp (SuSE SLES_10, 10.1) Proceed ? (y/n) y Installation finished. Module dependencies for kernel 2.6.16.46-0.12-smp Driver installation successful. -------------------------------------------------------------------------- ! You may have to adapt your system configuration ! - Please check the following configuration files: * /etc/modprobe.conf.local (set 'alias scsi_hostadapter <module>' or 'alias ethX <module>' to enable a module) * /etc/sysconfig/kernel (set 'INITRD_MODULES="... <module> ..."' to force loading a module) - Run mkinitrd or mk_initrd to rebuild your initial RAM disk if necessary. -------------------------------------------------------------------------- FSC Driver Installation Utility finished.



Page: 75 of 108

Check that the driver packages benet.ko and beiscsi.ko are now available on the target system and load the drivers into the kernel.

bx620s4pvt5:~ # find /lib -name benet.ko –print /lib/modules/2.6.16.21-0.8-smp/kernel/drivers/net/benet.ko bx620s4pvt5:~ # cd /lib/modules/2.6.16.21-0.8-smp/kernel/drivers/net bx620s4pvt5:/lib/modules/2.6.16.21-0.8-smp/kernel/drivers/net # insmod benet.ko bx620s4pvt5:~ # find /lib -name beiscsi.ko –print /lib/modules/2.6.16.21-0.8-smp/kernel/drivers/scsi/beiscsi.ko bx620s4pvt5:~ # cd /lib/modules/2.6.16.21-0.8-smp/kernel/drivers/scsi bx620s4pvt5:/lib/modules/2.6.16.21-0.8-smp/kernel/drivers/scsi # insmod beiscsi.ko

Check the driver installation.

bx620s4pvt5:~ # lsmod Module Size Used by beiscsi 142928 0 benet 124812 0 nls_utf8 6016 1 cifs 203908 1 ipv6 244448 26 ...

You can download the programs, packages and files required for the firmware flash from the Fujitsu Siemens download portal at http://support.fujitsu-siemens.com/com/support/downloads.html. The following example shows a firmware update in which the required programs, packages and files were stored on an internal server. Firmware updates using the FSC download portal http://support.fujitsu-siemens.com/com/support/downloads.html are carried out in the same way.

s75pvt319:~ # mkdir /biss s75pvt319:~ # mount -t cifs -o user=Administrator //biss.vlan10.qalab/Treiber_and_Tools /biss Password: ** s75pvt319:~ # s75pvt319: # cd /biss/Treiber-\ BIOS-\ FW/BX600\ 10GBit\ LAN/Software/BladeEngine/*707*/Firmware s75pvt319:/biss/Treiber- BIOS- FW/BX600 10GBit LAN/Software/BladeEngine/BE_Build_1.0.707.0/CD3/Firmware # cp flash.ufi /tmp s75pvt319:/biss/Treiber- BIOS- FW/BX600 10GBit LAN/Software/BladeEngine/BE_Build_1.0.707.0/CD3/Firmware # s75pvt319:~ # cd s75pvt319:~ # cd /biss/Treiber-\ BIOS-\ FW/BX600\ 10GBit\ LAN/Software/BladeEngine/*707* s75pvt319:/biss/Treiber- BIOS- FW/BX600 10GBit LAN/Software/BladeEngine/BE_Build_1.0.707.0 # ls BE Software v1.0.707.0.pdf BE_Build_1.0.707.0.zip CD1 CD2 CD3 readme.txt s75pvt319:/biss/Treiber- BIOS- FW/BX600 10GBit LAN/Software/BladeEngine/BE_Build_1.0.707.0 # cd CD3 s75pvt319:/biss/Treiber- BIOS- FW/BX600 10GBit LAN/Software/BladeEngine/BE_Build_1.0.707.0/CD3 # ls Firmware FlashUtility ManagementRPMs SEstats WindowsSilentInstall readme.txt s75pvt319:/biss/Treiber- BIOS- FW/BX600 10GBit LAN/Software/BladeEngine/BE_Build_1.0.707.0/CD3 # cd ManagementRPMs/*32 s75pvt319:/biss/Treiber- BIOS- FW/BX600 10GBit LAN/Software/BladeEngine/BE_Build_1.0.707.0/CD3/ManagementRPMs/linux_32 # ls SEMGMT_rhel4-1.0-707.0.i386.rpm SEMGMT_sles9-1.0-707.0.i386.rpm SEMGMT_rhel5-1.0-707.0.i386.rpm SEMGMT_suse10-1.0-707.0.i386.rpm s75pvt319:/biss/Treiber- BIOS- FW/BX600 10GBit LAN/Software/BladeEngine/BE_Build_1.0.707.0/CD3/ManagementRPMs/linux_32 # rpm -i SEMGMT_suse10-1.0-707.0.i386.rpm s75pvt319:/biss/Treiber- BIOS- FW/BX600 10GBit LAN/Software/BladeEngine/BE_Build_1.0.707.0/CD3/ManagementRPMs/linux_32 # cd s75pvt319:~ # /opt/ServerEngines/BladeEngine/smclp/client/client Welcome to ServerEngines SMCLP client version 1.0.707.0 Enter host name [127.0.0.1]: Enter user name: smclp





Page: 76 of 108

Enter user 'smclp' password: smclp / -> cd BladeEngine0/ /BladeEngine0 /BladeEngine0 -> show /BladeEngine0 : Manufacturer = ServerEngines, LLC ModelNumber = ServerEngines SE-EC3210 Description = BladeEngine 10Gb iSCSI Initiator SerialNumber = 001999110aab ActiveFirmwareVersion = 1.0.676.0 FirmwareVersionOnFlash = 1.0.676.0 BIOSVersion = 1.0.676.0 BootCodeVersion = 2.0.5.0 FirmwareStatus = Ready iSCSIVersionMin = 1 iSCSIVersionMax = 1 MaxCDBLength = 16 MaxMTUSize = 8342 LDTO = 20 DefaultETO = 30 VLDCapable = Yes VLDEnabled = Yes /BladeEngine0 -> oemse OEMSEEraseConfiguration OEMSESaveConfig OEMSEUpgradeFlashROM /BladeEngine0 -> OEMSEUpgradeFlashROM -OEMSEparams UFIFileName="/tmp/flash.ufi" BladeEngine firmware has been updated. Please reboot your system for the changes to take effect. Command executed successfully /BladeEngine0 -> exit s75pvt319:~ #

5.1.4 Installing under Red Hat Linux

The drivers and software for the 10 GbE I/O Module under Red Hat Linux are installed in the same way as under SuSE Linux.

5.2 Configuring iSCSI HBA In addition to its function as an Ethernet LAN controller, the 10 GbE I/O Module can also be used as an iSCSI host bus adapter (iSCSI HBA). This allows access to storage networks based on iSCSI.

5.2.1 Overview of iSCSI Storage Networks

iSCSI (Internet SCSI) is a standard for IP-based storage networks. With this method, the SCSI data is packed into TCP/IP packets and transported via IP networks.

An iSCSI storage network consists of one or more iSCSI storage units (iSCSI targets), which are connected via an IP-based network to servers that are allowed to exchange data with the iSCSI targets.

iSCSI targets can either process SCSI commands themselves or the commands are interpreted by routers which forward them to connected storage devices. iSCSI targets can be hard disks, RAID arrays or Fibre Channel Fabrics.

To be able to exchange data with an iSCSI target, a server must have an iSCSI initiator. The iSCSI initiator is responsible for packing and unpacking the SCSI data in TCP/IP packets and for authentication towards the iSCSI target. An iSCSI initiator can be a software implementation (iSCSI software initiator). In this case, the routines required for the iSCSI protocol are processed in the server’s CPU. An iSCSI initiator can also be implemented in the hardware of a network controller, as is the case in the 10 GbE I/O Module (iSCSI HBA). In this case, the CPU does not have to handle the iSCSI protocol.

iSCSI targets and iSCSI initiators are identified by their IP addresses and special iSCSI names in IQN or EUI format. (For information on IQN and EUI formats, see http://www.ietf.org/rfc/rfc3720.txt). An example of an IQN name is iqn.2003-07.com.serverengines:chapmutual. The iSCSI names are used for authentication to ensure that only authorized servers can establish an iSCSI connection.

This document describes how to explicitly set the IP addresses and IQN names of the iSCSI initiator and the iSCSI targets during configuration of the iSCSI initiator. This information can also be obtained via a DHCP server. For further details, see the manual “10GbE LAN I/O MODUL (PCIe) 1.0 – iSCSISelect Reference Guide V1.0.748.0“.

Additional security can be achieved using CHAP (Challenge Handshake Authentication Protocol).

Using the 10 GbE I/O Module as an iSCSI HBA also allows you to set up an iSCSI target as the boot medium for the server.

You can configure the 10 GbE I/O Module as an iSCSI HBA in any of the following ways:

http://www.ietf.org/rfc/rfc3720.txt



Page: 77 of 108

! CAUTION! Decide on one of the options and use this one exclusively for configuring the iSCSI HBA.

• iSCSISelect

iSCSISelect is an Int 13h setup menu that resides in the option ROM and that allows you to make all the settings required for operating the 10 GbE I/O Module in an iSCSI environment in the boot phase of the system. For more information on iSCSISelect, see the manual “10GbE LAN I/O MODUL (PCIe) 1.0 – iSCSISelect Reference Guide V1.0.748.0“.

• SMCLP SMCLP is a command line interface that allows you to set all the iSCSI configuration parameters of the 10 GbE I/O Module from a management client during operation. For more information on SMCLP, see the manual “BladeEngine iSCSI Initiator SM-CLP Command Reference“.

• Microsoft iSCSI Software Initiator The Microsoft iSCSI Software Initiator offers a graphical user interface via which you can set all the iSCSI parameters of the 10 GbE I/O Module except the IP address.

iSCSISelect and SMCLP are installed with the driver of the 10 GbE I/O Module, see section 0 on page 66. The Microsoft iSCSI Software Initiator must be installed separately. You can download it from the Microsoft homepage.

5.2.2 Integrating Additional Storage Media via iSCSI

The easiest case is integrating iSCSI storage as additional storage media. If the operating system is installed on the local hard disk, you can install and integrate the iSCSI target at any time later.

In the following example, the required settings are made with the SMCLP console, since this does not require a system restart as is the case when using iSCSISelect.

Starting an SMCLP session on the local computer

Under Microsoft Windows

cd /Program Files/ServerEngines/BladeEngine/smclp/client/ cd /Program Files x86/ServerEngines/BladeEngine/smclp/client/ client

Enter CLI at the prompt to start an SSH session.

Log in with the user name and password you specified during the installation, see page 66.

Under Linux

/opt/ServerEngines/BladeEngine/smclp/client/client

The remaining procedure is identical for both operating systems.

Setting the IP address of the iSCSI initiator

Disable DHCP: cd /BladeEngine0/iSCSIPhysicalHBA0/Network/TCPIPConfiguration OEMSEConfigureDHCP -OEMSEparams EnableDHCP="False"

Set Manual IP: cd /BladeEngine0/iSCSIPhysicalHBA0/Network/TCPIPConfiguration OEMSEConfigureIPAddress -OEMSEparams IPAddress="172.40.2.233",SubnetMask="255.255.255.0",Gateway="172.40.2.1"

Obtaining the IP address of the iSCSI initiator via a DHCP server

You can also have the IP address of the iSCSI initiator assigned via a DHCP server. In this case, the IP address is set when the system is restarted.

cd /BladeEngine0/iSCSIPhysicalHBA0/Network/TCPIPConfiguration OEMSEConfigureDHCP -OEMSEparams EnableDHCP=”True” DHCP will be enabled after reboot.

i

To start the SMCLP session from an SMCLP management client, you must specify with the CLI call the IP address of the server with the 10 GbE I/O Module that you want to configure: CLI –s IP_address

i Um die IP-Adresse des iSCSI Initiators manuell setzen zu können, muss DHCP deaktiviert sein.



Page: 78 of 108

Setting the name of the iSCSI initiator

cd /BladeEngine0/iSCSIPhysicalHBA0/LogicalHBA0/InitiatoriSCSIName set InitiatoriSCSIName="iqn.2005-08.com:se1" show

Checking the reachability of the iSCSI target

cd /BladeEngine0/iSCSIPhysicalHBA0/Network OEMSEPing -OEMSEparams IPAddress="172.40.2.1"

Establishing a connection to a target

cd /BladeEngine0/iSCSIPhysicalHBA0/LogicalHBA0/Discovery OEMSEAddSendTargetPortal -OEMSEparams IPAddress="172.40.2.106",PortNumber="3260" cd SendTargetPortal0/ OEMSEDiscoverTargets cd ../.. show (to see discovered targets)

The available iSCSI targets that can be reached under the specified IP address are displayed.

Making the login to a specific target and the connection to the target persistent • Login/Logout Target0:

cd /BladeEngine0/iSCSIPhysicalHBA0/LogicalHBA0/Target0 OEMSELogin show (session added) OEMSELogout (if persistent, persistent must be set to “false” before logout) show (session removed) • Making Target0/Portal0 persistent (available after restart):

cd /BladeEngine0/iSCSIPhysicalHBA0/LogicalHBA0/Target0/TargetPortal0 set Persistent=”True” show • Scanning for new LUNs:

cd /BladeEngine0/iSCSIPhysicalHBA0/LogicalHBA0/Target0 OEMSERScanLUNS

Other important SMCLP commands • Showing LUNs

cd /BladeEngine0/iSCSIPhysicalHBA0/LogicalHBA0/Target0 OEMSERescanLUNS Show (LUNs added) • Enabling iSNS:

cd /BladeEngine0/iSCSIPhysicalHBA0/iSNS set iSNSEnabled=”Yes” show • Adding an iSNS server:

cd /BladeEngine0/iSCSIPhysicalHBA0/iSNS OEMSEAddiSNSServer –OEMSEparams IPAddress=”172.40.1.2”,PortNum=”3205” cd /BladeEngine0/iSCSIPhysicalHBA0/iSNS/iSNSServer0 show cd /BladeEngine0/iSCSIPhysicalHBA0/LogicalHBA0 show (to see discovered targets) • Adding an iSNS server using DHCP:

cd /BladeEngine0/iSCSIPhysicalHBA0/iSNS OEMSEDiscoveriSNSServerThruDHCP show cd /BladeEngine0/iSCSIPhysicalHBA0/iSNS/iSNSServer0 show cd /BladeEngine0/iSCSIPhysicalHBA0/LogicalHBA0 show (to see discovered targets)



Page: 79 of 108

• Deleting an iSNS server

cd /BladeEngine0/iSCSIPhysicalHBA0/iSNS set iSNSEnabled=”no” show • Disabling iSNS:

cd /BladeEngine0/iSCSIPhysicalHBA0/iSNS OEMSEDeleteiSNSServer –OEMSEparams InstanceNumber=”0”

Displaying general initiator information • FW/ BIOS version:

cd /BladeEngine0 show • Driver version:

cd /BladeEngine0/iSCSIPhysicalHBA0/Driver show

g -OEMSEparams file="c:\filename.txt" // save to path given



Page: 80 of 108

Verifying the configuration

New iSCSI LUNs are automatically integrated as disks and are shown, for example, in the Disk Management section under MS Windows.



Page: 81 of 108

5.2.3 iSCSI Booting

Since the 10 GbE I/O Module used as an iSCSI HBA can be configured in the same way as a normal Int 13 storage adapter, you can boot the system from an iSCSI target via the 10 GbE I/O Module.

You make the required settings with iSCSISelect. iSCSISelect is a setup menu which is stored in the option ROM of the 10 GbE I/O Module.

To configure the iSCSI initiator for iSCSI boot, you require the following information about your network: • Target IP address (of the target from which the system is to be booted) • Initiator IQN name • Initiator IP address • Initiator subnet mask

To set up the system for iSCSI boot, carry out the following steps:

• Enable Option ROM scan for iSCSI HBA • Configure the IQN name of the initiator • Configure the network parameters of the initiator • Run Ping to check whether the target can be reached • Add the target • Configure the target for iSCSI booting

Enabling Option ROM scan for iSCSI HBA

Make sure that option ROM scan is enabled for the 10 GbE I/O Module in the system BIOS. option ROM scan must be enabled so that you can call iSCSISelect when booting the system. You use iSCSISelect to configure the other parameters for iSCSI booting.

How to enable option ROM scan for the 10 GbE I/O Module depends on the BIOS setup of your system. The following example refers to the server blade BX620 S4.

Change to the Advanced – PCI Configuration menu. On delivery, option ROM scan is disabled for the 10 GbE I/O Module (daughter board), see figure on the left.

i

To be able to start iSCSISelect, Option ROM scan must be enabled for the 10 GbE I/O Module in the system BIOS.



Page: 82 of 108

Change the option ROM scan setting for the 10 GbE I/O Module to Enabled.

The figure on the left shows the new setting.

Press <F10> to save the changed BIOS configuration.



Page: 83 of 108

Configuring the iSCSI initiator

Restart the system and press <Ctrl><S> when the following prompt is displayed: ServerEngines iSCSI Initiator BIOS v1.0.450.0 (c) 2005-2007 ServerEngines, LLC. All Rights Reserved. (c) 1998-2005 Adaptec, Inc. All Rights Reserved. Controller #0 Base 0xFE4A0000 at Bus:02 Dev:00 Fun:00 ◄◄◄ Press <Ctrl><S> for iSCSISelect(TM)Utility ►►►

Once the system BIOS has initialized, the iSCSISelect main menu is displayed.

Open the Adapter Configuration menu.

Configuring the IQN name of the initiator

Open the Controller Properties menu.



Page: 84 of 108

Confirm the suggested IQN name for the iSCSI initiator or change it.

i

The initiator IQN name must be listed in the access control list (ACL) of the target!

Save your settings (Save).

Quit the menu with <ESC>

Configuring the network parameters of the initiator

Open the Network Properties menu.

Open the Display/Set IP Address menu.



Page: 85 of 108

Enter the network parameters of the 10 GbE I/O Module.

Save your settings (Save).

Quit the menu with <ESC> and change to the iSCSISelect main menu.

Checking the reachability of the target with Ping

Change to the Target Configuration menu.

Open the Ping Target menu.

Enter the IP address of the target.

Run the Ping command.



Page: 86 of 108

This figure shows a successful Ping.

Quit the screen (<ESC>) and change back to the Target Configuration menu.

Adding a target

Open the Add Targets menu.

Enter the following parameters for the target:

• IQN name • IP address

Keep the default values for Port Number, Persistent and Login Parameters.

i Persistent must be set to Yes for the target to be available after booting.

Change to the Login field and press <Return> to log the initiator onto the target.

This figure shows the successful login to the target.

Quit the screen (<ESC>) and change back to the Target Configuration menu.

i In the Add Targets menu, you can configure further targets as storage media for the system, in addition to the boot target.



Page: 87 of 108

Configuring the target for iSCSI boot

Open the Manage Targets menu.

This menu lists all the connected targets.

Check the target settings. For the target from which the system is to be booted, Target Persistent must be set to YES and Bootable must be set to Primary. If required, correct the settings with the <F2>, <F3> and <F4> keys.

The figure on the left shows the settings for a bootable target.

Quit iSCSISelect and continue installing the operating system on the target.



Page: 88 of 108

5.3 Installing the Operating System on an iSCSI Target

i

Depending on the installation procedure you have selected, you might need a driver disk. The required files can be found in the Fujitsu Siemens software pool at http://support.fujitsu-siemens.com/com/support/downloads.html.

5.3.1 Installing MS Windows with ServerStart

If you are installing from the ServerStart DVD, you do not require an additional driver disk.

Boot your system from the ServerStart DVD.

Select “Quick installation“.




Page: 89 of 108

The iSCSI target which you have configured as the boot medium is displayed (possibly together with other available boot media).

Select the iSCSI target as the boot disk and specify the size of the boot partition.



Page: 90 of 108

Select the operating system you want to install.



Page: 91 of 108

Enter the requested basic configuration data.



Page: 92 of 108

The rest of the installation runs automatically.

5.3.2 Native MS Windows Installation

For a MS Windows installation without ServerStart, you require a driver disk for the iSCSI HBA function of the 10 GbE I/O Module. You will find the required files in the Fujitsu Siemens software pool at http://support.fujitsu-siemens.com/com/support/downloads.html.

Boot your system from the MS Windows installation CD.

Press <F6> when you are prompted to install proprietary SCSI or RAID drivers.

Once the drivers are installed, the iSCSI target is available as an installation target for the remainder of the installation process.

! CAUTION!

Once you have finished installing the operating system, you must run the installation program for the 10 GbE I/O Module, see page 66. This ensures that all the software and firmware components have the same up-to-date version and that the SMCLP console is installed.

5.3.3 Installing Linux with ServerStart

If you are installing from the ServerStart DVD, you do not require an additional driver disk.

Boot your system from the ServerStart DVD.

Select the operating system you want to install.

As the installation partition, select the LUN on the iSCSI target that you configured previously with iSCSISelect.

The rest of the installation procedure is the same as when you install the operating system on a local medium. At the end of the installation, ServerStart displays a dialog box indicating that additional software will be installed.

Confirm this message with OK.

ServerStart will now install the SMCLP management console and update the firmware of the 10 GbE I/O Module.




Page: 93 of 108

5.3.4 Native Linux Installation

For a Linux installation without ServerStart, you require a driver disk for the iSCSI HBA function of the 10 GbE I/O Module. You will find the required files in the Fujitsu Siemens software pool at http://support.fujitsu-siemens.com/com/support/downloads.html.

The Linux installation described below refers to Red Hat Enterprise Linux 5.0 U1. Other Linux distributions are installed in the same way.

Boot your system from the Linux installation DVD.

Enter dd as the boot option to include a driver disk.




Page: 94 of 108

Confirm the use of a driver disk.



Page: 95 of 108

Specify the drive with the driver disk.



Page: 96 of 108

Confirm that no other driver disks are to be loaded.

The rest of the installation process is the same as when you install the operating system on a local partition.



Page: 97 of 108

The iSCSI target which you have configured as the boot medium is displayed (possibly together with other available boot media).

Select the LUN on the iSCSI target as the installation target and click Next to continue the installation.



Page: 98 of 108

! CAUTION!

In the case of Red Hat EL 5.0 U1 (possibly also in later versions), you must intervene in the installation process again when the Congratulations screen is displayed (see above), in order to run the setup script from the driver disk.

Open a virtual console by pressing <Ctrl><Alt><F2>.

Mount the driver disk via a USB floppy disk drive and run the setup script. You will find the mknod parameters for the USB disk drive in /proc/partitions. Look for a line with the value 1440 in the third column. Use the values from the first two columns as the last two parameters of the mknod command.

#cat /proc/partitions ... 8 16 1440 sdb ... #mknod /tmp/floppy b 8 16 #/floppy/setup

Once the operating system installation is complete, update the drivers and firmware of the 10 GbE I/O Module and install the SMCLP server and client, see page 70 (SuSE Linux) or page 76 (Red Hat Linux). This ensures that the drivers and the firmware have the same up-to-date version.



Page: 99 of 108

5.4 Environment-Specific Settings To ensure that switching occurs as quickly as possible in the event of a failure, additional settings are required in multi-path configurations and in systems which use an MSCS cluster.

5.4.1 Settings for Multi-Path Configurations under Microsoft Windows and Linux

Timeouts

The driver of the 10 GbE I/O Module features internal error handling. This is designed to provide tolerance against connection interruptions and target failures within configured time limits, so that I/O errors remain imperceptible to applications and the operating system. This failover behavior is initiated under the following conditions:

a) Interruption of the direct connection to the initiator (cable unplugged / connection failure) The firmware of the 10 GbE I/O Module detects the interrupted connection and reports it to the driver. The driver queues the I/O jobs up to a timeout of 20 seconds so that the operating system does not detect any I/O errors. This timeout is referred to as Link Down Timeout (LDTO).

b) Interruption of the connection to the target due to an error on the target or in the network connection of the target. The firmware of the 10 GbE I/O Module detects the interrupted connection and reports it to the driver. The driver queues the I/O jobs up to a timeout of 20 seconds so that the operating system does not detect any I/O errors. This timeout is referred to as Extended Timeout (ETO).

Changing timeouts under MS Windows

Open the Windows Registry.

Double-click the Registry key HKLM\System\CurrentControlSet\Services\beiscsi\Parameters\Device\ DriverParameters

Enter the following new value: ldto=0;eto=0; Note that the entry is case-sensitive and that each parameter must end with a semicolon.

This setting sets the timeout values for LDTO and ETO to 0, so that the failover mechanism of the driver of the I/O module is disabled.

To re-enable the failover mechanism of the 10 GbE I/O Module, delete the relevant value from the Registry key HKLM\System\CurrentControlSet\Services\beiscsi\Parameters\Device\DriverParameters.

For the settings to take effect, you must restart the system (in the case of a boot target) or disable and re-enable the driver of the 10 GbE I/O Module (in the case of other targets).

Changing timeouts under Linux

The following command sets the values for the Link Down Timeout (LDTO) and the Extended Timeout (ETO) to 0, so that the failover mechanism of the driver of the I/O module is disabled.

#*insmod beiscsi.ko ldto=0,eto=0

To re-enable the failover mechanism, run the command again with values > 0.

For the settings to take effect, you must restart the system (in the case of a boot target) or disable and re-enable the driver of the 10 GbE I/O Module (in the case of other targets).

Setting multi-pathing for FSC DuplexDataManager or native mode under Linux

Use the following command to configure the 10 GbE I/O Module for multi-pathing with an FSC FibreCat CX-x0 using the DuplexDataManager (DDM).

#*insmod beiscsi.ko ddm=1

For Native MultiPath (Linux native), enter the command as follows:

#*insmod beiscsi.ko ddm=0



Page: 100 of 108

5.4.2 Additional Settings for Microsoft Windows

Setting the disk I/O timeout for Windows 2003

Under Windows 2003, you can increase the timeout value for SCSI subsystems to make frequent timeouts during accesses to storage devices imperceptible to the operating system. The default value for this timeout is 20 seconds.

For MSCS cluster installations, we recommend a timeout of 60 seconds.

Open the Windows Registry.

Select the Registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Disk

Add a new DWORD value TimeOutValue.

Set the value to 60 (0x3C).

Windows 2000

To ensure error-free operation of the 10 GbE I/O Module as an iSCSI HBA under Windows 2000, Service Pack 4 and the current Microsoft operating system updates must be installed.

Windows 2003 with STORPort

To ensure error-free operation of the 10 GbE I/O Module as an iSCSI HBA under Windows 2003 with STORPort, Service Pack 2 or higher must be installed. You should also install the new hotfixes of the Microsoft Knowledge Base at regular intervals.

5.4.3 Settings for Targets from Different Manufacturers

The 10 GbE I/O Module supports the use of NetApp targets, EMC Clarrion targets and an FSC FibreCat CX-x0. For information on configuring these targets, see the relevant target documentation.

! CAUTION! Please note the certifications of the storage-device manufacturers.

NetApp

For NetApp targets, both single-path and multi-path configurations are supported. The SNetDrive multi-path implementation is independent of software support and is already used by the iSCSI HBA. Connections to a target via different IP addresses are addressed with the same IQN name and are thus also shown as one target on the SMCLP console.

EMC Clarrion

For EMC Clarrion targets, both single-path and multi-path configurations are supported. For a multi-path implementation, EMC Powerpath is used. This does not require any software support by the Microsoft Software Initiator. The different connections to a target are configured as different targets (with different IQN names) and are thus also shown as such on, for example, the SMCLP console.

FSC DDM For multi-path connections to FSC targets, the FSC DuplexDataManager (DDM) is used. This is only supported for models of the FSC FibreCAT CX series.



Page: 101 of 108

Appendix: Network Driver Performance Tuning The following section includes examples of the various configuration parameters that affect performance of the Network and TCP Offload driver for the MS Windows Server operating system.

PCI-Express Bandwidth BladeEngine is an x8 PCI-Express device. If you are using a BladeEngine-based NIC, make sure that it is installed in an x8 or x16 slot that supports x8 link. Some system boards have x8 or x16 connectors with x4 link. BladeEngine can be used in such a slot, but it will not deliver performance close to a 10Gbps line rate. Each PCI-Express lane has a signaling rate of 2.5Gbps per direction and a data rate of 2Gbps after encoding in the physical layer. A significant portion of the bandwidth is used for packet headers in the Data Link Layer and Transaction Layer, which reduces the usable bandwidth to about 1.6Gbps per lane. An x8 device will have about 12.8Gbps of bandwidth (per direction) to support a 10Gbps line rate. But if installed in an x4 slot, the bandwidth is reduced to 6.4Gbps and performance will be substantially degraded. The MS Windows Server 2008 driver will report a warning message to the system event log if the device is not installed in an x8 PCI-E slot. A device installed in an x4 slot will work but with degraded performance. You can also install the device in an x16 PCI-E slot but this will not gain any additional performance. For information on viewing the event log, see Viewing Device Event Notifications BladeEngine performance may be improved by selecting a more efficient PCI-Express packet payload size. If the system BIOS allows selecting a larger PCI-Express packet size, selecting at least a 512-byte PCIe packet payload size provides the best efficiency for PCIe data transfers.

Memory and Processor Bandwidth BladeEngine requires substantial memory bandwidth in a system to support 10Gbps data streams. TCP offload helps the memory bandwidth significantly by eliminating the data copy of receive packets, but in all modes of operation, higher memory bandwidth leads to better network performance. Most computers offer multiple distinct memory channels, which may not be enabled by default. Check the manufacturer’s documentation and BIOS options for more details regarding enabling optimal memory bandwidth features. Typically, all the DIMM slots must be populated to make use of all the memory channels. As a general rule, more DIMMs will provide better performance. For DDR2, each memory channel is typically a pair of DIMMs. Therefore (4) 512MB modules may perform substantially better than (2) 1GB memory modules, because they can make use of both memory channels. Fully buffered DDR, used on Intel systems, offers even more scalability in terms of memory channels. However, the performance difference between (2) FB-DIMMs and (8) FB-DIMMs may be several Gbps of network performance. Some servers may allow a memory mirroring, where the total memory is divided in half and each location is actually stored twice. This allows fault recovery if one memory location detects an error, but it will greatly reduce the perceived memory bandwidth of the system. Almost any desktop or low-end server has enough memory bandwidth. However, most of the memory demands come from the processor accessing the data for either packet copies in the non-offloaded networking stack or application accesses. All processor memory accesses use the front side bus (FSB). The clock speed of this bus is critical for allowing efficient memory bandwidth. Ideal systems will use a 1333MHz FSB or greater.

Microsoft Windows Server Network Driver The following section discusses ways to use various BladeEngine properties and Microsoft Windows properties to performance tune a system. You can use the following table for tuning some typical servers: Situation Fixes There are a large number of short lived TCP connections such as Web Server or email server.

Enable RSS, disable TCP offload

There are large data transfers such as File Server, Web Server with file downloads, or FTP Server.

Use TCP offload

There are large data transfers such as to a Backup Server. Enable Jumbo Packets, use TCP offload There is a small server struggling to keep up with larger servers on the network.

Disable RSS, Enable TCP offload, Enable Jumbo Packets

There is a general purpose server such as Active Directory Server, DHCP Server, or a DNS Server.

Use TCP offload, Enable RSS

Viewing Device Event Notifications The BladeEngine device driver will report information through the system event log. The events may contain useful performance tuning information, such as details regarding memory allocation problems or reduced PCI-Express link speed. To enter the Event Viewer: 1. Click Start>Run, then type eventvwr and press the OK button. 2. Select Windows Logs > System and look for events from the bendis60 source.



Page: 102 of 108

Analyzing Performance Problems You can use the Windows Performance Monitor (perfmon) to view statistics for each network device. To do this, launch the Windows Performance Monitor by clicking start > run > perfmon. Add additional statistics by right-clicking and selecting “Add Counters”. For network performance, all the counters from the following sections are useful: Network Interface, TCPv4, IPv4, and Processor. The following table lists a few statistics to use for troubleshooting performance problems: Statistics Fixes Network Interface > Packets Received Errors If this is incrementing even a small amount, a physical

problem may exist on the network, such as a lose connection or bad cable, causing CRC errors in Ethernet packets. Find and eliminate the physical problem.

Network Interface > Packets Received Discarded

If this is incrementing dramatically, the computer system may be receiving a lot of unsolicited traffic that is using network resources.

IPv4 > Fragmented Datagrams / sec If this is greater than 0, the computer system is sending or receiving IP fragments. This is a serious performance problem. For details see MTU in the section “Jumbo Packets”.

TCPv4 > Segments Retransmitted / sec TCP retransmits indicate that packets are being dropped by the receiving system or in a network switch. Reducing retransmits to 0 is ideal.

Processor > % Processor Time If CPU usage is high, try to enable all available offloads, such as TCP Offload, Checksum Offloads, and use Jumbo Packets.

Jumbo Packets The Jumbo Packet setting in the registry determines the maximum Ethernet packet size. It includes Ethernet frame header (typically 14 bytes) but excludes the trailing Cyclic Redundancy Check (CRC). The standard packet size is 1514 bytes plus a 4 byte trailing CRC. Vendors use many terms that refer to this same quantity, such as packet size, Frame size, or MTU. The Maximum Transmission Unit (MTU) is the Ethernet packet payload size. This does not include the Ethernet frame header (typically 14 bytes) or the trailing Cyclic Redundancy Check (CRC). The standard MTU is 1500 bytes, corresponding to a 1514 packet size plus a 4 byte trailing CRC. Historically, any 1514 byte frame is a standard packet, while any frame larger than 1514 is called a jumbo packet. MS Windows Server 2008 is attempting to standardize the terminology across vendors so the Jumbo Packet parameter refers to the byte size of the packet.



Page: 103 of 108

The BladeEngine MS Windows Server driver supports 1514 byte and 8188 byte Jumbo Packet values. The larger packet size will provide better throughput and CPU usage. Typically, all devices on the network, including switches, must be configured for the larger size. The drawbacks of using Jumbo packets are interoperability and increased memory usage on the server. The path MTU is the maximum MTU that can be used before IP fragmentation occurs, taking into account the MTU for the endpoints and all routers between the endpoints. You can verify the path MTU by pinging a remote target with increasing payload sizes. Eventually, the IP packet length will exceed the path MTU, and the packet will be fragmented. This can be witnessed by using a packet sniffing application, such as Ethereal, Wireshark, or Microsoft Network Monitor. IP fragmentation will degrade performance dramatically, since all fragments must be received and reassembled before delivering the network packet to the upper layer protocol (ULP). In many cases, IP fragmentation may lead to a 10x performance degradation. The MTU parameter should be modified on all systems to avoid IP fragmentation for optimal network throughput. Typical Use Cases The following are typical cases for using the MTU:

• Server interconnects are typically deployed using Jumbo frames. This is the most efficient configuration for high bandwidth server-to-server communication, such as Network Attached Storage, iSCSI, and database transactions.

• Servers connected to client systems that run desktop operating systems typically use standard 1500 byte frames. Most desktop systems do not support Jumbo packets.

• Servers that need both high performance server-to-server communication and client access may be configured with Jumbo frames with Path MTU Discovery enabled. Path MTU Discovery is enabled by default in MS Windows Server 2008, and it allows TCP connections to negotiate the optimal packet size that avoids IP fragmentation.

Flow Control BladeEngine supports IEEE 802. 3x standard flow control, which uses control packets to temporarily pause the transmission of packets between two endpoints. These control messages are point-to-point - they are not forwarded by switches or routers. You must configure both endpoints for flow control. BladeEngine can either respond to flow control packets (by temporarily pausing transmits) or send flow control PAUSE packets when the transmitter is overwhelming the system’s receive bandwidth. Flow control has limited usefulness at 10Gb network speeds because it only allows the receiver to completely turn off the incoming flow of packets – it cannot just slow down the rate. Current 10Gb devices tend to cause a dip in performance because the exceedingly fast pipeline of packets will drain before the flow control can resume sending. The ideal situation is a continuous pipeline of packets flowing between the endpoints, not a start-and-stop flow. Flow Control Examples The following situations can use flow control. However in both of the situations, the TCP protocol generally does a better job at limiting the transmit rate. The TCP protocol has advanced algorithms for controlling network congestion and maintaining a rate that both endpoints can support. Flow control may help TCP in the following situations, but other protocols such as UDP will see greater benefit.

• BladeEngine is installed in a 4x PCI-Express slot or an underpowered server system. If the PCI-Express bus doesn’t provide 10Gbps of throughput due to chipset limitations or the bus width, BladeEngine cannot maintain 10Gbps of incoming receive data. It will start dropping packets quickly. In this situation it may be beneficial to enable RX flow control in BladeEngine, and enable flow control in the attached switch for all devices. This will help backpressure (i. e. slow down) the transmitters.

• BladeEngine is transmitting to 1G devices, especially non-TCP protocol. If BladeEngine is transmitting to a 10Gb switch with attached 1G clients, it may be possible for BladeEngine to overwhelm the switch. The switch will be forced to start dropping packets since it may receive a 10Gbps stream, but the client can only sink a 1Gbps stream. In this situation, it may be beneficial to enable TX flow control in BladeEngine, and enable flow control for the 10Gb switch port.

For information on modifying the Flow Control parameters, see Modifying Advanced Properties

Checksum Offloading and Large Send Offloading (LSO) BladeEngine supports IP, TCP, and UDP checksum offloading. All these protocols are enabled by default, but you can disable offloading through the Windows Device Manager Advanced Properties. Disabling checksum offloading is only useful for packet sniffing applications, such as Ethereal or Microsoft Network Monitor, on the local system where BladeEngine is installed and being monitored. When sniffing packets, transmit packets may appear to have incorrect checksums since the hardware has not calculated it yet. For information on modifying the Checksum Offload parameter, see Modifying Advanced Properties. BladeEngine supports transmit Large Send Offloading (LSO), Bladewhich allows the TCP stack to send one large block of data and the hardware segments it into multiple TCP packets. This is recommended for performance, but it can be disabled for packet sniffing applications. LSO sends will appear like giant packets in the packet sniffer, since the hardware has not segmented it yet. For information on modifying the Large Send Offload parameter, see Modifying Advanced Properties.

Receive Side Scaling (RSS) for Non-Offloaded IP/TCP Network Traffic BladeEngine can process TCP receive packets on four processors in parallel. This is ideal for applications that are CPU limited. Typically, these applications have numerous client TCP connections that may be short-lived. Web Servers or Database Servers are prime examples. RSS typically increases the number of transactions per second for these applications.



Page: 104 of 108

Understanding RSS To better understand RSS, it helps to understand the interrupt mechanism used in the network driver. Without RSS a network driver will receive an interrupt when a network packet arrives. This interrupt may occur on any CPU, or it may be limited to a set of CPUs for a given device depending on the server architecture. The network driver will launch one deferred procedure call (DPC) that will run on the same CPU as the interrupt. Only one DPC will ever run at a given moment in time. In contrast, with RSS enabled, the BladeEngine network driver will launch up to four parallel DPCs on four different CPUs. For example, if you have a four processor server that interrupts all processors, without RSS the DPC will jump from CPU to CPU, but it only ever runs on 1 CPU at a time. Each processor will only be busy 25 percent of the time. The total reported CPU usage of the system will be about 25 percent (perhaps more if other applications are also using the CPU). This is a sign that RSS may help performance. If the same four processor server uses RSS, there will be four parallel executing DPCs; one on each processor. The total CPU available for networking processing is increased from 25 percent to 100 percent. Some server machines and some network traffic profiles will not benefit from RSS. Since the non-offloaded TCP stack includes a data copy during receive processing, it is possible that memory bandwidth will limit performance before the CPU. In this situation, the CPU usage will be very high while all processors wait for memory accesses. To relieve this problem, you may reduce the number of RSS CPUs, or disable it entirely. Poor RSS behavior is only typical of network performance testing applications that receive data but perform no other processing. For other applications, RSS allows the application to scale other processing tasks across all CPUs and will improve overall performance. RSS offers the most benefit for applications that create numerous, short lived connections. These applications are typically CPU limited, instead of network bandwidth limited. For information on modifying the RSS Queues parameter, see Modifying Advanced Properties.

i

Microsoft currently does not schedule RSS processing on all hyperthreaded CPUs. Example: Only CPU 1 and 3 have RSS queues on a dual-core, hyperthreaded CPU. In contrast, a quad-core CPU has RSS queues on all four cores. The current Microsoft RSS model requires four consecutive CPUs for RSS queues, so a hyperthreaded system will never use all four RSS queues.

TCP Offloading TCP Offload is supported by MS Windows Server 2008. To monitor TCP offloads, you can type netstat –t in a command window. When this command is used, this program will indicate the offload state for each TCP connection of the system.

i

Packet sniffing applications such as Ethereal or Microsoft Network Monitor, do not see TCP offloaded packets.

MS Windows Server 2008 allows TCP offloads in more scenarios than MS Windows Server 2003. In particular, TCP offloads may occur with the Windows Firewall enabled. The following table displays common reasons why TCP offloads do not occur and suggested fixes:



Page: 105 of 108

Reasons for No TCP Offload Fixes Chimney is disabled on the system. To determine if Chimney is enabled or disabled, use

netsh interface tcp show global. To enable Chimney, use netsh interface tcp set global chimney=enabled. To disable Chimney, use netsh interface tcp set global chimney=disabled.

Offloads are disabled for specific ports or applications with netsh

Use the following commands to view any TCP ports or applications that may be configured to disable TCP offload: netsh interface tcp show chimneyports netsh interface tcp show chimneyapplications

A third-party firewall is running. Windows Firewall does not affect TCP offload, but third-party firewalls may prevent TCP offloads. Uninstall third-party firewall software to allow TCP offloads.

In the network properties, some intermediate drivers will prevent offloading.

Go to Network Connections>Properties, and uncheck boxes for unused drivers. In particular, “Network Load Balancing” and some third-party drivers will prevent offloads.

IPSec is enabled. Disable IPSec to use TCP offloading. IP Network Address Translation is enabled. Disable IP Network Address Translation. ServerEngines supports an Advanced Property to disable TCP offloading.

Make sure TCP offloading is enabled.

The TCP connection is using IPv6. BladeEngine only supports offloading TCP connections with IPv4.

TCP Offload Performance BladeEngine supports TCP offload, which provides significant performance improvements. These performance improvements can be:

• A zero copy receive data path exists. In contrast, all non-offloaded TCP packets are copied in the network stack. This copy dramatically increases the memory bandwidth and CPU requirements for receive data.

• Sending and receiving of ACK packets is handled entirely in hardware, reducing PCI Express bus usage and interrupts.

• TCP timers are implemented in hardware, including delayed ACK, push, retransmit, keep alive, etc. This reduces host CPU usage.

• Retransmits are handled entirely in hardware.

• Packetizing data, including segmenting, checksums, and CRC. This allows the network driver to receive huge send and receive buffers (>1MB) for maximum efficiency.

• BladeEngine drivers provide efficient parallel processing of multiple connections TCP on multiple CPU systems.

The BladeEngine receive path is zero copy for applications that prepost receive buffers. In other words, the application must issue the socket read before the data arrives. Ideal applications use Microsoft’s Winsock2 overlapped IO API, which allows posting multiple receive buffers with asynchronous completions. Applications that do not prepost buffers may incur the penalty of the data copy, and the performance improvement will be significantly less noticeable. Applications that transmit large amounts of data will show excellent CPU efficiency using TCP offload. TCP offload allows the network driver to accept large buffers of data to transmit. Each buffer is roughly the same amount of processing work as a single TCP packet for non-offloaded traffic. The entire process of packetizing the data, processing the incoming data acknowledgements, and potentially retransmitting any lost data is handled in hardware. TCP Offload Exclusions Microsoft provides a method to exclude certain applications from being offloaded to BladeEngine. There are certain types of applications that do not benefit that much from TCP offload. These include TCP connections that are short-lived, transfer small amounts of data at a time, exhibit fragmentation from end-to-end, or make use of IP options. If an application sends data less than the Max Segment Size (MSS), BladeEngine, like most TCP stacks, uses a Nagling algorithm. Nagling reduces the number of TCP packets on the network by combining small application sends into one larger TCP packet. Nagling typically reduces the performance of a single connection to allow greater overall performance for a large group of connections. During Nagling, a single connection may have long pauses (200ms) between sending subsequent packets as BladeEngine waits for more data from the application to append onto the packet. An application may disable Nagling using the TCP_NO_DELAY option. TCP offload does not improve the performance for connections that Nagle, since the performance is intentionally limited by the Nagling algorithm. Telnet and SSH consoles are examples of connections that typically use Nagling.



Page: 106 of 108

MS Windows Server 2008 has not optimized the connection offload path. Some applications that use numerous short lived TCP connections will not show a performance improvement using TCP offload. MS Windows Server 2008 provides control over the applications and TCP ports that are eligible for TCP offload using the netsh tool. Refer to the Microsoft documentation for these netsh commands: netsh interface tcp add chimneyapplication state=disabled application=<path> netsh interface tcp add chimneyport state=disabled remoteport=23 localport=*

i

The netsh commands require the Windows Firewall to be running. If the Firewall is disabled, all applications and ports added with the netsh commands will fail to connect.

Microsoft Windows TCP Parameters ServerEngines does not recommend modifying the TCP registry parameters provided by Microsoft, such as TcpAckFrequency. The default parameters are suitable for a wide variety of situations, with or without using TCP offloading. Support for Selective Acknowledgement in Offloaded TCP Connections Selective Acknowledgment (SACK) is an optional TCP feature described in RFC 2018, which is supported by MS Windows Server 2008 and by BladeEngine for offloaded connections. The use of selective acknowledgments may provide a slight throughput increase in some network use cases due to a reduction of retransmission traffic. For 10Gb links this will generally be less than 5 percent. SACK offload is disabled by default. To enable this support and for information on modifying the SACK parameter, see Modifying Advanced Properties.

Receive Window Auto Tuning and Compound TCP MS Windows Server 2008 adds several features to the host TCP stack, such as receive window autotuning and Compound TCP (CTCP). These features only affect non-offloaded TCP traffic. Some 10Gbps stress applications may actually suffer in performance with these features enabled. In particular, ServerEngines has seen some bi-directional data stream test performance degradation when the receive window auto-tuning is enabled. This is due to increased receive performance that adversely affects the same TCP connection’s transmit performance. To disable these features, use these commands:

netsh interface tcp set global autotuning=disabled

netsh interface tcp set global congestionprovider=none

Interrupt Coalescing The MS Windows Server network driver automatically performs adaptive interrupt coalescing. During periods of low network usage the interrupt delay is set to a minimum for lower latency. As the interrupt rate increases, the delay is increased. This allows the driver to perform more work in a single interrupt, which reduces the amount of wasted cycles from additional interrupts. The interrupt coalescing algorithm automatically tunes the system to maintain responsiveness and performance in a wide variety of situations, including RSS and TOE traffic. The interrupt coalescing algorithm cannot be modified through Advanced Properties.

CPU Binding Considerations MS Windows applications may set a processor affinity, which binds a program to a particular CPU in a multiple processor computer. You can also assign device interrupts to a particular processor, using an interrupt affinity filter driver, such as IntFilter. However, with the recent additions to the Windows networking stack, manually configuring CPU affinity is not recommended. BladeEngine uses multiple parallel DPCs that are explicitly assigned to particular CPUs for processing both RSS and TCP offloading tasks. Each TCP connection is assigned to a particular CPU for processing. This provides the advantage of interrupt filtering, increasing CPU cache hits, without any user configuration. Interrupt filtering is unnecessary in this situation since the bulk of the processing occurs on a different CPU than the interrupt. The advantage of application affinity for network applications revolves around choosing the ideal relationship between the DPC and application affinity. The ideal mapping may require that both the DPC and application run on the same processor, different processors, or different cores of a dual core processor that shares a common memory cache. Even when the best affinity relationship is determined, it is impossible to enforce this relationship since RSS or TCP offloading choose the DPC processor. The only reason to experiment with application and interrupt CPU affinity is when performing isolated networking benchmarks.

Single TCP Connection Performance Settings One common benchmark is to run a single TCP connection between two computers as fast as possible. The following are a few suggestions to deliver the best possible performance:

• Use TCP Window Scaling with a 256k or 512k window. This may be controlled with show socket applications, such as ntttcp from Microsoft.

• Use large send and receive buffers (>=128k) with an efficient application such as ntttcp.

• Disable RSS and use an interrupt filter driver as discussed in “CPU Binding Considerations”. Experiment with all relative CPU affinities to find the best combination.

• Disable timestamps and SACK since the test should run without dropping any packets.

• Unbind unused network protocols in the “Network Connections” property page.

• Disable any firewall services, IPSEC, or network address translation (NAT).



Page: 107 of 108

Appendix: Known Issues Known Issues – NIC/TOE

The following describes known issues seen in validation of this release of the BladeEngine Software for NIC/TOE.

1. If dynamic partitioning is supported by the system that is hosting a BladeEngine controller, and a processor is dynamically added to the running system (referred to as a "hot processor add") running Server 2008, the Receive Side Scaling (RSS) CPU indirection table is not updated by NDIS. The implication of this problem is that RSS cannot take advantage of any additional processors added to such a system if the "hot processoradd" feature is used. This is a known problem with the Microsoft Operating System. (9982)

2. With SuSE Guest OS, DHCP does not always work. The reason for this is that with vmxnet emulation, SuSe sometimes creates the sysconfig configuration file under /etc/sysconfig/network/ with the name ifcfg-eth-bus-pci-<bus#-slot#> instead of the usual name format ifcfg-eth-id-<mac-address>. The work around for this problem is to rename the configuration file. Follow these steps:

Delete all network cards configuration using yast, then save and exit.

Configure all network cards again using yast, then save and exit.

Rename the ifcfg-eth file corresponding to the vmxnet interface to ifcfg-eth-id-<mac-address>, where <mac-address> is the MAC address for the eth interface as shown by the ifconfig command (e.g., ifcfg-eth-id-00:0c:29:51:83:cd).

3. VI shows only e1000 as the emulation driver with some Guest OSes. (SuSE 10, Win2k8) and there is no way to choose vmxnet as the emulation driver. The work around for this problem is to choose vmxnet by editing the .vmx file for this VM. In the file /vmfs/volumes/*/<VM-NAME>/<VM-NAME>.vmx corresponding to the VM, replace the line: ethernet<N>.virtualDev = "e1000"

with ethernet0.features = "15"

where <N> is the interface number corresponding to the vmxnet emulation on the driver you want to configure. It may be necessary to reboot after this procedure.



Page: 108 of 108

Known issues – iSCSI

The following describes known issues seen in validation of this release of the BladeEngine Software for iSCSI.

1. During the installation of BladeEngine software on ESX 3.5, a four minute delay is encountered with ESX 3.5 Update 1. This delay is caused by ESX's Pegasus CIM provider. This delay is not seen when loading BladeEngine drivers with base ESX 3.5. (10132)

2. Under Microsoft Windows Server 2008, repeated log-in / log-out to multiple targets can cause a given LUN to appear more than once in Microsoft's Disk Manager. Both target instances in the disk manager refer to the same iSCSI LUN. Fortunately Disk Manager will only allow one of the instances to be brought on-line at a time. There are no other side effects or symptoms for this problem. (9847)

3. If the BladeEngine software installation utility is used to upgrade drivers for an iSCSI boot configuration, a second reboot may be required. This happens intermittently and the second reboot is requested only after running MS Windows device manager. (10037)

4. The BladeEngine installation utility installs an SSH client and creates an SM-CLP user for MS Windows 2008. However, Windows 2008 by default does not allow write permissions on all folders in the system. In order to use SM-CLP commands that will create a file on the system (e.g., SaveConfig) you will need to enable write permissions for the SSH user. (10162)

5. BladeEngine iSCSI drivers, firmware and iSCSISelect will always present an iSCSI target's LUN list in the order provided by the iSCSI target. Some iSCSI target implementations do not necessarily sort the LUN list by LUN numbers. This is an important consideration when configuring to boot from iSCSI so always double check the LUN number. (10399)

6. When SM-CLP Client is running on a Solaris system, it is running with root privileges. When you run the Client from a remote machine using telnet/ssh2, it is running with smclp user privileges and does not have write access to create a file in the cpe directory (/opt/…../cpe). One work around for this problem is to run the following command:

oemsesaveconfig –oemseparams File=”/tmp/abc.text”

since /tmp is world-writable.

Another work around for this problem is to have your administrator give you write access to the cpe directory. (10437)

7. In Solaris, the uninstaller does not remove the beiscsi driver with a target connected. Even after a reboot, the beiscsi driver will still be uninstalled. Since an iSCSI boot cannot be detected from a local boot, the driver will not be removed as fixing this behavior may cause unexpected loss of boot drive. (10527)

8. In Microsoft Windows Server 2008, the BladeEngine software installation utility installs COPSSH. However, there is an issue with COPSSH. If uninstalled independently, it will not remove the shortcuts in the Start->Program menu. These shortcuts will be cleaned up when BladeEngine software is uninstalled.

Information on this document On April 1, 2009, Fujitsu became the sole owner of Fujitsu Siemens Compu-ters. This new subsidiary of Fujitsu has been renamed Fujitsu Technology So-lutions.

This document from the document archive refers to a product version which was released a considerable time ago or which is no longer marketed.

Please note that all company references and copyrights in this document have been legally transferred to Fujitsu Technology Solutions.

Contact and support addresses will now be offered by Fujitsu Technology So-lutions and have the format …@ts.fujitsu.com.

The Internet pages of Fujitsu Technology Solutions are available at http://ts.fujitsu.com/... and the user documentation at http://manuals.ts.fujitsu.com.

Copyright Fujitsu Technology Solutions, 2009

Hinweise zum vorliegenden Dokument Zum 1. April 2009 ist Fujitsu Siemens Computers in den alleinigen Besitz von Fujitsu übergegangen. Diese neue Tochtergesellschaft von Fujitsu trägt seit-dem den Namen Fujitsu Technology Solutions.

Das vorliegende Dokument aus dem Dokumentenarchiv bezieht sich auf eine bereits vor längerer Zeit freigegebene oder nicht mehr im Vertrieb befindliche Produktversion.

Bitte beachten Sie, dass alle Firmenbezüge und Copyrights im vorliegenden Dokument rechtlich auf Fujitsu Technology Solutions übergegangen sind.

Kontakt- und Supportadressen werden nun von Fujitsu Technology Solutions angeboten und haben die Form …@ts.fujitsu.com.

Die Internetseiten von Fujitsu Technology Solutions finden Sie unter http://de.ts.fujitsu.com/..., und unter http://manuals.ts.fujitsu.com finden Sie die Benutzerdokumentation.

Copyright Fujitsu Technology Solutions, 2009

Documents

PRIMERGY BX600 10 GbE Switch Blade 10/2 and 10 …manuals.ts.fujitsu.com/file/3632/bx600-10gbe-wp-en.pdfEdition January 2009 PRIMERGY BX600 10 GbE Switch Blade 10/2 and 10 GbE LAN