NetXen, Inc. - Carnegie Mellon University€¦ · NetXen, Inc. Intelligent NIC User’s Guide - Revision 4.0 5 3.2 Windows NIC Driver Installation Table 6 provides quick-start instructions

NetXen, Inc.

NX3031 Intelligent NIC® User’s Guide

Version 4.0

January 7, 2009

NetXen, Inc.18922 Forge DriveCupertino, CA 95014-0701Telephone: (408) 861-4500Web Site: http://www.netxen.com

Copyright© Copyright 2009. NetXen, Inc. — All Rights Reserved.

DisclaimerThis is a preliminary document and is subject to change without prior notification and does not represent a commitment on the part of NetXen, Inc. The information contained herein is the proprietary and confidential information of NetXen or its licensors, and is supplied subject to, and may be used only by NetXen’s customer in accordance with, a written agreement between NetXen and its customer. Except as may be explicitly set forth in such agreement, NetXen does not make, and expressly disclaims, any representations or warranties as to the completeness, accuracy or usefulness of the information contained in this document.

Please contact NetXen, Inc. to obtain the latest revision of this document.

Trademarks Intelligent NIC is a registered trademark of NetXen, Inc. NetXen and the NetXen logo are trademarks of NetXen, Inc. Third Party brands and names are the property of their respective owners.

Revision History

Version Date Editor Notes

4.0 1/07/09 SR Initial release of document.

NetXen, Inc

Table of Contents

1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Obtaining the Latest Software Release Information . . . . . . . . . . . . . . . . . . 11.2 Related Documents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Document Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4 NetXen Intelligent NIC Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 System Requirements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.1 Hardware and Software Installation Requirements . . . . . . . . . . . . . . . . . . . 3

2.1.1 Cable Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.1.2 Optical Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.1.3 System and Software Requirements . . . . . . . . . . . . . . . . . . . . . . 3

3 Quick-Start Instructions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43.1 NetXen NIC Hardware Installation and Verification. . . . . . . . . . . . . . . . . . . 43.2 Windows NIC Driver Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53.3 Linux NIC Driver Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

4 Intelligent NIC Hardware Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74.1 NX3-4GBT Quad GbE Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

4.1.1 Ethernet Port Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74.1.2 Status LEDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84.1.3 Quad-GbE Bracket Removal and Replacement . . . . . . . . . . . . . . 8

4.2 NX3-20Gxx 10G SFP+ NIC Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 Windows Driver Installation Guidelines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

5.1 Windows Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105.1.1 Windows Service Pack Requirements . . . . . . . . . . . . . . . . . . . . 105.1.2 Dynamic Firmware Location . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

5.2 Linux Checklist and Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115.2.1 Linux Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115.2.2 Linux Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

6 Windows Driver Installation and Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . 136.1 Windows NIC Driver Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136.2 Windows Driver Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176.3 Configuring the NetXen NIC Card in Windows . . . . . . . . . . . . . . . . . . . . . 246.4 Uninstalling the NIC Driver in Windows. . . . . . . . . . . . . . . . . . . . . . . . . . . 266.5 Windows Diagnostic Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266.6 Windows Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

6.6.1 Steps for NIC Performance Testing using ntttcp on Windows . . 276.6.2 ntttcp Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286.6.3 ntttcps and ntttcpr Command Line Options . . . . . . . . . . . . . . . . 296.6.4 Tcp1323Opts Parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296.6.5 Set TcpWindowSize . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

7 Linux Installation and Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337.1 NetXen GPL Driver and Blacklisting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337.2 Preparing for NIC Driver Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337.3 NIC Driver Installation (RPM Format) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347.4 Linux NIC Driver Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

7.4.1 Ethtool Utility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357.5 Configuring the NetXen NIC on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367.6 Uninstalling the NetXen NIC Driver in Linux . . . . . . . . . . . . . . . . . . . . . . . 36

NetXen, Inc. i

Table of Contents

7.7 Linux Diagnostic Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367.8 Linux Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

7.8.1 Linux Performance Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377.8.2 Linux Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

8 Linux Configuration for Bonding Mode and Virtual LAN (VLAN). . . . . . . . . . . . . . 388.1 Linux Bonding Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

8.1.1 Active-Backup or Mode=1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 388.1.2 Round-Robin or Mode=0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398.1.3 Bonding Driver Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408.1.4 Bonding Driver Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

8.2 Configuring Virtual LAN (VLAN) under Linux . . . . . . . . . . . . . . . . . . . . . . 419 Firmware Update. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

9.1 Windows Dynamic Firmware Download . . . . . . . . . . . . . . . . . . . . . . . . . . 439.2 Linux Dynamic Firmware Download . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

10 Flash Update. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4410.1 Flash Update and Tools Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4410.2 Flash Update Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

10.2.1 nxflash Command. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4410.2.2 LiveCD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

10.3 Windows Flash Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4510.3.1 Using the NetXen Flash Utility in Windows. . . . . . . . . . . . . . . . . 4510.3.2 Windows Flash Update Procedure . . . . . . . . . . . . . . . . . . . . . . . 4610.3.3 Updating the Flash on Multiple Standalone Cards in Windows . 47

10.4 Linux Flash Update . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4810.4.1 Using the NetXen Flash Utility in Linux . . . . . . . . . . . . . . . . . . . 4810.4.2 Linux Flash Update Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . 49

11 Troubleshooting Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5011.1 Unable to Boot the Host System with the NIC Card Installed . . . . . . . . . . 5011.2 Unable to Send/Receive Network Traffic Using the NIC . . . . . . . . . . . . . . 5011.3 Gather System Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5111.4 Check for System Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5111.5 Caveats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

11.5.1 MSI-X Interrupts for Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5211.5.2 PCI-Express Slot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

11.6 Windows Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5311.6.1 Win2k3 RSS and SNP Support . . . . . . . . . . . . . . . . . . . . . . . . . 53

12 Quick Briefs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5412.1 Linux PXE Installation Requirements and Configuration. . . . . . . . . . . . . . 54

12.1.1 PXE Server Side Requirements . . . . . . . . . . . . . . . . . . . . . . . . . 5412.1.2 Validated in This Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5412.1.3 PXE Installation and Configuration. . . . . . . . . . . . . . . . . . . . . . . 55

12.2 iSCSI Installation Requirements and Configuration . . . . . . . . . . . . . . . . . 5512.2.1 iSCSI Boot Host Requirements . . . . . . . . . . . . . . . . . . . . . . . . . 5512.2.2 iSCSI Boot Client Requirements . . . . . . . . . . . . . . . . . . . . . . . . 5512.2.3 Validated in This Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5612.2.4 Main Steps Required . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

Appendix A Diagnostic Utility Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

ii NetXen, Inc.

1. Overview

1 OverviewThis guide describes the system requirements, installation, and set-up required for the NetXen Intelligent NIC® network interface card product release. This document should be used in conjunction with the Release Notes document.

1.1 Obtaining the Latest Software Release InformationRefer to the Release Notes document for more information on the software release package contents. The Release Notes document contains:

Software release package contents

Current driver and firmware release version

Known issues

Additional system software requirements

To download the latest version of the Release Notes for this product, refer to the NetXen web site at http://www.netxen.com/support/downloads/drivers_software.asp.

1.2 Related DocumentsRelease notes included in the software release.

Linux ‘man’ pages:

% man nx_nic

nxudiag utility help (diagnostics)

nxudiag -h or nxudiag --help (Windows)

% ./nxudiag -h or nxudiag --help (Linux)

Check NetXen web site for updates.

1.3 Document ConventionsThis document uses the following conventions:

The ‘%’ character is used to indicate the cursor prompt when executing Linux commands.

Actual commands are denoted in italics.

The call out <current release> is used to refer to the current release of the software. For example, release 4.0.222. Check the Release Notes document for the most up-to-date release version.

NetXen, Inc. Intelligent NIC User’s Guide - Revision 4.0 1

1. Overview

1.4 NetXen Intelligent NIC ProductsThe NIC products listed below support Windows, Linux, and other operating systems. Check the Release Notes document for updates to this table.

Table 1 NetXen Intelligent NIC Product Offerings

Product Number Product Definition Transfer Medium

NX3-4GBT Quad-port Gigabit Ethernet Adapter Copper

NX3-20GCU Dual-port 10-Gigabit Ethernet Adapter Twinax

NX3-20GSR Dual-port 10-Gigabit Ethernet Adapter (short range fiber)

Optical

NX3-20GLR Dual-port 10-Gigabit Ethernet Adapter (long range fiber)

Optical

NX3-20GLM1

1. Available Q1, 2009.

Dual-port 10-Gigabit Ethernet Adapter (legacy multimode fiber)

Optical

2 Intelligent NIC User’s Guide - Revision 4.0 NetXen, Inc.

2. System Requirements

2 System RequirementsBefore installing the product, verify the minimum system configuration requirements outlined below.

2.1 Hardware and Software Installation Requirements

2.1.1 Cable Specifications

2.1.2 Optical Modules

2.1.3 System and Software RequirementsRefer to the Release Notes document for more information on the particular system and software requirements for the current software release.

Table 2 Cable Specifications — Optical

Product Core Diameter (μm) Optical Mode Wavelength

(nm)Bandwidth

(MHz) Distance (m)

NX3-20GSR 50 OM2 850 400 66

500 82

OM3 2000 300

62.5 OM1 160 26

200 33

NX3-20GLM 50 OM2 1310 400 100

500 220

62.5 OM1 500 220

NX3-20GLR 9 n/a 1310 n/a 10,000

10 n/a n/a 10,000

Table 3 Cable Specifications — Copper

Product Cable Type Network Type Cable Length (meters)

NX3_4GBT Cat5 LAN 100m

NX3_20GCU Copper direct attach twinax LAN 10m

Table 4 SFP+ Optical Module Specifications

Manufacturer Name Part Number

JDS Uniphase Accelar™ RoHS-Compliant 10GBE 850 nm eSFP Linear Transceiver

PLRXPL-SC-S43-51-N

Finisar Corporation RoHS-6 Compliant 10Gb/s 850nm Multimode Datacom SFP+ Transceiver

FTLX8571D3BCL


3. Quick-Start Instructions

3 Quick-Start InstructionsThis section provides quick-start instructions for users already familiar with hardware and software installation procedures. Please refer to the appropriate subsection as follows:

Section 3.1, "NetXen NIC Hardware Installation and Verification" on page 4

Section 3.2, "Windows NIC Driver Installation" on page 5

Section 3.3, "Linux NIC Driver Installation" on page 6

3.1 NetXen NIC Hardware Installation and VerificationTable 5 shows the quick-start instructions for the NetXen NIC hardware installation and verification.

Table 5 NetXen NIC Hardware Installation and Verification

Pre-Installation Checklist1. Verify the system meets the requirements described in Section 2 and

the Release Notes.2. Power down the system.Installing the NetXen NIC Adapter1. Insert the adapter into an available PCI Express slot.2. Connect the appropriate cables to the adapter.3. Power on the system.Verify NIC Adapter InstallationAfter the host has been powered up, verify that the adapter is visible on the bus.

On Linux-based systems, execute the following command:% lspci -vvv -d 4040

On Windows-based systems, execute the following sequence:1. Right-click on My Computer2. Select Manage;3. Select Device Manager;4. Check under Other devices for an unknown device;5. Right-click and select Properties for the unknown device;6. Select the Details tab and look for VEN_4040 as part of the string.



3.2 Windows NIC Driver InstallationTable 6 provides quick-start instructions for the Windows driver installation.If the driver package has not been installed, download the driver package as described in Section 1.1 above and copy the files to the host system.

Table 6 Windows NIC Driver Installation Procedure

Driver Installation1

1. From the Device Manager, scan for new hardware.2. When prompted to connect to Windows Update, Click No, not at this time, then

click Next.3. In the next window, click Install from a list of specific location and then click Next.4. Select Search for the best driver in these locations. Select Include this location

in the search. 5. Provide the path to the directory where the driver <file name>2 is located.6. Double-click on My Computer.7. Navigate to the following directory path.

C:\WINDOWS\system32\drivers8. Locate the following three files in this directory:

nx3fwct.binnx3fwmn.bin<NetXen driver>2

NIC Card Configuration1. Open the Network Connections window from the Control Panel.2. Verify that NetXen NIC adapter icon says Connected. 3. Assign an IP address to the network interface.

1. Refer to Section 5.1 of this document for more information on Windows pre-installation guidelines.

2. nxnd5x86.sys — Windows 2003, 32-bit OS nxnd5x64.sys — Windows 2003, 64-bit OS nxnd6x86.sys — Windows 2008, 32-bit OS nxnd6x64.sys — Windows 2008, 64-bit OS



3.3 Linux NIC Driver InstallationTable 7 provides quick-start instructions for the Linux driver installation.If the driver package has not been installed, download the driver package as described in Section 1.1 above and copy the files to the host system.

Table 7 Linux NIC Driver Installation Procedure

Driver Installation1 (rpm)Redhat:1. Install the RPM source:

rpm -iv nx_nic-[version].src.rpm 2. Build the binary RPM:

rpmbuild -bb /usr/src/redhat/ SPECS nx_nic-[version].spec

3. Install the binary RPM: rpm -ivh /usr/src/redhat/RPMS [arch]2/nx_nic-[version].[arch].rpm

4. Install the tools RPM: rpm -ivh nx_tools-[version].rpm5. Load the driver:

modprobe -v nx_nicSUSE:1. Install the RPM source:

rpm -iv nx_nic-[version].src.rpm 2. Build the binary RPM:

rpmbuild -bb /usr/src/packages/ SPECS/nx_nic-[version].spec

3. Install the binary RPM: rpm -ivh /usr/src/packages/RPMS/ [arch]2/nx_nic-[version].[arch].rpm

4. Install the tools RPM: rpm -ivh nx_tools-[version].rpm5. Load the driver:

modprobe firmware_class modprobe -v nx_nic

NIC Card Configuration1. Identify the NetXen Ethernet interface:

cat /proc/net/nx_nic/dev0 | grep eth2. Assign an IP address:

ifconfig eth[n] [ip address]

1. Refer to Section 5.2 of this document for more information on Linux pre-installation guidelines.

2. The [arch] variable above designates the system architecture. Some [arch] variables are i386, i586, i686, x86_64, or others depending on the system.


4. Intelligent NIC Hardware Installation

4 Intelligent NIC Hardware InstallationThe following procedures are used to install the NetXen NX3-4GBT Quad GbE and NX3-20Gxx SFP+ adapters. Please follows all applicable ESD precautions when handling the NetXen NIC adapter.

4.1 NX3-4GBT Quad GbE InstallationThe following procedure is used to install the NetXen NX3-4GBT Quad-GbE NIC.

1 Verify that the system is powered off and unplug power cord.

2 Remove the system cover.3 Locate an empty x4 or higher PCIe slot.4 Insert the adapter into the x4 or higher PCIe slot. Apply uniform pressure on top of the

board until it is firmly seated in the PCIe bus connector.5 Secure the bracket to the chassis and replace the cover.6 Connect the appropriate cables to the NIC.7 Power on the system.The NX3-4GBT quad-GbE NIC adapter contains four ports that are organized as shown in Figure 1. Note that the portion of the bracket covering the four RJ45 jacks is not shown.

Figure 1 NX3-4GBT Quad-GbE Port Assignments

4.1.1 Ethernet Port MappingThe mapping of port numbers on the card is shown in Table 8.

Table 8 Mapping of Port Number, PCI Function, and Ethernet Interface

Port Number PCI Function Ethernet Interface

1 0 Ethernet interface numbers are dynamically allocated by the host. As such, any eth[n] interface can be connected to any port.

2 1

3 2

4 3

RJ45

RJ45

RJ45

RJ45

Port 1

Port 2

Port 3

Port 4



4.1.2 Status LEDsThe NX3-4GBT NIC has two status LED’s per port. These LEDs are located on the actual port itself as shown in Figure 2.

Figure 2 NX3-4GBT Status LEDs - Side View

4.1.3 Quad-GbE Bracket Removal and ReplacementDepending on the type of system, installation of the NX3-4GBT quad-GbE adapter card will require either a long bracket or a short bracket. NetXen provides both of these brackets with the product. The brackets for the NX3-4GBT boards are different from previous brackets in that they slide over the top of the four RJ45 Ethernet jacks. The bracket removal and replacement procedure is as follows:

1 Remove the screw at the top of the board that holds the bracket in place.

2 The bracket covers the top of the RJ45 Ethernet jacks. Slide the bracket upward towards the top of the board until it clears all of the Ethernet jacks.

3 To replace with a different bracket, reverse the process, sliding the new bracket over the Ethernet jacks. Then fasten in place with the screw.

Figure 3 Quad-GbE Bracket Removal and Replacement

Port 1 Port 2 Port 3 Port 4Top of board

Orange and Green Status LEDs

Both Off = 10 MbpsGreen = 100 MbpsOrange = 1000 Mbps

1. Remove screw usingPhilips screwdriver.

2. Slide the bracket upward

Bracket

until it clears the four RJ45Ethernet jacks.

Bracket Removal

2. Fasten screw usingPhilips screwdriver.1. Slide the bracket down

Bracket

over the four RJ45Ethernet jacks.

Bracket Replacement

RJ45

RJ45

RJ45

RJ45



4.2 NX3-20Gxx 10G SFP+ NIC InstallationThe following procedure is used to install the hardware for the NetXen NIC SFP+ optical adapter.The basic installation procedure for the SFP+ NIC adapter is as follows:

1 Verify that the system is powered off and unplug power cord.

2 Remove the system cover.3 Locate an empty x8 PCIe slot.4 If already inserted, remove the SFP+ optical modules from the NIC adapter. See

Figure 4 below.5 Insert the adapter into the x8 PCIe slot. Apply uniform pressure on top of the board

until it is firmly seated in the bus connector.6 Once the board is inserted into the PCIe slot, connect the SFP+ module. Make sure

an audible clicking sound can be heard when the optical module is inserted into the housing. See Figure 4 below.

7 Secure the bracket to the chassis and replace the cover.8 Connect the appropriate cables to the NIC.9 Power on the system.

Important Note: It is critical that the optical module be seated correctly in the housing to ensure proper operation.

Figure 4 NX3-20Gxx 10 GbE SFP+ Optical Module Installation and Removal

SFP+ NIC AdapterSFP+ module slides into

housing on the boardEnsure hinge is in upright positionwhen inserting module.

When the module is insertedcorrectly, an audible ‘clicking’

SFP+ Optical

To remove optical module, flip the hingedownward. Pull on the hinge to remove

the module from the housing.

Housing

Housing

Module

SFP+ Optical Module

sound will be heard.


5. Windows Driver Installation Guidelines

5 Windows Driver Installation GuidelinesThe following material provides some simple guidelines when using Windows.

5.1 Windows GuidelinesThe following is a brief summary of Windows installation guidelines and should be read before starting a Windows installation.

Windows service pack requirements.

Location of auto-load firmware files.

Windows host memory requirements. Refer to the Release Notes for more information.

Each of these guidelines is described in more detail below.

5.1.1 Windows Service Pack RequirementsSP2 must be installed for the Win2K3 operating system fixes.

5.1.2 Dynamic Firmware LocationThe firmware ROM image files (nx3fwct.bin and nx3fwmn.bin) are placed in the C:\WINDOWS\system32\drivers directory.NetXen provides the option of using either a GUI or command line interface for communicating with the flash utility.



5.2 Linux Checklist and GuidelinesThe following material provides a general checklist of items as well as some simple Linux guidelines.

5.2.1 Linux Checklist

Dynamic Firmware LocationAfter the installation of the binary RPM, the firmware ROM image files (nx3fwct.bin and nx3fwmn.bin) are automatically placed in the /lib/firmware directory (/usr/lib/hotplug/firmware for SLES9).When the driver is installed, the firmware file located the /lib/firmware directory is dynamically loaded onto the NetXen adapter.This firmware becomes the active firmware and supersedes the firmware that has been flashed on the NetXen adapter.

Driver File LocationThe NIC driver is located at:SUSE: /lib/modules/[kernel version]/update/drivers/netRedhat5: /lib/modules/[kernel version]/updateRedhat4: /lib/modules/[kernel version]/kernel/drivers/net

Tools Location

The tools can be found in the following directory:

% cd /opt/netxen

Verifying Firmware VersionTo verify which firmware version is loaded on the adapter and which firmware version is being used and active, run one of the following three commands:% ethtool -i eth[n]% ./nxflash -i eth[n] --info% dmesg | grep nx

5.2.2 Linux Guidelines

Building the Linux Driver Source CodeLinux kernel source/headers are required to compile the drivers provided in source code format. Some Linux OS distributions may not install the kernel source/headers by default. But the headers can be added from the Linux OS Distribution disk using the package manager.

Linux NIC - GPL Inbox Kernel and Commercial DriversThe NetXen Linux NIC driver is available in GPL inbox version and it is included in the Linux kernels available from kernel.org (netxen_nic).



Note that the commercial version of the NIC driver (nx_nic) contains the most recent release updates.

1 Refer to the Release Notes for more information regarding which Linux distributions will contain NetXen Linux GPL inbox driver.

Driver name: netxen_nic.ko

2 The netxen_nic.ko driver can be identified using the following command:

% lsmod | grep netxen_nic3 The Linux GPL driver, netxen_nic.ko can be disabled by:

i. Remove the driver from the operating system memory using the following command:

% rmmod netxen_nic

ii. Once the driver has been removed from operating system memory, remove or rename the netxen_nic.ko file from /lib/modules/<uname -r>/kernel/drivers/net/netxen directory.

In addition, the netxen_nic.ko file is automatically ‘blacklisted’ when the make install command is executed.

The NetXen commercial driver nx_nic.ko, which is included in this release, can be enabled after the Linux GPL driver is disabled.

MSI-X Initialization and SupportMSI and MSI-X are supported by the NetXen driver and firmware for the NX3031. Ensure that the operating system, kernel, and hardware supports MSI-X. MSI-X support can be confirmed using the following command:% cat /proc/interruptsThe following is an example output of this command.

Note that some older Linux kernel versions may not support MSI-X. To support Receive Side Scaling (RSS), there are 4 MSI-X interrupts per-port available to send interrupts to any CPU. Note that MSI-X support is required for RSS.

[root@apps4242 ~]# cat /proc/interruptsCPU0 CPU1

58: 164 7 PCI-MSI-X eth266: 0 0 PCI-MSI-X eth2:174: 6 0 PCI-MSI-X eth2:282: 0 0 PCI-MSI-X eth2:3122: 3195 0 PCI-MSI-X eth3130: 0 0 PCI-MSI-X eth3:1138: 0 0 PCI-MSI-X eth3:2146: 0 0 PCI-MSI-X eth3:3


6. Windows Driver Installation and Configuration

6 Windows Driver Installation and ConfigurationThis section describes the Windows installation and configuration procedure. For the Linux driver installation procedure, refer to Section 7. Note that throughout this section, some screen shots show both the Windows 2003 and the Windows 2008 versions.

6.1 Windows NIC Driver InstallationNote that the following procedure is for a new driver installation.

1 Copy the NIC release package to a directory on the host.

2 After the server system is powered on, Windows detects the new card and displays the Found New Hardware Wizard window.

When this window appears, select No, not at this time, and click Next.



3 Select Install from a list or specific location (Advanced) and click Next, as follows:

4 Select Search for the best driver in these locations.Select Include this location in the search.

Type the appropriate path to the NIC driver directory in the accompanying field and click Next. An example is shown below:

If the exact directory path is not known, click Browse and select the NIC driver directory.The NIC card supports both 32-bit and 64-bit operating systems. The x86 directory contains 32-bit driver and the x64 directory contains the 64-bit driver. Select the appropriate directory (x86 or x64) and click OK. Then click Next in the Hardware Update Wizard window to install the driver.

Windows 2003 Windows 2008

D:\windows\win2k3\x64




5 The ‘Please wait while the wizard installs the software” screen is displayed while the driver is being installed.





6 Once the driver installation is complete, the following window will appear. Click Finish to exit.




6.2 Windows Driver Configuration1 To verify that the driver was properly installed, Right-click on My Computer, then

select Device Manager. A listing of devices appears in right side of the window. Under Network Adapters, the NetXen 10Gb Ethernet Adapter should be listed as shown below:

Windows 2003

Windows 2008



2 In the above window, right-click on the NetXen 10Gb Ethernet adapter and select Properties. The General tab will be displayed as shown.




3 As shown above, there are numerous tabs such as General, Advanced, Driver, and Resources. Click on the Advanced tab to display the driver properties as shown below.




The Advanced tab selections are defined as follows. Note that the shaded entries are relative to Windows 2008 only.

Table 9 Description of Advanced Properties Parameters

Parameter Description

Completion Queue Size Indicates the size of the completion queue.Registry Key: CompletionQueueSizeDefault: 4 (16,384 entries).Values (entries): 0 (1024), 1 (2048), 2 (4096), 3 (8192), 4 (16,384), 5 (32,768)

Flow Control Sets flow control parameters. There are four flow control options: off, transmit, receive, and transmit/receive.Registry Key: FlowControlDefault: Transmit and receive flow control enabled.Values: Enabled/Disabled

Interrupt Moderation Interrupt moderation. When this function is enabled, interrupts are moderated automatically based on traffic patterns.Registry Key: InterruptModerationDefault: Disabled.Values: Enabled/Disabled

IPv4 Checksum Offload This parameter is used for enabling/disabling the IPv4 checksum offload. There are four flow checksum options: off (0), transmit (1), receive (2), and transmit/receive (3).Registry Key: IPChecksumOffloadv4Default: 3. Transmit and receive checksum enabled.Values: 0, 1, 2, 3

Large Receive Offload (LRO) Large receive offload – Enable/Disable options.Registry Key: LRODefault: Enabled.Values: Enabled/Disabled

Large Send Offload V1 IPv4 Large send offload – Enable/Disable options.Registry Key: LsoV1IPv4Default: Enabled.Values: Enabled/DisabledSize of LSO: 64K maximum





Locally Administered Address This is the Locally Administered Address (LAA) that users/administrators can set on this interface to override the permanent address of the adapter (that may have been flashed in to the hardware).Registry Key: NetworkAddressFormat: HexOptions: Value. (a value is required — enter the 12 hex bytes of the MAC address to be used, or not present)

Max Ethernet Frame Size Ethernet frame size for packet transmission and receive (does not include MAC Header).Reg Key: MaxFrameSizeDefault: 1500 (corresponds to 1514 bytes on the wire + 4 bytes of CRC)Max: 9000Min: 64Step: 1

Max Jumbo Buffers Maximum number of receive buffers per Jumbo frame.Reg Key: RxJumboRingSizeDefault: 2 (4096 buffers)Values: 0 (1024), 1 (2048), 2 (4096), 3 (8192).

Max Offload Connections Maximum number of offload connections.Reg Key: MaxOffloadConnectionsDefault: 512Max: 512Min: 0Step: 1

Maximum Receive Buffers (in K) This is packet buffers used by the driver for receive (DMA memory and indicated to stack). Size of each buffer is 1536 bytes.Registry Key: ReceiveBuffersDefault: 4 (16,384 entries).Values (entries): 0 (1024), 1 (2048), 2 (4096), 3 (8192), 4 (16,384), 5 (32,768)

Maximum Transmit Buffers (in K) This is packet buffers used by the driver for transmit (DMA memory and indicated to stack). Size of each buffer is 1536 bytes.Registry Key: TransmitBuffersMin: 1KMax: 16KDefault: 8192Values: 0, 1

Optimization Optimization and IP forwarding. This function should be enabled when the NIC adapter is used for IP forwarding.Registry Key: OptimizationDefault: DisabledValues: Disabled/Enabled

Table 9 Description of Advanced Properties Parameters (Continued)




Priority and VLAN Tag For supporting 802.1p/q priority tagging. Should be enabled to set VLAN ID. There are four options.0: Both 802.1p/q priority tagging and VLAN tagging disabled.1: Priority tagging enabled, VLAN tagging disabled.2: VLAN tagging enabled, priority tagging disabled.3: Both 802.1p/q priority tagging and VLAN tagging enabled.Registry Key: PriorityVLANTagDefault: 3Values: 0, 1, 2, 3

Receive Side Scaling This parameter is for enabling/disabling the Receive Side Scaling (RSS) feature.Registry Key: RSSDefault: EnabledValues: Enabled/Disabled

Receive Side Scaling Rings This parameter controls the number of rings used, the total is calculated by 1KB X 2n with a value of 5 for a maximum of 32KB.Registry Key: MaxStatusRingsDefault: 2Max: 4Min: 1Step: 1

TCP Checksum Offload IPv4 This parameter is for enabling/disabling the TCP checksum offload for IPv4. There are four TCP checksum options: off (0), transmit (1), receive (2), and transmit/receive (3).Registry Key: TCPChecksumOffloadv4Default: 3. Transmit and receive checksum enabled.Values: 0, 1, 2, 3

TCP Checksum Offload IPv6 This parameter is for enabling/disabling the TCP checksum offload for IPv6. There are four TCP checksum options: off (0), transmit (1), receive (2), and transmit/receive (3).Registry Key: TCPChecksumOffloadv6Default: 3. Transmit and receive checksum enabled.Values: 0, 1, 2, 3

TCP Connection Offload IPv4 This parameter is for enabling/disabling the TCP connection offload for IPv4.Registry Key: TCPConnectionOffloadDefault: DisabledValues: Enabled/Disabled

UDP Checksum Offload IPv4 This parameter is for enabling/disabling the UDP checksum offload. There are four UDP checksum options: off (0), transmit (1), receive (2), and transmit/receive (3).Registry Key: UDPChecksumOffloadv4Default: 3. Transmit and receive checksum enabled.Values: 0, 1, 2, 3





This completes the basic configuration of the NetXen NIC software. Refer to the next section for information on configuring the NIC. Note that the above step indicates how to use the Advanced Properties tab of the Adapter Properties sheet. As shown in the screen shot above, there are numerous other tabs that provide additional information.

UDP Checksum Offload IPv6 This parameter is for enabling/disabling the UDP checksum offload. There are four UDP checksum options: off (0), transmit (1), receive (2), and transmit/receive (3).Registry Key: UDPChecksumOffloadv6Default: 3. Transmit and receive checksum enabled.Values: 0, 1, 2, 3

VLAN ID For setting the VLAN ID for this interface (also exposed through the standard OID).Registry Key: VlanIdDefault: 0 (no VLAN)Max: 4094Min: 0Step: 1Range: 0 - 4094





6.3 Configuring the NetXen NIC Card in Windows1 Configure the network connections by opening the Network Connections window in

the control panel, or directly from the desktop if the icon exists. The following window is displayed.

Note that if an IP address has not been assigned, the NetXen adapter entry above will say “Limited or no connectivity”.If a red ‘X’ appears in the NetXen adapter icon, this indicates that the link is down. If this occurs, check the following:

Verify that the card is seated in the PCI slot properly.

Make sure the network cable is connected properly at both ends.

Is a known good cable being used?

Has an IP address been assigned to the interface?

For SFP+ optical adapters, verify that the SFP+ module is installed correctly.

Refer to Section 11, "Troubleshooting Guidelines" for more information.To check the status of the connection, right-click on the NetXen 10 Gb NIC adapter icon above and select properties. This causes the Local Area Connection Properties window to be displayed.



2 To assign an IP address manually, select the Internet Protocol (TCP/IP) entry below and click the Properties tab.

3 In the Internet Protocol (TCP/IP) Properties window, assign the appropriate IP, subnet mask, and default gateway values.

4 Using the ping utility, verify connectivity with another card or switch.



6.4 Uninstalling the NIC Driver in WindowsTo uninstall the NetXen NIC adapter driver software in Windows, perform the following steps:

1 Right-click on My Computer and click on Manage. The Computer Management window is displayed.

2 Double click on Device Manager. The list of devices in the computer is displayed on the right side of the window.

3 Locate and right-click on the NetXen adapter (e.g. NetXen NX3031 Quad GbE Intelligent Server P3 Adapter).

4 Click Uninstall to perform the operation.

6.5 Windows Diagnostic Tests The Windows user diagnostics utility performs the following tests using the nxudiag.exe command. The syntax for these tests is:

nxudiag.exe [command-options-and-arguments]Refer to Table 14 for a list of Windows diagnostic command options and arguments. The NIC adapter will be off line at this time.



6.6 Windows PerformanceThis section describes the Windows performance tuning.

6.6.1 Steps for NIC Performance Testing using ntttcp on WindowsThe following steps can be used to enhance Windows performance. The following subsections provide more information on each of these steps.

1 Enable Windows Scaling

Set the TCp1323Opts parameter value to 1. Note that this parameter is not created by default. For more information on modifying a previously created parameter, or creating a new parameter, Refer to Section 6.6.4 for more information.

2 Set TcpWindowSize

In the TcpWindowSize parameter window, set the value in the Value Data box to either 200000 hexadecimal, or 2,097,152 decimal. Be sure to click either hexadecimal or decimal in the Base box depending on the value entered.

If the TcpWindowSize parameter does not exist, it can be created using regedit. Refer to Section 6.6.5

3 Run ntttcp

Open a DOS prompt window by executing the cmd command at the Start -> Run window.

Change the directory path to the location to where ntttcp was downloaded or copied. An example directory tree is shown below.



For scripts, see the tools package included with the release.

6.6.2 ntttcp OverviewThere are two switches on ntttcp that boost performance: -a and -rb. Running ntttcp without an optimal value for the -a switch and without the -rb switch is acceptable for the Gigabit NIC product.In the 10G NIC product, however, and in particular with fewer threads on Windows, performance is limited by the receive window and by the outstanding I/O (-a) value, regardless of hardware limitations.For receive, an optimal value is 16 for -a and 512000 for -rb (or something similar). With these values, a single thread should perform closer to a multi-thread configuration. From a DOS command prompt, execute the following command. Note that the command must be executed from the same directory in which the executable files are located.

ntttcpr -m 16,0,<rcv ipaddr> -n <# of tcp segs for snd/rcv> -a 16

For transmit, -a should be a value less than 8. The more threads, the lower the number should be.

Note: The send and receive side ntttcp commands must use the receive side’s IP address.



6.6.3 ntttcps and ntttcpr Command Line Options

6.6.4 Tcp1323Opts ParameterThis parameter determines whether TCP uses the time-stamping and window scaling features described in RFC 1323, TCP Extensions for High Performance.For more efficient use of high bandwidth networks, a larger TCP window size may be used. The TCP window size field controls the flow of data and is limited to 2 bytes, or a window size of 65,535 bytes.Since the size field cannot be expanded, a scaling factor is used. TCP window scale is an option used to increase the maximum window size from 65,535 bytes to 1 Gigabyte. The TCP receive window is the amount of data the sending host can send at one time on a connection.When the value for window size is added to the registry and its size is larger than the default value, Windows attempts to use a scale value that accommodates the new window size.The following tables show the four Tcp1323Opts settings.

Data Type Range Value

REG_DWORD 0 | 1 | 2 | 3 1

Hex Value Meaning

0 Timestamps and window scaling are disabled.

1 Window scaling is enabled.

2 Timestamps are enabled.

3 Timestamps and window scaling are enabled.

Version 2.6ntttcpr.exe: [-l|-n|-p|-a|-x|-rb|-sb|-i|-f|-u|-w|-d|-t|-v|-6|-fr|-mb] -m <mapping> [mapping]

-l <Length of buffer> [default: 64K] -n <Number of buffers> [default: 20K] -p <Port base> [default: 5001] -a [outstanding I/O] [default: 2] -x [PacketArray size] [default: 1] -rb <Receive buffer size> [default: 64K] -sb <Send buffer size> [default: 0] -i Infinite Loop [Only UDP mode] -f <File Name> [default: output.txt] -u UDP send/recv -w WSARecv/WSASend -d Verify Flag -t <Runtime> in seconds -v enable verbose mode -6 enable IPv6 mode -fr Full buffers posted on reads -mb Multiple buffer post mode -m <mapping> [mapping] where a mapping is a session(s),processor,receiver IP set e.g. -m 4,0,1.2.3.4 sets up: 4 sessions on processor 0 to test a network on 1.2.3.4



The scale factor can be modified using the following procedure.To enable scaling, set the TCp1323Opts parameter value to 1. Note that this parameter is not created by default. To modify a previously created parameter, refer to the subsection below entitled “Modifying the Tcp1323Opts Parameter”. To create a new parameter, refer to the subsection below entitled “Creating the Tcp1323Opts Parameter”.

Modifying the Tcp1323Opts ParameterIf the parameter was created previously, it can be modified by executing the regedit command at the Start -> Run window.

Executing the regedit command opens the Registry Editor window.Click on the HKEY_LOCAL_MACHINE directory as shown below.

From the HKEY_LOCAL_MACHINE directory, navigate to the following directory path.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

Locate the TCp1323Opts entry and change the value to 1.

Creating the Tcp1323Opts ParameterIf the TCp1323Opts parameter does not exist, it can be created using the following procedure:Execute the regedit command and navigate to the following directory as described above:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

After navigating to the above directory, the following window is displayed.



Anywhere in the right-hand portion of the above window, right-click and create a new key with a DWORD value as shown below.

Rename the ‘New Value #1’ as ‘TCp1323Opts’.

Once this parameter is created, right-click on TCp1323Opts, select Modify, and edit the value from “0” to “1”.



6.6.5 Set TcpWindowSizeFrom regedit, create the parameter “TcpWindowSize” using the same procedure as described above. Rename the new value as TcpWindowSize.Once this parameter is created, right-click on TcpWindowSize and select Modify.

This displays the Edit DWORD Value screen. Set the value in the Value Data box to either 200000 hexadecimal, or 2,097,152 decimal. Be sure to click either hexadecimal or decimal in the Base box depending on the value entered.

For more information on setting the TCP window size, refer to Section 6.6.4 below.


7. Linux Installation and Configuration

7 Linux Installation and ConfigurationThis section describes the Linux installation and configuration procedure. For the Windows driver installation procedure, refer to Section 6.

7.1 NetXen GPL Driver and BlacklistingDuring the installation of the NetXen NIC RPM binary package, the NetXen GPL driver (netxen_nic.ko) will become blacklisted. This will stop the loading of the netxen_nic.ko driver when the system boots up. Upon using the RPM erase command (“rpm -e nx_nic-[version]”)1 and thereby removing the NetXen NIC RPM binary package, the blacklisting will revert back to the prior setting and loading of the netxen_nic.ko driver will be permitted.To check if the NetXen GPL driver has been installed on the system:

1 Check the /lib/modules/<kernel-version>/kernel/net/netxen directory for the netxen_nic.ko driver and either remove or rename this file then reboot.

2 Or to uninstall the driver, execute the following command before continuing with the following steps in this section.

% rmmod netxen_nic

The following sections describe the procedure for a new driver installation.

7.2 Preparing for NIC Driver InstallationCheck for prior versions of the NetXen driver and tools. Remove the packages from the system to prepare for installing the new release.

Use RPM to check for prior packages.% rpm -qa | grep nx_ nx_nic-[prior_version] nx_tools-[prior_version]

Remove each of the prior packages% rpm -e nx_nic-[prior_version]% rpm -e nx_tools-[prior_version]

1. [version] refers to the RPM package currently on the system. RPM packages can be located by executing the com-mand: “rpm -qa | grep nx_”.



7.3 NIC Driver Installation (RPM Format)The following subsections describe how to install the Linux source in RPM format, create the binary, and install the binary.

Install the Linux Source in RPM Format

To install the Linux source in RPM format, execute the following command.

% rpm -iv nx_nic-[version].src.rpm

Create the Binary

To create the binary RPM for Redhat systems, execute the following command:

% rpmbuild -bb usr/src/redhat/SPECS/nx_nic-[version].spec

To create the binary RPM for SuSE systems, execute the following command:

% rpmbuild -bb usr/src/packages/SPECS/nx_nic-[version].spec

Install the Binary

To install the binary RPM for Redhat systems, execute the following command:

% rpm -i usr/src/redhat/RPMS/[arch]/nx_nic-[version].[arch].rpm

To create the binary RPM for SuSE systems, execute the following command:

% rpm -i usr/src/packages/RPMS/[arch]/nx_nic-[version].[arch].rpm

Note that the [arch] variable above designates the system architecture. Some [arch] variables include i386, i586, i686, x86_64, or other depending on the system. The installed NetXen driver binary will be located in the either one of the following two directories:

/lib/modules/<uname -r>/updates/drivers/net/, or

/lib/modules/<uname -r>/kernel/drivers/net

Install the Tools

To install the binary tools, execute the following command:

% rpm -ivh nx_tools-[version].rpm

The tools can be found in the following directory:

% cd /opt/netxen

These tools include:

nxflash

nxudiag

phantomcore_P3



[card type]-romimage

The release may contain multiple ROM images. The [card type] variable in the above command refers to the exact ROM image. Refer to the Release Notes document for more information on the ROM image types.

7.4 Linux NIC Driver OptionsThe following optional parameters can be passed to the driver with the modprobe command:

% modprobe -v nx_nic <parameter>

The optional parameters are listed below.

7.4.1 Ethtool UtilityThe ethtool utility may also be used check and configure additional driver options. See the “man nx_nic” Linux man pages for examples on configuring driver options. See “man ethtool” for additional driver options. The following is an example command:% ethtool -K eth[n] tso off

Table 10 modprobe Command Optional Parameters

<parameter> Description Default Range

tx_desc Sets the size of the command descriptor rings (Tx). 1024 256 - 1024 (must be power of two)

jumbo_desc_1g Sets the size of the receive ring for jumbo frames (1 GbE ports).

1024 for x86_64

64 - 1024(must be power of two)

512 for x86_32

jumbo_desc Sets the size of the receive ring for jumbo frames (10 GbE ports).

1024 64 - 1024(must be power of two)

rdesc_1g Sets the size of the receive ring for normal packets (1 GbE ports).

For firmware version 4.0.xyz, max is 8192

8192 for x86_64

64 - 8192 (must be power of two)

2048 for x86

rdesc_10g Sets the size of the receive ring for normal packets (10 GbE ports).

For firmware version 4.0.xyz, max is 8192

8192 for x86_64

64 - 8192 (must be power of two)

4096 for x86

use_msi Enables MSI interrupts. 1 0: Disabled1: Enabled

use_msi_x Enables MSIx interrupts.Only available in kernel version greater than 2.6.8. If this option is enabled, it takes precedence over MSI interrupts and the use_msi option is ignored.

1 0: Disabled1: Enabled

rss_enable Enables receive side scaling. Only available if MSIx interrupts are enabled.

1 0: Disabled1: Enabled



7.5 Configuring the NetXen NIC on LinuxAssign an IP address to the interface from the command line. For example:

1 Identify the Ethernet interface (eth1, eth2, etc.) and the nx_nic driver:

% cat /proc/net/nx_nic/dev0/stats | grep eth

or

% ifconfig -a

After running ifconfig, look for the interface with a MAC address (HWaddr) prefix of 00:0E:1E:XX:XX:XX

2 Assign an IP Address and activate the interface:

% ifconfig eth[n] down

% ifconfig eth[n] <IP Address> netmask <Netmask Value> up

3 Verify network services are running, and ping:

% service network restart

ping a different IP address to verify connectivity

7.6 Uninstalling the NetXen NIC Driver in Linux

RPM FormatIn the Linux system, the RPM archive may be removed as follows:

% rpm -e nx_nic-[version]1

% ifconfig eth[n] down

% rmmod nx_nic.ko

7.7 Linux Diagnostic Tests The Linux user diagnostics utility performs the following tests using the ./nxudiag -i eth[n] command.

The syntax for these tests is:

% ./nxudiag [command-options-and-arguments]

Refer to Table 14 for a list of Linux diagnostic tests. Note that while running these tests, the network traffic will be interrupted.

1. [version] refers to the RPM package currently on the system. RPM packages can be located by executing the com-mand: “rpm -qa | grep nx_”.



7.8 Linux PerformanceThis section describes the Linux performance tuning.

7.8.1 Linux Performance ToolsAdd the following for performance tuning on Linux systems under --/etc/sysctl.conf

Note: Various factors including tools, kernel, stacks, and related parameters affect the performance of a system.

Using the IPERF ToolThe -P option determines the number of threads; default is one.Server Side:

% iperf -s -w256k -l 56k

Client Side:

% iperf -c <ipAddr of Server> -w256k -P12 -l 56k

7.8.2 Linux OptimizationWhile optimization of the Linux operating system is a topic beyond the scope of this guide, below are a few common guidelines.

PCIe x8 provides optimal performance.

MSI interrupts are preferred over legacy interrupts.

Linux Kernel versions 2.6.15 or later provide better performance optimization for the NIC.

Check CPU utilization on the host for CPU-bound bottlenecks.

The tunable performance parameters are system and application specific. Understanding the relationship between the parameter values, target workload, performance applications, OS and system configuration provides a better understanding of performance optimization and bottlenecks.

net.core.rmem_max = 16777216net.core.wmem_max = 16777216

net.ipv4.tcp_rmem = 4096 87380 16777216net.ipv4.tcp_wmem = 4096 87380 16777216

net.core.netdev_max_backlog = 250000

net.ipv4.tcp_timestamps = 0net.ipv4.tcp_sack = 1net.ipv4.tcp_low_latency = 0net.ipv4.tcp_window_scaling = 1net.ipv4.tcp_moderate_rcvbuf = 1net.ipv4.route.flush = 1

net.core.optmem_max = 524287


8. Linux Configuration for Bonding Mode and Virtual LAN (VLAN)

8 Linux Configuration for Bonding Mode and Virtual LAN (VLAN)

8.1 Linux Bonding ModeEnsure that the steps required for installing the RPM source, creating the binary, and installing the binary have been performed before setting up the Linux bonding mode. If not, refer to these steps in Section 7.3 before proceeding.In addition, verify that the NetXen driver has been installed using the following command:

% lsmod | grep nx_nic

If the driver has not been installed, execute the following command to install the driver:

% modprobe -v nx_nic

The steps for setting the Linux bonding mode are as follows. Note that these are example commands. The actual command may be different depending on the system configuration.

1 % modprobe bonding mode=1 miimon=100 updelay=50000 primary=eth16

2 % ifconfig bond0 25.145.1.199 netmask 255.255.255.0 broadcast 25.145.1.255 up

3 % ifenslave bond0 eth15 eth16

To bring down the bond use the following command:

% ifconfig bond0 down

Note: These commands and steps have been tested under RHEL5, but for other Linux distributions and versions there may be differences in how bonding is enabled and utilized.

8.1.1 Active-Backup or Mode=1Active-backup policy: Only one slave in the bond is active. A different slave becomes active if, and only if, the active slave fails.

By setting mode=1, when ifenslave is executed, the bonding driver calls into the bond_enslave() function and sets the slave link modes for the corresponding two Ethernet ports to BOND_LINK_DOWN.

Since the bond mode is 1, ie. BOND_MODE_ACTIVEBACKUP, and primary device is indicated, one slave device is configured as active slave. The dmesg command can be used to display the primary port. An example output of dmesg is shown below:

bonding: bond0: making interface eth16 the new active one 0 ms earlier

If the active connection goes down, then the slave port becomes the primary port. The following command can be used to monitor the bonding process during the switch-over or port priority.

% cat /proc/net/bonding/bond0

The following message is displayed:



MII Status: up (for eth16)

The dmesg command can be used to show the status of the slave port as shown below:

bonding: bond0: link status up for interface eth15, enabling it in 50000 ms.

Execute the following command to check the status of the primary port.


MII Status: down (for eth16)

After 50 seconds, dmesg will print:

bonding: bond0: link status definitely up for interface eth15.

bonding: bond0: making interface eth15 the new active one.

Execute the following command to check the status of the new primary port.


MII Status: up (for eth15)

8.1.2 Round-Robin or Mode=0This mode is used to implement a round-robin scheme between the two Ethernet ports. To set the round-robin mode, execute the following command:

% modprobe bonding mode=0 miimon=100 updelay=50000 primary=eth16

then both links delay for 50 secs before their states are set to BOND_LINK_UP state.

Execute the following command to check the status of the bonding process.


Ethernet Channel Bonding Driver: v3.0.1

Bonding Mode: load balancing (round-robin) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 50000 Down Delay (ms): 0

Slave Interface: eth16

MII Status: down <========== down state

Link Failure Count: 0

Permanent HW addr: 00:0e:1e:00:11:d2


MII Status: down <========= down state



after 50 secs:



Execute the following command again to verify the MII status of the Ethernet ports.


Ethernet Channel Bonding Driver: v3.0.1 (January 9, 2006)

Bonding Mode: load balancing (round-robin) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 50000 Down Delay (ms): 0


MII Status: up <=========== up state




MII Status: up <=========== up state



And dmesg will print:



8.1.3 Bonding Driver OptionsOptions for the bonding driver are supplied as parameters to the bonding module at load time. They may be given as command line arguments to the modprobe command, but are usually specified in either the /etc/modules.conf or /etc/modprobe.conf configuration file, or in a distribution-specific configuration file (some of which are detailed in the next section).

8.1.4 Bonding Driver ParametersThe available bonding driver parameters are listed below. If a parameter is not specified the default value is used. When initially configuring a bond, it is recommended "tail -f /var/log/messages" be run in a separate window to watch for bonding driver error messages.

It is critical that either the miimon or arp_interval and arp_ip_target parameters be specified, otherwise serious network degradation will occur during link failures. Very few devices do not support at least miimon, so there is really no reason not to use it.

Options with textual values will accept either the text name or, for backwards compatibility, the option value. E.g., "mode=802.3ad" and "mode=4" set the same mode.

The parameters are as follows:



arp_interval

Specifies the ARP link monitoring frequency in milliseconds. If ARP monitoring is used in an etherchannel compatible mode (modes 0 and 2), the switch should be configured in a mode that evenly distributes packets across all links.

If the switch is configured to distribute the packets in an XOR fashion, all replies from the ARP targets will be received on the same link which could cause the other team members to fail. ARP monitoring should not be used in conjunction with miimon. A value of 0 disables ARP monitoring. The default value is 0.

arp_ip_target

Specifies the IP addresses to use as ARP monitoring peers when arp_interval is > 0. These are the targets of the ARP request sent to determine the health of the link to the targets.

Specify these values in ddd.ddd.ddd.ddd format. Multiple IP addresses must be separated by a comma. At least one IP address must be given for ARP monitoring to function. The maximum number of targets that can be specified is 16. The default value is no IP addresses.

downdelay

Specifies the time, in milliseconds, to wait before disabling a slave after a link failure has been detected. This option is only valid for the miimon link monitor. The downdelay value should be a multiple of the miimon value. If not, it will be rounded down to the nearest multiple. The default value is 0.

lacp_rate

Option specifying the rate at which the link partner transmits LACPDU packets in 802.3ad mode. Possible values are:

- slow or 0: Request partner to transmit LACPDUs every 30 seconds (default)

- fast or 1: Request partner to transmit LACPDUs every 1 second

8.2 Configuring Virtual LAN (VLAN) under LinuxThe VLAN function can be configured as described below.

1 Check if the 8021q.ko driver has been loaded by executing the following command:

% lsmod | grep 8021q

If the driver has not been loaded, execute the following command:

% modprobe -v 8021q

2 The vconfig add command creates a VLAN-device on eth0 and will in turn create an eth0.5 interface. Add VLAN ID 5 with the following command for eth0:

% vconfig add eth0 5

3 Use the ifconfig command below to assign an IP address to the VLAN interface:



% ifconfig eth0.5 192.168.1.100 netmask 255.255.255.0 broadcast 192.168.1.255 up

4 Use the following command to get detailed information about the VLAN interface:

% cat /proc/net/vlan/eth0.5

Repeat steps 2 and 3 if more than one VLAN is required.To delete a VLAN interface, use the following commands:

% ifconfig eth0.5 down

% vconfig rem eth0.5


9. Firmware Update

9 Firmware UpdateThis section describes the firmware update process. During the driver installation process, the firmware files are installed onto the host. The dynamic firmware download takes place when the driver is loaded. As the driver is loaded the firmware file becomes the active firmware on the adapter.

9.1 Windows Dynamic Firmware DownloadDuring the driver installation process, the firmware ROM image files are placed in the C:\WINDOWS\system32\drivers directory. To determine if the firmware has loaded successfully in Windows, execute nxflash.exe -i “Local Area Connection n” --info. Check for the following entry:Active Firmware version: 4.0.229Note that the ‘n’ in “Local Area Connection n” refers to the port being flashed. To determine the port number:

Right-click on My Network Places

Select Properties

This will list all of the ports and the type of device to which they are connected. Note that the quotation marks shown in the command must be present for the command to execute correctly.

9.2 Linux Dynamic Firmware DownloadAfter the installation of the binary RPM, the firmware ROM image files are placed in the /lib/firmware directory (/usr/lib/hotplug/firmware for SLES9). When the driver is installed, the firmware file located in the /lib/firmware directory is dynamically loaded onto the NetXen adapter.To determine if the firmware has loaded successfully in Linux, execute the following command:./nxflash -i eth[n] --infoCheck for the following entry:Active Firmware version: 4.0.229


10. Flash Update

10 Flash UpdateUnder certain circumstances, it may be necessary to perform a Flash update on the NIC adapter. Refer to the Release Notes to determine if the NIC must be Flashed.

10.1 Flash Update and Tools Guidelines1 During the Flash update process do not reboot or stop the update process.

2 A backup image of the existing Flash will be created at the start of the Flash update process. The flash utility will use the backup image to recover if required.

3 The flash tool will not start the Flash update process if the NIC driver is not loaded.4 The flash tool auto-identifies the appropriate Flash image for the cards.

10.2 Flash Update OptionsThere are two ways to update the Flash. In both cases, these options should not be used unless specified in the Release Notes.

nxflash command

LiveCD

10.2.1 nxflash Command Run the appropriate command from Linux or Windows as shown below:For Linux:./nxflash -i eth[n] --allFor Windows:nxflash -i “Local Area Connection n” --allWith this method there will be a prompt asking for an Y(es) or N(o) to allow the upgrading of the Flash to continue. Press the Y key to continue.

10.2.2 LiveCDFrom the NetXen web site, download the NetXen LiveCD ISO image and burn that image onto a CD and boot the system with the CD. Use option 2 from the LiveCD menu shown below to update the flash on the card. Refer to the support page on the on NetXen web site for more information on downloading the LiveCD.

Netxen LiveCD menu screen:NetXen LiveCD Firmware UpdateMenu Options:1. Show NetXen board information2. Update to latest firmware on NetXen board3. Reboot system if firmware has been updatedEnter here:


10. Flash Update

10.3 Windows Flash UpdateThis section provides information on how to perform a Flash update in Windows. Note that the NetXen card is shipped with a default Flash version.

10.3.1 Using the NetXen Flash Utility in WindowsWhen using the nxflash.exe utility, the following syntax should be used.nxflash -i “Local Area Connection n” --allIn this command line, the -i indicates the interface command and the “Local Area Connection n” specifies the card being flashed. To determine the card number:

Right-click on My Network Places in Windows

Select Properties.

This will list all of the cards and the type of device to which they are connected. Note that the quotation marks shown in the command must be present for the command to execute correctly.Along with the port number, the available commands are listed in Table 11. Table 11 NetXen Flash Utility Commands

Command Definition Description

-i--interface

Interface Specifies the interface type [Local Area Connection 1, Local Area Connection 2, etc.]

--info Information Prints board information such as board type, chip revision, serial ID, firmware version, BIOS version, MAC address, etc.

-a--all

Flash update Complete update of BIOS and firmware. This command is used when the default rom image is used.

-a <image file>--all <image file>

Flash update Complete update of BIOS and Flash using user-renamed rom image. Refer to the Release Notes document for a listing of image files and their proper syntax.

-r <backup file>--restore

<backup file>

Restore When an update is performed, the previous firmware version is stored to a file in the current working directory. Executing this option restores the specified backup Flash version to the card. An example of a backup file name is as follows:flashbackup_[serial number]_[version}_[date-time]

--pxe-on PXE Enable PXE.(default).

--pxe-off PXE Disable PXE

-S--show

Show board information Displays board information for all installed NetXen adapter cards.

-h--help

Help Displays the help menu.

-v--version

Version Displays the version information.


10. Flash Update

10.3.2 Windows Flash Update ProcedureThe following steps are used for updating the Windows OS Flash.1 Open Command window.2 Go to the utils directory in the driver package provided.3 The following binaries are required to be in the same directory from where the Flash

command will be executed: nxflash.exe [card type]-romimageThe release may contain multiple ROM images. The [card type] variable in the above command refers to the exact ROM image. Refer to the Release Notes document for more information on the ROM image types.Note that the nxflash utility and ROM image files must be in the same directory from where the nxflash command is executed.Run the following command:

nxflash.exe -i “Local Area Connection n” --all

Writing of the flash requires a few minutes. Please do not interrupt or reboot during this time. Note that ‘n’ indicates the number of the port being flashed as described above.

The following is an example of the Windows nxflash --all output for local area connection 8.

4 Reboot the system. 5 Verify the Flash has been updated using the following command.

nxflash.exe -i “Local Area Connection n” --info

C:\netxen\4.0.223\utils>nxflash -i "Local Area Connection 8" --allNETXEN Port is Port 0Verifying flash...Done.0/9 - Init*** Currently in flash ***Board Type : Quad Gig LPFirmware Version : 4.0.222MAC Address 0 : 00:0E:1E:00:7F:24MAC Address 1 : 00:0E:1E:00:7F:25MAC Address 2 : 00:0E:1E:00:7F:26MAC Address 3 : 00:0E:1E:00:7F:27Serial Number : QG88BK0130NIC binary romimage found in C:\netxen\4.0.223\utilsRom Image : C:\netxen\4.0.223\utils\lp_qg_cut_thru_b1_romimage1/9 - Extracting RomimageFirmware version From Board: 4.0.222Firmware version From Romimage: 4.0.223WARNING: This operation will take the NIC off line.Do you wish to upgrade? (Y/N) y


10. Flash Update

Note that ‘n’ indicates the number of the port being flashed as described above.

The following is an example of a Windows nxflash --info output for local area connection 0:

10.3.3 Updating the Flash on Multiple Standalone Cards in WindowsThe Flash can be updated on multiple cards on the same host using one of the following procedures.

1 If the updates are being done from the same directory tree, the following commands must be run sequentially after the Flash update has completed.



Note that the in the above commands, the ‘n’ would represent two different port numbers, such as “Local Area Connection 4” and Local Area Connection 5”.

2 Updates can be done simultaneously from two different directory trees. Note that the following two commands are being executed from two separate directory trees.

c:\[version]-a\utils\ nxflash.exe -i “Local Area Connection n” --all

c:\[version]-b\utils\ nxflash.exe -i “Local Area Connection n” --all

Note that the in the above commands, the ‘n’ would represent two different port numbers, such as “Local Area Connection 4” and Local Area Connection 5”.

C:\Users\Administrator\Desktop\4.0.223\utils>nxflash.exe -i “Local Area Connection 0” --infoNETXEN Port is Port 0Verifying flash...Done.Board Type: 0x24 (Quad Gig LP)Board Chip Rev: B1Serial Id: QG87BK0092Efuse ID: 0x4e36343633340f08Firmware version (on Flash): 4.0.228Active Firmware version: 4.0.229Firmware in: CUT-THRU modeDriver version: 4.0.229BIOS version: 2.0.0MAC Addr 0: 00:0E:1E:00:79:DCMAC Addr 1: 00:0E:1E:00:79:DDMAC Addr 2: 00:0E:1E:00:79:DEMAC Addr 3: 00:0E:1E:00:79:DFSubsystem ID: 0x01244040PHY Type: UNKNOWN PHYPXE Status: Disabled


10. Flash Update

10.4 Linux Flash UpdateThis section provides information on how to perform a Flash update in Linux. Note that the NetXen card is shipped with a default Flash version.

10.4.1 Using the NetXen Flash Utility in LinuxThe NetXen Flash utility can be used to update the Flash. When using the nxflash.exe utility, the following syntax should be used. % ./nxflash -i eth[n] [command]In this command line, the [n] indicates the NetXen Ethernet interface number (eth[0], eth[1], etc).Along with the interface, the available commands are listed in Table 12.

Table 12 NetXen Flash Utility Commands

[command] Definition Description

-i--interface

Interface Specifies the interface type (eth0, eth1, etc,)

--info Information Prints board information such as board type, chip revision, serial ID, firmware version, BIOS version, MAC address, etc.

-a--all

Flash update Complete update of BIOS and firmware.

-a <image file>--all <image file>

Flash update Complete update of BIOS and firmware using user-renamed rom image.

-r <backup file>--restore

<backup file>

Restore When an update is performed, the previous Flash version is stored to a file in the current working directory. Executing this option restores the specified backup Flash version to the card. An example of a backup file name is as follows:flashbackup_[serial number]_[version}_[date-time]

-S--show

Show board information Displays board information for all installed NetXen adapter cards.

--pxe-on PXE Enable PXE.(default).

--pxe-off PXE Disable PXE

-h--help

Help Displays the help menu.

-v--version

Version Displays the version information.


10. Flash Update

10.4.2 Linux Flash Update ProcedureThe following steps are used for updating the Linux OS Flash.

1 Ensure that the steps required for installing the RPM source, creating the binary, and installing the binary to the file system have been performed. If not, refer to these steps in Section 7.3 before proceeding. The new driver is now ready to be loaded automatically the next time the system is booted up.

2 Update the Flash. The Flash update takes a few minutes to complete and MUST NOT BE INTERRUPTED. Note that only one port on each NIC adapter need be updated. Updates do not need to be performed on a per-port basis.

% cd /opt/netxen

% ./nxflash -i eth[n] --all

3 Reboot the host system.4 Get Flash related information

The --info option may be used to get information about the Flash.

% ./nxflash -i eth[n] --info

The following is an example of a Linux nxflash output:

NETXEN Port is Port 0Verifying flash...Done.Board Type: 0x24 (Quad Gig LP)Board Chip Rev: B1Serial Id: QG87BK0092Efuse ID: 0x4e36343633340f08Firmware version (on Flash): 4.0.228Active Firmware version: 4.0.229Firmware in: CUT-THRU modeDriver version: 4.0.229BIOS version: 2.0.0MAC Addr 0: 00:0E:1E:00:79:DCMAC Addr 1: 00:0E:1E:00:79:DDMAC Addr 2: 00:0E:1E:00:79:DEMAC Addr 3: 00:0E:1E:00:79:DFSubsystem ID: 0x01244040PHY Type: UNKNOWN PHYPXE Status: Disabled


11. Troubleshooting Guidelines

11 Troubleshooting Guidelines

11.1 Unable to Boot the Host System with the NIC Card InstalledRemove the NIC and reboot. Does the system boot without the NIC?

If so, power down the system, insert the NIC into a different PCIe slot and boot-up the system.

During the system boot up process, ensure that the system BIOS configuration is up to date.

Is the recommended minimum memory installed on the host system?

Note any error message during the OS boot sequence.

Check for system error messages:

Dbgview.exe (Windows event viewer)

% dmesg (Linux)

11.2 Unable to Send/Receive Network Traffic Using the NICIs the network cable plugged in?

Is the client machine configured correctly and able to communicate with other machines?

Type lspci -v and look for vendor ID ‘4040’.

- Is the NIC visible on the PCI bus?

- Is the NIC is plugged into the appropriate PCIe slot?

Check if the driver is loaded correctly and the card is initialized

- dmesg (display system messages on Linux)

- Windows Event Viewer tool

Check Network Configuration

- Check for valid IP Address and subnet

- Check for duplicate IP address and subnet numbers

- If connected via gateway, check gateway address and network path

- Check the NIC MAC address values

- Network link status as seen by the host OS

- Check for LED activity (using ping)

- Using network monitoring tools, analyze the Tx and Rx packets for the NIC

- Verify the ARP entry and response

If all of the above checks fail to resolve the problem, run the NetXen user diagnostics.



11.3 Gather System InformationThe following information can be gathered for general debug purposes.

NetXen Product Information

- Card type. For Windows, refer to step 1 in Section 6.3. For Linux, refer to Section 11.5.2.

- NIC firmware and driver version. For Windows, refer to step 2 of Section 10.3.1. For Linux, refer to step 1 of Section 10.4.1.

Operating System Information

- Windows OS (32 / 64) and Service Pack #.

- Linux Distribution (cat /etc/issue) and Kernel Release (uname –a)

Host System Information

- CPU, Cores, Chipset – (Run System Information tool)

- BIOS information

- Host memory

- PCIe x8, x4, x1 — electrical configuration (not physical)

- System setup description

- How many servers, clients etc

- Model of the host system

- Network setup (back to back, router, switch, backplane etc)

11.4 Check for System ErrorsLinux: Check for error messages from the kernel using the following commands:

% dmesg

% cat /var/log/messages

Linux: Check routing table, gateway, and IP Address conflicts for all interfaces using the following command:

% route

Linux: Check network interface TX and RX counts during pings using the following command:

% ifconfig <network interface>

Linux: Use the wireshark, ethereal, tethereal, or tcpdump utilities to capture packets:

Windows: Use the wireshark or ethereal utilities to capture packets:

Windows: Check for system error messages

Dbgview.exe (Windows event viewer)



11.5 Caveats

11.5.1 MSI-X Interrupts for LinuxNewer Linux OS distributions or kernel versions support MSI-X interrupts. However, older versions of Linux may not. If MSI-X is not available on the OS, then MSI or legacy mode may be used in the operating system.

On the Linux host, type the following command to check that PCI-MSI-X or PCI-MSI is enabled for the Ethernet interface:

% cat /proc/interrupts

Determine if the installed Linux kernel has the MSI configuration option enabled (earlier versions of the kernel may require an MSI patch).Type the following command from the directory /usr/src/kernels/<kernel version> and verify that MSI configuration option is supported/enabled.

% make menuconfig

11.5.2 PCI-Express SlotThe NetXen reference card can be located on the PCI-Express bus using the following Linux procedures.

Determine Vendor and Device IDTo check if the PCIe slot has an x8 link for optimal PCIe performance, first determine the PCI vendor ID and device ID codes using the following command:

% lspci -n

A message similar to the following should be displayed:

06:00.0 0200: 4040:0100 (rev 42)

The NetXen vendor ID is 0x4040. A list of NetXen device ID’s is shown below.

Determine Interface AttributesTo determine the attributes of the interface, execute the following command:

% lspci -n -d 4040:0100 -vvv | grep -i width

A message similar to the one below should be displayed.

Link: Supported Speed 2.5 Gb/s, Width x8, ASPM L0s, Port 0

Table 13 NetXen Device IDs

Product Device ID Description

NX3-4GBT 0124 Quad-port Gbit Ethernet NIC copper interface.

NX3-20GCU 0126 Dual-port 10 Gbit Ethernet NIC with twinax interface.

NX3-20GSR 0126 Dual-port 10 Gbit Ethernet NIC with short range fiber interface.

NX3-20GLR 0126 Dual-port 10 Gbit Ethernet NIC with long range fiber interface.

NX3-20GLM 0126 Dual-port 10 Gbit Ethernet NIC with multi-mode fiber interface.



Link: Speed 2.5Gb/s, Width x8

Determine Width ParametersCheck if the lspci output "negotiated width" (the second width in the output) reflects the actual "supported width". In an SFP+ system for example, execute the following command to determine if these two widths are the same:

% lspci -xxx -vvv -d 4040:0100

where 4040 indicates the NetXen vendor ID and 0124 indicates a 4GBT card device ID. The SFP+ card device ID is 0126.Examine the following hex output:

From the lspci hex output:

Note that some PCI-e chip sets with a x16 link may not support a x8 PCI-Express link width. A link is only required to support its maximum width and x1. Check the motherboard specification to determine if its PCI-e x16 slot can autonegotiate to x8. On some motherboards, the PCI-e slots are all the same physical size required by a x16 slot, but electrically they are different widths. Check the motherboard specification for the supported width of each of the PCI-e slots.

11.6 Windows Support

11.6.1 Win2k3 RSS and SNP SupportIf problems are encountered with server applications when RSS or SNP is enabled under Microsoft Win2k3 with SP2 (in Offload Mode), please follow the link below to see if Microsoft has addressed the specific issue.

http://support.microsoft.com/kb/948496

Read the bulletin entitled:

“An update to turn off default SNP features is available for Windows Server 2003-based and Small Business Server 2003-based computers”

...... 82 04 00 00 01 00 82

Supported link width

Negotiated link width

||_________ capability. "82 04" = x8 PCIe Gen2

c0: 10 00 02 00 02 80 00 00 00 10 0b 00 82 04 00 00


12. Quick Briefs

12 Quick BriefsThis section contains information about the following topics:

Linux PXE requirements and configuration

iSCSI boot requirements and configuration

Additional information on Linux PXE or iSCSI boot requirements can be found in the corresponding Tech Brief documents located on the NetXen support web site.

12.1 Linux PXE Installation Requirements and ConfigurationThis section includes the PXE side server requirements, an overview of the software used in this example, and a list of configuration steps.

12.1.1 PXE Server Side Requirements

The above services can be installed from the appropriate packages with the Linux Package Manager or by searching for the appropriate packages for each Linux distribution.

12.1.2 Validated in This Example

Installed Services Required / Optional

dhcp server Required

tftp-server Required

nfs server Required

Red Hat 5.2 x86_64 Install DVD Required

tftp (client) Optional

NetXen Driver Refer to release notes for latest version

NetXen Firmware Refer to release notes for latest version

DHCP server V3.0.5-RedHat (in-box)

NFS server Server nfs v3 (in-box)

TFTP server Default from RHEL5.2 (in-box)


12. Quick Briefs

12.1.3 PXE Installation and Configuration

1 Install on the PXE server:

Setting up PXE server with NetXen driver.

Configuring a static IP address for the interface to be used for PXE.

2 Configure DHCP service on the PXE server.

Set it to be enabled and started upon boot-up.

3 Customize the initird.img file with the latest NetXen driver.

Modify the modules.alias file to remove instances of the GPL driver.

Include information for the NetXen commercial NIC driver.

4 Configure xinetd and tftp services.

Prepare a Linux pre-install kick-start configuration file.

5 Configure NFS server on PXE server.

Create Linux install ISO image onto PXE server.

Create a customized Linux install kick-start configuration to automate the process for a network installation.

12.2 iSCSI Installation Requirements and ConfigurationThis section includes the iSCSI host and client server requirements, an overview of the software used in this example, and a list of configuration steps.

12.2.1 iSCSI Boot Host Requirements

12.2.2 iSCSI Boot Client Requirements

The above services can be installed from the appropriate packages with the Linux Package Manager or by searching for the appropriate packages for each Linux distribution. Microsoft packages can be retrieved from the Microsoft web site, www.microsoft.com.

Services Required / Optional

dhcp server required

iSCSI Enterprise Target required

Red Hat 5.2 x86_64 Install DVD optional

Programs Required / Optional

Windows Server 2003 32bit Install CD required

Microsoft iSCSI Initiator required


12. Quick Briefs

12.2.3 Validated in This Example

12.2.4 Main Steps Required

1 Configure DHCP service to be enabled and to start upon boot-up:

Refer to the NetXen Tech Brief regarding ‘Setting Up a Linux PXE Server’.

Go to the ‘Setting up the DHCP service’ section in the technical brief for this information.

2 Install and configure the iSCSI enterprise target on the iSCSI boot host system.3 Install and configure the iSCSI boot client system's NetXen driver and registry

specifically for booting via iSCSI.4 Install and configure the Microsoft iSCSI initiator on the iSCSI boot client system.5 Create and modify the iSCSI boot client's iSCSI boot image that contains Windows

and grub partition, then place it onto the iSCSI boot host system.6 Modify the iSCSI boot host's DHCP setting to support the boot image via iSCSI.For more information, please refer to the NetXen Tech Brief document on the NetXen support web site for the complete steps.

NetXen Driver Refer to release notes for latest version

NetXen Firmware Refer to release notes for latest version

DHCP server V3.0.5-RedHat (in-box)

iSCSI Enterprise Target Version 0.4.16

MSFT iSCSI Initiator Version 2.0.6


Appendix A. Diagnostic Utility Commands

Appendix A Diagnostic Utility Commands

Table 14 Diagnostic Utility Commands

Command Description

-i--interface

Specifies the interface type [Local Area Connection 1, Local Area Connection 2, etc.]

-a--all

Perform all test, regardless of default.

-D--default

Perform default test only.

-R--CRegs

Test all control registers (default).

-F-FLASH

Test the Flash (default).

-I--IRQS

Test interrupt mechanism (default).

-L--IntLB

Internal loopback test (default).

-H--Hw

Hardware test (default).

-S--LinkST

Link status test (default).

-I--LED

LED test (default).

-nR--noCRegs

No control registers test (combine -D or -a).

-nF--noFLASH

No Flash test (combine -D or -a).

-nl--noIRQS

No interrupt test (combine -D or -a).

-nL--noIntLB

No internal loopback test (combine -D or -a).

-nH--noHw

No hardware test (combine -D or -a).

-nS--noLinkSt

No Link status test (combine -D or -a).

-nl--noLED

No LED status test (combine -D or -a).

-E--ExtLB

External full duplex loop-back test.

-h--help

Prints the help message.


Appendix A. Diagnostic Utility Commands
