48
www.mellanox.com ConnectX ® -2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, MNPH29C-XTR, MNPH29D-XSR, MNPH29D-XTR Rev 1.6

ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

Embed Size (px)

Citation preview

Page 1: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

www.mellanox.com

ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual

P/N: MNPH29C-XSR, MNPH29C-XTR, MNPH29D-XSR, MNPH29D-XTR

Rev 1.6

Page 2: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

Mellanox Technologies350 Oakmead Parkway Suite 100 Sunnyvale, CA 94085 U.S.A.www.mellanox.comTel: (408) 970-3400Fax: (408) 970-3403

Mellanox Technologies, Ltd.PO Box 586 Hermon BuildingYokneam 20692IsraelTel: +972-4-909-7200Fax: +972-4-959-3245

© Copyright 2011. Mellanox Technologies. All rights reserved.Mellanox®, BridgeX®, ConnectX®, InfiniBlast®, InfiniBridge®, InfiniHost®, InfiniRISC®, InfiniScale®, InfiniPCI®, PhyX® and Virtual Protocol Interconnect® are registered trademarks of Mellanox Technologies, Ltd. CORE-Direct and FabricIT are trademarks of Mellanox Technologies, Ltd. All other marks and names mentioned herein may be trademarks of their respective companies.

ConnectX-2 Ethernet Card User Manual

Document Number: 3160

Rev 1.6

Mellanox Technologies2

NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Page 3: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 3

Table of Contents

Table of Contents 3List of Figures 5List of Tables 6Revision History 7About this Manual 8

Intended Audience 8Related Documentation 8Online Resources 8Document Conventions 8Technical Support 8Firmware and Software Updates 9

Chapter 1 Overview 101.1 Adapter Cards Covered in this Manual 111.2 Mellanox Part Numbering Legend 131.3 Finding the MAC and Serial Number on the Adapter Cards 141.4 Safety Warnings 14

Chapter 2 Adapter Card Interfaces 162.1 I/O Interfaces 162.2 Power 182.3 Memory 182.4 VPD Layout 19

Chapter 3 Driver Software and Firmware 223.1 Driver Software 223.2 RDMA over Converged Ethernet (RoCE) 223.3 Updating Card Firmware 233.4 ConnectX EN PXE 24

Chapter 4 Ethernet Adapter Card Installation 254.1 Hardware and Software Requirements 254.2 Installation Instructions 254.3 Set Up 264.4 Cables and Modules 31

Appendix A Specifications 34 A.1 MNPH29D-X[ST]R Specifications 34 A.2 MNPH29C-X[ST]R Specifications 35 A.3 Board Mechanical Drawing and Dimensions 35 A.4 EMC Certification Statements 37

Appendix B Interface Connectors Pinout 40 B.1 I2C-Compatible Connector Pinout 40 B.2 PCI Express x8Connector Pinout 40 B.3 PCI Express Connector Pinout 41 B.4 SFP+ Connector Pinout 41

Appendix C Replacing a Tall Bracket With a Short Bracket 43 C.1 Remove the Existing Bracket from the Adapter Card 43 C.2 Installing the New Bracket 43

Appendix D Avertissements de sécurité d’installation (Warnings in French) 45

Page 4: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

Rev 1.6

Mellanox Technologies4

Appendix E Sicherheitshinweise (Warnings in German) 46Appendix F Advertencias de seguridad para la instalación (Warnings in Spanish) 47

Page 5: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 5

List of Figures

Figure 1: SFP+ NIC Component Side and Bracket Views 12Figure 2: Card Product Label 14Figure 3: Port Numbering 16Figure 4: LED - Port Association 17Figure 5: Physical and Logical Link Indications 17Figure 6: I2C Connector 18Figure 7: Flash Jumper 19Figure 8: Support Download Assistant 24Figure 9: Hardware Devices 27Figure 10: PCI Device 28Figure 11: Module With Locking Mechanism Closed 32Figure 12: Module With Locking Mechanism Open 32Figure 13: Schematic of the ConnectX-2 Small Form Factor MNPH29D Adapter Card 36Figure 14: Schematic of the ConnectX-2 Card MNPH29C Adapter Card 37Figure 15: Compatible Connector Plug and Pinout 40Figure 16: Rear View of Module With Pin Placement 41Figure 17: Remove the Bracket 43Figure 18: Putting on the Bracket 44

Page 6: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

Rev 1.6

Mellanox Technologies6

List of Tables

Table 1: Revision History Table 7Table 2: Documents List 8Table 3: Network Interface Cards 11Table 4: Mellanox Cards Part Numbering Key 13Table 5: LEDs 17Table 6: Jumper Configuration 19Table 7: VPD Layout for MNPH29D-X[ST]R 19Table 8: VPD Layout for MNPH29C-X[ST]R 20Table 9: Hardware and Software Requirements 25Table 10: Specifications for MNPH29D-X[ST]R 34Table 11: Specifications for MNPH29C-X[ST]R 35Table 12: Adapter Cards Certification Status 37Table 13: I2C-Compatible Connector Pinout 40Table 14: SFP+ Connector Pinout 42

Page 7: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 7

Revision History

This document was printed on 2/15/11.

Table 1 - Revision History Table

Date Rev Comments/Changes

Jan. 2011 1.6 Changed Air flow requirementAdded to the Set up section

Dec. 19th, 2010

1.5 Minor formatting changes

Nov. 22nd, 2010

1.4 Added Generation D card Mini Form Factor

Oct. 28th, 2010

1.3 Added Technical Support and Firmware and Software Updates sections to the Pref-ace sectionAdded RoCEAdded ConnectX EN PXEAdded Set up section, This is a sanity check after card installation.Added KCC statement to Appendix AAdded QSFP to SFP+ QSA adapter Removed Gen 1 cardsFormatted EMC tableChanged EN 60950-1: 2001 to EN 60950-1: 2006Changed RDMA over EthernetAdded RDMA over EthernetNew template

April 2010 1.2 New power numbers

Feb. 2010 1.1 Added Section on PXE boot

Feb. 2010 1.0 Initial Release

Page 8: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

Rev 1.6

Mellanox Technologies8

About this Manual

This User Manual describes Mellanox Technologies ConnectX®-2 Dual Port 10Gigabit Ethernet PCI Express x8 adapter cards. It provides details as to the interfaces of the board, specifications, required software and firmware for operating the board, and relevant documentation.

Intended Audience

This manual is intended for the installer and user of these cards.

The manual assumes basic familiarity with Ethernet networks and architecture specifications.

Related Documentation

Table 2 - Documents List

Online Resources• Mellanox Technologies Web pages: http://www.mellanox.com• Mellanox Technologies Firmware download Web page:

http://www.mellanox.com =>

Document Conventions

When discussing memory sizes, MB and MBytes are used in this document to mean size in mega bytes. The use of Mb or Mbits (small b) indicates size in mega bits.

Technical Support

Customers who purchased Mellanox products directly from Mellanox are invited to contact us through the following methods.• URL: http://www.mellanox.com => Support • E-mail: [email protected]

Mellanox Firmware Tools (MFT) User ManualDocument no. 2204UG

User Manual describing the set of MFT firmware management tools for a single node. See http://www.mellanox.com => Firmware & Downloads => Download Firmware ToolsOr http://www.mellanox.com => Support => Download Firmware Tools

IEEE Std 802.3 Specification This is the IEEE Ethernet specification http://standards.ieee.org/getieee802

PCI Express 2.0 Specifications Industry Standard PCI Express 2.0 Card Electromechanical Specification, Rev 1.3.

Page 9: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 9

• Tel: +1.408.916.0055

Customers who purchased Mellanox M-1 Global Support Services, please see your contract for details regarding Technical Support.

Customers who purchased Mellanox products through a Mellanox approved reseller should first seek assistance through their reseller.

Firmware and Software Updates

The Mellanox support downloader contains software, firmware and knowledge database informa-tion for Mellanox products. Access the data base from the Mellanox Support Web page,

http://www.mellanox.com=>Support

or use the following link to go directly to the Mellanox Support Download Assistant page,

http://www.mellanox.com/supportdownloader/.

Page 10: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

OverviewRev 1.6

Mellanox Technologies10

1 Overview

This document is a User Manual for Mellanox Technologies cards based on the ConnectX®-2ENintegrated circuit device. The cards described in this manual have the following main features:• IEEE Std 802.3 compliant• 10 Gb/s Ethernet • Dual SFP+ ports for connection Ethernet traffic• PCI Express 2.0 (1.1 compatible) through an x8 edge connector up to 5GT/s• Fiber Channel encapsulation (FCoE)• RDMA over Converged Ethernet (RoCE)• RoHS-R6• Two bracket heights: short and tall

Page 11: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 11

1.1 Adapter Cards Covered in this Manual

Table 3 lists the Ethernet adapter cards described in this manual.

Table 3 - Network Interface Cards

Ordering Part Number (OPN)

PCI Express SERDES

Speed

Data Transmission Rate/

# of ports

Short / Tall Bracket RoHS Adapter IC Part Number

MNPH29D-XTR PCIe Gen2 5.0 GT/s

10 +10 Gb/sEthernet

2 ports

Tall R-6 MT25408B0-FCCR-TE

MNPH29D-XSR PCIe Gen2 5.0 GT/s

10 +10 Gb/sEthernet

2 ports

Short R-6 MT25408B0-FCCR-TE

MNPH29C-XTR PCIe Gen2 5.0 GT/s

10 +10 Gb/sEthernet

2 ports

Tall R-6 MT25408B0-FCCR-TE

MNPH29C-XSR PCIe Gen2 5.0 GT/s

10 +10 Gb/sEthernet

2 ports

Short R-6 MT25408B0-FCCR-TE

Page 12: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

OverviewRev 1.6

Mellanox Technologies12

Figure 1: SFP+ NIC Component Side and Bracket Views

J1

U1MNPH29C

J1

U1MNPH29D

Page 13: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 13

1.2 Mellanox Part Numbering Legend

Table 4 describes the Mellanox Technologies adapter cards part numbering legend.

For example, the part number MNPH29C-XTR describes Mellanox Technologies’ ConnectX EN adapter card with two SFP+ ports, a PCIe2.0 x8 5.0GT/s interface, no on-board memory (mem-free), a tall bracket, and RoHS R6 compliance. Using the legend,• field M = M to indicate a Mellanox Technologies product,• field H = N to indicate a Network Interface Card,• field T = P to indicate 10GBASE-SR/LR,• field S = H to indicate ConnectX • field # = 2 to indicate two ports,• field I = 9 to indicate PCI Express 2.0 x8 running at 5.0GT/s,• field G = C to indicate Generation C ConnectX-2• field X = X to indicate no on-board memory,• field B = T to indicate a tall bracket, and• field R = R to indicate RoHS R6 compliance

Table 4 - Mellanox Cards Part Numbering Key

Adapter Card OPN

MHTS#I-XBRField Decoder

M Mellanox Technologies

H Adapter Type H = InfiniBand Host Channel Adapter, N = Ethernet Network Interface Card

T Media P = SFP+,

S Adapter Architecture

H = ConnectX®or ConnectX-2

# # ports 1 = 1, 2 = 2

I Host Interface X = PCI-X, 4 = PCIe x4, 8 = PCIe Gen1 x8, 9 = PCIe (Gen2 x8),

G Generation <blank> = Initial product generation, B= generation B, C= generation C

- Separator

X Memory Size X = MemFree, 1=128MB, 2=256MB, 3=512MB

B Bracket S = Short, T = Tall, N = None

R RoHS <blank> = non RoHS, C = RoHS R-5 w/ Exemption, R = RoHS R-6 Lead-Free

Page 14: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

OverviewRev 1.6

Mellanox Technologies14

1.3 Finding the MAC and Serial Number on the Adapter Cards

All Mellanox adapter cards have a label on the printed side of the adapter card that has the card serial number, and the card MAC.

Figure 2: Card Product Label

1.4 Safety Warnings

For safety warnings in French see “Avertissements de sécurité d’installation (Warnings in French)” on page 45. For safety warnings in German see “Sicherheitshinweise (Warnings in German)” on page 46. For safety warnings in Spanish see “Advertencias de seguridad para la instalación (Warn-ings in Spanish)” on page 47.

1. Installation Instructions

2. Over-temperature

Port 1 uses the GUID or MAC ID described on the label, for port 2 GUID or MAC add 1 to port 1's description.

Read all installation instructions before connecting the equipment to the power source.

This equipment should not be operated in an area with an ambient temperature exceed-ing the maximum recommended: 55°C (131°F). To guarantee proper air flow, allow at least 8cm (3 inches) of clearance around the ven-tilation openings.

S/N:MT0744X00012

P/N:MNPH29C-XTRREV: X1

Made in IL

MAC: 0002C9020040DDB0

Page 15: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 15

3. During Lightning - Electrical Hazard

4. Copper Cable Connecting/Disconnecting

5. Equipment Installation

6. Equipment Disposal

7. Local and National Electrical Codes

During periods of lightning activity, do not work on the equipment or connect or dis-connect cables.

Some copper cables are heavy and not flexible, as such they should be carefully attached to or detached from the connectors. Refer to the cable manufacturer for spe-cial warnings and instructions.

This equipment should be installed, replaced, or serviced only by trained and qualified personnel.

Disposal of this equipment should be in accordance to all national laws and regula-tions.

This equipment should be installed in compliance with local and national electrical codes.

Page 16: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

Adapter Card InterfacesRev 1.6

Mellanox Technologies16

2 Adapter Card Interfaces

2.1 I/O Interfaces

Each card includes the following interfaces:

• SFP+ ports• I/O panel LEDs

• I2C compatible connector (for debug)

For dual port cards, port 1 connects to connector 1 of the device, while port 2 connects to connec-tor 2 of the device.

ConnectX-2 Ethernet Adapter Cards are compliant with the IEEE Std 802.3 Specification.

Figure 3: Port Numbering

2.1.1 Ethernet SFP+ Interface

The ConnectX®-2 device is compliant with the IEEE Std 802.3. The SFP+ port has one Tx/Rx pair of SerDes.Ethernet NIC cards listed in Table 3 on page 11 based on this device provide access to the Ethernet ports by means of SFP+ connectors.

2.1.2 PCI Express Interface

The ConnectX-2 adapter cards support PCI Express 2.0 (1.1 compatible) through an x8 edge con-nector. The device can be either a master initiating the PCI Express bus operations or a slave responding to PCI bus operations.

Port 2

Port 1

Page 17: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 17

2.1.3 LED Assignment

The board has I/O LEDs located on the I/O panel. The green LED, when lit, indicates that the driver is running and a valid physical connection between nodes exists. If the green LED is blink-ing, it indicates a problem with the physical link. The yellow LED when lit, indicates a valid data activity link, this is the logical link. The yellow LED lights up when the network is discovered over the physical link. A valid data activity link without data transfer is designated by a constant yellow LED indication. A valid data activity link with data transfer is designated by a blinking yel-low LED indication. If the LEDs are not active, either the physical link or the logical link (or both) connections have not been established.

Figure 4: LED - Port Association

Figure 5: Physical and Logical Link Indications

Port 1

Port 2

Table 5 - LEDs

Port Number LED Name

Port 1 Physical Link - Green Constant on indicates a good physical link

Blinking indicates a problem with the Physical link

Data Activity - YellowBlinking indicates Data Transfer

Constant on indicates no Data Transfer

Port 2 Physical Link - Green Constant on indicates a good physical link

Blinking indicates a problem with the Physical link

Data Activity - YellowBlinking indicates Data Transfer

Constant on indicates no Data Transfer

Port 1

Port 2

Page 18: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

Adapter Card InterfacesRev 1.6

Mellanox Technologies18

2.1.4 I2C Compatible Interface

A three-pin header on the card is provided as the I2C compatible interface. See Appendix A,“Specifications,” on page 34 for the location on the board.

Figure 6: I2C Connector

2.2 Power

All adapter cards receive 12V and 3.3V power from the PCI Express Edge connector. All other required power voltages are generated by on-board switch mode regulators. See “Specifications” on page 34.

2.3 Memory

The adapter cards support multiple memory devices through the PCI Flash, and I2C.

2.3.1 System Memory

The card utilizes the PCI Express interface to store and access Ethernet fabric connection informa-tion and packet data on the system memory.

2.3.2 Flash

Each of the cards include one SPI Flash device accessible via the Flash interface of theMT25408B0 ConnectX-2 EN device.

There is a jumper on each adapter card that indicates to the device whether an on-board Flash device is to be used. Table 6 provides information on this jumper. See the schematic in Appendix A,“Specifications,” on page 34 for the jumper location.

The short bracket has the same port and LED footprint as the tall bracket.

Page 19: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 19

2.3.3 EEPROM

Each board incorporates an EEPROM that is accessible through the I2C. The EEPROM is used for storing the Vital Product Data (VPD). The EEPROM capacity is 4Kb.

2.4 VPD Layout

The PCI VPD (Vital Product Data) layout, for each of the described Mellanox Technologies Con-nectX®-2 adapter cards complies with the format defined in the PCI 2.3 Specification, Appendix I. All ConnectX-2® adapter cards have the same PCI VPD layout.

Table 6 - Jumper Configuration

Description Option Card Default Configuration Comments

Flash present/ not present

connection open – Flash presentconnection shorted – Flash not present

Figure 7: Flash Jumper

connection open – Flash present

Header 1x2

Table 7 - VPD Layout for MNPH29D-X[ST]R

Offset (Decimal) Item Value Format Description

0 Large Resource Type ID String Tag (0x02)

0x82

1 Length [7:0] LSB 0xE

2 Length [15:8] MSB 0x0

3 Data Hawk Dual Port STR

17 Large Resource Type VPD-R Tag (0x10)

0x90

18 Length [7:0] LSB 0x4F

19 Length [15:8] MSB 0x00

20 VPD Keyword PN STR Add in Card Part Number

22 Length 0x15

23 PN PN %STR_SPC

44 VPD Keyword EC STR Engineering Change Level of the card (rev)

46 Length 0x2

47 Revision RV %STR PCB revision

49 VPD Keyword SN STR Serial Number

51 Length 0x18

52 SerialNumber %STR_SPC “00..00XXXX..XX”

Page 20: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

Adapter Card InterfacesRev 1.6

Mellanox Technologies20

76 VPD Keyword V0 STR Misc Information

78 Length 0x10

79 Data PCIe Gen2 x8 STR_SPC

95 VPD Keyword RV STR

97 Length 0x1

98 Data 0,97 %CS0

99 Large Resource Type VPD-W Tag (0x11)

0x91

100 Length [7:0] LSB 0x99

101 Length [15:8] MSB 0x00

102 VPD Keyword V1 STR EFI Driver version

104 Length 0x6

105 Data N/A STR_SPC

111 VPD Keyword YA STR Asset Tag

113 Length 0x20

114 Data N/A STR_SPC “N/A”

146 VPD Keyword RW STR Remaining read/write area

148 Length 0x6A

149 Data STR_ZERO Reserved (0x00)

255 Small Resource Type END Tag (0x11) 0x78

Table 8 - VPD Layout for MNPH29C-X[ST]R

Offset (Decimal) Item Value Format Description

0 Large Resource Type ID String Tag (0x02) 0x82

1 Length [7:0] LSB 0xE

2 Length [15:8] MSB 0x0

3 Data Hawk Dual Port STR

17 Large Resource Type VPD-R Tag (0x10) 0x90

18 Length [7:0] LSB 0x4F

19 Length [15:8] MSB 0x00

20 VPD Keyword PN STR Add in Card Part Number

22 Length 0x15

23 PN PN %STR_SPC

44 VPD Keyword EC STR Engineering Change Level of the card (rev)

46 Length 0x2

47 Revision RV %STR PCB revision

49 VPD Keyword SN STR Serial Number

51 Length 0x18

52 SerialNumber %STR_SPC “00..00XXXX..XX”

Table 7 - VPD Layout for MNPH29D-X[ST]R

Offset (Decimal) Item Value Format Description

Page 21: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 21

76 VPD Keyword V0 STR Misc Information

78 Length 0x10

79 Data PCIe Gen2 x8 STR_SPC

95 VPD Keyword RV STR

97 Length 0x1

98 Data 0,97 %CS0

99 Large Resource Type VPD-W Tag (0x11) 0x91

100 Length [7:0] LSB 0x99

101 Length [15:8] MSB 0x00

102 VPD Keyword V1 STR EFI Driver version

104 Length 0x6

105 Data N/A STR_SPC

111 VPD Keyword YA STR Asset Tag

113 Length 0x20

114 Data N/A STR_SPC “N/A”

146 VPD Keyword RW STR Remaining read/write area

148 Length 0x6A

149 Data STR_ZERO Reserved (0x00)

255 Small Resource Type END Tag (0x11) 0x78

Table 8 - VPD Layout for MNPH29C-X[ST]R

Offset (Decimal) Item Value Format Description

Page 22: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

Driver Software and FirmwareRev 1.6

Mellanox Technologies22

3 Driver Software and Firmware

3.1 Driver Software

3.1.1 Linux

For Linux, download and install the latest MLNX_EN driver software package available via the Mellanox Web site at: http://www.mellanox.com => Downloads => Ethernet SW/Drivers. Follow the installation instruc-tions included in the download package.

3.1.2 Windows

For Windows, download the MLNX EN – Ethernet driver. Download this package from the Mel-lanox Web site at: http://www.mellanox.com => Downloads => Ethernet SW/Drivers. Follow the installation instruc-tions included in the download package.

3.2 RDMA over Converged Ethernet (RoCE)

ConnectX-2 EN with RoCE utilizes advances in Data Center Bridging (DCB) to enable efficient and low cost implementations of RDMA over Ethernet, supporting the entire breadth of RDMA and low latency features. This includes reliable connected service, datagram service, RDMA and send/receive semantics, atomic operations, user level multicast, user level I/O access, kernel bypass, and zero copy.

ConnectX-2 EN with RoCE based network management is the same as that for any Ethernet and DCB-based network management, eliminating the need for IT managers to learn new technolo-gies.

3.2.1 Hardware and Software Requirements

Software:Operating System

• Mellanox OFED 1.5.1 or later

Hardware:• ConnectX-2 EN Adapter Card

To use RoCE you will need the following versions of firmware and software:• OFED 1.5.1 or later• firmware version 2.7.700 or later

3.2.2 Installation

For installation instructions visit: www.mellanox.com => Support => Ethernet SW/Driver

Page 23: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 23

3.3 Updating Card Firmware

Each card is shipped with the latest version of qualified firmware at the time of manufacturing. Firmware is updated occasionally, and the most recent firmware can be obtained from: http://www.mellanox.com =>Support > Download Center. Check that the firmware on your card is the latest found on the Mellanox site, if not update to the latest version found on the Mellanox website.

Firmware can be updated on the stand alone single card using the flint tool of the Mellanox Firm-ware Tools (MFT) package. This package is available for download, along with its user manual, from the Mellanox Firmware Tools page. See http://www.mellanox.com => Support > Download Center.

A firmware binaries table lists a binary file per adapter card. The file name of each such binary is composed by combining the firmware name, the firmware release version, and the card part num-ber. Please contact Mellanox system support if you cannot find the firmware binary for your adapter card.

To check the latest firmware:1. Go to Mellanox web site to check current firmware version.

Go to: http://www.mellanox.com/supportdownloader2. Enter your card PSID and compare the firmware versions.

Page 24: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

Driver Software and FirmwareRev 1.6

Mellanox Technologies24

Figure 8: Support Download Assistant

3.4 ConnectX EN PXE

The Mellanox ConnectX EN PXE solution enables the booting of servers from an IP based LAN environment.

To enable or disable PXE use the following procedure:

The user must use the MFT tools:

To disable one must delete the PXE ROM image from the flash.

To enable one must place the PXE ROM image to the flash.

This software is based on the open source Etherboot/gPXE project (see www.etherboot.org).

For more details go to http://www.mellanox.com Products => Ethernet SW/Drivers > ConnectX EN PXE.

Page 25: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 25

4 Ethernet Adapter Card Installation

4.1 Hardware and Software Requirements

Before installing the adapter card, please make sure that the system meets the hardware and soft-ware requirements listed in Table 9. Refer to Chapter 3,“Driver Software and Firmware” on page 22 for download and installation instructions.

4.2 Installation Instructions

To change a tall bracket to a short bracket see Replacing a Tall Bracket With a Short Bracket on page 43.

Read all installation instructions before connecting the equipment to the power source.

The adapter cards listed in Table 3 on page 11 are standard PCI Express cards, each with a stan-dard x8 edge connector. Please consult the host machine documentation for instructions on how to install a PCI Express card.

Table 9 - Hardware and Software Requirements

Requirement Description

Hardware • PCI Express x8 or x16 slots

Firmware and SoftwareOperating Systems/Distributions

• For the latest firmware available please check http://www.mellanox.com => Downloads => Firmware

Management Tools and Drivers• Mellanox management tools can be found at:

http://www.mellanox.com => Downloads => Firmware Tools• Linux Driver for ConnectX EN Based Network Interface Cards with 10GigE Sup-

port can be found at: http://www.mellanox.com => Downloads => Ethernet SW/Drivers

• Windows Driver for ConnectX EN Based Network Interface Cards with 10GigE. Support can be found at: http://www.mellanox.com => Downloads => Ethernet SW/Drivers

• Mellanox Ethernet Driver for Citrix XenServer 4.1, Mellanox ConnectX EN 10GbE Dual port NIC Support can be found at: http://www.mellanox.com => Downloads => Ethernet SW/Drivers

When more than one PCI slot is available make sure to use the PCI slot with the proper configuration.

Page 26: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

Ethernet Adapter Card InstallationRev 1.6

Mellanox Technologies26

Any PCI slot with the proper configuration is acceptable for connection. If the card is installed in a PCI slot with less lanes than the card requires then the adapter card will not provide the optimum data transfer.

4.3 Set Up

The basic steps to embed Mellanox cards in your computer are: 1. Identify the card in your system.2. Verify the computer is recognizing the new adapter. 3. Install the adapter drivers.4. Update the adapter firmware if needed.

The configuration of the card is dependant upon your choice to use InfiniBand or Ethernet.

4.3.1 Identify the Card in Your System

For instructions to identify your adapter card using specific tools see http://www.mellanox.com/content/pages.php?pg=firmware_HCA_FW_identification

Otherwise, use the instructions below relevant to your particular card and OS.

4.3.1.1 Windows1. Open Device Manager. Click start--> Run, and then enter “devmgmt.msc”.2. Check the Device Manager under “Other devices” for “PCI Device” (Windows 2003) or

“InfiniBand Controller” (Windows 2008/R2).

This section is valid for InfiniBand, Ethernet and VPI cards. Disregard sections that are not relevant to your card.

If you cannot find any PCI device, click Action --> Scan for hardware changes. If no PCI devices are detected, check that the network adapter card(s) is correctly installed in the PCI slot or try installing the adapter card into a different PCI slot.

Page 27: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 27

Figure 9: Hardware Devices

3. Select a PCI Device / InfiniBand Controller entry.4. Right-click. 5. Select “Properties to display the PCI Device Properties” window.6. Click the Details tab and select Device Instance Id (Windows 2003) or

Hardware Ids (Windows 2008/R2) from the Property pull-down menu.

Page 28: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

Ethernet Adapter Card InstallationRev 1.6

Mellanox Technologies28

Figure 10: PCI Device

7. In the Value display box, check the fields VEN and DEV (fields are separated by ‘&’). In the display example above, notice the sub-string “PCI\VEN_15B3&DEV_6368”: VEN is equal to 0x15B3 – this is the Vendor ID of Mellanox Technologies; and DEV is equal to 0x6368 – this is a valid Mellanox Technologies PCI Device ID.

8. If the PCI device does not have a Mellanox adapter ID, return to Step 3 to check another device.

The list of Mellanox Technologies PCI Device IDs can be found in the PCI ID reposi-tory at http://pci-ids.ucw.cz/read/PC/15b3.

Page 29: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 29

4.3.1.2 Linux

Get the device location on the PCI bus by running lspci and locating lines with the string “Mel-lanox Technologies”:

> lspci | grep Mellanox 2:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)

Make sure that either the MLNX_OFED driver or the MLNX_EN driver is loaded and configured.

Check the link status

First check the network interface name by running the “ifconfig –a” command.

To check the Ethernet link status, for EN only and VPI cards, run: ethtool <interface>Host# ethtool eth1 Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full Advertised auto-negotiation: Yes Speed: Unknown! (0) Duplex: Half Port: Twisted Pair PHYAD: 1 Transceiver: internal Auto-negotiation: on Supports Wake-on: g Wake-on: d Current message level: 0x000000ff (255) Link detected: yes

To check the IB link status, for IB and VPI cards, run “ibstat” and focus on the Physical state attributes.

Example:Host# ibstatCA 'mlx4_0' CA type: MT26428 Number of ports: 1 Firmware version: 2.7.616 Hardware version: b0 Node GUID: 0x0002c903000c8710 System image GUID: 0x0002c903000c8713 Port 1: State: Down Physical state: Polling

Page 30: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

Ethernet Adapter Card InstallationRev 1.6

Mellanox Technologies30

Rate: 10 Base lid: 6 LMC: 0 SM lid: 3 Capability mask: 0x0251086a Port GUID: 0x0002c903000c8711

Link layer: IB

Check the OFED version

To get the version of the running Mellanox OFED/BXOFED, run the following command:Host# ofed_info | head -1BXOFED-1.5.1-1.3.7-rc19:

Troubleshooting MLNX_OFED Installation

For troubleshooting driver installation, please check Mellanox OFED driver user manual: http://www.mellanox.com => Support > Adapter IB/VPI SW.

If you need to update your card firmware, download the new firmware.

Unzip the downloaded file and run flint/mstflint command:flint -d <device> -i <fw-file> [-guid <GUID> | -guids <4 GUIDS> | -mac <MAC> | -macs <2 MACs>] burn

Check VPI setup – Ethernet / InfiniBand configuration

For Example:flint -d /dev/mst/mt26428_pci_cr0 –i fw-25408-2_7_000-MNPH28B-XTC_A1-A2.bin burn

Loading the Ethernet Driver

By default, the Mellanox OFED stack loads mlx4_en. Run ‘lsmod’ to verify that the module is listed.

Example:Host# lsmod | grep mlx4_enmlx4_en 109708 0mlx4_core 136804 2 mlx4_en,mlx4_ib

If you don’t see the mlx4_en driver, run: ’ modprobe mlx4_en’

Another option is to use the command below to see which modules are active.Edit ”/etc/infiniband/openib.conf” which modules needs to loaded from a service.

For example:# Load MLX4_EN moduleMLX4_EN_LOAD=yes

Page 31: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 31

The “Usage: openibd start|stop|restart|status” command to modify this file and thereby control the drivers.

Ethernet Driver Usage and Configuration

To assign an IP address to the interface run: #> ifconfig eth<n> <ip>

where 'n' is the OS assigned interface number.• To check driver and device information run:

#> ethtool -i eth<n>

Example: #> ethtool -i eth2 driver: mlx4_en (MT_04A0140005) version: 1.5.1 (March 2010) firmware-version: 2.7.000 bus-info: 0000:13:00.0

• The mlx4_en parameters can be found under /sys/module/mlx4_en (or /sys/module/mlx4_en/parameters, depending on the OS) and can be listed using the command:

#> modinfo mlx4_en

To set non-default values to module parameters, the following line should be added to the file/etc/modprobe.conf:

"options mlx4_en <param_name>=<value> <param_name>=<value> ..."

Ethernet Network Tuning

To improve network perfomance by tuning your network see the Mellanox Performance Tuning Guide located at:

http://www.mellanox.com/related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters_rev_1_0.pdf

4.4 Cables and Modules

The SFP+ port can connect to the Ethernet through both direct attach cables and stand alone trans-ceivers with optical cables. SFP+ transceiver modules are available for both SR and LR protocols.

See www.mellanox.com => Products => Cables for certified and approved cable recommenda-tions.

4.4.1 Optical modules for SFP+

SFP+ transceiver modules can be used to connect fiber optic cables to the cards greatly increasing the cable reach.

Page 32: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

Ethernet Adapter Card InstallationRev 1.6

Mellanox Technologies32

The adapter cards are shipped without optical modules. Mellanox 10GBASE-SR (MFM1T02A-SR) and 10GBASE-LR (MFM1T02A-LR) optical modules are recommended. The figure below shows the Mellanox SFP+ module.

Inserting the Optical Transceiver Module

To insert the module into the cage:1. Open the module’s locking mechanism – see Figure 11 and Figure 12.2. Make sure that the male connectors on the module will align with the female connectors inside

of the cage. Also check that there is no dirt or foreign matter in the module or in the cage.

Figure 11: Module With Locking Mechanism Closed

Figure 12: Module With Locking Mechanism Open

3. Insert the module into the adapter card module cage. 4. Close the locking Mechanism.

To remove the module from the cage:1. Unlock the locking mechanism by opening the handle.2. Pull the module out of the cage.

4.4.2 Cable Installation

All cables can be inserted or removed with the unit powered on. To insert a cable, press the con-nector into the port receptacle until the connector is firmly seated. The GREEN LED indicator will light when the physical connection is established (that is, when the unit is powered on and a cable is plugged into the port with the other end of the connector plugged into a functioning port). After plugging in a cable, lock the connector using the latching mechanism particular to the cable ven-dor. When a logical connection is made the YELLOW LED will come on. When data is being transferred the yellow led will blink.

When installing cables make sure that the latches engage.

Page 33: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 33

Care should be taken not to impede the air exhaust flow through the ventilation holes. Cable lengths should be used which allow for routing horizontally around to the side of the chassis before bending upward or downward in the rack.

To remove a cable, disengage the locks and slowly pull the connector away from the port recepta-cle. Both LED indicators will turn off when the cable is unseated.

Always install and remove cables by pushing or pulling the cable and connector in a straight line with the card.

Cables, especially long copper cables, can weigh a substantial amount. Make sure that the weight of the cable is supported on its own and is not hanging from the adapter card.

Page 34: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

Rev 1.6

Mellanox Technologies34

Appendix A: Specifications

A.1 MNPH29D-X[ST]R SpecificationsTable 10 - Specifications for MNPH29D-X[ST]R

Physical Power and Environmental

Size:

Air Flow:

10Gb/s Connector:

2.71in x5.60in (68.90mm x 142.25mm)

200LFMa

SFP+

a. Air flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet.

Voltage:

Typ. Power:

Maximum Power:

Temperature:

12V, 3.3V

Passive Cables 6.43W Active Cables 7.83W

Passive Cables 7.08W Active Cables 8.48W

0°C to 55°C

Protocol Support Regulatory

Ethernet:

PCI Express

IEEE Std 802.3ae 10 Gigabit Ether-netIEEE Std 802.3ad Link Aggregation and FailoverIEEE Std 802.3x PauseIEEE Std 802.1Q VLAN tags, .1p prioritiesIEEE Std 802.1p PrioritiesIEEE P802.1au D2.0 Congestion NotificationIEEE P802.1az D0.2 Enhanced Transmission SelectionIEEE P802.1bb D1.0 Priority-based Flow ControlMulticast

2.0 SerDes @ 5.0 GT/s

EMC:

Safety:

EMI:

RoHS:

FCC 47 CFR part 15:2007, subpart B class A;ICES-003:2004 issue 4, class A;VCCI V-3/2008.04 class A;AS/NZ CISPR 22-2006, class A.EN 55022:1998 + A1:2000 + A2: 2003 class A,EN 55024:1998 + A1:2001 + A2: 2003EN 61000-3-2: 2006,EN 61000-3-3: 1995+A1:2001+A2:2005standards harmonized under EMC Directive 2004/108/EC Article 6(2);

UL, EU: IEC, TUV, and CBEN 60950-1:2006

CLASS 1 LASER PRODUCTIEC 60 825-1:1993 + A1:1997 + A2:2001 EN 60825-1:1994+A1:1996+ A2:2001”

RoHS R6

Page 35: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 35

A.2 MNPH29C-X[ST]R Specifications

A.3 Board Mechanical Drawing and Dimensions

All of the cards covered in this User Manual have the same mechanical drawing and share the same dimensions as depicted in Figure 13 or Figure 14.

Table 11 - Specifications for MNPH29C-X[ST]R

Physical Power and Environmental

Size:

Air Flow:

10Gb/s Connector:

2.71in x6.60in (68.90mm x 167.65mm)

200LFMa

SFP+

a. Air flow is measured ~1” from the heat sink between the heat sink and the cooling air inlet.

Voltage:

Typ. Power:

Maximum Power:

Temperature:

12V, 3.3V

Passive Cables 6.43W Active Cables 7.83W

Passive Cables 7.08W Active Cables 8.48W

0°C to 55°C

Protocol Support Regulatory

Ethernet:

PCI Express

IEEE Std 802.3ae 10 Gigabit Ether-netIEEE Std 802.3ad Link Aggregation and FailoverIEEE Std 802.3x PauseIEEE Std 802.1Q VLAN tags, .1p prioritiesIEEE Std 802.1p PrioritiesIEEE P802.1au D2.0 Congestion NotificationIEEE P802.1az D0.2 Enhanced Transmission SelectionIEEE P802.1bb D1.0 Priority-based Flow ControlMulticast

2.0 SerDes @ 5.0 GT/s

EMC:

Safety:

EMI:

RoHS:

FCC 47 CFR part 15:2007, subpart B class A;ICES-003:2004 issue 4, class A;VCCI V-3/2008.04 class A;AS/NZ CISPR 22-2006, class A.EN 55022:1998 + A1:2000 + A2: 2003 class A,EN 55024:1998 + A1:2001 + A2: 2003EN 61000-3-2: 2006,EN 61000-3-3: 1995+A1:2001+A2:2005standards harmonized under EMC Directive 2004/108/EC Article 6(2);

UL, EU: IEC, TUV, and CBEN 60950-1:2006

CLASS 1 LASER PRODUCTIEC 60 825-1:1993 + A1:1997 + A2:2001 EN 60825-1:1994+A1:1996+ A2:2001”

RoHS R6

Page 36: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

Rev 1.6

Mellanox Technologies36

Figure 13: Schematic of the ConnectX-2 Small Form Factor MNPH29D Adapter Card

All dimensions are in millimeters.All the mechanical tolerances are +/-0.1mm

J4 J3 U1

J1

U3

Q1

L8

U4

U14

45.013.65

1.90

15.00 45.96

142.25

56.15

12.75

64.40

Q2

J7 J8

Page 37: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 37

Figure 14: Schematic of the ConnectX-2 Card MNPH29C Adapter Card

A.4 EMC Certification Statements

Table 12 lists the approved certification status per in different regions of the world.

Table 12 - Adapter Cards Certification Status

Adapter Card P/N FCC VCCI EN ICES CE CB cTUVus KCC C-TICK

MNPH29C-XTR YES YES YES YES YES YES YES YES YES

MNPH29C-XSR YES YES YES YES YES YES YES YES

MNPH29D-XTR YES YES YES YES YES YES YES YES YES

MNPH29D-XSR YES YES YES YES YES YES YES YES

U14

L17

J5

J6J4

J7

U1J1U2 U

3J2

U18

L7

U19

U4

U25

15.00

33.3543.17

57.15

3.651.90

96.30

68.9056.97

J2 – I2C Connector J3 – Flash Jumper

167.65

56.1264.40

Page 38: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

Rev 1.6

Mellanox Technologies38

A.4.1 FCC Statements (USA)

Class A Statements:

§ 15.19(a)(4)

This device complies with Part 15 of the FCC Rules.

Operation is subject to the following two conditions:

1. This device may not cause harmful interference, and2. This device must accept any interference received, including interference that may cause

undesired operation.

§ 15.21

Statement

Warning!

Changes or modifications to this equipment not expressly approved by the party responsible for compliance (Mellanox Technologies) could void the user's authority to operate the equipment.

§15.105(a)

Statement

NOTE: This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to Part 15 of the FCC Rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial envi-ronment. This equipment generates, uses, and can radiate radio frequency energy and, if not installed and used in accordance with the instruction manual, may cause harmful interference to radio communications. Operation of this equipment in a residential area is likely to cause harmful interference in which case the user will be required to correct the interference at his own expense.

A.4.2 EN Statements (Europe)

EN55022 Class A Statement:

Warning

This is a class A product. In a domestic environment this product may cause radio interference in which case the user may be required to take adequate measures.

A.4.3 ICES Statements (Canada)

Class A Statement:

“This Class A digital apparatus complies with Canadian ICES-003. Cet appareil numérique de la classe A est conforme à la norme NMB-003 du Canada.”

Page 39: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 39

A.4.4 VCCI Statements (Japan)

Class A Statement:

(Translation - "This is a Class A product based on the standard of the Voluntary Control Council for Interference by Information Technology Equipment (VCCI). If this equipment is used in a domestic environment, radio interference may occur, in which case the user may be required to take corrective actions.")

A.4.5 KCC Certification (Korea)

Device User’s information

A급 기기

(업무용 방송통신기기 )

CLASS A device

(commercial broadcasting and communication equipment)

이 기기는업무용(A급)으로전자파적합등록을 한기기이오니 판매자또는 사용자는이 점을주의하시기바라며, 가정외의 지역에서사용하는 것을목적으로 합니다.

English Translation

This device has been approved by EMC registration. Distributors or users pay attention to this point . This device is usually aimed to be used in other area except at home .

Remark

Class A device: operated in a commercial area.

Page 40: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

Rev 1.6

Mellanox Technologies40

Appendix B: Interface Connectors Pinout

B.1 I2C-Compatible Connector Pinout

Figure 15: Compatible Connector Plug and Pinout

B.2 PCI Express x8Connector Pinout

These cards use a standard PCI Express x8 edge connector and the PCI Express x8 standard pinout according to the PCI Express 2.0 specification.

5 1 2 3 4

5 1 2 3 4

Table 13 - I2C-Compatible Connector

Connector Pin Number Signal Name

1 SPSDA

2 SPSCL

3 GND

4 NC

5 NC

Page 41: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 41

B.3 PCI Express Connector Pinout

B.4 SFP+ Connector Pinout

Figure 16: Rear View of Module With Pin Placement

13.70

8.50

Top

VeeT

VeeT

VeeT

TX_Fault

TX_D

isable

SDA

SC

L

MO

D_A

BS

RS0

RX_LO

S

RS

1

VeeR

VeeR

VeeR

VccT

VccR

TD+

TD-

RD

-

RD

+

1 2 3 4 5 6 7 8 9

1234567890 1111111112

10

Vee

T

Vee

T

Vee

T

TX_F

ault

TX_D

isab

le

SD

A

SC

L

MO

D_A

BS

RS

0

RX_

LOS

RS1

Vee

R

VeeR

Vee

R

Vcc

T

Vcc

R

TD+

TD-

RD

-

RD

+

123456789

1 2 3 4 5 6 7 8 9 01 1 1 1 1 1 1 1 1 2

10

SFP+ Cage

Page 42: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

Rev 1.6

Mellanox Technologies42

Table 14 - SFP+ Connector Pinout

Pin Symbol Name Description

1 VeeT Transmitter Ground (Common with Receiver Ground) a

a. Circuit ground is internally isolated from chassis ground.

2 TX_Fault Transmitter Fault.b

b. TFAULT is an open collector/drain output, which should be pulled up with a 4.7k – 10k Ohms resistor on the host board if intended for use. Pull up voltage should be between 2.0V to Vcc + 0.3V. A high output indicates a transmitter fault caused by either the TX bias current or the TX output power exceed-ing the preset alarm thresholds. A low output indicates normal operation. In the low state, the output is pulled to <0.8V.

3 TX_Disable Transmitter Disable. Laser output disabled on high or open. c

c. Laser output disabled on TDIS >2.0V or open, enabled on TDIS <0.8V

4 SDA 2-wire Serial Interface Data Line d

d. Should be pulled up with 4.7kΩ – 10kΩ on host board to a voltage between 2.0V and 3.6V. MOD_ABS pulls line low to indicate module is plugged in.

5 SCL 2-wire Serial Interface Clock Line d

6 MOD_ABS Module Absent. Grounded within the module d

7 RS0 No connection required

8 RX_LOS Loss of Signal indication. Logic 0 indicates normal operation. e

e. LOS is open collector output. Should be pulled up with 4.7kΩ – 10kΩ on host board to a voltage between 2.0V and 3.6V. Logic 0 indicates normal operation; logic 1 indicates loss of signal.

9 RS1 No connection required

10 VeeR Receiver Ground (Common with Transmitter Ground) a

11 VeeR Receiver Ground (Common with Transmitter Ground)a

12 RD- Receiver Inverted DATA out. AC Coupled

13 RD+ Receiver Non-inverted DATA out. AC Coupled

14 VeeR Receiver Ground (Common with Transmitter Ground) a

15 VccR Receiver Power Supply

16 VccT Transmitter Power Supply

17 VeeT Transmitter Ground (Common with Receiver Ground) a

18 TD+ Transmitter Non-Inverted DATA in. AC Coupled.

19 TD- Transmitter Inverted DATA in. AC Coupled.

20 VeeT Transmitter Ground (Common with Receiver Ground)a

Page 43: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 43

Appendix C: Replacing a Tall Bracket With a Short Bracket

This section provides instructions on how to remove the tall bracket of a standard Mellanox Tech-nologies adapter card and replace it with a short one. It includes the following sections:• Removing a bracket• Installing a new bracket

C.1 Remove the Existing Bracket from the Adapter Card

Figure 17: Remove the Bracket

1. Remove the two screws holding the bracket in place. 2. Push the bracket off using equal pressure at the top and bottom of the bracket. See Figure 17

3. Save the two screws and the two fiber washers.

C.2 Installing the New Bracket 4. Place the bracket onto the card until the screw holes line up.

5. Screw on the bracket using the screws and washers saved from the procedure above step 1.

Be careful not to put stress on the LEDs.

Do not force the bracket onto the card. You may have to gently push the LEDs using a small screwdriver to align the LEDs with the holes in the bracket.

Two screws with black fiber washers

Once the screws are removed push here with equal pressure on each side.

LEDs

Page 44: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

Rev 1.6

Mellanox Technologies44

Figure 18: Putting on the Bracket

6. Make sure that the LEDs are aligned onto the bracket holes. 7. Use a torque driver to apply up to 2 lbs-in torque on the screws.

Page 45: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 45

Appendix D: Avertissements de sécurité d’installation (Warnings in French)

1. Instructions d’installation

2. Température excessive

3. Orages – dangers électriques

4. Branchement/débranchement des câbles InfiniBand en cuivre

5. Installation du matériel

6. Elimination du matériel

7. Codes électriques locaux et nationaux

Lisez toutes les instructions d’installation avant de brancher le matériel à la source d’alimentation électrique.

Ce matériel ne doit pas fonctionner dans une zone avec une température ambiante dépassant le maximum recommandé de 55°C (131°F). Un flux d’air de 200LFM à cette température ambiante maximale est nécessaire. En outre, pour garantir un bon écoulement de l’air, laissez au moins 8 cm (3 pouces) d’espace libre autour des ouver-tures de ventilation.

Pendant un orage, il ne faut pas utiliser le matériel et il ne faut pas brancher ou débrancher les câbles.

Les câbles InfiniBand en cuivre sont lourds et ne sont pas flexibles, il faut donc faire très attention en les branchant et en les débranchant des connecteurs. Consultez le fab-ricant des câbles pour connaître les mises en garde et les instructions spéciales.

Ce matériel ne doit être installé, remplacé ou entretenu que par du personnel formé et qualifié.

L’élimination de ce matériel doit s’effectuer dans le respect de toutes les législations et réglementations nationales en vigueur.

Ce matériel doit être installé dans le respect des codes électriques locaux et nationaux.

Page 46: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

Rev 1.6

Mellanox Technologies46

Appendix E: Sicherheitshinweise (Warnings in German)

1. Installationsanleitungen

2. Übertemperatur

3. Bei Gewitter - Elektrische Gefahr

4. Anschließen/Trennen von InfiniBand-Kupferkabel

5. Geräteinstallation

6. Geräteentsorgung

7. Regionale und nationale elektrische Bestimmungen

Lesen Sie alle Installationsanleitungen, bevor Sie das Gerät an die Stromversorgung anschließen.

Dieses Gerät sollte nicht in einem Bereich mit einer Umgebungstemperatur über der maximal empfohlenen Temperatur von 55°C (131°F) betrieben werden. Es ist ein Luft-strom von 200 LFM bei maximaler Umgebungstemperatur erforderlich. Außerdem sollten mindestens 8 cm (3 in.) Freiraum um die Belüftungsöffnungen sein, um einen einwandfreien Luftstrom zu gewährleisten.

Arbeiten Sie während eines Gewitters und Blitzschlag nicht am Gerät, schließen Sie keine Kabel an oder ab.

InfiniBand-Kupferkabel sind schwer und nicht flexible. Deshalb müssen sie vorsichtig an die Anschlüsse angebracht bzw. davon getrennt werden. Lesen Sie die speziellen Warnungen und Anleitungen des Kabelherstellers.

Diese Gerät sollte nur von geschultem und qualifiziertem Personal installiert, ausge-tauscht oder gewartet werden.

Die Entsorgung dieses Geräts sollte unter Beachtung aller nationalen Gesetze Bestim-mungen erfolgen.

Dieses Gerät sollte unter Beachtung der regionalen und nationalen elektrischen Bestimmungen installiert werden.

Page 47: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

ConnectX-2 Ethernet Card User Manual Rev 1.6

Mellanox Technologies 47

Appendix F: Advertencias de seguridad para la instalación (Warnings in Spanish)

1. Instrucciones de instalación

2. Sobrecalentamiento

3. Cuando hay rayos: peligro de descarga eléctrica

4. Conexión y desconexión del cable Copper InfiniBand

5. Instalación de equipos

6. Eliminación de equipos

Antes de conectar el equipo a la fuente de alimentación, leer todas las instruc-ciones de instalación.

No se debe utilizar el equipo en un área con una temperatura ambiente superior a la máxima recomendada: 55°C(131°F). Además, para garantizar una circu-lación de aire adecuada, se debe dejar como mínimo un espacio de 8 cm (3 pul-gadas) alrededor de las aberturas de ventilación.

No utilizar el equipo ni conectar o desconectar cables durante períodos de activ-idad de rayos.

Dado que los cables de cobre InfiniBand son pesados y no son flexibles, su con-exión a los conectores y su desconexión se deben efectuar con mucho cuidado. Para ver advertencias o instrucciones especiales, consultar al fabricante del cable.

La instalación, el reemplazo y el mantenimiento de este equipo estarán a cargo únicamente de personal capacitado y competente.

La eliminación definitiva de este equipo se debe efectuar conforme a todas las leyes y reglamentaciones nacionales.

Page 48: ConnectX -2 EN Dual Port SFP+ Ethernet Adapter Card User ... · ConnectX®-2 EN Dual Port SFP+ Ethernet Adapter Card User Manual P/N: MNPH29C-XSR, ... SOFTWARE OR TEST SUITE PRODUCT

Rev 1.6

Mellanox Technologies48

Códigos eléctricos locales y nacionales

Este equipo se debe instalar conforme a los códigos eléctricos locales y nacio-nales.