Microsoft Windows 2016 Mellanox 100GbE NIC Tuning Guidedeveloper.amd.com/wp-content/resources/56288_1.00.pdf · Microsoft® Windows® 2016 Mellanox 100GbE NIC Tuning Guide 56288 Rev

Microsoft® Windows® 2016 Mellanox

100GbE NIC Tuning Guide

Publication # 56288 Revision: 1.00 Issue Date: June 2018

Advanced Micro Devices

© 2018 Advanced Micro Devices, Inc. All rights reserved.

The information contained herein is for informational purposes only, and is subject to change without notice. While

every precaution has been taken in the preparation of this document, it may contain technical inaccuracies, omissions

and typographical errors, and AMD is under no obligation to update or otherwise correct this information. Advanced

Micro Devices, Inc. makes no representations or warranties with respect to the accuracy or completeness of the

contents of this document, and assumes no liability of any kind, including the implied warranties of noninfringement,

merchantability or fitness for particular purposes, with respect to the operation or use of AMD hardware, software

or other products described herein. No license, including implied or arising by estoppel, to any intellectual property

rights is granted by this document. Terms and limitations applicable to the purchase or use of AMD’s products are

as set forth in a signed agreement between the parties or in AMD's Standard Terms and Conditions of Sale.

Trademarks

AMD, the AMD Arrow logo, EPYC, and combinations thereof, are trademarks of Advanced Micro Devices, Inc.

Other product names used in this publication are for identification purposes only and may be trademarks of their

respective companies.

Microsoft and Windows are registered trademarks of Microsoft Corporation.

PCIe is a registered trademark of PCI-SIG corporation.

56288 Rev. 1.00 June 2018 Microsoft® Windows® 2016 Mellanox 100GbE

NIC Tuning Guide

Revision History 3

Revision History

Date Revision Description

June 2018 1.00 Initial public release.

Microsoft® Windows® 2016 Mellanox 100GbE NIC

Tuning Guide

56288 Rev. 1.00 June 2018

4

Contents

Overview ......................................................................................................................................... 5

Initial Test Systems Preparation ..................................................................................................... 5

Download and Install the Mellanox Driver .................................................................................... 5

Initial Steps ..................................................................................................................................... 6

Determine the NUMA Node That the Mellanox Card is Installed On .......................................... 7

Find the Index Value of the Network Interface .............................................................................. 8

Adding Registry Values .................................................................................................................. 9

Changing Advanced Settings in the Driver .................................................................................. 10

Test Optimizations ........................................................................................................................ 13

Conclusion .................................................................................................................................... 13


NIC Tuning Guide

5

Overview

This guide is focused on tuning Mellanox ConnectX-4 cards and Windows® 2016 for high

bandwidth runs. First, the latest Mellanox driver will be installed, and then the driver is tuned.

Tuning the driver results in a significant performance improvement as we optimize the operating

system.

Initial Test Systems Preparation

1. Both systems should be configured as follows:

a. Hardware

i. AMD EPYC™ 7601 for best CPU performance.

ii. One DIMM per channel using the highest supported memory

configuration by your platform provider.

iii. ConnectX-4 in a 16-bit PCIe® slot running at PCIe Gen 3 (8GT/s) – one of

the steps below will show how to verify this

iv. Both ConnectX-4s directly connected and configured with IP addresses.

b. BIOS

i. C-States disabled (see next step)

c. Set the operating system power plan option found in the control panel to high

performance.

Download and Install the Mellanox Driver

1. Download the Mellanox driver from:

http://www.mellanox.com/page/products_dyn?product_family=32&mtag=windows_sw_drivers

You will need the latest WinOF-2 Driver version.

2. Double-click on the downloaded file and accept all the defaults. It may update the firmware

if the ConnectX-4 card that is installed is behind on its firmware version.

3. After it completes reboot the machine.

http://www.mellanox.com/page/products_dyn?product_family=32&mtag=windows_sw_drivers


Tuning Guide

56288 Rev. 1.00 June 2018

6

Initial Steps

1. Open Device Manager and select the Mellanox ConnectX-4 that you wish to tune. In this

example it is the ConnectX-4 #3.

2. Right-click on the card, select Properties, then select the Information tab.

3. Make sure the card negotiated to PCIe 8.0 GT/s x16 and it negotiated to 100.0 Gbps.


NIC Tuning Guide

7

Determine the NUMA Node That the Mellanox Card is

Installed On

1. Go to Device Manager and select the Mellanox ConnectX-4 that you wish to tune, in this

example it will be the Mellanox ConnectX-4 Adapter #3:

2. Right click on the selected card, go to Properties, then select the Details tab.

3. From there click the Property drop down and press the letter “N” which will then display the

NUMA node property. Document the NUMA node that the card is attached to. This

example shows NUMA node 6. This number is dependent on the PCIe slot the ConnectX-4

is installed in.


Tuning Guide

56288 Rev. 1.00 June 2018

8

Find the Index Value of the Network Interface

1. Open Device Manager

2. Right click on the selected card, go to Properties and then select the Details tab.

3. Select the Driver key and document the value.

4. The driver key value in your system may be slightly different from the example shown.


NIC Tuning Guide

9

Adding Registry Values

1. Adding these registry values is necessary with Mellanox v1.80 driver. The newer driver

v1.90 (and newer) automatically sets the registry values when installed

2. Add the registry values to the following key:

HKLM\SYSTEM\CurrentControlSet\Control\Class\{4d36e972-e325-11ce-bfc1-

08002be10318}\0006\

The part in bold text, 0006, will be specific to your machine.

3. Right-click on 0006 and then add the following new string values:

RscMaxPacketSize – 16384

RfdBufferSize - 16384

RscTimeoutPeriodUsecs – 64


Tuning Guide

56288 Rev. 1.00 June 2018

10

4. This example shows 0004. Select the folder which corresponds to driver key identified

earlier. In our case, it was 0006. The example photo shows 0004. It will likely be different

in your case.

Changing Advanced Settings in the Driver

1. Open Device Manager and select the Mellanox ConnectX-4 that you wish to tune. In this

example it is Mellanox ConnectX-4 Adapter #3.


NIC Tuning Guide

11

2. Right-click on the selected card, go to Properties, then select the Advanced tab.

3. Change "Receive Buffers" from Advanced property page to 4096.


Tuning Guide

56288 Rev. 1.00 June 2018

12

4. Change "Receive Completion Method" to Polling in the Advanced property page.


NIC Tuning Guide

13

Test Optimizations

On the server side, from a command prompt window enter “start /node 2 /affinity 0xAAAA iperf

-s” from the folder in which iperf resides. This example shows Node 2. You should select the

node that corresponds to the NUMA node identified above.

On the TX side run “start /Node 2 /Affinity 0xAAAA iperf -c <server ip> -P 8 -t <number of

seconds>”. Run this for at least three minutes as there is an important step to complete before

iperf completes.

Conclusion

Following these procedures, AMD achieved transfer rates of up to 94.5 Gb/s in our laboratories

using version 2.09 of IPERF.

Documents

Microsoft Windows 2016 Mellanox 100GbE NIC Tuning Guidedeveloper.amd.com/wp-content/resources/56288_1.00.pdf · Microsoft® Windows® 2016 Mellanox 100GbE NIC Tuning Guide 56288 Rev