Intel® Omni-Path Fabric Software Installation Guide · Intel® Omni-Path Fabric Software Installation Guide Rev. 8.0 ... 4.1.1 Fabric Design Prerequisites.....30 4.1.2 Fabric Setup

Intel® Omni-Path Fabric SoftwareInstallation Guide

Rev. 8.0

October 2017

Order No.: H76467-8.0

You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products describedherein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosedherein.

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest Intel product specifications androadmaps.

The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications.Current characterized errata are available on request.

Copies of documents which have an order number and are referenced in this document may be obtained by calling 1-800-548-4725 or visit http://www.intel.com/design/literature.htm.

Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation.Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer orretailer or learn more at intel.com.

No computer system can be absolutely secure.

Intel, the Intel logo, Intel Xeon Phi, and Xeon are trademarks of Intel Corporation in the U.S. and/or other countries.

*Other names and brands may be claimed as the property of others.

Copyright © 2015–2017, Intel Corporation. All rights reserved.

Intel® Omni-Path Fabric SoftwareInstallation Guide October 20172 Order No.: H76467-8.0

http://www.intel.com/design/literature.htm

http://www.intel.com/design/literature.htm

http://intel.com

Revision HistoryFor the latest documentation, go to: www.intel.com/omnipath/FabricSoftwarePublications.

Date Revision Description

October 2017 8.0 Document has been updated with the following:• Updated and added information to the Fabric Design Prerequisites

section• Added "Sandia-OpenSHMEM (hfi, gcc)" to the OPA-Basic Package

section• Updated section OS RPMs Installation Prerequisites to add RHEL*

7.4, SLES* 12.3 and an example of a install command• Added sandiashmem to the "Supported Component (comp) Names"

list in Using the ./INSTALL Command Line Options section• Updated Upgrade Prerequisites to include link to OS RPMs

Installation Prerequisites• Removed outdated appendix, "HFI UEFI PXE Installation and

Configuration"• Added new Appendix, Setting Up PXE Boot for Supported Operating

Systems• Moved Upgrade the HFI UEFI Firmware under Setting Up PXE Boot

for Supported Operating Systems and updated the section• The Intel® Omni-Path Fabric Suite FastFabricCommand Line

Interface Reference Guide has been merged into the Intel® Omni-Path Fabric Host Software User Guide and the Intel® Omni-PathFabric Suite FastFabric User Guide. In this document, all referenceshave been updated appropriately. See the Intel® Omni-PathDocumentation Library on page 11 for details.

August 2017 7.0 Document has been updated with the following:• Globally, updated the following filepaths:

— from /etc/sysconfig/opa to /etc/opa• Added high level installation flowchart to section Installation

Overview and updated the information• Updated section Fabric Manager GUI Packages• Updated section OS RPMs Installation Prerequisites• Added section Download and Install NVIDIA* Software (Optional)• Updated section Install the Intel® Omni-Path Software• Updated section Install Using the Linux* Distribution Software• Updated section Install Intel® Omni-Path Fabric Suite Fabric

Manager GUI

April 2017 6.0 Document has been updated with the following:• Document has been restructured and rewritten for usability.• Globally, updated the following filepaths:

— from /usr/lib/opa-fm/samples/ to /usr/share/opa-fm/samples/

— from /usr/lib/opa-fm/etc/opafm* to /usr/lib/opa-fm/bin/opafm*

— from /etc/sysconfig/opafm.xml to /etc/opa-fm/opafm.xml

— from /etc/sysconfig/* to /etc/*— from /usr/lib/opa/ to /usr/share/opa/— from /usr/lib/opa/src/* to /usr/src/opa/*

continued...

Revision History—Intel® Omni-Path Fabric

Intel® Omni-Path Fabric SoftwareOctober 2017 Installation GuideOrder No.: H76467-8.0 3

Date Revision Description

• Added Intel® Omni-Path Documentation Library to Preface.• Added new section Intel® Omni-Path Architecture Overview.• Added new section Install Using the Linux* Distribution Software.• Added new section Performance Tuning Prerequisites.• Deprecated opaxlattopology_cust in Fabric Setup Prerequisites.• Added new section Update the Fabric Manager opafm Configuration

File.• Added new section Thermal Monitoring Module Verification.• Updated Installation Checklist to include configuration and

verification pointers.

Dec 2016 5.0 Document has been updated with the following:• Removed all information pertaining to RHEL* 7.1• Updated "Prepare Fabric Manager for Fabric Manager GUI"• Added new section "Multi-Rail Usage" to Chapter "Additional

Installation and Setup Tasks"• Moved Appendix "Multi-Subnet Fabrics" under new Chapter

"Additional Installation and Setup Tasks"• Updated "HFI UEFI PXE Installation and Configuration" section• Removed Appendix "hfidiags User Guide". Moved to Intel® Omni-

Path Fabric Host Software User Guide.

Aug 2016 4.0 Document has been updated with the following:• Added information on UEFI firmware Prerequisites• Updated Install TUI menus• Updated ./INSTALL command information to include --user-space• Updated Fabric Manager GUI Installation information• Updated "HFI UEFI PXE Installation and Configuration" section• Updated "hfidiags User Guide" section

May 2016 3.0 Document has been updated with the following:• Updated OS RPMs Installation Prerequisites• Added MPI Compiler Prerequisites• Added information about minimum disk space requirement for the

Fabric Manager GUI installation• Replaced "Client/Server Configuration to Boot Over Fabric" section

with "HFI UEFI PXE Installation and Configuration"

February 2016 2.0 Document has been updated with the following:• Updated OS RPMs Installation Prerequisites• Added "Client/Server Configuration to Boot Over Fabric" appendix

November 2015 1.0 Initial Release

Intel® Omni-Path Fabric—Revision History


Contents

Revision History..................................................................................................................3

Preface............................................................................................................................. 11Intended Audience..................................................................................................... 11Intel® Omni-Path Documentation Library.......................................................................11Cluster Configurator for Intel® Omni-Path Fabric............................................................ 13Documentation Conventions........................................................................................ 13Laser Safety Information.............................................................................................14Electrostatic Discharge Sensitivity (ESDS) Precautions.................................................... 15License Agreements....................................................................................................15Technical Support.......................................................................................................15

1.0 Introduction................................................................................................................161.1 Document Organization......................................................................................... 16

2.0 Overview.................................................................................................................... 182.1 Intel® Omni-Path Architecture Overview.................................................................. 18

2.1.1 Host Fabric Interface................................................................................. 202.1.2 Intel® OPA Switches..................................................................................202.1.3 Intel® OPA Management............................................................................ 21

2.2 Intel® Omni-Path Software Overview.......................................................................212.3 Installation Overview.............................................................................................232.4 Installation Packages.............................................................................................24

2.4.1 OPA-Basic Package....................................................................................242.4.2 OPA-IFS Package...................................................................................... 242.4.3 Fabric Manager GUI Packages.....................................................................25

3.0 Checklists................................................................................................................... 263.1 OPA-Basic Checklists............................................................................................. 26

3.1.1 OPA-Basic Pre-Installation Checklist............................................................ 263.1.2 Install the OPA-Basic Software Checklist...................................................... 263.1.3 Upgrade the OPA-Basic Software Checklist................................................... 27

3.2 OPA-IFS Checklists................................................................................................273.2.1 OPA-IFS Pre-Installation Checklist...............................................................273.2.2 Install and Configure the OPA-IFS Software Checklist.................................... 273.2.3 Upgrade the OPA-IFS Software Checklist......................................................28

Part 1: Installing the Software...................................................... 29

4.0 Installation Getting Started........................................................................................ 304.1 Pre-Installation Requirements.................................................................................30

4.1.1 Fabric Design Prerequisites.........................................................................304.1.2 Fabric Setup Prerequisites..........................................................................314.1.3 OS RPMs Installation Prerequisites.............................................................. 334.1.4 MPI Compiler Prerequisites.........................................................................384.1.5 Performance Tuning Prerequisites............................................................... 38

4.2 Download the Intel® Omni-Path Software.................................................................394.3 Unpack the Tar File............................................................................................... 39

Contents—Intel® Omni-Path Fabric


4.4 Download and Install NVIDIA* Software (Optional)................................................... 40

5.0 Install the Intel® Omni-Path Software........................................................................ 415.1 Before You Begin.................................................................................................. 41

5.1.1 Using the ./INSTALL Command Line Options.................................................415.2 Install Using the TUI Menus....................................................................................455.3 Install Using CLI Commands...................................................................................465.4 Install Using the Linux* Distribution Software...........................................................475.5 Configure IPoIB IPV6.............................................................................................49

Part 2: Configuring the Software................................................... 51

6.0 Configuration Getting Started..................................................................................... 526.1 Pre-Configuration Requirements............................................................................. 52

6.1.1 Chassis Configuration Prerequisites............................................................. 526.1.2 Switch Configuration Prerequisites...............................................................53

6.2 How to Use the FastFabric TUI................................................................................53

7.0 Configure the Chassis................................................................................................. 567.1 Before You Begin.................................................................................................. 567.2 Configure the Chassis Using the FastFabric TUI Menu.................................................577.3 Configure the Chassis Using the FastFabric CLI Commands.........................................61

8.0 Configure the Externally-Managed Switches............................................................... 668.1 Before You Begin.................................................................................................. 668.2 Configure the Externally-Managed Switches Using FastFabric TUI Menu........................678.3 Configure the Externally-Managed Switches Using FastFabric CLI Commands................69

9.0 Install Host Software on Remaining Servers...............................................................719.1 Before You Begin.................................................................................................. 719.2 Install the Host Software on the Remaining Hosts Using the FastFabric TUI Menu.......... 739.3 Verify the Host Software on the Remaining Servers Using the FastFabric TUI Menu........769.4 Install the Host Software on the Remaining Hosts Using the FastFabric CLI Commands.. 799.5 Verify the Host Software on the Remaining Servers Using CLI Commands.................... 81

10.0 Set Up Additional Management Nodes.......................................................................8510.1 Before You Begin................................................................................................ 8510.2 Configure Additional Management Nodes Using FastFabric TUI Menus.........................8510.3 Configure Additional Management Nodes Using FastFabric CLI Commands.................. 87

11.0 Perform Initial Health Check.....................................................................................8811.1 Before You Begin................................................................................................ 8811.2 Configure and Initialize Health Check Tools Using FastFabric TUI Menu.......................8911.3 Configure and Initialize Health Check Tools Using FastFabric CLI Commands............... 90

12.0 Perform High Performance Linpack Benchmark ....................................................... 9212.1 Before You Begin................................................................................................ 9312.2 Run a Sample High Performance Linpack 2 Configuration..........................................9312.3 Generate and Run a Custom High Performance Linpack Configuration........................ 94

13.0 Install Intel® Omni-Path Fabric Suite Fabric Manager GUI........................................9613.1 Before You Begin................................................................................................ 96

13.1.1 System Requirements for a Windows* Environment.....................................96

Intel® Omni-Path Fabric—Contents


13.1.2 System Requirements for a Linux* Environment..........................................9613.1.3 Download the Fabric Manager Software......................................................97

13.2 Install the Fabric Manager GUI Software on Windows*............................................. 9713.3 Install the Fabric Manager GUI Software on Linux*.................................................. 9913.4 Prepare Fabric Manager for Fabric Manager GUI.................................................... 10013.5 Start the Fabric Manager GUI on Windows*.......................................................... 10113.6 Start the Fabric Manager GUI on Linux*............................................................... 10213.7 Uninstall Fabric Manager GUI.............................................................................. 102

13.7.1 Uninstall Fabric Manager GUI Software on Windows*..................................10313.7.1.1 Clear the Fabric Manager GUI Cache on Windows*.........................103

13.7.2 Uninstall Fabric Manager GUI Software on Linux*...................................... 10413.7.2.1 Clear the Fabric Manager GUI Cache on Linux*..............................104

14.0 Additional Installation and Setup Tasks.................................................................. 10614.1 Multi-Subnet Fabrics.......................................................................................... 106

14.1.1 Setting Up Primarily Independent Subnets................................................ 10614.1.2 Setting Up Overlapping Subnets.............................................................. 108

14.2 Multi-Rail Usage................................................................................................ 11014.2.1 Multi-Rail Overview................................................................................11014.2.2 Setting Up Dual Rails for a Single Subnet..................................................11214.2.3 Configuring Dual Rails for Dual Subnets....................................................114

15.0 Installation Verification and Additional Settings..................................................... 11715.1 LED Link and Data Indicators.............................................................................. 11715.2 Thermal Monitoring Module Verification................................................................ 11715.3 Adapter and Other Settings................................................................................ 11815.4 ARP Neighbor Table Setup for Large Clusters........................................................ 11815.5 SM Loop Test....................................................................................................119

Part 3: Upgrading the Software................................................... 120

16.0 Upgrade Getting Started......................................................................................... 12116.1 Upgrade Prerequisites........................................................................................ 12116.2 Download the Intel® Omni-Path Software............................................................. 12116.3 Unpack the Tar File............................................................................................122

17.0 Upgrade the Intel® Omni-Path Software................................................................. 12317.1 Before You Begin...............................................................................................12317.2 Upgrade Scenarios............................................................................................ 12317.3 Upgrade the OPA-Basic Software......................................................................... 12317.4 Upgrade the OPA-IFS Software............................................................................124

17.4.1 Upgrade the Software on the Management Nodes...................................... 12417.4.2 Update the Fabric Manager opafm Configuration File.................................. 12517.4.3 Upgrade the Software on the Remaining Servers....................................... 126

17.5 Upgrade from OPA-Basic to OPA-IFS Software Package.......................................... 128

Appendix A UEFI Secure Boot Support............................................................................ 129A.1 UEFI Firmware Support........................................................................................129A.2 Linux* Support...................................................................................................129

Appendix B Setting Up PXE Boot for Supported Operating Systems................................ 130B.1 PXE Server.........................................................................................................131

Contents—Intel® Omni-Path Fabric


B.1.1 Configure PXE Server with RHEL* 7.4........................................................ 131B.1.1.1 Before You Begin........................................................................ 131B.1.1.2 Configure Network Interface......................................................... 131B.1.1.3 Configure DHCP Server................................................................ 132B.1.1.4 Install a TFTP Server................................................................... 134B.1.1.5 Create Directory Structure for PXE Server ..................................... 135B.1.1.6 Configure the HTTP Server........................................................... 135B.1.1.7 Configure the Boot Loader............................................................136

B.1.2 Configure PXE Server with SLES* 12 SP3................................................... 136B.1.2.1 Before You Begin........................................................................ 136B.1.2.2 Configure Network Interface......................................................... 137B.1.2.3 Configure DHCP Server................................................................ 137B.1.2.4 Install a TFTP Server................................................................... 140B.1.2.5 Create Directory Structure for PXE Server ..................................... 140B.1.2.6 Configure the HTTP Server........................................................... 141B.1.2.7 Configure the Boot Loader............................................................141

B.2 Client Node........................................................................................................ 142B.2.1 Before You Begin.................................................................................... 143B.2.2 Upgrade the HFI UEFI Firmware................................................................ 143B.2.3 Install RHEL* 7.4 over Intel® OP HFI......................................................... 145B.2.4 Install SLES* 12 SP3 over Intel® OP HFI.................................................... 146

Intel® Omni-Path Fabric—Contents


Figures1 Intel® OPA Fabric.................................................................................................... 192 Intel® OPA Building Blocks........................................................................................203 Intel® OPA Fabric and Software Components...............................................................22

Figures—Intel® Omni-Path Fabric


Tables1 Red Hat* Enterprise Linux* OS Distribution RPMs Matrix...............................................342 SUSE* Linux* Enterprise Server OS Distribution RPMs Matrix........................................ 363 BIOS versus Boot Mode.......................................................................................... 142

Intel® Omni-Path Fabric—Tables


Preface

This manual is part of the documentation set for the Intel® Omni-Path Fabric (Intel®OP Fabric), which is an end-to-end solution consisting of Intel® Omni-Path Host FabricInterfaces (HFIs), Intel® Omni-Path switches, and fabric management anddevelopment tools.

The Intel® OP Fabric delivers a platform for the next generation of High-PerformanceComputing (HPC) systems that is designed to cost-effectively meet the scale, density,and reliability requirements of large-scale HPC clusters.

Both the Intel® OP Fabric and standard InfiniBand* are able to send Internet Protocol(IP) traffic over the fabric, or IPoFabric. In this document, however, it is referred to asIP over IB or IPoIB. From a software point of view, IPoFabric and IPoIB behave thesame way and, in fact, use the same ib_ipoib driver to send IP traffic over the ib0and/or ib1 ports.

Intended Audience

The intended audience for the Intel® Omni-Path (Intel® OP) document set is networkadministrators and other qualified personnel.

Intel® Omni-Path Documentation Library

Intel® Omni-Path publications are available at the following URLs:

• Intel® Omni-Path Switches Installation, User, and Reference Guides

http://www.intel.com/omnipath/SwitchPublications

• Intel® Omni-Path Software Installation, User, and Reference Guides (includes HFIdocuments)

http://www.intel.com/omnipath/FabricSoftwarePublications

• Drivers and Software (including Release Notes)

http://www.intel.com/omnipath/Downloads

Use the tasks listed in this table to find the corresponding Intel® Omni-Pathdocument.

Task Document Title Description

Key:Shading indicates the URL to use for accessing the particular document.

• Intel® Omni-Path Switches Installation, User, and Reference Guides: http://www.intel.com/omnipath/SwitchPublications

• Intel® Omni-Path Software Installation, User, and Reference Guides (includes HFI documents):http://www.intel.com/omnipath/FabricSoftwarePublications (no shading)

• Drivers and Software (including Release Notes): http://www.intel.com/omnipath/Downloads

continued...

Preface—Intel® Omni-Path Fabric









Setting up an Intel®OPA cluster

Intel® Omni-Path Fabric Setup Guide(Old title: Intel® Omni-Path FabricStaging Guide)

Provides a high level overview of the steps required to stagea customer-based installation of the Intel® Omni-Path Fabric.Procedures and key reference documents, such as Intel®Omni-Path user guides and installation guides are provided toclarify the process. Additional commands and BKMs aredefined to facilitate the installation process andtroubleshooting.

Installing hardware

Intel® Omni-Path Fabric SwitchesHardware Installation Guide

Describes the hardware installation and initial configurationtasks for the Intel® Omni-Path Switches 100 Series. Thisincludes: Intel® Omni-Path Edge Switches 100 Series, 24 and48-port configurable Edge switches, and Intel® Omni-PathDirector Class Switches 100 Series.

Intel® Omni-Path Host Fabric InterfaceInstallation Guide

Contains instructions for installing the HFI in an Intel® OPAcluster. A cluster is defined as a collection of nodes, eachattached to a fabric through the Intel interconnect. The Intel®HFI utilizes Intel® Omni-Path switches and cabling.

Installing hostsoftwareInstalling HFIfirmwareInstalling switchfirmware (externally-managed switches)

Intel® Omni-Path Fabric SoftwareInstallation Guide

Describes using a Text-based User Interface (TUI) to guideyou through the installation process. You have the option ofusing command line interface (CLI) commands to perform theinstallation or install using the Linux* distribution software.

Managing a switchusing Chassis ViewerGUIInstalling switchfirmware (managedswitches)

Intel® Omni-Path Fabric Switches GUIUser Guide

Describes the Intel® Omni-Path Fabric Chassis Viewergraphical user interface (GUI). It provides task-orientedprocedures for configuring and managing the Intel® Omni-Path Switch family.Help: GUI online help.

Managing a switchusing the CLIInstalling switchfirmware (managedswitches)

Intel® Omni-Path Fabric SwitchesCommand Line Interface ReferenceGuide

Describes the command line interface (CLI) task informationfor the Intel® Omni-Path Switch family.Help: -help for each CLI.

Managing a fabricusing FastFabric

Intel® Omni-Path Fabric SuiteFastFabric User Guide(Merged with: Intel® Omni-Path FabricSuite FastFabric Command LineInterface Reference Guide)

Provides instructions for using the set of fabric managementtools designed to simplify and optimize common fabricmanagement tasks. The management tools consist of TUImenus and command line interface (CLI) commands.Help: -help and man pages for each CLI. Also, all host CLIcommands can be accessed as console help in the FabricManager GUI.

Managing a fabricusing Fabric Manager

Intel® Omni-Path Fabric Suite FabricManager User Guide

The Fabric Manager uses a well defined management protocolto communicate with management agents in every Intel®Omni-Path Host Fabric Interface (HFI) and switch. Throughthese interfaces the Fabric Manager is able to discover,configure, and monitor the fabric.

Intel® Omni-Path Fabric Suite FabricManager GUI User Guide

Provides an intuitive, scalable dashboard and set of analysistools for graphically monitoring fabric status andconfiguration. It is a user-friendly alternative to traditionalcommand-line tools for day-to-day monitoring of fabrichealth.Help: Fabric Manager GUI Online Help.

Configuring andadministering Intel®HFI and IPoIB driver

Intel® Omni-Path Fabric Host SoftwareUser Guide

Describes how to set up and administer the Host FabricInterface (HFI) after the software has been installed. Theaudience for this document includes both cluster

continued...

Intel® Omni-Path Fabric—Preface



Running MPIapplications onIntel® OPA

administrators and Message-Passing Interface (MPI)application programmers, who have different but overlappinginterests in the details of the technology.

Writing and runningmiddleware thatuses Intel® OPA

Intel® Performance Scaled Messaging2 (PSM2) Programmer's Guide

Provides a reference for programmers working with the Intel®PSM2 Application Programming Interface (API). ThePerformance Scaled Messaging 2 API (PSM2 API) is a low-level user-level communications interface.

Optimizing systemperformance

Intel® Omni-Path Fabric PerformanceTuning User Guide

Describes BIOS settings and parameters that have beenshown to ensure best performance, or make performancemore consistent, on Intel® Omni-Path Architecture. If you areinterested in benchmarking the performance of your system,these tips may help you obtain better performance.

Designing an IP orstorage router onIntel® OPA

Intel® Omni-Path IP and StorageRouter Design Guide

Describes how to install, configure, and administer an IPoIBrouter solution (Linux* IP or LNet) for inter-operatingbetween Intel® Omni-Path and a legacy InfiniBand* fabric.

Building a Lustre*Server using Intel®OPA

Building Lustre* Servers with Intel®Omni-Path Architecture ApplicationNote

Describes the steps to build and test a Lustre* system (MGS,MDT, MDS, OSS, OST, client) from the HPDD master branchon a x86_64, RHEL*/CentOS* 7.1 machine.

Building Containersfor Intel® OPAfabrics

Building Containers for Intel® Omni-Path Fabrics using Docker* andSingularity* Application Note

Provides basic information for building and running Docker*and Singularity* containers on Linux*-based computerplatforms that incorporate Intel® Omni-Path networkingtechnology.

Writing managementapplications thatinterface with Intel®OPA

Intel® Omni-Path Management APIProgrammer’s Guide

Contains a reference for programmers working with theIntel® Omni-Path Architecture Management (Intel OPAMGT)Application Programming Interface (API). The Intel OPAMGTAPI is a C-API permitting in-band and out-of-band queries ofthe FM's Subnet Administrator and PerformanceAdministrator.

Learning about newrelease features,open issues, andresolved issues for aparticular release

Intel® Omni-Path Fabric Software Release Notes

Intel® Omni-Path Fabric Manager GUI Release Notes

Intel® Omni-Path Fabric Switches Release Notes (includes managed and externally-managed switches)

Cluster Configurator for Intel® Omni-Path Fabric

The Cluster Configurator for Intel® Omni-Path Fabric is available at: http://www.intel.com/content/www/us/en/high-performance-computing-fabrics/omni-path-configurator.html.

This tool generates sample cluster configurations based on key cluster attributes,including a side-by-side comparison of up to four cluster configurations. The tool alsogenerates parts lists and cluster diagrams.

Documentation Conventions

The following conventions are standard for Intel® Omni-Path documentation:

• Note: provides additional information.

• Caution: indicates the presence of a hazard that has the potential of causingdamage to data or equipment.

• Warning: indicates the presence of a hazard that has the potential of causingpersonal injury.



http://www.intel.com/content/www/us/en/high-performance-computing-fabrics/omni-path-configurator.html



• Text in blue font indicates a hyperlink (jump) to a figure, table, or section in thisguide. Links to websites are also shown in blue. For example:

See License Agreements on page 15 for more information.

For more information, visit www.intel.com.

• Text in bold font indicates user interface elements such as menu items, buttons,check boxes, key names, key strokes, or column headings. For example:

Click the Start button, point to Programs, point to Accessories, and then clickCommand Prompt.

Press CTRL+P and then press the UP ARROW key.

• Text in Courier font indicates a file name, directory path, or command line text.For example:

Enter the following command: sh ./install.bin• Text in italics indicates terms, emphasis, variables, or document titles. For

example:

Refer to Intel® Omni-Path Fabric Software Installation Guide for details.

In this document, the term chassis refers to a managed switch.

Procedures and information may be marked with one of the following qualifications:

• (Linux) – Tasks are only applicable when Linux* is being used.

• (Host) – Tasks are only applicable when Intel® Omni-Path Fabric Host Softwareor Intel® Omni-Path Fabric Suite is being used on the hosts.

• (Switch) – Tasks are applicable only when Intel® Omni-Path Switches or Chassisare being used.

• Tasks that are generally applicable to all environments are not marked.

Laser Safety Information

This product may use Class 1 laser optical transceivers to communicate over the fiberoptic conductors. The U.S. Department of Health and Human Services (DHHS) doesnot consider Class 1 lasers to be hazardous. The International ElectrotechnicalCommission (IEC) 825 Laser Safety Standard requires labeling in English, German,Finnish, and French stating that the product uses Class 1 lasers. Because it isimpractical to label the transceivers, the following label is provided in this manual.

Intel® Omni-Path Fabric—Preface


http://www.intel.com.

Electrostatic Discharge Sensitivity (ESDS) Precautions

The assemblies used in the switch chassis are ESD sensitive. Observe ESD handlingprocedures when handling any assembly used in the switch chassis.

License Agreements

This software is provided under one or more license agreements. Please refer to thelicense agreement(s) provided with the software for specific detail. Do not install oruse the software until you have carefully read and agree to the terms and conditionsof the license agreement(s). By loading or using the software, you agree to the termsof the license agreement(s). If you do not wish to so agree, do not install or use thesoftware.

Technical Support

Technical support for Intel® Omni-Path products is available 24 hours a day, 365 daysa year. Please contact Intel Customer Support or visit http://www.intel.com/omnipath/support for additional detail.



http://www.intel.com/omnipath/support

http://www.intel.com/omnipath/support

1.0 Introduction

This guide provides instructions for installing the Intel® Omni-Path Architecture(Intel® OPA) software and configuring the system for the first time. It also providesinstructions for upgrading the software.

For details about the other documents for the Intel® Omni-Path product line, refer to Intel® Omni-Path Documentation Library in this document.

You install the software using one of the following methods:

• Guided installation using Text User Interface (TUI) menus (recommended)

• Command Line Interface (CLI) commands

• Linux* Distribution Software

Note: The default Intel® OPA configuration during installation is for a single-subnet fabric.

To set up other fabric configurations, please contact your Intel Support Personnel. Foradditional information, refer to Multi-Subnet Fabrics on page 106.

Document Organization

This manual is organized as follows:

• Introduction

• Overview

• Checklists

Part 1: Installing the Software

• Installation Getting Started

• Install Intel® Omni-Path Software

Part 2: Configuring the Software

• Configuration Getting Started

• Configure the Chassis

• Configure the Externally Managed Switches

• Install Host Software on Remaining Servers

• Set Up Additional Management Nodes

• Perform Initial Health Check

• Perform High Performance Linpack Benchmark

• Install Intel® Omni-Path Fabric Suite Fabric Manager GUI

• Additional Installation and Setup Tasks

• Installation Verification and Additional Settings

1.1

Intel® Omni-Path Fabric—Introduction


Part 3: Upgrading the Software

• Upgrade Getting Started

• Upgrading the Intel® Omni-Path Software

Appendices

• UEFI Secure Boot Support

• HFI UEFI PXE Installation and Configuration

Introduction—Intel® Omni-Path Fabric


2.0 Overview

This section provides an overview of the Intel® Omni-Path Architecture and softwareinstallation process.

Intel® Omni-Path Architecture Overview

The Intel® Omni-Path Architecture (Intel® OPA) interconnect fabric design enables abroad class of multiple node computational applications requiring scalable, tightly-coupled processing, memory, and storage resources. Options for close "on-package"integration between Intel® OPA family devices, Intel® Xeon® Processors, and Intel®Xeon Phi™ Processors, enable significant system level packaging and networkefficiency improvements. When coupled with open standard APIs developed by theOpenFabrics Alliance* (OFA) Open Fabrics Interface (OFI) workgroup, host fabricinterfaces (HFIs) and switches in the Intel® OPA family systems are optimized toprovide the low latency, high bandwidth, and high message rate needed by large scaleHigh Performance Computing (HPC) applications.

Intel® OPA provides innovations for a multi-generation, scalable fabric, including linklayer reliability, extended fabric addressing, and optimizations for many-coreprocessors. High performance datacenter needs are also a core Intel® OPA focus,including link level traffic flow optimization to minimize datacenter-wide jitter for highpriority packets, robust partitioning support, quality of service support, and acentralized fabric management system.

The following figure shows a sample Intel® OPA-based fabric, consisting of differenttypes of nodes and servers.

2.1

Intel® Omni-Path Fabric—Overview


Figure 1. Intel® OPA Fabric

To enable the largest scale systems in both HPC and the datacenter, fabric reliability isenhanced by combining the link level retry typically found in HPC fabrics with theconventional end-to-end retry used in traditional networks. Layer 2 networkaddressing is extended for systems with over ten million endpoints, thereby enablinguse on the largest scale datacenters for years to come.

To enable support for a breadth of topologies, Intel® OPA provides mechanisms forpackets to change virtual lanes as they progress through the fabric. In addition, higherpriority packets are able to preempt lower priority packets to provide more predictablesystem performance, especially when multiple applications are runningsimultaneously. Finally, fabric partitioning is provided to isolate traffic between jobs orbetween users.

The software ecosystem is built around OFA software and includes four key APIs.

1. The OFA OFI represents a long term direction for high performance user level andkernel level network APIs.

Overview—Intel® Omni-Path Fabric


2. The Performance Scaled Messaging 2 (PSM2) API provides HPC-focused transportsand an evolutionary software path from the Intel® True Scale Fabric.

3. OFA Verbs provides support for existing remote direct memory access (RDMA)applications and includes extensions to support Intel® OPA fabric management.

4. Sockets is supported via OFA IPoFabric (also called IPoIB) and rSockets interfaces.This permits many existing applications to immediately run on Intel® Omni-Pathas well as provide TCP/IP features such as IP routing and network bonding.

Higher level communication libraries, such as the Message Passing Interface (MPI),and Partitioned Global Address Space (PGAS) libraries, are layered on top of these lowlevel OFA APIs. This permits existing HPC applications to immediately take advantageof advanced Intel® Omni-Path features.

Intel® Omni-Path Architecture is an end-to-end solution consisting of Intel® Omni-Path Host Fabric Interfaces (HFIs), Intel® Omni-Path switches, and fabricmanagement and development tools. These building blocks are shown in the followingfigure.

Figure 2. Intel® OPA Building Blocks

Switch

Node 0

HFI

Switch

(Service)Node y

HFI

Fabric Manager

Switch

Node x

HFI

Additional Links and Switches

Host Fabric Interface

Each host is connected to the fabric through a Host Fabric Interface (HFI) adapter.The HFI translates instructions between the host processor and the fabric. The HFIincludes the logic necessary to implement the physical and link layers of the fabricarchitecture, so that a node can attach to a fabric and send and receive packets toother servers or devices. HFIs also include specialized logic for executing andaccelerating upper layer protocols.

Intel® OPA Switches

Intel® OPA switches are OSI Layer 2 (link layer) devices, and act as packet forwardingmechanisms within a single Intel® OPA fabric. Intel® OPA switches are responsible forimplementing Quality of Service (QoS) features, such as virtual lanes, congestionmanagement, and adaptive routing. Switches are centrally managed by the Intel®Omni-Path Fabric Suite Fabric Manager software, and each switch includes a

2.1.1

2.1.2



management agent to handle management transactions. Central management meansthat switch configurations are programmed by the FM software, including managingthe forwarding tables to implement specific fabric topologies, configuring the QoS andsecurity parameters, and providing alternate routes for adaptive routing. As such, allOPA switches must include management agents to communicate with the Intel® OPAFabric Manager.

Intel® OPA Management

The Intel® OPA fabric is centrally managed and supports redundant Fabric Managersthat manage every device (server and switch) in the fabric through managementagents associated with those devices. The Primary Fabric Manager is an Intel® OPAfabric software component selected during the fabric initialization process.

The Primary Fabric Manager is responsible for:

1. Discovering the fabric's topology.

2. Setting up Fabric addressing and other necessary values needed for operating thefabric.

3. Creating and populating the Switch forwarding tables.

4. Maintaining the Fabric Management Database.

5. Monitoring fabric utilization, performance, and statistics rates.

The fabric is managed by sending management packets over the fabric. These packetsare sent in-band (that is, over the same wires as regular network packets) usingdedicated buffers on a specific virtual lane (VL15). End-to-end reliability protocols areused to detect lost packets.

Intel® Omni-Path Software Overview

For software applications, Intel® OPA maintains consistency and compatibility withexisting Intel® True Scale Fabric and InfiniBand* APIs utilizing the open sourceOpenFabrics Alliance* (OFA) software stack on Linux* distribution releases.

Software Components

The key software components and their usage models are shown in the followingfigure and described in the following paragraphs.

2.1.3

2.2



Figure 3. Intel® OPA Fabric and Software Components

Software Component Descriptions

Element Management Stack• Runs on an embedded Intel processor included in managed Intel® OP Edge Switch 100 Series and Intel®

Omni-Path Director Class Switch 100 Series switches.• Provides system management capabilities, including signal integrity, thermal monitoring, and voltage

monitoring, among others.• Accessed via Ethernet* port using command line interface (CLI) or graphical user interface (GUI).User documents:• Intel® Omni-Path Fabric Switches GUI User Guide• Intel® Omni-Path Fabric Switches Command Line Interface Reference Guide

Host Software Stack• Runs on all Intel® OPA-connected host nodes and supports compute, management, and I/O nodes.• Provides a rich set of APIs including OFI, PSM2, sockets, and OFA verbs.• Provides high performance, highly scalable MPI implementation via OFA, PSM2, and an extensive set of

upper layer protocols.• Includes Boot over Fabric mechanism for configuring a server to boot over Intel® Omni-Path using the

Intel® OP HFI Unified Extensible Firmware Interface (UEFI) firmware.User documents:• Intel® Omni-Path Fabric Host Software User Guide

continued...



Software Component Descriptions

• Intel® Performance Scaled Messaging 2 (PSM2) Programmer's Guide

Fabric Management Stack• Runs on Intel® OPA-connected management nodes or embedded Intel processor on the switch.• Initializes, configures, and monitors the fabric routing, QoS, security, and performance.• Includes a toolkit for configuration, monitoring, diagnostics, and repair.User documents:• Intel® Omni-Path Fabric Suite Fabric Manager User Guide• Intel® Omni-Path Fabric Suite FastFabric User Guide

Fabric Management GUI• Runs on laptop or workstation with a local screen and keyboard.• Provides interactive GUI access to Fabric Management features such as configuration, monitoring,

diagnostics, and element management drill down.User documents:• Intel® Omni-Path Fabric Suite Fabric Manager GUI Online Help• Intel® Omni-Path Fabric Suite Fabric Manager GUI User Guide

Installation Overview

The standard software installation process takes you through installing the Intel®Omni-Path Fabric (OPA-Basic) or Intel® Omni-Path Fabric Suite (OPA-IFS) softwarepackage, configuring the system, and verifying the system settings.

Perform all Prerequisites

Installing Management

Node?

Install the Omni-Path IFS Software

Configure the fabric

Install the OPA Basic Software on Remaining Nodes

Set up additional Management

Nodes (optional)

Install OPA FM GUI on

connected PC

Install the Omni-Path Basic Software

No

Yes

Intel recommends that you install the OPA-IFS software on the Management Nodeusing the Install TUI, and then use FastFabric to configure the Management Node.

Once the Management Node has been configured, the Basic software can be installedon all the remaining hosts using either the FastFabric TUI or a provisioning or disklessboot mechanism.

Note: If you are using a provisioning system, consult the documentation that comes with theprovisioning system.

2.3



Installation Packages

The following software installation packages are available for an Intel® Omni-PathFabric.

OPA-Basic Package

The IntelOPABasic.DISTRO.VERSION.tgz installation package (whereDISTRO.VERSION is the OS distribution and software version) installs the Intel®Omni-Path Fabric Host Software components needed to set up compute, I/O, andService nodes with drivers, stacks, and basic tools for local configuration andmonitoring.

The installation package includes the following components:

• Intel® Omni-Path Architecture (Intel® OPA) Stack (Deltas relative to thedistribution of the OpenFabrics Alliance* (OFA) stack)

• IBACM (Deltas relative to the distribution of the OFA stack)

• Intel HFI Components

• Intel® OPA Tools

• Intel® OPA OFA Development

• IP over IB (Deltas relative to the distribution of the OFA stack)

• MVAPICH2 (hfi, gcc)

• MVAPICH2 (hfi, Intel)

• OpenMPI (hfi, gcc)

• OpenMPI (hfi, Intel)

• GASNet (hfi, gcc)

• OpenSHMEM (hfi, gcc)

• Sandia-OpenSHMEM (hfi, gcc)

• MVAPICH2 (verbs, gcc)

• OpenMPI (verbs, gcc)

• MPI Source

• Pre-Boot components

• OFA Debug Info

Note: A separate Intel® Omni-Path Fabric Host Software installation package is available foreach of the supported Linux* distributions. Refer to the release notes of the packageversion being installed for a list of supported Linux* distributions.

OPA-IFS Package

The IntelOPA-IFS.DISTRO.VERSION.tgz installation package (whereDISTRO.VERSION is the OS distribution and software version) provides the Intel®Omni-Path Fabric Host Software package along with special features including theIntel® Omni-Path Fabric Suite FastFabric toolset and the Intel® Omni-Path FabricSuite Fabric Manager.

2.4

2.4.1

2.4.2



The installation package includes the following components:

• OPA-Basic package

For the list of components, refer to OPA-Basic Package.

• Intel® Omni-Path Fabric Suite FastFabric

Refer to the Intel® Omni-Path Fabric Suite FastFabric User Guide for more details.

• Intel® Omni-Path Fabric Suite Fabric Manager

Refer to the Intel® Omni-Path Fabric Suite Fabric Manager User Guide for moredetails.

Note: A separate IFS installation package is available for each of the supported Linux*distributions. Refer to the release notes of the version being installed for a list ofsupported Linux* distributions.

Fabric Manager GUI Packages

The following packages are used to install the Intel® Omni-Path Fabric Suite FabricManager GUI

• Linux* installation package:

IntelOPA-FMGUI-linux-x.x.x.x.x.noarch.rpm• Windows* installation package:

IntelOPA-FMGUI.windows-x.x.x.x.x.exeThe Intel® Omni-Path Fabric Suite Fabric Manager GUI provides a set of features forviewing and monitoring the fabric or multiple fabrics. The application is installed on acomputer outside of the fabric.

2.4.3



3.0 Checklists

This section includes checklists to help you track tasks during fabric installation,configuration, and upgrade.

OPA-Basic Checklists

OPA-Basic Pre-Installation Checklist

Step Description Complete

1. Ensure that hardware is installed, cabled, and powered.Refer to the Intel® Omni-Path Fabric Switches Hardware Installation Guide.

2. Ensure that an HFI is installed in each server.Refer to the Intel® Omni-Path Host Fabric Interface Installation Guide.

3. The hardware configuration should be reviewed to ensure everything is installed properly, according tothe plan.Refer to the local hardware configuration plan.

4. Ensure that the required operating system is installed on each server with the following options:• Root user command prompt ends in "#" or "$".

Note: A space must appear after "#" or "$".• Fancy and colored prompts must be disabled.• TCL and Expect packages installed on all Fabric Management Nodes.Refer to the Intel® Omni-Path Fabric Software Release Notes for supported operating systems.

5. Ensure the capability of remote login as root enabled.• SSH server enabled• All servers configured with the same root password

6. Ensure that there is a TCP/IP Host Name Resolution.• If using /etc/hosts, update the /etc/hosts file on the Fabric Management Node.• If using DNS, all Management Network and IPoIB host names are added to DNS.• The /etc/resolv.conf file is configured on Fabric Management Node.

7. Ensure that an NTP server is set up.

8. Ensure HFI Node Description Names are assigned.

Install the OPA-Basic Software Checklist


1. Complete the steps in the OPA-Basic Pre-Installation Checklist.

2. Download and Extract the OPA-Basic Software Package per Installation Getting Started.

3. Install the Intel® Omni-Path Software.

3.1

3.1.1

3.1.2

Intel® Omni-Path Fabric—Checklists


Upgrade the OPA-Basic Software Checklist


1. Complete the steps in the Upgrade Prerequisites on page 121.

2. Download and Unpack the new OPA-Basic Software Package per Download the Intel® Omni-PathSoftware on page 39 and Unpack the Tar File on page 39 respectfully.

3. Upgrade the OPA-Basic Software on each compute node per Upgrade the OPA-Basic Software.

OPA-IFS Checklists

OPA-IFS Pre-Installation Checklist


1. Ensure that hardware is installed, cabled, and powered.Refer to the Intel® Omni-Path Fabric Switches Hardware Installation Guide.

2. Ensure that an HFI is installed in each server.Refer to the Intel® Omni-Path Host Fabric Interface Installation Guide.

3. Ensure that all switch ports with management allowed are connected to the Management node.

4. The hardware configuration should be reviewed to ensure everything was installed and installedproperly, according to the plan. Refer to the local hardware configuration plan.

5. Ensure that the required operating system is installed on each server with the following options:• Root user command prompt ends in "#" or "$".

Note: A space must appear after "#" or "$".• Fancy and colored prompts must be disabled.• TCL and Expect packages installed on all Fabric Management Nodes.Refer to the Intel® Omni-Path Fabric Software Release Notes for supported operating systems.

6. Ensure the capability of remote login as root enabled.• SSH server enabled• All servers configured with the same root password

7. Ensure that there is a TCP/IP Host Name Resolution.• If using /etc/hosts, update the /etc/hosts file on the Fabric Management Node.• If using DNS, all Management Network and IPoIB host names are added to DNS.• The /etc/resolv.conf file is configured on Fabric Management Node.

8. Ensure that an NTP server is set up.

9. Ensure HFI Node Description Names are assigned.

Install and Configure the OPA-IFS Software Checklist


1. Complete the steps in the OPA-IFS Pre-Installation Checklist.

2. Download and extract the OPA-IFS Software Package per Installation Getting Started.

3. Install the Intel® Omni-Path Software.

4. Configure the Chassis.

5. Configure the Externally-Managed Switches.

continued...

3.1.3

3.2

3.2.1

3.2.2

Checklists—Intel® Omni-Path Fabric



6. Install the Host Software on the Remaining Hosts Using the FastFabric TUI Menu.

7. Set Up Additional Management Nodes (optional).

8. Install Intel® Omni-Path Fabric Suite Fabric Manager GUI (optional).

9. Verify the Host Software on the Remaining Servers Using the FastFabric TUI Menu.

10. Configure and Initialize Health Check Tools Using FastFabric CLI Commands.

11. Run a Sample High Performance Linpack 2 Configuration.

Upgrade the OPA-IFS Software Checklist


1. Complete the steps in the OPA-IFS Pre-Installation Checklist.

2. Download and extract the new OPA-IFS Software Packages per Upgrade Getting Started.

3. Upgrade the Software on the Management Nodes.

4. Upgrade the Software on the Remaining Servers.

5. Verify the Host Software on the Remaining Servers Using the FastFabric TUI Menu.

3.2.3

Intel® Omni-Path Fabric—Checklists


Part 1: Installing the Software

Installing the Software—Intel® Omni-Path Fabric


4.0 Installation Getting Started

This section provides instructions and information for getting started with the Intel®Omni-Path Software installation.

Pre-Installation Requirements

This section provides the information and procedures needed prior to installing thefabric software. Typically, the Site Implementation Engineer performs the setup tasksdescribed in this section to ensure that the fabric is ready for the software installation.

Fabric Design Prerequisites

Ensure that the following requirements are met prior to installing the software.

It is important that the design and installation of the hardware be planned carefullyprior to the installation and setup of the fabric. The design plan must include thefollowing information:

• Identification of servers that will function as the administration or ManagementNodes, where the Intel® Omni-Path Fabric Suite (IFS) will be installed.

— Server memory requirements based on the software being used:

• IFS, including the Intel® Omni-Path Fabric Suite Fabric Manager, requiresat least 2.5 GB of physical memory for each Fabric Manager instance.

When managing a cluster of 10000 nodes or more, 5 GB of memory perFabric Manager instance is required.

For very large fabrics, with more than 16000 nodes, 15 GB of memory perFabric Manager instance is required.

• When running multiple Fabric Manager instances on a single managementnode, the physical memory requirements should be multiplied by thenumber of Fabric Manager instances.

— Swap disk space allowance should follow recommendations for the givenversion of Linux*. Swap space should be twice the size of the physicalmemory on the server running the Fabric Manager. Minimum swap disk spaceshould be at least 5 GB + 34 GB (for PM short-term history)

— All servers with HFIs installed should have Unified Extensible FirmwareInterface (UEFI) firmware installed.

Refer to the Client Node on page 142 for UEFI BIOs versus Boot Modeinformation.

Note: Intel® Omni-Path Architecture Option ROM support requires UEFIfirmware; and, the HFI drivers are based on UEFI instead of Legacy-BIOS.

— Intel recommends, but does not require the following:

• Using Intel® Xeon® Processor dual-socket server.

4.1

4.1.1

Intel® Omni-Path Fabric—Installation Getting Started


• Processor SKUs that favor single threaded performance over maximumcores (i.e. higher base frequency).

• 32GB or more of ECC memory (ideally fully populating DIMM channels onprocessor to insure maximum memory bandwidth)

• 256GB or more of storage (ideally RAIDed for resiliency)

• Plan the cabling of the fabric and create a cable planning spreadsheet using thesample .cvs files installed into /usr/share/opa/samples/ on the managementnode. After a plan is established, opaxlattopology or opaxlttopology_custmay be used to convert the spreadsheet into a topology XML file that can be usedby FastFabric during fabric verification, as well as being potentially used by theFabric Manager for topology verification at runtime.

Note: The use of cable planning spreadsheets and the resulting topology XML filesis highly recommended by Intel as an effective and efficient way to ensurethe cluster is assembled and installed as intended.

For more information, refer to the opaxlattopology oropaxlattopology_cust sections of the Intel® Omni-Path Fabric SuiteFastFabric User Guide.

• Plan the naming conventions for hosts and switches in the fabric. Intelrecommends all switches and hosts be given unique names. Having unique namessimplifies operations that are performed using host and switch names.

Refer to Appendix C "Node Naming Recommendations" of the Intel® Omni-PathFabric Suite Fabric Manager User Guide for more information on naming theswitches and hosts.

Fabric Setup Prerequisites

Ensure that the following requirements are met prior to installing and setting up thefabric.

Note: For information about the configuration files used by FastFabric, refer to Intel® Omni-Path Fabric Suite FastFabric User Guide.

1. Ensure all hardware is installed:

• Servers

• Core and edge switches

• Fabric cables

Note: When installing externally-managed switches such as the Intel® Omni-PathEdge Switch 100 Series without a management module, the Node GUIDcould be required. The Node GUID is found on a label on the case of theswitch and may be needed to configure and manage the switches with theIFS. You can also use the opagenswitches command to request the nodeGUIDS if there is a topology input file supplied. Intel recommends that youcreate a topology input file using opaxlattopology,opaxlattopology_cust (deprecated), opagentopology, or other tools.In this case, the switch names and locations are figured out from the fabricand the GUIDs for each switch name are then determined automatically bytools such as opagenswitches.

2. Ensure an Intel® Omni-Path Host Fabric Interface (HFI) is installed in each server.

4.1.2

Installation Getting Started—Intel® Omni-Path Fabric


Refer to the Intel® Omni-Path Host Fabric Interface Installation Guide forinstructions.

3. The hardware configuration should be reviewed to ensure everything has beeninstalled according to the plan.

Following the software installation, Intel® Omni-Path Fabric Suite FastFabric toolsmay be used to help verify the installation.

4. (Linux) Ensure the required Operating System (OS) version (with the samekernel version) is installed on all hosts with the following options:

• Root user command prompt ends in "#" or "$" with a space after it.

• Fancy and colored prompts are disabled.

• TCL and Expect packages are installed on all Fabric Management Nodes.

The Management Node(s) that run FastFabric should have a full installation andmust include the TCL and Expect OS RPMs.

For MPI clusters, install the C and Fortran compilers, along with their associatedtools, on each Management Node.

Note: Refer to the Intel® Omni-Path Fabric Software Release Notes for a list ofsupported OS versions.

5. (Linux) Enable remote login as root on each host.

In order for FastFabric to manage the hosts, the Management Node must be ableto securely log in as root to each host. This can be accomplished using SSH.

Note: FastFabric includes the opasetupssh tool, which can help perform the keyexchange to establish secure password-less login from the FastFabric nodeto all other nodes in the fabric. To simplify the use of this tool, Intelrecommends initially configuring the same root password on all hosts. Afterroot access through SSH has been set up using FastFabric, the rootpasswords can be changed.

6. Resolve the TCP/IP Host Names.

FastFabric and TCP/IP must resolve host names to the management network andIPoIB IP addresses. If the management network is not IPoIB, each host must haveboth a management network name and an IPoIB network name. To do this, usethe actual host name as the management network name and HOSTNAME-opa asthe IPoIB network name, where HOSTNAME is the management network name ofthe given host.

Name resolution is accomplished by configuring a DNS server on the managementnetwork, with both management network and IPoIB addresses for each host andeach Intel internally-managed chassis.

Alternatively, an /etc/hosts file needs to be created on the Management Node;FastFabric can then propagate this /etc/hosts file to all the other hosts.

If you are using the /etc/hosts file approach and not using Domain NameSystem (DNS):

• On the master node, add all the Ethernet* and IPoIB addresses intothe /etc/hosts file.

• For the IPoIB convention, use HOSTNAME-opa.

• The localhost line should not be edited.

• The /etc/hosts file should not have any node-specific data.



• Copy the file to every node, as follows:

opascpall -p -f hostfile /etc/hosts /etc/hosts

If you are using Domain Name System (DNS):

• Refer to the documentation for the domain name system (DNS) server beingused. Make sure to edit the /etc/resolv.conf configuration file on theManagement Node to use the proper DNS server.

• Refer to the Linux* OS documentation for more information about configuringthe /etc/resolv.conf file. This file is typically configured during OSinstallation.

• If /etc/resolv.conf must be manually configured for each host, FastFabriccan aid in copying the file to all the hosts.

• The /etc/resolv.conf file created on the Management Node must not haveany node-specific data and must be appropriate for use on all hosts.

• Copying the /etc/resolv.conf file to all the nodes is accomplished duringthe OS installation.

• If the /etc/resolv.conf file was not set up on all the hosts during the OSinstallation, the FastFabric Copy a file to all hosts operation can be usedduring the Install the Host Software on the Remaining Hosts Using theFastFabric TUI Menu procedures to copy the /etc/resolv.conf file from theManagement Node to all the other nodes.

7. Set up a Network Time Protocol (NTP) server.

Configure an NTP server for the cluster, and set all Linux* hosts and internallymanaged chassis to sync to the NTP server.

The setup of the internally-managed chassis is described in Configure the Chassison page 56.

8. Assign HFI Node Description Names.

Node Description names can be configured in many ways. For Intel® Omni-Path,Intel recommends the use of the rdma-ndd daemon (part of the infinband-diagspackage) to keep the Node Description up to date with the hostname of the node.Once set up to assign node descriptions, rdma-ndd automatically assigns nodedescriptions to RDMA devices whenever a node is restarted, a RDMA device comesonline, or the hostname changes.

For details on rdma-ndd, see the man page.

Some distributions use rdma-ndd by default. For Intel® Omni-Path Fabric Suite-supported distributions that lack the rdma-ndd daemon, the latest version ofinfiniband-diags is supplied with the OPA-IFS package.

OS RPMs Installation Prerequisites


In addition to normal OS installation options, there are OS RPMs that must be installedbefore installing the Intel® Omni-Path software. Verify that all required OS RPMs areinstalled using one of the following tables:

• Red Hat* Enterprise Linux* OS Distribution RPMs Matrix

4.1.3



• SUSE* Linux* Enterprise Server OS Distribution RPMs Matrix

Note: Use the version distributed with your OS distribution.

Table 1. Red Hat* Enterprise Linux* OS Distribution RPMs Matrix

OS Distribution RPMs RHEL* 6.7 RHEL* 7.3 RHEL* 7.4

IB/OPA Centric

libibmad x x

libibumad x x

libibumad-devel x

libibverbs x x x

librdmacm x x x

libibcm x x x

ibacm x x

qperf x x x

perftest x x x

rdma x x

infinipath-psm x x x

opensm-devel x

opensm-libs x

rdma-core x

rdma-core-devel x

OPA Centric

libhfi1 x

System Centric

expat x x x

elfutils-libelf-devel x x x

libstdc++-devel x x x

gcc-gfortran x x x

atlas x x x

tcl x x x

expect x x x

tcsh x x x

sysfsutils x x x

pciutils x x x

bc (command line calculator for floating point math) x x x

rpm-build x x x

redhat-rpm-config x x x

continued...




kernel-devel x x x

c-ares x

json-c x x x

json-c-devel x

coreutils x x

elfutils-libelf x x

glibc x x

libgfortran x x

libgomp x x

libquadmath x x

libstdc++ x x

ncurses-libs x x

numactl-libs x x

openssl-libs x x

zlib x x

papi x x

Other/Generally Installed

chkconfig

bash x x

systemd x x

perl x x

perl-Getopt-Long x x

perl-Socket x x

irqbalance x x

kernel x x

kmod x x

libgcc x x

pkgconfig x x

python x x

Build Requirements

libibverbs-devel x x

libibmad-devel x x

librdmacm-devel x x

ibacm-devel x x

openssl-devel (1.0.1 or higher) x x

libuuid-devel x x

continued...




expat-devel x x

infinipath-psm-devel x x

valgrind-devel x x

json-c-devel x x x

libpfm x x

numactl-devel x

Table 2. SUSE* Linux* Enterprise Server OS Distribution RPMs Matrix

OS Distribution RPMs SLES* 12 SP2 SLES* 12 SP3

IB/OPA Centric

libibmad5 x x

libibumad-devel x

libibverbs1 x x

librdmacm1 x x

libibcm1 x x

ibacm x x

qperf x x

perftest x x

rdma x

opensm-devel x x

opensm-libs3 x x

libpsm_infinipath1 x x

libibumad3 x x

rdma-core x

rdma-core-devel x

OPA Centric

libhfi1verbs-rdmav2 x

System Centric

bc (command line calculator for floating pointmath)

x x

rpm-build x x

kernel-devel x x

libjson-c x x

libexpat1 x x

libelf-devel x x

gcc-fortran x x

libudev-devel x x

continued...




coreutils x

libelf1 x x

libgfortran3 x x

libgomp1 x x

libncurses5 x x

libnuma1 x x

libquadmath0 x x

libz1 x x

libopenssl1_0_0 x x

tcl x x

expect x x

tcsh x x

HFI ASIC-lite support

infiniband-diags x

libipathverbs-rdmav2 x

Other/Generally Installed

bash x x

glibc x x

grep x x

irqbalance x x

kmod x x

libedit0 x x

perl x x

perl-base x x

pkg-config x x

python-base x x

systemd x x

libgcc_s1 x x

perl-base x x

libstdc++6 x x

udev x x

Build Requirements

libibverbs-devel x

librdmacm-devel x

ibacm-devel x

libopenssl-devel (1.0.1 or higher) x x

continued...




libuuid-devel x x

libexpat-devel x x

libibumad-devel x

infinipath-psm-devel x

valgrind-devel x x

libjson-c-devel x x

libnuma-devel x

Note: Depending on the packages you choose, there may be additional prerequisites. Foradditional information, refer to the Release Notes for your specific release andinstallation type.

The following is only an example and is showing an install of a group of OS RPMs on aRHEL OS server. Not all required OS RPMs are included in this example and some OSRPMs in this example might not be needed. Intel recommends building your own list ofOS RPMs for installation.

# yum install libibmad libibverbs librdmacm libibcm qperf perftestrdma infinipath-psm expat elfutils-libelf-devel libstdc++-develgcc-gfortran atlas tcl expect tcsh sysfsutils pciutils bc libibumadlibibumad-devel libibumad libibumad-devel libibverbs-devel libibmaddevellibrdmacm-devel ibacm-devel openssl-devel libuuid-devel expatdevelinfinipath-psm-devel valgrind-devel libgnome libibverbs*opensm-libs libhfi1 papi ncurses-devel hwloc hwloc-gui

MPI Compiler Prerequisites


Two of the MPIs that are downloaded with the OPA-Basic Software package are listedin the Intel OPA Install Menu, MVAPICH2 and OpenMPI. These MPIs were built withIntel compilers.

• To use the MPIs for compiling and running your applications and benchmarks, youneed to install a version of the Intel® Parallel Studio XE 2016 Composer Editionthat includes the C++, and/or Fortran compilers according to your needs.

• Intel recommends, at a minimum, to have the Intel Composer Runtime librariesinstalled to be able to run MPI applications. These runtime libraries may beobtained from: https://software.intel.com/en-us/articles/redistributables-for-intel-parallel-studio-xe-2016-composer-edition-for-linux.

Performance Tuning Prerequisites

Intel recommends that you pre-configure servers and settings to tune fabricperformance to meet the needs of your system. These tasks can be performed beforeor after the installation. Refer to the Intel® Omni-Path Fabric Performance Tuning UserGuide which describes BIOS settings and parameters that have been shown toimprove performance, or make performance more consistent, on Intel® Omni-PathArchitecture. If you are interested in benchmarking the performance of your system,these tips may help you obtain better performance.

4.1.4

4.1.5



https://software.intel.com/en-us/articles/redistributables-for-intel-parallel-studio-xe-2016-composer-edition-for-linux

https://software.intel.com/en-us/articles/redistributables-for-intel-parallel-studio-xe-2016-composer-edition-for-linux

Download the Intel® Omni-Path Software

If the OS you installed did not include the OPA RPMs, download the software packagefrom an Intel web page or other Intel secured location using the following procedures.

1. Using a web browser, type downloadcenter.intel.com in the address fieldand press Enter.

2. In the "Search downloads" field, type Omni-Path and press Enter.

3. In the Description list select the "Intel® Omni-Path Fabric Software (IncludingIntel® Omni-Path Host Fabric Interface Driver)" for the version you are wanting toinstall .

Note: The latest version of each type of each download type is showing in the list.To show previous versions select "Show more" at the bottom of the list.

4. In the "Available Downloads" list, select the file(s) you need for the OS you haveinstalled on your fabric.

• For OPA-Basic software, select:

IntelOPA-Basic.DISTRO.VERSION.tgz

• For OPA-IFS software, select:

IntelOPA-IFS.DISTRO.VERSION.tgz

Note: DISTRO.VERSION refers to the distribution and CPU.

5. Review the Intel Software License Agreement.

6. Click "I accept the terms in the license agreement."

7. Save the download to your hard drive.

Unpack the Tar File

You unpack the tar file using the following procedure.

1. Open an SSH client session and log into the host where the package is beinginstalled.

2. Copy the tar file to the /root directory.

3. Change directory to /root.

cd /root

4. Unpack the tar file.

• For OPA-Basic, use:

tar xvfz IntelOPA-Basic.DISTRO.VERSION.tgz

• For OPA-IFS, use:

tar xvfz IntelOPA-IFS.DISTRO.VERSION.tgz

4.2

4.3



Download and Install NVIDIA* Software (Optional)

If you are installing the Intel® Omni-Path Software onto a machine that has a non-standard kernel, the software may need to recompile the Intel® Omni-Path HostFabric Interface (HFI) driver. If this is required while installing the GPUDirect*components, you will need the NVIDIA* driver source code and symbols available inorder to compile the HFI GPUDirect* enabled driver.

Use the following instructions to download and build the NVIDIA* driver:

1. Download the NVIDIA* driver source from:

http://www.nvidia.com/Download/index.aspx

It is also available in the CUDA* Toolkit package that can be downloaded from:

https://developer.nvidia.com/cuda-downloads

Note: Users of CUDA* applications will need the CUDA* runtime as well, which isalso part of the CUDA* Toolkit package. Intel recommends the full CUDA*Toolkit, but acknowledges that some administrators may only want thedriver itself.

2. Extract the driver source and build the driver module as per the instructions at:

http://docs.nvidia.com/cuda/gpudirect-rdma/index.html#linking-kernel-module-against-nvidia-ko

3. Install the NVIDIA* driver module if it's not already installed.

4. Run the command export NVIDIA_GPU_DIRECT=<path to NVIDIA driversrc build directory> so the Intel® Omni-Path Software installer knowswhere to find the NVIDIA* driver source when re-building the HFI driver.

Note: The <path to NVIDIA driver src build directory> is thedirectory where the Module.symvers file is present after the NVIDIA*driver is built.

Additionally for any runtime applications that need CUDA* runtime support you willneed to have CUDA* runtime installed for these applications. Intel recommendsdownloading the entire CUDA* Toolkit and installing it.

Download and Installation instructions from NVIDIA* can be found here:

http://docs.nvidia.com/cuda/cuda-installation-guide-linux/

4.4



HTTP://WWW.NVIDIA.COM/DOWNLOAD/INDEX.ASPX

https://developer.nvidia.com/cuda-downloads

HTTP://DOCS.NVIDIA.COM/CUDA/GPUDIRECT-RDMA/INDEX.HTML#LINKING-KERNEL-MODULE-AGAINST-NVIDIA-KO

HTTP://DOCS.NVIDIA.COM/CUDA/GPUDIRECT-RDMA/INDEX.HTML#LINKING-KERNEL-MODULE-AGAINST-NVIDIA-KO

HTTP://DOCS.NVIDIA.COM/CUDA/CUDA-INSTALLATION-GUIDE-LINUX/

5.0 Install the Intel® Omni-Path Software

This section provides information and procedures to install the Intel® Omni-PathSoftware on the Management Node or on a host node in the fabric.

You install the software using one of the following methods:

• Install using TUI menus (recommended).

• Install using CLI commands.

• Install Using the Linux* Distribution Software on page 47

Intel recommends that you install the OPA-IFS software on the Management Nodeusing the Install TUI, and then use FastFabric to configure the Management Node.

Once the Management Node has been configured, the Basic software can be installedon all the remaining hosts using either the FastFabric TUI or a provisioning or disklessboot mechanism.

Note: If you are using a provisioning system, consult the documentation that comes with theprovisioning system.

Use the Checklists on page 26 to track your installation of the software.

Before You Begin

Before starting the installation, perform the following:

• Refer to the Release Notes for the list of compatible operating systems.

• Be sure you have completed all Pre-Installation Requirements on page 30.

• You have downloaded and extracted the software package per Installation GettingStarted on page 30.

• If you are installing via Linux* Distribution (Distro), you have completed the OSRPMs Installation Prerequisites on page 33.

• If you are using a customized installation via the Install CLI command, prepareyour command line options. Refer to Using the ./INSTALL Command Line Options.

• For IPoIB IPV4, gather your list of the IP addresses and netmasks for eachinterface you are going to set up.

Using the ./INSTALL Command Line Options

The ./INSTALL command for the Basic and IFS installations are issued from thefollowing directories:

• Intel Basic directory: IntelOPA-Basic.DISTRO.VERSION• Intel IFS directory: IntelOPA-IFS.DISTRO.VERSION

5.1

5.1.1

Install the Intel® Omni-Path Software—Intel® Omni-Path Fabric


Syntax

./INSTALL [-r root] [-v|-vv] -R osver -B osver [-a|-n|-U|-u|-s|-O|-N|-i comp|-e comp] [-E comp] [-D comp] [--user-space] [--user_configure_options 'options'] [--kernel_configure_options 'options'] [--prefix dir] [--without-depcheck] [--rebuild] [--force] [--answer keyword=value]

Options

No option selected Displays the Intel OPA Software TUI.

-r root Specifies an alternate root directory. The defaultis /.

Note: This option permits boot images to beconstructed that include Intel® Omni-PathSoftware so that the boot images can later beused for network boot of Intel® Omni-PathFabric enabled nodes.

Note: Intel® Omni-Path Fabric Suite FastFabric useis not permitted in this environment.

-v Provides verbose logging. Logs to the /var/log/opa.log file.

-vv Provides very verbose debug logging. Logs tothe /var/log/opa.log file.

-R osver Force install for specific OS kernel version, ratherthan running kernel.

-B osver Run build for all components targetting specific OSkernel version.

-a Installs all Upper Layer Protocols (ULP) and driverswith the default options.

Note: This rebuilds the RPMs except for CUDA*which is not rebuilt unless -G is specified.

-n Installs all ULPs and drivers with the defaultoptions, but does not change the autostart options.

-U Upgrades/reinstalls all presently installed ULPs anddrivers with the default options, and does notchange the autostart options.

-u Uninstalls all ULPs and drivers with the defaultoptions.

Intel® Omni-Path Fabric—Install the Intel® Omni-Path Software


-s Enables autostart for all installed software.

-O Keeps the current modified rpm configuration file.

-N Uses a new default rpm configuration file.

-i comp Installs the given component with the defaultoptions. This option can appear multiple times on acommand line.

-e comp Uninstalls the given component with the defaultoptions. This option can appear multiple times on acommand line.

-E comp Enables autostart of given component. This optioncan appear with -D or multiple times on a commandline.

Note: To control which installed software isconfigured for autostart, combine this optionwith -a, -n, -i, -e, and -U options.

-D comp Disables autostart of the given component. Thisoption can appear with -E or multiple times on acommand line.

Note: To control which installed software isconfigured for autostart, combine this optionwith -a, -n, -i, -e, and -U options.

--user-space Skips kernel space components during installation.

--user_configure_optionsoptions

Specifies additional OFED build options for userspace srpms. This forces a rebuild of all user srpms.

--kernel_configure_options options

Specifies additional OFED build options for driversrpms. This forces a rebuild of all driver srpms.

--prefix dir Specifies an alternate directory prefix for theOFED_Delta installation. Default is /usr. Thiscauses a rebuild of needed srpms.

--without-depcheck Disables the check of OS dependencies.

--rebuild Forces a rebuild of OFED_DELTA srpms.

--force Forces the installation, even if the distributions donot match. Use of this option can result in undefinedbehaviors.



--answer keyword=value Provides an answer to a question which might occurduring the operation. Answers to questions that arenot asked are ignored. Invalid answers result inprompting for interactive installations or use of thedefault for non-interactive installations.

Possible Questions:

UserQueries Allows non-root users to access theUMAD interface.

Note: Allowing access to umadXdevice files may present asecurity risk. However, thisallows tools such asopasaquery andopaportinfo to be used bynon-root users.

Default options retain existingconfiguration files.

IrqBalance Set IrqBalance to Exact?

-C Shows the list of supported component names.

-V Outputs the version number of the software.

-G Install GPUDirect* components.

Note: Only RHEL* 7.3, RHEL* 7.4, and SLES* 12SP2 support GPUDirect*.

--user_queries Permits non-root users to query the fabric. This isthe default.

--no_user_queries Specifies non-root users cannot query the fabric.

Examples

./INSTALL

./INSTALL -n

./INSTALL -C

./INSTALL -V

./INSTALL -a -G



Other Information

Supported Component (comp) Names:

opa_stack, ibacm, mpi_selector, intel_hfi, oftools, opa_stack_dev, fastfabric, delta_ipoib, opafm, mvapich2_gcc_hfi, mvapich2_intel_hfi, openmpi_gcc_hfi, openmpi_intel_hfi, openmpi_gcc_cuda_hfi, gasnet, openshmem, sandiashmem, mvapich2, openmpi, delta_mpisrc, hfi1_uefi, delta_debug

Supported Component (comp) Name Aliases:

opa, ipoib, mpi, psm_mpi, verbs_mpi, pgas, mpisrc, opadev

Install Using the TUI Menus

You can install both the Intel® Omni-Path Software Basic and IFS software packagesusing the Intel OPA Software menu.

Assumptions

• You are logged in.

• You have your IPV4 list of IP addresses and netmasks for each interface you aregoing to set up.

Procedures

Perform the following steps to install the Intel® OP Software:

Step Task/Prompt Action

1. At the command prompt, change directory to the locationof the installation software package:

• For Basic, type the following and press Enter:cd IntelOPA-Basic.DISTRO.VERSION

• For IFS, type the following and press Enter:cd IntelOPA-IFS.DISTRO.VERSION

where DISTRO.VERSION is the distribution and CPU.

2. At the command prompt, start the install script. Type ./INSTALL and press Enter.

3. Select 1) Install/Uninstall Software. Type 1.

4. Review the items to be installed. Accept the defaults.Type N to go to the next page.NOTE: If you need to change any item, enter thealphanumeric character associated with the item to togglebetween Install or Don't Install.

5. Start the installation. Type P to perform the actions.

6. Preparing OFA VERSION release for Install... Rebuild OFASRPMs (a=all, p=prompt per SRPM, n=only as needed?)[n]:

Press Enter to accept the default.NOTE: The system will display prompts that require yourresponse throughout the installation.

7. For each system prompt... Accept the defaults by pressing Enter to continue.

8. When the Intel OPA Autostart Menu displays, review theitems.

Intel recommends leaving all of the Autostart selectionsset to the default values. However, on the FabricManagement Nodes where you plan to run the Intel®Omni-Path Fabric Suite Fabric Manager, make sure to setthe OPA FM to enable autostart.

continued...

5.2




NOTE: If you need to change any item, enter thealphanumeric character associated with the item to togglebetween Enable or Disable.

9. Run the OPA Autostart operations. Type P.

10. For each system prompt, "Hit any key to continue..." Press any key.NOTE: When the installation completes, you are returnedto the main menu.

Set up IPoIB IPV4 Configuration

11. Select 2) Reconfigure OFA IP over IB. Type 2.

12. Configure OFA IP over IB IPV4 addresses now?[n]:

Type y.

13. How many OFA IP over IB interfaces would youlike to configure? [1]:

Type the number of interfaces and press Enter.

14. For each interface, provide the following information:• OFA IP over IB interface name• IPV4 address in dot notation (or dhcp)• IPV4 netmask in dot notation

Provide the required information and press Enter.NOTE: After setup is complete, you are returned to themain menu.

15. Reboot the server. Type reboot and press Enter.

End Task

Next Steps

• If you need to set up IPoIB IPV6, go to Configure IPoIB IPV6.

• If you are ready to configure your IFS software, go to Configure the Chassis.

Install Using CLI Commands

You can install both the Intel® Omni-Path Software Basic and IFS software packagesusing the ./INSTALL.

The ./INSTALL command has many options including installing single components,and enabling and disabling autostart of components. This section provides instructionsfor the default installation, but you can append the install command with specificoptions for a more customized installation. For more information on the ./INSTALLsyntax and options, refer to Using the ./INSTALL Command Line Options.

Assumptions



Procedures

Perform the following steps to install the default Intel® OP Software configuration:


1. At the command prompt, change directory to the locationof the installation software package:

• For Basic, type the following and press Enter:cd IntelOPA-Basic.DISTRO.VERSION

continued...

5.3




• For IFS, type the following and press Enter:cd IntelOPA-IFS.DISTRO.VERSION

where DISTRO.VERSION is the distribution and CPU.

2. At the command prompt, start the install script. Type ./INSTALL -n and press Enter.

3. At the command prompt, reboot the server. Type reboot and press Enter.


4. Manually edit or create the /etc/sysconfig/network-scripts/ifcfg-ibX file.

Note: Use the OS distribution-supplied instructions forsetting up network interfaces.

For example:

DEVICE=ib0BOOTPROTO=staticIPADDR=1.1.1.1BROADCAST=1.255.255.255NETWORK=1.0.0.1NETMASK=255.0.0.1ONBOOT=yesCONNECTED_MODE=yesMTU=65520

5. Create the ifcfg-ibx configuration file for each hostusing the host IP addresses from the /etc/hosts file.

Type opahostadmin -f /etc/opa/hostsconfigipoib and press Enter.

End Task

Next Steps



Install Using the Linux* Distribution Software

You can install the Intel® Omni-Path Software package that is included with the OSdistribution.

Assumptions



Procedures

Perform the following steps to install the default Intel® OP Software configuration:


Install OPA-Basic Software

1. At the command prompt, enter the installation commandfor opa-basic-tools.

• For RHEL* 7.3, ... Type yum install -y opa-basic-tools and pressEnter.

• For SLES* 12sp2, ... Type zypper install -y opa-basic-tools andpress Enter.

continued...

5.4




2. At the command prompt, reboot the server. Type reboot and press Enter.

3. Check your link using opainfo Type opainfo and press Enter.Example output:

hfi1_0:1 PortGID:0xfe80000000000000:001175010163f931 PortState: Active LinkSpeed Act: 25Gb En: 25Gb LinkWidth Act: 4 En: 4 LinkWidthDnGrd ActTx: 4 Rx: 4 En: 3,4 LCRC Act: 14-bit En: 14-bit,16-bit,48-bit Mgmt: True LID: 0x00000010-0x00000010 SM LID: 0x0000000c SL: 0 QSFP: AOC , 5m FINISAR CORP P/N FCBN425QB1C05 Rev A Xmit Data: 0 MB Pkts: 251 Recv Data: 0 MB Pkts: 251 Link Quality: 5 (Excellent)

4. Install the rdma-core rpm.

• For RHEL* 7.3, ... Type yum install -y rdma-core and press Enter.

• For SLES* 12sp2, ... Type zypper install -y rdma-core and pressEnter.

5. On all compute nodes: install the PSM2 library.

• For RHEL* 7.3, ... Type yum install -y libpsm2 and press Enter.

• For SLES* 12sp2, ... Type zypper install -y libpsm2-2 and pressEnter.

Install Intel® Omni-Path Fabric Suite Components on the Management Node

6. Install FastFabric.

• For RHEL* 7.3, ... Type yum install -y opa-fastfabric and pressEnter.

• For SLES* 12sp2, ... Type zypper install -y opa-fastfabric and pressEnter.

7. For RHEL* 7.3 only:Install the opa-address-resolution rpm.

Type yum install -y opa-address-resolutionand press Enter.

8. Install Fabric Manager.

• For RHEL* 7.3, ... Type yum install -y opa-fm and press Enter.

• For SLES* 12sp2, ... Type zypper install -y opa-fm and press Enter.

9. Start the Fabric Manager. Type systemctl start opafm and press Enter.


10. Manually edit or create the ifcfg-ibX file. Note: Use the OS distribution-supplied instructions forsetting up network interfaces.

• For RHEL* 7.3, ... Type cat /etc/network-scripts/ifcfg-ib0 andpress Enter.Example output:

DEVICE=ib0BOOTPROTO=staticIPADDR=10.228.200.173

continued...




BROADCAST=10.228.203.255NETWORK=10.228.200.0NETMASK=255.255.252.0ONBOOT=yesCONNECTED_MODE=yesMTU=65520

• For SLES* 12sp2, ... Type cat /etc/network/ifcfg-ib0 and press Enter.Example output:

DEVICE=ib0BOOTPROTO=staticIPADDR=10.228.200.173BROADCAST=10.228.203.255NETWORK=10.228.200.0NETMASK=255.255.252.0ONBOOT=yesCONNECTED_MODE=yesMTU=65520

11. Bring up the ib0 interface. Type ifup ib0 and press Enter.

12. Perform a test ping. Type ping <remote IPoIB address> and pressEnter.For example:

ping 10.228.200.161PING 10.228.200.161 (10.228.200.161) 56(84) bytes of data.64 bytes from 10.228.200.161: icmp_seq=1 ttl=64 time=0.863 ms

(Optional) Install the Fabric Manager GUI

13. On one node in the fabric: install the Fabric Manager GUI. Note: Intel recommends not to install the FabricManager GUI on the Management Node where theFabric Manage is being used.

• For RHEL* 7.3, ... Type yum install -y opa-fmgui and press Enter.

• For SLES* 12sp2, ... Type zypper install -y opa-fmgui and pressEnter.

End Task

Next Steps


• If you are ready to configure your IFS software, go to Configuring the Software.

Configure IPoIB IPV6

You configure IPoIB for IPV6 on the management node. Instructions are dependent onthe particular OS installed on the node.

Using Red Hat* Enterprise Linux*

To configure IPoIB for IPV6, perform the following:

1. Edit file /etc/sysconfig/network to add the following line:

NETWORKING_IPV6=yes

5.5



2. Edit file ifcfg-ifname to add the following lines:

IPV6INIT=yesIPV6ADDR=”ipv6addr/prefix-length”

IPV6 address should look like the following example:

3ffe::6/64

3. Restart the network.

Using SUSE* Linux* Enterprise Server

To configure IPoIB for IPV6, perform the following:

1. Edit ifcfg-ifname to add the following line:

IPADDR=”ipv6addr/prefix-length”

IPV6 address should look like the following example:

3ffe::6/64

2. Restart the IPoIB network interfaces.

Next Steps

• At this time, you should tune Verb and IPoFabric performance. Refer to the Intel®Omni-Path Fabric Performance Tuning User Guide for instructions.




Part 2: Configuring the Software

Configuring the Software—Intel® Omni-Path Fabric


6.0 Configuration Getting Started

This section provides instructions and information for getting started with the Intel®Omni-Path Fabric Suite configuration.

You configure the fabric components using one of the following methods:

• Using the FastFabric TUI menus (recommended).

• Using the CLI commands.

Use the Checklists on page 26 to track your configuration and verification of thesoftware.

Pre-Configuration Requirements

This section provides the information and procedures needed prior to configuring andverifying the fabric software.

Chassis Configuration Prerequisites

Ensure that the following requirements are met prior to configuring the chassis.

Note: The Intel® Omni-Path Fabric Switches Hardware Installation Guide contains additionaldetails about steps that must be performed before completing this procedure.

1. (Switch) Connect each chassis to the management network through its Ethernetmanagement port. For chassis with redundant management, connect bothEthernet management ports.

2. (Switch) Set up the netmask and gateway addresses on each Intel chassis,following the procedures in the Intel® Omni-Path Fabric Switches GUI User Guide.

3. (Switch) Assign each Intel chassis a unique IP address, and appropriatelyconfigure the Ethernet management port network settings.

4. (Switch) For a chassis with redundant management, assign a unique IP addressfor each Intel Management Module or Intel Management Spine, and configure theirEthernet management port network settings.

5. (Switch) Select a unique name for each Intel chassis, Management Module, andSpine. This name should be configured in DNS or /etc/hosts as the TCP/IPname for the Ethernet management port.

Note: The chassis node description is set later in this process.

6. (Switch) Configure the administrator password on each Intel chassis.

6.1

6.1.1

Intel® Omni-Path Fabric—Configuration Getting Started


Note: All versions of Intel® Omni-Path Chassis 100 Series firmware permit SSHkeys to be configured within the chassis for secure password-less login.Intel® Omni-Path Fabric Suite FastFabric includes a tool (opasetupssh)that can help perform the key exchange to establish secure password-lesslogin from the Management Node to all the internally managed switches inthe fabric. For simplicity, Intel recommends you initially configure the sameadministrator password on all switches. (Another method is to leave thedefault “adminpass” until FastFabric has set up SSH keys.) After SSH hasbeen set up using FastFabric, Intel recommends you change the adminpassword.

7. (Switch) Copy the relevant chassis firmware files onto the Management Node.When performing the Chassis Configuration procedures, the *.spkg files are usedto upgrade the firmware on each chassis.

Note: Place all files at a given firmware level into a single directory. Intelrecommends that you include the firmware revision number in the directoryname.

Switch Configuration Prerequisites

Ensure that the following requirements are met prior to configuring the externally-managed switch.

1. (Switch) Select a unique name to be used for each switch. This name isconfigured as the Switch Description for the switch.

Note: Externally-managed switches do not have an Ethernet port and therefore donot have a TCP/IP name.

2. (Switch) Copy the relevant switch firmware files onto the Intel® Omni-PathFabric Suite FastFabric management node. The *.emfw files are used to upgradethe firmware on each switch.

Note: Place all files at a given firmware level into a single directory. Intelrecommends that you include the firmware revision number in the directoryname.

How to Use the FastFabric TUI

The FastFabric TUI menus are set up for ease of use. The submenus are designed topresent operations in the order they would typically be used during an installation.

Note: All FastFabric TUI menu alpha-based options are case-insensitive.

Selecting Menu Items and Performing Operations

1. From the Intel FastFabric OPA Tools main menu, select the target menu item(0-4).

Intel FastFabric OPA ToolsVersion: X.X.X.X.X

1) Chassis Setup/Admin 2) Externally Managed Switch Setup/Admin 3) Host Setup 4) Host Verification/Admin

6.1.2

6.2

Configuration Getting Started—Intel® Omni-Path Fabric


5) Fabric Monitoring

X) Exit

The target menu is displayed as shown in the example below:

FastFabric OPA Chassis Setup/Admin MenuChassis File: /etc/opa/chassisSetup:0) Edit Config and Select/Edit Chassis File [ Skip ]1) Verify Chassis via Ethernet Ping [ Skip ]2) Update Chassis Firmware [ Skip ]3) Set Up Chassis Basic Configuration [ Skip ]4) Set Up Password-Less SSH/SCP [ Skip ]5) Reboot Chassis [ Skip ]6) Get Basic Chassis Configuration [ Skip ]7) Configure Chassis Fabric Manager (FM) [ Skip ]8) Update Chassis FM Security Files [ Skip ]9) Get Chassis FM Security Files [ Skip ]Admin:a) Check OPA Fabric Status [ Skip ]b) Control Chassis Fabric Manager (FM) [ Skip ]c) Generate All Chassis Problem Report Info [ Skip ]d) Run a Command on All Chassis [ Skip ]Review:e) View opachassisadmin Result Files [ Skip ]

P) Perform the Selected Actions N) Select NoneX) Return to Previous Menu (or ESC)

2. Type the key corresponding to the target menu item (0-9, a-d) to toggle theSkip/Perform selection.

More than one item may be selected.

3. Type P to perform the operations that were selected.

Notes: • If more than one menu item is selected, the operations are performedin the order shown in the menu. This is the typical order desired duringfabric setup.

• If you want to perform operations in a different order, you must selectthe first target menu item, type P to perform the operation, thenrepeat this process for the next menu item operation to be performed,and so on.

4. Type N to clear all selected items.

5. Type X or press Esc to exit this menu and return to the Main Menu.

Aborting Operations

While multiple menu items are performing, you have an opportunity to abort individualoperations as they come up. After each operation completes and before the nextoperation begins, you are prompted as shown below:

Hit any key to continue...

• Press Esc to stop the sequence of operations return to the previous menu.

Any unperformed operations are still highlighted in the menu. To complete theselected operations, type P.

• Press any other key to perform the next selected menu item being performed.

Intel® Omni-Path Fabric—Configuration Getting Started


This prompt is also shown after the last selected item completes, providing anopportunity to review the results before the screen is cleared to display the menu.

Submenu Configuration Files

On each FastFabric submenu, item 0 permits a different file to be selected and edited(using the editor selected by the EDITOR environment variable). It also permitsreviewing and editing of the opafastfabric.conf file. The opafastfabric.conffile guides the overall configuration of FastFabric and describes cluster-specificattributes of how FastFabric operates.

At the top of each FastFabric submenu screen beneath the title, the directory andconfiguration file containing the components on which to operate are shown.

In the example below, the configuration file is noted in bold.

FastFabric OPA Host Setup MenuHost File: /etc/opa/hostsSetup:0) Edit Config and Select/Edit Host File [ Skip ]1) Verify Hosts Pingable [ Skip ]2) Set Up Password-Less SSH/SCP [ Skip ]

Note: During the execution of each menu selection, the actual FastFabric command line toolbeing used is shown. This can be used as an educational aid to learn the commandline tools.

The example snippet below shows how the CLI is displayed in the TUI execution.

Performing Chassis Admin: Verify Chassis via Ethernet PingExecuting: /usr/sbin/opapingall -C -p -F /etc/opa/chassis

Configuration Getting Started—Intel® Omni-Path Fabric


7.0 Configure the Chassis

This section provides information and procedures to configure the internally-managedswitches, such as the Intel® Omni-Path Director Class Switch 100 Series, using theIntel® Omni-Path Fabric Suite FastFabric toolset.

Note: For information about configuring switches made by other manufacturers, refer to thevendor's switch documentation.

Before You Begin

Before starting the chassis configuration, perform the following:

• Be sure you have completed all Chassis Configuration Prerequisites on page 52.

• Be sure you have the latest Intel® Omni-Path Fabric Switches Release Notes forreference.

• Gather your information for the configuration files, as needed.

Note that you can edit the files before you configure the chassis. Configurationfiles are located under the /etc/opa directory. Sample files are installedinto /usr/share/opa/samples with the suffix -sample.

The following files are used to configure the chassis:

— opafastfabric.conf: Lists the default settings for most of the FastFabriccommand line options.

— ports: Lists the local HFI ports to use to access the fabrics.

For typical single-subnet clusters, the default of “0:0” may be used. This usesthe first active port on the Management Node to access the fabric.

— chassis: Lists the TCP/IP names of the internally-managed switches in thecluster.

Intel recommends you use chassis names (the TCP/IP Ethernet managementport names assigned). Enter one chassis name or IP address per line. Note:Do not list externally-managed switches in this file.

For example:

Chassis1Chassis2

For more details about configuration files, refer to the Intel® Omni-Path FabricSuite FastFabric User Guide.

For more details about the file format of the configuration files, refer to the Intel®Omni-Path Fabric Suite FastFabric User Guide.

Note: If you are configuring multiple subnets, refer to Multi-Subnet Fabrics onpage 106 for additional instructions.

7.1

Intel® Omni-Path Fabric—Configure the Chassis


Configure the Chassis Using the FastFabric TUI Menu

You can configure the chassis using the FastFabric OPA Chassis Setup/Admin Menu.

Assumptions

• You are logged into the management node.

• You have the latest Intel® Omni-Path Fabric Switches Release Notes for reference.

Pre-Work

For the following menu items, prepare to answer questions or provide input as follows:

• Updating the Chassis FM Security Files

— Choose options:

• push – Ensures given security files are pushed to each chassis.

• restart – After push, restart FM on master, stop on slave.

• restartall – After push, restart FM on all MM.

— Select security files or directory containing pem files

— Choose parallel versus serial update

— Set chassis password (default is to have password in fastfabric.conf or to usepassword-less SSH)

• Checking the OPA Fabric Status

— Would you like to perform fabric error analysis? [y]:Allows you to start the analysis.

— Clear error counters after generating report? [n]:Allows you to clear the error counters after generating the report.

— Would you like to perform fabric link speed error analysis?[y]:Allows you to analyze fabric link speed errors.

— Check for links configured to run slower than supported?[n]:Allows you to check for Links running slower than expected.

— Check for links connected with mismatched speed potential?[n]:Allows you to check for links connected with mismatched speed.

— Enter filename for results [/root/linkanalysis.res]:Allows you to enter a filename for the results or use the default file.

• Control Chassis Fabric Manager

— Would you like to make sure the FM is not running? [n]:Allows you to ensure that the FM is not running.

— Would you like to make sure the FM is running? [n]:Allows you to ensure that the FM is running.

7.2

Configure the Chassis—Intel® Omni-Path Fabric


— Would you like to run FM on slave MMs? [n]:Allows you to run FM on slave management modules.

— Would you like to do the operation in parallel? [y]:Allows you to perform operations in parallel (on multiple chassis). Doing theoperation in parallel will finish the fastest.

— Would you like to change FM boot state to enable FM startat boot? [n]:Allows you to enable FM start on slave management modules at boot.

— Would you like to change FM boot state to disable FM startat boot? [n]:Allows you to disable FM start on slave management modules at boot.

— Would you like to be prompted for chassis' password? [n]:Allows you to be prompted for the chassis password.

Procedures

The following steps provide simplified instructions for first-time configuration of thechassis. For additional details, refer to the Intel® Omni-Path Fabric Suite FastFabricUser Guide.


Configuring the Chassis

1. Access the FastFabric OPA Chassis Setup/Adminmenu.

a) If you are not already logged into Intel FastFabricOPA Tools, at the command prompt...

Type opafastfabric and press Enter.

b) Access the FastFabric OPA Chassis Setup/Adminmenu.

Press 1.

c) Select menu items. Select all operations by entering each menu item 0 – e.NOTE: You can skip any items that are not needed foryour fabric. However, this procedure assumes that allitems have been selected.

d) Start the operations. Press P.NOTE: Each selected item is preformed in the order ofthe menu list.

2. Edit the Configuration and Select/Edit Chassis File(menu item 0)

a) Edit the opafastfabric.conf file. Review the file with a focus on the following:• FF_CHASSIS_LOGIN_METHOD• FF_CHASSIS_ADMIN_PASSWORD.• Select the location for the result files from FastFabric

with the FF_RESULT_DIR parameter.If you made any changes, save and close the file.Press any key to continue.

b) Edit the ports configuration file. Review the file.If you made any changes, save and close the file.Press any key to continue.

continued...




c) Edit the chassis configuration file. Create the file with a list of the chassis names (the TCP/IPEthernet management port names assigned) or IPaddresses.If you made any changes, save and close the file.

d) Do you want to edit/review/change thefiles? [y]:

Type n and Press Enter.

e) Continue to next step. Press any key to continue.

3. Verify Chassis via Ethernet Ping (menu item 1)

• If all chassis were found… Press any key to continue to Update the ChassisFirmware.

• If some chassis were not found… Press ESC to exit the menu and review the following listfor those chassis which were not found:• Is chassis powered on and booted?• Is chassis connected to management network?• Are chassis IP address and network settings consistent

with DNS or /etc/hosts ?• Is Management node connected to the management

network?• Are Management node IP address and network

settings correct?• Is management network itself up (including switches,

routers, and others)?• Is correct set of chassis listed in the chassis file? You

may need to repeat the previous step to review andedit the file.

4. (Optional)Update Chassis Firmware (menu item 2) NOTE: Before continuing, refer to the Intel® Omni-PathFabric Switches Release Notes for any prerequisites.

a) For Directories, all .spkg files in thedirectory tree will be used.Enter Files/Directories to use (or none):

Specify the directory where the relevant firmware fileshave been stored and press Enter.

b) Would you like to run the firmware now?[n]:

Type y and press Enter.NOTE: FastFabric ensures that all chassis are running thefirmware level provided, and installs and/or reboots eachchassis as needed. If any chassis fails to be updated, usethe View opachassisadmin Result Files option toreview the result files from the update. Refer to theIntel® Omni-Path Fabric Suite FastFabric User Guide formore details.

c) Continue to next step. Press any key to continue.

5. Set Up Chassis Basic Configuration (menu item 3)

a) For each prompt… Accept the defaults.

b) Continue to next step. Press any key to continue.

6. Set Up Password-less ssh/scp (menu item 4)

a) Would you like to override the defaultChassis password? [n]:

Press Enter to accept the default Chassis password.


7. Reboot the Chassis (menu item 5)

a) Performing Chassis Admin: Reboot Chassis Press Enter to accept the default.

continued...




Would you like to be prompted for chassis'password? [n]:


8. Get Basic Chassis Configuration (menu item 6)

a) Continue to next step. Press any key to continue.

9. Configure Chassis Fabric Manager (menu item 7) NOTE: For configuring multi-subnets: At least one subnetmanager is required per subnet. Refer to the Intel®Omni-Path Fabric Suite Fabric Manager User Guide formore information on how to configure a host SM node tomanage more than one subnet.

a) Performing Chassis Admin: ConfigureChassis Fabric Manager (FM)Enter FM Config file to use (or none orgenerate):

Type generate and press Enter.

b) For each prompt… Accept the defaults.

c) Would you like to restart the FM? [n]: Type y and press Enter.

d) Would you like to run the FM on slave MMs?[n]:

Choose:• Type y.

If your fabric has a single chassis running the FabricManager. You can run the Fabric Manager on the slavemanagement module (MM). This causes the FabricManager to be started in the applicable chassis.

• Type n.If our fabric has multiple chassis running the FabricManager. Intel recommends you run Fabric Manageron the master management module. This causes theFabric Manager to be started only on the mastermanagement module in the applicable chassis.

e) Would you like to do the operation inparallel? [y]:

Press Enter.

f) Would you like to enable FM start at boot?[n]:

Type y.

g) Would you like to enable FM start on slaveMMs at boot? [n]:

Choose:• Type y.

If your fabric has a single chassis running the FabricManager. You can run the Fabric Manager on the slavemanagement module. This causes the Fabric Managerto be started in the applicable chassis.

• Type n.If your fabric has multiple chassis running the FabricManager. Intel recommends you run Fabric Manageron the master management module. This causes theFabric Manager to only be started on the mastermanagement module in the applicable chassis.

h) Would you like to be prompted for chassis'password? [n]:

Press Enter.

i) Are you sure you want to proceed? [n]: Type y.

j) Continue to next step. Press any key to continue.

10. Updating the Chassis FM Security Files (menu item 8)

a) Enter Files/Directories to use (or none): NOTE: You must enter a file or the task ends.

continued...




b) For subsequent prompts… Provide the required information and press Enter.


11. Get the Chassis FM Security File (menu item 9)


Verifying the Chassis Configuration

12. Check OPA Fabric Status (menu item a)

a) For each prompt... Provide the required information and press Enter.


13. Control Chassis Fabric Manager (menu item b)


b) Are you sure you want to proceed? [n]: Select y.


14. Generate All Chassis Problem Report Information(menu item c)

a) Would you like to be prompted for chassis'password? [n]:

Press Enter.


15. Run a Command on All Chassis (menu item d)

a) If there are any other operations that need to beperformed on all chassis…

Use the Run a Command on All Chassis option. Eachtime this is executed, a single chassis CLI command maybe specified to be executed against all selected chassis.When using these commands, additional setup orverification of the chassis may be performed. Refer to theIntel® Omni-Path Fabric Suite FastFabric User Guide.


16. View the opachassisadmin Result Files (menu item e)

a) About to: vi /root/punchlist.csv /root/test.res /root/test.log

Press any key to review files or press ESC to abort.


End Task

Next Steps

• To configure the externally-managed switches, go to Configure the Externally-Managed Switches.

Configure the Chassis Using the FastFabric CLI Commands

You can configure the chassis using the CLI commands.

Assumptions



7.3



Procedures

The following steps provide simplified instructions for first-time configuration of thechassis. For additional details, refer to the Intel® Omni-Path Fabric Suite FastFabricUser Guide.


1. (Optional) Edit or review the configuration files.

a) Edit the /etc/opa/opafastfabric.conf file. Review the file with a focus on the following:• FF_CHASSIS_LOGIN_METHOD• FF_CHASSIS_ADMIN_PASSWORD.• Select the location for the result files from FastFabric

with the FF_RESULT_DIR parameter.If you made any changes, save and close the file.

b) Edit the /etc/opa/ports file. Review the file.If you made any changes, save and close the file.

c) Edit the /etc/opa/chassis file. Create the file with a list of the chassis names (the TCP/IPEthernet management port names assigned) or IPaddresses. Note: Do not list externally-managed switchesin this file.If you made any changes, save and close the file.

2. Verify the chassis is accessible over the managementnetwork.

Type opapingall -C -p -f /etc/opa/chassis.

• If all chassis were found… Continue to the next step.

• If some chassis were not found… Review the following list for those chassis that were notfound:• Is chassis powered on and booted?• Is chassis connected to management network?• Are chassis IP address and network settings consistent

with DNS or /etc/hosts ?• Is Management node connected to the management

network?• Are Management node IP address and network

settings correct?• Is management network itself up (including switches,

routers, and others)?• Is correct set of chassis listed in the chassis file? You

may need to repeat the previous step to review andedit the file.

3. Update the chassis firmware using either the parallel orserial method.

NOTE: Before continuing, refer to the Intel® Omni-PathFabric Switches Release Notes for any prerequisites.

a) For the parallel method... Type opachassisadmin -S -F chassisfile -Ppackage -a run upgrade.

b) For the serial method... Type FF_MAX_PARALLEL=0 opachassisadmin -S -Fchassisfile -P package -a run upgrade.

NOTES:During the firmware update, the following files are produced:• test.res – Appended with summary results of run• test.log – Appended with detailed results of run• save_tmp/ – Contains a directory per failed operation with detailed logs• test_tmp*/ – Intermediate result files while operation is runningIf any chassis fails to be updated, refer to the View opachassisadmin result files section to review the result filesfrom the update.

continued...




Refer to the Intel® Omni-Path Fabric Suite FastFabric User Guide for more details.

4. Set up the chassis with the basic configuration settingsusing the command.

Type opachassisadmin -S -F chassisfileconfigure.

a) Follow the system prompts to select and configure theitems required.

This wizard gives you the option of setting up thefollowing items on the chassis:• Syslog server• NTP server• Timezone and DST information• Chassis link width• Configure Node Desc to match Ethernet* chassis

name• Configure the Link CRC Mode

5. Set up secure password-less SSH, such that theManagement Node can securely log into all the chassis asadmin through the management network, withoutrequiring a password.

Type opasetupssh -p -S -C -F chassisfile.

6. Reboot all the selected chassis and ensure they rebootfully, as verified through ping over the managementnetwork.

Type opachassisadmin -S -F chassisfilereboot.

7. Retrieve basic information from chassis such as syslog,NTP configuration, time zone, node description, and otherinformation.

Type opachassisadmin -F chassisfilegetconfig.

8. Configure the Chassis Fabric Manager for any Intel®Omni-Path Chassis 100 Series chassis using either parallelor serial method.

NOTE: For configuring multi-subnets: At least one subnetmanager is required per subnet. Refer to the Intel®Omni-Path Fabric Suite Fabric Manager User Guide formore information on how to configure a host SM node tomanage more than one subnet.

a) For the parallel method... Type opachassisadmin -S -F chassisfile -Ppackage -a run fmconfig.

b) For the serial method... Type FF_MAX_PARALLEL=0 opachassisadmin -S -Fchassisfile -P package -a run fmconfig.

9. Update Chassis Fabric Manager Security Files to permitthe chassis security files to be verified and updated usingeither parallel or serial method, as needed.

Note: The FM security files are the private key, publickey, and certificate files required by the FM, tosupport secure socket connection via OpenSSL.Refer to the Intel® Omni-Path Fabric Suite FabricManager User Guide for instructions on theadministration tasks required to support thesefiles.

a) For the parallel method... Type opachassisadmin -S -F chassisfile -ssecurityFiles -a push fmsecurityfiles.

b) For the serial method... Type FF_MAX_PARALLEL=0 opachassisadmin -S -Fchassisfile -s securityFiles -a pushfmsecurityfiles.

10. Retrieve the Chassis Fabric Manager Security Files fromthe chassis.

Type opachassisadmin -F chassisfilefmgetsecurityfiles.

Verifying the Chassis Configuration

11. Check the OPA fabric status. Type opalinkanalysis -U -x snapshot_suffixreports verifyall > results_file 2>&1.NOTE: This step performs the following operations:• Perform a fabric error analysis.• Clear the error counters after generating a report.

continued...




• Perform a fabric link speed error analysis.• Check for links configured to run slower than

supported.• Check for links connected with mismatched speed

potential.• Enter a filename for the results or save the results to

the default location which is: /root/ffres/linkanalysis.res

12. To control the Chassis Fabric Manager, you can performany or all of the following optional steps:

a) Restart all the Chassis Fabric Managers and run theFabric Manager on the slave Management Module usingeither parallel or serial methods.

NOTE: There is a disruption as FMs are restarted. Usingthe serial method may reduce the disruption.

• For parallel method... Type opachassisadmin -S -F chassisfile -arestartall fmcontrol.

• For serial method... Type FF_MAX_PARALLEL=0 opachassisadmin -S -Fchassisfile -a restartall fmcontrol.

b) Restart the Master Chassis Fabric Managers and stopthe Fabric Manager on the slave Management Modulesusing either parallel or serial methods.

NOTE: There is a disruption as FMs are restarted. Usingthe serial method may reduce the disruption.

• For parallel method... Type opachassisadmin -S -F chassisfile -arestart fmcontrol.

• For serial method... Type FF_MAX_PARALLEL=0 opachassisadmin -S -Fchassisfile -a restart fmcontrol.

c) Stop the Master Chassis Fabric Managers from runningusing either parallel or serial methods.

• For parallel method... Type opachassisadmin -S -F chassisfile -astop fmcontrol.

• For serial method... Type FF_MAX_PARALLEL=0 opachassisadmin -S -Fchassisfile -a stop fmcontrol.

d) Ensure all of the Chassis Fabric Managers are running,including the slaves using either parallel or serialmethods.

• For parallel method... Type opachassisadmin -S -F chassisfile -arunall fmcontrol.

• For serial method... Type FF_MAX_PARALLEL=0 opachassisadmin -S -Fchassisfile -a runall fmcontrol.

e) Ensure the Master Chassis Fabric Manager is runningand the slaves are stopped using either parallel or serialmethods.

• For parallel method... Type opachassisadmin -S -F chassisfile -a runfmcontrol.

• For serial method... Type FF_MAX_PARALLEL=0 opachassisadmin -S -Fchassisfile -a run fmcontrol.

f) Change the Fabric Manager boot state to enable theMaster Chassis Fabric Manager to start at boot.

Type opachassisadmin -S -F chassisfile -Ienable.

g) Change the Fabric Manager boot state to enable the allof the Chassis Fabric Managers to start at boot.

Type opachassisadmin -S -F chassisfile -Ienableall.

continued...




13. Generate all chassis problem report information. Type opacaptureall -p -D 4 -f hostfile .

14. Run a command on all chassis. Type opacmdall -C -S -p -T timelimit -Fchassisfile STDIN.

15. View opachassisadmin result files Type editor result_dir/result_file.NOTE: In the line above, "editor" indicates thecommand line editor; for example, vi.The following default files are created:• punchlist.csv• test.res• test.log

End Task

Next Steps

• To configure the externally-managed switches, go to Configure the Externally-Managed Switches.



8.0 Configure the Externally-Managed Switches

This section provides information and procedures to configure the externally-managedswitches, such as Intel® Omni-Path Switch 100 Series using theIntel® Omni-PathFabric Suite FastFabric toolset.

Before You Begin

Before starting the switch configuration, perform the following:

• Be sure you have completed all Switch Configuration Prerequisites on page 53.



Note that you can edit the files before you configure the switch. Configuration filesare located under the /etc/opa directory. Sample files are installed into /usr/share/opa/samples with the suffix -sample.

The following files are used to configure the switch:


— ports: Lists the local HFI ports to use to access the fabric.

— switches: Lists the GUID, Node Description, and optional Distance of eachexternally-managed switch in the cluster.

Create the file with a list of the switch node GUID and required switch names.Enter one switch node GUID and required switch name per line. Do not useany spaces before or after the comma separating the switch node GUID andthe name. Note: Do not list internally-managed chassis in this file.

For example:

0x00117500d9000138,edge10x00117500d9000139,edge2




8.1

Intel® Omni-Path Fabric—Configure the Externally-Managed Switches


Configure the Externally-Managed Switches UsingFastFabric TUI Menu

You configure the externally-managed switches using the FastFabric OPA SwitchSetup/Admin menu.

Assumptions



Procedures

The following steps provide simplified instructions for first-time configuration of theswitch. For additional details, refer to the Intel® Omni-Path Fabric Suite FastFabricUser Guide.


Configuring the Externally-Managed Switches

1. Access the FastFabric OPA Switch Setup/Adminmenu.



b) Access the FastFabric OPA Switch Setup/Adminmenu.

Press 2.

c) Select menu items. Select all operations by entering each menu item 0 – a.NOTE: You can skip any items that are not needed foryour fabric. However, this procedure assumes that allitems have been selected.


2. Edit the Configuration and Select/Edit Switch File(menu item 0)

a) Edit the opafastfabric.conf file. Review the file.If you made any changes, save and close the file.Press any key to continue.

b) Edit the ports configuration file. Review the file.If you made any changes, save and close the file.Press any key to continue.

c) Edit the switches configuration file. Create the file with a list of the switch node GUID andrequired switch names.If you made any changes, save and close the file.

d) Do you want to edit/review/change thefiles? [y]:



3. Generate or Update Switch File (menu item 1)

a) For each prompt… Provide the required information and press Enter.


continued...

8.2

Configure the Externally-Managed Switches—Intel® Omni-Path Fabric



4. Test for Switch Presence (menu item 2)

• If all switches were found... Press any key to continue to Verify Switch Firmware.

• If some switches were not found... Press ESC to exit the menu and review the following listfor those switches that were not found:• Is switch powered on and booted?• Is switch connected to Intel® Omni-Path Fabric?• Is Subnet Manager running?• Is Management Node’s Port active?• Is Management Node connected to the correct Intel®

Omni-Path Fabric?• Is FM Switch LED activated on the switch port to which

the Fabric Management node is connected? For moreinformation, refer to the "FM Switch" section in theIntel® Omni-Path Fabric Switches HardwareInstallation Guide.

• Is the correct set of switches listed in the switchesfile? You may need to repeat the previous step toreview and edit the file.

5. Verify Switch Firmware (menu item 3)


6. Update Switch Firmware (menu item 4) Note: Refer to the Intel® Omni-Path Fabric SwitchesRelease Notes to ensure that any prerequisites forthe upgrade to the new firmware level have beenmet prior to performing the upgrade throughFastFabric.

a) Enter Files/Directories to use (or none): Enter the directory where the relevant firmware files arelocated.

b) Would you like to reboot the switch afterthe update? [n]:

Type y.

c) For subsequent prompts... Press Enter to accept the defaults.

d) Continue to next step. Press any key to continue.

7. Set Up Switch Basic Configuration (menu item 5)

a) For each prompt... Press Enter to accept the defaults.


8. Reboot Switch (menu item 6)


9. Report Switch Firmware & Hardware Info (menuitem 7)


10. Get Basic Switch configuration (menu item 8)


11. Report Switch VPD Information (menu item 9)


Verifying the Switch Configuration

12. View opaswitchadmin Result Files (menu item a)

continued...




a) About to: vi /root/punchlist.csv /root/test.res /root/test.log

Press any key to review files or press ESC to abort.

b) Would you like to remove test.res test.logtest_tmp* and save_tmp in /root ? [n]:

Press Enter to save or type y to remove the files.

End Task

Next Steps

• To install the host software on the remaining servers, go to Install Host Softwareon Remaining Servers.

Configure the Externally-Managed Switches UsingFastFabric CLI Commands

You configure the externally-managed switches using the FastFabric OPA SwitchSetup/Admin menu.

Assumptions



Procedures

The following steps provide simplified instructions for first-time configuration of theswitch. For additional details, refer to the Intel® Omni-Path Fabric Suite FastFabricUser Guide.



a) Edit the /etc/opa/opafastfabric.conf file. Review the file.If you made any changes, save and close the file.

b) Edit the /etc/opa/ports file. Review the file.If you made any changes, save and close the file.

c) Edit the /etc/opa/switches file. Create the file with a list of the switch node GUID andrequired switch names.If you made any changes, save and close the file.

2. (Optional) Generate or update the switches file.NOTE: You can also update switch names in theswitches file by comparing the actual fabric to topologyxml data.

Type opagenswitches -s -o switches.

3. Test to see if each externally-managed switch is present. Type opaswitchadmin -L switchfile ping.

a) If all switches were found... Continue to the next step.

b) I some switches were not found... Review the following list for those switches that were notfound:• Is switch powered on and booted?• Is switch connected to Intel® Omni-Path Fabric?• Is Subnet Manager running?• Is Management Node’s Port active?

continued...

8.3

Configure the Externally-Managed Switches—Intel® Omni-Path Fabric



• Is Management Node connected to the correct Intel®Omni-Path Fabric?

• Is FM Switch LED activated on the switch port to whichthe Fabric Management node is connected? For moreinformation, refer to the "FM Switch" section in theIntel® Omni-Path Fabric Switches HardwareInstallation Guide.

• Is the correct set of switches listed in the switchesfile? You may need to repeat the previous step toreview and edit the file.

4. Verify the switch firmware to check that each externallymanaged switch is operational and that its firmware isvalid and accessible.

Type opaswitchadmin -L switchfile fwverify.

5. Update the switch firmware version and the switch nodename set.

NOTE: Be aware that non-parallel operation for a fabricwith many externally-managed switches can take asignificant amount of time.

• For the parallel method... Type opaswitchadmin -O -L switchfile -Ppackages -a run upgrade.

• For the serial method... Type FF_MAX_PARALLEL=0 opaswitchadmin -O -Lswitchfile -P packages -a run upgrade.

6. Reboot the externally managed switches. Type opaswitchadmin -L switchfile reboot.

7. Set up the switch basic configuration for all externallymanaged switches.

Type opaswitchadmin -L switchfile configure.NOTE: This command runs a wizard to set up theexternally-managed switch configuration.

8. Run a report for the firmware and hardware versions foreach switch, along with the part number and data ratecapability of all of the externally managed switches.

Type opaswitchadmin -L switchfile info.

9. Run the command to get the externally managed switchconfiguration report for all of the ports.

Type opaswitchadmin -L switchfile getconfig.

10. Run the report that returns externally managed switchhardware vital product data (VPD) for all of the nodeslisted in the /etc/opa/switches file.

Type opaswitchadmin -L switchfile hwvpd.

Verifying the Switch Configuration

11. View the opaswitchadmin results files. Type editor result_dir/result_file.NOTE: In the line above, "editor" indicates thecommand line editor; for example, vi.The following default files are created:• punchlist.csv• test.res• test.log

End Task

Next Steps

• To install the host software on the remaining servers, go to Install Host Softwareon Remaining Servers.



9.0 Install Host Software on Remaining Servers

This section provides information and procedures to install, configure, and verify theIntel® Omni-Path Fabric Host Software on the remaining hosts.

Note: Intel® Omni-Path Fabric Suite FastFabric is also used to install the Intel® Omni-PathFabric Stack Tools on the remaining hosts when using other variations of OFA. In thiscase, OFA software must be installed on each host prior to installing the Intel® OPAsoftware.

Before You Begin

Before starting the host installation and configuration, perform the following:


Note that you can edit the files before you configure the hosts. Configuration filesare located under the /etc/opa directory. Sample files are installed into /usr/share/opa/samples with the suffix -sample.

The following files are used to configure the hosts:


Note: During setup of password-less SSH, FastFabric provides the opportunityto enter the host root password interactively when needed. Therefore,Intel recommends that you do not place it within theopafastfabric.conf file.

If you are required to keep the root password for the hosts in theopafastfabric.conf file, Intel recommends that you change theopafastfabric.conf permissions to be 0x600 (root-only access).

• Intel recommends that a FastFabric topology file is created as /etc/opa/topology.0:0.xml to describe the intended topology of the fabric. Thefile can also augment assorted fabric reports with customer-specificinformation, such as cable labels and additional details about nodes, SMs,links, ports, and cables. Refer to the Intel® Omni-Path Fabric SuiteFastFabric User Guide for more information about topology verificationfiles.

— ports: Lists the local HFI ports to use to access the fabric.

• For typical single-subnet clusters, the default of "0:0" may be used. Thisuses the first active port on the Management Node to access all externallymanaged switches.

— hosts: List of the hosts names (the TCP/IP management network names),except the Management Node from which FastFabric is presently being run.Enter one host’s name per line.

9.1

Install Host Software on Remaining Servers—Intel® Omni-Path Fabric


For example:

host1host2

Note: Do not list the Management Node itself (that is, the node whereFastFabric is currently running).

If additional Management Nodes are to be used, they may be listed atthis time, and FastFabric can aid in their initial installation andverification.

— allhosts: Lists the Management Node's hosts name (the TCP/IPmanagement network name, for example mgmthost) and includes the hostsfile.

For example:

mgmthostinclude /etc/opa/hosts

— hostverify.sh: Script to verify the configuration and performance of anindividual node. This should be run using opaverifyhosts. It can also berun on an individual node directly.




• For the following Host Setup menu item, prepare to answer questions or provideinput as follows:

— Building Test Apps and Copying to Hosts

Choose MPI Directory Selection:

• 0) /usr/mpi/gcc/mvapich2-x.x

• 1) /usr/mpi/gcc/mvapich2-x.x-hfi

• 2) /usr/mpi/gcc/openmpi-x.x.x

• 3) /usr/mpi/gcc/openmpi-x.x.x-hfi

• 4) /usr/mpi/intel/mvapich2-x.x-hfi

• 5) /usr/mpi/intel/openmpi-x.x.x-hfi

• 6) /usr/mpi/pgi/mvapich2-x.x-hfi

• 7) /usr/mpi/pgi/openmpi-x.x.x-hfi

• 8) Enter Other Directory

Choose MPI Directory Selection for SHMEM Job Launch

• 0) /usr/mpi/gcc/openmpi-x.x.x

• 1) /usr/mpi/gcc/openmpi-x.x.x-hfi

• 2) /usr/mpi/intel/openmpi-x.x.x-hfi

Intel® Omni-Path Fabric—Install Host Software on Remaining Servers


• 3) /usr/mpi/pgi/openmpi-x.x.x-hfi

• 4) Enter Other Directory

• 5) Skip MPI Directory Selection for SHMEM Job Launch

Note: "x.x" or "x.x.x" above refers to the latest supported versions defined in theIntel® Omni-Path Fabric Software Release Notes.

• For the following Host Verify menu item, prepare to answer questions or provideinput as follows:

— Perform Single Host Verification

• Prior to using this operation, you must have a copy of thehostverify.sh in the directory pointed to by FF_HOSTVERIFY_DIR. Ifthe file does not exist in that directory, copy the sample file /usr/share/opa/samples/hostverify.sh to the directory pointed to byFF_HOSTVERIFY_DIR.

Refer to the Intel® Omni-Path Fabric Suite FastFabric User Guide forinformation on opaverifyhosts.

• Determine the tests to be run and time limit (in minutes) for the tests.

Install the Host Software on the Remaining Hosts Usingthe FastFabric TUI Menu

You can configure the host software using the FastFabric OPA Host Setup menu.

Assumption


Procedures

The following steps provide simplified instructions for first-time configuration of thehosts. For additional details, refer to the Intel® Omni-Path Fabric Suite FastFabric UserGuide.


Configuring the Hosts

1. Access the FastFabric OPA Host Setup menu.



b) Access the FastFabric OPA Host Setup menu. Press 3.

c) Select menu items. • Select items 0 – 2 and 4 – 8.• Select item 3 if you are using /etc/hosts for name

resolution (as opposed to using DNS).


2. Edit Configuration and Select/Edit Host File (menuitem 0)

a) Edit the opafastfabric.conf file. Review the file with a focus on the following:• FF_IPOIB_SUFFIX

continued...

9.2




• FF_IPOIB_NETMASK• FF_IPOIB_CONFIG• FF_PRODUCT• FF_PACKAGES• FF_INSTALL_OPTIONS• FF_UPGRADE_OPTIONSIf you made any changes, save and close the file.Press any key to continue.

b) Edit the hosts configuration file. Create the file with a list of the hosts names (the TCP/IPmanagement network names), except the ManagementNode from which FastFabric is presently being run.If you made any changes, save and close the file.

c) Do you want to edit/review/change thefiles? [y]:



3. Verify Hosts Pingable (menu item 1)

• If all hosts were found... Press any key to continue to Set Up Password-LessSSH/SCP.

• If some hosts were not found... Press ESC to exit the menu and review the following listfor those hosts that were not found:• Host powered on and booted?• Host connected to management network?• Host management network IP address and network

settings consistent with DNS or /etc/hosts?• Management node connected to the management

network?• Management node IP address and network settings

correct?• Management network itself up (including switches,

routers, and others)?• Correct set of hosts listed in the hosts file? You may

need to repeat the previous step to review and editthe file.

4. Set Up Password-Less SSH/SCP (menu item 2)

a) Password for root on all hosts: Type the password for root on all hosts and press Enter.


5. Copy /etc/hosts to All Hosts (menu item 3)


6. Show uname -a for All Hosts (menu item 4) Note: Refer to the Intel® Omni-Path Fabric SwitchesRelease Notes to ensure that any prerequisites forthe upgrade to the new firmware level have beenmet prior to performing the upgrade throughFastFabric.


7. Install/Upgrade OPA Software (menu item 5) Note: An initial installation uninstalls any existing OFADelta or IFS software. Initial installs must beperformed when installing on a clean system or ona system that has stock OFA installed. To upgradethe fabric, refer to Upgrade the Software on theRemaining Servers on page 126.

continued...




a) Do you want to use ./IntelOPA-[Basic|IFS].DISTRO.VERSION.tgz? [y]:

Press Enter to accept the default.

b) Would you like to do a fresh [i]nstall, an[u]pgrade or [s]kip this step? [u]:

Type i and press Enter.

c) Are you sure you want to proceed? [n]: Type y and press Enter.

d) Complete the installation. Press any key to continue.

• If all hosts install... Press any key to continue.

• If any hosts fails to install... Use the View opahostadmin Results Files menu itemto review the result files from the update.

8. Configure IPoIB IP Address (menu item 6)


9. Build Test Apps and Copy to Hosts (menu item 7)

a) For each prompt... Provide your selections and press Enter.


10. Reboot Hosts (menu item 8)


11. Ensure the hosts fully reboot, as verified through pingover the management network.

Perform Step 3.

Optional TasksRefer to the Intel® Omni-Path Fabric Suite FastFabric User Guide for more information.

11. Refresh SSH Known Hosts (menu item 9) This menu item refreshes the SSH known hosts list onthis server for the Management Network. This may beused to update security for this host if hosts are replaced,reinstalled, renamed, or repaired.

12. Rebuild MPI Library and Tools (menu item a) This menu item prompts you for selection of which MPI torebuild, and provides choices as to which availablecompiler to use.

13. Run a Command on All Hosts (menu item b) Intel recommends that you run the date command on allhosts to verify that the date and time are consistent. Ifneeded, use the Copy a File to All Hosts menu item tocopy the appropriate files to all hosts to enable andconfigure NTP.

14. Copy a File to All Hosts (menu item c) A file on the local host may be specified to be copied to allselected hosts.

Verifying the Host Configuration

15. View opahostadmin Result Files (menu item d)

a) About to: vi /root/test.res /root/test.log Press any key to review files.



End Task

Next Steps

• To verify the host software is installed and running on the remaining servers, goto Verify the Host Software on the Remaining Servers Using the FastFabric TUIMenu.



Verify the Host Software on the Remaining Servers Usingthe FastFabric TUI Menu

You can verify the host software using the FastFabric OPA Host Verification/Adminmenu.

Note: As a result of running this sequence, a punchlist.csv file is produced. This fileprovides a cumulative summary of tests that failed and may be provided totechnicians for corrective action. The file can easily be imported into spreadsheets orother tools.

Assumption


Procedures

The following steps provide simplified instructions for first-time verification of thehosts. For additional details, refer to the Intel® Omni-Path Fabric Suite FastFabric UserGuide.


1. Access the FastFabric OPA Host Verification/AdminMenu.



b) Access the OPA Host Verification/Admin Menu. Press 4.

c) Select menu items. Select items 0 – 8.



a) Edit the opafastfabric.conf file. Review the file with a focus on the following:• FF_TOPOLOGY_FILE• FF_IPOIB_SUFFIX• FF_DEVIATION_ARGS• ff_host_basename_to_ipoib• ff_host_basenameReview the following parameters which are used foroverall fabric health checks:• FF_ANALYSIS_DIR• FF_ALL_ANALYSIS• FF_FABRIC_HEALTH• FF_CHASSIS_CMDS• FF_CHASSIS_HEALTH• FF_ESM_CMDSFF_ALL_ANALYSIS should be updated to reflect the typeof SM (esm or hostsm).If you made any changes, save and close the file.Press any key to continue.

b) Edit the ports configuration file. Review the file.If you made any changes, save and close the file.

continued...

9.3




Press any key to continue.

b) Create or edit the allhosts configuration file. Create the file with the Management Node's hosts name(the TCP/IP management network name, for examplemgmthost) and include the hosts file.

Note: If you have a cluster with mixed servers or HFIconfigurations, create a /etc/opa/*hosts filefor each type of server configuration

If you made any changes, save and close the file.




3. Summary of Fabric Components (menu item 1)

a) After the operation completes... Review the results against the expected configuration ofthe cluster.NOTE: If components are missing, or degraded oromitted links are found, they should be corrected.Subsequent steps aid in locating any such links.


4. Verify Hosts Pingable, SSHable and Active (menuitem 2)


b) After completion of the tests, you are prompted:Would you like to now use /etc/opa/good asHost File? [y]:

Press Enter to use the file or n to discard the file.

5. Perform Single Host Verification (menu item 3)

a) Would you like to edit /root/hostverify.shand copy to hosts? [y]:

Review the settings near the top and the list of TESTSselected.Note: If you have a cluster with mixed servers or HFI

configurations, ensure you add the proper settingsfor the server configuration (HFI PCIe bus, servermemory size, expected single node HPLperformance for server, etc) to the /root/hostverify.sh file


b) For each prompt... Provide the required information and press Enter.

c) Start the tests. Press any key to continue.

d) Review the results file. Press any key to view the file.Close the file.

e) Repeat the Perform Single Host Verification foreach of the host files.

Note: If you have a cluster with mixed servers or HFIconfigurations, Step 5 needs to be repeated foreach *host file

f) Continue to next step. Press any key to continue.

6. Verify OPA Fabric Status and Topology (menu item 4)



7. Verify Hosts See Each Other (menu item 5)

continued...





8. Verify Hosts Ping via IPoIB (menu item 6)


9. Refresh SSH Known Hosts (menu item 7)


10. Check MPI Performance (menu item 8) NOTE: This test identifies nodes whose performance isnot consistent with others in the fabric. It is not intendedas a benchmark of fabric latency and bandwidth. This testintentionally uses techniques to reduce test runtime.

a) For each prompt... Press Enter to select the defaults.

• If all hosts pass... Continue to the next step.

• If any hosts fail... • Carefully examine the failing hosts to verify the HFImodels, PCIe slot used, BIOS settings, and anymotherboard or BIOS settings related to devices onPCIe buses or slot speeds.

• Also verify that the HFI and any riser cards areproperly seated.

Refer to the Intel® Omni-Path Fabric Suite FastFabricUser Guide for more information.



11. Check Overall Fabric Health (menu item 9) This command permits the present fabric configuration tobe baselined for use in future fabric health checks.Perform this check after configuring any additionalManagement Nodes and establishing a healthy fabric viasuccessful execution of all the other tests discussed inthis section. If desired, a baseline of an incomplete orunhealthy fabric may be taken for future comparison aftermaking additions or corrections to the fabric. Refer to Configure and Initialize Health Check Tools UsingFastFabric CLI Commands on page 90 for moreinformation.

12. Start or Stop Bit Error Rate Cable Test (menu item a) This command performs host and/or ISL cable testing.The test allows for starting and stopping an extended BitError Rate test. The system prompts to clear hardwarecounters.

13. Generate All Hosts Problem Report Info (menu itemb)

This command collects configuration and statusinformation from all hosts and generates a single *.tgzfile that can be sent to an Intel support representative.

14. Run a Command on All Hosts (menu item c) This command runs the cmdall command. A Linux* shellcommand may be specified to be executed against allselected hosts. You may also specify a sequence ofcommands separated by semicolons.

Review the Host Verification

15. View opahostadmin Result Files (menu item d) A file on the local host may be specified to be copied to allselected hosts.

continued...







End Task

Next Steps

• To set up and configure the management software for additional managementnodes, go to Set Up Additional Management Nodes.

Install the Host Software on the Remaining Hosts Usingthe FastFabric CLI Commands

You can configure the host sofware using the FastFabric OPA Host Setup menu.

Assumption


Procedures

The following steps provide simplified instructions for first-time configuration of thehosts. For additional details, refer to the Intel® Omni-Path Fabric Suite FastFabric UserGuide.



a) Edit the /etc/opa/opafastfabric.conf file. Review the file with a focus on the following:• FF_IPOIB_SUFFIX• FF_IPOIB_NETMASK• FF_IPOIB_CONFIG• FF_PRODUCT• FF_PACKAGES• FF_INSTALL_OPTIONS• FF_UPGRADE_OPTIONSIf you made any changes, save and close the file.

b) Create or edit the /etc/opa/hosts configuration file. Create the file with a list of the hosts names (the TCP/IPmanagement network names), except the ManagementNode from which FastFabric is presently being run.If you made any changes, save and close the file.

2. Verify the hosts are pingable over the managementnetwork.

Type opapingall -p.

• If all hosts were found... Continue to the next step.

• If some hosts were not found... Review the following list for those hosts that were notfound:• Host powered on and booted?• Host connected to management network?• Host management network IP address and network


network?

continued...

9.4




• Management node IP address and network settingscorrect?

• Management network itself up (including switches,routers, and others)?

• Correct set of hosts listed in the hosts file? You mayneed to repeat the previous step to review and editthe file.

3. Set up secure password-less SSH, such that theManagement Node can securely log into all the hosts asroot through the management network, withoutrequiring a password.

Type opasetupssh -S -p -i "" -f hostfile.

4. (Optional) Copy the /etc/hosts file on this host to allthe other selected hosts.NOTE: If DNS is being used, skip this step.

Type opascpall -p -f hostfile /etc/hosts /etc/hosts.

5. (Optional) Copy the /etc/resolv.conf file on this hostto all the other selected hosts.

Type opascpall -p -f hostfile /etc/resolv.conf /etc/resolv.conf.

6. Show uname -a (OS version) on all the hosts. Type opacmdall -T 60 -f hostfile 'uname -a'.

7. Install the Intel® Omni-Path Fabric Host Software on allthe hosts.Note: An initial installation uninstalls any existing OFA

Delta or IFS software. Initial installs must beperformed when installing on a clean system or ona system that has stock OFA installed. To upgradethe fabric, refer to Upgrade the Software on theRemaining Servers on page 126.

Type opahostadmin -f hostfile -d dir load.By default, it looks in the current directory for theIntelOPA-Basic.DISTRO.VERSION.tgz file.

• If all hosts install... Continue to the next step.

• If any hosts fails to install... Use the View opahostadmin result files option to reviewthe result files from the update.

8. Configure IPoIB IP Address.This creates the ifcfg-ib0 files on each host.

Type opahostadmin -f hostfile configipoib.NOTE: The file is created with a statically assigned IPv4address. The IPoIB IP address for each host is determinedby the resolver (Linux* host command). If not foundthrough the resolver, /etc/hosts on the given host ischecked.

9. Build the MPI and/or SHMEM sample applications on theManagement Node and copy the resulting object files toall the hosts.Note: This is in preparation for execution of MPI and/or

SHMEM performance tests and benchmarks in alater step.

Note: This option is only available when using the Intel®Omni-Path Fabric Host Software packaging of OFA(OFA Delta).

Type MPICH_PREFIX=path_to_mpicd /usr/src/opa/mpi_apps; make clobber quickopascpall -t -p -f hostfile source_dirdest_dir.

10. Reboot all the selected hosts. Type opahostadmin -f hostfile reboot.

11. Ensure the hosts fully reboot, as verified through pingover the management network.

Perform Step 2.

Verifying the Host Configuration

15. View opahostadmin result files. Type editor result_dir/result_file.NOTE: In the line above, "editor" indicates thecommand line editor; for example, vi.The following default files are created:• test.res

continued...




• test.log

End Task

Next Steps

• To verify the host software is installed and running on the remaining servers, goto Verify the Host Software on the Remaining Servers Using CLI Commands.

Verify the Host Software on the Remaining Servers UsingCLI Commands

You can verify the host software using the FastFabric OPA Host Verification/Adminmenu.

Note: As a result of running this sequence, a punchlist.csv file is produced. This fileprovides a cumulative summary of tests that failed and may be provided totechnicians for corrective action. The file can easily be imported into spreadsheets orother tools.

Assumption


Procedures

The following steps provide simplified instructions for first-time verification of thehosts. For additional details, refer to the Intel® Omni-Path Fabric Suite FastFabric UserGuide.



a) Edit the /etc/opa/opafastfabric.conf file. Review the file with a focus on the following:• FF_TOPOLOGY_FILE• FF_IPOIB_SUFFIX• FF_DEVIATION_ARGS• ff_host_basename_to_ipoib• ff_host_basenameReview the following parameters which are used foroverall fabric health checks:• FF_ANALYSIS_DIR• FF_ALL_ANALYSIS• FF_FABRIC_HEALTH• FF_CHASSIS_CMDS• FF_CHASSIS_HEALTH• FF_ESM_CMDSFF_ALL_ANALYSIS should be updated to reflect the typeof SM (esm or hostsm).If you made any changes, save and close the file.Press any key to continue.

b) Edit the /etc/opa/ports configuration file. Review the file.If you made any changes, save and close the file.Press any key to continue.

continued...

9.5




b) Create or edit the /etc/opa/allhosts configurationfile.

Create the file with the Management Node's hosts name(the TCP/IP management network name, for examplemgmthost) and include the hosts file.

Note: If you have a cluster with mixed servers or HFIconfigurations, create a /etc/opa/*hosts filefor each type of server configuration


2. Provide a brief summary of the counts of components inthe fabric, including how many switch chips, hosts, andlinks are in the fabric.

Type opafabricinfo.

a) After the operation completes... Review the results against the expected configuration ofthe cluster.NOTE: If components are missing, or degraded oromitted links are found, they should be corrected.Subsequent steps aid in locating any such links.

3. (Optional) Verify each host is pingable. Type opapingall -p -f hostfile.

• If all hosts were found... Continue to the next step.

• If some hosts were not found... Review the following list for those hosts that were notfound:• Host powered on and booted?• Host connected to management network?• Host management network IP address and network


network?• Management node IP address and network settings

correct?• Management network itself up (including switches,

routers, and others)?• Correct set of hosts listed in the hosts file? You may

need to repeat the previous step to review and editthe file.

4. Verify each host is pingable, SSH-enabled, and active onthe Intel® Omni-Path Fabric and produce a list of goodhosts meeting all criteria.

Type opafindgood -R -A -Q -f hostfile.

a) Review the opasorthosts files. The following files are created in opasorthosts orderwith all duplicates removed in the OPA_CONFIG_DIR/directory:• good• alive• running• active• bad• quarantined

5. Perform a single host test on all hosts. Type opaverifyhosts -k -c -u hostverify.res -T timelimit -f hostfile test.

Note: If you have a cluster with mixed servers or HFIconfigurations, ensure you add the proper settingsfor the server configuration (HFI PCIe bus, servermemory size, expected single node HPLperformance for server, etc) to the /root/hostverify.sh file.

Repeat the this step for each of the *host files.

continued...




6. Verify OPA Fabric status and topology. Type opalinkanalysis -U -x snapshot_suffixall verifyall > $FF_RESULT_DIR/linkanalysis.res 2>&1.NOTE: The results can be seen in the $FF_RESULT_DIR/linkanalysis.res file. A punch list of issues isappended to the $FF_RESULT_DIR/punchlist.csvfile.

a) (Optional) To clear error counters after generating thereport...

Add clearerrors and optionally clearhwerrorsoptions to the opalinkanalysis run.

Note: Clearing of hardware counters (-A option) isoptional and may affect the PM and other toolsSee "PM Running Counters to Support opareport"section in the Intel® Omni-Path Fabric Suite FabricManager User Guide for more information.

7. Verify that each host can see all the others throughqueries to the Subnet Administrator.

Type opahostadmin -f hostfile sacache.

8. Verify that IPoIB is properly configured and running on allthe hosts.

Type opahostadmin -f hostfile ipoibping.

• If successful... Continue to next step.

• If not successful... Verify that the management host has IPoIB configured.

9. Refresh the SSH known_hosts file on the ManagementNode to include the IPoIB hostnames of all the hosts.

Type opasetupssh -p -U -f hostfile.

10. Perform a quick check of PCIe and MPI performancethrough end-to-end latency and bandwidth tests.NOTE: This test identifies nodes whose performance isnot consistent with others in the fabric. It is not intendedas a benchmark of fabric latency and bandwidth. This testintentionally uses techniques to reduce test runtime.

Type opahostadmin -f hostfilempiperfdeviation.

• If all hosts pass... Continue to the next step.

• If any hosts fail... • Carefully examine the failing hosts to verify the HFImodels, PCIe slot used, BIOS settings, and anymotherboard or BIOS settings related to devices onPCIe buses or slot speeds.

• Also verify that the HFI and any riser cards areproperly seated.

Refer to the Intel® Omni-Path Fabric Suite FastFabricUser Guide for more information.

Optional Tasks

11 Baseline the present fabric configuration for use in futurefabric health checks.NOTE: This should be performed after configuring anyadditional Management Nodes.

Type opaallanalysis -b.

12. Perform host and/or ISL cable testing.NOTE: The test allows for starting and stopping anextended Bit Error Rate test. The system prompts to clearhardware counters.NOTE: Intel recommends that you run this test for 20-60minutes for a thorough test. While the test is running,monitor the fabric for signal integrity or stability errorsusing opatop, opareport, and/or the Fabric ManagerGUI. Once the desired test time has elapsed, stop the testusing the command:

Type opacabletest -C -f hostfile start.

a) To stop the test... Type opacabletest -f hostfile stop.

continued...




Review the Host Verification

13. View opahostadmin result files. Type editor result_dir/result_file.NOTE: In the line above, "editor" indicates thecommand line editor; for example, vi.The following default files are created:• test.res• test.log

End Task

Next Steps

• To set up and configure the management software for additional managementnodes, go to Set Up Additional Management Nodes.



10.0 Set Up Additional Management Nodes

If the fabric has more than one Management Node, you need to set up additionalmanagement nodes. Previous sections in this installation guide described how toperform basic software installation, setup, and verification on the nodes in a fabric.This section describes how to install and configure the management software itself.

Note: The following procedures assume a symmetrical configuration where all ManagementNodes have the same connectivity and capabilities.

In asymmetrical configurations where the Management Nodes are not all connected tothe same set of management networks and subnets, the files copied to eachmanagement node may need to be slightly different. For example, configuration filesfor fabric analysis may indicate different port numbers, or host files used forFastFabric, and MPI may need to list different hosts.

For multiple-subnet configurations, refer to Multi-Subnet Fabrics on page 106.

Before You Begin

Before starting the set up of the additional management nodes, perform the following:

• If required, upgrade the software to add additional IFS components using theprocedure documented in Upgrade from OPA-Basic to OPA-IFS Software Packageon page 128.

— The Management Node must have at least Intel® Omni-Path Fabric SuiteFastFabric, the Intel® Omni-Path Fabric Stack, and should have IPoIB installedand configured.

— If this node is intended to also run a redundant fabric manager, install theIntel® Omni-Path Fabric Suite Fabric Manager.

— For MPI clusters, the Management Node should also include at least OFAopenmpi or OFA mvapich2. If you plan to rebuild MPI, the Intel® Omni-PathFabric Host Software Fabric Development package and MPI Source packagesare also required.

Note: Do not uninstall or replace existing configuration files that were previouslycreated, especially IPoIB-related configuration files.

Configure Additional Management Nodes Using FastFabricTUI Menus

You can configure additional management nodes using the FastFabric TUI menus.

Assumption

• You are logged into the target Management Node.

10.1

10.2

Set Up Additional Management Nodes—Intel® Omni-Path Fabric


Procedures

Repeat the following steps on each additional Management Node.


1. Copy the FastFabric configuration files from the initialManagement Node to each new management node.

At a minimum, the following files should be copied:

/etc/opa/opafastfabric.conf/etc/opa/ports/etc/opa/topology*.xml/etc/opa/hosts/etc/opa/allhosts/etc/opa/switches/etc/opa/chassis

2. Log into the FastFabric TUI.At the prompt...

Type opafastfabric.

3. From the main menu, access the FastFabric OPA HostSetup menu.

Press 3.

a) Edit Configuration and Select/Edit Host File Select 0 and press P to start the operation.

b) Edit the hosts file. Edit the hosts file such that the file on each ManagementNode omits itself.

c) Complete the task. Press X to return to the main menu.

4. Access the OPA Host Verification/Admin Menu. Press 4.

a) Edit Configuration and Select/Edit Host File Select 0 and press P to start the operation.

b) Edit the allhosts file. Edit the allhosts file such that the file on eachManagement Node specifies itself.

c) Complete the task. Press X to return to the main menu.

5. If Fabric Manager is to be run on the new managementnodes, copy the Fabric Manager configuration file (/etc/opa-fm/opafm.xml) from the initial Management Nodeto each new management node.

a) Edit the opafm.xml file on each Management Node asneeded.

Refer to the Intel® Omni-Path Fabric Suite FabricManager User Guide for more information on how toconfigure the Fabric Manager.

6. Access the FastFabric OPA Host Setup menu. Press 3.

a) (Linux) Set Up Password-Less SSH/SCP Select 2 and press P to start the operation.

b) Password for root on all hosts: Type the password for root on all hosts and press Enter.

c) Upon completing the task, you are returned to theFastFabric OPA Host Setup menu.

d) Refresh SSH Known Hosts Select 9 and press P to start the operation.

e) Complete the operation. Press any key to end the operation.

End Task

Next Steps

• To configure and run the health check tool, go to Perform Initial Health Check.

Intel® Omni-Path Fabric—Set Up Additional Management Nodes


Configure Additional Management Nodes Using FastFabricCLI Commands

You can configure additional management nodes using the FastFabric CLI commands.

Assumption

• You are logged into the target Management Node.

Procedures

Repeat the following steps on each additional Management Node.


1. Copy the FastFabric configuration files from the initialManagement Node to each new management node.

At a minimum, the following files should be copied:

/etc/opa/opafastfabric.conf/etc/opa/ports/etc/opa/topology*.xml/etc/opa/hosts/etc/opa/allhosts/etc/opa/switches/etc/opa/chassis

2. Edit the /etc/opa/hosts configuration file. Edit the hosts file such that the file on each ManagementNode omits itself.

3. Edit the /etc/opa/allhosts configuration file. Edit the allhosts file such that the file on eachManagement Node specifies itself.

4. If Fabric Manager is to be run on the new managementnodes, copy the Fabric Manager configuration file (/etc/opa-fm/opafm.xml) from the initial Management Nodeto each new management node.

5. Edit the opafm.xml file on each Management Node asneeded.

Refer to the Intel® Omni-Path Fabric Suite FabricManager User Guide for more information on how toconfigure the Fabric Manager.

6. Set up secure password-less SSH, such that theManagement Node can securely log into all the hosts asroot through the management network, withoutrequiring a password.

Type opasetupssh -S -p -i "" -f hostfile.

7. Refresh the SSH known_hosts file on the ManagementNode to include the IPoIB hostnames of all the hosts.

Type opasetupssh -p -U -f hostfile.

End Task

Next Steps

• To configure and run the health check tool, go to Perform Initial Health Check.

10.3

Set Up Additional Management Nodes—Intel® Omni-Path Fabric


11.0 Perform Initial Health Check

The Health Check tool performs the initial fabric verification and creates a baseline ofthe hardware and software configuration. Once a good baseline has been established,you use the tools to compare the present fabric against the baseline and check itshealth. Baselines are rerun when changes occur such as fabric upgrades, hardwarereplacements or changes and software configuration changes.

Before You Begin

Before starting the health check configuration, perform the following:



Note that you can edit the files before you configure the health check.Configuration files are located under the /etc/opa directory. Sample files areinstalled into /usr/share/opa/samples with the suffix -sample.

The following files are used to configure the health check:


Review and update the following parameters as needed:

• FF_ANALYSIS_DIR

This parameter should be updated to reflect the type of SM (esm or hsm).

• FF_ALL_ANALYSIS

NOTE: If you are running in a back-to-back configuration, this parametershould not contain chassis.

• FF_FABRIC_HEALTH

• FF_CHASSIS_CMDS

• FF_CHASSIS_HEALTH

• FF_ESM_CMDS

— esm_chassis: Lists the chassis names, using the assigned TCP/IP Ethernetmanagement port names or IP addresses, that are running SMs if usingEmbedded SM(s) in the Intel® Omni-Path Fabric Chassis.

Intel recommends you use chassis names (the TCP/IP Ethernet managementport names assigned). Enter one chassis name or IP address per line.

For example:

Chassis1Chassis2

11.1

Intel® Omni-Path Fabric—Perform Initial Health Check




Configure and Initialize Health Check Tools UsingFastFabric TUI Menu

The health check tools may be run on one or more Management Nodes within thecluster. You set up and use the Intel® Omni-Path Fabric Suite FastFabric health checktools using the FastFabric TUI Menu.

For more information about health check tools, see the detailed discussion in theIntel® Omni-Path Fabric Suite FastFabric User Guide.

Assumption

• You are logged into the target management node.

Procedures

Repeat the following steps on each Management Node that will use the health checktools.


1. Create the /etc/opa/esm_chassis file. Create the file with a list of the chassis names using theassigned TCP/IP Ethernet management port names.

2. Access the FastFabric OPA Host Verification/AdminMenu.



b) Access the OPA Host Verification/Admin Menu. Press 4.


Select item 0 and press P.

a) Edit opafastfabric.conf file. Review the following parameters:• FF_ANALYSIS_DIR

This parameter should be updated to reflect the typeof SM (esm or hsm).

• FF_ALL_ANALYSISNOTE: If you are running in a back-to-backconfiguration, this parameter should not containchassis.

• FF_FABRIC_HEALTH• FF_CHASSIS_CMDS• FF_CHASSIS_HEALTH• FF_ESM_CMDS

3. Check Overall Fabric Health (menu item 9) Select item 9 and press P.

a) Performing Host Admin: Check OverallFabric Health Baseline presentconfiguration? [n]:

Press Enter.

4. Check the results.

a) If no errors were encountered... Continue to next step.

continued...

11.2

Perform Initial Health Check—Intel® Omni-Path Fabric



b) If any errors are encountered... Perform the following:1. Resolve the errors.2. Rerun 3. and 4. until a clean run occurs.

5. Create a cluster configuration baseline.

a) Check Overall Fabric Health (menu item 9) Select item 9 and press P.

b) Performing Host Admin: Check OverallFabric Health Baseline presentconfiguration? [n]:

Press y and press Enter.

6. If required, schedule regular runs of opaallanalysisthrough cron or other mechanisms.

Refer to the Linux* OS documentation for moreinformation on cron.Also refer to the Intel® Omni-Path Fabric Suite FastFabricUser Guide for more information about opaallanalysisand its automated use.

End Task

Next Steps

• Before you run top500 HPL (High Performance Linpack) runs or customeracceptance tests, Intel recommends that you follow all steps outlined in the Intel®Omni-Path Fabric Setup Guide.

• To run top500 HPL2, go to Run a Sample High Performance Linpack 2Configuration.

Configure and Initialize Health Check Tools UsingFastFabric CLI Commands

The health check tools may be run on one or more Management Nodes within thecluster. You set up and use the Intel® Omni-Path Fabric Suite FastFabric health checktools using CLI commands.

For more information, see the detailed discussion in the Intel® Omni-Path Fabric SuiteFastFabric User Guide.

Assumption

• You are logged into the target management node.

Procedures

Repeat the following steps on each Management Node that will use the health checktools.


1. Edit /etc/opa/opafastfabric.conf file. Review the following parameters:• FF_ANALYSIS_DIR

This parameter should be updated to reflect the typeof SM (esm or hsm).

• FF_ALL_ANALYSISNOTE: If you are running in a back-to-backconfiguration, this parameter should not containchassis.

• FF_FABRIC_HEALTH

continued...

11.3

Intel® Omni-Path Fabric—Perform Initial Health Check



• FF_CHASSIS_CMDS• FF_CHASSIS_HEALTH• FF_ESM_CMDS

2. Create the /etc/opa/esm_chassis file. Create the file with a list of the chassis names using theassigned TCP/IP Ethernet management port names.

3. Perform a health check. Type opaallanalysis -e.

4. Check the results.

a) If no errors were encountered... Continue to next step.

b) If any errors are encountered... Perform the following:1. Resolve the errors.2. Rerun 3. and 4. until a clean run occurs.

5. Create a cluster configuration baseline.Note: This may also be done using the FastFabric TUI

menu by selecting Check Overall Fabric Healthand answering y to the question: Baselinepresent configuration? [n]:

Type opaallanalysis -b.

6. If required, schedule regular runs of opaallanalysisthrough cron or other mechanisms.

Refer to the Linux* OS documentation for moreinformation on cron.Also refer to the Intel® Omni-Path Fabric Suite FastFabricUser Guide for more information about opaallanalysisand its automated use.

End Task

Next Steps

• Before you run top500 HPL (High Performance Linpack) runs or customeracceptance tests, Intel recommends that you follow all steps outlined in the Intel®Omni-Path Fabric Setup Guide.

• To run top500 HPL2, go to Run a Sample High Performance Linpack 2Configuration.

Perform Initial Health Check—Intel® Omni-Path Fabric


12.0 Perform High Performance Linpack Benchmark

This section describes how to gather MPI benchmark data. A set of common MPIbenchmarks are installed as part of the installation process. A popular measure ofoverall performance is High Performance Linpack 2 (HPL2), which is used to ratesystems on the Top 500 list.

You run the initial HPL2 benchmark to gather preliminary baseline numbers. Thesample configurations provided in the /usr/src/opa/mpi_apps/hpl-configfolder should perform within 10 – 20% of optimal HPL2 results for the cluster. Forfurther performance tuning, refer to the Intel® Omni-Path Fabric Performance TuningUser Guide.

The basic HPL2 process is:

1. Perform the initial run.

This should be a very small problem size to determine if the run is successful.Performance of this run is expected to be low.

If the initial run is successful, you are ready to move onto full scale HPL2 runs.

2. Perform full scale runs based on your cluster size using sample orcustomized configurations.

Assorted sample HPL.dat files are provided in the /usr/src/opa/mpi_apps/hpl-config folder. These files are a good starting point for most clusters andshould get within 10 – 20% of the optimal performance for the cluster. Theproblem sizes used assume a cluster with 1GB of physical memory per processor.For example, for a two-processor node, 2 GB of node memory is assumed. Foreach cluster size, four files are provided:

• t – A very small test run (5000 problem size)

• s – A small problem size on the low end of optimal problem sizes

• m – A medium problem size

• l – A large problem size

The sample files can be selected using config_hpl2.

Customized files can be generated using hpl_dat_gen.

3. Review the results.

Use the top command on a node to monitor memory and CPU usage.

The xhpl should use 98 – 99% of the CPU. If any other processes are taking morethan 1 – 2%, review the host configuration and stop these extra processes ifpossible. HPL is very sensitive to swapping. If a lot of swapping is seen, and xhplis dropping below 97% for long durations, this may indicate a problem size that istoo large for the memory and OS configuration.

You can continue to tune HPL2 to refine performance. Parameters in HPL.dat canall affect HPL performance. In addition, the selection of compiler and BLAS Mathlibrary may also significantly affect performance. The new HPL.dat files may be

Intel® Omni-Path Fabric—Perform High Performance Linpack Benchmark


placed in /usr/src/opa/mpi_apps/hpl-config. Use config_hpl to selectthem and copy them to all nodes in the run. Alternately, scpall may be used tocopy the file to all nodes. Refer to Intel® Omni-Path Fabric Suite FastFabric UserGuide for more information on scpall.

Before You Begin

Before starting the High Performance Linpack benchmark, perform the following:

• Ensure that you have followed all steps outlined in the Intel® Omni-Path FabricSetup Guide.

• Create the file /usr/src/opa/mpi_apps/mpi_hosts listing the host names ofall the hosts.

Note: Use mpi_hosts.sample as a starting point for creating the mpi_hostsfile.

• Determine the HPL2 runs in advance, as needed.

— Select an initial, small configuration to verify HPL has been properly compiled.

— Select one or more full scale configurations based on the cluster size andproblem size.

Notes: You will need to the following commands to perform the funs:

— Select configuration command usage:

./config_hpl2 config_name[problem_size]— Run command usage:

run_hpl number_processesFor example, ./config_hpl2 256s selects the configuration and,subsequently, ./run_hpl2 256 runs the configuration.

Run a Sample High Performance Linpack 2 Configuration

You can run sample HPL2 configurations to gather baseline numbers based on pre-configured HPL.dat files found in the /usr/src/opa/mpi_apps/hpl-configfolder.

Assumptions

• You are logged into the host.

• You have determined the configurations you will run.

Procedure

Note: It is best to start with a small configuration to verify HPL has been properly compiled,then move on to full scale HPL2 runs.

(Host) To run a sample HPL2 configuration, perform the following:

12.1

12.2

Perform High Performance Linpack Benchmark —Intel® Omni-Path Fabric



1. Change directories to the mpi_apps folder. Type cd /usr/src/opa/mpi_apps

2. If you do not know which configuration file you will use,view the sample configuration files.

Type ./config_hpl2.

3. Select an initial run with a very small problem size todetermine if the run is successful.

Type ./config_hpl2 <number ofprocesses><problem size>.For example, ./config_hpl2 32s

4. Run the configuration. Type ./run_hpl2 <number of processes><problemsize>For example, ./run_hpl2 32

5. Review the results.NOTE: Performance of this run is expected to be low.

• If the run is successful... Continue to the next step.

• If it is not successful... Troubleshoot.

6. View the sample configuration files to find a configurationthat simulates your cluster.

Type ./config_hpl2.

7. Select a full scale run to that simulates your cluster. Type ./config_hpl2 <number ofprocesses><problem size>.For example, ./config_hpl2 9216m

8. Run the configuration. Type ./run_hpl2 <number of processes><problemsize>For example, ./run_hpl2 9216


End Task

Next Steps

• To install and start up the Fabric Manager GUI software, refer to Install Intel®Omni-Path Fabric Suite Fabric Manager GUI.

Generate and Run a Custom High Performance LinpackConfiguration

To generate a custom HPL2 configuration, you use hpl_dat_gen to probe the currentnode for information on the number of cores and the amount of memory and promptsyou for the desired characteristics of the run.

Assumptions

• You are logged into the host.

• You have determined the configurations you will run.

Procedures

(Host) To generate and run a custom HPL2 configuration, perform the following:

12.3

Intel® Omni-Path Fabric—Perform High Performance Linpack Benchmark



1. Change directories to the mpi_apps folder. Type cd /usr/src/opa/mpi_apps

2. Start the dat generator tool. Type ./hpl_dat_gen.

3. Provide the following information:

a) # of compute nodes [1]? Type a number and press Enter.

b) # of cores per node [72]? Type a number and press Enter.

c) # of RAM per node (in MB) [64156]? Type a number and press Enter.

d) Memory pressure (range between 0.1 and0.9) [0.3]?

Type a decimal number and press Enter.

e) From the results, note the number of processes. Thiswill be added to the run command: ./run_hpl2<number of processes>.

HPLinpack benchmark input file - 9216 processes, 0.5 memory sizeGenerated by hpl_dat_gen.sh.HPL.out output file name (if any)6 device out (6=stdout,7=stderr,file)1 # of problems sizes (N)733488 Ns1 # of NBs168 NBs0 PMAP process mapping (0=Row-,1=Column-major)1 # of process grids (P x Q)72 Ps128 Qs16.0 threshold1 # of panel fact1 PFACTs (0=left, 1=Crout, 2=Right)1 # of recursive stopping criterium4 NBMINs (>= 1)1 # of panels in recursion2 NDIVs1 # of recursive panel fact.2 RFACTs (0=left, 1=Crout, 2=Right)1 # of broadcast1 BCASTs (0=1rg,1=1rM,2=2rg,3=2rM,4=Lng,5=LnM)1 # of lookahead depth1 DEPTHs (>=0)2 SWAP (0=bin-exch,1=long,2=mix)168 swapping threshold0 L1 in (0=transposed,1=no-transposed) form0 U in (0=transposed,1=no-transposed) form1 Equilibration (0=no,1=yes)8 memory alignment in double (> 0)

Continue to the next step.

3. Run the configuration. Type ./run_hpl2 <number of processes>.For example: ./run_hpl2 9216


End Task

Next Steps

• To install and start up the Fabric Manager GUI software, refer to Install Intel®Omni-Path Fabric Suite Fabric Manager GUI.

Perform High Performance Linpack Benchmark —Intel® Omni-Path Fabric


13.0 Install Intel® Omni-Path Fabric Suite FabricManager GUI

Intel® Omni-Path Fabric Suite Fabric Manager GUI can be run on a Linux* orWindows* desktop/laptop system with TCP/IP connectivity to the Intel® Omni-PathFabric Suite Fabric Manager. Network connectivity to the Fabric Executive (FE)component of the Fabric Manager allows continuous, remote, “out of band”monitoring. If configured with alternate FE connection information, the Fabric ManagerGUI automatically fails over to an alternative FE if the connection to the original islost.

This section provides information and procedures to install and set up the FabricManager GUI on your desktop or laptop.

Before You Begin

The following sub-sections provide information and instructions that are necessarybefore you start the Intel® Omni-Path Fabric Suite Fabric Manager GUI installation.

System Requirements for a Windows* Environment

The minimum system requirements are as follows:

• Windows* operating system

— Windows Vista* Service Pack 2 or higher

— Windows* 7 Service Pack 1 or higher

— Windows* 8.x

— Windows* 10.x

— Windows Server* 2008 Service Pack 2 or higher

— Windows Server* 2012

• x86 or x64 processor architecture

• Oracle* Java Runtime Environment (JRE) 1.7 or higher

• Ethernet card/local network access

• Minimum disk space requirement is 100MB. This amount includes the disk spaceneeded for the application binaries and the application cache for topologyinformation.

• 2GB or greater of RAM

• 1280x800 resolution (65K color depth)

System Requirements for a Linux* Environment

The minimum system requirements for the Linux* installation are as follows:

• Linux* operating system

13.1

13.1.1

13.1.2

Intel® Omni-Path Fabric—Install Intel® Omni-Path Fabric Suite Fabric Manager GUI


— Red Hat* Enterprise Linux* 5.5 or higher

— SUSE* Linux* Enterprise Server 10 or higher

• x86 or x64 processor architecture

• Oracle* Java* Runtime Environment (JRE) 1.7 or higher

• X Window System

• Ethernet card/local network access

• Minimum disk space requirement is 100MB. This amount includes the disk spaceneeded for the application binaries and the application cache for topologyinformation.

• 2GB or greater of RAM

• 1280x800 resolution (65K color depth)

Download the Fabric Manager Software


2. In the "Search downloads" field, type Fabric Manager GUI and press Enter.

3. From the search drop-down, select the Intel® Omni-Path Fabric Suite FabricManager GUI software.

4. In the Available Downloads list, select the file(s) you need for the OS you haveinstalled on your fabric.

Note: You can download either the executable file to install or the source files tobuild your Intel OPA Fabric Manager GUI jar file. This document onlyaddresses installing through the executable.


6. Click "I accept the terms in the license agreement.


Install the Fabric Manager GUI Software on Windows*

You install the Fabric Manager GUI software on your Windows* desktop using theinstallation wizard.

Assumption

• You have downloaded the installation executable for Windows* to your desktop.

Procedure

To install the software, perform the following:


1. Start the wizard. Using File Explorer, navigate to the target folder anddouble-click the IntelOPA-FMGUI.windows-x.x.x.x.x.exe file on the desktop

continued...

13.1.3

13.2

Install Intel® Omni-Path Fabric Suite Fabric Manager GUI—Intel® Omni-Path Fabric



where x.x.x.x.x is the version number of the FabricManager GUI application being installed.

a) At the prompt: Do you want to allow thefollowing program to make changes to thiscomputer?

Click Yes.

b) Installer Language... Select the language to be used for the installation screensand then click OK.

c) Fabric Manager GUI Setup dialog...

• If this is a first-time installation... Click Next.

• If Fabric Manager GUI is already installed on yoursystem and you are re-installing...

Click Yes to overwrite the existing software and then clickNext.

d) Afer reviewing the Licence Agreement... Select the checkbox "I accept the terms of the LicenseAgreement" and click Next.

e) Choose Install Location... Intel recommends that you use the default file location.Click Install.

f) Completing Fabric Manager GUI Setup... Click Finish.

2. (Optional) Change the database location.

a) Change directory. Navigate to <app_data_path>\Intel\FabricManagerGUIwhere <app_data_path> is a user's application datafolder. Under Windows 7/8, it's C:\Users\<user_name>\AppData\Roaming

b) Create file settings.xml. 1. Right-click in the folder and select New ➤ TextDocument.

2. Name the file settings.xml.3. Click Yes to change the extension.

c) Open the file in a text editor and add the contents...<?xml version="1.0" encoding="UTF-8" standalone="no"?><!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd"><properties> <comment>FM GUI Application configuration settings</comment> <entry key="db.connection.url">jdbc:hsqldb:file:c:\temp\db\mydbname; hsqldb.result_max_memory_rows=1000;hsqldb.default_table_type=cached; hsqldb.log_size=50;</entry></properties>

Change c:\temp\db\mydbname to the location for thedatabase file.NOTE: Only change the database file path. Omitting theother options will cause database performance issues inHSQLDB.

Save and close the file.

End Task

Next Steps

• To prepare the Fabric Manager for Fabric Manager GUI, refer to Prepare FabricManager for Fabric Manager GUI.



Install the Fabric Manager GUI Software on Linux*

You install the Fabric Manager GUI software on your Linux* desktop using the rpmfile.

Assumption

• You have downloaded the installation rpm for Linux* to your desktop.

Procedure



1. Install the rpm.

a) Log into the system where the Fabric Manager GUI willbe installed.

Or become root by typing su and entering the super-userpassword.

b) Open a Terminal window in X Windows.

c) Change directories to the directory where youdownloaded the rpm file.

Type cd directory_path_name.

d) Install the package. Type rpm -iv IntelOPA-FMGUI-linux-x.x.x.x.x.noarch.rpmwhere x.x.x.x.x is the version number of the FabricManager GUI application being installed.

e) Wait for the successful completion of the command. Continue to next step.

2. (Optional) Change the database location.

a) Change directory. Type cd ~\.Intel\FabricManagerGUI.

b) Create file settings.xml. Type touch settings.xml

c) Using a text editor, add the following contents...<?xml version="1.0" encoding="UTF-8" standalone="no"?><!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd"><properties> <comment>FM GUI Application configuration settings</comment> <entry key="db.connection.url">jdbc:hsqldb:file:/var/temp/db/mydbname; hsqldb.result_max_memory_rows=1000;hsqldb.default_table_type=cached; hsqldb.log_size=50;</entry></properties>

d) Change /var/temp/db/mydbname to the location forthe database file.NOTE: Only change the database file path. Omitting theother options will cause database performance issues inHSQLDB.

Save and close the file.

End Task

Next Steps

• To prepare the Fabric Manager for Fabric Manager GUI, refer to Prepare FabricManager for Fabric Manager GUI.

13.3



Prepare Fabric Manager for Fabric Manager GUI

Fabric Manager GUI requires a running fabric with FE available. This task show youhow to prepare the Fabric Manager to work with the Fabric Manager GUI.

Assumptions

• You have completed the installation of the Fabric Manager GUI on your system.

• Before running Fabric Manager GUI, ensure that Fabric Manager has the properconfiguration and is running normally.

Procedure



1. On the management node, edit the /etc/opa-fm/opafm.xml file.

NOTE: The following steps contain the necessary edits toopafm.xml.

2. Change the <SubnetPrefix> for fm0 to0xfe80000000001000 <Shared>       <Start>1</Start>  

  <Name>fm0</Name>  <Hfi>1</Hfi>  <Port>1</Port>  <PortGUID>0x0000000000000000</PortGUID>  <SubnetPrefix>0xfe80000000001000</SubnetPrefix> 

To set up multiple rails in a single or multi-subnet, referto Multi-Rail Usage on page 110 for additionalinstructions.

3. Ensure that the FE is enabled. In the <FE> section, "Start" is 1.

<Fe> <Start>1</Start

4. Ensure that FE has proper security setup.

• If you want to run FE without SSL, ensure that theSslSecurityEnable is 0.Also, when you run Fabric Manager GUI, set up FEhost as unsecured (clear the "Secure" checkbox).

 <SslSecurityEnable>0</SslSecurityEnable>

continued...

13.4




• If you want to run FE with SSL, ensure that you setSslSecurityEnable to 1.To set up keys and opafm.xml properly, refer toadditional steps in sections and appendices found inthe Intel® Omni-Path Fabric Suite Fabric Manager UserGuide:— "Out-of-Band Security"— "Core-Level Public Key Infrastructure (PKI) Best

Practices Guidelines"— "Advanced-Level Public Key Infrastructure Best

Practices Guidelines"— "SSL Key Creation for Fabric Manager GUI"And, when you run Fabric Manager GUI, set up FEhost as "Secure" with the key files (select the "Secure"checkbox).

 <SslSecurityEnable>1</SslSecurityEnable>

5. Ensure that PM is enabled. In the <Pm> section, "Start" is 1.

<Pm>  <Start>1</Start> 

6. If you made any changes... Save and close the file.

7. Restart the Fabric Manager so that your changes takeeffect.

Type systemctl restart opafm.

End Task

Next Steps

• To start the Fabric Manager GUI on Windows*, go to Start the Fabric Manager GUIon Windows*.

• To start the Fabric Manager GUI on Linux*, go to Start the Fabric Manager GUI onLinux*.

Start the Fabric Manager GUI on Windows*

You start the Intel® Omni-Path Fabric Suite Fabric Manager GUI on Windows* throughthe Start menu.

Note: Intel recommends that you accept the default startup options (settings). Refer to theIntel® Omni-Path Fabric Suite Fabric Manager GUI Online Help for procedures to setuser preferences.

Using Windows Vista and Windows 7

To start the application:

1. From the Start menu, select All Programs.

2. Select Intel ➤ Omni-Path ➤ Fabric Manager GUI

The Fabric Manager GUI application starts.

13.5



Notes: It is recommended that you create a shortcut to put an icon on yourdesktop.

• For Windows Vista, right-click the application name in step 2 and selectSend To ➤ Desktop (Create Shortcut).

• For Windows 7, right-click the application name in step 2 and select"Pin to Start Menu".

Using Windows 8.x

To start the application:

1. From the Start menu, select the Search (magnifying glass) icon.

2. In the search box, begin typing Fabric Manager GUI.

The application will appear for selection.

3. Select the application.

The Fabric Manager GUI application starts.

Note: It is recommended that you pin the application to your Start menu. Right-click the application name in step 3 and select "Pin to Start".

Start the Fabric Manager GUI on Linux*

You can start the Fabric Manager GUI on Linux* from either a terminal window on XWindows or the K Desktop Environment (KDE).

Note: Intel recommends that you accept the default startup options (settings). Refer to theIntel® Omni-Path Fabric Suite Fabric Manager GUI Online Help for procedures to setuser preferences.

Using X Windows

To start the application from a terminal window on X Windows:

1. Open a terminal window in X Windows.

2. Type fmgui and press Enter.

The Fabric Manager GUI starts.

Using K Desktop Environment (KDE)

To start the application from the K Desktop Environment (KDE):

1. From the K Menu, select Applications.

2. Select the Fabric folder.

3. Select Fabric Manager GUI.

The Fabric Manager GUI starts.

Uninstall Fabric Manager GUI

This section provides instructions and information for uninstalling the Fabric ManagerGUI from your desktop. These instructions are provided for reference only.

13.6

13.7



Note: The Fabric Manager GUI must be closed for the uninstall to be successful. Theuninstall program does not warn you if the application is open. Warnings may bereceived at the end of the uninstall process stating that certain files have not beenremoved or it might just fail.

Uninstall Fabric Manager GUI Software on Windows*

Using Windows* Vista and Windows* 7

To uninstall the software, perform the following:

1. From the Start menu, select Control Panel.

2. Under the Programs category, select Uninstall a program.

3. In the program list, select Intel Fabric Manager GUI.

4. Click the Uninstall button at the top of the list.

5. Follow the instructions on the uninstall window.

The Fabric Manager GUI is uninstalled.

Using Windows* 8.x


1. Right-click the Start menu and select Programs and Features.

2. In the program list, select Intel Fabric Manager GUI.

3. Click the Uninstall button at the top of the list.

4. Follow the instructions on the uninstall window.


Clear the Fabric Manager GUI Cache on Windows*

The Fabric Manager GUI uses local disk storage to keep a cache of fabric information.The size of this cache depends on the size of the fabric(s) configured in the FabricManager GUI and the refresh rate used for performance monitoring. You can reclaimthis disk storage by using the Clear FM GUI Cache. Be aware that clearing the cachealso clears subnet configurations. Also, this option is only available to those users whohave used the Fabric Manager GUI.

Using Windows* Vista and Windows* 7

To clear the cache, perform the following:

1. Select Start menu.

2. Select All Programs.

3. Select Intel.

4. Select Omni-Path.

5. Select Clear FM GUI Cache.

A command window is displayed.

6. Type y and press Enter.

7. Press any key to complete the process.

13.7.1

13.7.1.1



Using Windows* 8.x


1. Select the Start button (window icon).

2. Click the Search (magnifying glass) icon.

3. Begin typing Clear FM GUI Cache.

The application will display in the results pane.


A command window is displayed.

5. Type y and press Enter.

6. Press any key to complete the process.

Uninstall Fabric Manager GUI Software on Linux*


1. Log in to the server where Fabric Manager GUI is installed as a user with rootaccess.

2. Open a Terminal window in X Windows.

3. Type # rpm -e IntelOPA-FMGUI-linux-x.x.x.x.x.noarch.rpmwhere x.x.x.x.x is the version number of the Fabric Manager GUI applicationbeing uninstalled.

4. Wait for the successful completion of the command.


Clear the Fabric Manager GUI Cache on Linux*

The Fabric Manager GUI uses local disk storage to keep a cache of fabric information.The size of this cache depends on the size of the fabric(s) configured in the FabricManager GUI and the refresh rate used for performance monitoring. You can reclaimthis disk storage by using the Clear FM GUI Cache. Be aware that clearing the cachealso clears subnet configurations. Also, this option is only available to those users whohave used the Fabric Manager GUI.

Using X Windows


1. Open a Terminal window in X Windows.

2. Enter the following commands:

cd ~/.Intel/FabricManagerGUIfmguiclear.sh

The Fabric Manager GUI cache is cleared.

Using K Desktop Environment (KDE)


1. In KDE or GNOME, select Applications.

13.7.2

13.7.2.1



2. Select Fabric; if not available, try the System category.


The Fabric Manager GUI cache is cleared.



14.0 Additional Installation and Setup Tasks

This chapter describes additional installation and setup tasks that, depending on yourconfiguration, you may need to perform during or after software installation.

Multi-Subnet Fabrics

Intel® Omni-Path Fabric Suite FastFabric supports management of both single-subnetfabric and multi-subnet fabrics. This section describes differences in the installationprocedure that are relevant for multi-subnet fabrics.

When operating a multi-subnet fabric, a subnet manager (SM) is required for eachsubnet. An SM may be run within switches within each subnet, or a host-based SMmay be run. A host-based SM can manage multiple subnets (assuming the host serveris connected to more than one subnet).

A number of combinations are possible for multi-subnet fabrics:

• All subnets are completely independent (except for any interconnectingrouters).

If a separate FastFabric node is being used per subnet, and servers are notinstalled in more than one subnet, the individual subnets can be treatedseparately. This single-subnet scenario is created using the default FastFabricinstructions previously described in this document.

• The subnets are primarily independent.

If the only components common to more than one subnet are the FastFabricnodes (and possibly SM nodes) and no servers are installed in more than onesubnet, refer to the following instructions for Setting Up Primarily IndependentSubnets.

• The subnets are overlapping.

If multiple components are common to more than one subnet, such as FastFabricnodes, servers, and others, refer to the following instructions for Setting UpOverlapping Subnets.

Setting Up Primarily Independent Subnets

This section provides additional instructions performed for specific steps during theinstallation and configuration process. Use the table below as both instructions and achecklist for setting up your primarily independent subnets.

Procedures

Perform the installation and configuration as described in this guide. Append thefollowing actions during the process.

14.1

14.1.1

Intel® Omni-Path Fabric—Additional Installation and Setup Tasks



1. Installation Prerequisites

a) Fabric Design Prerequisites Design the cabling such that the FastFabric node isconnected to each subnet it manages. The FastFabricnode must also have a management network path to allthe nodes in all the subnets that it manages. As part ofthe design, consider where routes between subnets arewanted between storage routers, IPoIB routers, and otherdevices.

b) Fabric Setup Prerequisites When installing the IFS software on the FabricManagement Node, IPoIB must be configured such thateach subnet is an independent IPoIB network interface,typically with different IP subnets.Refer to the Intel® Omni-Path Fabric Host Software UserGuide for more information on configuring IPoIB.

2. Install the Intel® Omni-Path Software As stated.

3. Configure the Software

a) Configure the Chassis • When creating the chassis file, list all Intelinternally-managed switches in all subnets. Ifrequired, additional files may also be created persubnet that list only the Intel chassis in each subnet.

• When editing the ports file, list all the FabricManagement Node ports that access the managedfabrics. If required, additional files may be created persubnet that list only the Fabric Management Node portconnected to the given managed fabric.

b) Configure the Externally-Managed Switches • When editing the ports file, list all the FabricManagement Node ports that access the managedfabrics. If required, additional files may be created persubnet that list only the Fabric Management Node portconnected to the given managed fabric.

• If required, the switches file may specify ahfi:port per switch. However, if hfi:port is notspecified, all the hfi:port specified in the ports fileare searched to locate the given switch’s Node GUID.

c) Install the Host Software on the Remaining Hosts Usingthe FastFabric TUI Menu

When creating the hosts file, list the hosts in all subnetsexcept the Fabric management node where FastFabric isbeing run. If required, additional files may be created persubnet that list the hosts in each subnet, except theFabric Management Node.

d) Verify the Host Software on the Remaining ServersUsing the FastFabric TUI Menu

• Edit Configuration and Select/Edit Host FileCreate the allhosts file as per the instructions.Next, create additional files per subnet that list all thehosts in each subnet including the Fabric ManagementNode. When editing the ports file, list all the FabricManagement Node ports that access the managedfabrics. If required, additional files may be created persubnet that list only the Fabric Management Node portconnected to the given managed fabric.

• Verify Hosts See Each OtherThis can be run for each subnet by using theallhosts files specific to each subnet (i.e., thoselisting only hosts in a single subnet).

• Check MPI PerformanceThis can be run for each subnet by using theallhosts files specific to each subnet (i.e., thoselisting only the hosts in a single subnet).

continued...

Additional Installation and Setup Tasks—Intel® Omni-Path Fabric



e) Configure Additional Management Nodes UsingFastFabric TUI Menus

When copying FastFabric configuration files to theadditional Fabric Management Nodes, be sure to also copythe additional hosts, chassis, and allhosts files thatwere created per subnet.Note: In asymmetrical configurations where the Fabric

Management Nodes are not all connected to thesame set of subnets, the files copied to eachmanagement node may need to be slightlydifferent. For example, configuration files foropafabricanalysis may indicate different portnumbers or host files used for FastFabric, and MPImay need to list different hosts.

f) Configure and Initialize Health Check Tools UsingFastFabric CLI Commands

Make sure the /etc/opa/ports file lists each of theFabric Management Node local HFIs and ports that areconnected to a unique subnet.When running opareports, opafabricinfo,opafabricanalysis, or opaallanalysis, the defaultis to use the ports file. If required, the -p and -toptions or the PORTS/PORTS_FILE environmentvariables may be used to specify all the HFIs and ports onthe Fabric Management Node such that all subnets arechecked. Similarly, the esm_chasssis and chassisfiles used must list all relevant Intel chassis in all subnets.

g) Run a Sample High Performance Linpack 2Configuration

Run HPL2 for each subnet by creating mpi_hosts filesspecific to each subnet, that is, only listing hosts in asingle subnet.

h) Install Intel® Omni-Path Fabric Suite Fabric ManagerGUI

At least one subnet manager is required per subnet.Refer to the Intel® Omni-Path Fabric Suite FabricManager User Guide for more information on how toconfigure a host SM node to manage more than onesubnet.

End Task

Setting Up Overlapping Subnets

This section provides additional instructions performed for specific steps during theinstallation and configuration process. Use the table below as both instructions and achecklist for setting up your overlapping subnets.

Procedures

Perform the installation and configuration as described in this guide. Append thefollowing actions during the process.


1. Installation Prerequisites

a) Fabric Design Prerequisites Design the cabling such that the FastFabric node isconnected to each subnet it manages. The FastFabricnode must also have a management network path to allthe nodes in all the subnets that it manages. As part ofthe design, consider where routes between subnets arerequired, between routers, IPoIB routers, and otherdevices.

continued...

14.1.2




b) Fabric Setup Prerequisites When installing the IFS software on the FabricManagement Node, IPoIB must be configured such thateach subnet is an independent IPoIB network interface,typically with different IP subnets.Refer to the Intel® Omni-Path Fabric Host Software UserGuide for more information on configuring IPoIB.

2. Install the Intel® Omni-Path Software As stated.

3. Configure the Software

a) Configure the Chassis • When creating the chassis file, list all Intelinternally-managed switches in all subnets. Ifrequired, additional files may be created per subnetthat list only the Intel chassis in each subnet.

• When editing the ports file, list all the FabricManagement Node ports that access the managedfabrics. If required, additional files may be created persubnet that list only the Fabric Management Node portconnected to the given managed fabric.

b) Configure the Externally-Managed Switches • When editing the ports file, list all the FabricManagement Node ports that access the managedfabrics. If required, additional files may be created persubnet that list only the Fabric Management Node portconnected to the given managed fabric.

• If required, the switches file may specify ahfi:port per switch. However, if hfi:port is notspecified, all the hfi:port entries specified in theports file are searched to locate the given switch’snode GUID.

c) Install the Host Software on the Remaining Hosts Usingthe FastFabric TUI Menu

• When creating the hosts file, list the hosts in allsubnets except the Fabric Management Node whereFastFabric is being run. If required, additional filesmay also be created per subnet that list the hosts ineach subnet, except the Fabric Management Node.

• For hosts that are connected to more than one subnet,IPoIB must be configured such that each subnet is anindependent IPoIB network interface, typically withdifferent IP subnets.

• Refer to the Intel® Omni-Path Fabric Host SoftwareUser Guide for more information on configuring IPoIB.

d) Verify the Host Software on the Remaining ServersUsing the FastFabric TUI Menu

• Edit Configuration and Select/Edit Host FileCreate the allhosts file as per the instructions.Next, create additional files per subnet that list all thehosts in each subnet including the Fabric ManagementNode. When editing the ports file, list all the FabricManagement Node ports that access the managedfabrics. If required, additional files may be created persubnet that list only the Fabric Management Node portconnected to the given managed fabric.

• Verify Hosts See Each OtherThis can be run for each subnet by using theallhosts files specific to each subnet (i.e., thoselisting only hosts in a single subnet).

• Check MPI PerformanceThis can be run for each subnet by using theallhosts files specific to each subnet (i.e., thoselisting only the hosts in a single subnet). This iscurrently not available for OFED.

continued...




e) Configure Additional Management Nodes UsingFastFabric TUI Menus

When copying FastFabric configuration files to theadditional Fabric Management Nodes, be sure to also copythe additional hosts, chassis, and allhosts files thatwere created per subnet.Note: In asymmetrical configurations where the Fabric

Management Nodes are not all connected to thesame set of subnets, the files copied to eachmanagement node may need to be slightlydifferent. For example, configuration files foropafabricanalysis may indicate different portnumbers or host files used for FastFabric, and MPImay need to list different hosts.

f) Configure and Initialize Health Check Tools UsingFastFabric CLI Commands

Make sure the /etc/opa/ports file lists each of theFabric Management Node local HFIs and ports that areconnected to a unique subnet.When running opareports, opafabricinfo,opafabricanalysis, or opaallanalysis, the defaultis to use the ports file. If required, the -p and -toptions or the PORTS/PORTS_FILE environmentvariables may be used to specify all the HFIs and ports onthe Fabric Management Node such that all subnets arechecked. Similarly, the esm_chasssis and chassisfiles used must list all relevant Intel chassis in all subnets.

g) Run a Sample High Performance Linpack 2Configuration

Run HPL2 for each subnet by creating mpi_hosts filesspecific to each subnet, that is, only listing hosts in asingle subnet.

h) Install Intel® Omni-Path Fabric Suite Fabric ManagerGUI

At least one subnet manager is required per subnet.Refer to the Intel® Omni-Path Fabric Suite FabricManager User Guide for more information on how toconfigure a host SM node to manage more than onesubnet.

End Task

Multi-Rail Usage

This section provides an overview and instructions for configuring the common multi-rail scenarios.

Multi-Rail Overview

A multi-rail configuration provides load balancing and failover capabilities, adding ahigher degree of fabric redundancy. If one HFI or an entire subnet fails, traffic can bemoved to the remaining switches.

The multi-rail feature can be applied to a single subnet or multiple subnets. Byenabling multi-rail, a process can use multiple network interface cards (HFIs) totransfer messages.

Note: Subnets can also be referred to as planes or fabrics. Rails are also referred to as HFIs.

Three basic scenarios include:

• Single-rail in a single subnet: This scenario, shown in the following figure, consistsof one HFI in a server connected to one subnet. This is the default configurationduring installation. This configuration provides the performance required by mostapplications in use today.

14.2

14.2.1



• Dual-rail in dual subnets: This scenario, shown in the following figure, consists oftwo HFIs in the same server connected to separate subnets. Depending on theplatform, this configuration may provide improved MPI message rate, latency, andbandwidth to the node as well as flexibility for configuring failover and load-balancing.

• Dual-rail in a single subnet: This scenario, shown in the following figure, consistsof two HFIs in the same server connected to the same subnet. This configurationalso provides improved MPI message rate, latency, and bandwidth to the node,but only basic HFI failover capabilities if configured.



Note: Other multi-rail scenarios can be configured. A single Host FM server can managemultiple subnets up to the supported number of FM instances.

Setting Up Dual Rails for a Single Subnet

Support for single rail or dual rails in a single subnet is the default scenario expectedduring installation. The driver will detect that there are one or two HFIs connected tothe host on the same fabric. To initiate the dual-rail configuration, you need to changethe SubnetPrefix for fm0 as described in the steps below.

Assumptions

• HFIs have been installed in the host servers.

• HFIs have been cabled as shown below:

Host Subnet

HFI1_1

HFI1_0

• The single subnet has already been configured during software installation.

Procedures

Perform the following steps to set the SubnetPrefix:

1. On the Host, open the /etc/opa-fm/opafm.xml file for editing.

2. Search on "<Shared>" to review the fm0 settings.

14.2.2



An example of the shared instance for fm0 with key items in bold is shown below.Note that this is the default configuration for a single-rail in a single subnet.

<Shared>       <Start>1</Start>  


   </Shared>

3. Change the <SubnetPrefix> for fm0 to a unique value other than0xfe80000000000000. A recommended value to use would be0xfe80000000001000.

An example of the change is shown below.




4. Edit fm1 settings as shown below:

• Start = 1 (enables FM to start an instance on the second HFI)

• HFI = 2• Port = 1



5. Verify that the SubnetPrefix is the same as fm0.

An example of the change for fm1 is shown below.

<Shared> <Start>1</Start>  <Name>fm1</Name>  <Hfi>2</Hfi>  <Port>1</Port>  <PortGUID>0x0000000000000000</PortGUID>  <SubnetPrefix>0xfe80000000001000</SubnetPrefix>      </Shared>

6. Save the opafm.xml file.

7. Restart the Fabric Manager using systemctl restart opafm.

Host 1 starts up as the master Fabric Manager.

8. Copy the opafm.xml file to other hosts using dual rail with single subnet.

9. Restart the Fabric Manager on the other hosts using systemctl restartopafm.

Configuring Dual Rails for Dual Subnets

A common multi-rail, multi-subnet scenario is to set up dual rails for dual subnetswhere you configure two HFIs in each host to two different subnets to provide failoverand load-balancing functions.

Assumptions

• HFIs have been installed in the host servers.

• HFIs have been cabled as shown below:

Host 1(Master FM)

Subnet 1Host 2

(Standby FM)

HFI1_1

HFI1_0 HFI1_0

HFI1_1Subnet 2

• Two subnets have already been configured during software installation.

Procedures

Perform the following steps to set up the master Fabric Manager on Host 1 andstandby Fabric Manager on Host 2:

1. On Host 1, open the /etc/opa-fm/opafm.xml file for editing.

2. Search on "<Shared>" to review the fm0 settings.

14.2.3



An example of the shared instance for fm0 with key items in bold is shown below.




3. Change the <SubnetPrefix> for fm0 to a unique value other than0xfe80000000000000. A recommended value to use would be0xfe80000000001000.

An example of the change is shown below.




4. Find the next occurrence of <Shared>.

An example of the shared instance for fm1 with key items in bold is shown below.

<Shared> <Start>0</Start>  <Name>fm1</Name> 



<Hfi>1</Hfi>  <Port>2</Port>  <PortGUID>0x0000000000000000</PortGUID>  <SubnetPrefix>0xfe80000000001001</SubnetPrefix>      </Shared>

5. Edit fm1 settings as shown below:

• Start = 1 (enables FM to start an instance on the second HFI)

• HFI = 2• Port = 1

6. Verify that the SubnetPrefix is the second, unique subnet ID set up duringsoftware installation.

An example of the final settings for fm1 is shown below.

<Shared> <Start>1</Start>  <Name>fm1</Name>  <Hfi>2</Hfi>  <Port>1</Port>  <PortGUID>0x0000000000000000</PortGUID>  <SubnetPrefix>0xfe80000000001001</SubnetPrefix>      </Shared>

7. Save the opafm.xml file.

8. Restart the Fabric Manager on Host 1 using systemctl restart opafm.

Host 1 starts up as the master Fabric Manager.

9. Run service opafm status to verify that the two instances of the FM arerunning.

10. Copy the opafm.xml file to Host 2.

11. Restart the Fabric Manager on Host 2 using systemctl restart opafm.

Host 2 starts up in standby mode.



15.0 Installation Verification and Additional Settings

This section provides instructions for verifying that the software has been properlyinstalled and configured, the Intel® Omni-Path Fabric drivers are loaded, and that thefabric is active and ready to use. Information on HFIs and performance tuning is alsoprovided.

LED Link and Data Indicators

For information on the LEDs function as beaconing, link, and data indicators, refer toIntel® Omni-Path Fabric Switches Hardware Installation Guide and/or Intel® Omni-Path Host Fabric Interface Installation Guide.

Thermal Monitoring Module Verification

The following procedure provides steps to check the version of your ThermalMonitoring Module (TMM) and to update it to the latest version.

Note the new field for TMM firmware version, an optional micro-controller for thermalmonitoring on vendor-specific HFI adapters using the SMBus.

1. Navigate to the Intel® Omni-Path Host Fabric Interface (HFI) firmware updatesdirectory:

cd /lib/firmware/updates

2. Check the file TMM firmware version in the hfi1_smbus.fw file using:

opatmmtool -f hfi1_smbus.fw fileversion

3. Check the current TMM firmware version using: opatmmtool -fwversion.

4. Check the TMM firmware version in the HFI revision snapshot:

# opahfirev

Output Example, TMM version is shown in bold type:

# opahfirev######################node145 - HFI 0000:81:00.0HFI: hfi1_0Board: ChipABI 3.0, ChipRev 7.17, SW Compat 3SN: 0x0063be82Location:Discrete Socket:1 PCISlot:00 NUMANode:1 HFI0Bus: Speed 8GT/s, Width x16GUID: 0011:7501:0163:be82SiRev: B1 (11)TMM: 10.0.0.0.696######################

15.1

15.2

Installation Verification and Additional Settings—Intel® Omni-Path Fabric


5. If the fwversion in step 3 and 4 is less than the fileversion in step 2, updatethe TMM firmware version in the HFI:

opatmmtool -f hfi1_smbus.fw update

Adapter and Other Settings

The following settings can be adjusted for better performance.

• Ensure that an MTU of 8k bytes, is used with the Intel® Omni-Path HostFabric Interface.

8K MTU is enabled in the Intel® Omni-Path driver by default. To change thissetting for the driver, refer to the Intel® Omni-Path Fabric Performance TuningUser Guide.

• Use a PCIe Max Read Request size of at least 512 bytes with the Intel®Omni-Path Host Fabric Interface.

The Intel® Omni-Path Host Fabric Interface can support sizes from 128 bytes to4096 bytes in powers of two. This value is typically set in the BIOS.

• Use a PCIe MaxPayload size of 256, where available, with the Intel®Omni-Path Host Fabric Interface.

The Intel® Omni-Path Host Fabric Interface can support 128, 256, or 512 bytes.This value is typically set by the BIOS as the minimum value supported both bythe PCIe card and the PCIe root complex.

• Check the PCIe bus width.

• If slots have a smaller electrical width than mechanical width, a lower thanexpected performance may occur. Use the following command to check PCIe Buslink speed and width.

lspci -vv -d :24f0 | grep LnkSta:

ARP Neighbor Table Setup for Large Clusters

On large clusters or subnets, the ARP neighbor table may overflow and produce aneighbor table overflow message to /var/log/messages along with other effectssuch as ping failing. The Intel® Omni-Path Fabric Suite includes a script, enabled bydefault during installation, that automatically tunes the ARP Neighbor Table wheninvoked. The script, opa-arptbl-tuneup, is run once by opa.service when a nodestarts or restarts, but it can also be run manually.

To run opa-arptbl-tuneup manually, it must execute at the root. The syntax forthe parameter options is as follows:

• opa-arptbl-tuneup start - adjust kernel ARP table size

• opa-arptbl-tuneup stop - restore previous configuration

• opa-arptbl-tuneup status - check if original table size was changed

• opa-arptbl-tuneup restart - stop then start

• opa-arptbl-tuneup force-reload - stop then start

• opa-arptbl-tuneup --help - usage information for the script

15.3

15.4

Intel® Omni-Path Fabric—Installation Verification and Additional Settings


There are two ARP/Neighbor Tables in the kernel, one for IPv4 networks and one forIPv6 networks. The operating system uses the table for the particular network in use.There are three threshold parameters for each table: gc_thresh1, gc_thresh2, andgc_thresh3. You can check the present threshold level 1 by entering the command:

cat /proc/sys/net/ipv4/neigh/default/gc_thresh1

You may repeat for gc_thresh2 and gc_thresh3.

SM Loop Test

The SM loop test is a diagnostic test provided in the Fabric Manager. As part of thistest, the Subnet Manager stress tests inter-switch links (ISLs) by continuously passingtraffic through them. Other tools, like FastFabric, can be used to monitor the links forsignal integrity issues or other errors. The advantage of the loop test is that itprovides a guaranteed way to test all of the ISLs in the fabric, without the need for alarge number of end hosts or applications. For information on the SM Loop Test andhow to use the test, refer to the Intel® Omni-Path Fabric Suite Fabric Manager UserGuide.

15.5

Installation Verification and Additional Settings—Intel® Omni-Path Fabric


Part 3: Upgrading the Software

Intel® Omni-Path Fabric—Upgrading the Software


16.0 Upgrade Getting Started

This section provides instructions and information for getting started with the Intel®Omni-Path Fabric Suite upgrade.

Upgrade Prerequisites

Prior to upgrading the IFS software, ensure the following items have been completed:

• Review the Release Notes for a list of compatible software.

• Uninstall all versions of third-party IB stacks.

• Back up the following configuration files in case the upgrade fails:

— /etc/opa-fm/opafm.xml— /etc/opa/*— /etc/sysconfig/opa/*— /var/usr/lib/opa/analysis/baseline/*— Refer to the OS documentation for a list of any other OS-specific files that

should be included in any backups.

• Add or remove OS RPMs per OS RPMs Installation Prerequisites on page 33.

Download the Intel® Omni-Path Software

If the OS you installed did not include the OPA RPMs, download the software packagefrom an Intel web page or other Intel secured location using the following procedures.


2. In the "Search downloads" field, type Omni-Path and press Enter.

3. In the Description list select the "Intel® Omni-Path Fabric Software (IncludingIntel® Omni-Path Host Fabric Interface Driver)" for the version you are wanting toinstall .

Note: The latest version of each type of each download type is showing in the list.To show previous versions select "Show more" at the bottom of the list.

4. In the "Available Downloads" list, select the file(s) you need for the OS you haveinstalled on your fabric.

• For OPA-Basic software, select:

IntelOPA-Basic.DISTRO.VERSION.tgz

• For OPA-IFS software, select:

IntelOPA-IFS.DISTRO.VERSION.tgz

16.1

16.2

Upgrade Getting Started—Intel® Omni-Path Fabric


Note: DISTRO.VERSION refers to the distribution and CPU.


6. Click "I accept the terms in the license agreement."


Unpack the Tar File

You unpack the tar file using the following procedure.

1. Open an SSH client session and log into the host where the package is beinginstalled.

2. Copy the tar file to the /root directory.

3. Change directory to /root.

cd /root

4. Unpack the tar file.

• For OPA-Basic, use:

tar xvfz IntelOPA-Basic.DISTRO.VERSION.tgz

• For OPA-IFS, use:

tar xvfz IntelOPA-IFS.DISTRO.VERSION.tgz

16.3

Intel® Omni-Path Fabric—Upgrade Getting Started


17.0 Upgrade the Intel® Omni-Path Software

This section provides information and procedures to upgrade to the Intel® Omni-PathSoftware.

Before You Begin

Before starting the upgrade, perform the following:

• Refer to the Release Notes for the list of compatible operating systems.

• Be sure you have completed all Upgrade Prerequisites on page 121.

• You have downloaded and extracted the software package per Upgrade GettingStarted on page 121.

• If your opafm.xml configuration file contains custom changes that you want tocarry forward, make a backup copy of it.

Upgrade Scenarios

The following upgrades are available:

• To upgrade to a new version of the Intel® Omni-Path Software for compute nodes(Basic), go to Upgrade the OPA-Basic Software.

• To upgrade to a new version of the Intel® Omni-Path Software for ManagementNodes (IFS), go to Upgrade the OPA-IFS Software.

• To upgrade a Management Node from OPA-Basic to OPA-IFS, go to Upgrade fromOPA-Basic to OPA-IFS Software Package.

Upgrade the OPA-Basic Software

You upgrade the Intel® Omni-Path Software using the IntelOPA-Basic.DISTRO.VERSION.tgz package file.

Assumption


Procedures

Perform the following steps to upgrade the Intel® OP Software:


1. At the command prompt, change directory to the locationof the new Basic software package.

Type the following and press Enter:cd IntelOPA-Basic.DISTRO.VERSIONwhere DISTRO.VERSION is the distribution and CPU.


continued...

17.1

17.2

17.3

Upgrade the Intel® Omni-Path Software—Intel® Omni-Path Fabric




4. Review the items to be upgraded. Accept the defaults.Type N to go to the next page.NOTE: If you need to change any item, enter thealphanumeric character associated with the item to togglebetween Upgrade or Don't Install.

5. Start the upgrade. Type P to perform the actions.



Intel recommends leaving all of the Autostart selectionsset to the default values.NOTE: If you need to change any item, enter thealphanumeric character associated with the item to togglebetween Enable or Disable.



10. Exit the installation menu. Type X.


End Task

Upgrade the OPA-IFS Software

You can upgrade the OPA-IFS software to a new version using the Intel OPA Softwareand FastFabric TUI menus. You begin by upgrading each Management Node and thenthe remaining servers.

Upgrade the Software on the Management Nodes

To upgrade the Intel® Omni-Path Software, you first upgrade the Management Nodesusing the IntelOPA-IFS.DISTRO.VERSION.tgz package file.

Assumption

• You are logged in to the target Management Node.

Procedures

Perform the following steps to upgrade the Intel® OP Software on each ManagementNode:


1. At the command prompt, change directory to the locationof the new IFS software package.

For IFS, type the following and press Enter:cd IntelOPA-IFS.DISTRO.VERSIONwhere DISTRO.VERSION is the distribution and CPU.



4. Review the items to be upgraded. Accept the defaults.

continued...

17.4

17.4.1

Intel® Omni-Path Fabric—Upgrade the Intel® Omni-Path Software



Type N to go to the next page.NOTE: If you need to change any item, enter thealphanumeric character associated with the item to togglebetween Upgrade or Don't Install.









End Task

Next Steps

• If you have customizations to add to your opafm.xml configuration file, go to Update the Fabric Manager opafm Configuration File.

• If you are ready to upgrade the fabric, go to Upgrade the Software on theRemaining Servers.

Update the Fabric Manager opafm Configuration File

The opafm is designed so that any new parameters that might not appear in your oldopafm.xml file will automatically be set to their defaults in the software. If you havecustomized your pre-upgrade opafm.xml file, it is recommended that you manuallyupdate your new opafm.xml file with the latest comments and defaults. This will aidyou with future customizations and make comparisons with the opafm.xml in futureupgrades easier.

Assumption

• You have upgraded the Fabric Manager, accepting all the defaults.

Procedures

Perform the following steps to transfer customizations into a new opafm.xml file:


1. Compare your retained customized /etc/opa-fm/opafm.xml file to the new default /usr/share/opa-fm/opafm.xml file.

Use the following command to compare files:opafmconfigdiff /etc/opa-fm/opafm.xml /usr/share/opa-fm/opafm.xml.NOTE: The resulting differences will highlight yourprevious customizations.

continued...

17.4.2




Refer to the Intel® Omni-Path Fabric Suite FabricManager User Guide for more information onopafmconfigdiff.

2. Create a new opafm.xml. Use /usr/share/opa-fm/opafm.xml file as a base forthe new file by copying this read-only file to a temporarylocation to make the subsequent edits.

3. Update the new opafm.xml with customizationsnecessary for your fabric.

Use the resulting differences file from Step 1 to locate thecustomizations and add them to the new opafm.xml.

4. Update /etc/opa-fm/opafm.xml. Use the new file from Step 3 to replace your /etc/opa-fm/opafm.xml file.

5. Restart the Fabric Manager Type reboot and press Enter.

End Task

Next Steps

• If you are ready to upgrade the fabric, go to Upgrade the Software on theRemaining Servers.

Upgrade the Software on the Remaining Servers

After upgrading the Management Nodes, you upgrade the fabric software on theremaining servers using the FastFabric OPA Host Setup menu.

Assumptions

• You have upgraded the Management Nodes.


Procedures

Perform the following steps to upgrade the servers:


Configuring the Hosts

1. Access the FastFabric OPA Host Setup menu.



b) Access the FastFabric OPA Host Setup menu. Press 3.

c) Select menu items. • Select items 0 (optional) and 5.


2. (Optional) Edit Configuration and Select/Edit HostFile (menu item 0)

a) Edit the opafastfabric.conf file. Review the file and change as needed.If you made any changes, save and close the file.Press any key to continue.

b) Edit the hosts configuration file. Review the file and change as needed.If you made any changes, save and close the file.

continued...

17.4.3







3. Install/Upgrade OPA Software (menu item 5)

a) Do you want to use ./IntelOPA-[Basic|IFS].DISTRO.VERSION.tgz? [y]:


b) Would you like to do a fresh [i]nstall, an[u]pgrade or [s]kip this step? [u]:


c) Are you sure you want to proceed? [n]: Type y and press Enter.

d) When the Intel OPA Autostart Menu displays, reviewthe items.


e) Run the OPA Autostart operations. Type P.

f) For each system prompt, "Hit any key to continue..." Press any key.NOTE: When the installation completes, you are returnedto the main menu.

g) Complete the installation. Press any key to continue.

• If all hosts install... Press any key to continue.

• If any hosts fails to install... Use the View opahostadmin Result Files menu item toreview the result files from the update.


4. Reboot Hosts (menu item 8) (Linux) This menu item reboots all the selected hostsand ensures they go down and come back up properly, asverified through ping over the management network.When the hosts come back up, they are running theinstalled Intel® Omni-Path Fabric Host Software.

5. Run a Command on All Hosts (menu item b) (Linux) For any other setup operations that need to beperformed on all hosts, this menu item executes thespecified Linux* shell command against all selected hosts.It can also execute a sequence of commands separatedby semicolons.Note: Check the relevant release notes for the new

Intel® Omni-Path Fabric Host Software releasebeing installed for any additional required steps.

6. Copy a File to All Hosts (menu item c) This menu item specifies a file on the local host to becopied to all selected hosts.Note: When using the Intel® Omni-Path Fabric software,

you can use FastFabric to upgrade the Intel®Omni-Path Fabric software stack on the remaininghosts.

7. View opahostadmin Result Files (menu item d)




End Task



Next Steps

• To verify the host software has been upgraded and running on the remainingservers, go to Verify the Host Software on the Remaining Servers Using theFastFabric TUI Menu.

Upgrade from OPA-Basic to OPA-IFS Software Package

You can upgrade from OPA-Basic to OPA-IFS in order to install the FastFabric andOPA FM software.

Assumptions

• OPA-Basic is installed on the server.


Procedures

Perform the following steps to upgrade to OPA-IFS:


1. At the command prompt, change directory to the locationof the new IFS software package.

For IFS, type the following and press Enter:cd IntelOPA-IFS.DISTRO.VERSIONwhere DISTRO.VERSION is the distribution and CPU.



4. Review the items to be upgraded.NOTE: Ensure that FastFabric and OPA_FM showInstall and all other selections show Up To Date.

Type N to go to the next page.NOTE: If you need to change any item, enter thealphanumeric character associated with the item to togglebetween Upgrade or Don't Install.









End Task

Next Steps

• If you are setting up additional Management Nodes, go to Set Up AdditionalManagement Nodes on page 85.

17.5



Appendix A UEFI Secure Boot Support

Intel® Omni-Path software includes support for Unified Extensible Firmware Interface(UEFI) Secure Boot, both in the UEFI firmware environment and in Linux*.

UEFI Firmware Support

All UEFI drivers shipped as part of the Intel® Omni-Path Fabric Host Software aresigned to support UEFI Secure Boot using the Microsoft* UEFI CA signing process.

Limitations

PXE Booting over the Intel® Omni-Path fabric is not supported when UEFI Secure Bootmode is enabled.

Linux* Support

Kernel modules shipped with major Linux* distributions RHEL* 7.3 and SLES* 12.2are signed using the keys for those distributions.

Kernel modules shipped as part of the Intel® Omni-Path IFS installation suite aresigned using an Intel key. The Intel public key is automatically added to the MachineOwner Key (MOK) list in the UEFI firmware environment with no action required bythe user. Refer to the Client Node on page 142 section in the Setting Up PXE Boot forSupported Operating Systems on page 130 appendix for UEFI firmware requirements.

The following MOK utility can be used to list the currently enrolled keys in the MOK listkeyring:

mokutil --list-enrolled

For more information about UEFI Secure Boot support in RHEL*, refer to the RHEL*documentation at https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/System_Administrators_Guide/sec-UEFI_Secure_Boot.html .

For more information about UEFI Secure Boot support in SLES*, refer to the SLES*documentation at https://www.suse.com/documentation/sled11/book_sle_admin/data/sec_uefi_secboot.html.

Limitations

When UEFI Secure Boot mode is enabled, the following Intel® Omni-Path Linux*utilities are not available:

• hfi1_eprom• hfidiags

A.1

A.2

UEFI Secure Boot Support—Intel® Omni-Path Fabric


https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/System_Administrators_Guide/sec-UEFI_Secure_Boot.html



https://www.suse.com/documentation/sled11/book_sle_admin/data/sec_uefi_secboot.html

https://www.suse.com/documentation/sled11/book_sle_admin/data/sec_uefi_secboot.html

Appendix B Setting Up PXE Boot for Supported OperatingSystems

Different operating systems require specific instructions for setting up PXE Boot on theIntel® Omni-Path Fabric.

The basic requirement for the preferred environment consists of a cluster with at leasttwo nodes connected to a switch. One of the nodes is the Client node, where the OSwill be installed over HFI. The other node will be a Preboot Execution Environment(PXE) Server to which the Client will connect to initialize and perform the OSinstallation.

The communication over Intel® Omni-Path (IP over InfiniBand*) proceeds as follows:

1. Client requests to DHCP Server and receives the IP address of TFTP Server andboot image (the name of bootloader).

2. Client contacts the TFTP Server and receives the boot image.

3. Client executes the boot image and downloads from PXE Server all other neededfiles like kernel and initial root file system (initramfs) image and loads them.

Note: The initramfs image does not contain all the modules and firmwarerequired to use IPoIB with HFI. Therefore it needs to be updated beforebooting over HFI.

4. The installation of the target OS starts on the Client machine using the ISO imageof the target OS downloaded from the HTTP Server.

The figure below shows a high-level overview of the communication between Clientand Server.

The following procedures describe how to configure and boot a server over Intel®Omni-Path using the HFI UEFI firmware. Note that it does not cover all possible PXEconfigurations. The examples given are targeted for servers running the latest,supported Red Hat* Enterprise Linux* (RHEL*) and SUSE* Linux* Enterprise Server(SLES*) operating systems.

Intel® Omni-Path Fabric—Setting Up PXE Boot for Supported Operating Systems


• For information on other PXE server configurations, please refer to your OS vendordocumentation.

• For PXE boot information on earlier supported RHEL* and SLES* OS versions,contact your Intel Support Personnel.

PXE Server

You use the Intel® Omni-Path Host Fabric Interface (HFI) Unified Extensible FirmwareInterface (UEFI) firmware to configure a server to boot over Intel® Omni-Path.

This sections describes the procedures for different, supported operating systems.

Configure PXE Server with RHEL* 7.4

Before You Begin

The OPA-IFS package contains the HFI driver and the required IP over IB softwarestack. The package also installs the Fabric Manager. Note that the Fabric Managermust be installed for the Intel® Omni-Path Boot client to be fully initialized.

In order to boot over Intel® Omni-Path, the following requirements must be met:

1. The Intel® Omni-Path Fabric Suite Fabric Manager software must be running.

Usually Fabric Manager is running on the Switch, but in a simple Back-to-Backconfiguration, Fabric Manager will be started on a head node (PXE server).

Notes: If you need to install and start the Fabric Manager, perform the followingsteps:

a. Install Intel® Omni-Path Fabric Suite Fabric Manager using thecommand:

yum install opa-fm

b. Start up Fabric Manager using the following command:

service opafm start

2. The HFI driver (hfi1) and IP over InfiniBand* modules must be running. Use thecommands:

modprobe hfi1modprobe ib_ipoib

Configure Network Interface

Before IP packets can be sent to and from the PXE server, an Intel® Omni-Pathnetwork interface must be configured. To configure an Intel® Omni-Path networkinterface, perform the following steps:

1. Add ib0 network interface configuration file: /etc/sysconfig/network-scripts/ifcfg-ib0

B.1

B.1.1

B.1.1.1

B.1.1.2

Setting Up PXE Boot for Supported Operating Systems—Intel® Omni-Path Fabric


2. Add the following content related to DHCP configuration to the configuration file:

DEVICE=ib0TYPE=InfinibandBOOTPROTO=staticIPADDR=192.168.100.1NETMASK=255.255.255.0NETWORK=192.168.100.0BROADCAST=192.168.100.255ONBOOT=yesUSERCTL=no

3. Run the following command, where ib0 is the name of the network interfaceconfigured in step 2:

ifup ib0

Configure DHCP Server

The following procedure describe how to configure DHCP for dynamic IP or fixed IPaddressing.

For Dynamic IP Addressing

To install a DHCP server for dynamic IP addressing, follow the steps below:

1. Install DHCP package using the command:

yum install dhcp

2. Add DHCP configuration file.

The following is an example of /etc/dhcp/dhcpd.conf

## DHCP Server Configuration file.#option space pxelinux;option pxelinux.magic code 208 = string;option pxelinux.configfile code 209 = text;option pxelinux.pathprefix code 210 = text;option pxelinux.reboottime code 211 = unsigned integer 32;option architecture-type code 93 = unsigned integer 16;allow booting;allow bootp;subnet 192.168.100.0 netmask 255.255.255.0 { range 192.168.100.2 192.168.100.230; option subnet-mask 255.255.255.0; option broadcast-address 192.168.100.255; option routers 192.168.100.1; class "pxeclients" { match if substring (option vendor-class-identifier, 0, 9) = "PXEClient"; next-server 192.168.100.1; if option architecture-type = 00:07 { filename "uefi/bootx64.efi"; } else { filename "pxelinux/pxelinux.0"; } }}

B.1.1.3



3. Start DHCP server using the command:

service dhcpd start

4. Enable DHCP service to be run always with OS start:

systemctl enable dhcpd.service

For Fixed IP Addressing

To install a DHCP server for fixed IP addressing, follow the steps below:

1. Use the GUID from the client HFI in this example:

001175010165abb2

2. Configure the dhcpd.conf file as the following, using the standard ff:00:00:00:00:00:02:00:00:01:57:

[root@phs1fnivd14u25 dhcp]# cat /etc/dhcp/dhcpd.confoption space pxelinux;option pxelinux.magic code 208 = string;option pxelinux.configfile code 209 = text;option pxelinux.pathprefix code 210 = text;option pxelinux.reboottime code 211 = unsigned integer 32;option architecture-type code 93 = unsigned integer 16;

allow booting;allow bootp;

subnet 192.168.100.0 netmask 255.255.255.0 { range 192.168.100.20 192.168.100.230; option subnet-mask 255.255.255.0; option broadcast-address 192.168.100.255; option routers 192.168.100.1;

host phs1fnivd14u23 { #dynamic; uid ff:00:00:00:00:00:02:00:00:01:57:00:11:75:01:01:65:ab:b2; fixed-address 192.168.100.15; supersede dhcp-client-identifier =ff:00:00:00:00:00:02:00:00:01:57:00:11:75:01:01:65:ab:b2;}

class "pxeclients" { match if substring (option vendor-class-identifier, 0, 9) = "PXEClient"; next-server 192.168.100.1; if option architecture-type = 00:07 { filename "uefi/bootx64.efi"; } else { filename "pxelinux/pxelinux.0"; } }}


service dhcp start



4. The output from /var/log/messages for the DHCP server will look somethinglike the following example:

Jun 30 03:16:13 phs1fnivd14u25 systemd: Started DHCPv4 Server Daemon.Jun 30 03:16:26 phs1fnivd14u25 dhcpd: DHCPDISCOVER fromff:00:00:00:00:00:02:00:00:01:57:00:11:75:01:01:65:ab:b2 via ib0Jun 30 03:16:26 phs1fnivd14u25 dhcpd: DHCPOFFER on 192.168.100.15 toff:00:00:00:00:00:02:00:00:01:57:00:11:75:01:01:65:ab:b2 via ib0Jun 30 03:16:29 phs1fnivd14u25 dhcpd: DHCPREQUEST for 192.168.100.15(192.168.100.1) from ff:00:00:00:00:00:02:00:00:01:57:00:11:75:01:01:65:ab:b2via ib0Jun 30 03:16:29 phs1fnivd14u25 dhcpd: DHCPACK on 192.168.100.15 toff:00:00:00:00:00:02:00:00:01:57:00:11:75:01:01:65:ab:b2 via ib0

Install a TFTP Server

Prerequisite: XINETD may have been installed with OS. If not, use the followingcommand:

yum install xinetd

To install a TFTP server, perform the following steps:

1. Install TFTP package using the command:

yum install tftp-server

2. Add or modify TFTP configuration file. An example is shown below: /etc/xinetd.d/tftp

service tftp { socket_type = dgram protocol = udp wait = yes user = root server = /usr/sbin/in.tftpd server_args = -s /var/lib/tftpboot disable = no per_source = 11 cps = 100 2 flags = IPv4}

Note: The disable field defaults to yes and should be changed to no beforestarting the XINETD server.

3. Start XINETD server using the command:

service xinetd start

4. Enable XINETD service to be run always with OS start:

systemctl enable xinetd.service

B.1.1.4



Create Directory Structure for PXE Server

The PXE Server is contained within a directory structure indicated by theserver_args argument of the TFTP configuration file.

1. Create the base directory structure using the command:

mkdir -p /var/lib/tftpboot/uefi

Configure the HTTP Server

To configure the HTTP Server, perform the following steps:

1. Install the TFTP package using the command:

yum install httpd

2. Add the configuration file /etc/httpd/conf.d/netinstall.conf with thefollowing content:

Alias /netinstall /var/www/netinstall<Directory /var/www/netinstall>Options Indexes FollowSymLinksOrder Deny,AllowAllow from all</Directory>

3. Create a new directory for iso image of the system that will be installed over theIntel® Omni-Path Host Fabric Interface. Use the following command:

mkdir -p /var/www/netinstall/ISO/

4. Start the HTTP server using the command:

service httpd start

5. Enable the HTTP service to be run always with OS start:

systemctl enable httpd.service

6. Whenever the content of /var/www/netinstall is modified use the command:

service httpd reload

Post-requisite

To see if the HTTP server was started successfully, open a web browser and enter <IPaddress of the PXE server>/netinstall.

For example:, if IP address of the PXE server is 100.100.200.200, use http://100.100.200.200/netinstall/.

B.1.1.5

B.1.1.6



Configure the Boot Loader

To configure a PXE server to use the Elilo boot loader, perform the following steps.

Note: For configuration details of other boot loaders, please refer to your OS vendordocumentation.

1. Download the latest source files for Elilo: https://sourceforge.net/projects/elilo/files/elilo/

2. Unpack the archive file.

tar xzf elilo-3.16-all.tar.gz

3. Rename the efi file and copy it to the PXE Server directory.

mv elilo-3.16-x86_64.efi bootx64.eficp bootx64.efi /var/lib/tftpboot/uefi/

4. Unpack the next archive file and copy the text menu message file to the PXEServer directory.

tar xzf elilo-3.16-source.tar.gzcp elilo-3.16-source/examples/textmenu_chooser/textmenu-message.msg /var/lib/tftpboot/uefi/

5. Create the Elilo configuration file: /var/lib/tftpboot/uefi/elilo.conf.

The following is an example for RHEL* 7.x using vmlinuz kernel file andinitramfs.img as a minimal file system image:

chooser=textmenumessage=textmenu-message.msgpromptdelay=20 default=rhel7image=vmlinuzlabel=rhel7description = "RHEL 7"initrd = initramfs.imgread-only

Configure PXE Server with SLES* 12 SP3

Before You Begin

The OPA-IFS package contains the HFI driver and the required IP over IB softwarestack. The package also installs the Fabric Manager. Note that the Fabric Managermust be installed for the Intel® Omni-Path Boot client to be fully initialized.

In order to boot over Intel® Omni-Path, the following requirements must be met:

1. The Intel® Omni-Path Fabric Suite Fabric Manager software must be running.

Usually Fabric Manager is running on the Switch, but in a simple Back-to-Backconfiguration, Fabric Manager will be started on a head node (PXE server).

B.1.1.7

B.1.2

B.1.2.1



https://sourceforge.net/projects/elilo/files/elilo/


Notes: If you need to install and start the Fabric Manager, perform the followingsteps:

a. Install Intel® Omni-Path Fabric Suite Fabric Manager using thecommand:

zypper install opa-fm

b. Start up Fabric Manager using the following command:

service opafm start

2. The HFI driver (hfi1) and IP over InfiniBand* modules must be running. Use thecommands:

modprobe hfi1modprobe ib_ipoib

Configure Network Interface

Before IP packets can be sent to and from the PXE server, an Intel® Omni-Pathnetwork interface must be configured. To configure an Intel® Omni-Path networkinterface, perform the following steps:

1. Add ib0 network interface configuration file: /etc/sysconfig/network-scripts/ifcfg-ib0

2. Add the following content related to DHCP configuration to the configuration file:

DEVICE=ib0TYPE=InfinibandBOOTPROTO=staticIPADDR=192.168.100.1NETMASK=255.255.255.0NETWORK=192.168.100.0BROADCAST=192.168.100.255ONBOOT=yesUSERCTL=no

3. Run the following command, where ib0 is the name of the network interfaceconfigured in step 2:

ifup ib0

Configure DHCP Server

The following procedure describe how to configure DHCP for dynamic or fixed IPaddressing.

B.1.2.2

B.1.2.3



For Dynamic IP Addressing

To install a DHCP server for dynamic IP addressing, follow the steps below:

1. Install DHCP package using the command:

zypper install dhcp-server

2. Add DHCP configuration file.

The following is an example: /etc/dhcpd.conf

## DHCP Server Configuration file.#option space pxelinux;option pxelinux.magic code 208 = string;option pxelinux.configfile code 209 = text;option pxelinux.pathprefix code 210 = text;option pxelinux.reboottime code 211 = unsigned integer 32;option architecture-type code 93 = unsigned integer 16;allow booting;allow bootp;subnet 192.168.100.0 netmask 255.255.255.0 { range 192.168.100.2 192.168.100.230; option subnet-mask 255.255.255.0; option broadcast-address 192.168.100.255; option routers 192.168.100.1; class "pxeclients" { match if substring (option vendor-class-identifier, 0, 9) = "PXEClient"; next-server 192.168.100.1; if option architecture-type = 00:07 { filename "uefi/bootx64.efi"; } else { filename "pxelinux/pxelinux.0"; } }}

3. Modify /etc/sysconfig/dhcpd by adding at the end of file line:

DHCPD_INTERFACE="ib0"


service dhcpd start

5. Enable DHCP service to be run always with OS start:

systemctl enable dhcpd.service

For Fixed IP Addressing

To install a DHCP server for fixed IP addressing, follow the steps below:

1. Use the GUID from the client HFI in this example:

001175010165abb2



2. Configure the dhcpd.conf file as the following, using the standard ff:00:00:00:00:00:02:00:00:01:57:

[root@phs1fnivd14u25 dhcp]# cat /etc/dhcp/dhcpd.confoption space pxelinux;option pxelinux.magic code 208 = string;option pxelinux.configfile code 209 = text;option pxelinux.pathprefix code 210 = text;option pxelinux.reboottime code 211 = unsigned integer 32;option architecture-type code 93 = unsigned integer 16;

allow booting;allow bootp;

subnet 192.168.100.0 netmask 255.255.255.0 { range 192.168.100.20 192.168.100.230; option subnet-mask 255.255.255.0; option broadcast-address 192.168.100.255; option routers 192.168.100.1;

host phs1fnivd14u23 { #dynamic; uid ff:00:00:00:00:00:02:00:00:01:57:00:11:75:01:01:65:ab:b2; fixed-address 192.168.100.15; supersede dhcp-client-identifier =ff:00:00:00:00:00:02:00:00:01:57:00:11:75:01:01:65:ab:b2;}

class "pxeclients" { match if substring (option vendor-class-identifier, 0, 9) = "PXEClient"; next-server 192.168.100.1; if option architecture-type = 00:07 { filename "uefi/bootx64.efi"; } else { filename "pxelinux/pxelinux.0"; } }}


service dhcp start

4. The output from /var/log/messages for the DHCP server will look somethinglike the following example:

Jun 30 03:16:13 phs1fnivd14u25 systemd: Started DHCPv4 Server Daemon.Jun 30 03:16:26 phs1fnivd14u25 dhcpd: DHCPDISCOVER fromff:00:00:00:00:00:02:00:00:01:57:00:11:75:01:01:65:ab:b2 via ib0Jun 30 03:16:26 phs1fnivd14u25 dhcpd: DHCPOFFER on 192.168.100.15 toff:00:00:00:00:00:02:00:00:01:57:00:11:75:01:01:65:ab:b2 via ib0Jun 30 03:16:29 phs1fnivd14u25 dhcpd: DHCPREQUEST for 192.168.100.15(192.168.100.1) from ff:00:00:00:00:00:02:00:00:01:57:00:11:75:01:01:65:ab:b2via ib0Jun 30 03:16:29 phs1fnivd14u25 dhcpd: DHCPACK on 192.168.100.15 toff:00:00:00:00:00:02:00:00:01:57:00:11:75:01:01:65:ab:b2 via ib0



Install a TFTP Server

Prerequisite: XINETD may have been installed with OS. If not, use the followingcommand:

zypper install xinetd

To install a TFTP server, perform the following steps:

1. Install TFTP package using the command:

zypper install tftp

2. Add or modify TFTP configuration file. An example is shown below: /etc/xinetd.d/tftp

service tftp{ socket_type = dgram protocol = udp wait = yes flags = IPv6 IPv4 user = root server = /usr/sbin/in.tftpd server_args = -u tftp -s /srv/tftpboot# per_source = 11# cps = 100 2 disable = no}

Note: The disable field defaults to yes and should be changed to no beforestarting the XINETD server.

3. Start XINETD server using the command:

service xinetd start

4. Enable XINETD service to be run always with OS start:

systemctl enable xinetd.service

Create Directory Structure for PXE Server

The PXE Server is contained within a directory structure indicated by theserver_args argument of the TFTP configuration file.

1. Create the base directory structure using the command.

mkdir -p /var/lib/tftpboot/uefi

2. Change the SUSE default access permission of the TFTP Server directory.

chmod 755 /srv/tftpboot/

B.1.2.4

B.1.2.5



Configure the HTTP Server

To configure the HTTP Server, perform the following steps:

1. Install the TFTP package using the command:

zypper install apache2

2. Add the configuration file /etc/apache2/conf.d/netinstall.conf with thefollowing content:

Alias /netinstall /var/www/netinstall<Directory /var/www/netinstall>Options Indexes FollowSymLinksRequire all granted</Directory>

Note: The configuration file above is valid for Apache* version 2.4 or higher.

3. Create a new directory for iso image of the system that will be installed overIntel® Omni-Path Host Fabric Interface. Use the following command:

mkdir -p /srv/www/netinstall/ISO/

4. Start the HTTP server using the command:

service apache2 start

5. Enable the HTTP service to be run always with OS start:

systemctl enable apache2.service

6. Whenever the content of /srv/www/netinstall is modified use the command:

service apache2 reload

Post-requisite

To see if the HTTP server was started successfully, open a web browser and enter <IPaddress of the PXE server>/netinstall.

For example:, if IP address of the PXE server is 100.100.200.200, use http://100.100.200.200/netinstall/.

Configure the Boot Loader

To configure a PXE server to use the Elilo boot loader, perform the following steps.

Note: For configuration details of other boot loaders, please refer to your OS vendordocumentation.

1. Download the latest source files for Elilo: https://sourceforge.net/projects/elilo/files/elilo/

B.1.2.6

B.1.2.7





2. Unpack the archive file.

tar xzf elilo-3.16-all.tar.gz

3. Rename the efi file and copy it to the PXE Server directory.

mv elilo-3.16-x86_64.efi bootx64.eficp bootx64.efi /srv/tftpboot/uefi/

4. Unpack the next archive file and copy the text menu message file to the PXEServer directory.

tar xzf elilo-3.16-source.tar.gzcp elilo-3.16-source/examples/textmenu_chooser/textmenu-message.msg /srv/tftpboot/uefi/

5. Create the Elilo configuration file: /srv/tftpboot/uefi/elilo.conf.

The following is an example for SUSE* Linux 12.x using vmlinuz kernel file andinitramfs.img as a minimal file system image:

chooser=textmenumessage=textmenu-message.msgpromptdelay=20 default=suse12image=vmlinuzlabel=suse12description = "SUSE Linux 12"initrd = initramfs.imgread-only

Client Node

You install the operating system over Intel® OP HFI for the Client node.

The following table shows the BIOS versus Boot Mode information.

Table 3. BIOS versus Boot Mode

BIOS Boot Mode UEFI DriverIs Run By

BIOS?

PCIe 8GT/s(Gen3)

Transition

PlatformConfigurationData for AOC

support

BootOptions

BoardDescription

Subsystem/SubVendor ID

UEFI UEFI BIOS Yes Achieved preboot

Provided byUEFI driver

Boot overfabricBoot local

Customizedby UEFIdriver

Customized byUEFI driver

continued...

B.2



BIOS Boot Mode UEFI DriverIs Run By

BIOS?

PCIe 8GT/s(Gen3)

Transition

PlatformConfigurationData for AOC

support

BootOptions

BoardDescription

Subsystem/SubVendor ID

Boot fromEthernet

UEFI LegacyBIOS

Yes (BIOSthat runs

UEFI driverin thismode)

Achieved preboot

Provided byUEFI driver

Boot localBoot fromEthernet

Default boarddescription

Customized byUEFI driver

UEFI LegacyBIOS

No (BIOSdoes not runUEFI driverin this mode– e.g. IntelUEFI BIOS)

Achieved byhost driverpost boot

Provided bydriver post boot(withlimitations)



Defaultsubsystemvalues

Legacy Legacy No Achieved byhost driverpost boot

Provided bydriver post boot(withlimitations)



Defaultsubsystemvalues

This sections describes the procedures for different, supported operating systems.

Before You Begin

Before installing the operating system over Intel® OP HFI, perform the following:

• Perform the steps in Upgrade the HFI UEFI Firmware if needed.

• Be sure to build the initrd.img image file on a machine that uses the same OSdistribution that it targets. For instance, you build RHEL* 7.2 initrd.img on amachine that has Linux RHEL* 7.2 running on it.

Upgrade the HFI UEFI Firmware

The HFI UEFI firmware is available as part of the hfi1-uefi.x86_64 RPM, released withthe Intel® Omni-Path Software package.

To upgrade the HFI UEFI firmware, perform the following steps:

1. Install the new HFI UEFI rpm.

# rpm -i hfi1-uefi-X.X-XX.x86_64.rpm

2. Change directory to /usr/share/opa/bios_images/.

# cd /usr/share/opa/bios_images/

3. Update the HFI firmware. The basic command is:

# hfi1_eprom [See Additional Options] -w -o HfiPcieGen3Loader_1.6.0RC0.1.rom -b HfiPcieGen3_1.6.0RC0.1.efi -c /lib/firmware/updates/hfi1_platform.dat

Note: If the cables configuration file needs to be updated, add the -cfull_file_path option at the end of the command as shown above.

B.2.1

B.2.2



Example output of update:

Using device: /sys/bus/pci/devices/0000:81:00.0/resource0Erasing loader file... doneWriting loader file... doneErasing driver file... doneWriting driver file... doneErasing config file... doneWriting config file... done

Additional Options:

The following information provides guidance for various firmware update options.

• To update all of the HFIs, add the -d all option before the -w option.

# hfi1_eprom -d all -w -o HfiPcieGen3Loader_1.6.0RC0.1.rom -b HfiPcieGen3_1.6.0RC0.1.efi -c /lib/firmware/updates/hfi1_platform.dat

• To update each HFI separately (using -d only), perform the following steps:

Note: This procedure needs to be repeated for every HFI card installed in thesystem (listed by lspci command).

a. Collect the PCIe addresses (first column in the output below) for each HFIto be updated using lspci |grep HFI.

Example output is:

# lspci |grep HFI05:00.0 Fabric controller: Intel Corporation Omni-Path HFI Silicon 100 Series [discrete] (rev 11)81:00.0 Fabric controller: Intel Corporation Omni-Path HFI Silicon 100 Series [discrete] (rev 11)

b. Add -d <resource0_file_full_path> as shown in the examplebelow:

# hfi1_eprom -d /sys/bus/pci/devices/<hfi_pcie_address>/resource0 -w -o HfiPcieGen3Loader_1.6.0RC0.1.rom -b HfiPcieGen3_1.6.0RC0.1.efi -c /lib/firmware/updates/hfi1_platform.dat

where <hfi_pcie_address> is the value read from the first column ofthe lspci command output. From the example output in 3.a, the<resource0_file_full_path> is: /sys/bus/pci/devices/0000:81:00.0/resource0.

4. Read loader file version.

# hfi1_eprom –V –o

Example output:

Using device: /sys/bus/pci/devices/0000:02:00.0/resource0loader file version: 1.6.0RC0.1

5. Read driver file version.

# hfi1_eprom –V –b



Example output:

Using device: /sys/bus/pci/devices/0000:02:00.0/resource0driver file version: 1.6.0RC0.1

6. Reboot the server.

# reboot

Install RHEL* 7.4 over Intel® OP HFI

To edit the initrd.img, perform the following steps:

1. Log in as root.

2. Decompress the .iso archive with RHEL* 7.4 distribution.

The initrd.img is located in the /images/pxeboot folder.

3. Create /home/images/initrd folder.

mkdir -p /home/images/initrd

4. Copy initrd.img to home/images/initrd.

cp <.iso decompressed archive path>/images/pxeboot/initrd.img /home/images/initrd

5. Change directories to /home/images/initrd.

cd /home/images/initrd

6. Check archive type.

For example:

file initrd.imginitrd.img: XZ compressed data

7. Extract initrd.img according to archive type:

• For XZ archive

xzcat ./initrd.img | cpio -d -i -m

• For GZ archive

gunzip < initrd.img | cpio -i --make-directories

8. Move initrd.img out of the folder.

mv ./initrd.img ../initrd.img

Note: From this point on in the instructions, all commands assume that thepresent working directory (pwd) is /home/images/initrd and allcommands are relative to this path.

B.2.3



9. Copy firmware files.

cp --recursive /lib/firmware/updates/ usr/lib/firmware/updates/

The list of required firmware files are:

/lib/firmware/updates/hfi1_dc8051_d.fw/lib/firmware/updates/hfi1_dc8051.fw/lib/firmware/updates/hfi1_fabric_d.fw/lib/firmware/updates/hfi1_fabric.fw/lib/firmware/updates/hfi1_pcie_d.fw/lib/firmware/updates/hfi1_pcie.fw/lib/firmware/updates/hfi1_platform.dat/lib/firmware/updates/hfi1_sbus_d.fw/lib/firmware/updates/hfi1_smbus.fw/lib/firmware/updates/hfi_dc8051.bin

10. Add ib_ipoib.conf file.

mkdir etc/modules-load.d/touch etc/modules-load.d/ib_ipoib.confecho "# Load ib_ipoib.ko at boot" >> etc/modules-load.d/ib_ipoib.confecho "ib_ipoib" >> etc/modules-load.d/ib_ipoib.conf

11. Generate initrd-new.img file and give permissions.

cd /home/images/initrdfind . -print |cpio -o -H newc | xz --format=lzma > ../initrd-new.imgchmod 777 ../initrd-new.img

Note: Remember to use appropriate Linux kernel file (vmlinuz) along withinitrd.img.

12. Sample elilo.conf.

promptchooser=textmenumessage=textmenu-message.msgdelay=20 default=rhel74image=vmlinuz label=rhel74 description = "RHEL" initrd=initrd-new.img read-only append="ip=ib0:dhcp rd.driver.post=ib_ipoib rd.net.timeout.carrier=50 inst.repo=http://192.168.100.1/netinstall/RHEL74"

Install SLES* 12 SP3 over Intel® OP HFI

To edit the initrd.img, perform the following steps:

1. Log in as root.

2. Mount the SLES* 12 SP3 .iso file.

sudo mkdir /mnt/isosudo mount -o loop SLE-12-SP2-Server-DVD-x86_64-GM-DVD1.iso /mnt/iso

B.2.4



3. Create a working directory /home/new_initrd.

mkdir /home/new_initrd

4. Copy linux file to home/new_initrd.

cp /mnt/iso/boot/x86_64/loader/linux /home/new_initrd

Note: linux (Linux kernel – corresponds to “vmlinuz” in RHEL) will not bemodified. Please remember to use this file as the kernel file provided forPXE Boot (in elilo.conf).

5. Copy initrd.img to home/new_initrd.

cp /mnt/iso/boot/x86_64/loader/initrd /home/new_initrd/initrd.img

Note: initrd (initial root filesystem image) will be modified in this procedure.

6. Change directories to /home/new_initrd.

cd /home/new_initrd

7. Create a new initrd folder and move initrd.img into it.

mkdir initrdmv initrd.img ./initrd

8. Extract initrd file.

cd initrdxzcat ./initrd.img | cpio -d -i -m

9. Move initrd.img out of the folder.

mv initrd.img ../cd ..

10. Go to initrd/parts directory.

cd initrd/parts

11. Copy 00_lib file into new_initrd directory (outside of initrd).

cp 00_lib ../..cd ../..

12. Extract 00_lib file.

unsquashfs 00_lib

Note: 00_lib will be extracted to a newly created ./squashfs-root directory.



After this step, the present working directory (/home/new_initrd) should looklike this:

# ls00_libinitrdinitrd.imglinuxsquashfs-root

13. Copy five firmware drivers to squashfs-root/lib/firmware directory.

cp /lib/firmware/updates/hfi1_fabric.fw ./squashfs-root/lib/firmware/cp /lib/firmware/updates/hfi1_pcie.fw ./squashfs-root/lib/firmware/cp /lib/firmware/updates/hfi1_dc8051.fw ./squashfs-root/lib/firmware/cp /lib/firmware/updates/hfi1_sbus.fw ./squashfs-root/lib/firmware/

14. Copy the following additional drivers from system that are not present in thedefault initrd.img.

cp /lib/modules/$(uname -r)/updates/ifs-kernel-updates/ib_mad.ko squashfsroot/lib/modules/$(uname -r)/initrd/cp /lib/modules/$(uname -r)/updates/ifs-kernel-updates/rdmavt.ko squashfsroot/lib/modules/$(uname -r)/initrd/cp /lib/modules/$(uname -r)/updates/ifs-kernel-updates/hfi1.ko squashfsroot/lib/modules/$(uname -r)/initrd/

15. In order to keep the existing module dependencies, temporarily copy somemodules from <initrd-root>/lib/modules/$(uname -r)/initrd intosquashfsroot/lib/modules/$(uname -r)/initrd.

cp initrd/lib/modules/$(uname -r)/initrd/lz4_decompress.ko squashfsroot/lib/modules/$(uname -r)/initrd/cp initrd/lib/modules/$(uname -r)/initrd/squashfs.ko squashfsroot/lib/modules/$(uname -r)/initrd/cp initrd/lib/modules/$(uname -r)/initrd/loop.ko squashfs-root/lib/modules/$(uname-r)/initrd/

16. Create a backup copy of an existing <initrd-root>/lib/modules directory.

cp -R initrd/lib/modules modules_backup

17. Temporarily remove the modules directory from initrd.

rm -rf initrd/lib/modules

18. Copy modules directory from squashfs to initrd.

cp -R squashfs-root/lib/modules initrd/lib

19. Change root to initrd root directory and rebuild module dependencies.

cd initrdchroot $(pwd)depmod -a -vexitcd ..



20. Go to <initrd-root>/lib/modules/$(uname -r)/initrd and remove allfiles added in 15.

rm initrd/lib/modules/$(uname -r)/initrd/lz4_decompress.korm initrd/lib/modules/$(uname -r)/initrd/loop.korm initrd/lib/modules/$(uname -r)/initrd/squashfs.ko

21. Remove modules directory from squashfs.

rm -rf squashfs-root/lib/modules

22. Copy a modified modules directory back to the squashfs.

cp -R initrd/lib/modules squashfs-root/lib

23. Remove modules directory from initrd.

rm -rf initrd/lib/modules

24. Restore (from the backup done in 16) the previous content of modules directory.

cp -R modules_backup initrd/lib/modules

25. Pack squashfs-root contents again to 00_lib.

mksquashfs squashfs-root 00_lib_new

26. Overwrite the existing 00_lib in the initrd directory.

cp 00_lib_new initrd/parts/00_lib

27. Compress again initrd contents (from initrd-root).

cd initrdfind . -print |cpio -ov -H newc | xz --format=lzma > ../initrd-sles.img

28. Change privileges of a new initrd.

chmod 777 ../initrd-sles.img

29. Update elilo.conf to include the line append="insmod=ib_ipoibnetdevice=ib0 netsetup=dhcp".

This forces an automatic ib_ipoib module start, uses ib0 as a default networkdevice, and configures IP address via DHCP.

promptchooser=textmenumessage=textmenu-message.msgdelay=20 default=sles123image=loader/linux label=sles123 description = "SLES" initrd=loader/initrd-new.img



read-only append="insmod=ib_ipoib netdevice=ib0 netsetup=dhcp install=http://192.168.100.1/netinstall/SLES_123"

Documents

Intel® Omni-Path Fabric Software Installation Guide · Intel® Omni-Path Fabric Software Installation Guide Rev. 8.0 ... 4.1.1 Fabric Design Prerequisites.....30 4.1.2 Fabric Setup