Page 2 - Document Revision History
Page ii IB0056101-00 G S QLogic HCA and InfiniPath ® Software Install Guide Version 2.2 Information furnished in this manual is believed to be accurate and reliable. However, QLogic Corporation assumes no responsibility for its use, nor for any infringements of patents or other rights of third parti...
Page 3 - Table of Contents; Introduction
IB0056101-00 G Page iii Table of Contents 1 Introduction Who Should Read this Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1 How this Guide is Organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1 Overview . . . . . . . . . . . . . ....
Page 4 - Software Installation
Page iv IB0056101-00 G QLogic HCA and InfiniPath ® Software Install Guide Version 2.2 S Hardware Installation for QLE7240, QLE7280, or QLE7140 with PCI Express Riser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-9 Dual Adapter Installation. . . . . . . . . . ....
Page 5 - Installation Troubleshooting
IB0056101-00 G Page v QLogic HCA and InfiniPath ® Software Install Guide Version 2.2 A Starting and Stopping the InfiniPath Software . . . . . . . . . . . . . . . . . . . . . . . 5-26 Rebuilding or Reinstalling Drivers After a Kernel Upgrade . . . . . . . . . . . . . 5-28 Rebuilding or Reinstalling ...
Page 6 - Configuration Files
Page vi IB0056101-00 G QLogic HCA and InfiniPath ® Software Install Guide Version 2.2 S B Configuration Files C RPM Descriptions InfiniPath and OpenFabrics RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-1 Different Nodes May Use Different RPMs . . . . . . . . . . . . . . . . ...
Page 7 - List of Figures; List of Tables
IB0056101-00 G Page vii QLogic HCA and InfiniPath ® Software Install Guide Version 2.2 A List of Figures Figure Page 4-1 QLogic QLE7280 with IBA7220 ASIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-7 4-2 QLogic QLE7140 Card with Riser, Top View . . . . . . . . . . . . . ....
Page 8 - Notes
Page viii IB0056101-00 G QLogic HCA and InfiniPath ® Software Install Guide Version 2.2 S Notes
Page 9 - Who Should Read this Guide; How this Guide is Organized
IB0056101-00 G 1-1 1 Introduction This chapter describes the contents, intended audience, and organization of the QLogic HCA and InfiniPath Software Install Guide. The QLogic HCA and InfiniPath Software Install Guide contains instructions for installing the QLogic Host Channel Adapters (HCAs) and th...
Page 10 - Overview
1 – IntroductionOverview 1-2 IB0056101-00 G S Section 3 , Step-by-Step Installation Checklist, provides a high-level overview of the hardware and software installation procedures. Section 4 , Hardware Installation, includes instructions for installing the QLogic QLE7140, QLE7240, QLE7280, QHT704...
Page 11 - Interoperability; Conventions Used in this Guide
1 – Introduction Interoperability IB0056101-00 G 1-3 A InfiniPath OpenFabrics software is interoperable with other vendors’ InfiniBand Host Channel Adapters (HCAs) running compatible OpenFabrics releases. There are several options for subnet management in your cluster: Use the embedded Subnet Mana...
Page 12 - Documentation
1 – IntroductionDocumentation 1-4 IB0056101-00 G S Documentation The product documentation includes: The QLogic HCA and InfiniPath Software Install Guide The QLogic HCA and InfiniPath Software User Guide The QLogic FastFabric Users Guide (for information on QLogic InfiniServ Tools) The OFED+...
Page 13 - Contact Information
1 – Introduction Contact Information IB0056101-00 G 1-5 A Contact Information Support Headquarters QLogic Corporation4601 Dean Lakes BlvdShakopee, MN 55379USA QLogic Web Site www.qlogic.com Technical Support Web Site support.qlogic.com Technical Support Email [email protected] Technical Training Em...
Page 15 - Feature Overview; What’s New in this Release
IB0056101-00 G 2-1 2 Feature Overview This section contains the features for this release, the supported QLogic adapter models, supported distributions and kernels, and a list of the software components. What’s New in this Release This release adds support for the QLE7240 and QLE7280 InfiniBand DDR ...
Page 16 - New Features
2 – Feature OverviewNew Features 2-2 IB0056101-00 G S New Features The following features are new to the 2.2 release: Expanded MPI scalability enhancements for PCI Express have been added. On the QLE7240 and QLE7280, up to 16 dedicated hardware contexts per node are available. The QHT7140 has eigh...
Page 17 - Supported Distributions and Kernels
2 – Feature Overview Supported Distributions and Kernels IB0056101-00 G 2-3 A A subset of the QLogic InfiniBand Fabric Suite, the enablement tools, are offered with this release. Two separate SCSI RDMA Protocol (SRP) modules are provided: the standard OpenFabrics (OFED) SRP, and QLogic SRP. QL...
Page 18 - Compiler Support; Software Components
2 – Feature OverviewCompiler Support 2-4 IB0056101-00 G S Compiler Support QLogic MPI supports a number of compilers. These include: PathScale Compiler Suite 3.0 and 3.1 PGI 5.2 , 6.0, and 7.1 Intel 9.x and 10.1 GNU gcc 3.3. x , 3.4. x , 4.0 and 4.1 compiler suites gfortran The PathScale C...
Page 21 - Hardware Installation
IB0056101-00 G 3-1 3 Step-by-Step Installation Checklist This section provides an overview of the hardware and software installation procedures. Detailed steps are found in Section 4 “ Hardware Installation ” and Section 5 “ Software Installation ”. Hardware Installation The following steps summariz...
Page 25 - Hardware Installation Requirements; Hardware
IB0056101-00 G 4-1 4 Hardware Installation This section lists the requirements and provides instructions for installing the QLogic InfiniPath Interconnect adapters. Instructions are included for the QLogic DDR PCI Express adapters, the QLE7240 and QLE7280; the QLogic InfiniPath PCIe adapter and PCIe...
Page 26 - Form Factors
4 – Hardware InstallationHardware Installation Requirements 4-2 IB0056101-00 G S Installation of the QLE7240, QLE7280, QLE7140, QHT7040, or QHT7140 in a 1U or 2U chassis requires the use of a riser card. See Figure 4-4 for an illustration of a PCI Express (PCIe) slot in a typical motherboard. See Fi...
Page 27 - Cabling and Switches
4 – Hardware Installation Hardware Installation Requirements IB0056101-00 G 4-3 A Cabling and Switches The cable installation uses a standard InfiniBand (IB) 4X cable. Any InfiniBand cable that has been qualified by the vendor should work. For SDR, the longest passive copper IB cable that QLogic has...
Page 28 - Optical Fibre Option; Configuring the BIOS
4 – Hardware InstallationHardware Installation Requirements 4-4 IB0056101-00 G S Optical Fibre Option The QLogic adapter also supports connection to the switch by means of optical fibres through optical media converters such as the EMCORE™ QT2400. Not all switches support these types of convertors. ...
Page 29 - Safety with Electricity; Verify the Package Contents
4 – Hardware Installation Safety with Electricity IB0056101-00 G 4-5 A Safety with Electricity Observe these guidelines and safety precautions when working around computer hardware and electrical equipment: Locate the power source shutoff for the computer room or lab where you are working. This is...
Page 32 - View; Unpacking the QLogic Adapter
4 – Hardware InstallationUnpacking Information 4-8 IB0056101-00 G S Figure 4-3. QLogic QHT7040/QHT7140 Full and Low Profile Cards with Riser, Top View Unpacking the QLogic Adapter Follow these steps when unpacking the QLogic adapter: 1. When unpacking, ground yourself before removing the QLogic adap...
Page 33 - with PCI Express Riser; Dual Adapter Installation; Installation Steps
4 – Hardware Installation Hardware Installation IB0056101-00 G 4-9 A Hardware Installation This section contains hardware installation instructions for the QLE7240, QLE7280, QLE7140, QHT7040, and QHT7140. Hardware Installation for QLE7240, QLE7280, or QLE7140 with PCI Express Riser Installation for ...
Page 34 - Figure 4-4. PCIe Slot in a Typical Motherboard
4 – Hardware InstallationHardware Installation 4-10 IB0056101-00 G S 4. Remove the cover screws and cover plate to expose the system’s motherboard. For specific instructions on how to do this, follow the hardware documentation that came with your system. 5. Locate the PCIe slot on your motherboard. ...
Page 35 - Figure 4-5. QLogic PCIe HCA Assembly with Riser Card
4 – Hardware Installation Hardware Installation IB0056101-00 G 4-11 A 9. Connect the QLogic adapter and PCIe riser card together, forming the assembly that you will insert into your motherboard. First, visually line up the adapter slot connector edge with the edge connector of the PCIe riser card (s...
Page 36 - Figure 4-6. Assembled PCIe HCA with Riser; Hardware Installation for QHT7140 with HTX Riser
4 – Hardware InstallationHardware Installation 4-12 IB0056101-00 G S 13. Insert the riser assembly into the motherboard’s PCIe slot, ensuring good contact. The QLogic adapter should now be parallel to the motherboard and about one inch above it (see Figure 4-6 ). Figure 4-6. Assembled PCIe HCA with ...
Page 38 - Figure 4-8. QLogic QHT7140 Adapter with Riser Card
4 – Hardware InstallationHardware Installation 4-14 IB0056101-00 G S 7. Remove the QLogic QHT7140 from the anti-static bag. 8. Locate the face plate on the connector edge of the card. 9. Connect the QLogic adapter and HTX riser card together, forming the assembly that you will insert into your mothe...
Page 39 - Figure 4-9. Assembled QHT7140 with Riser; Without a PCI Express Riser
4 – Hardware Installation Hardware Installation IB0056101-00 G 4-15 A 13. Insert the HT riser assembly into the motherboard’s HTX slot, ensuring good contact. The QLogic adapter should now be parallel to the motherboard and about one inch above it, as shown in Figure 4-9 . Figure 4-9. Assembled QHT7...
Page 40 - Hardware Installation for the QHT7140 Without an HTX Riser
4 – Hardware InstallationHardware Installation 4-16 IB0056101-00 G S Hardware Installation for the QHT7140 Without an HTX Riser Installing the QLogic QHT7140 without an HTX riser card requires a 3U or larger chassis. The card slot connectors on the QHT7140 fit into the HTX slot in a vertical install...
Page 41 - Figure 4-10. QHT7140 Without Riser Installed in a 3U Chassis; Switch Configuration and Monitoring
4 – Hardware Installation Switch Configuration and Monitoring IB0056101-00 G 4-17 A 8. Insert the card by pressing firmly and evenly on the top of the horizontal bracket and the top rear corner of the card simultaneously. The card should insert evenly into the slot. Be careful not to push, grab, or ...
Page 42 - Completing the Installation
4 – Hardware InstallationCompleting the Installation 4-18 IB0056101-00 G S The QLE7240, QLE7280, QLE7140, QHT7040, and QHT7140 adapters are all cabled the same way. To install the InfiniBand cables: 1. Check that you have removed the protector plugs from the cable connector ends. 2. Different vendor...
Page 43 - Cluster Setup; Types of Nodes in a Cluster Environment
IB0056101-00 G 5-1 5 Software Installation This section provides instructions for installing the InfiniPath and OpenFabrics software. The InfiniPath software includes drivers, protocol libraries, QLogic’s implementation of the MPI message passing standard, and example programs, including benchmarks....
Page 45 - Downloading and Unpacking the InfiniPath and; Choosing the Appropriate Download Files
5 – Software Installation Downloading and Unpacking the InfiniPath and OpenFabrics Software IB0056101-00 G 5-3 A Make sure that all previously existing (stock) OpenFabrics RPMs are uninstalled. See “Removing Software Packages” on page 5-33 for more information on uninstalling. If you are using RHE...
Page 47 - Check for Missing Files or RPMs
5 – Software Installation Downloading and Unpacking the InfiniPath and OpenFabrics Software IB0056101-00 G 5-5 A The RPMs are organized as follows: InfiniPath_license.txt,LEGAL.txt (top level)Documentation/ InfiniPath/ InfiniPath-Devel/ InfiniPath-MPI/ OpenFabrics/ OpenFabrics-Devel/OpenSM/OtherHCAs...
Page 48 - Installing the InfiniPath and OpenFabrics RPMs
5 – Software InstallationInstalling the InfiniPath and OpenFabrics RPMs 5-6 IB0056101-00 G S Installing the InfiniPath and OpenFabrics RPMs Linux distributions of InfiniPath and OpenFabrics software are installed from binary RPMs. RPM is a Linux packaging and installation tool used by Red Hat, SUSE,...
Page 49 - Choosing the RPMs to Install
5 – Software Installation Installing the InfiniPath and OpenFabrics RPMs IB0056101-00 G 5-7 A Choosing the RPMs to Install Although QLogic recommends that all RPMs are installed on all nodes, some are optional depending on which type of node is being used. To see which RPMs are required or optional ...
Page 51 - Installed Layout
5 – Software Installation Installed Layout IB0056101-00 G 5-9 A Installed Layout The default installed layout for the InfiniPath software is described in the following paragraphs. The InfiniPath shared libraries are installed in: /usr/lib for 32-bit applications/usr/lib64 for 64-bit applications MPI...
Page 52 - Starting the InfiniPath Service
5 – Software InstallationStarting the InfiniPath Service 5-10 IB0056101-00 G S Other OFED-installed modules may also be in this directory; these are also renamed if found during the install process. Starting the InfiniPath Service If this is the initial installation of InfiniPath (see “Lockable Memo...
Page 54 - Configuration on Red Hat
5 – Software InstallationConfiguring the InfiniPath Drivers 5-12 IB0056101-00 G S Configuring the ib_ipath Driver The ib_ipath module provides both low-level InfiniPath support and management functions for OpenFabrics protocols. The ib_ipath driver has several configuration variables that set reserv...
Page 56 - Configuration on SLES
5 – Software InstallationConfiguring the InfiniPath Drivers 5-14 IB0056101-00 G S 4. Check whether the Ethernet driver has been loaded with: $ lsmod | grep ipath_ether 5. Verify that the driver is up with: $ ifconfig -a ipath_ether Configuration on SLES The following procedure causes the ipath_ether...
Page 58 - OpenFabrics Drivers and Services Configuration
5 – Software InstallationOpenFabrics Drivers and Services Configuration and Startup 5-16 IB0056101-00 G S If you are using Dynamic Host Configuration Protocol (DHCP), add these lines to the file: STARTMODE=onbootBOOTPROTO=dhcpNAME=’InfiniPath Network Card’_nm_name=eth-id-$MAC Proceed to Step 6 . If ...
Page 59 - Configuring the IPoIB Network Interface
5 – Software Installation OpenFabrics Drivers and Services Configuration and Startup IB0056101-00 G 5-17 A Configuring the IPoIB Network Interface The following instructions show you how to manually configure your OpenFabrics IPoIB network interface. This example assumes that you are using sh or bas...
Page 60 - Configuring and Administering the VNIC Interface
5 – Software InstallationOpenFabrics Drivers and Services Configuration and Startup 5-18 IB0056101-00 G S Configuring and Administering the VNIC Interface The VirtualNIC (VNIC) Upper Layer Protocol (ULP) works in conjunction with firmware running on Virtual Input/Output (VIO) hardware such as the Si...
Page 66 - SRP; Using OFED SRP
5 – Software InstallationOpenFabrics Drivers and Services Configuration and Startup 5-24 IB0056101-00 G S For example: # Use the UPDN algorithm instead of the Min Hop algorithm.OPTIONS="-R updn" SRP SRP stands for SCSI RDMA Protocol. It was originally intended to allow the SCSI protocol to r...
Page 68 - MPI over uDAPL; Other Configuration: Changing the MTU Size; Starting and Stopping the InfiniPath Software
5 – Software InstallationOther Configuration: Changing the MTU Size 5-26 IB0056101-00 G S MPI over uDAPL Some MPI implementations, such as Intel MPI and HP-MPI, can be run over uDAPL. uDAPL is the user mode version of the Direct Access Provider Library (DAPL). Examples of these types of MPI implemen...
Page 70 - Rebuilding or Reinstalling Drivers After a Kernel
5 – Software InstallationRebuilding or Reinstalling Drivers After a Kernel Upgrade 5-28 IB0056101-00 G S An equivalent way to restart infinipath this is to use same sequence as above, except use the restart command instead of start and stop : # /etc/init.d/opensmd stop # ifdown eth2 # /etc/init.d/in...
Page 71 - Rebuilding or Reinstalling Drivers if a Different
5 – Software Installation Rebuilding or Reinstalling Drivers if a Different Kernel is Installed IB0056101-00 G 5-29 A Rebuilding or Reinstalling Drivers if a Different Kernel is Installed Installation of the InfiniPath driver RPM ( infinipath-kernel-2.2-xxx-yyy ) builds kernel modules for the curren...
Page 72 - Adapter Settings
5 – Software InstallationAdapter Settings 5-30 IB0056101-00 G S Adapter Settings The following adapter settings can be adjusted for better performance. Use taskset to tune CPU affinity on Opteron systems with the QLE7240, QLE7280, and QLE7140. Latency will be slightly lower for the Opteron socket ...
Page 73 - Customer Acceptance Utility
5 – Software Installation Customer Acceptance Utility IB0056101-00 G 5-31 A Use a PCIe MaxReadRequest size of at least 512 bytes with the QLE7240 and QLE7280. QLE7240 and QLE7280 adapters can support sizes from 128 bytes to 4096 byte in powers of two. This value is typically set by the BIOS. Use...
Page 75 - Removing Software Packages; Uninstalling InfiniPath and OpenFabrics RPMs; Uninstalling OFED 1.3 Software; Additional Installation Instructions
5 – Software Installation Removing Software Packages IB0056101-00 G 5-33 A Removing Software Packages This section provides instructions for uninstalling or downgrading the InfiniPath and OpenFabrics software. Uninstalling InfiniPath and OpenFabrics RPMs To uninstall the InfiniPath software packages...
Page 77 - Installing on an Unsupported Distribution
5 – Software Installation Additional Installation Instructions IB0056101-00 G 5-35 A For example, install all RPMs that relate to QLogic MPI in /usr/mpi/qlogic . Leave all remaining InfiniPath libraries and tools in their default installation location ( /usr) . This approach leaves InfiniPath librar...
Page 78 - Managing and Installing Software Using Rocks
5 – Software InstallationAdditional Installation Instructions 5-36 IB0056101-00 G S The following example installation is for a Red Hat Enterprise Linux 4 Update 4 compatible kernel, where the /etc/redhat-release file indicates another distribution. If you are a bash or sh user, type: # export IPATH...
Page 79 - Installing Rocks and InfiniPath RPMs
5 – Software Installation Additional Installation Instructions IB0056101-00 G 5-37 A Rocks is a way to manage the kickstart automated installation method created by Red Hat. By using the Rocks conventions, the installation process can be automated for clusters of any size. A Roll is an extension to ...
Page 83 - Further Information on Rocks and kickstart
5 – Software Installation Additional Installation Instructions IB0056101-00 G 5-41 A 6. The completion of the installation is done using the extend-compute.xml file. More specific instructions for completing the install process can be found under the Documentation link on the Rocks web site: http://...
Page 85 - Hardware Issues; Node Spontaneously Reboots; BIOS Settings
IB0056101-00 G A-1 A Installation Troubleshooting The following sections contain information about issues that may occur during installation. Some of this material is repeated in the Troubleshooting appendix of the QLogic HCA and InfiniPath Software User Guide . Many programs and files are available...
Page 86 - MTRR Mapping and Write Combining; Edit BIOS Settings to Fix MTRR Issues
A – Installation TroubleshootingBIOS Settings A-2 IB0056101-00 G S MTRR Mapping and Write Combining MTRR is used by the InfiniPath driver to enable write combining to the QLogic on-chip transmit buffers. Write combining improves write bandwidth to the QLogic chip by writing multiple words in a singl...
Page 87 - Issue with Supermicro; Software Installation Issues
A – Installation Troubleshooting Software Installation Issues IB0056101-00 G A-3 A Some BIOS’ do not have the MTRR mapping option. It may have a different name, depending on the chipset, vendor, BIOS, or other factors. For example, it is sometimes referred to as 32 bit memory hole . This setting mus...
Page 88 - Version Number Conflict with; OpenFabrics Dependencies; OpenFabrics Library Dependencies
A – Installation TroubleshootingVersion Number Conflict with opensm-* on RHEL5 Systems A-4 IB0056101-00 G S Version Number Conflict with opensm-* on RHEL5 Systems The older opensm-* packages that come with the RHEL5 distribution have a version number (3) that is greater the InfiniPath version number...
Page 89 - Missing Kernel RPM Errors
A – Installation Troubleshooting Version Number Conflict with opensm-* on RHEL5 Systems IB0056101-00 G A-5 A Missing Kernel RPM Errors Install the kernel-source , kernel-devel , and, if using an older release, kernel-smp-devel RPMs for your distribution before installing the InfiniPath RPMs, as ther...
Page 90 - Resolving Conflicts; mpirun
A – Installation TroubleshootingVersion Number Conflict with opensm-* on RHEL5 Systems A-6 IB0056101-00 G S Resolving Conflicts Occasionally, conflicts may arise when trying to install "on top of" an existing set of files that may come from a different set of RPMs. For example, if you instal...
Page 91 - Lockable Memory Error on Initial Installation of InfiniPath
A – Installation Troubleshooting Version Number Conflict with opensm-* on RHEL5 Systems IB0056101-00 G A-7 A In newer distributions, glibc is an RPM name. The 32-bit glibc is named similarly to: glibc-2.3.4-2.i686.rpm OR glibc-2.3.4-2.i386.rpm Check your distribution for the exact RPM name. ifup on ...
Page 95 - RPM Descriptions; InfiniPath and OpenFabrics RPMs; Different Nodes May Use Different RPMs; RPM Version Numbers and Identifiers
IB0056101-00 G C-1 C RPM Descriptions The following sections contain detailed descriptions of the RPMs for InfiniPath and OpenFabrics software. InfiniPath and OpenFabrics RPMs For ease of installation, QLogic recommends that all RPMs are installed on all nodes. However, some RPMs are optional. Since...
Page 96 - RPM Organization
C – RPM DescriptionsRPM Organization C-2 IB0056101-00 G S Non-InfiniPath components may also have their own version number: mvapich_gcc-2.2-33597.832.1_0_0.sles10_qlc.x86_64.rpm 1_0_0 is the 1.0.0 build for mvapich . In all of the tables in this appendix, the build identifier is xxx and the distribu...
Page 98 - OpenFabrics RPMs
C – RPM DescriptionsOpenFabrics RPMs C-4 IB0056101-00 G S The InfiniPath-MPI/RPMs are listed in Table C-4 . OpenFabrics RPMs OpenFabrics and OpenSM are optional components. For ease of installation, QLogic recommends that all of the OpenFabrics RPMs listed in Table C-5 and Table C-6 be installed on ...
Page 107 - Index
IB0056101-00 G Index-1 Index A ACPI 4-4, A-1Adapter, see HCA B BIOS configuring 4-4settings A-1settings to fix MTRR issues A-2 C -c 5-32 Cables supported 4-3Compiler support 2-4Configuration files B-1 ib_ipath 5-12 ipath_ether on Fedora, RHEL4, RHEL5 5-12 ipath_ether on SLES 5-14 OpenSM 5-23VNIC 5-1...
