QLogic IB0056101-00 G - Manuals

IB0056101-00 G Page v QLogic HCA and InfiniPath ® Software Install Guide Version 2.2 A Starting and Stopping the InfiniPath Software . . . . . . . . . . . . . . . . . . . . . . . 5-26 Rebuilding or Reinstalling Drivers After a Kernel Upgrade . . . . . . . . . . . . . 5-28 Rebuilding or Reinstalling ...

Page vi IB0056101-00 G QLogic HCA and InfiniPath ® Software Install Guide Version 2.2 S B Configuration Files C RPM Descriptions InfiniPath and OpenFabrics RPMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-1 Different Nodes May Use Different RPMs . . . . . . . . . . . . . . . . ...

IB0056101-00 G Page vii QLogic HCA and InfiniPath ® Software Install Guide Version 2.2 A List of Figures Figure Page 4-1 QLogic QLE7280 with IBA7220 ASIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-7 4-2 QLogic QLE7140 Card with Riser, Top View . . . . . . . . . . . . . ....

Page viii IB0056101-00 G QLogic HCA and InfiniPath ® Software Install Guide Version 2.2 S Notes

IB0056101-00 G 1-1 1 Introduction This chapter describes the contents, intended audience, and organization of the QLogic HCA and InfiniPath Software Install Guide. The QLogic HCA and InfiniPath Software Install Guide contains instructions for installing the QLogic Host Channel Adapters (HCAs) and th...

1 – IntroductionOverview 1-2 IB0056101-00 G S „ Section 3 , Step-by-Step Installation Checklist, provides a high-level overview of the hardware and software installation procedures. „ Section 4 , Hardware Installation, includes instructions for installing the QLogic QLE7140, QLE7240, QLE7280, QHT704...

1 – Introduction Interoperability IB0056101-00 G 1-3 A InfiniPath OpenFabrics software is interoperable with other vendors’ InfiniBand Host Channel Adapters (HCAs) running compatible OpenFabrics releases. There are several options for subnet management in your cluster: „ Use the embedded Subnet Mana...

1 – IntroductionDocumentation 1-4 IB0056101-00 G S Documentation The product documentation includes: „ The QLogic HCA and InfiniPath Software Install Guide „ The QLogic HCA and InfiniPath Software User Guide „ The QLogic FastFabric Users Guide (for information on QLogic InfiniServ Tools) „ The OFED+...

1 – Introduction Contact Information IB0056101-00 G 1-5 A Contact Information Support Headquarters QLogic Corporation4601 Dean Lakes BlvdShakopee, MN 55379USA QLogic Web Site www.qlogic.com Technical Support Web Site support.qlogic.com Technical Support Email [email protected] Technical Training Em...

IB0056101-00 G 2-1 2 Feature Overview This section contains the features for this release, the supported QLogic adapter models, supported distributions and kernels, and a list of the software components. What’s New in this Release This release adds support for the QLE7240 and QLE7280 InfiniBand DDR ...

2 – Feature OverviewNew Features 2-2 IB0056101-00 G S New Features The following features are new to the 2.2 release: „ Expanded MPI scalability enhancements for PCI Express have been added. On the QLE7240 and QLE7280, up to 16 dedicated hardware contexts per node are available. The QHT7140 has eigh...

2 – Feature Overview Supported Distributions and Kernels IB0056101-00 G 2-3 A „ A subset of the QLogic InfiniBand Fabric Suite, the enablement tools, are offered with this release. „ Two separate SCSI RDMA Protocol (SRP) modules are provided: the standard OpenFabrics (OFED) SRP, and QLogic SRP. „ QL...

2 – Feature OverviewCompiler Support 2-4 IB0056101-00 G S Compiler Support QLogic MPI supports a number of compilers. These include: „ PathScale Compiler Suite 3.0 and 3.1 „ PGI 5.2 , 6.0, and 7.1 „ Intel 9.x and 10.1 „ GNU gcc 3.3. x , 3.4. x , 4.0 and 4.1 compiler suites „ gfortran The PathScale C...

IB0056101-00 G 3-1 3 Step-by-Step Installation Checklist This section provides an overview of the hardware and software installation procedures. Detailed steps are found in Section 4 “ Hardware Installation ” and Section 5 “ Software Installation ”. Hardware Installation The following steps summariz...

IB0056101-00 G 4-1 4 Hardware Installation This section lists the requirements and provides instructions for installing the QLogic InfiniPath Interconnect adapters. Instructions are included for the QLogic DDR PCI Express adapters, the QLE7240 and QLE7280; the QLogic InfiniPath PCIe adapter and PCIe...

4 – Hardware InstallationHardware Installation Requirements 4-2 IB0056101-00 G S Installation of the QLE7240, QLE7280, QLE7140, QHT7040, or QHT7140 in a 1U or 2U chassis requires the use of a riser card. See Figure 4-4 for an illustration of a PCI Express (PCIe) slot in a typical motherboard. See Fi...

4 – Hardware Installation Hardware Installation Requirements IB0056101-00 G 4-3 A Cabling and Switches The cable installation uses a standard InfiniBand (IB) 4X cable. Any InfiniBand cable that has been qualified by the vendor should work. For SDR, the longest passive copper IB cable that QLogic has...

4 – Hardware InstallationHardware Installation Requirements 4-4 IB0056101-00 G S Optical Fibre Option The QLogic adapter also supports connection to the switch by means of optical fibres through optical media converters such as the EMCORE™ QT2400. Not all switches support these types of convertors. ...

4 – Hardware Installation Safety with Electricity IB0056101-00 G 4-5 A Safety with Electricity Observe these guidelines and safety precautions when working around computer hardware and electrical equipment: „ Locate the power source shutoff for the computer room or lab where you are working. This is...

4 – Hardware InstallationUnpacking Information 4-8 IB0056101-00 G S Figure 4-3. QLogic QHT7040/QHT7140 Full and Low Profile Cards with Riser, Top View Unpacking the QLogic Adapter Follow these steps when unpacking the QLogic adapter: 1. When unpacking, ground yourself before removing the QLogic adap...

4 – Hardware Installation Hardware Installation IB0056101-00 G 4-9 A Hardware Installation This section contains hardware installation instructions for the QLE7240, QLE7280, QLE7140, QHT7040, and QHT7140. Hardware Installation for QLE7240, QLE7280, or QLE7140 with PCI Express Riser Installation for ...

4 – Hardware InstallationHardware Installation 4-10 IB0056101-00 G S 4. Remove the cover screws and cover plate to expose the system’s motherboard. For specific instructions on how to do this, follow the hardware documentation that came with your system. 5. Locate the PCIe slot on your motherboard. ...

4 – Hardware Installation Hardware Installation IB0056101-00 G 4-11 A 9. Connect the QLogic adapter and PCIe riser card together, forming the assembly that you will insert into your motherboard. First, visually line up the adapter slot connector edge with the edge connector of the PCIe riser card (s...

4 – Hardware InstallationHardware Installation 4-12 IB0056101-00 G S 13. Insert the riser assembly into the motherboard’s PCIe slot, ensuring good contact. The QLogic adapter should now be parallel to the motherboard and about one inch above it (see Figure 4-6 ). Figure 4-6. Assembled PCIe HCA with ...

4 – Hardware InstallationHardware Installation 4-14 IB0056101-00 G S 7. Remove the QLogic QHT7140 from the anti-static bag. 8. Locate the face plate on the connector edge of the card. 9. Connect the QLogic adapter and HTX riser card together, forming the assembly that you will insert into your mothe...

4 – Hardware Installation Hardware Installation IB0056101-00 G 4-15 A 13. Insert the HT riser assembly into the motherboard’s HTX slot, ensuring good contact. The QLogic adapter should now be parallel to the motherboard and about one inch above it, as shown in Figure 4-9 . Figure 4-9. Assembled QHT7...

4 – Hardware InstallationHardware Installation 4-16 IB0056101-00 G S Hardware Installation for the QHT7140 Without an HTX Riser Installing the QLogic QHT7140 without an HTX riser card requires a 3U or larger chassis. The card slot connectors on the QHT7140 fit into the HTX slot in a vertical install...

4 – Hardware Installation Switch Configuration and Monitoring IB0056101-00 G 4-17 A 8. Insert the card by pressing firmly and evenly on the top of the horizontal bracket and the top rear corner of the card simultaneously. The card should insert evenly into the slot. Be careful not to push, grab, or ...

4 – Hardware InstallationCompleting the Installation 4-18 IB0056101-00 G S The QLE7240, QLE7280, QLE7140, QHT7040, and QHT7140 adapters are all cabled the same way. To install the InfiniBand cables: 1. Check that you have removed the protector plugs from the cable connector ends. 2. Different vendor...

IB0056101-00 G 5-1 5 Software Installation This section provides instructions for installing the InfiniPath and OpenFabrics software. The InfiniPath software includes drivers, protocol libraries, QLogic’s implementation of the MPI message passing standard, and example programs, including benchmarks....

5 – Software Installation Downloading and Unpacking the InfiniPath and OpenFabrics Software IB0056101-00 G 5-3 A „ Make sure that all previously existing (stock) OpenFabrics RPMs are uninstalled. See “Removing Software Packages” on page 5-33 for more information on uninstalling. If you are using RHE...

5 – Software Installation Downloading and Unpacking the InfiniPath and OpenFabrics Software IB0056101-00 G 5-5 A The RPMs are organized as follows: InfiniPath_license.txt,LEGAL.txt (top level)Documentation/ InfiniPath/ InfiniPath-Devel/ InfiniPath-MPI/ OpenFabrics/ OpenFabrics-Devel/OpenSM/OtherHCAs...

5 – Software InstallationInstalling the InfiniPath and OpenFabrics RPMs 5-6 IB0056101-00 G S Installing the InfiniPath and OpenFabrics RPMs Linux distributions of InfiniPath and OpenFabrics software are installed from binary RPMs. RPM is a Linux packaging and installation tool used by Red Hat, SUSE,...

5 – Software Installation Installing the InfiniPath and OpenFabrics RPMs IB0056101-00 G 5-7 A Choosing the RPMs to Install Although QLogic recommends that all RPMs are installed on all nodes, some are optional depending on which type of node is being used. To see which RPMs are required or optional ...

5 – Software Installation Installed Layout IB0056101-00 G 5-9 A Installed Layout The default installed layout for the InfiniPath software is described in the following paragraphs. The InfiniPath shared libraries are installed in: /usr/lib for 32-bit applications/usr/lib64 for 64-bit applications MPI...

5 – Software InstallationStarting the InfiniPath Service 5-10 IB0056101-00 G S Other OFED-installed modules may also be in this directory; these are also renamed if found during the install process. Starting the InfiniPath Service If this is the initial installation of InfiniPath (see “Lockable Memo...

5 – Software InstallationConfiguring the InfiniPath Drivers 5-12 IB0056101-00 G S Configuring the ib_ipath Driver The ib_ipath module provides both low-level InfiniPath support and management functions for OpenFabrics protocols. The ib_ipath driver has several configuration variables that set reserv...

5 – Software InstallationConfiguring the InfiniPath Drivers 5-14 IB0056101-00 G S 4. Check whether the Ethernet driver has been loaded with: $ lsmod | grep ipath_ether 5. Verify that the driver is up with: $ ifconfig -a ipath_ether Configuration on SLES The following procedure causes the ipath_ether...

5 – Software InstallationOpenFabrics Drivers and Services Configuration and Startup 5-16 IB0056101-00 G S If you are using Dynamic Host Configuration Protocol (DHCP), add these lines to the file: STARTMODE=onbootBOOTPROTO=dhcpNAME=’InfiniPath Network Card’_nm_name=eth-id-$MAC Proceed to Step 6 . If ...

5 – Software Installation OpenFabrics Drivers and Services Configuration and Startup IB0056101-00 G 5-17 A Configuring the IPoIB Network Interface The following instructions show you how to manually configure your OpenFabrics IPoIB network interface. This example assumes that you are using sh or bas...

5 – Software InstallationOpenFabrics Drivers and Services Configuration and Startup 5-18 IB0056101-00 G S Configuring and Administering the VNIC Interface The VirtualNIC (VNIC) Upper Layer Protocol (ULP) works in conjunction with firmware running on Virtual Input/Output (VIO) hardware such as the Si...

5 – Software InstallationOpenFabrics Drivers and Services Configuration and Startup 5-24 IB0056101-00 G S For example: # Use the UPDN algorithm instead of the Min Hop algorithm.OPTIONS="-R updn" SRP SRP stands for SCSI RDMA Protocol. It was originally intended to allow the SCSI protocol to r...

5 – Software InstallationOther Configuration: Changing the MTU Size 5-26 IB0056101-00 G S MPI over uDAPL Some MPI implementations, such as Intel MPI and HP-MPI, can be run over uDAPL. uDAPL is the user mode version of the Direct Access Provider Library (DAPL). Examples of these types of MPI implemen...

5 – Software InstallationRebuilding or Reinstalling Drivers After a Kernel Upgrade 5-28 IB0056101-00 G S An equivalent way to restart infinipath this is to use same sequence as above, except use the restart command instead of start and stop : # /etc/init.d/opensmd stop # ifdown eth2 # /etc/init.d/in...

5 – Software Installation Rebuilding or Reinstalling Drivers if a Different Kernel is Installed IB0056101-00 G 5-29 A Rebuilding or Reinstalling Drivers if a Different Kernel is Installed Installation of the InfiniPath driver RPM ( infinipath-kernel-2.2-xxx-yyy ) builds kernel modules for the curren...

5 – Software InstallationAdapter Settings 5-30 IB0056101-00 G S Adapter Settings The following adapter settings can be adjusted for better performance. „ Use taskset to tune CPU affinity on Opteron systems with the QLE7240, QLE7280, and QLE7140. Latency will be slightly lower for the Opteron socket ...

5 – Software Installation Customer Acceptance Utility IB0056101-00 G 5-31 A „ Use a PCIe MaxReadRequest size of at least 512 bytes with the QLE7240 and QLE7280. QLE7240 and QLE7280 adapters can support sizes from 128 bytes to 4096 byte in powers of two. This value is typically set by the BIOS. „ Use...

5 – Software Installation Removing Software Packages IB0056101-00 G 5-33 A Removing Software Packages This section provides instructions for uninstalling or downgrading the InfiniPath and OpenFabrics software. Uninstalling InfiniPath and OpenFabrics RPMs To uninstall the InfiniPath software packages...

5 – Software Installation Additional Installation Instructions IB0056101-00 G 5-35 A For example, install all RPMs that relate to QLogic MPI in /usr/mpi/qlogic . Leave all remaining InfiniPath libraries and tools in their default installation location ( /usr) . This approach leaves InfiniPath librar...

5 – Software InstallationAdditional Installation Instructions 5-36 IB0056101-00 G S The following example installation is for a Red Hat Enterprise Linux 4 Update 4 compatible kernel, where the /etc/redhat-release file indicates another distribution. If you are a bash or sh user, type: # export IPATH...

5 – Software Installation Additional Installation Instructions IB0056101-00 G 5-37 A Rocks is a way to manage the kickstart automated installation method created by Red Hat. By using the Rocks conventions, the installation process can be automated for clusters of any size. A Roll is an extension to ...

5 – Software Installation Additional Installation Instructions IB0056101-00 G 5-41 A 6. The completion of the installation is done using the extend-compute.xml file. More specific instructions for completing the install process can be found under the Documentation link on the Rocks web site: http://...

IB0056101-00 G A-1 A Installation Troubleshooting The following sections contain information about issues that may occur during installation. Some of this material is repeated in the Troubleshooting appendix of the QLogic HCA and InfiniPath Software User Guide . Many programs and files are available...

A – Installation TroubleshootingBIOS Settings A-2 IB0056101-00 G S MTRR Mapping and Write Combining MTRR is used by the InfiniPath driver to enable write combining to the QLogic on-chip transmit buffers. Write combining improves write bandwidth to the QLogic chip by writing multiple words in a singl...

A – Installation Troubleshooting Software Installation Issues IB0056101-00 G A-3 A Some BIOS’ do not have the MTRR mapping option. It may have a different name, depending on the chipset, vendor, BIOS, or other factors. For example, it is sometimes referred to as 32 bit memory hole . This setting mus...

A – Installation TroubleshootingVersion Number Conflict with opensm-* on RHEL5 Systems A-4 IB0056101-00 G S Version Number Conflict with opensm-* on RHEL5 Systems The older opensm-* packages that come with the RHEL5 distribution have a version number (3) that is greater the InfiniPath version number...

A – Installation Troubleshooting Version Number Conflict with opensm-* on RHEL5 Systems IB0056101-00 G A-5 A Missing Kernel RPM Errors Install the kernel-source , kernel-devel , and, if using an older release, kernel-smp-devel RPMs for your distribution before installing the InfiniPath RPMs, as ther...

A – Installation TroubleshootingVersion Number Conflict with opensm-* on RHEL5 Systems A-6 IB0056101-00 G S Resolving Conflicts Occasionally, conflicts may arise when trying to install "on top of" an existing set of files that may come from a different set of RPMs. For example, if you instal...

A – Installation Troubleshooting Version Number Conflict with opensm-* on RHEL5 Systems IB0056101-00 G A-7 A In newer distributions, glibc is an RPM name. The 32-bit glibc is named similarly to: glibc-2.3.4-2.i686.rpm OR glibc-2.3.4-2.i386.rpm Check your distribution for the exact RPM name. ifup on ...

IB0056101-00 G C-1 C RPM Descriptions The following sections contain detailed descriptions of the RPMs for InfiniPath and OpenFabrics software. InfiniPath and OpenFabrics RPMs For ease of installation, QLogic recommends that all RPMs are installed on all nodes. However, some RPMs are optional. Since...

C – RPM DescriptionsRPM Organization C-2 IB0056101-00 G S Non-InfiniPath components may also have their own version number: mvapich_gcc-2.2-33597.832.1_0_0.sles10_qlc.x86_64.rpm 1_0_0 is the 1.0.0 build for mvapich . In all of the tables in this appendix, the build identifier is xxx and the distribu...

C – RPM DescriptionsOpenFabrics RPMs C-4 IB0056101-00 G S The InfiniPath-MPI/RPMs are listed in Table C-4 . OpenFabrics RPMs OpenFabrics and OpenSM are optional components. For ease of installation, QLogic recommends that all of the OpenFabrics RPMs listed in Table C-5 and Table C-6 be installed on ...

IB0056101-00 G Index-1 Index A ACPI 4-4, A-1Adapter, see HCA B BIOS configuring 4-4settings A-1settings to fix MTRR issues A-2 C -c 5-32 Cables supported 4-3Compiler support 2-4Configuration files B-1 ib_ipath 5-12 ipath_ether on Fedora, RHEL4, RHEL5 5-12 ipath_ether on SLES 5-14 OpenSM 5-23VNIC 5-1...