QLogic OFED - Manuals

Draft x Preliminary IB0056101-00 G.02 QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 S Hardware Installation for QLE7240, QLE7280, or QLE7140 with PCI Express Riser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-9 Dual Adapter Install...

Draft IB0056101-00 G.02 Preliminary xi QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 A Uninstalling Software with Rocks or Platform OCS . . . . . . . . . . . . . . . 5-25 Downgrading RPMs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-25 6 ...

Draft xii Preliminary IB0056101-00 G.02 QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 S ibsrpdm Command Hangs When Two HCAs are Installed but Only Unit 1 is Connected to the Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-5 Outdated ipath_ethe...

Draft IB0056101-00 G.02 Preliminary xiii QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 A List of Figures Figure Page 4-1 QLogic QLE7280 with IBA7220 ASIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-7 4-2 QLogic QLE7140 Card with Riser, Top View...

Draft IB0056101-00 G.02 Preliminary 1-1 1 Introduction This chapter describes the contents, intended audience, and organization of the QLogic HCA and QLogic OFED Software Install Guide . The QLogic HCA and QLogic OFED Software Install Guide contains instructions for installing the QLogic Host Channe...

Draft IntroductionOverview 1-2 Preliminary IB0056101-00 G.02 S „ Section 3, Step-by-Step Installation Checklist , provides a high-level overview of the hardware and software installation procedures. „ Section 4, Hardware Installation , includes instructions for installing the QLogic QLE7140, QLE7240...

Draft Introduction Interoperability IB0056101-00 G.02 Preliminary 1-3 A QLogic OFED OpenFabrics software is interoperable with other vendors’ InfiniBand Host Channel Adapters (HCAs) running compatible OpenFabrics releases. There are several options for subnet management in your cluster: „ Use the em...

Draft IntroductionDocumentation 1-4 Preliminary IB0056101-00 G.02 S Documentation The product documentation includes: „ The QLogic HCA and QLogic OFED Software Install Guide „ The QLogic HCA and QLogic OFED Software Users Guide „ The QLogic InfiniBand Software Install Guide „ The QLogic ULP and Tool...

Draft Introduction Contact Information IB0056101-00 G.02 Preliminary 1-5 A Contact Information Support Headquarters QLogic Corporation4601 Dean Lakes BlvdShakopee, MN 55379USA QLogic Web Site www.qlogic.com Technical Support Web Site support.qlogic.com Technical Support Email [email protected] Tech...

Draft IB0056101-00 G.02 Preliminary 2-1 2 Feature Overview This section contains the features for this release, the supported QLogic adapter models, supported distributions and kernels, and a list of the software components. Features The QLogic OFED 1.4 software release contains the complete OFED 1....

Draft Feature OverviewFeatures 2-2 Preliminary IB0056101-00 G.02 S „ MVAPICH and OpenMPI compiled with newer versions of each of four different compilers (GCC, PGI, Intel and PathScale) are available. „ The QLogic InfiniBand Fabric Suite (IFS) is available separately for purchase. It includes FastFa...

Draft Feature OverviewSoftware Components 2-4 Preliminary IB0056101-00 G.02 S Software Components This release includes all of OFED 1.4 with enhancements (QLogic OFED 1.4), including a new version of the VNIC tools and driver, and support for the QHT7xxx and QLE7xxx adapters. The software includes t...

Draft IB0056101-00 G.02 Preliminary 3-1 3 Step-by-Step Installation Checklist This section provides an overview of the hardware and software installation procedures. Detailed steps are found in Section 4 “ Hardware Installation ” and Section 5 “ Software Installation ”. Hardware Installation The fol...

Draft IB0056101-00 G.02 Preliminary 4-1 4 Hardware Installation This section lists the requirements and provides instructions for installing the QLogic InfiniPath Interconnect adapters. Instructions are included for the QLogic DDR PCI Express adapters, the QLE7240 and QLE7280; the QLogic PCIe adapte...

Draft Hardware InstallationHardware Installation Requirements 4-2 Preliminary IB0056101-00 G.02 S Installation of the QLE7240, QLE7280, QLE7140, QHT7040, or QHT7140 in a 1U or 2U chassis requires the use of a riser card. See Figure 4-4 for an illustration of a PCI Express (PCIe) slot in a typical mo...

Draft Hardware Installation Hardware Installation Requirements IB0056101-00 G.02 Preliminary 4-3 A Cabling and Switches The cable installation uses a standard InfiniBand (IB) 4X cable. Any InfiniBand cable that has been qualified by the vendor should work. For SDR, the longest passive copper IB cabl...

Draft Hardware InstallationHardware Installation Requirements 4-4 Preliminary IB0056101-00 G.02 S Optical Fibre Option The QLogic adapter also supports connection to the switch by means of optical fibres through optical media converters such as the EMCORE™ QT2400. Not all switches support these type...

Draft Hardware Installation Safety with Electricity IB0056101-00 G.02 Preliminary 4-5 A Safety with Electricity Observe these guidelines and safety precautions when working around computer hardware and electrical equipment: „ Locate the power source shutoff for the computer room or lab where you are...

Draft Hardware InstallationUnpacking Information 4-8 Preliminary IB0056101-00 G.02 S Figure 4-3. QLogic QHT7040/QHT7140 Full and Low Profile Cards with Riser, Top View Unpacking the QLogic Adapter Follow these steps when unpacking the QLogic adapter: 1. When unpacking, ground yourself before removin...

Draft Hardware InstallationHardware Installation IB0056101-00 G.02 Preliminary 4-9 A Hardware Installation This section contains hardware installation instructions for the QLE7240, QLE7280, QLE7140, QHT7040, and QHT7140. Hardware Installation for QLE7240, QLE7280, or QLE7140 with PCI Express Riser I...

Draft Hardware InstallationHardware Installation 4-12 Preliminary IB0056101-00 G.02 S 13. Insert the riser assembly into the motherboard’s PCIe slot, ensuring good contact. The QLogic adapter should now be parallel to the motherboard and about one inch above it (see Figure 4-6 ). Figure 4-6. Assembl...

Draft Hardware InstallationHardware Installation IB0056101-00 G.02 Preliminary 4-15 A 13. Insert the HT riser assembly into the motherboard’s HTX slot, ensuring good contact. The QLogic adapter should now be parallel to the motherboard and about one inch above it, as shown in Figure 4-9 . Figure 4-9...

Draft Hardware InstallationHardware Installation 4-16 Preliminary IB0056101-00 G.02 S Hardware Installation for the QHT7140 Without an HTX Riser Installing the QLogic QHT7140 without an HTX riser card requires a 3U or larger chassis. The card slot connectors on the QHT7140 fit into the HTX slot in a...

Draft Hardware Installation Switch Configuration and Monitoring IB0056101-00 G.02 Preliminary 4-17 A 8. Insert the card by pressing firmly and evenly on the top of the horizontal bracket and the top rear corner of the card simultaneously. The card should insert evenly into the slot. Be careful not t...

Draft Hardware InstallationCompleting the Installation 4-18 Preliminary IB0056101-00 G.02 S The QLE7240, QLE7280, QLE7140, QHT7040, and QHT7140 adapters are all cabled the same way. To install the InfiniBand cables: 1. Check that you have removed the protector plugs from the cable connector ends. 2....

Draft IB0056101-00 G.02 Preliminary 5-1 5 Software Installation This section provides instructions for installing QLogic OFED 1.4, which includes QLogic InfiniPath and the OpenFabrics software. The software includes drivers, protocol libraries, QLogic’s implementation of the MPI message passing stan...

Draft Software InstallationCluster Setup 5-2 Preliminary IB0056101-00 G.02 S Distribution Identifiers Distribution identifiers for this release are listed in the table below. They are used in file naming conventions. Table 5-1. InfiniPath/OpenFabrics Supported Distributions and Kernels Distribution ...

Draft Software Installation Compiler Support IB0056101-00 G.02 Preliminary 5-3 A Compiler Support QLogic MPI supports use of a number of compilers. These include: „ GNU gcc 3.3. x , 3.4. x , 4.0, 4.1, 4.2.x, and 4.3.x compiler suites „ PathScale Compiler Suite 3.0, 3.1 and 3.2 „ PGI 5.2, 6.0. 7.1, 7...

Draft Software InstallationChoose the Appropriate Download Files 5-4 Preliminary IB0056101-00 G.02 S Some operating system packages are required for OpenFabrics; they are listed in Table 5-3 . There are also OS package requirements for some specific components in QLogic OFED. Choose the Appropriate ...

Draft Software Installation Install QLogicIB-Basic with the Installer Tool IB0056101-00 G.02 Preliminary 5-7 A Install QLogicIB-Basic with the Installer Tool The Installer tool has a Text User Interface (TUI) that affords easy installation of the software. Use this method if you have downloaded the ...

Draft Software Installation About rpm Installation IB0056101-00 G.02 Preliminary 5-13 A About rpm Installation Linux distributions of QLogic OFED (InfiniPath and OpenFabrics) software can be installed from binary RPMs. RPM is a Linux packaging and installation tool used by Red Hat, SUSE, and CentOS....

Draft Software InstallationUsing rpm to Install InfiniPath and OpenFabrics 5-14 Preliminary IB0056101-00 G.02 S RPMs contain config files. Your current config files will not be overwritten when new RPMs are installed. New config files will contain the suffix .rpmnew and can be found in /etc/sysconfi...

Draft Software InstallationUsing rpm to Install InfiniPath and OpenFabrics 5-16 Preliminary IB0056101-00 G.02 S Note that you need to install the InfiniPath-MPI/32bit/mpi-frontend-*.rpm even if you do not plan to use any other 32-bit RPMs. However, you can add the other 32bit subdirectories to the r...

Draft Software Installation Install QLogic OFED User-level Software with the rpm Command IB0056101-00 G.02 Preliminary 5-17 A OpenFabrics/32bit/OpenFabrics-Static/OpenFabrics-Static/32bit/OpenFabrics-Devel/OpenFabrics-Devel/32bit/OpenFabrics-Devel-Static/OpenFabrics-Debuginfo/OpenFabrics-Debuginfo/3...

Draft Software InstallationInstall QLogic OFED Using Rocks 5-20 Preliminary IB0056101-00 G.02 S Install QLogic OFED Using Rocks Rocks is a distribution designed for managing clusters from the San Diego Supercomputer center (SDSC). Rocks is a way to manage the kickstart automated installation method ...

Draft Software Installation Install QLogic OFED Using Rocks IB0056101-00 G.02 Preliminary 5-21 A Boot, Core, OS Roll DVD Note that you may also need updates; look for the latest files with the service-pack prefix. Make sure you downloaded the .iso images correctly; verify by checking the md5 checksu...

Draft Software InstallationInstall QLogic OFED Using a Platform OCS Kit 5-22 Preliminary IB0056101-00 G.02 S If you have a burned a CD version of InfiniPath roll from the .iso image: # mount /mnt/cdrom # rocks-dist --install copyroll # umount /mnt/cdrom # cd /home/install # rocks-dist dist You can u...

Draft Software Installation Install Additional Software IB0056101-00 G.02 Preliminary 5-23 A See also the current version of the FastFabric Users Guide, available with the software or from the QLogic download page. Install Additional Software This section contains information about additional third-...

Draft Software InstallationRemoving Software Packages 5-24 Preliminary IB0056101-00 G.02 S Documentation is found in: /usr/share/man/usr/share/doc/infinipath /usr/share/doc/mpich-infinipath Note that license information only is found in usr/share/doc/infinipath. InfiniPath user documentation can be ...

Draft Software Installation Removing Software Packages IB0056101-00 G.02 Preliminary 5-25 A For both InfiniPath and OpenFabrics, QLogic recommends that you remove all the packages at the same time. 1. Use the script ofed_uninstall.sh to uninstall the OFED software. See the OFED Installation release ...

Draft IB0056101-00 G.02 Preliminary 6-1 6 Configuring Drivers and Services This section provides instructions for configuring and using the drivers and services available with QLogic OFED 1.4. InfiniPath and OpenFabrics Driver Overview The InfiniPath ib_ipath module provides low level QLogic hardwar...

Draft Configuring Drivers and ServicesOpenFabrics Drivers and Services Configuration and Startup 6-2 Preliminary IB0056101-00 G.02 S Configuring the IPoIB Network Interface The following instructions show you how to manually configure your OpenFabrics IPoIB network interface. This example assumes th...

Draft Configuring Drivers and ServicesOpenFabrics Drivers and Services Configuration and Startup 6-4 Preliminary IB0056101-00 G.02 S You can stop opensmd again like this: # /etc/init.d/opensmd stop If you want to pass any arguments to the OpenSM program, modify the following file, and add the argume...

Draft Configuring Drivers and ServicesOpenFabrics Drivers and Services Configuration and Startup 6-6 Preliminary IB0056101-00 G.02 S 5. You can look for the newly created devices in the /proc/partitions file. The file will look similar to this example (the partition names may vary): # cat /proc/part...

Draft Configuring Drivers and Services OpenFabrics Drivers and Services Configuration and Startup IB0056101-00 G.02 Preliminary 6-13 A If you want to restart the QLogic VNIC interfaces, run the following command: # /etc/init.d/qlgc_vnic restart You can get information about the QLogic VNIC interface...

Draft Configuring Drivers and ServicesOther Configuration: Changing the MTU Size 6-14 Preliminary IB0056101-00 G.02 S 2. Check that you have a /etc/dat.conf file. It should be installed by the dapl- RPM. The file dat.conf contains a list of interface adapters supported by uDAPL service providers. In...

Draft Configuring Drivers and Services Managing the InfiniPath Driver IB0056101-00 G.02 Preliminary 6-15 A Managing the InfiniPath Driver The startup script for ib_ipath is installed automatically as part of the software installation, and normally does not need to be changed. It runs as a system ser...

Draft Configuring Drivers and ServicesManaging the InfiniPath Driver 6-16 Preliminary IB0056101-00 G.02 S The device files are: /dev/ipath /dev/ipath0, /dev/ipath1, ... The numbered device files allow access to a specific InfiniPath unit. See the ib_ipath man page for more details. Configure InfiniP...

Draft Configuring Drivers and Services Further Information on Configuring and Loading Drivers IB0056101-00 G.02 Preliminary 6-17 A The sequence of commands to restart the driver are as follows. # /etc/init.d/opensmd stop # /etc/init.d/ openibd stop ...# /etc/init.d/ openibd start # /etc/init.d/opens...

Draft IB0056101-00 G.02 Preliminary 7-1 7 Installation Verification and Additional Settings This section provides instructions for verifying that the software has been properly installed and that the drivers are loaded, and the fabric is active and ready to use. Information on adapter performance tu...

Draft Installation Verification and Additional SettingsCustomer Acceptance Utility 7-2 Preliminary IB0056101-00 G.02 S „ Make sure that write combining is enabled. The x86 Page Attribute Table (PAT) mechanism that allocates write-combining (WC) mappings for the PIO buffers has been added and is now ...

Draft IB0056101-00 G.02 Preliminary A-1 A Installation Troubleshooting The following sections contain information about issues that may occur during installation. Some of this material is repeated in the Troubleshooting appendix of the QLogic HCA and QLogic OFED Software User Guide . Many programs a...

Draft Installation TroubleshootingBIOS Settings A-2 Preliminary IB0056101-00 G.02 S BIOS Settings This section covers issues related to BIOS settings. You can check and adjust BIOS settings using the BIOS Setup utility. For specific instructions, follow the hardware documentation that came with your...

Draft Installation TroubleshootingSoftware Installation Issues A-4 Preliminary IB0056101-00 G.02 S Resolving Conflicts Occasionally, conflicts may arise when trying to install "on top of" an existing set of files that may come from a different set of RPMs. For example, if you install the QLo...

Draft Installation Troubleshooting Configuration Issues IB0056101-00 G.02 Preliminary A-5 A In older distributions, such as RHEL4, the 32-bit glibc is contained in the libgcc RPM. The RPM name will be similar to this: libgcc-<version>.EL4.i386.rpm In newer distributions, glibc is an RPM name. ...

Draft Write CombiningPAT and Write Combining B-2 Preliminary IB0056101-00 G.02 S .. If you do not see any of these messages on your console, but suspect this problem, check the /var/log/messages file. Some systems suppress driver load messages but still output them to the log file. Methods for enabl...

Draft IB0056101-00 G.02 Preliminary D-1 D Package Descriptions The following sections contain detailed descriptions of the packages for the InfiniPath and OpenFabrics software. In this release, software may be installed in several ways. With the QLogicIB-Basic download, which uses the Installer tool...

Draft Package DescriptionsInfiniPath RPM Version Numbers and Identifiers D-2 Preliminary IB0056101-00 G.02 S Although QLogic recommends installing all RPMs on all nodes, not all InfiniPath software is required on all nodes. See Table D-2 , Table D-3 , or Table D-4 for information on installation of ...

Draft Package Descriptions Documentation RPMs IB0056101-00 G.02 Preliminary D-3 A Documentation RPMs InfiniPath RPMs Table D-1. Documentation/RPMs RPM Name Front End Compute Development infinipath-doc- xxx_yyy .noarch.rpm InfiniPath man pages Optional Optional Optional mpi-doc- xxx_yyy .noarch.rpm M...

Draft Package DescriptionsOpenFabrics RPMs D-4 Preliminary IB0056101-00 G.02 S OpenFabrics RPMs For ease of installation, QLogic recommends that all of the OpenFabrics RPMs listed in Table D-5 be installed on all nodes. Version numbers are indicated by xxx . Table D-3. InfiniPath-Devel/RPMs RPM Name...

Draft Package Descriptions Other HCAs IB0056101-00 G.02 Preliminary D-9 A Other HCAs The packages in Table D-9 should be installed only if other HCAs are used. The development RPMs in Table D-10 are only needed on the nodes where programs are compiled Table D-8. OpenSM-Devel/RPM RPM Name Comments op...

Draft Package DescriptionsOther MPIs D-10 Preliminary IB0056101-00 G.02 S Other MPIs The packages in Table D-11 should be installed if other MPI implementations are desired.RPMs that include yyy in the name are supplied by QLogic; others are from OFED. libmlx4-devel- xxx .x86_64.rpm Development file...

Draft IB0056101-00 G.02 Preliminary Index-1 Index A ACPI 4-4, A-2Adapter, see HCA B BIOS configuring 4-4settings A-2settings to fix MTRR issues B-2 C -c 7-3 Cables supported 4-3Compiler support 5-3Configuration files C-1 ib_ipath 6-15 OpenSM 6-3VNIC 6-6 --continue 7-3 CPUs, HTX motherboards may requ...

Draft P r e l i m i n a r y © 2005-2009 QLogic Corporation. Specifications are subject to change without notice. All rights reserved worldwide. QLogic and the QLogic logo are registered trademarks of QLogic Corporation. QLA, SANsurfer, InfiniPath, and SilverStorm are trademarks or registered tradema...