Page 2 - Draft; Document Revision History
Draft Page ii Preliminary IB0056101-00 G.02 S QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 Information furnished in this manual is believed to be accurate and reliable. However, QLogic Corporation assumes no responsibility for its use, nor for any infringements of patent...
Page 8 - Notes
Draft Page viii Preliminary IB0056101-00 G.02 S QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 Notes
Page 9 - Table of Contents; Introduction
Draft IB0056101-00 G.02 Preliminary ix Table of Contents 1 Introduction Who Should Read this Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1 How this Guide is Organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1 Overview . . . . . . ...
Page 10 - Software Installation
Draft x Preliminary IB0056101-00 G.02 QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 S Hardware Installation for QLE7240, QLE7280, or QLE7140 with PCI Express Riser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-9 Dual Adapter Install...
Page 11 - Configuring Drivers and Services
Draft IB0056101-00 G.02 Preliminary xi QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 A Uninstalling Software with Rocks or Platform OCS . . . . . . . . . . . . . . . 5-25 Downgrading RPMs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-25 6 ...
Page 12 - Write Combining; Configuration Files
Draft xii Preliminary IB0056101-00 G.02 QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 S ibsrpdm Command Hangs When Two HCAs are Installed but Only Unit 1 is Connected to the Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-5 Outdated ipath_ethe...
Page 13 - List of Figures; List of Tables
Draft IB0056101-00 G.02 Preliminary xiii QLogic HCA and QLogic OFED Software Install Guide QLogic OFED Version 1.4 A List of Figures Figure Page 4-1 QLogic QLE7280 with IBA7220 ASIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-7 4-2 QLogic QLE7140 Card with Riser, Top View...
Page 15 - Who Should Read this Guide; How this Guide is Organized
Draft IB0056101-00 G.02 Preliminary 1-1 1 Introduction This chapter describes the contents, intended audience, and organization of the QLogic HCA and QLogic OFED Software Install Guide . The QLogic HCA and QLogic OFED Software Install Guide contains instructions for installing the QLogic Host Channe...
Page 16 - Overview
Draft IntroductionOverview 1-2 Preliminary IB0056101-00 G.02 S Section 3, Step-by-Step Installation Checklist , provides a high-level overview of the hardware and software installation procedures. Section 4, Hardware Installation , includes instructions for installing the QLogic QLE7140, QLE7240...
Page 17 - Interoperability
Draft Introduction Interoperability IB0056101-00 G.02 Preliminary 1-3 A QLogic OFED OpenFabrics software is interoperable with other vendors’ InfiniBand Host Channel Adapters (HCAs) running compatible OpenFabrics releases. There are several options for subnet management in your cluster: Use the em...
Page 18 - Documentation
Draft IntroductionDocumentation 1-4 Preliminary IB0056101-00 G.02 S Documentation The product documentation includes: The QLogic HCA and QLogic OFED Software Install Guide The QLogic HCA and QLogic OFED Software Users Guide The QLogic InfiniBand Software Install Guide The QLogic ULP and Tool...
Page 19 - Contact Information
Draft Introduction Contact Information IB0056101-00 G.02 Preliminary 1-5 A Contact Information Support Headquarters QLogic Corporation4601 Dean Lakes BlvdShakopee, MN 55379USA QLogic Web Site www.qlogic.com Technical Support Web Site support.qlogic.com Technical Support Email [email protected] Tech...
Page 21 - Feature Overview; Features
Draft IB0056101-00 G.02 Preliminary 2-1 2 Feature Overview This section contains the features for this release, the supported QLogic adapter models, supported distributions and kernels, and a list of the software components. Features The QLogic OFED 1.4 software release contains the complete OFED 1....
Page 22 - Other Changes
Draft Feature OverviewFeatures 2-2 Preliminary IB0056101-00 G.02 S MVAPICH and OpenMPI compiled with newer versions of each of four different compilers (GCC, PGI, Intel and PathScale) are available. The QLogic InfiniBand Fabric Suite (IFS) is available separately for purchase. It includes FastFa...
Page 24 - Software Components
Draft Feature OverviewSoftware Components 2-4 Preliminary IB0056101-00 G.02 S Software Components This release includes all of OFED 1.4 with enhancements (QLogic OFED 1.4), including a new version of the VNIC tools and driver, and support for the QHT7xxx and QLE7xxx adapters. The software includes t...
Page 25 - Hardware Installation
Draft IB0056101-00 G.02 Preliminary 3-1 3 Step-by-Step Installation Checklist This section provides an overview of the hardware and software installation procedures. Detailed steps are found in Section 4 “ Hardware Installation ” and Section 5 “ Software Installation ”. Hardware Installation The fol...
Page 29 - Hardware Installation Requirements; Hardware
Draft IB0056101-00 G.02 Preliminary 4-1 4 Hardware Installation This section lists the requirements and provides instructions for installing the QLogic InfiniPath Interconnect adapters. Instructions are included for the QLogic DDR PCI Express adapters, the QLE7240 and QLE7280; the QLogic PCIe adapte...
Page 30 - Form Factors
Draft Hardware InstallationHardware Installation Requirements 4-2 Preliminary IB0056101-00 G.02 S Installation of the QLE7240, QLE7280, QLE7140, QHT7040, or QHT7140 in a 1U or 2U chassis requires the use of a riser card. See Figure 4-4 for an illustration of a PCI Express (PCIe) slot in a typical mo...
Page 31 - Cabling and Switches
Draft Hardware Installation Hardware Installation Requirements IB0056101-00 G.02 Preliminary 4-3 A Cabling and Switches The cable installation uses a standard InfiniBand (IB) 4X cable. Any InfiniBand cable that has been qualified by the vendor should work. For SDR, the longest passive copper IB cabl...
Page 32 - Optical Fibre Option; Configuring the BIOS
Draft Hardware InstallationHardware Installation Requirements 4-4 Preliminary IB0056101-00 G.02 S Optical Fibre Option The QLogic adapter also supports connection to the switch by means of optical fibres through optical media converters such as the EMCORE™ QT2400. Not all switches support these type...
Page 33 - Safety with Electricity; Verify the Package Contents
Draft Hardware Installation Safety with Electricity IB0056101-00 G.02 Preliminary 4-5 A Safety with Electricity Observe these guidelines and safety precautions when working around computer hardware and electrical equipment: Locate the power source shutoff for the computer room or lab where you are...
Page 36 - Unpacking the QLogic Adapter
Draft Hardware InstallationUnpacking Information 4-8 Preliminary IB0056101-00 G.02 S Figure 4-3. QLogic QHT7040/QHT7140 Full and Low Profile Cards with Riser, Top View Unpacking the QLogic Adapter Follow these steps when unpacking the QLogic adapter: 1. When unpacking, ground yourself before removin...
Page 37 - with PCI Express Riser; Dual Adapter Installation; Installation Steps
Draft Hardware InstallationHardware Installation IB0056101-00 G.02 Preliminary 4-9 A Hardware Installation This section contains hardware installation instructions for the QLE7240, QLE7280, QLE7140, QHT7040, and QHT7140. Hardware Installation for QLE7240, QLE7280, or QLE7140 with PCI Express Riser I...
Page 40 - Hardware Installation for QHT7140 with HTX Riser
Draft Hardware InstallationHardware Installation 4-12 Preliminary IB0056101-00 G.02 S 13. Insert the riser assembly into the motherboard’s PCIe slot, ensuring good contact. The QLogic adapter should now be parallel to the motherboard and about one inch above it (see Figure 4-6 ). Figure 4-6. Assembl...
Page 43 - Without a PCI Express Riser
Draft Hardware InstallationHardware Installation IB0056101-00 G.02 Preliminary 4-15 A 13. Insert the HT riser assembly into the motherboard’s HTX slot, ensuring good contact. The QLogic adapter should now be parallel to the motherboard and about one inch above it, as shown in Figure 4-9 . Figure 4-9...
Page 44 - Hardware Installation for the QHT7140 Without an HTX Riser
Draft Hardware InstallationHardware Installation 4-16 Preliminary IB0056101-00 G.02 S Hardware Installation for the QHT7140 Without an HTX Riser Installing the QLogic QHT7140 without an HTX riser card requires a 3U or larger chassis. The card slot connectors on the QHT7140 fit into the HTX slot in a...
Page 45 - Switch Configuration and Monitoring
Draft Hardware Installation Switch Configuration and Monitoring IB0056101-00 G.02 Preliminary 4-17 A 8. Insert the card by pressing firmly and evenly on the top of the horizontal bracket and the top rear corner of the card simultaneously. The card should insert evenly into the slot. Be careful not t...
Page 46 - Completing the Installation
Draft Hardware InstallationCompleting the Installation 4-18 Preliminary IB0056101-00 G.02 S The QLE7240, QLE7280, QLE7140, QHT7040, and QHT7140 adapters are all cabled the same way. To install the InfiniBand cables: 1. Check that you have removed the protector plugs from the cable connector ends. 2....
Page 47 - Cluster Setup; Types of Nodes in a Cluster Environment; Supported Linux Distributions
Draft IB0056101-00 G.02 Preliminary 5-1 5 Software Installation This section provides instructions for installing QLogic OFED 1.4, which includes QLogic InfiniPath and the OpenFabrics software. The software includes drivers, protocol libraries, QLogic’s implementation of the MPI message passing stan...
Page 48 - Distribution Identifiers
Draft Software InstallationCluster Setup 5-2 Preliminary IB0056101-00 G.02 S Distribution Identifiers Distribution identifiers for this release are listed in the table below. They are used in file naming conventions. Table 5-1. InfiniPath/OpenFabrics Supported Distributions and Kernels Distribution ...
Page 49 - Compiler Support; Setting Up Your Environment
Draft Software Installation Compiler Support IB0056101-00 G.02 Preliminary 5-3 A Compiler Support QLogic MPI supports use of a number of compilers. These include: GNU gcc 3.3. x , 3.4. x , 4.0, 4.1, 4.2.x, and 4.3.x compiler suites PathScale Compiler Suite 3.0, 3.1 and 3.2 PGI 5.2, 6.0. 7.1, 7...
Page 50 - Choose the Appropriate Download Files
Draft Software InstallationChoose the Appropriate Download Files 5-4 Preliminary IB0056101-00 G.02 S Some operating system packages are required for OpenFabrics; they are listed in Table 5-3 . There are also OS package requirements for some specific components in QLogic OFED. Choose the Appropriate ...
Page 53 - Install QLogicIB-Basic with the Installer Tool
Draft Software Installation Install QLogicIB-Basic with the Installer Tool IB0056101-00 G.02 Preliminary 5-7 A Install QLogicIB-Basic with the Installer Tool The Installer tool has a Text User Interface (TUI) that affords easy installation of the software. Use this method if you have downloaded the ...
Page 59 - About
Draft Software Installation About rpm Installation IB0056101-00 G.02 Preliminary 5-13 A About rpm Installation Linux distributions of QLogic OFED (InfiniPath and OpenFabrics) software can be installed from binary RPMs. RPM is a Linux packaging and installation tool used by Red Hat, SUSE, and CentOS....
Page 60 - Using
Draft Software InstallationUsing rpm to Install InfiniPath and OpenFabrics 5-14 Preliminary IB0056101-00 G.02 S RPMs contain config files. Your current config files will not be overwritten when new RPMs are installed. New config files will contain the suffix .rpmnew and can be found in /etc/sysconfi...
Page 62 - RPM Organization
Draft Software InstallationUsing rpm to Install InfiniPath and OpenFabrics 5-16 Preliminary IB0056101-00 G.02 S Note that you need to install the InfiniPath-MPI/32bit/mpi-frontend-*.rpm even if you do not plan to use any other 32-bit RPMs. However, you can add the other 32bit subdirectories to the r...
Page 63 - Install QLogic OFED User-level Software with the
Draft Software Installation Install QLogic OFED User-level Software with the rpm Command IB0056101-00 G.02 Preliminary 5-17 A OpenFabrics/32bit/OpenFabrics-Static/OpenFabrics-Static/32bit/OpenFabrics-Devel/OpenFabrics-Devel/32bit/OpenFabrics-Devel-Static/OpenFabrics-Debuginfo/OpenFabrics-Debuginfo/3...
Page 66 - Install QLogic OFED Using Rocks; Install Frontend and Compute Nodes
Draft Software InstallationInstall QLogic OFED Using Rocks 5-20 Preliminary IB0056101-00 G.02 S Install QLogic OFED Using Rocks Rocks is a distribution designed for managing clusters from the San Diego Supercomputer center (SDSC). Rocks is a way to manage the kickstart automated installation method ...
Page 67 - Rocks Installation on an Existing Frontend Node
Draft Software Installation Install QLogic OFED Using Rocks IB0056101-00 G.02 Preliminary 5-21 A Boot, Core, OS Roll DVD Note that you may also need updates; look for the latest files with the service-pack prefix. Make sure you downloaded the .iso images correctly; verify by checking the md5 checksu...
Page 68 - Install QLogic OFED Using a Platform OCS Kit
Draft Software InstallationInstall QLogic OFED Using a Platform OCS Kit 5-22 Preliminary IB0056101-00 G.02 S If you have a burned a CD version of InfiniPath roll from the .iso image: # mount /mnt/cdrom # rocks-dist --install copyroll # umount /mnt/cdrom # cd /home/install # rocks-dist dist You can u...
Page 69 - Install Additional Software; Installing Lustre; Installed Layout
Draft Software Installation Install Additional Software IB0056101-00 G.02 Preliminary 5-23 A See also the current version of the FastFabric Users Guide, available with the software or from the QLogic download page. Install Additional Software This section contains information about additional third-...
Page 70 - Removing Software Packages; Uninstalling Using the Installer Tool; Uninstalling InfiniPath and OpenFabrics RPMs
Draft Software InstallationRemoving Software Packages 5-24 Preliminary IB0056101-00 G.02 S Documentation is found in: /usr/share/man/usr/share/doc/infinipath /usr/share/doc/mpich-infinipath Note that license information only is found in usr/share/doc/infinipath. InfiniPath user documentation can be ...
Page 71 - Uninstalling Software with Rocks or Platform OCS
Draft Software Installation Removing Software Packages IB0056101-00 G.02 Preliminary 5-25 A For both InfiniPath and OpenFabrics, QLogic recommends that you remove all the packages at the same time. 1. Use the script ofed_uninstall.sh to uninstall the OFED software. See the OFED Installation release ...
Page 73 - InfiniPath and OpenFabrics Driver Overview; OpenFabrics Drivers and Services Configuration
Draft IB0056101-00 G.02 Preliminary 6-1 6 Configuring Drivers and Services This section provides instructions for configuring and using the drivers and services available with QLogic OFED 1.4. InfiniPath and OpenFabrics Driver Overview The InfiniPath ib_ipath module provides low level QLogic hardwar...
Page 74 - Configuring the IPoIB Network Interface
Draft Configuring Drivers and ServicesOpenFabrics Drivers and Services Configuration and Startup 6-2 Preliminary IB0056101-00 G.02 S Configuring the IPoIB Network Interface The following instructions show you how to manually configure your OpenFabrics IPoIB network interface. This example assumes th...
Page 76 - SRP; Using QLogic SRP; Using OFED SRP
Draft Configuring Drivers and ServicesOpenFabrics Drivers and Services Configuration and Startup 6-4 Preliminary IB0056101-00 G.02 S You can stop opensmd again like this: # /etc/init.d/opensmd stop If you want to pass any arguments to the OpenSM program, modify the following file, and add the argume...
Page 78 - Configuring and Administering the VNIC Interface
Draft Configuring Drivers and ServicesOpenFabrics Drivers and Services Configuration and Startup 6-6 Preliminary IB0056101-00 G.02 S 5. You can look for the newly created devices in the /proc/partitions file. The file will look similar to this example (the partition names may vary): # cat /proc/part...
Page 85 - MPI over uDAPL
Draft Configuring Drivers and Services OpenFabrics Drivers and Services Configuration and Startup IB0056101-00 G.02 Preliminary 6-13 A If you want to restart the QLogic VNIC interfaces, run the following command: # /etc/init.d/qlgc_vnic restart You can get information about the QLogic VNIC interface...
Page 86 - Other Configuration: Changing the MTU Size
Draft Configuring Drivers and ServicesOther Configuration: Changing the MTU Size 6-14 Preliminary IB0056101-00 G.02 S 2. Check that you have a /etc/dat.conf file. It should be installed by the dapl- RPM. The file dat.conf contains a list of interface adapters supported by uDAPL service providers. In...
Page 87 - Managing the InfiniPath Driver
Draft Configuring Drivers and Services Managing the InfiniPath Driver IB0056101-00 G.02 Preliminary 6-15 A Managing the InfiniPath Driver The startup script for ib_ipath is installed automatically as part of the software installation, and normally does not need to be changed. It runs as a system ser...
Page 88 - Configure InfiniPath Driver State; Start, Stop or Restart InfiniPath
Draft Configuring Drivers and ServicesManaging the InfiniPath Driver 6-16 Preliminary IB0056101-00 G.02 S The device files are: /dev/ipath /dev/ipath0, /dev/ipath1, ... The numbered device files allow access to a specific InfiniPath unit. See the ib_ipath man page for more details. Configure InfiniP...
Page 89 - Unloading the Driver/Modules Manually; Further Information on Configuring and Loading
Draft Configuring Drivers and Services Further Information on Configuring and Loading Drivers IB0056101-00 G.02 Preliminary 6-17 A The sequence of commands to restart the driver are as follows. # /etc/init.d/opensmd stop # /etc/init.d/ openibd stop ...# /etc/init.d/ openibd start # /etc/init.d/opens...
Page 91 - LED Link and Data Indicators
Draft IB0056101-00 G.02 Preliminary 7-1 7 Installation Verification and Additional Settings This section provides instructions for verifying that the software has been properly installed and that the drivers are loaded, and the fabric is active and ready to use. Information on adapter performance tu...
Page 92 - Customer Acceptance Utility
Draft Installation Verification and Additional SettingsCustomer Acceptance Utility 7-2 Preliminary IB0056101-00 G.02 S Make sure that write combining is enabled. The x86 Page Attribute Table (PAT) mechanism that allocates write-combining (WC) mappings for the PIO buffers has been added and is now ...
Page 95 - Hardware Issues; Node Spontaneously Reboots
Draft IB0056101-00 G.02 Preliminary A-1 A Installation Troubleshooting The following sections contain information about issues that may occur during installation. Some of this material is repeated in the Troubleshooting appendix of the QLogic HCA and QLogic OFED Software User Guide . Many programs a...
Page 96 - BIOS Settings; Enable Advanced Configuration and Power Interface (ACPI); Issue with Supermicro; Software Installation Issues; Missing Kernel RPM Errors
Draft Installation TroubleshootingBIOS Settings A-2 Preliminary IB0056101-00 G.02 S BIOS Settings This section covers issues related to BIOS settings. You can check and adjust BIOS settings using the BIOS Setup utility. For specific instructions, follow the hardware documentation that came with your...
Page 98 - Resolving Conflicts; mpirun
Draft Installation TroubleshootingSoftware Installation Issues A-4 Preliminary IB0056101-00 G.02 S Resolving Conflicts Occasionally, conflicts may arise when trying to install "on top of" an existing set of files that may come from a different set of RPMs. For example, if you install the QLo...
Page 99 - Lockable Memory Error on Initial Installation of InfiniPath; Configuration Issues; Outdated
Draft Installation Troubleshooting Configuration Issues IB0056101-00 G.02 Preliminary A-5 A In older distributions, such as RHEL4, the 32-bit glibc is contained in the libgcc RPM. The RPM name will be similar to this: libgcc-<version>.EL4.i386.rpm In newer distributions, glibc is an RPM name. ...
Page 102 - PAT and Write Combining; MTRR Mapping and Write Combining; Edit BIOS Settings to Fix MTRR Issues
Draft Write CombiningPAT and Write Combining B-2 Preliminary IB0056101-00 G.02 S .. If you do not see any of these messages on your console, but suspect this problem, check the /var/log/messages file. Some systems suppress driver load messages but still output them to the log file. Methods for enabl...
Page 107 - Package Descriptions; Package Names with the QLogicIB-Basic
Draft IB0056101-00 G.02 Preliminary D-1 D Package Descriptions The following sections contain detailed descriptions of the packages for the InfiniPath and OpenFabrics software. In this release, software may be installed in several ways. With the QLogicIB-Basic download, which uses the Installer tool...
Page 108 - InfiniPath RPM Version Numbers and Identifiers; OpenFabrics RPM Names; InfiniPath and OpenFabrics RPMs
Draft Package DescriptionsInfiniPath RPM Version Numbers and Identifiers D-2 Preliminary IB0056101-00 G.02 S Although QLogic recommends installing all RPMs on all nodes, not all InfiniPath software is required on all nodes. See Table D-2 , Table D-3 , or Table D-4 for information on installation of ...
Page 109 - Documentation RPMs
Draft Package Descriptions Documentation RPMs IB0056101-00 G.02 Preliminary D-3 A Documentation RPMs InfiniPath RPMs Table D-1. Documentation/RPMs RPM Name Front End Compute Development infinipath-doc- xxx_yyy .noarch.rpm InfiniPath man pages Optional Optional Optional mpi-doc- xxx_yyy .noarch.rpm M...
Page 110 - OpenFabrics RPMs
Draft Package DescriptionsOpenFabrics RPMs D-4 Preliminary IB0056101-00 G.02 S OpenFabrics RPMs For ease of installation, QLogic recommends that all of the OpenFabrics RPMs listed in Table D-5 be installed on all nodes. Version numbers are indicated by xxx . Table D-3. InfiniPath-Devel/RPMs RPM Name...
Page 115 - Other HCAs
Draft Package Descriptions Other HCAs IB0056101-00 G.02 Preliminary D-9 A Other HCAs The packages in Table D-9 should be installed only if other HCAs are used. The development RPMs in Table D-10 are only needed on the nodes where programs are compiled Table D-8. OpenSM-Devel/RPM RPM Name Comments op...
Page 116 - Other MPIs
Draft Package DescriptionsOther MPIs D-10 Preliminary IB0056101-00 G.02 S Other MPIs The packages in Table D-11 should be installed if other MPI implementations are desired.RPMs that include yyy in the name are supplied by QLogic; others are from OFED. libmlx4-devel- xxx .x86_64.rpm Development file...
Page 119 - Index
Draft IB0056101-00 G.02 Preliminary Index-1 Index A ACPI 4-4, A-2Adapter, see HCA B BIOS configuring 4-4settings A-2settings to fix MTRR issues B-2 C -c 7-3 Cables supported 4-3Compiler support 5-3Configuration files C-1 ib_ipath 6-15 OpenSM 6-3VNIC 6-6 --continue 7-3 CPUs, HTX motherboards may requ...
Page 124 - P r e l i m i n a r y
Draft P r e l i m i n a r y © 2005-2009 QLogic Corporation. Specifications are subject to change without notice. All rights reserved worldwide. QLogic and the QLogic logo are registered trademarks of QLogic Corporation. QLA, SANsurfer, InfiniPath, and SilverStorm are trademarks or registered tradema...
