Page 3 - C O N T E N T S; Preface; About Host Drivers; Installing Host Drivers; IP over IB Protocol
iii Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 C O N T E N T S Preface vii Audience vii Organization vii Conventions viii Root and Non-root Conventions in Examples ix Related Documentation ix Obtaining Documentation, Obtaining Support, and Security Guidelines ix C H A P T E R...
Page 4 - SCSI RDMA Protocol; Sockets Direct Protocol; uDAPL
Contents iv Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Subinterfaces 3-2 Creating a Subinterface Associated with a Specific IB Partition 3-3 Removing a Subinterface Associated with a Specific IB Partition 3-4 Verifying IPoIB Functionality 3-5 IPoIB Performance 3-6 Sample Star...
Page 5 - MVAPICH MPI; HCA Utilities and Diagnostics; Acronyms and Abbreviations
Contents v Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 C H A P T E R 7 MVAPICH MPI 7-1 Introduction 7-1 Initial Setup 7-2 Configuring SSH 7-2 Editing Environment Variables 7-5 Setting Environment Variables in System-Wide Startup Files 7-6 Editing Environment Variables in the U...
Page 7 - Audience
vii Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Preface This preface describes who should read the Cisco SFS InfiniBand Host Drivers User Guide for Linux , how it is organized, and its document conventions. It includes the following sections: • Audience, page vii • Organizatio...
Page 8 - Conventions
viii Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Preface Conventions Conventions This document uses the following conventions: Chapter 8 HCA Utilities and Diagnostics Describes the fundamental HCA utilities and diagnostics. Appendix A Acronyms and Abbreviations Defines the acr...
Page 9 - Root and Non-root Conventions in Examples
ix Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Preface Root and Non-root Conventions in Examples Notes use the following convention: Note Means reader take note . Notes contain helpful suggestions or references to material not covered in the manual. Cautions use the following ...
Page 11 - Introduction
C H A P T E R 1-1 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 1 About Host Drivers This chapter describes host drivers and includes the following sections: • Introduction, page 1-1 • Architecture, page 1-2 • Supported Protocols, page 1-3 • Supported APIs, page 1-4 • HCA Utilit...
Page 12 - Architecture
1-2 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 1 About Host Drivers Architecture Note See the “Root and Non-root Conventions in Examples” section on page ix for details about the significance of prompts used in the examples in this chapter. Architecture Figure 1-1 dis...
Page 13 - Supported Protocols; IPoIB; SRP
1-3 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 1 About Host Drivers Supported Protocols Supported Protocols This section describes the supported protocols and includes the following topics: • IPoIB • SRP • SDP Protocol here refers to software in the networking layer i...
Page 14 - Supported APIs
1-4 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 1 About Host Drivers Supported APIs Supported APIs This section describes the supported APIs and includes the following topics: • MVAPICH MPI • uDAPL • Intel MPI • HP MPI API refers to software in the networking layer in ...
Page 16 - Contents of ISO Image
2-2 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 2 Installing Host Drivers Contents of ISO Image Contents of ISO Image The ISO image contains the following directories and files: • docs/ This directory contains the related documents. • tsinstall This is the installation...
Page 17 - Uninstalling Host Drivers from an ISO Image
2-3 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 2 Installing Host Drivers Uninstalling Host Drivers from an ISO Image topspin-ib-mpi-rhel4-3.2.0-136.x86_64 (MPI libraries, source code, docs, etc) topspin-ib-mod-rhel4-2.6.9-34.ELsmp-3.2.0-136.x86_64 (kernel modules) ins...
Page 20 - Manually Configuring IPoIB for Default IB Partition; Subinterfaces
3-2 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 3 IP over IB Protocol Manually Configuring IPoIB for Default IB Partition Manually Configuring IPoIB for Default IB Partition To manually configure IPoIB for the default IB partition, perform the following steps: Step 1 L...
Page 21 - Creating a Subinterface Associated with a Specific IB Partition
3-3 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 3 IP over IB Protocol Subinterfaces Creating a Subinterface Associated with a Specific IB Partition To create a subinterface associated with a specific IB partition, perform the following steps: Step 1 Create a partition ...
Page 22 - Removing a Subinterface Associated with a Specific IB Partition
3-4 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 3 IP over IB Protocol Subinterfaces NOARP MTU:1480 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (...
Page 23 - Verifying IPoIB Functionality
3-5 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 3 IP over IB Protocol Verifying IPoIB Functionality TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1024 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) lo Link encap:Local Loopback inet addr:127.0.0.1 ...
Page 24 - IPoIB Performance
3-6 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 3 IP over IB Protocol IPoIB Performance IPoIB Performance This section describes how to verify IPoIB performance by running the Bandwidth test and the Latency test. These tests are described in detail at the following URL...
Page 26 - Sample Startup Configuration File; IPoIB High Availability
3-8 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 3 IP over IB Protocol Sample Startup Configuration File Sample Startup Configuration File IP addresses that are configured manually are not persistent across reboots. You must use a configuration file to configure IPoIB w...
Page 27 - Unmerging Physical Ports
3-9 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 3 IP over IB Protocol IPoIB High Availability Step 3 Take the interfaces offline. You cannot merge interfaces until you bring them down. The following example shows how to take the interfaces offline: host1# ifconfig ib0 ...
Page 30 - Configuring ITLs when Using Fibre Channel Gateway
4-2 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 4 SCSI RDMA Protocol Configuring SRP This section contains information on how to configure your IB fabric to connect an SRP host to a SAN and includes the following topics: • Configuring ITLs when Using Fibre Channel Gate...
Page 34 - Configuring SRP Host
4-6 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 4 SCSI RDMA Protocol Configuring SRP Configuring SRP Host This section describes how to configure the SRP host. The SRP host driver exposes a Fibre Channel target (identified by a WWPN) as a SCSI target to the Linux SCSI ...
Page 35 - Verifying SRP; Verifying SRP Functionality
4-7 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 4 SCSI RDMA Protocol Verifying SRP Verifying SRP This section describes how to verify SRP functionality and verify SRP host-to-storage connections with the Element Manager GUI and includes the following sections: • Verify...
Page 36 - Verifying with Element Manager
4-8 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 4 SCSI RDMA Protocol Verifying SRP 512000 inodes, 1023996 blocks 51199 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=1048576000 32 block groups 32768 blocks per group, 32768 fragm...
Page 38 - Converting Sockets-Based Application; Explicit/Source Code Conversion Type
5-2 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 5 Sockets Direct Protocol Converting Sockets-Based Application Converting Sockets-Based Application This section describes how to convert sockets-based applications. You can convert your sockets-based applications to use ...
Page 39 - Log Statement; Match Statement
5-3 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 5 Sockets Direct Protocol Converting Sockets-Based Application Log Statement This section describes the log statement. The log directive allows the user to specify which debug and error messages are sent and where they ar...
Page 40 - SDP Performance
5-4 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 5 Sockets Direct Protocol SDP Performance The ip_port matches against an IP address, prefix length, and port range. The format is as follows: ip_addr [/ prefix_length ][: start_port [- end_port ]] The prefix length is opt...
Page 42 - Netperf Server with IPoIB and SDP
5-6 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 5 Sockets Direct Protocol Netperf Server with IPoIB and SDP The notable performance values in the example above are as follows: Client CPU utilization is 6.26 percent of client CPU. Server CPU utilization is 7.22 percent ...
Page 46 - uDAPL Throughput Test Performance
6-2 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 6 uDAPL uDAPL Throughput Test Performance The Throughput test measures RDMA WRITE throughput using uDAPL. To perform a uDAPL Throughput test performance, perform the following steps: Step 1 Start the Throughput test on th...
Page 47 - uDAPL Latency Test Performance
6-3 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 6 uDAPL Step 3 View the Throughput test results from the server. The following example shows the Throughput test results: Created an EP with ep_handle = 0x2a95f8a300 queried max_recv_dtos = 256 queried max_request_dtos = ...
Page 48 - Compiling uDAPL Programs
6-4 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 6 uDAPL host2$ /usr/local/topspin/bin/lat_client.x ib0 192.168.0.1 200000 1 0 • ib0 is the name of the device. • 192.168.0.1 is the IPoIB address of the server host. • 200000 is the number of RDMAs to perform for the test...
Page 50 - Initial Setup; Configuring SSH
7-2 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 7 MVAPICH MPI Initial Setup Initial Setup This section describes the initial MPI setup. MPI can be used with either IPoIB or Ethernet IP addresses. The drivers for MPI are automatically loaded at boot time if IPoIB or SDP...
Page 53 - Editing Environment Variables
7-5 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 7 MVAPICH MPI Editing Environment Variables Step 9 Return to host1 and copy the authorized keys file from Step 6 to the directory that you created in Step 8 . The following example shows how to return to host1 and copy th...
Page 54 - Setting Environment Variables in System-Wide Startup Files
7-6 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 7 MVAPICH MPI Editing Environment Variables Setting Environment Variables in System-Wide Startup Files This method is used to set a system-wide default for which MPI implementation is used. This method is the easiest for ...
Page 55 - Editing Environment Variables Manually; MPI Bandwidth Test Performance
7-7 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 7 MVAPICH MPI MPI Bandwidth Test Performance Editing Environment Variables Manually Typically, you edit environment variables manually when it is necessary to run temporarily with a given MPI implementation. For example, ...
Page 56 - MPI Latency Test Performance
7-8 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 7 MVAPICH MPI MPI Latency Test Performance When the test completes successfully, you see output that is similar to the following: # OSU MPI Bandwidth Test (Version 2.2) # Size Bandwidth (MB/s) 1 3.352541 2 6.701571 4 10.7...
Page 57 - Intel MPI Benchmarks (IMB) Test Performance
7-9 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 7 MVAPICH MPI Intel MPI Benchmarks (IMB) Test Performance • The name of the hostfile • The latency executable name The following example shows how to run the MVAPICH MPI Latency test: host1$ mpirun_rsh -np 2 -hostfile /tm...
Page 60 - Compiling MPI Programs
7-12 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 7 MVAPICH MPI Compiling MPI Programs Compiling MPI Programs This section describes how to compile MPI programs. Compiling MPI applications from source code requires adding several compiler and linker flags. MVAPICH MPI p...
Page 65 - tvflash Utility; Viewing Card Type and Firmware Version
8-3 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 8 HCA Utilities and Diagnostics tvflash Utility tvflash Utility This section describes the tvflash utility and includes the following topics: • Viewing Card Type and Firmware Version, page 8-3 • Upgrading Firmware, page 8...
Page 66 - Upgrading Firmware
8-4 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 8 HCA Utilities and Diagnostics tvflash Utility Upgrading Firmware To upgrade firmware on your host, perform the following steps: Note Upon installation of the host drivers, the firmware is automatically updated, if requi...
Page 67 - Diagnostics
8-5 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 8 HCA Utilities and Diagnostics Diagnostics Diagnostics This section includes diagnostics information. A few diagnostic programs are included with the Linux IB host drivers. The vstat utility prints IB information. The fo...
Page 71 - I N D E X
IN-1 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 I N D E X A architecture, HCA supported 1-2 audience vii authenticity messages 7-5 B Bandwidth test default 3-6, 5-7 MPI 7-7 Netperf 5-6 using 3-6 C card type, view 8-3 CLI 4-2 command-line interface. See CLI. compile MPI progra...