Page 3 - Restrictions and Disclaimers
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 3 Restrictions and Disclaimers The information contained in this document, including all instructions, cautions, and regulatory approvals and certifications, is provided by t...
Page 4 - Table of Contents
Rev 1.1 Mellanox Technologies 4 Table of Contents Restrictions and Disclaimers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Export Regulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Table of Contents ...
Page 8 - List of Tables
Rev 1.1 Mellanox Technologies 8 List of Tables Table 1: Revision History Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Table 2: Documents List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...
Page 9 - List of Figures
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 9 List of Figures Figure 1: Mellanox ConnectX-3 Dual Port 40GbE QSFP+ Network Adapter Full Height Bracket 16 Figure 2: Mellanox ConnectX-3 Dual Port 10GbE SFP+ Network Adapte...
Page 10 - Revision History; This document was printed on August 12, 2014.
Rev 1.1 Mellanox Technologies 10 Revision History This document was printed on August 12, 2014. Table 1 - Revision History Table Date Rev Comments/Changes August 2014 1.1 • Added Section 4.2, “Linux Driver Features,” on page 25 • Added Section 4.5, “WinOF Features,” on page 59 • Added Section 5, “Re...
Page 11 - About this Manual; This; Intended Audience; This manual is intended for the installer and user of these cards.; Related Documentation; Document Conventions; This document uses the following conventions:; Technical Support
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 11 About this Manual This User Manual describes Mellanox Technologies ConnectX ® -3 10/40 Gigabit Ethernet Adapter and Cards for Dell PowerEdge Servers. It provides details a...
Page 12 - Introduction; Functional Description
Introduction Rev 1.1 Mellanox Technologies 12 1 Introduction 1.1 Functional Description Mellanox Ethernet adapters utilizing IBTA RoCE technology provide efficient RDMA services, delivering high performance to bandwidth and latency sensitive applications. Applications utiliz-ing TCP/UDP/IP transport...
Page 14 - Supported Operating Systems/Distributions; OpenFabrics Enterprise Distribution (OFED)
Introduction Rev 1.1 Mellanox Technologies 14 instances of the device with separate resources. Mellanox adapters are capable of exposing up to 126 virtual instances called Virtual Functions (VFs). These virtual functions can then be provi-sioned separately. Each VF can be seen as an addition device ...
Page 15 - Adapter Card Interfaces; Each adapter card includes the following interfaces:; LED Assignment; Table 3; Function
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 15 2 Adapter Card Interfaces 2.1 I/O Interfaces Each adapter card includes the following interfaces: • High speed port: • QSFP+ for the 40GbE Network Adapters • SFP+ for the ...
Page 16 - Port 1 Activity
Adapter Card Interfaces Rev 1.1 Mellanox Technologies 16 Figure 1: Mellanox ConnectX-3 Dual Port 40GbE QSFP+ Network Adapter Full Height Bracket Table 4 - LED Assignment for 40GbE QSFP+ Network Adapters Link LED (Bicolor - Green and Yellow) Activity LED (Green) Function Off Off No link present Yello...
Page 17 - Port 2 Link
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 17 Figure 2: Mellanox ConnectX-3 Dual Port 10GbE SFP+ Network Adapter Full Height Bracket Port 2 Link Port 2 Activity Port 1 Link Port 1 Activity
Page 18 - Installing the Hardware; System Requirements; Hardware; Please refer to; Software Stacks; Safety Precautions; Remove any metallic objects from your hands and wrists.
Installing the Hardware Rev 1.1 Mellanox Technologies 18 3 Installing the Hardware 3.1 System Requirements 3.1.1 Hardware Dell PowerEdge Server with an available PCI Express x8 slot for installing the card is required. 3.1.2 Operating Systems/Distributions Please refer to Section 1.3, “Supported Ope...
Page 19 - Pre-installation Checklist; After shutting down the system, turn off power and unplug the cord.; Installation Instructions; For Adapters; The adapter can be placed in an available slot.; For Mezzanine Cards; The card can be placed in an available slot.
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 19 3.3 Pre-installation Checklist 1. Verify that your system meets the hardware and software requirements stated above. 2. Shut down your system if active. 3. After shutting ...
Page 20 - Secure the blade mezzanine card with the mezzanine card latch.; Connecting the Network Cables; Inserting a Cable into the Adapter Card; cable holder or tying the cable to the rack.; Removing a Cable from the Adapter Card; Identifying the Card in A System; On Linux
Installing the Hardware Rev 1.1 Mellanox Technologies 20 4. Expose the socket to be used for the new card. When replacing an existing card, remove the card from the socket. Grab the card on the edge on the side with UPC number and pull up while gently rocking the card back and forth. For a new insta...
Page 21 - Driver Installation and Configuration; Linux Driver; Installation Requirements; Required Disk Space for Installation; Downloading Mellanox OFED
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 21 4 Driver Installation and Configuration 4.1 Linux Driver For Linux, download and install the latest Linux Drivers for Mellanox ConnectX-3 Ethernet adapters software packag...
Page 22 - Installation Script; Table 5; mst
Driver Installation and Configuration Rev 1.1 Mellanox Technologies 22 4.1.3.1 Pre-installation Notes • The installation script removes all previously installed Mellanox OFED packages and installs the software release. 4.1.4 Installation Script Within each distribution specific subdirectory there is...
Page 23 - Installation Procedure; Login to the installation machine as root.
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 23 4.1.5 Installation Procedure Step 1. Login to the installation machine as root. Step 2. Copy the software release on your machine Step 3. Un-tar the software release. Step...
Page 24 - Installation Results; Software
Driver Installation and Configuration Rev 1.1 Mellanox Technologies 24 Step 6. The script adds the following lines to /etc/security/limits.conf for the userspace components such as MPI: These settings unlimit the amount of memory that can be pinned by a user space application. If desired, tune the v...
Page 25 - Post-installation Notes; Linux Driver Features; iSER Initiator
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 25 • The installation process unlimits the amount of memory that can be pinned by a user space application. See Step 6. • Man pages will be installed under /usr/share/man/ 4....
Page 26 - Plain Ethernet
Driver Installation and Configuration Rev 1.1 Mellanox Technologies 26 The following is the general mapping traffic to Traffic Classes flow: 1. The application sets the required Type of Service (ToS). 2. The ToS is translated into a Socket Priority ( sk_prio ). 3. The sk_prio is mapped to a User Pri...
Page 28 - “Strict Priority”; Strict Priority
Driver Installation and Configuration Rev 1.1 Mellanox Technologies 28 • When the user uses sk_prio , it is mapped into a UP by the ‘tc’ tool. This is done by the tc_wrap.py tool which gets a list of <= 16 comma separated UP and maps the sk_prio to the specified UP. For example, tc_wrap.py -ieth0...
Page 30 - Get Current Configuration:
Driver Installation and Configuration Rev 1.1 Mellanox Technologies 30 Get Current Configuration: Set ratelimit. 3Gbps for tc0 4Gbps for tc1 and 2Gbps for tc2: tc: 0 ratelimit: unlimited, tsa: strict up: 0 skprio: 0 skprio: 1 skprio: 2 (tos: 8) skprio: 3 skprio: 4 (tos: 24) skprio: 5 skprio: 6 (tos:...
Page 31 - to UP mapping, using the; In kernels that do not support; to UP
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 31 Configure QoS. map UP 0,7 to tc0, 1,2,3 to tc1 and 4,5,6 to tc 2. set tc0,tc1 as ets and tc2 as strict. divide ets 30% for tc0 and 70% for tc1: tc and tc_wrap.py The 'tc' ...
Page 32 - Additional Tools; tc tool compiled with the; interface is available.; Ethernet Time-Stamping; Time-stamping is off by default and should be enabled before use.
Driver Installation and Configuration Rev 1.1 Mellanox Technologies 32 Additional Tools tc tool compiled with the sch_mqprio module is required to support kernel v2.6.32 or higher. This is a part of iproute2 package v2.6.32-19 or higher. Otherwise, an alternative custom sysfs interface is available....
Page 34 - To enable time stamping for a net device:
Driver Installation and Configuration Rev 1.1 Mellanox Technologies 34 To enable time stamping for a net device: Admin privileged user can enable/disable time stamping through calling ioctl(sock, SIOCSHWT-STAMP, &ifreq) with following values: Send side time sampling: • Enabled by ifreq.hwtstam...
Page 35 - Receive side time sampling:
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 35 Receive side time sampling: • Enabled by ifreq.hwtstamp_config.rx_filter when 4.2.4.2 Getting Time Stamping Once time stamping is enabled time stamp is placed in the socke...
Page 36 - To display Time Stamping capabilities via ethtool:; RoCE Time Stamping
Driver Installation and Configuration Rev 1.1 Mellanox Technologies 36 a pending bounced packet is ready for reading as far as select() is concerned. If the outgoing packet has to be fragmented, then only the first fragment is time stamped and returned to the sending socket. 4.2.4.3 Querying Time St...
Page 37 - Polling a CQ for time stamp is done via the
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 37 For example: 4.2.5.2 Creating Time Stamping Completion Queue To get time stamps, a suitable extended Completion Queue (CQ) must be created via a special call to ibv_create...
Page 38 - Flow Steering; To enable Flow Steering:
Driver Installation and Configuration Rev 1.1 Mellanox Technologies 38 4.2.5.4 Querying the Hardware Time Querying the hardware for time is done via the ibv_query_values_ex verb. For example: To change the queried time in nanoseconds resolution, use the IBV_VALUES_HW_CLOCK_NS flag along with the hwc...
Page 39 - User Verbs
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 39 4.2.6.2 Flow Domains and Priorities Flow steering defines the concept of domain and priority. Each domain represents a user agent that can attach a flow. The domains are p...
Page 40 - Ethtool
Driver Installation and Configuration Rev 1.1 Mellanox Technologies 40 Output parameters: Returns 0 on success, or the value of errno on failure. For further information, please refer to the ibv_destroy_flow man page. • Ethtool Ethtool domain is used to attach an RX ring, specifically its QP to a sp...
Page 41 - All of the rest
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 41 RFS requires the kernel to be compiled with the CONFIG_RFS_ACCEL option. This options is available in kernels 2.6.39 and above. Furthermore, RFS requires Device Managed Fl...
Page 42 - PFs not included in the above list will not have SR-
Driver Installation and Configuration Rev 1.1 Mellanox Technologies 42 For example, to Intel systems, add: Step 7. Enable SRIOV number of functions in PCI. Note: by default 8 virtual functions are enabled in PCI level. Step 8. Create the text file /etc/modprobe.d/mlx4_core.conf if it does not exist,...
Page 43 - The protocol types are:; PFs not included in the above list will not activate any
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 43 The example above loads the driver with 5 VFs (num_vfs). The standard use of a VF is a single VF per a single VM. However, the number of VFs varies upon the working mode r...
Page 44 - To enable SR-IOV and Para Virtualization on the same setup:; Assigning the SR-IOV Virtual Function to the Red Hat KVM VM Server; Double click on the virtual machine and open its Properties.
Driver Installation and Configuration Rev 1.1 Mellanox Technologies 44 4.2.7.3 Enabling SR-IOV and Para Virtualization on the Same Setup To enable SR-IOV and Para Virtualization on the same setup: Step 1. Create a bridge. Step 2. Change the related interface (in the example below bridge0 is create...
Page 45 - If the Virtual Machine is up reboot it, otherwise start it.; Please be aware
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 45 Step 3. Go to Details->Add hardware ->PCI host device. Step 4. Choose a Mellanox virtual function according to its PCI device (e.g., 00:03.1) Step 5. If the Virtual ...
Page 49 - Options
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 49 The following are the ethtool supported options: Table 7 - ethtool Supported Options Options Description ethtool -i eth<x> Checks driver and device information. For ...
Page 50 - Ethernet Performance Counters
Driver Installation and Configuration Rev 1.1 Mellanox Technologies 50 4.2.9 Ethernet Performance Counters Counters are used to provide information about how well an operating system, an application, a service, or a driver is performing. The counter data helps determine system bottlenecks and fine-t...
Page 51 - Counter
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 51 • Physical Function can also read Virtual Functions' port counters through sysfs located under: • # /sys/class/net/eth*/vf*_statistics/ To display the network device Ether...
Page 55 - VMware Driver; Installing and Running the Driver; » To verify that the driver is loaded, run:
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 55 4.3 VMware Driver For VMware, download and install the latest Mellanox Ethernet Driver for VMware vSphere 5.X from the VMware support site: http://www.vmware.com/support ....
Page 56 - » To query network uplinks installed on your machine, run:
Driver Installation and Configuration Rev 1.1 Mellanox Technologies 56 » To query network uplinks installed on your machine, run: The number of uplinks claimed by MLX4_EN driver should be displayed. #> esxcli network nic list In Non Multifunction Mode, port 2 is identified as a pseudo device. The...
Page 57 - You can install the driver in one of the following ways:; Windows
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 57 4.3.2 Installing and Running the offline_bundle Driver on ESXi-5.x 1. Copy the offline_bundle zip file to ESXi 5.0 machine and extract its contents. 2. You can install the...
Page 58 - Microsoft Windows Server operating system; Downloading Mellanox WinOF; Verify that the machine architecture.; Installing Mellanox WinOF
Driver Installation and Configuration Rev 1.1 Mellanox Technologies 58 4.4.1 Installation Requirements 4.4.1.1 Required Disk Space for Installation • 200 MB 4.4.2 Software Requirements • Microsoft Windows Server operating system 4.4.2.1 Installer Privileges • The installation requires administrator ...
Page 59 - From a CMD console, execute the Dell Update Package silently.; Uninstalling Mellanox WinOF; affect Ethernet performance.; WinOF Features; To install the Data Center Bridging using the Server Manager:
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 59 4.4.4.2 Unattended Installation From a CMD console, execute the Dell Update Package silently. Network_Driver_NNNNN_WN_XX.XX.XX.EXE /s 4.4.5 Uninstalling Mellanox WinOF To ...
Page 60 - To configure QoS on the host:
Driver Installation and Configuration Rev 1.1 Mellanox Technologies 60 Step 1. Enable Data Center Bridging (DCB). To configure QoS on the host: Step 1. Change the Windows PowerShell execution policy. Step 2. Remove the entire previous QoS configuration. Step 3. Set the DCBX Willing parameter to fa...
Page 61 - To add the script to the local machine startup scripts:; RDMA over Converged Ethernet; Prerequisites
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 61 Step 9. Enable QoS on the relevant interface. Step 10. Enable PFC on priority 3. To add the script to the local machine startup scripts: Step 1. From the PowerShell invo...
Page 62 - If VLANs are used, Mark the egress traffic with the relevant VlanID.
Driver Installation and Configuration Rev 1.1 Mellanox Technologies 62 Step 4. The DCBX Willing parameter should be set to false as Mellanox drivers do not support this feature. Step 5. Create a Quality of Service (QoS) policy and tag each type of traffic with the relevant priority NOTE: In this exa...
Page 63 - Copying Port Control Protocol (PCP) between Subnets
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 63 Step 11. Enable QoS on the relevant interface. Using Global Pause Flow Control (GFC) To use Global Pause Flow Control (GFC) mode, disable QoS and Priority: 4.5.2.2 Confi...
Page 64 - To verify the SMB connection on the SMB client:
Driver Installation and Configuration Rev 1.1 Mellanox Technologies 64 The NETSTAT command confirms if the File Server is listening on the RDMA interfaces. 4.5.3.3 Verifying SMB Connection To verify the SMB connection on the SMB client: Step 1. Copy the large file to create a new session with the ...
Page 65 - Remote Boot; iSCSI Boot; To configure the iSCSI target:
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 65 5 Remote Boot 5.1 iSCSI Boot 5.1.1 RHEL6.4/RHEL6.5 5.1.1.1 Configuring the iSCSI Target Machine To configure the iSCSI target: Step 1. Download the IET target software f...
Page 66 - Start the iSCSI target service.; Specialized Storage Devices; Click on the; + Add Advanced Target
Remote Boot Rev 1.1 Mellanox Technologies 66 Step 4. Edit the /etc/sysconfig/iscsi-target file as follow. Step 5. Start the iSCSI target service. Step 6. Perform a sanity check by connecting to the iSCSI target from a remote PC on the 10GE network link. 5.1.1.2 Installing RHEL6.4/RHEL6.5 on a Remote...
Page 67 - In the Advanced Storage Options window perform the following:; Bind targets to network interfaces; Enter the IP address of iSCSI target.
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 67 Step 6. In the Advanced Storage Options window perform the following: Step a. Select the Add Iscsi Target option. Step b. Check the Bind targets to network interfaces chec...
Page 68 - Login; Make sure a new storage LUN appears in the; Other AN Devices; Next; Select; Fresh Installation; and proceed with the Installation
Remote Boot Rev 1.1 Mellanox Technologies 68 Step 9. Click Login . A successful login is mandatory to proceed. A failure at this stage is probably a result of a target or network configuration error and recovery/troubleshooting that is out of the scope of this document. Step 10. Make sure a new stor...
Page 70 - . Allow the installation to reach completion.; Booting Windows from an iSCSI Target; Configuring the WDS Server; To configure the WDS server:
Remote Boot Rev 1.1 Mellanox Technologies 70 Step 19. Click Next . Allow the installation to reach completion. 5.1.1.3 SAN-Booting the Diskless Client with FlexBoot When the installation process is completed, the client will ask to reboot. At that point, the DHCP server configuration for that client...
Page 71 - Add the Mellanox driver to install.wim; Configuring iSCSI Target; To configure iSCSI Target:; Configuring the DHCP Server; To configure the DHCP server:; Option
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 71 4. Add the Mellanox driver to install.wim 1 . 5. Add the new boot and install images to WDS. For additional details on WDS, please refer to: http://technet.microsoft.com/e...
Page 72 - firm that the Option ROM setting is set to Enabled.
Remote Boot Rev 1.1 Mellanox Technologies 72 7. In Device Settings for the adapter, confirm that Legacy Boot Protocol is set to iSCSI and con- firm that the Option ROM setting is set to Enabled. 5.1.2.3 Installing iSCSI 1. Reboot your iSCSI client. 2. Press F12 when asked to proceed to iSCSI boot. 3...
Page 73 - Choose the Operating System to install.
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 73 4. Choose the Operating System to install. 5. Run the Windows Setup Wizard. 6. Choose iSCSI target drive to install Windows and follow the instructions presented by the in...
Page 74 - Create the IQN in the ietd configuration file.
Remote Boot Rev 1.1 Mellanox Technologies 74 Step 2. Install iSCSI target and additional required software on target server. Step 3. Create the IQN in the ietd configuration file. Step 4. Edit the /etc/sysconfig/iscsi-target file as follow. Step 5. Start the iSCSI target service. Step 6. Perform a s...
Page 76 - Reboot the client and invoke PXE boot with the Mellanox boot agent.
Remote Boot Rev 1.1 Mellanox Technologies 76 Step 2. Reboot the client and invoke PXE boot with the Mellanox boot agent. Step 3. Select the "Install SLES11.3" boot option from the menu (see pxelinux.cfg example above). After about 30 seconds, the SLES installer will issue the notification be...
Page 77 - Configure iSCSI Disks; Choose; Connected Targets
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 77 Step 5. Click on the Configure iSCSI Disks button. Step 6. Choose Connected Targets tab.
Page 78 - Add; Enter the IP address of the iSCSI storage target.
Remote Boot Rev 1.1 Mellanox Technologies 78 Step 7. Click Add . Step 8. Enter the IP address of the iSCSI storage target. Step 9. Click Next . Step 10. Select the relevant target from the table (In our example, only one target exist so only one was discovered).
Page 79 - Connect; onboot; to exit the discovery screen.
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 79 Step 11. Click Connect . Step 12. Select onboot from drop-list. Step 13. Click Next to exit the discovery screen.
Page 81 - Physical Machine
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 81 Step 20. Select Physical Machine . Step 21. Click Next . Step 22. Click Install . Make sure "open-iscsi" RPM is selected for the installation under "Software&#...
Page 82 - Complete post installation configuration steps.; PXE Boot; The following are examples of PXE server configuration:; Discover and connect to a remote iSCSI storage.
Remote Boot Rev 1.1 Mellanox Technologies 82 Step 23. Complete post installation configuration steps. 5.1.3.4 Using PXE Boot Services for Booting the SLES11 SP3 from the iSCSI Target Once the installation is completed, the system will reboot. At this stage, it is expected from the client to perform ...
Page 83 - The following is known to work with Mellanox NIC:
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 83 If the iso method above is not used, two different PXE server configurations are required (PXELINUX booting labels) for each phase discussed herein (booting the installer ...
Page 84 - Firmware
Firmware Rev 1.1 Mellanox Technologies 84 6 Firmware Firmware and update instructions for these cards can be obtained from the Dell support web page: http://www.dell.com/support . Note: The firmware version on the adapter can be checked using the following methods: 1. System Setup > Device Settin...
Page 85 - Dell Lifecycle Controller; Troubleshooting; General
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 85 7 Dell Lifecycle Controller Troubleshooting 7.1 General Server unable to find the adapter • Ensure that the adapter is placed correctly • Make sure the adapter slot and th...
Page 86 - Linux
Dell Lifecycle ControllerTroubleshooting Rev 1.1 Mellanox Technologies 86 7.2 Linux 7.3 Windows Firmware Version Upgrade To download the latest firmware version refer to the Dell support site http://www.dell.com/support Environment Information cat/etc/issueuname –acat/proc/cupinfo | grep ‘model name...
Page 88 - Specifications; Physical; Protocol Support; Power and; Regulatory
Specifications Rev 1.1 Mellanox Technologies 88 8 Specifications Table 16 - Mellanox ConnectX-3 Dual 40GbE QSFP+ Network Adapter Specifications Physical Board Size: 2.71in. x 5.6in. (68.90 mm x 142.25 mm) Full height Bracket Size: 4.5 in. (116 mm) Low profile Bracket Size: 3.16 in. (80.3 mm) Connect...
Page 91 - Regulatory Statements; This device may not cause harmful interference, and
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 91 9 Regulatory 9.1 Regulatory Statements 9.1.1 FCC Statements (USA) Class A Statements: § 15.19(a)(4) This device complies with Part 15 of the FCC Rules. Operation is subjec...
Page 92 - Warning
Regulatory Rev 1.1 Mellanox Technologies 92 9.1.2 EN Statements (Europe) EN55022 Class A Statement: Warning This is a class A product. In a domestic environment this product may cause radio interference in which case the user may be required to take adequate measures. 9.1.3 ICES Statements (Canada) ...
Page 94 - Appendix A: Configuration for Mellanox Adapters through
Rev 1.1 Mellanox Technologies 94 Appendix A: Configuration for Mellanox Adapters through System Setup This section covers the main configuration options in Dell PowerEdge System Setup which can be accessed through BIOS or through Lifecycle Controller. Figure 3: System Setup Menu Setup menu options: ...
Page 95 - Shows general info regarding the adapter.
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 95 To configure the Mellanox Adapter, choose 'Device settings' and the relevant Mellanox adapter: Figure 4: Main Configuration Page Options 1. Shows general info regarding th...
Page 97 - Main Configuration Page - Firmware Image Properties; The below provides Firmware the uEFI versions numbers.; Main Configuration Page - NIC Configuration; Allows configuration of Virtual LAN Mode and Virtual LAN ID.
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 97 A.1 Main Configuration Page - Firmware Image Properties The below provides Firmware the uEFI versions numbers. 1 A.2 Main Configuration Page - NIC Configuration 1. Allows ...
Page 100 - Main Configuration Page - iSCSI Configuration
Rev 1.1 Mellanox Technologies 100 A.3 Main Configuration Page - iSCSI Configuration This section allows to override the default configurations of iSCSI and replaces DHCP configu-ration of iSCSI.
Page 103 - Main Configuration Page - Device Level Configuration
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 103 Figure 7: Main Configuration Page - iSCSI Configuration - iSCSI Target Parameters A.4 Main Configuration Page - Device Level Configuration Allows setting Global Flow cont...
Page 104 - SR-IOV Configuration
Rev 1.1 Mellanox Technologies 104 A.5 SR-IOV Configuration Enabling SRIOV requires configuration both for the system and the specific Mellanox adapter. To enable SR-IOV - follow the below steps 1-4. To disable SR-IOV - set the configuration in steps 1-3 to disabled.
Page 105 - Enable
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 105 Step 1. Enable “ Virtualization Technology” in System BIOS => Processor setting : Step 2 . Enable “ SR-IOV Global Enable” Go to: System BIOS => integrated Devices s...
Page 107 - . Reboot the server for the SR-IOV configuration to take effect.
User Manual for Mellanox ConnectX®-3 10/40 Gigabit Ethernet Adapters for Dell PowerEdge Servers Rev 1.1 Mellanox Technologies 107 Step 4 . Reboot the server for the SR-IOV configuration to take effect.
Page 108 - Appendix B: Safety Warnings; Installation Instructions
Rev 1.1 Mellanox Technologies 108 Appendix B: Safety Warnings Below is a list of safety warnings in English. For safety warnings in other languages, please refer to the appendices in this manual. 1. Installation Instructions 2. Over-temperature 3. During Lightning - Electrical Hazard 4. Copper Cable...
Page 110 - Appendix C: Avertissements de sécurité d’installation
Rev 1.1 Mellanox Technologies 110 Appendix C: Avertissements de sécurité d’installation (Warnings in French) 1. Instructions d’installation 2. Température excessive 3. Orages – dangers électriques 4. Branchement/débranchement des câbles en cuivre 5. Installation du matériel 6. Elimination du matérie...
Page 112 - Installationsanleitungen
Rev 1.1 Mellanox Technologies 112 Appendix D: Sicherheitshinweise (Warnings in German) 1. Installationsanleitungen 2. Übertemperatur 3. Bei Gewitter - Elektrische Gefahr 4. Anschließen/Trennen von -Kupferkabel 5. Geräteinstallation 6. Geräteentsorgung 7. Regionale und nationale elektrische Bestimmun...
Page 114 - Appendix E: Advertencias de seguridad para la insta-
Rev 1.1 Mellanox Technologies 114 Appendix E: Advertencias de seguridad para la insta- lación (Warnings in Spanish) 1. Instrucciones de instalación 2. Sobrecalentamiento 3. Cuando hay rayos: peligro de descarga eléctrica 4. Conexión y desconexión del cable Copper 5. Instalación de equipos 6. Elimina...
Dell 04YJJ6A00
User Manual
Dell 10 Pro
User Manual
Dell 10
Manual
Dell 1000
User Manual
Dell 1014
User Manual
Dell 1000W
User Manual
Dell 0JH552A01
User Manual
Dell 10g
User Manual
Dell 1100
User Manual
Dell 11
User Manual
Dell 11 Pro
User Manual
Dell 110T
User Manual
Dell 110T DLT1 Drive
User Manual
Dell 1100 Laser Mono Printer
User Manual
Dell 1100MP
User Manual