Sun Microsystems 10GBE XFP - Manuals

vi Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 A. Specifications 71 Connectors 71 Performance Specifications 72 Physical Characteristics 73 Power Requirements 73 B. Diagnostic Software 75 SunVTS Diagnostic Software 75 Updating SunVTS to Recognize the Adapter 76 ▼ To Update SunVTS...

vii Tables TABLE 1-1 Hardware and Software Requirements for Sun SPARC and X64 Servers 2 TABLE A-1 XFP Connector Characteristics 72 TABLE A-2 Performance Specifications 72 TABLE A-3 Physical Characteristics 73 TABLE A-4 Low-Profile Power Requirements 73 TABLE B-1 SunVTS Documentation 76

ix Figures FIGURE 1-1 Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter 2 FIGURE 8-1 Example of Servers Supporting Multiple VLANs With Tagging Adapters 68 FIGURE 8-2 Ethernet Tag Header Format 70 FIGURE A-1 Sun x8 Express Dual 10 Gigabit Ethernet XFP Low Profile Adapter Connector...

xi Declaration of Conformity EMC USA—FCC Class A This equipment complies with Part 15 of the FCC Rules. Operation is subject to the following two conditions:1. This equipment may not cause harmful interference.2. This equipment must accept any interference that may cause undesired operation. Europea...

xiii Preface This guide provides instructions for installing both the hardware and software forthe Sun Dual 10GbE XFP PCI Express card. This manual also describes how toconfigure the nxge driver, which controls the card. These instructions are designed for enterprise system administrators with exper...

xiv Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 Chapter 6 describes how to configure the Jumbo Frame feature. Chapter 7 describes how to configure link aggregation. Chapter 8 explains Virtual Local Area Networks (VLANs) in detail and provides configuration instructions and exampl...

Preface xv Shell Prompts Typographic Conventions Shell Prompt C shell machine-name % C shell superuser machine-name # Bourne shell and Korn shell $ Bourne shell and Korn shell superuser # Typeface * * The settings on your browser might differ from these settings. Meaning Examples AaBbCc123 The names...

xvi Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 Related Documentation The documents listed as online are available at: http://www.sun.com/documentation/ Documentation, Support, and Training Third-Party Web Sites Sun is not responsible for the availability of third-party web sites...

Preface xvii Sun Welcomes Your Comments Sun is interested in improving its documentation and welcomes your comments andsuggestions. You can submit your comments by going to: http://www.sun.com/hwdocs/feedback Please include the title and part number of your document with your feedback: Sun Dual 10Gb...

1 C H A P T E R 1 Product Overview This chapter provides an overview of the Sun x8 Express Dual 10 Gigabit EthernetFiber XFP Low Profile Adapter, including: ■ “Components” on page 1 ■ “Product Description” on page 1 ■ “Hardware and Software Requirements” on page 2 ■ “Features” on page 3 Components T...

2 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 FIGURE 1-1 Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter Hardware and Software Requirements Before installing the adapter, make sure your system meets the hardware andsoftware requirements. TABLE 1-1 lists the ...

Chapter 1 Product Overview 3 Features The Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapterprovides the following features: ■ Two full-duplex 10 gigabit Ethernet interface that use a 10 gigabit Ethernet smallform factor pluggable (XFP), optical transceiver with duplex LC fiber co...

5 C H A P T E R 2 Installing and Setting Up the Driver This chapter explains how to download and install the nxge driver. The nxge gigabit Ethernet driver ( nxge (7D)) is a multi-threaded, loadable, clonable, GLD-based STREAMS driver. The nxge driver is managed by the dladm (1M) command line utility...

Chapter 2 Installing and Setting Up the Driver 7 5. For x86 systems, install the software packages by typing the following at the command line: A menu similar to the following displays: 6. Select the packages you want to install: ■ Press Return or type all to accept the default and install all packa...

8 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 7. Verify that the nxge driver is installed on the system: ▼ To Remove the Driver from a Solaris Platform 1. To discover the driver packages run the pkginfo command: 2. To remove the driver packages run the pkgrm command: Sun PCI-E 10...

Chapter 2 Installing and Setting Up the Driver 9 Downloading and Installing the Driveron a Linux Platform 1. Login to your system. 2. Download the driver RPM for your operating system: http://www.sun.com/download/products.xml?id=44eb1efd For example: 3. Discover the network interfaces before adding ...

10 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 Note – In RHEL5.0 and later releases, the driver is packaged in the kmod driver binary package format. This packaging allows forward and backward driver binarycompatibility within the same flavors of RHEL5 releases. Driver packages n...

Chapter 2 Installing and Setting Up the Driver 11 7. Use the ethtool command to check the parameter configurations that apply to the nxge driver. ▼ To Remove the Driver From a Linux Platform ● To remove the driver packages from a Linux Platform use the rpm -e command: (For 10G) # ethtool -i eth4 dri...

13 C H A P T E R 3 Installing the Adapter This chapter describes how to install the Sun x8 Express Dual 10 Gigabit EthernetFiber XFP Low Profile Adapter in your system and verify that it has been installedcorrectly. This chapter contains the following section: ■ “Installing the Adapter” on page 13 ■...

14 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 Adapter in slot-2 of a Sun Fire T2000 system. The SATA cables from the adjacentcard slot prevent the proper installation of an Sun x8 Express Dual 10 GigabitEthernet Fiber XFP Low Profile Adapter in this slot. (Note that this issue d...

Chapter 3 Installing the Adapter 15 Note – These illustrations show the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter being installed into a Sun Fire T2000, which requires astandard height bracket for the low profile adapter. 6. Open the retaining clip at the edge of the PCI-...

16 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 9. Secure the retaining clip Caution – Do not use excessive force when installing the adapter into the PCI-E slot. You might damage the adapter ’s PCI connector. If the adapter does not seatproperly when you apply even pressure, remo...

Chapter 3 Installing the Adapter 17 11. Install the optical transceivers. Note – Illustrations for installing the adapter and installing the optical transceiver show different systems. This difference in systems provides the maximum clarity forinstalling the optical tranceiver and shows the most com...

18 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 ▼ To Install an Optical Transceiver 1. Pull the white locking handle into the full horizontal position. You will feel the handle click into position when it is fully opened. Note – If you are installing an optical transceiver in a Su...

Chapter 3 Installing the Adapter 19 4. Push the handle closed to lock the optical transceiver in place. 5. Repeat Step 1 through Step 4 to install the second optical transceiver 6. Detach the wrist strap. Caution – If you pull the locking handle down when the optical transceiver is installed, remove...

20 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 7. Connect the Ethernet cables. Verifying the Hardware Installation After you have installed the Sun 10-Gigabit Ethernet adapter, but before you boot your system, perform the following tasks to verify the installation. Refer to the y...

Chapter 3 Installing the Adapter 21 3. View the device that you installed. Using the previous example, type: 4. Use the .properties command to display a list of device properties. The . properties command displays the specific information about the device. If you are using the Sun x8 Express Dual 10...

22 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 5. Type the following when you finish looking at the .properties values: Rebooting the System After verifying the adapter installation, use the boot -r command to perform a reconfiguration boot on your system. ok device-end ok boot -...

23 C H A P T E R 4 Network Configuration This chapter describes how to edit the network host files after the adapter has beeninstalled on your system. This chapter contains the following sections: ■ “Configuring the Network Host Files” on page 23 ■ “Setting Up a 10-Gigabit Ethernet Network on a Disk...

24 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 1. At the command line, use the grep command to search the /etc/path_to_inst file for nxge interfaces. In this example, the device instance is from a Sun x8 Express Dual 10 GigabitEthernet Fiber XFP Low Profile Adapter installed in s...

Chapter 4 Network Configuration 25 ■ The host name must be different from any other host name of any other interface,for example: /etc/hostname.nxge0 and /etc/hostname.nxge1 cannot share the same host name. The following example shows the /etc/hostname.nxge number file required for a system called z...

26 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 ▼ To Set Up a 10-Gigabit Ethernet Port on aDiskless Client 1. Locate the root directory of the diskless client on the host server. The root directory of diskless client system is commonly installed in the hostserver ’s /export/root/ ...

Chapter 4 Network Configuration 27 Installing the Solaris Operating SystemOver a 10-Gigabit Ethernet Network The Solaris Advanced Installation Guide describes the full procedure for installing the Solaris Operating System over the network. The following procedure assumes thatyou have created an inst...

28 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 2. Find the root directory of the client system. The client system’s root directory can be found in the install server ’s /etc/bootparams file. Use the grep command to search this file for the root directory. In this example, the roo...

Chapter 4 Network Configuration 29 5. Shut down and halt the client system. Use the shutdown command to go to the OpenBoot ( ok ) prompt. 6. At the ok prompt, use the show-nets command to find the device path of the 10- Gigabit Ethernet device. The show-nets command lists the system devices. You sho...

30 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 9. After installing the Solaris Operating System, install the Sun x8 Dual 10 Gigabit Ethernet software on the client system. The software installed in Step 4 is required to boot the client system over the 10- Gigabit Ethernet interfa...

Chapter 4 Network Configuration 31 Booting Over the 10-Gigabit EthernetNetwork for Solaris x86 and LinuxSystems ▼ To Boot Over the Network on Solaris x86 andLinux Systems 1. Obtain the MAC address from the target Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter. 2. Set up the PX...

32 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 9. Press the F10 key to save the boot configuration changes and exit. System should reboot after saving the boot configuration.

Chapter 4 Network Configuration 33 10. Press the F12 key to install the OS from the network. If the cable is connected to the right port, you should see the MAC address that youassigned to your PXE server displayed by BIOS. 11. You can now install the nxge driver and configure the adapter. image : p...

35 C H A P T E R 5 Configuring the nxge Device Driver Parameters The nxge device driver controls the Sun x8 Express Dual 10 Gigabit Ethernet interfaces. You can manually set the nxge driver parameters to customize each device in your system. This chapter lists the available device driver parameters ...

36 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 processing. The Sun 10-Gigabit Ethernet Adapter utilizes Sun’s own innovativeMAC Controller to map the 10-Gigabit XAUI interface onto the PCI Express formfactor. It supports 10 Gb/sec bandwidth using eight transmit and eight receive ...

Chapter 5 Configuring the nxge Device Driver Parameters 37 ▼ To Specify Device Instances for the ndd Utility Before you use the ndd utility to get or set a parameter for a nxge device, you must specify the device instance for the utility. 1. Check the /etc/path_to_inst file to identify the instance ...

38 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 After you enter the parameter name, the ndd utility prompts you for the parameter value. 2. To list all the parameters supported by the nxge driver, type ? . Setting Parameters Using the nxge.conf File Specify the driver parameter pr...

Chapter 5 Configuring the nxge Device Driver Parameters 39 ▼ To Set Driver Parameters Using an nxge.conf File 1. Obtain the hardware path names for the nxge devices in the device tree. a. Check the /etc/driver_aliases file to identify the name associated with a particular device: b. Locate the path ...

42 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 b. The following parameters operate on a per port basis and can be set using the /platform/sun4u/kernel/drv/nxge.conf file. c. In the following example, the ports of all the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profi...

Chapter 5 Configuring the nxge Device Driver Parameters 43 d. The following example shows ports on two different cards being set. Only one node needs to be specified. 3. Save the nxge.conf file. Tuning for Maximum Performance on aSolaris Platform Tuning for maximum performance in a Solaris platform ...

44 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 3. Add the following to a startup script, or use ndd before plumbing the interface: Utilizing more soft-rings provided by the Solaris TCP/IP stack significantlyimproves bulk throughput for Rx. The default number of soft-rings is 2, b...

Chapter 5 Configuring the nxge Device Driver Parameters 45 ▼ To Obtain Higher Throughput Using the GenericTunables for the Solaris TCP/IP Stack ● To obtain higher throughput, add the following to a startup script: ndd -set /dev/tcp tcp_conn_req_max_q 8192 ndd -set /dev/tcp tcp_conn_req_max_q0 8192 n...

46 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 Setting Parameters on a Linux Platform ▼ To Set Parameters Using the ethtool Utility 1. Determine which parameters are available using the ethtool utility: # ethtool -help eth4 ethtool version 1.8 Usage: ethtool DEVNAME ethtool -a DE...

48 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 Following are some common parameters that can be changed: rx-usecs and rx-frames control the RX interrupt rate per RX DMA channel. RXinterrupt will be generated after rx-frames have been received or after rx-usecs timeinterval if few...

Chapter 5 Configuring the nxge Device Driver Parameters 49 2. To change RX interrupt Coalesce parameters use the ethtool -C command: 3. To get status of L4 HW checksumming, use the ethtool -k command: # ethtool -C eth4 rx-usecs 20 # ethtool -c eth4 Coalesce parameters for eth4: Adaptive RX: off TX: ...

50 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 ▼ To Set Parameters Using the Bundled configtool Utility 1. To get a list of tunable parameters, use the nxge_config if_name get command: These classification variables define how each IP class is configured. This parameteralso contr...

Chapter 5 Configuring the nxge Device Driver Parameters 51 3. To set a particular variable, use the /usr/local/bin/nxge_config if_name set param_name value : Tuning for Maximum Performance on aLinux Platform The following tunings will improve the performance of the Sun x8 Express Dual 10Gigabit Ethe...

53 C H A P T E R 6 Configuring the Jumbo FramesFeature This chapter describes how to configure the Jumbo Frames feature. It contains thefollowing sections: ■ “Jumbo Frames Overview” on page 53 ■ “Checking Jumbo Frames Configurations” on page 53 ■ “Enabling Jumbo Frames in a Solaris Environment” on p...

54 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 ▼ To Show the Driver Statistics in a SolarisEnvironment 1. Use the kstat command to display driver statistics, for example: The previous example diplays the receive packet counts on all of the eight ReceiveDMA channels on interface 1...

56 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 ▼ To Enable Jumbo Frames in a SolarisEnvironment Using nxge.conf 1. Enable Jumbo Frames for a port using the nxge.conf file. For example, 2. Reboot the system: 3. Set the maximum MTU for maximum performance: ▼ To Check Layer 2 Config...

Chapter 6 Configuring the Jumbo Frames Feature 57 ▼ To Check Layer 3 Configuration ● Check the Layer 3 configuration by using the dladm command with the show-link option, as shown in the following example: Enabling Jumbo Frames in a LinuxEnvironment This section describes how to enable jumbo frames ...

58 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 3. Set the MTU for maximum performance: Where x is the instance number of the interface you want to configure jumbo frames on. ▼ To Show the Driver Statistics in a LinuxEnvironment 1. Use the ifconfig utility to display driver statis...

Chapter 6 Configuring the Jumbo Frames Feature 59 3. To get more specific statitics, use the grep command on the output of ethtool -S: # ethtool -S eth9 |grep tx_pac tx_packets: 8645339 tx_packets: 3418334 tx_packets: 26 tx_packets: 0 tx_packets: 22 tx_packets: 401791 tx_packets: 1466540 tx_packets:...

61 C H A P T E R 7 Configuring Link Aggregation This chapter describes how to configure link aggregation. It contains the followingsections: ■ “Overview of Link Aggregation” on page 61 ■ “Configuring Link Aggregation in a Linux Environment” on page 64 Overview of Link Aggregation Link Aggregation al...

62 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 ▼ To Configure Link Aggregation in a SolarisEnvironment 1. Aggregate nxge0 and nxge1 to form an aggregation and a random number 33 as key. a. Unplumb the interfaces to be aggregated: b. Create a link-aggregation group with key 33 wit...

Chapter 7 Configuring Link Aggregation 63 2. Plumb up the interface aggr key , which is aggr33 is this case: 3. Show link-aggregation status again, and now the state should become attached: 4. Use the dladm show-aggr -s command to display statistics: # ifconfig aggr33 plumb # ifconfig aggr33 aggr33:...

64 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 5. Use the dladm show-aggr -L command to display LACP specific information: For more information refer to the man pages for dladm , man dladm . Configuring Link Aggregation in aLinux Environment This section explains how to configure...

Chapter 7 Configuring Link Aggregation 65 3. Configure bond0 interface. In this example, bond0 is the master of the two interfaces eth4 and eth5. # ip addr add 192.12.38.64/24 brd + dev bond0 # ip link set dev bond0 up # ifenslave bond0 eth4 eth5

67 C H A P T E R 8 Configuring VLANs This chapter explains Virtual Local Area Networks (VLANs) in detail and providesconfiguration instructions and examples. It contains the following sections: ■ “Overview of VLANs” on page 67 ■ “Configuring VLANs in a Solaris Environment” on page 69 ■ “Configuring ...

68 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 FIGURE 8-1 Example of Servers Supporting Multiple VLANs With Tagging Adapters The example network has the following features: The physical LAN network consists of a switch, two servers, and five clients. TheLAN is logically organized...

Chapter 8 Configuring VLANs 69 The Main Server is a heavily used server that needs to be accessed from all VLANsand IP subnets. The server has a Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFPLow Profile Adapter installed. All three IP subnets are accessed by means of thesingle physical adapter in...

70 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 Tagging an Ethernet frame requires the addition of a tag header to the frame. Theheader is inserted immediately following the destination MAC address and theSource MAC address. The tag header consists of two bytes of Ethernet Tag Pro...

Chapter 8 Configuring VLANs 71 2. Use the ifconfig(1M) to configure a VLAN virtual device, for example: The output of ifconfig -a on a system having VLAN devices nxge123000 and nxge224000 : 3. On the switch, set VLAN tagging and set VLAN ports to coincide with the VLANs you’ve set up on the server. ...

73 A P P E N D I X A Specifications This appendix lists the specifications for the Sun x8 Express Dual 10 Gigabit Ethernet FiberXFP Low Profile Adapter. It contains the following sections: ■ “Connectors” on page 73 ■ “Performance Specifications” on page 74 ■ “Physical Characteristics” on page 75 ■ “...

74 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 TABLE A-1 lists the characteristics of the fiber connectors used by the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter. Performance Specifications TABLE A-1 XFP Connector Characteristics Parameter Description C...

77 A P P E N D I X B Diagnostic Software This appendix provides an overview of the SunVTS diagnostic application andinstructions for updating the SunVTS software to recognize the adapter. Thisappendix contains the following sections: ■ “SunVTS Diagnostic Software” on page 77 ■ “Updating SunVTS to Re...

78 Sun Dual 10GbE XFP PCI Express Card User’s Guide • January 2009 http://www.sun.com/products-n-solutions/hardware/docs/Software /system_administration/tools/testing_valid/sunvts/index.html Select the document for the Solaris release on your system. Updating SunVTS to Recognize theAdapter Use SunVT...

Appendix B Diagnostic Software 79 Using the SunVTS netlbtest You must have the Ethernet card and the device driver installed, a loopbackconnector in place, and Intervention mode enabled before running netlbtest . netlbtest cannot run if the network interface is connected to a live network, and requi...

81 Index Symbols /etc/hostname.nxge number file , 24 A assigning an IP address, 24 C configuring the network host files, 23configuring VLANs, 69 D diskless clients, 25documentation SunVTS, 78 E editing the network host files, xiii, 23 H hardware components, 1requirements, 2 I ifconfig -a, 71ifconfig...