Page 3 - International Technical Support Organization
IBM Certification Study GuideAIX HACMP May 1999 SG24-5131-00 International Technical Support Organization
Page 5 - iii; C o n t e n t s
© Copyright IBM Corp. 1999 iii C o n t e n t s Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .ix Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xi Preface . . . . . . . . ...
Page 6 - iv
iv IBM Certification Study Guide AIX HACMP Chapter 3. Cluster Hardware and Software Preparation . . . . . . . . . . . 513.1 Cluster Node Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 3.1.1 Adapter Slot Placement . . . . . . . . . . . . . . . . . . . . . . . . ....
Page 8 - vi
vi IBM Certification Study Guide AIX HACMP 8.1.1 The clstat Command . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1528.1.2 Monitoring Clusters using HAView . . . . . . . . . . . . . . . . . . . . . . 1528.1.3 Cluster Log Files . . . . . . . . . . . . . . . . . . . . . . . . . . ....
Page 9 - vii
vii 9.3 VSDs - RVSDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 9.3.1 Virtual Shared Disk (VSDs) . . . . . . . . . . . . . . . . . . . . . . . . . . . 1909.3.2 Recoverable Virtual Shared Disk . . . . . . . . . . . . . . . . . . . . . . . 193 9.4 SP Switch a...
Page 10 - viii
Page 11 - ix; F i g u r e s
© Copyright IBM Corp. 1999 ix F i g u r e s 1. Basic SSA Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2. Hot-Standby Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3. Mutual Takeover Configuration . . . . . . ...
Page 13 - xi; Ta b l es
© Copyright IBM Corp. 1999 xi Ta b l es 1. AIX Version 4 HACMP Installation and Implementation . . . . . . . . . . . . . . . 4 2. AIX Version 4 HACMP System Administration . . . . . . . . . . . . . . . . . . . . . . 5 3. Hardware Requirements for the Different HACMP Versions . . . . . . . . . . . . ...
Page 14 - xii
Page 15 - xiii; Pr e f a ce; real world
xiii Pr e f a ce The AIX and RS/6000 Certifications offered through the Professional Certification Program from IBM are designed to validate the skills required of technical professionals who work in the powerful and often complex environments of AIX and RS/6000. A complete set of professional certi...
Page 16 - xiv; correct settings; T h e Te a m T h a t Wr o t e T h i s R e d b o o k
xiv IBM Certification Study Guide AIX HACMP • AIX parameters that are affected by an HACMP installation, and their correct settings • The cluster and resource configuration process, including how to choose the best resource configuration for a customer requirement • Customization of the standard HAC...
Page 17 - xv; C o m m e n t s We l c o m e; Your comments are important to us!; to the fax number shown on the form.
xv POWERparallel Systems area, known as the SP1 at that time. In 1997 he began working on HACMP as the Service Groups for HACMP and RS/6000 SP merged into one. He holds a diploma in Computer Science from the University of Frankfurt in Germany. This is his first redbook. Reinhard Zettler is an AIX So...
Page 18 - xvi
Page 19 - C h a p t e r 1 . C e r t i f i ca t i o n O ve r vi e w; . 1 I B M C e r t i f i e d S p e c i a l i s t - A I X H A C M P; C e rt if i ca ti o n R eq u ir em en t ( t w o Test s ) :; Test 181: AIX V4.3 System Administration; or; R e co m m en d e d P re re q u i s it es
© Copyright IBM Corp. 1999 1 C h a p t e r 1 . C e r t i f i ca t i o n O ve r vi e w This chapter provides an overview of the skill requirements for obtaining an IBM Certified Specialist - AIX HACMP certification. The following chapters are designed to provide a comprehensive review of specific top...
Page 20 - . 2 C e r t i f i c a t i o n E x a m O b j e c t i v e s; S ect i o n 1 - P re in s t al l at i o n
2 IBM Certification Study Guide AIX HACMP 1 . 2 C e r t i f i c a t i o n E x a m O b j e c t i v e s The following objectives were used as a basis for what is required when the certification exam was developed. Some of these topics have been regrouped to provide better organization when discussed i...
Page 21 - S ect i o n 3 - S ys t em M a n ag e m e n t
Certification Overview 3 • Create an application server. • Set up Event Notification. • Set up event notification and pre/post event scripts. • Set up error notification. • Post Configuration Activities. • Configure a client notification and ARP update. • Implement a test plan. • Create a snapshot. ...
Page 22 - . 3 C e r t i f i c a t i o n E d u c a t i o n C o u r s e s; Course Number
4 IBM Certification Study Guide AIX HACMP 1 . 3 C e r t i f i c a t i o n E d u c a t i o n C o u r s e s Courses and publications are offered to help you prepare for the certification tests. These courses are recommended, but not required, before taking a certification test. At the printing of this...
Page 25 - C h a p t e r 2 . C l u s t e r P l a n n i n g; . 1 C l u s t e r N o d e s; The following sections will deal with the various:; no single point of failure; server
© Copyright IBM Corp. 1999 7 C h a p t e r 2 . C l u s t e r P l a n n i n g The area of cluster planning is a large one. Not only does it include planning for the types of hardware (CPUs, networks, disks) to be used in the cluster, but it also includes other aspects. These include resource planning...
Page 26 - . 1 . 2 C l u s t e r N o d e C o n s i d e r a t i o n s
8 IBM Certification Study Guide AIX HACMP RISC System/6000 models as nodes in an HACMP 4.1 for AIX, HACMP 4.2 for AIX, or HACMP 4.3 for AIX cluster. Table 3. Hardware Requirements for the Different HACMP Versions 1 AIX 4.3.2 required For a detailed description of system models supported by HACMP/600...
Page 27 - Much of the decision centers around the following areas:
Cluster Planning 9 Much of the decision centers around the following areas: • Processor capacity • Application requirements • Anticipated growth requirements • I/O slot requirements These paradigms are certainly not new ones, and are also important considerations when choosing a processor for a sing...
Page 28 - The switch adapter is onboard and does not need an extra slot.
10 IBM Certification Study Guide AIX HACMP Your slot configuration must also allow for the disk I/O adapters you need to support the cluster’s shared disk (volume group) configuration. If you intend to use disk mirroring for shared volume groups, which is strongly recommended, then you will need to ...
Page 29 - . 2 C l u s t e r N e t wo r k s; • Generic IP
Cluster Planning 11 2 . 2 C l u s t e r N e t wo r k s HACMP differentiates between two major types of networks: TCP/IP networks and non-TCP/IP networks. HACMP utilizes both of them for exchanging heartbeats. HACMP uses these heartbeats to diagnose failures in the cluster. Non-TCP/IP networks are us...
Page 30 - • FDDI
12 IBM Certification Study Guide AIX HACMP • FDDI • SP Switch • SLIP • SOCC • Token-Ring As an independent, layered component of AIX, the HACMP for AIX software works with most TCP/IP-based networks. HACMP for AIX has been tested with standard Ethernet interfaces (en*) but not with IEEE 802.3 Ethern...
Page 31 - xxxxxxxxxxyy; is replaced with the first five pairs of digits; yy; can be chosen freely. There; • Token-Ring supports 4 or 16 Mbps, but 4 Mbps is very rarely used now. It
Cluster Planning 13 Network types also differentiate themselves in the maximum distance they allow between adapters, and in the maximum number of adapters allowed on a physical network. • Ethernet supports 10 and 100 Mbps currently, and supports hardware address swapping. Alternate hardware addresse...
Page 32 - • SP Switch is a high-speed packet switching network, running on the
14 IBM Certification Study Guide AIX HACMP • SP Switch is a high-speed packet switching network, running on the RS/6000 SP system only. It runs bidirectionally up to 80 MBps, which adds up to 160 MBps of capacity per adapter. This is node-to-node communication and can be done in parallel between eve...
Page 33 - ports
Cluster Planning 15 2 . 2 . 2 . 2 S p e ci a l C o n si d e r a t i o n s As for TCP/IP networks, there are a number of restrictions on non-TCP/IP networks. These are explained for the three different types in more detail below. Serial (RS232) A serial (RS232) network needs at least one available se...
Page 34 - Tar g et - m o d e S S A; . 3 C l u s t e r D i s k s; Monitoring and Managing IBM SSA Disk Subsystems,
16 IBM Certification Study Guide AIX HACMP 2 a PCI Multiport Async Card is required in an S7X model, no native ports 3 only one serial port available for customer use, i.e. HACMP In case the number of native serial ports doesn’t match your HACMP cluster configuration needs, you can extend it by addi...
Page 35 - initiator; issues commands, while a; target; responds with data and status information.; MBps; HOST; SSA Architecture
Cluster Planning 17 SSA subsystems are built up from loops of adapters and disks. A simple example is shown in Figure 1. Figure 1. Basic SSA Configuration Here, a single adapter controls one SSA loop of eight disks. Data can be transferred around the loop, in either direction, at 20 MBps. Consequent...
Page 40 - R A I D Te ch n o l o g y
22 IBM Certification Study Guide AIX HACMP 2 . 3 . 1 . 4 R A I D v s . N o n - R A I D R A I D Te ch n o l o g y RAID is an acronym for Redundant Array of Independent Disks. Disk arrays are groups of disk drives that work together to achieve higher data-transfer and I/O rates than those provided by ...
Page 41 - R A I D L e v e l s 2 a n d 3
Cluster Planning 23 R A I D L e v e l s 2 a n d 3 RAID 2 and RAID 3 are parallel process array mechanisms, where all drives in the array operate in unison. Similar to data striping, information to be written to disk is split into chunks (a fixed amount of data), and each chunk is written out to the ...
Page 42 - RAI D o n t h e 713 3 Dis k Su b s ys t em; • Mirroring is more expensive than RAID, but it provides higher data
24 IBM Certification Study Guide AIX HACMP As with RAID 3, in the event of disk failure, the information can be rebuilt from the remaining drives. RAID level 5 array also uses parity information, though it is still important to make regular backups of the data in the array. RAID level 5 stripes data...
Page 44 - D i s ks
26 IBM Certification Study Guide AIX HACMP 2 . 3 . 2 S C S I D i s k s After the announcement of the 7133 SSA Disk Subsystems, the SCSI Disk subsystems became less common in HACMP clusters. However, the 7135 RAIDiant Array (Model 110 and 210) and other SCSI Subsystems are still in use at many custom...
Page 45 - • Redundant Power Supply; Dual Active; mode, so that each controller controls the
Cluster Planning 27 • Enhanced SCSI-2 Differential Fast/Wide Adapter/A (MCA, FC: 2412, Adapter Label: 4-C); not usable with 7135-110 • SCSI-2 Fast/Wide Differential Adapter (PCI, FC: 6209, Adapter Label: 4-B) • DE Ultra SCSI Adapter (PCI, FC: 6207, Adapter Label: 4-L); not usable with 7135-110 2 . 3...
Page 46 - . 4 R e s o u r c e P l a n n i n g
28 IBM Certification Study Guide AIX HACMP withdraw the 7135 RAIDiant Systems from marketing because it is equally possible to configure RAID on the SSA Subsystems. 2 . 4 R e s o u r c e P l a n n i n g HACMP provides a highly available environment by identifying a set of cluster-wide resources esse...
Page 47 - Inactive Takeover
Cluster Planning 29 • Cascading • Rotating • Concurrent Each of these types describes a different set of relationships between nodes in the cluster, and a different set of behaviors upon nodes entering and leaving the cluster. Cascading Resource Groups: All nodes in a cascading resource group are as...
Page 48 - Cascading; All active nodes have access to the resource group.; Rotating; . 4 . 2 S h a r e d LV M C o m p o n e n t s; • Rotating Standby; H o t - S t an d b y C o n f i g u ra t i o n
30 IBM Certification Study Guide AIX HACMP reintegration, a node remains as a standby and does not take back any of the resources that it had initially served. Concurrent Resource Groups: A concurrent resource group may be shared simultaneously by multiple nodes. The resources that can be part of a ...
Page 49 - R o t a ti n g S t an d b y C o n f i g u r at i o n
Cluster Planning 31 Figure 2. Hot-Standby Configuration In this configuration, there is one cascading resource group consisting of the four disks, hdisk1 to hdisk4, and their constituent volume groups and file systems. Node 1 has a priority of 1 for this resource group while node 2 has a priority of...
Page 50 - Mu t u a l Ta ke o ve r C o n f ig u ra ti o n
32 IBM Certification Study Guide AIX HACMP the cluster becomes a standby node. You must choose a rotating standby configuration if you do not want a break in service during reintegration. Since takeover nodes continue providing services until they have to leave the cluster, you should configure your...
Page 51 - T h i r d - Pa r ty Ta ke o ve r C o n f i g u r at i o n
Cluster Planning 33 When a failed node reintegrates into the cluster, it takes back the resource group for which it has the highest priority. Therefore, even in this configuration, there is a break in service during reintegration. Of course, if you look at it from the point of view of performance, t...
Page 52 - . 4 . 3 I P A d d r e s s Ta k e o v e r
34 IBM Certification Study Guide AIX HACMP Here the resource groups are the same as the ones in the mutual takeover configuration. Also, similar to the previous configuration, nodes 1 and 2 each have priorities of 1 for one of the resource groups, A or B. The only thing different in this configurati...
Page 53 - • Design the network topology; S i n g l e N e t w o rk; one service adapter available to clients. In this setup, a
Cluster Planning 35 • Design the network topology • Define a network mask for your site • Define IP addresses (adapter identifiers) for each node’s service and standby adapters. • Define a boot address for each service adapter that can be taken over, if you are using IP address takeover or rotating ...
Page 54 - D u a l N et w o r k; The following figure shows a dual-network setup:; P o i n t - t o - Po i n t C o n n e ct i o n; only; as a private
36 IBM Certification Study Guide AIX HACMP D u a l N et w o r k A dual-network setup has two separate networks for communication. Nodes are connected to two networks, and each node has two service adapters available to clients. If one network fails, the remaining network can still function, connecti...
Page 55 - N e tw o rk N a m e; A TCP/IP network’s attribute is either public or private.; Public
Cluster Planning 37 The following diagram shows a cluster consisting of two nodes and a client. A single public network connects the nodes and the client, and the nodes are linked point-to-point by a private high-speed SOCC connection that provides an alternate path for cluster and lock traffic shou...
Page 56 - Private; A private network provides communication between nodes only it; Serial; A d a p t e r L a b el; is the name in the; A d a p t er F u n c t i o n
38 IBM Certification Study Guide AIX HACMP SLIP are considered public networks. Note that a SLIP line, however, does not provide client access. Private A private network provides communication between nodes only; it typically does not allow client access. An SOCC line or an ATM network are also priv...
Page 57 - should not; be configured for IP address; Boot Adapter
Cluster Planning 39 until it assumes the shared IP address. Consequently, Clinfo makes known the boot address for this adapter. In an HACMP for AIX environment on the RS/6000 SP, the SP Ethernet adapters can be configured as service adapters but should not be configured for IP address takeover. For ...
Page 58 - applicable, in the nameserver configuration.
40 IBM Certification Study Guide AIX HACMP service label (address) instead of the boot label. If the node should fail, a takeover node acquires the failed node’s service address on its standby adapter, thus making the failure transparent to clients using that specific service address. During the rei...
Page 59 - . 4 . 4 N F S E x p o r t s a n d N F S M o u n t s; Filesystems to Export; Filling in this field sets up what we call an; NFS; . Any file system defined in this; . 5 A p p l i c a t i o n P l a n n i n g
Cluster Planning 41 If you do not use Hardware Address Takeover, the ARP cache of clients can be updated by adding the clients’ IP addresses to the PING_CLIENT_LIST variable in the /usr/sbin/cluster/etc/clinfo.rc file. 2 . 4 . 4 N F S E x p o r t s a n d N F S M o u n t s There are two items concern...
Page 60 - HACMP for; . 5 . 1 P e r f o r m a n c e R e q u i r e m e n t s; HACMP for AIX Planning Guide; . 5 . 2 A p p l i c a t i o n S t a r t u p a n d S h u t d o w n R o u t i n e s
42 IBM Certification Study Guide AIX HACMP application on the takeover node when a fallover occurs. For more information about creating application server resources, see the HACMP for AIX, Version 4.3: Installation Guide , SC23-4278. 2 . 5 . 1 P e r f o r m a n c e R e q u i r e m e n t s In order t...
Page 61 - . 5 . 3 L i c e n s i n g M e t h o d s; must not; conflict; . 5 . 5 C r i t i c a l / N o n - C r i t i c a l P r i o r i t i z a t i o n s; Not e
Cluster Planning 43 2 . 5 . 3 L i c e n s i n g M e t h o d s Some vendors require a unique license for each processor that runs an application, which means that you must license-protect the application by incorporating processor-specific information into the application when it is installed. As a r...
Page 62 - . 6 C u s t o m i z a t i o n P l a n n i n g; HACMP; notify
44 IBM Certification Study Guide AIX HACMP 2 . 6 C u s t o m i z a t i o n P l a n n i n g The Cluster Manager ’s ability to recognize a specific series of events and subevents permits a very flexible customization scheme. The HACMP for AIX software provides an event customization facility that allo...
Page 63 - . 6 . 2 E r r o r N o t i f i c a t i o n; By specifying the following in a file:; errnotify; class through the
Cluster Planning 45 event to inform system administrators that traffic may have to be rerouted. Afterwards, you can use a network_up notification event to inform system administrators that traffic can again be serviced through the restored network. 2. 6.1 . 3 P r ed ic t i v e E v en t E r ro r Co r...
Page 65 - command, which shuts HACMP down
Cluster Planning 47 The above example screen will add a Notification Method to the ODM, so that upon appearance of the HPS_FAULT9_ER entry in the error log, the error notification daemon will trigger the execution of the /usr/sbin/cluster/utilities/clstop -grsy command, which shuts HACMP down gracef...
Page 66 - . 7 U s e r I D P l a n n i n g; rdist
48 IBM Certification Study Guide AIX HACMP 2 . 7 U s e r I D P l a n n i n g The following sections describe various aspects of User ID Planning. 2 . 7 . 1 C l u s t e r U s e r a n d G r o u p I D s One of the basic tasks any system administrator must perform is setting up user accounts and groups....
Page 67 - . 7 . 2 C l u s t e r P a s s w o r d s; As before, this can be done through; rcp; there are tools like; pcp; supper; to distribute information or better files.; . 7 . 3 U s e r H o m e D i r e c t o r y P l a n n i n g
Cluster Planning 49 2 . 7 . 2 C l u s t e r P a s s w o r d s While user and group management is very much facilitated with C-SPOC, the password information still has to be distributed by some other means. If the system is not configured to use NIS or DCE, the system administrator still has to distr...
Page 69 - C h a p t e r 3 . C l u s t e r H a r d w ar e a n d S o f t w a re Pr e p a ra t i o n; . 1 C l u s t e r N o d e S e t u p; PCI Adapter Placement Reference Guide,; . 1 . 2 R o o t v g M i r r o r i n g
© Copyright IBM Corp. 1999 51 C h a p t e r 3 . C l u s t e r H a r d w ar e a n d S o f t w a re Pr e p a ra t i o n This chapter covers the steps that are required to prepare the RS/6000 hardware and AIX software for the installation of HACMP and the configuration of the cluster. This includes con...
Page 70 - mirrorvg
52 IBM Certification Study Guide AIX HACMP mirroring rootvg in order to avoid the impact of the failover time involved in a node failure. In terms of maximizing availability, this technique is just as valid for increasing the availability of a cluster as it is for increasing single-system availabili...
Page 71 - Extend rootvg to hdisk1 by executing the following:; mklvcopy
Cluster Hardware and Software Preparation 53 mirrored. If the dump devices are NOT the paging device, that dump logical volume will not be mirrored. 3. 1. 2 . 1 P r o ce d u reThe following steps assume the user has rootvg contained on hdisk0 and is attempting to mirror the rootvg to a new disk: hdi...
Page 72 - N ote
54 IBM Certification Study Guide AIX HACMP “-m” option. You should consult documentation on the usage of the “-m” option for mklvcopy . 4. Synchronize the newly created mirrors with the following command: 5. Bosboot to initialize all boot records and devices by executing the following command: where...
Page 73 - . 1 . 3 A I X P r e r e q u i s i t e L P P s; The Prerequisites for the HACMP component HAView 4.2 are
Cluster Hardware and Software Preparation 55 3. 1. 2 . 2 N ec es sar y A PA R F i xe s Table 11. Necessary APAR Fixes To determine if either fix is installed on a machine, execute the following: 3 . 1 . 3 A I X P r e r e q u i s i t e L P P s In order to install HACMP and HACMP/ES the AIX setup must...
Page 74 - . 1 . 4 A I X P a r a m e t e r S e t t i n g s; • Network option settings
56 IBM Certification Study Guide AIX HACMP • nv6000.database.obj 4.1.0.0 • nv6000.Features.obj 4.1.2.0 • nv6000.client.obj 4.1.0.0 and for HAView 4.3 • xlC.rte 3.1.4.0 • nv6000.base.obj 4.1.2.0 • nv6000.database.obj 4.1.2.0 • nv6000.Features.obj 4.1.2.0 • nv6000.client.obj 4.1.2.0 3 . 1 . 4 A I X P ...
Page 75 - smit chgsys; AIX Performance Monitoring & Tuning Guide; thewall
Cluster Hardware and Software Preparation 57 and low-water marks. If a process tries to write to a file at the high-water mark, it must wait until enough I/O operations have finished to make the low-water mark. Use the smit chgsys fastpath to set high- and low-water marks on the Change/Show Characte...
Page 76 - rcnfs
58 IBM Certification Study Guide AIX HACMP 3. 1. 4 . 3 E d i t i n g t h e / e t c/ h o s t s F i l e a n d N a m es er ve r C o n f i g u ra t i o nMake sure all nodes can resolve all cluster addresses. See the chapter on planning TCP/IP networks (the section Using HACMP with NIS and DNS) in the HA...
Page 77 - file on each cluster node. Doing so allows the; rcmd; and; rsh; and thus require these
Cluster Hardware and Software Preparation 59 3 . 1 . 4 . 5 E d i t i n g t h e / . r h o s t s F i l e Make sure that each node’s service adapters and boot addresses are listed in the /.rhosts file on each cluster node. Doing so allows the /usr/sbin/cluster/utilities/clruncmd command and the /usr/sb...
Page 78 - . 2 N e t w o r k C o n n e c t i o n a n d Te s t i n g; events. Configuring a notify
60 IBM Certification Study Guide AIX HACMP 3 . 2 N e t w o r k C o n n e c t i o n a n d Te s t i n g The following sections describe important aspects of network connection and testing. 3 . 2 . 1 T C P / I P N e t w o r k s Since there are several types of TCP/IP Networks available within HACMP, th...
Page 79 - • All client traffic be carried over the service adapter
Cluster Hardware and Software Preparation 61 . Figure 9. Connecting Networks to a Hub 3. 2. 1 . 2 I P A d d r es se s a n d S u b n e t sThe design of the HACMP for AIX software specifies that: • All client traffic be carried over the service adapter • Standby adapters be hidden from client applicat...
Page 80 - must; netstat
62 IBM Certification Study Guide AIX HACMP To comply with these rules, pay careful attention to the IP addresses you assign to standby adapters. Standby adapters must be on a separate subnet from the service adapters, even though they are on the same physical network. Placing standby adapters on a d...
Page 81 - arp; TMSCSI
Cluster Hardware and Software Preparation 63 • Scan the /tmp/hacmp.out file to confirm that the /etc/rc.net script has run successfully. Look for a zero exit status. • If IP address takeover is enabled, confirm that the /etc/rc.net script has run and that the service adapter is on its service addres...
Page 82 - TMSSA; smit tty; fastpath to create a tty device on the nodes. On the resulting; disable; . You do not want a getty process being spawned on this interface.; with the; Defined; device is created on a node for each SCSI adapter on the same
64 IBM Certification Study Guide AIX HACMP TMSSA Target-mode SSA is only supported with the SSA Multi-Initiator RAID Adapters (Feature #6215 and #6219), Microcode Level 1801 or later. You need at least HACMP Version 4.2.2 with APAR IX75718. 3. 2. 2 . 2 C o n f i g u r i n g R S2 32Use the smit tty f...
Page 83 - To change the node number use the following command:; cfgmgr; command to configure the
Cluster Hardware and Software Preparation 65 3. 2. 2 . 4 C o n f i g u r i n g Ta rg e t M o d e SS AThe node number on each system needs to be changed from the default of zero to a number. All systems on the SSA loop must have a unique node number. To change the node number use the following comman...
Page 84 - on one node for reading from that device and:; . 3 C l u s t e r D i s k S e t u p; • Each SSA loop must be connected to a valid pair of connectors on the
66 IBM Certification Study Guide AIX HACMP cat /etc/environment > /dev/tmssay.im on the corresponding node for writing. x and y correspond to the appropriate opposite nodenumber. You should see the first command hanging until the second command is issued, and then showing its output. Target Mode ...
Page 85 - smit; A d a p t er R o u t e r
Cluster Hardware and Software Preparation 67 For more information regarding adapters and cabling rules see 2.3.1, “SSA Disks” on page 16 or the following documents: • 7133 SSA Disk Subsystems: Service Guide, SY33-0185-02 • 7133 SSA Disk Subsystem: Operator Guide, GA33-3259-01 • 7133 Models 010 and 0...
Page 86 - A d a p t er D e f in it i o n s; • Support the I/O Control (IOCTL) subroutine call for non service and
68 IBM Certification Study Guide AIX HACMP A d a p t er D e f in it i o n s By issuing the following command, you can check the correct adapter configuration. In order to work correctly, the adapter must be in the “Available” state: The third column in the adapter device line shows the location of t...
Page 87 - D i a g n o s ti cs; A Practical Guide to SSA for AIX; diag
Cluster Hardware and Software Preparation 69 SSA physical disks: • Are configured as pdisk0, pdisk1,...,pdiskN. • Have errors logged against them in the system error log. • Support a character special file (/dev/pdisk0, /dev/pdisk1,...,/dev/p.diskN). • Support the IOCTLl subroutine for servicing and...
Page 88 - U p g r ad e In s t ru c t io n s; smitty install
70 IBM Certification Study Guide AIX HACMP Configuration Verification This option enables you to display the relationships between physical (pdisk) and logical (hdisk) disks. Format Disk This option enables you to format SSA disk drives. Certify Disk This option enables you to test whether data on a...
Page 90 - To confirm that the upgrade was a success, type; smitty ssaraid; This will show you the following menu:; Select Add an SSA RAID Array to do the definitions.
72 IBM Certification Study Guide AIX HACMP 18.To confirm that the upgrade was a success, type lscfg -vl pdiskX where X is 0,1... for all SSA disks. Check the ROS Level line to see that each disk has the appropriate microcode level (for the correct microcode level see the above mentioned web-site). 3...
Page 91 - S C S I A d ap t er s
Cluster Hardware and Software Preparation 73 3. 3.2 . 1 C ab l i n gThe following sections describe important information about cabling. S C S I A d ap t er s A overview of SCSI adapters that can be used on a shared SCSI bus is given in 2.3.2.3, “Supported SCSI Adapters” on page 26. For the necessar...
Page 94 - Maximum total cable length: 25m
76 IBM Certification Study Guide AIX HACMP T T T T 6 bit) 6 (16-bit) #2416 (16- #2424 6-bit) 6 (16-bit ) #2426 #2416 (16-b #2416 (16-bit) #2426 Maximum total cable length: 25m
Page 95 - Controller 1
Cluster Hardware and Software Preparation 77 Figure 11. 7135-110 RAIDiant Arrays Connected on Two Shared 16-Bit SCSI Buses 3. 3. 2 . 3 A d ap t e r S C S I I D an d Ter m i n a t i o n c h an g eThe SCSI-2 Differential Controller is used to connect to 8-bit disk devices on a shared bus. The SCSI-2 D...
Page 97 - smit chgscsi
Cluster Hardware and Software Preparation 79 The ID of an SCSI adapter, by default, is 7. Since each device on an SCSI bus must have a unique ID, the ID of at least one of the adapters on a shared SCSI bus has to be changed. The procedure to change the ID of an SCSI-2 Differential Controller is: 1. ...
Page 98 - Reboot the machine to bring the change into effect.; command should be an; ascsi
80 IBM Certification Study Guide AIX HACMP 4. Reboot the machine to bring the change into effect. The same task can be executed from the command line by entering: Also with this method, a reboot is required to bring the change into effect. The procedure to change the ID of an SCSI-2 Differential Fas...
Page 99 - The command line version of this is:; . 4 S h a r e d LV M C o m p o n e n t C o n f i g u r a t i o n; Change/Show Characteristics of a SCSI Adapter
Cluster Hardware and Software Preparation 81 The command line version of this is: As in the case of the SCSI-2 Differential Controller, a system reboot is required to bring the change into effect. The maximum length of the bus, including any internal cabling in disk subsystems, is limited to 19 mete...
Page 100 - . 4 . 1 C r e a t i n g S h a r e d V G s; smit mkvg; fastpath to create a shared; VOLUME GROUP name; Set to yes; lvlstmajor
82 IBM Certification Study Guide AIX HACMP 3 . 4 . 1 C r e a t i n g S h a r e d V G s The following sections contain information about creating non-concurrent VGs and VGs for concurrent access. 3. 4. 1 . 1 C re at i n g N o n - C o n cu r re n t V G sThis section covers how to create a shared volum...
Page 101 - volume group. A concurrent-capable volume group can; with the following options to build the volume group:; Set this field to no.; Create VG concurrent capable?
Cluster Hardware and Software Preparation 83 C r ea t i n g a C o n c u r re n t A c ce ss Vo l u m e G r o u p o n S e ri al D i s kS u b sy st em s To use a concurrent access volume group, defined on a serial disk subsystem such as an IBM 7133 disk subsystem, you must create it as a concurrent-cap...
Page 102 - fastpath to create a shared volume group. Use the default; . 4 . 2 C r e a t i n g S h a r e d LV s a n d F i l e S y s t e m s; Use the; smit crjfs; fast path to create the shared file system on the source; R e n am i n g a j f s lo g a n d L o g i ca l Vo l u m es o n t h e S o u rc e N o d e; Options
84 IBM Certification Study Guide AIX HACMP Use the smit mkvg fastpath to create a shared volume group. Use the default field values unless your site has other requirements, or unless you are specifically instructed otherwise. Table 15. smit mkvg Options (Concurrent, RAID) 3 . 4 . 2 C r e a t i n g S...
Page 103 - lvsharefs; A d d i n g C o p i es t o L o g i ca l Vo l u m e o n t h e So u r ce N o d e; smit mklvcopy
Cluster Hardware and Software Preparation 85 the journaled file system log (jfslog) is a logical volume that requires a unique name in the cluster. To make sure that logical volumes have unique names, rename the logical volume associated with the file system and the corresponding jfslog logical volu...
Page 104 - Tes t in g a F i le Sy st e m; . 4 . 3 M i r r o r i n g S t r a t e g i e s; varyoffvg
86 IBM Certification Study Guide AIX HACMP That is, you enter this command for each disk. In the resulting display, locate the line for the logical volume for which you just added copies. For copies placed on separate disks, the numbers in the logical partitions column and the physical partitions co...
Page 105 - You can use the; smit importvg; fastpath to import the volume group.; smit chvg; fastpath to change the characteristics of a volume group.
Cluster Hardware and Software Preparation 87 The TaskGuide uses a graphical interface to guide you through the steps of adding nodes to an existing volume group. For more information on the TaskGuide, see 3.4.6, “Alternate Method - TaskGuide” on page 90. Importing the volume group onto the destinati...
Page 106 - command to deactivate the shared volume group so that it; command, and whether or not; VGDA; varyonvg; command reads information from; VGSA; varyonvg
88 IBM Certification Study Guide AIX HACMP 3. 4. 4 . 4 Va ry i n g O f f t h e Vo l u m e G ro u p o n t h e D e st i n a t i o n N o d esUse the varyoffvg command to deactivate the shared volume group so that it can be imported onto another destination node or activated as appropriate by the cluste...
Page 107 - command, or by using the; Q u o ru m E n ab l ed; Q u o ru m D i sa b l ed; With quorum disabled,; all; the physical volumes in the volume group must be
Cluster Hardware and Software Preparation 89 command succeeds. If exactly half the copies are available, as with two of four, quorum is not achieved and the varyonvg command fails. 3. 4. 5 . 2 Q u o r u m af t er Va ry O nIf a write to a physical volume fails, the VGSAs on the other physical volumes...
Page 108 - F o rc in g a Va ry o n; flag with the; command is issued; Q u o ru m i n N o n - C o n c u r re n t A c ce ss C o n f i g u ra t i o n s; . 4 . 6 A l t e r n a t e M e t h o d - Ta s k G u i d e
90 IBM Certification Study Guide AIX HACMP F o rc in g a Va ry o n A volume group with quorum disabled and one or more physical volumes unavailable can be “forced” to vary on by using the -f flag with the varyonvg command. Forcing a varyon with missing disk resources can cause unpredictable results,...
Page 109 - • You have a configured HACMP cluster in place.
Cluster Hardware and Software Preparation 91 conflict with the cluster ’s configuration. Online help panels give additional information to aid in each step. 3 . 4 . 6 . 1 Ta s k G u i d e R e q u i r em e n t sBefore starting the TaskGuide, make sure: • You have a configured HACMP cluster in place. ...
Page 111 - C h a p t e r 4 . H A C M P I n s t a ll a t i o n a n d C l u s t e r D e f i n i t i o n; SG24-5116, for details on installation or migration to that level.; Guide; . 1 I n s t a l l i n g H A C M P; must be confirmed. For parts; . 1 . 1 F i r s t Ti m e I n s t a l l s
© Copyright IBM Corp. 1999 93 C h a p t e r 4 . H A C M P I n s t a ll a t i o n a n d C l u s t e r D e f i n i t i o n This chapter describes issues concerning the actual installation of HACMP Version 4.3 and the definition of a cluster and its resources. It concentrates on the HACMP part of the i...
Page 113 - This fileset contains man pages and data for the HAView component:; Note
HACMP Installation and Cluster Definition 95 • cluster.vsm The Visual Systems Management Fileset contains Icons and bitmaps for the graphical Management of HACMP Resources, as well as the xhacmpm command: cluster.vsm HACMP X11 Dependent • cluster.haview This fileset contains the files for including ...
Page 114 - Application Heart Beat Daemon; The installation of CRM requires the following software:; AIX Run-time Executable; In s t all S er ve r N o d es; . 1 . 2 U p g r a d i n g F r o m a P r e v i o u s Ve r s i o n; of the; Administration Guide,; except for Version 4.2.2 already running
96 IBM Certification Study Guide AIX HACMP This fileset contains the Application Heart Beat Daemon, Oracle Parallel Server is an application that makes use of it: cluster.hc.rte Application Heart Beat Daemon The installation of CRM requires the following software: bos.rte.lvm.usr.4.3.2.0 AIX Run-tim...
Page 115 - U p g r ad e A I X o n O n e N o d e; Administration Guide; smit clstop; oslevel
HACMP Installation and Cluster Definition 97 HACMP software to HACMP for AIX, Version 4.3. The comments on upgrading the Operating System are not included. If you are already running AIX 4.3, see the special note at the end of this section. 4. 1. 2 . 1 U p g ra d i n g fr o m Ver si o n 4 . 1. 0 t h...
Page 116 - I n s t a l l H A C M P 4 .3 f o r A I X o n N o d e A; smit clstart; C h e ck U p g r ad e d C o n f i g u r at i o n; smit chgtty
98 IBM Certification Study Guide AIX HACMP I n s t a l l H A C M P 4 .3 f o r A I X o n N o d e A 5. After upgrading AIX and verifying that the disks are correctly configured, install the HACMP 4.3 for AIX software on Node A. For a short description of the filesets, please refer to 4.1.1, “First Tim...
Page 117 - clverify; C l i en t - o n l y M ig r at i o n
HACMP Installation and Cluster Definition 99 file on Node A using the following command: /usr/sbin/cluster/utilities/cllsif -x >> /.rhosts This command will append information to the /.rhosts file instead of overwriting it. Then, you can ftp this file to the other nodes as necessary. 12.Verify...
Page 118 - . 2 D e f i n i n g C l u s t e r To p o l o g y; No te
100 IBM Certification Study Guide AIX HACMP 2. If you wish to save your cluster configuration, see the chapter Saving and Restoring Cluster Configurations in the HACMP for AIX, Version 4.3: Administration Guide, SC23-4279. 3. Commit your current HACMP for AIX software on all nodes. 4. Shut down one ...
Page 119 - • The network modules; xhacmpm; utility, see the administrative facilities chapter of the; . 2 . 1 D e f i n i n g t h e C l u s t e r; smit hacmp; . Each of the cluster nodes; for more information.
HACMP Installation and Cluster Definition 101 • The network modules You define the cluster topology by entering information about each component into HACMP-specific ODM classes. You enter the HACMP ODM data by using the HACMP SMIT interface or the VSM utility xhacmpm . The xhacmpm utility is an X Wi...
Page 120 - A d d i n g o r C h a n g i n g a N o d e N am e a f t er t h e I n i ti al C o n f i g u ra t i o n; . 2 . 3 D e f i n i n g A d a p t e r s; Adapter IP Label; utility flags adapter; Network Type
102 IBM Certification Study Guide AIX HACMP A d d i n g o r C h a n g i n g a N o d e N am e a f t er t h e I n i ti al C o n f i g u ra t i o n If you want to add or change a node name after the initial configuration, use the Change/Show Cluster Node Name screen. See the chapter on changing the clu...
Page 121 - Network Name; Indicate whether the adapter's function is; service; . Press Tab to toggle the values. A node; HACMP Installation Guide; Version 4.3 for more
HACMP Installation and Cluster Definition 103 Network Name Enter an ASCII text string that identifies the network. The network name can include alphabetic and numeric characters and underscores. Use no more than 31 characters. The network name is arbitrary, but must be used consistently for adapters...
Page 122 - Adapter Identifier
104 IBM Certification Study Guide AIX HACMP Adapter Identifier Enter the IP address in dotted decimal format or a device file name. IP address information is required for non-serial network adapters only if the node’s address cannot be obtained from the domain name server or the local /etc/hosts fil...
Page 123 - A d d i n g o r C h a n g i n g A d ap te rs a f t er t h e I n i t i al C o n f i g u r at i o n; . 2 . 4 C o n f i g u r i n g N e t w o r k M o d u l e s; are changed to “
HACMP Installation and Cluster Definition 105 A d d i n g o r C h a n g i n g A d ap te rs a f t er t h e I n i t i al C o n f i g u r at i o n If you want to change the information about an adapter after the initial configuration, use the Change/Show an Adapter screen. See the chapter on changing t...
Page 124 - which is the acceptable number of missed heartbeats; Change / Show Topology; like the NIM tuning options.; . 2 . 5 S y n c h r o n i z i n g t h e C l u s t e r D e f i n i t i o n A c r o s s N o d e s
106 IBM Certification Study Guide AIX HACMP • SLIP • SP Switch • ATM It is highly unlikely that you will add or remove a network module. For information about changing a characteristic of a Network Module, such as the failure detection rate, see the chapter on changing the cluster topology in the HA...
Page 125 - file) on remote nodes during HACMP configuration. Instead, it uses a; chapter on verifying a cluster configuration in the; Emulate or Actual
HACMP Installation and Cluster Definition 107 configuration. If the cluster manager is active on some other cluster nodes but not on the local node, the synchronization operation is aborted. Before attempting to synchronize a cluster configuration, ensure that all nodes are powered on, that the HACM...
Page 126 - . 3 D e f i n i n g R e s o u r c e s
108 IBM Certification Study Guide AIX HACMP 4 . 3 D e f i n i n g R e s o u r c e s The HACMP for AIX software provides a highly available environment by identifying a set of cluster-wide resources essential to uninterrupted processing, and then by defining relationships among nodes that ensure thes...
Page 127 - NTX adapters are not supported by HACMP for AIX 4.3; You cannot; information on the Add a Resource Group screen.
HACMP Installation and Cluster Definition 109 4. 3. 1 . 1 C o n f i g u r i n g R es o u rc es f o r R e so u r ce G r o u p sOnce you have defined resource groups, you further configure them by assigning cluster resources to one resource group or another. You can configure resource groups even if a...
Page 128 - fsck; not set this field to parallel if you have shared, nested
110 IBM Certification Study Guide AIX HACMP These settings also have to be synchronized throughout the cluster. Therefore Synchronize Cluster Resources has to be chosen from the corresponding SMIT Menu. If the Cluster Manager is running on the local node, synchronizing cluster resources triggers a d...
Page 129 - . 4 S y n ch r o n i z i n g C l u s t er R e so u r ce s; . 4 I n i t i a l Te s t i n g; Running; smit clverify; file updated with
HACMP Installation and Cluster Definition 111 as the path locations for start and stop scripts for the application. These scripts have to be in the same location on every service node. Just as for pre- and post-events, these scripts can be adapted to specific nodes. They don’t need to be equal in co...
Page 130 - . 4 . 2 I n i t i a l S t a r t u p; ps; . 4 . 3 Ta k e o v e r a n d R e i n t e g r a t i o n; smitty clstop
112 IBM Certification Study Guide AIX HACMP 4 . 4 . 2 I n i t i a l S t a r t u p At this point in time, the cluster is not yet started. So the cluster manager has to be started first. To check whether the cluster manager is up, you can either look for the process with the ps command: ps -ef | grep ...
Page 131 - smitty clstart; and check again for the logfile, as well as the; . 5 C l u s t e r S n a p s h o t; administrative facilities chapter in the; and Facilities; , SC23-4276, and the online help information available with the
HACMP Installation and Cluster Definition 113 For cascading resource groups the failed node is going to reaquire its resources, once it is up and running again. So, you have to restart HACMP on it through smitty clstart and check again for the logfile, as well as the clusters status. Further and mor...
Page 132 - . 5 . 1 A p p l y i n g a C l u s t e r S n a p s h o t; as well as in the
114 IBM Certification Study Guide AIX HACMP Essentially, a snapshot saves all the ODM classes HACMP has generated during its configuration. It does not save user customized scripts, such as start or stop scripts for an application server. However, the location and names of these scripts are in an HA...
Page 135 - C h a p t e r 5 . C l u s t e r C u s t o m i z a t i o n; . 1 E v e n t C u s t o m i z a t i o n
© Copyright IBM Corp. 1999 117 C h a p t e r 5 . C l u s t e r C u s t o m i z a t i o n Within an HACMP for AIX cluster, there are several things that are customizable. The following paragraphs explain the customizing features for events, error notification, network modules and topology services. 5...
Page 137 - S e q ue n c e o f n o d e _ d o w n E v e n t s
Cluster Customization 119 event occurs only after a node_up_remote event has successfully completed. S e q ue n c e o f n o d e _ d o w n E v e n t s node_down This event occurs when a node intentionally leaves the cluster or fails. Depending on whether the exiting node is local or remote, this even...
Page 138 - Starts application servers.
120 IBM Certification Study Guide AIX HACMP node_down_local_completeInstructs the Cluster Manager to exit when the local node has left the cluster. This event occurs only after a node_down_local event has successfully completed. node_down_remote_completeStarts takeover application servers. This even...
Page 139 - dynamic reconfiguration has completed.
Cluster Customization 121 no actions since appropriate actions depend on the local network configuration. 5. 1.1 . 3 N et w o r k A d ap t e r Ev en t sswap_adapter This event occurs when the service adapter on a node fails. The swap_adapter event exchanges or swaps the IP addresses of the service a...
Page 140 - . 1 . 3 E v e n t N o t i f i c a t i o n; Notify; For example, a site may want to use a; . 1 . 4 E v e n t R e c o v e r y a n d R e t r y
122 IBM Certification Study Guide AIX HACMP reconfig_resource_completeThis event indicates that a cluster resource dynamic reconfiguration has completed. 5 . 1 . 2 P r e - a n d P o s t - E v e n t P r o c e s s i n g To tailor event processing to your environment, specify commands or user-defined s...
Page 141 - Retry; feature enables; . 1 . 5 N o t e s o n C u s t o m i z i n g E v e n t P r o c e s s i n g; You must declare a shell (for example; . 1 . 6 E v e n t E m u l a t o r; environment variable, or, use the default; . 2 E r r o r N o t i f i c a t i o n; notification objects.
Cluster Customization 123 For example, a file system cannot be unmounted, because of a process running on it. Then, you might want to kill that process first, before unmounting the file system, in order to get the event script done. Now, since the event script didn’t succeed in its first run, the Re...
Page 142 - notify method; can; . 3 N e t w o r k M o d u l e s / To p o l o g y S e r v i c e s a n d G r o u p S e r v i c e s; There are three values to choose from:
124 IBM Certification Study Guide AIX HACMP Each time an error is logged in the system error log, the error notification daemon determines if the error log entry matches the selection criteria. If it does, an executable is run. This executable, called a notify method , can range from a simple comman...
Page 143 - Installation and Administration Guide; . 4 N F S c o n s i d e r a t i o n s
Cluster Customization 125 The failure rate of networks varies, depending on their characteristics. For example, for an Ethernet, the normal failure detection rate is two keepalives per second; fast is about four per second; slow is about one per second. For an HPS network, because no network traffic...
Page 144 - Shared; . 4 . 2 E x p o r t i n g N F S F i l e S y s t e m s; flag from the; exportfs; . 4 . 4 C a s c a d i n g Ta k e o v e r w i t h C r o s s M o u n t e d N F S F i l e S y s t e m s
126 IBM Certification Study Guide AIX HACMP To prevent problems with NFS file systems in an HACMP cluster, make sure that each shared volume group has the same major number on all nodes. The lvlstmajor command lists the free major numbers on a node. Use this command on each node to find a major numb...
Page 145 - When Node A fails, Node B uses the; utility to close open files in; /afs locally mounted; C a ve at s ab o u t N o d e N a m es a n d N F S; To avoid this problem do one of the following:
Cluster Customization 127 Figure 14. NFS Cross Mounts When Node A fails, Node B uses the cl_nfskill utility to close open files in Node A:/afs, unmounts it, mounts it locally, and re-exports it to waiting clients. After takeover, Node B has: /bfs locally mounted /bfs nfs-exported /afs locally mounte...
Page 146 - • Ensure that node name and the service adapter label are the same on; . 4 . 5 C r o s s M o u n t e d N F S F i l e S y s t e m s a n d t h e N e t w o r k L o c k M a n a g e r; Add the code below between the following two lines (three places):
128 IBM Certification Study Guide AIX HACMP • Ensure that node name and the service adapter label are the same on each node in the clusteror • Alias the node name to the service adapter label in the /etc/hosts file. 5 . 4 . 5 C r o s s M o u n t e d N F S F i l e S y s t e m s a n d t h e N e t w o ...
Page 149 - C h a p t e r 6 . C l u s t e r Tes t i n g; . 1 N o d e Ve r i f i c a t i o n; more
© Copyright IBM Corp. 1999 131 C h a p t e r 6 . C l u s t e r Tes t i n g Before you start to test the HACMP configuration, you need to guarantee that your cluster nodes are in a stable state. Check the state of the: • Devices • System parameters • Processes • Network adapters • LVM • Cluster • Oth...
Page 150 - . 1 . 2 S y s t e m P a r a m e t e r s
132 IBM Certification Study Guide AIX HACMP 6 . 1 . 2 S y s t e m P a r a m e t e r s • Type date on all nodes to check that all the nodes in the cluster are running with their clocks on the same time. • Ensure that the number of user licenses has been correctly set (lslicense ). • Check high water ...
Page 152 - . 2 S i m u l a t e E r r o r s; echo; . 2 . 1 A d a p t e r F a i l u r e; errclear 0
134 IBM Certification Study Guide AIX HACMP • Verify the cluster configuration by running /usr/sbin/cluster/diag/clconfig -v ’-tr’ . • To show cluster configuration, run: /usr/sbin/cluster/utilities/cllscf . • To show the clstrmgr version, type: snmpinfo -m dump -o /usr/sbin/cluster/hacmp.defs clstr...
Page 153 - ifconfig en1 down
Cluster Testing 135 • Use ifconfig to swap the service address back to the original service interface back ( ifconfig en1 down ). This will cause the service IP address to failover back to the service adapter on NodeF. 6. 2. 1 . 2 E t h er n e t o r To ke n R i n g A d a p t er o r C a b l e F a i l...
Page 155 - . 2 . 2 N o d e F a i l u r e / R e i n t e g r a t i o n
Cluster Testing 137 • Verify that all sharedvg file systems and paging spaces are accessible ( df -k and lsps -a ). 6 . 2 . 2 N o d e F a i l u r e / R e i n t e g r a t i o n The following sections deal with issues of node failure and reintegration. 6 . 2 . 2 . 1 A I X C r a s hPerform the followin...
Page 156 - ping; . 2 . 3 N e t wo r k F a i l u r e
138 IBM Certification Study Guide AIX HACMP • Verify that failover has occurred ( netstat -i and ping for networks, lsvg -o and vi of a test file for volume groups, and ps -U <appuid > for application processes). • Power cycle NodeF. If HACMP is not configured to start from /etc/inittab (on re...
Page 159 - . 2 . 5 A p p l i c a t i o n F a i l u r e
Cluster Testing 141 • Monitor cluster logfiles on NodeT if HACMP has been customized to monitor 7133 disk failures. • Since the 7133 disk is hot pluggable, remove a disk from drawer 1 associated with NodeF's shared volume group. • The failure of the 7133 disk will be detected in the error log ( errp...
Page 161 - C h a p t e r 7 . C l u s t e r Tr o u b l e s h o o t i n g; In general, troubleshooting an HACMP cluster involves:; . 1 C l u s t e r L o g F i l e s; Log File Name
© Copyright IBM Corp. 1999 143 C h a p t e r 7 . C l u s t e r Tr o u b l e s h o o t i n g Typically, a functioning HACMP cluster requires minimal intervention. If a problem occurs, however, diagnostic and recovery skills are essential. Thus, troubleshooting requires that you identify the problem q...
Page 162 - mm indicates the month and dd indicates
144 IBM Certification Study Guide AIX HACMP For a more detailed description of the cluster log files consult Chapter 2 of the HACMP for AIX, Version 4.3: Troubleshooting Guide , SC23-4280. 7 . 2 c o n f i g _ t o o _ l o n g If the cluster manager recognizes a state change in the cluster, it acts up...
Page 163 - clruncmd; . 3 D e a d m a n S w i t c h; clstrmgr
Cluster Troubleshooting 145 hang. After a certain amount of time, by default 360 seconds, the cluster manager will issue a config_too_long message into the /tmp/hacmp.out file. The message issued looks like this: The cluster has been in reconfiguration too long;Something may be wrong. In most cases,...
Page 164 - AIX Performance Monitoring & Tuning; . 3 . 2 E x t e n d i n g t h e s y n c d F r e q u e n c y; syncd; . 3 . 3 I n c r e a s e A m o u n t o f M e m o r y f o r C o m m u n i c a t i o n s S u b s y s t e m; xxxxx
146 IBM Certification Study Guide AIX HACMP 7.3.1 Tuning the System Using I/O Pacing Use I/O pacing to tune the system so that system resources are distributed more equitably during large disk writes. Enabling I/O pacing is required for an HACMP cluster to behave correctly during large disk writes, ...
Page 165 - . 3 . 4 C h a n g i n g t h e F a i l u r e D e t e c t i o n R a t e; SMIT Change/Show a Cluster Network Module; . 4 N o d e I s o l a t i o n a n d P a r t i t i o n e d C l u s t e r s; Node isolation; Slow
Cluster Troubleshooting 147 7 . 3 . 4 C h a n g i n g t h e F a i l u r e D e t e c t i o n R a t e Use the SMIT Change/Show a Cluster Network Module screen to change the failure detection rate for your network module only if enabling I/O pacing or extending the syncd frequency did not resolve deadm...
Page 166 - . 5 T h e D G S P M e s s a g e; A DGSP message (short for Diagnostic Group Shutdown Partition
148 IBM Certification Study Guide AIX HACMP and control messages so that the Cluster Manager has accurate information about the status of its partner. When a cluster becomes partitioned, and the network problem is cleared after the point when takeover processing has begun so that keepalive packets s...
Page 167 - . 6 U s e r I D P r o b l e m s; . 7 Tr o u b l e s h o o t i n g S t r a t e g y; • Save the log files associated with the problem before they become
Cluster Troubleshooting 149 7 . 6 U s e r I D P r o b l e m s Within an HACMP cluster, you always have more than one node potentially offering the same service to a specific user or a specific user id. As the node providing the service can change, the system administrator has to ensure that the same...
Page 168 - • Do not make more than one change at a time. If you do, and one of the
150 IBM Certification Study Guide AIX HACMP • Go from the simple to the complex. Make the simple tests first. Do not try anything complex and complicated until you have ruled out the simple and obvious. • Do not make more than one change at a time. If you do, and one of the changes corrects the prob...
Page 169 - C h a p t e r 8 . C l u s t e r M a n a g e m e n t a n d A d mi n i s t r a t io n; . 1 M o n i t o r i n g t h e C l u s t e r; HAView; SMIT Show Cluster Services
© Copyright IBM Corp. 1999 151 C h a p t e r 8 . C l u s t e r M a n a g e m e n t a n d A d mi n i s t r a t io n This chapter covers all aspects of monitoring and managing an existing HACMP cluster. This includes a description of the different monitoring methods and tools available, how to start a...
Page 170 - . 1 . 1 T h e c l s t a t C o m m a n d; command for; . 1 . 2 M o n i t o r i n g C l u s t e r s u s i n g H AVi e w
152 IBM Certification Study Guide AIX HACMP Consult the HACMP for AIX, Version 4.3: Troubleshooting Guide , SC23-4280, for help if you detect a problem with an HACMP cluster. 8 . 1 . 1 T h e c l s t a t C o m m a n d HACMP for AIX provides the /usr/sbin/cluster/clstat command for monitoring a cluste...
Page 171 - . 1 . 3 C l u s t e r L o g F i l e s; for more information about these files.; high; which is the default.; m m d d; file contains timestamped,; mmdd; indicates the month and; dd
Cluster Management and Administration 153 More details on how to configure HAView and on how to monitor your cluster with HAView can be found in Chapter 3, “Monitoring an HACMP cluster” in HACMP for AIX, Version 4.3: Administration Guide , SC23-4279. 8 . 1 . 3 C l u s t e r L o g F i l e s HACMP for...
Page 172 - activity. This file is typically used by IBM support personnel.; . 2 S t a r t i n g a n d S t o p p i n g H A C M P o n a N o d e o r a C l i e n t
154 IBM Certification Study Guide AIX HACMP 8 . 1 . 3 . 5 / t m p / c m . l o g Contains timestamped, formatted messages generated by HACMP for AIX clstrmgr activity. This file is typically used by IBM support personnel. 8 . 1 . 3 . 6 / t m p / c s p o c . l o gContains timestamped, formatted messag...
Page 174 - . 2 . 2 S t a r t i n g C l u s t e r S e r v i c e s o n a N o d e; Start Cluster Services SMIT
156 IBM Certification Study Guide AIX HACMP 8. 2. 1 . 4 C l u st e r I n f o r m at i o n P ro g r am d ae m o n ( cl i n f o )This daemon provides status information about the cluster to cluster nodes and clients and invokes the /usr/sbin/cluster/etc/clinfo.rc script in response to a cluster event....
Page 175 - restart or both in the; . 2 . 3 S t o p p i n g C l u s t e r S e r v i c e s o n a N o d e; HACMP for AIX Stop Cluster; clstop
Cluster Management and Administration 157 are started in sequential order - not in parallel. The output of the command run on the remote node is returned to the originating node. Because the command is executed remotely, there can be a delay before the command output is returned. 8 . 2 . 2 . 1 A u t...
Page 176 - nodes for the changes to become active. For example, if; Graceful; script to halt the system. This
158 IBM Certification Study Guide AIX HACMP node. Because the command is executed remotely, there can be a delay before the command output is returned. 8. 2.3 . 1 W h e n t o S t o p C l u s te r se rv ic esYou typically stop cluster services in the following situations: • Before making any hardware...
Page 177 - . 2 . 4 S t a r t i n g a n d S t o p p i n g C l u s t e r S e r v i c e s o n C l i e n t s; command to; I mp ort ant N ote
Cluster Management and Administration 159 prevents unpredictable behavior from corrupting the data on the shared disks. See the clexit.rc man page for additional information. 8 . 2 . 4 S t a r t i n g a n d S t o p p i n g C l u s t e r S e r v i c e s o n C l i e n t s Use the /usr/sbin/cluster/etc...
Page 178 - . 3 R e p l a c i n g F a i l e d C o m p o n e n t s; • Slot capacity of the new node must be the same or better.
160 IBM Certification Study Guide AIX HACMP 8 . 3 R e p l a c i n g F a i l e d C o m p o n e n t s From time to time, it will be necessary to perform hardware maintenance or upgrades on cluster components. Some replacements or upgrades can be performed while the cluster is operative, while others r...
Page 179 - rmlvcopy
Cluster Management and Administration 161 • The new adapter must be of the same type or a compatible type as the replaced adapter. • When replacing or adding an SCSI adapter, remove the resistors for shared buses. Furthermore, set the SCSI ID of the adapter to a value different than 7. 8 . 3 . 3 D i...
Page 181 - . 4 C h a n g i n g S h a r e d LV M C o m p o n e n t s
Cluster Management and Administration 163 8 . 4 C h a n g i n g S h a r e d LV M C o m p o n e n t s Changes to VG constructs are probably the most frequent kind of changes to be performed in a cluster. As a system administrator of an HACMP for AIX cluster, you may be called upon to perform any of t...
Page 183 - • If the first disk in a sharedvg has been replaced, the; importvg; command
Cluster Management and Administration 165 Lazy Update has some limitations, which you need to consider when you rely on Lazy Update in general: • If the first disk in a sharedvg has been replaced, the importvg command will fail as Lazy Update expects to be able to match the hdisk number for the firs...
Page 185 - . 5 C h a n g i n g C l u s t e r R e s o u r c e s
Cluster Management and Administration 167 To use the SMIT shortcuts to C-SPOC, type smit cl_lvm or smit cl_conlvm for concurrent volume groups. Concurrent volume groups must be varied on in concurrent mode to perform tasks. 8 . 4 . 4 Ta s k G u i d e The TaskGuide is a graphical interface that simpl...
Page 186 - . 5 . 1 A d d / C h a n g e / R e m o v e C l u s t e r R e s o u r c e s; Use the following SMIT shortcuts:; . 5 . 2 S y n c h r o n i z e C l u s t e r R e s o u r c e s; Concepts and Facilities; ODMs stored in the DCDs on all cluster nodes.
168 IBM Certification Study Guide AIX HACMP To change the nodes associated with a given resource group, or to change the priorities assigned to the nodes in a resource group chain, you must redefine the resource group. You must also redefine the resource group if you add or change a resource assigne...
Page 187 - . 5 . 3 D A R E R e s o u r c e M i g r a t i o n U t i l i t y; command. The command lets you move the; cldare; flag to perform the migration.
Cluster Management and Administration 169 • If the Cluster Manager is active on the local node, synchronization triggers a cluster-wide, dynamic reconfiguration event. In dynamic reconfiguration, the configuration data stored in the DCD is updated on each cluster node, and, in addition, the new ODM ...
Page 188 - S ti ck y R es o u rc e Mi g ra t io n; You can attach the optional keyword; sticky; to any migration you perform,; N o n - S t i c ky R e so u r ce M ig r at i o n
170 IBM Certification Study Guide AIX HACMP 8. 5. 3 . 1 R es o u rc e M i g r at i o n Ty p esBefore performing a resource migration, decide if you will declare the migration sticky or non-sticky . S ti ck y R es o u rc e Mi g ra t io n A sticky migration permanently attaches a resource group to a s...
Page 189 - N o d e N a m e
Cluster Management and Administration 171 INACTIVE_TAKEOVER flag set to false and has not yet started because its primary node is down. In general, however, only rotating resource groups should be migrated in a non-sticky manner. Such migrations are one-time events and occur similar to normal rotati...
Page 190 - S to p L o ca t i o n
172 IBM Certification Study Guide AIX HACMP If you do not include a location specifier in the location field, the DARE Resource Migration utility performs a default migration, again making the resources available for reacquisition. S to p L o ca t i o n The second special location keyword, stop , ca...
Page 191 - S to p p i n g R es o u rc e G ro u p s
Cluster Management and Administration 173 Note that you cannot add nodes to the resource group list with the DARE Resource Migration utility. This task is performed through SMIT. S to p p i n g R es o u rc e G ro u p s If the location field of a migration contains the keyword stop instead of an actu...
Page 192 - clfindres; . 6 A p p l y i n g S o f t wa r e Ma i n t e n a n c e t o a n H A C M P C l u s t e r
174 IBM Certification Study Guide AIX HACMP Be aware that persistent sticky location markers are saved and restored in cluster snapshots. You can use the clfindres command to find out if sticky markers are present in a resource group. If you want to remove sticky location markers while the cluster i...
Page 193 - Restart the HACMP for AIX software on the node using the; Repeat Steps 1 through 5 on the remaining cluster nodes.; • Cluster nodes should be kept at the same AIX maintenance levels
Cluster Management and Administration 175 5. Restart the HACMP for AIX software on the node using the smit clstart fastpath and verify that the node successfully joined the cluster. 6. Repeat Steps 1 through 5 on the remaining cluster nodes. Figure 15 below shows the procedure: Figure 15. Applying a...
Page 194 - . 7 B a c k u p S t r a t e g i e s; backup; AIX Commands
176 IBM Certification Study Guide AIX HACMP • Cluster nodes should be running the same HACMP maintenance levels. There might be incompatibilities between various maintenance levels of HACMP, so you must ensure that consistent levels are maintained across all cluster nodes. The cluster must be taken ...
Page 195 - splitlvcopy
Cluster Management and Administration 177 8. 7. 1 . 1 H o w t o d o a s p l i t -m i r ro r b a ck u pThis same procedure can be used with just one mirrored copy of a logical volume. If you remove a mirrored copy of a logical volume (and file system), and then create a new logical volume (and file s...
Page 196 - . 7 . 2 U s i n g E v e n t s t o S c h e d u l e a B a c k u p; HACMP/6000 Customization Examples; . 8 U s e r M a n a g e m e n t
178 IBM Certification Study Guide AIX HACMP 9. After the backup is complete and verified, unmount and delete the new file system and the logical volume you used for it. 10.Use the mklvcopy command to add back the logical volume copy you previously split off to the fslv logical volume. 11. Resynchron...
Page 197 - . 8 . 1 L i s t i n g U s e r s O n A l l C l u s t e r N o d e s
Cluster Management and Administration 179 they don’t match, the user won’t get anything done after a failover happened. So, the administrator has to keep definitions equal throughout the cluster. Fortunately, the C-SPOC utility, as of HACMP Version 4.3 and later, does this for you. When you create a...
Page 198 - . 8 . 3 C h a n g i n g A t t r i b u t e s o f U s e r s i n a C l u s t e r
180 IBM Certification Study Guide AIX HACMP To add a user on one or more nodes in a cluster, you can either use the AIX mkuser command in a rsh to one clusternode after the other, or use the C-SPOC cl_mkuser command or the Add a User to the Cluster SMIT screen. The cl_mkuser command calls the AIX mk...
Page 199 - rmuser; . 8 . 5 M a n a g i n g G r o u p A c c o u n t s
Cluster Management and Administration 181 To remove a user account from one or more cluster nodes, you can either use the AIX rmuser command on one cluster node after the other, or use the C-SPOC cl_rmuser command or the C-SPOC Remove a User from the Cluster SMIT screen. The cl_rmuser command execut...
Page 201 - C h a p t e r 9 . S p ec i al R S / 60 00 SP To p i c s; . 1 H i g h Av a i l a b i l i t y C o n t r o l Wo r k s t a t i o n ( H A C W S ); IBM Parallel System Support Programs for; for a High Availability Workstation”, in the; IBM RS/6000 SP Planning Volume; • Continues running your SP system after a cws failure; . 1 . 1 H a r d w a r e R e q u i r e m e n t s
© Copyright IBM Corp. 1999 183 C h a p t e r 9 . S p ec i al R S / 60 00 SP To p i c s This chapter will introduce you to some special topics that only apply if you are running HACMP on the SP system. 9 . 1 H i g h Av a i l a b i l i t y C o n t r o l Wo r k s t a t i o n ( H A C W S ) If you are th...
Page 202 - . 1 . 2 S o f t w a r e R e q u i r e m e n t s; SP Ethernet
184 IBM Certification Study Guide AIX HACMP need to have the frame supervisors support dual tty lines in order to get both control workstations connected at the same time. Contact your IBM representative for the neccessary hardware (see Figure 16 on page 184). Both the tty network and the RS/6000 SP...
Page 203 - . 1 . 4 I n s t a l l H i g h Av a i l a b i l i t y S o f t w a r e; . 1 . 5 H A C W S C o n f i g u r a t i o n; spmgr splogd hardmon sysctld supfilesrv
Special RS/6000 SP Topics 185 The backup cws has to be installed with the same level of AIX and PSSP. Depending on the kerberos configuration of the primary cws, the backup cws has to be configured either as a secondary authentication server for the authentication realm of your RS/6000 SP when the p...
Page 204 - . 1 . 6 S e t u p a n d Te s t H A C W S; ifconfig; NOT; smit chinet
186 IBM Certification Study Guide AIX HACMP ordinary HACMP cluster, as it is described in Chapter 7 of the HACMP for AIX, Version 4.3: Installation Guide , SC23-4278. Now the cluster environment has to be configured. Define a cluster ID and name for your HACWS cluster and define the two nodes to HAC...
Page 205 - command, and verify the configuration with the; , are working as expected. If everything is fine, start up cluster; . 2 K e r b e r o s S e c u r i t y; To understand security, we have to clarify some definitions first.; Identification; is the process by which an entity tells another who it is.; Authorization; These issues are solved with kerberos.
Special RS/6000 SP Topics 187 After that, identify the HACWS event scripts to HACMP by executing the /usr/sbin/hacws/spcw_addevents command, and verify the configuration with the /usr/sbin/hacws/hacws_verify command. You should also check the cabling from the backup cws with the /usr/sbin/hacws/spcw...
Page 206 - Ker beros
188 IBM Certification Study Guide AIX HACMP The following is simply a shortened description on how kerberos works. For more details, the redbook Inside the RS/6000 SP , SG24-5145, covers the subject in much more detail. When dealing with authentication and Kerberos, three entities are involved: the ...
Page 207 - . 2 . 1 C o n f i g u r i n g K e r b e r o s S e c u r i t y w i t h H A C M P Ve r s i o n 4 . 3; kadmin
Special RS/6000 SP Topics 189 allow the clients to get service tickets to be used with other servers without the need to give them the password every time they request services. So, given a user has a ticket-granting ticket, if a user requests a kerberized service, he has to get a service ticket for...
Page 209 - buddy buffer; to temporarily store data for I/O; IBM Parallel System Support Programs for AIX
Special RS/6000 SP Topics 191 With reference to Figure 17 above, imagine two nodes, Node X and Node Y, running the same application. The nodes are connected by the switch and have locally-attached disks. On Node X’s disk resides a volume group containing the raw logical volume lv_X. Similarly, Node ...
Page 210 - Undefined
192 IBM Certification Study Guide AIX HACMP The VSDs in this scenario are mapped to the raw logical volumes lv_X and lv_Y. Node X is a client of Node Y’s VSD, and vice versa. Node X is also a direct client of its own VSD (lv_X), and Node Y is a direct client of VSD lv_Y. VSD configuration is flexibl...
Page 211 - . 3 . 2 R e c o v e r a b l e Vi r t u a l S h a r e d D i s k; N o d e Y; V S D; F a s t IP N e tw o r k
Special RS/6000 SP Topics 193 impact of servicing a local I/O request through VSD relative to the normal VMM/LVM pathway is very small. IBM supports any IP network for VSD, but we recommend the switch for performance. VSD provides distributed data access, but not a locking mechanism to preserve data...
Page 213 - private
Special RS/6000 SP Topics 195 9 . 4 S P S w i t c h a s a n H A C M P N e t w o r k One of the fascinating things with an RS/6000 SP is the switch network. It has developed over time; so, currently there are two types of switches at customer sites. The “older” HPS or HiPS switch (High Performance Sw...
Page 214 - . 4 . 2 E p r i m a r y M a n a g e m e n t; event, which in turn would shut down
196 IBM Certification Study Guide AIX HACMP 9 . 4 . 2 E p r i m a r y M a n a g e m e n t The SP switch has an internal primary backup concept, where the primary node, known as the Eprimary, is backed up automatically by a backup node. So, in case any serious failure happens on the primary, it will ...
Page 217 - C h a p t e r 1 0. H A C M P C l a s s ic v s . H A C M P / E S v s . H A N F S; 0 . 1 H A C M P f o r A I X C l a s s i c; Network Interface Modules (NIMs) monitor the nodes and network; 0 . 2 H A C M P f o r A I X / E n h a n c e d S c a l a b i l i t y
© Copyright IBM Corp. 1999 199 C h a p t e r 1 0. H A C M P C l a s s ic v s . H A C M P / E S v s . H A N F S So, why would you prefer to install one version of HACMP instead of another? This chapter summarizes the differences between them, to give you an idea in which situation one or the other be...
Page 218 - 0 . 2 . 1 I B M R I S C S y s t e m C l u s t e r Te c h n o l o g y (R S C T ); RSCT Services include the following components:; Event Manager
200 IBM Certification Study Guide AIX HACMP handling membership and event management by using heartbeats. On the SP, the original High Availability infrastructure was built on this technology, and HACMP/ES Version 4.3. is now another instance relying on it. As of AIX 4.3.2 and PSSP 3.1, the High Ava...
Page 219 - See Part 4 of; HACMP for AIX, Version 4.3: Enhanced Scalability Installation; 0 . 2 . 2 E n h a n c e d C l u s t e r S e c u r i t y; Standard; Synchronization is done through the; Enhanced; 0 . 3 H i g h Av a i l a b i l i t y f o r N e t w o r k F i l e S y s t e m f o r A I X; The HANFS for AIX software supports only two nodes in a cluster.
HACMP Classic vs. HACMP/ES vs. HANFS 201 See Part 4 of HACMP for AIX, Version 4.3: Enhanced Scalability Installation and Administration Guide , SC23-4284, for more information on these services. 1 0 . 2 . 2 E n h a n c e d C l u s t e r S e c u r i t y With HACMP Version 4.3 comes an option to switc...
Page 220 - 0 . 4 S i m i l a r i t i e s a n d D i f f e r e n c e s
202 IBM Certification Study Guide AIX HACMP 1 0 . 4 S i m i l a r i t i e s a n d D i f f e r e n c e s All three products have the basic structure in common. They all use the same concepts and structures. So, a cluster or a network, in the HACMP context, is the same, no matter what product is being...
Page 221 - haemd
HACMP Classic vs. HACMP/ES vs. HANFS 203 For switchless RS/6000 SP systems or SPs with the newer SP Switch, the decision will be based on a more functional level. Event Management is much more flexible in HACMP/ES, since you can define custom events. These events can act on anything that haemd can d...
Page 223 - Appendix A. Special Notices
© Copyright IBM Corp. 1999 205 Appendix A. Special Notices This publication is intended to help System Administrators, System Engineers and other System Professionals to pass the IBM HACMP Certification Exam. The information in this publication is not intended as the specification for any of the fol...
Page 224 - The following terms are trademarks of other companies:
206 IBM Certification Study Guide AIX HACMP been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk. Any pointers in ...
Page 225 - SUN Microsystems is a trademark of SUN Microsystems, Inc.
Special Notices 207 Java and HotJava are trademarks of Sun Microsystems, Incorporated. Microsoft, Windows, Windows NT, and the Windows 95 logo are trademarksor registered trademarks of Microsoft Corporation. PC Direct is a trademark of Ziff Communications Company and is usedby IBM Corporation under ...
Page 227 - Appendix B. Related Publications; B.1 International Technical Support Organization Publications
© Copyright IBM Corp. 1999 209 Appendix B. Related Publications The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this redbook. B.1 International Technical Support Organization Publications For information on ordering...
Page 228 - B.3 Other Publications
210 IBM Certification Study Guide AIX HACMP B.3 Other Publications These publications are also relevant as additional sources of information: • IBM RS/6000 SP: Planning, Volume 2, Control Workstation and Software Environment , GA22-7281 • IBM PSSP for AIX: Installation and Migration Guide , GA22-734...
Page 229 - H o w t o Ge t I T S O R e d b o o k s; H o w I B M E m p l o y e e s C a n G e t I T S O R e d b o o k s; R edpieces
© Copyright IBM Corp. 1999 211 H o w t o Ge t I T S O R e d b o o k s This section explains how both customers and IBM employees can find out about ITSO redbooks, CD-ROMs, workshops, and residencies. A form for ordering books and CD-ROMs is also provided. This information was current at the time of ...
Page 230 - H o w C u s t o m e r s C a n G e t I T S O R e d b o o k s; • Online Orders – send orders to:
212 IBM Certification Study Guide AIX HACMP H o w C u s t o m e r s C a n G e t I T S O R e d b o o k s Customers may request ITSO deliverables (redbooks, BookManager BOOKs, and CD-ROMs) and information about redbooks, workshops, and residencies in the following ways: • Online Orders – send orders t...
Page 231 - I B M R e d b o o k O r d e r F o r m
213 I B M R e d b o o k O r d e r F o r m Please send me the following: We accept American Express, Diners, Eurocard, Master Card, and Visa. Payment by credit card notavailable in all countries. Signature mandatory for credit card payment. Title Order Number Quantity First name Last name Company Add...
Page 233 - L i s t o f A b b r e v i a t i o n s
© Copyright IBM Corp. 1999 215 L i s t o f A b b r e v i a t i o n s AIX Advanced Interactive Executive APA All Points Addressable APAR Authorized Program Analysis Report The description of a problem to be fixed by IBM defect support. This fix is delivered in a PTF (see below). ARP Address Resolutio...
Page 239 - I T S O R e d b o o k E v a l u a t i o n; Overall Satisfaction; Please answer the following questions:; Was this redbook published in time for your needs?
© Copyright IBM Corp. 1999 221 I T S O R e d b o o k E v a l u a t i o n IBM Certification Study Guide AIX HACMP SG24-5131-00 Your feedback is very important to help us maintain the quality of ITSO redbooks. Please complete this questionnaire and return it using one of the following methods: • Use t...
Page 240 - IBM Certification Study Guide AIX HACMP
Pr int e d in the U .S.A . SG24- 5131- 00 IBM Certification Study Guide AIX HACMP SG24-5131-00