Download Compaq DS-DWZZH-05 Technical data
Transcript
TruCluster Server Cluster Hardware Configuration Technical Update for the AlphaServer ES45 May 2002 Product Version: TruCluster Server Version 5.1A Operating System and Version: Tru64 UNIX Version 5.1A This technical update describes how to configure an AlphaServer ES45 in a TruCluster Server environment, including configuration restrictions. Compaq Computer Corporation Houston, Texas © 2002 Compaq Computer Corporation Compaq, the Compaq logo, Alpha, AlphaServer, OpenVMS, StorageWorks, Tru64, and TruCluster are trademarks of Compaq Information Technologies Group, L.P. in the U.S and/or other countries. Microsoft, Windows, and Windows NT are trademarks of Microsoft Corporation in the U.S. and/or other countries. UNIX and The Open Group are trademarks of The Open Group in the U.S. and/or other countries. All other product names mentioned herein may be trademarks of their respective companies. Confidential computer software. Valid license from Compaq required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license. Compaq shall not be liable for technical or editorial errors or omissions contained herein. The information is provided “as is” without warranty of any kind and is subject to change without notice. The warranties for Compaq products are set forth in the express limited warranty statements accompanying such products. Nothing herein should be construed as constituting an additional warranty. Contents About This Technical Update 1 Overview 1.1 1.2 2 General Overview . . .. . .. . .. . . .. . .. . .. . .. . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . AlphaServer ES45 Overview . . .. . .. . .. . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . 1–1 1–1 Hardware Requirements and Restrictions 2.1 2.1.1 2.1.2 2.1.3 2.2 2.2.1 2.2.2 2.2.3 2.2.4 PCI Configuration . .. . .. . .. . . .. . .. . .. . .. . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . PCI Configuration Guidelines . . .. . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . PCI Modules Supported for Cluster Operations in a TruCluster Server Configuration . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . Differences in PCI Edge Connectors . .. . .. . .. . .. . . .. . .. . .. . .. . Hardware Restrictions . . .. . . .. . .. . .. . .. . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . Member System Restrictions .. . .. . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . Fibre Channel Restrictions . . .. . .. . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . Memory Channel Restrictions . .. . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . KZPBA-CB UltraSCSI Host Bus Adapter Restrictions . .. . 2–1 2–1 2–2 2–3 2–3 2–3 2–4 2–5 2–7 3 Installing an AlphaServer ES45 into a TruCluster Server Configuration 3.1 3.2 3.3 3.3.1 3.3.2 3.3.3 3.4 3.5 Preparing the AlphaServer ES45 for a TruCluster Server Configuration .. . . .. . .. . .. . .. . . .. . .. . .. . .. . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . Installing and Configuring the Cluster Interconnect . . .. . .. . .. . Installing and Configuring Shared Storage . .. . .. . .. . . .. . .. . .. . .. . TruCluster Server Configurations Using Fibre Channel for Storage . . . .. . .. . .. . .. . . .. . .. . .. . .. . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . TruCluster Server Configuration Using UltraSCSI Devices for Storage . .. . . .. . .. . .. . .. . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . TruCluster Server Configuration Using Non-UltraSCSI Devices for Storage . .. . . .. . .. . .. . .. . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . Displaying Adapters with the show Console Commands . .. . .. . Adding an ES45 with Dual Memory Channel Adapters to a Cluster with Memory Channel Adapters Jumpered for 512 MB . . . .. . .. . .. . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . 3–1 3–1 3–2 3–2 3–3 3–5 3–7 3–13 Contents iii Examples 3–1 3–2 3–3 3–4 Displaying Configuration on an AlphaServer ES45 in a Cluster Using a SCSI Bus for Shared Storage .. . .. . . .. . .. . .. . .. . Displaying Devices on an AlphaServer ES45 in a Cluster Using a SCSI bus for Shared Storage . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . Displaying Configuration on an AlphaServer ES45 in a Cluster Using Fibre Channel for Shared Storage .. . . .. . .. . .. . .. . Displaying Devices on an AlphaServer ES45 in a Cluster Using Fibre Channel for Shared Storage .. . .. . .. . .. . . .. . .. . .. . .. . 3–7 3–10 3–11 3–13 Figures 3–1 3–2 3–3 Fibre Channel HSG80 NSPOF Configuration . .. . .. . . .. . .. . .. . .. . TruCluster Server Configuration with UltraSCSI Components . . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . TruCluster Server Configuration with Non-UltraSCSI Devices Used for Storage .. . .. . .. . .. . . .. . .. . .. . .. . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . 3–3 3–4 3–6 Tables 2–1 2–2 3–1 3–2 iv Contents PCI Bus Configuration . . .. . . .. . .. . .. . .. . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . ES45 PCI Modules Supported in a TruCluster Server V5.1A Configuration for Cluster Operations . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . Hardware Components Shown in Figure 3–2 . .. . .. . . .. . .. . .. . .. . Hardware Components Used for Configuration Shown in Figure 3–3 . .. . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . .. . . .. . .. . .. . .. . . .. . .. . .. . .. . 2–1 2–2 3–4 3–7 About This Technical Update This technical update adds important information to the TruCluster™ Server Cluster Hardware Configuration manual about using the AlphaServer™ ES45 with the TruCluster Server Version 5.1A software product. Audience This technical update is for system administrators who plan to use an AlphaServer ES45 in a TruCluster Server configuration. New and Changed Features The following change has been made to this technical update since the 1 May 2002 edition: • Section 2.2.3 has been updated to reflect that the virtual hub restriction applies to systems that use the Memory Channel API for data transfers larger than 8 KB when loopback mode is enabled. Organization This technical update is organized as follows: Chapter 1 Provides a general overview of the AlphaServer ES45 system. Chapter 2 Provides the hardware requirements and restrictions that must be adhered to for an AlphaServer ES45 system configured in a TruCluster Server Version 5.1A cluster. Chapter 3 Provides directions for installing an AlphaServer ES45 in a TruCluster Server configuration. Related Documents Consult the following manuals for assistance in TruCluster Server hardware configuration, installation, administration, and programming tasks: • AlphaServer ES45 and AlphaStation ES45 Owner’s Guide — Provides basic operating instructions, describes how to boot an ES45, and shows how to install ES45 hardware components. • AlphaServer ES45 and AlphaStation ES45 Service Guide — Describes power-up and system reference manual (SRM) console diagnostics, explains how to interpret error logs, provides troubleshooting About This Technical Update v information, and provides procedures for removing and replacing field replaceable units (FRU). • AlphaServer ES45 Rackmount Installation Guide — Provides procedures for installing an AlphaServer ES45 rackmount system into a cabinet. • ES45 QuickSpecs — Describes the AlphaServer ES45 system. You can find the latest version of the QuickSpecs at: http://www.compaq.com/products/quickspecs/productbulletin.html Select U.S. QuickSpecs or World Wide QuickSpecs, then AlphaServers, then the Compaq AlphaServer ES45. • Supported Options List — Provides a list of options that are supported on the AlphaServer ES45. You can find the supported options list at: http://www.compaq.com/alphaserver/products/options.html • TruCluster Server QuickSpecs — Describes the TruCluster Server Version 5.1A product. You can find the latest version of the TruCluster Server QuickSpecs at: http://www.compaq.com/products/quickspecs/productbulletin.html Select U.S. QuickSpecs or World Wide QuickSpecs, then High Availability & Clustering, then Tru64 UNIX Clustering, and then Compaq TruCluster Server. • Cluster Release Notes — Provides important information about TruCluster Server Version 5.1A, including new features, known problems, and workarounds. • Cluster Technical Overview — Provides an overview of the TruCluster Server technology. • Cluster Hardware Configuration — Describes how to set up the processors that are to become cluster members, and how to configure cluster shared storage. • Cluster LAN Interconnect — Describes how to install and configure LAN hardware for the cluster interconnect. • Cluster Installation — Describes how to install the TruCluster Server product. • Cluster Administration — Describes cluster-specific administration tasks. • Cluster Highly Available Applications — Describes how to deploy applications on a TruCluster Server cluster and how to write cluster-aware applications. vi About This Technical Update 1 Overview This technical update to the TruCluster Server Cluster Hardware Configuration manual provides important information about support for the AlphaServer ES45 with the TruCluster Server Version 5.1A product. 1.1 General Overview The AlphaServer ES45 was recently qualified for use with the TruCluster Server Version 5.1A product. No TruCluster Server software release coincides with the availability of support for the AlphaServer ES45, so this technical update adds the AlphaServer ES45 configuration information that will appear in future revisions of the TruCluster Server Cluster Hardware Configuration manual. Use this technical update in conjunction with the TruCluster Server Cluster Hardware Configuration manual. 1.2 AlphaServer ES45 Overview The AlphaServer ES45 is a 1000 MHz follow-on to the AlphaServer ES40. It can employ from 1 through 4 processors. Only the ES45 Model 2 is presently supported. Other models, with different PCI/AGP bus configurations, will follow. A 10-slot PCI backplane provides six slots that support hot-swapping. ______________________ Note _______________________ The Tru64 UNIX Version 5.1A operating system does not support hot-swapping PCI modules. The ES45 Model 2 has four PCI buses (hoses). The PCI backplane supports four 5.0V 33-MHz PCI slots and six 3.3V 66-MHz PCI slots. The AlphaServer ES45 is available in a tower, pedestal, and rackmount (M-Series cabinet) configuration. Overview 1–1 2 Hardware Requirements and Restrictions This chapter describes the hardware requirements and restrictions for configuring an AlphaServer ES45 system in a TruCluster Server Version 5.1A cluster. For other restrictions not listed here, see the hardware requirements and restrictions chapter of Cluster Hardware Configuration. 2.1 PCI Configuration The AlphaServer ES45 Model 2 supports a 10-slot PCI backplane. It consists of both 3.3V 66-MHz and 5.0V 33-MHz slots over four PCI buses (hoses). Table 2–1 lists the PCI bus configuration. The fourth column lists the logical slot number that is displayed by a show config console command. Table 2–1: PCI Bus Configuration PCI Slota Voltageb Bus Numberc SRM Logical Slotd 1 3.3V 2 Hose 2 Slot 1 2 3.3V 2 Hose 2 Slot 2 3 5.0V 0 Hose 0 Slot 11 4 3.3V 3 Hose 3 Slot 2 5 3.3V 3 Hose 3 Slot 1 6 5.0V 0 Hose 0 Slot 10 7 3.3V 1 Hose 1 Slot 2 8 3.3V 1 Hose 1 Slot 1 9 5.0V 0 Hose 0 Slot 9 10 5.0V 0 Hose 0 Slot 8 a PCI slots 4 through 10 are hot-pluggable slots (not supported in Tru64 UNIX Version 5.1A). b The 3.3V slots operate at a bus speed of 66 MHz; the 5.0V slots operate at a bus speed of 33 MHz. c The bus number shows up as hose number in the show config command output. d A device installed in the physical PCI slot (shown in the first column) will have the logical slot ID (shown in the last column) in the show config command output. 2.1.1 PCI Configuration Guidelines Ensure that you abide by the following PCI configuration restrictions: Hardware Requirements and Restrictions 2–1 • PCI modules designed for 5.0V only operate in 5.0V slots (slots 3, 6, 9, and 10). • PCI modules designed for 3.3V only operate in 3.3V slots (slots 1, 2, 4, 5, 7, and 8). • Universal PCI modules can operate in either 3.3V or 5.0V slots. • A universal PCI module that operates at 66 MHz will only operate at 33 MHz if it is plugged into a 5.0V slot. • A universal PCI module that operates at 33 MHz will cause the entire 66-MHz bus to operate at 33 MHz if it is plugged into a 3.3V slot. 2.1.2 PCI Modules Supported for Cluster Operations in a TruCluster Server Configuration Table 2–2 lists the PCI modules that are supported on an AlphaServer ES45 Model 2 used in TruCluster Server V5.1A cluster configurations for shared storage and cluster communications over the cluster interconnect. Other adapters are supported for local storage and the public network that are not listed here. See the supported options list for information about other supported devices. Table 2–2: ES45 PCI Modules Supported in a TruCluster Server V5.1A Configuration for Cluster Operations PCI Module Description Number Supported Restricted to PCI Slots UltraSCSI Host Bus Adapter (KZPBA-CB) 5.0V/33 MHz 4a 3, 6, 9, and 10 Fibre Channel Host Bus Adapter (DS-KGPSA-CA) Universal/33 MHz 4b No slot restrictionc Memory Channel (CCMAB-AA) 5.0V/33 MHz 2d 3, 6, 9, and 10 3X-DE602-AA Universal/33 MHz 2 No slot restrictionsc DEGPA-SA Universal/66 MHz 3 1, 2, and 3e DEGPA-TA Universal/66 MHz 4 1, 2, 3, 4, 5, 6, 9, and 10ef PCI Module Shared Storage Cluster Interconnect a Although 4 KZPSA-CBs can be supported, the number of KZPBA-CB UltraSCSI host bus adapters that can be installed depends upon the number of Memory Channel modules installed in slots 3, 6, 9, and 10. b Four DS-KGPSA-CA Fibre Channel adapters are supported in a switched configuration; A loop configuration is not supported in a TruCluster Server configuration. c If you install a universal 33 Mhz PCI host bus adapter in a 3.3V 66 MHz slot (slots 1, 2, 4, 5, 7, or 8), the bus will only operate at 33 MHz. d Dual Memory Channel adapters must be jumpered for 128 MB. 2–2 Hardware Requirements and Restrictions Table 2–2: ES45 PCI Modules Supported in a TruCluster Server V5.1A Configuration for Cluster Operations (cont.) e If you install a universal 66 Mhz PCI host bus adapter in a 5.0V 33 MHz slot (slots 3, 6, 9, or 10), the module will only operate at 33 MHz. f The DEGPA-TA Gigabit Ethernet adapter cannot be installed in PCI bus 1. 2.1.3 Differences in PCI Edge Connectors The differences in PCI slots and module edge connectors prevent you from installing a 5.0V module into a 3.3V slot. _____________________ Caution _____________________ Do not attempt to plug a 5.0V module into a 3.3V PCI slot; you can damage the module or PCI backplane. A universal PCI module, which operates at either 3.3V or 5.0V, can be plugged into any PCI slot. 2.2 Hardware Restrictions The following sections describe the hardware requirements and restrictions particular to each device. 2.2.1 Member System Restrictions The requirements for member systems in a TruCluster Server cluster are as follows: • The minimum ES45 Alpha System Reference Manual (SRM) console firmware required is V5.9-10. You can obtain firmware information from the Compaq Web site at the following URL: http://www.compaq.com. In the Support column, select Software & Drivers, then in the Servers column, select AlphaServer, then select the AlphaServer ES45 system. • If the cluster has any system with SRM console firmware earlier than Version 5.7, that system may not be able to boot from a disk behind an HSZ80, HSG60, or HSG80 controller. The member may fail to boot, indicating "Reservation Conflict" errors. • TruCluster Server Version 5.1A supports eight-member cluster configurations as follows: Hardware Requirements and Restrictions 2–3 – Fibre Channel: Eight-member systems may be connected to common storage over Fibre Channel in a fabric (switch) configuration. – Parallel SCSI: Only four of the member systems may be connected to any one SCSI bus, but you can have multiple SCSI buses connected to different sets of nodes, and the sets of nodes may overlap. We recommend you use a DS-DWZZH-05 UltraSCSI hub with fair arbitration enabled when connecting four-member systems to a common SCSI bus using RAID array controllers. Illustrations of an externally terminated eight-node cluster are shown in the chapter about configuring an eight-member cluster using externally terminated shared SCSI buses in Cluster Hardware Configuration. The cluster shown is more appropriate for high performance technical computing (HPTC) customers who are looking for performance instead of availability. 2.2.2 Fibre Channel Restrictions For the latest information about supported hardware, see the AlphaServer ES45 supported options list. The following requirements and restrictions apply to the use of Fibre Channel with TruCluster Server Version 5.1A: • The AlphaServer ES45 only supports the KGPSA-CA PCI-to-Fibre Channel host bus adapters. A maximum of four adapters is supported. • Fibre Channel is only supported in a fabric configuration; TruCluster Server does not support an arbitrated loop configuration. A non-TruCluster Server configuration supports one KGPSA-CA in an arbitrated loop configuration. • The HSG60 and HSG80 require Array Control Software (ACS) Version 8.6 or later. • Eight member systems may be connected to common storage over Fibre Channel in a fabric (switch) configuration. • The only Fibre Channel switches supported are the DS-DSGGA-AA/AB 8/16 port, DS-DSGGB-AA/AB 8/16 port, and DS-DSGGC-AA/AB 8/16 port Fibre Channel switches. _____________________ Note _____________________ Although the ES45 was qualified with the DS-DSGGA-AA/AB Fibre Channel switches, they are not listed in the ES45 QuickSpecs or supported options list because they have gone end-of-life. 2–4 Hardware Requirements and Restrictions • The DSGGA, DSGGB, and DSGGC Fibre Channel switches support both shortwave (GBIC-SW) and longwave (GBIC-LW) Gigabit Interface Converter (GBIC) modules. Seven of the eight DSGGC-AA ports are fixed shortwave optical transceivers. Only one DSGGC-AA port is configured as a removable GBIC. It may be shortwave or longwave. The GBIC-SW module supports 50-micron, multimode fiber cables with the standard subscriber connector (SC) connector in lengths up to 500 meters (1640.4 feet). It also supports 62.5-micron multimode fiber cables in lengths up to 200 meters (656.2 feet). The GBIC-LW supports 9-micron, single-mode fiber cables with the SC connector in lengths up to 10 kilometers (6.2 miles). The KGPSA-CA PCI-to-Fibre Channel host bus adapter and the HSG60 and HSG80 RAID controller support the 50-micron Gigabit Link Module (GLM) for fiber connections. Therefore, only the 50-micron multimode fiber-optic cable is supported between the KGPSA and switch and between the switch and HSG60 or HSG80 for cluster configurations. You must install GBIC-SW GBICs in the Fibre Channel switches for communication between the switches and KGPSA or HSG60/HSG80. • The HSG60 and HSG80 supports transparent and multiple-bus failover mode when used in a TruCluster Server Version 5.1A configuration. Multiple-bus failover is recommended. • For other Fibre Channel restrictions, see the hardware requirements and restrictions chapter in Cluster Hardware Configuration. 2.2.3 Memory Channel Restrictions The Memory Channel interconnect is one method used for cluster communications between the member systems. Ensure that you abide by the following Memory Channel restrictions: • The AlphaServer ES45 systems only support Memory Channel 2 (MC2) hardware. • The Memory Channel adapter is a 5V 33-MHz module and is restricted to PCI bus 0 (slots 3, 6, 9, and 10). Installing a redundant Memory Channel adapter places both Memory Channel adapters on the same PCI bus. • If redundant Memory Channel adapters are used with an ES45, they must be jumpered for 128 MB and not the default of 512 MB. If there is only one Memory Channel adapter on the PCI bus, it may be jumpered for 512 MB. • An ES45 system with dual Memory Channel adapters cannot be added to an existing cluster that has Memory Channel adapters jumpered for 512 MB. The ES45 cannot join the cluster. The Hardware Requirements and Restrictions 2–5 startup error message on the ES45 jumpered for 128 MB states: panic: MC2 adapter has too little memory See Section 3.5 if you intend to add an ES45 system with dual Memory Channel adapters jumpered for 128 MB into an existing cluster with Memory Channel adapters jumpered for 512 MB. • The Memory Channel API is not supported for data transfers larger than 8 KB when loopback mode is enabled in two member clusters configured in virtual hub mode. See the TruCluster Server Version 5.1A QuickSpecs for more information. • A mixed-version dual Memory Channel configuration (MC1 and MC2 or MC1.5 and MC2) is not supported in a cluster containing an ES45 because the ES45 only supports MC2. • If you configure a cluster with a single rail Memory Channel in standard hub mode and the hub fails or is turned off, every cluster member panics. They panic because no member can see any of the other cluster members over the Memory Channel interface. A quorum disk does not help in this case, because no system is given the opportunity to obtain ownership of the quorum disk and survive. To prevent this situation in standard hub mode, install a second Memory Channel rail. A hub failure on one rail will cause failover to the other rail. When the Memory Channel is set up in standard hub mode, the Memory Channel hub must be visible to each member’s Memory Channel adapter. If the hub is powered off, no system is able to boot. • If a TruCluster Server cluster configuration utilizes multiple Memory Channel adapters in standard hub mode, the Memory Channel adapters must be connected to separate Memory Channel hubs. The first Memory Channel adapter (mca0) in each system must be connected to one Memory Channel hub. The second Memory Channel adapter (mcb0) in each system must be connected to a second Memory Channel hub. Also, each Memory Channel adapter on one system must be connected to the same linecard in each Memory Channel hub. • The maximum length of an MC2 BN39B link cable is 10 meters (32.8 feet). • You can use a CCMFB optical converter in conjunction with the MC2 CCMAB host bus adapter or a CCMLB hub line card to increase the distance between systems. If you use a CCMFB Memory Channel optical converter module, which is a 5.0V module, it must be plugged into PCI slot 3, 6, 9, or 10. The BN34R fiber-optic cable, which is used to connect two CCMFB optical converters, is available in 10-meter (32.8-foot) (BN34R-10) and 2–6 Hardware Requirements and Restrictions 31-meter (101.7-foot) (BN34R-31) lengths. Customers may provide their own fiber-optic cables to achieve greater separation of systems. • Always examine a Memory Channel link cable for bent or broken pins. Be sure that you do not bend or break any pins when you connect or disconnect a cable. 2.2.4 KZPBA-CB UltraSCSI Host Bus Adapter Restrictions KZPBA-CB UltraSCSI adapters have the following restrictions: • The KZPBA-CB UltraSCSI host bus adapter is a 5V 32-bit/33-MHz module and therefore is restricted to PCI bus 0 (slots 3, 6, 9, and 10). • Only four KZPBA-CBs are supported. If a Memory Channel module or any other modules are installed in PCI 0, the maximum number of KZPBA-CB modules is reduced by the number of other modules installed in PCI 0 slots. • The KZPBA-CB requires ISP 1020/1040 firmware Version 5.57 or higher, which is available with the system SRM console firmware on the Alpha Systems Firmware 5.3 Update CD-ROM (or later). • The maximum length of any differential SCSI bus segment is 25 meters (82 feet), including the length of the SCSI bus cables and SCSI bus internal to the SCSI adapter, hub, or storage device. A SCSI bus may have more than one SCSI bus segment. • See the KZPBA-CB UltraSCSI Storage Adapter Module Release Notes for more information. Hardware Requirements and Restrictions 2–7 3 Installing an AlphaServer ES45 into a TruCluster Server Configuration 3.1 Preparing the AlphaServer ES45 for a TruCluster Server Configuration The AlphaServer ES45 has PCI slot restrictions that must be adhered to, but otherwise, creating a cluster of ES45 systems, or adding an ES45 system to an existing cluster, is basically the same as any AlphaServer system supported in a TruCluster Server configuration. Cluster hardware installation can be divided into two steps: • Installing and configuring the cluster interconnect • Installing and configuring shared storage ______________________ Note _______________________ If you plan to add an ES45 with dual Memory Channel adapters (which must be jumpered for 128 MB) into an existing cluster with Memory Channel adapters jumpered for 512 MB, you must reset the jumpers on the existing Memory Channel adapters to 128 MB before booting the ES45 into the cluster. The following sections provide information about installing an AlphaServer ES45 into a TruCluster Server configuration. 3.2 Installing and Configuring the Cluster Interconnect A TruCluster Server configuration requires a cluster interconnect connecting all cluster members together. The cluster connect can be provided by Memory Channel (Memory Channel 2) hardware or a private local area network (LAN): • To install MC2 hardware as the cluster interconnect, review the restrictions listed in Section 2.2.3 and see the chapter about setting Installing an AlphaServer ES45 into a TruCluster Server Configuration 3–1 up a Memory Channel cluster interconnect in Cluster Hardware Configuration. • To install and configure a local area network as the cluster interconnect, see the Cluster LAN Interconnect manual. ______________________ Note _______________________ You may save time if you install host bus adapters for shared storage, cluster interconnect cards (either Memory Channel or Ethernet cards for a private LAN), and extra Ethernet cards all at the same time. 3.3 Installing and Configuring Shared Storage The shared storage for your cluster can be any of three different technologies: • Fibre Channel (HSG60 and HSG80 RAID array controllers). • UltraSCSI components (HSZ70, HSZ80, or RAID Array 3000 with an HSZ22 controller connected to a DWZZH-03 or DWZZH-05 UltraSCSI hub). Although the UltraSCSI BA356 could be considered in this group, it is not discussed here because it cannot be connected to an UltraSCSI hub. • Non-UltraSCSI components (BA350 and BA356). The UltraSCSI BA356 is discussed in this group. The following sections provide more information on a TruCluster Server configuration using each of the three categories of storage. 3.3.1 TruCluster Server Configurations Using Fibre Channel for Storage If your shared storage consists of HSG60 or HSG80 RAID array controllers connected to a storage array network (SAN) and KGPSA-CA Fibre Channel host bus adapter, review the restrictions in Section 2.2.2, see Section 3.4, and see the chapter about using Fibre Chanel storage in Cluster Hardware Configuration for installation information. Figure 3–1 shows an example of a no-single-point-of-failure (NSPOF) Fibre Channel configuration using HSG80 RAID array controllers. Either a Memory Channel interface or a private LAN can be used as the cluster interconnect. 3–2 Installing an AlphaServer ES45 into a TruCluster Server Configuration Figure 3–1: Fibre Channel HSG80 NSPOF Configuration Member System 1 Cluster Interconnects Member System 2 KGPSA KGPSA KGPSA KGPSA Fibre Channel Switch Port 1 Port 1 Fibre Channel Switch HSG 80 Controller A HSG 80 Controller B Port 2 Port 2 RA8000/ESA12000 ZK-1707U-AI 3.3.2 TruCluster Server Configuration Using UltraSCSI Devices for Storage If your shared storage consists of disks attached to an UltraSCSI RAID array controller (HSZ70, HSZ80, RAID array 3000) connected to a DWZZH-03 or DWZZH-05 UltraSCSI hub and KZPBA-CB UltraSCSI host bus adapters, see Section 3.4. Also, in Cluster Hardware Configuration, for installation procedures, see the chapter about using UltraSCSI hardware to configure a cluster and, for example configurations, see the chapter about shared SCSI bus requirements and configurations using UltraSCSI hardware. Figure 3–2 shows an example of a TruCluster Server configuration using UltraSCSI devices for storage. Installing an AlphaServer ES45 into a TruCluster Server Configuration 3–3 Figure 3–2: TruCluster Server Configuration with UltraSCSI Components Networks Member System 2 Member System 1 Cluster Interconnects T KZPBA-CB (ID 6) T KZPBA-CB (ID 7) KZPBA-CB (ID 6) T 1 1 T KZPBA-CB (ID 7) 1 1 T T DS-DWZZH-03 T DS-DWZZH-03 T T T 2 3 Port 1 4 4 T T Port 2 2 3 Port 1 Controller A HSZ80 Port 2 Controller B HSZ80 StorageWorks RAID Array 8000 ZK-1602U-AI Table 3–1 lists the components used to create the cluster shown in Figure 3–2. Table 3–1: Hardware Components Shown in Figure 3–2 Callout Number Description 1 BN38C or BN38D cablea 2 BN37A cableb 3 H8861-AA VHDCI trilink connector 4 H8863-AA VHDCI terminator a The maximum length of the BN38C (or BN38D) cable on one SCSI bus segment must not exceed 25 meters (82 feet). b The maximum length of the BN37A cables must not exceed 25 meters (82 feet). 3–4 Installing an AlphaServer ES45 into a TruCluster Server Configuration 3.3.3 TruCluster Server Configuration Using Non-UltraSCSI Devices for Storage If your shared storage consists of disks in BA350, BA356, or UltraSCSI BA356 storage units, or non-UltraSCSI RAID array controllers (HSZ40 and HSZ50) connected to a KZPBA-CB UltraSCSI host bus adapter on a SCSI bus using external termination, see Section 3.4. Also, in Cluster Hardware Configuration, for installation procedures, see the chapter about configuring systems for external termination or radial connections to non-UltraSCSI devices, and, for example configurations, the chapter about system configurations using external termination or radial connections to non-UltraSCSI devices. Figure 3–3 shows an example of a TruCluster Server configuration using non-UltraSCSI devices for storage. Installing an AlphaServer ES45 into a TruCluster Server Configuration 3–5 Figure 3–3: TruCluster Server Configuration with Non-UltraSCSI Devices Used for Storage Network Member System 1 Member System 2 Cluster Interconnect KZPSA-BB (ID 6) T KZPSA-BB (ID 7) 2 T 2 1 1 3 3 BA356 BA356 3 4 Do not use for data disk. May be used for redundant power supply. DWZZB-VW DWZZB-VW Clusterwide /, /usr, /var ID 1 Member 1 Boot Disk ID 2 Member 2 Boot Disk ID 3 Quorum Disk ID 4 Data disk ID 5 4 ID 9 ID 10 Data Disks ID 12 ID 13 ID 6 PWR (7) ID 11 PWR (15) ID 14 or redundant power supply ZK-1592U-AI Table 3–2 lists the components that are used to create the cluster that is shown in Figure 3–3. 3–6 Installing an AlphaServer ES45 into a TruCluster Server Configuration Table 3–2: Hardware Components Used for Configuration Shown in Figure 3–3 Callout Number Description 1 BN21W-0B Y cable 2 H879-AA terminator 3 BN21K, BN21L, or 328215-00X cablea 4 H885-AA trilink connector a The maximum combined length of the BN21K, BN21L, or 328215-00X cables must not exceed 25 meters (82 feet). 3.4 Displaying Adapters with the show Console Commands The Cluster Hardware Configuration manual does not have any examples that include the AlphaServer ES45. This section provides examples of using the show config and show device console commands to display ES45 system configuration information. Example 3–1 and Example 3–2 show a cluster configuration that uses a shared SCSI bus for storage. Example 3–3 and Example 3–4 show a cluster configuration that uses Fibre Channel for shared storage. Example 3–1 shows the output from the show config console command on an AlphaServer ES45 system in a cluster using a SCSI bus for shared storage. Example 3–1: Displaying Configuration on an AlphaServer ES45 in a Cluster Using a SCSI Bus for Shared Storage P00>>> show config Compaq Computer Corporation Compaq AlphaServer ES45 Model 2 Firmware SRM Console: PALcode: T6.1-5 OpenVMS PALcode V1.93-36, Tru64 UNIX PALcode V1.88-28 Serial ROM: V2.18-F RMC ROM: V1.0 RMC Flash ROM: T1.6 Processors CPU 0 CPU 1 CPU 2 CPU 3 Alpha Alpha Alpha Alpha EV68CB EV68CB EV68CB EV68CB pass pass pass pass 2 2 2 2 or or or or 2.1 2.1 2.1 2.1 1000 1000 1000 1000 MHz MHz MHz MHz 8MB 8MB 8MB 8MB Bcache Bcache Bcache Bcache Core Logic Installing an AlphaServer ES45 into a TruCluster Server Configuration 3–7 Example 3–1: Displaying Configuration on an AlphaServer ES45 in a Cluster Using a SCSI Bus for Shared Storage (cont.) Cchip Dchip PPchip 0 PPchip 1 TIG Memory Array --------0 1 2 3 Rev Rev Rev Rev Rev 17 17 17 17 2.6 Size ---------2048MB 2048MB 2048MB 2048MB Base Address ---------------0000000000000000 0000000080000000 0000000100000000 0000000180000000 Intlv Mode ---------4-Way 4-Way 4-Way 4-Way 8192 MB of System Memory Slot 7 8 9 10 11 12 16 Option Acer Labs M1543C NCR 53C895 DEC PCI MC QLogic ISP10x0 Hose 0, Bus 0, PCI - 33 MHz Bridge to Bus 1, ISA 1 pka0.7.0.8.0 SCSI Bus ID 7 dka0.0.0.8.0 COMPAQ BD018635CC dka100.1.0.8.0 COMPAQ BD018635CC mca0.0.0.9.0 Rev: 24, mca0 2 pkb0.5.0.10.0 SCSI Bus ID 5 3 dkb0.0.0.10.0 HSZ70 dkb1.0.0.10.0 HSZ70 dkb100.1.0.10.0 HSZ70 dkb101.1.0.10.0 HSZ70 . . . dkb205.2.0.10.0 dkb206.2.0.10.0 dkb207.2.0.10.0 dkb3.0.0.10.0 dkb300.3.0.10.0 dkb4.0.0.10.0 dkb5.0.0.10.0 dkb6.0.0.10.0 dkb7.0.0.10.0 mcb0.0.0.11.0 DEC PCI MC Yukon PCI Hot-Plug C Acer Labs M1543C IDE dqa.0.0.16.0 dqb.0.1.16.0 dqa0.0.0.16.0 Option Floppy HSZ70 HSZ70 HSZ70 HSZ70 HSZ70CCL HSZ70 HSZ70 HSZ70 HSZ70 Rev: 24, mcb0 2 Compaq CRD-8402B Hose 0, Bus 1, ISA dva0.0.0.1000.0 3–8 Installing an AlphaServer ES45 into a TruCluster Server Configuration Example 3–1: Displaying Configuration on an AlphaServer ES45 in a Cluster Using a SCSI Bus for Shared Storage (cont.) Slot 2 6 Option Hose 1, Bus 0, PCI - 33 MHz DEFPA fwa0.0.0.2.1 00-60-6D-93-A0-1A Yukon PCI Hot-Plug C Slot 1/0 Option Adaptec AIC-7899 1/1 Adaptec AIC-7899 Slot 6 P00>>> Option Hose 2, Bus 0, PCI - 66 MHz 4 pkc0.7.0.1.2 SCSI Bus ID 7 dkc0.0.0.1.2 COMPAQ BD036635C5 dkc100.1.0.1.2 COMPAQ BD036635C5 dkc200.2.0.1.2 COMPAQ BD0186349B dkc300.3.0.1.2 COMPAQ BD0096349A dkc400.4.0.1.2 COMPAQ BD0096349A dkc500.5.0.1.2 COMPAQ BD0186349B pkd0.7.0.101.2 SCSI Bus ID 7 dkd0.0.0.101.2 COMPAQ BD0186349B dkd100.1.0.101.2 COMPAQ BF00963643 dkd200.2.0.101.2 COMPAQ BD0096349A dkd300.3.0.101.2 COMPAQ BD0186349B dkd400.4.0.101.2 COMPAQ BD0096349A dkd500.5.0.101.2 COMPAQ BD0186349B Hose 3, Bus 0 Yukon PCI Hot-Plug C 1 KZPCA (internal, non-shared storage). 2 Memory Channel 2. 3 KZPBA (shared bus): Ensure that the KZPBA UltraSCSI host bus adapters on each system have different SCSI IDs. See the chapter about configuring a cluster using UltraSCSI hardware in Cluster Hardware Configuration. 4 KZPEA (local storage, non-shared bus). Example 3–2 shows the output from the show device console command entered on an AlphaServer ES45 system in a cluster using a SCSI bus for shared storage. Installing an AlphaServer ES45 into a TruCluster Server Configuration 3–9 Example 3–2: Displaying Devices on an AlphaServer ES45 in a Cluster Using a SCSI bus for Shared Storage P00>>> show device dka0.0.0.8.0 dka100.1.0.8.0 dkb0.0.0.10.0 dkb1.0.0.10.0 dkb100.1.0.10.0 dkb101.1.0.10.0 . . . dkb205.2.0.10.0 dkb206.2.0.10.0 dkb207.2.0.10.0 dkb3.0.0.10.0 dkb300.3.0.10.0 dkb4.0.0.10.0 dkb5.0.0.10.0 dkb6.0.0.10.0 dkb7.0.0.10.0 dkc0.0.0.1.2 dkc100.1.0.1.2 dkc200.2.0.1.2 dkc300.3.0.1.2 dkc400.4.0.1.2 dkc500.5.0.1.2 dkd0.0.0.101.2 dkd100.1.0.101.2 dkd200.2.0.101.2 dkd300.3.0.101.2 dkd400.4.0.101.2 dkd500.5.0.101.2 dqa0.0.0.16.0 dva0.0.0.1000.0 fwa0.0.0.2.1 pka0.7.0.8.0 pkb0.5.0.10.0 pkc0.7.0.1.2 pkd0.7.0.101.2 P00>>> DKA0 DKA100 DKB0 DKB1 DKB100 DKB101 COMPAQ BD018635CC COMPAQ BD018635CC HSZ70 HSZ70 HSZ70 HSZ70 BDC4 BDC4 V77Z V77Z V77Z V77Z DKB205 DKB206 DKB207 DKB3 DKB300 DKB4 DKB5 DKB6 DKB7 DKC0 DKC100 DKC200 DKC300 DKC400 DKC500 DKD0 DKD100 DKD200 DKD300 DKD400 DKD500 DQA0 DVA0 FWA0 PKA0 PKB0 PKC0 PKD0 HSZ70 HSZ70 HSZ70 HSZ70 HSZ70CCL HSZ70 HSZ70 HSZ70 HSZ70 BD036635C5 BD036635C5 BD0186349B BD0096349A BD0096349A BD0186349B BD0186349B BF00963643 BD0096349A BD0186349B BD0096349A BD0186349B CRD-8402B V77Z V77Z V77Z V77Z V77Z V77Z V77Z V77Z V77Z B017 B017 3B12 3B12 3B12 3B12 3B12 3B05 3B12 3B12 3B12 3B12 1.03 00-60-6D-93-A0-1A SCSI Bus ID 7 SCSI Bus ID 5 SCSI Bus ID 7 SCSI Bus ID 7 5.57 COMPAQ COMPAQ COMPAQ COMPAQ COMPAQ COMPAQ COMPAQ COMPAQ COMPAQ COMPAQ COMPAQ COMPAQ Compaq Example 3–3 shows the output from the show config console command on an AlphaServer ES45 system in a cluster using Fibre Channel for shared storage. 3–10 Installing an AlphaServer ES45 into a TruCluster Server Configuration Example 3–3: Displaying Configuration on an AlphaServer ES45 in a Cluster Using Fibre Channel for Shared Storage P00>>> show config Compaq Computer Corporation Compaq AlphaServer ES45 Model 2 Firmware SRM Console: PALcode: T6.1-5 OpenVMS PALcode V1.93-36 Tru64 UNIX PALcode V1.88-28 Serial ROM: V2.18-F RMC ROM: V1.0 RMC Flash ROM: T1.6 Processors CPU 0 CPU 1 CPU 2 CPU 3 Alpha Alpha Alpha Alpha Core Logic Cchip Dchip PPchip 0 PPchip 1 TIG Rev Rev Rev Rev Rev Memory Array --------0 1 2 3 EV68CB EV68CB EV68CB EV68CB pass pass pass pass 2 2 2 2 or or or or 2.1 2.1 2.1 2.1 1000 1000 1000 1000 MHz MHz MHz MHz 8MB 8MB 8MB 8MB Bcache Bcache Bcache Bcache 17 17 17 17 2.6 Size ---------2048MB 2048MB 2048MB 2048MB Base Address ---------------0000000000000000 0000000080000000 0000000100000000 0000000180000000 Intlv Mode ---------4-Way 4-Way 4-Way 4-Way 8192 MB of System Memory Slot 7 8 9 11 12 16 Option Acer Labs M1543C NCR 53C895 Hose 0, Bus 0, PCI - 33 MHz Bridge to Bus 1, ISA pka0.7.0.8.0 SCSI Bus ID 7 1 dka0.0.0.8.0 COMPAQ BD018635CC dka100.1.0.8.0 COMPAQ BD018635CC mca0.0.0.9.0 Rev: 24, mca0 2 mcb0.0.0.11.0 Rev: 24, mcb0 2 DEC PCI MC DEC PCI MC Yukon PCI Hot-Plug C Acer Labs M1543C IDE dqa.0.0.16.0 dqb.0.1.16.0 dqa0.0.0.16.0 Compaq CRD-8402B Installing an AlphaServer ES45 into a TruCluster Server Configuration 3–11 Example 3–3: Displaying Configuration on an AlphaServer ES45 in a Cluster Using Fibre Channel for Shared Storage (cont.) Option Floppy Hose 0, Bus 1, ISA dva0.0.0.1000.0 Slot 2 6 Option Hose 1, Bus 0, PCI - 33 MHz DEFPA fwa0.0.0.2.1 00-60-6D-93-A0-1A Yukon PCI Hot-Plug C Slot 1/0 Option Adaptec AIC-7899 1/1 Adaptec AIC-7899 Slot 1 2 6 P00>>> Hose 2, Bus 0, PCI - 66 MHz 3 pkb0.7.0.1.2 SCSI Bus ID 7 dkb0.0.0.1.2 COMPAQ BD036635C5 dkb100.1.0.1.2 COMPAQ BD036635C5 dkb200.2.0.1.2 COMPAQ BD0186349B dkb300.3.0.1.2 COMPAQ BD0096349A dkb400.4.0.1.2 COMPAQ BD0096349A dkb500.5.0.1.2 COMPAQ BD0186349B pkc0.7.0.101.2 SCSI Bus ID 7 dkd0.0.0.101.2 COMPAQ BD0186349B dkd100.1.0.101.2 COMPAQ BF00963643 dkd200.2.0.101.2 COMPAQ BD0096349A dkd300.3.0.101.2 COMPAQ BD0186349B dkd400.4.0.101.2 COMPAQ BD0096349A dkd500.5.0.101.2 COMPAQ BD0186349B Option KGPSA-C KGPSA-C Hose 3, Bus 0, PCI - 33 MHz pga0.0.0.1.3 WWN 2000-0000-c924-5d44 pgb0.0.0.2.3 WWN 2000-0000-c924-5012 dgb42.1001.0.2.3 HSG80 dgb42.1002.0.2.3 HSG80 dgb42.1003.0.2.3 HSG80 dgb42.1004.0.2.3 HSG80 Yukon PCI Hot-Plug C 4 4 1 KZPCA (internal, non-shared storage). 2 Memory Channel 2. 3 KZPEA (local storage, non-shared bus). 4 Fibre Channel host bus adapter (shared storage). KGPSA-CA pga is not connected to a Fibre Channel switch. KGPSA-CA pgb is connected to a Fibre Channel switch and has access to disks on the HSG80. 3–12 Installing an AlphaServer ES45 into a TruCluster Server Configuration Example 3–4 shows the output from the show device console command entered on an AlphaServer ES45 system in a cluster using Fibre Channel for shared storage. Example 3–4: Displaying Devices on an AlphaServer ES45 in a Cluster Using Fibre Channel for Shared Storage P00>>> show device P00>>>show device dgb42.1001.0.2.3 dgb42.1002.0.2.3 dgb42.1003.0.2.3 dgb42.1004.0.2.3 dka0.0.0.8.0 dka100.1.0.8.0 dkb0.0.0.1.2 dkb100.1.0.1.2 dkb200.2.0.1.2 dkb300.3.0.1.2 dkb400.4.0.1.2 dkb500.5.0.1.2 dkc0.0.0.101.2 dkc100.1.0.101.2 dkc200.2.0.101.2 dkc300.3.0.101.2 dkc400.4.0.101.2 dkc500.5.0.101.2 dqa0.0.0.16.0 dva0.0.0.1000.0 fwa0.0.0.2.1 pga0.0.0.1.3 pgb0.0.0.2.3 pka0.7.0.8.0 pkb0.7.0.1.2 pkc0.7.0.101.2 P00>>> $1$DGA42 $1$DGA42 $1$DGA42 $1$DGA42 DKA0 DKA100 DKB0 DKB100 DKB200 DKB300 DKB400 DKB500 DKC0 DKC100 DKC200 DKC300 DKC400 DKC500 DQA0 DVA0 FWA0 PGA0 PGB0 PKA0 PKB0 PKC0 COMPAQ COMPAQ COMPAQ COMPAQ COMPAQ COMPAQ COMPAQ COMPAQ COMPAQ COMPAQ COMPAQ COMPAQ COMPAQ COMPAQ Compaq HSG80 HSG80 HSG80 HSG80 BD018635CC BD018635CC BD036635C5 BD036635C5 BD0186349B BD0096349A BD0096349A BD0186349B BD0186349B BF00963643 BD0096349A BD0186349B BD0096349A BD0186349B CRD-8402B V86F V86F V86F V86F BDC4 BDC4 B017 B017 3B12 3B12 3B12 3B12 3B12 3B05 3B12 3B12 3B12 3B12 1.03 00-60-6D-93-A0-1A WWN 2000-0000-c924-5d44 WWN 2000-0000-c924-5012 SCSI Bus ID 7 SCSI Bus ID 7 SCSI Bus ID 7 3.5 Adding an ES45 with Dual Memory Channel Adapters to a Cluster with Memory Channel Adapters Jumpered for 512 MB If your ES45 has dual Memory Channel adapters, they must be jumpered for 128 MB because they are on the same PCI bus. Therefore, you cannot add the ES45 to an existing cluster if the Memory Channel adapters of the existing cluster member systems are jumpered for 512 MB. Installing an AlphaServer ES45 into a TruCluster Server Configuration 3–13 To add an ES45 with dual Memory Channel adapters into an existing cluster with Memory Channel adapters jumpered for 512 MB, follow this procedure: 1. Shut down all cluster member systems except one. ____________________ Notes ____________________ Do not shut down a cluster member that has an application with restricted placement policy unless the member that will not be shut down is on the hosting list for the application. See the Cluster Highly Available Applications manual for more information. Modify the quorum votes as necessary to support a one-node cluster. The Cluster Administration manual describes how quorum is calculated and how cluster membership is maintained. 2. On each cluster member that has been shut down, power off the system and move Memory Channel jumper J3 to pins 1 and 2 to select 128 MB. See the chapter about setting up a Memory Channel cluster interconnect in Cluster Hardware Configuration for more information. 3. Power up, but do not boot, each cluster member that has had the Memory Channel jumper modified. 4. Shut down the last cluster member system. 5. Boot one of the cluster members that has had its Memory Channel adapter re-jumpered for 128 MB. ____________________ Note _____________________ Readjust quorum votes and reset the placement policies as needed to maintain a one member cluster with the booted system. 6. Repeat step 2 for the last cluster member system. 7. Power on the last cluster member system and boot the remaining original cluster member systems. 8. Use the clu_add_member utility to add the ES45 system to the cluster. ____________________ Note _____________________ If you used the clu_add_member utility to add the ES45 system to the cluster before changing the Memory Channel 3–14 Installing an AlphaServer ES45 into a TruCluster Server Configuration jumpers on the other cluster member systems, the ES45 will not boot into the cluster. A system jumpered for 128 MB cannot join the cluster. You would receive a startup error message on the system jumpered for 128 MB: panic: MC2 adapter has too little memory 9. Boot the ES45 system into the cluster. ____________________ Note _____________________ Readjust quorum votes and reset the placement policies as needed. Installing an AlphaServer ES45 into a TruCluster Server Configuration 3–15