Page 1

Technical Whitepaper:

Architecture Considerations and Best Practices for Architecting an Oracle 10g R2 RAC Solution with Serviceguard and SGeRAC

Table of Content Introduction......................................................................................................................................... 3 Technologies Overview ........................................................................................................................ 3

Serviceguard and Serviceguard Extension for RAC (SGeRAC)............................................................... 3 Serviceguard ............................................................................................................................... 3 Serviceguard Extension for RAC..................................................................................................... 4

Oracle 10g R2 RAC......................................................................................................................... 5 Cluster Protocol ............................................................................................................................ 6 Networking ................................................................................................................................. 6 Storage ....................................................................................................................................... 7

Architecting an Oracle 10g R2 RAC Solution with SGeRAC ..................................................................... 8 Cluster Management – Who controls what and when? ......................................................................... 9 Network HA Configuration Choices ................................................................................................. 11

Network HA .............................................................................................................................. 12 Public Network Configuration ...................................................................................................... 13 Private Network Configuration ..................................................................................................... 14 Configuration Option 1 – Recommended Configuration....................................................................................... 15 Configuration Option 2 ................................................................................................................................... 16

Storage Management Choices......................................................................................................... 18 Serviceguard CFS....................................................................................................................... 19 Oracle Disk Manager ...................................................................................................................................... 20 Storage Checkpoints ....................................................................................................................................... 21 Storage Mapping............................................................................................................................................ 21 Best Practices when Using CFS for 10g RAC ...................................................................................................... 22

SLVM and CVM ......................................................................................................................... 23 Best Practices when Using SLVM for 10g RAC .................................................................................................... 23

ASM over SLVM......................................................................................................................... 23 Criteria in Choosing the Right Storage Management for RAC .......................................................... 24

Oracle 10g RAC with SGeRAC Integration Framework .......................................................................... 26 Why Integration Framework? .......................................................................................................... 26 Serviceguard 11.17 Multi-node Packages (MNPs) and Simple Package Dependencies .......................... 27 Framework Benefits ........................................................................................................................ 28

Page 2

Conclusion........................................................................................................................................ 29 References ........................................................................................................................................ 29

Figures

Figure 1. Serviceguard cluster .............................................................................................................. 3 Figure 2. SGeRAC cluster ..................................................................................................................... 4 Figure 3. SG and Oracle 10g R2 RAC integrated stack....................................................................... 8 Figure 4. 10g RAC HA Network ........................................................................................................ 11 Figure 5. Configuring Public network HA Option ............................................................................... 13 Figure 6. Private network configuration option 1 without CFS ............................................................ 15 Figure 7. Private network configuration option 1 with CFS................................................................. 15 Figure 8. Private network configuration option 2 without CFS ............................................................ 17 Figure 9. Private network configuration option 2 with CFS................................................................. 17 Figure 11. Oracle 10g RAC and Serviceguard CFS Alternative for Oracle 10g software ................ 22 Figure 13. Oracle Clusterware storage and database storage in SLVM/CVM .................................. 27 Figure 14. Oracle clusterware storage and database storage in Serviceguard CFS .......................... 28

Page 3

Introduction Beginning with its10g release, Oracle Real Application Cluster (RAC) will require pre-integrated clusterware from Oracle called Cluster Ready Services or CRS (also referred to as Oracle Clusterware in 10g R2). HP recommends a combination of Oracle Clusterware and Serviceguard/SGeRAC for Oracle 10g RAC solutions on HP-UX 11i. This whitepaper describes best practices for designing a combined Oracle Clusterware and Serviceguard/SGeRAC stack, and explains how Oracle Clusterware interacts with Serviceguard/SGeRAC.

Technologies Overview

Serviceguard and Serviceguard Extension for RAC (SGeRAC) Serviceguard Serviceguard is HP’s product for creating high availability clusters using a networked grouping of HP 9000 and HP Integrity servers. These servers are configured with redundant hardware and software components to eliminate single points of failure (SPOFs). Serviceguard is designed to keep application services running in spite of failures of hardware (for example System Processing Unit, disk, LAN, etc.) or software (e.g.,Operating System, user application, etc.). In the event of a hardware or software failure, Serviceguard and other high availability subsystems coordinate the transfer of functions from failed to working components. Serviceguard uses packages to group together sets of application services (e.g., individual HP-UX processes). Packages are typically configured to run on several nodes in the cluster, one at a time (Serviceguard implements a “shared nothing” cluster model). In the event of a service, node, network, or other monitored package resource failure on the node where the package is running, Serviceguard can automatically transfer control of the package to another node in the cluster, keeping the services available with minimal interruption. Figure 1 shows an example of Serviceguard cluster.

Figure 1. Serviceguard cluster

Page 4

•All systems are physically connected to each disk•Maximum cluster size is 16 nodes•Each application runs on only one host at a time•Hosts can run multiple applications•Failover is possible to any node that is physically connected to the data

App A

App B

App C

App D

P USH

READY

ALARM

MESSAGE

hp S to r a g e W o r k s x p 1 2 0 0 0 d i s k a r r a y

hp S to r a g e W o rk s xp 1 2 0 0 0 d i sk a r r a yhp S to r a g e W o r k s x p 1 2 0 0 0 d i s k a r r a y

A

B

C

D

2-nodecluster

Quorum ServiceHighly Available Quorum Devicethat is not a member of the clusterwhose quorum is being satisfied

Serviceguard Extension for RAC Serviceguard Extension for RAC (SGeRAC) is an add-on product specifically designed for running Oracle Real Application Cluster (RAC) on a Serviceguard cluster. SGeRAC provides the following features in the Oracle RAC environment:

• Manages the cluster, including the monitoring of cluster members and hardware and software components within each node

• Provides the cluster membership to RAC via a standard API defined by Oracle • Monitors and controls application packages for high availability • Detects and recovers from network interface card (NIC) failures • Provides shared storage between nodes via shared volumes and cluster file system

Figure 2 shows an example of SGeRAC cluster.

Figure 2. SGeRAC cluster

Page 5

Cache Fusion (Ethernet, Hyperfabric)

9i/10g RAC 9i/10g RAC 9i/10g RAC 9i/10g RAC

Hyperfabric is supported with 9i RAC only

Cluster heartbeat and Cache fusion can share the same Ethernet

PU SH

READY

ALARM

MESSAGE

hp S tor ageWor k s xp12000 d is k ar ray

hp S tor ageWor k s xp12000 d is k ar rayhp S tor ageWor k s xp12000 d is k ar ray

Oracle 10g R2 RAC In the 10g release, Oracle includes its own clusterware software with the database products. This clusterware in 10g R1 is known as Cluster Ready Services (CRS) and is included with Oracle 10g RAC software. CRS is required for 10g RAC databases. In 10g R2, Oracle uses the term Oracle Clusterware instead of CRS. There are three components in Oracle Clusterware: Cluster Synchronization Services (CSS), Cluster Ready Services (CRS), and Event Management (EVM). CSS manages the Oracle cluster membership and provides its own group membership service to RAC instances. CSS manages the cluster by controlling which nodes are members of the cluster and by notifying other members when a node joins or leaves the cluster. If third-party cluster software such as SGeRAC is used in the environment, CSS utilizes the group membership service provided by the third-party cluster software. CRS manages the high availability operation within a cluster. Anything CRS manages is known as a clustered resource. A clustered resource could be a database, a RAC instance, a listener, a Virtual IP (VIP) address, an application process, etc. CRS manages clustered resources using the resource’s configuration information that is stored in a shared Oracle Cluster Registry (OCR) file. This includes start, stop, monitor, and failover operations. EVM publishes events generated by CRS and may run scripts when certain events occur.

Page 6

Oracle Clusterware requires that the OCR and the voting disk be configured on shared raw devices, on shared raw volumes, or as files in a cluster file system.

Cluster Protocol CSS maintains two heartbeat mechanisms: 1) the disk heartbeat to the voting disk and 2) the network heartbeat, which is used to confirm valid node membership in the cluster. Each of these heartbeat mechanisms has a timeout value, expressed in seconds.

• The network heartbeat timeout value is known as CSS MISSCOUNT. On UNIX® platforms the default is 301 seconds.

• The default value for disk heartbeat timeout varies depending on the10g version. The default for 10.2.0.2 is to 200 seconds.

Even though Oracle Clusterware provides its own cluster membership services, Clusterware will use the node membership services provided by SGeRAC if SGeRAC is installed. If there is a network partition (meaning that nodes lose communication with each other), one or more nodes may be evicted from the cluster automatically to prevent data corruption. Depending on the type of failure, the eviction can be triggered by SGeRAC or by the Oracle Clusterware CSS daemon. See section “Cluster Management – Who controls what and when?” later in this document for more information.

Networking Three networks are required in an Oracle 10g RAC configuration: • A private network for cluster node heartbeat communication. This is referred to as the CSS

heartbeat. • A private network for the Global Cache Service (GCS) and Global Enqueue Service (GES). This

network is also known as Cache Fusion and DLM network. From here on, the term RAC interconnect will be used for this private network.

• A public network for client connections. Currently, Oracle Clusterware does not support redundant standby networks and does not monitor network interface cards (NICs). For the public network and private network high availability, Oracle relies either on platform network bonding software such as the HP-UX Auto-Port Aggregation (APA) product, or on networking features provided by third-party cluster software, such as Primary/Standby LANs in Serviceguard. Oracle Virtual IP (VIP) must be configured on the public network for each RAC instance. Clients use VIP to connect to the RAC database. If a node fails, CRS fails the VIP over to another cluster node to provide an immediate “node down” response to client’s connection request. This increases the availability of other RAC instances to the clients because the clients no longer have to wait for a network timeout before the connection request fails over to another instance in the cluster. When a RAC instance is configured on a cluster node, the VIP that is specified for that node during Oracle Clusterware installation will be used as the instance’s dependent resource. Note that all RAC instances (for different RAC databases) running in the same system depend on the same VIP configured for that system. If the VIP fails, all RAC instances that use it will also fail.

1 Based on Oracle’s Meta Link note 294430.1

Page 7

Storage In Oracle 10g, Oracle provides its own storage management with a new feature called Automatic Storage Management (ASM). ASM provides some filesystem and volume management capabilities for Oracle database files only. These include DB control files, redo logs, archived redo logs, data files, spfiles and Oracle Recovery Manager (RMAN) backup files. ASM cannot be used for Oracle executables and non-database files. ASM does not have multi-pathing capability. It assumes the underlying OS will provide this functionality. In HP-UX, multi-pathing is provided by a Volume Manager feature such as PVLinks in the HP-UX Logical Volume Manager (LVM) or DMP in Veritas Volume Manager from Symantec (VxVM), or by other third-party software such as Securepath or Powerpath. An ASM filesystem layer is implicitly created within a diskgroup. This filesystem is transparent to users and only accessible through ASM instance, interfacing databases, and ASM’s utilities. For example, database backups of ASM files can be performed only with RMAN. Since ASM supports only database data files and log files, the following storage management methods are required for non-database files if ASM is used:

• A local file system or Cluster File System for Oracle Clusterware binaries and configuration files and RAC binaries and configuration files.

• Shared storage for Oracle Clusterware data: voting disk and OCR files. This storage has to be configured either as shared raw devices, shared raw volumes, or files in the Cluster File System. Oracle Clusterware needs to be up and running before the ASM daemon can start. Therefore, ASM cannot be used for Oracle Clusterware data.

CAUTION: There is no shared awareness of LUN use between ASM and LVM or VxVM. This means tthat the system administrator must be careful not to accidentally allocate a LUN already allocated for LVM or VxVM use to ASM use (or vice-versa). See section “ASM over SLVM” later in this document for information on how to use ASM with SLVM to address this issue.

Page 8

Architecting an Oracle 10g R2 RAC Solution with SGeRAC Since the release of Oracle 10g, HP has recommended a combined Oracle Clusterware and SGeRAC configuration for RAC deployments on HP-UX. In this combined environment, the responsibilities of SGeRAC include:

• Providing cluster membership information to the Oracle Clusterware CSS daemon • Providing clustered storage to meet the needs of Oracle Clusterware and RAC database

instances o The Oracle Clusterware voting disk and OCR can be configured as shared raw

logical volumes managed by SGeRAC using Shared Logical Volume Manager (SLVM), or Cluster Volume Manager (CVM), or as shared files managed by SGeRAC using the Cluster File System (CFS). The Oracle Clusterware home directory, containing Oracle Clusterware executables, can also be stored in a CFS.

o The RAC database files can be configured as shared raw logical volumes managed by SGeRAC using SLVM or CVM. Beginning with Oracle 10g R2 and SGeRAC A.11.17, the RAC database files may also be configured as files in Oracle ASM Disk Groups. The members of the ASM Disk Groups are configured as shared raw logical volumes managed by SGeRAC using SLVM. Also, beginning with SGeRAC A.11.17, the RAC database files may be configured as shared files managed by SGeRAC using CFS. The Oracle database home directory, containing Oracle database server executables, can also be stored in a CFS.

The responsibilities of Oracle Clusterware in this combined environment include:

• Management of the database and associated resources (database instances, services, virtual IP addresses (VIPs), listeners, etc.)

• Management of Oracle ASM instances, if configured. Figure 3 shows the combined stack.

Figure 3. SG and Oracle 10g R2 RAC integrated stack

Page 9

HP supports and strongly recommends Oracle 10g RAC configurations that use:

• Oracle 10g R2 or later• If you must use 10g R1, use patch

release 10.1.0.4 or later

• SGeRAC cluster membership provides RAC membership

• HP-UX supported volume managers and file systems to store Oracle and application data.

• SLVM, CVM and CFS

• The SG package manager to provide HA for applications running on the same cluster as the RAC software

SG Clustering Components

Oracle components

10g RAC with HP ClusterMembership and Storage Mgmt

FileSystems

HPUX

SGeRAC: SG Cluster Membership

RAC

CRS

CR

Sde

finiti

on

SG

-Man

aged

App

licat

ions

SGPackageManager

SG

Inte

grat

ion

code

HP Volume ManagersSG/CFS

RawVolumes ASM

CAUTION: In the combined stack one needs to make sure that only SGeRAC or Oracle Clusterware manages the cluster membership at any given time, not both. If both pieces of software manage the cluster at the same time, there is the possibility of conflicting cluster membership that may cause the whole cluster to fail. The following sections discuss the interaction between cluster components in the combined stack, network HA configuration choices, and storage management choices.

Cluster Management – Who controls what and when? In an Oracle 10g R2 RAC with SGeRAC configuration, there are three cluster components: SGeRAC, Oracle Clusterware, and RAC. Each of these components has its own critical resources that keep the RAC cluster running. When one or more of these resources fail, the appropriate cluster component will do a recovery to maintain cluster integrity. A time delay is used to ensure only one cluster component controls the recovery at any given time. This mechanism works well but may also introduce a long recovery time if the configuration is not properly designed. The remainder of this section discusses the default time delay configuration for each component and how recovery is handled by these components when a fault occurs. When Oracle Clusterware is installed in a SGeRAC environment the CSS MISSCOUNT value is set to 600 seconds. This adjustment is to make sure Serviceguard has enough time to resolve the cluster membership before CSS decides its own cluster membership. Once the cluster reforms SGeRAC will present the new cluster membership to Oracle Clusterware via the Oracle supplied cluster membership API.

Page 10

RAC also has its own group membership. When you configure RAC, you choose all nodes or a subset of nodes in the cluster to run RAC instances for a specific RAC Database. By default, RAC Instance Membership Recovery (IMR) is set to 17 minutes. This means that when a critical RAC resource fails, RAC will wait for 17 minutes before performing its own recovery. This time delay is to ensure the underlying clustering technology (either Serviceguard or Oracle Clusterware) has enough time to perform cluster recovery before RAC handles it. Now lets look at critical resources for each component and the impact to the cluster when they fail. Serviceguard:

• Cluster heartbeat network

If all heartbeat networks fail for a cluster node, the impacted node will be evicted from the cluster via a system Transfer of Control (TOC). The time for node eviction depends on the cluster’s configurable parameter “NODE_TIMEOUT” value. The default value is 2 seconds. When the node is evicted, the remaining nodes reform the cluster and the new cluster membership is provided to Oracle Clusterware, which is then propagated to RAC as appropriate.

Oracle Clusterware:

• CSS heartbeat network If all CSS heartbeat neworks for a node fail and if Serviceguard doesn’t monitor the CSS heartbeat network, the impacted node will be evicted from the cluster by CSS via a TOC2. The time for the node eviction depends on the CSS MISSCOUNT value; the default value in a SGeRAC environment is 600 seconds. When the node TOCs, from Serviceguard’s perspective the node has crashed, and Serviceguard will reform the cluster. The new cluster membership is then provided to Oracle Clusterware via Serviceguard. • Voting disk If a node loses access to the voting disk, CSS evicts the impacted node from the cluster by causing a TOC on the node. The time for the node’s eviction depends on the disk heartbeat time out value; the default value is 200 seconds3. When the node TOCs, from Serviceguard’s perspective the node has crashed, and Serviceguard will reform the cluster. The new cluster membership is then provided to Oracle Clusterware via Serviceguard.

RAC:

• RAC interconnect network If all RAC interconnect networks fail for a node, RAC evicts the instance running on the impacted node from the group. The time for the instance eviction is dependent on the Instance Membership Recovery (IMR) value; the default value is 17 minutes. In this case, only the RAC instance is down not the node; therefore the cluster is not impacted.

As you can see, if Serviceguard detects the failure of a resource that it monitors, Serviceguard will provide the fastest cluster recovery, and the new cluster membership is propagated to Oracle Clusterware and RAC. Therefore, the design principle is to configure Oracle Clusterware and RAC critical resources to be the same as Serviceguard’s critical resources and be monitored by

2 Prior to patch 2 of release 2 (10.2.0.2), CSS uses reboot to evict the node. 3 Prior to patch 2 of release 2 (10.2.0.2), the disk I/O timeout value is the same as CSS MISSCOUNT value

Page 11

Serviceguard. For example, configure RAC interconnect and CSS heartbeat on the same network as Serviceguard heartbeat, and configure the voting disk as an EMS resource that is monitored by the HA Disk Monitor4 in Serviceguard.

Network HA Configuration Choices In a 10g RAC configuration there are three networks and a minimum of three IP addresses configured per cluster node:

• Private network for Oracle Clusterware CSS heatbeat

One IP address is required if configured on its own subnet. Oracle only supports one active sub-network for CSS heartbeat. Typically it would be configured on the same network as the RAC interconnect. There is more discussion on this in a later section.

• Private network for RAC interconnect One IP address is required if configured on its own subnet. Multiple active sub-networks can be configured for RAC interconnect, but they are only used for load balancing. The primary interface must be functioning for RAC to continue on the node.

• Public network for client connections to the RAC database This network requires two IP addresses: one station IP address is for the purpose of system administration, and one Virtual IP (VIP) address is for client connection to the RAC instance running in the system. The VIP address is required per node in 10g RAC, and is managed and monitored by Oracle Clusterware CRS component. The Oracle VIP and Serviceguard package IP (Relocatable IP) can be configured on the same Network Interface Card (NIC)5.

Since Oracle Clusterware doesn’t support redundant networks and doesn’t monitor NICs, either APA or Serviceguard Primary/Standby LANs should be used to provide network HA. Figure 4 shows an example of Servicguard Primary/Standby LANs that is used to provide network HA for public and private networks in an Oracle 10g RAC configuration.

Figure 4. 10g RAC HA Network

4 Currently, HA Disk Monitor only supports S/LVM resource 5 Required Oracle 10g R2 Patch 2 (10.2.0.2)

Page 12

Private cluster interconnect forCSS, GCS/GES (a.k.a. Cache Fusion or DLM)

Clients

PublicNetwork

CRS IP1 CRS IP2

Stationary IP3

Oracle VIP1

Stationary IP4

Oracle VIP2

Oracle VIP1

The remainder of this section discusses how to properly configure these networks for high availability as well as for fast recovery from network failure. First, lets look at methods available on HP-UX in configuring a highly available network. Network HA In general either APA6 or Serviceguard Primary/Standby LANs can be used to provide network HA for Oracle 10g RAC public and private networks in a 10g RAC with SGeRAC environment. The preferred method is Serviceguard Primary/Standby LANs for the following reasons:

• It has a much more robust network monitoring capability. The network sensor in SGeRAC is implemented as a clustered component. It has the ability to poll the status of a standby NIC in the local node as well as to use the remote NIC to poll status of the NICs in the local node. Therefore, it can pro-actively detect that a standby NIC is down before it is actually used.

• Allows one NIC to be a standby for multiple NICs which can reduce the total IO slots required relative to APA.

• Supports the failover to different NIC type in the same family, like 100B-T Ethernet failover to 1 Gb Ethernet.

• Monitors all links and can detect the loss of all links and take failover action.

HP-UX APA supports 3 configuration modes: Active/Standby, Active/Active, and LAN Monitor.

6 If Serviceguard CFS is used, APA is not supported for CFS interconnect; Serviceguard Primary/Standby LANs is required to provide network HA

Page 13

APA Active/Standby mode is supported in 10g RAC with SGeRAC configuration but has the following disadvantages:

• Only monitors the health of the standby NIC in the same system

• Requires one standby NIC per one active NIC

• Only supports the failover of the same NIC type

• Doesn’t monitor the loss of all links; they must be detected (after a timeout period) by the application.

• In some failure cases APA failover from active to standby LAN can take up to 30 seconds.

APA Active/Active mode provides the ability to combine the speed of multiple NICs into one single pipe (called a trunk) for larger bandwidth. This mode requires all APA links connected to the same network switch. This mode is not recommended due to the network switch being a Single Point of Failure (SPOF) I this configuration. If APA Active/Active mode is required for higher bandwidth it should be configured with multiple trunks in a primary/standby configuration in conjunction with Serviceguard Primary/Standby LANs feature for HA.

APA LAN Monitor mode is not supported and is disabled in a Serviceguard or SGeRAC environment.

Public Network Configuration Oracle only supports one active public network for client connections. Use APA or Servicerguard Local LAN failover to configure network HA for the public network. This is shown in the Figure 5.

Figure 5. Configuring Public network HA Option

Page 14

SG Local LAN Failover Configuration

Stationary IP 1 Stationary IP 2

VIP 1 VIP 2

Clients

NS NS

APA Active/Standby Configuration

Stationary IP 1 Stationary IP 2VIP 1 VIP 2

Clients

NS NS

Client access

Network

VIP: Oracle Virtual IP

NS: Network Switch

APA: Auto-Port Aggregation

The preferred method is Serviceguard local LAN failover. Even though Serviceguard local LAN failover works very fast (about 12 seconds or less), it may take up to 60 seconds for CRS to detect a local LAN failover so it can move the VIP to the standby LAN. The 60 seconds is CRS’s default polling interval on the VIP. This value can be configured with a smaller value, recommended by Oracle, when 10g RAC is running in SGeRAC environment.

Best practice for RAC public network HA is to use Serviceguard local LAN failover and do the following: • Configure VIP with a redundant network managed by Serviceguard7 • Configure the VIP polling interval to a smaller value than 60 seconds8 for faster VIP

local and remote failover.

Private Network Configuration There are a number of choices available when configuring the private network for the combined Oracle 10g RAC and SGeRAC stack. This section discusses two different options and the supported HA configuration for each option.

7 See Meta Link note # 296874.1 for instruction on how to configure Oracle VIP with redundant network managed by Serviceguard 8 See Oracle Metalink note # 294336.1 for instruction on how to modify the VIP monitoring interval.

Page 15

Configuration Option 1 – Recommended Configuration In this configuration, Serviceguard heartbeat, CFS interconnect (if CFS is used), Oracle Clusterware CSS heartbeat, and RAC interconnect are configured on the same network. Since Serviceguard’s heartbeat is sent over this network, Serviceguard will monitor this network. If a node’s connection to this network is down, Serviceguard will evict the impacted node from the cluster after the NODE_TIMEOUT value expires.

This is the recommended configuration because it provides the fastest cluster and RAC recovery if all private networks for a node fail.

Since APA is not supported for CFS interconnect, APA cannot be used in a CFS configuration.

Figure 6 shows the supported HA configurations for common private networks for Serviceguard cluster heartbeat, Oracle Clusterware CSS heartbeat, and RAC interconnect in a non-CFS environment. In this configuration either APA Active/Standby or Serviceguard Local LAN failover is supported.

Figure 6. Private network configuration option 1 without CFS

RECOMMENDED Configuration2 HA choices for common dedicated network for:

• Serviceguard heartbeat• Oracle Clusterware CSS heartbeat• RAC GCS/GES (Cache fusion/DLM)

SG Local LAN Failover Configuration

IP 1 IP 2

NS NS

APA Active/Standby Configuration

IP 1 IP 2

NS NS

NS: Network Switch

APA: Auto-Port Aggregation

Figure 7 shows the HA network configuration for common private network for Serviceguard cluster heartbeat, Oracle Clusterware CSS heartbeat, and RAC interconnect in a CFS environment. In this configuration only Serviceguard Local LAN failover is supported.

Figure 7. Private network configuration option 1 with CFS

Page 16

RECOMMENDED ConfigurationCommon dedicated network for:

• Serviceguard heartbeat• CFS interconnect• Oracle Clusterware CSS heartbeat• RAC GCS/GES (Cache fusion/DLM)IP 1

IP 2

NS NS

SG Local LAN Failover Configuration

NS: Network Switch

For fastest cluster recovery: • Use a common dedicated HA network for Serviceguard heartbeat, CFS interconnect (if

CFS is used), CSS heartbeat, and RAC interconnect • Use Serviceguard Local LAN failover for network HA • Configure the Serviceguard heartbeat on the same network as the CSS heartbeat, CSS

only supports one active network and Serviceguard heartbeat will fail and cause the node to be evicted if the network used by CSS fails. If you do not do this it will take the CSS timeout of 600 seconds before node failure and recovery will occur.

Configuration Option 2 When one common network is used for all private traffic, heavy RAC interconnect traffic may interfere with cluster heartbeat traffic and may cause heartbeat packets to be dropped . This can lead to a node being evicted from the cluster. Therefore, for this configuration, the RAC interconnect network can be configured on its own dedicated network separate from the cluster heartbeat network. In this configuration, to ensure a fast RAC group membership recovery in case a node loses all RAC interconnect networks, configure the RAC subnet to be monitored by Serviceguard. Otherwise, it will take up to 17 minutes for RAC to recognize the failure and do a recovery. Figure 8 shows the supported HA configurations for the two private networks in a non-CFS environment.

Either APA Active/Standby or Serviceguard local LAN failover can be configured to provide network HA.

Page 17

In addition, if the RAC interconnect traffic requires higher bandwidth, APA trunking can be configured to combine the bandwidth of multiple NICs in primary/standby configuration along with SG local LAN failover for HA.

Figure 8. Private network configuration option 2 without CFS

2 HA choices for common dedicated network for:• Serviceguard heartbeat• Oracle Clusterware CSS heartbeat

3 HA choices for dedicated network for:• RAC GCS/GES (Cache fusion/DLM)

NS NS

IP 3 IP 4

IP 1 IP 2

NS NS

NS NS

IP 3 IP 4

IP 1 IP 2

NS NSOR

SG Local LAN Failover Configuration

APA Active/Standby Configuration

NS: Network Switch

APA: Auto-Port Aggregation

IP 3 IP 4

APA trunking and SG Local

LAN failover Configuration

NS NS

Figure 9 shows the supported HA configurations for two private networks in a CFS environment.

APA is not supported for the CFS interconnect. Therefore, only SG local LAN failover can be used for the common private network used for Serviceguard heartbeat, Oracle clusterware CSS heartbeat, and CFS interconnect because they all share a common network.

For the RAC GCS/GES network, either APA Active/Standby or Serviceguard local LAN failover can be configured to provide network HA. If the RAC GCS/GES traffic requires additional bandwidth, APA trunking can be configured to combine the bandwidth of mutlitple NICs along with SG local LAN failover for HA.

Figure 9. Private network configuration option 2 with CFS

Page 18

One HA choice for dedicated network for:• Serviceguard heartbeat• CFS Interconnect• Oracle Clusterware CSS heartbeat

3 HA choices for dedicated network for:• RAC GCS/GES (Cache fusion/DLM)NS: Network Switch

APA: Auto-Port Aggregation

IP 1 IP 2

NS NS

NS NS

IP 3 IP 4

NS NS

IP 3 IP 4 IP 3 IP 4

SG Local LAN Failover Configuration APA trunking and SG Local

LAN failover Configuration

APA Active/Standby Configuration

NS NS

If the RAC interconnect needs to be configured on its own dedicated network: • Configure a common dedicated HA network for SG heartbeat, CFS interconnect (if

CFS is used), and CSS heartbeat • Configure the RAC interconnect on its own dedicated HA network, and configure

Serviceguard to monitor the RAC interconnect network for faster RAC group membership recovery.

• Use Serviceguard local LAN failover for network HA • Configure the Serviceguard heartbeat on the same network as the CSS heartbeat, CSS

only supports one active network and Serviceguard heartbeat will fail and cause the node to be evicted if the network used by CSS fails. If you do not do this it will take the CSS timeout of 600 seconds before node failure and recovery will occur.

Storage Management Choices Beginning with Serviceguard A.11.17, Serviceguard/SGeRAC supports the following storage management options for Oracle 10g R2 RAC:

• Serviceguard Cluster File System (CFS)

• Shared Logical Volume Manager (SLVM)

• Veritas Cluster Volume Manager (CVM)

• ASM with SLVM

Page 19

Both SLVM and CVM have been available for quite some time. The Serviceguard CFS storage option and ASM with SLVM option are available starting with Serviceguard A.11.17.

Table 1 shows the storage management choices for various data used in a RAC cluster. The Oracle binaries are for both Oracle Clusterware software and RAC software.

Table 1. Storage management options for RAC cluster

YesYesYesYesCluster File System (requires SGeRAC)

Yes**NoNoYesLocal File System (on each node)

No

No

No

Oracle Binaries

YesYesNoOracle ASM (10g only)

NoYesYesShared Raw Disk devices as presented to host (no SGeRAC)

NoYesYesShared Raw Logical Volumes –SLVM or CVM (requires SGeRAC)

Archive Log Files

Oracle Data Files

Ora10g Cluster Files (*)

Storage Option

(*) Oracle Cluster files = Voting Disk and Oracle Cluster Repository (OCR)(**) only if deployed on separate VG/DG which is failed over

As shown in Table 1, CFS is the only storage option that can be used for all kinds of data used in a RAC cluster. CFS also can be used for non-RAC applications running in the same cluster.

The following section provides further details on each of the storage management options.

Serviceguard CFS Prior to SGeRAC A.11.17, when deploying RAC on HP-UX, the user has no choice but to use shared raw volumes using SLVM or CVM for RAC database data. Beginning with Serviceguard/SGeRAC 11.17, Serviceguard CFS is also supported. While Oracle Clusterware OCR and Voting Disk may be placed on a raw device, Serviceguard Cluster File System (CFS) is preferred for better manageability and reliability. If a raw device is used, it must be accessible by all nodes in the cluster simultaneously. This can be accomplished by directly accessing a physical disk connected to all systems but this device has no protection from failure. However, devices under CFS have the inherited mirroring and multipathing features from CVM to increase device resiliency should any individual disk fail, or if the need arises to relocate or resize either component. Additionally, CFS greatly simplifies management of file growth by directly reporting currently allocated space in use for each file, while also reducing the number of devices to manage: the OCR file, and Voting Disk may both reside in the same CFS mount. CFS also simplifies backup and recovery in a RAC environment. Without CFS, the archive logs of each instance are stored in a local file system on the node where the instance is running.

Page 20

• Performing a backup of the database requires access to archive logs of all instances. The common method is to use NFS where the node running a backup job would do an NFS mount of all instances’ archive logs file system.

• When a node crashes or an instance is down and its logs are needed for recovery, a manual mount of the down instance’s archive logs is required.

By storing all instances’ archive logs on CFS, all nodes have instant access to each other’s archive logs. This eliminates the need to do NFS mounts or manual intervention for backup and recovery. The management of backup and recovery is greatly simplified. Some other advantages of using CFS for RAC include:

• Addition/deletion of files • Easy growth of Oracle data files • Use of standard file system backup utilities • Consistent storage management for

o Oracle RAC o Application(s) o Other shared files

• Provides support for installation of Oracle Clusterware binaries, and RAC binaries. Oracle 10g RAC is compatible with the cluster file system with all configuration file names being node/instance specific. Installing CRS software and RAC software on CFS reduces the amount of storage space needed in comparison to installing on a local file system of each node in the cluster.9

In addition to the above advantages, the following key features are included in the Serviceguard Storage Management Suite for RAC to further enhance the performance and manageability of Oracle RAC environment:

• Oracle Disk Manager • Storage Checkpoints • Storage Mapping

Oracle Disk Manager One of the major key features provided with Serviceguard CFS for RAC is the database accelerator Oracle Disk Manager or ODM. ODM is a standard API specified by Oracle for data base I/O. The VERITAS implementation of ODM improves performance by providing direct access for the database to the underlying storage without passing through the actual file system interface. The resulting performance is equivalent to performance when using raw devices. The performance improvements are based on the following:

• File System Direct I/O With this feature, Oracle disk I/O bypasses the HP-UX buffer cache to provide the following benefits:

o Improved system efficiency—Oracle uses its own buffer, System Global Area (SGA), to cache disk I/O. It has better knowledge than the HP-UX kernel about

9 For customers who need to do a rolling upgrades of Oracle software, they should install Oracle Clusterware binaries and RAC binaries in an individual file system on each node.

Page 21

which Oracle data objects must be buffered. Hence, it is more efficient to devote memory to the SGA than to the HP-UX buffer cache for Oracle databases.

o Improved I/O performance—Without File System Direct I/O, each Oracle I/O would be buffered twice: once in the SGA and then in the operating system buffer cache. In addition to using the memory in an inefficient way, this double buffering also consumes additional CPU resources (an additional memory to memory system call is required). Using File System Direct I/O avoids double buffering and its negative influence on I/O performance.

• Asynchronous File System I/O (/dev/async) Normally, direct I/O is synchronous, which can significantly downgrade write performance. For example, an application would block on a write I/O until the write completes. With Asynchronous File System I/O, the direct I/O is asynchronous in the same way as when the database is deployed on raw volumes.

For Oracle RAC on CFS, the ODM feature provides near-raw volume performance with the manageability of a file system. Storage Checkpoints Another key feature of Serviceguard CFS for RAC is Storage Checkpoints. Storage Checkpoints enable efficient backup and recovery of an Oracle database. The Storage Checkpoint facility is similar to the snapshot file system mechanism however a Storage Checkpoint persists after a system restart. A Storage Checkpoint creates an exact image of a database instantly and provides a consistent image of the database from the time the Storage Checkpoint is created. Like a snapshot file system, a Storage Checkpoint appears as an exact image of the snapped file system at the time the Storage Checkpoint is made. However, unlike a snapshot file system that uses separate disk space, all Storage Checkpoints share the same free space pool where the primary file system resides. A Storage Checkpoint can be mounted as read-only or read-write, allowing access to files as if it were a regular file system. VERITAS NetBackup also makes use of Storage Checkpoints to provide a very efficient Oracle backup mechanism. A direct application of the Storage Checkpoint facility is Storage Rollback. Because each Storage Checkpoint is a consistent, point-in-time image of a file system, Storage Rollback is the restore facility for these on-disk backups. Storage rollback rolls back blocks contained in a Storage Checkpoint into the primary file system for faster database recovery. Storage rollback restores a database, a tablespace or datafiles in the primary file system to the point-in-time image created during a Storage Checkpoint. Storage rollback is accomplished by copying the “before” images from the appropriate Storage Checkpoint back to the primary file system. As with Storage Checkpoints, Storage Rollback restores at the block level, rather than at the file level. Storage Mapping The Storage Mapping feature enables the mapping of data files to physical devices. Access to mapping information allows for a detailed understanding of the storage hierarchy in which files reside. This information is critical to effectively evaluating I/O performance. The VERITAS Storage Mapping option supports the Oracle set of storage APIs called Oracle Mapping (ORAMAP) that lets Oracle determine the mapping information for files and devices. Oracle provides a set of dynamic performance views (v$ views) that show the complete mapping of a file to intermediate layers of logical volumes and physical devices. These views enable you to

Page 22

locate the exact disk on which any specific block of a file resides. Use these mappings, along with device statistics, to evaluate I/O performance. The mapping can be viewed either via Oracle Enterprise Manage GUI or via Veritas Enterprise Administration GUI. Best Practices when Using CFS for 10g RAC

In 10g RAC environment, CFS can be used as storage for all kind of files: • Oracle Clusterware software and configuration files. This is referred as ORA_CRS_HOME • Oracle Clusterware data files: Voting disk and Oracle Cluster Registry (OCR) • Oracle RAC software and configuration files. This is referred as ORACLE_HOME • Oracle RAC database data files, control files, and Archive log files

However, if rolling upgrade of Oracle software is needed, use local file systems for ORA_CRS_HOME and ORACLE_HOME, as shown in the following figure 11.

Figure 10. Oracle 10g RAC and Serviceguard CFS Alternative for Oracle 10g software

Server 1

CFSshared storage

CFS/CVM

DB Accelerator (ODM)

Oracle RACinstance 1

SGeRAC

Serviceguard

CFS/CVM

DB Accelerator (ODM)

Oracle RACinstance 2

SGeRAC

Serviceguard

/cfs-redo logs

/cfs-crs data

/cfs-control files/cfs-data files

/cfs-archive redo logs

Server 2

/ORACLE_HOME (binaries, etc)

/ORA_CRS_HOME

/ORACLE_HOME

/ORA_CRS_HOME

Oracle 10g RAC supports the use of CFS for Oracle Clusterware software and Oracle RAC software. When the software is installed on CFS, node specific directories are created to hold log and trace files that are generated from Oracle Clusterware processes and RAC instance running on each node.

Page 23

CAUTION: All the log and trace files from all nodes in the cluster are stored in a shared CFS mount point; therefore, the file system can fill up quickly. When a file system is full; especially, the one that is used for ORA_CRS_HOME, Oracle Clusterware will not be able to read the file that contains CSS daemon PID, and will either hang or crash the node. So it’s important to periodically delete old log files, trace files, and core files in the following directories10 to free up disk space:

• $ORA_CRS_HOME/log/<nodename>/crsd • $ORA_CRS_HOME/log/<nodename>/cssd • $ORA_CRS_HOME/log/<nodename>/evmd • $ORA_CRS_HOME/log/<nodename>/racg

SLVM and CVM Prior to Serviceguard 11.17, SLVM and CVM were the only two storage management choices for RAC environments on HP-UX 11i. These two storage management choices will continue to be offered as another storage management option for Oracle 10g RAC with SGeRAC.

The current version of SLVM was enhanced to provide an online volume group reconfiguration feature known as Single Node Online Reconfiguration (SNOR). Before SNOR is available, reconfiguration of a volume group such as adding a physical device to increase the volume group size would require taking all RAC instances offline. SNOR improves online configuration by allowing one instance to be online accessing the data while the volume group holding the database data is reconfigured.

The key advantage of using SLVM and CVM with RAC is the I/O performance that raw volumes provide. The disadvantage is that it’s hard to manage because there is no filesystem. For example, you cannot use the file listing command like “ls” to see the database files listing; you cannot use the “bdf” command to see how much space has been used.

Best Practices when Using SLVM for 10g RAC An LVM logical volume (LV) by default has an infinite I/O timeout which would cause the CSS daemon to hang if the node has no access to the storage that houses either the Oracle Clusterware software or Oracle Clusterware voting disk. To avoid the hang problem, the LVs used for these components should be configured with an I/O timeout value. The LV’s time out value can be changed using the “lvchange” command. The recommended value is:

LV timeout value = (# of paths * PV Timeout) + 10 seconds Note that the PV timeout value can be changed, so make sure to use the configured value instead of the default PV timeout of 30 seconds. ASM over SLVM Starting with SGeRAC A.11.17, ASM over SLVM is also available as another storage option for 10g RAC. The advantages of using SLVM as an underlying volume manager for ASM are:

• An ASM with SLVM configuration provides multi-pathing for ASM disk groups by using LVM PV Links.

10 As observed in 10g R2. These directories may be changed in later version of Oracle.

Page 24

• ASM with SLVM enables the HP-UX devices used for disk group members to have the same names on all nodes easing ASM configuration.

• Protects ASM data against inadvertent overwrites from nodes inside or outside the cluster. If the ASM disk group members are raw disks, there is no protection from them being incorporated into LVM volume groups or VxVM disk groups.

Criteria in Choosing the Right Storage Management for RAC There are always pros and cons with each storage management option. The following list provides some criteria one should consider when making the choices:

• Ease of administration and management o CFS is best

o Provides consistent storage management for all components

• Initial and latent costs of software licenses

o SLVM/CVM raw volumes and ASM are the least expensive

• Desired performance and availability of the solution

o Raw volumes provide the best performance

• Hardware configurations

• Compatibility with existing applications

Table 2 shows a high level comparison between the solutions.

Table 2. Storage management options comparison

Page 25

NoNoYesUse of filesystemmanagement tool (like ls, cp, etc)

Good for managing Oracle data

GoodBestEase of management

NoNoYesUse of all normal backup utilities

Only Oracle db dataYesYesDatabase and non-db application data

Up to 8 (with SG framework)

Up to 16 (w/o SG framework)

Up to 16 (SLVM)Up to 8 (CVM)

Up to 4 (initial release)

Up to 8 (future)

Number of nodes supported

10g RAC9i RAC, 10g RAC9i RAC, 10g RACDatabase version compatibility

Yes

SLVM or CVM

Integration with DTS solutions

Yes

CFS

No

ASM on SLVM

Page 26

Oracle 10g RAC with SGeRAC Integration Framework

Why Integration Framework11? When Oracle 10g RAC is configured in a SGeRAC environment, the piecies of the combined stack start up and shut down in the proper sequence and the startup and shutdown sequences can be automated, if desired. In particular, the storage needed for the operation of Oracle Clusterware must be started before the Oracle Clusterware processes are started. Likewise the storage needed for the operation of a RAC database instance has to be started before the RAC database instance is started. On shutdown, the sequence is reversed, Oracle Clusterware and the RAC database instance must be terminated before shutting down the storage needed by these two entities. Traditionally, in the SGeRAC environment, these ordering requirements have been met using a package to encapsulate the startup and shutdown of an application as well as the startup and shutdown of storage needed by that application. In SGeRAC A.11.17, a different model is introduced for the case where the storage needs of an application are met by using a CFS. Here the CFS is started up and shutdown in a separate package from the one that starts up and shuts down the application. The ordering requirement is met by using the SGeRAC A.11.17 feature of simple package dependencies, discussed later in this document. Begin with Oracle 10.1.0.4, Oracle made improvements in coordination between Oracle Clusterware and third party clusterware, enabling the use of SGeRAC packages to manage the storage needs of Oracle Clusterware and RAC database instances:

• Support for on-demand startup and shutdown of Oracle Clusterware12 and RAC database Instances13

In addition to starting up and shutting down Oracle Clusterware automatically as HP-UX is taken up to init level 3 and taken down to a lower level respectively, we can start up and shutdown Oracle Clusterware on demand.

In addition to starting up and shutting down the RAC database instance automatically as Oracle Clusterware itself is started up and shut down, we can start up and shut down the RAC database instance on demand.

• Support for invocation of Oracle 10g Clusterware commands from customer-developed

scripts This includes invocation of such commands from SGeRAC package control scripts; hence SGeRAC packages can invoke commands to start up and shutdown Oracle Clusterware and/or RAC database instances.

These improvements makes it possible to use SGeRAC packages to properly sequence the startup and shutdown of Oracle Clusterware and RAC database instances with respect to the SGeRAC-managed storage used by these entities.

11 A SGeRAC toolkit for this integration framework will be available in December 2006 12 See Oracle Metalink note # 332257.1 for instruction on how to disable CRS to start up at system boot time. 13 See Oracle Metalink note # 298073.1 for instruction on how to disable CRS to automatically start up RAC instance

Page 27

Serviceguard 11.17 Multi-node Packages (MNPs) and Simple Package Dependencies As mentioned above, Oracle’s improvements on version 10.1.0.4 and later, Serviceguard packages can be used to manage the storage for Oracle Clusterware and RAC database instances. The best practice is to have separate packages for Oracle Clusterware and each RAC database. To ensure that these packages start and halt in the proper order, the combination of Serviceguard 11.17 MNPs and simple package dependencies coordinate Oracle 10g RAC and SGeRAC. The startup and shutdown of Oracle Clusterware is managed by one MNP. The startup and shutdown of the RAC database is managed by another MNP. The simple package dependency is used to define the database MNPs to depend on the Oracle Clusterware MNP. This is the core concept of the framework. The use of MNPs reduces the total package count and simplifies SGeRAC package configuration and administration. Figure 13 illustrates the use of the framework with SLVM/CVM as a storage management option for 10g RAC solution. In this configuration, the CRS MNP manages the shared storage for the voting disk and OCR, startup and shutdown CRS, and monitors cssd deamon14. The RAC instance MNP manages the shared storage for the database, startup and shutdown RAC instance, and monitor RAC instance15. The RAC instance MNP has a dependency on CRS MNP. This means, the RAC instance MNP can only start after CRS MNP is up and running. And, CRS MNP can only stop after the RAC instance MNP has stopped.

Figure 11. Oracle Clusterware storage and database storage in SLVM/CVM

Figure 14 shows the use of the framework with CFS as a management storage option for 10g RAC. In this configuration, the CRS MNP and the RAC instance MNP have dependencies on the corresponding CFS mount point MNPs. CRS MNP and RAC instance MNPs are configured the same as in the case of SLVM/CVM. 14 Due to Oracle’s restriction, if the monitor detects cssd process is down, it will trigger the CRS MNP package to stop, and let Oracle Clusterware do the appropriate clean up. 15 Same as foot note 12.

CRS MNP RAC Instance MNP

Page 28

Figure 12. Oracle clusterware storage and database storage in Serviceguard CFS

CRS MNPRAC Instance MNP

CFS Mount Point MNP for CRS

CFS Mount Point MNP for db

CFS/CVM 4.1 SMNP

CVM Disk Group MNP CVM Disk Group MNP

Framework Benefits The Oracle 10g RAC integration framework based on SGeRAC 11.17 Multi-Node Packages and Simple Package Dependencies provides a uniform, intuitive and easy-to-manage method to

• co-ordinate between SGeRAC and Oracle Clusterware • manage all the storage options supported by SGeRAC. CFS, SLVM, CVM and ASM (over

SLVM) The Oracle 10g RAC framework is a significant added value of SGeRAC. Although the concepts of resource dependency and resource aggregation, represented by SGeRAC 11.17 Multi-Node Package and Simple Package Dependencies, are present in other clusterware products, including Oracle Clusterware, this SGeRAC framework for Oracle 10g RAC is unique in the high level of multi-vendor (Oracle, Symantec, HP) and multi-storage platform (CFS, SLVM, CVM, ASM over SLVM) integration it offers.

Page 29

Conclusion Highlights of high availability features provided by SGeRAC for 10g RAC:

• Online node addition and replacement. • Advanced I/O fencing mechanism provides data integrity protection among nodes both

inside and outside of the cluster. • I/O fencing for a single instance DB or for the application itself (Oracle Clusterware does

not provide this). • Secure multiple paths to the data storage array (Oracle relies on the O/S vendor for this). • Protecting the Oracle virtual IP address (VIP) provides improved monitoring and protection

of all network interfaces including the cluster interconnect and IPv6 network (Oracle relies on the O/S vendor for this high availability protection).

• Enhanced faster failover time of the cluster than Oracle Cluster Services. • Support for rolling upgrades of HP-UX and Serviceguard. • Provides additional HA for 3rd party applications running on the same cluster as the RAC

software. • File system support for all application-related files including Oracle executables,

configuration files and error logs, any application-related files that are not part of the database itself such as executables, error log files, configuration files, flat files, etc.

• Activate/deactivate or mount/unmount filesystems without custom scripting. • Deploy cluster file system and/or raw volumes. For the CFS there is Oracle Disk Manager

(ODM) that provides near raw volume performance with manageability of a file system. Although combining Oracle Clusterware with Serviceguard/SGeRAC means deploying more cluster components than Oracle Clusterware alone, it provides a more robust HA environment and gives the RAC cluster administrator more storage-management choices. The Oracle 10g RAC with SGeRAC integration framework provides an intuitive and easy way to implement and manage the combined stack. Hewlett-Packard believes the unique capabilities of its Serviceguard suite, and the robustness it provides, best meet the high-availability and disaster-tolerance requirements of a large majority of Oracle 10g database deployments on HP-UX. This architecture is integrated with Hewlett-Packard’s Virtual Server Environment (VSE) solutions, and provides access to a wide range of Disaster Tolerant Solutions on HP-UX 11i.

References HP Documentation http://docs.hp.com/en/ha.html

• Managing Serviceguard, 12th Edition, March 2006 http://docs.hp.com/en/B3936-90100/B3936-90100.pdf

• Using Serviceguard Extension for RAC, May 2006 http://docs.hp.com/en/T1859-90038/T1859-90038.pdf

• Sample Configurations with SGeRAC and Oracle RAC, July 2006 http://docs.hp.com/en/8365/SGeRAC_10g_Sample_Config.pdf

Page 30

• VERITAS Storage Foundation™ 4.1 for Oracle RAC HP Serviceguard Storage Management Suite Extracts, December 2005 http://docs.hp.com/en/7412/T2771-90010.pdf

Oracle Documentation for 10g R2 http://www.oracle.com/technology/documentation/database10gr2.html • Oracle Clusterware and Oracle Real Application Clusters Installation Guide, 10g Release 2

(10.2) for HP-UX http://download-west.oracle.com/docs/cd/B19306_01/rac.102/b14202.pdf

• Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide, 10g Release 2 (10.2) http://download-west.oracle.com/docs/cd/B19306_01/rac.102/b14197.pdf