HP ProLiant DL980 Universal Database Solution: HP Serviceguard

Technical white paper

HP ProLiant DL980 Universal Database Solution HP Serviceguard for Linux and 3PAR StoreServ for Oracle Enterprise Database

Table of contents Executive summary ...................................................................................................................................................................... 2

Overview .......................................................................................................................................................................................... 3

HP Serviceguard for Linux ....................................................................................................................................................... 3

HP ProLiant DL980 G7 server................................................................................................................................................. 4

HP 3PAR StoreServ 10000 storage array ............................................................................................................................ 4

HP ProLiant DL980 Universal Database Solution .................................................................................................................. 5

Building the solution ..................................................................................................................................................................... 8

Software and hardware components ................................................................................................................................... 8

Hardware setup ......................................................................................................................................................................... 8

Install Red Hat Enterprise Linux Server .............................................................................................................................. 10

Install Oracle GRID Infrastructure ........................................................................................................................................ 11

Install Oracle Enterprise Database ...................................................................................................................................... 11

Install HP Serviceguard for Linux......................................................................................................................................... 11

Set up HP Serviceguard for Linux cluster .......................................................................................................................... 13

Set up HP Serviceguard for Linux ASM multinode package .......................................................................................... 13

Set up HP Serviceguard for Linux PROD failover package ............................................................................................. 13

Set up HP Serviceguard for Linux TEST failover package .............................................................................................. 13

High-Availability testing ............................................................................................................................................................. 14

Test description ....................................................................................................................................................................... 14

Test results ............................................................................................................................................................................... 15

Conclusion ..................................................................................................................................................................................... 16

Appendix A: Red Hat Enterprise Linux Server 6.2 installation ........................................................................................... 17

Appendix B: Oracle GRID Infrastructure 11gR2 installation ............................................................................................... 22

Appendix C: Oracle Enterprise Database Edition 11gR2 installation ............................................................................... 26

Appendix D: HP Serviceguard for Linux A.11.20 installation ............................................................................................. 29

Appendix E: HP Serviceguard for Linux cluster setup ......................................................................................................... 30

Appendix F: HP Serviceguard for Linux ASM package setup .............................................................................................. 32

Appendix G: HP Serviceguard for Linux PROD package setup ........................................................................................... 34

Appendix H: HP Serviceguard for Linux TEST package setup ............................................................................................ 37

Appendix I: Bill of Materials ....................................................................................................................................................... 39

For more information ................................................................................................................................................................. 41

Technical white paper | HP ProLiant DL980 Universal Database Solution

2

Executive summary

The HP ProLiant DL980 Universal Database Solution is HP’s large-scale, mission-critical, x86 database solution. Providing preconfigured, tested solutions with a choice of operating systems, storage, and databases. It delivers advanced reliability, availability, and serviceability (RAS) capabilities, enabling the high-end scalability and availability traditionally associated with RISC and mainframe systems at a fraction of the cost.

HP’s newest ProLiant DL980 Universal Database Solution for Oracle Databases combines high-performance HP ProLiant DL980 G7 server technology, HP 3PAR StoreServ 10000 storage technology and HP Serviceguard for Linux in a mission-critical solution designed specifically for large transactional (OLTP) and data warehousing (DSS) workloads. Additional information can be found on the HP ProLiant DL980 Universal Database Solution website. When compared to similar highly-available configurations, this solution:

• Lowers TCO by reducing power consumption, cooling, and space while improving performance.

• Reduce storage capacity as much as 50%.1

• Boosts system availability by as much as 200%.2

• Provides extreme performance for large-scale applications3 and virtualization environments.

The heart of this solution is the powerful HP ProLiant DL980 G7 server, the industry’s most popular, highly reliable, and comprehensively scalable eight-socket x86 server offering. The DL980 G7 server leverages HP’s years of experience designing mission-critical servers in RISC, EPIC, and UNIX® environments. This design capitalizes on over a hundred innovative availability features to deliver a portfolio of resilient and highly reliable scale-up x86 servers in a compact 8U rack form factor. Additional information can be found on the HP ProLiant DL980 G7 website. Some of the key scalability and reliability features of the DL980 G7 server are:

• 4 or 8 Intel® Xeon® Processors with up to 10 cores per central processing unit (CPU)

• From 128GB to 4TB of memory

• Up to 8 internal hot-swappable drives with built in hardware RAID

• Redundant system links to provide resilient data paths

Storage for this solution is delivered with the HP 3PAR StoreServ 10000 Series Tier 1 Storage Array designed to deliver enterprise IT as a utility service simply, efficiently, and flexibly. The arrays feature a tightly coupled clustered architecture, secure multi-tenancy, and mixed workload support for enterprise-class data centers. Additional information can be found on the HP 3PAR StoreServ website. Several key 3PAR StoreServ features are:

• Multi-tenant capabilities for managing unpredictable workloads.

• Save up to 90% of administrator time.4

• Silicon-based engine providing on-the-fly storage optimization.

• Full Mesh active-active cluster design for robust performance and high-availability.

The HP Serviceguard for Linux is the high availability clustering software used in this solution and is designed to protect applications and services from planned and unplanned downtime. The HP Serviceguard Solutions for Linux portfolio also includes numerous implementation toolkits that enable you to easily integrate various databases and open source applications into a Serviceguard cluster and three distinct disaster recovery options. Additional information can be found on the HP Serviceguard for Linux website. Some of the key features of HP Serviceguard for Linux are:

• Robust monitoring to protection against system, software, network and storage faults

• Advance cluster arbitration and fencing mechanisms to prevent data corruption or loss

• GUI and CLI management interfaces

• Quick and accurate cluster package creation

1 Requires the use of HP 3PAR Thin Conversion Software and HP 3PAR Thin Provisioning Software. For details, refer to the Get Thin Guarantee Terms and Conditions. More information is available at: hp.com/storage/getthin.

2 Based on system crash rates’ comparison between the ProLiant DL980 G7 servers to the ProLiant DL785 G5 servers. System crash rate is determined by availability features such as hot-swap components, resilient paths, ECC, and tolerant links such as QPI.

3 Based on the TPC-H price performance per query results for non-clustered systems at 3000 GB. 4 Based on HP Storage customer results, available from HP Storage Customer success stories.

http://www.hp.com/go/udb

http://www.hp.com/go/dl980

http://www.hp.com/go/3PAR

http://www.hp.com/go/sglx

http://www.hp.com/storage/getthin


3

These components, when combined with Oracle Database Enterprise Edition software, create a multi-node solution which protects against system and database instance failures by eliminating the single points of failure.

Many customers today desire to have highly performing, highly available Oracle database solutions but do not want the high cost or the inflexibility of “all-Oracle-stack” solutions. Indeed, our high-performance Oracle Database solution is much simpler and more flexible than competing Oracle solutions, and because it is based on open standards, the HP solution:

• Supports various database software versions, operating systems and patch levels.

• Allows you to update, expand or extend your server environment as your needs change.

• Gives you the choice of support and service offerings.

HP is a leading platform for Oracle solutions and continues to invest heavily with Oracle focused resources worldwide to provide cost effective open and standards-based architecture alternatives using converged infrastructure. Customer performance workload characteristics and requirements vary and HP has solutions tailored to provide maximum performance for each specific workload without compromising on required availability commitments to the business.

Target audience: This HP white paper was written for IT professionals who use, program, manage, or administer Oracle

databases that require high availability and specifically for those who design, evaluate, or recommend new IT high performance architectures.

This white paper describes testing performed between November 2012 and February 2013.

Overview

This paper will provide the details of the HP ProLiant DL980 Universal Database Solution for Oracle Enterprise Database Edition 11gR2. We will focus primarily on the configuration, build, and high-availability test results to aid in designing and deploying mission-critical, high performance database solutions for Oracle databases.

A brief overview of the major HP components used in this solution:

• HP Serviceguard for Linux

• HP ProLiant DL980 G7 server

• HP 3PAR StoreServ 10000 storage array

An overview of the HP ProLiant DL980 Universal Database Solution with details on building the solution:

• HP ProLiant DL980 Universal Database Solution

• Building the solution

A description of the high-availability testing performed, the database and workload used during that testing, results of the tests, and the final conclusion:

• Test description

• Database and workload defined

• Test results

• Conclusion

HP Serviceguard for Linux

HP Serviceguard for Linux is high-availability clustering software that manages the systems, software, network, and storage for a mission-critical solution. The environment is continuously monitored for any faults and when a failure or threshold violation is detected, Serviceguard for Linux can automatically and transparently resume normal operations in mere seconds without compromising data integrity and performance.

HP Serviceguard Solutions for Linux are designed to address the financial, business and operational impacts of planned and unplanned downtime in demanding mission critical Linux environments. With HP Serviceguard Solutions for Linux, you can define, implement, and manage solutions that help to secure your infrastructure, protect data and information, and align business continuity objectives to business requirements. See Table 1 for features and benefits of HP Serviceguard for Linux.



4

Table 1. HP Serviceguard for Linux features and benefits

Features Benefits

High data integrity HP offers Quorum Service, a robust fencing mechanism that allows Serviceguard for Linux to ensure the highest level of data integrity and reliability.

Zero downtime maintenance

A key feature called “Live Application Detach (LAD)” allows you to perform maintenance on the Serviceguard cluster infrastructure (including the heartbeat network) with zero application downtime.5

Shortened cluster setup

The Toolkits and Extensions portfolio offers predefined scripts that allow up to 93% of the setup time it takes to get an application protected.6

Disaster recovery capability

HP Serviceguard Disaster Recovery Solutions for Linux enable you to remain online even after the loss of a data center, regardless of the distance.

Simplified cluster management

HP Serviceguard Manager for Linux allows users to visually configure, monitor, manage and administer a cluster and its components. HP Serviceguard Manager is also integrated with HP Systems Insight Manager (HP SIM) to enable management of multiple clusters running on Linux or HP-UX from a single browser.

5 Based on HP Lab analysis while performing maintenance activities of the cluster including maintenance of the heartbeat network, it used

to be a common case where the application had to be brought down. However, with the LAD feature it is noticed that the application downtime has been reduced to none even if the heartbeat network is maintained and the cluster is brought down.

6 Based on HP Lab analysis that show typical manual effort for integrating Oracle Database into a cluster to require 30 engineering days. With Oracle toolkit from HP, this integration is achieved in two engineering days or less.

HP ProLiant DL980 G7 server

HP ProLiant DL980 G7 with HP PREMA Architecture is an eight socket server built to handle the largest x86 enterprise environments. Ideal choice for large scale databases with extreme workloads such as I/O intensive online transaction processing (OLTP) or compute intensive decision support system (DSS). Database consolidation or server virtualization is also an ideal fit for the DL980’s scale-up design.

The DL980 G7 blends industry standard economies with advanced mission critical capabilities to deliver balanced scaling, self-healing resiliency and breakthrough efficiencies essential for enterprise compute environments. Operating costs are reduced with less overall maintenance, lower energy use, reduced cooling, and floor space requirements. The HP PREMA Architecture features Smart CPU caching and a resilient system fabric to reduce bottlenecks, improve throughput and performance, as well as deliver enhanced reliability not previously available in an x86 environment

System management is provided by HP Integrated Lights-Out Management (iLO) and simplifies server setup, remote system management and integrates with HP Systems Insight Manager (SIM) and HP Insight Control.

HP 3PAR StoreServ 10000 storage array

The HP 3PAR StoreServ 10000 storage arrays are designed to deliver enterprise IT storage as a utility service simply, efficiently, and flexibly. The arrays feature a tightly coupled clustered architecture, secure multi-tenancy, and mixed workload support for enterprise-class data centers. Use of unique thin technologies reduces acquisition and operational costs by up to 50% while autonomic management features improve administrative efficiency by up to tenfold when compared with traditional storage solutions. The HP 3PAR StoreServ Gen4 ASIC in each of the system’s controller nodes provides a hyper-efficient, silicon-based engine that drives on-the-fly storage optimization to maximize capacity utilization while delivering high service levels.

Increase business agility

• Autonomic service level optimization through the use of optional HP 3PAR Dynamic Optimization (DO) and Adaptive Optimization (AO) software delivers the agility to react quickly to changing application and infrastructure requirements without the need for active management.

• Optional HP 3PAR Virtual Domains software, the first and only storage hypervisor-like technology on the market to deliver customized, secure, and “self-service” storage to multiple internal or external customers.

Reduce Total Cost of Ownership (TCO)

• Advanced internal virtualization, wide striping, and mixed workload support reduce physical capacity purchases, storage footprint, power usage, and cooling needs without compromising performance.

• The two HP 3PAR StoreServ Gen4 ASICs with Thin Built In inside every controller node have the ability to drive silicon-based thin conversion to reduce legacy storage capacity requirements and reclaim allocated-but-unused capacity.

http://h20195.www2.hp.com/V2/GetDocument.aspx?docname=4AA4-4108ENW




http://h20195.www2.hp.com/v2/GetDocument.aspx?docname=4AA3-2542ENW



5

• Optional HP 3PAR Thin Provisioning software eliminates capacity waste by allowing clients to purchase only the disk capacity they actually need, only as they actually need it for written data.

• Fast RAID 5 boosts RAID 5 performance to within 10% of RAID 1 but with significantly less capacity overhead; Fast RAID 6 (RAID MP) delivers enhanced protection while maintaining performance levels within 15% of RAID 10 and with capacity overheads comparable to popular RAID 5.

HP ProLiant DL980 Universal Database Solution

The development of the HP ProLiant DL980 Universal Database solution with Serviceguard for Linux focused on the building and testing of a large-scale, highly available Oracle Database Enterprise Edition 11gR2 for Red Hat Enterprise Linux 6.2 x86-64 environments. The large-scale platform consisted of HP ProLiant DL980 G7 servers and an HP 3PAR StoreServ 10800 storage array. High-availability was provided by HP Serviceguard for Linux, redundant 8Gb FC links for SAN connectivity, and two redundant 10GbE links for both private and public networks. To take advantage of the resources of the standby node, steps were included to configure an optional secondary database.

High-availability testing focused on HP Serviceguard’ s ability to monitor the health of the primary database instance and manage the failover of the instance to a standby node in the event of an active node failure. Failover tests were performed with the database under a heavy OLTP load and in an idle state.

The environment was isolated within a private network with access only through an HP ProLiant DL380 G7 gateway access server running Microsoft® Windows® Server 2008 R2. This server also provided the management console, DNS and NTP services for the environment. A second DL380 G7 installed with Red Hat Enterprise Linux Server 6.2 was configured as the client load generator and housed the database workload software.

The networks were comprised of two redundant 10GbE links for public and private communications and a single 1GbE link for system management. The redundant network links were set up in an active-passive mode using the Red Hat Linux NIC bonding driver. The public network was dedicated for client load and the private network was dedicated for Serviceguard cluster communications. The management LAN connected the server’s iLO system management ports and the management ports for the switches supporting the environment.

The Storage Area Network (SAN) comprised of two 8Gb fibre channel paths between the DL980 servers, fibre channel switches, and the 3PAR StoreServ 10800 array. Zoning was configured on the SAN switches to limit the server to storage fibre channel connections to two paths. The redundant FC links were set up using the Linux multipath driver in an active-active mode providing both redundancy and load balancing for the servers. See Figure 1 for the SAN and LAN diagram.


6

Figure 1. HP Universal Database solution LAN and SAN diagram

The HP ProLiant DL980 G7 servers were installed with Red Hat Enterprise Linux version 6.2 x86-64 and configured as HP Serviceguard cluster nodes. Both servers were equipped with eight Intel Xeon E7-4870 (2.4GHz/10-core/30MB/130W) processors, 1TB of memory, two dual port 8Gb FC HBAs, two dual port 10GbE NICs, and four internal 300GB disk drives configured as two RAID1 sets for the “boot disk” and the “/apps” directory.

The HP 3PAR StoreServ 10800 storage array provided the shared SAN storage for our solution. A 3PAR management domain was set up with 2TB of storage for the solution. Two host profiles were created using the World Wide Names (WWNs) of the server’s HBAs and the 3PAR controller/ports. A Serviceguard Lock LUN (quorum disk) and the Oracle Automatic Storage Management (ASM) LUNs were set up and presented to both host profiles. See Table 2 for the 3PAR LUN configuration used in this solution.

Table 2. 3PAR LUN configuration

LUN ID Size RAID type Description

1 256MB RAID5 Serviceguard Cluster Lock LUN

2, 3, 4, and 5 100GB RAID5 Primary Database (PROD) Data ASM LUNs

6, 7, 8, and 9 25GB RAID1 Primary Database (PROD) Log ASM LUNs

10, 11, 12, and 13 110GB RAID5 Secondary Database (TEST) Data ASM LUNs

14, 15, 16, and 17 30GB RAID1 Secondary Database (TEST) Log ASM LUNs


7

HP Serviceguard for Linux software was used to create the cluster and the HP Serviceguard Oracle Toolkit for Linux was used to create the three cluster packages for the solution. The ASM multi-node package was the first package created and ran on both nodes to manage the Oracle Automatic Storage Management (ASM) instances within the cluster. The primary PROD database instance failover package was then created to manage the PROD database instance in the cluster. The secondary TEST database instance failover package was created last to manage the TEST database instance in the cluster.

Both the PROD and the TEST packages were configured with dependence on the ASM package and to run exclusively on a node ensuring that only one database instance will be active on a single node at any given time. The PROD package was also configured with a higher priority than the TEST package thus ensuring that Serviceguard for Linux will halt the TEST package in favor of the PROD package failover. Figure 2 shows a fully operational Serviceguard cluster with Node1 running the PROD database and Node2 running the TEST database. Figure 3 shows what happens to Node2 when Node1 fails.

Figure 2. Fully operational cluster

Figure 3. Results of a node 1 failure


8

The Oracle Grid Infrastructure 11gR2 software was installed on both DL980 servers to provide the Oracle ASM services to manage the ASM volumes for both PROD and TEST database instances. The Oracle Enterprise Database 11gR2 software was installed in two separate Oracle Homes on each DL980 server, one for the PROD database, and one for the TEST database.

The database workload software was used to build the custom databases and to generate the OLTP load in order to stress the system while performing failover testing.

Building the solution

This section provides the steps to build the environment for the HP ProLiant DL980 Universal Database solution. This information can greatly reduce the time to plan and deploy your own large scale, highly available Oracle database solution for Red Hat Enterprise Linux environment.

In this solution the build sequence follows these steps:

• Software and hardware components

• Hardware setup

• Install Red Hat Enterprise Linux

• Install Oracle Grid Infrastructure

• Install the Oracle Enterprise Database

• Install HP Serviceguard for Linux

• Set up the HP Serviceguard for Linux cluster

• Set up the HP Serviceguard for Linux ASM multinode package

• Set up the HP Serviceguard for Linux PROD failover package

• Set up the HP Serviceguard for Linux TEST failover package

Software and hardware components

Acquire all the software, licenses, and hardware required to build the solution. A list of the HP equipment used in this solution is provided in the Appendix I: Bill of Materials.

Software

• Red Hat Enterprise Linux version 6 Update 2 64-bit

• HP Serviceguard for Linux 11.20

• HP Serviceguard Oracle Toolkit for Linux A.05.01.00

• Oracle Grid Infrastructure 11gR2

• Oracle Enterprise Database 11gR2 (11.2.0.3)

Hardware • Two HP ProLiant DL980 G7 servers

– 8x Intel Xeon E7-4870 (2.4GHz/10-core/30MB/130W) processors

– 1TB of memory

– 2x HP 82Q 8Gb Dual Port PCI-e FC HBAs

– 2x HP NC552SFP dual port 10GbE NICs

– 4x internal 300GB disk drives

• HP 3PAR StoreServ 10800

• Two HP 5800-24G 1/10GbE switches configured with redundant IRF 10Gb interconnects

• Two HP 8Gb SN6000 FC switches

Hardware setup

There are several options for setting up the hardware; HP factory built, HP Services sets up equipment at the customer site, or customer builds. See Figure 4 for the LAN and SAN wiring diagram for network and fibre channel connectivity used in the solution.


9

Figure 4. LAN and SAN wiring diagram

SAN configurations

The SAN consists of two HP SN6000 8/24 FC switches and an HP 3PAR StoreServ 10800 storage array. Redundant FC paths were configured for each DL980 G7 server to provide high-availability for shared storage I/O. Two dual port 8Gb FC HBAs were installed in each server but only one port per HBA was used in this configuration.

Each FC switch was configured with two single initiator (host WWNs) to target (3PAR controller-port WWNs) zones. The zones provided two single paths per server. This topology provides for complete redundancy across both servers, both FC switches, and both 3PAR controllers. See Table 3 for an overview of the FC zoning configuration. For additional information see the HP 3PAR StoreServ Storage best practices guide.

Table 3. FC switch zoning

FC switch Zone name Configuration

1 Node1Path1 World Wide Names (WWN):

DL980-1 HBA1 Port1

3PAR Controller 1 Port1


DL980-2 HBA1 Port1


http://h20195.www2.hp.com/V2/GetDocument.aspx?docname=/4AA4-4524ENW


10

FC switch Zone name Configuration


DL980-1 HBA2 Port1



DL980-2 HBA2 Port1


The 3PAR Management Console was used to configure the host profiles and shared LUNs for the Serviceguard lock LUN and the ASM disks. Each host file was configured using the WWNs for their HBAs and the 3PAR controller ports. The LUNs were created and presented to both hosts. See Table 4 for the 3PAR storage configuration.

Table 4. 3PAR SAN storage configuration

Host profile Controller #/Port # Host WWNs LUN ID #/LUN name/RAID type

DL980_1 1/1

2/1

DL980-1 HBA1 Port1

DL980-1 HBA2 Port1

1/Prod_lock/256MB/RAID5

2/Prod_data_ASM0000/100GB/RAID5




6/Prod_log_ASM0000/25GB/RAID1




10/Test_data_ASM0000/110GB/RAID5




14/Test_log_ASM0000/30GB/RAID1




DL980_2 1/1

2/1

DL980-2 HBA1 Port1

DL980-2 HBA2 Port1

Same as above

Install Red Hat Enterprise Linux Server

Red Had Enterprise Linux Server 6.2 was installed using iLO remote console. The installation ISO image was mounted as a virtual drive. When the server was started it treated the ISO file as a bootable DVD and automatically started the installation process. During the installation the 3PAR LUNs were specified for multipathing which eliminated the need to set up multipath after the installation. This was possible because the zoning for the FC switches and the SAN storage was previously setup. See Appendix A: Red Hat Enterprise Linux Server 6.2 installation for steps used in this solution to install the Linux operating system.

Post-installation steps

After the operating system was successfully installed the following post-installation steps were performed:

• Stage the installation media

• Modify the kernel parameters and set the ulimits

• Create the Oracle user

• Install the 10GbE driver

• Set up NIC bonding

• Set up FC multipath


11

Best practices for setting up DL980 G7 and Red Hat for Oracle

Below are generally used best practices for availability and performance on Oracle 11gR2 implementations.

Recommended BIOS settings • Enable/disable Hyper-Threading setting under System Options Processor Options for performance testing of your

applications. Increased performance is dependent on the individual workload types.

• Disable Intel Virtualization technology and Intel VT-d settings under System Options Processor Options.

• Select “Maximum Performance” setting under Power Management Options HP Power Profile

• Select “No Package State” setting under Power Management Options Advanced Power Management Options Minimum Processor Idle Power Core State. This prevents processors from powering down components when idle.

Recommended hardware settings • Distribute FC HBA and 10GbE NIC cards evenly across both I/O cages using the x8 designated PCI-x slots

• I/O slot 1 is a first generation PCI-x slot and should not be used for any FC HBA cards

• Install DIMMs evenly across all memory banks for optimum memory interleaving performance

Red Hat kernel parameters • Set the kernel parameters as recommended in the installation manual for Oracle Grid Infrastructure and Enterprise

database.

Install Oracle GRID Infrastructure

Oracle Automatic Storage Management (ASM) 11gR2 is now part of Oracle Grid Infrastructure 11gR2. ASM provides volume and file system management for Oracle databases. A single ASM instance supports one or more database instances. ASM is organized into logical disk groups, which is a collection of ASM disks. The database files are evenly striped across all the ASM disks presented to the disk group for performance. Oracle ASM disk groups can be configured using normal or external redundancy to determine how the data will be striped across ASM disks within a disk group. Normal redundancy will stripe and mirror everything (SAME) to protect against a single disk loss and is typically used for storage without hardware RAID capabilities. External redundancy will stripe data across the ASM disks and is typically used for storage arrays that provide their own hardware RAID capabilities. Oracle ASM operates in an instance configuration much the same as a database instance does. For more information regarding Oracle ASM, see the Oracle 11gR2 Database Concepts Guide.

ASM configuration

HP 3PAR storage has its own hardware RAID capabilities and ASM disk groups were configured with external redundancy. It is a best practice to create an ASM disk group with at least four ASM disks to improve I/O performance. For additional information please consult the Oracle Automatic Storage Management Administrator’s Guide.

In this solution four ASM disk groups were created using four ASM disks each. Two ASM disk groups for the PROD and the TEST database instances. Each database instance included two ASM disk groups one for data and the other for log files. See Appendix B: Oracle GRID Infrastructure 11gR2 installation for steps used in this solution to install Oracle Grid 11gR2.

Install Oracle Enterprise Database

Oracle Enterprise Database Edition 11gR2 (11.2.0.3) is the minimum supported version for Linux 6 and above. Install the Oracle Enterprise Database Edition software twice on each server creating ORACLE_HOMEs for the primary database PROD and the secondary database TEST. See Appendix C: Oracle Enterprise Database Edition 11gR2 installation for steps used in this solution to install the Oracle databases.

Install HP Serviceguard for Linux

This section describes the process for installing HP Serviceguard for Linux and the HP Serviceguard Toolkit for Oracle databases used in this solution. Before installing HP Serviceguard for Linux one must be have read and fully understand the HP Serviceguard for Linux Version A.11.20.00 Deployment guide. See Appendix D: HP Serviceguard for Linux A.11.20 installation for steps used in this solution to install the HP Serviceguard for Linux and the HP Serviceguard Oracle Toolkit.

Pre-installation steps

Prior to installing Serviceguard for Linux make sure that the following tasks have been completed:

• Networks defined

• JDK 7 installed per recommendation for HP Serviceguard Manager for Linux

• Supported Linux packages installed

http://docs.oracle.com/cd/E11882_01/server.112/e25789.pdf

http://docs.oracle.com/cd/E11882_01/server.112/e18951.pdf

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c03393068/c03393068.pdf


12

Networks defined Prior to configuring the Serviceguard cluster the networks should be defined. See Table 5 for the Serviceguard network configuration used in this solution.

Table 5. Serviceguard network configuration

Node IP address Type Description

DL980-1 192.168.1.201 Static Public network for client access

DL980-1 10.1.1.201 Static Private network for Serviceguard cluster communications

DL980-2 192.168.1.202 Static Public network for client access

DL980-2 10.1.1.202 Static Private network for Serviceguard cluster communications

Any 192.168.1.203 Virtual Serviceguard for Linux floating IP address for the PROD Database package

Any 192.168.1.204 Virtual Serviceguard for Linux floating IP address for the TEST Database package

Install the required Linux packages on both servers

Serviceguard for Linux depends on the following package:

• xinetd

Serviceguard for Linux SNMP subagent requires the following packages:

• lm_sensors

• net-snmp

Serviceguard Manager for Linux requires the following package:

• libXp

Serviceguard CIM provider requires the following package:

• tog-pegasus

To verify if a package is installed run the following command:

# rpm –qa | fgrep <package>

If not installed, locate the package on the DVD under the PACKAGE directory and install it:

# rpm –ivh <package><version>.rpm


13

Set up HP Serviceguard for Linux cluster

After the HP Serviceguard for Linux and the HP Serviceguard Oracle Toolkit have been successfully installed the next step is to set up the cluster. Two options are available to create the cluster; Serviceguard for Linux command line interface (CLI) or the Serviceguard for Linux Manager a graphical user interface (GUI). See Appendix E: HP Serviceguard for Linux cluster setup for steps used in this solution to set up the Serviceguard for Linux cluster.

In this solution the following items were specified during the setup of the cluster:

Cluster name: Prod_cluster

Cluster nodes: dl980-1, dl980-2

Subnet: 10.1.1.0

Type: Heartbeat

Subnet Configuration:

Node Network Address

Dl980-1 bond0 10.1.1.201

Dl980-2 bond0 10.1.1.202

Cluster Lock Type Lock LUN

Lock LUN Path DL980-1 /dev/mapper/mpathcp1, DL980-2 /dev/mapper/mpathcp1

Set up HP Serviceguard for Linux ASM multinode package

Serviceguard for Linux multinode packages are designed to run on one or more nodes at a time and do not failover. They simply start or stop the application on the node or the cluster. The ASM instances fit this model perfectly and were configured using the Serviceguard Oracle Toolkit for Linux multinode script to build the configuration file for the ASM packages.

The ASM instances were configured to startup without mounting any disk groups. The ASM multinode package will be set up to run on both nodes and manage all the disk groups. When a database package is started or halted it will mount or dismount the ASM disk groups associated with that particular database instance. See Appendix F: HP Serviceguard for Linux ASM package setup for steps used in this solution to building the ASM multinode package.

Set up HP Serviceguard for Linux PROD failover package

Serviceguard for Linux failover packages are designed to run on one node at a time and when a node failure occurs the package is moved over to a surviving node within the cluster. The PROD package configuration file was created using the Serviceguard Oracle Toolkit for Linux. This failover package will need to be customized in order to monitor and manage the PROD database instance. The PROD package was configured to have a dependency on the ASM instance, run exclusively to ensure that only one database instance can run on a single node at a given time, and have a higher priority than the TEST database package to ensure that Serviceguard for Linux will shut down the TEST instance prior to starting the PROD instance in the event of a failover. See Appendix G: HP Serviceguard for Linux PROD package setup for steps used in this solution to building the PROD failover package.

Set up HP Serviceguard for Linux TEST failover package

The TEST package configuration file was created using the Serviceguard Oracle Toolkit for Linux. This failover package will need to be customized in order to monitor and manage the TEST database instance. The TEST package was configured to have a dependency on the ASM instance, run exclusively to ensure that only one database instance can run on a single node at a given time, and have a lower priority then the PROD database package to ensure that Serviceguard for Linux will shut down the TEST instance prior to starting the PROD instance in the event of a failover. See Appendix H: HP Serviceguard for Linux TEST package setup for steps used in this solution to building the TEST failover package.


14

High-Availability testing

Test description

Our testing of the HP ProLiant DL980 Universal Database Solution focused on the downtime experienced by the database during a failover. Hours of testing were conducted while certain failure scenarios were administered to get an indication of the different failover times; tests were also conducted when the database was in an idle and loaded state. After each failover test the workload was restarted to verify that the database instance properly failed over to the other node.

The following failure scenarios were tested:

• Halting the active node using Serviceguard Manager for Linux halt node command

• Soft reboot of the active node using the Linux reboot command

• Hard Reset of the active node using the iLO system reset command

The following database states were also tested with each failover scenario:

• Active node database idle/Standby node only ASM running

• Active node database running a workload/Standby node only ASM running

• Active node database running a workload/Standby node with secondary database idle

• Active node database running a workload/Standby node with secondary database running a workload

Database and workload defined

The database workload software was used to build the databases and to generate a workload to stress the systems for the failover testing. In our solution we configured the lab software in a client-server mode with the server side components installed on DL980-1 and the client side components installed on a designated client load generator server. With the PROD database running we rebuilt PROD with a 100GB OLTP database separating the data files and the logs using the PROD_DATA and PROD_LOG ASM disk groups. The PROD database was shut down and the TEST database was started before rebuilding the TEST database with the same parameters.

The workload generated 1000 OLTP users for a duration of 10 minutes. This could have easily been 5000 OLTP users for a longer time but a shorter test was much more efficient and had similar results. There was very little tuning since our focus was on the database failover. The only turning was to accommodate the large number of users needed to generate a substantial workload to test failover scenarios of an Oracle database instance with Serviceguard for Linux.

The following system information was captured during the 1000 OLTP user workload using SAR and IOSTAT.

The average CPU utilization was around 60% during the run; 14% for system and 45% for user. The workload was fairly even across all 80 cores. The memory consumed grew from 77GB and leveled off after 5 minutes to 190GB.

Figure 5. CPU utilization and memory consumed


15

The network utilization was heavy for the public network with writes averaging 4.2MB/s and reads averaging 1.3MB/s. The private network utilization was very low with read and writes averaging around 190B/s.

Figure 6. Private and public networks

The I/O for the data disk group averaged 156MB/s for writes and 48MB/s for reads. The log disk group averaged 19MB/s for writes and 6MB/s for reads.

Figure 7. Data disk and log disk groups

Test results

All the tests have been verified to protect the Oracle ASM instance and the Oracle database before, during, and after a failover. Some of the specific recommendations that are a result of the extensive availability testing are:

• The FC SAN environment should be configured by zoning a single initiator to a single target.

• The udev rules file needs to be created since ASMLib is not available for Linux 6.

• The ASM PFILE needs to be created and move to the ASM $ORACLE_HOME/dbs directory and it should be configured to not mount any disk groups.

• The Private network used for cluster communication utilizes less than 0.001% of the available 10GbE bandwidth. A redundant pair of 1GbE links could fully support the Private network at a lower cost.

See Table 6 for a list of all the failover tests performed and their results. Outage results or downtime will vary depending on database size and recovery scenarios.

Table 6. Failover test results

Active node Action performed

Primary DB status

Standby node/ secondary DB status

Outage results mm:ss

Comments

Node Halted Primary Database Idle

ASM only 01:00 Serviceguard Manager for Linux was used to halt the active node

Node Halted Primary Database Idle

Secondary Database idle

01:18 Additional time for shutting down the 2nd Database


16

Active node Action performed

Primary DB status

Standby node/ secondary DB status

Outage results mm:ss

Comments

Node Halted 1K user workload

ASM only 02:39 Additional time for database recovery





1K user workload 03:14 Additional time for shutting down the 2nd Database during an active load

Operating System Soft Reboot

Primary Database Idle

ASM only 01:01 Operating System soft reboot command


1K user workload



1K user workload




1K user workload

1K user workload 03:05 Additional time for shutting down the 2nd Database during an active load

Server Hard Reset Primary Database Idle

ASM only 01:12 Server hard reset issued through iLO

Server Hard Reset 1K user workload






1K user workload 02:44 Database recovery time will vary

Conclusion

This HP ProLiant DL980 Universal Database Solution is an important part of the overall HP large-scale and highly-available reference architecture portfolio. Developed to provide high performance for I/O intensive transactional databases and compute intensive Decision Support Systems. This delivers business continuity for heavy loads, faster user response times, and increased throughput over traditional configurations. This solution delivers this at less than half the acquisition cost, takes up less data center footprint, and consumes one-quarter the power and cooling when compared to configurations delivering similar performance.

Key success factors in our extensive testing include:

• Successfully set up a high-availability configuration for an Oracle database environment using HP Serviceguard for Linux, HP DL980 G7 servers, HP 3PAR StoreServ 10000 storage, and HP LAN and SAN switches

• Defined the proper setup and validation procedures

• Conducted failover testing and recorded the outage time for the database

• Determined business continuance after each test

The HP ProLiant DL980 Universal Database Solution defined in this document was based on the HP ProLiant DL980 G7 server, HP 3PAR StoreServ 10800 storage array, HP Serviceguard for Linux, Red Hat Enterprise Linux, and Oracle Enterprise Database. Steps for building the solution were also included to reduce the time to deploy a similar configuration. The HP ProLiant DL980 Universal Database Solution can also be flexible, supporting different operating systems, storage arrays, and databases. This document can be used as a “steering guide” to assist in creating a configuration much like the one constructed by HP’s Oracle engineering team to evaluate and validate this solution.


17

Appendix A: Red Hat Enterprise Linux Server 6.2 installation

Install Red Hat Enterprise Linux Server 6.2

Red Had Enterprise Linux Server 6.2 was installed using iLO remote console. The installation ISO image was mounted as a virtual drive. When the server was started it treated the ISO file as a bootable DVD and automatically started the installation process. During the installation the 3PAR LUNs were specified for multipathing which eliminated the need to set up multipath after the installation. This was possible because the zoning for the FC switches and the SAN storage was previously set up. In the final installation steps the following server components were specified:

• Base system

– Compatibility Libraries

– Networking Tools

• Servers

– System admin tools

• Web services

– Web server

– Web servlet engine

• Desktop

– Desktop

– Graphical admin tools

– X Windows system

• Development

– Additional development

– Development tools

Post-installation steps

After the operating system was successfully installed the following post installation steps were performed:

• Stage the installation media

• Modify the kernel parameters and set the ulimits

• Create the Oracle user

• Install the 10GbE driver

• Set up NIC bonding

• Set up FC multipath

Download and stage the installation media

Both servers

• 10GbE device driver

• Java Development Kit (JDK) 7

• Oracle Grid Infrastructure 11gR2

• Oracle Enterprise Database 11gR2 (11.2.0.3)

Primary server

• HP Serviceguard for Linux 11.20

• HP Serviceguard Oracle Toolkit for Linux A.05.01.00


18

Modify the kernel parameters and set the ulimits

Edit the sysctl.conf file to support up to 5000 user connections.

# vi /etc/sysctl.conf

kernel.shmmni = 4096

kernel.msgmni = 64

kernel.sem = 2048 6291456 2048 3072

net.ipv4.ip_local_port_range = 9000 65500

net.core.rmem_default = 262144

net.core.rmem_max = 4194304

net.core.wmem_default = 262144

net.core.wmem_max = 1048576

fs.file-max = 6815744

fs.aio-max-nr = 1048576

Edit the rc.local file and add the following command to retain the settings after a system restart.

# vi /etc/rc.local

sysctl –p

Edit the limits.conf file to set the user limits.

# vi /etc/security/limits.conf

* hard nofile 65536

* soft nofile 4096

* hard nproc 16384

* soft nproc 2047

Edit the 90-nproc.conf file to increase the maximum number of nproc.

# vi /etc/security/limit.d/90-nproc.conf

* soft nproc 8191

Create the Oracle user

The Oracle user is the owner of the ASM, PROD, and TEST instances and the “/apps” file system.

# groupadd -g 600 dba

# useradd -u 601 -g dba -m oracle

# passwd oracle

# chown –R oracle:dba /apps

Install 10GbE device driver

Red Hat Enterprise Linux Server 6.2 operating system did not recognize the 10GbE NIC hardware and device drivers were not installed during the initial installation. Download the HP 10GbE driver for Linux on both systems and read the installation instructions before attempting to install the device driver.

Setup NIC bonding

NIC bonding is defined as grouping two or more network cards to form a single connection to achieve link redundancy for network high-availability. The public and private networks were configured with NIC bonding utilizing 10GbE links in an active-passive mode. The public network was set up for client access and the private network was set up for cluster communications.

Each DL980 G7 server came standard with four 1GbE embedded network interfaces eth0 through eth3. Two dual port 10GbE network cards were installed and provided interfaces for eth4 and eth5 for one NIC and eth6 and eth7 for the other.

Create the NIC bonding configuration files for the private network (10.1.1.0) and the public network (192.168.1.0) on both servers. When using dual-port network cards, NIC bonding should combine ports from different NICs to remove a single point of failure. It is recommended to make backup copies before modifying the network interfaces for eth4 through eth7 to utilize the NIC bonding devices.

http://www8.hp.com/us/en/support-drivers.html


19

Server1 For the private network:

# vi /etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE="bond0"

IPADDR="10.1.1.201"

NETMASK="255.255.255.0"

NETWORK="10.1.1.0"

BROADCAST="10.1.1.255"

ONBOOT="yes"

BOOTPROTO="none"

USERCTL="no"

BONDING_OPTS="miimon=100 mode=1"

For the public network:

# vi /etc/sysconfig/network-scripts/ifcfg-bond1

DEVICE="bond1"

IPADDR="192.168.1.201"

NETMASK="255.255.255.0"

NETWORK="192.168.1.0"

BROADCAST="192.168.1.255"

ONBOOT="yes"

BOOTPROTO="none"

USERCTL="no"

BONDING_OPTS="miimon=100 mode=1"

Server2 Identical to the private network for Server1 except for the IP address:

IPADDR="10.1.1.202"

Identical to the public network for Server1 except for the IP address:

IPADDR="192.168.1.202"

Both servers

Modify the network config files. The HWADDR or MAC addresses are uniquely assigned for each interface:

# vi /etc/sysconfig/network-scripts/ifcfg-eth4

DEVICE="eth4"

USERCTL="no"

ONBOOT="yes"

MASTER="bond0"

SLAVE="yes"

BOOTPROTO="none"


DEVICE="eth5"

USERCTL="no"

ONBOOT="yes"

MASTER="bond1"

SLAVE="yes"

BOOTPROTO="none"


DEVICE="eth6"

USERCTL="no"

ONBOOT="yes"

MASTER="bond0"

SLAVE="yes"

BOOTPROTO="none"


DEVICE="eth7"

USERCTL="no"

ONBOOT="yes"


20

MASTER="bond1"

SLAVE="yes"

BOOTPROTO="none"

Create the bonding conf file to map network device driver to network configuration file:

# vi /etc/modprobe.d/bonding.conf

alias eth0 netxen_nic

alias eth4 be2net

alias eth5 be2net

alias eth6 be2net

alias eth7 be2net

alias bond0 bonding

alias bond1 bonding

After making all the necessary modifications reboot the systems and verify the network bonding driver with the following commands:

# cat /proc/net/bonding/bond0

# cat /proc/net/bonding/bond1

Set up FC multipath

Device Mapper configuration

Device Mapper is a key component for a high-availability SAN configuration. Device Mapper Multipath enables a server to route disk I/O over multiple FC paths to provide redundant links to the shared SAN storage. Device Mapper runs a service called multipathd and is responsible for configuring and managing the devices by specifying a single multipath device to perform link redundancy. By default, the disk I/Os are also load balanced across both paths to improve bandwidth.

The infrastructure initially was provided with a base installation of Red Hat 6.2. The solution requires several Linux server components to be installed. Instead of installing the optional server components manually it would be more efficient to re-install the operating system at this time. Prior to the re-install, the WWNs (Worldwide Names) for the FC HBAs were identified and the FC switch zoning and 3PAR storage was configured. This provided a way to select the SAN storage and set up multipathing during the re-installation phase. See Table 7 for the device mapper configuration used in this solution. For manually configuring Multipath, please refer to the Red Hat Enterprise Linux 6 DM Multipath configuration guide.

Table 7. Device Mapper configuration

Device Mapper name Multipath disk files Description

/dev/mapper/mapthc sdc

sdd

Serviceguard Lock Disk

256MB / RAID1

/dev/mapper/mpathe sdg

sdh

PROD_DATA_ASM0000

100GB / RAID 5

/dev/mapper/mpathf sdi

sdj

PROD_DATA_ASM0001

100GB / RAID 5

/dev/mapper/mpathg sdk

sdl

PROD_DATA_ASM0002

100GB / RAID 5

/dev/mapper/mpathh sdn

sdm

PROD_DATA_ASM0003

100GB / RAID 5

/dev/mapper/mpathi sdo

sdp

PROD_LOG_ASM0000

25GB / RAID 1

/dev/mapper/mpathj sdq

sdr

PROD_LOG_ASM0001

25GB / RAID 1

/dev/mapper/mpathk sds

sdt

PROD_LOG_ASM0002

25GB / RAID 1

https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/pdf/DM_Multipath/Red_Hat_Enterprise_Linux-6-DM_Multipath-en-US.pdf


21

Device Mapper name Multipath disk files Description

/dev/mapper/mpathl sdu

sdv

PROD_LOG_ASM0003

25GB / RAID 1

/dev/mapper/mpathm sdw

sdae

TEST_DATA_ASM0000

110GB / RAID 5

/dev/mapper/mpathn sdx

sdaf

TEST_DATA_ASM0001

110GB / RAID 5

/dev/mapper/mpatho sdy

sdag

TEST_DATA_ASM0002

110GB / RAID 5

/dev/mapper/mpathp sdz

sdah

TEST_DATA_ASM0003

110GB / RAID 5

/dev/mapper/mpathq sdaa

sdai

TEST_LOG_ASM0000

30GB / RAID 1

/dev/mapper/mpathr sdab

sdaj

TEST_LOG_ASM0001

30GB / RAID 1

/dev/mapper/mpaths sdac

sdak

TEST_LOG_ASM0002

30GB / RAID 1

/dev/mapper/mpatht sdad

sdal

TEST_LOG_ASM0003

30GB / RAID 1

Setup the UDEV rules file

Oracle Grid Infrastructure does not provide the ASMLib package for Linux 6. ASMLib is a support library for the Automatic Storage Management to directly access disks. To resolve this issue use the Linux UDEV facility to provide Oracle permissions and ownership for device mapper files. The udev tool is a user-space file system that provides a dynamic mapping of device filenames to hardware devices.

Create the udev permission file to enable Oracle permissions and ownership of the device mapper files. Enter the mpath name of the volume, the oracle user and group as well as a permission value of 660. To determine the mpath name use the command “multipath –ll” to display the mpath to disk mappings. A system reboot will initiate this configuration. This must be done prior to installing Grid Infrastructure software.

Create the 12-dm-permissions.rules file to set the udev permission:

# vi /etc/udev/rules.d/12-dm-permissions.rules

ENV{DM_NAME}=="mpathe", OWNER:="oracle", GROUP:="dba", MODE:="660"

ENV{DM_NAME}=="mpathf", OWNER:="oracle", GROUP:="dba", MODE:="660"

ENV{DM_NAME}=="mpathg", OWNER:="oracle", GROUP:="dba", MODE:="660"

ENV{DM_NAME}=="mpathh", OWNER:="oracle", GROUP:="dba", MODE:="660"

ENV{DM_NAME}=="mpathi", OWNER:="oracle", GROUP:="dba", MODE:="660"

ENV{DM_NAME}=="mpathj", OWNER:="oracle", GROUP:="dba", MODE:="660"

ENV{DM_NAME}=="mpathk", OWNER:="oracle", GROUP:="dba", MODE:="660"

ENV{DM_NAME}=="mpathl", OWNER:="oracle", GROUP:="dba", MODE:="660"

ENV{DM_NAME}=="mpathm", OWNER:="oracle", GROUP:="dba", MODE:="660"

ENV{DM_NAME}=="mpathn", OWNER:="oracle", GROUP:="dba", MODE:="660"

ENV{DM_NAME}=="mpatho", OWNER:="oracle", GROUP:="dba", MODE:="660"

ENV{DM_NAME}=="mpathp", OWNER:="oracle", GROUP:="dba", MODE:="660"

ENV{DM_NAME}=="mpathq", OWNER:="oracle", GROUP:="dba", MODE:="660"

ENV{DM_NAME}=="mpathr", OWNER:="oracle", GROUP:="dba", MODE:="660"

ENV{DM_NAME}=="mpaths", OWNER:="oracle", GROUP:="dba", MODE:="660"

ENV{DM_NAME}=="mpatht", OWNER:="oracle", GROUP:="dba", MODE:="660"


22

Appendix B: Oracle GRID Infrastructure 11gR2 installation

Install Oracle Grid Infrastructure 11gR2 on both DL980 servers. Create the ASM disk groups for the PROD database instance on one server and the ASM disk groups for the TEST database instance on the other server. Oracle Grid Infrastructure only allows for a single ASM disk group to be created during the installation and the other disk groups will need to be created afterwards. Please refer to the Oracle Grid Infrastructure Installation Guide 11g Release 2 for Linux for further details.

Server1 Install Oracle Grid Infrastructure software and create the ASM disk groups for the PROD database instance.

The following answers were provided to the installation wizard:

Select Installation Option Configure Oracle Grid Infrastructure for a Standalone Server Disk Group Name: PROD_DATA Redundancy: External AU Size: 1 MB Disk Discovery Path: /dev/mapper/* Add Disks /dev/mapper/mpathe, /dev/mapper/mpathf, /dev/mapper/mpathg, /dev/mapper/mpathh Create the ORACLE ASM environment file.

Login as the oracle user and change to the oracle home directory and then create the asm.env file.

$ cd

$ vi asm.env

export ORACLE_HOME=/apps/oracle/product/11.2.0/grid

export ORACLE_BASE=/apps/oracle

export PATH=$PATH:$ORACLE_HOME/bin

export ORACLE_SID=+ASM

Source the newly created asm.env file and create the PROD_LOG ASM Disk Group.

$ . ./asm.env

$ sqlplus "/ as sysasm"

SQL> CREATE DISKGROUP PROD_LOG EXTERNAL REDUNDANCY

DISK

'/dev/mapper/mpathi',

'/dev/mapper/mpathj',

'/dev/mapper/mpathk,

'/dev/mapper/mpathl'

ATTRIBUTE

'au_size' = '1M',

'compatible.asm' = '11.2',

'compatible.rdbms' = '11.2'

/

SQL> alter diskgroup PROD_LOG mount;

SQL> select name, total_mb from v$asm_diskgroup order by name;

NAME TOTAL_MB

--------------------- ---------

PROD_DATA 409600

PROD_LOG 102400

SQL> select name, path from v$asm_disk order by name;

NAME PATH

---------------------- --------------------

PROD_DATA_0000 /dev/mapper/mpathe

PROD_DATA_0001 /dev/mapper/mpathf

PROD_DATA_0002 /dev/mapper/mpathg

PROD_DATA_0003 /dev/mapper/mpathh

PROD_LOG_0000 /dev/mapper/mpathi

PROD_LOG_0001 /dev/mapper/mpathj

PROD_LOG_0002 /dev/mapper/mpathk

PROD_LOG_0003 /dev/mapper/mpathl

http://download.oracle.com/docs/cd/E11882_01/install.112/e10812/toc.htm


23

Server2 Install Oracle Grid Infrastructure software and create the ASM disk groups for the TEST database instance.

The following answers were provided to the installation wizard:

Select Installation Option Configure Oracle Grid Infrastructure for a Standalone Server Disk Group Name: TEST_DATA Redundancy: External AU Size: 1 MB Disk Discovery Path: /dev/mapper/* Add Disks /dev/mapper/mpathm, /dev/mapper/mpathn, /dev/mapper/mpatho, /dev/mapper/mpathp Create the Oracle ASM environment file.

Login as the oracle user and change to the oracle home directory and then create the asm.env file.

$ cd

$ vi asm.env

export ORACLE_HOME=/apps/oracle/product/11.2.0/grid



export ORACLE_SID=+ASM

Source the newly created asm.env file and create the TEST_LOG ASM disk group.

$ . ./asm.env

$ sqlplus "/ as sysasm"

SQL> CREATE DISKGROUP TEST_LOG EXTERNAL REDUNDANCY

DISK

'/dev/mapper/mpathq',

'/dev/mapper/mpathr',

'/dev/mapper/mpaths,

'/dev/mapper/mpatht'

ATTRIBUTE

'au_size' = '1M',

'compatible.asm' = '11.2',

'compatible.rdbms' = '11.2'

/

SQL> alter diskgroup TEST_LOG mount;

SQL> select name, total_mb from v$asm_diskgroup order by name;

NAME TOTAL_MB

--------------------- ---------

TEST_DATA 450560

TEST_LOG 122880

SQL> select name, path from v$asm_disk order by name;

NAME PATH

---------------------- --------------------

TEST_DATA_0000 /dev/mapper/mpathm

TEST_DATA_0001 /dev/mapper/mpathn

TEST_DATA_0002 /dev/mapper/mpatho

TEST_DATA_0003 /dev/mapper/mpathp

TEST_LOG_0000 /dev/mapper/mpathq

TEST_LOG_0001 /dev/mapper/mpathr

TEST_LOG_0002 /dev/mapper/mpaths

TEST_LOG_0003 /dev/mapper/mpatht

Modify the ASM startup for Serviceguard for Linux The ASM instances will need to be modified so Serviceguard for Linux can manage the mounting and dismounting of the ASM disk groups. Configure both ASM instances to start without mounting any disk groups. By default ASM creates a SPFILE on the ASM disk group defined during the installation. Create a PFILE from the SPFILE and store it in the ORACLE_HOME/dbs directory. The ASM PFILE will be explicitly called by Serviceguard for Linux to start the ASM instances.


24

Both servers Startup the ASM instance and create a PFILE in the ASM ORACLE_HOME/dbs directory.

Login as the oracle user and source the asm.env file.

Log into the ASM instance and create the PFILE and then shut down the instance.

$ sqlplus “/ as sysasm”

SQL> create pfile='/apps/oracle/product/11.2.0/grid/dbs/init+ASM.ora' from

spfile;

SQL> shutdown

Edit the PFILEs to prevent ASM from mounting disk groups.

$ vi /apps/oracle/product/11.2.0/grid/dbs/init+ASM.ora

+ASM.asm_diskgroups=’ ‘ #Manual Mount

Validate the ASM instances. It is recommended at this time to validate that both ASM instances work as planned. This will limit the scope of troubleshooting to the Grid Infrastructure installation phase. After successfully completing all the following steps continue to the next phase installing Oracle Enterprise database.

Verify that the ASM instances can start up without mounting any disk groups.

Both servers Start up the ASM instances using the newly created PFILE and verify that no disk groups were mounted.


SQL> startup pfile=/apps/oracle/product/11.2.0/grid/dbs/init+ASM.ora;

SQL> select name, state from v$asm_diskgroup order by name;

NAME STATE

----------------- -----------

PROD_DATA DISMOUNTED

PROD_LOG DISMOUNTED

TEST_DATA DISMOUNTED

TEST_LOG DISMOUNTED

Server2 Manually mount and dismount all disk groups.


SQL> alter diskgroup PROD_DATA, PROD_LOG, TEST_DATA, TEST_LOG mount;


NAME STATE

----------------- --------------------

PROD_DATA MOUNTED

PROD_LOG MOUNTED

TEST_DATA MOUNTED

TEST_LOG MOUNTED

SQL> alter diskgroup PROD_DATA, PROD_LOG, TEST_DATA, TEST_LOG dismount;


NAME STATE

----------------- -----------


PROD_LOG DISMOUNTED


TEST_LOG DISMOUNTED


25

Server1 Mount and dismount all disk groups.


SQL> alter diskgroup PROD_DATA, PROD_LOG, TEST_DATA, TEST_LOG mount;


NAME STATE

----------------- --------------------

PROD_DATA MOUNTED

PROD_LOG MOUNTED

TEST_DATA MOUNTED

TEST_LOG MOUNTED

SQL> alter diskgroup PROD_DATA, PROD_LOG, TEST_DATA, TEST_LOG dismount;


NAME STATE

----------------- -----------


PROD_LOG DISMOUNTED


TEST_LOG DISMOUNTED

Mount the PROD instance disk groups to prepare for the next section, installing Oracle Enterprise database.

SQL> alter diskgroup PROD_DATA, PROD_LOG mount;


NAME STATE

----------------- --------------------

PROD_DATA MOUNTED

PROD_LOG MOUNTED


TEST_LOG DISMOUNTED

Server2 Mount the TEST instance disk groups to prepare for the next section, installing Oracle Enterprise database.


SQL> alter diskgroup TEST_DATA, TEST_LOG mount;


NAME STATE

----------------- --------------------


PROD_LOG DISMOUNTED

TEST_DATA MOUNTED

TEST_LOG MOUNTED


26

Appendix C: Oracle Enterprise Database Edition 11gR2 installation

On server1, install the Oracle Enterprise Database Edition 11gR2 software in two separate ORACLE_HOMEs. Create the temporary PROD database instance in one and install the software only for the TEST database instance. On server2, install the Oracle Enterprise Database Edition 11gR2 software in two separate ORACLE_HOMEs identical to server1. Create the temporary TEST database instance in one and install the software only for the PROD database instance.

Copy the Oracle parameter files created during the initial setup of the PROD and TEST database instance over to their corresponding ORACLE_HOMEs on the standby server.

The temporary database instances will be used to validate that the databases can be manually moved between servers prior to installing Serviceguard for Linux. The final PROD and TEST databases will be created after installing the database workload software. Reference the following Oracle documentation for installing Oracle database: Oracle Database Installation Guide 11g Release 2 11.2) for Linux and support note 169706.1 Oracle Database on UNIX AIX, HP-UX, Linux, Mac OS X, Solaris, Tru64 UNIX Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2).

The following steps outline the process for installing the Oracle database instances in our solution.

Server1 Install the Oracle software and create the PROD database.

Invoke the runInstaller as the oracle user and select the following options.

Select Installation Option: Create and configure a database

Grid Installation Options: Single instance db installation

Select Install Type: Advanced install

Select DB Edition: Enterprise Edition

Oracle Base: /apps/oracle

Software Location: /apps/oracle/product/11.2.0/prod

Global db name: prod.aps.com

SID: prod

Specify Storage Options: Oracle Automatic Storage Management

Select ASM Disk Group: PROD_DATA

Install Oracle software with the software only option to create the ORACLE_HOME for the TEST database.

Invoke the runInstaller as the oracle user and select the following options.

Select Installation Option: Software only

Grid Installation Options: Single instance db installation

Select DB Edition: Enterprise Edition

Oracle Base: /apps/oracle

Software Location: /apps/oracle/product/11.2.0/test

Server2 Install the Oracle software and create the TEST database. This is identical to installing the Oracle software and creating the PROD database on Server1 except for the following:

Software Location: /apps/oracle/product/11.2.0/test

Global db name: test.aps.com

SID: test

Select ASM Disk Group: TEST_DATA

http://docs.oracle.com/cd/E11882_01/install.112/e24321.pdf

http://docs.oracle.com/cd/E11882_01/install.112/e24321.pdf


27

Install Oracle software with the software only option to create the ORACLE_HOME for the PROD database. This is identical to installing the Oracle software and creating the TEST ORACLE_HOME on Server1 except for the following:

Software Location: /apps/oracle/product/11.2.0/prod

Both servers Copy the parameter files <ORACLE_HOME>/dbs/init<SID>.ora to the corresponding <ORACLE_HOME>/dbs located on the other server.

Create the PROD and TEST environment file in the Oracle user’s home directory on both servers.

$vi /home/oracle/prod.env

export ORACLE_HOME=/apps/oracle/product/11.2.0/prod



export ORACLE_SID=prod

$vi /home/oracle/test.env

export ORACLE_HOME=/apps/oracle/product/11.2.0/test



export ORACLE_SID=test

Validate the database instances It is recommended at this time to validate that both database instances work as planned. This will limit the scope of troubleshooting to the Oracle Enterprise database installation phase. After successfully completing all the following steps continue to Appendix D: HP Serviceguard for Linux A.11.20 installation.

The following steps were used to verify that the PROD and TEST database instances will run on either server.

Server1 Login as the oracle user and source the newly create prod.env file.

$ cd

$ . ./prod.env

Login to the PROD database instance, verify that it is up and then shut down the database.

$ sqlplus / as sysdba

SQL> show sga

SQL> shutdown

Source the ASM environment file and dismount the PROD_DATA and PROD_LOG ASM disk groups.

$ . ./asm.env


SQL> alter diskgroup PROD_DATA, PROD_LOG dismount;


NAME STATE

----------------- -----------


PROD_LOG DISMOUNTED


TEST_LOG DISMOUNTED

Server2 Login as the oracle user and source the newly create test.env file.

$ . ./test.env

Login to the TEST database instance, verify that it is up and then shut down the database.


SQL> show sga

SQL> shutdown


28

Source the ASM environment file and dismount the TEST_DATA and TEST_LOG ASM disk groups.

$ . ./asm.env


SQL> alter diskgroup TEST_DATA, TEST_LOG dismount;


NAME STATE

----------------- -----------


PROD_LOG DISMOUNTED


TEST_LOG DISMOUNTED

Source the ASM environment file and mount the PROD database disk groups.

$ . ./asm.env


SQL> alter diskgroup PROD_DATA, PROD_LOG mount;


NAME STATE

----------------- -----------

PROD_DATA MOUNTED

PROD_LOG MOUNTED


TEST_LOG DISMOUNTED

Source the prod.env file, startup the PROD database instance, and then shut down the database.

$ . ./prod.env


SQL> startup

SQL> shutdown

Source the ASM environment file, dismount the PROD database disk groups, and then shut down the instance.

$ . ./asm.env


SQL> alter diskgroup PROD_DATA, PROD_LOG dismount;


NAME STATE

----------------- -----------


PROD_LOG DISMOUNTED


TEST_LOG DISMOUNTED

SQL> shutdown

Server1 Source the ASM environment file and mount the TEST database disk groups.

$ . ./asm.env


SQL> alter diskgroup TEST_DATA, TEST_LOG mount;


NAME STATE

----------------- -----------


PROD_LOG DISMOUNTED

TEST_DATA MOUNTED

TEST_LOG MOUNTED

Source the test.env file, startup the TEST database instance, and then shut down the database.

$ . ./test.env


SQL> startup

SQL> shutdown


29

Source the ASM environment file, dismount the TEST database disk groups, and then shut down the ASM instance.

$ . ./asm.env


SQL> alter diskgroup TEST_DATA, TEST_LOG dismount;

SQL> shutdown

Appendix D: HP Serviceguard for Linux A.11.20 installation

Install Serviceguard for Linux

Server1

Mount the ISO file, and run the installation script.

# mkdir /dvd

# mount -o loop /stage/SGLX1120/SGLX1120_BB070-11001.iso /dvd

# cd /

# ./dvd/cmeasyinstall

The Serviceguard for Linux installation wizard will first check for all the pre-requisite files and packages required for installation. If any are missing, install the missing files and re-run the installation script.

After a successful installation the logfile can be found at /tmp/cmeasyinstall.log.

Append the root user’s profile to include the Serviceguard for Linux environment.

# vi .bash_profile

PATH=$PATH:/usr/local/cmcluster/bin

export PATH

. /etc/cmcluster.conf

Modify MAN pages.

# vi /etc/man.config

MANPATH /usr/local/cmcluster/doc/man

Create the cmclnodelist file for root level access between nodes.

# vi $SGCONF/cmclnodelist

dl980-1 root #PROD_CLUSTER, node1

dl980-2 root #PROD_CLUSTER, node2

Update the hosts file.

It is recommended to setup the hosts file for the cluster as a backup for the DNS server.

# vi /etc/hosts

192.168.1.201 dl980-1.aps.com dl980-1

192.168.1.202 dl980-2.aps.com dl980-2

192.168.1.203 prod.aps.com prod

192.168.1.204 test.aps.com test

Turn off IPTABLES.

By default, IPTABLES blocks access to the Serviceguard Manager for Linux. Stop IPTABLES prior to accessing the Serviceguard Manager for Linux.

# service iptables save

# service iptables stop

# chkconfig iptables off

Install HP Serviceguard Oracle Toolkit for Linux

Mount the ISO file and install the rpm:

# mount -o loop /stage/SGLX1120/Oracle_ToolKit_BB074-10001.iso /dvd

Install the toolkit RPM.

# cd /dvd/RedHat/Oracletoolkit/x86_64

# rpm -ivh serviceguard-oracle-toolkit-A.05.01.00-00.redhat.noarch.rpm


30

Appendix E: HP Serviceguard for Linux cluster setup

In the next steps, you will create the cluster, define the node membership, and configure the cluster heartbeat and cluster lock LUN device.

From an Internet browser invoke the HP System Management Homepage, https://[hostname]:2381.

Login as the root user.

Go to the Tools tab.

Click the “Serviceguard Manager” link to launch the Serviceguard Manager for Linux.

Click the “Create Cluster” button on the right.

Enter the Cluster Name, for example “Prod_cluster” and enter checkmarks in the boxes for both nodes, for example:

dl980-1, dl980-2

Go to the Network tab.

Enter in the “Subnets” section, for example:

Subnet: 10.1.1.0, Type: Heartbeat

Enter in the “Select Subnet Configuration” section, for example:

Node Network Address

Dl980-1 bond0 10.1.1.201

Dl980-2 bond0 10.1.1.202

Go to the Lock tab

For the Cluster Lock Type, select “Lock LUN”.

Enter the Lock LUN Path for each node, for example:

DL980-1 /dev/mapper/mpathcp1

DL980-2 /dev/mapper/mpathcp1

Select Finish.

Note When using Device Mapper Multipath, the path to the cluster Lock LUN, for example /dev/mapper/mpathcp1, must be the same on each node.

Select Check Configuration. Look for any errors and resolve them before going on to Appendix F: HP Serviceguard for Linux ASM package setup.

Note You may get a warning about the default NODE_TIMEOUT value. This warning can be ignored here, but refer to the documentation when finalizing your cluster.

Select Apply Configuration and select “OK” in the pop-up dialog box.

Note

During the creation of the cluster the Serviceguard for Linux files will be copied over to Server2. You must modify Server2’s root user’s profile, MAN pages, cmclnodelist, hosts, and set the IPTABLES to match Server1.


31

To verify the cluster configuration, run the following options from the Administration menu of the HP Serviceguard Manager for Linux Summary page to test that each node can run the cluster in the event that the other node fails:

a. Administration -> Run Cluster

b. Administration -> Halt Node

c. Administration -> Run Node

See Figure 8 for viewing the Serviceguard Manager for Linux screens for the cluster configuration. Figure 8. Serviceguard Manager for Linux screens showing the cluster configuration


32

Appendix F: HP Serviceguard for Linux ASM package setup

In this solution the following step were used to create the HP Serviceguard for Linux ASM multinode package to monitor and manage the Oracle ASM instances on both nodes.

Create the package configuration file.

# mkdir $SGCONF/asm_pkg

# cd $SGCONF/asm_pkg

# cmmakepkg -m sg/multi_node -m tkit/oracle/oracle asm_pkg.conf

Edit the multinode package configuration file prior to building the package.

# vi asm_pkg.conf

package_name asm_pkg

package_type multi_node

node_name *

auto_run yes

node_fail_fast_enabled no

tkit/oracle/oracle/TKIT_DIR ${SGCONF}/asm_pkg

tkit/oracle/oracle/INSTANCE_TYPE ASM

tkit/oracle/oracle/ORACLE_ADMIN oracle

tkit/oracle/oracle/ASM yes

tkit/oracle/oracle/ASM_DISKGROUP PROD_DATA

tkit/oracle/oracle/ASM_DISKGROUP PROD_LOG

tkit/oracle/oracle/ASM_DISKGROUP TEST_DATA

tkit/oracle/oracle/ASM_DISKGROUP TEST_LOG

tkit/oracle/oracle/ASM_HOME /apps/oracle/product/11.2.0/grid

tkit/oracle/oracle/ASM_USER oracle

tkit/oracle/oracle/ASM_SID +ASM

tkit/oracle/oracle/LISTENER no

tkit/oracle/oracle/PFILE ${ASM_HOME}/dbs/init${ASM_SID}.ora

tkit/oracle/oracle/MONITOR_PROCESSES asm_pmon_${ASM_SID}

tkit/oracle/oracle/MONITOR_PROCESSES asm_dbw0_${ASM_SID}

tkit/oracle/oracle/MONITOR_PROCESSES asm_ckpt_${ASM_SID}

tkit/oracle/oracle/MONITOR_PROCESSES asm_smon_${ASM_SID}

tkit/oracle/oracle/MONITOR_PROCESSES asm_lgwr_${ASM_SID}

tkit/oracle/oracle/MONITOR_PROCESSES asm_gmon_${ASM_SID}

tkit/oracle/oracle/MONITOR_PROCESSES asm_rbal_${ASM_SID}

tkit/oracle/oracle/MAINTENANCE_FLAG yes

tkit/oracle/oracle/MONITOR_INTERVAL 30

tkit/oracle/oracle/TIME_OUT 30

tkit/oracle/oracle/PARENT_ENVIRONMENT no

tkit/oracle/oracle/CLEANUP_BEFORE_STARTUP no

tkit/oracle/oracle/USER_SHUTDOWN_MODE abort

tkit/oracle/oracle/KILL_ASM_FOREGROUNDS yes

service_name oracle_asm_service

service_cmd

"$SGCONF/scripts/tkit/oracle/tkit_module.sh Oracle_monitor"

service_restart none

service_fail_fast_enabled no

service_halt_timeout 300

service_name oracle_asm_hang_service

service_cmd

"$SGCONF/scripts/tkit/oracle/tkit_module.sh

oracle_hang_monitor 30 failure"




Build the ASM multinode package.

# cmapplyconf -P asm_pkg.conf


33

Start the ASM multinode package to complete the configuration:

From your browser, on the “HP Serviceguard Manager Summary” page, select the “ASM” package.

From the “Administration” menu, select “Run Package”.

See Figure 9 for the Serviceguard Manager for Linux screens displaying the ASM package configuration.

Note You may need to start the cluster if it is not already running. To run the cluster, go to the “Administration” menu and select “Run Cluster”.

Validate the ASM package.

It is recommended to stop and start the ASM package to verify that it works as planned.

From your browser, on the “HP Serviceguard Manager Summary” page, select the “ASM” package.

From the “Administration” menu, select “Halt Package” and then “Start Package”.

Figure 9. Configuration of ASM package on Serviceguard Manager for Linux screens


34

Appendix G: HP Serviceguard for Linux PROD package setup

In this solution the following steps were used to create the HP Serviceguard for Linux PROD failover package to monitor and manage the Oracle PROD database instance within the cluster.

Set up and configure the database failover package.

# mkdir $SGCONF/prod_pkg

# cd $SGCONF/prod_pkg

# cmmakepkg -m tkit/oracle/oracle prod_pkg.conf

The prod_pkg.conf file must be edited before it can be used to build the PROD package.

# vi prod_pkg.conf

package_name prod_pkg

package_type failover

node_name *

auto_run yes

failback_policy manual

tkit/oracle/oracle/TKIT_DIR ${SGCONF}/prod_pkg

tkit/oracle/oracle/INSTANCE_TYPE database

tkit/oracle/oracle/ORACLE_HOME /apps/oracle/product/11.2.0/prod

tkit/oracle/oracle/ORACLE_ADMIN oracle

tkit/oracle/oracle/SID_NAME prod

tkit/oracle/oracle/START_MODE open

tkit/oracle/oracle/ASM yes

tkit/oracle/oracle/ASM_DISKGROUP PROD_DATA

tkit/oracle/oracle/ASM_DISKGROUP PROD_LOG

tkit/oracle/oracle/ASM_HOME /apps/oracle/product/11.2.0/grid

tkit/oracle/oracle/ASM_USER oracle

tkit/oracle/oracle/ASM_SID +ASM

tkit/oracle/oracle/LISTENER yes

tkit/oracle/oracle/LISTENER_NAME LISTENER

tkit/oracle/oracle/LISTENER_RESTART 2

tkit/oracle/oracle/PFILE ${ORACLE_HOME}/dbs/init${SID_NAME}.ora

tkit/oracle/oracle/MONITOR_PROCESSES ora_pmon_${SID_NAME}

tkit/oracle/oracle/MONITOR_PROCESSES ora_dbw0_${SID_NAME}

tkit/oracle/oracle/MONITOR_PROCESSES ora_ckpt_${SID_NAME}

tkit/oracle/oracle/MONITOR_PROCESSES ora_smon_${SID_NAME}

tkit/oracle/oracle/MONITOR_PROCESSES ora_lgwr_${SID_NAME}

tkit/oracle/oracle/MONITOR_PROCESSES ora_reco_${SID_NAME}

tkit/oracle/oracle/MONITOR_PROCESSES ora_mman_${SID_NAME}

tkit/oracle/oracle/MONITOR_PROCESSES ora_psp0_${SID_NAME}

tkit/oracle/oracle/MONITOR_PROCESSES ora_dbrm_${SID_NAME}

tkit/oracle/oracle/MONITOR_PROCESSES ora_vktm_${SID_NAME}

tkit/oracle/oracle/MONITOR_PROCESSES ora_rbal_${SID_NAME}

tkit/oracle/oracle/MONITOR_PROCESSES ora_asmb_${SID_NAME}

tkit/oracle/oracle/MAINTENANCE_FLAG yes

tkit/oracle/oracle/MONITOR_INTERVAL 30

tkit/oracle/oracle/TIME_OUT 30

tkit/oracle/oracle/PARENT_ENVIRONMENT no

tkit/oracle/oracle/CLEANUP_BEFORE_STARTUP no

tkit/oracle/oracle/USER_SHUTDOWN_MODE abort

tkit/oracle/oracle/KILL_ASM_FOREGROUNDS yes

tkit/oracle/oracle/DB_SERVICE all

service_name oracle_proddb_service

service_cmd "$SGCONF/scripts/tkit/oracle/tkit_module.sh

oracle_monitor"




service_name oracle_proddb_listener_service


oracle_monitor_listener"




35


service_name oracle_proddb_hang_service


oracle_hang_monitor 30 failure"




priority 10

dependency_name asm_dep

dependency_condition asm_pkg = up

dependency_location same_node

dependency_name prod_dep

dependency_condition test_pkg = down

dependency_location same_node

monitored_subnet 192.168.1.0

moinitored_subnet_access full

ip_subnet 192.168.1.0

ip_subnet_node dl980-1

ip_subnet_node dl980-2

ip_address 192.168.1.203

Build the PROD package.

# cmapplyconf -P prod_pkg.conf

Start the PROD package to complete the configuration.

From your browser, on the “HP Serviceguard Manager Summary” page, select the ”PROD” package. From the “Administration” menu, select “Run Package”.

Note

The PROD package is dependent on the ASM multinode package to be running before it can be started on that node.

See Figure 10 to view the Serviceguard for Linux PROD package configuration.

Validate the PROD packages.

It is recommended to run the PROD packages on both nodes to verify that it works as planned.

From your browser, on the “HP Serviceguard Manager Summary” page, select the ”PROD” package. From the “Administration” menu, select “Move Package” and then select “Node2”.

Halt the PROD package to prepare for building and verifying the TEST package.

From your browser, on the “HP Serviceguard Manager Summary” page, select the ”PROD” package. From the “Administration” menu, select “Halt Package”.


36

Figure 10. Configuration of the PROD package shown in Serviceguard Manager for Linux screens


37

Appendix H: HP Serviceguard for Linux TEST package setup

In this solution the following steps were used to create the HP Serviceguard for Linux TEST failover package to monitor and manage the Oracle TEST database instance within the cluster.

The processes are the same as creating the PROD package only the unique steps and settings are listed below.

# mkdir $SGCONF/test_pkg

# cd $SGCONF/test_pkg

# cmmakepkg -m tkit/oracle/oracle test_pkg.conf

The test_pkg.conf file must be edited before it can be used to build the TEST package.

# vi test_pkg.conf

package_name test_pkg

auto_run no

failback_policy manual

failover_policy manual

tkit/oracle/oracle/TKIT_DIR ${SGCONF}/test_pkg

tkit/oracle/oracle/ORACLE_HOME /apps/oracle/product/11.2.0/test

tkit/oracle/oracle/SID_NAME test

tkit/oracle/oracle/ASM_DISKGROUP TEST_DATA

tkit/oracle/oracle/ASM_DISKGROUP TEST_LOG

service_name oracle_testdb_service

service_name oracle_testdb_listener_service

service_name oracle_testdb_hang_service

priority no_priority

dependency_name test_dep

dependency_condition prod_pkg = down

ip_address 192.168.1.204

Build the TEST package.

# cmapplyconf -P test_pkg.conf

Start the TEST package to complete the configuration.

From your browser, on the “HP Serviceguard Manager Summary” page, select the “TEST” package. From the “Administration” menu, select “Run Package”.

Note

The TEST package is dependent on the ASM multinode package to be running before it can be started on that node. It is also dependent on the PROD package not running on that node before running the TEST package.


38

Figure 11 shows the Serviceguard for Linux TEST package configuration.

Validate the TEST packages.

It is recommended to run the TEST packages on both nodes to verify that it works as planned.

From your browser, on the “HP Serviceguard Manager Summary” page, select the “TEST” package. From the “Administration” menu, select “Move Package” and then select “Node2”.

Figure 11. Configuration of the TEST package on Serviceguard Manager for Linux screens


39

Appendix I: Bill of Materials

Table 8 shows the equipment and components used in the HP ProLiant DL980 Universal Database Solution.

Table 8. Bill of materials for the HP ProLiant DL980 Universal Database Solution

Quantity Product Description

HP rack and accessories

1 AF002A HP Universal Rack 10642 G2 Shock Rack

1 AF090A HP 10K Rack Airflow Optimization Kit

1 AF054A HP 10642 G2 Sidepanel Kit

2 252663-B24 HP 16A High Voltage Modular PDU

2 AF593A HP 3.6m C19 Nema L6-20P NA/JP Pwr Crd

HP DL980 G7

2 AM451A HP ProLiant DL980 G7 CTO system-E7 proc

2 AM450A HP DL980 CPU Installation Assembly for E7

2 650770-L21 HP DL980 G7 E7-4870 FIO 4-processor Kit

2 650770-B21 HP DL980 G7 E7-4870 4-processor Kit

16 A0R60A HP DL980G7 (E7) Memory Cartridge

256 A0R58A HP DL980 8GB 2Rx4 PC3L-10600R-9 Kit

2 AM434A HP DL980 LP PCIe I/O Expansion Module

4 AJ764A HP 82Q 8Gb Dual Port PCI-e FC HBA

4 614203-B21 HP NC552SFP 10GbE 2P Svr Adapter

8 455883-B21 HP 10Gb Short Range SFP Option

8 507127-B21 HP DL980 300GB 6G SAS 10K 2.5 DP ENT HDD

2 481043-B21 HP Slim 12.7mm SATA DVDRW Optical Kit

2 534562-B21 HP 1G Flash Backed Cache

8 AM470A HP DL980 1200W CS Plat Ht Plg Pwr Supply

2 631660-B21 HP DL580/DL585/DL980 G7 Power Cable Kit

HP 3PAR StoreServ 10800

1 QR585C HP 3PAR StoreServ 10800 single phase Rack Config Base

2 QR638C HP 3PAR StoreServ 10800 2.8-GHz Controller Node

12 QR591A HP 3PAR StoreServ 10000 4-Port 8Gb/s Fibre Channel Host/Disk Adapter

2 QR592C HP 3PAR StoreServ 10000 40-disk Drive Chassis

2 QR598A HP 3PAR StoreServ 10000 Rackmount Kit for 40-disk Drive Chassis

20 QW902A HP 3PAR StoreServ 10000 4x300GB 6Gb/s SFF 15K SAS Drive Mag

4 QL266B HP 3PAR 10M 50/125 (LC-LC) Fiber Cable


40

Quantity Product Description

Fibre Channel Switches

2 AW575B HP SN6000 Stackable 8Gb 24-port FC Switch

6 QK734A 5m PremierFlex OM4 LC/LC Multi-Mode Optical Cable

40 AJ718A HP 8 Gbps Short Wave FC SFP+

Ethernet Switches

2 JC100A HP 5800-24G Switch

2 JD362A HP 5800/A5500 150W AC Power Supply

2 JC092B HP 5800 2-port 10GbE SFP+ Module

4 BK837A HP 0.5 m PremierFlex OM3+ LC/LC Optical Cable


For more information

HP ProLiant DL980 Universal Database Solution website hp.com/go/udb

HP Serviceguard for Linux website hp.com/go/sglx

HP Serviceguard for Linux Deployment Guide http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c03393068/c03393068.pdf

HP Serviceguard Disaster Recovery Solutions Brochure http://h20195.www2.hp.com/V2/GetDocument.aspx?docname=4AA4-4108ENW

HP ProLiant DL980 G7 website hp.com/go/dl980

ProLiant DL980 G7 server QuickSpecs http://h18000.www1.hp.com/products/quickspecs/13708_div/13708_div.pdf

HP ProLiant DL980 G7 server with HP PREMA Architecture http://h20195.www2.hp.com/V2/GetDocument.aspx?docname=4AA3-0643ENW

HP 3PAR StoreServ website hp.com/go/3par

HP 3PAR StoreServ storage brochure http://h20195.www2.hp.com/v2/GetDocument.aspx?docname=4AA3-2542ENW

HP 3PAR Thin Technologies Solution Brief http://h20195.www2.hp.com/v2/GetDocument.aspx?docname=4AA3-2545ENW

HP 5800 Switch Series http://h17007.www1.hp.com/us/en/networking/products/switches/HP_5800_Switch_Series/index.aspx

HP SN6000 series FC switches http://www8.hp.com/us/en/products/switches/product-detail.html?oid=5306975

HP and Oracle alliance hp.com/go/oracle

HP portal for information on Oracle solutions hporacle.com

Open Source and Linux from HP hp.com/go/linux

Oracle 11gR2 Grid and Database documentation oracle.com/pls/db112/homepage

To help us improve our documents, please provide feedback at hp.com/solutions/feedback

Sign up for updates

hp.com/go/getupdated

© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for

HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as

constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation. Intel and Xeon are trademarks of Intel Corporation in the U.S. and other

countries. Oracle and Java are registered trademarks of Oracle and/or its affiliates. UNIX is a registered trademark of The Open Group.

4AA4-6631ENW, May 2013

http://www.hp.com/go/udb


http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c03393068/c03393068.pdf


http://www.hp.com/go/dl980

http://h18000.www1.hp.com/products/quickspecs/13708_div/13708_div.pdf


http://www.hp.com/go/3par



http://h17007.www1.hp.com/us/en/networking/products/switches/HP_5800_Switch_Series/index.aspx

http://www8.hp.com/us/en/products/switches/product-detail.html?oid=5306975

http://www.hp.com/go/oracle

http://www.hporacle.com/

http://www.hp.com/go/linux

http://www.oracle.com/pls/db112/homepage

http://www.hp.com/solutions/feedback

http://www.hp.com/go/getupdated

Documents

HP ProLiant DL980 Universal Database Solution: HP Serviceguard