DS3000 Oracle RAC 10gR2 ASM Windows

Deploying Oracle RAC 10g Release 2 on a Windows Server 2003 with an IBM DS3000 Series Storage Subsystem

Installation and Configuration Guide37478-00, Rev. BDecember 2008© Copyright 2008, IBM Corporation

Copyright © 2008 by International Business Machines Corporation. All rights reserved.

Contents

Chapter 1: INTRODUCTION

Benefits ................................................................................................................................................................1-2

Required Expertise ...........................................................................................................................................1-3

Required Documentation..............................................................................................................................1-3

Required Components ...................................................................................................................................1-4

Chapter 2: REFERENCE ARCHITECTURE

Chapter 3: CONFIGURING THE HOST AND THE STORAGE SUBSYSTEM

Configuring the Host ......................................................................................................................................3-1

Configuring the Storage Subsystem .........................................................................................................3-2

Checking the Setup of the Host and the Storage Subsystem..........................................................3-3

Chapter 4: ORACLE SETUP AND INSTALLATION

Preparing for Installation...............................................................................................................................4-1

Enabling Windows Auto Mounting ...........................................................................................................4-1

Formatting the Disk Drives ...........................................................................................................................4-2

Installing Oracle Clusterware .......................................................................................................................4-3

Updating Oracle Clusterware.......................................................................................................................4-3

Installing the Oracle Database Software..................................................................................................4-3

Updating the Oracle Database Software .................................................................................................4-3

Configuring ASM and Creating a Database with DBCA .....................................................................4-3

Chapter 5: CONCLUSION

Contact Information........................................................................................................................................5-1

Appendix A: ORACLE RAC TROUBLESHOOTING

Oracle Clusterware Complains That “Nodes Are Not Clusterable”................................................ A-1

Services Fail to Start at the End of Oracle Clusterware Installation............................................... A-1

Appendix B: REFERENCES

Deploying Oracle Database RAC 10g on a Windows Server with a DS3000 System iii© Copyright 2008, IBM Corporation

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

iv Deploying Oracle Database RAC 10g on a Windows Server with a DS3000 System© Copyright 2008, IBM Corporation

Chapter 1
Introduction
Oracle Real Application Clusters (Oracle RAC) is a high-availability option to the Oracle database application. Oracle RAC supports the deployment of a single database across a cluster of servers, providing excellent fault tolerance, performance, and scalability. The industry is taking note of RAC’s growing importance in a wide range of customer implementations across all industries that rely on transaction processing and data warehousing applications.

This document presents an overview and information for use when deploying the Oracle RAC 10g Release 2 database with Oracle RAC on the Windows Server 2003 operating system using an IBM DS3000 series storage subsystem. The database storage is administered using the Oracle Automatic Storage Manager (ASM) feature. The Clusterware logical drives are configured on shared raw devices.

The procedure for creating a simple configuration using a two-node cluster provides an example to illustrate the principles used when performing this type of implementation. Actual implementations vary in complexity and require significant expertise in all of the technologies involved. This document is intended to present an overview of the installation process. This document describes any deviations from the Oracle installation guide and any enhancements in implementation needed to fully use the IBM DS3000 series storage subsystem. It is not designed to provide a comprehensive reference for installing these components. For full implementation instructions, reference the product documentation for the hardware, the Windows operating system, and Oracle Database and Clusterware.

Oracle documentation can be obtained from:

http://www.oracle.com/technology/documentation/index.html

Windows Server 2003 documentation is available at:

http://www.microsoft.com/windowsserver2003/proddoc/default.mspx

Storage subsystem documentation is available from IBM.

Deploying Oracle Database RAC 10g on a Windows Server with a DS3000 System 1-1© Copyright 2008, IBM Corporation

http://www.oracle.com/technology/documentation/index.html

http://www.microsoft.com/windowsserver2003/proddoc/default.mspx

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

BenefitsIBM DS3000 series storage subsystems combine with Oracle RAC to provide a stable, reliable solution for the high-availability needs of the small-to-medium business (SMB). Deploying Oracle RAC 10g Release 2 on ASM with Windows Server 2003 provides many benefits.

Reliability of enterprise-level data protection at a fraction of the usual cost – The IBM DS3000 series storage subsystem provides high-end RAS (reliability, availability, serviceability) features with prices that are surprisingly lower than expected.

“Start Small, Grow Big” scalability – The IBM DS3000 series storage subsystem lets you start small and make additional investments in your infrastructure only as data growth requires it. This scalability lowers the acquisition cost of the IBM DS3000 series storage subsystem without sacrificing expansion in the future.

Ease of implementation and management – DS3000® Storage Manager provides a friendly user interface from setup to administration. The web-like and task-based management interface reduces the complexity of installation, configuration, and management.

Low-cost, high performance SAS disk drives – SAS disk drives are dual-ported like Fibre Channel (FC) disk drives. SAS disk drives are as robust as FC disk drives. SAS disk drives support dual paths, one from each controller, to deliver maximum data availability. SAS disk drives provide the majority of features and abilities of FC disk drives at a lower price.

Enterprise-level performance and environmental flexibility – The SAS-based back-end disk drive interfaces of the IBM DS3000 series storage subsystem deliver fast data performance, while the front-end host interfaces let it operate at speeds of 1Gb/s, 2 Gb/s, and 4 Gb/s in FC, SAS, or iSCSI environments. This fast performance is a result of years of custom firmware development combined with a hardware XOR engine. The XOR hardware handles all of the continuous RAID and parity calculations. You can move large data loads in and out of the disk drives of the IBM DS3000 series storage subsystem without requiring excessive amounts of expensive cache.

Advanced Replication premium features – Typically only found in higher-end systems, premium features such as FlashCopy® and VolumeCopy are available as options on the IBM DS3000 series storage subsystem. You can add the these premium features when your environment requires them:

• FlashCopy creates capacity-efficient, point-in-time logical drive images, providing the logical drive image for such uses as file restoration and backup.

• VolumeCopy creates a complete physical copy, or clone, of a logical drive within a storage subsystem. These unique copies can be assigned to any host on the SAN and used for application testing and development, information analysis, or data mining.

1-2 Deploying Oracle Database RAC 10g on a Windows Server with a DS3000 System© Copyright 2008, IBM Corporation

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Required Expertise

Enterprise level Storage Partitioning premium feature – Storage Partitioning lets you map individual logical drives to a specific host or to a group of hosts known as host groups. You can associate each host with an operating system. You can use the IBM DS3000 series storage subsystem to support logical drives mapped to multiple hosts with various operating systems safely and easily. You have to manage only a single storage subsystem used by these different operating systems. The IBM DS3000 series storage subsystem is ideal for IT departments seeking to obtain a powerful, flexible storage subsystem to create Oracle Database solutions. Oracle database administrators can appreciate the non-disruptive capabilities, low price-per-IOPS, and management ease-of-use of the IBM DS3000 series storage subsystem.

Required ExpertiseTo successfully plan, install, configure, and operate this solution, the implementation team must be qualified in these areas:

• Oracle RAC installation and administration

• Storage subsystem management for these storage subsystems with one or more of these components:

■ DS3200 controller module



• Familiarity with storage subsystems and DS3000 Storage Manager

• Expertise in Windows Server 2003 system administration

• Expertise in planning and implementing storage area network (SAN) infrastructure

Required DocumentationThis document does not include detailed information about how to install component software. For this information, refer to the installation guide for the appropriate software.

See Table 1-1 for documentation concerning component hardware.


Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Required ComponentsOracle RAC was tested and proven using this set of components with the stated version and release levels:

• Approved storage subsystems with appropriate firmware

• DS3000 Storage Manager Version 2.70 or later with premium feature keys to enable Storage Partitioning

• Windows Server 2003

IMPORTANT Similar software and firmware levels might not generate the same results. No implied warranty is granted for a different set of products or revisions.

Table 1-1 Required Documents

Storage Subsystem That You Are Installing Refer to This Document

DS3200 DS3200 Controller Module Initial Setup Guide



Table 1-2 Required Documents

If Installing Use This Installation Guide

DS3000 storage subsystem IBM TotalStorage DS3000 Storage Manager Installation and Support Guide for Intel-Based Operating System Environment

Table 1-3 Tested and Proven Models and Versions

Storage Subsystem Firmware Levels

DS3200 controller module 6.70 or later




Chapter 2
Reference Architecture
The reference architecture contains these components:

• Two servers

• Windows Server 2003 Service Pack 2 (SP2)

• Oracle RAC 10g Release 2 Clusterware, version 10.2.0.3

• Oracle RAC 10g Release 2 Database, version 10.2.0.3

• Oracle ASM – A feature in Oracle RAC 10g Release 2 that provides the database administrator with a simple storage management interface that is consistent across all server and storage platforms. As a vertically integrated file system and a logical drive manager, purpose-built for Oracle database files, ASM provides the performance of asynchronous I/O with the easy management of a file system. ASM provides capability that saves the database administrator time and provides flexibility to manage a dynamic database environment with increased efficiency.

• An IBM DS3000 series storage subsystem. This series includes the DS3400 Fibre Channel storage subsystem, the DS3200 SAS storage subsystem, and the DS3300 iSCSI storage subsystem.

• DS3000 Storage Manager, including an MPIO multipathing driver.

• Host bus adapters (HBA) appropriate for the host interconnect of the storage subsystem. Two hardware paths from each host are required for failover.

http://www.ibm.com/systems/support/storage/config/ssic/displayesssearchwithoutjs.wss?start_over=yes

NOTE The test setup uses the cluster software supplied with Oracle RAC 10g Release 2. In previous releases of Oracle, this cluster software was called Cluster Ready Services (CRS). The cluster software was renamed to Oracle Clusterware with this release; however, much of the Oracle product and documentation still refer to it as CRS. Therefore, the terms Clusterware and CRS are used interchangeably in this document.



Reference Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 2-1 Reference Architecture Using the DS3400 Controller Module

HB

A(1)

LargeHost (1)

HB

A(1)

LargeHost (1)

1 2 3 4 5 6 7 8

9 10 11 12 13 14 15 16

FC Switch (1)16 Ports

86123-05

a ba’ DS3400ControllerRight Ctlr A Left Ctrl B

b’

1 2 3 4 5 6 7 8 9 10 11

12

LUN 6(ASM DG 1)

LUN 8(ASM DG 1)

LUN 7(ASM DG 1)Oracle Cluster

Registry(OCR)LUN 2

Voting DiskLUN 3

ASM DG 1

Test DatabaseORACLE_SID=testdb

Voting DiskLUN 4

Voting DiskLUN 5

Oracle ClusterRegistry(OCR)LUN 1

A B A B


Chapter 3
Configuring the Host and the Storage Subsystem
This chapter describes how to configure the host, how to configure the storage subsystem, and how to validate the setup of the host and storage subsystem.

Configuring the HostTo configure the host, it must have these components installed:

• The operating system

• The HBA drivers

• DS3000 Storage Manager

1 Before you select the HBAs, make sure that the HBA is supported by IBM. Also check the approved version of the HBA driver for the release of firmware on your storage subsystem. This step is critical for your configuration to function correctly and to be supported.This information is available from the IBM Interop Matrix, which is available at this web site:


2 Configure the HBA BIOS, driver parameters, and Windows registry parameters as specified by the HBA Settings document. You can obtain this document from your OEM storage vendor.

IMPORTANT When you use the MPIO driver for the IBM DS3000 series storage subsystem, you must set the value of the DisableLunRebalance parameter in the Windows Registry to 3. Setting this parameter prevents logical drive thrashing and the resulting I/O failures. This feature can be changed using the Windows regedit tool. The key can be found at: HKLM\SYSTEM\Services\Ds4dsm\Parameters\DisableLunRebalance.



Configuring the Host and the Storage Subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Configuring the Storage SubsystemTo decide what layout best fits your environment, review all of the storage options available to you by referring to the Oracle Clusterware installation manual. After you have decided how much space you need and what logical drives must be created and mapped in the DS3000 Storage Manager, you are ready to configure the storage.

1 Set up the host mappings for the shared logical drives by performing these steps:

a Create a host group for the Oracle cluster hosts. This example uses a host group named ClusterHosts.

b Add a host entry in the cluster host group for each host in the cluster.

c Create host ports for each host definition for each HBA port on each host.

2 Use the DS3000 Storage Manager to create logical drives on the IBM DS3000 series storage subsystem for the Oracle CRS and data files. Create separate logical drives for voting, registry, and ASM disk drives.

NOTE As each logical drive is created, assign it to the host group that you created for the cluster hosts.

IMPORTANT The sizing and placement of these logical drives depend on many variables in the target configuration. Consult with an experienced Oracle database administrator to plan the best layout.

Figure 2-1 on page 2-2 shows the storage layout used in the reference architecture. All logical drives shown were assigned to the ClusterHosts host group. These logical drives were shared between the two hosts in this cluster. None of these logical drives were mapped to the individual hosts

IMPORTANT The LUN numbering for the logical drives that each host sees must always start with 0 (zero) on at least one storage subsystem. In addition, LUN numbers must be sequential. Do not skip LUN numbers when mapping. If these conditions are not met, problems can occur with hosts not seeing LUNs.


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Checking the Setup of the Host and the Storage Subsystem

Checking the Setup of the Host and the Storage Subsystem

Check the storage subsystem access by checking that the hosts have access to the mapped logical drives.

The %System Root%\Program Files\Storage Manager\util\ SMdevices.bat command displays the mapped logical drives that the hosts recognizes.

Following the initial mapping of the logical drives, you can perform one of these actions:

• Rescan the devices in the Disk Management under Computer Management

• Run the hot_add utility from the same directory that contains the SMdevices.bat batch file.

• Restart the server. This option makes sure that all possible disk drive ID changes are captured.


Configuring the Host and the Storage Subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


Chapter 4
Oracle Setup and Installation
To install Oracle RAC remotely, enable either VNC or Terminal Service/Remote Desktop services. If you use Windows Remote Desktop, you must connect to the system console when you start your session by typing this command:

mstsc -v:<servername> -console

For details, see Oracle Clusterware and Oracle Real Application Clusters Installation Guide, 10g Release 2 (10.2) for Microsoft Windows.

Preparing for Installation1 Check the Oracle Metalink web site for the latest hardware and software

requirements by visiting this web site:

http://metalink.oracle.com

2 Follow the Oracle Clusterware installation manual for how to configure the system for installation. The procedures might include these steps:

■ Synchronizing the system clocks

■ Configuring networking on the hosts

■ Disabling Windows Media Sensing in the system registry

■ Making sure that remote access is available between the cluster hosts

■ Using the CVU utility to check that all of the requirements for successful installation were met

Enabling Windows Auto MountingBefore you create the logical drives, enable Windows auto mounting by typing these commands.

1 On each host, run this command:

c:\> diskpartDISKPART> automount enableAutomatic mounting of new volumes enabled


http://metalink.oracle.com

Oracle Setup and Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2 After all hosts have had automount enabled, restart all hosts.

IMPORTANT You must restart all hosts at this point.

Formatting the Disk DrivesFormat all of the disk drives as raw devices initially, no matter what storage option (ASM, OCFS, or raw devices) you choose.

1 Create raw devices.

a Right-click My Computer and select Manage >> Disk Management.

All of the disk drives from the storage subsystem appear as unformatted logical drives.

b Initialize all of the disk drives as basic disk drives.

c Right-click a disk drive and select New Partition.

d In the wizard, select Extended Partition and specify that it uses all of the space that is available for the new partition.

e Right-click the partition that you just created and select New Logical Drive.

f Select Do not assign a drive letter or path and Do not format this partition.

If you check the disk drive properties, the File System column shows RAW.

2 Repeat step 1 for all of the shared disk drives in your setup.

3 Restart all of the hosts after creating raw devices.

4 Check all of the hosts to see that the disk drives are discovered and recognized as raw devices after you restart the storage subsystem.

5 If some hosts have automatically assigned disk drive letters to the raw disk drives after the first restart, manually remove the disk drive letters.

NOTE Although Oracle recommends disabling write caching for disk drive devices, this action is not necessary or desirable when using IBM DS3000 series storage subsystems. IBM DS3000 series storage subsystems provide battery backup for the cache data. For best performance using the IBM DS3000 series storage subsystem, do not disable write caching on the storage subsystem or on the Windows hosts.


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Installing Oracle Clusterware

Installing Oracle ClusterwareAfter you make sure that the essential devices are formatted correctly, install Oracle Clusterware according to the instructions in the Oracle Clusterware installation manual.

NOTE Depending on the public IP address of your host, you might have to run the vipca application manually to configure the Oracle VIP interfaces. This process is because of a known issue with Oracle 10.2.0.1 (4437727).

NOTE After the installation is completed, check the Oracle Clusterware status by running the crs_stat -t command on either host.

Updating Oracle ClusterwareAfter you install the Oracle Clusterware, update it to the latest patch set. The reference architecture updated CRS version 10.2.0.1 to 10.2.0.3.

Installing the Oracle Database SoftwareAfter you make sure that the Oracle Clusterware installation was successful and the cluster is up and running, you can install the Oracle Database software. Follow the installation instructions in the Oracle Database installation manual.

Updating the Oracle Database SoftwareAfter the installation of the Oracle Database software, update to the latest patch set. The reference architecture updated version 10.2.0.1 to version 10.2.0.3.

Configuring ASM and Creating a Database with DBCA

Use the Database Configuration Assistant (DBCA) to configure ASM and to create clustered databases.

1 As you go through the DBCA utility to configure ASM, you must stamp the disk drives before they can be added to ASM disk groups.

2 After you have configured ASM and created at least one disk group, you can create a database using DBCA.

3 Select the ASM disk groups on which you want the database to reside.


Oracle Setup and Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4 Implement the backup and recovery options by using the Flash Recovery Area (FRA) option when the database is being created. Select one of these features, which are set up on a separate area on disk drive:

■ Flashback capability

■ An area for the archive logs

■ A second copy of the multiplexed redo log files

■ The RMAN backups

This procedure is highly recommended.

5 Make sure that the FRA area has sufficient space to hold all of the data for the data retention periods that are set by the database.

6 Make sure that the FRA area has sufficient space to hold the different types of backups and determine how many backups this area can hold.

7 Make sure that the FRA area is set up correctly to handle the performance of the database.

For further information on using the FRA, refer to the Oracle documentation.


Chapter 5
Conclusion
By following the guidelines in this document, a system administrator can properly configure and install Oracle RAC using an IBM DS3000 series storage subsystems as the primary storage. Careful planning and execution of the prerequisites helps to make sure of a successful installation. If the host setup and the storage subsystem setup are completed correctly, the installation runs smoothly without errors. Plan to use an IBM DS3000 series storage subsystem to take advantage of the performance, value, and premium features.

Contact InformationFor more information and sales office locations, visit the IBM web site at:

www.ibm.com


www.ibm.com

Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


Appendix A
Oracle RAC Troubleshooting
This appendix lists some known issues and their resolutions.

Oracle Clusterware Complains That “Nodes Are Not Clusterable”

This error message means that Oracle cannot copy files between cluster nodes correctly.

To resolve the problem, make sure that all of the hosts have the same username and the same password and are logging into the same domain. Also check that all of the network settings and connections are configured properly.

Services Fail to Start at the End of Oracle Clusterware Installation

At the end of Oracle Clusterware installation, the Configuration Assistant reports that services are in an incorrect pending state and failed to start.

To correct this problem, perform these steps.

1 Make sure that the DisableDHCPMediaSense=1 registry key is set properly and in the correct location.

2 Make sure that the bind order for the network interfaces is in the correct order.

3 Make sure that the Distributed Transaction Coordinator service is disabled.

Deploying Oracle Database RAC 10g on a Windows Server with a DS3000 System A-1© Copyright 2008, IBM Corporation

Oracle RAC Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A-2 Deploying Oracle Database RAC 10g on a Windows Server with a DS3000 System© Copyright 2008, IBM Corporation

Appendix B
References
Automatic Storage Management - Oracle Technical Library

http://www.oracle.com/technology/products/database/asm/index.html

For the IBM compatibility list, go to:


HBA Settings

The HBA Settings document is not available on an external link. Customers who want it must contact the storage vendor providing the storage subsystem.

Oracle Documentation Library

http://www.oracle.com/pls/db102/homepage

Oracle Clusterware and Oracle Real Application Clusters Installation Guide, 10g Release 2 (10.2) for Microsoft Windows

Oracle Database Installation Guide, 10g Release 2 (10.2) for Microsoft Windows (32-bit)

Deploying Oracle Database RAC 10g on a Windows Server with a DS3000 System B-1© Copyright 2008, IBM Corporation

http://www.oracle.com/technology/products/database/asm/index.html

http://www.oracle.com/pls/db102/homepage


References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

B-2 Deploying Oracle Database RAC 10g on a Windows Server with a DS3000 System© Copyright 2008, IBM Corporation

Trademarks and Special Notices © Copyright IBM Corporation 2008. All rights reserved.

References in this document to IBM products or services do not imply that IBM intends to make them available in every country.

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at www.ibm.com/legal/copytrade.shtml.

Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a trademark of Linus Torvalds in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

Information is provided "AS IS" without warranty of any kind.

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.

Information concerning non-IBM products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the supplier of those products.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

www.ibm.com/legal/copytrade.shtml