Dep loying Oracle 11gR2 RAC on Dell EqualLogic FS7500 · PDF fileloying Oracle 11gR2 RAC on Dell EqualLogic FS7500 NAS Storage ... to configure and deploy an Oracle 11g R2 Real Application

Deploying Oracle 11gR2 RAC on Dell EqualLogic FS7500 NAS Storage

A Dell Technical White Paper

Database Solutions Engineering

Dell Product Group

Zafar Mahmood

Deploying Oracle 11g R2 RAC on Dell EqualLogic FS7500 Storage

Page ii

THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL

ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR

IMPLIED WARRANTIES OF ANY KIND.

© 2011 Dell Inc. All rights reserved. Reproduction of this material in any manner whatsoever without

the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell.

Dell, the DELL logo, and the DELL badge, PowerConnect, and EqualLogic are trademarks of Dell Inc.

Symantec and the SYMANTEC logo are trademarks or registered trademarks of Symantec Corporation or

its affiliates in the US and other countries. Microsoft, Windows, Windows Server, and Active Directory

are either trademarks or registered trademarks of Microsoft Corporation in the United States and/or

other countries. Other trademarks and trade names may be used in this document to refer to either the

entities claiming the marks and names or their products. Dell Inc. disclaims any proprietary interest in

trademarks and trade names other than its own.

July 2011


Page iii

Acknowledgements

The author would like to thank the following people for their contribution generating the lab results for

this whitepaper:

Dennis Lattka

Storage Development Engineering

Dell Product Group


Page 1

Contents Introduction ................................................................................................................ 2

Configuring FS7500 Network and Shares for Oracle 11GR2...................................................... 4

Oracle Real Applications Cluster Setup and Configuration ...................................................... 5

Performance .............................................................................................................. 12

Summary .................................................................................................................. 14

Tables

Table 1. Scalable Attributes of FS7500 ............................................................................ 5

Table 2. Required RPMs for Oracle RAC on Enterprise Linux .................................................. 6

Figures

Figure 1. Oracle RAC nodes .......................................................................................... 3

Figure 2. Setting up the FS7500 iSCSI Network ................................................................... 4

Figure 3. NAS Shares for Oracle RAC ............................................................................... 4

Figure 4. Selecting the File System Option for Oracle .......................................................... 9

Figure 5. Selecting the NFS mount point for OCR disk .......................................................... 9

Figure 6. Selecting the NFS mount point for the voting disk ................................................. 10

Figure 7. Selecting the NFS mounts for shared Grid Infrastructure Home ................................. 10

Figure 8. Selecting the NFS mount for the shared oracle home ............................................. 11

Figure 9. Average Query Response Time Comparison ......................................................... 13

Figure 10. Transactions per second Comparison .............................................................. 13


Page 2

Introduction Dell‟s EqualLogic product line is based on a virtualized peer storage architecture and is a leader in the

iSCSI storage arena. The recently announced EqualLogic FS7500 expands the EqualLogic ecosystem with

scalable NAS capability and works with all PS series arrays to provide a scale-out unified storage

solution for midsize deployments. The NAS capability is enabled by Dell Fluid File System (FluidFS), a

high-performance, high availability clustered file system that is based on IP acquired from Exanet.

The current release of the EqualLogic FS7500 product consists of up to two scalable pairs of FS7500

nodes and comes with a custom Backup Power Supply (BPS). Customers can purchase the FS7500 with

an EqualLogic backend storage array for new NAS deployments or plug it seamlessly into an existing

EqualLogic infrastructure to add NAS capability to the iSCSI based block storage. The key features of

this appliance include a scale-out architecture, NFS v3.0, SMB 1.0, Snapshots, and Quotas; the major

differentiator for database deployments is its capability to cache most recently used blocks into the

RAM of the scalable pairs of nodes/controllers. Since the current release of the FS7500 supports up to

two pairs in a single cluster, the caching capacity also scales as additional nodes pairs are added. The

cache is synchronized between nodes/controllers to deliver optimal database performance for the NAS

category of storage products. The backup power supply protects node/controller cache for safe flush to

the backend storage array in case of accidental power outage.

The document is intended to serve as a reference for IT administrators and Database Administrators

who are interested in deploying Oracle RAC database on EqualLogic FS7500 Network Attached Storage.

The purpose is to describe the steps to configure and deploy an Oracle 11g R2 Real Application Clusters

(RAC) environment on EqualLogic FS7500 product using NFS shares. Oracle 11G R2 RAC requires shared

storage access from multiple database nodes, each running its own set of Oracle processes. The RAC

nodes communicate with each other over a high speed private interconnect for heart beat and

database block traffic.

NFS is an abbreviation of Network File System, a platform independent technology that allows shared

access to files stored on computers that runs on top of TCP/IP. In this Oracle RAC deployment scenario,

the EqualLogic FS7500 will provide NFS shared storage for a RAC installation. This document describes

the procedure to deploy a two node Oracle 11g R2 RAC on the EqualLogic FS7500. It should be used in

conjunction with the EqualLogic FS7500 User Guide to fully accomplish the task of initial setup of the

product, which is beyond the scope of this document.


Page 3

The following diagram shows a two node Oracle RAC solution consisting of EqualLogic iSCSI and FS7500

storage:

Figure 1. Oracle RAC nodes

Oracle recommends the following for deploying Oracle RAC on NAS:

The performance of Oracle software and databases stored on NAS devices is highly dependent on the performance of the network connection between the Oracle server and the NAS device. It is recommended that Oracle nodes have redundant network connections available to the FS7500 storage. Up to 4 network links may be bonded together for high availability and fault tolerance. Currently, the FS7500 does not support Oracle Direct NFS client for database connections using NFS. Connect the Oracle nodes to the NAS device using a bonded private dedicated network connection consisting of Gigabit Ethernet.

For single-system installations (as opposed to RAC installations), you must create a separate Oracle home directory for each installation. Run the software in this Oracle home directory only from the system that you used to install it.

For Oracle Real Application Clusters (RAC) installations, use a single Oracle home directory mounted from each node in the cluster known as shared Oracle home. You must mount this Oracle home directory on each node so that it has the same directory path on all nodes.


Page 4

Configuring FS7500 Network and Shares for Oracle 11GR2 The initial configuration of FS7500 consists of setting up the following networks and IP addresses:

The Interconnect Network is a private network used by the FS7500 controllers. For this range

choose a subnet range that is not used by any other devices in your infrastructure.

iSCSI Network through which the FS7500 nodes/controllers communicate internally with

EqualLogic storage array. The following diagram shows an example of such setup.

Client Network through which external clients such as Oracle RAC nodes connect to the storage shares

Figure 2. Setting up the FS7500 iSCSI Network

Refer to the EqualLogic FS7500 Users Guide for detailed information on performing the initial setup.

The overall network is capable of delivering 4 Gb/s from each FS7500 node/controller resulting in a

total network bandwidth of 8Gb/s or approximately 800MB/s. For OLTP or transaction processing

solutions, the availability of 24GB of memory on each FS7500 node/controller can result in the

reduction of average query response time since most recently accessed database blocks are cached

inside the FS7500 RAM. For a 2 node/controller configuration, the available RAM is 48GB and can be

scaled to 96 GB for a configuration consisting of two pairs resulting in near linear scalability in

database performance in terms of response time.

Configure the shared storage for RAC using FS7500 NAS volumes:

Figure 3. NAS Shares for Oracle RAC


Page 5

For the purposes of this white paper, we configured the following NAS file systems on FS7500:

Oravol1: 10GB (For Oracle Grid Infrastructure files such as voting disk)

Oravol2: 20GB (for Oracle grid infrastructure shared binaries)

Oravol3: 20GB (For shared Oracle home binaries)

Oravol4: 400Gb (For Oracle datafiles)

Using UNIX security style is recommended for these volumes and also each volume should have an NFS

export. For enhanced security purposes, the NFS exports access should be restricted only to the Oracle

RAC nodes using the IP address restriction.

The following table describes the maximum capability comparison between 2 node/controller and 4

node/controller FS7500 configurations:

Table 1. Scalable Attributes of FS7500

Attribute 2-nodes/controllers 4-nodes/controllers

Max system size 509 TB ½ PB (509 TB)

(Max 8 EQL arrays)

Max file size 4 TB 4TB

Max size of Virtual Volumes 256TB 512TB

Max # virtual volumes 1024 2048

Max # snapshots 10,000 10,000

Cache size per cluster

(with 4 quad-core CPUs)

48GB (24GB each) 96GB (24GB each)

File name length 255 bytes 255 bytes

Max CIFS shares 1,024 2,048

Max CIFS client connections 1500 3000

Max quota rules per cluster 65,536 262,144

Once the FS7500 has been configured and is available on the network, follow the rest of the document

to install, and configure Oracle RAC and connect to the FS7500 storage array as described in the

following sections.

Oracle Real Applications Cluster Setup and Configuration 1. Perform a basic install of the Oracle Enterprise Linux operating system

2. Install the Oracle Validated RPM which can be found on the Oracle Enterprise Linux CD‟s.

Installing the Oracle validated RPM will result in an Operating System install which contains

all the prerequisite RPM‟s necessary to install and configure an Oracle 11gR2 RAC or single

node database.


Page 6

At the minimum, the following RPMs are required for a successful Oracle 11gR2 setup on OEL 5.5:

Table 2. Required RPMs for Oracle RAC on Enterprise Linux

Name Version Release

compat-db 4.2.52 5.1

compat-gcc-34 3.4.6 4

compat-gcc-34 3.4.6 4

compat-gcc-34-c++ 3.4.6 4

compat-libstdc++-33 3.2.3 61

compat-libstdc++-33 3.2.3 61

elfutils-libelf-devel 0.137 3.el5

gcc-c++ 4.1.2 48.el5

gdb 7.0.1 23.el5_5.1

libXp 1.0.0 8.1.el5

libaio-devel 0.3.106 5

libaio-devel 0.3.106 5

libstdc++-devel 4.1.2 48.el5

sysstat 7.0.2 3.el5

unixODBC 2.2.11 7.1

unixODBC 2.2.11 7.1

unixODBC-devel 2.2.11 7.1

unixODBC-devel 2.2.11 7.1

elfutils-libelf-devel-static 0.137 3.el5

3. Next, create the required Oracle users and groups to host both the Grid Infrastructure

home for an Oracle Real Applications Clusters deployment and the Oracle home for the

Oracle software

groupadd –g 1000 oinstall groupadd –g 1200 dba useradd –u 1100 –g oinstall –G dba oracle

4. For an Oracle 11gR2 deployment, which now comes with its own Cluster Time

Synchronization Service (CTSS), disable the OS NTP services.

service ntpd stop chkconfig ntpd off mv /etc/ntp.conf /etc/ntp.conf.orig rm /var/run/ntpd.pid


Page 7

5. Make sure that both the Grid Infrastructure user and Oracle users have sufficient resource

limits defined to be able to spawn Oracle processes.

a. Edit /etc/security/limits.conf with the following settings:

oracle soft nofile 131072

oracle hard nofile 131072

oracle soft nproc 131072

oracle hard nproc 131072

oracle soft core unlimited

oracle hard core unlimited

oracle soft memlock 50000000

oracle hard memlock 50000000

b. On each node, add or edit the following line in the /etc/pam.d/login file, if it does

not already exist:

session required pam_limits.so

c. Edit the /etc/profile file and add the following entry:

if [ $USER = "oracle" ] then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fi umask 022 fi

NOTE: Oracle 11gR2 can be configured with its new Grid Naming service (GNS) feature.

Although, This is not a requirement but it is useful when it comes to name resolution for

numerous RAC service names such as SCAN listeners. It is a good idea at this point to

configure the GNS dependencies. For detailed information on this topic, please refer to

Dell Tested and Validated configurations documentation available at

www.dell.com/oracle. For the purpose of this white paper we used GNS and configured the

domain name server to resolve the GNS VIP address, which will be used for our

configuration later.

6. On both node1 and node2 create the directories on which NAS volumes are mounted and

Oracle software installed.

mkdir -p /opt/app/11.2.0/grid mkdir -p /opt/app/oracle/product/11.2.0/db_1 mkdir -p /opt/shared_config mkdir -p /opt/oradata chown -R oracle:oinstall /opt/app /opt/app/oracle /opt/oradata /opt/shared_config chmod -R 775 /opt/app /opt/app/oracle /opt/oradata /opt/shared_config

7. Mount the NFS exports on database server nodes, which will be part of the Oracle 11GR2

cluster, and then do the following:

http://www.dell.com/oracle


Page 8

Add the following lines to the "/etc/fstab" file:

172.16.150.160:/oravol1 /opt/shared_config nfs

rw,bg,hard,nointr,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0 0 0

172.16.150.160:/oravol2 /opt/app/11.2.0/grid nfs


172.16.150.160:/oravol3 /opt/app/oracle/product/11.2.0/db_1 nfs


172.16.150.160:/oravol4 /opt/oradata nfs


NOTE: In the above configuration:

172.16.150.160 is the virtual IP (VIP) address of the FS700 client network assigned during

configuration of the NAS server.

Mount NFS volumes on RAC nodes:

mount /opt/shared_config mount /opt/app/11.2.0/grid mount /opt/app/oracle/product/11.2.0/db_1 mount /opt/oradata

Make sure the permissions on the shared directories are correct:

chown -R oracle:oinstall /opt/shared_config chown -R oracle:oinstall /opt/app/11.2.0/grid chown -R oracle:oinstall /opt/app/oracle/product/11.2.0/db_1 chown -R oracle:oinstall /opt/oradata

8. Install the shared Oracle Grid Infrastructure Home on FS7500 volume:

NOTE: Installing a shared Oracle Grid Infrastructure home on NFS is optional. In addition,

you can choose to install Oracle binaries locally on cluster nodes. Installing on a shared NFS

volume provides the benefits of one step patching and software configuration

management. The side effect is that the shared NFS volume becomes the single point of

failure for the entire cluster.

Start Oracle Universal Installer to install the Oracle 11GR2 Grid Infrastructure on all cluster nodes. The installation is run form one of the nodes only as user „grid‟:

./runInstaller

Follow the typical 11gR2 installation prompts while making sure of the following:

When prompted: Select the "Shared File System" option, then click the Next button


Page 9

Figure 4. Selecting the File System Option for Oracle

Select the required level of redundancy and enter the OCR File Locations under /opt/shared_config/ocr_configuration, then click the Next button:

Figure 5. Selecting the NFS mount point for OCR disk


Page 10

Select the required level of redundancy and enter the Voting Disk File Locations under /opt/shared_config/voting_disk, then click the "Next" button.

Figure 6. Selecting the NFS mount point for the voting disk

On the installation location page, specify "/opt/app/oracle" as the Oracle Base and "/opt/app/11.2.0/grid" as the software location, then click the Next button

Figure 7. Selecting the NFS mounts for shared Grid Infrastructure Home


Page 11

Follow the rest of the prompts for a typical 11gR2 grid infrastructure installation until completed. This results in successful installation and configuration of GID home on the shared NAS volume.

12. Installing the Shared Oracle Home on FS7500 volume:

NOTE: Installing a shared Oracle home on NFS is optional. In addition, choose to install Oracle

binaries locally on cluster nodes. Installing on a shared NFS volume provides the benefits of one

step patching and software configuration management benefits. The side effect is that the

shared NFS volume becomes the single point of failure for the entire cluster.

As user „oracle‟ start the Oracle Universal Installer:

./runInstaller

Follow typical Oracle 11gR2 database installation. Make sure that after choosing “typical install” option; choose the following directory locations for Oracle shared home binary installation.

Enter "/opt/app/oracle/product/11.2.0/db_1" for the software location. The storage type should be set to File System with the file location set to "/opt/oradata." Enter the appropriate passwords and database name, in this case RAC.localdomain as shown on the following screenshot:

Figure 8. Selecting the NFS mount for the shared oracle home

Complete the installation by following the typical Oracle 11gR2 installation instructions and complete the binaries installation on the NFS shared home as well as the creation of a seed database.

13. Check the Status of RAC database:

srvctl config database -d rac Database unique name: rac


Page 12

Database name: rac Oracle home: /opt/app/oracle/product/11.2.0/db_1 Oracle user: oracle Spfile: /opt/oradata/rac/spfilerac.ora Domain: localdomain Start options: open Stop options: immediate Database role: PRIMARY Management policy: AUTOMATIC Server pools: rac Database instances: rac1,rac2 Disk Groups: Services: Database is administrator managed $ $ srvctl status database -d rac Instance rac1 is running on node rac1 Instance rac2 is running on node rac2

Performance Traditional NFS servers are usually deployed for the purposes for testing and developing databases due

to their inherent performance limitations. Performance limitations arise due to additional overhead of

managing the network file systems and their lack of scalability options. The Dell EqualLogic FS7500

storage system is built with the idea of scalability and the added feature of offloading the file system

management to the clustered file system management nodes/controllers with synchronized cache for

the database volumes mounted by the database server nodes. Dell lab testing has shown that a single

FS7500 node pair performs at par with a block storage solution such as an EqualLogic iSCSI system with

same number of disks as shown in the figure 9. The diagram below shows that an OLTP workload

comprising of 900 concurrent users, the average query response time (which is one of the most

important performance criteria for online transaction processing database systems) stays below one

second and on par with an iSCSI block storage solution. For majority of Online Transaction Processing

workloads, a query response time of upto two seconds is acceptable, which means that we have

headroom for further user load without compromising the service level agreement (SLA) of a typical

OLTP system.

Figure 10 shows another comparison from the perspective of transactions per second. From figure 10

one can observe that as we increase the user load on the database server nodes, both iSCSI and FS7500

systems exhibit comparable performance levels up to a 900 user load after. We stopped increasing the

user load beyond 900 to stay well within an average query response time of one second.

From these test results, you can conclude that the minimum configuration of the FS7500 storage

system consisting of a single node pair can provide comparable performance to a block storage solution

with same number of spindles and network IO ports. Also, the solution is scalable such that you are not

limited to this performance level. You can add additional FS7500 nodes/controllers in pairs to achieve

the required level of performance.

NOTE: The current release only supports up to two FS7500 node pairs; this is expected to increase in

subsequent releases


Page 13

Figure 9. Average Query Response Time Comparison

Figure 10. Transactions per second Comparison


Page 14

NOTE: The results we have provided are intended only for the purpose of comparison between two

solutions consisting of specific configurations in a lab environment. The results do not portray the

maximum capabilities of any system, database software, or storage.

Summary Over the years, the database solution performance bottleneck has been shifting more and more

towards the backend storage as servers are getting increasingly powerful. Network attached storage,

although easy to deploy and configure, has traditionally been considered as the least favored storage

option for databases due to its additional overheads. For Network Attached Storage, the performance

of Oracle software and databases depends not only on the performance of the network connection

between the Oracle server and the NAS device but also on how scalable the NAS device is as the

database load increases.

The EqualLogic FS7500 addresses both of these bottlenecks. The minimum configuration consists of two

NAS server nodes/controllers and an EqualLogic backend storage array having eight gigabit Ethernet

connections for the database nodes. The solution scales with the same ratio as additional pairs of NAS

nodes/controllers and backend storage arrays are added. Also, the solution delivers database block

caching capability which scales as additional FS7500 node pairs are added. The lab test results also

indicated that for an online transaction processing database system, the performance of FS7500 based

Oracle solution meets the criteria of acceptable performance requirements and is comparable to block

based iSCSI storage.

Documents

Dep loying Oracle 11gR2 RAC on Dell EqualLogic FS7500 · PDF fileloying Oracle 11gR2 RAC on Dell EqualLogic FS7500 NAS Storage ... to configure and deploy an Oracle 11g R2 Real Application