RACGuides_Rac11gR1OnAIX

Table of ContentsRac11gR1OnAIX................................................................................................................................................1

1. Introduction.........................................................................................................................................2 1.1. What you need to know......................................................................................................2

1.1.1. Software required for install...............................................................................2 1.1.2. Processor Model.................................................................................................2

1.2. Installation Steps...........................................................................................................................................2 1.3. Schematic......................................................................................................................................................3

1.3.1. Hardware/software configuration BEFORE Oracle software install............................................3 1.3.2. Hardware/software configuration AFTER Oracle software install..............................................4

1.4. Installation Method.......................................................................................................................................4 2. Prepare the cluster nodes for Oracle RAC.......................................................................................................5

2.1. Required Hardware..........................................................................................................................5 2.2. User Accounts..................................................................................................................................5

2.2.1. Creating the OSDBA (DBA) Group................................................................................5 2.2.2. Creating an OSOPER Group (Optional)..........................................................................6 2.2.3. Creating an OSASM Group.............................................................................................6 2.2.4. Creating an OINSTALL Group.......................................................................................6 2.2.5. Determining Whether an Oracle Software Owner User Exists.......................................7 2.2.6. Creating Oracle Software Owner Users...........................................................................7 2.2.7. Modifying an Oracle Software Owner User....................................................................8

2.3. Networking...................................................................................................................................................8 2.3.1. Define Networks layout, Public, Virtual and Private Hostnames.................................................8 2.3.2. Network Ping Tests.....................................................................................................................10 2.3.3. Network Adapter configuration..................................................................................................10

2.4. Configuring SSH on Cluster Member Nodes.............................................................................................11 2.4.1. Checking Existing SSH Configuration on the System...............................................................11 2.4.2. To configure SSH, Create RSA Keys On Each Node using the following steps:......................11 2.4.3. Add All Keys to a Common authorized_keys File.....................................................................12 2.4.4. Enabling SSH User Equivalency on Cluster Member Nodes.....................................................13 2.4.5. Setting Display and X11 Forwarding Configuration..................................................................14

2.5. Time Synchronization, xntpd, Mandatory..................................................................................................15 2.6. Tuning the AIX Environment.....................................................................................................................16

2.6.1. Tuning Virtual Memory Manager (VMM).................................................................................16 2.6.2. Increasing System Block Size Allocation...................................................................................16 2.6.3. Configuring Shell Limits............................................................................................................16 2.6.4. Configuring User Process Parameters........................................................................................17 2.6.5. Configuring Network Tuning Parameters...................................................................................17

2.7. crs environment setup.................................................................................................................................19 2.8. Filesets, AIX APAR and other Operating System Fixes............................................................................20

2.8.1. Fixes............................................................................................................................................20 2.8.2. Filesets........................................................................................................................................20

2.9. Stage the Oracle Software...........................................................................................................................21 2.10. Cluster Verification Utility stage check....................................................................................................21 3. *Prepare the shared storage for Oracle RAC.................................................................................................33

3.1. Creating Storage for the OCR and Voting disks............................................................................34 3.1.1. How to Identify if a LUN is used or not ?.....................................................................34 3.1.2. Register LUN's at AIX level..........................................................................................36 3.1.3. Identify LUN's and corresponding hdisk on each node.................................................37

3.1.3.1. Removing reserve lock policy on hdisks from each node.............................40 3.1.3.2. Identify Major and Minor number of hdisk on each node.............................41 3.1.3.3. Create Unique Virtual Device to access the same LUN from each node......42 3.1.3.4. Set Ownership / Permissions on Virtual Devices..........................................43 3.1.3.5. Formating the virtual devices (zeroing).........................................................44

3.2. ASM disks...................................................................................................................................................45 3.2.1. *About ASM Instances parameters files, pfile or spfile ?..........................................................45

i

Table of ContentsRac11gR1OnAIX 3.2.2.Recommendations, hints and tips................................................................................................46

4. Oracle Clusterware Installation and Configuration.......................................................................................47 4.1. CVU Pre Oracle Clusterware install check....................................................................................48 4.2.Oracle Clusterware Install...............................................................................................................58

4.2.1. Installer screenshots.......................................................................................................59 4.3.Post Installation operations..........................................................................................................................72

4.3.1.Update the Clusterware unix user .profile...................................................................................72 4.3.2.Cluster Ready Services Health Check.........................................................................................72

5. Oracle Clusterware patching..........................................................................................................................74 5.1. Oracle Clusterware patching complete..........................................................................................81

6. Oracle ASM Home Software Install..............................................................................................................81 6.1. CVU check before creating ASM instance....................................................................................82 6.2. Oracle ASM Home Software Install..............................................................................................82 6.3. Post Install Actions........................................................................................................................92

7. Oracle ASM Software Home Patching..........................................................................................................93 7.1. Start the OUI for Oracle ASM Software Home Patching..............................................................93

8. Oracle RAC Database Listener Creation.....................................................................................................100 8.1. Create Node specific network listeners........................................................................................100 8.2. Verify the Listener resources are online......................................................................................106 8.3. Completed Node specific network listeners creation...................................................................106

9. Oracle ASM Instance and diskgroup Creation............................................................................................107 9.1. Verify ASM instances are online.................................................................................................114 9.2. Completed creation of ASM Instance and add of the DATA_DG diskgroup.............................114

10. Oracle RAC Database Home Software Install...........................................................................................114 10.1. CVU check -pre Database Instance Install................................................................................114 10.2. Database Software Install..........................................................................................................114 10.3.Symbolic links creation for listener.ora, tnsnames.ora and sqlnet.ora........................................126 10.4.Update the RDBMS unix user .profile........................................................................................127

11. Oracle RAC Software Home Patching......................................................................................................127 11.1. Using the installer to patch the Oracle RAC Software Home...................................................128

12. Oracle RAC Database Creation.................................................................................................................135 12.1. use dbca to create the RAC database.........................................................................................135

ii

Rac11gR1OnAIX

Rac11gR1OnAIX 1

1. Introduction

1.1. What you need to know

For those of you starting an Oracle RAC install for the first time, all this will be new. For those of you withOracle 10g RAC on AIX experience, there are just a few changes.

Disk requirements for Oracle Clusterware devices

The storage requirement for the OCR & vote devices used by Oracle Clusterware has been increased to 280MB

(256 MB, plus an additional buffer to allow for variation in space allocation).

Better support for IP address

The previous release had some restrictions that could be easily worked around, with regard to IP addressesfrom the non-routable range.

These restrictions have been removed in the Oracle Clusterware 11g version and the workarounds are nolonger required.

1.1.1. Software required for install

AIX Base Oracle Database 11g Release 1 DVD or you can download the software from the OracleTechnology Network at this link:

•

Oracle Database 11g for AIX Latest patches are available on Metalink•

This release is certified for the following distributions:

AIX (6.1) AIX (5.3)

Do not mix Operating Systems between nodes in a cluster. For example, do not configure a cluster with AIX5.3 on one node and AIX 6.1 on another node.

1.1.2. Processor Model

All nodes in an Oracle RAC Cluster must have :

The same processor Technology (Power)•

Any IBM System p Servers :

Power 4,5,5+,6 ...• From -1 to 64-core, 1 to 64 way• From p505 to p595, inclusing JS21 and JS22•

1.2. Installation Steps

To summarise the install steps are:

Preparation•

1. Introduction 2

http://www.oracle.com/technology/software/products/database/oracle11g/111060_aixsoft.html

http://metalink.oracle.com

Pre-reqs. to make sure the cluster is setup OK.♦ Stage all the software on one node, typically Node1♦

Establish Oracle Clusterware

Install the Oracle Clusterware (using the push mechanism to install on the other nodes in thecluster)

♦

Patch the Clusterware to the latest patchset♦

•

Establish ASM

Install an Oracle Software Home for ASM♦ Patch the ASM Home to the latest patchset♦ Create the Listeners♦ Create the ASM Instances and initial Disk Group♦

•

Establish RAC Database

Install an Oracle Software Home for RAC Database♦ Patch the RAC Database Home to the latest patchset♦ Create the RAC Database Instances♦

•

1.3. Schematic

The following is a schematic of the software & hardware layout of a 2node RAC cluster. As explained in thisdocument the actual number of LUN’s required will vary depending on your mirroring requirements.

1.3.1. Hardware/software configuration BEFORE Oracle software install

1.2. Installation Steps 3

1.3.2. Hardware/software configuration AFTER Oracle software install

Notice that there are a number of LUNs present to both machines from the shared storage.

5 of 290MB LUNs

2 are used for the OCR devices (/dev/sdb1,/dev/sdc1)♦ 3 are used for the vote devices (/dev/sdb5,/dev/sdc5, /dev/sdd5)♦

•

More LUNs for ASM storage and the ASM pfile.•

Also notice there is a dedicated switch on the cluster interconnect.

1.4. Installation Method

This document details one method of installing a 2-node Oracle Database 11g RAC cluster on AIX:

The Oracle Clusterware, ASM & Oracle Database Homes binaries are established on the local disk of each ofthe RAC nodes. The files required by Oracle Clusterware are on shared hdisks (explained later). The databasedata-files are on ASM. The ASM disks are also shared hdisks.

The local Homes for Oracle Clusterware, ASM and RAC take up a total of 6.5GB on each nodes local diskfilesystem.

It should be noted that there are other installation configurations. e.g. Files on GPFS, HACMP managed RawLogical Volumes or a NAS Device. This document does not cover these methods.

1.3.2. Hardware/software configuration AFTER Oracle software install 4

2. Prepare the cluster nodes for Oracle RACThis page includes hidden sections, use the and image for each section to show/hide the section

or you can 'Show' or 'Hide' all by selecting here : Show all Hide all

This installation routine presumes that you have a 2-node AIX cluster. There are a number of items thatrequire checking before the install commences. Getting this bit right will enhance your install experience.

User Accounts1. Networking2. Time Sync3. Stage the Oracle Software4. Run CVU5.

It is essential that these items are checked and are configured correctly before the install commences.

2.1. Required Hardware

All participants on an Oracle RAC Cluster must have :

The same processor Technology (Power)•

Any IBM System p Servers :

Power 4,5,5+,6 ...• From -1 to 64-core, 1 to 64 way• From p505 to p595, inclusing JS21 and JS22•

2.2. User Accounts

These commands must be issued as the 'root' user on all nodes. The user id and the group id must be the sameon all nodes. This is the user equivalence that the cluster verify utility will check later.

2.2.1. Creating the OSDBA (DBA) Group

You must create an OSDBA group in the following circumstances:

An OSDBA group does not exist, for example, if this is the first installation of Oracle Databasesoftware on the system

•

An OSDBA group exists, but you want to give a different group of operating system users databaseadministrative privileges in a new Oracle installation

•

To determine whether the OSDBA group exists, enter the following command:

# grep OSDBA_group_name /etc/group

If the OSDBA group does not exist or if you require a new OSDBA group, then create it as follows.In the following command, use the group name dba unless a group with that name already exists.

mkgroup -'A' id='502' adms='root' dba

Note: The default OSDBA group name is dba.

2. Prepare the cluster nodes for Oracle RAC 5

2.2.2. Creating an OSOPER Group (Optional)

Create an OSOPER group only if you want to identify a group of operating system users with a limited set ofdatabase administrative privileges(SYSOPER operator privileges). For most installations, it is sufficient to create only the OSDBA group.If you want to use an OSOPER group, then you must create it in the following circumstances:

If an OSOPER group does not exist, for example, if this is the first installation of Oracle Databasesoftware on the system

•

If an OSOPER group exists, but you want to give a different group of operating system users databaseoperator privileges in a new Oracle installation

•

If you require a new OSOPER group, then create it as follows. In the following command, use the group nameoper unless a group with that name already exists.

# mkgroup -'A' id='503' adms='root' oper

2.2.3. Creating an OSASM Group

Create an OSASM group only if you want SYSASM as a system privilege that enables the separation of theSYSDBAdatabase administration privilege from the ASM storage administration privilege.If you want to use an OSASM group, then you must create it in the following circumstances:

If an OSASM group does not exist, for example, if this is the first installation of Oracle Databasesoftware on the system

•

If an OSASM group exists, but you want to give a different group of operating system users databaseoperator privileges in a new Oracle installation

•

To determine whether the OSASM group exists, enter the following command:

# grep OSASM_group_name /etc/group

If the OSASM group does not exist or if you require a new OSASM group, then create it as follows.In the following command, use the group name asadmin unless a group with that name already exists.

# mkgroup -'A' id='504' adms='root' asm

2.2.4. Creating an OINSTALL Group

You must create an OINSTALL group in the following circumstances:

An OINSTALL group does not exist, for example, if this is the first installation of Oracle Databasesoftware on the system

•

An OINSTALL group exists, but you want to give a different group of operating system usersdatabase administrative privileges in a new Oracle installation

•

To determine whether the OINSTALL group exists, enter the following command:

# grep OINSTALL_group_name /etc/group

If the OINSTALL group does not exist or if you require a new OINSTALL group, then create it as follows.In the following command, use the group name dba unless a group with that name already exists.

# mkgroup -'A' id='500' adms='root' oinstall

2.2.2. Creating an OSOPER Group (Optional) 6

Note: The default OINSTALL group name is oinstall.

2.2.5. Determining Whether an Oracle Software Owner User Exists

For each user in 'crs, asm and rdbms'

To determine whether the appropriate Oracle software owners exist, enter the following command:

For example for a crs owner called 'crs' and an asm owner called 'asm' and an 'rdbms' user called oracle

# id crs# id asm# id oracle

If, say, the oracle user exists, then the output from this command is similar to the following:

uid=440(oracle) gid=200(oinstall) groups=201(dba),202(oper)

If the user(s) exist, then determine whether you want to use the existing user or create another oracle user.If you want to use the existing user, then ensure that the user’s primary group is the Oracle Inventory groupandthat it is a member of the appropriate OSDBA and OSOPER groups.

2.2.6. Creating Oracle Software Owner Users

In the following procedure, use the user name appropriate to the software component unless a user with thatname already exists. If the Oracle software owner user does not exist or if you require a new Oracle softwareowner user, then create it as follows:

1.To create the oracle user, enter a command similar to the following:

# mkuser id='500' pgrp='oinstall' groups='crs,dba,[oper],[asm],[asmdba],[staff]' home='/home/oracle' oracle

In this command:

The pgrp option specifies the primary group, which must be the Oracle Inventory group, for exampleoinstall

•

The groups option specifies the secondary groups, which must include the OSDBA group and ifrequired, the OSOPER group.dba or dba,oper

•

2.Set the password of the oracle user:

# passwd oracle

Note: The passwd command uses only the first eight characters of your password for local and NISpasswords.Only 7-bit characters are supported in passwords. For this reason, National Language Support (NLS) codepoints are not allowed in passwords.

2.2.4. Creating an OINSTALL Group 7

2.2.7. Modifying an Oracle Software Owner User

If the oracle user exists, but its primary group is not oinstall or it is not a member of the appropriate OSDBAor OSOPER groups, then enter a command similar to the following to modify it. Specify the primary groupusing the -g option and any required secondary group using the -G option:

# chuser pgrp=oinstall groups=dba[,oper] oracle</pre>

Note: Do not use the chuser command if you have a Network Information Service (NIS) database installed onyour system.

For each oracle software component owner

Ensure that the Oracle Clusterware oracle user has the capabilities CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM, and CAP_PROPAGATE.

To check existing capabilities, enter the following command as root; in this example,the Oracle Clusterwareoracle user is crs:

# lsuser -a capabilities crs

To add capabilities, enter a command similar to the following:

# chuser capabilities=CAP_NUMA_ATTACH CAP_BYPASS_RAC_VMM CAP_PROPAGATE crs

2.3. Networking

2.3.1. Define Networks layout, Public, Virtual and Private Hostnames

We need 2 different networks, with associated network interfaces on each node participating to the RACcluster :

Public Network to be used as Users network or reserved network for Application and Databasesservers.

•

Private Network to be used as a Reserved network for Oracle clusterware, and RAC.•

For each node, We have to define :

Public hostname and corresponding IP address on the “Public Network”• Virtual hostname and corresponding IP address, also on the “Public Network”. IP address must be ofsame range then Public hostname, but not used on the network prior to Oracle clusterware installation.

•

Private hostname and corresponding IP address on the “Private Network”. You will need a total of 3IP addresses per node:

•

The public IP address, which should be recorded in hosts file on each node and, if available, DNS.This IP Address should be bound to the public adapter before starting the install. It should be a static,not DHCP, address

•

The private IP address, which should be from a different subnet than the public IP address. Thisaddress does not require registering in DNS but you should place an entry in the hosts file on eachnode. This IP Address should be bound to the private adapter before starting the install. It should be astatic, not DHCP, address

•

2.2.7. Modifying an Oracle Software Owner User 8

A VIP address, which should be from the same subnet as the public IP address and should berecorded in DNS and the hosts file on each node. This IP Address should NOT be bound to the publicadapter before starting the install. Oracle Clusterware is responsible for binding this address. It shouldbe a static, not DHCP, address

•

If you do not have a DNS server then make sure both the public, private and the VIP addresses are enteredinto all the relevant hosts files that are normally found in the /etc/hosts file.

Let identify hostnames, and network interfaces …

Please make a table as follows to have a clear view of your RAC network architecture : PUBLIC NODENAME MUST BE NAME RETURNED BY “hostname”

Public Hostname VIP Hostname (Virtual IP) PrivateInterconect

en0 Public Network en1 PrivateNetwork

Node Name IP Node Name IP Node Name IPnode1 10.3.25.81 node1-rac 10.10.25.81 node1-vip 10.3.25.181node2 10.3.25.82 node2-rac 10.10.25.82 node2-vip 10.3.25.182

MANDATORY : Network interfaces must have same name, same subnet and same usage !!!

You should use ifconfig on each node to confirm the correct IP addresses are bound to the correct adapters

Show Networking setup information Show Networking setup information|

On node1

{node1:root}/ # ifconfig -aen0: flags=1e080863,80<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD,CHAIN> inet 10.3.25.81 netmask 0xffffff00 broadcast 10.3.25.255 inet 10.3.25.181 netmask 0xffffff00 broadcast 10.3.25.255 tcp_sendspace 131072 tcp_recvspace 65536en1: flags=1e080863,80<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD,CHAIN> inet 10.10.25.81 netmask 0xffffff00 broadcast 10.10.25.255 tcp_sendspace 131072 tcp_recvspace 65536en2: flags=1e080863,80<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD,CHAIN> inet 20.20.25.81 netmask 0xffffff00 broadcast 20.20.25.255 tcp_sendspace 131072 tcp_recvspace 65536lo0: flags=e08084b<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT> inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255 inet6 ::1/0 tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1{node1:root}/ #

On node 2 :{node2:root}/ # ifconfig -aen0: flags=1e080863,80<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD,CHAIN> inet 10.3.25.82 netmask 0xffffff00 broadcast 10.3.25.255 inet 10.3.25.182 netmask 0xffffff00 broadcast 10.3.25.255 tcp_sendspace 131072 tcp_recvspace 65536en1: flags=1e080863,80<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD,CHAIN> inet 10.10.25.82 netmask 0xffffff00 broadcast 10.10.25.255

2.3.1. Define Networks layout, Public, Virtual and Private Hostnames 9

http://globaldc.oracle.com/perl/twiki/genpdf/RACGuides/Rac11gR1OnAIX?skin=pattern;cover=print;sortcol=0;table=3;up=0#sorted_table




tcp_sendspace 131072 tcp_recvspace 65536en2: flags=1e080863,80<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD,CHAIN> inet 20.20.25.82 netmask 0xffffff00 broadcast 20.20.25.255 tcp_sendspace 131072 tcp_recvspace 65536lo0: flags=e08084b<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT> inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255 inet6 ::1/0 tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1

A sample /etc/hosts file

# Do not remove the following line, or various programs # that require network functionality will fail.127.0.0.1 localhost.localdomain localhost10.3.25.81 node1 10.3.25.82 node210.10.25.81 node1-rac10.10.25.82 node2-rac10.3.25.181 node1-vip10.3.25.182 node2-vip

It is important to confirm that networking is configured correctly before commencing install.

2.3.2. Network Ping Tests

There are a series of 'ping' tests that should be completed, and then the network adapter binding order shouldbe checked. You should ensure that the public IP addresses resolve correctly and that the private addresses areof the form 'nodename-priv' and resolve on both nodes via the hosts file.

Public Ping testPinging node1 from node1 should return node1's public IP addressPinging node2 from node1 should return node2's public IP addressPinging node1 from node2 should return node1's public IP addressPinging node2 from node2 should return node2's public IP address

•

Private Ping testPinging node1 private from node1 should return node1's private IP addressPinging node2 private from node1 should return node2's private IP addressPinging node1 private from node2 should return node1's private IP addressPinging node2 private from node2 should return node2's private IP address

•

VIP Ping test Pinging the VIP address at this point should fail. VIPs will be activated at the end of theOracle Clusterware install.

•

If any of the above tests fail you should fix name/address resolution by updating the DNS or local hosts fileson each node before continuing with the installation.

2.3.3. Network Adapter configuration

If your network adapters allow configuration you should make sure that they are configured for Full Duplexand at the 'fastest' speed consistent among nodes. They should not be left to 'auto-negotiate'.

2.3.2. Network Ping Tests 10

2.4. Configuring SSH on Cluster Member Nodes

To configure SSH, you must first create RSA or DSA keys on each cluster node, and then copy all the keysgenerated on all cluster node members into an authorized keys file that is identical on each node. Note that theSSH files must be readable only by root and by the software installation user (oracle, crs, asm), as SSHignores a private key file if it is accessible by others. When this is done, then start the SSH agent to load keysinto memory. In the examples that follow, the RSA key is used. You must configure SSH separately for eachOracle software installation owner that you intend to use for installation.

2.4.1. Checking Existing SSH Configuration on the System

To determine if SSH is running, enter the following command:

$ ps -ef | grep sshd

If SSH is running, then the response to this command is one or more process ID numbers. In the homedirectory of the software owner that you want to use for the installation (crs, oracle), use the command ls -al toensure that the .ssh directory is owned and writable only by the user.

You need either an RSA or a DSA key for the SSH protocol. RSA is used with the SSH 1.5 protocol, whileDSA is the default for the SSH 2.0 protocol. With OpenSSH, you can use either RSA or DSA. Theinstructions that follow are for SSH1. If you have an SSH2 installation, and you cannot use SSH1, then referto your SSH distribution documentation to configure SSH1 compatibility or configure SSH2 with DSA.

2.4.2. To configure SSH, Create RSA Keys On Each Node using the followingsteps:

1. Log in as the software owner (in this example, the crs user).

2. To ensure that you are logged in as the Oracle user, and that the user ID matches the expected user ID youhave assigned to the Oracle user, enter the commands id and id oracle. Ensure that Oracle user group and userand the terminal window process group and user IDs are identical. For example:

[crs@node1 ~]$ id uid=502(crs) gid=501(oinstall) groups=501(oinstall),502(crs) [crs@node1 ~]$ id crs uid=502(crs) gid=501(oinstall) groups=501(oinstall),502(crs)

3. If necessary, create the .ssh directory in the crs user's home directory, and set permissions on it to ensurethat only the crs user has read and write permissions:

[crs@node1 ~]$ mkdir ~/.ssh [crs@node1 ~]$ chmod 700 ~/.ssh

4. Enter the following command:

[crs@node1 ~]$ /usr/bin/ssh-keygen -t rsa

At the prompts:

Accept the default location for the key file (press Enter).• Enter and confirm a pass phrase unique for this installation user.•

This command writes the RSA public key to the ~/.ssh/id_rsa.pub file and the private key to the ~/.ssh/id_rsa

2.4. Configuring SSH on Cluster Member Nodes 11

file.

Never distribute the private key to anyone not authorized to perform Oracle software installations.

5. Repeat steps 1 through 4 on each node that you intend to make a member of the cluster, using the RSA key.

2.4.3. Add All Keys to a Common authorized_keys File

Complete the following steps:

1. Combine the contents of the id_rsa.pub files from each server. (You can do this all on one node)run the command below for each node that is a part of the cluster, replacing "stnsp001" with a different nodename each time. As the oracle user:

$ ssh node1 "cat ~/.ssh/id_rsa.pub" >> ~/.ssh/authorized_keys $ ssh node2 "cat ~/.ssh/id_rsa.pub" >> ~/.ssh/authorized_keys

* Each time you connect from any node to a new hostname for the first time, you will see a message similarto:

"The authenticity of host 'node1 (10.3.25.81' )' can't be established. RSA key fingerprint is ... Are you sure you want to continue connecting (yes/no)?"

Type "yes" and press ENTER. You will then see the message:

"Warning: Permanently added 'node1,10.3.25.81' (RSA) to the list of known hosts." "oracle@node1's password:"

Type password and press ENTER.

In the .ssh directory, you should see the id_rsa.pub keys that you have created, and the file authorized_keys.

2. On the local node, use SCP (Secure Copy) or SFTP (Secure FTP) to copy the authorized_keys file to theoracle user .ssh directory on a remote node. The following example is with SCP, on a node called node2, withthe Oracle Clusterware owner crs, where the crs user path is /home/crs:

[[crs@node1 crs]$ scp ~/.ssh/authorized_keys node2:~/.ssh/

You are prompted to accept an RSA key. Enter Yes, and you see that the node you are copying to is added tothe known_hosts file.

When prompted, provide the password for the oracle user, which should be the same on all nodes in thecluster. The authorized_keys file is copied to the remote node.

Your output should be similar to the following, where xxx represents parts of a valid IP address:

[crs@node1 .ssh]$ ~/.ssh/scp authorized_keys node2:~/.ssh/ crs@node2's password: authorized_keys 100% 828 7.5MB/s 00:00

3. Repeat steps 2 from each node to each other member node in the cluster.

2.4.2. To configure SSH, Create RSA Keys On Each Node using the followingsteps: 12

2.4.4. Enabling SSH User Equivalency on Cluster Member Nodes

After you have copied the authorized_keys file that contains all keys to each node in the cluster, complete thefollowing procedure, in the order listed. In this example, the Oracle Clusterware software owner is named crs:

1. On the system where you want to run OUI, log in as the crs user.

2. Use the following command syntax, where hostname1, hostname2, and so on, are the public hostnames(alias and fully qualified domain name) of nodes in the cluster to run SSH from the local node to each node,including from the local node to itself, and from each node to each other node:

[crs@node1 ~]$ ssh node1 date [crs@node1 ~]$ ssh node2 date . . .

For example: [crs@node1 crs]$ ssh node1 date The authenticity of host 'node1 (xxx.xxx.25.81)' can't be established. RSA key fingerprint is 7z:60:60:zz:48:48:z1:a0:f7:4e. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'node1,xxx.xxx.25.81' (RSA) to the list of known hosts. Enter passphrase for key '/home/crs/.ssh/id_rsa': Mon May 19 11:08:13 PST 2008

[crs@stnsp001 crs]$ ssh node1.us.oracle.com date The authenticity of host 'node1.us.oracle.com (xxx.xxx.25.81)' can't be established. RSA key fingerprint is 7z:60:60:zz:48:48:z1:a0:f7:4e. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'stnsp001.us.oracle.com,xxx.xxx.25.81' (RSA) to the list of known hosts. Enter passphrase for key '/home/crs/.ssh/id_rsa': Mon May 19 11:08:13 PST 2008

[crs@stnsp001 crs]$ ssh node2 date Enter passphrase for key '/home/crs/.ssh/id_rsa': Mon May 19 11:08:35 PST 2008 . .

At the end of this process, the public hostname for each member node should be registered in theknown_hosts file for all other cluster member nodes.

If you are using a remote client to connect to the local node, and you see a message similar to

"Warning: No xauth data; using fake authentication data for X11 forwarding,"

then this means that your authorized keys file is configured correctly, but your ssh configuration has X11forwarding enabled. To correct this, proceed to "Setting Display and X11 Forwarding Configuration" below.

3. Repeat step 2 on each cluster node member.

4. (Optional) If you entered passphrase above, and desire password-less login (needed for 10g RAC install),you need to inform the ssh-agent (already running for desktop) of the passphrase so that ssh clients would not

2.4.4. Enabling SSH User Equivalency on Cluster Member Nodes 13

be prompted for a passphrase. Once you notify ssh-agent the passphrase is cached for duration of the GUIdesktop login, and further client requests are satisfied by the ssh-agent.

$ ssh-add Enter passphrase for /home/oracle/.ssh/id_dsa: *TYPE YOUR PASSPHRASE* Identity added: /home/oracle/.ssh/id_dsa (/home/oracle/.ssh/id_dsa)

At the prompt, enter the pass phrase for each key that you generated. For example:

[crs@node1 .ssh]$ exec /usr/bin/ssh-agent $SHELL [crs@node1 .ssh]$ ssh-add Enter passphrase for /home/crs/.ssh/id_rsa Identity added: /home/crs/.ssh/id_rsa (/home/crs/.ssh/id_rsa)

These commands start the ssh-agent on the node, and load the RSA keys into memory so that you are notprompted to use pass phrases when issuing SSH commands.

If you have configured SSH correctly, then you can now use the ssh or scp commands without beingprompted for a password or a pass phrase. For example:

[crs@stnsp001 ~]$ ssh node2 date Mon May 19 23:34:42 UTC 2008 [crs@stnsp001 ~]$ ssh stnsp001 date Mon May 19 23:34:48 UTC 2008 [crs@stnsp001 ~]$ ssh node2

If any node prompts for a password or pass phrase, then verify that the ~/.ssh/authorized_keys file on thatnode contains the correct public keys, and that you have created an Oracle software owner with identicalgroup membership and IDs.

2.4.5. Setting Display and X11 Forwarding Configuration

If you are on a remote terminal, and the local node has only one visual (which is typical), then use thefollowing syntax to set the DISPLAY environment variable:

•

Bourne, Korn, and Bash shells

[crs@node1~]$ export DISPLAY=hostname:0

Note: You must run OUI from this session, or make a note of your SSH pass phrase, and remember to repeatstep 4 before you start OUI from a different terminal session.

C shell:

[crs@node1 ~]$ setenv DISPLAY hostname:0

For example, if you are using the Bash shell, and if your hostname is stnsp001, then enter the followingcommand:

$ export DISPLAY=node1:0

To ensure that X11 forwarding will not cause the installation to fail, create a user-level SSH clientconfiguration file for the Oracle software owner user, as follows:

•

2.4.5. Setting Display and X11 Forwarding Configuration 14

A. Using any text editor, edit or create the software installation owner’s

~/.ssh/config file.

B. Make sure that the ForwardX11 attribute is set to no. For example:

Host * ForwardX11 no

If you are not plannig to run the installer on the console you may need to run the following commandto allow the X session to be accessed remotely

xhost +

•

2.5. Time Synchronization, xntpd, Mandatory

There is a general requirement for Oracle RAC that the time on all the nodes be the same. Typically this canbe achieved using Network Time Server protocol.

Oracle recommends that you use xntpd for this purpose. xntpd is a complete implementation of the NTPversion 3 satndard and is more accurate than "timed".

To configure xntpd, follow these steps on each cluster node :

# touch /etc/ntp.drift /etc/ntp.trace /etc/ntp.conf # vi /etc/ntp.conf

Sample NTP Configuration file

# Specify the IP Addresses of three clock server systems.timeserver1 10.3.25.101timeserver2 10.3.25.102timeserver3 10.3.25.103<br># Most of the routers are broadcasting NTP time information. If your# router is broadcasting, then the following line enables xntpd# to listen for broadcasts.broadcastclient# Write clock drift parameters to a file. This enables the system# clock to quickly sychronize to the true time on restart.driftfile /etc/ntp.drifttracefile /etc/ntp.trace

A - Enter the following command:

# /usr/bin/smitty xntpd

B - Choose Start Using the xntpd Subsystem, then choose BOTH.

2.5. Time Synchronization, xntpd, Mandatory 15

2.6. Tuning the AIX Environment

2.6.1. Tuning Virtual Memory Manager (VMM)

Oracle recommends that you use the vmo command to tune virtual memory using the following values:

Parameter Valueminperm% 3 (default is 20)maxperm% 90 (default is 80)maxclient% = 90 90 (default is 80)lru_file_repage 0 (default is 1)strict_maxclient 1 (default is 1)strict_maxperm 0 (default is 0)page_steal_method 1 (default is ?)For example:

vmo -p -o minperm%=3vmo -p -o maxperm%=90vmo -p -o maxclient%=90vmo -p -o lru_file_repage=0vmo -p -o strict_maxclient=1vmo -p -o strict_maxperm=0vmo -r -o page_steal_method=1

You must restart the system for these changes to take effect.

2.6.2. Increasing System Block Size Allocation

Oracle recommends that you increase the space allocated for ARG/ENV list to 128.The size is specified by number of 4K blocks.For example:

chdev -l sys0 -a ncargs='128'

2.6.3. Configuring Shell Limits

To improve the performance of the software, you must increase the following shell limits:

Shell Limit Item in limits.conf Hard LimitMaximum number of open file descriptors nofile 65536Maximum number of processes available to a single user maxuproc 16384

2.6. Tuning the AIX Environment 16

To increase the shell limits:

1. Add the following lines to the /etc/security/limits file:

default:fsize = -1core = -1cpu = -1data = 512000rss = 512000stack = 512000nofiles = 2000If you will be running large queries or you see ORA-4030 errors set data, rss and stack equal to -1Thus:data = -1rss=-1stack = 512000

2. Enter the following command to list the current setting for the maximum number of processes allowed bythe Oracle software user:

/etc/lsattr -E -l sys0 -a maxuproc

If necessary, change the maxuproc setting using the following command:

/etc/chdev -l sys0 -a maxuproc = 16384

3. Repeat this procedure on all other nodes in the cluster

2.6.4. Configuring User Process Parameters

Verify that the maximum number of processes allowed for each user is set to 2048 or greater:1. Enter the following command:

# smit chgsys

2. Verify that the value shown for Maximum number of PROCESSES allowed for each user is greater than orequal to 2048.

Note: For production systems, this value should be at least 128 plus the sum of the PROCESSES andPARALLEL_MAX_SERVERS initialization parameters for each database running on the system.

If necessary, edit the existing value.

3. When you have finished making changes, press F10 to exit.

2.6.5. Configuring Network Tuning Parameters

Verify that the network tuning parameters shown in the following table are set to the values shown or highervalues.The procedure following the table describes how to verify and set the values.

2.6.3. Configuring Shell Limits 17

Network Tuning Parameter Recommended Valueipqmaxlen 512rfc1323 1sb_max 2*655360tcp_recvspace 65536tcp_sendspace 65536udp_recvspace 655360

Note: The recommended value of this parameter is 10 times thevalue of the udp_sendspace parameter. The value must be lessthan the value of the sb_max parameter.

udp_sendspace 65536Note: This value is suitable for a default database installation. Forproduction databases, the minimum value for this parameter is 4KB plus the value of the database DB_BLOCK_SIZE initializationparameter multiplied by the value of theDB_MULTIBLOCK_READ_COUNT initialization parameter:(DB_BLOCK_SIZE * DB_MULTIBLOCK_READ_COUNT) + 4 KB

To view the current value specified for these parameters, and to change them if necessary:

1. To check the current values of the network tuning parameters, enter commands similar to the following:

# no -a | more

2. If you must change the value of any parameter, then enter the following command to determine whether thesystem is running in compatibility mode:

# lsattr -E -l sys0 -a pre520tune

If the system is running in compatibility mode, then the output is similar to the following, showing that thevalue of the pre520tune attribute is enabled

pre520tune enable Pre-520 tuning compatibility mode True

3. If the system is running in compatibility mode, then follow these steps to change the parameter values:

a. Enter commands similar to the following to change the value of each parameter:

# no -o parameter_name=value

For example

# no -o udp_recvspace=655360

b. Add entries similar to the following to the /etc/rc.net file for each parameter that you changed in theprevious step

if [ -f /usr/sbin/no ] ; then/usr/sbin/no -o udp_sendspace=65536/usr/sbin/no -o udp_recvspace=655360/usr/sbin/no -o tcp_sendspace=65536/usr/sbin/no -o tcp_recvspace=65536/usr/sbin/no -o rfc1323=1

2.6.5. Configuring Network Tuning Parameters 18

/usr/sbin/no -o sb_max=2*655360/usr/sbin/no -o ipqmaxlen=512fi

By adding these lines to the /etc/rc.net file, the values persist when the system restarts.

4. If the system is not running in compatibility mode, then enter commands similar to the following to changethe parameter values:

ipqmaxlen parameter

# no -r -o ipqmaxlen=512

These commands modify the /etc/tunables/nextboot file, causing the attribute values to persist when thesystem restarts.

2.7. crs environment setup

AIX requires some environment variables to be set up either in the .profile file for the CRS and all otherOracle users (ASM and Oracle Database Home)or the changes can be made permanent by placing them in the "/etc/environment" file.

Oracle environment : $HOME/.profile file in the Oracle home directory

export ORACLE_BASE=/oracleexport AIXTHREAD_SCOPE=S #(S for system-wide thread scope)export TEMP=/tmpexportTMP=/tmpexport TMPDIR=/tmpumask 022if [ -t 0 ]; thenstty intr ^Cfi

Notes:

On AIX, when using multithreaded applications or LAN-free, especially when running on machines withmultiple CPUs:We strongly recommend setting AIXTHREADSCOPE=S in the environment before starting the application,for better performance and more solid scheduling.

For example:

export AIXTHREADSCOPE=S

Setting AIXTHREAD_SCOPE=S means that user threads created with default attributes will be placed intothe system-wide contention scope.If a user thread is created with system-wide contention scope, it is bound to a kernel thread and it is scheduledby the kernel.The underlying kernel thread is not shared with any other user thread.

2.7. crs environment setup 19

2.8. Filesets, AIX APAR and other Operating System Fixes.

You will need the following filesets and fixes:

2.8.1. Fixes

Installation Type or Product Requirement

All AIX 5L v5.3installations

All AIX 5L v. 5.3 installations Authorized Problem Analysis Reports (APARs) forAIX 5L v. 5.3 ML06, and the following AIX fixes:IY89080IY92037IY94343IZ01060, or efix for IZ01060IZ03260, or efix for IZ03260

Oracle JDBC/OCIDrivers

Note: These APARs are required only if you are using the associated JDK version

AIX 5L v5.3 APAR required for JDK 1.4.2 (64-bit):■ IY63533: JDK 1.4.2 64-bit SR1 caix64142-20040917APARs required for JDK 1.3.1.16 (32-bit):■ IY58350: SDK 1.3.1 32-Bit SR7P? : CA131IFX? -20040721A■ IY65305: JAVA142 32-bit PTF: CA142IFX? -20041203

2.8.2. Filesets

Before running CVU bear in mind that CVU may report quite a lot of packages and patches as missing. Thisis because CVU does not know if you are installing on GPFS or on top of third party clusterware. Thepackages you actually need for native Oracle clusterware with the vote disk and CRS on raw logical volumesare these:

bos.adt.basebos.adt.libbos.adt.libmbos.perf.libperfstatbos.perf.perfstatbos.perf.proctoolsrsct.basic.rtersct.compat.clients.rtexlC.aix50.rte 8.0.0.7 (or later)xlC.rte 8.0.0.7 (or later)

You must have the xlC C/C++ runtime filesets for installation, but you do not require the C/C++ compilers.You do not require a license for the xlC runtime filesets

IBM JDK 1.5 is installed with this release.

Any missing patches recomended by CVU can be obtained from:

http://www-912.ibm.com/eserver/support/fixes/

2.8. Filesets, AIX APAR and other Operating System Fixes. 20

http://globaldc.oracle.com/perl/twiki/edit/RACGuides/SR7P?topicparent=RACGuides.Rac11gR1OnAIX

http://globaldc.oracle.com/perl/twiki/edit/RACGuides/CA131IFX?topicparent=RACGuides.Rac11gR1OnAIX

http://globaldc.oracle.com/perl/twiki/edit/RACGuides/CA142IFX?topicparent=RACGuides.Rac11gR1OnAIX

http://www-912.ibm.com/eserver/support/fixes/

2.9. Stage the Oracle Software

It is recommended that you stage the required software onto a local drive on Node 1 of your cluster.Important. Ensure that you use only 32 bit versions of the Oracle Software on a 32bit OS and 64 bit versionsof the Oracle Software on a 64bit OS A suggested stage is as follows

Oracle Database 11g Release 1 Enterprise Edition (11.1.0.6) (1 DVD ) /mnt•

The 11gR1 now integrates the Oracle Clusterware, Database and Client install into one DVD with onerunInstaller program.

If you download the software from OTN you will not get the integrated installer. You will have separatedownloads for :

Oracle Clusterware and• ASM/Database•

2.10. Cluster Verification Utility stage check

Now you can run the CVU (Cluster Verification Utility) to check the state of the Operating Systemconfiguration. CVU can be run from the installation media, but it is recommended to download the latestversion from:http://www.oracle.com/technology/products/database/clustering/cvu/cvu_download_homepage.html

After the hardware and OS has been configured, it is recommended to run CVU to verify the nodes areconfigured correctly:

runcluvfy.sh stage -post hwos -n node1,node2 -verbose

Show CVU output Hide CVU output|

check ONLY for necessary requirements for ASM as explained below.

CVU is testing existence of all prerequirements needed for all implementations as :RAC implementationASM implementation GPFS implementation HACMP implementation (Concurrent raw devices or cohabitation between ASM or GPFS with HACMP)

runcluvfy.sh stage -pre crsinst -n node1,node2 -verbosePerforming pre-checks for cluster services setupChecking node reachability...Check: Node reachability from node "node1" Destination Node Reachable? ------------------------------------ ------------------------ node1 yes node2 yesResult: Node reachability check passed from node "node1".

Checking user equivalence...

Check: User equivalence for user "oracle"

2.9. Stage the Oracle Software 21

http://www.oracle.com/technology/products/database/clustering/cvu/cvu_download_homepage.html

Node Name Comment ------------------------------------ ------------------------ node2 passed node1 passed

Result: User equivalence check passed for user "oracle".

Checking administrative privileges...Check: Existence of user "oracle"

Node Name User Exists Comment ------------ ------------------------ ------------------------ node1 yes passed node2 yes passed

Result: User existence check passed for "oracle".

Check: Existence of group "oinstall"

Node Name Status Group ID ------------ ------------------------ ------------------------ node1 exists 501 node2 exists 501

Result: Group existence check passed for "oinstall".

Check: Membership of user "oracle" in group "oinstall" [as Primary]

Node Name User Exists Group Exists User in Group Primary Comment ---------------- ------------ ------------- ------------ ------------ ------------ node1 yes yes yes no passed node2 yes yes yes no passed

Result: Membership check for user "oracle" in group "oinstall" [as Primary] passed

Administrative privileges check passed

Checking node connectivity...

Interface information for node "node1" Interface Name IP Address Subnet ------------------------------ ------------------------------ ---------------- en0 10.3.25.81 10.3.25.0 en1 10.10.25.81 10.10.25.0 en2 20.20.25.81 20.20.25.0


Check: Node connectivity of subnet "10.3.25.0" Source Destination Connected? ------------------------------ ------------------------------ ----------------

2.10. Cluster Verification Utility stage check 22

node1:en0 node2:en0 yes

Result: Node connectivity check passed for subnet "10.3.25.0" with node(s) node1,node2.

Check: Node connectivity of subnet "10.10.25.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- node1:en1 node2:en1 yes


Check: Node connectivity of subnet "20.20.25.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- node1:en2 node2:en2 yes


Suitable interfaces for VIP on subnet "20.20.25.0":node1 en2:20.20.25.81node2 en2:20.20.25.82

Suitable interfaces for the private interconnect on subnet "10.3.25.0":node1 en0:10.3.25.81node2 en0:10.3.25.82


Result: Node connectivity check passed.

Checking system requirements for 'crs'...

Check: Kernel version Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- node1 AIX 5.3 AIX 5.2 passed node2 AIX 5.3 AIX 5.2 passed

Result: Kernel version check passed.

Check: System architecture

Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- node1 powerpc powerpc passed node2 powerpc powerpc passed

Result: System architecture check passed.

Check: Total memory

Node Name Available Required Comment ------------ ------------------------ ------------------------ ----------


node1 2GB (2097152KB) 512MB (524288KB) passed node2 2GB (2097152KB) 512MB (524288KB) passed Result: Total memory check passed.

Check: Swap space

Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- node1 1024GB (1073741824KB) 1GB (1048576KB) passed node2 1024GB (1073741824KB) 1GB (1048576KB) passed

Result: Swap space check passed.

Check: Free disk space in "/tmp" dir Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- node1 400.35MB (409960KB) 400MB (409600KB) passed node2 400.35MB (409960KB) 400MB (409600KB) passed

Result: Free disk space check passed.Check:

Free disk space in "/oracle" dir

Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- node1 4.63GB (4860044KB) 4GB (4194304KB) passed node2 4.63GB (4860044KB) 4GB (4194304KB) passed

Result: Free disk space check passed.

Checking system requirements for 'crs'... (Continued …)

Check: Package existence for "vacpp.cmp.core:7.0.0.2" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 vacpp.cmp.core:6.0.0.0 failed node2 vacpp.cmp.core:6.0.0.0 failed

Result: Package existence check failed for "vacpp.cmp.core:7.0.0.2".

Check: Operating system patch for "IY65361 "

Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- node1 unknown IY65361 failed node2 unknown IY65361 failed

Result: Operating system patch check failed for "IY65361 ".

Check: Package existence for "vac.C:7.0.0.2"

Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 vac.C:7.0.0.2 passed node2 vac.C:7.0.0.2 passed


Result: Package existence check passed for "vac.C:7.0.0.2".

Check: Package existence for "xlC.aix50.rte:7.0.0.4"

Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 xlC.aix50.rte:7.0.0.4 passed node2 xlC.aix50.rte:7.0.0.4 passed

Result: Package existence check passed for "xlC.aix50.rte:7.0.0.4".

Check: Package existence for "xlC.rte:7.0.0.1"

Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 xlC.rte:7.0.0.1 passed node2 xlC.rte:7.0.0.1 passed

Result: Package existence check passed for "xlC.rte:7.0.0.1".

Check: Package existence for "gpfs.base:2.3.0.3"

Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 gpfs.base:2.3.0.5 passed node2 gpfs.base:2.3.0.5 passed

Result: Package existence check passed for "gpfs.base:2.3.0.3".

Check: Operating system patch for "IY63969" Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- node1 IY63969:gpfs.baseIY63969:gpfs.docs.dataIY63969:gpfs.msg.en_US IY63969 passed node2 IY63969:gpfs.baseIY63969:gpfs.docs.dataIY63969:gpfs.msg.en_US IY63969 passed

Result: Operating system patch check passed for "IY63969".

Check: Operating system patch for "IY69911" Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- node1 IY69911:gpfs.base IY69911 passed node2 IY69911:gpfs.base IY69911 passed


Check: Operating system patch for "IY70276"

Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- node1 IY70276:gpfs.base IY70276 passed node2 IY70276:gpfs.base IY70276 passed



Check: Package existence for "cluster.license:5.2.0.0"

Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 missing failed node2 missing failed

Result: Package existence check failed for "cluster.license:5.2.0.0".



Result: Operating system patch check failed for "IY60759".


Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- node1 IY61034:bos.mpIY61034:bos.mp64 IY61034 passed node2 IY61034:bos.mpIY61034:bos.mp64 IY61034 passed


Check: Operating system patch for "IY61770" Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- node1 IY61770:rsct.basic.rteIY61770:rsct.core.errmIY61770:rsct.core.hostrmIY61770:rsct.core.rmcIY61770:rsct.core.secIY61770:rsct.core.sensorrmIY61770:rsct.core.utils IY61770 passed node2 IY61770:rsct.basic.rteIY61770:rsct.core.errmIY61770:rsct.core.hostrmIY61770:rsct.core.rmcIY61770:rsct.core.secIY61770:rsct.core.sensorrmIY61770:rsct.core.utils IY61770 passed


Check: Operating system patch for "IY62191" Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- node1 IY62191:bos.adt.profIY62191:bos.rte.libpthreads IY62191 passed node2 IY62191:bos.adt.profIY62191:bos.rte.libpthreads IY62191 passed


Check: Package existence for "ElectricFence-2.2.2-1:2.2.2"


Result: Package existence check failed for "ElectricFence-2.2.2-1:2.2.2".


Check: Package existence for "xlfrte:9.1"

Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 xlfrte:8.1.1.4 failed node2 xlfrte:8.1.1.4 failed Result: Package existence check failed for "xlfrte:9.1".

Check: Package existence for "gdb-6.0-1:6.0"


Result: Package existence check failed for "gdb-6.0-1:6.0".Check:

Package existence for "make-3.80-1:3.80"


Result: Package existence check failed for "make-3.80-1:3.80".

Check: Package existence for "freeware.gnu.tar.rte:1.13.0.0"


Result: Package existence check failed for "freeware.gnu.tar.rte:1.13.0.0".

Check: Package existence for "Java14_64.sdk:1.4.2.1"


Result: Package existence check failed for "Java14_64.sdk:1.4.2.1".

Check: Package existence for "Java131.rte.bin:1.3.1.16"


Result: Package existence check failed for "Java131.rte.bin:1.3.1.16".

Check: Package existence for "Java14.sdk:1.4.2.2"


Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 Java14.sdk:1.4.2.10 failed node2 Java14.sdk:1.4.2.10 failed

Result: Package existence check failed for "Java14.sdk:1.4.2.2".


Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- node1 IY65305:Java14.sdk IY65305 passed node2 IY65305:Java14.sdk IY65305 passed








Check: Package existence for "mqm.server.rte:5.3"


Result: Package existence check failed for "mqm.server.rte:5.3".

Check: Package existence for "mqm.client.rte:5.3"


Result: Package existence check failed for "mqm.client.rte:5.3".

Check: Package existence for "sna.rte:6.1.0.4"

Node Name Status Comment


------------------------------ ------------------------------ ---------------- node1 missing failed node2 missing failed

Result: Package existence check failed for "sna.rte:6.1.0.4".

Check: Package existence for "bos.net.tcp.server"

Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 bos.net.tcp.server:5.3.0.30 passed node2 bos.net.tcp.server:5.3.0.30 passed

Result: Package existence check passed for "bos.net.tcp.server".








Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- node1 IY58143:X11.Dt.libIY58143:X11.base.rteIY58143:bos.acctIY58143:bos.adt.includeIY58143:bos.adt.libmIY58143:bos.adt.profIY58143:bos.alt_disk_install.rteIY58143:bos.diag.comIY58143:bos.mp64IY58143:bos.mpIY58143:bos.net.ewlm.rteIY58143:bos.net.ipsec.keymgtIY58143:bos.net.ipsec.rteIY58143:bos.net.nfs.clientIY58143:bos.net.nfs.serverIY58143:bos.net.tcp.clientIY58143:bos.net.tcp.serverIY58143:bos.net.tcp.smitIY58143:bos.perf.libperfstatIY58143:bos.perf.perfstatIY58143:bos.perf.toolsIY58143:bos.rte.archiveIY58143:bos.rte.bind_cmdsIY58143:bos.rte.bootIY58143:bos.rte.controlIY58143:bos.rte.filesystemIY58143:bos.rte.installIY58143:bos.rte.libcIY58143:bos.rte.lvmIY58143:bos.rte.manIY58143:bos.rte.methodsIY58143:bos.rte.securityIY58143:bos.rte.serv_aidIY58143:bos.sysmgt.nim.clientIY58143:bos.sysmgt.quotaIY58143:bos.sysmgt.serv_aidIY58143:bos.sysmgt.sysbrIY58143:devices.chrp.base.rteIY58143:devices.chrp.pci.rteIY58143:devices.chrp.vdevice.rteIY58143:devices.common.IBM.atm.rteIY58143:devices.common.IBM.ethernet.rteIY58143:devices.common.IBM.fc.rteIY58143:devices.common.IBM.fda.diagIY58143:devices.common.IBM.mpio.rteIY58143:devices.fcp.disk.rteIY58143:devices.pci.00100f00.rteIY58143:devices.pci.14100401.diagIY58143:devices.pci.14103302.rteIY58143:devices.pci.14106602.rteIY58143:devices.pci.14106902.rteIY58143:devices.pci.14107802.rteIY58143:devices.pci.1410ff01.rteIY58143:devices.pci.22106474.rteIY58143:devices.pci.33103500.rteIY58143:devices.pci.4f111100.comIY58143:devices.pci.77101223.comIY58143:devices.pci.99172704.rteIY58143:devices.pci.c1110358.rteIY58143:devices.pci.df1000f7.comIY58143:devices.pci.df1000f7.diagIY58143:devices.pci.df1000fa.rteIY58143:devices.pci.e414a816.rteIY58143:devices.scsi.disk.rte


IY58143:devices.vdevice.IBM.llan.rteIY58143:devices.vdevice.IBM.vscsi.rteIY58143:devices.vdevice.hvterm1.rteIY58143:devices.vtdev.scsi.rteIY58143:sysmgt.websm.appsIY58143:sysmgt.websm.frameworkIY58143:sysmgt.websm.rteIY58143:sysmgt.websm.webaccess IY58143 passed

Node2 IY58143:X11.Dt.libIY58143:X11.base.rteIY58143:bos.acctIY58143:bos.adt.includeIY58143:bos.adt.libmIY58143:bos.adt.profIY58143:bos.alt_disk_install.rteIY58143:bos.diag.comIY58143:bos.mp64IY58143:bos.mpIY58143:bos.net.ewlm.rteIY58143:bos.net.ipsec.keymgtIY58143:bos.net.ipsec.rteIY58143:bos.net.nfs.clientIY58143:bos.net.nfs.serverIY58143:bos.net.tcp.clientIY58143:bos.net.tcp.serverIY58143:bos.net.tcp.smitIY58143:bos.perf.libperfstatIY58143:bos.perf.perfstatIY58143:bos.perf.toolsIY58143:bos.rte.archiveIY58143:bos.rte.bind_cmdsIY58143:bos.rte.bootIY58143:bos.rte.controlIY58143:bos.rte.filesystemIY58143:bos.rte.installIY58143:bos.rte.libcIY58143:bos.rte.lvmIY58143:bos.rte.manIY58143:bos.rte.methodsIY58143:bos.rte.securityIY58143:bos.rte.serv_aidIY58143:bos.sysmgt.nim.clientIY58143:bos.sysmgt.quotaIY58143:bos.sysmgt.serv_aidIY58143:bos.sysmgt.sysbrIY58143:devices.chrp.base.rteIY58143:devices.chrp.pci.rteIY58143:devices.chrp.vdevice.rteIY58143:devices.common.IBM.atm.rteIY58143:devices.common.IBM.ethernet.rteIY58143:devices.common.IBM.fc.rteIY58143:devices.common.IBM.fda.diagIY58143:devices.common.IBM.mpio.rteIY58143:devices.fcp.disk.rteIY58143:devices.pci.00100f00.rteIY58143:devices.pci.14100401.diagIY58143:devices.pci.14103302.rteIY58143:devices.pci.14106602.rteIY58143:devices.pci.14106902.rteIY58143:devices.pci.14107802.rteIY58143:devices.pci.1410ff01.rteIY58143:devices.pci.22106474.rteIY58143:devices.pci.33103500.rteIY58143:devices.pci.4f111100.comIY58143:devices.pci.77101223.comIY58143:devices.pci.99172704.rteIY58143:devices.pci.c1110358.rteIY58143:devices.pci.df1000f7.comIY58143:devices.pci.df1000f7.diagIY58143:devices.pci.df1000fa.rteIY58143:devices.pci.e414a816.rteIY58143:devices.scsi.disk.rteIY58143:devices.vdevice.IBM.llan.rteIY58143:devices.vdevice.IBM.vscsi.rteIY58143:devices.vdevice.hvterm1.rteIY58143:devices.vtdev.scsi.rteIY58143:sysmgt.websm.appsIY58143:sysmgt.websm.frameworkIY58143:sysmgt.websm.rteIY58143:sysmgt.websm.webaccess IY58143 passed









Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- node1 IY59386:bos.rte.bind_cmds IY59386 passed node2 IY59386:bos.rte.bind_cmds IY59386 passed



Check: Package existence for "bos.adt.base"

Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 bos.adt.base:5.3.0.30 passed node2 bos.adt.base:5.3.0.30 passed

Result: Package existence check passed for "bos.adt.base".

Check: Package existence for "bos.adt.lib"

Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 bos.adt.lib:5.3.0.30 passed node2 bos.adt.lib:5.3.0.30 passed

Result: Package existence check passed for "bos.adt.lib".

Check: Package existence for "bos.adt.libm"

Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 bos.adt.libm:5.3.0.30 passed node2 bos.adt.libm:5.3.0.30 passed

Result: Package existence check passed for "bos.adt.libm".

Check: Package existence for "bos.perf.libperfstat"

Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 bos.perf.libperfstat:5.3.0.30 passed node2 bos.perf.libperfstat:5.3.0.30 passed

Result: Package existence check passed for "bos.perf.libperfstat".

Check: Package existence for "bos.perf.perfstat"

Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 bos.perf.perfstat:5.3.0.30 passed node2 bos.perf.perfstat:5.3.0.30 passed

Result: Package existence check passed for "bos.perf.perfstat".

Check: Package existence for "bos.perf.proctools"

Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 bos.perf.proctools:5.3.0.30 passed node2 bos.perf.proctools:5.3.0.30 passed Result: Package existence check passed for "bos.perf.proctools".

Check: Package existence for "rsct.basic.rte"


Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 rsct.basic.rte:2.4.3.0 passed node2 rsct.basic.rte:2.4.3.0 passed

Result: Package existence check passed for "rsct.basic.rte".

Check: Package existence for "perl.rte:5.0005"

Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 perl.rte:5.8.2.30 passed node1 perl.rte:5.8.2.30 passed

Result: Package existence check passed for "perl.rte:5.0005".







Check: Package existence for "python-2.2-4:2.2"


Result: Package existence check failed for "python-2.2-4:2.2".

Check: Package existence for "freeware.zip.rte:2.3"


Result: Package existence check failed for "freeware.zip.rte:2.3".

Check: Package existence for "freeware.gcc.rte:3.3.2.0"

Node Name Status Comment


------------------------------ ------------------------------ ---------------- node1 missing failed node2 missing failed

Result: Package existence check failed for "freeware.gcc.rte:3.3.2.0".

Check: Group existence for "dba"

Node Name Status Comment ------------ ------------------------ ------------------------ node1 exists passed node2 exists passed Result: Group existence check passed for "dba".

Check: User existence for "nobody"

Node Name Status Comment ------------ ------------------------ ------------------------ node1 exists passed node2 exists passed

Result: User existence check passed for "nobody".

System requirement failed for 'crs'

Pre-check for cluster services setup was unsuccessful on all the nodes.

Show all Hide all

3. *Prepare the shared storage for Oracle RACThis section describes how to prepare the shared storage for Oracle RAC

Only IBM Storage is discussed here.

For IBM storage there are two possible configurations:

IBM ESS, DS6000 and DS8000 series. In this case the multi-pathing solution is IBM MPIO(Multi-Path I/O)

•

IBM DS4000 series (formerly "FasTt") In this case the multi-pathing solution is IBM RDAC(Redundant Disk Array Controller)

•

IBM DS300 and DS400 series storage is not supported.

For other storage such as EMC, HDS, HP and so on, you will have to check that the storage is compatible andsupported with RAC.Oracle does not support or certify storage (except NAS)

Storage Vendors will say if they support Oracle RAC or not, and with which Multi-pathing tool (IBMSDDPCM, EMC PowerPath? , HDLM, etc).

The SAN storage must be capable of implementing read/write concurency (shared disk).This means that the "reserve_policy" attibute of the disks must be able to be set to "no_reserve" or "no_lock"as appropriate.

3. *Prepare the shared storage for Oracle RAC 33

http://globaldc.oracle.com/perl/twiki/edit/RACGuides/PowerPath?topicparent=RACGuides.Rac11gR1OnAIX

Supported I/O configurations.

Advanced Power Virtualization hardware LPAR's are supported including Dynamic LPAR (DLPAR)or shared pool with micro partitions

•

VIOS Virtual SCSI for ASM data storage and associated raw hdisk based Voting and OCR• VIOS Virtual LAN for all public network and private network interconnect for supported data storageoptions

•

3.1. Creating Storage for the OCR and Voting disks.

Using the storage administration console, you have to create :

2 LUNs for OCR disks (300MB size)• 3 LUNs for Voting disks (300MB size)•

The following table shows the LUN mapping for nodes used in our cluster.

The LUN for OCR disks and Voting disks have ids 1 to 5. These IDs will help us on to identify which hdiskwill be used.

Disks LUN ID Number LUN SizeOCR1 L1 300 MBOCR2 L2 300 MBVoting1 L3 300 MBVoting2 L4 300 MBVoting3 L5 300 MB

3.1.1. How to Identify if a LUN is used or not ?

On node 1

# lspvhdisk0 00cd7d3e6d2fa8db rootvg activehdisk1 none Nonehdisk2 none Nonehdisk3 none Nonehdisk4 none Nonehdisk5 none Nonehdisk6 none Nonehdisk7 none Nsdhdisk8 none Nsd1hdisk9 none Nsd2hdisk10 none Nsd3hdisk11 none Nsd4hdisk12 none Nonehdisk13 none Nonehdisk14 none None

No PVIDs are assigned apart for the rootvg hdisk

Use the following command to read hdisk header :

On node 1 ... for hdisk0

3.1. Creating Storage for the OCR and Voting disks. 34

node1:root}# lspv | grep rootvg

hdisk0 00cd7d3e6d2fa8db rootvg active

{node1:root}# lquerypv -h /dev/rhdisk000000000 C9C2D4C1 00000000 00000000 00000000 |................|00000010 00000000 00000000 00000000 00000000 |................|00000020 00000000 00000000 00000000 00000000 |................|00000030 00000000 00000000 00000000 00000000 |................|00000040 00000000 00000000 00000000 00000000 |................|00000050 00000000 00000000 00000000 00000000 |................|00000060 00000000 00000000 00000000 00000000 |................|00000070 00000000 00000000 00000000 00000000 |................|00000080 00CD7D3E 6D2FA8DB 00000000 00000000 |..}>m/..........|00000090 00000000 00000000 00000000 00000000 |................|000000A0 00000000 00000000 00000000 00000000 |................|000000B0 00000000 00000000 00000000 00000000 |................|000000C0 00000000 00000000 00000000 00000000 |................|000000D0 00000000 00000000 00000000 00000000 |................|000000E0 00000000 00000000 00000000 00000000 |................|000000F0 00000000 00000000 00000000 00000000 |................|

When hdisk is not marked rootvg, but None, it's important to check that it's not used at all.


On node 1 ... for hdisk2

node1:root}# lquerypv -h /dev/rhdisk200000000 00000000 00000000 00000000 00000000 |................|00000010 00000000 00000000 00000000 00000000 |................|00000020 00000000 00000000 00000000 00000000 |................|00000030 00000000 00000000 00000000 00000000 |................|00000040 00000000 00000000 00000000 00000000 |................|00000050 00000000 00000000 00000000 00000000 |................|00000060 00000000 00000000 00000000 00000000 |................|00000070 00000000 00000000 00000000 00000000 |................|00000080 00000000 00000000 00000000 00000000 |................|00000090 00000000 00000000 00000000 00000000 |................|000000A0 00000000 00000000 00000000 00000000 |................|000000B0 00000000 00000000 00000000 00000000 |................|000000C0 00000000 00000000 00000000 00000000 |................|000000D0 00000000 00000000 00000000 00000000 |................|000000E0 00000000 00000000 00000000 00000000 |................|000000F0 00000000 00000000 00000000 00000000 |................|

If all lines are ONLY full of '0's THEN the hdisk is free to be used. If that is not the case, check first whichLUN is mapped to the hdisk (next pages) and if it's the LUN you are going to use, you must then use 'dd' tozero out the disk

BE CAREFULL !!!,

3.1.1. How to Identify if a LUN is used or not ? 35

Any hdisk not marked 'None' as a result of the lspv command may show also a blank header (GPFSused hdisk for example)

So make sure that hdisk has "None" value from the lspv command.

List of all hdisks not marked 'rootvg' or 'None' on node 1 :

On node 1

{node1:root} # lspv...

hdisk7 none Nsdhdisk8 none Nsd1hdisk9 none Nsd2hdisk10 none Nsd3hdisk11 none Nsd4

...


On node 1 for hdisk7

{node1:root}# lquerypv -h /dev/rhdisk7

00000000 00000000 00000000 00000000 00000000 |................|00000010 00000000 00000000 00000000 00000000 |................|00000020 00000000 00000000 00000000 00000000 |................|00000030 00000000 00000000 00000000 00000000 |................|00000040 00000000 00000000 00000000 00000000 |................|00000050 00000000 00000000 00000000 00000000 |................|00000060 00000000 00000000 00000000 00000000 |................|00000070 00000000 00000000 00000000 00000000 |................|00000080 00000000 00000000 00000000 00000000 |................|00000090 00000000 00000000 00000000 00000000 |................|000000A0 00000000 00000000 00000000 00000000 |................|000000B0 00000000 00000000 00000000 00000000 |................|000000C0 00000000 00000000 00000000 00000000 |................|000000D0 00000000 00000000 00000000 00000000 |................|000000E0 00000000 00000000 00000000 00000000 |................|000000F0 00000000 00000000 00000000 00000000 |................|

3.1.2. Register LUN's at AIX level

Before registration of LUN's :

On node 1

{node1:root}# lspvhdisk0 00cd7d3e6d2fa8db rootvg active

3.1.2. Register LUN's at AIX level 36

On node 2

{node2:root}/ # lspvhdisk1 00cd7d3e7349e441 rootvg active

*As root on each node, update the ODM repository using the following command : *

cfgmgr

You need to register and identify LUN's at AIX level, and LUN's will be mapped to hdisk and registered inthe AIX ODM

After registration of LUN's in ODM thru cfgmgr command :

On node 1

{node1:root}# lspvhdisk0 00cd7d3e6d2fa8db rootvg activehdisk2 none Nonehdisk3 none Nonehdisk4 none Nonehdisk5 none Nonehdisk6 none None

On node 2

{node2:root}# lspvhdisk1 00cd7d3e7349e441 rootvg activehdisk2 none Nonehdisk3 none Nonehdisk4 none Nonehdisk5 none Nonehdisk6 none None

3.1.3. Identify LUN's and corresponding hdisk on each node

We know the LUN's available for OCR and Voting disks are L1 to L5.

*Disks LUN ID NumberOCR 1 L1OCR 2 L2Voting 1 L3Voting 2 L4Voting 3 L5

Identify the disks available for OCR and Voting disks, on each node, knowing the LUN’s numbers.

Knowing the LUN's number to use, we now need to identify the corresponding hdisks on each node of thecluster as detailed in the following table :

Disks

3.1.3. Identify LUN's and corresponding hdisk on each node 37


LUN IDNumber

Node 1 Correspondinghdisk


Node... Correspondinghdisk

OCR 1 L1OCR 2 L2Voting 1 L3Voting 2 L4Voting 3 L5There are two methods to identify the corresponding hdisks :

Now identify LUN ID assigned to hdisk, using "lscfg -l hdisk?" command

Do not assign a PVID to any LUN intended for use by Oracle

The creation of a PVID on a disk intended for use by Oracle will cause corruption of any data placed there.

Get the List of the hdisks on node1 : Note: rootvg is hdisk0 !!!

On node 1

{node1:root}# lspvhdisk0 00cd7d3e6d2fa8db rootvg activehdisk2 none Nonehdisk3 none Nonehdisk4 none Nonehdisk5 none Nonehdisk6 none None

NO PVIDs are assigned apart from the rootvg hdisk

Using lscfg command, try to identify the hdisks in the list generated by lspv on node1 :

Identify the LUN ID assigned to hdisk, using the "lscfg -vl hdisk?" command

On node 1

node1:root} # for i in 2 3 4 5 6 do lscfg -vl hdisk$i done

hdisk2 U7879.001.DQD17GX-P1-C2-T1-W200800A0B812AB31-L1000000000000 1722-600 (600) Disk Array Devicehdisk3 U7879.001.DQD17GX-P1-C2-T1-W200800A0B812AB31-L2000000000000 1722-600 (600) Disk Array Devicehdisk4 U7879.001.DQD17GX-P1-C2-T1-W200800A0B812AB31-L3000000000000 1722-600 (600) Disk Array Devicehdisk5 U7879.001.DQD17GX-P1-C2-T1-W200800A0B812AB31-L4000000000000 1722-600 (600) Disk Array Devicehdisk6 U7879.001.DQD17GX-P1-C2-T1-W200800A0B812AB31-L5000000000000 1722-600 (600) Disk Array Device

Then, We get the following table :

Disks LUN IDNumber




OCR 1 L1 hdisk2OCR 2 L2 hdisk3



















Voting 1 L3 hdisk4Voting 2 L4 hdisk5Voting 3 L5 hdisk6

Get the List of the hdisks on node2

rootvg is hdisk1, which is not same hdisk as on node 1, which means that it could be the same for each hdisk.On both nodes, the same hdisk names might not be attached to same LUN.

On node 2

{node2:root}# lspvhdisk1 00cd7d3e7349e441 rootvg activehdisk2 none Nonehdisk3 none Nonehdisk4 none Nonehdisk5 none Nonehdisk6 none None

No PVIDs are assigned apart for the rootvg hdisk

Using lscfg command, try to identify the hdisks in the list generated by lspv on node2 :

Identify the LUN ID assign to hdisk, using 'lscfg -vl hdisk?' command

On node 2

node2:root}# for i in 2 3 4 5 6 do lscfg -vl hdisk$idone

hdisk2 U7879.001.DQD17GX-P1-C5-T1-W200800A0B812AB31-L1000000000000 1722-600 (600) Disk Array Device hdisk3 U7879.001.DQD17GX-P1-C5-T1-W200800A0B812AB31-L2000000000000 1722-600 (600) Disk Array Device hdisk4 U7879.001.DQD17GX-P1-C5-T1-W200800A0B812AB31-L3000000000000 1722-600 (600) Disk Array Devicehdisk5 U7879.001.DQD17GX-P1-C5-T1-W200800A0B812AB31-L4000000000000 1722-600 (600) Disk Array Devicehdisk6 U7879.001.DQD17GX-P1-C5-T1-W200800A0B812AB31-L5000000000000 1722-600 (600) Disk Array Device

Then, We get the following table :

Disks LUN IDNumber




OCR 1 L1 hdisk2 hdisk2 hdisk?OCR 2 L2 hdisk3 hdisk3 hdisk?Voting 1 L3 hdisk4 hdisk4 hdisk?Voting 2 L4 hdisk5 hdisk5 hdisk?Voting 3 L5 hdisk6 hdisk6 hdisk?Do the same again for any further nodes











3.1.3.1. Removing reserve lock policy on hdisks from each node

In AIX the default behaviour is for the disks to be opened exclusively for the first node that opens them.

This would mean that only one node in cluster would be operational.

A RAC cluster requires concuttent access to the disk storage by all nodes.

In AIX terms this means we must change what is called the "RESERVE_POLICY" for all disks used by theclusterware and the database.

Example for one hdisk :

Issue the command 'lsattr -El hdisk2' to display all attributes for hdisk2

{node1:root}# lsattr -El hdisk2

PR_key_value none Persistant Reserve Key Value Truecache_method fast_write Write Caching method Falseieee_volname 600A0B800012AB30000017F64787273E IEEE Unique volume name Falselun_id 0x0001000000000000 Logical Unit Number Falsemax_transfer 0x100000 Maximum TRANSFER Size Trueprefetch_mult 1 Multiple of blocks to prefetch on read Falsepvid none Physical volume identifier Falseq_type simple Queuing Type Falsequeue_depth 10 Queue Depth Trueraid_level 5 RAID Level Falsereassign_to 120 Reassign Timeout value Truereserve_policy single_path Reserve Policy Truerw_timeout 30 Read/Write Timeout value Truescsi_id 0x660500 SCSI ID Falsesize 300 Size in Mbytes Falsewrite_cache yes Write Caching enabled False

On IBM storage (ESS, FasTt? , DSXXXX) : Change the reserve_policy attribute to no_reserve

chdev -l hdisk? -a reserve_policy=no_reserve

On EMC storage : Change the reserve_lock attribute to no

chdev -l hdisk? -a reserve_lock=no

On HDS storage with HDLM driver, and no disks in Volume Group : Change the dlmrsvlevel attribute tono_reserve

chdev -l dlmfdrv? -a dlmrsvlevel=no_reserve

Change the reserve_policy attributes for each disks dedicated to OCR and voting disks, on each nodes of thecluster :

In this example for IBM storage.

On Node 1... On Node 2...

3.1.3.1. Removing reserve lock policy on hdisks from each node 40

http://globaldc.oracle.com/perl/twiki/edit/RACGuides/FasTt?topicparent=RACGuides.Rac11gR1OnAIX

{node1:root}# for i in 2 3 4 5 6 {node2:root}# for i in 2 3 4 5 6do dochdev -l hdisk$i -a reserve_policy=no_reserve chdev -l hdisk$i -a

reserve_policy=no_reservedone donechanged changedchanged changedchanged changedchanged changedchanged changed{node1:root}# {node2:root}#

3.1.3.2. Identify Major and Minor number of hdisk on each node

*As described before, disks might have different names from one node to another for example hdisk2 onnode1 might be hdisk3 on node2, etc… *

Disks LUN IDNumber

Device Name Node 1Corresponding

Disk

MajorNum.

MinorNum.

Node 2Corresponding

Disk

MajorNum.

MinorNum.

OCR 1 L1 /dev/ocr_disk1 hdisk2 hdisk2OCR 2 L2 /dev/ocr_disk2 hdisk3 hdisk3Voting1

L3 /dev/voting_disk1 hdisk4 hdisk4

Voting2


Voting3


The next steps will explain the procedure to identify the Major and Minor numbers necessary to createvirtual devices pointing to the right hdisk on each node.

On node1 :

{node1:root}/ # for i in 2 3 4 5 6dols -la /dev/*hdisk$i done

brw------- 1 root system 21, 6 Jan 11 15:58 /dev/hdisk2crw------- 1 root system 21, 6 Jan 11 15:58 /dev/rhdisk2brw------- 1 root system 21, 7 Jan 11 15:58 /dev/hdisk3crw------- 1 root system 21, 7 Jan 11 15:58 /dev/rhdisk3brw------- 1 root system 21, 8 Jan 11 15:58 /dev/hdisk4crw------- 1 root system 21, 8 Jan 11 15:58 /dev/rhdisk4brw------- 1 root system 21, 9 Jan 11 15:58 /dev/hdisk5crw------- 1 root system 21, 9 Jan 11 15:58 /dev/rhdisk5brw------- 1 root system 21, 10 Jan 14 21:18 /dev/hdisk6crw------- 1 root system 21, 10 Jan 14 21:18 /dev/rhdisk6

{node1:root}/ #

Values such as 21, 6 are the major and minor number for hdisk2 on node1.

3.1.3.2. Identify Major and Minor number of hdisk on each node 41



















21 is the major number, corresponding to this disk. 6 is the minor number.

Disks LUN IDNumber


Disk

MajorNum.

MinorNum.

Node 2Corresponding

Disk

MajorNum.

MinorNum.

OCR 1 L1 /dev/ocr_disk1 hdisk2 21 6 hdisk2OCR 2 L2 /dev/ocr_disk2 hdisk3 21 7 hdisk3Voting1

L3 /dev/voting_disk1 hdisk4 21 8 hdisk4

Voting2


Voting3


On node2 :

{node2:root}# for i in 2 3 4 5 6 do ls -la /dev/*hdisk$i done

brw------- 1 root system 21, 6 Jan 11 16:12 /dev/hdisk2crw------- 1 root system 21, 6 Jan 11 16:12 /dev/rhdisk2brw------- 1 root system 21, 7 Jan 11 16:12 /dev/hdisk3crw------- 1 root system 21, 7 Jan 11 16:12 /dev/rhdisk3brw------- 1 root system 21, 8 Jan 11 16:12 /dev/hdisk4crw------- 1 root system 21, 8 Jan 11 16:12 /dev/rhdisk4brw------- 1 root system 21, 9 Jan 11 16:12 /dev/hdisk5crw------- 1 root system 21, 9 Jan 11 16:12 /dev/rhdisk5brw------- 1 root system 21, 10 Jan 14 22:20 /dev/hdisk6crw------- 1 root system 21, 10 Jan 14 22:20 /dev/rhdisk6

Disks LUN IDNumber


Disk

MajorNum.

MinorNum.

Node 2Corresponding

Disk

MajorNum.

MinorNum.

OCR 1 L1 /dev/ocr_disk1 hdisk2 21 6 hdisk2 21 6OCR 2 L2 /dev/ocr_disk2 hdisk3 21 7 hdisk3 21 7Voting1

L3 /dev/voting_disk1 hdisk4 21 8 hdisk4 21 8

Voting2


Voting3


Diisks may have different names from one node to another for example L1 could correspond to hdisk2on node1, and hdisk1 on node2, etc

3.1.3.3. Create Unique Virtual Device to access the same LUN from each node

THEN from the table, for node1 we need to create virtual devices :

3.1.3.3. Create Unique Virtual Device to access the same LUN from each node 42





































Disks LUN IDNumber


Disk

MajorNum.

MinorNum.

Node 2Corresponding

Disk

MajorNum.

MinorNum.

OCR 1 L1 /dev/ocr_disk1 hdisk2 21 6 hdisk2 21 6OCR 2 L2 /dev/ocr_disk2 hdisk3 21 7 hdisk3 21 7Voting1


Voting2


Voting3


Now we will create create some virtual devices on each node called :

/dev/ocr_disk1/dev/ocr_disk2/dev/voting_disk1/dev/voting_disk2/dev/voting_disk3

We need to use major and minor number of the hdisks which will make the link between the virtual devicesand the hdisks.

Using the command :

mknod Device_Name c MajNum? MinNum?

On the first node, as root user

{node1:root}/ # mknod /dev/ocr_disk1 c 21 6{node1:root}/ # mknod /dev/ocr_disk2 c 21 7{node1:root}/ # mknod /dev/voting_disk1 c 21 8{node1:root}/ # mknod /dev/voting_disk2 c 21 9{node1:root}/ # mknod /dev/voting_disk3 c 21 10

On the second node, as root user

{node2:root}/ # mknod /dev/ocr_disk1 c 21 6{node2:root}/ # mknod /dev/ocr_disk2 c 21 7{node2:root}/ # mknod /dev/voting_disk1 c 21 8{node2:root}/ # mknod /dev/voting_disk2 c 21 9{node2:root}/ # mknod /dev/voting_disk3 c 21 10

3.1.3.4. Set Ownership / Permissions on Virtual Devices

For each node in the cluster :

Set the ownerships of the created virtual devices

{node1:root}/ # chown root:oinstall /dev/ocr_disk1

3.1.3.4. Set Ownership / Permissions on Virtual Devices 43



















http://globaldc.oracle.com/perl/twiki/edit/RACGuides/MajNum?topicparent=RACGuides.Rac11gR1OnAIX

http://globaldc.oracle.com/perl/twiki/edit/RACGuides/MinNum?topicparent=RACGuides.Rac11gR1OnAIX

{node1:root}/ # chown root:oinstall /dev/ocr_disk2{node1:root}/ # chown crs:dba /dev/voting_disk1{node1:root}/ # chown crs:dba /dev/voting_disk2{node1:root}/ # chown crs:dba /dev/voting_disk3

And repeat on node 2

THEN set read/write permissions on the created virtual devices

{node1:root}/ # chmod 640 /dev/ocr_disk1{node1:root}/ # chmod 640 /dev/ocr_disk2{node1:root}/ # chmod 660 /dev/voting_disk1{node1:root}/ # chmod 660 /dev/voting_disk2{node1:root}/ # chmod 660 /dev/voting_disk3

And repeat on node 2

THEN check that settings are applied on node1 :

{node1:root}/ # ls -la /dev/* | grep "21, 6"brw------- 1 root system 21, 6 Jan 11 15:58 /dev/hdisk2crw-r----- 1 root oinstall 21, 6 Feb 06 17:43 /dev/ocr_disk1crw------- 1 root system 21, 6 Jan 11 15:58 /dev/rhdisk2

And repeat for all the disks on both nodes.

Note: the important permissions are on the virtual device. The permissions on the underlying disks arenot relevant here.

This is true of Linux and most UNIXs

3.1.3.5. Formating the virtual devices (zeroing)

Now, we zero out the virtual devices :

On node 1

{node1:root}/ # for i in 1 2> do> dd if=/dev/zero of=/dev/ocr_disk$i bs=1024 count=300> done300+0 records in.300+0 records out.300+0 records in.300+0 records out.

On node 1

{node1:root}/ # for i in 1 2 3> do

3.1.3.5. Formating the virtual devices (zeroing) 44

> dd if=/dev/zero of=/dev/voting_disk$i bs=1024 count=300> done300+0 records in.300+0 records out.300+0 records in.300+0 records out.300+0 records in.300+0 records out.

{node1:root}/ #

3.2. ASM disks

3.2.1. *About ASM Instances parameters files, pfile or spfile ?

*Note the ASM spfile (unlike the database spfile) cannot be in ASM it must be a separate disk created likethose for the clusterware devices.*Permissions on this disk must be at least 'rw' for the asm user.

Using the storage administration console, you have to create :

LUN for the ASM spfile (100MB size)This disk will contain the shared ASM instance spfile

1.

A set of LUNs for disks to be used with ASM.Disks/LUNs for ASM can be either

/dev/rhdisk (raw disks)

or

/dev/ASM_DISK1 (virtual devices) for ease of administration.

The following screen shows the LUN mapping for nodes used in our cluster.These IDs will help us on to identify which hdisk will be used.

Disks LUN ID Number LUN SIZEASM Spfile L0 100 MBDisk 1 for ASM L6 4 GBDisk 2 for ASM L7 4 GBDisk 3 for ASM L8 4 GBDisk 4 for ASM L9 4 GBDisk 5 for ASM LA (meaning Lun 10) 4 GBDisk 6 for ASM LB (meaning Lun 11) 4 GBDisk 7 for ASM LC (meaning Lun 12) 4 GBDisk 8 for ASM LD (meaning Lun 13) 4 GBFollow the steps for identifying and mapping the LUNs on each node and creating the virtual devices asdescribed in the clusterware devices.

The only diffrences are that the spfile for ASM and the ASM data disks must have permissions of at least r/wfor the ASM and Oracle user.

When you are done:

3.2. ASM disks 45




To check all hdisks in one shot, use following shell script :

for i in 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 do lsattr -El hdisk$i | grep reserve_policy | awk '{print $1,$2 }'| read rp1 rp2 lsattr -El hdisk$i | grep pvid | awk '{print $1,$2 }'| read pv1 pv2 lsattr -El hdisk$i | grep lun_id | awk '{print $1,$2 }'| read li1 li2 if [ "$li1" != "" ] then echo hdisk$i' -> '$li1' = '$li2' / '$rp1' = '$rp2' / '$pv1' = '$pv2 fi done

hdisk1 -> lun_id = 0x0000000000000000 / reserve_policy = no_reserve / pvid = none hdisk2 -> lun_id = 0x0001000000000000 / reserve_policy = no_reserve / pvid = none hdisk3 -> lun_id = 0x0002000000000000 / reserve_policy = no_reserve / pvid = none hdisk4 -> lun_id = 0x0003000000000000 / reserve_policy = no_reserve / pvid = none hdisk5 -> lun_id = 0x0004000000000000 / reserve_policy = no_reserve / pvid = none hdisk6 -> lun_id = 0x0005000000000000 / reserve_policy = no_reserve / pvid = none hdisk7 -> lun_id = 0x0006000000000000 / reserve_policy = no_reserve / pvid = none hdisk8 -> lun_id = 0x0007000000000000 / reserve_policy = no_reserve / pvid = none hdisk9 -> lun_id = 0x0008000000000000 / reserve_policy = no_reserve / pvid = none hdisk10 -> lun_id = 0x0009000000000000 / reserve_policy = no_reserve / pvid = none hdisk11 -> lun_id = 0x000a000000000000 / reserve_policy = no_reserve / pvid = nonehdisk12 -> lun_id = 0x000b000000000000 / reserve_policy = no_reserve / pvid = nonehdisk13 -> lun_id = 0x000c000000000000 / reserve_policy = no_reserve / pvid = none

And a similar result is expected from node 2

3.2.2.Recommendations, hints and tips

How to free hdisks at the AIX level, when the hdisks are not needed anymore or need to be reset after afailed CRS installation ?

You must reset the hdisk header having the OCR or Voting disk stamp :

{node1:root}/ #dd if=/dev/zero of=/dev/rhdisk2 bs=8192 count=25000 & 25000+0 records in. 25000+0records out.

THEN query on the hdisk header will return nothing more than all lines full of ‘0’:

{node1:root}/ # lquerypv -h /dev/rhdisk2

00000000 00000000 00000000 00000000 00000000 |................|00000010 00000000 00000000 00000000 00000000 |................|00000020 00000000 00000000 00000000 00000000 |................|00000030 00000000 00000000 00000000 00000000 |................|00000040 00000000 00000000 00000000 00000000 |................|00000050 00000000 00000000 00000000 00000000 |................|00000060 00000000 00000000 00000000 00000000 |................|00000070 00000000 00000000 00000000 00000000 |................|

3.2.1. *About ASM Instances parameters files, pfile or spfile ? 46

00000080 00000000 00000000 00000000 00000000 |................|00000090 00000000 00000000 00000000 00000000 |................|000000A0 00000000 00000000 00000000 00000000 |................|000000B0 00000000 00000000 00000000 00000000 |................|000000C0 00000000 00000000 00000000 00000000 |................|000000D0 00000000 00000000 00000000 00000000 |................|000000E0 00000000 00000000 00000000 00000000 |................|000000F0 00000000 00000000 00000000 00000000 |................|

4. Oracle Clusterware Installation and ConfigurationThe guides include hidden sections, use the and image for each section to show/hide the section or youcan Expand all or Collapse all by clicking these buttons. This is implemented using the Twisty Plugin whichrequires Java Script to be enabled on your browser.

Next we will install the Oracle Clusterware layer.

Oracle Clusterware is an essential component of the Oracle RAC database infrastructure.

Oracle Clusterware does not require any other clustering software.

Oracle Clusterware will be installed in /crs/11.1.0 ($CRS_HOME, $ORA_CRS_HOME) on each node.

You may install the clusterware where you please with these exceptions.

Do not install the clusterware in a directory which is sub-directory of either the intended install locationfor ASM or the Database.

Failure to observe this rule could well mean that other people when removing an ASM or Oracle home willdestroy your clusterware installation.

Running rootpre.sh

When you start the installer it will ask if you have already run rootpre.sh and you should do this beforecontinuing

Show rootpre.sh output Hide rootpre.sh output

{node1:root}/distrib/SoftwareOracle/rdbms11gr1/aix/clusterware # ./rootpre.sh./rootpre.sh output will be logged in /tmp/rootpre.out_08-02-07.10:14:39Kernel extension /etc/pw-syscall.64bit_kernel is loaded.Unloading the existing extension: /etc/pw-syscall.64bit_kernel.... Oracle Kernel Extension Loader for AIX Copyright (c) 1998,1999 Oracle Corporation

Unconfigured the kernel extension successfullyUnloaded the kernel extension successfullySaving the original files in /etc/ora_save_08-02-07.10:14:39.... Copying new kernel extension to /etc.... Loading the kernel extension from /etc

Oracle Kernel Extension Loader for AIX Copyright (c) 1998,1999 Oracle CorporationSuccessfully loaded /etc/pw-syscall.64bit_kernel with kmid: 0x4245600

3.2.2.Recommendations, hints and tips 47

http://globaldc.oracle.com/perl/twiki/view/TWiki/JQueryTwistyPlugin

Successfully configured /etc/pw-syscall.64bit_kernel with kmid: 0x4245600

The kernel extension was successfuly loaded.Configuring Asynchronous I/O....Asynchronous I/O is already definedConfiguring POSIX Asynchronous I/O....Posix Asynchronous I/O is already definedChecking if group services should be configured....Nothing to configure.

4.1. CVU Pre Oracle Clusterware install check

If your deployment environment does not use HACMP, ignore the HACMP version and patch errors reportedby Cluster Verification Utility (CVU).

On AIX 5L version 5.2, the expected patch for HACMP v5.2 is IY60759. On AIX 5L version 5.3, theexpected patches for HACMP v5.2 are IY60759, IY61034, IY61770, and IY62191.

If your deployment environment does not use GPFS, ignore the GPFS version and patches errors reported byCluster Verification Utility (CVU). On AIX 5L version 5.2 and version 5.3, the expected patches for GPFS2.3.0.3 are IY63969, IY69911, and IY70276.

As the crs user

{node1:crs}./runcluvfy.sh stage -pre crsinst -n node1,node2 -r 11gR1 -verbose > /tmp/cluvfy_oracle.txt

Show CVU -pre crsinst output Hide CVU output

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "node1" Destination Node Reachable? ------------------------------------ ------------------------ node1 yes node2 yesResult: Node reachability check passed from node "node1".

Checking user equivalence...

Check: User equivalence for user "oracle" Node Name Comment ------------------------------------ ------------------------ node2 passed node1 passedResult: User equivalence check passed for user "oracle".

Checking administrative privileges...

Check: Existence of user "oracle" Node Name User Exists Comment ------------ ------------------------ ------------------------ node1 yes passed

4. Oracle Clusterware Installation and Configuration 48

node2 yes passed Result: User existence check passed for "oracle".

Check: Existence of group "oinstall" Node Name Status Group ID ------------ ------------------------ ------------------------ node1 exists 501 node2 exists 501 Result: Group existence check passed for "oinstall".

Check: Membership of user "oracle" in group "oinstall" [as Primary] Node Name User Exists Group Exists User in Group Primary Comment ---------------- ------------ ------------ ------------ ------------ ------------ tbas11b yes yes yes no failed tbas11a yes yes yes no failed Result: Membership check for user "oracle" in group "oinstall" [as Primary] failed.

Administrative privileges check failed

Checking node connectivity...



Check: Node connectivity of subnet "10.3.25.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- node1:en0 node2:en0 yesResult: Node connectivity check passed for subnet "10.3.25.0" with node(s) node1,node2.



4.1. CVU Pre Oracle Clusterware install check 49

Suitable interfaces for VIP on subnet "20.20.25.0":node1 en2:20.20.25.81node2 en2:20.20.25.82



Result: Node connectivity check passed.

Checking system requirements for 'crs'...

Check: Kernel version Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- node1 AIX 5.3 AIX 5.2 passed node2 AIX 5.3 AIX 5.2 passedResult: Kernel version check passed.

Check: System architecture Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- node1 powerpc powerpc passed node2 powerpc powerpc passed Result: System architecture check passed.

Check: Total memory Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- node1 2GB (2097152KB) 512MB (524288KB) passed node2 2GB (2097152KB) 512MB (524288KB) passed Result: Total memory check passed.

Check: Swap space Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- node1 1024GB (1073741824KB) 1GB (1048576KB) passed node2 1024GB (1073741824KB) 1GB (1048576KB) passed Result: Swap space check passed.

Check: Free disk space in "/tmp" dir Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- node1 400.35MB (409960KB) 400MB (409600KB) passed node2 400.35MB (409960KB) 400MB (409600KB) passed Result: Free disk space check passed.

Check: Free disk space in "/oracle" dir Node Name Available Required Comment ------------ ------------------------ ------------------------ ----------


node1 4.63GB (4860044KB) 4GB (4194304KB) passed node2 4.63GB (4860044KB) 4GB (4194304KB) passed Result: Free disk space check passed.

Check: Package existence for "vacpp.cmp.core:7.0.0.2" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 vacpp.cmp.core:6.0.0.0 failed node2 vacpp.cmp.core:6.0.0.0 failed Result: Package existence check failed for "vacpp.cmp.core:7.0.0.2".

Check: Operating system patch for "IY65361 " Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- node1 unknown IY65361 failed node2 unknown IY65361 failed Result: Operating system patch check failed for "IY65361 ".

Check: Package existence for "vac.C:7.0.0.2" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 vac.C:7.0.0.2 passed node2 vac.C:7.0.0.2 passed Result: Package existence check passed for "vac.C:7.0.0.2".Check: Package existence for "xlC.aix50.rte:7.0.0.4" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 xlC.aix50.rte:7.0.0.4 passed node2 xlC.aix50.rte:7.0.0.4 passed Result: Package existence check passed for "xlC.aix50.rte:7.0.0.4".

Check: Package existence for "xlC.rte:7.0.0.1" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 xlC.rte:7.0.0.1 passed node2 xlC.rte:7.0.0.1 passed Result: Package existence check passed for "xlC.rte:7.0.0.1".

Check: Package existence for "gpfs.base:2.3.0.3" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 gpfs.base:2.3.0.5 passed node2 gpfs.base:2.3.0.5 passedResult: Package existence check passed for "gpfs.base:2.3.0.3".

Check: Operating system patch for "IY63969" Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- node1 IY63969:gpfs.baseIY63969:gpfs.docs.dataIY63969:gpfs.msg.en_US IY63969 passed node2 IY63969:gpfs.baseIY63969:gpfs.docs.dataIY63969:gpfs.msg.en_US IY63969 passed Result: Operating system patch check passed for "IY63969".

Check: Operating system patch for "IY69911" Node Name Applied Required Comment


------------ ------------------------ ------------------------ ---------- node1 IY69911:gpfs.base IY69911 passed node2 IY69911:gpfs.base IY69911 passed Result: Operating system patch check passed for "IY69911".

Check: Operating system patch for "IY70276" Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- node1 IY70276:gpfs.base IY70276 passed node2 IY70276:gpfs.base IY70276 passed Result: Operating system patch check passed for "IY70276".

Check: Package existence for "cluster.license:5.2.0.0" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 missing failed node2 missing failed Result: Package existence check failed for "cluster.license:5.2.0.0".

Check: Operating system patch for "IY60759" Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- node1 unknown IY60759 failed node2 unknown IY60759 failed Result: Operating system patch check failed for "IY60759".

Check: Operating system patch for "IY61034" Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- node1 IY61034:bos.mpIY61034:bos.mp64 IY61034 passed node2 IY61034:bos.mpIY61034:bos.mp64 IY61034 passed Result: Operating system patch check passed for "IY61034".

Check: Operating system patch for "IY61770" Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- node1 IY61770:rsct.basic.rteIY61770:rsct.core.errmIY61770:rsct.core.hostrmIY61770:rsct.core.rmcIY61770:rsct.core.secIY61770:rsct.core.sensorrmIY61770:rsct.core.utils IY61770 passed Node2 IY61770:rsct.basic.rteIY61770:rsct.core.errmIY61770:rsct.core.hostrmIY61770:rsct.core.rmcIY61770:rsct.core.secIY61770:rsct.core.sensorrmIY61770:rsct.core.utils IY61770 passed Result: Operating system patch check passed for "IY61770".

Check: Operating system patch for "IY62191" Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- node1 IY62191:bos.adt.profIY62191:bos.rte.libpthreads IY62191 passed node2 IY62191:bos.adt.profIY62191:bos.rte.libpthreads IY62191 passed Result: Operating system patch check passed for "IY62191".

Check: Package existence for "ElectricFence-2.2.2-1:2.2.2" Node Name Status Comment


------------------------------ ------------------------------ ---------------- node1 missing failed node2 missing failed Result: Package existence check failed for "ElectricFence-2.2.2-1:2.2.2".

Check: Package existence for "xlfrte:9.1" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 xlfrte:8.1.1.4 failed node2 xlfrte:8.1.1.4 failed Result: Package existence check failed for "xlfrte:9.1".

Check: Package existence for "gdb-6.0-1:6.0" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 missing failed node2 missing failed Result: Package existence check failed for "gdb-6.0-1:6.0".

Check: Package existence for "make-3.80-1:3.80" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 missing failed node2 missing failed Result: Package existence check failed for "make-3.80-1:3.80".

Check: Package existence for "freeware.gnu.tar.rte:1.13.0.0" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 missing failed node2 missing failed Result: Package existence check failed for "freeware.gnu.tar.rte:1.13.0.0".

Check: Package existence for "Java14_64.sdk:1.4.2.1" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 missing failed node2 missing failed Result: Package existence check failed for "Java14_64.sdk:1.4.2.1".

Check: Package existence for "Java131.rte.bin:1.3.1.16" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 missing failed node2 missing failed Result: Package existence check failed for "Java131.rte.bin:1.3.1.16".

Check: Package existence for "Java14.sdk:1.4.2.2" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 Java14.sdk:1.4.2.10 failed node2 Java14.sdk:1.4.2.10 failed Result: Package existence check failed for "Java14.sdk:1.4.2.2".


Check: Operating system patch for "IY65305" Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- node1 IY65305:Java14.sdk IY65305 passed node2 IY65305:Java14.sdk IY65305 passed Result: Operating system patch check passed for "IY65305".



Check: Package existence for "mqm.server.rte:5.3" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 missing failed node2 missing failed Result: Package existence check failed for "mqm.server.rte:5.3".

Check: Package existence for "mqm.client.rte:5.3" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 missing failed node2 missing failed Result: Package existence check failed for "mqm.client.rte:5.3".

Check: Package existence for "sna.rte:6.1.0.4" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 missing failed node2 missing failed Result: Package existence check failed for "sna.rte:6.1.0.4".

Check: Package existence for "bos.net.tcp.server" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 bos.net.tcp.server:5.3.0.30 passed node2 bos.net.tcp.server:5.3.0.30 passed Result: Package existence check passed for "bos.net.tcp.server".

Check: Operating system patch for "IY44599" Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- node1 unknown IY44599 failed


node2 unknown IY44599 failed Result: Operating system patch check failed for "IY44599".


Check: Operating system patch for "IY58143" Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- node1 IY58143:X11.Dt.libIY58143:X11.base.rteIY58143:bos.acctIY58143:bos.adt.includeIY58143:bos.adt.libmIY58143:bos.adt.profIY58143:bos.alt_disk_install.rteIY58143:bos.diag.comIY58143:bos.mp64IY58143:bos.mpIY58143:bos.net.ewlm.rteIY58143:bos.net.ipsec.keymgtIY58143:bos.net.ipsec.rteIY58143:bos.net.nfs.clientIY58143:bos.net.nfs.serverIY58143:bos.net.tcp.clientIY58143:bos.net.tcp.serverIY58143:bos.net.tcp.smitIY58143:bos.perf.libperfstatIY58143:bos.perf.perfstatIY58143:bos.perf.toolsIY58143:bos.rte.archiveIY58143:bos.rte.bind_cmdsIY58143:bos.rte.bootIY58143:bos.rte.controlIY58143:bos.rte.filesystemIY58143:bos.rte.installIY58143:bos.rte.libcIY58143:bos.rte.lvmIY58143:bos.rte.manIY58143:bos.rte.methodsIY58143:bos.rte.securityIY58143:bos.rte.serv_aidIY58143:bos.sysmgt.nim.clientIY58143:bos.sysmgt.quotaIY58143:bos.sysmgt.serv_aidIY58143:bos.sysmgt.sysbrIY58143:devices.chrp.base.rteIY58143:devices.chrp.pci.rteIY58143:devices.chrp.vdevice.rteIY58143:devices.common.IBM.atm.rteIY58143:devices.common.IBM.ethernet.rteIY58143:devices.common.IBM.fc.rteIY58143:devices.common.IBM.fda.diagIY58143:devices.common.IBM.mpio.rteIY58143:devices.fcp.disk.rteIY58143:devices.pci.00100f00.rteIY58143:devices.pci.14100401.diagIY58143:devices.pci.14103302.rteIY58143:devices.pci.14106602.rteIY58143:devices.pci.14106902.rteIY58143:devices.pci.14107802.rteIY58143:devices.pci.1410ff01.rteIY58143:devices.pci.22106474.rteIY58143:devices.pci.33103500.rteIY58143:devices.pci.4f111100.comIY58143:devices.pci.77101223.comIY58143:devices.pci.99172704.rteIY58143:devices.pci.c1110358.rteIY58143:devices.pci.df1000f7.comIY58143:devices.pci.df1000f7.diagIY58143:devices.pci.df1000fa.rteIY58143:devices.pci.e414a816.rteIY58143:devices.scsi.disk.rteIY58143:devices.vdevice.IBM.l-lan.rteIY58143:devices.vdevice.IBM.vscsi.rteIY58143:devices.vdevice.hvterm1.rteIY58143:devices.vtdev.scsi.rteIY58143:sysmgt.websm.appsIY58143:sysmgt.websm.frameworkIY58143:sysmgt.websm.rteIY58143:sysmgt.websm.webaccess IY58143 passed Node2 IY58143:X11.Dt.libIY58143:X11.base.rteIY58143:bos.acctIY58143:bos.adt.includeIY58143:bos.adt.libmIY58143:bos.adt.profIY58143:bos.alt_disk_install.rteIY58143:bos.diag.comIY58143:bos.mp64IY58143:bos.mpIY58143:bos.net.ewlm.rteIY58143:bos.net.ipsec.keymgtIY58143:bos.net.ipsec.rteIY58143:bos.net.nfs.clientIY58143:bos.net.nfs.serverIY58143:bos.net.tcp.clientIY58143:bos.net.tcp.serverIY58143:bos.net.tcp.smitIY58143:bos.perf.libperfstatIY58143:bos.perf.perfstatIY58143:bos.perf.toolsIY58143:bos.rte.archiveIY58143:bos.rte.bind_cmdsIY58143:bos.rte.bootIY58143:bos.rte.controlIY58143:bos.rte.filesystemIY58143:bos.rte.installIY58143:bos.rte.libcIY58143:bos.rte.lvmIY58143:bos.rte.manIY58143:bos.rte.methodsIY58143:bos.rte.securityIY58143:bos.rte.serv_aidIY58143:bos.sysmgt.nim.clientIY58143:bos.sysmgt.quotaIY58143:bos.sysmgt.serv_aidIY58143:bos.sysmgt.sysbrIY58143:devices.chrp.base.rteIY58143:devices.chrp.pci.rteIY58143:devices.chrp.vdevice.rteIY58143:devices.common.IBM.atm.rteIY58143:devices.common.IBM.ethernet.rteIY58143:devices.common.IBM.fc.rteIY58143:devices.common.IBM.fda.diagIY58143:devices.common.IBM.mpio.rteIY58143:devices.fcp.disk.rteIY58143:devices.pci.00100f00.rteIY58143:devices.pci.14100401.diagIY58143:devices.pci.14103302.rteIY58143:devices.pci.14106602.rteIY58143:devices.pci.14106902.rteIY58143:devices.pci.14107802.rteIY58143:devices.pci.1410ff01.rteIY58143:devices.pci.22106474.rteIY58143:devices.pci.33103500.rteIY58143:devices.pci.4f111100.comIY58143:devices.pci.77101223.comIY58143:devices.pci.99172704.rteIY58143:devices.pci.c1110358.rteIY58143:devices.pci.df1000f7.comIY58143:devices.pci.df1000f7.diagIY58143:devices.pci.df1000fa.rteIY58143:devices.pci.e414a816.rteIY58143:devices.scsi.disk.rteIY58143:devices.vdevice.IBM.l-lan.rteIY58143:devices.vdevice.IBM.vscsi.rteIY58143:devices.vdevice.hvterm1.rteIY58143:devices.vtdev.scsi.rteIY58143:sysmgt.websm.appsIY58143:sysmgt.websm.frameworkIY58143:sysmgt.websm.rteIY58143:sysmgt.websm.webaccess IY58143 passed





Check: Operating system patch for "IY59386" Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- node1 IY59386:bos.rte.bind_cmds IY59386 passed node2 IY59386:bos.rte.bind_cmds IY59386 passed Result: Operating system patch check passed for "IY59386".

Check: Package existence for "bos.adt.base" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 bos.adt.base:5.3.0.30 passed node2 bos.adt.base:5.3.0.30 passed Result: Package existence check passed for "bos.adt.base".

Check: Package existence for "bos.adt.lib" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 bos.adt.lib:5.3.0.30 passed node2 bos.adt.lib:5.3.0.30 passed Result: Package existence check passed for "bos.adt.lib".

Check: Package existence for "bos.adt.libm" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 bos.adt.libm:5.3.0.30 passed node2 bos.adt.libm:5.3.0.30 passed Result: Package existence check passed for "bos.adt.libm".

Check: Package existence for "bos.perf.libperfstat" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 bos.perf.libperfstat:5.3.0.30 passed node2 bos.perf.libperfstat:5.3.0.30 passed Result: Package existence check passed for "bos.perf.libperfstat".

Check: Package existence for "bos.perf.perfstat" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 bos.perf.perfstat:5.3.0.30 passed


node2 bos.perf.perfstat:5.3.0.30 passed Result: Package existence check passed for "bos.perf.perfstat".

Check: Package existence for "bos.perf.proctools" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 bos.perf.proctools:5.3.0.30 passed node2 bos.perf.proctools:5.3.0.30 passed Result: Package existence check passed for "bos.perf.proctools".

Check: Package existence for "rsct.basic.rte" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 rsct.basic.rte:2.4.3.0 passed node2 rsct.basic.rte:2.4.3.0 passed Result: Package existence check passed for "rsct.basic.rte".

Check: Package existence for "perl.rte:5.0005" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 perl.rte:5.8.2.30 passed node1 perl.rte:5.8.2.30 passed Result: Package existence check passed for "perl.rte:5.0005".



Check: Package existence for "python-2.2-4:2.2" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 missing failed node2 missing failed Result: Package existence check failed for "python-2.2-4:2.2".

Check: Package existence for "freeware.zip.rte:2.3" Node Name Status Comment ------------------------------ ------------------------------ ---------------- node1 missing failed node2 missing failed Result: Package existence check failed for "freeware.zip.rte:2.3".

Check: Package existence for "freeware.gcc.rte:3.3.2.0" Node Name Status Comment


------------------------------ ------------------------------ ---------------- node1 missing failed node2 missing failed Result: Package existence check failed for "freeware.gcc.rte:3.3.2.0".

Check: Group existence for "dba" Node Name Status Comment ------------ ------------------------ ------------------------ node1 exists passed node2 exists passed Result: Group existence check passed for "dba".

Check: User existence for "nobody" Node Name Status Comment ------------ ------------------------ ------------------------ node1 exists passed node2 exists passed Result: User existence check passed for "nobody".

System requirement failed for 'crs'

Pre-check for cluster services setup was unsuccessful on all the nodes.

4.2.Oracle Clusterware Install

As root user on each node, create a symbolic link from /usr/sbin/lsattr to /etc/lsattr

ln -s /usr/sbin/lsattr /etc/lsattr

'/etc/lsattr' is used in the vip check action.

On each node :

Run the AIX command "/usr/sbin/slibclean" as "root" to clean all unreferenced libraries from memory.

{node1:root}/ # /usr/sbin/slibclean

Repeat on node2

Login as crs or oracle user (crs in our case) and follow the procedure below.

Setup and export your DISPLAY, TMP and TEMP variables.

Check your TMP destination e.g. /tmp has enough free space, about 500Mb on each node

If AIX5L? release 5.3 is used, modify the files oraparam.ini, and cluster.ini in Disk1/installer

update entries AIX5200 to AIX5300 in both files

As the crs user:

{node1:crs}./runInstaller

4.2.Oracle Clusterware Install 58

http://globaldc.oracle.com/perl/twiki/edit/RACGuides/AIX5L?topicparent=RACGuides.Rac11gR1OnAIX

4.2.1. Installer screenshots.

Notes•

Action• Click on Next•

Notes By default, the Oracle Universal installer may try to create /oraInventory directory,• but OUI has no rights to do so. Then you may see this message.• Action• Click 'OK' to modify Inventory location destination.•

4.2.1. Installer screenshots. 59

Action• Specify Inventory directory and group where you want to create the inventory directory. Typicallythis will be $ORACLE_BASE/oraInventory. Thus: /oracle/oraInventoryThe operating system group name should be set as oinstallThen click Next ... to Continue ...

•

Notes Specify Home DetailsSpecify an ORACLE_HOME name and destination directory for the CRS installation.The destination directory should not be within the $ORACLE_BASE subtree.

•


Add Product Languages if necessay.Action Then click Next ...•

Notes Product-Specific Prerequisite Checks :All checks should be have the status 'Succeeded'.

•

Action Then click Next ...•

Action• Click 'Edit' to check the network entries for node1.•


After checking the entries for node 1 click 'Add'•

Action• Specify the Cluster configuration :• You must give a unique name to the cluster (in our example rat_cluster)• Edit the default node to validate entries• Add entries for any new nodes, node2 in our case•

Notes If you have problems at this stage when clicking on Next (error messages)Check your network configuration, and User equivalence

•

For each cluster node, you must :specify one by one:Public Node Name: Public corresponds to the IP address linked to the public network,usually the IP returned by hostname.Private Node Name: Private corresponds to the IP address linked to the RAC interconnect.Virtual Host Name: The Virtual Host Name corresponds to a free IP address on the public networkto be used for the Oracle clusterware VIP.

•

Actions When done, click Next ...•


Notes• In our example:• en0 (10.3.25.0) must exist as 'Public' on each node.• en1 (10.10.25.0) must exist as 'Private' on each node.• en2 (20.20.25.0) 'Do Not Use'•

Action• Select an interface and click the edit button•

Action: Edit to set en1 to be private•


Action: set en2 to be 'Not Used'•

Action Check and then click Next•


Notes• Specify OCR Configuration, by selecting in our caseNormal Redundancy (OCR mirroring by Oracle Clusterware)Specify raw disk locations as follows :

•

/dev/ocr_disk1• /dev/ocr_disk2Remember we prepared these disks in chapter 3 of this guideIf problems happens at this stage, verify that locations specified exist,and are reachable from each AIX node, with right read/write access and user/group owner.

•

Action Then click Next ...•


Action• Specify Voting Disk Configuration, by selecting in our case:Normal Redundancy (Voting copies managed by Oracle Clusterware)If 'NORMAL Redundancy' is selected, specify the raw disk locations as follows:

•

/dev/voting_disk1• /dev/voting_disk2• /dev/voting_disk3• Then click Next ...•


Action• Click 'Install'•

Notes Now wait a while•

Notes Progressing•


Notes Nearly done...•

Action Execute the Configuration Script root.sh:• KEEP THIS WINDOWS OPEN• Execute the script on each node waiting for each to succeed before running the next one.•

On node1

As root, Execute /crs/11.1.0/root.sh

If all is well you should see output like this:

{node1:root}/crs/11.1.0 # ./root.sh


WARNING: directory '/crs' is not owned by rootChecking to see if Oracle CRS stack is already configured/etc/oracle does not exist. Creating it now.Setting the permissions on OCR backup directorySetting up Network socket directoriesOracle Cluster Registry configuration upgraded successfullyThe directory '/crs' is not owned by root. Changing owner to rootSuccessfully accumulated necessary OCR keys.Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.node <nodenumber>: <nodename> <private interconnect name> <hostname>node 1: node1 node1-rac node1 node 2: node2 node2-rac node2Creating OCR keys for user 'root', privgrp 'system'..Operation successful.Now formatting voting device: /dev/voting_disk1Now formatting voting device: /dev/voting_disk2Now formatting voting device: /dev/voting_disk3Format of 3 voting devices complete.Startup will be queued to init within 30 seconds.Adding daemons to inittabAdding daemons to inittabExpecting the CRS daemons to be up within 600 seconds.Cluster Synchronization Services is active on these nodes. node1Cluster Synchronization Services is inactive on these nodes. node2 Local node checking complete. Run root.sh on remaining nodes to start CRS daemons.{node1:root}/crs/11.1.0 #

Don't worry about 'WARNING: directory '/crs' is not owned by root'This is just an informational message.

On node2

{node2:root}/crs/11.1.0 # ./root.shWARNING: directory '/crs' is not owned by rootChecking to see if Oracle CRS stack is already configured/etc/oracle does not exist. Creating it now.

Setting the permissions on OCR backup directorySetting up Network socket directoriesOracle Cluster Registry configuration upgraded successfullyThe directory '/crs' is not owned by root. Changing owner to rootclscfg: EXISTING configuration version 4 detected.clscfg: version 4 is 11 Release 1.Successfully accumulated necessary OCR keys.Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.node <nodenumber>: <nodename> <private interconnect name> <hostname>node 1: node1 node1-rac node1node 2: node2 node2-rac node2clscfg: Arguments check out successfully.

NO KEYS WERE WRITTEN. Supply -force parameter to override.-force is destructive and will destroy any previous clusterconfiguration.


Oracle Cluster Registry for cluster has already been initializedStartup will be queued to init within 30 seconds.Adding daemons to inittabAdding daemons to inittabExpecting the CRS daemons to be up within 600 seconds.Cluster Synchronization Services is active on these nodes. node1 node2Cluster Synchronization Services is active on all the nodes.Waiting for the Oracle CRSD and EVMD to startOracle CRS stack installed and running under init(1M)Running vipca(silent) for configuring nodeapps

Creating VIP application resource on (2) nodes...Creating GSD application resource on (2) nodes...Creating ONS application resource on (2) nodes...Starting VIP application resource on (2) nodes...Starting GSD application resource on (2) nodes...Starting ONS application resource on (2) nodes...

Done.{node2:root}/crs/11.1.0 #

Notes•

IF CRS is not active at the end of the root.sh script :Check your network , shared disks configuration,and owner and access permissions (read/write) on OCR and Voting disks from each participatingnode. Re-execute the root.sh script on the node having the problems.If CRS starts on one node, but not on the others:Check shared Disks (OCR/Voting) for concurrent read/write access from all nodes, using unix ddcommand.

•

If ASM or GPFS is implemented with HACMP installed and configured for other purposesthen having a database on concurrent raw devices, You must declare disks ressources in HACMP to be able tostart the CRS (CSS).

If ASM or GPFS is implemented, and HACMP is installed but not used at all, then remove HACMP ordeclare disks resources in HACMP to be able to start the CRS (CSS).

The IBM AIX clustering layer, HACMP filesets, MUST NOT be installed if you've chosen an implementationwithout HACMP.

If this layer is implemented for other purpose, disks resources necessary to install and run CRS data will haveto be part of an HACMP volume group resource. If you have previously installed HACMP, you must remove:

HACMP filesets (cluster.es.*)• rsct.hacmp.rte• rsct.compat.basic.hacmp.rte• rsct.compat.clients.hacmp.rte•

If you ran a first installation of the Oracle Clusterware (CRS) with HACMP installed,


Check if /opt/ORCLcluster directory does exist and if so, remove it on all nodes.

To be able to run the root.sh script again you must :

First diagnose and fix the underlying problem.

Then follow Metalink Note 239998.1 - 10g RAC: How to Clean Up After a Failed CRS Install to properlyclean the system.

Then start the installation again.

However if all is well return to the installer screen and click OK

Notes• Configuration Assistants :If all or parts of assistants fail or are not executed, check for problems in thelog file:/oracle/oraInventory/logs/installActions?????.log (as shown on the Installer window) and solve them.Check also /crs/11.1.0/cfgtoollogs/configToolFailedCommands{node1:crs}/crs/11.1.0/cfgtoollogs

{node1:crs} cat configToolAllCommands Copyright (c) 1999, 2005, Oracle. All rights reserved. /crs/11.1.0/bin/racgons add_config node1:6200 node2:6200 /crs/11.1.0/bin/oifcfg setif -global en0/10.3.25.0:public en1/10.10.25.0:cluster_interconnect {node1:crs}/crs/11.1.0/cfgtoollogs #

•

If you miss those steps or have closed the previous screen of the Installer,you will have to run them manually before moving to the next step.Just adapt the lines with your own settings (node names, public/private network).On one node as crs user :For 'Oracle notification Server Configuration Assistant'


crs/11.1.0/bin/racgons add_config node1:6200 node2:6200• For 'crs Private Interconnect Assistant'*/crs/11.1.0/bin/oifcfg setif -global en0/10.3.25.0:public en1/10.25.25.0:cluster_interconnect

•

For 'crs Cluster Verification Utility'/crs/11.1.0/bin/cluvfy stage -post crsinst -n node1,node2

•

Action: click Exit ...•

4.3.Post Installation operations

4.3.1.Update the Clusterware unix user .profile

To be done on each node for the oracrs user (in our case). Add the following entries to the$HOME/.profile file

export CRS_HOME=/crs/11.1.0export ORACLE_CRS_HOME=$CRS_HOMEexport ORACLE_HOME=$ORA_CRS_HOMEexport LD_LIBRARY_PATH=$CRS_HOME/lib:$CRS_HOME/lib32export LIBPATH=$LD_LIBRARY_PATHexport PATH=$CRS_HOME/bin:$PATH

Disconnect from the oracrs user, and reconnect to load the modified $HOME/.profile

4.3.2.Cluster Ready Services Health Check

Check CRS processes on each nodes

{node1:crs} ps -ef|grep crs

crs 90224 393346 1 Mar 07 - 25:17 /crs/11.1.0/bin/ocssd.bin

crs 188644 237604 0 Mar 07 - 0:05 /crs/11.1.0/bin/evmlogger.bin -o /crs/11.1.0/evm/log/evmlogger.info -l /crs/11.1.0/evm/log/evmlogger.log

crs 225408 246014 0 Mar 07 - 1:57 /crs/11.1.0/bin/oclsomon.bin

4.3.Post Installation operations 72

crs 237604 327796 0 Mar 07 - 1:22 /crs/11.1.0/bin/evmd.bin

crs 246014 262190 0 Mar 07 - 0:00 /bin/sh -c cd /crs/11.1.0/log/node1/cssd/oclsomon; ulimit -c unlimited; /crs/11.1.0/bin/oclsomon || exit $?

root 282844 290996 5 Mar 07 - 251:00 /crs/11.1.0/bin/crsd.bin reboot

root 290996 1 0 Mar 07 - 0:00 /bin/sh /etc/init.crsd run

root 315392 331980 0 Mar 07 - 0:21 /crs/11.1.0/bin/oprocd run -t 1000 -m 500 -f

crs 393346 319628 0 Mar 07 - 0:00 /bin/sh -c ulimit -c unlimited; cd /crs/11.1.0/log/node1/cssd; /crs/11.1.0/bin/ocssd || exit $?

crs 397508 1 0 0:00 <defunct>

crs 622672 1 0 Mar 15 - 0:00 /crs/11.1.0/opmn/bin/ons -d

crs 626764 622672 0 Mar 15 - 0:01 /crs/11.1.0/opmn/bin/ons -d rdbms 4673770 1 0 Mar 15 - 0:00 /crs/11.1.0/bin/oclskd.bin

crs 4681948 4767816 0 16:34:59 pts/0 0:00 grep crs

crs 4767816 675942 0 16:06:06 pts/0 0:00 -ksh

crs 4796524 4767816 7 16:34:59 pts/0 0:00 ps -ef{node1:crs}/crs # ps -ef|grep d.bin

crs 90224 393346 1 Mar 07 - 25:17 /crs/11.1.0/bin/ocssd.bin

crs 237604 327796 0 Mar 07 - 1:22 /crs/11.1.0/bin/evmd.bin

root 282844 290996 7 Mar 07 - 251:00 /crs/11.1.0/bin/crsd.bin reboot

crs 4796528 4767816 0 16:35:11 pts/0 0:00 grep d.bin

You have completed the CRS install. Now you want to verify if the install is valid.

To Ensure that the CRS install on all the nodes is valid, the following should be checked on all thenodes.

Ensure that you have successfully completed running root.sh on all nodes during the install.

Please do not re-run root.sh, this is very dangerous and might corrupt your installation, The object of this stepis to only confirm if the root.sh was run successfully after the install

Run the command $ORA_CRS_HOME/bin/crs_stat. Please ensure that this command does not error out butdumps the information for each resource.It does not matter what CRS stat returns for each resource.If the crs_stat exits after printing information about each resource then it means thatthe CRSD daemon is up and the client crs_stat utility can communicate with it.

This will also indicate that the CRSD can read the OCR.

If the crs_stat errors out with CRS-0202: No resources are registeredThen this means that there are no resources registered, and at this stage you missed the VIP configuration.This is not an error but is mostly because at this stage you missed the VIP configuration.In this case run $ORA_CRS_HOME/bin/vipca as root, to configure the VIPs

If the crs_stat errors out with CRS-0184: Cannot communicate with the CRS daemon this means the CRSdeamons are not started.

On one node as the crs user

{node1:crs} /crs/11.1.0/bin/crs_stat -t

Name Type Target State Host------------------------------------------------------------ora.node1.gsd application ONLINE ONLINE node1ora.node1.ons application ONLINE ONLINE node1ora.node1.vip application ONLINE ONLINE node1ora.node2.gsd application ONLINE ONLINE node2

4.3.2.Cluster Ready Services Health Check 73

ora.node2.ons application ONLINE ONLINE node2ora.node2.vip application ONLINE ONLINE node2</pre>|

On one node as the crs user

{node1:crs} /crs/11.1.0/bin/crs_stat -ls

Name Owner Primary PrivGrp Permission-----------------------------------------------------------------ora.node1.gsd crs oinstall rwxr-xr--ora.node1.ons crs oinstall rwxr-xr--ora.node1.vip root oinstall rwxr-xr--ora.node2.gsd crs oinstall rwxr-xr--ora.node2.ons crs oinstall rwxr-xr--ora.node2.vip root oinstall rwxr-xr--

5. Oracle Clusterware patchingAt this point we have installed Oracle Clusterware 11.1.0.6. In this section we will patch the OracleClusterware to the latest release of Oracle Clusterware 11g Release 2 - 11.1.0.7. The patchset can bedownloaded from metalink.

Note: The Oracle Clusterware software must be at the same or newer level as any other ORACLE_HOMEdirectory. Therefore, you should always upgrade Oracle Clusterware before you upgrade any otherORACLE_HOME directory. This patch does not allow you to upgrade other ORACLE_HOME directories,before you upgrade Oracle Clusterware home directory. We will apply this patchset in a non rolling upgradefashion.

Before we start we can query the clusterware versions

# /crs/11.1.0/crs/bin/crsctl query crs softwareversionOracle Clusterware version on node [node1] is [11.1.0.6.0]

# /crs/11.1.0/crs/bin/crsctl query crs activeversionOracle Clusterware active version on the cluster is [11.1.0.6.0]

Enter the following commands to start Oracle Universal Installer, where patchset_directory is the directorywhere you unpacked the patch set software:

$ cd patchset_directory/Disk1$ ./runInstaller

5. Oracle Clusterware patching 74

Notes

The installer appears♦

•

Actions

Click Next♦

•

Notes•


You should ensure that the Clusterware home is selected in the first drop down list box♦ The installer should default the directory to the correct location♦

Actions

Ensure the correct home is selected♦ Click Next♦

•

Notes

the installer detects that this is a clustered home and automatically selects all the nodes in thecluster

♦

•

Actions

Click Next♦

•


Notes

Some parameters are validated by the installer♦

•

Actions

Click Next♦

•

Notes•


This is a summary of the actions the installer will complete♦ Actions

Click Install♦

•

Notes

the installer stages the patch on all the nodes in the cluster♦

•

Actions

No action required♦

•


Notes

At the end the installer lists the mandatory steps that must be completed to apply this patch♦

•

Actions

Log in as the root user and enter the following command to shut down the OracleClusterware:

♦

•

# CRS_home/bin/crsctl stop crs

Run the root111.sh script. It will automatically start the Oracle Clusterware on the patchednode:

♦ •

# CRS_home/install/root111.sh

Show root111.sh output (all nodes except last) Hide root111.sh output (all nodes except last){root@node1}# /crs/11.1.0/crs/install/root111.shCreating pre-patch directory for saving pre-patch clusterware filesCompleted patching clusterware files to /opt/crsRelinking some shared libraries.Relinking of patched files is complete.Preparing to recopy patched init and RC scripts.Recopying init and RC scripts.Startup will be queued to init within 30 seconds.Starting up the CRS daemons.Waiting for the patched CRS daemons to start.This may take a while on some systems..11107 patch successfully applied.clscfg: EXISTING configuration version 4 detected.clscfg: version 4 is 11 Release 1.


Successfully accumulated necessary OCR keys.Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.node <nodenumber>: <nodename> <private interconnect name> <hostname>node 1: node1 node1-priv node1Creating OCR keys for user 'root', privgrp 'root'..Operation successful.clscfg -upgrade completed successfullyCreating '/crs/11.1.0/crs/install/paramfile.crs' with data used for CRS configurationSetting CRS configuration values in /crs/11.1.0/crs/install/paramfile.crsSetting cluster unique identifierRestarting Oracle clusterwareStopping Oracle clusterwareStopping resources.This could take several minutes.Successfully stopped Oracle Clusterware resourcesStopping Cluster Synchronization Services.Shutting down the Cluster Synchronization Services daemon.Shutdown request successfully issued.Waiting for Cluster Synchronization Services daemon to stopCluster Synchronization Services daemon has stoppedStarting Oracle clusterwareAttempting to start Oracle Clusterware stackWaiting for Cluster Synchronization Services daemon to startWaiting for Cluster Synchronization Services daemon to startWaiting for Cluster Synchronization Services daemon to startWaiting for Cluster Synchronization Services daemon to startCluster Synchronization Services daemon has startedEvent Manager daemon has startedCluster Ready Services daemon has started

Show root111.sh output on last node Hide root111.sh output on last node{root@node2} # /crs/11.1.0/crs/install/root111.shCreating pre-patch directory for saving pre-patch clusterware filesCompleted patching clusterware files to /opt/crsRelinking some shared libraries.Relinking of patched files is complete.Preparing to recopy patched init and RC scripts.Recopying init and RC scripts.Startup will be queued to init within 30 seconds.Starting up the CRS daemons.Waiting for the patched CRS daemons to start.This may take a while on some systems..11107 patch successfully applied.clscfg: EXISTING configuration version 4 detected.clscfg: version 4 is 11 Release 1.Successfully accumulated necessary OCR keys.Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.node <nodenumber>: <nodename> <private interconnect name> <hostname>node 2: node2 node2-priv node2Creating OCR keys for user 'root', privgrp 'root'..Operation successful.clscfg -upgrade completed successfullyCreating '/crs/11.1.0/crs/install/paramfile.crs' with data used for CRS configurationSetting CRS configuration values in /crs/11.1.0/crs/install/paramfile.crs


Continue with these two steps on all nodes, one by one, this will achieve a rolling upgrade of the OracleClusterware. When done, verify the Oracle Clusterware is running on all nodes before exiting the installer.

$ CRS_home/bin/crsctl check crsCluster Synchronization Services appears healthyCluster Ready Services appears healthyEvent Manager appears healthy

# /crs/11.1.0/crs/bin/crsctl query crs softwareversionOracle Clusterware version on node [node1] is [11.1.0.7.0]

# /crs/11.1.0/crs/bin/crsctl query crs activeversionOracle Clusterware active version on the cluster is [11.1.0.7.0]

Notes

Once the Oracle Clusterware is running on all nodes, exit the installer♦

•

Actions

Click Yes♦

•

5.1. Oracle Clusterware patching complete

6. Oracle ASM Home Software InstallIn this chapter we are going to install the asm software (only). Creating the asm storage is in the next chapter.

The software will be installed in a directory separate from any crs or database home,preferably in the $ORACLE_BASE tree and not as a subtree in the database home.

On each node :

5.1. Oracle Clusterware patching complete 81


{node1:root}/ # /usr/sbin/slibclean{node2:root}/ # /usr/sbin/slibclean

On each node, set right ownership and permissions to following directories :

{node1:root}/ # chown asm:oinstall /asm{node1:root}/ # chmod 665 /asm

If AIX 5L release 5.3 is used, modify the files oraparam.ini, and cluster.ini in Disk1/installer.

Update entries AIX5200 to AIX5300 in both files.

Check that Oracle Clusterware (including VIP, ONS and GSD) is started on each node.

As the asm user from node1.

{node1:asm}/oracle/asm # $CRS_HOME/bin/crs_stat -tName Type Target State Host------------------------------------------------------------ora....E1.lsnr application ONLINE ONLINE node1ora.node1.gsd application ONLINE ONLINE node1ora.node1.ons application ONLINE ONLINE node1ora.node1.vip application ONLINE ONLINE node1

ora....E2.lsnr application ONLINE ONLINE node2ora.node2.gsd application ONLINE ONLINE node2ora.node2.ons application ONLINE ONLINE node2ora.node2.vip application ONLINE ONLINE node2

6.1. CVU check before creating ASM instance

Verify the cluster is configured correctly for an instance creation

./runcluvfy.sh stage -pre dbinst -n node1,node2 -r 11gR1 -verbose

Show CVU -pre dbinst output Hide CVU output

6.2. Oracle ASM Home Software Install

Login as the asm owner (asm in our case).

Setup and export your DISPLAY, TMP and TEMP variables With /tmp or other destination having enoughfree space, about 500Mb on each node.

Please note the actual display may be different. For instance if you are running from the console this willtypically be :0.0 The display setting and the xhost setting can be checked by typing 'xclock' if you see theclock then you are good to go.

{node1:asm} export DISPLAY=node1:1{node1:asm} export TMP=/tmp

6. Oracle ASM Home Software Install 82

{node1:asm} TEMP=/tmp {node1:asm} export TMPDIR=/tmp

As the user asm start the installer.

{node1:asm} cd /distrib/SoftwareOracle/rdbms11gr1/aix/database{node1:asm} ./runInstaller

Starting Oracle Universal Installer...

Checking Temp space: must be greater than 190 MB. Actual 1950 MB PassedChecking swap space: must be greater than 150 MB. Actual 3584 MB PassedChecking monitor: must be configured to display at least 256 colors. Actual 16777216 PassedPreparing to launch Oracle Universal Installer from /tmp/OraInstall2008-02-13_03-47-34PM. Please wait …{node1:asm}/distrib/SoftwareOracle/rdbms11gr1/aix/database # Oracle Universal Installer, Version 11.1.0.6.0ProductionCopyright (C) 1999, 2007, Oracle. All rights reserved.

Notes•

Action• Click on Next•

6.2. Oracle ASM Home Software Install 83

* Notes

Here we will create a ASM Software home on all the nodes in the cluster♦ •

Actions

Select the Oracle Database 11g radio button♦ Click Next♦

•


Notes

Here we can choose the type of database software to be installed♦

•

Actions

Select the Enterprise Edition radio button♦ Click Next♦

•

Notes

Here we specify the location of various components. An 11g install makes more use of theORACLE_BASE. Most logs will be stored in sub directories under the oracle base. Theoracle base will be common to all installs. Modify as required. Also this is where you specifythe location of the ASM software home. I usually modify the Home name and Home Path toinclude the word ASM. This makes it easier to identify later on. If you change the path youshould ensure that you do not use the exact same path as the Oracle Clusterware home

♦

•

Actions

Confirm entries are OK♦ Click Next♦

•


Notes

The installer has detected the presence of Oracle Clusterware and uses this to populate thisdialog box. To build a cluster which includes all nodes you must ensure that there arecheck-boxes next to the node names

♦

•

Actions

Click Select All♦ Click Next♦

•


Notes

The installer will then complete some Product-Specific Prerequisite checks. These should allpass OK – as you have already run the CVU check

♦

•

Actions

Click Next♦

•

Notes• • Actions•

Click Next•


Notes• Verify the UNIX primary group name of the asm user. (Use unix command id to find out• and set the Privileged Operating System Groups to the value found.• In our example, this must be :• 'dba' for Database Administrator (OSDBA) group• 'oper' for Database Operator (OSOPER) group• 'asm' for administrator (OSASM) group (Primary group of unix asm user)• to be set for both entries.• Actions• Click Next•


Notes

We are going to install a Software only home and then subsequently configure the software♦

•

Actions

Select the Install Software Only radio button♦ Click Next♦

•

Notes

♦

•


Action

♦ Click Install♦

•

%TBF%

Notes Install Progress• Action• Wait for the next screen.•

Notes• At this stage, you could see a message about 'Failed Attached Home'• If this happens just do the following specified node as the asm user• Check and adapt the script depending on your environment.• When done, click on 'OK':• From node2 (In this case)•

/oracle/asm/11.1.0/oui/bin/runInstaller -attachHome -noClusterEnabled ORACLE_HOME=/oracle/asm/11.1.0 ORACLE_HOME_NAME=OraAsm11g_home\

CLUSTER_NODES=node1,node2 INVENTORY_LOCATION=/oracle/oraInventory LOCAL_NODE=node2

Starting Oracle Universal Installer... No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /oracle/oraInventory

AttachHome was successful.

{node2:asm}/oracle/asm/11.1.0/bin $


* Notes

The installer pauses, some scripts need to be run as root on both nodes of the cluster♦ •

Action

open a shell window on each node♦ run the root.sh script♦

•

{node1:root}/oracle/asm/11.1.0 # ./root.sh

Running Oracle 11g root.sh script...The following environment variables are set as:

ORACLE_OWNER= asmORACLE_HOME= /oracle/asm/11.1.0

Enter the full pathname of the local bin directory: [/usr/local/bin]:The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: yCopying dbhome to /usr/local/bin ...The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: yCopying oraenv to /usr/local/bin ...The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: yCopying coraenv to /usr/local/bin ...Creating /etc/oratab file...Entries will be added to the /etc/oratab file as needed byDatabase Configuration Assistant when a database is createdFinished running generic part of root.sh script.Now product-specific root actions will be performed.Finished product-specific root actions.

Notes

The output from the root.sh script should be similar to this♦

•


it should only take a few seconds to run on each node♦ Action

run the scripts on all nodes♦ then return to the installer and Click OK♦

•

Notes

After the software install completes you will see this End of Installation dialog♦

•

Actions

Click Exit♦

•

6.3. Post Install Actions.

Update the ASM unix user .profile

To be done on each node for the asm user.

vi $HOME/.profile file in the asm users home directory. Add the entries in bold.

PATH=/usr/bin:/etc:/usr/sbin:/usr/ucb:$HOME/bin:/usr/bin/X11:/sbin:.export PATHif [ -s "$MAIL" ] # This is at Shell startup. In normalthen echo "$MAILMSG" # operation, the Shell checksfi # periodically.ENV=$HOME/.kshrcexport ENV#The following line is added by License Use Management installationexport PATH=$PATH:/usr/opt/ifor/ls/os/aix/bin

6.3. Post Install Actions. 92

export PATH=$PATH:/usr/java14/bin

export MANPATH=$MANPATH:/usr/local/manexport ORACLE_BASE=/oracleexport AIXTHREAD_SCOPE=Sexport TEMP=/tmpexport TMP=/tmpexport TMPDIR=/tmpumask 022export CRS_HOME=/crs/11.1.0export ORACLE_CRS_HOME=$CRS_HOME

export ORACLE_HOME=$ORACLE_BASE/asm/11.1.0export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$CRS_HOME/lib:$ORACLE_HOME/lib32:$CRS_HOME/lib32export LIBPATH=$LD_LIBRARY_PATH

export PATH=$ORACLE_HOME/bin:$CRS_HOME/bin:$PATHexport TNS_ADMIN=$ORACLE_HOME/network/adminexport ORACLE_SID=+ASM1if [ -t 0 ]; then stty intr ^C

On node2, ORACLE_SID will be set as below :

export ORACLE_SID=+ASM2

Disconnect from asm user, and reconnect to load modified $HOME/.profile

7. Oracle ASM Software Home PatchingOnce ASM software has been installed, the next step in the process is to apply the 11.1.0.7 patchset to theASM ORACLE_HOME.

7.1. Start the OUI for Oracle ASM Software Home Patching



7. Oracle ASM Software Home Patching 93

Notes

the Universal Installer screen appears♦

•

Action

On the Welcome screen, click Next.♦

•

Notes•

7.1. Start the OUI for Oracle ASM Software Home Patching 94

Specify the name and the location of the asm home♦ Action

Check that the name and location are correct♦

•

Notes:

here you can specify your metalik credentials for this install.♦ If you leave both fields blank you can opt out of notifications (see next screen).♦

•

Action

Enter your email address♦ Enter your Metalink password♦ Click Next♦

•


Notes:*

If you leave both fields blank, you will opt out of notifications and get the above screen.♦

•

Action

Click Yes♦

•

Notes•



♦

Actions

Click Next♦

•

Notes


•

Action

Click Next♦

•


Notes

This is a summary of the actions the installer will complete♦

•

Actions

Click Install♦

•

Notes•


The installer copies the patch to all the nodes in the cluster♦ Actions


•

Notes

The installer pauses, root.sh need to be run as root on both nodes of the cluster♦

•

Action

open a shell window on each node and run root.sh (one after the other)♦ then return to the installer and Click OK♦

•

{root@node1} # /oracle/asm/product/11.1.0/asm/root.sh Running Oracle 11g root.sh script...

The following environment variables are set as: ORACLE_OWNER= oracle11 ORACLE_HOME= /oracle/asm/product/11.1.0/asm

Enter the full pathname of the local bin directory: [/usr/local/bin]:

/usr/local/bin is read only. Continue without copy (y/n) or retry (r)? [y]: y

Warning: /usr/local/bin is read only. No files will be copied.

Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed.


Finished product-specific root actions.

Notes

The output from the root.sh script should be similar to this♦ it should only take a few seconds to run on each node♦

•

Action

run the scripts on all nodes♦ then return to the installer and Click OK♦

•

Notes

After the software install completes you will see this End of Installation dialog, exit theinstaller

♦

•

Actions

Click Yes♦

•

8. Oracle RAC Database Listener CreationThe Oracle network listeners traditionally run from the ASM home. Here we are going to create the listenersusing netca from the ASM home.

8.1. Create Node specific network listenerscd /oracle/asm/11.1.0/bin./netca

8. Oracle RAC Database Listener Creation 100

Notes

Netca detects that the Oracle Clusterware layer is running and offers Cluster or Single Nodeconfiguration

♦

•

Actions

Select the Cluster configuration radio button♦ Click Next♦

•

Notes

Netca uses Oracle Clusterware to determine all the nodes in the cluster♦

•

Actions

Click Select all nodes♦ Click Next♦

•

8.1. Create Node specific network listeners 101

Notes

You get various options - we need to configure listeners♦

•

Actions

Select the Listener configuration radio button♦ Click Next♦

•

Notes

We need to add a listener♦

•

Actions

Select the Add radio button♦ Click Next♦

•


Notes

Here you get the opportunity to name the listener - Do not change this. The listeners willeventually be called LISTENER_nodename1 & LISTENER_nodename2. This is importantfor RAC

♦

•

Actions

Click Next♦

•

Notes

Oracle Net supports various network protocols, although TCP is the most common.♦

•

Actions

Ensure the Selected Protocols list includes TCP♦ Click Next♦

•


Notes♦ It is possible to choose a non-default Port♦ Unless you have a good reason and are experienced in configuring network parameters:♦ Leave this as the default.♦

•

Actions

Ensure the Use the Standard port number of 1521 radio button is set♦ Click Next♦

•

Notes

Unless you are experienced with calling external procedures leave this as the default.♦

•

Actions

Click Next♦

•


Notes

You get the opportunity to configure another listener♦

•

Action

Click Next♦

•

Action• Click Next•


Notes

You get the opportunity to configre other networking components♦

•

Action

Click the Finish button to exit the tool♦

•

8.2. Verify the Listener resources are online

Checking if Listeners are registered in Oracle Clusterware:

[asm@node1 bin]$ cd /scratch/11.1.0/crs/bin[asm@node1 bin]$ ./crs_stat -tName Type Target State Host ------------------------------------------------------------ ora....e1.lsnr application ONLINE ONLINE node1 ora....de1.gsd application ONLINE ONLINE node1 ora....de1.ons application ONLINE ONLINE node1 ora....de1.vip application ONLINE ONLINE node1ora....e2.lsnr application ONLINE ONLINE node2ora....de2.gsd application ONLINE ONLINE node2ora....de2.ons application ONLINE ONLINE node2ora....de2.vip application ONLINE ONLINE node2

Notes

You can see the listener resources inside Oracle Clusterware♦

•

8.3. Completed Node specific network listeners creation

You have now completed the RAC cluster nodes Network listener configuration.A new managed resource will have been added to Oracle Clusterware OCR for each listener.

8.2. Verify the Listener resources are online 106

9. Oracle ASM Instance and diskgroup CreationNow we will create the ASM Instance.

Only one ASM instance per node is supported by Oracle

A few words before we proceed about ASM disks, failgroups and redundancy

The example in the screenshots is very simple.

A typical diskgroup would contain many more disks to give the desired I/O capacity andmore protection against failure.

We also choose 'external' redundancy.

This means that the responsibiliy of maintaining redundant copies of the data belongs to the disk array.If you have not implemented this by configuring the disks in the array, as, typically, raid 5 or raid1+0then any single disk falure will destroy your database.

*So don't specify 'external' redundancy unless your disks are configured reduntantly in the array.*

So what are the advantages of ASM 'Normal' redundancy?

In this case ASM itself will maintain two copies of the data (or even three! in the case of 'High' redundancy)

How is this better than 'hardware' mirroring.

Well in the first place ASM maintains duplicate copies of meta data,needed to access the data, in locations that are guaranteed not to be on the same disk.

So if a corruption happens in the meta data with external redundancy then the data is lost.With 'Normal' redundancy you always have a separately maintained copy of the meta data.So recovery is possible.

ASM 'Normal' redundancy is not a simple Disk-to-Disk mirror.Rather each extent in ASM (default 1 Megabyte but can be increased) has a copy maintained in a physicallyseparate location.

Additionally ASM uses a 'SAME' (Stripe and Mirror Everthing) methodology.So all data is striped across all the ASM 'disks' you have specified.

As we discussed in chapter 2 the 'ASM' disks may also represent 'striped sets' in the array.We have investigated the performance impact of this 'double striping' and it is negligible.

Also if you have storage that is not so 'high end' ASM can offer you similar levels of protectionand performance on inexpensive hardware.

As the asm user, from one node :

{node1:asm} export DISPLAY=node1 :0.0{node1:asm} export ORACLE_HOME=/oracle/asm/11.1.0{node1:asm} cd $ORACLE_HOME/bin{node1:asm} ./dbca &

9. Oracle ASM Instance and diskgroup Creation 107

Notes

We use the dbca from the ASM install to create the ASM instances♦

•

Actions

ensure the ORACLE_HOME environment variable is set to the ASM home directory♦ run ./dbca from the ASM home bin directory♦

•

Notes

dbca detects the Oracle Clusterware layer is running and offers to create either cluster orsingle instance database

♦

•

Actions

Select the Oracle Real Application Clusters database radio button♦ Click Next♦

•


Notes

We use dbca to create RAC databases and ASM instances♦

•

Actions

Select the Configure Automatic Storage Management radio button♦ Click Next♦

•


Notes

You need to make sure you create ASM instances on all the cluster nodes♦

•

Actions


•

Notes

Here we specify the password for the ASM Oracle SYS user♦

•

Actions

Enter the SYS password♦ Enter the same password for the Confirm SYS password♦ Click Next♦

•

* Notes

dbca will create and start ASM instances♦ •

Actions•


Click OK♦

Notes

This should take a few seconds♦

•

Actions

no action♦

•

Notes

ASM requires disks to be group together into diskgroups. This section will be used to create adisk group called DATA_DG

♦

Not illustrated here but you should also create separate disk group on separate disks for the'Flash Recovery Area'

♦

Typically this can be called FRA or FLASH♦

•

Actions

Click Create New♦

•


Notes• If you don't see any candidate disks:• Action• Click on 'Change Disk Discovery Path'•

Notes

Now we will assign disks to specific disk groups and create the DATA diskgroup♦

•

Actions•


In the Disk Group Name enter DATA_DG♦ Select the External Redundancy radio button♦ Select the Show All radio button♦ Select the disks to be used for the DATA_DG diskgroup♦ Click OK♦

Notes

A progress message – this should not take very long♦

•

Actions

None♦

•

Notes

Here we can see the DATA_DG diskgroup has been created mounted on 2/2 instances. Thiscompletes the ASM configuration.

♦

•

Actions

Click Finish♦

•


9.1. Verify ASM instances are online

[asm@node1 bin]$ cd $CRS_HOME/bin[asm@node1 bin]$ ./crs_stat -t

Name Type Target State Host ------------------------------------------------------------ ora....SM1.asm application ONLINE ONLINE node1ora....e1.lsnr application ONLINE ONLINE node1 ora....de1.gsd application ONLINE ONLINE node1 ora....de1.ons application ONLINE ONLINE node1 ora....de1.vip application ONLINE ONLINE node1ora....SM2.asm application ONLINE ONLINE node2 ora....e2.lsnr application ONLINE ONLINE node2 ora....de2.gsd application ONLINE ONLINE node2ora....de2.ons application ONLINE ONLINE node2 ora....de2.vip application ONLINE ONLINE node2

Notes

Here we can see the asm instances now as part of Oracle Clusterware♦

•

Action

change directory to the Oracle Clusterware bin directory♦ run ./crs_stat -t♦

•

9.2. Completed creation of ASM Instance and add of the DATA_DGdiskgroup

Congratulations you have installed ASM, Network listeners and created an ASM diskgroup.

10. Oracle RAC Database Home Software Install

10.1. CVU check -pre Database Instance Install

Before installing the RAC Database software you should use the CVU utility to confirm the status of thecluster.

./runcluvfy.sh stage -pre dbinst -n stnsp001,stnsp002 -r 11gR1 -verbose

Note: If you executed the CVU -pre dbinst command during the ASM instance installation, you donot have the execute the command again.

•

Show CVU -pre dbinst output Hide CVU output

10.2. Database Software Install

On each node


{node1:root}/ # /usr/sbin/slibclean{node2:root}/ # /usr/sbin/slibclean

9.1. Verify ASM instances are online 114

From node1 As root user,

Under VNC Client session, or other graphical interface, execute

{node1:root} # xhost +access control disabled, clients can connect from any hosts{node1:root} #

On each node, set the right ownership and permissions to following directories :

{node1:root}/ # chown rdbms:oinstall /oracle/rdbms{node1:root}/ # chmod 665 /oracle/rdbms{node1:root}/

{node2:root}/ # chown rdbms:oinstall /oracle/rdbms{node2:root}/ # chmod 665 /oracle/rdbms{node2:root}/ #

Setup and export your DISPLAY, TMP and TEMP variables

With /tmp or other destination having enough free space, about 500Mb on each node.

{node1:rdbms} export DISPLAY=node1:0.0If not set in rdbms .profile, then :{node1:rdbms} export TMP=/tmp{node1:rdbms} export TEMP=/tmp{node1:rdbms} export TMPDIR=/tmp

Check that Oracle Clusterware (including VIP, ONS, GSD, Listener's and ASM ressources) are started oneach node.

As rdbms user from node1 :

{node1:rdbms} $CRS_HOME/bin/crs_stat -t

Name Type Target State Host------------------------------------------------------------ora....SM1.asm application ONLINE ONLINE node1ora....E1.lsnr application ONLINE ONLINE node1ora.node1.gsd application ONLINE ONLINE node1ora.node1.ons application ONLINE ONLINE node1ora.node1.vip application ONLINE ONLINE node1ora....SM2.asm application ONLINE ONLINE node2ora....E2.lsnr application ONLINE ONLINE node2ora.node2.gsd application ONLINE ONLINE node2ora.node2.ons application ONLINE ONLINE node2ora.node2.vip application ONLINE ONLINE node2

{node1:rdbms} cd /distrib/SoftwareOracle/rdbms11gr1/aix/database

10.2. Database Software Install 115

{node1:rdbms} ./runInstaller

Starting Oracle Universal Installer...

Checking Temp space: must be greater than 190 MB. Actual 1933 MB PassedChecking swap space: must be greater than 150 MB. Actual 3584 MB PassedChecking monitor: must be configured to display at least 256 colors. Actual 16777216 PassedPreparing to launch Oracle Universal Installer from /tmp/OraInstall2008-02-14_02-41-49PM. Please wait ...

Oracle Universal Installer, Version 11.1.0.6.0 ProductionCopyright (C) 1999, 2007, Oracle. All rights reserved.

Notes

We use the same installer for Database homes as Clusterware homes♦

•

Actions

Select the Oracle Database 11g radio button♦ Click Next♦

•


|

Notes

Here we can choose the type of database software to be installed♦

•

Actions

Select the Enterprise Edition radio button♦ Click Next♦

•

Notes•


Here we specify the location of various components. If you change the path you shouldensure that you do not use the exact same path as the Oracle Clusterware or ASM homes

♦

Actions

Confirm entries are OK♦ Click Next♦

•

|

Notes

The installer has detected the presence of Oracle Clusterware and uses this to populate thisdialog box. To build a cluster which includes all nodes you must ensure that there arecheck-boxes next to the node names

♦

•

Actions


•


|

Notes• The installer will check some product-specific Prerequisites• Don't worry about the lines with checking at status 'Not executed',• These are justwarnings because the AIX maintenance level might be higher then 5300.• Which is the case in our example (ML03• Action• Click Next•

Show Details of the prerequisite checks done by runInstaller Hide Details of the prerequisite checks doneby runInstaller

Details of the prerequisite checks done by runInstaller

Checking operating system requirements ...Expected result: One of 5300.05,6100.00Actual Result: 5300.07Check complete. The overall result of this check is: Passed========================================================Checking operating system package requirements ...Checking for bos.adt.base(0.0); found bos.adt.base(5.3.7.0). PassedChecking for bos.adt.lib(0.0); found bos.adt.lib(5.3.0.60). PassedChecking for bos.adt.libm(0.0); found bos.adt.libm(5.3.7.0). PassedChecking for bos.perf.libperfstat(0.0); found bos.perf.libperfstat(5.3.7.0). PassedChecking for bos.perf.perfstat(0.0); found bos.perf.perfstat(5.3.7.0). PassedChecking for bos.perf.proctools(0.0); found bos.perf.proctools(5.3.7.0). PassedChecking for rsct.basic.rte(0.0); found rsct.basic.rte(2.4.8.0). PassedChecking for rsct.compat.clients.rte(0.0); found rsct.compat.clients.rte(2.4.8.0). PassedChecking for bos.mp64(5.3.0.56); found bos.mp64(5.3.7.1). PassedChecking for bos.rte.libc(5.3.0.55); found bos.rte.libc(5.3.7.1). PassedChecking for xlC.aix50.rte(8.0.0.7); found xlC.aix50.rte(9.0.0.1). PassedChecking for xlC.rte(8.0.0.7); found xlC.rte(9.0.0.1). PassedCheck complete. The overall result of this check is: Passed


========================================================

Checking recommended operating system patchesChecking for IY89080(bos.rte.aio,5.3.0.51); found (bos.rte.aio,5.3.7.0). PassedChecking for IY92037(bos.rte.aio,5.3.0.52); found (bos.rte.aio,5.3.7.0). PassedChecking for IY94343(bos.rte.lvm,5.3.0.55); found (bos.rte.lvm,5.3.7.0). PassedCheck complete. The overall result of this check is: Passed========================================================

Checking kernel parametersCheck complete. The overall result of this check is: Not executed <<<<

OUI-18001: The operating system 'AIX Version 5300.07' is not supported.

Recommendation: Perform operating system specific instructions to update the kernel parameters.========================================================

Checking physical memory requirements ...Expected result: 922MBActual Result: 2048MBCheck complete. The overall result of this check is: Passed========================================================

Checking available swap space requirements ...Expected result: 3072MBActual Result: 3548MBCheck complete. The overall result of this check is: Passed========================================================

Validating ORACLE_BASE location (if set) ...Check complete. The overall result of this check is: Passed========================================================

Checking maxmimum command line length argument, ncarg...Check complete. The overall result of this check is: Passed========================================================

Checking for proper system clean-up....Check complete. The overall result of this check is: Passed========================================================

Checking Oracle Clusterware version ...Check complete. The overall result of this check is: Passed========================================================

Checking uid/gid...Check complete. The overall result of this check is: Passed========================================================


|

Notes

We are going to install a Software only home and then subsequently configure the software♦

•

Actions

Select the Install Software Only radio button♦ Click Next♦

•

Notes

♦

•


Action

♦ Click♦

•

Notes

Here we can see a summary of the install.♦

•

Actions

Click Install♦

•


Notes

Here we can see a summary of the install.♦

•

Actions

Click Install♦

•

|

Notes•


The installer copies the software to all nodes in the cluster♦ Action

None♦

•

At this stage, you could hit message about Failed Attached Home

If this happens follow the instruction in the 'Install ASM Software' chapter adapting the commands for theORACLE Database Home.

Notes

The installer pauses, some scripts need to be run as root on both ndes of the cluster♦

•

Action

open a shell window on each node♦

•

Show the output of root.sh Hide the output of root.sh

{node1:root}/ # /oracle/rdbms/11.1.0/root.sh

Running Oracle 11g root.sh script...

The following environment variables are set as: ORACLE_OWNER= rdbms ORACLE_HOME= /oracle/rdbms/11.1.0

Enter the full pathname of the local bin directory: [/usr/local/bin]:The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n) [n]:The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]:The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]:

Entries will be added to the /etc/oratab file as needed byDatabase Configuration Assistant when a database is createdFinished running generic part of root.sh script.Now product-specific root actions will be performed.


Finished product-specific root actions.{node1:root}/

----------------------------------------------------------------{node2:root}/ # /oracle/rdbms/11.1.0/root.sh</b>Running Oracle 11g root.sh script...

The following environment variables are set as:

ORACLE_OWNER= rdbms ORACLE_HOME= /oracle/rdbms/11.1.0

Enter the full pathname of the local bin directory: [/usr/local/bin]:The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n) [n]:The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]:The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]:

Entries will be added to the /etc/oratab file as needed byDatabase Configuration Assistant when a database is createdFinished running generic part of root.sh script.Now product-specific root actions will be performed.Finished product-specific root actions.{node2:root}/ #

%TBF%

Notes• Coming back to this previous screen• Action• Click OK•


Notes

After the software install completes you will see this End of Installation dialog♦

•

Actions

Click Exit♦

•

10.3.Symbolic links creation for listener.ora, tnsnames.ora andsqlnet.ora

Create the symbolic links, make sure that the files 'listener.ora', 'tnsnames.ora' and 'sqlnet.ora' exist in/oracle/asm/11.1.0/network/admin/listener.ora on both nodes.

As rdbms user on each node :

{node1:rdbms}/ # ln -s /oracle/asm/11.1.0/network/admin/listener.ora /oracle/rdbms/11.1.0/network/admin/listener.ora

{node1:rdbms}/ # ln -s /oracle/asm/11.1.0/network/admin/tnsnames.ora /oracle/rdbms/11.1.0/network/admin/tnsnames.ora

{node1:rdbms}/ # ln -s /oracle/asm/11.1.0/network/admin/sqlnet.ora /oracle/rdbms/11.1.0/network/admin/sqlnet.ora

{node1:rdbms}/ # ls -l /oracle/rdbms/11.1.0/network/admin/*.ora

lrwxrwxrwx 1 oracle dba 47 Apr 23 10:19 listener.ora -> /oracle/ asm/11.1.0/network/admin/listener.ora

lrwxrwxrwx 1 oracle dba 47 Apr 23 10:19 tnsnames.ora -> /oracle/asm/11.1.0/network/admin/tnsnames.ora

lrwxrwxrwx 1 oracle dba 47 Apr 23 10:19 sqlnet.ora -> /oracle/asm/11.1.0/network/admin/sqlnet.ora

{node1:rdbms}/ #

Doing this will avoid dealing with multiple listener.ora, tnsnames.ora and sqlnet.ora files, as setting theTNS_ADMIN variablewill not be enough as some oracle assistant tools will not use this variable.

Do the same on node2

10.3.Symbolic links creation for listener.ora, tnsnames.ora and sqlnet.ora 126

10.4.Update the RDBMS unix user .profile

To be done on each node for rdbms (in our case) or oracle unix user.

vi the $HOME/.profile file in the rdbms owner's home directory.Add the entries in bold

PATH=/usr/bin:/etc:/usr/sbin:/usr/ucb:$HOME/bin:/usr/bin/X11:/sbin:.

export PATH

if [ -s "$MAIL" ] # This is at Shell startup. In normal then echo "$MAILMSG" # operation, the Shell checksfi # periodically.

ENV=$HOME/.kshrcexport ENV

#The following line is added by License Use Management installationexport PATH=$PATH:/usr/opt/ifor/ls/os/aix/binexport PATH=$PATH:/usr/java14/bin

export MANPATH=$MANPATH:/usr/local/manexport ORACLE_BASE=/oracleexport AIXTHREAD_SCOPE=Sexport TEMP=/tmpexport TMP=/tmpexport TMPDIR=/tmpumask 022export CRS_HOME=/crs/11.1.0export ORA_CRS_HOME=$CRS_HOMEexport ASM_HOME=$ORACLE_BASE/asm/11.1.0export ORA_ASM_HOME=$ASM_HOME

export ORACLE_HOME=$ORACLE_BASE/rdbms/11.1.0export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$CRS_HOME/lib:$ORACLE_HOME/lib32:$CRS_HOME/lib32export LIBPATH=$LD_LIBRARY_PATH

export PATH=$ORACLE_HOME/bin:$CRS_HOME/bin:$PATHexport TNS_ADMIN=$ASM_HOME/network/adminexport ORACLE_SID=

if [ -t 0 ]; then stty intr ^Cfi

Disconnect from the rdbms user, and reconnect to load modified $HOME/.profile

11. Oracle RAC Software Home PatchingOnce RAC software has been installed, the next step in the process is to apply the 11.1.0.7 patchset to theRAC ORACLE_HOME.

10.4.Update the RDBMS unix user .profile 127

11.1. Using the installer to patch the Oracle RAC Software Home



Notes

the Universal Installer screen appears♦

•

Actions

On the Welcome screen, click Next.♦

•

11.1. Using the installer to patch the Oracle RAC Software Home 128

Notes

Specify the name and the location of the Database home♦

•

Actions

Check that the name and location are correct, Click Next♦

•

Notes•


here you can specify your metalik credentials for this install.♦ If you leave both fields blank you can opt out of notifications (see next screen).♦

Actions

Enter your email address♦ Enter your Metalink password♦ Click Next♦

•

Notes

If you leave both fields blank, you will opt out of notifications and get the above screen.♦

•

Actions

Click Yes♦

•


Notes


♦

•

Actions

Click Next♦

•


Notes


•

Actions

Click Next♦

•

Notes

This is a summary of the actions the installer will complete♦

•

Actions

Click Install♦

•


Notes

The installer copies the patch to all the nodes in the cluster♦

•

Actions


•

Notes•


The installer pauses, root.sh need to be run as root on both nodes of the cluster♦ Actions

open a shell window on each node and run root.sh (one after the other), output is shownbelow

♦

then return to the installer and Click OK♦

•

{root@node1}# /oracle/product/11.1.0/db_1/root.sh Running Oracle 11g root.sh script...

The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /oracle/product/11.1.0/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:

/usr/local/bin is read only. Continue without copy (y/n) or retry (r)? [y]: y

Warning: /usr/local/bin is read only. No files will be copied.

Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. Finished product-specific root actions.

Notes

Exit the installer♦

•

Actions•


Click Yes♦

12. Oracle RAC Database CreationWe are nearly ready to use the Database Configuration Assistant (dbca) to create the Oracle RAC instances onall the nodes.

MANDATORY: Change permissions to allow the rdbms user to write to directories owned by the asm user,from node1 execute:

# chmod -R g+w /oracle/asm/11.1.0/network# chmod -R g+w /oracle/cfgtoollogs# chmod -R g+w /oracle/admin# chmod -R g+w /oracle/diag# rsh node2 chmod -R g+w /oracle/asm/11.1.0/network# rsh node2 chmod -R g+w /oracle/cfgtoollogs# rsh node2 chmod -R g+w /oracle/admin# rsh node2 chmod -R g+w /oracle/diag

12.1. use dbca to create the RAC database

Check that all resources from Oracle Clusterware are started on their home node.

From node1:

# /crs/bin/crs_stat -tName Type Target State Host------------------------------------------------------------ora....SM1.asm application ONLINE ONLINE node1ora....E1.lsnr application ONLINE ONLINE node1ora.node1.gsd application ONLINE ONLINE node1ora.node1.ons application ONLINE ONLINE node1ora.node1.vip application ONLINE ONLINE node1ora....SM2.asm application ONLINE ONLINE node2ora....E2.lsnr application ONLINE ONLINE node2ora.node2.gsd application ONLINE ONLINE node2ora.node2.ons application ONLINE ONLINE node2ora.node2.vip application ONLINE ONLINE node2

Connect as the Oracle user from first node, and setup your DISPLAY, then execute dbca to run the databaseconfiguration assistant

$ export ORACLE_HOME=/oracle/rdbms/11.1.0$ export DISPLAY=<machineip:0>$ $ORACLE_HOME/bin/dbca

12. Oracle RAC Database Creation 135

Notes

DBCA Welcome Screen♦

•

Actions

Select the 'Oracle Real Application Clusters Database' option.♦ Click Next♦

•

Notes

Operations♦

•

Actions•

12.1. use dbca to create the RAC database 136

Select the 'Create a Database' option.♦ Click Next♦

Notes

Node Selection♦

•

Actions

Make sure you select all RAC nodes.♦ Click Next♦

•

Notes•


Database Templates♦ Actions

Select 'General Purpose' Or 'Custom Database' if you want to more control.♦ Click Next♦

•

Notes

Database Identification♦

•

Actions

Specify the 'Global Database Name'♦ The 'SID Prefix' will be automatically updated. (by default it is the Global Database Name)For our example : JSC1DB

♦

Click Next♦

•


Notes

Management Options♦

•

Actions

Check 'Configure the database with Enterprise Manager' if you want to use Database Control(local administration), uncheck for global admin (Grid Control).

♦

You can also set Alerts and Backups (fill SMTP data and schedule):

◊

♦

Click Next♦

•


Notes

Database Credentials♦

•

Actions

Specify same password for all administrator users, or specify individual password for eachuser:

◊

♦

Click Next♦

•


Notes

Storage Options♦

•

Actions

Choose Automatic Storage Management (ASM)♦ Click Next♦ At password prompt, enter ASM sys password:

◊

♦

Click OK♦

•


Notes

Select Disk Group♦

•

Actions

Select the Disk Group to be used (DATA_DG1 in our example)♦ Click Next.♦

•

Notes

Database File Locations♦

•

Actions•


Select Use Oracle-Managed Files AND Select the Disk Group to use for the Database Files:

◊

♦

Click Next♦

Notes

Recovery Configuration♦

•

Actions

Select the DiskGroup and size to be used for the Flash Recovery Area (+DATA_DG1 &4096MB in our example). Oracle recommends that the Flash Recovery Area is placed in aseparate Disk Group, unlink this example.

♦

You may optionally enable archiving, by clicking on 'Edit Archive Mode parameters'♦

•


◊

Notes

If you do not set the archive log destination it will default to the FlashRecovery Area.

⋅

◊

Click Next•

Notes

Database Content♦

•

Actions * Optionally select sample schemas• Click Next•


Notes

Initialization Parameters♦

•

Actions * Change the database block size, connection mode, character set, if needed. * Select theneeded memory parameters, you may click on 'All Initialization Parameters' to view or modify, only

needed for experts: *

•

Click Next•


Notes

Security Settings♦

•

Actions * If you are upgrading an exiting database or have existing applications and theimplementation of the new security features has not been planned, select 'Revert to pre-11G...' Underall other circumstances select 'Keep the enhanced 11g'...

•

Click Next•

Notes

Automatic Maintenance Tasks♦

•

Actions•


If you know exactly what you are doing and have your own, well developed strategies forstatistics gathering, stored outlines etc, then you can disable this. Under all othercircumstances leave it enabled.

♦

Click Next•

Notes

Database Storage♦

•

Actions

Consider whether you need more and larger redo logs which will depend on your redogeneration rate. You will probably want to make your redo-logs bigger than the default (50Mbytes).

♦

•

Click 'Next'...•


Notes

Creation Options♦

•

Actions

Make sure the 'Create Database' is checked, also optionally select Creation Scripts'.♦

•

Click Finish•

Notes

Summary♦

•

Actions•


Check the description, save the HTML summary file if needed.♦ Click OK•

Notes

Database Creation script generation♦

•

Actions

Click OK on pop up (If scripts were selected):

◊

♦

Check /oracle/cfgtoollogs/dbca/JSC1DB for the logs in case of failure to create the database♦

•

Click OK•


Notes

End of Install and Password Management♦

•

Actions

Click password management if you need to change the passwords or unlock some useraccounts that are locked by default (for security purposes)

◊

♦

•


Click Exit and the instances will be started on all the nodes:

◊

♦


Documents

RACGuides_Rac11gR1OnAIX