Build Your Own Oracle RAC 11 g Release 2 Cluster on Red ... · If you have ever been tasked with extending an Oracle RAC cluster by adding a new node (or shrinking a RAC cluster by

Author – A.Kishore http:/www.appsdba.info

Build Your Own Oracle RAC 11g Release 2 Cluster on Red Hat 4 and iSCSI Here is my setup which I have used for Installing Oracle RAC 11gR2 High Level Steps > Install openfiler on linux1 using VMWARE > Install Grid Infrastructure on linux1 and linux2 > Install Oracle 11gR2 software on linux1 and linux2 > Install Cluster database Download the software from the below site http://www.oracle.com/technetwork/database/enterprise-edition/downloads/112010-linuxsoft-085393.html

Laptop – windows XP To access linux1 and linux2

Linux1 4 GB 250 GB

Linux2 4 GB 250 GB

openfiler1 using vmware with 1 GB RAM 250 GB

There are 2 operating systems running on linux1, one is the host and one is the guest for openfiler

http://www.oracle.com/technetwork/database/enterprise-edition/downloads/112010-linuxsoft-085393.html

http://www.oracle.com/technetwork/database/enterprise-edition/downloads/112010-linuxsoft-085393.html

http://appsdba.info/


1. Install Oracle 11gR2 Grid Infrastructure (Both the nodes should be ready) -- clean the old installation rm -rf /etc/oracle rm -rf /etc/oraInst.loc rm -rf /d01/oracle/RACDB/11gR2/*.* rm -rf /d01/oracle/RACDB/db/tech_st/grid/*.* rm -rf /d01/oracle/oraInventory/*.* rm -rf /etc/oratab rm -rf /home/oracle/app/*.* rm -rf /home/oracle/app/11.2.0/grid/*.* /etc/init.d/oracleasm deletedisk VOL1 /dev/sdc1 /etc/init.d/oracleasm deletedisk VOL2 /dev/sdd1 /etc/init.d/oracleasm deletedisk VOL3 /dev/sde1 /etc/init.d/oracleasm createdisk VOL1 /dev/sdc1 /etc/init.d/oracleasm createdisk VOL2 /dev/sdd1 /etc/init.d/oracleasm createdisk VOL3 /dev/sde1 /etc/init.d/oracleasm listdisks Run cluvfy to verify all the prereqs are available su - root cd /tmp rm -rf bootstrap/ su - oracle cd /d01/oracle/RAC/11gR2/grid sh runcluvfy.sh stage -pre crsinst -n linux1,linux2 –verbose See Appendix A for output Only one package has failed and that is fine, red hat comes with older version of glibc, but the cluvfy script is looking for the latest script Check: Package existence for "glibc-2.3.4-2.41" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 glibc-2.3.4-2.39 glibc-2.3.4-2.41 failed Result: Package existence check failed for "glibc-2.3.4-2.41"



./runInstaller





Check Network Requirements: Check Network Requirements: Check Network Requirements: Check Network Requirements: In this release there are two new network related components

– SCAN and GNS.

Single Client Access Name (SCAN) for the Cluster: Single Client Access Name (SCAN) for the Cluster: Single Client Access Name (SCAN) for the Cluster: Single Client Access Name (SCAN) for the Cluster:

During Typical installation, you are prompted to confirm the default Single Client Access

Name (SCAN), which is used to connect to databases within the cluster irrespective of which

nodes they are running on. By default, the name used as the SCAN is also the name of the

cluster. The default value for the SCAN is based on the local node name. If you change the

SCAN from the default, then the name that you use must be globally unique throughout your

enterprise.

In a Typical installation, the SCAN is also the name of the cluster. The SCAN and cluster

name must be at least one character long and no more than 15 characters in length, must be



alphanumeric, and may contain hyphens (-). If you require a SCAN that is longer than 15

characters, then select an Advanced installation.

Configure the following addresses:Configure the following addresses:Configure the following addresses:Configure the following addresses:

• A public IP address for each node

• A virtual IP address for each node

• A single client access name (SCAN) configured on the domain name server (DNS)domain name server (DNS)domain name server (DNS)domain name server (DNS) or Grid Grid Grid Grid

Naming Service (GNS)Naming Service (GNS)Naming Service (GNS)Naming Service (GNS) for Round Robin resolution to three addresses (recommended) or at

least one address (check documentation)

Some more information on Single Client Access Name ☺

Single Client Access Name (SCAN) for the Cluster

If you have ever been tasked with extending an Oracle RAC cluster by adding a new node (or shrinking a RAC cluster by removing a node), then you know the pain of going through a list of all clients and updating their SQL*Net or JDBC configuration to reflect the new or deleted node! To address this problem, Oracle 11g Release 2 introduced a new feature known as Single Client Access Name or SCAN for short. SCAN is a new feature that provides a single host name for clients to access an Oracle Database running in a cluster. Clients using SCAN do not need to change their TNS configuration if you add or remove nodes in the cluster. The SCAN resource and its associated IP address(s) provide a stable name for clients to use for connections, independent of the nodes that make up the cluster. You will be asked to provide the host name and up to three IP addresses to be used for the SCAN resource during the interview phase of the Oracle grid infrastructure installation. For high availability and scalability, Oracle recommends that you configure the SCAN name so that it resolves to three IP addresses. At a minimum, the SCAN must resolve to at least one address.

The SCAN virtual IP name is similar to the names used for a node's virtual IP addresses, such as racnode1-vip. However, unlike a virtual IP, the SCAN is associated with the entire cluster, rather than an individual node, and can be associated with multiple IP addresses, not just one address. Note that SCAN addresses, virtual IP addresses, and public IP addresses must all be on the same subnet.

The SCAN should be configured so that it is resolvable either by using Grid Naming Service (GNS) within the cluster, or by using Domain Name Service (DNS) resolution.

In this article, I will configure SCAN to resolve to only one, manually configured static IP address using the DNS method (but not actually defining it in DNS)

http://www.oracle.com/technology/pub/articles/hunter-rac11gr2-iscsi-3.html#Configure%20Grid%20Plug%20and%20Play%20Information%20-%20SCAN

http://www.oracle.com/technology/pub/articles/hunter-rac11gr2-iscsi-3.html#Configure%20Grid%20Plug%20and%20Play%20Information%20-%20SCAN

http://www.oracle.com/technology/pub/articles/hunter-rac11gr2-iscsi-3.html#Configuring%20SCAN%20without%20DNS



192.168.1.187 racnode-cluster-scan – make the entry in /etc/hosts







Enter Oracle password and click Setup



















The above error can safely be ignored







[root@linux1 ~]# /d01/oracle/app/oraInventory/orainstRoot.sh Changing permissions of /d01/oracle/app/oraInventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /d01/oracle/app/oraInventory to oinstall. The execution of the script is complete. [root@linux2 ~]# /d01/oracle/app/oraInventory/orainstRoot.sh Changing permissions of /d01/oracle/app/oraInventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /d01/oracle/app/oraInventory to oinstall. The execution of the script is complete.



Run root.sh on linux1 and linux2 cd /d01/oracle/app/11.2.0/grid/ [root@linux1 grid]# sh root.sh See Appendix B and C for outputs

Click OK





crs_stat -t

Post-Install Actions

By default, the Global Services Daemon (GSD) is not started on the cluster. To start GSD, change directory to the <CRS_HOME> and issue the following commands:

srvctl enable nodeapps –g srvctl start nodeapps



ora.oc4j is offine, that is fine Successful Oracle Clusterware operation can also be verified using the following command:



./crsctl check cluster -all



References: Thanks to Jeff Hunter, without his help I couldn’t have done this assignment http://www.rachelp.nl/index_kb.php?menu=articles&actie=show&id=61 11gR2 Grid: root.sh fails to start the clusterware on the second node (Doc ID 981357.1)

1. short-term: disable the firewall on all nodes, on Linux this can be done by running the following command(s) as the root user on each node of the cluster:

service iptables stop service ip6tables stop

To permanently disable the firewall, use:

chkconfig iptables off chkconfig ip6tables off

2. long-term: exclude all traffic on the private network from the firewall configuration.

http://coskan.wordpress.com/ http://coskan.wordpress.com/2009/12/07/root-sh-failed-after-asm-disk-creation-for-11gr2-grid-infrastructure/ http://oracle-base.com/forums/viewtopic.php?f=1&t=11307&start=0

1. How to Proceed from Failed 11gR2 Grid Infrastructure (CRS) Installation [ID 942166.1]

Issues and solutions cd /home/oracle/app/11.2.0/grid/log/linux2 cat alertlinux2.log [ctssd(27935)]CRS-2409:The clock on host linux2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

http://www.rachelp.nl/index_kb.php?menu=articles&actie=show&id=61

http://coskan.wordpress.com/2009/12/07/root-sh-failed-after-asm-disk-creation-for-11gr2-grid-infrastructure/

http://coskan.wordpress.com/2009/12/07/root-sh-failed-after-asm-disk-creation-for-11gr2-grid-infrastructure/



2010-07-24 05:26:20.471 [ctssd(27935)]CRS-2412:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /home/oracle/app/11.2.0/grid/log/linux2/ctssd/octssd.log. 2010-07-24 05:26:20.471 [ctssd(27935)]CRS-2409:The clock on host linux2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode. http://www.oracle.com/technology/pub/articles/hunter-rac11gr2-iscsi.html http://arjudba.blogspot.com/2010/03/in-11gr2-grid-rootsh-fails-with-crs.html How to Proceed from Failed 11gR2 Grid Infrastructure (CRS) Installation [ID 942166.1] -- If root.sh has failed, run the below commands to clean up -- first node cd /home/oracle/app/11.2.0/grid/crs/install ./rootcrs.pl -verbose -deconfig -force -- second node cd /home/oracle/app/11.2.0/grid/crs/install ./rootcrs.pl -verbose -deconfig -force -lastnode To add vip manually [root@linux2 bin]# ./srvctl add vip -n linux2 -A 192.168.0.201/255.255.255.0/eth0 -k 1 [root@linux2 bin]# ./srvctl config vip -n linux2 http://forums.oracle.com/forums/thread.jspa?messageID=4039293

http://forums.oracle.com/forums/thread.jspa?messageID=4039293



rpm -q binutils-2.15.92.0.2 \ compat-libstdc++-33.2.3 \ elfutils-libelf-0.97 \ elfutils-libelf-devel-0.97 \ gcc-3.4.6 \ gcc-c++-3.4.6 \ glibc-2.3.4-2.41 \ glibc-common-2.3.4 \ glibc-devel-2.3.4 \ glibc-headers-2.3.4 \ libaio-devel-0.3.105 \ libaio-0.3.105 \ libgcc-3.4.6 \ libstdc++-3.4.6 \ libstdc++-devel-3.4.6 \ make-3.80 \ pdksh-5.2.14 \ sysstat-5.0.5 \



unixODBC-2.2.11 \ unixODBC-devel-2.2.11 http://forums.oracle.com/forums/thread.jspa?messageID=4345930 http://www.oracle.com/technology/pub/articles/wartak-rac-vm_3.html http://wiki.oracle.com/page/11gR2+RAC+on+a+Mac+-+Part+6 –nice one

INFO: Checking Single Client Access Name (SCAN)... INFO: Checking name resolution setup for "racnode-cluster-scan"... INFO: ERROR: INFO: PRVF-4657 : Name resolution setup check for "racnode-cluster-scan" (IP address: 216.24.138.153) failed INFO: ERROR: INFO: PRVF-4657 : Name resolution setup check for "racnode-cluster-scan" (IP address: 192.168.1.187) failed

http://forums.oracle.com/forums/thread.jspa?messageID=4345930

http://www.oracle.com/technology/pub/articles/wartak-rac-vm_3.html

http://wiki.oracle.com/page/11gR2+RAC+on+a+Mac+-+Part+6



INFO: ERROR: INFO: PRVF-4664 : Found inconsistent name resolution entries for SCAN name "racnode-cluster-scan" INFO: Verification of SCAN VIP and Listener setup failed

Provided this is the only error reported by the CVU, it would be safe to ignore this check and continue by clicking the [Next] button in OUI and move forward with the Oracle grid infrastructure installation. This is documented in Doc ID: 887471.1 on the My Oracle Support web site.

If on the other hand you want the CVU to complete successfully while still only defining the SCAN in the hosts file, simply modify the nslookup utility as root on both Oracle RAC nodes as follows.

First, rename the original nslookup binary to nslookup.original on both Oracle RAC nodes:

[root@racnode1 ~]# mv /usr/bin/nslookup /usr/bin/nslookup.original

Next, create a new shell script named /usr/bin/nslookup as shown below while replacing 24.154.1.34 with your primary DNS, racnode-cluster-scan with your SCAN host name, and 192.168.1.187 with your SCAN IP address:

#!/bin/bash HOSTNAME=${1} if [[ $HOSTNAME = "racnode-cluster-scan" ]]; then echo "Server: 24.154.1.34" echo "Address: 24.154.1.34#53" echo "Non-authoritative answer:" echo "Name: racnode-cluster-scan" echo "Address: 192.168.1.187" else /usr/bin/nslookup.original $HOSTNAME fi

Finally, change the new nslookup shell script to executable:

[root@racnode1 ~]# chmod 755 /usr/bin/nslookup Remember to perform these actions on both Oracle RAC nodes.

The new nslookup shell script simply echo's back your SCAN IP address whenever the CVU calls nslookup with your SCAN host name; otherwise, it calls the original nslookup binary.

https://support.oracle.com/CSP/ui/flash.html#tab=KBHome%28page=KBHome&id=%28%29%29%2C%28page=KBNavigator&id=%28viewingMode=1143&bmDocID=887471.1&from=BOOKMARK&bmDocType=PROBLEM&bmDocDsrc=DOCUMENT&bmDocTitle=PRVF-4664%20:%20Found%20inconsistent%20name%20resolution%20entries%20for%20SCAN%20name%29%29

http://www.myoraclesupport.com/

http://www.myoraclesupport.com/



The CVU will now pass during the Oracle grid infrastructure installation when it attempts to verify your SCAN:

[grid@racnode1 ~]$ cluvfy comp scan -verbose Verifying scan Checking Single Client Access Name (SCAN)... SCAN VIP name Node Running? ListenerName Port Running? ---------------- ------------ ------------ ------------ ------------ ------------ racnode-cluster-scan racnode1 true LISTENER 1521 true Checking name resolution setup for "racnode-cluster-scan"... SCAN Name IP Address Status Comment ------------ ------------------------ ------------------------ ---------- racnode-cluster-scan 192.168.1.187 passed Verification of SCAN VIP and Listener setup passed Verification of scan was successful. =============================================================================== [grid@racnode2 ~]$ cluvfy comp scan -verbose Verifying scan Checking Single Client Access Name (SCAN)... SCAN VIP name Node Running? ListenerName Port Running? ---------------- ------------ ------------ ------------ ------------ ------------ racnode-cluster-scan racnode1 true LISTENER 1521 true Checking name resolution setup for "racnode-cluster-scan"... SCAN Name IP Address Status Comment ------------ ------------------------ ------------------------ ---------- racnode-cluster-scan 192.168.1.187 passed Verification of SCAN VIP and Listener setup passed Verification of scan was successful.

Now retry



Appendix A sh runcluvfy.sh stage -pre crsinst -n linux1,linux2 -verbose Performing pre-checks for cluster services setup Checking node reachability... Check: Node reachability from node "linux1" Destination Node Reachable? ------------------------------------ ------------------------ linux1 yes linux2 yes Result: Node reachability check passed from node "linux1" Checking user equivalence... Check: User equivalence for user "oracle" Node Name Comment ------------------------------------ ------------------------



linux2 failed linux1 passed Result: PRVF-4007 : User equivalence check failed for user "oracle" WARNING: User equivalence is not set for nodes: linux2 Verification will proceed with nodes: linux1 Checking node connectivity... Checking hosts config file... Node Name Status Comment ------------ ------------------------ ------------------------ linux1 passed Verification of the hosts config file successful Interface information for node "linux1" Name IP Address Subnet Gateway Def. Gateway HW Address MTU ------ --------------- --------------- --------------- --------------- ----------------- ------ eth0 192.168.1.3 192.168.1.0 0.0.0.0 192.168.1.1 00:08:A1:95:2F:61 1500 eth1 192.168.0.101 192.168.0.0 0.0.0.0 192.168.1.1 00:13:72:22:4C:78 1500 Check: Node connectivity of subnet "192.168.1.0" Result: Node connectivity passed for subnet "192.168.1.0" with node(s) linux1 Check: TCP connectivity of subnet "192.168.1.0" Result: TCP connectivity check passed for subnet "192.168.1.0" Check: Node connectivity of subnet "192.168.0.0" Result: Node connectivity passed for subnet "192.168.0.0" with node(s) linux1 Check: TCP connectivity of subnet "192.168.0.0" Result: TCP connectivity check passed for subnet "192.168.0.0" Interfaces found on subnet "192.168.1.0" that are likely candidates for VIP are: linux1 eth0:192.168.1.3 Interfaces found on subnet "192.168.0.0" that are likely candidates for a private interconnect are: linux1 eth1:192.168.0.101 Result: Node connectivity check passed



Check: Total memory Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 3.71GB (3895072.0KB) 1.5GB (1572864.0KB) passed Result: Total memory check passed Check: Available memory Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 3.32GB (3479748.0KB) 50MB (51200.0KB) passed Result: Available memory check passed Check: Swap space Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 5.86GB (6144820.0KB) 3.71GB (3895072.0KB) passed Result: Swap space check passed Check: Free disk space for "linux1:/tmp" Path Node Name Mount point Available Required Comment ---------------- ------------ ------------ ------------ ------------ ------------ /tmp linux1 / 6.82GB 1GB passed Result: Free disk space check passed for "linux1:/tmp" Check: User existence for "oracle" Node Name Status Comment ------------ ------------------------ ------------------------ linux1 exists passed Result: User existence check passed for "oracle" Check: Group existence for "oinstall" Node Name Status Comment ------------ ------------------------ ------------------------ linux1 exists passed Result: Group existence check passed for "oinstall" Check: Group existence for "dba" Node Name Status Comment ------------ ------------------------ ------------------------ linux1 exists passed Result: Group existence check passed for "dba" Check: Membership of user "oracle" in group "oinstall" [as Primary] Node Name User Exists Group Exists User in Group Primary Comment ---------------- ------------ ------------ ------------ ------------ ------------ linux1 yes yes yes yes passed Result: Membership check for user "oracle" in group "oinstall" [as Primary] passed Check: Membership of user "oracle" in group "dba" Node Name User Exists Group Exists User in Group Comment ---------------- ------------ ------------ ------------ ----------------



linux1 yes yes yes passed Result: Membership check for user "oracle" in group "dba" passed Check: Run level Node Name run level Required Comment ------------ ------------------------ ------------------------ ---------- linux1 5 3,5 passed Result: Run level check passed Check: Hard limits for "maximum open file descriptors" Node Name Type Available Required Comment ---------------- ------------ ------------ ------------ ---------------- linux1 hard 65536 65536 passed Result: Hard limits check passed for "maximum open file descriptors" Check: Soft limits for "maximum open file descriptors" Node Name Type Available Required Comment ---------------- ------------ ------------ ------------ ---------------- linux1 soft 65536 1024 passed Result: Soft limits check passed for "maximum open file descriptors" Check: Hard limits for "maximum user processes" Node Name Type Available Required Comment ---------------- ------------ ------------ ------------ ---------------- linux1 hard 61416 16384 passed Result: Hard limits check passed for "maximum user processes" Check: Soft limits for "maximum user processes" Node Name Type Available Required Comment ---------------- ------------ ------------ ------------ ---------------- linux1 soft 61416 2047 passed Result: Soft limits check passed for "maximum user processes" Check: System architecture Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 i686 x86 passed Result: System architecture check passed Check: Kernel version Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 2.6.9-67.EL 2.6.9 passed Result: Kernel version check passed Check: Kernel parameter for "semmsl" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- linux1 256 250 passed Result: Kernel parameter check passed for "semmsl" Check: Kernel parameter for "semmns" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- linux1 32000 32000 passed Result: Kernel parameter check passed for "semmns" Check: Kernel parameter for "semopm"



Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- linux1 100 100 passed Result: Kernel parameter check passed for "semopm" Check: Kernel parameter for "semmni" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- linux1 142 128 passed Result: Kernel parameter check passed for "semmni" Check: Kernel parameter for "shmmax" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- linux1 4294967295 536870912 passed Result: Kernel parameter check passed for "shmmax" Check: Kernel parameter for "shmmni" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- linux1 4096 4096 passed Result: Kernel parameter check passed for "shmmni" Check: Kernel parameter for "shmall" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- linux1 2097152 2097152 passed Result: Kernel parameter check passed for "shmall" Check: Kernel parameter for "file-max" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- linux1 6815744 6815744 passed Result: Kernel parameter check passed for "file-max" Check: Kernel parameter for "ip_local_port_range" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- linux1 between 9000 & 65500 between 9000 & 65500 passed Result: Kernel parameter check passed for "ip_local_port_range" Check: Kernel parameter for "rmem_default" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- linux1 262144 262144 passed Result: Kernel parameter check passed for "rmem_default" Check: Kernel parameter for "rmem_max" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- linux1 4194304 4194304 passed Result: Kernel parameter check passed for "rmem_max" Check: Kernel parameter for "wmem_default" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- linux1 262144 262144 passed Result: Kernel parameter check passed for "wmem_default"



Check: Kernel parameter for "wmem_max" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- linux1 1048576 1048576 passed Result: Kernel parameter check passed for "wmem_max" Check: Kernel parameter for "aio-max-nr" Node Name Configured Required Comment ------------ ------------------------ ------------------------ ---------- linux1 1048576 1048576 passed Result: Kernel parameter check passed for "aio-max-nr" Check: Package existence for "make-3.80" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 make-3.80-6.EL4 make-3.80 passed Result: Package existence check passed for "make-3.80" Check: Package existence for "binutils-2.15.92.0.2" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 binutils-2.15.92.0.2-24 binutils-2.15.92.0.2 passed Result: Package existence check passed for "binutils-2.15.92.0.2" Check: Package existence for "gcc-3.4.6" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 gcc-3.4.6-9 gcc-3.4.6 passed Result: Package existence check passed for "gcc-3.4.6" Check: Package existence for "gcc-c++-3.4.6" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 gcc-c++-3.4.6-9 gcc-c++-3.4.6 passed Result: Package existence check passed for "gcc-c++-3.4.6" Check: Package existence for "libaio-0.3.105" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 libaio-0.3.105-2 libaio-0.3.105 passed Result: Package existence check passed for "libaio-0.3.105" Check: Package existence for "glibc-2.3.4-2.41" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 glibc-2.3.4-2.39 glibc-2.3.4-2.41 failed Result: Package existence check failed for "glibc-2.3.4-2.41" Check: Package existence for "compat-libstdc++-33-3.2.3" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 compat-libstdc++-33-3.2.3-47.3 compat-libstdc++-33-3.2.3 passed Result: Package existence check passed for "compat-libstdc++-33-3.2.3" Check: Package existence for "elfutils-libelf-0.97" Node Name Available Required Comment



------------ ------------------------ ------------------------ ---------- linux1 elfutils-libelf-0.97.1-5 elfutils-libelf-0.97 passed Result: Package existence check passed for "elfutils-libelf-0.97" Check: Package existence for "elfutils-libelf-devel-0.97" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 elfutils-libelf-devel-0.97.1-5 elfutils-libelf-devel-0.97 passed Result: Package existence check passed for "elfutils-libelf-devel-0.97" Check: Package existence for "glibc-common-2.3.4" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 glibc-common-2.3.4-2.39 glibc-common-2.3.4 passed Result: Package existence check passed for "glibc-common-2.3.4" Check: Package existence for "glibc-devel-2.3.4" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 glibc-devel-2.3.4-2.39 glibc-devel-2.3.4 passed Result: Package existence check passed for "glibc-devel-2.3.4" Check: Package existence for "glibc-headers-2.3.4" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 glibc-headers-2.3.4-2.39 glibc-headers-2.3.4 passed Result: Package existence check passed for "glibc-headers-2.3.4" Check: Package existence for "libaio-devel-0.3.105" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 libaio-devel-0.3.105-2 libaio-devel-0.3.105 passed Result: Package existence check passed for "libaio-devel-0.3.105" Check: Package existence for "libgcc-3.4.6" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 libgcc-3.4.6-9 libgcc-3.4.6 passed Result: Package existence check passed for "libgcc-3.4.6" Check: Package existence for "libstdc++-3.4.6" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 libstdc++-3.4.6-9 libstdc++-3.4.6 passed Result: Package existence check passed for "libstdc++-3.4.6" Check: Package existence for "libstdc++-devel-3.4.6" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 libstdc++-devel-3.4.6-9 libstdc++-devel-3.4.6 passed Result: Package existence check passed for "libstdc++-devel-3.4.6" Check: Package existence for "sysstat-5.0.5" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 sysstat-5.0.5-16.rhel4 sysstat-5.0.5 passed Result: Package existence check passed for "sysstat-5.0.5"



Check: Package existence for "unixODBC-2.2.11" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 unixODBC-2.2.11-1.RHEL4.1 unixODBC-2.2.11 passed Result: Package existence check passed for "unixODBC-2.2.11" Check: Package existence for "unixODBC-devel-2.2.11" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 unixODBC-devel-2.2.11-1.RHEL4.1 unixODBC-devel-2.2.11 passed Result: Package existence check passed for "unixODBC-devel-2.2.11" Check: Package existence for "pdksh-5.2.14" Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 pdksh-5.2.14-30.6 pdksh-5.2.14 passed Result: Package existence check passed for "pdksh-5.2.14" Checking for multiple users with UID value 0 Result: Check for multiple users with UID value 0 passed Check: Current group ID Result: Current group ID check passed Checking Core file name pattern consistency... Core file name pattern consistency check passed. Checking to make sure user "oracle" is not in "root" group Node Name Status Comment ------------ ------------------------ ------------------------ linux1 does not exist passed Result: User "oracle" is not part of "root" group. Check passed Check default user file creation mask Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- linux1 0022 0022 passed Result: Default user file creation mask check passed Starting Clock synchronization checks using Network Time Protocol(NTP)... NTP Configuration file check started... The NTP configuration file "/etc/ntp.conf" is available on all nodes NTP Configuration file check passed Checking daemon liveness... Check: Liveness for "ntpd" Node Name Running? ------------------------------------ ------------------------ linux1 yes Result: Liveness check passed for "ntpd" Checking NTP daemon command line for slewing option "-x" Check: NTP daemon command line Node Name Slewing Option Set? ------------------------------------ ------------------------



linux1 yes Result: NTP daemon slewing option check passed Checking NTP daemon's boot time configuration, in file "/etc/sysconfig/ntpd", for slewing option "-x" Check: NTP daemon's boot time configuration Node Name Slewing Option Set? ------------------------------------ ------------------------ linux1 yes Result: NTP daemon's boot time configuration check for slewing option passed NTP common Time Server Check started... NTP Time Server "202.71.140.36" is common to all nodes on which the NTP daemon is running NTP Time Server "209.81.9.7" is common to all nodes on which the NTP daemon is running NTP Time Server ".INIT." is common to all nodes on which the NTP daemon is running NTP Time Server "218.45.21.199" is common to all nodes on which the NTP daemon is running Check of common NTP Time Server passed Clock time offset check from NTP Time Server started... Checking on nodes "[linux1]"... Check: Clock time offset from NTP Time Server Time Server: 202.71.140.36 Time Offset Limit: 1000.0 msecs Node Name Time Offset Status ------------ ------------------------ ------------------------ linux1 1.711 passed Time Server "202.71.140.36" has time offsets that are within permissible limits for nodes "[linux1]". Time Server: 209.81.9.7 Time Offset Limit: 1000.0 msecs Node Name Time Offset Status ------------ ------------------------ ------------------------ linux1 11.497 passed Time Server "209.81.9.7" has time offsets that are within permissible limits for nodes "[linux1]". Time Server: .INIT. Time Offset Limit: 1000.0 msecs Node Name Time Offset Status ------------ ------------------------ ------------------------ linux1 0.0 passed Time Server ".INIT." has time offsets that are within permissible limits for nodes "[linux1]". Time Server: 218.45.21.199 Time Offset Limit: 1000.0 msecs Node Name Time Offset Status ------------ ------------------------ ------------------------ linux1 23.78 passed



Time Server "218.45.21.199" has time offsets that are within permissible limits for nodes "[linux1]". Clock time offset check passed Result: Clock synchronization check using Network Time Protocol(NTP) passed Pre-check for cluster services setup was unsuccessful on all the nodes. _]0;oracle@linux1:/d01/oracle/RAC/11gR2/grid_[oracle@linux1 grid]$ Script done on Tue 27 Jul 2010 08:02:17 AM IST

Appendix B cd /d01/oracle/app/11.2.0/grid/ [root@linux1 grid]# script akr.txt Script started, file is akr.txt [root@linux1 grid]# sh root.sh Running Oracle 11g root.sh script... The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /d01/oracle/app/11.2.0/grid Enter the full pathname of the local bin directory: [/usr/local/bin]: The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y Copying dbhome to /usr/local/bin ... The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y Copying oraenv to /usr/local/bin ... The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y Copying coraenv to /usr/local/bin ... Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. 2010-07-27 08:46:51: Parsing the host name 2010-07-27 08:46:51: Checking for super user privileges 2010-07-27 08:46:51: User has super user privileges Using configuration parameter file: /d01/oracle/app/11.2.0/grid/crs/install/crsc onfig_params



Creating trace directory LOCAL ADD MODE Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. root wallet root wallet cert root cert export peer wallet profile reader wallet pa wallet peer wallet keys pa wallet keys peer cert request pa cert request peer cert pa cert peer root cert TP profile reader root cert TP pa root cert TP peer pa cert TP pa peer cert TP profile reader pa cert TP profile reader peer cert TP peer user cert pa user cert Adding daemon to inittab CRS-4123: Oracle High Availability Services has been started. ohasd is starting ADVM/ACFS is not supported on Redhat 4 CRS-2672: Attempting to start 'ora.gipcd' on 'linux1' CRS-2672: Attempting to start 'ora.mdnsd' on 'linux1' CRS-2676: Start of 'ora.gipcd' on 'linux1' succeeded CRS-2676: Start of 'ora.mdnsd' on 'linux1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'linux1' CRS-2676: Start of 'ora.gpnpd' on 'linux1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'linux1' CRS-2676: Start of 'ora.cssdmonitor' on 'linux1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'linux1' CRS-2672: Attempting to start 'ora.diskmon' on 'linux1' CRS-2676: Start of 'ora.diskmon' on 'linux1' succeeded CRS-2676: Start of 'ora.cssd' on 'linux1' succeeded CRS-2672: Attempting to start 'ora.ctssd' on 'linux1'



CRS-2676: Start of 'ora.ctssd' on 'linux1' succeeded ASM created and started successfully. DiskGroup DATA created successfully. clscfg: -install mode specified Successfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. CRS-2672: Attempting to start 'ora.crsd' on 'linux1' CRS-2676: Start of 'ora.crsd' on 'linux1' succeeded CRS-4256: Updating the profile Successful addition of voting disk 0482638d4e5d4f0fbfd40e025d6ca900. Successful addition of voting disk 63878fa4923f4f5dbfadf6a8be49a18a. Successful addition of voting disk 433cdc1d1b7f4f1fbf7f126bc0452222. Successfully replaced voting disk group with +DATA. CRS-4256: Updating the profile CRS-4266: Voting file(s) successfully replaced ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 0482638d4e5d4f0fbfd40e025d6ca900 (ORCL:VOL1) [DATA] 2. ONLINE 63878fa4923f4f5dbfadf6a8be49a18a (ORCL:VOL2) [DATA] 3. ONLINE 433cdc1d1b7f4f1fbf7f126bc0452222 (ORCL:VOL3) [DATA] Located 3 voting disk(s). CRS-2673: Attempting to stop 'ora.crsd' on 'linux1' CRS-2677: Stop of 'ora.crsd' on 'linux1' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'linux1' CRS-2677: Stop of 'ora.asm' on 'linux1' succeeded CRS-2673: Attempting to stop 'ora.ctssd' on 'linux1' CRS-2677: Stop of 'ora.ctssd' on 'linux1' succeeded CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'linux1' CRS-2677: Stop of 'ora.cssdmonitor' on 'linux1' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'linux1' CRS-2677: Stop of 'ora.cssd' on 'linux1' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'linux1' CRS-2677: Stop of 'ora.gpnpd' on 'linux1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'linux1' CRS-2677: Stop of 'ora.gipcd' on 'linux1' succeeded CRS-2673: Attempting to stop 'ora.mdnsd' on 'linux1' CRS-2677: Stop of 'ora.mdnsd' on 'linux1' succeeded CRS-2672: Attempting to start 'ora.mdnsd' on 'linux1' CRS-2676: Start of 'ora.mdnsd' on 'linux1' succeeded



CRS-2672: Attempting to start 'ora.gipcd' on 'linux1' CRS-2676: Start of 'ora.gipcd' on 'linux1' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'linux1' CRS-2676: Start of 'ora.gpnpd' on 'linux1' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'linux1' CRS-2676: Start of 'ora.cssdmonitor' on 'linux1' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'linux1' CRS-2672: Attempting to start 'ora.diskmon' on 'linux1' CRS-2676: Start of 'ora.diskmon' on 'linux1' succeeded CRS-2676: Start of 'ora.cssd' on 'linux1' succeeded CRS-2672: Attempting to start 'ora.ctssd' on 'linux1' CRS-2676: Start of 'ora.ctssd' on 'linux1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'linux1' CRS-2676: Start of 'ora.asm' on 'linux1' succeeded CRS-2672: Attempting to start 'ora.crsd' on 'linux1' CRS-2676: Start of 'ora.crsd' on 'linux1' succeeded CRS-2672: Attempting to start 'ora.evmd' on 'linux1' CRS-2676: Start of 'ora.evmd' on 'linux1' succeeded CRS-2672: Attempting to start 'ora.asm' on 'linux1' CRS-2676: Start of 'ora.asm' on 'linux1' succeeded CRS-2672: Attempting to start 'ora.DATA.dg' on 'linux1' CRS-2676: Start of 'ora.DATA.dg' on 'linux1' succeeded linux1 2010/07/27 08:54:48 /d01/oracle/app/11.2.0/grid/cdata/linux1/back up_20100727_085448.olr Configure Oracle Grid Infrastructure for a Cluster ... succeeded Updating inventory properties for clusterware Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 6000 MB Passed The inventory pointer is located at /etc/oraInst.loc The inventory is located at /d01/oracle/app/oraInventory 'UpdateNodeList' was successful. Appendix C sh root.sh Running Oracle 11g root.sh script... The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /d01/oracle/app/11.2.0/grid



Enter the full pathname of the local bin directory: [/usr/local/bin]: Copying dbhome to /usr/local/bin ... Copying oraenv to /usr/local/bin ... Copying coraenv to /usr/local/bin ... Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. 2010-07-27 08:57:25: Parsing the host name 2010-07-27 08:57:25: Checking for super user privileges 2010-07-27 08:57:25: User has super user privileges Using configuration parameter file: /d01/oracle/app/11.2.0/grid/crs/install/crsc onfig_params Creating trace directory LOCAL ADD MODE Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. Adding daemon to inittab CRS-4123: Oracle High Availability Services has been started. ohasd is starting ADVM/ACFS is not supported on Redhat 4 CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS d aemon on node linux1, number 1, and is terminating An active cluster was found during exclusive startup, restarting to join the clu ster CRS-2672: Attempting to start 'ora.mdnsd' on 'linux2' CRS-2676: Start of 'ora.mdnsd' on 'linux2' succeeded CRS-2672: Attempting to start 'ora.gipcd' on 'linux2' CRS-2676: Start of 'ora.gipcd' on 'linux2' succeeded CRS-2672: Attempting to start 'ora.gpnpd' on 'linux2' CRS-2676: Start of 'ora.gpnpd' on 'linux2' succeeded CRS-2672: Attempting to start 'ora.cssdmonitor' on 'linux2' CRS-2676: Start of 'ora.cssdmonitor' on 'linux2' succeeded CRS-2672: Attempting to start 'ora.cssd' on 'linux2' CRS-2672: Attempting to start 'ora.diskmon' on 'linux2' CRS-2676: Start of 'ora.diskmon' on 'linux2' succeeded CRS-2676: Start of 'ora.cssd' on 'linux2' succeeded CRS-2672: Attempting to start 'ora.ctssd' on 'linux2' CRS-2676: Start of 'ora.ctssd' on 'linux2' succeeded



CRS-2672: Attempting to start 'ora.asm' on 'linux2' CRS-2676: Start of 'ora.asm' on 'linux2' succeeded CRS-2672: Attempting to start 'ora.crsd' on 'linux2' CRS-2676: Start of 'ora.crsd' on 'linux2' succeeded CRS-2672: Attempting to start 'ora.evmd' on 'linux2' CRS-2676: Start of 'ora.evmd' on 'linux2' succeeded linux2 2010/07/27 09:01:36 /d01/oracle/app/11.2.0/grid/cdata/linux2/back up_20100727_090136.olr Preparing packages for installation... cvuqdisk-1.0.7-1 Configure Oracle Grid Infrastructure for a Cluster ... succeeded Updating inventory properties for clusterware Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 7994 MB Passed The inventory pointer is located at /etc/oraInst.loc The inventory is located at /d01/oracle/app/oraInventory 'UpdateNodeList' was successful.


Documents

Build Your Own Oracle RAC 11 g Release 2 Cluster on Red ... · If you have ever been tasked with extending an Oracle RAC cluster by adding a new node (or shrinking a RAC cluster by