http://www.oracle-base.com/articles/11g/OracleDB11gR2RACInstallationOnLinuxUsingNFS.php

http://www.oracle-base.com/articles/linux/UsingNFSWithASM.php

Setup two VM with redhat 5.5

To be consistent with the rest of the article, the following information should be set during the installation:

hostname: racdb1.localdomain

IP Address eth0: 191.192.192.128 (public address)

Default Gateway eth0: 191.192.192.254 (public address)

IP Address eth1: 192.168.0.101 (private address)

Default Gateway eth1: none

You are free to change the IP addresses to suit your network, but remember to stay consistent with those adjustments throughout the rest of the article.

Once the basic installation is complete, install the following packages whilst logged in as the root user. This includes the 64-bit and 32-bit versions of some packages.

# From Linux 5.5 DVDcd /media/cdrom/Serverrpm -Uvh kernel-headers*rpm -ivh glibc-2*x86_64.rpm glibc-common-*86_64.rpm glibc-devel* glibc-headers*rpm -Uvh binutils-2.*rpm -Uvh compat-libstdc++-33*.i386.rpmrpm -Uvh compat-libstdc++-33*.x86_64.rpmrpm -Uvh elfutils-libelf-0*.x86_64.rpmrpm -Uvh elfutils-libelf-devel*.x86_64.rpmrpm -Uvh gcc-4*.x86_64.rpmrpm -Uvh gcc-c++-4*.x86_64.rpmrpm -Uvh glibc-2.*rpm -Uvh glibc-common-2.*rpm -Uvh glibc-devel-2.*rpm -Uvh glibc-headers-2.*rpm -Uvh ksh-2*rpm -Uvh libaio-0.*rpm -Uvh libaio-devel-0.*rpm -Uvh libgcc-4.*rpm -Uvh libstdc++-4.*rpm -Uvh libstdc++-devel-4.*rpm -Uvh make-3.*rpm -Uvh libgomp-4.*rpm -Uvh sysstat-7.*rpm -Uvh unixODBC-2.*rpm -Uvh unixODBC-devel-2.*

#for Oracle 10g :rpm -Uvh setarch-2*rpm -Uvh make-3*rpm -Uvh glibc-2*

rpm -Uvh libaio-0*rpm -Uvh compat-libstdc++-33-3*rpm -Uvh compat-gcc-34-3*rpm -Uvh compat-gcc-34-c++-3*rpm -Uvh gcc-4*rpm -Uvh libXp-1*rpm -Uvh openmotif-2*rpm -Uvh compat-db-4*

Oracle Installation PrerequisitesPerform the following steps whilst logged into the "rac1" virtual machine as the root user.

Make sure the shared memory filesystem is big enough for Automatic Memory Manager to work.

# umount tmpfs# mount -t tmpfs shmfs -o size=1500m /dev/shmIf you are not using DNS, the "/etc/hosts" file must contain the following information.

127.0.0.1 localhost.localdomain localhost# Public191.192.192.128 racdb1.localdomain racdb1191.192.192.129 racdb2.localdomain racdb2# Private192.168.0.101 racdb1-priv.localdomain racdb1-priv192.168.0.102 racdb2-priv.localdomain racdb2-priv# Virtual191.192.192.228 racdb1-vip.localdomain racdb1-vip191.192.192.229 racdb2-vip.localdomain racdb2-vip# SCAN191.192.192.134 racdb-scan.localdomain racdb-scanNote. The SCAN address should not really be defined in the hosts file. Instead is should be defined on the DNS to round-robin between 3 addresses on the same subnet as the public IPs. For this installation, we will compromise and use the hosts file.

If you are using DNS, then only the first line should be present in the "/etc/hosts" file. The other entries are defined in the DNS, as described here.

Add or amend the following lines to the "/etc/sysctl.conf" file.

fs.aio-max-nr = 1048576fs.file-max = 6815744#kernel.shmall = 2097152#kernel.shmmax = 536870912kernel.shmmni = 4096# semaphores: semmsl, semmns, semopm, semmnikernel.sem = 250 32000 100 200net.ipv4.ip_local_port_range = 9000 65500net.core.rmem_default=262144net.core.rmem_max=4194304net.core.wmem_default=262144net.core.wmem_max=1048586Run the following command to change the current kernel parameters.

/sbin/sysctl -pAdd the following lines to the "vi /etc/security/limits.conf" file.

oracle soft nproc 2047

oracle hard nproc 16384oracle soft nofile 1024oracle hard nofile 65536Add the following lines to the "vi /etc/pam.d/login" file, if it does not already exist.

session required pam_limits.soDisable secure linux by editing the "/etc/selinux/config" file, making sure the SELINUX flag is set as follows.

SELINUX=disabledAlternatively, this alteration can be done using the GUI tool (System > Administration > Security Level and Firewall). Click on the SELinux tab and disable the feature.

Either configure NTP, or make sure it is not configured so the Oracle Cluster Time Synchronization Service (ctssd) can synchronize the times of the RAC nodes. In this case we will deconfigure NTP.

# service ntpd stopShutting down ntpd: [ OK ]# chkconfig ntpd off# mv /etc/ntp.conf /etc/ntp.conf.org# rm /var/run/ntpd.pidIf you are using NTP, you must add the "-x" option into the following line in the "/etc/sysconfig/ntpd" file.

OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"Then restart NTP.

# service ntpd restartCreate the new groups and users.

groupadd -g 1000 oinstallgroupadd -g 1200 dbauseradd -u 1100 -g oinstall -G dba oraclepasswd oracleCreate the directories in which the Oracle software will be installed.

mkdir -p /oracle/grid/11gGRID/gridmkdir -p /oracle/app/product/10gRDB/db_1mkdir -p /oracle/app/product/11gRDB/db_1chown -R oracle:oinstall /oraclechmod -R 775 /oracle/Login as the oracle user and add the following lines at the end of the .bash_profile file.

# Oracle SettingsTMP=/tmp; export TMPTMPDIR=$TMP; export TMPDIR

ORACLE_HOSTNAME=racdb1.localdomain; export ORACLE_HOSTNAMEORACLE_UNQNAME=RACDB; export ORACLE_UNQNAMEORA_CRS_HOME=/oracle/grid/11gGRID/grid; export ORA_CRS_HOMEORACLE_BASE=/oracle/app; export ORACLE_BASEORACLE_HOME=$ORACLE_BASE/app/product/10gRDB/db_1; export ORACLE_HOMEORACLE_SID=RACDB1; export ORACLE_SIDORACLE_TERM=xterm; export ORACLE_TERMPATH=/usr/sbin:$PATH; export PATHPATH=$ORACLE_HOME/bin:$PATH; export PATH

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH

CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

if [ $USER = "oracle" ]; then if [ $SHELL = "/bin/ksh" ]; then ulimit -p 16384 ulimit -n 65536 else ulimit -u 16384 -n 65536 fifi

alias 11g='export ORACLE_HOME=/oracle/app/product/11gRDB/db_1;export ORACLE_SID=FMS11G;export BASE_PATH=$ORACLE_BASE/scripts/general:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/maint/oracle;export PATH=${ORACLE_HOME}/bin:${BASE_PATH}:export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib'alias 10g='export ORACLE_HOME=/oracle/app/product/10gRDB/db_1;export ORACLE_SID=FMSDB;export BASE_PATH=$ORACLE_BASE/scripts/general:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/maint/oracle;export PATH=${ORACLE_HOME}/bin:${BASE_PATH}:export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib'alias asm='export ORACLE_HOME=$ORA_CRS_HOME;export ORACLE_SID=RACDB1;export BASE_PATH=$ORACLE_BASE/scripts/general:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/maint/oracle;export PATH=${ORACLE_HOME}/bin:${BASE_PATH}:export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib'

NFS Share SetupOn the NFS server create a directory to hold the shared files.

# mkdir -p /nfs/VM/nfs_shares/asmAdd the following line into the "/etc/exports" file. The export options are important, so don;t alter them unless you understand why you are doing it.

/nfs/VM/nfs_shares/asm_test *(rw,sync,no_wdelay,insecure_locks,no_root_squash)Make sure the NFS daemon is running.

# chkconfig nfs on# service nfs restart

NFS Mount SetupOn each machine that needs access to the NFS file system, perform the following operations.

Create a directory as a mount point for the NFS shares.

# mkdir -p /oracle/oradata# chown oracle:oinstall /oracle/oradataAdd the following line to the "/etc/fstab" file, where "nas1" should be the name of the NFS server. The mount options are important, so only alter them if you know what you are doing.

191.192.192.134:/nfs/VM/nfs_shares/asm /oracle/oradata nfs rw,bg,hard,nointr,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0 0 0Mount the file system and set the ownership.

mount /oracle/oradatachown -R oracle:oinstall /oracle/oradata

Create Shared FilesWe can use the "dd" command to create some empty files to present to ASM as devices. Since these are shared files this only needs to be done once regardless of whether this is a single instance or a RAC installation. Here I create 5 separate 10 Gig files.

su - oracledd if=/dev/zero of=/oracle/oradata/asm_dsk1 bs=1k count=10000000dd if=/dev/zero of=/oracle/oradata/asm_dsk2 bs=1k count=10000000dd if=/dev/zero of=/oracle/oradata/asm_dsk3 bs=1k count=10000000dd if=/dev/zero of=/oracle/oradata/asm_dsk4 bs=1k count=10000000#for OCR and VOTING disks and SPfile dd if=/dev/zero of=/oracle/oradata/asm_dsk5 bs=1k count=5000000

Using ASMLibIn Oracle 11gR2 it is recommended to use ASMLib. In order to use ASMLib we must use "losetup" to make the files look like block devices. Add the following lines to the "/etc/rc.local" file so they are run on reboot. You will also need to run the first 5 lines manually from the command line as the root user before you can proceed.

/sbin/losetup /dev/loop1 /oracle/oradata/asm_dsk1/sbin/losetup /dev/loop2 /oracle/oradata/asm_dsk2/sbin/losetup /dev/loop3 /oracle/oradata/asm_dsk3/sbin/losetup /dev/loop4 /oracle/oradata/asm_dsk4/sbin/losetup /dev/loop5 /oracle/oradata/asm_dsk5/usr/sbin/oracleasm scandisksNote. On reboot ASMLib automatically does a scan of the disks, but this happens before the rc.local is run, so we need to include the scan here to make sure the disks are visible when ASM starts up.

Before installing ASMLib you must first determine your current kernel.

# uname -rm2.6.18-164.el5 x86_64#Download the appropriate ASMLib RPMs from OTN. In this case we need:

These files can be found under /tmp/asmlib

oracleasm-support-2.1.3-1.el5.x86_64.rpm

oracleasmlib-2.0.4-1.el5.x86_64.rpm

oracleasm-2.6.18-164.el5-2.0.5-1.el5.x86_64.rpm

Install the packages using the following command.

# rpm -Uvh oracleasm*.rpmConfigure ASMLib using the following command.

# oracleasm configure -iConfiguring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM librarydriver. The following questions will determine whether the driver isloaded on boot and what permissions it will have. The current valueswill be shown in brackets ('[]'). Hitting <ENTER> without typing ananswer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: oracleDefault group to own the driver interface []: dbaStart Oracle ASM library driver on boot (y/n) [n]: yScan for Oracle ASM disks on boot (y/n) [y]: Writing Oracle ASM library driver configuration: done#Load the kernel module using the following command.

# /usr/sbin/oracleasm initLoading module "oracleasm": oracleasmMounting ASMlib driver filesystem: /dev/oracleasm# If you have any problems, run the following command to make sure you have the correct version of the driver.

# /usr/sbin/oracleasm update-driverMark the five shared disks as follows.

/usr/sbin/oracleasm createdisk DISK1 /dev/loop1/usr/sbin/oracleasm createdisk DISK2 /dev/loop2/usr/sbin/oracleasm createdisk DISK3 /dev/loop3/usr/sbin/oracleasm createdisk DISK4 /dev/loop4/usr/sbin/oracleasm createdisk DISK5 /dev/loop5Writing disk header: doneInstantiating disk: done# It is unnecessary, but we can run the "scandisks" command to refresh the ASM disk configuration.

# /usr/sbin/oracleasm scandisksReloading disk partitions: doneCleaning any stale ASM disks...Scanning system for ASM disks...#We can see the disk are now visible to ASM using the "listdisks" command.

# /usr/sbin/oracleasm listdisksDISK1DISK2DISK3DISK4DISK5#The disks are now configured and ready for the grid infrastructure installation. During the installation, instead of having to alter the disk discovery path, we are now presented with the disk we just marked as ASM disks.

Running Grid installation:

Step 1

Test