Installation SUN 10gR2 RAC ASM

Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH

Oracle Deutschland GmbH

Oracle RAC with ASM

Authors: Jürgen DömelOracle Deutschland GmbH

Michael Ahrens

SUN Microsystems GmbH

Created 05/05/2006 14:08:00

Last change: 29/11/2006 14:11:35 A11/P11

Version: 2.0

Page 1 of 141


Table of ContentsTable of Contents......................................................................................................................................................2Document Info...........................................................................................................................................................4

Content..............................................................................................................................................................4Cluster / host information..................................................................................................................................4RAC...................................................................................................................................................................4

Hardware Installation................................................................................................................................................5Architecture.......................................................................................................................................................5Drawing.............................................................................................................................................................5

Solaris Installation.....................................................................................................................................................6Solaris 10 u1......................................................................................................................................................6Setup slvm.........................................................................................................................................................6EIS-CD..............................................................................................................................................................7Setup IPMP groups on all nodes for all public network interfaces...................................................................8Modify NVRAM parameter:.............................................................................................................................8Setup Trunking for the Clusterinterconnect......................................................................................................9Last Systemanalyse...........................................................................................................................................9

Prepare Oracle Installation......................................................................................................................................10Set kernel parameters......................................................................................................................................10Create Users and Groups.................................................................................................................................10Share staging area............................................................................................................................................10Mount staging area..........................................................................................................................................10Modify /etc/hosts.............................................................................................................................................10Check SSH daemon.........................................................................................................................................11Setup SSH for oracle.......................................................................................................................................11Setup userenv for oracle..................................................................................................................................11Perform cluster check......................................................................................................................................11Workaround Bug.............................................................................................................................................12Prepare storage................................................................................................................................................12Verify device order..........................................................................................................................................12Trobleshooting device size..............................................................................................................................13Label devices...................................................................................................................................................13

Install Clusterware 10.2.0.1.....................................................................................................................................14Run installer....................................................................................................................................................14Trobleshooting Assistants...............................................................................................................................23Check Cluster Status.......................................................................................................................................25Install crsstat.sh...............................................................................................................................................26

Install ASM Home 10.2.0.1....................................................................................................................................27Run installer....................................................................................................................................................27

Create ASM Instance..............................................................................................................................................35Run DBCA......................................................................................................................................................35

Install Database Home 10.2.0.1..............................................................................................................................41Run installer ...................................................................................................................................................41

Install Management Agent......................................................................................................................................47Run installer....................................................................................................................................................47

Create Database ZDA..............................................................................................................................................55Run DBCA......................................................................................................................................................55Create login script for user oracle...................................................................................................................70

Install Patchset 10.2.0.2 on CRS Home..................................................................................................................71Stop all oracle processes.................................................................................................................................71Install binaries.................................................................................................................................................71Restart Custerware..........................................................................................................................................75

Install Patchset 10.2.0.2 on ASM Home.................................................................................................................76Stop Nodeapps.................................................................................................................................................76Install binaries.................................................................................................................................................76

Install Patchset 10.2.0.2 on DB Home....................................................................................................................81Verify service state..........................................................................................................................................81Install binaries.................................................................................................................................................81

Install Interim Patch 5117016 on DB Home...........................................................................................................87Install with opatch...........................................................................................................................................87Start services required to perform database upgrade......................................................................................89

Page 2 of 141


Upgrade database ZDA...................................................................................................................................89Restart cluster..................................................................................................................................................90How to Setup IPMP as Cluster Interconnect Doc ID: Note:368464.1..........................................................90

Add new node to cluster..........................................................................................................................................91Create links to devices.....................................................................................................................................91Change permissions.........................................................................................................................................91Modify /etc/hosts.............................................................................................................................................92Check SSH daemon.........................................................................................................................................92Enable user equivalence..................................................................................................................................92Perform cluster check......................................................................................................................................93Workaround OUI Bug.....................................................................................................................................93Install clusterware on new node......................................................................................................................93Install ASM home on new node......................................................................................................................97Install DB home on new node.......................................................................................................................101Create listener in DB Home..........................................................................................................................105Create ASM instance on new node...............................................................................................................110Create database instance on new node..........................................................................................................113End of Installation.........................................................................................................................................119

Remove Node from Cluster...................................................................................................................................120Remove instance...........................................................................................................................................120Remove Listener...........................................................................................................................................124Clean-up the ASM.........................................................................................................................................128Removing node from the database................................................................................................................128Removing node from the clusterware:..........................................................................................................128

Cluster Performance Tuning.................................................................................................................................130Apply Brownout Patches...............................................................................................................................130Set the CSS misscount to match the service level.........................................................................................130Run CSS in Real Time..................................................................................................................................130Set the check_interval for VIP addresses......................................................................................................130Remove the dependency between the VIP and ASM...................................................................................131Reduce the ping timeout................................................................................................................................131

Troubleshooting VIP Failover...............................................................................................................................132Adjusting Listener Configuration..................................................................................................................132Upgrade Oracle Home of Swingbench .........................................................................................................132Adjust SQLNet settings.................................................................................................................................132Set the ping timeout back to default..............................................................................................................132

Addendum.............................................................................................................................................................133Metalink Note 284752.1................................................................................................................................133Metalink Note 294430.1................................................................................................................................134Metalink Note 294430.1................................................................................................................................140

Page 3 of 141


Document Info

Content

This is an installation summary of the installation and patch process of the ZDA environment. It is based on the official procuct documentation provided by oracle cannot replace the those documents.

Cluster / host information

Hostnames: orac-rac-1, ora-rac-2, ora-rac,4

OS: Solaris 10

RAC

Oracle Cluster Name: crs

Oracle Versions 10.2.0.1

Page 4 of 141


Hardware Installation

Architecture

The cluster is based on two SF440 with 4 CPU's, 16GB RAM and 4 internal Disks. The internaldisks are mirrored with SLVM. Two disks are used for the OS and the other two are used for Oracle. Oracle will not be installed on shared disks because of the rolling upgrade capability. The shared disks are luns from a SAN and they are mirrored with Oracle ASM. The clusterinterconnect is realized with two trunked gigabitethernet connections.

Drawing

Page 5 of 141

2 x Interconnect

Storage A

Oracle RAC Cluster Node: ora-rac-1

Oracle RAC ClusterNode: ora-rac-2

Storage B

IPMP Groupipmp0

IPMP Groupipmp0

MpxIO MpxIO


Solaris Installation

Solaris 10 u1

Solaris 10 u1 SUNWCXall is installed.

• For SVM reserve slice 7 with 32MB (EIS recommendation) for metaDB/replicas.

• Ensure that the SAN Foundation Suite packages are installed and patched. (SAN is necessary for fibre server/storage)

change the home of root

#mkdir /root

#vi /etc/passwd

root:x:0:0:Super-User:/root:/sbin/sh

logout,login

Setup slvm

#cat /etc/lvm/md.tab

mddb1 -c 3 /dev/dsk/c3t0d0s7

d10 -m d11

d11 1 1 /dev/dsk/c3t0d0s0


d20 -m d21



d30 -m d31



Page 6 of 141


EIS-CD

Setup EIS environment

cd /cdrom/eis-dvd/sun/install

./setup-standard.sh

Install actual Solaris 10 patches

cd /cdrom/eis-dvd/patch/10

unpack-patches

cd /tmp/10

./install-patches

Analyse basic systemstatus

explorer

cd /opt/SUNWexplo/output/explorer*

/cdrom/eis-dvd/sun/tools/explominer/explominer

see the outputfile ExploMiner_EIS_Patchlist_nobody of the explominer and install all neccessary patches.

To enable MpxIO on Solaris 10 start

stmsboot -e before OracleCluster will be installed !!

For safety reasons clean the devices with:

devfsadm -C on all nodes.

Page 7 of 141


Setup IPMP groups on all nodes for all public network interfaces.

At least you need one test IP for each network adapter and one production IP for each IPMP group. e.g: IPMP group active-active with interfaces ce0 and ce4 with one physical IP.

ora-rac-1

Entry of /etc/hostname.ce0 :

ora-rac-1-1 netmask + broadcast + group ipmp0 up \

addif ora-rac-1 netmask + broadcast + deprecated -failover up


ora-rac-1-2 netmask + broadcast + group ipmp0 deprecated -failover up

ora-rac-2


ora-rac-2-1 netmask + broadcast + group ipmp0 up \addif ora-rac-3 netmask + broadcast + deprecated -failover up


ora-rac-2-2 netmask + broadcast + group ipmp0 deprecated -failover up

If the defaultrouter is NOT 100% available please read infodocs

70060 and 70062.

Notes:

• Do not use test IP for normal applications

• Test IP for all adapters in the same IPMP group must belong to a single IP subnet

• You need an addtional IP for each logical host.

• In the /etc/default/mpathd file, the value of

TRACK_INTERFACES_ONLY_WITH_GROUPS must be yes. (default)

Modify NVRAM parameter:

Set boot-device & diag-device to both sides of the mirror.

Suggested naming convention:

• Primary: rootdisk

• Secondary: rootmirror

Page 8 of 141


Setup Trunking for the Clusterinterconnect

Install Sun Trunking 1.3 for the ce interfaces.

pkgadd -d ./SUNWtrku

pkgadd -d ./SUNWtrkm

#vi /etc/opt/SUNWconn/bin/nettr.sh

Add at the end:

nttr -setup 1 device=ce members=1,3 policy=1

#reboot

Check the configuration:

#/etc/opt/SUNWconn/bin/nettr -conf

Name Head Policy DEV Type Original-Mac-Addr Speed Duplex Link

ce1 ce1 1 pci 8:0:20:8f:be:24 1000 Full Up

ce3 ce1 pci 8:0:20:8f:be:24 1000 Full Up

Last Systemanalyse

explorer

cd /opt/SUNWexplo/output/explorer*

/cdrom/eis-dvd/sun/tools/explominer/explominer

see the outputfile ExploMiner_EIS_Patchlist_nobody of the explominer and install all neccessary patches.

Page 9 of 141


Prepare Oracle Installation

Set kernel parameters

* Oracle settings *

set noexec_user_stack=1

set rlim_fd_cur=8192

* end of Oracle settings *

Create Users and Groups

>> ora-rac-1, ora-rac-2 as root

/usr/sbin/groupadd -g 61000 oinstall/usr/sbin/groupadd -g 61001 dba/usr/sbin/useradd -u 61000 -g oinstall -G dba oracle -s /usr/bin/bash -d /export/home/oracle

mkdir -p /export/home/oraclechown -R oracle:oinstall /export/home/oracle

passwd oracle> oracle> oracle

Share staging area

All Oracle software is located on Host ora-poc2:/oracle/install. The directory is shared as /install/oracle and can be mounted on every node if needed.

Share stage directory:

>> ora-poc2 as root

chown -R oracle:oinstall /oracleshare -rw /oracle/install

Mount staging area

In order to install oracle software, the staging area must be mounted on the instaation node of the clus-ter.

>> ora-rac-1 as root

mkdir /installmount ora-poc2:/oracle/install /install

Page 10 of 141


Modify /etc/hosts

127.0.0.1 localhost192.168.10.212 ora-rac-1 loghost ora-rac-1.192.168.10.234 ora-rac-1-1192.168.10.233 ora-rac-1-210.0.0.1 ora-rac-1-priv10.0.0.2 ora-rac-1-priv-110.0.0.3 ora-rac-1-priv-2192.168.10.225 ora-rac-1-vip

192.168.10.214 ora-rac-2 loghost ora-rac-2.192.168.10.232 ora-rac-2-1192.168.10.231 ora-rac-2-210.0.0.4 ora-rac-2-priv10.0.0.5 ora-rac-2-priv-110.0.0.6 ora-rac-2-priv-2192.168.10.229 ora-rac-2-vip

192.168.10.223 ora-rac-3-vip

192.168.10.239 ora-rac-4-vip

Check SSH daemon

ps -ef | grep sshd

Setup SSH for oracle

Note: This steps are different for any node in the cluster. Do complete the steps on the indicated nodes.

>> ora-rac-1, ora-rac-2 :

mkdir ~/.ssh

chmod 700 ~/.ssh

/usr/bin/ssh-keygen -t rsa/usr/bin/ssh-keygen -t dsa

>> ora-rac-1:

cat id_dsa.pub > authorized_keyscat id_rsa.pub >> authorized_keys

scp authorized_keys ora-rac-2:/export/home/oracle/.ssh/authorized_keys

>> ora-rac-2:

cat id_dsa.pub >> authorized_keyscat id_rsa.pub >> authorized_keys

>> ora-rac-1:

scp ora-rac-2:/export/home/oracle/.ssh/authorized_keys authorized_keys

>> ora-rac-1, ora-rac-2:

ssh ora-rac-1 datessh ora-rac-2 date

exec /usr/bin/ssh-agent $SHELL/usr/bin/ssh-add

Page 11 of 141


Setup userenv for oracle

>> ora-rac-1, ora-rac-2

Add line to ~/.profile

# Set Environment for oracle

. ~/set_oracle_env.sh

Perform cluster check

>> ora-rac-1

/install/10gR2/clusterware/runcluvfy.sh stage -pre crsinst -n ora-rac-1,ora-rac-2 -verbose

Workaround Bug

The OUI may fail if it does not find ssh and scp binaries in /usr/local/bin.


mkdir -p /usr/local/binln -s /usr/bin/ssh /usr/local/bin/sshln -s /usr/bin/scp /usr/local/bin/scp

Prepare storage

Create symbolic links for all disk devices needed for oracle


mkdir -p /dev/oracle

ln -s /dev/rdsk/c7t600015D00022B3000000000000005012d0s1 /dev/oracle/ocr1


ln -s /dev/rdsk/c7t600015D00022B300000000000000501Ed0s1 /dev/oracle/vote1

ln -s /dev/rdsk/c7t600015D00022B3000000000000005024d0s1 /dev/oracle/vote2

ln -s /dev/rdsk/c7t600015D00022B300000000000000502Ad0s1 /dev/oracle/vote3

ln -s /dev/rdsk/c7t600015D00022B3000000000000005042d0s1 /dev/oracle/asm01


ln -s /dev/rdsk/c7t600015D00022B300000000000000504Ed0s1 /dev/oracle/asm03


ln -s /dev/rdsk/c7t600015D00022B300000000000000505Ad0s1 /dev/oracle/asm05



ln -s /dev/rdsk/c7t600015D00022B300000000000000506Cd0s1 /dev/oracle/asm08



Change permissions for devices

chown oracle:dba /dev/oracle/*

Verify device order

Test if devices appear in same order on every node of the cluster.

Page 12 of 141


>> ora-rac-1

Create some textfiles for testing...

echo "This is orc1" > /tmp/ocr1echo "This is orc2" > /tmp/ocr2echo "This is vote1" > /tmp/vote1echo "This is vote2" > /tmp/vote2echo "This is vote3" > /tmp/vote3

Copy text to device

dd if=/tmp/ocr1 of=/dev/oracle/ocr1 dd if=/tmp/ocr2 of=/dev/oracle/ocr2 dd if=/tmp/vote1 of=/dev/oracle/vote1dd if=/tmp/vote2 of=/dev/oracle/vote2dd if=/tmp/vote3 of=/dev/oracle/vote3

>> ora-rac-2

Copy device contents to file on ervery node

dd if=/dev/oracle/ocr1 of=/tmp/ocr1 dd if=/dev/oracle/ocr2 of=/tmp/ocr2 dd if=/dev/oracle/vote1 of=/tmp/vote1dd if=/dev/oracle/vote2 of=/tmp/vote2dd if=/dev/oracle/vote3 of=/tmp/vote3

Check if content is the same as on the other node

cat /tmp/ocr1 cat /tmp/ocr2 cat /tmp/vote1cat /tmp/vote2cat /tmp/vote3

Trobleshooting device size

Count blocks

dd if=/dev/oracle/ocr1 of=/tmp/ocr1 bs=1k count=10

dd if=/dev/oracle/ocr2 of=/tmp/ocr2 bs=1k count=10

dd if=/dev/oracle/vote1 of=/tmp/vote1 bs=1k count=10



Label devices

Prepare devices for usage.

Template Device:

/dev/rdsk/c7t600015D00022B3000000000000005030d0s1

Label devices using template device:

prtvtoc /dev/rdsk/c7t600015D00022B3000000000000005030d0s2 | fmthard -s - /dev/rdsk/c7t600015D00022B3000000000000005012d0s2


prtvtoc /dev/rdsk/c7t600015D00022B3000000000000005030d0s2 | fmthard -s - /dev/rdsk/c7t600015D00022B300000000000000501Ed0s2


prtvtoc /dev/rdsk/c7t600015D00022B3000000000000005030d0s2 | fmthard -s - /dev/rdsk/c7t600015D00022B300000000000000502Ad0s2

Page 13 of 141


Install Clusterware 10.2.0.1

Run installer

>> ora-rac-1

/install/10gR2/clusterware/Disk1/runInstaller

Page 14 of 141


Page 15 of 141


Page 16 of 141


Page 17 of 141


Page 18 of 141


Page 19 of 141


Page 20 of 141


Page 21 of 141


Page 22 of 141


Trobleshooting Assistants

bash-3.00# ./root.sh

WARNING: directory '/oracle/product' is not owned by root

WARNING: directory '/oracle' is not owned by root

Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory

Setting up NS directories

Oracle Cluster Registry configuration upgraded successfully

WARNING: directory '/oracle/product' is not owned by root

WARNING: directory '/oracle' is not owned by root

clscfg: EXISTING configuration version 3 detected.

clscfg: version 3 is 10G Release 2.

Successfully accumulated necessary OCR keys.

Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.

node <nodenumber>: <nodename> <private interconnect name> <hostname>

node 1: ora-rac-1 ora-rac-1-priv ora-rac-1

node 2: ora-rac-2 ora-rac-2-priv ora-rac-2

clscfg: Arguments check out successfully.

NO KEYS WERE WRITTEN. Supply -force parameter to override.

-force is destructive and will destroy any previous cluster

configuration.

Oracle Cluster Registry for cluster has already been initialized

Startup will be queued to init within 30 seconds.

Adding daemons to inittab

root.shExpecting the CRS daemons to be up within 600 seconds.

CSS is active on these nodes.

ora-rac-1

ora-rac-2

CSS is active on all nodes.

Waiting for the Oracle CRSD and EVMD to start

Oracle CRS stack installed and running under init(1M)

Running vipca(silent) for configuring nodeapps

The given interface(s), "ce0, ce2" is not public. Public interfaces should be used to configure virtual IPs.

Page 23 of 141


/oracle/product/crs/bin/racgons add_config ora-rac-1:6200 ora-rac-2:6200

/oracle/product/crs/bin/oifcfg setif -global ce0/192.168.10.0:public ce1/10.0.0.0:cluster_interconnect ce2/192.168.10.0:public ce3/10.0.0.0:cluster_interconnect

/oracle/product/crs/bin/cluvfy stage -post crsinst -n ora-rac-1,ora-rac-2

Page 24 of 141


Check Cluster Status

>> ora-rac-1

ORACLE_HOME=/oracle/product/crs/bin

export ORACLE_HOME

PATH=$ORACLE_HOME/bin:$PATH

export PATH

$ crs_stat -t

Name Type Target State Host ----------------------------------------------------------- ora....c-1.gsd application OFFLINE OFFLINE ora....c-1.ons application ONLINE ONLINE ora-rac-1ora....c-1.vip application ONLINE ONLINE ora-rac-1 ora....c-2.gsd application OFFLINE OFFLINE ora....c-2.ons application ONLINE ONLINE ora-rac-2ora....c-2.vip application ONLINE ONLINE ora-rac-2

Page 25 of 141


Install crsstat.sh

Because of the unreadable output of „crs_stat -t“ setup a wrapper for crs_stat.

mkdir /oracle/tools

vi /oracle/tools/crsstat.sh

#!/usr/bin/ksh

#

# Sample 10g CRS resource status query script

#

# Description:

# - Returns formatted version of crs_stat -t, in tabular

# format, with the complete rsc names and filtering keywords

# - The argument, $RSC_KEY, is optional and if passed to the script, will

# limit the output to HA resources whose names match $RSC_KEY.

# Requirements:

# - $CRS_HOME should be set in your environment

RSC_KEY=$1

QSTAT=-u

AWK=/usr/xpg4/bin/awk

# Table header:echo ""

$AWK \

'BEGIN {printf "%-45s %-10s %-18s\n", "HA Resource", "Target", "State";

printf "%-45s %-10s %-18s\n", "-----------", "------", "-----";}'

# Table body:

$CRS_HOME/bin/crs_stat $QSTAT | $AWK \

'BEGIN { FS="="; state = 0; }

$1~/NAME/ && $2~/'$RSC_KEY'/ {appname = $2; state=1};

state == 0 {next;}

$1~/TARGET/ && state == 1 {apptarget = $2; state=2;}

$1~/STATE/ && state == 2 {appstate = $2; state=3;}

state == 3 {printf "%-45s %-10s %-18s\n", appname, apptarget, appstate; state=0;}'

Output of crsstat.sh

HA Resource Target State

----------- ------ -----

ora.ora-rac-1.gsd ONLINE ONLINE on ora-rac-1

ora.ora-rac-1.ons ONLINE ONLINE on ora-rac-1

ora.ora-rac-1.vip ONLINE ONLINE on ora-rac-1

ora.ora-rac-2.gsd ONLINE ONLINE on ora-rac-2

ora.ora-rac-2.ons ONLINE ONLINE on ora-rac-2

ora.ora-rac-3.vip ONLINE ONLINE on ora-rac-2

Page 26 of 141


Install ASM Home 10.2.0.1

Run installer

>> ora-rac-1

/install/10gR2/database/Disk1/runInstaller

Page 27 of 141


Ora_asmhome

Page 28 of 141


Page 29 of 141


Page 30 of 141


Page 31 of 141


Page 32 of 141


Page 33 of 141


Page 34 of 141


Create ASM Instance

Run DBCA

/oracle/product/asm/bin/dbca

Page 35 of 141


Page 36 of 141


Page 37 of 141


Page 38 of 141


Page 39 of 141


Page 40 of 141


Install Database Home 10.2.0.1

Run installer

>> ora-rac-1


Page 41 of 141


Page 42 of 141


Page 43 of 141


Page 44 of 141


Page 45 of 141


Page 46 of 141


Install Management Agent

Run installer

ora-rac-1


Page 47 of 141


Page 48 of 141


Page 49 of 141


Page 50 of 141


Page 51 of 141


Password: oracle1

Page 52 of 141


Page 53 of 141


Page 54 of 141


Create Database ZDA

Run DBCA

>> ora-rac-1

/oracle/product/db/bin/dbca

Page 55 of 141


Page 56 of 141


Page 57 of 141


Password: oracle

Page 58 of 141


Page 59 of 141


Page 60 of 141


Page 61 of 141


Page 62 of 141


Page 63 of 141


Page 64 of 141


Page 65 of 141


Page 66 of 141


Page 67 of 141


Page 68 of 141


Page 69 of 141


Create login script for user oracle

The script is located at ~/ set_oracle_env.sh

---------------This script starts here ------------

#!/usr/bin/bashclear

# Menu section. If user does not choose any oracle home, no actions takes place.

echo "Choose your Oracle Home and hit RETURN"echo "Enter 1 to use Oracle Home DB"echo "Enter 2 to use Oracle Home ASM"echo "Enter 3 to use Oracle Home Agent"echo "Enter 4 to use Oracle Home CRS"echo "Hit any other key to skip Oracle Home selection"

read answer

if [[ $answer = 1 ]] then ORACLE_HOME=/oracle/product/db ; ORACLE_SID=ZDA1elif [[ $answer = 2 ]] then ORACLE_HOME=/oracle/product/asm ; ORACLE_SID="+ASM1" ;elif [[ $answer = 3 ]] then ORACLE_HOME=/oracle/product/agent/agent10g ;elif [[ $answer = 4 ]] then ORACLE_HOME=/oracle/product/crs ;else returnfi

# If the user choose an oracle home, the following actions are performed:

ORACLE_BASE=/oracleexport ORACLE_BASE

export PATH=$ORACLE_HOME/bin:$PATHexport ORACLE_SIDexport ORACLE_HOME

EDITOR=viexport EDITOR

if [ -t 0 ]; then stty intr ^Cfi

# Alias section. Some usefull shortcuts ...

alias crsi='/oracle/product/crs/bin/crs_stat -t'alias xxx=". ~/set_oracle_env.sh"alias oh='cd $ORACLE_HOME'

echo "Tip: Enter xxx to change your current setting"

---------------The script ends here ------------

Page 70 of 141


Install Patchset 10.2.0.2 on CRS Home

Stop all oracle processes

As oracle:

srvctl stop database -d ZDA

srvctl stop asm -n ora-rac-1

srvctl stop asm -n ora-rac-2

srvctl stop nodeapps -n ora-rac-1

srvctl stop nodeapps -n ora-rac-2

As root:

/etc/init.d/init.crs stop

Install binaries

./runInstaller

Page 71 of 141


Page 72 of 141


Page 73 of 141


Page 74 of 141


Restart Custerware

>> ora-rac-1 root:

/etc/init.d/init.crs start

Page 75 of 141


Install Patchset 10.2.0.2 on ASM Home

Stop Nodeapps

>> ora-rac-1:

srvctl stop nodeapps -n ora-rac-1srvctl stop nodeapps -n ora-rac-2

Install binaries

./runInstaller

Page 76 of 141


Page 77 of 141


Page 78 of 141


Page 79 of 141


Page 80 of 141


Install Patchset 10.2.0.2 on DB Home

Verify service state

All Services must be down.

crs_stat -t

Name Type Target State Host------------------------------------------------------

ora....A1.inst application OFFLINE OFFLINEora....A2.inst application OFFLINE OFFLINEora.ZDA.db application OFFLINE OFFLINEora....SM1.asm application OFFLINE OFFLINEora....-1.lsnr application OFFLINE OFFLINEora....c-1.gsd application OFFLINE OFFLINEora....c-1.ons application OFFLINE OFFLINEora....c-1.vip application OFFLINE OFFLINEora....SM2.asm application OFFLINE OFFLINEora....-2.lsnr application OFFLINE OFFLINEora....c-2.gsd application OFFLINE OFFLINEora....c-2.ons application OFFLINE OFFLINEora....c-2.vip application OFFLINE OFFLINE

Install binaries

./runInstaller

Page 81 of 141


Page 82 of 141


Page 83 of 141


Page 84 of 141


Page 85 of 141


Page 86 of 141


Install Interim Patch 5117016 on DB Home

Install with opatch

>> ora-rac-3 as user oracle

#cd /install/10gR2/5117016

#opatch apply -local

Page 87 of 141


>> ora-rac-4 as user oracle

#cd /install/10gR2/5117016

#opatch apply -local

Page 88 of 141


Change /etc/init.d/init.crsd:

>> ora-rac-3 and ora-rac-4 as user root

from: exec $ORA_CRS_HOME/bin/crsd $BOOTFLAG $*

to : exec newtask $ORA_CRS_HOME/bin/crsd $BOOTFLAG $*

Setup system parameters on both Nodes:

>> ora-rac-3 and ora-rac-4 as user root

#projadd -p 61002 -c "Oracle RAC" -U oracle \-K 'project.max-sem-ids=(priv,1024,deny)' \-K 'process.max-sem-nsems=(priv,2048,deny)' \-K 'project.max-shm-ids=(priv,256,deny)' \-K 'project.max-shm-memory=(priv,18446744073709551615,deny)' \MAH

cat >> /etc/user_attroracle::::project=RAC<CTRL>-d

Check all by reboot

>> ora-rac-3 as user root

#scshutdown -g0 -y


ok boot


ok boot

Start services required to perform database upgrade

Restart the listener:

srvctl start listener -n ora-rac-1srvctl start listener -n ora-rac-2

Start asm:

srvctl start asm -n ora-rac-1srvctl start asm -n ora-rac-2

Upgrade database ZDA

sqlplus / as sysdbaSTARTUP NOMOUNTALTER SYSTEM SET CLUSTER_DATABASE=FALSE SCOPE=spfile;shutdown

STARTUP UPGRADESPOOL patch_10202.log@?/rdbms/admin/catupgrd.sqlSPOOL OFF

Check log for errors:

grep ORA- patch_10202.log

ORA-00600: internal error code, arguments: [qkacon:NFswrwo], [1], [], [], [],

This is probably the bug 5089217, fixed in version 11.1 and should be fixed in version 10.2.0.3.

Workaround bug 5089217:

Page 89 of 141


Set "_optimizer_connect_by_cost_based" = false

shutdown

startup@?/rdbms/admin/utlrp.sql

select distinct status from dba_objects;=> valid

ALTER SYSTEM SET CLUSTER_DATABASE=TRUE SCOPE=spfile;ALTER SYSTEM SET "_optimizer_connect_by_cost_based" = false SCOPE=spfile;

shutdown;startup;

Restart cluster

srvctl stop database -d ZDAsrvctl stop asm -n ora-rac-1srvctl stop asm -n ora-rac-2srvctl stop nodeapps -n ora-rac-1srvctl stop nodeapps -n ora-rac-2

srvctl start nodeapps -n ora-rac-1srvctl start nodeapps -n ora-rac-2srvctl start asm -n ora-rac-1srvctl start asm -n ora-rac-2srvctl start database -d ZDA

crs_stat -t

How to Setup IPMP as Cluster Interconnect Doc ID: Note:368464.1

In order to setup IPMP as cluster interconnect note 283107.1 should be used.

Page 90 of 141

https://metalink.oracle.com/help/usaeng/Search/search.html#file


Add new node to cluster

Create links to devices


mkdir -p /dev/oracle



ln -s /dev/rdsk/c9t600015D00022B300000000000000501Ed0s1 /dev/oracle/vote1

ln -s /dev/rdsk/c9t600015D00022B3000000000000005024d0s1 /dev/oracle/vote2

ln -s /dev/rdsk/c9t600015D00022B300000000000000502Ad0s1 /dev/oracle/vote3



ln -s /dev/rdsk/c9t600015D00022B300000000000000504Ed0s1 /dev/oracle/asm03


ln -s /dev/rdsk/c9t600015D00022B300000000000000505Ad0s1 /dev/oracle/asm05



ln -s /dev/rdsk/c9t600015D00022B300000000000000506Cd0s1 /dev/oracle/asm08



Change permissions


chown oracle:dba /dev/oracle/*

Page 91 of 141


Modify /etc/hosts

Add IP-Adresses.

>> ora-rac-1, ora-rac-2, ora-rac-4 as root

127.0.0.1 localhost192.168.10.212 ora-rac-1 loghost ora-rac-1.192.168.10.234 ora-rac-1-1192.168.10.233 ora-rac-1-210.0.0.1 ora-rac-1-priv10.0.0.2 ora-rac-1-priv-110.0.0.3 ora-rac-1-priv-2192.168.10.225 ora-rac-1-vip


192.168.10.223 ora-rac-3-vip


Check SSH daemon

>> ora-rac-4

ps -ef | grep sshd

Enable user equivalence

>> ora-rac-4

mkdir /export/home/oracle/.ssh

chmod 700 ~/.ssh

/usr/bin/ssh-keygen -t rsa/usr/bin/ssh-keygen -t dsa


cat id_dsa.pub >> authorized_keyscat id_rsa.pub >> authorized_keys



Page 92 of 141


Perform cluster check

/install/10gR2/clusterware/Disk1/cluvfy/runcluvfy.sh stage -pre crsinst -n ora-rac-1,ora-rac-2,ora-rac-4 -verbose

Workaround OUI Bug

>> ora-rac-4

The OUI may fail if it does not find ssh and scp binaries in /usr/local/bin.


mkdir -p /usr/local/bin

ln -s /usr/bin/ssh /usr/local/bin/sshln -s /usr/bin/scp /usr/local/bin/scp

Install clusterware on new node

>> ora-rac-1

/oracle/product/crs/oui/bin/addNode.sh

Page 93 of 141


<Enter node name>

Page 94 of 141


Page 95 of 141


Page 96 of 141


/oracle/product/crs/bin/crs_stat -t

Name Type Target State Host

------------------------------------------------------------

ora....DA1.srv application ONLINE ONLINE ora-rac-1

ora....atch.cs application ONLINE ONLINE ora-rac-1

ora....A1.inst application ONLINE ONLINE ora-rac-1

ora....A2.inst application ONLINE ONLINE ora-rac-2

ora.ZDA.db application ONLINE ONLINE ora-rac-2

ora....SM1.asm application ONLINE ONLINE ora-rac-1

ora....-1.lsnr application ONLINE ONLINE ora-rac-1

ora....c-1.gsd application ONLINE ONLINE ora-rac-1

ora....c-1.ons application ONLINE ONLINE ora-rac-1

ora....c-1.vip application ONLINE ONLINE ora-rac-1

ora....SM2.asm application ONLINE ONLINE ora-rac-2

ora....-2.lsnr application ONLINE ONLINE ora-rac-2







Install ASM home on new node

/oracle/product/asm/oui/bin/addNode.sh

Page 97 of 141


Page 98 of 141


Page 99 of 141


Page 100 of 141


Install DB home on new node

/oracle/product/db/oui/bin/addNode.sh

Page 101 of 141


Page 102 of 141


Page 103 of 141


Page 104 of 141


Create listener in DB Home

/oracle/product/db/bin/netca

Page 105 of 141


Fehler! Hier nur Node ora-rac-4 auswählen!

Page 106 of 141


LISTENER

LISTENER

Page 107 of 141


LISTENER

Page 108 of 141


Page 109 of 141


Create ASM instance on new node

>> ora-rac-1

/oracle/product/asm/bin/dbca

Page 110 of 141


Page 111 of 141


password: oracle

Page 112 of 141


Create database instance on new node

>> ora-rac-1


Page 113 of 141


Page 114 of 141


password sys / oracle

Page 115 of 141


Page 116 of 141


Page 117 of 141


Verify that group# is not already used (i.e. by standby redolos. In this case, alter group# to an unused value.

Page 118 of 141


End of Installation

Due to time constraints, the installtion was not completed. To be continued.

Page 119 of 141


Remove Node from ClusterSteps to perform a remove node from cluster following Note 317516.1

Remove instance

>> ora-rac-1


Page 120 of 141


Page 121 of 141


Page 122 of 141


Page 123 of 141


Remove Listener

>> ora-rac-1

/oracle/product/db/bin/dbca/netca

Page 124 of 141


Page 125 of 141


Page 126 of 141


Page 127 of 141


Clean-up the ASM

>> ora-rac-1

srvctl stop asm -n ora-rac-4srvctl remove asm -n ora-rac-4

>> ora-rac-4

rm -rf $ORACLE_HOME/admin/+ASMrm -f $ORACLE_HOME/dbs/*ASM*

Edit the /etc/oratab file and remove entries beginning with +ASM.

Removing node from the database

>> ora-rac-4

/oracle/product/db/oui/bin/runInstaller -updateNodeList ORACLE_HOME= /oracle/product/crs "CLUSTER_NODES={ora-rac-4}" -local

>> ora-rac-4

/oracle/product/crs/oui/bin/runInstaller #=> Choose de-install products and select the dbhome

Screenshots here!

>> ora-rac-1

/oracle/product/crs/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/oracle/product/crs "CLUSTER_NODES={node1,node2}"

Removing node from the clusterware:


Page 128 of 141


Get Remote Port number:

cat /oracle/product/crs/opmn/conf/ons.config

/oracle/product/crs/bin/racgons remove_config ora-rac-4:6200


/oracle/product/crs/install/rootdelete.sh


/oracle/product/crs/bin/olsnodes -n

/oracle/product/crs/install/rootdeletenode.sh ora-rac-4,3

>> ora-rac-4

/oracle/product/crs/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/oracle/product/crs/ "CLUSTER_NODES={ora-rac-4}" CRS=TRUE -local

/oracle/product/crs/oui/bin/runInstaller #=> Choose Deinstall software and re-move the CRSHOME

>> ora-rac-1

/oracle/product/crs/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/oracle/product/crs "CLUSTER_NODES={ora-rac-1,ora-rac-2}" CRS=TRUE

/oracle/product/crs/bin/olsnodes -n

Page 129 of 141


Cluster Performance TuningOverview and best practices for setting up a "zero brownout" configuration to achieve maximum high availability. More info in note 369101.1.

Note: Patch 5024639 is not yet available and skipped therefore.

Apply Brownout Patches

This patches are installed by using OPatch on DB and ASM Home:

5186679 (This became patch 4960705)

5186674 (This became patch 5016142)

5127434

5127472

5127482

5176453

Set the CSS misscount to match the service level

1. Shut down the clusterware stack on all nodes but one

2. On the remaining node, run as root:

$ORA_CRS_HOME/bin/crsctl set css misscount 5

$ORA_CRS_HOME/bin/crsctl set css disktimeout 200

3. Reboot the nodes for this to take affect.

4. Confirm the new values for the CSS parameters

$ORA_CRS_HOME/bin/crsctl get css misscount

$ORA_CRS_HOME/bin/crsctl get css disktimeout

Run CSS in Real Time

1. On each node as root use, crsctl set css priority 4

2. Restart each node in series, using crsctl stop crs followed by crsctl start crs

Set the check_interval for VIP addresses

>> ora-ra-1 as root

1. Generate the resource profile for the VIP resource

cd $ORA_CRS_HOME/crs/profile

crs_stat -p ora.ora-rac-1.vip > ora. ora-rac-1.vip.capcrs_stat -p ora.ora-rac-2.vip > ora. ora-rac-2.vip.cap

(current value of check_interval = 60 (default))

2. Edit ora.<hostname>.vip.cap and set the parameter CHECK_INTERVAL to 5

Page 130 of 141


vi ora.ora-rac-1.vip.capvi ora.ora-rac-2.vip.cap

3. Update the resource profile as the root user:

crs_register -u ora.ora-rac-1.vipcrs_register -u ora.ora-rac-2.vip

Note: The check interval should be long enough to sustain the refresh time for the VIP.

Remove the dependency between the VIP and ASM

1. Generate the resource profile for the ASM instance resource

cd $ORA_CRS_HOME/crs/profile

crs_stat -p ora.<hostname>.<ASM instance #>.asm > ora.<hostname>.<ASM instance #>.asm.cap

2. Edit ora.<hostname>.<ASM instance #>.asm.cap and remove 'ora.<hostname>.vip as the value of the REQUIRED_RESOURCES

3. Change the parameter REQUIRED_RESOURCES

vi ora.<hostname>.<ASM instance #>.asm.cap

REQUIRED_RESOURCES=

4. Update the resource profile

crs_register -u ora.<hostname>.asm

Note: There is a hard dependency from the ASM to the VIP. This means that when the virtual IP is relocated, ASM is shut down. It is fully supported to remove this CRS dependency, and is the default from 10.2.0.3 and later releases.

Reduce the ping timeout

1. Open the $CRS_HOME/bin/racgvip script.

(current value of PING_TIMEOUT = 3 (default))

2. Edit to set PING_TIMEOUT to:

PING_TIMEOUT="-w 1 -c 1"

This change shortens the delay between begin of CRS recovery and start of the VIP.

Note: If any patches are applied that change the racgvip script, this variable would need to be changed again.

alter system set disk_asynch_io=true scope=spfile ;alter system set fast_start_parallel_rollback=high scope=spfile ;alter system set "_imr_disk_voting_interval"=0 scope=spfile;alter system set "_first_spare_parameter"=0 scope=spfile ;alter system set "_imr_splitbrain_res_wait"=20 scope=spfile ;alter system set "_db_mttr_advice"=off scope=spfile ;

Page 131 of 141


Troubleshooting VIP Failover

Adjusting Listener Configuration

Make the local IP-address the first listening entry in the listener.ora. More infos in note 3589757 and 339107.1

>> All nodes

Example on node ora-rac-1. Before:

LISTENER_ORA-RAC-1 = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = ora-rac-1-vip) (PORT = 1521) (IP = FIRST)) (ADDRESS = (PROTOCOL = TCP) (HOST = 192.168.10.212) (PORT = 1521) (IP = FIRST)) ) )

[...]

After:

LISTENER_ORA-RAC-1 = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = 192.168.10.212) (PORT = 1521) (IP = FIRST)) (ADDRESS = (PROTOCOL = TCP) (HOST = ora-rac-1-vip) (PORT = 1521) (IP = FIRST)) ) )

[...]

Upgrade Oracle Home of Swingbench

Upgrade Oracle Home of Swingbench to Rel. 10.2.0.

Adjust SQLNet settings

Edit sqlnet.ora of client. Add / change the following line

SQLNET.OUTBOUND_CONNECT_TIMEOUT=10

Set the ping timeout back to default

1. Open the $CRS_HOME/bin/racgvip script.

(current value of PING_TIMEOUT = 3 (default))

2. Edit to set PING_TIMEOUT to:

PING_TIMEOUT="3"

Page 132 of 141


Addendum

Metalink Note 284752.1

Page 133 of 141



Page 134 of 141


Page 135 of 141


Page 136 of 141


Page 137 of 141


Page 138 of 141


Page 139 of 141



Page 140 of 141


Page 141 of 141

Documents

Installation SUN 10gR2 RAC ASM