This article will provide information on moving OCR-Voting devices to ASM diskgroups in newly upgraded 11.2.0.3.0 Grid infrastructure home from 10.2.0.4 CRS. Please check my earlier blog for upgrading 10.2.0.4 CRS to 11.2.0.3

As we can see from the below output that ASM resource is added to CRS resources but not online because ASM instances and DG resources are not created yet.[root@rac1 bin]# ./crsctl stat res ora.asm -t--------------------------------------------------------------------------------NAME TARGET STATE SERVER STATE_DETAILS --------------------------------------------------------------------------------Local Resources--------------------------------------------------------------------------------ora.asm OFFLINE OFFLINE rac1 OFFLINE OFFLINE rac2 OCR/VOTING disks are stored in Block devices as if now.[root@rac1 bin]# ./crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- --------- 1. ONLINE 31752502fce75fb6bfcbbe041949734c (/dev/raw/raw2) []Located 1 voting disk(s).[root@rac1 bin]# cat /etc/oracle/ocr.lococrconfig_loc=/dev/raw/raw1local_only=FALSEWe can see 6 DISKS for ASM however we'll create diskgroup DATADG1 using two disks only and later on we'll multiplex voting disks into different diskgroup DATADG2 using remaining four disks.[root@rac1 bin]# oracleasm listdisksDISK1DISK2DISK3DISK4DISK5DISK6Please make sure that same disks are available on both nodes before proceeding for running of asmca.[root@rac2 bin]# oracleasm listdisksDISK1DISK2DISK3DISK4DISK5DISK6

We can verify CRS version as 11.2.0.3 from below commands.[oracle@rac1 ~]$ crsctl query crs softwareversion

Oracle Clusterware version on node [rac1] is [11.2.0.3.0][oracle@rac1 ~]$ crsctl query crs activeversionOracle Clusterware active version on the cluster is [11.2.0.3.0]

Now we'll run asmca as oracle user from 11.2.0.3 home. We'll get first screen as shown below. Please enter sys and asmsnmp password for ASM instances and click on specify disk group button.

Once you click on specify disk group button, it'll show you below screen so please choose External in redundancy category, Specify diskgroup name as DATADG1 and select ORCL:DISK1 and ORCL:DISK2.

Once it's done then click ok and it'll come to first page where DISKGROUP will be mentioned as DATADG1 as shown below and now click on Create ASM button.

Once it'll create ASM instances and Diskgroup, it'll display message like below.

Once you click on ok in above screen, it'll display DG information as shown below so review it and exit from ASMCA.

Adding OCR into ASM diskgroup================================[root@rac1 bin]# ls -tlr /dev/rawtotal 0crw-rw-r-- 1 oracle oinstall 162, 2 Sep 6 10:29 raw2 -> used for Votingcrw-rw---- 1 root oinstall 162, 1 Sep 6 11:08 raw1 ->used for OCR

[root@rac2 bin]# ./ocrconfig -add +DATADG1[root@rac2 bin]# cat /etc/oracle/ocr.loc#Device/file getting replaced by device +DATADG1 ocrconfig_loc=/dev/raw/raw1ocrmirrorconfig_loc=+DATADG1local_only=false[root@rac2 bin]# ./ocrcheckStatus of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 1039908 Used space (kbytes) : 4328 Available space (kbytes) : 1035580 ID : 133883669 Device/File Name : /dev/raw/raw1 Device/File integrity check succeeded Device/File Name : +DATADG1 Device/File integrity check succeeded Device/File not configured Device/File not configured

Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded

Deleting OCR raw device and adding back

============================================

[root@rac2 bin]# ./ocrconfig -delete /dev/raw/raw1[root@rac2 bin]# cat /etc/oracle/ocr.loc#Device/file /dev/raw/raw1 getting replaced by device +DATADG1 ocrconfig_loc=+DATADG1local_only=false[root@rac2 bin]# ./ocrcheckStatus of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 1039908 Used space (kbytes) : 4328 Available space (kbytes) : 1035580 ID : 133883669 Device/File Name : +DATADG1 Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded

Now we'll add it back OCR block device and this time it'll be added as mirror device.

[root@rac2 bin]# ./ocrconfig -add /dev/raw/raw1[root@rac2 bin]# cat /etc/oracle/ocr.loc#Device/file getting replaced by device /dev/raw/raw1 ocrconfig_loc=+DATADG1ocrmirrorconfig_loc=/dev/raw/raw1local_only=false[root@rac2 bin]# ./ocrcheckStatus of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 1039908 Used space (kbytes) : 4328 Available space (kbytes) : 1035580 ID : 133883669 Device/File Name : +DATADG1 Device/File integrity check succeeded

Device/File Name : /dev/raw/raw1 Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded

Moving/Replacing Voting Devices into ASM DG

[root@rac2 bin]# ./crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- --------- 1. ONLINE 31752502fce75fb6bfcbbe041949734c (/dev/raw/raw2) []Located 1 voting disk(s).[root@rac2 bin]# ./crsctl add css votedisk +DATADG1 CRS-4671: This command is not supported for ASM diskgroups.CRS-4000: Command Add failed, or completed with errors.[root@rac2 bin]# ./crsctl replace votedisk +DATADG1CRS-4256: Updating the profileSuccessful addition of voting disk 0fd588d39f784fb8bf7d9fd3b7558dd3.Successful deletion of voting disk 31752502fce75fb6bfcbbe041949734c.Successfully replaced voting disk group with +DATADG1.CRS-4256: Updating the profileCRS-4266: Voting file(s) successfully replaced[root@rac2 bin]# ./crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- --------- 1. ONLINE 0fd588d39f784fb8bf7d9fd3b7558dd3 (ORCL:DISK1) [DATADG1]Located 1 voting disk(s).

Please note that we couldn't add voting devices to ASM while having voting device into raw devices as well. We could only replace existing voting devices to ASM DG using crsctl replace command as shown in above output.

Moving Voting files back to raw device===================================[root@rac2 bin]# ./crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- --------- 1. ONLINE 0fd588d39f784fb8bf7d9fd3b7558dd3 (ORCL:DISK1) [DATADG1]Located 1 voting disk(s).[root@rac2 bin]# ./crsctl replace votedisk /dev/raw/raw2

Now formatting voting disk: /dev/raw/raw2.CRS-4256: Updating the profileSuccessful addition of voting disk 56c5891e74094fbbbf9a62bc50ce872a.Successful deletion of voting disk 0fd588d39f784fb8bf7d9fd3b7558dd3.CRS-4256: Updating the profileCRS-4266: Voting file(s) successfully replaced[root@rac2 bin]# ./crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- --------- 1. ONLINE 56c5891e74094fbbbf9a62bc50ce872a (/dev/raw/raw2) []Located 1 voting disk(s).

So from above exercise, we can conclude that we can OCR devices into ASM DG and block devices at the same time as mirrored devices however for voting devices, we can either into Block devices or ASM DG not both at the same time.

Example of replacingVoting devices into from one ASM diskgroup to another.

Now we'll create second ASM diskgroup DATADG2 using remaining disks with normal redundancy. [oracle@rac1 ~]$ asmcmdASMCMD> lsdgState Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files NameMOUNTED EXTERN N 512 4096 1048576 10228 9071 0 9071 0 Y DATADG1/MOUNTED NORMAL N 512 4096 1048576 20456 20169 5114 7527 0 N DATADG2/Since Voting disk is already located in DATADG1 and we want to move it to DATADG2 while CRS is up, we get below error.[root@rac1 bin]# ./crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- --------- 1. ONLINE 43887170ea864f1cbf3c4008f21fc78a (ORCL:DISK1) [DATADG1]Located 1 voting disk(s).[root@rac1 bin]# ./crsctl add css votedisk +DATADG2CRS-4671: This command is not supported for ASM diskgroups.CRS-4000: Command Add failed, or completed with errors.[root@rac1 bin]# ./crsctl replace votedisk +DATADG2CRS-4602: Failed 27 to add voting file 6ac9fe19e3444fcfbf827a035155bd18.CRS-4602: Failed 27 to add voting file a4ffb5d52ad14f7ebf563726fbe15761.CRS-4602: Failed 27 to add voting file 92b962c4090c4f7abfd3184f721796cc.Failure 27 with Cluster Synchronization Services while deleting voting disk.

Failed to replace voting disk group with +DATADG2.CRS-4000: Command Replace failed, or completed with errors.[root@rac1 bin]# ./crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- --------- 1. ONLINE 43887170ea864f1cbf3c4008f21fc78a (ORCL:DISK1) [DATADG1]Located 1 voting disk(s).

Please stop CRS on both nodes using crsctl stop crs and then start CRS in exclusive mode on any of one node.[root@rac1 bin]# ./crsctl start crs -exclCRS-4123: Oracle High Availability Services has been started.CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeededCRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeededCRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeededCRS-2676: Start of 'ora.gipcd' on 'rac1' succeededCRS-2672: Attempting to start 'ora.cssd' on 'rac1'CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'CRS-2676: Start of 'ora.diskmon' on 'rac1' succeededCRS-2676: Start of 'ora.cssd' on 'rac1' succeededCRS-2672: Attempting to start 'ora.ctssd' on 'rac1'CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rac1'CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'rac1'CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'rac1' succeededCRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac1'CRS-2676: Start of 'ora.ctssd' on 'rac1' succeededCRS-2676: Start of 'ora.drivers.acfs' on 'rac1' succeededCRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac1' succeededCRS-2672: Attempting to start 'ora.asm' on 'rac1'CRS-2676: Start of 'ora.asm' on 'rac1' succeededCRS-2672: Attempting to start 'ora.crsd' on 'rac1'CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded[root@rac1 bin]# ps -ef |grep -i d.binroot 8102 1 2 10:55 ? 00:00:04 /u01/app/11.2.0/grid/bin/ohasd.bin exclusiveoracle 8223 1 1 10:55 ? 00:00:02 /u01/app/11.2.0/grid/bin/oraagent.binoracle 8235 1 0 10:55 ? 00:00:00 /u01/app/11.2.0/grid/bin/mdnsd.binoracle 8258 1 1 10:55 ? 00:00:01 /u01/app/11.2.0/grid/bin/gpnpd.binroot 8268 1 0 10:55 ? 00:00:00 /u01/app/11.2.0/grid/bin/cssdmonitororacle 8270 1 1 10:55 ? 00:00:02 /u01/app/11.2.0/grid/bin/gipcd.binroot 8290 1 0 10:55 ? 00:00:01 /u01/app/11.2.0/grid/bin/cssdagentoracle 8314 1 2 10:55 ? 00:00:03 /u01/app/11.2.0/grid/bin/ocssd.bin -X

root 8367 1 1 10:56 ? 00:00:02 /u01/app/11.2.0/grid/bin/orarootagent.binroot 8381 1 0 10:56 ? 00:00:00 /u01/app/11.2.0/grid/bin/octssd.binroot 8675 1 1 10:56 ? 00:00:01 /u01/app/11.2.0/grid/bin/crsd.bin rebootroot 8840 6677 0 10:58 pts/2 00:00:00 grep -i d.bin

[root@rac1 bin]# ./crsctl replace votedisk +DATADG2Successful addition of voting disk ae7a31fc941d4fa6bf813b58cbf4ae6d.Successful addition of voting disk 9a7cd52bf7e24fe4bfac1939a2433a2a.Successful addition of voting disk 75459ba3e9134f0abfa54468ac51610c.Successful deletion of voting disk 43887170ea864f1cbf3c4008f21fc78a.Successfully replaced voting disk group with +DATADG2.CRS-4266: Voting file(s) successfully replaced[root@rac1 bin]# ./crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- --------- 1. ONLINE ae7a31fc941d4fa6bf813b58cbf4ae6d (ORCL:DISK3) [DATADG2] 2. ONLINE 9a7cd52bf7e24fe4bfac1939a2433a2a (ORCL:DISK4) [DATADG2] 3. ONLINE 75459ba3e9134f0abfa54468ac51610c (ORCL:DISK5) [DATADG2]Located 3 voting disk(s).

From above output, we can see that now VOTING disk is moved into three disks out of 4 ASM disks since redundancy of ASM diskgroup is normal. now stop crs and start on both nodes. We've to use force option to shutdown CRS.[root@rac1 bin]# ./crsctl stop crsCRS-2796: The command may not proceed when Cluster Ready Services is not runningCRS-4687: Shutdown command has completed with errors.CRS-4000: Command Stop failed, or completed with errors.[root@rac1 bin]# ps -ef |grep -i d.binroot 8102 1 1 10:55 ? 00:00:06 /u01/app/11.2.0/grid/bin/ohasd.bin exclusiveoracle 8223 1 1 10:55 ? 00:00:06 /u01/app/11.2.0/grid/bin/oraagent.binoracle 8235 1 0 10:55 ? 00:00:00 /u01/app/11.2.0/grid/bin/mdnsd.binoracle 8258 1 0 10:55 ? 00:00:02 /u01/app/11.2.0/grid/bin/gpnpd.binroot 8268 1 0 10:55 ? 00:00:01 /u01/app/11.2.0/grid/bin/cssdmonitororacle 8270 1 0 10:55 ? 00:00:03 /u01/app/11.2.0/grid/bin/gipcd.binroot 8290 1 0 10:55 ? 00:00:01 /u01/app/11.2.0/grid/bin/cssdagentoracle 8314 1 1 10:55 ? 00:00:07 /u01/app/11.2.0/grid/bin/ocssd.bin -Xroot 8367 1 1 10:56 ? 00:00:05 /u01/app/11.2.0/grid/bin/orarootagent.binroot 8381 1 0 10:56 ? 00:00:01 /u01/app/11.2.0/grid/bin/octssd.binroot 8675 1 0 10:56 ? 00:00:01 /u01/app/11.2.0/grid/bin/crsd.bin rebootroot 9331 6677 0 11:03 pts/2 00:00:00 grep -i d.bin [root@rac1 bin]# ./crsctl stop crs -fCRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded

CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'CRS-2673: Attempting to stop 'ora.asm' on 'rac1'CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac1'CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeededCRS-2677: Stop of 'ora.ctssd' on 'rac1' succeededCRS-2677: Stop of 'ora.asm' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.cssd' on 'rac1'CRS-2677: Stop of 'ora.cssd' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'CRS-2677: Stop of 'ora.drivers.acfs' on 'rac1' succeededCRS-2677: Stop of 'ora.gipcd' on 'rac1' succeededCRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeededCRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completedCRS-4133: Oracle High Availability Services has been stopped.[root@rac1 bin]# ps -ef |grep -i d.binroot 9500 6677 0 11:04 pts/2 00:00:00 grep -i d.bin

[root@rac1 bin]# ./crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- --------- 1. ONLINE 44b55c5576154fc2bf3e54f08099a0e9 (ORCL:DISK3) [DATADG2] 2. ONLINE 0fcd8b592b604faabf9120ffef2a8e18 (ORCL:DISK5) [DATADG2] 3. ONLINE 00dc165a6d5c4ffebf4ba43496477d13 (ORCL:DISK6) [DATADG2]Located 3 voting disk(s).[root@rac1 bin]# ./crsctl check crsCRS-4638: Oracle High Availability Services is onlineCRS-4537: Cluster Ready Services is onlineCRS-4529: Cluster Synchronization Services is onlineCRS-4533: Event Manager is online

[root@rac2 bin]# ./crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- --------- 1. ONLINE 44b55c5576154fc2bf3e54f08099a0e9 (ORCL:DISK3) [DATADG2] 2. ONLINE 0fcd8b592b604faabf9120ffef2a8e18 (ORCL:DISK5) [DATADG2] 3. ONLINE 00dc165a6d5c4ffebf4ba43496477d13 (ORCL:DISK6) [DATADG2]Located 3 voting disk(s).[root@rac2 bin]# ./crsctl check crsCRS-4638: Oracle High Availability Services is onlineCRS-4537: Cluster Ready Services is onlineCRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

We've to perform same steps for OCR replacement like stopping CRS and starting in exclusive mode. Since we added/removed copy of OCR, we need not to performs stop/start(exclusive) operations.