Luci, Ricci and the RAC

...or Clustering on Centos 5 using Conga for an Oracle RAC install

Drawn up by Fauz Ghauri (@fauzg) for barcamp

Introduction

• Hardware

• Installation

• Custom udev rules for iscsi

• Cluster

• Heartbeat and fencing

• RAC Other things to consider for RAC.

• Acknowledgements

Hardware

• Network Cards

– DRAC (Dell Remote Access Card)

– ILO (Integrated Lights Out – HP kit)

– VMware Fence (Built into ESX – maybe server?)

– Etc.

• SAN – Are you using iSCSI?

• VMware – which version of ESX/server do you run?

Installation

• Mainly Defaults used for centos 5 install

• A few extra packages are required outside of the base install.

Installation

Installation

Installation

Installation

Installation

Installation

Installation

Installation – Conga Components

Installation – Conga Components

Installation – Luci

Installation – Luci

Custom udev rules for iSCSI

• If you’re using iSCSI – you’ll need to make sure that your drives are mapped.

# iscsiadm -m discovery -t sendtargets –p <sanIP.mydomain (or maybe ip address)>192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm1 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm2 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm3 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm4 192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.crs

Custom udev rules for iSCSI

• Manually log onto targets

– # iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.asm1 -p 192.168.2.195 -l

• Configure auto login

– # iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.asm1 -p 192.168.2.195 --op update -n node.startup -v automatic

Custom udev rules for iSCSI

• # (cd /dev/disk/by-path; ls -l *sanname* | awk '{FS=" "; print $9 " " $10 " " $11}')

• Returns:– ip-192.168.2.195:3260-iscsi-iqn.2006-

01.com.openfiler:racdb.asm1-lun-0 -> ../../sda– Etc.– Make a note!– scsi_id –g –u –s /block/sdX – find out which letters map to

which iSCSI device (where X is a, b, c, etc)– Note iSCSI strings.– Create a new rules file in /etc/udev/rules.d/22-

randomnumberandname.rules

Custom udev rules for iSCSI

• ACTION=="add", KERNEL=="sd*", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT==“<iscsi string goes here>", RUN+="/bin/raw /dev/raw/raw1 %N“

• ACTION=="add", KERNEL=="raw1", OWNER=="root", GROUP=="dba", MODE=="0660", SYMLINK+="oracle_ocr"

Cluster

Cluster

Cluster

Cluster

Cluster

Cluster

Cluster

Cluster

This turns yellow asthere’s an issue.

Cluster

Cluster

Cluster

Cluster

Heartbeat & Fencing

• Heartbeat – saying “hi! I’m here” to the cluster.

• Fencing – fencing off a dodgy node

• Fencing on ESX means the creation of a user for fencing.

– useradd and groupadd – insufficient permissions.

– Get it working on ESX before trying to fence remotely.

– Adding permissions can be done by doing the following:

Heartbeat & Fencing

– vmware-vim-cmd vimsvc/auth/role_add groupXVirtualMachine.Interact.PowerOn

– vmware-vim-cmdvimsvc/auth/entity_permission_addvim.Folder:ha-folder-root userX users groupXtrue

• Test it out with the command fence_vmware(syntax in wiki in ackowledgements section)

Things to consider with RAC

• Don’t reboot all nodes at the same time –unless you feel like losing your entire database!

• Bear in mind that the /udev rules should map to some raw disks if you’re using ASM.

• SSH key equivalency (sharing keys).

Acknowledgements

• Thanks to:– [redacted] for getting me going– [redacted] for helping me figure out that nasty little issue

with GSSAPI timing out my SSH connections

• Sources:– http://www.oracle.com/technology/pub/articles/hunter_r

ac10gr2_iscsi.html– http://www.redhat.com/docs/en-

US/Red_Hat_Enterprise_Linux/4.8/html/Cluster_Administration/s1-start-luci-ricci-conga-CA.html

– http://sources.redhat.com/cluster/wiki/VMware_FencingConfig

– http://www.vroem.co.za/?p=7

Bonus Slides!

Bonus Slides!

Bonus Slides!

Bonus Slides!

Bonus Slides!