Building Oracle Grid with Oracle

VM on Blade Servers and iSCSI

Storage

Kai Yu

Dell Oracle Solutions Engineering

• Kai Yu

– 15 years with Oracle technology, specialized in

Oracle Grid/RAC, Oracle Apps and Oracle VM

– Oracle ACE Director, Oracle papers author and

frequent presenter at OOW and Collaborate

– IOUG Oracle RAC SIG president

• Dell Oracle Solutions Engineering Lab

– Dell’s Oracle Technology solutions based on Dell

PE servers and Dell storages.

– The solutions stack: servers, storage, network,

OS, virtualization, Oracle RAC, Oracle Apps

About Author

2

Introduction to Oracle Grid

Grid Reference Configuration POC Project

Grid Implementation

Consolidate Applications on Grid

QA

3

Agenda

• Enterprise Grid Computing– Group independent hardware and software components in to a

single resource pool

– Can be provisioned, distributed and redistributed on demand

– Standardization, Virtualization, Automation

• Benefit of Grid Computing– Quickly meet the changing of business needs

– Dynamically allocate the resources for changing workloads

– Resource consolidation and sharing reduce the IT cost

– Providing Predictable service levels

• Key Technologies for Oracle Grid– Sever clustering: Real Applications Cluster (RAC)

– Server Virtualization: Oracle VM

– Storage Virtualization: Automatic Storage Management (ASM)

– Management and Provisioning: Oracle Enterprise Manager4

Introduction to Oracle Grid

5

Introduction to Oracle Grid

• Oracle Real Application Clusters– Multiple instances running on many hosts interconnected by

high speed private network share a single database

– Oracle Cache Fusion Technology for node synchronization

– All Instances can execute transactions simultaneously on the

single database in the shared storage

– Provide High Availability for the applications

– Scalability, special useful for commodity hardware.

• Automatic Storage Management (ASM)

– An integrated file system and volume manager

– Spread data evenly on disks; automatically rebalance

– Easy management of file system

– Provide shared storage for RAC nodes

6

Introduction to Oracle Grid

• Oracle VM Technology

– Virtual Server: Xen Hypervisor, management domain dom0

Linux kernel with support of devices, IO, networking, etc.

– Virtual Server Pool: an autonomous region of VM servers

– VM: guest OS with applications running on domU

7

Introduction to Oracle Grid

8

Introduction to Oracle Grid

• Enterprise Grid vs Cloud Computing– Dynamic shared infrastructure as the basis of Cloud computing

– Provide the key building blocks for cloud computing

– Help IT build private cloud and cloud-like infrastructure

• Engineering solution on Grid Computing – Provide customers a solution that builds cloud like services

• Complete stack of software and hardware platform

• Provide building, deploying and managing solutions

• Ensure customers the enterprise grade configuration

– Engineering Solution through the Oracle Grid Proof of Concept

project

9

Introduction to Oracle Grid

• Grid Reference POC Project – Engineering solution for Grid Architecture and implementation

Design/implement an enterprise grade Grid infrastructure

Dynamically provision multi-tier applications in a single Grid

Based on commodity hardware and SAN storage

• Grid Architecture– Physical Grid on clustering: 8 + nodes Oracle 11g RAC

– Virtual Grid: a set of virtual machines running on Oracle VM

servers based on Dell Blades

– Grid control as the unified management

• Grid Hardware Platform– Commodity Hardware: Dell 11g Blade Servers:

– Shared SAN storage: Dell EqualLogic iSCSI storage1

0

Grid Architecture Design

Physical View Logical View

Grid Architecture Design

Grid Architecture Design

Grid Architecture Design

Grid Implementation

1

4

• Implementation Tasks– Configure Grid Control Management Infrastructure

– Configure EqualLogic Shared storage

– Configure Physical Grid

– Configure Virtual Grid

• Grid Control Management Configuration– Oracle Enterprise Linux OEL 4.7 64 bit

– Oracle Enterprise Manager Grid control 10.2.0.3

– Grid control 10.2.0.5 upgrade

– Enable Virtual Management Pack Restart OMS server

Grid Implementation

Blade servers connect to EqualLogic SAN Storage

Grid Implementation

1

6

• EqualLogic shared storage configuration– Storage volumes for the Physical Grid

– Storage volumes for the Virtual Grid

Volume Size Raid Used for OS Mapping

blade_crs 2GB 10 OCR/Votingdisk Yes

blade_data1 100GB 10 Data for DB1 ASM diskgroup1

blade_data2 100GB 10 Data for DB2 ASM diskgroup2

blade_data3 150GB 10 Data for DB3 ASM diskgroup3

blade_data5 150GB 10 Data for DB4 ASM diskgroup5

Volume Size Raid Used for OS Mapping

blade_data4 400GB 10 VM Repositories /OVS

blade_data6 500GB 10 VM Repositories

/OVS/9A87460A7EDE4

3EE92201B8B7989DBA

6

vmcor1-5 1GB each 10 OCR/vogtingdisk 2XOCRs/3Xvotingdisks

vmracdb1 50GB 10 Data for RAC DB ASM diskgroup1

vmracdb2 50GB 10 Data for RAC DB ASM diskgroup2

Grid Implementation

1

7

EqualLogic storage management console

Grid Implementation

1

8

• Physical Grid Configuration : 8-node 11g RAC

– Network configuration

– iSCSI storage connections

• Configure host access to storage: eth2 and eth3, /etc/iscsi/iscsid.config

• iSCSI interfaces: i-eth2 on eth2, i-eth3 on eth3:

iscsiadm -m iface -I i-eth2 --op=new

iscsiadm -m iface -I i-eth2 --op=update -n iface.hwaddress -v <hwadd>

iscsiadm -m discovery -t st -p 10.16.7.100 --interface=i-eth2 --interface=i-eth3

iscsiadm -m node -p 10.16.7.100 --interface i-eth3 --login

iscsiadm -m node -p 10.16.7.100 --interface i-eth2 –login

Check Partitions : more /proc/partitions:

Network Interface IO modules Connections IP address

eth0 A1 Public Network 155.16.9.71-78

eth2 B1 iSCSI connection 10.16.7.241-255(odd)

eth3 B2 iSCSI connection 10.16.7.240-254(odd)

eth4,eth5 C1,C2 Bonded to bond0 192.168.9.71-78

VIP Virtual IP for 11g RAC 155.16.9.171-178

Grid Implementation

1

9

• Find iSCSI ID: /sbin/scsi_id -gus /block/sdd

• Linux Device Mapper to establish mutipath devices to storage alias:

/etc/multipath.conf .

multipaths {

multipath {

wwid 36090a0287065124c45a2941c7fc76340

alias ocr_crsp14

}

…….

}

service multipathd restart

– Check multipath devices in /dev/mapper:

ocr-crsp1, data1, data2, data3, data5

Make partitions: fdisk

Grid Implementation

2

0

– Use block devices for 11g Clusterware and RAC

databases: OCRs, Votingdisks and ASM diskgroups.

– Configure 11g RAC database Infrastructure

• 11g RAC clusterware configuration:

private interconnect : bond0

OCRs and votingdisks on mutipath devices

• 11g ASM configuration:

ASM instances provide the storage virtualization

• 11g RAC software to provide the database services

• Grid control agent 10.2.0.5 to connect to OMS server

Grid Implementation

2

1

– Consolidate Databases

• Sizing the DB: Storage, # of DB instances, which nodes to run

• Provision the storage/ASM diskgroup

• Create the database

• For some ERP applications, convert the installed DB to RAC

Grid Implementation

2

2

– Scale out the physical Grid Infrastructure• Add more databases to the Grid

• Increase capability of the existing databases

• Scale out the storage:

– add additional EqualLogic array

– expand existing or create new volumes

– make new volume accessible to the servers

– add new ASM diskgroup or add disk to the existing diskgroup or resize the disk of an diskgroup

• Add new node to the cluster

• Dynamically move the database to less busy node:

Grid Implementation

2

3

• Implementation Tasks Overview

– Virtual servers Installation

– Virtual server network and storage configuration.

– Connect VM servers to the Grid Control

– Create guest VM using VM template

– Configure the VMs using VM Management Pack with Oracle Enterprise Manager

Virtual Server Installation

– Prepare local disk and enable virtualization on BIOS

– Install Oracle VM server

– Change Dom0 memory : /boot/grub/menu.lst:

edit line: kernel /xen-64bit.gz dom0_mem=1024M

– Ensure VM agent working: #service ovs-agent status

Grid Implementation

2

4

• Oracle VM server network configuration

Grid Implementation

2

5

– Configure shared storages on dom0 • OVM repositories : blade_data4: 400GB,

• Shared disks for Oracle RAC on VMs

• Configure iSCSI and multipath devices on dom0

• /dev/mapper/ovs_data4p1

• Put OVM repository on blade_data4p1 (OVM 2.1)

mkfs.ocfs2 -b 4k -C 64k -L ovs /dev/mapper/ovs_data4p1

unmount /OVS, moun new OVS: mount -a -t ocfs2

edit /etc/fstab to include the new /OVS partiotion

create directories : iso_pool, proxy, running_pool,

seed_pool, sharedDisk under /OVS

Grid Implementation

2

6

Put OVM repository on blade_data4p1 (OVM 2.2)

service iptables stop; service o2cb configure,

mkfs.ocfs2 -Tdatafiles -N8 /dev/mapper/p1ovsdisk1p1

remove the local /OVS: /opt/ovs-agent-2.3/utils/repos.py -l

[ * ] 9e7012d5-69ba-4ac2-a58d-3400cd044efc => /dev/sda3

/opt/ovs-agent-2.3/utils/repos.py -d 9e7012d5-69ba-4ac2-a58d-3400cd044efc

create new repository on shared devices;

/opt/ovs-agent-2.3/utils/repos.py -n /dev/mapper/p1ovsdisk1p1[ NEW ] 567d581b-ec4a-4da8-94df-841dc0c5a390 => /dev/mapper/p1ovsdisk1p1

create the newly created repository the cluster root repository:

/opt/ovs-agent-2.3/utils/repos.py -r 567d581b-ec4a-4da8-94df-841dc0c5a390[ R ] 567d581b-ec4a-4da8-94df-841dc0c5a390 => /dev/mapper/p1ovsdisk1p1

Reboot all the VM servers

Grid Implementation

2

7

– VM Management tools:

• Oracle VM Manager

• Enterprise Manager: VM Management Pack

– Connect the VM servers to Enterprise Manager

Pre-requisites Steps: oracle user in oinstall group, oracle sshuser equivalence between dome0 and OMS server, Oracle user sudo privileges : /etc/sudoers

Grid Implementation

2

8

• Create VM server Pool: login to Grid control

Grid Implementation

2

9

• Add additional virtual servers to the VM server pool

Grid Implementation

3

0

• Create Guest VMs using VM template

– Virtual Machine as node of Virtual Grid

– Create guest VM using VM templates

– Import a VM template: download template; Discover VM

template

Grid Implementation

3

1

– Create VM using the template

Grid Implementation

3

2

• Prepare storage for VMs: virtual disks:From the Images files in repository:

• Local disk:

• Shared disk

• Disks as resources for VM:

Grid Implementation

3

3

• Add local disk to the VM

• Attach shared disk to the VM

• s

Grid Implementation

3

4

• Disks shown in VM as the virtual disk partitions:

• Attached the storage to the guest VM a) vm.cfg: disk = [''file:/OVS/sharedDisk/racdb.img,xvdc,w!', b) vm.cfg: disk = ['‘phy: /dev/mapper/vmracdbp1, xvdc,w!',

• Virtual network interface through Xen bridgesvm.cfg:vif = ['bridge=xenbr0,mac=00:16:3E:11:8E:CE,type=netfront','bridge=xenbr1,mac=00:16:3E:50:63:25,type=netfront',]xenbr0 eth0 in the guest VMxenbr1 eth1 in the guest VM

Grid Implementation

3

5

• Storage/Network configuration for 11g RAC on VMs:

3

6

Consolidate Applications

• Consolidate enterprise applications on the

Grid– Applications and middleware on the virtual Grid

• VMs Applications Application VM templates->New VMs

• Application template VMs Modification New application

template

• Import: existing VM: Copy VM image Import VM image

– Deploy database service on the physical Grid

• Provisiong Storage Attach storage to NodesASM

diskgroupDatabase

create Schema->connect to Database

– Deploy DEV/Test Application suite on the virtual Grid

• Multi-tier nodes are on the VMs

• Fast deployment based on templates

3

7

Consolidate Applications

• Grid Architecture to host applications

Oracle VM

Apps

1

Virtual Grid

Grid Structure and Applications

EqualLogic SAN

storage

Apps

2

ebsap

p1

Big_5

2vm2

nvmr

ac4

ebs2a

pp2

ebs2d

b1

11g RAC

Bann

1

Bann

2Banner RAC

DB

EBS R12 Test

Suite on VMs

Physical Grid

EBS RAC DB

EBS R12 Apps

Tier on VMs

11g RAC Databases Grid

Other Database

Other Database

VM server Pool

3

8

Consolidate Applications

• Oracle E-Business Suite Applications:

Consolidate Applications

3

9

• Virtual Grid shown on Grid Control Virtual Central

4

0

• Dell Grid POC: Oracle Grid on Dell Blades and EqualLogic

• Oracle Enterprise Manager Grid control for management

• Provides the infrastructure to consolidate the enterprise

applications and databases

• Acknowledgement: David Mar, Roger Lopez (Dell Oracle

Solutions Engineering), John Tao(Dell IT), Rajat

Nigam(Oracle)

• Related Dell Oracle Whitepapers:1) Building an Oracle® Grid with Oracle VM on Dell ™ Blade servers

and Dell EqualLogic iSCSI storage

2) Implementing Oracle® E-Business Suite in a Scalable and Reliable

Virtual System Architecture

3) Efficient Deployment of Oracle® Real Application Clusters (RAC) in a

Virtualized Environment Using Enterprise Manager Provisioning Pack

SUMMARY

References

4

1

1. Best Practices of Deployment of Oracle® ASM with Dell EqualLogic PS iSCSI Storage System, Dell White Paper

2 . Oracle Press Release: Oracle® Enterprise Manager Extends Management to Oracle VM Server Virtualization

3. Oracle® Enterprise Manager Concepts 10g Release 5 (10.2.0.5)Part Number B31949-10

4. Oracle Enterprise Manager Grid Control ReadMe for Linux x86-6410g Release 5 (10.2.0.5) April 2009

5. How to Enable Oracle VM Management Pack in EM Grid Control 10.2.0.5, Metalink Note #: 781879.1

6. Oracle VM: Converting from a Local to Shared OVS Repository, Metalink note # 756968.1

7. How to Add Shared Repositories to Oracle VM Pools with Multiple Servers, Metalink Note # 869430.1

8. Implementing Oracle Grid: A successful Customer Case Study, Kai Yu, Dan Brint, IOUG Select Journal Volume 16, Number 2, Second Quarter 2009

9. Deploying Oracle VM Release 2.1 on Dell PowewrEdge servers and Dell/EMC storage Dell white paper

10. Dell Reference Configuration Deploying Oracle® Database on Dell™ EqualLogic™ PS5000XV iSCSI Storage A Dell Technical White Paper

11. Technical Best Practices for Virtualization & RAC – Oracle RAC SIG webseminar slides, Michael Timpanaro-Perota & Daniel Dibbets

12. Kai Yu’s Oracle Blog

QA

4

2

4

3

Thank You

• Please Complete the Survey

(presentation #705, Building Oracle Grid

with Oracle VM on blade servers and iSCSI

storage, Presenter: Kai Yu)

• Visit Kai’s Oracle Blog:

http://kyuoracleblog.wordpress.com