Oracle ASM Reduces Cost of VLDB Deployment Hanan Hit, Principal Database Architect Lina Shabelsky, Senior Application and Database Engineer NOCOUG Winter

Oracle ASM Reduces Cost of VLDB Deployment

Hanan Hit, Principal Database Architect

Lina Shabelsky, Senior Application and Database Engineer

NOCOUG Winter Conference February 08, 2007

Confidential and Proprietary

Agenda

• Application description• What ASM is and is not• Protocol etc.• Deployment models• Network design • Physical architecture• Oracle operations to reduce required I/O• Index creation stats • Best practices• Lessons learned during VLDB implementation• Backup and recovery

Application Description


Logical Database Structure

Management Db

Operational Data Store

Historical Data Store

ArchiveData Store (SATA)

OLAP


General Loading Schema

Staging ODS –Last Hour

Data Warehouse

OLAP

Bus

Loader

Aggregators


High Level – Physical Layer

Storage Private Network

CISCOSYSTEMS 2G IP Switch

NFS

/Ora

AS

M

SkyRider Inc.RAC – Primary / Primary Configuration

With Management DB & ARCH DB

Cache Fusion Private Network

CISCOSYSTEMS 2G IP Switch

Public Network 1GB

FAS 3050 FAS 3050

AMD Dual CPU8GB Machine

AMD Dual CPU8GB Machine

AMD Dual CPU 16GB RAM

PRODDB1


PRODDB2


ARCHDB

Cache Fusion Private Network

What ASM is and is not


ASM - Flexible, Efficient, Time Saver

• Vertically integrated file system • Easy file management system • Management of:

-Raw device volume-Oracle data files-Online and archive logs-RMAN backup

• Multi platform availability• Stripes data across all raw volumes• Hot spot detection and correction• Optional mirroring (best with SAN/NAS) but striping is NOT• Oracle 10g Release 1 and above


What ASM is NOT

• A cluster file system• Available for non Oracle files• Available for OCR and voting disks

• General file system


Options prior to ASM

• RawWith LVM Without LVM

• Cooked With LVMWithout LVM


Why We Chose ASM in the First Place?

• New implementation using 10g • Linux RHEL shop• Mid range storage array - full enterprise business needs• “Just trust the hardware to handle it”

Not a workable solution (DBA’s)• Expected thousands of data files

Didn’t want to use the BIG File option (YET).• RAC & non RAC implementations• Obvious need for storage growth with unpredictable limit size• Do more with less

Very few DBAs & sys/storage admin• Block level access to storage• Easy storage provisioning

Protocol


Which Protocol to Use?

• FC – SAN (2/4 GBit/s)• iSCSI – IP-SAN (2/4 GBit/s)• NFS – NAS (2/4 GBit/s)

• FCIP


What is iSCSI Protocol?

• A network protocol standard that allows the use of the SCSI protocol over TCP/IP networks

• A transport layer protocol in the SCSI-3 specifications framework

• Expected to capture more than 10% of storage systems revenue and an even greater percentage of capacity by 2008 - Hot technologies for 2007 (Storage Magazine)

• “iSCSI SAN is definitely happening”- Stephen Foskett, GlassHouse Technologies Inc., Framingham, MA


Wikipedia iSCSI Definitions

• iSCSI initiator in client/server terminology, is akin to a client device that connects to some service offered by the server (in this case an iSCSI target).

• An iSCSI target is akin to a server, in that it provides block level access to its storage media (usually a hard drive, but can be other types of SCSI devices).

• Only one iSCSI initiator can talk to a given iSCSI target at a time (one-to-one).


Host Storage System

Initiator Target

Simple iSCSI system

Deployment Models


Deployment Models

• Separate Disk Groups Use storage base features for ASM deployment data management Backup/recovery, cloning etc.

• Shared Disk Groups Use Oracle tools exclusively for all data management methods


Separate Disk Groups


Shared Disk Groups


Shared Disk Group- Multiple RAC Instances

Network Design


Typical Performance in MB/s

System Component Throughput /Performance

Bits Bytes

16 Port Switch8 * 2GBit/s 1200 MB/s

Fibre Channel 2 GBit/s 200 MB/s

Disk Controller 2 GBit/s 180 MB/s

GigE NIC 2 GBit/s 80 MB/s

1 Gbit HBA 1 GBit/s 100 MB/s

2 Gbit HBA 2 GBit/s 200 MB/s


iSCSI End Point Options


Network Configuration

PROPERLY SHUT DOWN SYSTEM BEFORE OPENING CHASSIS.

PCI 1 PCI 3

PCI 4PCI 2

DISCONNECT AC POWER

CORD BEFORE REMOVAL

IO

AC

status

DISCONNECT AC POWER

CORD BEFORE REMOVAL

IO

AC

status

Console

Console

0a 0b

0a 0b

FC

FC

e0a

e0a e0b

e0b RLM

RLM

e0c

e0c e0d

e0d

0d

0d0c

0c

FC

FC

0e

0e

LVD SCSI

LVD SCSI

L

IN

K

L

IN

K

L

IN

K

L

IN

K

L

IN

K

L

IN

K

L

IN

K

L

IN

K

NetworkAppliance FAS3020


PCI 1 PCI 3

PCI 4PCI 2

DISCONNECT AC POWER

CORD BEFORE REMOVAL

IO

AC

status

DISCONNECT AC POWER

CORD BEFORE REMOVAL

IO

AC

status

Console

Console

0a 0b

0a 0b

FC

FC

e0a

e0a e0b

e0b RLM

RLM

e0c

e0c e0d

e0d

0d

0d0c

0c

FC

FC

0e

0e

LVD SCSI

LVD SCSI

L

IN

K

L

IN

K

L

IN

K

L

IN

K

L

IN

K

L

IN

K

L

IN

K

L

IN

K



PCI 1 PCI 3

PCI 4PCI 2

DISCONNECT AC POWER

CORD BEFORE REMOVAL

IO

AC

status

DISCONNECT AC POWER

CORD BEFORE REMOVAL

IO

AC

status

Console

Console

0a 0b

0a 0b

FC

FC

e0a

e0a e0b

e0b RLM

RLM

e0c

e0c e0d

e0d

0d

0d0c

0c

FC

FC

0e

0e

LVD SCSI

LVD SCSI

L

IN

K

L

IN

K

L

IN

K

L

IN

K

L

IN

K

L

IN

K

L

IN

K

L

IN

K


CONSOLE

Catalyst 2948G-L3

PSI RPSU

1 3 5 7 9 1311 15 211917 23 2725 29 31 39373533

AUX

21

65

87

109

1211

1413

1615

1817

2019

2221

2423

2625

2827

3029

3231

3433

3635

3837

4039

4241

41 43 45 47

4443

4645

4847

43

10/100/100 ETHERNET

LAYER 3 SWITCH

STATUS

49

50

5049

1000 Base - XCONSOLE

Catalyst 2948G-L3

PSI RPSU

1 3 5 7 9 1311 15 211917 23 2725 29 31 39373533

AUX

21

65

87

109

1211

1413

1615

1817

2019

2221

2423

2625

2827

3029

3231

3433

3635

3837

4039

4241

41 43 45 47

4443

4645

4847

43

10/100/100 ETHERNET

LAYER 3 SWITCH

STATUS

49

50

5049

1000 Base - X

MLS-A MLS-B

RLM

==== Ethernet Configuration ====Ethernet Ports: santana e0a à e0b à e0c à e0d à hendrix e0a à e0b à e0c à e0d à doors e0a à e0b à e0c à e0d à VLAN: 200Jumbo Frame 9216 Bytes

doorshendrixsantana

IP Configuration ====Domain Name: crightcorp.com (or skyrider.com)Default VRRP Router: 172.16.10.1

IP Addresses: MLS-A: 172.16.10.1 MLS-B: 172.16.10.2 santana V3: 172.16.10.16 hendrix V3: 172.16.10.17 doors V3: 172.16.10.18

==== General Info ====

Remote Power Strip (rps1) santana: Bus A: Plug xx Bus B: Port xx hendrix: Bus A: Plug xx Bus B: Plug xx doors: Bus A: Plug xx Bus B: Plug xx

Terminal Server (ts1: 9600/N/1) Port x: santana Port y: hendrix Port z: doors

VIF V1(Standby)

Static 2 Link LAG

VIF V3VIF V2(Active)

Active VRRP Router for VLAN 200

Backup VRRP Router for VLAN 200

9600/N/1

CONSOLE

Catalyst 2948G-L3

PSI RPSU

1 3 5 7 9 1311 15 211917 23 2725 29 31 39373533

AUX

21

65

87

109

1211

1413

1615

1817

2019

2221

2423

2625

2827

3029

3231

3433

3635

3837

4039

4241

41 43 45 47

4443

4645

4847

43

10/100/100 ETHERNET

LAYER 3 SWITCH

STATUS

49

50

5049

1000 Base - X

Management Switch

VIF V1(Standby)

VIF V3VIF V2(Active)

RLM RLMPwr

Bus-BPwr

Bus-A

Physical Architecture


FAS3050 #1

SMRAC2SMRAC1

FAS3050 #1

ASM (VOLMAINDATA)DISKGROUP

“DATA1”

LUN #1DATA #1 100GB

LUN #7TEMP #1 100GB

LUN #3UNDO # 1

200GB

LUN #2DATA # 2

100GB

LUN #5DATA #3 100GB

LUN #6DATA #4 100GB

LUN #4TEMP #2

100GB

iSCSI

iSCSI

MAIN RAC WITH ASM – DATABASE LAYOUT

VOLOCR

MAINOCR 2GB

MGMOCR 2GB

MAINVOTING 2GB

MAINVOTING 2GB

NFS

NFS


FAS3050 #2

SMRAC2SMRAC1

HENDRIX

ASM (VOLMAINDATA)DISKGROUP

“DATA2”

LUN #1INDEX #1

100GB

LUN #3UNDO # 1

200GB

LUN #2INDEX # 2

100GB

LUN #5INDEX #3

100GB

LUN #4TEMP #2

100GB

VOLMAINRECOVERY

LUN #1REDO #1

60GB

LUN #4REDO #4

60GB

LUN #3REDO # 3

60GB

LUN #2REDO # 2

60GB

LUN #5CONTROLD

20GB

iSCSIiSCSI

MAIN & MGM RAC WITH ASM – DATABASE LAYOUT

iSCSI

iSCSI

VOLDATADUMPS

UNC / NFS 600GB

NFS

NFS


FAS3020

SMARCHDB

DOORS

ASM (VOLARCHDBATA)DISKGROUP

“DATA1”

LUN #1DATA #1 512GB

LUN #3UNDO # 1

400GB

LUN #2INDEX #1

512GB

LUN #5TEMP

#2 300GB

VOLARCHDBRECOVERY

LUN #1REDO #1

60GB

LUN #4REDO

#4 60GB

LUN #3REDO #

3 60GB

LUN #2REDO # 2

60GB

LUN #5CONTRO

LD20GB

iSCSI

iSCSI

ARCHDB WITH ASM – DATABASE LAYOUT

VOLARCHDBARCHLOGS

512GB

NFS

LUN #4DATA # 2

512GB

LUN #6INDEX #

2512GB

VOLMISC

512GBORACLE_HOME

ASM_HOMEADMIN

NFS

SMRAC2SMRAC1

VOLMAINARCHLOGS

512GB

NFSNFS

NFS

NFS

ASM (VOLARCHDBATA)DISKGROUP

“DATA2”

LUN #1DATA #1 512GB

LUN #3UNDO # 1

400GB

LUN #2INDEX

#1 512GB

LUN #4DATA # 2

512GB

LUN #5INDEX #

2512GB

iSCSI

ASM (VOLARCHDB

ATA)DISKGROUP

“DATA3”

LUN #1DATA #1 400GB

LUN #2DATA #1

400B

LUN #3DATA # 2

300GB


Database size

Oracle operations to reduce required I/O


Oracle Operations - Reduce Disk I/O

• Index creation on many billions of rows New software version while supporting legacy data

• Gathering Oracle stats

Index creation statistics


Index Creation – New App Version

• Set/unset the SKIP_UNUSABLE_INDEXES When set to TRUE Oracle will not attempt to use or report errors when an index is marked as unusable

• System Level - ALTER SYSTEM SET SKIP_UNUSABLE_INDEXES = TRUE;

• Session Level ALTER SESSION SET SKIP_UNUSABLE_INDEXES = TRUE;


Index Creation – New App Version

• Step 1 - Create the new index with the UNUSABLE attribute• Step 2 - System Level - ALTER SYSTEM SET

SKIP_UNUSABLE_INDEXES = TRUE;• Step 3 - ALTER INDEX <index name> MODIFY PARTITION <partition

name> UNUSABLE; On all partitions. If using a Global Index then mark the entire index as UNUSABLE

• Step 4 - ALTER INDEX <index name> REBUILD PARTITION <new partition name> NOLOGGING;

On the most fresh partition this will enable the application to run with the new index ASAP.

• Step 5 - ALTER INDEX <index name> REBUILD PARTITION <partition name> NOLOGGING;

On all the legacy (non hotspot) partitions. This will be rebuild according the system workload.

• Final Step - System Level - ALTER SYSTEM SET SKIP_UNUSABLE_INDEXES = FALSE;


Gathering Oracle Stats

• Gather partitions statistics (First time) Exec dbms_stats.gather_table_stats(ownname=>‘<schema

name>', tabname => ‘<tab name>', PARTNAME=> ‘<part name>' , estimate_percent => DBMS_STATS.AUTO_SAMPLE_SIZE, cascade=>TRUE, method_opt=>'FOR ALL COLUMNS SIZE AUTO');

Exec dbms_stats.gather_table_stats (ownname=>‘<schema name>' , tabname=>‘<tab name>' , PARTNAME=> '<part name>' , estimate_percent=>1,CASCADE=> true);

• Unlock Statistics (If previously locked)

Exec dbms_stats.unlock_table_stats(OWNNAME=>‘<schema name>' , TABNAME=>‘<tab name>');


Gathering Oracle Stats – More

• Copy statistics Execdbms_stats.COPY_TABLE_STATS(OWNNAME=>‘<schema

name>' , TABNAME=>‘<tab name> ' , SRCPARTNAME=>‘<N’th partition name>' , DSTPARTNAME=>'<N’th+1 partition name>');

• Lock table statistics Exec dbms_stats.lock_table_stats(OWNNAME=>‘<schema name>' ,

TABNAME=>‘<tab name>');


Index Creation Example


Index Creation Example – More

Best Practices


NetApp/ASM – Our Best Practices

• Single aggregate • FlexVol • ASM external redundancy • Separate disk groups• Maximum volume size allowed 16TB while

recommended is not more then 3TB• Use RAID-DP with maximum 16 drives in a single

RAID group• Set minra to off – even on DSS • Jumbo frames – 9K MTU• Single mode VIF

Lessons learned during VLDB implementation


RAC 10gR2 Implementation Learnings

• Implementation of VLDB in RHEL 4.3, software iSCSI, NetApp and ASM environment

• Procedure of provisioning additional storage • Backup procedures with NetApp SnapManager for

Oracle


Network Implementation

• Isolate your database from the rest of the IP traffic• Use separate VLAN for access to the storage• Use oifcfg to verify the setup of your interfaces• Use IP bonding on both – server and filer to achieve

network stability and as an alternative solution to multipathing

• IP bonding allows you to aggregate multiple network interfaces into higher performance network link and provides failover solution


Oracle Clusterware

• Carefully plan your file system layout • If you use ASMLIB with iSCSI disks, don’t map OCR

and voting disk to raw devices in RAC environment – use OCFS or NFS

• Raw devices are not aware of ASMLIB • In RedHat 4 update 3, the use of

/etc/sysconfig/rawdevices is deprecated


Storage

• When planning for the large database consult your storage vendor about RAID configuration, storage provisioning and backup procedures.

• As an example – using NetApp as storage appliance allows DBAs to take advantage of RAID-DP (Double Parity) – fault tolerance of RAID 1 at the price of RAID 4.

• NetApp flexible architecture allows DBAs to provision additional storage with minimal downtime.

• SnapManager for Oracle allows DBAs to combine NetApp snapshot technology with Oracle RMAN to dramatically decrease time taken to backup and restore database.


Storage Provisioning with ASM

• Adding 1TB of storage takes no more than

30 minutes with Netapp• Step1 – Ask your system administrator to

plug in Netapp shelves• Step 2 – login to the filer and add new disks

to the aggregate leaving 2 spares per array• Step 3 – create new volume and LUNs, add

them to the initiator group


Filer view


Adding Storage to the Database

• Step 4 - After new LUNs created on the filer, reboot the database server to automatically discover new LUNS. You can verify that LUNs are accessible from the database server using NetApp host utilities:


Creating ASM Disks

• Step 5 – run fdisk on new devices. • Step 6 – run /etc/init.d/oracleasm createdisk.• Last step – connect to ASM instance and execute

“create diskgroup…” command. You can also use Enterprise Manager db control:

Backup and Recovery


SnapManager for Oracle Backups

• A management tool with a GUI and command-line interface for Oracle Database administrators that simplifies backup, recovery, and cloning for Oracle Databases

• Designed to leverage Oracle Database 10g data management and grid features such as ASM, RAC, and RMAN

• Utilizes NetApp Snapshot technology to create extremely fast and space-efficient backups

Snapshot copies are point-in-time copies of a database that are created nearly instantaneously. These backups can also be registered with Oracle RMAN, which facilitates the use of RMAN to restore and recover the database at finer granularities such as blocks.


How SnapManager Works

• To create a successful backup, SnapManager performs the following operations:

Determines the list of data files, control files, and log files that make up the database. These files can be located on one or more ASM disk groups.

Puts the database in hot backup mode Creates an atomic Snapshot copy of all the filer volumes

that make up the ASM disk group Ends the hot backup mode Clones the Snapshot copy and then renames the cloned

ASM disk group Verifies the backup Registers the cloned ASM disk group and storage

information in the RMAN repository


Disk Backup vs. SnapManager Backup

• To demonstrate the benefit of snapshot backup technology we measured the time taken to backup 70GB database to the disk. Here’s what we found:

• Backup to the disk:• Backup set complete, elapsed time: 00:25:20 • It would take about 18 hours to backup 3 TB

database.


Time Log for SnapManager Backup

• 2006-11-10 17:43:45,396 [INFO ]: SMO-07100: Placing database into online backup mode.

• 2006-11-10 17:46:00,724 [DEBUG]: SMO-12000: Executing SnapDriveCommand

• 2006-11-10 17:46:04,762 [DEBUG]:SMO-12001: Result SnapDriveResult (0:00:04.037) (took 4 seconds to take a snapshot of the volume)

• 2006-11-10 17:55:55,841 [INFO ]: SMO-13037: Successfully completed operation: Backup

• 2006-11-10 17:55:55,947 [INFO ]: SMO-13048: Operation Status: SUCCESS

• 2006-11-10 17:55:56,030 [INFO ]: SMO-13049: Elapsed Time: 0:12:33.238 (including all backup management procedures)

• From database alter log:Fri Nov 10 17:43:45 2006 ALTER DATABASE BEGIN BACKUP Fri Nov 10 17:46:42 2006 ALTER DATABASE END BACKUP


Restore Operation

• Similar to backups, SnapManager for Oracle relieves the DBA from having to worry about the underlying layout of data in terms of storage subsystems, disks, host volumes, or host file systems.

• DBAs just choose the backup they want to restore from, and SnapManager does the rest.

• DBAs can also specify the date and time or the Database SCN to which they would like the database restored.

Thank you

Hanan Hit, Principal Database Architect

Lina Shabelsky, Senior Application and Database Engineer

Documents

Oracle ASM Reduces Cost of VLDB Deployment Hanan Hit, Principal Database Architect Lina Shabelsky, Senior Application and Database Engineer NOCOUG Winter