Oracle Database Replications with PowerMax/VMAX Storage · Oracle Database Replications with PowerMax/VMAX Storage Dell IT Oracle E-Business Suite Database Replication Case Study

Oracle Database Replications with PowerMax/VMAX StorageDell IT Oracle E-Business Suite Database Replication Case Study

Kai Yu, Distinguished Engineer

Oracle Ready Database Engineering, Dell EMC

John Tao, Consultant, Database Administrator,

Dell IT Infrastructure

Oracle OpenWorld 2019, Sept 16th 2:30pm; Tuesday Sept 17th, 11:20am; Wednesday Sept 18th,

1:50pm, Moscone South, Dell Booth (#1011)

© Copyright 2019 Dell Inc.2Slide

Kai Yu

• Distinguished Engineer, Dell EMC Oracle Ready Solutions, Dell EMC Technical Leadership Community

• 25+ years specializing in IT Infrastructure, Oracle Database, Oracle EBS, Cloud and Virtualization

• Award winning author and frequent speaker at Global IT and Oracle technology community

John Tao• Consultant, Database Administrator, Dell IT Infrastructure

• 25+ years working in IT industry and 19+ years at Dell

• Specializing in Oracle Database and EBS architecture and administration

• Speaker in Oracle Conferences

• Co-recipient of the 2011 OAUG Innovator of Year


Dell IT Major Oracle EBS Database Replication Challenges

• Mission Critical 90TB Production Database on 16 Node Oracle RAC and Dell EMC VMAX Storage

• Remote Replication Across Data Centers: Production, DR, and Non-PROD for multiple purposes

– Production -> DR & Business Continuity->Production support (Scrambled Data)->Perf Test

• Local Replication from Perf Test to Test/Dev database environments

– 4 UAT test databases (Immediate need)

– 4 DTIs, 4 DEVs,1 PAT test and development databases (On Demand)

– Flexible to mount DEV/Test Databases to any DEV/Test DB hosts

• Cost for a full database copy/refresh

– Remote replication: weeks copy time

– Local Replication: 90TB storage and days copy time

• Looking for a storage based replication solution based on SRDF and SnapVX snapshot technology

– SRDF replicates the data to a remote data center

– SnapVX replicates data in local with multiple snapshots


PowerMax/VMAX SnapVX Snapshot Overview

• SnapVX snapshot characteristics– Create instant and storage-consistent point-in-time replicas of devices

– Space-efficient

▪ They are simply a set of pointers that points to the original data, or its own version when the source data is

modified

▪ Storage saving and memory saving by pointing to the same locations

– Both source data and snapshots can be updated separately

TDAT TDAT TDAT TDAT

SnapshotSource

TDAT TDAT TDAT TDAT

SnapshotSource


PowerMax SnapVX Snapshot Overview• Replicating Oracle database with SnapVX Snapshot

– Consistent snapshot with all data, control, redo logs files

– Take seconds to complete for any size

– Linked to target devices, accessed by another hosts

– Source, snapshots are in the same storage array

– A replicated database on the snapshot are restartable and recoverable (supported since Oracle 12c)

– Multiple snapshots linked to multiple sets of targets to form multiple snapshot databases

– Multiple snapshot relink and unlink with a set of targets

Linked

Target

Production

Volume

Snapshot

Snapshot

Snapshot

Source

Database

Host

Snapshot

Database

Host

Production

Devices

Snapshot 10am

Snapshot 11am

Snapshot 12pm

Establish

Snapshot 5pm

Auto-

expiration

Target

Devices

[Re-]Link

Unlink

Terminate

Secure


PowerMax/VMAX SRDF Overview

• SRDF for disaster recovery/ businesses continuity– Remote replication between storage arrays

– For different topologies including two, three and even four sites

– Four different modes

▪ SRDF Synchronous( SRDF/S)

▪ SRDF Asynchronous(SRDF/A)

▪ SRDF Adaptive Copy(SRDF/ACP)

▪ SRDF/Metro

– SRDF consistency group: put all the volumes in a SRDF consistency group

– Enabling SRDF consistency to preserved target devices consistency

– SRDF for Disaster Recovery or businesses continuity configuration


Oracle Database Replication Use Cases

1. Local Replication with Snapshot 2. Replication with Multiple Snapshot Databases


Oracle Database Replication Use Cases

3. Remote SRDF Replication 4. Remote SRDF plus Local Snapshot Replication


Dell EMC Ready Solution: Mixed OLTP, Snapshots and OLAPAll test/dev databases are based on snapshots of OLTP Production databases

Consolidated 50 Databases

snapshots


Dell IT Major Oracle EBS Database Replication SolutionArchitecture: SRDF Remote Replication plus Snapshot Local Replication:

Similar to the replication use case 4 above (remote plus local)

Production DB 90TB

PROD Support

(scrambled)

90TB

UAT

UAT

UAT

UAT

DIT

DIT

DIT

DIT

DEV

DEV

DEV

DEV

PAT

4xUATs on

snapshots

(90TB)

4xDITs on

snapshots

(90TB)

4xDEVs on

snapshots

(90TB)

1xPAT onsnapshot

(90TB)

Production Data Center Non-PROD Data Center

SRDF

Performance Test Database

90TB

SnapshotSnapshot Snapshot

Snapshot

Diaster Recovery Data Center

Diaster

Recovery DB 90TB

Business Continuity DB

90TB

Snapshot

Copy Mode

Notes

90TB: Database Size and actual storage capacity needed.

(90TB): Database size of snapshot database. The actual storage usage

is just for pointers and keeping data changes.

SRDF Remote

Replication


Dell IT Major Oracle EBS Database Replication Solution Summary

• DEV/test databases access the full size of scrambled production data

• Space saving: DEV/test databases are snapshots, which are just pointers, not additional space used initially.

Only small space is used to keep data changes made after snapshots

• Time saving: 5 seconds to take a snapshot of 90TB DB

• Snapshot databases mounted to any database hosts

• Multiple snapshots for multiple DEV/test environments

• Overcome the limitations of database based snapshot technologies

– Source and snapshot databases are mounted in the same host cluster

– Source and snapshot databases are stored in a special cluster file system

• Easy implementation

– Dell EMC Unisphere for PowerMax GUI: a few clicks to create snapshots and their links

– SnapVX CLI: commonly used by storage admins

– AppSync: completely automates the database snapshot replication process

•


Easy Steps to create and link snapshots


Backup Slides


Creation of Snapshot Database: Manual Process Overview:

1. Check prerequisites: PROD Database, Snapshot Hosts

2. Take snapshot of the PROD database Storage group: including data volumes, REDO volumes and FRA

volumes

3. Create the link targets of the snapshots

4. Map the snapshot link targets to the Snapshot Database Host(s)

5. Present the snapshot links as the Linux devices

6. Rename the ASM diskgroups and ASM disks of the snapshot devices

7. Mount the Snapshot database

8. Rename the Snapshot database and open the snapshot database.

Task Colors: Storage Engineer tasks

▪ Linux Engineer tasks

▪ Oracle DBA tasks


Step1: Check prerequisites: Source Database and Snapshot Hosts

• Prerequisites:

– Source connect to PowerMAX VMAX storage

– Source Database storage parent group includes all database volumes such as DATA volumes,

REDO Volumes and FRA volumes

– Snapshot host(s) connected to PowerMax storage ( with physical connectivity and FC zoning)

– OCR/votingdisk volumes and TEMP volumes configured and mapped to the Snapshot host(s)

– Oracle 12c Grid Infrastructure /ASM instance configured with diskgroup for OCR and Votingdisk

and TEMP diskgroup mounted in the ASM instance

– Oracle database software home installed


Easy Steps to create and link snapshots

ARCH

OMEGA

Taking

Snapshots

Links

OMEGA_LNK_SG_001

Source

Database

Hosts

Snapshot

Database

Hosts

OMEGA_SNP

DATA

FRA

ARCH snapshots

DATA snapshots

FRA snapshot

ARCH_SG_001

DATA_SG_001

FRA_SG_001

DATA_ADC

FRA snapshot

DATA_ADC_SG_001

Source Storage

Parent Group

OMEGA Snapshot

Snapshot Link

Targets

DATA_ADCsnapshot


Step2: take snapshot of the storage group of PROD Database

Create snapshots of the source database storage group


Step3: Create the link targets of the snapshot

Link Snapshot


Step4: Map the snapshot link targets to the Snapshot Database host(s)

Create masking view to map the snapshot link target to snapshot database hosts


Step5: Present the snapshot links as the Linux device(s)

1. Get the link target device WWn IDs for all the link target devices

sysdev –sid 457 list dev 00025, 00026, 00027, 00028, 00029 ….. -wwn

2. Rescanning the SCSI bus on the snapshot database host with this command:

[root#] rescan-scsi-bus.sh

3. Configure the multipath for the snapshot link targets by adding the corresponding entries in the

/etc/multipath.conf file and run $service multipathd reload

multipath {

wwid 360000970000197801276533030303342

alias OMEGA_SNP_FRA

4. Create a new udev rules file /etc/udev/rules.d/60-oracle-asmdevices.rules:

KERNEL=="dm-*", ENV{DM_NAME}==" ORA_SNP_FRA?", OWNER:="grid", GROUP:="asmadmin", MODE="0660“

……

5. Reload the udev rules as follows: [root#] udevadm trigger


Step6: Rename the ASM diskgroups and ASM disks of the snapshot(s)

1. Login to snapshot database host as the owner of the ASM instance (gird user)

2. Check ASM diskgroups of snapshot volumes are dismounted. SQL> select NAME, STATE from v$asm_diskgroup;

3. Rename the ASM diskgroups with these commands as the grid user :

$renamedg dgname=DATA newdgname=DATA_S asm_diskstring=/dev/oracleasm/disks/* verbose=TRUE

$renamedg dgname=REDO newdgname=REDO_S asm_diskstring=/dev/oracleasm/disks/* verbose=TRUE

$renamedg dgname=FRA newdgname=FRA_S asm_diskstring=/dev/oracleasm/disks/* verbose=TRUE

4. Mount the diskgroups in restrict mode with these commands as the grid user:

$asmcmd mount --restrict DATA_S

$ asmcmd mount --restrict FRA_S

$ asmcmd mount --restrict REDO1_S

5. Rename the disks of these diskgroups by logging to the ASM instance as sysasm

sqlplus / as sysasm

SQL> alter diskgroup DATA_S rename disks all;

SQL> alter diskgroup REDO_S rename disks all;

SQL> alter diskgroup FRA_S rename disks all;

6. Dismount and remount the diskgroup 7. check the diskgroup and disk status:

SQL> alter diskgroup DATA_S dismount;

SQL> alter diskgroup REDO_S dismount;

SQL> alter diskgroup FRSA_S dismount;

SQL> alter diskgroup DATA_S mount;

SQL> alter diskgroup REDO_S mount;

SQL> alter diskgroup FRSA_S mount;


Step7: Mount the Snapshot database(s)

1. Copy and edit the database init<prod>.ora file:

*.control_files='+DATA_S/OTC1S/CONTROLFILE/current.256.1006637651’

create audit_file_dest= /u01/app/oracle/admin/<dbname>/adump

2. Startup snapshot DB instance mount : startup nomount pfile=init<prod>.ora; alter database mount;

3. Rename datafile and logfile by changing ASM diskgroups names of all the datafiles and logfiles : generate

SQL scripts

eg: alter database rename file '+DATA/OTC1S/DATAFILE/apps_ts_tx_olad.334.1006640913’

'+DATA_S/OTC1S/DATAFILE/apps_ts_tx_olad.334.1006640913’ ;

alter database rename file '+REDO/OTC1S/ONLINELOG/group_23.442.1006647419' TO

'+REDO_S/OTC1S/ONLINELOG/group_23.442.1006647419' ;

4. Drop all tempfiles that point to the old temp diskgroup.

5. Open the snapshot database : SQL> alter database open;


Step8: Rename the snapshot database and open the snapshot database

1. Shutdown the snapshot database

2. Modify init<prod>.ora file :

*.db_create_file_dest='+DATA_S’

*.log_archive_dest_1='location="+FRA",valid_for=(ALL_LOGFILES,ALL_ROLES)’

3. Startup the snapshot database with new instance name <snapdb> with the same init<prod>o.ora

$export ORACLE_SID=<snapdb>

sqlplus / as sysdba

SQL> startup mount pfile=init<PROD>.ora

4. Use the nid utility to change the database name from <PROD> to <SNAPDB>

[oracle]$ /u01/app/oracle/product/12.1.0.2/db_1/bin/nid TARGET=/ DBNAME=<snapdb>

For example: <prod>=OTCS1S , <snapdb>= OTC1SDEV

command : $/u01/app/oracle/product/12.1.0.2/db_1/bin/nid TARGET=/ DBNAME=OTC1SDEV

**Note: this command will shutdown the database instance before it completes the task.

5. edit init<snapdb>.ora : Copy init<prod>.ora to init<snapdb>.ora

And modify init<snapdb>.ora by replacing <prod> with <snapdb>

Including changing *.dbname = <prod> to *.dbname=<snapdb>

6. Start new snapdb instance:

SQL> startup mount pfile=init<snapdb>.ora

SQL> alter database open resetlogs;

7. All the new temp file in the temp diskgroup:

SQL>CREATE TEMPORARY TABLESPACE TEMP TEMPFILE '+TEMP' SIZE 100M


Step8: Rename the snapshot database and open the snapshot database

Change the database name with nid utility: Start up the new database:

Documents

Oracle Database Replications with PowerMax/VMAX Storage · Oracle Database Replications with PowerMax/VMAX Storage Dell IT Oracle E-Business Suite Database Replication Case Study