MIGRATING FROM SINGLE INSTANCE TO RAC AND UPGRADING: A DATA

GUARD SUCCESS STORYFuad Arshad @fuadar

Walgreen Company

Me, Myself & I

•More than 15 years of Oracle experience

•Worked in textiles, telecom & retail/healthcare

•Currently work for Walgreens in the Enterprise Shared Services area

•Areas of Interest: backup & recovery, high availability, MAA, performance tuning

Socially Networkable • Twitter @fuadar

• Linkedin http://www.linkedin.com/in/fuadar

• G+ +Fuad Arshad

• Blog http://www.fuadarshad.com

Agenda• Problem

• Analysis

• Decision

• Strategy

• Migration

• Things to look out for!

• Credits

Community Access

• RAC Attack

• IOUG RAC SIG

• IOUG

Blogs / Blogs /Blogs .

• http://www.orana.info

• http://blogs.oracle.com

• http://www.oaktable.net

Ask Questions

• Freelists Oracle L

• Stack Overflow

• Quora

• MOS Communities

• Oracle Forums

The Problem

“It was very early, and we were still like beta or alpha stage, and so we started receiving a ton of download. The server became overloaded, and that's when I realized that this had a huge market.”

Shawn Fanning (Napster)

Paper #371 7

Our Problem

•The application was growing and adding features

•Customer access slowdown was reported

Paper #371 8

Our Problem

•System built in the early 2000’s to handle the new wave of technology

•More work as newer features were rolled out

•Architecture & design not intended to handle the rate of change

•Database is always the bottleneck

Paper #371 9

Analysis

•Active/Passive servers

•Oracle database version close to desupport

•Old OS version

•Old SAN infrastructure

•CPU and disk I/O issues

Result -----> IT WAS OLD Paper #371 10

Analysis

•Active/Passive Servers

•Oracle Database version close to desupport

•Old OS version

•Old SAN infrastructure

•Result ---------------- IT WAS OLD Paper #371 11

Paper #371 12

Veritas VolumeManager Upgrade

(Stage 2)

Veritas VolumeManager Upgrade

(Stage 1)

Veritas File SystemManager Upgrade

OS Upgrade toSolaris 10 (Stage 2)

OS Uprade toSolaris 10 (Stage 1)

Rebuild the stagingcluster (to test the

productionmigration)

Current serverNew ServerDependency (disk)

Veritas ClusterUpgrade

Migrate productionstorage to EMCVemax storageframe (here)

Expand Disk performance (IOPS)

Migrate HighPhysical IO objects

to SSD

Create a dedicatedPerformance Test

Environment

Migrate Storage toASM

ETA: Feb. / March 2011

Disk rehost case:DMX800 End of LifeAvailability8G network & additional IOPSSupport Oracle 11g upgrade

Time to Server CapacityEnd of Life (2011 Holidays) 6 - 7 Months

Active - Passive Server Failover Rolling Upgrades (6 hours of TTL Outages, 10 hours single node)Predecessors1) Setup ODS2) Migrate Data Guard

?

DBUpgrades

3 outages - 2 hours each

VDI & CodeRepository Backups

Decision

•Let’s migrate to RAC

•Can it be done yesterday?

•Can we do it with zero downtime?

•The application team doesn’t want to change anything

Paper #371 13

Strategy

•Export/Import

•Transportable Table spaces

•Disk based replication

•GoldenGate

•Data Guard - The Choice

Paper #371 14

Pre-Migration

•Build of servers

•Build of network

•RAC starter kits

Paper #371 15

Pre-Migration

• The strategy was to build a RAC database from a single instance environment

• Oracle provided instructions on how to prep the single instance environment so that it is RAC-aware. This includes running catclust scripts and adding undo tablespaces for the number of nodes that the new cluster will have

Paper #371 16

Pre-Migration

• The server builds need to have all the patches based on the OS that is being used

• Oracle provides a variety of starter kits that can be used to validate the configuration

• It is also important that the network be configured and available based on best practices

• 11.2.0.2 and above have added requirements like SCAN that need to be accounted for

Paper #371 17

Pre-Migration

• RAC Check is important tool to ensure that the environment is built based on best practices

• RAC Check can be downloaded at MOS # 1268927.1 and is updated frequently

– OS settings including kernel parameters and patches– CRS/Grid Infrastructure– RDBMS memory and database parameters– ASM parameters and statistics

Paper #371 18

Pre-Migration (Primary)Data Guard setup steps need to be followed which include :

• Enable supplemental logging & force logging on primary

• Add additional REDO threads (Based on # of RAC nodes)

• Add additional UNDO tablespaces (Based on # of RAC nodes)

• Backup the primary database

• Create a PFILE from the primary database

• Copy TNSNAMES and LISTENER files into a staging directory

• Configure the primary database for Data GuardPaper #371 19

Pre-Migration

• RAC Check is important tool to ensure that the environment is built based on best practices

• RAC Check can be downloaded at MOS # 1268927.1 and is updated frequently

– OS settings including kernel parameters and patches– CRS/Grid Infrastructure– RDBMS memory and database parameters– ASM parameters and statistics

Paper #371 20

Pre-Migration (Standby RAC)• Data Guard setup steps need to be followed which

include:

• NFS mount or copy backup and auxiliary files

• Configure Oracle Net which includes setting up TNSNAMES and LISTENER for 11.2.0.2 (This will be done in the Grid infrastructure home)

• Modify the TNSNAMES.ORA to include the primary as well as the standby tns entries

• Create a tns entry for the listener and make sure it is referenced in the REMOTE_LISTENER (REMOTE_LISTENER specifies the global list of listeners in a RAC configuration)

• Make sure the listener alias referenced by the LOCAL_LISTENER initialization parameter is in each standby tnsnames.ora file

• Ensure that the standby server has the same software version and patches installed as the primary

Paper #371 21

Pre-Migration (Standby RAC, cont.) Data Guard setup steps need to be followed which include:

• Ensure that a password file is configured on both the primary and the standby

• Copy the PFILE to the $ORACLE_HOME/dbs in the standby node

• Add DB_UNIQUE_NAME and CLUSTER_DATABASE parameters in the PFILE

• Modify all the other parameters outlined in the white paper

• Connect to the standby and create an SPFILE in the ASM diskgroup

• Startup the proposed standby environment in NOMOUNT mode

• Duplicate the database using “DUPLICATE TARGET DATABASE FOR STANDBY”

• Setup Server Control (SRVCTL) to register the instancesPaper #371 22

Pre-Migration (Standby RAC, cont.)Data Guard setup steps need to be followed which include:

• Add additional standby REDO logs (The number of standby redo logs should be one more than the number of online REDO logs per thread and should be the same size)

• Enable Managed Recovery Mode on the standby database

• Enable archive destination on the primary to start shipping logs to the standby

• Validate the transfer

• Setup Data Guard Broker

• Validate the configuration and ensure that logs are being applied

• Bring up the other nodes in the RAC environmentPaper #371 23

MigrationMigrating is simple after the Data Guard database is setup

• Ensure all ARCHIVELOGS have been applied

• Put the primary database in restricted mode and switch ARCHIVELOGS

• After all the outstanding logs have been applied, shutdown REDO apply on the standby

• Validate SCN across the primary and the standby

• Two options from here:

- Switchover & upgrade

- Shutdown & upgrade

**We Chose Shutdown & Upgrade**24Paper #371

Migration

• All standby RAC nodes can be up if using DBUA

• Ensure pre-checks are done

• DBUA is brutal if you don’t remove the obsolete parameters

• DST changes are important

• If manually upgrading DBMS_DST needs to be run to upgrade DST

Paper #371 25

Things to Look Out For Ensure Environments Are Validated

• RDA should be run and all issues fixed

• RAC check will give you a detailed overview of the system

• Ensure all starter kits and the RAC manuals are consulted: the result can be a good best practices evolving document

• Use services to better manage how the application connects and load balance your system

• Private network is key with RAC: follow Oracle’s best practices.

• XA changes in 11.2.0.2 & can cause issues: important to understand them

• “Compatible” parameter testing is important.

• Monitoring the environment after go-live is very importantPaper #371 26

CreditsI’d like to thank the following people for helping me with the presentation and accompanying white paper:

• Matthew Gallagher, Architect -- Walgreens

• Brice Lahl, Manager DB Architecture -- Walgreens

• Daniel Morgan, ACE Director -- Drugstore.com

A special shout out to Arup Nanda & Alex Gorbachev for encouraging me to present and Leighton Smith for reviewing my session

Also special credit to the RAC Assurance Team at Oracle for helping is get through this migration and subsequent issues. Some content in the whitepaper and presentation has been republished with permission from the Oracle RAC Assurance Team whose goal had been to proactively provide input to customers with new and existing CRS, RAC, ASM and MAA implementations.

Paper #371 27

Questions• Please fill

out evaluations

Paper #371 28