Dataguard Fsfo Implementation

DATAGUARD FAST START FAILOVER IMPLEMENTATION

1/99

/

Alejandro Vargas | Principal Support ConsultantOracle Advanced Customer Services

INDEX

Summary................................................................................................................................................................................3The environment....................................................................................................................................................................3

Database Checkup.................................................................................................................................................................4

The Primary database spfile...............................................................................................................................................5Names Related Parameters............................................................................................................................................5Network Related Parameters..........................................................................................................................................6Management Related Parameters..................................................................................................................................9

Standby Database Creation On Windows............................................................................................................................13

Enable Force Logging.......................................................................................................................................................13Create a Password File and Copy it Over to the Standby Site.........................................................................................14Create standby redologs...................................................................................................................................................14Enable Archiving...............................................................................................................................................................17Setup the network using netca or netmgr, edit the tnsnames.ora to register the listeners...............................................17Shutdown the Primary Database and copy it Over to the Standby Server.......................................................................22Create the Database Service on the Standby Server.......................................................................................................22Startup mount the Primary Database and create a standby controlfile...........................................................................22Create a copy of the primary spfile and modify it for the standby.....................................................................................23Copy the standby controlfile over to the Standby Server..................................................................................................26Generate the standby spfile from the pfile you prepared before.......................................................................................26Start Recovery on the standby database and check........................................................................................................27


2/99

/

Configuring Data Guard Broker............................................................................................................................................35

Enable flashback database on primary database.............................................................................................................35Enable database broker on primary database..................................................................................................................37

Enable database broker on standby database..............................................................................................................37Setup the Local_Listener parameter on the Primary Database....................................................................................38Setup the Local_Listener parameter on the Standby Database...................................................................................38Setup the Tnsnames.ora and Listener.ora parameter files on the Primary Database..................................................38Setup the Tnsmanes.ora and Listener.ora parameter files on the Standby Database.................................................42

Create the DataGuard Broker Configuration....................................................................................................................46Enable the Broker Configuration...................................................................................................................................50

Configure Data Guard Broker For Switchover..................................................................................................................51

Configuring the Observer Process.......................................................................................................................................56

Fast Start Failover Test....................................................................................................................................................58Observer Output During FSFO......................................................................................................................................58Database Monitoring During FSFO...............................................................................................................................58Database Monitoring During FSFO Using the Alert Logs..............................................................................................60SATISTD alert log (Standby converted to Primary).......................................................................................................60SATI alert log (Failed Primary Reinstated and converted to Standby)..........................................................................66

DGMGRL CHECKUPS.....................................................................................................................................................78

Client Failover Configuration................................................................................................................................................86Test client failover.............................................................................................................................................................88References.......................................................................................................................................................................98


3/99

/

SummaryThis document describes the implementation of a Dataguard Environment on Windows using Oracle RDBMS10.2.0.4. The setup includes 3 servers: Primary, Physical Standby and Observer.

The Configuration was setup in maximum availability mode and Fast Start Failover was configured with a 30seconds threshold.

The observer was setup to be started at server boot and was set to be run by system so that no user can shutdownit. A watchdog process run by a central control application takes care of restarting the observer process in casethat is shutdown; the observer process is also configured with the scheduler to be started up at boot time.

The environmentThe environment is setup of 3 Windows Server 2003 with 4 cpus and 2GB memory each.

Server DGONEIP 192.168.2.12Oracle Home C:\oracle\product\10.2.0\db_1Database SATIMain Role Primary

Server DGTWOIP 192.168.2.14Oracle Home C:\oracle\product\10.2.0\db_1 Database SATISTDMain Role Standby

Server DGTHREEIP 192.168.2.16


4/99

/

Oracle Home C:\oracle\product\10.2.0\db_1DatabaseMain Role Data Guard Broker Observer process site

Database CheckupThe Database is a standalone partial copy of the main production database, it is updated with specific informationat regular intervals and serves as a front end repository for a web application open to the general public so it doesrequire very high availability. In case of primary site crash the time to failover is about 40 seconds.

The database access is read only mostly, except for the periodic uploads of data that maintain it updated.

C:\Documents and Settings\AV\Desktop\SCRIPTS> sqlplus / as sysdba

SQL*Plus: Release 10.2.0.4.0 - Production on Sun Dec 14 09:43:57 2008

Copyright (c) 1982, 2007, Oracle. All Rights Reserved.

Connected to:Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - ProductionWith the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> show sga

Total System Global Area 1258291200 bytesFixed Size 1298304 bytesVariable Size 494928000 bytesDatabase Buffers 754974720 bytesRedo Buffers 7090176 bytes


5/99

/

The Primary database spfile

Configuring a Dataguard environment starts by preparing an Spfile that contains specific parameters that defineand control Dataguard operation. When Dataguard Broker is used to manage the environment an spfile must beused because the broker will take over the responsibility to setup and change specific parameters under specificcircumstances, like manual switchover, fast start failover, add or remove a standby, reinstate a failed primary, etc.

The Dataguard related parameters could be confusing at the beginning so it is convenient to use some time tobecome familiar with their meaning and use.

The next table explains the parameters required for configuring Dataguard, they are divided in groups to betterexplain their functionality.

Names Related Parameters

Parameter Functionality and comments

db_name

db_name contains the database name, this parameter has the same value on allmembers of the Dataguard environment, i.e.:

Primary Database : *.db_name='SATI'Physical standby : *.db_name='SATI'

db_unique_name

db_unique_name is used to identify a database created for reporting or aphysical standby. This database must have a unique DB_UNIQUE_NAME. The primarydatabase will have db_unique_name=db_name, while the standby will havedb_unique_name set to a unique value that is different from the db_name, i.e. :

Primary database : *.db_unique_name=SATIPhysical standby : *.db_unique_name=’SATISTD’


6/99

/

instance_name

Instance_name has the value of the System Identifier; The SID identifies theinstance's shared memory on a host. On Dataguard is important to take care thatthe instance_name is correctly defined on the physical standby with the samevalue as the db_unique_name, because the broker to establish communications usesthis value. A possible error here is to have correctly defined db_unique_name,but because the parameter file was copied from the primary instance name canremain set as the database name, that will make fail the automatic startup of thestandby on switchover or failover.

Primary database : *.instance_name='SATI'Physical standby : *.instance_name='SATISTD'

Network Related Parameters


fal_server

fal_server specifies the FAL (fetch archive log) server for astandby database. This parameter is relevant only when the databaseis open on the primary role. On The primary database the standby is configured as fal_server sothat in case of role change the primary, converted to standby, willbe able to get the required logs to close gaps if needed.

Primary database : *.fal_server='(DESCRIPTION= (ADDRESS_LIST=(ADDRESS= (PROTOCOL=TCP) (HOST=DGTWO) (PORT=1522))) (CONNECT_DATA=(SERVICE_NAME= SATISTD_XPT)(SERVER=dedicated)))'

Physical standby : *.fal_server='(DESCRIPTION= (ADDRESS_LIST=(ADDRESS= (PROTOCOL=TCP) (HOST=DGONE) (PORT=1522)))(CONNECT_DATA=(SERVICE_NAME= SATI_XPT)(SERVER=dedicated)))'

fal_client specifies the FAL (fetch archive log) client name that isused by the FAL_SERVER, to refer to the FAL client. The fal_clientwill point to the same database where it is configured, and will beactive only when this database will be mounted on the standby role.


7/99

/

fal_client Primary database : *.fal_client='(DESCRIPTION= (ADDRESS_LIST=(ADDRESS= (PROTOCOL=TCP) (HOST=DGONE) (PORT=1522))) (CONNECT_DATA=(SERVICE_NAME= SATI_XPT) (INSTANCE_NAME=SATI) (SERVER=dedicated)))'

Physical standby : *.fal_client='(DESCRIPTION= (ADDRESS_LIST=(ADDRESS= (PROTOCOL=TCP) (HOST=DGTWO) (PORT=1522)))(CONNECT_DATA=(SERVICE_NAME= SATISTD_XPT)(INSTANCE_NAME=SATISTD)(SERVER=dedicated)))'

archive_lag_target

archive_lag_target force a log switch after the specified amount oftime in seconds elapses.

A 0 value is used for real time apply. The recommended value, whennot using real time apply is 1800 (30 minutes).

Primary database : *.archive_lag_target=0Physical standby : *.archive_lag_target=0

local_listener

local_listener specifies the name of the listeners that are runningon the same machine as this instance. The address or address list isspecified in the TNSNAMES.ORA

Primary database : *.local_listener='(ADDRESS = (PROTOCOL =TCP)(HOST = DGONE)(PORT = 1522))'

Physical standby : *.local_listener='(ADDRESS = (PROTOCOL =TCP)(HOST = DGTWO)(PORT = 1522))'

log_archive_config

Log_archive_config enables or disables the sending of redo logs toremote destinations and the receipt of remote redo logs, andspecifies the unique database names (DB_UNIQUE_NAME) for eachdatabase in the Data Guard configuration. It will have the samevalue on both the primary and standby databases.

Primary database : *.log_archive_config='DG_CONFIG=(SATI,SATISTD)'Physical standby : *.log_archive_config='DG_CONFIG=(SATI,SATISTD)'


8/99

/

log_archive_dest_1

The LOG_ARCHIVE_DEST_1 parameter defines the first of up to 10 logarchive destinations, each one must specify either the LOCATION orthe SERVICE attribute to specify where to archive the redo data. There are several optional attributes. The LOCATION or the SERVICEattribute, must be the first attribute supplied.

The other attributes are used to control aspects of how redotransport services transfer data from the primary to the standbydatabase. V$ARCHIVE_DEST view can be queried to see the currentsettings for each destination.

In this example I’m using Location and because the directorystructure is identical on both the primary and standby database thelocation is identical.

Primary database : log_archive_dest_1= 'LOCATION=”G:\oracle\oradata\SATI\archive”

VALID_FOR=(ONLINE_LOGFILE,ALL_ROLES)'

Physical standby : log_archive_dest_1= 'LOCATION=”G:\oracle\oradata\SATI\archive”VALID_FOR=(ALL_LOGFILES,ALL_ROLES)'

log_archive_dest_state_1 For every log_archive_dest_state_n parameter must be configured alog_archive_dest_state_n parameter as well. This parameter specifieswhether the corresponding destination is currently enabled,deferred, disabled or can be used as an alternated location.

Primary database : log_archive_dest_state_1='enable'Physical standby : log_archive_dest_state_1='enable'

log_archive_dest_2

If using Dataguard Broker the broker will automatically setup on theprimary database a log_archive_dest_state_2

Primary database : log_archive_dest_state_2=’service="DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=DGTWO)(PORT=1522)))(CONNE CT_DATA=(SERVICE_NAME=SATISTD_XPT)(INSTANCE_NAME=SATISTD)(SERVER=ded icated)))", LGWR SYNC AFFIRM delay=0 OPTIONAL max_failure=0


9/99

/

log_archive_dest_state_2

max_connections=1 reopen=300 db_unique_name="SATISTD" registernet_timeout=180 valid_for=(online_logfile,primary_role)’

Primary database : log_archive_dest_state_2='enable'Physical standby : log_archive_dest_state_2='enable'

log_archive_max_processes

log_archive_max_processes specifies the number of archiverbackground processes (ARC0 through ARCn) Oracle initially invokes,the maximum is 30. Note that a high number of processes take longerto synchronize on switchover/failover. You can increase dynamicallythis parameter using alter system in case of load. For manualswitchover set it to 2. Increase it only if you have a lot of redoactivity and the standby is not updated enough fast.

Primary database : log_archive_max_processes=2Physical standby : log_archive_max_processes=2

log_archive_min_succeed_dest log_archive_min_succeed_dest defines the minimum number ofdestinations that must succeed in order for the online logfile to beavailable for reuse.

Management Related Parameters


dg_broker_start

dg_broker_start is set to true to enable automatic startup of the DataGuard broker DMON process.

Primary database : dg_broker_start=truePhysical standby : dg_broker_start=true

standby_file_management

standby_file_management set to AUTO enable automatic standby filemanagement so that operating system file additions and deletions on theprimary database are replicated on the standby database. This parameteris only applicable to Physical standby databases.


10/99

/

Primary database : standby_file_management=AUTOPhysical standby : standby_file_management=AUTO

SQL> show parameters spfile;

NAME TYPE VALUE------------------------------------ ----------- ------------------------------spfile string C:\ORACLE\PRODUCT\10.2.0\DB_1\DATABASE\SPFILESATI.ORA

SATI.__db_cache_size=754974720SATI.__java_pool_size=8388608SATI.__large_pool_size=16777216SATI.__shared_pool_size=209715200SATI.__streams_pool_size=50331648*.archive_lag_target=0*.background_dump_dest='Z:\oracle\admin\SATI\bdump'*.compatible='10.2.0.4.0'*.control_files='Z:\oracle\oradata\SATI\control01.ctl','X:\oracle\oradata\SATI\control02.ctl','G:\orac le\oradata\SATI\control03.ctl'*.core_dump_dest='Z:\oracle\admin\SATI\cdump'*.db_block_size=8192*.db_cache_size=0*.db_domain=''*.db_file_multiblock_read_count=16*.db_files=1024*.db_name='SATI'*.db_recovery_file_dest_size=26843545600*.db_recovery_file_dest='G:\FRA'*.db_unique_name='SATI'*.dg_broker_start=TRUE*.fal_client='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=DGONE)(PORT=1522)))(CONNECT_DATA =(SERVICE_NAME=SATI_XPT)(INSTANCE_NAME=SATI)(SERVER=dedicated)))'


11/99

/

*.fal_server='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=DGTWO)(PORT=1522)))(CONNECT_DATA =(SERVICE_NAME=SATISTD_XPT)(SERVER=dedicated)))'*.fast_start_mttr_target=300*.instance_name='SATI'*.java_pool_size=0*.job_queue_processes=50*.large_pool_size=0SATI.local_listener='(ADDRESS = (PROTOCOL = TCP)(HOST = DGONE)(PORT = 1522))'SATISTD.local_listener='(ADDRESS = (PROTOCOL = TCP)(HOST = DGONE)(PORT = 1522))'*.local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=DGONE)(PORT=1522))'*.LOG_ARCHIVE_CONFIG='DG_CONFIG=(SATI,SATISTD)'*.LOG_ARCHIVE_DEST_1='LOCATION=G:\oracle\oradata\SATI\archiveVALID_FOR=(ALL_LOGFILES,ALL_ROLES)DB_UNIQUE_NAME=SATI'SATI.log_archive_dest_1='location="G:\oracle\oradata\SATI\archive"','valid_for=(ONLINE_LOGFILE,ALL_ROL ES)'*.log_archive_dest_2=''SATI.log_archive_dest_2='location="dgsby"','valid_for=(STANDBY_LOGFILE,STANDBY_ROLE)'*.LOG_ARCHIVE_DEST_STATE_1='ENABLE'SATI.log_archive_dest_state_1='ENABLE'*.log_archive_dest_state_2='ENABLE'SATI.log_archive_dest_state_2='ENABLE'*.LOG_ARCHIVE_FORMAT='%t_%s_%r.arc'SATI.log_archive_format='%t_%s_%r.arc'*.log_archive_max_processes=2*.log_archive_min_succeed_dest=1SATI.log_archive_trace=0*.open_cursors=1000*.pga_aggregate_target=524288000*.processes=150*.query_rewrite_enabled='TRUE'*.REMOTE_LOGIN_PASSWORDFILE='EXCLUSIVE'*.remote_os_authent=TRUE*.session_max_open_files=20*.sessions=500*.sga_max_size=1258291200*.sga_target=1048576000*.shared_pool_size=0


12/99

/

SATI.standby_archive_dest='dgsby'*.standby_file_management='AUTO'*.star_transformation_enabled='TRUE'*.streams_pool_size=50331648*.timed_statistics=TRUE*.undo_management='AUTO'*.undo_retention=3600*.undo_tablespace='UNDOTBS1'*.user_dump_dest='Z:\oracle\admin\SATI\udump'*.utl_file_dir='Z:\oracle\admin\SATI\bdump'*.workarea_size_policy='AUTO'


13/99

/

Standby Database Creation On Windows

Enable Force Logging

In order to assure that all transactions are written to the online redologs we enable force logging, that optionprevents the execution of nologging operations.

ALTER DATABASE FORCE LOGGING;

By working in force logging mode there is a performance penalty that we must pay. In some circumstances wherewe would like to run massive changes that require nologging mode we can stop log apply and at the end of thebatch close the gap on the standby using an incremental backup.

For details on this technique please check chapter 12.7 of the Oracle® Data Guard Concepts and Administration10g Release 2 (10.2), Part Number B14239-05

http://download.oracle.com/docs/cd/B19306_01/server.102/b14239/scenarios.htm#CIHEGFEG

12.7 Using RMAN Incremental Backups to Roll Forward a Physical Standby Database

In some situations, RMAN incremental backups can be used to synchronize a physical standby databasewith the primary database. Using the RMAN BACKUP INCREMENTAL FROM SCN command, you cancreate a backup on the primary database that starts at the standby database's current SCN, which can thenbe used to roll the standby database forward in time.


14/99

/

Create a Password File and Copy it Over to the Standby Site

It is important to create the password file on the primary site and copy the generated file to the standby server. Thepassword file at both sites must be same one. Even creating the password file on the standby site in the same waythat was created on the primary site will not work.

C: >orapwdUsagZ: orapwd file=<fname> password=<password> entries=<users> force=<y/n>ignorecase=<y/n> nosysdba=<y/n>

wherefile - name of password file (required),password - password for SYS (optional),entries - maximum number of distinct DBA (required),force - whether to overwrite existing file (optional),ignorecase - passwords are case-insensitive (optional),nosysdba - whether to shut out the SYSDBA logon (optional Database Vault only).

There must be no spaces around the equal-to (=) character.

C: >orapwd file=orapwSATI password=<sys_password> force=y

Copy password file C:\oracle\product\10.2.0\db_1\database\ orapwSATI.ora From DGONE to the samedestination on DGTWO

Create standby redologs

When configuring FSFO we will use the synchronous redo transport mod, LogXptMode=SYNC, this mode doesrequire to use standby redo logs. Redo received via redo transport is written on the Standby Database to thecurrent standby redo log group by a RFS background process. When a log switch occurs on the Primary database,


15/99

/

the new redo is written to the next standby redo log group, and the previously used standby redo log group isarchived by an ARCn background process.

The process of filling and archiving redo log file groups at the Primary database is mirrored at the Standbydestination by the sequential filling and archiving of standby redo log groups.

Each standby redo log file must be the same size as the online logs of the Primary database, and in addition thestandby redo log must have at least one more group than the online redo logs on the Primary database.

On this case we have 3 online redo log groups with 2 members each on the primary

SQL> select * from v$log;

GROUP# THREAD# SEQUENCE# BYTES MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- ---------

1 1 297 104857600 2 NO CURRENT 4127112821 16-NOV-082 1 295 104857600 2 YES INACTIVE 4127112003 16-NOV-083 1 296 104857600 2 YES INACTIVE 4127112226 16-NOV-08

SQL> select * from v$logfile;

GROUP# STATUS TYPE MEMBER IS_---------- ------- ------- ---------------------------------------- ---

1 ONLINE Z:\ORACLE\ORADATA\SATI\REDO01.LOG NO1 ONLINE X:\ORACLE\ORADATA\SATI\REDO02.LOG NO2 ONLINE Z:\ORACLE\ORADATA\SATI\REDO11.LOG NO2 ONLINE X:\ORACLE\ORADATA\SATI\REDO12.LOG NO3 ONLINE Z:\ORACLE\ORADATA\SATI\REDO21.LOG NO3 ONLINE X:\ORACLE\ORADATA\SATI\REDO22.LOG NO

6 rows selected.

We will add another 4 standby redo log groups with 2 members each on the primary, when the database will becopied to create the standby the standby redologs will be already there.


16/99

/

SQL> alter database add standby logfile group 4('Z:\ORACLE\ORADATA\SATI\STDBYREDOG4_01.LOG','X:\ORACLE\ORADATA\SATI\STDBYREDOG4_02.LOG')2 SIZE 104857600;

Database altered.


Database altered.


Database altered.


Database altered.

SQL> SELECT GROUP#,THREAD#,SEQUENCE#,ARCHIVED,STATUS FROM V$STANDBY_LOG;

GROUP# THREAD# SEQUENCE# ARC STATUS---------- ---------- ---------- --- ----------

4 0 0 YES UNASSIGNED5 0 0 YES UNASSIGNED6 0 0 YES UNASSIGNED7 0 0 YES UNASSIGNED


17/99

/

Enable Archiving

On 10g we jaut need to mout the database and execute the command

SQL> alter database archivelog;

SQL> archive log listDatabase log mode Archive ModeAutomatic archival EnabledArchive destination G:\oracle\oradata\SATI\archiveOldest online log sequence 295Next log sequence to archive 297Current log sequence 297

Setup the network using netca or netmgr, edit the tnsnames.ora to register the listeners

One important aspect of preparing the Dataguard environment is to configure the network appropriately. A defaultconfiguration can be created and then both the listener.ora and the tnsnames.ora will need to be edited to matchthe standard requirements of Dataguard

On the listener.ora of each server you will need to manually register the corresponding database using thefollowing syntax:

On the Primary

SID_LIST_LISTENER_DGONE =(SID_LIST =(SID_DESC =


18/99

/

(SID_NAME = SATI)(GLOBAL_DBNAME=SATI_DGMGRL)(ORACLE_HOME = C:\oracle\product\10.2.0\db_1)

))

On the Standby

SID_LIST_LISTENER_DGTWO =(SID_LIST =(SID_DESC =(SID_NAME = SATISTD)(GLOBAL_DBNAME=SATISTD_DGMGRL)(ORACLE_HOME = C:\oracle\product\10.2.0\db_1)

))

These are the listener.ora files:

Listener.ora on primary database

# listener.ora Network Configuration FilZ: C:\oracle\product\10.2.0\db_1\network\admin\listener.ora# Generated by Oracle configuration tools.

LISTENER_DGONE =(DESCRIPTION_LIST =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = DGONE)(PORT = 1522))

) )

SID_LIST_LISTENER_DGONE =(SID_LIST =(SID_DESC =(SID_NAME = SATI)(GLOBAL_DBNAME=SATI_DGMGRL)


19/99

/

(ORACLE_HOME = C:\oracle\product\10.2.0\db_1))

)

Listener.ora on standby database

# listener.ora Network Configuration FilZ: C:\oracle\product\10.2.0\db_1\network\admin\listener.ora# Generated by Oracle configuration tools.

LISTENER_DGTWO =(DESCRIPTION_LIST =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = DGTWO)(PORT = 1522))

) )


))

Tnsnames.ora on primary database

# tnsnames.ora Network Configuration FilZ: C:\oracle\product\10.2.0\db_1\network\admin\tnsnames.ora# Generated by Oracle configuration tools.


) )


20/99

/

SATI =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = DGONE)(PORT = 1522))(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = SATI_DGMGRL)

))

SATISTD =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = DGTWO)(PORT = 1522))(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = SATISTD_DGMGRL)

))

EXTPROC_CONNECTION_DATA =(DESCRIPTION =(ADDRESS_LIST =(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC0))

)(CONNECT_DATA =(SID = PLSExtProc)(PRESENTATION = RO)

))

Tnsnames.ora on standby database# tnsnames.ora Network Configuration FilZ: C:\oracle\product\10.2.0\db_1\network\admin\tnsnames.ora# Generated by Oracle configuration tools.

LISTENER_DGTWO =


21/99

/

(DESCRIPTION_LIST =(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = DGTWO)(PORT = 1522)))

)

SATI =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = DGONE)(PORT = 1522))(CONNECT_DATA =

(SERVER = DEDICATED)(SERVICE_NAME = SATI_DGMGRL)

))

SATISTD =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = DGTWO)(PORT = 1522))(CONNECT_DATA =

(SERVER = DEDICATED)(SERVICE_NAME = SATISTD_DGMGRL)

))

EXTPROC_CONNECTION_DATA =(DESCRIPTION =(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC0)))(CONNECT_DATA =

(SID = PLSExtProc)(PRESENTATION = RO)

))


22/99

/

Shutdown the Primary Database and copy it Over to the Standby Server

On windows I just shared the directories and dragged the directories to the standby site.On Linux I use ftp or rman.

Create the Database Service on the Standby Server

When implementing Dataguard on Windows, after installing Oracle server on the server that will host the physicalstandby database, a service needs to be created to for the clone. Startmode is set to manual, because the mainrole of the database is to be physical standby on which case we will mount it only. In case that the database willassume the primary role, the broker will take care of starting it up and change the role.

oradim –NEW –SID SATISTD –STARTMODE manual

Startup mount the Primary Database and create a standby controlfile

C:\Documents and Settings\AV>SQLPLUS "/AS SYSDBA"

SQL*Plus: Release 10.2.0.3.0 - Production on Wed Nov 19 12:58:23 2008


Connected to an idle instance.

SQL> startup mount;ORACLE instance started.


23/99

/

Total System Global Area 1258291200 bytesFixed Size 1292180 bytesVariable Size 654313580 bytesDatabase Buffers 595591168 bytesRedo Buffers 7094272 bytesDatabase mounted.SQL> ALTER DATABASE CREATE STANDBY CONTROLFILE AS 'C:\SATISTD.CTL';

Database altered.

SQL> ALTER DATABASE OPEN;

Database altered.

Create a copy of the primary spfile and modify it for the standby

SQL> create pfile='c:\standby_pfile.ora' from spfile;

File created.

Edited FilZ:

SATISTD.__db_cache_size=545259520SATISTD.__java_pool_size=8388608SATISTD.__large_pool_size=8388608SATISTD.__shared_pool_size=427819008SATISTD.__streams_pool_size=50331648*.archive_lag_target=0*.background_dump_dest='Z:\oracle\admin\SATI\bdump'*.compatible='10.2.0.4.0'


24/99

/

*.control_files='Z:\oracle\oradata\SATI\control01.ctl','X:\oracle\oradata\SATI\control02.ctl','G:\orac le\oradata\SATI\control03.ctl'*.core_dump_dest='Z:\oracle\admin\SATI\cdump'*.db_block_size=8192*.db_cache_size=0*.db_domain=''*.db_file_multiblock_read_count=16*.db_files=1024*.db_name='SATI'*.db_recovery_file_dest_size=26843545600*.db_recovery_file_dest='G:\FRA'*.db_unique_name='SATISTD'*.dg_broker_start=TRUE*.fal_client='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=DGTWO)(PORT=1522)))(CONNECT_DATA =(SERVICE_NAME=SATISTD_XPT)(INSTANCE_NAME=SATISTD)(SERVER=dedicated)))'*.fal_server='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=DGONE)(PORT=1522)))(CONNECT_DATA =(SERVICE_NAME=SATI_XPT)(SERVER=dedicated)))'*.fast_start_mttr_target=300*.instance_name='SATISTD'*.java_pool_size=0*.job_queue_processes=50*.large_pool_size=0*.local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=DGTWO)(PORT=1522))'*.LOG_ARCHIVE_CONFIG='DG_CONFIG=(SATI,SATISTD)'*.LOG_ARCHIVE_DEST_1='LOCATION=G:\oracle\oradata\SATI\archiveVALID_FOR=(ALL_LOGFILES,ALL_ROLES)DB_UNIQUE_NAME=SATISTD'SATISTD.log_archive_dest_1='location="G:\oracle\oradata\SATI\archive"','valid_for=(ONLINE_LOGFILE,ALL_ ROLES)'*.log_archive_dest_2='service="(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=DGONE)(PORT=152 2)))(CONNECT_DATA=(SERVICE_NAME=SATI_XPT)(INSTANCE_NAME=SATI)(SERVER=dedicated)))"',' LGWR SYNCAFFIRM delay=0 OPTIONAL max_failure=0 max_connections=1 reopen=300 db_unique_name="SATI" registernet_timeout=180 valid_for=(online_logfile,primary_role)'*.log_archive_dest_3=''*.LOG_ARCHIVE_DEST_STATE_1='ENABLE'SATISTD.log_archive_dest_state_1='ENABLE'*.log_archive_dest_state_2='ENABLE'*.log_archive_dest_state_3='ENABLE'


25/99

/

*.LOG_ARCHIVE_FORMAT='%t_%s_%r.arc'SATI.log_archive_format='%t_%s_%r.arc'SATISTD.log_archive_format='%t_%s_%r.arc'*.log_archive_max_processes=2*.log_archive_min_succeed_dest=1SATISTD.log_archive_trace=0*.open_cursors=1000*.pga_aggregate_target=524288000*.processes=150*.query_rewrite_enabled='TRUE'*.REMOTE_LOGIN_PASSWORDFILE='EXCLUSIVE'*.remote_os_authent=TRUE*.session_max_open_files=20*.sessions=500*.sga_max_size=1258291200*.sga_target=1048576000*.shared_pool_size=0SATISTD.standby_archive_dest=''*.standby_file_management='auto'*.star_transformation_enabled='TRUE'*.streams_pool_size=50331648*.timed_statistics=TRUE*.undo_management='AUTO'*.undo_retention=3600*.undo_tablespace='UNDOTBS1'*.user_dump_dest='Z:\oracle\admin\SATI\udump'*.utl_file_dir='Z:\oracle\admin\SATI\bdump'*.workarea_size_policy='AUTO'


26/99

/

Copy the standby controlfile over to the Standby Server

On windows I just shared the directories and dragged the directories to the standby site.On Linux I usually use ftp.

Generate the standby spfile from the pfile you prepared before

To generate the Physical Standby spfile, we start the database in nomount mode with the pfile we created from theprimary and edited for the standby database, and we execute the create spfile command.

SQL> STARTUP NOMOUNT PFILE='C:\standby_pfile.txt';ORACLE instance started.

Total System Global Area 1258291200 bytesFixed Size 1292180 bytesVariable Size 520095852 bytesDatabase Buffers 729808896 bytesRedo Buffers 7094272 bytesSQL> CREATE SPFILE FROM PFILE='C:\standby_pfile.txt';

File created.

SQL> SHUTDOWN IMMEDIATE;ORA-01507: database not mounted

ORACLE instance shut down.


Total System Global Area 1258291200 bytesFixed Size 1292180 bytes


27/99

/

Variable Size 520095852 bytesDatabase Buffers 729808896 bytesRedo Buffers 7094272 bytes

SQL> show parameters spfile

NAME TYPE VALUE------------------------------------ ----------- ------------------------------spfile string C:\ORACLE\PRODUCT\10.2.0\DB_1\

DATABASE\SPFILESATISTD.ORASQL> show parameters local_li

NAME TYPE VALUE------------------------------------ ----------- ------------------------------local_listener string LISTENER_DGTWO

Start Recovery on the standby database and check

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;

Database altered.

Check standby alert.log

Wed Nov 19 14:31:59 2008ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSIONMRP0 started with pid=18, OS id=3540Managed Standby Recovery not using Real Time Applyparallel recovery started with 3 processesWed Nov 19 14:32:09 2008Clearing online redo logfile 1 Z:\ORACLE\ORADATA\SATI\REDO01.LOGClearing online log 1 of thread 1 sequence number 300Wed Nov 19 14:32:09 2008Errors in file Z:\oracle\admin\SATI\bdump\SATISTD_mrp0_3540.trc:


28/99

/

ORA-19527: physical standby redo log must be renamedORA-00312: online log 1 thread 1: 'Z:\ORACLE\ORADATA\SATI\REDO01.LOG'

Clearing online redo logfile 1 completeMedia Recovery Waiting for thread 1 sequence 300Wed Nov 19 14:32:10 2008Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSIONWed Nov 19 14:44:57 2008Using STANDBY_ARCHIVE_DEST parameter default value as G:\oracle\oradata\SATI\archiveRedo Shipping Client Connected as PUBLIC-- Connected User is ValidRFS[1]: Assigned to RFS process 4044RFS[1]: Identified database type as 'physical standby'Primary database is in MAXIMUM PERFORMANCE modeChanging standby controlfile to MAXIMUM PERFORMANCE modePrimary thread 1 already marked as open; setting 'closed'Wed Nov 19 14:44:57 2008RFS LogMiner: Client disabled from further notificationRedo Shipping Client Connected as PUBLIC-- Connected User is ValidRFS[2]: Assigned to RFS process 3136RFS[2]: Identified database type as 'physical standby'RFS[2]: Successfully opened standby log 4: 'Z:\ORACLE\ORADATA\SATI\STDBYREDOG4_01.LOG'Wed Nov 19 14:45:06 2008Redo Shipping Client Connected as PUBLIC-- Connected User is ValidRFS[3]: Assigned to RFS process 884RFS[3]: Identified database type as 'physical standby'Primary database is in MAXIMUM PERFORMANCE modePrimary database is in MAXIMUM PERFORMANCE modeRFS[3]: Successfully opened standby log 5: 'Z:\ORACLE\ORADATA\SATI\STDBYREDOG5_01.LOG'Wed Nov 19 14:45:09 2008Fetching gap sequence in thread 1, gap sequence 300-301Wed Nov 19 14:45:10 2008Redo Shipping Client Connected as PUBLIC-- Connected User is ValidWed Nov 19 14:45:10 2008Redo Shipping Client Connected as PUBLIC


29/99

/

Wed Nov 19 14:45:10 2008RFS[4]: Assigned to RFS process 1500Wed Nov 19 14:45:10 2008-- Connected User is ValidWed Nov 19 14:45:10 2008RFS[4]: Identified database type as 'physical standby'Wed Nov 19 14:45:10 2008RFS[5]: Assigned to RFS process 3976RFS[5]: Identified database type as 'physical standby'Wed Nov 19 14:45:10 2008RFS[4]: Archived Log: 'G:\ORACLE\ORADATA\SATI\ARCHIVE\1_300_646399178.ARC'Wed Nov 19 14:45:14 2008RFS[5]: Archived Log: 'G:\ORACLE\ORADATA\SATI\ARCHIVE\1_301_646399178.ARC'Wed Nov 19 14:45:39 2008Media Recovery Log G:\ORACLE\ORADATA\SATI\ARCHIVE\1_300_646399178.ARCMedia Recovery Log G:\ORACLE\ORADATA\SATI\ARCHIVE\1_301_646399178.ARCWed Nov 19 14:45:49 2008Media Recovery Log G:\ORACLE\ORADATA\SATI\ARCHIVE\1_302_646399178.ARCMedia Recovery Waiting for thread 1 sequence 303 (in transit)Wed Nov 19 14:47:36 2008Primary database is in MAXIMUM PERFORMANCE modeRFS[3]: Successfully opened standby log 4: 'Z:\ORACLE\ORADATA\SATI\STDBYREDOG4_01.LOG'Wed Nov 19 14:47:41 2008Media Recovery Log G:\ORACLE\ORADATA\SATI\ARCHIVE\1_303_646399178.ARCMedia Recovery Waiting for thread 1 sequence 304 (in transit)Wed Nov 19 14:50:15 2008Primary database is in MAXIMUM PERFORMANCE modeRFS[3]: Successfully opened standby log 4: 'Z:\ORACLE\ORADATA\SATI\STDBYREDOG4_01.LOG'Wed Nov 19 14:50:16 2008Media Recovery Log G:\ORACLE\ORADATA\SATI\ARCHIVE\1_304_646399178.ARCMedia Recovery Waiting for thread 1 sequence 305 (in transit)

Check redolog apply on the standby

SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;


30/99

/

SEQUENCE# FIRST_TIME NEXT_TIME---------- -------------- --------------

300 16/11/08 17:41 19/11/08 12:58301 19/11/08 12:58 19/11/08 13:03302 19/11/08 13:03 19/11/08 14:45303 19/11/08 14:45 19/11/08 14:48304 19/11/08 14:48 19/11/08 14:50

Switch a couple of sequences on the primary:


SQL> alter system switch logfile;System altered.





SQL> archive log listDatabase log mode Archive ModeAutomatic archival Enabled


31/99

/

Archive destination G:\oracle\oradata\SATI\archiveOldest online log sequence 308Next log sequence to archive 310Current log sequence 310

Check again redolog apply on the standby

SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME FROM V$ARCHIVED_LOG ORDER BY SEQUENCE# ;


300 16/11/08 17:41 19/11/08 12:58301 19/11/08 12:58 19/11/08 13:03302 19/11/08 13:03 19/11/08 14:45303 19/11/08 14:45 19/11/08 14:48304 19/11/08 14:48 19/11/08 14:50305 19/11/08 14:50 19/11/08 14:56306 19/11/08 14:56 19/11/08 14:56307 19/11/08 14:56 19/11/08 14:56308 19/11/08 14:56 19/11/08 14:56309 19/11/08 14:56 19/11/08 14:56

10 rows selected.


The apply process on the standby alert log

-- Connected User is ValidWed Nov 19 14:45:10 2008


32/99

/

RFS[4]: Identified database type as 'physical standby'Wed Nov 19 14:45:10 2008RFS[5]: Assigned to RFS process 3976RFS[5]: Identified database type as 'physical standby'Wed Nov 19 14:45:10 2008RFS[4]: Archived Log: 'G:\ORACLE\ORADATA\SATI\ARCHIVE\1_300_646399178.ARC'Wed Nov 19 14:45:14 2008RFS[5]: Archived Log: 'G:\ORACLE\ORADATA\SATI\ARCHIVE\1_301_646399178.ARC'Wed Nov 19 14:45:39 2008Media Recovery Log G:\ORACLE\ORADATA\SATI\ARCHIVE\1_300_646399178.ARCMedia Recovery Log G:\ORACLE\ORADATA\SATI\ARCHIVE\1_301_646399178.ARCWed Nov 19 14:45:49 2008Media Recovery Log G:\ORACLE\ORADATA\SATI\ARCHIVE\1_302_646399178.ARCMedia Recovery Waiting for thread 1 sequence 303 (in transit)Wed Nov 19 14:47:36 2008Primary database is in MAXIMUM PERFORMANCE modeRFS[3]: Successfully opened standby log 4: 'Z:\ORACLE\ORADATA\SATI\STDBYREDOG4_01.LOG'Wed Nov 19 14:47:41 2008Media Recovery Log G:\ORACLE\ORADATA\SATI\ARCHIVE\1_303_646399178.ARCMedia Recovery Waiting for thread 1 sequence 304 (in transit)Wed Nov 19 14:50:15 2008Primary database is in MAXIMUM PERFORMANCE modeRFS[3]: Successfully opened standby log 4: 'Z:\ORACLE\ORADATA\SATI\STDBYREDOG4_01.LOG'Wed Nov 19 14:50:16 2008Media Recovery Log G:\ORACLE\ORADATA\SATI\ARCHIVE\1_304_646399178.ARCMedia Recovery Waiting for thread 1 sequence 305 (in transit)Wed Nov 19 14:56:17 2008Primary database is in MAXIMUM PERFORMANCE modeRFS[3]: Successfully opened standby log 4: 'Z:\ORACLE\ORADATA\SATI\STDBYREDOG4_01.LOG'Primary database is in MAXIMUM PERFORMANCE modeRFS[3]: Successfully opened standby log 4: 'Z:\ORACLE\ORADATA\SATI\STDBYREDOG4_01.LOG'Primary database is in MAXIMUM PERFORMANCE modeRFS[3]: Successfully opened standby log 4: 'Z:\ORACLE\ORADATA\SATI\STDBYREDOG4_01.LOG'Wed Nov 19 14:56:21 2008Media Recovery Log G:\ORACLE\ORADATA\SATI\ARCHIVE\1_305_646399178.ARCMedia Recovery Log G:\ORACLE\ORADATA\SATI\ARCHIVE\1_306_646399178.ARCMedia Recovery Log G:\ORACLE\ORADATA\SATI\ARCHIVE\1_307_646399178.ARCMedia Recovery Waiting for thread 1 sequence 308 (in transit)


33/99

/

Wed Nov 19 14:56:22 2008Redo Shipping Client Connected as PUBLIC-- Connected User is ValidRFS[6]: Assigned to RFS process 3012RFS[6]: Identified database type as 'physical standby'RFS[6]: Successfully opened standby log 5: 'Z:\ORACLE\ORADATA\SATI\STDBYREDOG5_01.LOG'Wed Nov 19 14:56:23 2008Primary database is in MAXIMUM PERFORMANCE modeRFS[3]: Successfully opened standby log 4: 'Z:\ORACLE\ORADATA\SATI\STDBYREDOG4_01.LOG'Wed Nov 19 14:56:27 2008Media Recovery Log G:\ORACLE\ORADATA\SATI\ARCHIVE\1_308_646399178.ARCMedia Recovery Log G:\ORACLE\ORADATA\SATI\ARCHIVE\1_309_646399178.ARCMedia Recovery Waiting for thread 1 sequence 310 (in transit)

After a while recheck

SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;


300 16/11/08 17:41 19/11/08 12:58301 19/11/08 12:58 19/11/08 13:03302 19/11/08 13:03 19/11/08 14:45303 19/11/08 14:45 19/11/08 14:48304 19/11/08 14:48 19/11/08 14:50305 19/11/08 14:50 19/11/08 14:56306 19/11/08 14:56 19/11/08 14:56307 19/11/08 14:56 19/11/08 14:56308 19/11/08 14:56 19/11/08 14:56309 19/11/08 14:56 19/11/08 14:56310 19/11/08 14:56 19/11/08 15:25311 19/11/08 15:25 19/11/08 15:26312 19/11/08 15:26 19/11/08 15:26313 19/11/08 15:26 19/11/08 15:26314 19/11/08 15:26 19/11/08 15:26315 19/11/08 15:26 19/11/08 15:28316 19/11/08 15:28 19/11/08 15:31


34/99

/

317 19/11/08 15:31 19/11/08 15:40318 19/11/08 15:40 19/11/08 15:41319 19/11/08 15:41 19/11/08 15:41320 19/11/08 15:41 19/11/08 15:41

21 rows selected.

SELECT SEQUENCE#,APPLIED FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;

SEQUENCE# APP---------- ---

300 YES301 YES302 YES303 YES304 YES305 YES306 YES307 YES308 YES309 YES310 YES311 YES312 YES313 YES314 YES315 YES316 YES317 YES318 YES319 YES320 YES

21 rows selected.


35/99

/

Configuring Data Guard Broker

Enable flashback database on primary database

SQL> shutdown immediate;Database closed.Database dismounted.ORACLE instance shut down.SQL> startup mount;ORACLE instance started.

Total System Global Area 1258291200 bytesFixed Size 1292180 bytesVariable Size 637536364 bytesDatabase Buffers 612368384 bytesRedo Buffers 7094272 bytesDatabase mounted.

SQL> select name ,db_unique_name from v$database;

NAME DB_UNIQUE_NAME--------- ------------------------------SATI SATI

SQL> ALTER SYSTEM SET UNDO_RETENTION=3600 SCOPE=SPFILE;

System altered.

SQL> ALTER SYSTEM SET UNDO_MANAGEMENT='AUTO' SCOPE=SPFILE;

System altered.

SQL> ALTER DATABASE FLASHBACK ON;

Database altered.


36/99

/

On The Standby


Total System Global Area 1258291200 bytesFixed Size 1292180 bytesVariable Size 520095852 bytesDatabase Buffers 729808896 bytesRedo Buffers 7094272 bytesDatabase mounted.SQL> select name ,db_unique_name from v$database;

NAME DB_UNIQUE_NAME--------- ------------------------------SATI SATISTD

SQL> ALTER SYSTEM SET UNDO_RETENTION=3600 SCOPE=SPFILE;

System altered.

SQL> ALTER SYSTEM SET UNDO_MANAGEMENT='AUTO' SCOPE=SPFILE;

System altered.

SQL> ALTER DATABASE FLASHBACK ON;

Database altered.

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;

Database altered.


37/99

/

Enable database broker on primary database

The broker configuration files are automatically created when the broker is started using ALTER SYSTEMSET DG_BROKER_START=TRUE.

SQL> ALTER SYSTEM SET DG_BROKER_START=TRUE SCOPE=BOTH;

System altered.

SQL> show parameters dg_broker

NAME TYPE VALUE------------------------------------ ----------- ------------------------------dg_broker_config_file1 string C:\ORACLE\PRODUCT\10.2.0\DB_1\

DATABASE\DR1SATI.DATdg_broker_config_file2 string C:\ORACLE\PRODUCT\10.2.0\DB_1\

DATABASE\DR2SATI.DATdg_broker_start boolean TRUE

Enable database broker on standby database

SQL> ALTER SYSTEM SET DG_BROKER_START=TRUE SCOPE=BOTH;

System altered.

SQL> show parameters dg_broker

NAME TYPE VALUE------------------------------------ ----------- ------------------------------dg_broker_config_file1 string C:\ORACLE\PRODUCT\10.2.0\DB_1\

DATABASE\DR1SATISTD.DATdg_broker_config_file2 string C:\ORACLE\PRODUCT\10.2.0\DB_1\

DATABASE\DR2SATISTD.DATdg_broker_start boolean TRUE


38/99

/

Setup the Local_Listener parameter on the Primary Database

SQL> ALTER SYSTEM SET LOCAL_LISTENER='LISTENER_DGONE' SCOPE=BOTH;

System altered.

SQL> show parameters local_listener

NAME TYPE VALUE------------------------------------ ----------- ------------------------------local_listener string LISTENER_DGONE

Setup the Local_Listener parameter on the Standby Database

SQL> ALTER SYSTEM SET LOCAL_LISTENER='LISTENER_DGTWO' SCOPE =BOTH;

System altered.

SQL> show parameters local_listener;

NAME TYPE VALUE------------------------------------ ----------- ------------------------------local_listener string LISTENER_DGTWO

Setup the Tnsnames.ora and Listener.ora parameter files on the Primary Database

The listener.ora needs to include a service named global_db_name_DGMGRL to enable the brokerto start the databases on the event of switchover. This configuration needs to be includedon both servers. To setup shutdown the listener, make the changes and restart the listener.


39/99

/

Listener.ora


) )

SID_LIST_LISTENER_DGONE =(SID_LIST =(SID_DESC =(SID_NAME = SATI)(GLOBAL_DBNAME=SATI_DGMGRL)(ORACLE_HOME = C:\oracle\product\10.2.0\db_1)

))

Tnsnames.ora


))

SATISTD =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = DGTWO)(PORT = 1522))(CONNECT_DATA =(SERVER = DEDICATED)


40/99

/

(SERVICE_NAME = SATISTD_DGMGRL))

)

Check

C:\Documents and Settings\AV>lsnrctl status LISTENER_DGONE

LSNRCTL for 32-bit Windows: Version 10.2.0.4.0 - Production on 14-DEC-2008 10:19:22

Copyright (c) 1991, 2007, Oracle. All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=DGONE)(PORT=1522)))STATUS of the LISTENER------------------------Alias LISTENER_DGONEVersion TNSLSNR for 32-bit Windows: Version 10.2.0.4.0 - ProductionStart Date 08-DEC-2008 18:22:44Uptime 5 days 15 hr. 56 min. 42 secTrace Level offSecurity ON: Local OS AuthenticationSNMP OFFListener Parameter File C:\oracle\product\10.2.0\db_1\network\admin\listener.oraListener Log File C:\oracle\product\10.2.0\db_1\network\log\listener_DGONE.logListening Endpoints Summary...(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=DGONE)(PORT=1522)))

Services Summary...Service "SATI" has 1 instance(s).Instance "SATI", status READY, has 1 handler(s) for this service...

Service "SATI_DGB" has 1 instance(s).Instance "SATI", status READY, has 1 handler(s) for this service...

Service "SATI_DGMGRL" has 1 instance(s).Instance "SATI", status UNKNOWN, has 1 handler(s) for this service...

Service "SATI_XPT" has 1 instance(s).Instance "SATI", status READY, has 1 handler(s) for this service...

The command completed successfully

C:\Documents and Settings\AV>TNSPING SATI

TNS Ping Utility for 32-bit Windows: Version 10.2.0.4.0 - Production on 14-DEC-2008 10:20:13


41/99

/


Used parameter files:C:\oracle\product\10.2.0\db_1\network\admin\sqlnet.ora

Used TNSNAMES adapter to resolve the aliasAttempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = DGONE)(PORT = 1522)) (CONNECT_DATA =(SERVER= DEDICATED) (SERVICE_NAME = SATI_DGMGRL)))OK (20 msec)

C:\Documents and Settings\AV>TNSPING SATISTD




Used TNSNAMES adapter to resolve the aliasAttempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = DGTWO)(PORT = 1522)) (CONNECT_DATA =(SERVER= DEDICATED) (SERVICE_NAME = SATISTD_DGMGRL)))OK (0 msec)

C:\Documents and Settings\AV>SQLPLUS sys@SATI as sysdba



Enter password:


C:\Documents and Settings\AV>SQLPLUS sys@SATISTD as sysdba


42/99

/



Enter password:


Setup the Tnsmanes.ora and Listener.ora parameter files on the Standby Database

The listener.ora needs to include a service named global_db_name_DGMGRL to enable the brokerto start the databases on the event of switchover. This configuration needs to be includedon both servers. To setup shutdown the listener, make the changes and restart the listener.

Listener.ora

LISTENER_DGTWO =(DESCRIPTION_LIST =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = DGTWO)(PORT = 1522))

) )


))


43/99

/

Tnsnames.ora


))

SATISTD =(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = DGTWO)(PORT = 1522))(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = SATISTD_DGMGRL)

))

Check

C:\Documents and Settings\AV>lsnrctl status LISTENER_DGTWO

LSNRCTL for 32-bit Windows: Version 10.2.0.4.0 - Production on 14-DEC-2008 10:40:21


Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=DGTWO)(PORT=1522)))STATUS of the LISTENER------------------------Alias listener_DGTWOVersion TNSLSNR for 32-bit Windows: Version 10.2.0.4.0 - ProductionStart Date 08-DEC-2008 18:22:36Uptime 5 days 16 hr. 17 min. 46 secTrace Level off


44/99

/

Security ON: Local OS AuthenticationSNMP OFFListener Parameter File C:\oracle\product\10.2.0\db_1\network\admin\listener.oraListener Log File C:\oracle\product\10.2.0\db_1\network\log\listener_DGTWO.logListening Endpoints Summary...(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=DGTWO)(PORT=1522)))

Services Summary...Service "SATISTD" has 1 instance(s).Instance "SATISTD", status READY, has 1 handler(s) for this service...

Service "SATISTD_DGB" has 1 instance(s).Instance "SATISTD", status READY, has 1 handler(s) for this service...

Service "SATISTD_DGMGRL" has 1 instance(s).Instance "SATISTD", status UNKNOWN, has 1 handler(s) for this service...

Service "SATISTD_XPT" has 1 instance(s).Instance "SATISTD", status READY, has 1 handler(s) for this service...

The command completed successfully

C:\Documents and Settings\AV>TNSPING SATI




Used TNSNAMES adapter to resolve the aliasAttempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = DGONE)(PORT = 1522))(CONNECT_DATA = (SERVER= DEDICATED) (SERVICE_NAME = SATI_DGMGRL)))OK (0 msec)

C:\Documents and Settings\AV>TNSPING SATISTD




45/99

/


Used TNSNAMES adapter to resolve the aliasAttempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = DGTWO)(PORT = 1522))(CONNECT_DATA = (SERVER= DEDICATED) (SERVICE_NAME = SATISTD_DGMGRL)))OK (30 msec)

C:\Documents and Settings\AV>SQLPLUS SYS@SATI AS SYSDBA



Enter password:


C:\Documents and Settings\AV>SQLPLUS SYS@SATISTD AS SYSDBA



Enter password:



46/99

/

Create the DataGuard Broker Configuration

From within the Broker Command Line interface, DGMGRL create the configuration

C:\oracle\product\agent10g\BIN>dgmgrlDGMGRL for 32-bit Windows: Version 10.2.0.3.0 - Production


Welcome to DGMGRL, type "help" for information.DGMGRL> help

The following commands are availablZ:

add Add a standby database to the broker configurationconnect Connect to an Oracle instancecreate Create a broker configurationdisable Disable a configuration, a database, or Fast-Start Failoveredit Edit a configuration, database, or instanceenable Enable a configuration, a database, or Fast-Start Failoverexit Exit the programfailover Change a standby database to be the primary databasehelp Display description and syntax for a commandquit Exit the programreinstate Change a disabled database into a viable standby databaserem Comment to be ignored by DGMGRLremove Remove a configuration, database, or instanceshow Display information about a configuration, database, or instanceshutdown Shutdown a currently running Oracle instancestart Start Fast-Start Failover observerstartup Start an Oracle database instancestop Stop Fast-Start Failover observerswitchover Switch roles between the primary database and a standby database

Use "help <command>" to see syntax for individual commands

DGMGRL> CONNECT SYS/XXXXXX@SATI


47/99

/

Connected.

DGMGRL> CREATE CONFIGURATION fsfodg AS> PRIMARY DATABASE IS SATI> CONNECT IDENTIFIER IS SATI;Configuration "fsfodg" created with primary database "SATI"

DGMGRL> ADD DATABASE SATISTD AS> CONNECT IDENTIFIER IS SATISTD> MAINTAINED AS PHYSICAL;Database "SATISTD" added

DGMGRL> show configuration;

ConfigurationNamZ: fsfodgEnabled: NOProtection ModZ: MaxPerformanceFast-Start Failover: DISABLEDDatabases:SATI - Primary databaseSATISTD - Physical standby database

Current status for "SATI":DISABLED

DGMGRL> SHOW DATABASE VERBOSE SATISTD;

DatabaseNamZ: SATISTDRolZ: PHYSICAL STANDBYEnabled: NOIntended StatZ: ONLINEInstance(s):SATISTD

Properties:


48/99

/

InitialConnectIdentifier = 'SATISTD'LogXptMode = 'SYNC'Dependency = ''DelayMins = '0'Binding = 'OPTIONAL'MaxFailure = '0'MaxConnections = '1'ReopenSecs = '300'NetTimeout = '180'LogShipping = 'ON'PreferredApplyInstance = ''ApplyInstanceTimeout = '0'ApplyParallel = 'AUTO'StandbyFileManagement = 'auto'ArchiveLagTarget = '0'LogArchiveMaxProcesses = '30'LogArchiveMinSucceedDest = '1'DbFileNameConvert = ''LogFileNameConvert = ''FastStartFailoverTarget = 'SATI'StatusReport = '(monitor)'InconsistentProperties = '(monitor)'InconsistentLogXptProps = '(monitor)'SendQEntries = '(monitor)'LogXptStatus = '(monitor)'RecvQEntries = '(monitor)'HostName = 'DGTWO'SidName = 'SATISTD'LocalListenerAddress = '(ADDRESS=(PROTOCOL=TCP)(HOST=DGTWO)(PORT=1522))'StandbyArchiveLocation = 'G:\ORACLE\ORADATA\SATI\ARCHIVE'AlternateLocation = ''LogArchiveTrace = '0'LogArchiveFormat = '%t_%s_%r.arc'LatestLog = '(monitor)'TopWaitEvents = '(monitor)'

Current status for "SATISTD":DISABLED


49/99

/

DGMGRL> SHOW DATABASE VERBOSE SATI;

DatabaseNamZ: SATIRolZ: PRIMARYEnabled: NOIntended StatZ: ONLINEInstance(s):SATI

Properties:InitialConnectIdentifier = 'SATI'LogXptMode = 'SYNC'Dependency = ''DelayMins = '0'Binding = 'OPTIONAL'MaxFailure = '0'MaxConnections = '1'ReopenSecs = '300'NetTimeout = '180'LogShipping = 'ON'PreferredApplyInstance = ''ApplyInstanceTimeout = '0'ApplyParallel = 'AUTO'StandbyFileManagement = 'auto'ArchiveLagTarget = '0'LogArchiveMaxProcesses = '30'LogArchiveMinSucceedDest = '1'DbFileNameConvert = ''LogFileNameConvert = ''FastStartFailoverTarget = 'SATISTD'StatusReport = '(monitor)'InconsistentProperties = '(monitor)'InconsistentLogXptProps = '(monitor)'SendQEntries = '(monitor)'LogXptStatus = '(monitor)'RecvQEntries = '(monitor)'


50/99

/

HostName = 'DGONE'SidName = 'SATI'LocalListenerAddress = '(ADDRESS=(PROTOCOL=TCP)(HOST=DGONE)(PORT=1522))'StandbyArchiveLocation = 'G:\oracle\oradata\SATI\archive'AlternateLocation = ''LogArchiveTrace = '0'LogArchiveFormat = '%t_%s_%r.arc'LatestLog = '(monitor)'TopWaitEvents = '(monitor)'

Current status for "SATI":DISABLED

Enable the Broker Configuration

DGMGRL> ENABLE CONFIGURATION;Enabled.DGMGRL> show configuration;

ConfigurationNamZ: fsfodgEnabled: YESProtection ModZ: MaxPerformanceFast-Start Failover: DISABLEDDatabases:SATI - Primary databaseSATISTD - Physical standby database

Current status for "fsfodg":SUCCESS

DGMGRL> ENABLE DATABASE SATISTD;Enabled.



51/99

/

ConfigurationNamZ: fsfodgEnabled: YESProtection ModZ: MaxPerformanceFast-Start Failover: DISABLEDDatabases:SATI - Primary databaseSATISTD - Physical standby database


Configure Data Guard Broker For Switchover

First Check that the configuration is healthy, show database must return success

DGMGRL> show database SATI;

DatabaseNamZ: SATIRolZ: PRIMARYEnabled: YESIntended StatZ: ONLINEInstance(s):SATI

Current status for "SATI":SUCCESS

DGMGRL> show database SATISTD;

DatabaseNamZ: SATISTDRolZ: PHYSICAL STANDBYEnabled: YES


52/99

/

Intended StatZ: ONLINEInstance(s):SATISTD

Current status for "SATISTD":SUCCESS

Set the “LogXptMode” Property to SYNC

DGMGRL> EDIT DATABASE SATI SET PROPERTY 'LogXptMode'='SYNC';Property "LogXptMode" updatedDGMGRL> EDIT DATABASE SATISTD SET PROPERTY 'LogXptMode'='SYNC';Property "LogXptMode" updated

Set the “FastStartFailoverTarget” Property for Both the Primary and Physical Databases

DGMGRL> EDIT DATABASE SATI SET PROPERTY FastStartFailoverTarget='SATISTD';Property "faststartfailovertarget" updatedDGMGRL> EDIT DATABASE SATISTD SET PROPERTY FastStartFailoverTarget='SATI';Property "faststartfailovertarget" updated

Change the Protection Mode to Maximum Availability

DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS MAXAVAILABILITY;Succeeded.

Start The Observer to Enable Fast Start Failover

Note that the observer process needs to be run on the background of a third productions server allthe time in order to provide automatic fast start failover.

C:\oracle\product\agent10g\BIN>dgmgrlDGMGRL for 32-bit Windows: Version 10.2.0.3.0 - Production



53/99

/

Welcome to DGMGRL, type "help" for information.DGMGRL> connect sys/[email protected]> start observer;Observer started

Enable Fast Start Failover

DGMGRL> show configuration;ConfigurationNamZ: fsfodgEnabled: YESProtection ModZ: MaxAvailability Fast-Start Failover: DISABLEDDatabases:SATI - Primary databaseSATISTD - Physical standby database


DGMGRL> ENABLE FAST_START FAILOVER;Enabled.DGMGRL> show configuration;

ConfigurationNamZ: fsfodgEnabled: YESProtection ModZ: MaxAvailability Fast-Start Failover: ENABLEDDatabases:SATI - Primary databaseSATISTD - Physical standby database

- Fast-Start Failover target

Current status for "fsfodg":


54/99

/

SUCCESS

Check Status On Primary and Standby databases

SQL> select DB_UNIQUE_NAME,DATABASE_ROLE,OPEN_MODE,FS_FAILOVER_STATUS,FS_FAILOVER_CURRENT_TARGET fromv$database;

DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE FS_FAILOVER_STATUS FS_FAILOVER_CURRENT_TARGET--------------- ---------------- ---------- --------------------- ------------------------------SATI PRIMARY READ WRITE SYNCHRONIZED SATISTD

select DB_UNIQUE_NAME,DATABASE_ROLE,OPEN_MODE,FS_FAILOVER_STATUS,FS_FAILOVER_CURRENT_TARGET fromv$database;

DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE FS_FAILOVER_STATUS FS_FAILOVER_CURRENT_TARGET--------------- ---------------- ---------- --------------------- ------------------------------SATISTD PHYSICAL STANDBY MOUNTED SYNCHRONIZED SATISTD

Test Switchover

Switchover is used for maintenance on the primary, downtime is required to switch roles between primaryand standby databases.

Before starting a switchover is convenient to check that the configuration and the managed databases arehealthy

DGMGRL> show configuration

ConfigurationNamZ: fsfodgEnabled: YESProtection ModZ: MaxAvailabilityFast-Start Failover: ENABLEDDatabases:

SATISTD - Physical standby database- Fast-Start Failover target

SATI - Primary database


55/99

/


DGMGRL> show database SATI

DatabaseNamZ: SATIRolZ: PRIMARYEnabled: YESIntended StatZ: ONLINEInstance(s):

SATI


DGMGRL> show database SATISTD

DatabaseNamZ: SATISTDRolZ: PHYSICAL STANDBYEnabled: YESIntended StatZ: ONLINEInstance(s):

SATISTD


DGMGRL> switchover to SATISTDPerforming switchover NOW, please wait...Operation requires shutdown of instance "SATI" on database "SATI"Shutting down instance "SATI"...ORA-01109: database not open

Database dismounted.ORACLE instance shut down.Operation requires shutdown of instance "SATISTD" on database "SATISTD"Shutting down instance "SATISTD"...ORA-01109: database not open

Database dismounted.


56/99

/

ORACLE instance shut down.Operation requires startup of instance "SATI" on database "SATI"Starting instance "SATI"...ORACLE instance started.Database mounted.Operation requires startup of instance "SATISTD" on database "SATISTD"Starting instance "SATISTD"...ORACLE instance started.Database mounted.Switchover succeeded, new primary is "SATISTD"

After a successful switchover the primary and the standby needs to be synchronized

Execute this sql statement on both databases:

select DB_UNIQUE_NAME, DATABASE_ROLE, OPEN_MODE, FS_FAILOVER_STATUS, FS_FAILOVER_CURRENT_TARGET

from v$database;

DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE FS_FAILOVER_STATUS FS_FAILOVER_CURRENT_TARGET--------------- ---------------- ---------- --------------------- --------------------------SATISTD PRIMARY READ WRITE SYNCHRONIZED SATI

DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE FS_FAILOVER_STATUS FS_FAILOVER_CURRENT_TARGET--------------- ---------------- ---------- --------------------- --------------------------SATI PHYSICAL STANDBY MOUNTED SYNCHRONIZED SATI

Configuring the Observer ProcessThe Observer process will be in charge of monitoring the Primary and Standby databases; in case of Primary sitefailure, after checking that the Standby site is ready for the role change it will trigger the failover.


57/99

/

Once the failed Primary site became available again the Broker will convert it to the standby site and will resumenormal activity.The observer process needs to be run on a productions server that will provide assurance of continuousavailability. A watchdog process needs to be put in place to restart the observer immediately in the event that isstopped.

On most productions systems it is possible to define a script that will both check and restart the process in case offailure.

#!/bin/ksh# startobserverexport ORACLE_BASE=/vmractst3/app01/oracleexport ORACLE_HOME=/vmractst3/app01/oracle/product/10.2exportBASE_PATH=/vmractst3/app01/oracle/scripts/general:/opt/CTEact/bin:/usr/local/sbi n:/usr/local/bin:/sbin:/bin:/usr/bin:/etc:/usr/local/maint/oraclZ:/usr/ccs/bin:/ usr/openwin/bin:/usr/dt/bin:/usr/local/bin:.export PATH=$ORACLE_HOME/bin:$BASE_PATHdgmgrl << eofconnect sys/oracle@whiteowlSTOP OBSERVER;START OBSERVER;eof

On windows a script can be used to start the Observer process at server startup using the Windows Scheduler.

The same script can be invoked in case that the observer process crash to restart the Observer

set ORACLE_HOME=C:\oracle\product\10.2.0\db_1call dgmgrl -silent sys/xxxxxx@SATI "stop observer"


58/99

/

call dgmgrl -silent sys/xxxxxx@SATI "start observer"

Fast Start Failover Test

The FSFO can be tested by either shutting down the server where the primary database run or by shutting downthe database with the abort option.

The progress and efficiency of the FSFO can be monitored by doing tail to the Primary and Standby databasesalert logs. The whole process is also reflected on the Observer window if running on foreground.

These are the outputs registered during a FSFO test:

Observer Output During FSFO

On the observer output we see the following output:

1. At 17:36:32 the Observer detected a failure on the Primary database SATI and initiated failoverto the Standby database SATISTD

2. At 17:37:19 SATISTD is up and running as the Primary, total downtime was 47 seconds.3. At 17:38:33 the Observer detected that the failed database SATI was mounted and initiates the

reinstatement, the database is shut down to perform flashback and recovery until it issynchronized with the new Primary

4. at 17:40:01 SATI is up and synchronized

Database Monitoring During FSFO

Status of the databases before the Failure at 17:36:00


59/99

/

Primary Database SATIDB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE FS_FAILOVER_STATUS FS_FAILOVER_CURRENT_TARGET------------------------------ ---------------- ---------- --------------------- ------------------------------SATI PRIMARY READ WRITE SYNCHRONIZED SATISTD

Standby Database SATISTD

DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE FS_FAILOVER_STATUS FS_FAILOVER_CURRENT_TARGET------------------------------ ---------------- ---------- --------------------- ------------------------------SATISTD PHYSICAL STANDBY MOUNTED SYNCHRONIZED SATISTD

Primary Database SATI is ShutdownSQL> shutdown abortORACLE instance shut down.

Standby Database SATISTD is Converted to Primary

The new primary is up and ready in seconds, the new FSFO target is the failed database SATI and its status is“Reinstate Required”

DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE FS_FAILOVER_STATUS FS_FAILOVER_CURRENT_TARGET------------------------------ ---------------- ---------- ------------------ -------------------------------SATISTD PRIMARY READ WRITE REINSTATE REQUIRED SATI

The Failed Database is Mounted SQL> connect / as sysdbaConnected to an idle instance.SQL> startup mount;ORACLE instance started.

Total System Global Area 1258291200 bytesFixed Size 1298304 bytesVariable Size 494928000 bytes


60/99

/

Database Buffers 754974720 bytesRedo Buffers 7090176 bytesDatabase mounted.SQL> @chkstat

Immediately after the database is mounted the broker shuts it down and initiates reinstate, that’s why we got ora-3113 when trying to check status:

select DB_UNIQUE_NAME,DATABASE_ROLE,OPEN_MODE,FS_FAILOVER_STATUS,FS_FAILOVER_CURRENT_TARGET from v$database*ERROR at line 1:ORA-03113: end-of-file on communication channel

Once the database is mounted by the broker it is flashed back and synchronized with the new primary

SQL> connect / as sysdbaConnected.SQL> @chkstat

DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE FS_FAILOVER_STATUS FS_FAILOVER_CURRENT_TARGET------------------------------ ---------------- ---------- --------------------- ------------------------------SATI PHYSICAL STANDBY MOUNTED SYNCHRONIZED SATI

Database Monitoring During FSFO Using the Alert Logs

The alert logs provide the best information to monitor both the FSFO and the Reinstate processes; if available youcan use the command “tail -f <alert.log>” to monitor the progress.

This is a summary of the commands executed by the broker at both databases during the FSFO and the Reinstateprocesses.

SATISTD alert log (Standby converted to Primary)


61/99

/

Wed Dec 10 17:39:04 2008RFS[4]: Possible network disconnect with primary databaseWed Dec 10 17:39:07 2008RFS[3]: Possible network disconnect with primary databaseWed Dec 10 17:39:07 2008RFS[5]: Possible network disconnect with primary databaseWed Dec 10 17:39:34 2008Attempting fast-start failover because the threshold of 30 seconds has elapsed.Wed Dec 10 17:39:34 2008Initiated fast-start failover to database SATISTD.Wed Dec 10 17:39:35 2008ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH FORCEWed Dec 10 17:39:35 2008Terminal Recovery: Stopping real time applyWed Dec 10 17:39:36 2008MRP0: Background Media Recovery cancelled with status 16037Wed Dec 10 17:39:36 2008Errors in file Z:\oracle\admin\SATI\bdump\SATISTD_mrp0_2428.trc:ORA-16037: user requested cancel of managed recovery operation

Managed Standby Recovery not using Real Time ApplyRecovery interrupted!Recovered data files to a consistent state at change 4134609708Wed Dec 10 17:39:40 2008Errors in file Z:\oracle\admin\SATI\bdump\SATISTD_mrp0_2428.trc:ORA-16037: user requested cancel of managed recovery operation

Wed Dec 10 17:39:41 2008Terminal Recovery: Stopped real time applyManaged Standby Recovery not using Real Time Applyparallel recovery started with 3 processesTerminal Recovery timestamp is '12/10/2008 17:39:48'Terminal Recovery: applying standby redo logs.Terminal Recovery: thread 1 seq# 721 redo requiredTerminal Recovery: Wed Dec 10 17:39:48 2008Recovery of Online Redo Log: Thread 1 Group 5 Seq 721 Reading mem 0Mem# 0: Z:\ORACLE\ORADATA\SATI\STDBYREDOG5_01.LOG


62/99

/

Mem# 1: X:\ORACLE\ORADATA\SATI\STDBYREDOG5_02.LOGIdentified End-Of-Redo for thread 1 sequence 721Terminal Recovery: Updated next available block for thread 1 sequence 721 lno 5 to value 74605Wed Dec 10 17:39:49 2008Incomplete recovery applied all redo ever generated.Recovery completed through change 4134609709Terminal Recovery: successful completionBegin: Standby Redo Logfile archivalEnd: Standby Redo Logfile archivalResetting standby activation ID 1481388743 (0x584c32c7)Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH FORCEWed Dec 10 17:39:57 2008ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WAIT WITH SESSION SHUTDOWNIf media recovery active, switchover will wait 900 secondsStandby terminal recovery start SCN: 4134609708RESETLOGS after complete recovery through change 4134609709Online log Z:\ORACLE\ORADATA\SATI\REDO01.LOG: Thread 1 Group 1 was previously clearedOnline log X:\ORACLE\ORADATA\SATI\REDO02.LOG: Thread 1 Group 1 was previously clearedOnline log Z:\ORACLE\ORADATA\SATI\REDO11.LOG: Thread 1 Group 2 was previously clearedOnline log X:\ORACLE\ORADATA\SATI\REDO12.LOG: Thread 1 Group 2 was previously clearedOnline log Z:\ORACLE\ORADATA\SATI\REDO21.LOG: Thread 1 Group 3 was previously clearedOnline log X:\ORACLE\ORADATA\SATI\REDO22.LOG: Thread 1 Group 3 was previously clearedStandby became primary SCN: 4134609707Wed Dec 10 17:39:58 2008Setting recovery target incarnation to 3Wed Dec 10 17:39:58 2008Converting standby mount to primary mount.Completed: ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WAIT WITH SESSION SHUTDOWNWed Dec 10 17:39:58 2008ARC0: STARTING ARCH PROCESSESARC2: Archival startedARC2 started with pid=20, OS id=2744Wed Dec 10 17:39:58 2008ARC0: STARTING ARCH PROCESSES COMPLETEARC0: Becoming the 'no SRL' ARCHWed Dec 10 17:40:00 2008ALTER SYSTEM SET standby_archive_dest='' SCOPE=BOTH SID='SATISTD';Wed Dec 10 17:40:00 2008


63/99

/

ALTER SYSTEM SETlog_archive_dest_1='location="G:\oracle\oradata\SATI\archive"','valid_for=(ONLINE_LOGFILE,ALL_ROLES)'SCOPE=BOTH SID='SATISTD';Wed Dec 10 17:40:00 2008ALTER SYSTEM SET log_archive_dest_state_1='ENABLE' SCOPE=BOTH SID='SATISTD';Wed Dec 10 17:40:00 2008ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE AVAILABILITYWed Dec 10 17:40:00 2008Completed: ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE AVAILABILITYWed Dec 10 17:40:00 2008ALTER DATABASE OPENWed Dec 10 17:40:01 2008Assigning activation ID 1481384601 (0x584c2299)LGWR: Primary database is in MAXIMUM AVAILABILITY modeLGWR: Destination LOG_ARCHIVE_DEST_1 is not serviced by LGWRThread 1 opened at log sequence 1Current log# 1 seq# 1 mem# 0: Z:\ORACLE\ORADATA\SATI\REDO01.LOGCurrent log# 1 seq# 1 mem# 1: X:\ORACLE\ORADATA\SATI\REDO02.LOG

Successful open of redo thread 1Wed Dec 10 17:40:01 2008SMON: enabling cache recoveryWed Dec 10 17:40:04 2008Successfully onlined Undo Tablespace 1.Dictionary check beginningDictionary check completeStarting control autobackupControl autobackup written to DISK device

handle 'G:\ORACLE_BACKUP\SATI\RMAN\DB\SATI_CONTROL_SATIXXX_C-1342263826-20081210-00'Wed Dec 10 17:40:06 2008SMON: enabling tx recoveryWed Dec 10 17:40:06 2008*********************************************************************WARNING: The following temporary tablespaces contain no files.

This condition can occur when a backup controlfile hasbeen restored. It may be necessary to add files to thesetablespaces. That can be done using the SQL statement:

ALTER TABLESPACE <tablespace_name> ADD TEMPFILE


64/99

/

Alternatively, if these temporary tablespaces are no longerneeded, then they can be dropped.Empty temporary tablespacZ: TEMP_TOOLS

*********************************************************************Database Characterset is IW8ISO8859P8Opening with internal Resource Manager planwhere NUMA PG = 1, CPUs = 4replication_dependency_tracking turned off (no async multimaster replication found)Starting background process QMNCQMNC started with pid=22, OS id=2912Wed Dec 10 17:40:08 2008LOGSTDBY: Validating controlfile with logical metadataWed Dec 10 17:40:08 2008LOGSTDBY: Validation completeCompleted: ALTER DATABASE OPENWed Dec 10 17:40:13 2008ALTER SYSTEM SET standby_archive_dest='' SCOPE=BOTH SID='SATISTD';Wed Dec 10 17:40:13 2008ALTER SYSTEM SET log_archive_trace=0 SCOPE=BOTH SID='SATISTD';Wed Dec 10 17:40:13 2008ALTER SYSTEM SET log_archive_format='%t_%s_%r.arc' SCOPE=SPFILE SID='SATISTD';Wed Dec 10 17:40:14 2008ALTER SYSTEM SET standby_file_management='auto' SCOPE=BOTH SID='*';Wed Dec 10 17:40:14 2008ALTER SYSTEM SET archive_lag_target=0 SCOPE=BOTH SID='*';Wed Dec 10 17:40:14 2008ALTER SYSTEM SET log_archive_max_processes=2 SCOPE=BOTH SID='*';Wed Dec 10 17:40:14 2008ALTER SYSTEM SET log_archive_min_succeed_dest=1 SCOPE=BOTH SID='*';FSFP started with pid=27, OS id=460Wed Dec 10 17:40:20 2008Failover succeeded. Primary database is now SATISTD.Wed Dec 10 17:40:50 2008Shutting down archive processesWed Dec 10 17:40:55 2008ARCH shutting downARC2: Archival stopped


65/99

/

Wed Dec 10 17:41:35 2008ALTER SYSTEM SETlog_archive_dest_2='service="(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=DGONE)(PORT=1522) ))(CONNECT_DATA=(SERVICE_NAME=SATI_XPT)(INSTANCE_NAME=SATI)(SERVER=dedicated)))"',' LGWR SYNC AFFIRMdelay=0 OPTIONAL max_failure=0 max_connections=1 reopen=1 db_unique_name="SATI" registernet_timeout=180 valid_for=(online_logfile,primary_role)' SCOPE=BOTH;Wed Dec 10 17:41:35 2008ALTER SYSTEM SET log_archive_dest_state_2='RESET' SCOPE=BOTH;Wed Dec 10 17:42:23 2008ALTER SYSTEM SETlog_archive_dest_2='service="(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=DGONE)(PORT=1522) ))(CONNECT_DATA=(SERVICE_NAME=SATI_XPT)(INSTANCE_NAME=SATI)(SERVER=dedicated)))"',' LGWR SYNC AFFIRMdelay=0 OPTIONAL max_failure=0 max_connections=1 reopen=1 db_unique_name="SATI" registernet_timeout=180 valid_for=(online_logfile,primary_role)' SCOPE=BOTH;Wed Dec 10 17:42:23 2008ALTER SYSTEM SET log_archive_dest_state_2='RESET' SCOPE=BOTH;Wed Dec 10 17:42:47 2008ALTER SYSTEM SETlog_archive_dest_2='service="(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=DGONE)(PORT=1522) ))(CONNECT_DATA=(SERVICE_NAME=SATI_XPT)(INSTANCE_NAME=SATI)(SERVER=dedicated)))"',' LGWR SYNC AFFIRMdelay=0 OPTIONAL max_failure=0 max_connections=1 reopen=300 db_unique_name="SATI" registernet_timeout=180 valid_for=(online_logfile,primary_role)' SCOPE=BOTH;Wed Dec 10 17:42:47 2008ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=BOTH;Wed Dec 10 17:42:47 2008ALTER SYSTEM ARCHIVE LOGWed Dec 10 17:42:47 2008Destination LOG_ARCHIVE_DEST_2 is UNSYNCHRONIZED******************************************************************LGWR: Setting 'active' archival for destination LOG_ARCHIVE_DEST_2******************************************************************LNSb started with pid=31, OS id=388Wed Dec 10 17:42:55 2008LGWR: Standby redo logfile selected to archive thread 1 sequence 2LGWR: Standby redo logfile selected for thread 1 sequence 2 for destination LOG_ARCHIVE_DEST_2Wed Dec 10 17:42:55 2008Thread 1 advanced to log sequence 2 (LGWR switch)Current log# 2 seq# 2 mem# 0: Z:\ORACLE\ORADATA\SATI\REDO11.LOG


66/99

/

Current log# 2 seq# 2 mem# 1: X:\ORACLE\ORADATA\SATI\REDO12.LOGWed Dec 10 17:42:55 2008ARCH: LGWR is actively archiving destination LOG_ARCHIVE_DEST_2ARCH: Standby redo logfile selected for thread 1 sequence 1 for destination LOG_ARCHIVE_DEST_2Wed Dec 10 17:44:05 2008Destination LOG_ARCHIVE_DEST_2 is SYNCHRONIZEDLGWR: Standby redo logfile selected to archive thread 1 sequence 3LGWR: Standby redo logfile selected for thread 1 sequence 3 for destination LOG_ARCHIVE_DEST_2Wed Dec 10 17:44:05 2008Thread 1 advanced to log sequence 3 (LGWR switch)Current log# 3 seq# 3 mem# 0: Z:\ORACLE\ORADATA\SATI\REDO21.LOGCurrent log# 3 seq# 3 mem# 1: X:\ORACLE\ORADATA\SATI\REDO22.LOG

SATI alert log (Failed Primary Reinstated and converted to Standby)

Wed Dec 10 17:39:42 2008Shutting down instance (abort)License high water mark = 7Instance terminated by USER, pid = 3416Wed Dec 10 17:41:51 2008Starting ORACLE instance (normal)LICENSE_MAX_SESSION = 0LICENSE_SESSIONS_WARNING = 0Picked latch-free SCN scheme 2Autotune of undo retention is turned on. IMODE=BRILAT =55LICENSE_MAX_USERS = 0SYS auditing is disabledksdpec: called for event 13740 prior to event group initializationStarting up ORACLE RDBMS Version: 10.2.0.4.0.System parameters with non-default values:processes = 150sessions = 500timed_statistics = TRUEsga_max_size = 1258291200


67/99

/

__shared_pool_size = 209715200shared_pool_size = 0__large_pool_size = 16777216large_pool_size = 0__java_pool_size = 8388608java_pool_size = 0__streams_pool_size = 50331648streams_pool_size = 50331648sga_target = 1048576000control_files = Z:\ORACLE\ORADATA\SATI\CONTROL01.CTL,

X:\ORACLE\ORADATA\SATI\CONTROL02.CTL, G:\ORACLE\ORADATA\SATI\CONTROL03.CTLdb_block_size = 8192__db_cache_size = 754974720db_cache_size = 0compatible = 10.2.0.4.0log_archive_config = DG_CONFIG=(SATI,SATISTD)log_archive_dest_1 = location="G:\oracle\oradata\SATI\archive",

valid_for=(ONLINE_LOGFILE,ALL_ROLES)log_archive_dest_2 =

service="(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=DGTWO)(PORT=1522)))(CONNECT_DATA=(SER VICE_NAME=SATISTD_XPT)(INSTANCE_NAME=SATISTD)(SERVER=dedicated)))", LGWR SYNC AFFIRM delay=0OPTIONAL max_failure=0 max_connections=1 reopen=300 db_unique_name="SATISTD" registernet_timeout=180 valid_for=(online_logfile,primary_role)log_archive_dest_state_1 = ENABLElog_archive_dest_state_2 = ENABLElog_archive_max_processes= 2log_archive_min_succeed_dest= 1standby_archive_dest = log_archive_trace = 0log_archive_format = %t_%s_%r.arcfal_client =

(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=DGONE)(PORT=1522)))(CONNECT_DATA=(SERVICE_NAME =SATI_XPT)(INSTANCE_NAME=SATI)(SERVER=dedicated)))fal_server =

(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=DGTWO)(PORT=1522)))(CONNECT_DATA=(SERVICE_NAME =SATISTD_XPT)(SERVER=dedicated)))archive_lag_target = 0db_files = 1024


68/99

/

db_file_multiblock_read_count= 16db_recovery_file_dest = G:\FRAdb_recovery_file_dest_size= 26843545600standby_file_management = AUTOfast_start_mttr_target = 300undo_management = AUTOundo_tablespace = UNDOTBS1undo_retention = 3600remote_os_authent = TRUEremote_login_passwordfile= EXCLUSIVEdb_domain = instance_name = SATIlocal_listener = (ADDRESS = (PROTOCOL = TCP)(HOST = DGONE)(PORT = 1522))utl_file_dir = Z:\oracle\admin\SATI\bdumpjob_queue_processes = 50background_dump_dest = Z:\ORACLE\ADMIN\SATI\BDUMPuser_dump_dest = Z:\ORACLE\ADMIN\SATI\UDUMPcore_dump_dest = Z:\ORACLE\ADMIN\SATI\CDUMPsession_max_open_files = 20db_name = SATIdb_unique_name = SATIopen_cursors = 1000star_transformation_enabled= TRUEquery_rewrite_enabled = TRUEpga_aggregate_target = 524288000workarea_size_policy = AUTOdg_broker_start = TRUE

PMON started with pid=2, OS id=488PSP0 started with pid=3, OS id=548MMAN started with pid=4, OS id=3920DBW0 started with pid=5, OS id=2408LGWR started with pid=6, OS id=2440CKPT started with pid=7, OS id=2756SMON started with pid=8, OS id=3620RECO started with pid=9, OS id=3128CJQ0 started with pid=10, OS id=3800MMON started with pid=11, OS id=1144MMNL started with pid=12, OS id=3068


69/99

/

DMON started with pid=13, OS id=3384Wed Dec 10 17:41:52 2008ALTER DATABASE MOUNTWed Dec 10 17:41:56 2008Setting recovery target incarnation to 2Wed Dec 10 17:41:56 2008Successful mount of redo thread 1, with mount id 1481496160Wed Dec 10 17:41:56 2008Allocated 7966612 bytes in shared pool for flashback generation bufferStarting background process RVWRRVWR started with pid=15, OS id=2180Wed Dec 10 17:41:56 2008Database mounted in Exclusive ModeCompleted: ALTER DATABASE MOUNTWed Dec 10 17:41:59 2008Starting Data Guard Broker (DMON)NSV0 started with pid=16, OS id=568INSV started with pid=18, OS id=2840Wed Dec 10 17:42:06 2008ALTER SYSTEM SET log_archive_dest_state_2='RESET' SCOPE=BOTH;NSV0 started with pid=19, OS id=1528RSM0 started with pid=20, OS id=376Wed Dec 10 17:42:27 2008ALTER SYSTEM SET log_archive_dest_2='' SCOPE=BOTH;Wed Dec 10 17:42:27 2008ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=BOTH;Wed Dec 10 17:42:27 2008FLASHBACK DATABASE TO SCN 4134609707Flashback Restore StartFlashback Restore CompleteFlashback Media Recovery Startparallel recovery started with 3 processesWed Dec 10 17:42:35 2008Recovery of Online Redo Log: Thread 1 Group 1 Seq 721 Reading mem 0Mem# 0: Z:\ORACLE\ORADATA\SATI\REDO01.LOGMem# 1: X:\ORACLE\ORADATA\SATI\REDO02.LOG

Wed Dec 10 17:42:39 2008Incomplete Recovery applied until change 4134609708


70/99

/

Flashback Media Recovery CompleteCompleted: FLASHBACK DATABASE TO SCN 4134609707Wed Dec 10 17:42:41 2008alter database convert to physical standbyClearing standby activation ID 1481388743 (0x584c32c7)The primary database controlfile was created using the'MAXLOGFILES 37' clause.There is space for up to 34 standby redo logfilesUse the following SQL commands on the standby database to createstandby redo logfiles that match the primary databasZ:ALTER DATABASE ADD STANDBY LOGFILE 'srl1.f' SIZE 104857600;ALTER DATABASE ADD STANDBY LOGFILE 'srl2.f' SIZE 104857600;ALTER DATABASE ADD STANDBY LOGFILE 'srl3.f' SIZE 104857600;ALTER DATABASE ADD STANDBY LOGFILE 'srl4.f' SIZE 104857600;Completed: alter database convert to physical standbyWed Dec 10 17:42:42 2008Shutting down instancZ: further logons disabledWed Dec 10 17:42:42 2008Stopping background process CJQ0Wed Dec 10 17:42:43 2008Stopping background process MMNLWed Dec 10 17:42:44 2008Stopping background process MMONWed Dec 10 17:42:44 2008Shutting down instance (immediate)License high water mark = 7Wed Dec 10 17:42:44 2008Stopping Job queue slave processes, flags = 7Wed Dec 10 17:42:44 2008Job queue slave processes stoppedWed Dec 10 17:42:51 2008alter database CLOSE NORMALORA-1109 signalled during: alter database CLOSE NORMAL...Wed Dec 10 17:42:51 2008alter database DISMOUNTCompleted: alter database DISMOUNTARCH: Archival disabled due to shutdown: 1089Shutting down archive processes


71/99

/

Archiving is disabledArchive process shutdown avoided: 0 activeShutting down Data Guard Broker processesWed Dec 10 17:42:53 2008Completed: Data Guard Broker shutdownWed Dec 10 17:42:55 2008ARCH: Archival disabled due to shutdown: 1089Shutting down archive processesArchiving is disabledArchive process shutdown avoided: 0 activeWed Dec 10 17:42:56 2008Starting ORACLE instance (normal)LICENSE_MAX_SESSION = 0LICENSE_SESSIONS_WARNING = 0Picked latch-free SCN scheme 2Autotune of undo retention is turned on. IMODE=BRILAT =55LICENSE_MAX_USERS = 0SYS auditing is disabledksdpec: called for event 13740 prior to event group initializationStarting up ORACLE RDBMS Version: 10.2.0.4.0.System parameters with non-default values:processes = 150sessions = 500timed_statistics = TRUEsga_max_size = 1258291200__shared_pool_size = 209715200shared_pool_size = 0__large_pool_size = 16777216large_pool_size = 0__java_pool_size = 8388608java_pool_size = 0__streams_pool_size = 50331648streams_pool_size = 50331648sga_target = 1048576000control_files = Z:\ORACLE\ORADATA\SATI\CONTROL01.CTL,

X:\ORACLE\ORADATA\SATI\CONTROL02.CTL, G:\ORACLE\ORADATA\SATI\CONTROL03.CTL


72/99

/

db_block_size = 8192__db_cache_size = 754974720db_cache_size = 0compatible = 10.2.0.4.0log_archive_config = DG_CONFIG=(SATI,SATISTD)log_archive_dest_1 = location="G:\oracle\oradata\SATI\archive",

valid_for=(ONLINE_LOGFILE,ALL_ROLES)log_archive_dest_2 = log_archive_dest_state_1 = ENABLElog_archive_dest_state_2 = ENABLElog_archive_max_processes= 2log_archive_min_succeed_dest= 1standby_archive_dest = log_archive_trace = 0log_archive_format = %t_%s_%r.arcfal_client =

(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=DGONE)(PORT=1522)))(CONNECT_DATA=(SERVICE_NAME =SATI_XPT)(INSTANCE_NAME=SATI)(SERVER=dedicated)))fal_server =

(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=DGTWO)(PORT=1522)))(CONNECT_DATA=(SERVICE_NAME =SATISTD_XPT)(SERVER=dedicated)))archive_lag_target = 0db_files = 1024db_file_multiblock_read_count= 16db_recovery_file_dest = G:\FRAdb_recovery_file_dest_size= 26843545600standby_file_management = AUTOfast_start_mttr_target = 300undo_management = AUTOundo_tablespace = UNDOTBS1undo_retention = 3600remote_os_authent = TRUEremote_login_passwordfile= EXCLUSIVEdb_domain = instance_name = SATIlocal_listener = (ADDRESS = (PROTOCOL = TCP)(HOST = DGONE)(PORT = 1522))utl_file_dir = Z:\oracle\admin\SATI\bdumpjob_queue_processes = 50


73/99

/

background_dump_dest = Z:\ORACLE\ADMIN\SATI\BDUMPuser_dump_dest = Z:\ORACLE\ADMIN\SATI\UDUMPcore_dump_dest = Z:\ORACLE\ADMIN\SATI\CDUMPsession_max_open_files = 20db_name = SATIdb_unique_name = SATIopen_cursors = 1000star_transformation_enabled= TRUEquery_rewrite_enabled = TRUEpga_aggregate_target = 524288000workarea_size_policy = AUTOdg_broker_start = TRUE

PMON started with pid=2, OS id=2172PSP0 started with pid=3, OS id=2320MMAN started with pid=4, OS id=3324DBW0 started with pid=5, OS id=3820LGWR started with pid=6, OS id=3244CKPT started with pid=7, OS id=3984SMON started with pid=8, OS id=1872RECO started with pid=9, OS id=3768CJQ0 started with pid=10, OS id=3988MMON started with pid=11, OS id=2016MMNL started with pid=12, OS id=3840DMON started with pid=13, OS id=3280Wed Dec 10 17:42:56 2008alter database mountWed Dec 10 17:43:00 2008Setting recovery target incarnation to 2ARCH: STARTING ARCH PROCESSESARC0 started with pid=15, OS id=888Wed Dec 10 17:43:00 2008ARC0: Archival startedARC1 started with pid=16, OS id=756Wed Dec 10 17:43:00 2008ARC1: Archival startedARCH: STARTING ARCH PROCESSES COMPLETEWed Dec 10 17:43:00 2008ARC0: Becoming the 'no FAL' ARCH


74/99

/

ARC0: Becoming the 'no SRL' ARCHARC0: Thread not mountedWed Dec 10 17:43:00 2008Successful mount of redo thread 1, with mount id 1481482656Wed Dec 10 17:43:00 2008Allocated 7966612 bytes in shared pool for flashback generation bufferStarting background process RVWRRVWR started with pid=17, OS id=1828Wed Dec 10 17:43:01 2008Physical Standby Database mounted.Wed Dec 10 17:43:01 2008ARC1: Becoming the heartbeat ARCHCompleted: alter database mountWed Dec 10 17:43:04 2008Starting Data Guard Broker (DMON)INSV started with pid=18, OS id=1040NSV0 started with pid=19, OS id=3944RSM0 started with pid=20, OS id=3380Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DESTWed Dec 10 17:43:19 2008ALTER SYSTEM SET log_archive_dest_2='location="dgsby"','valid_for=(STANDBY_LOGFILE,STANDBY_ROLE)'SCOPE=BOTH SID='SATI';Wed Dec 10 17:43:19 2008ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=BOTH SID='SATI';Wed Dec 10 17:43:19 2008ALTER SYSTEM SET standby_archive_dest='dgsby' SCOPE=BOTH SID='SATI';Wed Dec 10 17:43:19 2008ALTER SYSTEM SET log_archive_trace=0 SCOPE=BOTH SID='SATI';Wed Dec 10 17:43:19 2008ALTER SYSTEM SET log_archive_format='%t_%s_%r.arc' SCOPE=SPFILE SID='SATI';Wed Dec 10 17:43:19 2008ALTER SYSTEM SET standby_file_management='AUTO' SCOPE=BOTH SID='*';Wed Dec 10 17:43:19 2008ALTER SYSTEM SET archive_lag_target=0 SCOPE=BOTH SID='*';Wed Dec 10 17:43:19 2008ALTER SYSTEM SET log_archive_max_processes=2 SCOPE=BOTH SID='*';Wed Dec 10 17:43:19 2008ALTER SYSTEM SET log_archive_min_succeed_dest=1 SCOPE=BOTH SID='*';


75/99

/

Wed Dec 10 17:43:19 2008ALTER SYSTEM SETfal_server='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=DGTWO)(PORT=1522)))(CONNECT_DATA=( SERVICE_NAME=SATISTD_XPT)(SERVER=dedicated)))' SCOPE=BOTH;Wed Dec 10 17:43:19 2008ALTER SYSTEM SETfal_client='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=DGONE)(PORT=1522)))(CONNECT_DATA=( SERVICE_NAME=SATI_XPT)(INSTANCE_NAME=SATI)(SERVER=dedicated)))' SCOPE=BOTH;Wed Dec 10 17:43:19 2008ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL SWITCHOVER DISCONNECT USING CURRENTLOGFILEMRP0 started with pid=23, OS id=2548Managed Standby Recovery starting Real Time Applyparallel recovery started with 3 processesWed Dec 10 17:43:28 2008Waiting for all non-current ORLs to be archived...Clearing online redo logfile 1 Z:\ORACLE\ORADATA\SATI\REDO01.LOGClearing online log 1 of thread 1 sequence number 721Wed Dec 10 17:43:28 2008Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL SWITCHOVER DISCONNECT USINGCURRENT LOGFILEWed Dec 10 17:43:30 2008Clearing online redo logfile 1 completeClearing online redo logfile 2 Z:\ORACLE\ORADATA\SATI\REDO11.LOGClearing online log 2 of thread 1 sequence number 719Wed Dec 10 17:43:31 2008Redo Shipping Client Connected as PUBLIC-- Connected User is ValidRFS[1]: Assigned to RFS process 3180RFS[1]: Identified database type as 'physical standby'Primary database is in MAXIMUM AVAILABILITY modeChanging standby controlfile to RESYNCHRONIZATION levelWed Dec 10 17:43:32 2008RFS LogMiner: Client disabled from further notificationWed Dec 10 17:43:32 2008Clearing online redo logfile 2 completeClearing online redo logfile 3 Z:\ORACLE\ORADATA\SATI\REDO21.LOGClearing online log 3 of thread 1 sequence number 720


76/99

/

Clearing online redo logfile 3 completeMedia Recovery Waiting for thread 1 sequence 721Wed Dec 10 17:43:35 2008Primary database is in MAXIMUM AVAILABILITY modeStandby controlfile consistent with primaryRFS[1]: Successfully opened standby log 4: 'Z:\ORACLE\ORADATA\SATI\STDBYREDOG4_01.LOG'Wed Dec 10 17:43:37 2008Redo Shipping Client Connected as PUBLIC-- Connected User is ValidRFS[2]: Assigned to RFS process 3536RFS[2]: Identified database type as 'physical standby'Wed Dec 10 17:43:37 2008Redo Shipping Client Connected as PUBLIC-- Connected User is ValidRFS[3]: Assigned to RFS process 3808RFS[3]: Identified database type as 'physical standby'Wed Dec 10 17:43:39 2008RFS[2]: Successfully opened standby log 5: 'Z:\ORACLE\ORADATA\SATI\STDBYREDOG5_01.LOG'RFS[2]: Detected missing archivals for Branch(resetlogs_id): 646399178RFS[2]: Last archived SCN: 0:-160415356 Last change SCN: 0:-160357587RFS[2]: New Archival REDO Branch(resetlogs_id): 673119597 Prior: 646399178RFS[2]: Archival Activation ID: 0x584c2299 Current: 0x0RFS[2]: Effect of primary database OPEN RESETLOGSRFS[2]: Managed Standby Recovery process is activeNew incarnation branch detected in ArchiveLog, filename Z:\ORACLE\ORADATA\SATI\STDBYREDOG5_01.LOGInspection of file changed rdi from 2 to 3Setting recovery target incarnation to 3Wed Dec 10 17:43:40 2008Setting recovery target incarnation to 3Wed Dec 10 17:43:44 2008RFS[3]: Archived Log: 'C:\ORACLE\PRODUCT\10.2.0\DB_1\DATABASE\DGSBY1_721_646399178.ARC'Wed Dec 10 17:43:45 2008MRP0: Incarnation has changed! Retry recovery...Wed Dec 10 17:43:45 2008Errors in file Z:\oracle\admin\SATI\bdump\SATI_mrp0_2548.trc:ORA-19906: recovery target incarnation changed during recovery

Managed Standby Recovery not using Real Time Apply


77/99

/

Recovery interrupted!Wed Dec 10 17:43:49 2008Errors in file Z:\oracle\admin\SATI\bdump\SATI_mrp0_2548.trc:ORA-19906: recovery target incarnation changed during recovery

Wed Dec 10 17:44:09 2008Managed Standby Recovery starting Real Time Applyparallel recovery started with 3 processesMedia Recovery start incarnation depth : 1, target inc# : 3, irscn : 4134609709Wed Dec 10 17:44:15 2008Waiting for all non-current ORLs to be archived...Media Recovery Log C:\ORACLE\PRODUCT\10.2.0\DB_1\DATABASE\DGSBY1_721_646399178.ARCIdentified End-Of-Redo for thread 1 sequence 721Wed Dec 10 17:44:15 2008Media Recovery End-Of-Redo indicator encounteredWed Dec 10 17:44:15 2008Media Recovery ContinuingMedia Recovery Log C:\ORACLE\PRODUCT\10.2.0\DB_1\DATABASE\DGSBY1_1_673119597.ARCMedia Recovery Waiting for thread 1 sequence 2 (in transit)Wed Dec 10 17:44:21 2008Recovery of Online Redo Log: Thread 1 Group 4 Seq 2 Reading mem 0Mem# 0: Z:\ORACLE\ORADATA\SATI\STDBYREDOG4_01.LOGMem# 1: X:\ORACLE\ORADATA\SATI\STDBYREDOG4_02.LOG

Wed Dec 10 17:44:46 2008Primary database is in MAXIMUM AVAILABILITY modeChanging standby controlfile to MAXIMUM AVAILABILITY levelRFS[1]: Successfully opened standby log 4: 'Z:\ORACLE\ORADATA\SATI\STDBYREDOG4_01.LOG'Wed Dec 10 17:44:47 2008Media Recovery Log C:\ORACLE\PRODUCT\10.2.0\DB_1\DATABASE\DGSBY1_2_673119597.ARCMedia Recovery Waiting for thread 1 sequence 3 (in transit)Wed Dec 10 17:44:47 2008Recovery of Online Redo Log: Thread 1 Group 4 Seq 3 Reading mem 0Mem# 0: Z:\ORACLE\ORADATA\SATI\STDBYREDOG4_01.LOGMem# 1: X:\ORACLE\ORADATA\SATI\STDBYREDOG4_02.LOG

Wed Dec 10 17:57:59 2008db_recovery_file_dest_size of 25600 MB is 4.70% used. This is auser-specified limit on the amount of space that will be used by thisdatabase for recovery-related files, and does not reflect the amount of


78/99

/

space available in the underlying filesystem or ASM diskgroup.

DGMGRL CHECKUPS

There are several checkup commands that can be executed from within DGMGRL to obtain the status of each oneof the Dataguard components. If at the last line of the output of any acomponent there is a line having SUCCESSstatus, that means the component is healthy.

C:\Documents and Settings\AV\Desktop\SCRIPTS>dgmgrl sys/xxxxxx@SATISTDDGMGRL for 32-bit Windows: Version 10.2.0.4.0 - Production


Welcome to DGMGRL, type "help" for information.Connected.DGMGRL> SHOW CONFIGURATION

ConfigurationNamZ: fsfodgEnabled: YESProtection ModZ: MaxAvailabilityFast-Start Failover: ENABLEDDatabases:SATISTD - Primary databaseSATI - Physical standby database




79/99

/

DGMGRL> SHOW DATABASE SATI

DatabaseNamZ: SATIRolZ: PHYSICAL STANDBYEnabled: YESIntended StatZ: ONLINEInstance(s):SATI


DGMGRL> SHOW DATABASE SATISTD

DatabaseNamZ: SATISTDRolZ: PRIMARYEnabled: YESIntended StatZ: ONLINEInstance(s):SATISTD


DGMGRL> SHOW DATABASE SATISTD

DatabaseNamZ: SATISTDRolZ: PRIMARY


80/99

/

Enabled: YESIntended StatZ: ONLINEInstance(s):SATISTD


DGMGRL> SHOW INSTANCE SATI

Instance 'SATI' of database 'SATI'Host NamZ: DGONE


DGMGRL> SHOW INSTANCE SATISTD

Instance 'SATISTD' of database 'SATISTD'Host NamZ: DGTWO


DGMGRL> SHOW DATABASE VERBOSE SATI

DatabaseNamZ: SATIRolZ: PHYSICAL STANDBYEnabled: YESIntended StatZ: ONLINE


81/99

/

Instance(s):SATI

Properties:InitialConnectIdentifier = 'SATI'ObserverConnectIdentifier = ''LogXptMode = 'SYNC'Dependency = ''DelayMins = '0'Binding = 'OPTIONAL'MaxFailure = '0'MaxConnections = '1'ReopenSecs = '300'NetTimeout = '180'LogShipping = 'ON'PreferredApplyInstance = ''ApplyInstanceTimeout = '0'ApplyParallel = 'AUTO'StandbyFileManagement = 'AUTO'ArchiveLagTarget = '0'LogArchiveMaxProcesses = '2'LogArchiveMinSucceedDest = '1'DbFileNameConvert = ''LogFileNameConvert = ''FastStartFailoverTarget = 'SATISTD'StatusReport = '(monitor)'InconsistentProperties = '(monitor)'InconsistentLogXptProps = '(monitor)'SendQEntries = '(monitor)'LogXptStatus = '(monitor)'RecvQEntries = '(monitor)'


82/99

/

HostName = 'DGONE'SidName = 'SATI'LocalListenerAddress =

'(ADDRESS=(PROTOCOL=TCP)(HOST=DGONE)(PORT=1522))'StandbyArchiveLocation = 'dgsby'AlternateLocation = ''LogArchiveTrace = '0'LogArchiveFormat = '%t_%s_%r.arc'LatestLog = '(monitor)'TopWaitEvents = '(monitor)'


DGMGRL> SHOW DATABASE VERBOSE SATISTD

DatabaseNamZ: SATISTDRolZ: PRIMARYEnabled: YESIntended StatZ: ONLINEInstance(s):SATISTD

Properties:InitialConnectIdentifier = 'SATISTD'ObserverConnectIdentifier = ''LogXptMode = 'SYNC'Dependency = ''DelayMins = '0'Binding = 'OPTIONAL'


83/99

/

MaxFailure = '0'MaxConnections = '1'ReopenSecs = '300'NetTimeout = '180'LogShipping = 'ON'PreferredApplyInstance = ''ApplyInstanceTimeout = '0'ApplyParallel = 'AUTO'StandbyFileManagement = 'auto'ArchiveLagTarget = '0'LogArchiveMaxProcesses = '2'LogArchiveMinSucceedDest = '1'DbFileNameConvert = ''LogFileNameConvert = ''FastStartFailoverTarget = 'SATI'StatusReport = '(monitor)'InconsistentProperties = '(monitor)'InconsistentLogXptProps = '(monitor)'SendQEntries = '(monitor)'LogXptStatus = '(monitor)'RecvQEntries = '(monitor)'HostName = 'DGTWO'SidName = 'SATISTD'LocalListenerAddress =

'(ADDRESS=(PROTOCOL=TCP)(HOST=DGTWO)(PORT=1522))'StandbyArchiveLocation = 'G:\oracle\oradata\SATI\archive'AlternateLocation = ''LogArchiveTrace = '0'LogArchiveFormat = '%t_%s_%r.arc'LatestLog = '(monitor)'TopWaitEvents = '(monitor)'


84/99

/


DGMGRL> SHOW INSTANCE VERBOSE SATI

Instance 'SATI' of database 'SATI'Host NamZ: DGONEPFILZ:

Properties:HostName = 'DGONE'SidName = 'SATI'LocalListenerAddress =

'(ADDRESS=(PROTOCOL=TCP)(HOST=DGONE)(PORT=1522))'StandbyArchiveLocation = 'dgsby'AlternateLocation = ''LogArchiveTrace = '0'LogArchiveFormat = '%t_%s_%r.arc'LatestLog = '(monitor)'TopWaitEvents = '(monitor)'


DGMGRL> SHOW INSTANCE VERBOSE SATISTD

Instance 'SATISTD' of database 'SATISTD'Host NamZ: DGTWOPFILZ:


85/99

/

Properties:HostName = 'DGTWO'SidName = 'SATISTD'LocalListenerAddress =

'(ADDRESS=(PROTOCOL=TCP)(HOST=DGTWO)(PORT=1522))'StandbyArchiveLocation = 'G:\oracle\oradata\SATI\archive'AlternateLocation = ''LogArchiveTrace = '0'LogArchiveFormat = '%t_%s_%r.arc'LatestLog = '(monitor)'TopWaitEvents = '(monitor)'



86/99

/

Client Failover ConfigurationIn order to provide ininterupted service in case of primary failure a failover configuration needs to be implemented,this can be achieved for single instances and RAC using database services, and a network configuration that pointto all servers that can have the primary role.

On the client side it is very important to set on the ‘sqlnet.ora’ file the parameterSQLNET.OUTBOUND_CONNECT_TIMEOUT = 3, to assure that the failed connections will not wait for TCPtimeout but will immediately proceed to the next host if the primary is unavailable

SQL > exec DBMS_SERVICE.CREATE_SERVICE ( service_name => 'fsfo', network_name => 'fsfo', failover_method => 'BASIC', failover_type => 'SELECT', failover_retries => 180, failover_delay => 1);

PL/SQL procedure successfully completed.

SQL > CREATE OR REPLACE TRIGGER manage_dgserviceafter startup on databaseDECLARErole VARCHAR(30);BEGINSELECT DATABASE_ROLE INTO role FROM V$DATABASE;IF role = 'PRIMARY' THENDBMS_SERVICE.START_SERVICE('fsfo');END IF;END;


87/99

/

/

Trigger created.

SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;

On tnsnames.ora define this entry

fsfo =(DESCRIPTION =(ADDRESS_LIST =(ADDRESS = (PROTOCOL = TCP)(HOST = DGONE)(PORT = 1522))(ADDRESS = (PROTOCOL = TCP)(HOST = DGTWO)(PORT = 1522))(LOAD_BALANCE = yes))(CONNECT_DATA=(SERVICE_NAME=fsfo)))

On sqlnet.ora add this line

SQLNET.OUTBOUND_CONNECT_TIMEOUT = 3


88/99

/

Test client failover

1. from dgmgrl check that configuration and database status return SUCCESS DGMGRL> connect sys@SATIPassword:Connected.


ConfigurationNamZ: fsfodgEnabled: YESProtection ModZ: MaxAvailabilityFast-Start Failover: ENABLEDDatabases:SATISTD - Physical standby database

- Fast-Start Failover targetSATI - Primary database





89/99

/


DGMGRL> show database SATISTD

DatabaseNamZ: SATISTDRolZ: PHYSICAL STANDBYEnabled: YESIntended StatZ: ONLINEInstance(s):SATISTD





2. Connect from a client server using the ‘fsfo’ service, check the instance namZ:

SQL> select instance_name from v$instance;


90/99

/

INSTANCE_NAME----------------SATI

3. From the client server, connected throug the ‘fsfo’ service start a long running query, i.e. ‘select 1,2,3 fromdba_source’ and check how long it takes to complete, you may repeat a couple of times to get it cached and get the best time.

SQL> select 1,2,3 from dba_source1 2 31 2 3

...

...1 2 31 2 3

140829 rows selected.

Elapsed: 00:0:30.12

4. From the client server, connected throug the ‘fsfo’ service issue again the same long running query,immediately move to the primary database and shut down it using the abort option.

SQL> select 1,2,3 from dba_source1 2 31 2 3

...SQL> shutdown abort

5. The ‘shutdown abort’ triggers a FSFO that will complete in about 30 seconds


91/99

/

6. At the client screen you will notice that the output from ‘select 1,2,3 from dba_source’ stops and then continueuntil finished

... <<<<<< Database shutdown abort here

...1 2 31 2 3

140829 rows selected.

Elapsed: 00:01:23.23 <<<<<< the select took longer but finished

7. at the client check the instance name ‘

SQL> select instance_name from v$instance;

INSTANCE_NAME----------------SATISTD <<<<<

Elapsed: 00:00:00.07

8. Connect to the failed primary, mount it up and check the FSFO status

SQL> startup mountORACLE instance started.

Total System Global Area 1258291200 bytesFixed Size 1298304 bytesVariable Size 436207744 bytes


92/99

/

Database Buffers 813694976 bytesRedo Buffers 7090176 bytesDatabase mounted.

SQL> select DB_UNIQUE_NAME, DATABASE_ROLE, OPEN_MODE, FS_FAILOVER_STATUS, FS_FAILOVER_CURRENT_TARGET From v$database;

DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE FS_FAILOVER_STATUS FS_FAILOVER_CURRENT_TARGET--------------- ---------------- ---------- --------------------- ------------------------------SATI PHYSICAL STANDBY MOUNTED BYSTANDER

SQL> /

DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE FS_FAILOVER_STATUS FS_FAILOVER_CURRENT_TARGET--------------- ---------------- ---------- --------------------- ------------------------------SATI PHYSICAL STANDBY MOUNTED REINSTATE IN PROGRESS SATI

SQL> /

DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE FS_FAILOVER_STATUS FS_FAILOVER_CURRENT_TARGET--------------- ---------------- ---------- --------------------- ------------------------------SATI PHYSICAL STANDBY MOUNTED UNSYNCHRONIZED SATI

SQL> /

DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE FS_FAILOVER_STATUS FS_FAILOVER_CURRENT_TARGET--------------- ---------------- ---------- --------------------- ------------------------------SATI PHYSICAL STANDBY MOUNTED SYNCHRONIZED SATI

9. From DGMGRL switchover back to the primary site

DGMGRL> connect sys@SATIPassword:Connected.


93/99

/

DGMGRL> show configurationConfigurationNamZ: fsfodgEnabled: YESProtection ModZ: MaxAvailabilityFast-Start Failover: ENABLEDDatabases:SATISTD - Primary databaseSATI - Physical standby database



DGMGRL> show database SATIDatabaseNamZ: SATIRolZ: PHYSICAL STANDBYEnabled: YESIntended StatZ: ONLINEInstance(s):SATI


DGMGRL> show database SATISTDDatabaseNamZ: SATISTDRolZ: PRIMARYEnabled: YES


94/99

/

Intended StatZ: ONLINEInstance(s):SATISTD


DGMGRL> switchover to SATIPerforming switchover NOW, please wait...Operation requires shutdown of instance "SATISTD" on database "SATISTD"Shutting down instance "SATISTD"...ORA-01109: database not open

Database dismounted.ORACLE instance shut down.Operation requires shutdown of instance "SATI" on database "SATI"Shutting down instance "SATI"...ORA-01109: database not open

Database dismounted.ORACLE instance shut down.Operation requires startup of instance "SATISTD" on database "SATISTD"Starting instance "SATISTD"...ORACLE instance started.Database mounted.Operation requires startup of instance "SATI" on database "SATI"Starting instance "SATI"...ORACLE instance started.Database mounted.Switchover succeeded, new primary is "SATI"


95/99

/

10. Check status of both sites

C: >sqlplus sys@SATI as sysdba

SQL*Plus: Release 10.2.0.4.0 - Production on Sun Dec 28 16:55:22 2008Copyright (c) 1982, 2007, Oracle. All Rights Reserved.


SQL> select DB_UNIQUE_NAME, DATABASE_ROLE, OPEN_MODE, FS_FAILOVER_STATUS, FS_FAILOVER_CURRENT_TARGET From v$database;

DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE FS_FAILOVER_STATUS FS_FAILOVER_CURRENT_TARGET--------------- ---------------- ---------- --------------------- ------------------------------SATI PRIMARY READ WRITE SYNCHRONIZED SATISTD

SQL> connect sys@SATISTD as sysdba

Enter password:Connected.

SQL> /

DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE FS_FAILOVER_STATUS FS_FAILOVER_CURRENT_TARGET--------------- ---------------- ---------- --------------------- ------------------------------

SATISTD PHYSICAL STANDBY MOUNTED SYNCHRONIZED SATISTD


96/99

/

Summary of main commands used on this documentFollowing there is a list of the main commands used on this document with a brief explanation of the functionalitythat is invoked with the command.

Command Description alter database force logging; Switch off no logging operations at the database

level orapwd file=orapwSATI password=<sys_password>force=y

Create the password file on the primary database

alter database add standby logfile group 4 ('Z:\ORACLE\ORADATA\SATI\STDBYREDOG4_01.LOG','X:\ORACLE\ORADATA\SATI\STDBYREDOG4_02.LOG')SIZE 104857600;

Create standby redologs, they need to be the samesize as online logs, we need to create at leastone standby group more than total number of onlinegroups.

startup mount;alter database archivelog;alter database open;

Mount the database, enable archive log mode andopen the database

oradim –NEW –SID SATISTD –STARTMODEmanual

Create a database service on Windows

alter database create standbycontrolfile as 'c:\SATISTD.ctl';

Create a stnadby controlfile on the primarydatabase, to be copied to the standby

create pfile='c:\standby_pfile.ora' fromspfile;

Create a pfile from the primary database to beedited for the physical standby

create spfile from pfile='c:\standby_pfile.txt'; Create an spfile on the default location,?/database on windows

alter database recover managed standby databasedisconnect from session;

Start recovery on the physical standby

select sequence#, first_time, next_time fromv$archived_log order by sequence#;

Check redo apply on the physical standby

alter system set dg_broker_start=truescope=both;

Enable the dataguard broker

dgmgrl sys/<passwd>@SATISTD Connect to the broker command line utility dgmgrl


97/99

/

create configuration fsfodg as primary databaseis SATI connect identifier is SATI;

Create Dataguard configuration form dgmngrl

add database SATISTD as connect identifier isSATISTD maintained as physical;

Add database to the broker configuration

show configuration; show broker configuration and status show database verbose SATISTD; Show broker database configuration show instance verbose SATISTD; Show broker instance configuration enable configuration; Enable broker configuration enable database SATISTD; Enable database in broker configuration edit database SATI set property'logxptmode'='sync';

Set database property logxptmode to sync onbroker, to enable FSFO

edit configuration set protection mode asmaxavailability;

Change configuration protection mode to maxavailability, to enable FSFO

enable fast_start failover; Enable FSFO select db_unique_name, database_role, open_mode,fs_failover_status, fs_failover_current_targetfrom v$database;

Check FSFO status on both Primary and StandbyDatabases

switchover to SATISTD; From within DGMGRL start switchover


98/99

/

References

1. Switchover and Failover Best Practices: Oracle Data Guard 10g Release 2http://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_FastStartFailoverBestPractices.pdf

2. Fast-Start Failover Best Practices: Oracle Data Guard 10g Releasehttp://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR2_FastStartFailoverBestPractices.pdf

3. Oracle Maximum Availability Architecture http://www.oracle.com/technology/deploy/availability/htdocs/maa.htm

4. Oracle Database High Availability Best Practices (Part #B25159)http://otn.oracle.com/pls/db102/db102.to_toc?partno=b25159

5. Workload Management with Oracle Real Application Clustershttp://www.oracle.com/technology/products/database/clustering/pdf/twpracwkldmgmt.pdf

6. Oracle Data Guard http://www.oracle.com/technology/deploy/availability/htdocs/DataGuardOverview.html

7. Oracle Database Oracle Clusterware and Oracle Real Application Clusters Administration and DeploymentGuide (Part #14197) http://download-west.oracle.com/docs/cd/B19306_01/rac.102/b14197/toc.htm

8. Workload Management with Oracle Real Application Clusters 10g (Provides a detailed explanation of theimplementation of Services, FAN and Fast Connection Failover in a RAC environment.):http://www.oracle.com/technology/products/database/clustering/pdf/twpracwkldmgmt.pdf


99/99

/

9. Oracle Data Guard Broker (Part #B14230) http://otn.oracle.com/pls/db102/db102.to_toc?partno=b14230

10. Oracle Database PL/SQL Packages and Types Reference (Part #B14261)http://otn.oracle.com/pls/db102/db102.to_toc?partno=b14261

11. Oracle Call Interface Programmer’s Guide (Part #B14250) http://otn.oracle.com/pls/db102/db102.to_toc?partno=b14250

12. Oracle Data Guard Concepts and Administration (Part #B14239)http://otn.oracle.com/pls/db102/db102.to_toc?partno=b14239