13
http://veera0109.blogspot.com/2014/03/how-to-recover-standby- database-from.html How to Recover Standby Database from a Missing Archive Log by applying the Incremental backup from Production Database Resync the standby database using an incremental RMAN backup from the primary database. Advantages: ----------- * This does not require a full rebuild of the standby database and can be a lot more efficient. * Time saving. Scenario: --------- Primary Database - pdb Standby Database - sdb One of the archivelog files is missing and there is no backup of that archivelog. Primary database and Standby database are not in SYNC. There is an archival gap. Resyncing the standby data using RMAN incremental backup taken from the primary database. On Primary Database: -------------------- connect as sys user and check the archive log list SQL> conn sys/oracle@sdb as sysdba Connected. SQL> select instance_name from v$instance; INSTANCE_NAME -------------- sdb SQL> archive log list Database log mode Archive Mode Automatic archival Enabled Archive destination /vol1/pdb/arch Oldest online log sequence 23 Next log sequence to archive 25 Current log sequence 25 Issue alter system switch logfile on primary database SQL> alter system switch logfile; System altered. SQL> archive log list Database log mode Archive Mode Automatic archival Enabled Archive destination /vol1/pdb/arch Oldest online log sequence 24 Next log sequence to archive 26

Recover Standby Database From a Missing Archive Log Incremental Backup

Embed Size (px)

DESCRIPTION

Oracle utils

Citation preview

Page 1: Recover Standby Database From a Missing Archive Log Incremental Backup

http://veera0109.blogspot.com/2014/03/how-to-recover-standby-database-from.html

How to Recover Standby Database from a Missing Archive Log by applying the Incremental backup from Production Database

Resync the standby database using an incremental RMAN backup from the primary database.

Advantages:-----------

* This does not require a full rebuild of the standby database     and can be a lot more efficient.

* Time saving.

Scenario:---------

Primary Database - pdb

Standby Database - sdb

One of the archivelog files is missing and there is no backup of that archivelog. Primary database and Standby database are not in SYNC. There is an archival gap.

Resyncing the standby data using RMAN incremental backup taken from the primary database.

On Primary Database:--------------------

connect as sys user and check the archive log list

SQL> conn sys/oracle@sdb as sysdbaConnected.

SQL> select instance_name from v$instance;

INSTANCE_NAME--------------sdb

SQL> archive log listDatabase log mode              Archive ModeAutomatic archival             EnabledArchive destination            /vol1/pdb/archOldest online log sequence     23Next log sequence to archive   25Current log sequence           25

Issue alter system switch logfile on primary database

SQL> alter system switch logfile;

System altered.

SQL> archive log listDatabase log mode              Archive ModeAutomatic archival             EnabledArchive destination            /vol1/pdb/archOldest online log sequence     24Next log sequence to archive   26Current log sequence           26

ON Standby database:--------------------

SQL> conn sys/oracle@sdb as sysdbaConnected.SQL> select instance_name from v$instance;

INSTANCE_NAME----------------sdb

SQL> archive log list

Page 2: Recover Standby Database From a Missing Archive Log Incremental Backup

Database log mode              Archive ModeAutomatic archival             EnabledArchive destination            /vol2/sdb/archOldest online log sequence     24Next log sequence to archive   0Current log sequence           26

ON Primary Database:--------------------

Create a user, connect to that user, create table and insert records in it.

SQL> conn veera/[email protected]>SQL> select * from tab;

no rows selected

SQL>SQL>SQL> create table emp (empid number(4), emp_name varchar2(12));

Table created.

SQL> insert into emp values(&empid,'&emp_name');Enter value for empid: 1111Enter value for emp_name: veeraold   1: insert into emp values(&empid,'&emp_name')new   1: insert into emp values(1111,'veera')

1 row created.

SQL> /Enter value for empid: 1309Enter value for emp_name: sureshold   1: insert into emp values(&empid,'&emp_name')new   1: insert into emp values(1309,'suresh')

1 row created.

SQL> commit;

Commit complete.

SQL> select * from emp;

     EMPID EMP_NAME---------- ------------      1111 veera      1309 suresh

You can see that we connected to a user "veera", created a table "emp" and inserted rows in to it.

ON Standby Database:--------------------

connect to the standby database and check whether the table has been replicated or not.

SQL> conn sys/oracle@sdb as sysdbaConnected.SQL> archive log listDatabase log mode              Archive ModeAutomatic archival             EnabledArchive destination            /vol2/sdb/archOldest online log sequence     25Next log sequence to archive   0Current log sequence           27

SQL> select * from veera.emp;

     EMPID EMP_NAME---------- ------------      1111 veera      1309 suresh

From the above we can see that the data has been replicated to the standby database from the primary database.

Page 3: Recover Standby Database From a Missing Archive Log Incremental Backup

Now cancel the media recovery in Standby database.

     alter database recover managed standby database cancel;

On Primary Database:--------------------

Insert in to the same table two more rows.

SQL> insert into emp values(&empid,'&emp_name');Enter value for empid: 1234Enter value for emp_name: mannojold   1: insert into emp values(&empid,'&emp_name')new   1: insert into emp values(1234,'mannoj')

1 row created.

SQL> insert into emp values(&empid,'&emp_name');Enter value for empid: 9876Enter value for emp_name: thiyaguold   1: insert into emp values(&empid,'&emp_name')new   1: insert into emp values(9876,'thiyagu')

1 row created.

SQL> commit;

Commit complete.

SQL> select * from veera.emp;

     EMPID EMP_NAME---------- ------------      1111 veera      1309 suresh      1234 mannoj      9876 thiyagu

Now connect as a sys user and switch the logfile.

SQL> alter system switch logfile;

System altered.

SQL> archive log listDatabase log mode              Archive ModeAutomatic archival             EnabledArchive destination            /vol1/pdb/archOldest online log sequence     26Next log sequence to archive   28Current log sequence           28

Now remove this archivelog file physically from the production database.

SQL> archive log listDatabase log mode              Archive ModeAutomatic archival             EnabledArchive destination            /vol1/pdb/archOldest online log sequence     26Next log sequence to archive   28Current log sequence           28

SQL> alter system switch logfile;

System altered.

SQL> archive log listDatabase log mode              Archive ModeAutomatic archival             EnabledArchive destination            /vol1/pdb/archOldest online log sequence     27Next log sequence to archive   29Current log sequence           29

ON Standby Database:--------------------

The archivelog file with sequence# 27 will be moved to standby database. Since the media recovery process has been canceled the archivelog will not be applied to the standby database.

Page 4: Recover Standby Database From a Missing Archive Log Incremental Backup

Now remove that archivelog with sequence# 27 physically from standby server also.

Once after deleting the archivelog file from both production and standby database servers, enable the media recovery process in standby database.

     alter database recover managed standby database disconnect from session;

SQL> archive log listDatabase log mode              Archive MoAutomatic archival             EnabledArchive destination            /vol2/sdb/archOldest online log sequence     26Next log sequence to archive   0Current log sequence           28

SQL> select * from veera.emp;

     EMPID EMP_NAME---------- ------------      1111 veera      1309 suresh

      1309 suresh

SQL> archive log listDatabase log mode              Archive MoAutomatic archival             EnabledArchive destination            /vol2/sdb/archOldest online log sequence     26Next log sequence to archive   0Current log sequence           28SQL>SQL>SQL> alter database recover managed standby database disconnect from session;

Database altered.

SQL> archive log listDatabase log mode              Archive MoAutomatic archival             EnabledArchive destination            /vol2/sdb/archOldest online log sequence     27Next log sequence to archive   0Current log sequence           29SQL>SQL>SQL> archive log listDatabase log mode              Archive MoAutomatic archival             EnabledArchive destination            /vol2/sdb/archOldest online log sequence     27Next log sequence to archive   0Current log sequence           29SQL>SQL>SQL>SQL> select sequence#,archived,applied from v$archived_log;

 SEQUENCE# ARC APPLIED---------- --- ---------         2 YES YES         3 YES YES         4 YES YES         5 YES YES         6 YES YES         7 YES YES         8 YES YES         9 YES YES        10 YES YES        11 YES YES        12 YES YES

 SEQUENCE# ARC APPLIED---------- --- ---------        13 YES YES        14 YES YES        15 YES YES        16 YES YES        17 YES YES        18 YES YES        19 YES YES        20 YES YES        21 YES YES        22 YES YES        23 YES YES

Page 5: Recover Standby Database From a Missing Archive Log Incremental Backup

 SEQUENCE# ARC APPLIED---------- --- ---------        24 YES YES        25 YES YES        26 YES YES        27 YES NO        28 YES NO

27 rows selected.

SQL> select * from veera.emp;

     EMPID EMP_NAME---------- ------------      1111 veera      1309 suresh

From the above we can clearly see that there is an archival gap sequence# 27 has not been applied to the standby database. But the archivelogs are moving to the standby database location but not applying.

ON Primary Database:--------------------

SQL> insert into veera.emp values(&empid,'&emp_name');Enter value for empid: 2345Enter value for emp_name: vivekold   1: insert into veera.emp values(&empid,'&emp_name')new   1: insert into veera.emp values(2345,'vivek')

1 row created.

SQL> insert into veera.emp values(&empid,'&emp_name');Enter value for empid: 3456Enter value for emp_name: shekold   1: insert into veera.emp values(&empid,'&emp_name')new   1: insert into veera.emp values(3456,'shek')

1 row created.

SQL> commit;

Commit complete.

SQL> select * from veera.emp;

     EMPID EMP_NAME---------- ------------      1111 veera      1309 suresh      1234 mannoj      9876 thiyagu      2345 vivek      3456 shek

6 rows selected.

SQL> alter system switch logfile;

System altered.

SQL> select * from veera.emp;

     EMPID EMP_NAME---------- ------------      1111 veera      1309 suresh      1234 mannoj      9876 thiyagu      2345 vivek      3456 shek

6 rows selected.

On Standby Database:--------------------

SQL> select * from veera.emp;

     EMPID EMP_NAME---------- ------------      1111 veera

Page 6: Recover Standby Database From a Missing Archive Log Incremental Backup

      1309 suresh

SQL> select sequence#,archived,applied from v$archived_log;

 SEQUENCE# ARC APPLIED---------- --- ---------         2 YES YES         3 YES YES         4 YES YES         5 YES YES         6 YES YES         7 YES YES         8 YES YES         9 YES YES        10 YES YES        11 YES YES        12 YES YES

 SEQUENCE# ARC APPLIED---------- --- ---------        13 YES YES        14 YES YES        15 YES YES        16 YES YES        17 YES YES        18 YES YES        19 YES YES        20 YES YES        21 YES YES        22 YES YES        23 YES YES

 SEQUENCE# ARC APPLIED---------- --- ---------        24 YES YES        25 YES YES        26 YES YES        27 YES NO        28 YES NO        29 YES NO

28 rows selected.

SQL> archive log listDatabase log mode              Archive ModeAutomatic archival             EnabledArchive destination            /vol2/sdb/archOldest online log sequence     28Next log sequence to archive   0Current log sequence           30

Overall Scenario:-----------------

Archivelog with sequence# 27 is missing from primary and also from standby database. Due to this there is an archival gap between primary database and standby database. The synchronization between primary database and standby database has been lost.

We have to recover / resync the standby database with the primary database.

The steps to resync the standby database using an incremental RMAN backup from the primary database are as follows.----------------------------------------------------------------------------------------------------

1. Get the current scn of the standby database.,

SQL> select current_scn from v$database;

CURRENT_SCN-----------    1053647

2. Connect to the primary database as the RMAN target and create an incremental backup from the current SCN (for a standby    lagging far behind the primary):

primary:/vol1 />rman catalog rman/rman@pdb target sys/oracle@pdb

Recovery Manager: Release 11.2.0.3.0 - Production on Wed Mar 13 15:54:00 2013

Page 7: Recover Standby Database From a Missing Archive Log Incremental Backup

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: PDB (DBID=3232692579)connected to recovery catalog database

RMAN> backup incremental from scn 1053647 database;

Starting backup at 13-MAR-13

allocated channel: ORA_DISK_1channel ORA_DISK_1: SID=14 device type=DISKbackup will be obsolete on date 20-MAR-13archived logs will not be kept or backed upchannel ORA_DISK_1: starting full datafile backup setchannel ORA_DISK_1: specifying datafile(s) in backup setinput datafile file number=00001 name=/VOL1/PDB/SYSTEM01.DBFinput datafile file number=00002 name=/VOL1/PDB/SYSAUX01.DBFinput datafile file number=00005 name=/VOL1/PDB/TBSRMAN.DBFinput datafile file number=00003 name=/VOL1/PDB/UNDOTBS01.DBFinput datafile file number=00004 name=/VOL1/PDB/USERS01.DBFchannel ORA_DISK_1: starting piece 1 at 13-MAR-13channel ORA_DISK_1: finished piece 1 at 13-MAR-13piece handle=/RMAN/DBF_BKUP/DBF_09O4EBAE_1_1 tag=TAG20130313T155700 comment=NONEchannel ORA_DISK_1: backup set complete, elapsed time: 00:00:35

using channel ORA_DISK_1backup will be obsolete on date 20-MAR-13archived logs will not be kept or backed upchannel ORA_DISK_1: starting full datafile backup setchannel ORA_DISK_1: specifying datafile(s) in backup setincluding current control file in backup setchannel ORA_DISK_1: starting piece 1 at 13-MAR-13channel ORA_DISK_1: finished piece 1 at 13-MAR-13piece handle=/RMAN/DBF_BKUP/DBF_0AO4EBBJ_1_1 tag=TAG20130313T155700 comment=NONEchannel ORA_DISK_1: backup set complete, elapsed time: 00:00:01Finished backup at 13-MAR-13

RMAN>

3. Transfer all the backup pieces created on the primary to the standby.

4. Create a standby controlfile on the primary and ship to the standby.

SQL> alter database create standby controlfile as '/oracle/CONTROL01.CTL';

Database altered.

Now on Standby database:------------------------

5. Recreate the standby controlfile on the standby:

a. shutdown abort the database.,

SQL> shut abortORACLE instance shut down.

b. Start the database in nomount.,

SQL> startup nomountORACLE instance started.

Total System Global Area 1686925312 bytesFixed Size                  2255824 bytesVariable Size             989856816 bytesDatabase Buffers          687865856 bytesRedo Buffers                6946816 bytesSQL>

c. connect to the rman prompt and connect to the target database (sdb).,

secondary:/vol2 />rman target sys/oracle@sdb

Recovery Manager: Release 11.2.0.3.0 - Production on Wed Mar 13 16:20:45 2013

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: PDB (not mounted)

d. restore the controlfile using the below command.,

     restore controlfile from '/oracle/CONTROL01.CTL';

Page 8: Recover Standby Database From a Missing Archive Log Incremental Backup

RMAN> restore controlfile from '/oracle/CONTROL01.CTL';

Starting restore at 13-MAR-13using target database control file instead of recovery catalogallocated channel: ORA_DISK_1channel ORA_DISK_1: SID=129 device type=DISK

channel ORA_DISK_1: copied control file copyoutput file name=/VOL2/SDB/STDCONTROL01.CTLFinished restore at 13-MAR-13

RMAN>

NOTE:-----

 Rename standby data/temp files and redo if different from the primary using ALTER DATABASE RENAME FILE

e. Mount the standby database.,

RMAN> alter database mount;

database mountedreleased channel: ORA_DISK_1

f. Catalog backup pieces if the location on the standby is different from the primary.

connect to rman as target as standby database (stb).,

secondary:/vol2 />rman target sys/oracle@sdb

Recovery Manager: Release 11.2.0.3.0 - Production on Wed Mar 13 16:01:03 2013

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: PDB (DBID=3232692579)

RMAN> catalog start with '/rman/rman_sdb';

searching for all files that match the pattern /rman/rman_sdb

List of Files Unknown to the Database=====================================File Name: /rman/rman_sdb/DBF_0BO4GTHJ_1_1File Name: /rman/rman_sdb/DBF_0CO4GTIO_1_1

Do you really want to catalog the above files (enter YES or NO)? YEScataloging files...cataloging done

List of Cataloged Files=======================File Name: /rman/rman_sdb/DBF_0BO4GTHJ_1_1File Name: /rman/rman_sdb/DBF_0CO4GTIO_1_1

g. Apply incremental backup to the standby.,

RMAN> recover database noredo;

Starting recover at 13-MAR-13allocated channel: ORA_DISK_1channel ORA_DISK_1: SID=195 device type=DISKchannel ORA_DISK_1: starting incremental datafile backup set restorechannel ORA_DISK_1: specifying datafile(s) to restore from backup setdestination for restore of datafile 00001: /VOL2/SDB/SYSTEM01.DBFdestination for restore of datafile 00002: /VOL2/SDB/SYSAUX01.DBFdestination for restore of datafile 00003: /VOL2/SDB/UNDOTBS01.DBFdestination for restore of datafile 00004: /VOL2/SDB/USERS01.DBFdestination for restore of datafile 00005: /VOL2/SDB/TBSRMAN.DBFchannel ORA_DISK_1: reading from backup piece /RMAN/DBF_BKUP/DBF_09O4EBAE_1_1channel ORA_DISK_1: piece handle=/RMAN/DBF_BKUP/DBF_09O4EBAE_1_1 tag=TAG20130313T155700channel ORA_DISK_1: restored backup piece 1channel ORA_DISK_1: restore complete, elapsed time: 00:00:03

Finished recover at 13-MAR-13

RMAN>

----------------------------------------------------------------------------------------------------You can see the Alert log in which the recovery process is clearly mentioned.,----------------------------------------------------------------------------------------------------ARC0: STARTING ARCH PROCESSES COMPLETE

Page 9: Recover Standby Database From a Missing Archive Log Incremental Backup

Lost write protection disabledARC2: Becoming the active heartbeat ARCHThu Mar 14 15:26:28 2013Completed: alter database mountThu Mar 14 15:26:34 2013Using STANDBY_ARCHIVE_DEST parameter default value as /vol2/sdb/archThu Mar 14 15:27:36 2013Incremental restore complete of datafile 4 /VOL2/SDB/USERS01.DBF  checkpoint is 1084371  last deallocation scn is 3Incremental restore complete of datafile 3 /VOL2/SDB/UNDOTBS01.DBF  checkpoint is 1084371  last deallocation scn is 1025328Incremental restore complete of datafile 5 /VOL2/SDB/TBSRMAN.DBF  checkpoint is 1084371  last deallocation scn is 994065Incremental restore complete of datafile 2 /VOL2/SDB/SYSAUX01.DBF  checkpoint is 1084371  last deallocation scn is 1005551Incremental restore complete of datafile 1 /VOL2/SDB/SYSTEM01.DBF  checkpoint is 1084371  last deallocation scn is 1004787Thu Mar 14 15:28:03 2013alter database recover managed standby database disconnect from sessionThu Mar 14 15:28:03 2013MRP0 started with pid=28, OS id=3188 started logmerger processThu Mar 14 15:28:08 2013Managed Standby Recovery not using Real Time ApplyParallel Media Recovery started with 4 slavesWaiting for all non-current ORLs to be archived...All non-current ORLs have been archived.Media Recovery Waiting for thread 1 sequence 32Completed: alter database recover managed standby database disconnect from sessionThu Mar 14 15:28:56 2013RFS[1]: Assigned to RFS process 1416RFS[1]: Opened log for thread 1 sequence 32 dbid -1062274717 branch 809697765Archived Log entry 1 added for thread 1 sequence 32 rlc 809697765 ID 0xc0af5d63 dest 2:Thu Mar 14 15:29:00 2013Media Recovery Log /VOL2/SDB/ARCH/ARC0000000032_0809697765.0001Media Recovery Waiting for thread 1 sequence 33Thu Mar 14 15:29:16 2013----------------------------------------------------------------------------------------------------

h. Start the media recovery process in standby database

     alter database recover managed standby database disconnect from session;

SQL> alter database recover managed standby database disconnect from session;

i. Check the archive log list on primary and standby databases.,

ON Primary :------------

SQL> archive log listDatabase log mode              Archive ModeAutomatic archival             EnabledArchive destination            /vol1/pdb/archOldest online log sequence     31Next log sequence to archive   33Current log sequence           33

ON Standby :------------

SQL> archive log listDatabase log mode              Archive ModeAutomatic archival             EnabledArchive destination            /vol2/sdb/archOldest online log sequence     31Next log sequence to archive   0Current log sequence           33

j. Cancel the recovery mode in Standby Database and open the database for read only access. Start the media recovery process.

SQL> alter database recover managed standby database cancel;

Database altered.

SQL> alter database open read only;

Page 10: Recover Standby Database From a Missing Archive Log Incremental Backup

Database altered.

SQL> select sequence#,archived,applied from v$archived_log;

 SEQUENCE# ARC APPLIED---------- --- ---------        32 YES YES

ON Primary Database:--------------------

SQL> alter system switch logfile;

System altered.

SQL> archive log listDatabase log mode              Archive ModeAutomatic archival             EnabledArchive destination            /vol1/pdb/archOldest online log sequence     31Next log sequence to archive   33Current log sequence           33

ON Standby Database:--------------------

SQL> select sequence#,archived,applied from v$archived_log;

 SEQUENCE# ARC APPLIED---------- --- ---------        32 YES YES        33 YES NO

SQL> select sequence#,archived,applied from v$archived_log;

 SEQUENCE# ARC APPLIED---------- --- ---------        32 YES YES        33 YES NO

SQL> alter database recover managed standby database disconnect from session;

Database altered.

SQL> select sequence#,archived,applied from v$archived_log;

 SEQUENCE# ARC APPLIED---------- --- ---------        32 YES YES        33 YES IN-MEMORY

SQL> select sequence#,archived,applied from v$archived_log;

 SEQUENCE# ARC APPLIED---------- --- ---------        32 YES YES        33 YES YES

k. Now finally check the synchronization of primary and standby databases.

ON Primary Database:--------------------

SQL> alter system switch logfile;

System altered.

SQL> archive log listDatabase log mode              Archive ModeAutomatic archival             EnabledArchive destination            /vol1/pdb/archOldest online log sequence     32Next log sequence to archive   34Current log sequence           34

ON Standby Database:--------------------

SQL> archive log listDatabase log mode              Archive ModeAutomatic archival             EnabledArchive destination            /vol2/sdb/archOldest online log sequence     32

Page 11: Recover Standby Database From a Missing Archive Log Incremental Backup

Next log sequence to archive   0Current log sequence           34

From the above check, we can clearly see that both primary and standby databases are in sync.