12
Backup and Recovery-Exchange 2010 DAG 1 Backup and Recovery Exchange 2010 DAG By Francis Sabarinath [frasa03]

Backup and Recovery-Exchange 2010 DAG.pdf - Zendesk · Backup and Recovery-Exchange 2010 DAG 3 BACKUP AND RECOVERY – EXCHANGE 2010 DAG 1. Exchange Backup Microsoft provides 2 ways

  • Upload
    dinhdan

  • View
    223

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Backup and Recovery-Exchange 2010 DAG.pdf - Zendesk · Backup and Recovery-Exchange 2010 DAG 3 BACKUP AND RECOVERY – EXCHANGE 2010 DAG 1. Exchange Backup Microsoft provides 2 ways

Backup and Recovery-Exchange 2010 DAG

1

Backup and Recovery – Exchange 2010 DAG

By Francis Sabarinath [frasa03]

Page 2: Backup and Recovery-Exchange 2010 DAG.pdf - Zendesk · Backup and Recovery-Exchange 2010 DAG 3 BACKUP AND RECOVERY – EXCHANGE 2010 DAG 1. Exchange Backup Microsoft provides 2 ways

Backup and Recovery-Exchange 2010 DAG

2

Table of Contents BACKUP AND RECOVERY – EXCHANGE 2010 DAG ........................................................................................ 3

1. Exchange Backup................................................................................................................................... 3

2. Backup Sequence Diagram – using Store Writer .................................................................................. 3

3. Backup Sequence Diagram – using Replica Writer ............................................................................... 4

4. Difference between VSS backup of Exchange 2010 using Windows Server Backup and that done by

CA ARCserve Agent for Microsoft Exchange ............................................................................................. 4

5. Important Tool for Troubleshooting Exchange Backup issues – MS Event Viewer .............................. 5

6. User Problems related to Backup ......................................................................................................... 8

6.1 User Problem 1 -Exchange Database Level Backup fails ................................................................. 8

6.2 User Problem 2 -Exchange Database Level Backup fails - Client halt error .................................... 8

6.3 User Problem 3 - Transaction Logs not truncated .......................................................................... 9

6.4 User Problem 4 – Unable to find Exchange Database .................................................................. 10

7. Exchange Restore ................................................................................................................................ 10

8. Restore Sequence Diagram – only Store Writer is used and not Replica Writer ................................ 11

9. User Problems related to Restore ....................................................................................................... 11

9.1 User Problem 1-Restore using Recovery Database – How to extract the Mailbox ...................... 11

9.2 User Problem 2 –Unable to find the Recovery Database ............................................................. 12

Page 3: Backup and Recovery-Exchange 2010 DAG.pdf - Zendesk · Backup and Recovery-Exchange 2010 DAG 3 BACKUP AND RECOVERY – EXCHANGE 2010 DAG 1. Exchange Backup Microsoft provides 2 ways

Backup and Recovery-Exchange 2010 DAG

3

BACKUP AND RECOVERY – EXCHANGE 2010 DAG

1. Exchange Backup Microsoft provides 2 ways of backup – legacy streaming and VSS. Legacy streaming backup is not

supported with Exchange 2010. The VSS communicates with the Windows Server File system and with

the mass storage device driver through a third-party (or customer) provider. The backup/restore

application is also called the VSS Requestor. The requestor communicates with the VSS to obtain

information about the Exchange 2010, to command the creation of shadow copies and to gain access to

the data for backup. To support the VSS, Exchange 2010 includes an Exchange writer that it built into the

Exchange Store. The Exchange Writer coordinates with the Exchange Store, to freeze and dismount the

database before backing it up, and then to unfreeze and mount the database after the backup is

complete. Backup from the active copy uses the Store Writer, and backup from the replica (copy

database) uses the Replica Writer. It is good to understand the Backup sequence using the Store and the

Replica writer which is diagrammatically provided in the following section.

2. Backup Sequence Diagram – using Store Writer

Page 4: Backup and Recovery-Exchange 2010 DAG.pdf - Zendesk · Backup and Recovery-Exchange 2010 DAG 3 BACKUP AND RECOVERY – EXCHANGE 2010 DAG 1. Exchange Backup Microsoft provides 2 ways

Backup and Recovery-Exchange 2010 DAG

4

3. Backup Sequence Diagram – using Replica Writer

Ref: http://msdn.microsoft.com/en-us/library/exchange/aa579076(v=exchg.140).aspx

4. Difference between VSS backup of Exchange 2010 using Windows Server

Backup and that done by CA ARCserve Agent for Microsoft Exchange

The sequence of backup will be the same except that a physical consistencies check that will be initiated

by ARCserve Exchange Agent. Our agent uses the exported functions from chksgfiles.dll to check the

consistency.

The CHKSGFILES API ships as part of Exchange 2010. The API can be used on a computer that has

Mailbox server role installed, or a computer that has the Administrator-only install. CHKSGFILES is

available only as a 64-bit DLL in Exchange 2010. By default, the CHKSGFILES DLL is installed in the

C:\Program Files\Exchange 2010\bin directory.

The backup/restore application may choose to validate the integrity of a shadow copy backup set.

During a backup operation managed by the Volume Shadow Copy Service, there is no opportunity for

Microsoft Exchange Server 2010 to read each database file in its entirety and verify its checksum

integrity. Therefore, database and transaction log file integrity should be verified by the backup

application. Microsoft recommends that the backup application verify the physical consistency of the

Page 5: Backup and Recovery-Exchange 2010 DAG.pdf - Zendesk · Backup and Recovery-Exchange 2010 DAG 3 BACKUP AND RECOVERY – EXCHANGE 2010 DAG 1. Exchange Backup Microsoft provides 2 ways

Backup and Recovery-Exchange 2010 DAG

5

shadow copy set prior to informing the Exchange Writer that the backup is complete. Upon successful

backup, Exchange store updates the headers of the backed-up databases to reflect the last successful

backup times and removes transaction logs from the server that are no longer required to roll forward

from the last successful backup.

In order to provide better granularity and control during the consistency check process, Exchange 2010

includes a set of unmanaged public APIs packaged into a DLL (chksgfiles.dll). The CHKSGFILES DLL is

installed with all Exchange server roles as well as part of the Admin-only option to enable running

consistency checks off of an Exchange server on a proxy backup machine.

More Reference:

http://msdn.microsoft.com/en-us/library/bb204049.aspx

http://msdn.microsoft.com/en-us/library/bb204219.aspx

5. Important Tool for Troubleshooting Exchange Backup issues – MS Event

Viewer Microsoft Event Viewer (application events) is an important tool to be used for troubleshooting an

Exchange Backup. All the events will be recorded in sequence – from the Backup initiation till the end.

Sequence of the Exchange Backup can be understood by looking at the application event during an

Exchange VSS backup. Provided below is a sample set of application events, from a backup taken from

the replica, and it is important to understand this sequence for troubleshooting purpose:

Replica Events:

SNo TIME SOURCE EVENT ID

TASK Category

DESCRIPTION

1 3:27:34 AM MSExchangeRepl 2021 Exchange VSS Writer

The Microsoft Exchange VSS Writer has successfully collected the metadata document in preparation for backup.

2 3:27:38 AM MSExchangeRepl 2110 Exchange VSS Writer

The Microsoft Exchange VSS Writer instance cfda266f-d071-47fd-9295-8ec31213f378 has successfully prepared for a full or a copy backup of database 'FRANCISTEST'. The following database will be backed up: FRANCISTEST.

3 3:27:38 AM MSExchangeRepl 2023 Exchange VSS Writer

The Microsoft Exchange Replication service VSS Writer (Instance cfda266f-d071-47fd-9295-8ec31213f378) successfully prepared for backup.

4 3:27:49 AM MSExchangeIS Public Store

9539 General General "The Microsoft Exchange Information Store database ""<NULL>"" was stopped. “

5 3:27:49 AM ESE BACKUP 960 General msexchangerepl (2444) This computer is performing a surrogate backup. The master server is USSDWIN2K8R1.

6 3:27:49 AM MSExchangeRepl 2027 Exchange VSS Writer

The Microsoft Exchange VSS Writer instance cfda266f-d071-47fd-9295-8ec31213f378 has successfully frozen the databases.

7 3:27:49 AM MSExchangeRepl 2025 Exchange VSS Writer

The Microsoft Exchange Replication service VSS Writer (Instance cfda266f-d071-47fd-9295-8ec31213f378) successfully prepared for a snapshot.

8 3:27:49 AM MSExchangeRepl 2027 Exchange VSS Writer

The Microsoft Exchange VSS Writer instance cfda266f-d071-47fd-9295-8ec31213f378 has successfully frozen the databases.

Page 6: Backup and Recovery-Exchange 2010 DAG.pdf - Zendesk · Backup and Recovery-Exchange 2010 DAG 3 BACKUP AND RECOVERY – EXCHANGE 2010 DAG 1. Exchange Backup Microsoft provides 2 ways

Backup and Recovery-Exchange 2010 DAG

6

9 3:27:49 AM MSExchangeRepl 2029

Exchange VSS Writer

The Microsoft Exchange VSS Writer instance cfda266f-d071-47fd-9295-8ec31213f378 has successfully thawed the databases.

10 3:27:50 AM MSExchangeRepl 2035 Exchange VSS Writer

The Microsoft Exchange Replication service VSS Writer (Instance cfda266f-d071-47fd-9295-8ec31213f378) has successfully processed the post-snapshot event.

11 3:27:54 AM Storage Group Consistency Check

100 Initialization Instance 1: A physical consistency check has been successfully initiated. Database '\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\Program Files\Microsoft\Exchange Server\V14\Mailbox\FRANCISTEST\FRANCISTEST.edb' and the transaction log files in '\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\Program Files\Microsoft\Exchange Server\V14\Mailbox\FRANCISTEST\' with a base name of 'E05' will be validated.

12 3:27:54 AM Storage Group Consistency Check

200 Database Header Validation

Instance 1: Database headers have been successfully validated. All databases are in a dirty shutdown state. To bring these databases to a clean shutdown state, log generations 3 (0x00000003) to 3 (0x00000003) will be required.

13 3:27:57 AM Storage Group Consistency Check

300 Log File Validation

Instance 1: Transaction log files in '\\? \GLOBALROOT\Device\HarddiskVolumeShadowCopy2\Program Files\Microsoft\Exchange Server\V14\Mailbox\FRANCISTEST\' with a base name of 'E05' have been successfully validated. No errors were detected.

14 3:27:57 AM ESE 103 General "caagstart (788) Instance 1: Log File Integrity Check (\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\Program Files\Microsoft\Exchange Server\V14\Mailbox\FRANCISTEST\E05): The database engine stopped the instance (0). Internal Timing Sequence: [1] 0.000, [2] 0.000, [3] 0.000, [4] 0.000, [5] 0.000, [6] 0.000, [7] 0.000, [8] 0.000, [9] 0.000, [10] 0.000."

15 3:27:57 AM ESE 105 General "caagstart (788) Instance 1: Log File Integrity Check (\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\Program Files\Microsoft\Exchange Server\V14\Mailbox\FRANCISTEST\E05): The database engine started a new instance (0). (Time=0 seconds) Internal Timing Sequence: [1] 0.000, [2] 0.000, [3] 0.000, [4] 0.000, [5] 0.000, [6] 0.000, [7] 0.000, [8] 0.000, [9] 0.000, [10] 0.000, [11] 0.000.“

16 3:27:57 AM ESE 102 General "caagstart (788) Instance 1: Log File Integrity Check (\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\Program Files\Microsoft\Exchange Server\V14\Mailbox\FRANCISTEST\E05): The database engine (14.00.0692.0000) is starting a new instance (0).

17 3:27:57 AM Storage Group Consistency Check

400 Termination Instance 1: The physical consistency check has completed successfully. All files were validated and no errors were detected.

18 3:28:47 AM MSExchangeRepl 2021 Exchange VSS Writer

The Microsoft Exchange VSS Writer has successfully collected the metadata document in preparation for backup.

19 3:29:03 AM ESE BACKUP 963 General msexchangerepl (2444) The surrogate backup procedure to USSDWIN2K8R1 has been successfully completed.

20 3:29:03 AM MSExchangeRepl 2046 Exchange VSS Writer

"The Microsoft Exchange Replication service VSS Writer instance cfda266f-d071-47fd-9295-8ec31213f378 has successfully completed the backup of database 'FRANCISTEST'. Database log truncation has been requested for this database. Log truncation will occur on the active copy after the next log generation is created. Log truncation will occur automatically on the passive copies after that log file is copied."

21 3:29:03 AM MSExchangeRepl 2033 Exchange The Microsoft Exchange Replication service VSS Writer (Instance cfda266f-

Page 7: Backup and Recovery-Exchange 2010 DAG.pdf - Zendesk · Backup and Recovery-Exchange 2010 DAG 3 BACKUP AND RECOVERY – EXCHANGE 2010 DAG 1. Exchange Backup Microsoft provides 2 ways

Backup and Recovery-Exchange 2010 DAG

7

VSS Writer d071-47fd-9295-8ec31213f378) has successfully processed the backup completion event.

22 3:29:03 AM MSExchangeRepl 2037 Exchange VSS Writer

The Microsoft Exchange Replication service VSS Writer (Instance cfda266f-d071-47fd-9295-8ec31213f378) backup has been successfully shut down.

Active Events:

SNo TIME SOURCE EVENT ID TASK Category DESCRIPTION

1 3:27:51 AM ESE 210 Logging/Recovery "Information Store (3400) FRANCISTEST: A full backup is starting.

2 3:27:51 AM ESE BACKUP 910 General Information Store (3400) A full surrogate backup is starting. This server is the master. The requestor is USSDWIN2K8R1.

3 3:29:04 AM MSExchangeRepl 3156 Service Active Manager successfully mounted database FRANCISTEST on server ussdwin2k8r1.gotir.ca.com.

4 3:29:04 AM ESE BACKUP 913 General Information Store (3400) The surrogate backup procedure has been successfully completed by USSDWIN2K8R1.

5 3:29:04 AM ESE 213 Logging/Recovery "Information Store (3400) FRANCISTEST: The backup procedure has been successfully completed.

Will the transaction logs be purged from the Active or the Replica – when an Exchange backup is

performed via the Replica?

The log truncation will happen on the Active Database only and not from the Copy databases.

Once the backup is completed from the Replica, it will signal the Active database. The logs will be

truncated from the Active database and sync will take place with the copies, which will affect the

transaction logs on the Copy databases.

Backup initiated transaction log file truncation will be triggered based on the type of backup chosen. In

non-DAG configurations, the Store Writer will truncate the transaction log files at the completion of

successful Full or Incremental backups. In DAG replicated configurations, log truncation will be delayed

by the Replication service until all necessary log files are replayed into all other copies. The Replication

service will delete the backed up log files both from the active and the copy log file paths after it verifies

that the to-be-deleted log files have successfully been applied to the copy database and both active

database and the database copies checkpoint has passed the log files to be deleted.

Log truncation doesn't occur on the active mailbox database copy when one or more passive copies are

suspended. If your planned maintenance activities are going to take an extended period of time (for

example, several days), you may have considerable log file buildup. To prevent the log drive from filling

Page 8: Backup and Recovery-Exchange 2010 DAG.pdf - Zendesk · Backup and Recovery-Exchange 2010 DAG 3 BACKUP AND RECOVERY – EXCHANGE 2010 DAG 1. Exchange Backup Microsoft provides 2 ways

Backup and Recovery-Exchange 2010 DAG

8

up with transaction logs, you can remove the affected passive database copy instead of suspending it.

When the planned maintenance is completed, you can re-add the passive database copy.

6. User Problems related to Backup

6.1 User Problem 1 -Exchange Database Level Backup fails

From the Application event, check at which step the backup failed.

Run a Windows Server Backup on that mailbox database to narrow down the case.

Refer the link below on how to run a VSS backup of Exchange using Windows Server Backup:

http://technet.microsoft.com/en-us/library/dd876854(v=exchg.141).aspx

6.2 User Problem 2 -Exchange Database Level Backup fails - Client halt error

The following errors will be observed from the Application event and it will occur during the backup

(after sequence 18 from the Application Events in section 5)

Active Events: Event ID: 215

Source: ESE Category: Logging/Recovery Level: Error Message: "Information Store (3084) EXCHDB1_08: The backup has been stopped because it was halted by the client or the connection with the client failed.

Replica Events: Source: ESE BACKUP Event ID: 914 Category: General Level: WarningMessage: General Information Store (3084) The surrogate backup by VMEXCHANGEMBS1A has stopped with error 0xFFFFFFFF.

In this case as well, run a Windows Server Backup of Exchange, as explained above to narrow down the case. If the Windows Server backup is successful, then do the steps below to disable TCP Offloading

TCP Offloading has to be disabled on the NIC from the Backup Server and the Exchange Nodes.

On the Backup servers:

IPv4 Checksum Offload Off

IPv4 Large Send Offload Disabled

IPv6 Checksum Offload Off

IPv6 Large Send Offload Disabled

Page 9: Backup and Recovery-Exchange 2010 DAG.pdf - Zendesk · Backup and Recovery-Exchange 2010 DAG 3 BACKUP AND RECOVERY – EXCHANGE 2010 DAG 1. Exchange Backup Microsoft provides 2 ways

Backup and Recovery-Exchange 2010 DAG

9

Receive Side Scaling Disabled

On the Exchange Nodes:

IPv4 Checksum Offload Disabled

IPv4 Giant TSO Offload Disabled

IPv4 TSO Offload Disabled

IPv4 TCP Segmentation Offload Disabled

Offload IP Options Disabled

TCP Checksum Offload (IPv4) Disabled

TCP Checksum Offload (IPv6) Disabled

6.3 User Problem 3 - Transaction Logs not truncated

Truncation of Transaction logs takes place after a full or an incremental backup of the Exchange

Database. If there is an issue where the transaction logs are not getting truncated, then the step is to

run a Windows Server backup of the Exchange Database to check if the logs are getting truncated.

If the transaction log truncation does not take place from the Windows Server Backup of the Exchange,

then check the status of the copy databases, in case of an Exchange 2010 DAG.

Log truncation doesn't occur on the active mailbox database copy when one or more passive copies are

suspended. If your planned maintenance activities are going to take an extended period of time (for

example, several days), you may have considerable log file buildup. To prevent the log drive from filling

up with transaction logs, you can remove the affected passive database copy instead of suspending it.

When the planned maintenance is completed, you can re-add the passive database copy.

Page 10: Backup and Recovery-Exchange 2010 DAG.pdf - Zendesk · Backup and Recovery-Exchange 2010 DAG 3 BACKUP AND RECOVERY – EXCHANGE 2010 DAG 1. Exchange Backup Microsoft provides 2 ways

Backup and Recovery-Exchange 2010 DAG

10

6.4 User Problem 4 – Unable to find Exchange Database

When the backup is initiated, you will get an error as “E3577 Cannot find DB on Exchange server”

In such cases, run the adsiedit.msc and use the same domain controller and credentials as provided in

the ARCserve Backup UI. Mostly, there will be ldap connectivity issues, or the Exchange containers

cannot be seen. The solution is to use a different domain controller or credentials, so that the Exchange

objects can be viewed correctly.

A sample screenshot of the adsiedit.msc is provided below:

7. Exchange Restore During a restore, the backup/restore application instructs the Exchange Writer to coordinate with the

Exchange Store to dismount the database, replace the database files, mount the database and then

replay the transaction logs as needed. When restoring, the requestor also communicates with the VSS to

prepare the system for the restore, and then to put the data back on to the mass storage device. The

backup/restore application is also responsible for working with the Windows Server to read data from

and write data to the backup storage media, whether a tape archive, a storage area network, or the

other backup medium. The restored database can be mounted either as a regular, active database, or as

the Exchange 2010 Recovery Database. Only one mounted database can be designated as a recovery

database on each Exchange Server.

Page 11: Backup and Recovery-Exchange 2010 DAG.pdf - Zendesk · Backup and Recovery-Exchange 2010 DAG 3 BACKUP AND RECOVERY – EXCHANGE 2010 DAG 1. Exchange Backup Microsoft provides 2 ways

Backup and Recovery-Exchange 2010 DAG

11

8. Restore Sequence Diagram – only Store Writer is used and not Replica

Writer

Ref: http://msdn.microsoft.com/en-us/library/exchange/bb204079(v=exchg.140).aspx

9. User Problems related to Restore

9.1 User Problem 1-Restore using Recovery Database – How to extract the Mailbox

After the mailbox database is restored to the Recovery Database, ensure that it is mounted.

Mount the Recovery Database either from Exchange Management Console or from Exchange Management Shell by running the command Mount-Database "Recovery Database"

Make sure the destination mailbox is enabled and existing in the Exchange Server. Make sure the database corresponding to the destination mailbox is mounted.

Before proceeding to recover the mailbox contents from Recovery Database, run the following command to get the details of mailboxes available in the Recovery Database.

Get-MailboxStatistics –Database "Recovery Database"

Page 12: Backup and Recovery-Exchange 2010 DAG.pdf - Zendesk · Backup and Recovery-Exchange 2010 DAG 3 BACKUP AND RECOVERY – EXCHANGE 2010 DAG 1. Exchange Backup Microsoft provides 2 ways

Backup and Recovery-Exchange 2010 DAG

12

The above command can be run to make sure that the mailbox that we are trying to restore is existing in the

Recovery Database.

Restore a single mailbox

1. Run the following command to restore the content of mailbox 'production mailbox' in the Recovery

Database to the production mailbox:

Restore-Mailbox –Identity "production mailbox" –RecoveryDatabase "Recovery Database"

2. Say 'Yes' for the confirmation asked. Restore process will start immediately.

3. When completed, you will get statistics about the restore mailbox job. You can see the information about

the source and target mailbox database etc.

Restore a single mailbox to a different mailbox

1. Run the following command to restore the content of 'production mailbox' in the Recovery Database to

'test mailbox' under the folder 'Recovered Contents Folder' within 'spare mailbox':

Restore-Mailbox -Identity "test mailbox" -RecoveryDatabase "Recovery Database" -

RecoveryMailbox "production mailbox" -TargetFolder "Recovered Contents Folder"

NOTE : The above command will create a new folder "Recovered Contents Folder" inside "test mailbox"

and will restore the content of "production mailbox" into the new folder "Recovered Contents Folder"

2. Say 'Yes' for the confirmation asked. Restore process will start immediately.

3. When completed, you will get statistics about the restore mailbox job. You can see the information about

the source and target mailbox database etc.

http://msexchangeguru.com/2010/05/23/exchange2010-recovery-database/

http://technet.microsoft.com/en-us/library/ee332351(v=exchg.141).aspx

9.2 User Problem 2 –Unable to find the Recovery Database

When a restore to a Recovery Database is attempted, it will fail with “unable to find the Recovery

Database”. In order to narrow down the problem do the following:

Have the adsiedit.msc installed. Then connect to the same Domain Controller and check if the

containers can be displayed, and check if the “Recovery Database” created can be viewed from the

Active Directory (Refer 6.4 User Problem 4 for a screenshot of the adsiedit.msc)