Embed Size (px)
Citation preview
Exchange 2003 Backup Restore & Recovery
Exchange 2003 Backup Restore & Recovery
Ronen GabbayRonen Gabbay
► ► Microsoft Exchange Regional Director ► Exchange MVPMicrosoft Exchange Regional Director ► Exchange MVP
► ► Microsoft Secure and Well Managed Infrastructure Specialist ► Microsoft IsraelMicrosoft Secure and Well Managed Infrastructure Specialist ► Microsoft Israel
► ► Microsoft Exchange Server User Group leader Microsoft Exchange Server User Group leader
► ► E-mail [email protected] [email protected]
INF 311
AgendaAgendaAre you awareAre you aware
Exchange is a mission critical applicationExchange is a mission critical application
How to prepare:How to prepare:
Exchange Database ArchitectureExchange Database Architecture
Backup and RestoreBackup and Restore
Recovery Storage GroupRecovery Storage Group
Dial Tone RecoveryDial Tone Recovery
Exchange Snapshot ArchitectureExchange Snapshot Architecture
How to avoid and what to do when things go badHow to avoid and what to do when things go bad
Exchange Errors and how to avoid themExchange Errors and how to avoid them
ESEUtil & ISInteg and all in betweenESEUtil & ISInteg and all in between
Server & Alternate Server Restore procedureServer & Alternate Server Restore procedure
Exchange is a mission critical application
Exchange is a mission critical application
Imagine how would life be without ExchangeImagine how would life be without Exchange
No inbound nor outbound mailsNo inbound nor outbound mails
No CalendarNo Calendar
No mails between company usersNo mails between company users
Have you considered how much money your Have you considered how much money your company losescompany loses
How long can you bear to be without ExchangeHow long can you bear to be without Exchange
Have you have any SLAHave you have any SLA
Have you got any written policy in case……Have you got any written policy in case……
Are you prepared for the worstAre you prepared for the worst
STORESTORE
LOGLOG
EDBEDBEDBEDB
STMSTMSTMSTM
LOGLOGLOGLOG
EDBEDBEDBEDB
STMSTMSTMSTM
ESE InstanceESE InstanceESE InstanceESE Instance
Storage Group 1Storage Group 1
LOGLOG
EDBEDBEDBEDB
STMSTMSTMSTM
LOGLOGLOGLOG
EDBEDBEDBEDB
STMSTMSTMSTM
ESE InstanceESE InstanceESE InstanceESE Instance
Storage Group 2Storage Group 2
EDBEDBEDBEDB
STMSTMSTMSTM
Storage GroupsStorage Groups
Exchange 2003 DatabasesExchange 2003 Databases
*.EDB Files*.EDB Files
Properties DatabaseProperties Database
MAPI Messages and AttachmentsMAPI Messages and Attachments
Headers for STM pagesHeaders for STM pages
*.STM Files*.STM Files
Raw ‘streaming’ data (MIME, documents, multimedia, Raw ‘streaming’ data (MIME, documents, multimedia, etc)etc)
Current database = EDB + STM + Unflushed Log Current database = EDB + STM + Unflushed Log entriesentries
Storing DataStoring DataTransaction
Log File
7155
254
15841
10
Database File
4 KB4 KB
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
18 19 20 21
22 23 24 25
Memory
7 15 25 4
17 8 3 1
10
Normal OperationNormal Operation
ESE Database ConsistencyESE Database Consistency
Memory
7 15 25 4
17 8 3 1
10
Memory
7 15 25 4
17 8 3 1
10
Database File
4 KB4 KB
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
18 19 20 21
22 23 24 25
7 15 25 4
17 8 3 1
10
TransactionLog File
7155
254
15841
10
Normal ShutdownNormal Shutdown
Database File
4 KB4 KB
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
18 19 20 21
22 23 24 25
Memory
7 15 25 4
17 8 3 1
10
TransactionLog File
7155
254
15841
10
Improper ShutdownImproper Shutdown
Database File
4 KB4 KB
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
18 19 20 21
22 23 24 25
Memory
7 15 25 4
17 8 3 1
10
TransactionLog File
7155
254
15841
10
Normal OperationNormal Operation
Database File
4 KB4 KB1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
18 19 20 21
22 23 24 25
Memory
7 15 25 4
17 8 3 1
10
TransactionLog File
7155
254
15841
10
Previous LogsPrevious Logs
Current LogCurrent Log
EXX00001.LOGEXX00002.LOG
.
.
.
Previous Log FilesPrevious Log FilesCurrent Logs Are Renamed After 5 MB of Data Is AcceptedCurrent Logs Are Renamed After 5 MB of Data Is Accepted
EXX00003.LOG
Edb.log
RenamedRenamedRenamedRenamed3 MB5 MB
New EXX.log(5 MB)
EXX0004.logEXX0003.logEXX0002.log
7155
25
110
41783
EXX0001.log
9442
18
1235
6131122
14401699
5221
19333120
4152342
6148
26344629
EXX0004.logEXX0003.logEXX0002.log
Circular LoggingCircular Logging
7155
25
110
41783
EXX0001.log
9442
18
1235
6131122
14401699
5221
19333120
4152342
6148
26344629
EXX0005.log
27306253
8271
36709447
In most cases Circular Logging should NOT be enabled
Checkpoint FileCheckpoint File
EXXnnnnn.log
EXX.chk
Transaction LogEntries Writtento the Database
Transaction LogEntries Not Yet Written to the
Database
Starting an ESE Database Starting an ESE Database
Locating the CheckpointLocating the Checkpoint
Check File Signatures matchCheck File Signatures match
Check Database File ConsistencyCheck Database File Consistency
Recovering a Database at StartupRecovering a Database at Startup
Online MaintenanceOnline Maintenance
Not clash with backup schedulesNot clash with backup schedules
Database size, Backup/Restore SLAsDatabase size, Backup/Restore SLAs
Purge Indices (public and mailbox stores) Purge Indices (public and mailbox stores)
Tombstone Maintenance (public and mailbox stores) Tombstone Maintenance (public and mailbox stores)
Dumpster Cleanup (public and mailbox stores) Dumpster Cleanup (public and mailbox stores)
Cleanup Deleted Mailboxes (mailbox stores)Cleanup Deleted Mailboxes (mailbox stores)
Events for each Database processedEvents for each Database processed
700 Defrag Started700 Defrag Started
701 Defrag Ended701 Defrag Ended
1221 The Defrag ended and the amount of free space in file1221 The Defrag ended and the amount of free space in file
Exchange & Active Directory ™ 2003 Exchange & Active Directory ™ 2003
Active Directory ™Active Directory ™
Directory stores Exchange server informationDirectory stores Exchange server information
The Exchange configuration is stored in theThe Exchange configuration is stored in theActive Directory configuration partitionActive Directory configuration partition
Mailboxes are not directory objectsMailboxes are not directory objects
A Mailbox is an attribute of a userA Mailbox is an attribute of a user
Mailboxes can be reconnected to other usersMailboxes can be reconnected to other users
Mailbox retention timeMailbox retention time
Deleted Items retention timeDeleted Items retention time
Dumping Headers using ESEUtilDumping Headers using ESEUtil
Configure Exchange EVS on ClusterConfigure Exchange EVS on Cluster
Exchange 2003 Backup Process flowExchange 2003 Backup Process flow
Backup APIs CalledBackup APIs CalledBackup APIs CalledBackup APIs Called
ESE Backup ModeESE Backup ModeESE Backup ModeESE Backup Mode Begin BackupBegin Backup• Backup program calls the Backup program calls the Exchange backup API Exchange backup API• The checkpoint file freezeThe checkpoint file freeze• Online maintenance pausesOnline maintenance pauses
• • Store informs ESE Store informs ESE each database to each database to backup backup
• • Agent requests DB pagesAgent requests DB pages sequentially 64K at timesequentially 64K at time• • Pages are check-summedPages are check-summed as readas read
End BackupEnd Backup
• • Page read completedPage read completed• • Logs copied to tapeLogs copied to tape• • Truncate logs Truncate logs • • Backup set closedBackup set closed
ESE Normal ModeESE Normal ModeESE Normal ModeESE Normal ModeBackup CompleteBackup CompleteBackup CompleteBackup Complete
• The checkpoint file unfreezesThe checkpoint file unfreezes• Online maintenance resumeOnline maintenance resume
BackupBackup
The IS must be running and all databases must be The IS must be running and all databases must be mountedmounted
Support for storage groups and for a single storeSupport for storage groups and for a single store
Best practice backup the entire storage group for Best practice backup the entire storage group for logs truncationlogs truncation
Concurrent backup/restore are supportedConcurrent backup/restore are supported
Truncate Log Files (*.log)Truncate Log Files (*.log)
Check-summingCheck-summing
Database files (*.edb)Database files (*.edb)
Stream files (*.stm)Stream files (*.stm)
Backup TypesBackup Types
Full(Normal)Full(Normal)
Copy/DailyCopy/Daily
IncrementalIncremental
DifferentialDifferential
SnapshotSnapshot
OfflineOffline
XX
XX
XX
XX
XX
XX
XX
XX
Type Copies DB Copy Logs Truncates LogsType Copies DB Copy Logs Truncates Logs
XX
XX
XX
Not Not recommendedrecommended
Dismount DatabaseDismount DatabaseDismount DatabaseDismount DatabaseESE Restore ModeESE Restore ModeESE Restore ModeESE Restore Mode Begin RestoreBegin Restore
End RestoreEnd RestoreESE Normal ModeESE Normal ModeESE Normal ModeESE Normal ModeRestore CompleteRestore CompleteRestore CompleteRestore Complete
• • Backup Backup application or application or Administrator Administrator dismounts dismounts databasedatabase
• • Store informs ESE andStore informs ESE and restore mode enteredrestore mode entered• • Restore SG createdRestore SG created
• • Agent copies EDB/STMAgent copies EDB/STM from Tape to DB path from Tape to DB path• • Log files from backup Log files from backup set are copied set are copied To temp restore locationTo temp restore location
• • Logs are processedLogs are processed By ESE restore instanceBy ESE restore instance• • Current logs processedCurrent logs processed By ESE restore instanceBy ESE restore instance• • Cleanup/Restore SG killedCleanup/Restore SG killed
• • DB is mounted by SGDB is mounted by SG• • Data deleted from Data deleted from temporary directory temporary directory
Exchange 2003 Restore Process FlowExchange 2003 Restore Process Flow
RestoreRestore
MSExchangeIS must be running to restoreMSExchangeIS must be running to restore
Databases to restore must be dismountedDatabases to restore must be dismountedand theand the
Db can be overridden by restore.Db can be overridden by restore.
Remaining databases can be mounted.Remaining databases can be mounted.
System Attendant not used in restore.System Attendant not used in restore.
Restore.env file createdRestore.env file created
Hard recovery via command lineHard recovery via command line
Eseutil /ccEseutil /cc
Restore: Restoring FilesRestore: Restoring Files
DatabasesDatabases
Backup Agent asks the Store where to place Backup Agent asks the Store where to place the database based on the the database based on the database GUIDdatabase GUID
Store has the databases placed on top of the Store has the databases placed on top of the existing databasesexisting databases
Log Log
Placed in a temporary log directoryPlaced in a temporary log directory
Location of temporary directory is specified by Location of temporary directory is specified by useruser
Restore.env file is createdRestore.env file is created
Restore.envRestore.env
Replaces Restore In Progress KeyReplaces Restore In Progress Key
Placed in temporary log directory during restorePlaced in temporary log directory during restore
Data included in Restore.envData included in Restore.env
Restore pathRestore path
Restore log file pathRestore log file path
Storage group Storage group
System parameters for the restore storage groupSystem parameters for the restore storage group
Log file rangeLog file range
Restore timeRestore time
Restore: Log File ReplayRestore: Log File Replay
Log file signature checkedLog file signature checked
Log files replayedLog files replayed
First from temporary log file locationFirst from temporary log file location
Then from running storage groupThen from running storage group
No Logs can be played if Circular Logging is enabledNo Logs can be played if Circular Logging is enabledIn this case the database will be rolled back to the date of In this case the database will be rolled back to the date of backupbackup
If there are multiple DB’s in a storage group only the If there are multiple DB’s in a storage group only the log records applicable to the failed DB are replayed, log records applicable to the failed DB are replayed, the others are ignoredthe others are ignored
On-line vs. Off-line BackupsOn-line vs. Off-line Backups
The IS must be running and the database mountedThe IS must be running and the database mounted
Page level checksum, during on-line backupPage level checksum, during on-line backup
Automatic Log files truncation indicates succesfull Automatic Log files truncation indicates succesfull backupbackup
Only online backups are SupportedOnly online backups are Supported
VSS or Off-line Backup needs manual integrity checkVSS or Off-line Backup needs manual integrity check
TIP: Moving large files (Eseutil or ESEFILE)TIP: Moving large files (Eseutil or ESEFILE)
NTBackup to disk use /FU switchNTBackup to disk use /FU switch
file unbuffered file unbuffered http://www.msexchange.org/tutorials/Optimize-Exchange-2http://www.msexchange.org/tutorials/Optimize-Exchange-2003-Performance-Part2.html003-Performance-Part2.html
Recovery Storage GroupRecovery Storage Group
Enable the Administrator to verify backup consistencyEnable the Administrator to verify backup consistency
Enables to recover a single item out of backupEnables to recover a single item out of backup
In case of a long recovery process enables an instant In case of a long recovery process enables an instant solution for mail connectivity – (Dial Tone)solution for mail connectivity – (Dial Tone)
It becomes the default location for the Backup API for It becomes the default location for the Backup API for restore purposerestore purpose
After using The RSG remove it ,or set the registry key to After using The RSG remove it ,or set the registry key to ignore it by the Backup APIignore it by the Backup API
Location: HKEY_LOCAL_MACHINE \ System \ CurrentControlSet Location: HKEY_LOCAL_MACHINE \ System \ CurrentControlSet \ Services \ MSExchangeIS \ ParametersSystem \ Services \ MSExchangeIS \ ParametersSystem Parameter: Recovery SG OverrideParameter: Recovery SG OverrideType: REG_DWORD Value: 0x00000001Type: REG_DWORD Value: 0x00000001
Dial Tone Recovery ExplainedDial Tone Recovery Explained
Restore mail service immediately; restore data later.Restore mail service immediately; restore data later.
New e-mail vs. historical dataNew e-mail vs. historical data
Why swap databases?Why swap databases?
When should you use this method?When should you use this method?
Q282496: Considerations and Best Practices When Q282496: Considerations and Best Practices When Resetting an Exchange Mailbox DatabaseResetting an Exchange Mailbox Database
Dramatically reduce EXMerged dataDramatically reduce EXMerged data
Preserve single instance storage and avoid database Preserve single instance storage and avoid database bloatbloat
Setting End User ExpectationsSetting End User Expectations
Send immediate status emailSend immediate status email
Data is being recovered, but service has been Data is being recovered, but service has been restored in the meantimerestored in the meantime
Time expectationsTime expectations
If you intend to swap back in the original database, If you intend to swap back in the original database, users should not reconfigure rules, views, offline filesusers should not reconfigure rules, views, offline files
Merge processMerge process
Outlook 2003 Exchange Recovery Mode
Outlook 2003 Exchange Recovery Mode
The MessageThe Message““Exchange is currently in recovery mode. You can either Exchange is currently in recovery mode. You can either connect to your Exchange server using the network, work connect to your Exchange server using the network, work offline, or cancel this logon”offline, or cancel this logon”
The OST problemThe OST problemLinked to a specific mailbox.Linked to a specific mailbox.Dependent on a matching key in the current user profile.Dependent on a matching key in the current user profile.Key is destroyed by connecting to a different mailbox.Key is destroyed by connecting to a different mailbox.
Q282496: Considerations and Best Practices When Q282496: Considerations and Best Practices When Resetting an Exchange Mailbox DatabaseResetting an Exchange Mailbox DatabaseExchange Recovery Mode OptionsExchange Recovery Mode Options
Offline – Access to current (Old) OST onlyOffline – Access to current (Old) OST onlyOnline _ Access to new (Rest) OST onlyOnline _ Access to new (Rest) OST only
Creating the Recovery Storage Group
Creating the Recovery Storage Group
Stop the entire storage groupStop the entire storage group
Why? So you can preserve all log filesWhy? So you can preserve all log files
COPY all log files and MOVE affected database filesCOPY all log files and MOVE affected database files
Start blank storeStart blank store
Send status message to all affected mailboxesSend status message to all affected mailboxes
Restoring to the Recovery Storage Group
Restoring to the Recovery Storage Group
If possible, set paths to the same logical drive as the If possible, set paths to the same logical drive as the original databaseoriginal databaseRecover the database in the Recovery Storage Recover the database in the Recovery Storage GroupGroupMount and dismount the database in the Recovery Mount and dismount the database in the Recovery Storage GroupStorage GroupNotify users of an impending outage Notify users of an impending outage Swap database methods:Swap database methods:
Move database files between folders on the same Move database files between folders on the same drivedriveMove database files between drivesMove database files between drivesSwap logical pathsSwap logical paths
Simple Backup & RestoreSimple Backup & Restore
Dump the restore.env fileDump the restore.env file
Using the Recovery Storage Group Using the Recovery Storage Group
Dial Tone Restore using RSGDial Tone Restore using RSG
Exchange Snapshot SupportExchange Snapshot Support
Windows Server™ 2003 (VSS service and framework)Windows Server™ 2003 (VSS service and framework)
RequestorRequestor
Third-party backup applicationsThird-party backup applications
NTBackup does not use VSS to back up Exchange NTBackup does not use VSS to back up Exchange
WriterWriter
Application-specific logic for participating in snapshot Application-specific logic for participating in snapshot process, and restore/recovery process, and restore/recovery
ProviderProvider
Third-party hardware control softwareThird-party hardware control software
Windows® includes a software providerWindows® includes a software provider
VSS components
Snapshot servicesSnapshot services
Exchange VSS BasicsExchange VSS Basics
ExWriter.dll Installed with Exchange 2003ExWriter.dll Installed with Exchange 2003
Exwriter and file system allows for a completely safe Exwriter and file system allows for a completely safe snapsnap
Fast restores (sometimes minutes)Fast restores (sometimes minutes)
During an extended snap:During an extended snap:
Clients may hour-glass during submitsClients may hour-glass during submits
Outlook 2003 in cached mode experiences zero Outlook 2003 in cached mode experiences zero production impactproduction impact
VSS and Exchange 2003 VSS and Exchange 2003
Two restore types:Two restore types:
Point in time restore (victimized restore)Point in time restore (victimized restore)
(Recovery is to (Recovery is to time of backuptime of backup))
Roll forward recoveryRoll forward recovery
(Recovery is to (Recovery is to time of failuretime of failure))
VSS can restore to:VSS can restore to:
Same locationSame location
Alternate forest and serverAlternate forest and server
What can VSS back up and restore?What can VSS back up and restore?BackupBackup
Only Read-only access is allowedOnly Read-only access is allowedStorage Group Storage Group Administration functions Administration functions prohibitedprohibitedBackup ChoicesBackup Choices
Minimum selection is the storage group (SG)Minimum selection is the storage group (SG)(to truncate log files)(to truncate log files)
Can snap multiple storage groups at the same timeCan snap multiple storage groups at the same time(best practice will be to snap individual SG)(best practice will be to snap individual SG)
On Backup Complete On Backup Complete Requestor validates Shadow image consistency using ESEUtil /kRequestor validates Shadow image consistency using ESEUtil /kLog files are truncatedLog files are truncatedStorage Group Storage Group Administration functions resumeAdministration functions resumeWriting to the database is allowedWriting to the database is allowed
RestoreRestoreRestore choicesRestore choices
Entire storage groupEntire storage groupSingle database Single database Multiple databases from a single SGMultiple databases from a single SG
VSS Best PracticeVSS Best Practice
Snapshot is not a complete backup replacement but Snapshot is not a complete backup replacement but an alternative to streaming you still need to backup an alternative to streaming you still need to backup the databasesthe databases
Makes sure requester does page check summing to Makes sure requester does page check summing to identify corruption Using Eseutil /K switchidentify corruption Using Eseutil /K switch
Exchange 2003 supports full backups and copy Exchange 2003 supports full backups and copy backupsbackupsMust restore to same logical drive letters Must restore to same logical drive letters Put each database at its own LUNPut each database at its own LUN
This allows snapshot restore of a single databaseThis allows snapshot restore of a single databaseNo native support for VSS restores to RSG No native support for VSS restores to RSG NTbackup does not support Exchange VSS NTbackup does not support Exchange VSS
Using OPC to extract data from snapshotUsing OPC to extract data from snapshot
Going from Pages to MailboxesGoing from Pages to Mailboxes
Database is composed of pages.Database is composed of pages.
Pages are linked together into B+-TreesPages are linked together into B+-Trees
B+-Trees are collected into tables.B+-Trees are collected into tables.
Tables cross-reference each other and store folders, Tables cross-reference each other and store folders, messages, mailboxes and database metadatamessages, mailboxes and database metadata
Eseutil ISInteg and all in betweenEseutil ISInteg and all in between
ESE LevelESE Level
Database is seen as tables and indexes, not folders, Database is seen as tables and indexes, not folders, messages, attachments.messages, attachments.
Application (Information Store) levelApplication (Information Store) level
Database is seen as folders, mailboxes, messages, Database is seen as folders, mailboxes, messages, etc.etc.
Eseutil understands the database at the ESE level Eseutil understands the database at the ESE level only.only.
ISInteg understands the database at the application ISInteg understands the database at the application level only.level only.
ESEUtil switchesESEUtil switches/D = Defrag Mode/D = Defrag Mode
New signaturesNew signaturesLog files mismatch, immediate backup requiredLog files mismatch, immediate backup required/T can be used to specify alternate location for the temp files/T can be used to specify alternate location for the temp files
/M = Dump headers (/ML, /MK, /MH)/M = Dump headers (/ML, /MK, /MH)/R = Recovery performs soft recovery/R = Recovery performs soft recovery/K = Integrity Check for Snapshots /K = Integrity Check for Snapshots /G = Checks Integrity at the ESE Level/G = Checks Integrity at the ESE Level
No Changes Are Made, This is a reporting Tool No Changes Are Made, This is a reporting Tool /F = Copy Mode/F = Copy Mode/C = Hard Recovery Mode (direct to restore.env)/C = Hard Recovery Mode (direct to restore.env)
/CC = Force Hard Recovery/CC = Force Hard Recovery/CM Dump the restore.env file header/CM Dump the restore.env file header
/P = Repair/P = Repair/P /P
/CreateSTM/CreateSTMWhen to perform a repairWhen to perform a repair/p /p /d /d “Isinteg –fix –test alltests” “Isinteg –fix –test alltests”
Dumping File Headers using ESEUtilDumping File Headers using ESEUtil
Database HeaderDatabase Header
Database signature, log signatureDatabase signature, log signature
State: Consistent or InconsistentState: Consistent or Inconsistent
Checkpoint HeaderCheckpoint Header
Shows checkpoint log file and signatureShows checkpoint log file and signature
Transaction Log File HeadersTransaction Log File Headers
Generation numberGeneration number
Log file signatureLog file signature
The attached databasesThe attached databases
ISIntegISInteg
-Dump: Dumps Database folders and indexes-Dump: Dumps Database folders and indexes
Isinteg –dumpIsinteg –dump
-Test: Test for integrity level errors-Test: Test for integrity level errors
-Fix: Fixes integrity level errors-Fix: Fixes integrity level errors
Isinteg –fix –test alltests.Isinteg –fix –test alltests.
Logical vs. Physical CorruptionLogical vs. Physical Corruption
You must understand the cause of the failureYou must understand the cause of the failure
Log file are missing or corruptedLog file are missing or corrupted
Hardware / Virus software etc…Hardware / Virus software etc…
Database file is corruptedDatabase file is corrupted
There are three layers of Information Store database There are three layers of Information Store database corruption corruption
Page levelPage level
ESE levelESE level
Store levelStore level
Strategies for removing corruption
Strategies for removing corruption
Restore an uncorrupted backup of the databaseRestore an uncorrupted backup of the database
If possibleIf possible
In case the backup is to old or can not be usedIn case the backup is to old or can not be used
Repair the databaseRepair the database
Expunge the corrupted pages from the databaseExpunge the corrupted pages from the database
Salvage data and generate a new databaseSalvage data and generate a new database
Repairing the DatabaseRepairing the Database
Remember that if the cause of the problems are Remember that if the cause of the problems are corrupted or lost log files then the assumption is that corrupted or lost log files then the assumption is that the database is undamaged the database is undamaged Sometimes simple check-disk would do the trickSometimes simple check-disk would do the trickRunning the Repair FunctionRunning the Repair Function
Attempts to repair linksAttempts to repair linksIf it finds physical corruption, it will delete the pageIf it finds physical corruption, it will delete the pageDatabase signature is changed if fixes are madeDatabase signature is changed if fixes are made
After Running RepairAfter Running RepairRun ISInteg -fixRun ISInteg -fixPerform a full backup of the databasePerform a full backup of the database
Lesson: Error -1018, -1019 and -1022
Lesson: Error -1018, -1019 and -1022
Error -1018: JET_errReadVerifyFailureError -1018: JET_errReadVerifyFailure
Bad checksumBad checksum
Wrong page numberWrong page number
Error -1019: JET_errPageNotInitializedError -1019: JET_errPageNotInitialized
Page expected in use is un-initialized (pgno = Page expected in use is un-initialized (pgno = 0x00000000)0x00000000)
Error -1022: JET_errDiskIOError -1022: JET_errDiskIO
Generic disk I/O failureGeneric disk I/O failure
-1018 Error ReportingExchange 2003 SP1
-1018 Error ReportingExchange 2003 SP1
Event ID : 474Event ID : 474Raw Event ID : 474Raw Event ID : 474Record Nr. : 34715Record Nr. : 34715Category : Database Page CacheCategory : Database Page CacheSource : ESESource : ESEType : ErrorType : ErrorGenerated : 10/10/2005 01:00:00Generated : 10/10/2005 01:00:00Written : 10/10/2005 01:00:00Written : 10/10/2005 01:00:00Machine : ExchangeMachine : ExchangeMessage : Message : Information Store (4884) The database page read from the Information Store (4884) The database page read from the
file "C:\EXCHSRVR\MDBDATA\C4SG2DB2.edb" at offset file "C:\EXCHSRVR\MDBDATA\C4SG2DB2.edb" at offset 14080122880 (0x00000003473da000) for 4096 (0x00001000) 14080122880 (0x00000003473da000) for 4096 (0x00001000) bytes failed verification due to a page checksum bytes failed verification due to a page checksum mismatch. The expected checksum was 1506336388 mismatch. The expected checksum was 1506336388 (0x59c8de84) and the actual checksum was 1237900932 (0x59c8de84) and the actual checksum was 1237900932 (0x49c8de84). The read operation will fail with error (0x49c8de84). The read operation will fail with error --10181018 (0xfffffc06). If this condition persists then please (0xfffffc06). If this condition persists then please restore the database from a previous backup.restore the database from a previous backup.
-1018 Error ReportingExchange 2003 SP2
-1018 Error ReportingExchange 2003 SP2
Root Causes of Error -1018?Root Causes of Error -1018?HardwareHardwareFirmwareFirmwareFile system corruptionFile system corruptionVirus Protection softwareVirus Protection softwareNot Exchange!Not Exchange!
With some exceptions--having to do with false positives and With some exceptions--having to do with false positives and negatives, not actually causing -1018snegatives, not actually causing -1018s
A -1018 would be Exchange’s fault if it:A -1018 would be Exchange’s fault if it:Constructed the wrong checksum for a pageConstructed the wrong checksum for a pageDropped a page in the wrong place in the databaseDropped a page in the wrong place in the database
This does not mean that Exchange has no corrupting bugs—but This does not mean that Exchange has no corrupting bugs—but these errors are not -1018.these errors are not -1018.How serious is a -1018? When do you see it? What’s on the How serious is a -1018? When do you see it? What’s on the page?page?
During normal operation? (somewhat serious)During normal operation? (somewhat serious)During startup? (likely fatal)During startup? (likely fatal)During backup (may be minor)During backup (may be minor)
What causes Error -1022?What causes Error -1022?
Any disk I/O failureAny disk I/O failureFile damage or truncationFile damage or truncationFile locked by another processFile locked by another processAnti-virus softwareAnti-virus softwareAlmost always fatal to the service, but does not Almost always fatal to the service, but does not necessarily indicate database damagenecessarily indicate database damageTroubleshootingTroubleshooting
Check file size/lock statusCheck file size/lock statusReboot to clear locks and other problemsReboot to clear locks and other problemsDo not conclude the database is damaged until you Do not conclude the database is damaged until you see that it issee that it is
Perform Soft Recovery Using Eseutil Perform Soft Recovery Using Eseutil
Repair Exchange Database Using EseutilRepair Exchange Database Using Eseutil
Running ISInteg /FixRunning ISInteg /Fix
Exchange DRA toolExchange DRA tool
Restore AD RequirementsRestore AD Requirements
Exchange 2003 disaster recovery assumes that the Exchange 2003 disaster recovery assumes that the Active Directory is available and if necessary fully Active Directory is available and if necessary fully recovered as wellrecovered as well
Running Setup /DisasterRecovery rebuilds the Running Setup /DisasterRecovery rebuilds the local box and does not re-write ANY data to the ADlocal box and does not re-write ANY data to the AD
Note: Exchange setup does not enforce or check Note: Exchange setup does not enforce or check that objects already exist in the AD that objects already exist in the AD
Full Exchange Server RestoreFull Exchange Server Restore
Reconfigure hardware drives similar to original serverReconfigure hardware drives similar to original server
Reinstall operating system using the old server nameReinstall operating system using the old server name
Install service packs and fixesInstall service packs and fixes
Install ExchangeInstall Exchange
Setup /DisasterRecoverySetup /DisasterRecovery
Install Exchange service packs and fixesInstall Exchange service packs and fixes
Restore Exchange databasesRestore Exchange databases
Any certificates must be restored separatelyAny certificates must be restored separately
Using the Setup /Disasterrecovery switchUsing the Setup /Disasterrecovery switch
Active Active DirectoryDirectory
ForestForest
Active Active DirectoryDirectory
ForestForest
ProductionServer
RestoreServer
.pst.pst.pst.pstCopyCopyCopyCopy
Restore
Alternate Server RestoresAlternate Server Restores
Alternate Server RestoresAlternate Server Restores
Configure new Windows 2003 Server with the latest Configure new Windows 2003 Server with the latest hotfixes/service packshotfixes/service packs
DCPROMO to create a new forestDCPROMO to create a new forest
Only one Exchange org per AD forestOnly one Exchange org per AD forest
alternatedomain.corp.mycompany.comalternatedomain.corp.mycompany.com
Install DNS as a standard primary, allowing dynamic Install DNS as a standard primary, allowing dynamic updates, and point to self as DNS serverupdates, and point to self as DNS server
Install Exchange using same org name and Install Exchange using same org name and Administrative Group nameAdministrative Group name
Configuring the restore server
LegacyExchangeDNs must matchLegacyExchangeDNs must match
/O=organization/OU=site/CN=/O=organization/OU=site/CN=container/CN=objectcontainer/CN=object
First Exchange 2003 admin group usually is “First First Exchange 2003 admin group usually is “First Administrative Group”Administrative Group”
AG display names can be changed, but this doesn’t AG display names can be changed, but this doesn’t change the legacyExchangeDNchange the legacyExchangeDN
To match DNsTo match DNs
Use Event 1088 on the Eventvwr or use ISInteg Use Event 1088 on the Eventvwr or use ISInteg /DUMP/DUMP
Use LegacyDN tool to modify legacyExchangeDNsUse LegacyDN tool to modify legacyExchangeDNs
Alternate Server RestoresAlternate Server Restores
Alternate Server RestoresAlternate Server Restores
Create a new storage group and database with Create a new storage group and database with matching display namesmatching display names
Or rename existing onesOr rename existing ones
Dismount database and mark “This database can be Dismount database and mark “This database can be overwritten by a restore”overwritten by a restore”
Restore backup sets and mount databaseRestore backup sets and mount database
Run Cleanup Agent and reconnectRun Cleanup Agent and reconnect
Log on and extract dataLog on and extract data
To .PST using OutlookTo .PST using Outlook
Or, use ExmergeOr, use Exmerge
Ronen Gabbay
Alternate Server RestoreAlternate Server Restore
Rename the Legacy DN using LegcyDN toolRename the Legacy DN using LegcyDN tool
Mounting the databaseMounting the database
Create all users using Mailbox Recovery CenterCreate all users using Mailbox Recovery Center
Bulk reconnect all users to their mailboxesBulk reconnect all users to their mailboxes
Disaster Recovery StrategiesDisaster Recovery Strategies
Service Level Agreements (SLA)Service Level Agreements (SLA)Monitoring and NotificationsMonitoring and NotificationsExchange Technical ExpertiseExchange Technical ExpertiseUse relative small database filesUse relative small database filesBuild Documentation / Change Control / Patch Build Documentation / Change Control / Patch ManagementManagement
Firmware UpdatesFirmware UpdatesSoftware UpdatesSoftware Updates
Documented Disaster Recovery PlanDocumented Disaster Recovery PlanRegular Recovery TestingRegular Recovery Testing
ReviewReview
Exchange is mission critical applicationExchange is mission critical application
Exchange Database ArchitectureExchange Database Architecture
Deleted items & Mailbox retention time.Deleted items & Mailbox retention time.
Backup and Restore ArchitectureBackup and Restore Architecture
Recovery Storage GroupRecovery Storage Group
Dial Tone RestoreDial Tone Restore
Exchange Snapshot ArchitectureExchange Snapshot Architecture
Eseutil & ISInteg and all in betweenEseutil & ISInteg and all in between
Full Server Restore procedureFull Server Restore procedure
Alternate Server Restore procedureAlternate Server Restore procedure
Thank You !Thank You !
Ronen GabbayRonen Gabbay
► ► Microsoft Exchange Regional Director ► Exchange MVPMicrosoft Exchange Regional Director ► Exchange MVP
► ► Microsoft Secure and Well Managed Infrastructure Specialist ► Microsoft IsraelMicrosoft Secure and Well Managed Infrastructure Specialist ► Microsoft Israel
► ► Microsoft Exchange Server User Group leader Microsoft Exchange Server User Group leader
► ► E-mail [email protected] [email protected]
This document is for informational purposes only.This document is for informational purposes only.
MICROSOFT MAKES NO WARRANTIES, EXPRESS OR MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENTIMPLIED, IN THIS DOCUMENT..
20022002 Microsoft Corporation. All rights reserved. Microsoft Corporation. All rights reserved. Microsoft, BackOffice, the BackOffice logo, Microsoft Microsoft, BackOffice, the BackOffice logo, Microsoft Internet Explorer logo, the Office logo, Where do you Internet Explorer logo, the Office logo, Where do you want to go today?, the Windows logo, and Windows NT want to go today?, the Windows logo, and Windows NT are either registered trademarks or trademarks of are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other Microsoft Corporation in the United States and/or other countries. Other product and company names mentioned countries. Other product and company names mentioned herein may be the trademarks of their respective owners.herein may be the trademarks of their respective owners.
© 2005 Microsoft Corporation. All rights reserved. This presentation is for informational © 2005 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only.purposes only.MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.
