Backup&Restore of SAP landscapes

8/7/2019 Backup&Restore of SAP landscapes

http://slidepdf.com/reader/full/backuprestore-of-sap-landscapes 1/64

© 2010 SAP AG

Best Practice

Backup and Restore for SAP SystemLandscapes

Dietmar-Hopp-Allee 16D-69190 Walldorf

CS STATUSCustomer Published

DATE VERSION

March 2010 2.0

SOLUTION MANAGEMENT PHASE SAP SOLUTION

Operations & Continuous Improvement Generic Best Practices

TOPIC AREA SOLUTION MANAGER AREA

Technical Operations Optimization



Best PracticeBackup and Restore for SAP System Landscapes

© 2010 SAP AG page 2/64

Table of Contents1 Management Summary 4

1.1 Goal of this Best Practice Document 41.2 Introduction 41.3 Key Messages 51.4 How to Work with this Best Practice 71.5 Staff and Skills Requirements 71.6 Duration and Timing 7

2 Basic Considerations for the Backup and Restore Concept 82.1 Backup Requirements in a System Landscape 8

2.1.1 Data Consistency in a System Landscape 82.1.2 SAP Communication Technology 8

2.1.3 Data Consistency after a Complete Recovery 102.1.4 Implications of an Incomplete Recovery 112.1.5 Conclusion 14

2.2 Designing a Backup and Restore Concept 152.2.1 Building Blocks of a Good B&R Concept 152.2.2 Required Information 16

2.3 Service Levels and B&R 172.3.1 Recovery Time 172.3.2 Recovery Point - Complete Recovery 192.3.3 Backup Time 20

3 B&R for Specific Components of an SAP System Landscape 213.1 Identify Components to be Backed Up 223.2 B&R for SAP NetWeaver Application Server-based Components 23

3.2.1 Backup of the Database (Application Data) 233.2.1.1 Database Log Backup 243.2.1.2 Database Checks 263.2.1.3 Full or Incremental Backup 27

3.2.2 Backup of the SAP and Database Filesystems (Software) 283.2.3 Server Backup (Bare Metal Information) 293.2.4 Restoring SAP NW AS 30

3.2.4.1 Restore Basic Server Operability 313.2.4.2 Restore Software Components 313.2.4.3 Repeat Any Lost Changes Done in the Filesystem 313.2.4.4 Restore Application Data 313.2.4.5 Recover Database 313.2.4.6 Resolve possible application-related inconsistencies 31

3.3 B&R for Standalone Engines 323.3.1 Standalone Engines with Application Data 32

3.3.1.1 Backing up the Application Data 323.3.1.2 Backing up the Standalone Engine Itself 33

3.3.2 Standalone Engines without Application Data 333.4 Clients 333.5 Further B&R Recommendations 34

3.5.1 Contents of a B&R Concept 34




© 2010 SAP AG page 3/64

3.5.2 Backup Schedule 353.5.3 Retention 36

3.5.4 Backup and System Maintenance 363.5.5 Backup Verification 363.5.6 Testing and Training 373.5.7 Continuous Updating and Change Control 37

4 Disaster Recovery - Basic Approaches using B&R 384.1 Disaster Recovery Tiers 384.2 Phases of a DR using the B&R Approach 38

4.2.1 Providing the DR Hardware 394.2.2 Providing the System Software 394.2.3 Providing the Application Software 39

4.2.4 Restoring the Application Data 394.3 Data Consistency with DR using the B&R Approach 40

5 Managing Incomplete Recovery 425.1 Avoiding Incomplete Recovery 42

5.1.1 Incomplete Recovery Due to Technical Problems 425.1.2 Approaches for Handling Logical Errors 435.1.3 Restore of a Filesystem Backup 45

5.2 Performing an Incomplete Recovery in a System Landscape 455.3 Handling Data Inconsistencies Between Systems 46

5.3.1 Performing Business Recovery 465.3.2 Preparing for Business Recovery 47

6 Consistent Landscape Backups 496.1 Possible Use Cases 496.2 Creating Consistent Landscape Backups 50

6.2.1 Application Level 516.2.1.1 Stop of All Applications During Backup 526.2.1.2 Stop of All-But-One Application During Backup 526.2.1.3 Stop of All-But-One Application During Log Backup 53

6.2.2 Database Level - Coordinated Suspend 556.2.3 Storage Level - Consistency Technology 57

6.3 Further Information on Consistency Technology 597 Tasks for Creating a B&R Concept 608 Appendix 62

8.1 Glossary 628.2 Further Information and References 638.3 Feedback 63




© 2010 SAP AG page 4/64

1 Management Summary

1.1 Goal of this Best Practice Document

Use this Best Practice to guide you in creating a backup and restore (B&R) concept for your SAP system

landscape. This document provides the necessary background information and directives needed to set up a

comprehensive B&R concept for a distributed system landscape.

This Best Practice covers, for example:

Questions regarding data consistency during B&R in a system landscape

General requirements and recommendations for B&R of individual components

Handling of incomplete recoveries

Options for creating consistent landscape backups

This Best Practice can be used to create a customer-specific B&R concept for any kind of SAP system

landscape.

1.2 Introduction

Today, our customer’s mission critical business processes do no longer run on a single system but usually

rely on the availability of multiple systems and components. To achieve the required availability level of all

involved systems and services is the task of Availability and Continuity Management, which becomes

increasingly important for our customers. But before optimizing system availability, the first and basic

responsibility of system operations is to protect the systems from data loss or even complete loss of the

systems . This is the task of backup and restore.

A reliable backup and restore concept must fulfill this basic requirement by regularly taking backups of all

relevant objects and by making sure that they can be restored in case of a loss of the primary data source or

hardware. Having achieved this basic prerequisite of system operations, customers can start to manage their

availability requirements by making sure that the implemented procedures are able to satisfy the demanded

service levels in case of a restore or in case of other failures. This will usually call for additional technologies

to be implemented for high availability and disaster recovery.

The goal of this document is to enable a customer to fulfill the backup and restore demand by providing

information on special questions arising in the context of today’s typical federated system landscapes

(landscapes of multiple connected systems exchanging data).

Note: When speaking of a “ System Landscape ”, this document always refers to the production

system landscape consisting of different systems serving productive purposes, for example a

production landscape consisting of a productive ERP system, a productive CRM system and other

production systems. This term is not used in the meaning of a ‘maintenance’ or ‘transport landscape’supporting the change management process.




© 2010 SAP AG page 5/64

1.3 Key Messages

This section briefly summarizes whether new backup requirements are needed by these federated

landscapes - like the need for a synchronized, consistent backup across all systems (called a consistent

landscape backup ). More background information explaining the details will be given in the following

chapters.

SAP generally stores all mission-critical application data in database systems. When restoring a database,

two scenarios have to be distinguished: Complete recovery without data loss and incomplete recovery

causing some amount of data loss. To determine the need for a consistent landscape backup in addition to

traditional backups of each individual system, both cases must be considered.

Complete Recovery

If a single system in the landscape needs to be restored for some technical reason like disk failures, only this

individual system would be restored from its backup, not the complete landscape. Data loss can usually be

avoided by rolling forward the database logs of the restored database exactly to the point of the error. Due to

the transactional and error-tolerant messaging technology used for data exchange with standard SAP

business scenarios, data consistency between the systems will be maintained after such a complete recovery

without data loss.

Incomplete Recovery

On the other hand, if any of the systems of a federated system landscape faces data loss, for example due to

an incomplete recovery, data consistency between the systems can no longer be maintained because

committed data and messages are lost. But even in this case, a consistent backup across all systems usually

does not solve the problem. Restoring a consistent landscape backup would result in data loss on all included

systems and would still leave inconsistencies in relation to further systems and the real world. Thus it is

generally preferable to restrict the data loss to a single system, keep it as small as possible and subsequently

work on resolving the resulting data inconsistencies. That will require close involvement of the application

experts because resolution of inconsistencies requires in-depth application knowledge and is no longer a

purely technical challenge.

Consistent Landscape Backups are Not Mandatory

So, for the overall backup concept for a system landscape, a consistent landscape backup is generally not

required to handle complete or incomplete recovery of a system. It is still sufficient to define the backup

procedure for each system independently of all others.

Avoiding Incomplete Recovery

To avoid data inconsistencies between systems, it is important to avoid data loss. For the backup and restore

concept of each individual system this means that special attention must be paid to safeguarding the

database logfiles, which are most critical for performing a complete recovery. Incomplete recovery, which

always causes data loss, must be avoided.




© 2010 SAP AG page 6/64

Alternate Ways of Resolving Logical Errors

Incomplete recovery or point-in-time recovery must also be avoided in case of logical errors . Incompleterecovery to a previous point in time is commonly considered as a possible approach to remove logical errors

from a system. In a system landscape, this approach is usually no longer preferable due to the impact on

overall data consistency between systems. It becomes important to define procedures that allow resolving

logical errors without restoring a productive system to a former point in time.

Preparing for Handling Data Inconsistencies

But there may also be other reasons for possible data loss or data inconsistencies between systems. Data

stored outside of databases could be a possible source of data inconsistency due to possible data loss after a

restore (filesystems do not offer the concept of forward recovery like databases do). Customer-specific

interfaces could cause inconsistencies if there is a lack in fault-tolerance and transactional consistency in

case of failures. Activating a disaster recovery plan can result in data inconsistencies between systems if no

special precautions were taken to ensure data consistency.

Since data inconsistencies can be critical for business operations, avoiding and handling inconsistencies is

an important subject of Business Continuity Management (BCM). At this point, a B&R concept correlates with

an overall BCM strategy. One of the main goals of BCM is to enable the recovery of critical business

processes within a predefined time. This is not restricted to the recovery of technical systems only but also

has to include application-related aspects like data consistency. As such, preparations and approaches for

handling possible inconsistencies as introduced later in this document should be part of an overall BCM

strategy. More information on BCM can be found in the Best Practice “ Business Continuity Management for

SAP System Landscapes ”. BCM is part of the Run SAP approach for SAP systems operations and includes

the topics Availability and Continuity Management.

Consistent Landscape Backups for Special Purposes

As a possible solution to address data consistency during disaster recovery, consistent landscape backups

can be considered. If, for example, the disaster recovery plan is based on shipping backup tapes to an

alternate site, shipping a consistent backup of the complete landscape will enable recovery of a consistent

state without the need to worry about possible inconsistencies between the systems. Thus, consistent

landscape backups can complement a continuity management concept.

https://service.sap.com/~sapidb/011000358700000575802008E








© 2010 SAP AG page 7/64

1.4 How to Work with this Best Practice

The following main topics are covered in the different chapters of this document:

Basic Considerations for the Backup and Restore Concept provides some background information that is

needed to understand the requirements of a B&R concept for a system landscape. The remaining chapters

can then be used selectively to provide more details on different aspects of the B&R concept.

A basic B&R concept for the systems of your system landscape can be defined with the help of B&R for

Specific Components of an SAP System Landscape and specific information on B&R procedures in further

SAP documentation.

Using the B&R concept also as a basic approach for disaster recovery when the complete environment

needs to be rebuilt is covered in Disaster Recovery - Basic Approaches using B&R .

Managing Incomplete Recovery provides further information extending the purely system-based technical

B&R concept into a more comprehensive BCM concept. That chapter addresses approaches to handle

logical errors and to plan for the recovery of business operations in case of data inconsistencies

caused by an incomplete recovery.

Consistent Landscape Backups gives information for customers who are interested in the implementation of

consistent landscape backups for serving special demands .

Tasks for Creating a B&R Concept provides a checklist for the different tasks to be performed when creating

a B&R concept for a system landscape.

In the Appendix you will find a short glossary of important terms in the context of this document and links to

further sources of information.

1.5 Staff and Skills Requirements

To apply this Best Practice you need a team consisting of:

System administrators, consultants or technical persons charged with planning, setting up, or

administrating an SAP system landscape

Application experts with knowledge of your business processes and interfaces

1.6 Duration and Timing

The initial B&R concept should be created during the implementation of an SAP solution and must be

finalized before the start of production. The time needed to create the B&R concept depends on the

complexity of the system landscape and business scenarios. Verifying, adapting, and testing the concept

should be an ongoing process during the complete lifecycle of the SAP solutions.




© 2010 SAP AG page 8/64

2 Basic Considerations for the Backup and Restore Concept

This Best Practice gives you a generic overview and advice for creating a backup and restore (B&R) conceptfor your system landscape. The following sections guide you through milestones and provide important

information you should keep in mind during this process.

2.1 Backup Requirements in a System Landscape

2.1.1 Data Consistency in a System Landscape

Business processes today are often built on services provided by different components or systems of a

system landscape. Business data created and used by these processes is stored in different locations of the

so-called ‘federated landscape’. For error-free business operations and correct business decisions, it is vital

that the data that is stored across these different locations is always consistent, thus providing a consistent

business view.

For more information on SAP standards for data consistency and data integrity see the whitepaper

“SAP Standards for Data Integrity and Transactional Consistency ”.

Data consistency between systems must also be maintained in case of system failures or system restores.

SAP business processes use a data exchange method (communication technology) that is able to guarantee

data consistency of all data that is created or modified by a business process in the different systems of the

landscape (see SAP Communication Technology ). But data consistency can only be ensured as long as no

committed data is lost.

For this reason, SAP generally stores all mission-critical application data in database systems. In contrast to

data stored in flat files in a filesystem, databases offer the advantage that the data of all committed

transactions can always be rebuilt based on a database backup in combination with the constantly generated

database logfiles (whereas filesystems can usually only be rebuild to the state of the last backup, meaning

that all data that was created or changed after the last backup will be lost when restoring from a backup).

To find out what requirements a B&R concept must fulfill, we need to look at the implications a databaserestore will have on data consistency between the systems. But firstly, we need to understand how SAP

systems communicate and exchange data.

2.1.2 SAP Communication Technology

When exchanging data between systems, the data exchange method must ensure that a consistent state of

the data can be maintained at all times. This includes that the data exchange must be tolerant against system

failures (reliable messaging).

In an SAP NetWeaver and SAP Business Suite system landscape, data is exchanged using the Remote

Function Call (RFC) or the Web services communication. Both techniques use a similar concept for messageexchange and can be distinguished into two major types: Synchronous and Asynchronous communication.

https://service.sap.com/~form/sapnet?_SHORTKEY=01100035870000711194&_SCENARIO=01100035870000000202&






© 2010 SAP AG page 9/64

With synchronous communication - synchronous RFC (sRFC) or Best Effort (BE) messaging, the caller of

a remote function (respectively the sender of a message) always waits for the answer or result from the target

system. If the remote operation cannot be executed completely (for example because the connection cannot

be established or the target system breaks during execution), the calling process receives an error code and

must take appropriate actions. Because of the synchronous nature, this communication protocol must not be

used for propagating changes to another system. When trying to modify data in two systems, the

synchronous protocol cannot ensure that the changes will really be done in both systems if the connection or

one of the systems breaks in the middle of communication. Synchronous remote communication may only be

used for ‘reading’-type functions.

With asynchronous communication - transactional RFC (tRFC) or Exactly Once (EO) messaging, the

remote function (respectively message) is always executed asynchronously while the calling process

continues without waiting. This same property applies to queued asynchronous communication – queued

RFC (qRFC) or Exactly Once in Order (EOIO) messaging, which are an extension to the asynchronous

protocol and allow a serialization of messages from different senders. Background RFC (bgRFC) provides

similar features for asynchronous processing with bgRFC type t (transactional) and bgRFC type q (queued).

The asynchronous communication protocols provide reliable messaging and ensure that the function (or

message) is executed exactly once in the target system and can be restarted at any stage of the

communication protocol. This is especially guaranteed in case of communication failures, for example if:

- the connection cannot be established

- the connection is terminated during the execution or during data transfer

- the target system crashes during the execution of the function or message

- the sending system crashes somewhere in the middle of processing the communication protocol

The asynchronous communication protocol exploits the reliable storage of the message states in the

databases of the sending and receiving SAP systems.

Note: All standard SAP communication channels and interfaces like CIF, ALE, IDOC, BDOC use the

underlying asynchronous communication protocols introduced above, thus inheriting reliability and

error-tolerance.

Because only asynchronous communication can provide this error-tolerance and restart-ability,

synchronous communication must not be used for creating or modifying data in multiple systems.

Customer-specific developments and third-party interfaces should be checked if they also

provide reliable messaging and error-tolerance to ensure transactional correctness and data

consistency between different data sources.

So during ‘normal’ operation (including shutdown or even crash of a single system), data consistency is notendangered because the asynchronous protocols are fault-tolerant. A vital prerequisite for this is the correct

handling of the queues that store information on ongoing communications. Queue entries should never be




© 2010 SAP AG page 10/64

deleted without being absolutely aware of the consequences (not even if the queues are hanging). If queue

entries are deleted, the corresponding function cannot be executed completely, which will most likely result in

inconsistencies between the sending and receiving systems!

While messages are still in transfer, application data may look inconsistent when being analyzed at that point

in time. For this reason, consistency checks have to distinguish between temporary differences (that will go

away once the message was processed completely) and real inconsistencies (that occurred for example due

to a loss or deletion of messages).

In SAP CRM Mobile Sales / Mobile Service, data exchange between the client computers and the

SAP CRM server is also tolerant against failures during communication. Data exchange to and from

the mobile clients is done via the SAP Communication Station. The communication station simplyconverts data packets between different formats and does not store any data. Sending a data packet

is implemented as an all-or-nothing operation: if the packet has not been sent completely when the

communication link is aborted, the packet will be resent again. All committed packets that reached

their destination will not be resent.

With SAP Mobile Infrastructure, mobile clients use a different data exchange mechanism which is

also error-tolerant and can be restarted at any point if there should be a failure during communication.

2.1.3 Data Consistency after a Complete Recovery

Normally, a database restore consists of two steps: Restore of a database backup followed by the recovery of

database logfiles to roll forward changes that have been done since the backup was taken. By applying all

generated logs to a restored database, the database can be recovered to the exact state at the time of a

crash. This is called a complete recovery .

A complete recovery will recover all committed transactions and all committed data while all open

transactions will be rolled back. All messages will be restored to exactly the state that was committed at the

time of the crash. This means that due to the error-tolerant asynchronous communication protocol used for

data exchange, all messages can be restarted in their previous (committed) state and data consistency

between the systems will be maintained.

If, in a system landscape, one system faces a problem and needs to be restored (for example after a disk

failure), only this single system would be restored from its backup and the database would be rolled forward

doing a complete recovery as shown in figure 1. After the complete recovery, all messages in the affected

system will be restarted from their previously committed states.

During the restore, the affected system will not be available but all other systems of the landscape

can continue operating and even create new messages that will be sent to the affected system once

it is back in operation.




© 2010 SAP AG page 11/64

Figure 1: Timeline of a Complete Recovery in a System Landscape

2.1.4 Implications of an Incomplete Recovery

Incomplete recovery occurs if a database or a system cannot be recovered to the state it had been at the

point of the crash. An incomplete recovery may be caused by:

A restore of a database (offline) backup without applying logs

A database point-in-time recovery recovering the database up to an earlier point in time

A database point-in-log recovery recovering the database to an earlier point in one of the logfiles

The restore of a filesystem to a previous backup

Reasons for an incomplete recovery of a database system may be corrupt logfiles, destroyed tapes or a

severe logical corruption of the database. Restore of a filesystem backup can also be considered as an

‘incomplete recovery’ because the filesystem will be restored to the time when the backup was taken. Any

data that was created after the backup will be lost.

An incomplete recovery of one component of a system landscape always causes data loss, which will

in most cases result in data inconsistencies between the systems. Data loss and data inconsistencies

will both have a serious impact on business operation. Therefore, an incomplete recovery in a

system landscape must be avoided as long as any other solution seems possible (see

Managing Incomplete Recovery ).

Logical errors in a system should be repaired without performing an incomplete recovery on a

productive system (see Approaches for Handling Logical Errors ).




© 2010 SAP AG page 12/64

The term ‘point-in-time recovery’ is usually used in the meaning of ‘incomplete recovery’ (although a

point-in-time recovery can technically be a complete recovery if all logs are applied).

A customer’s B&R concept should include precautions to avoid incomplete recovery and data loss. Here, the

B&R concept correlates with an overall Business Continuity Management (BCM) concept. The BCM concept

on the one hand needs to provide technical measures to protect from data loss and on the other hand has to

include considerations how to deal with possible data loss and data inconsistencies in case of an incomplete

recovery. Incomplete recovery needs to remain a very rare situation. The most common reason given for

incomplete recoveries, the resolution of logical errors, needs to be addressed by defining alternate

strategies for logical error handling. More information on these aspects will be found in Managing

Incomplete Recovery and in the Best Practice “ Business Continuity Management for SAP System

Landscapes ”.

The remaining part of this section will have a closer look at possible approaches in the rare case that an

incomplete recovery is actually unavoidable. The main consideration here is whether the B&R concept

requires new approaches for backing up the systems in a federated system landscape.

Figure 2 shows the system state after a database recovery using the example of two systems. It shows three

different alternatives that might be applicable if an incomplete recovery of one system of a federated system

landscape was needed. These alternatives ‘A’ to ‘C’ are:

(A) Only the affected system is restored doing an incomplete recovery.

In this case, inconsistencies in relation to other systems of the landscape will result for the period of

data loss in the affected system. These inconsistencies will have to be analyzed and resolved at

application level by comparing possibly affected data objects between all connected systems. This

subsequent recovery process at application level will be referred to as ‘ business recovery ’. For

more information on performing a business recovery and on available tools see Handling Data

Inconsistencies between Systems .

(B) A consistent backup of the complete system landscape is restored.

This option assumes that the customer regularly creates a consistent backup including all systems of

the landscape. Consistent Landscape Backups shows different options how such consistent landscape backups can be created. Using this alternative, all systems of the landscape could be

restored back to the last consistent landscape backup , which would usually be up to one day old.

There is data loss in all systems but there are no inconsistencies between the systems that were

included in this special kind of backup.

(C) All systems are restored to the point in time that the affected system had to be restored to.

This causes data loss in all systems and possibly some data inconsistencies due to the nature of the

point-in-time recoveries. Even when using a time synchronization server, there is no final guarantee

that the recoveries in the different systems will be synchronized up to the latest transactions; so therecould still be some inconsistencies (also see Creating Consistent Landscape Backups ).









© 2010 SAP AG page 13/64

Inc omp le t e Rec overy of OLTP - Al t e r na t i ves

Crash orProblem Detected

in System 1

Recover Pointfor System 1

CRMSystem 1

System 2

Data lossPoint-in-timerecoveryfor one system

Inconsistencies

A

Last backup (System 1)

/ Last Consistent

Landscape Backup

System 1

System 2

Data loss

Data loss

Restore of aconsistentlandscapebackup

B

System 1

System 2 Data loss

Point-in-timerecoveryfor all systems

Inconsistencies

C

Restore Recovery to t P1

t B t P1 t C

Restore

Restore Recovery to t P1

Data loss

Restore Recovery to t P2 tP1

Figure 2: System State with Different Alternatives in Case of an Incomplete Recovery

The three alternatives mainly differ with respect to the factors ‘data loss’, ‘inconsistencies’ and ‘downtime’, all

of which should be minimized. These factors must be carefully weighted when deciding on one of these

alternatives for a B&R concept. The following table compares the three alternatives:

Alternative:

Factor:

(A)Point-in-time Recoveryfor One System

(B)Restore a ConsistentLandscape Backup

(C)Point-in-time Recoveryfor All Systems

Data loss Minimum, only one systemaffected

Maximum, all systemsaffected, up to point of lastconsistent landscape backup

Medium, all systemsaffected

InconsistenciesMost inconsistencies,for all systems connectedto the affected system

No inconsistencies Little inconsistencies

Downtime

Only one system down,downtime depends onrestore time and the

amount of logs to recover.Business downtime maybe longer for resolvinginconsistencies.

All systems down,downtime possibly quiteshort when using storagetechnology for theconsistent landscape backup s

All systems down,downtime depends onrestore time and the

amount of logs to recover.Business downtime maybe longer for resolvinginconsistencies.




© 2010 SAP AG page 14/64

Although the decision on which alternative to use depends on the specific system landscape, the customer’s

business and the detailed error situation, in general, alternative A would be the preferred option when

facing an incomplete recovery because of the following reasons:

Data loss and downtime are kept to a minimum with alternative A, downtime on other systems can be

avoided.

Data loss in all other systems is not acceptable just because one system had a problem.

Restoring a consistent landscape backup causes too much data loss because everything is restored

back to the last such backup. Consistent landscape backups cannot be created in very short

intervals.

Even though business recovery requires high effort and time, there is a chance to get back data that

was lost in the affected system

Even when restoring a consistent landscape backup (or using alternative C), inconsistencies will

result for any other systems that are not included in this approach (legacy systems, partner systems,

systems of other business units or regions, mobile clients, etc.) and to the real world (deliveries that

left the warehouse, invoices that were printed and mailed, etc.). So there will always remain the needto perform business recovery for some systems and interfaces.

Managing Incomplete Recovery gives more details how to manage an incomplete recovery and the upcoming

business recovery .

2.1.5 Conclusion

Complete recovery maintains data consistency in the SAP system landscape due to the fault-tolerant

communication technology with tRFC, qRFC, EO or EOIO messaging. Any non-standard interfaces should

be checked whether they can provide the same fault-tolerance and restart-ability. To maintain data

consistency, it must be the most important goal of a B&R concept and the B&R operations to avoid data loss

and achieve a Recovery Point Objective (RPO) of 0 by enabling complete database recovery.

Incomplete recovery of a system in a federated landscape causes data loss and data inconsistencies

between systems and has to be avoided. The most appropriate approach after an incomplete recovery of a

single system will be to restrict the incomplete recovery to the affected system and leave all other systems

untouched. This will raise the need for business recovery - that is the process of identifying and resolving all

resulting data inconsistencies between the systems.

So, for complete and incomplete recovery, there is generally no need for a consistent landscape backup .

Thus a consistent landscape backup does not need to be part of the regular backup strategy. It is sufficient to

backup each system individually and to especially safeguard the database logfiles in order to avoid

incomplete recovery.

But apart from this, there are other use cases for consistent landscape backups which may justify including

consistent landscape backups into the backup strategy. Especially when using storage-based backuptechnologies, there are possibilities to combine the regular system backups with the creation of a consistent

landscape backup . For more information, see Consistent Landscape Backups .




© 2010 SAP AG page 15/64

2.2 Designing a Backup and Restore Concept

A detailed B&R concept is not only influenced by the system components actually in use in a system

landscape but also by many other customer-specific factors like service level requirements, available backup

technologies and backup tools, amount of data, implemented business processes and dependencies

between systems. This is why a B&R concept must always be tailored individually to suit the demands of

each specific implementation.

To design a backup and restore concept, the following aspects need to be considered:

Which system components and what data need to be backed up?

What are the demanded service levels for a restore and what is required to achieve them?

What are the demands on the backup process?Which backup and restore technologies are available, which of them shall be used?

What is required to protect the system and to safeguard restore and recovery?

Will the backups also be used for disaster recovery?

Is data consistency between the system components maintained for customer-specific developments

and non-standard interfaces?

What are the impacts of an incomplete recovery and which steps need to be taken?

What can be done to avoid data loss and incomplete recovery?Is a Consistent Landscape Backup desired and what will it be used for? How is it created?

2.2.1 Building Blocks of a Good B&R Concept

A comprehensive B&R concept for a system landscape should consider the following four building blocks:

1. Establish a B&R Concept for Each Individual System Component

This step will define the backup and restore process for each component of the system landscape. To

identify all components of the landscape, the system overview in SAP Solution Manager can assist.

Knowing all components that must be included in the B&R concept, the necessary backup objects can bedetermined based on the type of component and its type of data.

For more information on this basic part of the B&R concept, see B&R for specific components of an SAP

System Landscape .

2. Using the Backup for Disaster Recovery

Although there are more sophisticated Disaster Recovery (DR) solutions available, a basic approach for

DR relies on restoring a complete system on some completely new hardware. This requires a full backup

of the system and its application data. In addition to an ordinary restore of a backup onto the same

hardware where the backup was taken from, a restore on a different hardware may impose furtherrequirements to get the systems up and running. For a system landscape, data consistency within in the

target landscape also needs to be investigated and, if required, appropriate activities to reestablish data




© 2010 SAP AG page 16/64

consistency on application level need to be included in the DR procedure. Consistent landscape backups

can support a backup-based DR approach by avoiding possible inconsistencies.

For more information, see Disaster Recovery - Basic Approaches using B&R .

3. Provide Procedures and Tools for Avoiding and Handling Data Inconsistencies

Data inconsistencies between systems could result from an incomplete recovery in one of the systems.

Because of this, a B&R concept should, in addition to merely describing backup and restore procedures,

also introduce strategies on how to avoid and on how to deal with data inconsistencies. Avoiding

inconsistencies means establishing concepts to avoid incomplete recoveries, mainly by providing

alternate approaches for handling logical errors. Dealing with inconsistencies means providingapproaches for business recovery . Both will require a lot of application-specific know-how. This part of the

B&R considerations extends into the BCM area and requires people from the business departments and

system administrators working closely together on managing incomplete recoveries and data

inconsistencies.

For more information, see Managing Incomplete Recovery .

4. Implement Consistent Landscape Backups for Serving Special Purposes

If desired for special purposes like system copy or consistent disaster recovery across the landscape, you

could consider implementing consistent landscape backups . As discussed in Backup Requirements in a

System Landscape , consistent landscape backups are not a mandatory part of a regular backup strategy.

For more information on creating consistent landscape backups and possible benefits, see Consistent

Landscape Backups .

2.2.2 Required Information

The following information will be needed for designing a comprehensive B&R concept considering each of the

above building blocks:

The systems and components used by the business processes in your specific environment

The type of data that needs to be backed up for each of these systems and components

The applicable service levels (availability requirements) for the system landscape and each of its

components, translating into the demands on the B&R processes

Available technologies and tools for B&R

The overall DR approach

The service levels demanded for DR

For handling data inconsistencies (for example after an incomplete recovery or in case of DR), additional

application-specific information on the interfaces and data exchange between the systems will be required:

The core business processes exchanging data between the different system components

The data flow and the business objects being exchanged between these components




© 2010 SAP AG page 17/64

The leading system (original system) for each business object

For defining the scope of consistent landscape backups , this application-specific information will also beneeded to describe the dependencies between the systems of the landscape.

2.3 Service Levels and B&R

To ensure the demanded system availability, each system is required to meet its predefined service levels.

These service levels must be defined for each system or component based on the business requirements of

the business processes relying on that system or component (respectively its provided services).

The business requirements are highly customer-specific and the definition of the service levels should be

done based on a sound analysis of the business requirements involving the business process owners. SAP’s

proposed approach for defining service levels for system availability is based on ITIL (the IT Infrastructure

Library, see http://www.itil.co.uk ) and is described in the Best Practice “ Business Continuity Management for

SAP System Landscapes ”.

The B&R concept must adhere to the service levels with respect to the following factors:

Recovery Time Objective (RTO)

The RTO defines how much time is allowed for the recovery of a system. It should always be

possible to restore a system within the defined recovery time for that system.

Note that the service levels may distinguish different RTOs for different error types and that anerror scenario requiring a restore may allow for a special (usually longer) recovery time.

Recovery Point Objective (RPO)

The RPO defines how much data loss is acceptable in a specific error scenario. A complete

recovery (RPO = 0) should always be possible for databases (unless the customer’s service

levels give a different specification). We generally advise aiming for a complete recovery asdescribed in Backup Requirements in a System Landscape to avoid problems with data

consistency between systems.

Backup Window

The backup window defines the period which is available for backing up a system (or a number

of systems). It must be possible to complete the backups within the allowed backup window to

avoid conflicts with other activities in the system.

2.3.1 Recovery Time

To answer the question whether the backup infrastructure is able to meet the required recovery time, you

should calculate how long a restore of the system would take. For an existing system, this can be done most

reliably by performing a restore test.

http://www.itil.co.uk/











© 2010 SAP AG page 18/64

But for reliability of planning, you should also take the database growth and future database size into account

when calculating expected restore and recovery times. As a general guideline you should calculate with the

highest predicted volume and give a rolling forecast for the B&R concept.

It is also important to not just include the time for a database restore into this calculation but also consider the

time required to restore and apply all database logfiles performing the forward recovery from the backup to

the current point in time. Here, it is important to note that the time needed for log recovery might even exceed

the restore time itself because log recovery has to re-apply all changes to the restored image of the database.

The recovery time for restoring a system from its backup can be calculated from the following terms

(assuming that only the data and not the system itself need to be restored):

1. Restoring the Data from the Last Backup

Knowing the realistic restore throughput of your backup infrastructure (for example from a restore

test), the restore duration can be calculated based on the expected database size. In some cases,

the restore throughput may be smaller than the backup throughput; whereas in other cases, the

restore throughput may be higher, for example if backups are highly parallelized for a multitude of

systems competing for the resources while the whole capacity of the infrastructure can be exploited

when restoring a single system.

If incremental or differential backups are used, a longer restore time needs to be considered for

applying the incremental backup sets to the last full backup.

2. Restoring Logfiles from the Backup InfrastructureKnowing the overall size of archive logs being generated per day and the restore throughput from the

backup medium, you can estimate the time required to restore the archive logs from the backup

medium to primary storage. If it is possible to perform this restore in parallel to the database restore,

this time may not need to be considered. If archive logs are kept on primary storage for a sufficient

time, this time may also be omitted.

3. Recovering the Database by Applying Logfiles

A practical test should deliver information on the recovery throughput in your environment (e.g.

recovery time for one logfile of size x ). Knowing the maximum log volume generated by the database

per day you can estimate the maximum required recovery time needed once the last backup wasrestored (assuming that backups are created on a daily basis). Should the requirement arise to

restore from an older backup, the log recovery time will elongate accordingly.

If partial backups are used, backing up only different parts of a database with each backup, a longer

recovery time needs to be considered because of the different amount of recovery required on the

sets of a partial backup.

If this estimation of the recovery time in the case of a restore exceeds the permitted recovery time, you

should consider whether the restore and recovery can be accelerated to meet the RTO. There are many

factors that influence the achievable restore and recovery throughput like parallelism, number and speed of

available tape drives, bandwidth of backup network, and so on. SAP note 842240 (partly Oracle-specific) lists

several options how to possibly speed up restore and recovery.

https://service.sap.com/~iron/fm/011000358700000431401997/0842240E





© 2010 SAP AG page 19/64

Due to the increasing availability demands of today’s businesses and on the other hand due toincreasing database sizes, the backup infrastructure and the restore and recovery process is often no

longer able to fulfil the demanded RTO, even if special restore technologies and optimizations were

implemented. This raises the need to investigate and evaluate alternative recovery solutions that can

avoid restores for all critical error scenarios. This is the task of availability and continuity

management and is not further covered in this document.

Example:

The predicted database size of an SAP system will be about 1 TB after one year. A restore test

showed that the backup infrastructure can restore the data with an average throughput of 250 GB per

hour. On busy days, the system generates 50 GB of archive logs per day (200 files of 250 MB).

Recovery of one archive log takes about 1 minute, resulting in a recovery throughput of approx. 15 GB

per hour. With this information, the estimated recovery time for restoring the latest backup will be up to

7.5 hours (Restore of datafiles: 4 hours. Restore of logfiles: about 10 minutes. Forward recovery of the

database: up to 3 hours 20). If the RTO was 4 hours, it could not be met with the example

infrastructure.

Note that when talking about recovery time and RTO in this document, we are only referring to the

technical recovery time , which is the time required to perform the acutal recovery operations usingthe appropriate recovery solutions. This does neither include time required for error analysis or

decision making nor reaction time of administrators and technicians responsible for the recovery.

With respect to the actual business downtime possibly caused by a disruption, such times should

be considered as well in the overall RTO.

2.3.2 Recovery Point - Complete Recovery

The B&R concept should always enable a customer to perform a complete recovery of the database. This

involves that:

All archive logs are available and error-free

The most current online logs are available for the recovery.

For more information on safeguarding database logs, see Database Log Backup .

For a scenario when complete recovery is not possible for some component, the B&R concept needs to deal

with the possible consequences and describe what needs to be done in this situation.

Complete recovery may for example not be possible if data is stored in a filesystem and not in a database.

Restoring a filesystem will usually end up with an older state and possible inconsistencies in relation to other

systems that are interfaced by that component. For any such components, you should analyze and describewhat needs to be done after a restore.






© 2010 SAP AG page 21/64

3 B&R for Specific Components of an SAP System Landscape

A customer’s system landscape can contain systems or components of different types which may havedifferent B&R requirements. All main SAP systems with business data are based on an SAP NetWeaver

Application Server which stores this data in a database. In addition, there may be Standalone Engines and

Clients which provide specific functions in combination with one or more SAP NetWeaver systems. While

standalone engines generally reside on backend systems, clients usually reside on local front-end PCs.

Standalone engines and clients may store some application data of their own or may not have application

data at all. In addition to application data, the systems themselves with their software, configuration and

operating system may need to be backed up as well.

Altogether, the following types of objects may be relevant for backups on the different systems or

components:

System software (operating system, basic system software)

Application software (database, SAP)

Configuration files

Business data (application data)

o Data managed by databases

o Data in files which are managed directly by the application

Log files (‘error logs’)

To define a B&R concept, the first step is to identify the different systems and components in the landscape,

see Identify Components to be Backed Up . Then, the B&R concept needs to be defined for each of the

components. Sections, B&R for SAP NetWeaver Application Server-based Components through to Clients

provide information on B&R for the different types of components and their relevant backup objects. Further

basic recommendations regarding the B&R concept are given in Further B&R Recommendations .

Note: This chapter does not want to replace any database- or solution-specific operating handbooks

or documentation! It wants to provide the required background information to help you define anappropriate B&R concept for the components of your SAP system landscape and to summarize the

most important and critical recommendations regarding a B&R concept.

For more details on backing up an SAP NetWeaver-based system, see the system administration

topics in the SAP online help and SAP’s database-specific backup recommendations.

For more information on backing up specific systems or components also see the documentation or

operating handbooks of the respective systems or components.




© 2010 SAP AG page 22/64

3.1 Identify Components to be Backed Up

As a first step to define a B&R concept for a system landscape, it is necessary to create a list of all systems

or components used in the landscape.

You can use your Solution Manager and System Landscape Directory to find out which components

are part of your system landscape. You should always cross-check your backup concept to verify

whether it includes all systems mentioned in the Solution Manager.

The backup requirement depends on the type of system which also needs to be identified. In an SAP

environment, we will usually see systems based on SAP NetWeaver Application Server (SAP NW AS) and

so-called standalone engines like TREX, liveCache and IPC. Additionally, there can be clients like SAPMobile Infrastructure Client, SAP Business Explorer or SAP PI Adapter engines.

In addition to the type of system, you should list all sources of data (backup objects) as given above. This will

mainly determine the appropriate B&R strategy (technology, schedule, retention, etc.) as described in the

following sections. The following table provides an example with different SAP components:

Component

(Examples)

Type of

Component

Data Objects Data Source Possible Backup Options

(see 3.2 - 3.4)

SAP ERP SAP NW AS Application dataSystem

DatabaseFilesystem

Full / IncrementalFull / Incremental

SAP APO SAP NW AS Application data

System

Database

Filesystem

Full / Incremental

Full / Incremental

SAP liveCache Standalone Engine Application data

System

Database

Filesystem

Full / Incremental

Full / Incremental

SAP Optimizer Standalone Engine System Filesystem Full / Incremental / No backup

SAP Trex Standalone Engine Application data

System

Filesystem

Filesystem

Full / Incremental

Full / Incremental / No backup

SAP Gateway Standalone Engine System Filesystem Full / Incremental / No backup

SAP WebDispatcher Standalone Engine System Filesystem Full / Incremental / No backup

SAP MobileInfrastructure client

Client Application data

System

Database

Filesystem

Sync with back-end system

No backup

SAP Gui Client System Filesystem No backup




© 2010 SAP AG page 23/64

3.2 B&R for SAP NetWeaver Application Server-based Components

One of the most valuable assets of a company is the business data that is created and stored in the systems.

Only the concept of a database with its logging and forward recovery prevents the loss of committed data

when the system must be restored. For this reason, SAP generally uses an SAP NW AS-based system with

its underlying database to store business data.

A system based on an SAP NW AS consists of:

Database

Central instance and/or Central services instance

Additional application servers (dialog instances)

To avoid a loss of business data, the database and all database logs need to be backed up regularly, seeBackup of the Database (Application Data) .

All types of SAP NW AS (ABAP, Java and ABAP+Java) do not only store business data in the database but

also configuration data. Using the database mechanism, this configuration data is thus also protected from

data loss (unless you are doing changes directly in the filesystem).

The SAP software itself (kernel) on central instance and application servers, error logs or tracefiles, the

database software and operating system need to be backed up separately in addition to the regular database

backup, see Backup of the SAP and Database Filesystems (Software) and Server Backup (Bare Metal

Information) .

If you keep any application data outside the NW AS database, you must back up this data separately

by regular filesystem backups. This may apply for example to external or filesystem-based

repositories in Content Management, for TREX indices, for LDAP data, etc. Also, see B&R for

Standalone Engines and Clients .

Another example for application data in the filesystem, which you might want to backup regularly, are

transfer files used for data exchange by uploading data into the SAP system via f tp-interface or batch

input.

3.2.1 Backup of the Database (Application Data)

The B&R concept for a database consists of two parts: Regular backups of the database plus frequent

backups of the database logs (archive logs), which are written by the database. The tools and procedures for

database and log backup are comprehensively described in the database-specific administration guides

provided by SAP and the database vendors. For more information please see the respective guides.

Here, we only want to highlight some important aspects that are often neglected but are important to

safeguard the system and its availability.




© 2010 SAP AG page 24/64

3.2.1.1 Database Log Backup

The backup of the database logs (archive logs) is very important to prevent data loss. Whenever you have to

restore a system, the logs are needed to roll forward the database to the current point in time (see

Implications of an Incomplete Recovery regarding the consequences of incomplete recovery and data loss).

While there are multiple other database backups that could be used for a restore if one of the database

backups was destroyed, there is only one set of log backups that will be needed with every recovery !

Logs will be needed in a consecutive, uninterrupted series ranging from the backup to the current point. If a

logfile is missing or corrupt, recovery is not possible beyond that point! Complete recovery is only possible by

applying the most current online logs (which have not been archived yet!).

Data loss can only be avoided if you have a valid copy of all archive and online logs.

To protect the database logs and to make sure that you always have a correct copy of the logfiles, all

possible measures offered by the database or storage should be exploited so you are always able to perform

a complete recovery.

We generally recommend the following measures (which may differ on different database platforms):

1. Safeguard the online logs:

Typically, the database writes two copies of the online logs (on Oracle: origlog and

mirrlog). This feature should not be disabled. Both copies should be on separate physical

devices.

If possible, online logs should be replicated to a second storage system, as indicated by

(1) in figure 3. Synchronous replication allows avoiding data loss in case of a failure of the

primary storage and can provide an RPO of (or near) zero. (Note: This kind of data

replication is typically part of the DR concept.)

If log archiving occurs rarely due to low system activity, consider triggering logswitches at

fixed intervals . Unless you implement a replication of online logs, this interval (plus the time

needed to backup the archive file) should not be longer than your RPO.

2. Safeguard the archive logs:

Backup the archive logfiles in short intervals . In case the data is physically lost on the

storage system and the archive logs are not subject to replication to a second storage

system, this interval defines the RPO.

If possible, archive logs should be replicated to a second storage system, as indicated by

(1) in figure 3.

Typically, the database writes only one copy of archive logs into the filesystem. Most

database platforms support log duplexing by writing two (or more) copies into the filesystem

(on Oracle: log_archive_duplex_dest or log_archive_dest_2). This should be enabled toavoid the archive files being a single point of failure, as indicated by (2) in figure 3. Both

destinations should be on separate physical devices.




© 2010 SAP AG page 25/64

Log duplexing also offers the chance to protect from some types of corruptions in the archive

logs. In case one of the copies got corrupted, there is a chance that the second copy is ok.

When using log duplexing, both copies should be saved to the backup media.

Using brarchive on Oracle, there is a different approach. brarchive allows comparing the

duplexed copies during archiving. If the copies are not identical, brarchive will not delete

them from the filesystem and throw an alert.

Backup media (tapes) could become unreadable because of physical damage. You should

always have at least two backup copies of the archive logs on your backup media . This

can be achieved in different ways as indicated by (3) in figure 3:

o Saving two copies of the archive logs with the backup software, see figure 3 (B). On

Oracle, brarchive also allows double-saving, see figure 3 (A).o Duplicating the log tapes within the tape library or between two tape libraries, see

figure 3 (C).

o If archive logs are replicated to a second storage system, an additional copy of the

logfiles can be saved from the replica, see figure 3 (D).

No matter which method you use, it is important to ensure that both copies of the logs are

always located on different physical tapes . If possible, these should also be in different tape

libraries and different fire compartments.

If available, use checks offered by the backup tools to verify that the copy written to thebackup medium is identical to the original in the filesystem.

In contrast to possible corruptions that can be detected by comparing duplexed archive logs

or by the checks integrated in the backup tools, archive logs could still be internally corrupt.

Such logical corruptions cannot be detected by physical checks during the backup.

If available, use tools for logical validation of the archive logs . On Oracle, brarchive in

combination with RMAN allows some checks of internal log consistency.

A reliable verification of archive logs can only be done by applying them to a standby

database (log shipping).

3. Sufficient retention time

The retention time for archive log backups should follow your retention time for database backups.

For the oldest database backup that might be used for restore and recovery, a consecutive chain of

archive logs is required on the backup media as well. SAP recommends a retention time of 28 days

for database and archive log backups.

In addition to these backups allowing complete database recovery, there could be long-term backups

(for example monthly or yearly backups needed for legal reasons), which do not allow recovery. Note:

If these long-term backups are created as online backups, a specific number of archive logs will be

required with it to make this backup consistent.




© 2010 SAP AG page 26/64

RAID protection

oraarch

LGWR ARCH BRArchive

C

MirrlogAMirrlogB

Duplex_dest

Duplicate backup of archive logs by eitherA) BRArchive double-saving (Oracle)B) Log duplication through backup softwareC) Duplicate log tapesD) Save archive logs in both locationsMake sure that both copies are on different physical tapes !

A

Duplex Oracle archive logs using log_archive_duplex_destor log_archive_dest_2

Run binary comparison with BRArchive by setting“archive_dupl_del= check”

RAID protection

OriglogAOriglogB oraarch

MirrlogAMirrlogB

Duplex_dest

D

2

1

3

3

Backup SW Backup SW

B

Replicate online logs and archive logs synchronously (ifpossible)

2

1

OriglogAOriglogB

3

Synch.

Repl.

Figure 3: Possible Options for Log Safeguarding on Oracle

3.2.1.2 Database Checks

Regular database block checks are very important to protect your data. In the worst case, block corruptions

on the database can destroy the database and cause a loss of all business data . By running database

checks you should not only verify the integrity of your database but also ensure the integrity of your backups.

In case a check reports corruptions on the production database, you need to have an error-free backup

available. This would be the most recent backup taken before the last successful database check.

Database checks must be run at least once in a backup cycle to ensure that there always is a new corruption-

free backup before the oldest checked backup will be overwritten. A backup can be considered corruption-

free if a successful database check was run after that backup had finished.

Running the checks directly on the current database ensures that corruptions are detected in time so

that they can be repaired by applying a backup which is free of corruptions. But this approach leaves

a small risk that the backups might be corrupted by the backup hardware itself. To ensure both, the

correctness of the current database and the correctness of the backups, database checks can be

combined with regular restore tests, restoring a backup to an alternate location. Running the

database checks on a restored database additionally ensures that the backup hardware is not

causing corruptions.




© 2010 SAP AG page 27/64

We recommend running database checks on a weekly basis . See the database-specific SAP notes for

more information on database checks and the available tools (Oracle: 23345 , MSSQL: 142731 ,

MaxDB/liveCache: 940420 , 521870 , DB2/UDB: 98524 ). Service levels may require a higher frequency of

checks or additional measures to prevent a possibly severe outage resulting from block corruptions.

On very large databases, the DB check could take a long time to run so a regular check on the production

system may not be practicable. In this case you could run database checks on a physical, block-based copy

of the system (e.g. a storage-based mirror) or on a copy that was restored from the last full backup.

Never skip the DB checks completely as you risk loosing your system due to unusable

backups!

3.2.1.3 Full or Incremental Backup

In general, SAP suggests creating full backups of a database. For large databases, the need may arise to

reduce the backup runtime because the backup window is exceeded by the full backup or because the

backup infrastructure is no longer able to handle the backups of multiple systems in parallel (also see Backup

Time ). A common possibility to reduce the daily backup work is to perform incremental or partial backups of

the database.

A classic incremental backup routine is the so-called “Father-Son-Concept”. The “Father-Son-Concept”

describes the usage of full backups and incremental backups level 1. The “Father” is equivalent to a fullbackup and the son describes an incremental backup which depends on the “father” respectively the full

backup. Incremental backups with a higher level like the “Grandfather-Father-Son-Concept”, where one

incremental backup (level 2) depends on another incremental backup (level 1), are not advised for SAP

systems. If you plan to use incremental backups, always remember that in SAP terms it is an incremental

backup level 1.

Note: Incremental backups are only possible with online backups. Offline backups are always full

backups of the database.

Note: Some backup solution providers call an incremental backup level 1 a “differential” backup

because it always backs up the difference compared to the last full backup.

With incremental backups, the daily amount of backup data can be largely reduced, especially if you have a

large database with only a small volume of daily changes.

While incremental backups reduce the backup time, the restore and recovery time will increase

because in addition to restoring a full backup, the incremental backup needs to be applied to the

database before log recovery can be started. This tradeoff needs to be considered with respect to the

RTO that has to be covered by the service levels (see Recovery Time) . Very often full backups are

the fastest way for recovery because in most situations the latest backup of the past day is needed.


















© 2010 SAP AG page 28/64

Partial backups describe a concept where only a subset of all database objects (tablespaces, datafiles, etc.)

is backed up per day. A strategy for partial backups could be defined for example as:

o Full backup on Sundays

o Backup of the first third of the database objects on Mondays and Thursdays

o Backup of the second third of the database objects on Tuesdays and Fridays

o Backup of the third third of the database objects on Wednesdays and Saturdays

Similar as with incremental backups, there is a tradeoff between backup time and restore/recovery time. The

RTO will be significantly higher since forward recovery for parts of the database has to span archive logs for

multiple days (in the example up to three days). Using this approach, archive logs should be kept on disk for

some time to avoid the delay for restoring archive logfiles from tape.

In addition to the increase in RTO introduced with incremental and partial backups, administrative

complexity is also increased with these backup strategies. The procedures are more prone to

handling errors and thus may increase the duration of outages and the overall risk. If performed

incorrectly, there may be no valid backup available when required.

Failed backups could further increase complexity of a restore and recovery. A failed level 0 backup

for example would require either several level 1 backups or a large range of archivelogs to be

applied.

3.2.2 Backup of the SAP and Database Filesystems (Software)

Apart from business data located in the database, the SAP system and database software itself is worth

being backed up. While in principle, the SAP system and database can be installed newly, a restore from a

backup will be faster and will recover the system to the closest state it had before the error. As a general

recommendation, the application software should be backed up at least after it has been installed and after it

has been updated. Common practice is to take daily backups of all relevant filesystems. Additional backups

may be taken immediately after important changes were applied to the system.

When restoring a filesystem backup, all changes done since then are lost and need to be repeated. This may

apply to changes in the software (like SAP kernel patches or the SDM repository which contains information

on deployments on the J2EE engine). This does of course not apply to the configuration data which is kept in

the database and is either bootstrapped (as with NW AS Java) or can be exported to the filesystem (profiles

of NW AS ABAP).

The filesystem backup should include all shared directories and the Central instance respectively Central

services instance. Additional application servers (dialog instances) can be subject to backups as well but it is

also possible to just reinstall a dialog instance into the system. The recovery time for application servers is

usually not critical because there should be sufficient application servers installed to cope with the failure of

one of them.

To backup SAP and database software, the filesystem backup should include the following directories on

Unix (on Windows, it is recommended to backup the complete computer including the operating system):




© 2010 SAP AG page 29/64

/usr/sap/<SID>

/sapmnt/<SID>/usr/sap/trans

/<database-root-directory> (on the database server, excluding data files)

Home directories of SAP and database users (e.g. /home/<sid>adm, /home/ora<sid>, …)

SAP- and database-relevant data from /etc (services, password, groups, other configuration files)

Backing up these filesystems will automatically include SAP and database specific error logfiles and

configuration files.

Although it would be possible to selectively define backup strategies on a subdirectory level for directoriesbelow these high-level directories, we would not recommend this just to keep the effort low. If some

subdirectories should contain a large amount of irrelevant data, such directories could be specifically

excluded from the backups (e.g. log directories you would not want to back up).

In some situations it is advisable to stop or avoid special activities while performing a filesystem

backup. As such, it is recommended for example to stop the Software Deployment Manager (SDM)

during a backup to avoid changes in the filesystem.

Additional backups on special subdirectories could be performed after partial changes to the system. Forexample, an additional backup of the SDM directory on the central instance

(/usr/sap/<sid>/<instance_00>/SDM) is recommended to save the newest state of the SDM repository after

new deployments were done to the J2ee engine.

Any other directories you might use for application data outside of the database (like external repositories in

Content Management, ftp and batch input files) should be subject to the filesystem backups, too.

If other software is installed on the server, this may need to be considered for filesystem backups as well. For

low level (bare metal) backups of operating system and other system software, see Server Backup (Bare

Metal Information) .

3.2.3 Server Backup (Bare Metal Information)

If the server hardware of a system is lost completely (for example due to a server failure including its local

disks), the B&R concept has to provide a plan how to rebuild that server and the system (in addition to the

concept to recover its business data). The server can be either installed completely new or it can be restored

from a full system backup (including registry backup on Windows platforms). This is also called a bare metal

restore. A bare metal restore usually requires identical hardware for the restore.

The decision whether to backup server and software should mainly be based on an evaluation of the cost of

the backup and the duration of a restore compared to a new installation.

Whether you decide for a new installation or a bare metal restore, in case of a recovery, you will need some

basic information on the hardware and the system. This information should be documented in a server




© 2010 SAP AG page 30/64

handout providing a quick overview of the running systems and all necessary software including their

versions for all involved servers.

The server handout should include information like:

Current hardware:

If the circumstances force you to replace defect hardware, information is needed on server, local hard

disks, storage, partitions, hostnames, memory, etc.

Current operating system:

You need to know the type, version, patch levels.

Filesystem / storage layout :

When rebuilding a system, information is needed on filesystem structure, mount points, shares,

file/container sizes, layout of storage volumes, etc.

Systems running on the server:

The handout should include information on the SAP and/or database system on this server.

Third party software:

This should provide information on any installed software components like logical volume managers,

JDK, NFS, cluster software, and so on, that can be important during the recovery process.

The SAP Solution Manager could be used to document this kind of information - but note that this information

should also be at hand if all systems are not available.

3.2.4 Restoring SAP NW AS

The purpose of the backups is to enable a restore of the environment. A normal restore scenario assumes

that only a single system of the environment is affected by the restore. If the complete environment with

multiple systems needs to be restored, this would be considered as disaster recovery (see Disaster Recovery

- Basic Approaches using B&R ).

Based on the extent of the damage and on which parts of a system need to be restored, the restore of a

system can require different phases. The restore of a complete NW AS from scratch mainly consists of the

following phases:1. Restore basic server operability

2. Restore software components

3. Repeat any lost changes done in the filesystem

4. Restore application data

5. Recover database logs to current

6. Resolve possible application-related inconsistencies

If only parts of the system are affected, only specific phases of this process may need to be executed. Forexample if only the database data files were lost due to a storage failure, only phases 4 and 5 for restoring

and recovering the database or individual data files will be required.




© 2010 SAP AG page 31/64

3.2.4.1 Restore Basic Server Operability

In the case of a complete hardware loss of a server, the server needs to be rebuilt either by new installation

or by a bare metal restore, see Server Backup (Bare Metal Information) . This typically includes operating

system and basic system software (like logical volume managers, JDK, etc.) that was installed on this server.

3.2.4.2 Restore Software Components

Once the server is available, all application software (database and/or SAP) that was running on this server

needs to be rebuilt. This can be done by using the filesystem backup created according to Backup of the SAP

and Database Filesystems (Software) .

3.2.4.3 Repeat Any Lost Changes Done in the Filesystem

When restoring a filesystem backup, all changes that were done after the backup are lost and need to berepeated. This may apply to:

Changes in the software (like SAP kernel patches or SDM information on deployments on the J2ee

engine)

Specific administrative activities on the Enterprise Portal. The Portal stores transport packages of

Portal content and data of XML forms builder in the filesystem only. Some data of this kind may need

to be rebuilt if such activities were performed after the filesystem backup.

This does not apply to the configuration data, which is kept in the database and is either bootstrapped (as

with NW AS Java) or can be exported to the filesystem (profiles of NW AS ABAP). If configuration changeswere done directly in the filesystem, these changes may be lost as well and need to be redone. For NW AS

Java and ABAP+Java, the Java bootstrap program synchronizes the binary data from the Java database with

the local file system and updates a property file, which describes the processes of the Java instance

(instance.properties). The bootstrap section of instance.properties is required for the bootstrap to work. Data

consistency between DB and FS is guaranteed by the bootstrapping.

Depending on the nature of the changes that were applied, this step may also be executed at a later

time (unless there are dependencies between this information and the application data to be restored

in the next phase).

3.2.4.4 Restore Application Data

When the server and all software are available, the database can be restored from the database backup.

3.2.4.5 Recover Database

After the restore, the database forward recovery is done by recovering the archive and online logs. As

described before, the goal should be to perform a complete recovery avoiding data loss.

3.2.4.6 Resolve possible application-related inconsistencies

If any information could not be recovered to the current point in time (that is the point of the crash), there is a

risk of data inconsistencies. This could occur:




© 2010 SAP AG page 32/64

Within a system if there are for example dependencies between data in the database and data in the

filesystem; like data files used for data upload into an SAP system or like external repositories

(‘DBFS’ repositories) in SAP Content Management.

Between systems of the landscape if a complete recovery was not possible in the previous step.

For more background information, also see Backup Requirements in a System Landscape . For approaches

for resolving such inconsistencies, see Managing Incomplete Recovery .

3.3 B&R for Standalone Engines

In an SAP environment, you will also find non-NetWeaver AS-based components providing specific

functionality to the business. For the B&R concept, it is important to identify if such standalone engines hold

application data of their own. If they do, you need to know which type of data storage is used: Database or

flat file.

A standalone engine storing application data in a database is for example SAP liveCache. Standalone

engines storing application data in the filesystem are for example TREX (search indexes) or Content Server

(documents or other types of content). Standalone engines without application data are for example SAP

APO Optimizer and SAP WebDispatcher.

In addition to SAP standalone engines, your environment may also relay on other components like

external wikis, mailstorage or archiving systems, LDAP or UME which you have to consider in yourB&R concept.

3.3.1 Standalone Engines with Application Data

3.3.1.1 Backing up the Application Data

Application data of standalone engines should generally be backed up using a database respectively

filesystem backup (similar to Backup of the Database (Application Data) and Backup of the SAP and

Database Filesystems (Software) ), depending on the type of data storage used by that component. Although

in some cases, this data may completely be derived from leading data stored in other places like a NW AS

system, restoring such data is generally faster than rebuilding the data from its sources (as an example

consider TREX indexes which in principle could be rebuilt from the indexed data).

Nonetheless, in specific cases customers might decide not to take regular backups but instead rely on

rebuilding a component’s data from other sources, if the component should be lost due to a failure. In that

case, you need to be absolutely sure that no unique information would be lost on that component.

Examples:

1) SAP liveCache is a component that is initially loaded from the APO database. But in the course of time,

the liveCache creates application data that is not available on other sources. Since all this data would be

lost when reinitializing the liveCache, it is vital to have a backup concept for the liveCache database.2) TREX indexes can always be rebuilt from the indexed data. The recommendation for backups is mainly

based on the time it takes to restore a backup compared to completely rebuilding the indexes.




© 2010 SAP AG page 33/64

On the other hand, an advantage of rebuilding the data would be consistency. When restoring an engine’s

application data from a filesystem backup, some data created after the backup may be lost. This will cause

inconsistencies in relation to the sources where the data comes from. So in this case, additional activities

may be required to catch up and reestablish consistency. More information on this procedure needs to be

provided by the specific component.

In some situations it is advisable to stop or avoid special activities while performing a filesystem

backup. During an online backup of TREX indexes for example, it is recommended to ensure that no

write accesses take place though creation, deletion or resetting of indexes, changing of attributes,

indexing or changing of taxonomies.

3.3.1.2 Backing up the Standalone Engine Itself

Apart from the application data, a standalone engine has other types of objects to be considered for possible

backup: Software, configuration and logs. The decision whether to backup ‘the engine itself’ depends on an

evaluation of the cost of the backup and the duration of a restore compared to a new installation and

configuration. In many cases, setting up a standalone engine will not cause a large effort so there will be no

need for a backup. Backups will mainly be advisable if the installation requires a larger amount of time or

some complex configuration work. In those cases, you should perform regular filesystem backups of thesoftware and server (similar to Backup of the SAP and Database Filesystems (Software) and Server Backup

(Bare Metal Information) ). If the engine writes important logfiles of its activities, you might also want to take

regular backups of these logs.

3.3.2 Standalone Engines without Application Data

Standalone engines without application data can usually be reinstalled from scratch if the server should be

lost. A backup would only be preferable if the restore could provide a shorter recovery time than the

reinstallation or if the component requires complex configuration. A backup might also be desired if the

component generates important logfiles you might want to preserve.

For backing up such components, a regular filesystem backup and a backup after significant changes will

generally be sufficient.

3.4 Clients

Clients are additional programs or tools which typically reside on local front-end PCs accessed by the users.

Alternatively, they might be provided on back-end systems where they act as client programs within an SAP

system landscape. Clients are for example Adobe LiveCycle Designer, SAP Business Explorer, SAP Mobile

Infrastructure Client or SAP Gui.

Clients usually do not hold application data of their own, except for local files created by individual users.

Clients could be reinstalled if the local front-end or the back-end server should be lost, so in general there is

no backup required. A backup may only be desired if the reinstallation and configuration is time-consuming

and complex, especially if the client is installed on a common back-end system.




© 2010 SAP AG page 34/64

The SAP Mobile Infrastructure Client is a special kind of client which stores application data locally in a

database on a mobile computer. Data from this local database and the back-end system (for example SAP

CRM) is regularly synchronized when the mobile user connects to the office. If the mobile computer is lost,

the local database can be recreated from the data in the respective SAP back-end system. Since any data

that was created locally but not yet uploaded into the back-end system would be lost, we recommend

synchronizing with the back-end system on a frequent basis.

3.5 Further B&R Recommendations

For all kind of components there is general advice, which will help you lower the risk of a data loss after a

recovery and enable achievement of service levels.

3.5.1 Contents of a B&R Concept

The B&R concept must provide all information on performing the backups but it also has to provide all

information that will be needed when performing a restore and recovery. This information must be provided

for all components that are part of the landscape as identified in Identify Components to be Backed Up . Even

if a component is not backed up by intent, this fact must be documented in the B&R concept and the planned

approach to rebuild that component must be described.

The B&R concept should provide the following information:

General Information for B&R:

Components in the system landscape

Availability requirements and applicable service levels for components in the landscape

Backup infrastructure (backup hardware and software)

Tape lifetime and expiration

Policy for database checks

Backup requirements of development systems and non-productive systems (test, training, etc.)

Backup Plan (for Each Component):Objects to be backed up

o Application data (database, database logs, data in filesystem)

o System (backup of server, software, configuration, log files)

Backup method for the different objects

Backup technology, tools

Components or objects without backup

Backup schedule, start time and backup duration

Scheduling tool




© 2010 SAP AG page 35/64

Backup cycle (retention time)

Replicas of backups (especially of database archive logs)Backup checks for correctness, completeness and usability

Backup monitoring

Alerting on errors during backup

Operating instructions for the backup procedure

Special backups (backups after maintenance, backups for long-term retention, consistent landscape

backups , etc.)

Restore and Recovery Plan (for Each Component):Description of error scenarios with their respective solutions

Restore method for the different backup objects listed above

If there is no backup: all required information to rebuild that component (e.g. documentation on

installation and configuration)

Disaster recovery (if performed through restore and recovery)

Detailed operating instructions for restore and recovery

Options for restoring onto a spare hardware or sandbox (to build up an analysis system)Checks after restore

Restore dependencies (possible side-effects on other systems or components)

Measures to check or reestablish data consistency (if required)

Handling of incomplete recoveries (business recovery versus restore of a consistent landscape

backup )

Procedures and tools for business recovery or for restore of a consistent landscape backup

Testing and Change Control Management Plan (for Each Component):Test plan for restore and recovery testing and for administrator training

Verification of database archive logs

Test environment

Responsibilities for testing

Plans and responsibilities for review and updating the B&R concept

3.5.2 Backup Schedule

For the scheduling of your planned backups of filesystem and database, you need to know the backup

runtime as well as the service levels and the availability requirements of the system. A backup usually has an




© 2010 SAP AG page 36/64

impact on the load of your system (especially the I/O load on storage and network devices), unless you are

using non-impact backup technologies like split-mirror backup. You should plan the schedule of the backups

outside of the business hours when there is less (user) activity in the system.

The backup scheduling can also collide with planned jobs that are activated during the night. You should get

in touch with the responsible process owners to identify an acceptable backup window to avoid interference

with other important business processes on the system.

Sometimes it is not easy to identify a good backup window for the scheduled backups. There are possibilities

and technologies to reduce the time needed for the backup. While this will not necessarily reduce the needed

system resources, it may permit releasing the system back faster to normal operation. If this is not sufficient,

there are technologies that allow further reducing backup time and system resources needed for backup, see

Backup Time .

The backup scheduling should be documented in the B&R concept. The concept should also mention the

start and the estimated end time of the backup. The backup scheduling needs to be regularly checked since

backup runtimes will steadily increase with growing data volumes.

3.5.3 Retention

The retention period defines how long a backup is protected from being overwritten. The retention policy must

be defined according to the agreed service levels. Without a well-planned retention time you might

experience data loss if your last full backup was faulty or was overwritten accidently.

The retention can be critical for a recovery and must be defined in correlation with the database checks asdescribed in Database Checks . The retention policy for database archive logs should retain consecutive logs

allowing recovery from multiple full backups and reaching back sufficiently long into the past, see Database

Log Backup . With incremental backups, the retention time must make sure that there are always sufficient full

backups available and that recovery is also possible if some incremental backup became unusable.

3.5.4 Backup and System Maintenance

SAP Systems are maintained regularly with patches and support stacks. Maintenance activities should be

considered in the backup concept. It is good practice to take a full backup before and after every major

maintenance. Whenever changes have been applied to the software in the filesystem, we recommend taking

a timely filesystem backup so the most current state is saved and can be recovered from a backup.

You can use Change Management in SAP Solution Manager to plan and coordinate maintenance

and backup tasks. You should always plan a backup after a maintenance of a SAP system!

3.5.5 Backup Verification

When the need arises to restore a system from a backup, it is vital that the backup is correct and complete.

We recommend exploiting the different kinds of checks offered by the different backup tools. Using the SAPtool brbackup for Oracle for example, a verification is done whether all relevant database objects belonging to

an SAP system are actually included in the backup. Third-party backup tools offer backup checks for integrity




© 2010 SAP AG page 37/64

of the backups like a block-based verification of the data that is being written to tape or at least a verification

of the readability of the tape.

Similarly, some tools offer checks for the database archive logs, for example SAP brarchive provides a

feature to compare the backup of an archive log byte-by-byte against the original on disk.

Note that backup checks usually can only check the correctness of the data being written to the backup

medium. This does not verify whether the data itself is correct. The database or some archive log could stillbe internally corrupt. This can only be detected with database checks as described in Database Checks .

Since all backup checks cannot completely ensure that everything is correct with the backups and the backup

procedure, it is also important to regularly execute restore and recovery tests, see Testing and Training .

3.5.6 Testing and Training

A very important factor for the operability of a restore in case of an emergency is thorough testing and

training. Only tests can show whether a restore and recovery will actually be successful. Testing is also

needed to verify the operating instructions and point out possible deficiencies.

Testing will also provide regular training of administrators on the restore scenarios and procedures. Since

restore and recovery is a rare event, sufficient training is important to avoid delays and ensure an error-free

execution of a restore and recovery.

A test plan should describe:

Frequency and schedule of tests

Restore and recovery tests for application data (test of database restore and test of archive log

recovery)

Restore tests for non-database components

Restore of the system (filesystem and bare metal restore)

Consistency checks after restore

We recommend performing a full restore and recovery test at least twice per year. If the backups are used to

perform system copies with the database restore method, this may be considered as a test as well (if theprocedure includes the log recovery process).

3.5.7 Continuous Updating and Change Control

The B&R concept must be continuously updated to reflect changes in the environment, the technologies

being used and the operational procedures. This process must tightly integrate with B&R testing so that any

deficiencies detected during the tests will be incorporated into the B&R concept.




© 2010 SAP AG page 38/64

4 Disaster Recovery - Basic Approaches using B&R

4.1 Disaster Recovery Tiers

Disaster Recovery (DR) describes a situation where the complete production environment, often consisting of

multiple systems, needs to be rebuilt due to some catastrophic event. This is usually done at an alternate site

and we have to assume that the complete production hardware or even the complete primary site is no longer

available.

A basic DR solution is the so-called Pickup Truck Access Method . This solution relies on regularly shipping a

set of backup tapes to a secure location. In case of a major disaster, these tapes could be restored onto a

hardware infrastructure to be built newly. Speeding up DR, various tiers of disaster recovery options are

provided by partners, relying on a dedicated DR site with an in-place hardware infrastructure. Such moreadvanced DR solutions are based on data replication in combination with a secondary instance where

operations can be switched over to. This could be a standby database or a secondary database that can be

started on a replica of the data. Data replication can be realized on different layers like storage-based

replication, host-based replication or software-based replication.

Depending on the risk assessment and recovery requirements, companies can choose from several tiers of

DR options:

Externalization of backup sets, also called “Pickup truck access method” (PTAM)

Backup replication

Point in time copies

Log shipping to a standby database

Asynchronous data replication

Synchronous data replication

A more detailed description of these technology options or an assessment which of these technologies would

be appropriate for a specific customer environment is out of scope of this document. For further information

see the Best Practice “ Business Continuity Management for SAP System Landscapes ”.

4.2 Phases of a DR using the B&R Approach

In this document on B&R, we want to have a look at the restore as a basic approach for DR, that is, the first

two tiers from the list above. By discussing the requirements during the different phases of a DR, we also

want to show what needs to be considered when planning to use the B&R solution for DR too.

DR using a basic B&R approach will require the following phases that will be discussed in the following

sections:

1. Provide hardware2. Provide system software





© 2010 SAP AG page 39/64

3. Provide application software

4. Restore application data

4.2.1 Providing the DR Hardware

Depending on the DR approach demanded by a customer, the hardware for DR operations may already be

available at the DR site or it may only be provided once the disaster occurred. In the latter case, there should

be an agreement with the hardware provider defining the maximum provisioning time for the required

hardware. The hardware should basically be the same as the production hardware. Information about the

required hardware must be available at the DR site in case of a disaster (for example, in the form of a server

handout as described in Server Backup (Bare Metal Information) ).

4.2.2 Providing the System Software

If the DR hardware is already available at the DR site, the basic system software is usually already installed

on the servers. In this case, all software and configuration changes which are applied to the production

environment must also be repeated or transferred onto the DR environment. This requires a rigorous change

management process.

Alternatively, or if the hardware is newly provided, the system software must be either installed or be restored

from a bare metal backup as described in Server Backup (Bare Metal Information) and Restore Basic Server

Operability . All corresponding information about the infrastructure must be available at the DR site, too. In

case of an installation, it is important to make sure that only the same or compatible software versions areused and that the configuration is done in accordance with the previous configuration on production. In case

of a restore, the backup must have been shipped to the DR site (for example together with the database

backup).

4.2.3 Providing the Application Software

With preconfigured DR servers, the database and SAP application software is often also already installed on

the DR servers. In this case, it is important that the software is updated and kept in sync with the production

system. This requires that all software or configuration changes which are applied to the production

environment are also repeated or transferred onto the DR environment. This requires a rigorous changemanagement process. On SAP level, the tool SAPCPE can be used for keeping the binaries in sync, for

example by scheduling a regular synchronization from the central shares on the primary system.

An alternative would be to restore the application software from a filesystem backup or to perform a new

installation as described in Backup of the SAP and Database Filesystems (Software) and Restore Software

Components . This will of course imply a longer recovery time for DR.

4.2.4 Restoring the Application Data

The application data has to be restored from a backup that has been made available at the DR site. Here we

can distinguish the following approaches:




© 2010 SAP AG page 40/64

Regularly shipping a full backup to the DR site.

This can be done by physically shipping backup tapes from the primary to the DR site or to a safelocation (bunker) in between. This can also be achieved by replicating a backup over a network

connection to the DR site. For this, the backup can be either on tape (for example tape-based

replication between two tape libraries) or on disk. The RPO will be determined by the frequency of

shipment respectively replication.

To enable a successful restore, all basic database configuration files must also be available at the DR

site (should usually be included in a database backup).

If the backup is an online backup, it must be a ‘consistent online backup’, meaning that a specific

number of archive logs must be included in the backup set, allowing a database recovery to a

consistent state.

For filesystem backups, a similar approach can be used.

In addition to shipping full backups, regularly transferring database archive logs to the DR site.

The intent of this approach is to reduce the amount of data loss (the RPO) when activating the DR

site by applying all available archive logs to the restored database. See Data Consistency with DR

using the B&R Approach regarding landscape consistency with this approach.

As a prerequisite before restoring the backup on the DR host, the necessary filesystem structure must have

been created. All profiles and backup logfiles needed for the restore must be available on the DR host, for

example by extracting them from the backup set in a first step.

Once all prerequisites are met, the restore and recovery of the database can be performed at the DR site.

In case of DR, a system is restored to a different physical host. Backup tools in general support

restoring backups to a host having a different hostname.

For SAP NW AS Java, the possibility of restoring a backup is restricted to ‘the host from which you

performed the backup’ (according to SAP note 779708 ). This means that the same physical or virtual

hostname is demanded at the DR site that was used on the primary site.

4.3 Data Consistency with DR using the B&R Approach

In Phases of a DR using the B&R Approach , we were only talking about DR for a single system. In a system

landscape, we also have to analyze what this DR approach means for data consistency between the

systems.

Restoring a backup at the DR site will cause data loss (according to the RPO that was defined for DR).

Restoring the backups of multiple systems at a DR site may cause a different amount of data loss in the

different systems and thus leave these systems at different points in time. This means that there will be data

inconsistencies between the systems. This can only be avoided by shipping a consistent set of backups for allsystems. Consistent Landscape Backups shows how such consistent landscape backups can be created.






© 2010 SAP AG page 41/64

Note that even when shipping a consistent landscape backup to the DR site, there will probably still

be a need to deal with inconsistencies when actually activating the DR site: Inconsistencies to othersystems not included in this landscape as well as inconsistencies with respect to ‘the real world’.

Using backups for DR, it may be possible to reduce data loss at the DR site by applying additional archive

logs to the databases at the DR site. In this case, it is important to note that data consistency between the

systems cannot be ensured due to the nature of the point-in-time recoveries. Even when recovering all

databases to the same point in time and even when using a time synchronization server, there is no final

guarantee that the recoveries in the different systems will be synchronized up to the latest transactions; so

there could still be some inconsistencies (also, see C reating Consistent Landscape Backups ).

This also applies if a consistent landscape backup is available at the DR site but the systems are rolledforward by applying archive logs. Consistency will be lost.

Whenever the DR approach cannot guarantee consistency of the system landscape when activating

the DR site, the DR procedure must consider the resolution of data inconsistencies and should

include a procedure for business recovery into the DR plan. For more information on business

recovery and establishing this as part of a recovery plan, see Managing Incomplete Recovery .

Due to the implications DR can have on data consistency and thus on business continuity, activating

DR must be a careful decision to be confirmed by a ‘crisis management team’.

Similar considerations regarding data consistency must also be made in case of a ‘partial DR’, when

only some of the systems of a landscape are switched to the DR site.

The implications on data consistency between systems must also be considered with other DR tiers

as mentioned in Disaster Recovery Tiers .




© 2010 SAP AG page 42/64

5 Managing Incomplete Recovery

Databases offer the possibility to perform a point-in-time recovery, which means recovering the database withthe help of the database logs to any point in time between a backup and the current time. Previously, point-in-

time recovery (or incomplete recovery) was considered an acceptable option to remove logical errors from a

system by recovering the system to a point before that error occurred. In a system landscape of multiple

databases exchanging data, a point-in-time recovery is less feasible since it will cause data inconsistencies

between the systems, as discussed in Implications of an Incomplete Recovery . The inconsistencies need to

be resolved in an additional recovery phase we call business recovery. Because of this, incomplete

recovery needs to be avoided in a system landscape and alternate approaches for handling logical errors

should be exploited, see Avoiding Incomplete Recovery .

Before doing an incomplete recovery of a productive system, open a call at SAP and check if

the situation can be solved using SAP Note 434645 or the Best Practice “ Emergency Handling for

Recovery of SAP System Landscapes ”.

But still, in rare cases, there might be recovery situations when an incomplete recovery of a system cannot be

avoided. This raises the need to identify and deal with the resulting inconsistencies between the systems.

Since these activities need to be part of the recovery process, they should be introduced in a B&R concept,

see Performing an Incomplete Recovery in a System Landscape and Handling Data Inconsistencies between

Systems .

5.1 Avoiding Incomplete Recovery

As discussed above, incomplete recovery of a system in a system landscape will result in data

inconsistencies between the systems of a landscape and needs to be avoided whenever possible. This

section looks at possible reasons for incomplete recovery and shows ways of avoiding it.

Incomplete recovery may result mainly for the following reasons which will be regarded more closely in the

next sections:

Technical problems when trying to perform a complete database recovery

Incomplete recovery as a means to back out logical errors

Restoring the backup of a filesystem

5.1.1 Incomplete Recovery Due to Technical Problems

A main task of the B&R concept must be to avoid technical problems becoming a reason for an incomplete

recovery. The following measures should be taken to prevent incomplete recoveries due to technical

problems:











© 2010 SAP AG page 43/64

Possible Reasons for Incomplete Recovery Countermeasures

One of the logfiles in the consecutive chain starting

from the last backup is not available or corrupt

Specifically protect logfiles as described in Database

Log Backup

The online logs are not available Specifically protect logfiles as described in Database

Log Backup

There is no log backup at all Setup proper backup monitoring and alerting,

perform regular restore tests

The last full backup is unusable and there are not

sufficient logfiles to bridge the gap from the

previous backup

Have a sufficiently long retention time for archive logs

(reaching back to more than one full backup), see

Database Log Backup

An incremental backup is unusable and there are

not sufficient logfiles to bridge the gap from the

previous incremental or full backup

Have a sufficiently long retention time for archive logs

(reaching back to more than one full backup), see

Database Log Backup

A datafile or other database objects may be

missing in the backup

Use checks to ensure the completeness of the backup,perform regular restore tests, see Backup Verification

and Testing and Training

5.1.2 Approaches for Handling Logical Errors

Logical errors are errors in the application data which affect the correctness of the business functions

processing that data. Logical errors can usually not be detected by the database system or the SAP basis.

Logical errors can be caused for example by bugs in application programs, by user errors when entering data

or by administrator errors. Typical examples for logical errors are accidental deletion of data from a database

table, corruption of data due to a faulty program or import of wrong transports into a system.

To avoid an incomplete recovery, logical errors should be repaired directly in the affected system.

Even if the effort to fix the error may be very high, this is in most cases still preferable to the data loss and the

data inconsistencies that will be caused by an incomplete recovery. While logical errors usually only affect aspecific area of business operations; incomplete recovery will have an impact on the complete system and all

business processes using this system. Before deciding on an incomplete recovery, the tradeoff between

‘Data Repair’ and ‘Incomplete Recovery followed by b usiness recovery ’ needs to be carefully evaluated.

Figure 4 depicts both options.

Considering the examples for logical errors above, the following approaches should be evaluated for repair:

Deleted data should be recovered from a copy of the productive system that could for example be

built up by restoring to a sandbox.

Corruptions caused by programs should be fixed by developing a corrective program.

Wrong transports should be corrected by developing and importing a corrective transport.




© 2010 SAP AG page 44/64

Uptime Uptime

Uptimeavailability of some processes affected

System 1

System 2

Uptime Uptime

Uptime availability of many processes affected

System 1

System 2

ErrorDetection

time

Unavailability of affected business process

Downtime of complete system Unavailability of cross-system business processes

Incomplete Recovery to t O Business recovery between systems

Analysis System 1

Data Repair Phase

ErrorOccurrence

t O t D

Incomplete recovery+ Business recovery

Build ananalysis system

Data repair supportedby an analysis system

System reflects time t O

Data Repair

e.g. incompleterecovery to t O

Figure 4: “Data Repair” versus “Incomplete Recovery plus Business Recovery”

For further examples on how to avoid a point-in-time recovery in case of logical errors, see SAP note 434645 .

More information on a general approach to handle logical errors and a flowchart supporting with the repair of

logical errors can be found in section “Data Repair” of the Best Practice “ Emergency Handling for Recovery of

SAP System Landscapes ”.

A B&R concept should clearly state that an incomplete recovery should generally not be used

in case of logical errors.

A special approval process should be implemented for incomplete recoveries. Before

approving, careful evaluation of the alternatives and consequences should be done.

Providing a temporary ‘analysis system’ as a copy of production at the point before a logical error

occurred can support the repair of logical errors (as indicated in figure 4). This system can be built by

restoring a backup onto a sandbox or a spare environment. If you want to provide this option, the

prerequisites and procedure should be documented in the B&R concept.












© 2010 SAP AG page 45/64

5.1.3 Restore of a Filesystem Backup

Filesystems do not provide the possibility to restore data to the most current point in time because there is no

concept of continuous logging as with a database. Restoring a filesystem means restoring the latest backup,

which will result in some data loss if changes were applied to the data. This means that all changes that were

applied after this backup need to be repeated in the system. This may typically apply to software changes like

patches or updates. To avoid this, it is good practice to perform an immediate, additional backup after any

such changes.

If application data is stored in a filesystem, a restore will also cause a loss of all changes that were done to

the data since the last backup. Because of this, such data should not have “strong” dependencies to other

data stored in a database or there should be procedures available to reconcile that data with other dependant

data. Often, data in the filesystem is replicated or derived from source data in other systems (where it is

stored in a database). The reconciliation can then basically be done in two ways:

Using tools to resynchronize the data and ‘re-transfer’ any data that has been lost

Example: in SAP Knowledge Management, CM repository checks allow checking content and

metadata between filesystem and database

Reinitializing the replicated or derived data in the filesystem from the sources in other systems

Examples: Taxonomies in SAP Knowledge Management can be rebuilt; TREX indexes can be re-

indexed or rebuilt if required

Also, see Backup of the SAP and Database Filesystems (Software ), Server Backup (Bare Metal Information )and Restoring SAP NW AS regarding B&R of filesystems.

5.2 Performing an Incomplete Recovery in a System Landscape

If an incomplete recovery of a system in a system landscape is unavoidable and the decision was made after

weighing all the consequences discussed above, the most common approach will be as follows:

1. Shutdown the affected system. Other systems may possibly keep on running (depending on the

error).

2. Perform incomplete recovery of the affected system using the backup and a certain number of logs.

3. Perform business recovery for all interfaces to connected systems (see Handling Data

Inconsistencies Between Systems ).

4. Reconstruct other lost data which has not been recreated during business recovery.

5. Release the affected system back into operations once data quality has been reestablished to an

acceptable degree (to be confirmed by the business departments).

This approach corresponds to alternative (A) as described in Implications of an Incomplete Recovery and

figure 2. As discussed there, other alternatives like restoring a consistent landscape backup have other

drawbacks and are hardly practicable. If nonetheless the design of your B&R concept showed that one of the

other approaches would be more appropriate for your environment, follow that approach to perform the




© 2010 SAP AG page 46/64

incomplete recovery at this stage. In that case, there will probably still be a requirement to perform business

recovery for interfaces to other systems or the real world.

To relieve the impact of data loss and to provide options to at least partly recover lost data, we

generally recommend preserving a copy of the original system when doing a restore. This may

require preparatory thoughts to be considered in the B&R concept, providing the ability to copy any

system to some spare hardware or a backup device at any time without further delaying the acutal

recovery process (this can be achieved for example using storage technology). This copy may then

be used to track and recover lost data in the restored system.

5.3 Handling Data Inconsistencies Between SystemsIf one of the systems in a system landscape faces data loss due to an incomplete recovery, there will be

inconsistencies in the application data that has been exchanged with other systems of the landscape. In that

respect, these data inconsistencies are a kind of logical error in the application data, but in contrast to thelogical errors regarded in Approaches for Handling Logical Errors these data inconsistencies do not occur

within a single system but between different systems. This will raise the need to identify and remove these

inconsistencies in a special recovery phase that we call ‘b usiness recovery ’.

At this point it is important to note that business recovery will be required in any situation where the

different systems of a system landscape are recovered to different points in time and thus experiencepossibly different amounts of data loss. This could occur for example during disaster recovery (see

Disaster Recovery - Basic Approaches using B&R ).

Other causes for data inconsistencies can be for example incorrect programming, non-transactional

and non-restartable customer-specific interfaces (see SAP Communication Technology ), faulty user

input or data stored outside of databases (see Restore of a Filesystem Backup ).

For more background information on data consistency and data integrity in SAP environments also

see the whitepaper “ SAP Standards for Data Integrity and Transactional Consistency ”.

As the errors caused by inconsistencies might get worse the longer they stay uncorrected, thecorrection should be started as quickly as possible. If inconsistencies are detected, the business has

to decide whether work can continue in the system or whether all or some specific applications have

to be stopped.

5.3.1 Performing Business Recovery

Analyzing and fixing inconsistencies can be a complex process that requires very good knowledge of the

business processes and data objects. In case of an incomplete recovery, business recovery must be done for

all interfaces to all systems where data is exchanged and modified (interfaces that only implement reading

access to remote data need not be considered). Experts from the business departments and from basisadministration must work together closely on managing incomplete recoveries.







© 2010 SAP AG page 47/64

The following approaches are possible for business recovery :

Tools on application- or object-level:This approach is based on tools or reports that allow comparing data objects between different

systems. If inconsistencies are detected, such tools ideally allow fixing the problem by retransferring

or deleting inconsistent objects. SAP provides consistency check tools for various applications and

business objects. More information on available tools from SAP can be found in the Best Practice“ Data Consistency Check for Logistics ”.

Message-based approaches:

In some cases it may be possible to identify and correct inconsistencies by analyzing the message

transfer that took place between the systems and possibly re-send previously processed messages.

This is a possible approach mainly for ALE communication.

Initialization of data:

Inconsistent data that is completely replicated from another source can possibly be f ixed by reloading

that data. This may be possible for example for some master data objects.

For more information on these approaches to handle data inconsistencies between systems and for a

flowchart supporting with a business recovery see section “Business Recovery” in the Best Practice

“Emergency Handling for Recovery of SAP System Landscapes ”.

To perform a business recovery , comprehensive business information will be needed. You will need to knowwhich other systems were exchanging data with the affected system and which business objects (business

data) were exchanged by the business processes spanning these systems. This information should be

collected together with representatives of the different business units and should be mentioned in the backup

concept.

Usually, a lot of this required business information is already documented, for example in the Solution

Manager where a description of the system landscape, the business processes, the data flow and the

interfaces can be maintained.

Incomplete recovery can also affect data consistency with external non-SAP components. It will also

affect “the real world” for example in a way that the system does no longer know which documents

were printed and sent out during the period in question. All these aspects must as well be considered

during business recovery .

5.3.2 Preparing for Business Recovery

Incomplete recovery and business recovery can require a large amount of time and are thus a critical threat

to continuity of operations. Since this can be a consequence of a restore operation, a B&R concept should

touch on strategies on how to deal with data loss or data inconsistencies during the business recovery .

This includes providing documentation on:

http://service.sap.com/~sapidb/011000358700001955832008E









© 2010 SAP AG page 48/64

Business processes and the resulting dependencies between systems of a system landscape

Interfaces between systemsData objects being exchanged between systems

Available tools for data consistency checks

To prepare for a possible business recovery and to reduce potential recovery time, we recommend a more

comprehensive approach under the umbrella of Business Continuity Management as described in the Best

Practice “ Business Continuity Management for SAP System Landscapes ”. That approach incorporates

application-related threats like data inconsistencies into a business continuity concept by documenting

possibly affected business processes and business objects and by documenting available recovery options to

address these threats.

That document also includes an example of a recovery plan for the incomplete recovery of an SAP

system in a system landscape.







© 2010 SAP AG page 49/64

6 Consistent Landscape Backups

A “consistent landscape backup ” is a common backup of a predefined set of systems which enables restoringall included systems to a logically consistent state of the application data. Consistency means that all

application data that is exchanged between the systems reflects a common (consistent) point in time across

the system landscape.

As seen in Backup Requirements in a System Landscape , a consistent landscape backup is not mandatory

for a regular B&R strategy for an SAP system landscape. It is generally sufficient to individually backup each

system and all its information in order to be able to restore the system and all its application data.

But still there are several use cases where a consistent landscape backup can be beneficial for serving

special purposes.

6.1 Possible Use Cases

A consistent landscape backup can serve the following purposes:

System landscape copy:

When copying the systems of a system landscape onto a target landscape (for example to create a

non-productive environment for testing), a consistent copy of the landscape can be obtained by

restoring a consistent landscape backup onto all systems of the target landscape. Whether such a

consistent copy is actually needed in a specific customer situation depends on the nature and usage ofthe target landscape. While full consistency may usually be desirable for example for an integration test

landscape, a training landscape will often be sufficient without fully consistent test data.

For more information on system landscape copy, see the Best Practice “SAP System Landscape

Copy” .

Disaster recovery:

A disaster recovery strategy that is based on shipping a set of backups to an offsite location (PTAM

method), can benefit from consistent landscape backups because restoring such a backup would

automatically provide a consistent DR environment to restart operations without having to worry about

consistency between the systems.

Using backups for DR was discussed in Disaster Recovery - Basic Approaches using B&R .

Building a forensic environment for analysis and resolution of logical errors:

If logical errors in a system landscape should require an analysis of a previous state of the systems, aconsistent landscape backup could allow building up a forensic system landscape on some alternate

hardware. Problem analysis could then benefit from a consistent view onto the data provided across all

systems, reflecting a former point in time before the error landed up onto the environment.









© 2010 SAP AG page 50/64

Consistent fall-back point:

During upgrade or data migration projects or during other massive changes on the system landscape,consistent landscape backups could be exploited as an easy means to fall back the complete

landscape to a former state in case the changes turned out to be unsuccessful.

Golden copy:

Prior to go-live of a production landscape, it may be desired to repeat cutover testing or data loadmultiple times. In this case, a “golden copy” of the landscape could be created as a consistent

landscape backup after the installation and configuration has finished and before the first test run is

executed. After each test run, the landscape can be reset to the consistent landscape backup and the

tests can be repeated. Similarly, golden copies can be used in any other test environment where

repeatable tests are required.

Handling incomplete recovery in case of special consistency requirements:

In some special situations, a customer might prefer restoring a consistent landscape backup instead of

performing a single-system incomplete recovery which needed to be followed by a business recovery

(as depicted in figure 2). If this demand should arise during the creation of the B&R concept, thecustomer might want to create consistent landscape backups on a regular basis.

6.2 Creating Consistent Landscape Backups

A consistent landscape backup can be achieved on the communication layer (application level), on the

database level or on the storage level as indicated in figure 5.

System 1 System 2

Communication

Storage Storage

SAP Instance

DatabaseInstance

SAP Instance

DatabaseInstance

Figure 5: Schematic Representation of a 2-system Landscape




© 2010 SAP AG page 51/64

This section explains the different possibilities for each level. All options on application level require downtime

of at least some systems of the landscape while creating the consistent landscape backup . In contrast to that,

all options on database or storage level focus on ways of creating consistent landscape backups online,

without downtime of the involved systems.

Another way to achieve a consistent landscape backup is to install all components of the landscape in one

single database, but this method is not recommended for productive systems.

In general, it is not possible to perform a fully consistent restore of two or more systems by

performing a point-in-time recovery for all systems to the same point in time (unless all systems were

stopped at that specific point). It cannot be guaranteed that this image will be consistent because

there may be differences in the local machine time on the different systems. Even when using a timesynchronization protocol, there is no final guarantee that the systems are synchronized up to the

latest committed transactions. This even applies if the systems are running in different partitions on

the same server because each partition maintains its own internal time. But on the other hand, using

time synchronization, the number of possible inconsistencies can be expected to be very low.

Detailed background information about communication and consistency in an SAP system landscape

can be found in the document “Data Consistency with mySAP System Landscape Copy” .

Note: Customer landscapes often include external components not from SAP. Such components

could also be included in a consistent landscape backup using the concepts introduced below.

This section focusses on SAP systems with an underlying database. If you want to use a consistent

landscape backup to restore a complete system landscape, you might also have to consider other

components with data in a filesystem (see B&R for Specific Components of an SAP System

Landscape ) and make sure that a corresponding state of that data is also available for the restore (for

example by creating an offline backup of such components together with the consistent landscape

backup of the databases).

6.2.1 Application LevelOn application level, data consistency can be achieved by stopping the communication between all involved

components. New messages will have to be queued and can thus not cause inconsistencies when creating a

backup. Unfortunately there is no easy way of completely pausing communication between SAP systems. So

the only way to stop communication between the components is to stop the application (the SAP instances).

This does not necessarily include stopping the databases, which can stay online. There are several variations

of this approach, which have different effects on downtime in the landscape:

Stop of all applications during backup

Stop of all but one application during backupStop of all but one application during log backup







© 2010 SAP AG page 52/64

6.2.1.1 Stop of All Applications During Backup

The easiest method from a handling point of view is to stop all involved components and take an offline

backup. To achieve consistency from an application point of view it is not necessary to stop the underlying

databases as indicated in figure 6. By keeping the databases up and performing an online backup, a

performance gain is achieved because the database buffers are kept in place. So from an application point-

of-view this will be an offline backup while from a database point-of-view it can be an online backup.

The business downtime involved with this approach can be minimized by using a split-mirror or snapshot

technology for the backup. With this technology the downtime can be restricted to just the time it takes to

create the mirror or snapshot and can lie in the range of minutes.

System 1 System 2

RFC

Storage Storage

SAP Instance

DatabaseInstance

SAP Instance

DatabaseInstance

Figure 6: Stop All Applications for the Backup

Advantages Disadvantages

- Easy to implement and handle- Database buffers are preserved - Downtime for all involved systems needed (no7x24 operation possible). Downtime depends onbackup method being used.

- Performance loss due to buffer resets

6.2.1.2 Stop of All-But-One Application During Backup

To achieve application data consistency, it is not necessary to stop all SAP systems. Communication is also

completely interrupted if only one system (the most important application) stays up as indicated in figure 7.

There is no incoming communication from the stopped applications, outgoing communication will be buffered

in the database tables of the running application and will be processed later on.




© 2010 SAP AG page 53/64

This solution might be considered if the system that will stay online mostly does asynchronous

communication. If there is a lot of synchronous communication that relies on the availability of the other

systems this alternative will not be applicable because there will be lots of communication errors.

System 1 System 2

Storage Storage

SAP Instance

DatabaseInstance

SAP Instance

DatabaseInstance

Figure 7: Stop All-But-One Application

Advantages Disadvantages- All databases can stay online; buffer contents of

databases are preserved- Most important application is running (allowing

7x24 operation)- Easy to implement and handle

- Not suitable for business scenarios requiring 7x24operation on multiple systems

On running system:- Administrator activity needed to check RFC statusOn stopped systems:- Downtime depends on backup method being

used.- Performance loss due to buffer reset

6.2.1.3 Stop of All-But-One Application During Log Backup

To reduce the downtime of the involved systems with the two procedures from above (if there is no split-

mirror or snapshot technology available), the process can be modified as follows so the systems must only be

stopped for a short period to get a consistent state across the logfiles (see figure 8):

1. Start an online backup on all involved systems. The backups should be started in a way that the ends of

the backups are close to each other.

2. Once all systems have finished their online backups, stop either all or all-but-one application.




© 2010 SAP AG page 54/64

3. Take a backup of the logs, which have been filled since the online backup. If one system stays online,

perform a logswitch on this system while the others are shut down. Backup the logs up to the logswitch.

The consistent landscape backup then consists of the tapes of the online backups plus the log backups. The

application data is consistent once the applications are stopped (step 2) because at this point all

communication between the systems is stopped as well.

System 1

System 2

Systemssynchronized

Backuprunning

System 3

Shutdown n-1 systemsand start log backup

Online Backup

System 1

finished

System 2finished

Online Backup

System 3finished

Online Backup

Log Backup

Log Backup

Log Backup

Consistentbackup

Delta Data

Delta Data

Delta Data

Figure 8: Stop of All-But-One Application During Log Backup

Advantages Disadvantages- All databases can stay online; buffer contents of

databases are preserved- Most important application is running (allowing

7x24 operation)- Reduced downtime on all other applications- Less queue contents to be processed after startup

- Not suitable for business scenarios requiring 7x24operation on multiple systems

On running system:- Administrator activity needed to check RFC statusOn stopped systems:- Performance loss due to buffer reset- Procedure is more difficult to handle and thus

more liable to errors




© 2010 SAP AG page 55/64

6.2.2 Database Level - Coordinated Suspend

Using the " suspend write ” capability of the database management systems, it is possible to create a point of

consistency without stopping the applications. This feature has already been used for split-mirror backups of

single systems for a long time.

By issuing a suspend write command to the database, all modifications are blocked. This causes the

application to “freeze” for a short time, but it is not aborted. From the moment on that a database is in

suspend write mode, all outgoing and incoming communication between the systems is paused as well,

because communication always requires write accesses to the database. If there is communication which

only needs read access to the remote database, this cannot cause inconsistencies because no data is

modified on the remote database.

Suspending the databases of all systems that shall be included in the consistent landscape backup , a point ofconsistency is created across all systems. Now a fast copy of the storage volumes can be performed to

create a consistent image of the landscape as indicated in figure 9 and 10.

A fast copy of the storage volumes can be achieved with different technologies like split-mirror

(creating a full physical copy of the storage volumes) or snapshot (creating only a logical copy of the

data at a specific point in time). Both types of technologies are offered by storage systems or by

logical volume managers.

Figure 9: Coordinated Suspend on Database Level




© 2010 SAP AG page 56/64

The coordinated suspend procedure basically consists of the following steps as also depicted in figure 10:

1. Perform re-synchronization of the mirrors on all involved systems (if applicable, depending on the copytechnology being used)

2. Issue the “ suspend write ” command on all involved databases and wait until all databases are suspended

3. Split mirrors or create snapshots while all database systems are in suspend mode

4. Resume write activity on all databases

The mirrors or snapshots will now contain a consistent image of the landscape. This image could be used to

start up all databases. At startup, the databases will perform regular crash recovery, recovering all committed

data and rolling back all uncommitted data. Because all communication between the systems was suspended

when the image was created, the landscape will be brought up to a consistent state. It is important to notethat there will usually be pending messages waiting to be processed in the systems. Processing these

messages will complete application data consistency between the systems.

Note that consistency of the landscape will be destroyed if you roll forward any of the systems by

applying additional logs that were generated after creating the consistent landscape image .

Note that you should not start up clones of the systems on the mirror without performing the post-

processing activities of a regular system copy!

Figure 10: Coordinated Suspend – Process Flow

S stem 1

System 2

System 1suspended

Re-syncfinished

Fast disk copy

Startsuspend

System 2suspended

ResumeSystems

All systemssuspended

Split

Fast disk copy

Start re-sync(if applicable)




© 2010 SAP AG page 57/64

Advantages Disadvantages

- No downtime- Only a short time needed for freezing the systems- All components stay online; buffer contents is

preserved

- Relatively high implementation effort forcoordinating the operations across differentsystems

- The process gets more complicated the moresystems are involved

The following table lists the commands which implement the "suspend write" feature on the different database

systems:

DB system suspend WRITE resume WRITE

DB2 UDB OS/390 set log suspend set log resumeDB2 UDB open set write suspend for database set write resume for database

INFORMIX onmode –c block onmode –c unblock

ORACLE alter database… begin backupalter system suspend

alter system resumealter database… end backup

SQL Server via Virtual Device Interface(dbcc freeze_io (<db>) may notbe used for forward recovery)

via Virtual Device Interface(dbcc thaw_io (<db>) may notbe used for forward recovery)

MaxDB / liveCache dbmcli: util_execute suspendlogwriter

dbmcli: util_execute resumelogwriter

In SAP SCM, the suspend should always be performed on the liveCache first with a little delay before

the APO database is suspended (due to the special commit protocol used between SAP APO and

SAP liveCache).

6.2.3 Storage Level - Consistency TechnologyConsistency technology offers a very convenient way of creating a consistent image of a group of storage

volumes either within a single storage system or even across multiple storage systems. Consistency

technology is currently available for different storage systems but might in the future also be offered by logical

volume managers. Features like consistent split or consistency groups allow grouping arbitrary storage

volumes into a consistency group. A split or snapshot command issued to the consistency group can create a

consistent image of all involved volumes. Because the storage system takes care that the split is performed

atomically for all included volumes, this feature allows creating a consistent image of multiple application

systems by combining all their volumes in a common consistency group as indicated in figure 11. There is no

need to stop the applications or to suspend the databases.








© 2010 SAP AG page 60/64

7 Tasks for Creating a B&R Concept

This following overview briefly summarizes the required tasks for creating a comprehensive B&R concept fora system landscape in a kind of checklist and provides references to the sections of this document and to

further sources of information.

# Task Section Further information

(see 8.2)

1. Identify components of the landscape by analyzing

business processes and by looking up SAP Solution

Manager and SAP System Landscape Directory

3.1 [Business Continuity Mgmt]

2. Determine service levels (recovery requirements: RTO,

RPO) for all components (based on the requirements

of the business processes relying on the component)

2.3 [Business Continuity Mgmt]

3. Determine available B&R capabilities (already existing

technologies and tools for B&R)

4. Determine overall approach for DR (is this to be

handled by B&R processes?)

4 [Business Continuity Mgmt]

For all components:

5. Determine type of components 3.2, 3.3, 3.4

6. Determine type of application data (if applicable) 3.2, 3.3, 3.4

7. Determine possibly backup-relevant objects

(application data, software)

3, 3.2, 3.3 , 3.4

8. Determine recovery strategy for all objects (restore

from a backup or other method like reinstallation)

3.2, 3.3, 3.4

9. Determine backup strategy and backup tools for

backup-relevant objects

3.2, 3.3, 3.4,

3.5

10. Check if backup strategy is able to meet the recovery

requirements (RTO, RPO)

2.3, 3.2.4 ,

3.2.1.3

11. Determine additional measures to safeguard the

systems and the recovery

3.2.1.1, 3.2.1.2

12. Analyze and document possible follow-up activitiesafter a restore

3.2.4.3, 3.2.4.6












© 2010 SAP AG page 62/64

8 Appendix

8.1 Glossary

B&R: Backup and Restore

BCM: Business Continuity Management

Business continuity management : Business Continuity Management (BCM) is concerned with managing

risks to ensure that at all times an organization can continue operating to, at least, a pre-determined minimum

level (according to ITIL, the IT Infrastructure Library).

Business recovery : Business recovery describes the resolution of data inconsistencies between federated

systems on application level.Consistent landscape backup : A common backup of a specific number of systems which allows restoring

these systems (the landscape) to a logically consistent state considering the application data stored across

these systems.

Data inconsistency : An error affecting the integrity of application data between different systems. Data is

inconsistent if the states of an object that was exchanged between two (or more) systems do not match, for

example because they are not the same in both systems. While temporary inconsistencies (also called

differences) are normal when the data exchange is still in progress, real inconsistencies persist after all

related communication was completely processed. Real inconsistencies will cause problems in a company’s

business processes.

Disaster recovery : A situation where a complete system (or data center) has to be recovered or rebuilt after

some catastrophic event.

DR: Disaster Recovery

Federated system landscape / federated databases : A set of systems or databases (a system landscape)

providing services to one or more business processes. The term ‘federation’ is related to the consistency

requirements of the business data that is distributed over the different systems (respectively the underlying

databases).

Incomplete recovery : A system or component is recovered to a former point in time, leading to data loss in

the system.

Logical error : An error affecting the integrity of the application data within a single system. In contrast to data

inconsistencies, which describe integrity issues with application data due to the exchange between different

systems, we define logical errors as being local to a single system. Logical errors will cause problems in a

company’s business processes.

Point-in-time recovery : A database system is recovered to a specific point in time. This can be the most

current or any former point in time. In this document, point-in-time recovery is used as a synonym for

incomplete recovery (point-in-time recovery to the most current point is called complete recovery).




© 2010 SAP AG page 63/64

Recovery Point Objective : Maximum allowed data loss after a system failure (the point where the system

has to be recovered to). The Recovery Point Objective can be defined for a specific error scenario.

Recovery Time Objective : Maximum allowed recovery time after system failure. The Recovery Time

Objective can be defined for a specific error scenario.

RPO : Recovery Point Objective

RTO : Recovery Time Objective

SAP NW AS : SAP NetWeaver Application Server

8.2 Further Information and References

You can find further information related to the topics covered in this Best Practice in the following places:

Technical Operations for SAP NetWeaver 7.0 , or respectively for other SAP releases,

Online Documentation at http://help.sap.com

Business Continuity Management for SAP System Landscapes ,

Best Practice at http://service.sap.com/runsap

Emergency Handling for Recovery of SAP System Landscapes ,


SAP Standards for Data Integrity and Transactional Consistency ,

Whitepaper at http://service.sap.com/supportstandards .

Data Consistency Check for Logistics ,


Data Consistency with mySAP System Landscape Copy -- Why Consistency Technology is needed –

at http://service.sap.com/atg Federated backup

Further information on storage-technologies, split-mirror backups and consistency technologies:

http://service.sap.com/atg

SAP System landscape copy ,


8.3 FeedbackTo improve the quality of this Best Practice, we are depending on your feedback. Please send any feedbackconcerning errors or possible optimizations of this document to [email protected] , Subject “Backup and Restore -Feedback” . Note that we do not provide support for problems at this email address! Use SAP’scustomer support systems to get help with any issues with a B&R procedure.You can also click Feedback to send any comments on Best Practice documents.

http://help.sap.com/saphelp_nw70ehp1/helpdata/en/ec/9e1802dd600947abafd2248518697f/frameset.htm


http://help.sap.com/



http://service.sap.com/runsap






http://service.sap.com/supportstandards












mailto:[email protected]?subject=Backup%20and%20Restore%20-%20Feedback

https://websmp202.sap-ag.de/~form/alko/002006825000100142132000E/011000358700011083472001E?LOCURL=/~form/sapnet%3F&LOCURL=_HIER_KEY%3D501100035870000009816%26&LOCURL=_HIER_KEY%3D601100035870000116387%26&LOCURL=_HIER_KEY%3D601100035870000091575%26&LOCURL=_HIER_KEY%3D601100035870000066843%26&

mailto:[email protected]?subject=Backup%20and%20Restore%20-%20Feedback














http://help.sap.com/





© Copyright 2010 SAP AG. All Rights ReservedNo part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG.

The information contained herein may be changed without prior notice.Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.Microsoft, Windows, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation.IBM, DB2, DB2 Universal Database, OS/2, Parallel Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390, OS/400, iSeries, pSeries, xSeries,zSeries, z/OS, AFP, Intelligent Miner, WebSphere, Netfinity, Tivoli, and Informix are trademarks or registered trademarks of IBMCorporation.Oracle is a registered trademark of Oracle Corporation.UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of CitrixSystems, Inc.HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C®, World Wide Web Consortium, MassachusettsInstitute of Technology.Java is a registered trademark of Sun Microsystems, Inc.JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented byNetscape.MaxDB is a trademark of MySQL AB, Sweden.

SAP, R/3, SAP, SAP.com, xApps, xApp, SAP NetWeaver, and other SAP products and services mentioned herein as well as theirrespective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. Allother product and service names mentioned are the trademarks of their respective companies. Data contained in this document servesinformational purposes only. National product specifications may vary.

The information in this document is proprietary to SAP. No part of this document may be reproduced, copied, or transmitted in any formor for any purpose without the express prior written permission of SAP AG.This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This documentcontains only intended strategies, developments, and functionalities of the SAP® product and is not intended to be binding upon SAP toany particular course of business, product strategy, and/or development. Please note that this document is subject to change and maybe changed by SAP at any time without notice.SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of theinformation, text, graphics, links, or other items contained within this material. This document is provided without a warranty of any kind,either express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, or non-infringement.SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that

may result from the use of these materials. This limitation shall not apply in cases of intent or gross negligence.The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you mayaccess through the use of hot links contained in these materials and does not endorse your use of third-party Web pages nor provide anywarranty whatsoever relating to third-party Web pages.

Documents

Backup&Restore of SAP landscapes