1

06 November 2007 • 14:00 – 15:00

Platform: Linux, Unix, Windows

DB2 HADR – Case Study of Implementation

Jessica RockwoodIBM Toronto Lab

Session: D06

An in-depth look at using DB2 High Availability Disaster Recovery (HADR) in a large enterprise. This presentation will review the HADR technology and walk through the end-to-end process of implementing HADR and automation technology to manage failovers, including: architecture, setup and configuration, testing, and monitoring.

2

2

Session Objectives• A thorough understanding of the capabilities of High

Availability Disaster Recovery (HADR) • Impart my lessons learned in implementing HADR solutions

• How to build a plan for deploying HADR successfully in an enterprise environment

• An understanding of automation software and how it interacts with HADR

• How to build a test plan to ensure a complete availability solution before production deployment

• How to monitor and administer HADR databases (including “My Pack of Procedures”)

At the end of this presentation, the audience should be able to implement HADR in their environment, understanding how to build the project plan, assess risks and benefits, implement and thoroughly test the solution, and build monitoring capabilities.

3

3

Goals of HADR• Ultra-fast failover• Easy administration• Handling of site failures • Negligible impact on performance• Configurable degree of consistency • Protection against errant transactions• Software upgrades without interruption• Very easy integration with HA-software • Eventually, no need for HA-software at all• Transparent failover and failback for applications

(combined with “client re-route”)

Limitations/restrictions:No CONNECT access to standby databaseNot supported with Data Partitioning Feature (DPF), raw I/O for transaction logs, or redirected restorePrimary and standby databases must have the same name, the same operating system and DB2 levels and the same bit-ness (32-bit vs 64-bit)Rolling upgrades refer to Fix Pack upgrades onlyNon-logged operations are not replicated:

Updates to DB/DBM cfg and registryLOBs > 1GB or LOBs marked as NOT LOGGEDNOT LOGGED INITIALLY operations

Log archiving and self-tuning memory manager (STMM) can only be done on the primary database

4

4

Basic principles of HADR• Two active machines

• Primary• Processes transactions• Ships log entries to the other machine

• Standby (or Secondary)• Cloned from the primary• Receives and stores log entries from the primary• Re-applies the transaction

• If the primary fails, the standby can take over the transactional workload• The standby becomes the new primary

• If the failed machine becomes available again, it can try to be resynchronized• The old primary becomes the new standby

5

5

Scope of activity• HADR replication takes place at the database level

Database E Database E

HADR HADRTCP/IP

Primary Standby

Database A

HADR HADRTCP/IP

Standby Primary

Database A

Database B

Database C

Database D

6

6

HADR implementation details

LogsOld

Logs

Log Writer Log Reader

Tables

IndexesLogs

Old

Logs

Log Reader

Tables

Indexes

TCPIP

DB2 Engine DB2 EnginePRIMARY SERVER STANDBY SERVER

Log Writer

HADR HADR Shredder Replay Master

Redo SlavesRedo SlavesRedo SlavesReplay Slaves

Log Pages

Log Pages

Log Pages

Log RecordsLog Pages

Log

Records

Client ReroutePrimary Connection

7

7

HADR State Flow

HADR/DB2 Startup

Local catch-up

Remote catch-up Pending

Remote catch-up

Peer State

Connect

Con

nect

ion

lost

These states are as seen from the standby; on the primary only remote catch-up and peer state are valid states. All others are represented as ‘Disconnected”from the perspective of the primary.

8

8

HADR synchronization modes

Logs

Log writer

Logs

TCP/IP

Commit Succeeded

Synchronous

Near

- Syn

chro

nousAsynchronous

receive()send()

HADR HADR

Log writer

Near-Sync sends back the acknowledgement once data is received in memory.

9

9

Planning• Three components to examine:

• Client• Network• Server

• Goal is to identify the expected behavior in the case of a component failure

TCPIP

TSA detects failure of Site A within 5 seconds and invokes forced HADR takeover on Site B

e.g. AIX server failure

Site BSite A

Expected resultFailure description

Failure location

Location operating as primary

10

10

Planning stage - Client• Type of client?

• Uses DB2 Client or DB2 application driver (Java, CLI)?

• Type of workload?• Availability requirements?• Hours of operation?

TCPIP

Type of client:-When using an application driver, need additional attention to how to handle client reroute. When you specify the alternate client on the server, the Java or CLI client will pick up the alternate server information on first connect. If you need to support the clients being able to reroute without having successfully connected to the “primary” site, then need to harden the information in the client. For CLI, it’s an option on the definition of the data source. For JCC driver, need to implement the Type of workload:-Need to understand whether workload is constant or will have spikes-With spikes of utilization, it may affect the ability of the standby to keep up and could influence the choice of SYNC mode. With spikes in workload, likely you should consider NEARSYNC (or ASYNC if only using the solution for disaster recovery and not HA)Availability requirements:-What’s the priority on the system? Is it availability or ensuring no data loss…each have a tradeoff and can influence the choice between sync modes as well as options like HADR_TIMEOUTHours of operations:-Need to understand when and if there are down-times on the system-If there’s a period of little to no activity, then could do off-line maintenance activities or those with high impact to the standby (like offline reorg) during these periods and even consider deactivating the standby, doing maintenance,

d th f hi th t db ith b k i f th i

11

11

Planning stage - Network• What is the network topology?

• Same network used between clients & server as between HADR primary & standby?

• Bandwidth?• Must have bandwidth greater than the

database log generation rate• Redundancy of network adapters?

• Single network adapter is a possible point of failure

• Virtual IP?

TCPIP

Network topology:-It is recommended to isolate HADR communication on a network separate from the rest of the application communications. This ensures that spikes in application workload or other activity in the environment will not affect the HADR communication.-If the network is going to be shared, then may need to ensure optimal tuning of the network to handle all the data flow-Note that HADR requests tcp_nodelay for its connection for optimal performance; if ignored by the OS, it can negatively affect HADR performance-Increasing LOGBUFSZ can increase the amount of log flushed to the standby in each request… but more likely to need to increase DB2_HADR_BUF_SZ registry variable (to more than the default of 2xLOGBUFSZ) in order to handle peaks of log data flushed to standbyVirtual IP:- Using a virtual IP can simplify management of the HADR pair in cluster management software… in the event of a failure, the IP ownership moves from primary to standby and so no clients will be able to connect to the primary box (which can lead to split-brain if the primary was not shut down properly)- With a virtual IP, can still use client reroute by defining the same IP for the alternate client server as for the initial catalog entry

12

12

Planning stage - Server• DB2 levels (recommend minimum V8.1 FP12 or

V9 FP2)• Storage

• robustness of database storage?• location of archived logs?

• Server• operating system? release levels?• automation software?

TCPIP

Storage:-Storage is a possible point of failure on both the primary and the standby…disk mirroring or some other sort of HA strategy for the disks can avoid requiring a failover due to disk failure; there is no interruption of service if a failover is not required, so always want to look to repair in place when possible-If archived logs are on external storage or something like TSM, then both primary and standby need to be able to access the location; also, delays in retrieving logs can result in significant delays in catch-up by the standby so you may want to retrieve archived logs to the standby before starting HADR to avoid that network overhead (but bear in mind that you need to manage the deletion of those retrieved logs)Server:-Look to run the operating system levels prescribed by IBM and ensure that all OS patches are applied in a timely fashion-Tivoli System Automation for Multiplatforms is available for Linux and AIX. HACMP is also available for AIX.b

13

13

Case A: Bright Yellow Truck Company

•p5-575 AIX 5.2 64-bit ML6 with DB2 Viper 2•Tivoli Storage Manager (TSM) 5.3 is used to archive logs•HACMP 5.3 is the incumbent automation software

Server

Single public GB ethernet for all network traffic, heartbeat RSA connection between primary and disaster site

Network

24x7 proprietary application using DB2 clientClient

14

14

Bright Yellow Truck – Planning Analysis

… and so on, including network failure, DB2 failure, filesystem damage/corruption, TSM server failure

Local disk is mirrored, no impact to DB2; storage team to replace failed disk

Disk failureSite ASite A

No automatic action; need to repair server on site B, restart, and regain peer state

AIX server failure

Site BSite A

HACMP invokes rc.hadr.start script on Site B to takeover on standby database; clients reconnect to Site B primary through client reroute

AIX server failure

Site ASite A

Expected ResultFailure description

Failure location

Primary location

15

15

Case B: Little Blue Box Jewelers

•x3650 Linux SLES9 SP3 with DB2 9•Logs are archived to shared disk•Tivoli System Automation for Multiplatforms 2.1 (TSA) for automation

Server

Public GB ethernet for client-server traffic, private GB ethernet for HADR connection between primary and disaster site

Network

Java application running on WebSphere Application Server (WAS) using the JCC Type-4 driverApplication must be operational 8am-5pm weekdays and Saturday

Client

16

16

Decide on HADR configuration• Two decisions to make about HADR connection:

• hadr_timeout • Time (in seconds) that HADR waits before considering a

communication attempt as failed• Heartbeat interval will be ¼ of hadr_timeout or 30 seconds,

whichever is shorter• hadr_syncmode

• SYNC, NEARSYNC, or ASYNC• For disaster recovery, most choose between SYNC and

NEARSYNC -> performance testing is valuable to quantify the difference between the two

Note that, for maximal availability, as soon as the connection is closed (because the other end closed the connection, a network error is detected, or timeout is reached), the primary drops outof peer state to avoid blocking transactions.A recommended timeout value depends on the capacity of the data server and of the HADR network connection. If either are busy, then it can result in a delay in response between the HADR pair and that will start the countdown to the HADR_TIMEOUT value.In most enterprise implementations, an HADR_TIMEOUT between 30 and 60 seconds provides the best balance of a short timeout without risking falling out of peer due to too short a timeout. An HADR_TIMEOUT less than 30 seconds should be thoroughly tested to ensure the primary does not drop out of peer state as part of regular processing.

17

17

Big Yellow Truck Decisions• After testing SYNC and NEARSYNC, no significant performance

impact was detected between the two so set hadr_syncmode=SYNC

• Highest priority at Big Yellow Truck was data integrity – no transactions could be lost without a ripple effect through the organization• Risk even in SYNC mode; if the TCP/IP socket closed on the

primary, the primary could drop out of Peer mode immediately and if the primary failed and a forced takeover occurred before the standby reconnected and regained peer state there was the possibility of data lossChose to implement new Viper 2 feature: HADR peer window

18

18

HADR Peer Window• HADR_PEER_WINDOW can be used to cause the HADR pair

(primary and standby databases) to behave as though in peer state even if the primary loses connection with the standby database• Once connection to standby is lost, primary waits the number

of seconds specified by HADR_PEER_WINDOW before dropping out of Disconnected Peer state and continuing to process transactions

• Peer window end (meaning the time at which the primary is no longer guaranteed to be in peer state) is sent to the standby through the heartbeat - timestamp can be checked by automation software to determine whether or not to do forced takeover

• If standby does not reconnect with primary by the end of the peer window, the primary drops from Disconnected Peer state to Remote Catchup Pending

Note: important to synchronize clocks between primary and standby for effective usage of End of peer window

19

19

Setting HADR_PEER_WINDOW• DB2_HADR_PEER_WINDOW should be set to a value that fits the following

equation:

DB2_HADR_PEER_WINDOW >= response_time + fudge_factor + heartbeat_interval

response_time = estimated time for automation software to detect failure and invoke HADR takeover

fudge_factor = 5 sec, small delta time delay because detection/response is not instantaneous

heartbeat_interval = MIN(HADR_TIMEOUT/4, DB2_HADR_PEER_WINDOW/4,30 sec)

• Big Yellow Trucks determined their response time as a minimum of 45secs• HACMP detection of failure: 5 seconds• HACMP processing time until HADR takeover invoked: up to 40

seconds

20

20

HADR State FlowHADR/DB2 Startup

Local catch-up

Remote catch-up Pending

Remote catch-up

Peer State

Connect

Con

nect

ion

re-e

stab

lishe

d

Con

nect

ion

lost

Disconnected Peer State

Connection lost

Peer window expires

21

21

Implementing the HADR solution• Prepare and clone the primary database and

configure for HADR• Create the standby database from the clone• Configure the standby database for HADR• Configure client reroute • Start-up HADR• Set-up automation process

22

22

Prepare and clone the primary DB• Ensure archive logging is enabled on the primary database

Archive log location must be accessible by both primary and standby database

• Sample database configuration for Case A: LOGFILSIZ 16380LOGSECOND 12LOGARCHMETH1 TSM:DB2

• Sample database configuration for Case B:LOGFILSIZ 8192LOGSECOND 2LOGARCHMETH1 DISK:/shared/db2inst1/logarchives

• Take backup or split mirror copy of primary database and move the backup image to the standby location

23

23

Code snippet - Configure primary for HADRexport $dbname=TRUCKERSexport $pri=yellow1.bigtruck.comexport $sec=yellow2.bigtruck.comexport $svc=7000

db2 connect to $dbnamedb2 update db cfg for $dbname using LOGFILSIZ 16380db2 update db cfg for $dbname using LOGSECOND 12db2 update db cfg for $dbname using LOGARCHMETH1 TSM:DB2db2 update db cfg for $dbname using INDEXREC RESTARTdb2 update db cfg for $dbname using LOGINDEXBUILD ONdb2 update db cfg for $dbname using HADR_TIMEOUT 30db2 update db cfg for $dbname using HADR_PEER_WINDOW 90db2 update db cfg for $dbname using HADR_LOCAL_HOST $pridb2 update db cfg for $dbname using HADR_LOCAL_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_HOST $secdb2 update db cfg for $dbname using HADR_REMOTE_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_INST sitebdb2 update db cfg for $dbname using HADR_SYNCMODE SYNCdb2 deactivate db $dbnamedb2 activate db $dbname

Configuration for Case B would be:export $dbname=DIAMONDexport $pri=bluebox.jeweler.comexport $sec=bluebox2.jeweler.comexport $svc=7000

db2 connect to $dbnamedb2 update db cfg for $dbname using LOGFILSIZ 8192db2 update db cfg for $dbname using LOGSECOND 2db2 update db cfg for $dbname using LOGARCHMETH1 DISK:/shared/mydb/archivelogsdb2 update db cfg for $dbname using INDEXREC RESTARTdb2 update db cfg for $dbname using LOGINDEXBUILD ONdb2 update db cfg for $dbname using HADR_TIMEOUT 30db2 update db cfg for $dbname using HADR_LOCAL_HOST $pridb2 update db cfg for $dbname using HADR_LOCAL_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_HOST $secdb2 update db cfg for $dbname using HADR_REMOTE_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_INST sitebdb2 update db cfg for $dbname using HADR_SYNCMODE SYNCdb2 deactivate db $dbnamedb2 activate db $dbname

24

24

Create the standby DB and configure HADR• Restore the backup image or db2inidb the split mirror copy

database is left in roll forward pending state

db2 update db cfg for $dbname using LOGFILSIZ 16380db2 update db cfg for $dbname using LOGSECOND 12db2 update db cfg for $dbname using LOGARCHMETH1 TSM:DB2db2 update db cfg for $dbname using INDEXREC RESTARTdb2 update db cfg for $dbname using LOGINDEXBUILD ONdb2 update db cfg for $dbname using HADR_TIMEOUT 30db2 update db cfg for $dbname using HADR_PEER_WINDOW 90db2 update db cfg for $dbname using HADR_LOCAL_HOST $secdb2 update db cfg for $dbname using HADR_LOCAL_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_HOST $pridb2 update db cfg for $dbname using HADR_REMOTE_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_INST siteadb2 update db cfg for $dbname using HADR_SYNCMODE SYNCdb2 deactivate db $dbname

Strict symmetry of table space and container configuration is required on the standby. Name, path, size all must match. Relative container paths are allowed, and the full path may differ in this case.

If strict symmetry does not exist, HADR may fail to replicate a container operation on the standby. In such a case replication of the affected table space stops and the table space will be left in "roll forward in progress"

25

25

Configure automatic client reroute• On both the primary and standby database servers issue:

db2 update alternate server for db $dbname using hostname <host> port <port_num>

• Note: Alternate server is flowed to the client and cached on first connect; without a connection, the client will not know where to re-route• To protect against this on a DB2 client, the same

command can be run on the client to ‘harden’ the alternate server

• On a JCC-Type 4 client, use the clientRerouteServerListJNDIName DataSource property to persist alternate server information across JVMs and to ‘harden’ the alternate server

26

26

Start-up HADRSTANDBY

db2 start hadr on db $dbname as standby

db2 get snapshot for db on $dbname |grep –A 15 “HADR Status”

ORdb2 select * from

sysibmadm.snaphadr

PRIMARY

db2 start hadr on db $dbname as primary

db2 get snapshot for db on $dbname |grep –A 15 “HADR Status”

Start local catch-up

Validate connection

between primary & standby and start remote

catch-up

Upon starting the standby, the Local Catchup phase can begin:-Replay locally available log files (if any)-Replay log files that may be retrieved at the primary and shipped to the standby

Once the primary is started (which is an asynchronous operation), remote catchup can begin:-Replay log pages from the primary's archived logs-Replay log pages from the primary's active log to the standby until the standby is caught up to the tail of the log.-Replay log pages from the primary's in-memory log buffer to the standby whenever the primary flushes those pages to disk

At the conclusion of remote catchup, the standby is now in Peer state.

27

27

Start-up on standby – Expected ResultsDB20000I The START HADR ON DATABASE command completed successfully.

HADR StatusRole = StandbyState = Remote catchup pendingSynchronization mode = SyncConnection status = Disconnected, 03/22/2007 09:50:12.423522Peer window end = Null (0)Peer window (seconds) = 65Heartbeats missed = 0Local host = bbird007Local service = HADR_test2Remote host = bbird006Remote service = HADR_test1Remote instance = db2hdatimeout(seconds) = 30Primary log position(file, page, LSN) = S0000115.LOG, 0, 0000000748858000Standby log position(file, page, LSN) = S0000114.LOG, 1, 000000074485D0BCLog gap running average(bytes) = 67088191

28

28

Start-up on primary – Expected ResultsDB20000I The START HADR ON DATABASE command completed successfully.

HADR StatusRole = PrimaryState = PeerSynchronization mode = SyncConnection status = Connected, 03/22/2007 09:50:50.622363Peer window end = 03/22/2007 09:51:56.000000 (1174571516)Peer window (seconds) = 65Heartbeats missed = 0Local host = bbird006Local service = HADR_test1Remote host = bbird007Remote service = HADR_test2Remote instance = db2hdatimeout(seconds) = 30Primary log position(file, page, LSN) = S0000115.LOG, 0, 0000000748858000Standby log position(file, page, LSN) = S0000114.LOG, 1, 000000074485D0BCLog gap running average(bytes) = 67088191

29

29

Configure automation software• For TSA, use DB2-provided scripts hadr_start.ksh,

hadr_stop.ksh, and hadr_monitor.ksh• Reference whitepaper: “Automating DB2 HADR Failover

on Linux using Tivoli System Automation on Multiplatforms”(ftp://ftp.software.ibm.com/software/data/pubs/papers/hadr_tsa.pdf)

• For HACMP, use DB2-provided scripts rc.hadr.start, rc.hadr.stop, rc.hadr.monitor• Reference whitepaper: “Automating IBM DB2 UDB

HADR with HACMP”(ftp://ftp.software.ibm.com/software/data/pubs/papers/hadr-hacmp.pdf)

30

30

Next steps• Yippee! HADR is now up and running…

… but now you need to test it and develop procedures for the HADRenvironment

• Testing matrix should cover all the failure scenarios outlined during the planning phase to validate expected results

• Measurements to capture:• Time to detect failure, time for failover, and time until new primary

is open for business (any open transactions at time of failure rolled back and database ready to process new transactions)

• Performance of workload• Server utilization statistics• Consistency check between primary and standby data

31

31

My Pack of Procedures• Database start-up

• STANDBY: db2start; db2 activate db $dbname• PRIMARY: db2start; db2 activate db $dbname• Verify success with snapshot on PRIMARY

• If standby not available, PRIMARY: db2start; db2 start hadr on db $dbname as primary by force

• Database shut-down• PRIMARY: db2 deactivate db $dbname; db2stop force• STANDBY: db2 deactivate db $dbname; db2stop

Recommended aliases:alias gethadrhost=’db2 get db cfg for $dbname |grep HADR_LOCAL_HOST’alias getdbrole=’db2 get db cfg for $dbname |grep “HADR database role”’alias hadrstate=’snaphadr | grep State’alias snaphadr=’db2 get snapshot for db on $dbname |grep –A 15 “HADR Status”’

32

32

My Pack of Procedures• Database role reversal

• Verify peer state on primary with snapshot• STANDBY: db2 takeover hadr on db $dbname

• Verify success with snapshot on new PRIMARY

Primary Standby

33

33

My Pack of Procedures• Manual database failover

• With peer window configuration, issue on STANDBY: db2 takeover hadr on db $dbname by force peer window only

• Without, issue on STANDBY: db2 takeover hadr on db $dbname

• If this fails, need to evaluate ‘safe-ness’ of doing the forced takeover – risk of split-brain or losing transactions

34

34

Is the primary server (and its disk) available?

YES, OK to run a planned

takeover

On STANDBY, check the HADR state of the database, use alias:

hadrstate

On PRIMARY, run db2pdlog –l <outfile> from the database directory (where SQLOGCTL.LFH exists)

No

UNPLANNED EVENT: Task flow for determining ‘safe-ness’ to failover

Yes

Risk of data loss with

forced HADR takeover

Any other state

Remote catchuppending

state

Peer state

Did the primary server fail while the databases were still in peer (by comparing timestamps from automation software (AS)

logs and db2diag.log on standby)?Get timestamp from previous step’s message.Get timestamp from automation log for when the primary server failed (or whatever event

caused the failover)

On STANDBY, check diaglog. Was db last in Peer

before changing to remote catchup pending (“HADR state set to S-

RemoteCatchupPending(was S-Peer)”)?

Yes

No

YesNo

Compare db2diag timestamp (DT) and AS log timestamp (AT). Is the

DT within a “safe” (duration of failure detection) timespan?

YesNo

In <outfile>, is hdrPrimaryLastState

= 0x10 (Peer)

Note that the values returned from the db2pdlog command for hdrLastPrimaryState are internal values. The possible values are:0x01 – Booting0x02 – Waiting to enter remote catchup0x04 – Remote catchup0x08 – Nearly peer0x10 – Peer

35

35

My Pack of Procedures• Failed primary reintegration

• On old PRIMARY (now new STANDBY): db2 start hadr on db $dbname as standby

• HADR and reintegration starts asynchronously so success must be monitored

• If reintegration is not possible, details will be logged in the db2diag.log as to why it failed to synchronize

36

36

My Pack of Procedures• Rolling fix pack upgrade

1. On site B, upgrade the database• Deactivate standby database and stop DB2• Upgrade software, run db2iupdt• Restart DB2 and activate standby database• Wait for HADR pair to regain Peer state

2. Switch primary/standby role• Issue “TAKEOVER HADR ON DB $DBNAME” on site B

3. On site A, upgrade the database• Deactivate new standby database and stop DB2• Connect to site B, run BIND, db2updv8, etc on the new primary to complete the

upgrade on site B• Upgrade software, run db2iupt• Restart DB2 and activate new standby database• Wait for HADR pair to regain Peer state which completes the upgrade on site A

4. Optionally, switch primary/standby role

37

37

My pack of procedures - Alerts• HADR state

• Monitor hadr_state element for ‘Peer’ on the primary• Not logged objects exist

• On the primary, check SYSCAT.COLUMNS for any LOGGED=‘N’select tabschema, tabname, colname from syscat.columns

where logged=’N’

• Not logged initially table operations• Scan the administration notification log on the primary for

ADM5530W which indicates that a table has committed activity that was not logged (either created NOT LOGGED INITIALLY or altered to activate NOT LOGGED)

38

38

My pack of procedures - Alerts• Non-replicated load

• Monitor the diagnostic log on the standby• Load copy cannot be accessed on the standby:

FUNCTION: DB2 UDB, database utilities, sqluMCCheckDevType, probe:10MESSAGE : Media controller -- invalid device path:

/db2fs/db2hda/HDA.4.db2hda.NODE0000.CATN0000.20060406045810.001

FUNCTION: DB2 UDB, recovery manager, sqlpRecDbRedo, probe:2129MESSAGE : Tablespace 7 in restore pending state.

• Non-recoverable loadFUNCTION: DB2 UDB, High Availability Disaster Recovery,

hdrTakeoverCheckTablespaces, probe:51010 MESSAGE : Warning: Tablespace is not in NORMAL state following takeover TSID:DATA #1 : Hexdump, 8 bytes0x0000007FBFFBA2B0 : 0700 0000 0000 000 ........

39

39

My pack of procedures - Alerts• Non-normal table space state

• If any load operation has an inaccessible load copy• If there’s a disk error on the standby• On the standby, check the State column in the

Tablespace Statistics report generated from db2pd -db <dbname> -tablespaces

• Remember, some non-zero states are okay

40

40

My pack of procedures• Identifying which role a system should be

• Worst-case scenario where maybe automation failed to takeover and you have two standby databases .. or maybe two primary databases

db2diag -gi "funcname:=hdr" -fmt "%{tsmonth}-%{tsday}-%{tshour}.%{tsmin}.%{tssec} %{process} %{changeevent}\t %{message}"

04-11-13.09.57 db2hadrp (HDA) 0 HADR state set to P-Peer (was P-NearlyPeer)04-11-13.25.08 db2hadrp (HDA) 0 Info: Primary has started non-forced takeover request.04-11-13.25.08 db2hadrp (HDA) 0 Info: Primary is switching to standby role....04-11-13.59.16 db2hadrs (HDA) 0 HADR state set to S-Peer (was S-NearlyPeer)04-11-14.16.36 db2hadrs (HDA) 0 Info: Standby has initiated a takeover.04-11-14.16.36 db2hadrs (HDA) 0 Info: Standby switching roles to primary.04-11-14.16.40 db2hadrp (HDA) 0 HADR state set to P-Peer (was S-Peer)04-11-14.16.41 db2hadrp (HDA) 0 Info: Standby has completed takeover (now primary)....04-27-15.11.26 db2agent (HDA) 0 Info: HADR Startup has begun.04-27-15.11.26 db2hadrp (HDA) 0 Info: Primary Started.04-27-15.24.39 db2hadrs (HDA) 0 HADR state set to None (was None)04-27-15.24.39 db2hadrs (HDA) 0 HADR state set to S-Boot (was None)

On Unix, when an HADR database is operating as a primary database, the process name in diagnostic log entries is “db2hadrp”. When operating as a standby database, the process name is “db2hadrs”. On Windows, the state description for each HADR state set message uses “P-“to indicate primary and “S-“ to indicate standby.

41

41

Trouble-shooting

• Congestion• Can result in delay of transaction commits on

the primary if in SYNC or NEARSYNC• Due to competition for network or log receiving

blocked at standby• If due to peaks in workload and see an increase

in log gap, consider increasing DB2_HADR_BUF_SIZE to 4x or 8x LOGBUFSZ (default setting is 2xLOGBUFSZ)

42

42

Administration considerations• Restore:

• Table space level restore is not allowed• Database level restore can reset HADR role to STANDARD

• Table and index reorganization:• Online reorganization logs at fine granularity so replicated

frequently; cannot restart an online reorg started on site A after failover to site B

• Offline reorganization can have a significant impact on HADR as a single log record could encompass many updates .. the standby repeats the offline reorg after the primary completes

• Not logged initially (NLI) operations• NLI operations are not replicated; may not be an issue if it’s for

temporary or staging operations, but should monitor this scenario

43

43

Administration considerations• Load operations:

• Must be run with COPY YES option to be replicated• All other options result in standby becoming inconsistent with

primary and no longer valid for failover• User DB2_LOAD_COPY_NO_OVERRIDE registry variable to

avoid non-copied loads• Not-logged large objects:

• LOBs greater than 1GB or those defined as NOT LOGGED will not be replicated to the standby

• If these objects are required in DB, then will need regular refreshes of standby from primary backup to keep in sync

• Configuration updates and file system changes:• Recommended to keep configuration synchronized between

primary and standby by applying updates to both

When loading data into tables, a LOAD ... COPY YES ... is run on the primary and the data is replicated on the standby if the standby can access the copy through the path or device specified in the load command.

If the load copy cannot be accessed on the standby, the table space in which the table is stored is marked bad on the standby and the standby will skip future log records regarding this table space. In this situation, the standby database has to be rebuilt from a backup image taken after the LOAD operation was issued on the primary.

A LOAD ... NONRECOVERABLE is run on the primary as usual. On the standby, the table will be marked bad and the standby will skip future log records regarding this table. If the table is not going to be dropped as part of the load processing, the table can be recovered by issuing a LOAD COPY YES REPLACE for the table on the primary, or the standby can be rebuilt from a backup image taken after the LOAD ... NONRECOVERABLE was issued.

If a LOAD ... COPY NO is run on the primary, it is converted to a LOAD NONRECOVERABLE by default. However, this conversion can be overridden to convert to a COPY YES operation by setting DB2_LOAD_COPY_NO_OVERRIDE=”COPY YES TO <shared directory path>”. When the registry variable is used, all LOAD … COPY NO operations will be converted to and behave according to the LOAD … COPY YES behavior outlined above.

44

44

Summary• HADR is an easy-to-use technology• Early planning will lead to reaping rewards during

implementation• Pack of procedures will keep things simple and

enable smooth operation of HADR

45

45

Jessica RockwoodIBM Toronto Lab

jessicae@ca.ibm.com

Session D06DB2 HADR – Case Study of Implementation