Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
1
May 21, 2008 • 10:00 a.m. – 11:00 a.m.Platform: DB2 for Linux, Unix, Windows
Jessica Escott-RockwoodIBM Toronto Lab
Session: D09
DB2 HADR – Case Study of Implementation
An in-depth look at using DB2 High Availability Disaster Recovery (HADR) in a large enterprise. This presentation will review the HADR technology and walk through the end-to-end process of implementing HADR and automation technology to manage failovers, including: architecture, setup and configuration, testing, and monitoring.
2
2
Session objectives• High Availability Disaster Recovery (HADR)
• Functional overview • Restrictions/limitations • Set up
• How to plan and implement your HADR solution• Best Practices
• HADR operations• Monitoring & alerting• Troubleshooting• DB2 administration in an HADR database
3
3
Goals of HADR• Ultra-fast failover• Easy administration• Handling of site failures • Negligible impact on performance• Configurable degree of consistency • Protection against errant transactions• Software upgrades without interruption• Very easy integration with HA-software • Eventually, no need for HA-software at all• Transparent failover and failback for applications
(combined with “client re-route”)
4
4
Basic principles of HADR• Two active machines
• Primary• Processes transactions• Ships log entries to the other machine
• Standby (or Secondary)• Cloned from the primary• Receives and stores log entries from the primary• Re-applies the transaction
• If the primary fails, the standby can take over the transactional workload• The standby becomes the new primary
• If the failed machine becomes available again, it can try to be resynchronized• The old primary becomes the new standby
5
5
Limitations/Restrictions• No CONNECT access to standby database• Not supported with Data Partitioning Feature (DPF), raw I/O for
transaction logs, or redirected restore• Primary and standby databases must have the same name, the same
operating system and DB2 levels and the same bit-ness (32-bit vs 64-bit)
• Rolling upgrades refer to Fix Pack upgrades only• Non-logged operations are not replicated:
• Updates to DB/DBM cfg and registry• LOBs > 1GB or LOBs marked as NOT LOGGED• NOT LOGGED INITIALLY operations
• Log archiving and self-tuning memory manager (STMM) can only be done on the primary database
With no CONNECT access, utilities cannot be run on the standby database. If the standby database is deactivated, an offline backup can be taken but during the time of deactivation and the subsequent time to “catch-up” to peer state, the standby is not valid for failover.
HADR is also not supported with Data Links in V8, however, note that Data Links is deprecated functionality in DB2 9.
6
6
Scope of activity• HADR replication takes place at the database level
Database E Database E
HADR HADRTCP/IP
Primary Standby
Database A
HADR HADRTCP/IP
Standby Primary
Database A
Database B
Database C
Database D
7
7
HADR implementation details
LogsOld
Logs
Log Writer Log Reader
Tables
IndexesLogs
Old
Logs
Log Reader
Tables
Indexes
TCPIP
DB2 EngineDB2 EnginePRIMARY SERVER
STANDBY SERVER
Log Writer
HADR HADR Shredder Replay Master
Redo SlavesRedo SlavesRedo SlavesReplay Slaves
Log PagesLog Pages
Log Pages
Log RecordsLog Pages
Log
Records
Client ReroutePrimary Connection
8
8
HADR State FlowHADR/DB2 Startup
Local catch-up
Remote catch-up Pending
Remote catch-up
Peer State
ConnectC
onne
ctio
n lo
st
9
9
HADR synchronization modes
Logs
Log writer
Logs
TCP/IP
Commit Succeeded
Synchronous
Near
- Syn
chro
nousAsynchronous
receive()send()
HADR HADR
Log writer
10
10
Planning• Three components to examine:
• Client• Network• Server
• Goal is to identify the expected behavior in the case of a component failure
TCPIP
TSA detects failure of Site A within 5 seconds and invokes forced HADR takeover on Site B
e.g. AIX server failure
Site BSite A
Expected resultFailure description
Failure location
Location operating as primary
11
11
Planning• Client
• Uses DB2 Client or DB2 application driver (Java, CLI)?- With application driver, the Java or CLI client will pick up the alternate
client from the server on first connect. If you need to support the clients being able to reroute without having successfully connected to the “primary” site, then need to harden the information in the client.
• Type of workload? Constant or spikes of activity?- Spikes of utilization may affect the ability of the standby to keep up so
should consider NEARSYNC (or ASYNC if only using the solution for disaster recovery and not HA)
• Availability requirements? Hours of operation?- What’s the priority on the system? Is it availability or ensuring no data
loss - each have a tradeoff and can influence the choice between syncmodes as well as options like HADR_TIMEOUT
- Need to understand if/when there are down-times on the system
TCPIP
Client reroute:For CLI, it’s an option on the definition of the data source. For JCC driver, need to implement the <TBC>
If there are down-times on the system:-Could do off-line maintenance activities or those with high impact to the standby (like offline reorg) during these periods and even consider deactivating the standby, doing maintenance, and then refreshing the standby with a new backup image of the primary
12
12
Planning• Network
• What is the network topology.. same network used between clients & server as between HADR primary & standby?
- Recommended to isolate HADR communication on a separate network from regular database communication; if shared, then ensure optimal network tuning to handle the peak data flow
• Is bandwidth greater than the database log generation rate• Redundant network adapters to avoid single point of failure?• Virtual IP for database servers?
- In the event of primary failure, the IP ownership moves to standby; no clients will be able to connect to the primary box
- Use client reroute by defining the same IP for the alternate client server as for the initial catalog entry
• Server• DB2 and operating system release levels?• Storage … How robust? Where will logs be archived?• Automation software?
TCPIP
Network topology:-Note that HADR requests tcp_nodelay for its connection for optimal performance; if ignored by the OS, it can negatively affect HADR performance-Increasing LOGBUFSZ can increase the amount of log flushed to the standby in each request… but more likely to need to increase DB2_HADR_BUF_SZ registry variable (to more than the default of 2xLOGBUFSZ) in order to handle peaks of log data flushed to standbyStorage:-Storage is a possible point of failure on both the primary and the standby… disk mirroring or some other sort of HA strategy for the disks can avoid requiring a failover due to disk failure; there is no interruption of service if a failover is not required, so always want to look to repair in place when possible-If archived logs are on external storage or something like TSM, then both primary and standby need to be able to access the location; also, delays in retrieving logs can result in significant delays in catch-up by the standby so you may want to retrieve archived logs to the standby before starting HADR to avoid that network overhead (but bear in mind that you need to manage the deletion of those retrieved logs)Server:-Run recommended operating system levels prescribed by IBM ensuring that all OS patches are applied in a timely fashion-Tivoli System Automation for Multiplatforms is available for Linux and AIX. HACMP is available for AIX.
13
13
Case A: Bright Yellow Truck Co.
•p5-575 AIX 5.2 64-bit ML6 with DB2 Viper 2•Tivoli Storage Manager (TSM) 5.3 is used to archive logs•HACMP 5.3 is the incumbent automation software
Server
Single public GB ethernet for all network traffic, heartbeat RSA connection between primary and disaster site
Network
24x7 proprietary application using DB2 clientClient
Alternate Case B: Little Blue Box JewelersClient-Java application running on WebSphere Application Server (WAS) using the JCC Type-4 driver-Application must be operational 8am-5pm weekdays and SaturdayNetwork- Public GB ethernet for client-server traffic, private GB ethernet for HADR connection between primary and disaster siteServer- x3650 Linux SLES9 SP3 with DB2 9- Logs are archived to shared disk- Tivoli System Automation for Multiplatforms 2.1 (TSA) for automation
14
14
Bright Yellow Truck – Planning Analysis
… and so on, including network failure, DB2 failure, filesystem damage/corruption, TSM server failure
Local disk is mirrored, no impact to DB2; storage team to replace failed disk
Disk failureSite ASite A
No automatic action; need to repair server on site B, restart, and regain peer state
AIX server failure
Site BSite A
HACMP invokes rc.hadr.start script on Site B to takeover on standby database; clients reconnect to Site B primary through client reroute
AIX server failure
Site ASite A
Expected ResultFailure description
Failure location
Primary location
15
15
Planning - HADR configuration• Two decisions to make about HADR connection:
• hadr_timeout • Time (in seconds) that HADR waits before considering a
communication attempt as failed• Heartbeat interval will be ¼ of hadr_timeout or 30 seconds,
whichever is shorter• hadr_syncmode
• SYNC, NEARSYNC, or ASYNC• For disaster recovery, most choose between SYNC and
NEARSYNC -> performance testing is valuable to quantify the difference between the two
Note that, for maximal availability, as soon as the connection is closed (because the other end closed the connection, a network error is detected, or timeout is reached), the primary drops out of peer state to avoid blocking transactions.A recommended timeout value depends on the capacity of the data server and of the HADR network connection. If either are busy, then it can result in a delay in response between the HADR pair and that will start the countdown to the HADR_TIMEOUT value.In most enterprise implementations, an HADR_TIMEOUT between 30 and 60 seconds provides the best balance of a short timeout without risking falling out of peer due to too short a timeout. An HADR_TIMEOUT less than 30 seconds should be thoroughly tested to ensure the primary does not drop out of peer state as part ofregular processing.
16
16
Big Yellow Truck Decisions• After testing SYNC and NEARSYNC, no significant performance
impact was detected between the two so set hadr_syncmode=SYNC
• Highest priority at Big Yellow Truck was data integrity – no transactions could be lost without a ripple effect through the organization• Risk even in SYNC mode; if the TCP/IP socket closed on the
primary, the primary could drop out of Peer mode immediately and if the primary failed and a forced takeover occurred before the standby reconnected and regained peer state there was the possibility of data lossChose to implement new Viper 2 feature: HADR peer window
17
17
HADR Peer Window• HADR_PEER_WINDOW can be used to cause the HADR
pair (primary and standby databases) to behave as though in peer state even if the primary loses connection with the standby database• Once connection to standby is lost, primary waits the
number of seconds specified by HADR_PEER_WINDOW before dropping out of Disconnected Peer state and continuing to process transactions
• Peer window end (meaning the time at which the primary is no longer guaranteed to be in peer state) is sent to the standby through the heartbeat - timestamp can be checked by automation software to determine whether or not to do forced takeover
• If standby does not reconnect with primary by the end of the peer window, the primary drops from Disconnected Peer state to Remote Catchup Pending
Note: important to synchronize clocks between primary and standby for effective usage of End of peer window
18
18
Setting HADR_PEER_WINDOW• DB2_HADR_PEER_WINDOW should be set to a value that fits the
following equation:
DB2_HADR_PEER_WINDOW >= response_time + fudge_factor + heartbeat_interval
response_time = estimated time for automation software to detect failure and invoke HADR takeover
fudge_factor = 5 sec, small delta time delay because detection/response is not instantaneous
heartbeat_interval = MIN(HADR_TIMEOUT/4, DB2_HADR_PEER_WINDOW/4, 30 sec)
Big Yellow Trucks determined their response time as a minimum of 45secsHACMP detection of failure: 5 secondsHACMP processing time until HADR takeover invoked: up to 40 seconds
19
19
HADR State Flow with Peer WindowHADR/DB2 Startup
Local catch-up
Remote catch-up Pending
Remote catch-up
Peer State
Connect
Con
nect
ion
re-e
stab
lishe
d
Con
nect
ion
lost
Disconnected Peer State
Connection lost
Peer window expires
20
20
Set-up• Prepare and clone the primary database and
configure for HADR• Create the standby database from the clone• Configure the standby database for HADR• Configure client reroute • Start-up HADR• Set-up automation process
21
21
Prepare and clone the primary DB• Ensure archive logging is enabled on the primary database
Archive log location must be accessible by both primary and standby database
• Sample database configuration for Case A: LOGFILSIZ 16380LOGSECOND 12LOGARCHMETH1 TSM:DB2
• Sample database configuration for Case B:LOGFILSIZ 8192LOGSECOND 2LOGARCHMETH1 DISK:/shared/db2inst1/logarchives
• Take backup or split mirror copy of primary database and move the backup image to the standby location
22
22
Code snippet - Configure primaryexport $dbname=TRUCKERSexport $pri=yellow1.bigtruck.comexport $sec=yellow2.bigtruck.comexport $svc=7000db2 connect to $dbnamedb2 update db cfg for $dbname using LOGFILSIZ 16380db2 update db cfg for $dbname using LOGSECOND 12db2 update db cfg for $dbname using LOGARCHMETH1 TSM:DB2db2 update db cfg for $dbname using INDEXREC RESTARTdb2 update db cfg for $dbname using LOGINDEXBUILD ONdb2 update db cfg for $dbname using HADR_TIMEOUT 30db2 update db cfg for $dbname using HADR_PEER_WINDOW 90db2 update db cfg for $dbname using HADR_LOCAL_HOST $pridb2 update db cfg for $dbname using HADR_LOCAL_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_HOST $secdb2 update db cfg for $dbname using HADR_REMOTE_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_INST sitebdb2 update db cfg for $dbname using HADR_SYNCMODE SYNCdb2 deactivate db $dbnamedb2 activate db $dbname
Configuration for Case B would be:export $dbname=DIAMONDexport $pri=bluebox.jeweler.comexport $sec=bluebox2.jeweler.comexport $svc=7000
db2 connect to $dbnamedb2 update db cfg for $dbname using LOGFILSIZ 8192db2 update db cfg for $dbname using LOGSECOND 2db2 update db cfg for $dbname using LOGARCHMETH1 DISK:/shared/mydb/archivelogsdb2 update db cfg for $dbname using INDEXREC RESTARTdb2 update db cfg for $dbname using LOGINDEXBUILD ONdb2 update db cfg for $dbname using HADR_TIMEOUT 30db2 update db cfg for $dbname using HADR_LOCAL_HOST $pridb2 update db cfg for $dbname using HADR_LOCAL_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_HOST $secdb2 update db cfg for $dbname using HADR_REMOTE_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_INST sitebdb2 update db cfg for $dbname using HADR_SYNCMODE SYNCdb2 deactivate db $dbnamedb2 activate db $dbname
23
23
Create the standby DB and configure• Restore the backup image or db2inidb the split mirror copy
database is left in roll forward pending state
db2 update db cfg for $dbname using LOGFILSIZ 16380db2 update db cfg for $dbname using LOGSECOND 12db2 update db cfg for $dbname using LOGARCHMETH1 TSM:DB2db2 update db cfg for $dbname using INDEXREC RESTARTdb2 update db cfg for $dbname using LOGINDEXBUILD ONdb2 update db cfg for $dbname using HADR_TIMEOUT 30db2 update db cfg for $dbname using HADR_PEER_WINDOW 90db2 update db cfg for $dbname using HADR_LOCAL_HOST $secdb2 update db cfg for $dbname using HADR_LOCAL_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_HOST $pridb2 update db cfg for $dbname using HADR_REMOTE_SVC $svcdb2 update db cfg for $dbname using HADR_REMOTE_INST siteadb2 update db cfg for $dbname using HADR_SYNCMODE SYNCdb2 deactivate db $dbname
Strict symmetry of table space and container configuration is required on the standby. Name, path, size all must match. Relative container paths are allowed, and the full path may differ in this case.
If strict symmetry does not exist, HADR may fail to replicate a container operation on the standby. In such a case replication of the affected table space stops and the table space will be left in "roll forward in progress"
24
24
Configure automatic client reroute• On both the primary and standby database servers issue:
db2 update alternate server for db $dbname using hostname <host> port <port_num>
• Note: Alternate server is flowed to the client and cached on first connect; without a connection, the client will not know where to re-route• To protect against this on a DB2 client, the same
command can be run on the client to ‘harden’ the alternate server
• On a JCC-Type 4 client with JNDI, use the clientRerouteServerListJNDIName DataSource property to persist alternate server information across JVMs and to ‘harden’ the alternate server
For JDBC and SQLJ clients:http://publib.boulder.ibm.com/infocenter/db2luw/v9/index.jsp?topic=/com.ibm.db2.udb.apdv.java.doc/doc/r0021822.htm
You need to set properties forcom.ibm.db2.jcc.DB2BaseDataSource.clientRerouteAlternatePortNumbercom.ibm.db2.jcc.DB2BaseDataSource.clientRerouteAlternateServerName
25
25
Start-up HADRSTANDBY
db2 start hadr on db $dbname as standby
db2 get snapshot for db on $dbname |grep –A 15 “HADR Status”
ORdb2 select * from
sysibmadm.snaphadr
PRIMARY
db2 start hadr on db $dbname as primary
db2 get snapshot for db on $dbname |grep –A 15 “HADR Status”
Start local catch-up
Validate connection between primary & standby and start remote catch-up
Upon starting the standby, the Local Catchup phase can begin:-Replay locally available log files (if any)-Replay log files that may be retrieved at the primary and shipped to the standby
Once the primary is started (which is an asynchronous operation), remote catchup can begin:-Replay log pages from the primary's archived logs-Replay log pages from the primary's active log to the standby until the standby is caught up to the tail of the log.-Replay log pages from the primary's in-memory log buffer to the standby whenever the primary flushes those pages to disk
At the conclusion of remote catchup, the standby is now in Peer state.
26
26
Start-up on standby – Expected ResultsDB20000I The START HADR ON DATABASE command completed successfully.
HADR StatusRole = StandbyState = Remote catchup pendingSynchronization mode = SyncConnection status = Disconnected, 03/22/2007 09:50:12.423522Peer window end = Null (0)Peer window (seconds) = 65Heartbeats missed = 0Local host = bbird007Local service = HADR_test2Remote host = bbird006Remote service = HADR_test1Remote instance = db2hdatimeout(seconds) = 30Primary log position(file, page, LSN) = S0000115.LOG, 0, 0000000748858000Standby log position(file, page, LSN) = S0000114.LOG, 1, 000000074485D0BCLog gap running average(bytes) = 67088191
27
27
Start-up on primary – Expected Results
DB20000I The START HADR ON DATABASE command completed successfully.
HADR StatusRole = PrimaryState = PeerSynchronization mode = SyncConnection status = Connected, 03/22/2007 09:50:50.622363Peer window end = 03/22/2007 09:51:56.000000 (1174571516)Peer window (seconds) = 65Heartbeats missed = 0Local host = bbird006Local service = HADR_test1Remote host = bbird007Remote service = HADR_test2Remote instance = db2hdatimeout(seconds) = 30Primary log position(file, page, LSN) = S0000115.LOG, 0, 0000000748858000Standby log position(file, page, LSN) = S0000114.LOG, 1, 000000074485D0BCLog gap running average(bytes) = 67088191
28
28
Configure automation software (before DB2 9.5)• For TSA, use DB2-provided scripts hadr_start.ksh,
hadr_stop.ksh, and hadr_monitor.ksh• Reference whitepaper: “Automating DB2 HADR
Failover on Linux using Tivoli System Automation on Multiplatforms”(ftp://ftp.software.ibm.com/software/data/pubs/papers/hadr_tsa.pdf)
• For HACMP, use DB2-provided scripts rc.hadr.start, rc.hadr.stop, rc.hadr.monitor• Reference whitepaper: “Automating IBM DB2 UDB
HADR with HACMP”(ftp://ftp.software.ibm.com/software/data/pubs/papers/hadr-hacmp.pdf)
29
29
Configure automation software (in DB2 9.5)
• For Linux and AIX, Tivoli System Automation for Multiplatforms (TSA MP) is now bundled with and installed by DB2
• db2haicu utility is used to specify the cluster manager configuration• Create a cluster domain
• Define local DB2 instance resource groups on each node• Note that the default behavior is to restart the instance in place by default
• Define quorum device for the domain• Define DB2 HADR database resource across the cluster, online on
the primary and offline on the standby • Define the network interface to the cluster domain
• Optionally, define network resource for a floating virtual IP
• db2pd –ha shows the cluster manager status
Requires a sysadmin step to initialize the TSA environment, but the majority is DBA driven.Vendor interfaces have been provided, but it’s TSA that’s bundled. Work underway with VERITAS, but no immediate plans to include HACMP.
30
30
Next steps• Yippee! HADR is now up and running…
… but now you need to test it and develop procedures for the HADR environment
• Testing matrix should cover all the failure scenarios outlined during the planning phase to validate expected results
• Measurements to capture:• Time to detect failure, time for failover, and time until new
primary is open for business (any open transactions at time of failure rolled back and database ready to process new transactions)
• Performance of workload• Server utilization statistics• Consistency check between primary and standby data
31
31
Best Practices – HADR operations• Database start-up
• STANDBY: db2start; db2 activate db $dbname• PRIMARY: db2start; db2 activate db $dbname• Verify success with snapshot on PRIMARY
• If standby not available, PRIMARY: db2start; db2 start hadr on db $dbname as primary by force
• Database shut-down• PRIMARY: db2 deactivate db $dbname; db2stop force• STANDBY: db2 deactivate db $dbname; db2stop
Recommended aliases:alias gethadrhost=’db2 get db cfg for $dbname |grep HADR_LOCAL_HOST’alias getdbrole=’db2 get db cfg for $dbname |grep “HADR database role”’alias hadrstate=’snaphadr | grep State’alias snaphadr=’db2 get snapshot for db on $dbname |grep –A 15 “HADR Status”’
32
32
Best Practices – HADR operation• Database role reversal
• Verify peer state on primary with snapshot• STANDBY: db2 takeover hadr on db $dbname
• Verify success with snapshot on new PRIMARY
Primary Standby
33
33
Best Practices – HADR operations• Manual database failover
• With peer window configuration, issue on STANDBY: db2 takeover hadr on db $dbname by force peer window only
• Without, issue on STANDBY: db2 takeover hadr on db $dbname
• If this fails, need to evaluate ‘safe-ness’ of doing the forced takeover – risk of split-brain or losing transactions
34
34
Is the primary server (and its disk) available?
YES, OK to run a planned
takeover
On STANDBY, check the HADR state of the database, use alias:
hadrstate
On PRIMARY, run db2pdlog –l <outfile> from the database directory (where SQLOGCTL.LFH exists)
No
UNPLANNED EVENT: Task flow for determining ‘safe-ness’ to failover
Yes
Risk of data loss with
forced HADR takeover
Any other state
Remote catchuppending
state
Peer state
Did the primary server fail while the databases were still in peer (by comparing timestamps from automation software (AS)
logs and db2diag.log on standby)?Get timestamp from previous step’s message.Get timestamp from automation log for when the primary server failed (or whatever event
caused the failover)
On STANDBY, check diaglog. Was db last in Peer
before changing to remote catchup pending (“HADR state set to S-
RemoteCatchupPending(was S-Peer)”)?
Yes
No
YesNo
Compare db2diag timestamp (DT) and AS log timestamp (AT). Is the
DT within a “safe” (duration of failure detection) timespan?
YesNo
In <outfile>, is hdrPrimaryLastState
= 0x10 (Peer)
Note that the values returned from the db2pdlog command for hdrLastPrimaryState are internal values. The possible values are:0x01 – Booting0x02 – Waiting to enter remote catchup0x04 – Remote catchup0x08 – Nearly peer0x10 – Peer
35
35
Best Practices – HADR operations• Failed primary reintegration
• On old PRIMARY (now new STANDBY): db2 start hadr on db $dbname as standby
• HADR and reintegration starts asynchronously so success must be monitored
• If reintegration is not possible, details will be logged in the db2diag.log as to why it failed to synchronize
36
36
Best Practices – HADR operations• Rolling fix pack upgrade
1. On site B, upgrade the database• Deactivate standby database and stop DB2• Upgrade software, run db2iupdt• Restart DB2 and activate standby database• Wait for HADR pair to regain Peer state
2. Switch primary/standby role• Issue “TAKEOVER HADR ON DB $DBNAME” on site B
3. On site A, upgrade the database• Deactivate new standby database and stop DB2• Connect to site B, run BIND, db2updv8, etc on the new primary to complete the
upgrade on site B• Upgrade software, run db2iupt• Restart DB2 and activate new standby database• Wait for HADR pair to regain Peer state which completes the upgrade on site A
4. Optionally, switch primary/standby role
37
37
Best Practices – Alerts• HADR state
• Monitor hadr_state element for ‘Peer’ on the primary• Not logged objects exist
• On the primary, check SYSCAT.COLUMNS for any LOGGED=‘N’select tabschema, tabname, colname from syscat.columns
where logged=’N’
• Not logged initially table operations• Scan the administration notification log on the primary for
ADM5530W which indicates that a table has committed activity that was not logged (either created NOT LOGGED INITIALLY or altered to activate NOT LOGGED)
38
38
Best practices - Alerts• Non-replicated load
• Monitor the diagnostic log on the standby• Load copy cannot be accessed on the standby:
FUNCTION: DB2 UDB, database utilities, sqluMCCheckDevType, probe:10MESSAGE : Media controller -- invalid device path:
/db2fs/db2hda/HDA.4.db2hda.NODE0000.CATN0000.20060406045810.001
FUNCTION: DB2 UDB, recovery manager, sqlpRecDbRedo, probe:2129MESSAGE : Tablespace 7 in restore pending state.
• Non-recoverable loadFUNCTION: DB2 UDB, High Availability Disaster Recovery,
hdrTakeoverCheckTablespaces, probe:51010 MESSAGE : Warning: Tablespace is not in NORMAL state following
takeover TSID:DATA #1 : Hexdump, 8 bytes0x0000007FBFFBA2B0 : 0700 0000 0000
000 ........
39
39
Best practices - Alerts• Non-normal table space state
• If any load operation has an inaccessible load copy• If there’s a disk error on the standby• On the standby, check the State column in the
Tablespace Statistics report generated from db2pd -db <dbname> -tablespaces
• Remember, some non-zero states are okay
40
40
Trouble-shooting• db2diag.log is the primary tool for trouble-shooting• Example: Identifying which role a system should be
• Worst-case scenario where maybe automation failed to takeover and you have two standby databases .. or maybe two primary databases
db2diag -gi "funcname:=hdr" -fmt "%{tsmonth}-%{tsday}-%{tshour}.%{tsmin}.%{tssec} %{process} %{changeevent}\t %{message}"
04-11-13.09.57 db2hadrp (HDA) 0 HADR state set to P-Peer (was P-NearlyPeer)04-11-13.25.08 db2hadrp (HDA) 0 Info: Primary has started non-forced takeover request.04-11-13.25.08 db2hadrp (HDA) 0 Info: Primary is switching to standby role....04-11-13.59.16 db2hadrs (HDA) 0 HADR state set to S-Peer (was S-NearlyPeer)04-11-14.16.36 db2hadrs (HDA) 0 Info: Standby has initiated a takeover.04-11-14.16.36 db2hadrs (HDA) 0 Info: Standby switching roles to primary.04-11-14.16.40 db2hadrp (HDA) 0 HADR state set to P-Peer (was S-Peer)04-11-14.16.41 db2hadrp (HDA) 0 Info: Standby has completed takeover (now primary)....04-27-15.11.26 db2agent (HDA) 0 Info: HADR Startup has begun.04-27-15.11.26 db2hadrp (HDA) 0 Info: Primary Started.04-27-15.24.39 db2hadrs (HDA) 0 HADR state set to None (was None)04-27-15.24.39 db2hadrs (HDA) 0 HADR state set to S-Boot (was None)
On Unix, when an HADR database is operating as a primary database, the process name in diagnostic log entries is “db2hadrp”. When operating as a standby database, the process name is “db2hadrs”. On Windows, the state description for each HADR state set message uses “P-“ to indicate primary and “S-“ to indicate standby.
41
41
Trouble-shooting
• Congestion• Can result in delay of transaction commits on
the primary if in SYNC or NEARSYNC• Due to competition for network or log receiving
blocked at standby• If due to peaks in workload and see an increase
in log gap, consider increasing DB2_HADR_BUF_SIZE to 4x or 8x LOGBUFSZ (default setting is 2xLOGBUFSZ)
42
42
Administration considerations• Restore:
• Table space level restore is not allowed• Database level restore can reset HADR role to STANDARD
• Table and index reorganization:• Online reorganization logs at fine granularity so replicated frequently;
cannot restart an online reorg started on site A after failover to site B• Offline reorganization can have a significant impact on HADR as a
single log record could encompass many updates .. worst case, the standby repeats the offline reorg after the primary completes
• Not logged initially (NLI) operations• NLI operations are not replicated; may not be an issue if it’s for
temporary or staging operations, but should monitor this scenario
Re-clustering REORGOperations logged per hundreds or thousands of affected rowsSecondary may fall behind in this case.
Non re-clustering REORGEntire operation is a single record generated after REORG on PrimaryGreatest impact on secondary to keep up with Primary
43
43
Administration considerations• Load operations:
• Must be run with COPY YES option to be replicated• All other options result in standby becoming inconsistent with primary
and no longer valid for failover• User DB2_LOAD_COPY_NO_OVERRIDE registry variable to avoid non-
copied loads
• Not-logged large objects:• LOBs greater than 1GB or those defined as NOT LOGGED will not be
replicated to the standby• If these objects are required in DB, then will need regular refreshes of
standby from primary backup to keep in sync
• Configuration updates and file system changes:• Recommended to keep configuration synchronized between primary and
standby by applying updates to both
When loading data into tables, a LOAD ... COPY YES ... is run on the primary and the data is replicated on the standby if the standby can access the copy through the path or device specified in the load command.
If the load copy cannot be accessed on the standby, the table space in which the table is stored is marked bad on the standby and the standby will skip future log records regarding this table space. In this situation, the standby database has to be rebuilt from a backup image taken after the LOAD operation was issued on the primary.
A LOAD ... NONRECOVERABLE is run on the primary as usual. On the standby, the table will be marked bad and the standby will skip future log records regarding this table. If the table is not going to be dropped as part of the load processing, the table can be recovered by issuing a LOAD COPY YES REPLACE for the table on the primary, or the standby can be rebuilt from a backup image taken after the LOAD ... NONRECOVERABLE was issued.
If a LOAD ... COPY NO is run on the primary, it is converted to a LOAD NONRECOVERABLE by default. However, this conversion can be overridden to convert to a COPY YES operation by setting DB2_LOAD_COPY_NO_OVERRIDE=”COPY YES TO <shared directory path>”. When the registry variable is used, all LOAD … COPY NO operations will be converted to and behave according to the LOAD … COPY YES behavior outlined above.
44
44
Summary• HADR is an easy-to-use technology• Early planning will lead to reaping rewards with a
straight-forward implementation• Pack of procedures will keep things simple and
enable smooth operation of HADR
45
45
Jessica Escott-RockwoodIBM Toronto Lab
Session D09
DB2 HADR – Case Study of Implementation