Db2 Performance Tips And Tricks - WordPress.com · Explain plan to your right is a good plan for the delete that’s executing in 5 ms. Notice how the staging table to the bottom

Db2 Performance Tips And Tricks

Raghu Cherukuru (Sr. DB2 DBA, https://www.raghu-on-tech.com) 1

About Me

▪ I am Raghu Cherukuru and working with DB2 for close to 13 years. Started with V8.1 before HADR/STMM were features on DB2. Like to code in Shell/perl/Java but loves performance tuning and SQL tuning.

▪ I started blogging a couple of years ago at https://www.raghu-on-tech.com/ and I am also an IBM champion for 2019.

▪ I live on the Wisconsin side of Illinois-Wisconsin border with my wife and 2 year old son.


https://www.raghu-on-tech.com/

Agenda

▪ General best practices.

▪ Monitoring and tools.

▪ Reporting using WLM data.

▪ pureScale specific tips.

▪ DPF specific tips.

▪ Quick word about SORTING.

▪ SQL Tuning.


Signs Of Good Performance ?

▪ A system that is responsive → Good response time

▪ A system that processes a given number of requests maintaining the responsiveness → Throughput

▪ Most important of all happy users and business!

▪ Usually Response time and Throughput are inversely proportional.


General Best Practices (GBP)

▪ Transaction log buffer is shared by all the DB2 agents, proper buffer size and fast log disk are needed.

▪ Check for log write time per IO and also number of times DB2 had to flush the log buffer to disk because the buffer is full.


Locking GBP

▪ Incompatible locking serializes the requests, i.e. requester needs to wait until the owner releases locks.

▪ Don’t keep transactions open for longer than they need to be.

▪ Use the least restrictive isolation level that application needs.

▪ There are a number of ways to monitor anything these days, this including locks.


Monitoring Locks

SELECT SUBSTR(PRIMARY_AUTH_ID,1,20) as PRIMARY_AUTH_ID, SW.AGENT_ID,

SW.AGENT_ID_HOLDING_LK, LOCK_MODE, LOCK_OBJECT_TYPE, SI.APPL_STATUS,

TIMESTAMPDIFF(2,CHAR(CURRENT TIMESTAMP-LOCK_WAIT_START_TIME)) LOCK_WAIT_TIME_INSEC,

LOCK_MODE_REQUESTED, SW.MEMBER from SYSIBMADM.SNAPLOCKWAIT SW,

SYSIBMADM.SNAPAPPL_INFO SI WHERE SW.AGENT_ID=SI.AGENT_ID WITH UR;

SELECT DISTINCT A.LOCK_WAIT_START_TIME, B.APPLICATION_HANDLE,

A.HLD_APPLICATION_HANDLE, A.LOCK_NAME, A.HLD_MEMBER, A.LOCK_STATUS FROM

TABLE(MON_GET_APPL_LOCKWAIT(NULL,-2)) A JOIN

TABLE(MON_GET_LOCKS(CLOB('<lock_name>'||A.LOCK_NAME||'</lock_name>'),-2)) B ON

A.LOCK_NAME=B.LOCK_NAME WHERE A.HLD_APPLICATION_HANDLE IS NOT NULL;

Most comprehensive way to capture locking info is to use new Java based event monitors. They need

MON_LOCKTIMEOUT and MON_DEADLOCK DB CFG to be set to HIST_AND_VALUES to get the most out of

them. Only enable them when you are trying to troubleshoot a recurring problem.


OLTP GBP

▪ For transactional applications with simple queries, use parameter markers or repeated compilation of the same query is going to kill CPU and reduce throughput.

▪ Also helps improve package cache hit ratio.

▪ If packaged/vendor products come with literals and killing your system, take a look at STMT_CONC (statement concentrator).

▪ If your application is using parameter markers where it should really be using literals consider using REOPT ALWAYS.

▪ REOPT ALWAYS can be specified while binding you CLI packages or on the JDBC connection string.


Event Monitors GBP

• Great way to troubleshoot performance problems.

• DO NOT create indexes on tables they write to, especially when they are active or they are going to slowdown your database.

• If you need to create indexes, just archive the data on to a remote server and then start analyzing the data.


Federation GBP

▪ IMHO, federation and performance do not belong in the same sentence.

▪ When you write queries using federated nicknames you are limiting the optimizers capabilities.

▪ Queries does not always do what you expect them to do, for e.g. federated server might drop the isolation level specified depending on the data source.

▪ Should be used only in limited cases, such as when you need to perform a lookup against small master data table.


Federation GBP

▪ You could create indexes on nicknames but they are specification only, they are only there to help optimizer with indexing information on source.

▪ Use SYSPROC.NNSTAT stored procedure to collect statistics on nicknames.

▪ Try to avoid them in critical transactional systems.


Raghu Cherukuru (Sr. DB2 DBA, https://www.raghu-on-tech.com)

12

Monitoring And Tools

▪ MON_GET table functions are your friends.

▪ Activity event monitors are great for troubleshooting an issue indepth.

▪ WLM can replace your activity event monitor and provide long term trends of performance as well.

▪ If you don’t have a predefined set of monitoring scripts, you can’t go wrong with db2mon.

https://www.ibm.com/support/knowledgecenter/en/SSEPGG_11.5.0/com.ibm.db2.luw.admin.perf.doc/doc/t0070377.html


https://www.ibm.com/support/knowledgecenter/en/SSEPGG_11.5.0/com.ibm.db2.luw.admin.perf.doc/doc/t0070377.html

Gold Mine Query

select

substr(P.ROUTINENAME,1,48) as ROUTINENAME,

substr(P.SPECIFICNAME,1,48) as SPECIFICNAME,

case when P.ROWTYPE in ('B','O','P') then CHAR('IN',3) else CHAR('OUT',3) end as IN_OUT,

cast(p.ORDINAL as char(3)) as ORD,

substr(P.PARMNAME,1,40) as PARMNAME,

substr(P.TYPENAME,1,16) as TYPE

from sysibm.sysroutines r, sysibm.sysroutineparms p

where p.routineschema=r.routineschema

and p.routinename=r.routinename

and p.specificname=r.specificname

and r.function_type='T'

and substr(r.ROUTINENAME,1,4) in ('SNAP','MON_','ENV_','WLM_','COMP')

order by P.ROUTINESCHEMA,P.ROUTINENAME,P.SPECIFICNAME,IN_OUT,P.ORDINAL;


Monitoring And Tools (DISK)

▪ If you are running DB2 on linux, you may use iostat –xtc or newer dstat command.

▪ With in DB2, query MON_GET_TABLESPACE, MON_GET_TABLE, MON_GET_INDEX to determine which objects are driving the most IO.

▪ Once you identify the Table or Index driving the IO, try to identify the queries accessing these objects.

▪ You may use MON_GET_PKG_CACHE_STMT or an activity event monitor to identify the offending queries.



▪ If the query driving the IO references the table via a view, you may not be able to retrieve it from the package cache.

▪ A better way to identify queries accessing specific object(s) is to use “USAGE LISTS”. They were introduce in V10.1

https://www.ibm.com/support/knowledgecenter/SSEPGG_10.5.0/com.ibm.db2.luw.admin.dbobj.doc/doc/c0058647.html

▪ If nothing conclusive, is bufferpool too small ?


https://www.ibm.com/support/knowledgecenter/SSEPGG_10.5.0/com.ibm.db2.luw.admin.dbobj.doc/doc/c0058647.html


Tables involved in most number of table scans:

select substr(TABSCHEMA,1,50) as TABSCHEMA, substr(TABNAME,1,50) as TABNAME,TABLE_SCANS, OBJECT_DATA_P_READS from table(MON_GET_TABLE('','',-2)) as t order by TABLE_SCANS desc fetch first 10 rows only;

Tablespaces with most IO:

select substr(TBSP_NAME,1,30) as TBSP_NAME, POOL_DATA_L_READS, POOL_INDEX_L_READS, POOL_DATA_P_READS, POOL_INDEX_P_READS, POOL_TEMP_INDEX_P_READS FROM TABLE(SYSPROC.MON_GET_TABLESPACE(‘’, -2)) TBS ORDER BY (POOL_DATA_L_READS+POOL_INDEX_L_READS+POOL_TEMP_DATA_L_READS) DESC FETCH FIRST 10 ROWS ONLY WITH UR;

Statements Contributing to most IO:

select MEMBER, SUBSTR(STMT_TEXT,1,50) AS STMT_TEXT, ROWS_READ from TABLE(MON_GET_PKG_CACHE_STMT ( NULL, NULL, NULL, -2)) as T order by ROWS_READ DESC FETCH FIRST 10 ROWS ONLY with ur;


MON_GET_PKG_CACHE_STMT and ACTIVITY_METRICS


No better way to troubleshoot

different portion(s) of waits.

Total activity wait time

Monitoring And Tools (CPU)

▪ Could query MON_GET_CONNECTIONS using DGTT’s to measure differential over X number of minutes and then identify which connections are using most CPU.

▪ Repeated table scans or Index leaf level scans with in the bufferpool needs lot of CPU.

▪ SQL with out parameter markers burn CPU.

▪ MON_GET_PKG_CACHE_STMT


Few Words On WLM

▪ See if you can use WLM for performance monitoring and reporting, starting V11.5 its not restricted to advanced editions anymore.

▪ I use WLM on both my transactional and analytical databases and I like the data it provides for both long term and short term troubleshooting.

▪ It can be used to generate some great reports.



What kind of SQL provides below report ??


22


23

pureScale Specific Tips

▪ Monitor page reclaims, at the object level i.e. Tables and Indexes. Use MON_GET_PAGE_ACCESS_INFO.

▪ Indexes with monotonically increasing sequence numbers tend to cause problems due to bouncing of leaf page between the members.

▪ Consider adding CURRENT MEMBER special register as a leading column to the Index. You need to add a column to the table first and range partition it on CURRENT MEMBER.

▪ Use Read Stability (RS) and RR isolation levels very carefully.


pureScale Specific Tips(Contd…)

▪ Use 4K pagesize tablespaces for your databases on pureScale, to alleviate page reclaim issues and CF waits.

▪ Smaller page sizes also saves you on bufferpool memory.

▪ If you are running GPFS v4.1.1 or lower your cluster may slow down while creating indexes. Detective work at below blog post.

https://www.raghu-on-tech.com/2019/01/28/db2-purescale-cluster-slowness-during-index-creation/


https://www.raghu-on-tech.com/2019/01/28/db2-purescale-cluster-slowness-during-index-creation/


▪ Inplace reorgs are not supported on range partitioned tables (RPT) in a pureScale environments.

▪ Be vigilant about this restriction if you have plans to use RPT’s for partition roll in and roll out in a pureScale environment.

▪ pureScale lock escalations could happen locally at a member level and globally at a CF level. Lock escalations are always exclusive and kill your concurrency.


Contd…

▪ While writing/developing queries, be very careful if your application have DELETE’s or UPDATE’s with correlated sub-queries.

▪ You will see eternal slowness in query execution times if these queries use a HASH join, I am talking about 5ms vs 10 minutes for a two row delete ???


Contd…

▪ Explain plan to your right is a good plan for the delete that’s executing in 5 ms.

▪ Notice how the staging table to the bottom left of the plan have only 26 rows and a NLJOIN is used to match the rows on the inner table.


Contd…

▪ To your right is the BAD plan!

▪ Notice how the staging table got switched to the bottom right and the cardinality of the staging table went from 26 to 56831 ?

▪ Optimizer thought it’s a great thing to switch from NLJOIN to HSJOIN to speed up the query but NOT HERE!



WHY ?




▪ Complete analysis of the problem including how I was able to extract the plan that the application used while the slowness was happening is detailed in the blog post below.

https://www.raghu-on-tech.com/2019/08/29/slow-hash-joins-in-purescale/


https://www.raghu-on-tech.com/2019/08/29/slow-hash-joins-in-purescale/

DPF Specific Tips

▪ In a DPF environment your access patterns would be very different from a Transactional system. Some of the characteristics of your workload include

▪ Scanning, returning large volumes of data.

▪ Large number of HASH joins, MS Joins.

▪ Usage of large SORT HEAP and Temporary tablespaces.

▪ Resource utilization is driven by small but resource intensive requests.

▪ Less number of concurrent users compared to OLTP.


DPF Specific Tips (Contd…)

▪ Identifying a resource intensive query or utility is easier in DPF compared to your transactional database.

ps –mo THREAD –p $pid (AIX)

Ps –lLfp $pid (Linux)

Now you should be able to grep for only running threads and see how much CPU each of these Kernel threads are consuming. Then you can map these KTID’s to DB2 agent ID's.


$pid=pid of db2sysc



▪ If you identify that one of the KTID’s is consuming a lot of CPU, you could map the kernel thread to DB2 EDU by using below technique.

db2pd –edus|grep <Kernel Thread ID from screen shot>

db2pd –agents|grep <eduid from above command>

db2pd -db db2db -activestatements apphdl=59060115 (This needs to be run from coordinating partition otherwise no output will be returned).



▪ When you identify a resource consuming SQL, you may generate a formatted explain to analyze the access plan. Watch out for▪ Non-Collocated joins i.e. joins where the tables are joined on non-

distribution or hash key.

▪ BTQ (Broadcasted Table Queues) and DTQ (Directed Table Queues) indicate that joins are happening on a non-hash keys.

▪ Also understanding which sub-section of the access plan is holding up the query may be crucial to resolving the slowness of the query.



▪ How are BTQ’s and Subsections related ?In DPF environment, different parts of the access plan can be executed parallelly and these individual parts are separated by Table Queues. Individual parts are called subsections.

▪ How do you identify subsection information in explain plan ?db2expln is your friend, -i option displays subsection information in the plan. For e.g.

db2expln -d db2db -t -g -i -f temp.sql

Where temp.sql contains the SQL text.



▪ Once you identify all the subsections and tables involved in each of those subsections, you could monitor the progress by using a GLOBAL SNAPSHOT or SNAP_GET_SUBSECTION.

▪ I still don’t see the SUBSECTION monitoring capabilities via MON_GET table functions.

Below is a great IBM white paper on understanding and monitoring table queues/subsections.

https://www.ibm.com/developerworks/data/library/long/dm-1212db2tablequeues/index.html


https://www.ibm.com/developerworks/data/library/long/dm-1212db2tablequeues/index.html


QTY PRDID DTID

10 1 1

25 2 2

PRDID PRODUCT

1 PRD1

2 PRD2

DTID DATE

1 11/21/2019

2 11/27/2019

▪ How do you HASH these tables ?▪ If you HASH the Fact table by Product

ID assuming Product Dimension is HASH’ed by PRDID, the JOIN to DATE Dimension will be non-collocated.

▪ If you partition FACT by DTID, then join to PRODUCT DIM will be non-collocated

FACT TABLE

DIMENSION1 DIMENSION2


▪ Replicated MQT’s to the rescue.▪ Create a replicated MQT on the smallest of the dimensions and

partition the FACT inline with largest DIMENSION.

▪ Replicated MQT materializes the table on all the partitions of the database partition group.

▪ Ensure to create all these 3 tables in the same database partition group or the data will not be collocated.

▪ Don’t forget “ENABLE QUERY OPTIMIZATION” clause on the create replicated MQT statement.



▪ Try to understand if your existing data model is the right one for any new report requirements that come up.

▪ We had success by duplicating data and denormalizing the data per new report requirements rather than using the existing model.

▪ Some times you need to choose between ETL and reporting teams i.e. your design may only benefit one of the teams others might take a hit.

▪ In these situations throw ETL guys under the bus because they could mitigate some of these issues by breaking down SQL and creating work tables etc.


Is SORT really bad ?

▪ It depends on why DB2 is performing the sort! Take an e.g. of searching for ID=2 in unsorted data shown here.

▪ DB2 will start scanning the table for ID=2 and finds it as a second row in the heap but still keeps going.

▪ Since data is not sorted it does not know if another 2 exists down the list.

ID

3

2

5

1

7

3

9


Is SORT really bad ?

▪ Consider the sorted data here. To search for 2, I could use Binary Search.

▪ In binary search I get the size of the structure and divide it by 2. Here its 3.5, so I chose to go to 4th element, where ID 3 exists.

▪ Is 2 less than 3 ? Then eliminate all the elements below 3 since they are sorted and larger.

▪ In the upper half, again divide the size by and this time you find 2, and your search is done.


Scenarios of sorting

Valid Scenarios

▪ DB2 needs to sort data to build indexes.

▪ Needs to sort the Index RID’s to facilitate LIST PREFETCH.

▪ Facilitates Merge Scan join.

Invalid Scenarios

▪ I have an OCD, I can’t stand when the rows are unordered.

▪ Un-necessary SORT’s due to missing index.



TEST.TABLE1

▪ Tables in most RDBMS are heaps

of rows i.e. they are not ordered.

▪ Searching for the rows in a HEAP is

slow.

▪ Imagine searching for records with

ID=1.

▪ Db2 will scan the whole heap or

table even though it found the row

before reaching the end of the

table.

▪ That’s because data is not sorted!

▪ B-Tree Indexes are the magic data

structures that improves the search

efficiency.

ID NAME

5 Tom (RID1)

1 Anvith (RID2)

2 Cindy (RID3)

3 Raghu (RID4)

4 Jim (RID5)

7 Arvind (RID6)

6 Gene (RID7)

3 Raghu (RID8)

8 Scott (RID9)

9 Maria (RID10)

11 Greg (RID11)

10 Rajesh (RID12)

12 Sri (RID13)

13 Chris (RID14)


10Rajesh11Greg

RID(11,12)

5TomRID(1)

1Anvith2Cindy

RID (2,3)

12SriRID(13)

3RaghuRID(4)

3Raghu4Jim

RID(5,8)

6Gene7ArvindRID(7,6)

8Scott9Maria

RID(9,10)

13ChrisRID(14)

2Cindy 4Jim 9Maria 11Greg

7Arvind

▪ See how the leaf level is nicely sorted (True except for pureScale RANDOM).▪ Helps to speed up queries significantly.▪ Everything comes with a price, B-Trees slow down Writes.▪ Helps your order by queries, all DB2 needs to do is fetch the rows from the table in the order of Leaf level.


TEST

.TA

BLE

1

Unused/Redundant Indexes

https://www.raghu-on-tech.com/2019/06/30/how-to-find-redundant-unused-indexes/


https://www.raghu-on-tech.com/2019/06/30/how-to-find-redundant-unused-indexes/

SQL Tuning

▪ SQL is the language of RDBMS, but its not a complete programming language.

▪ It abstracts out lots of complexities of regular programming language, that’s why tuning might get challenging.

▪ Its declarative (what) but not imperative (how).

▪ Declarative: SELECT ID, NAME FROM TAB1 A, TAB2 B WHERE A.ID=B.ID AND A.NAME=‘Anvith’.

▪ Imperative: Write code to read file A, read file B, filter file A on 3rd

field for Name=‘Anvith’, build B-Tree indexes on file A and File B on ID and Name, then JOIN them on ID using a Nested Loop, SORT the results using advanced Radix sort then give me the results.


SQL Tuning(Contd…)

▪ DBA’s and developers write SQL (What).

▪ DB2 optimizer determines (how).

▪ Runstats help optimizer determine how to get data fast.

▪ Apart from Runstats, there are other ways to help optimizer by writing better SQL.

▪ The SQL you write will be rewritten by the optimizer during query rewrite phase of optimization.


SQL Tuning Tips

▪ Make sure statistics are current.

▪ Remove unnecessary joins and columns not needed by the application. Hibernate, Cognos and other tools are known to cause issues in this area.

▪ Try to avoid expression based predicates if possible.

▪ Look for objects that are causing the most Logical IO in the plan, try to fix them.

▪ Watch out for cardinality under/over estimations at various operators.

▪ Explore options such as moving predicates closer to the table, when you have a large SQL. I have seen them make a difference in the past.


SQL Tuning Tips(Contd…)

▪ If there is too much under/over estimation of cardinalities, try to fix them by making sure Runstats are current. You may use db2caem to troubleshoot the cardinality misestimations.

▪ If the cardinalities are messed up on a JOIN operator explore the use of Column Group Statistics and Statistical views.

▪ Explore options such as using CGTT to denormalize some of the joins to help the optimizer.

▪ If nothing appears to work, look at using OPTGUIDELINES.


SQL Tuning Tips(Contd…)

▪ More info on column groups could be found at below blog post.

https://www.raghu-on-tech.com/2019/05/02/what-are-column-group-statistics/


https://www.raghu-on-tech.com/2019/05/02/what-are-column-group-statistics/

How Does A Good Plan Look ?

▪ To me, a good plan should get the data you requested by accessing the least number of rows possible therefore reducing the logical IO.

▪ Always tune to reduce logical IO, no one can guarantee a data/index page to be in the buffer pools 100% of the time.

▪ When you reduce the IO, CPU and memory will follow. This approach gave me the most success while performance tuning.


Ways To Reduce Logical IO

▪ Table scans will cause a lot of IO. Especially when these scans reside on the INNER side of the nested loop Joins.

▪ Look out for any inefficient indexes in the plan.

▪ Ensure that the tables are being joined in the right order.

▪ Unnecessary sorting could cause IO.


Example Query

TAB1=10 ROWS (ID, COL1)



SELECT A.ID, B.ID, C.ID

FROM TAB1 A, TAB2 B, TAB3 C

WHERE A.ID=B.ID AND B.ID=C.ID AND C.ID=A.ID

AND C.ID=1;



ID2 COL2

1

2

3

4

5

6

7

8

9

10

ID3 COL3

1

2

3

4

5

6

7

8

9

10

NL JOIN

NL JOIN

ID1 COL1

1

2

3

4

5

6

7

8

9

10

Good Plan:

▪ Less IO

▪ Filtering Upfront

RETURN

TAB3 TAB2

TAB1

ID1 COL1

1

2

3

4

5

6

7

8

9

10

ID1 COL1

1

2

3

4

5

6

7

8

9

10

NL JOIN

ID1 COL1

1

2

3

4

5

6

7

8

9

10

NL JOIN

RETURNBad Plan:

▪ More IO

▪ Lot of Rows, through intermediary

operators!

TAB1 TAB2

TAB3

Good vs Bad Plan

▪ Good Plan:▪ Started by filtering on local predicate C.ID=1, which brings back 1 record with

1 ROW’s READ (really 3 IO’s for a 3 level index).

▪ Then Join that 1 row to TAB2, then that intermediary result set to the TAB3.

▪ A total of 3 ROWS READ and I got my result set back.

▪ Bad Plan:▪ started with joining tables TAB1 and TAB2 first, here 10 rows multiplied by 10

rows with a total of 100 Logical ROWS READ, then carry over those intermediary result set up the tree.

▪ Then join those 10 ROWS to TAB3, resulting in another 10 rows READ.

▪ A total of 110 ROWS READ, lots of IO, CPU and Memory


A Word About SLA’s

▪ Before designing you application service calls, think about what are the top 10 service requests and how they are going to manifest as SQL on the database.

▪ If a query is going to read 100,000 ROWS to return 100 ROWS, you may be able to meet your 1 second SLA with 99% hit ratio.

▪ If your bufferpools are not as hot and what will happen if your bufferpool hit ratio falls to 80% due to OS flushing out DB2 pages or what happens if you restart your server ? Is 10 second response time for the query acceptable ?



Why a DBA and a developer never gets engaged ?

Because the developer would not commit ☺

Documents

Db2 Performance Tips And Tricks - WordPress.com · Explain plan to your right is a good plan for the delete that’s executing in 5 ms. Notice how the staging table to the bottom