How to Influence the DB2 Query Optimizer Using Optimization Profiles

1

How to Influence the DB2 Query Optimizer Using Optimization Profiles

Tom [email protected]

Session: A1

Monday, May 8, 2006 • 10:20 a.m. – 11:30 a.m.

Platform: DB2 for Linux, UNIX, Windows

How to Influence the DB2 Query Optimizer Using Optimization GuidelinesAbstract: The DB2 optimizer is one of the most sophisticated in the industry; however, even after perfect performance tuning, it can occasionally "get it wrong." This talk teaches you how to use optimization guidelines to directly influence the optimizer's choice of access method, join method, and join order.Objective1: How to influence the optimizer's choice of access method, join method, and join orderObjective2: How to influence the query rewrite transformations applied by the optimizerObjective3: How to put optimization guidelines into effect without making application changesObjective4: How to determine which optimization guidelines were applied, andwhy others were notObjective5: Basic optimizer tuning tips

Note: This presentation includes information that will be presented in majority by a demonstration during the IDUG talk. The majority of the slides are included for your reference during and after the demonstration. They cover the demonstration material in more detail.

2

2

How to Influence the DB2 Query Optimizer Using Optimization Profiles

How to influence the optimizer's choice of access method, join method, and join order

How to influence the query rewrite transformations applied by the optimizer

How to put optimization guidelines into effect without making application changes

How to determine which optimization guidelines were applied, andwhy others were not

Some basic optimizer tuning tips

Hi. Thank you for reading the presentation notes for my talk. The majority of this talk will be presented as a demonstration, and the slides are available for your reference during and after the talk. They cover the demonstration material in more detail. I hope you enjoy the demonstration, the slides and the presentation notes. These notes are informal. I recommend reading the optimization profile documentation for more details about advanced syntax.

Have you ever wanted to tell DB2 exactly which index to use?Optimization Profiles help you control the rewrites and optimizations for your query.I will describe the power of DB2’s optimization profile.The different areas you can impact using profiles.Targeting a profile for a specific query.Targeting a profile to all queries executed while it is enabled.Cost-based guidelines (access and join methods and plan topology)Rewrite guidelines (subquery-to-join, inlist-to-join)Global guidelines (optimization class, reopt, MQTs, query degree)

I will describe how to write, enable, manage, and debug optimization profiles.Along the way, I will provide basic optimizer tuning tips which should help you avoid the need for profiles in all but the most desperate circumstances.

3

3

Motivation

The DB2 optimizer is responsible for choosing the best access plan for your query

It always does a fabulous job… usually

Your quiver of performance tools today:explain, runstats, optimization class, db and dbm configs, etc.

Goal of this talk:Add another, very powerful, tool to your quiver

Optimization Profiles

Plus a little something extra, watch for the

The running analogy in the talk is the quiver and the arrow. You have an existing set of performance tools to optimize your queries. These are arrows in your quiver. I hope to add optimization profiles as another tool, to be used only as a last resort.

For those reading the notes, I can let you in on the secret. The diamond represents XML. You’ll find out why soon!

4

4

The Big Picture – Optimization Profiles

You can influence the optimizations and plan for a query by providing a profile of your desired optimizations and plan elements to DB2.

Admissions:I like to have fun

ILIA Founder and Member(I Like Icons Association)

ILIA is the I Like Icons Association. As you can already see, I use icons everywhere in my slides. I think they’re fun and I hope you get a kick out of them.

5

5

Optimization Profile Overview

Used to directly influence the optimization of a DML statement

Disable transform of an IN list predicate to a joinUse index ISUPPKEY to access SUPPLIERS in the subqueryAlter join order of PARTS and PARTSUPPComposed using a simple XML specification

Can be used to impact queries without touching the application

Optimization profiles are stored in the databaseA profile can contain guidelines for one or more statementsIncoming queries dynamically mapped to stored profiles and guidelines

These are the main talking points of optimization profiles.The most important thing to remember is that you can impact your applications without altering them. The profiles you write are managed in the database, you can query and organize them.

A small rant: One of the main advances with SQL was the separation of the application from the schema, declarative query language. We worked hard to keep profiles out of the application. Profiles talk about your schema, your indexes, your data. Good application design means keeping these profiles out of your queries.

6

6

Optimization Profile Overview

Detailed diagnostics via explainQuery always executes, invalid or inapplicable profile registers warningsDiagnostics using db2exfmt and…Diagnostics using a SQL function

Should only be used after all other tuning options exploredAnalyze the query using db2exfmt or visual explainrunstats, indexes, explain, optimization class, db and dbm configs, etc.Warning, circumvents usual cost based optimization

Profiles are powerful things, and should be used with caution. You want to make sure you understand what’s wrong with your query before you twist the compiler’s arm.

7

7

Agenda

Big PictureAgendaOptimization profiles: first lookTypes of optimization profiles

Access (index…)JoinRewrites (subquery to join…)Global (opt class, reopt, MQTs)

Writing optimization profilesPutting optimization profiles into effectManaging optimization profilesDiagnostics

Lots of fun stuff to cover, but hopefully you can get the high level view fairly quickly.

8

8

Overview – How Can You Nudge The Optimizer

Understand what’s wrong with your queryUse today’s tools to tune DB2If those don’t work, use a profile as last resort

Write your profileCreate an XML file with a few optimization guidelinesIf needed, include information to tie a guideline to your query

Insert your profile into the user managed profile tableEnable your profile using a special register/bind optionUse explain and other tools to make sure you’ve fixed the problemMargarita!

Margarita: When things are so bad that you need a profile, then only a profile will do. And once you’ve overcome your major problem with a tiny bit of text, you’ll have enough time to relax at the pub. Enjoy your Margarita, it’s on the profiles team!

9

9

An Example QuerySELECT S.S_NAME, S.S_ADDRESS, S.S_PHONE, S.S_COMMENTFROM TPCD.PARTS, TPCD.SUPPLIERS S, TPCD.PARTSUPP PSWHERE P_PARTKEY = PS.PS_PARTKEY AND

S.S_SUPPKEY = PS.PS_SUPPKEY ANDP_SIZE IN (39,40,45,48) AND P_TYPE = ’BRASS’ AND S.S_NATION IN ('MOROCCO', 'SPAIN') AND PS.PS_SUPPLYCOST = (SELECT MIN(PS1.PS_SUPPLYCOST)

FROM TPCD.PARTSUPP PS1, TPCD.SUPPLIERS S1WHERE TPCD.PARTS.P_PARTKEY = PS1.PS_PARTKEYAND S1.S_SUPPKEY = PS1.PS_SUPPKEY AND S1.S_NATION = S.S_NATION)

ORDER BY S.S_NAME Index isn’t used

Should be first table

This will be my demonstration query for most of the presentation.We have some initial complaints:Why isn’t an index used on SUPPLIERS.S_NATION, and why isn’t it accessed first in the query block. If only I had that!

We can fix that right up, and much more, using profiles.

10

10

An Example QuerySELECT S.S_NAME, S.S_ADDRESS, S.S_PHONE, S.S_COMMENT

FROM TPCD.PARTS, TPCD.SUPPLIERS S, TPCD.PARTSUPP PS

WHERE P_PARTKEY = PS.PS_PARTKEY AND S.S_SUPPKEY = PS.PS_SUPPKEY AND

P_SIZE IN (39,40,45,48) AND P_TYPE = ’BRASS’ AND

S.S_NATION IN ('MOROCCO', 'SPAIN') AND

PS.PS_SUPPLYCOST = (SELECT MIN(PS1.PS_SUPPLYCOST)

FROM TPCD.PARTSUPP PS1, TPCD.SUPPLIERS S1

WHERE TPCD.PARTS.P_PARTKEY = PS1.PS_PARTKEY AND

S1.S_SUPPKEY = PS1.PS_SUPPKEY AND

S1.S_NATION = S.S_NATION)

ORDER BY S.S_NAME

Access the SUPPLIER table using the S_NATIONREGION index, and do it first

<IXSCAN TABLE='S' INDEX='S_NATIONREGION‘ FIRST='TRUE‘ />

Use an index oring to get start-stop keys on nation

<IXOR TABLE=‘S’ FIRST=‘TRUE’/>

Here I show pieces of optimization profile guidelines. Individual guidelines make up a profile. More on that later.

For now, the first bit of XML tells us all these things:-For the table Suppliers

-Use the index S_NATIONREGION-Make Suppliers the first table accessed in the query block

Uh oh, looking at the explain of the query after that first index shows that we don’t have start-stop keys to use on the index.

We can fix that if we do an index oring, and we still want it accessed first of course. We’ll subdivide the IN predicate, and each leg of the index or will use start stop keys. Wow! All with one line:

<IXOR TABLE=‘S’ FIRST=‘TRUE’/>

Also, here is our diamond back again, letting us know about XML. XML is great for profiles because you’ll be able to store and query your profiles using DB2.

11

11

Create The Profile XML File<?xml version="1.0" encoding="UTF-8"?><OPTPROFILE VERSION=‘9.0’>

<STMTPROFILE ID=‘TPCD Q1’><STMTKEY>SELECT S. _NAME, S.S_ADDRESS, S.S_PHONE,

S.S_COMMENT…</STMTKEY>

<OPTGUIDELINES><IXSCAN TABLE='S' INDEX='S_NATIONREGION‘ FIRST='TRUE‘/> </OPTGUIDELINES>

</STMTPROFILE>

</OPTPROFILE>

Which query should we change

What should we do to it?

This is what a complete optimization profile looks like when it modifies a single statement.

Most of this is template stuff we provide in the documentation.You drop in your query text and your small guideline, and we do the work of matching up your guideline to the exact query you wanted to impact.

There is great power stored in the extra XML, and I’ll describe more of it in presentation. For now lets start off with the simple common uses.

12

12

Insert and Use the ProfileIMPORT FROM profile_file OF DEL MODIFIED BY LOBSINFILE INSERT INTO SYSTOOLS.OPT_PROFILE

SET CURRENT OPTIMIZATION PROFILE TPCD.OPTPROFOr BIND option, CLI option, etc.

Re-execute query and check plan with explain

SELECT S.S_NAME, S.S_ADDRESS, S.S_PHONE, S.S_COMMENT










ORDER BY S.S_NAME

After you write a profile, you load it into a user-managed table in your database, and enable the profile.You can enable the profile for a particular bind, on the clp, in cli, sqlj, all of our DB2 interfaces.

Once you’ve enabled the profile, re-execute the query and make sure everything worked as you wanted. If you get a Warning 437 RC 13, that means something went wrong. I will describe diagnostics near the end of this talk.

13

13

Recap: Why Are Profiles Great?Get the plan you know you needUse them without changing the application!

In databaseCentrally managed, queriable, adapted as the database changesEasy to enable and disable as needed

FlexibleTarget specific queries or all queries runningChange access and join, change rewrites

High quality diagnostics, warnings, and you always get a plan

Margarita!

Why are profiles great?Everything, and then of course the margarita.

14

14

Anatomy of an Optimization Profile

Global Guidelines

Statement Specific Guidelines

Statement Identification

Rewrite Guidelines

Cost-based Guidelines



Rewrite Guidelines


<?xml version="1.0" encoding="UTF-8"?><OPTPROFILE VERSION=‘9.0’>

<OPTGUIDELINES><QRYOPT VALUE=‘3’/></OPTGUIDELINES>



<OPTGUIDELINES><IXSCAN TABLE='S' INDEX='S_NATIONREGION‘ FIRST='TRUE‘/>

</OPTGUIDELINES>

</STMTPROFILE></OPTPROFILE>

Statement Guides

Statement Key

Global Guides

This is an animated slide that shows that an optimization profile is composed of two major sections.First there is an optional Global Guidelines section, where you put guidelines that impact all queries that are run while the profile is enabled.Next, there are zero or more statement guidelines, each of which modify a single statement.DB2 matches a statement key you provide in the statement guideline with incoming queries in much the same way that the package cache maps compiled queries to incoming statements.

15

15

Types of Optimization GuidelinesCost-based Guidelines (zero or more per query)

Access methodsUse this index for this table, index-anding, access this table first, etc.

Join methodsJoin topology

Query Rewrite Guidelines (zero or more per query)Affect transformations applied to the original query to produce the optimized statementEg. Block in-list to join transformationEnable/disable query reoptimization, query opt level, degree

Global Guidelines (at most one per profile)Used to set Global optimization parametersEnable/disable query reoptimization, query opt level, degree

Optimization profiles include the following types of guidelines.Each is described in more detail in the presentation.Those in the blue square appear only in the statement level guidelines.Those with a star can appear in both the statement level and the Global guidelines.

16

16

First The Pieces, Then The Whole

In the next slides I will walk you through all the sections of the optimization profiles. On each slide I will show just the small piece of XML that corresponds to that topic. We will bring everything together near the end, but since each part of the profile is optional, your profile doesn’t need to be much more than any one of these fragments.

17

17

Cost-based Optimization GuidelinesAccess Requests

Applied for a specific querySpecify how to access a tableCorrespond to DB2 access methods Table reference identified using either exposed names in the original, or the optimized statement

IXSCAN, LPREFETCH, IXAND, IXOR, TBSCAN, ACCESS

These should look familiar (explain)

Access guidelines are probably the most useful profiles. We have an access guideline for each of our main access methods in DB2.

“Hey DB2, USE THIS INDEX”

18

18

SELECT S.S_NAME, S.S_ADDRESS, S.S_PHONE, S.S_COMMENTFROM TPCD.PARTS, TPCD.SUPPLIERS S, TPCD.PARTSUPP PSWHERE P_PARTKEY = PS.PS_PARTKEY AND S.S_SUPPKEY = PS.PS_SUPPKEY AND

P_SIZE = 39 AND P_TYPE = ’BRASS’ AND S.S_NATION IN ('MOROCCO', 'SPAIN') AND PS.PS_SUPPLYCOST =

(SELECT MIN(PS1.PS_SUPPLYCOST)FROM TPCD.PARTSUPP PS1, TPCD.SUPPLIERS S1WHERE TPCD.PARTS.P_PARTKEY = PS1.PS_PARTKEY ANDS1.S_SUPPKEY = PS1.PS_SUPPKEY AND S1.S_NATION = S.S_NATION)

ORDER BY S.S_NAME

You can tell us as much or as little in your profileUnspecified parts of the plan are determined by the optimizer as normal

Index Scan Guidelines

Access the PARTS table in the main subselect using the IPTKY index<IXSCAN TABLE=’TPCD.PARTS’ INDEX=‘IPTKY’/>

Access the PARTS table in the main subselect using an indexLet the optimizer choose the index, but access it first

<IXSCAN TABLE=’TPCD.PARTS’ FIRST=‘TRUE’/>

Yes, you can tell us which index to use(in a crunch)!

Take a break… relax… when the sky is fallingProfiles Come Calling!

And here’s the big party! It’s very easy to tell us to use a specific index, any index, and even tell us to access a table first in the query block.

If you leave something out like an index name, DB2 will pick the best.Your profile need not talk about the whole query. In fact we suggest you give very few guidelines, as few as are required to nudge DB2 into the right direction. Anything you do not talk about will be optimized as normal.

19

19

SELECT S.S_NAME, S.S_ADDRESS, S.S_PHONE, S.S_COMMENTFROM TPCD.PARTS, TPCD.SUPPLIERS S, TPCD.PARTSUPP PSWHERE P_PARTKEY = PS.PS_PARTKEY AND S.S_SUPPKEY = PS.PS_SUPPKEY AND

P_SIZE = 39 AND P_TYPE = ’BRASS’ AND S.S_NATION IN ('MOROCCO', 'SPAIN') AND PS.PS_SUPPLYCOST =

(SELECT MIN(PS1.PS_SUPPLYCOST)FROM TPCD.PARTSUPP PS1, TPCD.SUPPLIERS S1WHERE TPCD.PARTS.P_PARTKEY = PS1.PS_PARTKEY ANDS1.S_SUPPKEY = PS1.PS_SUPPKEY AND S1.S_NATION = S.S_NATION)

ORDER BY S.S_NAME

Access the PARTSUPP table in the main subselect using an index anding of PS_PARTKEY and PS_SUPPKEY

<IXAND TABLE='PS1'><INDEX IXNAME='PS_PARTKEY'/><INDEX IXNAME='PS_SUPPKEY'/>

</IXAND>

Why do you use PS1 here, instead of PARTSUPP?Use the exposed name. More on that at the end!

Index Anding Guidelines

We need to use the table name PS1 here because it is the name you’ve exposed for that table. The last slides describe the advanced topic of creating table references in complex situations such as views.

In general, use the exposed name if you’ve created one, else use the table name.

20

20

Access the PARTS table in the main subselect using a list prefetch on the IPTKY index

<LPRFETCH TABLE=’TPCD.PARTS’ INDEX=‘IPTKY’/></OPTGUIDELINES>

Access the PARTS table in any way you want, but do it first

<ACCESS TABLE=’TPCD.PARTS’ FIRST=‘TRUE’/>

List Prefetch Guidelines

Generic Access Guideline

Feeling pretty good yet?

The generic access guideline is very useful when building join queries.

21

21

Cost-based Optimization GuidelinesJoin Requests

Specify a join method and join orderCorrespond to DB2 join methods Composed of other access or join requests

NLJOIN, HSJOIN, MSJOIN, JOIN

Like access requests, we have a join guideline for each of our main join methods in DB2.

22

22




P_SIZE = 39 AND P_TYPE = ’BRASS’ AND







ORDER BY S.S_NAME

Join the PARTS and PARTSUPP tables in the main subselect using anested loop joinAccess the tables using the best index

<NLJOIN><IXSCAN TABLE=’TPCD.PARTS’/><IXSCAN TABLE=‘PS’/></NLJOIN>

Reverse the order<NLJOIN><IXSCAN TABLE=‘PS’/><IXSCAN TABLE=’TPCD.PARTS’/></NLJOIN>

Nested Loop Join Guidelines

Reversing the join order is great. In this case we specified that index accesses be used for the legs of the join, but we could have used an ACCESS guideline to give DB2 full freedom.

23

23











ORDER BY S.S_NAME

Join the SUPPLIERS and PARTSUPP tables in the subselect using a hash joinUse the best index for PARTSUPP, and the best access for SUPPLIERS.

<HSJN><ACCESS TABLE=‘S1’/><IXSCAN TABLE=‘PS1’/></HSJOIN>

Hash Join Guidelines

You may not always be able to get a hash join. For example, your query optimization class may disallow has joins, you may not have identical types on the sides of your predicate, etc. You can read more about hash join in the DB2 performance guides.

If request an impossible join, or in general if a guideline is not applied, DB2 will return a warning 437 and an RC 13. It will also log detailed diagnostics about your warning into a diagnostics table. You can use explain or SQL to read these diagnostics. Details are found later in the presentation.

24

24











ORDER BY S.S_NAME

Join the PARTS and PARTSUPP tables in the main subselect using anested loop joinJoin that result with the SUPPLIERS table using a merge joinChoose the best index for all table accesses

<MSJOIN><NLJOIN><IXSCAN TABLE=‘TPCD.PARTS’/><IXSCAN TABLE=‘PS’/></NLJOIN><IXSCAN TABLE=‘S’/>

</MSJOIN>

Merge Join Guidelines

LOOKIN’ GOOD!

Even complex graphs can be expressed very naturally in our XML syntax.Joins contain other accesses and joins.

25

25

Types of Optimization GuidelinesCost-based Guidelines



Query Rewrite GuidelinesAffect transformations applied to the original query to produce the optimized statementE.g. Block In-list to join transformationEnable/disable query reoptimization, query opt level, degree

Global GuidelinesUsed to set Global optimization parametersEnable/disable query reoptimization, query opt level, degree

Now we will look into the Query Rewrite Guidelines

26

26

Query Rewrite GuidelinesEnable or disable rewrite rules applied to a specific query

In-list to join (INLIST2JOIN)Transforms the constant list in an IN predicate to a table expressionCan enable/disable for a statement, or target a specific predicate

Subquery to join (SUBQ2JOIN)Applies to subqueries quantified by EXISTS, IN, =SOME, =ANY, <>SOME, <>ANY

Not-Exists to anti-join (NOTEX2AJ)Applies to subqueries quantified by NOT EXISTS

Not-In subquery to anti-join (NOTIN2AJ)Applies to subquery predicates quantified by NOT IN

Query rewrite guidelines help you alter the rewrite rules that transform your query. These can have a tremendous impact on your query, sometimes making other profiles inapplicable.

27

27

In-List to JoinSELECT S.S_NAME, S.S_ADDRESS, S.S_PHONE, S.S_COMMENT










ORDER BY S.S_NAME

Transform the list of constants in the predicate P_SIZE (39, 40, 45, 48) to a table expression

<INLIST2JOIN TABLE=‘P’/>

Ambiguous if multiple IN predicates on a target table, so add the COLUMN quantifier

<INLIST2JOIN TABLE=‘P’ COLUMN=‘P_SIZE’/>

In-list to join leaves the in-list as a predicate or transforms it to a join with a table expression. It can take a COLUMN attribute to target it to a specific predicate.

28

28

In-List to Join - DisableSELECT S.S_NAME, S.S_ADDRESS, S.S_PHONE, S.S_COMMENT










ORDER BY S.S_NAME

Disable the INLIST2JOIN transformation on P_SIZE

<INLIST2JOIN TABLE=‘P’ COLUMN=‘P_SIZE’ OPTION=‘DISABLE’/>

All the rewrite rules have an optional OPTION attributeOPTION=‘ENABLE’ is the defaultOPTION=‘DISABLE’ disables the rule, but may not work in all cases

Each of the rewrite guidelines can take an optional OPTION attribute, which allows you to DISABLE that transformation. ENABLE is the default if no OPTION is specified.

29

29

Subquery to Join

Perform subquery to Join transformations in the query if possibleCannot be targeted to a specific subquery

<SUBQ2JOIN />

Disable any subquery to join transformations in the query

<SUBQ2JOIN OPTION=‘DISABLE’/>

Identical to subquery to join:Not-Exists to anti-join (NOTEX2AJ)Not-In subquery to anti-join (NOTIN2AJ)

Due to space limitations, I don’t give the details for Not-Exists to anti-join and Not-In to anti-join. They are similar to subquery to join and can be found in our documentation.

30

30

Rewrite and Global GuidelinesWhen found at the statement level, applied only to the matched statementOverrides the values at the Global level, special register, bind option, etc.

REOPTWhen to reoptimize a query with parameters

Query Optimization Level0,1,3,5,9

Degree Query parallelism

Global Guidelines



Rewrite Guidelines




Rewrite Guidelines


So far I’ve shown you guidelines that only apply to individual statements, in the statement guidelines.These next few can be placed in a statement specific guideline, OR in the Global guidelines section to apply to all queries that are executed while the profile is active (all queries in a package if you bind with a profile, for example). I describe precedence rules later in the talk.

31

31

REOPTOverrides the REOPT bind

REOPT NONENo reoptimization of the query is done, optimizer chooses default values.

<REOPT VALUE=‘NONE’ />

REOPT ONCEThe execution plan is picked at the first OPEN for a query. Useful if initial values are representative of following executions of the query.

<REOPT VALUE=‘ONCE’ />

REOPT ALWAYSReoptimize the query on every execution. Chosen plans are optimal, at cost of compilation time and package cache activity.

<REOPT VALUE=‘ALWAYS’ />

REOPT is great for queries with parameter markers/host variables. Look into our documentation to find our more.

32

32

Query OptimizationOverrides the Query Optimization class (dft_queryopt)

You can set the query optimization class PER QUERY using profiles

QRYOPT 0Minimal query optimization

QRYOPT 1QRYOPT 3QRYOPT 5

Default. Significant optimization with limiting heuristicsQRYOPT 7QRYOPT 9

Maximal query optimization

<QRYOPT VALUE=’5’/>

0 - minimal query optimization. 1 - roughly comparable to DB2 Version 1. 2 - slight optimization.3 - moderate query optimization. 5 - significant query optimization with heuristics to limit the effort expended on selecting an access plan. This is the default. 7 - significant query optimization. 9 - maximal query optimization

33

33

Default DegreeOverrides the Current Degree (dft_degree)

ANY, -1 – Degree of query intra-partition parallelism determined by DB2<DEGREE VALUE=’ANY’/>

0 – Parallelism determined by value of CURRENT DEGREE register<DEGREE VALUE=’0’/>

1 – No intra-partition parallelism for this query<DEGREE VALUE=’1’/>

2 – 32767 – Amount of intra-partition parallelism<DEGREE VALUE=’10’/>

Please refer to the infocenter documentation to learn more about the degree parameter.

34

34

Types of Optimization GuidelinesCost-based Guidelines



Query Rewrite GuidelinesAffect transformations applied to the original query to produce the optimized statementE.g. Block In-list to join transformationEnable/disable query reoptimization, query opt level, degree

Global GuidelinesUsed to set Global optimization parametersEnable/disable query reoptimization, query opt level, degree

We will now look into global guidelines that impact all queries that are executed while the profile is enabled.

35

35

Global Guidelines

Up to now, all guidelines impacted a specific statementGlobal guidelines are applied to all queries issued while profile is in effect

Query Optimization LevelDegree REOPT

MQT Optimization ChoicesInclude or exclude MQTs from query matching

Global Guidelines



Rewrite Guidelines




Rewrite Guidelines


We have only a few global guidelines that cannot be applied to an individual statement. The one I will discuss is MQT optimization choices. Another lets you determine on which computational partition group your federated query executes.

36

36

MQT Optimization Choices GuidelineEnable or disable the use of MQTs in your queryProvide a list of MQTs considered for the query

Useful if you experience compile time problems with MQTsUsing the MQT is still a cost based decision, this adds/removes them from the pool

Disable all MQTs<MQTOPT OPTION=‘DISABLE’/>

Consider only a set of MQTs<MQT NAME=‘TPCD.PARTMQT’/><MQT NAME=‘COLLEGE.STUDENTS’/>

MQT Optimization choices can conflict (you can disable and then also list some MQTs). In these situations we try to take the most restrictive directive (disable) but in general this is not supported and you should not write such guidelines.

37

37

Statement Key, Statement What?<?xml version="1.0" encoding="UTF-8"?><OPTPROFILE VERSION=‘9.0’><OPTGUIDELINES><QRYOPT VALUE=‘3’/></OPTGUIDELINES>



<OPTGUIDELINES><IXSCAN TABLE=’TPCD.PARTS’ INDEX=‘IPTKY’/><INLIST2JOIN TABLE=‘TPCD.PARTS’ COLUMN=‘P_SIZE’ OPTION=‘DISABLE’/>

</OPTGUIDELINES></STMTPROFILE>

</OPTPROFILE>

An optimization profile can contain statement profiles for multiple statements. We use the key to map a statement level profile to a specific query.Normally, only the statement is required, and you can grab it from your application or from the Original Query section in the db2exfmt.

38

38

Statement Key

Each statement level profile must identify a corresponding queryusing a statement key

Statement Text (required)White space normalization

Default Schema (optional)Useful if single part tables are referenced in the query

Function Path (optional)Useful if functions are referenced in the query

Analogy is to the plan cache, we make sure we get the right statement matched up. If you don’t provide a value, then we do not use it as a criteria for matching. If you provide a value for schema or function path, then it must match the incoming statement’s schema or function path exactly.

39

39

Statement Key<STMTPROFILE ID=‘TPCD SIMPLE QUERY’>

<STMTKEY SCHEMA=‘TPCD’FUNCTIONPATH=‘SYSIBM,SYSFUN,SYSPROC,SYSIBMADM,TE’>

<![CDATA[SELECT C_NAME FROM CUSTOMERS]]></STMTKEY>

<OPTGUIDELINES><IXSCAN TABLE=’TPCD.PARTS’ INDEX=‘IPTKY’/><INLIST2JOIN TABLE=‘TPCD.PARTS’ COLUMN=‘P_SIZE’ OPTION=‘DISABLE’/>

</OPTGUIDELINES></STMTPROFILE>

Only queries matching exactly all the statement key information provided will match the statement profile

Use CDATA around the statement text to avoid XML-type parsing of comparisons ‘<‘ etc.

Here is an example of a statement key.

40

40

Create the Profile Tables

Profile Table DDLCREATE TABLE SYSTOOLS.OPT_PROFILE (SCHEMA VARCHAR(128) NOT NULL, NAME VARCHAR(128) NOT NULL, PROFILE BLOB (2M) NOT NULL, PRIMARY KEY ( SCHEMA, NAME ) );

SCHEMA - Specifies the schema qualifier of the optimization profile. NAME - Specifies the base-name of the optimization profile. PROFILE - The XML document defining the optimization profile.

DB2 documentation contains provides all DDL I use in the presentation. This is table is not created automatically, as most users do not use profiles.

41

41

Insert Into the Profile Table

Any way you want to load it is fine!I will show you how to use the IMPORT utility

Create a file on the clientprofile_file.load“TPCD”,”OPTPROF”,”tpcd_profile.prof”

Create the profile on the client named tpcd_profile.prof

IMPORT FROM profile_file.load OF DEL MODIFIED BY LOBSINFILE INSERT INTO SYSTOOLS.OPT_PROFILE

Check documentation for more IMPORT options

There are so many ways of inserting a LOB. My favorite is with a table function, but I chose to describe IMPORT here because it works from client to server.

42

42

Enabling DB2 for Profiles

Enable the DB2_OPTPROFILE registry variable

Profile related commands are not recognized without this set

db2set DB2_OPTPROFILE=YESdb2set DB2_OPTPROFILE=NO

DB2 restart required

Value listed via db2set -all

Often times this step is forgotten with profiles. DB2 will complain about the syntax of your other profile commands unless you tell it that profiles are enabled.

43

43

Enabling a Specific Profile

You may have many profiles in your systools.opt_profile tableOne profile enabled at a time

BIND option: OPTPROFILE SCHEMA.NAMESets the profile for all static SQL in the packageSets the default for all dynamic queries

Special registerSET CURRENT OPTIMIZATION PROFILE SCHEMA.NAMESets the profile for dynamic queries following it

“String”, Host Variable, or NULL (no profile in use, overrides bind option)

Also CLI (DB2_OPTPROF), SQLJ (OPTPROF), etc.

When you bind an application with a profile, it is the profile for all static queries. Also, if you have dynamic queries in your application, it is the default for those. You can specify a profile in the application using the registry variable, and that will change the default for following dynamic queries.

You can disable profiles by setting the current profile to NULL.

All DB2 interfaces can either explicitly support profile options, such as in the cli.inifile, or will pass through profile settings as needed.

44

44

How Can Things Go Wrong?Reference a nonexistent table or indexSyntax error in your guideline

Guidelines in the profile may become inapplicableRunstats has caused the query plan to change drasticallyRewrites have caused the query plan to change drastically

When DB2 cannot apply a guideline it returns a warning 437 reason code 13Messages written to a new diagnostic tables EXPLAIN_DIAGNOSTIC

DB2 will apply as much of your profile as possible

Sometimes things go wrong, or don’t go as planned. Perhaps you wrote a bad guideline, or perhaps the query has changed since you last explained it.

45

45

W 437 RC 13 example

S_NAME S_ADDRESS S_PHONE S_COMMENT ------------------------- ------------------------- --------------- -----------------SQL0437W Performance of this complex query may be sub-optimal. Reason code: "13". SQLSTATE=01602

Supplier#00004 pQskYdiqymGCKMpdh2rV3KLe2 25-8267647199 azedagedamwhofor

1 record(s) selected with 1 warning messages printed.

This is an example of a profile gone wrong on the DB2 CLP. Note the 437 warning still allows results to be returned. Profiles will always be associated with reason code 13. Please check the infocenter for other reason codes associated with SQL0437.

46

46

Warning, Now What?Explain the query with the warning

Human readable details in a new section!

Use a table function provided in DB2SELECT MSG FROM

TABLE( EXPLAIN_GET_MSGS('TPCDUSER', /*EXPLAIN_REQUESTER*/ TIMESTAMP( '2006-01-01-20.42.54.977268' ), /*EXPLAIN_TIME */ 'SQLC2E03', /*PACKAGE NAME */ CAST( NULL AS VARCHAR(128) ), /*SOURCE_SCHEMA */ CAST( NULL AS VARCHAR(64) ), /*SOURCE_VERSION */ CHAR( 'P' ), /*EXPLAIN_LEVEL */ 1, /*STMTNO */ CAST( NULL AS INTEGER ), /*SECTNO */ 'en_US') ) WARNINGS /*LOCALE */

ORDER BY DIAGNOSTIC_ID

Query EXPLAIN_DIAGNOSTIC tablesDiagnostics available via SQL to your applications!The profile and statement guideline used inserted as arguments to the RETURN operator

This table function is provided in our documentation.It is great to be able to access human-readable diagnostic messages using SQL. However, I normally use the db2exfmt explain tool.

47

47

Flushing?

Once used, profiles are stored in an internal cacheWhen updating or deleting a profile, the profile must be flushedfrom the cache

FLUSH OPTIMIZATION PROFILE CACHE SCHEMA.NAMEFLUSH OPTIMIZATION PROFILE CACHE ALL

Invalidates cached DML statementsPackages using profiles must be explicitly rebound, not automatic

Remember to flush? We’ll do it for you!DB2: Helping you avoid domestic disputes

Quick tip for note readers, don’t worry about flushing! I have a trigger that does it all for you in the next slide. It’s also in our documentation.

48

48

Flushing the Easy WayCREATE PROCEDURE SYSTOOLS.OPT_FLUSH_CACHE( IN SCHEMA VARCHAR(128),

IN NAME VARCHAR(128) ) LANGUAGE SQL MODIFIES SQL DATA BEGIN ATOMIC -- FLUSH stmt (33) + quoted schema (130) + dot (1) + quoted name (130) = 294 DECLARE FSTMT VARCHAR(294) DEFAULT 'FLUSH OPTIMIZATION PROFILE CACHE '; ---- Setup error handler to ignore error in case DB2_OPTPROFILE is not set DECLARE CONTINUE HANDLER FOR SQLSTATE VALUE '42601' BEGIN END; --IF NAME IS NOT NULL THEN IF SCHEMA IS NOT NULL THEN SET FSTMT = FSTMT || '"' || SCHEMA || '".'; --

END IF; --

SET FSTMT = FSTMT || '"' || NAME || '"'; --EXECUTE IMMEDIATE FSTMT; --

END IF; --END; CREATE TRIGGER SYSTOOLS.OPT_PROFILE_UTRIG AFTER UPDATE ON SYSTOOLS.OPT_PROFILE REFERENCING OLD AS O FOR EACH ROW

Here it is! Using this trigger you can forget there is any such thing as flushing the profile cache.

49

49

Opt guidelines can refer to tables, views, alias’s, table expressions using either

their exposed names in the original statement (TABLE attribute)their corresponding unique correlation names in the optimized statement (TABID attribute)

SELECT S.S_NAME, S.S_ADDRESS, S.S_PHONE, S.S_COMMENTFROM SM_TPCD.PARTS, SM_TPCD.SUPPLIERS S, SM_TPCD.PARTSUPP PSWHERE P_PARTKEY = PS.PS_PARTKEY AND S.S_SUPPKEY = PS.PS_SUPPKEY AND

P.P_SIZE = 39 AND P.P_TYPE = ’BRASS’ AND S.S_NATION IN ('MOROCCO', 'SPAIN') ANDPS.PS_SUPPLYCOST = (SELECT MIN(PS1.PS_SUPPLYCOST)

FROM SM_TPCD.PARTSUPP PS1, SM_TPCD.SUPPLIERS S1WHERE P.P_PARTKEY = PS1.PS_PARTKEY AND

S1.S_SUPPKEY = PS1.PS_SUPPKEY AND S1.S_NATION = S.S_NATION)

<OPTGUIDELINES> <HSJOIN> <TBSCAN TABLE='S1’/> <IXSCAN TABID=’Q2’ /> </HSJOIN><IXSCAN TABLE=‘SM_TPCD.PARTS’ />

</OPTGUIDELINES>

Optimized statement:

SELECT Q6.S_NAME AS "S_NAME", Q6.S_ADDRESS AS "S_ADDRESS", Q6.S_PHONE AS "S_PHONE", Q6.S_COMMENT AS "S_COMMENT"

FROM (SELECT MIN(Q4.$C0) FROM (SELECT Q2.PS_SUPPLYCOST

FROM SM_TPCD. SUPPLIERS AS Q1, SM_TPCD.PARTSUPP AS Q2WHERE Q1.S_NATION ='MOROCCO'AND Q1.S_SUPPKEY = Q2.PS_SUPPKEY AND Q7.P_PARTKEY = Q2.PS_PARTKEY) AS Q3

) AS Q4, SM_TPCD.PARTSUPP AS Q5, SM_TPCD. SUPPLIERS AS Q6, SM_TPCD.PARTS AS Q7WHERE P_SIZE = 39 AND Q5.PS_SUPPLYCOST = Q4.$C0 AND Q6.S_NATION IN ('MOROCCO', 'SPAIN') AND

Q7.P_TYPE = 'BRASS' AND Q6.S_SUPPKEY = Q5.PS_SUPPKEY AND Q7.P_PARTKEY = Q5.PS_PARTKEY

One important detail of optimization profiles is the creation of table references to link your guideline to a portion of your query. I will discuss these here through examples, but please also look at the documentation to learn the full story.

50

50

References Through ViewsCan use a sequence of exposed names to qualify references to tables in views

Can refer to tables indirectly referenced by views if the reference is not ambiguous

CREATE VIEW “Hamid".V1 as (SELECT * FROM EMPLOYEE WHERE SALARY > 50,000)CREATE VIEW “Laura".V2 AS (SELECT * FROM "Rick".V1 WHERE DEPTNO IN (‘52’, ‘53’,’54’)

SELECT *FROM “Laura".V2 AWHERE SALARY > (SELECT AVG(SALARY) FROM EMPLOYEE)

<OPTGUIDELINES><IXSCAN TABLE='A/“Hamid".V1/EMPLOYEE'/></OPTGUIDELINES>

CREATE VIEW "Rick".V1 as (SELECT * FROM EMPLOYEE E WHERE SALARY > 50,000)

CREATE VIEW "Gustavo".V2 AS (SELECT * FROM "Rick".V1 WHERE DEPTNO IN (‘52’, ‘53’,’54’)

SELECT *FROM "Gustavo".V2 AWHERE SALARY > (SELECT AVG(SALARY) FROM EMPLOYEE)

<OPTGUIDELINES><IXSCAN TABLE='E‘/><IXSCAN TABLE=‘EMPLOYEE’ /></OPTGUIDELINES>

References through views.

51

51

Summary

You can use Optimization Profiles to impact the DB2 Linux Unix and Windows optimizer’s rewrites and plans

You don’t need to change your applications!Optimization Profiles can influence many queries or be targeted at specific queriesOptimization Profiles are safely managed in the database

You have the power

I hope you enjoyed this presentation. For advanced topics I urge you to read our great documentation.

Profiles are a simple way to impact your query plans. We’ve worked hard to let you alter applications with modifying the application. Also, we’ve tried to support the best application design by separating the profiles from the queries themselves, and letting you manage and query the profiles as the extremely important objects they are.

52

52

Tom EliazIBM

[email protected]

Look for them in the Viper release!

Session: A1How to Influence the DB2 Query Optimizer Using Optimization Profiles

Once again I hope you enjoyed this presentation and I look forward to hearing your feedback. Check them out in DB2 Viper for Linux, Unix, and Windows!

Documents

How to Influence the DB2 Query Optimizer Using Optimization Profiles