SQL Plan Directives Explained

Mauro Pagano

2

Mauro Pagano• Consultant, working with both DBA and Devs• Oracle Enkitec Accenture• DBPerf and SQL Tuning• Training, Workshops, OUG• Free Tools (SQLd360, TUNAs360, Pathfinder)

3

Some background• CBO makes mistakes– Complex code with challenging job – Based on • Statistical formula, not perfect in 100% cases• Partial knowledge of the data (stats)

• Mistakes can translate into poor plans• Poor plans usually lead to poor performance

4

How to avoid mistakes?• Improve quality of the model (Oracle)– Hard, lots of corner cases for CBO to handle

• Improve quality of stats (kind of you)– Hard, need knowledge of data and queries

• Reactively, learn from them to avoid next time– Allow to adapt to specific situation– Requires mistake to be made first though

5

Cardinality Feedback• Oracle attempt of making CBO learning from its own mistakes

– Introduced in 11.2 (11.1 as one-off)– Enhanced in 12.1 to deal with joins (but disabled in 12.2)

• Might take a bit to learn– Kind of iterative approach, up to 5 “refined” attempts

• Lessons learned as OPT_ESTIMATE hints (like STA), very specific– Not persisted to dictionary (same mistakes over time)

• Few initial bugs gave it a bad reputation– SQL runs fine the first time, run again and runs slow!– Supposed to be transparent, not good when gives troubles – Some people got burned, decided to turn off system-wise

6

Table TAB1drop table tab1;create table tab1 (n1 number, n2 number, n3 number);-- the data is strongly correlatedinsert into tab1 select mod(rownum, 100), mod(rownum, 100), mod(rownum, 100) from dual connect by rownum <= 100000;commit;exec dbms_stats.gather_table_stats(user,'TAB1');

7

Cardinality Feedbackselect count(*) from tab1 where n1 = 1 and n2 = 1 and n3 = 1----------------------------------------------------------------------| Id | Operation | Name | Starts | E-Rows | A-Rows |----------------------------------------------------------------------| 0 | SELECT STATEMENT | | 1 | | 1 || 1 | SORT AGGREGATE | | 1 | 1 | 1 ||* 2 | TABLE ACCESS STORAGE FULL| TAB1 | 1 | 1 | 1000 |----------------------------------------------------------------------

----------------------------------------------------------------------| Id | Operation | Name | Starts | E-Rows | A-Rows |----------------------------------------------------------------------| 0 | SELECT STATEMENT | | 1 | | 1 || 1 | SORT AGGREGATE | | 1 | 1 | 1 ||* 2 | TABLE ACCESS STORAGE FULL| TAB1 | 1 | 1000 | 1000 |----------------------------------------------------------------------Note----- - statistics feedback used for this statement

OPT_ESTIMATE (@"SEL$1" TABLE "TAB1"@"SEL$1" ROWS=1000.000000 )

P(A and B and C) = P(A) * P(B) * P(C)

8

SQL Plan Directives • New feature introduced in 12.1 – Several changes introduced in 12.2

• Designed to be transparent – SPDs are CBO generated– SPDs are leveraged by CBO once in place

• Enabled by default in 12.1• Disabled by default in 12.2

9

What’s the goal?• Help CBO make better estimations• Not SQL-specific– Can be re-used by similar SQLs

• No specific adjustments like CFB• Keep track of ”risky paths” • Determine when to leverage other features– Dynamic Statistics– Instruct DBMS_STATS to create Column Groups

10

What does SPD track?• Cardinality misestimates (usually underestimates)• “Risky paths” are

– Data correlation on multi-column filter predicates– Uncommon join correlation – Data correlation on columns in GROUP BY

• Potentially other paths can be tracked– For example, cardinality estimation at query block level

• Many TYPEs could be supported– As of today, only existing TYPE is DYNAMIC_SAMPLING (*)

11

How does SPD track them?• Entry in the data dictionary

– opt_directive$, opt_finding_obj$– Externalized as DBA views

• Records table / columns involved– Different SQLs on same objects can use same SPD

• Info read at hard parse and consumed accordingly– For example, triggering dynamic sampling

• Used as warning “watch out when estimating X”

12

SPD creation exampleselect count(*) from tab1 where n1 = 1 and n2 = 1 and n3 = 1;

(from 10053)SINGLE TABLE ACCESS PATH Single Table Cardinality Estimation for TAB1[TAB1] Table: TAB1 Alias: TAB1 Card: Original: 100000.000000 Rounded: 1 Computed: 0.100000 Non Adjusted: 0.10…SPD: Generating finding id: type = 1, reason = 1, objcnt = 1, obItr = 0, objid = 135446, objtyp = 1, vecsize = 4, colvec = [1, 2, 3, ], fid = 29…48 SPD: Inserted felem, fid=29…48, ftype = 1, freason = 1, dtype = 0, dstate = 0, dflag = 0,…

Mistake! We know it’s 1K rows

13

SPD creation exampleselect directive_id, type, reason, notes from dba_sql_plan_directives where directive_id in (select directive_id from dba_sql_plan_dir_objects where owner = sys_context('userenv', 'session_user') and object_name = 'TAB1');

DIRECTIVE_ID TYPE REASON -------------------------- ---------------- ------------------------------------ 1149064327055029475 DYNAMIC_SAMPLING SINGLE TABLE CARDINALITY MISESTIMATE

NOTES -----------------------------------------------------<spd_note> <internal_state>NEW</internal_state> <redundant>NO</redundant> <spd_text>{EC(MPAGANO.TAB1)[N1, N2, N3]}</spd_text></spd_note>

14

SPD creation exampleselect * from dba_sql_plan_dir_objects where owner = sys_context('userenv', 'session_user') and object_name = 'TAB1';

SUBOBJECT_NAME OBJECT ---------------- ------ N1 COLUMNN2 COLUMNN3 COLUMN TABLE NOTES is at the TABLE level

NOTES --------------------------------------------------------------------------<obj_note> <equality_predicates_only>YES</equality_predicates_only> <simple_column_predicates_only>YES</simple_column_predicates_only> <index_access_by_join_predicates>NO</index_access_by_join_predicates> <filter_on_joining_object>NO</filter_on_joining_object> </obj_note>

15

SPD creation (another) example DIRECTIVE_ID TYPE REASON --------------------- ---------------- ------------------------------------ 7426975260533728013 DYNAMIC_SAMPLING JOIN CARDINALITY MISESTIMATE

NOTES---------------------------------------------------------------------------------<spd_note> <internal_state>NEW</internal_state> <redundant>NO</redundant> <spd_text>{(SYS.OPT_DIRECTIVE$) - (SYS.OPT_FINDING_OBJ$)}</spd_text> </spd_note>

DIRECTIVE_ID OBJECT_NAME OBJECT NOTES--------------------- ---------------- ------ ------------------------------------ 7426975260533728013 OPT_DIRECTIVE$ TABLE <obj_note> EVERYTHING NO </obj_note> 7426975260533728013 OPT_FINDING_OBJ$ TABLE <obj_note> EVERYTHING NO </obj_note>

16

SPD creation example• SPD_TEXT reports the “risky path”

– Specific format within { }, built on the fly– One or more tables – Columns included for single table– Format includes flag on predicate (where applicable)

• E = Equality predicates• C = Simple column predicates• J = Index access by join predicates• F = Filter on joining object

17

SPD creation example• INTERNAL_STATE shows where in the lifecycle SPD is

– NEW – SPD just created (flushed to disk), not used by SQLs yet– MISSING_STATS – CBO used to help estimations, likely used ADS– HAS_STATS – Stats (likely Column Group) helped CBO, SPD “off”– PERMANENT – Stats not enough, SPD will stay “on” (ADS)

• First two states are transitory• Last two states are definitive

– HAS_STATS -> PERMANENT if stats don’t help “enough”– SPD goes trough HAS_STATS before PERMANENT

18

How does CBO use SPD?• During hard parse applicable SPDs are looked at

– Search done at multiple levels– Objects looked up by obj#

• Many SPDs are not “current” anymore– Obsoleted / Superseded

• The valid ones are used– Action triggered at the specific level– As of today only CBO action is Dynamic Sampling (ADS)

19

SPD usage example – single tableselect count(*) from tab1 where n1 = 1 and n2 = 1 and n3 = 1;

Query Block SEL$1 (#0)Applicable DS directives: dirid = 16004857199080262683, state = 1, flags = 1, loc = 1 {EC(135484)[1, 2, 3]}…SINGLE TABLE ACCESS PATH Single Table Cardinality Estimation for TAB1[TAB1] SPD: Directive valid: dirid = 16004857199080262683, state = 1, flags = 1, loc = 1 {EC(135484)[1, 2, 3]} … Table: TAB1 Alias: TAB1 Card: Original: 100000.000000 >> Single Tab Card adjusted from 0.100000 to 1000.000000 due to adaptive dynamic sampling Rounded: 1000 Computed: 1000.000000 Non Adjusted: 0.100000

20

SPD usage example – single tableselect count(*) from tab1 where n1 = 1 and n2 = 1 and n3 = 1;

(from 10046 during hard parse)PARSING IN CURSOR #140175694530408 …SELECT /* DS_SVC */ /*+ … result_cache(snapshot=3600) */ SUM(C1) FROM (SELECT /*+ qb_name("innerQuery") NO_INDEX_FFS( "TAB1")*/ 1 AS C1 FROM "TAB1" "TAB1" WHERE ("TAB1"."N1"=1) AND ("TAB1"."N2"=1) AND ("TAB1"."N3"=1) ) InnerQuery

STAT #140175694530408 id=1 cnt=1 op='RESULT CACHE 145cmpkvg18at78z0dz8ptfwfw STAT #140175694530408 id=2 cnt=1 op='SORT AGGREGATE STAT #140175694530408 id=3 cnt=1000 op='TABLE ACCESS FULL TAB1

21

Use of SPD – single table conclusions• Allows CBO to get better estimation on the fly• Cool but not the best approach in the long run

– ADS even if fast still takes some time (overhead)– Longer parses cause most of the troubles in real-life

• Same result achieved with Column Group (often)• Real goal is

– Put a band-aid until Column Group in place– Instruct DBMS_STATS to collect Column Groups– Verify if Column Group helped, if not keep SPD active

22

SPD usage example – joinselect … from dba_sql_plan_directives a, dba_sql_plan_dir_objects b where a.directive_id = b.directive_id and a.created >= sysdate-1/24 order by a.create

Query Block SEL$AF75E632 (#0)Applicable DS directives: dirid = 10275726999217578058,state = 1,flags = 1,loc = 1{(612)[7]} dirid = 13360472976608779416,state = 1,flags = 1,loc = 1{(612)[]} dirid = 7426975260533728013,state = 1,flags = 1,loc = 2{(612)[]; (608)[]} dirid = 4318109285801377208,state = 1,flags = 1,loc = 1{(608)[]} dirid = 1215330379436061558,state = 1,flags = 1,loc = 2{(608)[]; (22)[]} dirid = 17607835686132426105,state = 5,flags = 1,loc = 1{C(22)[1]} dirid = 17030158486626056340,state = 1,flags = 1,loc = 2{(612)[];(608)[]; (22)[]}

23

SPD usage example – join(from 10053) SPD: Directive valid: dirid = 7426975260533728013,…,loc = 2 {(612)[]; (608)[]} …Join Card: 13.000000 = outer (9.000000) * inner (13.000000) * sel (0.111111)>> Join Card adjusted from 13.000000 to 16.000000 due to adaptive dynamic sampling, prelen=2Adjusted Join Cards: adjRatio=1.230769 cardHjSmj=16.000000 …Join Card - Rounded: 16 Computed: 16.000000

(from 10046)PARSING IN CURSOR #139704599064112 SELECT /* DS_SVC */ /*+ … result_cache(snapshot=3600) */ SUM(C1) FROM (SELECT /*+ qb_name("innerQuery") NO_INDEX_FFS( "F#0") */ 1 AS C1 FROM "SYS"."OPT_FINDING$" "F#0", "SYS"."OPT_DIRECTIVE$" "D#1" WHERE ("D#1"."F_ID"="F#0"."F_ID") ) innerQuery

24

Use of SPD – join conclusions• Allows CBO to get better estimations for joins– Provide adjustment ratio– No stored stats provide this info (really cool)– OPT_ESTIMATE is the closest thing, but still different

• Suffers from same side effects of single table– Magnified by longer ADS on joins– SPD from joins can be SUPERSEDED (HAS_STATS) too (*)

25

What happens to SPD after usage? (from 10053 of the second parse)SPD: Generating finding id: type = 1, reason = 1, objcnt = 1, obItr = 0, objid = 135484, objtyp = 1, vecsize = 4, colvec = [1, 2, 3, ], fid = 108…890SPD: Inserted felem, fid=108…890,ftype = 0,freason = 0,dtype = 1,dstate = 2,…SPD: qosdRecDSDirChange dirid = 16004857199080262683, retCode = UPDATED

(from dictionary)NOTES----------------------------------------------------<spd_note> <internal_state>MISSING_STATS</internal_state> <redundant>NO</redundant> <spd_text>{EC(MPAGANO.TAB1)[N1, N2, N3]}</spd_text></spd_note>

CBO is aware there is a “weak path” and it had to use ADS in order to limit

the damage

26

SPD instructs DBMS_STATS(from 10046 during parse)merge into sys.col_group_usage$ d using (select :1 obj#, :2 cols from dual) s on (d.obj# = s.obj# and d.cols = s.cols) when matched then update set d.timestamp = :3, d.flags = d.flags + :4 - bitand(d.flags, :4) when not matched then insert (obj#, cols, timestamp, flags) values (:1,:2,:3,:4)

:1 -> 135484 :2 -> “1,2,3” :3 -> ”1/12/2017 8:33:7” :4 -> 17

(from 10046 during DBMS_STATS run)SELECT ... FROM SYS.COL_GROUP_USAGE$ CU…UPDATE SYS.COL_GROUP_USAGE$ C SET C.FLAGS = FLAGS + 8 - BITAND(FLAGS, 8) WHERE OBJ# = :B2 AND COLS = :B1

COLUMN_NAME DATA_DEFAULT NUM_DISTINCT HISTOGRAM ------------------------------ ------------------------------------ ------------ ---------SYS_STSOYQUEIAZ7FI9DV53VLN$$$0 SYS_OP_COMBINED_HASH("N1","N2","N3") 100 FREQUENCY

27

Then what happens to SPD?SINGLE TABLE ACCESS PATH Single Table Cardinality Estimation for TAB1[TAB1] SPD: Directive valid: dirid=160…83,state=2,flags=1,loc=1{EC(…)[1, 2, 3]} ... ColGroup (#1, VC) SYS_STSOYQUEIAZ7FI9DV53VLN$$$0 Col#: 1 2 3 … Table: TAB1 Alias: TAB1 Card: Original: 100000.000000 Rounded: 1000 Computed: 1000.000000…SPD: Inserted felem, fid=10…890,…,dtype = 1, dstate = 3, dflag = 1,……NOTES------------------------------------------------------<spd_note> <internal_state>HAS_STATS</internal_state> <redundant>NO</redundant> <spd_text>{EC(MPAGANO.TAB1)[N1, N2, N3]}</spd_text></spd_note>

If the CG doesn’t help then SPD is marked as PERMANENT

and behaves like MISSING_STATS (ADS used)

No mistake

28

SPD DS results• DS results are stored

– Multiple SQL IDs (different parse) would trigger same DS SQL• In 12.1 Result Cache is used

– Results have lifetime of 3600 secs– Result Cache itself had some troubles in the past

• Mostly latch contention under heavy load

• In 12.2 SPD repository itself used to store results– Makes them more permanent– Search in the repo based on ADS recursive SQL ID

29

Is my SQL using SPD?• Many SPDs can reference just a few objects

– Even simple SQL can use many SPDs• CBO looks into them and pick up valid

– A subset of Valid are Used• OTHER_XML reports #valid/#used• DBMS_XPLAN to show if SQL is using SPDs

– DISPLAY (explain plan) reports which ones– DISPLAY_CURSOR (execution) reports just #used

30

DBMS_SPD• Oracle seeded package to manage SPDs• Move via pack/unpack in stgtab like baselines– Base 12.1.0.2 has bugs that impact xfr of SPDs

• Allow to manually flush SPDs to disk– Instead of waiting 15 minutes

• Can set basic preferences– For example, retention period before auto drop

31

Why the bad reputation?• Some issues caused large negative impact

– Hard to spot when the feature is “transparent”– Main symptoms mutex / latch contention

• SPDs generated based on workload– Performance tends get better “with time”

• Makes performance in Prod bad on day 1• Even though it was great in Test on day 200

– Upgrades need extra steps (move SPDs)• Limited number of Column Groups per table

– Less useful CG could be created instead of useful ones

32

Most common problems• Most issues are ultimately caused by ADS– ADS running for too long– ADS executed even for small app SQLs (no need)

• Longer parse caused by sampling– Parse is a serialized operation, parser hold X mutex– Other sessions with same SQL are stuck– Easy for snowball effect to trigger

• Result Cache maybe not the best decision?

33

Solutions?• I’m personally a big fan of SPD

– But I’m not a Production DBA • Many products require to turn it off in 12.1

– Bad habit was to turn off all Adaptive Features via parameter– Oracle patch split parameter in two (AdaptPlans vs AdaptStats)

• So just SPD and ADS can be disabled, Adaptive Plans stay ON• Oracle recommended solution in 12.1, SPD OFF with patch+param

• Disabled by default in 12.2– When enabled it seems a bit more conservative – Too early to judge how it will behave in 12.2 prod

34

Summary• Goal is improve CBO estimations over time– Tracking correlation in many areas– Putting stats / ADS in place to catch that

• Might take time and introduce instability– Makes comparing two systems harder

• Generated issues in 12.1– Oracle recommends to disable by default – Enable back specific systems might be beneficial

35

36

Contact Information• http://mauro-pagano.com– Email• mauro.pagano@gmail.com

– Free tools to download• SQLd360 • TUNAs360• PAthfinder