SQL Pivot and Prune Queries - Keeping an Eye on Performance

SQL PIVOT AND PRUNE QUERIES - KEEPING AN EYE ON PERFORMANCE

Author: Brendan Furey

Creation Date: 2 May 2011

Version: 1.3

Last Updated: 25 September 2012

document.doc . Page 1 of 53

Table of Contents

Introduction 4Hardware/Software Summary 4

Overview 5Problem Description 5Test Data Sets 6Pivot and Prune Query Strategies 6

Pivot Strategies 6Prune Strategies 7Strategy Modifiers 7Test Combinations 7

Testing Strategy 8Output Files 8

The Queries 9JNSQ: Join, Subquery 9

Query Text 9Query Diagram 10Execution Plan Example (W64-D16) 10Results 11

WJSQ: With, Subquery 12Query Text 12Query Diagram 13Execution Plan Example (W128-D64) 13Results 14

WJAN: With, Analytic 15Query Text 15Query Diagram 17Execution Plan Example (W128-D64) 17Results 18

WJKP: With, Keep 19Query Text 19Query Diagram 20Execution Plan Example (W128-D64) 20Results 21

PVAN: Pivot, Analytic 22Query Text 22Query Diagram 22Execution Plan Example (W128-D64) 22Results 23

PVANIV: Pivot, Analytic, View 24Query Text 24Query Diagram 24Execution Plan Example (W128-D64) 25Results 25

PVKP: Pivot, Keep 26Query Text 26Query Diagram 26Execution Plan Example (W128-D64) 27Results 27

PVKPIV: Pivot, Keep, View 28Query Text 28Query Diagram 28Execution Plan Example (W128-D64) 29Results 29

FNSC: Database Function 30document.doc Page 2 of 53

Query Text - Main 30Query Text - Function 30Query/Function Diagram 31Execution Plan Example - Main 31Execution Plan Example – Function 31Results 32

SSKP: Select Scalar Subquery, Keep33Query Text 33Query Diagram 34Execution Plan Example (W1-D1024) 34Results 34

SSSM: Select Scalar Subquery, Max 34Query Text 34Query Diagram 35Execution Plan Example (W1-D1024) 35Results 35

SSOB: Select Scalar Subquery, Order By 35Query Text 35Query Diagram 36Results 36

Analysis of Results 37Slice Analysis 37

Narrow Slice 37Wide Slice 38Shallow Slice 40Deep Slice 41

Summary Analysis 43Tables and Graphs of Relative Performance 43Which Query is Best? 44

Cost Based Optimizer 44CBO Cardinalities 44Inline View Modifier 45Subquery Factoring Modifier 45Performance of JNSQ 45Performance of Database Function and Scalar Subquery Strategies 46General Conclusions 46

Query Testing Program 47Call Structure Diagram 47Data Flow Diagram 48Table Structures 48

Generic Tables 48Specific Tables for our HR Queries 49

Program Logic 49Example Output 50

References 52

Change RecordDate Author Version Change Reference

02-May-2011 BPF 1.0 Initial03-May-2011 BPF 1.1 Typos and references20-Jun-2011 BPF 1.2 More typos and reference to new article25-Sep-2011 BPF 1.3 References now hyperlinks, and minor tidying

document.doc Page 3 of 53

IntroductionIt is a very common requirement in SQL to join two record sets where there is a one to many relationship between the two sets, but where the cardinality of the result set is the same as that of the set on the 'one' side. The obvious case is for standard grouping and aggregation querying, such as simply counting the number of records in the 'many' set for each record in the 'one' set. There are also some slightly less obvious cases where there may be various SQL techniques available, with varying performance and complexity characteristics. This article looks at two such cases: the first where one wishes to join multiple subtypes of a given entity – this is generally referred to as ‘pivoting’ from rows to columns; and the second where one wishes to join just one record from the 'many' set, but does not have a pure join condition to identify the record and so must use an ordering condition instead – I will call this ‘pruning’.

This work attempts to find the best SQL techniques for such queries in Oracle 11g, mainly in terms of performance. It does this by running a variety of queries within the context of an outbound interface against a deliberately simple data model across a two-dimensional range of data sizes. A simple generic PL/SQL package has been written to perform the testing efficiently, and it uses a previously described object type for timing (a more recent version is described here: Code Timing and Object Orientation and Zombies). Visio diagrams are provided for query structures, based on a similar approach previously described (A Structured Approach to SQL Query Design), and Microsoft Excel graphs are used to display comparative performances. The results reveal some interesting features of the behaviour of the Cost Based Optimiser in Oracle 11g.

The output file is provided here.

Hardware/Software SummaryComponent DescriptionDatabase Oracle Database 11g Express Edition Release 11.2.0.2.0 - BetaDiagrammer Microsoft Visio 2003 (11.3216.5606)Grapher Microsoft Excel 2007Operating System Microsoft Windows 7 Home Premium (32 bit)Computer Samsung X120, 3GB memory, Intel U4100 @ 1.3GHz x 2 (XE only uses 1 though)

document.doc . Page 4 of 53

http://www.scribd.com/doc/54425770/SQL-Pivot-and-Prune-Queries-%E2%80%93-Output

http://www.scribd.com/doc/15723877/A-Structured-Approach-to-SQL-Query-Design

http://www.scribd.com/doc/43588788/Code-Timing-and-Object-Orientation-and-Zombies


Overview

Problem DescriptionThe SQL problem is to obtain a single output record for each master record, to include a single ‘best’ value of a field in the detail record for each of several subtypes.

We consider a simple example scenario where the master table is employees in Oracle’s demo HR schema, and the detail table is a new table, phone_numbers, a simplified version of hz_contact_points in Oracle Applications. The phone_numbers table has records of four subtypes: HOME, WORK, MOBILE, FAX, and a valid from date, where the most recent number is preferred for a given type, with the maximum number used as a tie-breaker. We’ll assume that an employee should always be returned even if he has no phone numbers.

In the real world, this kind of query might be used in an outbound interface that supplies the latest contact details on a company’s customers to a third-party supplier for marketing or other purposes. In that kind of case, there would typically be a small number of detail records across a relative large table. This would be the wide-shallow scenario discussed below. Another kind of real world example might be


a CRM (Customer Relations Management) system in which one wants a report on the most recent customer interactions, and this could fall into the wide/narrow-deep scenarios.

Test Data SetsWe will test the performance of the queries across a range of sizes of the master table, denoted by a width dimension, and the detail table, denoted by a depth dimension. The maximum sizes are constrained by the execution time and storage capacity on a small laptop running Oracle 11g XE, so we will use a wide data set and a deep data set for better coverage. The data sets are shown in the table below, where the cells contain the number of detail records against each dimension pair, and the second row (# W =>) has the numbers of master records.

Starting from the base sizes (W1, D1) of 4 details (one of each subtype) for each of 107 master records (being the number of employees in the HR demo schema), the dimensions double successively. Test data are generated by copying the 107 seed employees with a sequence number added to the names, and generating random 9-digit phone numbers with the valid from date a random day within a given period. We used a single year for the wide data set, which would have a lot of duplicates, and a 50 year period for the deep data set, with fewer duplicates (the difference didn’t affect performance significantly). In order to test outer-joinery, one employee has all phone numbers deleted and another has all FAX numbers deleted (that’s why the cell values are slightly smaller than the products of the dimension values). Rows D1-D64 form the wide data set, while columns W1-W8 form the deep data set.

# D / W W1 W2 W4 W8 W16 W32 W64 W128# W => 107 214 428 856 1,712 3,424 6,848 13,696

D1 4 423 851 1,707 3,419 6,843 13,691 27,387 54,779D2 8 846 1,702 3,414 6,838 13,686 27,382 54,774 109,558

D4 16 1,692 3,404 6,828 13,676 27,372 54,764 109,548 219,116D8 32 3,384 6,808 13,656 27,352 54,744 109,528 219,096 438,232

D16 64 6,768 13,616 27,312 54,704 109,488 219,056 438,192 876,464D32 128 13,536 27,232 54,624 109,408 218,976 438,112 876,384 1,752,928

D64 256 27,072 54,464 109,248 218,816 437,952 876,224 1,752,768 3,505,856D128 512 54,144 108,928 218,496 437,632 875,904 1,752,448 3,505,536 7,011,712

D256 1,024 108,288 217,856 436,992 875,264 1,751,808 3,504,896 7,011,072 14,023,424D512 2,048 216,576 435,712 873,984 1,750,528 3,503,616 7,009,792 14,022,144 28,046,848

D1024 4,096 433,152 871,424 1,747,968 3,501,056 7,007,232 14,019,584 28,044,288 56,093,696

Pivot and Prune Query Strategies

Pivot Strategies

Oracle Database 11g introduced an explicit SQL syntax for pivoting rows to columns, using the new keyword PIVOT. Prior to 11g one could simulate this by the use of grouping and CASE expressions, but there seems little value in considering this approach now. There are other approaches however that are sufficiently different for it to be interesting to evaluate them.

Pivot Strategy Notes

Pivot Oracle 11g explicit PIVOT syntax.

Join

Repeat the tables and joins for each subtype. This is quite a commonly used strategy but might be expected to be inefficient due to additional table reads. Efficiency can be improved by combining with the Oracle 10g feature of subquery factoring, using the WITH keyword.

Select Scalar Subquery

Obtain each subtype column using a separate scalar subquery on the detail table in the Select clause, correlated to the master record. The detail table then does not appear in the main query. This might also be expected to be inefficient since multiple queries are effectively repeated for every row; however, there may be scenarios where this effect is outweighed by benefits such as improved index usage.

Database function

This is similar to the select scalar subquery strategy in that the detail table is removed from the main query, but here is moved into a separate function call. This might be


expected to have similar performance characteristics, but with some added overhead from the PL/SQL context switch; it also has a theoretical disadvantage in that read-consistency is not guaranteed. This strategy is quite widely used

Prune Strategies

Oracle Database version 9.0.1 introduced an explicit SQL syntax for pruning rows via aggregate functions, using the keyword KEEP, with FIRST or LAST. Prior to this, pruning could be achieved by combining various non-purpose-built techniques, and it seems that these alternatives continue to be widely used today.

Prune Strategy Notes

Keep

The maximum function can be applied to an expression on the detail record with the KEEP clause specifying that the maximum is over only those records ranking first (or last) in a given ordering. This is Oracle’s explicit pruning syntax and so might be expected to perform well. From the SQL manual (REF-1): When you need a value from the first or last row of a sorted group, but the needed value is not the sort key, the FIRST and LAST functions eliminate the need for self-joins or views and enable better performance.

Analytic sThe detail table can be placed within an inline view where the analytic function Row_Number can be selected to rank the records; a condition is then placed outside the view that the rank equals 1.

Select Scalar Subquery

The same approach as mentioned above for pivoting can also be used for pruning, where the pruning takes place within its own subquery in the Select clause; there are various possible subquery implementations

Join Scalar Subquery

Select an expression defining the preferred record in a subquery correlated to the master record, and join the main record. Aside from the obvious performance disadvantage of additional reads of the detail table, applying this strategy directly prevents outer joining; however, when used in conjunction with subquery factoring, outer joining to the factor is possible

Order By

The detail records are selected within an inline view in the preferred order, then a condition is applied outside the view that ROWNUM = 1. This has to be used within a database function: we tried to use it within a select scalar subquery but the SQL is invalid.

MaxWithin a select scalar subquery a maximum is taken of an expression that prepends the ordering fields to the desired select field, using appropriate type casting (e.g. taking the Julian format for a date)

Strategy Modifiers

There are some general SQL techniques that can be used in conjunction with the above pivot and prune strategies in some cases, and that may affect performance.

Strategy Modifier Notes

Subquery Factoring

This feature was introduced in Oracle 10g, and, using the WITH keyword, allows a subquery to be defined once and subsequently referenced several times by its alias. Where appropriate Oracle will run the subquery in advance and store the results in a temporary table to improve performance. Unlike with subqueries in the where clause of a query, outer joins can be used.

Inline View

An inline view is a subquery in brackets in the position of a table in the FROM clause of a query. It can affect performance by influencing the execution plan, as we’ll see.

Test Combinations

The table below lists the queries tested with the strategies employed. Note that not all combinations of strategies make sense, but we have tried to cover all that do.


Code Pivot Prune Modifier NotesMain query set tested on both wide and deep data sets, except JNSQ wide onlyJNSQ Join Join Scalar SubqueryWJSQ Join Join Scalar Subquery Subquery FactoringWJAN Join Analytic s Subquery FactoringWJKP Join Keep Subquery FactoringPVAN Pivot Analytic s

PVANIV Pivot Analytic s Inline ViewPruning is done within the inline view

PVKP Pivot Keep

PVKPIV Pivot Keep Inline ViewPruning is done within the inline view

FNSC Database Function Order By Inline ViewAdditional queries tested on W1-D1024 only - we tried to match the performance characteristics of FNSC on the narrow-deep data set without using a database function

SSSM Select Scalar Subquery Max

SSKP Select Scalar Subquery Keep

SSOB Select Scalar Subquery Order By Inline View

Same pruning strategy as FNSC but within query. The query is invalid owing to the need to correlate through two levels

Testing StrategyIn order to provide a realistic scenario, the queries are executed within the context of a simple outbound interface that writes each record to a file as a comma-separated string. A small PL/SQL package has been written to automate the testing. The program loops over width and depth dimensions, and for each data set point makes a call to a separate package to set up the test data and have the CBO statistics gathered; it then loops over a set of queries defined in the same separate package as strings that are executed by the main package. Detailed timings are made using the timing objects mentioned earlier and written to a generic log table. The total CPU and elapsed times for the query and data set point are written to a special run statistics table, from which they can be output in a form easily loaded into Microsoft Excel. The times can be viewed as a function of the 2-dimensional width-depth domain, and displayed as a 3-dimensional surface graph for each query.

The execution plan is obtained in each case, using an Oracle API, and is written to the generic log. The query string includes a random number that guarantees a hard-parse and thus recalculation of the execution plans at each data set point. Further details of the packages and supporting tables are given in a later section.

The testing strategy here seemed to me to provide much greater insight than the more conventional approach of testing on a single large data set, and was later used again in Forming Range-Based Break Groups with Advanced SQL, and other articles.

Output FilesThe complete output files are provided in SQL Pivot and Prune Queries – Output.

LST File NotesTest_Phone_8-7.LST Wide data set – first 9 queriesTest_Phone_4-11.LST Deep data set - first 9 queries, except JNSQTest_Phone_1-11.LST W1-D1024, SSKP, SSSM, FNSC – testing select scalar subqueries



http://www.scribd.com/doc/57696875/Forming-Range-Based-Break-Groups-With-Advanced-SQL


Test_Phone_1-11_3.LST W1-D1024, PVKP, PVKPG, PVKPIV – CBO grouping problem


The QueriesFor each query subsection, the following are provided (for the main queries, the last three were added later and have a bit less):

The query text in static form for readability; the program in fact uses dynamic SQL and consequently the random number at the end of the CSV string ensures that the query is hard-parsed for each data set point

A query diagram. The diagrams use extended versions of notation first described in A Structured Approach to SQL Query Design

The last Execution Plan output on the wide data set (usually)

A results section, split into wide and deep data sets, with tables of CPU times and 3-d graphs generated from Microsoft Excel. Analysis is reserved for a later section

JNSQ: Join, Subquery

Query TextSELECT /* JNSQ*/'"' || emp.first_name || ' ' || emp.last_name || '","' || pho_h.phone_number || '","' || pho_w.phone_number || '","' || pho_m.phone_number || '","' || pho_f.phone_number || '8098"' FROM employees emp JOIN phone_numbers pho_h ON pho_h.employee_id = emp.employee_id AND pho_h.phone_type = 'HOME' AND TO_CHAR (pho_h.valid_from, 'J') || pho_h.phone_number = (

SELECT MAX (TO_CHAR (sbq_h.valid_from, 'J') || sbq_h.phone_number) FROM phone_numbers sbq_h WHERE sbq_h.employee_id = pho_h.employee_id AND sbq_h.phone_type = pho_h.phone_type)

JOIN phone_numbers pho_w ON pho_w.employee_id = emp.employee_id AND pho_w.phone_type = 'WORK' AND TO_CHAR (pho_w.valid_from, 'J') || pho_w.phone_number = (

SELECT MAX (TO_CHAR (sbq_w.valid_from, 'J') || sbq_w.phone_number) FROM phone_numbers sbq_w WHERE sbq_w.employee_id = pho_w.employee_id AND sbq_w.phone_type = pho_w.phone_type)

JOIN phone_numbers pho_m ON pho_m.employee_id = emp.employee_id AND pho_m.phone_type = 'MOBILE' AND TO_CHAR (pho_m.valid_from, 'J') || pho_m.phone_number = (

SELECT MAX (TO_CHAR (sbq_m.valid_from, 'J') || sbq_m.phone_number) FROM phone_numbers sbq_m WHERE sbq_m.employee_id = pho_m.employee_id AND sbq_m.phone_type = pho_m.phone_type)

JOIN phone_numbers pho_f ON pho_f.employee_id = emp.employee_id AND pho_f.phone_type = 'FAX' AND TO_CHAR (pho_f.valid_from, 'J') || pho_f.phone_number = (

SELECT MAX (TO_CHAR (sbq_f.valid_from, 'J') || sbq_f.phone_number) FROM phone_numbers sbq_f WHERE sbq_f.employee_id = pho_f.em ployee_id AND sbq_f.phone_type = pho_f.phone_type)

ORDER BY 1




Query Diagram

Execution Plan Example (W64-D16)

In the example below, the top level cardinality estimate of 1 is clearly poor, since the number of rows returned is the number of employees less 2 due to the inner joins where phone numbers were deleted, giving 6,846. In addition, it is highly unlikely that not driving from the master table could be optimal.

--------------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |--------------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | 1777 (100)| |


| 1 | SORT ORDER BY | | 1 | 220 | 1777 (2)| 00:00:22 || 2 | NESTED LOOPS | | 1 | 220 | 1776 (1)| 00:00:22 || 3 | NESTED LOOPS | | 1 | 201 | 1710 (1)| 00:00:21 || 4 | NESTED LOOPS | | 1 | 182 | 1644 (2)| 00:00:20 || 5 | NESTED LOOPS | | 1 | 155 | 1642 (2)| 00:00:20 || 6 | NESTED LOOPS | | 1 | 128 | 1577 (2)| 00:00:19 || 7 | NESTED LOOPS | | 3 | 327 | 1379 (2)| 00:00:17 || 8 | NESTED LOOPS | | 1 | 82 | 1314 (2)| 00:00:16 ||* 9 | HASH JOIN | | 1 | 55 | 1313 (2)| 00:00:16 || 10 | VIEW | VW_SQ_1 | 4840 | 99K| 658 (2)| 00:00:08 || 11 | HASH GROUP BY | | 4840 | 122K| 658 (2)| 00:00:08 ||* 12 | TABLE ACCESS FULL | PHONE_NUMBERS | 109K| 2774K| 653 (1)| 00:00:08 ||* 13 | TABLE ACCESS FULL | PHONE_NUMBERS | 109K| 3628K| 654 (1)| 00:00:08 || 14 | TABLE ACCESS BY INDEX ROWID| EMPLOYEES | 1 | 27 | 1 (0)| 00:00:01 ||* 15 | INDEX UNIQUE SCAN | EMP_EMP_ID_PK | 1 | | 0 (0)| ||* 16 | TABLE ACCESS BY INDEX ROWID | PHONE_NUMBERS | 16 | 432 | 65 (0)| 00:00:01 ||* 17 | INDEX RANGE SCAN | PHONE_NUMBERS_N2 | 64 | | 2 (0)| 00:00:01 ||* 18 | VIEW PUSHED PREDICATE | VW_SQ_4 | 1 | 19 | 66 (0)| 00:00:01 || 19 | SORT GROUP BY | | 1 | 26 | 66 (0)| 00:00:01 ||* 20 | TABLE ACCESS BY INDEX ROWID| PHONE_NUMBERS | 16 | 416 | 66 (0)| 00:00:01 ||* 21 | INDEX RANGE SCAN | PHONE_NUMBERS_N2 | 64 | | 3 (0)| 00:00:01 ||* 22 | TABLE ACCESS BY INDEX ROWID | PHONE_NUMBERS | 16 | 432 | 65 (0)| 00:00:01 ||* 23 | INDEX RANGE SCAN | PHONE_NUMBERS_N2 | 64 | | 2 (0)| 00:00:01 ||* 24 | TABLE ACCESS BY INDEX ROWID | PHONE_NUMBERS | 16 | 432 | 2 (0)| 00:00:01 ||* 25 | INDEX RANGE SCAN | PHONE_NUMBERS_N2 | 64 | | 2 (0)| 00:00:01 ||* 26 | VIEW PUSHED PREDICATE | VW_SQ_3 | 1 | 19 | 66 (0)| 00:00:01 || 27 | SORT GROUP BY | | 1 | 26 | 66 (0)| 00:00:01 ||* 28 | TABLE ACCESS BY INDEX ROWID | PHONE_NUMBERS | 16 | 416 | 66 (0)| 00:00:01 ||* 29 | INDEX RANGE SCAN | PHONE_NUMBERS_N2 | 64 | | 3 (0)| 00:00:01 ||* 30 | VIEW PUSHED PREDICATE | VW_SQ_2 | 1 | 19 | 66 (0)| 00:00:01 || 31 | SORT GROUP BY | | 1 | 26 | 66 (0)| 00:00:01 ||* 32 | TABLE ACCESS BY INDEX ROWID | PHONE_NUMBERS | 16 | 416 | 66 (0)| 00:00:01 ||* 33 | INDEX RANGE SCAN | PHONE_NUMBERS_N2 | 64 | | 3 (0)| 00:00:01 |--------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):---------------------------------------------------

9 - access("VW_COL_1"=TO_CHAR(INTERNAL_FUNCTION("VALID_FROM"),'j')||"PHONE_NUMBER" AND "ITEM_1"="PHO_F"."EMPLOYEE_ID" AND "ITEM_2"="PHO_F"."PHONE_TYPE") 12 - filter("SBQ_F"."PHONE_TYPE"='FAX') 13 - filter("PHO_F"."PHONE_TYPE"='FAX') 15 - access("PHO_F"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 16 - filter("PHO_H"."PHONE_TYPE"='HOME') 17 - access("PHO_H"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 18 - filter("VW_COL_1"="PHO_H"."SYS_NC00007$") 20 - filter("SBQ_H"."PHONE_TYPE"='HOME') 21 - access("SBQ_H"."EMPLOYEE_ID"="PHO_H"."EMPLOYEE_ID") 22 - filter("PHO_M"."PHONE_TYPE"='MOBILE') 23 - access("PHO_M"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 24 - filter("PHO_W"."PHONE_TYPE"='WORK') 25 - access("PHO_W"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 26 - filter("VW_COL_1"="PHO_W"."SYS_NC00007$") 28 - filter("SBQ_W"."PHONE_TYPE"='WORK') 29 - access("SBQ_W"."EMPLOYEE_ID"="PHO_W"."EMPLOYEE_ID") 30 - filter("VW_COL_1"="PHO_M"."SYS_NC00007$") 32 - filter("SBQ_M"."PHONE_TYPE"='MOBILE') 33 - access("SBQ_M"."EMPLOYEE_ID"="PHO_M"."EMPLOYEE_ID")

Results

Wide Data SetTable of CPU TimesNote that the testing program iterates over the depth dimension within the width dimension loop, and stops the inner iteration once a CPU time of 300 seconds has been exceeded, hence the gaps.

Query W1 W2 W4 W8 W16 W32 W64 W128D1 1.35 1.73 2 2.15 2.34 4.01 7.3 13.53D2 1.29 1.94 2.71 3.49 5.37 10.26 20.02 34.59D4 1.51 2.39 4.02 8.99 11.63 21.29 34.43 67.66D8 1.89 4.28 7.47 24.91 15.56 28.7 154.86 298.62D16 2.95 7.83 163.68 320.74 608 85.89 878.53 324.23D32 5.98 16.54 55.29 213.4D64 14.59 59.48 159.65 292.08

Graph


Deep Data Set (not tested as JNSQ clearly uncompetitive)Notes

The performance across the data set points is very erratic. For example, for D16, doubling the width dimension from W16 to W32 caused the CPU time to drop from 608s to 86s. This clearly indicates that the CBO had followed a highly sub-optimal execution plan for the former. This query is the most complicated of those tested, having 9 table instances, but it may still be considered surprising that performance (of the CBO) was so poor for what in the real world would not be an especially complex query. It is not part of the scope to analyse the execution plans in detail, but one may observe that the worse ones do three full scans of the detail table while the better ones do only two.

It might be worth noting that the best execution plan would most likely decompose into one subplan for obtaining the detail records and joining them to an employee-based record set, repeated four times, plus one to obtain the employees (which would drive). However, the CBO would not be likely to see it that way.

WJSQ: With, Subquery

Query TextWITH wit AS (SELECT pho.employee_id,

pho.phone_type,pho.phone_number

FROM phone_numbers pho WHERE TO_CHAR (pho.valid_from, 'J') || pho.phone_number = (

SELECT MAX (TO_CHAR (sbq.valid_from, 'J') || sbq.phone_number) FROM phone_numbers sbq WHERE sbq.employee_id = pho.employee_id AND sbq.phone_type = pho.phone_type)

)SELECT /* WJSQ*/'"' || emp.first_name || ' ' || emp.last_name || '","' || wit_h.phone_number || '","' || wit_w.phone_number || '","' || wit_m.phone_number || '","' || wit_f.phone_number || '4121"' FROM employees emp LEFT JOIN wit wit_h ON wit_h.employee_id = emp.employee_id AND wit_h.phone_type = 'HOME' LEFT JOIN wit wit_w ON wit_w.employee_id = emp.employee_id AND wit_w.phone_type = 'WORK' LEFT JOIN wit wit_m ON wit_m.employee_id = emp.employee_id AND wit_m.phone_type = 'MOBILE' LEFT JOIN wit wit_f ON wit_f.employee_id = emp.employee_id AND wit_f.phone_type = 'FAX' ORDER BY 1


Query Diagram


The subquery factoring system table is actually named SYS_TEMP_0FD9D66F3_71C4A6, but I truncated it to avoid line wrapping below.

Notice how poor the cardinality estimate of 1 is on the subquery factor table. The actual number would be approximately four times the number of employees (13,696), i.e. 54,784, owing to the subquery correlation. On the other hand, the final cardinality estimate is about right.

-------------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |-------------------------------------------------------------------------------------------------------


| 0 | SELECT STATEMENT | | | | | 28090 (100)| || 1 | TEMP TABLE TRANSFORMATION | | | | | | || 2 | LOAD AS SELECT | | | | | | ||* 3 | HASH JOIN | | 1 | 64 | 1592K| 27704 (2)| 00:05:33 || 4 | VIEW | VW_SQ_1 | 38739 | 1134K| | 14699 (2)| 00:02:57 || 5 | SORT GROUP BY | | 38739 | 983K| 121M| 14699 (2)| 00:02:57 ||* 6 | TABLE ACCESS FULL | PHONE_NUMBERS | 3509K| 87M| | 5285 (2)| 00:01:04 || 7 | TABLE ACCESS FULL | PHONE_NUMBERS | 3509K| 113M| | 5265 (1)| 00:01:04 || 8 | SORT ORDER BY | | 13583 | 1313K| 1520K| 386 (2)| 00:00:05 ||* 9 | HASH JOIN RIGHT OUTER | | 13583 | 1313K| | 78 (3)| 00:00:01 ||* 10 | VIEW | | 1 | 18 | | 2 (0)| 00:00:01 || 11 | TABLE ACCESS FULL | SYS_TEMP_0FD9D66F| 1 | 19 | | 2 (0)| 00:00:01 ||* 12 | HASH JOIN RIGHT OUTER | | 13583 | 1074K| | 76 (3)| 00:00:01 ||* 13 | VIEW | | 1 | 18 | | 2 (0)| 00:00:01 || 14 | TABLE ACCESS FULL | SYS_TEMP_0FD9D66F| 1 | 19 | | 2 (0)| 00:00:01 ||* 15 | HASH JOIN RIGHT OUTER | | 13583 | 835K| | 73 (2)| 00:00:01 ||* 16 | VIEW | | 1 | 18 | | 2 (0)| 00:00:01 || 17 | TABLE ACCESS FULL | SYS_TEMP_0FD9D66F| 1 | 19 | | 2 (0)| 00:00:01 ||* 18 | HASH JOIN RIGHT OUTER| | 13583 | 596K| | 71 (2)| 00:00:01 ||* 19 | VIEW | | 1 | 18 | | 2 (0)| 00:00:01 || 20 | TABLE ACCESS FULL | SYS_TEMP_0FD9D66F| 1 | 19 | | 2 (0)| 00:00:01 || 21 | TABLE ACCESS FULL | EMPLOYEES | 13583 | 358K| | 68 (0)| 00:00:01 |-------------------------------------------------------------------------------------------------------


3 - access("VW_COL_1"=TO_CHAR(INTERNAL_FUNCTION("VALID_FROM"),'j')||"PHONE_NUMBER" AND "ITEM_0"="PHO"."EMPLOYEE_ID" AND "ITEM_1"="PHO"."PHONE_TYPE") 6 - filter(("SBQ"."PHONE_TYPE"='FAX' OR "SBQ"."PHONE_TYPE"='HOME' OR "SBQ"."PHONE_TYPE"='MOBILE' OR "SBQ"."PHONE_TYPE"='WORK')) 9 - access("WIT_F"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 10 - filter("WIT_F"."PHONE_TYPE"='FAX') 12 - access("WIT_M"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 13 - filter("WIT_M"."PHONE_TYPE"='MOBILE') 15 - access("WIT_W"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 16 - filter("WIT_W"."PHONE_TYPE"='WORK') 18 - access("WIT_H"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 19 - filter("WIT_H"."PHONE_TYPE"='HOME')

Results

Wide Data SetTable of CPU Times

Query W1 W2 W4 W8 W16 W32 W64 W128D1 0.13 0.22 0.24 0.37 0.67 1.17 2.16 4.44D2 0.14 0.19 0.28 0.41 0.68 1.29 2.26 4.62D4 0.15 0.18 0.28 0.45 0.75 1.39 2.75 5.05D8 0.16 0.24 0.31 0.56 0.93 1.67 3.3 6.14D16 0.16 0.28 0.38 0.63 1.17 2.19 4.34 8.7D32 0.22 0.3 0.5 0.93 1.73 3.4 6.47 13.22D64 0.3 0.42 0.79 1.42 2.73 5.7 11.6 22.62

Graph


Deep Data SetTable of CPU Times

Query W1 W2 W4 W8D1 0.19 0.21 0.23 0.38D2 0.19 0.23 0.25 0.41D4 0.14 0.2 0.26 0.42D8 0.15 0.21 0.29 0.48D16 0.17 0.22 0.39 0.61D32 0.22 0.3 0.54 0.88D64 0.28 0.47 0.73 1.42D128 0.44 0.67 1.28 2.5D256 0.62 1.19 2.38 4.59D512 1.11 2.15 4.45 9.54D1024 1.43 2.76 5.74 12.59

Graph

WJAN: With, Analytic

Query TextWITH wit AS (SELECT ilv.employee_id,

ilv.phone_type,ilv.pho_number


FROM (SELECT pho.employee_id,

pho.phone_type,pho.phone_number,Row_Number() OVER (PARTITION BY PHO.employee_id, pho.phone_type ORDER BY pho.valid_from

DESC, pho.phone_number) ind FROM phone_numbers pho

) ilv WHERE ilv.ind = 1)SELECT /* WJAN*/'"' || emp.first_name || ' ' || emp.last_name || '","' || wit_h.phone_number || '","' || wit_w.phone_number || '","' || wit_m.phone_number || '","' || wit_f.phone_number || '3006"' FROM employees emp LEFT JOIN wit wit_h ON wit_h.employee_id = emp.employee_id AND wit_h.phone_type = 'HOME' LEFT JOIN wit wit_w ON wit_w.employee_id = emp.employee_id AND wit_w.phone_type = 'WORK' LEFT JOIN wit wit_m ON wit_m.employee_id = emp.employee_id AND wit_m.phone_type = 'MOBILE' LEFT JOIN wit wit_f ON wit_f.employee_id = emp.employee_id AND wit_f.phone_type = 'FAX' ORDER BY 1


Query Diagram



Notice how poor the cardinality estimate of 3,509,000 (about the number of phone numbers) is on the subquery factor table. The actual number would be approximately four times the number of employees (13,696), i.e. 54,784. Also, the final cardinality estimate is very poor, presumably just because of the initial mistake - 63T means, I think, 63,000,000,000,000, when the actual number would be the number of employees (13,696),

-----------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |-----------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | | 14G(100)| || 1 | TEMP TABLE TRANSFORMATION | | | | | | || 2 | LOAD AS SELECT | | | | | | ||* 3 | VIEW | | 3509K| 107M| | 31925 (1)| 00:06:24 ||* 4 | WINDOW SORT PUSHED RANK| | 3509K| 90M| 134M| 31925 (1)| 00:06:24 || 5 | TABLE ACCESS FULL | PHONE_NUMBERS | 3509K| 90M| | 5265 (1)| 00:01:04 || 6 | SORT ORDER BY | | 63T| 5720T| 6574T| 14G (41)|999:59:59 ||* 7 | HASH JOIN RIGHT OUTER | | 63T| 5720T| 100M| 1494M (18)|999:59:59 ||* 8 | VIEW | | 3509K| 60M| | 5063 (1)| 00:01:01 || 9 | TABLE ACCESS FULL | SYS_TEMP_0FD9D6| 3509K| 63M| | 5063 (1)| 00:01:01 |


|* 10 | HASH JOIN RIGHT OUTER | | 242G| 17T| 100M| 4830K (21)| 16:06:11 ||* 11 | VIEW | | 3509K| 60M| | 5063 (1)| 00:01:01 || 12 | TABLE ACCESS FULL | SYS_TEMP_0FD9D6| 3509K| 63M| | 5063 (1)| 00:01:01 ||* 13 | HASH JOIN RIGHT OUTER | | 928M| 54G| 100M| 30789 (13)| 00:06:10 ||* 14 | VIEW | | 3509K| 60M| | 5063 (1)| 00:01:01 || 15 | TABLE ACCESS FULL | SYS_TEMP_0FD9D6| 3509K| 63M| | 5063 (1)| 00:01:01 ||* 16 | HASH JOIN OUTER | | 3552K| 152M| | 5146 (1)| 00:01:02 || 17 | TABLE ACCESS FULL | EMPLOYEES | 13583 | 358K| | 68 (0)| 00:00:01 ||* 18 | VIEW | | 3509K| 60M| | 5063 (1)| 00:01:01 || 19 | TABLE ACCESS FULL | SYS_TEMP_0FD9D6| 3509K| 63M| | 5063 (1)| 00:01:01 |-----------------------------------------------------------------------------------------------------


3 - filter("ILV"."IND"=1) 4 - filter(ROW_NUMBER() OVER ( PARTITION BY "PHO"."EMPLOYEE_ID","PHO"."PHONE_TYPE" ORDER BY INTERNAL_FUNCTION("PHO"."VALID_FROM") DESC ,"PHO"."PHONE_NUMBER")<=1) 7 - access("WIT_F"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 8 - filter("WIT_F"."PHONE_TYPE"='FAX') 10 - access("WIT_M"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 11 - filter("WIT_M"."PHONE_TYPE"='MOBILE') 13 - access("WIT_W"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 14 - filter("WIT_W"."PHONE_TYPE"='WORK') 16 - access("WIT_H"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 18 - filter("WIT_H"."PHONE_TYPE"='HOME')

Results



Graph


Query W1 W2 W4 W8D1 0.15 0.2 0.27 0.32D2 0.14 0.19 0.26 0.39D4 0.14 0.19 0.27 0.42D8 0.14 0.19 0.27 0.44


D16 0.17 0.23 0.32 0.61D32 0.18 0.25 0.47 0.89D64 0.24 0.44 0.86 1.58D128 0.42 0.78 1.62 3.09D256 0.8 1.56 3.18 6.44D512 1.56 3.17 6.68 9.89D1024 3.36 6.94 9.86 19.2

Graph

WJKP: With, Keep

Query TextWITH wit AS (SELECT pho.employee_id,

pho.phone_type, MAX (pho.phone_number) KEEP (DENSE_RANK LAST ORDER BY pho.valid_from) phone_number

FROM phone_numbers pho GROUP BY pho.employee_id, pho.phone_type)SELECT /* WJKP*/'"' || emp.first_name || ' ' || emp.last_name || '","' || wit_h.phone_number || '","' || wit_w.phone_number || '","' || wit_m.phone_number || '","' || wit_f.phone_number || '1970"' FROM employees emp LEFT JOIN wit wit_h ON wit_h.employee_id = emp.employee_id AND wit_h.phone_type = 'HOME' LEFT JOIN wit wit_w ON wit_w.employee_id = emp.employee_id AND wit_w.phone_type = 'WORK' LEFT JOIN wit wit_m ON wit_m.employee_id = emp.employee_id AND wit_m.phone_type = 'MOBILE' LEFT JOIN wit wit_f ON wit_f.employee_id = emp.employee_id AND wit_f.phone_type = 'FAX' ORDER BY 1


Query Diagram



Notice that the cardinality estimate of 38,739 on the subquery factor table is not too far off the actual number, that would be approximately four times the number of employees (13,696), i.e. 54,784. On the other hand, the final cardinality estimate of 943,000 is a long way off the actual number, 13,696.

------------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |------------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | | 38505 (100)| || 1 | TEMP TABLE TRANSFORMATION | | | | | | || 2 | LOAD AS SELECT | | | | | | || 3 | SORT GROUP BY | | 38739 | 1021K| 134M| 14919 (2)| 00:03:00 || 4 | TABLE ACCESS FULL | PHONE_NUMBERS | 3509K| 90M| | 5265 (1)| 00:01:04 || 5 | SORT ORDER BY | | 943K| 89M| 102M| 23586 (1)| 00:04:44 ||* 6 | HASH JOIN RIGHT OUTER | | 943K| 89M| 1136K| 2351 (1)| 00:00:29 ||* 7 | VIEW | | 38739 | 680K| | 40 (0)| 00:00:01 || 8 | TABLE ACCESS FULL | SYS_TEMP_0FD9D66| 38739 | 643K| | 40 (0)| 00:00:01 ||* 9 | HASH JOIN RIGHT OUTER | | 326K| 25M| 1136K| 812 (1)| 00:00:10 ||* 10 | VIEW | | 38739 | 680K| | 40 (0)| 00:00:01 || 11 | TABLE ACCESS FULL | SYS_TEMP_0FD9D66| 38739 | 643K| | 40 (0)| 00:00:01 ||* 12 | HASH JOIN RIGHT OUTER | | 113K| 6963K| 1136K| 312 (1)| 00:00:04 ||* 13 | VIEW | | 38739 | 680K| | 40 (0)| 00:00:01 || 14 | TABLE ACCESS FULL | SYS_TEMP_0FD9D66| 38739 | 643K| | 40 (0)| 00:00:01 ||* 15 | HASH JOIN OUTER | | 39210 | 1723K| | 109 (1)| 00:00:02 || 16 | TABLE ACCESS FULL | EMPLOYEES | 13583 | 358K| | 68 (0)| 00:00:01 ||* 17 | VIEW | | 38739 | 680K| | 40 (0)| 00:00:01 || 18 | TABLE ACCESS FULL | SYS_TEMP_0FD9D66| 38739 | 643K| | 40 (0)| 00:00:01 |------------------------------------------------------------------------------------------------------



6 - access("WIT_F"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 7 - filter("WIT_F"."PHONE_TYPE"='FAX') 9 - access("WIT_M"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 10 - filter("WIT_M"."PHONE_TYPE"='MOBILE') 12 - access("WIT_W"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 13 - filter("WIT_W"."PHONE_TYPE"='WORK') 15 - access("WIT_H"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 17 - filter("WIT_H"."PHONE_TYPE"='HOME')

Results


Query W1 W2 W4 W8 W16 W32 W64 W128D1 0.15 0.18 0.23 0.36 0.62 1.09 2.07 4.16D2 0.15 0.17 0.26 0.32 0.66 1.08 2.09 4.04D4 0.14 0.17 0.25 0.34 0.69 1.14 2.28 4.4D8 0.14 0.17 0.25 0.38 0.72 1.28 2.45 4.93D16 0.15 0.2 0.28 0.44 0.81 1.52 3.07 6.07D32 0.15 0.21 0.34 0.56 1.05 2 4.17 8.08D64 0.2 0.3 0.48 0.78 1.53 3.04 6.05 12.29

Graph


Query W1 W2 W4 W8D1 0.14 0.14 0.21 0.38D2 0.14 0.16 0.23 0.36D4 0.14 0.17 0.25 0.36D8 0.16 0.17 0.25 0.39D16 0.14 0.19 0.28 0.5D32 0.15 0.22 0.34 0.53D64 0.2 0.3 0.41 0.8D128 0.23 0.33 0.64 1.21D256 0.36 0.53 1.02 2D512 0.48 0.9 1.86 4.18D1024 0.82 1.7 3.51 7.31

Graph


PVAN: Pivot, Analytic

Query TextSELECT /* PVAN*/'"' || emp_name || '","' || h || '","' || w || '","' || m || '","' || f || '363"' FROM (

SELECT emp.first_name || ' ' || emp.last_name emp_name,pho.phone_type,pho.phone_number,Row_Number() OVER (PARTITION BY emp.employee_id, pho.phone_type ORDER BY pho.valid_from

DESC, pho.phone_number) ind FROM employees emp LEFT JOIN phone_numbers pho ON pho.employee_id = emp.employee_id

) PIVOT (MAX(phone_number) FOR phone_type IN ('HOME' AS h, 'WORK' AS w, 'MOBILE' AS m, 'FAX' AS f)) WHERE ind = 1 ORDER BY 1

Query Diagram


Notice that the final cardinality estimate of 3,480,000 is a long way off the actual number, 13,696. The CBO has clearly not allowed for pruning (or pivoting) affecting the cardinalities at all.

----------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |----------------------------------------------------------------------------------------------------


| 0 | SELECT STATEMENT | | | | | 51794 (100)| || 1 | SORT ORDER BY | | 3480K| 169M| | 51794 (1)| 00:10:22 || 2 | HASH GROUP BY PIVOT | | 3480K| 169M| | 51794 (1)| 00:10:22 ||* 3 | VIEW | | 3480K| 169M| | 51510 (1)| 00:10:19 ||* 4 | WINDOW SORT PUSHED RANK| | 3480K| 179M| 226M| 51510 (1)| 00:10:19 ||* 5 | HASH JOIN OUTER | | 3480K| 179M| | 5348 (2)| 00:01:05 || 6 | TABLE ACCESS FULL | EMPLOYEES | 13583 | 358K| | 68 (0)| 00:00:01 || 7 | TABLE ACCESS FULL | PHONE_NUMBERS | 3509K| 90M| | 5265 (1)| 00:01:04 |----------------------------------------------------------------------------------------------------


3 - filter("from$_subquery$_001"."IND"=1) 4 - filter(ROW_NUMBER() OVER ( PARTITION BY "EMP"."EMPLOYEE_ID","PHO"."PHONE_TYPE" ORDER BY INTERNAL_FUNCTION("PHO"."VALID_FROM") DESC ,"PHO"."PHONE_NUMBER")<=1) 5 - access("PHO"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID")

Results



Graph


Query W1 W2 W4 W8D1 0.14 0.14 0.19 0.33D2 0.12 0.13 0.21 0.34D4 0.11 0.14 0.23 0.37D8 0.12 0.17 0.28 0.47D16 0.15 0.2 0.34 0.67D32 0.17 0.29 0.53 1D64 0.27 0.49 0.97 1.94D128 0.46 0.9 1.85 3.65D256 0.9 1.88 3.91 6.03D512 1.89 3.84 6.26 11.58


D1024 3.88 6.55 11.65 22.13Graph

PVANIV: Pivot, Analytic, View

Query TextWITH wit AS (SELECT emp.first_name,

emp.last_name,ilv.phone_type,ilv.phone_number

FROM employees emp LEFT JOIN (

SELECT pho.employee_id,pho.phone_type,pho.phone_number,Row_Number() OVER (PARTITION BY pho.employee_id, pho.phone_type ORDER BY pho.valid_from

DESC,pho.phone_number) ind FROM phone_numbers pho

) ilv ON ilv.employee_id = emp.employee_id WHERE ilv.ind = 1)SELECT /* PVANIV*/'"' || first_name || ' ' || last_name || '","' || h || '","' || w || '","' || m || '","' || f || '3212"' FROM wit PIVOT (MAX(phone_number) FOR phone_type IN ('HOME' AS h, 'WORK' AS w, 'MOBILE' AS m, 'FAX' AS f))ORDER BY 1

Query Diagram



Notice that the final cardinality estimate is exactly right, although how the CBO gets there from id 3 to id 2 below is hard to understand.

-----------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |-----------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | | 66739 (100)| || 1 | SORT ORDER BY | | 13696 | 775K| 240M| 66739 (1)| 00:13:21 || 2 | HASH GROUP BY PIVOT | | 13696 | 775K| 240M| 66739 (1)| 00:13:21 ||* 3 | HASH JOIN | | 3480K| 192M| | 32009 (1)| 00:06:25 || 4 | TABLE ACCESS FULL | EMPLOYEES | 13583 | 358K| | 68 (0)| 00:00:01 ||* 5 | VIEW | | 3509K| 103M| | 31925 (1)| 00:06:24 ||* 6 | WINDOW SORT PUSHED RANK| | 3509K| 90M| 134M| 31925 (1)| 00:06:24 || 7 | TABLE ACCESS FULL | PHONE_NUMBERS | 3509K| 90M| | 5265 (1)| 00:01:04 |-----------------------------------------------------------------------------------------------------


3 - access("ILV"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 5 - filter("ILV"."IND"=1) 6 - filter(ROW_NUMBER() OVER ( PARTITION BY "PHO"."EMPLOYEE_ID","PHO"."PHONE_TYPE" ORDER BY INTERNAL_FUNCTION("PHO"."VALID_FROM") DESC ,"PHO"."PHONE_NUMBER")<=1)

Results



Graph


Query W1 W2 W4 W8D1 0.11 0.14 0.19 0.33D2 0.11 0.14 0.22 0.36D4 0.11 0.15 0.21 0.36D8 0.14 0.16 0.25 0.42D16 0.12 0.18 0.31 0.54


D32 0.18 0.26 0.46 0.78D64 0.3 0.42 0.78 1.57D128 0.39 0.78 1.54 3.04D256 0.77 1.53 3.17 6.3D512 1.5 3.18 6.66 10.01D1024 3.32 6.81 10.02 18.25

Graph

PVKP: Pivot, Keep

Query TextSELECT /* PVKP*/'"' || emp_name || '","' || h || '","' || w || '","' || m || '","' || f || '4895"' FROM (

SELECT emp.first_name || ' ' || emp.last_name emp_name,pho.phone_type,MAX (pho.phone_number) KEEP (DENSE_RANK LAST ORDER BY pho.valid_from) phone_number_last

FROM employees emp LEFT JOIN phone_numbers pho ON pho.employee_id = emp.employee_id GROUP BY emp.first_name || ' ' || emp.last_name, pho.phone_type

) PIVOT (MAX(phone_number_last) FOR phone_type IN ('HOME' AS h, 'WORK' AS w, 'MOBILE' AS m, 'FAX' AS f)) ORDER BY 1

Query Diagram



Notice how poor the final cardinality estimate of 2,132,000 is, when the actual number would be the number of employees (13,696). In this case, it was found later that changing the form of the Group By to separate out the two name fields caused the cardinality estimate to improve to be about ¾ the correct value at id 4, although the execution plan did not change (see Analysis section later).

------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | | 40121 (100)| || 1 | SORT ORDER BY | | 2132K| 77M| | 40121 (1)| 00:08:02 || 2 | HASH GROUP BY PIVOT | | 2132K| 77M| | 40121 (1)| 00:08:02 || 3 | VIEW | | 2132K| 77M| | 39952 (1)| 00:08:00 || 4 | SORT GROUP BY | | 2132K| 109M| 226M| 39952 (1)| 00:08:00 ||* 5 | HASH JOIN OUTER | | 3480K| 179M| | 5348 (2)| 00:01:05 || 6 | TABLE ACCESS FULL| EMPLOYEES | 13583 | 358K| | 68 (0)| 00:00:01 || 7 | TABLE ACCESS FULL| PHONE_NUMBERS | 3509K| 90M| | 5265 (1)| 00:01:04 |------------------------------------------------------------------------------------------------


5 - access("PHO"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID")

Results


Query W1 W2 W4 W8 W16 W32 W64 W128D1 0.14 0.17 0.2 0.31 0.58 0.98 2 3.84D2 0.14 0.15 0.22 0.32 0.61 1.04 2.14 4.16D4 0.12 0.14 0.22 0.35 0.63 1.17 2.4 4.74D8 0.09 0.15 0.24 0.4 0.7 1.44 2.83 5.66D16 0.14 0.17 0.3 0.5 1 1.87 3.8 7.84D32 0.14 0.21 0.41 0.71 1.34 2.8 5.76 12D64 0.2 0.33 0.54 1.11 2.24 4.77 9.56 20.19

Graph


Query W1 W2 W4 W8D1 0.12 0.14 0.19 0.38D2 0.13 0.12 0.2 0.32D4 0.12 0.14 0.25 0.38D8 0.12 0.15 0.22 0.39D16 0.14 0.19 0.3 0.49


D32 0.14 0.24 0.37 0.68D64 0.2 0.32 0.56 1.09D128 0.28 0.49 0.92 1.88D256 0.47 0.81 1.64 3.44D512 0.77 1.58 3.12 6.85D1024 1.45 2.86 6.2 13.06

Graph

PVKPIV: Pivot, Keep, View

Query TextSELECT /* PVKPIV*/'"' || first_name || ' ' || last_name || '","' || h || '","' || w || '","' || m || '","' || f || '599"' FROM employees emp LEFT JOIN (

SELECT pho.employee_id,pho.phone_type,MAX (pho.phone_number) KEEP (DENSE_RANK LAST ORDER BY pho.valid_from) phone_number_last

FROM phone_numbers pho GROUP BY pho.employee_id, pho.phone_type

) ilv ON ilv.employee_id = emp.employee_id PIVOT (MAX(phone_number_last) FOR phone_type IN ('HOME' AS h, 'WORK' AS w, 'MOBILE' AS m, 'FAX' AS f)) ORDER BY 1

Query Diagram



Notice that the final cardinality estimate is almost four times the actual number, while the intermediate estimates from id 7 up to id 3 are about right. It seems that the CBO has not allowed for the pivoting operation reducing cardinality.

------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | | 15867 (100)| || 1 | SORT ORDER BY | | 38419 | 1688K| 2128K| 15867 (2)| 00:03:11 || 2 | HASH GROUP BY PIVOT | | 38419 | 1688K| 2128K| 15867 (2)| 00:03:11 ||* 3 | HASH JOIN OUTER | | 38419 | 1688K| | 14988 (2)| 00:03:00 || 4 | TABLE ACCESS FULL | EMPLOYEES | 13583 | 358K| | 68 (0)| 00:00:01 || 5 | VIEW | | 38739 | 680K| | 14919 (2)| 00:03:00 || 6 | SORT GROUP BY | | 38739 | 1021K| 134M| 14919 (2)| 00:03:00 || 7 | TABLE ACCESS FULL| PHONE_NUMBERS | 3509K| 90M| | 5265 (1)| 00:01:04 |------------------------------------------------------------------------------------------------ Predicate Information (identified by operation id): --------------------------------------------------- 3 - access("ILV"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID")

Results



Graph


Query W1 W2 W4 W8D1 0.11 0.14 0.18 0.32D2 0.11 0.15 0.2 0.33D4 0.11 0.18 0.2 0.34D8 0.12 0.15 0.22 0.39D16 0.13 0.18 0.25 0.42D32 0.12 0.19 0.29 0.53


D64 0.18 0.25 0.42 0.73D128 0.19 0.32 0.61 1.17D256 0.29 0.52 1.01 2.05D512 0.45 0.9 1.84 4.04D1024 0.83 1.69 3.45 7.26

Graph

FNSC: Database Function

Query Text - MainSELECT /* FNSC*/ '"' || emp.first_name || ' ' || emp.last_name || '","' ||

Query_Test_Set.Last_Phone_Number (emp.employee_id, 'HOME') || '","'Query_Test_Set.Last_Phone_Number (emp.employee_id, 'WORK') || '","' ||Query_Test_Set.Last_Phone_Number (emp.employee_id, 'MOBILE') || '","' ||Query_Test_Set.Last_Phone_Number (emp.employee_id, 'FAX') || '4324"'

FROM employees empORDER BY 1

Query Text - FunctionSELECT /* FUNCTIONSQ */

phone_number FROM (SELECT pho.phone_number FROM phone_numbers pho WHERE pho.phone_type = p_phone_type AND pho.employee_id = p_employee_id ORDER BY pho.valid_from DESC) WHERE ROWNUM = 1


Query/Function Diagram

Execution Plan Example - Main---------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | ---------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | | 175 (100)| | | 1 | SORT ORDER BY | | 13583 | 358K| 544K| 175 (2)| 00:00:03 | | 2 | TABLE ACCESS FULL| EMPLOYEES | 13583 | 358K| | 68 (0)| 00:00:01 | ----------------------------------------------------------------------------------------

Execution Plan Example – Function

It was not possible to obtain the execution plan for the function from the v$ tables, for some reason, so we did an Explain Plan on the query with typical bind values.

---------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |---------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | 12 (100)| ||* 1 | COUNT STOPKEY | | | | | || 2 | VIEW | | 2 | 12 | 12 (0)| 00:00:01 ||* 3 | TABLE ACCESS BY INDEX ROWID | PHONE_NUMBERS | 1029 | 27783 | 12 (0)| 00:00:01 |


|* 4 | INDEX RANGE SCAN DESCENDING| PHONE_NUMBERS_N1 | 8 | | 3 (0)| 00:00:01 |--------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(ROWNUM=1) 3 - filter("PHO"."PHONE_TYPE"=:B2) 4 - access("PHO"."EMPLOYEE_ID"=:B1)

Results



Graph


Query W1 W2 W4 W8D1 0.3 0.58 0.81 1.44D2 0.34 0.58 0.77 1.48D4 0.38 0.78 0.84 1.46D8 0.47 0.44 0.83 1.45D16 0.28 0.47 0.83 1.46D32 0.27 0.48 0.78 1.47D64 0.25 0.44 0.84 1.48D128 0.25 0.42 0.78 1.45D256 0.25 0.43 0.76 1.47D512 0.26 0.42 0.81 1.74D1024 0.28 0.43 0.78 2.11

Graph


SSKP: Select Scalar Subquery, KeepThis query was added after analysis of the main results, and only run for a single data set point, W1-D1024.

Query TextSELECT /* SSKP*/ '"' || emp.first_name || ' ' || emp.last_name || '","' ||

(SELECT MAX (phone_number) KEEP (DENSE_RANK LAST ORDER BY valid_from) FROM phone_numbers WHERE phone_type = 'HOME' AND employee_id = emp.employee_id) || '","' ||(SELECT MAX (phone_number) KEEP (DENSE_RANK LAST ORDER BY valid_from) FROM phone_numbers WHERE phone_type = 'WORK' AND employee_id = emp.employee_id) || '","' ||(SELECT MAX (phone_number) KEEP (DENSE_RANK LAST ORDER BY valid_from) FROM phone_numbers WHERE phone_type = 'MOBILE' AND employee_id = emp.employee_id) || '","' ||(SELECT MAX (phone_number) KEEP (DENSE_RANK LAST ORDER BY valid_from) FROM phone_numbers WHERE phone_type = 'FAX' AND employee_id = emp.employee_id) ||'133"'

FROM employees emp ORDER BY 1


Query Diagram


Notice that the cardinalities look about right here, and the plan is the same as for SSSM.------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | 69 (100)| || 1 | SORT AGGREGATE | | 1 | 26 | | ||* 2 | TABLE ACCESS FULL | PHONE_NUMBERS | 1028 | 26728 | 619 (1)| 00:00:08 || 3 | SORT AGGREGATE | | 1 | 26 | | ||* 4 | TABLE ACCESS FULL | PHONE_NUMBERS | 1023 | 26598 | 619 (1)| 00:00:08 || 5 | SORT AGGREGATE | | 1 | 26 | | ||* 6 | TABLE ACCESS FULL | PHONE_NUMBERS | 1037 | 26962 | 619 (1)| 00:00:08 || 7 | SORT AGGREGATE | | 1 | 26 | | ||* 8 | TABLE ACCESS FULL| PHONE_NUMBERS | 1028 | 26728 | 619 (1)| 00:00:08 || 9 | SORT ORDER BY | | 107 | 2033 | 69 (2)| 00:00:01 || 10 | TABLE ACCESS FULL | EMPLOYEES | 107 | 2033 | 68 (0)| 00:00:01 |------------------------------------------------------------------------------------------

Results

The query ran in 44 seconds CPU time. Compare with FNSC, which took 0.3 seconds for this data set point.

SSSM: Select Scalar Subquery, MaxThis query was added after analysis of the main results, and only run for a single data set point, W1-D1024.

Query TextSELECT /* SSKP*/ '"' || emp.first_name || ' ' || emp.last_name || '","' ||

(SELECT Substr (Max (To_Char (valid_from,'j') || phone_number), 8) FROM phone_numbers WHERE phone_type = 'HOME' AND employee_id = emp.employee_id) || '","' ||(SELECT Substr (Max (To_Char (valid_from,'j') || phone_number), 8) FROM phone_numbers WHERE phone_type = 'WORK' AND employee_id = emp.employee_id) || '","' ||(SELECT MAX Substr (Max (To_Char (valid_from,'j') || phone_number), 8) FROM phone_numbers WHERE phone_type = 'MOBILE' AND employee_id = emp.employee_id) || '","' ||(SELECT Substr (Max (To_Char (valid_from,'j') || phone_number), 8) FROM phone_numbers


WHERE phone_type = 'FAX' AND employee_id = emp.employee_id) ||'133"'


Query Diagram


Notice that the cardinalities look about right here, and the plan is the same as for SSKP.------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | 69 (100)| || 1 | SORT AGGREGATE | | 1 | 25 | | ||* 2 | TABLE ACCESS FULL | PHONE_NUMBERS | 1028 | 25700 | 619 (1)| 00:00:08 || 3 | SORT AGGREGATE | | 1 | 25 | | ||* 4 | TABLE ACCESS FULL | PHONE_NUMBERS | 1023 | 25575 | 619 (1)| 00:00:08 || 5 | SORT AGGREGATE | | 1 | 25 | | ||* 6 | TABLE ACCESS FULL | PHONE_NUMBERS | 1037 | 25925 | 619 (1)| 00:00:08 || 7 | SORT AGGREGATE | | 1 | 25 | | ||* 8 | TABLE ACCESS FULL| PHONE_NUMBERS | 1028 | 25700 | 619 (1)| 00:00:08 || 9 | SORT ORDER BY | | 107 | 2033 | 69 (2)| 00:00:01 || 10 | TABLE ACCESS FULL | EMPLOYEES | 107 | 2033 | 68 (0)| 00:00:01 |------------------------------------------------------------------------------------------

Results

The query ran in 46 seconds CPU time. Compare with FNSC, which took 0.3 seconds for this data set point (but 0.7s in the last run in Test_Phone_1-11.LST).

SSOB: Select Scalar Subquery, Order ByThis query was added after analysis of the main results, and is not valid in Oracle SQL due to a technical limitation whereby an alias can only be referenced at one level down within a scalar subquery. It is of interest here, because had it been valid it would probably be the most efficient method for narrow deep cases, from consideration of FNSC.

Query TextSELECT /* SSOB*/ '"' || emp.first_name || ' ' || emp.last_name || '","' ||

(SELECT pho.phone_number FROM(SELECT pho.phone_number FROM phone_numbers pho WHERE pho.phone_type = 'HOME' AND pho.employee_id = emp.employee_id ORDER BY pho.valid_from DESC) WHERE ROWNUM = 1


) || '","' ||(SELECT pho.phone_number FROM(SELECT pho.phone_number FROM phone_numbers pho WHERE pho.phone_type = 'WORK' AND pho.employee_id = emp.employee_id ORDER BY pho.valid_from DESC) WHERE ROWNUM = 1) || '","' ||(SELECT pho.phone_number FROM(SELECT pho.phone_number FROM phone_numbers pho WHERE pho.phone_type = 'MOBILE' AND pho.employee_id = emp.employee_id ORDER BY pho.valid_from DESC) WHERE ROWNUM = 1) || '","' ||(SELECT pho.phone_number FROM(SELECT pho.phone_number FROM phone_numbers pho WHERE pho.phone_type = 'FAX' AND pho.employee_id = emp.employee_id ORDER BY pho.valid_from DESC) WHERE ROWNUM = 1) ||'999"'


Query Diagram

Results

The query fails with error:

ORA-00904: "EMP"."EMPLOYEE_ID": invalid identifier


Analysis of Results

Slice AnalysisWe would like to like to compare the performances of the queries graphically, but as it would be difficult to do this using the 3-dimensional graphs above, we will instead consider the four slices obtained by fixing one of the width and depth dimensions at its minimum and maximum with the other displayed across its range. For each slice and data set the CPU times are tabulated across all queries and a line graph is given, then the relative performances are assessed in an analysis subsection.

Narrow Slice


Query D1 D2 D4 D8 D16 D32 D64FNSC 0.35 0.36 0.41 0.46 0.28 0.28 0.31PVAN 0.15 0.11 0.13 0.14 0.15 0.19 0.29PVANIV 0.11 0.11 0.11 0.13 0.14 0.21 0.23PVKP 0.14 0.14 0.12 0.09 0.14 0.14 0.2PVKPIV 0.13 0.14 0.11 0.14 0.13 0.1 0.16WJAN 0.14 0.14 0.14 0.14 0.16 0.17 0.34WJKP 0.15 0.15 0.14 0.14 0.15 0.15 0.2WJSQ 0.13 0.14 0.15 0.16 0.16 0.22 0.3JNSQ 1.35 1.29 1.51 1.89 2.95 5.98 14.59

Graph of CPU Times, excluding JNSQ


Query D1 D2 D4 D8 D16 D32 D64 D128 D256 D512 D1024FNSC 0.3 0.34 0.38 0.47 0.28 0.27 0.25 0.25 0.25 0.26 0.28PVAN 0.14 0.12 0.11 0.12 0.15 0.17 0.27 0.46 0.9 1.89 3.88PVANIV 0.11 0.11 0.11 0.14 0.12 0.18 0.3 0.39 0.77 1.5 3.32PVKP 0.12 0.13 0.12 0.12 0.14 0.14 0.2 0.28 0.47 0.77 1.45PVKPIV 0.11 0.11 0.11 0.12 0.13 0.12 0.18 0.19 0.29 0.45 0.83WJAN 0.15 0.14 0.14 0.14 0.17 0.18 0.24 0.42 0.8 1.56 3.36WJKP 0.14 0.14 0.14 0.16 0.14 0.15 0.2 0.23 0.36 0.48 0.82WJSQ 0.19 0.19 0.14 0.15 0.17 0.22 0.28 0.44 0.62 1.11 1.43

Graph of CPU Times


AnalysisWide Data Set

This is a subset of the deep data set for the narrow slice (other than having wider date range), but was the only set that JNSQ was tested on (to save time).

JNSQ took 91 x best time at the maximum depth

Deep Data Set

FNSC is worst up to depth of 32 but best from 256, taking 34% of the next best time at maximum depth

FNSC shows essentially constant time; this is presumably because the main query scans the unchanging employees table, while the function query uses the index and is coded to stop after the first row and so may be able to obtain the rowid in the same small number of reads.

PVPKIV and WJKP show similar performance as next best at 2.9 x best

PVKP and WJSQ show similar performance as next best at 5.1 x best

PVANIV AND WJAN show similar performance as next best at 11.9 x best, with PVAN just behind at 13.9

Wide Slice

Wide Data Set – W128Table of CPU Times

Query D1 D2 D4 D8 D16 D32 D64FNSC 21.98 22.13 22.75 22.68 22.9 23.59 28.61PVAN 3.9 4.28 4.98 6.52 9.41 15.79 29.13PVANIV 3.81 4.17 4.7 5.91 8.19 13.03 22.97PVKP 3.84 4.16 4.74 5.66 7.84 12 20.19PVKPIV 3.83 3.97 4.23 4.79 5.86 7.79 11.99WJAN 4.31 4.39 4.97 6.19 8.72 12.49 21.31WJKP 4.16 4.04 4.4 4.93 6.07 8.08 12.29WJSQ 4.44 4.62 5.05 6.14 8.7 13.22 22.62JNSQ 13.53 34.59 67.66 298.62 324.23



Deep Data Set – W8Table of CPU Times

Query D1 D2 D4 D8 D16 D32 D64 D128 D256 D512 D1024FNSC 1.44 1.48 1.46 1.45 1.46 1.47 1.48 1.45 1.47 1.74 2.11PVAN 0.33 0.34 0.37 0.47 0.67 1 1.94 3.65 6.03 11.58 22.13PVANIV 0.33 0.36 0.36 0.42 0.54 0.78 1.57 3.04 6.3 10.01 18.25PVKP 0.38 0.32 0.38 0.39 0.49 0.68 1.09 1.88 3.44 6.85 13.06PVKPIV 0.32 0.33 0.34 0.39 0.42 0.53 0.73 1.17 2.05 4.04 7.26WJAN 0.32 0.39 0.42 0.44 0.61 0.89 1.58 3.09 6.44 9.89 19.2WJKP 0.38 0.36 0.36 0.39 0.5 0.53 0.8 1.21 2 4.18 7.31WJSQ 0.38 0.41 0.42 0.48 0.61 0.88 1.42 2.5 4.59 9.54 12.59

Graph of CPU Times


This is a subset of the deep data set for the wide slice (other than having wider date range), but was the only set that JNSQ was tested on (to save time).

JNSQ took 55 x best time at the maximum depth it was run at (D16)

Deep Data Set

FNSC is worst up to depth of 32 but best from 256, taking 29% of the next best time at maximum depth


FNSC shows essentially constant time for the first 9 depths; this is presumably because the main query scans the unchanging employees table, while the function query uses the index and is coded to stop after the first row and so may be able to obtain the rowid in the same small number of reads. The last two depths show increases of 18% and a further 21%; possibly the greater sizes of the index have increased the index depth and hence the numbers of blocks read, but I have not verified this.

PVPKIV and WJKP show similar performance as next best at 3.5 x best at maximum depth

PVKP and WJSQ show similar performance as next best at 6.2 x best

PVANIV AND WJAN show similar performance as next best at 8.9 x best, with PVAN just behind at 10.5

Apart from FNSC, the relative performances remain largely similar after divergence appears from about D8

Shallow Slice


Query W1 W2 W4 W8 W16 W32 W64 W128FNSC 0.35 0.56 0.87 1.49 2.85 5.6 12.02 21.98PVAN 0.15 0.15 0.19 0.32 0.56 1.08 2.07 3.9PVANIV 0.11 0.12 0.21 0.33 0.61 1.06 2.02 3.81PVKP 0.14 0.17 0.2 0.31 0.58 0.98 2 3.84PVKPIV 0.13 0.16 0.21 0.35 0.57 1.01 1.95 3.83WJAN 0.14 0.14 0.24 0.34 0.64 1.06 2.11 4.31WJKP 0.15 0.18 0.23 0.36 0.62 1.09 2.07 4.16WJSQ 0.13 0.22 0.24 0.37 0.67 1.17 2.16 4.44JNSQ 1.35 1.73 2 2.15 2.34 4.01 7.3 13.53

Graph of CPU Times


Query W1 W2 W4 W8FNSC 0.3 0.58 0.81 1.44PVAN 0.14 0.14 0.19 0.33PVANIV 0.11 0.14 0.19 0.33PVKP 0.12 0.14 0.19 0.38PVKPIV 0.11 0.14 0.18 0.32WJAN 0.15 0.2 0.27 0.32WJKP 0.14 0.14 0.21 0.38


WJSQ 0.19 0.21 0.23 0.38Graph of CPU Times


JNSQ is worst up to depth of 8, then second worst to FNSC, with JNSQ and FNSC taking 3.6 and 5.8 times the best time at maximum width

For the rest, performances divide into two groups:

o PVAN, PVANIV, PVKP and PVPKIV are best, taking about 90% of the times of the next group at maximum width

o WJAN, WJKP and WJSQ take about 1.1 x the times of the best group

Deep Data Set

This is a subset of the wide data set for the shallow slice.

Deep Slice

Wide Data Set – D64Table of CPU Times

Query W1 W2 W4 W8 W16 W32 W64 W128FNSC 0.31 0.5 0.78 1.53 3.04 5.88 11.72 28.61PVAN 0.29 0.47 0.97 1.9 3.84 6.88 13.67 29.13PVANIV 0.23 0.46 0.78 1.63 3.2 6.8 10.69 22.97PVKP 0.2 0.33 0.54 1.11 2.24 4.77 9.56 20.19PVKPIV 0.16 0.24 0.4 0.73 1.47 3.14 5.97 11.99WJAN 0.34 0.45 0.85 1.65 3.41 6.71 10.42 21.31WJKP 0.2 0.3 0.48 0.78 1.53 3.04 6.05 12.29WJSQ 0.3 0.42 0.79 1.42 2.73 5.7 11.6 22.62JNSQ 14.59 59.48 159.65 292.08



Deep Data Set – D1024Table of CPU Times

Query W1 W2 W4 W8FNSC 0.28 0.43 0.78 2.11PVAN 3.88 6.55 11.65 22.13PVANIV 3.32 6.81 10.02 18.25PVKP 1.45 2.86 6.2 13.06PVKPIV 0.83 1.69 3.45 7.26WJAN 3.36 6.94 9.86 19.2WJKP 0.82 1.7 3.51 7.31WJSQ 1.43 2.76 5.74 12.59

Graph of CPU Times


JNSQ is worst in all cases where it was run, having 96 times the best time at the maximum width it was run at (W32)

For the rest, performances divide into three groups:

o PVPKIV and WJKP are best, taking about 57% of the times of the next group at maximum width

o PVANIV, PVKP, WJAN, and WJSQ take about 1.8 x the times of the best group


o FNSC and PVAN take about 2.4 x the times of the best group

Deep Data Set

This is a subset of the wide data set for the shallow slice.

Summary AnalysisIn this section, the relative rankings and performance factors are tabulated across the query sets and slices for both data sets.

Tables and Graphs of Relative PerformanceTable of Rankings

DataSet> Wide DeepSlice -> Shallow Deep Narrow WideEnd -> Narrow Wide Narrow Wide Shallow Deep Shallow Deep

FNSC 8 9 6 7 8 1 8 1PVAN 6 1 5 7 4 8 1 8PVANIV 1 1 4 5 1 6 1 6PVKP 4 1 2 3 3 4 5 4PVKPIV 2 1 1 1 1 2 1 2WJAN 4 5 8 4 6 6 1 6WJKP 6 5 2 1 4 2 5 2WJSQ 2 6 6 5 7 4 5 4JNSQ 9 8 9 9 NA NA NA NA

Table of CPU Time Factors Relative to BestDataSet> Wide DeepSlice -> Shallow Deep Narrow WideEnd -> Narrow Wide Narrow Wide Shallow Deep Shallow Deep

FNSC 3.2 5.8 1.9 2.4 2.7 1 4.5 1PVAN 1.4 1 1.8 2.4 1.3 13.9 1 10.5PVANIV 1 1 1.4 1.9 1 11.9 1 8.6PVKP 1.3 1 1.3 1.7 1.1 5.2 1.2 6.2PVKPIV 1.2 1 1 1 1 3 1 3.4WJAN 1.3 1.1 2.1 1.8 1.4 12 1 9.1WJKP 1.4 1.1 1.3 1 1.3 2.9 1.2 3.5WJSQ 1.2 1.2 1.9 1.9 1.7 5.1 1.2 6JNSQ 12.3 3.6 91.2 0 NA NA NA NA

Graph of CPU Time Factors Relative to Best – Start (JNSQ excluded)

Graph of CPU Time Factors Relative to Best – End (JNSQ excluded)


Which Query is Best?

PVKPIV is best on 5 of the 8 data set points considered in the summary analysis above, and is significantly worse than only one other query, FNSC, on the two deepest data set points, where it is about equal second with WJKP. A couple of further points may be noted:

It is not surprising that the query performing best overall (PVKPIV) uses Oracle’s native syntax for both pivoting and pruning, being better in particular than those pruning by use of the more general technique of analytic functions. It is very surprising though that the inline view modifier is needed to obtain the best performance, as is the result that using a database function outperforms all others in some scenarios. These points are discussed in the next section.

WJKP shows a very similar performance profile to PVKPIV, but slightly worse; it seems that the execution plans, although superficially different, may amount to much the same but with the overhead of writing to the temporary table likely explaining the differences.

Cost Based OptimizerThe results throw light on the performance of the CBO. In this section we consider what they imply for query construction in general, as well as for our test scenarios. It should be noted that our results are for a single class of queries, and it may be that the CBO happens to be particularly well or badly suited to that class compared with other classes.

CBO Cardinalities

In order for the CBO to compare candidate execution plans effectively, it is of course very important that its cardinality estimates be reasonably good. However, it can be seen from the sections on query execution plans above that the accuracy of the cardinality estimates is frequently very poor: It seems, in some cases, to have trouble with both the pivoting and pruning operations. For example, with PVKPIV it gets the pruning side right, but not the pivoting. It is worth noting that the use of the inline view seems to help it get its estimates right. For example, with PVAN, it does not reduce cardinalities at all for either operation, while with PVANIV it does much better.

The PVKP query revealed an interesting CBO problem with the Group By clause. In the query as shown, the Group By for the main view merely copies the grouping fields from the select list, as is usual:

GROUP BY emp.first_name || ' ' || emp.last_name, pho.phone_type

By experiment, I found that replacing the employee names expression with its components led to the CBO obtaining an accurate estimate.

GROUP BY emp.first_name, emp.last_name, pho.phone_type

Note that the two forms are equivalent, as Oracle clearly knows at the syntax-checking level, or it would raise the error: ‘ORA-00979: not a GROUP BY expression’

The two forms were tested on the W1-D1024 data set point, along with PVKPIV (see Test_Phone_1-11_3.LST for full listing), and the two explain plans were:


Original------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | | 2573 (100)| || 1 | SORT ORDER BY | | 18565 | 688K| | 2573 (2)| 00:00:31 || 2 | HASH GROUP BY PIVOT | | 18565 | 688K| | 2573 (2)| 00:00:31 || 3 | VIEW | | 18565 | 688K| | 2570 (1)| 00:00:31 || 4 | SORT GROUP BY | | 18565 | 815K| 24M| 2570 (1)| 00:00:31 ||* 5 | HASH JOIN OUTER | | 433K| 18M| | 690 (2)| 00:00:09 || 6 | TABLE ACCESS FULL| EMPLOYEES | 107 | 2033 | | 68 (0)| 00:00:01 || 7 | TABLE ACCESS FULL| PHONE_NUMBERS | 433K| 10M| | 619 (1)| 00:00:08 |------------------------------------------------------------------------------------------------



Modified----------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |----------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | 708 (100)| || 1 | SORT ORDER BY | | 303 | 11514 | 708 (4)| 00:00:09 || 2 | HASH GROUP BY PIVOT | | 303 | 11514 | 708 (4)| 00:00:09 || 3 | VIEW | | 303 | 11514 | 706 (4)| 00:00:09 || 4 | SORT GROUP BY | | 303 | 13635 | 706 (4)| 00:00:09 ||* 5 | HASH JOIN OUTER | | 433K| 18M| 690 (2)| 00:00:09 || 6 | TABLE ACCESS FULL| EMPLOYEES | 107 | 2033 | 68 (0)| 00:00:01 || 7 | TABLE ACCESS FULL| PHONE_NUMBERS | 433K| 10M| 619 (1)| 00:00:08 |----------------------------------------------------------------------------------------



The timings were:

PVKP (original) 1.43

PVKPG (modified) 1.31

PVKPIV .84

Although the plans appear the same, the modified query appeared to be consistently slightly faster (Is the plan output hiding some differences ‘under the covers’? Incidentally, if the reader has not yet read Under the Net, the first, maybe best, book by the late English writer Iris Murdoch, it’s highly recommended). The change thus made little difference here, but in another query it might.

Inline View Modifier

Using an inline view to enclose the pruning operation is always at least as good as not, and up to 21% faster for the analytics strategy, and 59% faster for the keep strategy. The view guides the CBO to reduce the number of records being processed before joining them to the master records. This is extremely surprising because one would expect the CBO to work this out for itself, especially on our deliberately simple queries.

Subquery Factoring Modifier

Applying the subquery factor modifier proved extremely effective for the join subquery strategy, both in performance and in allowing outer joining.

It is interesting to note from the detailed timings in the output files (SQL Pivot and Prune Queries – Output) that, while Oracle normally defers execution of a query until the first fetch shows it’s needed, in all cases with a subquery factor execution occurs straight away on cursor opening.

Performance of JNSQ

One of the benefits of testing the queries across a 2 dimensional domain is that one can compare performances across data set points, and one sees immediately that the CBO has performed very poorly on some of these for JNSQ. In one case, doubling the number of master records reduced the CPU time by a factor of 7!




http://www.amazon.com/Under-Net-Iris-Murdoch/dp/0140014454

Performance of Database Function and Scalar Subquery Strategies

Using database functions within queries, while quite common, is often regarded as bad practice both for performance and theoretical reasons, and usually correctly. The supposed modularity gains can generally be better obtained through other means, including high level design (can complex SQL be placed in a (possibly transactional) API rather than duplicated?) and, sometimes, views.

In our case, however, we found that the best performance obtained on narrow, deep cases was through a database function. This was because only by this means was the relevant index used to exclude almost all the detail records straight away; generally, the other strategies continued to favour full table scans. We tried to replicate this performance advantage through scalar subqueries, but the ones that were valid again neglected the obvious indexing strategy. This seems to be a case of CBO imperfection – the queries are very simple, bypassing any question of combinatorial complexity, the data distribution is random, not skewed, and statistics were gathered. It ought to have been possible for the CBO to obtain the best strategy by itself from the native syntax query (PVKP) telling it what was required, not how to do it. This type of query may be an area where the CBO is underdeveloped.

General Conclusions

Oracle’s concepts manual (REF-2) says:

In contrast to procedural languages such as C, which describe how things should be done, SQL is nonprocedural and describes what should be done

This is true for basic classes of SQL, but is not always true for more complex classes. In many cases it is necessary to use inline views together with analytic, and other, functions to build a structured query in essentially a procedural manner, in order to obtain a desired result. The ability to combine non-procedural components in this way to build very powerful queries is of course a big strength of SQL (and is why I place a high value on diagrammatic design techniques for SQL, here and in A Structured Approach to SQL Query Design).

In our relatively simple queries, we have found that using modifiers such as inline views and subquery factoring to tell the CBO how (at a high level) to process the queries, while not strictly necessary, is often beneficial to performance. This perhaps surprising finding shows that keeping an eye on performance is still important for queries in Oracle 11g.




Query Testing ProgramThis section gives design information about the testing program. The modular design allows the generic package (Test Queries in the diagram below) and the specification of the Query Test Set package to be used for any set of queries provided in the body of Query Test Set. See REF-3 for the Oracle packages referenced.

Call Structure Diagram


Data Flow Diagram

Table Structures

Generic Tables

RUN_CONTROL

Column Name Type Notesid* Number Sequence generated primary keydescription Char(500) Description of runstatus Char(1) S – success, F - failuremessage Char(4000) Error message if anypoint_wide_max Number Maximum width pointpoint_deep_max Number Maximum depth point

cpu_time Number Total CPU timeelapsed_time Number Total elapsed timecreation_date Date Creation date

RUN_STATISTICS

Column Name Type Notesrun_control_id* Number Foreign key to RUN_CONTROL tablerun_type* Char(60) Query codepoint_wide* Number Width pointpoint_deep* Number Depth pointcpu_time Number CPU timeelapsed_time Number Elapsed timecreation_date Date Creation datestatus Char(1) S – success, F - failure


message Char(4000) Error message if any

OUTPUT_LOG

Column Name Type Notesline_ind Integer Line numberline_text Char(4000) Line textid Char(30) Identifier codecreation_date Date Creation date

Specific Tables for our HR Queries

EMPLOYEESThis is the Oracle HR demo table, with 107 seeded records that we copy with name changes.

Column Name Type Notesemployee_id* Number Sequence generated primary keyfirst_name Char(20) First name – copy number appended when copying last_name Char(25) Last nameemail Char(20) Email address – copy number appended when copying for

uniqueness, but not referenced in queriesphone_number Number

Populated as seeded but not referenced in queries

hire_date Datejob_id Numbersalary Numbercommission_pct Numbermanager_id Numberdepartment_id Number

Index EMP_EMP_ID_PK

employee_id

Indexes (various, not used, see Oracle’s HR demo schema)

PHONE_NUMBERSThis is a new table in the HR schema that stores phone numbers by employee and type in records, as in Oracle Applications modules, including the CRM modules.

Column Name Type Notesid* Number Sequence generated primary keyemployee_id Number Foreign key to EMPLOYEESphone_type Char(10) Phone number type (HOME, WORK, MOBILE, FAX)valid_from Date Valid from date (set randomly within a range)valid_to Date Valid to date (not set)phone_number Char(9) Phone number (set randomly within a range)

Index PHONE_NUMBERS_N1

employee_id

valid_from

Index PHONE_NUMBERS_N2

employee_id

To_Char (valid_from, 'J')

The idea here is to ensure that the queries have reasonable indexes to use if the CBO deems it advisable, but to avoid including all fields in an index, for the purpose of this testing. The function-based index is included to provide a usable index for the subquery strategies.

Program Logic Write header record (RUN_CONTROL table)

Get data set dimensions from Query_Test_Set package


Loop over width points

o Loop over depth points

Setup test data, and gather CBO statistics, using Query_Test_Set package

Call interface

Open output file

Get query string, using Query_Test_Set package

Loop over cursor for query

o Write line to file

End loop

Write execution plan to log, using Utils.Write_Plan (OUTPUT_LOG table)

Write statistics record (RUN_STATISTICS table)

Close output file

End Call interface

o End loop

End loop

Update header record (RUN_CONTROL table)

Example OutputThe output is included in SQL Pivot and Prune Queries – Output. Here is an extract, with indentation removed, and the first 100 characters shown thereafter per line (the version of code included also writes the number of lines written, but that was added after the example below was run):

Setup data: 7-1024 Phone numbers truncated3424 numbers added, (iteration 1 of 1024)...3424 numbers added, (iteration 1024 of 1024)3506176 numbers added in total4096 numbers deleted for emp 1211024 numbers deleted for emp 196, type FAX79014 duplicate dates527835 duplicate numbers856 total employees3501056 total numbersTimings Elapsed (CPU)Delete/Add employees 0.00 (0.00)Add numbers 5,166.15 (637.35)Gather stats 55.51 (20.32)Count duplicate dates 20.94 (12.53)Count duplicate numbers 9.94 (6.27)Count totals 4.46 (0.98)Total times: 5,257.00 (677.45)(Other): 0.00 (0.00)PVKPSELECT /* PVKP*/ '"' || emp_name || '","' || h || '","' || w || '","' || m || '","' || f || '8943"'ame emp_name, pho.phone_type, Max (pho.phone_number) KEEP (DENSE_RANK LAST ORDER BY pho.valid_from) ne_numbers pho ON pho.employee_id = emp.employee_id GROUP BY emp.first_name || ' ' || emp.last_name,R phone_type IN ('HOME' AS h, 'WORK' AS w, 'MOBILE' AS m, 'FAX' AS f)) ORDER BY 1SQL_ID 9xt2by4jcdg1x, child number 0-------------------------------------SELECT /* PVKP*/ '"' || emp_name || '","' || h || '","' || w || '","'|| m || '","' || f || '8943"' FROM ( SELECT emp.first_name || ' ' ||emp.last_name emp_name, pho.phone_type, Max (pho.phone_number) KEEP(DENSE_RANK LAST ORDER BY pho.valid_from) phone_number_last FROMemployees emp LEFT JOIN phone_numbers pho ON pho.employee_id =emp.employee_id GROUP BY emp.first_name || ' ' || emp.last_name,pho.phone_type ) PIVOT (Max(phone_number_last) FOR phone_type IN('HOME' AS h, 'WORK' AS w, 'MOBILE' AS m, 'FAX' AS f)) ORDER BY 1

Plan hash value: 3462645326

------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |



------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | | 22768 (100)| || 1 | SORT ORDER BY | | 148K| 5511K| | 22768 (1)| 00:04:34 || 2 | HASH GROUP BY PIVOT | | 148K| 5511K| | 22768 (1)| 00:04:34 || 3 | VIEW | | 148K| 5511K| | 22757 (1)| 00:04:34 || 4 | SORT GROUP BY | | 148K| 7686K| 227M| 22757 (1)| 00:04:34 ||* 5 | HASH JOIN OUTER | | 3495K| 176M| | 5348 (2)| 00:01:05 || 6 | TABLE ACCESS FULL| EMPLOYEES | 856 | 22256 | | 68 (0)| 00:00:01 || 7 | TABLE ACCESS FULL| PHONE_NUMBERS | 3495K| 90M| | 5265 (1)| 00:01:04 |------------------------------------------------------------------------------------------------



Timings Elapsed (CPU)Open cursor 0.04 (0.03)First fetch 13.34 (12.78)Remaining fetches 0.10 (0.04)Write to file 0.08 (0.08)Write plan 0.12 (0.05)Total times: 13.71 (13.06)(Other): 0.03 (0.08)Summary for 8 * 107 = 856 employees with 1024 * 4 = 4096 numbers per employeeTimings Elapsed (CPU)Run_One - Total 13.72 (13.06)Total times: 13.72 (13.06)(Other): 0.00 (0.00)PVAN...


ReferencesREF Document DetailsREF-1 Oracle® Database SQL Language Reference 11g Release

2 (11.2)http://www.oracle.com/pls/db112REF-2 Oracle® Database Concepts 11g Release 2 (11.2)

REF-3 Oracle® Database PL/SQL Packages and Types Reference 11g Release 2 (11.2)

REF-4 A Structured Approach to SQL Query Design BP Furey, May 2009REF-5 Code Timing and Object Orientation and Zombies BP Furey, November 2010REF-6 SQL Pivot and Prune Queries – Output BP Furey, May 2011

REF-7 Tom Kyte’s Oracle database forum I use it generally, and got the DBMS_XPlan reference there in particular

REF-8 Under the Net Iris Murdoch, 1954REF-9 Forming Range-Based Break Groups with Advanced SQL BP Furey, June 2011



http://www.amazon.com/Under-Net-Iris-Murdoch/dp/0140014454

http://asktom.oracle.com/




http://docs.oracle.com/cd/E11882_01/appdev.112/e25788/toc.htm

http://docs.oracle.com/cd/E11882_01/appdev.112/e25788/toc.htm

http://docs.oracle.com/cd/E11882_01/server.112/e25789/toc.htm

http://www.oracle.com/pls/db112



Documents

SQL Pivot and Prune Queries - Keeping an Eye on Performance