Sybase IQ 11 - IT Collegeenos.itcollege.ee/~gseier/RS/IQ P and T Documents/iq... · Web viewSybase Adaptive Server IQ Version 12.0 An Introduction to Query Engine Internals, Diagnostics

SSYBASEYBASE A ADAPTIVEDAPTIVE S SERVERERVER IQ IQVVERSIONERSION 12.0 12.0

An Introduction to Query Engine Internals,

Diagnostics and Performance

David WalrathSybase IQ engineeringMay 8, 2023

Draft CopyPlease note many sections are

incomplete or in need of further editing, or still refer to IQ 11.x

Sybase IQ V12.0 Query Engine Internals - © 1998 Sybase Inc. Page 1

Author’s notesThis paper is intended to provide a basic introduction to Sybase IQ’s query engine and query optimizer, with some notes and comments about bugs and on how to improve performance. It is not intended to provide detailed design information about the IQ optimizer or query engine internals. A fair amount of knowledge of relational databases, SQL queries and of Sybase IQ is assumed.

This document is based on a set of three earlier papers on joins, aggregation, and predicates, and a set of slides on Sybase IQ query diagnostics. This is an unpublished, living document that it is occasionally updated with additions, clarifications, and corrections. Please send comments, corrections or suggestions to the Sybase IQ query group.

AcknowledgmentsThis document grew out of many discussions within the Sybase IQ query group, internal memos and guides, and suggestions from various members of the IQ SWAT, PSE and tech support groups. In particular, Steve Kirk of the Sybase IQ query group is the source of much of this paper, and certain parts are directly plagiarized from his memos.

Distributing this document(Sybase Internal Notice) This document contains some proprietary information about Sybase products. Please notify the author or Guy Washburn in Sybase IQ Product Management before giving this document or proprietary information in this document to interested parties outside of Sybase.

(Sybase External Notice) This document is an unpublished internal document copyright 1998 by Sybase Inc. It contains Sybase IQ Internals information and should not be copied or distributed to other parties without the permission of Sybase. Contact Sybase IQ tech support or Sybase IQ product management for copies of this document.


Table of Contents

INTRODUCTION 6

PART 1 THE IQ DATAFLOW QUERY ENGINE 8Understanding queries : printing and drawing a query tree 9

Additional Query plan information 11

Query statistics 12Interactively Showing Command Statistics 12Dumping Command Statistics to a file 13

Long Running Queries 14

PART 2 VERTICAL CURSORS, FILTERS, AND LOCAL PREDICATES 15Vertical Cursor 15

More complex Vertical Cursors 15Root Filters and FP scans 16

Reading predicates in a query plan 18

Indexes and predicates 19

IQ SET parameters which effect local predicates 20

Tuning hints and optimizations 21Views and predicates 21Inferred Predicates 21IN lists vs. OR 21Range predicates 22Factoring out common predicates 22More complex factoring 23Precedence of AND versus OR 24

PART 3 AGGREGATION 25What is aggregation? 25

Aggregation and query plans 25Group By Single node 25Group By node 25Distinct node 26Vertical Cursor aggregation node 26

Ordering of results with aggregation 26

IQ SET parameters which effect aggregation 27

Tuning Hints 28

Count(distinct) 28

PART 4 JOINS 29Query blocks and joins 29

The different join algorithms in IQ 30Nested Loop 30Hash 30Sort/Merge 31


Nested Loop Push Down 31Hash Push Down 32Cartesian Nested Loop 33Prejoin indexsets and joins 33

Join algorithm conditions and limitations 35

Good and bad join conditions 36Primary equality join conditions: 36Primary relational (non-equality) join conditions: 36Secondary (or postfilter) join conditions: 37

Tuning Join Conditions 38

Join diagnostics 39

Indexes and Joins 42

Inferred join conditions 42

Documented parameters which effect Sybase IQ joins 43

Undocumented parameters which effect Sybase IQ joins 44

Some IQ join bugs fixed in 11.2.1 46

Some IQ join bugs fixed in 11.5.0 46

Known IQ bugs 46

Other Known Join Problems and Issues 46Indexes and data skew 46Sorts and Sort-Merge join 47Star schemas, star joins, and “interesting cartesians” 47

PART 5 VIEWS AND NESTED TABLES 48Examples 48

Tuning hints 49

PART 6 FUNCTIONS 50

PART 7 SUBQUERIES 51Uncorrelated Subqueries 51

Correlated Subqueries 51Tuning 52

Subqueries, query plans, and cmdstats 53

PART 8 UNIONS 54

PART 9 PARALLELISM 55Dataflow Paralellism 55

Parallelism and joins 55

PART 10 NOTES ON OPTIMIZER DESIGN 56

PART 11 INDEXES, HASHES, AND SORTS 57Indexes 57

Hashes 57


Sorts 57

PART 12 THE IQ BUFFER MANAGER 58What happens in which cache? 58

SHMEMMB, page size, and the number of cache pages 59

APPENDIX A QUERY PLAN NODE TYPES 60

APPENDIX B AGGREGATION EXAMPLES 61

APPENDIX C SYBASE IQ 11.5.1 65

APPENDIX D SYBASE IQ 12.0 66


Introduction

Sybase IQ is a Relational Database Management System (RDBMS), like Sybase ASE, Microsoft SQL Server, Oracle, and Informix. Sybase IQ grew out of a query accelerator product based on bitmapped indexes created by a company called Expressway Technologies. Sybase bought the company with the intention of integrating the technology into Sybase SQL Server, but because some of the most useful features of the Expressway product could not be integrated easily into a traditional RDBMS, Sybase IQ was born as an independent Sybase RDBMS focused on decision support.

Sybase IQ 11.0 through 11.5.x was a separate database engine and data store from Sybase SQL Server, and shipped with a custom version of Open Server (called open_iq). Sybase IQ 11.x is closely attached to SQL Server – for example it uses a SQL Server engine as a database catalog repository, which is why SQL Server is required even for a standalone IQ server. (The database catalog is where the database stores metadata, or data about the database, such as table and column names, the storage map, user information, et al.) Another feature is if IQ cannot run a query, it can pass the query to SQL Server, which will attempt to process the query.

In 1997, a plan was begun to separate Sybase IQ from Sybase SQL Server/ASE. Instead of creating the functionality of a catalog server, Sybase IQ would be built as a DLL attached to a version of Sybase’s SQL Anywhere (now Adaptive Server Anywhere). In this version called SAINT internally (SQL Anywhere INTegration), IQ stripped out its own ODBC/DBlib/CTlib interfaces, its parser, and the interface to SQL Server, and uses SQL Anywhere for these functions. SQL Anywhere acts as a sort of front-end to Sybase IQ, parsing DDL, DML, and query statements and passing parsed commands down to Sybase IQ, as well as acting as a catalog server. Many features which IQ does not handle natively (stored procedures, embedded SQL, Java, etc.) are transparently handled by SQL Anywhere as if IQ itself supported such abilities. As part of this process, much of the query engine and optimizer was rewritten, both to support the new parse structure passed from SQL Anywhere, and to add a number of new optimizations. As part of Saint, several other features were added which effect the query engine:

- TSQL functions not supported in 11.x

- most SQL Anywhere functions

- ANSI outer joins (left, right and full outer joins)

- Hash Push Down joins, an ad-hoc join especially useful for “star” joins

- local and global temp tables

- cursors

IQ 12.0 also added a number of other enhancements and features, such as incremental backup, auto-synchronizing prejoin indexes, a single temp database space for all users, and a single database server model, which are not discussed in this paper.

Sybase IQ has several features and characteristics that give it advantages in many types of queries over traditional relational database engines, particularly for ad-hoc (unplanned) queries. Three of IQ’s most notable features are its bitmapped indexes, how IQ stores all data in vertical


indexes with no traditional base table, and how IQ compresses data better than other databases. While none of these are part of the query engine, they do have an impact on how the query engine works.

The first feature, IQ’s bitmapped indexes, are called bitwise indexes in much of the Sybase external literature; this document (and most of IQ’s development organization) use the two terms interchangeably. Other databases have or are in the process of implementing a bitmapped index type; to the best of our knowledge, all of these other databases’ bitmapped indexes correspond approximately to Sybase IQ’s LowFast (LF) index, a bitmapped index useful for low cardinality (a small number of distinct values) data columns. Sybase IQ also has two bitmapped indexes - HighNonGroup (HNG) and HighGroup (HG) indexes - that are useful for high cardinality data; these indexes include unique bitmap index technology that has been patented by Sybase. A fourth index type, the Fast Projection (FP) index (called the “default” index in the 12.0 documentation), is not a bitmapped index. A fifth index type, LowDisk, is a bitmapped index designed for medium cardinality columns; due to poor performance in many areas it is only recommended for very limited uses.

Another feature that has major implications in the database engine is how the actual data rows are stored. Most traditional databases create a base table of data, stored as sequential rows of data, possibly clustered on a key index, with other auxiliary indexes layered on top. The exact equivalent of this base table in a traditional database does not exist in IQ; a base table of rows only exists as a virtual entity in the catalog. A useful way to think of IQ is a table with the data sliced vertically, rather than horizontally. The truth is somewhat more subtle; IQ requires an index on every column, stores the data in these indexes, and can use any index type either as a traditional index for looking up values or as a data source (like a base table column) for reading the base data. A FP index is closest to the idea of a vertical slice of the base table, but inside the IQ query engine even an FP index is treated just like any other index type - simply faster at some index operations and slower at others. And even with the FP index type, many columns are not actually stored as vertical slices of raw data and may be stored in an optimized and indexed representation for performance and compression.

Sybase IQ also advertises its ability to compress data very well. A general rule of thumb is that an IQ database will usually take the same or less space than the original raw ASCII data. In practice, most databases use anywhere from 60% to 120% of the raw ASCII data. This compression can provide some performance benefits. In some cases, the compression of data will allow IQ to store more data on a page than with more traditional databases, while in other cases, only a small disk I/O is needed to read the compressed page in.

On the other hand, IQ shares many characteristics with other databases. Once processing that can be done within the IQ indexes is complete, IQ will generate rows of data from the tables’ indexes. These rows are used for joins between tables, grouping, ordering, string and math functions not handled by the indexes, and so on. When data is processed by rows the algorithms and tools IQ uses are similar to many other databases in the industry.

Within Sybase IQ, when a piece of a query is processed within the indexes, it is called vertical processing. When a piece of a query is processed by rows it is called horizontal processing. A query that can be done entirely vertically is called a vertical query, while a query that can only be done with some amount of row processing is usually called a horizontal query.


Part 1 The IQ Dataflow Query EngineUnless intercepted and processed by CIS/OMNI (not discussed in this document), IQ will accept a query from ISQL or a front-end tool, parse the SQL statement, and pass this parsed query to the IQ optimizer. As IQ is optimizing a query, it builds a “tree” of objects (joins, group by clauses, subqueries, etc.). The conceptual model of this tree in IQ is somewhat upside-down compared to a real tree. Tables are “leaves” at the bottom of the tree, and rows of data flow up the tree from the leaves to a “root” node at the top, where the data is passed out of IQ to the user. This model with a tree of processing nodes is usually called a dataflow model.

An example of a dataflow tree might look like:

IQ’s query engine is based on the Volcano dataflow engine model developed by professors McKenna & Graefe at the University of Wisconsin. Like Sybase IQ, Informix and Oracle also have dataflow engines based on the Volcano model. Sybase SQL Anywhere, ASE, and OMNI all currently use different, non-dataflow query engine models.

In Sybase IQ, this tree of dataflow objects starts executing at the root node at the top of the tree. This node starts by requesting a first row from the next object below. The node below then "wakes up", and unless the node contains a base table, it begins asking for rows from the next node below it. This continues down the tree until the execution reaches the database tables, which read the data in from the disks.

A database table in IQ is accessed through what IQ calls a vertical cursor node. A vertical cursor represents two functions in a Sybase IQ query. It first contains those local table predicates - that is, the parts of the WHERE clause which access only one table - which can be processed “vertically” within the IQ indexes. After this is done, the second function is to project rows of data from the tables. Once data rows have been projected, IQ processes the data much like other dataflow query engines.


Understanding queries : printing and drawing a query treeFor understanding how IQ processes a query, the most useful tool is the ability to print out a query plan. The method of displaying a query plan in IQ is to turn IQ’s Query_Plan parameter on ( set [temporary] option query_plan = ‘on’ ). When a query is run, IQ will write the query plan to the iq message file – “<database name>.iqmsg”.

A simple example for the query :iq set queryinfo ongoselect s_suppkey, count(ps_partkey) from partsupp, supplier where ps_suppkey = s_suppkey

and s_acctbal < 4500 and ps_availqty > 100group by s_suppkey

go

The current query has been passed to the data flow engine*** Join optimizer has been invoked (Query Block Id #1).-- Query Plan:-- 1 #05: Group By Unordered-- Child Node 1: #03-- 2 . #03: Join (Hash)-- Left Child Node: #01-- Right Child Node: #02-- Left Input Table 1: partsupp-- Right Input Table 1: supplier-- Condition 1: (partsupp.ps_suppkey = supplier.s_suppkey)-- 3 . . #01: Vertical Cursor-- Table Name: partsupp-- Table Row Count: 80000-- Condition 1: (partsupp.ps_availqty > 100)-- 3 . . #02: Vertical Cursor-- Table Name: supplier-- Table Row Count: 1000-- Condition 1: (supplier.s_acctbal < 4500)

To map this query plan into a tree, you would normally start at the top node, and work down. The top node shown is a “Group By (Hash)” node; the true top node of the tree (the “root” node) only gets printed with additional query details turned on. Each node includes a node type, a node number, and a depth in the query tree. All nodes, except for those on the very bottom (Vertical Cursor leaf nodes) will also include a list of one or more child nodes.

The node type is the name of the type of node, like a join, group by, subquery, etc. The DFO (DtataFlow Operator) node number is a unique number to identify each individual node. There are some nodes used for internal housekeeping and not displayed in a query plan, so it is not a problem to have “missing”. The child or children nodes listed in a plan are the DFO node numbers of the nodes below which feed rows of data into the current node. The depth in the query tree is mostly just formatting to help read the tree – each level of children will have an incremented depth number. There is one top node, with a depth of 1, its child or children will have a depth of 2, their children will have a depth of 3, and so forth.


Node type

Depth in the Query Tree

DFO node number

Considering all of this, to map the rest of the tree, one would take the child of the group by node at the top, which happens to be a join, and place it below the group by, and then add the two vertical cursor nodes as children of the join. The join refers to its children as left and right children; this is both for convenience of drawing the tree, and because for joins and subqueries, what gets put on the left and right sides holds some useful meaning (discussed later).

So finally the query plan written as a tree might look like:

Usually the first step in debugging a complex query performance problem is to draw the query tree, and add to the picture those bits of information (local predicates, join predicates, table size, etc.) which may be relevant to the query’s performance. Often this can show various reasons why a query plan will run slow – a subquery executed many times, a complicated disjunction (OR or UNION) of tables, a missing join condition causing a many-to-many join between two tables, or similar problems.

Once the query plan has been drawn, often the next step is to fill in the actual post-query information (which can be supplied by the command statistics discussed later). These can be things such as the actual number of rows returned by each node, the number of subquery executions, or which indexes were used to process the query

A complete list of the IQ dataflow node types is found in appendix A.


DFO #5Group By Unordered

DFO #3hash join

DFO #1Vertical Cursor

partsupp – 80K rows

DFO #2Vertical Cursor

supplier – 1K rows

Additional Query plan informationTurning the hidden Query_Detail parameter on (set [temporary] option query_detail = ‘on’) will display additional information in the query plan, such as :

- grouping expressions- output columns for each node in the tree- the dataflow Root node- view columns which are not needed and trimmed off- certain parameter settings which effect query processing or output

The plan for the previous query with both QueryInfo and QueryDetail on would look like:-- Query Plan:-- 0 #06: Root-- Child Node 1: #05-- RowCount Setting: 1-- 1 #05: Group By Unordered-- Child Node 1: #03-- Grouping Expression 1: supplier.s_suppkey-- 2 . #03: Join (Hash)-- Left Child Node: #01-- Right Child Node: #02-- Left Input Table 1: partsupp-- Right Input Table 1: supplier-- Condition 1: (partsupp.ps_suppkey = supplier.s_suppkey)-- 3 . . #01: Vertical Cursor-- Table Name: partsupp-- Table Row Count: 80000-- Condition 1: (partsupp.ps_availqty > 100)-- Index Output 1 (Projected) : partsupp.ps_partkey-- Index Output 2 (Projected) : partsupp.ps_suppkey-- 3 . . #02: Vertical Cursor-- Table Name: supplier-- Table Row Count: 1000-- Condition 1: (supplier.s_acctbal < 4500)-- Index Output 1 (Projected) : supplier.s_suppkey


Columns produced from the

two tables

Grouping expression

iq set parameters

Root node

Query statisticsWhile the query plan provides the plan for a query before it is executed in a reasonably clear, readable format, it is often useful to know what the query actually did – for example, how many times was a subquery executed, or how many rows a join actually produced. During the execution of most commands various execution statistics are collected. After a command completes, these statistics can be shown interactively or dumped to a file.

Command Statistics (cmdstats) is the IQ name for those statistics collected during a query. Command statistics are not officially documented. There are two types of statistics collected - buffer manager statistics and dataflow statistics. The buffer manager statistics show the state of the buffer caches before and after a query. Buffer statistics will not be discussed in detail here; see the Sybase IQ administrator’s guide for more information on the IQ buffer caches. The dataflow statistics are primarily the query plan with the actual query information filled in, instead of just estimates.

Statistics are kept in memory until the start of the next command for which statistics are collected.

The types of statistics collected include:

Original command string Total execution time State of all buffer managers at the start of the command State of all buffer managers at the end of the command The set of indexes used in a query, and why The number of rows returned by each DFO node The number of times a subquery was re-executed The number of times that re-execution was not needed The average time to execute a subquery The maximum time to execute a subquery Number of groups found (or average number if in a subquery) All elements shown in the query planCommands for which statistics are not collected include:

set... iq set ... iq show ... iq dump ...

Interactively Showing Command Statistics

To interactively show command statistics for a query one would type after the query was executed:

iq show cmdstats [1|2|3] [“search string”]The optional integer argument is the verbosity level, where 1 is summary information only, 2 is medium detail, and 3 is the statistics in great detail. For a moderately complex query, verbosity 1 will produce roughly one page of output, verbosity 2 will produce four pages, and verbosity 3 will produce around twelve pages of output.


Verbosity 2 is the most common level used. It should contain enough information for most preliminary analysis.

If a search string is supplied, then the rows returned are only those name field contains the specified string, which is case insensitive. For example:

iq show cmdstats “predicate”Will return rows like:

Statistic Value___________________________________________ ____________DFO #1 : Predicate: l_partkey = <bind var> HG lidx02 DFO #3 : Predicate: l_partkey = <bind var> HG lidx02 DFO #4 : Predicate: p_brand = Brand#23 LF pidx04 DFO #4 : Predicate: p_container = MED BOX LF pidx07

This is all of the rows in the CMDstats output with the word “predicate” for a query.

Useful search strings to remember include:

Rows Returned - Rows returned by each DFO node Real Reads - Running totals of actual disk reads per query bufman - buffer statistics (requires verbosity >= 2) DFO - Prefix for all data flow statistics (more information at higher verbosities) VERT - Prefix for all vertical query statistics Predicate - Index usage for answering predicates Projection - Index usage for doing projections Execution Time - In case you forgot ShowElapsedTime. Note that this field seems to be

inaccurate when run through isql or front-ends.

Dumping Command Statistics to a file

To dump the statistics from the most recent command:iq dump cmdstats “my_stats_file.dmp”

Dump writes out all the statistics from the most recent command, using a tab-delimited format that easily can be loaded into a database for further processing. If a customer believes he is having a performance problem with some query, a statistics dump file created immediately after the query completes will contain similar information to an “iq show cmdstats 3” command.

It is also possible to set IQ to dump statistics to a file for every command entered. To enable automatic continuous dumping after every command, type:

iq set CmdStatsDumpFile “filename.dmp”To turn continuous dumping mode off do:

iq set CmdStatsDumpFile “”Dumping (via the iq dump or iq set CmdStatsDumpFile command) will overwrite any previous file of that name. It will not append output to an existing file.

In practice, “iq dump” and “iq set CmdStatsDumpFile” are not frequently used because of the volume of data produced. “IQ Show CMDstats 2” is the most common method of collecting post-query statistics.


Part 2 Vertical Cursors, Filters, and local predicates

Local predicates are conditions in the WHERE clause which only access one table - as separate from join predicates and subquery predicates. Local predicates are processed in the Sybase IQ dataflow tree in one of two places:

a vertical cursor nodea filter node

The bottom, or "leaf" nodes in a dataflow query tree are always one or more vertical cursors. A vertical cursor represents a table or prejoin index along with any predicates processed in the indexes. A filter node is created when there is a local predicate that cannot be processed in the IQ indexes.

Vertical CursorA vertical cursor is the dataflow query tree’s representation of a part of an IQ query that can be processed using only IQ’s indexes.

So if a local predicate is printed out in the query plan inside a vertical cursor, that means that IQ has decided that it is able to use IQ's indexes to process that predicate. This is something like Sybase SQL Server deciding that it can satisfy a predicate using only an index without having to do a table scan. Unlike many other databases though, IQ can satisfy any number of local predicates this way without having to actually scan the data. Other databases often only can use one index, or the DBA has to set up a compound index to process multiple predicates, or the database does a table scan.

After doing the various local predicate operations, the last thing a vertical cursor does is "project" the data. This is where IQ takes the list of rows which satisfy all of the conditions and reads those elements from the FP index for each column needed, and puts the elements together into rows to pass up to the next level in the query tree. At this point, IQ processes the remainder of the query in rows, much like most other SQL databases. The biggest difference is that since IQ stores the data in columns, IQ will read only each column it needs for the query, and construct rows from only those columns to pass up to the next level in the query tree.

More complex Vertical CursorsThere are three variants of vertical cursors that can occur in query plans:

Vertical Aggregation CursorsVertical Grouping CursorsVertical Distinct Cursor

These can be generated when there are other operations than local predicates being done within the IQ indexes, such as set functions (MIN, MAX, SUM, AVG, Count), group by clauses, or distinct. This will be discussed further in the aggregation part of the document.


Filters and FP scansAs mentioned above, the last thing a vertical cursor does is project the data as rows. Above a vertical cursor, everything is processed as rows, much like any other standard database engine. In Sybase IQ 11.x, if a local predicate could not be processed easily inside one of the indexes, it was processed in a filter node above the vertical cursor – so it was row by row, essentially doing a table scan. Of course IQ would still only project data from those columns it needed for that particular query, but the scan could still be quite expensive.

Sybase IQ 12.0 still has this feature, but has added the ability to do two types of predicates using indexes that it could not before. Complex LIKE predicates and very large IN lists can now be done using the FP index to filter the table, much like any other predicate. So in Sybase IQ 11.5, if a user entered a query such as:

select name, address, phone_number from Twhere T.flavor LIKE ‘%lemon%’and T.nation = ‘Greece’

IQ would generate a query plan that would appear like:

So we would project all of the names, addresses phone numbers, and flavors for everyone in Greece. We would then throw out every row where the flavor didn’t have ‘lemon’ in the string (we would also drop the whole flavor column at the top of the filter, since we don’t need it any more). In IQ 12.0, the LIKE predicate will be pushed down into an index, eliminating the filter node, and eliminating the need to project all of the names, addresses, and phone numbers of people who don’t have ‘lemon’ in their flavor column.

This is also true for very large IN lists. In Sybase IQ 11.x, if the optimizer decided that it was too expensive to process the IN list using IQ’s bitmapped indexes, we would process it using a filter. In 12.0, IQ will use the FP index to process the in list

In Sybase IQ 12.0, the ability to process any arbitrary single-column predicates in a FP index was added. What this means is that predicates with functions can still get benefits from IQ’s indexed storage, even if not quite as good as with the bitmapped indexes. So in IQ 11.x the local predicate

where datepart(mm, Table.date) = 1would be processed as a postfilter in a filter node much like a table scan, but in 12.0 the IQ query engine is able to process it in just the FP index for that one column.

There are still places where a filter node will be used. For example the predicate:where column_1 * column_2 < 100

cannot be processed in just one IQ column index, so it would be processed as a filter on top of the vertical cursor.


Root

FilterLIKE %lemon%

Vertical Cursornation = Greece

In other cases, such as multiple levels of views, IQ today is not always quite clever enough move predicates all of the way from the top query block down to the bottom block, where it could be put into a vertical cursor.

When a filter appears directly above a vertical cursor (a filter can also appear above a view , a subquery, or some other places) it can be thought of as something like a Sybase IQ table scan. Of course IQ will still only project data from the columns it needs for that query, though a query which would require a filter node and has a “select *” could be both similar to and as expensive as a table scan in SQL Server.

In still other places the optimizer determines that it would be faster to process the predicate without using the bitmap indexes - the most common example of this would be a very long IN list.


Reading predicates in a query planWhen the parameter “QueryInfo” is turned on, IQ will display a query plan generated by the IQ optimizer. When reading a query plan, the vertical cursor and filter nodes will have several parts. For example:

SQL> select count(*) from num_p18 where num_p18_s00 IN (1,2,3,4,5,6,7,8,9,10) and num_p18_s01 > 0 and num_p18_s01 < 5000goThe current query has been passed to the data flow engine-- Query Plan:-- 1 #03: Group By Single-- Child Node 1: #02-- 2 . #02: Filter-- Child Node 1: #01-- Condition 1: (num_p18.num_p18_s00 IN (1,2,3,4,5,6))-- 3 . . #01: Vertical Cursor-- Table Name: num_p18-- Table Row Count: 5000-- Condition 1: (num_p18.num_p18_s01 > 0)-- Condition 2: (num_p18.num_p18_s01 < 5000)

In the example query above there are three predicates - an IN list, a “column > 0” and a “column < 5000”. The optimizer has decided in this example to do the two range predicates (the < and > predicates) in the bitmap indexes, and the IN list in a postfilter.

We can tell this because in the query plan, node 1 (marked with the #1) is a vertical cursor with two conditions:

-- Condition 1: (num_p18.num_p18_s01 > 0)-- Condition 2: (num_p18.num_p18_s01 < 5000)

The vertical cursor also lists the table name and table row count.

Above the vertical cursor is a filter node. The optimizer has decided that processing the IN list in a filter node is cheaper than within the available indexes. The filter lists one condition, the IN list:

-- Condition 1: (num_p18.num_p18_s00 IN (1,2,3,4,5,6))The filter also lists its child node, the vertical cursor, which is where the rows flowing into the filter come from.

Above the filter is a "Group By Single" node; this is where the count(*) is processed.


Indexes and predicatesSome indexes are better than others for certain predicates. While the chart below is not always true for all cases, it is meant as a useful tool for deciding which indexes to put on which columns. It should be fairly consistent with the published IQ documentation.

FP LF HG HNG

=, <> slow fast fast medium<, > slow fast fast fastbetween slow fast fast fastIN medium fast fast slowLIKE slow N/A N/A mediumIS NULL fast fast fast fast

Notes:

A FP (default) index will exist on every column in Sybase IQ 12.0.

In Sybase IQ 11.x, a HG index was actually two indexes combined into one. There was something called a G-Array, which was something like a LF index optimized for many values, and also a HNG index embedded inside the HG. The embedded HNG was why it was not useful to have both a HG and a HNG index on the same column.

In 12.0, the embedded HNG has been removed from the HG index. Now a HG and a LF index are almost the same thing, with the HG optimized for many distinct values and the LF optimized for smaller numbers of distinct values.

The speed of range (<, >, and between) predicates in LF and HG index is dependent on the number of unique values selected in the range. It is fast with only a few distinct values, but can eventually get quite slow with many distinct values, which is why also having a HNG index for range predicates is advised.

For IN lists, if there are a small number of elements in an IN list, IQ will use a LF or HG index. If there are many IN list values or no HG/LF index, IQ will scan the FP index to process the IN list; this is in contrast to 11.x, where the larger IN lists would be processed as filter nodes above the vertical cursor.

Like predicates can be processed in one of two ways. If the like is a right-wildcard LIKE search string (column LIKE ‘foo%’), then IQ will internally convert the LIKE into a between. Left-wildcard (%foo) and full wildcard (%foo%) LIKE patterns will be processed using an FP index. In Sybase IQ 12.0.2, if the column is large IQ may decide to process the LIKE predicate in parallel.


IQ parameters that effect local predicates

Avoid_Vertical_Predicates [defaults to OFF]If this parameter is set to ON, all predicates will be forced into filter nodes, and no predicates will be processed using the indexes.

IN_List_Filter [defaults to 0]When set to the default, the IQ query optimizer will attempt to choose whether to use the indexes or a postfilter based on the speed of the indexes available and the number of in list elements.

The valid values are: 0 - Let the optimizer choose 1 - all IN lists are forced to use filter nodes instead of being processed in the indexes. -1 - the optimizer will always process IN lists using the indexes in a vertical cursor if it is legal to do so.

Index_Preference [defaults to 0]

This parameter is exactly as described in the 12.0 documentation:The Adaptive Server IQ optimizer normally chooses the best index available to process localWHERE clause predicates and other operations which can be done within an IQ index. Thisoption is used to override the optimizer choice for testing purposes; under most circumstancesit should not be changed. The following table describes the valid values for this option andtheir action.

0 Let the optimizer choose1 Prefer LF indexes -1 Avoid LF indexes2 Prefer HG indexes -2 Avoid HG indexes3 Prefer HNG indexes -3 Avoid HNG indexes4 Prefer LD indexes -4 Avoid LD indexes5 Prefer FP indexes -5 Avoid FP indexes

Except for some places with specific optimizations to use LF or HG indexes, IQ will always project data from a FP index regardless of this parameter.

FP_Predicate_Workunit_Pages [defaults to 500]

This parameter is documented. Note that regardless of the setting for this parameter, the IQ thread manager may reduce the number of threads available for parallelism. Even with a single user and a large number of threads, IQ will assign a maximum of 8 threads per CPU.


Tuning hints and optimizations

Views and predicates

See the section on Views and Nested Tables for information on predicate performance for views.

Inferred Predicates

IQ will automatically generate some inferred predicates. What IQ does is more formally called transitive closure of equality predicates. Transitive closure simply means a SQL application of the elementary school math class transitive law where if A=B and B=C then A=C.

This means for the query:select count(*) from TABLEwhere TABLE.a = 10 and TABLE.a = TABLE.b

the IQ optimizer will also automatically add “where TABLE.b = 10”. This also works across tables, so for

select count(*) from TABLE1, TABLE2where TABLE1.a = 10 and TABLE1.a = TABLE2.a

IQ would also add “where TABLE2.a = 10”

This also works for join predicates; e.g.:select count(*) from TABLE1, TABLE2, TABLE3where TABLE1.key = TABLE2.key and TABLE2.key = TABLE3.key

IQ will consider “and TABLE1.key = TABLE3.key”, although since there only needs to be two join conditions, IQ will eliminate the one it doesn’t use.

IQ will not generate inferred predicates for inequalities (<,>,<=,>=,<>) only equality (=) conditions.

IN lists vs. OR

The following queries will be processed almost identically in IQ

select * from T where T.a IN (1,2,3,4)and

select * from T where T.a = 1 or T.a = 2 or T.a = 3 or T.a = 4

So normally there is no reason to choose one over the other. The exception is if there are a large number of elements; IQ has a different, more efficient, type of predicate processing available for long IN lists, which is not available for OR. For large IN lists the IQ query engine will basically create something like a hash join of the IN list against the FP index.

IQ supports up to 16,000 elements in an IN list, so this can be significant performance factor.


Range predicates

A pair of range predicates on a single column, such as:where table.date >= ‘1-Jan-1997’ and table.date < ‘1-Jan-1998’

can take longer to process with HNG indexes thanwhere table.date between ‘1-Jan-1997’ and ‘31-Dec-1997’

With LF or HG indexes, two separate range operations may be even more than twice as many bitmap operations than a between. This is simply because, in the first example, IQ will do the first date predicate, and then do the second predicate. If the table is not large, or there are not a lot of other busy users running queries, the bitmaps from the first range operation may stay in the cache, and the cost is only a small CPU cost. If the bitmaps do not stay in memory, and especially if the table is tens or hundreds of millions of rows, the wall-clock difference for the query may be significant.

In Sybase IQ 12.0, an optimization was added to combine range operations into a single complex BETWEEN predicate. Normal betweens are inclusive on the end points; that is “Between 1 and 3” includes both 1 and 3. By complex, what is meant is that exclusive end points can be created and used within the query engine. So internally IQ will change:

where table.date >= ‘1-Jan-1997’ and table.date < ‘1-Jan-1998’

intowhere table.date between ‘1-Jan-1997’ and ‘1-Jan-1998’ excluding rows on ‘1-Jan-1998’

In the query plan, this will show up using the mathematical notations for inclusive endpoints [ and ], and the mathematical notations for exclusive endpoints ( and ). So the prior example will appear in the query plan as:

where table.date between [‘1-Jan-1997’ and ‘1-Jan-1998’)

A somewhat more obscure variant of this optimization, which Sybase IQ will not optimize in the query engine yet, is changing:

where table.a < 10 or table.a > 20

towhere table.a not between 10 and 20

Factoring out common predicates

If a query looks something like

select * from T1, T2where T1.key = T2.keyand ((T1.nation = “Spain” and T1.food = “apple” and T2.name = “Chen” ) or (T1.nation = “India” and T1.food = “apple” and T2.name = “Chen” and T2.car= “Fiat”))


The optimizer can never be sure whether it is legal to process the local predicates before or after the join, because there are two different tables inside each of the sets of local predicates on each side of the OR disjunction. So, to be safe, IQ will process them after the join. By doing this, IQ will do a table scan of both tables, and join every row of T1 to every row of T2.

Changing the query slightly toselect * from T1, T2where T1.key = T2.keyand (T1.nation = “Spain” or (T1.nation = “India” and T2.car= “Fiat” ))and T1.food = “apple” and T2.name = “Chen”

will allow the optimizer to easily decide that T1.food and T2.name can be processed in the indexes, while the ORed predicate would still be done after the join. This can (and probably will) cause the query to run much faster.

In Sybase IQ 11.5.1 and later, as well as Sybase IQ 12.0 this factoring should happen automatically.

More complex factoring

There is a somewhat harder variant of the previous factoring example that cannot be factored as before, but still can be optimized for better performance.

The queryselect * from T1, T2where T1.key = T2.keyand (T1.nation = “Spain” and T1.food = “apple” and T2.name = “Chen” ) or (T1.nation = “India” and T1.food = “banana” and T2.name = “Smith” )

cannot be factored because there are no common identical predicates. On the other hand, the same columns are referenced on both sides of the OR, so the individual predicates can be copied into additional predicates in a different order to prefilter the tables. The trick is to do this in a way that does not filter too many or too few rows. For example, the query above can be changed to the following:

select * from T1, T2where T1.key = T2.key-- old predicatesand (T1.nation = “Spain” and T1.food = “apple” and T2.name = “Chen” ) or (T1.nation = “India” and T1.food = “banana” and T2.name = “Smith” )-- new copied predicatesand (T1.nation = “Spain” or T1.nation = “India” )and (T1.food = “apple” or T1.food = “banana”)and (T2.name = “Chen” or T2.name = “Smith” )

The new query should produce the same result, only quite a bit faster. By keeping the original predicates where they were, we get the correct results, but by adding the individual table predicates listed by table the IQ query engine can also process those predicates in the IQ indexes. This will


filter many of the rows before the join (and even before projecting the data), instead of after the join, and thus after a lot of projection and join work.

When possible, this sort of optimization should usually happen automatically in IQ 11.5.1 and later.

Precedence of AND versus OR

Much like the rules of what order multiplication, division, addition, etc. are done in a calculator or with programming, there are similar precedence rules in SQL. By precedence, we mean that, for example, using C programming rules for operator precedence, the expression A+B*C is processed as A + (B*C), unless the user explicitly uses parenthesis to change the order of the math operations.

Much like multiplication having a higher precedence than addition, in ANSI SQL, AND has a higher precedence than OR. While this is less a performance issue and more of a correctness one, a symptom is usually very poor performance in a query with an OR. The practical issue is that to get both correct answers and reasonable performance, parenthesis must often be used around ORed conditions.

For example, the queryselect count(*) from T1, T2where T2.a = T2.a and T1.b = 10 or T1.b = 20 and T2.c > 1-Jan-1998

seems reasonable. But it will be interpreted by any SQL query engine as:

select count(*) from T1, T2where ( T2.a = T2.a and T1.b = 10 ) or ( T1.b = 20 and T2.c > 1-Jan-1998 )

This will be both quite slow, and will give a different answer than what the user actually wanted. What the user more likely wants is the query:

select count(*) from T1, T2where T2.a = T2.a and ( T1.b = 10 or T1.b = 20 ) and T2.c > 1-Jan-1998

where the parenthesis forces the ORed conditions to be processed together as a single unit of work.


Part 3 Aggregation

What is aggregation?Aggregation is a generic term for where data is collected or processed into groups. In Sybase IQ 11.x that is in Group By clauses, Select Distinct queries, and any query where there is a set functions (such as MIN, MAX, SUM, AVG and Count) in the select list. Group By and Select Distinct queries generate grouping based on the columns listed in the query, while a query with a set function but no group by or select distinct will return only one grouped row.

In Sybase IQ, there are two broad approaches to aggregating data. One is to take each row read from the table, join, or other query processing object beneath the aggregation, and process the rows by sorting them, hashing them, counting them, or with other methods common to most databases. The other method of aggregation is to use Sybase IQ's indexes to process the aggregate.

A query or part of a query that is processed using IQ's indexes is often referred to in Sybase IQ as vertical processing, since we are processing the data vertically by columns. When a query is being processed by row, like other databases usually do, it is usually referred to as being processed horizontally.

Aggregation and query plansAggregation in IQ will be processed in any of four types of query plan nodes: Group By single nodes Group By nodes Distinct nodes Vertical Cursor nodesSome examples of the how different types of aggregation nodes appear in query plans are listed in Appendix C.

Group By Single node

If a query has a set function (MIN, MAX, SUM, AVG, or Count), including if there is a COUNT(distinct), SUM(distinct), or AVG(distinct) in the select list, but there is no explicit group by clause, only one group will be produced, with one line of output. If the query is processed horizontally by rows (as opposed to processing the entire query vertically inside the IQ indexes), the query plan will have a "group by single" node to indicate that there is a type of grouping, but only the one single group will be produced.

Group By node

If there is an explicit GROUP BY clause that is not processed vertically, IQ will generate a "group by" node, which processes the groups horizontally, row by row. There are two types of horizontal


Group By nodes:

Group By (Hash) – this groups the data with a hashing function

Group By (Sort) – this groups the data with a sort

If the aggregation is done horizontally, and there is a count/sum/avg(distinct) with a group by the groups will always be found by sorting the data using the grouping columns and the count/sum/avg(distinct) will always be processed with a hash for each group. For example, finding the number of different pasta brands each store stocks:

select store, count(distinct pasta_brand) from Table1 group by storeTable1 will be sorted on key “store” and the count(distinct pasta_brand) will be processed with a hash for each group.

Distinct node

"SELECT DISTINCT" queries and subqueries will generate a "distinct" node when the distinct is processed horizontally by row. There are two types of horizontal distinct nodes:

Distinct (Hash) – this finds each distinct value with a hashing function

Distinct (Sort) – this will find each distinct value by sorting the data. A Distinct Ordered does not actually know how to sort data in IQ 11.2.0 and later; the optimizer will place an Order By node underneath a Distinct Ordered to sort the data for the distinct to process.

Vertical Cursor aggregation node

If a query accesses only one table, or all of the tables in a query can be processed using one prejoin indexset, the optimizer may choose to do the aggregation within a "vertical cursor". A vertical cursor in a query plan represents a table or prejoin indexset and all of the predicates (WHERE clause conditions) and aggregates that can be processed using only Sybase IQ's indexes.

There are 4 types of vertical cursors:

Vertical Cursor - this is a "vanilla" vertical cursor without any aggregation

Vertical Grouping Cursor - this is a vertical query with a group by clause.

Vertical Distinct Cursor - this is a vertical query with a “select distinct”.

Vertical Aggregation Cursor - this will be generated if a query has set functions in the select list which can be processed vertically, but no explicit group by clause (like a group by single node).

Indexes and aggregatesA query will only consider doing grouping or distincting in the IQ indexes if there are HG or LF indexes on the grouping columns.

For SUM, AVG, MIN and MAX, if there are a small number of distinct values then a LF or HG index is sufficient, but for a larger number of unique values, a HNG index will give best performance.


Ordering of results with aggregationIf there is a “group by” or “select distinct” in a query, and the user desires the results sorted in a particular order, the only way to guarantee that order is with an explicit “order by” clause.

If a customer does not have an “order by” clause and gets the results in one particular order, there should be no expectation that the results will be displayed in that order every time the query is run. Each of the different methods will be likely to return the data from a ”group by” or “select distinct” in a different order. There are many reasons that IQ may choose a different algorithm, and therefor a different ordering from a group by or distinct. For example, rows inserted or deleted from the database, more or fewer IQ users on the system, or an upgrade to a new version of IQ with different aggregation selection code in the optimizer can change the algorithm, and thus the ordering.

IQ SET parameters that effect aggregation

Aggregation_Preference [defaults to 0]

This parameter attempts to override the preferred method for doing aggregation in Sybase IQ. If a particular choice is illegal or not supported (such as attempting to group within the bitmap indexes using an expression), then the query optimizer will ignore this setting.

The settings are: 0 allow the optimizer to choose (default)

1 prefer aggregation by sorting rows -1 avoid sort

2 prefer vertical aggregation using IQ's bitmap indexes (Vertical Grouping, Vertical Distinct, or Vertical Aggregation Cursor)-2 avoid vertical aggregation

3 prefer aggregation by hashing rows -3 avoid hash

Max_Hash_Rows [defaults to 1,250,000]

If Sybase IQ decides that it will not process the group by or distinct vertically within the indexes, the optimizer will compute an estimated number of groups or distinct values. If that estimate is below MaxHashRows, the Sybase IQ optimizer may group or distinct the data with a hash function - though other factors are also considered. If the estimate is above MaxHashRows, IQ will process the aggregate with a sort.

Min_Hash_Buckets [defaults to 64]

When deciding how big a hash table will be, the optimizer will reserve a number of pages to be “pinned” in memory for the hash table. Modifying OptimizeForThisManyUsers and MaxPinnableHashPercent can increase the number of buffers available for the optimizer to consider a hash. If the optimizer chooses a hash but underestimates the number of rows needed for the hash,


the performance may be poor because of the overhead of locking and unlocking buffers (pinning and unpinning buffers). Increasing MinHashBuckets will improve hash performance.

Users must be careful, as increasing this parameter too much may pin too many buffers in memory, which may prevent other parts of the query, or even other users, from getting enough memory for their queries.

Tuning HintsThe three methods of aggregating data (vertically in the indexes, with a hash, or with a sort) are each best in some places, and worst in others.

Grouping and select distinct within the indexes is possible if there are no joins or all joins can be processed using only one prejoin index, and if there are no functions or expressions which cannot be processed inside the IQ indexes. There is also a costing in the optimizer of vertical grouping based on I/O cost and on the efficiency of certain bitmap operations, so even when grouping within the indexes is possible, it may be done horizontally if there are many groups.

If grouping or distinct processing is not possible in the indexes, but the optimizer estimates a relatively small number of groups, hash is often the best performer. Grouping or distinct with hashing is sensitive to the amount of memory available. With one user and large amounts of TEMPSHMEMMB, grouping with a hash might work well with hundreds of thousands of groups, but with many users or small amounts of memory it may be very slow with only a few tens of thousands of groups.

Grouping or distinct processing with a sort is often the fastest algorithm if there are several hundred thousand or millions of groups. While a sort is usually not very fast, a sort is the least sensitive to large amounts of data or constrained memory. Or put another way, when a hash has plenty of memory it can be very fast, but when it runs out, it performs terribly, while a sort chugs along at a moderate speed with large or small amounts of memory.

Count(distinct)Count(distinct) used with a GROUP BY (as well as the less common sum(distinct) and avg(distinct) functions) have several special performance issues different from other aggregates.

In Sybase IQ 11.5.1 and prior versions IQ had two algorithms available for computing a count(distinct). For a vertical query, IQ can use a HG or LF index to compute the aggregate. If the query is horizontal, IQ will use a hashing function to compute the aggregate. Unfortunately, this hash has the same limitations of any other hash; if there are too many values, the hash will become quite slow.


Part 4 Joins

A join is where rows from two tables are merged together into a result table.

An ad-hoc join is a join between two tables without a predefined join index. Sybase IQ 12 supports up to 20 tables in a FROM clause for ad-hoc joins.

A prejoin in Sybase IQ is a predefined join where the join relationship is defined as part of the schema. IQ treats prejoined tables much like a single base table in most cases.

Query blocks and joins…or how many tables can be joined together

Sybase IQ will optimize the main “select” as a single entity, or query block. Each view with a DISTINCT, aggregate expression, or GROUP BY, each arm of a UNION, and each subquery referenced in the query will be optimized individually as a separate query block.

The Sybase IQ optimizer can process a query with up to 20 tables in a FROM clause. Unfortunately, there is a factorial increase in the number of plans the optimizer must consider for each additional table in a query. This means that a query with 2, 3 or even 4 tables has only a few possible plans to look at, while a query with 15 or 20 tables has potentially huge numbers of possible plans.

In Sybase IQ 11.0 through 11.5.0 a limit of 8 or 9 tables is recommended not because more will not work, but because with more tables the optimizer could take excessive amounts of time to generate a plan. If there are very few combinations of ways to connect the tables through the joins, then more tables could be reasonable. If there are many combinations of ways to join the tables (sometimes called highly connected), it may take minutes, days, weeks, or longer to generate a plan with many tables, even if the final query only takes a few minutes to process.

In Sybase IQ 11.5.1, a new algorithm for optimizing tables was added. This allowed IQ to generate a join plan for up to 20 tables in a few seconds where IQ 11.5.0 and earlier versions might take as long as millions of years (really) to generate a 16 table query plan.

IQ 12.0 added a new feature to joining tables. Views without aggregates are now flattened into a single query block. So if there is one or more security views, or even unaggregated views with joins inside, IQ will be able to put them together into one large join tree and optimize them together as it feels is best.

The different join algorithms in IQSybase IQ has five different ad-hoc join algorithms.


Nested Loop

A nested loop join in Sybase IQ takes a row from the larger side/table of the join and compares it to each row of the smaller side.

The second row from the larger side is then compared to each row of the smaller side, and so on until all of the rows are compared. For acceptable performance, it is desirable for the rows from the smaller side of the join stay in memory after the first loop to reduce I/O.

IQ rarely chooses to use nested loop joins, since one of the other join types is usually faster when they are legal.

Hash

Hash joins are like Nested Loop joins in that there is a smaller side of the join that IQ wants to keep in memory, and a larger side that is compared row by row to the smaller side. The difference is that the smaller side is "hashed" into an in-memory table with a hashing function applied to the join key(s), and the keys of the rows on the larger side are used to "probe" the hash table to find any rows where the join keys are equal. For the Hash join, there are many fewer comparisons done as

compared to a nested loop, but it adds the cost of computing the hash key for each row on both sides of the join. In most cases though, this means that

a hash join will be a win.

For a simple example, take two tables tableA and tableB, where tableA has 1,000 rows and tableB has 100,000 rows. The Nested Loop join will do 1,000 x 100,000, or 100 million comparisons. The hash will do 101,000

hash computations - once for every row to hash the smaller table and once for each row in the large table to probe.

Since both Nested Loop and Hash joins read the small table into memory and read through the large table once, the I/O costs should be the same.

Current Sybase IQ Hash joins are sometimes referred to as "classic hash" joins, since the smaller


Large Table Small Table comparisons

Large Table Small Table hash probes keys stored in hash table

side needs to be kept mostly in memory for reasonable performance - there are other types of hash joins where this is less true.

Sort/Merge

Sort/merge joins sort the rows on both sides, and then compare the rows. The comparison (merge) phase takes advantage of knowing that the rows on both sides are now in an ascending or descending order, so the join no longer has to compare every row to every other row as is done for nested loops. Sort/Merge joins are good for very large amounts of data on both sides of a join, since it the least sensitive to keeping the data from both sides of a join in memory. Sort/merge joins can also be good when the entire set of rows from each side can be kept in memory.

Sort-merge joins make use of the MBsort parameter. An amount of process memory (from what is called heap memory) equal to MBsort is allocated, and ½ of this memory is assigned to each side of the sort-merge join. IQ will then fill these pieces of memory (called sort buckets) with rows from each side of the join; when a sort bucket is filled, it will then be written out to the temp indexspace cache (and perhaps eventually to disk). So if MBsort is set to 10, each side of a sort-merge join will be allocated 5 Mbytes as a sort bucket.

The above is also why, as the IQ Administration guide states, increasing MBsort can directly effect the speed of sorts, while increasing the TEMPSHMEMMB parameter will only indirectly effect the sort performance. On the other hand, since MBsort does not come from the two caches, it is added memory overhead beyond what is used by SHMEMMB and TEMPSHMEMMB. In the Saint (12.0) release of Sybase IQ, memory for sorts in queries (sorted Group By processing as well as sort-merge joins) will come directly from the temp indexspace cache, and MBsort will no longer be used for query processing. In Sybase IQ 11.2.1 and later versions, MBsort is no longer used for Order By and sorted Select Distinct processing.

Sort merge joins are the only join type in Sybase IQ today which is parallel within the node itself - that is, each of the sorts on both sides of the join can be themselves internally parallel. The hash, nested loop, and nested loop push down joins, as well as the final comparison/merge phase of the sort/merge join, only actually do one thing at a time inside. Note this is different from being able to start up one or both children at the same time. So with a simple 2-table join, a sort/merge join could be utilizing more than one CPU, while a hash join would only be able to use one CPU for most of the join.

Nested Loop Push Down

Nested Loop Push Down joins are an unusual join type, and somewhat similar to an index nested loop join in SQL Server and other databases. It makes use of the indexes of one of the tables, so a requirement is that the larger side – that is the side with the largest estimated number of rows after local predicates are applied from the WHERE clause - is an actual base table. IQ will take each row from the smaller side of a join, and "push" the key "down" into an index on the larger side, which will return a list, called a foundset, of rows matching the join conditions. IQ then projects the rows of this foundset as the rows returned for that element of the join. A way of thinking of this join is that we turn the keys of the smaller side of the join into an IN list on the large table, though in the query plan it shows up something like a correlated subquery.


IQ is conservative in using the nested-loop push-down join in Sybase IQ 11.x. It is a difficult join to cost properly, especially where walking through the indexes many times can thrash the main indexspace cache, and because of some restrictions placed on the join type in the code. This sort of join is popular in row-based databases like Sybase SQL Server/ASE which have clustered indexes on the join key; because of the way IQ stores data it is not always as useful as in some other databases.

Future versions of Sybase IQ may make more use of this join type as the proper costs and pitfalls become better known, and as other restrictions are removed.

Hash Push Down

This join algorithm is a new combination of a hash join and IQ’s special index processing that was added to Sybase IQ 12.0. With large tables the NL-PD join has the risk of thrashing the cache with huge numbers of I/Os if the larger table does not fit into memory. On the other hand, a hash join has the disadvantage of having to read every row (after local filtering predicates) from the big table; if there are many columns, this can be quite expensive.

An example of this might be:

Two tables T1 and T2- T1 has 100,000 rows- T2 has 100 million rows, and has 20 columns for a total of 250 bytes wide

select * from T1, T2where T1.a=T2.a and T1.b between 1 and 5,000

We will pretend that the “T1.b between” predicate returns 5,000 rows, so it filters 95% of T1. We will assume that there are exactly 1,000 rows in T2 for every row in T1, so the join will filters 95% of T2, the big table. Unfortunately, T2 is 100 million * 250 bytes, or 25 GBytes. Even with each join “push down” only reading 1/100,000 of T2, the data pages needed from T2 for a NL-PD join probably won’t fit in memory, and we don’t want to have to read a lot of table T2 5,000 times if it won’t fit in memory. On the other hand, reading all 25 GBytes even once for the Hash join will also be pretty bad.

What the Hash Push Down join will do is add another local predicate to T2, the big table. The predicate will make the query something like:

select * from T1, T2where T1.a=T2.a and T1.b between 1 and 5,000 and T2.a IN (select T1.a from T1 where T1.b between 1 and 5,000)

What this will do is pre-filter all of the rows which don’t match the join key, but we are only scanning one key column, which is quite a bit smaller than the full 20 columns of T2. This “IN list” will filter 95% of T2, so we only have to read the last 5%. This last 5% from T2 will still have to be joined to T1 to match up the full rows, as well as to account for duplicates in T1’s join keys.


What is even better about this join is for the star join cases. Given a query like:Three tables T1, T2, and T2

- T1 has 100,000 rows- T2 has 50,000 rows- T3 has 100 million rows, and has 20 columns for a total of 250 bytes wide

select * from T1, T2, T3where T1.a=T3.a and T2.b=T3.b and T1.x between 1 and 5,000 and T2.y IN (‘USA’, ‘Canada’, ‘Brazil’, ‘France’, ‘Japan’, ‘Germany’, ‘China’)

The IQ optimizer might decide that neither join would make a good NL-PD join, but could turn both join keys T1.a and T2.b into IN lists on the big T3 table. So the rows read from T3 to join with T1 and T2 will only be those rows that match both sets of keys, just as if there were two big IN lists on the big T3 table. Even if IQ decided to turn one of the joins into a NL-PD join, the other could still be a Hash Push Down join. And any number of join keys on a table can be processed this way, making it a very useful join optimization for star schemas.

Cartesian Nested Loop

There is an additional join type called a cartesian nested loop which is used for joins without a join condition in the WHERE clause - that is, a many-to-many join - but this is a minor variation of a nested loop join. A many-to-many or cartesian join will produce a row for every combination of rows from both sides, so if there are 50 rows on one side and 3,000 rows on the other, the join will produce 150,000 rows.

Prejoined tables and joins

The current prejoin indexes supported by Sybase IQ essentially work by denormalizing two or more tables. Inside the Sybase IQ query engine, IQ is able to treat a joined indexset as a single table, so a query plan will not display a join dataflow node for a prejoin. It will instead have a Vertical Cursor node just like a single table - a vertical cursor represents a table and all of the predicates IQ can do in the indexes of that table - with the join condition listed like an equality (or other comparison) between two columns in a table.

Normally if a joined indexset exists, and IQ finds the prejoin, IQ will automatically use the prejoin. The optimizer will not attempt to cost a prejoin against an ad-hoc join. It is possible that in a very complex query IQ might not be able to find a prejoin due to the number of complex relationships of the joins, particularly in versions of Sybase IQ prior to 11.2.0. In 11.2.0 and later, IQ should almost always find and use a prejoin if it is available.

The current IQ prejoins seem to perform best when they are between tables of similar sizes, and become less useful as the relative table sizes becomes greater.


Join algorithm conditions and limitationsSome kinds of joins only work today with certain join algorithms. If there are multiple join conditions, IQ can consider a join algorithm using the join condition that allows the largest number of options (which is usually a simple equality join).

join type nested loop hash sort/merge nested loop push-down

Hash Push Down Join

= X X X X X<, >, <=, >= XBetween, LIKE XOuter joins X X X X

Notes:

Hash and Hash Push Down joins must be equality joins where the join fields can be directly compared or easily converted without loss before being compared. For example, "unsigned integer column = signed int column" would not be a legal hash join in IQ 11.2.x (a negative integer cannot be turned into an unsigned integer) , but "short signed integer = signed integer" is, since a signed 16 bit integer can be always converted exactly to a signed 32 bit int.

Nested-Loop Push-Down joins require one side of the join (the larger) to be a base table or prejoin indexset, and the join condition on the larger table must be a base column, not an expression or function.

Obviously some of the join algorithms could work for types of joins where they don't today, and eventually this support will be added.


Good and bad join conditions…how to tell a fast join predicate from a slow one

The different join types listed in the table in the previous section can be split into three general categories:

Primary equality join conditions

Primary relational (or inequality) join conditions

Secondary (or postfilter) join conditions

These distinctions are important to allow the optimizer the widest variety of choices of join algorithms, as will be explained below.

Primary equality join conditions:

The simplest form a primary equality join condition would be is a join predicate something like <table-1.column> = <table-2.column>. It can also include more complicated join conditions of the form <expression-1> = <expression-2>, where all tables referenced in <expression-1> come from the same input side of the join, and all tables referenced in <expression-2> come from the other input side of the join.

This is the most useful type of join condition to have, since at least one primary equality join condition is required for the IQ query engine to be able to consider using all of the four join algorithms available. The Sort-Merge join, the Nested-Loop Push-Down, Hash, and Hash Push Down join algorithms all require at least one join condition to be a primary equality join condition.

For example, an equality expression such as:

select x from foo, barwhere foo.todaysdate = bar.shipdate + bar.maxdaystoship +1

could use any join algorithm, while an expression like

select x from foo, barwhere foo.todaysdate - bar.shipdate = bar.maxdaystoship +1

could not, since both tables appear on one side of the join, while there is an expression on the other side.

Primary relational (non-equality) join conditions:

This join condition is of the form <expression-1> <rel-op> <expression-2> where the <rel-op> is one of <, <=, <>, >, or >=, and where all tables referenced in <expression-1> come from the same input side of the join, and all tables referenced in <expression-2> come from the other input side of the join.

If there are no additional primary equality join conditions, then a join with a primary relational join condition is done using a nested-loop join.

For example:


select x from foo, bar where foo.a = bar.aand foo.b < bar.b

could use any join type (since foo.a = bar.a is a primary equality join condition), butselect x from foo, barwhere foo.b <= bar.b

can only be a nested-loop join.

If a join has any primary equality join conditions, then any primary relational join conditions will normally become postfilter conditions - that is, they are used merely to filter the output from a join operator, as opposed to being used to compute the initial join result. In this case, the optimizer can use any join algorithm. The only exception to this is for Nested-Loop Push-Down joins, where additional inequality join conditions will be "pushed" into the index(s) if that join type is preferred.

A reason why inequality join conditions can be slower than equality joins isn’t just the join algorithm available, but the number of rows returned. For example, take two tables T1 and T2:

a query

select count(*) from T1, T2 where T1.date < T2.datewould return for each row of the join:

T1.row 1 (1-Jan-1995) < T2.date returns 5 rows T1.row 2 (1-Feb-1995) < T2.date returns 4 rowsT1.row 3 (1-Mar-1995) < T2.date returns 3 rowsT1.row 4 (1-Apr-1995) < T2.date returns 2 rowsT1.row 5 (1-May-1995) < T2.date returns 1 row

There are a total of 15 rows returned from the join, even though both tables are only 5 rows.

Secondary (or postfilter) join conditions:

This is any condition that is not in one of the above categories because either the join predicate is not an equality or relational predicate (for example, it is a BETWEEN or LIKE join condition), or some table is referenced on both sides of the predicate.

If no primary join conditions (equality or relational) are present, then the optimizer must use a cartesian (many-to-many) Nested-Loop join and then apply any secondary join conditions as a postfilter.

For example, if there are two tables foo and bar, with foo=1,000 rows and bar=100,000 rows, and


Table T1rowT1.date11-Jan-

199521-Feb-199531-Mar-199541-Apr-199551-May-1995

Table T2rowT1.date15-Jan-

199525-Feb-199535-Mar-199545-Apr-199555-May-1995

there was a query:

select x from foo, barwhere foo.b LIKE bar.b

The query engine would generate a nested-loop join, which would produce a 100 million row (100,000 * 1,000) cartesian product, and apply the “foo.b LIKE bar.b” in a join postfilter to each of those hundred million rows.

Another example would be:

select x from foo, barwhere foo.todaysdate - bar.shipdate = bar.maxdaystoship +1

Since table “bar” is on both sides of the join condition, the join condition would be evaluated after joining every row of foo to every row of bar.

Tuning Join ConditionsThe most important thing to remember for join conditions is that if a join that has no primary equality join conditions, the optimizer can't choose any of the faster join algorithms. A join with a primary equality join condition can be easily ten times faster (and can be hundreds of times faster) than a join without a primary equality join condition.

The performance difference between a join with only a secondary join condition and a join with a primary relational join condition is typically between 1.5x and 3x.

It is sometimes possible to rewrite a condition that cannot be used as a primary condition into one that can be. For example:

A.one = B.two - A.threeBecause table "A" appears on both sides of the equality, this condition cannot be used as a primary join condition. However, it can be used as a primary join condition if it is rewritten as:

A.one + A.three = B.two Another example:

(A.width * A.length) + (B.width * B.length) = 100 can be rewritten to be a primary equality join condition as:

(A.width * A.length) = 100 - (B.width * B.length)Also note that whenever there are more than two tables involved in a join condition, the exact phrasing of the condition can significantly constrain the set of useful query plans we can use. In some cases the best plans which could be used may not be available because of the phrasing of the join conditions. In some cases, one join condition can be a primary equality condition for some of the possible join plans, and only a secondary condition for other possible join plans.


Join diagnosticsWhen reading a Sybase IQ query plan, some things can be inferred from the joins about the “left” and “right” sides of a join. Currently if a user has the IQ set parameter “queryinfo” set ON the IQ query optimizer prints out a plan with each join having one or more "right table" tables and "left table" tables. These are the tables that produced the rows flowing into the dataflow node on that side of the join. When looking at a query plan in IQ 11.2.1 and later, what IQ believes is the small side will be the right tables for all join types.

In IQ 11.2.0 and prior versions, the smaller side will be on the “right” for hash, nested loop, and sort/merge joins, and on the “left” for nested loop push down joins only.

For example, in Sybase IQ 11.5.1, with both QueryInfo and QueryDetail on:

select count(*) from nation, customer where n_name IN ("CHINA","INDIA") and n_nationkey=c_nationkey;

The current query has been passed to the data flow engine*** Join optimizer has been invoked (Query Block Id #1).-- Query Plan:-- 0 #06: Root-- Child Node 1: #05-- 1 #05: Group By Single-- Child Node 1: #03-- 2 . #03: Join (Hash)-- Left Child Node: #01-- Right Child Node: #02-- Left Input Table 1: customer-- Right Input Table 1: nation-- Condition 1: (customer.c_nationkey = nation.n_nationkey)-- 3 . . #01: Vertical Cursor-- Table Name: customer-- Table Row Count: 150000-- Index Output 1 (Projected) : customer.c_nationkey-- 3 . . #02: Vertical Cursor-- Table Name: nation-- Table Row Count: 25-- Condition 1: (nation.n_name IN ('CHINA','INDIA'))-- Index Output 1 (Projected) : nation.n_nationkey

COUNT(*) ------------ 12066

(1 row affected)

iq show cmdstats 2 "dfo"

Statistic Value


----------------------------------------------- ---------------------------- DFO #2 : Predicate: n_name in (list) LF n_name_lf DFO #1 : Projection: c_nationkey FP c_nationkey_fp DFO #2 : Projection: n_nationkey FP n_nationkey_fp DFO #1 : Node type vp_Cursor DFO #1 : implemented as: (1) SELECT customer.c_nationkey DFO #1 : (2) FROM tpcd_scale_1.dbo.custom DFO #1 : (3) er DFO #1 : using indexset for: tpcd_scale_1.dbo.customer DFO #1 : execution count: 1 DFO #1 : execution time: 146 DFO #1 : Node type Vertical Cursor DFO #1 : Result Rows 150000 DFO #1 : Table Name customer DFO #1 : Table Row Count 150000 DFO #1 : Index Output 1 (Projected) customer.c_nationkey DFO #2 : Node type vp_Cursor DFO #2 : implemented as: (1) SELECT nation.n_nationkey, n DFO #2 : (2) ation.n_name FROM tpcd_scale DFO #2 : (3) _1.dbo.nation WHERE (nation. DFO #2 : (4) n_name IN ('CHINA','INDIA')) DFO #2 : using indexset for: tpcd_scale_1.dbo.nation DFO #2 : execution count: 1 DFO #2 : execution time: 1 DFO #2 : Node type Vertical Cursor DFO #2 : Result Rows 2 DFO #2 : Table Name nation DFO #2 : Table Row Count 25 DFO #2 : Condition 1 (1) (nation.n_name IN ('CHINA',' DFO #2 : (2) INDIA')) DFO #2 : Index Output 1 (Projected) nation.n_nationkey DFO #3 : Node type Join (Hash) DFO #3 : Result Rows 12066 DFO #3 : Left Child Node #01 DFO #3 : Right Child Node #02 DFO #3 : Left Input Table 1 customer DFO #3 : Right Input Table 1 nation DFO #3 : Condition 1 (1) (customer.c_nationkey = nati DFO #3 : (2) on.n_nationkey) DFO #5 : Node type Group By Single DFO #5 : Result Rows 1 DFO #5 : Child Node 1 #03 DFO #6 : Node type Root DFO #6 : Result Rows 1 DFO #6 : Child Node 1 #05

If there is a query where the smaller side of the join is very large for any join except sort/merge, it might indicate an optimizer performance problem (either an optimizer bug or just a poor optimizer choice). Hash, nested loop, and Nested Loop Push-Down joins all assume that one side of the join is small compared to the other side. Hash and nested loop joins both also need the smaller side of the join to stay in memory for reasonable performance.

You can tell how many rows each join node actually produces by typing

iq show cmdstats 2directly after a query (you can and should reset the rowcount parameter first if it was set to something small) and looking through the cmdstats output for the "Result Rows" from the children on each side of the join.

For example, from the previous query, the statistics for the “nation” table appear as:


DFO #2 : Node type vp_Cursor DFO #2 : implemented as: (1) SELECT nation.n_nationkey, n DFO #2 : (2) ation.n_name FROM tpcd_scale DFO #2 : (3) _3.dbo.nation WHERE (nation. DFO #2 : (4) n_name = 'JAPAN') DFO #2 : using indexset for: tpcd_scale_3.dbo.nation DFO #2 : execution count: 1 DFO #2 : execution time: 4 DFO #2 : Node type Vertical Cursor DFO #2 : Result Rows 1 DFO #2 : Table Name nation DFO #2 : Table Row Count 25 DFO #2 : Condition 1 (nation.n_name = 'JAPAN') DFO #2 : Index Output 1 (Projected) nation.n_nationkey

If a query is running very long, it can usually be stopped with a control-c, though it may take several seconds, or even several minutes if it is in the middle of certain operations. If the user then collects cmdstats (“iq show cmdstats 2”), IQ will return the query statistics for the query up to the point of the control-c. This will often allow a user to compare the row count estimates from the optimizer to the number of rows actually produced, and can indicate places where the IQ optimizer estimates are off by a large amount.


Actual number of rows projected

Indexes and JoinsIndexes on join columnsWhen statistics are reported on a query with a join, we often find that no index on the join columns are reported as used, except for FP indexes for projection.

The main reason IQ recommends HG or LF indexes on join columns has to do with statistics. HG and LF indexes provide better statistics, such as easy access to MIN and MAX values of a column and the number of unique values in that column. This information can be used to generate a more aggressive query plan with potentially better performance.

So while LF and HG indexes on join columns may not be used for WHERE predicates or for projecting data and may not appear at all in the CMDSTATS output, they are heavily used by the optimizer in determining join plans.

A secondary reason for HG or LF indexes is that IQ will usually not consider a Nested Loop - Push Down join unless there is one of these indexes to “push” into.

Indexes on other predicatesSybase IQ can process predicates in two places in relation to join optimization. First it can process the predicate before join optimization is done, providing exact counts for that predicate, rather than just estimates. Very simple predicates on fast indexes, such as “column = constant” on a LF or HG can be done this way. Those predicates that are not done this way are processed after optimization with the rest of the query – this would include not just predicates done in a filter node, but also predicates done in indexes when the optimizer has decided that it would be expensive to process the predicate.

…needs work…

Inferred join conditionsIQ will automatically generate additional inferred predicates (called transitive closure), including inferred join conditions. Transitive closure simply means that elementary school math lesson “transitive property” where if A=B and B=C then A=C. So for the query:

select count(*) from T1, T2, T3where T1.key = T2.key and T2.key = T3.key

IQ will also consider “where T1.key = T3.key”. Since to process the query there only needs to have two joins, and we now have three join conditions, IQ will decide the best two join conditions to use, and eliminate the join condition it doesn’t use.

IQ does not generate inferred predicates for inequality conditions, or for complex expressions.


Parameters which effect Sybase IQ joins

Join_Preference [hidden, defaults to 0]

Sometimes there is a need to turn off a particular join method to avoid a bug, or we wish to see what happens when we use some other join than the optimizer's choice. The hidden IQ set parameter JoinAlgorithmPreference will cause the optimizer to try to avoid or force a particular join algorithm for all of the joins in an entire query, provided that it is legal to use (or not use) that particular join type.

Sybase IQ does not have the ability to control join algorithms for individual joins in a query with multiple joins; setting this parameter will effect every join in a query. This is also a "session" parameter like MBSort or QueryInfo, so if it is set to something other than 0, it will stay set to that value until either IQ is exited or it is set it to something else.

The syntax is “iq set JoinAlgorithmPreference # “ where # can be

0 normal behavior (default)-1 avoid sort/merge join 1 prefer sort/merge join-2 avoid nested loop join 2 prefer nested loop join-3 avoid nested loop push-down join 3 prefer nested loop push-down-4 avoid hash join 4 prefer hash join-5 avoid hash push-down join 5 prefer hash push-down join-6 avoid prejoin indexes 6 (not used)

Notes:

Although this parameter was not originally added as a tuning switch, but to help cost joins, it has proven useful both in working around occasional bugs and helping improve some poorly performing query plans. For example, a query is running very slow, and from the cmdstats statistics it is shown that IQ is trying to do a hash join with millions of rows on the smaller side of the join. The user could set this to avoid hash joins for that query – although any other hash joins in the query would also be changed to something else. Another caveat is that the query group has not tested the behavior extensively with this switch at anything other than zero (the default). So if a user, for example, set the switch to avoid sort/merge joins, we might not always pick the fastest join method of the remaining three (it usually should though).

With one exception, JoinAlgorithmPreference will not override the chart of join algorithm limitations. The only exception is that preferring Nested-Loop Push-Down joins will cause them to be also used for non-equijoins ( <,<=,>,>= joins).

Hash and nested loop joins assume you can keep the smaller side of the join in memory. So if a query is set to force a hash join on a join with 10 million rows on the smaller side of the join, expect to wait a long time for an answer, unless there is only one user and there are a very large number of temp buffers.

Unless the small side of a join is less than about 4 rows, hash joins should usually outperform nested loop joins.


Join_Optimization [defaults to ON]

If this parameter is set to OFF, IQ will essentially make no effort to optimize the joins in the query. It will generate a naïve right-deep plan, meaning it will take whatever is the rightmost table in the FROM clause, join it to the next-to-rightmost table, join those to the next table, and so on from right to left. It is up to the user to order the tables in the FROM clause into a reasonable join order. All joins will be Sort-Merge joins.

Max_Cartesian_Result [11.2.1 only, defaults to 10,000,000]

When creating a query plan, if the optimizer cannot find plan without a cartesian join, and that cartesian join has an estimated result larger than the value of MaxCartesianResult, then the query is rejected with the error:

"The optimizer was unable to find a query plan that avoided cartesian product joins larger than the MaxCartesianResult setting"

Usually this is caused by accidentally missing one or more of the join conditions when writing the query. Occasionally, it can be caused when a query joins three or more tables with a single non-binary join condition (e.g. A.depth * B.width = C.area).

It should be noted that if a cartesian join is used, the large number of rows generated can cause the query to be quite a bit slower than expected.

OptimizeForThisManyUsers [defaults to 5]

In Sybase IQ 11.2 and 11.2.1, this parameter helps the optimizer, query execution engine, and other parts of IQ schedule resources for better use with multiple users. Setting this parameter to the approximate number of concurrent users expected will help generate better query plans and help avoid buffer shortages.


Undocumented parameters which effect Sybase IQ joins

JoinMaxHashRows [11.2.0, hidden, defaults to 125,000]MaxHashRows [ 11.2.1, hidden, defaults to 200,000]

When determining the join plan, the optimizer estimates the number of rows the join will get from each of its children, which is used both for costing and for some join algorithm cutoff heuristics. This parameter (which changed its name slightly in 11.2.1) controls the maximum number of estimated rows on the smaller side it will consider a hash join for.

On systems with a large amount of TEMPSHMEMMB (temp buffers), setting this parameter larger may sometimes produce query plans that use fewer system resources and/or are faster.

This parameter also affects the cutoff between hashing and sorting for Distinct and Group By processing (thus the name change in 11.2.1).

MaxPinnableHashPercent [hidden, defaults to 20]

When optimizing a query, IQ takes the greater of OptimizeForThisManyUsers or the number of active queries running, and divides this into the number of temp buffers in the temp buffer cache. It then takes a percentage (MaxPinnableHashPercent) of that number to determine the maximum number of buffers which can be pinned for a single hash for that query. Even if MaxHashRows is set very high, this computation may internally cause MaxHashRows to be made significantly smaller.

For example, there is 100 MBytes of temp cache, and OptimizeForThisManyUsers is set to 5. 100 Mbytes/5 = 20 MBytes per user; MaxPinnableHashPercent=20 percent, so each hash will be allowed up to 4 MBytes. If there are a large number of joins, and/or Distincts or Group Bys, it may be necessary to lower this number. If a query has a smaller number of Distincts, Group Bys, and joins, it may be useful to increase this parameter to encourage more hash operations instead of sorts.

This parameter also affects the cutoff decisions between hashing and sorting for Distinct and Group By processing.

MinHashBuckets [hidden, defaults to 64]

When deciding how big a hash table will be, the optimizer will reserve a number of pages to be “pinned” in memory for the hash table. Modifying OptimizeForThisManyUsers and MaxPinnableHashPercent can increase the number of buffers available for the optimizer to consider a hash. If the optimizer chooses a hash but underestimates the number of rows needed for the hash, the performance may be poor because of the overhead of locking and unlocking buffers (pinning and unpinning buffers). Increasing MinHashBuckets will improve hash performance.

Users must be careful, as increasing this parameter too much may pin too many buffers in memory, which may prevent other parts of the query, or even other users, from getting enough memory for their queries.

HASH_PINNABLE_LIMIT


NOEXEC

PREFETCH_APPLICATION_PERCENT*

PREFETCH_BUFFER_LIMIT

integer

20

PREFETCH_BUFFER_PERCENT

0 to 100

0

PREFETCH_GARRAY_PERCENT

0 to 100

60

PREFETCH_SORT_PERCENT

QUERY_ROWS_RETURNED_LIMIT

integer

0

QUERY_TEMP_SPACE_LIMIT

integer

1000

QUERY_TIMING

SORT_PINNABLE_CACHE_PERCENT

Some IQ join bugs fixed in 11.5.0

Inferred IN lists:

Sybase IQ generates inferred predicates (often called transitive closure). For example in the query:

select count(*) from T1, T2 where T1.a=10 and T2.a=T1.aIQ will also add “where T2.a=10”. In 11.2.x IQ also supports very long IN lists using a hash in a filter node above a Vertical Cursor. Unfortunately, if there is an inferred IN list, and the optimizer


decides to use the filter option for the inferred predicate (usually because there is no HG or LF on the join column), IQ will forget to create the hash for the filter. The result is that the query aborts. This problem is fixed in IQ 11.5.0.

Known IQ bugs

Other Known Join Problems and Issues

Indexes and data skew

All of the join algorithms in IQ, except sort-merge, are very sensitive to cardinality (number of rows) on at least one side of the join. As a result, we depend on our cardinality estimates as well as the join costing to determine whether an algorithm is appropriate.

Sybase IQ’s cardinality estimates are sensitive to skewed data. For example, if a table has 100 distinct values, but one value is 90% of the rows, it may cause poor optimizer choices. While it is not listed in the query statistics, we are often dependent on information which the HG and LF indexes can easily provide (e.g. MIN, MAX, count(distinct) ) to generate a better query plan. This is the main reason why they are recommended for join fields.

Sorts and Sort-Merge join

For very large sorts (particularly large sort/merge joins) we do not pre-fetch the pieces of the sort (sort buckets) back into memory as we are completing the sort. It is believed that this can cause large portions of a query to run using less than one CPU.

Star schemas, star joins, and “interesting cartesians”

IQ, at this time, does not have a prejoin or join algorithm particularly designed for star schemas and star queries. While hash joins and NL-Pd joins can very efficiently join one dimension table to a fact table, this can cause unnecessarily large intermediate products between join nodes. An example of this would be a fact table of 100 million rows, and three dimensions, each of which would filter 80% of the rows from the fact table. The first join with the fact would still product 20 million rows, the second join 4 million rows, and the third, 800,000 rows, which is finally something which can fit easily into memory.

One possible performance workaround until IQ implements a star-join algorithm is to place two or more dimension tables in a view with no join condition. If the product of the rows from each dimension is small enough to fit in a hash join, this can, in some cases, dramatically improve the star join performance – this is sometimes referred to as an interesting cartesian. Continuing the previous example, dimension 1 has 100 filtered rows, dimension 2 has 25 filtered rows, and dimension 3 has 12 filtered rows. The cartesian product here would be 100*25*12 = 30,000 rows, easily allowing a hash join of all of the dimensions to the fact table at the same time. The 100 million row fact table will be scanned, like a table scan, but after joining all 3 dimensions together before joining them to the fact table, only 800,000 rows are produced from the big join.

In the SAINT (v12.0) release, IQ will have a new join type that is optimized for star schemas.


Part 5 Views and Nested Tables

Views and Nested (or derived) tables are both methods of creating virtual temporary tables for doing multiple levels of aggregation and other complex relations. Views may also be used for security; security access for tables, columns and views are processed before the optimizer or query engine are invoked, and will not be discussed in this document.

Nested tables are like views defined in the FROM clause of a query, and which only exist for the duration of the query. Nested tables allow users the flexibility of views, while permitting the DBA to restrict access for view creation or deletion. Use of derived tables also can reduce the number of forgotten undeleted user views left around in the catalog, and eliminates the problem of potential view name conflicts.

ExamplesThe following are the same query written using a view and a derived table:

ViewCREATE VIEW PERFORMANCE (YEAR, REVENUE) ASSELECT DATEPART(YEAR, O_ORDERDATE), L_EXTENDEDPRICE * (1-L_DISCOUNT)FROM LINEITEM, ORDERWHERE O_ORDERKEY = L_ORDERKEY

AND O_CLERK = 'Clerk 123456'AND L_RETURNFLAG = 'Defective'

go

SELECT YEAR, SUM(REVENUE) AS REVENUE

FROM PERFORMANCEGROUP BY YEARORDER BY YEAR

go

DROP VIEW PERFORMANCEgo

Nested TableSELECT

YEAR, SUM(REVENUE) AS REVENUEFROM (SELECT DATEPART(YEAR, O_ORDERDATE) AS YEAR,

L_EXTENDEDPRICE * (1-L_DISCOUNT) AS REVENUEFROM LINEITEM, ORDERWHERE O_ORDERKEY = L_ORDERKEY

AND O_CLERK = 'Clerk 123456'AND L_RETURNFLAG = 'Defective'

) AS PERFORMANCEGROUP BY YEARORDER BY YEAR

go


Tuning hintsIQ will attempt to trim unneeded columns and move local table predicates down through one level of a view or nested table. If there are multiple levels of views or nested tables, IQ may not be able to trim all unnecessary columns or move predicates all the way down into the IQ indexes, and IQ will not necessarily be as efficient as it could.

For example, take a simple view:create VIEW1 (A1, B1, C1, D1, E1)as select a, b, c, d, e from TABLE

The following query will only select columns a and b from TABLE. The query will be processed just as if there were no view:

select A1,B1 from VIEW1 where C1 = “dog”But if another view level is added:

create view VIEW2 (A2, B2, C2, D2)as select A1, B1, C1, D1 from VIEW1where VIEW1.D1 > 10

with the query: select A2,B2 from VIEW2 where C2 = “dog”

The optimizer will create 3 query blocks; one for VIEW1 (which acts on the actual base table), one for VIEW2, and one for the actual user’s select. The optimizer will understand that since VIEW2 only uses A1, B1, C1, and D1, it can get rid of column E1. It will also move the predicate “VIEW1.D1 > 10” into VIEW1, where it will get processed in the IQ indexes. The user’s query is one more level up though, so it’s column trimming decisions and local predicates will only be moved down into VIEW2, and not further down into VIEW1 where it will effect the index filtering and what is scanned from the disk.

The final pseudo-query plan will look something like:select a,b,c,d from TABLE where d > 10select a,b, from the results of the previous query where c=“dog”

so columns “c” and “d” are unnecessarily projected, and the predicate “c=‘dog’” is done as a filter, not within the IQ indexes.


Part 6 Functions


Part 7 Subqueries

There are two ways subqueries are categorized in IQ. The dataflow node types for a subquery are:

Quantified Subquery – Returns 0 or more rows. For example, an IN subquery.Unquantified Subquery – Always returns 1 row.Exists subquery – returns 0 or 1 row (true or false)

More interesting, though, are the more common catagories:

Uncorrelated Subquery - subquery is only executed onceCorrelated Subquery - subquery is executed 0 or more times (usually many times)

The latter catagories are more interesting because the performance characteristics of the subqueries are more closely connected with the latter catagories than the former.

A subquery has an outer block and an inner block. The inner query block is the subquery, while the outer block is the main select or the next higher level of subquery.

Uncorrelated SubqueriesUncorrelated subqueries are subqueries that could be cut out of a query and run on their own. Since the subquery is itself a standalone query, it is only executed once (unless, of course, it is inside another query block which is executed multiple times).

Tuning an uncorrelated subquery is much like tuning a separate query. If a subquery is on a single table, and meets all of the other conditions for a “vertical” query, the entire subquery can be exeuted within the IQ indexes.

Correlated SubqueriesA correlated subquery is executed 0 or more times; many correlated subqueries are executed thousands or even millions of times. A correlated subquery will have one or more columns in its where clause which are not from tables in the subquery’s FROM clause, but are from a higher level query block. Since these columns come from outside the subquery, it is referred to as an outer reference.

For example:

select count(*) from T1where T1.A > 10and T1.b = (select max(price) from T2 where T2.c = T1.c)

In this example, “(select max(price) from T2 where T2.c = T1.c)” is a correlated subquery, and T1.c is the outer reference, since T1 does not appear in the subquery’s FROM clause, but instead is supplied by the outer (main) query block.How many times a correlated subquery may be executed is dependent on how many values there are in the outer reference. In the example, the subquery could be executed once for every value of T1.c where T1.A >10.


Tuning

There are several methods to optimize correlated subqueries, beyond optimizing them as one would a separate query (or like an uncorrelated subquery):

caching outersflattening subqueriescaching invariant subquery parts

Sybase IQ is rather primitive in correlated subquery optimizations compared to other databases. In fact, correlated subqueries are the one area where Sybase IQ is very frequently slower than Sybase SQL Server / ASE.

Caching outer references

IQ caches the most recent outer reference. So if the outer references happen to be ordered on the outer reference column, the performance of IQ can be very good. In Sybase IQ 12.0, IQ will usually sort the outer references before running a correlated subquery. Sometimes IQ will not sort the outers if the optimizer decides it would not help, or if a sort/merge join or similar operation has already sorted the outer references.

Sorting the outers can actually have two benefits:

it will get maximum caching benefits, since IQ caches the most recent reference

If the outer reference is on a column with a HG index, ordering the outer references will cause the subquery to access the HG in sequential order…

Subquery Flattening

Flattening a subquery usually means rewriting the subquery so that it is a join. For example, the following query has a correlated EXISTS subquery, with order_key as an outer reference.

select order_priority, count(*) as order_count from orders where order_date between '1993-07-01' and '1993-10-01' and exists (select * from lineitem where l_orderkey = order_key and l_commitdate < l_receiptdate) group by order_priority

On the database this was run on, the subquery is executed 400,000 times. It can be rewritten as

select order_priority, count(*) as order_countfrom orderswhere order_key IN (select distinct order_key

from orders, lineitem where order_date between '1993-07-01' and '1993-10-01' and l_orderkey = order_key and l_commitdate < l_receiptdate)

group by order_priority


This does one join, and the uncorrelated IN subquery, rather than the correlated subquery. In IQ, the second query is significantly faster than the first.

Many databases, including Sybase SQL Server / ASE, will automatically flatten the first query into the second. In fact, correlated EXISTS subqueries are amongst the easiest to flatten, and are among the most common types of queries where IQ performance is significantly slower than SQL Server.

For best performance in Sybase IQ 11.x, it is recommended that when possible, correlated subqueries, and especially correlated EXISTS subqueries (because they are generally not too difficult to rewrite) should be flattened by the user into joins.

Caching invariant subquery parts

If there are multiple joins inside a correlated subquery, and the subquery cannot be flattened easily, a useful optimization is to find those tables and joins that only need to be executed once and to cache their output into a temporary table or similar structure. Then the query engine can simply replay the cached values each execution rather than re-run the tables/joins each subquery execution.

Sybase IQ 12.0 will look for opportunities to cache invariant subquery plan parts after the join plans have been generated. In a future version (most likely the release after 12.0.3) IQ will integrate searching for possible invariance into the generation of the query plans. In other words, today IQ will find the invariance if we are lucky and the optimizer arranged the plan to have significant invariant plan parts to cache. What we plan to add is to create the subquery plan specifically to maximize the invariance in the plans.

Subqueries, query plans, and cmdstats


Part 8 Unions


Part 9 Notes on Parallelism

Dataflow ParalellismA join node, subquery node, or union node will have two children, and IQ will either execute the children on one side first, and wait until all the rows are returned from that child before executing the other side, or it may choose to start both children in parallel. If it starts the children in parallel, it is sometimes called dataflow parallelism.

Parallelism and joinsWith the exception of sort-merge joins (discussed below), IQ only supports dataflow parallelism, not the more common parallelism possible within a single join. Dataflow parallelism, as mentioned in the introduction, means that the join can cause both children nodes to execute in parallel.

Hash joins and nested loop joins must get all of the rows on the small side of a join before it begins comparing keys and passing rows to the node above. So if it isn't parallel, it will fetch all the rows from the small side first. If it is parallel, the fetching of all rows from the small side will be done while the first row is fetched from the larger side.

A nested-loop push-down join must get each row from the small side before it uses the indexes to get the join results from the large side for that row.

Sort/merge joins get all of the rows from each side and sorts them before comparing any of the rows and passing results to nodes above. When it isn't parallel, we fetch the smaller side first.

Sort merge joins are the only join type which is parallel within the node itself - that is each of the sort objects on both sides of the join can be themselves internally parallel. The hash, nested loop, and nested loop push down joins (as well as the final comparison/merge phase of the sort/merge join) only actually do one thing at a time inside the join. This is different from being able to have both children on each side of the join process their work at the same time. So with a simple two table join, a sort/merge join dataflow node could be utilizing more than one CPU, while a hash join would only be using one.

There is a side effect of this intra-sort parallelism. For some single user queries we will choose a Sort/Merge join over a hash, because with parallelism within the join it may be faster, while with multiple users, we may choose a hash join for the same query, since a hash join will typically consume fewer resources.


Part 10 Notes on datatypes

Supported datatypes and the space they take:

tinyint 1 byte

integer, unsigned int, numeric(1…10,x) 4 bytes

bigint, numeric(11…18,x) 8 bytes

numeric (19…35) variable – numeric(20,x) takes 15 bytes

real 4 bytes

float 8 bytes

char, varchar variable – up to 255 bytes

The numeric(x,x) data type is stored in one of three formats; a scaled integer (32 bit integer), a scaled bigint (64 bit integer), or what is called internally a true numeric. A true numeric is a software emulated data type that can support internally up to 64 digits of precision.

Char and Varchar data types are padded out to the maximum length “under the covers”.

SCALE

PRECISION


Part 11 Notes on optimizer design

Ideally an optimizer will always choose the very best plan for a query. In practice, an optimizer has to often make choices based on estimates that are not perfect. There are many things that IQ will sometimes have to make estimates for: data skew of local predicates or join keys, how many rows a join will filter, how many groups will be produced by a group by, etc.

In the design of the IQ optimizer, instead of simply trying to be as aggressive or as conservative as possible, IQ, like most other databases, takes a middle road. IQ will attempt to be aggressive, but will not be aggressive to the point where the optimizer thinks there is a significant chance that it will be making a bad choice.

So, like any database, there will sometimes be possibilities to hand-tune queries to get more aggressive plans with better performance than IQ will produce on its own. Conversely, there are other (hopefully fewer) cases where IQ will be too aggressive, or will simply make bad estimates, and the performance of a query will be bad, and some of the parameters listed in this document can be used to make the query better.


Part 12 Indexes, Hashes, and Sorts

While not directly part of the query engine, these three “tools” are used extensively by the query engine for various functions, such as joins, grouping, or counting.

IndexesThe IQ indexes may be used by

Local predicates single table group by single table select distinct and count(distinct) single table set functions (Sum/AVG/Count/MIN/MAX) Nested Loop Push Down joins the optimizer for making estimates to create query plans

HashesHashes are used by

Long IN lists group by select distincts and count(distinct) hash joinsBecause the hash is not what is called a perfect hash, often more than one row from the smaller table will be hashed into the same place in the hash table (so elements are “chained together” into that hash “bucket”). Because of this chaining, even after finding the hash value of the row from the large table, the query engine needs to compare that row to each value chained together in that bucket. This means we could conservatively add 200,000 x 50, or 10 million comparisons to the hash.

SortsSorts are used by

HG inserts group by select distincts Sort/merge joins order by


Part 13 The IQ buffer manager

While the buffer manager is a completely separate piece of code from the optimizer and query engine, the two do interact. This section will attempt to describe some of the characteristics of the buffer manager.

The buffer manager is split into two parts, the main indexspace buffer manager, and the temp indexspace buffer manager. These two buffer managers get their space from the IQ set parameters SHMEMMB and TEMPSHMEMMB. These buffer managers are each further split into three parts. The first part is what most users think about, the data caches for the main indexspace and temp indexspace. The main and temp caches each also have a blockmap cache, which is very much like a Unix i-node cache. That is a sort of set of pages to tell the buffer manager how to map logical IQ pages into physical pages on the disk. Finally there is a shared space for the buffer managers to keep track of administrative information for the caches and indexspaces.

In addition, to add to the confusion, there is a main and temp freelist cache, which are each a list of pages that are free and used in the main and temp indexspaces. This cache is actually very small, and uses only up to about 640 Kbytes. This cache is only used for the main cache if there is a read/write user, and for the temp cache, this is taken from each user’s heap memory.

So to summarize:

Name Approximate size

main indexspace cache 80% of SHMEMMBmain indexspace blockmap cache 10% of SHMEMMBmain indexspace buffer manager overhead 10% of SHMEMMBmain indexspace freelist cache 10 or more pages from user’s heap memory

(only if there is a Read/Write user)temp indexspace cache 80% of TEMPSHMEMMBtemp indexspace blockmap cache 10% of TEMPSHMEMMBtemp indexspace buffer manager overhead 10% of TEMPSHMEMMBtemp indexspace freelist cache 10 or more pages from user’s heap memory

What happens in which cache?The caching policy is fairly simple. The main indexspace cache is simply a page cache for the pages in the main indexspace on disk. Except during inserts and deletes, no user will ever create, destroy, or modify a page in this cache. All intermediate products between joins, temporary bitmaps (almost every query with a WHERE clause creates at least one temporary bitmap), hashes, sorts1, and various other temporary objects use temp buffers to store the data.

1 IQ has two sorts. The older sort uses MBsort from heap memory, and as each MBsort bucket is filled and sorted, it is written to the temp indexspace. The newer “constrained” sort uses temp buffers directly as sort buckets, and does not use MBsort. In 11.5 and 11.5.1, only queries with ordered Select Distincts and Order By clauses use the new sort.


SHMEMMB, page size, and the number of cache pagesOne would reasonably guess that a simple formula for figuring out how many pages are created in the main cache would be something like:

SHMEMMB * 0.8 (from the chart above) / the page sizeThis is true for IQ databases with a page size of 32 KBytes or smaller. For indexspaces with 64KBytes or larger pages, IQ will create twice this number of page frames for the main indexspace cache2. A page frame can be thought of as the overhead information needed to access a page in the cache, and a pointer to where that page is in the cache. Sybase IQ stores a number of different types of objects in the main indexspace cache, such as fast projection index data pages, b-trees, and different kinds of bitmaps. Even uncompressed, many of these objects may take much less than 64 KBytes. By creating extra page frames, the buffer manager creates the opportunity to cache many more pages and better utilize the shared cache memory, compared to what it could cache if it only created “SHMEMMB * 0.8 / page size” worth of page frames.

There is a cost in terms of management overhead to do this though, and as the page size gets smaller, the amount of “wasted space” becomes less, so creating extra page frames becomes less useful. The performance cutoff for more pages cached versus greater cache overhead was found to be around a 32 Kbytes page size.

The day-to-day practical effect of this is that for the “IQ Status” command and the IQ buffer monitor, the number of indexspace cache buffers displayed is really the number of page frames. So for the main indexspace cache size, when the page size is greater then 32 Kbytes, this number will be twice the expected number. This can be even more confusing in the IQ monitor, where the monitor can show as little as 50% of the pages being used, when actually 100% of the data space in the cache is being used. This will happen if the pages are all full size, and so all of the extra page frames are unused.

2 The temporary indexspace cache does not have as many objects that fit naturally into smaller pages, so the Buffer Manager for the temporary indexspace cache does not create any extra page frames. This means that the temp cache buffer manager should always create approximately the expected number of pages.


Appendix A Query plan node typesSybase IQ 11.5.1 query plan nodes

Group By- Group By (Sort) group with a sort- Group By (Hash) group with a hash- Vertical Grouping Cursor group using the IQ indexes

Distinct- Distinct (Sort) find distinct with a sort- Distinct (Hash) find distinct with a hash- Vertical Distinct Cursor find distinct using the IQ indexes

Simple aggregation This is a set function without an explicit group by or distinct - for example “select sum(x) from Table”- Group By Single do the set function above the vertical cursor- Vertical Aggregation Cursor find result using IQ indexes

Order By- Order By sorts rows of data

Vertical Cursor- Vertical Cursor- Vertical Grouping Cursor- Vertical Distinct Cursor- Vertical Aggregation Cursor

Filter This has local table predicates from a WHERE clause which are not done within the IQ indexes. This node represents something like an IQ table scan, except that IQ only scans the columns it thinks it may need for the query.

Joins- Nested Loop- Nested Loop Push-Down- Hash- Hash Push-Down- Sort-Merge- Cartesian Nested Loop

Subqueries- Quantified Subquery subquery returns 0 or more rows- EXISTS subquery EXISTS or NOT EXISTS- Unquantified Subquery subquery returns 1 row - like a SQL Server expression subquery

Union Either an explicit UNION or an OR which is turned into a UNION inside IQ


Appendix B Aggregation examplesExamples of aggregation in IQ query plansThese results were generated using the IQ QueryInfo parameter; if the command “iq set queryinfo on” is entered before queries, the optimizer will display the query plan it generated to process the query.

Vertical CursorsSQL> select sum(o_totalprice) from orders+++> goThe current query has been passed to the data flow engine-- Query Plan:-- 1 #01: Vertical Aggregation Cursor-- Table Name: orders-- Table Row Count: 1500000

orders SUM(o_totalprice) ------------------------ 226710060385.0401

(1 row affected)

SQL> select o_orderpriority, sum(o_totalprice) from orders +++> where o_orderdate > '1-Jan-1990'+++> group by o_orderpriority+++> goThe current query has been passed to the data flow engine-- Query Plan:-- 1 #01: Vertical Grouping Cursor-- Table Name: orders-- Table Row Count: 1500000-- Condition 1: (orders.o_orderdate > Jan 01 1990)-- Grouping Column 1: orders.o_orderpriority

orders orders o_orderpriority SUM(o_totalprice) --------------- ------------------------ 1-URGENT 45434686570.0693 2-HIGH 45436378764.9099 3-MEDIUM 45132540163.9196 4-NOT SPECIFIED 45289775407.0799 5-LOW 45416679479.0599

(5 rows affected)

SQL> select distinct o_orderpriority from orders+++> goThe current query has been passed to the data flow engine-- Query Plan:-- 1 #01: Vertical Distinct Cursor-- Table Name: orders-- Table Row Count: 1500000-- Distinct Column 1: orders.o_orderpriority


orders o_orderpriority --------------- 1-URGENT 2-HIGH 3-MEDIUM 4-NOT SPECIFIED 5-LOW

(5 rows affected)

SQL> select count(distinct o_orderpriority) from orders+++> goThe current query has been passed to the data flow engine-- Query Plan:-- 1 #01: Vertical Aggregation Cursor-- Table Name: orders-- Table Row Count: 1500000

orders COUNT(DISTINCT o_orderpriority) ------------------------------- 5 (1 row affected)

Horizontal Group BySQL> select o_orderpriority, sum(o_totalprice) from orders +++> where o_orderdate > '1-Jan-1990'+++> group by o_orderpriority+++> goThe current query has been passed to the data flow engine-- Query Plan:-- 1 #02: Group By Ordered-- Child Node 1: #01-- AggregationAlgorithmPreference Setting: 1-- 2 . #01: Vertical Cursor-- Table Name: orders-- Table Row Count: 1500000-- Condition 1: (orders.o_orderdate > Jan 01 1990)

orders orders o_orderpriority SUM(o_totalprice) --------------- ------------------------ 1-URGENT 45434686570.0701 2-HIGH 45436378764.9104 3-MEDIUM 45132540163.9203 4-NOT SPECIFIED 45289775407.0799 5-LOW 45416679479.0594

(5 rows affected)

SQL> select o_orderpriority, sum(o_totalprice) from orders +++> where o_orderdate > '1-Jan-1990'+++> group by o_orderpriority+++> goThe current query has been passed to the data flow engine


-- Query Plan:-- 1 #02: Group By Unordered-- Child Node 1: #01-- AggregationAlgorithmPreference Setting: 3-- 2 . #01: Vertical Cursor-- Table Name: orders-- Table Row Count: 1500000-- Condition 1: (orders.o_orderdate > Jan 01 1990)

orders orders o_orderpriority SUM(o_totalprice) --------------- ------------------------ 4-NOT SPECIFIED 45289775407.0799 3-MEDIUM 45132540163.9196 1-URGENT 45434686570.0693 5-LOW 45416679479.0599 2-HIGH 45436378764.9099

(5 rows affected)

Horizontal DistinctSQL> select distinct o_orderpriority from orders+++> order by o_orderpriority+++> goThe current query has been passed to the data flow engine-- Query Plan:-- 1 #03: DistinctOrdered-- Child Node 1: #02-- AggregationAlgorithmPreference Setting: 1-- 2 . #02: Constrained Order By-- Child Node 1: #01-- 3 . . #01: Vertical Cursor-- Table Name: orders-- Table Row Count: 1500000


(5 rows affected)

SQL> select distinct o_orderpriority from orders+++> order by o_orderpriority+++> goThe current query has been passed to the data flow engine-- Query Plan:-- 1 #03: Constrained Order By-- Child Node 1: #02-- AggregationAlgorithmPreference Setting: 3-- 2 . #02: DistinctUnordered-- Child Node 1: #01


-- 3 . . #01: Vertical Cursor-- Table Name: orders-- Table Row Count: 1500000


(5 rows affected)

Group By SingleSQL> select sum(o_totalprice) from orders+++> goThe current query has been passed to the data flow engine-- Query Plan:-- 1 #02: Group By Single-- Child Node 1: #01-- AggregationAlgorithmPreference Setting: 1-- 2 . #01: Vertical Cursor-- Table Name: orders-- Table Row Count: 1500000

orders SUM(o_totalprice) ------------------------ 226710060385.0401

(1 row affected)

SQL> select count(distinct o_orderpriority) from orders+++> goThe current query has been passed to the data flow engine-- Query Plan:-- 1 #02: Group By Single-- Child Node 1: #01-- AggregationAlgorithmPreference Setting: 1-- 2 . #01: Vertical Cursor-- Table Name: orders-- Table Row Count: 1500000

orders COUNT(DISTINCT o_orderpriority) ------------------------------- 5 (1 row affected)


Appendix C Sybase IQ 11.5.1Sybase IQ 11.5.1 is primarily a bug fix release. There are only a couple of new features in 11.5.1:

Support for more tables in joins. Sybase IQ 11.5.0 has an absolute limit of 16 tables in a single FROM clause, and a recommended limit of 8-10 tables due to optimizer performance limits. In 11.5.1, the maximum limit is 24 tables, with up to 18 tables supported with good optimizer performance.


Appendix D Sybase IQ 12.0After Sybase IQ 11.5.0, which added full support for Component Integrated Services (CIS, also known as OMNIconnect), the next major release will be version 12.0, internally called SAINT. The major features of Saint will include (but are not limited to):

Separation from Sybase SQL Server / ASE. Sybase IQ 12.0 will no longer use SQL Server as a catalog server, and SQL Server will not be required to run Sybase IQ.

A completely new interface, which will resemble in many ways Sybase SQL Anywhere 6.0 (AKA Sybase Adaptive Server Anywhere). This will include new:

- system management interface- system tables- stored procedure language support- local variables- Java

A new server model. Sybase IQ 11.2 and later used a shared memory model with separate multithreaded processes for each user. In 12.0, Sybase IQ will have a single shared multithreaded server model, somewhat similar to SQL Anywhere or SQL Server.

Support for a more complete set of TSQL functionality (approximately the same TSQL support as Sybase SQL Anywhere). This includes the many TSQL math and string functions IQ does not support in 11.x.

Better compression. Some bitmap indexes and some FP columns will have significant space savings.

Automatic creation of FP indexes. The create table + create indexset DDL of Sybase IQ 11.x will be eliminated. When a table is created or loaded, it will be like creating or loading a SQL Server or SQL Anywhere table. When a table is created, each column will automatically have a Fast Projection index created.

Prejoin indexes will be updated when the tables in the prejoin are inserted into or deleted from. There will be no longer be separate prejoin index insert or delete statements.

Better backup support, including incremental backup.

A new join type, Hash Push-Down join, is added. This join helps in those places where a Nested Loop Push Down join would be useful, but would thrash the IQ cache (cause too much I/O). More importantly, it should provide significant performance enhancements for star schemas, and is comparable to (or, we expect, even better than) the “Star Join” types in other databases.

Local and global temp tables are supported natively in IQ.

ANSI outer joins are added. TSQL outer joins can lead to ambiguous meanings which are difficult for the query engine to interpret what the user wanted, while ANSI outer joins are clear statements of how to process the outer join. Left, Right, and Full outer joins will be supported


by all IQ join types.

Hash performance in general is improved significantly, and the places where hash joins and hash grouping is used should increase.
