1

Chapter 18

Strategies for Query Processing

We focus this discussion w.r.t RDBMS, however, they are applicable to

OODBS.

2

1. Translating SQL Queries into Relational Algebra and

Other Operators

- SQL is most commonly used query language

- Translated into equivalent extended relational algebra expression

- It is represented as a query tree data structure and optimized

- SQL queries are decomposed into query blocks; basic units;

translated into expressions and optimized

- A query block contains single select-from-where expression

including GROUP BY and HAVING

- Nested queries within a query are identified as separate blocks

Q1: Retrieve the names of employees who earn a salary that is greater

than the highest salary in department 5.

SELECT Lname, Fname

FROM EMPLOYEE

WHERE Salary > (SELECT MAX (Salary)

FROM EMPLOYEE

WHERE (Dno = 5);

The Q1 is a nested query, it is decomposed into two sub queries as

follows.

3

Q1a: (inner)

SELECT MAX (Salary)

FROM EMPLOYEE

WHERE (Dno = 5)

Q1b: (outer)

SELECT Lname, Fname

FROM EMPLOYEE

WHERE Salary > c (c represents the results of Q1a)

Q1a can be translated to: (inner)

ᵮMAX Salary (σ Dno=5 (EMPLOYEE)) (Selection operator)

Q1b can be translated to: (outer)

π Lname, Fname (σ Salary>c (EMPLOYEE)) (Projection operator)

- The query optimizer will choose the execution plan for each query

block

- The inner block needs to be evaluated only once

- The outer block has to iterate for each employee

- This type of nested queries are called non-correlated queries

4

5

Additional Operators

Semi-join and Anti-join

Semi-join is generally used for un-nesting EXISTS, IN and ANY

subqueries

Consider:

EMPLOYEE (Ssn, Bdate, Address, Sex, Salary, Dno)

DEPARTMENT (Dnumber, Dname, Dmgrssn, Zipcode)

Where a department is located in a specific zip code.

Q(SJ):

Counting the number of departments, that have employees, who make

more than 200000 salary.

SELECT COUNT(*)

FROM DEPARTMENT D

WHERE D.Dnumber IN (SELECT E.Dno

FROM EMPLOYEE E

WHERE E.Salary > 200000);

The nested query is joined by the connector IN operator.

6

To remove the nested query:

(SELECT E.Dno

FROM EMPLOYEE

WHERE E.Salary > 200000) is called as un-nesting. This query gives a set

of Dno with employees who make more than 200000. It leads to the

following query with an operation called semi-join, shown as “S=”

SELECT COUNT(*)

FROM EMPLOYEE E, DEPARTMENT D

WHERE D.Dnumber S = E.Dno and E.Salary > 200000;

Consider another query:

Count the number of employees who do not work in department

located in Zipcode 30332.

Q(AJ):

SELECT COUNT(*)

FROM EMPLOYEE

WHERE EMPLOYEE.Dno NOT IN (SELECT DEPARTMENT.Dnumber

FROM DEPATMENT

WHERE Zipcode = 30332);

7

SELECT COUNT(*)

FROM EMPLOYEE DEPARTMENT

WHERE EMPLOYEE.Dno A = DEPARTMENT AND Zipcode=30332

(Antijoin)

A row of T1 is rejected as soon as T1.X finds a match with any value of

T2.y. A row of T1 is returned, only if T1.x does not match with any value

of T2.y.

2.Algorithms for External Sorting

External sorting: sorting for large files or records stored on disk that do

not fit entirely in memory

Sorting most commonly used in databases. Ex. Whenever order by is

used, it sorts results. Join, Union, Intersection, …also use sorting.

Sorting may be avoided if primary or clustering index is available.

External sorting algorithms are used to sort large files on disk.

A typical sort used for this is sort-merge algorithm.

Requires buffer space in main memory, where sorting is done, which is

called DBMS cache that is controlled by DBMS.

Sort small sub files called runs, merge the sorted files.

Sorting Phase:

- Read a piece of file into buffer

- Sort using internal sort algorithm

- Write back to disk (as temporary disk space)

8

Merging Phase:

- The sorted runs are merged into one or more merge phases

- Each merge phase can have one or more merge steps

- One block is used to keep merged results

9

Available buffer space = nB blocks

Number of file blocks b

If nB = 5 and b = 1024

Number of initial runs (each of size 5 blocks except the last one)

= nR = гb/nB ˥ = 1024 / 5 = 205

After sort phase, 205 sorted runs are stored as temporary subfields on

disk

Degree of merging dM is the number of runs that can be merged in

each pass

dM = (nB – 1) or nR (smaller of)

dM = 5 - 1 = 4 (four way merging)

nP = гlogdM(nR) ˥ = number of passes = log4(205) = 4

Number of block accesses =

(2 * b) + (sort phase, once to read and once to write)

(2 * (b * logdM(nR))) (merge phase)

= 2 * 1024 + (2 * (1024 * 4))) = 2048+8192 = 10240

(205 -> 51 + 1 ->13 -> 3 +1 -> 1)

10

11

3.Algorithms for SELECT Operation

There are many algorithms for SELECT operation, basically a search

operation to locate records to satisfy certain conditions. They depend

on the access paths.

The following operations are used to illustrate the algorithms:

OP1: σ ssn=’123456789’ (EMPLOYEE)

OP2: σ Dnumber > 5 (DEPARTMENT)

OP3: σ Dno=5 (EMPLOYEE)

OP4: σ Dno=5 AND Salary > 30000 AND Sex=’F’ (EMPLOYEE)

OP5: σ Essn=’123456789’ AND Pno=10 (WORKS_ON)

OP6: An SQL Query

SELECT *

FROM EMPLOYEE

WHERE Dno IN (3, 27, 49)

OP7: An SQL Query

SELECT First_name, Lname

FROM EMPLOYEE

WHERE ((Salary * Commission_pct) + Salary) > 15000;

12

Search Methods

A number of search algorithms are possible for selecting records from a

file, called file scans.

(1) S1 – Linear Search; read all related disk blocks into memory

buffers and search for required conditions

(2) S2 – Binary Search: if equality condition on a key attribute; binary

search is efficient (file is ordered on this attribute), OP1

(3) S3a – Using a primary index; equality comparison and a key

attribute; OP1; use primary index to retrieve the record; retrieves

a single record

(4) S3b – Using a hash key: equality condition, use the hash key to

retrieve a record; OP1

(5) S4 – Using a primary index to retrieve multiple records; if the

comparison is > >= < <= , key field with a primary index, Dnumber

> 5; OP2; find a record satisfying the equality condition, then

retrieve all preceding records

(6) S5 – Using a clustering field to retrieve multiple records: equality

condition on a nonkey; OP3; retrieve all records

(7) S6 – Using a secondary (B+ index) on an equality comparison;

single record if a key, multiple records for non-key attribute; can

also be used for range queries; leaf nodes provide the ordering of

values to select multiple values in a range

SKIP other methods

Search methods for conjunctive selection

Made up of several conditions connected with AND

13

1. S8 – Using an individual index: if an attribute has an access path,

one of the methods S2 – S6 can be used to retrieve the record;

check if the retrieved record satisfies other conditions

2. S9 – Using a composite index: if a composite index created for one

or more conditions, then use it retrieve the records

3. S10 – Using intersection of record pointers: if secondary indexes

or access paths are available for more than one field, and the

indexes include a record pointer, then each record can be retried

and checked for a given condition; the intersection of these

records can only be retrieved to check for the conjunctive

condition

Search methods for disjunctive selection

OP4 example

Uses OR operator in the Selection. Require to retrieve records that satisfy

each condition and make a union of records and eliminate duplicates. If

one of the attribute does not have an access path, then we need to use

linear search to retrieve records. All the above steps for access paths can

be used if they exist.

Estimating the selectivity of a condition

- System catalog can maintain some statistics on database attributes

such as number of distinct values, number of records, frequency of

data changes etc…

- The above information can be used in query optimization

14

4.Implementing the JOIN Operation

- one of the most time consuming operation

- equijoin or natural join are most commonly used operations

- two-way JOIN involves joining two files

- multi-way JOIN involves multiple files (number of possible ways to

execute grows rapidly)

- we discuss two-way JOINS here

R A=B S A, B are the join attributes, domain compatible

OP6:

EMPLOYEE Dnumber=Dno DEPARTMENT

OP7:

DEPARTMENT Mgrssn=Ssn EMPLOYEE

Implementing Joins

(1) J1 – Nested Loop Join (or nested block join): Default, no need to

have access paths; test if this condition is true for each record;

t[A] = s[B]

(2) J2 – Index based nested loop Join: use an access structure to

retrieve the matching records; if an index or hash value exists for

one of the records, B of the S, retrieve each record t in R s[B] =

t[A]

15

(3) J3 – Sort merge join: The records of R and S are physically sorted

by the value of attributes A and B respectively; both files are

scanned concurrently and matched for the same values of A and B

(4) J4 – partitioned has join (or just hash-join): R and S are partitioned

into smaller files; the partitioning of each file is done using the same

hash function h on the attributes A of S and B of R (partitioning

phase); partitioning phase and probe phase

R S

……………… ……………………….

hash some records

h(A) h(B)

….. ….. (smaller file will fit into memory)

Match records from S

Continue the process….

Join Performance

OP6:

EMPLOYEE Dnumber=Dno DEPARTMENT

Assume buffer space is available

Nested loop approach

16

Available buffer space = nB blocks

Each memory buffer is same size as one disk block

DEPARTMENT file consists of rD = 50 records stored in bD = 10 disk

blocks

EMPLOYEE file consists of eD = 6000 records stored in bE = 2000 disk

blocks

Outer loop file records should be read to memory (as many as possible)

One block for the inner loop and one block to write the result and

remaining nb -2 blocks for outer loop

Read one block at a time for the inner loop file and use its records to

probe (that is search) that outer loop blocks that are currently in

memory for matching records;

The contents of the result block are appended to the result file on the

disk

In the nested loop join, it makes a difference which file is chosen for the

outer loop and which file is chosen for the inner loop;

If EMPLOYEE is chosen for outer loop, each block of employee is read

once; entire DEPARTMENT file (each of its blocks) read once for each

time we read in nB -2 blocks of the EMPLOYEE file.

Total number of blocks accessed for outer loop file = bE

Number of times (nB-2) blocks of outer file are loaded into memory =

гbE/(nB – 2) ˥

17

Total number of blocks accessed for inner loop file =

bD * гbE/(nB – 2) ˥

Total number of read accesses =

bE + bD * гbE/(nB – 2) ˥ = 2000 + 10(2000/5) = 6000 block accesses

Total number of read accesses (if DEPARTMENT is in outer loop) =

bD + bE * гbD/(nB – 2) ˥ = 10 + 2000(10/5) = 4010 block accesses

bRES block accesses are added to the above result for a total block

accesses.

JOIN Selection Factor and Performance

A fraction of records in one file is joined with other file.

It is called a join selection factor w.r.t equijoin

OP7:

DEPARTMENT Mgrssn=Ssn EMPLOYEE

rD = 50 (50 records in the department); 50 records will be joined with

6000 employee records, but 5950 will not be matched.

Suppose secondary indexes exist on Mgr_ssn and Ssn. The number of

index levels are Xssn = 4 and Xmgr_ssn = 2;

18

Retrieve each EMPLOYEE record and then use the index Xmgr_ssn to

find a matching record for the DEPARTMENT entries:

The number of accesses = bE + (rE * (Xmgr_ssn + 1)) = 2000 + (6000 * 3)) =

20000 block accesses

The second option is: retrieve each DEPARTMENT record and then use

the index on Ssn of EMPLOYEE to find a matching manager:

The number of accesses = bD + (rD * (Xssn + 1)) = 10 + (50 * 5)) = 260

SKIP 18.4.4 and 18.4.5

5.Algorithms for PROJECT and Set Operations

∏<attribute-list> (R) selects columns in the list of attributes

SELECT Salary

FROM EMPLOYEE;

Produces a list of salary for all employees.

It is linear search of all employees.

If attribute list does not include a key attribute, then you have duplicate

values in the result.

If duplicate tuples exist, simply sort and remove the duplicates.

Set operations UNION, INTERSECTION, DIFFERENCE, CARTESIAN

PRODUCT are expensive…

Sort merge can be used for set operations. Cartesian product should be

avoided and use JOIN instead.

SKIP 18.5.1

19

SKIP 18.6

7.Combining Operations Using Pipelining

A query gets translated into a relational algebraic expression with many

operations. A single operation at a time requires temporary files to hold

results; these temporary files can be passed to next operation; this is

called materialized evaluation.

To avoid large temporary files, a combination of operations used to

improve query execution.

A JOIN can be combined with select operations on two input files and

projection on one output file.

It is common to generate query execution code directly to implement

multiple operations.

The output of one operation is fed as input of another operation; a

series of pipeline to perform combined operations.

Iterator for implementing physical operations

- If the operator is implemented in such a way that it outputs one

tuple at a time, then it is regarded as a iterator (nested join)

- The iterator interfaces consists of the following methods:

o Open()

o GetNext()

o Close()

- Iterator concept can also be applied to access methods (B trees)

8.Parallel Algorithms for Query Processing

20

Parallel database architectures (distributed databases, big data and

NOSQL)

(1) Shared Memory Architectures

a. Interconnection network, common memory

b. Each processor has access to entire memory

c. Contention for memory and not scalable, interference

(2) Shared Disk Architectures

a. Every processor has its own memory

b. Every processor has access all disks through interconnection

network

c. Every processor may not have disk of its own (SANs, and

NAS)

(3) Shared-nothing Architectures

a. Commonly used

b. Its own memory and disk

c. Inexpensive to build (racks of storage with processors)

d. Allocating more processors and disks can achieve linear

speedup

(a) Operator-level Parallelism

- Partition data across disks

- Horizontal partitioning; distribution of tuples across disks based on

some method; ith tuple to (i mod n); range partitioning (10 disks, 10

ranges for storing employee records); hash partitioning ith record

assigned to h(i) disk

- Sorting: sort on age and partition data

- Selection based on some condition; range portioning

- Projection and duplicate elimination: sorting tuples and discarding;

sorting can use partitioning strategies

21

- JOIN: split the relations to be joined; join is divided into multiple

smaller joins to perform in parallel on n processors and take a union

of the result; Partitions must be non-overlapping on the join key

(b) Intraquery Parallelism

- A query execution graph can be modeled as a graph of operations

- Provide appropriate data in the partition to execute in parallel

- If operations do not depend upon one another, they can be run in

parallel

- If one of the operations can be generated tuple-by-tuple and fed to

the other operation, then result is pipelined parallelism

(c) Interquery Parallelism

- Scale up the parallelism

- Avoid cache coherency, concurrency control

- Increase the overall rate at which transactions can be processed in

parallel by increasing the processors

22