Spatial Indexing in Microsoft SQL Server 2008

Yi Fang, Marc Friedman, Giri Nair, Michael Rys, Ana-Elisa Schmidt Microsoft Corp.

{yfang,marcfr,gnair,mrys,anasc}@microsoft.com

ABSTRACT Microsoft SQL Server 2008 adds built-in support for 2-dimensional spatial data types for both planar and geodetic geometries to address the increasing demands for managing location-aware data. SQL Server 2008 also adds indexing capabilities that, together with the necessary plan selections done by the query optimizer, provide efficient processing of spatial queries. This paper will present an overview of the spatial indexing implementation in SQL Server 2008 and outline how the indexing is implemented and how the cost-based query optimizer chooses among the different plans.

Categories and Subject Descriptors H.2.2 [Database Management]: Physical Design – Access methods. H.2.4 [Database Management]: Systems – query processing. H.2.8 [Database Management]: Database Applications – spatial databases and GIS.

General Terms Algorithms, Performance, Design

Keywords Spatial databases, Spatial Indexing, Spatial Query Processing

1. Introduction Location-aware devices and services such as GPS devices and online mapping services are increasingly becoming commodity items that are used to help find everything from how to best drive from one place to another to the best location to eat dinner. These devices, attached to such things as pictures, buses, shipments and people, produce a wealth of location data that needs to be stored and managed efficiently. Since spatial data is often associated with other data that is already managed inside database systems, it makes sense to extend existing relational database systems with spatial data support. Many database systems such as IBM DB2, Informix, Oracle, and PostgreSQL provide extensions to deal with spatial data.

Spatial data support includes the following four items:

1. Native spatial types to represent the spatial shapes and objects that one would like to manage in a database system

2. Spatial operations such as testing for intersection of two spatial objects

3. Spatial access or indexing methods to enable efficient processing of the spatial operations on large spatial data sets

4. Cost-based optimizations that chose the right query plan (often including the spatial index) to efficiently apply the spatial operations

Starting in the upcoming release, SQL Server 2008 also adds spatial data support to manage location-aware data [13]. In particular it introduces two new built-in types to represent both 2-dimensional planar and geodetic vector data.

The planar data is managed with a new type called GEOMETRY that is used to represent points, lines, polygons and combinations thereof that live in a flat two-dimensional plane such as a state plane coordinate system or a factory floor plan.

The geodetic data is represented with the new GEOGRAPHY type that also represents points, lines and polygons on a 2 dimensional plane on the surface of a geoid that approximates the shape of the Earth.

Figure 1. (a) Planar and (b) geodetic representation of the Earth

(a) (b)

Both types support a plethora of spatial operations, including object composition and spatial relationship predicates. The set of functionality provided by the spatial types is aligned with the Open Geospatial Consortium (OGC) OpenGIS Simple Features Specification for SQL [14] and enables applications to perform operations in an industry-standard way.

In order to efficiently apply spatial relationship predicates, SQL Server 2008 also offers an indexing infrastructure and implements extensions to the query processor that allows the optimizer to make a cost-based decision to chose the appropriate query plan if one or more indices is present. Since SQL Server’s main access structure is based on B+-Trees, specific challenges had to be overcome to utilize the existing infrastructure to provide spatial index support.

This paper will present an overview of the spatial indexing implementation in SQL Server 2008. Section 2 reviews related work, while Section 3 provides a quick overview of the spatial data

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SIGMOD’08, June 9–12, 2008, Vancouver, BC, Canada. Copyright 2008 ACM 978-1-60558-102-6/08/06...$5.00.

1207

types and methods introduced by SQL Server 2008. Section 4 describes the spatial index infrastructure, while Section 5 explains how the SQL Server query optimizer builds spatial index plans, and decides whether to use them based on cost estimates. Section 6 describes some of the performance behaviors and provides some insight into the tuning of the index, before Section 7 concludes the paper.

2. Related Work Many spatial indexing methods and query processing techniques for operating on spatial data have been proposed in the literature and several have been implemented in both research prototypes and commercial database and GIS systems. Spatial indexing methods refer to the techniques that use hierarchical data structures to sort data based on its spatial occupancy. Spatial indexing decomposes the space from which the data is drawn into regions which can be further decomposed into sub-regions. These regions are grouped hierarchies and represented as a tree structure which facilitates fast spatial operations such as search or intersect. There are four principal approaches in spatial indexing:

• R-trees [3,8] organize spatial objects by summary enclosing rectangles. Each node in the tree corresponds to the smallest rectangle that encloses its child nodes. Leaf nodes contain pointers to the actual objects in the database. R-tree has the drawback that in the worst case, one has to search the entire database to determine which object a particular point belongs to.

• R+-tree [20,23] is a variant of the R tree. R+-trees use multiple bounding rectangles to cover an object so that the entries of any internal nodes do not overlap. An object ID may be stored in more than one leaf node. R+-tree has the drawback that the decomposition is data-dependent which means it is difficult to perform tasks that require composition of different data sets.

• Uniform grid method [6] decomposes the space into uniform grid. This approach is good for uniformly distributed data, but spatial data is usually not uniformly distributed;

• Quadtree [21,22] is a tree structure in which each internal node has up to four children, representing quadrants divided according to a fixed scheme. Quadtrees adapt the depth of decomposition to the distribution of the data, thus are suited for arbitrarily distributed data. But it they are sensitive to positioning – the placement of the objects relative to the decomposition lines of the space affects their storage and search efficiency.

R-trees do not partition the space into disjoint cells, which in the worst case can lead to searching the entire database to determine whether an object contains a particular point. The others use disjoint decomposition, which alleviate this problem, at the cost of additional tree nodes per object. Numerous query processing techniques have been proposed to improve the basic spatial data structures [4,5,10,17,18, etc.] The performance of various index structures has been studied extensively [23]. R-tree and quadtree variants are compared in [11] and from within Oracle Spatial in [19]. These studies show that the

index performance depends heavily on the nature of data and the type of applications. There is tremendous overhead associated with implementing and integrating any non-traditional indexing mechanism beyond B+-Trees. Extensible indexing frameworks that build on the existing B+-Tree infrastructures have been proposed to address this problem. In such frameworks, a variety of index structures can be efficiently instantiated without modifying the database engine. GiST (Generalized Search Trees) is an extensible framework for B+-Tree-like indexes [9]. SP-GiST (Space Partitioning Generalized Search Trees) is an extensible framework for the family of space-partitioning trees [1,7]. Commercial databases also support extensible indexing frameworks, e.g., DB2 extenders and Informix Datablades. Using extensible indexing frameworks, spatial indexing techniques have been realized in commercial and open source database systems, including Oracle Spatial, IBM DB2 Spatial Extender, Informix Spatial DataBlade Module and PostGIS in PostgreSQL. Microsoft SQL Server 2008’s implementation is also based on an extensible indexing framework based on its existing B+-Tree infrastructure and an adaptive quadtree-like multi-level grid and reuses the existing optimization framework of its relational query processor. Sections 5 and 6 will provide more details.

3. Spatial data types Microsoft SQL Server 2008’s new built-in GEOMETRY and GEOGRAPHY types are implemented as Microsoft .NET Framework Common Language Runtime (CLR) types. Both provide properties and methods that can be used to perform spatial operations such as calculating distances and areas, or finding geographical features that intersect one another, or composing new spatial objects from others, i.e., calculating the union of two polygons. These methods are exposed as type-associated CLR methods. Unlike some other spatial type libraries, these two types are able to represent a variety of spatial shapes under a single abstraction.

The planar GEOMETRY type represents points, lines, polygons, multipolygons (e.g., complex shaped polygons) and combinations thereof in a flat two-dimensional plane such as a state plane coordinate system or a factory floor plan. The geodetic GEOGRAPHY type represents points, lines, polygons, and multipolygons on a 2 dimensional surface of a geoid that approximates the shape of the Earth. Both types support generation of instances from the OGC Well-Known Text (WKT) and Well-Known Binary (WKB) formats as well as GML. The following SQL code example shows how to use the GEOMETRY type to define a table of city boundaries and a table of streets:

CREATE TABLE Cities

( CityId int IDENTITY (1,1),

CityName nvarchar(20),

CityGeo geometry);

CREATE TABLE Streets

( StreetId int IDENTITY (1,1),

StreetName nvarchar(20),

StreetGeo geometry);

Data, such as WKT descriptions of city polygons and streets, can be inserted using static constructors into the respective table:

1208

INSERT INTO Cities (CityName, CityGeo)

VALUES ('MyCity',

geometry::STGeomFromText

('POLYGON ((0 0, 150 0, 150 150, 0 150, 0 0))', 0));

INSERT INTO Streets (StreetName, StreetGeo)

VALUES ('First Avenue',

geometry::STGeomFromText

('LINESTRING (100 100, 20 180, 180 180)', 0))

Finally the following SQL query joins the streets with the cities that they geometrically intersect with:

SELECT s.StreetName, c.CityName

FROM Cities c, Streets s

WHERE s.StreetGeo.STIntersects(c.CityGeo)=1

4. Indexing 4.1 Indexing overview SQL Server 2008 Spatial Indexing component is built using the extensibility hooks available in the relational query processing engine. This means that the existing B+-Tree and access methods are used as the storage scheme for the index. Further, the basic principles of the XML indexing model introduced in SQL Server 2005 [16] are applied to the spatial implementation as well. While SQL Server 2008’s spatial indexing supports both spatial types, we focus the discussion on the indexing of the GEOMETRY type. The GEOGRAPHY type is indexed using similar principles with slight adjustments for the ellipsoidal domain. SQL Server’s spatial indexing method adopts the well-known strategy of hierarchical decomposition of space to implement the index. Some of the traditional limitations of fixed spatial decomposition systems are addressed in the implementation. The spatial indexing system is designed to be able to handle objects of varying shapes and sizes efficiently. Non-uniform data distribution patterns are the weak points of fixed decomposition schemes. In order to address this, the system allows for multiple localized indexes on a spatial column instance. The system decomposes the indexed space into an ordered collection of axes-aligned cells using a four level hierarchy of grids. The cells are disjoint except when they are part of ancestor-descendant relationships. The cells of all levels form an ordered domain, in which all of a cell’s descendents immediately follow it. The linear ordering is important and necessary because the underlying storage and access methods make use of existing 1-dimensional B+-Tree storage keyed on hierarchical cell identifiers. The linear ordering used provides for spatial locality in the index. The index is used to support both filter and join on a set of spatial methods, all of which are variants of the OpenGIS STIntersects operation, which compares two geometries and returns value 1 exactly if they share any points. There are two usage scenarios supported by the indexing scheme in 2008 server. Both scenarios involve table filter expressions. The first case is known as point or window queries, in which a particular geometric object, the query window W, is given, and the goal is to identify all intersecting indexed objects as fast as possible. When the spatial filter is used without a spatial index, assuming

there are no additional filters, every row is tested by the computationally intensive spatial filter. The indexing strategy adds an earlier filtering step that uses a restricted number of ordered range-seek operations on the index B+-Tree to prune the set of candidate objects. The second case is known as distance queries where given an object and distance, the goal is to find out the objects which are within the given distance. The system doesn’t support nearest neighbor queries in this release.

4.2 Structure of space decomposition The indexing space and decomposition structure need to be predefined at index creation time. The indexing space is defined using the axes-aligned rectangle known as a bounding box (note that the GEOGRAPHY index does not require a bounding box since it indexes the whole geoid). The hierarchical decomposition is defined using parameters that control the grid density. Level zero in the spatial decomposition, called the root level grid, consists of two cells: the interior and exterior of the bounding box. Objects can fall in one or the other or both. The objects that fall in the exterior cell don’t participate in the filtering operation unless the query window also falls in the exterior cell. There are 4 lower levels of hierarchical divisions applied to the bounding box cell. Each level divides cells of the previous level uniformly, according to one of the following user-specified configurations:

Table 1. Grid divisions

Index configuration Number of cells Grid formation

LOW 16 4 X 4

MEDIUM 64 8 X 8

HIGH 256 16 X 16

The decomposition approach is similar to Quad-Tree decomposition where 2, 3 or 4 2X2 division levels are collapsed into one level. Note that each of the four levels can have a different grid division setting. The four level division and grid settings are picked to sufficiently support indexing of whole geoid and to provide flexibility for geometric indexing. Figure 2 shows the grid decomposition of a polygon using 2x2 grid formations at each level where common children are collapsed into the higher-level grid:

Figure-2: Grid decomposition of a polygon using 2x2 grids

1209

4.3 Spatial tessellation When an object is inserted into a table with a spatial index, it is mapped to a set of cells known as its tessellation. Any set of index cells covering the entire input object is a legal tessellation.1 The tessellation is built by a CLR table-valued function called from the query processing layer, first during spatial index creation, and at query execution time for the query window W. The tessellation, in the form of <cell, primary key> pairs, is persisted in a B+-Tree. The query objects, which are typically run-time constant parameters or outer side references in a join on a spatial filter, are tessellated using the same table-valued function. This is called candidate mode tessellation and the cells are sometimes referred to as candidate cells. The tessellation cardinality is defined as the maximum number of cells output by the tessellation function for any object. The tessellation cardinality is also configurable at the index creation time and can be used to control the index size. Note that the higher the cardinality, the objects are shredded to greater detail in the index. However, the greater detail might not be useful in all cases, especially when the performance hinges on the query window size. In the candidate mode tessellation, the system automatically determines the cardinality so as to maximize the query performance. The tessellation is implemented using a top-down, breadth-first grid traversal algorithm. The basic building block of the algorithm is a coverage-mask computation method which generates the grid coverage-mask for the given object and grid. The coverage mask is a two-dimensional boolean array, implemented as a bit mask, with one value per each cell of the grid. There are three kinds of coverage masks: touched, partially covered or interior, each indicating whether the given cell touches, partially or fully covers the object. SQL Server distinguishes touched versus partially covered cells in order to deal with the precision differences in the tessellation and filter operations. A partially covered cell is guaranteed to intersect with the cell which is not the case with the touched cell. The tessellation operation is carried out in a lower precision than the spatial filter which, along with conservative detection of touched cells, enables the index to behave the same way as the table scan with the filter. The coverage information is persisted along with the cell-identification in the index. The hierarchical tessellation also uses the coverage mask, in order to determine whether to drill down the grids. Only partially covered cells are tessellated further using the child grids. Interior cells fully capture the spatial coverage and hence there is no need for tessellating them further. The algorithm is throttled further using the tessellation cardinality. If the desired output cardinality is exceeded while tessellating to a child grid, the parent cell is used to represent the area. The indexing system allows for multiple spatial indexes to be constructed on one column with different bounding box and grid settings. The spatial query processing system attempts to minimize the impact of the objects falling outside the bounding box when the index is used, enabling the applications to define multiple spatial indexes on a given column and work around severely uneven data distribution, if required. The application can provide index hints based on the location and size of the window query object. Multi-indexing capability of the indexing system is referred in the document as localized index support. 1 We produce tessellations consisting of cells from at most two

successive levels, plus the external cell when necessary. We do not produce the zero-level internal cell.

Algorithm 1: Index population query SELECT

tf.Cell_Id,

tf.Cell_Mask

FROM

base_table bt

CROSS APPLY

Sys.TessellationFunction (bt.indexed_col, index_params) tf

Algorithm 2: Hierarchical tessellation TessellationFunction(geometry g, cardinality card, out CellQ) Begin Init OutputCellQueue using CellQ Init ProcessQueue Insert TopCell into ProcessQueue While ProcessQueue is not empty Begin CellToTraverse = ProcessQueue.Remove ComputeCoverage(g, CellToTraverse, out CoverageMasks) If (card < CoverageMask.Count + OutputCellQueue.Count) Output CellToTraverse Else Begin Output Interior (fully covered) Cells Output Leaf-Level Cells Output Touched Cells Insert non-leaf PartiallyCoveredCells to ProcessQueue End End End

Algorithm 3: Grid Coverage Computation ComputeCoverage(geometry g, Cell c, out CoverageMasks cm) Begin [GridIntersectDetection Phase] For each edge determine the cells touched by the edge Mark the cells in coverageMaskTouched [Finalize Phase] Traverse the coverageMaskTouched row by row Keep track of number of cells touched in the row If the count is odd, mark coverageMaskInterior for the cell End

1210

4.4 Cell identification The cell identification generated by the tessellation consumes 5 bytes and is of the following format: L1.L2.L3.L4.D where Ln refers to the cell identifier at depth n and D is the depth (0 to 4) of the grid in which the cell belongs to in the decomposition hierarchy. L1 is the most significant byte in the binary value of the cell identifier. The value of cell identifier for a given grid is determined using a Hilbert space filling curve. The unused bytes in the identifier are zero'ed out. The byte D (depth byte) is the least significant byte. The cell-encoding scheme attempts to preserve spatial locality of the cells in the binary values of identifiers and in turn, the persisted rows in the index B+-Tree. Note that the encoding doesn’t allow for bounding box cell of the root grid to be represented. This means that tessellation cardinality is overridden at the root level grid. The disjoint of the bounding box, uses the identifier 0.0.0.0.0 and is sometimes referred to as the root cell. The mapping of the coverage mask to the cell identification is done without any run-time computation by using fixed label arrays. Even though the system allows for a Hilbert curve of the order 32, the label arrays are pre-computed. This is possible since Hilbert curve can be mapped to the grid hierarchy and the Hilbert identifier of the parent cells can be stitched together to form the Hilbert identifier of the cell. Each of the alternate parents in the grid hierarchy transposes the row and column values to match the Hilbert curve.

4.5 Primary and secondary filters The run-time processing of a window query using the spatial indexes has three logical stages: candidate mode tessellation of the query window W, primary filtering and secondary filtering. Primary filtering identifies primary keys of indexed objects associated with cells of W. Primary filtering is conservative, producing no false negatives. In some cases, there will be enough information to say with certainty that the query object intersects the target object by looking at the coverage mask information of the index cell and the candidate cell. In those cases, the row is directly returned without invoking the computationally expensive spatial filter method. When there is uncertainty, the computationally intensive spatial secondary filter is called. The index maintains <cell id, primary key>2 pairs with coverage mask values, clustered on the cell ids. During primary filtering, the index is filtered using a predicate that encodes overlap relationships between cells that can be expressed as IsAncestorOrDescendant(cell_id1,cell_id2)

cell_id1 = cell_id2

OR IsAncestor(cell_id1,cell_id2)

OR IsDescendant(cell_id1,cell_id2)

Since there are at most 3 ancestor levels, seek for the ancestor match is implemented using a fixed transformation of the candidate cell identifier. The descendant lookup, on the other hand, is handled by generating the descendant range of values for the given cell. IsAncestor and IsDescendant operations are normalized into a single IsAncestorOrDescendant operator for cost estimation purposes. It is clear from the description that the cell-

2 A candidate mask is also stored with each pair. Detailed

treatment of candidate masks would complicate the discussion without enhancing the exposition.

identifier is a hierarchical type and the implementation takes advantage of the built-in support for hierarchical types within the server.

Figure 2. Primary Filtering: One Dimensional View L1 L2 L3 L4

Before the secondary filter is called for a given row, the coverage masks are compared to determine whether the row can be preselected. However, since the filter deals with ancestor and descendant cells of the candidate cell, the coverage map of a candidate cell needs to be mapped into maps for ancestor and descendant cells maps first. The following tables capture the logic involved in pre-selection:

Table 2. Coverage Map Translation Candidate Cell Mask

Ancestor Cell Mask

Descendant Cell Mask

Touch Touch Touch

Partially covered Partially covered Touch

Interior Partially covered Interior

Table 3. Pre-selection by primary filter

Cell Mask 1 Cell Mask 2 Row Pre-selected

* Touch No

Interior Partially covered Yes

Interior Interior Yes

Partially covered Partially covered No

4.6 Extensibility hooks The spatial filter transformation to implement the spatial index is handled by building metadata driven extensibility hooks in the system. The hooks are built around the notion of an index extension scheme. An index extension scheme is defined as a set of query transformation rules that the system applies during the plan selection process. A transformation rule consists of an input plan fragment shape and an output implementation. The transformation is applied

Query Window in Candidate Cell

Object Tessellated in an Ancestor

Object Tessellated in a Descendant

1211

only if an index using index extension scheme exists and can be matched to the filter expression. Currently, the output transformation is entirely defined in terms of relational operators. In addition to STIntersects, the following methods are supported using the underlying indexing scheme: STDistance, STOverlaps, STWithin, STContains, STTouches and STEquals. SQL Server 2008 ships two built-in index extension schemes: one for geometry type and one for geography type. The transformation rules for the filter methods are similar. Some of the methods don’t support coverage mask based optimization. The system currently doesn’t support any disjoint spatial filters.

5. Query Optimization The optimizer builds spatial index query plans, and decides based on cost whether to use them or not. This is an important decision, as a wrong choice either way can be arbitrarily bad. The authors have seen 30x degradation from using the index when it should not have been used, and far worse for failing to use it when it should have been. Cost-based selection of non-relational index plans is new for SQL Server’s extended indexes. For XML and full-text indexes, the optimizer does not decide whether to use an index. The so-called Primary XML indexes are chosen heuristically at algebrization time [16], while full-text indexes are required based on query semantics. SQL Server has integrated the spatial indexing decisions into the existing generic optimization framework through a combination of generic and special-purpose extensions. It follows the attractive design principle that the relational optimizer should know nothing of the non-relational domains. In other words, the optimizer receives relational algebra, with non-relational details encapsulated in user-defined functions (in this case the tessellation function). However, some details of spatial indexing leak out into the relational algebra itself – into the cell id domain, the join predicate in the primary filter, and the relative position of the primary and secondary filters. In order to optimize I/O, and to correctly decide whether to use a spatial index or not, it was necessary to provide some integrated spatial optimization support. One key piece of spatial support is an index lookup plan tailored to the spatial primary filter. The cell id, the key of the spatial index, is a hierarchical id with variable length. There are examples of other hierarchical ids in the SQL Server XML PATH and PROPERTY indexes [16]. Matching the cells of the window object to an index of cell ids involves a complex join predicate IsAncestorOrDescendant. The join must find, for a spatial object W, all rows in the spatial index having cells which may overlap with cells in W’s tessellation: Join IsAncestorOrDescendant(T.cell, I.cell)

Spatial_Index(cell, attr) as I,

Tessellation(W, cell, attr) as T

Two cell ids match if one is either an ancestor or descendant of the other. This is equivalent to doing a prefix match (e.g. LIKE ‘ABC%’ in SQL), in both directions, and UNION-ing the results. Without further refining the IsAncestorOrDescendant predicate, spatial index access will be suboptimal. Naïvely joining on the predicate, or converting it to two prefix operations, results in a cross-product of the window cells with the entire spatial index. This violates the intuition that smaller, more selective windows should touch proportionally less data. Indeed, the whole point of mapping a two-dimensional space into a one-dimensional B+-Tree using a space-filling curve is to break two-

dimensional objects into the minimal number of disparate clumps of records, to reduce the number of seeks necessary to access them. SQL Server transforms the join into an index lookup plan that has the intuitive, near-minimal-cost behavior and is no worse than a single scan of the index in the worst case (Section 4.1). Cardinality estimation of the primary filter is the critical factor determining whether to use the index or not. It in turn depends on the statistics on Cell_Ids, and cost formulae to estimate the selectivity of the IsAncestorOrDescendant predicate. Costing and estimation of spatial plans required some new optimizer capabilities:

• Compile-time execution of small relational queries for estimation (Section 4.2.1).

• A selectivity estimator for IsAncestorOrDescendant (Section 4.2.2).

• First-class index hinting support for spatial indexes. The optimizer’s first goal is to pick a low-cost query plan automatically. However, since cost estimation is by nature imperfect, hinting is essential for user control.

• Late execution of expensive scalar filters. Optimally ordering an expensive filter with other relational operators adds costs to the relational query optimization problem by introducing scalar costing and increasing search space. SQL Server formerly assumed scalar computation was cheap, so it chose to apply scalar filters early. Spatial indexing plans require secondary filters to be delayed.

5.1 Seeking into the Spatial Index Although the IsAncestorOrDescendant predicate motivates special-purpose spatial join algorithms [15], the problem we face in window queries is different enough that SQL Server is able to effectively leverage pure relational operators. This simplification is possible because the tessellated window object being joined to the index has a limited number of rows (1024), cell divisions (256), and decomposition levels (4). The join between the tessellated window object and the spatial index is dominated by the cost of I/O on the index. The optimal serial I/O strategy would be to convert the set of Cell_Ids output by the tessellation of window W into a set of range seeks, in order, without overlap. SQL Server can get close to this ideal by converting IsAncestorOrDescendant(T.cell, I.cell) into an expression E with range and equality predicates seekable over index I, by constructing all the ancestor cells: E = Ancestor(T.cell, 1) = I.cell OR

Ancestor(T.cell, 2) = I.cell OR

…

Ancestor(T.cell, k-1) = I.cell) OR

I.cell BETWEEN T.cell and

DescendantLimit(T.cell))

where k is the level of T.cell, which is capped at k=4 in the implementation. The last part of the predicate captures both equality and descendant cells. The encoding is designed so that Ancestor and DescendantLimit (the highest-possible-rank descendant in the domain) can be computed very efficiently. As is, E is seekable, but it expands each row of T independently into up to 4 disparate range seeks. This may not be efficient, since it does not take advantage of the sharing of ancestors, or the likelihood (thanks to the Hilbert curve) of locality across ranges. For this E needs to be

1212

broken down further to collect the distinct ranges from all the cells. This constructs the computed table RANGES shown in Table 4.

Table 4. Relation RANGES

start end Ancestor(T.cell,1) Ancestor(T.cell,1)

Ancestor(T.cell,2) Ancestor(T.cell,2)

… …

Ancestor(T.cell,k-1) Ancestor(T.cell,k-1)

T.cell DescendantLmt(T.cell)

This table is sorted on {start,end} and duplicate rows are removed.3 Then the result is joined to I on the predicate I.cell BETWEEN start AND end.4

5.2 Estimation of primary filter selectivity The selectivity of the primary filter PF is the fraction of primary keys PK in the spatially-indexed table that survive primary filtering. This quantity is the key factor determining the cost-based spatial index selection decision. Assuming independence of the columns in the index, this can be computed as follows: selectivityPF = selectivityjoin * frequencyPK

The first term depends on W and the distribution of Cell_Ids, while the second term is a standard statistic that SQL Server creates and maintains automatically.5

5.2.1 Getting Statistics on Query Window W Primary filter selectivity is very sensitive to W. On, say, US businesses, selectivities will vary tremendously between a query for Manhattan and one for Eastern North America. In traditional compile-then-execute query processing, the compiler does not have statistics on the output of a runtime function like Tessellate(W). Therefore it is advantageous to interleave processing and execution somehow – through partial evaluation, feedback, or compile-time execution. SQL Server 2008 chose the latter for simplicity. Whenever the compiler considers a spatial index for a window query with fixed W, the compiler executes the tessellation of W and computes statistics on it. This is not SQL Server’s general practice for user-defined table-valued functions, due to both the cost/value tradeoff and side-effects. Since spatial support benefits from this compile-time execution, and the tessellation function is a system function for which extra executions are known to be benign, the extra cost during compile-time is paid.

5.2.2 Using Trie Statistics The critical operation for spatial indexing is to estimate the selectivity of the IsAncestorOrDescendant predicate on hierarchical cell ids. This problem is not well-suited for estimation

3 Candidate mask aggregation is also performed. 4 To eliminate range overlaps, strictly contained ranges would

need to be removed as well. It is not worthwhile to do that, however, because it hampers intermediate filtering. By construction, contained ranges can have more informative candidate masks than their containing ranges.

5 For single-column keys.

using histograms on columns. Histograms are ideal when there are a few tens or hundreds of contiguous regions of relatively uniform density. Index cells are more fractal; there are vast areas of no density with very condensed areas of high and variable density. Trie statistics, on the other hand, are well-suited for prefix matching because they model hierarchical range containment. Tries are an existing data structure the SQL Server optimizer uses to estimate selectivity of string predicates with wild cards [2]. Suffix trees [12], another popular technique for estimation of string matching with wild cards, is less useful, since it would wastefully preserve subsequence information. This application of tries to hierarchical ids is applicable to other domains such as XML HIDs and path expressions [16]. A trie R is a compact, lossless representation of a bag S = Trie-1(R) of strings, which stores shared prefixes at most once, with counts. Both Trie() and its inverse functions are computable. Currently, SQL Server uses tries for LIKE predicates (A LIKE B) when the pattern B is a parameter or constant. The spatial support extend this to the case where A and B are both columns with terminal wild cards.

5.2.3 Selectivity of the Join The new selectivity logic models the join predicate as two LIKE predicates: (T.cell LIKE I.cell + ‘%’)

OR (I.cell LIKE T.cell + ‘_%’)

where the ‘%’ represents any string and ‘_%’ represents any non-empty string.6 With both a trie for the string-encoded Cell_Ids of both T and a sample of I, SQL Server is able to estimate join selectivity by computing a new trie representing the result of the join. Let RT be Trie(T) and RI be Trie(Sample(I)). The new trie RIT can be defined as

Trie(Trie-1(RI) semijoinIsAncestorOrDescendant Trie-1(RT)).

The semijoin indicates that the optimizer is interested in output rows from one side, the spatial index I. RIT does not need to be produced; a count of its nodes is sufficient. Counts for join on (T.cell LIKE I.cell + ‘%’) (resp. (I.cell LIKE T.cell + ‘_%’)) can be computed in one tandem pass over RI and RT, using a bit vector VI over the nodes of trie RI. Each trie node N represents a single distinct string. It has an attribute N.Count(), a method to reconstruct the string, and a list of children nodes. For each node N of R = RT (resp. RI). Position on N’, which matches the maximum prefix of N in the other trie R’ = RI (resp. RT). If R = RI, set VI(N). Otherwise, set VI(A) for each ancestor A (optionally inclusive) of N’. One can remember the stack of ancestors while positioning strictly forward, so it costs a linear walk, times the depth of the deeper trie (the size of its stack). The result desired (call it C) is the count of the nodes of RI in VI.

6. Experiments The authors conducted several experiments with the completed product measuring the performance using a uniformly distributed synthetic dataset of points. This section describes some of the behaviors found, and provides some insight into the tuning of the index.

6 The way the predicate is estimated is independent of the way it

is implemented.

1213

6.1 Indexed vs. non-indexed query performance Using an index will improve performance in some cases but not others. The cost of lookups in the index trade-off against reduced secondary filtering. Figure 3 shows the elapsed times in milliseconds of a query using vs. not using a spatial index. The x-axis is the window edge length for a square window. Using the spatial index is effective for selective windows, but not for unselective windows.

Figure 3. Window Size vs. Window Query Times

6.2 Query performance and query window sizes This experiment measures the average costs per object in a window. It examines the impact of the query window size and the data density on the per row performance. The sampling interval for data density is 100 objects to 1900 objects per leaf level cell. The sampling interval for query window size is also defined using leaf-level cell size: window sizes 1, 4, 16 and 64 times the leaf-level cell are used for measurement. We take 20 samples for each query window size and the average warm-cache cost is used for plotting. The x-axis tracks the number of objects in the leaf level cell. The performance metric, per row elapsed time, is plotted in y-axis using microseconds as the units.

Figure 4. Object Density vs. Per-Row Cost

The per-row costs are higher when the window size is smaller. However, the data density doesn’t play a significant role in the per-row access cost for the measurement interval.

6.3 Primary and internal filter efficiencies Primary filter efficiency is defined as percentage of rows selected by the primary filter that appears in the output result-set. It is natural that the primary filter efficiency gets better with more granular grids. The same is true for internal filter efficiency which is defined as the percentage of rows in the output result-set that are pre-selected by the primary filter. The relationship of the index performance to primary and internal filter efficiencies was clear from subsequent experimental data. In this experiment, we focus on the query window size and the filter efficiencies. The data collected indicated that input data density didn’t play a significant role in the efficiency values. In this experiment, the measurement sampling interval for query window is 0.5 to 250 times the leaf level cell. We take 40 sample locations for each query window size and compute the filter efficiency values for each window. The average value is used for plotting in the y-axis. X-axis tracks the size of the query window in relation to the leaf-level cell.

The experiments suggest that the leaf cell sizes should be made small enough that common query window are larger by an order of magnitude.

7. Conclusions With the increasing proliferation of location-aware devices and services such as GPS devices and online mapping services, managing spatial data has become an important requirement for relational database systems. SQL Server 2008 joins its peers in offering support for spatial data with a simple to use, OGC aligned spatial feature set. Efficient processing is provided by the addition of an adaptive, multi-level grid based spatial index which builds on the existing B+-Tree infrastructure of SQL Server and its integration into SQL Server’s query optimizer. This paper has provided an overview of the spatial indexing in SQL Server 2008 and has shown the benefits of using the indexing and some insights into how to best configure and use the indexing mechanism. Future versions of SQL Server will build on top of the extensible framework and improve on the presented approach by offering more extension schemes based on customer-feedback.

1214

8. ACKNOWLEDGMENTS We would like to thank Ed Katibah and Isaac Kunen for reviewing the paper and appreciate the reviewers’ comments and suggestions to improve the presentation.

9. REFERENCES [1] Walid G. Aref, Ihab F. Ilyas: SP-GiST: An Extensible

Database Index for Supporting Space Partitioning Trees. J. Intell. Inf. Syst. 17(2-3): 215-240 (2001)

[2] Srikanth R Avadhanam, Nigel R. Ellis, Campbell Bryce Fraser, Rodger N. Kline. System and method for using a compressed trie to estimate like predicates. United States Patent # 6,829,602, 2004.

[3] N. Beckmann, H.-P. Kriegel, R. Schneider, and B. Seeger.The R*-tree: an efficient and robust access method for points and rectangles Proceedings of the 1990 ACM SIGMOD international conference on Management of data. pages 322-331, 1990

[4] S. Berchtold, C. Bohm, D. A. Keim, and H.-P. Kriegel. A cost model for nearest neighbor search in high-dimensional data space. In PODS, pages 78--86, 1997.

[5] Thomas Brinkhoff , Holger Horn , Hans-Peter Kriegel , Ralf Schneider, A Storage and Access Architecture for Efficient Query Processing in Spatial Database Systems, Proceedings of the Third International Symposium on Advances in Spatial Databases, p.357-376, June 23-25, 1993

[6] Wm. Randolph Franklin. Adaptive Grids for geometric operations. In Proc. Sixth International Symposium on Automated Cartography (Auto-Carto Six), pages 230-239, Ottawa, 1983

[7] Thanaa M. Ghanem, Rahul Shah, Mohamed F. Mokbel, Walid G. Aref, Jeffrey Scott Vitter: Bulk Operations for Space-Partitioning Trees. ICDE 2004: 29-41

[8] A. Guttman.R-Trees: A Dynamic Index Structure for Spatial Searching. Proc. ACM SIGMOD Int. Conf. on Management of Data. pages 47-57, 1984

[9] Joseph M. Hellerstein, Jeffrey F. Naughton, Avi Pfeffer: Generalized Search Trees for Database Systems. VLDB 1995: 562-573

[10] Gísli R. Hjaltason, Hanan Samet. Ranking in Spatial Databases. Proceedings of the 4th International Symposium on Advances in Spatial Databases. pages 83-95, 1995

[11] Erik G. Hoel, Hanan Samet: A Qualitative Comparison Study of Data Structures for Large Line Segment Databases. SIGMOD Conference 1992: 205-214

[12] P. Krishnan, Jeffrey Scott Vitter, Balakrishna R. Iyer. Estimating Alphanumeric Selectivity in the Presence of Wildcards. SIGMOD 1996: 282-293.

[13] Microsoft Corp. Delivering Location Intelligence with Spatial Data. Whitepaper. http://www.microsoft.com/sql/techinfo/whitepapers/spatialdata.mspx, 2007.

[14] Open Geospatial Consortium (OGC). OpenGIS® Simple Features Specification for SQL. http://www.opengeospatial.org/standards/sfs

[15] Jack A. Orenstein: A Comparison of Spatial Query Processing Techniques for Native and Parameter Spaces. SIGMOD Conference 1990: 343-352.

[16] Shankar Pal, Istvan Cseri, Gideon Schaller, Oliver Seeliger, Leo Giakoumakis, Vasili Zolotov: Indexing XML Data Stored in a Relational Database. VLDB 2004: 1134-1145.

[17] D. Papadias, Y. Theodoridis, T. Sellis, and M. Egenhofer. Topological relations in the world of minimum bounding rectangles: a study with R-trees. Proceedings of the ACM SIGMOD Conference, San Jose, California, 1995.

[18] K. V. Ravi Kanth, D. Agrawal, Ambuj K. Singh. Dimensionality Reduction for Similarity Searching in Dynamic Databases. SIGMOD Conference 1998: 166-176

[19] K. V. Ravi Kanth, Siva Ravada, Daniel Abugov. Quadtree and R-tree Indexes in Oracle Spatial: A Comparison using GIS Data SIGMOD Conference 2002: 546-557.

[20] J. T. Robinson. The K-D-B-Tree: A Search Structure For Large Multidimensional Dynamic Indexes. In Proceedings of ACM SIGMOD, pages 10--18, 1981.

[21] Hanan Samet, Robert E. Webber: Storing a Collection of Polygons Using Quadtrees. ACM Trans. Graph. 4(3): 182-222 (1985).

[22] Hanan Samet, C. A. Shatter, Randal C. Nelson, Y.-G. Huang, Kikuo Fujimura, A. Rosenteld: Recent developments in linear quadtree-based geographic information systems. Image Vision Comput. 5(3): 187-197 (1987).

[23] Timos K. Sellis, Nick Roussopoulos, Christos Faloutsos. The R+-Tree: A Dynamic Index for Multi-Dimensional Objects. Proceedings of 13th International Conference on Very Large Data Bases. page 507-518, 1987.

[24] Michael Stonebraker, “Inclusion of New Types in Relational Data Base Systems,” Proceedings of the Second International Conference on Data Engineering(ICDE), pages 262-269. 1986

[25] Yannis Theodoridis, Timos K. Sellis: A Model for the Prediction of R-tree Performance. PODS 1996: 161-171.

1215