Databases, Spatial Databases, and PostGIS

Databases, Spatial Databases, Spatial Databases, and Databases, and

PostGISPostGIS

Overview of relational Overview of relational database concepts and database concepts and

PostGISPostGIS

DBMS PerspectiveDBMS Perspective

From Spatial Databases, A Tour, by Shekhar and Chawla

Role of DBMSRole of DBMS

Typical small system architecture: Typical small system architecture:

Role of DBMSRole of DBMS

Larger spatial DBMSLarger spatial DBMS

From “Introduction to Spatial Data Management with Postgis,” by Arnulf Christl http://www.ccgis.de, http://www.mapbender.org/

DefinitionsDefinitions

Database Management Systems (DBMS) – Database Management Systems (DBMS) –

““Software that controls the organization, storage, Software that controls the organization, storage, retrieval, security and integrity of data in a database. It retrieval, security and integrity of data in a database. It accepts requests from the application and instructs the accepts requests from the application and instructs the operating system to transfer the appropriate data.” operating system to transfer the appropriate data.” (Computer Desktop Encyclopedia, 2007)(Computer Desktop Encyclopedia, 2007)

Key features: Key features:

– Data SecurityData Security– Data IntegrityData Integrity– Interactive QueryInteractive Query– Interactive Data ManipulationInteractive Data Manipulation– Data IndependenceData Independence

May be May be Network, Hierarchical, Object, RelationalNetwork, Hierarchical, Object, Relational. . RelationalRelational is by far the most commonly-used and well- is by far the most commonly-used and well-established, and handles most data management established, and handles most data management problems very well.problems very well.

DefinitionsDefinitions

Relational Database Management Systems Relational Database Management Systems (RDBMS)–(RDBMS)–

““A database that maintains a set of separate, related A database that maintains a set of separate, related files (tables), but combines data elements from the files (tables), but combines data elements from the files for queries and reports when required. The files for queries and reports when required. The concept was developed in 1970 by Edgar Codd, concept was developed in 1970 by Edgar Codd, whose objective was to accommodate a user's ad hoc whose objective was to accommodate a user's ad hoc request for selected data.” [See request for selected data.” [See Codd ArticleCodd Article for for details.] details.] (Computer Desktop Encyclopedia, 2007)(Computer Desktop Encyclopedia, 2007)

Data stored in separate tables, each containing Data stored in separate tables, each containing tabular data like a spreadsheet, joined together as tabular data like a spreadsheet, joined together as needed.needed.

In the early days of RDBMS, many vendors claimed to In the early days of RDBMS, many vendors claimed to offer relational databases when they did not – Codd offer relational databases when they did not – Codd came up with came up with 12 rules 12 rules defining the requirements for a defining the requirements for a database system to be truly relational.database system to be truly relational.

TablesTablesRelational tables have these properties:Relational tables have these properties:

Column Values Are of the Same Kind Column Values Are of the Same Kind

Each Row is Unique Each Row is Unique

The Sequence of Columns is not significant The Sequence of Columns is not significant

The Sequence of Rows is not significant The Sequence of Rows is not significant

Each column must have a unique name (within the table)Each column must have a unique name (within the table)

Represents a single Represents a single entityentity. Example:. Example:

ColumnsColumnsTable columns have these properties:Table columns have these properties:

They have a data type (similar to variables, but slightly different) – They have a data type (similar to variables, but slightly different) – char, varchar, number, float, text, BLOB (image), etc.char, varchar, number, float, text, BLOB (image), etc.

Column values should be independent from each other.Column values should be independent from each other.

Values may be required (not null), or nullable. (Some databases Values may be required (not null), or nullable. (Some databases differentiate between null and zero-length string).differentiate between null and zero-length string).

Columns may be indexed to improve access speed.Columns may be indexed to improve access speed.

A column may be used as the basis of the order of the physical data A column may be used as the basis of the order of the physical data (clustered index).(clustered index).

Each represents an independent Each represents an independent attributeattribute..

Primary KeysPrimary Keys The The primary keyprimary key is an attribute or a set of attributes that is an attribute or a set of attributes that

uniquely identify a specific instance of an entity. uniquely identify a specific instance of an entity.

Every entity in the data model must have a primary key Every entity in the data model must have a primary key whose values uniquely identify instances of the entity.whose values uniquely identify instances of the entity.

To qualify as a primary key for an entity, an attribute must To qualify as a primary key for an entity, an attribute must have the following properties:have the following properties:

– it must have a non-null value for each instance of the entity. it must have a non-null value for each instance of the entity. – the value must be unique for each instance of an entity. the value must be unique for each instance of an entity. – the values must not change or become null during the life of the values must not change or become null during the life of

each entity instance each entity instance

Composite and Artificial Composite and Artificial KeysKeys

Sometimes more than one attribute is required to Sometimes more than one attribute is required to uniquely identify an entity. A primary key that uniquely identify an entity. A primary key that made up of more than one attribute is known as a made up of more than one attribute is known as a composite keycomposite key..

An An artificial keyartificial key is one that has no intrinsic is one that has no intrinsic meaning. These can be very useful when no meaning. These can be very useful when no attribute has all the primary key properties, or attribute has all the primary key properties, or the meaning of the primary key is otherwise the meaning of the primary key is otherwise complicated or conditional. complicated or conditional.

Foreign KeysForeign Keys

A A foreign keyforeign key is an attribute that defines a relationship is an attribute that defines a relationship between two tables. Foreign keys provide a method for between two tables. Foreign keys provide a method for maintaining integrity in the data (referential integrity). maintaining integrity in the data (referential integrity). Every relationship between entities must be supported Every relationship between entities must be supported by a foreign key. by a foreign key.

A foreign key is an attribute in one table and a primary A foreign key is an attribute in one table and a primary key in another.key in another.

Typically, an RDBMS will let you specify whether Typically, an RDBMS will let you specify whether dependent entities are deleted (cascade delete) when a dependent entities are deleted (cascade delete) when a parent is deleted, or disallowed. This ensures the parent is deleted, or disallowed. This ensures the referential integrity of the database.referential integrity of the database.

Foreign keys make possible establishing relationships in Foreign keys make possible establishing relationships in an Entity-Relationship Diagram [see sample below].an Entity-Relationship Diagram [see sample below].

Primary and Foreign KeysPrimary and Foreign Keys

Example:Example:

http://msdn.microsoft.com/en-us/library/ms171971.aspx

IndexesIndexes Disk reads are expensive.Disk reads are expensive. Finding a record in a large table can be slow, Finding a record in a large table can be slow,

often requiring a full table scan, that is, often requiring a full table scan, that is, checking every row in the table.checking every row in the table.

Indexes allow for much more efficient Indexes allow for much more efficient searching.searching.

If the performance of a query is bad, most of If the performance of a query is bad, most of the time it can be fixed with better indexes.the time it can be fixed with better indexes.

Indexes tend to decrease query time, but Indexes tend to decrease query time, but increase insert/delete time, since they need increase insert/delete time, since they need to be adjusted (automatically) when the data to be adjusted (automatically) when the data changes.changes.

IndexesIndexes


IndexesIndexes

Example of an index minimizing disk Example of an index minimizing disk access:access:

IndexesIndexes Spatial data doesn’t have a simple natural ordering like Spatial data doesn’t have a simple natural ordering like

most traditional data types.most traditional data types. In PostGIS, interacting with spatial indexes is similar to In PostGIS, interacting with spatial indexes is similar to

other data types, but internally is handled differently using other data types, but internally is handled differently using the Generalized Search Tree algorithm. the Generalized Search Tree algorithm.

This allows flexible partitioning based on the kind of data This allows flexible partitioning based on the kind of data being indexed, rather than on an existing alphanumeric being indexed, rather than on an existing alphanumeric sequencesequence

PostGIS has implemented spatial partitioning algorithms for PostGIS has implemented spatial partitioning algorithms for GIST indexes.GIST indexes.

http://gist.cs.berkeley.edu/gist1.htmlhttp://gist.cs.berkeley.edu/gist1.html

Other Features of RDBMSOther Features of RDBMS TriggersTriggers Transaction SupportTransaction Support Stored ProceduresStored Procedures ViewsViews User-Defined FunctionsUser-Defined Functions User-Defined Data TypesUser-Defined Data Types Extended features:Extended features:

– Full-text searchFull-text search– Spatial dataSpatial data

ReplicationReplication Others…Others…

NormalizationNormalization The most elementary structure for a database is a single table, The most elementary structure for a database is a single table,

which corresponds to a spreadsheet or attribute table.which corresponds to a spreadsheet or attribute table.

In this model there is one kind of entity (consisting of rows) which In this model there is one kind of entity (consisting of rows) which has a uniform set of attributes (consisting of columns).has a uniform set of attributes (consisting of columns).

This is fast and simple, but very limited.This is fast and simple, but very limited.

Example of un-normalized data – NYC PLUTO tax lot data:Example of un-normalized data – NYC PLUTO tax lot data:

NormalizationNormalization While this might adequately capture the data we need, it has While this might adequately capture the data we need, it has

some drawbacks:some drawbacks:

– There will be lots of redundant data if many parcels are There will be lots of redundant data if many parcels are added. The borough and owner names, for example, added. The borough and owner names, for example, will all be replicated for each row where these values will all be replicated for each row where these values are the same. This makes maintaining the data are the same. This makes maintaining the data difficult.difficult.

– What happens if a single parcel is zoned for more uses What happens if a single parcel is zoned for more uses than the number of columns allow? Or if one parcel than the number of columns allow? Or if one parcel has several owners? has several owners?

– How do you know for sure which parcel is which? How do you know for sure which parcel is which?

NormalizationNormalization The solution is to begin normalizing the data. You can look at this as a three-step process The solution is to begin normalizing the data. You can look at this as a three-step process

implementing each of Codd’s “Normal Forms”.implementing each of Codd’s “Normal Forms”.

First normal form rules:First normal form rules:

– A row of data cannot contain repeating groups of similar data (atomicity); and A row of data cannot contain repeating groups of similar data (atomicity); and

– Each row of data must have a unique identifier or primary key. Each row of data must have a unique identifier or primary key.

Second normal form rules:Second normal form rules:

– No attribute can be dependent on only a portion of the primary key. No attribute can be dependent on only a portion of the primary key.

– Every column must depend only on the entire primary key; if it is dependent on one or Every column must depend only on the entire primary key; if it is dependent on one or more other columns, these should be moved into new tables.more other columns, these should be moved into new tables.

Third normal form:Third normal form:

– No dependencies within non-key attributes.No dependencies within non-key attributes.

NormalizationNormalization Normalization usually involves creating multiple tables, Normalization usually involves creating multiple tables,

each of which complies with the three normal forms. each of which complies with the three normal forms. A normalized version of the PLUTO data, for example, A normalized version of the PLUTO data, for example,

would likely have separate zoning and owner tables.would likely have separate zoning and owner tables. However, in practice it is often useful to de-normalize the However, in practice it is often useful to de-normalize the

database structure. The PLUTO data in its current form database structure. The PLUTO data in its current form is extremely usable, and de-normalizing it for usability is extremely usable, and de-normalizing it for usability makes sense.makes sense.

Typically reporting applications will use some form of de-Typically reporting applications will use some form of de-normalization to improve query performance, while more normalization to improve query performance, while more transactional applications will use normalized database transactional applications will use normalized database structures to minimize redundant updates.structures to minimize redundant updates.

SQLSQL Structured Query Language is the interface you use to communicate Structured Query Language is the interface you use to communicate

with an RDBMS.with an RDBMS.

It consists of It consists of QueriesQueries; ; DMLDML – data manipulation language; and – data manipulation language; and DDLDDL – – data definition language.data definition language.

SQL is standardized, but different DB vendors have different flavors SQL is standardized, but different DB vendors have different flavors and extensions (Oracle Spatial, for example, adds spatial keywords to and extensions (Oracle Spatial, for example, adds spatial keywords to SQL).SQL).

SQL is not really a full featured language like C#, although most SQL is not really a full featured language like C#, although most database vendors have SQL-based languages like PL/SQL or TSQL that database vendors have SQL-based languages like PL/SQL or TSQL that let you embed SQL statements directly in procedural code.let you embed SQL statements directly in procedural code.

Typically, a program will interact with a database by submitting SQL Typically, a program will interact with a database by submitting SQL statements, one by one, to a database using a data access layer that statements, one by one, to a database using a data access layer that sends the requests to the database and returns the results to the sends the requests to the database and returns the results to the program.program.

SQLSQL

DDL statements - examples:DDL statements - examples:

CREATE TABLECREATE TABLE - creates a new database table - creates a new database table

ALTER TABLE ALTER TABLE - changes a database table’s - changes a database table’s structurestructure

DROP TABLEDROP TABLE - deletes a database table - deletes a database table CREATE INDEXCREATE INDEX - creates an index - creates an index

DROP INDEXDROP INDEX - deletes an index - deletes an index

SQLSQL

Queries:Queries:

SELECTSELECT – extracts data from a database table – extracts data from a database table

DML statements:DML statements:

UPDATEUPDATE - updates data in a database table - updates data in a database table

DELETEDELETE - deletes data from a database table - deletes data from a database table

INSERT INSERT - inserts new data into a database table - inserts new data into a database table

SQLSQLReturn all the rows in a table:Return all the rows in a table:

SELECT * from TableNameSELECT * from TableName

Order by a field:Order by a field:

SELECT * from TableName SELECT * from TableName ORDER BY ColumnNameORDER BY ColumnName

Add a where clause:Add a where clause:

SELECT * from TableNameSELECT * from TableNameWHERE ColumnName < 5WHERE ColumnName < 5ORDER BY ColumnName ORDER BY ColumnName

Joining two tablesJoining two tables::

SELECT Table1.Column1, Table2.Column1SELECT Table1.Column1, Table2.Column1FROM Table1 INNER JOIN Table2 FROM Table1 INNER JOIN Table2 ON Table1.Id = Table2.IdON Table1.Id = Table2.Id

Others:Others:

SELECT DISTINCT Column1 from TableNameSELECT DISTINCT Column1 from TableNameSELECT COUNT(*) from TableNameSELECT COUNT(*) from TableNamesee this discussion for others.see this discussion for others.

http://db.grussell.org/imp.html

Data Access LayerData Access Layer

Databases come with interactive SQL query interfaces.Databases come with interactive SQL query interfaces. Programmatic interfaces are provided by the DB vendors Programmatic interfaces are provided by the DB vendors

as well as third party vendors.as well as third party vendors. There are many different standards and conventions There are many different standards and conventions

such as ODBC, ADO, ADO.NET, JDBC, OLE DB (to see such as ODBC, ADO, ADO.NET, JDBC, OLE DB (to see a list of Microsoft’s for example, see a list of Microsoft’s for example, see http://msdn2.microsoft.com/en-us/library/ms810810.aspxhttp://msdn2.microsoft.com/en-us/library/ms810810.aspx..

Data Access LayerData Access Layer

Typical application sequence:Typical application sequence:– Connect to database Connect to database

Requires editable string with database Requires editable string with database connection info; usually stored in connection info; usually stored in configuration file.configuration file.

Requires exception handling in case Requires exception handling in case connection is unsuccessful!connection is unsuccessful!

– Execute SQLExecute SQL– Iterate through results (if query)Iterate through results (if query)

Display/output dataDisplay/output data– Close connectionClose connection

Data Access LayersData Access Layers ADO.NET is commonly used in C#:ADO.NET is commonly used in C#:

SQLServerConnection Conn = new SQLServerConnection("host=nc-star;port=1433; User ID=test01;Password=test01;SQLServerConnection Conn = new SQLServerConnection("host=nc-star;port=1433; User ID=test01;Password=test01; Database Name=Test"); Database Name=Test"); try try { {

Conn.Open(); Conn.Open(); } } catch (SQLServerException ex) catch (SQLServerException ex) { {

Console.WriteLine(ex.Message); return; Console.WriteLine(ex.Message); return; } }

try try { {

string strSQL = "SELECT ename FROM emp WHERE sal>50000"; string strSQL = "SELECT ename FROM emp WHERE sal>50000"; SQLServerCommand DBCmd = new SQLServerCommand(strSQL, Conn); SQLServerCommand DBCmd = new SQLServerCommand(strSQL, Conn); SQLServerDataReader myDataReader; SQLServerDataReader myDataReader; myDataReader = DBCmd.ExecuteReader(); myDataReader = DBCmd.ExecuteReader();

while (myDataReader.Read()) while (myDataReader.Read()) { {

Console.WriteLine("High salaries: " + myDataReader["ename"].ToString()); Console.WriteLine("High salaries: " + myDataReader["ename"].ToString()); } } myDataReader.Close(); myDataReader.Close(); Conn.Close(); Conn.Close();

} } catch (Exception ex) catch (Exception ex) { {

Console.WriteLine(ex.Message); Console.WriteLine(ex.Message); return; return;

} }

Spatial DataSpatial Data Spatial data can be represented using a relational database Spatial data can be represented using a relational database

structure.structure.

But this is complex and inefficient, requiring multiple tables to But this is complex and inefficient, requiring multiple tables to capture topology.capture topology.

Usually spatial data can best be captured by a binary column that Usually spatial data can best be captured by a binary column that contains all of the geometry associated with a record.contains all of the geometry associated with a record.

A table in a spatial database will usually have a single geometry A table in a spatial database will usually have a single geometry column of a single type, although a generic Geometry column can column of a single type, although a generic Geometry column can support multiple geometry types.support multiple geometry types.


Spatial DatabasesSpatial Databases Spatial databases are usually built on top of RDBMS.Spatial databases are usually built on top of RDBMS.

Spatial requirements are more involved, including Spatial requirements are more involved, including referential integrity based on topological geometrical referential integrity based on topological geometrical relationships, not just foreign-key constraints.relationships, not just foreign-key constraints.

OGC has a SQL spec for simple features storage very OGC has a SQL spec for simple features storage very similar to the simple features spec used for geo tools: similar to the simple features spec used for geo tools: http://www.opengeospatial.org/standards/sfshttp://www.opengeospatial.org/standards/sfs

PostGIS complies with OGC spec.PostGIS complies with OGC spec.

Spatial DatabasesSpatial Databases

Sample join with spatial columns:Sample join with spatial columns:


Spatial DatabasesSpatial Databases

Mechanics of a spatial query:Mechanics of a spatial query:


PostGISPostGIS

PostGIS is the spatial add-on for PostgreSQLPostGIS is the spatial add-on for PostgreSQL Compares well to others but is free! Compares well to others but is free!

(www.postgresql.org).(www.postgresql.org). Based on relational structure with support for Based on relational structure with support for

spatial data types and spatial functions.spatial data types and spatial functions. Can be used to serve up map layers.Can be used to serve up map layers. Can also be used for typical database Can also be used for typical database

transactional functions. transactional functions. No intrinsic support for raster data, unlike ArcSDE No intrinsic support for raster data, unlike ArcSDE

or Oracle Spatial.or Oracle Spatial.

PostGISPostGIS RDBMS use database tables and columns to represent the RDBMS use database tables and columns to represent the

database structure itself.database structure itself. PostGIS supports GIS data by adding additional metadata PostGIS supports GIS data by adding additional metadata

tables:tables:

– spatial_ref_sysspatial_ref_sys – contains a definition for each – contains a definition for each

available coordinate system and projection. These can available coordinate system and projection. These can be added or edited as needed.be added or edited as needed.

– geometry_columnsgeometry_columns – contains a definition of each – contains a definition of each geometry column in the current database.geometry column in the current database.

http://www.opengis.org/techno/interop/EPSG2WKT.TXThttp://www.opengis.org/techno/interop/EPSG2WKT.TXT http://www.opengis.org/techno/specs.htmhttp://www.opengis.org/techno/specs.htm

PostGISPostGIS

PostGIS geometry is based on the Open PostGIS geometry is based on the Open Geospatial Consortium simple features Geospatial Consortium simple features specification:specification:

http://www.opengeospatial.org/standards/sfs

PostGISPostGIS

Uses human-readable well-known text format to Uses human-readable well-known text format to represent geometry (converts to binary represent geometry (converts to binary internally):internally):


PostGISPostGIS

Sample PostGIS SQL - note special role of Sample PostGIS SQL - note special role of AddGeometryColumn, which manages AddGeometryColumn, which manages geometry_columns table.geometry_columns table.


PostGISPostGIS Examples of supported functions – Examples of supported functions –

Comparisons:Comparisons:– ST_Equals, ST_Disjoint, ST_Intersects, ST_Touches, ST_Equals, ST_Disjoint, ST_Intersects, ST_Touches,

ST_Crosses, ST_Within, ST_Contains, and ST_Crosses, ST_Within, ST_Contains, and ST_Overlaps. ST_Overlaps.

Set operations: Set operations: – ST_Intersection, ST_Difference, ST_Union, ST_Intersection, ST_Difference, ST_Union,

ST_SymDifference, ST_Buffer, and ST_ConvexHullST_SymDifference, ST_Buffer, and ST_ConvexHull

Others:Others:– AsText, GeometryFromText, Transform (re-project).AsText, GeometryFromText, Transform (re-project).

PostGISPostGIS Non-spatial data can be loaded from the SQL interface using the Non-spatial data can be loaded from the SQL interface using the

Copy command.Copy command. Spatial data can be loaded using a command-line utility that Spatial data can be loaded using a command-line utility that

comes with the database, or a GUI shape file loader.comes with the database, or a GUI shape file loader.


PostGISPostGIS Data can be exported from PostGIS using the Data can be exported from PostGIS using the

query interface.query interface.

Spatial data can be exported to a shape file using Spatial data can be exported to a shape file using the pgsql2shp command-line utility. The basic the pgsql2shp command-line utility. The basic syntax is:syntax is:

psql2shp [<options>] <database> <SQL query>psql2shp [<options>] <database> <SQL query>

See http://www.bostongis.com/postgis_quickguide_1_4.bqg See http://www.bostongis.com/postgis_quickguide_1_4.bqg

PostGISPostGIS

PGAdmin is easy-to-use admin interface for PGAdmin is easy-to-use admin interface for Postgresql database. It includes a pane for DDL Postgresql database. It includes a pane for DDL and an interactive SQL query tool.and an interactive SQL query tool.

PostGISPostGIS

Sample application: NYC Solar MapSample application: NYC Solar Map– Vector data and raster tile metadata stored Vector data and raster tile metadata stored

in PostGIS databasein PostGIS database– Raster tiles stored on file systemRaster tiles stored on file system– Data accessed managed by C# web Data accessed managed by C# web

servicesservices– Web services communicate with web Web services communicate with web

application via JSON (Javascript object application via JSON (Javascript object notation), with GeoJSON extensions.notation), with GeoJSON extensions.

– Client application uses Javascript to display Client application uses Javascript to display map layers and query C# web servicemap layers and query C# web service

Documents

Databases, Spatial Databases, and PostGIS