Database Server Extension for managing and querying 4D gridded spatiotemporal data

Database Server Extension for

managing and querying 4D gridded

spatiotemporal data

Presented at the Edinburgh e-Science Institute Nov 1-2, 2005 conference on “Spatiotemporal Databases”

by

Ian BarrodaleBarrodale Computing Services Ltd. (BCS)

http://www.barrodale.com

Barrodale Computing Services Ltd. (BCS)

“At BCS we let the actual tasks that our clients are trying to accomplish guide our solutions, rather than producing software that dictates how clients can perform their work.”

Provides customized software and R&D services to technical clients

Successfully completed 450+ software development projects since incorporation in 1978

Long-term professional staff

IBM Business Partner

Major clients include:Canada - Province of BC (Elections BC, Ministry of Forests), DND,...USA - US Navy, NOAA, IBM, SPAWAR, Univ. of Mississippi,...


Some Application Areas:

• Defense Sciences (ASW, MCM, METOC)

• Elections/Census (Geo-Spatial Database)

• Forestry (Spatial Timber Supply Models)

• Terrain Modeling (Watershed Delineation)

• Seabed Monitoring (Gas Hydrates)

Some Skill Sets:

• Mathematical Analysis• Algorithm Development• Signal & Image Processing• Modeling & Simulation• Software Engineering• Spatial Data Analysis• Spatial Database Design• Database Server Extensions• Large Dataset Management• Graphical User Interfaces• Data Visualization• Web Map Services

Complex DataSimple Data

Query

No Query File System

RelationalDBMS

ObjectOrientedDBMS

ObjectRelational

DBMS

(Simplistic) Database Classification Matrix

File Server vs. RDBMS + File Server

Files for both metadata and data

vs. RDBMS for metadata & files for data.

File Server alone:

+ Simpler.

+ Less expensive.

± Metadata stored in data file name/directory or inside gridded data file.

RDBMS + File Server:

+ Integrity checking of metadata - integrity checking of metadata can be performed by built-in RDBMS features (check constraints, triggers, etc.).

+ Efficient access to metadata - e.g., indices can be used.

+ Easier to locate gridded data of interest - e.g., complicated queries on metadata can be performed.

− Metadata separated from gridded data - data inconsistencies possible.

Object Relational DBMS

RDBMS for metadata & fileserver for data

vs. ORDBMS (metadata & data integrated).

ORDBMS:

+ Improved concurrency - concurrent users can safely query the same gridded data.

+ Composite data types - gridded data bundled with their metadata.

+ Improved integrity - ability to reject bad gridded data before it is stored in ORDBMS.

+ Database extensibility - easy addition of data types and operations.

+ Uniform treatment of data items - SQL interface can perform complex queries based on any of these data items, e.g., metadata as well as gridded data; less need for custom 3GL programming.

+ Custom data access methods - e.g., R-tree indexes.

+ Point-in-time recovery of gridded data possible.

+ Built-in complex SQL functions for gridded data operations - e.g., aggregating, slicing, subsetting, reprojecting, subsampling, ...

BCS specializes in ORDBMS ApplicationsThe current main platforms for BCS database applications are IBM Informix Dynamic Server and PostgreSQL . Object Relational Data Base Management Systems (ORDBMSs) have four features that set them apart from traditional DBMSs:

User-defined abstract data types (ADTs). ADTs allow new data types with structures suited to particular applications to be defined.

User-defined routines (UDRs). UDRs provide the means for writing customized server functions that have much of the power and functionality expressible in C.

“SmartBLOBs”. These are disk-based objects that have the functionality of random access files. ADTs use them to store any data that does not fit into a table row.

Flexible spatial indexing. R-tree indexing for multi-dimensional data enables fast searching of particular ADTs in a table.

Example: Sum the “area” (UDR) of all “lakes” (ADT) contained (R-tree) in “British Columbia” (ADT)

Query: From a given point on a stream, what is the entire area from which drainage is

received?

“SQL” Example 1: Find the area of the watershed that is upstream from where a given road crosses a given stream.

SELECT Area(Watershed(streamElement, (Intersection(streamElement, roadElement))))

FROM streamNetwork, roadNetworkWHERE Overlap(Intersection(Box(streamElement),

Box(roadElement)), userDefinedArea);

Note:userDefinedArea is, say, a string provided by the user.UDRs:BOX - rectangle enclosing objectINTERSECTION - common areaOVERLAP - T or FWATERSHED - calculates watershed upstream from a pointAREA - calculates area

“SQL” Example 2: Find all side-scan sonar images, that are in a user-defined area, with a heading within one degree of 128.3 degrees and with an average slant range of less than 50 m.

SELECT image

FROM sonarImageArchive

WHERE Overlap(Box(image), userDefinedArea)

AND ABS(Heading(image) - 128.3) < 1.0

AND Average(SlantRange(image)) < 50.0;

Note:userDefinedArea is, say, a string provided by the user.UDRs:SLANTRANGE - calculates slant rangeAVERAGE - calculates averageHEADING - supplies heading of objectABS - absolute valueBOX - rectangle enclosing objectOVERLAP - T or F

“SQL” Example 3: In a user-defined area, overlay on a sea floor map all “West”-looking side scan sonar images of “sandy” sea floor bottom type.

SELECT Overlay(image, map)

FROM sonarImageArchive, seaFloorMapping

WHERE Overlap(Box(image), userDefinedArea)

AND Overlap(Box(map), Box(image))

AND SlantDirection(image) = “West”

AND surfaceType = “sandy”;

Note:userDefinedArea is, say, a string provided by the user,and surfaceType is a column in seaFloorMapping.UDRs:SLANTDIRECTION - calculates slant directionBOX - rectangle enclosing objectOVERLAP - T or FOVERLAY - overlays one image on another

• Gridded data occurs in meteorology, oceanography, the life sciences, non-destructive testing, exploration for oil, natural gas, coal & diamonds,…

• These datasets range from simple, uniformly spaced grid points along a single dimension (e.g., time series) to multidimensional grids containing several types of values (e.g., 4D cubes of meteorological attributes).

• Grids have typically been stored in simple files and then manipulated by programs that operated on these files. Nowadays there is increasing justification for storing and manipulating gridded data in DBMSs: the principal advantages are their ability to (i) ensure data integrity and consistency, and (ii) provide diverse users with independent and effective query-based access to these data across multiple applications and systems.

Gridded Data in Databases

• However, implementing an efficient gridded DBMS can be very challenging, particularly when it involves Binary Large Objects (BLOBs), user-defined abstract datatypes (ADTs) that encapsulate grid data structures and attributes, and user-defined routines (UDRs) with which applications can create, manipulate and access the gridded data stored in these new datatypes.

• BCS has developed an efficient technology that supports database storage, update, and fast retrieval of gridded data; it uses BLOBs, ADTs, and UDRs.

• Our first implementation of this technology was a Grid DataBlade for IBM Informix, and then a Grid Extension for PostgreSQL; we are currently developing an analogous Grid Cartridge for use with Oracle.

Gridded Data in Databases

• is designed to handle 1D, 2D, 3D, 4D (and “5D”) grids.

• stores grids using SmartBLOBS and a (user-controlled) tiling scheme that together permit very efficient generation of products (e.g., oblique slices or 1D sticks from 4D grids).

• sometimes provides more than 50-fold increases in speed of data product generation compared to the conventional approach that does not involve tiling or SmartBLOBs.

• can store the data in, and convert it between, hundreds of mapping projections.

• can handle irregularly spaced grids in any/all grid dimensions.

• can handle the presence of multiple vector and/or scalar values.

• provides several interpolation options.

• provides for convenient database loading and extraction of grid files via one form of the commonly used NetCDF format.

• provides C, Java, and SQL application programming interfaces.

• is supplied with full user/programmer documentation.

The BCS Grid DataBlade/Extension

U.S. Navy Solution

Worldwide weather grid

Used API to developgrid types, functions

& indexes

Grid types Grid functions

BCS Grid DataBlade

ORDBMS

SQL

User queryGet grid sampleof interest

Sample of interest

U.S. Navy: Tactical Environmental Data Services

Humidity-Humidity-Refractive EffectsRefractive Effects

Air TemperatureAir TemperatureAerosolsAerosolsDustDust

TrafficabilityTrafficability

FogFog

Soil MoistureSoil MoistureBeach ProfileBeach Profile

WavesWavesReefs, Bars, ChannelsReefs, Bars, Channels

Sediment TransportSediment Transport

Shelf /Shelf /InternalInternalWavesWaves

Swell / WaveSwell / WaveRefractionRefraction

Hydrography - Fine ScalesHydrography - Fine Scales

IceIce

BiologicsBiologics

Slope (Sea Floor)Slope (Sea Floor)

Coastal ConfigurationCoastal Configuration

Tidal PulseTidal Pulse

Sensible andSensible andLatent HeatLatent Heat

Wind Speed / DirectionWind Speed / Direction

Land CoverLand Cover

TerrainTerrain

SurfSurf

TurbidityTurbidity

Rain RateRain Rate

StraitsStraits

Island FlowIsland Flow

Wind - Driven CirculationWind - Driven Circulation

WrecksWrecks

Medical Application Demohttp://www.barrodale.com/grid_Demo/GridBladeApplet.html

Medical Application Demohttp://www.barrodale.com/grid_Demo/GridBladeApplet.html

Grid Fusing: Visualized through IDV




Grid Fusing: SQL for this example

SELECT GRDFuse( GRDFuseCollect(GRDPriorityGrid(image,1.0)), '((grdspec (translation -90.4 29.57 0 0) (affine_transformation 0 0 0 .001 0 0 .001 0

0 1 0 0 1 0 0 0) (dim_sizes 1 1 800 800)) (rules(weight)) )') FROM images i, places_of_interest p WHERE i.imageType = 'aerialPhoto' AND overlap(grdbox(i.image),grdbox(p.loc)) AND p.name = 'New Orleans';

SQL driving the Grid Fusion

SELECT GRDFuse( GRDFuseCollect(GRDPriorityGrid(image,1.0)), '((grdspec (translation -90.4 29.57 0 0) (affine_transformation 0 0 0 .001 0 0 .001 0 0 1 0 0 1 0 0 0) (dim_sizes 1 1 800 800)) (rules(weight)) )') FROM images i, places_of_interest p WHERE i.imageType = 'aerialPhoto' AND overlap(grdbox(i.image),grdbox(p.loc)) AND p.name = 'New Orleans';

UDR to resample a set of grids into a single grid.



Two UDRs to build a set of transient grids, associating a floating-point value with each of these grids. This floating-point value is later used to establish the relative weight of each grid’s elements in producing the fused grid. We’ve chosen each grid to have equal weight.



Each source grid is resampled at the same locations, using the source images’ spatial reference system, which is a Lat-Lon grid. The fused grid’s horizontal resolution is 0.001 degrees.



The source of the grids is a table called “images”.


SELECT GRDFuse( GRDFuseCollect(GRDPriorityGrid(image,1.0)), '((grdspec (translation -90.4 29.57 0 0) (affine_transformation 0 0 0 .001 0 0 .001 0 0 1 0 0 1 0 0 0) (dim_sizes 1 1 800 800)) (rules(weight)) )') FROM images i, places_of_interest p WHERE i.imageType = 'aerialPhoto' AND overlap(grdbox(i.image),grdbox(p.loc)) AND p.name = ‘New Orleans';

We use metadata stored in another column to pick only those images derived from aerial photographs.



A second table called “places_of_interest” is used to include only source grids that overlap a region called “New Orleans”.

Barrodale Grid Datablade for IBM Informix

select GRDExtract(grid, "((dim_sizes 1 1 600 600)(dim_names time level row column)(translation -1489986.000000 6574741.000000 0 0)(affine_transformation 0 0 0 3338.898164 0 0 3338.898164 0 0 1 0 0 1 0 0 0)(srtext 'PROJCS[@World_Hammer_Aitoff@,GEOGCS[@GCS_WGS_1984@,DATUM[@D_WGS_1984@,SPHEROID[@WGS_1984@,6378137.0,298.257223563]],PRIMEM[@Greenwich@,0.0],UNIT[@Degree@,0.0174532925199433]],PROJECTION[@Hammer_Aitoff@],PARAMETER[@False_Easting@,0.0],PARAMETER[@False_Northing@,0.0],PARAMETER[@Central_Meridian@,-65.0],UNIT[@Meter@,1.0]]'))"::GRDSpec)from grdImages where g_keytext = "world2k";

Resample and reproject a 386 MB raster image of the World

www.barrodale.com/projectionDemo/ProjectionApplet.html

Barrodale Grid Datablade for IBM InformixSample a 4D grid along a flight path

Head Wind

Humidity

Cross Wind

Ocean Applications?

10m,10m,-100m

40m,30m,-100m

40m,20m,-100m

45m,10m,-100m40m,10m,-100m20m,10m,-100m

40m,30m,-90m

40m,30m,-50m

(t1,s1,p1)

(t6,s6,p6)

(t3,s3,p3)

(t4,s4,p4) (t5,s5,p5)

(t2,s2,p2)

EASTING

NORTHING

`

DEPTH

10m 20m 5m` ` ` `` `

`

`

`

`

40m

10m

• Grids can have 1, 2, 3, or 4 dimensions.

• Each grid point can store several variables.

• Some grid point values can be NULL.

• Grid spacing along axes can be non-uniform.

BCS Grid DataBlade/Extension: Grids of Data

10m,10m,-100m

40m,30m,-100m

40m,20m,-100m

45m,10m,-100m40m,10m,-100m20m,10m,-100m

40m,30m,-90m

40m,30m,-50m

(t1,s1,p1)

(t6,s6,p6)

(t3,s3,p3)

(t4,s4,p4) (t5,s5,p5)

(t2,s2,p2)

EASTING

NORTHING

`

DEPTH

10m 20m 5m` ` ` `` `

`

`

`

`

40m

10m

BCS Grid DataBlade/Extension: Grids of Data

• Orthogonal ….

• Oblique………………………...

• Radial ….

10m,10m,-100m

40m,30m,-100m

40m,20m,-100m

40m,10m,-100m30m,10m,-100m20m,10m,-100m

40m,30m,-99m

40m,30m,-98m

EASTING

NORTHING

`

DEPTH

20m,10m,-100m 30m,10m,-100m

30m,20m,-100m

30m,30m,-100m

30m,30m,-98m

10m,10m,-100m

40m,30m,-100m

40m,20m,-100m

40m,10m,-100m30m,10m,-100m20m,10m,-100m

40m,30m,-99m

40m,30m,-98m

EASTING

NORTHING

`

DEPTH

Original Grid Position

Interpolated Grid Position

10m,20m,-99m

10m,20m,-99m

10m,20m,-99m

10m,20m,-99m

30m,30m,-99m

40m,20m,-99m

40m,20m,-98m

30m,30m,-98m

10m,10m,-100m

40m,30m,-100m

40m,20m,-100m

40m,10m,-100m30m,10m,-100m20m,10m,-100m

40m,30m,-99m

40m,30m,-98m

EASTING

NORTHING

`

DEPTH



10m,10m,-98m

10m,10m,-100m

BCS Grid DataBlade/Extension: Types of Extraction

10m,10m,-100m

40m,30m,-100m

40m,20m,-100m

40m,10m,-100m30m,10m,-100m20m,10m,-100m

40m,30m,-99m

40m,30m,-98m

EASTING

NORTHING

`

DEPTH

20m,10m,-100m 30m,10m,-100m

30m,20m,-100m

30m,30m,-100m

30m,30m,-98m

BCS Grid DataBlade/Extension: Orthogonal Extraction

10m,10m,-100m

40m,30m,-100m

40m,20m,-100m

40m,10m,-100m30m,10m,-100m20m,10m,-100m

40m,30m,-99m

40m,30m,-98m

EASTING

NORTHING

`

DEPTH

10m,20m,-99m

10m,20m,-99m

10m,20m,-99m

10m,20m,-99m

30m,30m,-99m

40m,20m,-99m

40m,20m,-98m

30m,30m,-98m



BCS Grid DataBlade/Extension: Oblique Extraction

10m,10m,-100m

40m,30m,-100m

40m,20m,-100m

40m,10m,-100m30m,10m,-100m20m,10m,-100m

40m,30m,-99m

40m,30m,-98m

EASTING

NORTHING

`

DEPTH

10m,10m,-98m

10m,10m,-100m



BCS Grid DataBlade/Extension: Radial Extraction

• Individual Points

• Appending

• Replacing Slices

Longitude

Latitude

Elevation

t000 t300t200t100

t001

t310

t311

t101

t021 t321t221t121

t320

t000 t300t200t100

t001

t310

t311

t101

t021 t321t'221t121

t320

Longitude

Time

Elevation

0 321 44 0 321

Longitude

Latitude

Elevation

Original Value

New Value

OriginalGrid

UpdatedGrid

NewGrid

Piece

BCS Grid DataBlade/Extension: Types of Updates

Longitude

Latitude

Elevation

t000 t300t200t100

t001

t310

t311

t101

t021 t321t221t121

t320

t000 t300t200t100

t001

t310

t311

t101

t021 t321t'221t121

t320

BCS Grid DataBlade/Extension: Updating Points

Longitude

Time

Elevation

0 321 44 0 321

BCS Grid DataBlade/Extension: Appending a Grid

Longitude

Latitude

Elevation

OriginalGrid

UpdatedGrid

NewGrid

Piece

Original Value

New Value

BCS Grid DataBlade/Extension: Replacing a Slice

• JoinNew

• JoinExisting

• Union

Y

X

Time = 0

0

1

2

1 32

Y

X

Time = 4

0

1

2

1 32

Y

X

Time = 3

0

1

2

1 32

Y

X

Time = 2

0

1

2

1 32

Y

X

Time = 1

0

1

2

1 32

Y

X

Time

0 1 32

1

2

3

4

X

Time

0 1 32

1

2

3

4

1

2

Y

X

Time

0 1 32

1

2

3

4

1

2

Y

X

Time

3 4 65

1

2

3

4

1

2

Y

4 65

00

s00

s21s11s01

s20s10

depth

time

0

1

0

1 2

s02

s23s13s03

s22s122

3

t00

t21t11t01

t20t10

depth

time

0

1

0

1 2

t02

t23t13t03

t22t122

3

depth

time

1

0

2

3

s00,t00

s21,t21

s11,t11

s10,t10

0 1 2

s02,t02

s23,t23

s13,t13

s03,t03

s22,t22

s12,t12

s20,t20

s01,t01

BCS Grid DataBlade/Extension: Types of Aggregation

Y

X

Time = 0

0

1

2

1 32

Y

X

Time = 4

0

1

2

1 32

Y

X

Time = 3

0

1

2

1 32

Y

X

Time = 2

0

1

2

1 32

Y

X

Time = 1

0

1

2

1 32

Y

X

Time

0 1 32

1

2

3

4

BCS Grid DataBlade/Extension: JoinNew

X

Time

0 1 32

1

2

3

4

1

2

Y

X

Time

0 1 32

1

2

3

4

1

2

Y

X

Time

3 4 65

1

2

3

4

1

2

Y

4 65

00

BCS Grid DataBlade/Extension: JoinExisting

s00

s21s11s01

s20s10

depth

time

0

1

0

1 2

s02

s23s13s03

s22s122

3

t00

t21t11t01

t20t10

depth

time

0

1

0

1 2

t02

t23t13t03

t22t122

3

depth

time

1

0

2

3

s00,t00

s21,t21

s11,t11

s10,t10

0 1 2

s02,t02

s23,t23

s13,t13

s03,t03

s22,t22

s12,t12

s20,t20

s01,t01

BCS Grid DataBlade/Extension: Union

Grid Extraction Size Limit as a Function of Available Physical Memory

0

50

100

150

200

250

300

350

256 512 1024 1536

Physical Memory On Server (MB)

Rec

om

men

ded

Max

imu

m

Ext

ract

ion

Siz

e (M

B)

Informix

PostgreSQL

How much memory does the server need when

extracting a large gridded derived product using the BCS Grid DataBlade (Informix) or the BCS Grid

Extension (PostgreSQL)?

BCS Grid DataBlade: The Effect of Tile Size

• A good choice of tile size allows larger grids to be extracted.

Extraction Time vs. Extraction Size Default and Optimized Tile Size

0.020.040.060.080.0

100.0120.0140.0160.0180.0200.0

10

60

11

01

60

21

02

60

31

03

60

41

04

60

51

05

60

Size of Extract (MB)

Ex

tra

cti

on

tim

e (

se

co

nd

s)

Default Tile Size

Optimized TileSize

…..is designed for applications where:

1. The data volumes are such that they can’t be kept in memory.

2. The amount of data extracted, in a particular query, is small relative to the amount stored.

3. The data needs some form of resampling.

SUMMARY

The BCS Grid DataBlade/Extension

CONTACT INFORMATION:

Dr. Ian Barrodale, President


P.O. Box 3075 STN CSC

Victoria BC V8W 3W2 Canada

(250) 472 4332 voice, (250) 472 4373 fax

e-mail: [email protected]

For more information about BCS projects, experience, and capabilities, please visit:

http://www.barrodale.com

Documents

Database Server Extension for managing and querying 4D gridded spatiotemporal data