Sensor Data Management with Model-based View

LSIR, EPFL

Motivation

•Building scientific models is essential to environmental science applications

•data cleaning (statistical analysis)

•visualization (interpolation models)

•event detection (statistical analysis, prediction models ... )

•simulations

Motivation

DatabaseSensor

Readings

Query

Data

FilesArcGIS / Matlab

Model Based View

View Specificat

ion database

access

Sensor Readings

Create Interpolation View…

ArcGIS / Matlab

Model Based View

•Building models as database views

•Convenient and elegant

•Uniform access to both raw data and model-derived data

•Real-time Visualization

•Optimize the computation procedure

•Parallel computation

Visualization

•Example – Snow Cover Distribution

Measured value

Interpolated value

Snowcover – in mm

> 1000mm

500 - 1000mm

200 - 500 mm

< 200 mm

User-defined model

Interpolation

•Raw sensor reading at sampled places and the sampled times.

•Build a interpolation model view

•Query values at any place any time

•Visualization

Interpolation Procedures

•Linear Interpolation

•Neighbor search

•Weight computation

•Value estimation

Interpolation Models

•Nearest Neighbor

•Average

•Inverse Distance

•Triangulation

•Kriging

Storage Management

•Materialized

•Non-materialized

•Partially Materialized

•Materializing Internal Variables

•weights

Historical Data Access

t=3ID Weight

ij

k

Wi

Wj

Wk

t=3

vi

vjvk

Real-time View

t=1

Real-time View

t=2

Real-time View

t=3

Data Cube for Interpolated Data

•Display/Visualize aggregate measurements

•a time interval, a particular area etc.

•quickly zoom in/out in both space and time

Data Cube for Interpolated Data

•Data cube

•Multi-dimensional and hierarchical aggregates

•time: 5 minute, 30 minute, hour, day, week, month, year, all

•sensor: sensor, region, site, all

•area: 10 m2 ,100 m2 , 1 km2

•Efficient drill down, roll up

Data Cube

all

site

sensor

Location Dimension Measurement Type Dimension

Time Dimension

measurement type

all

year

week

day

hour

minutedata measures

System Design (Alternative 1)

•Data cube with view materialization

ViewComputation

Data View

Sensor data stream

SQL Server

Database Engine

Analysis Service

Client Application

Archive

System Design (Alternative 1)

•Data cube with view materialization Storage explosion

Large region, fine granularity, high update rate

Cost inefficiency

What if data of interest only constitute a small portion of the entire cube

•How about materialize the view only when it is explicitly requested?

System Design (Alternative 2)

Utilize the cube on raw sensor data

Store internal variables

Compute the cube (i.e. compute the view) only when it is explicitly requested by the users

•Data cube without view materialization

The computation has to be simple and fast !

System Design (Alternative 2)

CubeConstruction

Sensor data stream

SQL Server

Database Engine

Analysis Service

Client Application

Internal Variables

•Data cube without view materialization (lazy evaluation)

System Design (Alternative 2)

•About internal variables weights

Cube built from measured data

ID Weight

ij

k

Wi

Wj

Wk

Internal Variables

•Support common modeling operations within database

•Interpolation

•Hidden Markov Model etc

•Interactions with external tools, s.a. Matlab / ArcGIS

Research Plan