Event-Based Fusion of Distributed Multimedia Data Sources

Vincent OriaDepartment of Computer Science

New Jersey Institute of TechnologyNewark, NJ 07102

2

Outline

Classical Data Integration Problem Multimedia Data An Architectural approach to Multimedia Data

Integration Event-Based Integration of Data Sources Conclusion

3

Classical Data Integration*

* Borrowed from M. Lenzerini

4

Classical Data Integration Issues How to construct the global schema? (Automatic) source wrapping How to discover mappings between the sources and the global

schema? Limitations in the mechanisms for accessing the sources Data extraction, cleaning and reconciliation How to process updates expressed on the global schema, and

updates expressed on the sources? The modeling problem: How to model the mappings between the

sources and the global schema? The querying problem: How to answer queries expressed on the

global schema? Query optimization

5

Multimedia Data

Multimedia data management is more than physical server designLogical data modeling is important

Multimedia data management is more than similarity search

“Show me all the images that are similar to this one [in terms of color, texture, shape].”

Querying is much more complicated Give me all the news items on Baghdad over the last 2

weeks

6

Multimedia Data … Multimedia data is heterogeneous in both format

and in access primitives and this has to be accommodated

You cannot store all the data in a single DBMS; the system has to be open

Query-based access to multimedia data is important as well as browsing and some transactional access

Some DBMS-like interface and control over multimedia data should be provided

8

Multimedia Database Processing

Multimedia Data Preprocessing System

Database Processing

MM DataPre-

processor

AdditionalInformation

<!ELEMENT ..>.....<!ATTLIST...>

Multimedia DBMS

Users

Que

ry I

nter

face

MM DataInstance

<article>.....</article>

Recognized components

MM DataInstance

MM Data

Meta-Data

9

Document Database Architecture

Document Processing System

Database Processing

DTD or XML Schema files

Sch

ema

Par

ser

<!ELEMENT ..>.....<!ATTLIST...>

DTD/ XML

Manager

TypeGenerator

Document DBMS

Users

Que

ry I

nter

faceDTD/

XML Schema

Document content

DocumentsXML or SGML

DocumentInstance

ParseTree

<article>.....</article>

<!ELEMENT ..>.....<!ATTLIST...>

DTD/ XML Schema

Types

Objects

DocumentParser

InstanceGenerator

10

Image Database Architecture

Image Processing System

Database Processing

Image Analysis and Pattern Recognition

ImageAnnotation

Image DBMS

Users

Que

ry I

nter

face

Image ContentDescription

Image

Image

SyntacticObjects

SemanticObjects

<article>.....</article>

Meta-Data

11

Video Database Architecture

VideoProcessing System

Database Processing

Video Analysis and Pattern Recognition

VideoAnnotation

Video DBMS

Users

Que

ry I

nter

face

Video

KeyFrames

Meta-Data

<article>.....</article>

Video ContentDescription

Video

12

Multimedia Data Integration: An Architectural Perspective

Simple Client-Server Integrated Server Database Server Middleware and Mediation

13

Simple Client-Server

Heavy-duty client Synchronization, user interface, QoS, …

Client has to access each server Scalability problems

client code has to be updated when new servers come on-line

Meta-data

DatabaseServer

DatabaseServer

TextServerText

Server

Text

ImageServerImageServer

Images

CMServerCM

Server

Video/Audio

Client

14

“Integrated” Server

Heavy-duty server DBMS should be able to handle multiple storage systems Real-time constraints on CM

Meta-data

Video/AudioImagesText

ImageServer

CMServer

Object StorageServer

DBMS Functions

Client

15

Database Server

Lighter client Client has to access only one server Scalability problems

server may become a bottleneck - distribute and interoperate

Meta-data

CMServerCM

Server

Video/Audio

ImageServerImageServer

ImagesText

TextServerText

Server

DatabaseServer

DatabaseServer

Client

16

Document Server

Document-centric view Multimedia objects are parts of documents

Might be suitable for, e.g., e-commerce catalogs

Video/AudioImagesText

StructuredDocument

DBMS

ImageDBMS

CMDBMS

Client

17

Interoperable System

Middleware

WrapperWrapper Wrapper Wrapper WrapperText

DBMSText

DBMSVideoDBMS

ImageDBMS

ImageDBMS

Mediator Mediator Mediator

Mediator

Mediator

Client Client

18

Event-Based Multimedia Data Integration An event aims at modeling any happening

Facts, context An event has 3 components

Time Space (location) Objects

19

Events: Temporal Dimension

Time Line and Temporal relationships

Event1

Image Video

Text

Event2

Image Video

Event3

Image Video

Image

Time Line

20

Events: Spatial Dimension

GIS (Location and Spatial Relationships)

Event1Event2

Event3

Directional and Topological relationships

21

Events: Object Dimension

Which real world objects are involved in the event? Object Recognition Classical Data Integration

22

Event: Spatio-Temporal Dimension Moving Objects and their Trajectories

Raw representation: The trajectory T of a moving object is defined as a sequence

of vectors

T=[t1, …, tn] Each ri show the successive positions of the moving object over a period of time. Movement sequence:The trajectory of a moving object is represented by a sequence of (movement direction, distance ratio) pairs. This representation is not affected by rotation, shifting or scaling.

M=[m1, …, mn-1]

Each mi is a pair of (movement direction, distance ratio).

23

Event Model

Events model interpretation context Example: KIMCOE 2006 is an event

Participants are objects Location: Hilton Garden Inn, Suffolk, Virginia Date/Time: October 24 - 27, 2006 Has sub-events like sessions or visit of Lockheed Martin's

Center For Innovation Event Properties

Discrete or continuous Local or distributed Simple or composite Descriptors: Data (classical and multimedia)

24

Event Querying

Objects: RDBM, XML

Time

Space: GIS

25

Event Querying

Objects: RDBM, XML

Time

Space: GIS

26

Event Querying

Objects: RDBM, XML

Time

Space: GIS

27

Event Operators

Temporal Operator Spatial Operators Spatio-Temporal Operator Aggregation

28

Aggregation and Concept Hierarchy Dimensions are hierarchical by nature: total

orders or partial orders Example: Location(continent country

province city) Time(yearquarter(month,week)day)

Industry Country Year

Category Region Quarter

Product City Month Week

Office Day

29

Aggregation and Concept Hierarchy: Operators

roll-up (increase the level of abstraction) drill-down (decrease the level of abstraction) slice and dice (selection and projection) pivot (re-orient the multi-dimensional view) drill-through (links to the raw data)

30

Aggregation and Concept Hierarchy: Roll-up Use of aggregation to summarize at different levels of a dimension hierarchy Ex: if we are given total sales per city we can

aggregate on the market to obtain sales per state

Dayton

Q1 Q2 Q4

Drama

Horror

Sci. Fi..

Comedy

Time (Quarters)

Market(city, state)

Q3Newark

S. OrangeN. York

Category

Roll-up on Market

Ohio

Q1 Q2 Q4

Drama

Horror

Sci. Fi..

Comedy

Time (Quarters)

Market(States,,USA) Category

Q3New Jersey

New YorkArizona

31

Aggregation and Concept Hierarchy: Drill-down Inverse of roll-up

Given a total sales by state, we can ask for more detailed presentation by drilling down on market

Dayton

Q1 Q2 Q4

Drama

Horror

Sci. Fi..

Comedy

Market(city, state)

Q3Newark

S. OrangeN. York

Category

Drill-down on Market

Ohio

Q1 Q2 Q4

Drama

Horror

Sci. Fi..

Comedy

Time (Quarters)

Market(States,,USA) Category

Q3New Jersey

New YorkArizona

32

Aggregation and Concept Hierarchy: Dice and Slice

January

Slice on January

Newark

Electronics

JanuaryDice onElectronics andNewark

33

Conclusion

Event model: A data Integration model This is a work in progress: We need to fully

define the event model We want to build on existing Technology

(RDBMS, XML, GIS,..)