SITUATION RECOGNITION: AN EVOLVING PROBLEM FOR HETEROGENEOUS

DYNAMIC BIG MULTIMEDIA DATA

Vivek K. Singh1,2, Mingyan Gao1, Ramesh Jain1

1 University of California, Irvine 2 MIT Media Lab

Presenter: jain@ics.uci.edu

Sandy in New York: Situation today

2

Weather forecast

What do I do?

Imagine This

An Interesting Problem

When we were data poor – we searched for words in documents.

Now that we are data rich – should we still search for words?

Time has come for us to stop thinking data poor; really start thinking and behaving data rich.

Data, Information, Knowledge, Wisdom

4

Data is Essential. But, we are really interested in products:

Information, Knowledge, and Wisdom.

BIG DATA

5

Volume

Varie

ty

Big Data offers Big Opportunities.

The Grand Challenge

Sense making from multimodal massive geo-social data-

streams.

6

Social Networks

Connecting

People

Fundamental Problem

Connecting People to Resources effectively, efficiently, and promptly

in given situations.

Connecting People

And Resources

Social Life Networks

Aggregation and

Composition

Situation Detection

Alerts

Queries

Information

11/1/12 9

Concept Recognition: Last Century

10

Environments

Real world Objects

Situations

Activities

Single M

edia

SPACE TIME

Scenes Location aware

Visual Objects

Trajectories

Visual Events

Location unaware

Static Dynamic

Location aware

Location unaware

Static Dynamic

Data = Text or Images or Video

Visual Concept Recognition: First research papers

• 1963: Object Recognition [Lawrence + Roberts] • 1967: Scene Analysis [Guzman] • 1984: Trajectory detection [Ed Chang+ Kurz] • 1986: Event Recognition [Haynes + Jain] • 1988: Situation Recognition [Dickmanns]

1960 1970 1980 1990 2000 2010

Object Scene Trajectory

Event

Situation

Concept Recognition: This Century

12

Environments

Real world Objects

Situations

Activities

SPACE TIME

Location aware

Location unaware

Static Dynamic

Heterogeneous M

edia

Location aware

Location unaware

Static Dynamic

Data is just Data. Medium and sources do not matter.

Situations: Definition

An actionable abstraction of observed spatio-temporal characteristics.

13

11/1/12 14

A) Situation Modeling

B) Situation Recognition

C) Visualization, Personalization, and Alerts

…

STT Stream

Emage

Situation

C1 ⊕

v2 v3 ⊕

v5 v6

@

∏  

Δ @

i) Visualization

ii) Personalization

+

+ Available resources

iii) Alerts

Personal context

Personalized

situation

Overall framework 15

Challenge: Unifying Multimodal Big Data

• Spatio-temporal-thematic (STT) real-time streams • E-mage as a unifying representation

16

(a) Pollen levels (Source: Visual) (b) Census data (Source: text file) (c) Reports on ‘Hurricanes’ (source: Twitter stream)  

d) Cloud cover (Source: Satellite imagery) (e) Predicted hurricane path (source: KML) (f) Open shelters coverage(Source: KML)  

Situation Modeling Get_components (v){ 1)  Identify output state space 2)  Identify S-T bounds 3)  Define component features:

v=f(v1, …, vk) •  If (type = imprecise)

•  identify learning data source, method

4)  ForEach (feature vi) { If (atomic)

•  Identify Data source. •  Type, URL, ST bounds

•  Identify highest Rep. level reqd. •  Identify operations

Else Get_components(vi)

} }

17

ϵ { Low, Mid, High}

v f1

v4 v2 v3

USA, 5 mins,

0.01x 0.01

@

D1

Emage

Δ

D2

∏

Emage

Δ

D3

Δ

@

Emage D2

∏

Emage

Δ

f2

v5 v6

Situation recognition: Workflow

Level  1:  Unified  representa3on  (STT  Data)

Level  3:  Symbolic  rep.  (Situa3ons)

Proper3es

Proper3es

Proper3es

Level  0:  Raw  data  streams    e.g.  tweets,  cameras,  traffic,  weather,  …  

Level  2:  Aggrega3on  (Emage)  

…

STT Stream

Emage

Situation

18

Operations

11/1/12 19

Billions of data sources. Selecting and combining appropriate sources to detect situations. Interactions with different types of Users

Decision Makers Individuals

Want to use: Contact jain@ics.uci.edu

Front  End  GUI

NewDataSource

NewQuery

E-­‐mageStream

E-­‐mage  Stream

E-­‐mage  Stream

Data  Cloud

Back  End  Controller

Stream  Query  Processor

Data  IngestorRegisteredData

Sources

RegisteredQueries

Raw  Spatial  Data  Stream

API  Calls

Raw  DataStorage

Personalized  Alert  Unit

AlertRequest

User  Info

11/1/12 20

11/1/12 21

Building Blocks: Operators 22

Δ Transform …Spatio-temporal

window

⊕ Aggregate +

γ Classification Classification method

@ Characterization Growth Rate = 125%

Property required

Pattern Matching ψ 72%

+Pattern

∏ Filter +Mask

Φ Learn Learning method

{Features}

{Situation}

f f

1) Data into right representation

2) Analyze data to derive features

3) Use features to evaluate situations

Supporting parameter(s) Data Output Operator Type

Personalized Alerts

Situation recognition and control

24

STT data

Tweet: ‘Urrgh… sinus’

Loc: NYC, Date: 3rd Jun, 2011

Theme: Allergy

Situation Detection

User-Feedback

‘Please visit Dr. Cureit at 4th St immediately’

Date: 3rd Jun, 2011

Aggregation,

1) Classification 2) Control action

Operations

Alert level = High

Allergy System: Beyond Twitter

EVENTSHOP: Recognizing situations from web streams

26

Thailand Flood Mitigation

27

In New York: https://twitter.com/researchrerere

28

Help today.

Connecting resources: Problems and Research Community

• Big Data is BIG in challenges and opportunities particularly for Multimedia research community.

• Situation recognition is the challenge for NOW. •  IF PUBLICATIONS motivate you, THEN this is a an

opportunity to grab. •  IF you want to make an IMPACT, THEN this is an

opportunity for you.

29

Thanks.

We Need Collaborators: EventShop.

30