Micro reports and Situation Recognition at social machines workshop

© Ramesh JainSlide 1

Micro-Reports and

Situation Recognition

Ramesh Jain

Computer Science@UCI

(with Vivek Singh, Mingyan Gao, Siripen Pongpaichet, and

Mengfann Tan)

and

Krumbs Inc

(Asquith Baily, Neil Jain, Pinaki Sinha)

[email protected]


Society exists only as a mental concept;

in the real world there are only

individuals.

-- Oscar Wilde


Humans are Smart Sensors.

3

Humans are Smart Actuators

Humans are the goal as well as the

source of Technology.


The Magic Device: Mobile Phone

Middle 4 Billion

Top 1.5

Billion

Bottom 1.5 Billion

MOP: Improving

Information

Environment

TOP: Strong Information

Environment

BOP: Deprived of

Information

4


This century is different from the last.

Should we think differently???


In 20th century, we tolerated photos in our textual documents.

In 21st century, you create visual documents that tolerate text.


Major Disruption in Photos: From Memories to Information Sources.

Photos are the most compelling source of information.


Most Fundamental Problem:

Connecting People’s Needs to

Resources Effectively, Efficiently,

and Promptly in given Situations.

8


Major transformation in human

history are a chronicle of building

Social Machines for

How People’s need are

connected to Resources.

9


Hunter Gatherer:

You went to Food.

Now food comes to you.


Until 2000, you

went to make a

call.

Now call finds you.

© Ramesh JainSlide 12 12

Human Needs remain

the same.

Resources and

distribution methods

are invented.


Designing Social Machines:

• Needs

• Resources

• Connecting Needs to Resources


How can we identify Needs?

Need = f (person, context)

14


Designing Social Machines

IoT

Social Media

Human Sensors

Environmental

Sensors

Human/Actuators

Open Data


We Live in Dynamic World.

16


What is Cyber Space?

Who invented it?

Animals

Machines

Societies

Published first in 1942


• Desired state (Goal)

• System model

• Control Signal (Act)

• Current State

(observe)

18


Social Life Networks: Important

Factors

• The world we live in.

– Knowing current situation.

– Knowing where resources are.

• Where needs are.

– Knowing each individual’s situation.

– Knowing what they may need.

• Matching/recommendation Engine

• Action


Social Life Networks

Physical

World

And

Informa

tion

Systems

Environment and Resources

Information

Personal Situation and Needs

Information

Match

ing

Action Signals


EventShop : Geospatial Situation Detection

Situation

RecognitionData Stream

Ingestion and

aggregation

Database

Predictive

Analytics

Personal EventShop: Life Event Detection

Personal

Situation

Recognition

Database

Personal

Data

Ingestion

Objective Self

Recommendation

Engine

Need- Resource Matcher

Identify Resources and Needs

Resources Needs

Evolving Global Situation

Evolving Personal Situation

Actionable Information


Billions of data sources.

Environment for

Selecting, and

Combining

appropriate sources to detect situations.

Prediction for Pro-active actions

Interactions with different types of Users

Inspired by Photoshop




Flood level - Shelter

Flood LevelShelter

Twitter

Classify (Flood level - Shelter)



Microblogs: Participatory Sensing

• Microblogging is a broadcast medium that

uses typically smaller form of blogging.

• Twitter, Status updates, Instagram, …


Micro-Blog Mining Process

• Extracting Data From Data Providers

• Parsing, Integrating, and Storing the

data

• Extract Information of interest

• Earthquake Analysis; Flu; Trends


Problem with Micro-Blogs

• Noisy

• Subjective

• Poor context

• Great concept but has limitations.

• New technology to overcome these

limitations.


Waze


First Principle in Journalism

• Truthfulness, accuracy, objectivity,

impartiality, fairness and public accountability

• Journalists cannot always guarantee ‘truth’,

but getting the facts right is the cardinal

principle of journalism. We should always

strive for accuracy, give all the relevant facts

we have and ensure that they have been

checked. When we cannot corroborate

information we should say so.

• Seek Truth and Report it as Fully as Possible


Most Reports or Information are report

from an Event.

• ‘Kodak Moment’

• Each event has interesting and important

moments.

• How do we capture a moment?


What is a camera?


Is a Smartphone camera still a

camera?

Camera collects all metadata related to the Event.

• Exposure Time

• Aperture Diameter

• Flash

• Metering Mode

• ISO Ratings

• Focal Length

• Time

• Location

• Face

Smartphone camera captures events.


Micro-Reports: Requirements

• Objective (Subjective comments put

explicitly)

• Spontaneous

• Compelling

• Universal


Micro-Reports

• What (Information)

• Who (Information)

• When (Time)

• Where (Location)

• Why (Causality)

• How (Experiential)Photo

What

Where

When

Who

Why

Sound


Krumbs: Capture and Report

experience of a moment.

What: ObjectsWho: PeopleWhen: EventsWhere: Location

Why: Intent/Emotions





Krumbs SDK


Dashboard


Micro Reports and Analytics SDK

1. Reports convey direct desire, feedback, and observations.

2. Real time Aggregation and Analytics for understanding reports.

Creation of reports Customer Reports 43


EVENT

Spatial

Temporal

Informa-tional

Experien-

tial

Structural

Causal

{"micro_reports":[{

"where":{

"geo_location":{

"latitude":32.90233332316081,

"longitude":-

117.2441166718801},

"when":{

"start_time":"Jun 14, 2009

11:25:19 AM",

"end_time":"Jun 14, 2009 11:25:19

AM",

"time_zone":"America/Los_Angeles"}

,

"what":[{

"concept_name":"people",

"confidence":0.9836078882217407,

"visual_concept_provider":"CLARIFA

I"},

… {

"concept_name":"food",

"confidence":0.8526291847229004,

"visual_concept_provider":"CLARIFA

I"}],

"tag":”#niceday #summer",

"source":{"default_src":"https://….jpg"}},

"sub_event":[],

"why":[]},

…]}

MediaJSON for each micro-report


STT

Emage

EventShop

Emage Stream

Processing Engine

STTbase Others

Visual

Analytics

Query &

Feedback

STT Data

Ingestion

Trend and

Correlation Analysis

Micro-

Reports

Wrappe

r

Data Streams fromOther sources i.e., satellites, IoT, and stationary sensors.

STT1 Stream

STT2 Stream

STTn Stream

…

Rule Engines and

Alert Units

Emage Generator

Dashboard and

External Applications

Notification


Krumbs SDK at work

UCI students in Next

Generation Search ClassService Connect MyUCIVizNotes Places

46


• Routine operations

• Surge conditions

• Situational awareness

• Multi-objective

resource optimization

• Service Routing

• Changing uses

Management Expectations

• Clean

• Safe

• Walkable

• Reliable operations

• Special events

• Livability

• Healthy environment

• Fresh food

• Web sources and blogs

• Realtime information

(traffic, weather, transit,

…)

• Citizen participation

vs

Our Challenge


The relevant Sustainable DC goals to this project

include:

• Develop a Zero Waste plan for the city

• Ban polystyrene from the city

• Decrease all citywide waste streams

• Increase recycling bins in public realm

• Coordinating a city-wide education programs

The Sustainable DC Plan, developed with extensive citizen

input, established 2032 goals and actions designed to make

Washington DC the most sustainable city in North

America.

How to Improve Public Space

Waste Management

Our Goals


DowntownDC plans have so far focused on:

• Refining routes for personnel and equipment

• Gaining an understanding of the timing of services

• Cataloging major events and activities

• Mapping all public space elements in GIS

Micro-reports Event dataGIS data Route information

Our Approach


Real-time Trash Situations from

Sensors and Micro-Reports

Trash Bin Sensors Data

Micro Reports from Krumbs

Filter

Aggregate

Filter

Real-Time Trash Fill Level Situation

in EventShop


Prediction based on Events History

Events Data

Real-Time Trash Fill Level Situation

5 32

Now

Predicted Trash Fill Level

in 30 minutes at a given location

4

595

30 minutes

10

3040

70

90100

20

0

20

40

60

80

100

120

7:30 8:00 8:30 9:00 9:30 10:00 10:30

Projected Trash Fill Level at a given location based on Event History

0

35

50

90

0

20

40

60

80

100

7:30 8:00 8:30 9:00 9:30 10:00 10:30

Real-Time Fill Level Situations at a given location of an event


Current Status: In the hands of

employees this week.


Smart Communities: Community Relationship Management

Most communities have Web Presence .

Improving community experience: facilities, local services,

participation.


Each Photo is a

Micro-Report

Today Flickr

Data.

Tomorrow All

Photos.


B

A

D

C

E


Number of Flickr Photos (in London)


Flickr Concepts Combined


#photos in each concepts

peoplesport

running swimming

Can you solve the mystery?


Current Status

Krumbs SDK Ready: Being used by multiple

groups.

EventShop is open source.

Photos as a report is an exciting topic for research.

Looking for collaborators to join the

adventure in building Social Machines.


Dream!

5.5 Billion People Reporting and

Contributing to Solve Societal Problems.

Even in Remotest parts of a Developing

Country!


Thanks for your time and attention.

For questions: [email protected]

Data & Analytics

Micro reports and Situation Recognition at social machines workshop