Ubiquitous Human Computation KSE 801 Uichin Lee. Outline Papers today: – Crowd-Sourced Sensing and...

Ubiquitous Human Computation

KSE 801Uichin Lee

Outline

• Papers today: – Crowd-Sourced Sensing and Collaboration Using

Twitter, WoWMOM 2010 – Earthquake Shakes Twitter User:

Analyzing Tweets for Real-Time Event Detection, WWW 2010

• Understand the potential of ubiquitous human computation (+social networking)

Crowd-Sourced Sensing and Collaboration Using Twitter

Murat Demirbas, Murat Ali Bayir, Cuneyt Gurcan Akcora, Yavuz Selim Yilmaz

SUNY BuffaloWoWMOM 2010

Slides are based on http://www.cse.buffalo.edu/~demirbas/presentations/twitter.pdf

Cellphones!

• 3-4B cellphone users worldwide

• 1.13 billion phones sold in 2009 (36 per sec) vs 0.3 billion PCs

• 174M were smartphones– 15% (up from 12.8% in 2008)– Expected to exceed # feature

phones

Status quo in cellphones

• Each device connects to the Internet – to download/upload data and – to accomplish a task that does not require

collaboration and coordination

What is missing?

• An infrastructure to assist mobile users to perform collaboration and coordination ubiquitously

• Any user should be able to search & aggregate the data published by other users in a region

Our goal

• To provide a crowdsourced sensing and collaboration service using Twitter

• To enable aggregation and sharing of data; dynamically assign sensing tasks to other cellphone users

Why Twitter?

• Open publish-subscribe system: 105 million users, over 30 million users in US, 55 million tweets 600 million search queries everyday

• Each tweet has 140 char limit• Twitter provides an open source search API and a

REST API (that enables developers to access tweets, timelines, and user data)

• Different actors may integrate published data differently and can offer new services in unanticipated ways

Crowdsourcing architecture

Sensweet

• Employs the smartphone’s ability to work in the background without distracting a mobile user– Sense the surrounding environment and send the resulting

data to Twitter • To search and process sensor values on Twitter, we

need to agree on a standard for publishing these sensor readings– Bio-code: Uses Twitter bio sections & allows users to search

for the sensors they are looking for on-the-fly– TweetML: Uses pre-defined hashtags to improve

searchability

Askweet

• Accepts a question from Twitter – tries to answer the question using the data on

Twitter, potentially data published by Sensweets– if that is not possible, Askweet finds experts on

Twitter and forwards the question to these experts (not clear how this was done in the paper)

• Parallelizable, easy to “cloudify” for scalable service provisioning

Applications

1. Crowdsourced weather2. Noise map application3. Location-based queries (with Foursquare)

1. Crowdsourced weather

• Current weather, everybody on Twitter can be an expert

• Question to Askweet: “?Weather Loc:Buffalo,NY”• Forwarded question:“How is the weather there now?

reply 0 for sunny, 1 for cloudy, 2 for rainy, and 3 for snowy

http://ubicomp.cse.buffalo.edu/rainradar

Experimental results for NYC in different

time slices

2. Noise map application

• Implemented a Sensweet client for the Nokia N97 Smartphone series

• Sensweet client detects a noise level of the surrounding environment and forwards this data to Twitter in the TweetML format

• Sound sample is classified into: Low, Medium, High state– Each level is modeled using normal distribution– Input signal is compared with 3 distributions (Low,

Medium, and High)

Noise map application

Noise levels for a user

3. Location based queries

• Factual vs. non-factual queries– Factual: “hotels in Miami”– Non-factual: “Anyone knows any cheap, good

hotel, price ranges between 100 to 200 dollars in Miami?” • Traditional search engine performs poorly!

• Significant fraction of location-based queries (in Twitter) is non-factual– e.g., 63% of the queries were non-factual, while only 37%

of them were factual (manual classification of 269 queries)Crowdsourcing Location-based Queries, Bulut et al., Pervasive Collaboration and Social Networking, 2011

http://www.percom.org/proceedings/workshops/papers/p490-bulut.pdf

Location based queries

• Aardvark uses a social network of the asker to find suitable answerers for the query and forwards this query to the answerers, and returns any answer back to the asker.

• How about Twitter + Foursquare?– Use Foursquare to determine users

that frequent the queried locale and that have interests on the queried category (e.g., food, nightlife)

– Find a right set of people to ask!

[Questions to be asked]

[Users]

[Valid questions] [Valid answers]

[Questions detected] [Answer detected]

[Answer to be forwarded]Moderator

Asker

tweet starting with ?keyword checking (anyone,

suggestion, where)

label the category and quality of questions

forwards validated questions to appropriate people (using

Twitter bio or Foursquare info)

Constantly polling Twitter account to check answers

1

2

3

5

6

74

Experiment Setup

• Question dataset consists of 269 questions that the system collected over Twitter and validated as acceptable by the moderators.

• Manually categorize questions as factual and nonfactual: 63% - non-factual; 37% factual

• Some examples of questions for each type.

Foursquare Reply Rate vs. Random User Reply Rate Foursquare

Response Time

• 13 minutes median response time which is comparable with Aardvark

• 50% of the answers were received within the first 20 minutes.

Earthquake Shakes Twitter User:Analyzing Tweets for Real-Time Event Detection

Takehi Sakaki Makoto Okazaki Yutaka Matsuo@tksakaki @okazaki117 @ymatsuo

Tokyo UniversityWWW 2010 Conference

What’s happening?

• Twitter– is one of the most popular microblogging services– has received much attention recently

• Microblogging – is a form of blogging

• that allows users to send brief text updates

– is a form of micromedia• that allows users to send photographs or audio clips

• In this research, we focus on an important characteristic real-time nature

Real-time Nature of Microblogging

– Twitter users write tweets several times in a single day.– There is a large number of tweets, which results in many

reports related to events

– We can know how other users are doing in real-time– We can know what happens around other users in real-time.

social events parties baseball games presidential campaign

disastrous events storms fires traffic jams riots heavy rain-falls earthquakes

Our Goals

• propose an algorithm to detect a target event– do semantic analysis on Tweet

• to obtain tweets on the target event precisely

– regard Twitter user as a sensor• to detect the target event• to estimate location of the target

• produce a probabilistic spatio-temporal model for – event detection– location estimation

• propose Earthquake Reporting System using Japanese tweets

Twitter and Earthquakes in Japan

a map of earthquake occurrences world wide

a map of Twitter userworld wide

The intersection is regions with many earthquakes and large twitter users.

Twitter and Earthquakes in Japan

Other regions: Indonesia, Turkey, Iran, Italy, and Pacific coastal US cities

Event detection algorithms

• do semantic analysis on Tweet – to obtain tweets on the target event precisely

• regard Twitter user as a sensor– to detect the target event– to estimate location of the target

Semantic Analysis on Tweet• Search tweets including keywords related to a

target event– Example: In the case of earthquakes

• “shaking”, “earthquake”

• Classify tweets into a positive class or a negative class– Example:

• “Earthquake right now!!” --- positive• “Someone is shaking hands with my boss” --- negative

– Create a classifier

Semantic Analysis on Tweet

• Create classifier for tweets– use Support Vector Machine(SVM)

• Features (Example: I am in Japan, earthquake right now!)– A: Statistical features (7 words, the 5th word) the number of words in a tweet message and the position of the query

within a tweet

– B: Keyword features ( I, am, in, Japan, earthquake, right, now) the words in a tweet

– C: Word context features (Japan, right) the words before and after the query word

Tweet as Sensor Data

・・・ ・・・ ・・・tweets

・・・・・・

Probabilistic model

Classifier

observation by sensorsobservation by twitter users

target event target object

Probabilistic model

values

Event detection from twitter Object detection in ubiquitous environment

the correspondence between tweets processing andsensor data processing for event detection

Tweet as Sensor Data

some users posts“earthquake right now!!”

some earthquake sensors

responses positive value

We can apply methods for sensory data detection to tweets processing

・・・ ・・・ ・・・tweets

Probabilistic model

Classifier

observation by sensorsobservation by twitter users

target event target object

Probabilistic model

values

Event detection from twitter Object detection in ubiquitous environment

・・・・・・

search and classify them into

positive class

detect an earthquake

detect an earthquake

earthquake occurrence

Tweet as Sensor Data• We make two assumptions to apply methods for observation by

sensors

• Assumption 1: Each Twitter user is regarded as a sensor– a tweet → a sensor reading– a sensor detects a target event and makes a report probabilistically– Example:

• make a tweet about an earthquake occurrence• “earthquake sensor” return a positive value

• Assumption 2: Each tweet is associated with time and location info– time : posting timestamp– location : GPS data or location information in user’s profile

By processing time and location information, we can detect target events and find events’ locations

Probabilistic Model

• Why we need probabilistic models?– Sensor readings are noisy and sometimes sensors work

incorrectly– We cannot judge whether a target event occurred or not

from a single tweet– We have to calculate the probability of an event

occurrence from a series of data

• We propose probabilistic models for– event detection from time-series data– location estimation from a series of spatial information

Temporal Model

• We must calculate the probability of an event occurrence from a set of sensor readings

• We examine the actual time-series data to create a temporal model

Aug 9

0..

.A

ug 9

0..

.A

ug 9

1..

.A

ug 9

2..

.A

ug 1

0..

.A

ug 1

0..

.A

ug 1

0..

.A

ug 1

1..

.A

ug 1

1..

.A

ug 1

1..

.A

ug 1

1..

.A

ug 1

2..

.A

ug 1

2..

.A

ug 1

2..

.A

ug 1

3..

.A

ug 1

3..

.A

ug 1

3..

.A

ug 1

3..

.A

ug 1

4..

.A

ug 1

4..

.A

ug 1

4..

.A

ug 1

5..

.A

ug 1

5..

.A

ug 1

5..

.A

ug 1

6..

.A

ug 1

6..

.A

ug 1

6..

.A

ug 1

6..

.A

ug 1

7..

.A

ug 1

7..

.0

20

40

60

80

100

120

140

160

num

ber

of

tweets

Oct

9 .

..O

ct 9

...

Oct

10..

.O

ct 1

0..

.O

ct 1

0..

.O

ct 1

0..

.O

ct 1

1..

.O

ct 1

1..

.O

ct 1

1..

.O

ct 1

2..

.O

ct 1

2..

.O

ct 1

2..

.O

ct 1

3..

.O

ct 1

3..

.O

ct 1

3..

.O

ct 1

3..

.O

ct 1

4..

.O

ct 1

4..

.O

ct 1

4..

.O

ct 1

5..

.O

ct 1

5..

.O

ct 1

6..

.O

ct 1

6..

.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

num

ber

of

tweets

Temporal Model with Exponential Dist. Example: Earthquake and Typhoon

Spatial Model

• We must calculate the probability distribution of location of a target

• We apply Bayes filters to this problem which are often used in location estimation by sensors– Kalman Filters– Particle Filters

Bayesian Filters for Location Estimation

• Kalman Filters– are the most widely used variant of Bayes filters– approximate the probability distribution which is

virtually identical to a uni-modal Gaussian representation

– advantages: computational efficiency– disadvantages: limited to accurate sensors or

sensors with high update rates

Bayesian Filters for Location Estimation

• Particle Filters– represent the probability distribution by sets of samples, or

particles– advantages: able to represent arbitrary probability densities

• particle filters can converge to the true posterior even in non-Gaussian, nonlinear dynamic systems.

– disadvantages: difficult to apply to high-dimensional estimation problems

Information Diffusion Related to Real-time Events

• Proposed spatiotemporal models need to meet one condition that– sensors are assumed to be independent

• We check if information diffusions about target events happen because– if an information diffusion happened among users,

Twitter user sensors are not independent, they affect each other (correlation!)

Information Diffusion Related to Real-time Events

Nintendo DS Game an earthquake a typhoonInformation Flow Networks on Twitter

In the case of an earthquake and a typhoon, very little information diffusion takes place on Twitter, compared to Nintendo DS Game→ We assume that Twitter user sensors are independent about earthquakes and typhoons

Experiments and Evaluation

• We demonstrate performances of– tweet classification– event detection from time-series data → 　 show this result in “application”– location estimation from a series of spatial

information

Evaluation of Semantic Analysis

• Queries– Earthquake query: “shaking” and “earthquake”– Typhoon query:”typhoon”

• Examples to create classifier– 597 positive examples

Evaluation of Semantic Analysis

• We obtain highest F-value when we use Statistical features and all features.

• Keyword features and Word Context features don’t contribute much to the classification performance

• A user becomes surprised and might produce a very short tweet

• It’s apparent that the precision is not so high as the recall

Features Recall Precision F-Value

Statistical 87.50% 63.64% 73.69%Keywords 87.50% 38.89% 53.85%Context 50.00% 66.67% 57.14%All 87.50% 63.64% 73.69%

Evaluation of Spatial Estimation• Target events

– earthquakes• 25 earthquakes from August.2009 to October 2009

– typhoons• name: Melor

• Baseline methods– weighed average

• simply takes the average of latitudes and longitudes

– median• simply takes the median of latitudes and longitudes

• Metric: distance from an epicenter – The smaller the better!

Evaluation of Spatial Estimation

Tokyo

Osaka

actual earthquake center

Kyoto

estimation by median

estimation by particle filter

balloon: each tweets color : post time

Evaluation of Spatial Estimation

Typhoon

Discussions of Experiments

• Particle filter performs better than other methods• If the center of a target event is in an oceanic area,

it’s more difficult to locate it precisely from tweets• It becomes more difficult to make good estimation in

less populated areas

Results of Earthquake DetectionJMA intensity scale 2 or more 3 or more 4 or more

Num of earthquakes 78 25 3Detected 70(89.7%) 24(96.0%) 3(100.0%)

Promptly detected* 53(67.9%) 20(80.0%) 3(100.0%)

Promptly detected: detected in a minutesJMA intensity scale: the original scale of earthquakes by Japan Meteorology Agency

Period: Aug.2009 – Sep. 2009Tweets analyzed : 49,314 tweetsPositive tweets : 6291 tweets by 4218 users

We detected 96% of earthquakes that were stronger than scale 3 or more during the period.

Conclusion

• We investigated the real-time nature of Twitter for event detection

• Semantic analyses were applied to tweets classification • We consider each Twitter user as a sensor and set a problem to

detect an event based on sensory observations• Location estimation methods such as Kaman filters and particle

filters are used to estimate locations of events • We developed an earthquake reporting system, which is a

novel approach to notify people promptly of an earthquake event

• We plan to expand our system to detect events of various kinds such as rainbows, traffic jam etc.