I-Semantics 2012 Keynote

Dial E for

Events

Lora Aroyo

Monday, September 10, 12

Observationevents are important

events are omni-presentevents carry different points of viewin the world, e.g. news, science etc.

in our personal lives, e.g. social

networking

iSemantics2012 Lora Aroyo @laroyoFlickr: elkabong Monday, September 10, 12

Position

The human disagreement & vagueness of events are part of

the event semantics

iSemantics2012 Lora Aroyo @laroyoFlickr: elkabong Monday, September 10, 12

Objects vs. Events

events perdure = their parts exist at different time pointsobjects endure = they have all their parts at all points in time

objects are wholly present at any point in time, events unfold over time

iSemantics2012 Lora Aroyo @laroyoFlickr: vanilllaph Monday, September 10, 12

http://www.flickr.com/photos/48921967@N03/


Events are importantcreate context for objects, e.g. people, locations, organizations, etc.

Lora Aroyo @laroyo iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Events are importantcreate meaning for objects, e.g. artifacts, pictures, videos.

iSemantics2012 Lora Aroyo @laroyoMonday, September 10, 12

Events are importantlink concepts, objects, and stories.


Events in the Worldevents anchor the information we

consume daily

iSemantics2012 Lora Aroyo @laroyoFlickr: craftydogma Monday, September 10, 12



Events @ Google News


Events @ Google News


Julian Assange’s Extradition Row








Lance Armstrong’s Doping Fight


Lance Armstrong’s Doping Fight


The Arab Spring


The Arab Spring


Events @ Social Web


Events @ Social Web


Events @ Social Web


Events @ Social Web


Events @ Social Web


Events are VagueHumans have no clear notion of what events are


“event is a significant "happening" or gathering of people. I would define a "happening" as an event if the group of people gathered were united in one common goal.”

We Asked the Crowd What an EVENT isiSemantics2012 Lora Aroyo @laroyoFlickr: massimo vitali




We Asked the Crowd What an EVENT is

Event is a happening, which can be scheduled or unscheduled. An earthquake or fire happens (unscheduled). A wedding or birthday party (scheduled). It is an occasion that is unusual and tends to be memorable.


“An event would be any occurrence where physical action has taken place. It may be a single, momentary instance (I sneezed), or it may span a period of time (the festival ran for four hours). An event may also be made up of a number of smaller events, such as a day at school is an event, but each individual class is also an event itself. Basically an event must have a physical action over any delimited time span.”

We Asked the Crowd What an EVENT isiSemantics2012 Lora Aroyo @laroyo


“A planned public or social get together or occasion.”

“an event is an incident that's very important or monumental”

“An event is something occurring at a specific time and/or date to celebrate or recognize a particular occurrence.”

“a location where something like a function is held. you could tell if something is an event if there people gathering for a purpose.”

“Event can refer to many things such as: An observable occurrence, phenomenon or an extraordinary occurrence.”

We Asked the Crowd What an EVENT isiSemantics2012 Lora Aroyo @laroyo


What do Experts think an EVENT is?

“an event is the exemplification of a property by a substance at a given time” Jaegwon Kim, 1966

“events are changes that physical objects undergo” Lawrence Lombard, 1981

“events are properties of spatiotemporal regions”, David Lewis, 1986

iSemantics2012 Lora Aroyo @laroyounder30ceo.comMonday, September 10, 12

Event-centric Projectshow events can be detected & extracted from

natural language text

how those extracted events are represented for use on the semantic web

how to identify the same events in different sources

how to capture different perspectives


Activists

prominent on the new web through different channels

by nature multi-perspective, biased & emotional


Activists

prominent on the new web through different channels

by nature multi-perspective, biased & emotional


Mapping Online Networks of Activism

“All protest events Greenpeace participated in.”

blogs, news, activists websites


Mapping Online Networks of Activism

“All protest events Greenpeace participated in.”

• build visualizations

• create appropriate analytics

• answer questions of end users & social scientists

blogs, news, activists websites


• What happened before/after?

• Who does what, when, and where?

• All bomb attacks in the 1950s

• In what events did Indonesia participate?

• ‘Grand narratives’

• objects (digitized artworks and artifacts)

• events (concrete particulars)

• entities (actors, locations, periods)

• narratives (organization of events)

Extracting Historical Events



• generate meaningful event sequences• capture the different perspectives• serve both end users & history researchers


Timelines from Text

4 right, 2 wrong, 3 missing eventstwo have no explicit times & are in the

wrong orderOne involved al-Qaeda but took place in

Jordan on the Syrian border

does a fuzzy task require fuzzier metrics?

“al-Qaeda activities in Syria”


Timelines from Text

4 right, 2 wrong, 3 missing eventstwo have no explicit times & are in the

wrong orderOne involved al-Qaeda but took place in

Jordan on the Syrian border

does a fuzzy task require fuzzier metrics?

“al-Qaeda activities in Syria”


Machine Readingbuild event timelines from text in 2 example domains

• NFL: news articles on football;

• Ontology: 4 classes, 20 relations

• Intel: news articles on terrorist events;

• Ontology: 20 Classes, 50 relations


Why is event semantics hard?


According to NLP traditionGather

your source material

1Extract

events and properties

2

Analyze

find links between events

3

Visualize

statistics, timelines, etc.

4


but for events we stumble Gather


1Extract


2

Analyze


3

Visualize


4


but for events we stumble Gather


1Extract


2

Analyze


3

Visualize


4

experts typically:

define a problemannotate ground truth

train evaluate


Closed World Dictatorship

1. domain experts define the meaning2. using limited vocabulary

3. aim for agreement

to fix the problem of high disagreement for eventsexperts enforce more tyranny - stricter rules

comparatively little annotated data for training & evaluation of eventdetection systems


But the World is Open

1. events have multiple dimensions2. each dimension has levels of granularity

3. people have different views on both

all this leads to very complex semantics


and our goal is ...

1. not to enforce agreement2. to capture different view points

3. to teach machines to reason in the disagreement space

iSemantics2012 Lora Aroyo @laroyoFlickr: elkabongMonday, September 10, 12

PositionArtificially restricting humans does not help machines to learn.

Machines will learn from diversity


• Museum, libraries, archives & researchers have been dominating the views.

• Controlled vocabularies & annotation schemes were leading.

• Professionals enforced agreement among themselves.

• End-users needs & tasks are not considered.

Professional Dictatorship of the Closed World


there is a tiny overlap between end-user terminology & professional annotations

the latter are typically coarse-grained & refer to entire object / topic

iSemantics2012 Lora Aroyo @laroyoFlickr: ganzelkaMonday, September 10, 12



only 1,900 tags (32,200 in total) match in vocabularies 257 in people (83 validated) 1,661 in geo (666 validated)

9,796 validated, but no match in professional vocabulary8% professional vocab

23 % lexical vocab 63% meaningful Google matches


Amateur democracy of the Open World

• Once the Web opened the world of information, professional dictatorship clashed with end-users democracy.

• What professionals consider interesting, relevant or important does not match what users think of it.

• Amateurs cannot find what they were searching for.iSemantics2012 Lora Aroyo @laroyo(c) banksy




people are interested in different annotation categories than the professionals

iSemantics2012 Lora Aroyo @laroyoFlickr: ganzelkaMonday, September 10, 12



Video aspects that are described by those tags:

non-visual (0)perceptual (11), e.g. color

conceptual (1,332)

Tag sample: 1,343 verified tags of 5 random video fragments

195 tags (adverbs & adjectives) couldn’t be classified

Abstract General Specific Total

Who 10 1665

17712

31%

What 73 563 12 57%

Where 0 6831

8 7%

When 4 31 6 5%

Total 7% 74% 9%

Object tags (1,313)Scene tags (30)


Harnessing the Crowd

• we include the user’s opinion as first class citizens.

• this brings the need to combine all these (different) opinions into a system of opinions that makes sense.

• new solutions are needed, e.g. crowdsourcing of perspectives on events that exploit disagreement

iSemantics2012 Lora Aroyo @laroyoFlickr: AmyJanelleMonday, September 10, 12



What do People Disagree on?

are sub-events always mere parts?are “mentions” meaningful for events?

are events coreferential across documents? (e.g. perspectives, observations)


the bombing targeted a housing development in Baghdad, killing 3 and injuring 13

indistinguishable by people, confusable: is bombing part of killing, or killing part of bombing?

What about targeting?

“merelogically extensional” (i.e arbitrary): container bursting into fragments as a result of explosion

some events don’t exist: an action by military forces prevented the bombing.


Disagreement Framework• ontology: disagreements on the basic status of events

themselves as referents of linguistic utterances, e.g. are people events or do events exist at all.

• granularity: disagreements that result from issues of granularity, e.g. the location being a country, region, or city, the time being a day, week, month, etc.

• interpretation: disagreements that result from (non-granular) ambiguity, differences in perspective, or error in interpreting an expression, e.g. classifying a person as a terrorist/hero, ”October Revolution” took place in September.


Disagreement Framework• ontology: disagreements on the basic status of events

themselves as referents of linguistic utterances, e.g. are people events or do events exist at all.

• granularity: disagreements that result from issues of granularity, e.g. the location being a country, region, or city, the time being a day, week, month, etc.

• interpretation: disagreements that result from (non-granular) ambiguity, differences in perspective, or error in interpreting an expression, e.g. classifying a person as a terrorist/hero, ”October Revolution” took place in September.


Granularity Disagreement

spatial, temporal, participants

compositional, classificationaliSemantics2012 Lora Aroyo @laroyo




Event Participants Disagreement

Prime minister Benjamin

Netanyahu

Benjamin Netanyahu

Israeli Prime minister

Cabinet

Benjamin Netanyahu’s

Cabinet

Israeli Cabinet

his Cabinet

Israeli Government

{TOLD}

50%

35%

15%

10%

15%

5%

45%

15%


Temporal Disagreement

Spring 1998

March 1, 1998

March 1998

SundayPrime minister

Benjamin Netanyahu

Benjamin Netanyahu

Israeli Prime minister

{TOLD}

50%

35%

15%

25%

15%

50%

5%


Spatial Disagreement

Lebanon

IsraelSouthern Lebanon

Israel's Northern Frontier

Middle East

{WILLING TO WITHDRAW}

35%

45%

10%

30%

65%


Approach Principles1. tolerate, capture & exploit disagreement

2. understand the range of disagreements by creating a space of possibilities with frequencies & similarities

3. score the machine output based on where it falls in this space4. adaptable to new annotation tasks

iSemantics2012 Lora Aroyo @laroyoFlickr: auroilleMonday, September 10, 12



it seems to refer to an inference or communicated feeling more than specific event.

does not refer to an event

a group of people did something specific at a specific point in 6me.

refers to an event

the actors in ques6on (top Israeli officials) performed an ac6on during a specified 6me (Sunday).

refers to an event

it refers to what the israelis did on sunday, a specific 6me.

Top Israeli officials SENT strong new SIGNALS Sunday that Israel wants to withdraw from southern Lebanon, ...


Because it is describing a historical issue concerning the resolu6on of 1978

refers to an event

That 1978 resolu6on calls for Israel's uncondi6onal WITHDRAWAL from the self-‐declared security zone it occupies in south Lebanon, ...

it is not a par6cular movement that has or is going on but a request that the country of Israel remove their forces from the zone they occupy.


the sentence is speaking of a demand for a withdrawal that had not yet occurred.



The Dark Side of Crowdsourcing Disagreement

• disagreement is beautiful, except when it results from spamming• crowdsourcing has to account for people that want to get paid for

not doing any work• spammers generate disagreement for the wrong reasons• most spam detection requires gold standard


Spam or not?

• cut & paste from text• identical to other explanations • much shorter time than the average• low trust value of the worker• shorter than 5-6 words


Spam or not?Prime Minister Benjamin Netanyahu TOLD his Cabinet on Sunday that Israel was willing to ...

Because being told something doesn't seem like an event.


Because the WAR is being described as a costly event.

refers to an event

Top Israeli officials sent strong new signals Sunday that Israel wants to withdraw from southern Lebanon, where a costly WAR of aTri6on has been claiming soldiers' lives.

Because Israel WANTS TO WITHDRAW from Lebanon.

refers to an event

Top Israeli officials sent strong new signals Sunday that Israel WANTS TO WITHDRAW from southern Lebanon, ...

+ low worker trust

+ low worker trust

+ low worker trust

Because WANTS TO WITHDRAW is an ac6on.

refers to an event

Top Israeli officials sent strong new signals Sunday that Israel WANTS TO WITHDRAW ...

+ short time


Motivation-Verification Method

• 2-stage method:• disagreement collection + motivation• spam filtering = motivation judgement

• Additionally:• sample the motivation stage to manually

extract gold standard for stage 2


Phase I:A. Collect event annotations +

motivations

Extraction of Putative Events

input: putative events

Phase I:C. Filtering spam

event annotations

input: output of A

Phase II:A. Collect event types

+ motivations

input: list of events

Phase II:B. Filtering spam

event types

input: output of A

Manual selection of Gold Questions

input: output of A

Manual selection of

Gold Questions

input: output of A

Phase III:A. Collect event

modalities + motivations


Phase III:B. Filtering spam event modalities

input: output of A Manual

selection of Gold Questions

input: output of A

Phase IV:A. Collect event

role fillers + motivations

Phase IV:B. Filtering spam event role fillers

input: output of A Manual

selection of Gold Questions

input: output of A


• a new way of measuring ground truth

• a new set of semantic features for learning in event extraction


PositionArtificially restricting humans does not help machines to learn.

Machines will learn from diversity


Position

The human disagreement & vagueness of events are part of

the event semantics


finally ...

end the tyranny

disagree

ment is beautifu

l


Acknowledgements

Michiel Hildebrand

Lotte Belice Baltussen

Jacco van Ossenbruggen

Marteen Brinkerink

Johan OomenGuus Schreiber

Riste Gligorov

Marieke van Erp

Lourens van der Meij

Roxane Segers

Piek Vossen

Susan Legêne

Thomas Ploeger

Chiel van den Akker Frank de Bakker

Bibiana Armenta

Iina Hellsten

Geertje Jacobs

Geert-Jan Houben

Chris Welty


Questions?

@laroyohttp://lora-aroyo.org


http://lora-aroyo.org

http://lora-aroyo.org

Technology

I-Semantics 2012 Keynote