TERQAS:Time and Event Recognition for

Question Answering Systems

TERQAS Group

Final ReviewARDA Workshop

NRRC/MITREJuly 22, 2002

TERQAS:2002 Workshop Schedule

time2002.org

1. January 30-31: Kick-off Meeting; Setting Agenda

2. March 11-15: Corpus Selection, Query Studies, TimeML

3. April 22-26: TimeML Specification, Corpus Work

4. May 8-15: Annotation Fest

5. June 10-20: Algorithm Specification, Annotation

6. July 15-22: Wrap-up and Evaluation

7. Aug-Sept: Prepare Final Report

Relevance to Question Answering Systems

• Is Gates currently CEO of Microsoft?

• Were there any meetings between the terrorist hijackers and Iraq before the WTC event?

• Did the Enron merger with Dynegy take place?

• How long did the hostage situation in Beirut last?

• When did the war between Iran and Iraq end?• When did John Sununu travel to a fundraiser for John Ashcroft?• How many Tutsis were killed by Hutus in Rwanda in 1994?• Who was Secretary of Defense during the Gulf War?• What was the largest U.S. military operation since Vietnam?• When did the astronauts return from the space station?

Workshop Goals

• TimeML: Define and Design a Metadata Standard for Markup of events, their temporal anchoring, and how they are related to each other in News articles.

• TIMEBANK: Given the specification of TimeML, create a gold standard corpus of 300 articles marked up for temporal expressions, events, and basic temporal relations.

Working Groups

• TimeML Definition and Specification • Algorithm Review and Development• Article Corpus Collection

Development• Query Corpus Development and

Classification• TIMEBANK Annotation• TimeML and Algorithm Evaluation

TERQAS Participants– James Pustejovsky, PI– Rob Gaizauskas– Graham Katz– Bob Ingria – José Castaño– Inderjeet Mani– Antonio Sanfilippo– Dragomir Radev– Patrick Hanks– Marc Verhagen– Beth Sundheim– Andrea Setzer

– Jerry Hobbs– Bran Boguraev– Andy Latto– John Frank– Lisa Ferro– Marcia Lazo– Roser Saurí– Anna Rumshisky– David Day– Luc Belanger– Harry Wu– Andrew See

Supported by

Presentation Outline• TimeML 1.0 Specification • T3PO Algorithm Development• Tool Development Effort• TIMEBANK Annotation Status• Query Corpus Development and

Classification• TIMEBANK Annotation• Future Projects

TimeML 1.0• Adopts the core of Setzer’s annotation framework

(Sheffield Temporal Annotation Guidelines, STAG)• Remains compliant (as much as possible) with

TIDES TIMEX2 annotation. • Introduces a TLINK tag: an object that links

events/times to events/times.• Introduces an ALINK tag: an object that associates

aspectual phases to events.• Introduces an SLINK tag: an object that

subordinates events within modality, negation, or another event.

• Enrich temporal relations: adds i-after, i-before, and aspectual relations.

• Introduces event identity.• Introduces Temporal functions for doing temporal

math without evaluation.• Introduces STATE as a possible event class.

How TimeML Differs from Previous Markups

• Extends TIMEX2 annotation;– Temporal Functions: three years ago– Anchors to events and other temporal expressions:

• Identifies signals determining interpretation of temporal expressions;

– Temporal Prepositions: for, during, on, at;– Temporal Connectives: before, after, while.

• Identifies event expressions; – tensed verbs; has left, was captured, will resign;– stative adjectives; sunken, stalled, on board;– event nominals; merger, Military Operation, Gulf War;

• Creates dependencies between events and times:– Anchoring; John left on Monday.– Orderings; The party happened after midnight.– Embedding; John said Mary left.

Annotation in an Extension of STAG

FAMILIES SUE OVER AREOFLOT CRASH DEATHSThe Russian airline Aeroflot has been<EVENT eid=1 relatedToTime=1 timeRelType=BEFORE tense=PRESENT

aspect=PERFECTIVE class=OCCURRENCE>hit</EVENT>with a writ for loss and damages,<EVENT eid=2 tense=NONE aspect=PERFECTIVE relatedToEvent=1

eventRelType=BEFORE class=OCCURRENCE>filed</EVENT>in Hong Kong by the families of seven passengers<EVENT eid=3 tense=NONE aspect=PERFECTIVE relatedToEvent=2

eventRelType=BEFORE class=OCCURRENCE relatedToEvent2=4 eventRel2Type=IS_INCLUDED signal2=1>

killed</EVENT><SIGNAL sid=1>In </SIGNAL>an air<EVENT eid=4 class=OCCURRENCE>crash</EVENT>.

STAG Annotation, cont.All 75 peopleon boardthe Aeroflot Airbus<EVENT eid=5 tense=PAST aspect=PERFECTIVE relatedToEvent=6

eventRelType=IAFTER signal=2>died</EVENT><SIGNAL sid=2>when </SIGNAL>it<EVENT eid=6 tense=PAST aspect=PERFECTIVE relatedToTime=2

timeRelType=IS_INCLUDED relatedToEvent=4 eventRelType=ID>ploughed</EVENT>into a Siberian mountain<SIGNAL sid=3>in</SIGNAL><TIMEX tid=2 type=DATE calDate=031994>March 1994</TIMEX>.

Drawbacks of Event-Internal Relations in STAG

• Triple attribute structure in EVENT: [([signalID] relatedToEvent eventRelType) |

([signalID] relatedToTime timeRelType)]

• Same attribute structure appears in TIMEX:

[(eid signalID relType)]

• These three attributes are logically linked, allowing eventRelType, eventRelType,and eventRelType to be collapsed into single attribute.

EVENT

attributes ::= eid class tense aspect eid ::= ID{eid ::= EventIDEventID ::= e<integer>}class ::= 'OCCURRENCE' | 'PERCEPTION' | 'REPORTING' |

'ASPECTUAL' | 'STATE' | 'I_STATE' | 'I_ACTION'

| 'MODAL'tense ::= 'PAST' | 'PRESENT' | 'FUTURE' | 'NONE'aspect ::= 'PROGRESSIVE' | 'PERFECTIVE' | 'PERFECTIVE_PROGRESSIVE' | 'NONE'

TimeML Event Classes• Occurrence:

– die, crash, build, merge, sell, take advantage of, ..

• State:– Be on board, kidnapped, recovering, love, ..

• Reporting:– Say, report, announce,

• I-Action:– Attempt, try,promise, offer

• I-State:– Believe, intend, want, …

• Aspectual:– begin, start, finish, stop, continue.

• Perception:– See, hear, watch, feel.

The young industry's rapid growth also is attracting regulators eager to police its many facets.

The young industry's rapid<EVENT eid="e1" class="OCCURRENCE"> growth </EVENT>also is <EVENT eid="e2" class="OCCURRENCE"> attracting </EVENT>regulators <EVENT eid="e4" class="I_STATE">eager </EVENT>to<EVENT eid="e5" class="OCCURRENCE"> police </EVENT>its many facets.

Israel will ask the United States to delay a military strike against Iraq until the Jewish state is fully prepared for a possible Iraqi attack.

Israel will <EVENT eid="e1" class="I_ACTION"> ask</EVENT>the United States to <EVENT eid="e2" class="I_ACTION"> delay</EVENT> a military <EVENT eid="e3" class="OCCURRENCE">strike </EVENT> against Iraq until the Jewish state is fully<EVENT eid="e4" class="I_STATE">prepared</EVENT>for a possible Iraqi<EVENT eid="e5" class="OCCURRENCE">attack</EVENT>

Attribute Function Example

VAL Contains a normalized form of the date/time.

VAL=“1964-10-16”

MOD Captures temporal modifiers.

MOD=“APPROX”

SET Identifies expressions denoting sets of times.

SET=“YES”

PERIODICITY Captures the period between regularly recurring times.

PERIODICITY=“P1M”

GRANULARITY Captures the unit of time denoted by each set member in a set of times.

GRANULARITY=“G3D”

ANCHOR_VAL Contains a normalized form of an anchoring date/time.

ANCHOR_VAL=“1964-10-16”

ANCHOR_DIR Captures the relative direction between VAL and ANCHOR_VAL.

ANCHOR_DIR=“BEFORE”

NON_SPECIFIC Identifies non-specific expressions.

NON_SPECIFIC=“YES”

COMMENT Contains any comments the annotator wants to add.

COMMENT=“context garbled”

TIMEX2 Tag Attributes

Temporal Functions

Temporal expressions where the calendar date is not referred to directly, but via an expression that acts as a temporal function over a TIMEX3 expression.

Examples:1. last week2. last Thursday3. the week before last4. next week

Pre-theoretic Treatment:DCT=DocCreationTime

• last week = (predecessor (week DCT)) That is, we start with a temporal anchor, in this case, the DCT, coerce it to a week, than find the week preceding it.

• last Thursday = (thursday (predecessor (week DCT)) Similar to the preceding expression, except that we pick out the day named 'thursday' in the predecessor week.

• the week before last = (predecessor (predecessor (week DCT))) Also similar to the first expression, except that we go back two weeks.

• next week = (successor (week DCT)) The dual of the first expression: we start with the same coercion, but go forward instead of back.

TIMEX2 Annotation

Sen. Alton Waldon, who served briefly in Congress <TIMEX2 VAL="199” MOD="BEFORE">more than a decade ago</TIMEX2>, is <TIMEX2 VAL="PRESENT_REF">Now</TIMEX2> retired.

TimeML Treatment of Temporal Functions

Sen. Alton Waldon, who served briefly in Congress more than a decade ago, is now retired.

Sen. Alton Waldon, who<EVENT eid="e1" class="OCCURRENCE" tense="PAST" aspect="NONE">served</EVENT><MAKEINSTANCE eiid="ei1" eventID="e1"/>briefly in Congress<TIMEX3 tid="t1" type=”DATE" value=”199" mod=“BEFORE” temporalfunction=“TRUE”>more than a decade ago</TIMEX3>is<TIMEX3 tid="t2" type="DATE" value="PRESENT_REF">now</TIMEX3><EVENT eid="e2" class="STATE" tense="NONE" aspect="NONE">retired</EVENT><MAKEINSTANCE eiid="ei2" eventID="e2"/>.<TLINK eventInstanceID="ei1" relatedToTime="t1" relType=”IS_INCLUDED" /><TLINK eventInstanceID="ei2" relatedToTime="t2" relType="HOLDS"/><TLINK eventInstanceID="ei1" relatedToEvent=”ei2" relType=”BEFORE"/>

Temporal Functions: Alternative AnalysisSen. Alton Waldon, who

<EVENT eid="e1" class="OCCURRENCE" tense="PAST" aspect="NONE">served</EVENT><MAKEINSTANCE eiid="ei1" eventID="e1"/>briefly in Congress<TIMEX3 tid="t1" type="DURATION" value="P1E" mod="MORE_THAN">more than a decade</TIMEX3><SIGNAL sid="s1">ago</SIGNAL>,is<TIMEX3 tid="t2" type="DATE" value="PRESENT_REF">now</TIMEX3><EVENT eid="e2" class="STATE" tense="NONE" aspect="NONE">retired</EVENT><MAKEINSTANCE eiid="ei2" eventID="e2"/>.<TLINK eventInstanceID="ei1" relatedToTime="t2" signalID="s1" relType="BEFORE"

magnitude="t1"/><TLINK eventInstanceID="ei2" relatedToTime="t2" relType="HOLDS"/>

TLINKTLINK or Temporal Link represents the temporal relationship holding between events or

between an event and a time, and establishes a link between the involved entities, making explicit if they are:

1. Simultaneous (happening at the same time)2. Identical: (referring to the same event)

John drove to Boston. During his drive he ate a donut. 3. One before the other:

The police looked into the slayings of 14 women. In six of the cases suspects have already been arrested.

4. One after the other: 5. One immediately before the other:

All passengers died when the plane crashed into the mountain 6. One immediately after than the other: 7. One including the other:

John arrived in Boston last Thursday.8. One being included in the other: 9. One holding during the duration of the other: 10. One being the beginning of the other:

John was in the gym between 6:00 p.m. and 7:00 p.m.11. One being begun by the other: 12. One being the ending of the other:

John was in the gym between 6:00 p.m. and 7:00 p.m.. 13. One being ended by the other:

SLINKSLINK or Subordination Link is used for contexts introducing relations between two events, or an event and a signal, of the following sort: 1. Modal: Relation introduced mostly by modal verbs (should, could, would, etc.) and events that introduce a reference to a possible world --mainly I_STATEs:

John should have bought some wine. Mary wanted John to buy some wine.

2. Factive: Certain verbs introduce an entailment (or presupposition) of the argument's veracity. They include forget in the tensed complement, regret, manage:

John forgot that he was in Boston last year. Mary regrets that she didn't marry John. John managed to leave the party

3. Counterfactive: The event introduces a presupposition about the non-veracity of its argument: forget (to), unable to (in past tense), prevent, cancel, avoid, decline, etc.

John forgot to buy some wine. Mary was unable to marry John. John prevented the divorce.

4. Evidential: Evidential relations are introduced by REPORTING or PERCEPTION: John said he bought some wine. Mary saw John carrying only beer.

5. Negative evidential: Introduced by REPORTING (and PERCEPTION?) events conveying negative polarity:

John denied he bought only beer. 6. Negative: Introduced only by negative particles (not, nor, neither, etc.), which will be marked as SIGNALs, with respect to the events they are modifying:

John didn't forgot to buy some wine. John did not wanted to marry Mary.

ALINKALINK or Aspectual Link represent the relationship between an aspectual event, which will

be annotated as a SIGNAL (section 2.3), and its argument event. Examples of the possible aspectual relations we will encode are:

1. Initiation: John started to read

2. Culmination: John finished assembling the table.

3. Termination: John stopped talking.

4. Continuation: John kept talking.

Causation: 1(1) The rains caused the flooding.

The<EVENT eid="e1" class="OCCURRENCE" tense="NONE" aspect="NONE">rains</EVENT><MAKEINSTANCE eiid="ei1" eventID="e1"/><EVENT eid="e2" class="OCCURRENCE" tense="PAST" aspect="NONE">caused</EVENT><MAKEINSTANCE eiid="ei2" eventID="e2"/>the<EVENT eid="e3" class="OCCURRENCE" tense="NONE" aspect="NONE">flooding</EVENT><MAKEINSTANCE eiid="ei3" eventID="e3"/>

<TLINK eventInstanceID="ei1" relatedToEvent="ei2" relType="IDENTITY"/><TLINK eventInstanceID="ei2" relatedToEvent="ei3" relType="BEFORE"/>

Causation: 2(2') Kissinger secured the peace at great cost.

Kissinger<EVENT eid="e1" class="OCCURRENCE" tense="PAST" aspect="NONE">secured</EVENT><MAKEINSTANCE eiid="ei1" eventID="e1"/>the<EVENT eid="e2" class="OCCURRENCE" tense="NONE" aspect="NONE">peace</EVENT><MAKEINSTANCE eiid="ei2" eventID="e2"/>at great cost.

<TLINK eventInstanceID="ei1" relatedToEvent="ei2" relType="BEFORE"/>

Causation: 3(3) He kicked the ball, and it rose into the air.

He<EVENT eid="e1" class="OCCURRENCE" tense="PAST" aspect="NONE">kicked</EVENT><MAKEINSTANCE eiid="ei1" eventID="e1"/>the ball, and it<EVENT eid="e2" class="OCCURRENCE" tense="NONE" aspect="NONE">rose</EVENT><MAKEINSTANCE eiid="ei2" eventID="e2"/>into the air.

<TLINK eventInstanceID="ei1" relatedToEvent="ei2" relType="BEFORE"/>

TLINK: 1(4) John taught 20 minutes every Monday.

John<EVENT eid="e1" class="OCCURRENCE" tense="PAST" aspect="NONE">taught</EVENT><MAKEINSTANCE eiid="ei1" eventID="e1" signalID="s1" cardinality="EVERY"/><TIMEX3 tid="t1" type="DURATION" value="PT20M">20 minutes</TIMEX3><SIGNAL sid="s1">every</SIGNAL><TIMEX3 tid="t2" type="DATE" value="XXXX-WXX-1">Monday</TIMEX3><TLINK eventInstanceID="ei1" relatedToTime="t1" relType="HOLDS"/><TLINK eventInstanceID="ei1" relatedToTime="t2" relType="IS_INCLUDED"/>

TLINK: 2(6) John taught twice on Monday but only once on Tuesday John<EVENT eid="e1" class="OCCURRENCE" tense="PAST" aspect="NONE">taught</EVENT><MAKEINSTANCE eiid="ei1" eventID="e1" signalID="s1"/><MAKEINSTANCE eiid="ei2" eventID="e1" signalID="s1"/><MAKEINSTANCE eiid="ei3" eventID="e1" signalID="s2"/><SIGNAL sid="s1">twice</SIGNAL><SIGNAL sid="s3">on</SIGNAL><TIMEX3 tid="t1" type="DATE" temporalFunction="true" value="XXXX-WXX-1">Monday </TIMEX3>But only<SIGNAL sid="s2">once</SIGNAL><SIGNAL sid="s4">on</SIGNAL><TIMEX3 tid="t2" type="DATE" temporalFunction="true" value="XXXX-WXX-2">Tuesday </TIMEX3><TLINK eventInstanceID="ei1" signalID="s3" relatedToTime="t1" relType="IS_INCLUDED"/><TLINK eventInstanceID="ei2" signalID="s3" relatedToTime="t1" relType="IS_INCLUDED"/><TLINK eventInstanceID="ei3" signalID="s4" relatedToTime="t2" relType="IS_INCLUDED"/>

TLINK: 3(7) John taught 5 minutes after the explosion.

<EVENT eid="e1" class="OCCURRENCE" tense="PAST" aspect="NONE">taught</EVENT><MAKEINSTANCE eiid="ei1" eventID="e1"/><TIMEX3 tid="t1" type="DURATION" value="PT5M">5 minutes</TIMEX3><SIGNAL sid="s1">after</SIGNAL>the <EVENT eid="e2" class="OCCURRENCE" tense="NONE" aspect="NONE">explosion</EVENT><MAKEINSTANCE eiid="ei2" eventID="e2"/><TLINK eventInstanceID="ei1" signalID="s1" relatedToEvent="ei2" relType="AFTER" magnitude="t1"/>

TLINK: 4(8) John taught from 1992 through 1995.

John<EVENT eid="e1" class="OCCURRENCE" tense="PAST" aspect="NONE">taught</EVENT><MAKEINSTANCE eiid="ei1" eventID="e1"/><SIGNAL sid="s1">from</SIGNAL><TIMEX3 tid="t1" type="DATE" value="1992">1992</TIMEX3><SIGNAL sid="s2">through</SIGNAL><TIMEX3 tid="t2" type="DATE" value="1995">1995</TIMEX3><TLINK eventInstanceID="ei1" signalID="s1" relatedToTime="t1" relType="BEGUN_BY"/><TLINK eventInstanceID="ei1" signalID="s2" relatedToTime="t2" relType="ENDED_BY"/>

TLINK: 5(9) John taught from September to December last year. John<EVENT eid="e1" class="OCCURRENCE" tense="PAST" aspect="NONE">taught</EVENT><MAKEINSTANCE eiid="ei1" eventID="e1"/><SIGNAL sid="s1">from</SIGNAL><TIMEX3 tid="t1" type="DATE" temporalFunction="true" value="XXXX-09">September</TIMEX3><SIGNAL sid="s2">To </SIGNAL><TIMEX3 tid="t2" type="DATE" temporalFunction="true" value="XXXX-12">December</TIMEX3><TIMEX3 tid="t3" type="DATE" temporalFunction="true" value="XXXX" anchorTimeID="t4">last year</TIMEX3><TIMEX3 tid="t4" type="DATE" functionInDocument="CREATION_TIME" value="1996-03-27">03-27-96</TIMEX3><TLINK eventInstanceID="ei1" signalID="s1" relatedToTime="t1" relType="BEGUN_BY"/><TLINK eventInstanceID="ei1" signalID="s2" relatedToTime="t2" relType="ENDED_BY"/>

SLINK: 1(12) John taught on Monday but not on Tuesday

John<EVENT eid="e1" class="OCCURRENCE" tense="PAST" aspect="NONE">taught</EVENT><MAKEINSTANCE eiid="ei1" eventID="e1" signalID="s3"/><MAKEINSTANCE eiid="ei2" eventID="e1" signalID="s4"/><SIGNAL sid="s3">on</SIGNAL><TIMEX3 tid="t1" type="DATE" temporalFunction="true" value="XXXX-WXX-1">Monday</TIMEX3>but<SIGNAL sid="s1">not</SIGNAL><SIGNAL sid="s4">on</SIGNAL><TIMEX3 tid="t2" type="DATE" temporalFunction="true" value="XXXX-WXX-2">Tuesday</TIMEX3><TLINK eventInstanceID="ei1" relatedToTime="t1" signalID="s3" relType="IS_INCLUDED"/><TLINK eventInstanceID="ei2" relatedToTime="t2" signalID="s4" relType="IS_INCLUDED"/><SLINK subordinatedEventInstance="ei2" signalID="s1" relType="NEGATIVE"/>

SLINK: 2(13) If Graham leaves today, he will not hear Sabine. <SIGNAL sid="s1">if</SIGNAL>Graham<EVENT eid="e1" class="OCCURRENCE" tense="PRESENT" aspect="NONE">leaves</EVENT><MAKEINSTANCE eiid="ei1" eventID="e1"/><SLINK subordinatedEvent="e1" signalID="s1" relType="MODAL"/><TIMEX3 tid="t1" type="DATE" temporalFunction="true" value="XXXX-XX-XX">today</TIMEX3>he <EVENT eid="e3" class="MODAL" tense="NONE" aspect="NONE">will</EVENT><MAKEINSTANCE eiid="ei3" eventID="e3"/><SIGNAL sid="s2">not</SIGNAL><EVENT eid="e2" class="OCCURRENCE" tense="FUTURE" aspect="NONE">hear</EVENT><MAKEINSTANCE eiid="ei2" eventID="e2"/>Sabine. <SLINK eventInstanceID="ei3" subordinatedEvent="e2" relType="MODAL"/><TLINK eventInstanceID="ei1" relatedToEvent="ei2" relType="BEFORE"/><SLINK subordinatedEvent="e2" signalID="s1" relType="NEGATIVE"/>

SLINK: 3(14) Bill denied that John taught on Monday.

Bill<EVENT eid="e1" class="OCCURRENCE" tense="PAST" aspect="NONE">denied</EVENT><MAKEINSTANCE eiid="ei1" eventID="e1"/>that<SLINK eventInstanceID="ei1" subordinatedEvent="e2" relType="NEG_EVIDENTIAL"/>John<EVENT eid="e2" class="OCCURRENCE" tense="PAST" aspect="NONE">taught</EVENT><MAKEINSTANCE eiid="ei2" eventID="e2"/><SIGNAL sid="s1">on</SIGNAL><TIMEX3 tid="t1" type="DATE" temporalFunction="true" value="XXXX-WXX-1">Monday</TIMEX3><TLINK eventInstanceID="ei2" relatedToTime="t1" relType="IS_INCLUDED"/>

SLINK: 4(15) Bill wants to teach on Monday.

Bill<EVENT eid="e1" class="I_STATE" tense="PRESENT" aspect="NONE">wants</EVENT><MAKEINSTANCE eiid="ei1" eventID="e1"/><SLINK eventInstanceID="ei1" signalID="s1" subordinatedEvent="e2" relType="MODAL"/><SIGNAL sid="s1">to</SIGNAL><EVENT eid="e2" class="OCCURRENCE" tense="NONE" aspect="NONE">teach</EVENT><MAKEINSTANCE eiid="ei2" eventID="e2"/><SIGNAL sid="s2">on</SIGNAL><TIMEX3 tid="t1" type="DATE" temporalFunction="true" value="XXXX-WXX-1">Monday</TIMEX3><TLINK eventInstanceID="ei2" relatedToTime="t1" relType="IS_INCLUDED"/>

ALINK: 1(17) The boat began to sink.

The boat<EVENT eid="e1" class="ASPECTUAL" tense="PAST" aspect="NONE">began</EVENT><MAKEINSTANCE eiid="ei1" eventID="e1"/><SIGNAL sid="s1">to</SIGNAL><EVENT eid="e2" class="OCCURRENCE" tense="NONE" aspect= "NONE">sink</EVENT><ALINK eventInstanceID="ei1" signalID="s1" relatedToEvent="e2" relType="INITIATES"/>

ALINK: 2(18) The search party stopped looking for the survivors.

The search party<EVENT eid="e1" class="ASPECTUAL" tense="PAST" aspect="NONE">stopped</EVENT><MAKEINSTANCE eiid="ei1" eventID="e1"/><EVENT eid="e2" class="OCCURRENCE" tense="NONE" aspect="PROGRESSIVE">looking</EVENT><ALINK eventInstanceID="ei1" relatedToEvent="e2" relType="TERMINATES"/>for the survivors

• DTD created– TimeML.dtd

• Schema created– TimeML.xsd

time2002.org

Confidence Measures

attributes ::= tagType tagID [attributeName confidenceValue

tagType ::= CDATA

tagID ::= IDREF

attributeName ::= CDATA

confidenceValue ::= CDATA

{confidenceValue ::= 0 < x < 1}

Use of Confidence Measure

The TWA flight<EVENT eid="e1" class="OCCURRENCE" tense="PAST" aspect="NONE">crashlanded</EVENT><MAKEINSTANCE eiid="ei1" eventID="e1"/><TLINK eventInstanceID="ei1" signalID="s1" relatedToTime="t2" relType="BEFORE"

magnitude="t1"/>on Easter Island<TIMEX3 tid="t1" type="DURATION" value="P2W">two weeks</TIMEX3><SIGNAL sid="s1">ago</SIGNAL>.

...

<TIMEX3 tid="t2" type="DATE" functionInDocument="CREATION_TIME" value="1999-12-20">12-20-1999</TIMEX3>

Domains and Data Sets

• Document Collection (300):– ACE– DUC– PropBank (WSJ)

• Query Corpus Collection:– Excite query logs– MITRE Corpus– TREC8/9/10– Queries from TIMEBANK

Corpus Analytics

• Concordanced and indexed all training data– DUC subset– ACE subset– WSJ subset

• Concordancing and indexing reference data– BNC– Brown Corpus– WSJ Corpus

Graphical Annotation Tools

• TimeML-Alembic:– Extensions to MITRE’s Alembic

Workbench• Semi-Graphical Annotation Tool

– Create links by ordering events and TIMEX3s

Text Segmented Closure• System-prompted queries (a la Setzer):

– Completes temporal ordering markup in a text

• Performed on document segments:– Decreases the number of queries required

to provide closure

• Enrichments to Closure:– Persistence of states – Negative events

Goals of Text Segmented Closure

• Too many temporal relations in a large document.

• The number of temporal relations is quadratic to the number of objects that are being linked temporally. An annotator may be prompted hundreds of times, especially for large documents where a lot of the relations are "unknown".

• Some temporal relations are not interesting.• There does not seem to be a need to relate all

time expressions to each other.

Architecture for TSC1. Perform initial closure on all links added by the annotator.

2. Alert the user to potential identity chains. This is the only occasion where a user may be asked to specify a non-local relation.

3. Create a sliding window of three sentences. Initially, the window will consist of sentences one through three. The sliding window implements the local context. The size of the sliding window can be parameterize.

4. Prompt the user to specify a relation type for two time objects that are not yet linked within the local context. If no temporal relation exists, the annotator may specify "unknown".

5. After each added relation, recompute the closure using the new fact. Do this till all time objects within the local context are related.

6. If all objects in the local context are related, move the window up one sentence. For example, if the previous local context was made up of sentences 3-5, then the next local context for the closure algorithm is sentences 4-6. Start prompting the user for the new context.

Temporal Axioms• The axioms work with a normalized set of temporal

relations (no axiom is needed for the relType "unknown"):

PRE before, after, ibefore, iafter

INC includes, is_included

SIM simultaneous

IDT identity

• For PRE, normalization works as follows:

Link<x,y,before> => Link<x,y,PRE>

Link<x,y,ibefore> => Link<x,y,PRE>

Link<x,y,after> => Link<y,x,PRE>

Link<x,y,iafter> => Link<y,x,PRE>

Precedence PRE1: [ x PRE y & y PRE z => x PRE z ]

----x---- ----y---- ----z----

PRE2: [ x PRE y & y SIM z => x PRE z ] PRE3: [ x PRE y & y IDT z => x PRE z ]

----x---- ----y---- ----z----

PRE4: [ x PRE y & x SIM z => z PRE y ] PRE5: [ x PRE y & x IDT z => z PRE y ]

----x---- ----y---- ----z----

PRE6: [ x PRE y & x INC z => z PRE y ]

----x---- ----y---- --z--

Inclusion INC1: [ x INC y & y INC z => x INC z ]

------x------

----y----

--z--

INC2: [ x INC y & y SIM z => x INC z ]

INC3: [ x INC y & y IDT z => x INC z ]

----x----

--y--

--z--

INC4: [ x INC y & z SIM x => z INC y ]

INC5: [ x INC y & z IDT x => z INC y ]

----x----

--y--

----z----

Identity and Simultaneity

SIM1: [ x SIM y & y SIM z => x SIM z ]

SIM2: [ x SIM y & y IDT z => x SIM z ]

IDT1: [ x IDT y & y IDT z => x IDT z ]

----x----

----y----

----z----

TIMEX3 Parser Objects (T3PO)

• Extends TIDES TIMEX2 annotation:– Broader Coverage of temporal expressions– Larger lexicon of temporal triggers

• Delays Computation of Temporal Math:– Annotation with Temporal Functions– Import Hobbs’ Semantic Web Temporal System

• Distinct Cascaded Processes:– TIMEX3 and signal recognizer; – Event Predicate recognizer– LINK creation transducer.

Algorithm Overview Preprocessing: POS,

Shallow Parsing Three Finite State modules:

– Temporal Expressions – Events– Links

Discourse Information

Slide 1

Temporal Expressions Extension to Timex2

– Coverage– Absolute ISO Values– Signals

Functional Representation:– Anchor Resolution – Suite of Temporal Functions

Event Recognition In Verbal uses VG chunks:

– Encodes Tense and Aspect information

Nominal Events using:– Morphological information– POS ambiguity– Signals– Semantic Information

Link Recognition Event -Timex Links

Use of heuristics.Extra-sentential (Event-DCT Links)

Event-Event Links:– Intrasentential

• SLINKS (evidential)• SLINKS (infinitivals)

– Extrasentential

Discourse Information

• Reference Resolution• Anchor Resolution• Tense Sequence and Discourse

Structure.

Current Development Status

90% Temporal Functions Definition

65% TIMEX RECOGNITION

20% SIGNAL RECOGNITION

90 % EVENT RECOGNITION.

20% EVENT CLASS RECOGNITION

30% LINK RECOGNITION

Preliminary Tests Estimation

(6 documents with human annotated version)Precision

Recall TASK

80% 73% TIMEX RECOGNITION

70% 64% TIMEX VALUE RECOGNITION

31% 25% SIGNAL RECOGNITION

88% 94 % EVENT RECOGNITION.

54% 58% EVENT CLASS RECOGNITION

50% TIMEX-EVENT LINK RECOGNITION

Next Steps• Complete Timex• Complete Event Recognition• Develop Signal Recognition• Develop Event Class Recognition• Reference/Anchor Time

Recognition • Evaluation against TimeBank

Graphical Visualization Tool

• Filled Yellow: event with just one instance• Yellow border: event without instance or

linked to multiple instances• Red border: Instance of the event• Dotted link: relation with a signal• Green Link: MAKEINSTANCE• Blue diamond: Timex3 tags• Purple link: TLINK• Blue link: SLINK• Orange link: ALINK• Signal are between [ ]

Visualization Process

1. Parses TimeML file2. Entity Extraction

1. Nodes (Events, Instances and Timex3)2. Links (Temporal, Subordinate,

Aspectual)

3. Graph generation (graphviz dot format)

4. Graphviz processing

Utility of Visualization

• Debugging Annotated documents– Use it as a syntax and semantic

validator

• Represent the timeline and partial ordering of the events

Temporal Math Closure

• " Two Russians and a Frenchman left the Mir and endured a rough landing on the snow-covered plains of Central Asia on [Thursday]. ... The two Russians arrived on the Mir [last August] ... Solovyov ... celebrated his 50th birthday during his [six-month] space voyage."

TIMEBANK STATUS

• 50 Articles Fully Annotated to TimeML 1.0

• 66 Articles Annotated with TIMEX3, Signals, and Events

• 100 Articles Annotated with Signals and Events

• 3 Articles from 50 with inter-annotator scores (for 2 annotators)

Final Report Leftover Items

• Add Generic Event Expressions

• Periodicity and set notation on TIMEX3

• Enhance Temporal Function Expressiveness

Near Term Projects

• Create TIMEBANK Gold Standard with Closure

• Adding axioms for computing closure for new LINK types

• Computing event ordering within articles:

– Text coherence models

• Computing event ordering between articles

Open Research Issues• Persistence and Entailed Events:

– The terrorists kidnapped the journalist.– The President resigned.

• Event Normalization and Quantification:– Three deaths occurred. – Three people died.

• Generalizing the Treatment of Negation:– No survivors were found. – The plane did not crash.

Impact of TERQAS

• Better understanding of limits of current technology

• Preliminary capabilities for answering questions using that output

• Standard for Temporal and Event Markup (TimeML)

• Gold Standard Corpus for use by anyone in the community (TIMEBANK)

• Add a new dimension to the kinds of Q&A possible

Practical Challenges of TimeML Human

Annotation• Density: Highly dense

annotation, mainly due to links

• Speed: Extremely slow process– 1K/hour per annotator

(8.61K took Inderjeet ~ 9 hours)

• Utility: Research communities carrying out other tasks need to adopt it

197240

618

2115

TIMEX3 SIGNAL EVENT LINK

TimeML tag frequencies in56.6K bytes (raw) dataset

Addressing the Challenges

• Density– move away from textual annotation for links:

Graphical Annotation

• Speed– use radical mixed-initiative architecture,

involving massive pre-processing and interactive post-processing and machine learning

• Relevance– build links to other communities, by showing

value (e.g., Q&A, summarization, MT)

Efficient Annotation Through Multi-stage

Mixed Initiative Method

Pre-processing

Human Tagging

Post-processing

Raw Corpus

Annotated Corpus

Machine Learning

Annotated Corpus

Annotated Corpus

Annotated Corpus

Efficiency,reliability gains

Multi-Document TimeML Annotation for Summarization

Even this simple summary is only possible using TimeML

Multi-doc TimeML anchors single-doc events, and merges events across multiple docs (via TimeML graphs)

TimeML for MT

• Extend to multilingual annotation (re TIMEX2 results on Spanish, French, and Korean)

• Address translation of specialized TimeML constructs

Human Annotation Accuracy

TIMEX2, 193 news docs5 annotators

TimeML elements, 3 news docs2 annotators

tag pos act corr prec rec f f-TempEx

extent 6421 6369 5064 0.795101 0.788662 0.791882 0.762098val 5324 5296 4578 0.864426 0.85988 0.862153 0.829071granularity 289 312 165 0.535168 0.486111 0.51064 0.440212mod 494 451 363 0.813776 0.723356 0.768566 0.778878non_specific 143 92 28 0.304348 0.195804 0.250076 0periodicity 244 253 208 0.822134 0.852459 0.837297 0.811268set 300 316 232 0.734177 0.773333 0.753755 0.81465

run 5/29/02 and 7/20/02

TimeML Event Warehousing

• Collects events 24X7 from news feeds

• Builds TimeML graphs from each document

• Aggregates events in database using graph

• Can be used for trend analysis, etc.

Conclusions:

• TimeML aims to provide a robust markup framework for multiple domains and applications

• Compliant and interoperable with Semantic Web standards

• Goal to integrate into Document Models and Models of narrative structure

• Algorithms can be compared and measured against common TimeML-marked up corpora, starting with TIMEBANK.

www.time2002.org