News Flash

At Heathrow Airport today an individual, later discovered to be a public school teacher, was arrested trying to board a flight while in possession of a compass, a protractor, and a graphical calculator. Authorities believe he is a member of the notorious al-Gebra movement. He is being charged with carrying weapons of math instruction.

Relationship Pilot Report

John Prager & Eric Nyberg

Pilot in a Nutshell• 100 “Relationship” questions• Teams were free to use any manual intervention

desired, as long as it was described• Three teams participated – CMU, Cycorp, IBM• Each team plus two NIST assessors judged each

run• Answers mostly scored on a 3- or 5-pt scale

– Lowest (0) = “no use at all”– Highest = “perfect”, or at least “perfectly satisfactory”– Intermediate values for incompleteness and/or presence

of too much irrelevant material• In evaluation presented here, binary scoring used:

Any use vs. No use

Sample Question/Answers• Q44: What is the given name of the international terrorist Carlos the Jackal?

• Cycorp: France's highest court confirmed the life sentence being served by Ilich Ramirez Sanchez, 50, the international terrorist who calls himself Carlos the Jackal, for the murder of two French police agents and a Lebanese informant in Paris in 1975.Sanchez was given a life sentence last December for three murders, and is under investigation for further terrorist attacks.He was held in La Sante prison where Carlos the Jackal, nickname for Venezuelan-born Ilich Ramirez Sanchez, is held.Whatever happened to cool terrorists like Carlos the Jackal?

• CMU: Illich Ramirez Sanchez

• IBM: Ilich Ramirez Sanchez

• IBM-Justification: France 's highest court confirmed the life sentence being served by Ilich Ramirez Sanchez , 50 , the international terrorist who calls himself Carlos the Jackal , for the murder of two French police agents and a Lebanese informant in Paris in 1975.

CMU Approach

• Add new question types, analysis, extractors for relationship questions

• Focus development on relationships between people initially

• Mostly the existing JAVELIN architecture & Planner was used(no pilot-specific control structure)

Cycorp Approach

• Use original question terms as IR query• Break top retrieved documents into sentences• Generate Bayesian network with words as nodes from

Sentence x Word matrix• Select ancestor terms to augment query

– E.g. “What is the connection between actor and comedian Chris Rock and former Washington, D.C. mayor Marion Barry?”

– Augmentation terms = {drug, arrested}

• Iterate but where new network has sentences as nodes• Output sentences that are neighbours of augmented

query

Single Strategy

IBM Approach

• No modification– “What group is Jonathan Tasini part of?”– “What is BERTS?”

• Extending question analysis– “Are there any black NASCAR racing notables?”– Yes/no -> list of instances

• Extending pattern-based agent– “What is the relationship between X and Y?” -> locate syntactic contexts with X

and Y:• conjunction • subject-verb-object• objects of prepositions.

• New profile-based agent– Local Context Analysis on documents containing either X or Y– Form vector of terms, normalize, intersect, sort– “What do Valentina Tereshkova and Sally Ride have in common?” ->

• Space• First• Woman• Collins (the first woman to ever fly the space shuttle)

4-part Strategy

Responses

• Responses were allowed to be a “blob” of ASCII text of arbitrary length

• Responses could be accompanied by an optional, not-to-be-judged, justification– Only IBM returned justifications

• Cycorp responses were typically several independent sentences

• CMU & IBM responses were typically short phrases or single sentences

• As a bonus evaluation, IBM’s justifications were judged (by IBM) as if they were the responses– Provides an extra data point on score vs. response-length curve

Evaluation Results

Evaluation - #Questions with some value

0

10

20

30

40

50

60

70

80

90

100

Cycorp CMU IBM IBM-J

Runs

Sco

res

NIST A

NIST B

Cycorp

CMU

IBM

Evaluation Results

Evaluation + Response Lengths

0102030405060708090

100

Cycorp CMU IBM IBM-J

Runs

Sco

res

Med L

Mean L

NIST A

NIST B

Cycorp

CMU

IBM

Evaluation Results

Evaluation Normalized by Median Response Length

010

2030

405060

7080

90100

Cycorp CMU IBM IBM-J

Runs

Sco

res

NIST A

NIST B

Cycorp

CMU

IBM

Thoughts about the Evaluation Process

• Need some penalty function for excessively long answers

• On the other hand, cryptically short answers aren’t so good either

• Not clear how to balance penalties for incompleteness vs. irrelevance or redundancy– 3-(or 5-)valued scale not liked by all assessors

• Task-based evaluation as measure of utility?

Thoughts about Relationship Questions

• Several different kinds of questions, possibly require different answer formats

• Too much simultaneous novelty:– Systems hadn’t tackled relationship questions

before– Assessors hadn’t judged them before– Assessment scheme unknown at run time– Questions were needlessly hard

(factoid influence vs. fundamental relations)

Relationship Pilot

Questions?

Sample Questions/Answers• Q14: What has been the relationship between Osama bin Laden and Sudan?

• Run A: President Clinton tied the factory to Osama bin Laden, the Saudi dissident blamed for the bombings of U.S. embassies in Kenya and Tanzania on Aug. 7 that killed 257 people.A front-page editorial in Beirut's Al Kifah Al Arabi paper said that ``Lewinsky's dress is no longer the preoccupation of the world after Clinton has discovered Osama bin Laden's dishdasha (Arab robe).''The president was careful in his speech to stress that his objection was to bin Laden, rather than to his religion, Islam.

• Run B: u.s. president bill clinton announced thursday he ordered cruise missile attacks on sites in afghanistan and sudan linked to osama bin laden, a saudi millionaire whom u.s. officials call a major sponsor of terrorism.

• Run C: Bin_Laden left Sudan in 1996 + for Afghanistan where he fought Soviet troops in the 1980s +

• Run C-J: KHARTOUM, Sudan (AP) _ Despite an American attack last week that wrecked Sudan's largest pharmaceutical plant, troubled relations between the two countries that soured almost a decade ago will soon become better, Sudan's parliament speaker says. Washington has said the plant, which provided at least one-third of Sudan's domestic drug market, was manufacturing precursors for VX nerve gas. The attack culminated years of deteriorating relations with Sudan's government, which took power in a military coup in 1989 that overthrew a democratically elected government. It later added Sudan to its list of countries supporting terrorism, imposed trade sanctions and withdrew its diplomats from the capital. Sudan is Africa's largest country, and Turabi said its economy would soon prosper by virtue of its fertile land, abundant water and natural resources. ``All the Sudanese now actually see the United States, generally, as the incarnation of the devil, of evil,'' Turabi said in the interview at his home, decorated with framed Koranic verses and a picture of the Kaaba, Islam's holiest shrine. He called the attack ``most criminal, most savage'' and, like other Sudanese leaders, he accused Clinton of launching the strike to divert attention from scandals at home. He has been a fixture of Sudanese politics since 1965, holding posts under five regimes. '' One militant who took refuge in Sudan was Osama bin Laden, the exiled Saudi millionaire the United States accused of orchestrating the bombings of American embassies in Kenya and Tanzania. In Sudan, his company built a road from the capital and an airport. Bin_Laden left Sudan in 1996 for Afghanistan, where he fought Soviet troops in the 1980s. His departure from Sudan was by choice, according to Turabi, who defended him as a hero. ''

Turabi defended Sudan's right to have chemical weapons, but said it had neither the attention nor the means to acquire them.

Sample Questions/Answers• Q45: In what country and by whom did Operation Turquoise take place?

• Run A: When France sent troops in June 1994 in Operation Turquoise to Rwanda under the name of humanitarian rescue, it actually let the former Hutu army and Hutu militias, held responsible for the 1994 genocide in Rwanda, to flee to eastern Zaire.PARIS (AP) A committee of French lawmakers on Tuesday presented their conclusions on the 1994 genocide in Rwanda, calling for French government reform but mainly faulting the United Nations and United States for letting the bloodbath take place.

• Run B: France

• Run C: France Rwanda

• Run C-Justification: France has been accused of mounting Operation Turquoise to protect its old Hutu friends .

Paris intervened in southwestern Rwanda towards the end of the killings with ``Operation Turquoise '' with the objective of halting the bloodshed and bringing in humanitarian aid.

Sample Questions/Answers• Q71: What was the goal of the U.S. 1998 Iraq Liberation Act?

• Run A: Separately, U.S. President Bill Clinton signed the Iraq Liberation Act of 1998 that makes clear the United States supports elements within the Iraqi opposition who advocate a democratic future for Iraq.The term, jargon for the overthrow of Saddam Hussein, has been used since President Clinton announced last November that the containment of the Iraqi president was no longer sufficient.There is no question that President Saddam Hussein's military capacity has been ``degraded,'' to use the Clinton administration's word of choice, but it is not clear how degraded, or for how long.Q: Sandy, half-a-dozen senators have sent a letter to the president, saying it's time to get serious about implementing the Iraq Liberation Act, and urging that Saddam Hussein be indicted for war crimes.US: Goals Unfilled on IraqThis is something that has to be seen as a long-term goal.''The anti-Iraq coalition assembled to oppose Saddam Hussein has disintegrated.''But because Iraq remains under sanctions for ignoring terms of the cease-fire, Saddam is not expected to cooperate.

• Run B: NIL

• Run C: change

• Run C-J: Kerry, who sponsored the 1998 Iraq Liberation Act providing for arming Iraqi opposition groups, called for a change in U.S. policy.