Question Answering Tutorial

Question Answering Question Answering TutorialTutorial

Based on:Based on:John M. PragerJohn M. Prager

IBM T.J. Watson Research CenterIBM T.J. Watson Research [email protected]

Taken fromTaken from (with deletions and adaptations): (with deletions and adaptations): RANLP 2003 tutorialRANLP 2003 tutorialhttp://lml.bas.bg/ranlp2003/Tutorials link, Prager tutorialTutorials link, Prager tutorial

John M. PragerJohn M. PragerRANLP 2003 Tutorial on Question AnsweringRANLP 2003 Tutorial on Question Answering

Part I - Anatomy of QAPart I - Anatomy of QA

A Brief History of QAA Brief History of QA

TerminologyTerminology

The Essence of Text-based QAThe Essence of Text-based QA

Basic Structure of a QA SystemBasic Structure of a QA System

NE Recognition and Answer TypesNE Recognition and Answer Types

Answer ExtractionAnswer Extraction


A Brief History of QAA Brief History of QANLP front-ends to Expert SystemsNLP front-ends to Expert Systems

SHRDLU (Winograd, 1972)SHRDLU (Winograd, 1972)User manipulated, and asked questions about, User manipulated, and asked questions about, blocks worldblocks worldFirst real demo of combination of syntax, semantics, and reasoningFirst real demo of combination of syntax, semantics, and reasoning

** NLP front-ends to Databases** NLP front-ends to Databases LUNAR (Woods,1973)LUNAR (Woods,1973)

User asked questions about moon rocksUser asked questions about moon rocksUsed ATNs and procedural semanticsUsed ATNs and procedural semantics

LIFER/LADDER (Hendrix et al. 1977)LIFER/LADDER (Hendrix et al. 1977)User asked questions about U.S. Navy shipsUser asked questions about U.S. Navy shipsUsed semantic grammar; domain information built into grammarUsed semantic grammar; domain information built into grammar

** NLP + logic** NLP + logic CHAT-80 (Warren & Pereira, 1982)CHAT-80 (Warren & Pereira, 1982)

NLP query system in Prolog, about world geographyNLP query system in Prolog, about world geographyDefinite Clause GrammarsDefinite Clause Grammars

** “Modern Era of QA” – answers from free text** “Modern Era of QA” – answers from free text MURAX (Kupiec, 2001)MURAX (Kupiec, 2001)

NLP front-end to EncyclopaediaNLP front-end to Encyclopaedia

** IR + NLP** IR + NLP TREC-8 (1999) (Voorhees & Tice, 2000)TREC-8 (1999) (Voorhees & Tice, 2000)

Today – all of the aboveToday – all of the above


Some “factoid” questions from TREC8-9Some “factoid” questions from TREC8-99: How far is Yaroslav from Moscow?9: How far is Yaroslav from Moscow?15: When was London's Docklands Light Railway constructed?15: When was London's Docklands Light Railway constructed?22: When did the Jurassic Period end?22: When did the Jurassic Period end?29: What is the brightest star visible from Earth?29: What is the brightest star visible from Earth?* 30: What are the Valdez Principles?* 30: What are the Valdez Principles?73: Where is the Taj Mahal?73: Where is the Taj Mahal?197: What did Richard Feynman say upon hearing he would receive 197: What did Richard Feynman say upon hearing he would receive the Nobel Prize in Physics?the Nobel Prize in Physics?198: How did Socrates die?198: How did Socrates die?199: How tall is the Matterhorn?199: How tall is the Matterhorn?200: How tall is the replica of the Matterhorn at Disneyland?200: How tall is the replica of the Matterhorn at Disneyland?* 227: Where does dew come from?* 227: Where does dew come from?269: Who was Picasso?269: Who was Picasso?298: What is California's state tree?298: What is California's state tree?


TerminologyTerminology

Question TypeQuestion Type

Answer TypeAnswer Type

Question TopicQuestion Topic

Candidate PassageCandidate Passage

Candidate AnswerCandidate Answer

Authority File/ListAuthority File/List


Terminology – Question TypeTerminology – Question Type

Question TypeQuestion Type: an idiomatic categorization of questions : an idiomatic categorization of questions for purposes of distinguishing between different for purposes of distinguishing between different processing strategies and/or answer formatsprocessing strategies and/or answer formats

E.g. TREC2003E.g. TREC2003 FACTOID: “How far is it from Earth to Mars?”FACTOID: “How far is it from Earth to Mars?” LIST: “List the names of chewing gums”LIST: “List the names of chewing gums” DEFINITION: “Who is Vlad the Impaler?”DEFINITION: “Who is Vlad the Impaler?”

Other possibilities:Other possibilities: RELATIONSHIP: “What is the connection between Valentina RELATIONSHIP: “What is the connection between Valentina Tereshkova and Sally Ride?”Tereshkova and Sally Ride?” SUPERLATIVE: “What is the largest city on Earth?”SUPERLATIVE: “What is the largest city on Earth?” YES-NO: “Is Saddam Hussein alive?”YES-NO: “Is Saddam Hussein alive?” OPINION: “What do most Americans think of gun control?”OPINION: “What do most Americans think of gun control?” CAUSE&EFFECT: “Why did Iraq invade Kuwait?”CAUSE&EFFECT: “Why did Iraq invade Kuwait?” … …


Terminology – Answer TypeTerminology – Answer Type

Answer TypeAnswer Type: the class of object (or rhetorical type of : the class of object (or rhetorical type of sentence) sought by the question. E.g.sentence) sought by the question. E.g.

PERSON (from “Who …”)PERSON (from “Who …”) PLACE (from “Where …”)PLACE (from “Where …”) DATE (from “When …”)DATE (from “When …”) NUMBER (from “How many …”)NUMBER (from “How many …”) ……

but alsobut also EXPLANATION (from “Why …”)EXPLANATION (from “Why …”) METHOD (from “How …”)METHOD (from “How …”) ……

Answer types are usually tied intimately to the classes Answer types are usually tied intimately to the classes recognized by the system’s Named Entity Recognizer.recognized by the system’s Named Entity Recognizer.


Broader Answer TypesBroader Answer Types

E.g.E.g. ““In what In what statestate is the Grand Canyon?” is the Grand Canyon?” ““What is the What is the populationpopulation of Bulgaria?” of Bulgaria?” ““What What colourcolour is a pomegranate?” is a pomegranate?”


Terminology – Question TopicTerminology – Question Topic

Question TopicQuestion Topic: the object (person, place, : the object (person, place, …) or event that the question is about. …) or event that the question is about. The question might well be about a The question might well be about a property of the topic, which will be the property of the topic, which will be the question focus.question focus.

E.g. E.g. “What is the height of Mt. Everest?”“What is the height of Mt. Everest?” Mt. EverestMt. Everest is the is the topictopic

Topic has to be mentioned in answer Topic has to be mentioned in answer passagepassage


Terminology – Candidate PassageTerminology – Candidate Passage

Candidate PassageCandidate Passage: a text passage : a text passage (anything from a single sentence to a (anything from a single sentence to a whole document) retrieved by a search whole document) retrieved by a search engine in response to a question.engine in response to a question.

Candidate passage expected to contain Candidate passage expected to contain candidate answers.candidate answers.

Candidate passages will usually have Candidate passages will usually have associated scores, from the search associated scores, from the search engine.engine.


Terminology – Candidate AnswerTerminology – Candidate Answer

Candidate AnswerCandidate Answer: in the context of a question, a small : in the context of a question, a small quantity of text (anything from a single word to a quantity of text (anything from a single word to a sentence or bigger, but usually a noun phrase) that is of sentence or bigger, but usually a noun phrase) that is of the same type as the Answer Type.the same type as the Answer Type.

In some systems, the type match may be approximateIn some systems, the type match may be approximate

Candidate answers are found in candidate passagesCandidate answers are found in candidate passagesE.g.E.g.

5050 Queen Elizabeth IIQueen Elizabeth II September 8, 2003September 8, 2003 by baking a mixture of flour and waterby baking a mixture of flour and water


Terminology – Authority ListTerminology – Authority ListAuthority List (or File)Authority List (or File): a collection of instances of a class of interest, used to test a : a collection of instances of a class of interest, used to test a term for class membership. <Answer type>term for class membership. <Answer type>Instances should be derived from an authoritative source and be as close to complete Instances should be derived from an authoritative source and be as close to complete as possible.as possible.Ideally, class is small, easily enumerated and with members with a limited number of Ideally, class is small, easily enumerated and with members with a limited number of lexical forms.lexical forms.Good:Good:

Days of weekDays of week PlanetsPlanets ElementsElements

Good statistically, but difficult to get 100% recall:Good statistically, but difficult to get 100% recall: AnimalsAnimals PlantsPlants ColoursColours

ProblematicProblematic PeoplePeople OrganizationsOrganizations

ImpossibleImpossible All numeric quantitiesAll numeric quantities Explanations and other Explanations and other clausalclausal quantities quantities


Essence of Text-based QAEssence of Text-based QA

Need to Need to find a passagefind a passage that answers the that answers the question. Steps:question. Steps: Find a candidate passage (search)Find a candidate passage (search) Check that semantics of passage and Check that semantics of passage and

question matchquestion match Extract the answerExtract the answer


Basic Structure of a QA-SystemBasic Structure of a QA-System

See for example Abney et al., 2000; Clarke et al., 2001; See for example Abney et al., 2000; Clarke et al., 2001; Harabagiu et al.; Hovy et al., 2001; Prager et al. 2000Harabagiu et al.; Hovy et al., 2001; Prager et al. 2000

QuestionAnalysis

AnswerExtraction

SearchCorpus

orWeb

Question

Answer

Documents/ passages

Query

AnswerType



For a very small corpus, can consider every For a very small corpus, can consider every passage as a candidate, but this is not passage as a candidate, but this is not interestinginterestingNeed to perform a search to locate good Need to perform a search to locate good passages.passages.If search is too broad, have not achieved that If search is too broad, have not achieved that much, and are faced with lots of noisemuch, and are faced with lots of noiseIf search is too narrow, will miss good passagesIf search is too narrow, will miss good passages

SearchSearch

Two broad possibilities:Two broad possibilities: Optimize searchOptimize search Use iterationUse iteration



Need to test whether semantics of Need to test whether semantics of passage match semantics of questionpassage match semantics of questionApproaches:Approaches: Count question words present in passageCount question words present in passage Score based on proximityScore based on proximity Score based on syntactic relationshipsScore based on syntactic relationships Prove matchProve match

MatchMatch



Find candidate answers of same type as Find candidate answers of same type as the answer type sought in question.the answer type sought in question.

Has implications for size of type hierarchyHas implications for size of type hierarchy




Have three broad locations in the system where Have three broad locations in the system where expansionexpansion takes place, for purposes of matching takes place, for purposes of matching passagespassagesWhere is the right trade-off?Where is the right trade-off?Question Analysis. Question Analysis.

Expand individual terms to synonyms (hypernyms, hyponyms, Expand individual terms to synonyms (hypernyms, hyponyms, related terms)related terms)

Reformulate question (paraphrases)Reformulate question (paraphrases)

In Search EngineIn Search EngineAt indexing timeAt indexing time

Stemming/lemmatizationStemming/lemmatization

High-Level View of RecallHigh-Level View of Recall



Have three broad locations in the system where Have three broad locations in the system where narrowing/filtering/narrowing/filtering/matchingmatching takes place takes placeWhere is the right trade-off?Where is the right trade-off?

Question Analysis. Question Analysis. Include all question terms in query, vs. allow partial matchingInclude all question terms in query, vs. allow partial matching Use IDF-style weighting to indicate preferencesUse IDF-style weighting to indicate preferences

Search EngineSearch Engine Possibly store POS information for polysemous termsPossibly store POS information for polysemous terms

Answer ExtractionAnswer Extraction Reward (penalize) passages/answers that (don’t) pass matching testReward (penalize) passages/answers that (don’t) pass matching test

High-Level View of PrecisionHigh-Level View of Precision


Answer Types and ModifiersAnswer Types and Modifiers

Most likely there is no type for Most likely there is no type for “French Cities”“French Cities” Cf. WikipediaCf. Wikipedia

So will look for So will look for CITYCITY include include “French/France”“French/France” in bag of words, and hope for the best in bag of words, and hope for the best include include “French/France”“French/France” in bag of words, retrieve documents, in bag of words, retrieve documents,

and look for evidence (deep parsing, logic)and look for evidence (deep parsing, logic) If you have a list of French cities, could eitherIf you have a list of French cities, could either

Filter results by listFilter results by listUse Answer-Based QA (see later)Use Answer-Based QA (see later)

Domain Model: Use longitude/latitude information of cities and Domain Model: Use longitude/latitude information of cities and countries – practical for domain oriented systems (e.g. countries – practical for domain oriented systems (e.g. geographical)geographical)

Name 5 French Cities Name 5 French Cities


Answer Types and ModifiersAnswer Types and Modifiers

Most likely there is no type for Most likely there is no type for “female figure skater”“female figure skater”Most likely there is no type for Most likely there is no type for “figure skater”“figure skater”Look for Look for PERSONPERSON, with query terms , with query terms {figure, skater}{figure, skater}What to do about “What to do about “female”female”? Two approaches.? Two approaches.

1.1. Include Include “female”“female” in the bag-of-words. in the bag-of-words. • Relies on logic that if “femaleness” is an interesting property, it Relies on logic that if “femaleness” is an interesting property, it

might well be mentioned in answer passages. might well be mentioned in answer passages. • Does not apply to, say “singer”.Does not apply to, say “singer”.

2.2. Leave out Leave out “female”“female” but test candidate answers for gender. but test candidate answers for gender. • Needs either an authority file or a heuristic test Needs either an authority file or a heuristic test

• e.g. look for e.g. look for she,her, …she,her, …• Test may not be definitive.Test may not be definitive.

Name a female figure skaterName a female figure skater


Named Entity RecognitionNamed Entity Recognition

BBN’s IdentiFinder (Bikel et al. 1999)BBN’s IdentiFinder (Bikel et al. 1999) Hidden Markov ModelHidden Markov Model

Sheffield GATE (Sheffield GATE (http://www.gate.ac.uk/)) Development Environment for IE and other NLP activitiesDevelopment Environment for IE and other NLP activities

IBM’s Textract/Resporator (Byrd & Ravin, 1999; IBM’s Textract/Resporator (Byrd & Ravin, 1999; Wacholder et al. 1997; Prager et al. 2000)Wacholder et al. 1997; Prager et al. 2000)

FSMs and Authority FilesFSMs and Authority Files

+ others+ others

Inventory of semantic classes recognized by NER Inventory of semantic classes recognized by NER related closely to set of answer types system can handlerelated closely to set of answer types system can handle


Named Entity RecognitionNamed Entity Recognition



Also called Answer Selection/PinpointingAlso called Answer Selection/PinpointingGiven a question and candidate passages, the process Given a question and candidate passages, the process of selecting and ranking candidate answers.of selecting and ranking candidate answers.Usually, candidate answers are those terms in the Usually, candidate answers are those terms in the passages which have the same answer type as that passages which have the same answer type as that generated from the questiongenerated from the questionRanking the candidate answers depends on assessing Ranking the candidate answers depends on assessing how well the passage context relates to the questionhow well the passage context relates to the question3 Approaches:3 Approaches:

Heuristic featuresHeuristic features Shallow parse fragmentsShallow parse fragments Logical proofLogical proof


Answer Extraction using FeaturesAnswer Extraction using FeaturesHeuristic feature sets (Prager et al. 2003+). See also (Radev at al. Heuristic feature sets (Prager et al. 2003+). See also (Radev at al. 2000)2000)Calculate feature values for each CA, and then calculate linear Calculate feature values for each CA, and then calculate linear combination using weights learned from training data.combination using weights learned from training data.

Features are generic/non-lexicalized, question independent (vs. supervised IE)Features are generic/non-lexicalized, question independent (vs. supervised IE)

Ranking criteria:Ranking criteria: Good global context:Good global context:

the the global context global context of a candidate answer evaluates the relevance of the of a candidate answer evaluates the relevance of the passage from which the candidate answer is extracted to the question.passage from which the candidate answer is extracted to the question.

Good local contextGood local contextthe the local context local context of a candidate answer assesses the likelihood that the of a candidate answer assesses the likelihood that the answer fills in the gap in the question.answer fills in the gap in the question.

Right semantic typeRight semantic typethe the semantic type semantic type of a candidate answer should either be the same as or a of a candidate answer should either be the same as or a subtype of the answer type identified by the question analysis component.subtype of the answer type identified by the question analysis component.

RedundancyRedundancythe degree of the degree of redundancy redundancy for a candidate answer increases as more for a candidate answer increases as more instances of the answer occur in retrieved passages.instances of the answer occur in retrieved passages.


Answer Extraction using Answer Extraction using Features (cont.)Features (cont.)

Features for Global ContextFeatures for Global Context KeywordsInPassagKeywordsInPassagee: the ratio of keywords present in a : the ratio of keywords present in a

passage to the total number of keywords issued to the search passage to the total number of keywords issued to the search engine.engine.

NPMatchNPMatch: the number of words in noun phrases shared by both : the number of words in noun phrases shared by both the question and the passage.the question and the passage.

SEScorSEScoree: the ratio of the search engine score for a passage to : the ratio of the search engine score for a passage to the maximum achievable score.the maximum achievable score.

FirstPassagFirstPassagee: a Boolean value which is true for the highest : a Boolean value which is true for the highest ranked passage returned by the search engine, and false for all ranked passage returned by the search engine, and false for all other passages.other passages.

Features for Local ContextFeatures for Local Context AvgDistancAvgDistancee: the average distance between the candidate : the average distance between the candidate

answer and keywords that occurred in the passage.answer and keywords that occurred in the passage. NotInQuerNotInQueryy: the number of words in the candidate answers that : the number of words in the candidate answers that

are not query keywords.are not query keywords.


Answer Extraction using Answer Extraction using RelationshipsRelationships

Can be viewed as additional featuresCan be viewed as additional features

Computing Ranking ScoresComputing Ranking Scores – –

Linguistic knowledge to compute Linguistic knowledge to compute passagepassage & & candidate answer scorescandidate answer scores

Perform syntactic processing on question and candidate passagesPerform syntactic processing on question and candidate passages

Extract predicate-argument & modification relationships from parseExtract predicate-argument & modification relationships from parse Question: “Who wrote the Declaration of Independence?”Question: “Who wrote the Declaration of Independence?”

Relationships: [X, write], [write, Declaration of Independence]Relationships: [X, write], [write, Declaration of Independence]

Answer Text: “Jefferson wrote the Declaration of Independence.”Answer Text: “Jefferson wrote the Declaration of Independence.”

Relationships: [Jefferson, write], [write, Declaration of Independence]Relationships: [Jefferson, write], [write, Declaration of Independence]

Compute scores based on number of question relationship matchesCompute scores based on number of question relationship matches


Answer Extraction using Answer Extraction using Relationships (cont.)Relationships (cont.)

Example: Example: When did Amtrak begin operations?When did Amtrak begin operations?

Question relationshipsQuestion relationships [Amtrak, begin], [begin, operation], [X, begin][Amtrak, begin], [begin, operation], [X, begin]

Compute passage scores: passages and relationshipsCompute passage scores: passages and relationships In 1971, Amtrak began operations,…In 1971, Amtrak began operations,…

[Amtrak, begin], [begin, operation], [1971, begin]…[Amtrak, begin], [begin, operation], [1971, begin]…

““Today, things are looking better,” said Claytor, expressing optimism Today, things are looking better,” said Claytor, expressing optimism about getting the additional federal funds in future years that will allow about getting the additional federal funds in future years that will allow Amtrak to begin expanding its operations.Amtrak to begin expanding its operations.

[Amtrak, begin], [begin, expand], [expand, operation], [today, look]…[Amtrak, begin], [begin, expand], [expand, operation], [today, look]…

Airfone, which began operations in 1984, has installed air-to-ground Airfone, which began operations in 1984, has installed air-to-ground phones…. Airfone also operates Railfone, a public phone service on phones…. Airfone also operates Railfone, a public phone service on Amtrak trains.Amtrak trains.

[Airfone, begin], [begin, operation], [1984, operation], [Amtrak, train]…[Airfone, begin], [begin, operation], [1984, operation], [Amtrak, train]…


Answer Extraction Answer Extraction using Logicusing Logic

Logical ProofLogical Proof Convert question to a goalConvert question to a goal Convert passage to set of logical forms Convert passage to set of logical forms

representing individual assertionsrepresenting individual assertions Add predicates representing subsumption Add predicates representing subsumption

rules, real-world knowledgerules, real-world knowledge Prove the goalProve the goal

See section on LCC nextSee section on LCC next


LCCLCC

Moldovan & Rus, 2001Moldovan & Rus, 2001Uses Uses Logic ProverLogic Prover for answer justification for answer justification

Question logical formQuestion logical form Candidate answers in logical formCandidate answers in logical form XWN glossesXWN glosses Linguistic axiomsLinguistic axioms Lexical chainsLexical chains

Inference engine attempts to verify answer by negating Inference engine attempts to verify answer by negating question and proving a contradictionquestion and proving a contradictionIf proof fails, predicates in question are gradually relaxed If proof fails, predicates in question are gradually relaxed until proof succeeds or associated proof score is below a until proof succeeds or associated proof score is below a threshold.threshold.


LCC: Lexical ChainsLCC: Lexical ChainsQ:1518 Q:1518 What year did Marco Polo travel to Asia?What year did Marco Polo travel to Asia?

AnswerAnswer:: Marco polo divulged the truth after returning in 1292 from his Marco polo divulged the truth after returning in 1292 from his travels, which included several months on Sumatratravels, which included several months on Sumatra

Lexical Chains:Lexical Chains: (1) travel_to:v#1 -> GLOSS -> travel:v#1 -> RGLOSS -> travel:n#1(1) travel_to:v#1 -> GLOSS -> travel:v#1 -> RGLOSS -> travel:n#1

(2) travel_to#1 -> GLOSS -> travel:v#1 -> HYPONYM -> return:v#1 (2) travel_to#1 -> GLOSS -> travel:v#1 -> HYPONYM -> return:v#1

(3) Sumatra:n#1 -> ISPART -> Indonesia:n#1 -> ISPART -> (3) Sumatra:n#1 -> ISPART -> Indonesia:n#1 -> ISPART -> Southeast _Asia:n#1 -> ISPART -> Asia:n#1Southeast _Asia:n#1 -> ISPART -> Asia:n#1

Q:1570 Q:1570 What is the legal age to vote in Argentina?What is the legal age to vote in Argentina?AnswerAnswer:: Voting is mandatory for all Argentines aged over 18.Voting is mandatory for all Argentines aged over 18.Lexical ChainsLexical Chains::

(1) legal:a#1 -> GLOSS -> rule:n#1 -> RGLOSS -> (1) legal:a#1 -> GLOSS -> rule:n#1 -> RGLOSS -> mandatory:a#1 mandatory:a#1(2) age:n#1 -> RGLOSS -> aged:a#3(2) age:n#1 -> RGLOSS -> aged:a#3(3) Argentine:a#1 -> GLOSS -> Argentina:n#1(3) Argentine:a#1 -> GLOSS -> Argentina:n#1


LCC: Logic ProverLCC: Logic ProverQuestionQuestion

Which company created the Internet Browser Mosaic?Which company created the Internet Browser Mosaic? QLF:QLF: (_organization_AT(x2) ) & company_NN(x2) & create_VB(e1,x2,x6) & (_organization_AT(x2) ) & company_NN(x2) & create_VB(e1,x2,x6) &

Internet_NN(x3) & browser_NN(x4) & Mosaic_NN(x5) & nn_NNC(x6,x3,x4,x5)Internet_NN(x3) & browser_NN(x4) & Mosaic_NN(x5) & nn_NNC(x6,x3,x4,x5)Answer passageAnswer passage

... Mosaic , developed by the National Center for Supercomputing Applications ... Mosaic , developed by the National Center for Supercomputing Applications ( NCSA ) at the University of Illinois at Urbana - Champaign ...( NCSA ) at the University of Illinois at Urbana - Champaign ...

ALF:ALF: ... Mosaic_NN(x2) & develop_VB(e2,x2,x31) & by_IN(e2,x8) & ... Mosaic_NN(x2) & develop_VB(e2,x2,x31) & by_IN(e2,x8) & National_NN(x3) & Center_NN(x4) & for_NN(x5) & Supercomputing_NN(x6) & National_NN(x3) & Center_NN(x4) & for_NN(x5) & Supercomputing_NN(x6) & application_NN(x7) & nn_NNC(x8,x3,x4,x5,x6,x7) & NCSA_NN(x9) & application_NN(x7) & nn_NNC(x8,x3,x4,x5,x6,x7) & NCSA_NN(x9) & at_IN(e2,x15) & University_NN(x10) & of_NN(x11) & Illinois_NN(x12) & at_IN(e2,x15) & University_NN(x10) & of_NN(x11) & Illinois_NN(x12) & at_NN(x13) & Urbana_NN(x14) & nn_NNC(x15,x10,x11,x12,x13,x14) & at_NN(x13) & Urbana_NN(x14) & nn_NNC(x15,x10,x11,x12,x13,x14) & Champaign_NN(x16) ... Champaign_NN(x16) ...

Lexical Chains Lexical Chains develop <-> makedevelop <-> make and and make <->createmake <->create exists x2 x3 x4 all e2 x1 x7 (develop_vb(e2,x7,x1) <-> make_vb(e2,x7,x1) & exists x2 x3 x4 all e2 x1 x7 (develop_vb(e2,x7,x1) <-> make_vb(e2,x7,x1) &

something_nn(x1) & new_jj(x1) & such_jj(x1) & product_nn(x2) & or_cc(x4,x1,x3) something_nn(x1) & new_jj(x1) & such_jj(x1) & product_nn(x2) & or_cc(x4,x1,x3) & mental_jj(x3) & artistic_jj(x3) & creation_nn(x3)).& mental_jj(x3) & artistic_jj(x3) & creation_nn(x3)).

all e1 x1 x2 (make_vb(e1,x1,x2) <-> create_vb(e1,x1,x2) & all e1 x1 x2 (make_vb(e1,x1,x2) <-> create_vb(e1,x1,x2) & manufacture_vb(e1,x1,x2) & man-made_jj(x2) & product_nn(x2)). manufacture_vb(e1,x1,x2) & man-made_jj(x2) & product_nn(x2)).

Linguistic axiomsLinguistic axioms all x0 (mosaic_nn(x0) -> internet_nn(x0) & browser_nn(x0)) all x0 (mosaic_nn(x0) -> internet_nn(x0) & browser_nn(x0))


USC-ISIUSC-ISI

Textmap system Textmap system Ravichandran and Hovy, 2002Ravichandran and Hovy, 2002 Hermjakob et al. 2003Hermjakob et al. 2003

Use of Surface Text PatternsUse of Surface Text PatternsWhen was X born ->When was X born ->

Mozart was born in 1756Mozart was born in 1756 Gandhi (1869-1948)Gandhi (1869-1948)Can be captured in expressionsCan be captured in expressions <NAME> was born in <BIRTHDATE><NAME> was born in <BIRTHDATE> <NAME> (<BIRTHDATE> -<NAME> (<BIRTHDATE> -

These patterns can be These patterns can be learnedlearned Similar in nature to DIRT, using Web as a corpusSimilar in nature to DIRT, using Web as a corpus Developed in the QA application contextDeveloped in the QA application context


USC-ISI TextMapUSC-ISI TextMapUse bootstrapping to learn patterns. Use bootstrapping to learn patterns. For an identified question type (“For an identified question type (“When was X born?”When was X born?”), start with known answers ), start with known answers for some values of Xfor some values of X

Mozart 1756Mozart 1756 Gandhi 1869Gandhi 1869 Newton 1642Newton 1642

Issue Web search engine queries (e.g. Issue Web search engine queries (e.g. “+Mozart +1756”“+Mozart +1756” ) )Collect top 1000 documentsCollect top 1000 documentsFilter, tokenize, smooth etc.Filter, tokenize, smooth etc.Use suffix tree constructor to find best substrings, e.g.Use suffix tree constructor to find best substrings, e.g.

Mozart (1756-1791)Mozart (1756-1791)FilterFilter

Mozart (1756-Mozart (1756-Replace query strings with e.g. Replace query strings with e.g. <NAME><NAME> and and <ANSWER><ANSWER>

Determine precision of each patternDetermine precision of each pattern Find documents with just question term (Mozart)Find documents with just question term (Mozart) Apply patterns and calculate precisionApply patterns and calculate precision


USC-ISI TextMapUSC-ISI TextMap

Finding AnswersFinding Answers Determine Question typeDetermine Question type Perform IR QueryPerform IR Query Do sentence segmentation and smoothingDo sentence segmentation and smoothing Replace question term by question tag Replace question term by question tag

i.e. replace i.e. replace MozartMozart with with <NAME><NAME> Search for instances of patterns associated with Search for instances of patterns associated with

question typequestion type Select words matching Select words matching <ANSWER><ANSWER> Assign scores according to precision of patternAssign scores according to precision of pattern


Additional Linguistic PhenomenaAdditional Linguistic Phenomena


Negation (1)Negation (1)

Q: Who invented the electric guitar?A: While Mr. Fender did not invent the electric guitar, he did revolutionize and perfect it.

Note: Not all instances of “not” will invalidate a passage.


Negation (2)Negation (2)

Name a US state where cars are manufactured.Name a US state where cars are manufactured. versusversus

Name a US state where cars are not manufactured.Name a US state where cars are not manufactured.

Certain kinds of negative events or instances are rarely asserted explicitly in text, but must be deduced by other means


Other Adverbial Modifiers Other Adverbial Modifiers (Only, Just etc.)(Only, Just etc.)

Name an astronaut who Name an astronaut who nearlynearly made it to the moonmade it to the moon

Name an astronaut who Name an astronaut who nearlynearly made it to the moonmade it to the moon

To satisfactorily answer such questions, need to know what are the different ways in which events can fail to happen. In this case there are several.


Attention to DetailsAttention to Details

TensesTenses Who Who isis the Prime Minister of Japan? the Prime Minister of Japan?

NumberNumber What What areare the largest snake the largest snakess in the world? in the world?

^


Jeopardy Examples - CorrectJeopardy Examples - Correct

Literary CharacterLiterary CharacterWanted for killing sir Danvers Carew ;Wanted for killing sir Danvers Carew ;Seems to have a split personalitySeems to have a split personalityHyde – correct ( Dr. Jekyll and Mr. Hyde)Hyde – correct ( Dr. Jekyll and Mr. Hyde)

category: olympic odditiescategory: olympic oddities Milrad Cavic almost upset this man's perfect 2008 Milrad Cavic almost upset this man's perfect 2008

olypmics, losing to him by 100th of a secondolypmics, losing to him by 100th of a second Michael PhelpsMichael Phelps

(identified name type – “man”)(identified name type – “man”)


Jeopardy Examples - WrongJeopardy Examples - Wrong

Name the decade:Name the decade:The first modern crossword puzzle is published The first modern crossword puzzle is published

& Oreo cookies are introduced& Oreo cookies are introduced Watson: wrong - 1920’s (57%), Watson: wrong - 1920’s (57%),

but the correct 1910’s with 30% but the correct 1910’s with 30%

largest US airport named after a World War II herolargest US airport named after a World War II hero Toronto, the name of a Canadian city.Toronto, the name of a Canadian city. (Missed that US airport means that the airport is in the (Missed that US airport means that the airport is in the

US, or that Toronto isn’t in the U.S.)US, or that Toronto isn’t in the U.S.)

General Perspective on General Perspective on Semantic ApplicationsSemantic Applications

Semantic applications as “text matching”Semantic applications as “text matching”

Matching between target texts andMatching between target texts and Supervised: training textsSupervised: training texts Unsupervised: user input (e.g. question)Unsupervised: user input (e.g. question)

Cf. the Cf. the textual entailment textual entailment paradigmparadigm
