Upload
horatio-king
View
220
Download
1
Tags:
Embed Size (px)
Citation preview
Automatic Answer Validation Automatic Answer Validation in Openin Open-Domain -Domain
Question AnsweringQuestion Answering Hristo Tanev
TCC,ITC - IRST
Open Domain Question AnsweringOpen Domain Question Answering
• Automatic extracting of the answer of a natural language question
• Related fields: – Information Extraction– Information Retrieval
• Deeper text analysis
Which is the capital of Italy?
ROME
Question processing
Which is the capital of
Italy?
Question type: Which-LOCATION
Keywords: capital
Italy
IR engine
Document collection
Document collection
Selected documents/paragraphs
Selected documents/paragraphs
Question type: Which-LOCATION
Keywords: capital
Italy
Answer extraction
Selected documents/paragraphs
Selected documents/paragraphs
Candidate answers
Candidate answers
Paris, Milan, Rome, TexasQuestion type: Which-
LOCATION
Keywords: capital
Italy
Answer evaluation
and validation
Candidate answers
Candidate answers
Knowledge bases, Abduction
Paris - 1, Milan - 1, Rome - 2
The The complexitycomplexity of the QA task of the QA task
• The variety of question classes
• The infinite number of answer formulations
• Anaphora, ellipsis, synonymy
• Sometimes syntactic and semantic analysis are necessary, also world knowledge
Answer inferenceAnswer inference
• The problem:
How to infer if a candidate answer is relevant with respect to the question?
• Filtering out the irrelevant answer candidates
• Score the candidate answers according to their relevance
Contemporary approaches for Contemporary approaches for answer inference answer inference
• Deducing the question logical form (QLF) from the text logical form
Sanda M. Harabagiu and Marius Pasca and Steven Maiorano “Experiments with Open-Domain Textual Question Answering”, COLING 2000,292-298
Example:
Q: Why did David Koresh ask FBI for a word processor?A: Mr. Koresh sent a request for word processor to FBI
to enable him to write his revelationsQLF: ask(Koresh, FBI, word processor,reason=?)ALF: sent request(Koresh, FBI, word processor,reason:
to write his revelations)Heuristic: send request => ask, ALF => QLF
Contemporary approaches for Contemporary approaches for answer inference (continued)answer inference (continued)
• Abduction, using pragmatic axioms, and semantic representation
Sanda Harabagiu , Steven Maiorano “Finding Answers in Large Collections of Texts: Paragraph Indexing + Abductive Inference”
– action1 (e1, Person1) & action1 (e2, Person2) & related_events(e1, e2) => related(Person1, Person2)
– Q:Who was Lincoln’s Secretary of State– A:Booth schemed to kill Lincoln, while his compatriots would
murder Vice President Andrew Johnson and Secretary of State William Seward.
– kill(e1, Lincoln) & murder(e2, Secretary of State William Seward) & related(e1 , e2) =>related(Lincoln, Secretary of State William Seward)
Contemporary approaches for Contemporary approaches for answer inference (continued)answer inference (continued)
• Lexico – syntactic patterns
Q: What forms of international crime exist?
A:…international forms of crime, including terrorism, blackmail and drug-related problems.
• These kinds of patterns are appropriate for certain type of questions, asking for taxonomic information
Contemporary approaches for Contemporary approaches for answer inference - disadvantages answer inference - disadvantages
• A very large open domain knowledge base is requisite
• The creation of knowledge bases is very expensive in time and resources
• The present world knowledge bases (such as WordNet or ThoughtTreasure) are far away from being comprehensive
• The question and its answer can be very different lexically, this poses the necessity from deep semantic analysis to infer the relation between the discourse entities
The simple approach – The simple approach – ask the oracle ask the oracle
Rome is the capital of
Italy
Rome is the capital of
Italy
• The database should be large enough to encode a great part of the human knowledge
• It should provide the necessary redundancy to contain different reformulations for the facts
• It should be changed dynamically to reflect recent state of thehuman knowledge about the world
• It should be easily accessible
World Wide Web as a source of knowledge
• Comprehensive
• Open domain nature
• Constantly updated and expanded
• Search indices and engines
• Implicit knowledge
My journey in Italy began in the capital Rome…
Disadvantages:
• Knowledge is in unstructured text form
• Access to the search engines may be slow
Web as a gigantic corpusWeb as a gigantic corpus
• Parameters:– 100.000.000 hosts– AltaVista indexes over 1.000.000.000 Web pages– Google 2.000.000.000 Web pages– 86% English language pages, 5.8% German, 2.36
French, 1.6% Italian
• AccessibilityDifferent public accessible search engines:
AltaVista, Fast, Google, Excite, Lycos, Yahoo!, Northern Light
Validation StatementsValidation Statements
Question Candidate Answer
• Who is Galileo? astronomer• Galileo is an astronomer• Which is the capital of Italy? Rome• Rome is the capital of Italy• Why the moon turns orange? because it
enters the Earth shadow• The moon turns orange because it enters the
Earth shadow
Validation Statements (continued)Validation Statements (continued)
The core of the data-driven answer validation is searching on-line texts, similar to the validation statement for a question-answer pair
The Answer Validation AlgorithmThe Answer Validation Algorithm
• Question + Answer = Validation Pattern– Q: How far is it from Denver to Aspen– A: 200 miles– QAP : [Denver … Aspen … 200 miles]
• Submit validation pattern to search engine• Infer the power of the relation between
Question & Answeron the basis of the search engine result
An ExampleAn Example
• QA pair: Who is Galileo? astronomer• Submit to AltaVista the query “Galileo”
– AltaVista returns 2000 hits• Submit to AltaVista the query “astronomer”
– AltaVista returns 10000 hits• Submit to AltaVista the query
Galileo NEAR astronomer– AltaVista returns 1000 hits
• PMI(Galileo, astronomer) = 14 > threshold
Validation Patterns Validation Patterns
The validation pattern is the base of the query which is submitted to the search engine to check if the question and the answer tend to appear together
Word Level Validation PatternsWord Level Validation Patterns
• Qk1, Qk2,…. The question keywords• A The Answer• The query to the search engine is formed by
linking the question keywords and answer with operators like AND or NEAR– Qk1 NEAR Qk2 NEAR …NEAR A– Qk1 AND Qk2 AND …AND A– (Qk1 AND Qk2 …) NEAR A
• This way co-occurrence between the question and answer keywords is searched in Intenet
Phrase Level Validation PatternsPhrase Level Validation Patterns
• Validation pattern is composed by syntactic phrases instead of separate keywords
• Example:– Q: What city had a world fair in 1900?– A: Paris– Query: ( city NEAR “world fair” NEAR “in 1900” ) NEAR Paris
• Pages found by these type of patterns are more likely to contain texts confirming the answer corectness
• Disadvantage: less probable, often obtain 0 hits even for the right answer
Phrase Level Validation Patterns Phrase Level Validation Patterns (continued)(continued)
• The phrases may be extracted by parser from the question
• More probable and coherent phrases should be preferred over the rare and non coherent phrases
• The phrase frequency may be measured using Web as a corpus
Sentence Level PatternsSentence Level Patterns
• If the question and the answer are short, the whole validation statement can be submitted to the search engine
• “When did Hawaii become a state?” – 1959• “Hawaii became a state in 1959”• Linguistic transformations are necessary to transform the
QA pair in a validation statement
Morphological Variations and Morphological Variations and Symonymy in PatternsSymonymy in Patterns
• The question and answer keywords may occur in different morphological forms
• Synonyms can also appear instead of the original keywords
• Most search engines (Google, AltaVista, Yahoo) allow the use of keyword variants by OR operator
• Q: What date did John Lenon die?• Question pattern:
John NEAR Lenon NEAR (die OR died)
Types of data driven answer Types of data driven answer inferenceinference
• Pure quantitative approach: only the number of hits, returned by the search engine are considered. Statistical techniques form the core of this class of approaches
• Qualitative approaches: the document content is processed
Statistical answer validationStatistical answer validation
• By search engine queries are obtained the frequencies of the question pattern, the answer and the question-answer validation pattern
• ExampleQuestion: How far is it from Denver to Aspen?Question Pattern : far NEAR Denver NEAR AspenAnswer : 200 milesQAP: far NEAR Denver NEAR Aspen NEAR 200 miles
• Search engine:– Frequency(Question Pattern)– Frequency(Answer)– Frequency(QAP)
Statistical answer validationStatistical answer validation
•Using the frequencies and the number approximating the pages indexed by the search engine are calculated the following probabilities for occurrence in Web:
P(Question Pattern), P(Answer), P(Question-Answer co-occurrence)
Statistical answer validationStatistical answer validation
•Thus calculated probabilities are combined in formulae, which are derived from classical co-occurrence formulae.
•The difference from the classical co-occurrence task is that we search how the appearance of the question pattern implies the appearance of the answer. Thus non symmetrical formulae are necessary.
•These formulae return a value, which is an indication for the answer corectness with respect to the question.
Statistical answer validationStatistical answer validation
3/2)(
)|(
AnswerP
QuestionAnswerP
Answer validation formulae
)(
)()|(
AnswerP
AnswerPQuestionAnswerP
Qualitative ApproachQualitative Approach
• The qualitative answer validation considers the content of the obtained documents as a result of the validation pattern submition to the search engine
• The distance between the question and answer keywords is considered
Qualitative ApproachQualitative Approach
• The use of document snippets can speed up this approach
• Certain search engines, like Yahoo! and Google return text snippets from the documents, where the keywords appear
Qualitative Approach. Extraction of Qualitative Approach. Extraction of data from the snippets.data from the snippets.
Q: Who is the first man to fly across the Pacific Ocean?A: PangbornQuery, submitted to Google:
first AND man AND fly OR flew AND Pacific AND Ocean AND Pangborn
Text snippets returned:“Pangborn became the first pilot to cross Pacific”“Pangborn with co-pilot Hew Herndon flew across
Pacific”
Qualitative Approach. Extraction of Qualitative Approach. Extraction of data from the snippets (continued).data from the snippets (continued).
Obtained co-occurrence relations:(Pangborn, first, Pacific) (Pangborn, fly, Pacific)Numerical values, obtained from the relations:
Proportion of question keywords, related to answer (0.6 in the example , 3 question keywords (first, Pacific, fly) related to answer from total of 5 question keywords)
Number of different relations and their length
Qualitative Approach. Calculating Qualitative Approach. Calculating answer relevanceanswer relevance
• Only the different co-occurrence relations are considered, co-occurrences, which are included in others are excluded
r
rlengthPQK 1)(2.
PQK percent of question keywords, related to the answer
r relations, obtained for the answer from the snippets
length(r) the number of words in the co-occurrence relation r
Qualitative Approach. Calculating Qualitative Approach. Calculating answer relevance (continued)answer relevance (continued)
• Keyword density in the co-occurrence relations may also be considered
• The formula may be the sum of the keyword densities for all the relations
r
rKeyDensity )(
Combining approachesCombining approaches
• The qualitative approach can be used to extract co-occurrences
• Statistical techniques can be used to evaluate these co-ocurrences
ExperimentExperiment
• The statistical approach was tested• The TREC10 question-answer list has been
used, provided by NIST– for total of 492 questions maximum three right and
three wrong answers are taken
• Two experiments were carried outPerformance of the system on the full set of questionsNamed entities questions
• A baseline model was introduced
Experiment (continued)Experiment (continued)
• For every 50 byte answer the algorithm extracts only the entities that correspond to the question type
• The pairs question – answer were evaluated using AltaVista
• Phrase level patterns and two types of word-level patterns has been used
Experiment. The Patterns.Experiment. The Patterns.
• Three types of patterns: – Phrase,Word level with NEAR,Word level with AND
• Example:– Q: “What city had a world fair in 1900?”– A: Paris– Phrase pattern: – (city NEAR “world fair” NEAR 1900) NEAR Paris– Word level with NEAR: – (city NEAR world NEAR fair NEAR 1900) NEAR Paris– World level with AND: – (city AND world AND fair AND 1900) NEAR Paris
ResultsResults
Test Set Success Rate
3000 question-answer pairs from TREC10
81%
1500 question-answer pairs for named entity questions from TREC10
86%
Baseline model 52%
Much more to do…Much more to do…
• Improvement of the statistical formulae
• Research on the search engine use
• Combining the qualitative and statistical approach
• Creation of reliable validation patterns
• Introducing new techniques for answer validation
• Integration in QA system