AnswerBus Question Answering System

Zhiping Zheng

School of Information, University of Michigan

HLT 2002

ABSTRACT

• AnswerBus is an open-domain question answering system based on sentence level Web information retrieval.

• It accepts users’ natural-language questions in English, German, French, Spanish, Italian and Portuguese and provides answers in English.

• Five search engines and directories are used to retrieve relevant Web pages.

• MRR=70.5% for TREC-8’s 200 questions.

Introduction

• Researchers have experimented with QA systems based on– closed, pre-tagged corpora– knowledge bases– Text REtrieval Conference (TREC) tasks

• Recent open-domain QA systems on WWW:– LCC, QuASM, IONAUT, START and Webclop

edia

AnswerBus

• Questions:– in natural language– in English, German, French, Spanish, Italian an

d Portuguese

• Answers:– from the Web– via Google, Yahoo, WiseNut, AltaVista, and Y

ahoo News

Working process of AnswerBus

• A simple language recognition module will determine whether the question is in English,

• If not, AltaVista’s translation tool BabelFish is used to translate it into English.

1. select two or three search engines among five for information retrieval 2. contact the search engines and retrieve documents referred at the top of the

hit lists

3. extract sentences that potentially contain answers from the documents

4. rank the answers and return the sentences of top choices with contextual URL links to the user.

Search Engine Selection• Different search engines or directories may suit dif

ferent types of questions.– for current events, Yahoo News may be a better choice

than Google

• Determination– pre-answer 2000 questions– record words in each question together with correct ans

wers returned by each search engine– given a new query “word1 word2”

word1: (Google, 7 answers), (AltaVista, 4 answers)

word2: (Google, 8 answers), (AltaVista, 6 answers)– Google (7+8) is chosen this time.

Relevant Document Retrieval• AnswerBus aims to retrieve enough relevant docu

ments from search engines within an acceptable response time.

• The main tasks are to select one or more appropriate search engines for a specific user question.

• Then form the queries.– Functional words deletion (of, in, …)– Frequently used words deletion– Special words deletion (give me, name one…)– Word form modification (Who did … end? → ended)

Candidate Answer Extraction

• AnswerBus first parses the documents into sentences and then determines whether sentence is an answer candidate.

• Two classes of words in a question:– matching words: words also in the query

– non-matching words: words not in the query

• Filtering– sentences not matching the following formula are filter

ed out.

Filtering Formula• • q is the number of matching words in the sentence• Q is the total number of matching words in the

query– Ex: if a query is of 3 words long, then only sentences

which match 2 or more words are kept for answer ranking.

• Sentences which contain no non-matching words are also dropped.

• Sentences ended with ‘?’ are also dropped.

11 Qq

Answer Ranking

• Other factors:– the determination of question type and use of a

QA specific dictionary– named entities extraction– coreference resolution

• The final score is a combination of the primary score and the influence of all the different factors.

Question Type andQA specific dictionary

• “How far …?” and “How close …?”– Qtype: DISTANCE

In QA specific dictionary:– “How close” unit: mile, kilometer, light year, in

ch, centimeter,…– “How far” unit: all above except short unit, suc

h as inch, centimeter…

Dynamic Named Entities Extraction

• The speed of a normal NE tagging technique is 100M/hour.– For one question, 50 HTML documents 1M by

tes needs 36 seconds.

• AnswerBus conducts dynamic named entities extraction, which extracts only the named entities that match question types.

Coreference Resolution

• AnswerBus only solves the coreferences in the adjacent sentences.– “he”, “they”…

• When this type of coreference is detected, the later sentence receives part of score from its previous sentence.

Hit Position andSearch Engine Confidence

• A sentence extracted from the first hit receives the highest score.

• The score decreases according to the position.

• Documents returned by different search engines may also receive different scores.

• Redundant sentences from different search engines are removed.

Evaluation

• Questions: 200 TREC-8 questions

• Comparing systems (via Internet): START, LCC, IONAUT, and QuASM

• Answers are judged manually.

• In the following table, T refers to Time, and L the Length of answers.

The Performance of Online Question Answering Systems

SystemsCorrectTOP 5

CorrectTOP 1

NISTScore

Tmax(s)

Tmin(s)

Tmean(s)

Tstddev

Lmean(byte)

AnswerBus 141 120 64.18% 15.06 3.79 7.2 3.07 141IONAUT 44.88 2.78 12.51 6.81 1312LCC 97 75 41.73% 342.52 4.3 44.24 32.63 178QuASM 13 7 4.45% 284.29 2.61 20.72 33.92 1766START 29 29 14.50% 62.07 2.02 9.84 7.45

Future Work

• Answer generation– An ideal QA system should be able to extract

the exact answer or summarize the potential answers.

• QA specific indexing– instead of general search engines

• New question set– TREC questions are not designed for Web-

based QA systems.