Web Searching with Multiple Correct

Answers

Steve O’Hara, Google Inc., CaliforniaTom Bylander, Univ. of Texas San Antonio

June 2 - 4, 2014, Thessaloniki, GreeceWeb Intelligence, Mining and Semantics

“LONGMONT, CO—The Information Age was dealt a stunning blow Monday, when a factual error was discovered on the Internet. The error was found on TedsUltimateBradyBunch.com, a Brady Bunch fan site that incorrectly listed the show's debut year as 1968, not 1969.”

http://www.theonion.com/articles/factual-error-found-on-internet,102/

Factual Error Found On Internet!

“What is the atomic mass of Copper?” “What is the diameter of Pluto?”

Factoid Questions

# Value Units Miles Delta Min Max

1 2300 KM 1,429 31.1 1,398 1,460

2 2390 KM 1,485 3.1 1,482 1,488

3 2170 MILES 2,170 5.0 2,165 2,175

4 1416 MILES 1,416 0.5 1,416 1,417

5 1444 MILES 1,444 0.5 1,444 1,445

6 1370 MILES 1,370 5.0 1,365 1,375

7 1500 MILES 1,500 50.0 1,450 1,550

8 2390 KM 1,485 3.1 1,482 1,488

9 2390 KM 1,485 3.1 1,482 1,488

10 1429 MILES 1,429 0.5 1,429 1,430

11 2300 KM 1,429 31.1 1,398 1,460

12 2274 KM 1,413 0.3 1,413 1,413

13 1865 MILES 1,865 2.5 1,863 1,868

# Value Units Miles Delta Min Max

14 2274KILO-

METERS1,413 0.3 1,413 1,413

15 1860 MILES 1,860 5.0 1,855 1,865

16 1860 MILES 1,860 5.0 1,855 1,865

17 2320 KM 1,442 3.1 1,438 1,445

18 1430 MILES 1,430 5.0 1,425 1,435

19 2274KILO-

METERS1,413 0.3 1,413 1,413

20 2274KILO-

METERS1,413 0.3 1,413 1,413

21 2270 KM 1,411 3.1 1,407 1,414

22 1485 MILES 1,485 2.5 1,483 1,488

23 1400 MILES 1,400 50.0 1,350 1,450

24300000

0METERS 1,864 310.7 1,553 2,175

25 1400 MILES 1,400 50.0 1,350 1,450

Select “Best” Answer

The two candidate answers of 1400 miles receive the most Support

The four candidate answers of 2274 km (1413 miles) give the most Support

What is the Diameter of Pluto?

zoom

Numeric Correlation

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%0

50

100

150

200

250

300

350

400

450

Reliability for Numeric Questions

Rightiness

Ra

nke

d S

up

po

rt

What is the population of New York City?

What is the area of Canada? When did Elizabeth Taylor get

married? What is the capital of Bolivia? Who coaches the Hoyas men’s

basketball team?

Motivating Questions

Numeric Hypothesis

99

200

301

400

500.1

100

400

403

497

500

300

300.9398.6

601 600

City Populations

City, State City MSA

Abilene, Texas 118,117 166,416

Akron, Ohio 198,402 701,456

Albuquerque, New Mexico 552,804 898,642

Wichita Falls, Texas 103,931 150,261

Wichita, Kansas 384,445 625,526

Wilmington, North Carolina 108,297 369,685

Some Candidate Answers(for the population of Akron, Ohio)

th a population of 217,074 (2000 census).<div/

n, OH. Population: 214,556. Compare Akron to T

ent from 69,000 to 210,000. By the mid 20&#39;

. 1920, Population 208,435 . Warren G. Harding

with approximately 207,934 people within the c

as a population of 207934 people this year. Ak

Total population, 200,181 ...<div/><div><span

as a population of 200,000. Through the 1900s,

any) was producing 200,000 "Diamond" brand lam

ad a population of 199,110. The Akron Metropol

OH, population is 199,110.<div/><div><span st

Candidate Answer Distribution

0

5

10

15

20

25

Num

ber o

f Can

dida

te A

nsw

ers

Population of Akron, Ohio (zoomed)

Target answers: 198,402 city, 701,456 MSA

Abilene Results

Rank

CandidateAnswer

SupportCity

RightinessMSA

Rightiness

1 112,205 4.600 0.431 0.162

2 114,757 4.572 0.510 0.169

RankCandidate

Answer Pair

Paired Suppo

rt

City Rightines

s

MSA Rightine

ss

1 106,654 - 108,157 4.273 0.356 0.148

2975,619 - 1,005,656

3.394 0.018 0.027

Rank

CandidateAnswer

SupportCity

Rightiness

MSA Rightine

ss

1 116,966 6.889 0.656 0.175

2 1,005,656 5.021 0.018 0.026

Single Score(0.431/1 + 0.169/2) / 2

0.258

Paired Score(0.356 + 0.148) / 2

0.252

Combined Score(0.656 + 0.026) / 2

0.341

Target: 118,117 Cityand 166,416 MSARan

kCandidate

AnswerSupport

CityRightiness

MSA Rightiness

1 117,063 2.000 0.668 0.175

2 11,928,500 2.000 0.001 0.002

Baseline Score(0.668 + 0.002) / 2

0.335

City Population Results

 Baselin

e Results

Single Result

s

Paired Result

s

Combined Results

Mean Score 0.233 0.297 0.291 0.319

Improvement - 31% 29% 41%

Multi-Valued Named Entities

Ann

Bob

Carl

Dan

Eva

Anna

Dana

Dana

Eve

Eve

Carl

CarlDana

Fred Fred

Adapt Levenshtein’s Algorithm to whole words

Count #insertions, #deletions and #changes Instead of character matching, do whole word

matching Which uses regular Levenshtein’s edit distance

Handles typos Handles middle names Does not handle Bill / William

Proper Name Matching

Two Levels of Searches

SearchMovie

Directors

SearchMovieStars

MovieDirectors

MovieStars

Directorsand Stars

Movie Target Answers

Movie Director(s) Stars

Ben-Hur William Wyler Charlton Heston, Jack Hawkins,Stephen Boyd, Haya Harareet

Gone with the Wind Victor Fleming Clark Gable, Vivien Leigh,

Thomas Mitchell, Barbara O'Neil

Time of Violence Ludmil StaikovIossif Surchadzhiev, Rousy

Chanev,Ivan Krystev, Anya Pencheva

The English Patient

Anthony Minghella

Ralph Fiennes, Juliette Binoche,Willem Dafoe, Kristin Scott

Thomas

Schindler's List Steven Spielberg Liam Neeson, Ralph Fiennes,Ben Kingsley, Caroline Goodall

GWTW Results

Support Candidate Answer Result

37 Victor Fleming correct

21 George Cukor wrong

12 Sam Wood (skip)

Support Candidate Answer Result

10 Vivien Leigh correct

5 Clark Gable (skip, correct)

3 Victor Fleming (skip, director)

3 Evelyn Keyes (skip)

2 Rhett Butler (skip)

2 Rand Brooks (skip)

MovieDirectors

MovieStars

Of 500 movies, 32 instances of co-occurring

pairs 23 additional correct answers 5 duplicate correct answers 4 incorrect answers:

Movie Results

Movie Errors and Rationalization

JFK Tom Hanks is the director of a second JFK movie.Lee Harvey Oswald was the assassin.

The Fighter The movie is about Mickey Ward.

V for Vendetta

Alan Moore is the author of the book.

Is Wikipedia Always Right?

“The number of factual errors shows just how unreliable it can be to use the online resource as a sole means of digging up information.

Yet millions base everything from school homework to corporate presentations using facts and figures they have gleaned from the site.

A study into, specifically, company information on the massively popular website discovered 60 per cent of articles had factual errors.”

Source: www.dailymail.co.uk/sciencetech/article-2131458/Up-articles-Wikipedia-contain-factual-errors.html#ixzz2sDrWl4RH