1
GLOW- Global and Local Algorithms for Disambiguation to Wikipedia Lev Ratinov, Doug Downey, Mike Anderson, Dan Roth This research is sponsored by DARPA Machine Reading Program under Air Force Research Laboratory (AFRL) prime contract No. FA8750-09-C-0181 and by the Army Research Laboratory (ARL) under agreement W911NF-09-2-0053. Problem Definition: Find all the “interesting” concepts in the input text, and disambiguate them to the corresponding Wikipedia titles”. Problem Formulation: Γ * is a solution to the problem, a set of mention-title pairs (m,t). Evaluate the local matching quality using Φ(m,t). Evaluate the global structure based on (1) pair-wise coherence scores Ψ(t i ,t j ) (2) a solution-proxy Γ’ (Γ’ can be unambiguous mentions (such as Jiangsu), or local solutions to all mentions). Γ’ allows solving the problem locally while taking into account the global structure.) Chicago was used by default for Mac menus through MacOS 7.6 , and OS 8 was released mid-1997..” Combining local and global matchers: Font or City ? Prior: P(title| surface form) Text Context Title co- occurrenc e Font 0.001 0.01 0.23 0.004 City 0.99 0.00 1 0.03 0 We train an SVM ranker which makes use of the difference between the feature vectors. It abstracts over the text and learns which signals to trust more: SVM(font Vs. city)=(0.99-0.001,0.001- 0.01,0.03-0.23,0-0.004) End-to-end system. experimental results: GLOW Vs. Previous State of the Art (Milne 08) 1 It’s a version of Chicago the standard classic Macintosh menu font, with that distinctive thick diagonal in the ”N”. Chicago was used by default for Mac menus through MacOS 7.6 , and OS 8 was released mid-1997.. Chicago VIII was one of the early 70s-era Chicago albums to catch my ear, along with Chicago II . F1: Disambiguation Candidates Generation T2: Ranking Accuracy on “Solvable Mentions” Discussion: Table T1 summarizes dataset statistics. We fail to detect some mentions. We are only investigating “identifiable mentions” here. Figure F1 shows that P(title|surface form) is a very strong baseline. Choosing the link based on this decision alone leads to over 80% accuracy on all datasets except MSNBC. We add disambiguation candidates based on P(title|surface form). After 5 candidates, adding more candidates is inefficient. We cut off candidate generation at 20. If the correct solution appears in top 20 candidates, then the mention is “solvable”. Table T2 compares ranking accuracy with different strategies on “solvable mentions”. We see that the global approaches are better, and combining local and global leads to minor improvements. It is important, however, to identify “non-solvable mentions”. For example, we want to know that “Michael Rush, a 16-year-old hunter” does not have a Wikipedia page. We train a linker, a classifier for that task and test its performance in Table T3. It shows that the local features are better for this task, but the end performance is brittle. Identifying names that do not appear in Wikipedia is currently unsolved problem. T3: Impact of Training a Linker on End Performance Visit our demo: http://cogcomp.cs.illinois.edu/demo/wikify/ Local Match Scorers: P(title|surface form) : Baseline PrTFIDF(title text | input document) PrTFIDF(title context | input document) PrTFIDF(title text | surface form context) PrTFIDF(title context | surface form context) PrTFIDF is a probabilistic version of TFIDF (Joachims 97) Global Structure Scorers: Average, Minimal, Maximal relatedness of the title to: - unambiguous mentions. - baseline disambiguation of the NER mentions. - baseline disambiguation of all the mentions. Using PMI, Normalized Google Distance. T1: Datasets

GLOW- Global and Local Algorithms for Disambiguation to Wikipedia

  • Upload
    azra

  • View
    51

  • Download
    0

Embed Size (px)

DESCRIPTION

GLOW- Global and Local Algorithms for Disambiguation to Wikipedia. Lev Ratinov, Doug Downey, Mike Anderson, Dan Roth. Problem Definition: “ Find all the “interesting” concepts in the input text, and disambiguate them to the corresponding Wikipedia titles”. Problem Formulation: - PowerPoint PPT Presentation

Citation preview

Page 1: GLOW- Global and Local Algorithms for Disambiguation to Wikipedia

GLOW- Global and Local Algorithms for Disambiguation to WikipediaLev Ratinov, Doug Downey, Mike Anderson, Dan Roth

This research is sponsored by DARPA Machine Reading Program under Air Force Research Laboratory (AFRL) prime contract No. FA8750-09-C-0181 and by the Army Research Laboratory (ARL) under agreement W911NF-09-2-0053.

Problem Definition:

“Find all the “interesting” concepts in the input text, and disambiguate them to the corresponding Wikipedia titles”.

Problem Formulation:

Γ* is a solution to the problem, a set of mention-title pairs (m,t). Evaluate the local matching quality using Φ(m,t). Evaluate the global structure based on (1) pair-wise coherence scores Ψ(ti,tj) (2) a solution-proxy Γ’ (Γ’ can be unambiguous mentions (such as Jiangsu), or local solutions to all mentions). Γ’ allows solving the problem locally while taking into account the global structure.)

Problem Definition:

“Find all the “interesting” concepts in the input text, and disambiguate them to the corresponding Wikipedia titles”.

Problem Formulation:

Γ* is a solution to the problem, a set of mention-title pairs (m,t). Evaluate the local matching quality using Φ(m,t). Evaluate the global structure based on (1) pair-wise coherence scores Ψ(ti,tj) (2) a solution-proxy Γ’ (Γ’ can be unambiguous mentions (such as Jiangsu), or local solutions to all mentions). Γ’ allows solving the problem locally while taking into account the global structure.)

”Chicago was used by default for Mac menus through MacOS 7.6, and OS 8 was released mid-1997..”

Combining local and global matchers: Font or City ?

Prior:P(title| surface

form)

Text Context

Title co-occurrenc

e

Font 0.001 0.01 0.23 0.004City 0.99 0.00

10.03 0

We train an SVM ranker which makes use of the difference between the feature vectors. It abstracts over the text and learns which signals to trust more:SVM(font Vs. city)=(0.99-0.001,0.001-0.01,0.03-0.23,0-0.004)

End-to-end system. experimental results: GLOW Vs. Previous State of the Art (Milne 08)

1

It’s a version of Chicago – the standard classic Macintosh menu font, with that distinctive thick diagonal in the ”N”.

Chicago was used by default for Mac menus through MacOS 7.6, and OS 8 was released mid-1997..

Chicago VIII was one of the early 70s-era Chicago albums to catch myear, along with Chicago II.

F1: Disambiguation Candidates Generation

T2: Ranking Accuracy on “Solvable Mentions”

Discussion:

Table T1 summarizes dataset statistics. We fail to detect some mentions. We are only investigating “identifiable mentions” here. Figure F1 shows that P(title|surface form) is a very strong baseline. Choosing the link based on this decision alone leads to over 80% accuracy on all datasets except MSNBC. We add disambiguation candidates based on P(title|surface form). After 5 candidates, adding more candidates is inefficient. We cut off candidate generation at 20. If the correct solution appears in top 20 candidates, then the mention is “solvable”. Table T2 compares ranking accuracy with different strategies on “solvable mentions”. We see that the global approaches are better, and combining local and global leads to minor improvements. It is important, however, to identify “non-solvable mentions”. For example, we want to know that “Michael Rush, a 16-year-old hunter” does not have a Wikipedia page. We train a linker, a classifier for that task and test its performance in Table T3. It shows that the local features are better for this task, but the end performance is brittle. Identifying names that do not appear in Wikipedia is currently unsolved problem.T3: Impact of Training a Linker on End Performance

Visit our demo: http://cogcomp.cs.illinois.edu/demo/wikify/Local Match Scorers: P(title|surface form) : Baseline PrTFIDF(title text | input document) PrTFIDF(title context | input document) PrTFIDF(title text | surface form context) PrTFIDF(title context | surface form context) PrTFIDF is a probabilistic version of TFIDF (Joachims 97)

Global Structure Scorers:Average, Minimal, Maximal relatedness of the title to:- unambiguous mentions.- baseline disambiguation of the NER mentions.- baseline disambiguation of all the mentions.Using PMI, Normalized Google Distance.

T1: Datasets