Upload
others
View
18
Download
0
Embed Size (px)
Citation preview
Design Challenges for Entity Linking
Xiao Ling, Sameer Singh, Daniel S. Weld
Entity Linking
2
Seattle beat Portland yesterday.
Entity Linking
3
Seattle beat Portland yesterday.
Entity Linking
4
Seattle beat Portland yesterday.
Seattle (city)
Seattle Sounders
Sea-Tac(airport)
Entity Linking
5
Seattle beat Portland yesterday.
Seattle (city)
Seattle Sounders
Sea-Tac(airport) ~3-4 M
entries
Applications• Relation Extraction
(e.g. Koch et al. 2014)
• Coreference Resolution (e.g. Hajishirzi et al. 2013, Durrett & Klein 2014)
• Question Answering (e.g. Sun et al. 2015)
• Web Search (e.g. Knowledge Graph)
• many others… (see Shen et al. 2014; Roth et al. 2014)
6
Ambiguity
• Seattle beat Portland yesterday.
• Seattle scores high in the latest report of startup hubs.
• The Emerald City Council To Make Decision on Antibiotic Resolution
7
Ambiguity
• Seattle beat Portland yesterday.
• Seattle scores high in the latest report of startup hubs.
• The Emerald City Council To Make Decision on Antibiotic Resolution
Seattle Sounders
8
Ambiguity
• Seattle beat Portland yesterday.
• Seattle scores high in the latest report of startup hubs.
• The Emerald City Council To Make Decision on Antibiotic Resolution
Seattle Sounders
Seattle (city)
9
Variability• Seattle scores high in the latest report of startup
hubs.
• The Emerald City Council To Make Decision on Antibiotic Resolution
10
Variability• Seattle scores high in the latest report of startup
hubs.
• The Emerald City Council To Make Decision on Antibiotic Resolution
Seattle (city)
11
Related Work• Cucerzan (2007)
• Milne and Witten (2008)
• Kulkarni et al. (2009)
• Ratinov et al. (2011)
• Hoffart et al. (2011)
• Han and Sun (2012)
• He et al. (2013a)12
• He et al. (2013b)
• Cheng and Roth (2013)
• Sil and Yates (2013)
• Li et al. (2013)
• Cornolti et al. (2013)
• … many others
Related Work• Cucerzan (2007)
• Milne and Witten (2008)
• Kulkarni et al. (2009)
• Ratinov et al. (2011)
• Hoffart et al. (2011)
• Han and Sun (2012)
• He et al. (2013a)13
• He et al. (2013b)
• Cheng and Roth (2013)
• Sil and Yates (2013)
• Li et al. (2013)
• Cornolti et al. (2013)
• … many others
Joint Inference
Related Work• Cucerzan (2007)
• Milne and Witten (2008)
• Kulkarni et al. (2009)
• Ratinov et al. (2011)
• Hoffart et al. (2011)
• Han and Sun (2012)
• He et al. (2013a)14
• He et al. (2013b)
• Cheng and Roth (2013)
• Sil and Yates (2013)
• Li et al. (2013)
• Cornolti et al. (2013)
• … many others
Joint Inference
Learning to rank
Related Work• Cucerzan (2007)
• Milne and Witten (2008)
• Kulkarni et al. (2009)
• Ratinov et al. (2011)
• Hoffart et al. (2011)
• Han and Sun (2012)
• He et al. (2013a)15
• He et al. (2013b)
• Cheng and Roth (2013)
• Sil and Yates (2013)
• Li et al. (2013)
• Cornolti et al. (2013)
• … many others
Joint Inference
Deep Neural Networks
Learning to rank
Popular Data Sets
16
Dataset
# of Mentions Knowledge Base
UIUCACE 244 Wikipedia
MSNBC 654 WikipediaAIDA
(Hoffart et al. 2011)
AIDA-D 5917 YagoAIDA-T 5616 Yago
TAC KBP
TAC09 3904 Wikipedia 2008TAC10 2250 Wikipedia 2008TAC10T 1500 Wikipedia 2008TAC11 2250 Wikipedia 2008TAC12 2226 Wikipedia 2008
Unfortunately…
17
ACE MSNBC AIDA-D AIDA-T KBP09 KBP10 KBP10T KBP11 KBP12
Cucerzan (2007) ⎷Milne & Witten (2008)Kulkarni et al. (2009) ⎷Ratinov et al. (2011) ⎷ ⎷Hoffart et al. (2011) ⎷Han & Sun (2012) ⎷He et al. (2013a) ⎷ ⎷He et al. (2013b) ⎷ ⎷
Cheng & Roth (2013) ⎷ ⎷ ⎷Sil & Yates (2013) ⎷ ⎷ ⎷
Li et al. (2013) ⎷ ⎷Cornolti et al. (2013) ⎷ ⎷TAC-KBP participants ⎷ ⎷ ⎷ ⎷ ⎷
Unfortunately…
18
ACE MSNBC AIDA-D AIDA-T KBP09 KBP10 KBP10T KBP11 KBP12
Cucerzan (2007) ⎷Milne & Witten (2008)Kulkarni et al. (2009) ⎷Ratinov et al. (2011) ⎷ ⎷Hoffart et al. (2011) ⎷Han & Sun (2012) ⎷He et al. (2013a) ⎷ ⎷He et al. (2013b) ⎷ ⎷
Cheng & Roth (2013) ⎷ ⎷ ⎷Sil & Yates (2013) ⎷ ⎷ ⎷
Li et al. (2013) ⎷ ⎷Cornolti et al. (2013) ⎷ ⎷TAC-KBP participants ⎷ ⎷ ⎷ ⎷ ⎷
Joint Inference
Deep Neural Networks
Learning to rank
Metonymy
19
ACE MSNBC AIDA-D AIDA-T KBP09 KBP10 KBP10T KBP11 KBP12
Cucerzan (2007) ⎷Milne & Witten (2008)Kulkarni et al. (2009) ⎷Ratinov et al. (2011) ⎷ ⎷Hoffart et al. (2011) ⎷Han & Sun (2012) ⎷He et al. (2013a) ⎷ ⎷He et al. (2013b) ⎷ ⎷
Cheng & Roth (2013) ⎷ ⎷ ⎷Sil & Yates (2013) ⎷ ⎷ ⎷
Li et al. (2013) ⎷ ⎷Cornolti et al. (2013) ⎷ ⎷TAC-KBP participants ⎷ ⎷ ⎷ ⎷ ⎷
… Moscow ’s as yet undisclosed
proposals …
Moscow (city)
Russia (country)
Government of Russia
Nested Entities
20
ACE MSNBC AIDA-D AIDA-T KBP09 KBP10 KBP10T KBP11 KBP12
Cucerzan (2007) ⎷Milne & Witten (2008)Kulkarni et al. (2009) ⎷Ratinov et al. (2011) ⎷ ⎷Hoffart et al. (2011) ⎷Han & Sun (2012) ⎷He et al. (2013a) ⎷ ⎷He et al. (2013b) ⎷ ⎷
Cheng & Roth (2013) ⎷ ⎷ ⎷Sil & Yates (2013) ⎷ ⎷ ⎷
Li et al. (2013) ⎷ ⎷Cornolti et al. (2013) ⎷ ⎷TAC-KBP participants ⎷ ⎷ ⎷ ⎷ ⎷
… Florida Green Party …
Green Party of the US
Green Party of Florida
Contributions• Vinculum: a simple, deterministic, modular EL sys.
• comprehensive evaluation over nine data sets
• candidate conditional prob. can work quite well
• entity types are important to the final performance
• comparable results with two state-of-the-art sys.
21
Agenda
• Introduction
• Vinculum
• Experiments
• Conclusion
22
Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Vinculum ArchitectureSeattle beat Portland yesterday.
23
Input:
Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Mention ExtractionSeattle beat Portland yesterday.
24
Mention Extraction
Candidate GenerationSeattle beat Portland yesterday.
25
Candidate Entities
- Seattle (city)
- Seattle Sounders
- Seattle-Tacoma (airport)
Candidate Generation
Mention Extraction
Conditional probability
26
… capital of the state of Washington .
In 1990, Washington starred as Bleek Gilliam …
Washington refused to run for a third term …
… Washington …
p(e | m) = # [m -> e]
# m
Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
27
… capital of the state of Washington .
In 1990, Washington starred as Bleek Gilliam …
Washington refused to run for a third term …
… Washington …
p( | “Washington”) = # “W” ->
# “W”
Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Conditional probability
Candidate GenerationSeattle beat Portland yesterday.
28
Candidate Entities
- Seattle (city)
- Seattle Sounders
- Seattle-Tacoma (airport)
Candidate Generation
Mention Extraction
Candidate GenerationSeattle beat Portland yesterday.
29
Candidate Entities
- Seattle (city)
- Seattle Sounders
- Seattle-Tacoma (airport)
0.6
0.2
0.1
Candidate Generation
Mention Extraction
Entity TypesSeattle beat Portland yesterday.
30
Candidate Entities
- Seattle (city)
- Seattle Sounders
- Seattle-Tacoma (airport)
0.6
0.2
0.1
Entity Type Prediction• city • sports_team • facility/airport
0.10.4
0.1
Entity Type
Candidate Generation
Mention Extraction
Entity TypesSeattle beat Portland yesterday.
31
Candidate Entities
- Seattle (city)
- Seattle Sounders
- Seattle-Tacoma (airport)
0.6
0.2
0.1
Entity Type Prediction• city • sports_team • facility/airport
0.10.4
0.1
p(e | m) = ∑t p(e,t | m) p(e | m) = ∑t p(e | t,m) p(t | m)
Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Entity TypesSeattle beat Portland yesterday.
32
Candidate Entities
- Seattle (city)
- Seattle Sounders
- Seattle-Tacoma (airport)
0.6
0.2
0.1
p(e | m) = ∑t p(e,t | m) p(e | m) = ∑t p(e | t,m) p(t | m)
p(e | t,m) : re-normalization of cond. prob.
Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Entity TypesSeattle beat Portland yesterday.
33
Candidate Entities
- Seattle (city)
- Seattle Sounders
- Seattle-Tacoma (airport)
0.6
0.2
0.1
p(e | m) = ∑t p(e,t | m) p(e | m) = ∑t p(e | t,m) p(t | m)
p(e | t,m) : re-normalization of cond. prob. e.g. t = LOC p(Seattle-city | LOC, “Seattle”) = 0.6 / 0.7 p(Sea-Tac | LOC, “Seattle”) = 0.1 / 0.7
Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Entity TypesSeattle beat Portland yesterday.
34
Candidate Entities
- Seattle (city)
- Seattle Sounders
- Seattle-Tacoma (airport)
0.6
0.2
0.1
Entity Type Prediction• city • sports_team • facility/airport
0.10.4
0.1
p(e | m) = ∑t p(e,t | m) p(e | m) = ∑t p(e,t | m) p(t | m)
p(t | m)
Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Entity TypesSeattle beat Portland yesterday.
35
Entity Type Prediction
• city • sports_team • facility/airport
0.10.4
0.1
Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Candidate Entities
- Seattle (city)
- Seattle Sounders
- Seattle-Tacoma (airport)
0.6
0.2
0.1
Candidate Generation Candidate Entities
- Seattle (city)
- Seattle Sounders
- Seattle-Tacoma (airport)
0.2
0.4
0.1
Entity Type
Entity TypesSeattle beat Portland yesterday.
36
Entity Type Prediction
• city • sports_team • facility/airport
0.10.4
0.1
Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Candidate Entities
- Seattle (city)
- Seattle Sounders
- Seattle-Tacoma (airport)
0.6
0.2
0.1
Candidate Generation Candidate Entities
- Seattle Sounders
- Seattle (city)
- Seattle-Tacoma (airport)
0.2
0.4
0.1
Entity Type
Coreference
37
Seattle Sounders head coach Sigi Schmid has some ideas … Seattle beat Portland yesterday.
Entity Type
Candidate Generation
Coreference
Mention Extraction
Coherence
38
Candidate Entities
- Seattle (city)
- Seattle Sounders
- Seattle-Tacoma (airport)
Candidate Entities
- Portland, OR
- Univ. of Portland
- Portland Timbers
Seattle beat Portland yesterday.
0.2
0.4
0.1
0.2
0.2
0.1
Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Normalized Google Distance (NGD)
39
(Milne & Witten, 2008)Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Normalized Google Distance (NGD)
40
(Milne & Witten, 2008)
George Washington
Denzel Washington
President of the US
US Constitution
American Revolutionary
War
Tony Award
Golden Globe Training Day
Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Normalized Google Distance (NGD)
41
(Milne & Witten, 2008)
George Washington
Denzel Washington
President of the US
US Constitution
American Revolutionary
War
Tony Award
Golden Globe Training Day
Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Normalized Google Distance (NGD)
42
(Milne & Witten, 2008)
George Washington John Adams
President of the US
US Constitution
American Revolutionary
WarQuasi-WarPresident
of the USUS
Constitution
Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Relational Score
• Relation triples from Freebase
• A binary score =
• 1, if two entities appear in a triple
• 0, otherwise
• E.g. (Barack Obama, birthplace, United States) => r (Barack Obama, United States) = 1
43
(Cheng & Roth, 2013)Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Context
44
Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Agenda
• Introduction
• Vinculum
• Experiments
• Conclusion
45
ACE MSNBC AIDA-D AIDA-T KBP09 KBP10
KBP10T
KBP11
KBP12Cucerzan (2007) ⎷
Milne & Witten (2008)Kulkarni et al. (2009) ⎷
Ratinov et al. (2011) ⎷ ⎷
Hoffart et al. (2011) ⎷
Han & Sun (2012) ⎷
He et al. (2013a) ⎷ ⎷
He et al. (2013b) ⎷ ⎷
Cheng & Roth (2013) ⎷ ⎷ ⎷
Sil & Yates (2013) ⎷ ⎷ ⎷
Li et al. (2013) ⎷ ⎷
Cornolti et al. (2013) ⎷ ⎷
TAC-KBP participants ⎷ ⎷ ⎷ ⎷ ⎷
Data Sets
46
Dataset # of Mentions Knowledge BaseACE 244 Wikipedia
MSNBC 654 WikipediaAIDA-D 5917 YagoAIDA-T 5616 YagoTAC09 3904 Wikipedia 2008TAC10 2250 Wikipedia 2008TAC10T 1500 Wikipedia 2008TAC11 2250 Wikipedia 2008TAC12 2226 Wikipedia 2008
Mention based F1
Official Eval.
Candidate Generation
• intra-Wikipedia
• CrossWikis(Spitkovsky & Chang, 2012)
• Freebase Search API
47
Entity Type
Candidate Generation
Coreference
Coherence
Mention ExtractionConditional Probability p(e | m) e.g. p( | “Washington”)
Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Aggregate Recall
48
1 3 5 10 20 30 50 100 Inf0.5
0.6
0.7
0.8
0.9
1
k
Recall@k
CrossWikisIntra−WikipediaFreebase Search
49
• Coarse-grained NER(Stanford NER)
• Fine-grained Entity Types (Ling & Weld, 2012) Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Effect of Entity Types
p(e | m) = ∑t p(e,t | m) p(e | m) = ∑t p(e,t | m) p(t | m)
Entity Type Probability
FIGER
• 112 entity types
• multi-label multi-class
50
(Ling & Weld, 2012)
Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Predicted Entity Types
51
F1
50
62.5
75
87.5
100
ACE MSNBC AIDA-D AIDA-T TAC09 TAC10 TAC10T TAC11 TAC12 Overall+NER +FIGER
Overall Performance
52
F1
50
62.5
75
87.5
100
ACE MSNBC AIDA-D AIDA-T TAC09 TAC10 TAC10T TAC11 TAC12Candidate Generation +Entity Types +Coref. +Coherence
Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Overall Performance
53
F1
50
62.5
75
87.5
100
AverageCand +Entity Type +Coref +Coherence
79.078.076.775.0
Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Overall Performance
• AIDA (Hoffart et al. 2011)
• Illinois Wikifier 2.0 (Cheng & Roth, 2013)
54
F1
50
62.5
75
87.5
100
AverageCand +Entity Type +Coref +Coherence AIDA Wikifier
79.6
72.279.078.076.775.0
Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Error Analysis
55
Misc 10%
Specific Labels 14%
Context 33%
Coreference 10%
Types 14%
Metonymy 19%
Conclusion• a modular deterministic system achieves good performance
56
Vinculum
Conclusion• a modular deterministic system achieves good performance• a comprehensive evaluation over nine data sets
56
Vinculum
Conclusion• a modular deterministic system achieves good performance• a comprehensive evaluation over nine data sets
• CrossWikis provides better cond. prob.
56
Vinculum
Conclusion• a modular deterministic system achieves good performance• a comprehensive evaluation over nine data sets
• CrossWikis provides better cond. prob.• Fine-grained entity types are very useful
56
Vinculum
Conclusion• a modular deterministic system achieves good performance• a comprehensive evaluation over nine data sets
• CrossWikis provides better cond. prob.• Fine-grained entity types are very useful• Coreference and Coherence also improve the performance
56
Vinculum
Conclusion• a modular deterministic system achieves good performance• a comprehensive evaluation over nine data sets
• CrossWikis provides better cond. prob.• Fine-grained entity types are very useful• Coreference and Coherence also improve the performance
• http://github.com/xiaoling/vinculum
56
Vinculum
• a modular deterministic system achieves good performance • a comprehensive evaluation over nine data sets
• CrossWikis provides better cond. prob. • Fine-grained entity types are very useful • Coreference and Coherence also improve the performance
• http://github.com/xiaoling/vinculum
Conclusion
57
Thanks!
Questions?
Vinculum
Entity Type
Candidate Generation
Coreference
Coherence
Mention Extraction
Oracle Entity Types
58
F1
50
62.5
75
87.5
100
ACE MSNBC AIDA-D AIDA-T TAC09 TAC10 TAC10T TAC11 TAC12 Overall+NER (Gold) +FIGER (Gold)
Implementation Details
59
Component Implementation
Mention Extraction Stanford NER
Candidate Generation CrossWikis
Entity Type Prediction Fine-grained Entity Types
Coreference Stanford Coreference
Coherence NGD + relational triples
System Comparison
60
VINCULUM AIDA WIKIFIER
Mention Extraction NER NER NER, noun
phrasesCandidate Generation CrossWikis intra-Wikipedia intra-Wikipedia
Entity Types FIGER NER NER
Coreference representative mention - re-rank the
candidates
Coherence NGD, relational NGD NGD, relational
Learning deterministic trained on AIDA trained on Wiki
Error Analysis: Metonymy
• South Africa managed to avoid a fifth successive defeat in 1996 at the hands of the All Blacks …
• Prediction : South Africa
• Label : South Africa national rugby union team
61
Error Analysis: Entity Types
• Instead of Los Angeles International, for example, consider flying into Burbank or John Wayne Airport ...
• Prediction : Burbank, California
• Label : Bob Hope Airport
62
Error Analysis: Coreference
• It is about his mysterious father, Barack Hussein Obama, an imperious if alluring voice gone distant and then missing.
• Prediction : Barack Obama
• Label : Barack Obama Sr.
63
Error Analysis: Context
• Scott Walker removed himself from the race, but Green never really stirred the passions of former Walker supporters, nor did he garner outsized support “outstate”.
• Prediction : Scott Walker (singer)
• Label : Scott Walker (politician)
64