Fang Xu- Enriching content with Knowledge Base by Search Keywords and Wikidata

1

© Searchmetrics. All rights reserved. Do not distribute without permission.

Enriching content with Knowledge Base by Search Keywords and Wikidata

Fang [email protected]@allxufang

2


Data Science@Searchmetrics

Data driven search and content optimization marketing

• Learning from keywords

• Content optimization

• Data visualization

3


Looooots of Data

• 120 Million Domains

• 600 Million Keywords

• 120 Billion Links

• 25,000 Billion Social Signals

• 25 PB raw data

4


Authors submit content üRate the content’s effectiveness ü Feedback to optimize and enrich it

Content Production in Real-time

5


Beyond keywords

• Keyword • Typos• Ambiguous• Sparse

• Entity • Augmented with

metadata• Relations among entities

6


Q64

Entity

7


8

© Searchmetrics. All rights reserved. Do not distribute without permission.http://brendangriffen.com/blog/gow-programming-languages

Knowledge Base (KB)

9


20012012

20142008

Knowledge vaults

2012

2005

KB Timeline

10


• Free collaborative KB• Continuous evolution• Open multilingual Data• mapping to other KBs

Why Wikidata

11


Link content to KB• Entity Linking -- free text to entities

• Blog posts • Tweets • Keywords• User-generated Contents

• Entities from a knowledge base• Wikipedia• Wikidata• Domain-specific KBs

12


Image from Milne and Witten (2008b). Learning to Link with Wikipedia. In CIKM 2008

Entity Linking

13


• Identify important keywords to link in the text

• Link to right entity

Main Problems

14


Dictionary of keywords to KB entitiesSearch keyword mentions in text

15


Keyword to wiki uris in top SERP

16


Not all keywords are useful

Keyword Cleaning:

• Navigational or factual words

• Non-frequent words

• Non-latin letters

17


Keyword Filtering: • Starting or ending tokens • Stopwords • Part-of-speech tags• Wikipedia popularity:

• popular wiki uris for one keyword• Search popularity:

• popular keywords for one wiki uri

Not all keywords are useful

18


Search Popularity Filtering Keyword Search Popularity (Volume)

germany 268583

germany facts 4291

germany article 24

german encyclopedia 23

germany encyclopedia 19

germany t 18

ger many 16

19


parse wikidata dump & extract entities as json

Entity data{ entity: "Berlin", Freebase Id: "/m/0156q", OpenStreetMap Relation identifier: 62422, alias: ["Berlin, Germany"], capital of: [ "Germany", "Kingdom of Prussia", "Weimar Republic", "Brandenburg-Prussia", "Free State of Prussia", ... ], contains administrative territorial entity: [ "Mitte", "Friedrichshain-Kreuzberg", "Pankow", "Charlottenburg-Wilmersdorf", "Spandau", "Steglitz-Zehlendorf", "Tempelhof-Schöneberg", "Neukölln", "Treptow-Köpenick", ... ], coordinate location: [ { altitude: null, latitude: 52.516666666667, longitude: 13.383333333333, precision: 0.016666666666667 } ], country: "Germany", ... ... }

20


Link to the right Wikipedia entityWord Sense Disambiguation

21


d

Tree 92.82%

Tree (graph theory) 2.94%

Tree (data structure) 2.57%

Tree (set theory) 0.15%

Phylogenetic tree 0.07%

Christmas tree 0.07%

Binary tree 0.04%

Family tree 0.04%

… ...

Link to Most Common Entities

e ew

ew

LL

i

,

,

ew entity , text surface with LinksofNumber

Entity Wikipedia Commnoness

(Milne and Witten 2008b)tree

22


https://en.wikipedia.org/wiki/Tree_data_structure

https://en.wikipedia.org/wiki/Tree

Disambiguation

23


Disambiguation using context

24


• Build a Word2Vec model for Wikiepdia entity

• Calculate Word2Vec similarity to contextual entities

contextcontext

TreestructureTree_data_ )(similarity)(similarity

Entity Disambiguation

25


Relatedness between Entities

26

© Searchmetrics. All rights reserved. Do not distribute without permission.Image from Milne and Witten (2008a). An Effective, Low-Cost Measure of Semantic Relatedness Obtained from Wikipedia Links

Entity Relatedness

27


• Jaccard similarity

• Word2Vec similarity of entity to context

eeee

and entity tolinks of Union and entity tolinks of onIntersecti

Relatedness Score

28


Wikipedia Data Parsing

29


Wikipedia Dump

'''Berlin''' is the [[Capital city|capital]] of [[Germany]] and one of its 16 [[states of Germany|states]]. With a population of approximately 3.5 million people,<ref name="Population" /> Berlin is the second [[Largest cities of the European Union by population within city limits|most populous city proper]] and the seventh [[Largest urban areas of the European Union|most populous urban area]] in the [[European Union]].

30


Wikipedia Article as Json

31


Word2Vector Training• Collection of plain article text

... ...can4linux ||open_source|| ||controller_area_network|| ||linux_kernel|| ||device_driver|| development started 1990s philips 82c200 controller stand chip 1995 version created bus linux laboratory automation project linux lab project ||freie_universität_berlin|| nxp sja1000 successor supported controller philips 82c200 intel 82527 development powerful ||microcontroller||s integrated controllers capable ... ...

32


Linking vectors• Pairs of uri, annotations

outlink vector [Capital_City, Germany , States_of_Germany, European_Union,Spree, Havel, Berlin-Brandenburg_Metropolitan_Region, ... ... ]

inlink vector [Germany, Prussia, Berlin_Wall, Albert_Einstein, Kosmos_(Berlin), Berlin_International_Film_Festival, .. .. ]

33


Wikipedia Popularity• Aggregation of annotations

Surface text Wiki entity Popularity

United States United_States 174338

World War II World_War_II 106483

India India 95966

France France 94666

American United_States 85976

Iran Iran 83249

Australia Australia 76655

Germany Germany 76384

34


Overall SystemKeywordDatabase

KeywordProcessing

Parser

UserContent

KeywordMatching

Disam-biguation

Relatedness calculation Result

Wikipedia Popularity

Entity Linking API

WikiParser

W2VModel

WikiLinksKeyword

to KB entities

35


• https://github.com/piskvorky/gensim• https://github.com/jodaiber/Annotated-WikiExtractor• https://dumps.wikimedia.org/• https://dumps.wikimedia.org/wikidatawiki/entities/

Resources

36


Thank you

37


Questions?

[email protected]

We are hiring

Data & Analytics

Fang Xu- Enriching content with Knowledge Base by Search Keywords and Wikidata