TowardsUnderstandingtheGeometryofKnowledgeGraph

EmbeddingsChandrahas1,AdityaSharma2,Partha Talukdar1,2

1ComputerScienceandAutomation2ComputatationalandDataSciencesIndianInstituteofScience,Bangalore

KnowledgeGraphs(KG)

HasFootballTeam

LionelMessi

FCBarcelona

ArgentinaNationalFootballTeam

1

KnowledgeGraphs(KG)

HasFootballTeam

LionelMessi

FCBarcelona

ArgentinaNationalFootballTeam

Entities

2

KnowledgeGraphs(KG)

HasFootballTeam

LionelMessi

FCBarcelona

ArgentinaNationalFootballTeam

Entities

Relations

3

KnowledgeGraphs(KG)

4

• ExampleKGs

KnowledgeGraphs(KG)

5

• ExampleKGs • Applications

• Search

KnowledgeGraphs(KG)

6

• ExampleKGs • Applications

• Search

• QuestionAnswering

KGEmbeddings

• Representsentitiesandrelationsasvectorsinavectorspace

ℝ𝑑

7

TransE1

1.TranslatingEmbeddings forModelingMulti-relationalData,Bordes etal.

KGEmbeddings

• Representsentitiesandrelationsasvectorsinavectorspace

ℝ𝑑

𝒅

PlaysFor

...

...

...

8

TransE1

1.TranslatingEmbeddings forModelingMulti-relationalData,Bordes etal.NIPS2013.

GeometryofEmbeddings

9

• Arrangementofvectorsinthevectorspace.

GeometryofEmbeddings

10

• Arecentworkby(Mimno andThompson,2017)1 presentedananalysisofthegeometryofwordembeddings andrevealedinterestingresults.

• However,geometricalunderstandingofKGembeddings isverylimited,despitetheirpopularity.

1.Thestrangegeometryofskip-gramwithnegativesampling,Mimno andThompson,EMNLP2017

Problem

• StudythegeometricalbehaviorofKGembeddings learntbydifferentmethods.

• Studytheeffectofvarioushyper-parametersusedduringtrainingonthegeometryofKGembeddings.

• StudythecorrelationbetweenthegeometryandperformanceofKGembeddings.

11

KGEmbeddingMethods

• Learnsd-dimensionalvectorsforentities𝓔 andrelations𝓡 inaKG.

12

KGEmbeddingMethods

• Learnsd-dimensionalvectorsforentities𝓔 andrelations𝓡 inaKG.

• Ascorefunction𝛔 :𝓔⨉𝓡⨉𝓔→ℝ distinguishescorrecttriples𝑇 +

fromincorrecttriples𝑇 −.Forexample,𝛔(Messi,plays-for-team,Barcelona)>𝛔(Messi,plays-for-team,Liverpool)

13

KGEmbeddingMethods

• Learnsd-dimensionalvectorsforentities𝓔 andrelations𝓡 inaKG.

• Ascorefunction𝛔 :𝓔⨉𝓡⨉𝓔→ℝ distinguishescorrecttriples𝑇 +

fromincorrecttriples𝑇 −.Forexample,𝛔(Messi,plays-for-team,Barcelona)>𝛔(Messi,plays-for-team,Liverpool)

• Alossfunction𝐿(𝑇+, 𝑇−) isusedfortrainingtheembeddings (usuallylogisticlossormargin-basedrankingloss).

14

KGEmbeddingMethods

15

KGEmbeddingMethods

• AdditiveMethods

• MultiplicativeMethods

• NeuralMethods

16

KGEmbeddingMethods

17☉ Entry-wiseproduct ★ Circularcorrelation

GeometricalMetrics

• AverageVectorLength

18

GeometricalMetrics

• AverageVectorLength

• AlignmenttoMean

19

GeometricalMetrics

• Conicity

20

GeometricalMetrics

• Conicity

• VectorSpread

21

GeometryofEmbeddings

22

HighConicity LowConicity

Experiments

• WestudytheeffectoffollowingfactorsonthegeometryofKGEmbeddings• Typeofmethod(AdditiveorMultiplicative)• NumberofNegativeSamples• DimensionofVectorSpace

• Wealsostudythecorrelationofperformanceandgeometry.

• Forexperiments,weusedFB15kdataset.

23

AdditivevsMultiplicative(EntityVectors)

24

Additive

Multip

licative

AdditivevsMultiplicative(RelationVectors)

25

Additive

Multip

licative

26

ModelType Conicity VectorSpreadAdditive Low High

Multiplicative High Low

AdditivevsMultiplicative

Effectof#NegativeSamples(EntityVectors)

27

Effectof#NegativeSamples(EntityVectors)

28

Additive

Effectof#NegativeSamples(EntityVectors)

29

Multiplicative

Effectof#NegativeSamples(EntityVectors)

30

AdditiveNochange

Effectof#NegativeSamples(EntityVectors)

31

AdditiveNochange

MultiplicativeConicity Increases

Effectof#NegativeSamples(EntityVectors)

32

AdditiveNochange

Effectof#NegativeSamples(EntityVectors)

33

AdditiveNochange

MultiplicativeAVLdecreases

Effectof#NegativeSamples

34

ModelType Vector Type Conicity AVL

AdditiveEntity NoChange NoChangeRelation NoChange NoChange

MultiplicativeEntity Increases DecreasesRelation Decreases NoChange

exceptHolE

SGNS(Word2Vec1)asMultiplicativeModel

• Similarobservationwasmadeby(Mimno andThompson,2017)2 forSGNSbasedwordembeddings wherehigher#negativesresultedinhigherconicity.

• Word2Vec1 maximizeswordandcontextvectordotproductforpositiveword-contextpairs.

• Thisbehaviorisconsistentwiththatofmultiplicativemodels.

351.Distributedrepresentationsofwordsandphrasesandtheircompositionality,Mikolov etal.NIPS20132.Thestrangegeometryofskip-gramwithnegativesampling,Mimno andThompson,EMNLP2017

Effectof#Dimensions(EntityVectors)

36

AdditiveNochange

Effectof#Dimensions(EntityVectors)

37

AdditiveNochange

MultiplicativeConicity decreases

Effectof#Dimensions(EntityVectors)

38

AdditiveNochange

Effectof#Dimensions(EntityVectors)

39

AdditiveNochange

MultiplicativeAVLIncreases

Effectof#Dimensions

40

ModelType Vector Type Conicity AVL

AdditiveEntity NoChange NoChangeRelation NoChange NoChange

MultiplicativeEntity Decreases IncreasesRelation Decreases Increases

Correlationb/wGeometryandPerformance

41

Correlationb/wGeometryandPerformance

42

Additive

Correlationb/wGeometryandPerformance

43

HolE performsbadwithhigher

negatives

Correlationb/wGeometryandPerformance

44

NegativeSlope-NegativeCorrelation

Correlationb/wGeometryandPerformance

45

NegativeSlope-NegativeCorrelation

HigherNegatives-HigherSlopeMagnitude

Correlationb/wGeometryandPerformance

46

Correlationb/wGeometryandPerformance

47

AdditiveandHolE

Correlationb/wGeometryandPerformance

48

PositiveSlope-PositiveCorrelation

Correlationb/wGeometryandPerformance

49

PositiveSlope-PositiveCorrelation

HigherNegatives-HigherSlopeMagnitude

Correlationb/wGeometryandPerformance

50

• Additive:Nocorrelationbetweengeometryandperformance.

• Multiplicative:Forfixednumberofnegativesamples,• Conicity hasnegativecorrelationwithperformance• AVLhaspositivecorrelationwithperformance

ConclusionandFutureWorks

• WeinitiatedthestudyofgeometricalbehaviorofKGembeddings andpresentedvariousinsights.

• Explorewhetherotherentity/relationfeatures(eg entitycategory)haveanycorrelationwithgeometry.

• ExploreothergeometricalmetricswhichhavebettercorrelationwithperformanceanduseitforlearningbetterKGembeddings.

51

Acknowledgements

• WethankGoogleforthetravelgrantforattendingACL2018.

• WethankMHRDIndia,Intel,Intuit,GoogleandAccentureforsupportingourwork.

• Wethankthereviewersfortheirconstructivecomments.

52

Thankyou

53

Effectof#NegativeSamples(RelationVectors)

54

AdditiveNochange

Effectof#NegativeSamples(RelationVectors)

55

AdditiveNochange

MultiplicativeConicity decreases

Effectof#NegativeSamples(RelationVectors)

56

AdditiveNochange

Effectof#NegativeSamples(RelationVectors)

57

AdditiveNochange

MultiplicativeNochangeexcept

HolE

Effectof#Dimensions(RelationVectors)

58

AdditiveNochange

Effectof#Dimensions(RelationVectors)

59

AdditiveNochange

MultiplicativeConicity decreases

Effectof#Dimensions(RelationVectors)

60

AdditiveNochange

Effectof#Dimensions(RelationVectors)

61

AdditiveNochange

MultiplicativeAVLIncreases