Research ArticleLink Trustworthiness Evaluation over Multiple HeterogeneousInformation Networks

Meng Wang 12 Xu Qin 13 Wei Jiang 45 Chunshu Li 1 and Guilin Qi 12

1School of Computer Science and Engineering Southeast University Nanjing 210096 China2Key Laboratory of Computer Network and Information Integration (Southeast University) Ministry of EducationNanjing 211189 China3Southeast University-Monash University Joint Research Institute Suzhou 215123 China4Laboratory for Complex Systems Simulation Institute of Systems Engineering AMS Beijing 100083 China5North China Institute of Computing Technology Beijing 100083 China

Correspondence should be addressed to Meng Wang mengwangseueducn

Received 5 December 2020 Revised 16 February 2021 Accepted 25 February 2021 Published 8 March 2021

Academic Editor Jianxin Li

Copyright copy 2021 Meng Wang et al is is an open access article distributed under the Creative Commons Attribution Licensewhich permits unrestricted use distribution and reproduction in any medium provided the original work is properly cited

Link trustworthiness evaluation is a crucial task for information networks to evaluate the probability of a link being true in aheterogeneous information network (HIN) is task can significantly influence the effectiveness of downstream analysisHowever the performance of existing evaluation methods is limited as they can only utilize incomplete or one-sided informationfrom a single HIN To address this problem we propose a novel multi-HIN link trustworthiness evaluation model that leveragesinformation across multiple related HINs to accomplish link trustworthiness evaluation tasks inherently and efficiently Wepresent an effective method to evaluate and select informative pairs across HINs and an integrated training procedure to balanceinner-HIN and inter-HIN trustworthiness Experiments on a real-world dataset demonstrate that our proposed model out-performs baseline methods and achieves the best accuracy and F1-score in downstream tasks of HINs

1 Introduction

Heterogeneous information network (HIN) is an efficienttechnique to model complicated real-world informationthrough a network structure in which network nodes in-dicate multi-typed real-world nodes and edges indicaterelationships between nodes For example MovieLens [1]and IMDB are typical examples of HIN in movie domainsmodeling relationships among movies actors directors etcContaining richer structural information HINs bring aboutplenty of opportunities for data mining in complicatedapplications However erroneous information commonlyexists in HINs Because HINs are mainly constructed byinformation retrieving techniques nowadays the unreli-ability of web sources and the biases of techniques [2] maylead to subtle errors Downstream analysis tasks employingthese biased data will cause accumulated errors in the ap-plications erefore the task of evaluating the

trustworthiness of HINs should be stressed to maintain thereliability of HIN data and improve the accuracy ofdownstream tasks [3] Specifically let t (Ns Ep No)G bea link in the HIN G starting from the subject node Ns andending with the objective node No with the correspondingedge Epus worthiness evaluation tasks aim at computinga probabilistic trustworthiness T(wt

G) isin [0 1] for every linkt inG Here T(wt

G) 1 indicates link t is completely correctwhile T(wt

G) 0 when t is totally incorrectCurrent HIN trustworthiness evaluation methods are

mainly based on knowledge base (KB) methods [4] eyfocus on inferring a linkrsquos trustworthiness in a single HINwhere the trustworthiness metric should evaluate how re-liable this link is In addition to basic characteristics likenode types [5] attributed values [6] and confidence ofextracted source trustworthiness is typically evaluated fromother perspectives Network embedding methods [7] con-sider whether nodes and edges satisfy in the network

HindawiComplexityVolume 2021 Article ID 6615179 11 pageshttpsdoiorg10115520216615179

topology For instance R-GCNmodels relational data in theGCN framework On the other hand edge learning methods[3 8ndash10] measure the acceptability of a link composed of anedge and two nodes corresponding to it For exampleTransE [9] considers whether the embedding nodes andedges of multi-relational data converge in low-dimensionalvector spaces

Almost all the previous trustworthiness evaluationmethods are focusing on only one HIN however the in-formation in a single HIN tends to be uncompleted or one-sided Notice that multiple HINs may capture informationfrom the same resources and it is crucial to accomplishtrustworthiness evaluation with the help of multiple HINsthat model the same or similar domains which is shown inFigure 1 For example HIN 1 has a description of LeonardoDiCaprio starring in e Great Gatsby where no connectionbetween actors are shown in this HIN while HIN 2 has linksconnecting actors for example Leonardo DiCaprio hascooperated with MatthewMcConaughey us we can infermore information about Leonardo DiCapriorsquos starred filmsDifferent HINs usually have different emphasis even on thesame domain For example two HINs constructed based onIMDB and YouTube respectively are different in manyaspects even when they all model the information of videosIn one HIN some links have ample evidence while relatedlinks in another HIN only have little evidence such that it isdifficult to evaluate trustworthiness in single HIN us inthe multi-HIN situation trustworthiness evaluation can bedifferent as links in one HIN can influence their counterpartsin another one ese related links from different HINs arecalled interactive links and shared information of interac-tive links should improve the effectiveness of evaluation

However evaluating trustworthiness with multiple HINsis a nontrivial task because of several challenges

(1) Aligning Interactivity e first challenge is how todetermine the interactivity If interactive links areonly based on the same nodes or edges the influencescope will be restricted and the error introduced bythe alignment procedure will deduce the evaluationresult

(2) Evaluating Influence across Multiple HINs esecond challenge is to design an influential methodacross HINs Having determined the interactivedegree between a pair of interactive links how toupdate this pair should be carefully processed

(3) Balancing Inner-HIN and Inter-HIN TrustworthinessIt is difficult to balance the inner-HIN and inter-HINtrustworthiness It will be redundant and laboriousto initially evaluate a linkrsquos trustworthiness in everysingle graph and then evaluate it again in the co-embedding space

ese three difficulties make multi-HIN trustworthinessproblems different from and much more difficult thantraditional single HIN settings

To address challenges in multi-HIN trustworthinessevaluation tasks we propose a multi-HIN trustworthinessevaluation (multi-HITE) model aiming at leveraging inner-

HIN characteristics and inter-HIN information in the co-embedding space to evaluate link trustworthiness reeaforementioned challenges of multi-HIN tasks are solvedaccordingly through the following

(1) Multiple Interactive Links Effect Functions We de-sign three kinds of functions to evaluate the inter-active links effect (ILE) across different HINs eseILE functions evaluate the interactivity from literallevel semantic level and graph level respectivelyAltogether these functions help our model aligninteractivity across multiple HINs

(2) Iterative Procedure to Update Trustworthiness In-fluential information across HINs is updated iter-atively in the training process through threshold-based influence function

(3) Integrated Learning with Multi-HINs e lossfunction of our model measures the inner-HIN andinter-HIN trustworthiness of multiple HINs at thesame time Additionally the trade-off between thesetwo parts is controlled by hyperparameters

Specifically this paper has made the followingcontributions

(1) To our best knowledge our method is the first oneconsidering evaluating the trustworthiness of HINlinks in multi-HIN settings

(2) We propose a novel multi-HIN trustworthinessevaluation method combining inner-HIN and inter-HIN characteristics in a co-embedding space

(3) We conduct experiments on a real-world dataset thatproves the effectiveness of our model Based ondifferent kinds of interactive metrics we conductseveral experiments using a real-world datasetcompared with modified single-HIN evaluationmethods on the tasks of error detection and linkprediction

Notations A HIN is denoted by G and t (Ns Ep No)G

represents one link in G A set of multiple HINs is repre-sented as S G1 G2 GK1113864 1113865 Let Ns No and Ep denotesubjective node objective node and node relations in theoriginal spaces and let Ns No and Ep respectively denotetheir representations in the embedding space Given a vectorx isin Rp x

x21 + x2

2 + middot middot middot + x2p

1113969denotes its l2-norm and

given a matrix X isin Rm times n its l2-norm is defined in anelementwise manner as X

11139361le ilem 1le jle n(Xij)

21113969

2 Materials and Methods

21 Trustworthiness in a SingleNetwork Due to the concernsabout the influence of data quality on downstream tasks linktrustworthiness evaluation in an HIN and of a HIN sourcehas quickly gained popularity among researchers [11] Mostexisting methods are based on knowledge bases Earlymethods are rule-based or statistical For example Ma et al[12] learned disjointness axiom via association rule mining

2 Complexity

to dynamically update rule patterns A statistical example isSDValidate [5] which detects noise in a HIN via the sta-tistical distribution of the properties and types

Another popular approach is to integrate the KB rep-resentation learning into the models to evaluate the trust-worthiness in embedding space Typical examples are thetranslation models including TransE [9] and its variants likeTransH [8] and TransR [13] In Trans-series model if theenergy function of a link in the embedding space

A wtG1113872 1113873 Ns + Ep minus No

(1)

scores larger than a desired value it is assumed that this linkhas little local context proof in original HIN space iehaving low trustworthiness

In addition to translation models CKRL [11] exploits thestructure information by defining local link confidences andglobal path confidence to detect noises in HINsis methodexploits the structure information of the network KGTtm[10] is a multi-level knowledge graph link trustworthinessevaluation method It separates evaluation into nodes edgesand graph levels combining the internal semantic infor-mation of links and the global inference information of theKB However none of these methods consider utilizingother HINs that model similar domains erefore theirperformance is limited as the information from a single HINtends to be incomplete or one-sided

22LearningacrossMultipleHINs In the multi-HIN settingas nodes and edges in different HINs are extracted fromdifferent sources network alignment methods are proposedto find corresponding nodes across multiple networksTypically network alignment can be categorized into localalignment and global alignment Local alignment aims tofind similar local regions across networks [14] Globalalignment focuses on exploiting the global topologicalconsistency such as in iNEAT [15] and CAlign [16]

Conventional KB methods briefly use handcraftedldquosameAsrdquo link literal information and attribute features to

generate linkages between different HINs [17 18] Recent worksfocused on representation learning across multi-relationalgraphs noticing that aligned nodes should be similar in space[19] MTransE [20] tries to merge graphs and uses relationallearning methods to discover aligned nodes It combines threedifferent transformation methods to learn embedding forsupporting multi-lingual learning LinkNbed [21] learns latentrepresentations of nodes and edges in multiple graphs where aunified graph embedding is constructed avoiding the biascaused by the transformation between different HINembeddings

We naturally consider utilizing similar methods in nodealignment in the multi-HIN trustworthiness evaluationproblem However these methods of the node alignmentproblem lack global structure inference information whichprevents these methods from being directly generalized tomulti-HIN trustworthiness evaluation

23 General Structure of Trustworthiness Evaluation Toaddress the limitations of previous evaluation methods wepropose a novel method inherently considering inner-HINand inter-HIN effects e trustworthiness evaluation inmulti-HIN settings can be separated into two parts egeneral idea is shown in Figure 2

Firstly given the overall trustworthiness VGiof the i-th

HIN evaluate trustworthiness score of all links tj isin Gi becausein the multi-HIN situation different HINs have different re-liabilities e trustworthiness of a link in graph Gi is not onlybased on the inner-HIN trustworthiness A(w

tj

Gi) calculated by

inner characteristics of Gi but also dependent on the HINsource confidence VGi

is influence can be denoted as

T wtj

Gi1113874 1113875 VGi

A wtj

Gi1113874 1113875 (2)

Initially all global confidence VGiis assigned to 1 us

according to the representations for head nodes tail nodesand edges which are generated by inner characteristics andinitial source trustworthiness the trustworthiness of linkscan be calculated for every HIN

LeonardoDiCaprio

Inception

ChristopherNolan

Thegreat

gatsby

LeonardoDiCaprio

AnneHathaway

Interstellar

HIN 1 HIN 2

MatthewMcConaughey

Related node

Figure 1 General situation of multi-HIN problems

Complexity 3

en in the second step we use link trustworthinessscore to conversely update VGi

based on T(wtj

Gi) In the

opposite view of the multi-HIN problem we assume that thetrustworthiness of a HIN source is determined by all linksrsquotrustworthiness in the source and is computed via

VGi

1nGi

1113944tjisinGi

T wtj

Gi1113874 1113875 (3)

Here nGimeans the total number of links in graph Gi

which acts as a normalization factor At each iteration VGiis

firstly fixed to calculate single linkrsquos trustworthinessaccording to equation (2) followed by fixing T(w

tj

Gi) to

calculate VGi according to equation (3) After the iterativetraining has converged a steady trustworthiness evaluationof HIN source and link can be obtained

e above procedure is commonly used in no-mutual-in-fluence multi-HIN settings As for mutual-influence multi-HINcases the influence between differentHINs needs to bemodeledwhere our method models this influence by the trustworthinessinfluences between interactive links across HINs is will bediscussed in ldquoInter-HIN Evaluationrdquo Section Specifically dif-ferent HINs are placed on the co-embedding space to representthe influence between the sources If HINs are separatelyevaluated in their own space this will cause redundant proce-dure in the transformation betweenHIN spaces whichmake themodel difficult to train en having obtained inner scores forlinks in both HINs interactive influence will update corre-sponding linksrsquo scores e final step is to update the sourcetrustworthiness is process will be iteratively conducted AllHIN sourcesrsquo trustworthiness values are initially set to 1 and afteriterative influencing between sources and links the final scoresof links and sources will be obtained

Because the process of trustworthiness evaluation is thesame in all the HINs without loss of generality we use G andt to denote an arbitrary HIN and a certain edge in itWithoutexplicit statements we drop the subscripts i and j forsimplicity in the rest of this paper

e following sections will be organized as follows wedetail our modelrsquos trustworthiness evaluation in a single HINin Subsection 24 the details of the multi-HIN mutual

influence method in Subsection 25 and the training methodtailed for multi-HIN evaluation in Subsection 26

24 Inner-HIN Evaluation We start by considering how tocompute the link trustworthiness score with inner-HINcharacteristics Based on the current single HIN trust-worthiness evaluation methods [22] HINrsquos inner charac-teristics including semantic and graphic information arestrong proofs to evaluate the reliability of links In ourmodel we utilize four inner-HIN characteristics includingnode attributes node-type constraints contextual em-beddings and graphic energy transmission information toevaluate links

241 Node-Attribute Information Node attributes are thebasic and effective information to assess whether a node isconsistent with its value Here we use doc2vec [23] to embedattribute values and then integrate the value with its one-hotattribute key index to represent hidden information ofnodes If a node has several attributes the hidden attributeinformation will be averaged

242 Node-Type Constraint For a certain edge the nodesappearing near it normally have prevalent and uniformtypes us for each edge we can summarize the node typesthat emerge around it en the node-type information andedge embedding information are integrated into the hiddenlayer

243 Contextual Embedding Information Trans-seriesmodels embed nodes and edges in low dimension repre-sentation spaceese embeddings contain global contextualinformation where outliers can be found based on the in-consistency However we are not directly using the energyfunction to evaluate the consistency Here we utilize thefully connected layer to unite other inner-HIN character-istics with embeddings to get a hidden layerrsquos representation

Node attributes

Node types

Graphicattributes

Nodeembedding

Edgeembedding

Head hiddenrepresentation

Edge hiddenrepresentation

Tail hiddenrepresentation

Trustworthinessscore

HINtrustworthiness

updation

Interactiveinfluence

HIN 1

HIN 2

Innercharacteristics

Interactive links from HIN 1

Interactive links from HIN 2

Co-embeddingspace

Figure 2 General structure of multi-HIN trustworthiness evaluation method

4 Complexity

244 Graph Energy Transferring Information HIN is aspecial heterogeneous network where basic network anal-ysis techniques can be utilized If a link has strong con-nectivity or other favourable graphic features it alsoindicates sufficient proofs in the HIN Here we mainlyconcentrate on the energy flow between the head node andthe tail node In this situation we can construct a singlegraph for each node in the HIN where the node is the centralnode All nodes associated with the central node are takeninto consideration e energy transferring can be modeledwith page-rank-like method [11 24] However iteratingthrough a graph will have a long tail phenomenon where themajority of nodes have low energy Hence we add severalfeatures like in-degree and out-degree into consideration as

g Ns No( 1113857 ε timese Ns No( 1113857

e Ns( 1113857+(1 minus ε) times fNsNo

din dout( 1113857

(4)

Here g(Ns No) is the combined energy transmittedbetween embedded nodes Ns and No e(Ns) represents theenergy thatNs holds in the graph and e(Ns No) denotes theenergy flowing from Ns to No fNsNo

(din dout) is the energyfunction determined by the in-degree and out-degree of Ns

and No Besides ϵ is a hyperparameter balancing the page-rank-like energy and degree-determined function

After obtaining the above four different features eachfeature is transmitted to the hidden dimension by a separatelinear forward layer followed by the combination of allfeatures of the same node or link as shown in Figure 2 Afterobtaining the hidden representation z isin Rn (n is the size ofhidden states) of the head node tail node and edge we usethe following function to model the trustworthiness

T wtG1113872 1113873 VG times σ z

⊤p zs ⊙ zo( 11138571113872 1113873 (5)

in which σ(middot) is the activation function of the neural networkand ⊙ denotes the elementwise multiplication e scorefunction is related to the score of the linkrsquos located HIN so itneeds to multiply the located HINrsquos score VG Part of thisfunction is initially adopted by DistMult [25]

With all these we can calculate traditional trans-mission loss in every single HIN of S is loss is used tomodel local information of a HIN For a specific link t

shown in the original dataset we call it a positive link enegative links set c(G) is generated by replacing the headand tail node of t with other nodes so that negative linksdo not appear in the original dataset Specifically for aHIN G isin S let T(wt

G) and T(wtprimeG) be the positive score and

negative score respectively for the positive link t andnegative link tprime in this HIN and the relational loss formultiple HINs is

Lrel 1113944GisinS

1113944tisinG

1113944

tprimeisinc(G)

max 0 c minus T wtG1113872 1113873 + T w

tprimeG1113874 11138751113874 1113875)⎛⎝ (6)

Here c is the margin between positive links and negativelinks and c(G) is a set of corrupted links tprime corresponding tolinks t in the HIN G

25 Inter-HINEvaluation In multi-HIN settings inter-HINinfluence is another crucial component of the evaluationerefore we further include inter-HIN influence into theloss function for trustworthiness evaluation If a link(Ns Ep No)G in one HIN G has influence on a link(Nsprime Epprime Noprime)Gprime in another HIN Gprime we call these two linksinteractive links e way how interactive links influenceeach other in different HINs is called Interactive Links Effect(ILE) We use h(middot) to denote ILE and propose three differentkinds of ILEs that are utilized in our method for modelinginteractivity from three levels literal level semantic leveland graph level

251 Value ILE If two nodes refer to the same real-worldnode they should share similar attributes [26] Based on thisassumption from the literal level the value ILE can bedefined to evaluate influence for nodes of a pair of interactivelinks Let p(N) denote the attribute value set of the node N

and c(N1 N2) represent the common attributes shared bynodes N1 and N2 in the embedding space value ILE iscomputed via

hvalue Ns Ep No1113872 1113873G Nsprime Epprime Noprime1113872 1113873

Gprime1113872 1113873

c Ns Nsprime( 1113857

11138681113868111386811138681113868111386811138681113868

p Ns( 11138571113868111386811138681113868

1113868111386811138681113868 + p Nsprime( 11138571113868111386811138681113868

1113868111386811138681113868 minus c Ns Nsprime( 11138571113868111386811138681113868

1113868111386811138681113868

+c No Noprime( 1113857

11138681113868111386811138681113868111386811138681113868

p No( 11138571113868111386811138681113868

1113868111386811138681113868 + p Noprime( 11138571113868111386811138681113868

1113868111386811138681113868 minus c No Noprime( 11138571113868111386811138681113868

1113868111386811138681113868

(7)

in which | middot | is the number of elements in a set

252 Alignment ILE If two links have aligned nodes oredges they should be interactive Additionally alignednodes or edges are close in embedding space [19] us wecan assume that if two links are interactive at least one of thenodes or edges should be close in the embedding spaceBesides only one aligned part is not sufficient as many linksdescribe the same node from a different perspective like ldquoYaoMing-wife-Ye Lirdquo and ldquoYao Ming- teammate-TracyMcGradyrdquo us we add separate distances of co-spaceembedding of nodes and edges together and restrict thedistance of interactive links of this embedding distance todetermine whether two links are interactive or not Withthis we define the alignment ILE as

halignment Ns Ep No1113872 1113873G Nsprime Epprime Noprime1113872 1113873

Gprime1113872 1113873

Ns minus Nsprime

2 + No minus Noprime

2 + Ep minus Epprime

2

(8)

253 Neighbour ILE According to the semantic similaritywork [27] if nodes share similar surroundings there exists apossibility that they have semantic similarity in other wordsthey are interactive us we use neighbouring nodes of thehead and tail nodes to evaluate the ILE Let n(N) denote theaverage embedding for neighbouring nodes of node N thenwe have

Complexity 5

hneighbour Ns Ep No1113872 1113873G Nsprime Epprime Noprime1113872 1113873

Gprime1113872 1113873

n Ns( 1113857 minus n Nsprime( 1113857

2 + n No( 1113857 minus n Noprime( 1113857

2(9)

With all three ILE functions we can inherently use themto determine if two links from two different HINs haveinfluence on each other Specifically we design a matrix I torepresent the interaction of two HINs For a certain linki isin G and another link j isin Gprime their interactive status isdefined as

Iij 1 hvalue(i j)lt θvalue( 1113857 + 1 halignment(i j)lt θalignment1113872 1113873

+ 1 hneighbour(i j)lt θneighbour1113872 1113873

(10)

where 1(c) is the indication function with 1(c) 1 ifstatement c is true otherwise 1(c) 0 ree ILE thresholdsincluding θvalue θalignment and θneighbour are playing the rolesof hyperparameters

Intuitively the trustworthiness of two interactive linksshould be close to each other However there exist caseswhere two links are interactive but one has large trust-worthiness while that of another link is low us trust-worthiness scores of two links should be deductedSpecifically for i isin G and j isin Gprime we design the influencebetween links i and j as R(i j) 0 if Iij 0 otherwise

R(i j) T w

iG1113872 1113873 minus T w

j

G1113872 111387311138681113868111386811138681113868

11138681113868111386811138681113868 if T wiG1113872 1113873 minus T w

j

G1113872 111387311138681113868111386811138681113868

11138681113868111386811138681113868 le ρ

T wiG1113872 1113873 + T w

j

G1113872 111387311138681113868111386811138681113868

11138681113868111386811138681113868 if T wiG1113872 1113873 minus T w

j

G1113872 111387311138681113868111386811138681113868

11138681113868111386811138681113868 gt ρ

⎧⎪⎨

⎪⎩

(11)

Combining all the inter-HIN influences of interactivelinks we introduce the interactive loss

Lint 1113944(ij)isinIn

R(i j)(12)

with a set of interactive links In

26 Model Training in Multi-HIN Settings To obtain thetrustworthiness both in single-HIN and multi-HIN per-spectives we introduce the multi-task objective functionwhich models the contextual information of local and in-fluential information We integrate relational loss with in-teractive influential loss by using a hyperparameter β as thetrade-off and obtain

L β Lrel +(1 minus β)Lint (13)

e training process of our model can be separated intotwo main procedures

In the first phase we minimize the loss function with aneural network We put inner graph characteristics into theneural network including node attributes node-type con-straints node and edge embeddings and graph energytransferring information en we separately combinenode-related features into hidden representations of nodesand combine node-type constraints with edge embeddingsinto hidden representations of edges e detailed structure

of this neural network is shown in Figure 3 In the networke1(middot) is the node energy function for a certain node e2(middot) isthe node embedding e3(middot) denotes the attribute embeddinge4(middot) represents the edge embedding and e5(middot) is the typeembedding of a node Moreover weight matricesWe1h We2h We3h We4h andWe5h project different featuresto its hidden representations Function r(middot) projects em-beddings in the same space Besides Wer Wnr and War

combine node-related features to generate hidden repre-sentations of nodes while Wlr and Wtr combine edge-relatedfeatures to generate hidden representations of edges Inaddition we further attach an extra regularizer to control thesize of all the parameters Let Ω We1h We2h1113864

We3h We4h We5h Wer Wnr War Wlr Wtr denote a set ofall the weight parameters in the neural network and theregularized loss function becomes

L β Lrel +(1 minus β)Lint + 1113944W isinΩ

W2 (14)

After evaluating the score for all links in multiple HINswe can iteratively use equations (2) and (3) to update thetrustworthiness of HIN and each link until results convergee pseudocode of our method is shown in Algorithm 1

3 Results and Discussion

In this section we implement comprehensive experiments totest the performance of ourmodel from various perspectives

31 Dataset We choose to use two HIN alignment datasetsDBP-WD and DBP-YG [28] in our experiments e firstdataset is sampled from DBpedia and Wikidata (ie twoHINs) and contains 100 thousand aligned nodes while thesecond dataset is extracted from DBpedia and YAGO withsame 100 thousand aligned nodes Information about nodetypes is attached to the data according to DBpedia type in-formation Tables 1 and 2 provide statistics of these datasets inthe experimentWe split this dataset into training testing andvalidation set with the ratio as 10 1 1 Besides DBP-WDdataset is used in all experiments while DBP-YG is only usedin Experiment III DBpedia in DBP-WD is noted as DBpediawhile DBpedia in DBP-YG is noted as DBpedia2

Furthermore we generate some negative links in thedataset to simulate noisy data with erroneous links Spe-cifically for one negative link we replace the head or tailnode of a positive link so that it does not appear in theoriginal dataset Other characteristics including node at-tributes node types and neighbouring nodes remain thesame with the replaced node To obtain a different level ofnoise similar to the simulation implemented in [11]multiple noisy datasets are simulated with the percentage ofreplaced negative links of 10 20 and 40 Notice thatthe noises are only added to the training set

32 Baseline Methods As there is no trustworthiness eval-uation method previously proposed in multi-HIN settingswe directly extend traditional embedding methods intomulti-HIN cases as baseline methods We use embedding

6 Complexity

methods including TransE [9] R-GCN [29] DistMult [25]and ComplexE [30] However we are not using originalversions of these algorithms Instead we use refined modelsfrom OpenKE [31] to compare our model with currentfrequently used model versions Because these methods donot generate trustworthiness scores directly we calculate thetrustworthiness score according to their threshold where alink score that is larger than the threshold is thought as 0otherwise as 1 Moreover because these methods are notproposed for multi-HIN cases they are not able to integrally

analyse multiple HINs us we use these methods toevaluate trustworthiness scores separately on different HINs

Additionally we simplify our model for comparisonsbetween our model with its simplified version We use twosimplified models to show the effectiveness of our modelsetting ese two versions of methods are denoted asMattribute- and Minfluence- respectively Mattribute- is a versionignoring node attribute inner-HIN characteristic where inthe inner-HIN evaluation stage only node-type constraintcontextual embedding information and graph energy

Typeembedding

Edgeembedding

Attributeembedding

Nodeembedding

Nodeenergyfeatures

Scorefunction

Relationalloss in source

Head node hiddenrepresentation

Tail node hiddenrepresentation

Edge hiddenrepresentation

Influentialtriplet loss

We1h

We1h

We2h

We3h

We3h

We4h

We5h

We2h

WerWnrWar

WerWnrWar

e1 (Ns)

e1 (No)

e2 (Ns)

e2 (No)

e3 (Ns)

e3 (No)

e4 (Ep)

e5 (Ep)

WlrWtr

r (e1(Ns)))r (e2(Ns)))r (e3(Ns)))

r (e1(No)))r (e2(No)))r (e3(No)))

r (e4(Ep)))r (e5(Ep)))

Zs

Zo

Zp

Figure 3 Detailed structure of neural network

REQUIRE S maximal epochs of training T source trustworthiness Vfor G isin S do

VGlarr1end forwhile Tgt 0 dofor G isin S dofor t isin G doCalculate link score according to (2)

end forend forfor G isin S doUpdate source trustworthiness VG

end forTlarrT minus 1Backpropagate loss and update parameters in the neural network

end while

ALGORITHM 1 Multi-HIN trustworthiness evaluation

Table 1 Statistics of HINs constructed based on DBpedia and Wikidata

HIN of nodes of edges of attributes of node types of linksDBpedia1 100000 686 622331 225 463294Wikidata 100000 946 990517 295 448774

Complexity 7

transferring information are taken into considerationMinfluence- is the simplified model neglecting influencemethods through ILE betweenHINs where in the inter-HINevaluation stage all ILEs are set to 0 so that there exist nointeractive links

33 Hyperparameter Setup In the neural network of ourmodel the size of the node embedding is 256 the edgeembedding size is 64 and the attribute embedding size is512 e hidden size and combined nodeedge size is 512We use the ReLU function as the activation function in themodel e sigmoid function is used in the output layer forproducing the trustworthiness score e value of learningrate is selected from 01 001 0001 and set as 0001 andthe batch size is selected from 10 40 100 and set as 0001ese hyperparameters are selected through cross-validationwith the best performance in multi-HIN situations

e value of margin c in our function is selected from05 08 1 Moreover we use l2-norm as the regularizer tocontrol all the trainable parameters in the neural networkInner-HIN loss and cross-HIN balance parameter β is selectedfrom 03 05 07 and set as 05 Graphic balancing parameterϵ is selected from 03 05 07 and set as 05 Influencemethodcontrol threshold parameter ρ is selected from 001 01 05

and set as 01 Value influence threshold θvalue is selected from05 08 09 and set as 08 Alignment influence thresholdθalignment is selected from 0 01 02 04 and set as 02Neighbouring influence threshold θneighbor is selected from0 01 02 04 and set as 01 ese hyperparameters areselected through cross-validation with the best performance inmulti-HIN situations

34 Experiment I Errorless Link Classification Link classi-fication is the task of testing the model trustworthinessevaluation ability in new data In our experiment this ex-periment is implemented on the testing set It is a simplifiedtask compared with error link classification and link errordetection [11] where there is no noise in the training set ecorrect link is labelled as 1 while a false link is labelled as 0e model output lies in the range of [0 1] representing theprobability of one link being a correct onee trustworthinessscore is binarized into 0 (false link label) or 1 (correct link label)based on a threshold value Namely given a link the trust-worthiness score is calculated according to equation (5) If thetrustworthiness score is larger than the threshold this link islabelled as the correct link otherwise the false link

Here we compare our model with other models in basicsituations namely in separate HIN space and in combinedHIN spaces Besides in this experiment we train and test onthe original dataset not adding false links into training Alsoassuming we do not know where the link comes from theHIN source trustworthiness VG is set to 1

Experimental results are shown in Table 3 Obviouslyour model outperforms baseline methods in separate HINand combined HIN space which indicates that the hiddeninformation representing the reliability of links has been welllearned in our model We can analyse this result from twodifferent perspectives (1) single HIN and multi-HIN (2)

influence of inner-HIN characteristics From the view of singleand multi-HIN we can notice that when HINs are combinedin other words when data are augmented evaluation resultsare improving compared with those in single HIN situationis can be also proved from the threshold dropping in TransEDistMult and partial ComplexE results is proves thatthough previous methods do not directly consider the con-nection between HINs they can model the inherent rela-tionship between multiple HINs in other implicit ways whiletheir results are no better than our explicit relationshipmodeling method From the view of influence from inner-HINcharacteristics attribute characteristics are rather important inthe link classification task as we can see the accuracy ofMattribute- drops by nearly 5 compared with normal multi-HITE model Our model can achieve convergence by runningtwo or three epochs on the training data In the DBP-WDdataset it takes 45 minutes for one epoch e network em-bedding model usually takes two hours to network embeddingfollowed by the training of the classification task which costsmore time than ours

35 Experiment II Error Link Classification and Link ErrorDetection in TwoHINs e evaluation metric is the same asthe error detection task e first task error link classifi-cation is similar to the errorless link classification edifference is in the training set where error link classificationis trained on the noise training set while testing set andvalidation set are the same A small change in our modeltakes place here as our model cannot converge optimal inthe noise training set which means our model detects noiseWe first train and evaluate our model on the training set(both positive and negative-sampled links) and then retrainour model on the generated training set where labels arepredicted from the first trained modelrsquos evaluation esecond task link error detection is conducted on thetraining set e results are shown in Figure 4 in the formatof PR curve to test different modelsrsquo performance on thebinary classification task of detecting error links

In Table 4 we can see our model achieves the best ac-curacy and F1-score when compared with other models inthe multi-HIN setting for error link classification in 10 and40 neg situation In detail our model thought 786856816749 and 895114 links may be correct in the first train andevaluation round separately (where 757126 757135 and757139 are truly correct) where 10 20 and 40 noisedatasets contain links of 834973 910880 and 1062693When the noise links rise our model is more likely to ruleout false examples It reveals that our model achieves ef-fectiveness in modeling the inter-influence of HINs by usinginteractive links For TransE DistMult and ComplexE theyare robust to errors in the training set where errors slightlyreduce modelsrsquo performance It reveals that the traditionalembedding model can handle the problem of error links inmulti-HIN trustworthiness situations Two simplifiedmodels indicate the importance of inner characteristics andwe can conclude that the attributes are an important in-dicator of whether a link is true or not We think that thereason why attribute factors play a more important role in

8 Complexity

the evaluation is the number of attributes For 200000entities in this paperrsquos dataset there are 1612848 attributedescriptions in total while the interactions between HINsare subject to mutual influence conditions where thenumber of influence restrictions is far smaller than that ofattribute description information erefore deleting at-tribute information will affect the performance of the modelmore than deleting interaction information

In Figure 4 we can see our modelrsquos PR curve is aboveTransErsquos PR curves in the multi-HIN setting for link errordetection in all neg situation DistMult and ComplexE cannotdistinguish noise links in the training set so they are not plottedin Figure 4 It reveals that our model achieves effectiveness indetecting noise in multi-HIN situation Figure 4 also explainsthat there is a difference between the distributions of true linksand noise linksrsquo scores both in multi-HITE and TransE modelFor lines of TransE in Figure 4 there exist sharp drops in the leftpart is can be explained that the scores for true links andnoisy links do not have a large gap especially near extremethreshold where all links are thought as positive When thethreshold drops few links are thought to be noise while positivelinks are mistaken as noise which will influence the curve

sharply Also it can be observed that the PR curve rises whenthe number of noise links increases as it is easier to detect errorswhen errors are more frequent

36 Experiment III Error Link Classification in DifferentCombinations of HINs In this part we try to combinedifferent HINs with different numbers of HINs to comparethe performance of our model for error link classificatione results are shown in Table 5 In the case of combiningdifferent data sources our model proves robust from theresults At the same time with the increment of data sourcesour model can make more use of the interaction informationbetween different data sources to provide more sufficientevidence for link classification

37 Experiment IV Influential Method EvaluationFinally we test how the performance of two ILEs varies withdifferent values of the threshold and test the model in er-rorless link classification tasks Based on the results in Ta-ble 6 we can conclude that alignment ILE is more sensitiveto the threshold value because its threshold is close to 0

Table 2 Statistics of HINs constructed based on DBpedia and YAGO

HIN of nodes of edges of attributes of node types of linksDBpedia2 100000 643 760062 225 428952YAGO 100000 64 827682 295 502563

10

08

06

04

02

00100806040200

Prec

ision

10 negative

Recall

Multi-HITETransE

(a)

10

08

06

04

02

00

Prec

ision

20 negative

100806040200Recall

Multi-HITETransE

(b)

10

09

08

07

06

05

04

03Pr

ecisi

on

40 negative

100806040200Recall

Multi-HITETransE

(c)

Figure 4 Trade-off between precision and recall according to trustworthiness threshold

Table 3 Performance of link prediction from all the models on multiple HINs

DataMulti-HITE Minfluence- Mattribute- TransE R-GCN DistMult ComplexEacc F1 acc F1 acc F1 acc F1 acc F1 acc F1 acc F1

DBpedia1 mdash mdash 092 092 087 088 091 091 088 088 090 089 091 090Wikidata mdash mdash 094 094 093 089 090 090 089 089 088 088 089 088Combined 095 095 093 093 090 090 092 092 090 090 091 091 091 091ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

Complexity 9

compared with neighbour ILE It should be explained thatneighbour ILE (see equation (9)) has fewer componentscompared with alignment ILE (see equation (8))

4 Conclusions

In this paper we focus on the multi-HIN trustworthinessevaluation problem and propose a co-embedding spaceevaluation method Our model incorporates inner-HIN andinter-HIN characteristics into the loss function A deepneural network is used in our model to solve the problemWe compare our model with other single-HIN trustwor-thiness evaluation methods on a multi-HIN dataset Ex-perimental results demonstrate that our model can wellaccomplish the trustworthiness evaluation task and out-performs baseline models For future works we will focus onhow to incorporate truth value discovery into our modelBesides the processing of our model is complicated com-pared with other models such as TransE DistMult andComplexE erefore we need to find a concise way toevaluate trustworthiness while maintaining the current levelof accuracy

Data Availability

e HIN alignment dataset DBP-WD used to support thefindings of this study can be accessed through httpsgithubcomnju-websoftBootEA

Conflicts of Interest

e authors declare that there are no conflicts of interestregarding the publication of this paper

Acknowledgments

is study was supported by the National Key Research andDevelopment Program of China (grant nos2018YFC0830201 and 2017YFB1002801) the FundamentalResearch Funds for the Central Universities (grant nos4009009106 and 22120200184) and the CCF-Baidu OpenFund

References

[1] F M Harper and J A Konstan ldquoe movielens datasetshistory and contextrdquo ACM Transactions on Interactive In-telligent Systems vol 5 no 4 2015

[2] H Paulheim ldquoKnowledge graph refinement a survey ofapproaches and evaluation methodsrdquo Semantic Web vol 8no 3 pp 489ndash508 2017

[3] Y Lin Z Liu H Luan et al ldquoModeling relation paths forrepresentation learning of knowledge basesrdquo in Proceedings ofthe 2015 Conference on Empirical Methods in Natural Lan-guage Processing pp 705ndash714 Lisbon Portugal September2015

[4] C Yang Y Xiao Y Zhang et al ldquoHeterogeneous networkrepresentation learning survey benchmark evaluation andbeyondrdquo 2020 httparxivorgabs200400216

Table 4 Performance of all the methods over data with different levels of noise

DataMulti-HITE Minfluence- Mattribute- TransE R-GCN DistMult ComplexEacc F1 acc F1 acc F1 acc F1 acc F1 acc F1 acc F1

10 neg 091 092 090 090 090 089 091 090 090 090 091 091 091 09120 neg 090 091 089 089 089 089 091 090 089 089 090 090 090 09040 neg 092 092 090 089 089 089 090 090 089 089 088 088 088 088ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

Table 6 Results (accuracyF1-score) of multi-TW using different values of thresholds for ILE functions

Alignment neighbour 00 01 02 0400 0899508981 0899409020 0896808997 084330849501 0872308735 0893708975 0905809027 088830890402 0863908673 0948009482 0923309253 091040907804 0843808453 0922509237 0940409395 0907509017

Table 5 Performance of multi-HITE over different datasets with different levels of noise

DatasetDBP1-WD DBP2-YAGO YAGO-WD DBP1-YAGO-

DBP2-WDacc F1 acc F1 acc F1 acc F1

10 neg 091 092 090 090 091 091 091 09220 neg 090 091 090 090 089 089 091 09240 neg 092 092 089 089 089 089 090 090ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

10 Complexity

to dynamically update rule patterns A statistical example isSDValidate [5] which detects noise in a HIN via the sta-tistical distribution of the properties and types

Another popular approach is to integrate the KB rep-resentation learning into the models to evaluate the trust-worthiness in embedding space Typical examples are thetranslation models including TransE [9] and its variants likeTransH [8] and TransR [13] In Trans-series model if theenergy function of a link in the embedding space

A wtG1113872 1113873 Ns + Ep minus No

(1)

scores larger than a desired value it is assumed that this linkhas little local context proof in original HIN space iehaving low trustworthiness

In addition to translation models CKRL [11] exploits thestructure information by defining local link confidences andglobal path confidence to detect noises in HINsis methodexploits the structure information of the network KGTtm[10] is a multi-level knowledge graph link trustworthinessevaluation method It separates evaluation into nodes edgesand graph levels combining the internal semantic infor-mation of links and the global inference information of theKB However none of these methods consider utilizingother HINs that model similar domains erefore theirperformance is limited as the information from a single HINtends to be incomplete or one-sided

22LearningacrossMultipleHINs In the multi-HIN settingas nodes and edges in different HINs are extracted fromdifferent sources network alignment methods are proposedto find corresponding nodes across multiple networksTypically network alignment can be categorized into localalignment and global alignment Local alignment aims tofind similar local regions across networks [14] Globalalignment focuses on exploiting the global topologicalconsistency such as in iNEAT [15] and CAlign [16]

Conventional KB methods briefly use handcraftedldquosameAsrdquo link literal information and attribute features to

generate linkages between different HINs [17 18] Recent worksfocused on representation learning across multi-relationalgraphs noticing that aligned nodes should be similar in space[19] MTransE [20] tries to merge graphs and uses relationallearning methods to discover aligned nodes It combines threedifferent transformation methods to learn embedding forsupporting multi-lingual learning LinkNbed [21] learns latentrepresentations of nodes and edges in multiple graphs where aunified graph embedding is constructed avoiding the biascaused by the transformation between different HINembeddings

We naturally consider utilizing similar methods in nodealignment in the multi-HIN trustworthiness evaluationproblem However these methods of the node alignmentproblem lack global structure inference information whichprevents these methods from being directly generalized tomulti-HIN trustworthiness evaluation

23 General Structure of Trustworthiness Evaluation Toaddress the limitations of previous evaluation methods wepropose a novel method inherently considering inner-HINand inter-HIN effects e trustworthiness evaluation inmulti-HIN settings can be separated into two parts egeneral idea is shown in Figure 2

Firstly given the overall trustworthiness VGiof the i-th

HIN evaluate trustworthiness score of all links tj isin Gi becausein the multi-HIN situation different HINs have different re-liabilities e trustworthiness of a link in graph Gi is not onlybased on the inner-HIN trustworthiness A(w

tj

Gi) calculated by

inner characteristics of Gi but also dependent on the HINsource confidence VGi

is influence can be denoted as

T wtj

Gi1113874 1113875 VGi

A wtj

Gi1113874 1113875 (2)

Initially all global confidence VGiis assigned to 1 us

according to the representations for head nodes tail nodesand edges which are generated by inner characteristics andinitial source trustworthiness the trustworthiness of linkscan be calculated for every HIN

LeonardoDiCaprio

Inception

ChristopherNolan

Thegreat

gatsby

LeonardoDiCaprio

AnneHathaway

Interstellar

HIN 1 HIN 2

MatthewMcConaughey

Related node

Figure 1 General situation of multi-HIN problems

Complexity 3

en in the second step we use link trustworthinessscore to conversely update VGi

based on T(wtj

Gi) In the

opposite view of the multi-HIN problem we assume that thetrustworthiness of a HIN source is determined by all linksrsquotrustworthiness in the source and is computed via

VGi

1nGi

1113944tjisinGi

T wtj

Gi1113874 1113875 (3)

Here nGimeans the total number of links in graph Gi

which acts as a normalization factor At each iteration VGiis

firstly fixed to calculate single linkrsquos trustworthinessaccording to equation (2) followed by fixing T(w

tj

Gi) to

calculate VGi according to equation (3) After the iterativetraining has converged a steady trustworthiness evaluationof HIN source and link can be obtained

e above procedure is commonly used in no-mutual-in-fluence multi-HIN settings As for mutual-influence multi-HINcases the influence between differentHINs needs to bemodeledwhere our method models this influence by the trustworthinessinfluences between interactive links across HINs is will bediscussed in ldquoInter-HIN Evaluationrdquo Section Specifically dif-ferent HINs are placed on the co-embedding space to representthe influence between the sources If HINs are separatelyevaluated in their own space this will cause redundant proce-dure in the transformation betweenHIN spaces whichmake themodel difficult to train en having obtained inner scores forlinks in both HINs interactive influence will update corre-sponding linksrsquo scores e final step is to update the sourcetrustworthiness is process will be iteratively conducted AllHIN sourcesrsquo trustworthiness values are initially set to 1 and afteriterative influencing between sources and links the final scoresof links and sources will be obtained

Because the process of trustworthiness evaluation is thesame in all the HINs without loss of generality we use G andt to denote an arbitrary HIN and a certain edge in itWithoutexplicit statements we drop the subscripts i and j forsimplicity in the rest of this paper

e following sections will be organized as follows wedetail our modelrsquos trustworthiness evaluation in a single HINin Subsection 24 the details of the multi-HIN mutual

influence method in Subsection 25 and the training methodtailed for multi-HIN evaluation in Subsection 26

24 Inner-HIN Evaluation We start by considering how tocompute the link trustworthiness score with inner-HINcharacteristics Based on the current single HIN trust-worthiness evaluation methods [22] HINrsquos inner charac-teristics including semantic and graphic information arestrong proofs to evaluate the reliability of links In ourmodel we utilize four inner-HIN characteristics includingnode attributes node-type constraints contextual em-beddings and graphic energy transmission information toevaluate links

241 Node-Attribute Information Node attributes are thebasic and effective information to assess whether a node isconsistent with its value Here we use doc2vec [23] to embedattribute values and then integrate the value with its one-hotattribute key index to represent hidden information ofnodes If a node has several attributes the hidden attributeinformation will be averaged

242 Node-Type Constraint For a certain edge the nodesappearing near it normally have prevalent and uniformtypes us for each edge we can summarize the node typesthat emerge around it en the node-type information andedge embedding information are integrated into the hiddenlayer

243 Contextual Embedding Information Trans-seriesmodels embed nodes and edges in low dimension repre-sentation spaceese embeddings contain global contextualinformation where outliers can be found based on the in-consistency However we are not directly using the energyfunction to evaluate the consistency Here we utilize thefully connected layer to unite other inner-HIN character-istics with embeddings to get a hidden layerrsquos representation

Node attributes

Node types

Graphicattributes

Nodeembedding

Edgeembedding

Head hiddenrepresentation

Edge hiddenrepresentation

Tail hiddenrepresentation

Trustworthinessscore

HINtrustworthiness

updation

Interactiveinfluence

HIN 1

HIN 2

Innercharacteristics

Interactive links from HIN 1

Interactive links from HIN 2

Co-embeddingspace

Figure 2 General structure of multi-HIN trustworthiness evaluation method

4 Complexity

244 Graph Energy Transferring Information HIN is aspecial heterogeneous network where basic network anal-ysis techniques can be utilized If a link has strong con-nectivity or other favourable graphic features it alsoindicates sufficient proofs in the HIN Here we mainlyconcentrate on the energy flow between the head node andthe tail node In this situation we can construct a singlegraph for each node in the HIN where the node is the centralnode All nodes associated with the central node are takeninto consideration e energy transferring can be modeledwith page-rank-like method [11 24] However iteratingthrough a graph will have a long tail phenomenon where themajority of nodes have low energy Hence we add severalfeatures like in-degree and out-degree into consideration as

g Ns No( 1113857 ε timese Ns No( 1113857

e Ns( 1113857+(1 minus ε) times fNsNo

din dout( 1113857

(4)

Here g(Ns No) is the combined energy transmittedbetween embedded nodes Ns and No e(Ns) represents theenergy thatNs holds in the graph and e(Ns No) denotes theenergy flowing from Ns to No fNsNo

(din dout) is the energyfunction determined by the in-degree and out-degree of Ns

and No Besides ϵ is a hyperparameter balancing the page-rank-like energy and degree-determined function

After obtaining the above four different features eachfeature is transmitted to the hidden dimension by a separatelinear forward layer followed by the combination of allfeatures of the same node or link as shown in Figure 2 Afterobtaining the hidden representation z isin Rn (n is the size ofhidden states) of the head node tail node and edge we usethe following function to model the trustworthiness

T wtG1113872 1113873 VG times σ z

⊤p zs ⊙ zo( 11138571113872 1113873 (5)

in which σ(middot) is the activation function of the neural networkand ⊙ denotes the elementwise multiplication e scorefunction is related to the score of the linkrsquos located HIN so itneeds to multiply the located HINrsquos score VG Part of thisfunction is initially adopted by DistMult [25]

With all these we can calculate traditional trans-mission loss in every single HIN of S is loss is used tomodel local information of a HIN For a specific link t

shown in the original dataset we call it a positive link enegative links set c(G) is generated by replacing the headand tail node of t with other nodes so that negative linksdo not appear in the original dataset Specifically for aHIN G isin S let T(wt

G) and T(wtprimeG) be the positive score and

negative score respectively for the positive link t andnegative link tprime in this HIN and the relational loss formultiple HINs is

Lrel 1113944GisinS

1113944tisinG

1113944

tprimeisinc(G)

max 0 c minus T wtG1113872 1113873 + T w

tprimeG1113874 11138751113874 1113875)⎛⎝ (6)

Here c is the margin between positive links and negativelinks and c(G) is a set of corrupted links tprime corresponding tolinks t in the HIN G

25 Inter-HINEvaluation In multi-HIN settings inter-HINinfluence is another crucial component of the evaluationerefore we further include inter-HIN influence into theloss function for trustworthiness evaluation If a link(Ns Ep No)G in one HIN G has influence on a link(Nsprime Epprime Noprime)Gprime in another HIN Gprime we call these two linksinteractive links e way how interactive links influenceeach other in different HINs is called Interactive Links Effect(ILE) We use h(middot) to denote ILE and propose three differentkinds of ILEs that are utilized in our method for modelinginteractivity from three levels literal level semantic leveland graph level

251 Value ILE If two nodes refer to the same real-worldnode they should share similar attributes [26] Based on thisassumption from the literal level the value ILE can bedefined to evaluate influence for nodes of a pair of interactivelinks Let p(N) denote the attribute value set of the node N

and c(N1 N2) represent the common attributes shared bynodes N1 and N2 in the embedding space value ILE iscomputed via

hvalue Ns Ep No1113872 1113873G Nsprime Epprime Noprime1113872 1113873

Gprime1113872 1113873

c Ns Nsprime( 1113857

11138681113868111386811138681113868111386811138681113868

p Ns( 11138571113868111386811138681113868

1113868111386811138681113868 + p Nsprime( 11138571113868111386811138681113868

1113868111386811138681113868 minus c Ns Nsprime( 11138571113868111386811138681113868

1113868111386811138681113868

+c No Noprime( 1113857

11138681113868111386811138681113868111386811138681113868

p No( 11138571113868111386811138681113868

1113868111386811138681113868 + p Noprime( 11138571113868111386811138681113868

1113868111386811138681113868 minus c No Noprime( 11138571113868111386811138681113868

1113868111386811138681113868

(7)

in which | middot | is the number of elements in a set

252 Alignment ILE If two links have aligned nodes oredges they should be interactive Additionally alignednodes or edges are close in embedding space [19] us wecan assume that if two links are interactive at least one of thenodes or edges should be close in the embedding spaceBesides only one aligned part is not sufficient as many linksdescribe the same node from a different perspective like ldquoYaoMing-wife-Ye Lirdquo and ldquoYao Ming- teammate-TracyMcGradyrdquo us we add separate distances of co-spaceembedding of nodes and edges together and restrict thedistance of interactive links of this embedding distance todetermine whether two links are interactive or not Withthis we define the alignment ILE as

halignment Ns Ep No1113872 1113873G Nsprime Epprime Noprime1113872 1113873

Gprime1113872 1113873

Ns minus Nsprime

2 + No minus Noprime

2 + Ep minus Epprime

2

(8)

253 Neighbour ILE According to the semantic similaritywork [27] if nodes share similar surroundings there exists apossibility that they have semantic similarity in other wordsthey are interactive us we use neighbouring nodes of thehead and tail nodes to evaluate the ILE Let n(N) denote theaverage embedding for neighbouring nodes of node N thenwe have

Complexity 5

hneighbour Ns Ep No1113872 1113873G Nsprime Epprime Noprime1113872 1113873

Gprime1113872 1113873

n Ns( 1113857 minus n Nsprime( 1113857

2 + n No( 1113857 minus n Noprime( 1113857

2(9)

With all three ILE functions we can inherently use themto determine if two links from two different HINs haveinfluence on each other Specifically we design a matrix I torepresent the interaction of two HINs For a certain linki isin G and another link j isin Gprime their interactive status isdefined as

Iij 1 hvalue(i j)lt θvalue( 1113857 + 1 halignment(i j)lt θalignment1113872 1113873

+ 1 hneighbour(i j)lt θneighbour1113872 1113873

(10)

where 1(c) is the indication function with 1(c) 1 ifstatement c is true otherwise 1(c) 0 ree ILE thresholdsincluding θvalue θalignment and θneighbour are playing the rolesof hyperparameters

Intuitively the trustworthiness of two interactive linksshould be close to each other However there exist caseswhere two links are interactive but one has large trust-worthiness while that of another link is low us trust-worthiness scores of two links should be deductedSpecifically for i isin G and j isin Gprime we design the influencebetween links i and j as R(i j) 0 if Iij 0 otherwise

R(i j) T w

iG1113872 1113873 minus T w

j

G1113872 111387311138681113868111386811138681113868

11138681113868111386811138681113868 if T wiG1113872 1113873 minus T w

j

G1113872 111387311138681113868111386811138681113868

11138681113868111386811138681113868 le ρ

T wiG1113872 1113873 + T w

j

G1113872 111387311138681113868111386811138681113868

11138681113868111386811138681113868 if T wiG1113872 1113873 minus T w

j

G1113872 111387311138681113868111386811138681113868

11138681113868111386811138681113868 gt ρ

⎧⎪⎨

⎪⎩

(11)

Combining all the inter-HIN influences of interactivelinks we introduce the interactive loss

Lint 1113944(ij)isinIn

R(i j)(12)

with a set of interactive links In

26 Model Training in Multi-HIN Settings To obtain thetrustworthiness both in single-HIN and multi-HIN per-spectives we introduce the multi-task objective functionwhich models the contextual information of local and in-fluential information We integrate relational loss with in-teractive influential loss by using a hyperparameter β as thetrade-off and obtain

L β Lrel +(1 minus β)Lint (13)

e training process of our model can be separated intotwo main procedures

In the first phase we minimize the loss function with aneural network We put inner graph characteristics into theneural network including node attributes node-type con-straints node and edge embeddings and graph energytransferring information en we separately combinenode-related features into hidden representations of nodesand combine node-type constraints with edge embeddingsinto hidden representations of edges e detailed structure

of this neural network is shown in Figure 3 In the networke1(middot) is the node energy function for a certain node e2(middot) isthe node embedding e3(middot) denotes the attribute embeddinge4(middot) represents the edge embedding and e5(middot) is the typeembedding of a node Moreover weight matricesWe1h We2h We3h We4h andWe5h project different featuresto its hidden representations Function r(middot) projects em-beddings in the same space Besides Wer Wnr and War

combine node-related features to generate hidden repre-sentations of nodes while Wlr and Wtr combine edge-relatedfeatures to generate hidden representations of edges Inaddition we further attach an extra regularizer to control thesize of all the parameters Let Ω We1h We2h1113864

We3h We4h We5h Wer Wnr War Wlr Wtr denote a set ofall the weight parameters in the neural network and theregularized loss function becomes

L β Lrel +(1 minus β)Lint + 1113944W isinΩ

W2 (14)

After evaluating the score for all links in multiple HINswe can iteratively use equations (2) and (3) to update thetrustworthiness of HIN and each link until results convergee pseudocode of our method is shown in Algorithm 1

3 Results and Discussion

In this section we implement comprehensive experiments totest the performance of ourmodel from various perspectives

31 Dataset We choose to use two HIN alignment datasetsDBP-WD and DBP-YG [28] in our experiments e firstdataset is sampled from DBpedia and Wikidata (ie twoHINs) and contains 100 thousand aligned nodes while thesecond dataset is extracted from DBpedia and YAGO withsame 100 thousand aligned nodes Information about nodetypes is attached to the data according to DBpedia type in-formation Tables 1 and 2 provide statistics of these datasets inthe experimentWe split this dataset into training testing andvalidation set with the ratio as 10 1 1 Besides DBP-WDdataset is used in all experiments while DBP-YG is only usedin Experiment III DBpedia in DBP-WD is noted as DBpediawhile DBpedia in DBP-YG is noted as DBpedia2

Furthermore we generate some negative links in thedataset to simulate noisy data with erroneous links Spe-cifically for one negative link we replace the head or tailnode of a positive link so that it does not appear in theoriginal dataset Other characteristics including node at-tributes node types and neighbouring nodes remain thesame with the replaced node To obtain a different level ofnoise similar to the simulation implemented in [11]multiple noisy datasets are simulated with the percentage ofreplaced negative links of 10 20 and 40 Notice thatthe noises are only added to the training set

32 Baseline Methods As there is no trustworthiness eval-uation method previously proposed in multi-HIN settingswe directly extend traditional embedding methods intomulti-HIN cases as baseline methods We use embedding

6 Complexity

methods including TransE [9] R-GCN [29] DistMult [25]and ComplexE [30] However we are not using originalversions of these algorithms Instead we use refined modelsfrom OpenKE [31] to compare our model with currentfrequently used model versions Because these methods donot generate trustworthiness scores directly we calculate thetrustworthiness score according to their threshold where alink score that is larger than the threshold is thought as 0otherwise as 1 Moreover because these methods are notproposed for multi-HIN cases they are not able to integrally

analyse multiple HINs us we use these methods toevaluate trustworthiness scores separately on different HINs

Additionally we simplify our model for comparisonsbetween our model with its simplified version We use twosimplified models to show the effectiveness of our modelsetting ese two versions of methods are denoted asMattribute- and Minfluence- respectively Mattribute- is a versionignoring node attribute inner-HIN characteristic where inthe inner-HIN evaluation stage only node-type constraintcontextual embedding information and graph energy

Typeembedding

Edgeembedding

Attributeembedding

Nodeembedding

Nodeenergyfeatures

Scorefunction

Relationalloss in source

Head node hiddenrepresentation

Tail node hiddenrepresentation

Edge hiddenrepresentation

Influentialtriplet loss

We1h

We1h

We2h

We3h

We3h

We4h

We5h

We2h

WerWnrWar

WerWnrWar

e1 (Ns)

e1 (No)

e2 (Ns)

e2 (No)

e3 (Ns)

e3 (No)

e4 (Ep)

e5 (Ep)

WlrWtr

r (e1(Ns)))r (e2(Ns)))r (e3(Ns)))

r (e1(No)))r (e2(No)))r (e3(No)))

r (e4(Ep)))r (e5(Ep)))

Zs

Zo

Zp

Figure 3 Detailed structure of neural network

REQUIRE S maximal epochs of training T source trustworthiness Vfor G isin S do

VGlarr1end forwhile Tgt 0 dofor G isin S dofor t isin G doCalculate link score according to (2)

end forend forfor G isin S doUpdate source trustworthiness VG

end forTlarrT minus 1Backpropagate loss and update parameters in the neural network

end while

ALGORITHM 1 Multi-HIN trustworthiness evaluation

Table 1 Statistics of HINs constructed based on DBpedia and Wikidata

HIN of nodes of edges of attributes of node types of linksDBpedia1 100000 686 622331 225 463294Wikidata 100000 946 990517 295 448774

Complexity 7

transferring information are taken into considerationMinfluence- is the simplified model neglecting influencemethods through ILE betweenHINs where in the inter-HINevaluation stage all ILEs are set to 0 so that there exist nointeractive links

33 Hyperparameter Setup In the neural network of ourmodel the size of the node embedding is 256 the edgeembedding size is 64 and the attribute embedding size is512 e hidden size and combined nodeedge size is 512We use the ReLU function as the activation function in themodel e sigmoid function is used in the output layer forproducing the trustworthiness score e value of learningrate is selected from 01 001 0001 and set as 0001 andthe batch size is selected from 10 40 100 and set as 0001ese hyperparameters are selected through cross-validationwith the best performance in multi-HIN situations

e value of margin c in our function is selected from05 08 1 Moreover we use l2-norm as the regularizer tocontrol all the trainable parameters in the neural networkInner-HIN loss and cross-HIN balance parameter β is selectedfrom 03 05 07 and set as 05 Graphic balancing parameterϵ is selected from 03 05 07 and set as 05 Influencemethodcontrol threshold parameter ρ is selected from 001 01 05

and set as 01 Value influence threshold θvalue is selected from05 08 09 and set as 08 Alignment influence thresholdθalignment is selected from 0 01 02 04 and set as 02Neighbouring influence threshold θneighbor is selected from0 01 02 04 and set as 01 ese hyperparameters areselected through cross-validation with the best performance inmulti-HIN situations

34 Experiment I Errorless Link Classification Link classi-fication is the task of testing the model trustworthinessevaluation ability in new data In our experiment this ex-periment is implemented on the testing set It is a simplifiedtask compared with error link classification and link errordetection [11] where there is no noise in the training set ecorrect link is labelled as 1 while a false link is labelled as 0e model output lies in the range of [0 1] representing theprobability of one link being a correct onee trustworthinessscore is binarized into 0 (false link label) or 1 (correct link label)based on a threshold value Namely given a link the trust-worthiness score is calculated according to equation (5) If thetrustworthiness score is larger than the threshold this link islabelled as the correct link otherwise the false link

Here we compare our model with other models in basicsituations namely in separate HIN space and in combinedHIN spaces Besides in this experiment we train and test onthe original dataset not adding false links into training Alsoassuming we do not know where the link comes from theHIN source trustworthiness VG is set to 1

Experimental results are shown in Table 3 Obviouslyour model outperforms baseline methods in separate HINand combined HIN space which indicates that the hiddeninformation representing the reliability of links has been welllearned in our model We can analyse this result from twodifferent perspectives (1) single HIN and multi-HIN (2)

influence of inner-HIN characteristics From the view of singleand multi-HIN we can notice that when HINs are combinedin other words when data are augmented evaluation resultsare improving compared with those in single HIN situationis can be also proved from the threshold dropping in TransEDistMult and partial ComplexE results is proves thatthough previous methods do not directly consider the con-nection between HINs they can model the inherent rela-tionship between multiple HINs in other implicit ways whiletheir results are no better than our explicit relationshipmodeling method From the view of influence from inner-HINcharacteristics attribute characteristics are rather important inthe link classification task as we can see the accuracy ofMattribute- drops by nearly 5 compared with normal multi-HITE model Our model can achieve convergence by runningtwo or three epochs on the training data In the DBP-WDdataset it takes 45 minutes for one epoch e network em-bedding model usually takes two hours to network embeddingfollowed by the training of the classification task which costsmore time than ours

35 Experiment II Error Link Classification and Link ErrorDetection in TwoHINs e evaluation metric is the same asthe error detection task e first task error link classifi-cation is similar to the errorless link classification edifference is in the training set where error link classificationis trained on the noise training set while testing set andvalidation set are the same A small change in our modeltakes place here as our model cannot converge optimal inthe noise training set which means our model detects noiseWe first train and evaluate our model on the training set(both positive and negative-sampled links) and then retrainour model on the generated training set where labels arepredicted from the first trained modelrsquos evaluation esecond task link error detection is conducted on thetraining set e results are shown in Figure 4 in the formatof PR curve to test different modelsrsquo performance on thebinary classification task of detecting error links

In Table 4 we can see our model achieves the best ac-curacy and F1-score when compared with other models inthe multi-HIN setting for error link classification in 10 and40 neg situation In detail our model thought 786856816749 and 895114 links may be correct in the first train andevaluation round separately (where 757126 757135 and757139 are truly correct) where 10 20 and 40 noisedatasets contain links of 834973 910880 and 1062693When the noise links rise our model is more likely to ruleout false examples It reveals that our model achieves ef-fectiveness in modeling the inter-influence of HINs by usinginteractive links For TransE DistMult and ComplexE theyare robust to errors in the training set where errors slightlyreduce modelsrsquo performance It reveals that the traditionalembedding model can handle the problem of error links inmulti-HIN trustworthiness situations Two simplifiedmodels indicate the importance of inner characteristics andwe can conclude that the attributes are an important in-dicator of whether a link is true or not We think that thereason why attribute factors play a more important role in

8 Complexity

the evaluation is the number of attributes For 200000entities in this paperrsquos dataset there are 1612848 attributedescriptions in total while the interactions between HINsare subject to mutual influence conditions where thenumber of influence restrictions is far smaller than that ofattribute description information erefore deleting at-tribute information will affect the performance of the modelmore than deleting interaction information

In Figure 4 we can see our modelrsquos PR curve is aboveTransErsquos PR curves in the multi-HIN setting for link errordetection in all neg situation DistMult and ComplexE cannotdistinguish noise links in the training set so they are not plottedin Figure 4 It reveals that our model achieves effectiveness indetecting noise in multi-HIN situation Figure 4 also explainsthat there is a difference between the distributions of true linksand noise linksrsquo scores both in multi-HITE and TransE modelFor lines of TransE in Figure 4 there exist sharp drops in the leftpart is can be explained that the scores for true links andnoisy links do not have a large gap especially near extremethreshold where all links are thought as positive When thethreshold drops few links are thought to be noise while positivelinks are mistaken as noise which will influence the curve

sharply Also it can be observed that the PR curve rises whenthe number of noise links increases as it is easier to detect errorswhen errors are more frequent

36 Experiment III Error Link Classification in DifferentCombinations of HINs In this part we try to combinedifferent HINs with different numbers of HINs to comparethe performance of our model for error link classificatione results are shown in Table 5 In the case of combiningdifferent data sources our model proves robust from theresults At the same time with the increment of data sourcesour model can make more use of the interaction informationbetween different data sources to provide more sufficientevidence for link classification

37 Experiment IV Influential Method EvaluationFinally we test how the performance of two ILEs varies withdifferent values of the threshold and test the model in er-rorless link classification tasks Based on the results in Ta-ble 6 we can conclude that alignment ILE is more sensitiveto the threshold value because its threshold is close to 0

Table 2 Statistics of HINs constructed based on DBpedia and YAGO

HIN of nodes of edges of attributes of node types of linksDBpedia2 100000 643 760062 225 428952YAGO 100000 64 827682 295 502563

10

08

06

04

02

00100806040200

Prec

ision

10 negative

Recall

Multi-HITETransE

(a)

10

08

06

04

02

00

Prec

ision

20 negative

100806040200Recall

Multi-HITETransE

(b)

10

09

08

07

06

05

04

03Pr

ecisi

on

40 negative

100806040200Recall

Multi-HITETransE

(c)

Figure 4 Trade-off between precision and recall according to trustworthiness threshold

Table 3 Performance of link prediction from all the models on multiple HINs

DataMulti-HITE Minfluence- Mattribute- TransE R-GCN DistMult ComplexEacc F1 acc F1 acc F1 acc F1 acc F1 acc F1 acc F1

DBpedia1 mdash mdash 092 092 087 088 091 091 088 088 090 089 091 090Wikidata mdash mdash 094 094 093 089 090 090 089 089 088 088 089 088Combined 095 095 093 093 090 090 092 092 090 090 091 091 091 091ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

Complexity 9

compared with neighbour ILE It should be explained thatneighbour ILE (see equation (9)) has fewer componentscompared with alignment ILE (see equation (8))

4 Conclusions

In this paper we focus on the multi-HIN trustworthinessevaluation problem and propose a co-embedding spaceevaluation method Our model incorporates inner-HIN andinter-HIN characteristics into the loss function A deepneural network is used in our model to solve the problemWe compare our model with other single-HIN trustwor-thiness evaluation methods on a multi-HIN dataset Ex-perimental results demonstrate that our model can wellaccomplish the trustworthiness evaluation task and out-performs baseline models For future works we will focus onhow to incorporate truth value discovery into our modelBesides the processing of our model is complicated com-pared with other models such as TransE DistMult andComplexE erefore we need to find a concise way toevaluate trustworthiness while maintaining the current levelof accuracy

Data Availability

e HIN alignment dataset DBP-WD used to support thefindings of this study can be accessed through httpsgithubcomnju-websoftBootEA

Conflicts of Interest

e authors declare that there are no conflicts of interestregarding the publication of this paper

Acknowledgments

is study was supported by the National Key Research andDevelopment Program of China (grant nos2018YFC0830201 and 2017YFB1002801) the FundamentalResearch Funds for the Central Universities (grant nos4009009106 and 22120200184) and the CCF-Baidu OpenFund

References

[1] F M Harper and J A Konstan ldquoe movielens datasetshistory and contextrdquo ACM Transactions on Interactive In-telligent Systems vol 5 no 4 2015

[2] H Paulheim ldquoKnowledge graph refinement a survey ofapproaches and evaluation methodsrdquo Semantic Web vol 8no 3 pp 489ndash508 2017

[3] Y Lin Z Liu H Luan et al ldquoModeling relation paths forrepresentation learning of knowledge basesrdquo in Proceedings ofthe 2015 Conference on Empirical Methods in Natural Lan-guage Processing pp 705ndash714 Lisbon Portugal September2015

[4] C Yang Y Xiao Y Zhang et al ldquoHeterogeneous networkrepresentation learning survey benchmark evaluation andbeyondrdquo 2020 httparxivorgabs200400216

Table 4 Performance of all the methods over data with different levels of noise

DataMulti-HITE Minfluence- Mattribute- TransE R-GCN DistMult ComplexEacc F1 acc F1 acc F1 acc F1 acc F1 acc F1 acc F1

10 neg 091 092 090 090 090 089 091 090 090 090 091 091 091 09120 neg 090 091 089 089 089 089 091 090 089 089 090 090 090 09040 neg 092 092 090 089 089 089 090 090 089 089 088 088 088 088ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

Table 6 Results (accuracyF1-score) of multi-TW using different values of thresholds for ILE functions

Alignment neighbour 00 01 02 0400 0899508981 0899409020 0896808997 084330849501 0872308735 0893708975 0905809027 088830890402 0863908673 0948009482 0923309253 091040907804 0843808453 0922509237 0940409395 0907509017

Table 5 Performance of multi-HITE over different datasets with different levels of noise

DatasetDBP1-WD DBP2-YAGO YAGO-WD DBP1-YAGO-

DBP2-WDacc F1 acc F1 acc F1 acc F1

10 neg 091 092 090 090 091 091 091 09220 neg 090 091 090 090 089 089 091 09240 neg 092 092 089 089 089 089 090 090ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

10 Complexity

en in the second step we use link trustworthinessscore to conversely update VGi

based on T(wtj

Gi) In the

opposite view of the multi-HIN problem we assume that thetrustworthiness of a HIN source is determined by all linksrsquotrustworthiness in the source and is computed via

VGi

1nGi

1113944tjisinGi

T wtj

Gi1113874 1113875 (3)

Here nGimeans the total number of links in graph Gi

which acts as a normalization factor At each iteration VGiis

firstly fixed to calculate single linkrsquos trustworthinessaccording to equation (2) followed by fixing T(w

tj

Gi) to

calculate VGi according to equation (3) After the iterativetraining has converged a steady trustworthiness evaluationof HIN source and link can be obtained

e above procedure is commonly used in no-mutual-in-fluence multi-HIN settings As for mutual-influence multi-HINcases the influence between differentHINs needs to bemodeledwhere our method models this influence by the trustworthinessinfluences between interactive links across HINs is will bediscussed in ldquoInter-HIN Evaluationrdquo Section Specifically dif-ferent HINs are placed on the co-embedding space to representthe influence between the sources If HINs are separatelyevaluated in their own space this will cause redundant proce-dure in the transformation betweenHIN spaces whichmake themodel difficult to train en having obtained inner scores forlinks in both HINs interactive influence will update corre-sponding linksrsquo scores e final step is to update the sourcetrustworthiness is process will be iteratively conducted AllHIN sourcesrsquo trustworthiness values are initially set to 1 and afteriterative influencing between sources and links the final scoresof links and sources will be obtained

Because the process of trustworthiness evaluation is thesame in all the HINs without loss of generality we use G andt to denote an arbitrary HIN and a certain edge in itWithoutexplicit statements we drop the subscripts i and j forsimplicity in the rest of this paper

e following sections will be organized as follows wedetail our modelrsquos trustworthiness evaluation in a single HINin Subsection 24 the details of the multi-HIN mutual

influence method in Subsection 25 and the training methodtailed for multi-HIN evaluation in Subsection 26

24 Inner-HIN Evaluation We start by considering how tocompute the link trustworthiness score with inner-HINcharacteristics Based on the current single HIN trust-worthiness evaluation methods [22] HINrsquos inner charac-teristics including semantic and graphic information arestrong proofs to evaluate the reliability of links In ourmodel we utilize four inner-HIN characteristics includingnode attributes node-type constraints contextual em-beddings and graphic energy transmission information toevaluate links

241 Node-Attribute Information Node attributes are thebasic and effective information to assess whether a node isconsistent with its value Here we use doc2vec [23] to embedattribute values and then integrate the value with its one-hotattribute key index to represent hidden information ofnodes If a node has several attributes the hidden attributeinformation will be averaged

242 Node-Type Constraint For a certain edge the nodesappearing near it normally have prevalent and uniformtypes us for each edge we can summarize the node typesthat emerge around it en the node-type information andedge embedding information are integrated into the hiddenlayer

243 Contextual Embedding Information Trans-seriesmodels embed nodes and edges in low dimension repre-sentation spaceese embeddings contain global contextualinformation where outliers can be found based on the in-consistency However we are not directly using the energyfunction to evaluate the consistency Here we utilize thefully connected layer to unite other inner-HIN character-istics with embeddings to get a hidden layerrsquos representation

Node attributes

Node types

Graphicattributes

Nodeembedding

Edgeembedding

Head hiddenrepresentation

Edge hiddenrepresentation

Tail hiddenrepresentation

Trustworthinessscore

HINtrustworthiness

updation

Interactiveinfluence

HIN 1

HIN 2

Innercharacteristics

Interactive links from HIN 1

Interactive links from HIN 2

Co-embeddingspace

Figure 2 General structure of multi-HIN trustworthiness evaluation method

4 Complexity

244 Graph Energy Transferring Information HIN is aspecial heterogeneous network where basic network anal-ysis techniques can be utilized If a link has strong con-nectivity or other favourable graphic features it alsoindicates sufficient proofs in the HIN Here we mainlyconcentrate on the energy flow between the head node andthe tail node In this situation we can construct a singlegraph for each node in the HIN where the node is the centralnode All nodes associated with the central node are takeninto consideration e energy transferring can be modeledwith page-rank-like method [11 24] However iteratingthrough a graph will have a long tail phenomenon where themajority of nodes have low energy Hence we add severalfeatures like in-degree and out-degree into consideration as

g Ns No( 1113857 ε timese Ns No( 1113857

e Ns( 1113857+(1 minus ε) times fNsNo

din dout( 1113857

(4)

Here g(Ns No) is the combined energy transmittedbetween embedded nodes Ns and No e(Ns) represents theenergy thatNs holds in the graph and e(Ns No) denotes theenergy flowing from Ns to No fNsNo

(din dout) is the energyfunction determined by the in-degree and out-degree of Ns

and No Besides ϵ is a hyperparameter balancing the page-rank-like energy and degree-determined function

After obtaining the above four different features eachfeature is transmitted to the hidden dimension by a separatelinear forward layer followed by the combination of allfeatures of the same node or link as shown in Figure 2 Afterobtaining the hidden representation z isin Rn (n is the size ofhidden states) of the head node tail node and edge we usethe following function to model the trustworthiness

T wtG1113872 1113873 VG times σ z

⊤p zs ⊙ zo( 11138571113872 1113873 (5)

in which σ(middot) is the activation function of the neural networkand ⊙ denotes the elementwise multiplication e scorefunction is related to the score of the linkrsquos located HIN so itneeds to multiply the located HINrsquos score VG Part of thisfunction is initially adopted by DistMult [25]

With all these we can calculate traditional trans-mission loss in every single HIN of S is loss is used tomodel local information of a HIN For a specific link t

shown in the original dataset we call it a positive link enegative links set c(G) is generated by replacing the headand tail node of t with other nodes so that negative linksdo not appear in the original dataset Specifically for aHIN G isin S let T(wt

G) and T(wtprimeG) be the positive score and

negative score respectively for the positive link t andnegative link tprime in this HIN and the relational loss formultiple HINs is

Lrel 1113944GisinS

1113944tisinG

1113944

tprimeisinc(G)

max 0 c minus T wtG1113872 1113873 + T w

tprimeG1113874 11138751113874 1113875)⎛⎝ (6)

Here c is the margin between positive links and negativelinks and c(G) is a set of corrupted links tprime corresponding tolinks t in the HIN G

25 Inter-HINEvaluation In multi-HIN settings inter-HINinfluence is another crucial component of the evaluationerefore we further include inter-HIN influence into theloss function for trustworthiness evaluation If a link(Ns Ep No)G in one HIN G has influence on a link(Nsprime Epprime Noprime)Gprime in another HIN Gprime we call these two linksinteractive links e way how interactive links influenceeach other in different HINs is called Interactive Links Effect(ILE) We use h(middot) to denote ILE and propose three differentkinds of ILEs that are utilized in our method for modelinginteractivity from three levels literal level semantic leveland graph level

251 Value ILE If two nodes refer to the same real-worldnode they should share similar attributes [26] Based on thisassumption from the literal level the value ILE can bedefined to evaluate influence for nodes of a pair of interactivelinks Let p(N) denote the attribute value set of the node N

and c(N1 N2) represent the common attributes shared bynodes N1 and N2 in the embedding space value ILE iscomputed via

hvalue Ns Ep No1113872 1113873G Nsprime Epprime Noprime1113872 1113873

Gprime1113872 1113873

c Ns Nsprime( 1113857

11138681113868111386811138681113868111386811138681113868

p Ns( 11138571113868111386811138681113868

1113868111386811138681113868 + p Nsprime( 11138571113868111386811138681113868

1113868111386811138681113868 minus c Ns Nsprime( 11138571113868111386811138681113868

1113868111386811138681113868

+c No Noprime( 1113857

11138681113868111386811138681113868111386811138681113868

p No( 11138571113868111386811138681113868

1113868111386811138681113868 + p Noprime( 11138571113868111386811138681113868

1113868111386811138681113868 minus c No Noprime( 11138571113868111386811138681113868

1113868111386811138681113868

(7)

in which | middot | is the number of elements in a set

252 Alignment ILE If two links have aligned nodes oredges they should be interactive Additionally alignednodes or edges are close in embedding space [19] us wecan assume that if two links are interactive at least one of thenodes or edges should be close in the embedding spaceBesides only one aligned part is not sufficient as many linksdescribe the same node from a different perspective like ldquoYaoMing-wife-Ye Lirdquo and ldquoYao Ming- teammate-TracyMcGradyrdquo us we add separate distances of co-spaceembedding of nodes and edges together and restrict thedistance of interactive links of this embedding distance todetermine whether two links are interactive or not Withthis we define the alignment ILE as

halignment Ns Ep No1113872 1113873G Nsprime Epprime Noprime1113872 1113873

Gprime1113872 1113873

Ns minus Nsprime

2 + No minus Noprime

2 + Ep minus Epprime

2

(8)

253 Neighbour ILE According to the semantic similaritywork [27] if nodes share similar surroundings there exists apossibility that they have semantic similarity in other wordsthey are interactive us we use neighbouring nodes of thehead and tail nodes to evaluate the ILE Let n(N) denote theaverage embedding for neighbouring nodes of node N thenwe have

Complexity 5

hneighbour Ns Ep No1113872 1113873G Nsprime Epprime Noprime1113872 1113873

Gprime1113872 1113873

n Ns( 1113857 minus n Nsprime( 1113857

2 + n No( 1113857 minus n Noprime( 1113857

2(9)

With all three ILE functions we can inherently use themto determine if two links from two different HINs haveinfluence on each other Specifically we design a matrix I torepresent the interaction of two HINs For a certain linki isin G and another link j isin Gprime their interactive status isdefined as

Iij 1 hvalue(i j)lt θvalue( 1113857 + 1 halignment(i j)lt θalignment1113872 1113873

+ 1 hneighbour(i j)lt θneighbour1113872 1113873

(10)

where 1(c) is the indication function with 1(c) 1 ifstatement c is true otherwise 1(c) 0 ree ILE thresholdsincluding θvalue θalignment and θneighbour are playing the rolesof hyperparameters

Intuitively the trustworthiness of two interactive linksshould be close to each other However there exist caseswhere two links are interactive but one has large trust-worthiness while that of another link is low us trust-worthiness scores of two links should be deductedSpecifically for i isin G and j isin Gprime we design the influencebetween links i and j as R(i j) 0 if Iij 0 otherwise

R(i j) T w

iG1113872 1113873 minus T w

j

G1113872 111387311138681113868111386811138681113868

11138681113868111386811138681113868 if T wiG1113872 1113873 minus T w

j

G1113872 111387311138681113868111386811138681113868

11138681113868111386811138681113868 le ρ

T wiG1113872 1113873 + T w

j

G1113872 111387311138681113868111386811138681113868

11138681113868111386811138681113868 if T wiG1113872 1113873 minus T w

j

G1113872 111387311138681113868111386811138681113868

11138681113868111386811138681113868 gt ρ

⎧⎪⎨

⎪⎩

(11)

Combining all the inter-HIN influences of interactivelinks we introduce the interactive loss

Lint 1113944(ij)isinIn

R(i j)(12)

with a set of interactive links In

26 Model Training in Multi-HIN Settings To obtain thetrustworthiness both in single-HIN and multi-HIN per-spectives we introduce the multi-task objective functionwhich models the contextual information of local and in-fluential information We integrate relational loss with in-teractive influential loss by using a hyperparameter β as thetrade-off and obtain

L β Lrel +(1 minus β)Lint (13)

e training process of our model can be separated intotwo main procedures

In the first phase we minimize the loss function with aneural network We put inner graph characteristics into theneural network including node attributes node-type con-straints node and edge embeddings and graph energytransferring information en we separately combinenode-related features into hidden representations of nodesand combine node-type constraints with edge embeddingsinto hidden representations of edges e detailed structure

of this neural network is shown in Figure 3 In the networke1(middot) is the node energy function for a certain node e2(middot) isthe node embedding e3(middot) denotes the attribute embeddinge4(middot) represents the edge embedding and e5(middot) is the typeembedding of a node Moreover weight matricesWe1h We2h We3h We4h andWe5h project different featuresto its hidden representations Function r(middot) projects em-beddings in the same space Besides Wer Wnr and War

combine node-related features to generate hidden repre-sentations of nodes while Wlr and Wtr combine edge-relatedfeatures to generate hidden representations of edges Inaddition we further attach an extra regularizer to control thesize of all the parameters Let Ω We1h We2h1113864

We3h We4h We5h Wer Wnr War Wlr Wtr denote a set ofall the weight parameters in the neural network and theregularized loss function becomes

L β Lrel +(1 minus β)Lint + 1113944W isinΩ

W2 (14)

After evaluating the score for all links in multiple HINswe can iteratively use equations (2) and (3) to update thetrustworthiness of HIN and each link until results convergee pseudocode of our method is shown in Algorithm 1

3 Results and Discussion

In this section we implement comprehensive experiments totest the performance of ourmodel from various perspectives

31 Dataset We choose to use two HIN alignment datasetsDBP-WD and DBP-YG [28] in our experiments e firstdataset is sampled from DBpedia and Wikidata (ie twoHINs) and contains 100 thousand aligned nodes while thesecond dataset is extracted from DBpedia and YAGO withsame 100 thousand aligned nodes Information about nodetypes is attached to the data according to DBpedia type in-formation Tables 1 and 2 provide statistics of these datasets inthe experimentWe split this dataset into training testing andvalidation set with the ratio as 10 1 1 Besides DBP-WDdataset is used in all experiments while DBP-YG is only usedin Experiment III DBpedia in DBP-WD is noted as DBpediawhile DBpedia in DBP-YG is noted as DBpedia2

Furthermore we generate some negative links in thedataset to simulate noisy data with erroneous links Spe-cifically for one negative link we replace the head or tailnode of a positive link so that it does not appear in theoriginal dataset Other characteristics including node at-tributes node types and neighbouring nodes remain thesame with the replaced node To obtain a different level ofnoise similar to the simulation implemented in [11]multiple noisy datasets are simulated with the percentage ofreplaced negative links of 10 20 and 40 Notice thatthe noises are only added to the training set

32 Baseline Methods As there is no trustworthiness eval-uation method previously proposed in multi-HIN settingswe directly extend traditional embedding methods intomulti-HIN cases as baseline methods We use embedding

6 Complexity

methods including TransE [9] R-GCN [29] DistMult [25]and ComplexE [30] However we are not using originalversions of these algorithms Instead we use refined modelsfrom OpenKE [31] to compare our model with currentfrequently used model versions Because these methods donot generate trustworthiness scores directly we calculate thetrustworthiness score according to their threshold where alink score that is larger than the threshold is thought as 0otherwise as 1 Moreover because these methods are notproposed for multi-HIN cases they are not able to integrally

analyse multiple HINs us we use these methods toevaluate trustworthiness scores separately on different HINs

Additionally we simplify our model for comparisonsbetween our model with its simplified version We use twosimplified models to show the effectiveness of our modelsetting ese two versions of methods are denoted asMattribute- and Minfluence- respectively Mattribute- is a versionignoring node attribute inner-HIN characteristic where inthe inner-HIN evaluation stage only node-type constraintcontextual embedding information and graph energy

Typeembedding

Edgeembedding

Attributeembedding

Nodeembedding

Nodeenergyfeatures

Scorefunction

Relationalloss in source

Head node hiddenrepresentation

Tail node hiddenrepresentation

Edge hiddenrepresentation

Influentialtriplet loss

We1h

We1h

We2h

We3h

We3h

We4h

We5h

We2h

WerWnrWar

WerWnrWar

e1 (Ns)

e1 (No)

e2 (Ns)

e2 (No)

e3 (Ns)

e3 (No)

e4 (Ep)

e5 (Ep)

WlrWtr

r (e1(Ns)))r (e2(Ns)))r (e3(Ns)))

r (e1(No)))r (e2(No)))r (e3(No)))

r (e4(Ep)))r (e5(Ep)))

Zs

Zo

Zp

Figure 3 Detailed structure of neural network

REQUIRE S maximal epochs of training T source trustworthiness Vfor G isin S do

VGlarr1end forwhile Tgt 0 dofor G isin S dofor t isin G doCalculate link score according to (2)

end forend forfor G isin S doUpdate source trustworthiness VG

end forTlarrT minus 1Backpropagate loss and update parameters in the neural network

end while

ALGORITHM 1 Multi-HIN trustworthiness evaluation

Table 1 Statistics of HINs constructed based on DBpedia and Wikidata

HIN of nodes of edges of attributes of node types of linksDBpedia1 100000 686 622331 225 463294Wikidata 100000 946 990517 295 448774

Complexity 7

transferring information are taken into considerationMinfluence- is the simplified model neglecting influencemethods through ILE betweenHINs where in the inter-HINevaluation stage all ILEs are set to 0 so that there exist nointeractive links

33 Hyperparameter Setup In the neural network of ourmodel the size of the node embedding is 256 the edgeembedding size is 64 and the attribute embedding size is512 e hidden size and combined nodeedge size is 512We use the ReLU function as the activation function in themodel e sigmoid function is used in the output layer forproducing the trustworthiness score e value of learningrate is selected from 01 001 0001 and set as 0001 andthe batch size is selected from 10 40 100 and set as 0001ese hyperparameters are selected through cross-validationwith the best performance in multi-HIN situations

e value of margin c in our function is selected from05 08 1 Moreover we use l2-norm as the regularizer tocontrol all the trainable parameters in the neural networkInner-HIN loss and cross-HIN balance parameter β is selectedfrom 03 05 07 and set as 05 Graphic balancing parameterϵ is selected from 03 05 07 and set as 05 Influencemethodcontrol threshold parameter ρ is selected from 001 01 05

and set as 01 Value influence threshold θvalue is selected from05 08 09 and set as 08 Alignment influence thresholdθalignment is selected from 0 01 02 04 and set as 02Neighbouring influence threshold θneighbor is selected from0 01 02 04 and set as 01 ese hyperparameters areselected through cross-validation with the best performance inmulti-HIN situations

34 Experiment I Errorless Link Classification Link classi-fication is the task of testing the model trustworthinessevaluation ability in new data In our experiment this ex-periment is implemented on the testing set It is a simplifiedtask compared with error link classification and link errordetection [11] where there is no noise in the training set ecorrect link is labelled as 1 while a false link is labelled as 0e model output lies in the range of [0 1] representing theprobability of one link being a correct onee trustworthinessscore is binarized into 0 (false link label) or 1 (correct link label)based on a threshold value Namely given a link the trust-worthiness score is calculated according to equation (5) If thetrustworthiness score is larger than the threshold this link islabelled as the correct link otherwise the false link

Here we compare our model with other models in basicsituations namely in separate HIN space and in combinedHIN spaces Besides in this experiment we train and test onthe original dataset not adding false links into training Alsoassuming we do not know where the link comes from theHIN source trustworthiness VG is set to 1

Experimental results are shown in Table 3 Obviouslyour model outperforms baseline methods in separate HINand combined HIN space which indicates that the hiddeninformation representing the reliability of links has been welllearned in our model We can analyse this result from twodifferent perspectives (1) single HIN and multi-HIN (2)

influence of inner-HIN characteristics From the view of singleand multi-HIN we can notice that when HINs are combinedin other words when data are augmented evaluation resultsare improving compared with those in single HIN situationis can be also proved from the threshold dropping in TransEDistMult and partial ComplexE results is proves thatthough previous methods do not directly consider the con-nection between HINs they can model the inherent rela-tionship between multiple HINs in other implicit ways whiletheir results are no better than our explicit relationshipmodeling method From the view of influence from inner-HINcharacteristics attribute characteristics are rather important inthe link classification task as we can see the accuracy ofMattribute- drops by nearly 5 compared with normal multi-HITE model Our model can achieve convergence by runningtwo or three epochs on the training data In the DBP-WDdataset it takes 45 minutes for one epoch e network em-bedding model usually takes two hours to network embeddingfollowed by the training of the classification task which costsmore time than ours

35 Experiment II Error Link Classification and Link ErrorDetection in TwoHINs e evaluation metric is the same asthe error detection task e first task error link classifi-cation is similar to the errorless link classification edifference is in the training set where error link classificationis trained on the noise training set while testing set andvalidation set are the same A small change in our modeltakes place here as our model cannot converge optimal inthe noise training set which means our model detects noiseWe first train and evaluate our model on the training set(both positive and negative-sampled links) and then retrainour model on the generated training set where labels arepredicted from the first trained modelrsquos evaluation esecond task link error detection is conducted on thetraining set e results are shown in Figure 4 in the formatof PR curve to test different modelsrsquo performance on thebinary classification task of detecting error links

In Table 4 we can see our model achieves the best ac-curacy and F1-score when compared with other models inthe multi-HIN setting for error link classification in 10 and40 neg situation In detail our model thought 786856816749 and 895114 links may be correct in the first train andevaluation round separately (where 757126 757135 and757139 are truly correct) where 10 20 and 40 noisedatasets contain links of 834973 910880 and 1062693When the noise links rise our model is more likely to ruleout false examples It reveals that our model achieves ef-fectiveness in modeling the inter-influence of HINs by usinginteractive links For TransE DistMult and ComplexE theyare robust to errors in the training set where errors slightlyreduce modelsrsquo performance It reveals that the traditionalembedding model can handle the problem of error links inmulti-HIN trustworthiness situations Two simplifiedmodels indicate the importance of inner characteristics andwe can conclude that the attributes are an important in-dicator of whether a link is true or not We think that thereason why attribute factors play a more important role in

8 Complexity

the evaluation is the number of attributes For 200000entities in this paperrsquos dataset there are 1612848 attributedescriptions in total while the interactions between HINsare subject to mutual influence conditions where thenumber of influence restrictions is far smaller than that ofattribute description information erefore deleting at-tribute information will affect the performance of the modelmore than deleting interaction information

In Figure 4 we can see our modelrsquos PR curve is aboveTransErsquos PR curves in the multi-HIN setting for link errordetection in all neg situation DistMult and ComplexE cannotdistinguish noise links in the training set so they are not plottedin Figure 4 It reveals that our model achieves effectiveness indetecting noise in multi-HIN situation Figure 4 also explainsthat there is a difference between the distributions of true linksand noise linksrsquo scores both in multi-HITE and TransE modelFor lines of TransE in Figure 4 there exist sharp drops in the leftpart is can be explained that the scores for true links andnoisy links do not have a large gap especially near extremethreshold where all links are thought as positive When thethreshold drops few links are thought to be noise while positivelinks are mistaken as noise which will influence the curve

sharply Also it can be observed that the PR curve rises whenthe number of noise links increases as it is easier to detect errorswhen errors are more frequent

36 Experiment III Error Link Classification in DifferentCombinations of HINs In this part we try to combinedifferent HINs with different numbers of HINs to comparethe performance of our model for error link classificatione results are shown in Table 5 In the case of combiningdifferent data sources our model proves robust from theresults At the same time with the increment of data sourcesour model can make more use of the interaction informationbetween different data sources to provide more sufficientevidence for link classification

37 Experiment IV Influential Method EvaluationFinally we test how the performance of two ILEs varies withdifferent values of the threshold and test the model in er-rorless link classification tasks Based on the results in Ta-ble 6 we can conclude that alignment ILE is more sensitiveto the threshold value because its threshold is close to 0

Table 2 Statistics of HINs constructed based on DBpedia and YAGO

HIN of nodes of edges of attributes of node types of linksDBpedia2 100000 643 760062 225 428952YAGO 100000 64 827682 295 502563

10

08

06

04

02

00100806040200

Prec

ision

10 negative

Recall

Multi-HITETransE

(a)

10

08

06

04

02

00

Prec

ision

20 negative

100806040200Recall

Multi-HITETransE

(b)

10

09

08

07

06

05

04

03Pr

ecisi

on

40 negative

100806040200Recall

Multi-HITETransE

(c)

Figure 4 Trade-off between precision and recall according to trustworthiness threshold

Table 3 Performance of link prediction from all the models on multiple HINs

DataMulti-HITE Minfluence- Mattribute- TransE R-GCN DistMult ComplexEacc F1 acc F1 acc F1 acc F1 acc F1 acc F1 acc F1

DBpedia1 mdash mdash 092 092 087 088 091 091 088 088 090 089 091 090Wikidata mdash mdash 094 094 093 089 090 090 089 089 088 088 089 088Combined 095 095 093 093 090 090 092 092 090 090 091 091 091 091ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

Complexity 9

compared with neighbour ILE It should be explained thatneighbour ILE (see equation (9)) has fewer componentscompared with alignment ILE (see equation (8))

4 Conclusions

In this paper we focus on the multi-HIN trustworthinessevaluation problem and propose a co-embedding spaceevaluation method Our model incorporates inner-HIN andinter-HIN characteristics into the loss function A deepneural network is used in our model to solve the problemWe compare our model with other single-HIN trustwor-thiness evaluation methods on a multi-HIN dataset Ex-perimental results demonstrate that our model can wellaccomplish the trustworthiness evaluation task and out-performs baseline models For future works we will focus onhow to incorporate truth value discovery into our modelBesides the processing of our model is complicated com-pared with other models such as TransE DistMult andComplexE erefore we need to find a concise way toevaluate trustworthiness while maintaining the current levelof accuracy

Data Availability

e HIN alignment dataset DBP-WD used to support thefindings of this study can be accessed through httpsgithubcomnju-websoftBootEA

Conflicts of Interest

e authors declare that there are no conflicts of interestregarding the publication of this paper

Acknowledgments

is study was supported by the National Key Research andDevelopment Program of China (grant nos2018YFC0830201 and 2017YFB1002801) the FundamentalResearch Funds for the Central Universities (grant nos4009009106 and 22120200184) and the CCF-Baidu OpenFund

References

[1] F M Harper and J A Konstan ldquoe movielens datasetshistory and contextrdquo ACM Transactions on Interactive In-telligent Systems vol 5 no 4 2015

[2] H Paulheim ldquoKnowledge graph refinement a survey ofapproaches and evaluation methodsrdquo Semantic Web vol 8no 3 pp 489ndash508 2017

[3] Y Lin Z Liu H Luan et al ldquoModeling relation paths forrepresentation learning of knowledge basesrdquo in Proceedings ofthe 2015 Conference on Empirical Methods in Natural Lan-guage Processing pp 705ndash714 Lisbon Portugal September2015

[4] C Yang Y Xiao Y Zhang et al ldquoHeterogeneous networkrepresentation learning survey benchmark evaluation andbeyondrdquo 2020 httparxivorgabs200400216

Table 4 Performance of all the methods over data with different levels of noise

DataMulti-HITE Minfluence- Mattribute- TransE R-GCN DistMult ComplexEacc F1 acc F1 acc F1 acc F1 acc F1 acc F1 acc F1

10 neg 091 092 090 090 090 089 091 090 090 090 091 091 091 09120 neg 090 091 089 089 089 089 091 090 089 089 090 090 090 09040 neg 092 092 090 089 089 089 090 090 089 089 088 088 088 088ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

Table 6 Results (accuracyF1-score) of multi-TW using different values of thresholds for ILE functions

Alignment neighbour 00 01 02 0400 0899508981 0899409020 0896808997 084330849501 0872308735 0893708975 0905809027 088830890402 0863908673 0948009482 0923309253 091040907804 0843808453 0922509237 0940409395 0907509017

Table 5 Performance of multi-HITE over different datasets with different levels of noise

DatasetDBP1-WD DBP2-YAGO YAGO-WD DBP1-YAGO-

DBP2-WDacc F1 acc F1 acc F1 acc F1

10 neg 091 092 090 090 091 091 091 09220 neg 090 091 090 090 089 089 091 09240 neg 092 092 089 089 089 089 090 090ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

10 Complexity

244 Graph Energy Transferring Information HIN is aspecial heterogeneous network where basic network anal-ysis techniques can be utilized If a link has strong con-nectivity or other favourable graphic features it alsoindicates sufficient proofs in the HIN Here we mainlyconcentrate on the energy flow between the head node andthe tail node In this situation we can construct a singlegraph for each node in the HIN where the node is the centralnode All nodes associated with the central node are takeninto consideration e energy transferring can be modeledwith page-rank-like method [11 24] However iteratingthrough a graph will have a long tail phenomenon where themajority of nodes have low energy Hence we add severalfeatures like in-degree and out-degree into consideration as

g Ns No( 1113857 ε timese Ns No( 1113857

e Ns( 1113857+(1 minus ε) times fNsNo

din dout( 1113857

(4)

Here g(Ns No) is the combined energy transmittedbetween embedded nodes Ns and No e(Ns) represents theenergy thatNs holds in the graph and e(Ns No) denotes theenergy flowing from Ns to No fNsNo

(din dout) is the energyfunction determined by the in-degree and out-degree of Ns

and No Besides ϵ is a hyperparameter balancing the page-rank-like energy and degree-determined function

After obtaining the above four different features eachfeature is transmitted to the hidden dimension by a separatelinear forward layer followed by the combination of allfeatures of the same node or link as shown in Figure 2 Afterobtaining the hidden representation z isin Rn (n is the size ofhidden states) of the head node tail node and edge we usethe following function to model the trustworthiness

T wtG1113872 1113873 VG times σ z

⊤p zs ⊙ zo( 11138571113872 1113873 (5)

in which σ(middot) is the activation function of the neural networkand ⊙ denotes the elementwise multiplication e scorefunction is related to the score of the linkrsquos located HIN so itneeds to multiply the located HINrsquos score VG Part of thisfunction is initially adopted by DistMult [25]

With all these we can calculate traditional trans-mission loss in every single HIN of S is loss is used tomodel local information of a HIN For a specific link t

shown in the original dataset we call it a positive link enegative links set c(G) is generated by replacing the headand tail node of t with other nodes so that negative linksdo not appear in the original dataset Specifically for aHIN G isin S let T(wt

G) and T(wtprimeG) be the positive score and

negative score respectively for the positive link t andnegative link tprime in this HIN and the relational loss formultiple HINs is

Lrel 1113944GisinS

1113944tisinG

1113944

tprimeisinc(G)

max 0 c minus T wtG1113872 1113873 + T w

tprimeG1113874 11138751113874 1113875)⎛⎝ (6)

Here c is the margin between positive links and negativelinks and c(G) is a set of corrupted links tprime corresponding tolinks t in the HIN G

25 Inter-HINEvaluation In multi-HIN settings inter-HINinfluence is another crucial component of the evaluationerefore we further include inter-HIN influence into theloss function for trustworthiness evaluation If a link(Ns Ep No)G in one HIN G has influence on a link(Nsprime Epprime Noprime)Gprime in another HIN Gprime we call these two linksinteractive links e way how interactive links influenceeach other in different HINs is called Interactive Links Effect(ILE) We use h(middot) to denote ILE and propose three differentkinds of ILEs that are utilized in our method for modelinginteractivity from three levels literal level semantic leveland graph level

251 Value ILE If two nodes refer to the same real-worldnode they should share similar attributes [26] Based on thisassumption from the literal level the value ILE can bedefined to evaluate influence for nodes of a pair of interactivelinks Let p(N) denote the attribute value set of the node N

and c(N1 N2) represent the common attributes shared bynodes N1 and N2 in the embedding space value ILE iscomputed via

hvalue Ns Ep No1113872 1113873G Nsprime Epprime Noprime1113872 1113873

Gprime1113872 1113873

c Ns Nsprime( 1113857

11138681113868111386811138681113868111386811138681113868

p Ns( 11138571113868111386811138681113868

1113868111386811138681113868 + p Nsprime( 11138571113868111386811138681113868

1113868111386811138681113868 minus c Ns Nsprime( 11138571113868111386811138681113868

1113868111386811138681113868

+c No Noprime( 1113857

11138681113868111386811138681113868111386811138681113868

p No( 11138571113868111386811138681113868

1113868111386811138681113868 + p Noprime( 11138571113868111386811138681113868

1113868111386811138681113868 minus c No Noprime( 11138571113868111386811138681113868

1113868111386811138681113868

(7)

in which | middot | is the number of elements in a set

252 Alignment ILE If two links have aligned nodes oredges they should be interactive Additionally alignednodes or edges are close in embedding space [19] us wecan assume that if two links are interactive at least one of thenodes or edges should be close in the embedding spaceBesides only one aligned part is not sufficient as many linksdescribe the same node from a different perspective like ldquoYaoMing-wife-Ye Lirdquo and ldquoYao Ming- teammate-TracyMcGradyrdquo us we add separate distances of co-spaceembedding of nodes and edges together and restrict thedistance of interactive links of this embedding distance todetermine whether two links are interactive or not Withthis we define the alignment ILE as

halignment Ns Ep No1113872 1113873G Nsprime Epprime Noprime1113872 1113873

Gprime1113872 1113873

Ns minus Nsprime

2 + No minus Noprime

2 + Ep minus Epprime

2

(8)

253 Neighbour ILE According to the semantic similaritywork [27] if nodes share similar surroundings there exists apossibility that they have semantic similarity in other wordsthey are interactive us we use neighbouring nodes of thehead and tail nodes to evaluate the ILE Let n(N) denote theaverage embedding for neighbouring nodes of node N thenwe have

Complexity 5

hneighbour Ns Ep No1113872 1113873G Nsprime Epprime Noprime1113872 1113873

Gprime1113872 1113873

n Ns( 1113857 minus n Nsprime( 1113857

2 + n No( 1113857 minus n Noprime( 1113857

2(9)

With all three ILE functions we can inherently use themto determine if two links from two different HINs haveinfluence on each other Specifically we design a matrix I torepresent the interaction of two HINs For a certain linki isin G and another link j isin Gprime their interactive status isdefined as

Iij 1 hvalue(i j)lt θvalue( 1113857 + 1 halignment(i j)lt θalignment1113872 1113873

+ 1 hneighbour(i j)lt θneighbour1113872 1113873

(10)

where 1(c) is the indication function with 1(c) 1 ifstatement c is true otherwise 1(c) 0 ree ILE thresholdsincluding θvalue θalignment and θneighbour are playing the rolesof hyperparameters

Intuitively the trustworthiness of two interactive linksshould be close to each other However there exist caseswhere two links are interactive but one has large trust-worthiness while that of another link is low us trust-worthiness scores of two links should be deductedSpecifically for i isin G and j isin Gprime we design the influencebetween links i and j as R(i j) 0 if Iij 0 otherwise

R(i j) T w

iG1113872 1113873 minus T w

j

G1113872 111387311138681113868111386811138681113868

11138681113868111386811138681113868 if T wiG1113872 1113873 minus T w

j

G1113872 111387311138681113868111386811138681113868

11138681113868111386811138681113868 le ρ

T wiG1113872 1113873 + T w

j

G1113872 111387311138681113868111386811138681113868

11138681113868111386811138681113868 if T wiG1113872 1113873 minus T w

j

G1113872 111387311138681113868111386811138681113868

11138681113868111386811138681113868 gt ρ

⎧⎪⎨

⎪⎩

(11)

Combining all the inter-HIN influences of interactivelinks we introduce the interactive loss

Lint 1113944(ij)isinIn

R(i j)(12)

with a set of interactive links In

26 Model Training in Multi-HIN Settings To obtain thetrustworthiness both in single-HIN and multi-HIN per-spectives we introduce the multi-task objective functionwhich models the contextual information of local and in-fluential information We integrate relational loss with in-teractive influential loss by using a hyperparameter β as thetrade-off and obtain

L β Lrel +(1 minus β)Lint (13)

e training process of our model can be separated intotwo main procedures

In the first phase we minimize the loss function with aneural network We put inner graph characteristics into theneural network including node attributes node-type con-straints node and edge embeddings and graph energytransferring information en we separately combinenode-related features into hidden representations of nodesand combine node-type constraints with edge embeddingsinto hidden representations of edges e detailed structure

of this neural network is shown in Figure 3 In the networke1(middot) is the node energy function for a certain node e2(middot) isthe node embedding e3(middot) denotes the attribute embeddinge4(middot) represents the edge embedding and e5(middot) is the typeembedding of a node Moreover weight matricesWe1h We2h We3h We4h andWe5h project different featuresto its hidden representations Function r(middot) projects em-beddings in the same space Besides Wer Wnr and War

combine node-related features to generate hidden repre-sentations of nodes while Wlr and Wtr combine edge-relatedfeatures to generate hidden representations of edges Inaddition we further attach an extra regularizer to control thesize of all the parameters Let Ω We1h We2h1113864

We3h We4h We5h Wer Wnr War Wlr Wtr denote a set ofall the weight parameters in the neural network and theregularized loss function becomes

L β Lrel +(1 minus β)Lint + 1113944W isinΩ

W2 (14)

After evaluating the score for all links in multiple HINswe can iteratively use equations (2) and (3) to update thetrustworthiness of HIN and each link until results convergee pseudocode of our method is shown in Algorithm 1

3 Results and Discussion

In this section we implement comprehensive experiments totest the performance of ourmodel from various perspectives

31 Dataset We choose to use two HIN alignment datasetsDBP-WD and DBP-YG [28] in our experiments e firstdataset is sampled from DBpedia and Wikidata (ie twoHINs) and contains 100 thousand aligned nodes while thesecond dataset is extracted from DBpedia and YAGO withsame 100 thousand aligned nodes Information about nodetypes is attached to the data according to DBpedia type in-formation Tables 1 and 2 provide statistics of these datasets inthe experimentWe split this dataset into training testing andvalidation set with the ratio as 10 1 1 Besides DBP-WDdataset is used in all experiments while DBP-YG is only usedin Experiment III DBpedia in DBP-WD is noted as DBpediawhile DBpedia in DBP-YG is noted as DBpedia2

Furthermore we generate some negative links in thedataset to simulate noisy data with erroneous links Spe-cifically for one negative link we replace the head or tailnode of a positive link so that it does not appear in theoriginal dataset Other characteristics including node at-tributes node types and neighbouring nodes remain thesame with the replaced node To obtain a different level ofnoise similar to the simulation implemented in [11]multiple noisy datasets are simulated with the percentage ofreplaced negative links of 10 20 and 40 Notice thatthe noises are only added to the training set

32 Baseline Methods As there is no trustworthiness eval-uation method previously proposed in multi-HIN settingswe directly extend traditional embedding methods intomulti-HIN cases as baseline methods We use embedding

6 Complexity

methods including TransE [9] R-GCN [29] DistMult [25]and ComplexE [30] However we are not using originalversions of these algorithms Instead we use refined modelsfrom OpenKE [31] to compare our model with currentfrequently used model versions Because these methods donot generate trustworthiness scores directly we calculate thetrustworthiness score according to their threshold where alink score that is larger than the threshold is thought as 0otherwise as 1 Moreover because these methods are notproposed for multi-HIN cases they are not able to integrally

analyse multiple HINs us we use these methods toevaluate trustworthiness scores separately on different HINs

Additionally we simplify our model for comparisonsbetween our model with its simplified version We use twosimplified models to show the effectiveness of our modelsetting ese two versions of methods are denoted asMattribute- and Minfluence- respectively Mattribute- is a versionignoring node attribute inner-HIN characteristic where inthe inner-HIN evaluation stage only node-type constraintcontextual embedding information and graph energy

Typeembedding

Edgeembedding

Attributeembedding

Nodeembedding

Nodeenergyfeatures

Scorefunction

Relationalloss in source

Head node hiddenrepresentation

Tail node hiddenrepresentation

Edge hiddenrepresentation

Influentialtriplet loss

We1h

We1h

We2h

We3h

We3h

We4h

We5h

We2h

WerWnrWar

WerWnrWar

e1 (Ns)

e1 (No)

e2 (Ns)

e2 (No)

e3 (Ns)

e3 (No)

e4 (Ep)

e5 (Ep)

WlrWtr

r (e1(Ns)))r (e2(Ns)))r (e3(Ns)))

r (e1(No)))r (e2(No)))r (e3(No)))

r (e4(Ep)))r (e5(Ep)))

Zs

Zo

Zp

Figure 3 Detailed structure of neural network

REQUIRE S maximal epochs of training T source trustworthiness Vfor G isin S do

VGlarr1end forwhile Tgt 0 dofor G isin S dofor t isin G doCalculate link score according to (2)

end forend forfor G isin S doUpdate source trustworthiness VG

end forTlarrT minus 1Backpropagate loss and update parameters in the neural network

end while

ALGORITHM 1 Multi-HIN trustworthiness evaluation

Table 1 Statistics of HINs constructed based on DBpedia and Wikidata

HIN of nodes of edges of attributes of node types of linksDBpedia1 100000 686 622331 225 463294Wikidata 100000 946 990517 295 448774

Complexity 7

transferring information are taken into considerationMinfluence- is the simplified model neglecting influencemethods through ILE betweenHINs where in the inter-HINevaluation stage all ILEs are set to 0 so that there exist nointeractive links

33 Hyperparameter Setup In the neural network of ourmodel the size of the node embedding is 256 the edgeembedding size is 64 and the attribute embedding size is512 e hidden size and combined nodeedge size is 512We use the ReLU function as the activation function in themodel e sigmoid function is used in the output layer forproducing the trustworthiness score e value of learningrate is selected from 01 001 0001 and set as 0001 andthe batch size is selected from 10 40 100 and set as 0001ese hyperparameters are selected through cross-validationwith the best performance in multi-HIN situations

e value of margin c in our function is selected from05 08 1 Moreover we use l2-norm as the regularizer tocontrol all the trainable parameters in the neural networkInner-HIN loss and cross-HIN balance parameter β is selectedfrom 03 05 07 and set as 05 Graphic balancing parameterϵ is selected from 03 05 07 and set as 05 Influencemethodcontrol threshold parameter ρ is selected from 001 01 05

and set as 01 Value influence threshold θvalue is selected from05 08 09 and set as 08 Alignment influence thresholdθalignment is selected from 0 01 02 04 and set as 02Neighbouring influence threshold θneighbor is selected from0 01 02 04 and set as 01 ese hyperparameters areselected through cross-validation with the best performance inmulti-HIN situations

34 Experiment I Errorless Link Classification Link classi-fication is the task of testing the model trustworthinessevaluation ability in new data In our experiment this ex-periment is implemented on the testing set It is a simplifiedtask compared with error link classification and link errordetection [11] where there is no noise in the training set ecorrect link is labelled as 1 while a false link is labelled as 0e model output lies in the range of [0 1] representing theprobability of one link being a correct onee trustworthinessscore is binarized into 0 (false link label) or 1 (correct link label)based on a threshold value Namely given a link the trust-worthiness score is calculated according to equation (5) If thetrustworthiness score is larger than the threshold this link islabelled as the correct link otherwise the false link

Here we compare our model with other models in basicsituations namely in separate HIN space and in combinedHIN spaces Besides in this experiment we train and test onthe original dataset not adding false links into training Alsoassuming we do not know where the link comes from theHIN source trustworthiness VG is set to 1

Experimental results are shown in Table 3 Obviouslyour model outperforms baseline methods in separate HINand combined HIN space which indicates that the hiddeninformation representing the reliability of links has been welllearned in our model We can analyse this result from twodifferent perspectives (1) single HIN and multi-HIN (2)

influence of inner-HIN characteristics From the view of singleand multi-HIN we can notice that when HINs are combinedin other words when data are augmented evaluation resultsare improving compared with those in single HIN situationis can be also proved from the threshold dropping in TransEDistMult and partial ComplexE results is proves thatthough previous methods do not directly consider the con-nection between HINs they can model the inherent rela-tionship between multiple HINs in other implicit ways whiletheir results are no better than our explicit relationshipmodeling method From the view of influence from inner-HINcharacteristics attribute characteristics are rather important inthe link classification task as we can see the accuracy ofMattribute- drops by nearly 5 compared with normal multi-HITE model Our model can achieve convergence by runningtwo or three epochs on the training data In the DBP-WDdataset it takes 45 minutes for one epoch e network em-bedding model usually takes two hours to network embeddingfollowed by the training of the classification task which costsmore time than ours

35 Experiment II Error Link Classification and Link ErrorDetection in TwoHINs e evaluation metric is the same asthe error detection task e first task error link classifi-cation is similar to the errorless link classification edifference is in the training set where error link classificationis trained on the noise training set while testing set andvalidation set are the same A small change in our modeltakes place here as our model cannot converge optimal inthe noise training set which means our model detects noiseWe first train and evaluate our model on the training set(both positive and negative-sampled links) and then retrainour model on the generated training set where labels arepredicted from the first trained modelrsquos evaluation esecond task link error detection is conducted on thetraining set e results are shown in Figure 4 in the formatof PR curve to test different modelsrsquo performance on thebinary classification task of detecting error links

In Table 4 we can see our model achieves the best ac-curacy and F1-score when compared with other models inthe multi-HIN setting for error link classification in 10 and40 neg situation In detail our model thought 786856816749 and 895114 links may be correct in the first train andevaluation round separately (where 757126 757135 and757139 are truly correct) where 10 20 and 40 noisedatasets contain links of 834973 910880 and 1062693When the noise links rise our model is more likely to ruleout false examples It reveals that our model achieves ef-fectiveness in modeling the inter-influence of HINs by usinginteractive links For TransE DistMult and ComplexE theyare robust to errors in the training set where errors slightlyreduce modelsrsquo performance It reveals that the traditionalembedding model can handle the problem of error links inmulti-HIN trustworthiness situations Two simplifiedmodels indicate the importance of inner characteristics andwe can conclude that the attributes are an important in-dicator of whether a link is true or not We think that thereason why attribute factors play a more important role in

8 Complexity

the evaluation is the number of attributes For 200000entities in this paperrsquos dataset there are 1612848 attributedescriptions in total while the interactions between HINsare subject to mutual influence conditions where thenumber of influence restrictions is far smaller than that ofattribute description information erefore deleting at-tribute information will affect the performance of the modelmore than deleting interaction information

In Figure 4 we can see our modelrsquos PR curve is aboveTransErsquos PR curves in the multi-HIN setting for link errordetection in all neg situation DistMult and ComplexE cannotdistinguish noise links in the training set so they are not plottedin Figure 4 It reveals that our model achieves effectiveness indetecting noise in multi-HIN situation Figure 4 also explainsthat there is a difference between the distributions of true linksand noise linksrsquo scores both in multi-HITE and TransE modelFor lines of TransE in Figure 4 there exist sharp drops in the leftpart is can be explained that the scores for true links andnoisy links do not have a large gap especially near extremethreshold where all links are thought as positive When thethreshold drops few links are thought to be noise while positivelinks are mistaken as noise which will influence the curve

sharply Also it can be observed that the PR curve rises whenthe number of noise links increases as it is easier to detect errorswhen errors are more frequent

36 Experiment III Error Link Classification in DifferentCombinations of HINs In this part we try to combinedifferent HINs with different numbers of HINs to comparethe performance of our model for error link classificatione results are shown in Table 5 In the case of combiningdifferent data sources our model proves robust from theresults At the same time with the increment of data sourcesour model can make more use of the interaction informationbetween different data sources to provide more sufficientevidence for link classification

37 Experiment IV Influential Method EvaluationFinally we test how the performance of two ILEs varies withdifferent values of the threshold and test the model in er-rorless link classification tasks Based on the results in Ta-ble 6 we can conclude that alignment ILE is more sensitiveto the threshold value because its threshold is close to 0

Table 2 Statistics of HINs constructed based on DBpedia and YAGO

HIN of nodes of edges of attributes of node types of linksDBpedia2 100000 643 760062 225 428952YAGO 100000 64 827682 295 502563

10

08

06

04

02

00100806040200

Prec

ision

10 negative

Recall

Multi-HITETransE

(a)

10

08

06

04

02

00

Prec

ision

20 negative

100806040200Recall

Multi-HITETransE

(b)

10

09

08

07

06

05

04

03Pr

ecisi

on

40 negative

100806040200Recall

Multi-HITETransE

(c)

Figure 4 Trade-off between precision and recall according to trustworthiness threshold

Table 3 Performance of link prediction from all the models on multiple HINs

DataMulti-HITE Minfluence- Mattribute- TransE R-GCN DistMult ComplexEacc F1 acc F1 acc F1 acc F1 acc F1 acc F1 acc F1

DBpedia1 mdash mdash 092 092 087 088 091 091 088 088 090 089 091 090Wikidata mdash mdash 094 094 093 089 090 090 089 089 088 088 089 088Combined 095 095 093 093 090 090 092 092 090 090 091 091 091 091ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

Complexity 9

compared with neighbour ILE It should be explained thatneighbour ILE (see equation (9)) has fewer componentscompared with alignment ILE (see equation (8))

4 Conclusions

In this paper we focus on the multi-HIN trustworthinessevaluation problem and propose a co-embedding spaceevaluation method Our model incorporates inner-HIN andinter-HIN characteristics into the loss function A deepneural network is used in our model to solve the problemWe compare our model with other single-HIN trustwor-thiness evaluation methods on a multi-HIN dataset Ex-perimental results demonstrate that our model can wellaccomplish the trustworthiness evaluation task and out-performs baseline models For future works we will focus onhow to incorporate truth value discovery into our modelBesides the processing of our model is complicated com-pared with other models such as TransE DistMult andComplexE erefore we need to find a concise way toevaluate trustworthiness while maintaining the current levelof accuracy

Data Availability

e HIN alignment dataset DBP-WD used to support thefindings of this study can be accessed through httpsgithubcomnju-websoftBootEA

Conflicts of Interest

e authors declare that there are no conflicts of interestregarding the publication of this paper

Acknowledgments

is study was supported by the National Key Research andDevelopment Program of China (grant nos2018YFC0830201 and 2017YFB1002801) the FundamentalResearch Funds for the Central Universities (grant nos4009009106 and 22120200184) and the CCF-Baidu OpenFund

References

[1] F M Harper and J A Konstan ldquoe movielens datasetshistory and contextrdquo ACM Transactions on Interactive In-telligent Systems vol 5 no 4 2015

[2] H Paulheim ldquoKnowledge graph refinement a survey ofapproaches and evaluation methodsrdquo Semantic Web vol 8no 3 pp 489ndash508 2017

[3] Y Lin Z Liu H Luan et al ldquoModeling relation paths forrepresentation learning of knowledge basesrdquo in Proceedings ofthe 2015 Conference on Empirical Methods in Natural Lan-guage Processing pp 705ndash714 Lisbon Portugal September2015

[4] C Yang Y Xiao Y Zhang et al ldquoHeterogeneous networkrepresentation learning survey benchmark evaluation andbeyondrdquo 2020 httparxivorgabs200400216

Table 4 Performance of all the methods over data with different levels of noise

DataMulti-HITE Minfluence- Mattribute- TransE R-GCN DistMult ComplexEacc F1 acc F1 acc F1 acc F1 acc F1 acc F1 acc F1

10 neg 091 092 090 090 090 089 091 090 090 090 091 091 091 09120 neg 090 091 089 089 089 089 091 090 089 089 090 090 090 09040 neg 092 092 090 089 089 089 090 090 089 089 088 088 088 088ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

Table 6 Results (accuracyF1-score) of multi-TW using different values of thresholds for ILE functions

Alignment neighbour 00 01 02 0400 0899508981 0899409020 0896808997 084330849501 0872308735 0893708975 0905809027 088830890402 0863908673 0948009482 0923309253 091040907804 0843808453 0922509237 0940409395 0907509017

Table 5 Performance of multi-HITE over different datasets with different levels of noise

DatasetDBP1-WD DBP2-YAGO YAGO-WD DBP1-YAGO-

DBP2-WDacc F1 acc F1 acc F1 acc F1

10 neg 091 092 090 090 091 091 091 09220 neg 090 091 090 090 089 089 091 09240 neg 092 092 089 089 089 089 090 090ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

10 Complexity

hneighbour Ns Ep No1113872 1113873G Nsprime Epprime Noprime1113872 1113873

Gprime1113872 1113873

n Ns( 1113857 minus n Nsprime( 1113857

2 + n No( 1113857 minus n Noprime( 1113857

2(9)

With all three ILE functions we can inherently use themto determine if two links from two different HINs haveinfluence on each other Specifically we design a matrix I torepresent the interaction of two HINs For a certain linki isin G and another link j isin Gprime their interactive status isdefined as

Iij 1 hvalue(i j)lt θvalue( 1113857 + 1 halignment(i j)lt θalignment1113872 1113873

+ 1 hneighbour(i j)lt θneighbour1113872 1113873

(10)

where 1(c) is the indication function with 1(c) 1 ifstatement c is true otherwise 1(c) 0 ree ILE thresholdsincluding θvalue θalignment and θneighbour are playing the rolesof hyperparameters

Intuitively the trustworthiness of two interactive linksshould be close to each other However there exist caseswhere two links are interactive but one has large trust-worthiness while that of another link is low us trust-worthiness scores of two links should be deductedSpecifically for i isin G and j isin Gprime we design the influencebetween links i and j as R(i j) 0 if Iij 0 otherwise

R(i j) T w

iG1113872 1113873 minus T w

j

G1113872 111387311138681113868111386811138681113868

11138681113868111386811138681113868 if T wiG1113872 1113873 minus T w

j

G1113872 111387311138681113868111386811138681113868

11138681113868111386811138681113868 le ρ

T wiG1113872 1113873 + T w

j

G1113872 111387311138681113868111386811138681113868

11138681113868111386811138681113868 if T wiG1113872 1113873 minus T w

j

G1113872 111387311138681113868111386811138681113868

11138681113868111386811138681113868 gt ρ

⎧⎪⎨

⎪⎩

(11)

Combining all the inter-HIN influences of interactivelinks we introduce the interactive loss

Lint 1113944(ij)isinIn

R(i j)(12)

with a set of interactive links In

26 Model Training in Multi-HIN Settings To obtain thetrustworthiness both in single-HIN and multi-HIN per-spectives we introduce the multi-task objective functionwhich models the contextual information of local and in-fluential information We integrate relational loss with in-teractive influential loss by using a hyperparameter β as thetrade-off and obtain

L β Lrel +(1 minus β)Lint (13)

e training process of our model can be separated intotwo main procedures

In the first phase we minimize the loss function with aneural network We put inner graph characteristics into theneural network including node attributes node-type con-straints node and edge embeddings and graph energytransferring information en we separately combinenode-related features into hidden representations of nodesand combine node-type constraints with edge embeddingsinto hidden representations of edges e detailed structure

of this neural network is shown in Figure 3 In the networke1(middot) is the node energy function for a certain node e2(middot) isthe node embedding e3(middot) denotes the attribute embeddinge4(middot) represents the edge embedding and e5(middot) is the typeembedding of a node Moreover weight matricesWe1h We2h We3h We4h andWe5h project different featuresto its hidden representations Function r(middot) projects em-beddings in the same space Besides Wer Wnr and War

combine node-related features to generate hidden repre-sentations of nodes while Wlr and Wtr combine edge-relatedfeatures to generate hidden representations of edges Inaddition we further attach an extra regularizer to control thesize of all the parameters Let Ω We1h We2h1113864

We3h We4h We5h Wer Wnr War Wlr Wtr denote a set ofall the weight parameters in the neural network and theregularized loss function becomes

L β Lrel +(1 minus β)Lint + 1113944W isinΩ

W2 (14)

After evaluating the score for all links in multiple HINswe can iteratively use equations (2) and (3) to update thetrustworthiness of HIN and each link until results convergee pseudocode of our method is shown in Algorithm 1

3 Results and Discussion

In this section we implement comprehensive experiments totest the performance of ourmodel from various perspectives

31 Dataset We choose to use two HIN alignment datasetsDBP-WD and DBP-YG [28] in our experiments e firstdataset is sampled from DBpedia and Wikidata (ie twoHINs) and contains 100 thousand aligned nodes while thesecond dataset is extracted from DBpedia and YAGO withsame 100 thousand aligned nodes Information about nodetypes is attached to the data according to DBpedia type in-formation Tables 1 and 2 provide statistics of these datasets inthe experimentWe split this dataset into training testing andvalidation set with the ratio as 10 1 1 Besides DBP-WDdataset is used in all experiments while DBP-YG is only usedin Experiment III DBpedia in DBP-WD is noted as DBpediawhile DBpedia in DBP-YG is noted as DBpedia2

Furthermore we generate some negative links in thedataset to simulate noisy data with erroneous links Spe-cifically for one negative link we replace the head or tailnode of a positive link so that it does not appear in theoriginal dataset Other characteristics including node at-tributes node types and neighbouring nodes remain thesame with the replaced node To obtain a different level ofnoise similar to the simulation implemented in [11]multiple noisy datasets are simulated with the percentage ofreplaced negative links of 10 20 and 40 Notice thatthe noises are only added to the training set

32 Baseline Methods As there is no trustworthiness eval-uation method previously proposed in multi-HIN settingswe directly extend traditional embedding methods intomulti-HIN cases as baseline methods We use embedding

6 Complexity

methods including TransE [9] R-GCN [29] DistMult [25]and ComplexE [30] However we are not using originalversions of these algorithms Instead we use refined modelsfrom OpenKE [31] to compare our model with currentfrequently used model versions Because these methods donot generate trustworthiness scores directly we calculate thetrustworthiness score according to their threshold where alink score that is larger than the threshold is thought as 0otherwise as 1 Moreover because these methods are notproposed for multi-HIN cases they are not able to integrally

analyse multiple HINs us we use these methods toevaluate trustworthiness scores separately on different HINs

Additionally we simplify our model for comparisonsbetween our model with its simplified version We use twosimplified models to show the effectiveness of our modelsetting ese two versions of methods are denoted asMattribute- and Minfluence- respectively Mattribute- is a versionignoring node attribute inner-HIN characteristic where inthe inner-HIN evaluation stage only node-type constraintcontextual embedding information and graph energy

Typeembedding

Edgeembedding

Attributeembedding

Nodeembedding

Nodeenergyfeatures

Scorefunction

Relationalloss in source

Head node hiddenrepresentation

Tail node hiddenrepresentation

Edge hiddenrepresentation

Influentialtriplet loss

We1h

We1h

We2h

We3h

We3h

We4h

We5h

We2h

WerWnrWar

WerWnrWar

e1 (Ns)

e1 (No)

e2 (Ns)

e2 (No)

e3 (Ns)

e3 (No)

e4 (Ep)

e5 (Ep)

WlrWtr

r (e1(Ns)))r (e2(Ns)))r (e3(Ns)))

r (e1(No)))r (e2(No)))r (e3(No)))

r (e4(Ep)))r (e5(Ep)))

Zs

Zo

Zp

Figure 3 Detailed structure of neural network

REQUIRE S maximal epochs of training T source trustworthiness Vfor G isin S do

VGlarr1end forwhile Tgt 0 dofor G isin S dofor t isin G doCalculate link score according to (2)

end forend forfor G isin S doUpdate source trustworthiness VG

end forTlarrT minus 1Backpropagate loss and update parameters in the neural network

end while

ALGORITHM 1 Multi-HIN trustworthiness evaluation

Table 1 Statistics of HINs constructed based on DBpedia and Wikidata

HIN of nodes of edges of attributes of node types of linksDBpedia1 100000 686 622331 225 463294Wikidata 100000 946 990517 295 448774

Complexity 7

transferring information are taken into considerationMinfluence- is the simplified model neglecting influencemethods through ILE betweenHINs where in the inter-HINevaluation stage all ILEs are set to 0 so that there exist nointeractive links

33 Hyperparameter Setup In the neural network of ourmodel the size of the node embedding is 256 the edgeembedding size is 64 and the attribute embedding size is512 e hidden size and combined nodeedge size is 512We use the ReLU function as the activation function in themodel e sigmoid function is used in the output layer forproducing the trustworthiness score e value of learningrate is selected from 01 001 0001 and set as 0001 andthe batch size is selected from 10 40 100 and set as 0001ese hyperparameters are selected through cross-validationwith the best performance in multi-HIN situations

e value of margin c in our function is selected from05 08 1 Moreover we use l2-norm as the regularizer tocontrol all the trainable parameters in the neural networkInner-HIN loss and cross-HIN balance parameter β is selectedfrom 03 05 07 and set as 05 Graphic balancing parameterϵ is selected from 03 05 07 and set as 05 Influencemethodcontrol threshold parameter ρ is selected from 001 01 05

and set as 01 Value influence threshold θvalue is selected from05 08 09 and set as 08 Alignment influence thresholdθalignment is selected from 0 01 02 04 and set as 02Neighbouring influence threshold θneighbor is selected from0 01 02 04 and set as 01 ese hyperparameters areselected through cross-validation with the best performance inmulti-HIN situations

34 Experiment I Errorless Link Classification Link classi-fication is the task of testing the model trustworthinessevaluation ability in new data In our experiment this ex-periment is implemented on the testing set It is a simplifiedtask compared with error link classification and link errordetection [11] where there is no noise in the training set ecorrect link is labelled as 1 while a false link is labelled as 0e model output lies in the range of [0 1] representing theprobability of one link being a correct onee trustworthinessscore is binarized into 0 (false link label) or 1 (correct link label)based on a threshold value Namely given a link the trust-worthiness score is calculated according to equation (5) If thetrustworthiness score is larger than the threshold this link islabelled as the correct link otherwise the false link

Here we compare our model with other models in basicsituations namely in separate HIN space and in combinedHIN spaces Besides in this experiment we train and test onthe original dataset not adding false links into training Alsoassuming we do not know where the link comes from theHIN source trustworthiness VG is set to 1

Experimental results are shown in Table 3 Obviouslyour model outperforms baseline methods in separate HINand combined HIN space which indicates that the hiddeninformation representing the reliability of links has been welllearned in our model We can analyse this result from twodifferent perspectives (1) single HIN and multi-HIN (2)

influence of inner-HIN characteristics From the view of singleand multi-HIN we can notice that when HINs are combinedin other words when data are augmented evaluation resultsare improving compared with those in single HIN situationis can be also proved from the threshold dropping in TransEDistMult and partial ComplexE results is proves thatthough previous methods do not directly consider the con-nection between HINs they can model the inherent rela-tionship between multiple HINs in other implicit ways whiletheir results are no better than our explicit relationshipmodeling method From the view of influence from inner-HINcharacteristics attribute characteristics are rather important inthe link classification task as we can see the accuracy ofMattribute- drops by nearly 5 compared with normal multi-HITE model Our model can achieve convergence by runningtwo or three epochs on the training data In the DBP-WDdataset it takes 45 minutes for one epoch e network em-bedding model usually takes two hours to network embeddingfollowed by the training of the classification task which costsmore time than ours

35 Experiment II Error Link Classification and Link ErrorDetection in TwoHINs e evaluation metric is the same asthe error detection task e first task error link classifi-cation is similar to the errorless link classification edifference is in the training set where error link classificationis trained on the noise training set while testing set andvalidation set are the same A small change in our modeltakes place here as our model cannot converge optimal inthe noise training set which means our model detects noiseWe first train and evaluate our model on the training set(both positive and negative-sampled links) and then retrainour model on the generated training set where labels arepredicted from the first trained modelrsquos evaluation esecond task link error detection is conducted on thetraining set e results are shown in Figure 4 in the formatof PR curve to test different modelsrsquo performance on thebinary classification task of detecting error links

In Table 4 we can see our model achieves the best ac-curacy and F1-score when compared with other models inthe multi-HIN setting for error link classification in 10 and40 neg situation In detail our model thought 786856816749 and 895114 links may be correct in the first train andevaluation round separately (where 757126 757135 and757139 are truly correct) where 10 20 and 40 noisedatasets contain links of 834973 910880 and 1062693When the noise links rise our model is more likely to ruleout false examples It reveals that our model achieves ef-fectiveness in modeling the inter-influence of HINs by usinginteractive links For TransE DistMult and ComplexE theyare robust to errors in the training set where errors slightlyreduce modelsrsquo performance It reveals that the traditionalembedding model can handle the problem of error links inmulti-HIN trustworthiness situations Two simplifiedmodels indicate the importance of inner characteristics andwe can conclude that the attributes are an important in-dicator of whether a link is true or not We think that thereason why attribute factors play a more important role in

8 Complexity

the evaluation is the number of attributes For 200000entities in this paperrsquos dataset there are 1612848 attributedescriptions in total while the interactions between HINsare subject to mutual influence conditions where thenumber of influence restrictions is far smaller than that ofattribute description information erefore deleting at-tribute information will affect the performance of the modelmore than deleting interaction information

In Figure 4 we can see our modelrsquos PR curve is aboveTransErsquos PR curves in the multi-HIN setting for link errordetection in all neg situation DistMult and ComplexE cannotdistinguish noise links in the training set so they are not plottedin Figure 4 It reveals that our model achieves effectiveness indetecting noise in multi-HIN situation Figure 4 also explainsthat there is a difference between the distributions of true linksand noise linksrsquo scores both in multi-HITE and TransE modelFor lines of TransE in Figure 4 there exist sharp drops in the leftpart is can be explained that the scores for true links andnoisy links do not have a large gap especially near extremethreshold where all links are thought as positive When thethreshold drops few links are thought to be noise while positivelinks are mistaken as noise which will influence the curve

sharply Also it can be observed that the PR curve rises whenthe number of noise links increases as it is easier to detect errorswhen errors are more frequent

36 Experiment III Error Link Classification in DifferentCombinations of HINs In this part we try to combinedifferent HINs with different numbers of HINs to comparethe performance of our model for error link classificatione results are shown in Table 5 In the case of combiningdifferent data sources our model proves robust from theresults At the same time with the increment of data sourcesour model can make more use of the interaction informationbetween different data sources to provide more sufficientevidence for link classification

37 Experiment IV Influential Method EvaluationFinally we test how the performance of two ILEs varies withdifferent values of the threshold and test the model in er-rorless link classification tasks Based on the results in Ta-ble 6 we can conclude that alignment ILE is more sensitiveto the threshold value because its threshold is close to 0

Table 2 Statistics of HINs constructed based on DBpedia and YAGO

HIN of nodes of edges of attributes of node types of linksDBpedia2 100000 643 760062 225 428952YAGO 100000 64 827682 295 502563

10

08

06

04

02

00100806040200

Prec

ision

10 negative

Recall

Multi-HITETransE

(a)

10

08

06

04

02

00

Prec

ision

20 negative

100806040200Recall

Multi-HITETransE

(b)

10

09

08

07

06

05

04

03Pr

ecisi

on

40 negative

100806040200Recall

Multi-HITETransE

(c)

Figure 4 Trade-off between precision and recall according to trustworthiness threshold

Table 3 Performance of link prediction from all the models on multiple HINs

DataMulti-HITE Minfluence- Mattribute- TransE R-GCN DistMult ComplexEacc F1 acc F1 acc F1 acc F1 acc F1 acc F1 acc F1

DBpedia1 mdash mdash 092 092 087 088 091 091 088 088 090 089 091 090Wikidata mdash mdash 094 094 093 089 090 090 089 089 088 088 089 088Combined 095 095 093 093 090 090 092 092 090 090 091 091 091 091ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

Complexity 9

compared with neighbour ILE It should be explained thatneighbour ILE (see equation (9)) has fewer componentscompared with alignment ILE (see equation (8))

4 Conclusions

In this paper we focus on the multi-HIN trustworthinessevaluation problem and propose a co-embedding spaceevaluation method Our model incorporates inner-HIN andinter-HIN characteristics into the loss function A deepneural network is used in our model to solve the problemWe compare our model with other single-HIN trustwor-thiness evaluation methods on a multi-HIN dataset Ex-perimental results demonstrate that our model can wellaccomplish the trustworthiness evaluation task and out-performs baseline models For future works we will focus onhow to incorporate truth value discovery into our modelBesides the processing of our model is complicated com-pared with other models such as TransE DistMult andComplexE erefore we need to find a concise way toevaluate trustworthiness while maintaining the current levelof accuracy

Data Availability

e HIN alignment dataset DBP-WD used to support thefindings of this study can be accessed through httpsgithubcomnju-websoftBootEA

Conflicts of Interest

e authors declare that there are no conflicts of interestregarding the publication of this paper

Acknowledgments

is study was supported by the National Key Research andDevelopment Program of China (grant nos2018YFC0830201 and 2017YFB1002801) the FundamentalResearch Funds for the Central Universities (grant nos4009009106 and 22120200184) and the CCF-Baidu OpenFund

References

[1] F M Harper and J A Konstan ldquoe movielens datasetshistory and contextrdquo ACM Transactions on Interactive In-telligent Systems vol 5 no 4 2015

[2] H Paulheim ldquoKnowledge graph refinement a survey ofapproaches and evaluation methodsrdquo Semantic Web vol 8no 3 pp 489ndash508 2017

[3] Y Lin Z Liu H Luan et al ldquoModeling relation paths forrepresentation learning of knowledge basesrdquo in Proceedings ofthe 2015 Conference on Empirical Methods in Natural Lan-guage Processing pp 705ndash714 Lisbon Portugal September2015

[4] C Yang Y Xiao Y Zhang et al ldquoHeterogeneous networkrepresentation learning survey benchmark evaluation andbeyondrdquo 2020 httparxivorgabs200400216

Table 4 Performance of all the methods over data with different levels of noise

DataMulti-HITE Minfluence- Mattribute- TransE R-GCN DistMult ComplexEacc F1 acc F1 acc F1 acc F1 acc F1 acc F1 acc F1

10 neg 091 092 090 090 090 089 091 090 090 090 091 091 091 09120 neg 090 091 089 089 089 089 091 090 089 089 090 090 090 09040 neg 092 092 090 089 089 089 090 090 089 089 088 088 088 088ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

Table 6 Results (accuracyF1-score) of multi-TW using different values of thresholds for ILE functions

Alignment neighbour 00 01 02 0400 0899508981 0899409020 0896808997 084330849501 0872308735 0893708975 0905809027 088830890402 0863908673 0948009482 0923309253 091040907804 0843808453 0922509237 0940409395 0907509017

Table 5 Performance of multi-HITE over different datasets with different levels of noise

DatasetDBP1-WD DBP2-YAGO YAGO-WD DBP1-YAGO-

DBP2-WDacc F1 acc F1 acc F1 acc F1

10 neg 091 092 090 090 091 091 091 09220 neg 090 091 090 090 089 089 091 09240 neg 092 092 089 089 089 089 090 090ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

10 Complexity

methods including TransE [9] R-GCN [29] DistMult [25]and ComplexE [30] However we are not using originalversions of these algorithms Instead we use refined modelsfrom OpenKE [31] to compare our model with currentfrequently used model versions Because these methods donot generate trustworthiness scores directly we calculate thetrustworthiness score according to their threshold where alink score that is larger than the threshold is thought as 0otherwise as 1 Moreover because these methods are notproposed for multi-HIN cases they are not able to integrally

analyse multiple HINs us we use these methods toevaluate trustworthiness scores separately on different HINs

Additionally we simplify our model for comparisonsbetween our model with its simplified version We use twosimplified models to show the effectiveness of our modelsetting ese two versions of methods are denoted asMattribute- and Minfluence- respectively Mattribute- is a versionignoring node attribute inner-HIN characteristic where inthe inner-HIN evaluation stage only node-type constraintcontextual embedding information and graph energy

Typeembedding

Edgeembedding

Attributeembedding

Nodeembedding

Nodeenergyfeatures

Scorefunction

Relationalloss in source

Head node hiddenrepresentation

Tail node hiddenrepresentation

Edge hiddenrepresentation

Influentialtriplet loss

We1h

We1h

We2h

We3h

We3h

We4h

We5h

We2h

WerWnrWar

WerWnrWar

e1 (Ns)

e1 (No)

e2 (Ns)

e2 (No)

e3 (Ns)

e3 (No)

e4 (Ep)

e5 (Ep)

WlrWtr

r (e1(Ns)))r (e2(Ns)))r (e3(Ns)))

r (e1(No)))r (e2(No)))r (e3(No)))

r (e4(Ep)))r (e5(Ep)))

Zs

Zo

Zp

Figure 3 Detailed structure of neural network

REQUIRE S maximal epochs of training T source trustworthiness Vfor G isin S do

VGlarr1end forwhile Tgt 0 dofor G isin S dofor t isin G doCalculate link score according to (2)

end forend forfor G isin S doUpdate source trustworthiness VG

end forTlarrT minus 1Backpropagate loss and update parameters in the neural network

end while

ALGORITHM 1 Multi-HIN trustworthiness evaluation

Table 1 Statistics of HINs constructed based on DBpedia and Wikidata

HIN of nodes of edges of attributes of node types of linksDBpedia1 100000 686 622331 225 463294Wikidata 100000 946 990517 295 448774

Complexity 7

transferring information are taken into considerationMinfluence- is the simplified model neglecting influencemethods through ILE betweenHINs where in the inter-HINevaluation stage all ILEs are set to 0 so that there exist nointeractive links

33 Hyperparameter Setup In the neural network of ourmodel the size of the node embedding is 256 the edgeembedding size is 64 and the attribute embedding size is512 e hidden size and combined nodeedge size is 512We use the ReLU function as the activation function in themodel e sigmoid function is used in the output layer forproducing the trustworthiness score e value of learningrate is selected from 01 001 0001 and set as 0001 andthe batch size is selected from 10 40 100 and set as 0001ese hyperparameters are selected through cross-validationwith the best performance in multi-HIN situations

e value of margin c in our function is selected from05 08 1 Moreover we use l2-norm as the regularizer tocontrol all the trainable parameters in the neural networkInner-HIN loss and cross-HIN balance parameter β is selectedfrom 03 05 07 and set as 05 Graphic balancing parameterϵ is selected from 03 05 07 and set as 05 Influencemethodcontrol threshold parameter ρ is selected from 001 01 05

and set as 01 Value influence threshold θvalue is selected from05 08 09 and set as 08 Alignment influence thresholdθalignment is selected from 0 01 02 04 and set as 02Neighbouring influence threshold θneighbor is selected from0 01 02 04 and set as 01 ese hyperparameters areselected through cross-validation with the best performance inmulti-HIN situations

34 Experiment I Errorless Link Classification Link classi-fication is the task of testing the model trustworthinessevaluation ability in new data In our experiment this ex-periment is implemented on the testing set It is a simplifiedtask compared with error link classification and link errordetection [11] where there is no noise in the training set ecorrect link is labelled as 1 while a false link is labelled as 0e model output lies in the range of [0 1] representing theprobability of one link being a correct onee trustworthinessscore is binarized into 0 (false link label) or 1 (correct link label)based on a threshold value Namely given a link the trust-worthiness score is calculated according to equation (5) If thetrustworthiness score is larger than the threshold this link islabelled as the correct link otherwise the false link

Here we compare our model with other models in basicsituations namely in separate HIN space and in combinedHIN spaces Besides in this experiment we train and test onthe original dataset not adding false links into training Alsoassuming we do not know where the link comes from theHIN source trustworthiness VG is set to 1

Experimental results are shown in Table 3 Obviouslyour model outperforms baseline methods in separate HINand combined HIN space which indicates that the hiddeninformation representing the reliability of links has been welllearned in our model We can analyse this result from twodifferent perspectives (1) single HIN and multi-HIN (2)

influence of inner-HIN characteristics From the view of singleand multi-HIN we can notice that when HINs are combinedin other words when data are augmented evaluation resultsare improving compared with those in single HIN situationis can be also proved from the threshold dropping in TransEDistMult and partial ComplexE results is proves thatthough previous methods do not directly consider the con-nection between HINs they can model the inherent rela-tionship between multiple HINs in other implicit ways whiletheir results are no better than our explicit relationshipmodeling method From the view of influence from inner-HINcharacteristics attribute characteristics are rather important inthe link classification task as we can see the accuracy ofMattribute- drops by nearly 5 compared with normal multi-HITE model Our model can achieve convergence by runningtwo or three epochs on the training data In the DBP-WDdataset it takes 45 minutes for one epoch e network em-bedding model usually takes two hours to network embeddingfollowed by the training of the classification task which costsmore time than ours

35 Experiment II Error Link Classification and Link ErrorDetection in TwoHINs e evaluation metric is the same asthe error detection task e first task error link classifi-cation is similar to the errorless link classification edifference is in the training set where error link classificationis trained on the noise training set while testing set andvalidation set are the same A small change in our modeltakes place here as our model cannot converge optimal inthe noise training set which means our model detects noiseWe first train and evaluate our model on the training set(both positive and negative-sampled links) and then retrainour model on the generated training set where labels arepredicted from the first trained modelrsquos evaluation esecond task link error detection is conducted on thetraining set e results are shown in Figure 4 in the formatof PR curve to test different modelsrsquo performance on thebinary classification task of detecting error links

In Table 4 we can see our model achieves the best ac-curacy and F1-score when compared with other models inthe multi-HIN setting for error link classification in 10 and40 neg situation In detail our model thought 786856816749 and 895114 links may be correct in the first train andevaluation round separately (where 757126 757135 and757139 are truly correct) where 10 20 and 40 noisedatasets contain links of 834973 910880 and 1062693When the noise links rise our model is more likely to ruleout false examples It reveals that our model achieves ef-fectiveness in modeling the inter-influence of HINs by usinginteractive links For TransE DistMult and ComplexE theyare robust to errors in the training set where errors slightlyreduce modelsrsquo performance It reveals that the traditionalembedding model can handle the problem of error links inmulti-HIN trustworthiness situations Two simplifiedmodels indicate the importance of inner characteristics andwe can conclude that the attributes are an important in-dicator of whether a link is true or not We think that thereason why attribute factors play a more important role in

8 Complexity

the evaluation is the number of attributes For 200000entities in this paperrsquos dataset there are 1612848 attributedescriptions in total while the interactions between HINsare subject to mutual influence conditions where thenumber of influence restrictions is far smaller than that ofattribute description information erefore deleting at-tribute information will affect the performance of the modelmore than deleting interaction information

In Figure 4 we can see our modelrsquos PR curve is aboveTransErsquos PR curves in the multi-HIN setting for link errordetection in all neg situation DistMult and ComplexE cannotdistinguish noise links in the training set so they are not plottedin Figure 4 It reveals that our model achieves effectiveness indetecting noise in multi-HIN situation Figure 4 also explainsthat there is a difference between the distributions of true linksand noise linksrsquo scores both in multi-HITE and TransE modelFor lines of TransE in Figure 4 there exist sharp drops in the leftpart is can be explained that the scores for true links andnoisy links do not have a large gap especially near extremethreshold where all links are thought as positive When thethreshold drops few links are thought to be noise while positivelinks are mistaken as noise which will influence the curve

sharply Also it can be observed that the PR curve rises whenthe number of noise links increases as it is easier to detect errorswhen errors are more frequent

36 Experiment III Error Link Classification in DifferentCombinations of HINs In this part we try to combinedifferent HINs with different numbers of HINs to comparethe performance of our model for error link classificatione results are shown in Table 5 In the case of combiningdifferent data sources our model proves robust from theresults At the same time with the increment of data sourcesour model can make more use of the interaction informationbetween different data sources to provide more sufficientevidence for link classification

37 Experiment IV Influential Method EvaluationFinally we test how the performance of two ILEs varies withdifferent values of the threshold and test the model in er-rorless link classification tasks Based on the results in Ta-ble 6 we can conclude that alignment ILE is more sensitiveto the threshold value because its threshold is close to 0

Table 2 Statistics of HINs constructed based on DBpedia and YAGO

HIN of nodes of edges of attributes of node types of linksDBpedia2 100000 643 760062 225 428952YAGO 100000 64 827682 295 502563

10

08

06

04

02

00100806040200

Prec

ision

10 negative

Recall

Multi-HITETransE

(a)

10

08

06

04

02

00

Prec

ision

20 negative

100806040200Recall

Multi-HITETransE

(b)

10

09

08

07

06

05

04

03Pr

ecisi

on

40 negative

100806040200Recall

Multi-HITETransE

(c)

Figure 4 Trade-off between precision and recall according to trustworthiness threshold

Table 3 Performance of link prediction from all the models on multiple HINs

DataMulti-HITE Minfluence- Mattribute- TransE R-GCN DistMult ComplexEacc F1 acc F1 acc F1 acc F1 acc F1 acc F1 acc F1

DBpedia1 mdash mdash 092 092 087 088 091 091 088 088 090 089 091 090Wikidata mdash mdash 094 094 093 089 090 090 089 089 088 088 089 088Combined 095 095 093 093 090 090 092 092 090 090 091 091 091 091ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

Complexity 9

compared with neighbour ILE It should be explained thatneighbour ILE (see equation (9)) has fewer componentscompared with alignment ILE (see equation (8))

4 Conclusions

In this paper we focus on the multi-HIN trustworthinessevaluation problem and propose a co-embedding spaceevaluation method Our model incorporates inner-HIN andinter-HIN characteristics into the loss function A deepneural network is used in our model to solve the problemWe compare our model with other single-HIN trustwor-thiness evaluation methods on a multi-HIN dataset Ex-perimental results demonstrate that our model can wellaccomplish the trustworthiness evaluation task and out-performs baseline models For future works we will focus onhow to incorporate truth value discovery into our modelBesides the processing of our model is complicated com-pared with other models such as TransE DistMult andComplexE erefore we need to find a concise way toevaluate trustworthiness while maintaining the current levelof accuracy

Data Availability

e HIN alignment dataset DBP-WD used to support thefindings of this study can be accessed through httpsgithubcomnju-websoftBootEA

Conflicts of Interest

e authors declare that there are no conflicts of interestregarding the publication of this paper

Acknowledgments

is study was supported by the National Key Research andDevelopment Program of China (grant nos2018YFC0830201 and 2017YFB1002801) the FundamentalResearch Funds for the Central Universities (grant nos4009009106 and 22120200184) and the CCF-Baidu OpenFund

References

[1] F M Harper and J A Konstan ldquoe movielens datasetshistory and contextrdquo ACM Transactions on Interactive In-telligent Systems vol 5 no 4 2015

[2] H Paulheim ldquoKnowledge graph refinement a survey ofapproaches and evaluation methodsrdquo Semantic Web vol 8no 3 pp 489ndash508 2017

[3] Y Lin Z Liu H Luan et al ldquoModeling relation paths forrepresentation learning of knowledge basesrdquo in Proceedings ofthe 2015 Conference on Empirical Methods in Natural Lan-guage Processing pp 705ndash714 Lisbon Portugal September2015

[4] C Yang Y Xiao Y Zhang et al ldquoHeterogeneous networkrepresentation learning survey benchmark evaluation andbeyondrdquo 2020 httparxivorgabs200400216

Table 4 Performance of all the methods over data with different levels of noise

DataMulti-HITE Minfluence- Mattribute- TransE R-GCN DistMult ComplexEacc F1 acc F1 acc F1 acc F1 acc F1 acc F1 acc F1

10 neg 091 092 090 090 090 089 091 090 090 090 091 091 091 09120 neg 090 091 089 089 089 089 091 090 089 089 090 090 090 09040 neg 092 092 090 089 089 089 090 090 089 089 088 088 088 088ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

Table 6 Results (accuracyF1-score) of multi-TW using different values of thresholds for ILE functions

Alignment neighbour 00 01 02 0400 0899508981 0899409020 0896808997 084330849501 0872308735 0893708975 0905809027 088830890402 0863908673 0948009482 0923309253 091040907804 0843808453 0922509237 0940409395 0907509017

Table 5 Performance of multi-HITE over different datasets with different levels of noise

DatasetDBP1-WD DBP2-YAGO YAGO-WD DBP1-YAGO-

DBP2-WDacc F1 acc F1 acc F1 acc F1

10 neg 091 092 090 090 091 091 091 09220 neg 090 091 090 090 089 089 091 09240 neg 092 092 089 089 089 089 090 090ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

10 Complexity

transferring information are taken into considerationMinfluence- is the simplified model neglecting influencemethods through ILE betweenHINs where in the inter-HINevaluation stage all ILEs are set to 0 so that there exist nointeractive links

33 Hyperparameter Setup In the neural network of ourmodel the size of the node embedding is 256 the edgeembedding size is 64 and the attribute embedding size is512 e hidden size and combined nodeedge size is 512We use the ReLU function as the activation function in themodel e sigmoid function is used in the output layer forproducing the trustworthiness score e value of learningrate is selected from 01 001 0001 and set as 0001 andthe batch size is selected from 10 40 100 and set as 0001ese hyperparameters are selected through cross-validationwith the best performance in multi-HIN situations

e value of margin c in our function is selected from05 08 1 Moreover we use l2-norm as the regularizer tocontrol all the trainable parameters in the neural networkInner-HIN loss and cross-HIN balance parameter β is selectedfrom 03 05 07 and set as 05 Graphic balancing parameterϵ is selected from 03 05 07 and set as 05 Influencemethodcontrol threshold parameter ρ is selected from 001 01 05

and set as 01 Value influence threshold θvalue is selected from05 08 09 and set as 08 Alignment influence thresholdθalignment is selected from 0 01 02 04 and set as 02Neighbouring influence threshold θneighbor is selected from0 01 02 04 and set as 01 ese hyperparameters areselected through cross-validation with the best performance inmulti-HIN situations

34 Experiment I Errorless Link Classification Link classi-fication is the task of testing the model trustworthinessevaluation ability in new data In our experiment this ex-periment is implemented on the testing set It is a simplifiedtask compared with error link classification and link errordetection [11] where there is no noise in the training set ecorrect link is labelled as 1 while a false link is labelled as 0e model output lies in the range of [0 1] representing theprobability of one link being a correct onee trustworthinessscore is binarized into 0 (false link label) or 1 (correct link label)based on a threshold value Namely given a link the trust-worthiness score is calculated according to equation (5) If thetrustworthiness score is larger than the threshold this link islabelled as the correct link otherwise the false link

Here we compare our model with other models in basicsituations namely in separate HIN space and in combinedHIN spaces Besides in this experiment we train and test onthe original dataset not adding false links into training Alsoassuming we do not know where the link comes from theHIN source trustworthiness VG is set to 1

Experimental results are shown in Table 3 Obviouslyour model outperforms baseline methods in separate HINand combined HIN space which indicates that the hiddeninformation representing the reliability of links has been welllearned in our model We can analyse this result from twodifferent perspectives (1) single HIN and multi-HIN (2)

influence of inner-HIN characteristics From the view of singleand multi-HIN we can notice that when HINs are combinedin other words when data are augmented evaluation resultsare improving compared with those in single HIN situationis can be also proved from the threshold dropping in TransEDistMult and partial ComplexE results is proves thatthough previous methods do not directly consider the con-nection between HINs they can model the inherent rela-tionship between multiple HINs in other implicit ways whiletheir results are no better than our explicit relationshipmodeling method From the view of influence from inner-HINcharacteristics attribute characteristics are rather important inthe link classification task as we can see the accuracy ofMattribute- drops by nearly 5 compared with normal multi-HITE model Our model can achieve convergence by runningtwo or three epochs on the training data In the DBP-WDdataset it takes 45 minutes for one epoch e network em-bedding model usually takes two hours to network embeddingfollowed by the training of the classification task which costsmore time than ours

35 Experiment II Error Link Classification and Link ErrorDetection in TwoHINs e evaluation metric is the same asthe error detection task e first task error link classifi-cation is similar to the errorless link classification edifference is in the training set where error link classificationis trained on the noise training set while testing set andvalidation set are the same A small change in our modeltakes place here as our model cannot converge optimal inthe noise training set which means our model detects noiseWe first train and evaluate our model on the training set(both positive and negative-sampled links) and then retrainour model on the generated training set where labels arepredicted from the first trained modelrsquos evaluation esecond task link error detection is conducted on thetraining set e results are shown in Figure 4 in the formatof PR curve to test different modelsrsquo performance on thebinary classification task of detecting error links

In Table 4 we can see our model achieves the best ac-curacy and F1-score when compared with other models inthe multi-HIN setting for error link classification in 10 and40 neg situation In detail our model thought 786856816749 and 895114 links may be correct in the first train andevaluation round separately (where 757126 757135 and757139 are truly correct) where 10 20 and 40 noisedatasets contain links of 834973 910880 and 1062693When the noise links rise our model is more likely to ruleout false examples It reveals that our model achieves ef-fectiveness in modeling the inter-influence of HINs by usinginteractive links For TransE DistMult and ComplexE theyare robust to errors in the training set where errors slightlyreduce modelsrsquo performance It reveals that the traditionalembedding model can handle the problem of error links inmulti-HIN trustworthiness situations Two simplifiedmodels indicate the importance of inner characteristics andwe can conclude that the attributes are an important in-dicator of whether a link is true or not We think that thereason why attribute factors play a more important role in

8 Complexity

the evaluation is the number of attributes For 200000entities in this paperrsquos dataset there are 1612848 attributedescriptions in total while the interactions between HINsare subject to mutual influence conditions where thenumber of influence restrictions is far smaller than that ofattribute description information erefore deleting at-tribute information will affect the performance of the modelmore than deleting interaction information

In Figure 4 we can see our modelrsquos PR curve is aboveTransErsquos PR curves in the multi-HIN setting for link errordetection in all neg situation DistMult and ComplexE cannotdistinguish noise links in the training set so they are not plottedin Figure 4 It reveals that our model achieves effectiveness indetecting noise in multi-HIN situation Figure 4 also explainsthat there is a difference between the distributions of true linksand noise linksrsquo scores both in multi-HITE and TransE modelFor lines of TransE in Figure 4 there exist sharp drops in the leftpart is can be explained that the scores for true links andnoisy links do not have a large gap especially near extremethreshold where all links are thought as positive When thethreshold drops few links are thought to be noise while positivelinks are mistaken as noise which will influence the curve

sharply Also it can be observed that the PR curve rises whenthe number of noise links increases as it is easier to detect errorswhen errors are more frequent

36 Experiment III Error Link Classification in DifferentCombinations of HINs In this part we try to combinedifferent HINs with different numbers of HINs to comparethe performance of our model for error link classificatione results are shown in Table 5 In the case of combiningdifferent data sources our model proves robust from theresults At the same time with the increment of data sourcesour model can make more use of the interaction informationbetween different data sources to provide more sufficientevidence for link classification

37 Experiment IV Influential Method EvaluationFinally we test how the performance of two ILEs varies withdifferent values of the threshold and test the model in er-rorless link classification tasks Based on the results in Ta-ble 6 we can conclude that alignment ILE is more sensitiveto the threshold value because its threshold is close to 0

Table 2 Statistics of HINs constructed based on DBpedia and YAGO

HIN of nodes of edges of attributes of node types of linksDBpedia2 100000 643 760062 225 428952YAGO 100000 64 827682 295 502563

10

08

06

04

02

00100806040200

Prec

ision

10 negative

Recall

Multi-HITETransE

(a)

10

08

06

04

02

00

Prec

ision

20 negative

100806040200Recall

Multi-HITETransE

(b)

10

09

08

07

06

05

04

03Pr

ecisi

on

40 negative

100806040200Recall

Multi-HITETransE

(c)

Figure 4 Trade-off between precision and recall according to trustworthiness threshold

Table 3 Performance of link prediction from all the models on multiple HINs

DataMulti-HITE Minfluence- Mattribute- TransE R-GCN DistMult ComplexEacc F1 acc F1 acc F1 acc F1 acc F1 acc F1 acc F1

DBpedia1 mdash mdash 092 092 087 088 091 091 088 088 090 089 091 090Wikidata mdash mdash 094 094 093 089 090 090 089 089 088 088 089 088Combined 095 095 093 093 090 090 092 092 090 090 091 091 091 091ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

Complexity 9

compared with neighbour ILE It should be explained thatneighbour ILE (see equation (9)) has fewer componentscompared with alignment ILE (see equation (8))

4 Conclusions

In this paper we focus on the multi-HIN trustworthinessevaluation problem and propose a co-embedding spaceevaluation method Our model incorporates inner-HIN andinter-HIN characteristics into the loss function A deepneural network is used in our model to solve the problemWe compare our model with other single-HIN trustwor-thiness evaluation methods on a multi-HIN dataset Ex-perimental results demonstrate that our model can wellaccomplish the trustworthiness evaluation task and out-performs baseline models For future works we will focus onhow to incorporate truth value discovery into our modelBesides the processing of our model is complicated com-pared with other models such as TransE DistMult andComplexE erefore we need to find a concise way toevaluate trustworthiness while maintaining the current levelof accuracy

Data Availability

e HIN alignment dataset DBP-WD used to support thefindings of this study can be accessed through httpsgithubcomnju-websoftBootEA

Conflicts of Interest

e authors declare that there are no conflicts of interestregarding the publication of this paper

Acknowledgments

is study was supported by the National Key Research andDevelopment Program of China (grant nos2018YFC0830201 and 2017YFB1002801) the FundamentalResearch Funds for the Central Universities (grant nos4009009106 and 22120200184) and the CCF-Baidu OpenFund

References

[1] F M Harper and J A Konstan ldquoe movielens datasetshistory and contextrdquo ACM Transactions on Interactive In-telligent Systems vol 5 no 4 2015

[2] H Paulheim ldquoKnowledge graph refinement a survey ofapproaches and evaluation methodsrdquo Semantic Web vol 8no 3 pp 489ndash508 2017

[3] Y Lin Z Liu H Luan et al ldquoModeling relation paths forrepresentation learning of knowledge basesrdquo in Proceedings ofthe 2015 Conference on Empirical Methods in Natural Lan-guage Processing pp 705ndash714 Lisbon Portugal September2015

[4] C Yang Y Xiao Y Zhang et al ldquoHeterogeneous networkrepresentation learning survey benchmark evaluation andbeyondrdquo 2020 httparxivorgabs200400216

Table 4 Performance of all the methods over data with different levels of noise

DataMulti-HITE Minfluence- Mattribute- TransE R-GCN DistMult ComplexEacc F1 acc F1 acc F1 acc F1 acc F1 acc F1 acc F1

10 neg 091 092 090 090 090 089 091 090 090 090 091 091 091 09120 neg 090 091 089 089 089 089 091 090 089 089 090 090 090 09040 neg 092 092 090 089 089 089 090 090 089 089 088 088 088 088ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

Table 6 Results (accuracyF1-score) of multi-TW using different values of thresholds for ILE functions

Alignment neighbour 00 01 02 0400 0899508981 0899409020 0896808997 084330849501 0872308735 0893708975 0905809027 088830890402 0863908673 0948009482 0923309253 091040907804 0843808453 0922509237 0940409395 0907509017

Table 5 Performance of multi-HITE over different datasets with different levels of noise

DatasetDBP1-WD DBP2-YAGO YAGO-WD DBP1-YAGO-

DBP2-WDacc F1 acc F1 acc F1 acc F1

10 neg 091 092 090 090 091 091 091 09220 neg 090 091 090 090 089 089 091 09240 neg 092 092 089 089 089 089 090 090ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

10 Complexity

the evaluation is the number of attributes For 200000entities in this paperrsquos dataset there are 1612848 attributedescriptions in total while the interactions between HINsare subject to mutual influence conditions where thenumber of influence restrictions is far smaller than that ofattribute description information erefore deleting at-tribute information will affect the performance of the modelmore than deleting interaction information

In Figure 4 we can see our modelrsquos PR curve is aboveTransErsquos PR curves in the multi-HIN setting for link errordetection in all neg situation DistMult and ComplexE cannotdistinguish noise links in the training set so they are not plottedin Figure 4 It reveals that our model achieves effectiveness indetecting noise in multi-HIN situation Figure 4 also explainsthat there is a difference between the distributions of true linksand noise linksrsquo scores both in multi-HITE and TransE modelFor lines of TransE in Figure 4 there exist sharp drops in the leftpart is can be explained that the scores for true links andnoisy links do not have a large gap especially near extremethreshold where all links are thought as positive When thethreshold drops few links are thought to be noise while positivelinks are mistaken as noise which will influence the curve

sharply Also it can be observed that the PR curve rises whenthe number of noise links increases as it is easier to detect errorswhen errors are more frequent

36 Experiment III Error Link Classification in DifferentCombinations of HINs In this part we try to combinedifferent HINs with different numbers of HINs to comparethe performance of our model for error link classificatione results are shown in Table 5 In the case of combiningdifferent data sources our model proves robust from theresults At the same time with the increment of data sourcesour model can make more use of the interaction informationbetween different data sources to provide more sufficientevidence for link classification

37 Experiment IV Influential Method EvaluationFinally we test how the performance of two ILEs varies withdifferent values of the threshold and test the model in er-rorless link classification tasks Based on the results in Ta-ble 6 we can conclude that alignment ILE is more sensitiveto the threshold value because its threshold is close to 0

Table 2 Statistics of HINs constructed based on DBpedia and YAGO

HIN of nodes of edges of attributes of node types of linksDBpedia2 100000 643 760062 225 428952YAGO 100000 64 827682 295 502563

10

08

06

04

02

00100806040200

Prec

ision

10 negative

Recall

Multi-HITETransE

(a)

10

08

06

04

02

00

Prec

ision

20 negative

100806040200Recall

Multi-HITETransE

(b)

10

09

08

07

06

05

04

03Pr

ecisi

on

40 negative

100806040200Recall

Multi-HITETransE

(c)

Figure 4 Trade-off between precision and recall according to trustworthiness threshold

Table 3 Performance of link prediction from all the models on multiple HINs

DataMulti-HITE Minfluence- Mattribute- TransE R-GCN DistMult ComplexEacc F1 acc F1 acc F1 acc F1 acc F1 acc F1 acc F1

DBpedia1 mdash mdash 092 092 087 088 091 091 088 088 090 089 091 090Wikidata mdash mdash 094 094 093 089 090 090 089 089 088 088 089 088Combined 095 095 093 093 090 090 092 092 090 090 091 091 091 091ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

Complexity 9

compared with neighbour ILE It should be explained thatneighbour ILE (see equation (9)) has fewer componentscompared with alignment ILE (see equation (8))

4 Conclusions

In this paper we focus on the multi-HIN trustworthinessevaluation problem and propose a co-embedding spaceevaluation method Our model incorporates inner-HIN andinter-HIN characteristics into the loss function A deepneural network is used in our model to solve the problemWe compare our model with other single-HIN trustwor-thiness evaluation methods on a multi-HIN dataset Ex-perimental results demonstrate that our model can wellaccomplish the trustworthiness evaluation task and out-performs baseline models For future works we will focus onhow to incorporate truth value discovery into our modelBesides the processing of our model is complicated com-pared with other models such as TransE DistMult andComplexE erefore we need to find a concise way toevaluate trustworthiness while maintaining the current levelof accuracy

Data Availability

e HIN alignment dataset DBP-WD used to support thefindings of this study can be accessed through httpsgithubcomnju-websoftBootEA

Conflicts of Interest

e authors declare that there are no conflicts of interestregarding the publication of this paper

Acknowledgments

is study was supported by the National Key Research andDevelopment Program of China (grant nos2018YFC0830201 and 2017YFB1002801) the FundamentalResearch Funds for the Central Universities (grant nos4009009106 and 22120200184) and the CCF-Baidu OpenFund

References

[1] F M Harper and J A Konstan ldquoe movielens datasetshistory and contextrdquo ACM Transactions on Interactive In-telligent Systems vol 5 no 4 2015

[2] H Paulheim ldquoKnowledge graph refinement a survey ofapproaches and evaluation methodsrdquo Semantic Web vol 8no 3 pp 489ndash508 2017

[3] Y Lin Z Liu H Luan et al ldquoModeling relation paths forrepresentation learning of knowledge basesrdquo in Proceedings ofthe 2015 Conference on Empirical Methods in Natural Lan-guage Processing pp 705ndash714 Lisbon Portugal September2015

[4] C Yang Y Xiao Y Zhang et al ldquoHeterogeneous networkrepresentation learning survey benchmark evaluation andbeyondrdquo 2020 httparxivorgabs200400216

Table 4 Performance of all the methods over data with different levels of noise

DataMulti-HITE Minfluence- Mattribute- TransE R-GCN DistMult ComplexEacc F1 acc F1 acc F1 acc F1 acc F1 acc F1 acc F1

10 neg 091 092 090 090 090 089 091 090 090 090 091 091 091 09120 neg 090 091 089 089 089 089 091 090 089 089 090 090 090 09040 neg 092 092 090 089 089 089 090 090 089 089 088 088 088 088ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

Table 6 Results (accuracyF1-score) of multi-TW using different values of thresholds for ILE functions

Alignment neighbour 00 01 02 0400 0899508981 0899409020 0896808997 084330849501 0872308735 0893708975 0905809027 088830890402 0863908673 0948009482 0923309253 091040907804 0843808453 0922509237 0940409395 0907509017

Table 5 Performance of multi-HITE over different datasets with different levels of noise

DatasetDBP1-WD DBP2-YAGO YAGO-WD DBP1-YAGO-

DBP2-WDacc F1 acc F1 acc F1 acc F1

10 neg 091 092 090 090 091 091 091 09220 neg 090 091 090 090 089 089 091 09240 neg 092 092 089 089 089 089 090 090ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

10 Complexity

compared with neighbour ILE It should be explained thatneighbour ILE (see equation (9)) has fewer componentscompared with alignment ILE (see equation (8))

4 Conclusions

In this paper we focus on the multi-HIN trustworthinessevaluation problem and propose a co-embedding spaceevaluation method Our model incorporates inner-HIN andinter-HIN characteristics into the loss function A deepneural network is used in our model to solve the problemWe compare our model with other single-HIN trustwor-thiness evaluation methods on a multi-HIN dataset Ex-perimental results demonstrate that our model can wellaccomplish the trustworthiness evaluation task and out-performs baseline models For future works we will focus onhow to incorporate truth value discovery into our modelBesides the processing of our model is complicated com-pared with other models such as TransE DistMult andComplexE erefore we need to find a concise way toevaluate trustworthiness while maintaining the current levelof accuracy

Data Availability

e HIN alignment dataset DBP-WD used to support thefindings of this study can be accessed through httpsgithubcomnju-websoftBootEA

Conflicts of Interest

e authors declare that there are no conflicts of interestregarding the publication of this paper

Acknowledgments

is study was supported by the National Key Research andDevelopment Program of China (grant nos2018YFC0830201 and 2017YFB1002801) the FundamentalResearch Funds for the Central Universities (grant nos4009009106 and 22120200184) and the CCF-Baidu OpenFund

References

[1] F M Harper and J A Konstan ldquoe movielens datasetshistory and contextrdquo ACM Transactions on Interactive In-telligent Systems vol 5 no 4 2015

[2] H Paulheim ldquoKnowledge graph refinement a survey ofapproaches and evaluation methodsrdquo Semantic Web vol 8no 3 pp 489ndash508 2017

[3] Y Lin Z Liu H Luan et al ldquoModeling relation paths forrepresentation learning of knowledge basesrdquo in Proceedings ofthe 2015 Conference on Empirical Methods in Natural Lan-guage Processing pp 705ndash714 Lisbon Portugal September2015

[4] C Yang Y Xiao Y Zhang et al ldquoHeterogeneous networkrepresentation learning survey benchmark evaluation andbeyondrdquo 2020 httparxivorgabs200400216

Table 4 Performance of all the methods over data with different levels of noise

DataMulti-HITE Minfluence- Mattribute- TransE R-GCN DistMult ComplexEacc F1 acc F1 acc F1 acc F1 acc F1 acc F1 acc F1

10 neg 091 092 090 090 090 089 091 090 090 090 091 091 091 09120 neg 090 091 089 089 089 089 091 090 089 089 090 090 090 09040 neg 092 092 090 089 089 089 090 090 089 089 088 088 088 088ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

Table 6 Results (accuracyF1-score) of multi-TW using different values of thresholds for ILE functions

Alignment neighbour 00 01 02 0400 0899508981 0899409020 0896808997 084330849501 0872308735 0893708975 0905809027 088830890402 0863908673 0948009482 0923309253 091040907804 0843808453 0922509237 0940409395 0907509017

Table 5 Performance of multi-HITE over different datasets with different levels of noise

DatasetDBP1-WD DBP2-YAGO YAGO-WD DBP1-YAGO-

DBP2-WDacc F1 acc F1 acc F1 acc F1

10 neg 091 092 090 090 091 091 091 09220 neg 090 091 090 090 089 089 091 09240 neg 092 092 089 089 089 089 090 090ldquoaccrdquo and ldquoF1rdquo denote the accuracy and F1-score of estimations respectively

10 Complexity

[5] H Paulheim and C Bizer ldquoImproving the quality of linkeddata using statistical distributionsrdquo International Journal onSemantic Web and Information Systems vol 10 no 2pp 63ndash86 2014

[6] V Chandola A Banerjee and V Kumar ldquoAnomaly detec-tionrdquo ACM Computing Surveys vol 41 no 3 pp 1ndash58 2009

[7] Y Dong N V Chawla and A Swami ldquoMetapath2vecscalable representation learning for heterogeneous networksrdquoin Proceedings of the 23rd ACM SIGKDD InternationalConference on Knowledge Discovery and Data Miningpp 135ndash144 Halifax Canada August 2017

[8] Z Wang J Zhang J Feng et al ldquoKnowledge graph em-bedding by translating on hyperplanesrdquo AAAI vol 14pp 1112ndash1119 2014

[9] A Bordes N Usunier A Garcia-Duran et al ldquoTranslatingembeddings for modeling multi-relational datardquo in Pro-ceedings of the 26th International Conference on Neural In-formation Processing Systems-Volume 2 pp 2787ndash2795 NewYork NY USA December 2013

[10] S Jia Y Xiang X Chen et al ldquoTriple trustworthinessmeasurement for knowledge graphrdquo in Proceedings of theWorld Wide Web Conference pp 2865ndash2871 San FranciscoCA USA May 2019

[11] R Xie Z Liu F Lin et al ldquoDoes william shakespeare reallywrite hamlet Knowledge representation learning with con-fidencerdquo in Proceedings of the 32nd AAAI Conference onArtificial Intelligence pp 4954ndash4961 New Orleans LA USAFebruary 2018

[12] Y Ma H Gao T Wu et al ldquoLearning disjointness axiomswith association rule mining and its application to incon-sistency detection of linked datardquo in Proceedings of ChineseSemantic Web and Web Science Conference pp 29ndash41Wuhan China August 2014

[13] H Lin Y Liu W Wang Y Yue and Z Lin ldquoLearning entityand relation embeddings for knowledge resolutionrdquo ProcediaComputer Science vol 108 pp 345ndash354 2017

[14] S Zhang H Tong R Maciejewski et al ldquoMultilevel networkalignmentrdquo in Proceedings of the World Wide Web Confer-ence pp 2344ndash2354 San Francisco CA USA May 2019

[15] S Zhang H Tong J Tang et al ldquoIneat incomplete networkalignmentrdquo in Proceedings of the 2017 IEEE InternationalConference on Data Mining (ICDM) pp 1189ndash1194 NewOrleans LA USA November 2017

[16] K Chao ldquoCalign Aligning sequences with restricted affinegap penaltiesrdquo Bioinformatics vol 15 no 4 pp 298ndash3041999

[17] S Auer C Bizer G Kobilarov et al ldquoDbpedia a nucleus for aweb of open datardquo in Proceedings of the 6th InternationalSemantic Web Conference pp 722ndash735 Busan Korea No-vember 2007

[18] J Hoffart F M Suchanek K Berberich and G WeikumldquoYago2 a spatially and temporally enhanced knowledge basefrom wikipediardquo Artificial Intelligence vol 194 pp 28ndash612013

[19] L Y Wu A Fisch S Chopra et al ldquoStarspace embed all thethingsrdquo in Proceedings of the 32nd AAAI Conference on Ar-tificial Intelligence pp 5569ndash5577 New Orleans LA USAFebruary 2018

[20] M Chen Y Tian M Yang et al ldquoMultilingual knowledgegraph embeddings for cross-lingual knowledge alignmentrdquo inProceedings of the 26th International Joint Conference onArtificial Intelligence pp 1511ndash1517 Melbourne AustraliaAugust 2017

[21] R Trivedi B Sisman X L Dong et al ldquoLinknbed multi-graph representation learning with entity linkagerdquo in Pro-ceedings of the 56th Annual Meeting of the Association forComputational Linguistics (Volume 1 Long Papers)pp 252ndash262 Melbourne Australia July 2018

[22] E Huaman E Karle and D Fensel ldquoKnowledge graphvalidationrdquo 2020 httparxivorgabs200501389

[23] Q Le and T Mikolov ldquoDistributed representations of sen-tences and documentsrdquo in Proceedings of the 31st Interna-tional Conference on International Conference on MachineLearning-Volume 32 pp 1188ndash1196 Beijing China June2014

[24] L Page S Brin R Motwani et al ldquoe pagerank citationranking bringing order to the webrdquo Technical reportStanford InfoLab Stanford CA USA 1998

[25] B Yang W-t Yih X He et al ldquoEmbedding entities andrelations for learning and inference in knowledge basesrdquo inProceedings of the International Conference on LearningRepresentations (ICLR) San Diego CA USA May 2015

[26] W Liu J Liu M Wu et al ldquoRepresentation learning overmultiple knowledge graphs for knowledge graphs alignmentrdquoNeurocomputing vol 320 pp 12ndash24 2018

[27] M A Rodrıguez and M J Egenhofer ldquoDetermining semanticsimilarity among entity classes from different ontologiesrdquoIEEE Transactions on Knowledge and Data Engineeringvol 15 no 2 pp 442ndash456 2003

[28] Z Sun W Hu and C Li ldquoCross-lingual entity alignment viajoint attribute-preserving embeddingrdquo in Proceedings of the16th International Semantic Web Conference pp 628ndash644Vienna Austria October 2017

[29] M Schlichtkrull T N Kipf P Bloem et al ldquoModeling re-lational data with graph convolutional networksrdquo in Pro-ceedings of the European Semantic Web Conferencepp 593ndash607 Portoroz Slovenia June 2018

[30] T Trouillon J Welbl S Riedel et al ldquoComplex embeddingsfor simple link predictionrdquo in Proceedings of the 33rd In-ternational Conference on International Conference on Ma-chine Learning-Volume 48 pp 2071ndash2080 New York NYUSA June 2016

[31] X Han S Cao X Lv et al ldquoOpenke an open toolkit forknowledge embeddingrdquo in Proceedings of the 2018 Conferenceon Empirical Methods in Natural Language Processing SystemDemonstrations pp 139ndash144 Brussels Belgium November2018

Complexity 11