Using ontologies to model human navigation behavior in information

Semantic Web 0 (2014) 1 1IOS Press

Using ontologies to model human navigationbehavior in information networks:A study based on WikipediaEditor(s): Werner Kuhn, University of Münster, GermanySolicited review(s): Carsten Keßler, Hunter College, USA; Tobias Weigel, Deutsches Klimarechenzentrum, Germany; Hartwig Hochmair,University of Florida, USA

Daniel Lamprecht a, Markus Strohmaier a, b ∗, Denis Helic a, Csongor Nyulas b, Tania Tudorache b,Natalya F. Noy b and Mark A. Musen b

a Knowledge Technologies Institute, Graz University of Technology, Inffeldgasse 13, 8010 Graz, Austriab Stanford Center for Biomedical Informatics Research, Stanford University, 1265 Welch Road, Stanford, CA94305, USA

Abstract. The need to examine the behavior of different user groups is a fundamental requirement when building informationsystems. In this paper, we present Ontology-based Decentralized Search (OBDS), a novel method to model the navigationbehavior of users equipped with different types of background knowledge. Ontology-based Decentralized Search combinesontologies and decentralized search, an established method for navigation in social networks, to model navigation behaviorin information networks. The method uses ontologies as an explicit representation of background knowledge to inform thenavigation process and guide it towards navigation targets. By using different ontologies, users equipped with different types ofbackground knowledge can be represented. We demonstrate our method using four biomedical ontologies and their associatedWikipedia articles. We compare our simulation results with base line approaches and with results obtained from a user study andfind that our method produces click paths that have properties similar to those originating from human navigators. The resultssuggest that our method can be used to model human navigation behavior in systems that are based on information networkssuch as Wikipedia.

Keywords: Navigation, Decentralized Search, Ontology

1. Introduction

One of the challenges in building information sys-tems is the need to develop interfaces suited to a rangeof different types of users. Different types of users,such as novices, experts, generalists or specialists will,in general, display considerably different knowledgeabout a given domain. This specific knowledge in turninfluences their interactions with an information sys-tem. Gaining insight into human navigation behaviorsupports the construction of easy-to-use software and

*Corresponding author – [email protected]

information systems that are ready to accommodate abroad range of user types.

In this paper, we investigate ways of modeling nav-igational behavior of human users in information net-works. Humans navigating an information network(such as Wikipedia) generally do not know the net-work topology in its entirety. They are therefore notalways familiar with the global network structure butnavigate based on assumptions and local informationonly. Experiments by Stanley Milgram and others [31][21] have shown that humans are very effective at find-ing short paths based on local information in offline aswell as in online social networks.

1570-0844/14/$27.50 c© 2014 – IOS Press and the authors. All rights reserved

2 Using ontologies to model human navigation behavior in information networks: A study based on Wikipedia

In this paper we present a novel method for simu-lating human navigational click behavior in informa-tion networks, using ontologies as background knowl-edge and examine its suitability to model actual hu-man navigation behavior. The method, which we callOntology-based Decentralized Search (OBDS), buildson decentralized search [16], a well-established nav-igation method in social networks which is based onlocal information only. Decentralized search has beensuccessfully applied to navigation in information net-works in previous research, where it has been usedto model the behavior of users and to produce simu-lated click data [12]. OBDS uses decentralized searchwith ontologies as background knowledge to model thesearch process and to point an algorithmic searcher to-wards the direction of the target.

This method is new in that it uses an explicit repre-sentation of the background knowledge in the form ofan ontology. Research in psychology suggests that hu-mans store concepts in their minds hierarchically [7].

Research questions: In this work, we will address thefollowing three research questions:

RQ1 Can ontologies contribute useful informationto modeling navigation in information networks? Andhow does OBDS using real-world ontologies performin comparison to OBDS using randomly generated on-tologies and random walks?

RQ2 Does Ontology-based Decentralized Search(OBDS) produce valid results, i.e., are the simulatednavigation paths similar to those produced by humannavigation?

RQ3 When using OBDS, what ontology is bestsuited to produce human-like navigation results?

To demonstrate our method, we use the informationnetwork formed by a set of biomedical Wikipedia ar-ticles and the connections (hyperlinks) between them.We show that several different biomedical ontologiescan be used as background knowledge to inform navi-gation simulations, much as humans use their acquiredknowledge for navigation.

Contributions: Our main contribution is the demon-stration of the general suitability of existing real-worldontologies to inform models of human navigation,specifically decentralized search on information net-works such as Wikipedia. To the best of our knowl-edge, our work presents a novel method and a novel

application of ontologies. By comparing the naviga-tional paths generated by our simulations with severalbaseline approaches and with data obtained from a userstudy, we show that our method yields results similarto those produced by actual human users. The resultssuggest that OBDS can be used to simulate humannavigational behavior in information networks, whichcan be useful for addressing issues arising in the de-velopment of systems that are based on networked in-formation. These findings are relevant for researchersinterested in new applications for ontologies and forresearchers interested in modeling navigation in in-formation networks using ontologies as backgroundknowledge.

The rest of this paper is structured as follows: InSection 2 we place our work in the context of previ-ous research and related work. In Section 3 we discussmaterials and methods, and we present the results inSection 4. We end with a discussion.

2. Related work

In the context of this paper, the related areas can bedivided in three: navigation in social networks, naviga-tion in information networks and ontologies.

2.1. Navigation in social networks

This paper particularly addresses navigation in so-cial networks via decentralized search algorithms.Fundamentally, decentralized search describes a wayof solving a pathfinding problem in a social network.Starting from an arbitrary start node (i.e., a person)within the network, the objective of decentralizedsearch is to find a way to a given target node. Thealgorithm, however, does not possess global knowl-edge of the network and can therefore only take deci-sions based on local knowledge. The term decentral-ized stems from the fact that the search proceeds byforwarding the search problem from one node to an-other, which, in a social network, involves a differentperson taking the decisions at every node.

The idea of decentralized search, as used in our nav-igation simulations, was made popular by Stanley Mil-gram’s widely discussed small-world experiment [31][21] in the 1960s. In the experiment, participants inBoston and Nebraska received a letter containing in-formation about a target person (a Boston stock bro-ker). They were then asked to forward the letter to oneof their acquaintances, to bring the letter closer to the

Using ontologies to model human navigation behavior in information networks: A study based on Wikipedia 3

target person. The resulting median chain length of sixintermediates for successful chains of letters coinedthe term "six degrees of separation". By taking onlythe limited knowledge of each participant into accountat each step, the search effectively constituted a formof decentralized search. The result illustrated the so-called small world phenomenon, as it seemed possi-ble to connect two arbitrary persons across the UnitedStates through a very small number of hops.

In 1998, Watts and Strogatz [33] characterized net-works exhibiting small-world characteristics as ex-hibiting a high clustering coefficient and a low charac-teristic path length and demonstrated the actual exis-tence of this type of small-world networks in a film ac-tor collaboration network, the power grid of the west-ern United States and the neural network of C. elegans,a small roundworm.

In 2000, Jon Kleinberg proved that for the type ofsmall-world networks proposed by Watts and Strogatz[14], no effective decentralized search algorithm couldexist that always found a path connecting two nodesin subpolynomial time. However, Kleinberg presenteda more generalized version of the model for which hethen proved that a decentralized algorithm capable offinding short paths existed.

Kleinberg later extended his model of decentralizedsearch to include hierarchies [15], where the term hi-erarchy denotes a tree that includes all network nodes(and may contain more nodes). He showed that whenthe network nodes were embedded as the leaf nodes ofa hierarchy and links in a network were formed pro-portional to distances in this hierarchy, the resultingnetwork was also efficiently searchable. To form an ef-fectively searchable graph, nodes were connected witha probability proportional to their distance in the tree,i.e., the height of their closest common ancestor. A de-centralized search algorithm with knowledge of the hi-erarchy (i.e., the background knowledge) could thenefficiently find targets. In this paper, we use ontologiesas this type of background knowledge.

Miao et al. [20] have studied decentralized searchin collaboration networks. Collaboration networks dif-fer from information or social networks in that the in-formation flow in them is driven by tasks. This meansthat the edges in the network are formed by collabo-ration on tasks. In their study, the tasks were softwarebugs. Developers who were assigned a bug they couldnot eliminate themselves forwarded it to another de-veloper who they believed could handle it. By estab-lishing several forwards in a row, this workflow can beviewed as a type of decentralized search, as all deci-

sions about the next hop were taken independently bymultiple participants.

Adamic and Adar [4] studied decentralized searchin the e-mail network of the HP labs and found thatdecentralized search according to hierarchies based onconnectedness and office cubicle distance worked best.

Decentralized search is also used in peer-to-peer filesharing protocols such as Gnutella or KaZaA. With alow characteristic path length and a high cluster co-efficient, the Gnutella network displayed small-worldcharacteristics in 2003 [18].

2.2. Navigation in information networks

In this paper, decentralized search, a navigationmodel originally developed for social networks, is ap-plied to information networks.

One of the most prominent models related to searchin information networks is information foraging [26].Information foraging is based on foraging theory inbiology. In order to survive, animals have adoptedmethods which maximize the energy gained from foodsources. In the theory of information foraging, searchin information networks is not guided by backgroundknowledge but by information scent, with each articleand link emanating a distinct scent, which is dependenton the target of the search. For instance, when search-ing for information on penguins, a link leading to anarticle about Antarctica would provide more scent thana link leading to an article about the Sahara desert.

In this paper, information networks are studied onthe example of Wikipedia. However, genuine naviga-tion paths from Wikipedia are difficult to obtain, asthe goals of users are often hidden and not explic-itly visible, and logs of click trails are hard to obtain.With 60 − 70%, the fraction of teleports is further-more significantly higher on Wikipedia than on gen-eral web sites [8]. This might be due to the fact thatusers visit Wikipedia to satisfy specific information de-mands rather than to browse articles. However, thereexist valid reasons to navigate Wikipedia, which willbe detailed in the description of the navigation scenar-ios in Section 3.5.

Due to the difficulty of obtaining Wikipedia nav-igation paths, wiki games have been a popular re-placement for Wikipedia navigation paths in recent re-search. Wiki games, such as Wikispeedia1, Wiki Aata2

1www.wikispeedia.net2www.wikiaata.com


or Wiki Game3 allow users to play games on the infor-mation network formed by the Wikipedia articles andlinks between them. Click trails from wiki games haveenabled researchers to gain insight into navigationalbehavior on Wikipedia. In 2009, West et al. [35] usedwiki game data to infer semantic distances betweenconcepts by studying game click paths. In 2012, Westand Leskovec [34] found that in wiki games, playerstend to navigate to hubs (articles with a large numberof outlinks) first, and subsequently home in on targetnodes.

In our own group, we have used decentralizedsearch (with non-ontological background knowledge)in different contexts:

In 2011, Helic and Strohmaier compared the naviga-bility of different tag hierarchy generation algorithmson data from Bibsonomy, CiteULike, Delicious, Flickrand LastFm [11]. The paper evaluated the suitability oftag hierarchies for navigation on tagging networks andproposed a novel tag hierarchy generation algorithm.

In 2012, Strohmaier, Helic et al. compared differ-ent folksonomy induction algorithms through decen-tralized search [29]. They showed that, based on evalu-ation through navigation, clustering algorithms devel-oped for social tagging systems performed better thanstandard hierarchical clustering algorithms.

Helic et al. applied decentralized search to broadand narrow folksonomies on data from Mendeley [10]and found broad folksonomies better suited to support-ing navigation.

Trattner et al [30] compared decentralized searchand human navigation behavior in information net-works and showed that the simulation of decentralizedsearch yielded very similar results to actual humannavigation data on Wikipedia. In their work, Trattner etal. investigated different types of hierarchies as back-ground knowledge and found that decentralized searchbased on a hierarchy generated from network featuressuch as in- and outdegree simulated human navigationbetter than comparable hierarchies generated from ex-ternal knowledge.

In ongoing research, Helic, Strohmaier et al. arestudying the influence of stochasticity and differentmethods of selecting the next hop in decentralizedsearch [12].

The previous work did not tap into existing on-tologies as background knowledge, but used other ap-proaches (such as automated methods) for this pur-

3www.thewikigame.com

pose. This paper goes beyond previous research by ex-tending the simulation framework with ontologies andby applying Ontology-based Decentralized Search tothe case of Wikipedia and to concrete ontologies forthe biomedical domain.

2.3. Ontologies

Ontologies have been used in previous research tofacilitate navigation in digital libraries. Papazoglouand Hoppenbrouwers [24] have used ontologies to re-trieve related work when searching digital libraries.The research of Rajapakse et al. [28] shows effortsto navigate the digitally available literature related todengue fever. Villela Dantes et al. [32] have studiedthe ontology-guided insertion of links into web pages.In their work, they classified web pages according toan ontology and subsequently inserted links to relatedtopics into web pages to facilitate navigation.

These research papers share the effort to use ontolo-gies to aid navigation. The objective of this paper liesin explaining and modeling user behavior by using on-tologies as background knowledge. The ontologies arehence not used to guide human users but to simulateand possibly explain behavior.

This paper uses three ontologies from the biomed-ical domain. Biomedical ontologies play an impor-tant role in biomedical research [6] and are used for arange of purposes. In the biomedical domain, ontolo-gies have been adapted more frequently than in otherdisciplines [23].

3. Materials and methods

3.1. Introductory example

To illustrate our work, let us introduce the followingexample, depicted in Figure 1: Alice accompanied herfather to a physician, who diagnosed him with a cer-tain cardiovascular disease. Back at home, Alice re-alizes that she forgot the exact name of the condition.However, she remembers that the disease was some-how related to heart rhythm problems. Trying to re-cover the exact name, she goes to Wikipedia; but sinceshe does not know the exact name of the target arti-cle she cannot use the search function to jump to thearticle directly. Alice instead starts from a (hypotheti-cal) Wikipedia portal containing links to a number ofcommon diseases. She first chooses to click the portallink leading to the article on Cardiovascular disease, as


ICD-10

D E I

I10-I15 I30-I52 I60-I69

I50 I51

Hypertension

Heart Failure

Cardiovascular

disease

Vascular disease

I63 I62 I61 I64

Stroke

I47 I48 I49

Cardiac

dysrythmia I47.1

Supraventricular

tachycardia

E65-E68

E66

Obesity

D10-D36

D35

D35.0

Pheochromocytoma

I51.6

Portal

ICD-10 relation

Wikipedia mapping

ICD-10 path

(a)

(b)

(c)

(d)

(e)

4

4

5 1

0

(a) Background knowledge (ICD-10)

Cardiovascular disease

Portal

Pheochromocytoma

Vascular disease

Stroke Cardiac Dysrhytmia

Supraventricular tachycardia

Hypertension

Obesity

Heart Failure

Wikipedia linkICD-10 pathshortest path

(a)

(b)

(c)

(d)

(e)

(b) Wikipedia graph

Fig. 1. Alice’s Wikipedia Navigation Scenario. Looking for a disease, Alice goes to Wikipedia and starts from a hypothetical portal containinglinks to a number of common diseases. Alice then navigates her way through the Wikipedia network.In the figure, we assume that Alice’s background knowledge is represented by ICD-10, a classification of diseases (see Section 3.2 for a detaileddescription). Figure a) shows a part of ICD-10 and the corresponding Wikipedia articles. Figure b) shows a subgraph of the Wikipedia linknetwork. Alice’s path in the graph (red, dashed) is guided by ICD-10, which differs from the shortest path (green, solid). The numbers along theICD-10 path show the distance to the target, according to ICD-10.

this seems to be a good starting point. Next, she navi-gates to the article on Vascular disease, then to Stroke,clicks the link to Cardiac dysrythmia and finally arrivesat Supraventricular tachycardia, which she recognizes asthe disease the doctor had diagnosed her father with.

At each step, Alice is only aware of the links lead-ing away from the current article. She is familiar with

some of the article titles, and is able to relate themto one another through what we refer to as her back-ground knowledge. She recognizes some of the linksand knows what their target article could likely beabout. Since Alice is only making use of the local ar-ticle content and its outgoing links at each step, sheperforms what is called decentralized search.


(a) ICD-10 (b) MeSH (c) SNOMED CT

Fig. 2. Structure of the four top levels of the ontologies used in our research. The figure shows the structure for ICD-10, MeSH and SNOMEDCT. The root node is displayed in black and bold in the center of each plot. The figures show all ontology concepts up until a distance of four fromthe root node. Color indicates distance, with red being close to the root and blue being farther away. SNOMED CT contains roughly 40, 000

nodes in the subgraph displayed, which is significantly more than ICD-10 (∼ 12, 000) and MeSH (∼ 10, 000).

To simulate Alice’s usage of Wikipedia, we firstmapped a subset of biomedical Wikipedia articlesto their corresponding ontology concepts in threebiomedical ontologies. Given these mappings, the sim-ulation could then calculate distance information onthe ontology. For each potential outgoing link that Al-ice could click, the simulation computed the shortestpath between the article behind that link and the targetarticle. This distance information was used as a proxymeasure to estimate the distance to the target articlein the Wikipedia network. The distance informationgained this way was not necessarily optimal or evencorrect, but generally provided a good guess to guidethe navigation.

In this manner, the simulator was able to make aneducated guess about what link to follow, just as Al-ice could roughly place the outgoing articles into cate-gories.

In the rest of the Materials and Methods section, wedescribe the ontologies used to inform the simulator(Section 3.2), the Wikipedia articles and how we ob-tained them (Section 3.3), our method of Ontology-based Decentralized Search (Section 3.4), the naviga-tion scenarios (Section 3.5), the user study (Section3.6) and the simulator implementation (Section 3.7).

3.2. Biomedical ontologies

We used the following three ontologies and termi-nologies (all from the biomedical domain) as back-ground knowledge. With these ontologies, we wereable to i) extract articles from Wikipedia and ii) guidethe next-step selection in the simulator.

The International Classification of Diseases, 10th re-vision (ICD-10) is a classification of diseases, signsand symptoms first published in 1992 and maintainedby the World Health Organization (WHO). ICD-10 hadits origins in the classification of causes of deaths andis presently used by over 100 countries to report mor-tality statistics. It is also widely used for epidemiol-ogy, health management as well as clinical purposesand is available in 46 languages [1]. The version weused contained 12,417 concepts. ICD-10 consists of22 top-level nodes termed chapters and assigns a code(or a range of codes) to every disease in its domain. Inour experiments, we used Wikipedia articles mappingto concepts from all 22 chapters.

Medical Subject Headings (MeSH) is a controlledvocabulary thesaurus for journal articles in the medi-cal domain. MeSH is maintained by the U.S. NationalLibrary of Medicine. The ontology forms a tree struc-ture with 16 top-level concepts and contains 26,142terms (dubbed descriptors) [2]. Descriptors are graphleaves and attached to one or more tree nodes (whichare not descriptors). As such, the complete graph weused contained 80,689 nodes. MeSH extends beyondbiomedical concepts and comprises terms from otherdomains such as Geography, Technology or Publica-tion Characteristics. In our experiments, 96% of theWikipedia articles mapped to the subgraph representedby the Diseases concept, and the rest to the Psychiatryand Psychology subgraph.


Table 1Characteristics of the data sets used for our work. The tablesdisplay statistics about the examined ontologies as well as the set ofWikipedia articles mapping to those articles.

ICD-10 MeSH SNOMED CTOntology concepts 12,417 80,689 295,482

top-level 22 16 19relations 12,416 112,463 440,408

density 8.05×10−5 1.73×10−5 5.04×10−6

depth 4 14 16relation is-a is-a, part-of is-a

Wikipedia articles 1,593links 14,539

density 5.73×10−3

The Systematized Nomenclature of Medicine -Clinical Terms (SNOMED CT) [27] is a clinicalhealthcare terminology used in electronic health recordsystems. The revision we used contained 295,482 con-cepts, which made it by far the largest ontology inour simulations. SNOMED CT consists of 19 top-level concepts. In our experiments, 98% percent of theWikipedia articles mapped to the Clinical finding sub-tree.

Table 1 displays statistics about the data sets used forthis paper. The row denoted density was calculated as

D =|relations|

|concepts|(|concepts| − 1)(1)

for the ontologies (which were regarded as undirectedgraphs) and as

D =|links|

|articles|(|articles| − 1)(2)

for the Wikipedia article network, which formed a di-rected graph. Figure 2 depicts the examined ontologygraphically for the first four hops from the root node.SNOMED CT contains roughly 40, 000 nodes in thesubgraph displayed, which is significantly more thanICD-10 (∼ 12, 000) and MeSH (∼ 10, 000). Still, over-all SNOMED CT is less dense than the other ontolo-gies.

3.3. Wikipedia articles

We used a dump of the English Wikipedia from De-cember 2011 to extract articles from the biomedicaldomain corresponding to ontology concepts. We thenmapped the articles to the ontologies by parsing thearticles’ info boxes.

In disease articles, the Infobox disease4 iscommonly used. It offers several options to referencemedical ontologies such as ICD-10 or MeSH (see Fig-ure 3 for an example). We used template fields in theInfobox disease as well as two other infoboxtemplates to map Wikipedia articles to their ontologycounterparts in ICD-10 and MeSH.

SNOMED CT is proprietary and as such not presentin Wikipedia info boxes. As a consequence, we couldnot directly relate Wikipedia articles to the ontologyconcepts. We therefore used semantic mappings fromBioPortal [36] to map Wikipedia articles to SNOMEDCT. We mapped a total of 1,593 Wikipedia articlesfrom both ICD-10 and MeSH to SNOMED CT withthis method.

3.4. Ontology-based Decentralized Search

3.4.1. IntroductionDecentralized search is a method of solving a

pathfinding problem in a network without a centralcontrol unit. Starting from an arbitrary start nodewithin the network, the objective of decentralizedsearch is to find a way to a given target node. Theterm decentralized stems from the fact that the searchproceeds by forwarding the search problem from onenode to the next, until the target is reached. In Stan-ley Milgram’s small world experiment [31], decentral-ized search was established through humans forward-ing letters to acquaintances with the objective of reach-ing an unfamiliar target person. Each human along thechain of letters acted independently of all others andthus made the search decentralized, i.e., acting with-out a central control unit involved in the decisions atevery step. Further examples for decentralized searchinclude bug forwarding in a developer network, where

4http://en.wikipedia.org/wiki/Template:Infobox_disease


Fig. 3. Example for an infobox template used in disease articles on Wikipedia. Disease articles commonly make use of an Infobox diseasetemplate, which offers fields for ontology codes. We used template fields in the Infoboxes to map Wikipedia articles to their ontology counterparts.

software bugs are assigned to a starting person, andthen forwarded to other developers until they are fixed[20], or job recommendations in social networks [4].

In a social network, the decision of where to forwardthe problem is generally based on the expected knowl-edge and capability of that particular next node (per-son). For our simulations, we assumed that all nodesshared a common background knowledge expressed asan ontology. This assumption made our algorithm less"decentralized" in a certain sense because all the de-cisions were now made by the same entity (our sim-ulator). Just like in the original decentralized searchhowever, at each node the simulator could only ac-cess information about that particular node’s local net-work neighborhood. The background knowledge rep-resented additional knowledge about the network nec-essary to efficiently find a short path to the target.When looking for an employee in a company for exam-ple, this knowledge could represent the organizationalhierarchy - with the restriction that the search can onlybe forwarded to acquainted employees, which wouldfor instance be the case with personal recommenda-tions.

In the theory of network navigability, Jon Klein-berg showed that networks that are formed accordingto a background hierarchy (i.e., a tree) are efficientlynavigable [15], provided the search agent has accessto that background hierarchy during the search. Thismethod, called Hierarchical Decentralized Search, has

been successfully applied in previous research ([12],[29]). This paper extends this application by using on-tologies as the background knowledge.

3.4.2. SimulationsBased on Alice’s navigation example, we imple-

mented our navigation simulations based on decen-tralized search. Recall that Alice did not rememberthe exact name of her navigation target, but she couldroughly place it in a category. She then found her wayto this target using her own background knowledge.

To be able to represent this scenario in the computersimulations, the title of the target article (but not itsposition in the article network) was directly known tothe simulations. This modeled the same scenario - thesomewhat familiar article Alice was trying to reach.The next link to click was determined by calculatingthe distance from the current node to the target nodeon the background knowledge. For instance, in the ex-ample shown in Figure 1, if we assume the current po-sition to be at the Stroke article there are two options:clicking the link to Heart Failure or to Cardiac dysryth-mia. The objective would be to select the one lead-ing closer to the target article Supraventricular tachycar-dia. While Alice decides this based on her backgroundknowledge, the simulations made use of an ontology(e.g., ICD-10) and calculated the distances from eachof the linked concepts to the target article in the ontol-ogy. In the example at hand, the distance was smaller


for Cardiac dysrythmia, leading to a simulated click onthe link to this concept.

To avoid loops, the simulation explored each node inthe network only once. However, the simulation couldbacktrack to the last visited nodes (up until the start-ing portal, if necessary), just as Alice would use herbrowser’s back button. This was used in case of deadends (articles with no unvisited outgoing links) or atarticles providing only links leading further away fromthe target (according to the ontology information). Atany given point, the simulation could also jump backto the starting portal directly, modeling a home buttonin an information system.

3.4.3. Usage of ontologiesThis use of existing ontologies represents a substan-

tial change in the motivation of the background knowl-edge: As opposed to previous work in this area, thebackground knowledge is now exogenous to the net-work. What this implies is that the hierarchy is basedon knowledge independent of the network that theagent navigates on. All ontologies used in the appli-cation of Ontology-based Decentralized Search in thispaper play a key role for their corresponding domain intheir research fields. They are hence representative fora good part of the knowledge in these domains. Thisprovides OBDS with a foundation to more accuratelyrepresent the intuitions of human navigation behavior.

The use of ontologies and the associated semanticinformation open up a range of new possibilities forthe application of the background knowledge:

– Filtering by relations and properties: Ontologiesare (in general) made up of different types of rela-tions (such as is-a or part-of, or regulates), whichcan be used to extract different varieties of back-ground knowledge from one and the same ontol-ogy. For example, a hierarchical version of the on-tology could be extracted by following only theis-a relations. Furthermore, ontologies may as-sign properties to their concepts. A backgroundknowledge can hence also be restricted to ontol-ogy concepts with a certain property. An ontologycould be filtered to contain only contain conceptsstemming from a single domain, such as geogra-phy. This could then be compared with the nav-igation behavior based on other filtered versionsof the ontology, such as politics or economics.

– Modeling different user groups: Ontologies canalso be used to model different types of users.A good example for this is the case of ICD-10,which provides a classification of diseases. In the

ontology, the depth of a disease (i.e., its distancefrom the root node) corresponds to its specificity.This could be used to model the knowledge of dif-ferent hospital personnel. For instance, a medicalspecialist could be modeled by the entire depth ofknowledge of one section of the ontology, and adepth-limitation in the other sections. A layper-son could be modeled by having a certain depth-limitation in all areas. This could be effectivelyused to simulate different user groups in medicalinformation systems, without having to carry outactual human user studies. Analogously, differentgroups of users could be represented by differ-ent ontologies. To correctly model heterogeneoususer groups, such as a group consisting of doc-tors, nurses and laypersons, each of the subgroupsneeds to be modeled separately using several ap-propriate ontologies or adapted version thereof.

– Inference: Ontologies permit inference on theirentities. For hierarchical relations this could meanthat subconcepts could be assigned the type oftheir superconcepts (e.g., the perhaps unfamil-iar Supraventricular tachycardia is a subconcept ofHeart Disease in ICD-10, which is more com-monly known). In the case of the cut-off back-ground knowledge, more specific ontology con-cepts could then be substituted by their inferredsuperconcepts and provide more information tothe navigation process than a pure random guess.

In the experiments conducted for this paper, Ontology-based Decentralized Search was used with three dif-ferent ontologies (ICD-10, MeSH and SNOMED CT)that were not filtered. This meant that all conceptsand relation types present in the ontologies (and map-ping to the data sets) were used as the backgroundknowledge. In terms of relation types, ICD-10 andSNOMED CT consist of is-a relations. For MeSH,which is more a controlled vocabulary than a properontology, the relations are of the types is-a and part-ofand are both used without clear distinction. For thispaper, Ontology-based Decentralized Search ignoredany properties associated with the ontology concepts.

The results of the simulations were then comparedon the same information network, which meant thatthe three ontologies effectively modeled different usergroups on the same set of data. While all three ontolo-gies represent expert knowledge, they still serve differ-ent purposes: ICD-10 is a disease classification widelyused by insurance companies, physicians and hospi-tals. SNOMED CT is a terminology of clinical terms,


Fig. 4. Starting portals used in navigation simulations. For ICD-10, MeSH and SNOMED CT we used a portal obtained by mapping navigationbar articles from WebMD.com to Wikipedia articles (subfigure a).

and MeSH is a terminology for the indexing of journalarticles in the medical domain. We hence assume theseontologies to be representative of expert knowledge intheir respective biomedical fields.

3.4.4. LimitationsPrevious research on decentralized search has stud-

ied the stochasticity of human navigation behavior[12]. As the aspiration for this work was not to improvethe navigation model but to examine the suitability ofontologies to support navigation, this more complexnavigation model was beyond the scope of this article.

For this work, we assumed the ontologies to be staticrepresentations of user knowledge. As the number oftasks in the users study was limited to 15 and all taskswere independent of each other, we assumed learningto be negligible. In future efforts, we would also liketo include a learning component, such as users memo-rizing preferred paths through the network or makingnew associations while navigating.

The model used for this paper is also limited to a sin-gle domain, and hence to a single ontology at a time.Fulfilling multiple constraints at once is not possibleas of now. Future work could look into more extensivedomains that need to be covered by multiple ontolo-gies.

3.5. Navigation scenarios

We studied two different search scenarios, both ofwhich started from a hypothetical Wikipedia portal.

Starting portal We started the navigation from ahypothetical Wikipedia portal featuring a selectionof suitable articles: the 25 health conditions listedin the main navigation toolbar of WebMD.com (seeFigure 4). We manually mapped these conditions toWikipedia articles from our dataset and used the arti-cles as the outgoing links from our artificial portal. In away, the artificial portal thereby resembled the naviga-tional structure of the WebMD front page - a popularhealth information web site. Medical web sites, suchas WebMD are frequently used to obtain informationabout diseases, or as a first information before consult-ing a medical doctor [5].

Single-target search The first scenario was analo-gous to Alice’s introductory example. This models thescenario of having a vague concept in mind (but notthe exact term), or having a concept on the tip of one’stongue. The goal of single-target search is then to re-discover this concept via navigation on the Wikipediaarticle network.


In single-target search, the simulation started at theportal and proceeded to a single target article usingOntology-based Decentralized Search.

Multiple-target search In the second scenario, westudied Ontology-based Decentralized Search in thecontext of exploratory search. In exploratory search,users explore a space of resources rather than trying tofind one specific target [19]. The objective of multiple-target search is therefore to gain an overview of a cer-tain field of concepts. One example could be to esti-mate a range of diseases associated with certain symp-toms (such as "What diseases could show the symp-toms of heart burn, fatigue and headaches?").

In the simulations for multiple-target search, the dif-ference to single-target search was in the targets, whichconsisted of target sets of two to fifteen articles (in-stead of only a single one). The rest of the simula-tion (starting portal, decentralized search, backgroundknowledge) was conducted in the same way as thesingle-target search.

We used clusters of semantically similar Wikipediaarticles as our target sets and applied k-means clus-tering to arrange similar articles into clusters basedon TF-IDF features (using scikit-learn [25]). We usedthose resulting clusters containing two to fifteen arti-cles in our simulations. Examples for clusters are givenin Table 2.

Maximum number of clicks For all evaluations, amaximum number of 20 clicks for the single-targetscenario and 40 clicks for the multiple-target scenariowere assumed. While these limits are certainly vari-able in the setting of navigation, the chosen numbersrepresent a reasonable limit of potential clicks that auser might undergo to find the target article on an infor-mation network such as Wikipedia. Analysis of Wikigames has revealed average path lengths of fewer thanseven clicks, with some games taking up to 30 clicks[9] [35]. Longer paths might also skew the results, asthey are more likely a result of guesswork and not ofan exploitation of background knowledge.

3.6. User study

To evaluate our simulations, we carried out a userstudy on Wikipedia navigation. Eight participantswithout any particular background in medicine wereasked to navigate Wikipedia, modeling the scenario ofnavigating to find diseases. All of them were gradu-ate students in different fields (but not in medicine) atStanford University at the time of the user study.

With the OBDS approach, we assume that the on-tology represents an expert with profound knowledgeof the entire domain. As such, we expect experts tobe able to be able to classify diseases to their approx-imate position in the ontology, which is precisely whythe simulations have access to the entire ontologies asbackground knowledge. As we conducted our studywith layperson users, however, we could only expectthem to be only familiar with a smaller set of diseases.

The study used the data set of ICD-10, SNOMEDCT and MeSH, containing 1,593 Wikipedia articles.Naturally, a large share of these articles turned out tobe too specialized for participants not particularly fa-miliar with the medical domain (with article namessuch as Halitosis, Aniseikonia or Milroy’s disease, whichleft users puzzled in a pilot study). In order for the on-tologies to be compatible with the non-expert users,we limited the target articles to 100 manually selectedand generally better known targets (such as Pneumo-nia, Stomach cancer or Asthma), out of which we alsomanually formed 20 clusters of four articles each.

The setup for the user study consisted of a servercontaining the subset of Wikipedia with an interfacesimilar to Internet encyclopedia (see Figure 4). The in-terface showed the articles used by the study, as wellas information about the current task. As in our sim-ulations, backtracking (using the back button in thebrowser) and jumping back to the portal by clicking ahome link were enabled at all times. Each step of theusers was logged.

We asked each of our participants to complete a to-tal of 15 navigation tasks. A task consisted of finding agiven target node (or a set of target nodes) in the sub-set of the Wikipedia network. As in the simulator, thestarting point for a task was always the portal, and par-ticipants could only click on links to articles within thedata set. To deal with potential frustration, participantswere given the possibility to abort the current task ifthey had not found the target(s) after half of the maxi-mum number of steps (20 for single targets and 40 formultiple targets).

3.7. Implementation

The experiments presented in this paper were con-ducted on a decentralized search simulator. This sim-ulator was an extension of previous work by Helic,Strohmaier et al ([11], [29]) and implemented inC++ based on the Stanford Network Analysis Projectframework [3]. It permitted the simulation of decen-tralized search on a given network and used a provided


Table 2Examples for clusters of Wikipedia articles used in exploratory search. The table shows three examples of clusters used in our simulations.We used TF-IDF features and k-means clustering to automatically group Wikipedia articles into semantically related groups of two to fifteenarticles.

Nausea-related Stomach-related Cough-relatedVomiting Linitis plastica BronchitisNausea Stomach cancer Chronic bronchitisMotion sickness Gastritis Acute bronchitisMorning sickness Atrophic gastritis CoughDrooling Ménétrier’s disease SputumHyperemesis gravidarum Achlorhydria

GastroparesisDuodenal cancerGastric dumping syndromeStomach disease

background knowledge to calculate the distances. Thesimulator was used to run a total of 1794 simulationsof decentralized search with two navigation scenarios.

4. Results

4.1. Evaluation metrics

Based on work by Krioukov and Papadopoulos [17]we used success ratio and stretch to evaluate naviga-tion paths.

In accordance with Strohmaier and Helic [12], wedefine the success ratio s to be the fraction of targetnodes found and the stretch τ to be the average ratio offound path lengths to shortest path lengths.

Let P be the set of target nodes and W be the set oftarget nodes that were successfully navigated to by oursimulator. Then we have that the success ratio s is

s =|W ||P |

. (3)

Thus, the success ratio measures the extent to whichthe simulator is successful in finding targets, e.g., asuccess ratio of 0.9 states that 90% of the targets werefound. Furthermore, let l(t) be the length of the short-est path from the portal to the target node t and leth(t) be the length of the path to the target found by theagent. The stretch τ is then defined as

τ =1

|W |∑t∈W

h(t)

l(t). (4)

Stretch measures the efficiency of search. For example,a stretch of 1.2 states that the paths an agent was ableto find are - on average - 20% longer than the shortestpaths for these targets. As in work by Helic, Trattneret al ([11], [30]), we report success ratio and stretchsplit by path length of the underlying node pairs. Thesemetrics give us a means of analyzing what paths werefound by the simulator and how much longer than theshortest paths they were.

We further extend these metrics with the accumu-lated success ratio as, which we define as the fractionof nodes found up until a certain number n of steps.Let

as(n) =|Wn||P |

, (5)

whereWn is the set of target nodes reached by the sim-ulation in up to n steps.

For all our evaluations, we assumed a maximumnumber of 20 clicks for the single-target scenario and40 clicks for the multiple-target scenario.

4.2. Comparison with random baselines and optimalsolutions

We established comparisons with random and opti-mal solutions by including a random walk, randomlygenerated ontologies and a shortest-path solution.

Random Walk The random walk consisted of follow-ing a random link (or tracking back) at each step, nottaking already visited nodes or potential targets amongthe neighboring nodes into account. The comparisonwith the random walk showed us how much more in-


formation the OBDS approach provided to the naviga-tion in comparison to a completely random behavior.

Randomly generated Ontologies For this compari-son, we constructed a randomly generated ontologycounterpart to each ontology used in our simulations.To this end, we used the number of nodes and edges asinput for the configuration model approach of generat-ing a random graph with the same number of nodes andedges [22]. As the resulting graph was not necessar-ily connected, we subsequently randomly connectedall graph components and then removed the numberof additional edges created in this process from otherparts of the resulting graph without deconnecting it.

This comparison showed us how much informa-tion the OBDS approach gained by taking the struc-ture of the ontologies into account (but not yet thecorrect mappings). Furthermore, evaluating with ran-domly generated ontologies took the structured searchbehavior of decentralized search into account: De-centralized search, in our implementation, did not re-explore already visited nodes, could backtrack and al-ways recognized links leading to a target node amongthe current node’s neighbors. This gave this method adistinct advantage over the pure random walk.

Shortest-path solution Finally, for the comparisonwith the optimal approach, we computed a shortest-path solution. In the single-target scenario, this meantthat we always used the shortest possible path in thegraph for connecting the portal to the target node. Forthe multiple-target scenario, an exact solution wouldhave required solving an instance of the traveling-salesman problem, which is computationally expen-sive. To circumvent this issue, we approximated theperfect solution with a nearest-neighbor approach thatalways took the shortest possible path to the nearestneighbor. This allowed us to compare to the (approx-imately) optimal solution. It is important to note thatthis was only possible with global knowledge of thegraph topology, which users do not posses in a decen-tralized search scenario.

4.3. Evaluation

To compare the performance of OBDS with differ-ent ontologies as background knowledge, we evaluatedmultiple ontologies on the same set of Wikipedia ar-ticles. This allowed us to inspect the ontologies sideby side, facilitating comparison. Figure 5 (left column)displays the results for OBDS using ICD-10, MeSHand SNOMED CT.

Figure 5 shows that the success ratio for OBDS withICD-10, MeSH and SNOMED CT was well aboveboth the random walk and OBDS with the randomlygenerated ontologies. When comparing the biomedi-cal ontologies, the results show that OBDS with ICD-10 performed best, followed by MeSH and SNOMEDCT for the success ratio. Cochran’s Q test showed sta-tistically significant differences for the success ratios(p < 0.01). Post-hoc McNemar tests (with Bonferronicorrection) showed that ICD-10 and MeSH were sig-nificantly better than SNOMED CT, that the biomed-ical ontologies were better than the random baselinesand that OBDS with the randomly generated ontolo-gies was better than the random walk (all significant atp < 0.01).

For the stretch, ICD-10 lead to the best results fol-lowed by SNOMED CT, which in turn fared slightlybetter than MeSH (all differences statistically sig-nificant by Friedman tests with Post-hoc Wilcoxonsigned-rank tests with Bonferroni correction, all forp < 0.01).

For the accumulated success ratio, ICD-10 andMeSH informed the navigation significantly betterthan SNOMED CT. In addition, ICD-10 was betterthan MeSH up until close to the maximum numberof steps. OBDS with the biomedical ontologies per-formed better than with randomly generated ontolo-gies, which in turn were better than the random walk(all differences statistically significant by Friedmantests with Post-hoc Wilcoxon signed-rank tests withBonferroni correction, all for p < 0.05, all tested atN = 5, 10, 20, 40).

One explanation for the variations in the perfor-mance for ICD-10, MeSH and SNOMED CT couldbe the different popularity of the ontologies. As themost widely used statistical classification system inthe world, ICD-10 evidently has the highest influ-ence on Wikipedia editing behavior. Not only do mostdisease articles reference their corresponding ICD-10code, but also disease categories are arranged based onthe ICD-10 model. This could explain why ICD-10 isslightly better suited to navigate the encyclopedia’s ar-ticle network in these evaluations. Many disease arti-cles on Wikipedia also reference MeSH, making thesame arguments apply for it as well. While both MeSHand ICD-10 are available freely online, SNOMED CTis not available in all countries and perhaps less-knownin the Wikipedia community. SNOMED CT is also notreferenced in the infobox templates of disease articles,making it perhaps a poorer candidate to inform thenavigation.


1 2 3 4 5 6

Shortest Path (#hops)

0

20

40

60

80

100

Succ

ess

Rat

io(%

)

Single-target: Success Ratio

ICD-10 (0.55)MeSH (0.53)SNOMED CT (0.45)Optimal (1.00)Rand. Walk (0.01)Rand. ICD-10 (0.11)Rand. Mesh (0.10)Rand. SNOMED CT (0.11)

(a) Success Ratio for ICD-10/MeSH/SNOMED CT

1 2 3 4 5 6


0

20

40

60

80

100

Succ

ess

Rat

io(%

)

Single-target: Success Ratio

ICD-10 (0.89)MeSH (0.91)SNOMED-CT (0.79)User Study (0.92)Optimal (1.00)Rand. Walk (0.05)Rand. ICD-10 (0.45)Rand. Mesh (0.44)Rand. SNOMED CT (0.46)

(b) Success Ratio for the User Study

1 2 3 4 5 6


0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Stre

tch

Single-target: Stretch

ICD-10 (2.11)MeSH (2.49)SNOMED CT (2.45)Optimal (1.00)

(c) Stretch for ICD-10/MeSH/SNOMED CT

1 2 3 4 5 6


0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0St

retc

hSingle-target: Stretch

ICD-10 (1.84)MeSH (1.78)SNOMED-CT (1.81)User Study (1.74)Optimal (1.00)

(d) Stretch for the User Study

0 5 10 15 20 25 30 35 40

Step

0

20

40

60

80

100

Foun

dTa

rget

s(%

)

Multiple-target: Accumulated Success Ratio

ICD-10MeSHSNOMED CTOptimal

(e) Accumulated Success Ratio for ICD-10/MeSH/SNOMEDCT

0 5 10 15 20 25 30 35 40

Step

0

20

40

60

80

100

Foun

dTa

rget

s(%

)

Multiple-target: Accumulated Success Ratio

ICD-10MeSHSNOMED-CTUser StudyOptimal

(f) Accumulated Success Ratio for the User Study

Fig. 5. Success ratio, stretch and accumulated success ratio for ICD-10/MeSH/SNOMED CT and the user study. The first column showsthe results for ICD-10, MeSH and SNOMED CT, the second column the results for the user study. The rows show success ratio and stretch (bothwith overall values in parentheses) and accumulated success ratio, respectively. The legends in the first row are valid for the entire columns. Theplots include the computer-generated random baselines (dashed and dotted) as well as the optimal solution. Note that the stretch plots do notinclude the random baselines, as this measure can only be usefully applied to compare simulations with a similar number of found paths. Thefigures show that the results produced by Ontology-based Decentralized Search with biomedical ontologies are noticeably better than the resultsfor randomly generated ontologies. The figures also show that the results of Ontology-based Decentralized Search on a limited data set are in therange of the results produced by human participants.


0 5 10 15 20

Path Length

0

10

20

30

40

50

Pro

duce

dPa

ths

(%)

Single-target: Path Lengths


(a)

0 5 10 15 20 25 30 35 40

Path Length

0

10

20

30

40

50

Pro

duce

dPa

ths

(%)

Multiple-target: Path Lengths


(b)

Fig. 6. Path lengths produced by the user study and the simulator. The figures show the resulting path lengths for the single-target (a) andmultiple-target (b) search scenarios. Navigation was limited to 20 and 40 steps, respectively, hence the high number of paths for these lengths(i.e., not all targets were found). The path distributions for the random walk and the randomly generated ontologies were left out for reasons ofclarity.

4.4. User study

For the user study, we evaluated the results of humannavigators on a set of 100 manually selected targetsand 20 manually selected clusters. We then comparedthe performance of OBDS, the computer-generatedrandom baselines and the optimal solution on the sametargets. Note that the subset of targets was a maximumdistance of three hops away from the portal. The eval-uations hence do not include any data points for longershortest paths.

4.4.1. Success Ratio, Stretch and AccumulatedSuccess Ratio

Figure 5 (right column) shows that the success ratiofor the user study was fairly close to the simulator. Forthe single-target scenario, the overall success ratio was92% for the user study and ranged from 79% to 91%for OBDS supported by the three biomedical ontolo-gies. Cochran’s Q test showed statistically significantdifferences (p < 0.01). Post-hoc McNemar tests (withBonferroni correction) showed that statistically signif-icant differences exist from all groups to OBDS withthe randomly generated ontologies, as well as from thelatter to the random walk. This showed that, as for theevaluation of ICD-10, MeSH and SNOMED CT onthe unrestricted target set, OBDS with these ontolo-gies outperforms random approaches. The user studywas significantly different only to the random base-lines (p < 0.01), but not to the biomedical ontologies

or to the optimal solution (which might be due to thethe users finding almost all targets in the study).

For the single-target stretch, with an overall stretchof 1.74 the user study performed slightly better thanthe simulator, which displayed stretches between 1.78and 1.84. A Friedman test showed that groups weresignificantly different (p < 0.01). Post-hoc Wilcoxonsigned-rank tests (with Bonferroni correction) showeda significant difference of the user study and SNOMEDCT (p < 0.05) but no statistically significant differ-ences between the user study and ICD-10 or MeSH.

For the multiple-target scenario, the accumulatedsuccess ratio showed that the user study fell within orjust below the range of OBDS with one of the threebiomedical ontologies. A Friedman test showed thatstatistically significant differences existed between thegroups (p < 0.01), tested for N = 5, 10, 20, 40.

For post-hoc Wilcoxon signed-rank tests with Bon-ferroni correction, results varied over the tested val-ues of N . The random walk was significantly differ-ent to all other groups at all values of N (p < 0.01).The user study was significantly different only to therandom walk (for all N ) and to the optimal solution(for all N except N = 5), all at p < 0.01. This sug-gests that the users found near optimal paths for thefirst part of their click trails. It is worth noting that after20 steps, the users in our study did not find any moretargets. This coincides with the point from where onusers were given the possibility to abort a search taskif they could not find the target.


Table 3Kullback-Leibler divergence for the path length distributions produced by the simulator and the user study. The table shows the KLdivergence from the user study to the ontologies and the optimal and random solutions. This metric measures the number of additional bitsrequired to encode the original distribution, if another distribution is used in its place. The Randomly Generated Ontology column was computedusing an average over the three randomly generated ontologies considered. The table shows that the user study was more similar to the ontologiesthan to the base lines for the single-target scenario. The results closest to the user study are displayed in bold.

User Study ICD-10 MeSH SNOMED CT Optimal RandomlyGeneratedOntology

Random Walk

single-target 0.12 0.08 0.18 0.46 0.97 2.56multiple-target 1.01 0.74 0.84 1.63 0.55 1.29

4.4.2. Path lengthsTo obtain qualitative insight into the navigation pro-

cess, we compared the resulting distribution of pathlengths produced by both the user study and the sim-ulations. This distribution can be seen in Figure 6. Wethen computed the Kullback-Leibler (KL) divergencefrom the user study distribution to the other distribu-tions. The Kullback-Leibler divergence measures thenumber of additional bits needed to encode the pathlength distribution, if the other distribution is used inplace of the original (user study) path length distri-bution. The resulting values can be seen in Table 3.For the single-target search scenario, it is clearly vis-ible that only OBDS with biomedical ontologies pro-duced path length distributions close to the user study:All three ontology path length distributions had a verysmall KL divergence (0.08 - 0.18 bits) to the userstudy. This means that, as far as produced path lengthsare considered, it appears justifiable to replace humannavigation data with data produced by OBDS and a fit-ting ontology. The same cannot be asserted about ran-domly generated ontologies (nor the random walk orthe optimal solution), which cannot be easily taken inlieu of the biomedical ontologies and yield similar re-sults.

For the multiple-target search scenario, these claimscannot be made this clearly. However, this might bedue to the path length distribution for the multiple-target scenario being rather sparse, which meant thata single path accounted for five percent of the pathlengths (which is also reflected in Figure 6 b).

4.4.3. Further evaluationIn addition to the path length distributions, we ana-

lyzed further aspects of the user study in comparisonwith OBDS (see Table 4).

First, we looked into the found targets and the firstclick after the portal. To compare these, we arrangedthe nodes into vectors and computed cosine similari-ties.

For the found targets, all three ontologies displayedhigh cosine similarity values. This reflects the resultsfrom Figure 5, and is caused by high success ratiosfor the limited target set used in our user study whichleads to the majority of the vectors containing onesat the same positions. As discussed in the examina-tion of the success ratios, the user study differed sta-tistically significantly from the random walk in bothsingle and multiple-target search and in addition fromthe randomly generated ontologies in the single-targetcase (all p < 0.01).

For the first hops (i.e., the very first click in thesearch), the clicks were distributed rather evenly. Auniformly random distribution would see each linkclicked 3.7% of times. Our results showed distribu-tions ranging from 1 to 17% and were thus fairlyevenly distributed, explaining the values of the cosinesimilarity being in the same range for all groups. Forthe single-target scenario, ICD-10 displayed the mostsimilar values to the user study in the single-target sce-nario, where it was also close to the optimal solution.This suggests that the participants were able to selectthe best first hop in most cases. In the multiple-targetscenario, the user study was less similar to the othergroups.

In addition to calculating similarities, we also in-spected the average per-step probability of back-tracking or clicking the home button. Statistical testswere conducted with Friedman tests with post-hocWilcoxon signed-rank tests with Bonferroni correc-tion.

Both the simulation and the users had access to aback button (leading to the previously visited page)and a home button (leading back to the portal) at alltimes. As it turned out, the simulations used the homebutton only immediately after having found a target inthe multiple-target search scenario. In all other cases,the best strategy given by our simulation constraintswas backtracking. The user study showed differentbehavior from the simulator in several aspects: For


Table 4Details of the user study and the compared data sets. The table displays statistics about the user study and the ontologies. The most similarvalues to the user study are displayed in bold face. For the first three measures, we viewed the information about found targets and first hops as avector of values, for which we calculated the angle to the vector containing the information for the user study (i.e., the cosine similarity). For therandom walk, we averaged over 1000 random walks for each portal-target pair. The last two measures display the average per step usage of theback and home buttons for the different scenarios. In summary, the results confirm that what has appeared somewhat apparent from the successratios and the stretch, i.e., that ICD-10 and MeSH displayed the most similar behavior to the user study. The randomly generated ontologycolumn was computed using an average over the three randomly generated ontologies considered. The number of clicks ranged from 195 to2000 for the single-target scenario (user study: 485) and from 111 to 800 fro the multiple-target scenario (user study: 419).

User ICD-10 MeSH SNOMED Optimal Random RandomStudy CT Ontology Walk

Found targets Single 1.00 0.93 0.95 0.89 0.95 0.78 0.72(Cosine Similiarity) Multiple 1.00 0.94 0.94 0.91 0.95 0.90 0.67

First Hops Single 1.00 0.89 0.85 0.69 0.88 0.77 0.80(Cosine Similiarity) Multiple 1.00 0.64 0.62 0.56 0.68 0.64 0.71Back Button Uses Single 0.09 0.13 0.11 0.13 0.00 0.26 0.07(average per step) Multiple 0.27 0.17 0.18 0.18 0.01 0.21 0.09

Home Button Uses Single 0.02 0.00 0.00 0.00 0.00 0.01 0.00(average per step) Multiple 0.01 0.03 0.02 0.03 0.00 0.05 0.00

single-target search, users backtracked less frequently(9% of clicks were back button clicks, versus 11-13%for the simulations, significant at the p < 0.05 level)but used the home button in 2% of clicks, versus never(0%) for OBDS with the biomedical ontologies (sig-nificant at p < 0.01).

For the multiple-target search, users backtrackedmore frequently (27% versus 17-18% for the simulatorand used the home button less frequently (1% versus2-3%). However, this was not statistically significant,possibly due to the small sample size.

In conclusion, backtracking appeared to be the mostwidely applied strategy for navigating out of dead endsand backtrack from less promising areas of the net-work.

5. Discussion

In this work, we studied simulated user naviga-tion behavior via decentralized search. We intro-duced Ontology-based Decentralized Search (OBDS),a novel navigation simulation method based on decen-tralized search which uses ontologies as backgroundknowledge. We showed that our method can be suc-cessfully applied to navigation in information net-works, and demonstrated that it can be applied onWikipedia when supported by biomedical ontologies.

In the following, we want to focus our discussion onthe research questions raised in this work:

RQ1 Can ontologies contribute useful informationto modeling navigation in information networks? Andhow does OBDS using real-world ontologies performin comparison to OBDS with randomly generated on-tologies and random walks?

We found that ontologies can indeed inform naviga-tion in information networks. OBDS with medical on-tologies as background knowledge was able to outper-form the random baseline approaches significantly.

RQ2 Does Ontology-based Decentralized Search(OBDS) produce valid results, i.e., are the simulatednavigation paths similar to those produced by humannavigation?

We addressed this question by comparing proper-ties of the simulated navigation paths with propertiesproduced by humans in a study. We found that theclick paths produced by OBDS matched success ra-tios, stretches and accumulated success ratios of hu-man paths better than pure random walks and OBDSwith randomly generated ontologies. Furthermore, theresulting path length distribution of the user study wasbest matched by OBDS with biomedical ontologies.

RQ3 When using OBDS, what ontology is bestedsuited to produce human-like navigation results?

From our results, it seems that ICD-10 and MeSHseem to perform best. However, the overall differencesbetween the ontologies were not very strong, and it issubject of ongoing research to further identify differ-ences in the performance of OBDS with different on-tologies.


5.1. Further comments

We’ve limited our work to ontologies and Wikipediaarticles from the biomedical domain in this paper. Inthis domain, ontologies have been adapted more fre-quently than in other disciplines [23], play an impor-tant role in biomedical research [6] and are used fora range of purposes. Without this frequent use, the re-sults might be different in other domains. Another im-portant aspect was the ready availability of infoboxtemplates on Wikipedia articles, which facilitated themappings to the ontologies. However, the principles ofour method apply for other domains as well.

Influence of ICD-10 The International Classificationof Diseases (ICD-10) has found widespread use andprobably influences and inspires Wikipedia editors. OnWikipedia, disease articles are almost always indexedby ICD-10 as the first entry in the article infoboxes.Furthermore, the category system for the disease arti-cles of the English Wikipedia is organized accordingto ICD-10. These two facts and the wide use of ICD-10 have quite possibly also influenced the link creationbehavior on the encyclopedia as well as the generalknowledge of the test participants. This might be anexplanation of why ICD-10 seems to be best suited tomodel human navigation behavior in our case study.

User study In comparison to the simulator’s perfor-mance, participants in the user study performed betterfor single-target search and worse for multiple-targetsearch. This is also influenced by the fact that usersaborted 30% of their multiple-target navigation tasksbefore having found all the targets, while the simula-tions ran for whole number of possible steps.

Building user models By using different ontologiesas background knowledge, our results could help re-searchers and engineers build and evaluate user inter-faces with different user types. The ontologies com-pared in the results were rather similar and mostlyshared the same domain. In future work, it will be in-teresting to compare ontologies that do not cover theentire domain, modeling specialist users, or combiningontologies to form a more complete coverage of thedomain. Another idea might be to prune the ontologiesat a certain depth, modeling broad generalist knowl-edge that does not extend beyond a certain depth.

Action selection The simulations in the form we pre-sented followed a deterministic greedy action selectionmodel, in that it always selected the most profitablelink according to the background knowledge. Related

research has shown that users might be better mod-eled using epsilon-greedy action selection mechanismswith dynamically changing epsilons [12]. In follow-up research, our work could be extended with stochas-tic action selection mechanism such as epsilon-greedy.This would also lead to another potentially crucial as-pect of the present simulations, namely the need toevaluate games multiple times with potentially vary-ing results. One could expect that these adaptationswould help to finetune and validate any future attemptsat modeling human navigation. However, we leave thistask to future research.

5.2. Future work

The user study we presented was limited in that itwas restricted to a subset of target nodes because of therequirement that the target be familiar to participantswithout a medical background. Since the simulationresults for these targets were very close to the resultsof the participants, it can be hypothesized that the userbehavior for the whole set of targets is likewise similar.It is up to future research to show more details of thecomparison of human users and decentralized search.

Another aspect was the limitation of the user studyto a subset of the target articles. The user studytried to approximate non-medical students with expertbiomedical ontologies. While this worked to a certainextent, it will be interesting to see further user studieswith medical experts and compare their results on theentire data. Since the user study was limited to eightparticipants, conducting a similar study with a largernumber of users could reveal more interesting results,especially for the multiple-target scenario.

The chosen portal, based on WebMD.com, undoubt-edly influenced the navigation results. It is up to futurework to compare different portals and shed a light onpossible differences.

The idea to navigate to one single predefined tar-get might seem somewhat artificial in the case of userbehavior concerning explorative tasks. However, oneidea to improve on this might be to calculate the TF-IDF features of the target node beforehand and subse-quently navigate until a page (or a number of pages)similar enough to the TF-IDF features has been found(which does not need to be the predefined target page).This could model the case of users exploring areas ofthe network.

Other potential research questions might include thelimitation of visible links to links in the upper part ofWikipedia articles and comparing the results on non-


English editions of the encyclopedia. Past research[13] has already compared different methods of ex-tracting background knowledge from the actual net-work used for navigation. These background knowl-edges were based on network features such as central-ity or degree. It would be interesting to directly com-pare these extracted background knowledges with on-tologies and analyze the differences.

6. Conclusions

In this work, we have presented a novel, ontology-based method (Ontology-Based Decentralized Search)for simulating human navigation in information net-works such as Wikipedia. Our results provide tech-nical answers to several questions regarding the useof ontologies in decentralized search: We have notonly presented a method to integrate ontological back-ground knowledge into decentralized search, but alsofound that ontologies can serve as efficient back-ground knowledge. With appropriate ontologies andWikipedia link networks, our simulations using OBDS(i) found targets more efficiently than two baseline ap-proaches (random walks or randomly generated on-tologies) and (ii) produced navigational paths that weremore similar to actual human navigational paths thanto baseline approaches. While our human study waslimited in terms of - for example - size, the results re-ported in this paper are encouraging in several ways.First, our method opens up ways to explore the effectsof assuming different kinds of background knowledgeof users in a navigation task. For example, swappingdifferent kinds of ontologies in future work would al-low us to explore their impact on the efficiency of de-centralized search in information networks. Second,our results can be seen as additional corroboration thatontologies indeed capture useful knowledge about adomain. In some of our experiments, OBDS with med-ical ontologies as background knowledge was able tooutperform baseline approaches significantly.

Summarizing, our findings are relevant for re-searchers interested in new applications for ontologiesor interested in modeling navigation in informationnetworks using ontologies as background knowledge.

7. Acknowledgements

This research was supported by the Austrian Sci-ence Fund (FWF) grant P24866 and by NIH grant

U54-HG004028. Part of the research of the first au-thor was carried out during a visit at the Stanford Cen-ter for Biomedical Informatics Research and was sup-ported by a Marshall Plan Scholarship granted by theAustrian Marshall Plan Foundation.

References

[1] International classification of diseases, revision 10.http://www.who.int/classifications/icd/en, 2012. last accessed29-May-2014.

[2] Medical subject headings. http://www.nlm.nih.gov/mesh/,2012. last accessed 29-May-2014.

[3] Stanford network analysis project. http://snap.stanford.edu,2012. last accessed 29-May-2014.

[4] L. Adamic and E. Adar. How to search a social network. SocialNetworks, 27(3):187–203, 2005.

[5] L. Baker, T. H. Wagner, S. Singer, and M. Bundorf. Use of theinternet and e-mail for health care information: Results from anational survey. JAMA: The Journal of the American MedicalAssociation, 289(18):2400–2406, 2003.

[6] O. Bodenreider et al. Biomedical ontologies in action: Role inknowledge management, data integration and decision support.Yearbook Medical Informatics, 47(Suppl 1):67–79, 2008.

[7] J. Evermann and J. Fang. Evaluating ontologies: Towards acognitive measure of quality. Information Systems, 35(4):391– 403, 2010.

[8] D. F. Gleich, P. G. Constantine, A. D. Flaxman, and A. Gu-nawardana. Tracking the random surfer: empirically mea-sured teleportation parameters in PageRank. In M. Rappa andP. Jones, editors, Proceedings of the 19th International Con-ference on World Wide Web, WWW ’10, pages 381–390, NewYork, NY, USA, 2010. ACM.

[9] D. Helic. Analyzing user click paths in a wikipedia navigationgame. In M. Baranovic, M. Golfarelli, B. Vrdoljak, and R. San-dri, editors, Proceedings of the 35th International Conventionon Information and Communication Technology, Electronicsand Microelectronics, MIPRO ’12’, page 374–379, New York,NY, USA, 2012. IEEE.

[10] D. Helic, C. Körner, M. Granitzer, M. Strohmaier, and C. Trat-tner. Navigational efficiency of broad vs. narrow folksonomies.In E. Munson, editor, Proceedings of the 23rd ACM Confer-ence on Hypertext and Social Media, HT ’12, pages 63–72,New York, NY, USA, 2012. ACM.

[11] D. Helic and M. Strohmaier. Building directories for social tag-ging systems. In B. Berendt, A. de Vries, W. Fan, C. Macdon-ald, I. Ounis, and I. Ruthven, editors, Proceedings of the 20thACM International Conference on Information and Knowl-edge Management, CIKM ’11, pages 525–534, New York, NY,USA, 2011. ACM.

[12] D. Helic, M. Strohmaier, M. Granitzer, and R. Scherer. Mod-els of human navigation in information networks based on de-centralized search. In G. Stumme, editor, Proceedings of the24th ACM Conference on Hypertext and Social Media, HT ’13,pages 89–98, New York, NY, USA, 2013. ACM.

[13] D. Helic, M. Strohmaier, C. Trattner, M. Muhr, and K. Ler-man. Pragmatic evaluation of folksonomies. In S. Sadagopan,K. Ramamritham, A. Kumar, and M. P. Ravindra, editors, Pro-


ceedings of the 20th International Conference on World WideWeb, WWW ’11, pages 417–426, New York, NY, USA, 2011.ACM.

[14] J. M. Kleinberg. The small-world phenomenon: an algorithmperspective. In F. Yao and E. Luks, editors, Proceedings of the32nd Annual ACM Symposium on Theory of Computing, STOC’00, pages 163–170, New York, NY, USA, 2000. ACM.

[15] J. M. Kleinberg. Small-world phenomena and the dynamics ofinformation. In T. G. Dietterich, S. Becker, and Z. Ghahramani,editors, Proceedings of the 20th Conference on Neural Infor-mation Processing Systems, NIPS’12, pages 431–438, Cam-bridge, MA, 2001. MIT Press.

[16] J. M. Kleinberg. Complex networks and decentralizedsearch algorithms. In M. Sanz-Solé, J. Soria, J. L. Varona,and J. Verdera, editors, Proceedings of the InternationalCongress of Mathematicians, ICM’06, pages 1019–1044,Zurich, Switzerland, 2006. European Mathematical Society.

[17] D. Krioukov, F. Papadopoulos, M. Kitsak, A. Vahdat, andM. Boguñá. Hyperbolic Geometry of Complex Networks.Physical Review E, 82(036106), Oct 2010.

[18] Z. Li, X. Zhao, D. Huang, and J. Huang. An improved networkbroadcasting method based on gnutella network. In M. Li,X.-H. Sun, Q. Deng, and J. Ni, editors, Grid and Coopera-tive Computing, Second International Workshop, GCC 2003,Shanghai, China, December 7-10, 2003, Revised Papers, PartII, volume 3033 of Lecture Notes in Computer Science, pages404–407. Springer, Heidelberg, Germany, 2003.

[19] G. Marchionini. Exploratory search: From finding to under-standing. Communications of the ACM, 49(4):41–46, Apr.2006.

[20] G. Miao, S. Tao, W. Cheng, R. Moulic, L. E. Moser, D. Lo,and X. Yan. Understanding task-driven information flow incollaborative networks. In T. Yu, editor, Proceedings of the21st International Conference on World Wide Web, WWW ’12,pages 849–858, New York, NY, USA, 2012. ACM.

[21] S. Milgram. The small world problem. Psychology Today,1(1):61–67, 1967.

[22] M. E. J. Newman. The structure and function of complex net-works. SIAM Review, 45(2):167–256, 2003.

[23] N. F. Noy and T. Tudorache. Collaborative ontology develop-ment on the (semantic) web. In D. Sleeman and M. Musen,editors, Proceedings of the AAAI Spring Symposium on Symbi-otic Relationships between Semantic Web and Knowledge En-gineering, pages 63–68, Menlo Park, CA, 2008. AAAI.

[24] M. Papazoglou and J. Hoppenbrouwers. Knowledge naviga-tion in networked digital libraries. In D. Fensel and R. Studer,editors, Knowledge Acquisition, Modeling and Management,11th European Workshop, EKAW ’99, Dagstuhl Castle, Ger-many, May 26-29, 1999, Proceedings, volume 1621, pages 13–

32. Springer, 1999.[25] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel,

B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss,V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau,M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Ma-chine Learning in Python . Journal of Machine Learning Re-search, 12:2825–2830, 2011.

[26] P. Pirolli. Information Foraging Theory: Adaptive Interactionwith Information. Oxford University Press, 2007.

[27] C. Price and K. Spackman. Snomed clinical terms. BJHC&IM-British Journal of Healthcare Computing & Information Man-agement, 17(3):27–31, 2000.

[28] M. Rajapakse, R. Kanagasabai, W. T. Ang, A. Veeramani, M. J.Schreiber, and C. J. Baker. Ontology-centric integration andnavigation of the dengue literature. Journal of Biomedical In-formatics, 41(5):806–815, 2008.

[29] M. Strohmaier, D. Helic, D. Benz, C. Körner, and R. Kern.Evaluation of folksonomy induction algorithms. ACM Trans-actions on Intelligent Systems and Technology, 3(4):74:1–74:22, Sept. 2012.

[30] C. Trattner, P. Singer, D. Helic, and M. Strohmaier. Explor-ing the differences and similarities between hierarchical de-centralized search and human navigation in information net-works. In S. Lindstaedt and M. Granitzer, editors, Proceedingsof the 12th International Conference on Knowledge Manage-ment and Knowledge Technologies, i-KNOW ’12, pages 14:1–14:8, New York, NY, USA, 2012. ACM.

[31] J. Travers and S. Milgram. An experimental study of the smallworld problem. Sociometry, 32:425–443, 1969.

[32] J. R. Villela Dantas and P. P. Muniz Farias. Conceptual navi-gation in knowledge management environments using navcon.Information processing & management, 46(4):413–425, 2010.

[33] D. J. Watts and S. H. Strogatz. Collective dynamics of small-world networks. Nature, 393(6684):440–442, June 1998.

[34] R. West and J. Leskovec. Human wayfinding in informationnetworks. In T. Yu, editor, Proceedings of the 21st Interna-tional Conference on World Wide Web, WWW ’12, pages 619–628, New York, NY, USA, 2012. ACM.

[35] R. West, J. Pineau, and D. Precup. Wikispeedia: An onlinegame for inferring semantic distances between concepts. InC. Boutilier, editor, Proceedings of the 21st International JontConference on Artifical Intelligence, IJCAI’09, pages 1598–1603, San Francisco, CA, USA, 2009. Morgan Kaufmann Pub-lishers Inc.

[36] P. L. Whetzel, N. F. Noy, N. H. Shah, P. R. Alexander, C. Nyu-las, T. Tudorache, and M. A. Musen. Bioportal: Enhancedfunctionality via new web services from the national centerfor biomedical ontology to access and use ontologies in soft-ware applications. Nucleic Acids Research, 39(Web-Server-Issue):541–545, 2011.

Documents

Using ontologies to model human navigation behavior in information