Balancing the Trade-Offs Between Diversity and Precision ... · diversiﬁcation and precision among the image search results. Using this knowledge, we have developed a simple method

Balancing the Trade-Offs Between Diversity andPrecision for Web Image Search Using

Concept-Based Query ExpansionEnamul Hoque, Orland Hoeber, and Minglun Gong

Department of Computer ScienceMemorial University of Newfoundland

St. John’s, NL, Canada A1B 3X5Email: {enamulp, hoeber, gong}@mun.ca

Abstract— Even though Web image search queries are oftenambiguous, traditional search engines retrieve and presentresults solely based on relevance ranking, where only themost common and popular interpretations of the queryare considered. Rather than assuming that all users areinterested in the most common meaning of the query, amore sensible approach may be to produce a diversified setof images that cover the various aspects of the query, underthe expectation that at least one of these interpretationswill match the searcher’s needs. However, such a promotionof diversity in the search results has the side-effect ofdecreasing the precision of the most common sense. Inthis paper, we evaluate these competing factors in thecontext of a method for explicitly diversifying image searchresults via concept-based query expansion using Wikipedia.Experiments with controlling the degree of diversificationillustrate this trade-off between diversity and precision forboth ambiguous and more specific queries. As a resultof these experiments, an automatic method for tuning thediversification parameter is proposed based on the degreeof ambiguity of the original query.

I. INTRODUCTION

The primary method for performing image retrieval onthe Web is based on document search techniques [1].Images are indexed based on the text that is related totheir use on the Web (keywords, tags, and/or associateddescriptions). User-supplied queries are matched to thistext to produce a set of images, which are ranked based onthe relevance of their associated textual information to thequery. This approach can work well when the contents ofthe images are concisely and accurately described withintheir source Web pages, and when the searchers’ needsare clearly specified.

Recent studies on user behaviour with respect to imagesearch have found that queries are often very short andambiguous [2]. This ambiguity comes from the difficultythat searchers experience in finding the words to describean idea or image they have in their mind. From the

This paper is an extended version of “Evaluating the Trade-OffsBetween Diversity and Precision for Web Image Search Using Concept-Based Query Expansion,” by E. Hoque, O. Hoeber, and M. Gong,which appeared in the Proceedings of the IEEE/WIC/ACM InternationalConference on Web Intelligence Workshops (International Workshop onIntelligent Web Interaction), Lyon, France, August 2011.

perspective of image retrieval, the difficulty with ambigu-ous queries is that they can be open to many differentinterpretations. It is possible that different searchers mayenter the same query, but their intentions and needs mayvary significantly from one another. In situations suchas this, the matching algorithms used by image searchengines promote the interpretation that is most commonand popular. However, a more sensible approach maybe to produce a diversified set of images that cover thevarious aspects of the query, under the expectation thatat least one of the interpretations matches the searcher’sintent. Providing searchers with an overview of the imagesand allowing them to zoom-in and focus on a particularinterpretation [3] may improve the effectiveness and effi-ciency of the image search process.

Moving beyond traditional relevance ranking, diversi-fication approaches aim to improve the coverage of thesearch results set with respect to the different senses ofthe original query. A common method for diversificationis query expansion, whereby additional terms are addedto the query to generate a collection of new queries that,when taken together, are more broad than the original[4]. However, in doing so there is a danger in broadeningthe query too much, resulting in a potentially significantdecrease in precision. That is, the more broad and diversethe search results are, the less chance that a particularsearch result will be relevant to the searcher’s informationneed. As such, this trade-off between diversity and preci-sion must be studied in order to understand the situationswhere more or less diversification is beneficial.

Maintaining a balance between diversity and precisionrequires an automatic modelling of the searcher’s queryto determine an appropriate degree of diversification topromote. In many cases, image search queries are inher-ently ambiguous. For example, “Washington” might beinterpreted as “Washington (state)”, “Washington D.C.”,or “George Washington”. Even within a more specificquery, searchers might have an interest in seeing imagesthat are related to but not explicitly identified in the query.For example, if a searcher submits a query such as “HongKong”, they may wish to see some representative imagesof different landmarks in the Hong Kong area. In othercases, a query might be very specific, and the scope for

26 JOURNAL OF EMERGING TECHNOLOGIES IN WEB INTELLIGENCE, VOL. 4, NO. 1, FEBRUARY 2012

© 2012 ACADEMY PUBLISHERdoi:10.4304/jetwi.4.1.26-34

broadening the query is limited. For example, queries fora particular landmark within a specific setting like “EiffelTower Bastille Day” may leave little room for diversifyingthe search results.

In previous research, we presented a method for diver-sifying image search results using concept-based queryexpansion, which is based on information derived fromWikipedia [3]. In this work, we have modified our ap-proach such that the degree of diversification can becontrolled, allowing us to evaluate the trade-offs betweendiversification and precision among the image searchresults. Using this knowledge, we have developed asimple method for automatically determining the degreeof diversification based on the level of ambiguity of thequery, resulting in a balance between diversification andprecision.

The remainder of this paper is organized as follows.Section II provides an overview of the existing methodsfor search results diversification. Section III outlines ourmethod for diversifying image search results, with aspecific focus on a parameter that we use to controlthe degree of diversification. Section IV describes anevaluation that explores how the diversification parametercan be used to find a balance between diversity andprecision within the search results, and proposes a formulafor automatically determining this parameter. Section Vprovides a discussion on the outcomes of this study andthe implications for diversifying image search results.The paper concludes with a summary of the researchcontributions in Section VI.

II. RELATED WORK

The problem of enhancing diversity in search resultshas been recognized as an important topic in the researchcommunity. Diversification can be stated as an optimiza-tion problem, where the objective is to maximize theprobability of finding a relevant document/image amongthe top selected results, while minimizing its redundancywith respect to different aspects of the query. The generalproblem is an instance of maximum coverage problem,which is NP-hard [5].

In the literature of document search techniques, most ofthe previous approaches to search results diversificationcan be categorized as either implicit or explicit [6].Implicit search results diversification approaches performdirect comparison between the retrieved documents, underthe assumption that similar documents will cover similaraspects. In an early work, Carbonell et al. [7] proposeda greedy approximation based ranking method calledMaximal Marginal Relevance (MMR), which attempted tomaximize the relevance of a search result while minimiz-ing its similarity to higher ranked documents. In anotherimplementation of an MMR-based method, Zhai et al.[8] proposed a subtopic search method using statisticallanguage models, which aimed to return results that covermore subtopics.

An alternative approach to diversification is to explic-itly utilize different aspects associated with a query by

directly modelling these aspects. For instance, Agrawalet al. [5] investigated the diversification problem basedon the assumption that a taxonomy is available forboth queries and documents. In their work, documentsretrieved for a query were considered similar if they wereconfidently classified into one or more common categoriescovered by the query. They applied a greedy algorithmthat started with an empty list of results and selectedthe next result with the highest marginal utility until kresults were selected. Here, the marginal utility measuredthe probability that the result satisfied a category thecurrent result set did not yet cover. In a related approach,Radlinski and Dumais [9] proposed to filter the resultsretrieved for a given query so as to limit the number ofthose satisfying the same aspect of the query, representedas different query reformulations obtained from a largequery log from a commercial search engine. Recently,Santos et al. [10] introduced a probabilistic framework forsearch results diversification, which explicitly representedquery aspects as “sub-queries”. They defined diversitybased on the estimated relevance of documents to multiplesub-queries and the relative importance of each sub-query.

The diversity problem is more challenging in Webimage search for a number of reasons. First of all, imagesearch may involve not only semantic diversity but alsovisual diversity of the search results [11]–[14]. Althoughthere may be some benefit to promoting both types ofdiversity, the focus of our research is on the semanticside. Secondly, in the document centric retrieval methods,the documents themselves provide some useful ways toallow direct comparisons and similarity calculations whenpromoting diversification in the search results. But inimage retrieval, the limited amount of textual metadataassociated with the retrieved images are not reliableenough nor sufficient to allow for the computation ofsimilarities between their associated images.

A somewhat common approach to dealing with thediversification problem for image search results is to usesemantic and/or visual clustering [11]–[13], [15], [16].Although these approaches differ in the information usedand the methods employed for clustering, the end result isa grouping of images that represent the different aspectsof the image collection. There are, however, a numberof challenges associated with clustering approaches, in-cluding determining a suitable similarity measure, theefficiency of clustering algorithms, determining an ap-propriate number of clusters to use, and deciding howto order or organize the clusters.

Re-ranking methods have also been proposed with anaim to enhance topic coverage in search results. Song etal. [17] addressed the diversity problem using a re-rankingmethod based on topic richness analysis. The goal here isto enhance topic coverage in the image search results,while maintaining acceptable retrieval performance. Atopic richness score was computed by analyzing thedegree of mutual topic coverage between an image pairbased on the assumption that images are annotated byseveral words.

JOURNAL OF EMERGING TECHNOLOGIES IN WEB INTELLIGENCE, VOL. 4, NO. 1, FEBRUARY 2012 27

© 2012 ACADEMY PUBLISHER

Some have also begun to study the trade-off betweenprecision and diversification within the context of imagesearch. For example, van Zwol et al. proposed a methodfor optimizing this trade-off by estimating a query modelfrom the distribution of tags that favour the dominantsense of an image search query [18].

A promising direction for enhancing the diversity ofimage search results is the use of query expansion meth-ods based on concepts that are related to the query. Unlikethe general domain of document-centric information re-trieval, query expansion has been studied in only a fewresearch works in Web image retrieval. Those that haveexplored such techniques have shown them to be promis-ing [3], [19]. However, one of the challenges associatedwith query expansion is to find an appropriate sourceof knowledge required for the expansion process. It hasbeen noted that many image search queries are associatedwith conceptual domains that include proper nouns (i.e.,people’s names and locations) [2], [20]. As such, findinga suitable knowledge base that has sufficient coverage ofa realistic conceptual domain is very important first stepin this approach to query expansion. Wikipedia is a goodcandidate for such a knowledge base since it includesa large number of articles describing people, places,landmarks, animals, and plants. The challenge in usingWikipedia is to design efficient and effective algorithmsthat can process the semi-structured knowledge to derivemeaningful terms for use in the query expansion process.

III. IMAGE SEARCH RESULTS DIVERSIFICATION

For the diversification framework used in this research,image search results are explicitly diversified based onconcept-based query expansion. For the short and am-biguous queries that are common in image search, queryexpansion attempts to capture the various aspects ofthe query. We model the original query by discoveringdifferent possible senses, and for each sense a numberof concepts pertaining to the query are discovered fromwithin Wikipedia. These concepts are ranked accordingto the semantic relatedness to the original query, andonly the top-N most related concepts are used within thequery expansion process to retrieve a range of images thatprovides a broad view of what is available.

Within this process, the value of N is an explicitindicator of the degree of diversification, which we callthe diversification parameter. With a smaller value ofN , fewer concepts will be used, and the search resultswill remain more focused. If we increase N , then moreconcepts will be used for query expansion, and in turnthe search results will be more diversified covering moreaspects associated with the query. The fundamental trade-off between diversification and precision is based on thefact that as we increase N , there is a higher chancethat a concept will be selected for the query refinementprocess that is not relevant to the searcher’s informationneeds, resulting in the associated irrelevant images beingincluded in the search results. As such, our goal is to fulfillthe diversification objective by setting N sufficiently high

to capture the broad range of concepts associated withthe query, but not so high as to have a significant adverseaffect on the average precision across all of the senses ofthe query.

In this precision-diversity trade-off context, we canstate our diversification objective as follows: Given aquery Q, perform query expansion based on N conceptsto retrieve a results set R, so that it will maximize thevalue of N , and at the same time maximize the precisionP with respect to each of the possible senses of the queryover the results set R. Note that an inherent feature ofsuch diversification is that it will increase the precision ofthe search results with respect to the less common sensesof the query, at the expense of the most common sense.

To achieve this objective in our diversification method,the possible senses of the query, the number of con-cepts to be selected for each sense, and the number ofimages to be retrieved for each concept are determinedautomatically based on an analysis of the original queryand the candidate concepts extracted from Wikipedia. Thepurpose here is to distribute the concepts and the numberof images retrieved for each concept unevenly, so thatwe can alleviate the problem of harming precision due tothe images retrieved for irrelevant concepts that are onlyloosely related to the senses of the original query.

The process of diversifying the image search resultsusing concept-based query expansion follows three steps:extracting concepts from Wikipedia; ranking the extractedconcepts to select the top-N related concepts; and retriev-ing images based on the expanded queries derived fromthese concepts. The details for each of these steps areexplained in the remainder of this section.

A. Concept Extraction Using Wikipedia

In order to use Wikipedia as the source knowledgebase in this work, a dump of the Wikipedia collectionwas obtained and preprocessed to support the type ofknowledge extraction required for our purposes. Matchinga user-supplied query Q to this knowledge base is simplya matter of selecting the best matching article (referredas the home article) using Wikipedia’s search feature. Inthe case where the query is ambiguous and Wikipediasuggests multiple senses of interpretations, the ones withhigher commonness values are used as the home articles.Here the commonness value of an article is calculatedbased on how often it is linked by other articles.

In analyzing Wikipedia, we observed that the in-linkarticles (ones having links to a home article) and out-linkarticles (ones to which a home article links) often providemeaningful information that is closely related to theconcept of the home article, and hence the user-specifiedquery. Therefore, these linked articles are located and theirtitles are extracted as candidates for related concepts.

We also found that the captions surrounding the imagespresent within a given article can often provide a valuableperspective on the visual features associated with the ar-ticle’s concept. Due to the importance of this informationfor the purposes of image retrieval, it is important to



ensure that all relevant concepts associated with the imagecaptions are extracted. We use Wikifier [21] to augmentthe captions with links to relevant Wikipedia articles thatmay have been missed by the author of the article, anduse these links to extract their associated concepts.

The end result of this process is the selection of a setof home articles {hs|1 s q} (for q senses of givenquery Q), along with a list of all the candidate articlesChs for each home article hs that originate from the in-link articles, out-link articles, and the image captions.These concepts provide the basis for the automatic queryexpansion process.

B. Ranking the Extracted Concepts

Due to the rich and interconnected nature of Wikipedia,the number of concepts obtained in the process describedabove may become very large. Thus, a filtering step isnecessary to ensure the quality of the concepts that areextracted. Here, our objective is to select the top-N con-cepts from among all the candidate articles. Consideringthe difference in the importance of each of the senses,we distribute these top-N concepts among the candidateconcepts Chs of each home article hs. As such, thenumber of concepts Nhs that are to be selected for aparticular home article hs is determined as follows:

Nhs =|Chs |⇥NPq

j=1 |Chj |

Note that the sum of all Nhs values equals N .To select these Nhs concepts for each home article

hs, it is necessary to rank the candidate concepts Chs

based on their relevance to their associated home article.Our approach to this problem is to measure the semanticrelatedness between the home article and each of thecandidate concepts. A number of different methods havebeen devised to use Wikipedia for this purpose, includingWikiRelate! [22], Explicit Semantic Analysis (ESA) [23],and Wikipedia Link-based Measure (WLM) [24]. Giventhe computationally efficiency and accuracy of WLM, weuse this approach in our work.

For each of the candidate articles ci 2 Chs extractedfrom the home article, WLM is applied between thehome article hs and the candidate articles. WLM takesadvantage of the hyperlink structure of the associatedarticles to find out how much they share in common.In order to give preference to the concepts that havebeen extracted from the image captions within the homearticle, we use a re-weighting function to determine therelatedness score:

r(ci, hs) = min(WLM(ci, hs)(1 + ↵s), 1)

Since WLM provides a value in the [0,1] range, weensure that the relatedness score remains in this rangewith the min function. The re-weighting factor ↵s isprovided according to the following function:

↵s =

(k ChsNhs

if concept ci originates from a caption0 otherwise

Here, Chs and Nhs are as defined above, and k isa system parameter that controls the importance of theconcepts derived from the captions. In our prototypeimplementation k = 0.01, which results in a 10 -20% increase in the score for the concepts derived fromthe captions, with proportionally more importance beinggiven when there are more concepts extracted from thehome article.

The outcome of this process is that the top-N conceptsare selected from among the candidate articles, suchthat the ones that came from the image captions aregiven preference over those from in-link and out-linkarticles. These concepts are used as the source for thequery expansion process. The value of N serves here asan explicit diversification parameter. How it affects thediversification and precision of search results are discussin Section IV.

C. Concept-Based Query Expansion and Image Retrieval

In order to ensure that the expanded queries remainfocused on the topic of the query itself, the top-N relatedconcepts {cr|0 r N} are prepended with theirassociated home article hr, resulting in queries of theform < hr, cr >. We define c0 to be null and h0 to thethe original query Q, producing the original query plusN expanded queries.

Given that individual expanded queries have differingdegrees of relevance to the original query, we dynami-cally determine how many images to retrieve for eachexpanded query based on their relatedness score to thehome articles. This way we can ensure that more imagesare retrieved for concepts that are most similar to the senseassociated with their home article, even when the originalquery has multiple meanings. This is done to minimizethe number of images retrieved for concepts that are onlyloosely associated with the sense of the query.

The number of images to retrieve for each expandedquery is given by the following formula:

Ir =r(cr, hs)⇥ ItPNk=0 r(ck, hs)

Here, r is the same function used to generate therelatedness score in the concept ranking process, and Itis the total number of images to be retrieved by all of thequeries. We set It = 60 for the purposes of performingthe evaluation within this paper, but it can be set to anyreasonable number of images. Since the null expandedquery (c0) is the original query, we define r(c0, hs) = 1in the above calculation. All of the queries are sent to theGoogle AJAX Search, and the desired number of imagesare retrieved. Duplicate images are deleted based on theURL of the source image (as provided by the underlyingsearch engine).

IV. EVALUATION

The goal of this approach is to automatically diversifythe images retrieved for a given query, using Wikipediaas the source for a query expansion process. However, it



is unclear to what degree such diversification should bepromoted during the search process. In this evaluation ofthe approach, our goal is to study the inherent trade-offbetween precision and diversity in detail. In particular, fora set of queries, we explore how the precision changesas diversity in the image search results is promoted.Using this information, we propose a simple approachto automatically determining the degree of diversificationbased on features of the user-supplied query.

A. Experimental Setup

For these experiments, we chose 12 query topics, splitbetween those we deemed to be highly ambiguous (hav-ing four senses), moderately ambiguous (three senses),slightly ambiguous (two senses), and non-ambiguous (onesense). This distribution of different degrees of ambiguityallowed us to examine the effect of the experimentalcondition (i.e., the varying of the degree of diversification)in the context of ambiguity. For each of these differentdegrees of ambiguity we selected two queries, except forthe moderate ambiguity, for which we selected six queries.The moderate degree of ambiguity was examined morecarefully since it represents the most common case ofambiguity.

To evaluate the effect of diversification on precision,we retrieved the top 60 search results from Google ImageSearch using our concept-based query expansion methodwith ten different values of N , ranging from 0 to 40(N = 0, 2, 4, 6, 8, 10, 15, 20, 30, 40). Here, N = 0 impliesthat no query expansion has occurred (i.e., the searchresults are not diversified, and are simply the resultsprovided by the underlying image search engine). At theother extreme, N = 40 causes the system to return ahighly diversified set of image search results from 40different associated concepts chosen in the query expan-sion procedure. Data was collected more frequently in thelow end of this range in order to more closely observethe effect of a low degree of diversification. Preliminaryexperiments illustrated that the effect of the degree ofdiversification at the higher range became rather stable[25].

For each of the different senses of the query, asses-sors were asked to judge the relevance of each image.This assessment of relevance provided the ground truthinformation in the calculation of the precision scores (theratio of relevant images to the total number of imagesretrieved). Since there were ten trails (i.e., ten differentvalues of N ) and 60 images retrieve with each trial, thisresulted in the evaluation of a total of 600 images for eachtest query.

B. Results

In these experiments we measured the precision foreach of the test queries as the diversification parameterN was varied from 0 to 40. Our hypothesis was that asN increased, the distribution of the senses would becomemore balanced across all of the meanings of the query.

This would result in a reduction in the precision for themost common senses of the query, and an increase in theprecision for the less common senses. This feature canbe readily identified in the graphs in Figures 1, 2, and 3.To further understand this effect, we plotted the averageprecision (the red lines with the triangle markers) andthe total precision (the dark red lines with the x marker)across all of the senses.

Figures 1 and 2 show the results from the highly andmoderately ambiguous queries. The general effect that canbe seen here is that the precision for the most commonsense (i.e., the blue line in each graph) was automaticallyreduced as a result of the diversification. In most casesthis occurred in a more or less smooth fashion even withvery low values of N . At the same time, the precision forall other senses increased. In some cases, it was necessaryfor the value of N to be set higher than six for imagesfrom some of the less common senses of the query to berepresented in the search results.

Although there was, in some cases, a minor reductionin the total precision for very low values of N , this wasoften accompanied by a subsequent increase in the totalprecision as more diversification occurred. This effect isa result of the method by which the number of imagesretrieved for each expanded query is dynamically deter-mined. With very few expanded queries, more images areretrieved for each (which may result in the inclusion ofsome less relevant images deeper in the search resultslist). As N is increased and more expanded queriesare generated, the images that are retrieved have higherrankings with respect to their source query.

For all of these queries, the precision of the imagesover each of the senses of the query did not changesignificantly once the value of N was set beyond six toten, depending on the specific query. Furthermore, if thevalue of N was set too large (i.e., 30 or 40), the averageand total precision started to decrease, indicating thatsome non-relevant concepts and their associated imageswere being included in the search results set due to over-diversification.

Our expectation when designing these experiments wasthat for queries that have a high degree of ambiguity, itwould be necessary to set the diversification parameterrather high in order to capture enough information on allof the different senses. However, it is clear that even witha diversification parameter set at N = 10, the desiredeffect appears.

In some cases, the most common sense remains themost common regardless of the level of diversification(e.g., Figure 1a and b, and 2d, and e). In the othercases, senses that were less common in the original searchresults become dominant. The reason for this changeis that there may be disagreement between what theunderlying search engine assumes is the desired informa-tion need (i.e., the most common sense of the query),and the amount of information that can be extractedfrom Wikipedia on the other senses and their associatedconcepts.



(a) Tiger (b) Washington

Figure 1. The effect of varying the degree of diversification (N) on precision (P) for highly ambiguous queries with four different senses.

(a) Beetle (b) Fuji

(c) Jaguar (d) Jim Clark

(e) Sharks (f) Tivoli

Figure 2. The effect of varying the degree of diversification (N) on precision (P) for moderately ambiguous queries with three different senses.

Figure 3 shows the results from the slightly ambiguousqueries. As with the highly ambiguous queries, evenwith a low degree of diversification (from two to six),the outcome of the diversification is a balancing of theprecision between the two senses of the query. In addition,

the effect of a dropping then increasing total precisionover low values of N is present, as is the dropping averageand total precision when N is set too large. However, thelater effect occurs with lower values of N than with thehighly or moderately ambiguous queries (15 - 20 for the



(a) Apple (b) Beckham

Figure 3. The effect of varying the degree of diversification (N) on precision (P) for slightly ambiguous queries with two different senses.

(a) Eiffel Tower (b) Washington Monument

Figure 4. The effect of varying the degree of diversification (N) on precision (P) for non-ambiguous queries with only one sense.

slightly ambiguous queries).For the queries where there was only one sense (Figure

4), it is clear that diversifying the search results can veryquickly have a negative effect on the precision. However,for small values of N (e.g, 2-4) this effect is negligible.This indicates that even for very specific queries, a smalldegree of diversification might be tolerable and perhapseven beneficial as highly related images are drawn intothe search results set.

As a result of this analysis, we conclude that diver-sifying the image search results can be very useful foraddressing the situation where an ambiguous query hasmultiple senses. Rather than relying on the search engineto choose the most common sense, we can diversify theimage search results and let the user focus on thoseimages that match their needs. The more senses thatcan be inferred from a query, the more diversification isnecessary to sufficiently balance all of these senses inthe search results. However, when there are few differentsenses, the degree of diversification should be limited toavoid including irrelevant concepts and their associatedimages in the search results.

C. Automatically Determining the Degree of Diversifica-tion

One of our goals in this research is to automaticallydetermine the degree of diversification needed to enhancethe searcher’s ability to find relevant documents. Asillustrated in the experiments, the degree of ambiguity of

the query has an impact on the value of diversification.That is, a highly ambiguous query can benefit from morediversification, whereas a very specific query may beharmed by diversifying the search results too much.

Based on this, we propose a linear scaling of the degreeof diversification based on the number of senses of thequery (as determined by Wikipedia). One such automaticfunction for determining the how aggressively to diversifythe search results is N = a ⇥ q where q is the numberof senses for query Q (following the terminology fromSection III). Based on the experiments reported in thispaper, setting a = 3 will produce reasonable results. Thatis, if a query has one sense, the diversification parameterwill be set to 3, performing a relatively low degree ofdiversification that does not harm the precision of theone sense of the query. For the slightly, moderately, andhighly ambiguous queries, this results in the diversifica-tion parameter being set to 6, 9, and 12, respectively. Byinspecting the graphs in Figures 1 - 4, we can see thatthis is a near-optimal value for N .

This simple approach to determining N could benefitfrom further research in fine-tuning this formula and itsparameters for automatically determining the value of Nbased on some measure of query ambiguity.

V. DISCUSSION

Without diversification, the most common sense of anambiguous query can dominate the image search results.Images that match other senses of the query may not



be very common or even represented at all within thetop search results. This is a preferential outcome if thesearcher’s information needs match the underlying searchengine’s interpretation of the query. However, when thisis not the case, it can be very difficult for the searcher tofind images that match their needs. Their only recourseis to attempt to reformulate the query into somethingmore specific. However, studies on Web image searchbehaviour indicate that this may be a rather difficult taskfor searchers to perform.

The key benefit of performing image search diversifi-cation is that instead of assuming a single interpretationof the query, we retrieve images from different sensesdiscovered using Wikipedia, providing a more balanceddistribution of the senses within the set of search results.As a result, this diversified set of search results may besuitable to a wider range of users with a wider range ofinformation needs.

A common problem with query expansion in general,and search result diversification in particular, is that itcan reduce the precision by including documents (images)that might not be relevant to all searchers. This remainsthe case for our work on image search diversification. Inparticular, we have shown how the precision for the mostcommon sense will invariably be reduced. This is becauseimages from the less common senses are being included inthe search results as a result of the diversification process.However, even amid this reduction in the precision of themost common sense, and the increase in the precisionof the less common senses, the average precision is notchaining noticeably when the degree of diversification isnot set unreasonably large. On the contrary, in some casesthe total precision actually increases, when calculated overall senses of the query. This indicates that the proposedapproach is effective in keeping the expanded queriesfocused on the intended senses of the query, even whenthere are multiple such senses.

Because the process of diversification will commonlyresult in a mix of relevant and irrelevant images for agiven interpretation of an ambiguous query, it is importantthat an interface to the image search results be used thatmakes it easy for the searcher to ignore the senses ofthe query that are not relevant to their needs, and focuson those that are. We have developed such an interfacein previous research that organizes the search resultsaccording to both visual and conceptual similarities [26],allowing the searcher to filter the search results both vi-sually through a pan-and-zoom operation [27], and basedon a hierarchy of the concepts that produced the expandedqueries [3]. Using this visual interface to explore thesearch results, the precision for a particular sense of thequery may be improved as irrelevant images are movedout of the field of view or are filtered based concepts thatare uninteresting to the searcher. An evaluation of thisinterface from the perspective of improving the precisionof the search results through these interactive operationshas shown not only the benefit of the interactive features,but also the benefit of organizing the images based on

both visual and conceptual features [28].It is also worth noting that the benefit that this ap-

proach to image search diversification can provide de-pends greatly on the completeness and interconnectednature of information in Wikipedia. If there is a sense of agiven query that is not well represented in Wikipedia, thenit will not be well represented within the diversified imagesearch results set. As such, as Wikipedia continues togrow and be enhanced, it will become a better and bettertool for providing knowledge for information retrievalresearch such as what has been discussed in this paper.

VI. CONCLUSIONS

In this paper, we present a novel approach for explicitlydiversifying image search results using concept-basedquery expansion. This approach is based on our previousresearch, but has been modified such that the degree ofdiversification can be controlled through the diversifica-tion parameter N . Using this system, we have evaluatedthe trade-off between diversification and precision, usinga set of test queries of varying ambiguity.

From these experiments, we can see that the degreeto which diversification needs to be promoted in orderto provide a balance across all of the senses of a querydepends on the ambiguity of the query itself. That is, ahighly ambiguous query can benefit from a high degree ofdiversification more so than a very specific query. Basedon this, we propose a simple automatic calculation of thediversification parameter based on the level of ambiguityof the query that balances the increased diversificationagainst the decrease in precision.

Instead of only satisfying the information needs for themost common interpretation of the query, our methodprovides a more balanced view of the different sensesof the query, without a significant negative impact onthe average precision across all senses. Furthermore, ourmethod of diversification provides an effective meansfor ensuring that the expanded queries that produce thediversified search results remain focused on at least onesense of the query. Coupled with a visual interface fororganizing the image search results based on their visualand conceptual similarity, and interaction methods thatsupport dynamic filtering, this approach to search resultsdiversification can be a very powerful tool for enhancingthe image search experience.

ACKNOWLEDGMENTS

This research was supported by Research DevelopmentCorporation (RDC) IgniteR&D and Natural Sciences andEngineering Research Council of Canada (NSERC) Dis-covery Grants held by the last two authors.

REFERENCES

[1] M. L. Kherfi, D. Ziou, and A. Bernardi, “Image retrievalfrom the world wide web: Issues, techniques, and systems,”ACM Computing Surveys, vol. 36, no. 1, pp. 35–67, 2004.



[2] P. Andre, E. Cutrell, D. S. Tan, and G. Smith, “Designingnovel image search interfaces by understanding uniquecharacteristics and usage,” in Proceedings of the IFIPConference on Human-Computer Interaction, 2009, pp.340–353.

[3] E. Hoque, G. Strong, O. Hoeber, and M. Gong, “Concep-tual query expansion and visual search results explorationfor Web image retrieval,” in In Proceedings of the AtlanticWeb Intelligence Conference, 2011, pp. 73–82.

[4] C. J. van Rijsbergen, Information Retrieval. Butterworths,1979.

[5] R. Agrawal, S. Gollapudi, A. Halverson, and S. Ieong,“Diversifying search results,” in Proceedings of the ACMInternational Conference on Web Search and Data Mining,2009, pp. 5–14.

[6] R. Santos, J. Peng, C. Macdonald, and I. Ounis, “Ex-plicit search result diversification through sub-queries,” inProceedings of the European Conference on InformationRetrieval, 2010, pp. 87–99.

[7] J. Carbonell and J. Goldstein, “The use of MMR, diversity-based reranking for reordering documents and producingsummaries,” in Proceedings of the Annual InternationalACM SIGIR Conference on Research and Development inInformation Retrieval, 1998, pp. 335–336.

[8] C. X. Zhai, W. W. Cohen, and J. Lafferty, “Beyondindependent relevance: methods and evaluation metricsfor subtopic retrieval,” in Proceedings of the AnnualInternational ACM SIGIR Conference on Research andDevelopment in Information Retrieval, 2003, pp. 10–17.

[9] F. Radlinski and S. Dumais, “Improving personalized Websearch using result diversification,” in Proceedings of theAnnual International ACM SIGIR Conference on Researchand Development in Information Retrieval, 2006, pp. 691–692.

[10] R. L. Santos, C. Macdonald, and I. Ounis, “Exploitingquery reformulations for Web search result diversification,”in Proceedings of the International Conference on theWorld Wide Web, 2010, pp. 881–890.

[11] D. Cai, X. He, Z. Li, W.-Y. Ma, and J.-R. Wen, “Hierarchi-cal clustering of WWW image search results using visual,textual and link information,” in Proceedings of the ACMInternational Conference on Multimedia, 2004, pp. 952–959.

[12] B. Gao, T.-Y. Liu, T. Qin, X. Zheng, Q.-S. Cheng, and W.-Y. Ma, “Web image clustering by consistent utilization ofvisual features and surrounding texts,” in Proceedings ofthe ACM International Conference on Multimedia, 2005,pp. 112–121.

[13] S. Wang, F. Jing, J. He, Q. Du, and L. Zhang, “IGroup:presenting Web image search results in semantic clusters,”in Proceedings of the SIGCHI Conference on HumanFactors in Computing Systems, 2007, pp. 587–596.

[14] R. H. van Leuken, L. Garcia, X. Olivares, and R. vanZwol, “Visual diversification of image search results,” inProceedings of the International Conference on the WorldWide Web, 2009, pp. 341–350.

[15] P.-A. Moellic, J.-E. Haugeard, and G. Pitel, “Image clus-tering based on a shared nearest neighbors approach fortagged collections,” in Proceedings of the InternationalConference on Content-Based Image and Video Retrieval,2008, pp. 269–278.

[16] H. Ding, J. Liu, and H. Lu, “Hierarchical clustering-basednavigation of image search results,” in Proceedings of theACM International Conference on Multimedia, 2008, pp.741–744.

[17] K. Song, Y. Tian, W. Gao, and T. Huang, “Diversifyingthe image retrieval results,” in Proceedings of the ACMInternational Conference on Multimedia, 2006, pp. 707–710.

[18] R. van Zwol, V. Murdock, L. Garcia Pueyo, andG. Ramirez, “Diversifying image search with user gen-erated content,” in Proceeding of the ACM InternationalConference on Multimedia Information Retrieval, 2008,pp. 67–74.

[19] D. Myoupo, A. Popescu, H. L. Borgne, and P.-A. Moellic,“Multimodal image retrieval over a large database,” inProceedings of the International Conference on Cross-Language Evaluation Forum: Multimedia Experiments,2009.

[20] B. J. Jansen, A. Spink, and J. Pedersen, “An analysisof multimedia searching on AltaVista,” in Proceedings ofthe ACM SIGMM International Workshop on MultimediaInformation Retrieval, 2003, pp. 186–192.

[21] D. Milne and I. H. Witten, “Learning to link withWikipedia,” in Proceedings of the ACM Conference onInformation and Knowledge Management, 2008, pp. 509–518.

[22] M. Strube and S. P. Ponzetto, “WikiRelate! computingsemantic relatedness using Wikipedia,” in Proceedings ofthe AAAI Conference on Artificial Intelligence, 2006, pp.1419–1424.

[23] E. Gabrilovich and S. Markovitch, “Computing semanticrelatedness using Wikipedia-based explicit semantic anal-ysis,” in Proceedings of the International Joint Conferenceon Artificial Intelligence, 2007, pp. 1606–1611.

[24] D. Milne and I. H. Witten, “An effective, low-cost measureof semantic relatedness obtained from Wikipedia links,”in Proceedings of the AAAI Workshop on Wikipedia andArtificial Intelligence, 2008, pp. 25–30.

[25] E. Hoque, O. Hoeber, and M. Gong, “Evaluating thetrade-offs between diversity and precision for Web imagesearch using concept-based query expansion,” in Proceed-ings of the IEEE/WIC/ACM International Conference onWeb Intelligence - Workshops (International Workshop onIntelligent Web Interaction), 2011, pp. 130–133.

[26] G. Strong, E. Hoque, M. Gong, and O. Hoeber, “Organiz-ing and browsing image search results based on conceptualand visual similarities,” in Proceedings of the InternationalSymposium on Visual Computing, 2010, pp. 481–490.

[27] G. Strong, O. Hoeber, and M. Gong, “Visual image brows-ing and exploration (Vibe): User evaluations of imagesearch tasks,” in Proceedings of the International Confer-ence on Active Media Technology, 2010, pp. 424–435.

[28] E. Hoque, O. Hoeber, and M. Gong, “Combining concep-tual query expansion and visual search results explorationfor Web image retrieval,” Journal of Ambient Intelligenceand Humanized Computing, in press.

Enamul Hoque is currently an M.Sc. student in the Departmentof Computer Science at Memorial University of Newfoundland.His research interests include image retrieval, search interfaces,and information visualization.

Orland Hoeber received his Ph.D. from the University ofRegina in 2007, and is currently an Assistant Professor in theDepartment of Computer Science at Memorial University ofNewfoundland. His primary research interests include informa-tion visualization, Web search interfaces, Web search personal-ization, human-computer interaction, and geo-visual analytics.

Minglun Gong received his Ph.D. from the University ofAlberta in 2003, and is currently an Associate Professor inthe Department of Computer Science at Memorial Universityof Newfoundland. His research interests include a variety oftopics in computer graphics, computer vision, image processing,pattern recognition, and optimization techniques.



Documents

Balancing the Trade-Offs Between Diversity and Precision ... · diversiﬁcation and precision among the image search results. Using this knowledge, we have developed a simple method