Tag Embedding Based Personalized Point Of Interest RecommendationSystem

Suraj Agrawala,∗, Dwaipayan Royb, Mandar Mitraa

aIndian Statistical Institute, Kolkata, IndiabGESIS - Leibniz Institute for the Social Sciences, Cologne, Germany

Abstract

Personalized Point of Interest recommendation is very helpful for satisfying users’ needs at new places.In this article, we propose a tag embedding based method for Personalized Recommendation of Point OfInterest. We model the relationship between tags corresponding to Point Of Interest. The model providesrepresentative embedding corresponds to a tag in a way that related tags will be closer. We model Pointof Interest-based on tag embedding and also model the users (user profile) based on the Point Of Interestrated by them. finally, we rank the user’s candidate Point Of Interest based on cosine similarity betweenuser’s embedding and Point of Interest’s embedding. Further, we find the parameters required to model userby discrete optimizing over different measures (like ndcg@5, MRR, ...). We also analyze the result whileconsidering the same parameters for all users and individual parameters for each user. Along with it wealso analyze the effect on the result while changing the dataset to model the relationship between tags. Ourmethod also minimizes the privacy leak issue. We used TREC Contextual Suggestion 2016 Phase 2 datasetand have significant improvement over all the measures on the state of the art method. It improves ndcg@5by 12.8%, p@5 by 4.3%, and MRR by 7.8%, which shows the effectiveness of the method.

Keywords:Information Retrieval, Contextual Suggestion, Recommender System

1. Introduction

Due to easy access to the internet, more than 55% of the world’s population are using worldwide web 1

at present with a massive 4 billion users. The rapid production of information (in the form of text, images,audio, video, etc.) initiated the necessity for efficient handling (to store and search) of the information.One always requires assistance to get important information from this extensively large amount of data.Recommender systems play the role of this apprentice in suggesting essential instructions to the end-user.A recommender system suggests relevant items to users depending on his/her preferences [1].

With the advancement of technology, electronic devices have become smart and are capable of performingpowerful computations and assisting us in different ways. The advent of GPS has initiated the foundingstone for a plethora of technologies as well as new research domains related to navigation. One of theresearch domain that has emerged is the Point of Interest (POI) Recommendation System. A majorityof these recommender systems work based on preferences along with contextual information [2, 3, 4, 5].Collaborative filtering based methods were also used, which are using other user’s behavior to get the POIrecommendation [6, 7, 8, 9, 10]. Even some of the methods take proximity and temporal information of theuser to recommend the POIs [11, 12, 13, 14].

∗Corresponding author.Email addresses: s.agrawal1993@gmail.com (Suraj Agrawal), dwaipayan.roy@gmail.com (Dwaipayan Roy),

mandar@isical.ac.in (Mandar Mitra)1https://www.internetworldstats.com/emarketing.htm

Preprint submitted to Journal of Information Processing and Management April 15, 2020

arX

iv:2

004.

0638

9v1

[cs

.IR

] 1

4 A

pr 2

020

The Collaborative filtering based methods have generally suffered from the problems of data sparsity,while preference-based system required the information regarding what the user like/dislike about the places,which generally is captured using reviews. Reviews contain important information about the places, butmost of the users only rate the places without writing reviews. An alternative for the reluctant user is toprovide her with a list of predefined tags that the user may choose from in order to reflect her preference;nonetheless this is a challenging task to model a user because of the sparsity of different tags.

In this work, we have used tags to model users in order to perform POI recommendation. Our proposedapproach can be divided in four steps:

• Step I: Modeling the relationship between tags using word embedding.

• Step II: Representing users using tag embedding.

• Step III: Scoring of candidate Point of Interest on the basis of vector similarity between user profileand represented vectors for POI.

• Step IV: Refinement of ranking using binary classification on the basis of context.

2. Our Research Contribution

Research Objective. The purpose of this study is to build a recommender system based on onespreferences. With the emerging popularity of services like Yelp, Tripadvisor etc., research on POI recom-mendation surged its popularity among researchers. Development of an effective recommender system is astep towards a smarter contextualized search system that would be able to appropriately suggest places tovisit, or activities to the users and would be a steep jump towards development of a ‘zero query search’system.

Theoretical Contribution. Word embedding techniques have shown their proficiency in capturingsemantic regularities on the basis of word co-occurrence in fairly sized text corpora. The improvements,when applied for various text classification tasks, have been remarkable. In this study, we argue thatembeddings, when learned using a dataset containing short tag-texts without proper sentences, may ableto capture term relatedness similar to the scenario when the embeddings are trained using fairly sized textdump.

Practical Implication. The practical implication of this work lies in empirical validation of whetherapplying embeddings on less formal, tag text could actually be helpful in suggestion tasks. Experimentationon TREC contextual suggestion data and comparison with some of the best performing systems, validatesthe significant superiority of our proposed methods over baselines. The relatively less information of usersgetting exposed to the end point (server side) makes the approach more secure as well.

3. Terminologies

We use the term POI, Point of Interest and Place interchangeably to specify a potential place ofattraction for a user.

Tags are a taxonomical representation for places and POI s. For example, The Colosseum is a placewhich will be represented by the tags: {history, architecture}.

Profile is a single user’s preferences (list of POI s rated by user with their tags/endorsements andratings), their gender and age. For example, consider a male user having age 29, rated Victoria Palace withrating 3, Indian Museum with rating 4 and Bistro with rating 1 will represent his profile, where IndianMuseum, Victoria Palace and Bistro has tags {history, museum}, {history, architecture}, {pub, restaurant}respectively. The profile corresponding to this user would contain the places (associated tags) along withthe scores for the places rated by the user.

Context specifies the information about the destination city (i.e target location) of the visit along withtrip type (leisure, work), trip duration (one week, two weeks etc.), type of group the person is traveling with

2

(family, friends, colleagues) and season of the trip (spring, summer etc.). For example, a user visitingAmsterdam, with his friends on a weekend in spring will represent the context.

In this work, we propose a noble point-of-interest recommender system by embedding the tag informationof the places in an abstract space and finally, utilizing the embedded representation, to suggest potentialpoint of interest based on the profile and the context.

4. Related work

Research on recommender system has been a popular domain for quite some time. An extensive literaturesurvey is beyond the scope of this article. A comprehensive survey on recommender systems in general can befound in [15] and [16]. In this section, we discuss some of the works on recommender system in the followingtwo sub-domains. i) Recommender system for e-Tourism, discussed the work for tourism recommendationand privacy leak issues and ii) Embedding based recommender system, discussed different methods basedon POI embedding

4.1. Recommender Systems for e-Tourism

Researchers propose various methods for tourism recommendations. Borras et al. [17] provides a surveyof the field, considering the different kinds of interfaces, the diversity of recommendation algorithms andalso provides some guidelines for the construction of tourism recommenders. Renjith et al. [18] conducteddetailed study on the evolution of travel recommender systems, their features and limitation. Gavalas etal. [19] provide the classification of recommendation techniques in mobile recommendations used in tourism.Refanidis et al. [20] suggested the system myVisitPlanner, which recommends the activity along with theschedule, based on visit duration and timing, geographical areas of interest and visit profiling. Pliakos etal. [21] give a novel approach for an image-based recommendation. Tsekeridou et al. [22] proposed the iGuidesystem which will give recommendations with multimedia content. Park et al. [23] take context information,location, time, weather, and user request from the mobile device and uses Bayesian Networks to infers themost preferred recommendations. Y Ge. et al. [24] works on the energy-efficient recommendation system.Yang et al. [25] proposed iTravel, a recommendation system in a peer to peer mobile network. Researchersalso worked on the privacy leak issues in the mobile recommendation system. Efraimidis et al. [26] useprivacy by design approach to minimize the privacy leak. Danezis et al. [27] propose a method to map legalobligations to design strategies for implementing privacy requirements.

4.2. Embedding based Recommender Systems

Researchers explored the various embedding based method for POI recommendations. Hao et al. [28]uses real-time POI embedding to mine real-time information of the places and learn the latent representa-tions according to the corresponding geo-tagged posts using CNN, multi-modal embedding. The scientificcommunity also worked on the sequential influences of POIs for recommendations. Xie et al. [29] proposedthe Graph-based metric embedding, which converts POI to low dimension metrics and uses sequential check-in behavior for POI embedding(see [30] for graph-based embedding using check-in data). Tang et al. [29]embed a sequence of recent POIs into an image in the time, latent spaces and learn sequential patterns aslocal features of the image using convolutional filters. Wang et al. [31] use word2vec for POI embedding,then use RNN to model users’ successive transition behavior. Ozsoy et al. [32] use word2vec with check-insto the venues for making recommendations. Manotumruksa et al. [33] use the word embedding techniqueto infer the vector space representation of venues, user preferences, user’s contextual preferences. Xue etal. [34] proposed a deep Item-based collaborative filtering method for suggesting items, where they used adeep neural network to capture high order relationship among items. Manotumruksa et al. [35] proposedContextual Recurrent Collaborative Filtering framework(CRCF) to leverage the user’s preferred contextand the context info associated with the user’s sequence of check-ins in order to model user’s short termpreferences. Researchers also explored, Embedding technique using temporal information. Zhao et al. [36]propose Geo-Teaser, a temporal POI embedding model to learn POI representations under some particular

3

Figure 1: A sample tweet (anonymous) where the hashtags are indicative in expressing the basic information contained in thetweet.

temporal state. Chen et al. [37] propose Trip2Vec, a deep embedding based approach to find the purpose ofthe trip.

There have been some studies which uses TREC Contextual Suggestion dataset. Amrampatizis et al. [38]shows content-based methods (uses k-NN and weighted Rocchio), collaborative filtering methods and hybridmethods and find out content-based methods to be best. They also mentioned the privacy leak of thesemethods. Aliannejadi et al. [39, 40, 41] get a mapping between user-generated tags and taste keywordsusing the probabilistic model. They find context relevance of the place based its category. They calculatefrequency-based score for the category and tags to represent the relevance of places to user profile. Alongwith it they also train a classifier on given review text for relevance score of the place. Then combinethese scores to rank candidate POI( uses methods like RankBoost, AdaRank, LambdaMART). They alsopresent a CR framework that focuses on the top of the ranked list while integrating an arbitrary number ofsimilarity functions between venues as it learns the model’s parameters. Manotumruksa et al. [33] proposea method to find the appropriateness of the venue to the context using user-generated data on LSBN likefoursquare. Yang et al. [42] construct the methods to mine opinion about the places visited by the user andthen construct the opinion based user profiling in a collaborative manner. The result of methods based onTREC CS dataset is shown in Table 3.

5. Tag Based POI recommendation

We are currently in the middle of the expeditiously growing era of social media. Mediums like Facebook,Instagram and Twitter have made people less talking, more texting. After the inception of the practice ofusing hashtags in 20072, the general praxis has become even-less writing, more hash-tagging. Using tags,users can easily express their feelings as well as the tags can also be used for grouping similar thoughts3. Insum, tags have become a popular way of communication in the present age of micro-blogging.

In context of reviewing, tags have become crucial in understanding ones preferences (likes or dislikes). Ithas also become a social trend in posting photos and publishing in social-media from what a person is havingfor dinner, to the places he is visiting4. For example, consider a sample tweet presented in Figure 1 wherethe tags used by the user (nightout, dinner, friends) are specifying the trip type (nightout, dinner), as wellas the accompanying type of group (friends). Tags can also be useful in representing places on the basis ofits category (e.g. restaurant, museum) or the popular type of companion to visit with (e.g. friends, family).For example, Keen’s Steakhouse is a famous restaurant cum bar in the New York City, United State, thatserves good drinks as well as foods. Tags like pub, bar-hopping or restaurant can be used to represent thisplace.

Our hypothesis is based on the observation that there is a relationship between the ‘tags of places’ andthe ratings (implicit or explicit) for similar places assigned by people. Also, there could be some latentrelationships between tags as well using which, we can group similar places also likes and dislikes. Forexample, tags such as healthy food and yoga may have a co-relation, because a person having interest inhealthy foods is likely to be health conscious as well and likely to be interested in yoga5.

2https://twitter.com/chrismessina/status/2231154123For example, people discussing on topics related to Brexit use #Brexit to imprint their statement for easy understanding

of their thoughts.4https://www.newstatesman.com/culture/food-drink/2016/08/why-does-food-taste-better-when-we-instagram-it5Although seems too optimistic, we formulate this argument after initial manual observation of the data

4

5.1. Modelling Tags, POIs and Users

5.1.1. Modelling Tags

We need to model tags in a way that captures relationships between tags. Usually each POI has a setof tags associated with it (e.g. historical, sports etc.). POIs with similar theme share a set of commontags, along with a set of tags which are related (but not exactly same). For example, one point of interestNicco Park has tags: Outdoor activities, family friendly, parks, entertainment and another point of interestPrinsep Ghat has tags: Outdoor Activities, Citywalks, Scenery. The person interested in the place havingtags outdoor activities, parks may also be interested in the place having tags citywalks and scenery, whichshows that citywalks, parks, scenery are related.

Word embedding strategies, which project each word of the given vocabulary into an abstract spacekeeping the semantic relatedness of the words, have been proven very effective for different text processingtasks. If we train the tag information using an embedding technique, the linguistic relationship betweenterms are expected to be captured in the vector embeddings of the tags. For example, given the pointof interest Nicco Park having associated tags: Outdoor activities, family friendly, parks, entertainment,embedding techniques have the ability to predict tag Outdoor activities based on the associated tags familyfriendly, parks, entertainment considered as input.

5.1.2. Modelling POIs

Each POI contains tags, which describe the particular POI. For example, the POI Indian Museum hastags museum and history. These tags can be used to model the associated POIs and the vector for a POIcan be approximated by summing the vectors of all tags corresponding to that POI. Formally, let POI Pihave tags TG(Pi) =< tg1, tg2, . . . >, then the vector corresponds to Pi is:

Pi =∑

tgεTG(Pi)

tg (1)

Instead of taking summation of tag’s embedding to represent POIs, another approach can be to normalizethe representation by dividing the number of tags in the POI. Further approach in representing POIs asvectors is using the Doc2Vec model [43], where the POIs can be considered as a document having a sentencegenerated from tags (as described in Section 5.1.1). The Doc2Vec model can then be trained over all thePOIs having associated tags.

As our initial effort, we experimented with all the above approaches; in this article, we report theresults following the first approach (Equation 1) which was performing best in comparison to the rest of theapproaches.

5.1.3. Modelling Users

Depending on past experience, a user may want to visit similar places in future or may entirely avoidtouring some places. Note that, these two extreme preferences are expressed when the user is overwhelmedby the place (either positively or negatively). A neutral possibility is when the user has neither liked nordisliked the place. We represent a user based on the feelings she expressed for the places she visited andrated. The modelling is done based on three representations: whether the user liked, disliked or has nostrong opinion about the place and the representations are called positive profile, negative profile, andneutral profile respectively.

The preference of a user is rated in a scale of 0 (highly disliked) to 4 (greatly liked). Further, the POIwith a rating greater than 2 are relevant/positive or liked, rating 2 denotes neutral and a rating of less than2 is negative or disliked by a user. All the profiles can be modeled in two ways. All the positive (likewisenegative) profiles may be assigned equal importance (un-weighted). Alternatively, we may choose to assigngreater importance to strongly positive profiles (i.e. those rated 4) than to weakly positive profiles (i.e.those rated 3); and likewise for strongly and weekly negative profiles (weighted). A formal description ofthese two modeling types are discussed in the following:

5

• Un-weighted: In the un-weighted representation of users, we give equal importance to POIs whichare rated either as either strongly positive or as weakly positive and model the positive profile usingPOIs liked by the user. The positive profile for a user is created by summing vectors corresponding toPOIs relevant to the user. Formally, let the relevant POIs for user u be: prof+(u) =< P1, P2, . . . >.

Then, the profile of a user u, denoted by−−−−−−→prof+(u) is represented following Equation 2.

−−−−−−→prof+(u) =

∑Pεprof+(u) P

|prof+(u)|(2)

Similarly, we can represent a negative profile vector−−−−−−→prof−(u), as well as a neutral profile vector

−−−−−−→profo(u) by taking the neutral POIs to the user.

• Weighted: Strongly positive POIs are more liked by the user then weakly positive POIs. Thus a wayof incorporating a stronger opinions is by giving extra importance to strongly positive (or negative)POIs than weakly positive (or negative) POIs and model the positive (or negative) profile using POIsliked by the user. The importance (preferences) of the POI for the user is determined by the ratinggiven by the same user. We scale the rating to give the same amount of weight in positive and negativePOIs for strongly and weakly rated POIs. The scaling of ratings is defined in Table 1. Let relevantPOIs to user u be: prof+(u) =< P1, P2, . . . >. Then a positive profile is created following Equation 3.

−−−−−−→prof+(u) =

∑Pεprof+(u) P ∗ rating(P )

|prof+(u)|(3)

Similarly we can get a negative profile vector−−−−−−→prof−(u) and neutral profile vector

−−−−−−→profo(u) by taking

negative and neutral ratings given to POIs by the user.

Ratings given by user 0 1 2 3 4Scaled rating -3 -2 1 2 3

Table 1: Rating correspondence

5.2. Ranking method

We use the idea of Rocchio feedback algorithm [44] to rank the candidate POIs. In the Rocchio feedbackmethod, which works on the vector space model (VSM), the initial query vector Qi is modified to Qm

considering the centroid of the set of relevant (DR) and non-relevant (DNR) documents. Mathematically,the Rocchio’s feedback algorithm is presented in Equation 4.

Qm = α1

|DR|∑

di∈DR

di + βQO − γ1

|DNR|∑

dj∈DNR

dj (4)

In Equation 4, the parameters α and γ are weights associated with the set of relevant and non-relevantdocuments while β corresponds to the weight for the original query. In our case, we have created user vectorin the same manner from its positive (relevant), negative (non-relevant) and neutral profile representation.Formally, following the exposition of Rocchio model, we take a linear combination of positive user profilevector, neutral profile vector and negative profile vector to create an overall user profile vector [45] as follows:

−−−−−→prof(u) = α ∗

−−−−−−→prof+(u) + β ∗

−−−−−−→profo(u)− γ ∗

−−−−−−→prof−(u) (5)

In Equation 5, α, β and γ are parameters of the model respectively indicating weights associated withpositive, neutral or negative profiles, similar to Equation 4. Finally candidate suggestions are ranked based

on the cosine similarity between user profile vector−−−−−→prof(u) and the POI vector Pi.

6

TREC CS 2016 TREC CS 2015Number of requests 442 221Number of requests evaluated by TREC 58 211Number of Point Of Interest 18,808 8,794Number of Users 238 209Number of Users in evaluated request 27 209Number of POI rated per user 30 or 60 30 or 60Number of unique POI rated per user 60 4102

(Users rated same POI)Number of Candidate POI per request 79-119 (AVG 96.5) 30Number of rated POIs with Tags in evaluated request 2273/2310 6599/11400Number of candidate POI with Tags in evaluated request 4791/5599 0/6330Number of Unique Tags 150 186

Table 2: TREC CS Dataset comparison

6. Evaluation

6.1. Dataset

For experimental validation of our proposed model, we use the dataset used in TREC Contextual Sugges-tion (CS) track6. Similar to the recommendation scenario discussed in Section 1 and 2, the objective of thetrack, which ran for consecutive five years (from 2012 to 2016), was the study of techniques for addressing“complex information needs that are highly dependent on context and user interests” [46, 47, 48, 49, 50].The open web and ClueWeb12 were used as the dataset for selection of the point of interest during thefirst three years of this track. From 2015, the organizers introduced a dedicated dataset for the task havingeach attractions containing the title, the city as well as an URL of the place. An additional tag informationfor each of the attractions in the dataset was introduced in TREC 2016 CS track 7 which was generatedthrough crowd-sourcing [46]. We evaluated our methods on this latest benchmark dataset proposed in TREC2016. Table 2 presents the basic statistics of both TREC CS 2015 and 2015 datasets. For a comprehensivediscussion on the datasets please refer to [46].

6.2. Learning Embedded Representation

To train word2vec, we consider all the tags assigned to a particular POI as a single sentence. For exampletags Beer, Tourism, Outdoor Activities, Culture, History, Family Friendly, Food, Parks, Entertainment, LiveMusic are assigned to the document. The sentence corresponding to this place is supposed to be ‘beer tourismoutdoor-activities culture history family-friendly food parks entertainment live-music’. We create sentencesfor all the POIs in the request file and train word2vec. To this end, our assumption is the natural similarityof the tags would also be captured in the embedded representations. We expect related tags’ (for example,healthy food and yoga, romantic and boating) vector to have higher cosine similarity between them. As thetraining data is relatively limited, we further extend the set of sentences by permuting the terms in a sentenceto reflect the semantic similarity space. We train word2vec using two different text dumps: (i) The TRECContextual Suggestion 2016 dataset, and (ii) TREC Contextual Suggestion 2016 along with 2015 dataset.After varying the different parameters, the word2vec model is trained with the following settings : (i) CBOWmodel, (ii) window size of 5 (varying from 3 to 10), (iii) min count of 3 (varying from 3 to 7). As reportedin [52, 53], we experience similar observation in varying this three parameters. Due to the fewer number oftag terms (precisely 150) that has been considered as the vocabulary for training, the vector dimension ofis set to 9 after varying it from 5 to 20. Note that, this is significantly less than the usual settings for text

6https://trec.nist.gov/data/context.html7https://sites.google.com/site/treccontext/trec-2016

7

Figure 2: A two dimensional projection of the embedded vector representations corresponding to the tags using t-SNE [51].

retrieval where researchers vary the dimension from somewhere between 100 to 500 [54, 55, 56, 57, 58]. Thenumber of iteration is varied in the range {5, 10, 50, 100, 200, 500, 1000, 1500, 2000} and finally set it to 1000based on the initial results. A graphical representation of the trained model using tSNE [51] in presentedin Figure 2. In agreement with our assumption, it can be observed from the figure that the similar tags aresituated in near vicinity to each other, forming clusters based on the intent. For example, related tags likefine art museums, art galleries and art are seen to be very close to the vector of museum in the abstracttwo-dimensional projection of the vectors. similarly, the tags for food like healthy food, local food, fast food,etc are also close. The tags related to the same concept as food, drinking places, shopping places, naturalplaces, tourist places, etc are closer. This empirically shows that ability of word2vec [59] model in capturingthe relationships between tags.

8

6.3. Tuning parameters

There are three hyper-parameters associated with the tag-based proposed model: (α, β, γ) of Equa-tion 5 which respectively indicates the strength of positive, neutral and negative impressions on the overallrepresentation of the user profile. Instead of setting these parameters heuristically, we adapt a discreteoptimization technique to find the optimal values of these parameters with the scores considered as theoptimization function. In order to avoid overfiting parameters on one metric, we select various measurescores like NDCG@5, P@5, and MRR for optimization and to analyze the result.

Optimization function score. Consider the preferences of a user u, based on the point of interestsrated by user with tags/endorsements and ratings. For example, in case u rated Victoria Palace with rating3, Indian Museum with rating 4 and Bistro with rating 1, the corresponding will represent his user profile,where Indian Museum, Victoria Palace and Bistro has tags {History, Museum}, {History, Architecture},{Pub, Restaurant} respectively. Our approach generates positive profile vector

−−−−−−→prof+(u), neutral profile

vector−−−−−−→profo(u) and negative profile vector

−−−−−−→prof−(u) using procedure discussed in Section 5.1.3. Equation 5

took value of parameter α, β, γ in the search space and generate user profile vector−−−−−→prof(u) representing

user u. The method also generates POI vector for the places rated by user u (Victoria Palace, IndianMuseum, Bistro), using the procedure discussed in Section 5.1.2. Finally we rank these POI, based on thecosine similarity with user profile vector and calculate the measure score based on the rating given by theuser, which in turn will be the optimization function score.

In order to find the optimal values of the parameters (α, β, γ), we employed two different techniquestogether. In our initial experimentation, we applied genetic algorithms [60] to find the optimal parameters,then realized that we could search exhaustively in a certain subspace. Finally we use exhaustive search.we consider parameter β to be 1.0 and find α, γ in the search space. We consider −8.0 <= α <= 8.0 and−8.0 <= γ < 8.0 and exhaustively check for each α, γ in interval of 0.2. We choose α, γ that improves theperformances in terms of NDCG, P@5, MRR. Along with the optimizing the value of measure to get theparameters, we also tried two ways to get the parameters for user profiles.

6.3.1. Same parameter for all User profiles

We use the same parameters for all user profiles and rank the POIs rated by the user to tune theparameter instead of using cross-validation. We set the parameter such that the measure’s score of thePOIs ranked by the contextual suggestion system is maximized. In this case, the optimization score is thecombined score for all queries, which means it is the sum of all individual user’s scores, considering the sameparameters. This assigns the same parameter to all the user’s profile.

6.3.2. Parameter Per User Profile

If we use the same parameter for all the user profiles, then every user is given the same weights topositive, neutral and negative profiles. It might also be possible that different users have different weightsto positive, neutral and negative profiles. This can be captured if we can tune the parameter per profile andgive weights accordingly based on its profile. Also, in this case, we set the parameter such that the measure’sscore of the POIs ranked by the contextual suggestion system is maximized, but here we optimize the scorefor each user individually. This fixes different parameters value to a user’s profile, which will provide weightsaccording to the individual.

6.4. Evaluation Measures

In this study, we report three evaluation metrics to compare the effectiveness of the methods. These arealso the official metrics that has been used in the TREC CS track evaluations [61]. The final evaluation isperformed using the official ground truth provided by the TREC organizers8.

8https://trec.nist.gov/data/context.html

9

6.5. Experiments

We have tried all combinations of user profile generation methods and parameter tuning methods.

6.5.1. Weighted User Profile with the Same Parameter for all queries(WUPSame)

We created user profiles using the weighted user profile model and ranked the candidate suggestion, whileduring parameter tuning we consider the same parameter for all queries. The parameters are α, β, γ, whichshows the strength of positive, neutral and negative profile in the overall profile representation of the user.

6.5.2. Un-weighted User Profile with Same Parameter for all queries(UnWUPSame)

We created user profiles using an un-weighted user profile model and ranked the candidate suggestion,while during parameter tuning we consider the same parameter for all queries. The parameters are α, β, γ,which shows the strength of positive, neutral and negative profile in the overall profile representation of theuser.

6.5.3. Weighted User Profile with Unique Parameter for each query(WUPUniq)

We created user profiles using the weighted user profile model and ranked the candidate suggestion, whileparameters are tuned per query. The parameter is α, β, γ, which shows the strength of positive, neutral andnegative profile in the overall profile representation of the user.

6.5.4. Un-weighted User Profile with Unique Parameter for each query(UnWUPUniq)

We created user profiles using an un-weighted user profile model and ranked the candidate suggestion,while parameters are tuned per query. The parameters are α, β, γ, which shows the strength of positive,neutral and negative profile in the overall profile representation of the user.

6.5.5. Refine ranking

We also tried some methods along with the previous experiments to improve the ranking.

• Use context information: ‘Backstage’ is a pub that is not popular being visited with family,even if the user likes pub. Similarly, visiting tracks near Mumbai in monsoon season rather thanin summer is a general choice tourist often considered. This shows the importance of context indetermining the relevance of POI. We try to model this information, by training a classifier that findsif a POI is relevant to the context. Once we have the relevant POI’s to the context, we rank eachclass (relevant/not relevant) individually using UnWUPSame/WUPSame/WUPUniq/UnWUPUniq.Finally, we put non-relevant POIs after relevant POIs. For the Classifier, we create training datausing TREC Contextual Suggestion 2015 request file and qrels. It contains two context information:(i) Season (winter, summer, autumn, spring), and (ii) Group (friends, family, alone, others). In thedataset, the request file provides the feature vector and qrel gives relevance information.

The features of the classifier are POI vector and an 8-dimensional one-hot encoded vector representingseason, group. One hot encoded vector representing the season, group defined as following:

<′ winter′, ‘summer′, ‘autumn′, ‘spring′, ‘friends′, ‘family′, ‘alone′, ‘others′ >

We use kNN [62] as a classifier with k=5 to model the contextual relevance. As shown in Table 5,adding contextual information further makes a little improvement on the WUPUniq method, while allof the methods have better results on ndcg@5 from state of the art method.

• Learning to rank: In the previous experiments, we are combining a positive profile, negative profile,and neutral profile to generate user profile vector, then rank the POIs based on the cosine similaritywith the user profile vector. We are using discrete optimization to find the parameters to generatethe user profile vector. We tried to combine the cosine similarity of POI with an individual pro-file(positive/neutral and negative) instead of generating a user profile vector and then taking cosinesimilarity with it. We use learning the rank algorithms to combine the similarity with an individual

10

NDCG@5 P@5 MRRDUTH rocchio [64] 0.3306 0.4724 0.6801USI5 [66] 0.3265 0.5069 0.6796UAmsterdamDL [65] 0.2824 0.4448 0.5924Venue appropriateness prediction [41] 0.3603 0.5379 0.7054Personalized context aware point of interest [39] 0.3526 0.531 0.6800WUPSame 0.3932 0.5138 0.6969UnWUPSame 0.3982 0.5241 0.6952WUPUniq 0.3891 0.5345 0.6969UnWUPUniq 0.4064 0.5310 0.7106

Table 3: Comparison with base line method

NDCG@5 P@5 MRRWUPSame 0.4067 0.5586 0.7415UnWUPSame 0.4046 0.5621 0.7445WUPUniq 0.3973 0.5448 0.7413UnWUPUniq 0.3796 0.5172 0.7608

Table 4: Score of methods considering 2015+2016 Tags for tag embedding

profile(positive/neutral and negative). We take cosine similarity of POIs with positive profile, negativeprofile and neutral profile individually. This produces a 3-dimension vector, which can be used as a fea-ture vector for learning to rank algorithms. We use both weighted and un-weighted method to create auser profile and LambdaMART [63] algorithm for learning to rank. As shown in the Table 6, Learningto rank doesn’t improve the result, where WL2R represents weighted profile generation along withLambdaMART [63] and UnWL2R represents un-weighted profile generation with LambdaMART [63].

6.6. Baseline Methods

DUTH [64] collects more data that corresponds to POIs from the Location-based social networking site.They use K-NN to find the rating of candidate POIs, Rocchio to generate a query to get POI in the indexcorresponds to context and output result based on the combined score. UAmsterdam [65] learns languagemodel per user profile and POIs using tags, word embedding, and rank based on the KL Divergence score.USI [66] generates multiple scores based on the review of POIs rated by the user, category normalizedfrequency score, keywords normalized frequency score, tag normalized frequency score, context appropriate-ness score and combine them linearly. They got parameters using cross-validation. Venue appropriatenesspredictions [41] are the extension work of USI, which uses Learning to rank model to combine the score,which improves the result. Personalized Context-Aware Point of Interest Recommendation [39] finds thetags correspond to the keywords in the POIs. It finds out tags for each POIs(without tags) using multiplemethods. It also generates multiple scores similar to USI and combined them.

We run given experiments on two tag embedding (i) Tag embedding trained only on 2016 request file Tags,(ii) Tag embedding trained on 2015+2016 request file tags. Table 3 shows the result of all our experimentsalong with base line methods. Table 4 shows the result of our methods while adding 2015 request file’s tagalong with 2016 request file’s tag to train tag embedding.

Our all experiments gain significant improvement on main measure ndcg@5 over all existing methods.We also get significant improvement on P@5 as well as MRR. Along with it, we also get the improvementover other measures on the TREC runs. These results show the impact of the tag relationship model. Theresults with the bold font in the tables show the improved results.

11

2015 + 2016 Tag Embedding 2016 Tag EmbeddingNDCG@5 P@5 MRR NDCG@5 P@5 MRR

WUPSame 0.3951 0.5586 0.6990 0.3776 0.5138 0.6464UnWUPSame 0.3979 0.5552 0.7299 0.3835 0.5069 0.6792WUPUniq 0.4014 0.5448 0.7625 0.3858 0.5172 0.7041UnWUPUniq 0.3836 0.5310 0.7216 0.3963 0.5310 0.6992

Table 5: Score of methods after adding context relevance classifier

2015 + 2016 Tag Embedding 2016 Tag EmbeddingNDCG@5 P@5 MRR NDCG@5 P@5 MRR

WL2R 0.3531 0.4931 0.6514 0.3363 0.4828 0.6471UnWL2R 0.3644 0.4897 0.6964 0.3148 0.4172 0.5981

Table 6: Score of methods using Learning To Rank

7. Discussion

Our approaches to tackle this problem are mainly based on the tag relationship model. On top of the tagrelationship model, we apply various methods to model POIs and users, then we rank the candidate POIsusing ranking methods. We also use two methods to tune the parameters. We tried most of the combinationof these methods. While modeling tag relationships, POI’s, users and ranking the candidate POIs, thereare variables like the number of iteration, POI’s tags data to train the model for modeling tags relationshipand measures for optimization to get the parameters for getting user’s embedding.

Heatmap in the Table 7- 12 : We have tried to see, how scores (P@5, ndcg@5 and MRR) are varyingon changing optimization measures (e.g. ndcg@5, MRR, bpref, etc) to get parameters, dataset of tags (2016request file tags or 2015 request file along with 2016) and Number of iteration (500/1000) used to trainthe tag embedding across different methods. To show this variation, we have plotted the heat map acrossthese (Table 7- 12 ). In Table 7- 12 row shows the optimization measure (eg. ndcg@5, MRR, bpref, etc) toget parameters (α, β, γ), two major column representing the number of iteration used to train model to gettag embedding and four sub column to each major column shows the result for experiments UnWUPSame,WUPSame, UnWUPUniq, WUPUniq respectively. There are separate table for each output score (P@5,ndcg@5 and MRR) and dataset of tags (2016 request file tags or 2015 request file along with 2016) used totrain the tag embedding. The Table 7, 8 shows the Heatmap for P@5, Table 9, 10 shows the Heatmap forndcg@5 and Table 11, 12 shows Heatmap for MRR. Table 7, 9, 11 shows the methods results over 2015+2016tags data to train tag relationship model, while Table 8, 10, 12 shows the result over 2016 tags data.

Bar chart in Figure 3-5: We have also plotted a bar chart showing the average value of the resultswhile choosing a different measure to optimize for getting parameters for user profile generation. Alongwith the average value, we show the standard deviations by a vertical line on the top of the bar. We alsoshow the maximum value by the point on the top of the bar. Figure 3, 4, 5, shows the bar charts, for P@5,ndcg@5 and MRR respectively. Each Figure has two subfigure (a), (b), similar to the heatmap. Subfigure(a) shows the results on 2016 tags data for the training tag relationship model, while subfigure (b) showsresults on 2015+2016 tags data. Each subfigure x-axis shows the experiments and the y-axis show the valueof the result and there are two bars for each experiment. Each bar shows the result on the different numberof iteration on which tag relationship model is trained.

7.1. Impact of number of iteration and data points to train tag embedding

As our method mainly depends on the model to capture tag relations. We use word2vec to capture thisrelation. Changing the number of iteration could affect the results. Increasing the number of iteration to

12

(a) (b)

Figure 3: Effect on P@5 by changing the number of iterations to train the Tag embedding

train word2vec, for modeling tag relations, works differently based on the technique to tune the parameters.If we consider the same parameters for all users, the results using 1000 iteration tag embedding has a lesserstandard deviation. While if we consider unique parameters per user, increasing the number of iteration,increases the standard deviation in the results. In Figure 3, 4, 5 blue bar show the results from 1000 iterationtag embedding, while the green one shows the result using 500 iterations. We can see the standard deviationby the vertical bar in black at the top of bars.

We have also seen that adding tag data from TREC CS 2015 dataset have improved the performance ofthe system. Table 7, 8 shows the Heatmap for P@5, which show Table 7 is darker then Table 8 in most ofthe cases. Similarly Table 9, 10 and 11, 12 shows the Heatmap for ndcg@5 and MRR respectively. It is alsoshown that, Table 9, 11 are darker then 10, 12 respectively. Table 7, 9, 11 shows the results for P@5, ndcg@5and MRR using 2015+2016 datasets, which is better then the result shown in Table 8, 10, 12 respectively.Table 8, 10, 12 shows the result using 2016 data. This suggest that adding more tag data corresponds toPOI could improve the result further.

7.2. Impact of measures used to tune the parameters

The result also varies on changing the measure to optimize to get the parameters. As we notice heat mapshown in Table 7, 8, 9, 10, 11, 12 is darker in ndcg, MAP rows. This shows that, choosing ndcg, MAP or itsvalue till rank k, as the measure for optimization works better than other measures. These measures(MAP,ndcg) depends on the rank of relevant POI, which could be the reason why they work better then others.

7.3. Impact of parameter tuning methods on results

As we could see from heatmap show in the Table 7, 8, 9, 10, 11, 12 that it is more darker towardsUnWUPSame, WUPSame, then UnWUPUniq, WUPUniq side, which means that for most of the measurefor optimization, considering same parameters for all user profile generation is better than unique parametersper user profile. As we could also see in the Figure 3, 4, 5 bar representing UnWUPSame, WUPSameexperiments is dominating over UnWUPUniq, WUPUniq. But it is also noticeable that some time themaximum value for UnWUPUniq, WUPUniq is better then UnWUPSame and WUPSame.

The Unique parameters per profile have a lesser number of POIs. As it will optimize the measure on thePOIs rated by the user, it will have 30/60 POIs for TREC 2016 CS dataset. It is possible, that due to lessnumber of POIs, parameters get over-fitted over the POIs rated by the user, while in the case of the sameparameter for all users, due to different type of users, the parameters get generalized and worked better. Itis also noticeable in the Figure 3, 4, 5 that maximum values at the bar are very much close to average valuefor the experiments heaving same parameters for all profiles. This shows that experiments heaving the sameparameters for all profiles has a lesser deviation in the scores while changing the measure for optimization.

13

Embedding Trained on 2015+2016 Tags 500 Iteration Embedding Trained on 2015+2016 Tags 1000 Iteration

P@5 UnWUPSame WUPSame UnWUPUniq WUPUniq UnWUPSame WUPSame UnWUPUniq WUPUniq

bpref 0.5379 0.4966 0.4828 0.4759 0.5345 0.5276 0.4483 0.4862map 0.5379 0.4966 0.5172 0.5207 0.5379 0.5586 0.5034 0.5069

map cut 10 0.5414 0.5138 0.5276 0.5172 0.5241 0.5552 0.5172 0.5172map cut 15 0.5379 0.4966 0.5414 0.5379 0.5276 0.5379 0.4966 0.4966map cut 20 0.5379 0.4966 0.5552 0.5586 0.531 0.531 0.5 0.5172map cut 30 0.5379 0.5138 0.5172 0.5207 0.531 0.5586 0.5 0.4724map cut 5 0.5448 0.5138 0.4448 0.469 0.531 0.5241 0.469 0.4759

ndcg 0.5414 0.5172 0.4793 0.4966 0.5586 0.5586 0.5103 0.5069ndcg cut 10 0.5414 0.5172 0.5069 0.4862 0.5552 0.5586 0.469 0.5034ndcg cut 15 0.5414 0.5172 0.5103 0.4897 0.5276 0.5552 0.4862 0.5138ndcg cut 20 0.5414 0.4966 0.4931 0.4621 0.5552 0.5586 0.4724 0.5448ndcg cut 30 0.5414 0.5172 0.4517 0.4862 0.5552 0.5586 0.4828 0.4966ndcg cut 5 0.5448 0.5103 0.4931 0.5138 0.5517 0.5586 0.4828 0.5103

P 10 0.5241 0.5172 0.5483 0.5138 0.5517 0.5586 0.5103 0.5138P 15 0.5379 0.4931 0.5552 0.531 0.5379 0.5345 0.4897 0.4897P 20 0.5379 0.4931 0.5069 0.4103 0.531 0.5483 0.4241 0.5207P 30 0.4931 0.5138 0.4138 0.3966 0.4552 0.4966 0.4069 0.4138P 5 0.5345 0.5172 0.4414 0.4448 0.5621 0.5241 0.4483 0.4759

recall 10 0.5448 0.5172 0.5483 0.5138 0.5517 0.5586 0.5103 0.5138recall 15 0.5379 0.4931 0.5552 0.531 0.5379 0.5448 0.4897 0.4897recall 20 0.5379 0.4966 0.5069 0.4103 0.531 0.5586 0.4241 0.5207recall 30 0.4931 0.5138 0.4138 0.3966 0.4552 0.4966 0.4069 0.4138recall 5 0.5379 0.4931 0.4414 0.4448 0.5034 0.5241 0.4483 0.4759

recip rank 0.5379 0.4966 0.3517 0.3759 0.5 0.5483 0.3552 0.3276Rprec 0.5414 0.4931 0.4276 0.4517 0.5483 0.5172 0.4207 0.4034

Table 7: Effect on P@5 on varying the measures to tune the parameters considering 2015+2016 dataset

Embedding Trained on 2016 Tags, 500 Iteration Embedding Trained on 2016 Tags 1000 Iteration

P@5 UnWUPSame WUPSame UnWUPUniq WUPUniq UnWUPSame WUPSame UnWUPUniq WUPUniq

bpref 0.5276 0.5138 0.4310 0.4621 0.4966 0.4966 0.4621 0.4862map 0.5000 0.5034 0.4690 0.5034 0.4931 0.4966 0.4897 0.4862

map cut 10 0.5000 0.5034 0.4448 0.4931 0.5103 0.5138 0.4828 0.4897map cut 15 0.5069 0.5034 0.4586 0.4552 0.4931 0.4966 0.4724 0.4862map cut 20 0.5138 0.4897 0.4828 0.4828 0.4931 0.4966 0.4931 0.4966map cut 30 0.4931 0.5034 0.4724 0.5000 0.4931 0.5138 0.4966 0.4897map cut 5 0.4897 0.4897 0.4103 0.4448 0.5207 0.5138 0.4241 0.4517

ndcg 0.4828 0.5034 0.4793 0.4759 0.5172 0.5172 0.5345 0.5310ndcg cut 10 0.4759 0.5034 0.4793 0.4862 0.5138 0.5172 0.5310 0.5138ndcg cut 15 0.5138 0.5034 0.4586 0.5103 0.4931 0.5172 0.5207 0.5000ndcg cut 20 0.4793 0.5034 0.4793 0.4966 0.5172 0.4966 0.5379 0.5138ndcg cut 30 0.4793 0.5034 0.4586 0.4621 0.5207 0.5172 0.5448 0.5276ndcg cut 5 0.4759 0.5034 0.4310 0.4552 0.5241 0.5103 0.4828 0.4793

P 10 0.4724 0.5034 0.4414 0.4448 0.4897 0.5172 0.4897 0.4414P 15 0.5069 0.4759 0.4759 0.4207 0.4931 0.4931 0.4586 0.4793P 20 0.5207 0.5138 0.4897 0.4379 0.4897 0.4931 0.4690 0.4690P 30 0.4931 0.4793 0.3793 0.3621 0.4793 0.5138 0.4069 0.4069P 5 0.4621 0.4897 0.4241 0.4448 0.5138 0.5172 0.4207 0.4483

recall 10 0.4724 0.5034 0.4414 0.4448 0.4897 0.5172 0.4897 0.4414recall 15 0.5069 0.5069 0.4759 0.4207 0.4931 0.4931 0.4586 0.4793recall 20 0.5103 0.4897 0.4897 0.4379 0.4897 0.4966 0.4690 0.4690recall 30 0.4793 0.4793 0.3793 0.3621 0.4793 0.5138 0.4069 0.4069recall 5 0.4621 0.5207 0.4241 0.4448 0.5138 0.4931 0.4207 0.4483

recip rank 0.4828 0.4828 0.4000 0.3690 0.4552 0.4966 0.4414 0.3448Rprec 0.4828 0.4655 0.4310 0.4793 0.5000 0.4931 0.4828 0.4793

Table 8: Effect on P@5 on varying the measures to tune the parameters considering 2016 dataset

Embedding Trained on 2015+2016 Tags 500 Iteration Embedding Trained on 2015+2016 Tags 1000 Iteration

ndcg@5 UnWUPSame WUPSame UnWUPUniq WUPUniq UnWUPSame WUPSame UnWUPUniq WUPUniq

bpref 0.3973 0.3462 0.3474 0.341 0.3936 0.3918 0.3171 0.349map 0.3973 0.3668 0.3598 0.3694 0.3954 0.4067 0.3683 0.3722

map cut 10 0.4006 0.3813 0.3786 0.3678 0.3866 0.4009 0.3796 0.3769map cut 15 0.3948 0.3668 0.387 0.3916 0.3905 0.3932 0.3622 0.3564map cut 20 0.3973 0.3668 0.4077 0.4061 0.3882 0.3915 0.3613 0.3796map cut 30 0.3948 0.3813 0.3631 0.371 0.3903 0.4067 0.3699 0.3409map cut 5 0.4023 0.3813 0.3142 0.3288 0.3898 0.3921 0.3377 0.3381

ndcg 0.4008 0.3888 0.3379 0.3364 0.4013 0.4067 0.3592 0.3545ndcg cut 10 0.4006 0.3888 0.3515 0.3325 0.3975 0.4067 0.3286 0.3533ndcg cut 15 0.4012 0.3888 0.3551 0.3397 0.3905 0.4056 0.3598 0.3843ndcg cut 20 0.4004 0.3668 0.3455 0.321 0.3996 0.4067 0.335 0.3973ndcg cut 30 0.4008 0.3888 0.307 0.3272 0.3989 0.4067 0.3412 0.3507ndcg cut 5 0.3995 0.3861 0.3267 0.3535 0.3968 0.4067 0.3462 0.3614

P 10 0.3847 0.3822 0.3812 0.3579 0.3947 0.4068 0.3766 0.3711P 15 0.3956 0.3671 0.4017 0.377 0.3979 0.3925 0.3488 0.3544P 20 0.3977 0.3429 0.3596 0.2727 0.3873 0.4027 0.289 0.3731P 30 0.3259 0.3932 0.2805 0.2635 0.3088 0.3646 0.2902 0.3041P 5 0.3942 0.3839 0.3136 0.3063 0.4046 0.3921 0.3075 0.3368

recall 10 0.4012 0.3822 0.3812 0.3579 0.3947 0.4068 0.3766 0.3711recall 15 0.3956 0.3671 0.4017 0.377 0.3979 0.4019 0.3488 0.3544recall 20 0.3977 0.3671 0.3596 0.2727 0.3873 0.4068 0.289 0.3731recall 30 0.3259 0.3932 0.2805 0.2635 0.3088 0.3646 0.2902 0.3041recall 5 0.3953 0.3649 0.3136 0.3063 0.3607 0.3921 0.3075 0.3368

recip rank 0.3938 0.3437 0.2307 0.2464 0.3532 0.3984 0.2369 0.2135Rprec 0.4003 0.3401 0.2825 0.3131 0.3947 0.386 0.283 0.276

Table 9: Effect on ndcg@5 on varying the measures to tune the parameters considering 2015+2016 dataset

14

Embedding Trained on 2016 Tags 500 Iteration Embedding Trained on 2016 Tags 1000 Iteration

ndcg@5 UnWUPSame WUPSame UnWUPUniq WUPUniq UnWUPSame WUPSame UnWUPUniq WUPUniq

bpref 0.3751 0.3642 0.2963 0.3189 0.3587 0.3462 0.3231 0.3164map 0.3666 0.3637 0.3433 0.3732 0.3783 0.3668 0.37 0.3443

map cut 10 0.3666 0.3637 0.3071 0.3615 0.3887 0.3813 0.3572 0.3412map cut 15 0.372 0.3637 0.3283 0.3424 0.3793 0.3668 0.3381 0.3429map cut 20 0.374 0.3604 0.3521 0.346 0.3783 0.3668 0.3667 0.3472map cut 30 0.3689 0.3637 0.3483 0.372 0.3783 0.3813 0.3682 0.3445map cut 5 0.3655 0.3596 0.2916 0.3308 0.3946 0.3813 0.2891 0.2864

ndcg 0.3604 0.3637 0.3551 0.3525 0.3924 0.3888 0.4064 0.3891ndcg cut 10 0.3505 0.3637 0.3514 0.3599 0.3909 0.3888 0.4008 0.3656ndcg cut 15 0.3756 0.3637 0.3375 0.3736 0.3793 0.3888 0.3772 0.352ndcg cut 20 0.3591 0.3637 0.3418 0.3614 0.3935 0.3668 0.385 0.3646ndcg cut 30 0.3591 0.3637 0.337 0.3421 0.3957 0.3888 0.4048 0.3747ndcg cut 5 0.3523 0.3622 0.3291 0.3495 0.3982 0.3861 0.3709 0.3445

P 10 0.3499 0.3622 0.297 0.3081 0.3749 0.3822 0.3577 0.3014P 15 0.372 0.35 0.3234 0.2928 0.3784 0.3671 0.3331 0.3459P 20 0.3691 0.3777 0.3471 0.2955 0.3504 0.3429 0.3404 0.3366P 30 0.341 0.3534 0.2552 0.2671 0.3309 0.3932 0.2905 0.2886P 5 0.3393 0.3593 0.2907 0.3117 0.389 0.3839 0.2862 0.2811

recall 10 0.3499 0.3622 0.297 0.3081 0.3749 0.3822 0.3577 0.3014recall 15 0.372 0.3813 0.3234 0.2928 0.3784 0.3671 0.3331 0.3459recall 20 0.3654 0.3308 0.3471 0.2955 0.3504 0.3671 0.3404 0.3366recall 30 0.3558 0.3534 0.2552 0.2671 0.3309 0.3932 0.2905 0.2886recall 5 0.3393 0.384 0.2907 0.3117 0.389 0.3649 0.2862 0.2811

recip rank 0.3604 0.3507 0.2662 0.2418 0.3214 0.3437 0.3048 0.2181Rprec 0.3573 0.32 0.2906 0.3363 0.3645 0.3401 0.3485 0.339

Table 10: Effect on ndcg@5 on varying the measures to tune the parameters considering 2016 dataset

Embedding Trained on 2015+2016 Tags 500 Iteration Embedding Trained on 2015+2016 Tags 1000 Iteration

MRR UnWUPSame WUPSame UnWUPUniq WUPUniq UnWUPSame WUPSame UnWUPUniq WUPUniq

bpref 0.7457 0.6163 0.6887 0.6866 0.7311 0.7067 0.6991 0.7093map 0.7457 0.6515 0.637 0.6419 0.7357 0.7415 0.7397 0.6948

map cut 10 0.744 0.6655 0.6441 0.6471 0.7553 0.7357 0.7608 0.7306map cut 15 0.7658 0.6515 0.6414 0.6796 0.7371 0.7239 0.7234 0.6742map cut 20 0.7457 0.6515 0.7084 0.6772 0.7429 0.7068 0.7253 0.7311map cut 30 0.7658 0.6655 0.637 0.642 0.7253 0.7415 0.7316 0.6646map cut 5 0.7723 0.6655 0.5618 0.5872 0.7363 0.7167 0.6646 0.65

ndcg 0.7723 0.6843 0.6518 0.6166 0.7448 0.7415 0.7097 0.7187ndcg cut 10 0.744 0.6843 0.6747 0.6212 0.7318 0.7415 0.705 0.7425ndcg cut 15 0.7723 0.6843 0.66 0.6263 0.7371 0.7365 0.718 0.7631ndcg cut 20 0.7723 0.6515 0.6609 0.6005 0.7319 0.7415 0.7045 0.7413ndcg cut 30 0.7723 0.6843 0.6499 0.6271 0.7298 0.7415 0.687 0.7152ndcg cut 5 0.7295 0.6859 0.6007 0.6653 0.7264 0.7415 0.7022 0.7198

P 10 0.7258 0.6669 0.6641 0.623 0.7318 0.7386 0.7119 0.7011P 15 0.7723 0.6647 0.6798 0.6947 0.7472 0.7235 0.6811 0.704P 20 0.7486 0.6288 0.6719 0.5918 0.734 0.742 0.6319 0.6929P 30 0.7118 0.6968 0.6045 0.572 0.6223 0.7251 0.5694 0.5951P 5 0.7384 0.667 0.5627 0.5604 0.7445 0.7167 0.6113 0.6753

recall 10 0.7254 0.6669 0.6641 0.623 0.7318 0.7386 0.7119 0.7011recall 15 0.7723 0.6647 0.6798 0.6947 0.7472 0.7336 0.6811 0.704recall 20 0.7486 0.6564 0.6719 0.5918 0.734 0.7386 0.6319 0.6929recall 30 0.7118 0.6968 0.6045 0.572 0.6223 0.7251 0.5694 0.5951recall 5 0.7723 0.6592 0.5627 0.5604 0.7244 0.7167 0.6113 0.6753

recip rank 0.7342 0.6231 0.5336 0.5143 0.7195 0.7288 0.49 0.4839Rprec 0.7312 0.6145 0.5795 0.6197 0.734 0.7143 0.5509 0.5968

Table 11: Effect on MRR on varying the measures to tune the parameters considering 2015+2016 dataset

Embedding Trained on 2016 Tags 500 Iteration Embedding Trained on 2016 Tags 1000 Iteration

MRR UnWUPSame WUPSame UnWUPUniq WUPUniq UnWUPSame WUPSame UnWUPUniq WUPUniq

bpref 0.6899 0.6959 0.6389 0.6309 0.6892 0.6163 0.5899 0.5935map 0.7025 0.6516 0.6618 0.6776 0.6994 0.6515 0.6595 0.6349

map cut 10 0.7025 0.6516 0.6539 0.6644 0.6878 0.6655 0.6398 0.5846map cut 15 0.7044 0.6516 0.6479 0.6361 0.6992 0.6515 0.604 0.6105map cut 20 0.6996 0.6735 0.6589 0.6221 0.6994 0.6515 0.6755 0.615map cut 30 0.7129 0.6516 0.6615 0.6776 0.6994 0.6655 0.6675 0.6395map cut 5 0.7254 0.6741 0.6194 0.6818 0.688 0.6655 0.597 0.5898

ndcg 0.7084 0.6516 0.6939 0.6735 0.6878 0.6843 0.7106 0.6969ndcg cut 10 0.6995 0.6516 0.6771 0.6923 0.6849 0.6843 0.716 0.6578ndcg cut 15 0.703 0.6516 0.6613 0.6541 0.6992 0.6843 0.6594 0.6436ndcg cut 20 0.7153 0.6516 0.6242 0.6552 0.6878 0.6515 0.6721 0.6856ndcg cut 30 0.7153 0.6516 0.6659 0.6621 0.688 0.6843 0.7122 0.6837ndcg cut 5 0.6999 0.6515 0.6477 0.7022 0.6952 0.6859 0.6878 0.6727

P 10 0.6949 0.6515 0.6112 0.6135 0.6943 0.6669 0.6339 0.5497P 15 0.7044 0.6636 0.6191 0.6063 0.6995 0.6647 0.6566 0.6419P 20 0.7134 0.7295 0.624 0.602 0.6882 0.6288 0.6399 0.6258P 30 0.6843 0.6675 0.5419 0.5495 0.6642 0.6968 0.5675 0.5704P 5 0.6976 0.6649 0.6164 0.644 0.6856 0.667 0.597 0.5915

recall 10 0.6949 0.6515 0.6112 0.6135 0.6943 0.6669 0.6339 0.5497recall 15 0.7044 0.6949 0.6191 0.6063 0.6995 0.6647 0.6566 0.6419recall 20 0.6782 0.6694 0.624 0.602 0.6882 0.6564 0.6399 0.6258recall 30 0.7127 0.6675 0.5419 0.5495 0.6642 0.6968 0.5675 0.5704recall 5 0.6976 0.706 0.6164 0.644 0.6856 0.6592 0.597 0.5915

recip rank 0.7084 0.6472 0.5321 0.5512 0.68 0.6231 0.5442 0.4618Rprec 0.7207 0.6778 0.6201 0.6619 0.6997 0.6145 0.6587 0.6443

Table 12: Effect on MRR on varying the measures to tune the parameters considering 2016 dataset

15

(a) (b)

Figure 4: Effect on ndcg@5 by changing the number of iterations to train the Tag embedding

(a) (b)

Figure 5: Effect on MRR by changing the number of iterations to train the Tag embedding

16

7.4. Privacy Issue

There is a privacy concern in the mobile recommender systems, as large sensitive (personal preference)data is there on the mobile device that can be accessed and sent over the remote server by a mobileapplication. This is a concern with respect to the user’s identity and location privacy, as discussed byArampatzis et al. [38].

Our methods can achieve user privacy by protecting sensitive user information. Today’s mobile deviceare computationally intensive with sufficient storage capacity. We can store information regarding a largenumber of POIs and create user specific profile vector in a mobile device, and the personal information ofthe user need not be exposed. It will only expose the context information of the user. Based on the contextinformation (e.g. location / city), the recommender system could provide a list of POIs along with their tags/ POIs representation using which ranking can be done on mobile devices. All of the information is storedat the mobile device, which only requires tags representation and parameters’ value to generate the users’representation. Tags’ representation and parameters could be passed to the mobile device, which does notreveal any information about mobile device users.

8. Conclusions and future work

Recommending point of interest is an important problem for mobile users. In this article, we havepresented a word embedding based model to capture the relationship between tags. We argue that therating-based tagged information can also be embedded in an abstract space and will also capture the se-mantic relatedness among the tags. Using word2vec [59] model to embed the tags in an abstract space, wegraphically present the evidence in favour of our intuition, that embedded tag-vectors able to reflect thesemantic relatedness of the individual tags. Modifying the Rocchio based feedback technique, we proposea novel tag-embedding based recommendation technique to model users and point of interests. Further wepresent approaches on refining the ranking using learning to rank techniques, adding context relevance. Weuse discrete optimization in order to find optimal parameters instead of choosing it heuristically. Empiricalevaluation on TREC contextual suggestion data validates the superiority of the proposed method in com-parison to the state-of-the-art techniques. The results indicate that increasing the data on tags regardingpoint of interests can further increase the performance. We also show the variation of results on changingthe optimization measure to get the parameters for modeling users. The proposed method can reduce theprivacy concern for mobile devices while suggesting personalized POIs. Our approach is dependent on tags,therefore not immediately applicable for TREC CS 2015 dataset. As a part of future work, we plan onexploring transfer learning techniques for the same tasks with missing tags.

References

[1] P. Resnick, H. R. Varian, Recommender systems - introduction to the special section, Commun. ACM 40 (3) (1997) 56–58.doi:10.1145/245108.245121.URL https://doi.org/10.1145/245108.245121

[2] G. Adomavicius, A. Tuzhilin, Context-aware recommender systems, in: F. Ricci, L. Rokach, B. Shapira, P. B. Kantor(Eds.), Recommender Systems Handbook, Springer, 2011, pp. 217–253. doi:10.1007/978-0-387-85820-3\_7.URL https://doi.org/10.1007/978-0-387-85820-3_7

[3] H. Park, J. Yoo, S. Cho, A context-aware music recommendation system using fuzzy bayesian networks with utility theory,in: L. Wang, L. Jiao, G. Shi, X. Li, J. Liu (Eds.), Fuzzy Systems and Knowledge Discovery, Third International Conference,FSKD 2006, Xi’an, China, September 24-28, 2006, Proceedings, Vol. 4223 of Lecture Notes in Computer Science, Springer,2006, pp. 970–979. doi:10.1007/11881599\_121.URL https://doi.org/10.1007/11881599_121

[4] L. Baltrunas, B. Ludwig, F. Ricci, Matrix factorization techniques for context aware recommendation, in: B. Mobasher,R. D. Burke, D. Jannach, G. Adomavicius (Eds.), Proceedings of the 2011 ACM Conference on Recommender Systems,RecSys 2011, Chicago, IL, USA, October 23-27, 2011, ACM, 2011, pp. 301–304. doi:10.1145/2043932.2043988.URL https://doi.org/10.1145/2043932.2043988

[5] K. Verbert, N. Manouselis, X. Ochoa, M. Wolpers, H. Drachsler, I. Bosnic, E. Duval, Context-aware recommender systemsfor learning: A survey and future challenges, TLT 5 (4) (2012) 318–335. doi:10.1109/TLT.2012.11.URL https://doi.org/10.1109/TLT.2012.11

17

[6] A. Karatzoglou, X. Amatriain, L. Baltrunas, N. Oliver, Multiverse recommendation: n-dimensional tensor factorizationfor context-aware collaborative filtering, in: X. Amatriain, M. Torrens, P. Resnick, M. Zanker (Eds.), Proceedings of the2010 ACM Conference on Recommender Systems, RecSys 2010, Barcelona, Spain, September 26-30, 2010, ACM, 2010,pp. 79–86. doi:10.1145/1864708.1864727.URL https://doi.org/10.1145/1864708.1864727

[7] J. B. Schafer, D. Frankowski, J. L. Herlocker, S. Sen, Collaborative filtering recommender systems, in: P. Brusilovsky,A. Kobsa, W. Nejdl (Eds.), The Adaptive Web, Methods and Strategies of Web Personalization, Vol. 4321 of LectureNotes in Computer Science, Springer, 2007, pp. 291–324. doi:10.1007/978-3-540-72079-9\_9.URL https://doi.org/10.1007/978-3-540-72079-9_9

[8] B. M. Sarwar, G. Karypis, J. A. Konstan, J. Riedl, Item-based collaborative filtering recommendation algorithms (2001)285–295doi:10.1145/371920.372071.URL https://doi.org/10.1145/371920.372071

[9] M. Balabanovic, Y. Shoham, Content-based, collaborative recommendation, Commun. ACM 40 (3) (1997) 66–72. doi:

10.1145/245108.245124.URL https://doi.org/10.1145/245108.245124

[10] Z. Zheng, H. Ma, M. R. Lyu, I. King, Wsrec: A collaborative filtering based web service recommender system, in: IEEEInternational Conference on Web Services, ICWS 2009, Los Angeles, CA, USA, 6-10 July 2009, IEEE Computer Society,2009, pp. 437–444. doi:10.1109/ICWS.2009.30.URL https://doi.org/10.1109/ICWS.2009.30

[11] V. L. Wijeratne, Local area preference determination system and method, uS Patent 7,302,468 (Nov. 27 2007).URL https://patentimages.storage.googleapis.com/d6/b1/59/e67b4409aafd73/US7302468.pdf

[12] Q. Yuan, G. Cong, Z. Ma, A. Sun, N. Magnenat-Thalmann, Time-aware point-of-interest recommendation, in: G. J. F.Jones, P. Sheridan, D. Kelly, M. de Rijke, T. Sakai (Eds.), The 36th International ACM SIGIR conference on research anddevelopment in Information Retrieval, SIGIR ’13, Dublin, Ireland - July 28 - August 01, 2013, ACM, 2013, pp. 363–372.doi:10.1145/2484028.2484030.URL https://doi.org/10.1145/2484028.2484030

[13] J. Roberts, System and method for capturing, integrating, discovering, and using geo-temporal data, uS Patent App.11/871,842 (Apr. 16 2009).URL https://patentimages.storage.googleapis.com/4f/65/cd/bb0582ceaf26ef/US20090100018A1.pdf

[14] A. Hinze, A. Voisard, Locations- and time-based information delivery in tourism, in: T. Hadzilacos, Y. Manolopoulos,J. F. Roddick, Y. Theodoridis (Eds.), Advances in Spatial and Temporal Databases, 8th International Symposium, SSTD2003, Santorini Island, Greece, July 24-27, 2003, Proceedings, Vol. 2750 of Lecture Notes in Computer Science, Springer,2003, pp. 489–507. doi:10.1007/978-3-540-45072-6\_28.URL https://doi.org/10.1007/978-3-540-45072-6_28

[15] J. Bobadilla, F. Ortega, A. Hernando, A. Gutierrez, Recommender systems survey, Knowl. Based Syst. 46 (2013) 109–132.doi:10.1016/j.knosys.2013.03.012.URL https://doi.org/10.1016/j.knosys.2013.03.012

[16] F. Ricci, L. Rokach, B. Shapira, Introduction to recommender systems handbook, in: F. Ricci, L. Rokach, B. Shapira,P. B. Kantor (Eds.), Recommender Systems Handbook, Springer, 2011, pp. 1–35. doi:10.1007/978-0-387-85820-3\_1.URL https://doi.org/10.1007/978-0-387-85820-3_1

[17] J. Borras, A. Moreno, A. Valls, Intelligent tourism recommender systems: A survey, Expert Syst. Appl. 41 (16) (2014)7370–7389. doi:10.1016/j.eswa.2014.06.007.URL https://doi.org/10.1016/j.eswa.2014.06.007

[18] S. Renjith, A. Sreekumar, M. Jathavedan, An extensive study on the evolution of context-aware personalized travelrecommender systems, Information Processing & Management 57 (1) (2020) 102078. doi:https://doi.org/10.1016/j.

ipm.2019.102078.URL http://www.sciencedirect.com/science/article/pii/S0306457319300111

[19] D. Gavalas, C. Konstantopoulos, K. Mastakas, G. E. Pantziou, Mobile recommender systems in tourism, J. Netw. Comput.Appl. 39 (2014) 319–333. doi:10.1016/j.jnca.2013.04.006.URL https://doi.org/10.1016/j.jnca.2013.04.006

[20] I. Refanidis, C. Emmanouilidis, I. Sakellariou, A. Alexiadis, R. Koutsiamanis, K. Agnantis, A. Tasidou, F. Kokkoras,P. S. Efraimidis, myvisitplanner gr: Personalized itinerary planning system for tourism, in: A. Likas, K. Blekas, D. Kalles(Eds.), Artificial Intelligence: Methods and Applications - 8th Hellenic Conference on AI, SETN 2014, Ioannina, Greece,May 15-17, 2014. Proceedings, Vol. 8445 of Lecture Notes in Computer Science, Springer, 2014, pp. 615–629. doi:

10.1007/978-3-319-07064-3\_53.URL https://doi.org/10.1007/978-3-319-07064-3_53

[21] K. Pliakos, C. Kotropoulos, Simultaneous image clustering, classification and annotation for tourism recommendation, in:A. Likas, K. Blekas, D. Kalles (Eds.), Artificial Intelligence: Methods and Applications - 8th Hellenic Conference on AI,SETN 2014, Ioannina, Greece, May 15-17, 2014. Proceedings, Vol. 8445 of Lecture Notes in Computer Science, Springer,2014, pp. 630–640. doi:10.1007/978-3-319-07064-3\_54.URL https://doi.org/10.1007/978-3-319-07064-3_54

[22] S. Tsekeridou, V. Tsetsos, A. Chalamandaris, C. Chamzas, T. Filippou, C. Pantzoglou, iguide: Socially-enriched mobiletourist guide for unexplored sites, in: A. Likas, K. Blekas, D. Kalles (Eds.), Artificial Intelligence: Methods and Applica-tions - 8th Hellenic Conference on AI, SETN 2014, Ioannina, Greece, May 15-17, 2014. Proceedings, Vol. 8445 of LectureNotes in Computer Science, Springer, 2014, pp. 603–614. doi:10.1007/978-3-319-07064-3\_52.

18

URL https://doi.org/10.1007/978-3-319-07064-3_52

[23] M. Park, J. Hong, S. Cho, Location-based recommendation system using bayesian user’s preference model in mobiledevices, in: J. Indulska, J. Ma, L. T. Yang, T. Ungerer, J. Cao (Eds.), Ubiquitous Intelligence and Computing, 4thInternational Conference, UIC 2007, Hong Kong, China, July 11-13, 2007, Proceedings, Vol. 4611 of Lecture Notes inComputer Science, Springer, 2007, pp. 1130–1139. doi:10.1007/978-3-540-73549-6\_110.URL https://doi.org/10.1007/978-3-540-73549-6_110

[24] Y. Ge, H. Xiong, A. Tuzhilin, K. Xiao, M. Gruteser, M. J. Pazzani, An energy-efficient mobile recommender system, in:B. Rao, B. Krishnapuram, A. Tomkins, Q. Yang (Eds.), Proceedings of the 16th ACM SIGKDD International Conferenceon Knowledge Discovery and Data Mining, Washington, DC, USA, July 25-28, 2010, ACM, 2010, pp. 899–908. doi:

10.1145/1835804.1835918.URL https://doi.org/10.1145/1835804.1835918

[25] W. Yang, S. Hwang, itravel: A recommender system in mobile peer-to-peer environment, J. Syst. Softw. 86 (1) (2013)12–20. doi:10.1016/j.jss.2012.06.041.URL https://doi.org/10.1016/j.jss.2012.06.041

[26] P. S. Efraimidis, G. Drosatos, A. Arampatzis, G. Stamatelatos, I. N. Athanasiadis, A privacy-by-design contextual sug-gestion system for tourism, J. Sensor and Actuator Networks 5 (2) (2016) 10. doi:10.3390/jsan5020010.URL https://doi.org/10.3390/jsan5020010

[27] G. Danezis, J. Domingo-Ferrer, M. Hansen, J. Hoepman, D. L. Metayer, R. Tirtea, S. Schiffner, Privacy and data protectionby design - from policy to engineering, CoRR abs/1501.03726. arXiv:1501.03726.URL http://arxiv.org/abs/1501.03726

[28] P. Hao, W. Cheang, J. Chiang, Real-time event embedding for POI recommendation, Neurocomputing 349 (2019) 1–11.doi:10.1016/j.neucom.2019.04.022.URL https://doi.org/10.1016/j.neucom.2019.04.022

[29] M. Xie, H. Yin, F. Xu, H. Wang, X. Zhou, Graph-based metric embedding for next POI recommendation, in: W. Cellary,M. F. Mokbel, J. Wang, H. Wang, R. Zhou, Y. Zhang (Eds.), Web Information Systems Engineering - WISE 2016 - 17thInternational Conference, Shanghai, China, November 8-10, 2016, Proceedings, Part II, Vol. 10042 of Lecture Notes inComputer Science, 2016, pp. 207–222. doi:10.1007/978-3-319-48743-4\_17.URL https://doi.org/10.1007/978-3-319-48743-4_17

[30] M. Xie, H. Yin, H. Wang, F. Xu, W. Chen, S. Wang, Learning graph-based POI embedding for location-based recom-mendation, in: S. Mukhopadhyay, C. Zhai, E. Bertino, F. Crestani, J. Mostafa, J. Tang, L. Si, X. Zhou, Y. Chang, Y. Li,P. Sondhi (Eds.), Proceedings of the 25th ACM International Conference on Information and Knowledge Management,CIKM 2016, Indianapolis, IN, USA, October 24-28, 2016, ACM, 2016, pp. 15–24. doi:10.1145/2983323.2983711.URL https://doi.org/10.1145/2983323.2983711

[31] M. Wang, Y. Lu, J. Huang, SPENT: A successive POI recommendation method using similarity-based POI embedding andrecurrent neural network with temporal influence, in: IEEE International Conference on Big Data and Smart Computing,BigComp 2019, Kyoto, Japan, February 27 - March 2, 2019, IEEE, 2019, pp. 1–8. doi:10.1109/BIGCOMP.2019.8679431.URL https://doi.org/10.1109/BIGCOMP.2019.8679431

[32] M. G. Ozsoy, From word embeddings to item recommendation, CoRR abs/1601.01356. arXiv:1601.01356.URL http://arxiv.org/abs/1601.01356

[33] J. Manotumruksa, C. MacDonald, I. Ounis, Modelling user preferences using word embeddings for context-aware venuerecommendation, CoRR abs/1606.07828. arXiv:1606.07828.URL http://arxiv.org/abs/1606.07828

[34] F. Xue, X. He, X. Wang, J. Xu, K. Liu, R. Hong, Deep item-based collaborative filtering for top-n recommendation, ACMTrans. Inf. Syst. 37 (3) (2019) 33:1–33:25. doi:10.1145/3314578.URL http://doi.acm.org/10.1145/3314578

[35] J. Manotumruksa, C. Macdonald, I. Ounis, A contextual recurrent collaborative filtering framework for modelling se-quences of venue checkins, Information Processing & Management (2019) 102092doi:https://doi.org/10.1016/j.ipm.2019.102092.URL http://www.sciencedirect.com/science/article/pii/S0306457319301876

[36] S. Zhao, T. Zhao, I. King, M. R. Lyu, Geo-teaser: Geo-temporal sequential embedding rank for point-of-interestrecommendation, in: R. Barrett, R. Cummings, E. Agichtein, E. Gabrilovich (Eds.), Proceedings of the 26th Inter-national Conference on World Wide Web Companion, Perth, Australia, April 3-7, 2017, ACM, 2017, pp. 153–162.doi:10.1145/3041021.3054138.URL https://doi.org/10.1145/3041021.3054138

[37] C. Chen, C. Liao, X. Xie, Y. Wang, J. Zhao, Trip2vec: a deep embedding approach for clustering and profiling taxi trippurposes, Personal and Ubiquitous Computing 23 (1) (2019) 53–66. doi:10.1007/s00779-018-1175-9.URL https://doi.org/10.1007/s00779-018-1175-9

[38] A. Arampatzis, G. Kalamatianos, Suggesting points-of-interest via content-based, collaborative, and hybrid fusion methodsin mobile devices, ACM Trans. Inf. Syst. 36 (3) (2018) 23:1–23:28. doi:10.1145/3125620.URL https://doi.org/10.1145/3125620

[39] M. Aliannejadi, F. Crestani, Personalized context-aware point of interest recommendation, ACM Trans. Inf. Syst. 36 (4)(2018) 45:1–45:28. doi:10.1145/3231933.URL https://doi.org/10.1145/3231933

[40] M. Aliannejadi, D. Rafailidis, F. Crestani, A collaborative ranking model with multiple location-based similarities forvenue suggestion, in: D. Song, T. Liu, L. Sun, P. Bruza, M. Melucci, F. Sebastiani, G. H. Yang (Eds.), Proceedings of the

19

2018 ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR 2018, Tianjin, China, September14-17, 2018, ACM, 2018, pp. 19–26. doi:10.1145/3234944.3234945.URL https://doi.org/10.1145/3234944.3234945

[41] M. Aliannejadi, F. Crestani, Venue appropriateness prediction for personalized context-aware venue suggestion, in:N. Kando, T. Sakai, H. Joho, H. Li, A. P. de Vries, R. W. White (Eds.), Proceedings of the 40th International ACMSIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017,ACM, 2017, pp. 1177–1180. doi:10.1145/3077136.3080754.URL https://doi.org/10.1145/3077136.3080754

[42] P. Yang, H. Wang, H. Fang, D. Cai, Opinions matter: a general approach to user profile modeling for contextual suggestion,Information Retrieval Journal 18 (6) (2015) 586–610. doi:10.1007/s10791-015-9278-7.URL https://doi.org/10.1007/s10791-015-9278-7

[43] J. H. Lau, T. Baldwin, An empirical evaluation of doc2vec with practical insights into document embedding generation,CoRR abs/1607.05368. arXiv:1607.05368.URL http://arxiv.org/abs/1607.05368

[44] J. J. Rocchio, Relevance feedback in information retrieval, The SMART retrieval system: experiments in automaticdocument processing (1971) 313–323.

[45] G. Salton, C. Buckley, Improving retrieval performance by relevance feedback, JASIS 41 (4) (1990) 288–297. doi:

10.1002/(SICI)1097-4571(199006)41:4<288::AID-ASI8>3.0.CO;2-H.URL https://doi.org/10.1002/(SICI)1097-4571(199006)41:4<288::AID-ASI8>3.0.CO;2-H

[46] S. H. Hashemi, C. L. Clarke, J. Kamps, J. Kiseleva, E. Voorhees, Overview of the trec 2016 contextual suggestion track,in: Proceedings of TREC, Vol. 2016, 2016.URL https://trec.nist.gov/pubs/trec25/papers/Overview-CS.pdf

[47] A. Dean-Hall, C. L. A. Clarke, J. Kamps, J. Kiseleva, E. M. Voorhees, Overview of the TREC 2015 contextual suggestiontrack, in: E. M. Voorhees, A. Ellis (Eds.), Proceedings of The Twenty-Fourth Text REtrieval Conference, TREC 2015,Gaithersburg, Maryland, USA, November 17-20, 2015, Vol. Special Publication 500-319, National Institute of Standardsand Technology (NIST), 2015.URL http://trec.nist.gov/pubs/trec24/papers/Overview-CX.pdf

[48] A. Dean-Hall, C. L. A. Clarke, J. Kamps, P. Thomas, E. M. Voorhees, Overview of the TREC 2014 contextual suggestiontrack, in: E. M. Voorhees, A. Ellis (Eds.), Proceedings of The Twenty-Third Text REtrieval Conference, TREC 2014,Gaithersburg, Maryland, USA, November 19-21, 2014, Vol. Special Publication 500-308, National Institute of Standardsand Technology (NIST), 2014.URL http://trec.nist.gov/pubs/trec23/papers/overview-context.pdf

[49] A. Dean-Hall, C. L. A. Clarke, N. Simone, J. Kamps, P. Thomas, E. M. Voorhees, Overview of the TREC 2013 contextualsuggestion track, in: E. M. Voorhees (Ed.), Proceedings of The Twenty-Second Text REtrieval Conference, TREC 2013,Gaithersburg, Maryland, USA, November 19-22, 2013, Vol. Special Publication 500-302, National Institute of Standardsand Technology (NIST), 2013.URL http://trec.nist.gov/pubs/trec22/papers/CONTEXT.OVERVIEW.pdf

[50] A. Dean-Hall, C. L. A. Clarke, J. Kamps, P. Thomas, E. M. Voorhees, Overview of the TREC 2012 contextual suggestiontrack, in: E. M. Voorhees, L. P. Buckland (Eds.), Proceedings of The Twenty-First Text REtrieval Conference, TREC 2012,Gaithersburg, Maryland, USA, November 6-9, 2012, Vol. Special Publication 500-298, National Institute of Standards andTechnology (NIST), 2012.URL http://trec.nist.gov/pubs/trec21/papers/CONTEXTUAL12.overview.pdf

[51] L. v. d. Maaten, G. Hinton, Visualizing data using t-sne, Journal of Machine Learning Research (JMLR) 9 (November)(2008) 2579–2605.URL http://www.jmlr.org/papers/v9/vandermaaten08a.html

[52] G. Zuccon, B. Koopman, P. Bruza, L. Azzopardi, Integrating and evaluating neural word embeddings in information re-trieval, in: Proceedings of the 20th Australasian Document Computing Symposium, ADCS 15, Association for ComputingMachinery, New York, NY, USA, 2015. doi:10.1145/2838931.2838936.URL https://doi.org/10.1145/2838931.2838936

[53] D. Roy, D. Ganguly, S. Bhatia, S. Bedathur, M. Mitra, Using word embeddings for information retrieval: How collection andterm normalization choices affect performance, in: Proceedings of the 27th ACM International Conference on Informationand Knowledge Management, CIKM 18, Association for Computing Machinery, New York, NY, USA, 2018, p. 18351838.doi:10.1145/3269206.3269277.URL https://doi.org/10.1145/3269206.3269277

[54] D. Ganguly, D. Roy, M. Mitra, G. J. Jones, Word embedding based generalized language model for information retrieval,in: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval,SIGIR 15, Association for Computing Machinery, New York, NY, USA, 2015, p. 795798. doi:10.1145/2766462.2767780.URL https://doi.org/10.1145/2766462.2767780

[55] M. Grbovic, N. Djuric, V. Radosavljevic, F. Silvestri, N. Bhamidipati, Context- and content-aware embeddings for queryrewriting in sponsored search, in: Proceedings of the 38th International ACM SIGIR Conference on Research and Devel-opment in Information Retrieval, SIGIR 15, Association for Computing Machinery, New York, NY, USA, 2015, p. 383392.doi:10.1145/2766462.2767709.URL https://doi.org/10.1145/2766462.2767709

[56] D. Roy, D. Ganguly, M. Mitra, G. J. F. Jones, Estimating gaussian mixture models in the local neighbourhood of embeddedword vectors for query performance prediction, Inf. Process. Manage. 56 (3) (2019) 1026–1045. doi:10.1016/j.ipm.2018.

20

10.009.URL https://doi.org/10.1016/j.ipm.2018.10.009

[57] D. Roy, D. Ganguly, M. Mitra, G. J. Jones, Word vector compositionality based relevance feedback using kernel densityestimation, in: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management,CIKM 16, Association for Computing Machinery, New York, NY, USA, 2016, p. 12811290. doi:10.1145/2983323.2983750.URL https://doi.org/10.1145/2983323.2983750

[58] B. Mitra, F. Diaz, N. Craswell, Learning to match using local and distributed representations of text for web search,in: Proceedings of the 26th International Conference on World Wide Web, WWW 17, International World Wide WebConferences Steering Committee, Republic and Canton of Geneva, CHE, 2017, p. 12911299. doi:10.1145/3038912.

3052579.URL https://doi.org/10.1145/3038912.3052579

[59] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, J. Dean, Distributed representations of words and phrases andtheir compositionality, in: C. J. C. Burges, L. Bottou, Z. Ghahramani, K. Q. Weinberger (Eds.), Advances in NeuralInformation Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedingsof a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, 2013, pp. 3111–3119.URL http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality

[60] K. Deb, S. Agrawal, A. Pratap, T. Meyarivan, A fast and elitist multiobjective genetic algorithm: NSGA-II, IEEE Trans.Evolutionary Computation 6 (2) (2002) 182–197. doi:10.1109/4235.996017.URL https://doi.org/10.1109/4235.996017

[61] J. Manotumruksa, C. MacDonald, I. Ounis, University of glasgow terrier team at TREC 2016: Experiments in contextualsuggestions, in: E. M. Voorhees, A. Ellis (Eds.), Proceedings of The Twenty-Fifth Text REtrieval Conference, TREC 2016,Gaithersburg, Maryland, USA, November 15-18, 2016, Vol. Special Publication 500-321, National Institute of Standardsand Technology (NIST), 2016.URL http://trec.nist.gov/pubs/trec25/papers/uogTr-CX.pdf

[62] M. Zhang, Z. Zhou, ML-KNN: A lazy learning approach to multi-label learning, Pattern Recognit. 40 (7) (2007) 2038–2048. doi:10.1016/j.patcog.2006.12.019.URL https://doi.org/10.1016/j.patcog.2006.12.019

[63] C. J. Burges, From ranknet to lambdarank to lambdamart: An overview, Tech. Rep. MSR-TR-2010-82 (June 2010).URL https://www.microsoft.com/en-us/research/publication/from-ranknet-to-lambdarank-to-lambdamart-an-overview/

[64] G. Kalamatianos, A. Arampatzis, Recommending points-of-interest via weighted knn, rated rocchio, and borda countfusion, in: E. M. Voorhees, A. Ellis (Eds.), Proceedings of The Twenty-Fifth Text REtrieval Conference, TREC 2016,Gaithersburg, Maryland, USA, November 15-18, 2016, Vol. Special Publication 500-321, National Institute of Standardsand Technology (NIST), 2016.URL http://trec.nist.gov/pubs/trec25/papers/DUTH-CX.pdf

[65] S. H. Hashemi, J. Kamps, N. O. Amer, Neural endorsement based contextual suggestion, in: E. M. Voorhees, A. Ellis (Eds.),Proceedings of The Twenty-Fifth Text REtrieval Conference, TREC 2016, Gaithersburg, Maryland, USA, November 15-18, 2016, Vol. Special Publication 500-321, National Institute of Standards and Technology (NIST), 2016.URL http://trec.nist.gov/pubs/trec25/papers/Uamsterdam-CX.pdf

[66] M. Aliannejadi, I. Mele, F. Crestani, Venue appropriateness prediction for contextual suggestion, in: E. M. Voorhees,A. Ellis (Eds.), Proceedings of The Twenty-Fifth Text REtrieval Conference, TREC 2016, Gaithersburg, Maryland, USA,November 15-18, 2016, Vol. Special Publication 500-321, National Institute of Standards and Technology (NIST), 2016.URL http://trec.nist.gov/pubs/trec25/papers/USI-CX.pdf

21