Ryen W. White, Peter Bailey, Liwei ChenMicrosoft Corporation

MotivationInformation behavior is embedded in external context

Context motivates the problem, influences interactionIR community theorized about context

Context sensitive search, user studies of search context

User interest models can enhance post-query behavior & general browsing by leveraging contextual info.e.g., personalization, information filtering, etc.

Little is known about the value of different contextual sources for user interest modeling

OverviewA systematic, log-based study of five contextual sources

for user interest modeling during Web interactionAssume user has browsed to URLEvaluate the predictive value of five contexts of URL:

Interaction: recent interactions preceding URLCollection: pages that link to URLTask: pages sharing search engine queries with URLHistoric: long term interests of current userSocial: combined long-term interests those who visit URL

Domain is website recommendation not search results

Data SourcesAnonymized URLs visited by users of a widely-

distributed browser toolbar4 months of logs (Aug 08 – Nov 08 inclusive):

Past: Aug-Sep used to create user historiesPresent: Oct-Nov used for current behavior and

future interests250K users randomly selected from a larger

user pool once most active users (top 1%) were removedChosen users with at least 100 page visits in Past

Trails and Terminal URLsFrom logs we extracted millions of browse trails

Temporally-ordered sequence of URLs comprising all pages visited by a user per Web browser instance

Terminate with 30-minute inactivity timeout

A set of 5M terminal URLs (ut) obtained by randomly-sampling all URLs in the trailsTerminal URLs demarcate past and future events

Task = Learn user interest models from contexts for ut, use those models to predict future user interests

Building User Interest ModelsClassified context URLs in the Open Directory

Project human-edited Web directory (ODP, dmoz.org)

Automatically assigned category labels via URL matchURL back-off used if no exact match obtained

Represent interests as list of ODP category labelsLabels ranked in descending order by frequencyFor example, for a British golf enthusiast, the top of

their user interest profile might resemble:ODP Category Labels FrequencySports/Golf/Courses/Europe/United Kingdom 102Sports/Golf/Driving Ranges 86Sports/Golf/Instruction/Golf Schools 63

Selecting ContextsIngwersen and Järvelin (2005) developed

nested model of context stratification representing main contextual influences on people engaged in information behavior

Dimensions usedOthers challenging to

model via logse.g., cognitive and

affective state, infra- structures, etc.

Defining ContextsNone (ut only): Interest model for terminal URL

Interaction (ut-5 … ut-1): Interest model for five Web pages immediately preceding ut

Task: Interest model for pages encountered during the same or similar tasksWalk on search engine click

graph from ut to queries and then back out to pages

Defining ContextsCollection: Interest model pages linking to ut

We obtained a set of in-links for each ut from a search engine index, built model from pages linking to ut

Historic: Interest model for each user based on their long-term Web page visit history

Social: Interest model from combination of the historic contexts of users that also visit ut

What is the effectiveness of different context sources for user interest modeling?

MethodologyFound instances of ut in Present set (Oct-Nov 08

logs)Used all actions after ut as source of future behavior

Futures specific to each user and each ut

Used to gauge predictive value of each contextCreated three interest models representing future

interests (ranked list of ODP labels & frequencies):Short: within one hour of ut

Medium: within one day of ut

Long: within one week of ut

Filtered {ut} to help ensure experimental integritye.g., no more than 10 ut per user

MethodologyDivided filtered {ut} into 10 equally-sized runs

Each run contained at most one ut from each user

Experimental procedure:For each ut in each run:

Build ground truth for short-, medium-, and long-term future interest models

Build interest models for different contexts (and combinations)

Determine predictive accuracy of each model

Used five measures to determine prediction accuracyP@1, P@3, Mean Reciprocal Rank, nDCG, and F1F1 tracked well with others - focus on that here

Findings – Context comparisonPredictive performance of contextual sources for different futures

Interaction context & Task context most predictive of short-term interests

Task context most predictive of medium-term interests

Historic context most predictive of long-term interests

Findings – Handling near missesNear miss between prediction and ground truth regarded as

total miss Use one/two/three-level back-off on both ground truth and

prediction

No back-offBack-off to top two ODP levels

Findings – Improved confidence

No back-offPredicted and ground truth labels based on ≥ 5 pages

Basing predictions & ground truth on small # page visits may skew results Repeat experiment & ignore labels based on < 5 page visits

Findings – Combining contexts

Overlap beats single contextual sourcesKey contexts still important

Short = Interaction (i) and Task (t)Medium = Task (t)Long = Historic (h)

Supports polyrepresentation theory (Ingwersen, 1994)Overlap between sources boosts predictive accuracy

RankShort Medium Long

SourcesF1 score

SourcesF1

scoreSources

F1 score

1 n, i, t, h, s, c

0.72** n, i, t, h, s, c 0.53** n, i, t, s, h, c 0.45**

2 n, i, s, h, c 0.71** n, i, t, h, c 0.52** n, i, s, h, c 0.43**

3 n, i, t, h, c 0.71** n, i, t 0.49** n, i, t, h, c 0.43*

4 n, i, h, c 0.71** n, i, s, h, c 0.48* s, h 0.43*

5 n, i, s, t, c 0.69** n, i, h, t 0.48* n, i, s, h, t 0.42*

Summary of FindingsPerformance of context dependent on distance between

ut and end of prediction windowShort-term interests predicted by task/interaction

contexts Topical interest may not be highly dynamic, even if queries and

information needs areMedium-term interests best predicted by task context

More likely to include task variants appearing in next dayLong-term interests predicted by historic/social contexts

Interest may be invariant over time, users visiting same pages may have similar interests

Overlap effective - many contexts reinforce key interests

Conclusions and Take-awaySystematic study of context for user interest modelingStudied predictive value of five context sources

Value varied with duration of predictionShort: interaction/task, Medium: task, Long:

historic/socialOverlap was more effective than any individual sourceSource must be tailored to modeling taskSearch/recommendation systems should not treat all

contextual sources equallyWeights should be assigned to each source based on the

nature of the prediction task