[WI 2017] Context Suggestion: Empirical Evaluations vs User Studies

Context Suggestion: Empirical Evaluations vs User Studies

Yong ZhengSchool of Applied Technology

Illinois Institute of TechnologyChicago, IL, 60616, USA

The 2017 IEEE/WIC/ACM Conference on Web Intelligence (WI)August 23-26, 2017, Leipzig, Germany

Agenda

• Intro: Context-aware Recommender Systems

• Motivations: Context Suggestion

• Methodologies and Research Problems

– Direct Context Prediction

– Indirect Context Suggestion

• Experimental Results and Findings

• Conclusions and Future Work

2

Agenda








3

Recommender System (RS)

• RS: item recommendations tailored to user tastes

4

Context-Aware Recommendation

5

Companion

User’s decision may vary from contexts to contexts

• Examples:➢ Travel destination: in winter vs in summer

➢ Movie watching: with children vs with partner

➢ Restaurant: quick lunch vs business dinner

➢ Music: for workout vs for study

Terminology in CARS

6

• Example of Multi-dimensional Context-aware Data set

➢Context Dimension: time, location, companion

➢Context Condition: Weekend/Weekday, Home/Cinema

➢Context Situation: {Weekend, Home, Kids}

User Item Rating Time Location Companion

U1 T1 3 Weekend Home Kids

U1 T2 5 Weekday Home Partner

U2 T2 2 Weekend Cinema Partner

U2 T3 3 Weekday Cinema Family

U1 T3 ? Weekend Cinema Kids

What is Context?

7

The most common contextual variables:

➢Time and Location

➢User intent or purpose

➢User emotional states

➢Devices

➢Topics of interests, e.g., apple vs. Apple

➢Others: companion, weather, budget, etc

Usually, the selection/definition of contexts is a domain-specific problem

Agenda








8

Motivations: Context-Drive Applications

1) Context is necessary to maximize the user experience. A list of good item recommendations is NOT enough.

2) It is difficult to collect context information on the Web!!!! Context suggestion provides a way for context acquisition

9

Motivation: User Experience

10

• San Diego Zoo • San Diego Zoo Safari Park


11


12

13

Motivation: Context Acquisition

• Google Music

Agenda








14

Solution for Context Suggestion

• Direct Context PredictionThe output is a binary predictionThe value “1” indicates appropriate suggestionThe value “0” tells inappropriate suggestion

• Indirect Context SuggestionThe output is a top-N recommendations

Task: Suggest appropriate contexts for a user to enjoy a given item

15

Direct Context Prediction

16

Treat context conditions as binary labelsUtilize multi-label classification as solution

Indirect Context Suggestion

17

• Context-aware Recommendation

Task:Given a user and context infoRecommend a list of items

Indirect Context Suggestion

18

• Indirect Context Suggestion

Task:Given a user and an itemRecommend a list of appropriatecontexts for the users to enjoythe items

Item-Aware Context Recommendation

Research Problems

• Direct Context Prediction was explored in WI’14,but Indirect Context Suggestion was neverdiscussed and compared with the direct contextprediction

• Previous research infers user preferences on contexts from contextual ratings on the items – it is not validated that whether contextual ratings can tell whether a user prefers a given contexts to enjoy the items

• There are no evaluation standards for context suggestion

19

Agenda








20

Data Collection

• It’s first time to collect user’s tastes on contexts

• 5043 ratings by 97 users on 79 movies

21

Experimental Settings

• 5-fold Cross Validation

• Direct Context Prediction

– Classification Chains (MLC_CC)

– Label Powerset (MLC_LP)

• Indirect Context Suggestion

– Tensor Factorization (TF)

– Context-aware Matrix Factorization (CAMF)

– Contextual Sparse Linear Methods (CSLIM)

22

Evaluation Mechanisms

We propose the evaluation standards for context suggestion

• Top-N Context Prediction

– N varies from 1 to the number of context conditions

– Any available N value is fine

– It does not matter if two contexts from a same variable are suggested. For example, {weekend, weekday, home, kids}

• Exact Context Suggestion

– Top-N, but N = the number of context condition

– For each dimension, we only suggest one conditionFor example, {weekend, home, kids}

23

Results (Top-N Suggestion)

24

By using contextual ratings as ground truth for evaluation purpose

Results (Top-N Suggestion)

25

By using user tastes on contexts (from user surveys) as ground truth

Results (Exact Suggestion)

26

Results (Exact Suggestion)

27

Conclusions and Findings

28

• “General” indicates user’s general preferences on contexts for movie watching, without considering which movie it is

• Personalization is required, since many algorithms outperform the “General” method

• UISplitting and TF are the best ones

• Indirect context suggestion is better to offer better suggestions than direct context prediction

• The results by using contextual ratings and usertastes on contexts are consistent.

Future Work

29

• We will try to collect more data and evaluate these solutions on larger data set

• We will try to utilize the context suggestionmethods to predict emotional states

• We will seek solutions to improve the indirectcontext prediction

Context Suggestion: Empirical Evaluations vs User Studies

Yong ZhengSchool of Applied Technology

Illinois Institute of TechnologyChicago, IL, 60616, USA

The 2017 IEEE/WIC/ACM Conference on Web Intelligence (WI)August 23-26, 2017, Leipzig, Germany