Carrots for Couch Potatoes: Improving recommendations by motivating the crowd

Carrots for Couch PotatoesImproving recommendations by

motivating the crowd

@fabianabel

Definition 1“Recommender system = black box that knows the answer to the ultimate question…of life, the universe and everything.”

Hypothesis 1 “The more obscure a recommender system, the higher the chance that its users are happy with the system.”

Definition 2.1 “Data Scientist = folks that can program the smartest recommender systems.”

Hypothesis 2 “Nobody needs an interaction designer.”

Definition 2.2 “Interaction Designer = folks that think about what users actually want to do.”

Definition 3 “Couch potatoes = users who do not provide input to a recommender system, but have high expectations towards the quality of the system.”

Hypothesis 3 “The quality of a recommender system increases with the number of couch potatoes that are *using* the system.”

Goals of Recommender Systems

Make users happy and surprise them with new and relevant content.

[user perspective]

Deliver content so that monetary success of the business is maximized.

[business perspective]

Problem space

Challenges

• Understanding the users

• Understanding the items

• Coding a good (ensemble of) recommendation algorithm(s)

• Evaluation

• Presentation of recommendations

• …

recommender

system

users

items

recommender system

Example from xing.com

Delete item

Hide entire box

(1) Less-Like-This(2) Collaborative

filtering

deletions?

Clicks + bookmarks

(1) More-Like-This(2) Collaborative

filtering

positive feedback

interactions exploited by…

negative feedback

AB test* resultsC

TR

Control group

Group with Less-Like-This

filtering

-3%

?

*AB tests on XING- are done in front-end and back-end

components- typically 50:50 random splits (others:

specific groups; inter-leaving)- Run for days to weeks significance

level: p-value < 0.01- Validation includes AA comparison,

BA/BA test, repeating AB test

More cookies!

People used “delete” to get more recommendations.

Hypothesis 2 is wrong!

Hypothesis 2 “Nobody needs an interaction designer.”

How can we collect more valuable explicit feedback from

our couch potatoe users?

Related Work

1. First explicit feedback is collected right from the beginning during on-boarding, e.g.: select 3 favorites rate 10 items (5-star rating scale)

2. Continuous collection of explicit feedback user control, e.g.: ratings (lightweight) revising ratings, taste preference questionnaire (advanced)

3. Understanding why a user liked or disliked an item, e.g.: emphasizing topics blocking topics

Explicit feedback is key!

1. From the beginning2. Continuously 3. Understanding why…

Hypothesis 3 is wrong!

Hypothesis 3 “The quality of a recommender system increases with the number of couch potatoes that are *using* the system.”

We need to motivate our couch potatoes to contribute to improve

our recommender sytems!

Feedback app

Only means of motivation:1. Promise: “this will enhance your recommendations”2. Progress bar

Feedback App

Pros• Continuous stream of

explicit feedback

• Decoupled from the actual system

Cons• Attracts “Haters” more than

fans

• Decoupled from the actual system

In addition, we also want feedback mechanisms that are more integrated into the natural interaction flow of the system.

explanations feedback

Is this job for me?

Is Hypothesis 1 wrong as well?

Hypothesis 1 “The more obscure a recommender system, the higher the chance that its users are happy with the system.”

Conclusions

Thank you!

@fabianabel