Adapting Recommendation Diversity to Openness to Experience: A Study of Human BehaviourNava Tintarev, Matt Dennis and Judith Masthoff

University of Aberdeen

UMAP’2013. Rome, Italy

2

Outline

Personality and recommender systems

Experiment – openness to experience and diversity

Results

Limitations

Implications for recommender systems design

UMAP’2013. Rome, Italy

3

Personality traits

Generally it is assumed that: a) traits are relatively stable over time, b) traits differ among individuals (for instance, some people like to

try new things while others prefer to stick to known options), and c) traits influence behaviour (e.g. ordering familiar food at a

restaurant).

Five factor model (Big Five): Extraversion, Agreeableness, Conscientiousness, Emotional Stability, and Openness to Experience

Openness to Experience: active imagination, aesthetic sensitivity, attentiveness to inner feelings, preference for variety, and intellectual curiosity

UMAP’2013. Rome, Italy

4

Personality + Recsys == TRUE?

Recommendation is not all about accuracy.

The tailoring of recommender systems to personality has been found to improve accuracy for sparse data sets and

new users (Hu and Pu 2011), to help predict choices for presidential

candidates (Nunes 2008), to positively impact the acceptance of a

system and recommendations (Hu and Pu 2009, Wu et al 2013)

Openness to experience may make users more receptive to more diverse and potentially serendipitous recommendations.

UMAP’2013. Rome, Italy

5

So what makes for “good” diversity

UMAP’2013. Rome, Italy

6

Diversity != Serendipity

Unexpected and helpful (Ge et al 2010)

Topic diversification approach based on taxonomy-based dissimilarity (Ziegler et al. ,2005). Impacted accuracy negatively.

Re-rank a list of top items was found to improve diversity without a great loss in accuracy (Adomavicius and Kwon, 2011)

Users preferred recommendations from a diversified set of clusters (categorical diversity?), rather than within clusters (thematic diversity?). (Abbassi et al., 2012)

UMAP’2013. Rome, Italy

7

Research Questions

1) there may be a difference in preference for the degree of diversity in recommendations among users,

and

2) within category vs across category diversity in recommendations has not received a great deal of weight in previous literature and would benefit from empirical testing with users.

UMAP’2013. Rome, Italy

8

Experiment: User-as-wizard

UMAP’2013. Rome, Italy

9

Experiment - participants

Amazon Mechanical Turk

120 participants (128 excluded)

57% female, 41% male, 2% undisclosed

Openness to Experience within range for the normal population (TIPI).

Average completion time 5 minutes (up to 30)

UMAP’2013. Rome, Italy

10

Meet Oliver (Dennis et al 2010)

openness_low

Oliver is not interested in abstract ideas, as he has difficulty understanding them. He does not like art, and dislikes going to art galleries. He avoids philosophical discussions. He tends to vote for conservative political candidates. He does not like poetry and rarely looks for a deeper meaning in things. He believes that too much tax money goes to supporting artists. He is not interested in theoretical discussions. Oliver is quite a nice person, and tends to enjoy talking with people.

openness_high

Oliver believes in the importance of art and has a vivid imagination. He tends to vote for liberal political candidates. He enjoys hearing new ideas and thinking about things. He enjoys wild flights of fantasy, getting excited by new ideas.

baseline (no story)

UMAP’2013. Rome, Italy

11

Procedure

Recommend three items (books)

Vary along three dimensions author (0,1)

Same or different genre (0,0.3, 1)

Same, similar or different themes (0,0.3,1)

Almost all themes in common, some themes in common, or no themes in common

Had to justify their choice before moving on to the next recommendation.

UMAP’2013. Rome, Italy

12

UMAP’2013. Rome, Italy

13

Results

UMAP’2013. Rome, Italy

14

Results

No statistically significant effect of story

Effect of order in sequence

But a tendency toward a difference in application of thematic and categorical diversity

UMAP’2013. Rome, Italy

15

Effect of story

Condition DiversityAvg DiversityMax

baseline 1.42 (0.31) 2.00 (0.42)

openness_low 1.41 (0.38) 2.08 (0.51)

openness_high 1.46 (0.30) 2.14 (0.44)

overall 1.44 (0.33) 2.08 (0.46)

• 2-3 things changed!• Slightly higher for openness_high• Difference not reliable – possible ceiling effect• No correlation between the participants (aggregated TIPI

score on) openness to experience and diversity.

UMAP’2013. Rome, Italy

16

Order

Condition Book1 Book2 Book3

baseline 1.04 (0.58) 1.56 (0.49) 1.67 (0.67)

openness_low 1.18 (0.68) 1.35 (0.66) 1.72 (0.74)

openness_high 1.33 (0.63) 1.38 (0.61) 1.68 (0.72)

overall 1.18 (0.64) 1.43 (0.60) 1.69 (0.71)

Book2 >div Book2Book3 >div Book3

p < 0.01 (Bonferroni corrected)

Openness_low starts lowest but `catches up’ by Book3!

UMAP’2013. Rome, Italy

17

Ways of applying diversity

Condition Author Genre Theme

baseline 1.92 (0.70) 1.22 (0.70) 1.12 (0.69)

openness_low 1.83 (0.71) 1.17 (0.63) 1.25 (0.78)

openness_high 1.88 (0.75) 1.45 (0.58) 1.06 (0.49)

overall 1.88 (0.72) 1.28 (0.64) 1.14 (0.66)

• Tendency toward more genre diversity for openness_high• Tendency toward more theme diversity for openness_low• We need to repeat this study!

UMAP’2013. Rome, Italy

18

Results (again)

No statistically significant effect of story

Effect of order in sequence

But a tendency toward a difference in application of thematic and categorical diversity

UMAP’2013. Rome, Italy

19

Limitations

Domain

Is this what people need or what people do?

Predicting openness to experience

UMAP’2013. Rome, Italy

20

Possible implications for recommender systems design

Start narrow go broader

Shift focus toward more thematic diversity (within cluster) for people who are low on openness to experience.

Diversity across genres (across clusters) is still relevant for the majority of users

Worth looking into predicting openness to experience

UMAP’2013. Rome, Italy

21

Wrap-up

We studied the effect of openness to experience on rec diversity

People like to expand each other’s horizons

They start narrow and go wide

Do not really consider personality

But tends toward more thematic diversity for low OE vs more categorical diversity for low OE.

UMAP’2013. Rome, Italy

22

Questions?

n.tintarev@abdn.ac.uk

This research has been funded by the Engineering and Physical Sciences Research Council (EPSRC, UK), grant ref. EP/J012084/1