Data Mining - Volinsky - 2011 - Columbia University 1 Topic 12 – Recommender Systems and the Netflix Prize

Data Mining - Volinsky - 2011 - Columbia University 1

Topic 12 – Recommender Systems and the Netflix Prize

Outline

• Intro to Recommender Systems• The Netflix Prize – structure and

history• Models for Recommendation• Extensions of model for other

domains


Recommender Systems

• Systems which take user preferences about items as input and outputs recommendations

• Early examples• Bellcore Music Recommender (1995)• MIT Media Lab: Firefly (1996)

Best example: Amazon.comWorst example: Amazon.com

Also:NetflixeBayGoogle ReaderiTunes Geniusdigg.comHulu.com

3Data Mining - Volinsky - 2011 - Columbia University

Recommender Systems

• Basic idea– recommend item i to user u for the purpose of

• Exposing them to something they would not have otherwise seen• Leading customers to the Long Tail• Increasing customers’ satisfaction

• Data for recommender systems (need to know who likes what)– Purchase/rented– Ratings– Web page views

– Which do you think is best?


Recommender Systems

• Two types of data:• Explicit data: user provides information about their

preferences– Pro: high quality ratings– Con: Hard to get: people cannot be bothered

• Implicit data: infer whether or not user likes product based on behavior– Pro: Much more data available, less invasive– Con: Inference often wrong (does purchase imply preference?)

• In either case, data is just a big matrix – Users x items– Entries binary or real-valued

• Biggest Problem:– Sparsity: most users have not rated most products.


1 3 5 5 4

5 4 4 2 1 3

2 4 1 2 3 4 3 5

2 4 5 4 2

4 3 4 2 2 5

1 3 3 2 4

Recommender Systems: ModelsTwo camps on how to make recommendations:

– Collaborative Filtering (CF)• Use collective intelligence from all available rating information to make

predictions for individuals• Depends on the fact that user tastes are correlated and commutative:• If Alice and Bob both like X and Alice likes Y then Bob is more likely to

like Y

– Content based• Extracts “features” from items for a big regression or rule-based model• See www.nanocrowd .com

– 15 years of research in the field– Conventional wisdom:

• CF performs better when there is sufficient data• Content-based is useful when there is little data


Recommender Systems: Evaluation

• Narrow focus on accuracy sometimes misses the point– Prediction Diversity, Prediction Context, Order of

predictions

– For explicit ratings: use training – test setup and use some form of MSE score

• OK, but treats all ratings the same (care most about top k)• Doesn’t reflect interventional impact of recommender

– For implicit, we don’t have a score to calculate MSE.• Can still do training test• Create a ranked vector of preference – really only interested

in top-k for a recommendation algorithm• use “discounted cumulative gain”

– Gives more weight to those predicted at the top


Evaluation: Discounted Cumulative Gain

• Recommending stories on a web site:– Ten stories to recommend in test set: ABCDEFGHIJ– Only room on page for 5 recommendations– Algorithm 1 ranks: AECHI– Algorithm 2 ranks: CDBFG– User actually viewed AEBFGJ

– rel = {0,1}

– DCG1= 1+1/log2(2)+0+0+0 = 2

– DCG2=0+0+1/log2(3)+1/log2(4)+1/log2(5) = 1.56


The Netflix Prize


Netflix

• A US-based DVD rental-by mail company• >10M customers, 100K titles, ships 1.9M DVDs per day

Good recommendations = happy customers


Netflix Prize • October, 2006:

• Offers $1,000,000 for an improved recommender algorithm

•Training data• 100 million ratings• 480,000 users• 17,770 movies• 6 years of data: 2000-2005

• Test data• Last few ratings of each user (2.8 million)• Evaluation via RMSE: root mean squared error • Netflix Cinematch RMSE: 0.9514

• Competition• $1 million grand prize for 10% improvement• If 10% not met, $50,000 annual “Progress Prize”

for best improvement

user movie score date

1 21 1 2002-01-03

1 213 5 2002-04-04

2 345 4 2002-05-05

2 123 4 2002-05-05

2 768 3 2003-05-03

3 76 5 2003-10-10

4 45 4 2004-10-11

5 568 1 2004-10-11

5 342 2 2004-10-11

5 234 2 2004-12-12

6 76 5 2005-01-02

6 56 4 2005-01-31

user movie score date

1 212 ? 2003-01-03

1 1123 ? 2002-05-04

2 25 ? 2002-07-05

2 8773 ? 2002-09-05

2 98 ? 2004-05-03

3 16 ? 2003-10-10

4 2450 ? 2004-10-11

5 2032 ? 2004-10-11

5 9098 ? 2004-10-11

5 11012 ? 2004-12-12

6 664 ? 2005-01-02

6 1526 ? 2005-01-31


Netflix Prize

• Competition design

•Hold-out set created by taking last 9 ratings for each user

– Non-random, biased set

• Hold-out set split randomly three ways:

– Probe Set – appended to training data to allow unbiased estimation of RMSE

– Submit ratings for the (Quiz+Test) Sets – Netflix returns RMSE on the Quiz Set

only

– Quiz Set results posted on public leaderboard, but Test Set used to determine the winner!

» Prevents overfitting12Data Mining - Volinsky - 2011 - Columbia University

Data Characteristics

0

5

10

15

20

25

30

35

40

1 2 3 4 5

Rating

Perc

en

tag

e

Training (m = 3.60)

Probe (m = 3.67)


Ratings per movie/user

User ID # Ratings Mean Rating

305344 17,651 1.90

387418 17,432 1.81

2439493 16,560 1.22

1664010 15,811 4.26

2118461 14,829 4.08

1461435 9,820 1.37

Avg #ratings/user: 208

Avg #ratings/movie: 5627


Data Characteristics

• Most Loved MoviesMost Loved Movies Avg rating Count

The Shawshank Redemption 4.593 137812

Lord of the Rings: The Return of the King 4.545 133597

The Green Mile 4.306 180883

Lord of the Rings: The Two Towers 4.460 150676

Finding Nemo 4.415 139050

Raiders of the Lost Ark 4.504 117456

Most Rated MoviesMiss Congeniality

Independence Day

The Patriot

The Day After Tomorrow

Pretty Woman

Pirates of the Caribbean

Highest VarianceThe Royal Tenenbaums

Lost In Translation

Pearl Harbor

Miss Congeniality

Napolean Dynamite

Fahrenheit 9/11


The competition progresses…

• Cinematch beaten in two weeks

• Halfway to 10% in 6 weeks• Our team, BellKor (Bob Bell

and Yehuda Koren) took over the lead in the summer…

• With 48 hours to go to the $50K Progress Prize, we had a comfortable lead…with another submission in pocket 16Data Mining - Volinsky - 2011 - Columbia University

8pm6am10/18pm

Leaderboard05:00 pm

Sept 30


Leaderboard06:00 pm

Sept 30

wanna split 50/50?


8pm6am10/18pm

• ARRRRGH! We have one more chance….


A Nervous Last Day

• Start in virtual tie on Quiz data• Unclear who leads on Test data

• Can Gravity/Dinosaurs improve again?• More offers for combining• Can we squeeze out a few more points?

• Improved our mixing strategy• Created a second team, KorBell, just to be safe


8pm6am10/18pm

Our final submission(s)…



So The DramaTimeline:

– 2008:• BellKor merges with BigChaos to win 2nd $50K Progress Prize

(9.4%)

– 2009:• June 26: BellKor, Big Chaos and Pragmatic Theory merge

passing 10% threshold, begins 30 day ‘last call’ period. • July 24: All is quiet. We crawl up to 10.08%...• July 25: The Ensemble, a coalition of 23 independent teams,

combine to pass us on the public leaderboard. • July 26: Bellkor pulls back into a tie. Then Ensemble pulls

ahead by 0.01 %• July 26: Netflix Prize ends: winner (determined on Test Set) is

unclear.• September 21: Netflix announces BellKor’s Pragmatic Chaos as

winner of $1M prize.


So The Drama

Timeline:– 2009:

• June 26: BellKor, Big Chaos and Pragmatic Theory merge passing 10% threshold, begins 30 day ‘last call’ period.

• Lead next best team by 0.37%• Only two teams within 0.80%

– But soon, the cavalry charges• Mega-mergers start to form.

– July 25 (25 hours left):• We crawl up to 10.08%...• The Ensemble, a coalition of 23 independent

teams, combine to pass us on the public leaderboard



26

Test Set Results

• BellKor’s Pragmatic Theory: 0.8567• The Ensemble: 0.8567• Tie breaker was submission date/time

• We won by 20 minutes!

But really:• BellKor’s Pragmatic Theory: 0.856704• The Ensemble: 0.856714

• Also, a combination of BPC (10.06%) and Ensemble (10.06%) scores results in a 10.19% improvement!

Data Mining - Volinsky - 2011 - Columbia University

Our Approach

• Our Prize winning solutions were an ensemble of many separate solution sets

• Progress Prize 2007: 103 sets• Progress Prize 2008 (w/Big Chaos): 205 sets• Grand Prize 2009 (w/ BC and Pragmatic Theory): >

800 sets!!

– We used two main classes of models• Nearest Neighbors• Latent Factor Models (via Singular Value

Decomposition)• Also regularized regression, not a big factor• Teammates used neural nets and other methods• Approaches mainly algorithmic, not statistical in nature


Data representation (excluding dates)

1 2 3 4 5 6 7 8 9 10 11 12

1 1 3 5 5 4

2 5 4 4 2 1 3

3 2 4 1 2 3 4 3 5

4 2 4 5 4 2

5 4 3 4 2 2 5

6 1 3 3 2 4

users

mo

vies

-unknown rating -rating between 1 to 5


Nearest Neighbors


Nearest Neighbors

1 2 3 4 5 6 7 8 9 10 11 12

1 1 3 5 5 4

2 5 4 4 2 1 3

3 2 4 1 2 3 4 3 5

4 2 4 5 4 2

5 4 3 4 2 2 5

6 1 3 3 2 4

users

mo

vies

-unknown rating -rating between 1 to 5


Nearest Neighbors

1 2 3 4 5 6 7 8 9 10 11 12

1 1 3 ? 5 5 4

2 5 4 4 2 1 3

3 2 4 1 2 3 4 3 5

4 2 4 5 4 2

5 4 3 4 2 2 5

6 1 3 3 2 4

users

mo

vies

-estimate rating of movie 1 by user 5


Nearest Neighbors

1 2 3 4 5 6 7 8 9 10 11 12

1 1 3 ? 5 5 4

2 5 4 4 2 1 3

3 2 4 1 2 3 4 3 5

4 2 4 5 4 2

5 4 3 4 2 2 5

6 1 3 3 2 4

users

Neighbor selection:Identify movies similar to 1, rated by user 5

mo

vies


Nearest Neighbors

1 2 3 4 5 6 7 8 9 10 11 12

1 1 3 ? 5 5 4

2 5 4 4 2 1 3

3 2 4 1 2 3 4 3 5

4 2 4 5 4 2

5 4 3 4 2 2 5

6 1 3 3 2 4

users

Compute similarity weights:s13=0.2, s16=0.3

mo

vies


Nearest Neighbors

1 2 3 4 5 6 7 8 9 10 11 12

1 1 3 2.6 5 5 4

2 5 4 4 2 1 3

3 2 4 1 2 3 4 3 5

4 2 4 5 4 2

5 4 3 4 2 2 5

6 1 3 3 2 4

users

Predict by taking weighted average:(0.2*2+0.3*3)/(0.2+0.3)=2.6

mo

vies


Nearest Neighbors

– To predict the rating for user u on item i:• Use similar users’ ratings for similar movies:

rui = rating for user u and item i

bui= baseline rating for user u and item I

sij = similarity between items i and j

N(i,u) = neighborhood of item i for user u (might be fixed at k)


Nearest Neighbors

• Useful to “center” the data, and model residuals

• What is sij ???– Cosine distance– Correlation

• What is N(i,u)??– Top-k– Threshold

• What is bui

• How to deal with missing values?• Choose several different options and throw

them in! 36Data Mining - Volinsky - 2011 - Columbia University

Nearest Neighbors, cont• This is called “item-item” NN

– Can also do user-user– Which do you think is better?

• Advantages of NN– Few modeling assumptions– Easy to explain to users– Most popular RS tool


Nearest Neighbors, Modified

• Problem with traditional k-NN:• Similarity weights are calculated globally, and• do not account for correlation among the neighbors

– We estimate the weights (wij) simultaneously via a least squares optimization :

Basically, a regression using the ratings in the nbhd. – Shrinkage helps address correlation– (don’t try this at home)


Geared towards females

Geared towards

males

serious

escapist

The PrincessDiaries

The Lion King

Braveheart

Lethal Weapon

Independence Day

AmadeusThe Color Purple

Dumb and Dumber

Ocean’s 11

Sense and Sensibility

Latent factor models – Singular Value Decomposition


SVD finds concepts

Matrix Decomposition - SVD


1 3 5 5 4

5 4 4 2 1 3

2 4 1 2 3 4 3 5

2 4 5 4 2

4 3 4 2 2 5

1 3 3 2 4

item

s

.1 -.4 .2

-.5 .6 .5

-.2 .3 .5

1.1 2.1 .3

-.7 2.1 -2

-1 .7 .3

1.1 -.2 .3 .5 -2 -.5 .8 -.4 .3 1.4 2.4 -.9

-.8 .7 .5 1.4 .3 -1 1.4 2.9 -.7 1.2 -.1 1.3

2.1 -.4 .6 1.7 2.4 .9 -.3 .4 .8 .7 -.6 .1

~

~

item

s

users

users

?

D3

Example with 3 factors (concepts

Each user and each item is described by a feature vector across concepts

Factorization-based modeling

1 3 5 5 4

5 4 4 2 1 3

2 4 1 2 3 4 3 5

2 4 5 4 2

4 3 4 2 2 5

1 3 3 2 4

.1 -.4 .2

-.5 .6 .5

-.2 .3 .5

1.1 2.1 .3

-.7 2.1 -2

-1 .7 .3

1.1 -.2 .3 .5 -2 -.5 .8 -.4 .3 1.4 2.4 -.9

-.8 .7 .5 1.4 .3 -1 1.4 2.9 -.7 1.2 -.1 1.3

2.1 -.4 .6 1.7 2.4 .9 -.3 .4 .8 .7 -.6 .1~

• This is a strange way to use SVD!– Usually for reducing dimensionality, here for filling

in missing data!– Special techniques to do SVD w/ missing data

• Alternating Least Squares = variant of EM algorithms

• Probably most popular model among contestants– 12/11/2006: Simon Funk describes an SVD based method


Latent Factor Models, Modified

• Problem with traditional SVD:– User and item factors are determined globally– Each user described as a fixed linear combination across factors– What if there are different people in the household?

• Let the linear combination change as a function of the item rated.

• Substitute pu with pu(i), and add similarity weights

• Again, don’t try this at home!


First 2 Singular Vectors


Incorporating Implicit Data

• Implicit Data: what you choose to rate is an important, and separate piece of information than how you rate it.

• Helps incorporate negative information, especially for those users with low variance.

• Can be fit in NN or SVD


Temporal Effects

It took us 1.5 years to figure out how to use the dates!

Account for temporal effects in two ways

• Directly in the SVD model – allow user factor vectors pu to vary with time pu(t)

– Many ways to do this, adds many parameters, need regularization

– DTTAH

• Observed: number of user ratings on a given date is a proxy for how long ago the movie was seen:

– Some movies age better than others


46

Memento vs Patch Adams

ratin

gratin

g

frequency

frequency

Data Mining - Volinsky - 2011 - Columbia University

Netflix Prize: Other Topics

• Ensembles of models• Application to TV data


The Power of Ensembles

• Ensembles of models• Over 800 in Grand Prize solution!• Some of our “models” were blends of other models!

– or models on residuals of other models– Why: because it worked

• Black Box magic at its worst• However, mega blends are not needed in practice

– Best model: complex model with implicit and explicit data, and time varying coefficients, millions of parameters and regularization.

– This did as well as our 2007 Progress Prize winning combo of 107 models!

– Even a small handful of ‘simple’ models gets you 80-85% of the way

• In practice, .01% does not matter much

Courtesy LingPipe Blog


Blending Models

• Figuring out the right way to blend models was an important piece of our team strategy

• Subsequent slides courtesy of Michael Jahrer (BigChaos)…

• Keep in mind…


Probe Blending < Quiz Blending• A few facts before we go on:

1)Every model I can be reduced to its prediction vector on the probe set (pi). That can be compared to the true probe answers (r)

2)The model can also be applied to the quiz set (qi), but can only be compared to the true answers (s) by submission

3)We submitted all individual models to get the RMSE for every model i.

4)The distribution of the quiz vector (how many 1s, 2s, 3s, etc) was well known. This gives the variance of the vector


Probe Blending


Quiz Blending


Documents

Data Mining - Volinsky - 2011 - Columbia University 1 Topic 12 – Recommender Systems and the Netflix Prize