The Netflix Prize Contest

The Netflix Prize Contest1) New Paths to New Machine Learning Science

2) How an Unruly Mob Almost Stolethe Grand Prize at the Last MomentJeff Howbert

February 6, 2012Netflix Viewing Recommendations

Recommender SystemsDOMAIN: some field of activity where users buy, view, consume, or otherwise experience items

PROCESS:users provide ratings on items they have experiencedTake all < user, item, rating > data and build a predictive modelFor a user who hasnt experienced a particular item, use model to predict how well they will like it (i.e. predict rating)Roles of Recommender SystemsHelp users deal with paradox of choice

Allow online sites to:Increase likelihood of salesRetain customers by providing positive search experience

Considered essential in operation of:Online retailing, e.g. Amazon, Netflix, etc.Social networking sites

Amazon.com Product Recommendations

Recommendations on essentially every category of interest known to mankindFriendsGroupsActivitiesMedia (TV shows, movies, music, books)News storiesAd placementsAll based on connections in underlying social network graph and your expressed likes and dislikes Social Network RecommendationsTypes of Recommender SystemsBase predictions on either:

content-based approachexplicit characteristics of users and items

collaborative filtering approachimplicit characteristics based on similarity of users preferences to those of other usersThe Netflix Prize ContestGOAL: use training data to build a recommender system, which, when applied to qualifying data, improves error rate by 10% relative to Netflixs existing system

PRIZE: first team to 10% wins $1,000,000Annual Progress Prizes of $50,000 also possible

The Netflix Prize ContestCONDITIONS:Open to publicCompete as individual or groupSubmit predictions no more than once a dayPrize winners must publish results and license code to Netflix (non-exclusive)

SCHEDULE:Started Oct. 2, 2006To end after 5 yearsThe Netflix Prize ContestPARTICIPATION:51051 contestants on 41305 teams from 186 different countries44014 valid submissions from 5169 different teamsThe Netflix Prize DataNetflix released three datasets480,189 users (anonymous)17,770 moviesratings on integer scale 1 to 5

Training set: 99,072,112 < user, movie > pairs with ratingsProbe set: 1,408,395 < user, movie > pairs with ratingsQualifying set of 2,817,131 < user, movie > pairs with no ratings

Model Building and Submission Process99,072,1121,408,3951,408,7891,408,342training setquiz settest setqualifying set(ratings unknown)probe setMODELtuningratingsknownvalidatemake predictionsRMSE onpublicleaderboardRMSE keptsecret forfinal scoring

Why the Netflix Prize Was HardMassive datasetVery sparse matrix only 1.2% occupiedExtreme variation in number of ratings per userStatistical properties of qualifying and probe sets different from training set

Dealing with Size of the DataMEMORY:2 GB bare minimum for common algorithms4+ GB required for some algorithmsneed 64-bit machine with 4+ GB RAM if seriousSPEED:Program in languages that compile to fast machine code64-bit processorExploit low-level parallelism in code (SIMD on Intel x86/x64)Common Types of AlgorithmsGlobal effectsNearest neighborsMatrix factorizationRestricted Boltzmann machineClusteringEtc.Nearest Neighbors in Action

Identical preferences strong weightSimilar preferences moderate weightMatrix Factorization in Action

< a bunch of numbers >

< a bunch ofnumbers >reduced-ranksingularvaluedecomposition(sort of)+Matrix Factorization in Action

multiply and addfeatures(dot product)for desired< user, movie >prediction+The Power of BlendingError function (RMSE) is convex, so linear combinations of models should have lower errorFind blending coefficients with simple least squares fit of model predictions to true values of probe setExample from my experience:blended 89 diverse modelsRMSE range = 0.8859 0.9959blended model had RMSE = 0.8736Improvement of 0.0123 over best single model13% of progress needed to winAlgorithms: Other Things That MatteredOverfittingModels typically had millions or even billions of parametersControl with aggressive regularization

Time-related effectsNetflix data included date of movie release, dates of ratingsMost of progress in final two years of contest was from incorporating temporal informationThe Netflix Prize: Social PhenomenaCompetition intense, but sharing and collaboration were equally soLots of publications and presentations at meetings while contest still activeLots of sharing on contest forums of ideas and implementation detailsVast majority of teams:Not machine learning professionalsNot competing to win (until very end)Mostly used algorithms published by othersOne Algorithm from Winning Team(time-dependent matrix factorization)

Yehuda Koren, Comm. ACM, 53, 89 (2010)

Netflix Prize Progress: Major MilestonesDATE:Oct. 2007Oct. 2008July 2009WINNER:BellKorBellKor in BigChaos???me, starting June, 2008June 25, 2009 20:28 GMT

June 26, 18:42 GMT BPC Team Breaks 10%

meGenesis of The EnsembleThe Ensemble33 individualsOpera and Vandelay United19 individualsGravity4 indiv.DinosaurPlanet3 indiv.7 other individualsVandelayIndustries5 indiv.OperaSolutions5 indiv.Grand Prize Team14 individuals9 other individualswww.the-ensemble.comJune 30, 16:44 GMT

July 8, 14:22 GMT

July 17, 16:01 GMT

July 25, 18:32 GMT The Ensemble First Appears!

24 hours, 10 minbefore contest ends

#1 and #2 teamseach have onemore submission !

July 26, 18:18 GMTBPC Makes Their Final Submission24 minutes before contest ends

The Ensemble can make one more submission window opens 10 minutes before contest endsJuly 26, 18:43 GMT Contest Over!

Final Test Scores

Final Test ScoresNetflix Prize: What Did We Learn?Significantly advanced science of recommender systemsProperly tuned and regularized matrix factorization is a powerful approach to collaborative filteringEnsemble methods (blending) can markedly enhance predictive power of recommender systems

Crowdsourcing via a contest can unleash amazing amounts of sustained effort and creativityNetflix made out like a banditBut probably would not be successful in most problems

Netflix Prize: What Did I Learn?Several new machine learning algorithmsA lot about optimizing predictive modelsStochastic gradient descentRegularizationA lot about optimizing code for speed and memory usageSome linear algebra and a little PDQEnough to come up with one original approach that actually worked

Money makes people crazy, in both good ways and bad

COST: about 1000 hours of my free time over 13 months

Sheet1movie 1movie 2movie 3movie 4movie 5movie 6movie 7movie 8movie 9movie 10movie 17770user 1123user 22334user 3534user 42322user 54534user 62user 72423user 8344user 93user 10122user 480189433

Sheet2

Sheet3

Sheet1movie 1movie 2movie 3movie 4movie 5movie 6movie 7movie 8movie 9movie 10movie 17770movie 1movie 2movie 3movie 4movie 5movie 6movie 7movie 8movie 9movie 10movie 17770user 1123user 1123user 22334user 22334?user 3534user 353user 42322user 42322user 54534user 5235424user 62user 62user 72423user 7242user 8344user 8313454user 93user 93user 10122user 10122user 480189433user 480189433

Sheet2

Sheet3

Sheet1movie 1movie 2movie 3movie 4movie 5movie 6movie 7movie 8movie 9movie 10movie 17770movie 1movie 2movie 3movie 4movie 5movie 6movie 7movie 8movie 9movie 10movie 17770user 1123user 1123user 22334user 22334?user 3534user 353user 42322user 42322user 54534user 5235424user 62user 62user 72423user 7242user 8344user 8313454user 93user 93user 10122user 10122user 480189433user 480189433

Sheet2

Sheet3

Sheet1movie 1movie 2movie 3movie 4movie 5movie 6movie 7movie 8movie 9movie 10movie 17770movie 1movie 2movie 3movie 4movie 5movie 6movie 7movie 8movie 9movie 10movie 17770user 1123user 1123user 22334user 22334?user 3534user 353user 42322user 42322user 54534user 5235424user 62user 62user 72423user 7242user 8344user 8313454user 93user 93user 10122user 10122user 480189433user 480189433movie 1movie 2movie 3movie 4movie 5movie 6movie 7movie 8movie 9movie 10movie 17770feature 1feature 2feature 3feature 4feature 5feature 1feature 2feature 3feature 4feature 5user 1user 2user 3user 4user 5user 6user 7user 8user 9user 10user 480189

Sheet2

Sheet3

Sheet1movie 1movie 2movie 3movie 4movie 5movie 6movie 7movie 8movie 9movie 10movie 17770movie 1movie 2movie 3movie 4movie 5movie 6movie 7movie 8movie 9movie 10movie 17770user 1123user 1123user 22334user 22334?user 3534user 353user 42322user 42322user 54534user 5235424user 62user 62user 72423user 7242user 8344user 8313454user 93user 93user 10122user 10122user 480189433user 480189433movie 1movie 2movie 3movie 4movie 5movie 6movie 7movie 8movie 9movie 10movie 17770feature 1feature 2feature 3feature 4feature 5feature 1feature 2feature 3feature 4feature 5user 1user 2user 3user 4user 5user 6user 7user 8user 9user 10user 480189

Sheet2

Sheet3

Sheet1movie 1movie 2movie 3movie 4movie 5movie 6movie 7movie 8movie 9movie 10movie 17770movie 1movie 2movie 3movie 4movie 5movie 6movie 7movie 8movie 9movie 10movie 17770user 1123user 1123user 22334user 22334?user 3534user 353user 42322user 42322user 54534user 5235424user 62user 62user 72423user 7242user 8344?user 8313454user 93user 93user 10122user 10122user 480189433user 480189433

Sheet2

Sheet3

Sheet1movie 1movie 2movie 3movie 4movie 5movie 6movie 7movie 8movie 9movie 10movie 17770movie 1movie 2movie 3movie 4movie 5movie 6movie 7movie 8movie 9movie 10movie 17770user 1123user 1123user 22334user 22334?user 3534user 353user 42322user 42322user 54534user 5235424user 62user 62user 72423user 7242user 8344user 8313454user 93user 93user 10122user 10122user 480189433user 480189433movie 1movie 2movie 3movie 4movie 5movie 6movie 7movie 8movie 9movie 10movie 17770feature 1feature 2feature 3feature 4feature 5feature 1feature 2feature 3feature 4feature 5user 1user 2user 3user 4user 5user 6user 7user 8user 9user 10user 480189

Sheet2

Sheet3

Sheet1movie 1movie 2movie 3movie 4movie 5movie 6movie 7movie 8movie 9movie 10movie 17770movie 1movie 2movie 3movie 4movie 5movie 6movie 7movie 8movie 9movie 10movie 17770user 1123user 1123user 22334user 22334?user 3534user 353user 42322user 42322user 54534user 5235424user 62user 62user 72423user 7242user 8344user 8313454user 93user 93user 10122user 10122user 480189433user 480189433movie 1movie 2movie 3movie 4movie 5movie 6movie 7movie 8movie 9movie 10movie 17770feature 1feature 2feature 3feature 4feature 5feature 1feature 2feature 3feature 4feature 5user 1user 2user 3user 4user 5user 6user 7user 8user 9user 10user 480189

Sheet2

Sheet3