Recommendations 101

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ N

Jim Shur

26 Jan 2009

RECKED

Amsterdam, The Netherlands

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ O

^ÖÉåÇ~

• qçé=päáÅÉW=tÜ~í=áë=píê~åÇë\

• qÜÉ=jÉ~í=Eçê=îÉÖÖáÉëFW=

====J=_ìáäÇáåÖ=~=oÉÅçããÉåÇÉê

• _çííçã=päáÅÉW=píê~åÇë=pçÅá~ä=oÉÅçããÉåÇÉê

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ P

tÜ~í=áë=píê~åÇë\

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ

tÜ~í=áë=píê~åÇë\

Q

píê~åÇë=ÇÉîÉäçéë=êÉÅçããÉåÇ~íáçå=íÉÅÜåçäçÖáÉë=íç=ÜÉäé=éÉçéäÉ=ÇáëÅçîÉê=åÉï=íÜáåÖë

båÇ=ìëÉêëW=píê~åÇëKÅçã

ãçåÉóKpíê~åÇëKÅçã=

_ìëáåÉëëÉëW=qÜêçìÖÜ=~=ÜçëíÉÇ=ëçäìíáçåI=píê~åÇë=ÜÉäéë=ÄìëáåÉëëÉë=ÇÉäáîÉê=íÜÉ=êáÖÜí=ÅçåíÉåí=íç=íÜÉ=êáÖÜí=éÉçéäÉ=~í=íÜÉ=

êáÖÜí=íáãÉ

aÉîÉäçéÉêëW=cêÉÉ=~åÇ=É~ëó=íç=~ÇÇ=êÉÅçããÉåÇ~íáçå=

ïáÇÖÉíë=ïáíÜ=çìê=^mfë=

cçê=ÇÉîÉäçéÉêë=~åÇ=ëã~ää=éêçàÉÅíë

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ R

qÜÉ=låÉJäáåÉJçÑJ`çÇÉ=oÉÅçããÉåÇÉê

m êÉëÉåí áåÖ

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ S

my $recdata = `grep -h -e "$seed" $dataDir/alf_*.txt | tr , '\n' | grep -v -e "$seed" -e "^imdb_" | sort | uniq -c | sort -k1,1nr | head`;

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ T

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ U

^ic• ^ëëçÅá~íáçå=iáëí=cçêã~í

• iáëíë=çÑ=áíÉãë=Ñçê=ïÜáÅÜ=óçì=Ü~îÉ=ëçãÉ=êÉ~ëçå=íç=ÄÉäáÉîÉ=íÜ~í=É~ÅÜ=áíÉã=ÅçìäÇ=ëÉêîÉ=~ë=~=êÉÅçããÉåÇ~íáçå=Ñçê=íÜÉ=çíÜÉê

• _ÉÜ~îáçê~ä

• jìëáÅ=éä~óäáëíëI=éä~ó=ëíêÉ~ãëI=Ñ~îçêáíÉëI=KKK

• mìêÅÜ~ëÉ=Ä~ëâÉíëI=é~ÖÉ=îáëáíëI=KKK

• `çããçå=ãÉí~Ç~í~=EÅçäÇ=ëí~êíF

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ V

labs:~/cgi-bin/data/movie$ wc -l alf_*.txt 214711 alf_movie_imdb_actors.txt 129865 alf_movie_imdb_actresses.txt 24970 alf_movie_imdb_cinematographers.txt 19018 alf_movie_imdb_composers.txt 7120 alf_movie_imdb_costume-designers.txt 47145 alf_movie_imdb_directors.txt 28048 alf_movie_imdb_distributors.txt 24544 alf_movie_imdb_editors.txt 24886 alf_movie_imdb_keywords.txt 13691 alf_movie_imdb_locations.txt 11639 alf_movie_imdb_miscellaneous-companies.txt 65584 alf_movie_imdb_miscellaneous.txt 57846 alf_movie_imdb_producers.txt 34519 alf_movie_imdb_production-companies.txt 7574 alf_movie_imdb_production-designers.txt 1333 alf_movie_imdb_special-effects-companies.txt 57309 alf_movie_imdb_writers.txt 769802 total

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ NM

imdb_actors_Bogaert%2C Frans,Bleeke Bet (1923),Bulldog Drummond (1922),Fate's Plaything (1920)imdb_actors_Bogaerts%2C Sam,Ellektra (2004),Sicilia%2C La (1997),Steve + Sky (2004),Witte%2C De (1980)imdb_actors_Bogan%2C Hugh,Men Cry Bullets (1997),Zoo Radio (1990)imdb_actors_Bogard%2C Jerrod,Gatorman (2006),LovecraCked! The Movie (2006)imdb_actors_Bogart%2C Humphrey,Across the Pacific (1942),Action in the North Atlantic (1943),African Queen%2C The (1951),...

imdb_costume-designers_Adamcova%2C Ruzena,Frona (1954),Igelfreundschaft%2C Die (1962),Jestrab kontra Hrdlicka (1953)imdb_costume-designers_Adams%2C Annalisa,Billy Schulz (2006),Coffee Date (2006),Confessions of a Late Bloomer (2005)...imdb_costume-designers_Adams%2C Katherine,Safe Side%2C The (2005),Trip%2C The (2003)imdb_costume-designers_Adams-Galligan%2C Kimberly,House Divided%2C A (1998),Minotaur (1994),Minus Man%2C The (1999)imdb_costume-designers_Adams-Santos%2C Stephanie,Last Good Man%2C The (2006),Model Man (2006)

imdb_directors_Spielberg%2C Steven (I),,Amistad (1997),Artificial Intelligence: AI (2001),Close Encounters of the Third Kind ...imdb_directors_Spierig%2C Peter,Big Picture%2C The (2000),Daybreakers (2008),Undead (2003)imdb_directors_Spires%2C Tony,Tears of a Clown (2005),Two Degrees (2001)imdb_directors_Spiridakis%2C Tony,Last Word%2C The (1995),Noise (2004),Tinseltown (1997)imdb_directors_Spiridellis%2C Evan,Ahnuld for Governor (2003),Big Box Mart (2005),Second Term (2005),This Land (2004)

mdb_keywords_dog-sled,Alaska: Spirit of the Wild (1997),Alaska's Silver Millions (1936),Arizona Dream (1993)...imdb_keywords_newlywed,10 (1979),Above Suspicion (1943),Another Day%2C Another Man (1966)imdb_keywords_surrealism,2001: A Space Odyssey (1968),300 (2006),8? (1963),8 X 8: A Chess Sonata in 8 Movements (1957)...imdb_keywords_whimsical,10 (1979),10 Things I Hate About You (1999),13 Going on 30 (2004),Addams Family%2C The ,,,

^ëëçÅá~íáçå=iáëíë

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ

fíÉã=J=fíÉã=• eÉêÉ=ïÉ=~êÉ=ÇáëÅìëëáÄäÉåÖ=?áíÉãJáíÉã?=

êÉÅçããÉåÇ~íáçåë

• mÉçéäÉ=ïÜç=äáâÉ=EÄçìÖÜíI=ìëÉI=KKKF=íÜáëI=ïÉ=íÜáåâ=ïáää=äáâÉ=EÄìóI=ìëÉI=KKKF=íÜ~í

• iáâÉ=~=ÄáÖ=ã~íêáñ=ïáíÜ=~ää=íÜÉ=áíÉãë=~ë=ÄçíÜ=êçïë=~åÇ=Åçäìãåë

• b~ÅÜ=Éåíêó=E~IÄF=áë=~=äáâÉäáÜççÇ=íÜ~í=Ä=áë=~=ÖççÇ=êÉÅçããÉåÇ~íáçå=Ñçê=~

NN

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ NO

grep -h -e "$seed" $dataDir/alf_*.txt

The seed is that from which we are generating the recommendations, in this

case “Casablanca (1942)”

Hence this first step returns all the lists from all the ALF files that contain “Casablanca (1942)”

(imagine huge list of lists, here)

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ NP

... | tr , '\n' | grep -v -e "$seed" -e "^imdb_" • Pipe all those lists into tr, converting the commas into newlines,

putting each movie on its own line, then remove the list headers and the seed itself

...13th Letter%2C The (1951)633 Squadron (1964)Mr. Brown Comes Down the Hill (1965)Fox%2C The (1967)imdb_writers_Epstein%2C Julius J.Letter from an Unknown Woman (1948)Happy Deathday (1968)In Our Time (1944)Casablanca (1942)In This Our Life (1942)Intimate Stranger%2C The (1956)Journey Out of Darkness (1967)Letter%2C The (1940)Mission to Moscow (1943)Greengage Summer%2C The (1961)...

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ NQ

... | sort | uniq -c

• Sort the list and qive to uniq -c to replace duplicate lines with a single line plus the count of occurrences

... 1 Angel's Flight (1965) 1 Angel's Holiday (1937) 1 Angela's Ashes (1999) 1 Angels One Five (1952) 2 Angels Over Broadway (1940) 1 Angels Wash Their Faces%2C The (1939) 1 Angels of Death Island%2C The (2003)14 Angels with Dirty Faces (1938) 3 Anger Management (2003) 3 Angst essen Seele auf (1974) 1 Angus (1995) 2 Anima (1998) ...

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ NR

... | sort -k1,1nr | head • Now sort again, numerically in reverse order, by the

counts. Then take the top of the list, and those are your recommendations!

14 Angels with Dirty Faces (1938)14 To Have and Have Not (1944)13 Conspirators%2C The (1944)13 Now%2C Voyager (1942)12 Maltese Falcon%2C The (1941)12 Passage to Marseille (1944)12 Special Agent K-7 (1936)11 Big Sleep%2C The (1946)11 Dark Victory (1939)11 Key Largo (1948)

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ NS

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ NT

oÉîáÉï

• fíÉã=ñ=áíÉã=êÉÅçããÉåÇ~íáçåë=êÉíìêå=íÜÉ=ãçëí=~ëëçÅá~íÉÇ=áíÉãë=íç=~=ÖáîÉå=ëÉÉÇ

• lêÖ~åáòÉ=óçìê=áíÉãë=áå=~ëëçÅá~íáçåë=äáëíë

• cáåÇ=~ää=íÜÉ=äáëíë=ïáíÜ=íÜÉ=ëÉÉÇ

• oÉíìêå=íÜÉ=áíÉãë=íÜ~í=~ééÉ~ê=çå=íÜÉ=ãçëí=äáëíë===

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ NU

bñéä~å~íáçåë

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ NV

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ OM

`çåëáÇÉê~íáçåë

• qÜÉ=a~í~

• tÉáÖÜíáåÖ

• mçéìä~êáíó=L=råéçéìä~êáíó

• mÉêëçå~äáò~íáçå=C=mêÉëÉåí~íáçå

• j~âáåÖ=áí=c~ëí

• qÉëíáåÖ

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ ON

qÜÉ=a~í~• qÜÉ=a~í~=Ü~ë=íç=ÅçãÉ=Ñêçã=ëçãÉïÜÉêÉI=~åÇ=

óçì=éêçÄ~Ääó=åÉÉÇ=~=äçí=çÑ=áí

• qÜÉ=ÄÉëí=~äÖçêáíÜã=ïáíÜ=åç=Ç~í~=áë=~ää=ÇêÉëëÉÇ=ìé=ïáíÜ=åç=éä~ÅÉ=íç=Öç=EÄìí=ÅçåîÉêëÉäóKKKF

• håçïáåÖ=ïÜÉå=íïç=áíÉãë=~êÉ=?íÜÉ=ë~ãÉ?

• j~íÅÜáåÖ=íç=~=Å~í~äçÖ

• _É=ÅêÉ~íáîÉ

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ OO

tÉáÖÜíáåÖ• pÅçêÉë=ÇçåÛí=Ü~îÉ=íç=ÄÉ=N=éçáåí=éÉê=çÅÅìêêÉåÅÉ

• iáëí=ïÉáÖÜíáåÖ

• fë=ëÜ~êáåÖ=~=ÇáêÉÅíçê=ãçêÉ=áãéçêí~åí=íÜÉå=ëÜ~êáåÖ=~=ÅçëíìãÉ=ÇÉëáÖåÉê\

• qÉêã=ïÉáÖÜíáåÖ

• fë=NM=éä~óäáëí=ÅçJçÅÅìêêÉåÅÉë=ïáíÜ=`çäÇéä~ó=ïçêíÜ=íÜÉ=ë~ãÉ=~ë=NM=ÅçJçÅÅìêêÉåÅÉë=ïáíÜ=gçå~íÜ~å=oáÅÜã~å\

• j~óÄÉI=ã~óÄÉ=åçí=JJJ=ãçêÉ=ä~íÉê==

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ OP

mçéìä~êáíó• _ÉÜ~îáçê~ä=Ç~í~

• qÜÉ=?`çäÇéä~ó?=mêçÄäÉã

• ^ëëçÅá~íáçåë=îëK=`çêêÉä~íáçåë

• jçêÉ=~ëëçÅá~íÉÇ=Äìí=äÉëë=ÅçêêÉä~íÉÇ\

800

100

25

2723

AB C

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ OQ

mçéìä~êáíó• pçäìíáçåW=qÉêã=tÉáÖÜíáåÖ=Et^icF

• list1,A,B:0.2,C:0.3

• qÉêã=cêÉèìÉåÅó=J=fåîÉêëÉ=açÅìãÉåí=cêÉèìÉåÅó

• qcJfac

• ^äëç=ëçäîÉë=aáäìíÉÇ=iáëí=mêçÄäÉã

• ^ëëçÅá~íáçåë=ã~ó=ÄÉ=äÉëë=ãÉ~åáåÖÑìä=áå=îÉêó=ä~êÖÉ=äáëíë

• pÉÉ=~äëçW=mÉêëçå~äáò~íáçå

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ OR

råéçéìä~êáíó• oÉÅçããÉåÇ~íáçåë=Ñçê=~å=áíÉã=çå=çåÉ=çê=îÉêó=

ÑÉï=äáëíë

• ^â~=íÜÉ=?ïÜó=íÜÉ=ÜÉÅâ=ÇáÇ=óçì=êÉÅçããÉåÇ=íÜáë\?=éêçÄäÉã

• jççåäáÖÜí=pçå~í~=JJ[=qçñáÅ=E_êáíåÉó=péÉ~êëF

• pçäìíáçåW=qeobpelia

• oÉèìáêÉ=ëçãÉ=ãáåáãìã=ëÅçêÉ=L=ÉîáÇÉåÅÉ

• qÜêÉëÜçäÇ=J=`çîÉê~ÖÉ=íê~ÇÉçÑÑ

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ OS

mÉêëçå~äáò~íáçå• êÉÅEìëÉêJNI=ëÉÉÇF=≠=êÉÅEìëÉêJOI=ëÉÉÇF

• rëÉ=Üáëíçêó=~åÇ=éêçÑáäÉ=Ñçê=áãéäáÅáí=ÑáäíÉêáåÖ=çê=ïÉáÖÜíáåÖ=E~äêÉ~Çó=ÄçìÖÜíI=îáëáíÉÇ=êÉÅÉåíäóI=KKKF

• ^ääçï=ÉñéäáÅáí=ÑáäíÉêáåÖI=Åìëíçãáò~íáçå=EëÉÉW=éçéF

• cÉÉÇÄ~Åâ=J=Ñ~îçêáíÉëI=Ä~ååáåÖ

• pÉÉÇ=ÖÉåÉê~íáçå=L=~ìÖãÉåí~íáçå

• =x=ÑEìëÉêF=Z=êÉÅçããÉåÇ~íáçåë=z

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ OT

mêÉëÉåí~íáçå• dççÇ=îëK=rëÉÑìä

• lÄîáçìëåÉëë

• péêáåÖëíÉÉå=“_çêå=íç=oìåÒ=J[=péêáåÖëíÉÉå=“qÜìåÇÉê=oç~ÇÒ=Eilqo=f=J[=ilqo=ffF

• aáîÉêëáíó

• kçí=~ää=ëÜçÉë=Äó=íÜÉ=ë~ãÉ=ÇÉëáÖåÉê

• cçê=ÑêÉèìÉåí=îáëáíÉêI=ÇçåÛí=ëÜçï=ë~ãÉ=êÉÅçããÉåÇ~íáçåë=ÉîÉêó=íáãÉ

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ

• fÑ=åÉÅÉëë~êó>

• mêÉJÅçãéìí~íáçå

• fåÅêÉãÉåí~ä=ìéÇ~íÉë

offline

péÉÉÇ

listsassociation

matrix top n correlations

per item

recommender

OU

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ OV

qÉëíáåÖ• pìÄàÉÅíáîÉ=~å~äóëáë

• eçäÇJçìíë

• _ÉÑçêÉ=L=^ÑíÉê

• ^J_=qÉëíáåÖ

• fåÇáêÉÅí=ãÉíêáÅë

• jÉ~ëìêáåÖ=ïÜ~íÉîÉê=?ëìÅÅÉëë?=ãÉ~åë=Ñçê=óçìê=ëáíÉ

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ PM

píê~åÇë=pçÅá~ä=oÉÅçããÉåÇÉê

• qÜÉ=ÅçêÉ=ÅçåÅÉéí=áë=ëíê~áÖÜíÑçêï~êÇ

• ^=ÑÉï=êÉ~äáíáÉë=ã~âÉ=áí=ãçêÉ=ÇáÑÑáÅìäí

• píê~åÇë=áë=áå=íÜÉ=ÄìëáåÉëë=çÑ=ã~âáåÖ=áí=~ë=É~ëó=~ë=éçëëáÄäÉ

• áÜÉ~êíãçîáÉëKÅçã=~ÇÇÉÇ=êÉÅçããÉåÇ~íáçåë=íç=íÜÉáê=ëáíÉ=áå=çåÉ=Üçìê

• q~äâ=íç=gÉëúë=çê=Åçåí~Åí=ìë=Ñçê=ãçêÉ=áåÑç

• ÜííéWLLÄáòëçäìíáçåëKëíê~åÇëKÅçã

`lkcfabkqf^i=C=molmofbq^ov

`çéóêáÖÜí=OMMPJOMMUI=píê~åÇëI=fåÅ PN

dÉåÉêáÅ=ÉåÖáåÉ=Ñçê=ÉJÅçããÉêÅÉI=îáÇÉçI=~åÇ=ëçÅá~ä=ÅçåíÉåí=ëáíÉë

p~~p=oÉÅçããÉåÇÉê

`lkcfabkqf^i=C=molmofbq^ov

`çéóêáÖÜí=OMMPJOMMUI=píê~åÇëI=fåÅ PO

Capturing behavior in a retail site

Item visited Recommended item visited Item reviewed Item added to shopping cart Item added to favorites Item added to wishlist Item purchased

Capturing behavior in a content site

Content visited Recommended content visited Content reviewed Content searched Content uploaded Content downloaded

Getting recommendations Get item recommendation Get user recommendation Explain a recommendation

Session start/end Session start Session end

Rating items Rate an item Show item rating

Tagging users and items Set user tags Get user tags Set item tags Get item tags

pçÅá~ä=oÉÅçããÉåÇÉê=^mf

`lkcfabkqf^i=C=molmofbq^ov

`çéóêáÖÜí=OMMPJOMMUI=píê~åÇëI=fåÅ PP

Item visited/tracking/event_visited.sbs

‣ customerId‣ itemId‣ userId‣ sessionId‣ needResult

Tracks an item that has been visited by a user

Recommended item visited

/tracking/event_clickedrecommendation.sbs

‣ customerId‣ itemId‣ userId‣ reqId‣ sessionId‣ needResult‣ widgetId

Tracks a recommended item that has been clicked

Item reviewed/tracking/event_reviewed.sbs

‣ customerId‣ itemId‣ userId‣ sessionId‣ needResult

Tracks an item reviewed or commented by a user

Item added to shopping cart

/tracking/event_addshoppingcart.sbs

‣ customerId‣ itemId‣ userId‣ sessionId‣ needResult

Tracks an item that has been added to the shopping cart

bñ~ãéäÉW=ÄÉÜ~îáçê=áå=êÉí~áä

`lkcfabkqf^i=C=molmofbq^ov

`çéóêáÖÜí=OMMPJOMMUI=píê~åÇëI=fåÅ PQ

aÉîÉäçéÉêë=páíÉÜííéWLLÄáòëçäìíáçåëKëíê~åÇëKÅçãLíÉÅÜ

oÉÖáëíÉê=C`~í~äçÖ=ìéäç~Ç

`~éíìêÉ_ÉÜ~îáçê~ä=bîÉåíë

dÉí=oÉÅçããÉåÇ~íáçåë

`lkcfabkqf^i=C=molmofbq^ov

`çéóêáÖÜí=OMMPJOMMUI=píê~åÇëI=fåÅ PR

bñ~ãéäÉ=rëÉêë

`lkcfabkqf^i=C=molmofbq^ov

`çéóêáÖÜí=OMMPJOMMUI=píê~åÇëI=fåÅ PS

a~ëÜÄç~êÇ=^å~äóíáÅë

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ

^ÅâåçïäÉÇÖÉãÉåíë

• qÜ~åâë=íç=hÉå=_~äÇïáå=ïÜç=çêáÖáå~ääó=ÅçåÅÉáîÉÇ=çÑ=íÜÉ=?çåÉJäáåÉ=êÉÅçããÉåÇÉê?=áå=íÜáë=í~äâ

• qÜ~åâë=íç=t~âççé~=~åÇ=píê~åÇë=Ñçê=ëéçåëçêáåÖ=ob`hba

• qÜ~åâë=íç=~ää=Ñçê=~ííÉåÇáåÖ>

PT

`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ PU