Upload
john-rogers
View
11.635
Download
1
Embed Size (px)
DESCRIPTION
Recked is an informal event for engineers interested in recommender systems
Citation preview
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅN
Jim Shur
26 Jan 2009
RECKED
Amsterdam, The Netherlands
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅO
^ÖÉåÇ~
! qçé=päáÅÉW=tÜ~í=áë=píê~åÇë\
! qÜÉ=jÉ~í=Eçê=îÉÖÖáÉëFW=
====J=_ìáäÇáåÖ=~=oÉÅçããÉåÇÉê
! _çííçã=päáÅÉW=píê~åÇë=pçÅá~ä=oÉÅçããÉåÇÉê
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅP
tÜ~í=áë=píê~åÇë\
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
tÜ~í=áë=píê~åÇë\
Q
píê~åÇë=ÇÉîÉäçéë=êÉÅçããÉåÇ~íáçå=íÉÅÜåçäçÖáÉë=íç=ÜÉäé=éÉçéäÉ=ÇáëÅçîÉê=åÉï=íÜáåÖë
båÇ=ìëÉêëW=píê~åÇëKÅçã
ãçåÉóKpíê~åÇëKÅçã=
_ìëáåÉëëÉëW=qÜêçìÖÜ=~=ÜçëíÉÇ=ëçäìíáçåI=píê~åÇë=ÜÉäéë=ÄìëáåÉëëÉë=ÇÉäáîÉê=íÜÉ=êáÖÜí=ÅçåíÉåí=íç=íÜÉ=êáÖÜí=éÉçéäÉ=~í=íÜÉ=
êáÖÜí=íáãÉ
aÉîÉäçéÉêëW=cêÉÉ=~åÇ=É~ëó=íç=~ÇÇ=êÉÅçããÉåÇ~íáçå=
ïáÇÖÉíë=ïáíÜ=çìê=^mfë=
cçê=ÇÉîÉäçéÉêë=~åÇ=ëã~ää=éêçàÉÅíë
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅR
qÜÉ=låÉJäáåÉJçÑJ`çÇÉ=oÉÅçããÉåÇÉê
m êÉëÉåí áåÖ
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅS
my $recdata = `grep -h -e "$seed" $dataDir/alf_*.txt | tr , '\n' | grep -v -e "$seed" -e "^imdb_" | sort | uniq -c | sort -k1,1nr | head`;
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅT
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅU
^ic! ^ëëçÅá~íáçå=iáëí=cçêã~í
! iáëíë=çÑ=áíÉãë=Ñçê=ïÜáÅÜ=óçì=Ü~îÉ=ëçãÉ=êÉ~ëçå=íç=ÄÉäáÉîÉ=íÜ~í=É~ÅÜ=áíÉã=ÅçìäÇ=ëÉêîÉ=~ë=~=êÉÅçããÉåÇ~íáçå=Ñçê=íÜÉ=çíÜÉê
! _ÉÜ~îáçê~ä
! jìëáÅ=éä~óäáëíëI=éä~ó=ëíêÉ~ãëI=Ñ~îçêáíÉëI=KKK
! mìêÅÜ~ëÉ=Ä~ëâÉíëI=é~ÖÉ=îáëáíëI=KKK
! `çããçå=ãÉí~Ç~í~=EÅçäÇ=ëí~êíF
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅV
labs:~/cgi-bin/data/movie$ wc -l alf_*.txt 214711 alf_movie_imdb_actors.txt 129865 alf_movie_imdb_actresses.txt 24970 alf_movie_imdb_cinematographers.txt 19018 alf_movie_imdb_composers.txt 7120 alf_movie_imdb_costume-designers.txt 47145 alf_movie_imdb_directors.txt 28048 alf_movie_imdb_distributors.txt 24544 alf_movie_imdb_editors.txt 24886 alf_movie_imdb_keywords.txt 13691 alf_movie_imdb_locations.txt 11639 alf_movie_imdb_miscellaneous-companies.txt 65584 alf_movie_imdb_miscellaneous.txt 57846 alf_movie_imdb_producers.txt 34519 alf_movie_imdb_production-companies.txt 7574 alf_movie_imdb_production-designers.txt 1333 alf_movie_imdb_special-effects-companies.txt 57309 alf_movie_imdb_writers.txt 769802 total
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅNM
imdb_actors_Bogaert%2C Frans,Bleeke Bet (1923),Bulldog Drummond (1922),Fate's Plaything (1920)
imdb_actors_Bogaerts%2C Sam,Ellektra (2004),Sicilia%2C La (1997),Steve + Sky (2004),Witte%2C De (1980)
imdb_actors_Bogan%2C Hugh,Men Cry Bullets (1997),Zoo Radio (1990)
imdb_actors_Bogard%2C Jerrod,Gatorman (2006),LovecraCked! The Movie (2006)
imdb_actors_Bogart%2C Humphrey,Across the Pacific (1942),Action in the North Atlantic (1943),African Queen%2C The (1951),...
imdb_costume-designers_Adamcova%2C Ruzena,Frona (1954),Igelfreundschaft%2C Die (1962),Jestrab kontra Hrdlicka (1953)
imdb_costume-designers_Adams%2C Annalisa,Billy Schulz (2006),Coffee Date (2006),Confessions of a Late Bloomer (2005)...
imdb_costume-designers_Adams%2C Katherine,Safe Side%2C The (2005),Trip%2C The (2003)
imdb_costume-designers_Adams-Galligan%2C Kimberly,House Divided%2C A (1998),Minotaur (1994),Minus Man%2C The (1999)
imdb_costume-designers_Adams-Santos%2C Stephanie,Last Good Man%2C The (2006),Model Man (2006)
imdb_directors_Spielberg%2C Steven (I),,Amistad (1997),Artificial Intelligence: AI (2001),Close Encounters of the Third Kind ...
imdb_directors_Spierig%2C Peter,Big Picture%2C The (2000),Daybreakers (2008),Undead (2003)
imdb_directors_Spires%2C Tony,Tears of a Clown (2005),Two Degrees (2001)
imdb_directors_Spiridakis%2C Tony,Last Word%2C The (1995),Noise (2004),Tinseltown (1997)
imdb_directors_Spiridellis%2C Evan,Ahnuld for Governor (2003),Big Box Mart (2005),Second Term (2005),This Land (2004)
mdb_keywords_dog-sled,Alaska: Spirit of the Wild (1997),Alaska's Silver Millions (1936),Arizona Dream (1993)...
imdb_keywords_newlywed,10 (1979),Above Suspicion (1943),Another Day%2C Another Man (1966)
imdb_keywords_surrealism,2001: A Space Odyssey (1968),300 (2006),8? (1963),8 X 8: A Chess Sonata in 8 Movements (1957)...
imdb_keywords_whimsical,10 (1979),10 Things I Hate About You (1999),13 Going on 30 (2004),Addams Family%2C The ,,,
^ëëçÅá~íáçå=iáëíë
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
fíÉã=J=fíÉã=! eÉêÉ=ïÉ=~êÉ=ÇáëÅìëëáÄäÉåÖ=?áíÉãJáíÉã?=
êÉÅçããÉåÇ~íáçåë
! mÉçéäÉ=ïÜç=äáâÉ=EÄçìÖÜíI=ìëÉI=KKKF=íÜáëI=ïÉ=íÜáåâ=ïáää=äáâÉ=EÄìóI=ìëÉI=KKKF=íÜ~í
! iáâÉ=~=ÄáÖ=ã~íêáñ=ïáíÜ=~ää=íÜÉ=áíÉãë=~ë=ÄçíÜ=êçïë=~åÇ=Åçäìãåë
! b~ÅÜ=Éåíêó=E~IÄF=áë=~=äáâÉäáÜççÇ=íÜ~í=Ä=áë=~=ÖççÇ=êÉÅçããÉåÇ~íáçå=Ñçê=~
NN
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅNO
grep -h -e "$seed" $dataDir/alf_*.txt
The seed is that from which we are generating the recommendations, in this
case “Casablanca (1942)”
Hence this first step returns all the lists from all the ALF files that contain “Casablanca (1942)”
(imagine huge list of lists, here)
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅNP
... | tr , '\n' | grep -v -e "$seed" -e "^imdb_"
• Pipe all those lists into tr, converting the commas into newlines, putting each movie on its own line, then remove the list headers and the seed itself
...
13th Letter%2C The (1951)
633 Squadron (1964)
Mr. Brown Comes Down the Hill (1965)
Fox%2C The (1967)
imdb_writers_Epstein%2C Julius J.Letter from an Unknown Woman (1948)
Happy Deathday (1968)
In Our Time (1944)
Casablanca (1942)In This Our Life (1942)
Intimate Stranger%2C The (1956)
Journey Out of Darkness (1967)
Letter%2C The (1940)
Mission to Moscow (1943)
Greengage Summer%2C The (1961)
...
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅNQ
... | sort | uniq -c
• Sort the list and qive to uniq -c to replace duplicate lines with a single line plus the count of occurrences
... 1 Angel's Flight (1965) 1 Angel's Holiday (1937) 1 Angela's Ashes (1999) 1 Angels One Five (1952) 2 Angels Over Broadway (1940) 1 Angels Wash Their Faces%2C The (1939) 1 Angels of Death Island%2C The (2003)14 Angels with Dirty Faces (1938) 3 Anger Management (2003) 3 Angst essen Seele auf (1974) 1 Angus (1995) 2 Anima (1998) ...
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅNR
... | sort -k1,1nr | head
• Now sort again, numerically in reverse order, by the counts. Then take the top of the list, and those are your recommendations!
14 Angels with Dirty Faces (1938)
14 To Have and Have Not (1944)
13 Conspirators%2C The (1944)
13 Now%2C Voyager (1942)
12 Maltese Falcon%2C The (1941)
12 Passage to Marseille (1944)
12 Special Agent K-7 (1936)
11 Big Sleep%2C The (1946)
11 Dark Victory (1939)
11 Key Largo (1948)
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅNS
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅNT
oÉîáÉï
! fíÉã=ñ=áíÉã=êÉÅçããÉåÇ~íáçåë=êÉíìêå=íÜÉ=ãçëí=~ëëçÅá~íÉÇ=áíÉãë=íç=~=ÖáîÉå=ëÉÉÇ
! lêÖ~åáòÉ=óçìê=áíÉãë=áå=~ëëçÅá~íáçåë=äáëíë
! cáåÇ=~ää=íÜÉ=äáëíë=ïáíÜ=íÜÉ=ëÉÉÇ
! oÉíìêå=íÜÉ=áíÉãë=íÜ~í=~ééÉ~ê=çå=íÜÉ=ãçëí=äáëíë===
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅNU
bñéä~å~íáçåë
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅNV
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅOM
`çåëáÇÉê~íáçåë
! qÜÉ=a~í~
! tÉáÖÜíáåÖ
! mçéìä~êáíó=L=råéçéìä~êáíó
! mÉêëçå~äáò~íáçå=C=mêÉëÉåí~íáçå
! j~âáåÖ=áí=c~ëí
! qÉëíáåÖ
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅON
qÜÉ=a~í~
! qÜÉ=a~í~=Ü~ë=íç=ÅçãÉ=Ñêçã=ëçãÉïÜÉêÉI=~åÇ=óçì=éêçÄ~Ääó=åÉÉÇ=~=äçí=çÑ=áí
! qÜÉ=ÄÉëí=~äÖçêáíÜã=ïáíÜ=åç=Ç~í~=áë=~ää=ÇêÉëëÉÇ=ìé=ïáíÜ=åç=éä~ÅÉ=íç=Öç=EÄìí=ÅçåîÉêëÉäóKKKF
! håçïáåÖ=ïÜÉå=íïç=áíÉãë=~êÉ=?íÜÉ=ë~ãÉ?
! j~íÅÜáåÖ=íç=~=Å~í~äçÖ
! _É=ÅêÉ~íáîÉ
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅOO
tÉáÖÜíáåÖ! pÅçêÉë=ÇçåÛí=Ü~îÉ=íç=ÄÉ=N=éçáåí=éÉê=çÅÅìêêÉåÅÉ
! iáëí=ïÉáÖÜíáåÖ
! fë=ëÜ~êáåÖ=~=ÇáêÉÅíçê=ãçêÉ=áãéçêí~åí=íÜÉå=ëÜ~êáåÖ=~=ÅçëíìãÉ=ÇÉëáÖåÉê\
! qÉêã=ïÉáÖÜíáåÖ
! fë=NM=éä~óäáëí=ÅçJçÅÅìêêÉåÅÉë=ïáíÜ=`çäÇéä~ó=ïçêíÜ=íÜÉ=ë~ãÉ=~ë=NM=ÅçJçÅÅìêêÉåÅÉë=ïáíÜ=gçå~íÜ~å=oáÅÜã~å\
! j~óÄÉI=ã~óÄÉ=åçí=JJJ=ãçêÉ=ä~íÉê==
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅOP
mçéìä~êáíó! _ÉÜ~îáçê~ä=Ç~í~
! qÜÉ=?`çäÇéä~ó?=mêçÄäÉã
! ^ëëçÅá~íáçåë=îëK=`çêêÉä~íáçåë
! jçêÉ=~ëëçÅá~íÉÇ=Äìí=äÉëë=ÅçêêÉä~íÉÇ\
800
100
25
2723
AB C
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅOQ
mçéìä~êáíó! pçäìíáçåW=qÉêã=tÉáÖÜíáåÖ=Et^icF
! list1,A,B:0.2,C:0.3
! qÉêã=cêÉèìÉåÅó=J=fåîÉêëÉ=açÅìãÉåí=cêÉèìÉåÅó
! qcJfac
! ^äëç=ëçäîÉë=aáäìíÉÇ=iáëí=mêçÄäÉã
! ^ëëçÅá~íáçåë=ã~ó=ÄÉ=äÉëë=ãÉ~åáåÖÑìä=áå=îÉêó=ä~êÖÉ=äáëíë
! pÉÉ=~äëçW=mÉêëçå~äáò~íáçå
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅOR
råéçéìä~êáíó! oÉÅçããÉåÇ~íáçåë=Ñçê=~å=áíÉã=çå=çåÉ=çê=îÉêó=
ÑÉï=äáëíë
! ^â~=íÜÉ=?ïÜó=íÜÉ=ÜÉÅâ=ÇáÇ=óçì=êÉÅçããÉåÇ=íÜáë\?=éêçÄäÉã
! jççåäáÖÜí=pçå~í~=JJ[=qçñáÅ=E_êáíåÉó=péÉ~êëF
! pçäìíáçåW=qeobpelia
! oÉèìáêÉ=ëçãÉ=ãáåáãìã=ëÅçêÉ=L=ÉîáÇÉåÅÉ
! qÜêÉëÜçäÇ=J=`çîÉê~ÖÉ=íê~ÇÉçÑÑ
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅOS
mÉêëçå~äáò~íáçå! êÉÅEìëÉêJNI=ëÉÉÇF=!=êÉÅEìëÉêJOI=ëÉÉÇF
! rëÉ=Üáëíçêó=~åÇ=éêçÑáäÉ=Ñçê=áãéäáÅáí=ÑáäíÉêáåÖ=çê=ïÉáÖÜíáåÖ=E~äêÉ~Çó=ÄçìÖÜíI=îáëáíÉÇ=êÉÅÉåíäóI=KKKF
! ^ääçï=ÉñéäáÅáí=ÑáäíÉêáåÖI=Åìëíçãáò~íáçå=EëÉÉW=éçéF
! cÉÉÇÄ~Åâ=J=Ñ~îçêáíÉëI=Ä~ååáåÖ
! pÉÉÇ=ÖÉåÉê~íáçå=L=~ìÖãÉåí~íáçå
! =x=ÑEìëÉêF=Z=êÉÅçããÉåÇ~íáçåë=z
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅOT
mêÉëÉåí~íáçå! dççÇ=îëK=rëÉÑìä
! lÄîáçìëåÉëë
! péêáåÖëíÉÉå="_çêå=íç=oìåÒ=J[=péêáåÖëíÉÉå="qÜìåÇÉê=oç~ÇÒ=Eilqo=f=J[=ilqo=ffF
! aáîÉêëáíó
! kçí=~ää=ëÜçÉë=Äó=íÜÉ=ë~ãÉ=ÇÉëáÖåÉê
! cçê=ÑêÉèìÉåí=îáëáíÉêI=ÇçåÛí=ëÜçï=ë~ãÉ=êÉÅçããÉåÇ~íáçåë=ÉîÉêó=íáãÉ
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
! fÑ=åÉÅÉëë~êó>
! mêÉJÅçãéìí~íáçå
! fåÅêÉãÉåí~ä=ìéÇ~íÉë
offline
péÉÉÇ
listsassociation
matrix top n correlations
per item
recommender
OU
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅOV
qÉëíáåÖ! pìÄàÉÅíáîÉ=~å~äóëáë
! eçäÇJçìíë
! _ÉÑçêÉ=L=^ÑíÉê
! ^J_=qÉëíáåÖ
! fåÇáêÉÅí=ãÉíêáÅë
! jÉ~ëìêáåÖ=ïÜ~íÉîÉê=?ëìÅÅÉëë?=ãÉ~åë=Ñçê=óçìê=ëáíÉ
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅPM
píê~åÇë=pçÅá~ä=oÉÅçããÉåÇÉê
! qÜÉ=ÅçêÉ=ÅçåÅÉéí=áë=ëíê~áÖÜíÑçêï~êÇ
! ^=ÑÉï=êÉ~äáíáÉë=ã~âÉ=áí=ãçêÉ=ÇáÑÑáÅìäí
! píê~åÇë=áë=áå=íÜÉ=ÄìëáåÉëë=çÑ=ã~âáåÖ=áí=~ë=É~ëó=~ë=éçëëáÄäÉ
! áÜÉ~êíãçîáÉëKÅçã=~ÇÇÉÇ=êÉÅçããÉåÇ~íáçåë=íç=íÜÉáê=ëáíÉ=áå=çåÉ=Üçìê
! q~äâ=íç=gÉë#ë=çê=Åçåí~Åí=ìë=Ñçê=ãçêÉ=áåÑç
! ÜííéWLLÄáòëçäìíáçåëKëíê~åÇëKÅçã
`lkcfabkqf^i=C=molmofbq^ov
`çéóêáÖÜí=OMMPJOMMUI=píê~åÇëI=fåÅ PN
dÉåÉêáÅ=ÉåÖáåÉ=Ñçê=ÉJÅçããÉêÅÉI=îáÇÉçI=~åÇ=ëçÅá~ä=ÅçåíÉåí=ëáíÉë
p~~p=oÉÅçããÉåÇÉê
`lkcfabkqf^i=C=molmofbq^ov
`çéóêáÖÜí=OMMPJOMMUI=píê~åÇëI=fåÅ PO
Capturing behavior in a retail site
Item visited Recommended item visited Item reviewed Item added to shopping cart Item added to favorites Item added to wishlist Item purchased
Capturing behavior in a content site
Content visited Recommended content visited Content reviewed Content searched Content uploaded Content downloaded
Getting recommendations Get item recommendation Get user recommendation Explain a recommendation
Session start/end Session start Session end
Rating items Rate an item Show item rating
Tagging users and items
Set user tags Get user tags Set item tags Get item tags
pçÅá~ä=oÉÅçããÉåÇÉê=^mf
`lkcfabkqf^i=C=molmofbq^ov
`çéóêáÖÜí=OMMPJOMMUI=píê~åÇëI=fåÅ PP
Item visited/tracking/event_visited.sbs
! customerId
! itemId
! userId
! sessionId
! needResult
Tracks an item that has been visited by a user
Recommended item visited
/tracking/event_clickedrecommendation.sbs
! customerId
! itemId
! userId
! reqId
! sessionId
! needResult
! widgetId
Tracks a recommended item that has been clicked
Item reviewed/tracking/event_reviewed.sbs
! customerId
! itemId
! userId
! sessionId
! needResult
Tracks an item reviewed or commented by a user
Item added to shopping cart
/tracking/event_addshoppingcart.sbs
! customerId
! itemId
! userId
! sessionId
! needResult
Tracks an item that has been added to the shopping cart
bñ~ãéäÉW=ÄÉÜ~îáçê=áå=êÉí~áä
`lkcfabkqf^i=C=molmofbq^ov
`çéóêáÖÜí=OMMPJOMMUI=píê~åÇëI=fåÅ PQ
aÉîÉäçéÉêë=páíÉ
ÜííéWLLÄáòëçäìíáçåëKëíê~åÇëKÅçãLíÉÅÜ
oÉÖáëíÉê=C`~í~äçÖ=ìéäç~Ç
`~éíìêÉ_ÉÜ~îáçê~ä=bîÉåíë
dÉí=oÉÅçããÉåÇ~íáçåë
`lkcfabkqf^i=C=molmofbq^ov
`çéóêáÖÜí=OMMPJOMMUI=píê~åÇëI=fåÅ PR
bñ~ãéäÉ=rëÉêë
`lkcfabkqf^i=C=molmofbq^ov
`çéóêáÖÜí=OMMPJOMMUI=píê~åÇëI=fåÅ PS
a~ëÜÄç~êÇ=^å~äóíáÅë
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅ
^ÅâåçïäÉÇÖÉãÉåíë
! qÜ~åâë=íç=hÉå=_~äÇïáå=ïÜç=çêáÖáå~ääó=ÅçåÅÉáîÉÇ=çÑ=íÜÉ=?çåÉJäáåÉ=êÉÅçããÉåÇÉê?=áå=íÜáë=í~äâ
! qÜ~åâë=íç=t~âççé~=~åÇ=píê~åÇë=Ñçê=ëéçåëçêáåÖ=ob`hba
! qÜ~åâë=íç=~ää=Ñçê=~ííÉåÇáåÖ>
PT
`çéóêáÖÜí=OMMPJOMMVI=píê~åÇëI=fåÅPU