Comparing Recommendaon Algorithms for Social Bookmarkingtoinebogers.com/content/slides/201001-comparing-recsys-for-bookm… · Social bookmarking • Way of storing, organizing, and

ComparingRecommenda/onAlgorithmsforSocialBookmarking

ToineBogers

RoyalSchoolofLibraryandInforma/onScience

Copenhagen,Denmark

Aboutme

•  Ph.D.fromTilburgUniversity  “RecommenderSystemsforSocialBookmarking”  Promotor:Prof.dr.AntalvandenBosch

•  Currently@RSLIS(Copenhagen,DK)  Researchassistantonretrievalfusionproject

•  Researchinterests  Recommendersystems  Socialbookmarking

  Expertsearch  Informa/onretrieval

Outline

1.  Introduc/on2.  Collabora/vefiltering3.  Content‐basedfiltering4.  Recommendersystemsfusion

5.  Conclusions

Socialbookmarking

•  Wayofstoring,organizing,andmanagingbookmarksofWebpages,scien/ficar/cles,books,etc.  Alldoneonline  Canbemadepublicorkeptprivate  Allowuserstotag(=label)theiritems

  Manydifferentwebsitesavailable:

Socialbookmarking

•  Differentdomains  Webpages  Scien/ficar/cles  Books

•  Stronggrowthinpopularity  Millionsofusers,items,andtags

  Forexample:Delicious-  140,000+posts/dayonaveragein2008(Keller,2009)-  7,000,000+posts/monthin2008(Wetzkeretal.,2009)

Contentoverload

•  Problemswiththisgrowth  Contentoverload  Increasingambiguity

•  Howcanwedealwiththis?  Browsing  Search

•  Apossiblesolu/on  Takeamoreac/verole:recommenda,on

Canbecomelesseffec/veascontentincreases!

Recommenda/ontasks

!"#$%"&%'("&

)"

*+")&

,"-#))"./

012#.

!"#$%"&

$,#3%'.4

*+")&

"5$",+6

7#,"&

%'("&+8'6&

914&

6:44"62#.

;#)1'.

"5$",+6

!",6#.1%'<"0&

6"1,-8

;"$+8&

=,#>6'.4

?@AB

*9A7

9CD

?@AB *9A7 9CD

!"#$%&$'''

()*&"$+$$'''

Itemrecommenda/on

•  Ourfocus:itemrecommenda,on   Iden/fysetsofitemsthatarelikelytobeofinteresttoacertainuser-  Returnarankedlistofitems

-  ‘FindGoodItems’task(Herlockeretal.,2004)

  Basedondifferentinforma/onsources-  Transac/onpajerns(usagedata,purchaseinforma/on)

–  Explicitra/ngs–  Implicitfeedback

- Metadata

-  Tags

Relatedwork

•  Workonsocialbookmarkingmostlyfocusedon  Improvingbrowsingexperience

-  clustering,dealingwithambiguity

  Incorpora/ngtagsinsearchalgorithms

  Tagrecommenda/on

•  Problemswithworkonitemrecommenda/on  Differentdatasets  Differentevalua/onmetrics

  Nocomparisonofalgorithmsundercontrolledcondi/ons  Hardlyeverpubliclyavailabledatasets  Nouser‐basedevalua/on

Collec/ngdata

•  Fourdatasetsfromtwodifferentdomains  Webbookmarks

- Delicious-  BibSonomy

  Scien/ficar/cles-  CiteULike-  BibSonomy

~78%ofuserspostedonlytypeofcontent(bookmarksorscien/ficar/cles)

Whatdidwecollect?

•  Usagedata  User‐item‐tagtripleswith/mestamps

•  Metadata  Varieswiththedomain

Scien,ficar,cles  Item‐intrinsic

-  TITLE,DESCRIPTION,JOURNAL,AUTHOR,TAGS,URL,etc.

  Item‐extrinsic-  CHAPTER,DAY,EDITION,

YEAR,INSTITUTION,etc.

Webbookmarks  TITLE,DESCRIPTION,TAGS,

URL

Filtering

•  Why?  Toreducenoiseinourdatasets  Commonprocedureinrecommendersystemsresearch

•  How?  ≥20itemsperuser

  ≥2usersperitem(nohapaxlegomenaitems)  Nountaggedposts

•  Comparedtorelatedwork  Stricterfiltering  Morerealis/c

Datasets

Delicious BibSonomy CiteULike BibSonomy

#users 1,243 192 1,322 167

#items 152,698 11,165 38,419 12,982

#tags 42,820 13,233 28,312 5,165

#posts 238,070 29,096 84,637 29,720

Scien,ficar,clesBookmarks

Experimentalsetup

•  Backtes/ng  Withholdrandomlyselecteditemsfromtestusers  Useremainingmaterialfortrainingrecommendersystem  Successispredictedtheuser’sinterestinhis/herwithhelditems

•  Details  Overall90%‐10%splitonusers  Withhold10randomlyselecteditems ofeachtestuser  Parameterop/miza/on

- Used10‐foldcross‐valida/on-  90‐10splits-  10withhelditems

  Macro‐averagingofevalua/onscores

Evalua/on

•  ‘FindGoodItems’taskreturnsarankedlist  Needmetricthattakeintorankingofitems

•  Precision‐orientedmetric  MeanAveragePrecision(MAP)

-  AveragePrecision(AP)isaverageofprecisionvaluesateachrelevant,retrieveditem

- MAPisAPaveragedoverallusers

-  “singlefiguremeasureofqualityacrossrecalllevels”(Manning,2009)

•  Testeddifferentmetrics  Allprecision‐orientedmetricsshowedthesamepicture

Collabora/vefiltering

•  Ques/on  Howcanweusetheinforma/oninthefolksonomytogeneratebejerrecommenda/ons? - Users-  Items-  Tags

•  Collabora/vefiltering(CF)  Ajemptstoautomate“word‐of‐mouth”recommenda/ons  Recommenditemsbasedonhowlike‐mindedusersratedthoseitems

  Similaritybasedon- Usagedata-  Taggingdata

usagepajerns

Collabora/vefiltering

•  Model‐basedCF  ‘Eager’recommenda/onalgorithms  Trainapredic/vemodeloftherecommenda/ontask

  Quicktoapplytogeneraterecommenda/ons

•  Memory‐basedCF  ‘Lazy’recommenda/onalgorithms  Simplystoreallpajernsinmemory

  Deferpredic/onefforttowhenuserrequestsrecommenda/ons

Relatedwork

•  Model‐based  HybridPLSA‐basedapproach(Wetzkeretal.,2009)  Tensordecomposi/on(Symeonidisetal.,2008)

•  Memory‐based  Tag‐awarefusion(Tso‐Sujeretal.,2008)

•  Graph‐based  FolkRank(Hothoetal.,2006)  Randomwalk(Clementsetal.,2008)

Algorithms

•  User‐basedk‐NNalgorithm  Calculatesimilaritybetweentheac/veuserandallotherusers  Determinethetopknearestneighbors

-  I.e.,themostsimilarusers

  Unseenitemsfromnearestneighborsarescoredbythesimilaritybetweentheneighborandtheac/veuser

•  Item‐basedk‐NNalgorithm  Calculatesimilaritybetweentheac/veuser’sitemsandallotheritems

  Determinethetopknearestneighbors-  I.e.,themostsimilaritemsforeachoftheac/veuser’sitems

  Unseenneighboringitemsarescoredbythesimilaritybetweentheneighborandtheac/veuser’sitem

Usagedata

•  Baseline:CFusingusagedata•  Profilevectors

  Userprofiles  Itemprofiles

•  Noexplicitra/ngsavailable  Onlybinaryinforma/on(1or0)  Orrather:unary!

•  Similaritymetric  Cosinesimilarity

•  10‐foldcross‐valua/ontoop/mizek

UI

items

users

Results(usagedata)

BibSonomy Delicious BibSonomy CiteULike

UBCF+usagedata 0.0277 0.0046 0.0865 0.0746

IBCF+usagedata 0.0244 0.0027 0.0737 0.0887


•  Tagsareshorttopicaldescrip/onsofanitem(oruser)

•  Profilevectors  Usertagprofiles  Itemtagprofiles

•  Similaritymetrics  Cosinesimilarity

  Jaccardoverlap  Dice’scoefficient

Taggingdata

UT

tags

users

IT

tags

items

Results(taggingdata)


UBCF+usagedata 0.0277 0.0046 0.0865 0.0746

IBCF+usagedata 0.0244 0.0027 0.0737 0.0887

UBCF+taggingdata 0.0102 0.0017 0.0459 0.0449

IBCF+taggingdata 0.0370 0.0101 0.1100 0.0814


Findings(taggingdata)

•  CFwithtagoverlap  User‐basedCFperformssignificantlyworse  Item‐basedCFperformsmuchbejer

- Ouensta/s/callysignificantimprovements

  ExceptonCiteULike:CFwithouttagsbejer•  Similaritymetricrela/velyunimportant

  Cosinesimilarityslightlybejer

Comparisontorelatedwork

•  Randomwalkmodel(Clementsetal.,2008)  Createtransi/onmatrixbasedontripar/tefolksonomygraph  SimilartoFolkRank,butnowalksofinfinitelength

  Walklengthnisaparameter

•  Tag‐awarefusion(Tso‐Sujeretal.,2008)  Fusionofalgorithmsanddatarepresenta,ons  Usagedataandtaggingdata

- User‐basedCFextendUImatrixwithtagsasextraitems

-  Item‐basedCFextendUImatrixwithtagsasextrausers

  User‐basedCFanditem‐basedCF-  Fusetogetherpredic/ons

Comparisontorelatedwork

!"#$%&'"#()

*+,#$-./

0,#1%&'"#()

*+,#$-./

! "#

$#2

%!"#$"

%&#'"

&()"

%&#'"

%&#'"

&()"

!"#$"

&()"

Results


UBCF+usagedata 0.0277 0.0046 0.0865 0.0746

IBCF+usagedata 0.0244 0.0027 0.0737 0.0887

UBCF+taggingdata 0.0102 0.0017 0.0459 0.0449

IBCF+taggingdata 0.0370 0.0101 0.1100 0.0814

UBCF+fuseddata 0.0303 0.0057 0.0829 0.0739

IBCF+fuseddata 0.0468 0.0125 0.1280 0.1212

Tag‐awarefusion 0.0474 0.0166 0.1297 0.1268

Randomwalkmodel 0.0182 0.0003 0.0608 0.0536


Metadata‐basedrecommenda/on

•  Ques/on  Howcanweusethemetadatatogenerate(bejer)itemrecommenda/ons?

•  Content‐basedfiltering  Buildrepresenta/onsofthecontentinasystem

  Learnaprofileoftheuser’sinterests  Matchcontentrepresenta/onsagainsttheuser’sprofile

Reminder:whatdidwecollect?

•  Twotypesofmetadata  Intrinsicmetadata,i.e.,directlyrela/ngtothecontent

-  E.g.,<TITLE>,<DESCRIPTION>,<JOURNAL>,<AUTHOR>,...  Extrinsicmetadata,i.e.,administra/veinforma/on

-  E.g.,<PAGES>,<MONTH>,<EDITION>,…

Relatedwork

•  Commonapproaches  Informa/onretrieval  Machinelearning

•  Examples  TF∙IDFweigh/ng(Lang,1995;Whitman&Lawrence,2002)

  Personalinforma/onagents(Balabanovic,1998;Joachimsetal.,1997;Chiritaetal.,2006)

  NaiveBayes(Mooneyetal.,2000;DeGemmisetal.,2008)

  Linearregression(Alspectoretal.,1997)•  Nothingappliedtosocialbookmarkingsofar!

•  TakeanIRapproach:profile‐centricmatching  Buildrepresenta/onsofthecontentinasystem

-  Allmetadataassignedtoanitem→itemprofile

  Learnaprofileoftheuser’sinterests-  Collateallofuser’smetadataintoauserprofile

  Matchandrankitemprofilestouserprofiles-  LanguagemodelingwithJelinek‐Mercersmoothing

-  Stopwordfiltering,nostemming

Profile‐centricmatching

Profile‐centricmatching

!"#$%$%&'$()*'+",-.)/0123)'4/)"'+",-.)/

!"#$%&'(&)*"+(,-.*(/+)0

/$*$.#"$(5

*#(16$%&78 9

: 0 ;

< = ;

> ;

()/('+#$"/("#$%$%&'+#$"/

7

0

9

0

:

0

9

=

<

=

7

;

:

;

>

;

<

;

7

8

9

8

:

8

>

8

•  Problem  Biguserprofilewillmatchnearlyanything  Sacrificingprecisionforrecall

•  Differentlevelofgranularity:post‐centricmatching  Constructmetadatarepresenta/onsofeachpost

  Matcheachoftheuser’spostsagainstallotherposts  Match,rank,andaggregateallretrievedposts

Post‐centricmatching

Post‐centricmatching

!"#$%$%&'()*+*,-./0'1*0"2*'()*+*

!"#$%&'()*+,(-.*$/0(*1.,2

*$3$4#"$+5

3#+-6$%&

7

7

7

8

9'9'9

:

:

:

:

8

;

8

8

9'9'9

,

,

<

,

+0*+'(#$"*+"#$%$%&'(#$"*

7

,

8

,

;

,

8

<

=

<

7

>

;

>

?

>

=

>

7

:

8

:

;

:

?

:

Results


Profile‐centricmatching 0.0402 0.0014 0.1279 0.0987

Post‐centricmatching 0.0259 0.0036 0.1190 0.0455


•  Problemwithpost‐centricmatching:datasparseness

Hybridfiltering

•  Similaritybetweenusersanditemsbasedonmetadata  Plugthesesimilari/esintostandardk‐NNCFapproach!  User‐basedCFwithmetadata‐basedsimilari/es

-  Textualsimilaritybetweenuserprofiles

  Item‐basedCFwithmetadata‐basedsimilari/es-  Textualsimilaritybetweenitemprofiles

Results




Hybrid(UBCF+metadata) 0.0218 0.0039 0.0410 0.0608

Hybrid(IBCF+metadata) 0.0399 0.0017 0.1510 0.0746


Results(comparison)






BestCFrun 0.0370 0.0101 0.1100 0.0887


Results(comparison)






BestCFrun 0.0370 0.0101 0.1100 0.0887

Tag‐awarefusion 0.0474 0.0166 0.1297 0.1268


Findings

•  Content‐basedfiltering  Profile‐levelmatchingbejerthanpost‐level

•  Hybridfiltering  Item‐basedCFwithmetadatasimilari/esworksbest

•  Noclearwinneroveralldatasets

Datafusion

•  Ques/on  Canweimproveperformancebycombiningdifferentrecommenda/onalgorithms?

  Tenta/veanswer:yes!

•  Datafusionusedindifferentfields  Machinelearning  Informa/onretrieval

-  Collec/onfusion-  Resultsfusion

Combina/ontaxonomy

•  Burke(2002)definessevendifferenttechniques1.  Mixed(allshowntogether,interleaved)2.  Switching(pickone,dependingonthesitua/on)

3.  Featurecombina/on(combinesourcesforasinglealgorithm)

4.  Cascade(outputofalgorithm1isinputofalgorithm2)

5.  Featureaugmenta/on(outputalg.1isinputfeaturealg.2)6.  Meta‐level(modelalg.1isinputforalg.2)

7.  Weightedcombina/on(outputcombina/onof≥2alg.)-  SameasresultsfusioninIR

Whydoesdatafusionwork?

•  Problem  Recommenda/onistoocomplex  Individualsolu/oncannevercapturethiscompletely

•  Solu/on  Combinedifferentalgorithmsanddatarepresenta/ons

  Eachhighlightsadifferentaspectofthetask  Overlapbetweentheindividualrunsisevidenceofrelevance

Howdowecombine?

•  Score‐basedfusion  Differentalgorithmshavedifferentscoredistribu/ons  Scorenormaliza/oninto[0,1]range

•  Sixstandardcombina/ontechniquesfromIR  CombMAX(maxscoreperitem)

  CombMIN(minscoreperitem)  CombMED(medianscoreperitem)

  CombSUM(sumofscoresperitem)  CombMNZ(sumofscoresperitem×no.ofretrievingruns)

  CombANZ(sumofscoresperitem÷no.ofretrievingruns)

Howdowecombine?

•  Unweightedvs.weightedcombina/on  “Notallrecommenda/onalgorithmsarecreatedequal!”  Linearweigh/ngofindividualruns  Weightop/miza/onusingrandom‐restarthillclimbing

-  Stepsof0.1-  100itera/ons- Using10‐foldcross‐valida/on

Whatdowecombine?

•  Whataspectsofthetaskcanwevary?  Algorithms

- User‐basedCF-  Item‐basedCF

-  Content‐basedfiltering(profile‐andpost‐centricmatching)

- Hybridfiltering(CFwithmetadataoverlap)

  Datarepresenta/on- Usagedata-  Tags- Metadata

  Numberofrunscombined-  Canvaryfromtwotoeight

Whatdowecombine?

RunID #runs Descrip,on

FusionA 2 BestUBCFandIBCFrunswithusagedata

FusionB 2 BestUBCFandIBCFrunswithtagggingdata

FusionC 2 BestCFrunswithusageand/ortaggingdata(A+B)

Whatdowecombine?





FusionD 2 Bestprofile‐centricandpost‐centricmatchingruns

FusionE 2 BestUBCFandIBCFrunswithmetadatasimilarity

FusionF 2 Bestmetadata‐basedruns(D+E)

Whatdowecombine?








FusionG 2 Bestfolksonomicandbestmetadata‐basedrun(C+F)

Whatdowecombine?








FusionG 2 Bestfolksonomicandbestmetadata‐basedrun(C+F)

FusionH 4 AllfourbestCFrunswithusageand/ortaggingdata(A+B)

FusionI 4 Allfourbestmetadata‐basedruns(D+E)

FusionJ 8 Alleightbestruns(A+B+D+E)

Results

RunID BibSonomy Delicious BibSonomy CiteULike

FusionA 0.0362 0.0065 0.1017 0.0949

FusionB 0.0434 0.0105 0.1196 0.0952

FusionC 0.0482 0.0115 0.1593 0.1278

FusionD 0.0388 0.0038 0.1303 0.1008

FusionE 0.0514 0.0051 0.1596 0.0945

FusionF 0.0494 0.0056 0.1600 0.1136

FusionG 0.0539 0.0109 0.1539 0.1556

FusionH 0.0619 0.0092 0.1671 0.1286

FusionI 0.0565 0.0065 0.1749 0.1188

FusionJ 0.0695 0.0090 0.1983 0.1531


Comparison


UBCF+usage 0.0277 0.0046 0.0865 0.0757

UBCF+tags 0.0102 0.0017 0.0459 0.0449

IBCF+usage 0.0244 0.0027 0.0737 0.0887

IBCF+tags 0.0370 0.0101 0.1100 0.0814

Content‐based+profile 0.0402 0.0014 0.1279 0.0987

Content‐based+post 0.0259 0.0036 0.1190 0.0455



Bestfusionrun 0.0695 0.0115 0.1983 0.1556

%Improvement +72.9% +13.9% +31.3% +57.6%


Findings

•  Fusionworks!Butwhatworksbest?  Weightedfusion  Combiningdifferentalgorithms

  Combiningdifferentdatarepresenta/ons  Combiningahighernumberofruns

  CombMNZandCombSUM

•  Addi/onalanalysesshowedthat  Improvementsmostlyaprecision‐enhancingeffect  Duetobejerrankingofdocuments

•  Newques/on:whereisthesweetspot?  Performancevs.computa/on

•  Usingtagoverlapinitem‐basedCFworkswell  Easytoimplement/adapt

•  Metadata‐basedrecommenda/onouenbejerthanCF  Notsignificantly  Noclearwinningalgorithm

  Easiesttoimplementusingexis/ngsearchengine

•  Recommenderfusionispromising  Combinerunsthatcoverdifferentaspects

  Weightedfusionworksbest

  Combiningmore(butdifferent)runsworksbejer

Overallfindings

•  Large‐scalecomparisonofalgorithms

•  Online,user‐basedevalua/onofalgorithms•  Exploringotherrecommenda/ontasks

Futurework

Ques/ons?

Metadatafindings

•  Whatdidwetestintermsofmetadatafields?  Individualintrinsicfields  Allintrinsicfieldscombined

  Allintrinsicfields+allextrinsicfieldscombined

•  Metadata  Allintrinsicmetadatacombinedworksbest

  Bestfields:TAGS,TITLE,AUTHOR,URL,ABSTRACT  Extrinsicmetadatacontributeslijle

Documents

Comparing Recommendaon Algorithms for Social Bookmarkingtoinebogers.com/content/slides/201001-comparing-recsys-for-bookm… · Social bookmarking • Way of storing, organizing, and