Embed Size (px)
DESCRIPTION
This presentation is about our paper which was presented at the Hypertext conference 2012. In this paper, we investigate a methodology for measuring the influence of tag recommenders on the indexing quality in collaborative tagging systems. We propose to use the inter-resource consistency as an indicator of indexing quality. The inter-resource consistency measures the degree to which the tag vectors of indexed resources reflect how the users understand the resources. We use this methodology for evaluating how tag recommendations coming from (1) the popular tags at a resource or from (2) the user's own vocabulary influence the indexing quality. We show that recommending popular tags decreases the indexing quality and that recommending the user's own vocabulary increases the indexing quality. Links to the paper: http://dx.doi.org/10.1145/2309996.2310009 http://www.west.uni-koblenz.de/files/publications/dellschaft2012mti.pdf
Citation preview
Web Science & Technologies
University of Koblenz ▪ Landau, Germany
Measuring the Influence of Tag Recommenders on the Indexing Quality in Tagging Systems
Klaas Dellschaft [email protected]
Steffen Staab
Measuring the Influence of Tag Recommenders Klaas Dellschaft ([email protected])
Slide 2 of 21 http://west.uni-koblenz.de
Collaborative Tagging Systems
Objectives of tag recommenders: Improve indexing quality retrieval results Reduce tagging effort
Measuring the Influence of Tag Recommenders Klaas Dellschaft ([email protected])
Slide 3 of 21 http://west.uni-koblenz.de
Outline
Measures of indexing quality What to understand under “indexing quality”? Inter-resource consistency inter-indexer consistency
Evaluation of the measures Are the measures correlated with each other? User study: Apply measures for two recommenders
Evaluation results
Conclusions
Measuring the Influence of Tag Recommenders Klaas Dellschaft ([email protected])
Slide 4 of 21 http://west.uni-koblenz.de
Measures of Indexing Quality
Measuring the Influence of Tag Recommenders Klaas Dellschaft ([email protected])
Slide 5 of 21 http://west.uni-koblenz.de
What does “indexing quality” mean?
Tag
Vec
tors
des
crib
e
r1r1 r2
r2 r3r3
patents
humor
news
science
0
0
10
4
1v
0
6
8
0
2v
5
9
0
0
3v
Res
ou
rces
sim(v1, v2) sim(v2, v3)
user perceived similarity
Measuring the Influence of Tag Recommenders Klaas Dellschaft ([email protected])
Slide 6 of 21 http://west.uni-koblenz.de
Measures of indexing quality
Inter-resource consistency Compare resource similarity to the tag vector distance Requires external knowledge about similarity of resourcesDirect but sophisticated measure of indexing quality
Inter-indexer consistency Do users agree on common description for a resource? Assumption: Users select tags independent of each other Indirect but easy measure of indexing quality
Which measure to use for evaluating tag recommenders?
Measuring the Influence of Tag Recommenders Klaas Dellschaft ([email protected])
Slide 7 of 21 http://west.uni-koblenz.de
Research Hypotheses
Hypothesis: Inter-indexer consistency does not measure the influence of tag recommenders on the indexing quality!
Popular Tags: Suggest most popular tags of a resource H1a: Popular Tags increase the inter-indexer consistency H1b: Popular Tags decrease the inter-resource consistency
User Tags: Suggest all tags previously applied by the user H2a: User Tags lead to a decreased or unchanged inter-indexer
consistency H2b: User Tags increase the inter-resource consistency
The measures do not correlate when evaluating tag recommenders
Measuring the Influence of Tag Recommenders Klaas Dellschaft ([email protected])
Slide 8 of 21 http://west.uni-koblenz.de
Measuring Inter-Resource Consistency
Idea: Compare resource similarity and tag vector distanceai: Average distance to resources in the same cluster
bi: Average distance to resources in the closest other cluster
0-1 +1
resource
cluster of similar resources
inconsistent consistent even moreconsistent
),max( ii
iii ba
abs
Measuring the Influence of Tag Recommenders Klaas Dellschaft ([email protected])
Slide 9 of 21 http://west.uni-koblenz.de
Measuring Inter-Indexer Consistency
Idea: Do users agree on common description for a resource?Tag Reuse Rate
Average number of users who apply a tag Used in the related work
patents
fun
humor
news
0
2
4
8
0
2
6
8
Tag Reuse Rate: 4.7 5.3 7
0
0
6
8
Measuring the Influence of Tag Recommenders Klaas Dellschaft ([email protected])
Slide 10 of 21 http://west.uni-koblenz.de
Evaluation
Measuring the Influence of Tag Recommenders Klaas Dellschaft ([email protected])
Slide 11 of 21 http://west.uni-koblenz.de
Experimental Setup
Objective: Are inter-resource and inter-indexer correlated if tag
recommendations are given?
Task given to users: Assign keywords to 10 web pages. After tagging, cluster web pages according to their
similarity ( inter-resource consistency).
Three different experimental conditions:1) No Suggestions2) User Tags3) Popular Tags
Further divided into an English and German user group
Measuring the Influence of Tag Recommenders Klaas Dellschaft ([email protected])
Slide 12 of 21 http://west.uni-koblenz.de
Suggestion of Popular Tags – Screenshot
Measuring the Influence of Tag Recommenders Klaas Dellschaft ([email protected])
Slide 13 of 21 http://west.uni-koblenz.de
Clustering of Similar Web Pages – Screenshot
Measuring the Influence of Tag Recommenders Klaas Dellschaft ([email protected])
Slide 14 of 21 http://west.uni-koblenz.de
Results
Measuring the Influence of Tag Recommenders Klaas Dellschaft ([email protected])
Slide 15 of 21 http://west.uni-koblenz.de
Sizes of the Tagging Data Set
#Users #Tags #TAS #TAS / #User
No Suggestions 74 706 2134 28.84
Popular Tags 78 531 2228 28.56
User Tags 79 466 1507 19.08
German User Group:
English User Group:
#Users #Tags #TAS #TAS / #User
No Suggestions 115 973 3150 27.39
Popular Tags 118 550 3003 25.45
User Tags 118 819 2919 24.74
Measuring the Influence of Tag Recommenders Klaas Dellschaft ([email protected])
Slide 16 of 21 http://west.uni-koblenz.de
The Clustering Data Set
In average, each user identified 4.59 clusters Overall, 146 distinct clusters have been identified 11 most frequent clusters 70% of the data
The web pages cover ~7 topics 3 web pages are on the border between two topics
Measuring the Influence of Tag Recommenders Klaas Dellschaft ([email protected])
Slide 17 of 21 http://west.uni-koblenz.de
Differences in the Topical Clusters
English Popular Tags condition has to be excluded
The Onion + BBC News
The Onion + Patents Humor
No SuggestionsPopular TagsUser Tags
Cluster probabilities in English experiment
Measuring the Influence of Tag Recommenders Klaas Dellschaft ([email protected])
Slide 18 of 21 http://west.uni-koblenz.de
Measuring the Inter-Resource Consistency
H1a: Popular Tags decrease the inter-resource consistency H2a: User Tags increase the inter-resource consistency
Expectation: E(spt,i) < E(sns,i) < E(sut,i)
E(spt,i) E(sns,i) E(sut,i)
German Users 0.1474 0.1847 0.2367
English Users N/A 0.1713 0.1915
(All differences are significant!)
Measuring the Influence of Tag Recommenders Klaas Dellschaft ([email protected])
Slide 19 of 21 http://west.uni-koblenz.de
Measuring the Inter-Indexer Consistency
H1b: Popular Tags increase the inter-indexer consistency H2b: User Tags lead to a decreased or unchanged
inter-indexer consistency
Expectation: E(trpt,i) > E(trns,i) ≥ E(trut,i)
E(trpt,i) E(trns,i) E(trut,i)
German Users 3.60 2.44 2.39*
English Users 4.67 2.76 2.68*
* Differences between E(trns,i) and E(trut,i) not significant
Measuring the Influence of Tag Recommenders Klaas Dellschaft ([email protected])
Slide 20 of 21 http://west.uni-koblenz.de
Conclusions
Measures of indexing quality Inter-resource consistency Inter-indexer consistencyMeasures do not correlate if recommendations are givenOnly inter-resource consistency can be used
Popular Tags Do not lead to consistent descriptions across resources Are rather counterproductive for indexing resources
User Tags Lead to consistent descriptions across resource Consolidate the personomy of users
Measuring the Influence of Tag Recommenders Klaas Dellschaft ([email protected])
Slide 21 of 21 http://west.uni-koblenz.de
Paper:K. Dellschaft & S. Staab. Measuring the Influence of Tag
Recommenders on the Indexing Quality in Tagging Systems. Proceedings of the Hypertext Conference, 2012http://dl.acm.org/citation.cfm?id=2310009
Experimental Interface:http://userpages.uni-koblenz.de/~klaasd/experiment/
Data Set:http://west.uni-koblenz.de/Research/DataSets/tagging-experiment/
