View
153
Download
1
Category
Preview:
Citation preview
How do Gain and Discount Functions Affect the
Correlation between DCG and User Satisfaction?Julián Urbano Mónica Marrero
ECIR 2015
Vienna, March 30th
Discount d(i ; k) Gain g(r)
Zipfian: 1/� Linear: �
Linear: �� � � � 1/� Exp(2): 2� � 1
Constant: 1 Exp(3): 3� � 1
Log(2): 1/ log� � � 1 Exp(5): 5� � 1
Log(3): 1/ log� � � 2 Bin(1): I�� � 1�
Log(5): 1/ log��� � 4 Bin(2): I�� � 2�
Discount functions
Rank i
Dis
count d(i)
0.0
0.2
0.4
0.6
0.8
1.0
1 2 3 4 5
ZipfianLinear
Constant
Log(2)
Log(3)
Log(5)
Gain functions
Relevance r
Gain
g(r
)
05
10
15
20
25
0 1 2
LinearExp(2)
Exp(3)
Exp(5)
Bin(1) Bin(2)
Documents
Information Need
Real World Cranfield
IR System
Topic
Relevance Judgments
IR System
Documents
GAP
DCG
ERR
StaticComponent
Dynamic Component
TestCollection
EffectivenessMeasures
Time to complete task, Idle time, Success rate, Frustration, Satisfaction,
Ease of use, Ease of learning…
Precision, Average Precision, Reciprocal Rank,Q-measure, Discounted Cumulative Gain,Rank-Biased Precision, Time-Biased Gain…
Live Observation
What Gain and Discount for DCG arebetter to predict user satisfaction?• First, let’s normalize DCG scores (this is not nDCG!)
• One system with DCG=φ. What does it mean?• Intuition: φ·100% of users will be satisfied• P(Sat|DCG= φ)= φ
• Two systems with ΔDCG=Δφ. What does it mean?• Intuition: users will prefer the (supposedly) better one• P(Pref|ΔDCG=Δφ)=1
P(Sat) and P(Pref) depend on the systems,not on how we evaluate them. Yet, there are
many different ways to compute effectiveness
Experiment• Collect user preferences between two systems• Map DCG onto P(Sat)• Map ΔDCG onto P(Pref)
• Music recommendation task• Ad-hoc, informational, enjoyable by assessors• Preferences less confounded by interface effects
• All data from MIREX (TREC-like for Music IR)• Datasets from 2007–2012• 3-point relevance scale: {0, 1, 2}
• 4115 examples• Uniformly covering the [0,1] range of ΔDCG• 432 unique queries• 5636 unique documents
• Crowdsourced with Crowdflower• Trap examples for quality control• 113 subjects
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Zipfian discount
Difference in DCG
Pro
bability
that users
agre
e
Gains
Linear
Exp(2)
Exp(3)
Exp(5)
Bin(1)
Bin(2)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Linear discount
Difference in DCG
Pro
bability
that users
agre
e
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Log(2) discount
Difference in DCG
Pro
bability
that users
agre
e
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Log(3) discount
Difference in DCG
Pro
bability
that users
agre
e
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Log(5) discount
Difference in DCG
Pro
bability
that users
agre
e
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Constant discount
Difference in DCG
Pro
bability
that users
agre
e
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Zipfian discount
DCG
Pro
bability
of user
satis
factio
n
Gains
Linear
Exp(2)
Exp(3)
Exp(5)
Bin(1)
Bin(2)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Linear discount
DCG
Pro
bability
of user
satis
factio
n
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Log(2) discount
DCG
Pro
bability
of user
satis
factio
n
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Log(3) discount
DCG
Pro
bability
of user
satis
factio
n
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Log(5) discount
DCG
Pro
bability
of user
satis
factio
n
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Constant discount
DCG
Pro
bability
of user
satis
factio
n
Results (1 system): DCG predicting user satisfaction Results (2 systems): ΔDCG predicting user preference
Results: bias of Gain and Discount functions
• Diagonal: how far is P(Sat|DCG) from the ideal diagonal?• Intuitiveness of DCG scores
• Endpoint: how far is P(Sat|DCG) from the ideal 0% and 100%?• User disagreement and goodness of the DCG user model
• Top: how far is P(Pref|ΔDCG) from the ideal 100%?• Discriminative power
Summary and Implications
• New method to map system effectiveness onto user satisfaction• Sample application to DCG for a music recommendation task
• Gain functions that emphasize highly relevant documents underestimate user satisfaction. Linear gain is better than exponential
• All discount functions bias the prediction of user satisfaction• This task might be too enjoyable to observe discount effect
• Size (of the DCG difference) does matter• Non-parametric statistics (eg. Sign test, Wilcoxon test) and just looking at
the ranking of systems (eg. Kendall τ) oversimplify the evaluation problem• Zero-point null hypothesis testing (ie. H
0 : ΔDCG=0) is not reasonable
• Future work will investigate this method for Text IR• Provide a common framework, based on P(Sat) and P(Pref), to evaluate
with informational and navigational queries using appropriate measures
Data and code
available online
���@� �∑ � �� ⋅ � �; ��
���
∑ � �� ⋅ � �; ��
���
0.06
0.10
0.14
Diagonal bias
Bias
Bin(2)
Bin(1)
Exp(5)
Exp(3)
Exp(2)
Linear
0.06
0.10
0.14
Zipfia
n
Linear
Log(2)
Log(3)
Log(5)
Constant
0.06
0.10
0.14
Gain Discount
0.14
0.18
0.22
Endpoint bias
Bias
Bin(2)
Bin(1)
Exp(5)
Exp(3)
Exp(2)
Linear
0.14
0.18
0.22
Zipfia
n
Linear
Log(2)
Log(3)
Log(5)
Constant
0.14
0.18
0.22
Gain Discount
0.46
0.50
0.54
Top bias
Bias
Bin(2)
Bin(1)
Exp(5)
Exp(3)
Exp(2)
Linear
0.46
0.50
0.54
Zipfia
n
Linear
Log(2)
Log(3)
Log(5)
Constant
0.46
0.50
0.54
Gain Discount
Recommended