Embed Size (px)
Citation preview
Elici%ng Informa%on from the Crowd a
Part of the EC’13 Tutorial on Social Compu%ng and User-‐Generated Content
Yiling Chen Harvard University
June 16, 2013
Roadmap
▶ Elici%ng informa%on for events with verifiable outcomes ▶ Proper scoring rules ▶ Market scoring rules
▶ Elici%ng unverifiable informa%on ▶ Peer predic%on ▶ Bayesian Truth Serum ▶ Robust Bayesian Truth Serum
How to Evaluate Weather Forecasts
▶ Specifica%on of a scale of goodness for weather forecasts
▶ “… the forecaster may oTen find himself in the posi%on of choosing to … let it (the verifica%on system) do the forecas%ng for him by ‘hedging’ or ‘playing the system.’ ”
▶ “one essen%al criterion for sa%sfactory verifica%on is that the verifica%on scheme should influence the forecaster in no undesirable way”
[Brier 1950]
p 1� p
The Brier Scoring Rule [Brier 1950]
p 1� p
▶
▶ Expected score of predic%on given belief
▶ Predic%ng maximizes the expected score
S(p, rainy) = 1� p2S(p, sunny) = 1� (1� p)2
S(p, q) = q S(p, sunny) + (1� q)S(p, rainy)
= 1� q(1� p)2 � (1� q)p2
= 1� q + q2 � (p� q)2
p q
argmax
pS(p, q) = q
q
S(p, q) increases as |p� q| decreases.
The Brier Scoring Rule
▶ State ▶ Predic%on ▶ Brier score for predic%on in state is
▶ Also called the quadra%c scoring rule
! 2 {1, 2, . . . , n}
p
p = (p1, p2, . . . , pn)
Parameters, b >0. Affine transforma%on does not change incen%ves.
! -‐th indicator vector !
S(p,!) = a! � b ||e! � p||2
Other Strictly Proper Scoring Rules
▶ Logarithmic scoring rule
▶ Spherical scoring rule
S(p,!) =p!||p||
S(p,!) = log p!
Proper Scoring Rules
▶ Scoring rule ▶ Expected score for predic%on given belief :
S(p,!)
p q
S(p,q) :=nX
!=1
q!S(p,!)
A scoring rule is proper if and only if .
It is strictly proper if the inequality is strict unless .
S(p,!)S(q,q) � S(p,q)
q = p
predic%on
belief
Proper scoring rules are dominant-‐strategy incen%ve compa%ble for risk-‐neutral agents.
Geometric Interpreta%on S(p, sunny) = 1� (1� p)2
S(p, rainy) = 1� p2G(p) = S(p, p) = (p� 1/2)2 + 3/4
10 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
1
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
Probability for Sunny
pq
G
G(p) = S(p, p)
G(q) = S(q, q)
S(p, q)
S(p, sunny)
S(p, rainy)
Brier Scoring Rule
Bregman Divergence DG(q, p) = G(q)�G(p)�rG(p)(q � p)
McCarthy, Savage Characteriza%on ［McCarthy 1956］[Savage 1971]
▶ Expected score
▶ In terms of Bregman Divergence, for differen%able ，
A scoring rule is (strictly) proper if and only if
,
where is a (strictly) convex func%on and is a subgradient of and is its component.
S(p,!)
rG(p)
S(p,!) = G(p)�rG(p) · p+rG!(p)
rG!(p) !-thGG
S(p,!) = G(p)�rG(p) · p+rG(p) · e!
= G(e!)� (G(e!)�G(p)�rG(p) · (e! � p))
= G(e!)�DG(e!,p)
S(p,p) =X
!
p! (G(p)�rG(p) · p+rG!(p)) = G(p)
G
Common Strictly Proper Scoring Rules
▶ Brier scoring rule
▶ Logarithmic scoring rule
▶ Spherical scoring rule
S(p,!) = 1� ||e! � p||2
G(p) = ||p||2 DG(q,p) = ||q� p||2
S(p,!) = log p!
G(p) =X
!
p! log p! DG(q,p) =X
!
q! log
q!p!
S(p,!) =p!||p||
G(p) = ||p|| DG(q,p) = ||q||� q · p||p||
Other Work on Scoring Rules
▶ Proper scoring rules for con%nuous variables [Matheson and Winkler 1976; Gnei%ng and RaTery 2007]
▶ Proper scoring rules for proper%es of distribu%ons (e.g. mean, quin%les) [Savage 1971; Lambert et al. 2008; Abernethy and Frongillo 2012]
▶ Scoring rules for more complex environments ▶ Agents can affect the event outcome [Shi et al. 2009; Bacon et al. 2012] ▶ A decision will be made based on the elicited informa%on [Othman and Sandholm. 2010; Chen and Kash 2011; Bou%lier 2012]
One Expert à Mul%ple Experts
Market Scoring Rules (MSR) [Hanson 2003, 2007] ▶ Sequen%al, shared version of proper scoring rules ▶ Reward improvements on predic%on
▶ Select a proper scoring rule ▶ Market opens with an ini%al predic%on ▶ Par%cipants sequen%ally change the market predic%on ▶ Par%cipant who changes the predic%on from to receives payment
S(p,!)
p0
pt�1 pt
S(pt,!)� S(pt�1,!)
An Example: LMSR Market
5log(0.6)-‐5log(0.5)
5log(0.4)-‐5log(0.5)
5log(0.8)-‐5log(0.6)
5log(0.4)-‐ 5log(0.8)
5log(0.9)-‐ 5log(0.4)
5log(0.2)-‐5log(0.4)
5log(0.6)-‐ 5log(0.2)
5log(0.1)-‐ 5log(0.6)
The mechanism only pays the final par%cipant!
0.5 0.6 0.8 0.9 0.4
0.5 0.4 0.2 0.6 0.1
5log(0.9)-‐5log(0.5)
5log(0.1)-‐5log(0.5)
t
S(p,!) = 5 log p!
Proper%es of MSR
▶ Bounded loss (subsidy) ▶ Eg. Loss bound of LMSR with is
▶ Incen%ve compa%ble for myopic par%cipants
▶ Recent work studies informa%on aggrega%on in MSR with forward-‐looking par%cipants [Chen et al. 2010; Ostrovsky 2012; Gao et al. 2013] ▶ Market as a Bayesian extensive-‐form game ▶ Condi%ons under which informa%on is fully aggregated in the
market
S(p,!) = b log p! b log n
Predic%on and Trading
Buy this contract if
Sell this contract if
p 1� p
$1 iff
price < p
price > p
MSR as Automated Market Makers [Chen and Pennock 07] ▶ One contract for each outcome
▶ Payments are determined by a cost poten%al func%on ▶ is the current number of shares of the contract for
outcome that have been purchased ▶ Current cost of purchasing a bundle of shares is
▶ Instantaneous prices Market Predic%on
C(Q)
R
C(Q+R)� C(Q)
Qi
!
$1 iff !
p!(Q) =@C(Q)
@Q!
Example ▶ LMSR with
▶ Cost func%on
▶ Price func%ons
S(p,!) = b log p!
p!(Q) = log
eQ!/b
P!0 eQ!0/b
C(Q) = b logX
!
eQ!/b
LMSR Equivalence
▶ Profit of a par%cipant who changes to in state (
¯Q! �Q!)� (C(
¯Q)� C(Q))
= (
¯Q! � C(
¯Q))� (Q! � C(Q))
= (log eQ! � log
X
!0
eQ!0)� (log eQ! � log
X
!0
eQ!0)
= log
eQ!
P!0 eQ!0
� log
eQ!
P!0 eQ!0
= log p!( ¯Q)� log p!(Q)
= S(p( ¯Q),!)� S(p(Q),!)
C(Q) = log
X
!
eQ!
p!(Q) =
eQ!
P!0 eQ!0
S(p,!) = log p!Q Q !
Cost Func%on MM Strictly Proper MSR [Abernethy et al. 2013]
▶ An MSR using a strictly proper scoring rule ▶ The expected scoring func%on ▶ The corresponding cost func%on MM
▶ Profit equivalence holds when is in the rela%ve interior of the probability simplex
S(p,!)
G(p) = S(p,p)
C(Q) = supp2�n
Q · p�G(p)
p(Q) = rG(Q) = arg max
p2�n
Q · p�G(p)
Conjugate Duality
p(Q)
Elici%ng Unverifiable Informa%on
Unverifiable Informa%on
Is this website age-‐appropriate for children? • Yes • No
The General Sesng
▶ State ▶ E.g. whether this website is really age-‐appropriate or not
▶ agents ▶ Each agent receives a private signal ▶ E.g. whether agent thinks the website age-‐appropriate
▶ Prior distribu%on is common knowledge for all agents
▶ Goal: truthfully revealing their private signal is a strict Bayesian Nash equilibrium
! 2 {1, 2, . . . , n}
i si 2 Sm � 2
Pr(!, s1, . . . , sm)
si
i
The Peer Predic%on Mechanism [Miller et al. 2005]
▶ Require that the mechanism knows the prior ▶ Each agent has a reference agent
sj
si Mechanism
Pr(s1, . . . , sm)sj
si
Pr(si|sj)
Pr(sj |si)
S(Pr(si|sj), si)
S(Pr(sj |si), sj)Payment
Payment
The Peer Predic%on Mechanism [Miller et al. 2005]
▶ is a strictly proper scoring rule ▶ Technical assump%on: Stochas%c Relevance
sj
si Mechanism
Pr(s1, . . . , sm)sj
si
Pr(si|sj)
Pr(sj |si)
S(Pr(si|sj), si)
S(Pr(sj |si), sj)Payment
Payment
S(p, s)
Pr(sj |si) 6= Pr(sj |s0i), 8si 6= s0i
Truthful repor%ng is a strict Bayesian Nash equilibrium.
Requiring the mechanism know the prior is undesirable!
Bayesian Truth Serum (BTS) [Prelec 2004]
▶ Each agent is asked for two reports ▶ Informa%on report: is an indicator vector, having
value 1 at its element if agent reports signal ▶ Predic%on report: is agent ’s predic%on of the
frequency of reported signals in the popula%on
▶ The mechanism calculates
▶ Score for agent :
Is this website age-‐appropriate for children? (Yes, No)
x
i
k-th kiyi i
xk = limn!1
1
n
nX
i=1
x
ik log
¯yk = lim
n!1
1
n
nX
i=1
logyik
X
k
x
ik log
¯
xk
¯
yk+
X
k
¯
xk logy
ik
¯
xk
i
Bayesian Truth Serum (BTS)
▶ Technical assump%ons: condi%onal independence of signals and stochas%c relevance
X
k
x
ik log
¯
xk
¯
yk+
X
k
¯
xk logy
ik
¯
xk
When , truthful repor%ng is a strict Bayesian Nash equilibrium.
For sufficiently large , truthful repor%ng is a strict Bayesian Nash equilibrium. But depends on the prior.
n ! 1
n
Predic%on Score Informa%on Score
Log scoring rule! Truthful predic%on reports
BTS: “Surprisingly Common” ▶ True signal maximizes in expecta%on
Consider an agent with “yes” signal
0
0.2
0.4
0.6
0.8
1
Prior Agents with "Yes"
Agents with "No"
Geometric mean
Predic%on for "Yes"
Predic%on for "No"
X
k
x
ik log
¯
xk
¯
yk
Robust Bayesian Truth Serum (RBTS) [Witkowski and Parkes 2012a]
▶ Works for small popula%ons ▶ RBTS assumes binary signals ▶ Each agent is asked for two reports ▶ Informa%on report: reported signal ▶ Predic%on report: is the predic%on of the
frequency of signal 1
▶ Each agent has two reference agents
S = {0, 1}
xi 2 {0, 1}i
yi 2 [0, 1]
Robust Bayesian Truth Serum (RBTS) [Witkowski and Parkes 2012a]
sj
si
sk
Mechanism
(xk, yk)
(xi, yi)
(xj , yj) {y0i =yj � d if xi = 0
yj + d if xi = 1
d = min{yj , 1� yj}
S(y0i, xk) + S(yi, xk), S being the Brier scoring rule
Truthful repor%ng is a strict Bayesian Nash equilibrium for . n � 3
Shadowing ▶ S is the Brier scoring rule
S(y0i, xk) + S(yi, xk) Predic%on Score Informa%on Score
{y0i =yj � d if xi = 0
yj + d if xi = 1
d = min{yj , 1� yj}
Shadowing
d d
0 1 yjx x
Predic%on of agent given signal 0 and yj
i Predic%on of agent given signal 1 and yj
i
Other Related Work
▶ Many improvements on the original peer predic%on [Jurca and Fal%ngs 2006, 2007, 2008, 2011]
▶ Private prior peer predic%on [Witkowski and Parkes 2012b]
▶ RBTS for non-‐binary signals [Radanovic and Fal%ngs, 2013]
References ▶ J. Abernethy, Y. Chen, and J. W. Vaughan, 2013. Efficient Market Making via Convex
Op%miza%on, and a Connec%on to Online Learning. ACM Transac%ons on Economics and Computa%on. Vol. 1, no. 2, pp. 12:1-‐12:39.
▶ J. D. Abernethy and R. M. Frongillo. 2012. A Characteriza%on of Scoring Rules for Linear Proper%es. COLT’12.
▶ D. F. Bacon, Y. Chen, I. A. Kash, D. C. Parkes, M. Rao, and M. Sridharan. 2012. Predic%ng Your Own Effort. AAMAS’12.
▶ C. Bou%lier. 2012. Elici%ng forecasts from self-‐interested experts: scoring rules for decision makers. AAMAS’12.
▶ G. W. Brier. 1950. Verifica%on of forecasts expressed in terms of probability. Monthly Weather Review 78(1): 1-‐3.
▶ Y. Chen, S. Dimitrov, R. Sami, D. M. Reeves, D. M. Pennock, R. D. Hanson, L. Fortnow, and R. Gonen. 2010. Gaming predic%on markets: equilibrium strategies with a market maker. Algorithmica, 58: 930–969.
▶ Y. Chen and D. M. Pennock, 2007. A U%lity Framework for Bounded-‐Loss Market Makers. UAI'07.
▶ Y. Chen and I. A. Kash. 2011. Informa%on elicita%on for decision making. AAMAS’11.
References (Cont.) ▶ X. A. Gao, J. Zhang, Y. Chen. 2013. What You Jointly Know Determines How You Act -‐ Strategic
Interac%ons in Predic%on Markets, EC’13. ▶ T. Gnei%ng and A. E. RaTery. 2007. Strictly proper scoring rules, predic%on, and es%ma%on.
Journal of the American Sta%s%cal Associa%on 102, 477, 359–378. ▶ R. D. Hanson. 2003. Combinatorial informa%on market design. Informa%on Systems Fron%ers
5(1), 107–119.
▶ R. D. Hanson. 2007. Logarithmic market scoring rules for modular combinatorial informa%on aggrega%on. Journal of Predic%on Markets, 1(1), 1–15.
▶ R. Jurca and B. Fal%ngs, 2006. Minimum Payments that Reward Honest Reputa%on Feedback. EC'06.
▶ R. Jurca and B. Fal%ngs, 2007. Collusion Resistant, Incen%ve Compa%ble Feedback Payments. EC'07.
▶ R. Jurca and B. Fal%ngs.Incen%ves For Expressing Opinions in Online Polls, 2008. EC'08. ▶ R. Jurca and B. Fal%ngs 2011. Incen%ves for Answering Hypothe%cal Ques%ons. Workshop on
Social Compu%ng and User Generated Content at EC'11. ▶ N. S. Lambert, D. M. Pennock, and Y. Shoham. 2008. Elici%ng proper%es of probability
distribu%ons. EC '08.
▶ J. E. Matheson and R. L. Winkler. 1976. Scoring rules for con%nuous probability distribu%ons. Management Science 22, 1087–1096.
References (Cont.) ▶ J. McCarthy. 1956. Measures of the value of informa%on. PNAS: Proceedings of the Na%onal
Academy of Sciences of the United States of America, 42(9): 654-‐655.
▶ N. Miller, P. Resnick, and R. Zeckhauser 2005, Management Science, 51(9): 1359-‐1373.
▶ A. Othman and T. Sandholm. 2010. Decision rules and decision markets. AAMAS’10.
▶ M. Ostrovsky. 2012. Informa%on aggrega%on in dynamic markets with strategic traders. Econometrica, 80(6): 2595-‐2648.
▶ D. Prelec, 2004. A Bayesian Truth Serum for Subjec%ve Data. Science, 306 (5695): 462-‐466.
▶ G. Radanovic and B. Fal%ngs, 2013. A Robust Bayesian Truth Serum for Non-‐Binary Signals. AAAI’13.
▶ L. J. Savage. 1971. Elicita%on of personal probabili%es and expecta%ons. Journal of the American Sta%s%cal Associa%on, 66(336): 783-‐801.
▶ P. Shi, V. Conitzer, and M. Guo. 2009. Predic%on mechanisms that do not incen%vize undesirable ac%ons. WINE’09.
▶ J. Witkowski and D. C. Parkes, 2012a. A Robust Bayesian Truth Serum for Small Popula%ons. AAAI'12.
▶ J. Witkowski and D. C. Parkes, 2012b. Peer Predic%on without a Common Prior. EC'12.
