Robust Bayesian Truth Serum - Harvard · PDF fileRobust Bayesian Truth Serum Presentation by Mark Bun and Bo Waggoner 2012-12-03 Presentation by Mark Bun and Bo Waggoner Robust Bayesian

Introduction and SettingRBTS and Shadowing

PP Without Common PriorSummary: Human Computation Mechanisms

Robust Bayesian Truth Serum

Presentation by Mark Bun and Bo Waggoner

2012-12-03

Presentation by Mark Bun and Bo Waggoner Robust Bayesian Truth Serum



Outline

1 Introduction and SettingRecap: Human Computation Mechanisms so farThe RBTS ApproachSetting

2 RBTS and ShadowingShadowingRBTS

3 PP Without Common Prior

4 Summary: Human Computation MechanismsAssumptions and Results




Recap: Human Computation Mechanisms so farThe RBTS ApproachSetting

Recap

Peer prediction:

Elicit?

Rewards?

Bayesian Truth Serum:

Elicit?

Rewards?





Recap

Peer prediction:

Elicit?

Rewards?

Bayesian Truth Serum:

Elicit?

Rewards?





Big problem with implementing PP?

Mechanism needs to know the information structure!

Big problem with BTS?

Requires n→∞!

Other problems with BTS? Does RBTS resolve them?








Requires n→∞!









Requires n→∞!









Requires n→∞!









Requires n→∞!






Goal: Truthful mechanism for any n that doesn’t rely on themechanism knowing the information structure.

Approach:

Recall: Elicit two-part report (prediction and signal)

Which one is easy to incentivize and which is difficult?

To elicit the signal: Use shadowing!






Approach:









Approach:








Belief system

Consists of state T drawn from {1, . . . ,m} and signals Sdrawn from {l , h} = {0, 1}Agents have common prior Pr[T = t] and Pr[S = h|T = t].

“Impersonally informative” : For every i , j , k , write

p{si} = Pr[Sj = h|Si = si ]

p{si ,sj} = Pr[Sk = h|Si = si ,Sj = sj ]





Belief system

Consists of state T drawn from {1, . . . ,m} and signals Sdrawn from {l , h} = {0, 1}

Agents have common prior Pr[T = t] and Pr[S = h|T = t].








Belief system









Belief system


“Impersonally informative”

: For every i , j , k , write







Belief system









Belief system









Admissibility

Admissible prior:

1 At least two states (m ≥ 2)

2 Every state has positive probability (Pr[T = t] > 0)

3 Assortative property:Pr[S = h|T = 1] < · · · < Pr[S = h|T = m]

4 Fully mixed: 0 < Pr[S = h|T = t] < 1

Lemma 6: For an admissible prior,

1 > p{h,h} > p{h} > p{h,l} = p{l ,h} > p{l} > p{l ,l} > 0





Admissibility

Admissible prior:











Admissibility

Admissible prior:











Admissibility

Admissible prior:











Admissibility

Admissible prior:











Admissibility

Admissible prior:











Admissibility

Admissible prior:










ShadowingRBTS

What does shadowing solve?

Truthfully elicit signals instead of beliefs

ω ∈ {0, 1} a binary future event

Agent i draws a signal Si ∈ {0, 1} = {l , h}How do we get agent i to truthfully reveal Si?




ShadowingRBTS








ShadowingRBTS








ShadowingRBTS




Agent i draws a signal Si ∈ {0, 1} = {l , h}

How do we get agent i to truthfully reveal Si?




ShadowingRBTS








ShadowingRBTS

Naıve approach

1 Choose a strictly proper binary scoring rule R(y , ω)

2 Ask agent i for signal report xi ∈ {0, 1} using R(xi , ω)

Might not work if i ’s beliefs p{Si} about ω are far from xi !




ShadowingRBTS

Naıve approach







ShadowingRBTS

Naıve approach







ShadowingRBTS

Naıve approach







ShadowingRBTS

Solution: Use a reference prediction

1 Let y ∈ (0, 1) be an arbitrary prediction report

2 Set δ = min(y , 1− y)

3 Create a shadow report for i :

y ′i =

{y + δ if xi = 1y − δ if xi = 0

4 Score agent xi using the quadratic scoring rule Rq(y ′i , ω)




ShadowingRBTS



2 Set δ = min(y , 1− y)


y ′i =

{y + δ if xi = 1y − δ if xi = 0





ShadowingRBTS



2 Set δ = min(y , 1− y)


y ′i =

{y + δ if xi = 1y − δ if xi = 0





ShadowingRBTS



2 Set δ = min(y , 1− y)


y ′i =

{y + δ if xi = 1y − δ if xi = 0





ShadowingRBTS



2 Set δ = min(y , 1− y)


y ′i =

{y + δ if xi = 1y − δ if xi = 0





ShadowingRBTS

When and why does shadowing work?

Intuition: Truthfully reporting xi = Si pulls the referenceprediction toward i ’s posterior

Selten ’98: Let Y ⊂ [0, 1]. If agent i has beliefs p, then shemaximizes her expected score Rq(y , ω) over y ∈ Y byminimizing |y − p|.Lemma 8: Suppose agent i has observed signals I ∈ {l , h}kand Si . If p{l ,I} < y < p{h,I}, then agent i should truthfullyreport xi = Si .




ShadowingRBTS







ShadowingRBTS



Selten ’98: Let Y ⊂ [0, 1]. If agent i has beliefs p, then shemaximizes her expected score Rq(y , ω) over y ∈ Y byminimizing |y − p|.

Lemma 8: Suppose agent i has observed signals I ∈ {l , h}kand Si . If p{l ,I} < y < p{h,I}, then agent i should truthfullyreport xi = Si .




ShadowingRBTS







ShadowingRBTS

What’s missing?

How do we pick y to guarantee p{l ,I} < y < p{h,I}?

How do we pick ω when there is no ground truth?




ShadowingRBTS

What’s missing?






ShadowingRBTS

What’s missing?






ShadowingRBTS


Idea: Shadowing + reference raters

Information report: xi ∈ {0, 1} represents agent i ’s signal

Prediction report: yi ∈ [0, 1] represents agent i ’s predictionfor the frequency of h signals




ShadowingRBTS








ShadowingRBTS








ShadowingRBTS








ShadowingRBTS

Scoring

For each agent i , pick reference raters j , k

Set y = yj and ω = xk .

Create a shadow report for i :

y ′i =

{yj + δ if xi = 1yj − δ if xi = 0

where δ = min(yj , 1− yj)

RBTS score:

Rq(y ′i , xk)︸︷︷︸information score

+ Rq(yi , xk)︸︷︷︸prediction score




ShadowingRBTS

Scoring




y ′i =



RBTS score:






ShadowingRBTS

Scoring




y ′i =



RBTS score:






ShadowingRBTS

Scoring




y ′i =



RBTS score:






ShadowingRBTS

Scoring




y ′i =



RBTS score:






ShadowingRBTS

Incentive compatibility

If n ≥ 3 agents have signals drawn from an admissiblecommon prior, then reporting truthfully is a strictBayes-Nash equilibrium.

Why should agents report their predictions truthfully?

How about signals? Enough to show that admissibility=⇒ p{l ,I} < yj < p{h,I}.




ShadowingRBTS








ShadowingRBTS








ShadowingRBTS




How about signals?

Enough to show that admissibility=⇒ p{l ,I} < yj < p{h,I}.




ShadowingRBTS








ShadowingRBTS




Case 1: yj = p{h} =⇒ p{l ,h} < yj = p{h} < p{h,h}

Case 2: yj = p{l} =⇒ p{l ,l} < yj = p{l} < p{h,l}

Recap: p{l ,Sj} < yj < p{h,Sj}, so agent i should truthfullyreveal xi = Si




ShadowingRBTS










ShadowingRBTS










ShadowingRBTS










ShadowingRBTS










ShadowingRBTS

Properties of RBTS

Works for any number of agents n ≥ 3

Scores are well-defined between 0 and 2. Participation is “expost individually rational”.

Numerically robust




ShadowingRBTS

Properties of RBTS



Numerically robust




ShadowingRBTS

Properties of RBTS


Scores are well-defined between 0 and 2.

Participation is “expost individually rational”.

Numerically robust




ShadowingRBTS

Properties of RBTS



Numerically robust




ShadowingRBTS

Properties of RBTS



Numerically robust




ShadowingRBTS

Extensions

Adapting to a budget constraint. Scaling, randomizedexclusion.

More than two outcomes?

Other proper scoring rules?




ShadowingRBTS

Extensions

Adapting to a budget constraint.

Scaling, randomizedexclusion.






ShadowingRBTS

Extensions







ShadowingRBTS

Extensions







ShadowingRBTS

Extensions







Peer Prediction Without a Common Prior. Witkowski, Parkes.EC-2012.

Same setting as RBTS except each agent may have a differentinformation structure.

Elicit both the agent’s prior and posterior probability thatanother agent will get a high signal.

Infer whether agent’s signal was high or low

Score = score(prior) + score(posterior)

OR elicit prior and signal, and shadow to get a posterior(complicated)

















































Assumptions and Results

PP PP-CP BTS RBTS

common prior exists

designer knows CP

information structure

# of players

# of signals

designer learns?




Assumptions and Results

PP PP-CP BTS RBTS

common prior exists

designer knows CP

information structure

# of players

# of signals

designer learns?


Documents

Robust Bayesian Truth Serum - Harvard · PDF fileRobust Bayesian Truth Serum Presentation by Mark Bun and Bo Waggoner 2012-12-03 Presentation by Mark Bun and Bo Waggoner Robust Bayesian