45
Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) [email protected] Ron K. Cytron Washington University in St. Louis [email protected] COMSOC ’08 3 September 2008

Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) [email protected]

  • View
    215

  • Download
    2

Embed Size (px)

Citation preview

Page 1: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

Approval-rating systems that never

reward insincerity

Rob LeGrandWashington University in St. Louis

(now at Bridgewater College)

[email protected]

Ron K. CytronWashington University in St. Louis

[email protected]

COMSOC ’083 September 2008

Page 2: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

2

Approval ratings

Page 3: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

3

Approval ratings

• Aggregating film reviewers’ ratings– Rotten Tomatoes: approve (100%) or disapprove (0%) – Metacritic.com: ratings between 0 and 100– Both report average for each film– Reviewers rate independently

Page 4: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

4

Approval ratings

• Online communities– Amazon: users rate products and product reviews– eBay: buyers and sellers rate each other– Hotornot.com: users rate other users’ photos– Users can see other ratings when rating

• Can these “voters” benefit from rating insincerely?

Page 5: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

5

Approval ratings

Page 6: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

6

Average of ratings

9.0,8.0,8.0,7.0,4.0v 9.0,8.0,8.0,7.0,4.0r

0 172.0

outcome: 72.0)( vfavg

data from Metacritic.com: Videodrome (1983)

Page 7: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

7

Average of ratings

9.0,8.0,8.0,7.0,0v 9.0,8.0,8.0,7.0,4.0r

0 164.0

outcome: 64.0)( vfavg

Videodrome (1983)

Page 8: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

8

Another approach: Median

9.0,8.0,8.0,7.0,4.0v 9.0,8.0,8.0,7.0,4.0r

0 18.0

outcome: 8.0)( vfmed

Videodrome (1983)

Page 9: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

9

Another approach: Median

9.0,8.0,8.0,7.0,0v 9.0,8.0,8.0,7.0,4.0r

0 18.0

outcome: 8.0)( vfmed

Videodrome (1983)

Page 10: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

10

Another approach: Median

• Immune to insincerity– voter i cannot obtain a better result by voting– if , increasing will not change– if , decreasing will not change

• Allows tyranny by a majority– – – no concession to the 0-voters

ii rv imed vvf )(

imed vvf )( iv

iv

1,1,1,1,0,0,0v1)( vfmed

)(vfmed

)(vfmed

Page 11: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

11

Declared-Strategy Voting[Cranor & Cytron ’96]

electionstate

cardinal

preferences

rational

strategizer

ballot

outcome

Page 12: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

12

Declared-Strategy Voting[Cranor & Cytron ’96]

electionstate

cardinal

preferences

rational

strategizer

ballot

outcome

• Separates how voters feel from how they vote• Levels playing field for voters of all sophistications• Aim: a voter needs only to give sincere preferences

sincerity strategy

Page 13: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

13

Average with Declared-Strategy Voting?

• Try using Average protocol in DSV context

• But what’s the rational Average strategy?• And will an equilibrium always be found?

electionstate

cardinal

preferences

rational

strategizer

ballot

outcome

Page 14: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

14

Rational [m,M]-Average strategy

• Allow votes between and• For , voter i should choose to move

outcome as close to as possible• Choosing would give• Optimal vote is

• After voter i uses this strategy, one of these is true:– and– – and

0m

)),,min(max( Mmvnrvij jii

iavg rvf )(

ij jii vnrv

iavg rvf )(

Mvi

mvi iavg rvf )(

iavg rvf )(

1Mni 1 iv

ir

Page 15: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

15

Equilibrium-finding algorithm

0 1

9.0,8.0,8.0,7.0,4.0r

72.0

9.0,8.0,8.0,7.0,4.0v

Videodrome (1983)

Page 16: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

16

Equilibrium-finding algorithm

01

9.0,8.0,8.0,7.0,4.0r

0,0,0,0,0v

Page 17: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

17

Equilibrium-finding algorithm

0 1

9.0,8.0,8.0,7.0,4.0r

2.0

1,0,0,0,0v

Page 18: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

18

Equilibrium-finding algorithm

0 1

9.0,8.0,8.0,7.0,4.0r

4.0

1,1,0,0,0v

Page 19: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

19

Equilibrium-finding algorithm

0 1

9.0,8.0,8.0,7.0,4.0r

6.0

1,1,1,0,0v

Page 20: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

20

• Is this algorithm guaranteed to find an equilibrium?

Equilibrium-finding algorithm

0 1

9.0,8.0,8.0,7.0,4.0r

7.0

1,1,1,5.0,0vequilibrium!

Page 21: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

21

• Is this algorithm guaranteed to find an equilibrium?• Yes!

Equilibrium-finding algorithm

0 1

9.0,8.0,8.0,7.0,4.0r

7.0

1,1,1,5.0,0vequilibrium!

Page 22: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

22

• These results generalize to any range

Expanding range of allowed votes

1 2

9.0,8.0,8.0,7.0,4.0r

8.0

2,2,2,1,1 v

Page 23: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

23

• Will multiple equilibria always have the same average?

Multiple equilibria can exist

outcome in each case:

7.0)( vfavg

1,1,1,5.0,0v 9.0,8.0,7.0,7.0,4.0r

1,1,9.0,6.0,0v

1,1,75.0,75.0,0v

Page 24: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

24

• Will multiple equilibria always have the same average?• Yes!

Multiple equilibria can exist

outcome in each case:

7.0)( vfavg

1,1,1,5.0,0v 9.0,8.0,7.0,7.0,4.0r

1,1,9.0,6.0,0v

1,1,75.0,75.0,0v

Page 25: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

25

Average-Approval-Rating DSV

9.0,8.0,8.0,7.0,4.0v 9.0,8.0,8.0,7.0,4.0r

0 17.0

outcome: 7.0)1,0,( vfaveq

Videodrome (1983)

Page 26: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

26

• AAR DSV is immune to insincerity in general

Average-Approval-Rating DSV

9.0,8.0,8.0,7.0,0v 9.0,8.0,8.0,7.0,4.0r

0 1

outcome: 7.0)1,0,( vfaveq

7.0

Page 27: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

27

• Expanded vote range gives wide range of AAR DSV systems:

• If we could assume sincerity, we’d use Average• Find AAR DSV system that comes closest• Real film-rating data from Metacritic.com

– mined Thursday 3 April 2008– 4581 films with 3 to 44 reviewers per film– measure root mean squared error

• Perhaps we can come much closer to Average than Median or [0,1]-AAR DSV does

Evaluating AAR DSV systems

10 a 10 b)(, vba

Page 28: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

28

Evaluating AAR DSV systems

5.0,aRMSE

a

3240.0aminimum at

5.0b

Page 29: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

29

Evaluating AAR DSV systems: hill-climbing

a

3647.0aminimum at

4820.0b

4820.0,aRMSE

Page 30: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

30

Evaluating AAR DSV systems: hill-climbing

4820.0bminimum at

3647.0a

bRMSE ,3647.0

b

Page 31: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

31

Evaluating AAR DSV systems

)(4820.0,3647.0 v

)(vfavg

Page 32: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

32

AAR DSV: Future work

• New website: trueratings.com– Users can rate movies, books, each other, etc.– They can see current ratings without being tempted to

rate insincerely– They can see their current strategic proxy vote

• Richer outcome spaces– Hypercube: like rating several films at once– Simplex: dividing a limited resource among several uses– How assumptions about preferences are generalized is

important

Thanks! Questions?

Page 33: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

33

What happens at equilibrium?

• The optimal strategy recommends that no voter change

• So• And

– equivalently,

• Therefore any average at equilibrium must satisfy two equations:– (A)– (B)

1)( ii vrvi

ii rvvi 0)(0)( ii vrvi

irvinv : nvrvi i :

Page 34: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

34

Proof: Only one equilibrium average

irinA :)( nriB i :)(

212211 )()()()( BABA

• Theorem:

• Proof considers two symmetric cases:– assume– assume

• Each leads to a contradiction

21 12

Page 35: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

35

Proof: Only one equilibrium average

21 case 1:

ii rri 12)( ii riri 12 :: ii riri 12 ::

irin 22 : nri i 11:

nririn ii 1122 :: nn 12

12 21 , contradicting

)( 2A)( 1B

Page 36: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

36

Proof: Only one equilibrium average

21 Case 1 shows that

Case 2 is symmetrical and shows that 12

21 Therefore

Therefore, given , the average at equilibrium is uniquer

Page 37: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

37

An equilibrium always exists?

• At equilibrium, must satisfy

Given a vector , at least one equilibrium indeed always exists.

A particular algorithm will always find an equilibrium for any . . .

)),,min(max()( Mmvnrviij jii

v

r

r

Page 38: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

38

An equilibrium always exists!

Equilibrium-finding algorithm:• sort so that• for i = 1 up to n do

• Since an equilibrium always exists, average at equilibrium is a function, .

• Applying to instead of gives a new system, Average-Approval-Rating DSV.

r

)),,)(min(max( Mmminvnrvik kii

ji rrji )(

),,( Mmrfaveq

v

r

aveqf

(full proof and more efficient algorithm in dissertation)

Page 39: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

39

• What if, under AAR DSV, voter i could gain an outcome closer to ideal by voting insincerely ( )?

• It turns out that Average-Approval-Rating DSV is immune to strategy by insincere voters.

• Intuitively, if , increasing will not change .

ii rv

Average-Approval-Rating DSV

iaveq vMmvf ),,(

iv),,( Mmvfaveq

Page 40: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

40

• If ,– increasing will not change .– decreasing will not increase .

• If ,– increasing will not decrease .– decreasing will not change .

• So voting sincerely ( ) is guaranteed to optimize the outcome from voter i’s point of view

AAR DSV is immune to strategy

iiaveq rvMmvf ),,(

),,( Mmvfaveq

),,( Mmvfaveq

iviv

iiaveq rvMmvf ),,(

iv

iv

),,( Mmvfaveq

),,( Mmvfaveq

ii rv

(complete proof in dissertation)

Page 41: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

41

• [m,M]-AAR DSV can be parameterized nicely using a and b, where and :

mMa

1

mM

mb

1

a

bbM

1

a

bbm

Parameterizing AAR DSV

x

bb

x

bbvfv aveq

axba

1,,lim)(,

10 a 10 b

Page 42: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

42

• For example:

Parameterizing AAR DSV

)1,0,()(,1 vfv aveqb

vfv med

)(

2

1,0

11,10,)(2

1,

21

1 vfv aveq

vv

min)(1,0

vv

max)(0,0

2,1,)(2

1,3

1 vfv aveq

Page 43: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

43

• Real film-rating data from Metacritic.com– mined Thursday 3 April 2008– 4581 films with 3 to 44 reviewers per film

Evaluating AAR DSV systems

10 a 10 b

2,, vfvvSE avgbaba

V

VV

v

vba

ba v

vSEvRMSE

,

,

Page 44: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

44

Higher-dimensional outcome space

• What if votes and outcomes exist in dimensions?

• Example:• If dimensions are independent, Average, Median

and Average-approval-rating DSV can operate independently on each dimension– Results from one dimension transfer

1d

1010:, 2 yxyx

Page 45: Approval-rating systems that never reward insincerity Rob LeGrand Washington University in St. Louis (now at Bridgewater College) legrand@cse.wustl.edu

45

Higher-dimensional outcome space

• But what if the dimensions are not independent?– say, outcome space is a disk in the plane:

• A generalization of Median: the Fermat-Weber point [Weber ’29]

– minimizes sum of Euclidean distances between outcome point and voted points

– F-W point is computationally infeasible to calculate exactly [Bajaj ’88] (but approximation is easy [Vardi ’01])

– cannot be manipulated by moving a voted point directly away from the F-W point [Small ’90]

1:, 222 yxyx