Pricing and Incentives I

KSE 801Uichin Lee

Financial Incentives and the “Performance of Crowds”

Winter Mason and Duncan J. WattsKDD-HCOMP ‘09, June 28, 2009

Paris, France

Motivation of Crowdsourcing: Monetary incentive vs. Performance

• Motivation of crowdsourcing and peer production– Intrinsic motivation: enjoyment, desire to help out– Social rewards (Flickr) – Monetary incentives (M-Turk)

• “Rational choice” in economic theory– Rational workers will choose to improve their performance in response

to a scheme that rewards such improvements with financial gain• Example: an autoglass factory where workers install windshields

on a production line and switched from time-rate wage (pay per hour) to piece-rate wage (pay per unit)– Over the course of a year and half, individual productivity has

increased 20%, concluding that performance based pay scheme is a powerful tool for eliciting improved performance

Motivation of Crowdsourcing: Monetary incentive vs. Performance

• Yet, under certain circumstance the provision of financial incentives can undermine “intrinsic motivation” (e.g., enjoyment, altruism), possibly leading to poorer outcome

• Workers may ignore rational incentives to work longer when they have accomplished pre-set targets (e.g., taxi driver examples)

• Recent experiments demonstrate that financial incentives undermine actual performance (called chocking effect); e.g., hampering innovations

Experiments

• Study 1: Image ordering– Image ordering: sorting a set of images taken from a traffic camera at

2-second intervals in chronological order– To understand the impact of monetary compensation on

performance (quantity and quality)– Differentiate between quantity of work (output) and quality of work

(accuracy)– Use image ordering tasks where output can vary widely and accuracy

can be measured objectively

• Study 2: Word puzzles– To further investigate the absence of an effect on accuracy (will see

what it means later)

Study 1: Image ordering

At the beginning of a task, you will be presented with a list of images taken from traffic cameras. An example list is shown below

Your goal is to reorder the list chronologically from left to right and top to bottom. The sorted list is shown below.

Drag and drop for ordering

Study 1: Image ordering• All participants (total 611 workers) were paid an initial fee of $0.10 to

complete the introductory survey and training set

• Then, participants were given information about (1) how much they would be paid (pay level) and (2) what difficulty level they will be given– Pay level:

• No pay• Low pay: $0.01 per image set• Medium pay: $0.05 per image set• High pay: $0.10 per image set

– Difficulty level:• Easy (two images per set)• Medium (three images per set)• Hard (four images per set)

Study 1: Image ordering – Results

• Across all difficulty levels participants chose to complete more tasks on average when the pay was higher– F(3, 607) = 15.73, p<0.001

• Across all payment levels, the number of completed tasks decreased with increasing difficulty

Num

ber o

f tas

ks c

ompl

eted

Pay per task

Hard: 4 images

Medium: 3 images

Easy: 2images

Study 1: Image ordering – Results • Accuracy measure:

– A: Proportion of image sets correctly sorted

– B: Spearman’s rank correction between the correct order and the sorted order• Perfect match: ρ = 1• Perfect mismatch: ρ = 0

• One-way ANOVA test confirms that increasing wage did not improve accuracy– A: F(3, 607) = 0.66, p<0.001– B: F(3, 607) = 0.82, p<0.001

Accu

racy

(pro

porti

on o

f im

ages

se

ts c

orre

ctly

labe

led)

Pay per task

Hard: 4 images

Medium: 3 images

Easy: 2 images

Discussion

• Another test: single wage of $0.01, yet alerting participants that accuracy is measured, with the following 4 variants1) Only informed that accuracy would be measured2) Shown feedback on their accuracy after every fourth image set3) Told explicitly that their pay would be contingent on their

performance4) (2) and (3) are used in combination

• With this test, they found that quantity and quality results were indistinguishable, suggesting that workers were treating their pay as performance dependent

Discussion: Anchoring effect?

• Workers in all conditions generally felt that the expected wage for the work they performed is greater than what they received – They felt they are paid less than

expected– X2=243.61, p<0.001

• Thus, workers were no more motivated to perform better no matter how much they were actually paid

Perc

eive

d Ta

sk V

alue

$0.10$0.01$0.05

$0.20

$0.10

Actual Task Value$0.05

Study 2: Word Puzzle• Want to further investigate absence of an effect on accuracy

* Shown a list of 15 possible words (not all of the words listed are in the puzzle)* Select a word: click the first and last letter (if correct, it will turn red)

* Two wage models: quota vs. piece rate* Quota: every puzzle successfully completed* Piece: every word they found* Pay levels: low, medium, high, (no pay) -- Puzzle: $0.01, $0.05, $0.10 -- Word: $0.01, $0.02, $0.03

Study 2: Word Puzzle – Results

• Payment results in higher # of completions

Incentive Framing

Num

ber o

f Puz

zles

Com

plet

ed

Study 2: Word Puzzle – Results

• # of puzzles completed did not differ by payment strategies --- no significant impact on quantity!

Pay per Puzzle

Num

ber o

f Puz

zles

Com

plet

ed

Pay per Word

Num

ber o

f Puz

zles

Com

plet

ed

F(2, 108) = 0.71, not significant F(2, 124) = 1.82, not significant

66

Possible Reason: Intrinsic motivation of enjoyment

Study 2: Word Puzzle – Results Ac

cura

cy

(fra

ction

of w

orld

s fo

und

per p

uzzl

e)

Cost

per

Wor

d

No Contingent Pay Pay per Puzzle Pay per Word

Accuracy

Cost per word

High accuracy per puzzle

means low cost per word

Low accuracy per puzzle, but workers find as many

words as they can Intrinsic

motivation (enjoyment)

Discussion• Quota requires more efforts than piece rate• Quota elicits more work: (1) greater marginal difficulties of

finding next words to complete a puzzle, (2) implicit goal setting – harder puzzles become more salient goals

Perc

eive

d Ta

sk V

alue

Perc

eive

d Ta

sk V

alue

Actual Task Value (Per Puzzle)Actual Task Value (Per Word)

Anchoring effects

Discussion• When there is no expectation of financial reward, effort is

motivated by other kinds of rewards (e.g., social); but when monetary compensation is expected, as in the AMT framework, the anticipated financial value of the effort will be the driving mechanism.

• Although paid workers generally did more work than unpaid workers, how they were paid had a larger impact on their output and accuracy than how much they were paid.

• Moreover, paying workers a low rate led to them to perceive their work as less valuable than not paying them at all (previous slide)

Summary

• Investigated the relationship between financial incentives and performance in the novel setting of online peer production systems

• Main findings: – Increased payments increased the quantity of work

performed, but not its quality– The particular design of the compensation scheme (a quota

scheme vs. a piece rate, for example) can have a significant effect on quality even to the point where better work can be accomplished for less pay (or even without any payment)