Insights into Mechanical Turk (or, “Mistakes Requesters Make”)

© 2009 Amazon.com, Inc. or its Affiliates.

Insights into Mechanical Turk(or, “Mistakes Requesters Make”)

Adam D. Bradleyabradley@amazon.com

© 2009 Amazon.com, Inc. or its Affiliates.

What is Mechanical Turk?

Amazon Mechanical Turk is a marketplace for work.

© 2009 Amazon.com, Inc. or its Affiliates.

Feedback

Mechanical Turk provides a set of primitives • HIT properties (reward, instructions,

requirements)• Assignment can be approved or rejected• Worker can be bonused• Worker can be blocked• Qualification can be assigned or revoked

You use these primitives to execute your “policies”

Your policies shape your Reputation

© 2009 Amazon.com, Inc. or its Affiliates.

Law of Unintended Consequences is True

© 2009 Amazon.com, Inc. or its Affiliates.

Plurality to the Rescue (?)

© 2009 Amazon.com, Inc. or its Affiliates.

When Plurality Diverges

Possible Root Causes

– Worker skills/training/knowledge– Ergonomics– Worker effort– Ambiguity in the Instructions– Ambiguity in the Data

© 2009 Amazon.com, Inc. or its Affiliates.

When Plurality Diverges

Worker Quality and Worker History

– 1000 questions, 5 answers each = 5000 independent answers

– 5 Workers each answer 1000 questions

See Ipeirotis et al in HCOMP 2010

© 2009 Amazon.com, Inc. or its Affiliates.

When Plurality Diverges

Which Workers do you Trust?

– “Gold Standard”, “Ground Truth”, “Trap”•“Meaningful” Sample?

– Approval History…?

– Is “Trust” closed or open?

© 2009 Amazon.com, Inc. or its Affiliates.

Turnaround time for Requester X’s HITs

0

20

40

60

80

100

120

140

160

avg(min)

© 2009 Amazon.com, Inc. or its Affiliates.

Turnaround time for Requester X’s HITs

0

20

40

60

80

100

120

140

160

avg(min)

Anyone try out these HITs.

I stopped doing these HITS since the rejection rate has increased tremendously compared to the initial stage

I have done about 700 with 5 rejects …. Honestly I feel like my rejects have come from being tired and selecting the wrong answer

thanks, for telling me about these.

I found my new favorite HIT

© 2009 Amazon.com, Inc. or its Affiliates.

Comparison of Turnaround Time

6/1/20

10

6/2/20

10

6/3/20

10

6/4/20

10

6/5/20

10

6/6/20

10

6/7/20

10

6/8/20

10

6/9/20

10

6/10/2

010

6/11/2

010

6/12/2

010

6/13/2

010

6/14/2

010

6/15/2

010

6/16/2

010

6/17/2

010

6/18/2

010

6/19/2

010

6/20/2

010

6/21/2

010

6/22/2

010

6/23/2

010

6/24/2

010

6/25/2

010

6/26/2

010

6/27/2

010

6/28/2

010

6/29/2

010

6/30/2

010

1

10

100

1000

10000

Requester XRequester Y

Aver

age

Turn

Tim

e - M

inut

es

© 2009 Amazon.com, Inc. or its Affiliates.

Optimizing For Best Results

Accuracy

Price

Speed

© 2009 Amazon.com, Inc. or its Affiliates.

By the way – We’re Hiring!

turk-jobs@amazon.com

© 2009 Amazon.com, Inc. or its Affiliates.

Questions?

abradley@amazon.com