Embed Size (px)
DESCRIPTION
Insights into Mechanical Turk (or, “Mistakes Requesters Make”). Adam D. Bradley [email protected]. What is Mechanical Turk?. Amazon Mechanical Turk is a marketplace for work. . Feedback. Mechanical Turk provides a set of primitives HIT properties (reward, instructions, requirements) - PowerPoint PPT Presentation
Citation preview
© 2009 Amazon.com, Inc. or its Affiliates.
Insights into Mechanical Turk(or, “Mistakes Requesters Make”)
Adam D. [email protected]
© 2009 Amazon.com, Inc. or its Affiliates.
What is Mechanical Turk?
Amazon Mechanical Turk is a marketplace for work.
© 2009 Amazon.com, Inc. or its Affiliates.
Feedback
Mechanical Turk provides a set of primitives • HIT properties (reward, instructions,
requirements)• Assignment can be approved or rejected• Worker can be bonused• Worker can be blocked• Qualification can be assigned or revoked
You use these primitives to execute your “policies”
Your policies shape your Reputation
© 2009 Amazon.com, Inc. or its Affiliates.
Law of Unintended Consequences is True
© 2009 Amazon.com, Inc. or its Affiliates.
Plurality to the Rescue (?)
© 2009 Amazon.com, Inc. or its Affiliates.
When Plurality Diverges
Possible Root Causes
– Worker skills/training/knowledge– Ergonomics– Worker effort– Ambiguity in the Instructions– Ambiguity in the Data
© 2009 Amazon.com, Inc. or its Affiliates.
When Plurality Diverges
Worker Quality and Worker History
– 1000 questions, 5 answers each = 5000 independent answers
– 5 Workers each answer 1000 questions
See Ipeirotis et al in HCOMP 2010
© 2009 Amazon.com, Inc. or its Affiliates.
When Plurality Diverges
Which Workers do you Trust?
– “Gold Standard”, “Ground Truth”, “Trap”•“Meaningful” Sample?
– Approval History…?
– Is “Trust” closed or open?
© 2009 Amazon.com, Inc. or its Affiliates.
Turnaround time for Requester X’s HITs
0
20
40
60
80
100
120
140
160
avg(min)
© 2009 Amazon.com, Inc. or its Affiliates.
Turnaround time for Requester X’s HITs
0
20
40
60
80
100
120
140
160
avg(min)
Anyone try out these HITs.
I stopped doing these HITS since the rejection rate has increased tremendously compared to the initial stage
I have done about 700 with 5 rejects …. Honestly I feel like my rejects have come from being tired and selecting the wrong answer
thanks, for telling me about these.
I found my new favorite HIT
© 2009 Amazon.com, Inc. or its Affiliates.
Comparison of Turnaround Time
6/1/20
10
6/2/20
10
6/3/20
10
6/4/20
10
6/5/20
10
6/6/20
10
6/7/20
10
6/8/20
10
6/9/20
10
6/10/2
010
6/11/2
010
6/12/2
010
6/13/2
010
6/14/2
010
6/15/2
010
6/16/2
010
6/17/2
010
6/18/2
010
6/19/2
010
6/20/2
010
6/21/2
010
6/22/2
010
6/23/2
010
6/24/2
010
6/25/2
010
6/26/2
010
6/27/2
010
6/28/2
010
6/29/2
010
6/30/2
010
1
10
100
1000
10000
Requester XRequester Y
Aver
age
Turn
Tim
e - M
inut
es
© 2009 Amazon.com, Inc. or its Affiliates.
Optimizing For Best Results
Accuracy
Price
Speed
