Upload
ericblattberg
View
216
Download
0
Embed Size (px)
Citation preview
7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk
http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 1/14
©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$
!"#$%&'#$("(%&)*+,-(++
amazonweb services
7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk
http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 2/14
©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$
!"#$%&'#$("(%&)*+,-(++22
Crowdsourcing myths
7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk
http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 3/14
©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$
!"#$%&'#$("(%&)*+,-(++
_ It’s cheaper _
It’s actually more efficient
_ It’s faster _ It’s actually more scalable
_ It’s not accurate _ It’s actually more accurate
3
7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk
http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 4/14
©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$
!"#$%&'#$("(%&)*+,-(++44
Crowdsourcing BestPractices
7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk
http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 5/14
©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$
!"#$%&'#$("(%&)*+,-(++
_ Consider the question
carefully _ Workers answer what you ask
_ Select your workers _
Perspective and skills vary
_ Monitor and improve _ Adjust for optimal results
5
7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk
http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 6/14
©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$
!"#$%&'#$("(%&)*+,-(++6
The question[ ]6
You will get an answer to thequestion that you ask. Focus onasking the right question
7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk
http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 7/14
©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$
!"#$%&'#$("(%&)*+,-(++7
Choosing your workers[ ]7
Workers are different– from
language and culturaldifferences – to varying skills.Test and monitor.
7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk
http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 8/14
©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$
!"#$%&'#$("(%&)*+,-(++8
Monitor and Improve[ ]8
Monitor key metrics, adjust and
measure key attributes impacton those metrics.
7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk
http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 9/14
©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$
!"#$%&'#$("(%&)*+,-(++
_ Accuracy
_ Know your current
_ Throughput
_ Understand both turnaround and scale requirements
_ Cost _ Measure against a budget – as cost can impact the
other two
9
Key Metrics
[ ]9
7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk
http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 10/14
©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$
!"#$%&'#$("(%&)*+,-(++10
Accuracy[ ]10
error has two sources: human
error and systematic error.Isolating human error and solving for systematic error gives
a better chance for success.
7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk
http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 11/14
©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$
!"#$%&'#$("(%&)*+,-(++11
Accuracy[ ]11
After solving for systematic error
choosing the best workers and monitoring those workersprovides the next step towards
high accuracy.
7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk
http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 12/14
©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$
!"#$%&'#$("(%&)*+,-(++12
Throughput[ ]12
Many factors impact throughput;
Reputationergonomics
clarity of task
7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk
http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 13/14
©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$
!"#$%&'#$("(%&)*+,-(++13
Cost[ ]13
Many factors impact cost;
Accuracy and throughput tradeoff
ergonomics
sociology
7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk
http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 14/14
©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$
!"#$%&'#$("(%&)*+,-(++14
http://www.mturk.com
14
JohnHoskins,AmazonMechanicalTurk
amazonweb services