14
7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 1/14 ©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved !"#$ !"#$%&'#$("(%&)*+,-(++ amazon web services

Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

Embed Size (px)

Citation preview

Page 1: Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 1/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++

amazonweb services

Page 2: Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 2/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++22

Crowdsourcing myths

Page 3: Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 3/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++

 _  It’s cheaper _ 

 It’s actually more efficient

 _  It’s faster _   It’s actually more scalable

 _  It’s not accurate _   It’s actually more accurate

3

Page 4: Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 4/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++44

Crowdsourcing BestPractices

Page 5: Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 5/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++

 _  Consider the question

carefully _   Workers answer what you ask 

 _  Select your workers _ 

 Perspective and skills vary

 _  Monitor and improve _   Adjust for optimal results

5

Page 6: Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 6/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++6

The question[  ]6

You will get an answer to thequestion that you ask. Focus onasking the right question

Page 7: Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 7/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++7

Choosing your workers[  ]7

Workers are different– from

language and culturaldifferences – to varying skills.Test and monitor.

Page 8: Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 8/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++8

Monitor and Improve[  ]8

Monitor key metrics, adjust and 

measure key attributes impacton those metrics.

Page 9: Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 9/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++

 _   Accuracy

 _   Know your current

 _   Throughput

 _   Understand both turnaround and scale requirements

 _   Cost _   Measure against a budget – as cost can impact the

other two

9

Key Metrics

[ ]9

Page 10: Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 10/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++10

Accuracy[  ]10

error has two sources: human

error and systematic error.Isolating human error and solving for systematic error gives

a better chance for success.

Page 11: Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 11/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++11

Accuracy[  ]11

After solving for systematic error

choosing the best workers and monitoring those workersprovides the next step towards

high accuracy.

Page 12: Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 12/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++12

Throughput[  ]12

Many factors impact throughput;

Reputationergonomics

clarity of task 

Page 13: Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 13/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++13

Cost[  ]13

Many factors impact cost;

Accuracy and throughput tradeoff

ergonomics

sociology

Page 14: Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

7/29/2019 Massolution NYC 2013: Crowdsourcing Best Practices | Amazon Mechanical Turk

http://slidepdf.com/reader/full/massolution-nyc-2013-crowdsourcing-best-practices-amazon-mechanical-turk 14/14

©2013, Amazon Web Services, Inc. or its affiliates. All rights reserved!"#$

!"#$%&'#$("(%&)*+,-(++14

http://www.mturk.com

14

JohnHoskins,AmazonMechanicalTurk

[email protected]

amazonweb services