Striving to Earn More: A Survey of Work Strategies and Tool Use Among Crowd Workers

Toni Kaplan*, Susumu Saito*, Kotaro Hara, and Jeffrey P. Bigham

(* Equal contribution)

Earning a decent wage is difficult for crowd workers on Amazon Mechanical Turk• Most workers are not paid well [Martin et al., 2014]

• Workers’ median hourly wage is ~$2/h, while average requester pays $11+/h [Hara et al., 2018]

Supporting workers for a better wage is a crucial challenge

What is needed to improve wage?(i.e., increasing !"#$% &$'(&) '&*$')

!"#$% #+,& -.&(# $- $*"'/&' )

1. Minimizing spent time 2. Maximizing “pay per work time”

[Hanrahan et al., 2015]

(Given )

…But not everyone can do this*Task listing page on Amazon Mechanical Turk*

✘ Requires time to find tasks [Chilton et al. 2010]

✘ Likely to waste working time [McInnis et al. 2016]

- Too small reward- Broken task (= unable to submit)- Answer rejection after submission

Work Strategies

• Website forums and tools/scripts [Mason and Suri, 2012]

→ Some workers are successful, but others are still not

! No formal research on tool types and theirprevalence per workers’ earnings

Our future project: Support by automation using Artificial Intelligence (AI)?→ Modeling successful workers’ behavior on their work

can possibly give all workers good capabilities

<

Research Questions

a) What are workers’ current strategies/tools/information for improving wage?

b) What are design considerations for future tool development for supporting crowd workers?

Survey

• Period … January 23, 2018 – January 31, 2018

• Subjects … 360 US-based workers on Amazon Mechanical Turk• 40/400 spam answers are filtered out• 100 each to workers who has completed {100 | 1,000 | 5,000 | 10,000} HITs

• Task … 67 survey questions (~10-30 minutes) for $3.50• General demographic information and income as workers• Workers’ working tips• Workers’ tools/forums usage• Sentiment towards AI …

Analysis

• Worker categorization by their reported total earnings in 2017

High-earning Extremes … Top 10%

High-earners … Top 50%

Low-earners … Bottom 50%

Outline

1. Tools and strategies for workers

2. Sentiment towards automation with AI

3. Discussion: How can we assist workers better?

Outline

1. Tools and strategies for workers

2. Sentiment towards automation with AI

3. Discussion: How can we assist workers better?

0

20

40

60

80

Always Most of the time About half the time Occasionally Never

High-earning extremes High-earners Low-earners

Website forums

[%]

Q: How often do you browse or post in Mechanical Turk related websites?

Website forums

Q: Which of the following Mechanical Turk related websites do you regularly read / browse / reference?

• MTurkgrind• MTurk Forum• Turker Nation• HIT Notifier

• MTurk Crowd• Turker Hub• [Reddit] AMT• [Reddit] HITsWorthTurkingFor• [Reddit] hNOTwtf

Website forums

0

20

40

60

80

100

MTurk Crowd Turker Hub [Reddit]mturk

[Reddit]HWTF

…

[%]

• Relatively new and active• Various thread categories

(e.g., “Daily Work”, “MTurk Help & Resources” …)

• Active• Mainly Q&A / chatting• Single thread category

0

50

100

Extremes High-earners Low-earners

Q: Are you using any browser extensions or scripts to improveyour work experience/pay on Mechanical Turk?

Tools

[%]

Tools

Q: Which, if any, of the following extensions and scripts do you regularly use while working on Mechanical Turk?

• HIT Scraper

• Turkmaster

• MTurk Engine

• Turkopticon

• MTurk Suite

• Panda Crazy

HIT Scraper

Tools

0

20

40

60

80

100

Turkopticon MTurk Suite Panda Crazy

Turkopticon [Irani et al., 2013]

MTurk Suite

Panda Crazy:• Userscript with interface

for automated “PandA”ing of

prespecified HIT batches

HIT Scraper:• Auto-refreshing task search

with extra filters

(Turkopticon ratings,

customizable blocklist…)

Outline

1. Tools and strategies for workers

2. Sentiment towards automation with AI

3. Discussion: How can we assist workers better?

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

StronglyAgree

SomewhatAgree

Neither Agreenor Disagree

SomewhatDisagree

StronglyDisagree

“I would use a tool that automates some of the work in a HIT, which then lets me verify if the work was done correctly.”

Why NOT agree?

Why would you NOT use it?

• 78 workers expressed concerns• The role of a human in Human Intelligence Tasks (18)

• “The whole purpose of a HIT is to complete a Human Intelligence Task, which by definition is a task that cannot or should not be automated.”

• Mistrust in the quality of AI output (17)• “I don’t trust it, it would add more time to go back and check to see if it was right”

• Violating the AMT terms of service (12)• Violation of their personal ethics (9)• Unfairness to requesters (12)

Outline

1. Tools and strategies for workers

2. Sentiment towards automation with AI

3. Discussion: How can we assist workers better?

Is there room for improvement?STATUS QUO: “Sharing” information among workers

Microtasks are doneall by workers

Tools that assistsearching tasks(e.g., Turkopticon)

(Choose tasks)

Tools that assistaccepting tasks(e.g., Panda Crazy)

(Popular tasks)

Is there room for improvement?

2. Tools that supportdoing microtasks

POSSIBLE FUTURE: Collecting data for automation with AI

Workers do microtasks with the support

1. Tools that automatesearch & accepting microtasks

(Popular tasks)

(Similar tasks)

Proposal 1: Automatic microtask search

Find and reserve recommendable tasks as soon as they are posted

SupervisedML algorithms

Output:- Hourly wage- How possible to be

approved?- How long will they

be available?

?Newly posted

microtask batch

Input:

Training dataFeature: microtask HTMLLabel: completion time,

approval/rejection…

Worker preferences- Microtask type- Work time- Reward target …

Auto-reservemicrotasks

Proposal 2: Iterative answer suggestion

Augmenting work by automating user action• Opening links as a task starts (e.g., survey tasks)• Copy & pasting specified keywords in microtasks (e.g., external search tasks)

Suggesting answers for repeated or commonly asked questions• “Input your AMT worker ID” etc.

|Answer suggestions:

AXBCDEFG01234

(Repeatedlyinputting

worker ID)

(Repeatedlyinputting

worker ID)

SupervisedML algorithms?

(Training data) (Output)

Conclusion

a) What are workers’ current strategies/tools/information for improving wage efficiency?→ Use active forums and tools that evaluate requesters/tasks and

reserve task batches

b) What are design considerations for future tool development for supporting crowd workers?→ Use collected data for automation with machine learning

Striving to Earn More: A Survey of Work Strategies and Tool Use Among Crowd WorkersToni Kaplan*, Susumu Saito*, Kotaro Hara, and Jeffrey P. Bigham(* Equal contribution)

Questions?

0

20

40

60

80

Always Most of the time About half the time Occasionally Never

Extremes High-earners Low-earners

Website forums

[%]

Q: Do you browse or post in any Mechanical Turk related websites?

Website forums

0

20

40

60

80

100

MTurkCrowd

Turker Hub [Reddit]mturk

[Reddit]HWTF

[Reddit]hNOTwtf

Mturkgrind MTurkForum

TurkerNation

HIT Notifier Other

[%]

Website forums

• MTurk Crowd and Turker Hub• Relatively new and rapidly growing• Has many discussion threads

• Subreddits• Popular among most workers, but less attractive to high-earning extremes• Dunno why?

• Other forums• Used to be active but not anymore.

• HIT Notifier• Still new and seems pretty easy to use, but dunno why it’s not popular?

Tools

0

20

40

60

80

100

Turkopticon MTurkSuite

Tampermonkey/Greasemonkey

Panda Crazy

(Userscript-related)(No userscript-related)

Tools

• Turkopticon and MTurk Suite• Just a chrome extension and intuitive, which makes it very easy to use for everyone• Used mainly for seeing requesters’ reputation

• Greasemonkey, Tampermonkey• A base extension tool for userscript

• Panda Crazy• Not very intuitive but has high functionality w/ helper scripts, once getting used

• HIT Scraper• Intuitive, but not always good for high-earners who prefers native AMT page

• Turkmaster• Often recognized as a substitute of Panda Crazy. Seems less useful

• MTurk Engine• A new script being developed recently, that combines Panda Crazy and HIT Scraper• Maybe not widely known yet?

High-earners had more access to

• Masters Qualification• Access to more profitable HITs• More opportunity, higher reward

• “PANDA” technique (“Preview and Accept”)• Technique for task batch reservation• More opportunity, less task search time

0

50

100

Extremes High-earners Low-earners

0

50

100[%]

[%]