Do Mechanical Turks Dream of Square Pie Charts?

Robert Kosara and Caroline Ziemkiewicz

University vs. Online Studies

University Online

Controlled environment

Easy to recruit Standard incentives

Homogenous users Slow, work-intensive Small samples

Broader demographics Fast, low effort Larger numbers

Environmental control Vote flooding Recruitment and

advertising

2 Caroline Ziemkiewicz

Mechanical Turk Amazon’s micro-payment job market Addresses many issues with online studies!

Prevents vote flooding Manages payment automatically Handles recruitment

3 Caroline Ziemkiewicz

Our experiences Turking studies We’ve done a lot of these by now!

2 years 7 studies Over 550 participants

What have we learned?

4 Caroline Ziemkiewicz

Participants Who are the Turkers?

57.4% female, 42.6% male (in our studies) Contrast with student populations

Mean age of 32, and a broad distribution…

5 Caroline Ziemkiewicz

Participants Generally faster at

tasks than student populations

Some performance differences

Personality

6 Caroline Ziemkiewicz

Designing studies Preventing cheaters

Bonus system allows for accuracy/quality incentives

Building checks into data collection Pre-testing/calibration

Other issues Informed consent Good instructions Feedback Audit trails

7 Caroline Ziemkiewicz

Running studies Participation varies over time of day…

Highest rate during the afternoon + evening EST

8 Caroline Ziemkiewicz

Running Studies Timing affects speed of completion Rate of participation starts high and

decreases over time Placement on front page is crucial!

Longer tasks get fewer participants Turkers like doing studies!

9 Caroline Ziemkiewicz

To Turk or not to Turk Studies that lend themselves well to Turking

Simple interactions and data collection Short performance time Responses that are hard to fake

Studies that don’t Accuracy of self-reported data is crucial Complex tasks Very long task duration

10 Caroline Ziemkiewicz

Thanks!

Let’s discuss!