Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
DELIVERING WEB EXPERIMENTS USINGAMAZON MECHANICAL TURK
CHUN CHAN
18 OCT 2017
INTRODUCTION
¡ Software developer with the Linguistics Department and CSD
¡ Developing MTurk based experiments for about 3 years
MTURK URLS
¡ Sandbox
¡ https://requestersandbox.mturk.com/begin_signin
¡ https://workersandbox.mturk.com/mturk/beginsignin
¡ Production
¡ https://requester.mturk.com/mturk/beginsignin
¡ https://www.mturk.com/mturk/beginsignin
https://acceptability-judgements-demo.firebaseapp.com/
1. External Link 2. Internal Single Page Survey 3. Embedded
Easy setupDifficult close out
Complex setupEasier close out
EXTERNAL LINK – WORKER’S VIEWS
¡ Live demo
EXTERNAL LINK – REQUESTER’S VIEW
¡ Tutorial
¡ https://blog.mturk.com/tutorial-getting-great-survey-results-from-mturk-and-qualtrics-f5366f0bd880
1. External Link 2. Internal Single Page Survey 3. Embedded
Easy setupDifficult close out
Complex setupEasier close out
SINGLE PAGE SURVEY
¡ Worker doesn’t have to visit an external URL
¡ No confirmation codes to validate
¡ Involves writing some HTML
¡ https://bradylab.ucsd.edu/ttt/
¡ Data gets saved in Mturk
SINGLE PAGE SURVEY – WORKER’S VIEWS
¡ Live demo
1. External Link 2. Internal Single Page Survey 3. Embedded
Easy setupDifficult close out
Complex setupEasier close out
EMBEDDED
¡ Worker doesn’t have to visit external URL
¡ No confirmation codes to validate
¡ Data is saved to Firebase or Qualtrics
¡ Better functionality from worker’s perspective
EMBEDDED SURVEY – WORKER’S VIEWS
¡ Live demo
EMBEDDED
¡ Mturk Command Line Tools (CLI)
¡ https://requester.mturk.com/developer/tools/clt
¡ http://web.mit.edu/cathywu/Public/AMT/aws-mturk-clt-1.3.0/UserGuide.html#externalhit
¡ AMT Manager
¡ https://nusubdb.ling.northwestern.edu/AMTManager/
¡ Invite only for now
EMBEDDED
¡ Firebase or Qualtrics compatible
¡ Some special code “hooks” required
¡ Templates available if interested
¡ Speech In Noise frameworks
https://acceptability-judgements-demo.firebaseapp.com/
workerId = ABC1234567
https://www.mturk.com/mturk/externalSubmit?workerId = ABC1234567
IN SUMMARY
External URL¡ Worker must visit an external URL
¡ Can’t close original window
¡ Worker enters confirmation code (possible typos)
¡ Data is saved to Firebase or Qualtrics
¡ Requester matches confirmation codes to workers
Single Page Survey¡ Worker doesn’t have to visit an external URL
¡ Long scrolling page, difficult for worker
¡ No confirmation codes to validate
¡ Involves writing some HTML
¡ Data gets saved in Mturk
IN SUMMARY
Embedded Survey¡ Worker doesn’t have to visit external URL
¡ No confirmation codes to validate
¡ Data is saved to Firebase or Qualtrics
¡ Better functionality from worker’s perspective
¡ Little more work to get set up initially (Firebase/Qualtrics templates)
¡ Can embed games, other types of interesting tasks
¡ Speech in Noise frameworks
¡ https://northwestern.box.com/s/9g2rigz1iqh4ymfkgunq6t31u3iycbpr
THE LIFE OF AN MTURK WORKER
https://turkopticon.ucsd.edu/
WHY USE MTURK
¡ Easy set up
¡ Diverse population
¡ More data over a short period of time
¡ Everyone else is doing it
REASONS NOT TO USE MTURK
¡ You have to manage everything
¡ Payments
¡ Technical issues
¡ Angry workers
¡ Preventing the same workers from repeating your HIT
¡ Can get expensive
¡ Not all workers are naive
¡ https://doi.org/10.1016/j.tics.2017.06.007)
¡ Not all languages and countries are evenly represented
¡ https://www.semanticscholar.org/paper/The-Language-Demographics-of-Amazon-Mechanical-Tur-Pavlick-Post/034bd3a6f63caf2bb2dc2762e43401cd204b86b5)
ALTERNATIVES
¡ Turkprime
¡ https://www.turkprime.com
¡ Prolific
¡ https://www.prolific.ac/
QUESTIONS?