AMT: Requester UI

Amazon Mechanical TurkRequester User Interface Guide

Version

Amazon Mechanical Turk: Requester User Interface GuideCopyright © 2010 Amazon Web Services LLC or its affiliates.

This book was last updated 2010-11-03.

Amazon Mechanical Turk Requester User Interface Guide

Table of ContentsWelcome ............................................................................................................................................................. 1Introduction to Mechanical Turk .......................................................................................................................... 3

Overview of Mechanical Turk .................................................................................................................. 3Advantages ................................................................................................................................... 3

Mechanical Turk Concepts ...................................................................................................................... 3Requester ...................................................................................................................................... 4Human Intelligence Task (HIT) ...................................................................................................... 4Assignment .................................................................................................................................... 4Worker ........................................................................................................................................... 4Approval and Payment .................................................................................................................. 4Qualification Type .......................................................................................................................... 4Using the Requester User Interface .............................................................................................. 5

Getting Started ................................................................................................................................................... 6Creating Your Batch of HITs ............................................................................................................................... 7

The Design Tab ....................................................................................................................................... 7Creating a HIT Template ......................................................................................................................... 8

Publishing Your Batch of HITs .......................................................................................................................... 15Publishing Batches ............................................................................................................................... 15

Managing Batches ............................................................................................................................................ 18Displaying Batch Progress, Details, and Results .................................................................................. 18Configuring the Review Results Page .................................................................................................. 20

Filtering Results .......................................................................................................................... 21Approving and Rejecting Assignments ................................................................................................. 21

Reviewing Assignments in the RUI ............................................................................................. 22Reviewing Assignments Offline ................................................................................................... 22

Deleting a Batch ................................................................................................................................... 24Managing Workers ............................................................................................................................................ 26

Viewing Worker Statistics ..................................................................................................................... 26Blocking a Worker ................................................................................................................................. 27Unblocking a Worker ............................................................................................................................. 28Awarding a Bonus ................................................................................................................................. 29Assigning a Qualification to a Worker ................................................................................................... 30Revoking a Worker's Qualification ........................................................................................................ 30Managing Worker Details Offline .......................................................................................................... 31

Managing Qualification Types ........................................................................................................................... 33Creating a Qualification Type ................................................................................................................ 33Viewing Existing Qualification Types ..................................................................................................... 34Deleting Qualification Types .................................................................................................................. 35

Glossary ........................................................................................................................................................... 37Document History ............................................................................................................................................. 39Index ................................................................................................................................................................. 40


Welcome

Topics

• Prerequisites (p. 1)

• How This Guide Is Organized (p. 1)

• Amazon Mechanical Turk Resources (p. 2)

Amazon Mechanical Turk gives your business access to a scalable, on-demand workforce. By helpingyou break your work into smaller tasks that's done by multiple Workers simultaneously, Mechanical Turkhelps you get results faster and at a lower cost than was previously possible.

This is the Amazon Mechanical Turk Requester User Interface Guide. It is for people who want to accessAmazon Mechanical Turk functionality using a graphical user interface.

This guide occasionally refers to Amazon Mechanical Turk as "Mechanical Turk," and the AmazonMechanical Turk Requester User Interface as "RUI." All copyrights and legal protections still apply.

PrerequisitesThis guide does not assume that you have prior knowledge of Amazon Mechanical Turk. We recommendthat you visit the Resource Center to learn about best practices, user case studies, and other usefulinformation about Amazon Mechanical Turk.

How This Guide Is OrganizedThe following table describes the organization of this guide.

Relevant SectionsInformation

Mechanical Turk Concepts (p. 3)Basic concepts

Creating Your Batch of HITs (p. 7)Designing your HITs

Publishing Your Batch of HITs (p. 15)Publishing your HITs

Managing Batches (p. 18)Managing your batches

Version1

Amazon Mechanical Turk Requester User Interface GuidePrerequisites

https://requester.mturk.com/mturk/resources

Relevant SectionsInformation

Managing Workers (p. 26)Managing Workers

Managing Qualification Types (p. 33)Managing Qualifications

Document History (p. 39)What's new in this release.

Amazon Mechanical Turk ResourcesThe following table lists related resources that can help you learn more about Amazon Mechanical Turk.

DescriptionResource

Contains links to helpful guides, videos, webinars, casestudies, and other tools to help you get started and manageyour HITs.

Amazon Mechanical Turk ResourceCenter

Includes important information about your account and taxes.Amazon Mechanical Turk FAQ

Consists of the terms and conditions for using the MechanicalTurk site.

Participation Agreement

Describes the information Amazon gathers from users.Privacy Notice

Provides email support for using the Requester User Interface.Contact Us

Version2

Amazon Mechanical Turk Requester User Interface GuideAmazon Mechanical Turk Resources



https://requester.mturk.com/mturk/help?helpPage=requester

https://requester.mturk.com/mturk/conditionsofuse

https://requester.mturk.com/mturk/privacynotice

https://requester.mturk.com/mturk/contactus

Introduction to Mechanical Turk

Topics

• Overview of Mechanical Turk (p. 3)

• Mechanical Turk Concepts (p. 3)

This introduction to Mechanical Turk provides a detailed summary of this user interface. After readingthis section, you should have a good idea what the Amazon Mechanical Turk Requester User Interface(RUI) offers and how it can fit in with your business.

Overview of Mechanical TurkMechanical Turk provides a marketplace for work and gives you access to Workers around the world andaround the clock. Mechanical Turk enables you to complete a variety of tasks, including data categorization,moderation, verification, and tagging.

AdvantagesThe major advantages of Mechanical Turk are:

• On demand workforce—Workers are available whenever you need them

• Scalable workforce—You can use one or thousands of Workers to complete your assignments

• Qualified workforce—You can qualify Workers to work on your assignments to get the best results

• Pay only for satisfactory work—You don’t pay Workers or Mechanical Turk fees until you accepttheir assignments; if you reject their work, they do not get paid

Mechanical Turk ConceptsTopics

• Requester (p. 4)

• Human Intelligence Task (HIT) (p. 4)

• Assignment (p. 4)

• Worker (p. 4)

Version3

Amazon Mechanical Turk Requester User Interface GuideOverview of Mechanical Turk

• Approval and Payment (p. 4)

• Qualification Type (p. 4)

• Using the Requester User Interface (p. 5)

This section describes the concepts and terminology you need to understand to use Mechanical Turkeffectively.

RequesterA Requester creates tasks on Mechanical Turk for Workers to work on. As a Requester, you use theRequester User Interface (RUI) to create tasks, check the status of your tasks, and accept or reject workperformed on tasks. Workers see your account name (specified by your Amazon.com account) whenthey view and accept your tasks.

Human Intelligence Task (HIT)A Human Intelligence Task (HIT) is a single, self-contained task a Requester creates on Mechanical Turk,for example, "Identify the color of the car in the photo."

AssignmentYou can assign many Workers to work on the same HIT. This is a useful way of getting agreement onwhat is correct. A Worker can only accept a HIT once and can only submit one assignment per HIT. Thisguarantees that multiple Workers must complete a HIT that has multiple assignments.

If a Worker fails to complete an assignment before the time frame specified (i.e., the Worker abandonsthe HIT), or if the Worker chooses not to complete it after accepting it (i.e., the Worker returns the HIT),the assignment becomes available for other Workers to work on.

WorkerA Worker is a person who completes assignments. Workers use the Mechanical Turk web site(http://mturk.amazon.com/) to find assignments to work on, submit responses, and manage their account.A Requester sees the Worker's account ID for every assignment submitted.

Approval and PaymentWorkers submit responses for assignments. If you approve their work, Mechanical Turk transfers the HITreward from your Mechanical Turk account to their Amazon Payments accounts. Workers don't get paidif you reject their work.

When you post a batch of HITs, you agree to approve or reject work by a specified deadline. If thatdeadline passes, Mechanical Turk automatically approves the assignments and pays the Workers whosubmitted work.

Before you can post your batch of HITs, you must have enough money in your account to pay for all ofthe work.You can deposit money into your Mechanical Turk account at any time using the Requesterweb site (http://Requester.mturk.amazon.com/).

Qualification TypeMechanical Turk provides System Qualification Types that automatically keep track of a Worker's accountstatistics and attributes. For example, one System Qualification Type is the Worker's approval rating for

Version4

Amazon Mechanical Turk Requester User Interface GuideRequester

http://mturk.amazon.com/

http://requester.mturk.amazon.com/

all approved and rejected assignments.You can use System Qualification types to control who can andcannot work on your HITs. For example, you can require that Workers have a 95% or greater approvalrating to work on your HITs.

You can also create your own Qualification Types and assign them to Workers.You can stipulate thatonly Workers assigned your Qualification Type can work on your HITs. Each Qualification Type you createincludes a name, description, and a score, which is an integer from 0 to 100.You use the score todifferentiate Workers that have the same Qualification Type.

Using the Requester User InterfaceThe Requester User Interface (RUI) makes it easy to create a HIT template, publish HITs, manage batchresults, and manage Workers. The following graphic and process show the basic workflow.

Using the RUI

Define the properties and design the layout of the HIT template.1

Publish your batch of HITs.2

Approve or reject the work done on your HITs.You can download the results, review them offline,and republish rejected HITs.

3

View a Worker’s approval rating, award bonuses, assign qualifications, or block Workers from workingon your HITs.

4

Version5

Amazon Mechanical Turk Requester User Interface GuideUsing the Requester User Interface

Getting Started

To get started, you must create a Mechanical Turk account, or sign in if you have one.

To use the Requester User Interface

1. Go to the Amazon Mechanical Turk Resource Center and do one of the following.

Do This...To...

Click Sign Up Now and follow the on-screeninstructions.

Create a Mechanical Turk account

Click Sign In and follow the on-screeninstructions.

Sign in using your Mechanical Turk account

2. Click the Design tab.

Version6


https://mturk-requester.integ.amazon.com/mturk/welcome

Creating Your Batch of HITs

Topics

• The Design Tab (p. 7)

• Creating a HIT Template (p. 8)

This section shows you how to create a batch of HITs.

The Design TabYou need to create a HIT template to create a batch of HITs.You can create a HIT template by customizingan existing template. The Design tab lists available templates in two tables. The Your HIT Templatestable lists templates you created.The Sample HIT Templates table lists templates supplied by MechanicalTurk.

Version7

Amazon Mechanical Turk Requester User Interface GuideThe Design Tab

Designing Your HIT TemplateA HIT template is an HTML page that presents HITs to Workers.There are two kinds HIT templates; onesthat:

• Don't contain variablesIn this case, all Workers see the same HTML page.You might use this to create a survey.

• Contain variables so that every HIT has the same format but each HIT variesFor example, you might ask Workers to provide key words for photos and each HIT would have adifferent photo.

If your HIT template contains variables, you must provide a data file that contains the values for thevariables. The column headings in the data file must match the variable names in the HIT template. Onerow of data in the data file creates one HIT. The values can be text, or the URLs of images or videos.

The RUI merges the HIT template and the values in the data file to create a batch of HITs.

Creating a HIT TemplateThe following process describes the steps you take to create a HIT template.

Creating a HIT Template

Define the HIT template properties. These properties are HIT metadata, for example, how longWorkers can work on a HIT.

1

Design the HTML HIT template layout. Take into consideration the variables in your HIT data file.In this step, you create the template the RUI uses to create the HITs in the batch.

2

Preview the HIT template.3

Version8

Amazon Mechanical Turk Requester User Interface GuideDesigning Your HIT Template

To create a HIT template

1. In the HIT Sample Templates table, choose one of the provided templates by clicking one of the Startwith this template buttons.The following image shows the HIT Sample Templates table.

The following image shows the RUI display for the Image Tagging HIT template.

Version9

Amazon Mechanical Turk Requester User Interface GuideCreating a HIT Template

2. Use the following table to enter the HIT's properties.

DescriptionField

Enter a template name. The RUI does not display this name to Workers. If youuse a supplied template, the RUI fills in this field.

Template Name

Enter the name of the HIT. Make sure to be specific, such as "Tag landmarkimages" instead of "Tag photos." The RUI displays this title to Workers.

Title

Describe the HIT. The search mechanism searches through this description souse words you think would help Workers find your HITs.

Description

Enter a comma-separated list of words that Workers can use to find your HIT.Include your Requester name.

Keywords

Specify the maximum amount of time a Worker can work on an assignment.TheRUI withdraws the assignment from the Worker once this time expires so otherscan work on it. The countdown starts once the Worker accepts the assignment.

Time Allowed PerAssignment

Specify how long Workers can accept HITs in the batch. Workers can't acceptHITs in the batch once this time expires. Workers can finish working onassignments they previously accepted even though the batch is no longeravailable for others to work on.

HIT Expires in

Use the drop-down menu to specify Qualification Types the Worker must haveto qualify for working on the batch.You can use System Qualification Typesprovided by Mechanical Turk or custom Qualification Types you create.

Worker must meetthe followingcriteria..

Click this button to specify additional qualifications.Add anothercriteria

Select this check box to require that a Worker have the specified qualificationsto preview these HITs.

Required forpreview

Specify how much you'll pay the Worker if you approve an assignment.Reward perassignment

Specify the number of Workers you want to work on each HIT. One assignmentper HIT means that only one Worker works on a HIT.You might want multipleWorkers to work on a HIT to see if there is agreement, which can increase yourtrust in the results. A Worker can only work on one assignment per HIT. If a HIThas multiple assignments, the RUI guarantees that multiple Workers will workon the HIT.

Number ofassignments perHIT

Specify when the RUI will automatically approve HITs you haven't reviewed.This limit ensures that Workers get paid in timely manner.

Results areautomaticallyapproved in

3. Click Design Layout.The Design Layout tab appears.

Version10


4. Use the controls in the formatting toolbar to edit the text on the page.The icons are similar to those used in most word processors. If you hover your mouse over an icon,the RUI displays a description of it. To author in HTML, click Edit HTML Source.

5. Choose the names for the variables in your template.

If you use a data file to populate your HITs, you must make the variable names match the columnheadings in your data file. For example, if the column heading is continent, ${continent} is thevariable name you'd use in your HIT template.You use a dollar sign ($) and curly brackets to createa text variable.

Each row in your HIT data file must contain all of the values for one HIT, as shown in the followingfigure.

Version11


6. Enter variables for images and videos in the template.

If your template includes variables for images or videos, you must edit the HTML directly by clickingEdit HTML Source.

Refer to the following table to see how to create variables for images and videos.

FormatVariable

HTML image tag, <img>, where the alt attribute defines the name of the variable, forexample:

<img width="200" height="200" alt="imagevariableName"style="margin-right: 10px;" src="${image_url}" />

image

HTML object tag, <object>, where the name attribute defines the name of the variable, forexample:

<object width="425" height="344"><param name="movie"value="${video_url}"></param><param name="allowFullScreen"value="true"></param><embed src="${video_url}"type="application/x-shockwave-flash" allowfullscreen="true"width="425" height="344"></embed></object>

video

7. Create your data file.Separate the values in the HIT data file with commas and save it using the .csv format. Manyspreadsheet applications, including Microsoft Excel, can save files as .csv. One row of data shouldexactly match the number of values needed in each HIT. The order in which you present the variablesin the HIT template does not need to match the order of columns in the .csv file.

Version12


Note

The RUI doesn't support line breaks between cells nor does it support "\r" as a line breakcharacter. Apple Macintosh computers insert this character when they convert a MicrosoftExcel table into a .csv file.

If your HITs contain images or videos, you must include links to them in the data file, and the imagesand videos must be publicly accessible. The RUI does not provide a tool for uploading images orvideos. Consider using one of the publicly-available tools, such as Bucket Explorer, to upload yourimages into Amazon S3.

8. Create answer fields in your template where Workers can enter answers.

You must include the HTML name attribute in the answer field definition because the name becomesa column heading in the results table. In the following example, the answer field name is Landmarkand its size is 25.

<input type="text" name="Landmark" id="Field1" size="25" /></td>

The RUI returns results in a table stored in a comma-separated-value file format with the extension,.csv.The column headings are the variable names prepended by Input, and the answer field names,prepended by Answer. For example, if the variable is continent, the column heading isInput.continent. The following is a sample results table.

One row in the results table contains the responses for one assignment.

9. Save the HTML of your HIT template.

Mechanical Turk deletes the HIT templates if you don't use them for 120 consecutive days. If you needto access your template longer, we recommend that you copy the HTML and save it on your ownsystem.

10. Click Preview and Finish.

The RUI displays the HIT Preview page.

Version13


11. Click Finish if you're satisfied with the HIT properties and template.

You have set the HIT properties for your HIT template and designed the HTML page that Workers willsee. Now, you need to publish the HIT template to create the batch of HITs and make them available forWorkers to work on.

Version14


Publishing Your Batch of HITs

Publishing your HITs on the Mechanical Turk web page gives Workers the opportunity to work on them.If you're publishing a HIT without variables, you only need to publish the HTML page you created in theprevious section. If your HIT template contains variables, you must upload a .csv data file that containsthe values for the variables in each HIT.

Publishing BatchesUse the following procedure to publish your batch of HITs.

Note

The following procedure assumes that you will upload a HIT data file.

To publish a batch of HITs

1. Click the Publish tab.The RUI displays the Select HIT Template page.

2. Click Select beside the HIT template you want to publish.The RUI displays the Upload Input Data page if your HIT template contains variables. If it doesn't,the RUI skips to task 3, Preview.

Version15

Amazon Mechanical Turk Requester User Interface GuidePublishing Batches

https://www.mturk.com/mturk/findhits

3. Click Browse to locate the .csv data file that you want to upload, and click Upload.The RUI displays the HIT Preview page.

Version16


4. Carefully review your HITs so that you catch mistakes before publishing.

a. To see the next HIT, click Next HIT.You can preview up to 200 HITs in your data file.

Note

To use a different, previously-loaded .csv file, click Select a Different Input File.

b. Click Next HIT after previewing your HITs.The RUI displays the Batch Summary page.

This page shows the total amount you will pay Workers and Mechanical Turk if you approve all ofthe assignments. The following table explains the sections of the Batch Summary page.

DescriptionSection

Contains the values you set on the Design tab for the batch properties, including thenumber of days the batch can exist before expiring, and the number of days you haveto approve assignments before the RUI automatically approves the assignments.

BatchProperties

Calculates the number of assignments per batch. In this case, there are threeassignments per HIT and three HITs for a total of nine assignments.

HITs

Calculates the cost of the batch assuming you approve all assignments.The total costis the number of assignments multiplied by the price per assignment plus theMechanical Turk fee.You must have the total cost in your account before you canpublish the HIT. If you don't, the RUI asks you to add money to your account.

Cost

5. Click Publish HITs to publish the batch of HITs.

Version17


Managing Batches

Topics

• Displaying Batch Progress, Details, and Results (p. 18)

• Configuring the Review Results Page (p. 20)

• Approving and Rejecting Assignments (p. 21)

• Deleting a Batch (p. 24)

On the Manage tab, you can manage your batch results, the Workers who work on your batches, andyour Qualification Types. On the Batches page on the Manage tab, you can view:

• The progress of your batch

• The batch details

• The batch results

Displaying Batch Progress, Details, and ResultsThe following procedure describes how to display information related to your batches.

To display the batch progress, details, and results

1. Click Batches on the Manage tab.The RUI displays headings for batches in different states.

2. Click the batch state you want to view.The RUI displays batches in that state.

Version18

Amazon Mechanical Turk Requester User Interface GuideDisplaying Batch Progress, Details, and Results

The progress bar shows two quantities:

• % submitted—Percentage of completed HITs

• % published—Percentage of published HITs in your batchMost batches publish quickly, but extremely large batches can take several minutes to publish.

3. Click the name of the batch you want to view.The RUI displays the Batch Details page.

This page contains all of the batch properties you entered when creating this batch.

4. Click Results to see the submissions for the assignments in the batch.

The RUI displays the Review Results page.

Version19

Amazon Mechanical Turk Requester User Interface GuideDisplaying Batch Progress, Details, and Results

This page shows detailed information about the HIT results, such as the Worker who completed theHIT, the Worker's approval rating, and the Worker's submission for your HITs.

Configuring the Review Results PageThe RUI enables you to display the columns you want in the order you want on the Review Resultspage.

To configure the Review Results page

1. Click Customize View on the Batches page.The RUI displays the Configure Data Results page.

2. Do one or more of the following.

Do This..To..

Drag field names from the Available fields list into the Show these fieldsin this order list.

Add fields listed in theresults

Version20

Amazon Mechanical Turk Requester User Interface GuideConfiguring the Review Results Page

Do This..To..

Drag field names from the Show these fields in this order list into theAvailable fields list.

Remove fields listed inthe results

Drag the fields up or down in the Show these fields in this order list.Change the order ofthe fields in the results

3. Click Apply Settings.

Filtering ResultsThe RUI enables you to display a subset of the results.You can filter out results based on the amount oftime it took to do the assignment, and on the state of the result (submitted, approved, or rejected). Bydefault, the RUI displays only submitted work.

To filter the results

1. Click Filter Results on the Review Results page.The RUI displays the Filter Results page.

2. To show assignments in a specified state, select the Status Filter check box, and select the statein the drop-down menu that you want to view.

3. Select the Velocity Filter check box and specify a number of seconds if you only want to showassignments completed in a specified amount of time or less.

4. Click Apply Filters.

Approving and Rejecting AssignmentsTopics

• Reviewing Assignments in the RUI (p. 22)

• Reviewing Assignments Offline (p. 22)

A Requester approves or rejects assignments that Workers submit. When you approve an assignmentthe Worker gets paid; when you reject an assignment the Worker does not get paid.

You can approve results individually or all at once. Also, you can sort the table based on the Worker IDso you can review the results from a specific Worker all at once.

Version21

Amazon Mechanical Turk Requester User Interface GuideFiltering Results

Reviewing Assignments in the RUIIt's convenient to use the RUI to review relatively small numbers of assignments.

To approve or reject results using the RUI

1. On the Review Results page, do one of the following.

Do This..To..

Click the check box in the row(s) of results youwant to approve, and click Approve.

Approve individual results

Click the check box in the row(s) of results youwant to reject, and click Reject.

Reject individual results

Click Approve All, or click the check box in thecolumn heading of check boxes and clickApprove.

Approve all results in the batch

Click the check box in the column heading ofcheck boxes and click Reject.

Reject all results in the batch

2. If you want to sort the list by Worker, click the Worker ID column heading.

Tip

You can easily approve or reject all of a Worker's work. Over time you might find a Workerwho consistently submits excellent work and therefore has a very high approval rating. In thatcase, you could approve all of the Worker's work without reviewing all of it.

Mechanical Turk processes payments several times a day so there is always a small delay betweenapproval and payment. When there is a delay, Workers see HITs as Approved-pending payment intheir dashboard. The dashboard displays Paid for processed payments.

Reviewing Assignments OfflineIf a batch is large, it's often easier to download batch results, review them using another application, suchas Microsoft Excel, and then upload the revised file.

Note

Results are available for 120 days after you approve or reject them. If you need to access theresults longer than that, use the following procedure to download and archive them.

To download and review assignments offline

1. On the Review Results page, click Download CSV.The RUI displays the following dialog box.

Version22

Amazon Mechanical Turk Requester User Interface GuideReviewing Assignments in the RUI

2. Click the word, here.The RUI displays the following dialog box.

3. In this dialog box, do one of the following.

Do This..To..

Select Open with and choose the application from the pull down list thatyou want to use to view the results. The application you select must beable to display .csv files.

To open and view theresults

Select Save to Disk.To save the results

4. Click OK and open the results file with a .csv compatible application, such as Microsoft Excel.

5. For each result, enter an X in the Approve column, or a reason to reject the result in the Rejectcolumn.

6. Save the spreadsheet as a .csv file.

Version23

Amazon Mechanical Turk Requester User Interface GuideReviewing Assignments Offline

Now that you've reviewed the results, you must upload your file so Mechanical Turk can approve or rejectwork based on the changes you made to the file.

To upload a .csv file into the RUI

1. On the Review Results page, click Upload CSV.The RUI displays the Review Results Offline dialog box.

2. Click Browse, select the .csv file you saved, and click Upload CSV.The following dialog box appears.

3. To republish rejected assignments, select the Republish rejected assignment(s) for other Workersto complete check box.

4. Click Yes to confirm your choices.

Note

Some versions of Microsoft Excel do not display international characters by default. If your HITtitle, description, or HIT results contain international characters, you must follow the instructionsfor your version of Microsoft Excel to import or activate international characters.

Deleting a BatchIf the batch you published isn't working the way you'd like, you can delete it.

To delete a batch

1. On the Batches page, click Delete on the batch you want to delete.

Version24

Amazon Mechanical Turk Requester User Interface GuideDeleting a Batch

The Delete Batch dialog box appears.

2. Click Yes.

Note

It can take several minutes to delete a batch. All Workers who accepted assignments beforeyou deleted the batch can continue working on them.The batch will not be completely deleteduntil all assignments accepted by Workers have either been returned, submitted, orabandoned.

Version25

Amazon Mechanical Turk Requester User Interface GuideDeleting a Batch

Managing Workers

Topics

• Viewing Worker Statistics (p. 26)

• Blocking a Worker (p. 27)

• Unblocking a Worker (p. 28)

• Awarding a Bonus (p. 29)

• Assigning a Qualification to a Worker (p. 30)

• Revoking a Worker's Qualification (p. 30)

• Managing Worker Details Offline (p. 31)

The Manage tab in the RUI enables Requesters to track Worker performance and take appropriate actions,including blocking Workers, awarding bonuses, and assigning qualifications. Requesters can view thefollowing details on all of the Workers who have worked for them:

• Worker’s approval rating on your HITs

• Worker’s blocked status

• Worker’s qualification score for up to five of your Qualification Types

Mechanical Turk automatically keeps track of the number of assignments you've approved and rejectedfor each Worker.You might choose to block a Worker whose assignments you consistently rejected inthe past.You might choose to give a bonus to Workers whose work is consistently excellent.

The following procedures begin on the Manage tab on the Workers page.

Viewing Worker StatisticsMechanical Turk enables you to view a Worker's statistics, which characterize what the Worker is goodat.

To view a Worker's statistics

1. Click Workers.The Manage Workers page appears.

Version26

Amazon Mechanical Turk Requester User Interface GuideViewing Worker Statistics

The Block Status column can have the following values:

• Never Blocked—Worker has never been blocked by you

• Blocked—Worker is not allowed to work for you

• Unblocked—Worker was blocked by you at one time, but no longer is

2. To take a specific action on an individual Worker, click a Worker ID.The RUI displays the Manage Individual Worker page.

On this page you can view the Worker's approval rating, as well as the number of assignments youapproved and rejected.

Blocking a WorkerIf Workers aren't performing to your standards, you can block them from working on your HITs.

To block a Worker

1. On the Worker page, double click the Worker you want to block.The Manage Individual Worker page appears.

Version27

Amazon Mechanical Turk Requester User Interface GuideBlocking a Worker

2. Click Block Worker.The Block Worker dialog box appears.

3. Enter a reason for blocking the Worker, and then click Block.The RUI sends this message to the Worker.

Unblocking a WorkerIf you mistakenly blocked a Worker, you can unblock them.

To unblock a Worker

1. On the Worker page, double click the Worker you want to unblock.The Manage Individual Worker page appears.

2. Click Unblock Worker.

Version28

Amazon Mechanical Turk Requester User Interface GuideUnblocking a Worker

The Unblock Worker dialog box appears.

3. Enter a reason for unblocking the Worker, and click Unblock.The RUI sends this message to the Worker.

Awarding a BonusYou can award bonuses to Workers to encourage them to keep working for you.

To give a Worker a bonus

1. On the Worker page, double click the Worker you want to award a bonus.The Manage Individual Worker page appears.

2. Click Bonus Worker.The Bonus Worker page appears.

Version29

Amazon Mechanical Turk Requester User Interface GuideAwarding a Bonus

3. Enter the amount of the bonus, the ID of the assignment worked on, and the reason for the bonus,and then click Pay Bonus Now.

Assigning a Qualification to a WorkerThis section shows how to assign a Worker a Qualification Type. For information about creating aQualification Type, see How to Manage Qualification Types.

To assign a Qualification Type to a Worker

1. On the Manage Individual Worker page, click Assign Qualification Type.The RUI displays the Assign Qualification Type page.

2. Select the check box(es) next to the Qualification Type you want to assign to the Worker.The Score text box appears beneath each selected Qualification Type.

3. Enter a score (0 to 100) and click Assign.Use scores to differentiate Workers that have the same Qualification Type.The Worker's qualificationappears at the bottom of the window.

The edit link in the Score column enables you to change the qualification score.

Revoking a Worker's QualificationThe following procedure revokes a Worker's qualification.

To revoke a Worker's qualification

• On the Manage Individual Worker page, click the X next to the Qualification Type you want torevoke.

Version30

Amazon Mechanical Turk Requester User Interface GuideAssigning a Qualification to a Worker

Managing Worker Details OfflineIf you have a large number of Workers, it's easier to manage them offline using another application, suchas Microsoft Excel. This section shows how to download Worker information, edit it, and upload it intothe RUI.

To download and modify Worker details

1. On the Manage Workers page, click Download CSV.The RUI displays the Download Workers Results page.

2. Click the word, here to download the Worker data file.The Worker data downloads and opens in Microsoft Excel. The CURRENT-QualName column showsthe Worker's current qualification score. If the cell is blank, you haven't assigned the QualificationType to the Worker. The far, right column, CURRENT-BLOCK STATUS, shows the Worker's blockstatus.

3. To update the values, do one or more of the following:

• Indicate which Qualification Type to assign a Worker by putting a Qualification score in theUPDATE-QualName column, for example, UPDATE-Good Tagger. Qualification scores must be0 to 100, inclusive.

• To revoke the Qualification Type, enter Revoke in the UPDATE column for your Qualification Type.

• Block or unblock a worker by putting Block or Unblock in the UPDATE-BlockStatus column.

4. Save the .csv file, and click Workers.The Manage Workers page appears.

5. On the Manage Workers page, click Upload CSV.The Manage Workers Offline window appears.

Version31

Amazon Mechanical Turk Requester User Interface GuideManaging Worker Details Offline

6. Click Browse to find the .csv file you saved, and then click Upload CSV.

The Offline Processing window appears.

7. Click Yes to confirm that you would like to save the changes for your Workers.

Version32

Amazon Mechanical Turk Requester User Interface GuideManaging Worker Details Offline

Managing Qualification Types

Topics

• Creating a Qualification Type (p. 33)

• Viewing Existing Qualification Types (p. 34)

• Deleting Qualification Types (p. 35)

You can create your own Qualification Types, or use the ones supplied by Mechanical Turk.

Mechanical Turk provides System Qualification Types that automatically keep track of a Worker's accountstatistics and attributes.You can use System Qualification types to control who can and cannot work onyour HITs. For example, you can require that Workers have a 95% approval rating or greater to work onyour HITs.

You can create new Qualification Types to select Workers based on any criteria you want.You can assigna Qualification Type and a score to Workers who work for you.Then when creating a HIT, you can specifythe Qualification Type and the minimum score a Worker must have to be eligible to work on your HITs.

Note

The RUI does not support Qualification Tests that a Worker must take to achieve a qualification.Use the Mechanical Turk APIs or the command line tools to use tests. For more information, goto http://aws.amazon.com/documentation/mturk/.

The following procedures start on the Manage tab on the Qualification Types page.

Creating a Qualification TypeThe following procedure shows you how to create your own Qualification Type.

To create a new Qualification Type

1. Click Create New Qualification Type.The Create New Qualification Type dialog box appears.

Version33

Amazon Mechanical Turk Requester User Interface GuideCreating a Qualification Type

http://aws.amazon.com/documentation/mturk/

2. In the Friendly Name text box, enter a name that describes the Qualification Type.

3. In the Description text box, enter a description of the Qualification Type and click Create.The new Qualification Type appears in the list of Qualification Types on the Manage QualificationTypes page.

There is a short delay before the new Qualification Type appears in the list.You can refresh your browserto update the list.

To assign Workers your new Qualification Type, see Assigning a Qualification Type to a Worker (p. 30).

Viewing Existing Qualification TypesThe following procedure shows you the Qualification Types you created.

To view Qualification Types

• Click Qualification Types from the Manage tab.The Qualification Types page appears and shows you all of your Qualification Types.

Version34

Amazon Mechanical Turk Requester User Interface GuideViewing Existing Qualification Types

Deleting Qualification TypesThis section explains how to delete Qualification Types that you created.

1. Click Qualification Types on the Manage tab.The Qualification Types page appears.

2. Click the X next to the Qualification Type you want to delete.

3. When the RUI asks you to confirm the deletion, confirm it.

Version35

Amazon Mechanical Turk Requester User Interface GuideDeleting Qualification Types

There is a short delay before the Qualification Type disappears from the list.You can refresh yourbrowser to update the list.

When you delete a Qualification Type the RUI automatically removes it from all of your Workers andHIT templates. The RUI does not remove the Qualification Type from HITs that Workers are workingon.

Version36

Amazon Mechanical Turk Requester User Interface GuideDeleting Qualification Types

Glossary

approval If a Worker’s response satisfies your HIT’s task, you approve theassignment. When you approve an assignment Mechanical Turktransfers the HIT reward from your Mechanical Turk account to theWorker's Amazon Payments account.

assignment A HIT can have many assignments, that is, you can assign manyWorkers the same task. This is a useful way of getting agreement onwhat is correct. A Worker can only accept a HIT once and can onlysubmit one assignment per HIT.This guarantees that multiple Workersmust complete a HIT that has multiple assignments.

batch A batch is one or more HITs created and published using theMechanical Turk Requester User Interface.

HIT A Human Intelligence Task (HIT) is a task that a Requester createson Mechanical Turk. A HIT represents a single, self-contained task,for example, "Identify the color of the car in the photo."

keyword A keyword is a word that Mechanical Turk Workers use to search fora HIT.

Qualification Type Mechanical Turk provides System Qualification Types that keep trackof a Worker's account statistics and attributes.You can use SystemQualification types to control who can or cannot work on your HITs.For example, you can require that Workers have a 95% approvalrating or greater to work on your HITs.

You can also create your own Qualification Types and assign themto Workers.You could stipulate that only Workers assigned yourQualification Type can work on your HITs. Each Qualification Typeyou create includes a name, description, and a score.

Requester A Requester creates tasks on Mechanical Turk for Workers to workon. As a Requester, you use the Requester User Interface (RUI) tocreate tasks, check the status of your tasks, and accept or rejectsubmitted tasks. Workers see your account name (specified by yourAmazon.com account) when they view and accept your tasks.

Version37


Workers A Worker is a person who completes assignments. A Worker usesthe Mechanical Turk web site (http://mturk.amazon.com/) to find HITsto work on, submit responses for HITs, and manage his or her account.A Requester sees the Worker's account ID for every assignmentsubmitted.

Version38


http://mturk.amazon.com/

Document History

The following table describes the important changes since the last major release of this guide.This guidewas last updated on 03 November 2010.

Release DateDescriptionChange

2010-10-04The Manage tab now has a page called ManageBatches. On that page, you can approve and rejectassignments, import and export results, view batch details,and delete a batch. For more information, see CreatingYour Batch of HITs (p. 7).

Manage Batches

2010-10-04The Manage tab now has a page called ManageWorkers. On that page, you can block Workers, givethem bonuses, and assign them a Qualification Type andscore. For more information, see ManagingWorkers (p. 26).

Manage Workers

2010-10-04The Manage tab now has a page called ManageQualification Types. On that page, you can view thedetails of Qualification Types and create new ones. Formore information, see Managing QualificationTypes (p. 33).

Manage QualificationTypes

2010-02-25Now you can award a bonus while you're reviewing aWorker's results. For more information, see ManagingBatches (p. 18).

Bonus Workers

2010-02-25Now you can reject work and republish an assignmentwhile you're reviewing a Worker's results. For moreinformation, see Managing Batches (p. 18).

Reject and Republish

Version39


IndexAadvantage, 3approve assignment, 21approve payment, 4assign qualification, 30assignment, 4

approve, 21award, 29

Bbatch

delete, 24manage, 18publish, 15results, 18

block a Worker, 27bonus, 29

Cconcepts, 4

Ddata file, 12delete a batch, 24delete Qualification Type, 35Design tab, 7document history, 39

Ffilter results, 21

Gget started, 6glossary, 37guide organization, 1

HHIT, 4

properties, 10template, 8

create, 8

Mmanage batches, 18manage Workers, 26

offline, 31

Ooffline, Workers, 31organization of guide, 1

Ppublish batch, 15

Qqualification

assign, 30revoke, 30

Qualification Type, 4, 33create, 33delete, 35view, 34

Rreject assignment, 21related resources, 2Requester, 4results

batch, 18configure page, 20download, 22filter, 21upload reviewed, 24

reviewresults, 22

revoke qualification, 30RUI, 5

review, 22

Uunblock a worker, 28

WWorker, 4Worker details, 26

Version40


Technology

AMT: Requester UI