Embed Size (px)
DESCRIPTION
Amazon Mechanical Turk (https://requester.mturk.com), a crowdsourcing marketplace, adds value by processing work that cannot be computerized.Typically, these include large-scale [data] projects - think Amazon.com's 500MM product database - which requires some form of human judgment to help digitize, validate, moderate, tag, cleanse, categorize, dedupe, etc...This is accomplished by breaking down the project into micro-tasks, distributing to qualified cloud-based workers, who then deliver the results.
Citation preview
Introducing Amazon Mechanical Turk
A Brief Overview of What, Why, and How
John Hoskins
Agenda
What is Mechanical Turk?
When would you use it?
How are others using it - Case Studies
How – Implementation options
What is Mechanical Turk?
Mechanical Turk is a Marketplace.
Mechanical Turk is a marketplace for work.
Mechanical Turk gives businesses and developers access to an on-demand, scalable workforce.
• Flexibility: Scale your workforce up and down quickly• Accuracy: Get high quality, efficient and cost effective results.• Price: Pay only when you are satisfied with the results.• Speed: Start receiving results in minutes
How it works:. “Design & Publish” “Validate,
Pay & Go”
Workforce
500,000 Workers
190+ Countries
Who are the Workers?• Workers are global• Workers represent numerous skill sets• 24X7 Follow the sun availability
Managing Your Workforce• You can choose who is allowed to work on your assignments• Can narrow to US based (i.e. when you need western culture competence) • Leverage qualifications to find your best Workers
When would you useMechanical Turk?
Business Challenges
Our Community is generating 10,000 comments per day – how can we eliminate SPAM?
We pull information from hundreds of sources – how do we normalize and cleanse?
We have thousands of legacy assets – how do we make them searchable?
Our heuristics need jump started – how can I generate thousands of reference data points fast?
We need to add a new attribute to our data?
Business Challenges
Requires Human Judgment: Recognition, Reasoning, Context, Prior Knowledge
Cannot be completely solved algorithmically
Has unpredictable fluctuations in volume
How are businesses using it? Data Management
Data Verification Data Entry & Collection Data De-duplication Algorithm Training
Content & Media Moderate Photos & Content Content Creation & Editing Transcription
Categorization Classification Tagging Keywords Sentiment Analysis
Business Services Search Relevancy Data Gathering Enhancement
How are others usingMechanical Turk?
Search Enhancement
Make your assets more discoverable• Categorize them into your search tree• Add descriptions, attributes and keywords• Enhance listing data
How it works: Item description
is sent to Mechanical Turk
for review by Workers
Workers select the best category and sub-category
for the item
Items are placed in that branch of the search tree
Content is more easily discovered through search
Data Handling
Resolve Data conflicts directly in your business process• Remove duplicates• Normalize data entries• Resolve conflicts and collisions• Enhance entries
How it works:Business
Process Ingests Data Elements from multiple
sources
Items flagged for conflict are sent to Mechanical
Turk
Workers review item and resolve
conflict
Business Process
continues
How to apply your task to Mechanical Turk?
Your Project Tasks
“Working Backwards”• What are the business results that you want? • How will you use those results? • What format do the results need to be in? • Which tasks benefits from human intelligence?
Break your overall project into smaller “tasks”• Consolidate like items and processes
Define instructions for each task
Example Workflow
Content Moderation with API integration
The Key components of your HIT
Instruction• Guidelines for how to do
a HIT
Data • “batch” concept
Question(s)
What kind of dress is this?
Cocktail
Sun dress
Bridal dress
Work Catalog
Workers discover available HITs through the listings catalog
Name of Requester, Name of HIT, Price and Qualifications are displayed
HIT Example
Specific instructions on what to do – and how to do it
Providing as much information as available
Adjudication Strategies
Leverage different methods of determining the correct answer by designing for accuracy in your HIT:• Multiple Choice
• Plurality• Known Answers (“Gold Standards”)
• Free-form• Multiple HIT Workflow’s where a new Worker “edits” or “grades” previous
Worker submissions
• Creative Tasks• Bonus Payments
Your Workers
Identify and manage your best Workers
All Workers
Your Workers
Your Trusted Workers
• All available Workers• 500,000 + • 24x7, 365
• Any Worker who completes a HIT
• Evaluate performance on your HITS
• Your best Workers with proven quality and availability
Balances & Levers
Leverage market dynamics to optimize speed, price and accuracy to meet your business needs.
Accuracy
Price
Speed
Requester access:
Web UI
• Easy Access • Create HITS in
HTML• Upload and
manage data via CSV
API & CLT
• Programmatic Access
• Multiple SDKS• Massive scale
Apps & Solution Providers
• Experts at Workflow design
• They do the training and management of Workers for specific kinds of tasks
Self Service Applications
Mechanical Turk supports a robust ecosystem of providers that have built or can build solutions to meet your needs.
Custom Solution Providers
Advantages of Mechanical Turk
Converts a fixed cost into a variable cost
Significantly reduces ongoing headcount expenses
Offers a pay-as-you-go staffing model
LowersCosts
Eliminates lead time of interviewing and staffing
Work gets done faster, by working in parallel
Work continues around the globe, around the clock
Reduces work turnaround time
Gives you access to an on-demand workforce
100% Efficiency
Maintain the flexibility your business demands
Use only the capacity you need, when you need it
Eliminates Staffing Issues
