Upload
montana
View
51
Download
0
Tags:
Embed Size (px)
DESCRIPTION
CrowdFlow Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility. Alex Quinn, Ben Bederson, Tom Yeh , Jimmy Lin. Human Computation. Translation Photo tagging Face recognition Human detection Speech recognition Text analysis Planning. Things COMPUTERS can do. - PowerPoint PPT Presentation
Citation preview
CrowdFlowIntegrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility
Alex Quinn, Ben Bederson, Tom Yeh, Jimmy Lin
Human Computation
ThingsHUMANS
can do
ThingsCOMPUTERS
can do
TranslationPhoto tagging
Face recognitionHuman detection
Speech recognitionText analysis
Planning
Human Computation
ThingsHUMANS
can do
ThingsCOMPUTERS
can do
TranslationPhoto tagging
Face recognitionSpeech recognitionHuman detection
Text analysisPlanning
Example: Human detection
Trade-off space
Quality
Spee
d, A
fford
abili
ty
Computers
Human Workers(traditional)
Human Computation
Trade-off space
Quality
Spee
d, A
fford
abili
ty
Computers
Human Computation
Human Workers(traditional)
Man-Computer Symbiosis
Automation with human
post-correction
Supervisedmachinelearning
humans
computer
speedcostquality
computer
humans
speedcostquality
Man-Computer Symbiosis
CrowdFlowAutomation with human
post-correction
Supervisedmachinelearning
humanscomputer
speedcostquality
humans
computer
speedcostquality
computer
humans
speedcostquality
Mechanical Turk
Human Detection – Starting point
Human Detection – Task
Human Detection – Results
Quality
Spee
d, A
fford
abili
ty
60% 90%
119 images took3 hrs 50 mins and cost $2.38
Human Detection – Scenarios
Quality
Spee
d, A
fford
abili
ty
60% 90%
1000 photos at 72% accuracy would take 12 hrs 20 mins and cost $8.00
119 images took3 hrs 50 mins and cost $2.38
Vision: Richer modelInput with computer results
Validator
Appraiser
Fixer Worker
Correct Incorrect
Fix Start over
Output
Lessons Learned Design for overall needs/constraints
Practical advice: Pay consistently and reasonably Reject only work that is definitely cheating Build in fair cheating deterrence from the start Keep instructions short, but always clear
Contact: Alex Quinn [email protected]