CrowdFlowIntegrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility

Alex Quinn, Ben Bederson, Tom Yeh, Jimmy Lin

Human Computation

ThingsHUMANS

can do

ThingsCOMPUTERS

can do

Translation

Photo tagging

Face recognition

Human detection

Speech recognition

Text analysis

Planning

Human Computation

ThingsHUMANS

can do

ThingsCOMPUTERS

can do

Translation

Photo tagging

Face recognition

Speech recognition

Human detection

Text analysis

Planning

Example: Human detection

Trade-off space

Quality

Spee

d, A

fford

abili

ty

Computers

Human Workers(traditional)

Human Computation

Trade-off space

Quality

Spee

d, A

fford

abili

ty

Computers

Human Computation

Human Workers(traditional)

Man-Computer Symbiosis

Automation with human

post-correction

Supervisedmachinelearning

humans

computer

speedcostquality

computer

humans

speedcostquality

Man-Computer Symbiosis

CrowdFlowAutomation with human

post-correction

Supervisedmachinelearning

humanscomputer

speedcostquality

humans

computer

speedcostquality

computer

humans

speedcostquality

Mechanical Turk

Human Detection – Starting point

Human Detection – Task

Human Detection – Results

Quality

Spee

d, A

fford

abili

ty

60% 90%

119 images took3 hrs 50 mins

and cost $2.38

Human Detection – Scenarios

Quality

Spee

d, A

fford

abili

ty

60% 90%

1000 photos at 72% accuracy would take 12 hrs 20 mins and cost $8.00

119 images took3 hrs 50 mins

and cost $2.38

Vision: Richer modelInput with computer results

Validator

Appraiser

Fixer Worker

Correct Incorrect

Fix Start over

Output

Lessons Learned Design for overall needs/constraints

Practical advice: Pay consistently and reasonably Reject only work that is definitely cheating Build in fair cheating deterrence from the start Keep instructions short, but always clear

Contact: Alex Quinn aq@cs.umd.edu