Investigating Factors Influencing Crowdsourcing Tasks withHigh Imaginative Load

Raynor VliegendhartMartha LarsonChristoph KoflerCarsten Eickhoff (speaker)Johan Pouwelse

WSDM 2011 Workshop on Crowdsourcing for Search and Data Mining (CSDM 2011),Hong Kong, China, February 9–12, 2011.

2

OFirst Things First

3

• That title sounds pretty esoteric. What is this all about?

OFirst Things First

4

• That title sounds pretty esoteric. What is this all about?

• We are dealing with two phenomena:• HIT titles that try to prepare the worker for the task• HITs that require the worker to project into different roles or

situation

• We refer to this property of HITs as “Imaginative Load”

OFirst Things First

5

• Why “Imaginative Load”?• Observations• Further Investigation• Conclusions

OOutline

6

• Evaluation context:

• Novel search-related feature for a file-sharing system

• Term clouds as content descriptors

• Required the workers to project themselves into the role of

a user

IWhy “Imaginative Load”?

7

The actual HIT was preceded by a recruitment step:

RWhy “Imaginative Load”?

Recruitment HIT

100/100

Evaluation HIT0/405

QualifiedWorkers

8181

8

The turnout:

RWhy “Imaginative Load”?

Recruitment HIT

100/100

Evaluation HIT10/4054

QualifiedWorkers

81

9

Perhaps we need more eligible workers?

RWhy “Imaginative Load”?

Recruitment HIT

100/100

Evaluation HIT12/4055

Recruitment HIT

100/100

79

QualifiedWorkers

160

10

• HIT uptake is slow• Most workers do not do the actual HIT• If they do, then they don't do many iterations

• Hypothesis:“The recruitment task and the HIT titles were misleading. Once

the workers realised what they were supposed to do they lost interest.”

IObservation I

11

Can workers guess which types of content are available?

• HITs with and without term clouds• Several variations of term clouds

EProjection Into Different Roles

12

Jim and his large circle of friends have a huge collection of files that they are sharing with a very popular file-sharing program. The file-sharing program is a make-believe program. Please imagine that it looks something like this sketch:

FProjection Into Different Roles

13

(1) If you could download one of these files, which one would it be?

(1) Why would you choose this particular file for download and viewing?

(1) Think again about the file that you chose.Why did you guess that Jim or one of his friends would have this file in their collection?

VQuestions

14

• Some workers match literally between the mock-up frame and the questions

• The majority of workers is able to generalize from the situation or the mock-up frame

• Hypothesis:“HIT design can enhance the workers' success at completing

projection tasks.”

IObservation II

15

• Answer quality and HIT uptake of under the influence of

• Title• Questionnaire design

• 5 experimental conditions:• 5 HITs per condition• 10 workers per HIT

• $0.10 reward per assignment

EFurther Investigation

16

Title conditions1) A: Jim, his friends and a make-believe file-sharing program2) B: Jim, his friends and digital stuff to download3) C: Jim, his friends and interesting stuff to download

EFurther Investigation

17

• All HITs yielded serious results(two assignments rejected due to cheating)

• Title A: more than 2 days to complete

• Title B+C: each completed within a day

RFurther Investigation

18

Question conditions (using Title B):1) No preference questions at all,

only an explanation of all given judgments

2) No justification of preference asked

EFurther Investigation

19

Absence of preference questions:

• Cut and paste strategies possible due to generality of

question

• Serious answers become more verbose to capture the

generalized situation

No explanation of preference:

• Slight decrease in the degree to which the workers

managed to project into Jim and his friends

RFurther Investigation

20

• “High imaginative load” tasks can be successfully run on MTurk

• The key appears to be a combination of:• Signaling to workers the unique nature of the tasks

(which are possibly quite different from those they generally choose)results in faster HIT uptake

• Making each HIT require individualized free-text justification response improves the workers ability of projection and generalization

CConclusions

21

QQuestions?