Understanding crowdworkers’ learning practices

Anoush MargaryanCaledonian Academy

Glasgow Caledonian University, UK

@anoush

Vakharia, D. & Lease, M. (2015)Beyond Mechanical Turk: An analysis of paid crowdwork platforms.

Types of crowdwork• Microwork (AMT, CrowdFlower):

– microtasks (automated and routine)– short timeframe (seconds or minutes) – algorithm-governed– anonymous– small pay – lower barriers to entry

• Online freelancing (Upwork – formerly oDesk, Freelancer):– simple or complex projects– longer timeframe (hours, days or months)– worker and employer identities more transparent – professional skills often required

Kuek, S. C., et al. (2015). The global opportunity in online outsourcing. Washington, DC: World Bank.

Examples of crowdwork tasks• Data entry• Graphic design (logos, brochures, web products

and mobile apps, drawing and illustration)• Video production• Transcription• Writing texts• Rating sentiment about a product• Screening and tagging images• Searching and extracting information from

websites• Completing surveys

Microwork vs online freelancing

TASKS Microwork Online freelancing

High complexity

Medium complexity

Low complexity

Economic significance

• Estimated gross revenue (in 2013) - $2 billion

• Forecast for 2016 - $4.8 billion ($4.4 billion for online freelancing and $0.4 billion for microwork)

• Estimated number of crowdworkers worldwide – 48 million

Kuek, S. C., et al. (2015). The global opportunity in online outsourcing. Washington, DC: World Bank.

Demographics• 2/3 of crowdworkers come from:

– USA– India– Philippines

• Predominately male, below 35 y.o.

• Education - university degree:– 75% of online freelancers– 33% of microworkers

• Hours worked: majority work for fewer than 20 hrs/w

• Motivation -income generation– For microworkers – supplementary income– For online freelancers – the only means of income

Kuek, S. C., et al. (2015). The global opportunity in online outsourcing. Washington, DC: World Bank.

Why study crowdworker learning?

• Worker self-organisation to learn when there is no access to traditional workplace learning support (eg training, access to experienced colleagues)

• Social dimension of workplace learning in crowdwork

• Learning-intensity of crowdwork

• Enhancing the learning potential of crowdwork

Research design1. Self-regulated Learning at Work Questionnaire, SRLWQ

(Fontana et al, 2015):– Personal details (education, profession, employment status,

country, experience level, etc)– SRL strategies: planning, performance and reflection– Workplace Learning Activities– Nature of work tasks (routinisation, decision-making, expertise

& collaboration required, etc)2. Interviews with crowdworkers (learning pathways,

learning and work goals, motives, aspirations)3. Interviews with platform providers and employers (to

contextualise crowdworkers’ perspective)4. Document analysis (training provisions, policies)

Hypotheses

• There are qualitative and quantitative differences in learning practices within microwork and online freelancing

• Learning-intensity of microwork is lower than that of online freelancing

SRLW Questionnaire

Fontana, P., Milligan, C., Littlejohn, A., & Margaryan, A. (2015). Measuring self-regulated learning in the workplace. International Journal of Training and Development, 19(1), 32-52

Margaryan, A., Milligan, C., & Littlejohn, A. (2011). Validation of Davenport’s classification structure of knowledge-intensive processes. Journal of Knowledge Management, 15(4), 568-581.

1. Demographic questions – 10 items 2. Workplace learning activity – 14

items 3. Self-regulated learning – 33 items4. Typology of work tasks – 10 items

Pilot survey response

Microwork (CrowdFlower)–Total n=127–Incomplete n=30–Suspicious n=4–Usable n=93

Online freelancing (Upwork):

–Total n=30–Incomplete n=8–Suspicious n=2–Usable n=20

DemographicsUpwork (n=20)• Age range: 1949-1997

– 75% Millennials (1980+)– 20% Generation X (1965-79)– 5% Baby Boomers (1946-64)

• Gender:– Male 40%– Female 60%

• Countries:– 25% USA– 20% Philippines– 10% each India, Romania,

Serbia– 5% each Bosnia, Canada,

Pakistan, Thailand, UK

CrowdFlower (n=93)• Age range: 1950-1996

– 61% Millennials (1980+)– 32% Generation X (1965-79)– 7% Baby Boomers (1946-64)

• Gender:– Male 74%– Female 26%

• Countries:– 10% each India, Serbia, Bosnia– Bulgaria, Venezuela, Greece,

Indonesia, Macedonia, Russia, Turkey, Bangladesh, Canada, Italy, Romania, Ukraine, Croatia, Spain, Vietnam, Algeria, Brazil, Czech Republic, Estonia, Mexico, Portugal, Slovakia, Sri Lanka, Uruguay, USA

Education

Secondary School

Undergrad Masters Professional Qualifications

Doctorate0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

UpworkCrowdFlower

Employment status

Freelancer FT Employee PT Employee Retired Diasbled Student Homemaker0%

10%

20%

30%

40%

50%

60%

70%

80%

UpworkCrowdFlower

Professional background: Online freelancers

Professional background: microworkers

Experience on platform

1 yr or less 2-3 yrs 4-10 yrs 11+ yrs0%

10%

20%

30%

40%

50%

60%

70%

Upwork

ClowdFlower

WLA: Online freelancersAcquiring new information

Working alone

Working with others

Following new developments in your field

Performing new tasks

Asking others for advice

Attending a F2F training course

Attending free online courses/tutorials

Using paid online tutorials

Self-studying professional literature

Replicating others’ strategies

Trial and error

Reflecting on your actions

Receiving feedback on your work

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

NeverSometimesRegularly

WLA: microworkersAcquiring new information

Working alone

Working with others

Following new developments in your field

Performing new tasks

Asking others for advice

Attending a F2F training course

Attending free online courses/tutorials

Using paid online tutorials

Self-studying professional literature

Replicating others’ strategies

Trial and error

Reflecting on your actions

Receiving feedback on your work

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

NeverSometimesRegularly

My work is/involves…Mostly routine tasks

Highly reliant on formal rules/procedures

Limited/no freedom to decide what should be done

Mostly systematically repeatable

Reliant on formal processes/standards

Dependent on integration across disciplinary boundaries

Improvisational

Highly reliant on my deep expertise/judgment

Dependent on collaborating with others

Highly reliant on my individual expertise/experience

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

CrowdFlower

Upwork

Conclusions

• Considerable learning and professional development appears to occur in both microwork and online freelancing

• Crowdwork is not lonely work - considerable amount of collaboration appears to take place

• Some quantitative differences in learning activities within microwork and online freelancing

• More analysis and data collection needed…

Next steps• Full study (2016-2018)

– Survey with a larger sample and more comprehensive analyses

– In-depth interviews with crowdworkers– Interviews with platform providers and employers– Study of platform documentation and policies

• Compare results to data from previous survey samples in traditional occupations– Financial analysts (WLBK project, 2012-2013)– Researchers, scientists and consultants (SRL@W

project, 2014-2015)– Medics and clinicians (Gates Foundation study, 2014-

2015)