Crowdsourcing for Business:An Emerging Paradigm

Shourya RoyArea Manager, Human ComputationXerox Research Centre India, Bangaloreshourya.roy@xerox.com

Workshop on Social Computing, IIT Kharagpur5th Oct, 2012

Crowdsourcing : What is it? The act of taking a task traditionally performed by

an employee or contractor, and outsourcing it to an undefined, generally large group of people, in the form of an open call Digitization, image labeling, user studies, machine

translation evaluation, logo design, EDA simulation, innovation contests, . . .

Services Thrust – Xerox Confidential –

Handwriting Recognition Problem*

Many tasks are easy/feasible/doable for humans, but difficult/challenging/impossible for computer programs

1. Make progress towards deciphering this handwriting

2. Put words which you are unsure about in parenthesisInstructions to Crowd

Services Thrust

Examples (1/5)

Services Thrust

Examples (2/5)

Services Thrust

Examples (3/5)

Services Thrust

Examples (4/5)

It has been Existing

Humans were the first “computers,” computers, used for math computations

9 examples of crowdsourcing, before ‘crowdsourcing’ existed : http://bit.ly/mXFdRp

– Xerox Confidential –

Internet and Mobile Have Made it More Common and Promising

Services Thrust

Increasing Activities and Popularity

Page 10

“2M contributors who does more than 4PY of work on an average day!” --

Increasing Popularity as Depicted by Google Trends

Crowdsourcing on Google Scholar Over the Last Few Years

CEO

Services Thrust

Changing Demographics

What is the Problem Given a computational problem, design a solution

using human computers and automated computers

– Xerox Confidential –

Human in the loop (and not Guinea Pigs)

Main doer is Human (and not Machines as in Assembly Lines)

Humans are actively computing(not merely carrier of sensors)

The outcome is determined by an algorithm (and not the natural dynamics of the crowd)

Why is it Different?

Where is Research?

– Xerox Confidential –

Quality Estimation and Assurance (Redundancy and voting; Gold data; joint estimation of worker quality and task difficulty; Symbiosis with Machine Learning)

Complex Tasks (No discrete answer; Exploration and exploitation; crowd workflows;)

Task Design (Optimize cost, quality and time; infinite completion time; Real time)

Incentive and Motivation (Payment vs. non-payment; Optimal payment; Payment and quality; )

Market Design (Reputation Mechanism; Monitoring and feedback; Task Discovery;

Behavioral Aspects (Noisy behaviour; Non-reproducible;)

An Emerging Research Field

An Interdisciplinary Research Field

That’s Alright – but Xerox!!?

We have transformed…

… into the world’s leading enterprise for Business Process and Document Management

Revenue

Market Opportunity

2011Services-led~50% Services

Document Outsourcing Business Process Outsourcing Information Tech Outsourcing

~$23 billion

$500 billion +

Services Leadership In

$15.2 billion

2009Technology-led

~25% Services

$132 billion

Document Outsourcing

18

Xerox Revenue by Business Segment*

* http://www.fastcompany.com/magazine/161/ursula-burns-xerox

Page 20

Is Crowdsourcing a Viable Alternative to Outsourcing? Outsourcing is

Focus on the core business while partnering with 3rd party vendors to tackle the non-core operations Tasks requiring human intelligence and skills Data and process migration by smart use of technology

Heavily human intensive; typically with the help of computing technologies

Large distributed workforce enabled by technology executing tiny pieces of work requiring human intelligence

Page 21

Data Entry by Crowd We started by considering

a typical outsourced process (Data Entry)

Objective is to understand a process in detail and identify implications for crowdsourcing

Digitisation of insurance forms and medical records for US based insurance companies

Typing in, validation/ correction of information from scanned forms

Outsourced, distributed process

Features that make Form Digitization process amenable to crowd sourcing

Page 22

Relatively low skill data entry work, known as ‘key what you see’

Already an outsourced process requiring a low level of interactivity between sequential steps

Strong workflow tool to manage work, which flows through a series of system and human steps

Between sitesBetween sequential tasksBetween agents (given their known skill set)

Findings from Work-Practice Study (1/2)

Findings from Work-Practice Study (2/2)

Page 24

Workplace Ecology : Data security is physical, technical & social Crowdsourcing: lose physical and social enforcement, reduced

control of workforce. Need technical solutions. Skills and Knowledge 1)‘key what you see’ data entry actually

involves extensive rule set. 2) Form difficulty is situational. 3) Non-standard means non-standard.

Crowdsourcing: Situational-based incentives and supporting learning Being a Corporate Employee Pay alone not enough to achieve

SLA. Agents made accountable. Crowdsourcing: reduced accountability could increase rejections of

difficult work. Making the Workflow Work: Push model of work

Crowdsourcing: Pull model of work raises coordination and completion issues.

Collaborative Working: Work is not collaborative at workflow level; but it is at claim level (floorwalkers & colleagues).

Crowdsourcing: building collaboration in? Pull models of supervision?

Conclusion

Page 25

Crowdsourcing is an emerging Research areaIt requires expertise and research

competencies from a number disciplinesCrowdsourcing can be applied in various

domains to solve problems in a more effective manner

Finally, a large fraction of the crowd comes from India

Focused research and technologies will be highly relevant

Services Thrust

References• TurKit: Tools for Iterative Tasks on Mechanical Turk; Greg

Little, Lydia B. Chilton, Robert C. Miller, and Max Goldman• Matt Lease Tutorial• Soylent – A cr• Fold.it – S. Cooper et. al