Crowdsourcing

7/18/2019 Crowdsourcing

1/7

A Survey of Crowdsourcing as a means of Collaboration and the Implications

of Crowdsourcing for Interaction Design

Yue Pan & Eli Blevis

School of Informatics & ComputingIndiana University at Bloomington

{panyue,eblevis}@indiana.edu

ABSTRACT

Crowdsourcing has emerged as a new learning and

online collaborative paradigm, in which crowds of

people can collaborate and complete a specific task. In

this paper, we provide a survey of crowdsourcing as a

means of collaboration in three different contexts,

namely academic, enterprise, and social. The survey

also includes a short inventory of how different factors

contribute to the development of crowdsourcing. Finally,we develop and outline open questions, insights, as well

as a research agenda for enhancing crowdsourcing

development in the perspective of HCI.

KEYWORDS: Crowdsourcing, HCI, interactive design,

collaboration.

1. INTRODUCTION

Crowdsourcing is a new paradigm that relies on theintelligence of crowds of people to solve a specific

problem or complete a tasksometimes with a

monetary reward [10, 14, 16]. Jeff Howe invented the

term crowdsourcing in 2006. The most recent

definition of crowdsourcing he offers is:

Crowdsourcing is the act of taking a job traditionally

performed by a designated agent (usually an employee)

and outsourcing it to an undefined, generally large

group of people in the form of an open call. (Howe

2011)

Crowdsourcing has prompted models and applicationsfor problem solving and knowledge building in a wide

range of domains, from science to enterprise to social

networks. The development of crowdsourcing has been

leveraged in a way that largely satisfies social demands

for working with people in a larger geographic scope, as

well as task demands for accomplishing human

intelligence tasks (HIT). [12] Successful

crowdsourcing applications, such as Amazons

Mechanical Turk (AMT), Everything2, Wikipedia,

Innocentive etc., serve as innovative learning and

network collaboration paradigms.

The role of interaction design and research in facilitating

the development of online collaborative practices has

not been fully explored so far. Of interest is the question

of how can interaction design play the role of motivating

and sustaining participation and collaboration indifferent contexts (academic, enterprise, social)? This

paper provides a survey of crowdsourcing as a means of

collaboration, especially related to interaction design,and proposes a research agenda of interaction design

insights and methods of promoting applicability of

crowdsourcing in different domains.

This paper is divided into three parts. First, we review

the literature about crowdsourcing in different contexts,

namely academic, enterprise, and social. The second

part of the paper describes crowdsourcing in more

detailed dimensions, including role, behavior and media,

and with different variables, including leadership,ownership, coordination and conflict, integration and

attribution, mobile crowdsourcing, and ubiquitous

crowdsourcing. In the last part, based on the literature

review in the first and second sections, this paperconcludes by developing several open questions about

crowdsourcing and puts forward several design insights

and considerations, as well as a research agenda for

further investigation and research in crowdsourcing

development.

2. CROWDSOURCING & INTERACTION

DESIGN IN DIFFERENT CONTEXTS

Crowdsourcing has become a very essential tool and is

quite popular among various areas. In order to betterunderstand various motivations behind different groups

of contributors, this paper provides a general review ofcrowdsourcing apropos of the HCI literature in three

different contexts, namely academic, enterprise and

social.

978-1-61284-639-2/11/$26.00 2011 IEEE 397


2/7

2.1. Crowdsourcing for Enterprise

Stewart et al. in [14] state that design principles can be

used to influence two domains in crowdsourcing:enterprise (inside a firewall) and the public

domain.They raise the notion of extrinsic motivationmonetary rewards that can draw a crowd without any

number limitation. Stewart et al. employ a large

crowdsourcing community across a wide range of

influence at IBM with about 400,000 employees located

in 160 countries. The value of Stewart et al.s work lies

in three hypotheses and tests about motivating

crowdsourcing, as well as two design principlestask

decomposition and incentive model to motivate and

sustain participation in enterprise level.

In [10] Maja Vukovic talks about the usage of

crowdsourcing in the software development domain at

the enterprise level. She categorizes crowdsourcing

platforms along two dimensions: crowdsourcedfunction and crowdsourcing mode. She also proposes

22 required features for crowdsourcing and administers

an assessment of the four most famous crowdsourcing

systems illustrative of each feature, namely MTurk

which illustrates features, such as Comprehensive

model of the crowdsourcing request, and Automated

provisioning of crowdsourced request deliverable etc.;

Innocentive, which illustrates features, such as

Crowdsourcing platforms, and Collaboration

services etc.; iStockphoto, which illustrates features,such as Comprehensive model of the crowdsourcing

request, and Registration facility etc. ; Threadless,

which illustrates features, such as Registrationfacility, and Collaboration services etc. Finally, she

proposes 7 research challenges which can be a good

guidance for system developers and interaction

designers to improve crowdsourcing systems. For

instance, one of the research challenges is that

Designing a mechanism for virtual team formation,

incorporating not only skill-set, but also discovered

social networks.[10]

2.2. Crowdsourcing for Academic

Regarding the application and impact of crowdsourcing

in academic and science field, Buecheler et al. in their

2010 paper [15] describe how crowdsourcing canbenefit the Research Value Chain and they provide a

research agenda about the applicability of

crowdsourcing in fundamental science based on

Artificial Intelligent research. The research brings

insights and methods for facilitating the processes of

crowdsourcing in scientific and academic fields. It also

supports the idea that non-scientists can supportscientific projects with high quality contributions. The

same idea is also held by [3], which is targeted

specifically at the academic purpose. He holds that data

collected by volunteers (non-scientists) can be of great

educational value in a larger geographic scale and over a

longer period of time. These two papers hold the belief

that the collaboration between professionals (scientists)

and non-professionals (general public) deserves moreattention from academic and scientific field.

In [13], Nov et al. introduce a term to represent internet-

based crowdsourcing in the field of science, namely

SciSource and they explore the motivations behind

what drives people to make contributions to share

resources voluntarily. Based on findings from two

preliminary studies (SETI@home and CWOP), they list

several prominent factors that serve as motivations for

continuous crowdsourcing participation in academic and

scientific related fields, including Enjoyment,

Reputation, Values, Enhancement, andAffiliation to a team and tenure. Understanding

different aspects of motivations that contribute toscientific crowdsourcing can be a first-class

consideration for designing scientific projects.

2.3. Crowdsourcing for Social Network

Chung et al. [6] introduce the Stress OutSourced

(SOS) system which is designed to enable people to

relieve stress by sending each other therapeutic

massages as an instance of the concept of

crowdsourcing. They state: The SOS system consists of

members wearing customizable signal modules for

sending and responding to an anonymous SOS call, and

wearable massage actuation modules to receive themassage strokes sent by others. Chung et al. discuss

three innovative design concepts of the SOS system.

Based on their prototype of the massage module, they

describe a design framework which can be useful to help

better understand and develop scalable social media and

haptic networks with crowdsourcing. The prototype of

the SOS system is achieved by wearable garments (such

as sweater, shirt etc.) that can send and receive signals

through an embedded or attachable signal module and

a massage module.

Huberman et al. [4] describe social networking as a

construction of the interaction that people have with

their friends, families and acquaintances. From datagathered from Twitter, they found that the degree of

peoples contributions to crowdsourcing depends on the

attention people receive from others. Based on this

notion, they describe two types of networks, a very

dense one made up of followers and followees and a

sparser and simpler network of actual friends. This

distinction can be part of the design and research agenda

for design online collaboration network based on social

aspects that we describe in the last section of this paper.

398


3/7

3. CROWDSOURCING IN DIFFERENT

DIMENSIONS

In this part, we examine crowdsourcing in more detailed

dimensions but from a different angle. We list a set of

factors that contribute to motivations and participationof volunteers at different levels. The set of factors can

be classified into following categories: (i) role-oriented

crowdsourcing with leadership and ownership, (ii)

behavior-oriented crowdsourcing with attribution,

coordination and conflict, and (iii) media-oriented

crowdsourcing with mobile computing and ubiquitous

computing.

3.1. Role-oriented CrowdsourcingLeadership & Crowdsourcing

Crowdsourcing requires communication and

cooperation from people who are geographically-distributed [7] and who have diverse backgrounds. At

the same time, Kittur and Kraut [2] argue that it can be

very ineffective for peer-to-peer communication if

there are too many people involved in a free and open

environment. They also conducted research whichsuggests that it is highly beneficial to have people who

can set direction and provide a structure to which others

can contribute. Under this circumstance, high-quality

leadership is quite essential and necessary to guarantee

successful crowdsourcing participation and online

collaboration at different levels. Based on empirical

qualitative studies of three online communities, Luther

and Bruckman introduce two major challenges that

online collaboration leaders face in the entertainmentdomain. [7] They identify four themes, namely

originality, completion, subjectivity and

ownership, which can be used to guide us onimproving design to deal with these challenges that are

faced by online collaboration leaders.

Based on [8], Luther also explores the role of leadership

in online collaboration in the context of movie-making.

In [8], he offers concrete design and research principles

and insights about the potential and limitations of online

creative collaboration.

3.2. Ownership & Crowdsourcing

Regarding existing features of Wikipedia, Thom-

Santelli et al. [5] raises a notion of territoriality, namely

the expression of ownership towards an object.This

can be valuable regarding a successful hierarchical style

of collaboration, and present design implications for

supporting expressive territory behaviors such asmarking, control, and defense. These design

insights and implicationscan be used for later research

on how to maintain online collaborative system in a

healthy and productive way, as well as providing

support and encouragement to further user experience

design of collaborative systems.

3.3. Behavior-oriented Crowdsourcing

Integration & Attribution withinCrowdsourcing

In [9], Luther et al. hold based on their research that

recognizing integration and attribution of online

reputations in a proper way can contribute to motivating

creative online collaborations, as an instance of

collaborations that are possibly in the sense of

crowdsourcing. Based on a qualitative study (involving

17 experienced animators) on Newgroundsthe largest

Flash animation portal on the Web, this paper also

provides novel insights on promoting online creative

collaboration by identifying frustrations with existing

practices and systems, as well as proposing design

considerations for alleviating them. Especially, threedesign insights, namely cr-editing, collaborative

authoring, and community value can be crucial

guidelines for better understanding and designing for

online creative practices that prompt collaboration.

3.4. Coordination & Conflict within

Crowdsourcing

Kittur and Kraut in their [2] emphasize the importance

of coordination by distinguishing (i) explicit

coordination, which is based on direct

communication and verbal planning and (ii) implicit

coordination,which is workshop structure, unspokenexpectations and shared mental models of the task to be

accomplished. They adopt different research

methodologies (such as cross-sectional analysis and

longitudinal approach) to examine different factors that

contribute to the quality of crowdsourcing, including the

number of contributors and the differing

coordination techniques.Their findings can be helpful

in providing a better understanding on what kind of role

that coordination plays in improving crowdsourcing

quality.

Recognizing a phenomenon with Wikipedia that the

rate of creation of new articles and content isdecreasing, while levels of maintenance and indirect

work are increasing, Kittur et al. [1] build a

characterization model and a user conflict model to

investigate motivations and conflict at the local article

level. Their models have been testified by adopting

several scientific methods, such as human-labeled

controversy tags and history flow visualization etc.

as well as by conducting an online survey of 28 articles.

The scientific methods, conflict models, visualization

399


4/7

tools and techniques that they used to investigate and

predict conflict and coordination costs are very intuitive

and can be applied to analyze and generalize

crowdsourcing systems in a larger scale.

3.5. Media-oriented Crowdsourcing

Mobile Crowdsourcing

Mobile computing has potential implications forcrowdsourcing. Mobile computing can facilitate the

development and expansion of crowdsourcing by

enabling mobile users to work on crowdsourcing tasks

with their mobile devices. In [16], Yan and his

colleagues investigate existing crowdsourcing services

and develop a mobile application mCrowd based on

sensor-rich mobile devices which allow users to post

sensing tasks, such as posting text or image tagging

queries. The idea and design of the mCrowd

application can serve as a good demonstration of the

ways in which mobile technology can facilitate the

development of crowdsourcing in a sustainable way.

Considering crowdsourcing applications using mobiletechnology, another good example is Txteagle by

Nathan Eagle [12]. Txteagle is a very typical mobile

application using mobile computing technology in the

crowdsourcing domain that enables people from

developing countries to conduct real-time, peer-to-

peer financial transactions. In their paper, they

describe four interaction scenarios that demonstrate how

Txteagle facilitates financial earnings in rural settings.

For example, in one user scenario a home-maker in

Butare Rwanda was unable to afford a phone herself.

But now, she gets paid by completing txteagletransaction tasks for a village phone operator, and

eventually she will be able to buy her own cell phone.

Txteagle can be a good design insight and implication

on applying mobile technology and crowdsourcing in

providing financial sources to people at remote and

impoverished areas of the world.

3.6. Ubiquitous Crowdsourcing

Vukovic et al. in their Ubicomp 10 workshop paper[11]describe several important aspects of ubiquitous

crowdsourcing, including the interaction model,

crowdsourcing applications enabled by ubiquitous

computing systems, andprotocol between ubiquitous

computing and crowds. They raise three critical

challenges about crowdsourcing and ubiquitous

computing, from the perspective of (i) creating

interaction model and protocol, (ii) facing challenges of

ensuring crowdsourcing quality, and (iii) developing

crowdsourcing application within ubiquitous computing

domain. All of these three challenges serve as novel

insights for HCI and interaction design and they also

contribute to part of our research agenda described in

the last section of the paper.

4. RESEARCH AGENDA

In this section of the paper, we propose several open

questions that interaction designers may face apropos ofHCI and interaction design in crowdsourcing, and

develop insights that own directly to our secondary

research in the previous sections. More specifically, the

design insights include a summary of crowdsourcing

values in different context (academic, enterprise, and

social), the assessment of different variables on

crowdsourcing, as well as the motivation model, all ofwhich can lead to concepts and principles for interaction

design.

4.1. Design Problems

We provide the following inventory of open research

questions based on our literature review of

crowdsourcing within HCI and interaction domain.

1.

What are the factors spurring creative online

collaboration in different contexts (academic,

enterprise, social) as well as targeted at different

dimensions (role, behavior, and media)?

2.

What are the challenges that interaction design is

facing according to improving and maintaining

development of crowdsourcing?

3.

What are human-computer interaction designer and

researcher roles in motivating, enhancing and

sustaining motivations and participation in

crowdsourcing?

4.

Are there ways in which HCI design can improve

creative collaborative crowdsourcing?

5.

What values are advanced by crowdsourcing systems

in promoting social values, such as diversity, social

awareness, respect, sustainability etc.?

4.2. Insight Based on Surveys

This part of the paper describes crowdsourcing values in

different contexts, roles of different variables in

crowdsourcing, as well as the motivation model that arederived from the survey of crowdsourcing as a means of

collaboration in the previous part.

4.2.1. Crowdsourcing Values in Different Contexts

Generally speaking, the role of crowdsourcing is tocollect intelligence from the crowd to complete a

specific task. With the purpose of better understanding

the contribution of crowdsourcing, we derive

400


5/7

crowdsourcing values separately in the three contexts of

academic, enterprise, and social that we identified in the

previous part, which is displayed in Table 1.

4.2.2. Crowdsourcing in Different DimensionsIn the previous section, we identify a list of factors that

contribute to crowdsourcing from three differentdimensions, including role, behavior and media. In

Table 2, we describe the role that each factor may play

in crowdsourcing. Each variable according to different

dimension can contribute to crowdsourcing from

different perspectives. More specifically, some variables

may act as intrinsic factors to motivate and direct

collaborators, such as to motivate ones personal values,

enhance ones reputations, etc.; some variables may

influence the quality and development of crowdsourcing

in a more direct way, such as affecting the cost and

development process of crowdsourcing; some variables

may contribute from the perspective of technology and

communication media.

It is useful to be familiar with each factor and its

respective roles as part of any design process involving

crowdsourcing. Keeping these factors in mind can shape

the design of crowdsourcing applications with more

specific targets and objectives.

4.2.3. Motivation Model

Considering the values people have that prompt them to

participate in crowdsourcing, we created the IncentiveModel shown as Table 3 that presents an inventory of

motivating factors from different aspects.

4.3. Research Agenda Based on Insights

Crowdsourcing as a means of collaboration is a very

promising model of interactivity that can be applied in

different disciplines. At the same time, it also prompts

wide-ranging research challenges along the contexts of

academic, enterprise, and social. What follows is the setof challenges that interaction design and HCI may face

when designing and developing crowdsourcing

applications:

1.

Design interfaces that can attract and sustain a large

Table 1.Crowdsourcing Value in Different Contexts

Table 2:The Roles of Different Variables on Crowdsourcing

Table 2.The Roles of Different Variables on Crowdsourcing

401


6/7

number of people to join in crowdsourcing.

2.

Design mechanisms that support efficient

crowdsourcing formation as well as avoid conflict

among collaborators.

3.

Design experience models that evaluate the quality

of experience in crowdsourcing process. [10]

4.

Design a mechanism for balancing communication

and independency within crowdsourcing. [15]

5.

Design in a way to increase the robustness of

crowdsourcing in dealing with collaborations between

unknown people within unknown situations and/orenvironments. [15]

5. CONCLUSION

We have surveyed several key articles about

crowdsourcing as a means of collaboration and created

three tables that serve as a classification scheme for thecontent of this survey.

No matter if it is in the context of academic, enterprise,

or social, the benefits of crowdsourcing merit research.

The survey of crowdsourcing in this paper can be a first

step in exploring crowdsourcing values and listingvarious factors that determine the development and

expansion of crowdsourcing. How to address the

research challenges enumerated above for developing

further crowdsourcing services from HCI and design

perspectives remains as future related work.

REFERENCE

[1] Aniket Kittur, Bongwon Suh, Bryan A. Pendleton, andEd H. Chi. 2007. He says, she says: conflict and

coordination in Wikipedia, In Proceedings of the

SIGCHI conference on Human factors in computing

systems (CHI '07). ACM, New York, NY, USA, 453-462.

[2] Aniket Kittur and Robert E. Kraut. 2008. Harnessing

the wisdom of crowds in wikipedia: quality through

coordination, In Proceedings of the 2008 ACMconference on Computer supported cooperative work

(CSCW '08). ACM, New York, NY, USA, 37-46.

[3]

Cohn, J.P. (2008) Citizen science: Can volunteers doreal research? Bioscience 58(3): 192197.

[4] Huberman, B. A., Romero, D. M., and Wu, F.. Social

networks that matter: Twitter under the microscope,arXiv: 0812.1045v1, Dec 2008.

[5] Jennifer Thom-Santelli, Dan R. Cosley, and Geri Gay.

2009. What's mine is mine: territoriality incollaborative authoring, In Proceedings of the 27th

international conference on Human factors in computing

systems (CHI '09). ACM, New York, NY, USA, 1481-

1484.

[6] Keywon Chung, Carnaven Chiu, Xiao Xiao, and Pei-Yu

(Peggy) Chi. 2009. Stress outsourced: a haptic socialnetwork via crowdsourcing, In Proceedings of the 27thinternational conference extended abstracts on Human

factors in computing systems (CHI '09). ACM, New

York, NY, USA,

[7] Kurt Luther and Amy Bruckman. 2008. Leadership in

online creative collaboration, In Proceedings of the

2008 ACM conference on Computer supported

Table 3.Motivation Model

402


7/7

cooperative work (CSCW '08). ACM, New York, NY,USA, 343-352.

[8] Kurt Luther. 2010. Supporting and transforming

leadership in online creative collaboration, InProceedings of the 28th of the international conference

extended abstracts on Human factors in computing

systems (CHI EA '10). ACM, New York, NY, USA,2931-2934.

[9] Kurt Luther, Nicholas Diakopoulos, and Amy Bruckman.

2010. Edits & credits: exploring integration and

attribution in online creative collaboration, InProceedings of the 28th of the international conference

extended abstracts on Human factors in computing

systems (CHI EA '10). ACM, New York, NY, USA,

2823-2832.

[10]

Maja Vukovic. 2009. Crowdsourcing for Enterprises,

in Proceedings of the 2009 Congress on Services - I

(SERVICES '09). IEEE Computer Society, Washington,DC, USA, 686-692.

[11]Maja Vukovic, Soundar Kumara, and Ohad Greenshpan.

2010. Ubiquitous crowdsourcing, in Proceedings ofthe 12th ACM international conference adjunct papers

on Ubiquitous computing (Ubicomp '10). ACM, New

York, NY, USA, 523-526.

[12]

Nathan Eagle. 2009. txteagle: Mobile Crowdsourcing,

In Proceedings of the 3rd International Conference on

Internationalization, Design and Global Development:

Held as Part of HCI International 2009 (IDGD '09),Nuray Aykin (Ed.). Springer-Verlag, Berlin, Heidelberg,

447-456.

[13]Oded Nov, Ofer Arazy, & David Anderson. (2010).

Crowdsourcing for science: understanding andenhancing SciSourcing contribution, ACM CSCW

2010 Workshop on the Changing Dynamics of Scientific

Collaborations.

[14]Osamuyimen Stewart, Juan M. Huerta, and Melissa

Sader. 2009. Designing crowdsourcing community for

the enterprise, In Proceedings of the ACM SIGKDD

Workshop on Human Computation (HCOMP '09), PaulBennett, Raman Chandrasekar, Max Chickering, Panos

Ipeirotis, Edith Law, Anton Mityagin, Foster Provost,

and Luis von Ahn (Eds.). ACM, New York, NY, USA,

50-53.

[15]Thierry Bcheler, Rudolf M. Fchslin, Rolf Pfeifer, Jan

Henrik Sieg. (2010). In Proceedings of Artificial LifeXII -- Proceedings of the Twelfth InternationalConference on the Synthesis and Simulation of Living

Systems. Pages 679686. MIT Press.

[16]Tingxin Yan, Matt Marzilli, Ryan Holmes, DeepakGanesan, and Mark Corner. 2009. mCrowd: a platform

for mobile crowdsourcing, In Proceedings of the 7th

ACM Conference on Embedded Networked Sensor

Systems (SenSys '09). ACM, New York, NY, USA,347-348.

403

Documents

Crowdsourcing