Upload
paul-resnick
View
1.364
Download
0
Embed Size (px)
DESCRIPTION
Slides for ICWSM tutorial MP1: EVIDENCED-BASED SOCIAL DESIGN OF ONLINE COMMUNITIES: GETTING TO CRITICAL MASS AND ENCOURAGING CONTRIBUTIONS. http://icwsm.org/2012/program/tutorial/
Citation preview
Evidenced-Based Social Design of Online Communities
Robert E. KrautCarnegie Mellon University
Paul ResnickUniversity of Michigan
http://slidesha.re/ResnickICWSM12
Agenda
• Our approach & nature of design claims (10 minutes)• The challenge of contribution (90 min)
– Requests, goals & motivation (55 minutes)– Design challenge (15 minutes)– Break– Debrief (20 minutes)
• Starting a community (80 minutes)– Network externalities & getting to critical mass (45 minutes)– Design challenge (15 minutes)– Debrief (20 minutes)
Today’s goals
• Introduction to view of social design based on social science theory and empirical results
• Application to – Challenges of encouraging contribution to online groups– Challenges of starting a community from scratch
• Format: Lecture combined with break-out groups for design exercises
Instructors
• Paul Resnick– Professor, School of Information,
University of Michigan– Computer scientist by training
• Economics orientation– 2 years in industry at AT&T
• Robert Kraut– Herbert A. Simon Professor of HCI
at Carnegie Mellon– Social psychologist by training– 12 years in industry at Bell Labs
and Bellcore– Emphasis on social computing
Online communities face challenges typical of off-line groups
• Community start-up• Recruit, select and socialize members• Encourage commitment• Elicit contribution• Regulate behavior• Coordinate activity
But anonymity, weak ties, high turnover, & lack of institutionalization make challenges more daunting online
Evidence-based Social Design
• Mine the rich empirical and theoretical literatures in psychology and economics
• Develop design claims – Hypotheses about the effects of social design decisions
• Sometimes directly tested in the online context and sometimes only extensions of empirically tested theories developed in offline settings
Inspiration
Kurt Lewin
“There is nothing so practical as a good theory”
“If you want to understand something, try to change it”
The Roles of Theory and Evidence
• Identify Challenges• Generate Solution Ideas• Predict Consequences
Design Claims
• Our approach is to translate relevant social science theory and empirical research to design claims
• Alternative X helps/hinders achievement of goal Y under conditions Z
• E.g., – Coupling goals with specific deadlines leads to increases in
contributions as the deadlines approach– Group goals elicit contribution most among people who
identify with the group
Design Claims Differ from Pattern Languages
• Design pattern: a formal way of documenting a solution to a design problem in a particular field of expertise.
• May or may not document the reasons why a problem exists and why the solution is a good one
• Captures the common solutions, but not necessarily the effective ones
Design Levers
• Community structure• Content, tasks & activities• Selection, sorting & highlighting• External communication• Feedback & rewards• Roles, rules, policies and procedures• Access controls• Presentation and framing
Encouraging Contributions
Online Communities Face Challenges Typical of Off-line Groups
• Community start-up• Recruit, select and socialize members• Encourage commitment• Elicit contribution• Regulate behavior• Coordinate activity
But anonymity, weak ties, high turnover, & lack of institutionalization make challenges more daunting online
Reasons To Care
• Overall goal. Creating sufficient volume of contribution of the resources the group values to provide benefits to group members and others who rely upon the online community
• Different communities require different types of contribution– Social support forums: Conversational acts, empathy, offers of help– Recommender systems: Votes, opinions, comments– Facebook: Invites, accepts, wall posts, pictures – WoW guild: Time, particular skills– Threadless: T-shirt designs– OSS: Patches, code, translations, documentation– Wikipedia: New articles, facts, copy-editing, administration work,
cash (& recently, letters to congressmen)
Under Contribution Is Rampant
• Across many Internet domains, a small fraction of participants contribute the majority of material– Code in open source projects– Edits in Wikipedia– Illegal music in Gnutella– Answers in technical support groups
• Often leads to a power-law/Zipf curve distribution
• In many cases uneven contribution leads to an under supply of needed content. E.g., – Assessments and content in Wikipedia – Reviews of art movies in MovieLens
Wikipedia Stubs & Unassessed Articles• Many Wikipedia articles haven’t been assessed for
quality or importance• 58% of important ones are of low quality
Association for Psychological Science Wikipedia Initiative
APS/WI Reviewing Goal
• Subgoal: Get psychologists & grad students to review Wikipedia articles by adding comments to article talk pages describing problems with an article
• ~300 have signed up for the APSWPI, improving > 700 articles
• But fewer than 15 have reviewed• How you can apply any of the design claims
presented here to increase these reviews from APS members?
First attempt – Directed messages
0 re
view
s fro
m
107
reques
ts
Second attempt
• Simplifying the task– Direct link to where the action is needed
• Highlighting “social identity” in the invitation message• Personalizing the message
– Specifying users’ expertise • Phrasing the task as “rating” instead of “reviewing”
Current Interface
61 re
view
s fro
m
250
reques
ts
(21
revi
ewer
s)
61 re
view
s fro
m
250
reques
ts
Naïve Task Analysis of Online Contribution
To get people to contribute needed content :
1. They need to understand what is wanted
2. They have to be competent to provide it
3. They have to be motivated to provide it
Naïve Task Analysis of Online Contribution
To get people to contribute needed content :
1. They need to understand what is wanted
2. They have to be competent to provide it
3. They have to be motivated to provide it
Requests, matching, and goal setting
theories of motivation• Extrinsic• Intrinsic• Effects of
social situations
Requests, Matching, and Goal Setting
Requests Focus Attention on Needed Contributions
• Make the list of needed contributions easily visible to increase the likelihood that the community will provide them
Identify Who Should Make The Contribution• Request help in a chat room• “Can you tell me how to see someone’s profile”
– 400 Chat rooms– DV=Time to response
• People are slower to respond when others are present• Diffusion of responsibility is reduced when people are called by name
0
10
20
30
40
50
60
70
80
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Others present
Tim
e to
res
po
nd
(se
con
ds)
No name
0
10
20
30
40
50
60
70
80
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Others present
Tim
e to
res
po
nd
(se
con
ds)
No name Name
Markey(2000)
Email Request to Contribute to Movielens Quadruples Ratings
• In week after email reminder, contributes quadrupled, to ~ 20 ratings/person from ~5.4
• Is this sustainable?
Ask: Explicitly Asking for Needed Contributions Increases Likelihood of Getting Them
• News site with a “Leave a comment” form at the end of each article
• Fewer than 0.1% leave comments
• Experiment to estimate the value of explicit requests– No ask: “Leave a comment”
form at end of article– Immediate: Pop-up “Leave
a comment” when user opens article
– Delayed: Pop-up “Leave a comment” on closing article
Delayed
Immediate
No ask
0 20 40 60 80 100 120
Comments by Type of Request
Number of comments
(Wash & Lampe, 2012)
Ask Someone Who Is Willing & Able to Help: Intelligent Task Routing (Cosley, 2007)
SuggestBotSuggestions
Suggestions Quadruple Editing Rates
Goal Setting Theory• Goals motivate effort, perseverance & performance
– Trigger for both self-reward (e.g., self-efficacy) & external reward (e.g., money, reputation, promotion)
• Goals are more effective if– Specific & challenging rather than easy goals or vague ‘do your
best’– Immediate, with feedback– People commit selves to the goals – because of importance,
incentives, self-esteem, …– People envision the specific circumstance & method they will
use to achieve them
• Design claim: Providing members with specific and highly challenging goals, whether self-set or system-suggested, increases contribution.
Experiment Showing that Goals Work:
• Send email to ~900 MovieLens subscribers– Gave non-specific, do your best goal or specific, numerical
contribution goals – Assigned goal to individual subscribers or a nominal group
of 10 subscribers (the “Explorers”)
Goal Experiment Results
• Results– Specific, challenging goals increased contribution– Group assignment increased contributions
In-game Goals in WoW
• In WoW players receive extra powers each 10-levels implicit goals setting
• Ducheneaut, N., et al.(2007). The life and death of online gaming communities: A look at guilds in world of warcraft. in SIGCHI conference on Human factors in computing systems. San Jose, California, USA.
Weekly minutes playing World of Warcraft, by level
Featured Status in Wikipedia as a Challenge
Wikipedia edits before and after reaching featured status
WikiProjects Use Collaborations of the Week (COTW) as Time-Delimited Goals
39
A COTW announcement in a project page
An example template identifying an article as a COTW
Get designated to good status in a defined period (e.g., a week or a month)
Goal doubles contribution
40
Pre-Collaboration Collaboration Post-Collaboration
Edits per person on the collaboration articles
Non self-identified members
Self-identified group members
Goal has much larger effect on group members
41
Pre-Collaboration Collaboration Post-Collaboration
Edits per person on the collaboration articles
Non self-identified members
Self-identified group members
Goals and Identity
• Design claim: Goals have a more powerful effects when achieving them benefits a group the target identifies with
• Association for Psychological Science Wikipedia Initiative appeals to PhD psychologists with this technique
Design Claims Re: Goals
• Providing members with specific and highly challenging goals will increase their contributions.
• Goals have larger effects when people receive frequent feedback about their performance with respect to the goals.
• Externally imposed goals can be as effective as self-imposed ones, as long as the goals are important to community members
• Time-delimited challenges enhance the effects of goals
• Combining goals with appeals to social identity enhances their effects
Motivations for Contributing
What Motivates Contributors?• Intrinsic value of task (e.g., fun, curiosity,
challenge)• External personal value
– Reinforcement– Pay– Privilege
…• Social utility
– Reputation– Identification with the group– Reciprocity– Altruism
…
These are leverage points for interventions to increase motivation
Value-Expectancy Model Provides Leverage Points for Reducing Social Loafing
individual effort
individual performance
individual outcome
individual motivation
group performance
group outcome
individual utility
6 6
3, 4
4
5
3
Value-Expectancy Model Provides Leverage Points for Reducing Social Loafing
Liking for group membersIdentification with groupHistory of interaction with group
individual effort
individual performance
individual outcome
individual motivation
group performance
group outcome
individual utility
6 6
3, 4
4
5
3
Number of othersOwn competenceOwn unique skillsGroup’s incompetence
Frame request consistent with user’s values
Create incentives user valuesExtrinsicIntrinsic
Imagine you wanted labels for web images. How can you motivate people?
Extrinsic and Intrinsic Motivations
• Individual motivation influences behavior through external motivators (e.g., rewards, incentives, reputation) and intrinsic motivators (e.g., fun & curiosity)
Increase contributions by manipulating extrinsic incentives & intrinsic motivations
– Extrinsic motivators: Offer rewards as incentive (e.g., money, reputation, perks, grades)
• Larger rewards induce more contribution than smaller rewards. • Luxury goods create better incentives than money as rewards for
more difficult tasks. • Rewards of status, privileges, money, or prizes that are task-
contingent but not performance-contingent will lead to gaming by performing the tasks with low effort.
• People won't game the system for private verbal reward – Intrinsic motivators: Make the task fun or intrinsically interesting
Amazon’s Mechanical Turk
Typical Task: $.03
Financial Incentives on Threadless
Incentives vs. Reinforcements
• Incentives are promises given before the behavior to cause people to produce it
• Reinforcements are rewards given after a behavior that make it persist
Social Incentives on Amazon
Reinforcement: Barnstars
Design Claims Re: Incentives & Reinforcement
• Incentive Effects– People do more of the behaviors that they anticipate will be
rewarded. – Task non-contingent rewards will not create incentive to do more of
a task or exert more effort in doing it – Larger rewards induce more contribution than smaller rewards – Small gifts create more effective incentives than small payments
• Reinforcement effects– Rewards delivered in response to behaviors cause people to do
more of those behaviors – Rewards work better as reinforces if they are delivered right after
the desired behavior – Rewards generate more consistent performance over time if they
are unpredictable
Intrinsic Motivators
• Intrinsic motivation is the process of working to achieve the rewards that that come from carrying out an activity rather from as a result of the activity.
• Comes from the pleasure one gets from the task itself or from the sense of satisfaction in completing or working on a task.
Redesign the task to make it more fun or interesting
ESP Game To Label Images
• Example of playing the game• Taboo words
Truck
Red school bus
Red school bus
Red
How would you make a contribution task more fun?
What Makes a Contribution Fun?
Flow Criteria Principles of game design
Concentration Games should require concentration and the player should be able to concentrate on the game
Challenge Be sufficiently challenging and match the player’s skill level
Skills Support player skill development and mastery
Control Support players sense of control over their actions
Clear Goals Provide the player with clear goals at appropriate time
Feedback Provide appropriate feedback at appropriate times
Immersion Players should experience deep but effortless involvement in the game
Social Interaction Games should support and create opportunities for social interaction
Mapping flow to principles of game design (from Sweetser & Wyeth, 2005)
Lessons from game design
Support Opportunities for Social Interaction
Make tedious tasks fun via social interaction
Gamification• Applying game-design thinking to non-game
applications • Is the effect via fun (internal motivation) or
incentives (external motivations)?
Design Claims Re: Trade-offs Btw Intrinsic & Extrinsic Motivation
• Adding a reward to an already interesting task will cause people to be less interested in the task and to perform it less often.
• While tangible rewards reduce intrinsic motivations for interesting activities, verbal rewards enhance intrinsic motivation.
• Verbal rewards will not enhance intrinsic motivation and may undermine it while they are judged as controlling.
• Verbal rewards enhance intrinsic motivations most when they enhance the target’s perceptions of competence
Design Jam
• Groups of 4• Task
– Redesign of one thing of • request for review,• reviewing page on APS/WI site
– Sample Interface for reviewing page:http://hciresearch2.hcii.cs.cmu.edu/~rfarzan/wikipedia/tool/review/review.php?&cond=1 (also on next slide)
– Say which slides justify your proposal. – Mockup your proposal. – 15 minutes for Jam– Readout
• Show your mockup and narrate it: 1 minute!
Current Interface
Community Startup
A Startup Challenge
Externalities
• Alice's adoption or production decisions have direct or indirect effects on Bob's adoption or production decisions• Alice's decisions create costs and benefits external to her
• e.g., Size of telephone, email, and fax networks• e.g., Complementary products—hardware & software• e.g., Second hand smoke & other pollution
• Probability of a song appearing on Napster increased with the number of users, at a declining rate
Positive Externalities in Napster
• Measures of congestion in Napster increased with the number of users, at an increasing rate
Negative Externalities in Napster
Network Externalities in OLCs
• Negative externalities– Server congestion; competition for attention
• Positive externalities– People to interact with– Content they produce– Identity value
Implications of Positive Network Externalities
• Winner-take-all competition between networks• Need for critical mass
– minimum number of users that makes others want to join (or not quit)
Startup Phase Challenges
• Identifying a Niche• Defending the Niche• Getting to Critical Mass
Startup Phase Challenges
• Identifying a Niche• Defending the Niche• Getting to Critical Mass
Getting to Critical Mass
• Leveraging Early Members• Attracting Early Members
Getting to Critical Mass
• Leveraging Early Members• Attracting Early Members
Join Now or Wait?
• Early members especially important• Model provides insights into how to attract
• Join now if utility(join_now) > utility(wait)
utility (join now) = participation_benefitstage1
- startup_cost + success_probability * (participation_benefitstage2 + early_adopter_benefit)
utility(wait) = success_probability * (participation_benefitstage2 - startup_cost)
utility(join now) - utility(wait) = participation_benefitstage1 – startup_cost * (1 – success_probability) + early_adopter_benefit * success_probability
Implications: Where to Look for Solutions
• Increase immediate benefits
util(join now) - util(wait) =
participation_benefitstage1 – startup_cost * (1 – success_probability) + (early_adopter_benefit *
success_probability)
Implications: Where to Look for Solutions
• Increase immediate benefits• Reduce effort to join
util(join now) - util(wait) = participation_benefitstage1
– startup_cost * (1 – success_probability) + (early_adopter_benefit *
success_probability)
Implications: Where to Look for Solutions
• Increase immediate benefits• Reduce effort to join• Promise future benefits to
early adoptersutil(join now) - util(wait) =
participation_benefitstage1 – startup_cost * (1 – success_probability) + (early_adopter_benefit * success_probability)
Implications: Where to Look for Solutions
• Increase immediate benefits• Reduce effort to join• Promise future benefits to
early adopters• Set expectations: probability
of success
util(join now) - util(wait) = participation_benefitstage1 – startup_cost * (1 – success_probability) + (early_adopter_benefit *
success_probability)
Implications: Where to Look for Solutions
• Increase immediate benefits• Reduce effort to join• Promise future benefits to
early adopters• Set expectations: probability
of success• Conditional commitments
util(join now) - util(wait) = participation_benefitstage1
– startup_cost * (1 – success_probability) + (early_adopter_benefit *
success_probability)
Implications: Where to Look for Solutions
• Increase immediate benefits• Reduce effort to join• Promise future benefits to
early adopters• Set expectations: probability
of success• Conditional commitments• What's not worth focusing
on– Expectation setting: stage 2
value if successful
util(join now) - util(wait) = participation_benefitstage1 – startup_cost * (1 – success_probability) + (early_adopter_benefit *
success_probability)
Implications: Where to Look for Solutions
• Increase immediate benefits
• Reduce effort to join• Promise future benefits to
early adopters• Set expectations:
probability of success• Conditional commitments• What's not worth focusing
on– Expectation setting: stage 2
value if successful
util(join now) - util(wait) = participation_benefitstage1 – startup_cost * (1 – success_probability) + (early_adopter_benefit *
success_probability)
Increase Immediate Benefits
• DC25: Productivity, Entertainment or Commerce– E.g., Flickr offers picture storage and management,
services that are useful to the user even if nobody else is using Flickr.
• DC26: Professionaly generated content• DC27: Syndicated content• DC28: Professional staff contributions
Feb
15
Con
test
1
Con
test
2
Con
test
s 3,
4T
hrea
d 1
May
15
Thr
ead
2
Con
test
5
Fis
htan
ks
Con
test
6
Aug
15
0
5
10
15
20
25
30
35
40
45
0
50
100
150
200
250
300
Staff posts
Member posts
Active members
Key Dates
Mes
sag
es p
ost
ed
Cu
rren
t m
emb
er c
ou
nt
Resnick, Paul, Janney, Adrienne, Buis, Lorriane R, and Caroline R Richardson, “Adding an online community to an Internet-mediated walking program. Part 2: Strategies for encouraging community participation”. Journal of Medical Internet Research. 2010. 12(4):e72.
Increase Immediate Benefits
• DC25: Productivity, Entertainment or Commerce• DC26: Professionally generated content• DC27: Syndicated content• DC28: Professional Staff contributions• DC30: (as a last resort)• DC31: Bots
Promise future benefits to early adopters
• DC 32: Future discounts to the early adopters– E.g., lower rates for life
• DC 35: Limited resources that tempt users to join early– E.g., status & recognition with being an early adopter– E.g., users sign up first to claim their username
• DC 36: Privileges– E.g., administrator status
• Identity rewards– "Won't you be proud that you helped this get off the ground?"
Expectation Setting: Presenting Success at Different Stages
• DC43: Small and slow growing– Display new members and content
• DC44: Small and fast growing– Display percentage growth
• DC45: Big– Display absolute numbers
Conditional Commitments
Summary
• The Challenges– Identifying a Niche– Defending the Niche– Getting to Critical Mass
• Leveraging Early Members• Attracting Early Members
Attracting Early Members
• Increase immediate benefits
• Reduce effort to join• Promise future benefits
to early adopters• Set expectations:
probability of success• Conditional
commitments• What's not worth focusing
on– Expectation setting: stage 2
value if successful
util(join now) - util(wait) = participation_benefitstage1 – startup_cost * (1 – success_probability) + (early_adopter_benefit *
success_probability)
Challenge: Make Design Suggestions for Getting a Specific Community to Critical Mass• SuccessfulOnlineCom
munities.com• Or a community that
someone in your group is trying to launch
• Increase immediate benefits
• Promise future benefits to early adopters
• Set expectations: probability of success
• Conditional commitments
A Startup Challenge