The great promise of Wiki Science - MemberClicks · SuggestBot: using intelligent task routing to...

This prsentation is published under the Creative Commons Attribution Share-alike 4.0 International (CC-BYSA 4.0) license. Anyone is free todistribute and re-use this work on the conditions that the original authors are appropriately credited and that any derivative work is madeavailable under the same, similar, or a compatible license. © 2019 Copyright held by the owner/author(s)

The great promise of Wiki

ScienceOpen Problems of Strategic Importance

by Aaron Halfaker

Sociological history of Wikipedia

Open data & scholarship

Strategic new research directions

Part 1: Sociological history

of Wikipedia

What is Wikipedia?

The world’s largest encyclopedia

~5 million articles

in English

A wiki

● Anyone can edit

● Shared authorship

Flipped publication model

● publish first

● review later (maybe)

An online community

● ~100k active volunteer editorsWikiProject Video

WikiProject Medicine

A system

Available

Attention

A system

Available

Attention

Output

A system

Available

Attention

Output

Socio-technical

Social Technical

https://commons.wikimedia.org/wiki/File:Commodore_

PET_4016,_Google_NY_office_computer_museum.jp

https://commons.wikimedia.org/wiki/File:Kadokawa_office_(2).jpg

Social Technical

https://commons.wikimedia.org/wiki/File:Marcus_Bai

ns_Line.png https://commons.wikimedia.org/wiki/File:Balsa-gpg.png

Socio-technical

Technologist

Techno-biologist?

BacteriumParamecium

ribosomes

vacuole

BacteriumParamecium

ribosomes

vacuole

contractile vacuole

endoplasmic

reticulumoral groove

macronucleus

micronucleus

Bacterium Paramecium

Dunbar’s number: ~150

Via commons. CC-BY-SA 2.0

Bacterium ParameciumBare-amecium Organelles

Robots

& tools

Policies &

Guidelines

socio - technical

socio-technical

System with specialized

sub-systems

System with specialized

sub-systems

Work allocation● Identify, prioritize and assign tasks

Work allocation

● Largely for free due to Linus’ Law

Linus’s Law?

Raymond

Linus’s Law?

“given enough eyeballs, all bugs are shallow”

Linus’s Law?

visibility is critical to open collaboration

A corollary for Wikipedia

Given that enough people see an incomplete

article, all potential contributions to that article

will be easy for someone.

Works in theory -- Stvilia, B., et al. (2008). Information quality work organization in

Wikipedia. JASIST, 59(6), 983-1001.

Part of becoming a Wikipedian -- Bryant, S. L., Forte, A., & Bruckman, A. (2005,

November). Becoming Wikipedian. GROUP (pp. 1-10). ACM.

We can support visibility with technology -- Cosley, D., et al. (2007, January).

SuggestBot: using intelligent task routing to help people find work in wikipedia. IUI

(pp. 32-41). ACM.

Bad things happen when we take it away -- Schneider, J., et al. (2014, August).

Accept, decline, postpone. OpenSym(p. 26). ACM.

Regulation of behavior● Norm formation, propagation and enforcement

Prescriptive

Idea for a Norm

Descriptive

Observation of

consistent behavior

Prescriptive

Idea for a Norm

Descriptive

Observation of

consistent behavior

Prescriptive

Idea for a Norm

Descriptive

Observation of

consistent behavior

Requests for

CommentsESSAY

rabblerabble

rabble

Prescriptive

Idea for a Norm

Descriptive

Observation of

consistent behavior

Requests for

Comments

GUIDELINE

POLICY

rabblerabble

rabble

GUIDELINE

POLICY

Informal FormalizedMorgan, J. T., & Zachry, M. (2010,

November). Negotiating with angry

mastodons: the wikipedia policy

environment as genre ecology. In

Proceedings of the 16th ACM

international conference on Supporting

group work (pp. 165-168). ACM.

Halfaker, A., Geiger, R. S., Morgan, J.

T., & Riedl, J. (2012). The rise and

decline of an open collaboration system:

How Wikipedia’s reaction to popularity is

causing its decline. American Behavioral

Scientist, 0002764212469365.

Informal Formalized

Growth of regulations

Halfaker, A., Geiger, R. S., Morgan, J. T., & Riedl, J. (2012). The rise and decline of an open collaboration system: How Wikipedia’s reaction

to popularity is causing its decline. American Behavioral Scientist, 0002764212469365.

Citation distribution

Beschastnikh, I., Kriplean, T., & McDonald, D. W. (2008, March). Wikipedian Self-Governance in Action: Motivating the Policy Lens. In

ICWSM.

Citation distribution

Beschastnikh, I., Kriplean, T., & McDonald, D. W. (2008, March). Wikipedian Self-Governance in Action: Motivating the Policy Lens. In

ICWSM.

“Wikipedia’s governance structure is

inclusionary”

Quality control● Identify and remove damage

Quality control

Fully automated

Machine learning

● Fast (~ 5 seconds)[1]

● No human effort

● Only obvious vandalism

Semi automated

Human computation

● Still pretty fast (~ 30

seconds)[1]

● Minimizes human effort

● Humans catch most

vandalism at a glance

1. R. Stuart Geiger & Aaron Halfaker. When the Levee Breaks (2013). WikiSym.

Quality control

Fully automated

Semi automated

Banning (Admins)

Quality control

Fully automated

Semi automated

Banning (Admins)

Innate- Fast

- General

- Local

Adaptive- Slow

- Specific

- Global

Halfaker, A. & Riedl, J. (2012) Bots and Cyborgs,

IEEE Computing 45(3) (p. 79-82)

Geiger, R. S., & Ribes, D. (2010, February). The

work of sustaining order in wikipedia: the banning

of a vandal. CSCW (pp. 117-126). ACM.

Community management ● Newcomer socialization, dispute mediation & training

https://commons.wikimedia.org/wiki/File:Flickr_-_Official_U.S._Navy_Imagery_-

_Sailor%27s_daughter_operates_a_fire_hose_with_crew_member_assistance..jpg

6,000 newcomers per day

vandals

good-faith

vandals

Regulation of behavior● Form, propagate and enforce norms

Community management ● Socialize & train newcomers; mediate disputes

Reflection (Adaptation) ● Where are we going? Where do we want to go? How do we want to get there?

Reflection (Adaptation)

Where are we going?

Where do we want to go?

How do we get there?

Reflection

Adaptation

https://commons.wikimedia.org/wiki/File:Morphine_to_Apomorphine.png CC-SY-SA 4.0

https://commons.wikimedia.org/wiki/File:Gear_reducer.gif CC-BY-SA 3.0

User:ClueBot NG

Available

Attention

Part 2: Open data and

scholarship

Planet

Earth(7.6 billion)

Planet

Earth(7.6 billion)

Monthly

Wikipedia

Readers

(1 billion)

Monthly Wikipedia Editors

(113,304)

Planet

Earth(7.6 billion)

1:10000

Monthly

Wikipedia

Readers

(1 billion)

(113,304)

Planet

Earth(7.6 billion)Monthly

Wikipedia

Readers

(1 billion)

1:10000

(113,304)

Planet

Earth(7.6 billion)Monthly

Wikipedia

Readers

(1 billion)

1:10000

[edit]

(100,000)

1:10,000

Wikimedia Foundation Staff

(~225)

1:1000

Monthly Wikipedia

Readers (1B)

Wikimedia

Research

Wikimedia

Research

● Google research: 1953 (Alexa #1)

● Facebook research: 580 (Alexa #3)

● Wikimedia research: 6 (Alexa #5)

● Yahoo Research: 84 (Alexa #7)

https://www.alexa.com/topsites (Gathered: 2019-03-08)

● Google research: 1953 (Alexa #1)

● Facebook research: 580 (Alexa #3)

● Wikimedia research: 6 (Alexa #5)

● Yahoo Research: 84 (Alexa #7) https://www.alexa.com/topsites (Gathered: 2019-03-08)

… but Wikipedia is a volunteer organization.

Wiki Researchers

… but Wikipedia is a volunteer organization.

Wikimedia

… in the end, we might rival Google Research.

Why is Wikipedia so interesting?

1. Huge. Alexa #5. Read by 1/10th of the

planet. Trusted source of information.

2. Weird. Flips publication model.

Decentralized governance. Shouldn’t have

worked at all.

2. Weird. Flips publication model.

Decentralized governance. Shouldn’t have

worked at all.

3. Open. Datasets, proposals, and initiatives

are made freely available.

Story time

Why isn’t anyone looking at Google Plus?

Google

Research

Wikipedia Research Wikipedia Research

2008 2010 2012 2014 2016 2018

So much great research

out there about

Wikipedia

Manager

out there about

Wikipedia

Who cares about that?

Why don’t people study

Google Plus?

Manager

out there about

Wikipedia

Google Plus?

Manager

out there about

Wikipedia

Google Plus?

datasets

openness

benefits humanity

Manager

out there about

Wikipedia

Google Plus?

datasets

openness

benefits humanity

walled garden

publishing

embargo

benefits google

Manager

Halfaker, A., Geiger, R. S., Morgan, J. T., & Riedl, J. (2013). The rise

and decline of an open collaboration system: How Wikipedia’s reaction

to popularity is causing its decline. American Behavioral Scientist,

57(5), 664-688.

Geiger, R. S., & Halfaker, A. (2013, August). When the levee

breaks: without bots, what happens to Wikipedia's quality control

processes? OpenSym (p. 6). ACM.

Black Holes - Monsters in Space (PD)

Google,

Facebook,

Twitter,

Google,

Facebook,

Twitter,

Profits

Breaking News:

Google’s Social

Site is Dying!

Lost profit

Profit

Donation Totals by Fiscal Year and Average Gift Size

Part 3: Strategic new

research directions

3 key areas

● Gaps in representation and content

● Growth in newcomer retention

● 3rd party re-use

NCI_swiss_cheese (PD)

Amazon_Echo_Plus_02 by Asivechowdhury (CC-BY-SA 4.0)[[en:Harrods]] (CC-BY-SA 4.0)

Gaps in representation

● Mostly Male

● Mostly from North America and Western Europe

● Mostly citing English Language sources

Sen, S. W., Ford, H., Musicant, D. R., Graham, M., Keyes, O. S., & Hecht, B. (2015, April). Barriers to the localness of volunteered geographic

information. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 197-206). ACM. pdf

Sen, S. et al.. (2015, April). Barriers to the localness of volunteered geographic information. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (pp. 197-206). ACM.

Halfaker, A. (2017, August). Interpolating quality dynamics in wikipedia and

demonstrating the keilana effect. In Proceedings of the 13th International

Symposium on Open Collaboration (p. 19). ACM.

What other content areas are under-developed?

Newcomer retention

TeBlunthuis, N., Shaw, A., & Hill, B. M. (2018, April). Revisiting The Rise and Decline in a Population of Peer Production Projects. In

Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (p. 355). ACM.

Newcomer

retention

Quality

control

Time →

● :m:Research:The_Rise_and_Decline● Halfaker, A., Geiger, R. S., Morgan, J. T., & Riedl, J. (2012). The rise and decline of an open collaboration

system: How Wikipedia’s reaction to popularity is causing its decline. American Behavioral Scientist,

0002764212469365.

User:ClueBot NG

Newcomer retentionQuality control

How do we have both?

Newcomer

retention

Quality

control

Time →

3rd party re-use

How long do goats live?

According to

Wikipedia, goats

can live 15 - 18

● Amazing! Wikipedia is open licensed for

exactly this reason.

● Amazing! Wikipedia is open licensed for

exactly this reason.

● But! 2 problems:○ Ecosystem

○ Tracking usage

Eco system

Contributors

Content

Readers

Search results

Eco system

Contributors

Content

Readers

Search results

Eco system

Contributors

Content

Readers

Search results

Eco system

Contributors

Content

Readers

Search results

Eco system

Contributors

Content

Readers

Search results

Eco system

Contributors

Content

Readers

Search results

Eco system

Contributors

Content

Readers

Search results

Eco system

Contributors

Content

Readers

Search results

With the Knowledge panel

● Users are 50% less likely to click a

Wikipedia link

● Users are 5X as likely to assume the

information *comes from* Google.

McMahon, C., Johnson, I., & Hecht, B. (2017, May). The substantial interdependence of wikipedia and google: A case study on the relationship

between peer production communities and information technologies. In Eleventh International AAAI Conference on Web and Social Media.

Tracking usage

Resources

https://upload.wikimedia.org/wikipedia/commons/d/d

0/Wikimedia_Public_Research_Resources.pdf

https://quarry.wmflabs.org

https://paws.wmflabs.orghttps://ores.wikimedia.org/v3/scores/enwiki/5187334

19/articlequality

https://github.com/wikimedia-research/iwsc-2017-

workshop

The great promise of Wiki Science - MemberClicks · SuggestBot: using intelligent task routing to...

Documents

tion ( IUI )? · An Intra Uterine Insemination (IUI) is a type of fertility treatment. In IUI, hormonal medications are given to the woman prior to the procedure, to stimulate , are

IUI Yearbook 1995-96

Controversies in Iui

Ovarian Stimulation in IUI- Overview

IUI Yearbook 1997-98

How to make IUI cost effective

Context Relevance Assessment for Recommender Systemsricci/papers/iui-2011-baltrunas.pdf · Context-aware, recommender systems, user preferences. ACM Classiﬁcation Keywords H.3.3

CALL FOR PAPERSiui.acm.org/2017/files/IUI2017_CfP06102016.pdf · ACM IUI is where the Human-Computer Interaction(HCI) community meets the Artificial Intelligence (AI), with contributions

10 secrets of success of IUI

Response modeling-iui-2013-talk

Iui treatment

tiifrhrCGGL/ BackaiT^cwL iUi-OiacijUrivd5

IUI Yearbook 2000-01

IUI Review

Setting Up of IUI Lab

Iui 1.0 clean

scope - MemberClicks

IVF Centre in Delhi-IUI Treatment

IVF,GIFT,ICSI,IUI BABY

Monitoring in iui