Internet The open - Wikimedia€¦ · Internet Using AI to keep Wikipedia open Aaron Halfaker -- Wikimania 2018 ... (MIT license) #1 reason listed: “This would reduce the workload

New content

The Internet

Using AI to keep Wikipedia open

Aaron Halfaker -- Wikimania 2018

Wikipedia-logo-v2.svg (CC-BY-SA 3.0)

https://commons.wikimedia.org/wiki/File:Wikipedia-logo-v2.svg

https://meta.wikimedia.org/wiki/User:Halfak_(WMF) (CC-BY-SA 3.0)

Halfaker,_Aaron_Sept_2013.jpg (CC-BY-SA 3.0)

https://meta.wikimedia.org/wiki/User:Halfak_(WMF)

https://commons.wikimedia.org/wiki/File:Halfaker,_Aaron_Sept_2013.jpg

https://meta.wikimedia.org/wiki/User:Halfak_(WMF) (CC-BY-SA 3.0)


https://meta.wikimedia.org/wiki/User:Halfak_(WMF)






1. Openness vs. quality control

AHHHHHH!Turn it OFF!

New contentNew editors


2. Scaling review process



5s 10m 24h


2. Scaling review process

3. Using AI to re-shape new article review



5s 10m 24h

ORES

Part 1Openness vs. quality control


New content

New contentThe

Internet

OMG NEW CONTENT (enwiki)

● 160k edits to review / day[1]

● 1-2k new Article creations / day○ ~40k drafts need review & 200 new / day[2]

● 330k articles need cleanup[3]

● 1600 new editors / day[4]

1. http://quarry.wmflabs.org/query/4387




http://quarry.wmflabs.org/query/4387




quality?

New contentThe

Internet

New contentThe

Internet

QUALITY!

New contentThe

Internet

QUALITY!“curators”

Bad stuff

Good stuffNew content

The Internet


Bad stuff


The Internet


...backlog...

Bad stuff


The Internet


...backlog...

Good stuff

Bad stuff

New contentThe

Internet


...backlog...

Bad stuff

Good stuff

New contentThe

Internet


...backlog...

Bad stuff

Good stuff

New contentThe

Internet

“curators”

...backlog...

Bad stuff

Good stuff

New contentThe

Internet

“curators”

BACKLOG!


In summary...

● Few curators + Small amount of new content → Small backlog

In summary...


● More curators + Large amount of new content → Small backlog

In summary...


● More curators + Large amount of new content → Small backlog

● More content than curators can handle → Pressure to close the wiki AHHHHHH!

Turn it OFF!

If we don’t get this right, it’s very tempting to close Wikipedia off.

OOjs_UI_icon_lock.svg (MIT license)

https://commons.wikimedia.org/wiki/File:OOjs_UI_icon_lock.svg

#1 reason listed:“This would reduce the workload on New Page Patrollers”

Part 2Scaling review process

5seconds

1-10minutes

24+hours

New contentThe

Internet

5seconds

1-10minutes

24+hours

Good stuff

Subtle problemsBad stuff

Obviously bad stuff

New contentThe

Internet

Strategy: Filter out everything in Special:RecentChanges

5seconds

1-10minutes

24+hours

Good stuff


Obviously bad stuff

New contentThe

Internet


Racial slurs, Curse words,Key mashes

5seconds

1-10minutes

24+hours

Good stuff


Obviously bad stuff

New contentThe

Internet


“___ is gay”,dubious claims,unexplained deletions

5seconds

1-10minutes

24+hours

Good stuff


Obviously bad stuff

New contentThe

Internet


Hoaxes,Undue weight,MOS issues

5seconds

1-10minutes

24+hours

Good stuff


Obviously bad stuff

New contentThe

Internet


RC Patrollers

Hoax or NotIs this edit good or bad?

Can you tell?

HOAX

HOAX

✓

Insight:Routing based on interest makes subtle judgement calls easier.

New content Watchlists

Topic-interested individuals







5seconds

1-10minutes

24+hours

Good stuff


Obviously bad stuff

New contentThe

Internet


RC Patrollers

5seconds

1-10minutes

24+hours

RC Patrollers

Bad stuffObviously bad stuff

Probably good

Subtle problems

New contentThe

Internet

Goodstuff


Watchlists

RC Patrollers

Watchlists

https://commons.wikimedia.org/wiki/File:Flickr_-_Official_U.S._Navy_Imagery_-_Sailor%27s_daughter_operates_a_fire_hose_with_crew_member_assistance..jpg

RC Patrollers

Watchlists





RC Patrollers

Watchlists



ORES.editquality.recentchanges_split_(enwiki).svg (CC-BY-SA 3.0)

https://commons.wikimedia.org/wiki/File:ORES.editquality.recentchanges_split_(enwiki).svg

Subtle problems

Bad stuff

Obviously bad stuff

Not obviously bad

New contentThe

Internet

Goodstuff

Probably good

5seconds

1-10minutes

24+hours

RC patrollerswatch-list-ersbots

Edit review filter

Subtle problems

Bad stuff

Obviously bad stuff

Not obviously bad

New contentThe

Internet

Goodstuff

Probably good

5seconds

1-10minutes

24+hours

RC patrollers

watch-list-ers

bots

Edit review filter

Artificial.intelligence.jpg (public domain)

https://commons.wikimedia.org/wiki/File:Artificial.intelligence.jpg

RC patrollerswatch-list-ers

bots

Geiger, R. S., & Halfaker, A. (2013, August). When the levee breaks: without bots, what happens to Wikipedia's quality control processes?. In Proceedings of the 9th International Symposium on Open Collaboration (p. 6). ACM.

Edit review filter

Strainer_MET_98065.jpg (CC0)

Edit review process

● Fast● High capacity

https://commons.wikimedia.org/wiki/File:Strainer_MET_98065.jpg

Edit review process

● Fast● High capacity

Reduces the pressure to close off contribution

Firehose of 1500 newarticles per

day

The Internet

New page patrol process


day

The Internet

patrollersNew page patrol process


Really bad stuff


day

The Internet

Goodstuff



Really bad stuff


day

The Internet

Goodstuff


Bad stuffReally bad stuff


day

The Internet Backlog

patrollers

Good stuff

Subtle problems


Good stuff

Really bad stuff


day


patrollers

Subtle problems

Bad stuff


Good stuff

Really bad stuff


day


patrollers



Good stuff

Really bad stuff


day

The Internet

patrollersNew page patrol process AHHHHHH!

Turn it OFF!

Backlog


https://commons.wikimedia.org/wiki/File:Strainer_MET_98065.jpg (CC0)

https://commons.wikimedia.org/wiki/File:Gootsteenstop.png (PD)

Bad stuffReally bad stuff

The Internet

Subtle problems

Edit review process


Really bad stuff



https://commons.wikimedia.org/wiki/File:Strainer_MET_98065.jpg

https://commons.wikimedia.org/wiki/File:Gootsteenstop.png

Part 3Using AI to re-shape new

article review

ORES

Obviously bad stuff Bad stuff


● G3: Pure vandalism and blatant hoaxes● G11: Unambiguous advertising or promotion● G10: Attack pages


Obviously bad stuff

1.5k new pages

Subtle problems

Subtle problems

Subtle problems

Probably good edits Watchlists


Subtle problems



Very difficult to watch pages before they are created...

Subtle problems



Topic modeling in ORES

https://en.wikipedia.org/w/index.php?title=Alan_Turing&oldid=234785 (CC-BY-SA)

ORES

https://en.wikipedia.org/w/index.php?title=Alan_Turing&oldid=234785



Biography

Technology

Philosophy

Europe

ORES




Biography

Technology

Philosophy

Europe

ORES


Culture and arts Geography History and society STEM

Arts Music Performing Plastic VisualBroadcastingCrafts and hobbiesEntertainment Games and toysFood and drinkInternet cultureLanguage and lit. Linguistics BiographyMediaPhilosophy and rel.Sports

Bodies of waterCitiesCountries Africa Americas Asia Europe OceaniaLandformsMapsParks & cons.

History and societyBusiness and econ.EducationMilitary and warfarePolitics and gov.Transportation

ScienceBiologyChemistryEconomicsEngineeringGeosciencesMedicineInformation scienceMathematicsMeteorologyPhysicsSpaceTechnologyTimeWomen's health
















Hand-wavey Feature Engineering for my ML Friends● Features: 300 cell vector based on word2vec

…

● Estimator: GradientBoosting (one vs rest strategy)

Hand-wavey statistics for my ML Friends

TL;DR: It really works well -- even on the drafty first revisions of articles!

The ORES vision for article draft review


Released:July, 2017


Released:July, 2017

Released:March, 2018

Roger_Bamkin_y_Rosie_Stephenson-Goodknight_en_Wikimania_2015_22.JPG (CC-BY-SA 4.0)

https://commons.wikimedia.org/wiki/File:Roger_Bamkin_y_Rosie_Stephenson-Goodknight_en_Wikimania_2015_22.JPG

Mary Wollstonecraft by John Opie (c. 1797).jpg (PD)

https://commons.wikimedia.org/wiki/File:Mary_Wollstonecraft_by_John_Opie_(c._1797).jpg

Rescuesquad.png (CC-BY-SA)

https://commons.wikimedia.org/wiki/File:Rescuesquad.png

New Editors

+MoreNew Editors

=People like:

New Editors

+MoreNew Editors

=

andLess hard work for page patrollers

People like:

Disclaimer: Rosiestep has not reviewed or endorsed this presentation or the modeling project I have described.

I had one conversation with Rosiestep at Wikimania’17 and I happen find her and her work inspiring -- so I thought she made for a great example.

She deserves some of the credit, but none of the blame for what I’m talking to you about.

We have the models...


Now we need product integration.






Now we need product integration.User:SQLBot

New content

Thanks!Props to Sumit Asthana for doing most of the hard work.

Aaron Halfaker

[email protected]

EpochFail/halfak

SCIENCE

SCIENCE

SCIENCESCIENCE

SCIENCESCIENCE SCIENCE

mailto:[email protected]

● Expanding beyond English Wikipedia

● Other workflows that are struggling

● Demo some of these AIs -- what is ORES anyway?

Documents

Internet The open - Wikimedia€¦ · Internet Using AI to keep Wikipedia open Aaron Halfaker -- Wikimania 2018 ... (MIT license) #1 reason listed: “This would reduce the workload