Lessons learned building machine learning models ...€¦ · Lessons learned building machine...

Lessons learned building machine learning models

for WikidataWikimania 2016, Esino Lario, 24.06.20016

Amir Sarabadani User:Ladsgroup@AmirSarabadani

Wikidata?

● free knowledge base by the Wikimedia movement

● structured data● user created content● linked data● people, places, events, …● powered by Wikibase (open source)● data: CC-0

Quality control in Wikidata

● Quality of Wikidata is important● Vandalism is a threat!● But the backlog is too big

Wikidata gets 80,000 human edits per day, 320 will be reverted and 80 of them will have been vandalism.

0.4% of edits are reverted, 0.1% of reverted edits are vandalism

80,000 edits per day(133 hours per day)

320 reverted

80 vandal

https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2015-12-09/Op-ed

“For Wikidata to truly give more people more access to more knowledge, the data in Wikidata needs to be of high quality.”

“And probably the most important part is machine-learning tools like ORES that help us find bad edits and other issues.”

Machine learning classifiers

● Machine learning in Wikipedia● But there is no infrastructure ● Here it comes the ORES

is_anonchrs_addedchrs_removedcust_commentrepeated_chrslongest_tokenbadwords_added

80kHumanEditsPer Day

Need review

Probably OKhttps://commons.wikimedia.org/wiki/File:Flickr_-_Official_U.S._Navy_Imagery_-_Sailor%27s_daughter_operates_a_fire_hose_with_crew_member_assistance..jpg

https://ores.wikimedia.org/scores/wikidatawiki/damaging/267233560

“Wikidata”

Is this edit damaging?

{ "267233560": { "prediction": true, "probability": { "false": 0.111, "true": 0.889 } }}

{ "286961313": { "prediction": false, "probability": { "false": 0.946, "true": 0.054 } }}

ORES in Wikipedia vs. Wikidata

● Wikitext-based vandalism detection● Wikibase-based vandalism detection● Percentage of vandalism

Wikipedia: 1-10% (Mostly ~7%)Wikidata: ~0.1%

ORES extension

● Database cached = speed and filtering● Hit record

Problem of harvesting data

● Different languages● Type of statements● Why NLP and categories are not options

How Kian works

● Extracting features● Training

high_density_settlement_cats = 1

medium_density_settlement_cats = 0

low_density_settlement_cats = 0

Settlement

Not a settlement

Questions?

Revision Scoring team:Aaron Halfaker (ahalfaker@wikimedia.org)Amir Sarabadani

Me?User:LadsgroupLadsgroup@gmail.com@AmirSarabadani (twitter)

Lessons learned building machine learning models ...€¦ · Lessons learned building machine...

Documents

Lessons&Learned&in&Deploying&Akka& Persistencedownloads.typesafe.com/website/presentations/lessons-learned-akk… · Lessons&Learned&in&Deploying&Akka& Persistence Duncan&K.&DeVore

Lessons Learned Handbook · 2011-10-12 · • Lessons Learned, an adjective, describes anything related to a Lessons Learned procedure. E.g. Lessons Learned process, Lessons Learned

ASSESSMENT & LESSONS LEARNED FROM AFRICAN DIASPORA MARKETPLACEpdf.usaid.gov/pdf_docs/PDACT256.pdf · ASSESSMENT & LESSONS LEARNED FROM AFRICAN ... ASSESSMENT & LESSONS LEARNED FROM

Lessons learned after 190M lessons served

DDM UPDATE LESSONS LEARNED & LESSONS SHARED …

Lessons Learned Events at SLAC · 2012-09-14 · Lessons Learned Events: Summary Comments . Important to identify and communicate lessons learned! Lessons learned and near misses

Wikipedia, Wikimania, Wikimania Esino Lario

Lessons learned

Lessons learned & not learned at MSU

Lessons learned from accident investigation Seminar/IHS - Day 2... · LESSONS LEARNED FROM OTHERS LESSONS LEARNED FROM UAE ... Lessons Learned . ... Dealing with survivors and bereaved

Lessons Learned and Lessons to Be Learned: Investment Law & … · 2014. 4. 30. · 1 Lessons Learned and Lessons to Be Learned: Investment Law & Development for Developed Countries

Lessons Learned Globally - Campaign for Tobacco-Free Kids...Process lessons learned 2. Content lessons learned See next page for lists of these lessons learned. 7 | Lessons Learned

Detecting Financial Reporting Fraud – Lessons learned · PDF fileDetecting Financial Reporting Fraud – Lessons learned by ... Detecting Financial Reporting Fraud – Lessons learned

Lessons Learned:

Lessons learned from Asian tsunami disaster: Sharing knowledgeusir.salford.ac.uk/9805/1/lessons.pdf · Title Lessons learned from Asian tsunami disaster: ... Lessons learned from

Lessons Learned from “Lessons Learned” · 2 LESSONS LEARNED FROM “LESSONS LEARNED” 2. The NRC set safety goals in its Safety Goal Policy Statement, initiated not long after

LEARNED LESSONS:

Lessons Learned, Opportunities and Innovations in …pdf.usaid.gov/pdf_docs/PBAAD103.pdf · Lessons Learned, Opportunities and Innovations in Human ... Lessons Learned, Opportunities

LESSONS LEARNED AND GOOD PRACTICES...Lessons Learned Lessons are learned for sharing and applying to similar projects. Project lessons are learned from experiences, both inspiring

The Apollo Experience Lessons Learned for Constellation ... · PDF fileThe Apollo Experience Lessons Learned for Constellation Lunar Dust Management ... Experience Lessons Learned