Digital Humanities Benelux 2017: Keynote Lora Aroyo

http://lora-aroyo.org @laroyo

Harnessing Human Semantics at Scale

Measurable, Reproducible, Engaging, SustainableCrowdsourcing & Nichesourcing

Lora Aroyo


20071998 2006 2009

from DVDs to data science


20071998 2006 2009

Team BellKor wins Netflix Prize


20061994 2003 2016 2017

from books to data science


20061994 2003 2016 2017



20061994 2003 2016 2017



20061994 2003 2016 2017



20061994 2003 2016 2017



data is at the centre of every process


data is essential to evolve with users


Ceci n'est pas … la mona lisa


Ceci n'est pas … la mona lisa

Louvre’s Mona Lisa

is only #14


the battle of two worlds

9,3 million

Louvre

visitors 2014

14 million

website visitors

2,3 million

social media


in the (very near) future

most visitors will be digital-born

not bound by time or location

native to new forms of co-makership

native to new mediaSiebe Weide, Max Meijer and Marieke Krabshuis (2012).

Agenda 2026: Study on the Future of the Dutch Museum Sector


variety of meaningsmultitude of perspectivesabundance of sourcesendless contexts

know your data


crowdsourcing to know your data at scale


variety of typesmultitude of platformsabundance of interactionsendless characteristics

know your crowds


https://www.rijksmuseum.nl/en/rijksstudio

Engage with Co-creation


Engage with Co-creativity


Engage with Co-curation


Engage the Expert Niche

http://annotate.accurator.nl



expertise of Rijksmuseum professionals is in annotating their collection

with art-historical information, e.g. when they were created, by whom, etc.


detailed domain-specific information about depicted objects, e.g. which species the

animal or plant belongs to,is in most cases not available


use nichesourcing, i.e. niches of people with the right expertise, to add more specific

information


Keep Reproducing




Engage with Games

training the general crowd to be a niche:game in which players can carry out an expert

annotation tasks with some assistance


http://waisda.nl

Engage with Games


http://waisda.nl

Engage with Games


http://spotvogel.vroegevogels.vara.nl

Keep Reproducing


CrowdTruth.org

Experiment with Paid Crowds


CrowdTruth.org



CrowdTruth.org



http://crowdtruth.org/


http://data.crowdtruth.org/


Challenges


Low reproducibility ratesDifficult to estimate & control the time to complete Difficult to assess & compare quality Demands continuous promotional effortActive learning (human-in-the-loop) needs different expertiseDifficult to incorporate results into existing content infrastructure

Challenges

Crowdsourcing typically undertaken in isolation


Assess Impact of Task Design


InstructionsLayoutSequenceCrowdsPaymentCampaign

Assess Impact of Task Design

experiment with different designs


for example

mapping music to mood


Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Otherpassionate, rollicking, literate, humorous, silly, aggressive, fiery, does not fit into

rousing, cheerful, fun, poignant, wistful, campy, quirky, tense, anxious, any of the 5

confident, sweet, amiable, bittersweet, whimsical, witty, intense, volatile, clusters

boisterous, good-natured autumnal, wry visceral

rowdy brooding

Choose one:

Which is the mood most appropriate

for each song?

Goal:

(Lee and Hu 2012)

1 song - 1 mood???


If “One Truth” & “No Disagreement”

Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5

W1 1

W2 1

W3 1

W4 1

W5 1

W6 1

W7

W8

W9 1

W10 1

Totals 1 3 1 2 1


Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 Other

W1 1 1 1

W2 1 1 1

W3 1 1 1

W4 1 1

W5 1 1

W6 1 1 1

W7 1 1 1

W8 1 1 1

W9 1 1

W10 1 1 1 1 1

Totals 3 5 6 5 2 8

If “Many Truths” & “Disagreement”

Web & Media Group


simplification of context

this all results in

Web & Media Group



● Identify Crowdsourcing Goals through user log analysis

○ # queries, #unique queries, #queries of specific type

○ ranked by popularity

○ ranked by popularity and with error, e.g.

■ # queries entered over 50 times with 0 results

■ # queries of specific type with 0 results

○ which will have biggest impact

○ which has biggest urgency

● … or through other user analysis

Assess Impact of Results


for example

in video search


people search for fragmentsexperts annotate full videos

35% of search queries result in not found

people search for fragmentsexperts annotate full videos

35% of search queries result in not found

for example

in video search


Measure Quality

“On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011


Measure Quality


time-based annotationbernhard

88% of the tags usefulfor specific genres

describe short segmentsoften not very specificdon’t describe program as a whole


for example

in video search

video annotation is time-consuming5 times the video duration

experts use a specific vocabularythat is unknown to general audiences

video annotation is time-consuming5 times the video duration

experts use a specific vocabularythat is unknown to general audiences


user vocabulary8% in professional vocabulary23% in Dutch lexicon89% found on Google

locations (7%)

engeland

persons (31%)

objects (57%)

Measure Quality


Web & Media Group


human subjectivity, ambiguity & uncertainty of expression

natural part of human semantics


measure quality

quality is not just about spamquality is typically multi-dimensionalunderstand the diversity in crowd answers do not ignore multitude of interpretationsunderstand the variety of contextsidentify cases with high ambiguity, similarity, …experiment with explicit metricsexperiment with different designs


Measure Progress

6 months 2 years

340,551 tags 36,981 tags

137.421 matches

602 items 1.782 items

555 registered players 2,017 users (taggers)

thousands of anonymous players

12,279 visits (3+ min online)

44,362 pageviews

Riste Gligorov, Michiel Hildebrand, Jacco van Ossenbruggen, Guus Schreiber, Lora Aroyo

(2011). On the role of user-generated metadata in audio visual collections. International

conference on Knowledge capture K-CAP '11, Pages 145-152


campaign, campaign, campaign





Measurable qualityReproducible resultsSustainable settingsEngaging interaction

Goals

Technology

Digital Humanities Benelux 2017: Keynote Lora Aroyo