Crowdsourcing Disagreement for Collecting Semantic Annotation

Anca Dumitrache@anouk_anca

Extended Semantic Web ConferencePhD Symposium

June 1st 2015

annotatorannotation

example

annotatorannotation

exampleproblem:

annotatorannotation

exampleproblem:

●

annotatorannotation

exampleproblem:

●

●

annotatorannotation

exampleproblem:

●

●

●

annotatorannotation

exampleproblem:

●

●

●

quality = inter-annotator agreement

crowdannotation

examplesolution:

annotation

examplesolution:

●

●

●

crowd

annotation

examplesolution:

●

●

●

crowd

Hypothesis:

annotation

exampleapproach:

crowd

annotation

example ambiguityapproach:

crowd

annotation clarity

exampleapproach:

crowd

worker qualityannotation

exampleapproach:

annotation

exampleapproach:

crowd

annotation

exampleapproach:

crowd

annotation

exampleapproach:

●

●

crowd

annotation

example

crowd

approach:

annotation

example

●

●

crowd

approach:

Crowdsourcing Methodology

annotation

example

experimental plan:

crowd

annotation

example

experimental plan:

crowd

annotation

example

experimental plan:

crowd

annotation

example

experimental plan:

crowd

annotation

example

experimental plan:

crowd

annotation

example

crowd

current work:

annotation

example

approach:

crowd

Crowdsourcing task workflow

annotation

example

evaluation:

crowd

Classifier evaluation results

Try it out!

CrowdTruth.org

github.com/CrowdTruth.org

@anouk_anca

#CrowdTruth