23
10/24/2011 www.insemtives.eu 1 Ten ways to make your semantic app addicted - REVISITED Elena Simperl Tutorial at the ISWC2011, Bonn, Germany

Insemtives iswc2011 session1

Embed Size (px)

DESCRIPTION

Slides of the Insemtives tutorial at the ISWC 2011 in Bonn, Germany. More slides available at http://www.insemtives.eu/iswc2011-tutorial/

Citation preview

Page 1: Insemtives iswc2011 session1

10/24/2011 www.insemtives.eu 1

Ten ways to make your semantic app addicted - REVISITED

Elena Simperl

Tutorial at the ISWC2011, Bonn, Germany

Page 2: Insemtives iswc2011 session1

Executive summary • Many aspects of semantic content authoring naturally rely

on human contribution. • Motivating users to contribute is essential for semantic

technologies to reach critical mass and ensure sustainable growth.

• This tutorial is about

– Methods and techniques to study incentives and motivators applicable to semantic content authoring scenarios.

– How to implement the results of such studies through technology design, usability engineering, and game mechanics.

www.insemtives.eu 2

Page 3: Insemtives iswc2011 session1

Incentives and motivators

• Motivation is the driving force that makes humans achieve their goals.

• Incentives are ‘rewards’ assigned by an external ‘judge’ to a performer for undertaking a specific task. – Common belief (among

economists): incentives can be translated into a sum of money for all practical purposes.

• Incentives can be related to both extrinsic and intrinsic motivations.

• Extrinsic motivation if task is considered boring, dangerous, useless, socially undesirable, dislikable by the performer.

• Intrinsic motivation is driven by an interest or enjoyment in the task itself.

Page 4: Insemtives iswc2011 session1

Examples of applications

www.insemtives.eu 4

Page 5: Insemtives iswc2011 session1

Extrinsic vs intrinsic motivations • Successful volunteer crowdsourcing is difficult

to predict or replicate. – Highly context-specific. – Not applicable to arbitrary tasks.

• Reward models often easier to study and control.* – Different models: pay-per-time, pay-per-unit, winner-

takes-it-all… – Not always easy to abstract from social aspects (free-

riding, social pressure…). – May undermine intrinsic motivation.

* in cases when performance can be reliably measured

Page 6: Insemtives iswc2011 session1

Examples (ii)

Mason & Watts: Financial incentives and the performance of the crowds, HCOMP 2009.

Page 7: Insemtives iswc2011 session1

Amazon‘s Mechanical Turk

• Types of tasks: transcription, classification, and content generation, data collection, image tagging, website feedback, usability tests.*

• Increasingly used by academia. • Vertical solutions built on top. • Research on extensions for complex tasks.

* http://behind-the-enemy-lines.blogspot.com/2010/10/what-tasks-are-posted-on-mechanical.html

Page 8: Insemtives iswc2011 session1

Tasks amenable to crowdsourcing

• Tasks that are decomposable into simpler tasks that are easy to perform.

• Performance is measurable. • No specific skills or expertise are required.

Page 9: Insemtives iswc2011 session1

Patterns of tasks* • Solving a task

– Generate answers – Find additional information – Improve, edit, fix

• Evaluating the results of a task – Vote for accept/reject – Vote up/down to rank

potentially correct answers – Vote best/top-n results

• Flow control – Split the task – Aggregate partial results

• Example: open-scale tasks in Mturk – Generate, then vote. – Introduce random noise to

identify potential issues in the second step

* „Managing Crowdsourced Human Computation“@WWW2011, Ipeirotis

Gene

rate

ans

wer

Label image

Vote

ans

wer

s

Correct or not?

Page 10: Insemtives iswc2011 session1

Examples (iii)

www.insemtives.eu 10

Page 11: Insemtives iswc2011 session1

What makes game mechanics successfull?*

• Accelerated feedback cycles. – Annual performance appraisals vs immediate feedback to

maintain engagement. • Clear goals and rules of play.

– Players feel empowered to achieve goals vs fuzzy, complex system of rules in real-world.

• Compelling narrative. – Gamification builds a narrative that engages players to

participate and achieve the goals of the activity.

• But in the end it’s about what task users want to get better at.

*http://www.gartner.com/it/page.jsp?id=1629214 Images from http://gapingvoid.com/2011/06/07/pixie-dust-the-mountain-of-mediocrity/ and http://www.hideandseek.net/wp-content/uploads/2010/10/gamification_badges.jpg

Page 12: Insemtives iswc2011 session1

Guidelines • Focus on the actual goal and incentivize related

actions. – Write posts, create graphics, annotate pictures, reply

to customers in a given time… • Build a community around the intended actions.

– Reward helping each other in performing the task and interaction.

– Reward recruiting new contributors. • Reward repeated actions.

– Actions become part of the daily routine.

Image from http://t1.gstatic.com/images?q=tbn:ANd9GcSzWEQdtagJy6lxiR2focH2D01Wpz7dzAilDuPsWnL0i4GAHgnm_0hyw3upqw

Page 13: Insemtives iswc2011 session1

What tasks can be gamified?* • Tasks that are decomposable into simpler

tasks, nested tasks. • Performance is measurable. • Obvious rewarding scheme. • Skills can be arranged in a smooth learning

curve.

*http://www.lostgarden.com/2008/06/what-actitivies-that-can-be-turned-into.html Image from http://www.powwownow.co.uk/blog/wp-content/uploads/2011/06/gamification.jpeg

Page 14: Insemtives iswc2011 session1

What is different about semantic systems?

• It‘s still about the context of the actual application.

• User engagement with semantic tasks in order to – Ensure knowledge is

relevant and up-to-date. – People accept the new

solution and understand its benefits.

– Avoid cold-start problems. – Optimize maintenance

costs.

Page 15: Insemtives iswc2011 session1

Tasks in knowledge engineering • Definition of vocabulary • Conceptualization

– Based on competency questions – Identifying instances, classes, attributes,

relationships

• Documentation – Labeling and definitions. – Localization

• Evaluation and quality assurance – Matching conceptualization to documentation

• Alignment • Validating the results of automatic methods

www.insemtives.eu 15

Page 16: Insemtives iswc2011 session1

16

http://www.ontogame.org http://apps.facebook.com/ontogame

Page 17: Insemtives iswc2011 session1

OntoGame API • API that provides several methods that are

shared by the OntoGame games, such as: – Different agreement types (e.g. selection

agreement). – Input matching (e.g. , majority). – Game modes (multi-player, single player). – Player reliability evaluation. – Player matching (e.g., finding the optimal

partner to play). – Resource (i.e., data needed for games)

management. – Creating semantic content.

• http://insemtives.svn.sourceforge.net/viewvc/insemtives/generic-gaming-toolkit

www.insemtives.eu 17 10/24/2011

Page 18: Insemtives iswc2011 session1

OntoGame games

10/24/2011 www.insemtives.eu 18

Page 19: Insemtives iswc2011 session1

Case studies

• Methods applied – Mechanism design. – Participatory design. – Games with a purpose. – Crowdsourcing via MTurk.

• Semantic content authoring scenarios – Extending and populating

an ontology. – Aligning two ontologies. – Annotation of text, media

and Web APIs.

Page 20: Insemtives iswc2011 session1

Lessons learned • Approach is feasible for mainstream domains, where a

(large-enough) knowledge corpus is available. • Advertisement is important. • Game design vs useful content.

– Reusing well-kwown game paradigms. – Reusing game outcomes and integration in existing workflows

and tools.

• But, the approach is per design less applicable because – Knowledge-intensive tasks that are not easily nestable. – Repetitive tasks players‘ retention?

• Cost-benefit analysis.

Page 21: Insemtives iswc2011 session1

Using Mechanical Turk for semantic content authoring

• Many design decisions similar to GWAPs. – But clear incentives structures. – How to reliably compare games and MTurk results?

• Automatic generation of HITs depending on the types of tasks and inputs.

• Integration in productive environments.

– Protégé plug-in for managing and using crowdsourcing results.

Page 22: Insemtives iswc2011 session1

Outline of the tutorial Time Presentation 14:00 – 14:45

Human contributions in semantic content authoring

14:45 – 15:30

Case study: motivating employees to annotate enterprise content semantically at Telefonica

15:30 – 16:00

Coffee break

16:00 – 16:45

Case study: Crowdsourcing the annotation of dynamic Web content at seekda

16:45 – 17:30

Case study: Content tagging at MoonZoo and MyTinyPlanets

17:30 – 18:00

Ten ways to make your semantic app addicted - revisited www.insemtives.eu 22

Page 23: Insemtives iswc2011 session1

10/24/2011 www.insemtives.eu 23

Realizing the Semantic Web by encouraging millions of end-users to

create semantic content.