Gamification of crowdsourcing tasks:

What motivates a medical expert?

18 July 2014Rens van Honschooten

Lora Aroyo (VU) & Robert-Jan Sips (IBM)

What motivates medical experts?

Gamification of crowdsourcing tasks: What motivates a medical expert?

Why incentives for a medical expert?

● Train Watson in the medical domain

● To train cognitive systems you need data○ Lots of it

● To get annotated data:

Gamification of crowdsourcing tasks: What motivates a medical expert?

Crowdsourcing games to collect data

● Large amounts of data, fast and cheap [1]

Gamification of crowdsourcing tasks: What motivates a medical expert?

Crowdsourcing games to collect data

● One thing they all have in common?

● They motivate their crowd

● The challenge: find out how to motivate busy medical experts

Gamification of crowdsourcing tasks: What motivates a medical expert?

Dr. Detective: finding clues for a diagnosis

● Learning features

Gamification of crowdsourcing tasks: What motivates a medical expert?

Dr. Detective: finding clues for a diagnosis

● Learning features● Most experts liked playing it

● Downsides:○ The scoring was not clear○ No rewards for completing a task

● Incentives○ Competition○ Entertainment○ Learning

● Are these all the incentives?

● Which are the most important?

Gamification of crowdsourcing tasks: What motivates a medical expert?

Crowdsourcing incentives (literature)

Moon, H., & Baek, Y. (2009). Exploring variables affecting player’s intrinsic motivation in educational games. In Proceedings of the 17th International Conference on Computers in Education (pp. 718-722).

Fang, Y., & Neufeld, D. (2009). Understanding sustained participation in open source software projects. Journal of Management Information Systems, 25(4), 9-50.

Incentive Example

Personal interest Interested in science, medicine, etc.

Contributing to the community Helping diabetes patients to eat healthy

Status & recognition Being recognized as a good doctor

Personal growth Learning new thing in their medical domain

Enjoyment Solving challenging medical problems

Choice autonomy Choosing what to do and when, at work

A monetary reward Receiving money for helping patients

Gamification of crowdsourcing tasks: What motivates a medical expert?

Gamification of crowdsourcing tasks: What motivates a medical expert?

Helping patientsImprove medical field

Gamification of crowdsourcing tasks: What motivates a medical expert?

● Can we incorporate these incentives to motivate the experts?

● Created a design for next version of Dr. Detective, called Dr. Watson○ evaluated mockup by interviewing and testing with 6 experts

Gamification of crowdsourcing tasks: What motivates a medical expert?

The design of Dr. Watson

Gamification of crowdsourcing tasks: What motivates a medical expert?

The design of Dr. Watson● Personal growth,

learning

● Reading most preferred

● Provide overview of literature

● Spark their interest

Gamification of crowdsourcing tasks: What motivates a medical expert?

The design of Dr. Watson● To unlock articles:

○ play games to gather reputation points

● Articles motivate medical experts○ Except researchers and students○ Medical books, videos, giveaways and

accreditation points should motivate these experts

Gamification of crowdsourcing tasks: What motivates a medical expert?

Gamification of crowdsourcing tasks: What motivates a medical expert?

● Select game mode

● Cannot explore all content

● Make the experts

curious

● More difficult tasks, to keep the game challenging

Gamification of crowdsourcing tasks: What motivates a medical expert?

Gamification of crowdsourcing tasks: What motivates a medical expert?

● Make game more challenging

● Motivate the experts to keep playing○ motivate them to play less popular tasks and

annotate articles without annotations

Gamification of crowdsourcing tasks: What motivates a medical expert?

Gamification of crowdsourcing tasks: What motivates a medical expert?

● Goal: beat the score of the crowd○ average amount of points earned by others○ matching the score also means you win

● To get points○ Agreement with other experts○ Pioneer by suggesting new answer○ Three or five in a row bonus

● The scoring was clear for the experts○ streaks motivate them to annotate correctly

● To increase game difficulty use topics the expert is less familiar with

Gamification of crowdsourcing tasks: What motivates a medical expert?

Gamification of crowdsourcing tasks: What motivates a medical expert?

Gamification of crowdsourcing tasks: What motivates a medical expert?

● To support learning goals more topic selection is required

● Medical experts stated to play again○ To learn more○ To improve their score○ Because they enjoyed playing it

● The experts could easily navigate through the UI○ elements at the bottom of the sidebar

● Experts wanted to see more clearly how Watson is improving

Gamification of crowdsourcing tasks: What motivates a medical expert?

Future work - improving Dr. Watson

● Topic selection for tasks

● Personalization of:○ The difficulty - take specialization in account○ The homepage - change background image, reorder elements

● Incorporate additional unlockables/rewards○ Videos○ Medical books○ Accreditation points

● Add ‘Watson status’ page○ track the progress of the crowd and what watson achieved

Gamification of crowdsourcing tasks: What motivates a medical expert?

What motivates medical experts?

Helping patientsImprove medical field

Questions?