Gamification of crowdsourcing tasks:
What motivates a medical expert?
18 July 2014Rens van Honschooten
Lora Aroyo (VU) & Robert-Jan Sips (IBM)
What motivates medical experts?
Gamification of crowdsourcing tasks: What motivates a medical expert?
Why incentives for a medical expert?
● Train Watson in the medical domain
● To train cognitive systems you need data○ Lots of it
● To get annotated data:
Gamification of crowdsourcing tasks: What motivates a medical expert?
Crowdsourcing games to collect data
● Large amounts of data, fast and cheap [1]
Gamification of crowdsourcing tasks: What motivates a medical expert?
Crowdsourcing games to collect data
● One thing they all have in common?
● They motivate their crowd
● The challenge: find out how to motivate busy medical experts
Gamification of crowdsourcing tasks: What motivates a medical expert?
Dr. Detective: finding clues for a diagnosis
● Learning features
Gamification of crowdsourcing tasks: What motivates a medical expert?
Dr. Detective: finding clues for a diagnosis
● Learning features● Most experts liked playing it
● Downsides:○ The scoring was not clear○ No rewards for completing a task
● Incentives○ Competition○ Entertainment○ Learning
● Are these all the incentives?
● Which are the most important?
Gamification of crowdsourcing tasks: What motivates a medical expert?
Crowdsourcing incentives (literature)
Moon, H., & Baek, Y. (2009). Exploring variables affecting player’s intrinsic motivation in educational games. In Proceedings of the 17th International Conference on Computers in Education (pp. 718-722).
Fang, Y., & Neufeld, D. (2009). Understanding sustained participation in open source software projects. Journal of Management Information Systems, 25(4), 9-50.
Incentive Example
Personal interest Interested in science, medicine, etc.
Contributing to the community Helping diabetes patients to eat healthy
Status & recognition Being recognized as a good doctor
Personal growth Learning new thing in their medical domain
Enjoyment Solving challenging medical problems
Choice autonomy Choosing what to do and when, at work
A monetary reward Receiving money for helping patients
Gamification of crowdsourcing tasks: What motivates a medical expert?
Gamification of crowdsourcing tasks: What motivates a medical expert?
Helping patientsImprove medical field
Gamification of crowdsourcing tasks: What motivates a medical expert?
● Can we incorporate these incentives to motivate the experts?
● Created a design for next version of Dr. Detective, called Dr. Watson○ evaluated mockup by interviewing and testing with 6 experts
Gamification of crowdsourcing tasks: What motivates a medical expert?
The design of Dr. Watson
Gamification of crowdsourcing tasks: What motivates a medical expert?
The design of Dr. Watson● Personal growth,
learning
● Reading most preferred
● Provide overview of literature
● Spark their interest
Gamification of crowdsourcing tasks: What motivates a medical expert?
The design of Dr. Watson● To unlock articles:
○ play games to gather reputation points
● Articles motivate medical experts○ Except researchers and students○ Medical books, videos, giveaways and
accreditation points should motivate these experts
Gamification of crowdsourcing tasks: What motivates a medical expert?
Gamification of crowdsourcing tasks: What motivates a medical expert?
● Select game mode
● Cannot explore all content
● Make the experts
curious
● More difficult tasks, to keep the game challenging
Gamification of crowdsourcing tasks: What motivates a medical expert?
Gamification of crowdsourcing tasks: What motivates a medical expert?
● Make game more challenging
● Motivate the experts to keep playing○ motivate them to play less popular tasks and
annotate articles without annotations
Gamification of crowdsourcing tasks: What motivates a medical expert?
Gamification of crowdsourcing tasks: What motivates a medical expert?
● Goal: beat the score of the crowd○ average amount of points earned by others○ matching the score also means you win
● To get points○ Agreement with other experts○ Pioneer by suggesting new answer○ Three or five in a row bonus
● The scoring was clear for the experts○ streaks motivate them to annotate correctly
● To increase game difficulty use topics the expert is less familiar with
Gamification of crowdsourcing tasks: What motivates a medical expert?
Gamification of crowdsourcing tasks: What motivates a medical expert?
Gamification of crowdsourcing tasks: What motivates a medical expert?
● To support learning goals more topic selection is required
● Medical experts stated to play again○ To learn more○ To improve their score○ Because they enjoyed playing it
● The experts could easily navigate through the UI○ elements at the bottom of the sidebar
● Experts wanted to see more clearly how Watson is improving
Gamification of crowdsourcing tasks: What motivates a medical expert?
Future work - improving Dr. Watson
● Topic selection for tasks
● Personalization of:○ The difficulty - take specialization in account○ The homepage - change background image, reorder elements
● Incorporate additional unlockables/rewards○ Videos○ Medical books○ Accreditation points
● Add ‘Watson status’ page○ track the progress of the crowd and what watson achieved
Gamification of crowdsourcing tasks: What motivates a medical expert?
What motivates medical experts?
Helping patientsImprove medical field
Questions?