R. Tinati, M. Luczak-Rösch, E. Simperl, W. Hall, N. Shadbolt

University of Southampton, SOCIAM, Web & Internet Science

Web Science 2015

‘/Command and Conquer’

Analysing Discussion in a Citizen

Science Game

background

related work

data

findings

conclusions

background

related work

data

findings

conclusions

Citizen Science

It’s not new! In the 16th century, scientific discovery was a collaborative process

The Web/crowdsourcing now helps this happen again

Solving scientific problems using crowdsourcing techniques

Emphasis on communities that are volunteers, unpaid, and potentially

untrained in the domain

It has been applied to various domains: astronomy, medicine, nature, history

EyeWire

“A game to map the brain”

A gamified citizen science project

Started: Jan 2012 - present

160,000+ active volunteers today

Players from 145 Countries

Sponsored by National

Telecommunications Organisation

in South Korea

EyeWire’s Task

Players are tasked with filling in the 3D structures of Brain neurons

Uses gamification components to turn the task into gaming activities

Gamification features as a way to increase interest in the task

Leader boards, points, badges, roles

The original task is still visible, so not full gamification (e.g. FoldIT)

EyeWire’s Talk

Players can talk in real-time with each other

The Real-time chat interface is embedded in the main user

interface

Players can issue commands using specific pre-assigned terms

report or list stats and current leaders, communicate directly with

other players, join teams, ask for help, or silence each other

EyeWire’s Competitions and Teams

EyeWire runs gaming competitions to speed up the classification

process

Competitions were a grassroots movement led by the active

players

Originally only during competitions, players can join teams and

compete to gain the most points

Real-time chat facilitates the process of coordination and

progress.

We seek to understand real-time chat

what is the relationship between real-time chat and the

gaming process?

what do players use real-time chat for?

does real-time chat facilitate discourse and collaboration

between players?

Past studies have shown that passive discussion plays an

important role in the success of citizen projects

Based on these findings, how can EyeWire better support its community

Study Motivations

background

related work

data

findings

conclusions

related work

Motivation of citizen scientists (Raddick et al. 2010, 2013; Rotman, 2012)

Discussions related to task completion and success

(Hassman et al. 2013, Mugar et al. 2014)

Cross-project participation bolsters engagement and sustained activity

(Luczak-Rösch et al. 2014)

The use of gamification in citizen science projects (Bowser et al. 2013,2014)

background

related work

data

findings

conclusions

data

EyeWire player gaming and chat log activity between

January 2012 and August 2014

97,000+ Players

4,000,000+ completed games

835,000+ chat messages

Analysis Process

General platform analysis

Player analysis (Churn, Player Types)

Real-time chat analysis

Real-time Chat and Task Completion

background

related work

data

findings

conclusions

general overview

68.2% of games were completed by 10.9% of EyeWire players

… and just over 1% of EyeWire players were responsible for over 50% of the total

games (2 million)

‘active’ players on average completed a higher number of games in comparison to

gaming only players (255 games compared to 15).

‘active’ players on average took longer to complete games in comparison to gaming

only players (293s compared to 258s).

The overall account length (the total time they were active on EyeWire) of ’active’

players was nearly 4 times longer.

general platform characteristics

Re

la-t

ime

ch

atm

essa

ge

s

Games Completed

Player Churn (method)

Purpose:

Measure if players remain active after a given period of time

We separate a player’s gaming session based on task activity

We also separate player chat activity into the corresponding timeframe

(of their gaming)

Activity is segmented into monthly (m) time slices

We examine if players in m are in m-1. If not, players are identified as

non-participating players.

gaming activity churn

real-time chat churn

Types of players

Purpose:

Distinguish different types of players based on how they

interaction with they EyeWire features

How does these types of players relate to their gaming activity

We looked at features including

use of chat messages

use of chat commands

types of commands used

timing of command use during gaming

Types of players

Chat + Gaming Players

[‘active players’]

(10.9%)

Command

Players

(3.2%)

All Players

(100%)

‘Highly Active’

(1.1%)

Players using commands

12% of messages were made during gaming sessions

29.5% of ‘active players’ used commands

60.3% of ‘highly active players’ used commands

players that used commands within their chat completed over 6

times as many games on average (642 in comparison to 94)

Command-using player were slower to complete games in

comparison to non-command using players (317s compared to

286s)

Players using commands

Com

ma

nds U

sed (

log)

Games Completed (log)

Chat and command use during playing

Purpose:

Can we examine how the use of chat at different points in time affects

gaming sessions?

The 5 stages that a chat message occurs during the completion of a

game.

Chat and command use during playing

12% of chat messages were made at the same time as chatting

Games took on average 464 seconds in comparison to 364 seconds

Stage Messages Associated

Games

Avg. Message

Len. (Chars)

Avg Class.

Duration (Secs)

Commands

Used

Before (Q0 ) 19,942 11,811 26 142.6 11,271

Start (Q1) 13,154 23,583 25 865.6 17,070

During (Q2-3) 29,783 50,143 30 461.9 45,540

End (Q4) 14,497 21,075 30 344.0 22,660

After (Q5) 18,735 12,236 20 177.51 8,972

Chat and command use during playing

Stage Messages Associated

Games

Avg. Message

Len. (Chars)

Avg Class.

Duration (Secs)

Commands

Used

Before (Q0 ) 19,942 11,811 26 142.6 11,271

Start (Q1) 13,154 23,583 25 865.6 17,070

During (Q2-3) 29,783 50,143 30 461.9 45,540

End (Q4) 14,497 21,075 30 344.0 22,660

After (Q5) 18,735 12,236 20 177.51 8,972

Messages before gaming sessions (Q0) and at the start (Q1) are shorter in length

than during (Q2-3) or at the end (Q4)

Messages at the start (Q1) or during (Q2-3) increased game completion time

Chat and command use during playing

before start during end after

“Introductions,

Welcome”

“/silence,

/team

“Questions,

Help”

“/team,

Interface

“Farewells,

/gm

background

related work

data

findings

conclusions

conclusions

what is the relationship between real-time chat and the gaming process?

real-time chat appear to have a positive effect on player activity

what do players use real-time chat for?

game commands are the predominate use of real-time chat, especially for the

‘highly active’ players

conversations are short, without scientific vocab (unlike other CS platform)

does real-time chat facilitate discourse and collaboration between players?

real-time chat may provide a mechanism for players to interact with each other,

and work in teams, but we did not detect user flows in chat sessions

Implications for citizen science

Real-time chat and gamified features may be beneficial to engagement and player

success

the use of game commands appears to be related with player activity and length of

account activity.

real-time chat may facilitate co-operative playing between members

if co-ordination within the game/platform is required, then a communication

mechanism such as real-time chat is essential.

real-time chat may not be a sufficient mechanism to facilitate in-depth scientific

discourse

additional communication channels may be required (Such as Zooniverse talk)

thank you

r.tinati@soton.ac.uk

sociam.org