, M. Van Kleek, E. Simperl, M. Lucsak-Roesch, R. Simpson, N. Shadbolt y of Southampton, SOCIAM Project, Web & Internet Science y of Oxford, Department of Astrophysics signing for Citizen Data Analysis: Cross-Sectional Study of Multi-Domain Citizen Science Platform
1. R. Tinati, M. Van Kleek, E. Simperl, M. Lucsak-Roesch, R.
Simpson, N. Shadbolt University of Southampton, SOCIAM Project, Web
& Internet Science University of Oxford, Department of
Astrophysics CHI 2015 Designing for Citizen Data Analysis: A
Cross-Sectional Study of A Multi-Domain Citizen Science
Platform
2. background method findings related work conclusions
3. background method findings related work conclusions
4. The Zooniverse (July 2007 - present) 1.31m+ active
volunteers today 35+ projects space, climate, humanities nature,
health 87+ academic papers
5. Solar Stormwatch GZ Supernova* GZ Hubble* Moon Zoo Old
Weather Milky Way Planet Hunters Ancient Lives Ice Hunters NEEMO*
Whale FM SETI Live* Galaxy Zoo 4 Seafloor Explorer Cyclone Center
Bat Detective Cell Slider Andromeda Project* Snapshot Serengeti
Planet Four Notes from Nature Space Warps* Worm Watch Lab Plankton
Portal Radio Galaxy Zoo Operation War Diary Disk Detective Condor
Watch Sunspotter Asteroid Zoo Whats the Score Floating Forests
Chicago Wildlife Watch Galaxy Zoo Quench Snapshot Supernova Penguin
Watch Higgs Hunters space (astrophysics) nature & biology
humanities & history climatology
6. Citizen-led investigations Serendipitous Discoveries
Persistent Cross-Project Community
7. Hannys Voorwerp Galaxy Zoo [2007] Green Pea Galaxies Galaxy
Zoo [2007] Yellow Balls Milky Way [2009] Circumbinary Planet Ph1b
Planet Hunter [2012] Convict Worm Seafloor Explorer [2012] Spanish
Flu Operation War Diaries [2014]
8. factors that most differentiated project outcomes that led
to rapid project completion & adoption on some, slower uptake
on others that fostered citizen-led investigations &
serendipitous discovery? what tacit knowledge gained by the
Zooniverse team can be elicited to establish an understanding of:
patterns and anti-patterns toward the design of future Citizen
Science (CS) Study Motivations
9. background method findings related work conclusions
10. method Structured reflection (based on Kolb, DA. 1984) with
2 core members of the Zooniverse team project founder and lead
project manager Explore the Zooniverse with Chris Lintott Setting
Zooniverse HQ Oxford Data access to lab notes digital notes code
repository logs blog posts historical usage log statistics
11. Thematic analysis Structured reflection sessions w/
Zooniverse Team Recordings transcribed Transcriptions coded and
reduced into themes Themes were then re-presented to the Zooniverse
team members for discussion and refinement Final themes
derived
12. Thematic analysis Structured reflection sessions w/
Zooniverse Team Recordings transcribed Transcriptions coded and
reduced into themes Themes were then re-presented to the Zooniverse
team members for discussion and refinement Final themes
derived
13. Thematic analysis Structured reflection sessions w/
Zooniverse Team Recordings transcribed Transcriptions coded and
reduced into themes Themes were then re-presented to the Zooniverse
team members for discussion and refinement Final themes
derived
14. Thematic analysis Structured reflection sessions w/
Zooniverse Team Recordings transcribed Transcriptions coded and
reduced into themes Themes were then re-presented to the Zooniverse
team members for discussion and refinement Final themes
derived
15. Thematic analysis Structured reflection sessions w/
Zooniverse Team Recordings transcribed Transcriptions coded and
reduced into themes Themes were then re-presented to the Zooniverse
team members for discussion and refinement Final themes
derived
16. method http://mitpress.mit.edu/boo ks/building-successful-
online-communities comparison with online communities Design Claims
Kraut, Resnick et al.s Building Successful Online Communities Ch.
2. Encouraging Contribution to Online Communities Ch. 3.
Encouraging Commitment in Online Communities Ch. 5. The Challenges
of Dealing with Newcomers Ch. 6. Starting New Online
Communities
17. background method findings related work conclusions
18. findings We organised our Design Claims (DCs) around the
following 4 themes:* TASK-COMMUNITY INTEGRATION COMMUNITY
DEVELOPMENT TASK DESIGN PUBLIC RELATIONS & ENGAGEMENT *themes
were for structuring purposes, rather than theoretically
driven
19. TASK-COMMUNITY INTEGRATION a) citizen science participants
benefit from discussing many aspects of projects, and discussion
compels prolonged engagement & peripheral participation b)
standard message boards work well for small online citizen science
communities, but tend not to scale as communities grow c)
separation between discussion & task interfaces can hinder
community growth & discussion by making tasks disconnected
& decontextualised from discussions (more on this next!) DC1:
let them talk! - Discussion facilitation essential to CS, even from
Day 1.
20. TASK-COMMUNITY INTEGRATION a) citizen science participants
benefit from discussing many aspects of projects, and discussion
compels prolonged engagement & peripheral participation b)
standard message boards work well for small online citizen science
communities, but tend not to scale as communities grow c)
separation between discussion & task interfaces can hinder
community growth & discussion by making tasks disconnected
& decontextualised from discussions (more on this next!) DC1:
let them talk! - Discussion facilitation essential to CS, even from
Day 1.
21. DC2: Discussion enables citizen-led discovery Talking about
subjects enables people to ask questions and share observations
about what they see, outside the task workflows. Such affordances
open up pathways to user-initiated, user-directed processes of
inquiry that can sometimes lead to discovery. Linking discussions
in situ with task workflow offers many advantages: establishes
necessary task context framing for discussion, lowers engagement
barriers to participate, & enhances overall discoverability of
relevant posts. TASK-COMMUNITY INTEGRATION
22. DC2: Discussion enables citizen-led discovery Talking about
subjects enables people to ask questions and share observations
about what they see, outside the task workflows. Such affordances
open up pathways to user-initiated, user-directed processes of
inquiry that can sometimes lead to discovery. Linking discussions
in situ with task workflow offers many advantages: establishes
necessary task context framing for discussion, lowers engagement
barriers to participate, & enhances overall discoverability of
relevant posts. TASK-COMMUNITY INTEGRATION
23. DC3: (a) Granting roles and privileges to participants with
experience can motivate contributors to assume community leadership
& essential moderator roles. (b) Volunteer moderators can
effectively & quickly solve many kinds of issues without
science team intervention. [Adapted from Kraut et al.] 300+
posts/day across the Zooniverse DC4*: Timely support from science
team members fosters citizen-led discovery follow up on potential
leads by bringing in expertise, external data resources not
available to citizens, extra analytical tools & techniques.
COMMUNITY DEVELOPMENT
24. Encouraging Contribution DC23: Rewards, whether in the form
of status, privileges, or material benefits, will motivate
contributions. Starting New Online Communities DC33: Promoting the
status or readiness benefits of being early to an online community
can attract early adopters to the community. Regulating Behaviour
DC26: Reputation systems, which summarize the history of someones
online behaviour, help to encourage good behaviour and deter norm
violations. Comparing to Kraut et al.s Building Successful Online
Communities
25. Any method to skip forward in tasks, (e.g. I dont know /
Skip This) discouraged attempts at best guesses and undermined
motivation. (Being able to see new subjects served as a form of
implicit reward; thus any Skip mechanism devalued such rewards.)
Contributions from best guesses had substantial cumulative effect
on project completion. DC5*: Design tasks to encourage best guesses
to avoid the dont know effort trap TASK DESIGN
26. DC6*: factors that seemed to most differentiate
participants perceptions of Zooniverse projects included intrinsic
interestingness of domain & subjects, task difficulty (effort
& skill required), time required to complete each task, &
frequency of targets in rare-item search tasks. However, other
factors including: type of task (e.g. classification,
transcription), domain of study (humanities, astrophysics, climate
science, natural science), research cause (e.g. cancer research),
or source of data had no perceived impact. TASK DESIGN
27. DC7*: get new participants contributing straight away
Avoiding introductory materials, (videos, tutorials) and allowing
participants to get to work straight away on tasks with real data
increased engagement, reduced bounce rate, enabled leveraging work
from 1-visit wonders. (6,620,423 classifications, ~15% came from
transient users who never signed up) Mandatory tutorials slowed
down participant uptake, and, in turn, project completion. In-task
guidance, interposed gold-standard tasks worked well. TASK
DESIGN
28. DC9: Providing context to tasks (e.g. background info about
subjects being classified) can provide value by making tasks more
interesting, thereby increasing engagement. Initially, it was
thought that such background information might serve as a
distraction from the primary task. However, in moderation, it led
to people getting more interested in tasks, which more often led to
citizen-led lines of inquiry, and increased chances of citizen-led
discoveries. TASK DESIGN DC8*: performance feedback can motivate
continued participation. Interspersing gold standard tasks among
the real tasks, offering participants feedback, caused an increase
in sustained engagement & improved performance. Participants
became more accurate and faster at completing tasks in the presence
of feedback.
29. DC9: Providing context to tasks (e.g. background info about
subjects being classified) can provide value by making tasks more
interesting, thereby increasing engagement. Initially, it was
thought that such background information might serve as a
distraction from the primary task. However, in moderation, it led
to people getting more interested in tasks, which more often led to
citizen-led lines of inquiry, and increased chances of citizen-led
discoveries. TASK DESIGN DC8*: performance feedback can motivate
continued participation. Interspersing gold standard tasks among
the real tasks, offering participants feedback, caused an increase
in sustained engagement & improved performance. Participants
became more accurate and faster at completing tasks in the presence
of feedback.
30. First 24 hrs of a project served as an accurate indicator
of how the project would continue to perform. Problems could be
detected early, PR strategies could be evaluated & tuned DC10*:
Project launch performance indicative of future success; pre-launch
PR important for recruiting new & existing members. PUBLIC
RELATIONS & ENGAGEMENT DC11*: Citizen science communities can
be extremely effective at promoting projects through social media
campaigns.
31. First 24 hrs of a project served as an accurate indicator
of how the project would continue to perform. Problems could be
detected early, PR strategies could be evaluated & tuned DC10*:
Project launch performance indicative of future success; pre-launch
PR important for recruiting new & existing members. PUBLIC
RELATIONS & ENGAGEMENT DC11*: Citizen science communities can
be extremely effective at promoting projects through social media
campaigns.
32. background method findings related work conclusions
33. related work Motivation of citizen scientists (Raddick et
al. 2010, 2013) Discussions related to task completion and success
(Hassman et al. 2013, Mugar et al. 2014) Cross-project
participation bolsters engagement and sustained activity
(Luczak-Rsch et al. 2014) Changes in participants Motivations over
time (Rotman, 2012)
34. background method findings related work conclusions
35. conclusions Designing successful citizen science projects
and platforms can be improved by knowing domain knowledge and
expertise, which primarily involves understanding the needs of
citizen scientists. Contributing to the existing literature
concerning the citizen science design space, we have provided
patterns and anti-patterns of design which can improve the
likelihood of sustaining an active community of participants
Citizen Scientists are not crowd workers, they need to be
considered as part of the scientific process. The discussions and
extra work that they contribute may be extremely beneficial to the
overall project success.