Embed Size (px)
Citation preview
Report from Dagstuhl Seminar 17272
Citizen Science: Design and EngagementEdited byIrene Celino1, Oscar Corcho2, Franz Hölker3, and Elena Simperl4
1 CEFRIEL - Milan, IT, [email protected] Polytechnic University of Madrid, ES, [email protected] IGB - Berlin, DE, [email protected] University of Southampton, GB, [email protected]
AbstractThis report documents the program and the outcomes of Dagstuhl Seminar 17272 "Citizen Science:Design and Engagement". In this report, we briefly summarise the content of three invited keynotetalks and two invited tutorials. We further outline the findings of five parallel working groups,which met on the first and third day of the workshop, in the areas of: sustainability, measuringsuccess, community engagement, linking and quality.
Seminar July 2–5, 2017 – http://www.dagstuhl.de/172721998 ACM Subject Classification I.2 Artificial Intelligence, I.2.9 Robotics, Society, Human-
computer InteractionKeywords and phrases Citizen science, Crowdsourcing, Data Analytics, Gamification, Human
Computation, Incentives Engineering, Online Community, Open ScienceDigital Object Identifier 10.4230/DagRep.7.7.22Edited in cooperation with Neal Reeves (University of Southampton)
1 Executive summary
Irene CelinoOscar CorchoFranz HölkerElena Simperl
License Creative Commons BY 3.0 Unported license© Irene Celino, Oscar Corcho, Franz Hölker, Elena Simperl
Citizen science is an approach to science that is enlisting the help of millions of volunteersacross a range of academic disciplines to complete tasks that would have otherwise beenunfeasible to tackle using expert time or computational methods [2]. While it is a popularand effective way to solve various problems, with many examples of incredible success [3, 1],there remains a number of ongoing challenges that must be addressed in order to ensure thevalidity of citizen science as a widespread approach to research. The aim of this workshop –organised in partnership with the SOCIAM1 and Stars4All2 projects – was to discuss andexplore aspects of the future of citizen science, focusing on design factors and engagementstrategies, although this naturally required a holistic assessment of citizen science projects,platforms and applications as a whole.
1 https://sociam.org/about2 http://stars4all.eu/
Except where otherwise noted, content of this report is licensedunder a Creative Commons BY 3.0 Unported license
Citizen Science: Design and Engagement, Dagstuhl Reports, Vol. 7, Issue 7, pp. 22–43Editors: Irene Celino, Oscar Corcho, Franz Hölker, and Elena Simperl
Dagstuhl ReportsSchloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstuhl Publishing, Germany
Irene Celino, Oscar Corcho, Franz Hölker, and Elena Simperl 23
References1 Jeehung Lee, Wipapat Kladwang, Minjae Lee, Daniel Cantu, Martin Azizyan, Hanjoo
Kim, Alex Limpaecher, Snehal Gaikwad, Sungroh Yoon, Adrien Treuille et al. RNA DesignRules from a Massive Open Laboratory. Proceedings of the National Academy of Sciences,National Academy of Sciences, 111; 6, 2122–2127, 2014.
2 Chris J Lintott, Kevin Schawinski, Anže Slosar, Kate Land, Bamford Steven, DanielThomas, Jordan M Raddick, Robert C Nichol, Alex Szalay and Dan Andreescu et al.Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan DigitalSky Survey. Monthly Notices of the Royal Astronomical Society, Blackwell Publishing LtdOxford, UK, 389; 3, 1179–1189, 2008.
3 Chris J Lintott, Kevin Schawinski, William Keel, Hanny Van Arkel, Nicola Bennert, Ed-ward Edmondson, Daniel Thomas, Daniel JB Smith, Peter D Herbert, Matt J Jarvis etal. Galaxy Zoo:‘Hanny’s Voorwerp’, a quasar light echo?. Monthly Notices of the RoyalAstronomical Society, Blackwell Publishing Ltd Oxford, UK, 399; 1, 129–140,2009.
17272
24 17272 – Citizen Science: Design and Engagement
2 Table of Contents
Executive summaryIrene Celino, Oscar Corcho, Franz Hölker, Elena Simperl . . . . . . . . . . . . . . 22
Introduction
Overview of Keynote TalksCrowdsourcing for Smart Cultural Heritage: Harnessing Human Semantics at ScaleLora Aroyo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Open Citizen Science: What, how and with whom?Claudia Göbel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
The state of the art of OpenStreetMap: technologies, community and researchchallengesMaurizio Napolitano . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Overview of TutorialsCitizen Science as a New Way To Do ScienceMarisa Ponti . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Citizen Science as a Social MachineElena Simperl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Overview of Working groupsWorking Group One - Sustainability . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Working Group Two - Measuring Success . . . . . . . . . . . . . . . . . . . . . . . 30
Working Group Three - Community Engagement . . . . . . . . . . . . . . . . . . . 31
Working Group Four - Linking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Working Group Five - Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Working Group Six - Manifesto for Citizen Science . . . . . . . . . . . . . . . . . . 35
Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Appendix A - Lightning Talk Slides . . . . . . . . . . . . . . . . . . . . . . . . . 37
Irene Celino, Oscar Corcho, Franz Hölker, and Elena Simperl 25
3 Introduction
While amateur involvement in science began long before the establishment of modern academicinstitutions, the web and digital technologies have fundamentally revitalized and expandedthe ways and scale in which untrained citizens can participate in scholarly research. These‘Citizen Science’ projects have thus far enlisted the help of millions of volunteers in a widearray of scientific inquiries, ranging from the taxonomic classification of galaxies and thecreation of an online encyclopedia of all living species on Earth, to the derivation of solutionsto protein folding problems and the tracking and measuring the population and migratorypatterns of animals in the Serengeti national park [2]. This new, more inclusive way ofpursuing science is proving successful in many ways: it gives scientists around the worldan effective, affordable way to collect and analyze large amounts of data in a short periodof time, popularizes scientific topics to wider audiences, and encourages the formation ofamateur scientific communities, which initiate their own projects and deliver notable results.
The seminar was arranged as a platform to discuss and explore the aspects of and tooutline the future of the citizen science research, platforms, and applications. The seminarwas organized around to the following three perspectives:1. A crowdsourcing perspective that views citizen science as a large-scale volunteer-driven
human computation system. Relevant aspects include:Task and workflow designUser experience designAnswer validationTask assignment and contributor performanceCrowd learning, feedback, and tutorialsGamification and rewards
2. An online community perspective that considers social and other communication andinteraction activities that support task-centered efforts. This is related to quantitativeand qualitative approaches for content and community analysis, including:
Analysis of discussion forum and chat activitySocial network analysisInterplay with other community spaces such as social mediaAnalysis of community trajectoriesLurker behavior and more general contribution patternsConflict and collaborationSurveys of motivation and incentives
3. An open science perspective that focuses on citizen science as an emerging model ofcollaborative research. In particular:
Open, participatory approaches to all stages of the scientific lifecycleCrowdfunding for scienceOpen access publishing of research ideas and outcomesOpenness in data acquisition and sharingParticipation of young volunteers in citizen science activitiesScientific publishing for crowdsourced science
In order to discuss the design and engagement of citizen science, the workshop consistedof a number of talks and working groups that incorporated these differing perspectivesthroughout.
17272
26 17272 – Citizen Science: Design and Engagement
4 Overview of Keynote Talks
4.1 Crowdsourcing for Smart Cultural Heritage: Harnessing HumanSemantics at Scale
Lora Aroyo (Vrije Universiteit Amsterdam, NL)
License Creative Commons BY 3.0 Unported license© Lora Aroyo
The state of the art in machine learning and information extraction has advanced thedetection and recognition of concepts and objects, like people, locations, and various othertypes of named entities. However, still there is various types of human knowledge thatcannot yet be captured by machines, especially when dealing with wide ranges of real-worldtasks and contexts. The key scientific challenge is to provide an approach to capturinghuman knowledge in a way that is scalable and adequate to real-world needs. HumanComputation has begun to scientifically study how human intelligence at scale can be usedto methodologically improve machine-based knowledge and data management. My researchfocuses on understanding human computation for improving how machine-based systemscan acquire, capture and harness human knowledge and thus become even more intelligent.In this talk I will present use cases related to smart culture, e.g. enrichment of culturalheritage collections of artworks, videos, newspapers, etc. I will show how the CrowdTruthcrowdsourcing framework http://crowdtruth.org facilitates data collection, processing andanalytics of human computation knowledge. Processing real-world data with the crowd leavesone thing absolutely clear - there is no single notion of truth, but rather a spectrum thathas to account for context, opinions, perspectives and shades of grey. CrowdTruth is a newframework for processing of human semantics drawn more from the notion of consensus thenfrom set theory.
4.2 Open Citizen Science: What, how and with whom?Claudia Göbel (Museum für Naturkunde Berlin/ European Citizen Science Association, DE)
License Creative Commons BY 3.0 Unported license© Claudia Göbel
The presentation focused on unpacking relations of open and collaborative aspects in CitizenScience. On a conceptual level, I identified synergies and tensions between Citizen Scienceand Open Science by mapping agendas from the European research policy discourse againsteach other. What is done in Citizen Science practice was explored by looking at findings ofan international stakeholder analysis on Citizen Science data interoperability and examplesfrom other areas of Open Science. Finally, I sketched some cornerstones of an analyticalperspective focusing on stakeholder networks and organizations for further analysis.
Irene Celino, Oscar Corcho, Franz Hölker, and Elena Simperl 27
4.3 The state of the art of OpenStreetMap: technologies, communityand research challenges
Maurizio Napolitano (FBK - Centre for Information Technology/ Open Street Map, IT)
License Creative Commons BY 3.0 Unported license© Maurizio Napolitano
OpenStreetMap - OSM is known as the free world map created on a voluntary basis. ButOSM is not just a map, it is much more: one of the biggest geo-referenced open data resources,a community of people who are able to give voice to a territory, an important resource forentrepreneurial initiatives, and much more. This talk introduced the project from its historyand technological aspects in order to highlight what challenges science could help solvingand the benefits deriving from it.
5 Overview of Tutorials
5.1 Citizen Science as a New Way To Do ScienceMarisa Ponti (University of Göteborg, SE)
License Creative Commons BY 3.0 Unported license© Marisa Ponti
Main reference Marisa Ponti, “Citizen Science as a New Way To Do Science”, SocArXiv, 2017.URL http://dx.doi.org/10.17605/OSF.IO/KGXSQ
Citizen science has received increasing attention because of its potential as a cost-effectivemethod of gathering massive data sets and as a way of bridging the intellectual divide betweenlayperson and scientists. Citizen science is not a new phenomenon, but is implemented in newways in the digital age, offering opportunities to shape new interactions between volunteers,scientists and other stakeholders, including policymakers. Arguably, citizen science restson two main pillars: openness and participation. However, openness remains unexploitedif we do not create the technical and social conditions for broader participation in morecollaborative citizen science projects, beyond collecting and sharing data with scientists.“Public participation” has too often accounted for the assumed ease with which hierarchiesin science can be horizontalized, and economic and geographic barriers can be removed.However, public participation is a contested term, which should be problematized. TheScandinavian tradition of participatory design can help explore conceptually the challengesrelated to participation and to design for participation.
5.2 Citizen Science as a Social MachineElena Simperl (University of Southampton, GB)
License Creative Commons BY 3.0 Unported license© Elena Simperl
Tim Berners-Lee envisaged the web as a platform for large-scale participation. Social machinesare socio-technical constructs that enable all this. They define and support processes wherepeople contribute creative work and algorithms carry out the more routine, predictable or“engineer-able” tasks to bring the results of that work together. Citizen science is itself an
17272
28 17272 – Citizen Science: Design and Engagement
example of a social machine: bringing groups and communities of volunteers together to helpadvance science by collecting or analysing data relevant to scientific experiments. To date,lots of the research has focused on showcasing how and where citizen science can help orstudying specific properties of the systems using a variety of methods from social computing,HCI, or scientific communication. We need to appreciate that citizen science can be studiedfrom several angles, and that more research is needed in understanding how citizen science canmove away from being just an approach useful for the professional scientist to a fully-fledgedsocial machine. In the same time, we need to design better tools and teach the scientiststo understand how to improve and maintain community health and determine whether thevolunteers are happy and producing enough data. This talk introduced a number of researchquestions and solutions, such as how to design platforms for meaningful contributions, how tomanage the data produced by the volunteers, studying emerging communities, and designingincentives for participation. The ultimate question is ‘what makes citizen science successful?’and we need a interdisciplinary, inclusive approach to solve it.
6 Overview of Working groups
Working groups were held on days 1 and 3 of the workshop in order to facilitate discussionson topics related to the areas outlined above. In the following sections we provide briefsummaries of the main outputs of these working groups.
6.1 Working Group One - SustainabilityReported by Alessandro Bozzon. This group discussed various dimensions of sustainabilityrelated to citizen science. These are presented in the table below, along with a series ofchallenges for each that must be overcome in order to ensure sustainable projects andpractices. In addition the group discussed a number of developments that may help toincrease sustainability, such as a ‘graveyard’ of projects that have run their course, addressedtheir hypothesis and wrapped up engagements with their community, along with a repositoryof workflows that have been annotated with what problems they’ve worked well on previously.This would allow other citizen science teams to see what has worked, and build from there.Other suggestions were a high-school level curriculum module on citizen science, to increaseawareness of this from a younger age, and a re-usable set of personas to help in the designprocess.
1. Openness and Reuse of DataData and Metadata
Incentivising data sharingDealing with diverse standardsLicensingProvenance and tracking
TechnologyCreating a Github for open citizen science hardware and softwareDesigning projects for re-useDescribing workflowsStandardisationDocument deployment configurations and conditions
Irene Celino, Oscar Corcho, Franz Hölker, and Elena Simperl 29
Discovering existing initiativesCreating a citizen science directoryhttp://firstmonday.org/ojs/index.php/fm/article/view/5520/4194https://scistarter.com/http://vgibox.eu/https://www.citizensciencealliance.org/https://en.wikipedia.org/wiki/List_of_citizen_science_projects
MethodologyCo-creation and knowledge sharing for project design and implementationPeopleIncentivesTasksTrade-offsLessons LearnedFor example, repository of case studies, personas, design guidelines, etc.
2. Socio-economic aspectsEngagement/Participation
Limited number of potential participantsReaching motivated participants
Fitness for purpose/availabilityEvidence of required expertiseTrustworthinessSpatial and temporal activity patternsResponsiveness
Agency/EmpowermentCommunity tools for building citizen scienceCo-creation of projects and research
EducationAccepted as curriculumHands-on and built on practiceRevisit pedagogical approaches
Social and Ethical goalsEncourage empowerment of under-represented groups (e.g., migrants)Increase enthusiasm for science in lay public
3. EcosystemicsProject Variety
Ensuring ecosystem is fit for different timescales and project culturesProject Lifecycle
Funding - connecting funding for infrastructure and for doing scienceEnd of Life - data archival, communities and connectionsCreating bridges to other projects
17272
30 17272 – Citizen Science: Design and Engagement
6.2 Working Group Two - Measuring SuccessReported by Paul Groth. The success working group discussed a range of topics such asinvestigating what constitute ‘success’ in a citizen science context, the different types ofoutcome, and different frameworks for assessing various facets of success. The group thenmoved on to outline a range of grand challenges for this topic area. Each challenge wassummarised with an impact statement of where, ideally, the state of play would be in tenyears time.
Predict the success of a project before starting, based on a manual of best practices.For this challenge, the different criteria for different stakeholder groups were discussed, alongwith what features can be looked at to understand success. The scientists themselves, funders,and citizens may all have differing expectations about what success means in a citizen scienceproject, and going even further there are planetary scale stakeholders that are concernedwith the positive ecological impacts that many of these initiatives may have. Success may bemeasured using a huge rage of metrics and features, going beyond measures of the numberof classifications or volunteers to look at how many visitors the project has attracted, thenumber of scientists and developers engaged in the process, the quality of the input data,the number of countries reached, etc. As a grand vision, in ten years time, we will be able topredict successful citizen science projects and tell you why.
Expand the diversity of contributors to citizen science. In order for citizen science as anapproach to be truly successful, there needs to be a diverse range of volunteers who participatein projects to ensure that there is no cultural, socio-economic, age, educational or languagebias recorded in the data. We identified a number of steps required such as identifying whatthe current bias is, and eliciting methods of motivating more people to participate, fromdifferent backgrounds. There were also discussions around what automatic adjustments todesign could be made so that an existing project or tasks could easily be replicated or madesuitable for a different demographic group. The proposed impact statement for this groupwas: in ten years time, we will have citizen science projects that reflect the demographicbreakdown of the world.
Radically scaling up co-created citizen science. Citizen science already reaches largenumbers of volunteers in some cases, with this an essential aspect of many projects. Howeverthe role that the volunteers play in the design process is less established, and one of theareas of further potential is a true collaborative, co-creation process at all stages of thescientific process (as opposed to the majority of current projects that incorporate volunteerinvolvement in the data collection and/or analysis phase). Foundational work needs to becarried out to understand what is currently done in co-creation process so that experiencesand lessons learned may be documented. A testing and development process for componentsof the scientific method workflow will then allow the creation of processes that work acrossmultiple scientific domains, before over time things are scaled up so that performance may betested against a baseline to determine the successfulness of running truly co-created citizenscience projects. If it is possible for a completely citizen-led project to match or even beatthe performance of professional scientists’ own research then citizen science will have reacheda high level of success as an approach to executing the scientific method. In ten years time,a citizen science platform with over 100,000 people will beat the performance of professionalscientists for the full scientific process.
Irene Celino, Oscar Corcho, Franz Hölker, and Elena Simperl 31
6.3 Working Group Three - Community EngagementReported by Oscar Corcho and Christopher Kyba. Community engagement is crucial in orderto ensure that volunteers continue to contribute to a project, as well as return to both thecurrent project and future projects that may be of interest to them. This working groupinvestigated two challenges as outlined below.
A framework and contextual conditions for citizen science projects
The current status-quo. Funding agencies have not adapted their funding strategies/callsto account for applying Citizen Science in any type of project. Either a project is submittedas a specific Citizen Science project for a Citizen Science call, or if it is a general project callthen there is no clear space for the funding characteristics of Citizen Science projects.
In some cases (e.g., some Ministries in Germany), there are no possibilities of havingfollow-up projects, and for a successful citizen science project this may not be adequate(you make a community orphan). The usual term for a research project (3 years) is notenough for building a successful community (which requires between 3 and 6 years).Private and public foundations (e.g., Wellcome Trust, Knight Foundation, World Devel-opment Bank, UN) in some countries are providing some smaller funding for this type ofinitiatives.There is not infrastructure-type of funding for Citizen Science projects, as it is done forother types of large-scale research infrastructure-related projects (e.g., ESFRI projects).
There are costs associated to Citizen Science projects that are not easy to claim for (e.g.,creating a good brochure to reach a large set of citizens - 1 month of work instead of a coupleof days needed for a scientific community -, to do scientific outreach/transfer) or that arenot easy to get into the accounts of scientific organisations (e.g., problems with audits). Howdo you pay freelancers or crowdworkers?
Citizen Science projects are generally more risky (e.g., engagement with people maynot work and they may not show up). We should be also more open to the fact thatexperiments may generate less (or more) data than originally expected, or with a differentquality to the original expectations. So there must be more room for flexibility (even changingthe hypotheses during the project execution), and some more knowledge inside researchorganisations of the characteristics of Citizen Science projects.
Research organisations have already some budget for scientific outreach (e.g., open days,education activities for secondary schools). However, Citizen Science projects are not normallyconsidered as fundable under this budget, even if they cover some of these needs (creatingawareness about Science for Society), but only fundable under the budgets for researchprojects.
Challenges. The following challenges were identified as necessary to overcome in order toadvance this aspect of citizen science research:
Change the funding (and accounting) mechanisms from funding agencies to adapt betterto the funding needs for Citizen Science projects (e.g., to be more flexible)Expand the collection of organisations that can provide funding for these projects byproviding a better explanation of what is done in Citizen Science projects and theirpotential benefitsConnect Citizen Science to evidence-based decision making (e.g., fighting against fakenews in newspapers)
17272
32 17272 – Citizen Science: Design and Engagement
Create Citizen Science infrastructure helpdesks inside research organisations or at thenational/international level.Create specific budgets inside research organisations for funding Citizen Science projects.Adapt administrative procedures - e.g. possibilities of paying volunteers for contributionsor participation in the project; modalities of payment (paying upfront)Appoint Citizen Science coordinators in research organisations, so that they can helpscientists to contact and engage citizens.
Actions. In the short term, we need to create a clear set of administrative proceduresthat can be applied by research institutions when paying volunteers for their contributions.To support this, helpdesks/support units could be created for Citizen Science projects atdifferent levels, including online training and appointing Citizen Science coordinators.
In the medium term, there needs to be influence on public funding bodies in orderto incorporate Citizen Science into their usual funding streams. In order to drive this,foundations should be engaged to disseminate information about the opportunities thatcitizen science can offer, and alongside this we should exploit the social corporate responsibilityof many large companies as they may become potential funders. Future research shouldexplore how citizen science could fit new business models for publishers, and there shouldbe work to create budgets within organisations for funding citizen science projects. Finally,citizen science project involvement should be aligned to university curricula so that (forexample) students can participate in projects and earn credits towards their degree.
Longer term, the group discussed the creation of citizen science ‘wallets’ where volunteerscould collect remuneration for their contributions and develop a record of how they’ve helpedin various science projects. Additionally, there is the need to create an ESFRI-like structurefor large scale projects.
Upscaling and diversity
The current status-quo. As discussed in the diversity section of the ‘Success’ workinggroup, citizen science project participants are not currently representative of societies ingeneral. They are often drawn from higher educated groups. To some extent this reflectsself-selection, but this may be because of lack of awareness in other demographic groups. Notall projects collect participant information, so we don’t necessarily have the full picture ofwho is participating and aggregated statistical information about participant characteristicsis not currently available or accessible. Many citizens don’t realise that they could be involvedin science projects such as these.
Furthermore, in some projects the data often has very strong clustering - with someareas having a lot of data and others having little or none. Participatory approaches have avery strong bias towards Europe and North America and reflect cultural differences: withinEurope there is considerable diversity in the tools that are available, and approaches such asbioblitz [1], (popular in e.g. Germany and UK, but not taking place in e.g. Spain). Thereare cultural differences particularly between Eastern and Western European countries withregards to participatory research. Even within countries, there are differences in participationrates among the population. In Spain for example, participation varies strongly betweennorthern and southern Spain. Language creates a barrier for expanding participation acrossEuropean countries and also to countries outside of Europe.
Research questions in projects that are labeled Citizen Science are far more commonlyinitiated by academic researchers, rather that co-created although this is less of the casein “digital social innovation”. Citizen science projects are however rarely being initiated by
Irene Celino, Oscar Corcho, Franz Hölker, and Elena Simperl 33
citizens directly. Not all projects are being funded with a clear “closing strategy” for whenthe project will be handed off to e.g. a foundation, or how it will be properly ended (e.g.final communications, publications, etc.). As such, there is currently duplication of effortaround these areas.
Individual citizen scientists are very seldomly represented at discussions of citizen scienceAccess to paywalled articles is strongly differentiated by classParticipants are not necessarily getting enough feedback and education from the projectsthey participate in: missing a chance to expand understanding of critical thinking
Challenges. Subsequently, there are a number of challenges in terms of upscaling andincreasing diversity in projects:
Attraction of participants: how do you get them to know about your project?Overcoming language barriersThere is no one “European media”, so getting messages out is difficultFinding translators and funding for citizen science projects (apps, lesson plans, officialcommunication and feedback)How do you find “gatekeepers” who can promote your project to members of theirgroup?
Community management is very difficult with an international participant groupThere can also be cultural issues in how to communicate (e.g. with students versuselderly people)
If you want to expand a project into a neighboring country, how do you find who youneed to contact?There is a lack of funding for community management, funders don’t realize that the truecost of a project is much larger than the cost of an app/platform.Citizen science funding is often put out in specific calls, and not more generally acceptedwithin disciplinary funding programsHow could you allow for citizen project initiation, while still dealing with questions offunding oversight?
Actions. To address these challenges, we first need to provide tools and resources to citizenscientists so that they can participate more easily in projects. Funding needs to be allocated tocommunity management and towards a strong communication strategy in order to draw in amore diverse set of participants. To help with this further, online science literacy classes couldbe provided to give volunteers the skills they need in order to help out, while seminars couldbe given to scientists on how to interact with citizen volunteers. One larger-scale suggestionwas to provide European funding for a citizen scientists conference where participants ofprojects can attend if they have demonstrated a level of participation in a project. Therealso needs to be efforts to share data among projects, and disseminate different engagementstrategies.
In the medium term, this should go further and demographic information needs to beshared across projects so that we have a better understanding of what communities arebeing engaged. Citizen science practitioners also need to be up-to-date with what digitaltransformations are impacting citizen science.
Finally, longer term, there needs to be open access to research papers so that all interestedparties can find out about the research opportunities and outcomes in citizen science. Wediscussed the idea of making citizen science part of the junior high school curriculum so thateveryone becomes aware of its existence and so that students have the option to participatethroughout their lives.
17272
34 17272 – Citizen Science: Design and Engagement
6.4 Working Group Four - LinkingReported by Andrea Wiggins. The linking group discussed the complexities involved in linkingcommunities across citizen science projects. Currently, many projects want to link andconnect with others in order to enhance the wider citizen science network. However there arecurrently no incentives for such networking and it is not seen as innovation in citizen sciencedesign. There is little funding available for these activities and therefore most attemptsgenerally start from the beginning rather than building on the records of previous attempts.
In the short term, the idea of networking communities within citizen science shouldbe made clear to all projects so that existing and established communities may be usedrather than always creating new ones. This should be detailed in the early stages of projectformation, including the grant-writing process and project planning. There should also be anemphasis on sharing technology including the tools, user interfaces, data and profiles usedwithin projects, and technologies such as OpenID that are easy to use and already have welldocumented APIs should be adopted.
In the longer term, there needs to be greater effort to network co-ordinators together.There needs to be an increase in funding allocated towards linking communities, consistingof tasks such as administration, arrangement of workshops and planning strategies. Thereshould also be specific incentivisation mechanisms for linking so that this is not just tackedon to a project but instead planned appropriately. In order to encourage this, a change isrequired at the grant application stage so that policies and funding opportunities encouragelinking to be included in project proposal and if this can be promoted internationally forcross-border sharing then the potential of citizen science will be realised faster.
6.5 Working Group Five - QualityReported by Elena Simperl. Data quality is a matter of primary concern for researchers whouse citizen science. There are generally three stages at which data quality typically becomesa concern: the input data (e.g. the collection of objects or the results of a machine learningclassifier), the data collected or produced by citizens, and the final results of the task. Thisworking group considered mechanisms that projects employ to ensure quality in the dataproduced by these projects, and discussed frameworks and mechanisms for managing these -in particular a framework of 18 mechanisms presented by Wiggins et al. (2011) that are basedon direct experience and existing surveys [3]. The working group took into considerationsources of error in the data, as well as different stages of the citizen science project processwhich may introduce such errors. Using this as a starting point for our discussions, weconsidered what the current debate in this area should be in citizen science.
There are different aspects of data quality that need to be considered. Relevance andtrustworthiness are required in order to ensure that valid and reliable data has been collected.There then needs to be a representative presence covered by the data, which may be throughthe density of coverage (e.g. spatially). There also needs to be information to ensure that weknow the degree to which a volunteer followed the research protocol - and therefore thereneeds to be a process of recording this so that it can be assessed. Finally, the data should bereproducible so that the same distributions of data can be obtained if the same process isfollowed again.
Common standards may help with ensuring a base level of quality across projects, butit is likely that there are too many different project types in order to develop a reliablestandard in this area. As such it may be more appropriate to develop standards around the
Irene Celino, Oscar Corcho, Franz Hölker, and Elena Simperl 35
tools, question types or methods used, and to also consider the domain specific concernsinvolved in specific projects as many of these already have existing standards. While some ofthese may be shared between domains and projects, there may also exist a conflict betweenwhat constitutes ’good‘ data.
Because of the complexity of devising a common benchmark for citizen science dataquality, a number of issues were listed as things that need to considered in order to proceed.In terms of data quality, it must be established exactly how citizen science differs fromconventional science - and whether or not this requires a new way of thinking compared totraditional approaches. Furthermore, the different technologies and tools used by differentprojects will introduce unique data quality problems, and therefore an understanding of theseneeds to be shared so that project designers may anticipate and prevent potential problems.Noisy data may be caused by inconsistent or inaccurate sensors (either physical, or human),or through ambiguity and bias in the process. This may originate from the task design, wherespecific labelling requirements or semantics mean that bias is introduced into the process. Itis therefore crucial to understand these so that future projects can be designed to alleviatethese problems. Because of the impact that the citizens themselves have on the data, wemust also consider the impact from using different participatory or engagement frameworks,and how these in turn determine the resulting data quality.
6.6 Working Group Six - Manifesto for Citizen ScienceReported by Paul Groth, Edited by Neal Reeves A final working group developed a manifesto forthe future of Citizen Science, intended to support different stakeholders in producing projectsand in publishing scientific results. This document can be viewed at https://goo.gl/ojuZiR.
References1 Cathy Lundmark. BioBlitz: Getting into Backyard Biodiversity. BioScience, JSTOR, 53;4,
329–329, 2003.2 Alexandra Swanson, Margaret Kosmala, Chriss Lintott, Robert Simpson, Arfon Smith and
Craig Packer. Snapshot Serengeti, High-Frequency Annotated Camera Trap Images of 40Mammalian Species in an African Savanna. Scientific Data, Nature Publishing Group, 2,150026, 2015.
3 Andrea Wiggins, Greg Newman, Robert D. Stevenson and Kevin Crowston. Mechanismsfor Data Quality and Validation in Citizen Science. Proceedings of the 2011 IEEE SeventhInternational Conference on e-Science Workshops (ESCIENCEW ’11), IEEE ComputerSociety, Washington, DC, USA, 14-19, 2011.
17272
36 17272 – Citizen Science: Design and Engagement
Participants
Lora AroyoFree University Amsterdam, NL
Alessandro BozzonTU – Delft, NL
Maria Antonia BrovelliPolytechnic University ofMilan, IT
Irene CelinoCEFRIEL – Milan, IT
Oscar CorchoPolytechnic University ofMadrid, ES
Dominic di FranzoCornell University – Ithaca, US
Claudia GöbelMuseum für Naturkunde –Berlin, DE
Esteban González GuardiaTechnical University ofMadrid, ES
Paul GrothElsevier Labs – Amsterdam, NL
Lynda HardmanCWI – Amsterdam, NL
Franz HölkerIGB – Berlin, DE
Tomi KauppinenAalto University, FI
Christopher KybaGFZ – Potsdam, DE
Dave Murray-RustUniversity of Edinburgh, GB
Maurizio NapolitanoBruno Kessler Foundation –Trento, IT
Jasminko NovakHochschule Stralsund, DE
Christopher PhetheanUniversity of Southampton, GB
Marisa PontiUniversity of Göteborg, SE
Lisa PoschGESIS - Cologne, DE
Gloria Re CalegariCEFRIEL – Milan, IT
Neal ReevesUniversity of Southampton, GB
Sven SchadeEC Joint Research Centre –Ispra, IT
Sibylle SchroerIGB – Berlin, DE
Elena SimperlUniversity of Southampton, GB
Alice VerioliBSDesign – Milan, IT
Christopher A. WeltyGoogle – New York, US
Andrea WigginsUniversity of Nebraska –Omaha, US
Amrapali ZaveriMaastricht University, NL
Irene Celino, Oscar Corcho, Franz Hölker, and Elena Simperl 37
8 Appendix A - Lightning Talk Slides
Alessandro Bozzon, TU Delft
Christopher Kyba, GFZ - Potsdam
17272
38 17272 – Citizen Science: Design and Engagement
Maria Antonia Brovelli, Polytechnic University of Milan
Lisa Posch, GESIS - Köln
Irene Celino, Oscar Corcho, Franz Hölker, and Elena Simperl 39
Sven Schade, EC Joint Research Centre
Sibylle Schroer, IGB - Berlin
17272
40 17272 – Citizen Science: Design and Engagement
Andrea Wiggins, University of Nebraska
Amrapali Zaveri, Maastricht University
Tomi Kauppinen, Aalto University
Irene Celino, Oscar Corcho, Franz Hölker, and Elena Simperl 41
Neal Reeves, University of Southampton
Irene Celino, CEFRIEL
Gloria Re Calegari, CEFRIEL
17272
42 17272 – Citizen Science: Design and Engagement
Dave Murray-Rust, University of Edinburgh
Esteban González Guardia, Technical University of Madrid
Irene Celino, Oscar Corcho, Franz Hölker, and Elena Simperl 43
Chris Phethean, University of Southampton
Jasminko Novak, Hochschule Stralsund
17272
