Environmental Science & Policy 66 (2016) 47–61

QUICKScan as a quick and participatory methodology for problemidentification and scoping in policy processes

Peter Verweij*, Sander Janssen, Leon Braat, Michiel van Eupen, Marta Pérez Soba,Manuel Winograd, Wim de Winter, Anouk CormontWageningen University and Research Centre, Droevendaalsesteeg 3, 6708 PB Wageningen, The Netherlands, The Netherlands

A R T I C L E I N F O

Article history:Received 10 December 2015Received in revised form 14 June 2016Accepted 20 July 2016Available online 16 August 2016

Keywords:ParticipatoryDecision makingTrade-offEnvironmental policySpatial planningImpact assessment

A B S T R A C T

Policy making is required in cases in which a public good needs to be either maintained or created, andprivate or civil initiatives cannot deal alone with this. Policy making thus starts with a phase of problemidentification and determining whether there is a problem that needs to be dealt with. Rapidly evolvingcontexts exert influence on policy makers who have to take decisions much faster and more accuratelythan in the past, also facing greater complexity. There is a need for a method that lowers the lead time ofthe exploratory phase of the policy cycle. At the same time the method should create a jointunderstanding of the most important interactions. This paper proposes QUICKScan, a method, processand spatially explicit tool, to jointly scope policy problems in a participatory setting, investigate the mostimportant interactions and feedbacks and assesses the state of knowledge and data of relevance to theproblem. QUICKScan uses strongly moderated participatory workshops bringing together a wide range ofstakeholders relevant to the policy issue. These moderated workshops jointly build an expert system in aspatially explicit tool using functionality of bayesian belief networks, python programming, simple mapalgebra and knowledge matrices, with a strong focus on visualization of results. QUICKScan has beenapplied in 70 different applications in a range of different policy contexts, stakeholders and physicallocations. Through these applications participants were able to internalize the knowledge that wasusually handed to them in briefs and reports, to develop a joint understanding of the main interactionsand their link to impacts and to develop a problem statement and solution space in a reduced lead time.Ultimately, QUICKScan demonstrates another role of science, not solely as a knowledge production, butalso facilitating the knowledge consumption.

ã 2016 Elsevier Ltd. All rights reserved.

Contents lists available at ScienceDirect

Environmental Science & Policy

journal homepage: www.elsevier .com/ locat e/e nvsci

1. Introduction

It has become clear that it is extremely difficult to have societaland economic development without compromising environmentalsustainability, which is the eco-social system that humanitymaintains and depends upon (PEER, 2010). Drivers of change, suchas demographic development, resource depletion, loss of ecosys-tem services, natural hazards and climate change have becomethreats to social and policy issues such as water- and food security,social wellbeing, energy security and a prosperous economy (United Nations, 2014). The spatial distribution, scale and com-plexity of the interactions between these issues and driversrepresent a challenge for policy makers, spatial planners,researchers and the public at large. While the scientific community

* Corresponding author.E-mail address: peter.verweij@wur.nl (P. Verweij).

http://dx.doi.org/10.1016/j.envsci.2016.07.0101462-9011/ã 2016 Elsevier Ltd. All rights reserved.

tries to find testable explanations between drivers and issues, thepublic sector sets societal goals such as sustainable development,nature conservation and environmental quality. Spatial plannersorganize the distribution of human activities across territories ofdifferent scales according to an overall strategy (United Nations,1987). It is the role of policy makers at different levels ofgovernment to facilitate and encourage mitigation, adaptation andprepare for likely changes by achieving the level of transparencyneeded to obtain the public support for taking far reachingmeasures. For both it is a challenge to formulate initiatives whichbring together as many, often conflicting, interests as achievable.

Policy making is required in cases in which a public good needsto be either maintained or created, and private or civil initiativescannot deal (alone) with this. Policy making is typicallyconceptualized as a cyclical process (Fig. 1), that goes throughdifferent stages of analysis, design of policy options, implementa-tion and review (Zamparutti et al., 2012; Jansen et al., 2007;Winsemius, 1989). Especially in the first stages of problem

http://crossmark.crossref.org/dialog/?doi=10.1016/j.envsci.2016.07.010&domain=pdfmailto:peter.verweij@wur.nlhttp://dx.doi.org/10.1016/j.envsci.2016.07.010http://dx.doi.org/10.1016/j.envsci.2016.07.010http://www.sciencedirect.com/science/journal/14629011www.elsevier.com/locate/envsci

Fig. 1. Policy cycle of dealing with a problem.

48 P. Verweij et al. / Environmental Science & Policy 66 (2016) 47–61

identification, evidence gathering and design of different options,science has a role to play, and is traditionally seen as the supplier ofevidence (Gibbons et al., 1994; Sterk et al., 2009), that can then beconsumed by policy makers.

As an example of a step in the policy cycle and its relation toevidence, Impact Assessment (IA) is a decision support method toensure that sustainability concerns are taken into consideration byidentifying a problem, setting an objective and choosing betweenalternative options to reach that objective. An evidence based IA isbecoming increasingly important in societal decision making andpolicy development (Turnpenny et al., 2009). It enables policymakers, decision makers and spatial planners to maximize benefitsto society and minimize unwanted side-effects. The analysisshould cover the impacts in the targeted domain and regions, aswell as unintended impacts, side effects and trade-offs in adjacentdomains and regions.

Rapidly evolving contexts exert influence on policy makers whohave to take decisions much faster and more accurately than in thepast. Current practise of IA is often found to be ‘an expensive andtime consuming regulatory hurdle’ (Pope et al., 2013), while alsomethods of evidence provisioning in science through modelling orexperimental work are time and resource intensive. Often by thetime the evidence is produced through scientific methods, the(policy) context has changed, and is concerned with other items(Adelle et al., 2012). “Increasingly science is expected to supportdecisions by providing urgent answers to complex, uncertainquestions. Typical complaints are that science takes too long, orprovides unreliable answers that turn out to contradict stake-holders’ experiences resulting in stakeholder disappointment.Stakeholders must necessarily work together to define the rightquestion, and delineate how approximate the answer can be, andstill be useful. Scientists must define how vague the question can

be, and still be studied. Both require certainty � of expectations fora given question, and of reliability of the answer (contingent oncurrent understanding)” (Guillaume and Jakeman, 2012). Wherethe integral character of policy making and planning hampers aresponsive adaptation to new circumstances a demand for moreagility exists. Especially steps requiring ‘scientific evidence’ and‘consultation with external stakeholders’ need to be streamlinedinto the process.While policies are often conceived on the basis ofcurrent trends, there is a growing need to improve anticipatorythinking to capture both the future risks and opportunities(European Commission, 2013a,b).

In response to the demand for shorter lead times and more agility,scientific methods have been developed for the early phases inpolicy making and spatial planning, which are exploratory by nature.In these phases, problems and stakeholders are identified, objectivesare set and alternative options (i.e. scenarios, (spatial) strategies)defined. Scientific methods available in the exploratory phase areexpert groups (European Commission, 2010), Rapid (Participatory)Appraisal (McCracken et al., 1988; Ison and Ampt, 1992), qualitativedeliberative participatory methods (Davies and Dwyer, 2008),preference elicitation (Kodikara et al., 2010; Aloysius et al., 2006)or fuzzy cognitive mapping (Kosko, 1986; Jetter and Kok, 2014).These methods result in storylines, preference functions, scoretables, or concept maps showing linkages and directions of influencebetween major problems, drivers, valuations and other concepts.However, additional steps such as modelling are required to quantifyimpacts and use those to iterate, fine tune or improve preferences,options and storylines. Ideally, this would be done during theparticipatorysessions, resulting in an understandingof the influenceof key drivers on key outputs as perceived by the stakeholdersengaged in the participatory process. Thus, there is a need for amethod that lowers the lead time of the exploratory phase of the

P. Verweij et al. / Environmental Science & Policy 66 (2016) 47–61 49

policy cycle and that results in a joint understanding of the mostimportant interactions in a participatory setting, as a way of capacitybuilding across actors.

This paper introduces a method, process and spatially explicitmapping and assessment tool, named QUICKScan, to jointly scopepolicy problems in a participatory setting, investigate the mostimportant interactions and feedbacks and assesses the state ofknowledge and data of relevance to the problem (see Fig. 1). Thepaper demonstrates the usability and usefulness of the QUICKScanthrough an overview of a large number of applications withdifferent policy contexts and questions considered across a rangeof spatial and temporal scales.

2. Methods

2.1. Overview

QUICKScan is a participatory modelling method (KorfMacher,2001; Voinov and Brown Gaddis, 2008) that links stakeholder- anddecision maker knowledge and preferences to available spatial-and spatio-statistical data, and is designed for group use, e.g. in amulti-stakeholder workshop setting.

During suchworkshops aniterativeapproach is followed,startingwith simple (knowledge-based) rules (equations) and step-by-stepadding complexity, using the participants’ interpretation of model-results. Results are visualized in interactive maps (McCall, 2003;Jankowski, 2009), and summary charts and trade-off diagrams.Successive iterations are used to 1) improve the quality of the model,2) tryoutalternative(spatial)plans andpolicyoptionsand,3) includedifferent stakeholder values and perspectives.

Fig. 2. –Sequence of QUICKScan phases: scoping, preparation, workshop and reporting. Tcomplexity. Several tools are used to support knowledge exchange between participan

Knowledge of the participants is captured in a computerprogram and encrypted in a conditional (e.g. ‘if A then B’), mostlyqualitative form, as is common in expert systems; humans tend torepresent their knowledge qualitatively rather than quantitatively(Newell and Simon, 1972) (e.g. ‘Mary is small, but Clarissa is smaller’as opposed to ‘Mary is 1.68 m and Clarissa is 1.62 m’). The computerprogram can show how a conclusion is reached by visualising thechain of knowledge and the data. The knowledge is separated fromthe reasoning and from the data on which it is applied.(Negnevitsky, 2002; Buchanan and Smith, 2003; Yuchuan Chenet al., 2012).

2.2. Process

Each QUICKScan follows a number of logical steps: scoping,workshop preparation, the workshop itself and reporting on resultsand observations (Fig. 2).

The scoping phase starts with clarifying the decision context(Gregory et al., 2012) and defining the objectives. It ends with theformulation of key questions by the client. Examples of keyquestions are: ‘what are Ecosystem service impacts of ecologicalreconstruction plans? Which are relevant ecosystem services? ’, or‘what management options are available for increasing agriculturalproduction? Which ones are acceptable? ’.

In the preparation phase participants are identified, evidenceand potential alternatives are gathered and data is collected. Thereare various techniques to identify participants. The choice of aspecific participant identification technique strongly depends onthe project context, the project phase and the available resources(Luyet et al., 2012). To ensure inclusion of all relevant stakeholders,

he workshop phase is characterised by many iterations in which each iteration addsts: whiteboard, post-it, flipchart, computer and video projector.

50 P. Verweij et al. / Environmental Science & Policy 66 (2016) 47–61

to avoid bias and to minimize the complexity of the process, weidentify key participants together with the donor and problemholder. Subsequently additional participants may be identified byconsulting the key stakeholders. Typically we aim at a variety ofparticipants including decision makers, interest groups, topicexperts and data experts that have different attitudes, conflictingperspectives, power, urgency and proximity to the key question(Mitchell et al., 1997).

Evidence is gathered by studying background information andinterviewing participants aiming at new ideas (Ampt and Ison,1989; Ison and Ampt, 1992). Together with the backgroundinformation these semi-structured interviews provide the basisfor data collection as the interviews provide insight on participantperspectives on criteria, consequences, trade-offs, alternatives,estimations and perceived values. Data may refer to bio-physical(e.g. soil, elevation), classified Remote Sensing (e.g. land cover),census data (e.g. population density), results from model runs (e.g.climate projections), or spatial plans. If required data are notavailable a proxy might be used (e.g. when in need of informationabout accessibility of forests, e.g. for timber harvesting, slope mayfunction as a proxy).

The workshop is setup following iterations of model conceptu-alisation, make stakeholder knowledge explicit, compute indicatorsand model evaluation based on the resulting indicators. Theevaluation is used to adapt the model in the successive iterations.

� Develop model concept—The participants jointly inventoryrelevant indicators, indicator metrics and alternatives. i.e. theindicator ‘timber production’ might be measured qualitatively interms of {low, medium, high}, or quantitative in tons/hectare/year. That indicator ‘timber production’ might be derivedfollowing different alternatives, such as: from land cover map,or from forest management, growing stock and forest type. Otheralternatives might include: timber production in the currentsituation and in a possible future (e.g. from spatial plans, orclimate projections); or compare different stakeholder perspec-tives.

� Make stakeholder knowledge explicit—The participants relateindicator concepts to available data by building a causal chain ofparticipants’ knowledge. Their knowledge can be a mix of formalscience, local and indigenous knowledge (Pert et al., 2015;Thaman et al., 2013), tacit knowledge, assumptions andperceived values.

� Compute indicators—The tool operator calculates indicator mapsand summary charts as requested by the participants (e.g.average per administrative unit, or trade-off of a number ofindicators per administrative unit).

� Evaluate—the participants evaluate the performance of theindicators in a single alternative, or evaluate the performanceof summaries of indicators across alternatives. The evaluationmight trigger another iteration in which participants identifyadditional indicators, perspectives and refining knowledge.

After the workshop has ended the results and the participants’evaluations are documented in a report to secure progress, andestablish agreements and disagreements.

2.3. People

Several people are involved in a QUICKScan workshop with thefollowing roles:

� Participants—decision makers, interest groups and topic experts.� Discussion facilitator—guiding the group with a focus on howthings are discussed and securing that tasks are done andspecified problems are addressed.

� Modeller—analysing the participants’ discussion, extracts spo-ken knowledge and transfers it into modelling terms.

� Computer program operator—puts modelling terms into thecomputer program and, initiates calculations, shows maps andsummary graphs, keeps it all organised and ensures everyparticipant understands the model. Often the role of operatorand modeller are combined in one individual.

2.4. The tool

The QUICKScan computer program encompasses a modellingenvironment that needs to be filled with spatial and statistical dataduring the preparation phase. The tool is not restricted to a specificgeographic location or spatial resolution. Knowledge rules,capturing participant knowledge, are used to combine data andderive indicators. Typically the rules use classifications to describequantitative data and typologies to give qualitative data meaning.Rules may be linked together to form a chain of rules. Alternative(chains of) rules are used to capture different options. Derived datafrom alternatives can be aggregated (e.g. by administrative units,or biophysical units such as catchments, or climatic zones) to bedisplayed in tables and charts for overviews (Fig. 3). Additionalfunctionality is listed in Table 1.

2.5. Tool development process

The development of the QUICKScan started with a scopingphase in which the strategic aims, short term objectives andboundaries were set. The development process focused on usersand their needs. User involvement was organised by identifyingseveral sounding boards in order to gain: mutual understanding,insights in the user needs and support from the targetedcommunities. The different sounding boards had different meetingfrequencies depending on their role.

The QUICKScan concept was shaped via one-on-one semi-structured interviews (Wilson, 2013) and workshops with thesounding boards. This conceptualisation phase resulted inguidance on the workshop process and a software concept interms of wire frames and a technical architecture. ‘Wire frames’ areprototypes addressing the layout of a screen and deal withinformation, structure, relationships between information andflow between screens (Verweij et al., 2014a).

The actual software development followed an agile approachwith a sequence of time-boxed activities: design, develop, test,deliver, elicit feedback and the planning for another iteration(Verweij et al., 2010a,b). After several iterations we’d built enoughfunctionality to start using it in actual workshops. Each workshopprovided insight on new software functionality to build anddeepened and broadened the guidance on the workshop process.

2.6. Approach to evaluate QUICKscan performance

The findings described in this paper are based on an analysis oftwo sources of information and data:

1 QUICKScan has been applied in a multitude of situations overthe past few years, all with some policy dimension and with adiversity of problems, options considered and spatial andtemporal scale. These applications were prepared and facilitatedby the author team and some others over the past years.Strategic reflection occurred with representatives of theEuropean Environment Agency over the years to specify thesteps in the process and organisation required to reach theexpected outcomes. All these applications represent a process oflearning by doing and gradual refinement of the approach.

Fig. 3. –Screen shot compilation of the QUICKScan tool. A typical QUICKScan exercise starts by populating the system’s data and rule library ‘10 with spatial and statistical datarelevant for the study (e.g. elevation and forest management). ‘20 is an example of an if.then.else rule defining potential timber production based on the growing stock andforest management. Data and rules are dragged onto the canvas and linked together forming a chain (see ‘30). Rules are applied to the data to create maps (‘40). Results ofalternative chains may be compared in aggregated bar charts (e.g. potential timber production profit per administrative unit, or climatic zone).

P. Verweij et al. / Environmental Science & Policy 66 (2016) 47–61 51

2 Next to these applications, feedback from participants wascollected after the workshops, in some cases in structuredformats, in other cases by discussion and reflection. Thefeedback from participants is summarised below to explainaspects of the functioning of QUICKScan and highlight strengthsand weaknesses. The feedback of participants has also been usedin improvements of the methodology.

3. Results

3.1. Overview

Since 2010 successive versions of QUICKScan have been appliedin approximately 70 workshops in 20 countries (see Annex 1), e.g.China, Romania, Darfur, Hungary, Brazil, France and theNetherlands. More than 40 were in a setting with 5–30participants. The remaining applications have been done by anindividual �usually scientist– as a desk study with regularconsultations with fellow scientists and/or stakeholders on resultsand modelling approach. The participatory workshops varied inturnaround time from 3 h to 25 days. The latter involved 5workshops of 5 days each with a time lag of 3 weeks betweeneach consecutive workshop. The shorter workshops were explor-ative, while the longer ones focused on getting more accuracy into

the assessment. Most of the workshops took a single day. Theapplication domain ranges from environmental planning, ecosys-tem service assessment, sustainable management, natural capitaland green infrastructure to crop production, water management,outdoor recreation, nature development, land use restoration andmineral exploitation. The scale of the applications varied from localto continental with a spatial resolution from 5 � 5 m2 to 1 �1 km2.Most applications have been carried out at regional, national andcontinental scales with a resolution ranging from 100 � 100 m2 to1 �1 km2. In the following paragraphs results from three differentworkshops are described that vary in objective, duration andnumber of participants.

3.2. Sample result 1, explorative assessment-potential timberproduction of France

In the context of the EU Biodiversity Strategy to 2020 MemberStates map and assess the state of ecosystems and their services(Braat and de Groot, 2012)in their national territory with theassistance of the Commission (European Commission, 2011) tohelp decide on what ecosystems to restore with priority where(Maes et al., 2013). 17 Member States were trained in mappingecosystem services (Braat et al., 2015; Pérez-Soba et al., 2015). The

Table 1–Listing of software functionality and its rationale.

Function Rationale description

Standardisation Bring all indicators in the same domain space. Standardize quantitative and ordinal data between 0.100.Spider diagram Trade-off analysis between indicators, alternatives

and regions.Display multiple indicators of multiple alternatives in a single spider. Eachindicator is standardised before display.

Linked maps Facilitate the visual comparison of severalindicator maps.

Show multiple driver and indicator maps in separate, but spatially synchronizedwindows. Zooming and panning in one map makes the other window follow.Moving your cursor on one window makes the cursor in the other maps follow.

Difference map Compare alternatives. Highlight the differences from two alternatives that specified the same indicator(e.g. different times, or with different assumptions).

Difference chart Compare alternatives. Show areal loss and gain between two alternatives.Bar chart Compare alternatives and regions. Show indicator scores summarised per spatial unit (e.g. administrative units) and

alternatives.Sustainability limits Show how sustainable a location, or spatial

aggregation is from a limit. Either below, or abovethe limit.

Sustainability limits include thresholds, standards and policy targets (Paracchiniet al., 2011). Limits can be defined per indicator and may vary per spatial unit (e.g.administrative unit, or biophysical stratification).

Weighted average Create a composite indicator (for Multi-CriteriaAnalysis).

Do a weighted sum on two or more indicators. The indicators are standardisedbefore summing them up.

Bayesian Belief Networks(Stelzenmueller et al., 2010;Haines-Young, 2011;Gret-Regamy et al., 2013)

Support reasoning with uncertainties. Include uncertainties in the knowledge rules and visualise the propagation the(un)certainties.

ArcPy Support map algebra (Burrough et al., 1998). A set-based algebraic language to manipulate geographic data, such assubtraction, multiplication, or shortest path analysis.

Tracing Model transparency. Clarify the causal pathwaysfrom drivers and (management) options to theimpacts.

From every location in an indicator map the chain of reasoning can be shownfollowing the chain of participant knowledge and data. The path of reasoning islocation specific. This tool is commonly used to iterate and tune specific causalrelationships.

52 P. Verweij et al. / Environmental Science & Policy 66 (2016) 47–61

description below illustrates the mapping of a single ecosystemservice by France.

During a three hours session a policymaker, an expert onEcosystem Services and a GIS data expert of France set out to mapestimates of ‘potential timber production’ supported by a QUICKS-can modeller. Initially they explored available maps of Franceaccompanied by storytelling to get a shared understanding of thelocation of forests, the circumstances under which they grew andthe earnings of selling the timber. Maps included: CORINE landcover (EEA, 2013), forest management (Hengeveld et al., 2012), theroad network for accessibility to harvest timber, and climate zones(Metzger et al., 2005) influencing growth rates.

The participants discussed the metric to use for measuring theamount of timber production, including ordinal qualities (‘a lot’,‘moderate’, ‘little’) and quantities in tons/hectare/year. Given theobjective, data availability and time availability they chose to usequantitative ranges expressed as ordinal qualities (‘3000 tons/hectare/year). Iteratively theparticipants developed four alternatives: 1. Map timber productiondirectly from CORINE land cover; 2. Map timber production basedon growing stock (EEA, 2014) and forest management; 3. Includeaccessibility using slope as proxy under the assumption that toosteep places are unfavourable to harvest; 4. Include tree species(Brus et al., 2012) to correct for species characteristics influencingthe extractable net timber. In the last alternative the averagespecies price per ton was used to calculate the profits peradministrative unit for all of France. Fig. 3 shows part of the rulesforming the model as created by the participants.

The participants assessed their modelled results positivelyusing their personal knowledge and official reports with statisticsper administrative units as comparison. The monetary valuationwas evaluated as a coarse proxy. The government officials clarifiedthat the experienced learning-by-doing (Gavrel et al., 2016)created a much deeper understanding than what they typicallyget from written, or spoken form. This workshop demonstratedhow the Member State can map ecosystem services to help decideon what ecosystems to restore with priority, and where. Theworkshop clarified the mapping expectations of the EuropeanCommission and it enabled the participants to produce additionalrequested maps independently.

3.3. Sample result 2, participatory model development-wetlandmanagement in the Chinese Yellow River delta

The Yellow River Delta (YRD) is located between Bo Sea Bay andLaizhou Bay in China. It is a delta with weak tide, much sedimenttransport, frequent displacements and forms the most completeand extensive young wetland ecological system in China. On theeast-Asian migration routes it offers breeding, wintering and stop-over places for many migratory birds, among which are very rarespecies like the Red-crowned crane and the Saunders’s gull. TheYRD is also an important base for aqua-culture and has beenappointed as national agricultural development area. The deltafaces influences of urbanization, pollution and fragmentationcaused by oil development. In recent years regulation of the rivercourse to the delta and decreased sediment loads have led tosalinization and a trend of rapid decrease of wetlands. Thefreshwater wetland area has decreased half in size in the last 20years, destroying the connectivity and integrity of the wetlandecosystems. The habitats that are used by rare birds are facing thedanger of disappearance.

What would be a more balanced water allocation for sustain-able development of the wetland nature reserves, dealing with theeffects of land use changes and variations in the flooding regime?

During one and a half year 5 10-day workshops were organisedwith the Yellow River Conservancy Commission (YRCC), hydrolog-ical and ecological experts from the University of Najing and theChinese Academy of Science, Dutch consultants and local stake-holders to define scenarios, spatial strategies, indicators andcompare scenario and strategy impacts. Stakeholders wereselected by the YRCC based on their dependency of water fromthe Yellow River and included the Nature Reserve Authority andurban planning of Dongying municipality. Both also representingagriculture and aqua-culture farmers within their territory. Since itwas argued that the oil industry predominates all other interests itwas decided not to include it in the workshops. Stakeholderpresence varied with relevance per workshop.

The study started with an inception workshop resulting in adiagnosis of the problems, defining the boundary conditions andapproach of the study in detail, and including indicators formeasuring ecological performance. Four additional workshops

P. Verweij et al. / Environmental Science & Policy 66 (2016) 47–61 53

were planned. In each workshop focus groups were formed with aspecific objective, such as the definition and refinement ofscenarios, spatial strategies, ecological qualitative rule-basedmodelling and hydrological modelling (to be denoted as watermodels). During each workshop the focus groups worked in dailyiterations. At the end of each day each focus group presented theirprogress for plenary discussion and acceptance by officials.

In the first workshop sessions were organised to: 1) definescenarios, spatial strategies and indicators based on the proposalsby YRCC, 2) do an inventory of required available spatial data, 3)choose water management options and, 4) model the ecologicaleffects based on expert rules. In consecutive workshops scenarios,spatial strategies and the knowledge rules were refined.

Each workshop involved modelling. Due to their complexityand data needs the water models were run once, or twice during aworkshop. At the start of a workshop parameters for a scenario(water volume per unit of time) and spatial strategy (location ofdams) were chosen to be fed to the models. Resulting ground waterlevel and flood duration maps were discussed afterwards.

The semi-quantitative ecological model was built with thestakeholders keeping the targeted indicators constantly in mindand using those as a starting point for back reasoning the causalrelationship from habitat suitability towards the inputs generatedby the water models (Eupen et al., 2007). The ecological know-howwas gathered and implemented during the workshops andincluded the definition of ecotope-, vegetation and physiotopetypologies and rules for vegetation development. During a dailysession multiple iterations of ecological model adaptation,execution and result analysis were made.

During the workshop the participant awareness of possible andfeasible water allocation increased. Later, part of the wetlandnature reserve was given the Ramsar status as result of this study(Ramsar Convention Secretariat, 2013).

3.4. Sample result 3, scientific method development—ecosystemintegrity in the Brazilian Amazon

Deforestation and climate change heavily impact the ecosystemof the Amazon rainforest threatening its resilience and thesustainability of many human activities. The notion of EcosystemIntegrity is used as a synonym for intactness, completeness andintegration of ecosystems. Land protection may prevent ecosys-tems and their services to deteriorate from the pressures ofagricultural expansion, population growth and wood harvesting. Inthe Brazilian Amazon land protection occurs in several forms suchas environmental conservation, setting biodiversity priority areasand the delineation of indigenous lands. Still, the effects are notclear as understanding of the ecosystems is incomplete andresponses to human actions are highly uncertain.

Bayesian Belief Networks (BBN) are models that probabilisti-cally represent correlative and causal relationships amongvariables. BBNs have been successfully applied to natural resourcemanagement to address environmental management problemsand to assess the impact of alternative management measures.While BBN’s are used to study results from deliberative participa-tory questionnaires linked to GIS-data (e.g. Gret-Regamy et al.,2013) and in preference elicitation methods with a very littleamount of spatial entities (e.g. Haines-Young, 2011), few studieshave fully integrated BBNs and GIS and explored the resultingbenefits (Stelzenmueller et al., 2010). By training the probabilisticrelationships using field data, Remote Sensing data and GIS datathe BBN can provide information on the ecosystems: theecosystem integrity and their likely response to climate changeor alternative management actions. For this study the QUICKScansoftware was extended with BBN functionality to allow BBN’s to beapplied on spatial data without the need for time consuming and

error prone manual conversion of data between GIS software andBBN software.

During an initial tele-conference ecosystem experts and spatialmodellers set up a conceptual map (Novak, 1991) of ecosystemintegrity that fit the perceived reality of the local experts. Based onthe identified drivers satellite imagery was used to create drivermaps of leaf area index (Watson, 1947), Gross Primary Production(Prince and Goward, 1995), evapotranspiration and vegetationcover (Amthor and Baldocchi, 2001). The conceptual map wastransferred to a prototype ecosystem integrity BBN-model and wastested against experts’ expectations. To test the effect of theinclusion of probabilities mechanistic rules were developedsimultaneously. The results of both approaches were compared.The statistical BBN relationships and the mechanistic rules in bothmodels were iterated upon during several tele-conferences withthe Brazilian ecosystem experts, Brazilian Remote Sensing expertsand Dutch ecosystem modellers and QUICKScan experts. Inbetween the tele-conferences more Remote Sensing- and GISdata was gathered by the Brazilian experts. which was integratedduring the tele-conferences. The iterations stopped when the localexperts were satisfied with the result and identified the necessityto further tune and proof the model with field data.

The study showed that the concept of Ecosystem Integrity canbe mapped using high resolution satellite imagery. Both themechanistic rules and the BBN resulted in a similar statisticaloverall distribution of the Ecosystem Integrity. However, themodelled spatial patterns were quite different. The local expertsjudged the BBN to better fit reality. The BBN model showed moregradual integrity transitions and better positioned the well-knownbiodiversity hotspots. This study is input for the evaluation ofexisting and assessment of potential future conservation areas andindigenous lands. The study has been published in Verweij et al.,2014b and Simões et al., 2015.

3.5. Participant feedback

At the end of workshops participants were asked to shortlyreflect upon how they perceived the workshop. Annex 2 provides alist of the feedback. Based on this feedback the following topicssupporting the approach were extracted:

� The method speeds up the first stages of the policy cycle (Fig. 1):gaining understanding, finding evidence, identifying data andknowledge gaps and the rapid evaluation of strategies whendoing impact assessments.

� The method stimulates to truly work interdisciplinary. Eachindividual responds to the visualisations of modelled results,which is then discussed by the group

� This proves it is possible to do an assessment without complex,time consuming and expensive modelling.

Critical reflections include

� If the stakeholders don’t bring in important information youmight miss out the effects that make a difference.

� How strong will the evidence-base of the results of a workshopbe back in the political arena?

� The method heavily relies on the availability of spatial data. Ifthe data is of poor quality you will also get poor results.

4. Discussion and conclusion

As demonstrated above, the QUICKScan methodology operateson the science-policy interface and can be employed in a range ofdifferent circumstances to jointly develop an understanding ofthe problem and solution space in early phases of policy

54 P. Verweij et al. / Environmental Science & Policy 66 (2016) 47–61

development. QUICKScan has matured via a large number ofapplications (Annex 1) to an off-the shelf methodology for policy-science interaction in the exploratory phase of policy develop-ment. We demonstrated that the methodology is capable ofdeveloping storylines, selecting indicators for measuring theobjective achievement, gaining and processing of stakeholderknowledge and jointly create new model(s) as is done inparticipatory modelling (Voinov and Brown Gaddis, 2008).QUICKScan offers access to spatially distributed phenomenaand provides interactive zooming, overlaying, temporal compar-isons and many visualization options as used in participatory GISas part of its tool (McCall, 2003; Jankowski, 2009; Cutts et al.,2011). QUICKScan is applicable in situations that Ittersum et al.,1998 calls explorative; a situation with high uncertainty and highcausality.

4.1. Three main benefits of QUICKScan emerged during theapplications.

First, the use of QUICKScan resulted in a reduction of lead timefor the problem scoping phase of the policy cycle. In situations withuncertainty on the precise definition of the problem, theimplications in different futures and the possible responses inscenarios, it produced rapidly a joint understanding of the mainrelevant interactions, the impact on indicators and commitmentfrom different stakeholders for future steps. Even if the lead timeincludes time for data preparation and initial discussions onproblem formulation before the main event in the workshop, in allcases it was still faster as a policy officer contracting out extensiveresearch on a specific problem for evidence gathering, or as expertgroup consultations. As an added benefit the results of theworkshops often provided pointers to questions in which moreevidence has to be gathered, or a more extensive stock-take of theavailable evidence is required in further development of thepolicy options. Such next steps could for example be executedwith more detailed system dynamics models including feedbackloops.

Second, the application of QUICKScan resulted in a better jointunderstanding across stakeholders. Rodela et al. (2015) found thatQUICKScan performs well on knowledge integration, learning andshared understanding. Particularly in the workshops, participantscould be carefully selected to represent different perspectives,while alternatively the approach to the problem could be adaptedto the stakeholders available in some applications where therewere more representatives from science seeking a thoroughunderstanding from a scientific point-of-view. Participants areforced to listen to another, and jointly develop model inputmatrices and relationships between variables, on which they allhad their views individually, while at the same time getting anunderstanding of the impact on indicators, that were jointlyagreed as crucial reference points. In future discussions andinteractions, the stakeholders could thus have more targetedexchanges on what they see as the most relevant interactions andindicators.

Third, participants emphasized the importance of internalizingthe (scientific) knowledge and data, as it was before only presentedto them in reports, visualisations and publications. By workingwith the knowledge, explicitly using it in constructing mentalmodels, and defining the relationships between variables, partic-ipants obtained an active understanding of the implications of theknowledge and data, as impacts could be visualized, and changes incausal pathways immediately resulted in changes in indicatorvalues. For this not only the mental model itself was crucial (ascaptured in other methods such as Fuzzy Cognitive Mapping), butalso the computation of indicator values as part of the mentalmodel.

The QUICKScan methodology still has some limitations.

First, a clear limitation is its link to spatial thinking, as the tool isspatially explicit, which excludes any non-spatial problems.Arguably all problems will have a spatial dimension, however,this may not be as important nor as apparent as the emphasis itreceives through the QUICKScan methodology.

Second, if the logical model has to include feedback loops andfocuses on explaining the systemic functioning, then more detailedmethods based on system dynamics are required. Arguably aninteractive and participatory setting of problem explorations is notappropriate for such investigations in systemic functioning, as thesystem description will likely soon be too complex for allparticipants to follow.

Third, a possible drawback of the use of this type of flexible modelsetup is that important drivers may be overlooked if no expertise, ordata of the topic is available. This makes the modelled values ofindicators less accurate or incomplete. To some extent this can beremedied by already identifying variables early on in the processfrom a problem perspective and finding appropriate data at thatstage. If data is not available, suitable proxies can then be identified.

Fourth, participants skills and predispositions may be limitativein some cases. Participants do usually not spend a great deal oftime on preparation for the workshop, unless actively involvedearly on, which may not be possible for all participants. Someparticipants might then not agree with the approach as importantdetails are overlooked from their perspective, or data was notincluded in the preparation that they believe is crucial.

This all emphases the importance of skilled facilitators who canalso mediate the use of technology and spatial data and thinking inparticipatory settings.

Further extensions of the QUICKScan methodology are continu-ously being worked on. As an example, a link to a map table is beingexplored, inwhich the map table can be used as an interactive tool forsome of the discussions by participants and by directly outliningareas on a map (e.g. conservation areas). Also an online platform iscontinuously build to document the different applications, whichcould in the future be used to bring data, results and models together,but also allow for continued discussion and exchanges betweenparticipants remotely. Finally, more computational tools are beingaddedtothe libraryof functions available inthespatiallyexplicit tool,including land use and land cover projections (Verweij et al., inprep.)and an extension of Multi Criteria Analysis.

In conclusion, QUICKScan speeds up the early phases of thepolicy cycle by facilitating knowledge uptake and internalizationthrough a strongly mediated participatory process. In these multi-stakeholder processes, science is not merely a messenger of dataand knowledge products through reports and briefings, but isintegrated together with local and tacit knowledge to reachbroader support for policy making. QUICKScan is relevant to theproblem and solution scoping phase in policy processes whenthere is a clear spatial component. Similar methodologies could bedeveloped in other policy processes.

Acknowledgements

The authors gratefully acknowledge the funding received forthis work. Alterra undertook the QUICKScan projects at the requestof and funded by the European Environment Agency (EEA/IEA/10/001, EEA/IEA/11/001 and ETC SIA 2012) and Alterra, WageningenUR. It has also been supported by the Dutch Ministry of EconomicAffairs and through contributions from the European UnionSeventh Framework Programme (FP7/2007-2013) under grantagreement no. 283093–The Role Of Biodiversity In climate changemitigatioN (ROBIN) and 308428–OPErationalisation of Naturalcapital and EcoSystem Services (OpenNESS).

Annex 1 : Listing of case studies and their characteristics

Case Study Objective Data Type Participants Setting Impact

Green Infrastructure ofEurope (EEA, 2011;Verweij et al., 2012)

What part of the Natura 2000 areascould be seen as GreenInfrastructure? And as Natura 2000is the core, what other areas mightbe included based on whatassumptions?

Pan-European spatial datasets witha 1 km2 resolution: protected natureareas, land cover, High Nature Valuefarmland, eco-tones and variousadministrative and bio-geographical mappings.

Explorativeassessment

European policyassessors anddomain expertsfrom across Europe.

During two days, three half-dayworkshops were organised withEuropean policy assessors anddomain experts from across Europe.Within workshop 1 the policycontext was delineated andalternatives and indicators defined.The experts used previouslygathered maps to derive theindicators for all alternatives inWorkshop 2. The next morning theresults were presented to theassessors in workshop 3 anditerated upon.

Helping the demandarticulation for definingGreen Infrastructure

Wetland Conservation inthe Yellow River delta(Eupen et al., 2007;Verweij et al., 2010b;Wang et al., 2012)

What would be a more balancedwater allocation for sustainabledevelopment of the wetland naturereserves, dealing with the effects ofland use changes and variations inthe flooding regime?

50�50m2 resolution: land cover,topography (incl. oil pump jacks),soil, water table, hydrological flow,vegetation and elevation.

Participatorymodel

development

Municipalityofficials,conservationcommissioners,farmers andhydrological andecological experts.

During one and a half year 5 ten-dayworkshops were organised to definescenarios, spatial strategies,indicators and compare scenarioand strategy impacts.

Short term: increasedparticipant awareness ofpossible and feasiblewater allocation;Long term: gave RAMSARstatus to part of thewetland nature reserve

Soybean Expansion inBrazil (Barreto et al.,2012)

What are likely areas for futuresoybean expansion? What is theeffect of that future expansion onindicator species as birds and largemammals? What areas needprotection?

250�250m2 resolution: land use(current situation and futureprojections), topography, soil,elevation

Facilitatescientificmethod

development

Biological experts,scientific ecologicaland land usemodellers

One year postdoc desk study withregular feedback rounds from fellowscientists

Raised awareness withinthe scientific community

Resettlement ofdisplaced persons inSouth Darfur(Eshitera, 2013)

Howmuch agricultural area is beingconverted to urban? Is the SouthDafur agricultural system able tosupport the population?

100�100m2 resolution: soil,rainfall, land cover (incl. agriculturalcrops and livestock grazing areas),water access points. Statistical data:consumption per capita, population,actual and maximum agriculturalproduction.

Explorativeassessment

Local farmers,agriculturalexperts, humansettlement expertsand municipalityofficials

Half year Msc study includingseveral group discussions andinterviews to gather expertknowledge (and data).

Increased localawareness: the studyresults indicated that thelivestock productionsubsystem is beyond thepotential sustainablecarrying capacity.

Landscape attractivenessof the Dutchcountryside (Roos-Klein Lankhorst et al.,2016; Losekoot, 2013)

How do citizens value the scenicbeauty of the Dutch livingenvironment different? Can thispurely be based on physicalcharacteristics of the landscape?And can citizens be grouped onsocietal background given theirvaluation?

50�50m2 resolution: land use andtopography including highbuildings, glass houses, powerpylons, wind turbines, lines of trees,ditches, etc.

Facilitatescientificmethod

development

Policy assessors,social scientists,statisticians, spatialmodellers

A group of citizen several locationsstatisticians determined locations totest for landscape attractiveness andidentified citizens to be interviewedto capture their perception of theselocations.

Validated model is used inannual reportingobligations of the Dutchgovernment

Urban Sprawl inEuropean cities(Winograd et al., 2013)

Where do we expect urban areas togrow?What are the biophysical andsocio-economic implications forurban, peri-urban and rural areas inrelation with land cover, greeninfrastructure?

Pan European 1�1km2 resolution:land cover (historic data and presentsituation), urban nigh light,protected nature areas, elevation,economic and population density,accessibility to cities, agriculturalsoil production, soil suitability forconstruction, administrativeboundaries

Explorativeassessment

European urbanexperts and policyassessors.

Three workshops with Europeanurban experts and policy assessors.Scopingwas performed inworkshop1. Workshop 2, took half a day andresulted in the definition of threealternatives and the identification ofrequired maps and statistics. Duringthe last workshop the alternativeswere built and linked to indicatorsusing knowledge of bothparticipating experts and policyassessors.

Results included inEuropean reportingobligation

Find relation between climate andhuman imposed drivers and

Pan Amazonia data at 1�1km2resolution: Remote sensing

Facilitatescientific

development Ecosystem experts, biologists,statisticians

Several tele-meetings andworkshops with Brazilian

P. Verw

eij et

al. /

Environmental

Science &

Policy 66

(2016) 47

–61

55

(Continued)

Case Study Objective Data Type Participants Setting Impact

Ecosystem Integrity ofthe Brazilian Amazon(Verweij et al., 2014b)

ecosystem integrity to find likelyfuture impacts on biodiversity

products (like leaf area index andvegetation cover), land use, landcover, environmental protectionzones and zonation of indigenouslands

method

development

scientists and Dutchecosystem experts andmodellers. Fieldwork wascarried out to validate themodel.

Found quantitativerelationships of thecomponents forming theecosystem integrityHigh Nature ValueForests within Europe(Winograd et al., 2013)

High Nature Value forests are ahotspot for biodiversity. How canthese forests be best characterisedand where are they located?

Pan European 1�1km2 resolution:tree species, forest types, land cover,growing stock, wilderness of naturalvegetation, forest connectivity,precipitation, slope, protectionstatus, ecotones.

Explorativeassessment

European forestryexperts, ecologistsand policyassessors.

Three workshops with Europeanexperts and policy assessors.Scopingwas performed inworkshop1. Workshop 2, took half a day andresulted in the definition of fouralternatives and the identification ofrequired spatial data. During the lastworkshop the alternatives werebuilt and linked to indicators usingknowledge of both participatingexperts and policy assessors.

Results included inEuropean reportingobligation

Risk mapping for soilCarbon under climatechange (Hijbeek et al.,2016, in prep.)

Find hotspots of soil carbon stockthat are sensitive to climate changeendangering the sustainability offarming systems.

1�1km resolution: soil texture,aridity, organic matter, slope andfarming systems.

Facilitatescientificmethod

development

Geographers,biological experts,soil scientists,spatial modellersand representativesof local farmers

One month scientific expert deskstudy with regular feedback roundsfrom fellow scientists

Improving expertknowledge using localknowledge

Impact of climate changeon biodiversity in Pan-European protectedareas

The aim of the European Natura2000 network is to assure the long-term survival of Europe's mostvaluable and threatened species andhabitats (European Commission,2013a,b). How does the futureclimate variability change and howvulnerable are the protected areas tothis change?

1�1km2 resolution: maps ofNatura 2000 areas (EEA, 2013),climate projections (fromEU FP6Integrated Project ENSEMBLES,Contract number 505539), DigitalElevation Model, Land cover,population density and accessibilityto markets

Explorativeassessment

European policymakers and expertson climateadaptation.

During a half a day workshop weevaluated the impact of variousclimate projections (REF TOENSEMBLES) on the protected areason basis of the participantsexpertise

Created more in depthquestions, createdawareness of usefulness ofIT tools for climate changeimpacts exploration

Vulnerability andadaptation assessmentfor Central America(Winograd, 2013)

What are the main vulnerabilitiesand risks to climate variability andclimate change at local, regional andnational scale? What are the bestmitigation and adaptation options?

Population and agricultural censusdata, 1�1km2 resolution: landcover, land se, temperature andprecipitation (both actual andprojections), topography, elevation,administrative areas, accessibility tomarkets

Participatorymodel

development

United NationsEnvironmentProgram � Climatechange andadaptation teamand all REGATTAproject (UNEP,2013) members forlatin America

One month expert desk study withregular feedback rounds fromdecision makers

Strengthened capacity onhow to do a vulnerabilityassessment

Pantanal River, Brazil(Jongman et al., 2005)

Build capacity to develop a coherentriver management organisation toreduce unwanted effects in theBrazilian Pantanal, like: permanentinundation caused by sanding up ofthe Rio Taquari.

100�100m resolution: land useand flooding patterns (based on30�30m multi-temporal LANDSATsatellite images), geomorphology,vegetation, geology, topography

Participatorymodel

development

Biologists, watermanagers,hydrologists,ecologists, regionalpolicy maker andlocal (large scale)farmers

Three workshops of a week withscientific experts to build the modeland two workshops with farmersand policy maker to evaluate andimprove the rules by including theirtacit knowledge.

Local farmers changeposition and join forceswith policy makers topush forward theenforcement of regulation

Integrated floodmitigation strategies,Taiwan (Yang et al.,2011)

What spatial plan is optimal for bothflood prevention and habitatrestoration?

25�25m resolution: land use,flooding frequency, elevation, hydronetwork, man-made watermanagement structures (e.g. dikes)

Explorativeassessment

Water managers,conservationorganisation,municipalityofficials andlandscapeecologists

Preparatory workshop withstakeholders to develop floodpreventive scenarios.4 months scientific expert deskstudy with regular feedback roundsfrom fellow scientists.

Raised political awarenessfor the importance ofintegrated assessments

56

P. Verw

eij et

al. /

Environmental

Science &

Policy 66

(2016) 47

–61

Agricultural value-of-use (Boogaard et al.,2003)

What is the value of land fordifferent agricultural functions?How can we optimally allocateownership of lands?

30�30m resolution: soil type,water level, organic matter, waterstorage, soil acidity

Participatorymodel

development

Experts on farm-level agriculturalpractise, soil andagricultural crops

7 months scientific expert deskstudy with regular feedback roundsfrom fellow scientists.

Created valuation of landforming the basis forexchange of landownership by farmers

Suitability for naturedevelopment (Runhaaret al., 2003)

Determine nature target suitabilitybased on site conditions

25�25m resolution: water type,seepage, acidity infiltration,groundwater levels, other soilcharacteristics

Participatorymodel

development

Experts on locallevel watermanagement,vegetationspecialists andecologists

5 months a year scientific expertdesk study with regular feedbackrounds from fellow scientists

Several Dutch watermanagers use the NATLEScriteria for analysingimpacts of hydrologicalchanges on terrestrialecosystems, such asspatial planning to adaptfor climate change andnature conservation

Wetland restoration inthe Liaohe delta, China,(Xiaowen et al., 2012;Knol and Verweij, 1999)

Develop scenarios and identifymeasures to realize the landscapetargets, locate the spatial areasinvolved in these measures, anddetermine the ecological impacts onflagship species.

25�25m resolution: vegetation,land cover, soil, wetness, landscapetarget scenarios, measures, roads,waterways and oil plants.

Facilitatescientificmethod

development

Conservationorganisationrepresentatives,watermanagement,municipalities,fisheries, farmers,consultancy (forcosting measures),ecologists andhydrologists.

Preparatory interviews andworkshops with stakeholders tounderstand stakes and preferences.Half a year scientific expert deskstudy with regular feedback roundsfrom fellow scientists andstakeholders.

Mitigated the competingland-use needs betweenthe ecologicalconservation and humanneeds, and tomaintain the“no-net-loss” of wetlandhabitats

Integrated watermanagement optionsfor East Africa (Eupenet al., 2014)

What are the costs and what is theeffectiveness of measures for (1)minimizing the number of people atrisk for flooding and (2) to minimizeyield gaps in crop production?

Study area 1500�1000km,1�1 kmresolution: land cover, flooded area,soil texture, precipitation, yields,conservation areas, accessibility,grazing density, population density,elevation, slope, river basinboundaries, local measures

Facilitatescientificmethod

development

Policy advisors,experts on floodrisk modelling,landscape ecologistand an agriculturaleconomist

During one and a half year 10 one-day workshops were organised todefine scenarios, spatial strategies,indicators and compare scenarioand strategy impacts. Theseworkshops were followed up withdesktop improvements.

Learned that linking localmeasures to global datadoes not provide plausibleinformation.Participatory sessionshave little added value ifcomplex modellingapproach is a prerequisite.Software must be robust.

Adaptive managementplan for the lowerDanube river, Romania

What are ecosystem service impactsof different ecologicalreconstruction plans?

10�10m resolution: land use,flooding regimes, protection statusand administrative units

Explorativeassessment

EnvironmentalNGO, watermanager,municipalityofficial, farmers,fisheriesorganisation,tourist organisation

1day workshop with 12 individualsin which ESS were identified,prioritized and rules defined forquantifying the value of these ESS.Two scenarios were developed andthe implications for the ESS assessed

Shared understanding ofthe stakes. Joint agendasetting

Central area of theKiskunság NationalPark, Hungary

Local experts and naturemanagement organisations together(re)thinking the ongoing land usedevelopments, develop sustainableland use and water managementoptions, and consequently helpreveal conflicts over land use changeand management.

Study area is part of a long termsocio ecological research networksite (LTSER) 40� 40km, 25�25mresolution: land cover,topographical wetness, accessibility,elevation, distance from roads,administrative units andtopography

Explorativeassessment

Forest managers,natureconservation, waterauthorities andecologists

1 day workshop in which ecosystemservices (ESS) were identified,prioritized and rules defined forquantifying the value of these ESS.Prioritize at the local level relevantservices for five different birdgroups by evaluation of preference,according to this landscape ofpastures (pollen, nectar) servicecapacity of the local population interms of priority services.

Organised working groupthat met on a regular basisusing QUICKScan forfurther exploration andassessment resulting inchanging the LTSERmanagement plan

Map current and futureecosystem services inGlenlivit, Scotland

What are currently priorityecosystem services? How will theychange under different land usescenarios? Which are the trade-offs?

10�10m resolution: land cover,topographical wetness, accessibility,elevation, distance from rivers,administrative units andtopography

Explorativeassessment

Natureconservation,tourism, foresters,farmers,sociologist,ecologist,hydrologist,business developer

1day workshop in which ecosystemservices (ESS) were identified,prioritized and rules defined forquantifying the value of these ESS.three scenarios were developed andthe implications for the ESS assessed

Safe environment forstakeholders to putforward and try outextreme scenarios andevaluate based on impactvisualization

P. Verw

eij et

al. /

Environmental

Science &

Policy 66

(2016) 47

–61

57

(Continued)

Case Study Objective Data Type Participants Setting Impact

Effects of land usechange on landscapequalities (Roos-KleinLankhorst et al., 2013)

Determine status and likely impactsof policy scenarios on landscapequalities for the Netherlands(cultural history, landscape scale,historical landscape, recreationcapacity, green infrastructure, visualdisturbance, morphology)

250�250m resolution: land cover,land use, topography, elevation,vegetation, management ofagricultural and natural areas,building- and vegetation height

Participatorymodel

development

Policy makers,policy assessors,thematic experts

4 Month study with intensecollaboration of thematic expertsand policy assessors. Multipleworkshop sessions, many bilateralmeetings and emails. The model isupdated annually by a small groupof experts and irregular groupdiscussions.

Validated model is used inannual reportingobligations of the Dutchgovernment

Mapping of EcosystemServices of 17EUMember States (Maeset al., 2013; Braat et al.,2015; Pérez-Soba et al.,2015).

Train EU Member States on how tomap and assess the state ofecosystems and their services intheir national territory to helpdecide on what ecosystems torestore with priority where.

Varies per Member State. Dataresolution ranges from 25�25m to1�1km on land cover, forest types,management, elevation, floods,accessibility of rural areas andcoastal waters, fishing activities,standardised prices (e.g. per treespecies), protected areas, etc.

Explorativeassessment

Nationalpolicymaker,national EcosystemServices expert andnational (spatial)data expert withthe support of anEcosystem Serviceschampion,representativesfrom the EuropeanCommission and aQUICKScanmodeller.

A one hour preparatory tele-meeting to identify key EcosystemServices and available data perMember State.Two day workshop with eachMember State to map their threepreferred ecosystem services.

Raised awareness on howto map ecosystemservices; Created 3ecosystems service mapsper Member State(17*3 = 51 maps in total);Clarification of theEuropean Commission’sobjective to obtain themaps; Participants areable to create their own(improved) maps.

Planning for ecosystemservices in Cities: theAmsterdamcase (Zardoet al., 2016, in prep.)

Make an explicit link betweenphysical features of GreenInfrastructure and provisioningEcosystem Services

5�5m resolution: land cover, treecrown coverage, tree species, treeelement type (e.g. line of trees,single tree, etc.) NDVI, soil, height ofartificial areas, wind direction andwind strength

Facilitatescientificmethod

development

Scientists: urbanfabric designer,ecologist,ecosystem servicesexperts and a(spatial) dataexpert

3 Month study in which regularexpert discussions and participatorymodelling sessions took place.

Understanding how whaturban ecosystems servicesmay be mapped.

Planning for greenfunctions in the Dutchcity of Utrecht (Maeset al., 2016)

The main societal functions forwhich municipalities design greenspaces are the aesthetic value ofgreen spaces and recreation. TheDutch municipality of Utrecht isinterested in making more use ofgreen infrastructure in the search ofmeasures that can help to achieve ahealthy city; for regulatingtemperature, air quality, waterstorage and drainage and noisereduction.

10�10m resolution: land cover,road patterns and road usage, treecrown coverage, tree species, heatpeaks, particulates from traffic,green index and noise levels.

Explorativeassessment

Municipallandscapearchitects and civilservant experts onhealth, water,cateringestablishments,recreation, citizencomplaints (e.g.stress by noise) andecology

Half a day workshop with 8individuals for identifying functionsof green and doing trade-offanalysis. New functions andindicators where identified, definedand validated via group discussion.

Shared understanding onthe effect and scope ofgreen measures.

Safeguarding access tomineral deposits in 8European memberState regions (Murguiaet al., 2015)

The exploitation of minerals inEurope is an indispensable activityto ensure that the present andfuture needs of the European societycan be met. Access may be hinderedby legislative, biophysical orcommunity opposition constraints.Perform participatory land useplanning to overcome theseproblems.

Varies per Member State. Dataresolution ranges from 25�25m to1�1km on geology, sea vesselroutes, accessibility, culturalheritage, protected natural areas,land cover, fishing grounds.

Explorativeassessment

Nationalpolicymaker,national EcosystemServices expert andnational (spatial)data expert withthe support of anEcosystem Serviceschampion,representativesfrom the EuropeanCommission and aQUICKScanmodeller.

A one hour preparatory tele-meeting to identify key mineralsand available data per MemberState.Two day workshop with eachMember State to map theirpreferred minerals.

Deeper understanding ofdifferent uses of land andsea influencing the(future) extraction ofminerals.

58

P. Verw

eij et

al. /

Environmental

Science &

Policy 66

(2016) 47

–61

Understan

ding

Eco

system

Services

andwellbeingin

Mab

alan

ewoo

dland,

Moza

mbique

(Moh

aman

eet

al.,20

16,

inprep.)

Despitetheke

yrole

thech

arco

alindustry

plays

intheliv

elihoo

dof

thepeo

ple,thech

arco

alprovision

ingec

osystem

servicehas

dec

reased

considerab

lybe

cause

ofch

ange

sin

landuse

andlandco

ver.

Thesech

ange

saffect

dev

elop

men

tal

initiative

sin

Mab

alan

e.Th

isstudy

iden

tifies

theco

mmunitiesmos

tsu

scep

tibleto

chan

gesan

dhelps

targetinghow

tobe

stsu

pportru

ral

communities.

30�30

mreso

lution

:landco

ver,

elev

ation,s

lope,

topog

raphy

,acce

ssibilityto

reso

urces,

acce

ssibilityto

marke

ts,

topog

raphic

wetnessindex

Facilitate

scientific

method

dev

elop

men

t

Com

munities,

park

range

rsan

dgo

vern

men

tal

officials,

sociolog

ists

and

ecolog

ists

Seve

ralpreparatoryworks

hop

sto

iden

tify

ecos

ystem

services,

tech

nical

interven

tion

s,institution

alinterven

tion

san

dforest

man

agem

entop

tion

sto

geta

localpersp

ective

ofissu

essu

rrou

ndinglanduse,e

cosystem

services

andwell-be

ing.

The

QUICKSc

anmod

el(includingthe

Bay

esianBeliefNetwork)

was

built

byseve

ralscientistsan

dtested

against

factsan

dpreference

sga

thered

duringpreparation.

Enhan

cedunderstan

ding

effectsof

landuse

and

landco

verch

ange

asa

resu

ltof

land

man

agem

ent,marke

tan

dpolicy,

therelation

ship

withEc

osystem

Services

andtheim

pacton

the

livelihoo

dof

villag

ersto

target

aid.

Coffee

pro

ductionin

Colombia

(Bec

erra

etal.,20

15)

Whicharea

sare,

production-w

ise,

mos

taffec

tedby

clim

ateva

riab

ility?

How

andwheredowhat

man

agem

entop

tion

saffect

water

qualityan

d�q

uan

tity?W

hat

are

social

andfarm

erprofitability

impacts?

500�50

0an

d30

�30

mreso

lution

:altitude,

temperature,p

recipitation

pattern

s,harve

stperiods,

administrativeregion

s,farm

types

andfarm

man

agem

ent

Participatory

mod

el

dev

elop

men

t

Prob

lem

owner,

marke

ting

spec

ialist,b

usiness

expert,field

experts,s

cien

tific

experts

Oneday

preparatoryworks

hop

todeterminetheob

jective,

iden

tify

participan

tsan

dreleva

ntdatasets.

Follow

edby

atw

o-wee

kworks

hop

Uptake

ofQUICKSc

anin

theman

agem

entproce

ssof

theorga

nisationto

supportea

rlywarning

dec

isionmak

ing

P. Verweij et al. / Environmental Science & Policy 66 (2016) 47–61 59

Annex 2 : Participant feedback

� Feedback that confirms the QUICKScan approach, include:

� ‘QUICKScan speeds up taking management decisions. It provided uswith the management information we require for taking decisions intwo days. It took our analysts 2 months to do detailed analysis andthen aggregate it to the indicators relevant for our job’ (Businessmanager, November 26th 2015)

� ‘This workshop pushes us to truly work interdisciplinary, which athome we don’t manage to do although we have got the same peoplearound’ (Business operational manager, November 18th 2015)

� ‘Great to rapidly conduct a semi-qualitative analysis and create mapand graph products while doing so’ (Regional policy maker,November 5th, 2015)

� ‘Although the data is sometimes of poor quality and knowledge of allunderlying processes incomplete. Let’s work with it and give adviceto the best of our capabilities, as lobbyists will push forward theiragenda’s and decisions are going to be taken anyway’ (Marinemineral consultant, November 3, 2015)

� ‘The rather extreme scenarios we set up and assessed clarified wherewe had to refine and which scenarios didn’t have a relevant impact.It helped us identify the scenarios that were of potential interest’(spatial planner, October 13th, 2015)

� ‘Love the possibility to smoothly shift between scales, numbers,relations and dialogue. Very stimulating’ (municipal official,October 13th, 2015)

� ‘For several months we have had the idea that our proposed policywould have huge impacts, but within these few hours we have cometo understand that it will never have the magnitude we hadpresumed. We need to adjust our strategy.’ (policy maker, February19th, 2015)

� ‘We don’t always need to initiate an expensive and time consumingtender to hire a consultant. This can speed up our work considerably’(policy maker, February 19th 2015)

� ‘The storytelling of my colleagues at the start of the workshop andtheir choice of maps to illustrate it was very interesting indeed.’(conservationist, January 22nd 2015)

� ‘It is so easy and fast. It feels a bit like a game, but it really makesme think. Very stimulating.’ (policy advisor, September 10th2014)

� ‘QUICKScan provides relevant results and is easy to use. GIS tools aremore complex. I am happy to find out it is possible to do anassessment without complex, time consuming and expensivemodelling’ (policy advisor, February 13th 2014)

� ‘The iterative approach of starting simple and adding complexitylater on is very useful. QUICKScan is very practical and easy. It is agood communication tool’ (scientist, February 13th 2014)

� ‘We are no longer afraid of modellers who say everything is complex.We can use a far simpler approach and have a useful result. We nowsee we should substitute missing data by expert estimates, but thisrequires courage against the critique of hard scientists.’ (scientistand policy advisor, February 13th 2014)

� ‘the use of Knowledge Tables is quite easy and handy as it is helpedus local experts to participate in decision making where weallocated the available land resources to various livelihood zones’(local expert, November 2012)

� ‘Information which we can read from internet or from reports bringsus intellectual knowledge. In this workshop we gained hands-onknowledge. That experience based knowledge goes much deeperthan the theoretical intellectual one’ (policy maker, February 13th2014)

� ‘QUICKScan is crisp and clear. A very elegant tool’ (policymaker,March 22nd 2013)

60 P. Verweij et al. / Environmental Science & Policy 66 (2016) 47–61

� Critical reflections include:

� ‘This approach uses constant expert gut-feeling assessment ofknowledge, results and uncertainties. This is no objective assess-ment.’ (Policy advisor from industry, November 5th, 2015)

� ‘There is too little time to study themeta-data to objectively assessthe results’ (Geological scientist, November 4th, 2015)

� ‘The method heavily relies on the availability of spatial data. If thedata is of poor quality you will also get poor results.’ (scientist andpolicy advisor, October 2014)

� ‘The mechanistic approach is too simplistic. In the real world it isoften the sudden unexpected changes or the sum of many smallchanges that make a difference.’ (scientific modeller, July 2014)

� ‘You only include the perceptions of the participating stakeholdersat the time of the workshop. Isn’t that too narrow and toosusceptible to change?’ (scientist, February 2016)

� ‘How strong will the evidence-base of the results of this workshop beback in the political arena?’ (scientist and policy advisor, February2016)

� ‘Complex spatial interactions like spill over cannot be modelledwithin a few hours. You’ll miss out on just the effects that make adifference’ (scientific economic modeller, 2012)

� ‘If your stakeholders don’t bring in that peak water levels occurevery 100 year you’ll miss out the effects that make a difference’(hydrological consultant, 2010)

References

Adelle, C., Jordan, A., Turnpenny, J., 2012. Proceeding in parallel or drifting apart? Asystematic review of policy appraisal research and practice. Environ. Plann. C 30(3), 400–414.

Aloysius, J., Davis, F., Wilson, D., Taylor, A., Kottemann, J., 2006. User acceptance ofmulti-criteria decision support systems: the impact of preference elicitationtechniques. Eur. J. Oper. Res. 169 (1), 273–285 (16, Pages).

Ampt, P.R., Ison, R.L.,1989. Rapid Rural Appraisal for the identification for agronomicresearch. Proc. XIV Int Grassland Congress, Nice 1291–1292.

Amthor, J.S., Baldocchi, D.D., 2001. Terrestrial Higher Plant Respiration and NetPrimary Production. Terrestrial Global Productivity. Academic Press, pp. 33–59.

Barreto, L., van Eupen, M., Kok, K., Jongman, R., Ribeiro, M., Veldkamp, A., Hass, A.,Oliveira, T., 2012. The impact of soybean expansion on mammal and bird, in theBalsas region north Brasilian Cerrado. J. Nat. Conserv. 20, 374–383.

Becerra, J.P., Carvajal, F., Winograd, M., Verweij, P., Miguel-Ayala, L., 2015,QUICKScan: hands-on training to assess what it can for the Federacion Nacionalde Cafeteros and Intelligent Water Management, internal technical report.

Boogaard, H.L., Verweij, P., Runhaar, J., 2003. Agricultural land suitability in theNetherlands (Dutch: Een nieuwe applicatie voor het bouwen en toepassen vangeografische kennissystemen: nu is het MENES). Agro Inf. 16, 5–8.

Braat, L.C., de Groot, R., 2012. The ecosystem services agenda: bridging the worlds ofnatural science and economics, conservation and development, and public andprivate policy. Ecosyst. Serv. 1 (1), 4–15.

Braat, L.C., Bouwma, B. Delbaere, S. Jacobs, N. Dendoncker, M. van Eupen, A. Grêt-Regamy, M. Perez-Soba, J. Peterseil, F. Santos-Martin, P. Scholefield, A. Torre-Marin, P. Verweij and B. Weibel (2015), Mapping of Ecosystems and theirServices in the EU and its Member States (MESEU), report no. ENV.B2/SER/2012.0016.

Brus, D.J., Hengeveld, G.M., Walvoort, D.J.J., Goedhart, P.W., Heidema, A.H., Nabuurs,G.J., Gunia, K., 2012. Statistical mapping of tree species over Europe. Eur. J. For.Res. 131 (1), 145–157.

Buchanan, B.G., Smith, R.G., 2003. Fundamentals of expert systems. Ann. Rev.Comput. Sci. 3 (1), 23–58.

Burrough, P.A., McDonnell, R.A., Lloyd, C.D., 1998. Principles of GeographicalInformation Systems. Oxford university press, pp. p 327.

Cutts, B., White, D., Kinzig, A., 2011. Participatory geographic information systemsfor the co-production of science and policy in an emerging boundaryorganization. Environ. Sci. Policy 14 (8), 977–985.

Davies, G., Dwyer, C., 2008. Qualitative methods II: minding the gap. Prog. Hum.Geogr. 32 (3), 399–406.

EEA, 2011. Green infrastructure and territorial cohesion. EEA Techn. Rep. 18–2011.EEA, 2013. Corine Land Cover 2006 Vector v17. available at: http://www.eea.europa.

eu/data-and-maps.EEA,, 2014. Developing a forest naturalness indicator for Europe. Concept and

Methodology for a High Nature Value (HNV) Forest Indicator, EEA TechnicalReport No 13/2014, European Environmental Agency, Copenhagen.

Eshitera, A., Assessing Agricultural Land Carrying Capacity for SustainableLivelihoods and Resettlement of Internally Displaced Persons in South Darfur,2013, Msc thesis, supervisores: Boerboom,L., Dopheide, M., Faculty of Geo-Information Science and Earth Observation of the University of Twente.

Eupen M van, B. Pedroli, C Huang, X. Wang, Impact modelling of scenarios onvegetation and fauna in the Yellow River Delta. Yellow River DeltaEnvironmental Flow Study Sino-Dutch Study Programme Final Report 2007YRCC, Yellow River Water Resources Protection Bureau, Alterra, DelftHydraulics, Arcadis Euroconsult, 2007.

Eupen, M., van Hoek, S.B., Boogaard, H., 2014. Linking a Database on Land and WaterManagement Options with a Spatial Knowledge Rule Based ModellingEnvironment. Alterra, Wageningen-UR, The Netherlands.

European Commission, 2010, Framework for commission expert groups: horizontalrules and public register, 1360.

European Commission, 2011, Our life insurance, our natural capital: an EUbiodiversity strategy to 2020.

European Commission, 2013, digital agenda for europe, a europe 2020 initiative,why do we need policy making 3.0?, http://ec.europa.eu/digital-agenda/en/why-do-we-need-policy-making-30.

European Commission, 2013, digital agenda for europe, a europe 2020 initiative,why do we need policy making 3.0?, http://ec.europa.eu/digital-agenda/en/why-do-we-need-policy-making-30.

Gavrel, F., Lebon, I., Rebiere, T., 2016. Formal education versus learning-by-doing: onthe labor market efficiency of educational choices. Econ. Modell. 54, 545–562.

Gibbons, M., Limoges, C., Nowotny, H., Schwartzmann, S., Scott, P., Trow, M., 1994.The New Production of Knowledge: The Dynamics of Science and Research inContemporary Societies. SAGE publishing, London (p 192).

Gregory, R., Failing, L., Harstone, M., Long, G., McDaniels, T., Ohlson, D., 2012.Structured Decision Making—A Practical Guide to Environmental ManagementChoices. Wiley-Blackwell (p 312).

Gret-Regamy, A., Brunner, S., Altwegg, J., bebi, P., 2013. Facing uncertainty inecosystem services-based resource management. J. Environ. Manage. 127, 145–154.

Guillaume, J., Jakeman, A., 2012. Providing certainty in predictive decision support:the role of closed questions. In: Seppelt, R., Voinov, A.A., Lange, S., Bankamp, D.(Eds.), International Environmental Modelling and Software Society (iEMSs)2012 International Congress on Environmental Modelling and SoftwareManaging Resources of a Limited Planet, Sixth Biennial Meeting, Leipzig,Germany. www.iemss.org/society/index.php/iemss-2012-proceedings.

Haines-Young, R., 2011. Exploring ecosystem service issues across diverseknowledge domains using Bayesian Belief Networks. Prog. Phys. Geogr. 35, 681–699.

Hengeveld, G.M., Nabuurs, G.-J., Didion, M., Van den Wyngaert, I., Clerkx, A.P.P.M.,Schelhaas, M.-J., 2012. A forest management map of European forests. Ecol. Soc.17 (4), 53.

Hijbeek, R., Cormont, A., Hazeu, G., Janssen, B., Bechini, L., Zavattaro, L., Werner, M.,Schlatter, N., Guzman, G., Bijttebier, J., Pronk, A., van Ittersum, M., in preparation,Do farmers perceive a shortage of soil organic matter? A European and farmlevel analysis.

Ison, R.L., Ampt, P.R., 1992. Rapid rural appraisal: a participatory problemformulation method relevant to australian agriculture. Agric. Syst. 38, 363–386.

Ittersum van, M., Rabbinge, R., Latesteijn van, H., 1998. Exploratory land use studiesand their role in strategic policy making. Agricult. Syst. 58 (3), 309–330.

Jankowski, P., 2009. Towards participatory geographic information systems forcommunity—based environmental decision making. J. Environ. Manage. 90 (6),1966–1971.

Jansen, J., Verweij, P., Wien, J., 2007, Policy lifecycle related tool development inenvironmental sciences, In: MODSIM 2007 International Congress on Modellingand Simulation, Australia and New Zealand, December 2007. Modelling andSimulation Society of Australia and New Zealand, ISBN 9780975840047—p.335–341.

Jetter, A.J., Kok, K., 2014. Fuzzy cognitive maps for futures studies—a methodologicalassessment of concepts and methods. Futures 61, 45–57.

Knol, W.C., Verweij, P.J.F.M., 1999. A spatial decision support system for riverecosystems. In: Wiens, J.A., Moss, M.R. (Eds.), Issues in Landscape Ecology,Proceedings of the Fifth World Congress, International Association forLandscape Ecology. Snowmass Village, Co, USA.

Kodikara, P., Perera, B., Kularathna, M., 2010. Stakeholder preference elicitation andmodelling in multi-criteria decision analysis—a case study on urban watersupply. Eur. J. Oper. Res. 206 (1), 209–220.

KorfMacher, K.S., 2001. The politics of participation in watershed modeling. Environ.Manage. 27 (2), 161–176.

Kosko, B., 1986. Fuzzy cognitive maps. Int. J. Man Mach. Stud. 24, 65–75.S. Losekoot, The Landscape appreciation model: construction and evaluation of two

prototypes, Technical report Van Hall-Larenstein—wildlife managementLeeuwarden. 2013.

Luyet, V., Schlaepfer, R., Parlange, M., Buttle, A., 2012. A framework to implementstakeholder participantion in environmental projects. J. Environ. Managent 11,213–219.

Maes, J., Hauck, J., Paracchini, M.L., Ratamäki, O., Hutchins, M., Termansen, M.,Furman, E., Perez-Soba, M., Braat, L., Bidoglio, G., 2013. Mainstreamingecosystem services into EU policy. Terr. Ecosyst. 5, 128–134.

Maes J, Zulian G, Thijssen M, Castell C, Baró F, Ferreira AM, Melo J, Garrett CP, DavidN, Alzetta C, Geneletti D; Cortinovis C, Zwierzchowska I, Alves FL, Cruz CS, BlasiC, Alós Ortí, MM, Attorre F, Azzella MM, Capotorti G, Copiz R, Fusaro L, Manes F,Marando F, Marchetti M, Mollo B, Salvatori E, Zavattero L, Zingari PC, GiarratanoMC, Bianchi E, Duprè E, Barton D, Stange E, Perez-Soba M, van Eupen M, VerweijP, de Vries A, Polce C, Cugny-Seguin M, Erhard M, Nicolau R, Fonseca A, Fritz M,Teller A (2016) Mapping and Assessment of Ecosystems and their Services.Urban Ecosystems. Publications Office of the European Union, Luxembourg.

http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0005http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0005http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0005http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0010http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0010http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0010http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0015http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0015http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0020http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0020http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0025http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0025http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0025http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0035http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0035http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0035http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0040http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0040http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0040http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0050http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0050http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0050http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0055http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0055http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0060http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0060http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0065http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0065http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0065http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0070http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0070http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0075http://www.eea.europa.eu/data-and-mapshttp://www.eea.europa.eu/data-and-mapshttp://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0085http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0085http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0085http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0100http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0100http://refhub.elsevier.com/S1462-9011(16)30438-5/sbref0100http://ec.europa.eu/digital-agenda/en/why-do-we-need-policy-making-30http://ec.europa.eu/digital-agenda/en/why-do-we-need-policy-making-30http://ec.europa.eu