Comparison of Methods to Assess the Sustainability

7/29/2019 Comparison of Methods to Assess the Sustainability

1/16

Comparison of Methods to Assess the Sustainabilityof Agricultural Systems: A Review

Christian Bockstaller, Laurence Guichard, Olivier Keichinger, Philippe Girardin, Marie-Batrice Galan,

and Grard Gaillard

Abstract Since the 1990s, numerous agri-environ-mental indicators and indicator-based methods havebeen developed to assess the adverse effects of crop-

ping and farming systems such as water pollution bynitrates and pesticides, and gaseous emissions due tonitrogen inputs. This wealth of environmental indica-tors and assessment methods based on indicators raisesissues on the quality of the methods and of the in-dicators, and on the relevancy of results. Evaluationand comparative studies are therefore needed to an-swer such issues. Here, we present four recent compar-ative studies selected for their illustrative value, first,to analyse the methodologies used for comparison ofmethods, and second, to highlight the main results of

the four comparisons. The first study involves 23 indi-cators to address nitrate leaching. The second study in-volves 43 indicators to address pesticide risk. The thirdand fourth studies compare environmental assessmentmethods based on a set of indicators used in French andUpper Rhine plains (France, Germany and Switzer-land). Both studies also compare the outputs of themethods and highlight the low degree of convergencebetween them. The approach proposed in the last studyis the most elaborate among the four case studies. Itcould be used to develop a generic evaluation and com-

parison methodology. The review of those four casestudies shows the need to formalise the methodologyunderlying any comparison work of indicators or eval-uation methods.

C. Bockstaller ()INRA, UMR 1121 Nancy-Universits INRA Agronomieet Environnement Nancy-Colmar, BP 20507, 68021 ColmarCedex, Francee-mail: [email protected]

Keywords Environmental assessment r Indicators r

Nitrate r Nitrogen r Pesticide

1 Introduction

During the 1990s, there was a growing concern forenvironmental issues in agriculture, e.g. water pol-lution by nitrates, pesticides, erosion, or more re-cently, greenhouse gas emissions and biodiversitylosses (Kirchmann and Thorvaldsson 2000). This ledto the demand for operational assessment tools consid-ered as a prerequisite to the development of new farm-

ing or cropping systems (Bockstaller et al. 1997). Thiswas favoured by the popularisation of the concept ofenvironmental management approaches like the ISO-14000 which rests on the four steps of the qualityspiral of continuous process improvement: to plan, todo, to check, to act (Meynard et al. 2002). The stepcheck requires an assessment method of environ-mental impacts. The use of indicators has appeared asan alternative to direct impact measurement (Mitchellet al. 1995; Bockstaller et al. 2008), and is linked tomethodological difficulties (impossibility of measure-

ment, complexity of the system) or practical reasons(time, costs) for carrying out direct measurements.Another reason is the use of such tools for prospec-tive goals (development of new agricultural strategies,prevention of environmental damage) in an ex anteassessment for which it is per definition not possibleto perform measurements.

An indicator explosion (Riley 2001a) has oc-curred for the last two decades with the developmentof numerous indicator-based methods which are aimed

E. Lichtfouse et al. (eds.), Sustainable Agriculture, DOI 10.1007/978-90-481-2666-8_47, 769c Springer Science+Business Media B.V. - EDP Sciences 2009. Reprinted with permission of EDP

Sciences from Bockstaller et al., Agron. Sustain. Dev. 29 (2009) 223235. DOI: 10.1051/agro:2008058
http://localhost/var/www/apps/conversion/tmp/scratch_1/[email protected]://localhost/var/www/apps/conversion/tmp/scratch_1/[email protected]


2/16

770 C. Bockstaller et al.

at assessing environmental impacts of agriculturalactivities, or the whole sustainability of agriculturalsystems (Rosnoblet et al. 2006). Among the workson indicators, one can distinguish those on a spe-cific theme, on one hand, like the environmental riskdue to pesticide application (Maud et al. 2001; Reus

et al. 2002; Devillers et al. 2005) and multi-criteriaapproaches based on a set of indicators addressingdifferent environmental issues, on the other. Exam-ples at regional (Payraudeau and van der Werf 2005),farm (Eckert et al. 2000; van der Werf and Petit 2002;Hlsbergen 2003; Meyer-Aurich 2005), and croppingsystem levels (Bockstaller et al. 1997; Lpez-Ridauraet al. 2005; Nemecek et al. 2005) can be given for both.

This multiplicity and variety of indicators and meth-ods raise questions. Riley (2001b) pointed out that it isa source of confusion which is increased by the fact

that many methods are not evaluated for their scien-tific relevance and feasibility. The potential user, eithera researcher working on innovative cropping systems,or an adviser working with farmers or a stakeholderinvolved in an environmental debate, will have ques-tions about the selection of a given method adaptedto his needs and how to make this selection. A sec-ond group of questions deals with the stability of theoutputs of the different methods: do they provide thesame conclusions? Answers to such questions requirean evaluation and comparison study which provides

information, not only about the strengths and draw-backs of each method, its field of use and validity,but also about the comparison of the conclusions de-rived from the outputs of the methods. Some authors(Meynard et al. 2002; Bockstaller et al. 2008) havealready pointed out the requirement of a comparativeanalysis and validation of the various indicators avail-able. To answer this concern of potential users, differ-ent kinds of comparative works have been undertaken.Comparison works of assessment methods based ona set of indicators, such as those at farm level (van

der Werf and Petit 2002; Halberg et al. 2005) or re-gional level (Payraudeau and van der Werf 2005) arebased on a descriptive approach. In other compara-tive studies on impact assessment (Thompson 1990;Hertwich et al. 1997) or more specific to the agricul-tural sector (Gebauer and Buerle 2000; Thomassenand de Boer 2005), authors use a set of qualitativeor semi-quantitative evaluation criteria to compare themethods. No information is given on the comparison

of the outputs or conclusions of the methods by allthose authors, except by Thomassen and de Boer 2005.They also study correlation between results of compa-rable indicators belonging to the inputoutput account-ing approach and Life Cycle Analysis for a datasetobtained on eight dairy farms.

This short review of the literature points out the di-versity of approaches and a lack of formalised com-parison methodology. The first goal of this article isto analyse the methodologies used in four compar-ative studies (CORPEN 2006; Devillers et al. 2005;Galan et al. 2007; Bockstaller et al. 2006), selectedto derive some methodological principles for potentialusers who need to undertake such a comparison. Sec-ond, the main results of the four comparisons will behighlighted to guide potential users of indicators or anevaluation method in their choice. Attention is paid to

agronomists working on the design of innovative crop-ping systems and to environmental impact due to pes-ticides and nitrogen issues, for which many indicatorsare available. The four case studies structuring the ar-ticle were selected for their diversity and illustrativevalue. The type of indicators and methods covered bythe case studies and their target users, agronomists as-sessing and designing cropping systems, was anotherreason for their selection.

2 Presentation of the Four Case Studies:

Context and Method of Comparison

2.1 Comparison of Indicators Assessing

Nitrogen Losses

2.1.1 Context of the Work

The work was initiated by the CORPEN organisation,which depends on the French Ministry for Ecology andSustainable Development and has the mission to bringtogether experts and stakeholders involved in the issueof water quality and agriculture in order to deliver rec-ommendations (CORPEN 2006). The objective was tohelp users to choose and to implement indicators de-pending on the question and the scale of study. It wascarried out by a group of experts on nitrogen fertiliza-tion and losses, from research and technical institutes.


3/16

Comparison of Methods to Assess the Sustainability of Agricultural Systems: A Review 771

The expert group listed 23 indicators currently usedby agricultural advisors, farmers or even public policyagents to assess nitrogen losses in France, and espe-cially nitrate leaching at farm and regional levels. Forthe sake of concision, we will restrict the presentationof this work to the field and farm scale since our arti-

cle addresses the evaluation of cropping and farmingsystems.

2.1.2 Method of Comparison

A descriptive sheet was filled in for each indicatorwith a list of descriptors: reference values, calculationmethod, time and spatial scale, periodicity of calcu-lation, time for implementation, recommendations forinterpretation and similar indicators, etc. In the reportof the project, a synthetic table was added to presentthe assessment of two evaluation criteria for 15 indica-tors: (1) the feasibility, i.e. easiness of implementationdue to accessibility of data and cost of implementa-tion expressed on a qualitative scale between 1 (low)and 4 (high), and (2) the relevance assessed by expertson a four-class scale, from 1 (indicator not to be im-plemented alone) to 4 (indicator advised). Indicatorsheets as well as the two evaluation criteria were filledin by members of the group of experts and validated bythe group of experts. A selection of descriptors and theassessment of the two criteria are presented in Table 3in Sect. 3.1 of the results chapter.

2.2 Comparison of 43 Pesticide

Risk Indicators


This work followed the studies of Maud et al. (2001)and Reus et al. (2002) who compared, respectively,six and eight pesticide risk indicators. The study wasordered by the French Ministry for Ecology and Sus-tainable Development and was expected to be as ex-haustive as possible to make the review available to alarge panel of users and to help the ministry to choosethe best indicators for the assessment of its policy(Devillers et al. 2005).


Each indicator was presented in a descriptive sheet,with a list of 25 criteria, a short presentation of thecalculation, and the list of the parameters and variablesused for calculation (Devillers et al. 2005). The follow-

ing criteria were used (1) some general descriptors onthe use, users and planned use, (2) others on the spa-tial scale, the environmental compartments taken intoaccount and the calculation method, (3) some informa-tion useful for assessing the qualities of the indicators,the calculation time, and the existence of a scientificvalidation procedure according to the framework ofBockstaller and Girardin (2003), and (4) finally, fourevaluation criteria expressed on a qualitative four-levelscale: , , +, ++ covering the readability, the feasi-bility, the reproducibility and the relevance for the end-

users. All the indicator sheets as well as the evaluationcriteria were filled in by the same person and validatedby a group of experts. Information sources were thereferences from grey and scientific literature. No im-plementation test was presented in this book. For thesake of concision, the number of indicators presentedin this article was reduced to a selection of indicatorschosen for their illustrative value or because they arealready implemented (see Table 4 in Sect.3.2 of theresults chapter).

2.3 Comparison of Five Assessment

Methods of Sustainability in France


This work was launched by a regional organisation,Agro-Transfert, at the request of the agricultural sec-tors representatives to develop a quality managementand environmental management approach in the Pi-

cardie region, North of France (Galan et al. 2007). Thefirst step was to develop a regional benchmark for goodfarming practice Qualiterre (Aubry et al. 2005).The second step (developed as an extension to theQualiterre programme) is the development of anenvironmental management system which is relevantand user-friendly. In order to have a state of the art ofthe existing tools and to choose the best fitted tool,Agro-Transfert performed a comparison of the five


4/16


methods used most frequently in France to evaluate theenvironmental impacts of agriculture. Those methodsare all based on a set of indicators addressing differentenvironmental themes, whereas the first method belowalso includes the economic and social dimensions ofsustainability. The five methods are:

(1) IDEA (Indicateur de Durabilit des ExploitationsAgricoles), which was developed by a workinggroup under the patronage of the French Ministryof Agriculture (Vilain et al. 2008).

(2) DIAGE (DIAgnostic Global dExploitation),which was developed by the Regional Federa-tion of Agricultural Cooperatives (FRCA) in theFrench Centre administrative region, in partner-ship with agricultural technical institutes (FRCA-Centre 2002).

(3) DIALECTE (DIAgnostic Liant Environnementet CTE), which was developed by the Solagroassociation (Solagro 2000) as well as the nextmethod.

(4) DIALOGUE (Diagnostic agri-environnementalglobal dexploitation), which addressed morethemes than DIALECTE at field level (Sola-gro 2001).

(5) INDIGOr (indicateurs de diagnostic global laparcelle), which was developed by the INRAsSustainable Agriculture Research Unit in Colmar

(Bockstaller et al. 1997).


As for previous work, a set of criteria was selectedby the authors to compare the methods: (1) generalcriteria: type of agricultural production evaluated, spa-tial scales, implementation time, target users, spread-ing and developers; (2) environmental themes andimpacts, (3) main activities, crop rotation, nitrogen

fertilization, etc., (4) aggregation levels, calculationmethod, rating scores and thresholds, and (5) type ofdata required (field data, management at farm level,sensitivity of the environment). Unlike the second casestudy on 43 pesticide indicators where each indicatorwas described and evaluated in a separate sheet, themethods are here compared directly in tables.

To get some of those data, e.g. implementation time,the authors tested the five methods on a set of 15 farmsin Picardie (all with cereals and sugar beet,C450 ewes

for 1 farm, C50 beef for 1 farm, C potatoes for 3farms, C vegetables for 1 farm, size ranging between93 and 460 ha). The results obtained with each methodon the 15 farms were compared in two ways:

For a single impact, the results for all 15 farms werecompared with each of the five tools. The effect ofcrop protection on water quality was selected.

For each method, the individual result for fourdifferent activities (management of inert waste,nitrogen fertilization, crop protection and energymanagement) within the water pollution themewere compared on one particular farm.

The results of the indicators were normalised byexpressing them as a percentage of the maximumpossible rating for the indicator, so that they can becompared (Nardo et al. 2005).

2.4 Comparison of Four Farm

Management Tools in the Upper

Rhine Plain (COMETE Project)


The last work was initiated in a transregional con-text, in the upper Rhine plain by French, German andSwiss partners in 2003. The French and Swiss methodswere compared with two German tools widely used inGermany. As in the previous study, the four selectedmethods based on a set of environmental indicators are:

1. INDIGOr, also compared in the previous project(see Sect. 2.3.2).

2. SALCA (Swiss Agricultural Life Cycle Assess-ment), developed at the Agroscope ART Recken-holz in Zurich (Switzerland), (Rossier and Gaillard2004).

3. KUL/USL (Criteria and Standards for SustainableAgriculture), developed at the state agriculturalinstitute of Thuringe in Iena (Germany), (Eckertet al. 2000).

4. REPRO, developed at the University of Halle(Germany), (Hlsbergen 2003).

The tools were assessed according to the versionvalid in mid-2004. For REPRO, only a subset of thewhole indicator set with high relevance for environ-mental items was analysed.


5/16



Since no adapted methodological framework forcomparison was found in the literature, the work-ing group of the COMETE project developed its ownapproach, which consists of two stages:

First, a comparative evaluation using a list of criteriawhich were grouped into three domains (scientificsoundness, feasibility and utility) (Table 1).

Second, the test of the implementation of the meth-ods in a set of 13 farms. For the first step, for eachcriterion, a score between 1 (the lowest) and 5 (thehighest) was defined by a set of decision rules. Anexample is given in Table 2, the details being avail-able in Bockstaller et al. (2006). The criteria ad-dressing the users needs and the whole list were

discussed during a workshop with the three iden-tified user groups: farmers, advisers and agents ofadministration.

The four methods were evaluated by the authorsthemselves for INDIGOr and SALCA and validated

by the whole working group. For the German methodsthe authors did not take part in the project, so the groupdecided to send the evaluation carried out by the Ger-man partner to the developers of the two methods. Thefeedback of the latter was validated by the workinggroup. The previous evaluation was completed by a

test of the methods on a group of 13 farms (three inSwitzerland, five in France and five in Germany) fortwo years. The type of production was various, arablefarms (maize monoculture, cereals), arable farms withspecial crops and mixed farms (arable crops and cattleor milk).

Following the evaluation with a set of criteria, theresults obtained on the group of farms were com-pared in two ways. First, an aggregated indicator wascalculated by means of an average value which wasweighted for SALCA according to the experience

gained by sensitivity analysis by the authors, withouta weighting procedure for INDIGOr and KUL, anda sum of scores for REPRO. The ranking of farmsobtained with each aggregated indicator was com-pared by means of Spearmans correlation coefficient.

Table 1 List of evaluation criteria used in the COMETE project (Bockstaller et al. 2006)

Scientific soundness Feasibility Utility

Coverage of environmental issues Accessibility of dataa Coverage of needsa

Coverage of agricultural Qualification of user Clearness of conclusion from resultsproduction branches

Coverage of production factor Need for external support Quality of communication of resultsIndicator type,b depth of User-friendliness

environmental analysisAvoidance of incorrect conclusions Integration with existing farming softwareTransparency Time requirementaFor three user groups: farmers, advisers, administrationbBased on the driving-force, pressure, state, impact, response framework (EEA 2005)

Table 2 Example ofassessment for the criterion:avoidance of incorrectconclusions

Decision rules for the assessment of the Scorecriterion avoidance of incorrect conclusions (15)Lack of data on evaluation of the indicator and 1

and criteria indicator typeD 1Indicator based on a non-validated model 1No agreement of indicator value with observed data 1

Indicator criticised in a peer-reviewed article 2Indicator based on a partially validated model 2Lack of data on evaluation of the indicator and 2

and criteria indicator typeD 2 to 5

Medium agreement of indicator value with observed data 3Calculation method recommended by experts 3

Scientific peer-reviewed article on the indicator 4Indicator based on a validated model 4

Good agreement of indicator value with observed data 5


6/16


Second, the conformity of recommendation derivedfrom the indicators was compared by means of a newlydeveloped index of conformity (IK) ranging between 0(no conformity between the methods) and 1 (total con-formity between the methods):

IK D 1 24 X

pD1n

XqD1b

XrD1vk

ipqr jpqr

=.2nb/

35

with:

ipqr: degree of achievement of recommendation r forthe production factor q for farm p for method 1;jpqr: degree of achievement of recommendation r forthe production factor q for farm p for method 2;n, b, vk: respectively, number of farms, production fac-tors and recommendations per production factor.

For example, the production factor nitrogen man-agement was decomposed into recommendations likereduce the amount of fertilizer; increase the amount

of fertilizer, change the type of fertilizer andchange the date, method of fertilization. If a methodgives the recommendation reduce the amount of fer-tilizer, the degree of achievement will be 1 for thisrecommendation and 0 for the other recommendation.It should be noticed that a value inferior to 1 can be

given if more than one recommendation is given.

3 Main Results of the Four Case Studies

3.1 Comparison of Indicators Assessing

Nitrogen Losses

Several groups of indicators can be distinguished

in Table 3: (1) a first group of simple indicators(Bockstaller et al. 2008) focusing on nitrogen inputmanagement, mainly mineral/organic fertilization, but

Table 3 Comparison of nitrogen indicators (CORPEN 2006)

Time for AgronomicIndicator Spatial scale Threshold value interpretation relevance Feasibility

FertilizationAmount of applied nitrogen Field/farm/region Local per crop Year 1 4Amount of available nitrogen Field/farm/region Local per crop Year 1 3Number of nitrogen applications Field/farm/region Local per crop Year 1 3

(organic and mineral)Deviation from the recommendation Field Zero Year 2 3

of nitrogen ratePeriod of application Field/farm/region Local Year 1 3Number of grazing days Field/farm/region Local Year 3 2

Soil coverArea with bare soil during drainage Farm/region Local 34 years 2 4

periodArea with catch crops Farm/region Local 34 years 2 4Assessment of surpluses or lossesInput/output budget (CORPEN) Field/farm/region Local per >5years 2 3

croppingsystem

N supply / requirement budget Field/farm/region Local: close to zero Year 3 2

(EQUIF)Soil mineral nitrogen at harvesta Field Local per soil type Year 3 2Soil mineral nitrogen at beginning Field Local per soil type Year 4 2

wintera

Model predicting N losses: IN Field/farm/region 7 (matching a YearINDIGOr concentration 4 3b

below roots of50 NO3 mg L1)

Model predicting nitrate lixiviation: Field/farm/region Local per Year 4 3b

DEAC cropping systemaMeasured, or assessed by a modelbWhen the parametrization has been achieved


7/16


also organic input due to grazing. They are consid-ered as descriptors of practices; (2) a second groupaddressing soil cover in winter, assessing nitrogen up-take during the period after harvest until winter, and(3) a third group resulting from the combination ofvariables such as nitrogen balance or model-based.

Some of them are based on calculation of the inputoutput balance to estimate surplus. Others include ni-trogen cycle processes to estimate fluxes/emission ofnitrogen. Among them, the nitrogen indicator fromthe INDIGOr method (IN/, based on an operationalmodel, provides the amount of nitrogen lost to wa-ter and air (Bockstaller et al. 2008), whereas DEACfocuses on nitrate leaching in winter (Cariolle 2002).The evaluation of the relevance and feasibility showsa relative discrepancy between the feasibility and rele-vance for the first and the last group in Table 3. Indi-

cators from the first group are straightforward to cal-culate (high feasibility) but not really relevant if theyare used alone. In contrast, indicators including in theirequation nitrogen cycle processes gain in relevancy tothe detriment of feasibility. In the description sheet ofeach indicator, recommendations are given to the usersabout interpretation of results and the domain of valid-ity, and propositions of complementary indicators aregiven to improve the relevance of the first group. Anexample can be given for indicators based on the cal-culation of a balance (input minus output) used by sev-

eral authors and institutions as an indicator for nitrogenlosses (e.g. Goodlass et al. 2003; EEA 2005). However,several authors (Lord et al. 2002; Oenema et al. 2005;ten Berge et al. 2007) pointed out by comparison withmeasurements of nitrate leaching that such nitrogenbalances are bad estimators of nitrate leaching risk, ifthey are used on an annual basis (Laurent et al. 2000).Thus, the report recommended an interpretation basedon pluriannual calculation.

3.2 Comparison of 43 Pesticide

Risk Indicators

The output of the work was a book describing the 43indicators, 24 in a detailed way and 19 in a simplifiedway. Several groups of indicators can be distinguished:(1) indicators resulting from transformation of vari-ables into scores and summed up or aggregated in an

empirical way, among them EIQ, one of the first indi-cators published (Table 4); (2) a second group of indi-cators uses outputs from model calculation. 14 indica-tors among the 43 are based on the risk ratio approachwhich is used in registration of pesticides (Vercruysseand Steurbaut 2002): it is the quotient of the estimated

human exposure or predicted concentration and toxico-logical reference value used for different environmen-tal compartments, e.g. EPRIP, POCER. (3) The thirdgroup contains specific approaches such as the qualita-tive one based on decision rules associated with fuzzylogic (e.g. I-Phy) or based on a multicriteria rankingmethod (Vaillant et al. 1995; Aurousseau 2004).

Other trends which can be pointed out throughthis comparison is the lack of indicators which werevalidated by comparison with experimental data (12among the 43), only one (EYP) being validated by

end-users (Bockstaller and Girardin 2003). Most of theindicators are calculated on the field scale and only3 among 43 on the watershed scale, which is rele-vant for assessment of surface water quality. The im-plementation of the indicator requires in general lessthan 1 h per calculation, except for EPRIP and EYP,which need more time because of the high number ofdata for calculation. Only 8 among 43 propose refer-ence values which help users in the interpretation ofthe outputs. No specific focus was put on the use of theindicators.

3.3 Comparison of Five Assessment

Methods of Sustainability in France

The first part of the work is descriptive. A synthesis ofthe results is given in Table 5. Besides general infor-mation, Galan et al. (2007) assess on a qualitative scalethe degree of coverage of environmental themes andfarm activities (practices) at field as well at farm level.For the first item, water quality (sporadic pollution),air quality and social environment (noise, odours)are not covered by a majority of methods, whereas forthe second item, most of the methods neglect or poorlyintegrate the activities construction/modificationof buildings or storage, production of renewableenergy and management of inert waste. Additionalinformation is given on the type of data needed forwhich INDIGOr differentiates from the others by


8/16


Table

4

Examplesfrom

thecomparisonof43pesticideriskindicators(Dev

illersetal.2005)

Indicator

Developer/

reference

Targetuserb

Spatialscale

Environ.

compartm

ent

addresseda

Calculation

method

Validation

(D,

O,U

)cTimefor

da

ta(for

calculation)Readability

dFeasibility

d

Reproducibility

d

Relevance

foruser

ADSCOR

OCDEgroup

DM

Field/region/

country

Sw

Sumofscores

D,

O


9/16


10/16


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Farm number

INDIGO

DIAGE

IDEA

DIALECTE

DIALOGUE

Tool grade (% of the maximum possible rating for the given tool)

Fig. 1 Comparison of the output of pesticide indicators for water quality from five assessment methods. Indicators are calculatedon 15 farms and their outputs are normalised as a percentage of the maximum value (Galan et al. 2007)

using detailed field practice data and data on the sensi-tivity of the environment, soil and climate, but no sitedata such as maintenance of the storage tank or sprayer,or building management. About the aggregation of theinformation, most of the methods use a simple methodbased on the sum of scores, and product (for DIAGE),whereas indicators in INDIGOr are based on modelsand expert systems (Bockstaller et al. 2008).

The authors go a step further by comparing thefive assessment methods for water quality. They com-pare the impact of pesticide use on 15 farms. The

normalised values of the pesticide indicators are rep-resented in Fig. 1. All the methods except DIAGE, andDIALOGUE to a lesser extent, show significant vari-ations between farms. IDEA yields in general higherresults, showing less impact on water quality, thanthe other methods. In any case, no correlation be-tween methods appears on the sample of farms, whichmeans that the recommendations for pesticide man-agement will not be the same between methods fora given farm. This can be explained by the differ-ence between methods in: (1) the integration of as-

pects of sporadic pollution (point source), as it is thecase for IDEA and DIAGE, (2) type of data used,pesticide properties (INDIGOr and DIALOGUE),and soil and environment sensitivity (INDIGOr andDIAGE), and (3) the aggregation method. Similardiscrepancies between the five methods are foundfor one particular farm when they are compared onfour different activities (management of inert waste,nitrogen fertilization, crop protection and energymanagement).

3.4 Comparison of Four FarmManagement Tools in the Upper

Rhine Plain (COMETE Project)

Based on the versions available in mid-2004 for thefour methods and on a subset of indicators for RE-PRO, the results yielded by each method for the 15criteria are shown in Fig. 2. For the domain scientificsoundness, SALCA presents the best environmentalscores, but none of the methods was able to cover

all relevant environmental issues, especially regardingbiodiversity. The low scores of INDIGOr for the cri-teria coverage of agricultural production and con-sideration of production factors result from its spe-cialisation in plant production. However, this methodallows a detailed analysis of a cropping system, en-abling the user to trace the cause of an environmen-tal risk to the management, e.g. risk analysis of eachpesticide application, taking into account the field con-ditions, tillage, spraying techniques and active ingredi-ent properties. The depth of environmental analysis

is low for REPRO due to the fact that this method con-siders for each environmental issue all types of indica-tors without priority despite the risk of redundancy be-tween them; and for KUL, due to the type of indicator(mainly only driving forces). Those take into accountonly farmers practices and not emissions or impacts.The low score of KUL/USL for the criterion trans-parency reflects the non-accessibility of the software,which is balanced by the score in the domain feasibil-ity for which KUL/USL receives the best score as a


11/16


1

2

3

4

5

coverageo

fenvironmentalissues

coverageofagricultural

productionbranches

coverageof

productionfactors

indicatortype(DPSIR),

depthofenvironmental

analysis

avoidanceof

incorrectconclusions

transparency

accessibilityofdata

qualificationofuser

ne

edforexternalsupport

user-friendliness

in

tegrationwithexisting

farmingsoftware

timerequirement

coverageofneeds

clearnessofconclusion

fromr

esults

qualityofco

mmunicationofresults

Feasability UtilityScientific soudness

INDIGO SALCA KUL REPRO

Fig. 2 Comparison of four farm management tools in the upper Rhine plain with the help of 15 criteria (see Table 1) in the frameof the COMETE project (Bockstaller et al. 2006)

result of its cleverly devised organisation form. On thecontrary, SALCAs electronic entry data form was notuser-friendly. The evaluation with REPRO is compara-

tively more time-consuming. For the domain utility,no great differences were observed between the fourmethods. The better score of KUL/USL is due to thecriterion communicability thanks to the possibilityof labelling, which is compensated for by the lack ofspecific recommendations at field level.

There was a high correlation between SALCA, RE-PRO and INDIGOr (not enough farms for KUL/USL)regarding the environmental ranking of the analysedfarms (Spearman coefficients range between 0.72 and0.88, see Fig. 3a). In other words, for the four methods,

there is no reason to fear that the choice of the environ-mental management tool determines whether a farmperforms well or badly from an environmental point ofview. On the other hand, the conformity index showsa low convergence between the recommendations forthe four methods (index range between 0.48 and 0.64,see Fig. 3b).

These discrepancies are explained by major con-ceptual differences between the investigated methods,namely: (1) in the different environmental issues

considered. This can be illustrated by the phosphorusmanagement: INDIGOr addresses soil fertility issueswhich can lead to a recommendation increase the

amount of fertilizer, whereas SALCA focuses oneutrophication (of soil and water) and environmentalsoil quality aspects (here linked to heavy metalspresent in some fertilizers). Provided that a minimalyield is reached, SALCA does not recommend from anenvironmental point of view to increase the amount offertilizer, whereas INDIGO can do it to maintain soilfertility. (2) In the production factors which are usedfor the calculations of indicators dealing with similarissues. INDIGOr and SALCA take into accountamount of nitrogen, crop management, e.g. soil cover

in winter, and soil mineralisation to assess nitrateleaching, whereas KUL and REPRO, for the indicatorconsidered in the study, only take into account nitro-gen input and output, and (3) to a lesser extent in thebenchmark used to derive a recommendation for somesimilar indicators.

Besides the evaluation with criteria and the com-parison of outputs, some general qualitative aspectswere pointed out through the experience gained by im-plementing the method on farms. Two deserve more


12/16


b Conformity index (Ik)

Ik=0.66

SALCA

INDIGO

REPRO

Ik=0.46

Ik=0.59

SALCA

INDIGO

REPRO

a Spearman correlation (rs)

rs=0.75 rs=0.79

rs=0.76

Fig. 3 Comparison of outputs based on (a) the ranking of farmsby means of the Spearman correlation coefficient (rs/, (b) theconformity of recommendations by means of the conformity in-dex (Ik/, (Bockstaller et al. 2006). A value 1 indicates a perfect

correlation for rs and conformity for Ik . Both comparisons aremade with a sample of 13 farms with the data of 2002 (KUL notincluded in the comparison because it was implemented on threefarms only)

attention. The implementation of a method outside thecountry where it was developed raises several prob-lems such as the accessibility of data or different de-scription schemes for the same issue (especially forsoil description) and bugs in the software due to na-tional parametrization. On the other side of the chain,the user stands alone for the interpretation of resultsand is not provided by any methods with an interpreta-tion system in the software to interpret the results ex-cept for KUL. In this case, the user receives a written

report with the interpretation and recommendations toimprove the system. However, the user does not haveaccess to the calculations and has to pay for thoserecommendations.

4 Discussion

In this discussion we will not discuss the resultsobtained for each method but focus on the methodol-ogy used to compare and evaluate assessment meth-ods or thematic indicators. First, it should be noticedthat if such a study is in many cases user-oriented, itcan also help indicator or method developers to im-prove their methods. For example, the work on pesti-cide risk indicators was followed by a second projecton indicator validation (Girardin et al. 2007) and onthe improvement of two of them, e.g. introduction ofa risk component on biodiversity. The developers ofSALCA took into account the poor assessment of their

method according to the criteria integration with ex-isting farming software and user-friendliness (seeFig. 2) for the SALCA version of mid-2004. They in-tegrated the use of commercial farm management soft-ware for the data collection and the implementation ofa new software program for data validation and prepa-ration before calculation, for the last two years. Thecomparison of the five assessment methods in the thirdcase study led the authors to develop a new methodmore fitted to the need of the local users.

In Table 6 we synthesise the main features of thecomparison and evaluation approaches used for thefour case studies of this article. It highlights the vari-ability between the approaches, explainable by a lackof a generic methodology. The criteria and their organ-isation vary between the case studies. Criteria on feasi-bility and relevance (or soundness) can be found in thefour cases. This can be compared with previous stud-ies. Hertwich et al. (1997) proposed only three criteria:information requirement, tolerance for imperfectinformation and potential for undesirable outcome.

Other authors such as Gebauer and Buerle (2000) orThomassen and de Boer (2005) developed a longerlist organised, respectively, into different groups: im-plementation and utility, and, relevance for user,quality and availability of data. Other compar-ative studies remained mainly descriptive, includinginformation on the time needed for data collectionand recommendations on the type of indicators andlinked issues, e.g. choice of threshold, scale of re-sult expression (van der Werf and Petit 2002; Halberg


13/16


Table 6 An overview of approaches used to compare indicators and assessment methods in the four case studies

et al. 2005; Payraudeau and van der Werf 2005). Suchdescriptive comparison studies allow the users to knowthe construction methods better, and to appropriate thetools and complete the evaluation step which high-lights strong and weak points of each method.

It should be noticed that the cost of implementa-tion is not used in the four case studies or by all theauthors previously quoted, although it is an importantcriterion (Romstad 1999). This can be explained by thefact the studied methods were at an experimental stage,and that most costs are internalised by the method de-velopers so that no realistic assessment of this criterioncould be achieved.

From the list of criteria presented in Table 6 or usedby other authors, it appears that the meaning of theword can in some cases vary between authors. Withregard to the feasibility, Hertwich et al. (1997), likeThomassen and de Boer (2005), linked it mainly tothe availability of data, whereas it covers more aspectsin the fourth case study (COMETE project), like inthe work ofGebauer and Buerle (2000). Even withina working group like this of the CORPEN, the as-sessment of the criterion relevance was not so easy.It refers to a synthesis or even compromise of cri-teria such as sensitivity, representativeness, legibilityand robustness, which are not so easy to specify. Thisexplains the reason why the group of the COMETEproject prefers to increase the number of criteria with

the risk of providing too much information to the user.A solution to this inflation of criteria would be to syn-thesise the outcome of the evaluation with a multi-criteria analysis, as was proposed for social validationof indicators (Cloquell-Ballester et al. 2006).

The objective of the CORPEN group (CORPEN2006) was to guide the users in the selection of indi-cators addressing the nitrogen leaching issue in orderto avoid misuse outside the domain of use, or mis-interpretation. In the study of the CORPEN group,an evaluation of indicators is briefly presented in themain text but no criteria are given in the descrip-tive sheet, whereas a synthesis in the form of textbut no comparative tables are given in the book ofDevillers et al. (2005). A database with queries tohelp to choose a pesticide indicator is in develop-ment (Girardin, personal communication). The thirdcomparative case study (Galan et al. 2007) providesseveral tables comparing the French assessment meth-ods for their technical features regarding their calcula-tion method, the domain of use, etc., which could beused for an evaluation work. The time for implemen-tation is quantified but not valued like in the last casestudy, the COMETE project (Bockstaller et al. 2006).The last case study, the COMETE project, clearly dif-ferentiates description and evaluation and proposes amethod based on a set of criteria with decision rulesto assess them (see Table 2). This should increase the


14/16


transparency. However, a degree of subjectivity mayremain in the criteria of the COMETE project as somecriteria are the results of a scoring procedure withoutdecision rules, e.g. coverage of an environmental is-sue. The cross-validation which was done in the projectcould help to reduce the subjectivity. Another point to

notice is the effort to make the evaluation more preciseby differentiating different user groups. The authors ofthe COMETE project (Bockstaller et al. 2006) iden-tify three groups (farmers, advisers and employers ofadministration) which are differentiated for the evalu-ation of two criteria, accessibility of data and coverageof needs. This was also done by Thomassen and deBoer (2005), who added a fourth group of scientists tothe three groups for one criterion, comprehensibility.A criterion such as accessibility of data also has to beadapted to the context of use. Some data, such as those

describing soils, vary a lot between countries or evenregions (Bockstaller et al. 2006).

An interesting output of the third case study (Galanet al. 2007) is the comparison of the outputs of themethods, which is rarely done according to our knowl-edge. Examples can be found in the literature on com-parison of outputs for pesticide risk indicators (Maudet al. 2001; Reus et al. 2002). However, those authorscompared the ranking of pesticides but did not takeinto account the absolute value of the indicator, so thatthe actual difference between the results of two indi-

cators is not assessed. In the work of Galan et al., as-sessment methods based on different sets of indicatorsare compared. Consequently, Galan et al. (2007) re-stricted the analysis to comparisons farm by farm orindicator by indicator. In the COMETE project, resultsof the individual indicators are aggregated althoughthe developers (except for REPRO) do not proposeit for users because of methodological problems due,for example, to the addition of scores (Schrlig 1985).The second approach based on a conformity indexis original and avoids this problem. However, it re-

quires an effort of formalisation of the potential rec-ommendations for each indicator within an evaluationmethod. Comparisons of outputs in Galan et al. (2007),like the comparison of recommendations in COMETE,yielded poor convergence between the compared meth-ods, which can be explained by the ground differencein assumptions and choices in the calculation methods.The potential users should be aware of this, which isonly possible if those assumptions are transparent.

5 Conclusion

This article highlights through the four case studies thevariability in approaches used to compare indicators orassessment methods. The first two studies focus on, re-spectively, 23 and 43 indicators addressing the nitrate

leaching issue and pesticide risk, respectively. Thosestudies provide a lot of descriptive information aboutthe indicators summarised in the article. Few evalu-ation criteria are used to point out strong and weakpoints of those indicators. The third and fourth stud-ies compare environmental assessment methods basedon indicators, respectively, five used in France andfour tested in the upper Rhine plain (France, Germanyand Switzerland, COMETE project). Both studies alsocompare the outputs of the methods and highlight alow degree of convergence among them. The approach

developed in the COMETE project appears to be themost elaborate. It should be tested in other compara-tive studies like the third case study. An adaptation tothe comparison of pesticide risk indicators is ongoingin the Endure network (Kgi et al. 2008).

Our study can contribute to developing a meta-method which should help with the selection ofindicators or of assessment methods. Such a meta-method could rest on a list of criteria like thoseof COMETE which would require local adaptation:which criteria are relevant for a given context, but alsohow they should be assessed, e.g. availability of soildata, which can change between countries or even re-gions. It should include descriptive information, evalu-ation criteria based not only on theoretical informationbut also on a test in practice. Basic assumptions, thepotentialities of the methods, e.g. environmental issuescovered, factors addressed, should in any case be statedclearly because they strongly influence the final resultsand explain the divergence between methods in termsof recommendations. Further work is needed to helpusers to cope with those potential discrepancies be-tween indicators for the same issue, or between assess-ment methods.

Acknowledgment The comparison of 43 pesticide indica-tors was sponsored by the French Ministry for Ecology andSustainable Development. The comparison work in Picardiereceived financial support from the Picardie administrative re-gion (Conseil Rgional de Picardie) and the French govern-ments ADEME agency (Agence pour lEnvironnement et laMatrise de lnergie). The research within the COMETEproject was partly funded by the ITADA (Institut transfrontalier


15/16


dapplication et de dveloppement agronomique) which is spon-sored by the EU (Programme INTERREG 3) as well as the SwissConfederation and the cantons of Basel-Stadt, Basel-Land andAargau.

References

Aubry C., Galan M.B., Maz A. (2005) Garanties de qualitdans les exploitations agricoles: exemples de llaborationdu rfrentiel QualiTerre R en Picardie, Cah. Agric. 14,313321.

Aurousseau P. (2004) Agrgation des paramtres et bases math-matiques de combinatoire de facteurs de risque, in: BarriusoE. (Ed.), Estimation des risques environnementaux des pesti-cides, INRA Editions, Paris, pp. 5874.

Bockstaller C., Girardin P. (2003) How to validate environmentalindicators, Agr. Syst. 76, 639653.

Bockstaller C., Girardin P., Van der Werf H.G.M. (1997) Use

of agro-ecological indicators for the evaluation of farmingsystems, Eur. J. Agron. 7, 261270.

Bockstaller C., Gaillard G., Baumgartner D., FreiermuthKnuchel R., Reinsch M., Brauner R., Unterseher E. (2006)Mthodes dvaluation agri-environnementale des exploita-tions agricoles : Comparaison des mthodes INDIGO,KUL/USL, REPRO et SALCA, ITADA, Colmar, p. 112.

Bockstaller C., Guichard L., Makowski D., Aveline A., GirardinP., Plantureux S. (2008) Agri-environmental indicators to as-sess cropping and farming systems. A review, Agron. Sus-tain. Dev. 28, 139149.

Cariolle M. (2002) DEAC Nitrogen: means to diagnosenitrogen leaching on a mixedfarm scale, in: Proceedingsof the 65th Institut International de Recherches Betterav-ires Congress, Brussels, Belgium, 1314 February 2002,pp. 6774.

Cloquell-Ballester V.A., Monterde-Diaz R., Santamarina-Siurana M.C. (2006) Indicators validation for theimprovement of environmental and social impact quan-titative assessment, Environ. Impact Assess. Rev. 26,79105.

CORPEN (2006) Des indicateurs AZOTE pour grer des ac-tions de matrise des pollutions lchelle de la parcelle,de lexploitation et du territoire, Ministre de lcologieet du Dveloppement Durable, http://www.ecologie.gouv.fr/IMG/pdf/maquette_azote29_09.pdf, Paris, p. 113.

Devillers J., Farret R., Girardin P., Rivire J.-L., Soulas G.

(2005) Indicateurs pour valuer les risques lis lutilisationdes pesticides, Lavoisier, Londres, Paris, New York.Eckert H., Breitschuh G., Sauerbeck D. (2000) Criteria and stan-

dards for sustainable agriculture, J. Plant Nutr. Soil Sci. 163,337351.

EEA (2005) Agriculture and environment in EU-15; the IRENAindicator report, European Environmental Agency (EEA),Copenhagen (Danemark), p. 128.

FRCA Centre (2002) DIAGE, manuel dutilisation et logiciel.Galan M.B., Peschard D., Boizard H. (2007) ISO 14 001 at the

farm level: analysis of five methods for evaluating the envi-ronmental impact of agricultural practices, J. Environ. Man-age. 82, 341352.

Gebauer J., Buerle A.S. (2000) Betriebliche Umweltinforma-tionstechniken fr die Landwirschaft, Ber. Landwirtsch. 78,354392.

Girardin P., Devillers J., Thybaud E., Soulas G. (2007) Pro-gramme Indicateurs et pesticides Phase II : Validation etproposition damlioration dindicateurs pesticides, Min-istre de lcologie et du Dveloppement Durable, p. 71.

Goodlass G., Halberg N., Verschuur G. (2003) Input output ac-counting systems in the European community an appraisalof their usefulness in raising awareness of environmentalproblems, Eur. J. Agron. 20, 1724.

Halberg N., van der Werf H.M.G., Basset-Mens C., Dalgaard R.,de Boer I.J.M. (2005) Environmental assessment tools for theevaluation and improvement of European livestock produc-tion systems, Livest. Prod. Sci. 96, 3350.

Hart A., Brown C.D., Lewis K.A., Tzilivakis J. (2003) p-EMA(II): evaluating ecological risks of pesticides for a farm-levelrisk assessment system, Agronomie 23, 7584.

Hertwich E.G., Pease W.S., Koshland C.P. (1997) Evaluating theenvironmental impact of products and production processes:a comparison of six methods, Sci. Total Environ. 196, 1329.

Hlsbergen K.J. (2003) Entwicklung und Anwendung einesBilanzierungsmodells zur Bewertung der Nachhaltigkeitlandwirtschaftlicher Systeme, Shaker (Halle, Univ., Habil.-Schr., 2002), Aachen.

Kgi T., Bockstaller C., Gaillard G., Hayer F., Mamy L., Strasse-meyer J. (2008) Multicriteria evaluation of RA and LCAassessment methods considering pesticide application, Eu-ropean Network for Durable Exploitation of crop protec-tion strategies (ENDURE), Internal report, p. 43, http://www.endure-network.eu/.

Kirchmann H., Thorvaldsson G. (2000) Challenging targets forfuture agriculture, Eur. J. Agron. 20. 12, 145161.

Kovach J., Petzoldt C., Degni J., Tette J. (1992) A method to

measure the environmental impact of pesticides, New YorksFood and Life Sciences Bulletin, 8 p.Laurent F., Verts F., Farrugia A., Kerveillant P. (2000) Effets

de la conduite de la prairie pture sur la lixiviation du ni-trate. Proposition pour une matrise du risque la parcelle,Fourrages 164, 397420.

Lpez-Ridaura S., van Keulen H., van Ittersum M.K., LeffelaarP.A. (2005) Multi-scale methodological framework to derivecriteria and indicators for sustainability evaluation of peasantnatural resource management systems, Environ. Dev. Sus-tain. 7, 5169.

Lord E.I., Anthony S.G., Goodlass G. (2002) Agricultural nitro-gen balance and water quality in the UK, Soil Use Manage.18, 363369.

Maud J., EdwardsJones G., Quin F. (2001) Comparative eval-uation of pesticide risk indices for policy development andassessment in the United Kingdom, Agr. Ecosyst. Environ.86, 5973.

Meyer-Aurich A. (2005) Economic and environmental analysisof sustainable farming practices a Bavarian case study, Agr.Syst. 86, 190206.

Meynard J.M., Cerf M., Guichard L., Jeuffroy M.H., MakowskiD. (2002) Which decision support tools for the environmen-tal management of nitrogen? Agronomie 22, 817829.

Mitchell G., May A., Mc Donald A. (1995) PICABUE: amethodological framework for the development of indicators
http://www.endure-network.eu/http://www.endure-network.eu/http://www.endure-network.eu/http://www.endure-network.eu/http://www.endure-network.eu/


16/16


of sustainable development, Int. J. Sust. Dev. World 2,104123.

Nardo M., Saisana M., Saltelli A., Tarantola S. (2005) Toolsfor composite indicators building. Joint Research Center,European Commission, Ispra (Italy), p. 134.

Nemecek Th., Huguenin-Elie O., Dubois D., Gaillard G. (2005)kobilanzierung von Anbausystemen im schweizerischen

Acker- und Futterbau, Schriftenreihe der FAL 58, AgroscopeFAL Reckenholz, 155 p., Zurich.Oenema O., van Liere L., Schoumans O. (2005) Effects of low-

ering nitrogen and phosphorus surpluses in agriculture on thequality of groundwater and surface water in the Netherlands,J. Hydrol. 304, 289301.

Payraudeau S., van der Werf H.M.G. (2005) Environmental im-pact assessment for a farming region: a review of methods,Agr. Ecosyst. Environ. 107, 119.

Reus J., Leendertse P.C. (2000) The environmental yardstick forpesticides: a practical indicator used in the Netherlands, CropProt. 19, 637641.

Reus J., Leenderste P., Bockstaller C., Fomsgaard I., GutscheV., Lewis K., Nilsson C., Pussemier L., Trevisan M., vander Werf H., Alfarroba F., Blmel S., Isart J., McGrath D.,Seppl T. (1999) Comparing environmental risk indicatorsfor pesticides. Results of the European CAPER project. Cen-tre for Agriculture and Environment, Utrecht, p. 183.

Reus J., Leenderste P., Bockstaller C., Fomsgaard I., GutscheV., Lewis K., Nilsson C., Pussemier L., Trevisan M., vander Werf H., Alfarroba F., Blmel S., Isart J., McGrath D.,Seppl T. (2002) Comparing and evaluating eight pesti-cide environmental risk indicators developed in Europe andrecommandations for future use, Agr. Ecosyst. Environ. 90,177187.

Riley J. (2001a) The indicator explosion: local needs and inter-national challenges, Agr. Ecosyst. Environ. 87, 119120.

Riley J. (2001b) Indicator quality for assessment of impactof multidisciplinary systems, Agr. Ecosyst. Environ. 87,121128.

Romstad E. (1999) Theorical considerations in the develop-ment of environmental indicators, in: Brouwer F.M., Crab-tree J.R. (Eds.), Environmental indicators and agriculturalpolicy, CAB International, Wallingford, pp. 1323.

Rosnoblet J., Girardin P., Weinzaepflen E., Bockstaller C. (2006)Analysis of 15 years of agriculture sustainability evalua-tion methods, in: Fotyma M., Kaminska B. (Eds.), 9th ESACongress, Warsaw, Poland, pp. 707708.

Rossier D., Gaillard G. (2004) kobilanzierung des Land-wirtschaftsbetriebs: Mthode und Anwendung in 50 Land-wirtschaftsbetrieben, 53, p. 142.

Schrlig A. (1985) Dcider sur plusieurs critres. Panorama delaide la dcision multicritre, Presses polytechniques etuniversitaires romandes, Lausanne.

Solagro (2000) DIALECTE, Diagnostic Liant Environnementet Contrat Territorial dExploitation ; manuel dutilisation etlogiciel.

Solagro (2001) DIALOGUE, Diagnostic Agri-environnementalGlobal dexploitation agricole ; manuel et logiciel.

ten Berge H.F.M., Burgers S.L.G.E., Van der Meer H.G.,Schrder J.J., Van der Schoot J.R., Van Dijk J.R. (2007)Residual inorganic soil nitrogen in grass and maize on sandysoil, Environ. Pollut. 145, 2230.

Thomassen M.A., de Boer I.J.M. (2005) Evaluation of indica-tors to assess the environmental impact of dairy production

systems, Agr. Ecosyst. Environ. 111, 185199.Thompson M.A. (1990) Determining Impact significance inEIA: a review of 24 methodologies, J. Environ. Manage. 30,235250.

Vaillant M., Jouany J., Devillers J. (1995) A multicriteria esti-mation of the environmental risk of chemicals with the SIRISmethod, Toxicol. Model. 1, 5772.

van der Werf H.G.M., Petit J. (2002) Evaluation of environmen-tal impact of agroculture at the farm level: a comparison andanalysis of 12 indicator-based methods, Agr. Ecosyst. Envi-ron. 93, 131145.

van der Werf H.M.G., Zimmer C. (1998) An indicator of persti-cide environmental impact based on a fuzzy expert system,Chemosphere 36, 22252249.

Vercruysse F., Steurbaut W. (2002) POCER, the pesticide oc-cupational and environmental risk indicator, Crop Prot. 21,307315.

Vilain L., Boisset K., Girardin P., Guillaumin A., Mouchet C.,Viaux P., Zahm F. (2008) La mthode IDEA: indicateur dedurabilit des exploitations agricoles: guide dutilisation, 3rdedition, Educagri, Dijon.

Documents

Comparison of Methods to Assess the Sustainability