Evaluation of Rural Development Requires Clarity on Expected Outcomes L’évaluation du développement rural demande à ce que les résultats attendus soient clairement identifiés

They say of number 24 buses thatyou wait for ages, then several comealong at once. For me, the subject ofevaluation has been like that recently.In just the last few weeks: I wasasked to evaluate the rural andenvironmental progress of agovernment; I found myselfchallenged to say how sustainabledevelopment could be evaluated; andI chaired a conference on PostOccupation Evaluation (POE) ofbuildings.

ƒ‘‘Ohne eine gute

Evaluation verzichtet

man bewusst auf

Information, anhand

derer zukunftige

Politikmaßnahmen

ausgewahlt werden

sollten.,,POE is an essentially technical processthat helps ensure that a new buildingonce delivered works well – in termsof construction, capability of use andadaptability by future users. Thecrucial thing is to be absolutely clearabout the outcome you want beforestarting the project. Does the buildingdo what the users want, economically;and will it be adaptable in use to servethe needs of future users too?

For a building, defining desiredoutcomes is not too difficult if thearchitect and partners in the

development work properly with theclient from the start. On the otherhand, evaluation of whether society isachieving sustainable development ispretty near impossible, given so manycompeting views of what successmight look like in terms of outcomes.Evaluation of rural developmentpolicy falls somewhere between thesetwo extremes.

For the OECD (where I chair theRural Working Party), all work shouldbe focussed on the outcome of abetter world economy. For farmpolicy, that means a world marketthat produces sufficient foodefficiently and sustainably, which has

a lot to do with getting fair trade andcompetition policies right. Ruraldevelopment policy is, however, amuch wider concept. Outcomes needto be defined in terms of benefits torural businesses and rural residents ofstronger, fairer, cleaner economiesand good services for citizens.Incidentally, ‘cleaner’ in this contextmeans better for the environmentand free of corruption.

In the case of the European Union’sCommon Agricultural Policy, theobjectives have always been broadand devoid of specific indicators ofsuccess. The goals of the currentpolicy (as set out in Agenda 2000) are:

point devuebyRichard Wakeford

Evaluation of Rural Development Requires Clarity onExpected Outcomes

L’evaluation du developpement rural de-mande a ce que les resultats attendus soientclairement identifies

Die Evaluation der Entwicklung deslandlichen Raums erfordert Klarheithinsichtlich der erwarteten Ergebnisse

ª 2010 The Author EuroChoices 9(1)ƒ 37

Journal compilation ª The Agricultural Economics Society and the European Association of Agricultural Economists 2010

• To increase competitivenessinternally and externally in orderto ensure that European Unionfood producers take full advantageof positive world marketdevelopments;

• To achieve food safety and foodquality, which are both fundamentalobligations towards consumers;

• To ensure a fair standard of livingfor the agricultural community andcontribute to the stability of farmincomes;

• Integration of environmental goalsinto agricultural policy;

• Promotion of sustainableagriculture;

• To create alternative job andincome opportunities for farmersand their families;

• To simplify European Unionlegislation.

Each of these holds a resonance forrural development, and the Europeanmodel is supposed to permit a widerange of action to achieve thesebroad outcomes. The explicitpromotion of rural development nowtakes place under the CommonAgricultural Policy’s Pillar II (coveringa wide range of economic andenvironmental measures directed atoperators of farms and forestholdings and landowners, and someto diversify rural activities andimprove quality of life there).

National ⁄ regional authorities havesubsidiarity to put their ownemphasis on particular Pillar IIobjectives (subject to minimum spendrequirements in certain areas – note,NOT minimum outcomerequirements). Nevertheless, thedirect support of agricultural incomesby Pillar I support, principally via theSingle Farm Payment system, alsocarries implications for ruraldevelopment.

How does one start to evaluatesomething with goals so broad anddelivery instruments so complex?There are, of course, mandatory mid-term and post-hoc evaluations of all

EU Rural Development Programmes.These focus on the national andregional levels. This begs twoquestions – whether the MemberStates defined the desired outcomessufficiently well from the start; andwhether a sum of evaluations willdeliver a compelling picture ofprogress towards the Europe-wideobjectives which justified a CommonAgricultural Policy in the first place.

ƒ‘‘Sans une bonne

evaluation, nous nous

privons des

informations qui

devraient influencer les

choix en matiere

d’instruments de

politique de

developpement rural

pour l’avenir.,,It is tempting to take eachcomponent of each Programme andevaluate it separately. But evaluatingthe individual components doesn’tmuch help with the bigger picture.What has been the outcome ofreplacing production subsidies with acombination of Single Farm Payment

38ƒEuroChoices 9(1) ª 2010 The Author


and a rural development programmewith different, potentially interactiveelements under four axes? And inaddressing structural disparitiesbetween regions, is it right toseparate evaluation of support fordiversification from that of othercomplementary (or even overlapping)assistance measures for economicdiversification and regionaldevelopment?

At the European level, in the lightof world trade negotiations, it wasimportant to wean farmers offproduction subsidies deliveringsupport without reference to need.One way was to divert money toencourage more general ruralbusiness growth – even at the riskof trespass into regionaldevelopment policy objectives. Soinvestment was incentivised totransform farms, on the one handseducing individuals otherwisereluctant to grow their businesses,and on the other moving somefarmers on to make way for thosewho would make more of the landresource. So the EuropeanCommission duly broadened therural development programme toopen up that possibility. Better todo that, I have heard it said inBrussels, than to leave all ruraldevelopment outcomes in the handsof regional developmentprogrammes that generally focus on

urban development. And for the(mainly) producer focused farmMinisters who agreed to the ruraldevelopment programme, better toaccept some leakage into non-farming businesses than to lose aslice of finance altogether.

I suspect that a survey of the 27current ‘Rural Development’ Ministerswould not reveal much consensusabout the desired outcomes of theEU’s rural development policy. Yetthese are the people responsible fortranslating Regulations agreed at theEuropean level into practice.

ƒ‘‘Without good

evaluation, we deny

ourselves the evidence

that should influence

choices of rural

development policy

instrument in the

future.,,So, whose outcomes should be takenas the basis for evaluation – those ofDG Agri or the Member State? Andsince we are talking regionaldevelopment, where do DG Regio

programmes factor in achieving theoutcomes? Furthermore, at thenational level, regional developmentdepends on the decisions of a widerseries of Ministers – transport,education, health, housing etc –decisions which should in the viewof the OECD be ‘rural proofed’. Inshort the alignment of desiredoutcomes against policies andprogrammes is difficult and imposessignificant constraints on evaluationof programmes – either individuallyor for their overall impact in ruralareas.

So, evaluation of rural developmentpolicies is complex. There isn’tsufficient clarity about the outcomesthat were expected. It’s not evenclear who should own ruraldevelopment outcomes andresponsibility for assessingprogrammes to achieve them. Butgood evaluation of any programmethat spends as much as the CAP andits components is needed right now,in a world still recovering from thefinancial crisis, with public spendingunder increased scrutiny. Withoutgood and timely evaluation, we denyourselves the evidence that shouldinfluence choices of policy instrumentin the future.

In short, have the decisions Ministerstook in shaping the programmes wenow have across Europe started todeliver the outcomes they intended –collectively and individually? Is therenow a level playing field for ruraldevelopment?

Ministers and officials may find ithard to bring themselves to doevaluation. After all, it may show thattheir policies haven’t delivered. And ifthis were an article about PostOccupancy Evaluation of buildings,there would be similar questions –why should an architect pay for anindependent evaluation of whethertheir building has proved to be fit forpurpose, which may provide reasonsfor others not to employ them in thefuture!

So, in respect of rural developmentprogrammes, some questions needaddressing, and not just in the EU:



• Should the evaluation be ofindividual programmes orcombinations of programmes?

• Should the evaluation include theimpact of other rules, regulationsand fiscal measures as well as whatthe financial incentives delivered?

• At a higher level, how can theevaluation address whetherlegislators’ other goals have beenmet, e.g. how much green box;how much trade distorting; howmuch grant made that privatesector would have provided?

• Should evaluation look at theEU ⁄ US as a whole; or at eachnation ⁄ region ⁄ state individually?

• How much of the ruraldevelopment programme needs tobe determined at theEuropean ⁄ US level, and how muchcan be left to states and regionswithout distorting the desired EUor US wide outcomes?

• Is the analysis of good practicehighlighted by programmeadvisory committees being spreadadequately throughnational ⁄ regional networks andmonitoring committees (in a kindof continuous improvementprocess)?

• Are current policies ⁄ programmesreally achieving outcomes (ratherthan outputs)?

• Can the outcomes of potentiallyoverlapping measures andprogrammes be teased out (e.g.regional and agriculturalprogrammes) and were thoseoutcomes properly made clear tostart with – at nation ⁄ region ⁄ statelevel?

• Who owns responsibility forsuccess?

• How to get the best timing: ideallyto influence the next programmereview, but also perhaps learningfrom what’s called ‘soft’ PostOccupancy Evaluation of buildings,where the building is given aninitial review really early in its life.

Finally, in shaping evaluation Isuggest that attention is given to

World Bank principles. These are thatevaluations should:

• review against measurableindicators of performanceestablished at the start –including outputs, outcomes andimpacts;

• be conducted every three to fiveyears;

• be jointly owned and utilised by allpartners – in the EU contextincluding DG Agri, DG Regio,national governments, regionalgovernments, and otherstakeholders;

• be overseen by the programme’sgoverning body and conductedwith the assistance of programmemanagers (nations ⁄ regions);

• be conducted by individuals orcompanies that have been at‘arm’s length’ from the initiative;

• address the performance of thegoverning body as well as ofmanagement; and

• make recommendations forfinancial sustainability as well asprogramme delivery.

Respecting these principles will go along way towards producing anevaluation system that servessociety well. The crucial thing is to beabsolutely clear about the outcomeyou want before starting the project.

Disclaimer

The views expressed in this article arethose solely of the author.

Further Reading

n Higher Education Funding Council for Europe (HEFCE), Association of UniversityDirectors of Estates (AUDE) and University of Westminster (2006). Guide to Post-Occupancy Evaluation, London. Available online at www.aude.ac.uk/uploads/POE%20Brochure%20Final%2006.pdf.

n World Bank. Reports of the Independent Evaluation Group, Washington, DC: TheWorld Bank. Available at www.worldbank.org/oed

Guide to Post Occupancy Evaluation: Higher Education Funding Council forEngland

Richard Wakeford, Scottish Government’s Director General, Rural Futures and Chair ofthe OECD’s Rural Working Party.

Email: [email protected]

40ƒEuroChoices 9(1) ª 2010 The Author


summaryEvaluation of RuralDevelopment RequiresClarity onExpected Outcomes

Without good evaluation, wedeny ourselves the evidence that

should influence choices of policyinstrument in the future. Evaluation ofrural development is complex. Thecrucial thing is to be absolutely clearabout the outcome you want beforestarting the project. For a building,defining desired outcomes is not toodifficult if the architect and partners indevelopment work properly with theclient from the start. Evaluation ofwhether society is achieving sustainabledevelopment is pretty near impossible,given so many competing views of whatsuccess might look like in terms ofoutcomes. Evaluation of ruraldevelopment policy falls somewherebetween these two extremes. Thesuspicion is that, despite Regulationsagreed at Council level, the EU’s 27‘Rural Development’ Ministers wouldnot reveal much consensus about thedesired outcomes of the EU’s ruraldevelopment policy. Ministers andofficials may find it hard to bringthemselves to do evaluation as it mayshow that their policies haven’tdelivered. Nevertheless, usefulevaluation can take place. It has to facea series of technical issues, such as‘Should the evaluation be of individualprogrammes or combinations ofprogrammes’. It also has to respectcertain principles, such as the ‘armslength’ status of the evaluator.

L’evaluation dudeveloppement ruraldemande a ce que lesresultats attendus soientclairement identifies

Sans une bonne évaluation, nousnous privons des informations

qui devraient influencer les choix enmatière d’instrument de politique pourl’avenir. L’évaluation du développementrural est complexe. Avant de démarrerun projet, il est crucial d’êtreabsolument clair sur les résultatsdésirés. Définir les résultats désirésn’est pas tellement difficile pour uneconstruction si les architectes et lespartenaires du développementtravaillent correctement avec le clientdès le début. Évaluer si la sociétéatteint un développement durable estpratiquement impossible du fait dugrand nombre de vues contradictoiressur ce que pourrait constituer unsuccès en termes de résultats.L’évaluation de la politique dedéveloppement rural se situe quelquepart entre ces extrêmes. On peutsoupçonner qu’en dépit des règlementsacceptés au niveau du Conseil, lesministres en charge du ‘‘développementrural’’ des pays de l’Union européenneà 27 n’atteindraient pas un grandniveau de consensus sur les résultatsdésirés de la politique dedéveloppement rural. Les ministres etles autorités pourraient trouver difficilede procéder à une évaluation car ellepourrait conduire à la conclusion queles politiques n’ont pas eu de succès. Ilest pourtant possible de réaliser uneévaluation utile. Elle devrait s’affronterà une série de questions techniquestelles que ‘‘l’évaluation devrait-elleconcerner des programmes individuelsou une combinaison de programmes’’.Elle devrait aussi respecter certainsprincipes comme le statut ‘‘en retrait’’de l’évaluateur.

Die Evaluation derEntwicklung deslandlichen Raumserfordert Klarheithinsichtlich dererwarteten Ergebnisse

Ohne eine gute Evaluationverzichtet man bewusst auf

Information, anhand derer zukünftigePolitikmaßnahmen ausgewählt werdensollten. Die Evaluation der Entwicklungdes ländlichen Raums ist kompliziert: Eskommt darauf an, das verlangteErgebnis im Vorfeld ganz genaufestzulegen. Beim Hausbau z.B. ist esnicht allzu schwierig, die gewünschtenErgebnisse zu definieren, wenn derArchitekt und alle beteiligtenVertragspartner von Anfang an mit demBauherrn Hand in Hand arbeiten. Es istdagegen nahezu unmöglich zuevaluieren, ob die Gesellschaft einenachhaltige Entwicklung erreicht, wennso viele widersprüchliche Vorstellungenhinsichtlich eines erfolgreichenErgebnisses existieren. Die Evaluationder Politik des ländlichen Raums liegtirgendwo zwischen diesen beidenExtremen. Obwohl im MinisterratEinigkeit über die entsprechendenVerordnungen erzielt wurde, würdendie Minister für die ‘‘Entwicklung desländlichen Raums’’ der EU27 vermutlichnicht in vielen Fragen, welche diegewünschten Ergebnisse der Politik zurEntwicklung des ländlichen Raums derEU betreffen, übereinstimmen. Ministerund Beamte können sichmöglicherweise nur schwer dazumotivieren, Evaluationendurchzuführen, da sich ihrePolitikmaßnahmen eventuell nicht alserfolgreich herausstellen könnten.Dennoch kann eine nutzbringendeEvaluation stattfinden. Sie steht einerReihe von technischen Fragengegenüber wie z.B. ‘Sollte sich dieEvaluation auf einzelne Programmeoder auf eine Kombination vonProgrammen beziehen’. Sie mussebenfalls auf gewissen Grundsätzenberuhen, wie z.B. der Unabhängigkeitdes Evaluators.



Documents

Evaluation of Rural Development Requires Clarity on Expected Outcomes L’évaluation du développement rural demande à ce que les résultats attendus soient clairement identifiés