POLICY IMPACT EVALUATIONBiogas in Rwanda

Willem Cornelissen – ERBS, Erasmus University Rotterdam

Outline

The central questions of the CDI seminar Policy evaluation in relation to impact evaluationIOB’s role; its evaluation of renewable energy; IE biogas in Rwanda: results chainComing back to the central questions

Central questions

Who defines what is IE? Who defines what is to be evaluated? How is utilization of IE findings influenced by IE design and communication?

What influences IE design? What influences communication of IE

findings? What can be done in design and

communication to enhance utilization of IE findings?

4

What policy evaluation is all about

Policy is about what a government wants to achieve – it is specific, measurable and time bound (the target)A strategy is how to go about achieving the particular policy objectives (the direction)Plans and programmes – who, when and resources required (the means)

Review processesInstitutions and capacities

Public Budget of the Netherlands

Policy articles; these are –in principle- subject to evaluation (or parts thereof)Ref. to renewable energy: Chapter V (Foreign Affairs); policy article 6: “better protected and improved environment”, art 6.1 “Protection and sustainable use of the environment in its global context and national context of developing countries”In 2011 the art. 6.1 was amended to: “sustainable use of the environment in the world” (the national context was eliminated).

The budget law defines

Governmental Regulation for Policy Evaluation research (RPE)

Periodic policy evaluation Study of net policy effects

Objective of RPE: improvement of the quality of policy information; more ‘value for money’

Justification of public expenditure Effectiveness Efficiency Sustainability

Purpose: Contribute to the Minister’s accountability to the

Parliament Learning function

OECD

IOB: Policy and Operations Evaluation Department

Mandate: evaluations of policies and operations of the Min. of Foreign AffairsIts findings are public - through parliament

(Renewable) energy: policy evaluation underpinned by eight Impact Evaluations Amongst these: biogas in RwandaMotivation: Overall policy objective: developmental + environmental; strategy objective: biogas in Africa; programme: Dutch funded programme in Rwanda

Biogas results chain

Choice of methods

Statistical requirements: robustness, representativeness (sample size), confidence levels; attributable resultsPragmatic: time constraint, limited financial resources (excludes option for DiD and RCTs)

Mixed methods IE:Quantitative: 600 hh (300 with; 300 without); cross-sectional with propensity score matchingQualitative: village research, stakeholder analysis, semi structured survey masons

Coming back to the central questions (1)

DesignPerception of accountability -through Parliament- to Dutch tax payerDesign in part determined by budget law (RPE)and in part by OECD evaluation questionsand by pragmatic constraints

Communication Accountability:

Ministry for Dev.Coop and Foreign Trade + Foreign Affairs towards ParliamentBudget accountability

Learning:Government of RwandaPolicy makers Min.Dev.Co. + Foreign TrProgramme and project implementers, practioners

Coming back to the central questions (2)

Dissemination (not done yet)

Publication (book)Newsletter and websiteScientific publicationPresentation in seminars

Utilization

Formalized feed-back within ministry, and with implementersBut:….

Utilization depends on WHO communicates and the convenience of the moment

HOW it relates to mainstream thinking and norms (the logic of appropriateness)

The appeal for what is NOVEL The importance of personal gain

(saving face; prestige; office politics)

Dilemma’s

• Elliot Stern stated that evaluators sharpen their tools & techniques to evaluate more and more rigorously and hence less and less policies. However, IE is needed to address the effectiveness question of the policy evaluation

• Policies are volatile... IE measures past policies, not current ones

• Impact evaluation is an artisanal skill…..overkill and undermining of quality