Joint Evaluation of the Paris Declaration, Phase 2

Core Team

Joint Evaluation of the Paris Declaration, Phase 2

Evaluation Framework & Workplan

Presentation to International Reference Group30 Nov 09

Joint Evaluation of the Paris Declaration, Phase 2

Core Team

Umbrella Framework for All Elements in Phase 2• Overview • Evaluation Profile: Reasons for the Evaluation, Objectives,

Audiences and Stakeholders, Approach • Approaches to Methodology: Basis, Principles, The

Evaluation Framework and Core Questions, Core components of the methodology, Anticipated methods, Ensuring comparability, Ensuring validity and reliability.

• Accountabilities and Responsibilities• Workplan and Schedule

A.The Overarching Evaluation Matrix for Phase 2 B.Draft Generic TORs for Country Evaluations C.Draft Generic TORs for Donor/Agency HQ EvaluationsD.First Draft Outline for Evaluation Synthesis Report E.Two framework diagrams from preparatory studies

Joint Evaluation of the Paris Declaration, Phase 2

Core Team

Building blocks of the synthesisSYNTHESIS

PHASE 1 RESULTS

EVALUATION QUESTIONS4. Alternatives3. Development outcomes2. Process and intermediate outcomes1. Context

COUNTRY STUDIESDONOR STUDIES

SUPPLEMENTARYSTUDIES

Joint Evaluation of the Paris Declaration, Phase 2

Core Team

Relationship to ToRs• The Framework is the umbrella that sets out all

substantive elements of the Evaluation, its governance & broad approaches to methodology;

• Generic ToRs for Country and Donor/Agency HQ evaluations will operationalize the main components, with direct guidance to teams;

• The whole structure and synthesis is to be based on a single, clear set of evaluation questions and a simple but rigorous approach to ensure comparative findings that are credible, relevant and useable by its key audiences.

Joint Evaluation of the Paris Declaration, Phase 2

Core Team

Overall evaluation approach1. Accepts special methodological challenges;2. Process and results designed for accountability (Seoul & at home)

and learning (to strengthen policies & practices);3. Fully joint, relying primarily in Phase 2 on the 20+ country

evaluations, including donors on the ground ;4. Seven donor/agency HQ-level studies, plus possible updating of

Phase I. A small number of “supplementary studies”.5. Evaluating a) to what extent the PD has been implemented, and b)

in so far as it has been implemented, what the results have been in terms of aid effectiveness and contribution to development results;

6. Focus on the workings of country-donor partnerships and their development outcomes at country-level;

7. Implementation or “process;” and8. “Outcomes” or “Results” in terms of aid effectiveness and

contributions to development results.

Joint Evaluation of the Paris Declaration, Phase 2

Core Team

Approaches to Evaluation Methodology

1. Summative and formative; 2. A clear, simple and straightforward common

approach, language, and methods for the evaluations;

3. Apply existing data sources, avoid duplicative and unnecessary demands;

4. Take account of early preparatory work on the many complex factors and relationships at work (see diagrams in Appendix E). Matrix to Mgt. Group in Sept/Oct 2009.

Joint Evaluation of the Paris Declaration, Phase 2

Core Team

The Evaluation Framework and Core Questions1. A manageable set of agreed common Core Evaluation

Questions and sub-questions, with common methods, for robust comparative findings;

2. Focus on the most important, results-oriented questions of most interest to most users;

3. If needed, partner country and Donor/Agency HQ evaluations to supplement their answers to the common Core Questions with questions of special interest/priority to them;

4. Allow for the integration of Donor/Agency HQ Evaluations; key cross-cutting assessments, e.g. on adherence to the five PD principles and the Accra Agenda for Action priorities; and the results of supplementary studies to fill gaps.

Joint Evaluation of the Paris Declaration, Phase 2

Core Team

The Paris Declaration in Perspective

Q1: PD in context

Aid influenced by PD commitments

The Aid Partnership

Overall development processes

Other international & national influences &

forces

Q1

Q2Q3

Q4Conclusions: Compared against pre-PD or alternative approaches

Q3: Effects of PD on development results

Q2: Effects of PD on aid effectiveness

Joint Evaluation of the Paris Declaration, Phase 2

Core Team

The Core Questions(Refined through regional workshops and inputs from other IRG

members)

1. “What are the important factors that have shaped and limited Paris Declaration implementation and its potential effects on aid effectiveness and development results?” (The Paris Declaration in context)

2. “To what extent and how has the implementation of the PD led to an improvement in the efficiency of aid delivery, the management and use of aid and better partnerships?” (Process and intermediate outcomes)

3. “Has the implementation of PD strengthened the contribution of aid to sustainable development results? How?” (Development outcomes)

Joint Evaluation of the Paris Declaration, Phase 2

Core Team

Framework for conclusions1. To what extent have the 5 principles of the PD been observed and

implemented and the Accra Agenda priorities reflected? Why? Have there been conflicts or trade-offs between them?

2. What has the Paris Declaration achieved for aid effectiveness and development results? How significant are these contributions? How sustainable? Is there evidence of better ways to make aid more effective and contribute more to development results?

3. What has Paris Declaration-style development cooperation added compared with the pre-PD situation and alongside other drivers of development in the country, other sources of development finance and other development partners not so far endorsing the Declaration?

4. What are the key messages for a) national stakeholders, and b) donor countries and agencies?

5. What are the key implications for aid effectiveness in the future, taking account of new challenges and opportunities (e.g. climate change) and new actors and relationships?

Joint Evaluation of the Paris Declaration, Phase 2

Core Team

Core components of the methodology1. Questions, and the framework for conclusions,

follow the criteria of relevance, efficiency, effectiveness and sustainability;

2. Include assessment of “upstream” or precursor steps to Paris in 2005;

3. Not expect clear-cut or one-dimensional causality, seek “plausible contributions”;

4. Searching analysis of (vital) context;5. Sub-questions: descriptive, analytical, normative

and evaluative.

Joint Evaluation of the Paris Declaration, Phase 2

Core Team

Common specifications and suggestions for country evaluations

On:• Types of evidence and, where applicable, indicators • Anticipated availability and (probable) reliability of

data sources • Proposed sources, methods & techniques for data

collection, analysis, triangulation and validation.

Joint Evaluation of the Paris Declaration, Phase 2

Core Team

Anticipated methods (To be finalized at Inception Report stage)

1. Syntheses and meta-analyses of existing evidence;2. The normal arsenal of evaluation tools and techniques (see

p. 12);3. Comparative studies, esp. on one “tracer” and other

sectors;4. Backward tracking to past Paris Declaration-like initiatives

and their results;5. Analysis of time-series data, and synthesis studies;6. ‘Theory based’ (longitudinal) studies that are forward

looking, incl. “direction of travel” and “distance travelled”;7. Examine “mechanisms of change”: help explain results 8. Ensuring comparability (See p.12)9. Ensuring validity and reliability (See p.13)

Joint Evaluation of the Paris Declaration, Phase 2

Core Team

Accountabilities and Responsibilities

• International Reference Group• National Evaluation Coordinator• National Reference/Advisory Group• Country Evaluation Team• Donor Headquarters Evaluation Team• Core Evaluation Team• Evaluation Management Group and Secretariat

Joint Evaluation of the Paris Declaration, Phase 2

Core Team

Workplan and Schedule• See “Critical Milestones” Document

Joint Evaluation of the Paris Declaration, Phase 2

Core Team

Appendix A. The Overarching Evaluation Matrix for Phase 2

• Proposed Core Evaluation Questions & Sub-questions

• Overview of sources of evidence for answering the Evaluation questions (elaborated in Generic TORs)