INTRODUCTION TO MONITORING AND EVALUATION SYSTEMS POVERTY REDUCTION AND EQUITY...

1

INTRODUCTION TO MONITORING AND

EVALUATION SYSTEMS

POVERTY REDUCTION AND EQUITYGACEVEDO@WORLDBANK.ORG

2

Why M&E Systems?

1. Improve the effectiveness of public policy

2. Improve the allocation of public expenditure for higher welfare

3. Increase accountability4. Strengthen design, implementation,

and results of programs and policies

3

Outline of the Presentation

1. What is M&E?

2. How M&E influences Policy?

3. Country M&E Systems

4

1.- What is M&E

Monitoring

Provides regular information on performance

A continuing function 1) Indication of extent of progress towards goals

- poverty rates - coverage of basic services

2) systematic data collection3) usually conducted in-house

5

What to Monitor?

1. Identify few indicators that can be measured and are of interest to policy makers

2. Prioritize input, output, outcome and impact indicators for monitoring

3. Develop a data collection system to sustain monitoring

6

Inputs Outputs Outcomes Impact

How to Monitor?

• Use different types of information: surveys and censuses, administrative data, and participatory exercises

• Draw on logical framework and arrangements for results monitoring which identifies:-Data sources for each indicator-Frequency of measurements-The organization responsible for collecting information.

• Different types of Monitoring: -Annual progress report-Database for continuous monitoring-Desk reviews

7

Evaluation

Evaluation helps to analyze deviations from targets and goals ( e.g Why learning achievement increased?

Why poverty decreased?) Usually conducted by independent

evaluators A systematic and objective

measurement of the results achieved by a project/program/policy

8

What to Evaluate?

Evaluation can be time and resource intensive Plan ahead evaluation and identify questions

Strategy, operation, and learning

Reserve impact evaluation in cases such as Strategic significance of a policy or program Contributing to close knowledge gap Innovative nature of the policy or program

9

High

Low

Low High

Participantjudgment andexpert opinion

Complexity and sometimes cost

Strength of causal inference

Before-and-after

Quasi-experimental

with statisticalcontrols

Experimentalwith randomassignment

Trade Offs in Evaluation

Summing up…

Monitoring assesses progress in implementation of ongoing programs

Evaluation provides a snapshot against some benchmarks

Monitoring looks at progress relative to targets and assumes there is causality

Evaluation seeks to establish and prove causality

11

12

2. How M&E influences Policy?

13

M&E and the Policy Cycle

M&E needs to accompany the policy cycle so decisions can be based on evidence

Define what you want set goals and targets

Analyze what might work to reach targets design programs and evaluate ex-ante

Implement set performance indicators, a monitoring system , and implement evaluations

14

But we are far from there…

4 years ago, more than half of programs in Mexico had no logical frameworks No explicit objectives, no definition of target population

Countries seldom set targets using scientific methods: Reduce violent crimes from x to y in 5 yrs… is that a

meager performance or a great performance? Countries rarely define the outcome (say, nutrition),

and then decide in which sector the correct intervention is

Countries seldom analyze ex ante who the winners and losers of a policy are

Definition of Objective

s and Targets

Inputs Activities

Outputs

Policy and

Program

Design

Outcomes Impacts

Implementation Results

The Results Chain

Amount of services provided

Health literacy

Consumption, Life expectancy, poverty

Definition of Objective

s and Targets

Inputs Activities

Outputs

Policy and

Program

Design

Outcomes Impacts

Implementation Results

Pla

nn

ing

S

tag

e • Benchmarking• Logic Framework• Poverty Analysis• Ex-Ante Distributional and Poverty analysis

The Results Chain

•What are the extent and causes of poverty in a given country? •How is a proposed policy expected to affect the poor?•How to set strategic targets for a given policy/program?

Definition of Objective

s and Targets

Inputs Activities

Outputs

Policy and

Program

Design

Outcomes Impacts

Implementation Results

Monitoring Stage

• Logical Framework• Performance Indicators• Dashboards

The Results Chain (cont)

•What is a given policy/program trying to achieve? •What are the causal links through which the project is supposed to achieve its goals? •Is the project achieving its objectives? How to measure outputs and outcomes?

Definition of Objective

s and Targets

Inputs Activities

Outputs

Policy and

Program

Design

Outcomes Impacts

Implementation Results

Evaluation Stage

•Design Evaluation•Process Evaluation•Consistency Evaluation•Impact Evaluation•Economic Analysis/Modelling

The Results Chain•Is the execution of the project activities conducive to the expected results? •Are the chosen indicators good indicators? •What is the impact of the intervention? •Are the changes in well-being indeed attributable to the intervention?

19

Several methods to support evidence policy

Performance indicatorsThe logical framework (logframe) approachTheory-based evaluationFormal surveysRapid appraisal methodsParticipatory methodsPublic expenditure tracking surveysImpact evaluationCost-benefit and cost-effectiveness analysis

Example: Consistency and Results

Consistency and Results covers: Design, Strategic Planning, Target Population

and Coverage, Operation, Perception of Beneficiaries and Final Results.

Instrument applied by external institutions under terms of reference prepared by a Central Unit

The evaluation includes a Summary Strengths, Challenges and Recommendations

for every program

DESIGN Does de program identifies clearly the problem it’s trying to

solve? Is it clear that the products or services produced by the

program logically contribute to reduce the problem the program is tacking?

STRATEGIC PLANNING Does the program have updated strategic plans for the short,

middle and long-term? In the S-plan, does the final results are clearly established?

TARGET POPULATION AND COVERAGE Does the program have a method to identify and quantify the

potential and target population? Does the program have a coverage strategy for the short,

middle and long-term?

Consistency and Results: Questions

OPERATION Are there standard and adequate procedures for

the selection of projects, beneficiaries? Are there documents showing that the procedures

are done according to the rules of operation. PERCEPTION OF BENEFICIARIES

Does the program have instruments to measure the satisfaction of the beneficiaries with the program?

FINAL RESULTS Does the program systematically collect objective

information about its Goal (Fin y Propósito)? Does the program have rigorous impact

evaluations?

Consistency and Results Questions (cont)

Findings CONEVAL-Mexico

Topic Findings

DESIGN

67%

have a Purpose and Goal that relate to the problem’s solution.

50%

of the programs defined necessary and enough outputs to achieve the Purpose.

STRATEGIC PLANNING

21%

of the plans clearly establish the results they want to accomplish.

23%

of the programs have strategic plans in the short, middle and long-term.

TARGET POPULATION AND

COVERAGE

24%

of the programs quantify the target and potential population.

23%

of the programs have a coverage strategy for the short, middle and long-term.

Findings CONEVAL-Mexico

Topic Findings

OPERATION

72%

have standardized and adequate procedures for the project/ beneficiaries selection.

84%

of the programs have efficacy indicators in their operation.

BENEFICIARIES PERCEPTION

50%

of the programs have instruments to measure the satisfaction level of the target population.

FINAL RESULTS

34%

of the programs collect objective information about their final results.

26%

have external evaluations that allow to measure their impact.

25

3. Country M&E Systems

Country M&E Systems: Keys for success

Systems which permit the generation of information (basic data, indicators, evaluations, etc), reliable, credible and high quality

A high level of utilization of the information generated by the M&E system

Sustainability for the future

Improve the supply

Incentives from the demand side

The demand side: specific uses of an M&E system (or..why it’s needed?)

To support government planning at the national, sub national, local and sectoral level in the establishment of targets/goals and permanently ask the question why are targets not being met? focus on results

To support management public programs through monitor and evaluate performance and effectiveness

To informing the redesign and design of programs and interventions

To inform budget allocation decisions

To facilitate accountability

Countries choose one or all of these roles for M&E systems

Contradictoy?

Increasing effective demand and use of M&E

Pressure from civil society for budget accountability and vigilance of state´s performance

Need to maintain macro - equilibrium and simultaneous pressures to extend coverage of public services efficiency

Produce information (data , indicators, evaluations) which can effectively be used. Worry about the trust on the quality of information

Incentives for sectoral and budget officials, congress, etc. so that they use the information

Leaders who persuade and motivate the government to use the information generated a champion (person or institution) of results-orientation

All this is usually more important than legislation

Building blocks on the supply side

1. Improve Statistics:

household surveys, sectoral surveys and censuses

Better administrative records and sectoral statistics

Integration of systems to generate input and outcome indicators

Regional and local level data

to feed the production of M&E

Sectoral/program monitoring systems

Government –wide monitoring systems (Systems of presidential goals, PRS monitoring).

Performance evaluation systems for public services

Ex-ante and Ex-post Evaluations (Process evaluations , Rapid Assessments, Impact E.)

Ensuring sustainability

Permanently redefine the system. M&E tools have to be effectively used to improve performance and effectiveness of the state at all levels: Are we spending right? Are programs being well implemented? Are programs attaining its targets? Are we spending on the right things? Are programs being effective

Establishing strategic alliances: M&E a permanent function of different entities that should work together Finance ministries Planning agencies, Supreme Audit Institutions National Statistical Offices Evaluation offices ( embedded in an high leverage institutions) Sector ministries and its statistical offices Local governments

32

Thanks you