Upload
rexcris
View
6.854
Download
3
Tags:
Embed Size (px)
Citation preview
1
Project Cycle Management
Training Course on "Evaluation"
2
Definitions
Evaluation is the systematic assessment of the worth and merit of some object.
Evaluation is the systematic acquisition and assessment of information to provide useful feedback about some object.
3
Evaluation: A DefinitionAn evaluation is an assessment, as systematic and objective as possible, of an on-going or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, develop-mental efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of both partner countries and donors.
OECD, DAC, 1991
a
4
The Project CycleKey characteristics: breaks down the life of a project/
programme into phases defines key decisions, information
requirements, & responsibilities at each phase
each phase must be completed for the next to be tackled with success
allows for corrective measures during implementation, based on monitoring and mid-term evaluation
draws on evaluation experience for the design of future programmes & projects
a
5
When to do evaluations?
Mid-term Mid-term evaluationevaluation
End of End of project/ project/ programme programme evaluationevaluation
Ex-post Ex-post evaluationevaluation
During implemen-tation
At the end of implemen-tation
After imple-mentation
Also used as term: Ex ante evaluation
a
6
Evaluations: What for?
To improve
Decision making
Resource allocation
Accountability
Through:
Informing the public
Informed key decision-making processes
Encouraging on-going organisational learning
7
Logframe Basics
‘... IF results are delivered, AND assumptions hold true, THEN the project purpose will be achieved ...’
Objectively Verifiable Indicators
Intervention Logic
Sources of Verification
Assumptions
Overall Objectives
Project Purpose
Results
Activities Means Cost
Pre-condi-tions
8
Good indicators
Do partners and stakeholders agree that this indicator makes sense to use?
Owned
Will the data have utility for decision-making and learning?
Useful
Can data be collected easily, on a timely basis at reasonable costs?
Accessible
Is the data consistent or comparable over time?Reliable
Is the definition precise and unambiguous about what is to be measured?
Objective
Does the indicator directly represent the objectice it is intended to measure?
Valid
9
Indicators: An Example
Objective: Pollution load of wastewater discharged into the Blue river is reduced
Select the indicator: Concentration of heavy metal compounds (Pb, Cd, Hg)
Define the targets:• Define the quantity: Concentration of heay metal compounds (Pb,
Cd, Hg) is reduced by 75% compared to year x levels …(particular attention should be paid to the availability of baseline information)
• Define the quality: ... to meet the limits for irrigation water ...• Define the target group: ... , used by the farmers of Blue village, ...• Define the place : ... in the Blue river section of the District ...• Determine the time: ... 2 years after the project has started
10
Special challenges facing the donor agencies (1)
Work in many different countries and contexts
Have a wide diversity of projects in multiple sectors
Often focus on capacity building and policy reform, which are harder to measure than direct service delivery activities
11
Special challenges facing the donor agencies (2)
Are moving into new areas such as good governance, where there's little performance measurement experience
Often lack standard indicators on results that can easily be aggregated across projects
Are usually only minor actors affecting impacts, with subsequent problems in attributing them to their agency's activities
12
Special challenges facing the donor agencies (3)
Typically rely on outcome and impact data collected by partner countries, who have limited technical capacity and resources, with subsequent quality, coverage and timeliness problems.
13
Criteria for Evaluating Development Assistance
Relevance Efficiency Effectiveness Impact Sustainability
14
The Logframe and the 3 basicevaluation criteria
Overall Objectives
Project Purpose+ Assumptions
Results+ Assumptions
Activities+ Assumptions
+ Pre-conditions
allocation
action
utilisation
change
Efficiency
Means
Impact
Overall Objectives
+ Assumptions
Results+ Assumptions
Activities+ Assumptions
allocation
action
utilisation
change
+ Pre-conditionsMeans
Overall Objectives
Project Purpose+ Assumptions
Results+ Assumptions
Activities+ Assumptions
allocation
action
utilisation
change
+ Pre-conditionsMeans
Effective-ness
Project Purpose
15
Logframe and basic evaluation criteria: Major questions
Overall Objectives
Project Purpose+ Assumptions
Results+ Assumptions
Activities+ Assumptions
+ Pre-conditions
allocation
action
utilisation
change
Means
Efficiency
Effective-ness
Impact
How were inputs and activities converted into Results?
How well did the Results contribute to the achievement of the Project Purpose?
Which benefits on society and sector?
16
The 4th evaluation criterion: Relevance
Overall Objectives
Project Purpose+ Assumptions
Results+ Assumptions
Activities+ Assumptions
+ Pre-conditions
allocation
action
utilisation
change
Means
Efficiency
Effective-ness
Impact
Quality of planning and adaptation, including relevance of problems to correct beneficiaries, OVIs, means, cost, assumptions, risks
How were inputs and activities converted into Results?
How well did the Results contribute to the achievement of the Project Purpose?
Which benefits on society and sector?
Relevance
17
The 5th evaluation criterion: Sustainability
Have and will services and benefits be maintained?
Quality of planning and adaptation, including relevance of problems to correct beneficiaries, OVIs, means, cost, assumptions, risks
How were inputs and activities converted into Results?
How well did the Results contribute to the achievement of the Project Purpose?
Which benefits on society and sector?
Overall Objectives
Project Purpose+ Assumptions
Results+ Assumptions
Activities+ Assumptions
+ Pre-conditions
allocation
action
utilisation
change
Means
Efficiency
Effective-ness
Impact
Relevance
Sustain-ability
18
Summary: The 5 evaluation criteria
The fact that the results were obtained at reasonable cost, i.e. the cost, speed and management efficiency with which Means/Inputs and Activities were converted into Results, and the quality of the Results achieved.
Efficiency
The effect of the project on its wider environment, and its contribution to the wider objectives summarised in the Overall Objectives.
Impact
The appropriateness of project objectives to the real problems of the intended beneficiaries that it was supposed to address, and to the physical and policy environment within which it operated.
Relevance
The likelihood of a continuation in the stream of benefits produced by the project after the period of external assistance has ended.
Sustainability
How well the Results contribute to the achievement of the Project Purpose, and how Assumptions have affected project achievements.
Effectiveness
19
Relevance= The extent to which the aid intervention is suited to the priorities and policies of
the target group, partner country and donor
Possible questions: To what extent are the objectives of the programme still valid? Are the activities and outputs of the programme consistent with the
broader objectives and do they contribute to the attainment of these objectives?
Are the activities and outputs of the programme consistent with the intended impacts and effects?
… … …
20
Efficiency= Efficiency measures the outputs – qualitative and quantitative – in relation to the
inputs. It is a term which signifies that the aid uses the least costly resources in order to achieve the desired results. This generally requires comparing alternative approaches to achieving the same outputs, to see whether the most efficient process has been adopted
Possible questions: Were the activities cost-efficient? Were objectives achieved on time? What were the major factors influencing the achievement or non-
achievement of the objectives? … … …
21
Effectiveness= A measure of the extent to which an aid intervention attains its objectives
Possible questions: To what extent were the objectives achieved/are likely to
be achieved? What were the major factors influencing the achievement
or non-achievement of the objectives? … … …
22
Impact= The positive and negative changes produced by an intervention, directly or
indirectly, intended or unintended.
Possible questions: What has happened as a result of the programme or project? What real difference has the intervention made to the
beneficiaries? How many people have been affected? …… …… ……
23
Sustainability= Sustainability is concerned with measuring whether the benefits of an
intervention are likely to continue after donor funding has been withdrawn.
Possible questions: To what extent did the benefits of a programme or project continue
after donor funding ceased? What were the major factors which influenced the achievement or
non-achievement of sustainability of the programme or project? … … …
24
Principles of evaluation Integrative part of planning and management: closely connected to decision
making processes Impartiality & independence of the evaluation process in its function from the
process concerned with policy making, the delivery and management of assistance (= separation of evaluation and responsibility for the project/ programme/policy)
Credibility depending on expertise and independence of the evaluators & transparency to be seeked through an open process, wide availability of results, distinction between findings and recommendations
Usefulness: relevant, presented in a clear and concise way, reflecting the interests and needs of the parties involved, easily accessible, timely and at the right moment improved decision-making
Participation of stakeholders (donor, recipient...); if possible: views and expertise of groups affected should form integral part of the evaluation; involving all parties capacity building
DAC 1991
25
Evaluation, Monitoring and Audit (1)
once or twice, essentially at the end or 'ex-post' drawing lessons from the past in order to orient future policies and actions but also during implementation: mid-term evaluation to (re-) orient implementation
When?
external evaluators specialised in the subjects evaluatedWho?
in-depth analysisHow?
mainly analysis of the efficiency, effectiveness, impact, relevance and sustainability of aid policies and actions
What?
Evaluation:
26
Evaluation, Monitoring and Audit (2)
regularly, several times per yearWhen?
internal and external (staff, monitors.....)Who?
rapid and continuous analysis, immediately useful to improve on-going actions; of key importance to improving performance
How?
mainly analysis of efficiency and effectiveness (i.e. measuring actual against planned deliverables); is a systematic management activity
What?
Monitoring:
27
Evaluation, Monitoring and Audit (3)
.
during or after implementationWhen?
external, professional auditorsWho?
verification of financial records (financial audit)How?
traditionally checks whether financial operations and statements are in compliance with the legal and contractual obligations. More concerned with compliance, but better financial management can also contribute to improving current and future actions. More recently: Performance audit is strongly concerned with questions of efficiency amd good management
What?
Audit:
28
Evaluation
Evaluation is the systematic assessment of the value and merit of some object (in this case, a project or a programme)
Evaluation is the systematic acquisition and assessment of information to provide useful feedback about the evaluated object.
29
Evaluation: major criteria
Relevance Efficiency Effectiveness Impact Sustainability
30
Relevance= The extent to which the aid intervention is suited to the priorities
and policies of the target group, partner country and donor
Possible questions: To what extent are the objectives of the programme valid for the
beneficiaries? Are the activities and results of the programme consistent with the
overall objectives? … …
31
Efficiency= Efficiency measures the outputs – qualitative and quantitative – in relation
to the inputs. It is a term which signifies that the aid uses the least costly resources in order to achieve the desired results. This generally requires comparing alternative approaches to achieving the same outputs, to see whether the most efficient process has been adopted
Possible questions: Were the activities cost-efficient? Are the unit costs comparable to … … …
32
Effectiveness= A measure of the extent to which an aid intervention reaches its
objectives
Possible questions: To what extent was the project purpose achieved/is likely to be achieved? What were the major factors influencing the achievement or non-
achievement of the project purpose? … …
33
Impact= The positive and negative changes produced by an intervention,
directly or indirectly, intended or unintended.
Possible questions: What real difference has the activity made to the beneficiaries? How many people have been affected? … …
34
Sustainability= Sustainability is concerned with measuring whether the benefits of
an activity are likely to continue after donor funding has been withdrawn.
Possible questions: To what extent did the benefits of a programme or project continue after
donor funding ceased? What were the major factors which influenced the achievement or non-
achievement of sustainability of the programme or project? … …