Embed Size (px)
Citation preview
Monitoring and Evaluation, and the
Potential for Civil Society
Participation in Yemen
An IRDPG Civil Society
Workshop
December 16, 2009
1
Monitoring defined
(World Bank OD 10.70)
• [T]he continuous assessment of project
implementation in relation to agreed schedules,
and of the use of inputs, infrastructure, and
services by project beneficiaries. …Its main
objectives are to provide continuous feedback
on implementation, and to identify actual or
potential successes and problems as early as
possible to facilitate timely adjustments to
project operation.
2
Evaluation Defined (OD 10.70)
• The periodic assessment of the relevance,
performance, efficiency, and impact (both
expected and unexpected) of the project in
relation to stated objectives.
– Interim evaluation: undertaken by project managers
during implementation as a first review of progress
and a prognosis of the likely effects of the project.
– Terminal evaluation: at the end, assesses the
project's effects and their potential sustainability.
– Impact evaluation: usually several years after final
disbursement, measuring changes attributable to the
project in terms of both direct and indirect causality. 3
Link of monitoring and evaluation to
poverty alleviation
• Human development outcomes mediocre today in MENA… and worse for poor people– How to increase effectiveness of public spending and make
services work -- to reduce inequalities and improve quality?
• Traditional approach is not working– Four reasons why public spending does not translate into good
services
• A new approach to service delivery– Empower (poor) people to monitor and discipline service
providers and to raise their voice in policy-making
– Strengthen incentives for service providers to serve the poor
• A new role for information: Examples of “cutting-edge”research across the world monitoring and evaluation
4
A Framework of Accountability
Source: WDR 2004
Monitoring and Evaluation information can enhance accountability. 5
Example: Information interventions in education improve accountability of policymakers to citizens
Policymakers
Citizens/Clients
Intervention Type Country
Information Provided by Policymakers >> Citizens/Clients
National Newspaper Campaign*** Uganda***
Textbook Count 1-2-3 Philippines
Test-Based School Rankings Chile
School-Level Report Cards Brazil, Uganda, Nigeria
School Performance Appraisal Meetings Ghana
Information Provided by Citizens/Clients >> Policymakers
Education Sector Report Cards
(Transparency Int’l Surveys)Bangladesh
Citizen Report Cards Bangalore, India
Pro-Poor Services Report Cards Philippines
√ Rig. IE
√ Rig. IE
6
Example: Newspaper Campaign in Uganda
In 1995 in Uganda, only 13 percent of non-wage recurrent spending on primary education reached primary schools.
From 1997, amounts of per capita grants sent to local governments have been published in local and national newspapers to promote client power. 7
Schools in Uganda received more of what
they were due, post-intervention:
Source: Reinikka and Svensson (2001), Reinikka and Svensson (2003a)8
There are many techniques for monitoring
and evaluation.
Source: World Bank and OECD, MONITORING & EVALUATION: Some Tools, Methods
& Approaches (Washington: World Bank, 2004)
Impact Evaluation
Cost-Benefit and Cost-
Effectiveness Analysis
Performance Indicators
Theory-Based Evaluation
Rapid Appraisal Methods
Public Expenditure
Tracking Surveys (PETS)
Participatory Methods
Formal Surveys
Logical Framework
Approach
Some offer more entry points
for CSO participation….
9
CSOs can engage at multiple stages.Method
Phase
Perfo
rman
ce
Ind
icato
rs
Form
al
Su
rvey
s
Citizen
Rep
ort C
ard
s
Rap
id
Ap
pra
isal
Meth
od
s
Partic
ipato
ry
Meth
od
s
PE
TS
Imp
act
Evalu
atio
n
Identify
Indicators
Design
Instruments
Raise Awareness/
Response
Implement
Analyze, Report
Data
Develop policy,
action plan
Disseminate
Findings 10
The role NGOs/CSOs can play
• In all cases, can contribute to select important
indicators and adapt to local conditions.
• In many cases, can contribute to developing
instrument (e.g. questionnaire, template, etc.)
• In the case of Citizen Report Cards, NGOs have
been primary implementers.
• In many cases, can facilitate or participate in
implementation of rapid appraisals.
• Participatory monitoring and evaluation engages
NGOs and beneficiaries directly at multiple stages.
• In all cases, CSOs can participate in policy
discussions and dissemination. 11
Implications for CSOs
• Level of engagement depends in part on
capacity of NGOs/CSOs to participate in
various approaches.
• In some countries, development of CSOs is
limited, and membership small.
• In some countries, knowledge of local
conditions may be strong, but technical
capacity or experience may be limited.
• Technical assistance, training and
incremental engagement may each contribute
to building capacity.12
Lessons of experience for CSO
engagement
• Working together in a network is more effective and
efficient than working alone.
• CSOs need to avoid simplistic positions.
– Invest time, resources to fully understand issues.
– Use the process of learning as networking
• It is better to work with government because it is more
accountable compared to IFIs
• But it is also important to maintain independence.
– CSOs can support needed reforms, but be alert to
token benefits for the poor, unsustainable quick fixes,
corruption, profiteering, etc.
• Can integrate reform engagement with service delivery.
– May have more credibility and better communication
channels with communities. Source: WSP 13
Performance Indicators
• Measures of inputs, processes, outputs,
outcomes and impacts
• Used for assessing progress towards targets,
early warning.
• Measure progress towards objectives of
different project components.
• Participation of stakeholders helps them
understand and use them.
• Costs depend on number of indicators,
frequency and quality of information sought,
comprehensiveness of system. 14
Indicators in Yemen Institutional Reform
Development Policy Grant Program Document
15
Example of Performance Indicators for
Tax Component from Enterprise Surveys
12 15
23 24
28 28
31 31
34 37
40 47
52 54 54
61 71
73
0 10 20 30 40 50 60 70 80 Percent of respondents evaluating constraints as "major" or "very severe"
Labor regulations Transportation
Skills/education of workers Business licensing/operating permits
Crime, theft & disorder Access to finance ( collateral)
Regulatory policy uncertainty Legal system/conflict resolution
Cost of financing Access to land
Customs & trade regulations Electricity
Smuggling or dumping Anticompetitive or informal practice
Tax administration Corruption
Tax rates Macroeconomic Uncertainty
2005 Survey Evaluation of Constraints by Yemeni Entrepreneurs
2010 ?
2010 ?
16
Example of Performance Indicators on
Tax Component from ―Doing Business‖
Requirements for prototypical firm to comply with tax requirements –
number of payments, time spent, effective tax rate on profits.
17
Use of formal surveys for Monitoring
and Evaluation
• Client satisfaction surveys
– Citizen Report Cards
• Household surveys
• Enterprise surveys
Source: World Bank and OECD, Monitoring and Evaluation: Some Tools,
Methods & Approaches (Washington: World Bank, 2004) 18
Absence rates among teachers and health workers
Note: Surveys were all fielded in 2002 or 2003. Sources: Chaudhury et al (2006) except for PNG,
World Bank (2004) and Zambia, Das et al (2005).
0
10
20
30
40
50
Bangladesh Ecuador India Indonesia Papua New
Guinea
Peru Zambia Uganda
Primary schools Primary health facilities
Increasing accountability through quantitative service delivery surveys.
19
Example: Use of Citizen Report Cards
Simple evaluation tool, structured for simple communication
Credible user feedback on public services
An effective diagnostic tool for service providers that encourages citizen friendly practices
Creates opportunities for partnerships and opens space for dialogue and discussions
Report always in PUBLIC DOMAIN.
20
What a Citizen Report Card can do.
Collects client experiences to assess and rank performance of government services
Identifies clients’ constraints in accessing services, views about quality and adequacy of services, responsiveness of public officials, extent of corruption encountered.
Usually conducted by NGOs and think-tanks.
Generates benchmarks for tracking reforms
Widespread publication of the findings.
Leads to practical ACTIONS
21
5
41
73
6
47
94
4
42
73
9
67
92
34
78
1
16
85
32
96
0
20
40
60
80
100
BMP BESCOM BWSSB BSNL POLICE BDA BMTC
1994 1999 2003
Citizen Report Cards measure,
benchmark and compare over time.
agency
N/aN/a
22
The Bribery Matrix (All Households)
Percent Claiming to have paid a bribe
Average payment per transaction in Rupees
SERVICES / AGENCIES
1994 1999 1994 1999
Water Supply Board 12 11 275 561
Power Corporation 11 09 206 1563
Municipal Corporation
21 52 656 3759
Telephones 04 26 110 245
Transport Office 33 57 648 637
Development Authority
33 35 1850 1653
23
Do Citizen Report Cards Work? (India)
24
Do Citizen Report Cards Work? (Uganda)
• Bjorkman and Svensson (2007): Impact evaluation of a
citizen report cards rural primary health project.
– Randomized field experiment – very strong method
• Citizen report cards impact:
– Improved access to information
– Improved local organizational capacity to effectively
use the information
• Impact: better quality and quantity of health service
• One year into the program:
– 16 percent increase in utilization
– Significant weight gains for infants
– Markedly lower deaths among children under 5 25
Rapid Appraisal methods
• Focus groups
• Use of key informants
• Community group interviews
• Direct observation
Source: World Bank and OECD, Monitoring and Evaluation: Some Tools,
Methods & Approaches (Washington: World Bank, 2004) 26
Rapid Appraisal – quick and qualitative
• Rapid appraisal methods are quick, low-cost
ways to gather the views and feedback of
beneficiaries and other stakeholders.
• Provide rapid information for management decision-
making, especially at the project or program level.
• Offer qualitative understanding of complex and
dynamic social situations, or people’s values,
motivations, and reactions.
• Provides context for quantitative, formal data.
• Findings are specific to groups interviewed, not
statistically valid.
27
Rapid Appraisal – examples
Key informant interview—in-depth, open-ended, semi-
structured interviews of individuals selected for their
knowledge and experience in a topic of interest.
Focus group discussion— Guided, facilitated discussion
among 8–12 selected participants with similar back-
grounds, such as project beneficiaries or program staff.
Community group interview— a series of structured
questions and a facilitated discussion in a meeting open
to all community members.
Direct observation— uses a detailed observation form to
record what is seen and heard at a program site.
Mini-survey— a structured, short questionnaire
administered to 50–75 people (e.g. project beneficiaries). 28
Example: Rapid Appraisal of West
African Livestock Marketing
• Economic anthropologist fluent in local language
interviewed traders, producers.
• Identified policy, regulatory and procedural barriers to the
long-distance livestock trade; quantified formal and
informal costs of livestock marketing.
• Action plan emphasized transport efficiency and
competitiveness. It recommended deregulating
international trucking between coastal West African
countries and the Sahelian interior countries to reduce
costs. Policy reforms supported by WB operations.
• Utilized widespread and systematic dissemination of
information to private livestock traders about reforms.
• Periodic rapid appraisals were used subsequently to
monitor discretion and corruption among public officials. 29
Public Expenditure Tracking Survey
(PETS)
• Surveys service-providers to assess efficiency of
public spending, quality and quantity of services.
• Tracks the flow of resources through the various
layers of government bureaucracy, down to the
service facilities, to determine:
– Percent of allocated resources reaching each level;
– How long resources take to get to each level.
– Location and extent of impediments to resource flows
(financial, staff, equipment).
– Performance of mechanisms and incentives
responsible for public expenditure leakages, capture
and deployment impediments. Source: World Bank and OECD, Monitoring and Evaluation: Some Tools, Methods & Approaches
(Washington: World Bank, 2004)
30
Steps in PETS
• Consultations with key stakeholders, including
government agencies, donors and civil society
organizations to define the objectives of the
survey, identify key issues, determine the
structure of resource flows and the institutional
setup, review data availability, outline hypotheses
and chose the appropriate survey tool.
• Design and implementation of survey instruments
using a ―multi-angular data collection strategy‖
and carefully considering incentives of
respondents to misreport.
Reinikka and Svensson (2001, 2002, 2003) 31
Public Expenditure Tracking Surveys (PETS):
What percent of disbursed public spending on
school grants actually reach schools ?
Percent GNI per capita
(2000)
GNI per capita
PPP (2000)
Ghana 1997/98 51 330 1880
Kenya 2004 (secondary school bursary funds)
78250 810
Madagascar 2002 88 2050 4610
Peru 2001 (utilities) 70 / 97 670 2280
PNG (2001/2002) 72 / 93 280 510
Tanzania 2002-2003 62 270 1250
Uganda 1991-1995/2001 <20 / 80
Zambia 2001 (discretion/rule) 24 / 90 320 740
Ye and Canagarajah (2002) for Ghana; Republic of Kenya (2005) for Kenya; Francken (2003) for Madagascar; Instituto
Apoyo and World Bank (2002) for Peru; World (Bank 2004) for PNG; MOF, Government of Tanzania (2005) for
Tanzania; Reinikka and Svensson (2005) for Uganda; Das et al. (2002) for Zambia.
32
33
Participatory Monitoring and Evaluation
Involves stakeholders at different levels working together to
identify problems, collect and analyze information and
generate recommendations. Methods include:
1. Stakeholder analysis: Develops an understanding of
power relationships, influence, and interests of the various
people involved in an activity, shape participation.
2. Participatory rural appraisal: Shares learning between
local people and outsiders, to facilitate collaborative
planning of appropriate interventions.
3. Beneficiary assessment: Systematic consultation with
project beneficiaries and other stakeholders to identify
and design development initiatives, signal constraints to
participation, and provide feedback to improve services.
Source: World Bank and OECD, Monitoring and Evaluation: Some Tools, Methods & Approaches
(Washington: World Bank, 2004)
34
4. Community Scorecards
What can they do?
• Qualitative tool, very participative.
• Gathers feedback from consumers,
beneficiaries and providers on public services
• Raises awareness of causes of poor
performance and low customer satisfaction
• Usually conducted by NGOs and think-tanks.
• Generates benchmarks for tracking reforms
• Findings are published in the public domain
• Leads to practical ACTION PLANS to improve
service performance and satisfaction 35
36
Citizen Report Cards vs. Community
Scorecards
Aggregated data – city, region or country
Requires survey and quantitative skills.
Better for comparing and measuring, as basis
for wide publicity, mobilization of demand for
accountability..
Requires skilled facilitation.
Better for providing feedback within project
context.
Source: World Bank, “The Community Score Card Process: Introducing the Concept and Methodology,”
http://www.worldbank.org/wbi/attackingpoverty/events/Tanz_0603/TheCommunityScoreCardProcess_June0
3.pdf37
Impact Evaluation – Slow and Precise
• The systematic identification of the effects – positive or
negative – on households, institutions, and the environ-
ment caused by a development program or project.
• Helps to understand effects on welfare of poor.
• Can range from large scale sample surveys comparing
project populations and control groups before and after;
to small-scale rapid assessments or participatory
appraisals, estimating impacts based on group
interviews, informants, case studies and available data.
• Distinguishes effects of project from that of other factors.
• Helps clarify whether costs for an activity are justified,
and draw lessons for improving future activities.38
Why aren’t there more World Bank impact
evaluations?
• Cost ranged from $200,000-$900,000
• Rapid impact evaluations – under $100,000.
• Requires strong technical skills in social
science research design, management,
analysis and reporting.
• Can take 2 or more years. Rapid assessments
can be conducted in less than 6 months.
Source: World Bank and OECD, Monitoring and Evaluation: Some Tools,
Methods & Approaches (Washington: World Bank, 2004) 39
EXAMPLES OF IMPACT EVALUATION
DESIGNS
“The Gold Standard”: Randomized evaluation designs,
involving the collection of information on project and
control groups at two or more points in time, provide the
most rigorous statistical analysis of project impacts and
the contribution of other factors. Rarely possible.
Most impact evaluations use less expensive designs:
• Quasi-experimental design using a "non-equivalent"
control group to match as closely as possible the
characteristics of the project population.
• Comparison of project population with a non-equivalent
control group after the project has been implemented.
• Rapid assessment combining group interviews, key
informants, case studies and secondary data. 40
World Bank best practice: engage stakeholders
in Poverty and Social Impact Assessment (PSIA)
41
Stakeholder questions answered in PSIA.
42
Summary: Information-for-accountability
• Better information through monitoring and evaluation is relatively new in some areas of public services.
• Currently, there is mostly qualitative evidence that these approaches are effective.
• These can be relatively inexpensive ways to improve the efficacy of ongoing programs.
• However, information alone does not necessarily lead to improved outcomes for the poor: enforcement mechanisms are essential to create incentives to perform well and deliver good services.
43
Monitoring and Evaluation, and the
Potential for Civil Society
Participation in Yemen
An IRDPG Civil Society
Workshop
December 16, 2009
44
