Social Development DepartmentThe World Bank
Participatory Approaches in
Impact Evaluation
Asli GurkanSocial Development DepartmentWorld BankDubai – Impact Evaluation workshopMay 31-June 4
Demand for Good Governance
Setting the Context Heightened attention to governance issues at
the World Bank since adoption on GAC strategy Increasing emphasis on outcomes/impact to
enhance development effectiveness More attention to transparency, access to
information, citizen-participation at all levels in World Bank operations (mandatory with Investment Lending reform)
Key focus of Social Development department-strengthening demand-side of governance (including PM&E methods)
Demand for Good Governance
What is demand for good governance (DFGG)?
The ability of citizens, civil society organizations and other non state actors to hold the state accountable and to make it responsive to their needs
In turn, DFGG enhances the capacity of the state to become transparent, accountable and participatory in order to respond to these demands
DFGG mechanisms can be initiated and supported by the state, citizens or both
but very often they are demand-driven and operate from the bottom-up.
The people have a right to know, a right to question, a collective Constitutional right to receive an answer.” Aruna Roy, MKSS Rajasthan, India
Demand for Good Governance
Key Demand-side mechanisms to address governance challenges
Transparency/Access to InformationConsultation/ParticipationGrievance Redress MechanismsThird party Monitoring/ Independent
verification of outcomes Participatory Impact Assessments
Demand for Good Governance
6
Possible methods/tools under PM&E:
Visual techniques Activity monitoring chart Participatory Rural Appraisal Citizens report cards Community scorecards SARAR (participatory problem solving tool) Participatory Impact Assessments
Demand for Good Governance
7
What’s Participatory Impact Assessment?
“Involves the adaptation of participatory tools combined with more conventional
statistical approaches specifically to measure the impact of humanitarian
assistance and development projects on people’s lives.”
Source: Feinstein International Center: Participatory Impact Assessment” Guide for Practitioners
Demand for Good Governance
8
Participatory versus Conventional IE
Conventional IE Participatory IEWho? External experts, IE
specialistsStakeholders, including communities and project staff; service providers and users
What? Predetermined indicators, to measure impact
Indicators identified by stakeholders, to measure impact
How? Questionnaire surveys, by outside “neutral” evaluators, distanced
Simple, qualitative and quantitativemethods, by stakeholders themselves guided by facilitator
Why? To make project and staffaccountable to funding agency
To empower stakeholders to takecorrective action
Method?
Pre-determined AdaptableSource: Adapted from Deepa Narayan, World Bank.
Demand for Good Governance
At the project level-3 key questions…
1. What changes have there been in the community since the start of the project?
2. Which of these changes are attributable to the project?
3. What difference have these changes made to people’s lives?
Demand for Good GovernanceDesigning and
Implementing PIA: StepsStage 1: Define QuestionsStage 2: Define the geographical and time-limits of
the projectStep 3: Identify locally defined impact indicatorsStep 4: Decide on ranking/scoring methods
on and testing sampling methodsStage 5: Choose Sampling MethodsStage 6: Assess project attributionStage 7: TriangulationStage 8: Feedback and verify results with
communitySource: Feinstein International Center: Participatory Impact Assessment” Guide for Practitioners
Demand for Good Governance
Step 2, Tool #1: Defining the project boundary: participatory mapping
A map of Zipwa Site, Zimbabwe Community members drawing a map in the sand
Demand for Good Governance
Step 2- Tool # 2: Define the project period by timelines established by the communities
Creating a timeline--- Identify a Knowledgeable person (or persons) in a community- Ask them to describe the
history of the community.- - In many rural
communities, such descriptions usually refer to key events such as drought, periods of conflict or disease epidemics
- The project start and end time should be related to these key events.
Demand for Good Governance
Step 3: Indicators-identified by communities themselves…
Ex: Drought projects in Zimbabwe and Niger
Impact indicator by project M&E specialist
Impact indicators by beneficiaries
increased crop production The ability to pay for school fees using project derived income (education benefits)
dietary diversity The ability to make home improvements Improved skills and knowledge from the projects training activities
Improved social cohesion
Tips for practitioners: Make sure to capture the views of different groups ofpeople within the community. (Women will often have different priorities and expectations of project impact than men.)
Demand for Good Governance
project participants identify all the food sources that contribute to thehousehold food basket.
Practitioner Tips- Where informants are literate you may choose to simply write the name ofThe indicator on a card.
Step 4: Methods for Scoring/EvaluationTool #1: Scoring of Food sources using counters- Evaluating the impact of a community garden
Demand for Good Governance
Step 4-Tool # 2: Impact calendars-post-harvest food balance
Monthly household utilization of the harvested maize until depletion (using 25 counters)
exercise -done with project participants for the agricultural year before and after the project and again for the agricultural year. The exercise then repeated with community members who had not participated in the project
Demand for Good Governance
Challenges with community-level PM&E approaches (from Bolivia
and Nepal) Clash of incentives: pressure from donors to ‘prove’ impacts
vs. adoption of a bottom-up, participatory approach based on ‘improving’ programs in ways that meet community needs and aspirations
Resistance from the project teams to changing their existing M&E practices
Project deadlines prevent project-staff from establishing a consistent PM&E practice
Availability of PME expertise within the field staff to facilitate the PM&E exercises, Insufficient transfer of community-level PME skills to interest
groups and grassroots organizations Lack of sufficient training/capacity-building programs
Human resource problems and lack of capacity in analyzing qualitative data and reporting results.
Demand for Good Governance
Possible recommendations to improve PM&E activities
Develop ready-to-use templates, a detailed Community Researcher manual to improve the research data and reports.
Conduct regular follow up visits to each case study sites to review the work of the Community researchers and provide feedback, advice and support
Identify mentors, encourage the community researchers to phone their mentors on a regular basis to share their progress and any problems they had.
Encourage the community researchers to contact each other regularly to share their experiences and reports. Promote “peer-to-peer learning’
Demand for Good Governance
Key messages Keep participatory evaluation methodologies
simple and practical
Develop your methods, standardize and repeat.
the more repetitions, or the larger the sample size, the more statistically reliable the results will be.
Remember to field test your methods with community members before the assessment–most methods look easy on paper but require fine tuning once you start to use them in the field.
Demand for Good Governance
Participatory Impact Monitoring Booklet I- V GTZ
World Bank Participation and Civic Engagement Website:
Feinstein International Center-Tufts University Participatory Impact Assessment: A Guide for Practitioners
Useful links/Resources
World Bank Social Accountability Sourcebook
Demand for Good Governance
Thank you