Upload
moses-hawkins
View
219
Download
1
Tags:
Embed Size (px)
Citation preview
1
Measuring Results and Evaluating Impact:
What You Need to Know
Paul Gertler and Barbara BrunsHuman Development Network
November 2006November 2006
2
How do we turn this teacher…
3
…into this teacher?
4
What do we need to know??
• How many teachers • How many are performing like teacher A, like
teacher B, or absent altogether• How many kids:
– enrolled in school– attending school– repeating grades– dropping out – completing primary school
• Trends over time: are things moving in right direction?
5
What do we need to know??
• Student learning – which kids are learning and how much? – who is benefiting?
• Costs (per student) • What inputs, policies and incentives have greatest
impact on learning? – remedial teachers, multi-grade teacher training,
deworming pills– higher teacher standards, curriculum focused
on early grade reading– contract teachers, school-based management,
vouchers• Which are most cost-effective?
6
Message 1: Monitoring and Evaluation is our core business
• All TTLs need tools and techniques for– Better monitoring of Bank/IDA operations:
• defining project development objectives clearly• credible results chains • selecting indicators
– Building client capacity for M&E – Impact evaluation
• Both are important, and some complementarities– Standardized student learning data and good
administrative data (health facility surveys, school census data) lower costs and improve quality of impact evaluations
7
Message 2: We’re not doing a very good job at present
• All Networks struggling with need for better monitoring and more impact evaluation– INF study: over 200 different indicators in water supply
projects, measuring similar things, most w/only 1 obs.
– OPCS study: 78 education projects exiting in FY04-05, • Only 3 reported MDG (primary completion rate)• Only 5 had any data on student learning – and none
comparable• Less than 50% tracked enrollments (mix of GER,
NER, raw numbers, gender)– In S Asia HNP projects, avg. operation had 26 indicators,
only 33% with clear definition, baseline and collection schedule
8
Little Progress of M&E Over Time(from Ben Loevinsohn and Aaka Pande)
0
10
20
30
40
50
60
70
80
90
Indicatorsmeasurable
Clear target Baseline datacollected
Follow-up datacollected
Data collectionroughly in
keeping withPAD
Before 2000
After 2000
9
Message 3: HD under special pressure with “results agenda”
• Greater accountability for major global goals– 2/3 of MDGs are HD outcomes– HD outcomes important in IDA 14 results
• build country capacity to measure these outcomes
• link PDOs to “higher level” sectoral or MDG outcomes
• Greater role in “selling” aid – OECD electorates focused on HD issues – selling IDA 15 on “results”, esp. HD results
10
Message 4: there is help….• Tools
– Checklist for M&E in HNP operations, also education– Guidance on indicators (AFR Generic Results
Framework for HIV projects)– Common instruments for classroom observation and
student learning assessment • Technical advice
– “on demand” impact evaluation clinics– FPSI “on demand” PDO/monitoring clinics
• Financing– BNPP funding to launch prospective impact evaluations
• Training
11
Plan for next 3 daysTODAY• Understanding the value of impact evaluation• Understanding the results agenda from an institutional perspective• TTL experience in improving M and E• 4 hour module on Improving “M” -- results focus in projects
TUESDAY • TTLs will meet in MC-5-100• Researchers will meet in IFC (F-5-P100)• All day on impact evaluation – methods, operational issues
WEDNESDAY• Everyone will meet in MC-4-100• Teams organized along sector lines will prepare evaluation designs
for projects of their choice & write concept notes• Teams present in evaluation clinics