The WSP Global Scaling Up

Handwashing Behavior Project

Six month team meeting May 29, 2007

Monitoring and Evaluation

2

M&E Organization

Impact Evaluation

– Objectives defined primarily by Gates

– IE Team is responsible for technical design, But:

• coordination/integration with county programs is essential

• Overlaps with project monitoring are significant Project Monitoring

– Primarily a tool for project managers

– Managers should have ultimate authority (Is this true? Or should it be consultative?) over design, resources, strategic design and data collection

– Overlaps with IE should benefit both activities

3

Impact Evaluation

Objective: Learn what works, why and how well:

What are impacts of HW-BC interventions? • Behavior • Environment• Health, social, and economic welfare

What are the antecedents to behavioral change?

How did the interventions affect outcomes?

What is the cost-effectiveness of these interventions?

Share lessons learned with countries, donors and other partners.

4

Project Monitoring/MIS

Project Monitoring system(s) serves the project managers.

It helps managers ensure that the project is functioning as it should.

If done well, it expands, contracts, and refocuses at the discretion of project manager.

5

IE: Outcomes of Interest

Intermediate – Sanitation coverage and open defecation– Reduced fecal contamination of houses, hands,

food, water (? - not yet funded – country permission is not yet clear)

Final– Reduce diarrhea– Reduce parasite loads(?)– Reduce wasting, stunting and anemia(?)– Improve cognitive development(?)– Improve wages and productivity– Social welfare (security, female schooling, etc.)

6

IE: TS-SM Logical Chain of Evidence

A compelling logical chain of evidence improves credibility. We hope to demonstrate:

A plausible behavioral change/ consumer demand model (closely linked to the Results Framework) through the BCF/model;

Microbiological evidence of infection transmission channels (not in our current RF);

Statistical links between intermediate and final outcomes (sanitation diarrhea reductions);

7

IE: Data Requirements

Two-round household survey– Household demographics, economics– Household handwashing, sanitation and water– Intervention Process indicators (knowledge, etc.)– Measures of microbiological contamination– Health, nutrition and cognitive status– Employment, wages, productivity– School enrollment & attendance; sense of security

Monthly monitoring– Diarrhea among children <5 yrs. – Community Sanitation– Handwashing behavior– Where useful, institution-based disease/absentee reporting

8

IE: Use of Intermediate Outputs

The Results Framework should tell us whether the intervention worked as expected

Some program diagnostic information can be gathered, but most of these will be available only after the project. Usefulness to Gates is limited

9

IE: Implementation Implications

Baseline - the baseline survey needs to be in place prior to full-scale implementation. But can trial implementation be monitored before full-scale roll-out?

Treatments & controls - choosing treatment and control groups requires:

– Buy-in/agreement from field, Gov’t, stakeholders, implementing institution(s), other development partners

– Detailed information on roll out, target population, unit of intervention (hh / community / village / district…), number of units to be covered, inhabitants per unit, geographic area, etc.

Country-Specific IE Concept notes - the specific interventions determine the IE design – specific implementation details at country level are crucial to the design of the country-specific concept notes

10

Impact Evaluation: Where are we?

Steps already taken Organizational model: variations on a common theme Experimental designs

• Mass-media and EE generally not randomized• Direct Consumer Contact activities randomized

Country PIs and supporting teams • Peru – Paul is leading, w/Sebastian Galiani, Alex and

others;• Senegal – no decision yet;• Tanzania – Sebastian Martinez will lead; supporting staff

proposed; Initial visit scheduled;• Vietnam – Pascaline Dupas will lead.

11

IE: Where are we? (cont’d)

Steps to be taken Country PIs – Senegal’s needs to be proposed. Peru’s and

Vietnam’s needs to be contracted; Tanzania done in-house Nearly all country supporting teams must be identified; Survey firms

• Tanzania – really only one leading candidate; non-competitive contract may be needed

• Peru – already discussing w/survey firms• Vietnam and Senegal – nothing known

Questionnaire development• July 9-10 meeting to prepare final x-country draft• Translation/reformatting July/August• Interviewer training August/September

Fieldwork starting mid-late September at earliest

12

IE: Implications for implementation

Baseline - the baseline needs to be in place previous to implementation in the treatment areas. Note that the initial/trial roll-out can (and probably should) be implemented in non treatment/control areas before the baseline survey.

IE Concept note - the intervention defines the IE design – implementation details at country level are crucial to design country-specific concept notes

Treatments & controls - choosing treatment and control groups requires:

– Buy-in/agreement from field, Gov’t, stakeholders, implementing institution(s), other development partners

– Detailed information on roll out, unit of intervention (hh / community / village / district…), number of units to be covered, habitants per unit, geographic area, etc

13

Objective of Project Monitoring/MIS

Determine whether inputs (from RF) are implemented as and when planned (according to workplan/milestones)– Identify problems early, when they occur– Facilitate problem diagnoses and effective solutions

How? Assess, in real time, whether inputs are achieving intended outputs– Do HHs know of interventions?– Have they heard behavior-change messages?– Do they intend to change their behavior?– Is it reasonable to expect that they will?

Allows for intervention improvements in real time

14

General Model of Project Monitoring

Inputs

Intermed. Outputs: Awareness/ Intent

Final Outputs: Behavior Activity

HH Health Outcomes

Routine Monitoring: can be 100% coverage

(if contracted)

Verification/QA5% - 25%

Behavior monitoring?

Eval. & Review

15

MIS: data collection tools

Contract-based (Tanz; Vietnam)– Build routine reports into field activities

(strike a balance between informative and non-disruptive reports)

– Quality Assurance (QA) required (objective: unbiased routine reports)

Public sector institution-based (Peru schools, Vietnam MOH facilities)– Incomplete/biased reporting concerns and solutions

Independent Surveys– Most costly data collection, but needed for IE and some

QA

16

MIS: Where are we (cont’d)?

Program monitoring data collection systems Peru: School program + contract-based community

DCCs – two systems needed Tanzania – All contract-based? Vietnam – at least two systems probably needed

(one for contract-based work; one for government-based work)

Senegal?

Do we need a strategic plan for intensive early KAP monitoring?

All should establish QA Plans

17

MIS tools (cont)

Other monitoring activities

QR / AR

– Gates required; Frequency: quarterly and annually

– Based on country-specific RFs / milestones → need to review / revise (if appropriate) for the 2nd QR

HAMR

– Project required; Frequency: biweekly

– Crucial for lessons learnt

Monitor Enabling Environment

Monitor Implementation Effectiveness

18

MIS tools (cont)

Other monitoring tools

Budget Monitoring

– WSP required; Frequency: currently every QR and AR

– New TF structure under discussion: only one TF per country. Implications → simpler but calls for more rigor to not exceed (+/-) 10% limit imposed by Gates

– Budget needs to be revised at country and global level before submission of 2nd QR (mid June)