Program Evaluation: a potential platform for cross site analyses

Program Evaluation: Program Evaluation: a potential platform for cross site analyses a potential platform for cross site analyses

Louise C. Ivers, MD, MPH and Joia S. Mukherjee, MD, MPHPartners In Health,

Division of Global Health Equity, Brigham and Women’s Hospital, Harvard Medical School

Context: Reason for Program Context: Reason for Program EvaluationEvaluation

• GHI spending may not be distributed evenly throughout countries

• Sub-recipients / partners at district, local or grassroots level are involved

• Program-level evaluation gives opportunity to examine variability in impact that may not otherwise be seen at a macro level

Program Evaluation: AimsProgram Evaluation: Aims

Evaluate performance of local health system– impact of GHI on HS– Impact of HS on GHI

Facility level evaluationCommunity level evaluation

MethodsMethods

• Quantitative data at program level – Comparative analysis– Historical control– Projected targets

• Qualitative data via ethnographic surveys and focus groups with patients and providers

Performance of Health SystemPerformance of Health System

• Scale• Scope• Distribution• Quality/safety of services• System capabilities (response to emerging

and changing challenges)

Process: Human ResourcesProcess: Human Resources

NumberDistribution / per population servedSkill levelSkill mixAccreditation

OutcomesOutcomes

• Targeted disease/health outcomes– GHI specific

• Non-targeted health outcomesChild Health

DTP3Diarrheal illness (2 weeks) Under 5 malnutrition (diagnosis and treatment)

• Maternal HealthBirths assisted by skilled workerFamily planning uptake

Outcomes: Patient-focused lensOutcomes: Patient-focused lens

• Quality of services• Diversity of services• Health literacy• Fairness

Delivery / access / linkagesDelivery / access / linkages

• Attendance at antenatal clinic• Attendance at primary care clinics• Number of children weighed• Existence of functional referral systems

Community health clinic secondary/tertiary community

Lascahobas, Haiti: Lascahobas, Haiti: Attribution of FinancingAttribution of Financing

% GHI % MOH % external % private

34% 6% 60% <1%*

Lascahobas, Haiti:Lascahobas, Haiti:Work force, 2002 vs. 2007Work force, 2002 vs. 2007

Before After

MD 0 2

Nurse 2 3

Nurse aux 1 4

Pharmacist 0 1

Lascahobas, Haiti:Lascahobas, Haiti:Health Service Utilization, Health Service Utilization,

2002 vs. 20072002 vs. 2007Before After

PHC visits daily 10 400

ANC visits annual 10 16920HIV cases total 0 1200

TB cases annual 5 150

Family planning ? 5178

Care TargetsCare Targets

• MOH targets for primary health care utilization: 10% of population per month, 1.2 visits per capita per annum

• WHO targets for ANC: 4% of total population is pregnant in a given year

• Global TB report TB incident and prevalence• UNAIDS and country reports for HIV prevalence

Lascahobas, Haiti:Lascahobas, Haiti:Is it stronger?Is it stronger?

Utilization % of target

PHC visits daily 400 110%

ANC visits annual

16920 130%

HIV cases total 1200 100%

TB cases annual 150 120%

Family planning 5178 47%

Using the Yard StickUsing the Yard Stick

• Disease specific: HIV detection, ART enrollment, TB case detection and treatment, pMTCT

• Total number : general visits, ANC, family planning uptake, immunization

• Health personnel: numbers and cadres• Drugs: % of full “essential drug package”• Diagnostics: number of diseases “diagnosable”

Rusumo, Rwanda: Rusumo, Rwanda: 2006 vs. 20072006 vs. 2007

% GHI % MOH % external % private

42% 7% 48% 3%

Before After

MD 0 2/5

Nurse 0 7

Nurse aux 5 7

Pharmacist 0 0

Rusumo, Rwanda:Rusumo, Rwanda:Is it stronger?Is it stronger?

Before After % of Target

PHC visits daily

20 84 55

ANC visits annual

200 1925 26

HIV cases total

4 263 36

TB cases annual

7 40 71

Family planning

? ?

Further ChallengesFurther Challenges

• Outcomes• Temporal changes—strengthening,

integraton• Evaluating quality of services• Evaluating counterfactual—clinics that have

not had the benefits of GHI monies OR the GHI monies were used only vertically

Next steps:Next steps:

• Developing a data abstraction tool Aug 28, 2009

• Evaluation of 2-3 PIH supported MOH clinics in Haiti, Rwanda, Malawi, Lesotho and one counterfactual in each Jan 1, 2009

• Visits to 4-5 other countries to analyze a clinics with tool Mar 1, 2009

• Analyze and interpret and write May 1, 2009