Futures Group and the World Bank Institute in collaboration
with Abt Associates, OHanlon Health Consulting, University of
California at San Francisco and Tropical Health LLP JANUARY
2014
Slide 2
MARKETS FOR HEALTH SESSION 11 What do We Really Need to Know?
M&E and its Application to M4H Ruth Berg Futures Group Futures
Group and the World Bank Institute in collaboration with Abt
Associates, OHanlon Health Consulting, University of California at
San Francisco and Tropical Health LLP
Slide 3
MARKETS FOR HEALTH OVERVIEW M&E FUNDAMENTALS M4H
CONSIDERATIONS REAL WORLD EXAMPLES SUMMARY
Slide 4
MARKETS FOR HEALTH Evaluation Types (the Short List) Types of
evaluation Answers the questionBasic requirements Process
evaluation / monitoring Is the programme on track? and Is change
occurring according to plan? Construct good indicators Valid
accurately measures behavior Reliable minimizes measurement error
Precise defined in clear terms Measureable - quantifiable Timely in
step with the programme Performance evaluation Is change occurring
according to plan? And Can we plausibly attribute change in outputs
or outcomes to the intervention? Triangulate information Program
records and monitoring data Data from secondary sources Expert
interviews Stakeholder opinions Impact evaluation Can we rigorously
attribute change in outputs or outcomes to the intervention?
Construct a counterfactual - estimate what the outcome would have
been without intervention Experimental designs Quasi experimental
designs Non-experimental designs
Slide 5
Evaluation Purpose Spectrum Process Evaluation/ Monitoring
Performance evaluation Impact evaluation Throughout implementation
Before implementation & post implementation Learning and
adapting programme Learning and decision making (scale/replication)
Improving Proving Type Timing Use Internal team External team (
close collaboration with internal team) External team ( close
collaboration with internal team) Distance Adapted from ITAD 2013
Learning and adapting Mid-project & end of project
Slide 6
MARKETS FOR HEALTH OVERVIEW M&E FUNDAMENTALS M4H
CONSIDERATIONS REAL WORLD EXAMPLES RECOMMENDATIONS SUMMARY
Slide 7
An ITAD (2013) review of 14 M4P evaluations concluded: The
nature of the [M4P] approachpresents a number of challenges for
evaluation The M4P evaluationswere often weak in terms of
Consideration of systemic, sustainable changes in market systems
Triangulation practices Theories of change (often linear)
Considerations for Market System Approaches
Slide 8
InputProcess Health Output Health Outcome Health Impact Why?
Dynamic Effects of Market System Approaches at Odds with
Traditional Linear Results Chains Traditional Results Chain
Large-scale
Slide 9
Requires More Adaptation and Change During the Process Stage:
Less is Known at the Theory of Change Stage M4P Component E M4P
Component D M4P Component B M4P Component A M4P Component C Setting
the Strategic Framework Understanding Market Systems Defining
Sustainable Outcomes Facilitating Systemic Change Assessing Change
Monitoring & Evaluation Vision & Rationale Identification
& Research Monitoring & Evaluation Implementation &
Adaptation Information & Feedback
Slide 10
Systems Approaches are Often Unpredictable So the Learning
Process can Look More Like This Learning MonitoringStrategy Adapted
from Preskill and Beer, 2012
Slide 11
InputProcess Health Output Health Outcome Health Impact Linear
Logic Model vs. Systemic Logic Model Systemic Large-scale
Sustainable Linear logic model Systemic logic model for M4H Adapted
from ITAD 2013 M4H Intervention Market Change Health Output or
Outcome Sustainable Large-scale Systemic Adaptive Process
Slide 12
Impact Trajectory for Impact is Likely to Be Non-Linear
Non-systemic approach Impact Time Systemic approach Start of
projectEnd of project Baseline Endline Post-endline Source: Adapted
from DCED 2011
Slide 13
Explicitly monitor Systemic change Whether changes in rules or
supporting functions occurred (yes/no) Sustainability Whether
market system change continues after intervention without external
assistance Whether health outputs/outcomes continue after
intervention without external support at the same or improved
levels (yes/no) Scale Whether intervention crowds-in of other
actors Whether intervention leads to copying by other actors % of
targeted population reached by the intervention Market system,
outputs or outcomes after the project ends What Does this Mean for
M4H M&E? Process evaluation/monitoring
Slide 14
Plausible attribution Usually best option because it allows
Non-linear impact trajectory Adaptive approach to implementation
Expected spill-over effects Rigorous attribution through
experimental designs Faces important challenges for systems
approaches They dont allow Spill-over effects Contamination Need to
stick to your original intervention (no adaptation as you go)
Possible at pilot stage if Beneficiaries can be randomly assigned
to treatment & control groups (RCTs) Baseline equivalency can
be established in key variables (quasi-experiments) What Does this
Mean for M4H M&E? Evaluation & Attribution
Slide 15
MARKETS FOR HEALTH OVERVIEW M&E FUNDAMENTALS M4H
CONSIDERATIONS REAL WORLD EXAMPLES SUMMARY
Slide 16
MARKETS FOR HEALTH Using Plausible Attribution to Measure M4H
Light in Morocco
Slide 17
R&D Rules Regulations Standards Laws Informal rules &
norms SD Information Quality Assurance Subsidy Infrastructure
Purchase Partnerships Health services & products Markets for
Health Framework (M4H) Related Services Invest Private Sector
Representative bodies Government Providers, Sellers Consumers,
Patients Not-for-profits
Slide 18
Ministry of Healths system goal: Change incentives of OC
manufacturers in enduring way to Ensure commercial sector reaches
low & middle income MoH developed four-way OC partnership
(1992) With USAID, Wyeth Pharmaceuticals, & Schering, &
IPPF affiliate In consultation with Pharmaceutical Association The
negotiation MOH Changed regulation to permit OC advertising
Required brands to have common logo: Kinat al Hilal Wyeth and
Schering Price brands 30% lower than nearest commercial product
Nation-wide to achieve scale USAID Manage advertising fund &
campaign during start-up Transition to IPPF affiliate to sustain
campaign post-graduation Supporting Function Oral Contraceptive
(OC) Partnership Morocco (1992 2003)
Slide 19
How did it work and how do we know (checking and adapting)
Performance evaluation showed plausible attribution using
Triangulation of different data sources Monitoring data from four
rounds of Demographic and Health Surveys Pricing data Examination
of different variables Do they tell a story consistent with the
strategy and activities? From launch to post-project period OC use
increased commercial share increased reduced prices remained stable
equity improved How do we know if it worked?
Slide 20
% of MWRA Who Use OCs OC Demand Steadily Increased Percent
Sources: Demographic and Health Surveys 1992, 1995, 2003:
Commercial Market Strategy Survey: 2000
Slide 21
Commercial Share of OC Use Increased 19922003 Public & NGO
Commercial OC Source Mix
Slide 22
72% 118% Commercial Share of OC Use by Wealth Quintile 18%
Commercial Sector Reach among Poor Increased Percent
Slide 23
Current modern contraceptive use among women in union by wealth
quintile, Morocco, 1987 2003. Agha S, and Do M Health Policy Plan.
2008;23:465-475 Published by Oxford University Press in association
with The London School of Hygiene and Tropical Medicine The Author
2008; all rights reserved. Crowding-in effect?
Slide 24
Kinat Al Hilal s Prices Remained Stable Average Price (USD) per
OC Cycle
Slide 25
MARKETS FOR HEALTH Take-aways Plausible attribution requires
looking at multiple data sources and variables can be inexpensive
by using secondary data National data sets allowed assessment of
scale Post-endline review allowed assessment of sustainability
Missed opportunity Potential crowding in could have been checked
with interviews Systemic changes take time, but potential pay off
is large Impact took over 10 years with extensive planning and
negotiation
Slide 26
MARKETS FOR HEALTH Rigorous Attribution to Measure M4H Pilot in
Nicaragua
Slide 27
Situation: 93% of informal sector workers lack health insurance
Nicaraguan Social Security Institutes (INSS) goal: Extend health
insurance programme to informal sector Increase health care
utilization INSS contracted with 3 microfinance institutions (MFIs)
to market voluntary insurance enroll beneficiaries collect premiums
17 predominantly private clinics to provide services Under
capitated payment system Used an RCT to Measure impact of health
insurance coverage on service utilization Remove selection bias
(sick people more likely to sign up) Financing Function Health
Insurance Pilot Managua, Nicaragua (20072008)
Slide 28
2007 baseline study conducted with 4,000 market vendors Vendors
(sellers in market) randomized to receive Brochure about the health
insurance program 6 months of free INSS insurance Nothing Subsidy
winners randomized to register for insurance at INSS office or
Local MFI 2008 endline study conducted with 2,608 of these vendors
Two-stage least squares regression analysis conducted To measure
effect of insurance on health service utilization and expenditures
Controlling for endogeneity of insurance Study Design
Slide 29
Baseline Suggests Balanced Randomization ControlInfo
Only6-Month INSS 6-Month MFI Age383738 Male.38.34.36.35 Years of
Education9999 Married.78.71.69.68 # of Children2.22.02.22.0
Smokes.18.16.14.15 Income ($US)287255298263
Observations1111051659623 * Table refers to 2,608 respondents who
participated in both baseline & endline
Slide 30
Main Findings Signing up Subsidized vendors more likely to sign
up for insurance than controls 29% versus less than 1% Time &
costs were main reason for not signing up Service utilization No
net increase in healthcare utilization Total health expenditures
for insurees decreased by 36% For those with chronic illness,
health expenditures decreased by 89% Retention Less than 10% of
enrollees were still paying for insurance a year later Those with
subsidies were the least likely to be retained over time Main
reasons for dis-enrolling Expense of premiums
Slide 31
MARKETS FOR HEALTH Take-aways RCTs allow rigorous attribution
Remove selection bias and give accurate answers to important
questions In general, systems approaches do not lend themselves to
RCT evaluation due to scale & complexity RCTs are possible for
some types of M4H pilots Pilot must allow randomization of
beneficiaries into treatment and control groups Determine whether
system intervention is worthy of scale-up Reporting failures makes
important contribution to knowledge base
Slide 32
MARKETS FOR HEALTH OVERVIEW M&E FUNDAMENTALS M4H
CONSIDERATIONS REAL WORLD EXAMPLES SUMMARY
Slide 33
MARKETS FOR HEALTH Session Summary M4H and other systems
approaches pose important M&E considerations Need to explicitly
monitor systemic change, scale, and sustainability Need to monitor
after project ends to assess Expected J-curve impact Sustainability
results Rely on plausible attribution most of the time Using mixed
methods and triangulation Use rigorous attribution when it makes
sense Pilot phase Testing an innovation Randomization of
beneficiaries is possible (RCTS) Baseline equivalence possible
(QEDs) Spill-over effects and contamination are unlikely Cost can
be kept down