Upload
fiona-preston
View
4
Download
0
Embed Size (px)
DESCRIPTION
Saving Moms and Babies; What Does the Impact Evaluation Evidence Show?. Jeffery C Tanner, Team [email protected]. 1. Introduction to Systematic Reviews 2. SR on Maternal & Child Mortality 3. Results 4. Knowledge Gaps 5. Summing Up. - PowerPoint PPT Presentation
Citation preview
Saving Moms and Babies;What Does the Impact Evaluation Evidence Show?
Jeffery C Tanner, Team [email protected]
1. Introduction to Systematic Reviews2. SR on Maternal & Child Mortality3. Results4. Knowledge Gaps5. Summing Up
Evidence-Based Decision-Making
the contribution of Systematic Reviews► “Sum up the best available research on a
specific question” (The Campbell Collaboration)
► Make strong claims on comprehensiveness of search
► Are a form of research• Unit of Analysis: Secondary observations
(Studies)• Follow basic steps of research process• Aim to minimize bias and error
3
• Sheer amount and flow of information/ research
• Variable quality of research outputs
• Need to ‘separate the wheat from the chaff’
• Problems of publication bias
• Limitations of single studies
Why do we need systematic reviews?
2. SR on Maternal & Child Mortality
MDGs 4 and 5 Continue to Lag
► MDG5: ¾ Reduction in Maternal Mortality by 2015• Main Indicator: Proportion of births attended by skilled health
personnel
► MDG4: 2/3 Reduction in Under-Five Mortality► Knowing what to do is no longer the problem; knowing
how to do it remains a challenge
MD
G 1
.a E
xtre
me
pove
rty
(%
popu
latio
n be
low
$1.
25 a
day
, 20
05 P
PP
)
MD
G 2
.1 P
rim
ary
com
plet
ion
rate
(%
rel
evan
t age
gro
up)
MD
G 3
.1 R
atio
of
girl
s to
boy
s in
pr
imar
y an
d se
cond
ary
educ
atio
n (%
)
MD
G 4
.a M
orta
lity
rate
, inf
ants
(p
er 1
,000
live
bir
ths)
MD
G 4
.a M
orta
lity
rate
, chi
ldre
n un
der
5 (p
er 1
,000
)
MD
G 5
.a M
ater
nal m
orta
lity
ratio
(m
odel
ed e
stim
ate,
per
100
,000
live
bi
rths
)
MD
G 7
.c I
mpr
oved
wat
er s
ourc
e (%
pop
ulat
ion
with
out a
cces
s)
MD
G 7
.c I
mpr
oved
san
itatio
n fa
cil
-iti
es (
% p
opup
latio
n w
ithou
t acc
ess)0
20406080
100 7894 96
80 8068
87 87100
87 96
51 5238
100
72
Corresponding target Latest available value
Objective of this Systematic Review► Scope: Reviews impact evaluations of interventions
to improve five MCH outcomes (SBA, MM, NM, IM, U5M) and those of SBA as an intervention from scalable programs in IDA/IBRD countries
► Outcome-oriented approach: Include full range of interventions
► Aims to answer the following questions:What interventions demonstrate reductions in
maternal and child mortality and increase skilled birth attendance?
What do we know about the effects of increasing skilled birth attendance?
What important knowledge gaps remain on interventions to reduce maternal and child mortality?
Frequency of Impact Evaluations by Outcome and Quality
► AAA-quality Impact Evaluations: Established Causality• Few, if any, remaining threats to internal validity.
► AA-quality Impact Evaluations: Likely Causality• Some identifying assumptions untested or unclear
► Consistency Analysis. Key Results driven by AAA-rated IES
► External Validity, Construct Validity also considered
Skilled birth attendant
Neonatal mortality
Infant mortality
Under-5 mortality
Maternal mortality
10
9
11
5
3
23
17
12
12
5
33
8
26
23
17
AAA rating AA ratingTotal
68 Impact Evaluations, 62 Studies
3. Results
Standardized Effect Size: SBA—Outcome
Results: Increasing Skilled Birth Attendance—Outcome
► Skilled Birth Attendance rates can be improved through • Conditional Cash Transfers and Vouchers• Interventions that bundle quality
improvements with increased accessibility
► Solely training health workforce or increasing awareness of safe motherhood was not observed to yield significant results on SBA rates.
► Where reported, effects are larger for more disadvantaged households
Results: Skilled Birth Attendance – Intervention
► No Robust Evidence that solely increasing proportion of births with SBA affects mortality• Only evaluated program is JSY in India: Null
results for NM– 2 IEs, AAA and AA quality, both high-powered– No effect even in areas with high (or low) quality of
health services
• Critical Knowledge gap: Need more IEs on this MDG indicator
► SBA “+” Can affect mortality and intermediate outcomes• PLUS=Provision & Utilization: quality of care, knowledge,
access• But evidence is mixed across outcomes, even within a given
study• Unclear what explains variation in results• Consistent, if thin, evidence on better U5M, Breastfeeding,
Family planning, Postnatal visits, Immunization, Anthropometric outcomes
Intermediate Outcomes of SBA as an Intervention► SBA PLUS = Provision & Utilization► Quality of care, knowledge, access► Consistent, if thin, evidence that SBA+
results in better • U5M• Breastfeeding• Family planning• Postnatal visits• Immunization• Anthropometric outcomes
Results: Maternal Mortality
► Few (8) studies exist, concentrated in SA (5)
► Most studies underpowered to detect effects in MM
► Interventions bundling components of both health care provision and utilization can reduce maternal mortality. • Specifically, bundling health worker training
and mothers’ knowledge and information (with and without insurance)
► More evaluations are needed• 3 Delay Model, especially transport and
referral systems• Family planning, universal health
Results: Neonatal Mortality
► Health: Knowledge & Information interventions in the sample which change home-based care practices at the community level reduced mortality
► Non-health: Interventions in non-health sectors associated with maternal education consistently lowered neonatal mortality
► More IEs are needed in• 3 Delay Model, esp. Transportation and
Referral Systems• Improvements in Quality and Availability of
Health Infrastructure for newborns
Infant Mortality
► Interventions in non-health sectors consistently reduced IM• Water and Sanitation• Energy• Education
► Governance interventions report significant effects in lowering infant mortality
► Training health workers to provide continuum of care services within communities can reduce IM
► Where reported, households from lower SES benefited more
Under-Five Mortality
► Interventions in non-health sectors consistently report large reductions in under-five mortality.
► Public Participation, Service Packages may reduce U5M
► Insecticide Treated Nets are only intervention targeting three main causes of mortality that has IE evidence on U5M
4. Knowledge Gaps
Gaps by Region
15 IEs3 SBA1 NM9 IM7 U5
9 IEs5 SBA3 NM3 U5
0 IEs
1 IE1 MM1 IM
28 IEs15 SBA5 MM18 NM6 IM4 U5
15 IEs10 SBA2 MM4 NM7 IM3 U5
Gaps by Severity
[98] [94] [93][84]
[49] [49]
[32]
[83] [81] [80]
[220]
[500]
[8][11] [10]
[14]
[32][34]
[14][17] [16]
[24]
[49]
[69]
[17] [20] [20][29]
[63]
[108]
0
10
20
30
40
50
60
70
80
90
100
0
5
10
15
20
25
30
35
40
ECA EAP LAC MENA SA SSA ECA EAP LAC MENA SA SSA ECA EAP LAC MENA SA SSA ECA EAP LAC MENA SA SSA ECA EAP LAC MENA SA SSA
Seve
rity I
ndex
(100
=wor
se in
dicato
r valu
e)
Impa
ct Ev
aluati
ons
Impact Evaluations Severity
Skilled birth attendant Maternal mortality Neonatal mortality Infant mortality Under 5 mortality
Other Gaps in Impact Evaluation Evidence
► By Outcome• Maternal Mortality: limited number of studies
but highly concentrated in South Asia (5 out of 8)
► By Intervention Type• 3 Delay Model (especially transport and
referral systems)• Governance• Health information systems, infrastructure,
financing• Income generating / Labor market
interventions• Transportation infrastructure
External Validity Implications: Beneficial Impacts are more likely in problematic areas
Skilled Birth Attendance
Infant Mortality
Neonatal Mortality
Under-Five Mortality
5. Summing Up
Key Messages
► There is no IE evidence that increasing skilled birth attendance alone reduces maternal or neonatal mortality: • Importance of EVIDENCE-BASED INDICATORS for post-MDGs
► Slow progress on MDGs 4 & 5, but evidence of effective interventions• SBA: vouchers, CCTs, bundled interventions• MM: SBA+ combining provision and utilization elements• NM: knowledge & information, maternal education • IM: Governance, Energy, WASH, Ed; training community
health workers• U5: Gov & Participation, WASH, Education; health Service
Packages, ITNs
► Countries & households with higher burdens may see larger results
► Important knowledge gaps remain• Intervention: including SBA, Nutrition, 3 Delays Model
(esp transportation)• Evaluation components (subgroup analysis)
25
The Systematic Review can be downloaded from
https://ieg.worldbankgroup.org/Data/reports/mch_eval.pdf
The Database of all IEs is at
https://ieg.worldbankgroup.org/Data/mch/mch_dataset.xlsx
Thank You!
General Systematic Review resources and international bodies
► Cochrane Collaboration; 1993; www.cochrane.org• Producing high quality information about the effectiveness of health
care (> 5000 published online – Cochrane library)
► Campbell Collaboration; 2000; www.campbellcollaboration.org• Producing systematic reviews of the effects of social interventions
(>200 published online – Campbell library)
► International Development Coordinating Group (IDCG); 2010 www.campbellcollaboration.org/international_development• Producing systematic reviews of high policy-relevance focusing on
social and economic development interventions in LMICs
► International Initiative for Impact Evaluation, 3ie; 2008 www.3ieimpact.org/en/evidence/systematic-reviews
► EPPI Centre - An Institute of Education centre focusing on systematic reviews in education, health and social policy
► Collaboration for Environmental Evidence producing systematic reviews for environmental management
Backup Slides
Overview
1. Introduction
2. Methods
3. Results
4. Knowledge Gaps
5. Summing Up
28
1. Introduction
MDGs 4 and 5 Continue to Lag
► MDG5: ¾ Reduction in Maternal Mortality by 2015• Main Indicator: Proportion of births attended by skilled health
personnel
► MDG4: 2/3 Reduction in Under-Five Mortality► Knowing what to do is no longer the problem; knowing
how to do it remains a challenge
MD
G 1
.a E
xtre
me
pove
rty
(%
popu
latio
n be
low
$1.
25 a
day
, 20
05 P
PP
)
MD
G 2
.1 P
rim
ary
com
plet
ion
rate
(%
rel
evan
t age
gro
up)
MD
G 3
.1 R
atio
of
girl
s to
boy
s in
pr
imar
y an
d se
cond
ary
educ
atio
n (%
)
MD
G 4
.a M
orta
lity
rate
, inf
ants
(p
er 1
,000
live
bir
ths)
MD
G 4
.a M
orta
lity
rate
, chi
ldre
n un
der
5 (p
er 1
,000
)
MD
G 5
.a M
ater
nal m
orta
lity
ratio
(m
odel
ed e
stim
ate,
per
100
,000
live
bi
rths
)
MD
G 7
.c I
mpr
oved
wat
er s
ourc
e (%
pop
ulat
ion
with
out a
cces
s)
MD
G 7
.c I
mpr
oved
san
itatio
n fa
cil
-iti
es (
% p
opup
latio
n w
ithou
t acc
ess)0
20406080
100 7894 96
80 8068
87 87100
87 96
51 5238
100
72
Corresponding target Latest available value
Role of IEG
► Independent Evaluation in the World Bank Group
► Impact Evaluations in the World Bank Group
► Why this Systematic Review• Are we doing the right things to achieve
MDGs?• Compare Causal evidence vs Bank Portfolio• Compare stock vs need of evidence, regionally
31
2. Methods
Search Process
► 3 Search Rounds• Electronic, “hand”
and snowball search strategies
► Review and coding into 300+ fields
► 7,000 62 studies► Quality ratings by
Internal Validity• Elements of
Construct and External Validity also considered
Challenges and Cautions for Systematic Reviews► Representativeness of Interventions—non-random selection
• Overrepresentation of easily evaluable interventions• Focus on “reduced form” studies excludes those with intermediate
outcomes• Lack of evidence does not imply no effect
► Representativeness of Impact Evaluations• Includes only existing studies• Publication bias (file drawer bias)
► Interpretation of Results• IEs measure partial equilibrium; general equilibrium may be
different• Null results must be interpreted carefully—we never “accept” zero• External validity—changes to time, place, or scale may affect
results
Standardized Effect Size: Maternal Mortality
Standardized Effect Size:Neonatal Mortality
Search Criteria
► Impact Evaluations• Experimental or Quasi-Experimental design• Counterfactual
► Completed 1995 – Present► Effectiveness / Policy / Field studies
• (Rather than bio-medical and efficacy trials)
► Low and Middle-Income Countries► Representative Sample of population of interest► Peer Review► Report impacts on at least 1 outcome of interest
• Skilled Birth Attendance, Maternal Mortality• Neonatal, Infant, Under-five Mortality
What of Nutrition?
► No studies on nutrition explicitly; some “bundling”• Mortality Outcomes• Effectiveness, not clinical/efficacy
► 18/93 studies mentioned “nutrition” or supplements (or variants)• 3 AAA—Intervention: Supplements to moms/kids
– 2 on Progresa CCT in Mexico—Impacts on IM; not significant for NM, SBA
• 7 AA, only 3 Interventions (4 Outcomes) – Bangladesh—Converted nutrition workers to Kangaroo Care; not
significant for NM, IM– Bangladesh—Family planning through Community Health Workers;
highly significant for U5– Vietnam—Provided training in child malnutrition; not significant for SBA
► 6/93 with Breastfeeding: 3 AAA (all India), 3AA (India, Pakistan, Bangladesh)• 5 Bundled Interventions—all reduced NM, none improved
SBA– CHW in India also reduced IM– Women’s Group in India not significant for MM
38
Standardized Effect Size:Infant Mortality
Standardized Effect Size:Under-Five Mortality
IEs of World Bank Funding and Projects: Mortality
► No significant results for maternal or neonatal mortality► Significant, but often small effects on infant mortality.
Larger for U5.
Income increasing; India; SME(N=429,445)
Monitoring/Evaluation/Accountability, Service management; Uganda; SME(N=50)
Planning/Policy, Delivery Modality; Brazil; SME(N=38,762)
Planning/Policy, Delivery Modality; Brazil; SME(N=38,762)
Water/Sanitation; Brazil; SME(N=3,568)
Delivery modality, Health workforce, Service package; Indonesia; SME(N=1,590)
Delivery modality, Health workforce, Service package; Indonesia; SME(N=1,590)
Health workforce, Knowledge/Information, Service Package; Indonesia; OR(N=80)
Health workforce, Knowledge/Information, Service Package; Indonesia; OR(N=63)
Health Infrastructure; Planning/Policy; Water/Sanitation; Bolivia; SME(N=8,009)
Income Increasing; India; SME(N=182,869 )
Income Increasing; India; SME(N=182,869 )
Planning/Policy; Delivery modality; Brazil; SME(N=3,336)
[50] Newman and others 2002; F(U5)
[57] Rocha and Soares 2010; F(U5)
[4] Baird abd others 2011; P(U5)
[12] Bjorkman and Svensson 2009; P(U5)
[43] Macinko and others 2007; F(IM)
[57] Rocha and Soares 2010; F(IM)
[58] Shrestha 2010; F(IM)
[4] Baird abd others 2011; P(IM)
[28] Gamper-Rabindran 2010; P(IM)
[58] Shrestha 2010; F(NM)
[47] Mazumdar and others 2011; F(NM)
[39] Lim and others 2010; F(NM)
[39] Lim and others 2010; F(MM)
-1 -.5 0 .5 1size effect
estimate (AAA) estimate(AA) 95% conf. int.
IEs of World Bank Funding & Projects:Skilled Birth Attendance (Outcome)
► SBA often significant, but small effect size► Of 15 IEs on World Bank, 11 from 3 countries ► SBA & IM IEs concentrated in regions with 2nd –lowest
burden
Health financing; Planning/Policy ; Indonesia; SME(N=12,000)
Health financing; Planning/Policy ; Indonesia; SME(N=12,000)
Income increasing; India; SME(N=429,445)
Monitoring/Evaluation/Accountability, Service management; Uganda; SME(N=50)
Health Financing; Rwanda; OR(N=2,108)
Health workforce, Delivery modality, Service Package; Indonesia; OR(N=6,730)
Health workforce, Knowledge/Information, Service Package; Indonesia; SME(N=52)
Health Infrastructure; Planning/Policy; Water/Sanitation; Bolivia; SME(N=8,009)
Income Increasing; India; SME(N=182,869 )
[7] Basinga and others 2011; F(SBA)
[26] Frankenberg and others 2009; F(SBA)
[39] Lim and others 2010; F(SBA)
[47] Mazumdar and others 2011; F(SBA)
[50] Newman and others 2002; F(SBA)
[55] Olken and others 2012a; F(SBA)
[55] Olken and others 2012b; F(SBA)
[4] Baird abd others 2011; P(SBA)
[12] Bjorkman and Svensson 2009; P(SBA)
-1 0 1 2size effect
estimate (AAA) estimate(AA) 95% conf. int.
Specific Knowledge Gaps
► SBA: no robust evidence that solely increasing proportion of births with SBA affects mortality
► Mortality: • Maternal Mortality: limited number of studies but
highly concentrated in South Asia (5 out of 8)• Child Mortality: needs attention to IE quality and
intervention details
► More high quality evaluations are needed• Family planning, universal health• Improvements in quality and availability of health
Infrastructure for newborns• “3 Delay” models (especially Transport and Referral
Systems)
1. Scoping: defining answerable question, methods set out in study protocol
2. Rigorous search to identify published and unpublished sources, in any language
3. Application of study inclusion criteria (PICOS)
4. Critical appraisal of study quality, to assess how reliable is the evidence
5. Data extraction and organisation
6. Synthesis of evidence (outcomes along causal chain)
7. Interpreting results (policy and practice, research recommendations)
8. Improving and updating reviews as new evidence emerges
What makes a systematic review ‘systematic’?