Assessing impact of health aid: how can public health

sciences contribute to mutual accountability?

Moving from evaluating the impact of programmes and interventions to

health systems and capacity strengthening

Phyllida Travis, 6 February 2012

IHP+ experience with

1. Issues and challenges in communicating results of health aid

2. Some examples of linking aid effectiveness to 'results'

3. Of strengthening mutual accountability for results

4. Approaches to making health aid for system strengthening more effective / efficient

Who is IHP+?

2007 2012

Developing countries: 8 30

Bilateral donors: 8 13

Int'l agencies and foundations*: 11 12

TOTAL 27 55

*African Development Bank, Bill and Melinda Gates Foundation, European Commission, GAVI Alliance, Global Fund, International Labour Organization, UNAIDS, UNICEF, UNDP, WHO, UNFPA, World Bank

1. Issues, challenges in communicating results

Want results that are memorable, persuasive, credible, useful, but

Politicians / public v public health experts Country policymakers/managers v global perspective Stories v numbers Progress v impact Impact of aid per se v impact of way give aid Joint v individual results Plausible v causal links 'Quick wins' v sustained gains Improved messaging now v in 5 years, w new methods

Communicating benefits, results

IHP+ country teams meeting 2010: Clear progress on putting aid effectiveness principles into practice Need to better document results / outcomes, find ways of

conveying better. Need mix of good evidence, good stories, moderate claims

In 2011 IHP+Results annual report on signatories' progress - visual; what

not why OECD TTHATS report for Busan - sound; not sexy Accumulation of narrative country examples - what, why, how;

distillation needed Commission on Information and Accountability - new catalyst

May 2011 IHP+ Results report Individual scorecards

And comparisons across partners

2. Some country specific examples linking aid effectiveness and results

Ethiopia, Nepal, Mali (all of which have positive trends in service delivery and health outcomes) plus DRC, Benin

Asking 3 questions To what extent have aid effectiveness principles been

put into practice? Has this contributed to better results?

Has health aid actually become more effective? Have health systems been strengthened? Have health services improved?

What was critical in achieving these results? Where were the barriers and how have they been overcome?

Aid effectiveness and results…different perspectives

Can trace plausible links from 'putting Paris into practice' via stronger systems to improved services and then …more tenuously….health outcomes

All note 'two speed implementation' of Paris agenda Is also an argument that goes

Outcomes are important Proving link between AE and results (outcomes) is more

of a global than country concern Risks excluding other important, common sense benefits to

countries of working to 'one plan, one budget, one report'

Need frame the debate to reflect wider range of benefits/results

Links between effective aid and results: examples Nepal: scale up of free maternal health care from a few districts

to nation-wide over 2-3 years was possible because government and donors acted collectively – no one could have done it alone. Institutional deliveries risen from18% 2006 to 28% 2011.

Ethiopia: Health extension worker programme went nation-wide over 5 years, faster than many thought possible, because FMOH provided a clear vision; credible plan and indicated where support was needed from DPs

DRC: A new MOH single donor coordination unit led to threefold reduction in management costs for donor funds, from 28% - 9%. Liberated funds for other uses. Message: better coordination of resources can deliver greater value for money.

DRC: Reform of health financing – some results MOH wanted to see

The reform benefits from the support, both technical and financial, of all partners even if its utilisation is gradual

AE principles in practice? Other types of results one MOH is

interested inIndicators Target for 2009/10

(Compact)Achievement 2009/10

% DPs providing confirmation of long term commitments

95% 63%

(no baseline available)

% DPs which don't ask Government for a separate plan

95% 58%

(baseline 23%)

% DPs' activities and budgets reflected in the government's plan

100% 76%

(baseline 44%)

% of Funds provided through Government

preferred modalities

60%

19%

(baseline 24%)

3. Strengthening mutual accountability

MA important route towards greater aid effectiveness, but slow progress

Mutual accountability efforts in health Countries

MOH and EDPs: Compacts; some MOHs incorporating indicators of DP aid behaviour into JAR agendas eg Mozambique, Ethiopia, Nepal…

MOH and parliaments: politicians getting involved eg Nigeria MOH and CSOs, research institutes – mixed; efforts to increase;

Uganda Global / regional level

3rd round of monitoring by IHP+Results – 19 countries in 2012 (10 last time); 15 DPs

Upcoming meeting of southern health CSOs – will discuss scorecards as one tool for MA

Note: mutual v joint accountability / responsibility crops up a lot in-country

4. Making health aid more effective in health system and capacity strengthening: Nepal

Existing instruments 5 year health sector programme (NHSP2), results framework;

partnership agreements; joint annual review (JAR) 2012 JAR: systems problems prominent

E.g. only 15% of facilities had no drug stock outs (target 70%) Getting TA better aligned with NHSP-2 priority problems a major theme

New actions to harmonize/align TA with priority problems TA matrix against NHSP2 objectives identifies gaps / overlaps Joint technical co-operation agreement agreed To match TA to priorities in Annual Work Plan and Budget

Ie not just about systems knowledge/ research but practical instruments

Factors enabling aid to effectively contribute to

health systems and capacity strengthening

Critical enabling factors seem to be Government ownership Leadership from MOH and at least some DPs Strategic plans and review processes established and taken

seriously Joint financing arrangements that are adhered to (including

more predictable funding), complemented by more joined up TA?

Some home-grown form of mutual accountability

Common barriers DP own organizational incentives/adaptability Willingness to synchronize work Trust

spares

Moving from impact evaluation to health systems and capacity

strengthening An aid effectiveness perspective…joint responsibility Context

post Busan; Old chestnuts

One does not follow the other – move in parallel what politicians and the public listen to – and when Impact has to be more than health Benefits of 'more effective aid' often on more proximal

aspects of system performance Existing approaches – 2 country examples New developments

Increased emphasis Utility of compacts – what they contain

Common monitoring framework of health progress and performance

Data collection

Indicatordomains

Analysis & synthesis

Communication & use

Administrative sourcesFinancial tracking system; NHADatabases and records: HR, infrastructure, medicines etc.Policy data

Facility assessments Service readiness

Population-based surveysCoverage, health status, equity, risk protection, responsiveness

Clinical reporting systemsquality, coverage, health status

Civil registration

Data quality assessment; Synthesis of progress and performance; Evaluation

Regular country health sector review processes; Global reporting

Improved health outcomes

& equity

Social and financial risk protection

Responsiveness

Efficiency

Fina

ncin

gInfrastructure;

ICT

Health workforce

Supply chain

Information

Interventionaccess & services

readiness

Interventionquality, safety

Coverage of interventions

Prevalence risk behaviours &

factorsGov

erna

nce

Inputs & processes Outputs Outcomes Impact

International funding remains unpredictable

Donor commitments for health as % of total expenditure on health

0

10

20

30

40

50

60

1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008

Mali

Mauritania

Tanzania

Eritrea

Donor commitments for health as % of total expenditure on health

0

10

20

30

40

50

60

1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008

Benin

Burundi

Guinea

Liberia

Growing numbers of players in health: Chad

Nations Unies

UE

France

Suisse

Allemagne

Egypte

USAIDChine

Cuba

ONUSIDA

UNFPA

UNICEF

PAM

OMS

Membres de l'UE

Libye UNHCR

PNUD

OCHA

FAO

Initiatives mondiales

Fonds mondial contre le Sida, la tuberculose et le

paludisme

GAVI

ONG nationales et internationales

MSFs CCSIDED

Croix Rouge

COOPI

MDM

MENTHOR

BASE

UNAD EMEET

CSAI

ACF

ECHO

CICR

Ordre

de Malte

BM

BADInstit.

fin.

int.

issues

Focus on (quick) results does not need to compromise HSS / CB efforts, but needs to be managed

Results need to be expressed in different ways

Traditional approaches to HS and capacity strengthening mixed record

AE p's apply to this as

Links back to impact evaluations / public health research

One of the 9 standard JAR background documents is a research report, summarising major health related studies in Nepal in last year.

In 2011 research report covered findings from Major periodic surveys eg DHS, NLSS, service tracking survey Financing studies commissioned as part of strategy

development Studies on effectiveness of new policies eg reducing barriers

to maternal health services Studies on aid effectiveness and results

Improving background documents is an iterative process Some research topics identified in JAR eg on

procurement

Snapshot of Mali

Progress with putting AE principles into practice Coherent planning framework (PRODESS+, MTEF);

stronger MOH ownership; more alignment with strategies; less with systems; greater management for results; some greater donor harmonization, MA

Results 1: more effective aid Mixed: more predictable pledges; disbursement worse;

some transaction costs seen as investment (but many still negative); TA more focused on systemic priorities

Result 2: health system strengthening

27 October 2011 Mali Case Study 25

• Important progress in implementing aid effectiveness principles practice and behavior changes highly probable contribution to HSS and sector results

• BUT “two-speed implementation” of the principles of the Paris Declaration and IHP+: – Only 13/50 donors signed the Compact and 3 do SBS– Sharp decrease in executed ext. financing since

2008– Still many earmarked funding (thematic./geograph.)– Specific procedures and PIUs– Aid not predictable– Individual missions / audits in addition to joint ones

Expectations must be realistic

Mali: Conclusion and perspectives