Verification of Results

Findings and Recommendations

from a Cross-Case Analysis

Petra Vergeer, Tawab Hashemi, Martin Sabignoso, Olivier Basenya,

Catherine Mugeni, Eubert Vushoma, Chenjerai Sisimayi

• Relatively new function • Verification is the first-order substantiation of results paid for in RBF, including

coverage rates or quantities of patients seen, quality of services provided, and patient satisfaction.

• Counter-verification is the second order substantiation of the above, i.e., it requires that a first order verification has been carried out and is verifying the accuracy of it.

• Donors and government acutely sensitive to potential for

“over-payments” for inflated service reporting

• Avoid appearance of, or actual conflict of interest: contracted party has incentive to over report; separate actor must verify reporting

Verification: Essential element of

implementation

Not to be confused with M&E!!

• Process for ensuring the consistency of routine reporting on the volume (i.e., quantity) of purchased services provided (recount of data)

• Process for confirming with patients the provision of purchased services (patient tracing)

• Direct observation of conditions of service delivery and actual care to assess quality

• Assessment of satisfaction of patients

• RBF mechanisms often include multiple approaches

Different methods used

Objectives of Cross-Case Analysis • Expand knowledge about verification processes and

practices to address the design and implementation needs of RBF projects.

• Add to available knowledge by comparing the characteristics of verification strategies as well as available data on costs (using level of effort as a proxy), savings, and verification results to date in six countries: Afghanistan, Argentina, Burundi, Panama, Rwanda, and the UK.

• Country cases written with a common outline to describe major characteristics of the verification method, the verification results, the use of the verification results, costs, and key lessons and recommendations.

Overview of Verification Structures

and Interactive Quiz

Overview of Verification Structures

Afghanistan, Argentina, Burundi, Rwanda (community RBF)

Afghanistan: Verification structure There are four types of verification activities:

• Quantity of services verified in facilities, conducted

quarterly, ex-ante, by a third party, of 25% of providers;

• Patient tracing, conducted quarterly, ex-ante, by a third

party, of 25% of providers;

• Quality of services assessed by the Provincial Health Office

(PHO) jointly with the NGO

• Counter-verification of quality through health facility

assessment by a third party, sample basis.

There are three types of verification activities:

• Beneficiary enrollment verification, conducted monthly, ex-ante, internally at national level, with electronic data validation all records, no field visits;

• Beneficiary enrollment counter-verification, conducted every two months, ex-post, by a third party, with electronic data validation all records, and a sample is checked to ensure existence of enrollment form;

• Tracer indicator verification, conducted every four months, ex-post, third party, data validation all records, risk-based sample of health facilities (primarily facilities with higher numbers of patients).

Argentina: Verification structure

Burundi: Verification structure There are 4 types of verification activities in Burundi:

• Quantity verification, conducted monthly, ex-ante, jointly by verifiers from the MOH and civil society organizations (public –private partnership), at all sites;

• Technical quality verification, conducted quarterly, ex-ante, internally with civil society engagement, at all sites;

• Patient tracing (including patient satisfaction) as part of the quality score, conducted bi-annually, ex-ante, by a third party, each facility, sample basis; (n.b. for quarters with no patient tracing,

the previous quarter’s score is used in the calculation)

• Quantity, quality and patient tracing counter-verification, conducted quarterly, ex-post, by third party, sample basis.

Rwanda: Verification structure for

community RBF There are four types of verification activities:

• Verification by the health facility of the quantity of referrals

based on information in submitted reports, monthly, ex-

ante, internally, all indicators, all CHWs. The number of

referrals is cross-checked against health center records.

• Quarterly counter-verification, by a sector steering

committee (mostly comprised of health center staff - but

with some community members, considered independent).

Rwanda: Community RBF

verification structure (cont.) • All cooperatives’ reports are assessed each quarter for data

completeness and report submission timeliness, internal.

The evaluation of cooperative management is carried out by

the district hospital. Each quarter, 100% of cooperatives are

evaluated.

• Verification of the demand-side scheme is not systematic

and is integrated into the monitoring of the health center by

the district hospital.

Interactive Quiz

How long does it take to observe improvements in

RBF data due to quantity verification?

1. One (1) year

2. Two (2) years

3. Five (5) years

Findings: Level of agreement between HMIS and

facility data increases over time in Afghanistan

• Structure o Verification of the quantity of services in facilities is conducted quarterly, ex-ante, by a

third party. 25% of providers are sampled each quarter.

• Findings o Error rates in quantity verification declined from 17% to 8% between 2010 and 2013

% a

gre

em

en

t

83 83 83 83

86 87

89

93

89

95 94

91 92

76

78

80

82

84

86

88

90

92

94

96

Figure: Trends in Level of Agreement between HMIS and Facility-Level

Verification Data for the Quantity of Services Delivered

Findings: Error rates in beneficiary enrollment

decline over time in Argentina • Structure

Beneficiary enrollment:

o Verification through electronic data validation of all records (no field visits) monthly, ex-ante,

internally at the national level.

o Counter-verification, every two months, ex-post, third party, electronic data validation all

records, sample checked to ensure existence of enrollment form;

• Findings o Error rates in beneficiary enrollment declined from 20% to less than 1% in 2 years

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

2004 2005 2006 2007 2008 2009 2010 2011 2012

% R

ecord

s R

eje

cte

d

Phase 1 Provinces Phase 2 Provinces

Figure: Counter-verification of

beneficiary enrollment compared to

records submitted by provinces in

Argentina, 2004-2012

Findings: Counter-verification shows <10% errors

in Burundi after 2 years

• Structure: o Quantity verification, conducted monthly, ex-ante, jointly by verifiers from the MOH

and civil society organizations (public –private partnership), at all sites;

o Quantity counter-verification, conducted quarterly, ex-post, by a third party, sample

basis.

• Findings: o Internal verification found that 31% of declarations for health centers were reported

with error and 38% for hospitals. The average error size was 5% for health centers

and 4% for hospitals.

o Difference between verification and third party counter-verification of quantity after 2

years of nationwide PBF implementation were small (<1%) for health centers.

Differences for hospitals were substantially larger (9%) but can be explained in part

by a lack of standardized registers among hospitals.

Findings: Decline in size of indicator over-reporting

in Rwanda

• Structure

o Verification by the health facility of quantity of referrals based on

information in submitted reports is done on a monthly basis, ex-ante, by an

internal verifier. All indicators and all CHWs are verified. The number of

referrals is cross-checked against health center records.

• Findings

o The percentage of service indicators that contained error did not change

dramatically, with 49% of indicator reports inaccurate in Q4 2012 (23%

over-reported and 26% under-reported ). However, the size of the error for

over-reporting declined substantially (from over 140% to around 7%) in two

year’s time.

o This refers to the discrepancies between the performance as self-assessed

by CHWs at cell level and the performance after the verification process is

complete.

After 2 years of implementation, what was the average

percentage of traced patients who could not be found?

1. 20%-30% of all patients traced could not be found

2. 10% of patients traced from health centers and 15% of patients traced from hospitals could not be found

3. <10% of patients traced from health centers and 15% of patients traced from hospitals could not be found

Findings: Percentage of missing patients reduced

over time in Afghanistan

• Structure o Patient tracing is conducted quarterly, ex-ante, by a third party. 25% of providers are

sampled.

• Findings o “Missing patients” reduced from 33% to 7% between 2010 and 2013

% a

gre

em

en

t

67 77

83 86 89 89 91 92 94 95 96 93 93

0

20

40

60

80

100

120

Trends in Level of Agreement between HMIS and Community -Level Verification

Data for the Quantity of Services Delivered

Findings: Most patients traced from health centers

but more difficult in hospitals in Burundi • Findings

o In Q1 2012, 7.4% of patients traced from health

centres (see below graph) and 15.4% of

patients from hospitals were not found.

o More than 98% of those found (both for health

centers and hospitals) confirmed receiving the

services recorded.

o Counter-verification of patient tracing is

performed but with a newly-taken sample of

patients and hence no comparison can be done

between verification and counter-verification.

• Structure

o Patient tracing (including patient satisfaction),

conducted bi-annually, ex-ante, third party, each

facility, sample basis;

o Counter-verification of patient tracing counter-

verification, conducted quarterly, ex-post, third

party, sample basis.

Figure: Health centre patient tracing results 2011-2012

Findings: National study in Rwanda identifies most

patients in community

• Structure:

• A national study, which is to be distinguished from regular patient tracing as in

Burundi and Afghanistan, was conducted by the MoH in 2012 and included,

among other things, patient tracing in the community

• Findings:

• 97% of the patients could be identified in the community.

• Of those found, 97% confirmed having been treated at the facility for the

services for which the CHW referred them.

• In addition, 98% of eligible women confirmed to have received in-kind

incentives.

What are the links between patient confidentiality

and verification?

1. There are no concerns about patient confidentiality in

verification

2. Concerns about protecting patient confidentiality result in

the exclusion of certain indicators from verification (e.g.,

family planning)

3. Electronic checks of records helps to protect patient

confidentiality because patient data are de-identified

4. 2 and 3 are correct

Findings: Indicators excluded because of patient

confidentiality concerns in Burundi

• Findings – Burundi o Only 9 of 22 indicators at the health center and 8 out of 24 for hospitals are verified as

part of patient tracing for confidentiality reasons. Indicators on HIV, tuberculosis, and

family planning are excluded.

o As a result, the existence of “phantom patients” for almost half of the health centre

indicators, and one third of hospital indicators, is never assessed.

What type of indicators have high error rates?

1. Indicators with high patient volume

2. Indicators with complex definitions

3. Indicators with registration difficulties

4. Indicators with a high incentive attached

5. 1 and 2 are correct

6. 1, 2, and 3 are correct

Findings: Indicators with a high rate of

occurrence and with complex definitions

have higher error rates in Burundi

• Findings o Indicators with a high rate of

occurrence have the highest level of

error. These are indicators where

the risk of errors when counting may

be greater.

o Indicators with definitions that are

complicated also have higher errors.

Facilities have greater chances of

counting a service that does not

match the definition and that will

not be validated by the verification

team

o Indicators with high incentives do

not have high error rates

10 Indicators with highest error rates identified during

verification in health centers

(January-August 2012)

Indicator % accuracy of

declared data

n

Consultation (child) 22% 4,085

Consultation (adult) 30% 4,085

Observation day (child) 44% 3,802

Consultation (pregnant woman) 44% 4,062

Small surgery 53% 3,987

Family planning 58% 3,458

Completely vaccinated child 67% 4,065

Anti-tetanus vaccination (TT2-TT5) 68% 4,055

Observation day (adult) 69% 2,896

Prenatal consultation 67% 4,074

Findings: Indicators with a high rate of occurrence

have higher error rates in Rwanda • Findings

o Malnutrition monitoring had the highest error rate in Rwanda which may have been due to

the large number of children involved.

Table: Percentage of inaccurate reports detected by the health centers and by the sector steering committees

in the 4 sectors visited for 8 paid indicators, during Q4 2010, Q4 2011 and Q4 2012

Indicator

Errors detected by the health center (comparison between cell and sector reports)

Errors detected by the sector steering

committee (comparison between sector reports and national db)

% inaccurate indicators Nb. of reports % inaccurate

indicators Nb. of reports

Woman accompanied for delivery 51% 35 14% 35

Woman accompanied for antenatal care 43% 35 14% 35

Patients accompanied for VCT 49% 35 20% 35

Children monitored for nutrition status 59% 34 29% 34

Family planning users referred 23% 35 23% 35

TB-cases followed per month 23% 35 37% 35

TB suspects referred 37% 35 41% 34

Women referred for PMTCT 52% 33 36% 33

Total 42% 277 27% 276

Findings: Higher error rates are associated

with registration difficulties in Argentina • Structure

o Tracer indicator verification is

conducted every four months, ex-post

by a third party. Data validation is

conducted for all records, and a risk-

based sample of health facilities is

selected (primarily facilities with

higher numbers of patients).

• Finding o Higher error rates are associated with:

• Lack of registration tools (tracer

VII)

• Weak adherence to registration

norms (tracer IX)

• More than one source of data

needs to be utilized (tracer IV).

Tracer Percentage of results with error

rates >20% of declared

Percentage of results with error

rates >40% of declared

I Early detection of pregnant

w omen 23% 3%

II Effectiveness of childbirth and

neonatal care 6% 1%

III Effectiveness of prenatal care and

prevention of prematurity 7% 1%

IV Effectiveness of prenatal and

delivery care 19% 5%

V Case assessments in child and

maternal deaths out of all child

and maternal deaths 5% 3%

VI Immunization coverage 10% 1%

VII Sexual and reproductive care 18% 6%

VIII Tracking healthy child up to 1 year 11% 3%

IX Tracking healthy children betw een

1 and 6 years 24% 7%

X Inclusion of the indigenous

population 6% 2%

Table: Error rates >20% and >40% by indicator 2008-2012

identified counter verification

Why is there a large difference between

verification and counter-verification of quality?

1. Time delay between verification and counter-verification

2. Potential conflict of interest between those assessing

quality and those contracted to provide services

3. Objectivity of measuring tool is compromised

4. Sanctions for discrepancies between verification and

counter-verification are not applied

5. All of the above are correct

Findings: Afghanistan quality verification

• National Monitoring Checklist (NMC) is used for quality verification.

o Interviewees understand payment is linked to quality.

o However, they are not necessarily clear about which specific indicators are linked to quality

payment (e.g., clinic infrastructure, facility health information system HMIS data and

essential drugs from the NMC all are part of the indicators making up the payment for

quality.

• NGO and MoPH – SM supervisors fill NMC checklist as part of their routine supervision visit to

health facilities.

o However, PPHOs often do not join the health facility visits which can lead to conflict of

interest.

• BSC is used for verification at the provincial level hospitals.

o Due to delays in the implementation of the BSC, bonus payments to hospitals were also

delayed.

• BSC was also intended as a way of triangulating the NMC results at provincial level.

o Due to delays in the implementation of the BSC, this has not been operationalized.

Findings: Burundi quality verification

• Systematic difference between quality verification and counter-verification (79% of health centers and 84% of hospitals)

• Technical quality was overestimated (by 11% in health centres and by 17% in hospitals).

• Overestimation can be explained by three factors: o Time lag between verification and counter-verification

o Counter-verification team is more rigorous

o Possible conflict of interest as peer hospitals review other hospitals and provincial health teams verify their own health facilities. Sanctions were not applied for discrepancies found during counter-verification

Average

difference

% with

over-

estimation

Average

over-

estimation

% with

under-

estimation

Average

under-

estimation n

Health

centres -11% 79% -20% 21% 24% 101

Hospitals -17% 84% -24% 16% 20% 32

Table. Difference between technical quality assessment performed by the BPS, BDS or peers and counter-verification by HDP in all

hospitals and health centres counter-verified during the 8 counter-verification rounds, 2010-2012

Key Recommendations 1. Consider context to determine whether merging functions

is appropriate (be mindful of conflict of interest)

2. Analyze and use data available from verification and

counter-verification

3. Verification strategies should be dynamic, not static, and

use a risk-based approach

Factors influencing verification: a

conceptual framework Context

Verification Characteristics Impact on accuracy,

cost, sustainability

RBF Characteristics RATIONALE FOR RBF CONTRACT TYPE

USE OF RBF RESULTS

Improving health outcomes/HSS Relational

Payment, improving performance

Financial accountability/Cost control Classic

Transparency, Naming and Shaming

Monthly Annual

Yes

Large

Whole universe Risk-based approach Internal

Verification Results and Their Use

FREQUENCY

ALLOWABLE ERROR MARGIN

SAMPLE SIZE INSTITUTIONAL SETUP ADVANCE WARNING No

Small

Third party

Learning, Error correction Cost recovery, Sanction

PAYMENT FREQUENCY Monthly Annual

POLITICAL ENVIRONMENT GOVERNANCE CULTURE

When and how to change your verification

strategy…examine quantity verification error rates

1

Facility-level patterns

Indicator-level patterns

Generalized

across all

contracted

parties

Localized to specific

facilities (by

geographic area)

Localized to

specific types

of facilities

Indicators with

complex

compliance

criteria

Indicators that are

rewarded more

frequently and/or

have higher patient

volume

Indicators

rewarded at a

higher level

Examine quantity

verification error rates

Activities to

explore

Activities to

explore

Risk-based sampling for verification • Using a risk-based sampling approach likely more cost-

effective

• Sample contracted parties (e.g., facilities) with selection

criteria, such as: o Higher volume (like in Argentina)

o Outliers in performance relative to province- or national- averages (as in the UK)

• Sample indicators with selection criteria such as: o Higher volume (possibly more prone to error as in Burundi and Rwanda example)

o More complex (possibly more prone to error as in Argentina example)

o Higher $ value

• Always ensure a credible threat of verification remains for

all contracted parties

-15

-10

-5

0

5

10

15

HF1 HF2 HF3 HF4 HF5 HF6

Difference Between Declared and Verified 6 Month Totals

Within 5% Difference

Risk-based verification: Zimbabwe model

Green Category:

• Verified on a quarterly

basis

Amber Category

• Verified bi-monthly -

randomly selected 2

months

Red Category

• Verified on a monthly basis

• Also incorporates new

facilities

Difference above 5%

but below or equal to

10% Difference above 10%

• Model based on three risk levels

• Comparison between declared

and verified values for 6-month

totals

Example at District level

Health

Facility

Total

Declared

Total

Verified

%

Difference

Hoyuyu

1 1011 1016 0%

Matedza 344 325 6%

Kawazva 327 417 -28%

Sample Facilities (Mutoko District)

• Districts have a mix of facilities at different risk levels

Total facilities by risk-category

85

63

244

0

50

100

150

200

250

300

Red Amber Green

Num

ber

of F

acili

ties

Category

Areas for further research in verification

• Costs, savings, and cost-effectiveness of

verification and counter-verification

• Application of technology for verification and

potential for cost savings

• Ensuring patient confidentiality is protected

• Patient tracing

• Measuring quality

Conclusions • The conceptual framework and is intended to assist RBF

implementers and policymakers in their deliberations about

the consequences of various verification characteristics on

the accuracy, cost, and sustainability of a chosen approach.

• Verification strategies are not static but should be a

dynamic process. The pathway tool can provide guidance

for how to adapt verification strategies.

• While there is no optimal verification method appropriate to

all settings, the recommendations provided can be useful to

consider in different contexts.

References • Cashin, Cheryl and Lisa Fleisher. Verification of performance in results-based financing:

the case of Afghanistan. World Bank: Washington, DC. Forthcoming.

• Cashin, Cheryl and Petra Vergeer. (2013). Verification in results-based financing: the case

of the United Kingdom. World Bank: Washington, DC.

https://openknowledge.worldbank.org/handle/10986/13567

• Perazzo, Alfredo. Verification of performance in results-based financing: the case of

Panama. World Bank: Washington, DC. Forthcoming.

• Perazzo, Alfredo and Erik Josephson. Verification of performance in results-based

financing: the case of Argentina. World Bank: Washington, DC. Forthcoming.

• Renaud, Adrien. Verification of performance in results- based financing: the case of

Burundi. World Bank: Washington, DC. Forthcoming.

• Renaud, Adrien and Jean-Paul Semasaka. Verification of performance in results-based

financing: the case of the Rwanda community RBF interventions: community PBF and

demand side scheme. World Bank: Washington, DC. Forthcoming.

• Vergeer, Petra, Anna Heard, Erik Josephson, and Lisa Fleisher. Verification in results-

based financing for health: findings and recommendations from a cross-case analysis.

World Bank: Washington, DC. Forthcoming.

Thank You