1

AN IMPACT EVALUATION OF PERFORMANCE BASED CONTRACTING FOR GENERAL HEALTH SERVICES IN RWANDA

Evaluation team:Paulin BasingaPaul GertlerJennifer Sturdy Christel VermeerschDamien de Walque

Project teamAgnes Soucat

Miriam Schneidman+ team

2

04/20/23

Presentation plan

Country and program information

Impact evaluation design

Descriptive results from the baseline study

3

SECTION 1: COUNTRY AND PROGRAM INFORMATION

4

04/20/23

RWANDA general & health sector

Total population : 9,038,001 (2005) 30 Administrative districts GDP per capita of $230 33 District Hosp. and 369 Health centers HDI: ranked 158th (2004) MMR: 750 per 100,000 (DHS 2005) IMR: 86 per 1,000 (DHS 2005) HIV: 3.1% (DHS 2005)

5

04/20/23

Relevance and Severity of the Health Issue Addressed Diagnostic

Shortage of human resources for health services Low levels of productivity and motivation High levels of absenteeism Low user satisfaction & poor quality of service lead

to low use. Increase morbidity and mortality

Goal Increase number of trained medical personnel Increase motivation Improve quality of services Increase personnel income

Policy Response Performance Based Contracting & Financing

6

National PBC model for Health Centers

16 Primary Health Care indicators, eg: New Curative Consultation = $0.27 Delivery at the HC = $3.63 Completely vaccinated child = $ 1.82

14 HIV/AIDS indicators, eg: One Pregnant woman tested (PMTCT) =

$1.10 HIV+ women treated with NVP = $1.10

Separation of Functions between stakeholders

7

04/20/23

Fig: Implementing organizations

Ministry of Health (CAAC) Ministry of Finance

Stakeholders Administrative districts Steering committee Purchasers District hospital

Purchase contract Provider: Health facilities

Head of the Health center ---- Management Committee Motivation contract Employee

8

SECTION 2 : IMPACT EVALUATION DESIGN

10

04/20/23

Evaluation Design

Make use of expansion of PBC schemes over time The rollout takes place at the District level Treatment and control facilities were allocated as

follow: Identify districts without PBC in health centers

in 2005 Group the districts based on characteristics:

rainfall population density livelihoods

Flip a coin to assign districts to treatment and control groups.

11

12

04/20/23

Roll-out plan

Phase 0 districts (white) are those districts in which PBF was piloted NOT part of the impact evaluation

Phase 1 districts (yellow) are districts in which PBF is being implemented in 2006, following the ‘roll-out plan’

Phase 2 districts (green) are districts in which PBF is phased in later; these are the so-called ‘Phase 2’ or ‘control districts’ following the roll-out plan. PBF is being introduced in these districts in 2008.

13

04/20/23

Program Implementation Timeline

Jan-06 Mar 06 Jun-06 – Sept 06 2007 Feb-08-Apr 08 Apr-08

Phase I Start of intervention

Program Implementation

Phase II

Start of intervention

Impact Evaluation

SURVEYS Baseline

(General Health) Baseline

(HIV/AIDS)

Follow-up

14

Quality assurance in comparisons

Law of large numbers does not apply here… Proposed solution:

Propensity scores matching of communities in treatment and comparison based on observable characteristics

Over-sample “similar” communities in Phase I & Phase II

It turned out Couldn’t find enough characteristics to predict

assignment to Phase I Took a leap of faith and did simple stratified

sampling

15

More money vs. More incentives

Incentive based payments increase the total amount of money available for health center, which can also affect services

Phase II area receive equivalent amounts of transfers average of what Phase I receives Not linked to production of services Money to be allocated by the health center Preliminary finding: most of it goes to

salaries

16

Surveys

General Health facility survey (168 centers)

General Health household survey

HIV/AIDS facility survey (64 centers) HIV/AIDS household survey

Note: HIV/AIDS study is for another presentation

17

Design and sample

04/20/23

18

04/20/23

Socio-economic information Anemia finger prick test: children 12-71

months old Malaria dip stick test: children under age 6 Anthropometrics: <6 years old Mental health: mothers, pregnant women,

adults over age 20 Sexual history and preventative behavior

knowledge Pre-natal care utilization and results Parents or caretakers were asked for

information regarding child (<5 years) health status

General Health Household Survey: content

19

04/20/23

Health Centers Survey

All 168 centers in General Health; 64 in HIV/AIDS

General characteristics Human resources module: Skills, experience

and motivations of the staff Services and pricing Equipment and resources Vignettes: Pre-natal care, child care, adult

care VCT, PMTCT, AIDS detection services Exit interviews: Pre-natal care, child care,

adult care, VCT, PMTCT

20

04/20/23

Analysis Plan

All analyses will be clustered at the district level Compare the average outcomes of facilities and

individuals in the treatment group to those in the control group 24 months after the intervention began.

Use of multivariate regression (or non-parametric matching) : control confounding factors

Test for differential individual impacts by: Gender, poverty level Parental background (If infant : maternal

education, HH wealth)

21

SECTION 3: DESCRIPTIVE RESULTS FROM BASELINE SURVEY

22

HEALTH FACILITIES BASELINE SURVEY

23

Classification of facilities

04/20/23

24

Financial Capital Resources

Main Sources of Funding at Facility Level Average Annual Funding at Facility Level

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

User Fees Drug Sales Religious Other Donor GovernmentOther

Major Source

Second Source

Third Source

RWF

2,416,597.00

RWF 727,169.90

RWF

2,069,715.00

RWF 297.47

RWF 295,745.90

Health

District

Religious Consultation

Fees

Drug

FeesLab Fees

Main Sources of Funding

04/20/23

25

Human resources

On average:

• 1 doctor for every 31,190 individuals,

• 1 nurse for every 4,835 individuals

Staff Obsv Mean Std dev. Min Max

Nurses A1 165 0.24 0.66 0 5

Nurses A2 164 5.32 2.74 0 18

Social Ass A2 165 0.16 0.93 0 11

Lab tech 166 0.80 1.02 0 10

Total staff 165 15.05 7.28 3 43

04/20/23

Availability of laboratory tests

04/20/23

Availability of equipment

04/20/23

Availability of drugs

04/20/23

29

Prenatal Care procedures:Comparison between health provider & patient responses

92%

86%

80%

21%

18%

13%

64%

85%

85%

21%

17%

8%

Number of previous miscarriages

Number of previous pregnancies

Late menstrual dates

Lost/gain of weight, nausea, vomiting

Medicaments that is taking right now

Family history of genetic problems

HealthProvider Patient

04/20/23

30

Prenatal care Physical ExaminationComparison between health provider and patient responses

52%

58%

72%

81%

81%

86%

83%

19%

17%

97%

61%

93%

Weight measured

Pelvic examination

Height measured

Abdomen examination

Blood presure measured

Listen featal heatbeat

Health Provider Patient

04/20/23

31

Quality index: Descriptive statistics

INDEX Mean SE Min Max

Process quality

Prenata care_Vignette 0.48 0.002 0.17 0.83

Child care_Vignette 0.44 0.002 0.20 0.71

Exit interv satisfaction 0.80 0.004 0.00 1.00

Structural quality

HF Infrastructure 0.76 0.004 0.17 1.00

HF Equipment 0.74 0.003 0.32 1.00

HF: Laboratory 0.52 0.005 0.00 1.0004/20/23 31

32

PATIENT SATISFACTION

Negative significant correlation between satisfaction and:

• The length of wait time • Total time spent attending the facility, • The cost of the visit • The cost of the medications.

No difference between satisfaction of mutuelle members and non-members

• But mutuelle membership is significantly positively correlated with cost

Prenatal visit: positive correlation with the cleanliness of the facility

Child care: lower levels of satisfaction in

• Waiting time, • Total time spent for visit, • Cleanliness of the facility, • Privacy during the exam, • Attitude of the staff, • Explanation of the child’s condition

during the exam

04/20/23

33

HOUSEHOLD BASELINE SURVEY

34

Sample description

2159 HH, 10,880 individuals Gender balance (49,3% male and 50,7

female) Average HH size is 5.71 individuals, Age range of <12 months to 96 years

old. Half of the sample is below 12 years old 75% of the sample is under the age of 30

years old. (Sampling strategy)04/20/23

35

HH Education by phase

Phase I (Intervention district)

Phase II (Control districts)

Variable obs % std obs % std. tstat

Ever_attended_school 3258 79.6 0.402 4172 80.6 0.395 1.048

No_schooling 2588 10.5 0.307 3356 11.5 0.319 1.208

Primary 2588 82.1 0.383 3356 79.8 0.401 -2.227

Secondary_above 2588 6.9 0.255 3356 7.9 0.270 1.309

04/20/23

36

Household assets Phase I (Intervention district) Phase II (Controldistricts) Variable Obs Std Obs std tstat Number of rooms in dwelling 960 3.59 1.09 1173 3.39 1.16 -4.12 Complete sofa set 960 1.88% 0.14 1173 2.90% 0.17 1.53 Radio 960 49.90% 0.50 1173 50.04% 0.50 0.07 Radio-cassette or music system 960 3.65% 0.19 1173 5.80% 0.23 2.31 Telephone – 960 1.04% 0.10 1173 3.32% 0.18 3.51 Mosquito nets 960 22.50% 0.42 1173 28.73% 0.45 3.27 Sewing machine 960 0.83% 0.09 1173 1.19% 0.11 0.82 A bed 960 63.23% 0.48 1173 58.14% 0.49 -2.39 Wardrobe 960 4.38% 0.20 1173 5.54% 0.23 1.23 Metallic library 960 0.63% 0.08 1173 1.36% 0.12 1.68 Table 960 63.02% 0.48 1173 62.49% 0.48 -0.25 Chair 960 85.31% 0.35 1173 83.89% 0.37 -0.91 Car 960 0.63% 0.08 1173 0.43% 0.07 -0.64 Motorcycle 960 0.42% 0.06 1173 0.51% 0.07 0.32 A bicycle 960 17.92% 0.38 1173 16.88% 0.37 -0.63 Dwelling - High Quality Materials 960 0.21% 0.05 1173 0.17% 0.04 -0.20 Dwelling – Low Quality Materials 960 44.17% 0.50 1173 38.45% 0.49 -2.67 Uses Electrogaz for Cooking (No) 960 99.38% 0.08 1173 99.57% 0.07 0.64 Main source of lighting is electricity 956 0.84% 0.09 1170 1.37% 0.12 1.15 Well water 954 71.91% 0.45 1172 62.97% 0.48 -4.38 Other source of water 954 19.81% 0.40 1172 25.26% 0.43 2.98 Open source of water 954 8.07% 0.27 1172 11.35% 0.32 2.52 Owns flush toilet 960 0.10% 0.03 1173 0.26% 0.05 0.80 Owns latrine 960 80.63% 0.40 1173 76.47% 0.42 -2.32

36

37

Maternal health service utilization

Age PNC provide with the last child

Preventive injection for Tetanus

Assisted delivery

Delivery at the health facility

<20 28 (98.86%) 27 (77.78%) 13 (43.33) 15 (50%)

20-34 1154 (96.7%) 1117 (84.15%) 595 (29.04) 664 (31.91)

35+ 515 (94.95%) 489 (46.01%) 153 (20.45) 162 (21.34)

Overall 1,681 (95.84)

1,227 (72.86)

837 (26.01) 929 ( 28.47)

04/20/23

38

Maternal health service utilization

Phase I (Intervention district) Phase II (Control districts)

obs Perc Std e obs Perc Std e tstat

Assisted delivery 1492 24.66 0.01 1726 27.17 0.01 1.61

Delivery at the health facility 1508 27.4 0.01 1755 29.3 0.01 1.19Times received PNC 752 2.781 1.138 927 2.709 1.159 -1.274

Preventative injection for Tetanus 757 73.45% 0.442 927 72.38% 0.447 -0.488

04/20/23

39

Immunization

Phase I (Intervention district)

Phase II (Control districts)

Obs Perc Std Obs Perc Std tstat

Fully immunized (12-23 months old) 272 73.53% 0.442 302 76.49% 0.425 -0.818

Fully immunized (12-71 months old) 1225 61.14% 0.488 1459 64.08% 0.480 1.570

04/20/23

40

Conclusion

HH Results comparables to the recent DHS

Validity of treatment – control groups Of 110 key characteristics and output variables of

HF, the sample is balanced on 104 of the indicators.

Of 80 key HH output variables, the sample is

balanced on 73 of the variables.

Majority of the indicators which differ between

Phase I and Phase II are results from patient exit

interview 04/20/23