Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Midline Evaluation of Hospitals Selected from a Quality-Based Pay for Performance
Scheme in Liberia
Luke Bawo
Kenneth L Leonard
Rianna Mohammed
Manasi Sharma
June 2016
2
Table of Contents
I. Health Systems Strengthening Project in Liberia ................................................................................. 7
A. Background ....................................................................................................................................... 7
B. The Project Development Objective ............................................................................................... 10
C. Theory of Change ........................................................................................................................... 12
II. Data Collection Effort ......................................................................................................................... 13
A. Instruments filled ............................................................................................................................ 17
B. Health workers observed ................................................................................................................. 17
III. Instrument summary ....................................................................................................................... 22
A. Motivation surveys .......................................................................................................................... 22
B. Competence/Capacity Vignettes ..................................................................................................... 29
1. Facility and health worker type averages for vignettes ............................................................... 29
2. Validating the theory of vignettes ............................................................................................... 34
C. Direct observation ........................................................................................................................... 37
D. Exit surveys ..................................................................................................................................... 41
IV. Evaluating the theory of change within HSSP facilities ................................................................. 50
A. The Three Gap Framework ............................................................................................................. 50
1. The Three Gaps in our Sample ................................................................................................... 53
2. Analyzing the three gaps by facility ........................................................................................... 54
B. Changes between 2013 Baseline Data Collection and 2015 Baseline Data Collection .................. 58
V. Conclusion .......................................................................................................................................... 65
VI. Appendices ...................................................................................................................................... 67
A. Acronyms ........................................................................................................................................ 67
B. Data Collection Schedule ................................................................................................................ 68
3
C. Quality Assessment Tool ................................................................................................................ 69
A. Instrument Summaries .................................................................................................................... 71
1. Surgical Vignettes ....................................................................................................................... 71
2. Pediatric Vignettes ...................................................................................................................... 80
3. Obstetric Vignettes ...................................................................................................................... 85
List of Tables
Table 1 Summary of Instruments by Service and Location ........................................................................ 16
Table 2 Number of Instruments Collected by Facility ................................................................................ 19
Table 3 Credentials of Health Workers on Hospital Rosters ...................................................................... 21
Table 4 Motivations Factors and Factor Loadings of Items ....................................................................... 23
Table 5 Motivations Factors across Facilities and wards............................................................................ 26
Table 6 Factor Means by Facility ............................................................................................................... 28
Table 7 Average Vignette Scores by Inputs, Diagnosis and Treatment...................................................... 30
Table 8 Regression coefficients for vignette scores by facility for assessment, diagnosis, and treatment
factors .......................................................................................................................................................... 32
Table 9 Regression Coefficients for Vignettes by Health Worker Type for Assessment, Diagnosis, and
Treatment Factors ....................................................................................................................................... 33
Table 10 Relationship between Inputs, Diagnosis, Detailed Diagnosis, and Treatment ............................ 35
Table 11 Direct Observation of Obstetric Care........................................................................................... 38
Table 12 Direct Observation of Pediatric Care ........................................................................................... 39
Table 13 Direct Observation of Surgical Care ............................................................................................ 40
Table 14 Patient Characteristics .................................................................................................................. 42
Table 15 Satisfaction As Expressed Across All Three Types of Exit Surveys ........................................... 44
4
Table 16 Factor Loading for Patient Exit Interviews .................................................................................. 46
Table 17 Satisfaction Summary Scores ...................................................................................................... 47
Table 18 Satisfaction Summary Scores by Facility and Ward .................................................................... 48
Table 19 Change in Average General Hospital Infrastructure Observation Scores, by Facility ................. 59
Table 20 Change in Average Motivation Factor Scores, by Facility .......................................................... 60
Table 21 Change in Key Indicators for Obstetrics Ward Observation ....................................................... 61
Table 22 Change in Key Indicators for Pediatrics Ward Observation ........................................................ 62
Table 23 Change in Key Indicators for Pediatrics Ward Observation, by Facility ..................................... 62
Table 25 Change in Key Indicators for Surgery Ward Observation ........................................................... 63
Table 26 Change in Vignette Scores, by Facility ........................................................................................ 63
Table 27 Change in Patient Characteristics for All Patient Exit Interviews, by Facility ............................ 64
Table 28 Change in Satisfaction Summary Scores for All Patient Exit Interviews, by Facility ................. 65
Table 29 Data Collection Schedule ............................................................................................................. 68
Table 30 Management and Structural Checklist Overview ........................................................................ 69
Table 31: Quality Checklist Overview ........................................................................................................ 69
Table 32: Quantity of Services.................................................................................................................... 70
Table 32: Surgery Vignette Scores by Facility (S_V_1_A) ........................................................................ 71
Table 2: Surgery Vignette Scores by Facility (S_V_1_B) .......................................................................... 72
Table 3: Surgery Vignette Scores by Facility (S_V_2_A).......................................................................... 74
Table 4: Surgery Vignette Scores by Facility (S_V_2_B) .......................................................................... 75
Table 5: Surgery Vignette Scores by Facility (S_V_3) .............................................................................. 77
Table 6: Surgery Vignette Scores by Facility (S_V_5_A).......................................................................... 78
Table 7: Surgery Vignette Scores by Facility (S_V_5_B) .......................................................................... 79
Table 8: Pediatric Vignette Scores by Facility (P_V_1) ............................................................................. 80
Table 9: Pediatric Vignette Scores by Facility (P_V_2) ............................................................................. 81
Table 10: Pediatric Vignette Scores by Facility (P_V_3) ........................................................................... 82
5
Table 11: Pediatric Vignette Scores by Facility (P_V_4) ........................................................................... 84
Table 12: Obstetrics Vignette Scores by Facility (O_V_1_A) ................................................................... 85
Table 13: Obstetrics Vignette Scores by Facility (O_V_1_B) .................................................................... 86
Table 14: Obstetrics Vignette Scores by Facility (O_V_2_A) ................................................................... 88
Table 15: Obstetrics Vignette Scores by Facility (O_V_2_B) .................................................................... 89
Table 16: Obstetrics Vignette Scores by Facility (O_V_3) ........................................................................ 90
Table 17: Obstetrics Vignette Scores by Facility (O_V_4_A) ................................................................... 91
Table 18: Obstetrics Vignette Scores by Facility (O_V_4_B) .................................................................... 92
List of Figures
Figure 1 Theoretical Framework: Theory of Change ................................................................................. 13
Figure 2 Probability of Being Correct Across All Vignettes by Facility .................................................... 32
Figure 3 Probability of Being Correct by Health Worker Type .................................................................. 33
Figure 4 Proportion of Possible Inputs into Correct Diagnosis and Treatment by Health Worker Type ... 34
Figure 5 Diagnoses are closely related to the use of key inputs ................................................................. 35
Figure 6 Correct treatment is related to the correct use of Physical Examination ...................................... 36
Figure 7 The Three-Gap Framework .......................................................................................................... 52
Figure 8 The Three Gap Framework for our sample (pediatrics) ............................................................... 54
Figure 9 Four Gaps across facilities ............................................................................................................ 55
Figure 10 Motivation factors and the four gaps with facility fixed effects ................................................. 57
Figure 11 Motivation factors and the four gaps with facility fixed effects ................................................. 58
6
Abstract:
Background: Improving the quality of care at hospitals is a key next step in rebuilding Liberia’s
health system. In order to improve the efficiency, effectiveness, and quality of care at the secondary
hospital level, the country is developing a system to upgrade health worker skills and competencies,
and shifting towards improved provider-accountability for results through a twin-pronged
approach involving the development of a Graduate Medical Residency Program (GMRP) and the
introduction of performance-based financing (PBF) at the hospital level.
Methods/Design: This document examines the assessment of quality in pediatric, obstetrics,
emergency and surgical wards in five hospitals enrolled in the Health Systems Strengthening
Project (HSSP) and five control hospitals. We examine first the overall level of quality in these
facilities as measured by our teams in intensive four to five days visits that examine quality using
competence and capacity vignettes as well as direct observation of patient care. Second, we
examine the data for evidence of patterns linking competence, capacity and performance as
outlined in the original proposal: the three gap model. As part of this, we also look at the gaps in
each facility and the differences in these gaps across facilities. Finally, four of the facilities
examine in this measurement exercise have been enrolled in the program now for almost two years
and although the implementation of the program was slowed by the Ebola crises, we examine the
data for evidence of improvements in the quality of care provided in these four facilities.
Findings: The quality of care remains relatively low in our sample, with important differences
across facilities. In examining the three gap model, we find that there is a very small competence-
capacity gap but a large can-do gap. Although we find that competence is low, suggesting
possibilities for more training, we find the link between competence (or capacity) and performance
is so weak as to call into question the value of improvements in competence. It is far more useful
to focus on the reasons why health workers do not use their knowledge and capacity in practice.
Fortunately, addressing this weak link was precisely the intention behind the HSSP. Finally, we
look at the changes across the four facilities that have been in the program for almost two years
and find no evidence of improvements in quality. There are some cases in which things have
improved but they fit not overall pattern that would suggest that they were caused by the HSSP.
Rather, many of the changes seen (for example in surgical preparation) are likely to have been
caused by the response to the Ebola crises.
7
I. Health Systems Strengthening Project in
Liberia
A. Background
The Liberian civil wars (1989-1996 and 1999-2003) completely destroyed the country’s
infrastructure and health care system. This included many hospitals and clinics being burned to
the ground; healthcare workers being killed or fleeing the country; equipment and supplies being
stolen or destroyed; a dramatic reduction in the number of functional laboratories; poor quality of
sanitation and basic supplies like water and electricity; and major disruptions in transport and
communication mechanisms.
Consequently, with the presence of the United Nations Mission in Liberia (UNMIL) and several
national and international organizations and countries providing humanitarian aid, the post-
conflict period saw improvements in efforts to rebuild the healthcare system. Despite impressive
gains in overall health systems management and in health services delivery since the end of the
war, Liberia continues to face significant challenges in improving maternal and child health
outcomes, as well as other health-related Millennium Development Goal (MDG) outcomes.
The country’s progress was severely disrupted when its healthcare system suffered a devastating
blow caused by the Ebola Virus Disease (EVD) outbreak between 2014 and 2015. At least 4,800
persons are reported to have died directly due to Ebola in Liberia, with thousands more dead due
to indirect causes such as overburdened healthcare facilities; decreased funding and attention on
other serious conditions like malaria, cholera, and Lassa fever; reduced supplies and infrastructure
in hospitals and clinics; and reduction in health care staff. Liberia was said to be the worst hit by
Ebola among the three affected West African countries (Sierra Leone and Guinea being the other
two), with 300-400 new cases being reported weekly during the peak of the outbreak in August-
September, 2014. On 13 January, 2016, the World Health Organization (WHO) declared Liberia
Ebola-free for the third time, due to the emergence of one or two cases in between. This was after
42 days of no new reported cases, as per WHO guidelines.
8
In terms of the country’s health indicators, although the maternal mortality ratio (MMR) declined
from close to 1000 per 100,000 births in 2007, to an estimated 770 per 100,000 in 2010, current
statistics indicate an increase to 994 per 100,000, possibly due to the Ebola outbreak. Only 46
percent of births are attended by skilled health workers. Approximately 18 percent of children
under five are underweight, and the under-five mortality rate is 114 deaths per 1,000 live births.
Only 59.7 percent one year olds are vaccinated against Measles. Malaria continues to be the
leading cause of mortality for children under age 5 (22 percent in 2013). For all age groups, Lower
Respiratory Infections is the leading cause of death (12 percent), followed by Malaria (8 percent),
Tuberculosis and HIV (6 percent).
Gains, however, remain skewed in favor of urban populations. For example, 63 percent of
deliveries in urban areas are facility-based, compared with 25 percent in rural areas; similarly, 77
percent of urban deliveries are by a skilled service provider compared with only 32 percent of rural
deliveries. While over one in ten children will die before the age of five, infant and under five
mortality rates have almost halved to 71 and 110 per 1,000 births respectively over the last 20
years due to improved access resulting from the Government of Liberia’s (GoL) free health care
policy, and restoration of a number of key child health services like immunizations. Malaria,
however, continues to be a major source of morbidity and mortality; 38 percent of outpatient
attendance and 42 percent of inpatient deaths was attributable to malaria in 2007.
Hospitals in Liberia remain in generally poor physical condition; are understaffed in key areas of
competence; and, have long waiting times and a lack of equipment and drugs. As a consequence,
hospitals in general provide low quality of care. This is reflected in high levels of post-surgery
complications and infection rates; very limited maternal and child death audits; and no systematic
use of clinical guidelines and protocols. Notably, accreditation scores on the quality of services
are worse in secondary vis a vis primary facilities. Poor quality is a particularly critical concern at
the severely resource-constrained hospital-level in Liberia because it can obviate all the implied
benefits of good access and effective treatment, and lead to sub-optimal and wasteful use of
resources.
9
In addition to restoring essential health services disrupted during the EVD crisis, improving the
quality of care at hospitals is a key step in strengthening Liberia’s health system, and ensuring
resilience to future health shocks. Hospitals in Liberia remain in generally poor physical condition;
are understaffed in key areas of competence with insufficient numbers of productive, responsive,
and qualified staff; and, have long waiting times and a lack of equipment and drugs. As a
consequence, hospitals in general provide low quality of care. This is reflected in high levels of
post-surgery complications and infection rates, low quality data on clinical outcomes, very limited
maternal and child death audits, and no systematic use of clinical guidelines and protocols. Poor
quality is a particularly critical concern at the severely resource-constrained hospital-level in
Liberia because it can obviate the implied benefits of good access and effective treatment; frustrate
the positive achievements at the primary health care system by not being able to respond to referral
patients with complications; and lead to sub-optimal and wasteful use of resources.
In order to improve the efficiency, effectiveness, and quality of care at the secondary hospital level,
the country is developing a system to upgrade health worker skills and competencies, and shifting
towards improved provider-accountability for results.
The Post Graduate Medical Council (PGMC) is tasked to develop the Graduate Medical
Residency Programme (GMRP) to facilitate in-country specialization of core MDG-related
hospital-level competencies. Residents will be selected from the existing pool of medical school
graduates based on standardized criteria. The GMRP requires both a critical stream of specialist
faculty to support the program, as well as the upgrading of essential equipment and supplies at
target teaching facilities.
In addition to the development of an MDG-related GMRP, the GoL is also moving towards
provider-accountability for improvements in quality through performance-based financing (PBF)
at target hospitals. The PBF approach is an innovative, results-oriented mechanism to incentivize
providers to achieve agreed-upon, measurable performance targets, with incentives typically
including financial payments, bonuses, other material goods, or public recognition. The shift
towards PBF is influenced by experiences and evidence in a number of high-, middle-, and low-
income countries which show that performance-based approaches can be effectively deployed to:
(i) clearly signal health priorities and ensure that there is adequate focus on corresponding
10
interventions; (ii) ensure that health facilities focus on delivering targeted and cost-effective health
services; (iii) strengthen monitoring and evaluation systems; (iv) empower decision-makers in the
field to set priorities and improve health facilities according to more local needs; (v) motivate staff
to change behavior and improve performance; and, (vi) increase provider autonomy and enhancing
accountability for better results. The latter can stimulate innovations in an effort to overcome
implementation constraints.
A rigorous, randomized controlled evaluation in Rwanda showed a positive impact of PBF at the
primary care level on utilization for institutional deliveries, growth monitoring consultations, and
increased levels of quality of care (Basinga et al., 2011). Rigorous evaluation of PBF programs is
essential for generating solid evidence that can inform the GoL and more broadly other
governments and partners to effectively design and use PBF mechanisms. At this time, solid
evidence on the impact of PBF in a hospital setting is scarce.
A. The Project Development Objective
The project development objective of the Liberia Health Systems Strengthening Project (HSSP) is
to “improve the quality of maternal health, child health, and infectious disease services in selected
secondary-level health facilities.” The project aims to strengthen the institutional capacity needed
to improve Maternal and Child Health (MCH), and infectious disease-related health outcomes at
target facilities through an innovative approach involving systematic and coordinated
improvements to the quality of services delivered at target facilities (through performance-based
financial incentives), and an expansion of health worker skills (through the provision of specialized
training, and the decentralization of specialist availability).
Specifically, the project aims to: (a) improve the quality of care standards (in both diagnosis and
treatment) for services with proven effectiveness; (b) increase the availability of qualified graduate
physicians (pediatricians, obstetricians, general surgeons, and internal medicine specialists, with
cross-cutting focus on anesthesiology); (c) enhance the clinical capabilities and competencies of
mid-level cadres - nurses, midwives, and physician assistants- in emergency obstetrics, surgery,
11
pediatrics, and internal medicine; and, (d) improve provider-accountability mechanisms related to
both the achievement of results, and health-worker performance at selected facilities. These
improvements should provide a thrust towards improved outcomes.
The HSSP is focused on improving: (a) the quality of care (in terms of both diagnosis and
treatment) for key services of proven effectiveness (e.g. maternity, pediatric/ neonatal, surgery,
management hygiene and patient satisfaction, and health worker performance); (b) structural and
management quality (related to, for example, availability of drugs and commodities, and health
facility rehabilitation); and (c) utilization across a limited number of clearly defined services. The
Quality Assessment Tool is outlined in Appendix C (Table 29, Table 30, Table 31). The project
incentivizes an increase in the quantity of services delivered for a only a limited number of services
because, as hospitals, the goal is not to draw patients away from more appropriate locations (e.g.
the primary health care level), but to provide the best care for those patients who need hospital
care. Health facilities will receive 75 percent of incentive payments based on quality
improvements, and twenty-five percent based on improved utilization of incentivized services. The
level of incentives will be adjusted to take into account equity considerations; for example, the
remoteness of a health facility.
Although the PBF aspect of the program focuses on specific services, the levers of the HSSP are
expected to have a much broader impact beyond these targeted services. In particular, the Project
aims to enhance the clinical capabilities and competencies of mid-level cadres - nurses, midwives,
and physician assistants- in emergency obstetrics, surgery, pediatrics, and internal medicine
through the provision of in-service training and direct coaching support; upgrade the equipment
and infrastructure levels at target facilities through both direct investments, and PBF incentives;
increases the motivation of health workers through both improved opportunities for training, and
individual performance-related bonuses; and improve provider-accountability mechanisms related
to the achievement of results, health-worker performance at selected facilities, and improved
access to information. These changes are expected to lead to improvements in the quality of
services and in addition, to increases in utilization for key services.
12
In examining the expected impact of the HSSP, the following set of research questions (RQs) will
be addressed:
RQ1: Did the program work, that is, did the HSSP achieve its goal of improving the quality of
service delivery in target hospitals?
RQ2: How did facilities improve their performance and meet the objective of improved quality?
Did levels of motivation, capacity and competence change as a result of the HSSP?
Which of the intervention levers (management, information, structural improvements, incentives
or training) made an important contribution to the success of the HSS project?
RQ3: What role did existing human resources in the health sector play in the success/failure of the
program? Did existing levels of skills and experience or motivation hinder or facilitate the
interventions?
B. Theory of Change
Figure 1 diagrammatically demonstrates the theory of change linking changes in skills, equipment,
and motivation brought about by the levers of the HSSP. The skills and experience of a health
worker determines their competence. Competence can be increased by training. The equipment
and infrastructure available to a health worker, combined with their competence, determines their
capacity. Capacity can be improved by changes in competence or by structural and supply
improvements to the facility. Finally, health worker effort, combined with capacity, determines the
performance of health workers. Effort is driven by motivation and can be improved through better
management, increased information, and/or improved incentives. Performance improves
measurable outcomes directly and indirectly through improved quality scores across the process
of care and structural and management checklists, and increased utilization of health care services
from patients, who are more likely to trust the services provided.
13
Figure 1 Theoretical Framework: Theory of Change
II. Data Collection Effort
The training and data collection for the baseline evaluation of the HSSP Impact Evaluation took
place from June 2 to July 20, 2015. The objectives of the baseline-evaluation effort were to: (a)
train enumerators on baseline data collection instruments; and (b) conduct baseline data collection
in 10 health facilities (5 intervention hospitals targeted by the performance-based financing (PBF)
program and 5 control hospitals) (c) prepare to begin the continuous data collection phase of the
project in Redemption hospital. The ten hospitals for data collection were: Redemption Hospital,
Liberian Government Hospital, CH Rennie Hospital, St. Francis Hospital, F.J.Grante Hospital,
Performance
(Quality,
Responsiveness,
Skills,
Experience
Equipment,
infrastructure
Motivation
Training
Competence Capacity
Structural and
supply
Improvements
Management,
Information,
Incentives
Baseline level of
health inputs
HSS Levers
Inputs
Measurable
Outcomes
Utilization and
Trust
Effort
Outputs
Outcomes
14
Phebe Hospital, Curren Hospital, Tellewoyan Hospital, G.W.Harley Hospital, and J.F.Doe
Hospital.
In the first week of June, 2015, the mission successfully undertook the training of the 32
enumerators and 2 supervisors mobilized by the MOHSW and World Bank. The training covered
topics including the objectives of the data collection, the ethics of research in the field, the main
guidance rules for quality assurance during data collection, the organization and logistics of the
data collection, and familiarization and practice with the data collection instruments. The
enumerators were recruited and specifically trained to administer certain instruments based on
their area of expertise. This is because a large part of the data collection instruments require
medical training to be successfully administered.
The data-collection was done in two groups for 5 hospitals each, with 16 enumerators and a
supervisor in each group. Each group’s enumerators were divided into 5 teams
(Obstetrics/Gynecology, Outpatient focused on Pediatrics, Surgery, Emergency/Triage, and
Hospital Assessment) to focus on specific data collection instruments based on their medical
training and/or area of expertise. Group 1 visited the following hospitals: F.J. Grante hospital
(Sinoe), St. Francis hospital (Rivercess), C.H.Rennie hospital (Margibi), Liberian Government
Hospital (Bomi), and Redemption hospital (Montserrado). Group 2 visited the following hospitals:
J.F.Doe (Nimba), G.W.Harley (Nimba), Tellewoyan (Lofa), Curren Hospital (Lofa), and Phebe
Hospital (Bong). The data collection schedule for both groups is presented in the appendix as Table
28. At each hospital, the groups spent 5 days for data collection, and often an additional sixth day
to observe staffing on weekends
Each enumerator and supervisor was assigned a unique identifier (ID) and was responsible for
remembering their ID, and recording it on each of the instruments they fill out. A quality check
process was established to ensure that all groups properly filled out all forms properly, to insure
that there was minimal missing data due to non-response or skipped questions. Due to possible
overlap of health worker shifts between the different wards, a protocol was established to ensure
that supervisors and enumerators constantly coordinated and checked to make sure all attendance
sheets, health worker rosters, and consent forms were periodically updated throughout each day.
15
Procurement of several goods and services were completed prior to the baseline survey. In
particular:
Personal Protective Equipment (PPE) was procured for the enumerators. Specifically, scrubs and
boots were procured for the enumerators in the OB and surgical wards, and gloves, masks, and
hand sanitizers were procured for all enumerators and supervisors.
Stationery (pencils, erasers, clipboards) were procured for all enumerators and supervisors.
Transportation to and from hospitals outside of Monrovia for the data collection team was arranged
for both groups. Inside Monrovia, transport was organized to and from Redemption hospital with
the Ministry of Health and Social Welfare (MOHSW) as a pick-up/drop-off point.
Printing and photocopying of all data collection instruments was done at the end of the training,
and then subsequently at the beginning of each hospital data collection where required.
The data collection comprised 33 separate instruments, which were designed to measure the
presence of health workers, their skills, capacity to perform, performance, and motivation. In
addition, the infrastructure and supplies of each facility were also assessed. All baseline survey
instruments were used in an earlier evaluation of the project in the 2013, and have been reviewed
since by the PIs in consultation with the clinical MOHSW staff in order to verify their medical
accuracy and local appropriateness. The instruments were revised accordingly prior to the training
of the enumerators MOHSW. Table 1 shows the breakdown of the services evaluated, and the
types of survey instruments used in each service.
16
Table 1 Summary of Instruments by Service and Location
17
A. Instruments filled
Table 2 shows the number of instruments completed per facility, disaggregated by motivation
surveys, staff registers, infection prevention and control forms, vignettes, direct observation and
exit surveys for each facility, as well as the total number of vignettes administered in each facility.
A total of 3,693 instruments were completed across the ten facilities. Overall, the maximum
number of instruments was collected for FJ Grante and Phebe hospitals. Data collection was lower
at Curran hospital, possibly due to low patient turnout (since the hospital is not free of charge) as
well as limited staff present. Hospital names have been abbreviated in all tables and represent the
following: Jackson F. Doe Hospital (JFD), Redemption Hospital (Redem), Phebe Hospital
(Phebe), St. Francis Hospital (St. F), Tellewoyan Memorial Hospital (Tell), Liberian Government
Hospital (LGH), C.H. Rennie Hospital (CHR), Curran Lutheran Hospital (Curr), F.J. Grante
Hospital (FJG), G.W. Harley Hospital (GWH). The instruments in the first column for Table 2
represent the following: General Observation of the Hospital Infrastructure (F_DO_1), Hospital
Staff Rosters (F_DO_2 and F_DO_3), Staff Motivation Survey (F_V_2), Infection, Prevention
and Control (IPC), Obstetrics Labor and Delivery Observation Checklist (O_DO_1), Obstetrics
Post Natal Care Observation (O_DO_2), Obstetrics Vignettes (O_V_1_A, O_V_1_B, O_V_2_A,
O_V_2_B, O_V_3, O_V_4_A, O_V_4_B), Obstetrics Patient Exit Survey (O_X), Pediatrics
Inpatient and Outpatient Direct Observation (P_DO_1), Emergency and Pediatrics Post Triage
Ward Care Observation (P_DO_2), Pediatrics Vignettes (P_V_1, P_V_2, P_V_3, P_V_4),
Pediatrics Patient Exit Survey (P_X), Surgical Safety Observation Checklist (S_DO_1),
Emergency and Surgery Post Triage Post-Surgical Ward Care Observation (S_DO_2), Surgery
Vignettes (S_V_1_A, S_V_1_B, S_V_2_A, S_V_2_B, S_V_3, S_V_4, S_V_5_A, O_V_5_B),
and Surgery Patient Exit Survey (S_X).
B. Health workers observed
Table 3 shows the total number of health workers (per facility) that participated in the data-
collection effort by being observed, by participating in a vignette or by answering survey questions
about their job. We looked at these numbers according to the credentials of the health workers who
18
were on the rosters of the hospital. We also looked at the proportion present on the first day of the
research at the facility. The rosters were sent by the hospital in advance of the visit and the names
were verified with the director of human resources (or staff responsible for human resources).
Health workers not present on the original rosters were added and names from the roster that were
no longer on the staff were deleted. The proportion present is the response to the question “is this
person present here at the facility today?” and was not independently verified. In general, the
medical officers present in the hospital rosters were not present for many procedures observed in
the four to five days. Physician assistants, midwives, and nurses provided most of the care in the
hospitals. Some of the hospitals had very few specialists and surgeons to perform procedures, often
leading to delays and referrals, for example, GW Harley, FJ Grante, and Liberian Government
Hospital. In addition, many of the staff in the hospitals were performing tasks much higher than
what they were qualified for, indicating an urgent need for more specialized and qualified health
workers.
19
Table 2 Number of Instruments Collected by Facility
JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH Total
F_DO_1 1 1 1 1 1 1 1 1 1 1 10
F_DO_2 1 1 1 1 1 1 1 1 1 1 10
F_DO_3 1 1 1 1 1 1 1 1 1 1 10
F_V_2 72 106 79 43 53 82 61 41 72 63 672
IPC 10 31 19 19 9 39 35 17 25 25 229
O_DO_1 0 11 8 4 12 6 8 4 11 9 73
O_DO_2 0 18 8 5 10 9 8 4 10 8 80
O_V_1_A 2 6 10 3 4 2 5 4 6 8 50
O_V_1_B 6 4 7 4 7 5 7 5 7 3 55
O_V_2_A 5 6 9 3 5 2 5 4 6 8 53
O_V_2_B 6 4 6 4 6 5 7 5 7 3 53
O_V_3 10 10 8 7 9 7 12 9 13 11 96
O_V_4_A 4 6 7 3 4 2 5 4 6 8 49
O_V_4_B 3 4 6 4 6 5 7 5 7 3 50
O_X 0 32 11 8 16 14 14 9 14 13 131
P_DO_1 72 60 60 65 66 60 61 26 76 40 586
P_DO_2 24 23 20 22 24 20 20 20 21 35 229
P_V_1 11 10 14 7 7 6 7 8 10 9 89
P_V_2 12 10 13 7 7 6 7 8 10 9 89
20
P_V_3 12 10 16 7 7 6 7 8 10 10 93
P_V_4 11 10 16 7 7 6 7 8 10 10 92
P_X 59 60 60 60 55 62 62 12 68 40 538
S_DO_1 12 5 3 2 6 9 6 1 5 13 62
S_DO_2 11 5 3 3 6 10 6 1 5 13 63
S_V_1_A 4 0 3 3 2 0 1 3 3 4 23
S_V_1_B 4 1 2 1 3 3 1 4 2 1 22
S_V_2_A 3 1 2 1 4 3 1 2 2 1 20
S_V_2_B 5 0 2 3 3 0 1 4 3 4 25
S_V_3 7 1 7 4 7 3 2 6 5 3 45
S_V_4 0 1 0 2 1 0 0 1 0 1 6
S_V_5_A 4 0 3 3 3 0 1 3 3 2 22
S_V_5_B 4 1 2 1 4 3 1 2 2 1 21
S_X 7 0 1 3 1 9 14 0 2 10 47
Total 298 299 307 246 292 263 283 170 324 280 3693
21
Table 3 Credentials of Health Workers on Hospital Rosters
JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH
Cadre N Pres N Pres N Pres N Pres N Pres N Pres N Pres N Pres N Pres N Pres
Certified Midwife 2 50% 27 82% 20 75% 4 100
% 9 22% 7 86%
1
4
100
% 8 63% 4
100
% 5 80%
Registered Nurse 7 43% 47 64% 20 60% 1
0
100
%
2
8 46% 21 76%
1
2 83% 8 38% 20
100
% 8 25%
Registered Nurse /
Certified Midwife 1
100
% 3 33% 2 50% 2 50% 1
100
%
Anesthetist
Registered
Nurse
2 0% 3 100
% 7 29% 1
100
% 2
100
% 1 0% 1
100
% 2 50%
Associate Degree
Nurse 3 67% 12 75% 4 75% 2
100
% 4
100
% 2
100
% 5 20% 1
100
%
Bachelors of Science
Nurse 2
3 39% 38 79% 37 43% 2
100
% 4 75% 4
100
% 4
100
% 5 60% 1
100
%
1
2 50%
Physician Assistant 1 0% 14 86% 3 100
% 4
100
% 8 63% 19 95%
1
5 87% 2 50% 6 83% 8 38%
Medical Doctor 6 67% 4 100
% 55 51% 3
100
% 6 33% 7 71% 4 50% 4
100
% 8 38%
Specialist 6 100
% 1
100
% 2
100
% 6
100
% 3
100
% 2
100
% 2 50% 2
100
% 5
100
%
Other 4
5 44% 69 70%
2
6 73%
3
7 30% 41 76%
3
3 88%
1
8 22% 66 55%
2
6 50%
Total 9
5 47%
21
6 74%
14
9 54%
5
4 87%
9
8 43%
11
0 82%
8
9 87%
4
9 39%
10
4 70%
7
5 51%
22
III. Instrument summary
In this section, we present the summary statistics for each instrument used in the survey by facility and ward type. We also show indices
of inputs, correct treatments and diagnoses, etc. The raw data (i.e. summary statistics for each question) of each instrument is contained
in the appendix.
A. Motivation surveys
Health workers were asked to answer a series of questions about their attitudes and motivations, with Likert options that ranged on a
five-point scale from “Strongly Agree” to “Strongly Disagree”. A factor analysis using varimax rotation was conducted on the full list
of questions showed a number of significant factors in the data.1 Using the cutoff of eigenvalues above 1, we reduced the data to five
factors listed in Table 4, along with the factor loadings for each item. As is always the case with factors, the labels attached are somewhat
arbitrary, however by looking at the items given the largest weights in each of the five factors we can develop simple labels, that,
although not unique, can be used to interpret the results. The factors are described as follows, and in order of the proportion of the data
they explain:
1. I can make a difference.
1 Some of the questions are negative in tone, and the score was inverted so that a higher score always reflected a more positive evaluation. In Table 4, these
questions are indicated with a *.
23
2. This hospital and management provide the right tools.
3. In this organization, I am taken seriously.
4. I am keen to improve.
5. Feedback, teamwork, recognition are important.
Table 4 Motivations Factors and Factor Loadings of Items
Factor 1
I can make
a difference
Factor 2
This
hospital
and
manageme
nt provide
the right
tools
Factor 3
In this
organizatio
n, I am
taken
seriously
Factor 4
I am keen
to improve
Factor 5
Feedback,
teamwork,
recognition
are
important
3.10: Good performance is recognized by our superiors 0.40 0.22 0.082 0.13 -0.095
3.11: This facility has a fair system for rewarding staff 0.19 0.43 0.19 0.081 -0.046
3.12: My performance is appraised regularly 0.30 0.42 0.22 0.082 -0.12
3.13: * Some of the team members work well, yet others do
not and so this facility doesn't perform well overall
0.14 0.21 -0.067 0.20 0.27
3.14: * We do not know how our facility is performing
compared to others in the district
0.18 0.20 -0.055 0.12 0.28
3.15: Our facility has clear goals that we are working towards 0.38 0.26 -0.071 0.33 -0.15
3.16: I understand how my work contributes to the facility's
overall goals
0.42 0.066 -0.13 0.32 -0.24
3.17: I am keen use any new tools to improve my performance 0.31 0.036 -0.24 0.35 -0.22
3.18: This facility has a good reputation in the community 0.35 0.40 -0.011 0.20 -0.051
3.19: This facility provides everything I need to perform well
at work
0.19 0.47 0.17 0.24 -0.035
24
3.20: There are enough health providers to do the work in this
facility
0.18 0.26 -0.019 0.046 -0.14
3.21: * Too often the referral system does not work efficiently 0.092 0.15 -0.057 0.027 0.15
3.22: Maintenance of broken equipment at this facility is
prompt and reliable
0.16 0.50 0.13 0.23 0.097
3.23: * I do not get feedback from my superiors so it is hard to
improve my performance
0.13 0.27 0.028 0.12 0.40
3.24: My job duties and responsibilities are clear and specific 0.34 0.18 -0.073 0.10 -0.042
3.25: Relevant guidelines are easy to access at this facility 0.30 0.33 0.10 0.13 -0.079
3.26: * I often feel left alone when I have to make difficult
decisions about a patient’s care
0.20 0.13 -0.19 0.14 0.21
3.27: I regularly have access to relevant trainings to keep my
skills up to date
0.18 0.23 0.18 0.11 -0.12
3.28: It makes me feel appreciated when patients are grateful 0.42 -0.23 -0.083 0.11 -0.083
3.29: I usually cope well with changes at work 0.25 0.086 -0.013 0.028 -0.12
3.30: * It is difficult for me to speak openly to my superiors
about how things are really going at work
0.21 0.28 0.026 0.17 0.27
3.31: * Suggestions made by health workers on how to
improve the facility are generally ignored
0.16 0.34 0.03 0.18 0.21
3.32: * I intend to leave this facility as soon as I can find
another position
0.20 0.37 0.060 -0.21 0.11
3.33: I would recommend to my children that they choose the
health profession
0.15 0.23 0.17 -0.28 -0.12
3.34: I am willing to put in a great deal of effort to make this
facility successful
0.60 -0.15 -0.15 -0.071 -0.013
3.35: I am proud to be working for this health facility 0.57 0.23 -0.089 -0.23 0.084
3.36: This hospital inspires me to do my best on the job 0.38 0.46 0.10 -0.22 -0.033
3.37: I am proud to tell others that I work in this ward / part of
the hospital
0.59 0.22 -0.094 -0.27 -0.046
3.38: I am glad that I work for this facility rather than other
facilities in the country
0.33 0.38 0.015 -0.36 -0.083
25
3.39: These days I feel motivated to work as hard as I can 0.34 0.43 -0.0016 -0.21 -0.067
3.40: My profession helps me to achieve my goals in life 0.46 0.10 -0.11 -0.21 0.19
3.41: Overall, I am very satisfied with my work in this ward
/part of the hospital
0.53 0.32 0.017 -0.30 -0.034
3.42: I am very satisfied to have a position where I can work
closely with the community
0.45 0.093 -0.16 -0.32 -0.073
3.43: I am satisfied with the opportunity to use my abilities in
my job.
0.55 0.073 -0.26 -0.19 -0.054
3.44:I am punctual about coming to work 0.61 -0.064 -0.14 -0.041 0.056
3.45:I am a hard worker 0.61 -0.18 -0.20 0.035 0.059
3.46:I work hard to make sure that no patient has to wait a long
time before being seen
0.58 -0.17 -0.21 0.13 0.015
3.47: I always complete my tasks efficiently and correctly 0.54 -0.10 -0.09 0.13 0.0027
3.48: When I am not sure how to treat a patient's condition I
look for information or ask for advice
0.61 -0.25 -0.29 -0.037 0.028
3.49: I try to get on well with the other health staff because it
makes the work
0.62 -0.19 -0.32 0.087 -0.040
3.50: I get along well with my superiors at work 0.50 0.13 -0.099 0.0040 0.024
4.10: I feel that I am a person of worth, at least on an equal
plane with others
0.36 -0.071 -0.23 0.079 -0.11
4.11: I feel that I have a number of good qualities 0.53 -0.25 -0.22 -0.013 0.10
4.12: * All in all, I am inclined to feel that I am a failure 0.15 -0.024 -0.011 -0.076 0.21
4.13: I am able to do things as well as most other people 0.47 -0.25 -0.11 0.078 -0.15
4.14: * I feel I do not have much to be proud of 0.079 -0.092 -0.082 -0.0062 0.35
4.15: I take a positive attitude toward myself 0.50 -0.32 0.057 -0.074 -0.028
4.16: On the whole, I am satisfied with myself 0.53 -0.18 0.20 -0.031 0.0025
4.20: In this organization, I am taken seriously 0.41 -0.0041 0.44 0.022 -0.079
4.21: In this organization, I am trusted 0.57 -0.26 0.37 0.063 -0.10
4.22: In this organization, I am important 0.62 -0.33 0.42 -0.047 0.094
4.23: In this organization, I can make a difference 0.65 -0.43 0.35 0.055 0.065
4.24: In this organization, I am valuable 0.62 -0.42 0.37 0.060 0.091
26
4.25: In this organization, I am helpful 0.65 -0.39 0.31 0.021 0.12
Table 5 shows the levels of these scores across the 5 facilities, by ward type. In addition, we show the mean Likert score (with all
negative questions reversed) in Table 6. Overall, the differences between the five facilities are small, but there are differences in the
factors. Curran and Redemption hospitals had the lowest overall factor scores, with especially low scores on Factor 1: “I can make a
difference.” For Curran hospital, one reason for this may be the low patient flow due to hospital fees. At Redemption, it was the opposite
situation with extremely high patient flow and workload, perhaps leading to tiredness and low motivation levels. Staff motivation levels
were highest for JF Doe and St. Francis hospitals. For example, during data collection at St. Francis hospital, the enumerators noted that
although the facility does not have many health workers, the few they have are very hard working and thus need more
motivation/incentives to stay in the hospital. Since JF Doe is a referral hospital mainly for complicated cases, it too has fewer staff, but
mostly specialists who were found to be hardworking, cooperative and motivated.
Table 5 Motivations Factors across Facilities and wards
Ward JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH Total
Means for Factor 1: I can make a difference
ER 0.25 -2.62 0.15 0.19 0.11 -1.3 -1.67 -7.42 -0.04 -0.3 -1.33
OB 0.64 -1.86 0.6 0.1 0.38 - -3.24 -6.98 0.19 0.56 -0.42
PE 0.49 -2.24 -0.3 -0.16 -0.05 0.56 -2.29 -7.33 -0.22 0.09 -1.03
SU 0.55 -4.87 -0.29 -0.21 0.37 - -0.7 -6.38 -0.33 0.01 -1.39
Total 0.49 -2.84 0.04 -0.08 0.19 0.25 -2.14 -6.96 -0.03 0.13 -1.05
Ward JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH Total
Means for Factor 2: This hospital and management provide the right tools
ER 0.58 1.25 -0.03 0.17 0.35 0.61 0.88 3.14 -0.09 -0.02 0.76
OB 0.15 0.91 0.11 0.06 0.18 - 2.09 3.41 -0.59 0.39 0.47
PE 0.47 1.18 0.2 0.2 0.05 -0.46 1.11 2.88 -0.07 0.23 0.6
27
SU 0.56 2.52 0.1 -0.26 -0.39 - 0.63 3.86 -0.39 0.23 0.86
Total 0.47 1.43 0.11 0.01 0.06 -0.28 1.2 3.38 -0.3 0.23 0.67
Ward JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH Total
Means for Factor 3: In this organization, I am taken seriously
ER 0.07 0.09 -0.52 -0.23 -0.04 0.54 -0.1 1.16 -0.36 -0.02 -0.02
OB 0.11 0.19 -0.08 0.07 -0.03 - 0.61 0.56 -0.06 -0.04 0.08
PE -0.33 0.03 -0.28 0.14 0.02 0.26 -0.02 0.96 -0.2 -0.22 -0.05
SU 0.01 0.53 -0.51 0.26 -0.01 - -0.01 1.19 -0.26 0.35 0.16
Total -0.03 0.18 -0.31 0.12 -0.01 0.31 0.09 1.03 -0.19 0.03 0.04
Ward JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH Total
Means for Factor 4: I am keen to improve
ER 0.72 -0.05 -0.15 -0.27 0.27 0.56 0.16 -0.04 -0.2 0.32 0.09
OB 0.23 0.08 0.05 -0.36 -0.09 - -0.31 0.56 -0.15 -0.03 -0.01
PE 0.64 -0.03 0.07 -0.14 0.34 0.6 0.15 -0.15 -0.06 -0.05 0.11
SU 0.61 0.02 -0.15 -0.57 0.38 - -0.09 0.18 -0.01 0.4 0.1
Total 0.57 -0.01 -0.02 -0.35 0.22 0.6 0.03 0.11 -0.12 0.16 0.08
Ward JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH Total
Means for Factor 5: Feedback, teamwork, recognition
ER 0.14 -0.04 0.44 0.41 0.24 -0.4 0 0.2 -0.11 0.43 0.13
OB 0.04 -0.25 -0.09 0.04 -0.39 - -0.11 -0.34 0.46 -0.14 -0.12
PE -0.04 -0.13 0.01 0 0.26 -0.24 -0.33 0.75 0.24 0.3 0.01
SU 0.22 0.05 -0.41 0.17 -0.1 - 0.05 0.9 0.11 0.04 0.06
Total 0.11 -0.08 -0.04 0.12 0.01 -0.27 -0.14 0.49 0.22 0.13 0.02
28
Table 6 Factor Means by Facility
JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH
Factor Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N
I can make a
difference 0.56 64 -6.8
9
6 -0.15 72 0.61 19 0.19 47 -0.28 80 -2.22 60 -6.9
3
2 -0.03 35 0.13 57
This hospital
and
management
provide the right
tools
0.34 64 3.98 9
6 0.15 72 0.03 19 0.08 47 -0.37 80 1.26 60 3.33
3
2 -0.3 35 0.21 57
In this
organization, I
am taken
seriously
-0.14 64 0.58 9
6 -0.47 72 0.13 19 -0.16 47 -0.42 80 0.1 60 0.99
3
2 -0.19 35 0 57
I am keen to
improve 0.66 64 -0.06
9
6 0.02 72 0.34 19 0.24 47 0.17 80 0.05 60 0.12
3
2 -0.12 35 0.14 57
Feedback,
teamwork,
recognition
0 64 0.02 9
6 -0.18 72 0.21 19 0.09 47 0.08 80 -0.16 60 0.44
3
2 0.22 35 0.08 57
Unweighted
average positive
responses
4.01 71 2.22 9
6 3.72 80 4.01 20 3.9 53 3.62 81 3.26 60 2.1
4
1 3.74 35 3.86 61
29
B. Competence/Capacity Vignettes
To measure competence and capacity, we used vignettes that were designed to test knowledge,
and to simultaneously assess the availability of equipment and materials that would allow health
workers to use that knowledge. Thus, many of the vignettes administered measure competence and
capacity, where capacity is equal to knowledge, after subtracting the things that health workers
knew how to do, but could not do because the required equipment was not available or not
functioning.
In addition, vignettes form an essential part of our identification strategy because they demonstrate
the link between key health inputs (physical examination for example) and important outcomes. If
we can demonstrate that a health worker who does more in trying to get to the correct diagnosis,
is indeed, more likely to obtain the correct diagnosis, then when we see health workers doing more
in practice with their patients, we can know that this will lead to probabilistically better outcomes.
1. Facility and health worker type averages for vignettes
Table 7 shows the average scores on inputs, diagnosis, and treatment, by vignette type (Obstetrics,
Pediatrics, or Surgery). Some of the scores are available only for a subset of vignettes, so the total
number of observations is not constant for all facilities. For many of the vignettes, the health
worker was attempting to obtain the correct diagnosis or treatment, and did so by using history
taking, physical examination, and laboratory tests. Each of these vignettes was associated with a
list of possible history-taking questions, physical-exam procedures and laboratory tests, with
health workers receiving a higher score if they used more of these items. These inputs make more
sense in the pediatric and surgical vignettes where health workers are trying to assess the health
worker’s condition. However, for the obstetric vignettes, the inputs are less standardized and
therefore difficult to summarize.
30
Table 7 Average Vignette Scores by Inputs, Diagnosis and Treatment
History Physical Correct
Assessment
Partial Correct
Diagnosis
Completely
Correct Diagnosis
Correct Treatment
N Mean N Mean N Mean N Mean N Mean N Mean
OB 0 - 0 - 59 0.64 61 0.39 61 0.3 61 0.67
Peds 301 0.33 301 0.39 0 - 212 0.36 212 0.54 176 0.65
Surg 169 0.45 169 0.52 125 0.41 169 0.66 85 0.34 104 0.21
Total 470 0.37 470 0.44 184 0.48 442 0.48 358 0.45 341 0.52
31
For OB, we do not have any input scores, however, OB health workers have higher average scores
for correct assessment and correct treatment but lower correct diagnosis scores, compared to
Pediatrics and Surgery. For Surgery vignettes, the average correct treatment score was lowest
compared to those for health workers completing Pediatrics and OB vignettes. This may in part be
due to the fact that there were very few specialist surgeon staff available to complete vignettes,
and some of the surgery vignettes were completed by GPs or qualified nursing staff, who may have
gotten some of the more complex cases wrong, reflecting the need for more highly skilled and
qualified surgeons in the hospitals.
To better understand the differences across facilities and across health worker types in the outputs
of the vignettes, we show the result of an analysis of each score by both facility and cadre type.
The results are presented in two sets of tables and graphs, but they come from the same regression.
Table 8 presents the probability of the correct assessment, partially correct diagnosis, completely
correct diagnosis, and treatment for each facility with JF Doe as the reference or control hospital.
This table includes the surgical, pediatric, and obstetric simulated patient vignettes. The same
results are shown graphically in Figure 2.
The results indicate that, compared to JF Doe, in terms of ability to do the assessment correctly,
Tellewoyan was the best performing and FJ Grante was the worst performing. In terms of
achieving partially or completely correct diagnoses, almost all the hospitals performed worse than
JF Doe, with the exception of Tellewoyan, Curran, and CH Rennie hospitals, although the slight
differences between them and JF Doe were not statistically significant. Health workers from
Phebe, St. Francis, Redemption, Tellewoyan, and Bomi, all performed statistically significantly
worse than JF Doe hospital, in terms of making correct diagnosis, signaling the need for better
training in screening and diagnosis for healthcare professionals. For doing the correct treatment as
well, almost all the hospitals performed worse than JF Doe, with the exception of CH Rennie.
Health workers from Phebe, St. Francis, Redemption, Tellewoyan, and Bomi, Curran, and GW
Harley, all performed statistically significantly worse than JF Doe hospital, in terms of going about
the correct treatment, revealing a serious gap to be addressed.
32
Table 8 Regression coefficients for vignette scores by facility for assessment, diagnosis, and treatment factors
Assessment Partially Correct
Diagnosis
Completely
Correct Diagnosis
Treatment
b (SE) b (SE) b (SE) b (SE)
JF Doe (ref) (ref) (ref) (ref)
Redemption -0.041 (0.087) -0.025 (0.062) -0.122 (0.061)** -0.202 (0.088)**
Phebe 0.06 (0.088) -0.108 (0.060)* -0.021 (0.060) -0.188 (0.089)**
St. Francis -0.139 (0.142) -0.17 (0.099)* -0.03 (0.102) -0.234 (0.125)*
Tellewoyan 0.126 (0.086) 0.044 (0.057) -0.14 (0.059)** -0.266 (0.082)***
Bomi 0.038 (0.168) -0.046 (0.123) -0.172 (0.103)* -0.207 (0.128)
CH Rennie -0.097 (0.170) -0.151 (0.122) 0.031 (0.121) 0.065 (0.167)
Curran -0.103 (0.141) 0.059 (0.092) -0.054 (0.103) -0.247 (0.124)**
FJ Grante -0.207 (0.112)* -0.036 (0.099) -0.147 (0.092) -0.179 (0.128)
GW Harley 0.084 (0.138) -0.047 (0.101) -0.1 (0.101) -0.371 (0.090)***
Note: *p<0.05, **p<0.01, ***p<0.001
Figure 2 Probability of Being Correct Across All Vignettes by Facility
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Assessment Partial Diag Complete Diag Treatment
33
Error! Not a valid bookmark self-reference. presents the same results but with health worker
category as predictors, with the values also shown in Figure 3. Nurses with bachelors or the
equivalent served as the reference category. The other two categories were 1) Medical Doctors
(MDs) including specialists, general practitioners and medical students and 2) Registered Nurses
(RN), including nurse’s aides, and nursing certificate and diploma holders. We found that MDs
made statistically significantly greater correct diagnoses and treatment calls compared to their
nurse counterpart, as expected due to their greater years in specialized training. However, RNs
also made statistically significantly more partially correct diagnosis compared to nurses, but none
of the other outcomes were significantly different between the two nursing categories.
Table 9 Regression Coefficients for Vignettes by Health Worker Type for Assessment, Diagnosis, and Treatment Factors
Assessment Partially Correct
Diagnosis
Completely
Correct Diagnosis
Treatment
b (SE) b (SE) b (SE) b (SE)
Nurse (ref) (ref) (ref) (ref)
MD 0.071 0.451*** 0.209*** 0.223**
RN -0.09 0.128*** -0.041 0.025
Note: *p<0.05, **p<0.01, ***p<0.001
Figure 3 Probability of Being Correct by Health Worker Type
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Assessment Partial Diag Complete Diag Treatment
MD RN Nurse
34
2. Validating the theory of vignettes
The superior training of MDs (and of RNs) might be the reason that they achieve better outcomes
in these vignettes, but Figure 4 shows that their superior knowledge and training is not the direct
cause of their better outcomes. In particular, MDs provide more history taking and physical
examination than either RNs or other nurses.
Figure 4 Proportion of Possible Inputs into Correct Diagnosis and Treatment by Health Worker Type
Table 10 presents the results of the regression analyses with inputs (history taking, physical, and
lab tests) as predictors, and correct assessment, diagnosis, and treatment as outcomes, across all
vignettes. We see a similar pattern, except now there is no strong relationship between inputs and
the ability to obtain the correct general diagnosis. Overall physical examination is important for
the detailed diagnosis and treatment, whereas history taking is important for the partial diagnosis
of the condition.
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
History Physical Lab Tests
MD RN Nurse
35
Table 10 Relationship between Inputs, Diagnosis, Detailed Diagnosis, and Treatment
Assessment Partial Correct
Diagnosis
Completely
Correct Diagnosis
Treatment
b (SE) b (SE) b (SE) b (SE)
History 0.069 (0.113) 0.217 (0.092)** 0.077 (0.082) 0.143 (0.131)
Physical 0.088 (0.115) 0.241 (0.090)*** 0.305 (0.073)*** 0.244 (0.121)**
Lab tests 0.027 (0.174) -0.194 (0.161) 0.162 (0.134) 0.081 (0.191)
N 324 403 403 265
As we can see in Figure 5 and Figure 6, the increased use of key inputs improves the probability
of obtaining the correct diagnosis and the use of physical examination increases the probability of
the correct treatment. It is not the case that training allows one to shortcut the diagnosis process
and get to the correct diagnosis quicker; training teaches the health workers how to properly use
physical examination, history taking and laboratory tests to obtain the correct diagnoses and to
prescribe the correct treatment.
Figure 5 Association between the Use of Key Inputs and Correct Diagnosis
.4.4
5.5
.55
.6.6
5
Pro
ba
bili
ty o
f co
rrect d
iagn
osis
0 .2 .4 .6 .8 1Proportion of Correct Inputs
History Taking Physical Examination
HT + PE
Relationship of Correct Diagnosis and Inputs
36
Figure 6 Association between the Use of Physical Examination and Correct Treatment
.35
.4.4
5.5
.55
Pro
ba
bili
ty o
f co
rrect tr
ea
tme
nt
0 .2 .4 .6 .8 1Proportion of Correct Inputs
History Taking Physical Examination
HT + PE
Relationship of Correct Treatment and Inputs
37
C. Direct observation
The researchers observed health workers in four different settings, the obstetric ward, the pediatric
ward, the emergency ward and the surgical ward. In these settings, the activities of health workers
were recorded with three different sets of instruments: obstetric, pediatric, and surgical.
Table 11 examines markers of quality in obstetric care. Not all patients were observed in all stages,
and of course, not all patients experienced complications, so the numbers of observations are
relatively low. We do not expect to observe many cases on neonatal asphyxiation, for example,
which is one of the reasons why vignettes were designed to test for health worker responses in
these cases. The variance is quite high for different aspects of care. In general, the quality of care
is much higher when there are complications such as pre-eclampsia, across all hospitals. On the
other hand, the use of the partograph is generally low, with proper use of partograph being
inadequate or inappropriately timed (in some cases, after the delivery). There are not enough
observations to draw strong conclusions about the differences between hospitals, but it seems clear
that there is ample opportunity to improve quality in all hospitals.
Table 12 examines the quality of care in pediatric care, both in the pediatric outpatient ward and
the emergency ward. There are significantly more observations for these instruments allowing us
to make stronger statements about the difference between hospitals. However, average
observation scores on different aspects of pediatric care were fairly similar, with little variation.
Table 13 examines key features of the few surgical procedures observed. Evaluating the quality of
a surgical process is difficult, particularly since most of them are quite different from each other.
The table therefore focuses on certain key markers in the preparation for surgery, including items
such as politeness, the use of a surgical check list and the proper administration of antibiotic
prophylaxis before the incision.
38
Table 11 Direct Observation of Obstetric Care
Facility
JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH
Avg N Avg N Avg N Avg N Av
g
N Avg N Avg N Avg N Avg N Avg N
Obstetric Intake:
History Taking
1 2 0.33 6 0.96 6 0 4 0.75 12 0.36 3 0.13 3 0.97 2 0.16 4 0.09 6
Obstetric Intake:
Presence of
Complications
0.06 2 0.33 3 0.16 6 1 3 0.19 10 3.13 3 6 3 0.44 2 1.63 4 5.9 5
Obstetric Intake:
Overall Quality
0.6 2 0.44 6 0.66 6 0.29 4 0.59 12 0.39 3 4.06 3 0.61 2 0.33 4 0.47 6
First Stage of Labor
Observation
Quality
0.7 2 0.64 1
6
0.73 13 1.68 8 0.56 14 1.09 4 7.13 5 0.57 4 0.48 5 0.9 8
Second and third
stage observation
quality
0.75 4 0.83 2
2
0.88 15 0.75 11 0.94 19 1.96 4 1.36 5 1.03 4 0.78 8 0.98 8
Episiotomy:
Usually
unnecessary
0 4 0.09 2
2
0.13 15 0.18 11 0.05 19 0 4 0.4 5 0 4 0.13 8 0 8
Newborn
Resuscitation
Quality
0 0.68 5 0.92 1 0.78 2 0.87 1 0.87 1 0 0 0.29 2 0.8 1
Potentially Harmful
Practices
0 4 0.08 2
5
0 17 0.05 12 0.02 19 0.01 6 0.05 8 0 4 0.09 12 0.01 8
Partograph Quality 0 4 0.49 2
4
0.67 17 0.73 12 0.79 19 0.15 6 1.08 8 0 4 0.42 12 0.71 8
Filling Partograph
After Delivery
0 0.67 1
5
0.83 12 0.67 9 1.88 8 1 1 0.57 7 0 0.33 6 1 6
Obstetric Care PPH
Quality
0 1 1 0 0.88 1 0.88 1 0 0 0 0.86 1
39
Obstetric Care: Pre-
Eclampsia Quality
1 4 0.97 2
7
0.97 17 0.95 13 0.97 21 1 6 1 8 1 4 1 12 1 9
Table 12 Direct Observation of Pediatric Care
Facility
JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH
Avg N Av
g
N Av
g
N Avg N Avg N Avg N Avg N Avg N Avg N Avg N
Pediatric
Observatio
n Physical
Examinatio
n Score
0.48 151 0.37 120 0.41 111 0.41 118 0.2 124 0.29 59 0.36 60 0.38 26 0.29 74 0.24 74
Pediatric
Observatio
n History
Taking and
Physical
Exam
0.41 151 0.35 121 0.36 113 0.38 118 0.19 125 0.28 59 0.37 60 0.36 26 0.27 74 0.23 74
Pediatric
Observatio
n Health
Education
Score
0.29 124 0.36 112 0.34 90 0.34 107 0.27 104 0.31 46 0.44 56 0.23 22 0.39 55 0.27 67
Pediatric
Observatio
n General
Effort
Score
0.25 139 0.22 106 0.26 106 0.17 79 0.24 100 0.17 57 0.18 45 0.21 27 0.22 71 0.22 56
Pediatric
Observatio
n Physical
Examinatio
n Score
0.48 151 0.37 120 0.41 111 0.41 118 0.2 124 0.29 59 0.36 60 0.38 26 0.29 74 0.24 74
40
Table 13 Direct Observation of Surgical Care
Facility
JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH
avg N avg N avg N avg N avg N avg N avg N avg N avg N avg N
cleanliness of operating
theater 0.92 27 0.9 16 0.92 14 0.56 11 0.83 7 0.8 9 0.77 6 0.56 1 0.5 5 0.8 13
Staff was polite,
respectful 1 24 0.94 18 1 12 0.64 11 1 6 1 9 1 6 1 1 0.8 5 1 13
Preparation of materials
pre-surgery 0.83 25 0.68 18 0.86 12 0.7 12 0.86 6 0.71 9 0.77 6 0.91 1 1.23 5 0.78 13
Preparation of the
primary surgeon 0.87 49 0.59 28 0.82 27 0.77 22 0.76 13 0.59 18 0.67 12 0.81 2 0.65 10 0.82 25
Surgeon was polite,
respectful 1 24 0.94 18 1 12 0.64 11 1 6 1 9 1 6 1 1 0.8 5 1 13
use of surgical check list 0.16 58 0 34 0.04 27 0 25 0.29 14 0 18 0 12 0.5 2 0.1 10 0.04 25
antibiotic prophylaxis
given or confirmed 0.36 25 0.27 15 0.14 14 0 9 0.43 7 0.11 9 0 6 1 1 0.2 5 0 12
antibiotic prophylaxis
known given within 60
minutes of incision
0.08 25 0 15 0 14 0 9 0 7 0.11 9 0 6 0 1 0 5 0 12
General surgical Quality
(before incision) 0.77 27 0.61 15 0.74 14 0.65 9 0.78 7 0.48 9 0.53 6 0.92 1 0.6 5 0.71 13
The Surgeon broke the
surgical field during
surgery
0.1 58 0 34 0.04 27 0.08 25 0 14 0.06 18 0.08 12 0 2 0 10 0 25
Medical Staff Post
Surgery Clean Up 0.61 24 0.45 15 0.56 12 0.41 11 0.7 7 0.5 9 0.66 6 0.69 1 0.71 5 0.74 12
Housekeeping Staff Post
Surgery Clean Up 0.68 25 0.65 13 0.74 12 0.32 11 0.89 7 0.64 9 0.82 6 1 1 0.77 5 0.92 12
41
D. Exit surveys
Patients were also interviewed about their experiences and observations of the hospitals. Table 14
shows the socio-demographic characteristics of the patient sample by facility and ward (pediatrics,
obstetrics, and surgery).
42
Table 14 Patient Characteristics
Facility
JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH
Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N
Age 26 125 27.29 180 25.68 154 29.16 143 26.77 146 27.34 77 27.64 80 17.59 22 29.92 83 21.78 65
Patients without
any schooling
0.33 146 0.24 199 0.37 164 0.39 155 0.42 158 0.34 85 0.34 88 0.27 22 0.3 86 0.18 74
Patients completed
primary schooling
0.32 146 0.22 199 0.32 164 0.25 155 0.22 158 0.22 85 0.32 88 0.5 22 0.26 86 0.32 74
Patients completed
secondary schooling
0.33 146 0.45 199 0.24 164 0.31 155 0.28 158 0.34 85 0.25 88 0.23 22 0.35 86 0.42 74
Patients completed
vocational training
0 146 0.02 199 0 164 0.01 155 0.01 158 0.02 85 0.01 88 0 22 0.05 86 0.04 74
Patients completed
post-secondary schooling
0.02 146 0.06 199 0.04 164 0.02 155 0.07 158 0.02 85 0.02 88 0 22 0.02 86 0.01 74
Patients reporting
being able to read easily
0.26 146 0.4 199 0.18 164 0.26 155 0.23 158 0.22 85 0.22 88 0.32 22 0.31 86 0.36 74
Patients reporting being
able to read with difficulty
0.32 146 0.29 199 0.39 164 0.3 155 0.29 158 0.35 85 0.39 88 0.32 22 0.26 86 0.39 74
Patients reporting not
Being able to read at all
0.42 146 0.29 199 0.43 164 0.44 155 0.44 158 0.4 85 0.4 88 0.36 22 0.42 86 0.23 74
Travel Time: Less than
30 minutes
0.17 146 0.27 199 0.18 164 0.23 155 0.34 158 0.38 85 0.36 88 0.36 22 0.53 86 0.51 74
Travel time:
30 - 60 minutes
0.12 146 0.22 199 0.16 164 0.14 155 0.09 158 0.32 85 0.4 88 0.23 22 0.33 86 0.16 74
Travel Time: More
than 60 Minutes
0.21 146 0.1 199 0.17 164 0.17 155 0.13 158 0.28 85 0.19 88 0.36 22 0.13 86 0.19 74
Index of Assets
Owned
-0.15 145 0.89 197 -0.33 164 -0.17 152 -0.19 158 -0.45 83 -0.04 88 -0.1 22 0.12 85 -0.16 72
43
Table 15 shows the responses given by facility about general satisfaction across the obstetric,
pediatric and surgery/emergency services. Respondents scored a number of characteristics on a
Likert score and were asked, for each characteristic, if it was important to them. Likert options
ranged on a five-point scale from “Strongly Agree” to “Strongly Disagree”.
44
Table 15 Satisfaction As Expressed Across All Three Types of Exit Surveys
Facility
JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH
Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N
It is convenient to
travel from your house
to the health facility.
2.67 85 3.78 134 2.98 107 3.53 95 3.43 103 3.9 21 3.41 22 3.89 9 3.13 15 3.78 23
The health facility is
clean. 4.24 85 4.02 134 3.93 107 3.86 95 4.17 103 4.24 21 4.41 22 4.11 9 3.2 15 3.91 23
The health staff is
courteous and
respectful.
4.01 85 4.01 134 3.93 107 3.88 95 4.15 103 4.71 21 4.59 22 4.22 9 3.53 15 4.26 23
The health workers did
a good job of
explaining your
condition.
3.82 85 3.6 134 3.64 107 3.65 95 3.81 103 4.19 21 4.64 22 4.22 9 3.13 15 4.17 23
It is easy to get
medicine that health
workers prescribe.
3.47 85 3.84 134 3.31 107 3.56 95 3.32 103 3.95 21 4.05 22 3.22 9 3.53 15 3.61 23
The amount of time
you spent waiting to be
seen by a health
worker was not
3.04 85 3.72 134 3.46 107 3.27 95 3.69 103 3.76 21 4 22 4.22 9 3.07 15 3.52 23
You had enough
privacy during your
visit.
3.94 85 3.46 134 3.82 107 3.59 95 3.93 103 3.95 21 4.5 22 4.22 9 3.07 15 3.57 23
The health worker
spent a sufficient
amount of time with
you.
3.85 85 3.98 134 3.73 107 3.72 95 3.96 103 4.33 21 4.32 22 4.33 9 2.93 15 3.65 23
45
The overall quality of
services provided was
satisfactory.
3.95 146 4.01 195 3.89 164 3.95 154 4.01 158 4.01 83 4.12 84 4.05 22 3.68 84 3.75 73
You felt that the
facility maintained
confidentiality of your
personal info
3.92 146 3.99 195 3.93 164 3.91 154 3.98 158 4.01 83 4.07 84 4 22 3.74 84 3.85 73
You were asked for
consent (permission)
before any procedures.
3.36 145 3.43 195 3.38 164 3.36 154 3.25 158 3.86 83 3.86 84 3.32 22 3.61 84 3.01 73
You felt respected by
your healthcare
provider.
4 144 4.08 195 4.01 164 3.99 154 4.11 158 4.08 83 4.15 84 4.05 22 3.92 84 4.03 73
You had a warm and
compassionate
healthcare provider.
3.91 145 4.1 195 3.99 164 4.03 154 4.04 158 4.1 83 4.15 84 4.09 22 3.94 84 3.92 72
46
Using factor analysis, we reduced this list of responses to two significant factors: quality and
convenience. The factor scores are presented in Table 16. The summary scores for these factors
are reported in Table 17, alongside the average Likert score. Note that the overall Likert score
(which shows little variation) masks the important differences in the two factors of quality and
convenience. For example J.F. Doe is ranked as much higher than St. Francis in quality, but the
facilities are nearly opposite in convenience. Note also that the ranking of facilities by health
workers and patients is not the same. Health workers were the most satisfied at Liberian
Government Hospital, followed by GW Harley, FJ Grante, Tellewoyan, JF Doe, and Redemption
(all attaining positive overall satisfaction scores). These were followed by St. Francis, Curran,
Phebe, and finally CH Rennie, with all these hospitals having negative overall satisfaction scores.
Table 16 Factor Loading for Patient Exit Interviews
Factor1:
Quality
Factor2:
Convenience
It is convenient to travel from your house to the
health facility.
0.77 0.31
The health facility is clean. 0.91 -0.077
The health staff is courteous and respectful. 0.95 -0.14
The health workers did a good job of explaining
your condition.
0.88 0.07
It is easy to get medicine that health workers
prescribe.
0.84 0.24
The amount of time you spent waiting to be seen by
a health worker was not
0.80 0.30
You had enough privacy during your visit. 0.84 0.01
The health worker spent a sufficient amount of time
with you.
0.90 0.01
The overall quality of services provided was
satisfactory.
0.94 -0.13
You felt that the facility maintained confidentiality
of your personal info
0.90 -0.15
You were asked for consent (permission) before
any procedures.
0.77 0.12
You felt respected by your healthcare provider. 0.95 -0.21
You had a warm and compassionate healthcare
provider.
0.95 -0.20
47
Table 17 Satisfaction Summary Scores
Facility
JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH
Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N
Quality 0.16 85 0.27 134 0.03 107 -0.07 95 0.41 103 1.61 21 1.46 22 0.87 9 -1.61 15 0.52 22
Convenie
nce -0.18 85 0.31 134 -0.15 107 0.07 95 -0.01 103 0.05 21 -0.04 22 -0.01 9 0.14 15 0.11 22
Average
Likert
scale score 3.73 146 3.88 195 3.73 164 3.78 154 3.82 158 3.96 83 4.06 84 3.91 22 3.76 84 3.67 73
Important:
overall
satisfactio
n 0.09 78 0.08 134 -0.14 106 -0.01 92 0.11 102 1.6 12 -0.15 10 -0.13 9 0.75 13 1.49 13
Important:
individual
attention 0.66 78 -0.2 134 -0.25 106 -0.01 92 -0.17 102 -0.49 12 -0.01 10 -0.51 9 -0.31 13 -0.38 13
48
Table 18 Satisfaction Summary Scores by Facility and Ward
Ward JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH Total
Means for Factor 1: Quality
SU 0.11 0.18 -0.1 0.3 0.43 1.87 2.08 - -1.34 -0.2 0.49
OB 1.18 0.7 0.12 0.37 1.02 1.41 0.72 0.87 -1.65 1.12 0.58
PE 0.01 -0.12 0.01 -0.32 0.1 - - - - - -0.06
Total 0.16 0.27 0.03 -0.07 0.41 1.61 1.46 0.87 -1.61 0.52 0.25
Means for Factor 2: Convenience
SU -0.25 -0.34 -0.43 -0.17 -0.27 -0.18 -0.39 - 0.74 -0.17 -0.26
OB 0.36 0.61 0.09 0.15 0.22 0.22 0.38 -0.01 0.05 0.34 0.3
PE -0.24 0.24 -0.22 0.08 -0.08 - - - - - -0.05
Total -0.18 0.31 -0.15 0.07 -0.01 0.05 -0.04 -0.01 0.14 0.11 0.03
Means Total Score
SU 3.76 3.78 3.56 3.83 3.79 4.29 4.46 - 3.31 3.61 3.86
OB 4.09 3.98 3.71 3.87 4.07 4.23 4.02 4.04 3.24 4.12 3.93
PE 3.7 3.85 3.74 3.76 3.76 3.86 3.99 3.82 3.88 3.56 3.78
Total 3.73 3.88 3.73 3.78 3.82 3.96 4.06 3.91 3.76 3.67 3.82
Mean Importance Factor 1: Overall Satisfaction
SU -0.03 0.03 -0.06 -0.01 0.05 - - - - - 0
OB -0.23 0.21 -0.43 -0.29 0.41 1.6 -0.15 -0.13 0.75 1.49 0.24
49
PE 0.17 -0.02 -0.05 0.09 -0.04 - - - - - 0.03
Total 0.09 0.08 -0.14 -0.01 0.11 1.6 -0.15 -0.13 0.75 1.49 0.1
Mean Importance Factor 2: Respect and Warmth
SU -0.05 0.11 -0.16 0.09 0.31 - - - - - 0.08
OB -0.54 -0.46 -0.7 -0.35 -0.58 -0.49 -0.01 -0.51 -0.31 -0.38 -0.47
PE 1.01 -0.04 -0.09 0.11 -0.05 - - - - - 0.17
Total 0.66 -0.2 -0.25 -0.01 -0.17 -0.49 -0.01 -0.51 -0.31 -0.38 -0.07
50
I. Impact Evaluation Strategy
The HSSP will take place in five pre-selected hospitals. Given the small number of hospitals in
Liberia, it was determined at early stages that a full randomized controlled trial was not feasible.
Thus, for the evaluation at the hospital level, the impact evaluation takes the project as a given,
without control over the selection of treated or control hospitals, the timing, or structure of any
interventions. Five control facilities have been chosen to match the treated hospitals. The original
criteria were that hospitals be drawn from urban and rural areas, from different regions of Liberia
and from public and private services. Thus, each control facility is matched to the paired treatment
facility based on these characteristics. Facilities were not selected on the basis of initial quality
levels or on capacity to absorb the Project.
These restrictions on the number of hospitals suggest a simple before-and-after, matched pair
comparison using measurable outcomes directly and indirectly tied to the project. However, with
only five units in treatment and control groups, this strategy has very limited power to detect
statistically significant improvements in outcomes. Although we will report these outcomes, the
evaluation strategy focuses, instead, on the theory of change within treated facilities and on a series
of augmented PBF interventions at the health-worker level within treated facilities. The external
validity of the Project as a whole is limited because the design is specific to Liberia, but the lessons
about how things change within hospitals should have more generalizable findings.
IV. Evaluating the theory of change within
HSSP facilities
In order to better understand how PBF, as defined by the HSSP, is functioning within hospitals,
we propose to collect the data necessary to measure “the three gaps” for health workers in the
treated and control facilities.
A. The Three Gap Framework
51
The three-gap framework focuses on four levels of care: 1) the competence to perform; 2) the
capacity to perform; 3) actual performance; and 4) the target levels of performance. The three gaps
defined by these four levels are the know gap, the know-can gap and the can-do gap. Figure 7
graphically demonstrates the three-gap framework. Performance originates with knowledge,
education, and skills as measured by competence (C), shown on the vertical axis extending below
the origin: increases in competence are shown by points closer to the bottom of the figure (further
from the origin). Capacity (K) comes from competence, taking into account the infrastructure,
equipment, and medicines needed to appropriately use training, education, and skill, shown on the
horizontal axis to the right of the origin. Finally, performance (P) comes from taking capacity and
combining it with effort: health workers must choose to use their knowledge and equipment in
order to perform, shown on the vertical axis above the origin.
An important feature of this model is the fact that performance is limited by capacity, and that
capacity is limited by competence: a health worker cannot do better than what he or she knows
how to do, for example. This is shown in the graph by what we call the capacity and performance
barriers. In the translation from competence to capacity, and again from capacity to performance,
we take into account the capacity barrier and performance barrier, shown as 45-degree lines from
the origin in the lower right and upper right quadrants. These barriers reflect the fact that better
equipment cannot produce capacity beyond the level of competence, and that greater effort cannot
produce performance beyond the level of capacity. In other words, competence limits capacity,
and capacity limits performance.
52
Figure 7 The Three-Gap Framework
Target competence is the level required to perform according to the target level of performance,
marked in Figure 1 as CT. In the ideal world, any health worker with target competence would also
have target capacity KT: if health workers had all the equipment and medicines to work according
to their training, capacity and competence would be the same. In addition, this capacity (KT) would
ideally translate directly into targeted performance, PT.
The idea of the three-gap framework is that, in the real world, not all health workers have target
competence, competence does not always translate into capacity and capacity does not always
translate into performance. In addition to this ideal triplet (CT, KT, PT), Figure 7 shows an example
of another possible triplet (C, K, P). Competence (C) is lower than targeted performance, and
because some equipment or medicine is not available, this particular health worker has capacity
(K), which is lower than his competence. This is shown by the pair (K, C) in the lower right
quadrant and the fact that (K, C) is to the left of the capacity barrier. Furthermore, because effort
is not ideal, the health worker does not fully transform capacity into performance (P), as shown by
the pair (K, P) in the upper right quadrant. Thus, performance is significantly below target
performance. Importantly, we can divide the shortfall into three gaps:
53
1) The know gap (shown as G1), which is the difference between targeted performance and the
competence to perform;
2) The know-do gap (shown as G2), which is the difference between competence and capacity; and
3) The can-do gap (shown as G3), which is the difference between capacity and performance.
Note that these gaps are important for policy reasons. A large know-gap suggests deficiencies in
training. A large know-do gap suggests deficiencies in infrastructure, equipment, or medicines,
and a large can-do gap suggests deficiencies in motivation. However, the size of the gaps by
themselves is not enough to understand the potential gains from changes in policy. In order to do
this, we need to understand the relationship between competence, capacity, and performance.
1. The Three Gaps in our Sample
Figure 8 shows the actual pattern of the three gap framework across all the facilities in the sample
in pediatric care. We focus on pediatric care because the correspondence between competence and
capacity vignettes and performance observations is the most straightforward. In other words, there
are a large number of items for which we can observe whether or not a health worker would
perform the action if presented with a case study vignette (competence), whether or not they have
the equipment present that would allow them to perform the action in practice (capacity) and
whether or not the same health worker does perform the action in practice. In addition, most care
in pediatrics can be linked to one health worker, whereas in obstetrics, in particular, multiple health
workers are responsible for the same patient making it more difficult to link three types of outputs
across one health worker.
Note that Figure 8 shows all health workers measured across all facilities, so we observe the widest
possible variety of abilities. What is readily apparent in this graph is that the competence capacity
gap is not a major factor. Except at the lowest end of competence, health workers are not limited
by the availability of equipment and the line connecting competence and capacity is not
statistically different from the 45 degree line representing a one to one mapping of competence
onto capacity. On the other hand the relationship between capacity and performance is quite
different. Except that it slopes slightly upwards (greater capacity leads to greater performance) the
relationship between these two measures is very week. In fact, health workers with very low levels
54
of capacity actually perform at levels above that expected for the competence, and health workers
with very high levels of capacity perform at levels significantly below their level of capacity.
Figure 8 The Three Gap Framework for our sample (pediatrics)
Overall, this suggests that across all facilities, performance is a matter of habit, not a matter of
deliberate application of knowledge. More importantly, it suggests that improvement in
competence or capacity will not automatically result in any improvements in performance. Even
when competence is low, it is not the limiting factor in performance.
2. Analyzing the three gaps by facility
Figure 9 shows the size of four gaps for each facility in the sample. The four gaps we examined
are the target to performance gap (the target-do gap), the target-know gap, the know-can gap and
the can-do gap. Recall that the overall gap between the targeted behavior and the actual behavior
can be broken up into the three subsequent gaps. In general it is clear that the target-know gap is
the largest gap: health worker knowledge of what they should be doing (as demonstrated on the
55
vignettes) is very low. Recall that MDs do better on vignettes than other cadres, and recall that in
general a very small proportion of all health care in a hospital is directly provided by an MD.
Importantly the know-can gap is very small across all facilities. This shows that health workers
have the capacity to do what they know how to do: they are not obviously lacking in equipment or
medicines. The can-do gap is larger than the know-do gap and demonstrates some interesting
variation. St Francis and Curran hospitals, for example, demonstrate a very low level of the can-
do gap: health workers in these hospitals do what they can do. On the other hand the can-do gap
at Redemption is much larger. In fact, redemption hospital demonstrates the smallest target-
knowledge gap as well as the largest can-do gap: they know much more than other hospitals but
do much less of what they know.
Figure 9 Four Gaps across facilities
Figure 10 examines the links between these four gaps and the five motivation factors looking at
all health workers across all facilities. It helps explain why these gaps exist and what are some of
the factors that help explain their size. For example if the management at one facility is very good,
then health workers are likely to be highly motivated and to do their job well, certainly compared
to health workers at a facility that is less well managed. It is difficult to say that they do their job
because they are motivated, or conversely that they are motivated because they do their job.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%JF Doe
Redemption
Phebe
St. Francis
Tellewoyan
Bomi
CH Rennie
Curran
FJ Grant
GW Harley
56
Nonetheless the patterns we see can be informative. Figure 10 can also be considered a graphical
representation of the t-statistics of a regression of each of the four gaps on all five of the motivation
factors. The size of the coefficient has no natural interpretation, but the t-statistic shows how
important the factor is. In particular, if the t-statistic is larger than 1.9 (or smaller than -1.9) then
there is only a 5% chance that the factor is not statistically important. Thus, for the overall gap (the
target-performance gap) there are two factors that are significantly related to the size of the gap:
“This hospital and management provide the right tools” and “In this organization, I am taken
seriously.” The other three factors have t-statistics below the 5% cutoff, and are therefore not
shown in the graph. Note that the first factor (“This hospital and management provide the right
tools”) has a negative association with the gap, which means an increase in this factor closes the
gap. On the other hand “In this organization, I am taken seriously” has a positive association,
which means that for health workers and hospitals where this factor is more present, the gap is
larger.
57
Figure 10 Motivation factors and the four gaps with facility fixed effects
Figure 11 shows similar results but each regression includes fixed effects for each facility. As a
result the differences in gaps and motivations across facilities are eliminated and we can interpret
this as comparing health workers within facilities. Here we see that the second factor (“This
hospital and management provide the right tools”) still plays an important role in eliminating the
overall gap and the can-do performance gap. Note that, since we have included facility fixed
effects, this compares, for example, two workers within the same hospital: the one who is more
likely to believe that the hospital provides the right tools is the one more likely to provide better
quality care.
Note that because the gaps are connected to each other, the relationships to the factors look counter
intuitive at first glance. “I am keen to improve,” for example increases the target-know gap and
the know-can do gap, but decreases the can-do gap. This is because, by increasing the two previous
-6
-4
-2
0
2
4
6
target-do target-know know-can can-do
T-s
tati
stic
(ab
ov
e 5
% c
uto
ff)
The four gaps
The role of motivation factors in 4 gaps (without facility effects)
I can make a difference
This hospital and management provide the right tools
In this organization, I am taken seriously
I am keen to improve
Feedback, teamwork, recognition
58
gaps, it results in a low level of capacity for the health worker and with a low level of capacity the
can-do gap is generally smaller.
Figure 11 Motivation factors and the four gaps with facility fixed effects
Overall, we see that two factors play a consistently important role in these gaps. “This hospital and
management provide the right tools” tends to reduce gaps and “In this organization I am taken
seriously” tends to increase the gaps. The interpretation of this finding is less straightforward. It is
interesting that positive perceptions about the role of the individual within the hospital do not play
a stronger role in improving performance. Rather, it is the perceptions that the hospital is
adequately supporting the health worker that improves performance.
B. Changes between 2013 Baseline Data Collection and
2015 Baseline Data Collection
-6
-4
-2
0
2
4
6
target-do target-know know-can can-do
T-s
tati
stic
s (a
bo
ve
5%
cu
toff
)
The four gaps
The role of motivation in 4 gaps (with facility fixed effects)
I can make a difference
This hospital and management provide the right tools
In this organization, I am taken seriously
I am keen to improve
Feedback, teamwork, recognition
59
The initial baseline was collected in four of the facilities (JF Doe, Redemption, Phebe and
Tellowayan) in 2013 and the follow up was collected in a total of 10 facilities in 2015. Initially,
the design was to use this second round of data collection as a mid-line review of results. However,
because of the intervening EVD crisis it is not possible to interpret the results as being caused by
the quality PBF program. Nonetheless is it is instructive to examine the changes that took place in
the four facilities over this gap.
Each table, with two exceptions, shown below examines a t-test of the difference in means for the
four repeated facilities from 2013 to 2015. For the obstetric and surgical direct observation, there
are not enough observations in each of the facilities to allow a test of changes across time and
therefore we combine all four hospitals in a before and after for the combination of treated
facilities. In each table, in addition to the actual difference, we report the p-value of that difference.
Differences below a p-value of 0.1 can be seen as statistically important, though of course the p-
value does not say why something has changed.
Table 19 Change in Average General Hospital Infrastructure Observation Scores, by Facility
Change in overall scores between 2014 and
2015
Indicators for General Observation of the Hospital
Infrastructure
JFD Rede
m
Phebe Tell Total
Average of necessary equipment present 0.15 0.33 0.11 0.04 0.16
Average of necessary equipment working 0.13 0.32 0.09 0.02 0.14
Proportion of infrastructure present that is not working 0.02 0.01 0.03 0.06 0.03
Note: *p<0.10, **p<0.05, ***p<0.001
Table 19 examines the changes in the summary infrastructure scores for each facility. In general
each facility has more equipment present than in 2013, but more of the equipment is not presently
working. However, overall there are small increases in the proportion of equipment that is working.
Table 20 examines the changes in motivation scores (with the factor weights fixed at the 2013
level). Across all facilities there is a significant switch from factor 1 to factor 2. Health workers
are significantly less likely to believe “I can make a difference” and significantly more likely to
believe “this hospital and management provide the right tools.” Although this might be caused by
60
the PBF program, it seems to align well with what we might expect from the experience of the
EVD crises, particularly since this was most acutely felt at Redemption and Phebe hospitals.
Health workers feel somewhat powerless in the face of such a crises, but now have come to believe
that resources will be made available to help them, if necessary.
Table 20 Change in Average Motivation Factor Scores, by Facility
Change in overall scores between 2014 and 2015
Motivation Survey Factors JFD Redem Phebe Tell Total
I can make a difference 0.18 -6.6*** -0.37* 0.05 -2.27***
This hospital and management provide the right tools -0.26 4.23*** 0.07 0.05 1.49***
In this organization, I am taken seriously -0.27 0.66*** -0.3 -0.26 0.05
I am keen to improve 0.19 -0.09 0.07 0.05 0.06
Feedback, teamwork, recognition -0.24 0.16 -0.27* 0.15 -0.01
Total Score -0.01 -1.47*** -0.16** 0.08 -0.47***
Note: *p<0.10, **p<0.05, ***p<0.001
Table 21 shows the changes across all four facilities in key measures of quality in the obstetric
ward. There are a number of significant changes over the two years in question, but no overall
pattern. History taking increases and the quality of observation at the second and third stage also
increases, but the quality at the first stage decreases. There are no significant changes in the use of
harmful practices over this period, although the final category, an increase in no harmful practices
observed (“none of the above”), is almost significant at the 10% level. Overall, it would be
dangerous to try and draw conclusions about the impact of the program (or EVD) from the changes
observed in this table.
61
Table 21 Change in Key Indicators for Obstetrics Ward Observation
Obstetrics Observation Category Change in overall scores between 2014
and 2015
p-value
Obstetric Intake: History Taking 0.29 0.08 Obstetric Intake: Presence of Complications 0.1 0.52 Obstetric Intake: Overall Quality 0.07 0.37 First Stage of Labor Observation Quality -0.1 0.06 Second and third stage observation quality 0.08 0.08 Episiotomy: Usually unnecessary -0.02 0.76 Newborn Resuscitation Quality -0.09 . Potentially Harmful Practices 0.01 0.72 Partograph Quality 0.01 0.98 Filling Partograph After Delivery 0.6 0.24 Obstetric Care PPH Quality 0.13 . Obstetric Care: Pre-Eclampsia Quality 0.02 0.45 Itemized harmful practices Use of enema -0.03 0.35 Pubic shaving 0.03 0.29 Woman routinely forced to push during second stage
of labor -0.03
0.35 Apply fundal pressure to hasten delivery of baby or
placenta -0.03
0.35 Lavage of uterus after delivery 0 0.95 Minor tears stitched when not bleeding -0.03 0.64 Episiotomy / tears not repaired with local anesthesia -0.09 0.1 Vagina swabbed with antiseptics after delivery -0.03 0.35 Disinfectant put on the perineum after delivery -0.06 0.18 Bladder catheterization performed postpartum 0.03 0.3 Cervix checked after delivery -0.03 0.64 Slap newborn 0 0.93 Hold newborn upside down 0.07 0.13 Milking the newborn's chest -0.03 0.35 Excessive stretching of the perineum -0.08 0.22 Shout, insult or threaten the woman during labor or
after 0
. Slap, hit or pinch the woman during labor or after 0 . None of the above 0.17 0.1
62
Table 22 examines the changes under direct observation in the pediatric ward over all four facilities
and Table 23 examines the same scores by facility. Here, the quality of care (as seen in physical
examination and the combination of physical examination and history taking) actually falls over
the period in question.
Table 22 Change in Key Indicators for Pediatrics Ward Observation
Pediatrics Observation Category Change in overall scores between 2014 and 2015
Pediatric Observation Physical Examination Score -0.2***
Pediatric Observation History Taking and Physical Exam -0.14***
Pediatric Observation Health Education Score 0.04
Pediatric Observation General Effort Score -0.02
Note: *p<0.10, **p<0.05, ***p<0.001
When we examine these numbers for each facility we see that the decrease is common to all four
facilities. In fact, no score increases over this period.
Table 23 Change in Key Indicators for Pediatrics Ward Observation, by Facility
Change in overall scores between 2014 and 2015
Pediatrics Observation Category JFD Redem Phebe Tell Total
Pediatric Observation Physical Examination Score -0.29*** -0.16*** -0.24*** -0.06* -0.2***
Pediatric Observation History Taking and Physical Exam -0.24*** -0.06 -0.2*** 0 -0.14***
Pediatric Observation Health Education Score 0.12*** 0.09 -0.06 -0.03 0.04
Pediatric Observation General Effort Score 0.04* -0.13*** -0.02 0.02 -0.02
Note: *p<0.10, **p<0.05, ***p<0.001
Table 24 examines the overall changes for the surgical direct observation. Here we see a number
of improvements across all four facilities. The preparation of the patient (obtaining charts,
histories, and consent) is improved and the team is much more likely to insure that the proper
protocol is followed for antibiotic prophylaxis. The quality of the surgery prep in general is also
improved and the cleanup both by the surgical team and the hospital staff is also improved. These
improvements are much more likely to be the result of the EVD crises than the PBF intervention.
Although at the time of the survey Liberia was declared Ebola free, the process of surgery is a
particularly sensitive and dangerous place likely to be impressed by recent memory.
63
Table 24 Change in Key Indicators for Surgery Ward Observation
Surgery Observation Category Change in overall scores between 2014 and 2015
Cleanliness of operating theater 0.07
Preparation of patient for surgery 0.13***
Surgeon preparation for surgery -0.05
Antibiotic prophylaxis given or confirmed 0.41***
General surgery quality 0.22***
The surgeon broke the surgical field during surgery 0.05
Medical staff post-surgery clean up 0.3***
Housekeeping staff post-surgery clean up 0.37***
Note: *p<0.10, **p<0.05, ***p<0.001
Table 25 takes advantage of the design of our data collection by examining quality in the vignettes.
These do not suffer when there are not enough patients and should show immediate gains in
competence (or capacity). Looking across all vignettes we slight improvements in the use of
history taking and the use of lab tests, but the gains are very small. Importantly, we see no evidence
that the drop in scores in the pediatric setting is due to a decline in knowledge about how to
properly diagnose cases.
Table 25 Change in Vignette Scores, by Facility
Change in overall scores between 2014 and 2015
Vignette Scores JFD Redem Phebe Tell Total
vignette history taking score -0.06 0.12* 0.04 0.1* 0.05*
vignette physical examination score -0.15*** -0.02 0 0.05 -0.02
vignette: proportion of lab tests 0.03 -0.21*** 0.16*** 0.05 0.04*
Correct Assessment of Condition -0.21 -0.44* -0.03 -0.04 -0.12
Correct overall diagnosis -0.04 -0.08 -0.01 0.08 -0.01
Correct detailed diagnosis -0.14 0.12 0.12 -0.14 0
Correct Treatment -0.09 0.09 0.07 -0.14 0
Note: *p<0.10, **p<0.05, ***p<0.001
Table 26 examines the changes in characteristics of patients reporting at facilities. We might see
important changes if there is a change in the perception of quality at the facility. Indeed we see
that patients are younger, less likely to have no schooling and less likely to report being unable to
64
read. Patients are not wealthier however, as measured by the index of assets owned. In addition,
patients at Phebe and Tellowayan are less likely to have traveled great distances to seek care
(although there is no overall difference across all facilities).
Table 26 Change in Patient Characteristics for All Patient Exit Interviews, by Facility
Change in overall scores between 2014 and 2015
Pediatrics Observation Category
JFD Redem Phebe Tell Total
Age -6.47*** 1.58 -2.82 -2.48 -2.1*
Patients without any schooling -0.43*** -0.02 -0.02 0.01 -0.11***
Patients completed primary schooling 0.31*** 0.08 -0.06 -0.01 0.07*
Patients completed secondary
schooling
0.06 -0.06 0.05 0.04 0.03
Patients completed vocational training 0 -0.01 0 -0.01 0
Patients completed post-secondary
schooling
0.04 0.01 0 -0.02 0.01
Patients reporting being able to read
easily
0.11 -0.02 0.12* -0.05 0.04
Patients reporting being able to read
with difficulty
0.31*** -0.01 -0.07 0.17* 0.08*
Patients reporting not being able to
read at all
-0.43*** 0.03 -0.05 -0.05 -0.12***
Travel Time: Less than 30 minutes -0.06 -0.05 0.11 0.25** 0.04
Travel time: 30 - 60 minutes -0.03 -0.02 0.16 -0.02 0.01
Travel Time: More than 60 Minutes 0.03 0.07 -0.29** -0.23** -0.07
Index of Assets Owned 0.36 -0.11 0.23 0.06 0.15
Note: *p<0.10, **p<0.05, ***p<0.001
65
Table 27 examines the changes in the satisfaction of patients by facility. Here we see that, overall
the patients are more pleased with the quality and convenience of facility and have a greater overall
opinion. They are slightly less likely to feel that they have received the right amount of individual
attention.
Table 27 Change in Satisfaction Summary Scores for All Patient Exit Interviews, by Facility
Change in overall scores between 2014 and 2015
Pediatrics Observation Category JFD Redem Phebe Tell Total
Quality -0.24 1.18*** 1.22*** 1.6*** 1.1***
Convenience -0.25 0.48*** -0.03 0.08 0.27***
Average Likert scale score -0.01 0.28*** 0.14 0.07 0.14***
Important: overall satisfaction -0.38 -0.21 0.06 1.47* 0.28
Important: individual attention -1.98 -0.02 -0.03 -0.32 -0.3**
Note: *p<0.10, **p<0.05, ***p<0.001
V. Conclusion
This report discusses the data collected in the spring of 2015, evaluating the quality of 10 hospitals
providing care in Liberia. Although there are important differences between the hospitals studied,
the report focuses on validating the measures of quality collected and examining the changes that
have been seen in the past two years. Overall, we find that low performance of health workers in
the collection of hospitals is linked to a large can-do gap: the difference between what a health
worker is capable of doing and what he actually does. There exists a significant knowledge gap in
all of the facilities observed, but since there is very little connection between knowledge and
practice, it seems unlikely that improving knowledge will improve quality. Our measures of quality
show that there is much room to improve, but also show that the quality based PBF might find
fertile ground to make progress for exactly this reason.
We also examine the evidence for changes to date in the hospitals that are part of the program.
Here we find little statistically significant evidence of changes. Some things have improved, others
66
have not. If we could associate all of the positive changes observed to the implementation of the
program, the results would still be disappointing. However, due to the intervening EVD crises and
the lack of a proper control, we cannot associate even these modest gains with the implementation
of the program.
Fortunately, we now have a more robust identification strategy through the inclusion of five control
facilities. Although Redemption hospital does stand out in some categories, these five control
facilities span the range of characteristics seen in the other four treatment facilities and will help
us to establish, more precisely the changes we expect to see in the next round of evaluation.
67
VI. Appendices A. Acronyms
EVD Ebola Virus Disease
GMRP Graduate Medical Residency Programme
GoL Government of Liberia
HSSP Health Systems Strengthening Project
MCH Maternal and Child Health
MDG Millennium Development Goal
PGMC Post Graduate Medical Council
PGMC Post Graduate Medical Council
PBF Performance-Based Financing
PPE Personal Protective Equipment
MOHSW Ministry of Health and Social Welfare
WHO World Health Organization
68
B. Data Collection Schedule
Table 28 Data Collection Schedule
Departure Date Arrival Date Hospital for data
collection
Date for data collection at
each Hospital
Group 1
Leave from
Monrovia on 14
June
15 June F.J. Grante (Sinoe)
16 June – 20 June
21 June 21 June St.Francis Hospital
(Rivercess)
22 June –26 June
27 June 28 June C.H.Rennie (Margibi) 29 June – 3 July
5 July 5 July Liberian Government
Hospital (Bomi)
6 – 10 July
Leave for
Monrovia on 11
July
11 July
Redemption
(Montserrado)
13—17 July
Take off from MoH by 7am
daily.
Arrive at Redemption by 8am
daily.
Group II
14 June 14 June J.F.Doe (Nimba)
15—19 June
20 June 20 June G.W.Harley (Nimba)
22—26 June
27 June 27 June Tellewoyan (Lofa) 29 June – 3 July
4 July 4 July Curren Hospital (Lofa)
6 – 10 July
11 July 11 July Phebe Hospital (Bong)
13 – 17 July
Depart from
Bong to
Monrovia
18 -- 19 July
69
C. Quality Assessment Tool
Table 29 Management and Structural Checklist Overview
Category Component
I. Management 1. General Management
2. Human Resources for Health
II. Structural
3. Hygiene and medical waste disposal management
4. Drugs Management
5. Equipment and Supplies
Table 30: Quality Checklist Overview
Service-delivery
Type Priority Clinical Focus Areas
Childbirth:
Maternal-
Newborn
(intra- and post-
partum)
Routine intra- and post-partum high-impact care:
Intra-partum (labor/immediate post-partum)Post-partum (mother &
newborn):
Maternal Complications:
1. Obstructed Labor
2. Hemorrhage
3. Sepsis
4. Eclampsia
Neonatal Complications:
5. Asphyxia,
6. Sepsis,
7. Prematurity
Pediatric
(in-patient care)
8. Maternal Newborn Best Practices Chart review
9. ETAT: Emergency Triage, initial Assessment & Treatment
(Routine triage & initial stabilization all children presenting to hospital)
10. Malaria
11. Pneumonia
12. Acute Diarrhea
13. Severe Acute Malnutrition
Other: Pediatric Outcome and Processes
(Note: Neonatal Sepsis included above)
Surgical Care
14. WHO Surgical Safety Checklist (based on modified WHO content)
-Pre-operative assessment of patient
70
(General Surgery
& Caesarian)
-Anesthesia assessment & safety planning
-Management of intra-operative complications (e.g. bleeding, breathing,
cardiovascular (BP, arrhythmias, etc)
-Post-operative care
-Management of intra-operative complications
Table 31: Quantity of Services
PBF Services Definition
Quality of complicated and assisted
pregnancy and delivery (including C-
section)
Any labor that is made more difficult or complex by a
deviation from the normal procedure. Complicated
delivery is defined as: assisted vaginal deliveries
(vacuum extraction or forceps), C-section, episiotomy
and other procedures.
Quantity of normal deliveries for at
risk referrals
High-risk pregnant women referred by health center to
the hospital but delivered normally. A high-risk
pregnancy is defined as: evidence of edema, mal
presentation, increased BP, multi-parity, etc.
Quantity of counter referral letters
returned to health centers
Hospital returns counter referrals letter with feedback
on the referred patient to the referring health center. The
counter referral letter is completed in triplicate, with
one also given to the patient, and one retained by the
hospital.
Quantity of newborns referred for
emergency neonatal care treatment
Newborns referred for emergency neonatal care due to:
perinatal complications, low birth weight, congenital
malformation, asphyxia, etc.
Quantity of Referred under-fives with
fever
Infants and under-fives with fever who were referred to
the hospital for management of Malaria and Pneumonia.
Quality of Minor surgical intervention Any surgical procedure that does not involve anesthesia
or respiratory assistance.
Quality of major surgery (excluding
CS, including major trauma)
Any surgery in which the patient must be put under
general spinal/anesthesia and given respiratory
assistance. Major surgery in the case of this package of
services is defined as any of the following:
Herniarraphy, Appendectomy, Myomectomy,
Sleenectomy, Salpingectomy, Hysterectomy,
Thyrodectomy, and Mastectomy.
Quantity of patients transported by
ambulance
Patients transferred from a lower-level facility (health
center or health clinic) to the hospital for emergency
treatment.
71
D. Instrument Summaries
In the following sections, we look at the scores for each vignette in turn.
1. Surgical Vignettes
Table 32: Surgery Vignette Scores by Facility (S_V_1_A)
Facility
JFD Redem JFD St. F JFD LGH JFD Curr JFD GWH
Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N
Surgical vignette history
taking score
0.47 10 0.44 16 0.45 11 0.32 7 0.31 9 0 0 0.63 2 0.42 3 0.63 4
Surgical vignette
physical examination
score
0.52 10 0.56 16 0.44 11 0.54 7 0.29 9 0 0 3.3 2 0.53 3 0.5 4
Surgical vignette:
proportion of lab tests
0.25 10 0.26 16 0.22 11 0.27 7 0.21 9 0 0 0.61 2 0.37 3 0.33 4
Mean Capacity Vignette
Score (average of items
correct and available)
0.16 6 0.16 16 0.15 8 0.29 4 0.09 7 0 0 0 0 0
Mean Capacity
Competence Gap
(competence-capacity)
0.06 6 0.1 16 0.06 8 0.08 4 0.1 7 0 0 0 0 0
Correct Assessment of
Condition: Stable
0.91 11 0.3 20 0.82 11 0.67 9 0.89 9 0 0 1 3 0.67 3 0.75 4
Correct overall diagnosis:
hernia
0.73 11 0.7 20 0.64 11 0.67 9 0.44 9 0 0 0 3 1 3 0.5 4
72
Correct detailed
diagnosis: Inguinal
Direct or Indirect Hernia
0.64 11 0.3 20 0.64 11 0.78 9 0.44 9 0 0 1 3 1 3 1 4
Correct Treatment:
Elective Herniorrhaypy
0.71 7 0.18 11 0.63 8 0.5 8 1 1 0 0 0 1 1 0 1
Surgical vignette:
proportion of rational lab
tests
0.38 10 0.38 16 0.16 11 0.33 7 0.29 9 0 0 0.8 2 0.53 3 0.4 4
Surgical vignette:
proportion of irrational
lab tests
0.26 11 0.28 20 0.25 11 0.31 9 0.17 9 0 0 1.44 3 0.22 3 0.25 4
Table 33: Surgery Vignette Scores by Facility (S_V_1_B)
Facility
JFD Redem JFD St. F JFD LGH JFD Curr JFD GWH
Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N
Surgical vignette history
taking score 0.54 7 0.28 8 0.42 10 0.35 5 0.27 12 0.33 3 0 0 0.25 3 0
Surgical vignette
physical examination
score 0.68 7 0.56 8 0.4 10 0.5 5 0.46 12 0.92 3 0 0 0.5 3 0
Surgical vignette:
proportion of lab tests 0.3 7 0.31 8 0.2 10 0.3 5 0.19 12 0.41 3 0 0 0.15 3 0
Mean Capacity Vignette
Score (average of items
correct and available) 0.16 2 0.17 8 0.13 8 0.26 5 0.12 7 0 0 0 0 0
73
Mean Capacity
Competence Gap
(competence-capacity) 0.03 2 0.15 8 0.03 8 0.04 5 0.05 7 0 0 0 0 0
Correct Assessment of
Condition: Unstable 0.63 8 0.7 10 0.4 10 0.67 6 0.67 12 0.33 3 0 0 0 3 0
Correct overall diagnosis:
hernia 0.75 8 0.5 10 0.5 10 0.33 6 0.33 12 1 3 0 0 0.33 3 0
Correct detailed
diagnosis: Femoral
Hernia 0.5 8 0.4 10 0.3 10 0.67 6 0.5 12 0.33 3 0 0 0.33 3 0
Correct Treatment:
Monitor, Emergency
Surgery 0.67 6 0.83 6 0.75 4 0.83 6 1 3 1 3 0 0 1 1 0
Dangerous Treatment:
Discharge 0.25 8 0.5 10 0.7 10 0.33 6 0.75 12 0 3 0 0 0.67 3 0
74
Table 34: Surgery Vignette Scores by Facility (S_V_2_A)
Facility
JFD Redem JFD St. F JFD LGH JFD Curr JFD GWH
Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N
Surgical vignette history
taking score 0.29 6 0.44 8 0.19 9 0.38 6 0.45 11 0.67 3 0 0.25 2 0.25 2 1 1
Surgical vignette
physical examination
score 0.43 6 0.48 8 0.42 9 0.6 6 0.4 11 0.67 3 0 0.4 2 0.4 2 1 1
Surgical vignette:
proportion of lab tests 0.17 6 0.16 8 0.13 9 0.32 6 0.17 11 0.4 3 0 0.4 2 0.1 2 0.8 1
Mean Capacity Vignette
Score (average of items
correct and available) 0.13 3 0.16 6 0.06 7 0.23 5 0.09 7 0 0 0 0 0
Mean Capacity
Competence Gap
(competence-capacity) 0.02 3 0.05 6 0.02 7 0.07 5 0.01 7 0 0 0 0 0
Correct Assessment of
Condition: Stable 0.25 8 0.45 11 0.11 9 0.29 7 0.36 11 0.33 3 0 0.5 2 0 2 1 1
Correct overall diagnosis:
hernia 0.88 8 0.91 11 1 9 1 7 0.91 11 1 3 0 1 2 0.5 2 0 1
Correct detailed
diagnosis: Inguinal
Direct Hernia 0.38 8 0.27 11 0 9 0.29 7 0 11 0.33 3 0 0 2 0 2 0 1
Correct Treatment:
Elective Herniorrhaypy 0.29 7 0.09 11 0 5 0.29 7 0 4 0.33 3 0 0 0 1 0 1
75
Table 35: Surgery Vignette Scores by Facility (S_V_2_B)
Facility
JFD Redem JFD St. F JFD LGH JFD Curr JFD GWH
Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N
Surgical vignette history
taking score 0.41 11 0.52 11 0.42 12 0.25 8 0.28 9 0 0 0.75 4 0.25 3 0.5 4
Surgical vignette
physical examination
score 0.55 11 0.47 11 0.43 12 0.6 8 0.33 9 0 0 0.5 4 0.27 3 0.45 4
Surgical vignette:
proportion of lab tests 0.21 11 0.25 11 0.21 12 0.3 8 0.11 9 0 0 0.35 4 0 3 0.25 4
Mean Capacity Vignette
Score (average of items
correct and available) 0.17 6 0.2 11 0.1 9 0.29 5 0.1 6 0 0 0 0 0
Mean Capacity
Competence Gap
(competence-capacity) 0.01 6 0.05 11 0.02 9 0 5 0 6 0 0 0 0 0
Correct Assessment of
Condition: Stable 0.67 12 0.45 11 0.42 12 0.33 9 0.56 9 0 0 0.25 4 0 3 0.5 4
76
Correct overall diagnosis:
hernia 0.92 12 0.82 11 0.75 12 0.89 9 0.78 9 0 0 0.75 4 0.33 3 0.75 4
Correct detailed
diagnosis: Inguinal
Direct Hernia 0.25 12 0.18 11 0 12 0.11 9 0 9 0 0 0 4 0 3 0 4
Correct Treatment:
Elective Herniorrhaypy 0 11 0.1 10 0 8 0.11 9 0 3 0 0 0 2 0 1 0 1
77
Table 36: Surgery Vignette Scores by Facility (S_V_3)
Facility
JFD Redem JFD St. F JFD LGH JFD Curr JFD GWH
Avg N Avg N Avg N Avg N Avg N Avg N Av
g
N Avg N Avg N Avg N
Surgical vignette history
taking score 0.53 9 0.42 17 0.42 15 0.45 5 0.4 15 0.67 3
0.7
5 1 0.5 6 0.2 5 0.42 3
Surgical vignette
physical examination
score 0.49 9 0.4 17 0.35 15 0.6 5 0.36 15 0.6 3 0.8 1 0.37 6 0.44 5 0.33 3
Surgical vignette:
proportion of lab tests 0.08 9 0.04 17 0.11 15 0.18 5 0.05 15 0.07 3 0.2 1 0.2 6 0.04 5 0.15 3
Mean Capacity Vignette
Score (average of items
correct and available) 0.3 1 0.03 14 0.05 10 0 0.05 9 0 0 0 0 0
Mean Capacity
Competence Gap
(competence-capacity) 0 1 0.02 14 0.02 10 0 0 9 0 0 0 0 0
Correct overall diagnosis:
hernia 0.61 18 0.43 21 0.53 15 0.4 15 0.53 15 0.33 3 1 1 0.83 6 0.8 5 0.67 3
Correct Treatment:
Elective Herniorrhaypy 0 6 0 8 0 11 0.38 8 0 8 0.5 2 0 0 4 0 4 0 1
78
Table 37: Surgery Vignette Scores by Facility (S_V_5_A)
Facility
JFD Redem JFD St. F JFD LGH JFD Curr JFD GWH
Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N
Surgical vignette history
taking score 0.34 8 0.35 10 0.4 12 0.42 9 0.35 10 0 0 0 0.25 3 0.5 2
Surgical vignette
physical examination
score 0.48 8 0.44 10 0.53 12 0.67 9 0.32 10 0 0 0 0.47 3 0.5 2
Surgical vignette:
proportion of lab tests 0.17 8 0.21 10 0.19 12 0.19 9 0.09 10 0 0 0 0.17 3 0.17 2
Mean Capacity Vignette
Score (average of items
correct and available) 0.13 4 0.17 10 0.12 9 0.17 6 0.1 7 0 0 0 0 0
Mean Capacity
Competence Gap
(competence-capacity) 0.05 4 0.05 10 0.06 9 0.03 6 0.01 7 0 0 0 0 0
Correct Assessment of
Condition: Stable 0.33 9 0.73 11 0.42 12 0.36 11 0.4 10 0 0 0 0 3 0.5 2
Correct overall diagnosis:
hernia 0.44 9 0.27 11 0.17 12 0.64 11 0.1 10 0 0 0 0.33 3 0.5 2
Correct Treatment:
Elective Herniorrhaypy 0.17 6 0 10 0 11 0 11 0 7 0 0 0 0 3 0 2
79
Table 38: Surgery Vignette Scores by Facility (S_V_5_B)
Facility
JFD Redem JFD St. F JFD LGH JFD Curr JFD GWH
Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N
Surgical vignette history
taking score 0.38 6 0.44 9 0.17 9 0.18 7 0.47 9 0.42 3 0.25 1 0.25 2 0.13 2 0.25 1
Surgical vignette
physical examination
score 0.7 6 0.6 9 0.47 9 0.51 7 0.42 9 0.73 3 0.6 1 0.5 2 0.2 2 0.2 1
Surgical vignette:
proportion of lab tests 0.29 6 0.15 9 0.17 9 0.18 7 0.15 9 0.4 3 0.4 1 0.3 2 0 2 0.2 1
Mean Capacity Vignette
Score (average of items
correct and available) 0.1 2 0.13 5 0.1 7 0.21 6 0.08 5 0 0 0 0 0
Mean Capacity
Competence Gap
(competence-capacity) 0.07 2 0.06 5 0.03 7 0 6 0 5 0 0 0 0 0
Correct Assessment of
Condition: Stable 0.43 7 0.42 12 0.44 9 0.75 8 0.44 9 0.67 3 0 1 0.5 2 0 2 1 1
Correct overall diagnosis:
hernia 0.71 7 0.75 12 0.56 9 0.88 8 0.89 9 1 3 1 1 1 2 0.5 2 1 1
Correct Treatment:
Elective Herniorrhaypy 0.83 6 0.42 12 0.17 6 0.75 8 0 6 0 3 0 1 0.5 2 1 1 0
80
2. Pediatric Vignettes
Table 39: Pediatric Vignette Scores by Facility (P_V_1)
Facility
JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH
Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N
Correct treatment: refer,
IV or naso-gastric
feeding plus ORS 0.81 31 0.68 22 0.66 37 0.7 25 0.63 27 0.67 6 0.6 5 0.75 8 0.61 9 0.61 9
Correct and equipment
available 0.74 31 0.64 22 0.51 37 0.62 25 0.56 27 0.58 6 0.5 5 0.5 8 0.28 9 0.56 9
Mean Capacity
Competence Gap
(competence-capacity) 0.06 31 0.05 22 0.15 37 0.08 25 0.07 27 0.08 6 0.1 5 0.25 8 0.33 9 0.06 9
81
Table 40: Pediatric Vignette Scores by Facility (P_V_2)
Facility
JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH
Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N
Average proportion of
history taking items 0.41 53 0.4 34 0.31 57 0.33 38 0.28 43 0.35 6 0.41 5 0.36 8 0.28 9 0.29 10
Average proportion of
physical examination
items 0.53 53 0.45 34 0.48 57 0.47 38 0.33 43 0.49 6 0.43 5 0.46 8 0.34 9 0.61 10
Diagnosis correct:
Malaria + anemia 0.43 53 0.21 34 0.46 57 0.5 38 0.23 43 0.33 6 0.2 5 0.13 8 0.22 9 0.1 10
Diagnosis partially
correct: Malaria 0.53 53 0.68 34 0.51 57 0.45 38 0.7 43 0.67 6 0.8 5 0.75 8 0.78 9 0.7 10
Diagnosis wrong:
Meningitis 0.04 53 0.09 34 0.04 57 0.05 38 0.09 43 0 6 0 5 0 8 0 9 0 10
Capacity to perform
physical examination 0.5 53 0.45 34 0.48 57 0.43 38 0.33 43 0.47 6 0.42 5 0.43 8 0.33 9 0.57 8
Physical Examination
Competence Capacity
Gap 0.03 53 0 34 0.01 57 0.03 38 0.01 43 0.01 6 0.02 5 0.03 8 0.01 9 0.02 8
82
Table 41: Pediatric Vignette Scores by Facility (P_V_3)
Facility
JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH
Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N
Average proportion of
history taking items 0.39 28 0.43 21 0.33 35 0.33 25 0.3 31 0.29 6 0.32 5 0.27 8 0.32 9 0.38 11
Average proportion of
physical examination
items 0.56 28 0.6 21 0.47 35 0.5 25 0.44 31 0.39 6 0.56 5 0.43 8 0.45 9 0.44 11
Diagnosis correct:
Acute Diarrhea 0.82 28 0.81 21 0.91 35 0.92 25 0.94 31 0.67 6 1 5 0.75 8 0.78 9 0.91 11
Diagnosis partially
correct: Dehydration,
dysentery,
gastroenteritis 0.07 28 0.05 21 0 35 0 25 0.03 32 0 6 0 5 0 8 0 9 0 11
Diagnosis wrong:
malaria 0.57 28 0.48 21 0.4 35 0.24 25 0.48 31 0.5 6 0.2 5 0.75 8 0.44 9 0.64 11
Diagnosis: Capacity to
perform physical
examination 0.52 28 0.6 21 0.46 35 0.48 25 0.41 31 0.39 6 0.56 5 0.38 8 0.43 9 0.4 11
Diagnosis: Physical
Examination
Competence Capacity
Gap 0.04 28 0 21 0.01 35 0.03 25 0.02 31 0 6 0 5 0.06 8 0.02 9 0.03 11
Treatment correct: ORS
or IV fluids 0.93 28 0.76 21 0.73 33 0.72 25 0.68 31 0.67 6 0.6 5 0.63 8 0.89 9 0.5 10
Treatment partially
correct 0.5 28 0.43 21 0.42 33 0.32 25 0.52 31 0.5 6 0.2 5 0.63 8 0.33 9 0.82 11
83
Treatment wrong: only
Anti-malarials and/or
PCM 0.61 28 0.57 21 0.67 33 0.8 25 0.87 31 0.67 6 0.4 5 0.88 8 0.78 9 0.45 11
Treatment dangerous:
antidiarrheal 0.46 28 0.38 21 0.42 33 0.4 25 0.52 31 0.67 6 0.4 5 0.88 8 0.67 9 0.45 11
Treatment increment:
prescribing zinc 0 28 0.05 21 0.09 35 0 25 0.13 32 0 6 0 5 0 8 0 9 0 11
Health Education:
Proportion of good
health education done 0.59 28 0.54 21 0.46 35 0.51 24 0.44 31 0.4 6 0.48 5 0.65 8 0.36 9 0.61 11
84
Table 42: Pediatric Vignette Scores by Facility (P_V_4)
Facility
JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH
Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N
Average proportion of
history taking items 0.41 23 0.43 21 0.37 36 0.33 25 0.29 26 0.39 6 0.2 5 0.28 8 0.3 9 0.28 10
Average proportion of
physcial examination
items 0.45 23 0.32 21 0.36 36 0.38 25 0.28 27 0.34 6 0.19 5 0.26 8 0.4 9 0.23 10
Capacity to perform
physcial examination 0.43 19 0.3 21 0.34 36 0.35 25 0.26 24 0.32 6 0.15 5 0.18 8 0.37 8 0.23 9
Diagnosis: Physical
Examination
Competence Capacity
Gap 0.01 19 0.02 21 0.03 36 0.03 25 0.03 24 0.02 6 0.04 5 0.08 8 0.02 8 0.01 9
85
3. Obstetric Vignettes
Table 43: Obstetrics Vignette Scores by Facility (O_V_1_A)
Facility
JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH
Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N
Mean Vignette Score
(average of items
correctly
used/mentioned) 0.5 2 0.52 6 0.71 11 0.58 3 0.55 4 0.41 3 0.51 9 0.71 4 0.52 7 0.67 10
Mean Capacity Vignette
Score (average of items
correct and available) 0 0.45 4 0.88 1 0.94 1 0 0 0.62 1 0.68 3 0.51 6 0
Mean Capacity
Competence Gap
(competence-capacity) 0 0.03 4 0 1
-
0.24 1 0 0 0.03 1 0 3 0.02 6 0
86
Table 44: Obstetrics Vignette Scores by Facility (O_V_1_B)
Facility
JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH
Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N
Partially Correct
Diagnosis (Pre-
eclampsia) 0.67 6 0.5 4 0.33 6 0 4 0.75 8 0.4 5 0.15 13 0.4 5 0.43 7 0.33 3
Correct Diagnosis
(Severe Pre-eclampsia) 0.17 6 0 4 0.67 6 0.25 4 0.13 8 0.4 5 0.46 13 0.4 5 0.14 7 0 3
average of correct
treatment responses
(MgSO4,
antihypertensive,
document) 0.72 6 0.25 4 0.61 6 0.33 4 0.58 8 0.47 5 0.49 13 0.47 5 0.39 6 0.44 3
Correct Treatment
(stabilize with MgSO4) 0.67 6 0.5 4 0.83 6 0.25 4 0.63 8 0.6 5 1 13 0.6 5 0.43 7 0.67 3
Correct response to
convulsions (MS,
oxygen and protect from
injury or lay on si 0.5 6 0 4 0.5 6 0.25 4 0.25 8 0 5 0 13 0.4 5 0.29 7 0.33 3
Wrong response to
convulsions (Diazepam,
antihypertensives or
active restraint) 1 6 1 4 1 6 0.5 4 0.88 8 1 5 0.85 13 1 5 1 7 1 3
Number of essential
items listed (a, b, c, f, g,
h, i) 0.69 6 0.44 4 0.71 6 0.09 4 0.45 8 0.55 5 0.46 13 0.65 5 0.42 6 0.71 3
Correct response to
stabilization (repeat 0.64 6 0.33 4 0.5 6 0.17 2 0.23 8 0.37 5 0.33 13 0.43 5 0.33 6 0.58 3
87
MgSO4, partograph,
record, reflexes)
88
Table 45: Obstetrics Vignette Scores by Facility (O_V_2_A)
Facility
JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH
Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N
Mean Vignette Score
(average of items
correctly
used/mentioned) 0.71 9 0.61 8 0.72 18 0.71 9 0.65 12 0.7 3 0.46 7 0.76 4 0.44 9 0.74 10
Mean Capacity Vignette
Score (average of items
correct and available) 0.76 4 0.71 2 0.66 9 0.74 6 0.55 7 0 0 0 0 0
Essential items: open
airway, suction,
ventilate 0.67 9 0.38 8 0.67 18 0.89 9 0.75 12 0.67 3 0 7 0.75 4 0.44 9 0.7 10
Mean availability
score(average of
equipment available) 2.44 9 1 8 2.33 18 3.33 9 2.25 12 0 3 0 7 0 4 0 9 0 10
Mean Capacity
Competence Gap
(competence-capacity) 0 4 0.07 2 0.03 9 0.01 6 0.03 7 0 0 0 0 0
89
Table 46: Obstetrics Vignette Scores by Facility (O_V_2_B)
Facility
JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH
Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N
Health worker follows
correct process for
resuscitation 0.83 10 0.67 6 0.7 11 1.49 8 1.33 13 0.44 4 0.43 7 0.76 5 2.26 7 2.97 3
Health worker is able to
properly ventilate on
first try 0.5 10 0.5 6 0.45 11 0.29 7 1.46 13 0.25 4 0.14 7 0.8 5 0.43 7 0.5 2
Health worker is able to
properly ventilate on
second try 0.33 6 1 4 0.67 6 0.43 7 3.67 3 0.33 3 0 6 1 2 0.25 4 0 1
Health worker cannot
properly ventilate 0.3 10 0.17 6 0.18 11 0.25 8 0 13 0.5 4 0.71 7 0 5 0.43 7 0.33 3
90
Table 47: Obstetrics Vignette Scores by Facility (O_V_3)
Facility
JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH
Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N
Mean Vignette Score
(average of items
correctly
used/mentioned) 0.69 18 0.61 23 0.72 18 0.53 16 0.96 22 0.62 8 0.53 13 0.69 10 0.48 15 0.67 13
Mean availability
score(average of
equipment available) 0.21 18 0.21 23 0.21 18 0.22 16 0.18 22 0 8 0 13 0 10 0 15 0 13
Mean Capacity Vignette
Score (average of items
correct and available) 0.51 18 0.46 23 0.5 18 0.43 15 0.42 19 0.38 8 0.32 13 0.4 8 0.28 15 0.35 10
Mean Capacity
Competence Gap
(competence-capacity) 0.18 18 0.15 23 0.21 18 0.11 15 0.15 19 0.24 8 0.21 13 0.3 8 0.21 15 0.29 10
91
Table 48: Obstetrics Vignette Scores by Facility (O_V_4_A)
Facility
JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH
Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N
Mean Vignette Score
(average of partograph
items correct) 0.68 9 0.65 14 0.81 20 0.8 8 0.56 11 0.15 2 0.71 6 0.85 4 0.51 8 0.84 9
92
Table 49: Obstetrics Vignette Scores by Facility (O_V_4_B)
Facility
JFD Redem Phebe St. F Tell LGH CHR Curr FJG GWH
Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N Avg N
Mean Vignette Score
(average of partograph
items correct) 0.53 5 0.45 9 0.78 13 0.58 10 0.65 15 0.41 5 0.62 6 0.89 5 0.33 7 0.68 3