Measuring What Counts in HIS - Balanced Scorecards

international journal of health planning and management

Int J Health Plann Mgmt 2010; 25: 74–90.

Published online in Wiley InterScience

(www.interscience.wiley.com) DOI: 10.1002/hpm.1004

Designing a balanced scorecard for a tertiarycare hospital in Pakistan: a modified Delphigroup exercise

Fauziah Rabbani1,2*, Syed M. Wasim Jafri3y,z, Farhat Abbas4x,�,Mairaj Shah5k,#, Syed Iqbal Azam1yy, Babar Tasneem Shaikh1yy,Mats Brommels6,7zz,xx,�� and Goran Tomson2,7kk,##

1Department of Community Health Sciences, Aga Khan University, Karachi, Pakistan2Department of Public Health Sciences, IHCAR Div International Health, KarolinskaInstitutet, Stockholm, Sweden3Department of Medicine and Department of Continuing Professional Education , Aga KhanUniversity, Karachi, Pakistan4Department of Surgery, Aga Khan University, Karachi, Pakistan5Aga Khan University Hospital, Karachi, Pakistan6Department of Public Health, University of Helsinki, Finland7Medical Management Centre at Karolinska Institutet, Stockholm, Sweden

SUMMARY

Balanced Scorecards (BSC) are being implemented in high income health settings linkingorganizational strategies with performance data. At this private university hospital in Pakistanan elaborate information system exists. This study aimed to make best use of available data forbetter performance management. Applying the modified Delphi technique an expert panel ofclinicians and hospital managers reduced a long list of indicators to a manageable size.Indicators from existing documents were evaluated for their importance, scientific soundness,appropriateness to hospital’s strategic plan, feasibility and modifiability. Panel membersindividually rated each indicator on a scale of 1–9 for the above criteria. Median scores wereassigned. Of an initial set of 50 indicators, 20 were finally selected to be assigned to the four

*Correspondence to: Dr F. Rabbani, Professor Dept of Community Health Sciences, PO BOX 3500,Stadium Road, Karachi, Pakistan and doctoral student Dept of Public Health Sciences IHCAR DivInternational Health, Nobels vag 9, Karolinska Institutet, SE 171 77 Stockholm, Sweden.E-mails: [email protected]; [email protected] of Medicine.zAssociate Dean.xProfessor in the Section of Urology.�The COO (Chief Operating Officer—on site).kManager Clinical Affairs.#CME.yyAssistant Professor.zzProfessor of Health Services Management.xxGuest Professor.��Director.kkProfessor.##Director.

Copyright # 2010 John Wiley & Sons, Ltd.

DESIGNING BSC USING MODIFIED DELPHI 75

BSC quadrants. These were financial (n¼ 4), customer or patient (n¼ 4), internal business orquality of care (n¼ 7) and innovation/learning or employee perspectives (n¼ 5). A need forstringent definitions, international benchmarking and standardized measurement methods wasidentified. BSC compels individual clinicians and managers to jointly work towards improvingperformance. This scorecard is now ready to be implemented by this hospital as a performancemanagement tool for monitoring indicators, addressing measurement issues and enablingcomparisons with hospitals in other settings Copyright # 2010 John Wiley & Sons, Ltd.

key words: balanced scorecard; performance management; indicators; modified Delphi;

private hospital in Pakistan

INTRODUCTION

Hospital performance assessment is becoming increasingly important for different

stakeholders such as health care providers, decision makers, and purchasers of

health care. This is in response to growing demands to ensure transparency,

control, and reduce variations in clinical practice (Groene et al., 2008). With

hospitals consuming more than half of overall health care budget (McKee et al.,

2002), recent hospital reforms are highlighting a quest for achieving more efficient

and effective hospital care. This can be achieved through generalizable,

standardized interpretable, and useable information for clinicians or health service

managers (Willis et al., 2008).

Hospitalmanagement teams receivevoluminous information fromawidevariety of

sources. Despite thewidespread use of performance indicators, there is little research

evidence on how to select the essential data to make evidence-informed decision

making (Ovretveit and Al Serouri, 2006). The worldwide health community,

therefore, needs to focus on improving measurement of a small set of priority areas

(Murray, 2007). Formal consensus methods are a set of techniques that synthesize

expert or stake holder opinion to guide and prioritize group decisions in situations

where information is lacking, contradictory or where there is an overload of

information (Campbell et al., 2002). Threemainmethods have been used in the health

field: the Delphi, the Nominal Group Technique and the Consensus Development

Conference. The comparative advantage of the Delphi technique over other strategies

is the enhanced opportunity for all participants to contribute greater number of ideas

than other group processes, minimizing domination of the process by more confident

or outspoken individuals, the ease of interpreting the results (ideas aregenerated, voted

on/ranked, aggregated, and evaluated at the session itself), a greater sense of

accomplishment formembers (results are available immediately after the session), and

minimal resource requirements with efficient use of time (Murphy et al., 1998).

Conceptualization of hospital functioning is a diverse and complex phenomenon.

WHO strategic orientations are encompassed into six interrelated dimensions:

clinical effectiveness, safety, patient centeredness, responsive governance, staff

orientation, and efficiency (Veillard et al., 2005). Though no performance

management tool is ideal, this multidimensional approach of hospital performance

is captured in the balanced scorecard (BSC), in four different perspectives with an

equal weightage: (i) labeled learning and growth (staff orientation and satisfaction),

(ii) internal processes (clinical outcomes and management of health services),

Copyright # 2010 John Wiley & Sons, Ltd. Int J Health Plann Mgmt 2010; 25: 74–90.

DOI: 10.1002/hpm

76 F. RABBANI ET AL.

(iii) customer (patient) satisfaction, and (iv) financial efficiency/performance

(Castaneda-Mendez et al., 1998). BSC serves as a dashboard for meaningful decision

making and quality improvement and relates results to external references while

promoting internal comparisons overtime (Veillard et al., 2005).

The advantage that BSC has over other performance measurement tools is that it is

less of a diagnostic control system for highlighting abnormal activities and more of

an interactive system for providing signals to the organization about management

objectives, stimulating debate, improving quality, and achieving organizational learn-

ing (Gordon and Geiger, 1999). Through its use various healthcare organizations in

high income countries (HICs) improved in their recruitment and retention processes

and employees gained a better understanding of organizational strategies leading to

overall improvement in performance including reduction in costs, better clinical

outcomes, increased staff, and patient satisfaction (Curtright et al., 2000; Kaplan and

Norton, 2000; Hospital Report, 2003; Mannion et al., 2005).

The implementation of management models is considered a step towards

maturity and a change discourse aiming for an efficient and modern organization

(Schalm, 2008). In the context of low income countries (LICs), however, we know

little about successful models to promote greater management effectiveness at the

hospital level (Hartwig et al., 2008). Evidence about BSC usage in LICs is

deficient mainly due to lack of committed leadership, cultural readiness, quality

information systems, viable strategic plans, and optimum resources (Rabbani et al.,

2007). Simple dissemination of written guidelines in LICs is proving to be often

ineffective (Rowe et al., 2005) and health managers face significant challenges in

developing and managing appropriate systems (Green and Collins, 2003). Such

faulty information systems result in a clear lack of knowledge regarding where to

focus priorities; where improvement is needed and whether ongoing initiatives

were having a positive impact (Murray, 2007; Malqvist et al., 2008). Although a

partnership-mentoring model for enhancing management capacity in Ethiopian

hospitals has been tested (Hartwig et al., 2008), to our knowledge BSC specifically

has not been implemented in hospital settings in LICs. Recently BSC was applied

at a national (macro) level to demonstrate how provinces and the country are doing

in delivering the basic package of health services in Afghanistan (Peters et al.,

2007). This innovative adaptation of the BSC in Afghanistan at a macro level has

provided a useful tool to summarize the multidimensional nature of health services

and enabled managers to benchmark performance and identify strengths and

weaknesses in the Afghan context.

There is heavy emphasis on curative services and hospitals consume 45% of the

meager health budget in Pakistan (Abrejo et al., 2008). Despite this the quality of

health care in public hospitals is dismal and 70% health care is being provided by

private facilities (Ghaffar et al., 2000). We conducted this study at a private

university hospital in Karachi Pakistan to assess the feasibility of the modified Delphi

group technique to reach consensus about the indicators of an institutional level BSC,

identify the strengths, and weaknesses of data being generated and recommend ways

to improve hospital performance measurement.


DOI: 10.1002/hpm


METHODS

This study was conducted at a large private university hospital in Karachi (largest and

most populous city of Pakistan) in 2006. The hospital offers quality care to

outpatients and inpatients of all socio-economic classes (Rafique et al., 2006). It

operates with 542 beds in operation and offers a broad range of secondary and

tertiary services to over 38 000 hospitalized patients and to over 500 000 outpatients

annually. Inpatients have an average length of stay of 3.9 days (AKUH Quality

Manual, 2007: internal document). There are currently 400 trainees (interns,

residents, and fellows) affiliated with the hospital. Clinical services offered by this

university hospital (with staffing details) are listed in Table 1. According to the

hospital’s strategic plan (Health Sciences Centre Committee, 2002; internal

document), the vision is to provide (i) compassionate, accessible, and good quality

care that meets or exceeds expectations (ii) provision of a work environment that

fosters committed and motivated staff, and (iii) enabling leadership in research and

education that improves national health.

This hospital has an extensive health information system in place. An internal

situation analysis, however, identified the need for better integration of information

collected for evidence informeddecisionmaking (Health SciencesCentreCommittee,

2002: internal document). This report recommended that academicians and

administrators develop a road map together and foster a culture of team work,

shared vision, and institutional ownership. BSC serves as a road map for self-

assessment and continuous improvement towards excellence (Ruiz et al., 1999).

Table 1. University hospital–clinical services

Anesthesia �(28) Surgery (52)� Cardiothoracic surgery

Family medicine (12) � Dental-oral and maxillofacial surgery� General surgery

Medicine (54) � Neurosurgery� Cardiology � Ophthalmology� Diabetes, Endocrinology, and Metabolism � Orthopedic surgery� Gastroenterology � Otolaryngology� General internal medicine � Pediatric Surgery� Hematology and Oncology � Urology� Neurology� Pulmonary and Critical care medicine Ambulatory care services� Emergency medicine

Allied health servicesObstetrics and Gynecology (15) � Pharmacy

� PhysiotherapyPediatrics (23) � NutritionPathology and Microbiology (33) Diagnostic services

� CardiopulmonaryPsychiatry (7) � Clinical laboratories

� Neurophysiology� Radiology (21)

�Number in parenthesis represents total full time faculty. Non-faculty employees are not listed.


DOI: 10.1002/hpm


Therefore, in 2006 a multidisciplinary team comprising of hospital leadership

(including DG and CEO of the hospital, Medical Director, Chief Operating Officer)

agreed that the hospital needs to produce aBSC incorporating clinical and non-clinical

metrics for better clinical outcomes andperformancemanagement. In 2008anewVice

President (VP) was appointed for health services with the past experience of working

as an Executive Director at Guy’s and St. Thomas NHS Foundation Trust in London.

The newly appointed VP was responsible for corporate and clinical governance,

clinical operations and organization-wide performance measurement and manage-

ment. Under the leadership of VP, BSC was envisaged as an organizational

performance management pyramid (Figure 1) empowering all levels (from executive

to operational) with varying metrics and details. It would serve to link the hospital’s

strategic plan and individual department objectives. The frontline level was to look at

details with a large set of indicators tracked on a monthly/quarterly basis and

concerned with problem solving and improvements whereas the Board and executive

management would be more aligned towards long-term global trends, summary

reports generated biannually and concerned with overall strategy and governance.

Following an assessment for cultural readiness to implement the BSC (Rabbani

et al., 2008), a systematic development plan was used to design the BSC at this

hospital. The cultural assessment showed that the required prerequisites for BSC

implementation particularly conducive leadership, viable strategic plan, and a

functional management information system already existed at this hospital. The steps

used to design the BSC were in line with those outlined in earlier studies (Kaplan and

Figure 1. Proposed approach to develop a balanced scorecard


DOI: 10.1002/hpm


Norton, 1996; Wachtel et al., 1999; Oliveira, 2001; Peters et al., 2007). These are;

(i) building the business case at the executive leadership level of the hospital (Vice

President hospital services, Chief Operating Officer and Medical Director of the

hospital were involved in this study) (ii) identifying strategies and tactical objectives,

(iii) identifying performance measurements, (iv) identifying data sources, (v),

building consensus around the indicators to create the BSC based on the background

material obtained from internal documents (vi) develop communication tools and

targets for each measure based on benchmarking (vii) implementation, refinement,

evaluation, and reusing the BSC.

After working through initial steps this study focused on the creation of BSC for

use at the Medical Directorate level (Figure 1). A subsequent study (in progress) will

report on the adaptation and implementation of BSC at the frontline departmental

level. It is anticipated that results of the latter would be available in 2010 after which

the BSC would be cascaded upward to the CEO and Board level.

In order to select indicators for BSC, a multistage modified Delphi consensus

process developed by RAND (Marshall et al., 2006) was used. We used the modified

Delphi technique so that face-to-face panel discussions with experts in the field could

be conducted and face validity of indicators established. The face validity of the

indicators was defined as whether its meaning and relevance to the assessment under

consideration was self-evident and it superficially appeared to measure what it was

supposed tomeasure (McBurney, 2001).Apanel of nine expertswas selected based on

guidelines of the Delphi technique (Campbell et al., 2002). The group of experts was

identified from a variety of professional disciplines and the required range of

professional backgrounds. The panel represented hospital domains of marketing

(managers who conduct quarterly patient satisfaction surveys), clinical quality

assurance (clinicians, physician, and nurse managers who monitor quality care

indicators), human resource management (staff and managers who conduct annual

staff satisfaction surveys), and budget and planning (financial managers furnishing

financial reports). As per recommendation of other studies (Chung et al., 2008) it was

ensured that experts committed time and involvement until the process was complete.

Ethical approval for the study was received from the Ethical Review Committee of

the Pakistani hospital where this study was implemented.

RESULTS

Short listing of indicators by the expert panel

Following an extensive review of existing internal documents (periodical quality

assurance, patient and employee satisfaction surveys, and financial reports), a

preliminary list of 50 indicators was formulated in line with hospital’s strategic plan.

No indicators were removed from consideration at this phase of the activity. The next

step was to prioritize key performance indicators based on the criteria of importance,

scientific soundness (credibility), appropriateness to hospital’s strategic plan,

feasibility (i.e., whether the measure was available easily as part of management

information system, could be collected accurately, reliably and at a reasonable cost)

and modifiability of the clinical outcome measures.


DOI: 10.1002/hpm


The panel used the modified Delphi technique (over a period of 6 months) during

face to facemeetings to individually rate each indicator on a scale of 1–9 for the above

criteria. All criteria were given equal weightage. If an indicator was thought not to be

amenable to action it was dropped. Median scores and measures of disagreement for

the whole panel and individual ratings were discussed, in subsequent meetings. Panel

members were given an opportunity to change their ratings after the discussions.

Indicators receiving final scores of 7–9were regarded as robust, 4–6 as equivocal, and

1–3asweak (Figure 2).All indicators receiving scores of 7ormore (facevalidity)were

included in the final set. In addition, a small number of indicators which received

scores of 4–6 were retained if the panelists considered the indicators essential to

contribute to the overall balance and comprehensiveness of the final set.

Twenty indicators (receiving a median score of 7 or more) were finally selected

(Table 2) and organized by the expert panel into the four BSC quadrants: financial,

customer (patient), internal business, and innovation/learning.

Indicators for innovation and learning quadrant of the BSC

The indicators for employee satisfaction (innovation/learning quadrant of BSC) were

selected from the annual faculty and staff surveys and included (i) satisfaction with

job; dimensions of training and skills, work load including double duties performed,

maximum use of staff abilities, decision-making authority, and motivation to strive

for excellence, (ii) collegial satisfaction (helping each other in times of need, respect

from the colleagues, discussion with colleagues to mutually resolve issues),

(iii) satisfaction with supervisor; dimensions of friendly working relationship, regular

feedback, satisfaction with appraisal system, recognition for doing a good job,

openness to suggestions, and good ideas, (iv) satisfaction with organization; annual

faculty and staff turnover, fair treatment without gender and religious discrimination,

opportunities for growth and improvement, proud to work, viewing organization as a

Figure 2. Short listing indicators for BSC using a modified Delphi process


DOI: 10.1002/hpm

Table 2. Shortlisted set of indicators for the BSC using the modified Delphi technique

Indicators

Financial perspective (FP) Median Mean Std. deviation

� Average charges (inpatient) 8.00 8.22 0.44� Length of stay (inpatient) 7.00 6.89 0.78� Daily census (inpatient) 8.00 8.11 0.60� Net operating margin 9.00 8.67 0.50� Overall FP Median 8.00Internal business perspective (IBP): clinical outcomes (efficiency and quality)� Laboratory report turnaround time 8.00 7.89 0.60� Radiology film rejection rate 8.00 8.33 0.50� Unplanned stay after day care procedure 7.00 6.78 1.20� Incidence of blood transfusion reaction 7.00 7.22 1.09� Nosocomial infection 8.00 8.00 0.71� Cross match to transfusion ratio 7.00 7.33 0.87� Needle stick injuries 8.00 7.89 1.17� Overall IBP Median 8.00Human resource perspective (HRP)� Satisfaction with job 7.00 7.33 0.71� Satisfaction with colleagues 8.00 7.78 0.83� Satisfaction with on campus facilities 8.00 7.44 0.73� Satisfaction with organization 7.00 6.89 0.93� Satisfaction with supervisors 7.00 7.00 1.12� Overall HRP Median 7.00Patient satisfaction perspective (PSP)� Satisfaction with physicians 7.00 7.11 0.78� Patient Complaints (inpatient) 8.00 7.56 0.88� Satisfaction with nursing services 7.00 7.33 0.87� Proportion of patients recommending

this hospital to their families and friends8.00 7.89 0.60

� Overall PSP Median 7.50


long-term career choice, balance between work and personal life, and (v) satisfaction

with various on- campus staff facilities (sports and gymnasium, utility shops, child

day care center, cash withdrawal facilities, payment of utility bills etc.).

Some of the indicators reviewedwere not finally selected as theywere considered to

be more specific to the Human Resources Department (HRD) and not directly

influencingclinical serviceprovisionat thehospital.These included (i) theperformance

of HRD obtained through staff surveys (e.g., assistance provided to new employees,

managing employee discipline cases, promptness in responding to queries etc.), (ii)

indicators for employee safety and emergency preparedness obtained through Safety

andSecurityDepartment reports (e.g., fireemergency response time,numberof injuries

per100full-timeemployees) and (iii) indicatorsofworkforcemanagement suchas staff

absenteeism (doctors and nurses), number and type of employee illnesses.

Indicators for internal business quadrant of the BSC

Indicators for this quadrant were shortlisted from a larger list of quality care

indicators which the hospital (medical directorate) is monitoring through various


DOI: 10.1002/hpm


quality assurance teams for ongoing accreditation by the Joint Commission on

Accreditation of Healthcare Organizations USA (JCAHO). The selected indicators

included three indicators of efficiency; laboratory report turnaround time (number of

samples reported within acceptable time limit per total number of samples analyzed),

radiology film rejection rate (number of rejected films out of total number used and

indicates film wastage rate with implications for appropriate training of radiology

staff in patient positioning and exposure techniques) and cross match to transfusion

ratio (proxy for indicating the actual need for carrying out blood transfusions). There

were two indicators of quality of care and efficiency; unplanned stay after day care

procedure (number of unplanned overstay following day care surgery out of patients

undergoing day care surgery) and incidence of blood transfusion reactions (number

of blood transfusion associated reactions out of total units of blood transfused).

Needle stick injuries were selected as an indicator of staff compliance with safety

techniques and quality procedures and is also an indicator of employee safety. Rate

of nosocomial infections (central line associated blood stream infections in intensive

care units in relation to device days (following CDC guidelines of National Nosocomial

Infection Surveillance) was selected as the indicator of infection control.

The Health Management Information System provided information on mortality

related to anesthesia (using American Society of Anaesthesia guidelines), however,

desirable information on adjusted case fatality rate for tracer conditions was not

available. Similarly peri-operative mortality (in elective procedures) and returns to

operation theatre during the same episode were considered as indicators of operative

safety however due to residual confounding effects (age, severity of illness etc.), these

were not finally selected. Hospital acquired pressure ulcers, adverse event reporting,

patient fall rates, unplanned descents to the floor /1000 patient days were initially

selected as indicators of nursing quality but the panelists were of the opinion that in-

depth discussion was required with Division of Nursing Services and individual

clinical units should consider these while developing their customized scorecards.

Indicators for customer (patient) satisfaction quadrant of the BSC

Quarterly patient satisfaction surveys by the hospital’s Marketing department and

quality reports from the Department of Clinical Affairs were used to select four

indicators of patient satisfaction The patient satisfaction survey captures, analyzes and

monitors patient satisfaction with outpatient, inpatient, diagnostic and emergency

services. For the purpose of this study only inpatient service indicatorswere discussed.

These indicators are; (i) satisfaction with nursing services (dimensions: provision of

adequate information on health condition/medicines/ follow up care, courtesy,

listening, prompt response to call bell, respect for privacy, skilful insertion of cannulas/

IV lines, proper dressing, provision of special help when needed) (ii) satisfaction with

physicians (dimensions: daily visit of consultant, proper explanation and information

given and respect for privacy) (iii) recommending this service to family and friends (a

proxy indicator for a satisfied client), and (iv) percentage of patient complaints

(dimensions: care, delays, environment, attitude, availability of health staff,

communication, billing system, quality of food, cleanliness of washrooms, level of

noise, scheduled tests and investigation procedures on time etc.).


DOI: 10.1002/hpm


It is important to mention that the Department of Clinical Affairs and the

Hospital’s Risk Management Forum are already taking specific actions against those

patient complaints classified as ‘sensitive’ (resulting in potential defamation,

litigation, or compensation) and hence the latter was excluded from our list of

generic BSC indicators. Moreover ‘overall patient satisfaction’ was a composite

indicator shortlisted but not finally selected. The patients included in the sample are

requested to respond to the question’ were you overall satisfied with the quality of the

service you received? in terms of strongly agree, agree, neutral, disagree, and

strongly disagree. Responses obtained on a likert scale are converted to mean values

and reported as a percentage. Those who respond to this question with strongly agree

and agree are classified as overall satisfied patient. Sample size calculations for this

indicator needed statistical refinement and the panel could not rule out the element of

recall bias in these telephonically conducted interviews. Rates of medication error

(including prescribing, administration, and dispensing) and rationale use of

antibiotics were considered as indicators of patient safety, however, since there

were already specific quality control committees monitoring these on a quarterly

basis they were dropped as key indicators for the institutional level BSC.

Indicators for financial quadrant of the BSC

Four indicators were finally selected (i) average charges per inpatients (total inpatient

revenue earned against total number of patients admitted over a period) is a measure

of the accessibility of patients to hospital and also the cost-effectiveness of services

provided. It is computed by comparing the increase in average charges per inpatient

with the average price increase and the inflation rate, (ii) inpatient length of stay

(indicator of efficiency), (iii) average daily census (average number of patients

occupying bed per day), and (iv) net operating margin (margin on gross revenues

before interest and depreciation: indicator of cost and productivity) were some

indicators routinely generated by the department of Budget and Planning and later

shortlisted for the financial quadrant of the BSC.

FTE per adjusted occupied bed (an indicator of patient staff ratios and efficiency

monitored for JCAHO), % capex expenditure versus planned (highlights the total

capital budget consumed against the annual budget), percentage of referrals from

CHC; the low cost outpatient clinic (another indicator of financial accessibility of

this hospital for all socioeconomic groups) were selected in the initial list of 50

indicators but were dropped later due to lack of available national and regional

benchmarking and because some of this information was not considered relevant for

general public disclosure.

DISCUSSION

To the best of our knowledge, this was the first time that experts (managers, academicians,

and clinicians) were together involved in a scientific process (the modified Delphi group

technique) to develop a BSC for a hospital in a LIC setting. Integrating the activities


DOI: 10.1002/hpm


of different departments is a difficult task for the management of the organization

(Axelsson and Axelsson, 2006). Designing the BSC was possible in our study

because most of the necessary prerequisites for successful BSC implementation in

LICs (committed leadership, viable strategic plans and information systems etc.)

were already in place and cultural readiness for BSC usage had previously been

assessed (Rabbani et al., 2007; Rabbani et al., 2008). The modified Delphi method

successfully incorporated views from health personnel and specialists in the

development of a BSC. The same has been reported from Canadian hospitals

(Robinson et al., 2003). Another study of nine health provider organizations in USA

emphasized the importance of using a lot of teaching, discussion and consensus

building to ensure a successful BSC implementation (Inamdar et al., 2002).

Moreover consensus techniques such as modified Delphi have been utilized in

Thailand to develop trauma care indicators (Suwaratchai et al., 2008). The recent use

of formal consensus methods in Pakistan’s neighboring Muslim country Iran to

identify outcome-based indicators for rational drug prescribing and effective

academic leadership in order to increase the validity of the findings is also

encouraging (Bikmoradi et al., 2008; Esmaily et al., 2008). It is noteworthy that the

criteria (importance, scientific soundness, feasibility etc.) used by other studies

(Idanpaan-Heikkila, 2006; Marshall et al., 2006; McLoughlin et al., 2006) to short

list indicators also worked well in our setting.

Interestingly the composition of the multidisciplinary panel in our study was

found to be quite similar to other studies on development of a BSC for hospitals in

HICs. In an Australian study, experts in the Delphi panel included both hospital

managers and clinical practitioners (Xiao et al., 1997). In Taiwan (Huang et al.,

2004) the team that worked to develop the BSC included president, Vice President

and all department directors of the study hospital. In another recent study on

implementation of BSC in a community hospital at USA besides others the panel of

experts consisted of directors from patient care services and quality management

(Lorden et al., 2008). It is envisaged that this involvement of staff at different levels

within an organization during the development of BSC will enhance the acceptance

of the scorecard when it is implemented.

The Balanced Scorecard provided an opportunity to capture indicators in four

aspects of hospital performance. This demonstrates that data which are routinely

collected by the hospital can be used to develop an integrated core of

multidisciplinary indicators. This institutional level BSC has been designed for

use at the level of medical directorate. Subsequent studies can contextualize and

customize BSC by each of the implementing frontline clinical units so that specialty–

specific BSCs can be developed. Other studies have also used existing documents in a

similar fashion to create effective BSC (Wachtel et al., 1999; Idanpaan-Heikkila,

2006; Marshall et al., 2006; McLoughlin et al., 2006). In a recently concluded study

BSCwas used to track certain nursing indicators in acute care Ontario hospitals using

secondary data (Hall et al., 2008).

There was relatively a high level of agreement about the usefulness of the 20

indicators which were finally selected in our study. These indicators were distributed

across all 4 quadrants of BSC: financial perspective (n¼ 4), internal business (n¼ 7),

human resource perspective (n¼ 5) and patient satisfaction perspective (n¼ 4). The


DOI: 10.1002/hpm


indicators are similar to the ones shortlisted in both high (Baker and Pink, 1995) and

low income (Hansen et al., 2008) health settings. In the former study revenue

generated, patient volumes, patient and employee satisfaction, nosocomial

infections, average length of stay, and routine laboratory test turnaround time were

included among the 23 indicators shortlisted for the BSC developed for Canadian

hospitals. In the latter study in Afghanistan domains of patient and community, staff,

capacity for service, service provision, financial system, and overall vision were used

to monitor 29 indicators at provincial level. The latter are quite similar to the ones

selected in our setting. In another recent Canadian study following initial steps of

building executive commitment and strategic alignment, 23 indicators were

shortlisted which could be compared across various care centers using the same BSC

dimensions (Schalm, 2008).

The results from this investigation also reflect limitations of routinely collected

data. The Delphi process highlighted that certain indicators selected for BSC

(Table 3) had relatively lower face validity as assessed by their median ratings. This

was mainly due to lack of standardized definitions and measurement techniques,

reliable instruments, adequate sample sizes and response rates etc. Other studies

(Baker and Pink, 1995; Zeitlin et al., 2003) have also noted that methodological

shortcomings of many indicators have generated skepticism about the data sources,

consistency of reporting, derivation of the numbers, and their usefulness in offering

analogous estimates. It is to be noted that Pakistan does not have a national hospital

database. Comparable national/regional targets, benchmarking and a balance

between process and outcome indicators was recommended. It was also noted that

for subsequent designing of BSC for front line clinical departments disaggregation of

information by each clinical specialty would be needed.

It is possible that despite efforts to capture all relevant indicators through publicly

available surveys and documents, certain valuable indicators may have been

overlooked. Some of the shortlisted indicators in the western studies included

allocative efficiency, vertical equity, survival rates and age, sex and disease specific

mortality, and morbidity ratios (Wachtel et al., 1999; Robinson et al., 2003; ten

Asbroek et al., 2004; Schalm, 2008). The BSC developed in our setting did not have

some of the more analytical indicators listed above. Such lack of in-depth outcome

data has been listed as a BSC implementation barrier elsewhere (Schalm, 2008). It is

important to mention that to date at this hospital only diagnostic services (laboratory,

radiology) and pharmacy (data on medications prescribed and dispensed) are fully

computerized. Although each patient visiting this hospital has a unique medical

record number and information related to patient characteristics (age, gender,

diagnosis, length of stay, and clinical intervention performed) is computerized,

however, presenting complaints, co-morbid conditions, and discharge summaries are

still available only on paper files. Similar issues of patient data records have been

reported from Ethiopian hospitals (Hartwig et al., 2008). Moreover it has been shown

in Iran that hospitals are collecting a lot of financial and clinical information in a

fairly computerized but not in a well-organized format (Ghaffari et al., 2008). A need

for an electronic patient record system (an e–health initiative) was, therefore,

emphasized to overcome some of these methodological barriers and come up with

more robust indicators in our setting.


DOI: 10.1002/hpm

Table

3.Measurementissues

highlightedduringtheDelphiprocess

Nam

eofindicator

Sourceofdata

Issues

Financial

perspective

�Length

ofstay

(inpatient)

Monthly

reportsofdepartm

ent

ofbudget

andplanning

�Cannotstratify

byseverityofillness,number

ofcomplications

andco-m

orbidities

�Thisindicatorcanbeinfluencedbyfactors

beyondthehospital

environment(e.g.,absence

ofnursinghomes

andhomecare

facilities

postdischarge)

�Length

ofstay

canalso

increase

because

ofmanagem

entdelays

(schedulingofinvestigationprocedures,timingofconsultant

wardrounds,weekendadmissionsetc.)

Internal

business(efficiency

andquality)

�Unplanned

stay

afterday

care

procedure

Quarterlyqualityim

provem

ent

reports

�Cannotdifferentiatebetweensurgical

complicationsand

managem

entdelays(e.g.,delay

inavailabilityofoperation

theatreandlow

staffinglevels)

�Cannotcommentonwhichprocedure

causesmaxim

um

delay

�Incidence

ofbloodtransfusion

reaction

Quarterlyqualityim

provem

ent

reports

�Needto

setatarget

based

oninternational

comparison

�Cross

match

totransfusionratio

�Only

trendbeingmonitoredcurrently

Human

resourceperspective

�Satisfactionwithjob

Annual

employee

satisfaction

survey

�Low

response

rate

tothesurvey

�Notdisaggregated

bydepartm

entordesignation

�Only

quantitativeinform

ationcollectedonaslidingscale

�Satisfactionwithorganization

�Datacollectioninstrumentnotvalidated,questionslikelyto

evoke

apositiveresponse

�Satisfactionwithsupervisors

�Nointernational/regional

comparisonsortargets

Patientsatisfactionperspective

�Satisfactionwithphysicians

Quarterlypatientsatisfaction

surveys

�Does

notconvey

underlyinginform

ationaboutsatisfactionwith

physician

andnursingservices

byeach

clinical

departm

entand

therefore

delaysaction

�Satisfactionwithnursing

services

Measurementissues

relatedto

9/20indicators

oftheBSC

arehighlighted.Theseindicators

received

arelativelylower

rating(m

edian¼7).


DOI: 10.1002/hpm



Despite these practical limitations, the Delphi group process led to a pragmatic

interpretation of existing data resulting in the design of a scorecard with

comprehensive indicators in multiple dimensions. It has been reported that for

hospital managers and those developing health policies, studies such as this provide

insight into the factors influencing hospital performance (Xiao et al., 1997). The 20

indicators which emerged from our study using the modified Delphi process have

highlighted the methodological challenges faced during the design of BSC. As a next

step this scorecard will now be customized for individual clinical departments of this

hospital and later implemented at the executive and board level. The BSC would

facilitate rational organization and management of data collection systems and serve

as an evaluation framework for monitoring improvement of clinical outcomes and

quality. A greater cohesion within the hospital units is expected simultaneously.

Lessons learnt will have important bearing for hospital performance measurement

initiatives in other settings.

ACKNOWLEDGEMENTS

The authors thank the senior Aga Khan University (AKU) and hospital (AKUH)

leadership—President Firoz Rasul, Vice President Health Services Dallas Ariotti,

Dean of the Medical college AKU, Mohammad Khurshid, Director General and

CEOAKUH Nadeem Khan, Dean of Research and Graduate Studies AKU, El-Nasir

Lalani, Director Hiuman Resources AKU, Navroz Surani, and chair Dept of

Community Health Sciences (CHS) Dr Gregory Pappas—for encouraging us to

proceed with the work related to Balanced Scorecard (BSC) at AKU. This study is a

component of BSC studies underway. We thank Dr Naushaba Mobeen, Senior

Instructor Community Health Sciences and Dr Wasif Shahzad Manager Dept of

Medicine for assisting in initial meetings. We also express our gratitude to Mr Zafar

Tahir (CHS), Aslam Fareed and Muhammad Feisal (Marketing Department AKUH),

Ms Salma Jaffer (Manager JCIA Coordination AKUH), Ms Shamim Nayani (Senior

Manager Employee Relations, Rehman Hirani, and Khurram Jamal (Dept of Budget

and Planning AKU). Ms Saira Nigar (CHS) assisted us in data analysis and

Ms Shafaq Ambreen, administrative officer (CHS) rendered untiring secretarial assist-

ance. We thank Bo Badr Saleem Lindblad, Professor Emeritus of International Child

Health, Department of Public Health Sciences, Division of International Health

(IHCAR), Karolinska Institutet Medical University, Stockholm, Sweden, and visit-

ing professor, AKU, Karachi, Pakistan, for his overall support. Thanks also go to

Mr Thomas Mellin at IHCAR, Department of Public Health Sciences, Karolinska

Institutet (KI), Sweden for connecting the first author to various information

technology resources during her visits to KI. The authors acknowledge our various

grant sources: Swedish South Asian Network (SASNET: grant ID; EPG05S:06),

WHO EMRO (project ID #: RPC 04/60) and Swedish Institute (Si: Id # 05655/2005).

The major support for this study came from AKU University Research Council

(URC, project ID 052013 CHS).


DOI: 10.1002/hpm


REFERENCES

Abrejo FG, Shaikh BT, Saleem S. 2008. ICPD to MDGs: missing links and common grounds.Reprod Health 5(4): DOI: 10.1186/1742-4755-5-4

Axelsson R, Axelsson SB. 2006. Integration and collaboration in public health—a conceptualframework. Int J Health Plann Manage 19: 383–398. DOI: 10.1002/hpm.826

Baker GR, Pink GH. 1995. A balanced scorecard for Canadian hospitals. Healthc ManageForum 8: 7–21.

Bikmoradi A, Brommels M, Shoghli A, Sohrabi Z, Masiello I. 2008. Requirements foreffective academic leadership in Iran: a nominal group technique exercise. BMC Med Educ8: 24.

Campbell SM, Braspenning J, Hutchinson A, Marshall M. 2002. Research methods used indeveloping and applying quality indicators in primary care. Qual Saf Health Care 11: 358–364. DOI:10.1136/qhc.11.4.358

Castaneda-Mendez K, Mangan K, Lavery AM. 1998. The role and application of the balancedscorecard in healthcare quality management. J Healthc Qual 20: 10–13.

Chung KP, Lai MS, Cheng SH, et al. 2008. Organization-based performance measures ofcancer care quality: core measure development for breast cancer in Taiwan. Eur J CancerCare (Engl) 17: 5–18. DOI: 10.1111/j.1365-2354.2007.00796.x

Curtright JW, Stolp-Smith SC, Edell ES. 2000. Strategic performance management: devel-opment of a performance measurement system at mayo clinic. J Healthc Manag 45: 58–68.

Esmaily HM, Savage C, Vahidi R, Amini A, Zarrintan MH, Wahlstrom R. 2008. Identifyingoutcome-based indicators and developing a curriculum for a continuing medical educationprogramme on rational prescribing using a modified Delphi process. BMCMed Educ 8(33):DOI: 10.1186/1472-6920-8-33

Ghaffar A, Kazi BM, SalmanM. 2000. Health care systems in transition III. Pakistan, Part I. anoverview of the health care system in Pakistan. J Public Health Med 22: 38.

Ghaffari S, Doran C, Wilson A, Aisbett C, Jackson T. 2008. Investigating DRG cost weightsfor hospitals in low resource countries: an Iranian example. Int J Health Plann Manage1–14. DOI: 10.1002/hpm.948

Gordon D, Geiger G. 1999. Strategic management of an electronic patient record project usingthe balanced scorecard. J Healthc Inf Manag 13: 113–123.

Green A, Collins C. 2003. Health systems in developing countries: public sector managers andthe management of contradictions and change. Int J Health Plann Manage 18: S67–S78.DOI: 10.1002/hpm.721

Groene O, Skau JKH, Frolich A. 2008. An international review of projects on hospitalperformance assessment. Int J Qual Health Care 20(162): DOI: 10.1093/intqhc/mzn008

Hall LM, Peterson J, Baker GR, et al. 2008. Nurse staffing and system integration and changeindicators in acute care hospitals: evidence from a balanced scorecard. J Nurs Care Qual 23:242–252. DOI: 10.1097/01.NCQ.0000310655.60187.92

Hansen PM, Peters DH, Niayesh H, Singh LP, Dwivedi V, Burnham G. 2008. Measuring andmanaging progress in the establishment of basic health services: the Afghanistan healthsector balanced scorecard. Int J Health PlannManage 23: 107–117. DOI: 10.1002/hpm.931

Hartwig K, Pashman J, Cherlin E, et al. 2008. Hospital management in the context of healthsector reform: a planning model in Ethiopia. Int J Health Plann Manage 23: 203–218. DOI:10.1002/hpm.915

Hospital Report. 2003. Acute Care. A joint initiative of the University of Toronto, OntarioHospital. Association and the Government of Ontario. Canadian Institute for HealthInformation Ontario.

Huang SH, Chen PL, Yang MC, Chang WY, Lee HJ. 2004. Using a balanced scorecard toimprove the performance of an emergency department. Nurs Econ 22: 140–146, 107.

Idanpaan-Heikkila U. 2006. Selecting indicators for the quality of cardiac care at the healthsystem level in organization for economic co-operation and development countries. Int JQual Health Care 18: 39–344. DOI: 10.1093/intqhc/mzl028


DOI: 10.1002/hpm


Inamdar N, Kaplan RS, Bower M. 2002. Applying the balanced scorecard in healthcareprovider organizations. J Healthc Manag 47: 179–196.

Kaplan RS, Norton DP. 1996. The balanced scorecard: translating strategy into action.Harvard Business School Press: Boston, MA.

Kaplan RS, Norton DP. 2000. The Strategy-Focused Organization: How Balanced ScorecardCompanies Thrive in the New Business Environment. Harvard Business School Press:Boston MA.

Lorden A, Coustasse A, Singh KP. 2008. The balanced scorecard framework-A case study ofpatient and employee satisfaction: what happens when it does not work as planned? HealthCare Manage Rev 33(145): DOI: 10.1097/01.HMR.0000304503.27803

Malqvist M, Eriksson L, Nga NT, et al. 2008. Unreported births and deaths, a severe obstaclefor improved neonatal survival in low-income countries; a population based study. BMC IntHealth Hum Rights 8: 1–7. DOI: 10.1186/1472-698X-8-4

Mannion R, Davies H, Marshall M. 2005. Impact of star performance ratings in English acutehospital trusts. J Health Serv Res Policy 10: 18–24.

Marshall M, Klazinga N, Leatherman S, et al. 2006. OECD health care quality indicatorproject. The expert panel on primary care prevention and health promotion. Int J QualHealth Care 18: 21–25. DOI: 10.1093/intqhc/mzl021

McBurney D. 2001. Research methods. Wadsworth Thomson Learning: Belmont, CA,Stamford, USA.

McKeeM, Healy J, Edwards N, Harrison A. 2002. Pressures for change.Hospitals in changingEurope. World Health Organisation (WHO): European observatory on health care systemsseries 36–58.

McLoughlin V, Millar J, Mattke S, et al. 2006. Selecting indicators for patient safety at thehealth system level in OECD countries. Int J Qual Health Care 18: 14–20. DOI: 10.1093/intqhc/mzl030

Murphy MK, Black NA, Lamping DL, et al. 1998. Consensus development methods, and theiruse in clinical guideline development. Health Technol Assess 2: 1–88.

Murray CJL. 2007. Towards good practice for health statistics: lessons from the millenniumdevelopment goal health indicators. The Lancet 369: 862–873. DOI: 10.1016/S0140-6736(07)60415-2

Oliveira J. 2001. The balanced scorecard: an integrative approach to performance evaluation.Healthc Financ Manage 55: 42–46.

Ovretveit J, Al Serouri A. 2006. Hospital quality management system in a low income Arabiccountry: an evaluation. Int J Health Care Qual Assur 19: 516–532. DOI: 10.1108/09526860610686999

Peters DH, Noor AA, Singh LP, Kakar FK, Hansen PM, Burnham G. 2007. A balancedscorecard for health services in Afghanistan. Bull World Health Organ 85: 146–151.

Rabbani F, Jafri W, Abbas A, et al. 2009. Culture and quality care perceptions in a Pakistanihospital. Int J Health Care Qual Assur 22: 498–513. DOI: 10.1108/09526860910975607

Rabbani F, Jafri W, Abbas F, Pappas G, Brommels M, Tomson G. 2007. Reviewing theapplication of the balanced scorecard with implications for low-income health settings.J Healthc Qual 29: 21–34.

Rafique G, Azam SI, White F. 2006. Diabetes knowledge, beliefs and practices among peoplewith diabetes attending a university hospital in Karachi, Pakistan. East Mediterr Health J12: 590–598.

Robinson VA, Hunter D, Shortt SE. 2003. Accountability in public health units: using amodified nominal group technique to develop a balanced scorecard for performancemeasurement. Can J Public Health 94: 391–396.

Rowe AK, de Savigny D, Lanata CF, Victora CG. 2005. How can we achieve and maintainhigh-quality performance of health workers in low-resource settings? The Lancet 366:1026–1035. DOI: 10.1016/S0140-6736(05)67028-6

Ruiz U, Simon J, Molina P, Jimenez J, Grandal J. 1999. A two-level integrated approach toself-assessment in healthcare organisations. Int J Health Care Qual Assur 12: 135–142.


DOI: 10.1002/hpm


Schalm C. 2008. Implementing a balanced scorecard as a strategic management tool in a long-term care organization. J Health Serv Res Policy 13: 8–14. DOI: 10.1258/jhsrp.2007.007013

Suwaratchai P, Sithisarankul P, Sriratanban J, Chenvidhyamd D, Phonburee W. 2008. Utilizethe modified Delphi technique to develop trauma care indicators. J Med Assoc Thai 91: 99–103.

ten Asbroek AHA, Arah OA, Geelhoed J, Custers T, Delnoij DM, Klazinga NS. 2004.Developing a national performance indicator framework for the Dutch health system. Int JQual Health Care 16: i65–i71. DOI: 10.1093/intqhc/mzh020

Veillard J, Champagne F, Klazinga N, Kazandjian V, Arah OA, Guisset AL. 2005.A performance assessment framework for hospitals: the WHO regional office for EuropePATH project. Int J Qual Health Care 17(487): DOI: 10.1093/intqhc/mzi072

Wachtel TL, Hartford CE, Hughes JA. 1999. Building a balanced scorecard for a burn center.Burns 25: 431–437. DOI: 10.1016/S0305-4179(99)00024-8

Willis CD, Stoelwinder JU, Cameron PA. 2008. Interpreting process indicators in trauma care:construct validity versus confounding by indication. Int J Qual Health Care 20: 1–8. DOI:10.1093/intqhc/mzn027

Xiao J, Douglas D, Lee AH, Vemuri SR. 1997. A Delphi evaluation of the factors influencinglength of stay in Australian hospitals. Int J Health Plann Manage 12: 207–218. DOI:110.1002/(SICI)1099-1751(199707/09)12:3

Zeitlin J, Wildman K, Breart G, et al. 2003. Selecting an indicator set for monitoring andevaluating perinatal health in Europe: criteria, methods and results from the PERISTATproject. Eur J Obstet Gynecol Reprod Biol 111: 5–14. DOI: 10.1016/j.ejogrb.2003.09.002


DOI: 10.1002/hpm

Documents

Measuring What Counts in HIS - Balanced Scorecards