Measuring Health System Performance

Lecture 3

Health Service OutcomesHealth Service Outcomes

Reinhard Busse, Prof. Dr. med. MPHDepartment of Health Care Management

Berlin University of Technology/

(WHO Collaborating Centre for Health Systems Research and Management)

European Observatory on Health Systems and Policies

Outline

• Concept and background

– Population health vs. Health service indicators

– Health service outcomes = quality?

• Dimensions of quality

• Outcome vs process

• Different type of Health Service Outcome indicators• Different type of Health Service Outcome indicators

– Mortality indicators (general, condition-specific)

– Patient Safety Indicators

– Readmissions

– Ambulatory-care sensitive hospitalisations

– Patient Reported Outcome Measures

– Process indicators

Population health indicators (denominator: population)

Life expectancy/

overall mortality

Health-adjusted

life expectancy

YLD

Tracer: Condition-

specific mortality (e.g.

AMI, breast cancer)

YLL

Amenable/ avoidable

mortality (group of tracers)

Ambulatory-care

sensitive

hospitalisations

Health service indicators

Condition-specific

inpatient mortality

Condition-

specific 5-year

survival (e.g.

breast cancer)

Specific

(tracer)

(denominator: patients)

hospitalisations

Infant mortality

Hospital

mortality

Hospital

readmissions

Patient

safety

indicators

inpatient mortality

(e.g. AMI)

Condition-specific

processes

Patient-reported

outcomes (function,

quality-of-life)

Attributability to

health care provider

generic

„... quality of care is that component of the difference

between efficacy and effectiveness that can be attributed to

care providers, taking account of the environment in which

they work.“

(Brook RH & Lohr KN. Efficacy, effectiveness, variations, and quality. Medical

Health service performance = quality?

• no uniform definition of quality; an often quoted definition is:

(Brook RH & Lohr KN. Efficacy, effectiveness, variations, and quality. Medical

Care 1985; 25: 710-722)

• This definition implies focus on effectiveness (health) as outcome

• OECD definition is wider as it also includes patient safety and

patient-centeredness (which we summarise under patient experience)

Most commonly used quality dimensionsDimension Definition

Safety The degree to which health care processes avoid, prevent

and ameliorate adverse outcomes or injures that stem

from the processes of health care itself

Effectiveness The degree of achieving desirable outcomes, given the

correct provision of evidence-based healthcare services to

all who could benefit but not to those who would not

benefit. Includes appropriateness of care.benefit. Includes appropriateness of care.

Responsiveness How a system treats people to meet their legitimate non-

health expectations

Accessibility The ease with which services are reached

Equity The extent to which a system deals fairly with all concerned

Efficiency The system’s optimal use of available resources to yield

maximum benefits or results.

Source: Kelley, E & Hurst, J. (2006) Health Care Quality Indicators Project: Conceptual

Frameowrk Paper. OECD Working Paper No. 23

Elements of quality and potential

problems

Elements of Quality Care Type of Quality Problem

People get the care they need Underuse

People need the care they get Overuse

Provided safely ErrorProvided safely Error

Timely Delays

Patient-centered Unresponsive

Delivered equitably Disparities

Delivered efficiently Waste

IOM, Crossing the Quality Chasm (2001)

Population

health status

(need)

Other sectors

Nutrition/ agriculture

Environment

Personnel sufficient

and well qualified?

Institutions of high standards?

Technologies effective?

Coverage &

needs-based,

equitable

access?

Patients satisfied,

services safe and

of high quality?

Health care

outcome:

satisfaction,

complications

etc.Structures and

organisation

Patients

Process

Health

gain/

Outcome

Human

resources

Technologies

Financial

resources

Fair and sustainable funding?

Utilization responsive,

appropriate, coordinated …?

Health care system

Re

sou

rce

cre

ati

on

ad

eq

ua

te?

How much?

How equitable?

Are the services delivered efficiently?

Donabedian Quality Assurance Model

Structure:Material Resources

Operational CharacteristicsOrganizational Characteristics

Process:Clinical Care

Policy and ProcedureAdherence to standardsOrganizational Characteristics Adherence to standards

Outcome:Health status of patients

Clinical measures

Suitability of condition-specific mortality

(and amenable mortality) for performance assessment:

the example of AMI mortality in England, 2002-2010

Smolina et al (BMJ, 2012) http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3266430/

Authors ask: how much of this decline is due to a fall in incidence and how much to declines in case fatality?

Contribution of average annual trends in event

rate and case fatality rate to average annual trend

in mortality for AMI by region, 2002-10, England

From a policy From a policy

maker’s perspective:

why is distinguishing

between these

causes important?

Mortality measurement: a valid

indicator for quality of care?

• Increased publication of risk adjusted

hospital mortality rates

• What is the relationship between

mortality rates and quality of care?

• Systematic Review of evidence (2007)• Systematic Review of evidence (2007)

– A positive correlation between better

quality of care and risk adjusted mortality

was found in under half the relationships

• Shahian et al. (2010)

– Risk adjustment method can produce

substantially different results

Methodological Challenges

Reliability of Data

• mortality statistics, registries, administrative data-bases,

electronic health records, survey data

Validity of IndicatorsValidity of Indicators

• -face validity

• content validity

• construct validity

• criterium validity

Hospital Standardised Mortality Ratio (HSMR)

Summary-level In-Hospital Mortality Indicator (SHMI)

Admissions

and deaths included

• All in-hospital deaths

• For the 56 diagnostic groups

(e.g. CCS = clinical

classification software or DRGs)

• All in-hospital deaths and

deaths 30d post discharge

• For all diagnosis super groups

except births, stillbirths

• Age group

• Diagnosis/procedure subgroup

• Comorbidities

• Age group

• Diagnosis

• Comorbidities

14

Risk adjustment

• Comorbidities

• Admission method

• Gender

• Palliative care coding

• Deprivation

• Number of previous emergency

admissions

• Discharge year

• Month of admission

• Source of admission

• Comorbidities

• Admission method

• Gender

Dutch hospital standardised mortality ratios

2001-03 (standardised for age, sex, urgency/readmission, LOS within 50 CCS

groups leading to 80% all deaths, excluding small hospitals and those with poor

data recording, using year 2000 standard)

100

120

140

HS

MR

s (9

5%

CIs

) 200

1-2

003

0

20

40

60

80

96

35

68

14

83

81

51

25

89

50

103 3

52

44

85 5

78

36

12

100

72

94

13

104

65

33

34

95

101

39

93

82

79

23

61

47

37

20

87

97

45

31

107

19

98

54

102

Hospital number (assigned by BJ)

HS

MR

s (9

5%

CIs

) 200

1-2

003

Mortality for particular conditions?

• Select conditions where quality of care is associated with

mortality (AMI, Stroke, Hip)

• Challenges:

– sample size

(ex. Dimick et al. 2004 – studies 7 operations that the Agency for Healthcare (ex. Dimick et al. 2004 – studies 7 operations that the Agency for Healthcare

Research and Quality in the US recommended surgical mortality as a quality

indicator – found that only for 1 (CABG surgery) was sample size large

enough to make quality assessments)

– generalizability of results

(Challenges include: International comparisons of mortality, timing,

measurement)

Source: Dimick et al. (2004) Surgical Mortality as an Indicator of Hospital Quality: The Problem with Small Sample Size. JAMA 292(7):847-851.

Patient Safety Indicators

• Common: infection rates (MRSA, C-difficile)

• Death among surgical inpatients with serious treatable

complications

• Adverse post-op outcomes (i.e. pressure ulcer, hip fracture,

hemorrhage etc)hemorrhage etc)

• Composite - Complication/patient safety for selected

indicators

• Never events, non events (wrong site surgery, wrong

prosthesis, etc)

• Medication errors

http://www.qualityindicators.ahrq.gov/Downloads/Modules/PSI/V41/TechSpecs/PSI%2004%20Death%20among%20Surgical%20Inpatients.pdfhttp://www.england.nhs.uk/wp-content/uploads/2013/12/nev-ev-list-1314-clar.pdf

Patient Safety Indicators

Readmission indicators:

a valid health services outcome measure?

• Various studies suggest Readmissions may not be always indicative of poor quality

• McClellan & Staiger (1999), Papanicolas & Mcguire (2011) – found some conditions had negative correlations between mortality and readmissions – USA and UK samples.

• Laudicella et al. (2013) hospitals’ performance in readmissions is • Laudicella et al. (2013) hospitals’ performance in readmissions is determined in part by their difference in the quality of care and in part by their difference in the share of unobservably sicker patients. (UK sample)

• Fischer et al. (2011): Systematic review of readmission indicators identified only 21 out of 486 studies addressed validity of indicator when using it as an outcome measure.

– Little consensus over time-frame, type of readmission and case-mix adjustment applied.

Ambulatory Care Sensitive Conditions:good ambulatory care should prevent or reduce hospitalisations

What are ACSCs?

Chronic conditions that include congestive heart failure, diabetes, asthma, angina, epilepsy and hypertension.

Actively managing patients Actively managing patients with ACS conditions – through vaccination; better self-management, disease-management or case-management; or lifestyle interventions – prevents acute exacerbations and reduces the need for emergency hospital admission.

http://www.dartmouthatlas.org/data/topic/topic.aspx?cat=25

ACSCs: comparison of Canadian provinces

But is it a valid indicator?

Trends in ACSCs – is primary care in the

UK improving?

Bardsley et al (2012) Is secondary preventive care improving? Observational study of 10-year

trends in emergency admissions for conditions amenable to ambulatory care

http://bmjopen.bmj.com/content/3/1/e002007?cpetoc

Patient Reported Outcome Measures

(PROMs)

• Instruments which gain information about health, illness and

the effects of health care interventions from the perspective

of the patient (Fitzpatrick et al, 1997)

• Four types of PROM:

– Generic– Generic

– Utility

– Disease specific

– Individual

Generic proms are applicable to the

widest possible range of health problems

Utility instruments assign utilities to

respondent’s health states – the EQ5D

Disease-specific proms are tailored to the

specific disease for which they are intended

Individual level proms are tailored to the

specific disease for which they are intended

PROM results in England 2013/14

Participation and

Coverage: There have

been 128,759 PROMs-

eligible procedures

carried out in hospitals

and 98,695 pre-operative

questionnaires returned

so far, a participation rate so far, a participation rate

of 76.7%.

For the 98,695 pre-

operative questionnaires

returned, 44,460 post-

operative questionnaires

were sent out, of which

11,423 have been

returned so far - a return

rate of 25.7% (73.3%).

Yet, how effective are PROMs?

• Fitzpatrick (2009) assess PROMs with respect to 7 criteria:– Reliability

– Validity

– Responsiveness

– Precision

– Interpretability

– Acceptability– Acceptability

– Feasibility

• PROMs are increasingly being used as performance instruments

• Hold much promise – but realising their potential requires: – Credible data collection, instruments and analysis

– Good reporting so that information is useful for different users and for different types of decisions

– Recognitions of limitations

– An open mind to this type of information

Outcome vs Process Measures

The case for measuring outcomes of care

• Central indicator of the success of health care

• Essential for determining “what works” in health care

• Can nurture innovation

The case for measuring processes of care

• Certain aspects of process (such as waiting

times or patient experience) are often valued

by patients

• Certain processes are known to be

associated with desired health outcomes.

• Measuring outcomes can be difficult, costly

and takes a long time -

• Can nurture innovation

• Are universal and do not become easily obsolete

• Clinical attention is focused on securing improved health rather than ‘checklists of activities’

• Harder to manipulate than process measures

Measuring outcomes can be difficult, costly

and takes a long time -

• Process measures are almost instantaneous

and can be acted on quickly.

• Process measures are usually readily

attributable to the provider of care and so

more easily interpretable (as opposed to

outcome measures which display a lot of

random noise).

• It is easier to devise incentive schemes

associated with process rather than

outcomes.

Process Measures

• Process Measures (waiting times) vs Clinical Process Measures (Measuring blood pressure for Hypertensive patients)

• Advantages: fast to collect, easier to attribute • Advantages: fast to collect, easier to attribute directly to health services, reflect compliance with good practice.

• Disadvantages: may be less relevant when considered alone, not always applicable, may become dated.

Developing Clinical Process Measures

• Selecting topics

• Reviewing clinical evidence

• Identifying quality indicators

• Constructing process measures

• Creating scoring methods

Process indicators (examples)

Measure Set Measures

Acute Myocardial Infarction (AMI)

■ Aspirin on Arrival

■ Aspirin Prescribed at Discharge

■ ACEI or ARB for LVSD

■ Beta Blocker Prescribed at Discharge

■ Beta Blocker on Arrival

■ Thrombolytic agent received within 30 minutes of hospital arrival

■ Percutaneous Coronary Intervention within 120 minutes of hospital arrival

■ Adult Smoking Cessation Advice/Counseling

■ LVF Assessment ■ Discharge Instructions

Heart Failure (HF)

■ LVF Assessment

■ ACEI for LVSD

■ Discharge Instructions

■ Adult Smoking Cessation Advice/Counseling

Pneumonia (PN)

■ Initial antibiotic received within 4 hours of hospital arrival

■ Oxygenation Assessment

■ Pneumoccoccal Screening and/or Vaccination

■ Blood Cultures

■ Adult Smoking Cessation Advice/Counseling

■ Appropriate initial antibiotic selection

Surgical Infection Prevention (SIP)

■ Prophylactic antibiotic received within 1 hour prior surgical incision

■ Prophylactic antibiotic discontinued within 24 hours after surgical infection

Patient Safety Indicators (PSI)

■ Postoperative Septicemia

■ Postoperative PE/DVT

■ Infection due to medical care

■ OB trauma without instruments

Interpretation issues: what measures of

quality can and cannot tell you

• MOST indicators require further investigation or

validation before one can be confident that it

indicates ‘good’ or ‘bad’ quality.

• Often our assessment depends on where an Often our assessment depends on where an

organization/physician/unit is placed in relation to

others rather than to an absolute standard, but this

can also be influenced by OTHER factors:

– Data issues, differences in clinical practice, external

factors, random variation, all of the above