Does pay for performance save lives? Martin Roland University of Cambridge UK

Does pay for performance save lives?

Martin Roland

University of Cambridge

UK

Does pay for performance save lives?

University of Manchester

• Matt Sutton, Ruth Boaden, Søren Rud Kristensen, Rachel Meacock, Alex Turner

Warwick Business School

• Ruth MacDonald

University of Cambridge

• Martin Roland

All authors declare that they have no conflict of interest. The study was funded by the National Institute of Health Research

What’s the evidence on pay for performance?

Current literature suggests:

• Clear evidence that pay for performance can improve the process of care

• Not a ‘magic bullet’ – effects likely to be limited in the absence of other quality improvement measures

• Very limited evidence of impact on outcome

• P4P scheme introduced in 2008 in all 24 NHS hospitals

in the North West of England (population 6.8m)

• Process indicators for acute myocardial infarction,

heart failure, pneumonia, hip and knee surgery and

coronary artery bypass surgery

Evaluation of ‘Advancing Quality’

• Months 1-12: Tournament system in which only the

top performers received a bonus (top quartile

received extra 4% second quartile extra 2%)

• Months 13-18: Bonuses for both achieving targets

and improving score

• Months 19-42: Penalties if hospitals failed to achieve

targets

‘Advancing Quality’ payment rules

Impact on mortality for acute myocardial infarction, heart failure and pneumonia in:

• First 18 months (Sutton et al NEJM 2012;367:1821–8)

• Next 24 months (NEJM ‘revise and resubmit’)

We did not include hip and knee surgery (low mortality) or

coronary bypass surgery (not carried out in most hospitals)

Evaluation of ‘Advancing Quality’ over 42 months

Study design

We studied 30 day in-hospital mortality comparing mortality for acute myocardial infarction, heart failure and pneumonia using a difference in difference analysis with three sets of controls:

• Mortality for these conditions in all other 132 hospitals in England

• Mortality for non incentivised conditions in: a) hospitals in the North West of England and b) all other hospitals in England

Selection of non-incentivised conditions

• not clinically linked to any incentivized condition

• sufficient volume (over 9,000 admissions in England per year)

• 30-day mortality over 6%

• more than 80% of deaths within 30 days occurring in a hospital

• Six conditions met these criteria: acute renal failure, alcoholic liver disease, intracranial injury, and intestinal obstruction, pulmonary embolism and duodenal ulcer

Data available for analysis

• Used routine hospital activity data collected from NHS hospitals

• All admissions for 18 months before and 42 months after the introduction of the scheme

• Patients treated at the 24 NHS hospitals in the North West and the 132 NHS hospitals in all other regions of England

Number of admissions in each analysis

18 months 48 months

Pneumonia 410,384 761,954

Heart failure 201,003 338,921

Acute myocardial infarction

245,187 390,652

Non-incentivised conditions

241,009 333,991

Includes admissions for 18 months prior to P4P scheme

Difference in differences analyses controlled for:

• Gender

• Age

• Type of admission (planned/unplanned, transfer)

• Location from which the patient was admitted

• 31 co-morbidities (Elixhauser) derived from secondary diagnostic codes

Incentivised conditions


Non -incentivised conditions

Comparison 2


Non -incentivised conditions

Comparison 3


Comparison 1Incentivised conditions

24 hospitals in North West England

132 hospitals in the rest of England

Difference-in-differences analyses

Changes in mortality summarised as triple difference in differences analysis

Sensitivity analyses

• Controlling for baseline mortality

• Controlling for changes in patient volumes

• Controlling for hospital quality and financial rating (data from national regulator)

• Controlling for hospital type (teaching)

• Different method of classifying co-morbidities (Charlson )

• Checking pre-intervention trends in mortality no different between incentivised and non-incentivised conditions or between areas.

• Comparing North West with a subset of similar English districts instead of the whole of England

Summary of analysis of first 18 months

• Significant reduction in absolute mortality of 1.3% (relative reduction of 6%)

• Driven by significant reduction for pneumonia and non-significant reductions for MI and heart failure

• Equivalent to 890 fewer deaths in the North West in the 18 month period

Sutton M, Nikilova S, Boaden R, Lester H, McDonald R, Roland M. Reduced mortality with hospital pay for performance in England. New England Journal of Medicine 2012;367:1821–8

Analysis of the next 24 months

• Between-region difference-in-differences analysis

• Time trends for each of the 20 quarter year periods

• Separate estimates of impact of short and long term periods

What happened in the second time period?

• Mortality continued to fall in the North West, but there was no longer a significant difference compared to controls

• Mortality for incentivized conditions fell more in control hospitals than in Advancing Quality hospitals (0.7 percentage points more, 95% CI 0.3 to 1.2)

• Mortality for non-incentivized conditions fell more in Advancing Quality hospitals than in control hospitals (1.2 percentage points more, 95% CI 0.4 to 2.0).

What happened in the second time period?


Non-incentivised conditions

Advancing Quality hospitals (North West)

First 18 months -1.6% -0.5%

Next 24 months -1.8% -2.9%

Rest of England (controls)

First 18 months -0.8% -1.2%

Next 24 months -2.3% -1.7%

Estimates are from weighted least-squares regression models Results are robust to a range of specifications of standard errors and weights

Possible explanations

• Mortality for incentivized conditions fell more in control hospitals than in Advancing Quality hospitals

Did the other hospitals start to use the AQ quality indicators?


• Mortality for incentivized conditions fell more in control hospitals than in Advancing Quality hospitals


• Mortality for non-incentivized conditions fell more in Advancing Quality hospitals than in control hospitals

Were there ‘spillover effects’ on other aspects of care in the intervention hospitals?



• The results of the AQ pilot were widely disseminated and given national publicity

• Two other regions of the country developed P4P schemes using the AQ indicators

• The reduction in mortality in these regions was more than in other regions of England


Were there ‘spillover effects’ on other aspects of care in the intervention hospitals?

• We looked at which doctors cared for patients with the non-incentivised conditions

• There was substantial overlap between doctors caring for the incentivised conditions and two of the non-incentivised ones – acute renal failure and alcoholic liver disease

• The fall in mortality for these two non-incentivised conditions was greater than for the other non-incentivised conditions

Lessons for evaluation

Impact on outcomes may not be long-lasting

Spillover effects may hinder evaluation by masking true effects, e.g.

• into un-incentivised hospitals

• onto un-incentivised conditions

(spillover effects are deliberate)

How did we explain the reduction in mortality?

• Not solely on the basis of changes in process measures

• Wide range of quality improvement strategies

• New data collection systems and feedback on performance

• Employment of new specialist nurses

• Regular face to face learning events between hospitals

Why was it different from the US (HQID)?

• Self-selected hospitals in US

• Larger bonuses

• Greater probability of earning a bonus

• More face to face learning between hospitals

Pay for performance can have a beneficial impact on outcomes

The causal pathway is likely to be multi-factorial and not just related to improvements in incentivised aspects of care

The impact may not be long-lasting, but spillover effects (which are welcome) may mask longer term improvement

Quality indicators in the Advancing Quality P4P scheme

Pneumonia• Blood cultures performed in the ER prior to initial antibiotics • Adult smoking cessation advice/counselling• Initial antibiotic received within six hours of hospital arrival• Initial antibiotic selection in immuno-competent patients• Oxygenation assessment

Heart failure• Discharge instructions• Adult smoking cessation advice/counselling• ACEI or angiotensin receptor blocker for left ventricular systolic dysfunction• Left ventricular systolic function assessment

Acute myocardial infarction• Adult smoking cessation advice/counselling• Beta blocker prescribed at discharge• Aspirin at arrival• Aspirin prescribed at discharge• Fibrinolytic therapy received within 30 minutes of hospital arrival• ACEI or angiotensin receptor blocker for left ventricular systolic dysfunction

Table 1: Average hospital achievement on the incentivized indicators in the North West of England in the short and long-term periods

Condition Indicator name Short Term Long Term Difference

Pneumonia

Blood cultures performed in the emergency department prior to initial antibiotics received in hospital 63.1 81.9 18.7 Adult smoking cessation advice/counselling 44.9 62.7 17.8 Initial antibiotic received within six hours of hospital arrival 66.2 76.2 10.0 Initial antibiotic selection in immunocompetent patients 82.7 91.2 8.5 Oxygenation assessment 97.6 99.5 1.9

Heart Failure

Discharge instructions 29.2 55.1 25.9 Adult smoking cessation advice/counselling 55.6 75.7 20.1 Angiotensin converting enzyme inhibitor or angiotensin receptor blocker for left ventricular systolic dysfunction 90.1 94.9 4.8 Left ventricular systolic function assessment 89.3 93.6 4.3

Acute Myocardial Infarction

Adult smoking cessation advice/counselling 86.8 94.1 7.3 Beta blocker prescribed at discharge 93.8 97.6 3.7 Aspirin at arrival 97.1 98.8 1.8 Aspirin prescribed at discharge 98.3 99.3 1.1 Fibrinolytic therapy received within 30 minutes of hospital arrival 84.4 85.3 0.9 Angiotensin converting enzyme inhibitor or angiotensin receptor blocker for left ventricular systolic dysfunction 97.5 98.0 0.5

Note: The table shows the percentage of patients for whom an indicator was met on the incentivized indicators in the North West of England. The short term period covers the first 18 months of the program. The long term period includes months 19-42 of the program. Within each condition the measures are ranked in descending order of the magnitude of the increase in achievement between the short- and long-term periods.

Documents

Does pay for performance save lives? Martin Roland University of Cambridge UK