Key Performance Indicators, Centre Reports, and more

Stephen McDonald

Barbecue talk

20

40

60

80

100

120

per m

illio

n pe

r yea

r

1970 1980 1990 2000 2010Year

Rate 95% CIIncident rate

Incident RRT, Australia only

More “good” news

5

10

15

20

30

40

50

60

70

50

100

150

200

0

500

0

200

400

600

0

100

200

300

1980 1990 2000 2010 1980 1990 2000 2010 1980 1990 2000 2010

0-24 25-34 45-54

65-74 75-84 85+

Rate 95% CI

per m

illio

n pe

r yea

r

Year

Graphs by age group

AustraliaAge-specific incident RRT rates

Indigenous incidence rates

0

100

200

300

400

500

Per

mill

ion

per y

ear

1985 1990 1995 2000 2005 2010Year

Rate 95% CI

Aboriginal & TSI, Australia

Background

• A number of ongoing work themes exist within ANZDATA for generating output– Stock and flow figures– Annual Report– Contributor requests

• Responses to information needed for various projects

– Research projects (internal and external analyses)

– Outcomes reporting

Outcomes reporting

• Recent years have seen a growth of interest in outcomes reporting

• Centre reports have been part of ANZDATA for many years, with increasing emphasis in recent years– At “parent hospital level”– Limited distribution historically

Why measure outcomes?

.2

.5

1

2

4

Obs

erve

d / E

xpec

ted

mor

talit

y

0 20 40 60 80units

O/E 98% CI

All Australia & NZ Dialysis Units, 98% confidence intervals

Dialysis outcome

.2

.5

1

2

4

Adj

uste

d re

lativ

e ris

k

0 20 40 60 80

Units, ranked by RR

RR 95% CI

Mortality rate during dialysis treatment in Australia 2006-10, adjustedfor demographics and comorbidities

Variation in transplant outcomes

.25

.5

1

2

5

10

20

50R

R g

raft

failu

re

0 5 10 15 20Units

RR 95% CI

Fully adjusted 1 year graft survival, by unitAll transplant units, Australia and New Zealand, patients transplanted2005-2019, followup to 2010

What is happening to centre reports?

• Greater reporting of demographics and comorbidities

• Adjusted analyses in transplanting centre and dialysis reports– Details of models supplied

• Graphs– Funnel plots– CUSUM plots (transplant)

Centre reports – graph 1

0.00

0.25

0.50

0.75

1.00

Pat

ient

Sur

viva

l

0 1 2 3 4 5Years

CNARAustraliaNew Zealand

Survival from 90th Day of Treatment

Centre reports – graph 2

0.00

0.25

0.50

0.75

1.00

Tech

niqu

e S

urvi

val

0 1 2 3 4 5Years

CNARAustraliaNew Zealand

Technique Survival - PD at 90 days

But....

0

20

40

60

80

100

Age

(of p

reva

lent

pat

ient

s)

Everywhere else CNARTS

Adjusted graphs

.7

.8

.9

1

1.1

1.2

1.3

Adj

uste

d S

MR

CNAR Australia New Zealand

Adjusted SMR (95% CI)

Adjusted graphs

CNAR

0

.5

1

1.5

2

2.5

SM

R

0 50 100 150 200Expected Number of Deaths

How are reports derived?

You need a model

• Logistic regression model (transplant), Poisson model (dialysis)

• Adjusted for demographics, comorbidities (donor and XM variables)

• With this model, derive a probability of “expected” failure for each person / graft based on covariate matrix

• Compare this with actual outcomes

www.anzdata.org.au

Which predictors are important?

Recipi

ent a

ge ge

nder

& graft

numbe

r

+com

orbidi

ties

+ HLA

matc

hing

+ isc

haem

ic tim

e

+ don

or ag

e

+ cau

se do

nor d

eath

00.10.20.30.40.50.60.7

Harrell's CSomer's D

Predictive power of multivariate Cox model predicting graft survival, all DD transplants 2001-2009, with sequential addition of covariate groups

www.anzdata.org.au

• Factors within the control of centre– These may be why a particular centre gets

good or bad results• Factors that occur as a result of treatment

decisions• For example, don’t adjust for

– Choice of dialysis modality, HD access– Use of immunosuppressives, rejection, 1

month graft function…

Don’t adjust for…

Other graphical demonstrations of output

• Funnel plots are a static measure and summarise performance (relative to a comparator) over a fixed period of time.– Lack a dynamic element– Weight recent and distant results equally

Adding time – CUSUM

0

100

200

300

400

Num

ber o

f tx

-4

-2

0

2

4

Cum

ulat

ive

sum

O-E

01jan2004

01jan2005

01jan2006

01jan2007

01jan2008

01jan2009

Tx date

Twoway CUSUM for a transplant centre

Removing credit for good deeds

0

1

2

3

4

5

Cum

ulat

ive

sum

O-E

0 100

200

300

400

Tx number

Oneway CUSUM for for a hospital

Do we need to do more?

Why KPIs?

• Mortality is an insensitive and late indicators of problems– Hopefully rare– Outcome of complex series of events

• Incompletely ascertained– Important to monitor as best we can

• Key Process indicators– Simpler to understand, easier to address– Need to be valid and correctable (and related

to meaningful outcomes)

KPI Project

• Dialysis KPI project commenced 2011 – At instigation of DNT committee

• 2 markers chosen – Peritonitis and HD access at first treatment– Deliberately limited to existing data collection

• NO additional data collected– Based on real time ANZDATA data collection

Variation in HD access

8

5 5

40

31

65

18

66

28

10

57

2522

113

17

3668

810

15

7

47

19

12

295275

43445229

11

34344144

12

4464

20

1017

3144331923

139

2815

54

6 8

0

.2

.4

.6

.8

1

Pro

porti

on o

f lin

es

0 20 40 60Centres

Proportion 95% CI

ANZDATA, access at first HD where first dialysis

Variation in peritonitis rate

27

24

12

6

4

Pat

ient

-mon

ths

per e

piso

de

.5

1

1.5

2

2.5

3

Epi

sode

s pe

r pat

ient

-yea

r

Confidence intervals not shown where upper limit >3Units with <5 person-years PD over 2009 not shown

2009 onlyPeritonitis rates by treating unit

KPI reporting -- access

• Quarterly identified feedback to units

Peritonitis reporting

Where to from here?

• COMMUNICATE• Improve data collection• Improve access to results• Enhance reporting

– Add peritonitis rates– Access subdivided by late referral– Graphs etc etc

• Or is it all just too hard?

How do we view quality?

Centre reports -- SMR