Tom MarshallDepartment of Public Health & Epidemiology,

University of BirminghamT.P.Marshall@bham.ac.uk

Understanding VariationUsing Registry data to drive improvement - what makes

clinicians take note of statistics

Acknowledgements

Mohammed A. Mohammed,

Department of Public Health & Epidemiology, University of Birmingham

Quality & Variation• Conventional tools

– Standard setting• Clinical Audit

– Ranking or League tables

– Hypothesis testing

• Effect– Pass or fail

– Action on those that fail

• Another way ..

Shewhart’s Concepts• Letter a

COMMON CAUSEACTION: PROCESS

SPECIAL CAUSEACTION:

FIND & ELIMINATE

PROCESS OF WRITING

Basis of Control Limits

• Tchebycheff’s theorem

X mean +/- t SD

P > 1 - 1/t2

• t=3

• Economic– common cause vs special cause

Special cause variation - action

1. Data • Accuracy• Definition of errors

2. Raw materials• Difficulty of tasks

3. Equipment, facilities, staffing• Typewriters, workload, lighting

4. Processes, procedures• How is the work organised?

5. People• Skill levels and techniques

Application to Health Care

• Case studies

Surgeon Variability• Surgeon Survived Died %• A 82 16 16• B 58 8 12• C 49 9 16• D 45 7 13• E 37 15 29• F 41 5 11• G 35 3 8• H 26 11 30• I 31 5 14• J 27 7 21• K 28 4 13• L 19 2 10• M 18 3 14

McArdle & Hole BMJ 1991;302:1501-5

Conclusions: “There were significant variations in patient outcome among surgeons after surgery forcolorectal cancer; such differences compromisesurvival. A considerable improvement in overallsurvival might be achieved if such surgery wereundertaken by surgeons with a special interest incolorectal surgery or surgical oncology.”

Surgeon Variability

McArdle & Hole BMJ 1991;302:1501-5

A

BC

D

E

F

G

H

IJ

K

LM

X Number alive

Y N

um

nber

die

d

..

Common cause variation

Fractured Hips

• 90 day mortality (N=580; 104 deaths 18%)– Hospital Mortality

• 1 19/79 24%

• 2 5/24 21%

• 3 16/79 20%

• 4 19/80 24%

• 5 12/80 15%

• 6 4/81 5%

• 7 14/79 18%

• 8 15/63 19%

Todd et al BMJ 1995;301:904-8

Conclusions: “Uncritical acceptance of the "advantages" of hospital 6 should, however, be avoided as random variationalmost certainly plays some part in these findings.”

Fractured Hips

X Number alive

Y N

umbe

r di

ed

Todd et al BMJ 1995;301:904-8

Special cause variation

Western Electric Company Rules

• Additional rules for detecting special causes– 1 data point >3 sigma from mean– 2 out of 3 data points >2 sigma from mean– 4 out of 5 data points >1 sigma from mean– 9 successive data points on one side of mean– Trend of 6 successive data points

Special cause variation: nine successive data points below the mean

Renal Registry Data

• Average haemoglobin per quarter

Run chart of quarterly mean Hb

9

10

11

12

13

14

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

Quarter

Qu

arte

rly

mea

n H

b

UK

Average

Run Chart – sequential data points + mean

Trend of 6 data points

Interpretation

• Rising trend in mean Hb nationally

• Difficult to interpret changing Hb in a single centre except in relation to rising trend nationally

Run chart of quarterly mean Hb

9

10

11

12

13

14

Jan-

97

May

-97

Sep-9

7

Jan-

98

May

-98

Sep-9

8

Jan-

99

May

-99

Sep-9

9

Jan-

00

May

-00

Sep-0

0

Jan-

01

May

-01

Sep-0

1

Jan-

02

May

-02

Sep-0

2

Jan-

03

May

-03

Sep-0

3

Jan-

04

May

-04

Sep-0

4

Jan-

05

May

-05

Sep-0

5

Jan-

06

May

-06

Sep-0

6

Quarter

Qu

arte

rly

mea

n H

b

Middlb

Average

XMR chart of difference between quarterly mean Hb and national average

-2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0Ja

n-97

Jul-9

7

Jan-

98

Jul-9

8

Jan-

99

Jul-9

9

Jan-

00

Jul-0

0

Jan-

01

Jul-0

1

Jan-

02

Jul-0

2

Jan-

03

Jul-0

3

Jan-

04

Jul-0

4

Jan-

05

Jul-0

5

Jan-

06

Jul-0

6

Qu

arte

rly

mea

n H

b

Middlb

Average

+3 sig

-3 sig

Consistent with national average

9 data points above mean i.e. own long term average

Below national average

10 data points below mean

XMR chart of difference between quarterly mean Hb and national average

-2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0Ja

n-97

Jul-9

7

Jan-

98

Jul-9

8

Jan-

99

Jul-9

9

Jan-

00

Jul-0

0

Jan-

01

Jul-0

1

Jan-

02

Jul-0

2

Jan-

03

Jul-0

3

Jan-

04

Jul-0

4

Jan-

05

Jul-0

5

Jan-

06

Jul-0

6

Qu

arte

rly

mea

n H

b

Middlb

Average

+3 sig

-3 sig

XMR chart of quarterly mean Hb

9

10

11

12

13

14

Jan-

97

May

-97

Sep-9

7

Jan-

98

May

-98

Sep-9

8

Jan-

99

May

-99

Sep-9

9

Jan-

00

May

-00

Sep-0

0

Jan-

01

May

-01

Sep-0

1

Jan-

02

May

-02

Sep-0

2

Jan-

03

May

-03

Sep-0

3

Jan-

04

May

-04

Sep-0

4

Jan-

05

May

-05

Sep-0

5

Jan-

06

May

-06

Sep-0

6

Quarter

Qu

arte

rly

mea

n H

b

Truro

Average

Run Chart – difference between this centre + UK Average

9 data points above mean

Average determined from first 8 data points

Control Chart – Run Chart + 3 sigma limits

XMR chart of quarterly mean Hb

9

10

11

12

13

14

Jan-

97

May

-97

Sep-9

7

Jan-

98

May

-98

Sep-9

8

Jan-

99

May

-99

Sep-9

9

Jan-

00

May

-00

Sep-0

0

Jan-

01

May

-01

Sep-0

1

Jan-

02

May

-02

Sep-0

2

Jan-

03

May

-03

Sep-0

3

Jan-

04

May

-04

Sep-0

4

Jan-

05

May

-05

Sep-0

5

Jan-

06

May

-06

Sep-0

6

Quarter

Qu

arte

rly

mea

n H

b

Truro

Average

+3 sig

-3 sig

XMR chart of quarterly mean Hb

9

10

11

12

13

14

Jan-

97

May

-97

Sep-9

7

Jan-

98

May

-98

Sep-9

8

Jan-

99

May

-99

Sep-9

9

Jan-

00

May

-00

Sep-0

0

Jan-

01

May

-01

Sep-0

1

Jan-

02

May

-02

Sep-0

2

Jan-

03

May

-03

Sep-0

3

Jan-

04

May

-04

Sep-0

4

Jan-

05

May

-05

Sep-0

5

Jan-

06

May

-06

Sep-0

6

Quarter

Qu

arte

rly

mea

n H

b

Truro

Average

+3 sig

-3 sigConsistent with two stable processes:

before Sept 04& after Sept 04

Special cause variation - action

1. Data (including definitions)

2. Raw materials (case-mix)

3. Equipment, facilities, staffing

4. Processes, procedures

5. People

Monitoring Many Centres

Walter A Shewhart 1931

“The central problem in management and leadership …is failure to understand the

information in variation”

William E Deming 1986 Out of the Crisis MIT pg 309

Summary• Shewhart’s concepts

– Understand variation– Simple & powerful– Guide action– Wide application

• Continual improvement• Clinical governance• Other implications ..

Implications• Prediction

– Limits of common cause variation– Statistical control

• Action– Common cause variation

• League tables, ranking, hypothesis testing all misleading• Improve process/system as a whole

– Special cause variation• Investigate and eliminate (or learn lessons)

• Data order important

How It Works In Industry

• Balanced set of measures

Balanced Set Of Measures

Customer

Financial

Internal ExternalAim

Four or five measures for each box

Balanced Set Of Measures

Patient Experience

Financial / Resources

Clinical effectiveness

Strategic Effectiveness

Aim

Special Causes

• Identify special causes in each domain

• Collate & prioritise for action– Low hanging fruit first