Leeds Institute of Health Sciences

Getting better evidence

Stephen Morley

Why bother with trials … ?

From: Moore & McQuay ‘Bandolier’s little book of making sense of the evidence’ 2006

Knowledge

Wisdom

Sys Reviews& Meta-anal

Evidence in clinicalpractice

DistillationIntegration

Quality

ExperienceValues

Conditions

InformationSingle RCTs

Trials

Good Poor

ReviewsGood Ideal May

helpPoor Can

repeatWill

mislead

From: Moore & McQuay ‘Bandolier’s little book of making sense of the evidence’

Issues

• Trial quality– Design, size matters– Quality and effect size

• Outcomes– Variety, validity and ‘clinical’ relevance– Efficacy and effectiveness

• Treatment content and coherence– Is there a model?– Mediation?

Trial quality

Cumulative trials over years

Hoffman et al 2007

Morley et al 1999

Words of caution …

What to count?Quality …Content …

CBT on the label ‘may not be CBT in the tin’

What is CBT in this context ?

Tools for assessing quality

Ideal May help

Can repeat

Will mislea

dFilter out poor quality trials by setting cut-offs

Investigate influences of feature on

conclusions – Meta-regression

WHY?

Trials

+ -

+

MA

-

Quality scales: The Jadad Scale

1. Is the trial randomised (1 point)+1 point if method described and appropriate

2. Is the trial double blind (1 point)+1 point if method given and appropriate

3. Is there a description of withdrawals and drop outs (1 point)

Suggested cut-off = 3

It’s simple Quick Captures major biases Can be reliable with basic training

But

Criterion 2 eliminates all complex interventions

Doesn’t capture important features of psychological trials

Quality scale for psychological trials

Yates et al, Pain 2005: 117; 314-325

Identify and recruitDelphi panel

Panel generates and agreesItems: 3 rounds

Expert panel writes QS

Reliability and validity studies using novice and expert raters

Datafrom 31

published trials

Final QS

Quality scale for psychological treatment trials

32 parts

Is there a good description of the sample in the trial?

Sample characteristics 0 1

Group equivalence 0 1

44 parts

Have adequate steps been taken to minimise biases?

Randomisation 0 1 2

Allocation Bias 0 1

Measurement Bias 0 1

Treatment expectations 0 1

Reliability

ICC absolute agreement

Full scale > 0.9

Treatment quality > 0.9

Design quality = 0.85

Kappa for items

range 0.0 to 0.74

Agreement coefficient for items

>80%

Strengths and weakness of psychological trials

Yates et al, Pain 2005: 117; 314-325

Design

Treatment

Quality over time – the good news

Morley, Eccleston & Williams, unpublished

Effect size and quality

Yates et al, Pain 2005: 117; 314-325

TotalQSβ = -.35 , p =.057

Treatment QS ns

Design QS, β = -.4 , p <.05

Size matters

Data from Hoffman et

al 2007

Outcomes

Outcomes

underlying scale

x►

y►

z►►

x►

dysfunctional or clinical sample

functional or normal sample

a bc

►

CSC criteria

Morley in McQuay et al ‘Systematic reviews in pain research’ 2008, IASP press

Turning continuous outcomes into dichotomous ones

0

5

10

15

20

25

30

35

40

0 5 10 15 20 25 30 35 40

Pre-treatment

Po

st-

treatm

en

t

Deteriorated from pre-treatment good functioning

Reliably deteriorated

Reliable improvement but not clinically significant

Reliable and clinically significant improvement

No reliable change

Reliable improvement but not clinically significant

Reliably deteriorated

Heterogeneity of outcomes in trials

0

5

10

15

20

25

30Pain experience

Mood/ affect

Social role

Cognitive

Behavioural activity

Biological

Health care use

Miscellaneous

Number oftrials usingthe measure

Mean numberof measuresper trial

Data from Morley et al, Pain 1999: 80; 1-13

IMMPACT core outcomes

1. Pain

2. Physical Functioning (interference/disability)

3. Emotional functioning

4. Global improvement

5. Symptoms/adverse effects

6. CONSORT data

Dworkin et al, Pain 2005: 113; 9-19

Stakeholders and outcomes:who wants what change?

0

5

10

15

20

25

30Pain experience

Mood/ affect

Social role

Cognitive

Behavioural activity

Biological

Health care use

Miscellaneous

Health care provider

Researcher

The patient

Employers

What outcome do you want?

Sleep

Weakness

Fatigue

Emotional well-being

Enjoyment of life

Doing tasks

IMMPACT, 2008 Pain:137; 276-285

How much change do you want?

Mdn % change desired

ES(d) %meeting RCI

%meeting CSC

Severity 60 1.45 61.2 20.9

Impact 75 1.70 73.2 57.7

Interfere 66 1.82 75.0 63.2

Activity 44 1.38 36.8 16.2

Thorne & Morley in preparation

How much change do you want?

Thorne & Morley in preparation

Interference Pain severity

The evidence cycle

Efficacy studiesRandomisedControlled

Trials

Evidence-based practiceas policy

Practitioners

Practice-based evidence

Effectiveness studiesRoutineClinical

Treatment

Practitioners

Practice based evidence

Morley, Williams & Hussain, Pain 2008; 137: 670-680

Outcome categories - efficacy

8 58

3 5 4

20 6 20

Crude NNT values

Morley, Williams & Hussain, Pain 2008; 137: 670-680

Benchmarking from RCT data

From Minami et al J Consult Clin Psychol 2007;75: 232-43

Effectiveness + benchmark

WLC group

Tx

Group

RCT is: Williams et al. Pain 1996;66(1):13-22.

Treatment

Quality controlling treatment

• Manuals – protocols?• Training for therapist and teams?• Supervision?

• Patient monitoring systems?

Is there a model: what’s in the tin?

Unpublished data from Morley et al, Pain 1999: 80 1-13

CBT treatment components across trials

What’s the model?

• Generic– Principles of engagement and delivery

• Collaborative, information provision / education – Changing key ‘cognitive appraisals’ through behavioural

experimentation– Techniques: principled application or self service store?

• In PMP/CPM programmes embedded within pharmacotherapy, functional restoration, medical management– How coherent and integrated are they?

• Developing more specific models? A debate (JV)

Thanks to …

• Chris Eccleston• Amanda Williams• Henry McQuay• Andrew Moore

• Wendy Callaghan

• Johan Vlaeyen• Lance McCracken

• Shona Yates• Sumerra Hussain• Fiona Thorne

Seen in Leeds ….

PAINis just

weakness

leaving

the

BODY