Application of research to quality improvement Helen CrispThe Health Foundation, London UK

2

Being ‘scientific’ about health care improvement

Improvement needs to be as ‘evidence-based’ as any other aspect of health care provision

Effective, theory-based interventions using tested methods

Change demonstrated by robust measures: • process• clinical outcomes• patient experience

3

Why is research important for improvement?

To understand what we’re trying to do

To measure what we are doing and if it’s working

To report our work so others can learn

4

Understanding what we’re trying to doTo boost chances of getting desired results - improvement interventions need to:

be focused on a well-defined issue

use research evidence on approaches that have been tried

build on previous work that showed good results

and

learn from interventions demonstrated as ineffective

5

E.g. You want to improve hand washing rates

What approaches have been tried elsewhere?

What has shown little effect?

What works?

Where did it work?

Is it likely to work for us?

6

Finding relevant research

It can overwhelming!

Refine your search terms

Look for:

systematic reviews

meta analyses

Work with your local research leads

Sign up for regular research up-dates:

7

Health Systems EvidenceMcMaster University - Canada

8

Other sources

Good practice repositories

Examples:

NICE Quality and Productivity proven case studies

BMJ Quality Reports

Other national patient safety and quality improvement agencies

9

10

11

12

PollTo date, has your improvement work been based on research evidence?

• YES, TOTALLY

• YES, TO SOME EXTENT

• NO, WE GENERATE OUR OWN IDEAS

• NOT SURE

13

Benefits of research-based improvement

Not starting from scratch

Benchmark your results against reported findings

Counters the inevitable question from clinicians when introducing a change:

“What’s the evidence for this?”

14

THE EVIDENCE IS OUT THERE

15

Using research methods to do improvement work

To improve effectiveness of implementation; improvement programmes need to be based on evaluated methods, based on explicit theories about how and why the intervention is expected to work.

 

16

Introducing ‘Theory of change’

Image credit: Sidney Harris

“I think you should be more explicit here in step two”

17

What is a ‘Theory of change?A comprehensive description and illustration of how and why a desired change is expected to happen in a particular context.

The theory of change sets out explicit statements on the components of an improvement programme (its activities or interventions) and how these are expected to lead to achieving the desired goals.

First identifying the desired long-term goals, the theory of change works back from these to identify all the conditions (outcomes) that must be in place (and how these related to one another causally) to reach the goals.

Behaviour change:- Chronic conditions more effectively managed

OUTPUTS

Fewer visits to surgery

ACTIVITIES OUTCOMESGOAL

Advertising campaign

Equitable access to resources

and interventions

Pain clinics

Reduction in medications

over time

# Self Referrals

Recognising & reporting of adverse

effects

Physical activity

workshops

ENABLING FACTORS

Interventions targeted at specific population; Local stakeholders buy-in to champion, Funding continues to sustain, Effective monitoring and evaluation

THEORY OF CHANGE : Managing chronic conditions; pain, fatigue, shortness of breath

Dietary and

Nutritional advice

Lower BMI

Individuals feel more

empowered

Accuracy in self

administering

Reduction in pain

< Negative emotion

s

Psychotherapy

Smoking cessation

workshops

19

Developing a theory of change

Keep it simple • Use language that’s easy to understand• Keep your diagram on one side of paper/screen view

Keep it relevant • Focus on the key elements of the intervention • Explore the assumptions linking action and expected

outcomesKeep it updated

• Return to the theory of change at regular intervals• What has changed during the implementation?• Does the theoretical link between actions and outcomes

hold up in practice?

20

Measurement is key to quality improvement research

21

Measuring improvementUnless you measure you do not know if there has been an improvement

What to measure and how to measure it?

Measure key elements of the intervention

Use routinely collected data where possible

Be precise about data definitions

Measure over time

Use robust techniques such as statistical process control charts

22

Typical measuresProcess measures• error rates e.g. prescription errors• compliance rates e.g. completing a checklist

Clinical outcome measures• complication rates e.g. of a surgical procedure• infection rates – healthcare associated infections

Resource use • length of hospital stay• number of medications prescribed

Patient experience• involved in decisions about care• treated with dignity and respect

23

Evaluating improvement interventions

Key evaluation question:

Did the improvement intervention fulfil its intended objectives?

Sub-questions:

How was this achieved?

What resources did it take?

What unintended results were there?

24

Evaluation approaches

Summative• Summarises the intervention effect at the end

Formative• Findings shared and help to shape the intervention

Rapid cycle• Frequent review of effectiveness of intervention

Developmental • Intervention is still developing, all aspects reviewed

and changes made in response

25

External evaluation

Often part of large scale, national or multi-site change programmes

Independently commissioned from specialist teams

Great learning opportunity (NOT a threat!)

Share experience - the good and the challenging

26

27

PollDo you think it is important that quality improvement work is written up for publication?

• YES

• NO

• NOT SURE

28

Helping to build the evidence base

This section based on presentations by Dr Kaveh Shojania, Editor in Chief, BMJ Quality and Safety

Better reporting of improvement work will:

Help spread successful improvement interventions

Prevent wasted effort on repeating interventions that don’t work

29

Sharing and learning moreReports on improvement work need not only results but also:• how the initiative was designed• the setting where it was implemented• detail on the core components• measures and data used to measure the change• challenges overcome along the way• how they were overcome• what the team would do differently in the future

30

Reporting bias

When reporting a successful intervention many improvement reports amount to:

“See, we did X!”

versus

“Here’s what we had to do to

achieve X”

Higher tendency to write up reports and submit papers and abstracts when the improvement is ‘successful’ We can a lot from what didn’t work

31

Reporting to facilitate spread

Improvement reports need to provide enough detail:

to convey credibly that something worked

to give insight on the action needed to replicate the results in another clinical setting

32

Credibility and replication• Too often improvement reports lack important

details about key components of intervention and

institutional context

− Readers cannot know if it’s worth trying in their setting

• No information is given on barriers or problems to

implementation

− No improvement effort works immediately, this

absence decreases credibility

33

A typical QI reportIntroduction

Hospital infections affect thousands each year

Hospital staff do not wash their hands consistently

We implemented a multi-faceted strategy:

• Staff education

• Clinical champions

• Empowering patients to ask staff if they have washed their hands

Methods

Briefly stated design, data collection strategy and main outcomes, plus some mention of PDSA

Results

We improved hand hygiene by 50%

Discussion

Patient empowerment can be effective

34

What is lacking here?Introduction

Hospital infections affect thousands each year

Hospital staff do not wash their hands consistently

We implemented a multifaceted strategy:• Staff education

• Clinical champions

• Empowering patients to ask staff if they have washed their hands

No connection between the introduction material and specific features of the intervention

35

A better approachIntroductionCommonly identified barriers to hand hygiene compliance include A, B, and C 

Staff education, clinical champions, and empowering patients address A, B, and C by doing X, Y, and Z

• This introduction makes clear what factors explain poor hand hygiene

• And, it makes explicit why the intervention includes these ingredients

• This “theory for the intervention” will pay off in writing the report and interpreting the results

36

Is it clear what you did?Methods

Briefly stated design, data collection strategy and main outcomes, plus some mention of PDSA

Results

We improved hand hygiene by 50%

Discussion

Patient empowerment can be effective

‘PDSA’ needs context to make sense!

Simply saying; ‘We carried out three PDSA cycles’ is not informative. What did the ‘study’ of what you had ‘done’ reveal and how did you ‘act’ as a result?

37

A better approachMethodAfter the first round of staff education we reviewed the delivery mechanism and feedback from participants, using PDSA methodology. It was reported that timing of training sessions was an issue in getting staff attendance, so the next sessions were planned with ward managers.

Participants wanted more visual material to illustrate key points - these were designed with staff and used in subsequent sessions.

• This provides more detail which makes the report credible

• Others are likely to have the same issues and could avoid making the same mistakes

38

Points to consider:

When do you start ‘writing up’?

How to capture the key components of the improvement initiative?

When barriers arise - how do you record these- and the action to overcome

them?

39

Some tips for writing up

Writing always takes longer than you think – don’t leave it to the last few weeks

Robust data collection from the outset is vital

Keep an ‘improvement diary’ to help capture information as you go along, particularly the adjustments

40

Involve others

Include different perspectives in your write up;

not just the improvement lead

other staff involved

staff not involved but affected by the change

service users

41

Consider the audience and where to publish

Core information remains but different emphasis for:

A report to the funder

Academic publication in a peer reviewed journal

Publication in a professional practice magazine

Beyond text:

Photos, videos, animations - bring the work to life

42

A tool to help

SQUIRE guidelines

Standards for Quality Improvement Reporting Excellence

Checklist of points to consider when writing up improvement work

BUT• Don’t leave it until you’ve completed the work• Use guidelines to consider what data to capture as you go

along

43

44

45

Download SQUIRE guidelines from:

http://www.squire-statement.org/

46

Spreading the word

Professionals listen to their peers

Think of a range of approaches - and use every opportunity:

Professional seminars

Conferences

E-mail bulletins and newsletters

Blogs

Twitter

47

Conclusion

Find and use existing evidence

Actively use robust research methods

Contribute to building the evidence

48

Conclusion

Find and use existing evidence

Actively use robust research methods

Build the evidence