Knowledge Translation:The steep path between evidence

generation and application

Brian HaynesHealth Information Research Unit

Dep’t of Clinical Epidemiology and BiostatisticsMcMaster University

KNOWLEDGE IS THE ENEMY OF DISEASE

The application of what we know will have a bigger impact on health and disease than any single drug or technology likely to be introduced in the next decade. Sir Muir Gray, UK National Library for Health

Knowledge Translation…

…the organization, retrieval, appraisal, refinement, dissemination, and uptake of knowledge (eg, important new knowledge from health research)

Generalizable knowledge for better clinical practice and

healthcare • knowledge from research

(sometimes called evidence)• knowledge from the analysis of

routinely collected and audit data (sometimes called statistics)

• knowledge from the experience of clinicians and patients.

Cost-effectiveness of warfarin*

• Warfarin for atrial fibrillation– $25CDN saved per stroke

averted

• Aspirin for atrial fibrillation– $65CDN saved per stroke

*Gustafsson C, et al. Cost effectiveness of primary stroke prevention in atrial

fibrillation: Swedish national perspective.

BMJ. 1992;305:1457-60.

What proportion of patients with atrial fibrillation do not

receive anticoagulants?

50%

Bradley BC, et al. Frequency of anticoagulation for atrial fibrillation and reasons for its non-use at a Veterans Affairs medical center. Am J Cardiol. 2000 Mar 1;85(5):568-72.

In Hamilton, Ontario, “The Clot Capital of the Universe,”

the proportion of medical inpatients receiving clot prevention according to

guidelines is…

…33%

Current guideline adherence for diabetes

Intervention:Ophthalmology assessment… 46% - 80%Proteinuria assessment… 35% - 82%Foot assessment… 30% - 72%HbA1c… 16% - 87%

Cholesterol assessment… 55% - 68%Smoking status assessment… 25% - 87%

In all, 73% of microalbuminuric patients were not on ACE-I/ARB. Hypertensive type II diabetic patients were often left untreated and only a minority of those treated were optimally controlled. The importance of an elevated systolic pressure is underestimated and the number of antihypertensive drugs prescribed, insufficient. Screening and treatment of albuminuria are inadequate.

The routine application of what we know can prevent

or minimise:• unknowing variation in clinical practice• errors of commission and omission• unsatisfactory patient experience

Evidence (from research) is necessary but, of course, not sufficient…

...it has to be combined with the circumstances of the individual patient and the values of each patient. But without evidence it is improbable that patients, professionals, and those who manage resources, will to make good decisions.

researchers

decision makers

Steps from evidence generation to clinical application

Steps: 1. generation of evidence from research; 2. evidence summary and synthesis; 3. forming clinical policy; 4. application of policy; 5. individual clinical decisions, including a) patient’s circumstances, b) patient’s wishes, and c) evidence from research

a

bc

1

generation

2

synthesis

3

policy

4

application

5

decisions

Knowledge Translation

CIHRMRC

Barrier Solutions

• too little research addressing “real world” problems

• large, simple randomized trials• “head to head” comparisons

Step 1. Generating Research Evidence

Barrier Solutions• size and noise of the research enterprise

• research into rating, abstracting, and synthesizing research

Step 2. Synthesizing Research Evidence

How much synthesis do we need?

“..at least 10 000 Cochrane reviews are needed to cover a substantial proportion of the studies relevant to health care that have already been identified”

Susan Mallett & Mike Clarke

ACP Journal Club. 2003 Jul-Aug;139:A11.

When will we have our 10,000 reviews?

“…between 2010 and 2015”.Mallett&Clarke, ACPJC 2003

Growth of Cochrane Reviews and Protocols

1995

2003

2000 completed mid-2004

reviews

protocols2500 completed mid-2005Non-Cochrane reviews: >50% of all reviews

Barrier Solutions

• problems in developing evidence-based clinical and health policy

• national drug and technology assessment agencies• local leadership

Step 3. Developing Policy

Step 4. Applying evidence in practice

Barrier Solutions• poor access to current best evidence and guidelines

• development and testing of information systems that integrate evidence and guidelines with patient care(eg Diabetes In-CHARGE)

The McMaster PLUS project

• only a tiny proportion of all research is “ready for application”

• only a tiny fraction of the “ready”

research is “relevant” to the practice of a given clinician

• only a tiny proportion of the “relevant” research for a given practitioner is “interesting” in the sense of being something new, important, and actionable.

50,000 articles/yfrom 120 journals

~2,500 articles/ymeet critical appraisaland content criteria(95% noise reduction)

Evidence-Based Journals

Critical Appraisal Filters

~2,500 articles/y meet critical appraisaland content criteria(95% noise reduction)

McMaster PLUS Project

Clinical Relevancy Filter (MORE)

~20 articles/yr for clinicians (99.96%noise reduction)

~5-50 articles/y for authors of evidence-based clinical topic reviews

Dear Dr. Jones,

We want to alert you to NEW articles in the PLUS system. These articles that have received very high relevancy and newsworthiness scores:

1. Bohlius J, et al. Erythropoietin for patients with malignant disease. Cochrane Database Syst Rev 2004;(3):CD003407.  

Rated by: IM/General (patients referred from Primary Care)

Relevance: 6 of 7

Newsworthiness: 6 of 7

 

2. Gourlay S, et al. Clonidine for smoking cessation. Cochrane Database Syst Rev 2004;3:CD000058.  

Rated by: IM/General (patients referred from Primary Care)

Relevance: 6 of 7

Newsworthiness: 6 of 7

We hope that you will find these articles of value in your clinical practice.

Best wishes from the PLUS Team

203 randomized: 10 communities

6 small clusters 4 large clusters

Group 1 (3) Group 2 (3) Group 1 (2) Group 2 (2)

7 refused consent

344 consent eligible

2 left study

134 non-respondent

PLUS Trial – Northern Ontario Physicians

Intervention

Self Serve Version• Ovid• Stat!Ref• Pyramid of Evidence

Full Serve Version• Ovid• Stat! Ref• Pyramid of Evidence• PLUS Email Alerts• PLUS Search Engine

• Randomization to 2 different trial interfaces

PLUS Preliminary Findings: % of Participants Using PLUS by Month

Pe

rce

nta

ge

Us

ing

PL

US

Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun

03 03 04 04 04 04 04 04 04 04 04 04 04 04 05 05 05 05 05 05Month

70

60

50

40

30

20

10

0

Baseline (5 mo) Self-serve vs Full-serve Full-Serve

Self-serve Full-Serve

Relative increase 58.7%, P=0.001RCT begins Control cross-over begins

Free EBM literature updating service

http://www.bmjupdates.com

Free at www.bmjupdates.com! (sponsored by BMJ Publishing Group)

Step 4. Applying evidence in practice

Barrier Solution

• ineffectual continuing education

• effective continuing education and quality improvement programs for practitioners

Step 4. Applying evidence in clinical decisions

Barrier Solution

• ignorance about barriers and their solutions

• shift a portion of health investment from services to quality improvement

WHO estimates US$100B/yr for health-related research

• not enough is for application research

• the balance is shifting slowly• should there be a Nobel Prize for

applied research?

Step 5. Making better clinical decisions

Barrier Solutions

• not having the right information at the right time

• Computerized decision support

Effects of Computerized Clinical Decision Support Systems on Practitioner Performance and Patient Outcomes

A Systematic Review

Amit Garg MD, Neill Adhikari MD, Heather McDonald MSc,

Patricia Rosas-Arellano MD,PhD, Phillip J. Devereaux MD,, Joseph Beyene PhD, Justina Sam, R. Brian Haynes MD, PhD

Departments of Clinical Epidemiology and Biostatistics, McMaster UniversityDepartments of Medicine, McMaster University, University of Toronto, and University of Western OntarioDepartment of Biostatistics and Epidemiology, University of Western Ontario

Ref: Garg et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005;293:1323-38.

Context – Computerized Clinical Decision Support

Systems

Software designed to directly aid in clinical decision making in which characteristics of individual patients are matched to a computerized knowledge base for the purpose of generating patient specific assessments or recommendations.

Rules / Algorithms

Computer

INPUTPatient characteristics• Automated through EMR• By extra research staff• By existing health care staff• By the patient• By the practitioner

OUTPUTRecommendations delivered to health care provider• Directly by computer• By pager• By extra research staff• By existing health care staff

Outcomes• Provider performance• Patient outcomes integrate into

workflow

Are CDSSsclinically effective?

Did CDSS improve practitioner performance?100 studies “counting positive results on ≥ 50% outcomes measured”

In 16 of 21 (76%) reminder systems

In 24 of 37 (65%) disease management systems

In 19 of 29 (66%) drug dosing or prescribing systems

In 4 of 10 (38%) diagnostic systems

Examined in 97 studies, 63 cited improvement (65%)

Did CDSS improve patient outcome?Update 100 studies

most had inadequate power to detect important difference

none proven to improve definitive outcome such as mortality

surrogate outcomes such as BP and HbA1C not meaningfully improved in most studies

Examined in 52 studies, 7 cited improvement (13%)

Screening, counseling, vaccination, testing, medication use, or the identification of at-risk behaviors

CDSS successes were typically demonstrated in ambulatory care, although one successful system was used in hospitalized patients

Improved Practitioner

Performance- 76% -

Improved Patient Outcome

- 0% -

Reminder Systems40 studies

Most are RECOMMENDATIONS.

Range of problems, for example: - diabetes care - cardiovascular prevention- incontinence in the elderly - advanced directives - ventilator support - infertility - corollary orders - reduce unneeded health care utilization

Improved Practitioner

Performance- 62% -

Improved Patient Outcome

- 19% -

Disease Management Systems37 studies

Step 5. Improving health care decisions

Barrier Solutions• low patient adherence to treatments

• adoption of effective strategies to assist patients to follow evidence-based health care

The weakest links

• Policy - especially at the local level

• Coordination - 4P• Helping practitioners to

recommend effective treatments• Helping patients to follow

effective treatments

The strongest link

• Organization of health care knowledge according to the hierarchy of evidence (evidence-based medicine)

Systems

Summaries

Synopses

Syntheses

Studies

Examples

Computerized decision support

Evidence-based textbooks

Evidence-based journal abstracts

Systematic reviews

Original journal articles

The evolution of Evidence-Based information systems

KNOWLEDGE IS THE ENEMY OF DISEASE

The application of what we know will have a bigger impact on health and disease than any single drug or technology likely to be introduced in the next decade.

Sir Muir Gray, UK National Library for Health