Embedding Evidence in Context:

Approaches Used by an Embedded

Research and Evaluation UnitColleen J. Metge, BSc(Pharm), PhD

Director, Research & Evaluation Unit

Winnipeg Regional Health Authority

Evidence in Context

Flower class corvettes left from St.

John’s NFLD 1941-1945 during

the Battle of the Atlantic escorting

25,000 merchant ships during

WW2.

They were ill-suited as anti-

submarine, open-ocean vessels

but were easy to build. They were

largely served by men from the

Prairies.

HMCS Trillium on the open seas with an American crew

(row boat) returning to their ship after a U-boat attack

K141:

HMCS Summerside

Creds for the Session

The Winnipeg Regional Health Authority’s

(WRHA) Research & Evaluation Unit:

• Produces HTAs on request

• Is an embedded unit established for Mode

2 (II) knowledge production

• It’s key value: collaborative research &

evaluation in specific health care contexts

The Forks at Sunset, Winnipeg

What is the WRHA?

• comprised of six hospitals (two tertiary), 35

personal care homes, and 12 community

areas

Hospital T

Hospital C

Communities

CEO

VP VP VP VP

Programs Programs Programs Programs

The art of policy and practice involves

reconciling the strength of published evidence

with its relevance based on the experience of

those who know, live and work with the

problem that the evidence is designed to

solve.Lawrence W. Green, 2001

20 January 2009 8

Mode 2 (II) Knowledge Production

• Mode 1 (I): traditional, academic norms of

scholarship in the disciplines & institutions

in which researchers work

• Mode 2 (II): non-hierarchical relationships

with stakeholders to collaborate on a

research issue situated in a specific health

care context

Mode 1 Mode 2

Academic context

Disciplinary

Homogeneity

Autonomy

Traditional quality

control (peer review)

Context of application

Transdisciplinary

Heterogeneity

Reflexivity / social

accountability

Novel quality control

Knowledge Production

Gibbons M et al. (1994) New Production of Knowledge

Nowotny H et al. (2001) Re-thinking Science: Knowledge & the Public in an Age of Uncertainty

Mode 2 research includes a wider,

more temporary and heterogeneous

set of practitioners, collaborating on

a problem defined in a specific and

localized context

Gibbons 2005

…for those who have never heard of Mode 2

knowledge production, “think” integrated KT and the

KTA or Knowledge to Action cycle

12

and, we are back to discussing how

we embed evidence in context

WRHA’s Research & Evaluation Unit’s

experience, including their expertise in

critical appraisal, proposal development and

evaluation and knowledge translation

activities.

em·bed ( ěm-běd ) v. em·bed·ded also v.tr.

1. To fix firmly in a surrounding mass: embed a post

in concrete; fossils embedded in shale.

2. To enclose snugly or firmly.

3. To cause to be an integral part of a surrounding

whole:

4. To assign (a journalist) to travel with a military unit

during an armed conflict.

WRHA’s Research & Evaluation Unit

Critical Literature Synthesis

ResearchEvaluation

Capacity

Building

WRHA

Board & Senior

Management

Knowledge

Translation

(External)CHA

Hypotheses generated

Ongoing scan of evidence

important to SNR MGT

Knowledge Translation

(internal)

Examples:

• MH-CRC

• BGSC

• CareLinks

• ACSS

• Pre-hab

Leverage

Funding

Opportunities

WRHA’s

Research & Evaluation

Unit: A functionalapproach (2009)

The most precious gift we can offer

others is our presence

Thich Nhat Hanh

WRHA R&E TEAM members

Research Credentials

Disciplinary Area

Liping Zhang MD, MSc Data analystStatistics

Sara Kreindler D.Phil. (Oxford)Rhodes Scholar

Social PsychologyResearcher/KT/KS

Ashley StruthersShannon Winters

MSc (OT)MSc BA (Honors)

EvaluationKnowledge Synthesis

Paul Beaudin MSc PhDSpeech Pathology

QualitativeQuantitative

Catherine Charette PhDMultidisciplinary

QualitativeEvaluation

Yang Cui MSc (Econ)Doctoral candidate

Data analystEconomics

Sunita Bapuji RN, MSc(Nursing) QualitativeKnowledge Synthesis

Ola Norrie BSc(Pharm), MSc, PhD(Economics)

QuantitativeSurvey Developer

Xibiao Ye MD, PhD(Epidemiology)

QuantitativeKnowledge Synthesis

Critical Appraisal

& Synthesis of

Evidence

Community

Health

Assessment

Research Evaluation Capacity-

building

Consultations

Evidence scans (ongoing):

Physician Assistants

(SB/SW/CJM)

Focused Report

**Promoting Health Equity

(XY/PB/CJM)

Predictive Hospitalization

Using MCHP Remote Access

System (YC/XY/CJM)

Birth Centre

**Consulting (SW/CJM)

MPH Students:

Tracey Dowson (Bell Hotel

+ Food Access) CC/CJM

Enhancing Collaborative Pt-

Centred Care

(WCIHC: SK/CJM)

eMental Health Knowledge

Synthesis (also research)

CIHR funded (XY +++)

Community Engagement

(meeting with CADs)

Producing Profiles (ON)

Patient-flow initiatives

(SK/CJM)dia with pregnant

women who smoke (HSFM)

MACHs (Prenatal

Connect)

Needs Assessment &

Evaluation (SW/CJM

Forecasting, Projections &

Predictive Modelling Group

CJM/XY/YC/LZ

Free First Visit (LZ)

Acute Care Surgical Services:

Cochrane protocol developed &

underway

**Promoting Health Equity

Describing the Problem (XY,LZ,YC,CJM)

Recommendation Task Team (PB)

Communication

HSFM: ENCOURAGE

project (T.Duhamel)

CJM

Cardiac Services:

Developing projection

models for future service

use (XY/CJM)

SMD Peer-led groups integrated

with the HC system (AS)

Crowdsourcing (CJM) CHA-Indicator Review

Committee (XY/CJM)

PATHS:

Teen pregnancy(CC/CJM)

Suicide

Advanced Care Planning

Dignity in Care

(PB/CC/SB)

Briefing Note advisory

function (see PBMA)

CJM

After Hours Access

(Outpatient Tx of Latent & Active

TB (ON)

evidence-review)

CHAN: Change in format of

CHA (XY/CJM/CC)

Bell Hotel Supportive

Housing Project

(CC/CJM)

Web-site development

(KC/CJM)

Latent TB data collection issues

in 2-WRHA clinics

- ACCESS for decision support

(ON/CJM)

Patient-reported Outcome

Measures (PROMs) CJM “Silos”

paper in revision

CHAN: Evaluation committee

(CC)

CIHR application:Est Effect of

Neighbor Deprivation on Mental

Health Disorder Disparity

(Child/Adolesc):

Life-course Epidemiology

XYe, C.Metge (resubmit?)

Orthopedic Surgery

Consolidation

SK/LZ/CJM

Priority setting

Program-based marginal

analysis (PBMA)

CJM (possible Unit in

future)

Integration of Tuberculosis

Services &

TB Services Stakeholder

EngagementCHAN Engagement processes

(CJM)

P.I.E.C.E.S. Evaluation

(SB/SW)

Social Identity & Physician

Participation in Primary Care (SK)

CHA-Advisory Committee

(reconvene Fall 2012)

Lawson Fdn: Pre-cardiac surg

diabetes care (CJM/ON)

PPHU Evaluation Planning

(SB/PB/CJM)

Birth projections for

Women’s program

Patient-centred care discourse

document

SK

RLEI: Lang/Ethnic Ques

+ language/ethnicity Indicators

(development & evaluation)

CJM/CC

MMF Regional Adaptation

Strategy (PB)

Community Mental Health

Crisis Response Centre

(CRC)

CC/ON/CJM

Happy Child Happy Smile

(LZ)

Ante-natal Home care program

Food Security Profile Downtown

Community Area (TD; CC)

Cancer Patient Journey (WRHA/CCMB)

CJM

Regional Research Policy Review

(PB)

Rehab (AS)

CIHR OT Instrument Devl

French Language services

(PB)

CCSD Census Data

Consortium (LZ; CC)

PATHS: Case study evaluation of interventions for child health

with an equity lens (CJM/CC)

FSAS-ED(OT: linked to ER

projects)

• Creating unity

Creating a group mandala is a unifying

experience in which people can

express themselves individually within

a unified structure.

Bamboo forest on the way to

Waimoku Falls near Hana, Maui,

Hawaii

Outline of eight steps informing Knowledge to Action evidence summary approach knowledge

users. Khangura et al. Systematic Reviews 2012, 1:10

Rapid Reviews: Evidence Summaries

Knowledge to

Action step

Task

Step 1

Step 2

Step 3

Step 4

Step 5

Step 6

Step 7

Step 8

Needs assessment

Question development and refinement

Proposal development and approval

Systematic literature search

Screening and selection of studies

Narrative synthesis of included studies

Report production

Ongoing follow-up and dialogue with knowledge users

Pulmonary Rehabilitation (PR)

Evidence Scan

What is the effectiveness of outpatient & community-based

pulmonary rehabilitation (PR) programs for persons with chronic

obstructive pulmonary disease (COPD)?

SUMMARY

PR is an effective intervention for patients with COPD and improves

exercise tolerance, health related quality of life and symptoms of

dyspnea. There is also some evidence that PR is cost-effective and

reduces health care utilization. However, the benefits of PR tend to

diminish over time.

BACKGROUNDPulmonary rehabilitation has been defined as “a multidisciplinary program of care

for patients with chronic respiratory impairment that is individually tailored and

designed to optimize …

My friend, drop all your

preconceived and fixed ideas and

be neutral. Do you know why this

cup is useful? Because it is empty.

Bruce Lee

Healthcare Quarterly 2009 v12(2)

Br Med Bull 2010

95: 7-32.

Critical Literature Synthesis

ResearchEvaluation

Capacity

Building

WRHA

Board & Senior

Management

Knowledge

Translation

(External)CHA

Hypotheses generated

Ongoing scan of evidence

important to SNR MGT

Knowledge Translation

(internal)

Examples:

• MH-CRC

• BGSC

• CareLinks

• ACSS

• Pre-hab

Leverage

Funding

Opportunities

WRHA’s

Research & Evaluation

Unit: A functionalapproach (2009)

A planning tool for the

WRHA

Being able to

compare

Critical Literature Synthesis

ResearchEvaluation

Capacity

Building

WRHA

Board & Senior

Management

Knowledge

Translation

(External)CHA

Hypotheses generated

Ongoing scan of evidence

important to SNR MGT

Knowledge Translation

(internal)

Examples:

• MH-CRC

• BGSC

• CareLinks

• ACSS

• Pre-hab

Leverage

Funding

Opportunities

WRHA’s

Research & Evaluation

Unit: A functionalapproach (2009)

Strategies for Patient-oriented

Research (SPOR) Networks

• a collaboration of patients, health service providers,

policy/decision makers and health researchers across

Canada (CIHR)

– Conduct research that produces information that responds to the

needs of patients

• TWO foci at the moment:

– Patient-oriented network in adolescent and youth mental health

• KS (CIHR-funded):E-mental health in youth and adolescents

(X.Ye et al.) …a start

– Community-based Primary Health Care

• More research in this area in Manitoba would support our current

and changing environment

George & Fay Yee Centre for Healthcare Innovation lead by

Dr. Terry Klassen: Vision for a SUPPORT Unit (to support SPOR)

Critical Literature Synthesis

ResearchEvaluation

Capacity

Building

WRHA

Board & Senior

Management

Knowledge

Translation

(External)CHA

Hypotheses generated

Ongoing scan of evidence

important to SNR MGT

Knowledge Translation

(internal)

Examples:

• MH-CRC

• BGSC

• CareLinks

• ACSS

• Pre-hab

Leverage

Funding

Opportunities

WRHA’s

Research & Evaluation

Unit: A functionalapproach (2009)

Importance of Evaluation

• Evaluation helps to identify, describe, and understand the

context in which an initiative is implemented in order to

understand (and, sustain) outcomes

• Take a “realist approach”…one that is

1. Comprehensive in identifying how, and for whom, a change

in process might work and,

2. Understanding the implementation of the program as

proposed

• Conduct evaluation assessments (utilization focused) early on

– Allows for modifications prior to full-scale implementation

Evaluations span the scope of

healthcare delivery

• Community Mental Health Crisis Response Centre

• The Bell Hotel

• Women travelling for birth (Pre-natal connections)

• Birth Centre

• P.I.E.C.E.S.: skills development for working with

persons with dementia

• Evaluating how program planning is accomplished

within a WRHA agency

= new / innovative

What is RE-AIM?…Reach, Effectiveness, Adoption, Implementation, Maintenance

• Originally posited by Russell E. Glasgow in 1999.

• The premise?

– “Multilevel interventions that incorporate policy,

institutions & individual components should be evaluated

with measurements suited to their settings, goals &

purpose”

• The RE-AIM Framework uses 5 dimensions (which

occur at multiple levels) to determine & describe

the impact of a program or policy.

FIVE (5) dimensions…help to describe a successful impact & to design

interventions that:

Reach: …reach a broad and representative proportion of

the target population

Effectiveness: …leads to positive changes that are robust across

diverse groups

Adoption: …be adopted across a broad & representative

proportion of settings

Implementation: …lead to consistent implementation of strategies

at a reasonable cost

Maintenance: …lead to sustained delivery of intervention/change

Environmental ImpetusPersons seeking or

needing services

System Vision, Culture, Strategy & Leadership

Functional

Integration

Physician

Integration

Integration(Clinical/System/Person-based)

Horizontal

Vertical

Relational

Coordination

Relational

Coordination

Performance

(Outcomes)

Integration: Precursors Processes Outcomes

Use of RE-AIM Planning &

Evaluation Framework

for Working Groups

Model for Studying

INTEGRATION

Measures of Functional (FI) & Physician Integration

(PI), Relational Coordination (RC) & Integration are

available for use Before-After in the CPJI

60-days fromHigh suspicion to

1st Treatment

Knowing measures

can help in planning

Embedding Evidence in Context: Approaches

Used by an Embedded Research & Evaluation Unit

As the great sub-systems of modernity (state, market,

culture and science), became more transgressive, the

resulting fuzziness helped to create the transaction

spaces in which Mode 2 knowledge production

developed…Nowotny H, 2003

..we believe that our Unit was created to be a

transaction space for Mode 2 knowledge production

…or, using a bamboo forest as a metaphor for Mode 2 knowledge

production by a unit embedded in a health authority