Implementation of an Evaluation Model for Evaluating Complex Health Research Outcomes

American Evaluation Association November 1-5 2011

Kathryn Graham (PhD), Heidi Chorzempa (MSc)

Overview of Presentation

•CAHS Implementation and Results

•Lessons Learnt

•Next Steps

Our Organization and Mandate

•Alberta Innovates Health Solutions has a three decades long history of supporting medical/health research

•“Support, for the economic and social well-being of Albertans, health research and innovation activities aligned to meet Government of Alberta priorities, including, without limitation, activities directed at the development and growth of the health sectors, the discovery of new knowledge and the application of that knowledge”.

Why We Evaluate Health Research?

Source: Canadian Academy of Health Sciences 2009

“…..economic and social well-being….

Implementation Approach and Considerations

ResultsInputs

Org

an

iza

tio

n’s

M

iss

ion

1. Evidence-Based Practice• Environmental scans • Theoretical & Conceptual Models (CAHS)• Retrospective Data Mine Studies• Prospective reporting• Psychometric Testing

2. Practice-Based Evidence• Integated into the business model• Stakeholder/Actors need, values, use• Evaluation tools e.g. collaborative logic models

3. Top Down and or Bottom Up• Policy/Priority Driven and or Discovery Driven

4. Context • Strategy, External Factors, Clusters, Domains

5. Levels of Aggregation

6. Target Audience• Tailor results to stakeholder needs & values

Macro, Meso, Micro

Organizational and Health Research

Inputs, Processes,

Reach, Outputs & Outcomes

Socio-Economic

Impact Categories

Mixed Methodology Approach

Indicators & Metrics

Indicators & Metrics

TIMING – SHORT TO LONG TERMTIMING – SHORT TO LONG TERM

Drivers for Retrospective Study

• Verification: Using CAHS - does it make sense? Were the impact categories, sub categories and measures meaningful?

• Applying Theory to Practice “in Context” - Alberta’s story

• Evidence Informed Decision Making: Lessons learned informing design of new AIHS programs

Retrospective Data Mine Study

Pillar I: Biomedical Research

Pillar II: Clinical

Research

Pillar III: Health

Services

Pillar IV: Population

Health

5 years

5 years

3+3 years

3+3 years

• Scholar• Scientist• Sr. Scholar

• Scholar• Scientist• Sr. Scholar

• Clinical Investigator

• Population Health Investigator

Program Duration

242 health researchers supported

How we did it….

• A retrospective review of finalized grants from 2004-2008

• Population of a database

• Review 855 annual reports across 8 types of personnel grants

• Recognizing project limitations:

-Attribution and contribution

-Self-report

-Data variability

-Outcome underestimation

-Timing of measurement

-Retrofitting

Retrofitting the data to the model

Advancing Knowledge

CapacityBuilding

InformingDecision Making

Socioeconomic Impact

Health Impact

# publications

# book

chapters

# abstracts

# of invited

talks

# trainees - Grad students - Post Docs - Summer students - Other trainees

* Infrastructure

* External PI funding amount

# of collaborations

# of influences in guidelines

# of influences in policies

# of media citations

# of influences in public education

# patents

# spin-off companies

# products in development

# of diagnostic/prognosis advancements

# of treatment advancements

# of access improvements

# of safety improvements

Annual Report Template

Impact Summary

Alberta Innovates Health Solutions

•Across 4 Research Pillars

Research Capacity

•$217,381 in attracted funding

•990 trainees on average per award year

•14 new laboratories

Res

ea

rch

Res

ult

s Knowledge Pool

11

96

Co

lla

bo

rati

on

s w

ith

oth

er

Re

se

arc

her

s

11 collaborations with Health Authorities

•5 guidelines

25 collaborations with Industries

•17 products developed

17 collaborations with Government

•4 policies•3 data services

18 collaborations with Non-profits

•8 guidelines

Health Care•42 improvements to health care effectiveness•8 improvements to health care efficiency & accessibility

Improvements in:•30 therapeutics •12 diagnosis/•prognosis techniques

Determinants of Health•Personal behaviour•Social/cultural determinants•Environmental

Improvements in health and wellbeing •Qualitative Results

Economic & Social Impact

Impacts feed back into inputs for future research

National/ Global Research

Research activity

That produces results

That influences decision making in….

That affects healthcare and health risk factors

That contribute to changing health, social and economic wellbeing

Health Status and function, well-being, economic conditions

Research Diffusion

AdvancingKnowledge

Capacity Building

InformingDecisionMaking

HealthImpact

SocioeconomicImpact

•3833 publications

•4398 abstracts

•386 book chapters

•2789 invited talks

•107 patents •10 products in development •5 spin-off companies

100%

100%

90%

15%

21%

The Public•16 media events

2001 2002 2003 2004 2005 2006 2007 2008

“I have designed a new approach where a target gene can be cloned into a large number of expression vectors by combinatorial assembly of the target gene with a limited number of pre-made DNA fragments.” Scholar, year 2

Significance: II created a procedure that can disperse many copies of one gene with limited resources. This could be used to increase research productivity (for example, placing the gene responsible for insulin in many different cell types to try to determine the cell that produces insulin the most efficiently.

“The former was developed in collaboration with _____& _____and represents the first imaging tool to assess disease activity in both spine and sacroiliac joints of patients with AS.” Sr. Scholar, year 2

Significance: II developed a tool that can be used to assess progression of anklosying spondylitis (a chronic and painful condition). This can help determine guidelines for treatment (ex. When to use a specific therapy)

“The data in this manuscript were the first to show that isoprostanes have the ability to modulate inflammatory processes with relevance to airway respiratory diseases. “Sr. Scholar, year 3

Significance: II found evidence for the use of isoprostanes (a form of anti-inflammatories) in the treatment of conditions such as asthma and COPD, where inflammation is the main source of disease progression.

“The second paper in J Neurophysiol was cited by the Faculty of 1000 Biology as highly innovative. This paper explained why thalamic stimulation is not broadcast widely to the motor cortex and why DBS does not disrupt motor control. It also helped explain the mechanism of action of subcortical DBS for severe refractory depression. “ CI, year 4

Significance: Helps explain how deep brain stimulation works, which could lead to

improvements on this treatment modality.

*Please note that this timeline only represents a small proportion of the 320 innovation statements captured during the review.

“This lab contains state-of-the-art computing infrastructure… and likely the first in a medical imaging lab in Canada)” Bio.Scholar, year 2.

Significance: II has cutting edge technology which could be used to improve diagnostic/prognostic modalities or be used for research (higher resolution images may display more information that is needed for progression).

“We were the first lab in the world to describe the functional consequences of mutations in T-type channels linked to childhood absence epilepsy in humans” Sr. Scholar, year 3

Significance: II found differences in the calcium channels in epileptic patients compared to non-epileptics. This could elucidate the pathophysiology of epilepsy as well as potential therapeutic venues.

“In addition, these studies showed for the first time, that chimerism induces a dominant form of humoral tolerance, allowing B cell tolerance to multiple donors.” Bio. Scholar, year 5

Significance: II showed that a combined protein can improve tolerance of donated pancreatic tissue, which could improve long-term success rates of islet transplantation in diabetics.

Timeline of Innovation 2001-2008

“In 2007, we published the first major report from the Alberta Diabetes Surveillance System (ADSS), in collaboration with the Health Strategies and Surveillance Branches of Alberta Health and Wellness (AHW).” H.Scholar, year 4

Significance: The increased surveillance abilities resulting from ADSS will help inform health authorities’ efforts to reduce the burden of diabetes on Albertans.

© P

erfo

rman

ceM

easu

rem

ent.c

aWhat we did next

• Applied to Trainees Program

ProspectivePilot Studies

Science of Science

Integrated ProgramDesign

Retrospective Data Mining Studies

• Collaborative Logic Models

• Created pilot reports for 3 programs

• Collaborative Balanced Scorecard at the organizational Level

• Partnered in a science of science network

• New program design

• Link to Goals/Objectives and reach

• Psychometric testing for key inidicators

Example of the Balanced Scorecard

Strategy Map

Linking and Ranking ProjectsLinking and Rating Processes

Scorecard Information – Indicators

Scorecard Data – Dash board

Courtesy PM2

Lessons Learnt • Verification – “YES WE COULD DO IT”

• Theory to Practice Lessons – Implementation- Complemented CAHS model with other models- Imbed knowledge translation – decision making rubrics – progress markers- Link collaborative activity to outcomes- Tell the story through qualitative and quantitative indicators

• Context Lessons- Alberta Innovates System – innovation- Link organizational and program goals and objectives to CAHS categories - Tease out Innovation

• Challenges• Change Management – theory of change • Utilization – is this a useful model for evaluating health research funding

Values and Valuing of Measurement

“We are noticing a shift around the measurement of scientific value versus social/public value as well as the concept of collective action and shared value. Using a staged implementation approach, the CAHS model provided us with a best practice framework to measure benefits across the spectrum.

Contact Information

Kathryn Graham (PhD)

Director, Performance Management

Alberta Innovates Health Solutions

1500 – 10104 103 Avenue N.W.

Edmonton, Alberta, Canada T5J 4A7

Email: kathryn.graham@albertainnovates.ca

Heidi Chorzempa (MSc)

Manager, Performance Management

Email: heidi.chorzemp@albertainnovates.ca

THANK YOU