UCLA CTSI-Evaluation Metrics for Monitoring

Transformation & Performance

Pamela Davidson, PhD CTSI-Evaluation Leader

Y2 Progress Report

1

Presentation Outline

I. Evaluation Aims, Design and Methods

II. Response to 2012 EAB Recommendations

III. UCLA CTSI Phases of Change • 2011-2013 progress

– Restructuring and metrics for monitoring transformation

• Future plans for 2013 and beyond – Emerging vision/strategic framework and metrics for

monitoring performance

2

I. Evaluation Aims, Design & Methods

Evaluation Aims 1. Document the transformation 2. Monitor progress against plan 3. Investigate longitudinal impact using key performance

and outcome metrics 4. Align evaluation metrics with national and regional CTSA

metrics and priorities

Evaluation design & methods – Quasi-experimental trend analysis – Mixed methods using quantitative and qualitative

analysis – Measure transformational change and monitor

organizational performance 3

II. Response to 2012 EAB Recommendations

• Using a more “utilization-focused evaluation” that intertwines Evaluation + Development may support the progressive unfolding of the UCLA CTSI as it builds capacities across the components and brings them to scale. – This utilization-focus implies that the “evaluator

role” is to both assess and support the development of CTSI strategies and activities using rapid feedback of the data.

4

Utilization Focused/ Developmental Evaluation • Developed prototype for Office of Investigator Services (OIS) tracking system

that informed design of the CTSI Team Science (Ticketing) System • Ongoing collaboration with BioInformatics to refine Ticketing System; indicated

need for outreach campaign to train people

• Analyzed and reported results from the BioInformatics survey to inform and refine investigator services

• Conducted Org Effectiveness Survey: indicated need for more intensive

communication and outreach led to Information Session Series

Utilization-Focused Evaluation in 2012-13 (examples)

Supplemental materials: contain comprehensive outline of utilization-focused evaluation activities, refer to slides 24-25.

II. Response to 2012 EAB Recommendations

6

Emerging opportunities and unanticipated outcomes (examples)

• Involvement in PALM/Operations Committee – Emerging leadership strategy – Organizational restructuring – Entrepreneurial opportunities

• CTSI-Evaluation Faculty Co-Leaders reviewed and commented on the new governance structure

• Tracking the emerging vision/strategic

framework – think futuristically about dashboard metrics

PALM: Program Area Leaders Meeting 7

Environmental Analysis/ Summary

• 2016 CTSI will look very different than the 2011-2015 CTSI

• Environmental trends will inform and shape the 2016 CTSI Vision – Regionalization/ Consortial Arrangements – Stagnant government funding – Public-Private Partnerships – Standardizing CTSA Metrics – Integration w/ Patient-centered Outcomes Research

Institute – Institutionalization – State-supported universities consolidating – Other IOM Recommendations (TBD) 8

III. UCLA CTSI Phases of Change

2011-2013 Progress • CTSI Structuring and Restructuring • Selecting metrics for monitoring

transformation

June 2011 through Feb. 2013 9

Evaluation

Create an academic home for translational

science

Advance team science

Train the next generation of translational

scientists

Build bi-directional community

partnerships

Promote inter-institutional

collaborations

Community

CCRR

Biostatistics

Bioinformatics

Regulatory

Technology Transfer

Pilot / CTT

Education

10

Goal-focused Metrics (2011-2012 Example)

CTSI Goal 2: Build transdisciplinary research teams to accelerate and translate discovery Metrics for longitudinal tracking

Metric definition Data source Baseline measures

Rapid Response Team (RRT)1 performance

- Number of scientific teams using RRT services to develop a grant application in a time period.

- Number of scientific

teams using RRT services that receive funding in a time period

- Number of scientific

teams using RRT services that received Patient Centered Outcomes Research Institute (PCORI) funding.

Y2 Pilot Grants Annual Progress Report (APR) RRT Annual Report, FY 2012 RRT tracking database

Number of teams using RRT services 2011: 13 2012: 23 Number of teams receiving funding 2011: 1 ($307,985) 2012: 2 ($4.9M) Number of PCORI proposals awarded 2011: 1 ($307,985) – Pilot Project Award 2012: 1 ($1.7M) – Cycle I

Supplemental materials: Table 1 reports metrics, definitions, data sources, and baseline data for CTSI Institute Goals 1-5. 11

2013 UCLA CTSI Exec Dashboard Initiative

• CTSI-Evaluation

– Define, measure, and monitor a set of enterprise-level metrics to improve organizational performance and outcomes.

• data2insight

– Facilitate creation of strategic dashboard to inform CTSI decision-making and actions to improve performance.

• Engage CTSI Program Leaders

– Select among candidate metrics for designing the dashboard

– Co-create strategies for improving/expanding metric performance

12

13

Evaluation planning

Need dashboard?

Design measures

Data collection

and analysis

Build buy in

Define measures and data sources

1) Design 2) Build 3) Populate

5) Refresh

4) Publish

Yes

6) Evaluate and refine

Dashboarding Process

Metric Selection Criteria

• Alignment with national CTSA metrics • Alignment with regional UC BRAID metrics • Cross-cutting metrics are preferred over

component based metrics • Signal indicates strengths or gaps in

performance • Mix of leading and lagging metrics • Actionable for performance improvement

planning • Reflect emerging vision/strategic framework

14

III. UCLA CTSI Phases of Change

Future Plans: 2013 and beyond • Emerging vision/strategic framework, and • Metrics for monitoring performance

16

Emerging vision/strategic framework and metrics for monitoring performance

1. Managing complex change and organizational transitions involves a phased approach

2. Transformation continues with emerging vision/ strategic framework

3. Strategic framework will be implemented using a planned change strategy

4. In parallel, CTSI-Evaluation identifies leading metrics that track with emerging strategic framework

17

Strategic Framework Defined

Formal strategies contain 3 essential elements: – The most important goals to be achieved – The most significant policies guiding or limiting

action – The major action sequences that are to accomplish

the defined goals

Creating strategic framework will focus and align CTSI:

– Operational priorities – Budget allocations – Performance monitoring and improvement

18

matchmaking T1-T4 teams

access CTSI core services

accelerate research

processes

scientific achievement/ health impact

19

Build high performing teams

Linking research efforts, translational capabilities and program outcomes

funding opportunities

19

Metric-Driven Team Science Process

20

Build high performing teams Metric-Driven Team Science Process

20 Linking research efforts, translational capabilities and program outcomes

ROI ▪ external funding ▪ publications

biostatistical consults ▪ pilot funding

team science grants ▪ RRT applications

▪ awarded team science grants

OIS consults ▪ core services

▪ business/regulatory consults ▪ CTT/vouchers

funding opportunities

matchmaking T1-T4 teams

access CTSI core services

accelerate research

processes

scientific achievement/ health impact

In Summary

I. Evaluation Aims, Design and Methods II. Response to 2012 EAB Recommendations

III. UCLA CTSI Phases of Change

• 2011-2013 progress – Structuring, Restructuring and – Metrics for monitoring transformation

• Future plans for 2013 and beyond – Emerging vision/strategic framework, and – Metrics for monitoring performance

21

UCLA CTSI-Evaluation Faculty Co-Leaders

Pamela Davidson, PhD, UCLA-Westwood (Chair) Ronald Andersen, PhD, UCLA, Public Health/HSR Mohsen Bazargan, PhD, Charles Drew University Lourdes Guerrero, EdD, UCLA-Medicine Moira Inkelas, PhD, UCLA, Public Health Gerald Kominski, PhD, UCLA, Public Health Deborah Koniak-Griffin, EdD, RN, UCLA-Nursing Loren Miller, MD, Harbor LA-Biomed Jack Needleman, PhD, UCLA, Public Health Martin Shapiro, MD, PhD, UCLA-Medicine/HSR Michael Weisman, MD, Cedars Sinai Health System Alisa Wilson, PhD, Cedars Sinai Health System

22

Supplemental Information

• UCLA CTSI Self-Evaluation Annual Progress Report

• UCLA CTSI Evaluation Program Annual Progress Report

• Table 1. Metrics, definitions, data sources, and baseline data (2011-2012)

• Additional slides 24-30

23

24

Utilization-Focused Evaluation in 2012-13

Utilization-Focused Evaluation in 2012-13

25

OUTCOMES

Intermediate • Team Science

• Scientific Achievement

• Organizational

Performance

Longer-Term • Scientific / Health Impact

• CTSI Sustainability

Context

• 60 NCATS CTSAs • 5 UC CTSAs (UC BRAID) • Public/Private Partnerships

Evaluate and improve systems and subsystems

T1 Basic &

Bio Medical

T2 Clinical Studies

T3 Community Practice

Five UCLA CTSI Goals

T4 Systems/Policy

Change

Planned Change Strategies

Macro Structure UCLA CTSI Enterprise

• 4 Institutional Partners • Medical Centers/Clinics • Professional Schools • Community Partners

Micro Structure

CTSI Programs

• Community • CCRR (Former GCRCs) • Design and Biostatistics • Bioinformatics • Regulatory/OIS • Technology Transfer • Pilot & Collaborative Grants • Education/Career Development • Evaluation

UCLA CTSI Evaluation Framework for Transforming Systems and Subsystems

26

Create an academic home for translational science

Advance team science

Train the next generation of translational scientists

Build bi-directional community partnerships

Promote inter-institutional collaborations

UCLA CTSI Goal Focused Evaluation

27

Evaluation Questions

How has the UCLA CTSI transformed the translational research enterprise?

Goal 1. How did we create and sustain an academic home?

Goal 2. Did we accelerate team science, scientific productivity, and

health impact? Goal 3. How did the CTSI improve the quality, effectiveness and/or

efficiency of research training and career development?

Goal 4. To what extent did we create a sustainable infrastructure for conducting community-partnered research?

Goal 5. How did we expand national, state and regional CTSA collaborations?

28

Data dashboard value proposition

• Sustain focus on high impact, cross-cutting performance measures

• Facilitate strategic, productive conversations

• Link measurement to strategy

• Communication and decision making

• High level performance measures

• Snapshot for CTSI leaders

29

15 Most Promising Metrics from national CTSA Evaluation KFC F2F Annual Meeting (Oct 2012)

1. Time from IRB submission to approval 2. Studies meeting accrual goals 3. Time from notice of grant award to study opening (e.g., investigator initiated studies) 4. Number of technology transfer products 5. Volume of investigators who used services 6. Volume of types of services used 7. Satisfaction/needs assessment 8. Time to publication 9. Influence of research publication (e.g., observed/expected citations) 10. Researcher collaboration (e.g., team science; collaboration index) 11. ROI of pilot and KL2 scholars 12. Time from publication to a research synthesis 13. Career development 14. Career trajectory (e.g., K-R transition) 15. Institutional collaboration (public-private; cross-institutional; community)

30