Evaluation of NCI’s Strategy for 2017 I… · Title of Presentation Program Mission: To support the development, maturation, and dissemination of novel and potentially transformative

Title of Presentation

Evaluation of NCI’s Strategy for

Early-Stage Technology

Development Research

Tony Dickherber, Ph.D.

Center for Strategic Scientific Initiatives, NCI

August 2017

2

Outline

1. Overview of the IMAT Program

2. Evaluation Design/Approach

3. Evaluation Findings

4. NIH-wide Technology

Development

5. Conclusions and Next Steps

2


“Progress in science depends on new techniques, new discoveries and new ideas; probably in that order.”

-Sydney Brenner

Title of PresentationNCI Support for Technology Development

Low Cost Global Health Technologies (UH2/UH3, R01, R03)

Common Fund Initiatives

(e.g. Single Cell Technologies or Tissue Chip)

Bioengineering Research Grants (R21, R01)

Clinical Proteomics Technology Assessment (U01, U24)

Nano (IRCN-U01 & Alliance-U24)

ITCR (R21, U01, U24)

SBIR/STTR (R41-44)

Academic-Industrial Partnerships (R01)

Genome Technology Program (R21, R01, SBIR/STTR)

Technology Development Pipeline

ConceptPrototyping &

Feasibility Demonstration

Advanced Development

towards Context of Use

Scaling/Optimization within Context of Use

Hardening and Validation

Dissemination

Typical NIH barrier for technology


Program Mission:

To support the development, maturation, and dissemination of novel and potentially transformative next-generation technologies

through an approach of balanced but targeted innovation in support of clinical, laboratory, or epidemiological research on cancer.


R43

• Feasibility study• Clear commercial

potential

R44

• Development & (regulatory) validation• Manufacturing & marketing plan• Requires proof of feasibility and

commercialization plan• Demonstration of transformative utility

Fast-Track

≤ $225k over 6m total cost support

≤ $1.5M over 2 yearstotal cost support







Dissemination

Typical NIH barrier for technology

R21

R33

• Feasibility/Proof-of-principle study• Highly innovative technology• No preliminary data required

• Advanced development• Demonstration of transformative utility• Requires proof of feasibility

≤$400k over 3 yearsdirect cost support


Two Tracks: 1. Molecular/Cellular Analysis Technologies (MCA)

2. Biospecimen Science Technologies (BST)

IMAT Program Structure

Title of PresentationWhat is a “biospecimen science” technology?

• Sample Quality Control (e.g., RNALater)

– Focus on preserving the biological integrity of the molecular and

cellular targets to be assessed

– Spans the preanalytical time period from patient management

variables, through sample procurement, immediate handling and

preservation, and processing and sample preparation in advance of

analysis

• Sample Quality Assessment (e.g., RIN)

– Focus on verifying/assessing the biological integrity of the

molecular and cellular targets to be tested/measured


• Solicitation:

– Emphasis on innovative technology with transformative potential (i.e.

high-risk, high-impact)

– Focus exclusively on technology development (NOT biological/clinical

hypothesis-driven research)

– 100% investigator-initiated research grants

• Review:

– Special emphasis panels recruited based on focus of submissions, drawing

heavily from former IMAT grantees

– Milestone-based applications to quantitatively assess the capabilities of the

technology (e.g., specificity, sensitivity, and speed) and characterize the

improvement over state-of-the-art

Distinguishing Features of IMAT


8

IMAT New Award Distribution by FYTota

l # o

f A

ward

s

Success R

ate

Fiscal Year

BST: Biospecimen Science Technologies MCA: Molecular/Cellular Analysis Technologies

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

0

5

10

15

20

25

30

35

40

45BST R33 BST R21 MCA R33 MCA R21 Success Rate

Issued as PAR

Title of PresentationGeneral Breakdown of the IMAT Portfolio

Cancer Biology Technologies

• Molecular fingerprinting (-omic discovery)

• Molecular interactions

• Cancer modeling

• Imaging/spectroscopy probes

• Sample preparation

• Mechanobiololgy/microrheology

• et cetera…

Technologies for Clinical

Treatment and Diagnosis

• Drug screening platforms

• Patient-derived tumor modeling

• Diagnostic imaging agents

• Cancer-targeting

• Drug delivery vehicles

• Point-of-care diagnostics

• et cetera…

Early Detection Screening

• Point-of-care detection

• Field sample collection and storage

• Liquid biopsy platforms

• et cetera…

Molecular Epidemiology Tools

• Population-scale analysis

• Low-resource setting point-of-care screening

• et cetera…

47%

4%

9%

40%

Title of PresentationActive IMAT PortfolioApplication & Validation of Emerging Technologies for Cancer Research (R33)o Optimization/scaling or other further developmento Analytical/technical validation in biological context of use

Innovative Technologies for Cancer Research (R21)o Initial proof-of-concepto Quantifiable milestone driven development plan

8

4

6

3

3

6

283

6

4

5

4

5

1 4 2 1

clinical diagnosticsdrug screeningepigenomicsgenomicsglycomicsimagingimmunotherapyliquid biopsymetabolomicsmodelingnovel biosensorpathway toolsproteomicssample prepsample QAsingle celltranscriptomicstreatment

Current R21 Portfolio (75 Active Projects)

6

2

4

4

313

14

6

2

2

3

15

2

Current R33 Portfolio (49 Active Projects)

11

Outline




4. NIH-wide Technology Development


Title of PresentationIMAT FOA & Evaluation History

RFAs Renewed for 5 years• 3 R21 (1 is a 3-yr award)• 2 R33• 2 STTR• 2 SBIR RFAs Renewed for 1 year

• 2 R21 (3 yr awards)• 2 R33

RFAs Renewed for 3 years• 2 R21 (3 yr awards)• 2 R33

Evaluation Feasibility Study

Full Program Evaluation

IMAT RFAs Approved for 3 years• 3 R21/R33• 2 STTR/SBIR

RFAs Renewed for 2 years• 2 R21 (both 3-yr awards)• 2 R33

Evaluative Update

Targeted Evaluation

IMAT PAR Renewed• 2 R21/R33• 2 STTR/SBIR

IMAT PAR Renewed• 2 R21/R33• 1 STTR/SBIR

IMAT PAR Released• 1 R21/R33• 1 STTR/SBIR

Full Program Evaluation

(until FY2016)

Ongoing Evaluation

RFA Renewed for 3 years• 2 R21• 2 R33• Competitive Revisions

Title of PresentationEvaluation Criteria

• number of publications that cite a specific IMAT award number;

• number of patent applications submitted to the US Patent & Trademark Office (USPTO);

• number of patent applications granted or approved by the USPTO based on patent

applications that cite a specific IMAT award number in one of four government interest

fields;

• number of IMAT‐funded technologies now used in other NCI and NIH strategic

initiatives; and

• follow‐up case studies on previously funded technology development projects and

platforms, including their current use by and utility to the extramural scientific and

clinical communities.

13


• Group: Ripple Effect Communications

• Design: From Macro International during a prior Evaluation

Feasibility Study for the IMAT program (2007)

– Key: Follow each technology from before target award to after to

understand how the program/intervention affected the outcome

• Focus: Outcomes for all IMAT project prior to 2014

– 705 unique awards (including SBIR and STTR)

– archival data records, web-surveys, and phone interviews

– Included web-survey of and archival data analysis for a comparison

group and phone interviews with IMAT technology end-users

14

2015-16 IMAT Evaluation Overview

Title of PresentationLogic Model (Conceptual Framework)

15Courtesy Ripple Effect Communications

Title of PresentationEvaluation Objectives


Logic Model

AreaEvaluation Question

Archival

Data

Web-based

SurveyInterview

Initial

Investment

What were the pre-existing technologies that served as the basis for technology developed by IMAT? ✓ ✓

What technologies were proposed and what technologies were funded? ✓ ✓

Program

Activities

How did the application process, FOA/solicitation, and IMAT funding structure (mechanisms) impact

the development of the technology?✓

How were the technologies developed during the funding period for IMAT grantees? ✓

How did interactions with NIH, NCI, or other organizations impact the development of the technology

for IMAT grantees?✓ ✓

How did the research environment (e.g. institutional support; other related research activities) impact

the development of the technology?✓ ✓

Short-,

Medium-, and

Long-term

Outcomes

What was the technology development path after IMAT funding? ✓ ✓ ✓

How were the details of the technology spread to scientific and/or clinical audiences? ✓ ✓

To what extent and in what setting(s) is the technology or methodology being used? ✓ ✓

Are there common themes for those grantees that did not achieve their aims within the IMAT funding

period? If so, what are the themes?✓

Did the short-term and medium-to-long-term outcomes differ from the comparison group? Stage of

Development✓

Dissemination of technology via publications and patent ✓

Self-reported long-term impact ✓

Title of PresentationDevelopment of Comparison Group

17

C = specific aims contains “cancer” or synonym

IN = specific aims contains “innovate” or derivatives

C&IN = specific aims contains both “cancer” and “innovate”

Courtesy Ripple Effect Communications

Trans-NIH Eval

Advisory Group

Title of PresentationDevelopment of Comparison Group

18

$0

$100,000

$200,000

$300,000

$400,000

$500,000

$600,000

R21 R33 R41 R42 R43 R44

Average Award AmountComparison

IMAT

113

146

161152

182 178

127

111

173 176

190 188

0

20

40

60

80

100

120

140

160

180

200

R21 R33 R41 R42 R43 R44

Average Priority ScoreComparison

IMAT


Title of PresentationData Collection Overview



20

Evaluation Findings: Productivity

24 48

91

281

0%

20%

40%

60%

80%

100%

Grants

Perc

enta

ge o

f IM

AT

Gra

nts

Total IMAT Grants(N=444 or ~73% of all awards)

Accounts for 75-100% of publications



Accounts for top 25% of publications

10032097

30313814

44805646

6583

84209758

1114211655

0

2000

4000

6000

8000

10000

12000

14000

2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013Nu

mb

er o

f C

itat

ion

s

Year of Citation

6447

11939

5540

10674

5467 4865 6850

6086

4408

7246

2115

2572

678

91

24

0

50

100

150

200

250

300

350

Nu

mb

er o

f P

ub

licat

ion

s

Fiscal Year of Grant

100

176

108

171142146157

200162

275

173171153

49 59

0

50

100

150

200

250

300

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

20

11

20

12

20

13Nu

mb

er o

f P

ub

licat

ion

s


0

10

20

30

40

50

Nu

mb

er

of

Aw

ard

s



21

IMAT vs. Comparison Group

57

53

60

70

23

6

16

12

22

69

1

11

15

65

111

2

5

0 100 200 300

FDA approval

Clinical trials

International approval

Licenses

Patents

Planned

Submitted/Initiated

Approved/Completed

Rejected

45

45

43

49

31

315

211

42

6

16

10

27

51

1

1

0

2

3

0 50 100 150

FDA approval

Clinical trials

International approval

Licenses

Patents

Planned

Submitted/Initiated

Approved/Completed

Rejected

IMAT Grant Outcomes

Comparison Group Grant Outcomes

0

10

20

30

40

50

60

70

80

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

20

11

20

12

20

13

Nu

mb

er o

f G

ran

ts A

war

ded


IMAT

Comparison


100

176

108

171

142 146157

200

162

275

173 171153

49 59

1 025 28

60

101

36

104

79 79 78

2538

58

23

0

50

100

150

200

250

300

1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

Num

ber

of

Public

ations

Fiscal Year of Associated Grant


22

Self-Reported Outcomes

45

54

57

54

26

49

50

22

17

33

75

59

29

44

110

65

41

33

57

93

61

91

154

122

34

0 100 200 300 400

Advancement of ability to diagnose

Advancement of ability to treat

Improve quality of biospecimensused in clinical management

Improve utility of biospecimensused in research

Improve standards/methods forconducting cancer research

Number of Grants

Are

a o

f Im

pac

t

No Impact

Little Impact

Moderate Impact

Great Impact

N/A (Not a goal of this technology)

44

29

58

51

38

22

37

12

11

16

34

52

4

17

20

29

41

8

19

22

77

47

121

106

107

0 50 100 150 200 250

Advancement of ability to diagnose

Advancement of ability to treat

Improve quality of biospecimens usedin clinical management

Improve utility of biospecimens usedin research

Improve standards/methods forconducting cancer research

Number of Grants

Are

a o

f Im

pac

t

No Impact

Little Impact

Moderate Impact

Great Impact

N/A (Not a goal of this technology)

IMAT Grant Outcomes (310 responses) Comparison Group Grant Outcomes (211 responses)

Survey Responses



• Certified the value of archival data for addressing evaluation

questions

• Certified the value of survey and interview data for

developing more comprehensive appreciation for the impact

of the program/intervention

• Recommendations:– Make better use of the annual PI meeting

– Improve options for supporting commercialization efforts

– Develop standardized review requests at close of awards that includes

downstream development information

– Incorporate measures of risk during initial award to better quantify differences

within IMAT and among comparison groups

– Continue pursuing end-user interviews for evaluation of impact

23

Ripple Effect Report

24

Outline





Development



• Performance Measures study from the Science and Technology Policy

Institute (STPI) developed during 2015 [link]

• The study had three components:

– Developing a comprehensive catalog of NIH Funding Opportunity

Announcements (FOAs) that are focused on technology development for

achieving a specific goal

– Developing case studies of a representative sample of FOAs based on

discussions with FOA program officers to identify measures being used and

lessons learned

– Identifying candidate outcome measures for assessing technology

development initiatives and a data collection infrastructure that would be required

to implement these measures in a consistent and ongoing manner

25

NIH-wide Technology Development Evaluation

https://www.ida.org/idamedia/Corporate/Files/Publications/STPIPubs/2016/D-5712.ashx

Title of PresentationTech Dev FOA Catalog

26

Type of Solicitation

FOA Type Number Percentage

RFA 190 67%

PA 58 20%

PAR 30 11%

PAS 6 2%

Activity Codes

Activity Code Number Percentage

R43/R44 76 27%

R21 64 23%

R41/R42 41 14%

R01 37 13%

R33 26 9%

U01 16 6%

R21/R33 12 4%

R43 only 7 2%

R41 only 6 2%

Also: P01, U24, U54, DP3, R03, UH2, P41

• Included 284 distinct FOAs, organized into 83 distinct groups

• FOAs supported 1,956 distinct awards• $1.83 billion in total NIH spending

over 10 years

Courtesy of the Science and Technology Policy Institute

Title of PresentationLargest and Smallest FOA Groups (by Total Cost)

Largest:• Instrument Development for Biomedical

Applications (165 awards, $174 million)

• Revolutionary Genome Sequencing Technologies:

$1000 Genome (100 awards, $171 million)

• Clinical Proteomic Technologies for Cancer

Initiative (CPTC): Proteome Characterization

Centers (61 awards, $161 million)

• Validation and Advanced Development of

Emerging Molecular Analysis Technologies for

Cancer Research (187 awards, $118 million)

• Early-Stage Innovative Molecular Analysis

Technology Development for Cancer Research

(233 awards, $116 million)

• National Technology Centers for Networks and

Pathways Program (6 awards, $82 million)

• Exceptionally Innovative Tools and Technologies

for Single Cell Analysis (42 awards, $66 million)

Smallest:• New Technology to Screen for Mild Hearing Loss

in Children (1 application, no awards)

• In vivo Methods for Assessing Placental

Development and Function (2 applications, no

awards)

• Development of Diagnostic Screening Test for

Salt Sensitivity (3 applications, no awards)

• Technologies To Assess Sleep Health Status in

Populations (7 applications, 1 award, $0.1 million)

• Innovative Technologies and Assays in Support of

HIV Cure Research (9 applications, 1 award, $0.3

million)

• Methods Development for Obtaining

Comprehensive Genomic Information from Human

Specimens that Are Easy to Collect and Store (7

applications, 2 awards, $0.4 million)

• Development of a Vestibular Neural Prosthesis (6

applications, 2 awards, $2.7 million)


Title of PresentationTech Dev FOA Catalog

28

Technology Area Stage of DevelopmentStage of

DevelopmentNumber of FOA

Groups

Early Only 15

Early/Intermediate 26

Intermediate Only 4

Intermediate/Late 13

Late Only 3

Early to Late 22

• Majority address a defined area within biomedical research

or clinical care

• Majority encompass a diverse set of products

• Equally divided between technologies intended for research

versus clinical use

• Few have defined performance requirements

Categorization Number of FOA Groups

Medical Devices 19

Molecular Analysis 11

Information Technology 9

Cells/Tissues Analysis 6

Point of Care Devices 6

Proteomic Analysis 6

Genomic Analysis 5

Imaging 5

Low Cost Medical Devices 5

Implantable Devices 4

Biospecimen Technologies 3

Others 4


Title of PresentationLessons Learned: Program Management Best Practices

• Award flexibility beneficial. Because technology development projects often require more time and money than typical of discovery research projects, FOAs need to take advantage of opportunities for longer award periods and larger award sizes; multiple acceptance dates are also valuable.

• Tailored review necessary. Because technology development projects often involve engineering and physical sciences disciplines and have more applied goals, tailored review is essential.

• Milestones valuable. Because technology development projects are intended to result in a defined entity for use in research or the clinic, milestones help in charting progress.

• Grantee meetings valuable. Grantee meetings open to industry, other researchers, and potential investors are valuable for sharing information among awardees, facilitating collaborations, and exploring potential commercial relationships.

• Program officer expertise critical. Technology development program officers require three critical characteristics:

– Clear understanding of requirements for commercializing or otherwise disseminating technologies

– Expertise in the technology field

– Familiarity with the relevant investigator community



• Commercialization a hurdle, especially for clinical technologies. Technologies for clinical use almost uniformly require more funding than available through standard NIH award mechanisms. As a result, clinical technologies often languish even if early-stage clinical testing has been completed.

• Funding “blue-sky” technology development difficult. Only the R21 mechanism was viewed as being tailored to fund truly high-risk projects and additional approaches for encouraging such projects need to be developed.

• Greater coordination of technology development efforts is needed. Program officers were generally aware of other ongoing technology development initiatives, but indicated that a forum where they could share lessons learned and best practices would be beneficial.

30Courtesy of the Science and Technology Policy Institute

Lessons Learned: Ongoing Challenges

Title of PresentationProposed Outcome Measures

31

• Ultimate Objectives

– Dissemination and use of the technology

– Increase in the overall level of NIH-funded research activity in the technology development domain

• Intermediate Outcomes

– Achievement of technical milestones

– Conversion of exploratory awards

– Technology licensing or other pre-commercialization activity

– Progress toward clinical use

– Data and software downloads


Title of PresentationRequired Data Collection Infrastructure for Outcome Measures

• Post-award reporting

– Publications and grants reporting use of the technology

– Appearance in clinical practice guidelines

– Conversion of exploratory awards

– Software downloads (for information technologies), and

– Additional data on commercialization activities and progress toward clinical use

• Bibliometric analysis

– Number and rate of citations

– Manual analysis of citations for relevance

• Publication/grant application analysis

– MEDLINE and NIH grant database searches for the name of the technology or the

technology development domain or for references to publication(s)



• Commercial activity in technology development domain

– Manual searching of publicly available information based on product names

– Queries to relevant NIH program officers

• Expert panels

– Assessment of whether a technology has achieved widespread dissemination

and use

– Assessment of whether there is increased research activity in the relevant

technology development domain

• Milestone analysis

– Standard template for recording milestones in grant applications

– Standard Research Performance Progress Report (RPPR) template for

reporting progress toward milestones

Required Data Collection Infrastructure for Outcome Measures


34

Outline





Development



Program Mission:

To support the development, maturation, and dissemination of novel and potentially transformative next-generation technologies

through an approach of balanced but targeted innovation in support of clinical, laboratory, or epidemiological research on cancer.


R43

• Feasibility study• Clear commercial

potential

R44

• Development & (regulatory) validation• Manufacturing & marketing plan• Requires proof of feasibility and

commercialization plan• Demonstration of transformative utility

Fast-Track

≤ $225k over 6m total cost support

≤ $1.5M over 2 yearstotal cost support







Dissemination

R21

R33

• Feasibility/Proof-of-principle study• Highly innovative technology• No preliminary data required

• Advanced development• Demonstration of transformative utility• Requires proof of feasibility



Competitive Revisions• Validation within the context of a

compelling hypothesis • Pursued in collaboration with

end-users≤$300k over 2 years direct cost support

(R01, U01, U54, P01, P50)

New IMAT Program Structure


• NCI has approved IMAT to continue making new awards through

2020 & Cancer MoonshotSM Initiative has a focus on New Enabling

Technologies

• NIBIB reissued EBRG/BRG/BRP in 2016

• NHGRI novel NGS RFAs and genome technology PARs still open

• NIGMS launched an exploratory R21/R01 tech dev program in 2017

• NINDS & NIMH coordinated BRAIN Initiative efforts issued a variety

of research and training efforts in 2017

• NIAID is currently deliberating launch of their own version of IMAT

• …time to initiate a P&E version of the NIH-wide Bioengineering

Consortium (BECON)?

Next steps…

37

Questions?

(301) 547-9980

[email protected]

mailto:[email protected]

Documents

Evaluation of NCI’s Strategy for 2017 I… · Title of Presentation Program Mission: To support the development, maturation, and dissemination of novel and potentially transformative