Responding to your Grant ReviewPart 1

October 27, 2016Gloria Hoffman, PhDMollie Lange, MA

Institutional Development CoreASCEND Center for Biomedical Research Morgan State University

1

You just got your scores and the critiques.

You have had yourmelt down.

(It’s ok, everyone has a melt down)

Now what?2

In 2014, “NIH received 51,073 research project grant (RPG) applications, out of which we funded 9,241, resulting in a success rate of 18.1 percent. ”

You are not alone!

3

If you did not get funded,

Remember the 5 Stages of Grief

Copied from David C. Steffens, M.D., MHS 2012https://medschool.duke.edu/sites/default/files/field/attachments/Responding%20to%20a%20Grant%20Review%204-10-12%20(2).pdf

• Denial: “there was nothing wrong with my grant proposal.”• Anger “the reviewers are a bunch of idiots.” • Bargaining: “Maybe if I respond to a few of the concerns, it

won’t be too difficult to resubmit.” • Depression: “OMG I actually do have to go through this process

again!”• Acceptance: “I have got to deal with it, so I might as well start

preparing.”

4

Today: Dealing with it and moving on

• Research grants only today• What is the NIH scoring system?• What is the guidance given to reviewers?• Reread your proposal from a reviewer’s point of view.• Systematically build your response and revisions.

5

Scoring System for NIH Research Grants

A score from 1-9 is given for each of five criteria with “1” being the best score and “9” the weakest. A score of “5” is considered average.

There is no grade inflation and there is no curve. Not every grant gets scored; those that are considered unlikely to be funded after preliminary review are “triaged”.

6

Proposals are evaluated and scored on the following criteria:1. Significance2. Investigator(s)3. Innovation4. Approach5. Environment

In addition reviewers provide an overall impact score to reflect their assessment of the likelihood of the project to exert a “powerful, sustained influence on the field of research.”

Scoring continued

7

1. SignificanceDoes the proposal address an important problem or critical barrier to progress in the field?

Is there a strong scientific premise for the project?

If the goals are achieved, how will scientific knowledge, technical capability and/or clinical practice be improved?

How will successful completion change the concepts, methods, technologies, treatments, services or preventive services that drive the field?Remember we are talking about National Institutes of Health

8

2. Investigators

• Are the PI, CoI’s, and other key personnel well suited for the project?

• Do new investigators have appropriate training and experience?

• Have established investigators demonstrated accomplishments that have advanced their fields of study?

• If multi-PI (and/or multi-disciplinary), do the investigators have complementary and integrated expertise?

• Is their leadership approach defined and appropriate?9

3. Innovation

• Does the proposal seek to challenge current research paradigms by using novel theories, approaches or methods?

• Are the concepts, methods and approaches novel to the field or novel in a broad sense?

• Is a refinement , improvement or novel application of existing theory, methods, or instrumentation proposed?

10

4. Approach

• Are the strategy, methods and analyses well reasoned and appropriate to achieve the specific aims?

• Does the approach predict a robust result?• Does the plan address ways to eliminate bias?• Are potential problems, alternative strategies and benchmarks for

success discussed?• For early stage work, will the plans establish feasibility? How will risk

be managed?• Are there adequate plans to address relevant biological variables, such

as sex or age?

11

Research plan, methods, analysis

Approach: human subjects

• Does the research plan address the protection of human subjects from research risk?

• Is the inclusion or exclusion of individuals on the basis of gender, race, age, and ethnicity justified in terms of the scientific goals and research strategy proposed?

• Children?

12

This will figure into the Approach score

5. Environment

• Will the scientific environment of the performance site contribute positively to the success of the project?

• Are the institutional support, equipment, and facilities available to the investigators appropriate for the work described?

• Will the project benefit from the unique features of the scientific environment , subject populations, or collaborative arrangements?

13

Overall Impact Score

14

• Takes into account, but is distinct from, the other scored criteria.

• The overall impact score is not a mean or a sum of the other scores.

• It is not a summary of the strengths and weaknesses.

Significance vs Impact: NIH clarificationSignificance: Does the proposal address an important problem or critical barrier to progress in the field? Is there a strong scientific premise for the project? If the goals are achieved, how will scientific knowledge, technical capability and/or clinical practice be improved? How will successful completion change the concepts, methods, technologies, treatments, services or preventive services that drive the field?

Overall Impact: Reviewers will provide an overall impact score to reflect their assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved, in consideration of the following five core review criteria, and additional review criteria (as applicable for the project proposed).

15

16

Significance vs. Overall ImpactCase Study handout

Additional Criteria- not scoredVertebrate animals: Are numbers, species ,strains, ages, sex and total number appropriate? Has the use of animals vs. alternative models (e.g. cell culture) been

justified? Have interventions to minimize pain, discomfort and distress been

addressed?

Biohazards: Are any procedures or materials potentially hazardous to research

personnel and/or the environment? If yes, has adequate protection been discussed?

17

Other Review Considerations

• Select Agent Research• Resource Sharing Plans• Authentication of Key Biological and/or Chemicals: If key biological or

chemical resources will be used reviewers will comment on the plans for identifying and validating these items. More on this one later.

• Budget: Are the expenses reasonable, adequate and justified for the work described?

18

Scientific Rigor and Transparency

This is a new NIH initiative. The goal is to enhance reproducibility of research through rigor and transparency in 4 areas.

• Scientific Premise• Scientific Rigor• Consideration of Relevant Biological Variables• Plan for resource authentication

19

Scientific Premise

Scientific premise will be addressed in the Significance section. There is no separate section for it.

NIH defines this as the key existing data that are referenced and used to justify the project.

Reviewers are asked to look for a well-reasoned evaluation of existing (yours and that of others) data that supports the application.

How does the proposed research fill any gaps or weaknesses in the current state of the science?

20

Scientific Rigor

• Reviewers will look for scientific rigor in the Approach section and will figure it into the score for this section.

• Is there a strict use of the scientific method in the proposed work to assure effective and unbiased experimental design, methodology, data analysis and interpretation?

• How will the new data be collected, analyzed, interpreted?

• Will the experiments actually test the hypothesis?

21

Consideration of Relevant Biological Variables

• Reviewers will look for this in Research Strategy and will figure it into the score for Approach.

• An RBV is a critical factor affecting the health or disease in a vertebrate animal or human; e.g. sex, age, source, strain, possibly race or ethnicity.

• Each of the above is known to be a variable in normal biology and predisposition or response to injury or disease.

• If you are studying prostate cancer, you don’t need to include females, but you need to address the subject.

22

Plan for Resource Authentication

• There is now a specific section for this.

• Biological or chemical reagents that may vary from lab to lab, under different conditions, over time, could influence the experimental data and are integral to the proposed research.

• Examples: cell lines, specialty chemicals, antibodies. Consumables only?

• The plan should be less than one page, should NOT include authentication data, should discuss existing guidelines for validation or the need for the research community to develop such guidelines or standards.

Reviewers will not include this evaluation in scoring. They should comment on the adequacy of the plan. 23

In terms of review criteria, what’s true for NIH is true for any other agency

• Today we are talking about NIH grants. These are (or have been) the bread and butter of the biomedical research field. They are prestigious, and one RO1 can fully support (almost) a small laboratory’s research expenses. These are hard to get these days.

• But funding is available from other federal (DoD, NSF) and state (TEDCO) agencies and many private foundations.

• What we discuss today is applicable to nearly any proposal for funding. All granting agencies want to fund significant, innovative work done by experienced investigators.

24

Guess what?

Knowing the review criteria and the scoring system should help you tremendously when you write your next grant.

But for now, we need to work on the earlier application that was reviewed.

25

Ok, now you know what the reviewers are looking for and what sections they found to be weak.

How to proceed?

Let’s assume you will revise and resubmit. (We won’t cover details of the actual resubmission process)

What now?

26

Re-read the evaluations then make a list

Make a comprehensive list for each reviewer of each criticism

27

Reread your proposal with eagle eyes

• Be the reviewer, not the author; detach emotionally• Look for holes or gaps or jumps in topics• Identify the areas (very specifically) that require attention,

matching up as well as possible the criticism with the page(s) or paragraph(s) that need fixing

See David C. Steffens, M.D., MHS 2012https://medschool.duke.edu/sites/default/files/field/attachments/Responding%20to%20a%20Grant%20Review%204-10-12%20(2).pdf

28

Identify the criticisms that are easy to correct • Correct spelling and grammar• Clarify by reformatting or adding headings or indents. Add a little white space

if you have room. • Think about using tables and diagrams for lists or research plans.• Add a consultant or other expertise. Get a letter of support.• Rework the budget if you have added personnel. You may need to come back

to this one after fixing major problems.

• Make sure all text in figures is readable by a 55 year old reviewer who wears reading glasses.

Start working on the easy fixes

29

Identify the major changes needed

Think scientific premise, significance, innovation

1. More preliminary data!!!!2. Rework the proposal question.3. Is there a flaw in reasoning or application of the scientific method?4. Are your references up to date? Are you up to date on the current

state of the science? Does updating your knowledge change your question or approach?

30

Get help• Ask colleagues to read the reviews and your grant (carefully) and give their

honest take on both.• Build a relationship with people in your field who are successful at getting

funding. Learn from them.• These investigators are likely also reviewers. Talk about the review process with

them.

https://grants.nih.gov/grants/grants_process.htmhttp://grants.nih.gov/grants/peer/critiques/rpg_D.htm

Online resources

31

What we did not cover today, but is still important

• The timeline and for NIH review and decision• Resubmission policies and procedures• NIH Study Sections• Resubmission or New Application?• Review guidance for the different NIH Research Grant

mechanisms (e.g. RO3, R21, RO1)

Information on all of the above can be found online. Ask Mollie or Dr. Hoffman for help in navigating.

32

For next weekWe will evaluate and score according to NIH guidelines a small grant application.

It is a public health related topic, something that everyone will understand and be able to evaluate.

33

34

Important:

Much of the material used for this presentation comes from http://grants.nih.gov/grants/peer/critiques/rpg_D.htmand from a presentation by David C. Steffens, M.D., MHS https://medschool.duke.edu/sites/default/files/field/attachments/Responding%20to%20a%20Grant%20Review%204-10-12%20(2).pdf

Some of the NIH language is presented verbatim.