Merit Review Process - NSF Grants Conferences · Becoming a Reviewer ... Combination Internal Program Officer ... effectiveness, and evenhandedness of NSF’s merit review process

Merit Review Process

National Science Foundation - Spring Grants ConferenceLouisville, KY - June 5&6 - Kentucky EPSCoR

Panelists

2

Jeremy EpsteinDeputy Division Director, Directorate for Computer & Information Science & Engineering; Division of Computer & Network Systems

Francisco RubenAssistant General Counsel and Alternate Designated Agency Ethics Official, Office of the Director; Office of the General Counsel

Mary ToneyProgram Director, Directorate for Engineering; Division of Civil, Mechanical & Manufacturing Innovation

Jennifer WellerProgram Director, Directorate for Biological Sciences; Division of Biological Infrastructure

Topics Covered

3

“Ask Early, Ask Often!”

• Proposal and Award Timeline• Proposal Preparation and Submission

Reminders When Preparing Proposals

• Proposal Review and Processing Program Officer Review

Proposal Review Criteria

Types of Reviews

Becoming a Reviewer

Managing Conflicts of Interest

Funding Decisions

• Award Processing Issuing the Award

4

NSF Proposal & Award Process Timeline

4

90 Days 6 Months 30 Days

Proposal Receipt at NSF

Proposal PreparationProposal Receipt to DD Concurrence of PO Recommendation

DD Concur

DGA Review & Process

Can be returned without review/withdrawn

Award

NSF Announces Opportunity

Research & Educational

Communities

Submit

NSF Program Officer

Ad Hoc

Panel

Combination

Internal

Program Officer Analysis and

RecommendationsDD Concur

Via DGA

Organization

Reminders When Preparing Proposals

5

• Read the funding opportunity; ask a Program Officer for clarifications if needed

• Address all the proposal review criteria

• Understand the NSF merit review process

• Avoid omissions and mistakes

• Check your proposal to verify that it is complete!

Program Officer Review

6

• Upon receipt at NSF, proposals are routed to the PI-designated program office.

• NSF staff conducts a preliminary review to ensure they are:

Complete;

Timely; and

Conform to proposal preparation requirements.

• NSF may not accept a proposal or may return it without review if it does not meet the requirements above.

• If the proposal is outside the scope of the program, the program officer usually tries his/her best to transfer it to the most appropriate program for evaluation.

Proposal & Award Policies & Procedures Guide (PAPPG)

7

The PAPPG contains detailed guidelines on proposal preparation and a description of the Merit Review Criteria:

Proposals Not Accepted or Returned Without Review

8

• Per the PAPPG Project Summary Requirement: Must include an overview and separate statements on Intellectual

Merit and Broader Impacts.

• Per the PAPPG Project Description Requirement: Must contain, as a separate section within the narrative, a section

labeled “Broader Impacts of the Proposed Work.”

Must include results from prior NSF support with start date in the past 5 years.

• Per the PAPPG Data Management Plan Requirement: Must be included as a supplementary document.

• Postdoctoral Researcher Mentoring Requirement (if applicable): Proposals that include postdoctoral researchers must include a

description of the mentoring activities that will be provided for such individuals.

If it does not contain all of the required sections, as described in PAPPG Chapter II.C.2.

Other Reasons for Return of Proposals Without Review

9

• It is inappropriate for funding by the National Science Foundation.

• It is submitted with insufficient lead time before the activity is scheduled to begin.

• It is a full proposal that was submitted by a proposer that has received a “not invited” response to the submission of a preliminary proposal.

• It is a duplicate of, or substantially similar to, a proposal already under consideration by NSF from the same submitter.

Other Reasons for Return of Proposals Without Review

10

• It does not meet NSF proposal preparation requirements, such as page limitations, formatting instructions, and electronic submission, as specified in the PAPPG or program solicitation.

• It is not responsive to the PAPPG or program announcement/solicitation.

• It does not meet an announced proposal deadline date (and time, where specified).

• It was previously reviewed and declined and has not been substantially revised.

• It duplicates another proposal that was already awarded.

11

NSF Proposal & Award Process Timeline

11




DD Concur



Award



Communities

Submit

NSF Program Officer

Ad Hoc

Panel

Combination

Internal



Via DGA

Organization

Merit Review Criteria: Guiding Principles

12

• All NSF projects should be of the highest quality and have the potential to advance, if not transform, the frontiers of knowledge.

• NSF projects, in the aggregate, should contribute more broadly to achieving societal goals.

• Meaningful assessment and evaluation of NSF funded projects should be based on appropriate metrics, keeping in mind the likely correlation between the effect of broader impacts and the resources provided to implement projects.

Merit Review Criteria

13

• When evaluating NSF proposals, reviewers should consider what the proposers want to do, why they want to do it, how they plan to do it, how they will know if they succeed, and what benefits would accrue if the project is successful. These issues apply both to the technical aspects of the proposal and the way in which the project may make broader contributions. To that end, reviewers are asked to evaluate all proposals against two criteria:

Intellectual Merit: The Intellectual Merit criterion encompasses the potential to advance knowledge; and

Broader Impacts: The Broader Impacts criterion encompasses the potential to benefit society and contribute to the achievement of specific, desired societal outcomes.

Five Review Elements

14

• The following elements should be considered in the review for both criteria:

• What is the potential for the proposed activity to:

advance knowledge and understanding within its own field or across different fields (Intellectual Merit); and

benefit society or advance desired societal outcomes (Broader Impacts)?

• To what extent do the proposed activitiessuggest and explore creative, original, orpotentially transformative concepts?

• Is the plan for carrying out the proposedactivities well-reasoned, well-organized, andbased on a sound rationale? Does the planincorporate a mechanism to assess success?

• How well qualified is the individual, team,or institution to conduct the proposed

activities?

• Are there adequate resources available to the PI (either at the home institution or through collaborations) to carry out the proposed activities?

Review Format in FastLane

15

Reviewers provide feedback to NSF based on the Review Criteria and the Review Elements

Review Criteria and Elements are available as reviewers provide feedback

16

Proposal Review and Processing

16




DD Concur



Award



Communities

Submit

NSF Program Officer

Ad Hoc

Panel

Combination

Internal



Via DGA

Organization

Types of Reviews

17

• Ad hoc: Proposals sent out for review

Ad hoc reviewers usually have specific expertise in a field related to the proposal.

Some proposals may undergo ad hoc review only.

• Panel: Face-to-face sessions conducted by reviewers mainly at NSF but also in other settings

Panel reviewers usually have a broader scientific knowledge.

Some proposals may undergo only a panel review.

Some proposals may undergo reviews by multiple panels (especially for those proposals with crosscutting themes).

Types of Reviews

18

• Combination: Some proposals may undergo supplemental ad hoc reviews before or after a panel review.

• Internal: Review by NSF Program Officers only

Examples of internally reviewed proposals:• Proposals submitted to Rapid Response

Research Grants (RAPID)

• Proposals submitted to Early-concept Grants for Exploratory Research (EAGER)

• Proposals submitted to Research Advanced by Interdisciplinary Science and Engineering (RAISE)

• Proposals for conferences under $50,000

How are Reviewers Selected?

19

• Types of Reviewers Recruited:

Reviewers with specific content expertise

Reviewers with general science or education expertise

• Sources of Reviewers:

Program Officer’s knowledge of the research area

References listed in proposal Recent professional society programs Computer searches of S&E journal

articles related to the proposal Former reviewers Reviewer recommendations included in

proposal or sent by email

• Three or more external reviewers per proposal are selected.

How Do I Become a Reviewer?

20

• Contact the NSF Program Officer(s) of the program(s) that fit your expertise:

Introduce yourself and your research experience.

Tell them you want to become a reviewer for their program.

Ask them when the next panel will be held.

Offer to send a 2-page CV with current contact information.

Stay in touch if you don’t hear back right away.

What is the Role of the Reviewer?

21

• Review all proposal material and consider:

The two NSF merit review criteria and any program specific criteria.

The adequacy of the proposed project plan including the budget, resources, and timeline.

The priorities of the scientific field and of the NSF program.

The potential risks and benefits of the project.

• Make independent written comments on the quality of the proposal content.

What is the Role of the Review Panel?

22

• Discuss the merits of the proposal with the other panelists

• Write a summary based on that discussion

• Provide some indication of the relative merits of different proposals considered

Why Serve on an NSF Panel?

23

• Gain first-hand knowledge of the merit review process

• Learn about common problems with proposals

• Discover proposal writing strategies

• Meet colleagues and NSF Program Officers managing the programs related toyour research

Managing Conflicts of Interest in the Review Process

24

• The primary purpose is to remove or limit the influence of ties to an applicant institution or investigator that could affect reviewer advice.

• The secondary purpose is to preserve the trust of the scientific community, Congress, and the general public in the integrity, effectiveness, and evenhandedness of NSF’s merit review process.

Affiliations with Applicant Institutions

25

• Examples:

Current employment at the institution

Other association with the institution, such as being a consultant

Being considered for employment or any formal or informal reemployment arrangement at the institution

Any office, governing board membership, or relevant committee membership at the institution

Personal Relationships with Investigator or Project Director

26

• Examples:

Known family or marriage relationship

Business partner

Past or present thesis advisor or thesis student

Collaboration on a project or book, article, or paper within the last 48 months

Co-edited a journal, compendium, or conference proceedings within the last24 months

27


27




DD Concur



Award



Communities

Submit

NSF Program Officer

Ad Hoc

Panel

Combination

Internal



Via DGA

Organization

Funding Decisions

28

• The merit review panel provides:

Review of the proposal and a recommendation on funding.

Feedback (strengths and weaknesses) to the proposers.

• NSF Program Officers make fundingrecommendations guided by programgoals and portfolio considerations.

• NSF Division Directors eitherconcur or reject the Program Officers’ fundingrecommendations.

Feedback from Merit Review

29

• Reviewer ratings (such as: E, V, G, F, P)

• Analysis of how well proposal addresses both review criteria: Intellectual Merit and Broader Impacts

• Proposal strengths and weaknesses

• Reasons for a declination (if applicable)

If you have any questions, contact the cognizant Program Officer

Documentation from Merit Review

30

• Verbatim copies of individual reviews, excluding reviewer identities

• Panel Summary or Summaries (if panel review was used)

• Context Statement (usually)

• PO to PI comments (formal or informal, written, email or verbal) as necessary to explain a decision

Examples of Reasons for Declines

31

• The proposal was not considered to be competitive based on the merit review criteria and the program office concurred.

• The proposal had flaws or issues identified by the program officer.

• The program funds were not adequate to fund all competitive proposals.

Revisions and Resubmissions

32

• Points to consider:

Do the reviewers and the NSF Program Officer identify significant strengths in your proposal?

Can you address the weaknesses that reviewers and the Program Officer identified?

Are there other ways you or your colleagues think you can strengthen a resubmission?

Again, if you have questions, contact the cognizant Program Officer.

NSF Reconsideration Process

33

Explanation from Program Officer and/or Division Director

Written request for reconsideration to Assistant Director within 90 days of the decision

Request from organization to Deputy Director of NSF

Possible Considerations for Funding a Competitive Proposal

34

• Addresses all review criteria

• Likely high impact

• Broadening participation

• Educational impact

• Impact on institution/state

• Special programmatic considerations(e.g. CAREER/RUI/EPSCoR)

• Other support for PI

• “Launching” versus “Maintaining”

• Portfolio balance

35


35



DD Concur



Award



Communities

Submit

NSF Program Officer

Ad Hoc

Panel

Combination

Internal



Via DGA

Organization

Issuing the Award

36

• NSF’s Division of Grants and Agreements (DGA) reviews the recommendation from the program office for business, financial, and policy implications.

• NSF’s grants and agreements officersmake the official award as long as:

The institution has an adequategrants management capacity.

The PI/Co-PIs do not have overdue annual or final reports.

There are no other outstanding issues with the institution or PI.

For More Information

37

Go to NSF’s Home Page (www.nsf.gov)

For More Information

38

Ask Early, Ask Often!

nsf.gov/staff

nsf.gov/staff/orglist.jsp

nsf.gov/about/career_opps/rotators/index.jsp

Documents

Merit Review Process - NSF Grants Conferences · Becoming a Reviewer ... Combination Internal Program Officer ... effectiveness, and evenhandedness of NSF’s merit review process