Arlington, VA March 31, 2004

Arlington, VAMarch 31, 2004

Presentation for theAdvisory Committee for Business & Operations

Effective Practices Research Overview For Merit Review

This document is confidential and is intended solely for the use and information of the client to whom it is addressed.

NSF BUSINESS ANALYSIS

2


Today, we will provide…

Brief update on the status of the NSF Business Analysis

Summary of the key findings from the Effective Practices research for Merit Review

3


NSF Business Analysis has recently competed the following activities and is currently moving into the design phase of the project

Completed as of 3/31/04Completed as of 3/31/04

Completed researching effective practices for the Merit Review and Award Management & Oversight processes

Completed workload study that assessed the amount and distribution of workload across NSF

Completed first full draft of the Target Enterprise Architecture for Applications, Data, Network and Security

Completed researching effective practices for the Merit Review and Award Management & Oversight processes

Completed workload study that assessed the amount and distribution of workload across NSF

Completed first full draft of the Target Enterprise Architecture for Applications, Data, Network and Security

“Works in Progress”“Works in Progress”

Developing process scenarios and preliminary business cases for the Merit Review and Award Management & Oversight processes

Conducting an eJacket study to determine how the electronic-jacket technology tool impacts human capital issues

Developing an IT Technology Governance Framework and IT Implementation Plan that considers the 2-7 year transition plan necessary to move to the Target Enterprise Architecture

Developing process scenarios and preliminary business cases for the Merit Review and Award Management & Oversight processes

Conducting an eJacket study to determine how the electronic-jacket technology tool impacts human capital issues

Developing an IT Technology Governance Framework and IT Implementation Plan that considers the 2-7 year transition plan necessary to move to the Target Enterprise Architecture

Quick Snapshot of Project Status

4


For the “effective” practices research, the team researched the practices of grant-making organizations in order to identify potential alternative practices

Research Objectives:

To understand how other federal and non-federal organizations review proposals and manage awards, and the relative emphasis that is placed on either activity

To provide NSF with insights into how other organizations address challenges similar to those faced by the Foundation in the proposal review and award management processes

To identify practices or principles that could be leveraged or modified by NSF in the redesign of its core business processes

5


The team interviewed representatives of 14 federal and non-federal organizations

Organizations Interviewed

6


Important to note is that few effective practices were identified in the areas of performance measurement, “working across the organization,” and technology usage

Few organizations apply performance metrics to their review or award management practices

– Customer satisfaction surveys are performed ‘ad hoc’

– Processing times tend to be the same or longer than NSF

– Few agencies have monitoring plans in place to address the degree of award oversight that is applied

Few organizations had a significant need for ‘working across the organization’

Many agencies were either “developing” or “progressing” in their eBusiness capabilities with availability of resources cited as the greatest barrier to development efforts

7


Effective practices emerged in each of the following areas

Process for how an organization receives, processes, and streamlines activities associated with proposal intake, review, and decision-making

Reviewer Community

Management

Proposal Workflow

Management

Review Methodologies

Description

Various practices an organization applies to review proposals and make funding decisions with limited resources

Management practices that an organization applies to attract, recruit, and reward its reviewers, as well as how the organization maintains a relationship with its community

Merit Review Areas

8


Reviewer Community

Management

Proposal Workflow Management


Strategy Description

Apply “filters” to reduce the number of proposals that receive reviews and to manage future workload requirements

Leveraging Letters of Intent (LOI), pre-applications or other filters that enable organizations to better manage workload

Identifying early in the process proposals that are noncompetitive or inappropriate, and begin panel coordination earlier in the review process

Coordinate the scheduling of program deadlines to spread the workload more evenly across the fiscal year

For organizations with multiple programs, centrally coordinating the deadlines for programs to reduce the burden on both internal resources and the reviewer community

Employ temporary staff during peak periods or outsource some administrative functions to mitigate workloads

Using temporary employees to provide staff with administrative relief during periods of peak workload. Services range from minor administrative support (i.e., jacket assembly) to outsourcing all proposal support (i.e., panel coordination, soliciting reviewers, conflicts of interest, etc.)


Apply “filters” to reduce the number of proposals that receive reviews and to manage future workload requirements

Leveraging Letters of Intent (LOI), pre-applications or other filters that enable organizations to better manage workload

Identifying early in the process proposals that are noncompetitive or inappropriate, and begin panel coordination earlier in the review process

Coordinate the scheduling of program deadlines to spread the workload more evenly across the fiscal year

For organizations with multiple programs, centrally coordinating the deadlines for programs to reduce the burden on both internal resources and the reviewer community

Employ temporary staff during peak periods or outsource some administrative functions to mitigate workloads

Using temporary employees to provide staff with administrative relief during periods of peak workload. Services range from minor administrative support (i.e., jacket assembly) to outsourcing all proposal support (i.e., panel coordination, soliciting reviewers, conflicts of interest, etc.)

Many organizations employ different strategies to mitigate workload demands

9


Reviewer Community

Management




Structure decision-focused review process; allow reviewers to spend more time on competitive proposals

Posting reviews prior to panel or pre-ranking proposals based on reviewer feedback, then focusing panels on those proposals that require additional feedback upon which to base a decision (e.g., proposals “on the fence”)

Reduce degree of “budget” negotiations that must occur throughout the post-review process

Employing strategies to mitigate the degree of post-review budget discussions include:

– Embedding “cost realism” as a review criteria

– Adjusting budgets and issuing ‘conditional awards’

– Employing policies of non-negotiable awards

Enable reviewers to participate remotely in panels through the use of technology

Providing tools that: facilitate involvement of panelists who are unable to physically be present through remote participation; allow NSF to utilize expert reviewers for the appropriate panels; and, broaden the accessible reviewer pool


Structure decision-focused review process; allow reviewers to spend more time on competitive proposals

Posting reviews prior to panel or pre-ranking proposals based on reviewer feedback, then focusing panels on those proposals that require additional feedback upon which to base a decision (e.g., proposals “on the fence”)

Reduce degree of “budget” negotiations that must occur throughout the post-review process

Employing strategies to mitigate the degree of post-review budget discussions include:

– Embedding “cost realism” as a review criteria

– Adjusting budgets and issuing ‘conditional awards’

– Employing policies of non-negotiable awards

Enable reviewers to participate remotely in panels through the use of technology

Providing tools that: facilitate involvement of panelists who are unable to physically be present through remote participation; allow NSF to utilize expert reviewers for the appropriate panels; and, broaden the accessible reviewer pool

Most organizations structure the review process to optimize the time of reviewers

10


Reviewer Community

Management



Most organizations treat reviewers as “customers” throughout the process


Treat reviewers as “customers” throughout the review process

Coordinating all panel logistics including travel arrangements

Providing social/networking opportunities while on-site

Pay them well Offering attractive honorariums, paying ‘ad hoc’ reviews

Educate reviewers on their duties and responsibilities

Providing training to new panelists. Training could include on-line tutorials, mentoring, or “mock reviews”

Structure “terms of service” to ensure continuity

Requiring panelists to serve for a period of time -- generally for a 3-year term and not necessarily consecutive years

Reach out to the full ‘pool’ of possible reviewer candidates

Providing targeted marketing and outreach efforts to potential reviewers (i.e., soliciting universities, societies, etc.)

Integrating of databases for applicants, reviewers, and awardees

Accepting on-line applications


Treat reviewers as “customers” throughout the review process

Coordinating all panel logistics including travel arrangements

Providing social/networking opportunities while on-site

Pay them well Offering attractive honorariums, paying ‘ad hoc’ reviews

Educate reviewers on their duties and responsibilities

Providing training to new panelists. Training could include on-line tutorials, mentoring, or “mock reviews”

Structure “terms of service” to ensure continuity

Requiring panelists to serve for a period of time -- generally for a 3-year term and not necessarily consecutive years

Reach out to the full ‘pool’ of possible reviewer candidates

Providing targeted marketing and outreach efforts to potential reviewers (i.e., soliciting universities, societies, etc.)

Integrating of databases for applicants, reviewers, and awardees

Accepting on-line applications

11


Based on the findings in the effective practices research, NSF should consider the following in the scenario design

Broadening the use of Letters of Intent (LOIs) for determining program fit and begin planning and coordinating panels earlier in the process

More pro-actively coordinating program deadlines taking into account the needs of the research community

Developing piloting opportunities to broaden the use of temporary help during peak proposal times or outsourcing certain process functions

Incorporating process changes that reduce the degree of proposal budget rework that occurs throughout the Merit Review process

Compensating off-site reviews, broadening the concept of panel “terms of service”, and providing training for how best to perform a review

Developing strategies that “reach out” to a greater pool of reviewers (e.g., integrating databases for applicants, awardees, reviewers; enabling better remote access; accepting on-line reviewer applications, etc.)

12


Questions & Answers

For additional questions, please contact:

– Abe Zwany, [email protected], 703.902.5342

– Tim Koch, [email protected], 703.377.0389

Documents

Arlington, VA March 31, 2004