SHORTENING THE REVIEW CYCLE Archived File The file below has been archived for historical reference purposes only. The content and links are no longer

SHORTENING THE REVIEW CYCLESHORTENING THE REVIEW CYCLE

Archived FileArchived File

The file below has been archived for historical reference purposes only. The content and links are no longer

maintained and may be outdated. See the OER Public Archive Home Page for more details about

archived files.


SSTRUCTUREDTRUCTURED CCRITIQUESRITIQUES

September 26, 2005

Associate Director forScientific Review and Policy

DEA, NIAID

Hortencia Hornbeak, Ph.D.Hortencia Hornbeak, Ph.D.

PPEER REVIEW EER REVIEW AADVISORY DVISORY CCOMMITTEE OMMITTEE MMEETINGEETING


Outline for Today’s DiscussionOutline for Today’s DiscussionMulti-pronged approach to shortening review cycle

Principal drivers of structured critiques

Structured critique formats

Structured critiques-award mechanisms

Reviewer guidance

Changing reviewer behavior

Resources needed


Features of NIH Peer Review thatFeatures of NIH Peer Review thatMeet the Test of TimeMeet the Test of Time

Management of conflict of interest (core value)

Secure appropriate expertise (core value)

Develop collective expert advice through discussion

Provide guidance/feedback to applicant


Multi-pronged ApproachMulti-pronged Approach Electronic submission of applications

Knowledge management solutions in referral/recruitment

Electronic recruitment of reviewers for SEPs

Internet assisted review (IAR)

Structured critiques

Abbreviated summary statements

Council approval independent of scheduled meetings


Structured Critiques: Principal DriversStructured Critiques: Principal Drivers

Accelerated review for AIDS initiatives (mid 1980s)

Hyper-accelerated review of Innovation grants for HIV/AIDS hyper-accelerated review

Large increase in funds ($1.5 B) for biodefense research (2003)

Five review cycles (73 days per cycle) per FY60 days from receipt to review


Structured Critiques: Principal Drivers Structured Critiques: Principal Drivers (cont.)(cont.)

Hyper-accelerated review of Bioshield initiatives- Project Bioshield Legislation 2004No FACA rulesOutside IMPAC II

Constraints on FY funding

Lag time to hire and train staff


Five Review Cycles: FY03-04Five Review Cycles: FY03-04 73 days per cycle for most reviews

SRA team work

Flexible SRA assignments

Advance electronic recruitment of reviewers for SEPs

based on letters of intent

Structured critiques

IAR assembly of abbreviated summary statements

Administrative review report to GMB and Program


Five Review Cycles: FY03-04 (cont.)Five Review Cycles: FY03-04 (cont.) Lessons learned

Secure appropriate expertise with large numbers of potential reviewers in conflict

Tools developed to manage large reviews Structured critiquesPractical guidance for reviewersAdministrative workbooksReviewer Support SitePublication search macro for conflicts


Five Review Cycles: FY03-04 (cont.)Five Review Cycles: FY03-04 (cont.)Effective flexible working teams

Conflict of interest management

Development of staff training resources

Effective communications with stakeholders

Pre-meeting teleconferences to educate reviewers

Increase use of teleconferences for reviews


STRUCTURED CRITIQUE FORMATSTRUCTURED CRITIQUE FORMAT


Structured Critique - BenefitsStructured Critique - Benefits

Goal Improve utility for stakeholders by:

More focused discussionsMore concise written evaluationsFacilitated preparation of abbreviated summary

statements


Structured Critique - ChallengesStructured Critique - ChallengesRequires change in behavior of reviewers

May adversely affect “tutorial” aspect

May not capture adequately complexity and/or subtleties

May lead to applicant appeal


Structured Critiques: Simple and Structured Critiques: Simple and Complex MechanismsComplex Mechanisms

R03

R21

R01/U01

P54 Regional Centers - two-tiered review

P01/U19

UC6 Biocontainment laboratories


Structured Critique TemplateStructured Critique Template

Tailored for each initiative

Initiative-specific review criteria

Use short phrases to describe each review criteria

Based on the five NIH review criteria as a scaffold for initiative specific criteria


Critique Structured FormatCritique Structured Format

Significance:

Strength # 1 [strongest] (1-2 lines)Justification/explanation (2-4 lines)

Strength # 2 (1-2 lines) Justification/explanation (2-4 lines)

Strength # X (1-2 lines)Justification/explanation (2-4 lines)


Weakness # 1 [most serious weakness] (1-2 lines)Justification/explanation (2-4 lines)

Weakness # 2 (1-2 lines) Justification/explanation (2-4 lines)

Weakness # X (1-2 lines) Justification/explanation (2-4 lines)

Critique Structured Format (cont.)Critique Structured Format (cont.)


Continue with format in previous slides for:

Approach

Innovation

Investigator

Environment Criteria

Critique Structured Format (cont.)Critique Structured Format (cont.)


EXAMPLE OF AEXAMPLE OF A STRUCTURED CRITIQUE STRUCTURED CRITIQUE


Example of Structured CritiqueExample of Structured Critique

RESUME AND SUMMARY OF DISCUSSION: This (adjectival

rating) application entitled “ " was submitted by

(organization), Principal Investigator. The applicants propose to (one

sentence summary of specific aim(s). The principal strengths identified by

reviewers include , , , and . The

major weaknesses are , , , and .

After discussion, the panel concluded that , and

scored the application in the range.


Example of Structured Critique (cont.)Example of Structured Critique (cont.)

OVERALL EVALUATION

This is an excellent application in response to the RFA…... It is based on

solid hypotheses and preliminary evidence that XX inhibitors can protect

from toxin-mediated pathogenicity of B. anthracis. The team is highly

experienced and has worked together to establish the background for the

project. Goals are clearly laid out, experiments are generally well described,

and there is a high probability of significant advances in therapy of anthrax-

related toxicity in patients.



CRITIQUE

Significance Strength 1 - novel approach to block enzymatic conversion of a

bacterial XX protoxin in several bacteria, including B. anthracis

Strength 2 - excellent likelihood of successful development of lead compounds

Strength 3 - excellent opportunity to test lead compounds directly in toxin-treated animals

Weakness 1 - unknown toxicity related to furin inhibitors


Approach

Strength 1 - solid rationale and preliminary evidence for Pseudomonas-related toxin XX, including protection of toxin-induced death in mice

Strength 2 - lead compound XXX has potent binding to target

Strength 3 - excellent team to conduct studies




Approach (cont.)

Weakness 1 - Is furin block in vivo sufficient to cure infection or only protect from XX-mediated toxicity?

Weakness 2 - in preliminary results, the PI concludes that “XXX actually enters cells”, but the data only demonstrate that “biotinylated XX” enters cells - they are quite different substances. To further understand the apparent lack of cytotoxicity of XXX, permeabilized cells should be used.


Innovation Strength 1 - detailed mechanistic study of XXX effect on XX

toxicity to RAW cells

Strength 2 - study of kinetics and time course of protection from XX and cell-surface bound iodinated XX

Strength 3 - design of XX analogs including synthetic peptides, polyamines and related peptidomimetics

Strength 4 - in vivo studies examining efficacy of XXX on TNF production and on XX induced toxicity in animal models

Weakness 1 –the approach not innovative and could benefit from collaborative expertise for these studies.



Investigator The PI is experienced in peptide hormone studies and peptide processing. Collaborator Dr. XX is experienced in peptide synthesis. The team has successfully worked together in the discovery of polyarginine furin inhibitors. The application would be strengthen by additional collaborative expertise for XX studies

Environment Lab facilities of the PI are adequate for the studies. Presumably

those of Dr. XX’s institution are also adequate.



Budget

Appropriate for the level of effort proposed.

Animal Welfare

Comments: A letter of approval of an animal protocol was included. It does not indicate if both mice and rats are approved for the study. The PI does not address the “five questions” about animal usage. Nature of anesthesia is not indicated.



STRUCTURED CRITIQUES: STRUCTURED CRITIQUES: MECHANISM-SPECIFIC EXAMPLESMECHANISM-SPECIFIC EXAMPLES


Structured Critiques: Structured Critiques: Simple to Complex InitiativesSimple to Complex Initiatives

R21- Rapid Response Grant Program on Bioterrorism-Related Research

U01/U19 RFA Cooperative Research for the Development of Vaccines, Adjuvants, Therapeutics, Immunotherapeutics, and Diagnostics for Biodefense

Biodefense Countermeasure Development


Structured Critiques: Structured Critiques: Benefits – All MechanismsBenefits – All Mechanisms

More focused assessments of the applications

Increased time for discussion of competitive applications

Enabled committee to manage larger number of applications per SEP.

Faster SS completion for awarded grants


Structured Critiques - R21Structured Critiques - R21 RFA Rapid Response Grant Program on Bioterrorism-

Related Research (2002)

Mechanism R21

Number of applications 303

Review committees 3 SEPs

Number of members/SEP 25

Streamlined applications 33 percent


Structured Critiques - R21 (cont.)Structured Critiques - R21 (cont.)

Electronic critique templates

3 SRAs assigned

Resume and summary of discussion templates

Faster SS completion – 4 weeks


Structured Critiques - U01s and U19sStructured Critiques - U01s and U19s

RFA Cooperative Research for the Development of Vaccines, Adjuvants, Therapeutics, Immunotherapeutics, and Diagnostics for Biodefense (2003)

Mechanism U01, U19 Number of applications 186

Review committees 3 SEPs

Number of members/SEP 40

Streamlined applications 33 percent



Structured Critiques - U01s and U19s (cont.)Structured Critiques - U01s and U19s (cont.)

Resume and summary of discussion templates

Templates for resume and summary of discussion

3 SRAs assigned

Faster SS completion - 3 weeks


Structure Critiques - Project BioshieldStructure Critiques - Project Bioshield

Biodefense Countermeasure Development Hyper-accelerated Review (2004)

Mechanism N/A (small grants)Number of applications 66Letter of intent/program

Pre-approvalReview committees 1 SEPNumber of members/SEP 23Streamlined applications 60 percent


Structure Critique - Project Bioshield (cont.)Structure Critique - Project Bioshield (cont.)

Teleconference review

Receipt to award 60 day


Enabled committee to manage the large workload in a hyper-accelerated time frame

Faster SS completion - 2 weeks

Not standard summary statements


Guidance Provided to ReviewersGuidance Provided to ReviewersOrientation teleconference with members from

multiple committee for standardization

Electronic critique template tailored to the initiative.

Instructions for Reviewer Manual tailored for each initiative or SEP

Overlapping reviewers among the review panels


Changing Behavior of ReviewerChanging Behavior of ReviewerClearly stated expectations

Focused rational rationale and objectives

Required buy-in from all stake holdersNIHNIAIDreviewers

Make it easy across the board


Resources Needed to Decrease TimeResources Needed to Decrease Time

Sufficient numbers of trained staff

Effective SRA/GC Work teams

Flexible staff assignments

Versatile and state of the art IT support


SummarySummaryPreserve features that make NIH Peer Review work

with any changes adaptedManage conflict of interest (core value)Secure appropriate expertise (core value)Develop collective expert advice through

discussionProvide guidance/feedback to applicant

Structured critiques are applicable to many award mechanisms: simple to complex


Summary (Cont.)Summary (Cont.)

Reviewers must be properly oriented in pre-review teleconferences

Reviewers do adapt to change

Structured critiques decrease time for summary statement preparation

Resulting product (SS) more useful to stakeholders


Summary (cont.)Summary (cont.) Structured critiques along with other innovations

can shorten the review cycle

Electronic submission of applications

Knowledge management solutions in referral

Electronic recruitment of reviewers for SEPs

Internet assisted review (IAR)

Abbreviated summary statements

Council approval independent of scheduled meetings


Questions?Questions?

Documents

SHORTENING THE REVIEW CYCLE Archived File The file below has been archived for historical reference purposes only. The content and links are no longer