Upload
baina
View
18
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Peer Review of ARS Research Projects. A Brief Overview of the Process. Office of Scientific Quality Review Steven Huber, Scientific Quality Review Officer. OSQR Web Site http://www.osqr.ars.usda.gov. National Programs Schedules of Reviews Forms and Resources (TIPS!) Reviewer Information - PowerPoint PPT Presentation
Citation preview
Peer Reviewof ARS Research Projects
Peer Reviewof ARS Research Projects
Office of Scientific Quality ReviewSteven Huber, Scientific Quality Review
Officer
A Brief Overview of the Process
OSQR Web Sitehttp://www.osqr.ars.usda.gov
OSQR Web Sitehttp://www.osqr.ars.usda.gov
National ProgramsSchedules of ReviewsForms and Resources (TIPS!)Reviewer InformationPeer Review statusManual (online comments soon)Staff
Topics to CoverTopics to Cover
BackgroundRole of the Research LeaderThe Prospectus & Project PlanGeneral writing tipsThe panel processOutcomes
New Mandate for this processNew Mandate for this process
The Agricultural Research, Extension and Education Reform Act of 1998 (public law 105-185):
ARS projects be evaluated by panels composed predominantly of non-ARS scientists
With oversight from the REE Advisory Committee
Intent of Peer ReviewIntent of Peer Review
To obtain constructive, independent, expert feedback and advice on ways to improve the scientific and technical merit of each project plan
NOT funding issues
Fundamental QuestionsFundamental Questions
Panelists are asked to evaluate:
MERIT AND SIGNIFICANCE (Relevance to NP) APPROACHES AND PROCEDURES LIKELIHOOD OF SUCCESS
(feasibility)
Suggestions for improvements or solutions to problems are solicited
Steps in the ProcessSteps in the Process
RL and Scientists develop Prospectus
RLs Prepare Project Plan with LSsSubmit Plan for approval
Peer Review
Response and RevisionSubmit revised project plan
Re-review?
NPS
AD
OSQR
AD/NPS
OSQR
16 wk(24 wk)
18 wk(14 wk)
6 wk
6 wk
6 wk
Scientist’s ResponsibilitiesScientist’s Responsibilities
• Communicate with the NP Team.• Scope of the project.• Coordination with other projects.
• QUALITY of the Project Plan.• Many are ‘poorly’ written.
Pro
ject
Pla
n T
em
pla
tePro
ject
Pla
n T
em
pla
te Title and Investigator(s) TOC (coming soon!) Project summary Objectives (< ½ p) Need for research (1p) Scientific Background (5-6 p) Approach & Procedures (6-12 p) Milestones & Outcomes ( 1 p) Literature Cited Past Accomplishments Issues of Concern statements Appendices
15 - 25 pages
Scientific Background Scientific Background
Focus on the pertinent literature
Show linkage/coordination with other similar projects in the NP (or other NPs) and with minority-coded projects in the MU.
GENERAL TIPGENERAL TIP
Use illustrations (figures, schemes, etc) to enhance the Plan. (Up to 2 pages does not count against page limit).
Preliminary dataHow your project fits with othersWorking modelsExperimental design/treatments
Approach and Procedures—TipsApproach and Procedures—Tips
Lack of necessary detail is the most common criticism.
Tell what will be done, by whom, and what will result. Identify staff involved with key portions of the project.
If an objective is supported by a grant, mention that.
The PanelThe Panel
PanelistsPrimary reviewersSecondary reviewers
Panel Chair
Ad-hoc reviewers may be used
OSQR Officer
Provides orientation ObservesDebriefsReceives results
Panel Chair SelectionPanel Chair Selection
SUGGESTIONSfor Non-ARS Panel Chair
from NPS, ADs, etc.
OSQR OfficerSelects Chair
OSQR OfficerConsults inside
and outsideARS
Panelist SelectionPanelist Selection
SUGGESTIONS for Panelistsfrom NPS, ADs, RLs, Lead Scientists, University Scientists, Customers, etc.
OSQR OfficerApproves Panel*
Panel Chair (PC)Makes
recommendations toOSQR Officer
*Based on PC recommendations and diversity requirements
Suggest
ed P
anelis
tsSuggest
ed P
anelis
tsSUGGESTED PANEL MEMBER
Full Name of Prospective Panelist:
Full Title:
Business Address:
Tele: FAX: Email:
Scientific Area(s) of Expertise:
National Program(s):
101 Animal Genomes, Germplasm, Reprod. 205 Grazinglands Management 102 Animal Production Systems 206 Manure and Byproduct Utilization 103 Animal Health 207 Integrated Farming Systems 104 Arthropod Pests of Animals & Humans 301 Plant Microbial & Insect Germplasm... 105 Animal Well-Being & Stress Contr. Syst. 302 Plant Biological & Molecular Processes 106 Aquaculture 303 Plant Diseases 107 Human Nutrition Reqs., Food Cons... 304 Crop & Commodity Pest Biology... 108 Food Safety 305 Integrated Crop Production & Protection 201 Water Quality 306 New Uses, Quality, & Marketability of 202 Soil Quality & Management Plant & Animal Products 203 Air Quality 307 Bioenergy & Energy Alternatives 204 Global Change 308 Methyl Bromide Alternatives
Please provide further pertinent information on this prospect (such as prior panels served, etc.):
PICReturn this form with your Project Outline to your National Program Team LeaderNPLCForward this form with the Project Outline to the OSQRADCForward this form to the OSQR
Use this form!
The Panel Meeting Process
Panel Discussion
Primary reviewer presents the plan input from
secondary reviewer, panelists, and ad hoc reviewer(s).
Panelists and Panel Chair assign an action class to the project.
Turn into SQR Officer
Primary reviewer modifies the review recommendation on disk
1 hour
On breaks, evenings
Panel Chair Approval
Turn into SQR Officer
Action Classes*Action Classes*
Each panelist individually provides overall ‘Action Class’ assignment: No revision required. Minor revision might be made.
Minor revision required. Objectives fit the NP action plan; approaches to all objectives are sound. Project is feasible.
Moderate revision required. Moderate revision of an objective and/or one approach needed. Project is feasible.
Major revision required. Should be sound and feasible after major revision. (May be re-reviewed).
Not feasible. Deficiencies in expertise or facilities or major flaw.
*Action Class Matrix will be provided
After the Panel MeetingAfter the Panel Meeting
If the composite Action Class is: No revision required Minor revision required Moderate revision required
Major revision required
Not feasible
Scientists solicit comments and revise the Plan within 6 weeks
Revised Plans will be revised AND re-reviewed
Revise for immediate re-review; delay revision until major resource changes (e.g., personnel) are made; or terminate
Responding to the Review Recommendations
Project Title:_____________________________________________________________
CRIS #_______________________ National Program__________________________
Lead Scientist:_________________________________ Reviewer #_________________
PEER REVIEW OF ARS RESEARCH PROJECT
2. Adequacy of Approach and Procedures:
Please comment on the following: Are the hypotheses and/or plan of work well conceived? Are the experiments, analytical methods, and approaches and procedures appropriate and sufficient to accomplish the objectives? How could the approach or research procedures be improved?
Please list and number each significant recommendation being made. Be sure to briefly state the rational or basis for suggestions made or questions raised. Each recommendation can include several specific questions you believe should be addressed by the lead scientist.
1. Objective 1--The hypothesis being tested is only one of several that should be considered. For example……
ARS Response:
ARS Responses to Review Recommendations
ARS Responses to Review Recommendations
ARS scientists must respond to each major recommendation.
• TONE of the response: should be receptive, not defensive or condescending.
•CONTENT: make all reasonable efforts to accommodate suggestions made. Lack of adoption must be justified. Should ‘stand alone.’
The completed forms go back to the panelists!
‘Re-Review’ of Project Plans‘Re-Review’ of Project Plans
Purpose:
1) independent evaluation of response, and
2) maintain credibility with community.
NP108, NP302, NP202 Panel StatisticsNP108, NP302, NP202 Panel Statistics
7 Panel Chairs and 44 Panelists (51 total)
1 ARS employee 33 employed by universities 7 employed by private industry, trades groups, or
self-employed 10 Other federal agencies (U.S. and Canada)
NP108, NP302, NP202 Panel results
NP108, NP302, NP202 Panel results
0
10
20
30
40
50
NO MIN MOD MAJ NOT
Action Class
Num
ber
of P
roje
cts
Comments from PanelistsComments from Panelists
Think the peer review process is important for the agency
Appreciated the positive approach of the ARS peer review process.
Enjoyed the experience (opportunity for them to discuss science with peers).
Hope the review recommendations will be useful and incorporated.
Benefits of Peer ReviewBenefits of Peer Review
Better short- and long-term thinking and planning of research (if done properly!).
Increased communication among scientists, ADs and NPLs.
Increased collaboration (especially among ARS labs).
Increased quality—perspectives from peer reviewers.
Greater ability to compete for competitive funds.
OSQR Web Sitehttp://osqr.ars.usda.gov/
OSQR Web Sitehttp://osqr.ars.usda.gov/
• National Programs
• Schedules of Reviews
• Forms and Resources
• Reviewer Information
• Peer Review Status
• Manual
• Staff