The Review of Epidemiologic Research The Review of Epidemiologic Research Applications at the Applications at the

National Institutes of Health: National Institutes of Health: Process, Products, and Best Possible Process, Products, and Best Possible

ScienceScience

Investigator Viewpoint, from Idea to Submission, Investigator Viewpoint, from Idea to Submission, andand

Study Section Viewpoint, from Submission to Study Section Viewpoint, from Submission to AssessmentAssessment

Presented at the 44Presented at the 44thth Annual Conference on Annual Conference on Cardiovascular Disease Epidemiology and Cardiovascular Disease Epidemiology and

Prevention of the American Heart Association, Prevention of the American Heart Association, March 3, 2004, San Francisco, CAMarch 3, 2004, San Francisco, CAJ. Scott Osborne, III, Ph.D., M.P.H.J. Scott Osborne, III, Ph.D., M.P.H.

Scientific Review AdministratorScientific Review Administrator

Epidemiology of Chronic Diseases Study SectionEpidemiology of Chronic Diseases Study Section

Center for Scientific Review, National Institutes of Center for Scientific Review, National Institutes of HealthHealth

Department of Health and Human ServicesDepartment of Health and Human Services

PROCESS OF REVIEWPROCESS OF REVIEW

From Idea to Application, From Idea to Application, Submission, and AssessmentSubmission, and Assessment

PRODUCTS OF REVIEWPRODUCTS OF REVIEW Assessment of Scientific Merit Assessment of Scientific Merit and Summary Statementand Summary Statement

BEST POSSIBLE SCIENCEBEST POSSIBLE SCIENCE

PROCESS OF REVIEWPROCESS OF REVIEW IDEA proposed in Application (PHS 398)IDEA proposed in Application (PHS 398) SUBMISSION of APPLICATION to NIH SUBMISSION of APPLICATION to NIH

Division of Receipt and Referral, Center Division of Receipt and Referral, Center for Scientific Reviewfor Scientific Review

ASSIGNMENT to Institute(s) and Study ASSIGNMENT to Institute(s) and Study Section (suggestions welcome)Section (suggestions welcome)

REVIEW by NIH STUDY SECTIONREVIEW by NIH STUDY SECTION ASSESSMENT OF SCIENTIFIC MERIT ASSESSMENT OF SCIENTIFIC MERIT

(score and percentile rank) for Institute (score and percentile rank) for Institute Advisory Councils and Center Advisory Advisory Councils and Center Advisory BoardsBoards

PRODUCTS OF REVIEWPRODUCTS OF REVIEW

PRE-REVIW: APPLICATION – development PRE-REVIW: APPLICATION – development of an idea into a meaningful projectof an idea into a meaningful project

REVIEW: RECOMMENDATION (score; REVIEW: RECOMMENDATION (score; percentile rank) regarding ASSESSMENT percentile rank) regarding ASSESSMENT of Scientific and Technical Merit of of Scientific and Technical Merit of ApplicationApplication

POST-REVIEW: SUMMARY STATEMENTPOST-REVIEW: SUMMARY STATEMENT

BEST POSSIBLE SCIENCE:BEST POSSIBLE SCIENCE:Contributions of the Process of Contributions of the Process of

ReviewReview PRODUCTS OF REVIEW LEAD TO BEST PRODUCTS OF REVIEW LEAD TO BEST

POSSIBLE SCIENCE.POSSIBLE SCIENCE. ASSESSMENTS regarding Scientific and ASSESSMENTS regarding Scientific and

Technical Merit guide decision making.Technical Merit guide decision making. SUMMARY STATEMENTS provide SUMMARY STATEMENTS provide

written feedback, allowing written feedback, allowing improvement of projects and improvement of projects and meaningful revisions of applications for meaningful revisions of applications for resubmissions.resubmissions.

NIH STAFF provide guidance. NIH STAFF provide guidance.

BEST POSSIBLE BEST POSSIBLE SCIENCE:SCIENCE:

QUALITIESQUALITIES Moves field forward.Moves field forward.

Has clinical and pubic health Has clinical and pubic health significance.significance.

May provide clues in areas outside May provide clues in areas outside specific framework of study.specific framework of study.

BEST POSSIBLE SCIENCE:BEST POSSIBLE SCIENCE:“The Impossible Dream”“The Impossible Dream”

Perfectly Justified AimsPerfectly Justified Aims Perfect Clarity of Exposures, Perfect Clarity of Exposures,

Confounders, Effect Modifiers, and Confounders, Effect Modifiers, and OutcomesOutcomes

Perfect Design for a Perfect SamplePerfect Design for a Perfect Sample Perfect Methods for Ascertainment Perfect Methods for Ascertainment

and Measurement, Perfect Validity and Measurement, Perfect Validity and Reliability of Dataand Reliability of Data

Perfect AnalysisPerfect Analysis Perfect Interpretation and ResponsePerfect Interpretation and Response

BEST POSSIBLE BEST POSSIBLE SCIENCESCIENCE

CONVENIENCE vs. QUALITY – Select the CONVENIENCE vs. QUALITY – Select the best (e.g., valid, generalizable, etc.) best (e.g., valid, generalizable, etc.) possible sample and approach to possible sample and approach to collecting data; do not opt for collecting data; do not opt for convenience.convenience.

The “Ideal” or “Gold Standard” – Fully The “Ideal” or “Gold Standard” – Fully explore strengths and limitations of an explore strengths and limitations of an approach, and consider a selected approach, and consider a selected approach in comparison to alternatives.approach in comparison to alternatives.

““Best Approximation” – Propose a study Best Approximation” – Propose a study well justified in the context of the current well justified in the context of the current state of the field.state of the field.

3 “TIERS” OF3 “TIERS” OFEPIDEMIOLOGIC EPIDEMIOLOGIC

RESEARCHRESEARCH

DESCRIPTIONDESCRIPTION ETIOLOGYETIOLOGY INTERVENTION (including INTERVENTION (including

clinical trial epidemiology as well clinical trial epidemiology as well as primary, secondary, and as primary, secondary, and tertiary prevention)tertiary prevention)

INVESTIGATORINVESTIGATORVIEWPOINTVIEWPOINT

From IDEA to From IDEA to SUBMISSIONSUBMISSION

From Idea to Application: From Idea to Application: IDEAIDEA

IDEA: Significance -- “So what”?IDEA: Significance -- “So what”? IDEA: Appropriately focused or too broad? IDEA: Appropriately focused or too broad? IDEA: Innovative, standard, or already in IDEA: Innovative, standard, or already in

progress or done? (Ask colleagues, and do progress or done? (Ask colleagues, and do a thorough search of the literature. Check a thorough search of the literature. Check CRISPCRISP at the NIH Website for funded NIH at the NIH Website for funded NIH research.)research.)

IDEA: Evidence to support in literature IDEA: Evidence to support in literature (animal and human studies)? Need pilot (animal and human studies)? Need pilot data to support?data to support?

IDEA: Plausibility; MechanismIDEA: Plausibility; Mechanism

INNOVATIONINNOVATION

New Question, New ApproachNew Question, New Approach

New Question, Old ApproachNew Question, Old Approach

Old Question, New ApproachOld Question, New Approach

From Idea to Application: From Idea to Application: IDEAIDEA

Is the NIH interested in this area? Is the NIH interested in this area? (Check for a Program Announcement (Check for a Program Announcement or Request for Applications. Call NIH or Request for Applications. Call NIH Program Staff at different relevant Program Staff at different relevant Institutes to discuss your idea and Institutes to discuss your idea and how it fits with their programs.)how it fits with their programs.)

SELECT APPROPRIATE MECHANISMSELECT APPROPRIATE MECHANISM

From Idea to Application: From Idea to Application: SUPPORTSUPPORT

STANDARD NIH MECHANISMS OF STANDARD NIH MECHANISMS OF GRANT SUPPORT REVIEWED IN GRANT SUPPORT REVIEWED IN

ECDECD

FELLOWSHIPSFELLOWSHIPS SMALL GRANTS (R03)SMALL GRANTS (R03) R21R21 R01R01

From Idea to Application: From Idea to Application: INVESTIGATIVE TEAMINVESTIGATIVE TEAM

Investigative Team: Convenience vs. Investigative Team: Convenience vs. Quality. NETWORK!!!Quality. NETWORK!!!

The Principal Investigator should The Principal Investigator should develop an appropriate team. This develop an appropriate team. This may include one or more Co-may include one or more Co-Principal Investigators as well as Co-Principal Investigators as well as Co-Investigators, Collaborators, and Investigators, Collaborators, and Consultants. Persons on the Consultants. Persons on the investigative team are often from investigative team are often from different institutions.different institutions.

From Idea to Application: From Idea to Application: INVESTIGATIVE TEAMINVESTIGATIVE TEAM

INTELLECTUAL PROPERTY: An application INTELLECTUAL PROPERTY: An application is submitted by an “applicant institution” is submitted by an “applicant institution” (e.g., your university) but is the “intellectual (e.g., your university) but is the “intellectual property” of the investigators.property” of the investigators.

CONSULTANT ROLES vary according to the CONSULTANT ROLES vary according to the extent of involvement, but they are generally extent of involvement, but they are generally viewed as a service and do not involve a viewed as a service and do not involve a proprietary intellectual contribution to a proprietary intellectual contribution to a project.project.

SERVICES GENERALLY AVAILABLE (e.g., SERVICES GENERALLY AVAILABLE (e.g., assays) are not viewed as involving a assays) are not viewed as involving a proprietary intellectual contribution to a proprietary intellectual contribution to a project.project.

From Idea to Application:From Idea to Application:ENVIRONMENTENVIRONMENT

General Institutional SupportGeneral Institutional Support Specific Resources Available for Specific Resources Available for

ProjectProject Evidence of Departmental Level Evidence of Departmental Level

SupportSupport Potential for Success in Overall Potential for Success in Overall

EnvironmentEnvironment

From Idea to Application:From Idea to Application:METHODSMETHODS

DESIGNDESIGN SAMPLESAMPLE

REPRESENTATIVENESS AND REPRESENTATIVENESS AND GENERALIZABILITY (External Validity)GENERALIZABILITY (External Validity)

POWER (and justification for Effect Size)POWER (and justification for Effect Size) DATA COLLECTIONDATA COLLECTION ANALYSIS OF DATA (linked to ANALYSIS OF DATA (linked to

Specific Aims)Specific Aims) POTENTIAL BIASES AND POTENTIAL BIASES AND

LIMITATIONSLIMITATIONS

From Idea to Application:From Idea to Application:Critical ThinkingCritical Thinking

Be your own worst critic. (About Be your own worst critic. (About Everything!)Everything!)

““So what”? (Significance)So what”? (Significance) Would you believe findings from Would you believe findings from

this study? (Methods)this study? (Methods)

From Idea to Application:From Idea to Application:PHS 398 APPLICATION PHS 398 APPLICATION

FORMFORMRESEARCH PLANRESEARCH PLAN

SPECIFIC AIMSSPECIFIC AIMS BACKGROUND AND BACKGROUND AND

SIGNIFICANCESIGNIFICANCE PRELIMINARY STUDIES or PRELIMINARY STUDIES or

PROGRESS REPORTPROGRESS REPORT RESEARCH DESIGN AND RESEARCH DESIGN AND

METHODSMETHODS HUMAN SUBJECTSHUMAN SUBJECTS

From Idea to Application:From Idea to Application:SUBMISSIONSUBMISSION

RECEIPT DATESRECEIPT DATES

REQUEST STUDY SECTION REQUEST STUDY SECTION ASSIGNMENTASSIGNMENT

STUDY SECTION STUDY SECTION VIEWPOINTVIEWPOINT

From SUBMISSION to From SUBMISSION to ASSESSMENT OF SCIENTIFIC ASSESSMENT OF SCIENTIFIC

MERITMERIT

NIH SUPPORTNIH SUPPORT

CONTRACTS (Solicited with CONTRACTS (Solicited with Request for Proposals -- RFP)Request for Proposals -- RFP)

COOPERATIVE AGREEMENTSCOOPERATIVE AGREEMENTS SOLICITED SET-ASIDES: RFA SOLICITED SET-ASIDES: RFA

(Request for Applications; 1 (Request for Applications; 1 submission, 1 review)submission, 1 review)

Program Announcements (PA)Program Announcements (PA) UNSOLICITED APPLICATIONSUNSOLICITED APPLICATIONS

Epidemiology of Chronic Epidemiology of Chronic Diseases (ECD) Study Diseases (ECD) Study

SectionSection Unsolicited applicationsUnsolicited applications 3 cycles (“rounds”) per year3 cycles (“rounds”) per year

Submission/Receipt Submission/Receipt (FEB2004/June2004/Oct2004)(FEB2004/June2004/Oct2004)

Review (JUNE 2004/Oct2004/Feb2005)Review (JUNE 2004/Oct2004/Feb2005) Council (OCT 2004/Jan2005/May 2005)Council (OCT 2004/Jan2005/May 2005) Award by Institute to “Applicant Award by Institute to “Applicant

Institution” vs. “Principal Investigator”Institution” vs. “Principal Investigator” “ “New Investigator” Status – A “new New Investigator” Status – A “new

investigator” has not previously been a investigator” has not previously been a principal investigator for an R01.principal investigator for an R01.

APPLICATION ASSIGNMENT APPLICATION ASSIGNMENT NUMBERSNUMBERS

ORIGINAL SUBMISSIONORIGINAL SUBMISSION 1 R01 HL 012345-011 R01 HL 012345-01

REVISED SUBMISSIONSREVISED SUBMISSIONS 1 R01 HL 012345-01A11 R01 HL 012345-01A1 1 R01 HL 012345-01A21 R01 HL 012345-01A2

CONTINIUATIONSCONTINIUATIONS 2 R01 HL 012345-062 R01 HL 012345-06 2 R01 HL 012345-06A12 R01 HL 012345-06A1 2 R01 HL 012345-06A22 R01 HL 012345-06A2

SUPPLEMENTS (within project period)SUPPLEMENTS (within project period) 3 R01 HL 012345-01, -01A1, -01A23 R01 HL 012345-01, -01A1, -01A2

Function of ECD Study Function of ECD Study SectionSection

ECD assesses NIH applications for ECD assesses NIH applications for scientific and technical merit; this is scientific and technical merit; this is the first level of peer review.the first level of peer review.

NIH Institute Advisory Councils NIH Institute Advisory Councils assess applications for relevance to assess applications for relevance to NIH goals and public health needs; NIH goals and public health needs; this is the second level of peer this is the second level of peer review.review.

The Institute Directors use these The Institute Directors use these assessments to make funding assessments to make funding decisions.decisions.

Criteria to Evaluate Criteria to Evaluate Scientific and Technical Scientific and Technical

MeritMerit Significance: Is it? Why? Advance Significance: Is it? Why? Advance

science?science? Approach: Design/Population and Approach: Design/Population and

Sample/Data Collection/AnalysisSample/Data Collection/Analysis Innovation: Aims or approach?Innovation: Aims or approach? Investigator(s)Investigator(s) EnvironmentEnvironment

Study Section Study Section Perspective:Perspective:Pre-MeetingPre-Meeting

Submission/Receipt of ApplicationsSubmission/Receipt of Applications Self-Referral: Request Study Section in Self-Referral: Request Study Section in

cover lettercover letter SRA decides if appropriate for Study SRA decides if appropriate for Study

Section after initial referralSection after initial referral Institutes determine their interest and Institutes determine their interest and

roles as primary, secondary, etc.roles as primary, secondary, etc. Additional Information (contact SRA)Additional Information (contact SRA) Human Subjects: IRB approval not Human Subjects: IRB approval not

needed for review needed for review

Study Section Study Section Perspective:Perspective:Pre-MeetingPre-Meeting

Administrative IssuesAdministrative Issues AssignmentsAssignments

Chartered members (20 for 4 year Chartered members (20 for 4 year terms each; about 5 members terms each; about 5 members nominated by SRA each year; Chair nominated by SRA each year; Chair nominated by SRA, usually for 2 years)nominated by SRA, usually for 2 years)

Special reviewers (as needed)Special reviewers (as needed) Outside opinions/Mail ReviewersOutside opinions/Mail Reviewers

Confidentiality andConfidentiality andConflict of InterestConflict of Interest

Confidentiality: in perpetuityConfidentiality: in perpetuity Conflict of InterestConflict of Interest

Assessed by SRAAssessed by SRA Assessed by reviewersAssessed by reviewers

Member Conflicts: If a member is Member Conflicts: If a member is on your application, it will be on your application, it will be reviewed by a special emphasis reviewed by a special emphasis panel or another study section.panel or another study section.

Study Section Study Section Perspective:Perspective:

Meeting Procedure of Meeting Procedure of ReviewReview Initial levels of enthusiasm from Initial levels of enthusiasm from

assigned reviewersassigned reviewers Description and critique from first Description and critique from first

reviewer (including assessment of reviewer (including assessment of potential risks to human subjects and potential risks to human subjects and the inclusion of women, minorities, the inclusion of women, minorities, and children)and children)

Critiques from other reviewersCritiques from other reviewers Discussion by entire Study SectionDiscussion by entire Study Section Final levels of enthusiasmFinal levels of enthusiasm Vote conscienceVote conscience

Study Section Study Section Perspective:Perspective:

Meeting Procedure of Meeting Procedure of ReviewReview After scoringAfter scoring

Budget recommendations from reviewers, Budget recommendations from reviewers, discussed by entire study sectiondiscussed by entire study section

Discussion of Administrative issues (e.g., Discussion of Administrative issues (e.g., overlap)overlap)

Deferral and Site VisitsDeferral and Site Visits An application may be deferred if it is viewed An application may be deferred if it is viewed

as highly significant and if a small amount of as highly significant and if a small amount of additional information (obtained by the SRA) additional information (obtained by the SRA) would address concerns.would address concerns.

Site visits are conducted ONLY if there is no Site visits are conducted ONLY if there is no other way to acquire such needed information.other way to acquire such needed information.

ScoringScoring

Judgment by reviewers based on Judgment by reviewers based on assessment of scientific and technical assessment of scientific and technical meritmerit

Criteria not equally weightedCriteria not equally weighted Budget not a factorBudget not a factor Other administrative issues not factorsOther administrative issues not factors HS/Gender/Minority/Children HS/Gender/Minority/Children

inclusion may be factor (as related to inclusion may be factor (as related to aims)aims)

ScoringScoring

Streamlining (the “UN”score): about Streamlining (the “UN”score): about 50% of applications reviewed in a 50% of applications reviewed in a “normal” round“normal” round

Range: 1.0 - 5.0 (ideal median = 3.0)Range: 1.0 - 5.0 (ideal median = 3.0) Percentile Rank calculated using Percentile Rank calculated using

the current and previous two rounds the current and previous two rounds as a baseas a base

Exception: F-series: 1.0-5.0, NRException: F-series: 1.0-5.0, NR

Study Section Study Section Perspective:Perspective:Post-MeetingPost-Meeting Scores/codes/budgets verified by SRA and Scores/codes/budgets verified by SRA and

entered in NIH computer systementered in NIH computer system Summary Statements prepared by SRASummary Statements prepared by SRA SRA advises investigators & discusses reviewSRA advises investigators & discusses review SRA represents the assessments of scientific SRA represents the assessments of scientific

and technical merit by Study Section at the and technical merit by Study Section at the National Advisory Councils of InstitutesNational Advisory Councils of Institutes

Note: We are receiving the next round’s Note: We are receiving the next round’s applications while preparing summaries for applications while preparing summaries for the current round.the current round.

Reorganization of Reorganization of Epidemiology Study Epidemiology Study

Sections in October 2003Sections in October 2003 1965-2003: Epidemiology and Disease 1965-2003: Epidemiology and Disease Control-1 (EDC-1) Study Section and EDC-2Control-1 (EDC-1) Study Section and EDC-2

2001: Added EDC-32001: Added EDC-3 October 2003 reorganization:October 2003 reorganization: EDC-1 EDC-1 Epidemiology of Chronic Diseases Epidemiology of Chronic Diseases

(ECD) Study Section (ECD) Study Section EDC-2 EDC-2 Epidemiology of Cancer (EPIC) Epidemiology of Cancer (EPIC)

Study Study Section Section EDC-3 EDC-3 Epidemiology of Clinical Epidemiology of Clinical

Disorders and Disorders and Aging (ECDA) Study Aging (ECDA) Study SectionSection

Inside the NIH Grant Review Inside the NIH Grant Review Process VideoProcess Video

http://www.csr.nih.gov/video/http://www.csr.nih.gov/video/video.aspvideo.asp

CSR has developed CSR has developed a video of a mock a video of a mock study section study section meeting to show meeting to show how NIH grant how NIH grant applications are applications are reviewed. reviewed.

CSR Web Site: CSR Web Site: http://www.csr.nih.govhttp://www.csr.nih.gov

News and EventsNews and Events

Resources for Resources for ApplicantsApplicants

Study Section Study Section InformationInformation

Employment Employment OpportunitiesOpportunities

Contact Contact InformationInformation

Information on the World Information on the World Wide WebWide Web

Selected Sites of InterestSelected Sites of Interest National Institutes of Health (http://www.nih.gov)National Institutes of Health (http://www.nih.gov) Office of Extramural Research Office of Extramural Research

(http://www.nih.gov/grants/oer.htm)(http://www.nih.gov/grants/oer.htm) Grants Policy Grants Policy

(http://www.nih.gov/grants/policy/policy.htm)(http://www.nih.gov/grants/policy/policy.htm)

Center for Scientific Review (http://www.csr.nih.gov)Center for Scientific Review (http://www.csr.nih.gov) Referral and Review (http://www.csr.nih.gov/refrev.htm)Referral and Review (http://www.csr.nih.gov/refrev.htm) CSR Study Section Rosters CSR Study Section Rosters

(http://www.csr.nih.gov/committees/rosterindex.asp)(http://www.csr.nih.gov/committees/rosterindex.asp) Review Group Meeting Dates Review Group Meeting Dates

(http://www. (http://www. csr.nih.gov/committees/meetings/ssmeet1.asp)csr.nih.gov/committees/meetings/ssmeet1.asp)

CSR Reorganization News CSR Reorganization News (http://www.csr.nih.gov/review/reorgact.asp)(http://www.csr.nih.gov/review/reorgact.asp)