Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov

Best Practices: Leveraging Existing Data Resources to Evaluate NIH Research Training Programs

American Evaluation Association October 18, 2013

Jennifer Sutton, MS, Deepshikha RoyChowdhury, PhD, Cassandra Spears, Katrina Pearson, and Robin Wagner, PhD

Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov

Overview

Background

Research Questions

Data and Methodology

Results

Conclusions

2

Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov

Background

3

In evaluating its research training programs, NIH regularly utilizes information from existing national resources

This presentation will cover: The methods used to match NIH and external databases The broader range of evaluation questions that can be

answered as a result How the evaluation results are being used to inform

NIH policies and planning

Data provided by the Division of Statistical Analysis & Reporting (DSAR)/OPAC/OER Contact: OERStats@mail.nih.gov

Research Questions

What is NIH’s role in doctoral education nationally?

How are NIH investments in research training allocated across scientific fields? Has that distribution changed over time?

How do NIH-supported predoctoral trainees and fellows compare to other PhD recipients in related fields in terms of time to degree and completion rates? Diversity?

What are the post-PhD plans and research activities of NIH-supported trainees and fellows?

4

Data Sources

5

IMPAC II: NIH’s administrative database, which includes information on research grants, contracts, and research training and fellowship awards

Doctorate Records File (DRF): The consolidated results of the Survey of Earned Doctorates, an annual census of all individuals receiving US research doctorates since 1957

The survey is coordinated by the National Science Foundation (NSF) and co-sponsored by the NIH and other federal agencies

NIH receives a copy of the DRF under a licensing agreement with the NSF

Methodology

NIH receives an updated copy of the DRF every fall and matches it to its IMPAC II database The latest version of the DRF includes US PhD recipients

through June of 2011

For individuals listed in NIH administrative files as predoctoral trainees or fellows, appearing in the DRF: Confirms whether they completed the PhD Provides information on their doctoral training, such as

field of study, time to degree, and post-PhD plans

6

Data Matching Process

7

DRF File is received and uploaded in

IMPACII Database

DRF File is received and uploaded in

IMPACII Database

Matching Algorithm Executed

Matching Algorithm Executed

Validity of the data is confirmed by running manual queries. Existing auxiliary mapping tables are updated, if needed.Validity of the data is confirmed by running manual queries. Existing auxiliary mapping tables are updated, if needed.

DRF and NIH data in IMPACII are uploaded to Oracle staging tables and matched on basis of main matching attributes such as last four digit of the SSN, date of birth and name.

DRF and NIH data in IMPACII are uploaded to Oracle staging tables and matched on basis of main matching attributes such as last four digit of the SSN, date of birth and name.

“Non-matches” are identified and filtered out.“Non-matches” are identified and filtered out.

Various permutations of the matching attributes such as SSN+DOB, SSN+last name, DOB+last name etc. are used to calculate weighted scores. Cases of flip-flopped names are accounted for as well.

Various permutations of the matching attributes such as SSN+DOB, SSN+last name, DOB+last name etc. are used to calculate weighted scores. Cases of flip-flopped names are accounted for as well.

Most common names in DRF and NIH data sets are identified and the weighted score for those is deducted in the matching table. Most common names in DRF and NIH data sets are identified and the weighted score for those is deducted in the matching table.

Matches with weights below minimum threshold are removed and matched table is uploaded in IMPACII.Matches with weights below minimum threshold are removed and matched table is uploaded in IMPACII.

Individuals with matching fields such as state address information, expertise, PhD institution, foreign citizenship etc. are identified and weighted score is increased each time a match is found for a field.

Individuals with matching fields such as state address information, expertise, PhD institution, foreign citizenship etc. are identified and weighted score is increased each time a match is found for a field.

Weighted scores are rescaled. Previously identified “bad matches” are removed from the matching table.Weighted scores are rescaled. Previously identified “bad matches” are removed from the matching table.

Step 1

Step 2

Step 3 Inspection and Testing

Inspection and Testing

Manual Inspection and testing of the matches are performed before finalizing the table.Manual Inspection and testing of the matches are performed before finalizing the table.

Results

8

Broad Fields of Study of NIH-Supported Trainees and Fellows, 2011

9

Fields of Study

NIH-Supported

PhDs

Total Number of PhDs

% Supported by NIH

Biological/Biomedical Sciences 2,174 8,072 26.9%

Health Sciences 216 2,124 10.2%

Psychology 179 3,297 5.4%

Engineering 162 6,710 2.4%

Physical Sciences 141 5,655 2.5%

Other 99 6,931 1.4%

Total 2,971 32,789 9.1%

Doctoral Fields With the Highest Concentrationof NIH-Supported Trainees and Fellows, 2011

10

=

Arrows indicate fields where the percentage of NIH-supported PhDs has increased >10% since 2006

Trends in PhD Fields of Study ofNIH-Supported Trainees and Fellows

11

Length of Doctoral Study

Time to Degree

Median Time to Degree for NIH-Supported Trainees in 2011 6.3 years

Median Time to Degree in the Life Sciences in 2011 6.9 years

Median Time to Degree in the Social Sciences in 2011 7.7 years

Age at PhD

NIH-Supported Trainees and Fellows Receiving PhDs in 2011 30.0

All Life Science PhDs in 2011 31.3

All Social Science PhDs in 2011 32.5

12

Completion Rates

Data from the Council of Graduate Schools’ Doctorate Completion Project provides context for assessing the completion rates of NIH trainees

Completion Rates

NIH-Supported Trainees Completing PhDs Within 10 Years 80.1%

10-Year Completion Rate for Doctoral Students in the Life Sciences 62.9%

10-Year Completion Rate for Doctoral Students in the Social Sciences 55.9%

13

Trends in NIH-Supported Traineesand Fellows Receiving PhDs, by Sex

14

Trends in NIH-Supported Trainees and Fellows Receiving PhDs, by Race and Ethnicity

15

The percentage of NIH trainees and fellows from groups underrepresented in science (i.e., African Americans, Hispanics, Native Americans/ Alaska Natives) increased from 5% in 1986 to 14% in 2011.

The percentage of NIH trainees and fellows from groups underrepresented in science (i.e., African Americans, Hispanics, Native Americans/ Alaska Natives) increased from 5% in 1986 to 14% in 2011.

Post-PhD Plans of NIH-SupportedTrainees and Fellows, 2011

16

Year of PhD

Perc

enta

ge o

f PhD

Rec

ipie

nts

Subsequent NIH Grant Activity

17

Within 15 years of their degrees:

Applied for NIH Grants

ReceivedNIH Grants

NIH-Supported Trainees and Fellows 36.7% 23.6%

Other PhDs from the Same Fields and Institutions 12.8% 7.0%

Other PhDs from the Same Fields and Institutions Without NIH Training Grants 5.9% 2.6%

Conclusions

18

Matching NIH records with data from the DRF and other sources enhances NIH’s capacity to evaluate its research training and fellowship programs, by providing:

–Access to information not available in NIH databases

–Context and comparison groups

–An understanding of how NIH programs contribute to US doctoral education overall

Conclusions, cont.

19

Our analyses indicate:

– NIH-supported trainees and fellows are more likely to complete the PhD than other graduate students and do so in less time

– The percentages of women and underrepresented minorities among the NIH-supported trainees and fellows receiving PhDs have steadily increased over time

– NIH-supported trainees and fellows are more likely to remain active in research careers and be successful in obtaining research funding from the NIH than other PhD recipients However, there are signs of uncertainty in the job market that

bear watching

Contact Information

Office of Extramural Programs

Jennifer Sutton – suttonj@mail.nih.gov

Office of Planning, Analysis and Communications

Deepshikha RoyChowdhury – roychowdhuryd@mail.nih.gov

Cassandra Spears – spearsc@mail.nih.gov

Katrina Pearson – pearsonk@mail.nih.gov

Robin Wagner – wagnerr2@mail.nih.gov

20

Questions?

Thank you!

21