Evaluating Postgraduate Public Health and BiomedicalTraining Program Outcomes:Lost Opportunities and Renewed Interest

Jessica Faupel-Badger & David E. Nelson &

Stephen Marcus & Aisha Kudura & Elaine Nghiem

Published online: 8 December 2012# Springer Science+Business Media New York (outside the USA) 2012

Abstract To identify recent studies in the scientific litera-ture that evaluated structured postgraduate public health andbiomedical training programs and reported career outcomesamong individual trainees, a comprehensive search of sev-eral databases was conducted to identify published studiesin English between January 1995 and January 2012. Studiesof interest included those that evaluated career outcomes fortrainees completing full-time public health or biomedicaltraining programs of at least 12 months duration, withstructured training offered on-site. Of the over 600 articlesidentified, only 13 met the inclusion criteria. Six studiesevaluated US federal agency programs and six were ofuniversity-based programs. Seven programs were solely orpredominantly of physicians, with only one consistingmainly of PhDs. Most studies used a cohort or cross-sectional design. The studies were mainly descriptive, withonly four containing statistical data. Type of employment

was the most common outcome measure (n012) and num-ber of scientific publications (n06) was second. The lack ofoutcomes evaluation data from postgraduate public healthand biomedical training programs in the published literatureis a lost opportunity for understanding the career paths oftrainees and the potential impact of training programs. Sug-gestions for increasing interest in conducting and reportingevaluation studies of these structured postgraduate trainingprograms are provided.

Keywords Career . Evaluation . Fellowship . Postdoctoral

Introduction

One of the major roles of educational and research institu-tions is training the next generation of scientists and practi-tioners [1–7]. There are an estimated 37,000–68,000persons with PhDs and 5,000 physicians in postdoctoraltraining positions in the USA [8]. Given the size and scopeof the postgraduate public health and biomedical workforcetraining enterprise, one might expect reports of evaluatingcareer outcomes of individuals who complete these trainingprograms to be plentiful in the published literature. Onoccasion, institutions and foundations that support individ-ual fellowships, such as career development awards, con-duct and publish career outcome evaluations of awardees[9–11]. These evaluations include awardees across multipleinstitutions and the career development training, being indi-vidualized, is highly variable across awards. This is incontrast to evaluation of structured, postgraduate trainingprograms offered on-site to a cadre of trainees. Evaluationsof these structured programs in the literature are rare andpredominantly limited to medical students and physicians[12–15].

J. Faupel-Badger (*) :D. E. Nelson :A. Kudura : E. NghiemCancer Prevention Fellowship Program,National Cancer Institute,National Institutes of Health,6120 Executive Blvd, Suite 150E,Bethesda, MD 20892-7105, USAe-mail: badgerje@mail.nih.gov

S. MarcusNational Institute of General Medical Sciences,National Institutes of Health, Bethesda, MD, USA

Present Address:A. KuduraPublic Health & Preventive Medicine,Oregon Health & Science University, Portland, OR, USA

Present Address:E. NghiemUniversity of Maryland, College Park, MD, USA

J Canc Educ (2013) 28:18–26DOI 10.1007/s13187-012-0437-x

As an example, in a comprehensive literature review ofstudies from 1966 to 2006 reporting relationships betweencareer development (broadly defined) and mentoring pro-grams in medicine, Sambunjak and colleagues found only39 evaluation studies, 34 (87 %) of which used a cross-sectional design [15]. A review conducted by Buddeberg-Fischer of the literature from 1966 to 2002 identified only16 evaluation studies of mentoring programs in medicine,and found most lacked structure (i.e., a defined curriculum)and included no defined short-term or long-term outcomemeasures [12]. In addition, most studies included in thesetwo reviews had weak evaluation research designs [16],including small samples sizes, no comparison populations,and use of self-reported data collected through one-timecross-sectional surveys of program graduates with the em-phasis largely on measures of satisfaction with the programrather objective career outcomes [12, 14, 15]. Of particularconcern with the cross-sectional design is that this capturesonly one point in time with no ability to follow accomplish-ments over a longer period.

The number and quality of studies focused on postgrad-uate training programs in the public health and biomedicaldisciplines represent a major lost opportunity. This lack ofeasily accessible evaluation data has become increasinglyvisible with national efforts to evaluate the biomedicalworkforce and postdoctoral experience [8, 17, 18]. A recentreport on the future of the biomedical workforce from anAdvisory Committee to the National Institutes of Health(NIH) Director highlighted the lack of data as one ofthe impediments to projecting career paths for postdoc-toral scholars [8]. Partially in response to this, membersof the committee issued a recommendation to evaluatetraining programs supported by the NIH and reportthese results so prospective trainees could have accessto outcomes data.

With the broad and evolving changes in biomedical andpublic health research and practice, a constrained fundingenvironment, and growing calls for accountability [8, 19],this presents a good opportunity to examine the recentliterature focused specifically on evaluating individual-level career outcomes of participants in postgraduate publichealth and biomedical training programs with structuredtraining provided on-site at the trainee’s home institutionand to provide a framework for future evaluations of theseprograms. Herein, the results from a comprehensive litera-ture review of postgraduate public health and biomedicaltraining programs outcome evaluations published from 1995to 2012 are presented. The focus of this literature reviewwas studies of full-time structured programs conductedwithin a given institution and reporting career outcomes oftrainees. A secondary purpose of this report is to providespecific recommendations on how public health and bio-medical training programs can take steps to improve or

expand their evaluation efforts and the benefits likely toaccrue from doing so.

Methods

The focus of this literature review was outcomes studies offull-time postgraduate structured public health and biomed-ical training programs (e.g., postgraduate programs in epi-demiology, biostatistics, basic/laboratory science related tohealth, medical fellowships, nursing, health policy, or socialand behavioral science related to health) with structuredtraining provided on-site at the home institution. Onlystudies with career-oriented outcome data were considered,e.g., scientific publications, type of employment, careeradvancement, or receipt of grants. To help differentiatecareer-oriented from short-term training programs, partici-pants had to be full-time postgraduate trainees (i.e.,performing at least 40 h of work per week) in a structuredprogram of at least 12 months’ duration. Studies by institu-tions or foundations awarding individual fellowships wereexcluded because trainees work in a multitude of academicinstitutions with highly variable program structures andtraining experiences and often with no specific or definedcurriculum across awardees. Studies based solely on gradu-ate students, such as those of medical students, were exclud-ed. In addition, evaluations of skills-based or similartraining programs with outcomes, e.g., communication orsurgical techniques, were excluded because they wereshort term in nature without career outcomes.

The authors worked closely with an experienced biomedi-cal librarian/informationist at the NIH Library to select searchterms and databases. Because there is no standard searchstrategy to identify published studies about postgraduate pub-lic health and biomedical training programs, the followingsearch terms were utilized to search MEDLINE databases:

Search Strategy 1—(“Evaluation Studies as Topic”[Mesh]OR “Evaluation Studies”[Publication Type] OR “ProgramEvaluation”[Mesh] OR “Data Collection”[Mesh]) AND(“Fellowships and Scholarships”[Majr] OR internship,nonmedical[majr] OR fellowship*[ti] OR mentor*[ti] ORpreceptorship*[ti])ORSearch Strategy 2—(“Career Mobility”[Mesh] OR “CareerChoice”[Mesh] OR “Job Satisfaction”[Mesh] OR “Profes-sional Competence”[Mesh]) AND (“Fellowships andScholarships”[Majr] OR internship, nonmedical[majr]OR fellowship*[ti] ORmentor*[ti] OR preceptorship*[ti]).

Similar strategies were used to identify English-languagearticles describing evaluations of biomedical and publichealth-related training programs published between January1, 1995 and January 31, 2012 from the following resources:

J Canc Educ (2013) 28:18–26 19

Scopus, Web of Science, ERIC (Educational ResourceInformation Center) and CINAHL (Cumulative Index toNursing and Allied Health Literature) databases, as well asthe LexisNexis business and news databases. Using thesearch strategies above, 598 articles of interest were identified.

Two junior members of the research team (A.K. andE.N.) received extensive training on the inclusion and ex-clusion criteria (Table 1) for this literature review from thesenior authors (J.F-B. and D.N.), which included reviewinga test set of articles. Each of the junior members independentlyreviewed and coded half of the initial 598 articles. They thenswitched lists to determine if there was concurrence on articlesto include. Articles identified for potential inclusion wereprovided to the senior members of the research team, whoreviewed them independently of each other. Any disagree-ments about which articles to include were quickly resolvedbased on discussions between the senior authors. A total ofseven articles met the above inclusion criteria. The referencelists from each of these articles was then systematicallyreviewed, which successfully identified six additional articles,for an overall total of 13 articles.

The senior authors then systematically abstracted infor-mation from each article, which included descriptions aboutthe training program, demographic data on participants,years evaluated, study design, outcome measure(s), datasource(s), presence of a comparison population, and inclusionof statistical testing results. Sources of evaluation data wereclassified as survey or archival; archival sources includedprogram or other organization-specific databases (e.g., spread-sheets maintained by staff), scientific publication databases(e.g., PubMed), grants databases, or curriculum vita.

Results

Brief descriptions of the characteristics of each trainingprogram and their participants are provided in Table 2. Ten

studies were of programs in the USA [13, 20–28], and mostwere at least 2 years in length [13, 21, 23, 25–28, 30, 31].Six were evaluations of federal government-affiliated pro-grams [13, 21, 27, 29–31], six were university-based [20,23–26, 28], and one had another affiliation [22]. The timeperiod for examining career outcomes ranged from 1 year inseveral studies to up to 50 years.

Participants in seven studies were solely or predominatelyphysicians [20, 23–27, 29], one consisted predominantly ofpersons with PhDs [13], with the rest having persons from avariety of disciplines or with disciplines not specified. Out-come data were available from a high percentage of programparticipants (range, 65–100 %; data not shown in tables).Five studies [13, 21–23, 27] included current program par-ticipants in their data analyses. In terms of demographics,participants’ sex was mentioned in ten studies (range, 14 %to 72 % female), with mean age (range, late twenties to latethirties) or race/ethnicity mentioned in six and five studies,respectively.

A summary of evaluation methods and results for each ofthe 13 studies is described in Table 3. Cohort or cross-sectional designs were used in ten studies [13, 21–29, 31],one study used a case–control design [22], and study designinformation was missing for the other two. Four studies usedsurveys as their sole source of data [22, 24, 26, 29], fourused archival sources only [13, 21, 25, 28], one used asurvey and archival sources [23], and four did not describethe source(s) of data. Only the study by Jones et al.contained a comparison population [22]. It is worth notingoutcome data was predominantly descriptive in nature only,as only four studies contained any statistical testing resultssuch as p values [13, 22, 23, 26].

Type of employment was the most common outcomemeasure utilized, with 12 studies having at least one mea-sure of this type. There was wide variability, however, in thetype of employment measure utilized. Studies reportingemployment varied on either reporting current employment

Table 1 Exclusion and inclusion criteria for articles in postgraduate public health and biomedical training program literature review

Article EXCLUDED if met ONE of the following conditions: Article INCLUDED if contained ALL of the following conditions:

Population Majority of population were faculty members, graduate andmedical school students, participants whose appointment is nota full-time (i.e., 40+ hours/week) training position, participantsin part-time training programs

Postgraduate trainees in full-time (i.e., 40+ hours/week) researchor health practice training programs

Duration Programs completed in <12 months Programs requiring 12+ months for completion

Location Trainees in multiple sites (e.g., grantees located at variousinstitutions with variable training program structures andexperiences)

Structured training program provided on-site to a cadre oftrainees

Outcome Only satisfaction measures without also including careeroutcomes or skills-based outcomes (e.g., surgical techniquesor communication)

Contained at least one career outcome, such as currentemployment information or information on employmentimmediately following participation in training program,publication counts, grant funding, etc.

20 J Canc Educ (2013) 28:18–26

or employment immediately after completion of the pro-gram. Within these studies, some highlighted the employ-ment sector of program graduates, e.g., government,university, another training program, or private medicalpractice. Others mentioned specific type of job, rank, orarea of specialization, or job rank, e.g., epidemiologist,associate professor, or health policy involvement. Twostudies from outside the USA reported data on countryof employment.

The second most common outcome measure was scientificpublications of program graduates, which was mentioned insix studies. Publications were categorized in different ways,such as total or average number of first-authored publications,peer-reviewed journal articles, book chapters, etc. Only twostudies reported scientific conference presentations as an out-come measure. Among studies of university-based programs

only, a few additional measures were used, which includedreceipt of grant research funding (three studies), time spentconducting research (two studies), and teaching or men-toring responsibilities (two studies).

Discussion

From the outset, this literature review of evaluation studiesof postgraduate public health and biomedical training pro-grams published in the past 17 years was designed to beexploratory in nature since the breadth or depth of researchin this area was unclear, and there is no standard terminol-ogy for search terms (i.e., it is possible additional or morerefined terms may have uncovered more publications). Nev-ertheless, despite an attempt to identify evaluations of

Table 2 Characteristics of postgraduate health training programs

Source Program description and objectives Program duration Study time frame Participants

Betts, 1998 [29] Foreign Epidemiology Training Program in Mexico,Thailand, Philippines, Spain, and Uganda that providesapplied epidemiology training for health professionals

2 years 1980–1998a 136 postgraduate trainees(95 % physicians)

Simon, 1999 [26] Research-intensive fellowship at Harvard Universitydesigned to increase medical school faculty trainedin general internal medicine

2 years 1979–1997 103 physicians

Thacker, 2001 [27] Epidemic Intelligence Service (EIS) fellowship trainingprogram in applied epidemiology for healthprofessionals through the U.S. Centers for DiseaseControl and Prevention

2 years 1951–2000 2,338 postgraduate trainees(78 % physicians)

Waterbor, 2002 [28] Fellowship in cancer prevention and control for pre-and postdoctoral students at the University ofAlabama–Birmingham School of Public Health

2–3 years 1988–2001 39 pre- and postdoctoralrainees (80 %predoctoral trainees)

Dores, 2006 [13] Postdoctoral fellowship program in cancer preventionresearch and leadership at the U.S. National CancerInstitute, National Institutes of Health

2–3 years 1987–1997 64 postdoctoral trainees(71 % PhDs)

Gordon, 2007 [21] Postdoctoral clinical research fellowship program at theU.S. National Institute for Dental and CraniofacialResearch, National Institutes of Health

1–3 years 2000–2005 11 postdoctoral trainees(82 % dentists)

Jones, 2007 [22] Postgraduate fellowship program in health policy inWashington DC area

1 year 1987–2005 18 postgraduate traineeswith varied backgrounds

Landrigan, 2007 [23] Fellowship at five U.S. research institutions to trainpediatricians to become physician-scientists andacademic leaders in pediatric environmental health

3 years 2001–2007 13 physicians

Lopez, 2008 [30] Foreign Epidemiology Training Program in CostaRica, El Salvador, Guatemala, Honduras, Nicaragua,Panama, and the Dominican Republic that providesapplied epidemiology training for health professionals

2 years 2000–2007a 58 postgraduate trainees(disciplines NA)

Cronholm, 2009 [20] Fellowship for family physicians at University ofPennsylvania designed to enhance research trainingand academic career development

2–3 years 1997–2007 15 physicians

Rivera, 2010 [24] Fellowship in clinical nutrition education for physiciansat the Cleveland Clinic

1 year 1994–NA 14 physicians

Rose, 2011 [25] Fellowship at the Mayo Clinic (MN) designed to trainphysicians in anesthesiology critical care medicine

2–3 years 2000–2010 28 physicians

Matovu, 2011 [31] Postgraduate fellowship training program at theMakerere University School of Public Health (Uganda)designed to build HIV/AIDS leadership andmanagement capacity

2 years 2002–2008 54 postgraduate traineeswith varied backgrounds

NA not availablea Number of years varied across countries

J Canc Educ (2013) 28:18–26 21

postgraduate training programs with outcome measuresfrom a wide variety of health fields, there is a dearth ofpublished, easily accessible evaluation research in this area,particularly for PhD postdoctoral programs. The small num-ber of studies we found makes comparisons across reportsdifficult. For example, more data is needed to compare andcontrast outcomes by discipline, size of program, or dura-tion of training. All of these factors will likely influencecareer outcomes.

The studies reported here were identified through a com-prehensive literature review and, therefore, career outcomesdata reported by training programs to funding agencies werenot included. Some of these reports may not be publiclyavailable but could be a plentiful source of data. Two mech-anisms supporting a large number of trainees with NIHfunding are the T32 and R25 training grants. Principalinvestigators on both of these grants are required to report10 years of data on trainees including, at a minimum, currentposition and source of support. This reporting requirementsuggests that there is much more data available than can befound through traditional literature searches.

The studies we identified herein had at least one substantialdesign flaw, and it is likely that these internal evaluationswould be susceptible to these same issues. The most serious

concern was the lack of a comparison population to assesswhether outcomes were associated with participation in atraining program [16]. Evaluating outcomes data capturedonly after the program but not before and including outcomesof current trainees in the “alumni” group were also commonproblems. In addition, statistical analyses were largely absent.

There are multiple explanations for the number and qual-ity of the studies found during this literature search. Limitedresources are the most likely explanation for why there areso few studies of career-oriented outcome studies of post-graduate trainees [8, 32–34]. Newer social media technolo-gies may lower the cost of finding and contacting pasttrainees, but the scope and complexity of the evaluationdesign will be proportional to the level of investment inthese activities. In addition, for training programs withoutstable funding sources, there are no assurances funding willbe available for a long-term evaluation effort. This uncer-tainty can result in evaluation being a low priority. This alsolimits the types of questions (especially in regard to careeroutcomes) that can be asked if data need to be obtained,analyzed, and reported in a limited funding cycle.

If evaluation is a lower priority, this could result intraining program directors, early in the development of atraining program, failing to consider or collect data that

Table 3 Postgraduate training program evaluation study parameters

Source Study design Data source Comparisongroup

Statisticaltesting

Outcome measure(s)

Betts, 1998 [29] Cross-sectional Survey None No Employment before and after completingfellowship, scientific publications,conference presentations

Simon, 1999 [26] Cross-sectional Survey None Yes Current employment (academic appointment),academic rank, teaching and mentoring, timespent in research

Thacker, 2001 [27] Cohort NA None No Current employment

Waterbor, 2002 [28] Cohort Archival None No Current employment, scientific publications,conference presentations

Dores, 2006 [13] Cohort Archival None Yes Scientific publications

Gordon, 2007 [21] Cohort Archival None No Scientific publications, current employment,involvement in research, receipt of NIHresearch funding

Jones, 2007 [22] Case–control Survey 10 unselectedapplicants

Yes Employment upon completing fellowship,involvement in health policy

Landrigan, 2007 [23] Cross-sectional Archival and Survey None Yes Current employment, scientific publications,receipt of research grant funding

Lopez, 2008 [30] NA NA None No Current employment

Cronholm, 2009 [20] NA NA None No Current employment, scientific publications,receipt of research grant funding

Rivera, 2010 [24] Cross-sectional Survey None No Current employment, subsequent fellowshiptraining, clinical nutrition practice, income

Rose, 2011 [25] Cohort Archival None No Current employment

Matovu, 2011 [31] Cohort NA None No Current employment, country of employment

NA not available

22 J Canc Educ (2013) 28:18–26

would be useful for subsequent evaluation purposes. If long-term follow-up data are not available, the evaluation islimited to using one-time surveys (cross-sectional designs)of program alumni, which was the most common methodseen in this literature search. Other issues included that moststudies had small sample sizes, which limited statisticalanalyses and generalizability. Finally, training program lead-ers may be unsupportive of evaluation because of fear,believing that evaluation findings could be interpreted asshowing their programs are “unsuccessful” and placing theirprograms (or even their own jobs) at risk for funding reduc-tions or elimination.

These barriers to conducting and reporting evaluation oflong-term postgraduate training programs need to be over-come since greater emphasis is being placed on collectingand reporting outcomes from postgraduate training programparticipants. Evaluation data will become even more impor-tant in justifying new or sustained funding of training pro-grams. Several ongoing activities at the national level [8, 17]have renewed the call for increased evaluation of theseprograms. In the absence of substantial progress in devel-oping comprehensive outcomes evaluations of postgraduatetraining programs, both national and local efforts at improv-ing postgraduate training may suffer. The national commit-tees making workforce recommendations will continue tooperate on limited data and without a baseline from which toassess progress and future needs. Individual programs willrun the risk of using training methods or activities that arenot effective, eliminating or failing to include methods oractivities that are, and not having adequate data available todemonstrate the impact of the training program and justifycontinued resources.

With the current review of the literature demonstratingmost training program evaluation study designs are relative-ly weak at the same time there is increased interest inevaluation of these programs, a few suggestions for improv-ing evaluation efforts are provided here. First and foremost,including more information on demographics (e.g., age,gender, degree) of trainees in evaluation reports would en-able comparisons across training programs. At a minimum,reports should also include the type of evaluation studydesign and the source of the data (e.g., archival records,surveys, CV review) used for the evaluation. The specificevaluation design will depend on the goals and resources ofeach training program; however, several principles of eval-uation are presented here. Table 4 outlines a few evaluationdesign elements to consider, options for each element, andpotential limitations.

For postgraduate training programs, the issues of com-parison populations and length of training complicate thestudy designs that can be selected. One conundrum is indi-viduals are not randomized to training programs but ratherselect training programs based on their preferences and are,

in turn, selected by program directors to join the trainingprogram [35]. In the literature review described here, onlyone study had a comparison group. It is essential for validattribution and causal inference regarding the impact of thetraining program on subsequent career outcomes to haveappropriate comparison groups [16]. “Compared to what?”is the question that needs to be at all times upmost in theevaluator’s mind.

Identifying and including comparison groups increasesthe expense of the evaluation. To decrease costs, two ormore programs could combine resources and serve as eachother’s comparison population, if the programs have similarstructure and trainee populations. It may be difficult to findwilling participants to serve as a comparison group if thereare concerns the evaluation results will frame one programas inferior to the other. Options for different types of com-parison groups are explored in Table 4. If no comparisongroup is utilized, one needs to be mindful of the effect oftime on certain outcomes and how this may bias the results[35]. For example, in research-intensive environments, pub-lication number should increase with duration of career andmay not be directly attributable to participation in the train-ing program.

Meaningful and appropriate (including time-appropriate)outcome measures that are valid, reliable, relevant to thegoals and objectives of the program, and reflect the natureand timing (e.g., what outcomes are to be expected and inwhat timeframe) of the underlying processes and phenome-na need to be selected. It is common mistake to chooseoutcomes that could not have occurred in the timeframe ofthe evaluation. Table 4 presents several choices for outcomemeasures. Ideally, a longer-term outcomes evaluation woulduse a mixed-methods approach to obtain qualitative andquantitative data. This could include not only quantifyingpublications, grants awarded, or other metrics but also con-ducting interviews with past trainees to understand what rolethe trainees think the training program had in their subse-quent career trajectory.

Hand in hand with deciding on outcome measures, thedata sources must be determined (Table 4). Often the archi-val data sources available for postgraduate training programevaluations contain outcomes most relevant to academiccareer paths (e.g., grants, publications, patents). Utilizingexisting data sources also limits the questions that can beasked during an evaluation to only those for which dataalready exists. Developing new data collection instrumentsprovides the opportunity to select the most important eval-uation questions and then capture the data necessary toanswer those questions.

In Fig. 1, a diagram containing elements that couldbe universally applied to evaluating postgraduate publichealth and biomedical training programs is presented. Thisevaluation model is “scale-able” depending on funding and

J Canc Educ (2013) 28:18–26 23

timeframe. One could begin with evaluating only individualcomponents of the training program and/or start with only asubset of alumni (e.g., completed program within the last 5years). The first column contains examples of questions thatwould need to be answered to determine if the trainingprogram, both as a whole and the individual componentsof the training program, are meeting the overall goals of theprogram. The second column contains examples of compo-nents of structured training programs. It is important to notetraining programs may include only a few of these compo-nents, and these components could focus on specific areas ofresearch or on professional development (e.g., grant-writingskills, leadership/management skills). The third columnindicates types of career outcomes that could be assessedduring an evaluation. These outcomes focus on both indi-vidual and collaborative contributions and recognition.

Implicit within this framework is the need to capture asmany career outcomes as possible and not focus solely onmeasures aligned with academic careers. Most postdoctoralfellows will not go on to academic positions [36, 37] where

publishing and grantsmanship are the primary measures ofproductivity. For postdoctoral fellows who choose not topursue an academic career, there are limited data on theprofessional fields where postdoctoral fellows find employ-ment and their reasons for pursuing these options. Otheroutcomes that could be obtained and would apply to alumniacross different career paths include assessing career trajec-tory (e.g., promotions and timing between promotions) andpeer recognition. Peer recognition could include appoint-ment to editorial boards, professional society and/or institu-tional committee service, serving on advisory panels, ornomination for or receipt of awards. This captures a levelof connectedness and expertise. For those alumni in aca-demic settings, this will likely track closely with publica-tions and academic rank. It may be less correlated for thosealumni in other career paths.

For future outcomes evaluations to provide more insightinto the career paths selected by postgraduate trainees, thesestudies should include methodology for collecting data di-rectly from past trainees rather than relying on archival data

Table 4 Considerations for training program outcomes evaluation design

Evaluation elements Approaches Feasibility Additional notes

1. Comparisonpopulations

• Unsuccessful applicants Requires competitive application processand detailed records to find unsuccessfulapplicants

Perhaps most similar to alumni since appliedfor same opportunity

• Trainees in similar trainingprogram/environment

Need to collaborate with directors of trainingprogram that will be the comparison group,requires both programs kept detailed records

Training program directors may fear oneprogram will appear less desirable and willbe less likely to collaborate

• No comparison population Difficult to discern whether outcomes arerelated to training environment; could usepre–post training comparisons

In pre–post comparison, need to account foreffects of time on career outputs andtrajectory

2. Outcomemeasures

• Satisfaction/processmeasures

Can be collected immediately using shortsurveys; quickly implemented

May be important for program to knowsatisfaction but not a career outcome

• Quantitative outcomes fromexisting data

Limits evaluation questions that can beasked and may not be applicable to allalumni of training program

Most relevant to academic career paths(e.g., publications, grants, patents);existing data caveats

• Qualitative and quantitativeoutcomes for diverse careerpaths

Data collection more complicated but caninclude measures of career trajectory, peerrecognition, professional connectedness

Evaluation measures will be applicable to allalumni; diversity of career paths growing

3. Data sources Existing data: Depends on quality and completeness ofrecords

Evaluation designed around questions thatcan be answered with existing data

• Archival data Informationwill be available on only a limitednumber of outcomes (e.g., publications)

No need to contact alumni directly; someoutcomes may be incomplete

New data collection: More resource intensive but can insure datais complete

Can select the most important questions andthen collect relevant data to answer them

• CV review Relying on alumni to periodically sendupdated CVs; no standard CV format

Can examine career outcomes such ascommittee service, current job title

• Surveys Can design survey to specifically addressevaluation questions; relying on alumni tocomplete

Survey can include open-ended questions

• Interviews In-depth information can be collected;questions can be directed to futureprogram planning

Information collected on limited number ofindividuals; subjective depending on whoparticipates

24 J Canc Educ (2013) 28:18–26

sources such as grant and publication databases. In fact,even among the studies included in this literature search,most of the 13 studies included surveys, requested a currentCV from participants, and/or another form of personal con-tact to obtain career outcomes information. Broadening theoutcomes explored to include measures that would be ap-plicable to a variety of career paths would provide a moreaccurate profile from which to assess trainee success acrossthe multitude of career options available to postgraduatetrainees in the public health and biomedical sciences.

In conclusion, this literature review demonstrated thelack of published outcomes evaluations of postgraduatepublic health and biomedical training programs, which isparticularly concerning given the number of individuals andinvestment in these training programs. The recommenda-tions for future evaluations are meant to serve as a startingpoint for developing a more sophisticated approach, whilebalancing enthusiasm for the ideal evaluation with the chal-lenges inherent in evaluating long-term postgraduate train-ing programs. A principal message to take from thisdiscussion is to be as rigorous as possible with the evalua-tion design, within the boundaries imposed or resourcesavailable, and to acknowledge the caveats of each decisionmade. In addition, many training programs have a longhistory. If evaluation was not considered at the initiation ofthe program, it is better to start now than not at all, keepingthe limitations of retrospective study design in mind. Thereis a great need to expand on the outcomes collected inevaluation studies so that these studies can be a true reflectionof career options and success of past trainees in the postgraduatepublic health and biomedical workforce. Finally, these results

need to be published to inform the design of future trainingprograms and increase the data available to those interested inunderstanding the postgraduate public health and biomedicalworkforce career trajectories.

Acknowledgments We thank Mary Ryan, biomedical librarian/informationist at NIH library, for her help with search terms andconducting literature searches. Aisha Kudura acknowledges summer2011 fellowship support from the National Cancer Institute’s Introductionto Cancer Research Careers and Elaine Nghiem acknowledges summer2011 fellowship support from the NIH Office of Research on Women’sHealth and Foundation for Advanced Education in the Sciences. We alsothank Drs. Julie Mason and Jonathan Wiest for critical reading of themanuscript and insightful comments.

References

1. Gibbons M (1998) Higher education relevance in the 21st century.The World Bank, Washington, DC

2. Goldberger, Marvin L., Brendan A. Maher, Pamela Ebert Flattau,National Research Council (U.S.). Committee for the Study ofResearch-Doctorate Programs in the United States, ConferenceBoard of the Associated Research Councils, and National ResearchCouncil (U.S.). Office of Scientific and Engineering Personnel.Studies and Surveys Unit (1995) Research-doctorate programs inthe United States: continuity and change. National Academy Press,Washington, D.C.

3. Guelich JM, Singer BH, Castro MC, Rosenberg LE (2002) Agender gap in the next generation of physician-scientists: medicalstudent interest and participation in research. J Investig Med 50(6):412–418

4. Hotez PJ (2008) Training the next generation of global healthscientists: a school of appropriate technology for global health.PLoS Negl Trop Dis 2(8):e279

How well is the training

program supporting career

development for diverse

outcomes?

How well are the structured

elements of the training

program contributing to the

overall goal of the program?

Is the program supporting

successful career transitions

through professional

development training?

Professional development activities within training program

Issues to consider for successof training program

Mentoring model

Didactic courses

Seminars

Workshops

Opportunities for

visibility of trainees

Connectedness of

alumni with each other

and current fellows

Career outcomes

Career choices

Career advancement

Peer recognition

Scientific productivity

Transdisciplinary

collaboration or team

science contributions

Fig. 1 Framework demonstrating examples of questions important to a wide variety of postgraduate training programs, specific components ofthese training programs, and subsequent evaluation of the impact of the training program on the career outcomes of alumni

J Canc Educ (2013) 28:18–26 25

5. Laredo P (2007) Revisiting the third mission of universities:toward a renewed categorization of university activities? HigherEducation Policy 20(4):441–456

6. Rip A (2004) Strategic research, post-modern universities, andresearch training. Higher Education Policy 17(2):153–166

7. Summers MF (2011) Training the next generation of proteinscientists. Protein Sci 29:1786–1801

8. Advisory Committee to the National Institutes of Health Director(2012) Biomedical research workforce working group report.National Institutes of Health, Bethesda

9. National Institutes of Health (2006) The career achievements ofnational research service award postdoctoral trainees and fellows:1975–2004.

10. Pion G, Ionescu-Pioggia M (2003) Bridging postdoctoral trainingand a faculty position: initial outcomes of the Burroughs WellcomeFund Career Awards in the Biomedical Sciences. Acad Med 78(2):177–186

11. National Institute of General Medical Sciences (1998) MSTPstudy: the careers and professional activities of graduates of theNIGMS medical scientist training program.

12. Buddeberg-Fischer B, Herta KD (2006) Formal mentoringprogrammes for medical students and doctors—a review ofthe Medline literature. Med Teach 28(3):248–257

13. Dores GM, Chang S, Berger VW, Perkins SN, Hursting SD, WeedDL (2006) Evaluating research training outcomes: experience fromthe cancer prevention fellowship program at the National CancerInstitute. Acad Med 81(6):535–541

14. Sakalys JA, Stember ML, Magilvy JK (2001) Nursing doctoralprogram evaluation: alumni outcomes. J Prof Nurs 17(2):87–95

15. Sambunjak D, Straus SE, Marusic A (2006) Mentoring in academicmedicine: a systematic review. JAMA 296(9):1103–1115

16. Shadish WR, Cook TD, Campbell DT (2002) Experimentaland quasi-experimental designs for causal inference, 2nd edn.Houghton-Mifflin, Boston

17. National Academy of Sciences. 2012. Committee to review thestate of the postdoctoral experience for scientists and engineers.http://sites.nationalacademies.org/PGA/COSEPUP/Postdoc-2011/.Accessed August 6 2012.

18. National Academy of Sciences, National Academy of Engi-neering, and Institute of Medicine. (2000) Enhancing thepostdoctoral experience for scientists and engineers: a guidefor postdoctoral scholars, advisers, institutions, funding organ-izations, and disciplinary societies. The National AcademiesPress, Washington, DC.

19. Cooksey D (2006) A review of UK health research funding.Stationary Office, London

20. Cronholm PF, Straton JB, Bowman MA (2009) Methodology andoutcomes of a family medicine research fellowship. Acad Med 84(8):1111–1117

21. Gordon SM, Dionne RA (2007) Development and interimresults of a clinical research training fellowship. J Dent Educ71(8):1040–1047

22. Jones SR, Filerman GL (2007) An analysis of the impact of theDavid A Winston Health Policy Fellowship. J Health Adm Educ24(1):33–41

23. Landrigan PJ, Woolf AD, Gitterman B, Lanphear B, Forman J,Karr C, Moshier EL, Godbold J, Crain E (2007) The ambulatorypediatric association fellowship in pediatric environmental health:a 5-year assessment. Environ Health Perspect 115(10):1383–1387

24. Rivera R, Kirby DF, Steiger E, Seidner DL (2010) Professionaloutcomes of completing a clinical nutrition fellowship: Clevelandclinic’s 16-year experience. Nutr Clin Pract 25(5):497–501

25. Rose SH, Long TR, Kor DJ, Onigkeit JA, Brown DR (2011)Anesthesiology critical care medicine: a fellowship and facultyrecruitment program. J Clin Anesth 23:261–264

26. Simon SR, Shaneyfelt TM, Collins MM, Cook EF, Fletcher RH(1999) Faculty training in general internal medicine: a survey ofgraduates from a research-intensive fellowship program. AcademicMedicine 74(11):1253–1255

27. Thacker SB, Dannenberg AL, Hamilton DH (2001) Epidemicintelligence service of the centers for disease control and preven-tion: 50 years of training and service in applied epidemiology. AmJ Epidemiol 154(11):985–992

28. Waterbor JW, Heimburger DC, Fish L, Etten TJ, Brooks CM(2002) An interdisciplinary cancer prevention and control trainingprogram in public health. J Cancer Educ 17(2):85–91

29. Betts CD, Abed JP, Butler MO, Gallogly MO, Goodman KJ,Orians C et al (1998) Evaluation of the Field EpidemiologyTraining Program (FETP). Battelle, Arlington

30. Lopez A, Caceres VM (2008) Central America Field EpidemiologyTraining Program (CA FETP): a pathway to sustainable publichealth capacity development. Hum Resour Heal 6:27

31. Matovu JK, Wanyenze RK, Mawemuko S, Wamuyu-Maina G,Bazeyo W, Olico O, Serwadda D (2011) Building capacity forHIV/AIDS program leadership and management in Ugandathrough mentored fellowships. Glob Health Action 4:5815

32. Frechtling J (2002) The 2002 user friendly handbook for projectevaluation. National Science Foundation, Gaithersburg

33. Council NR (2005) Advancing the nation’s health needs:NIH research training programs. National Academies Press,Washington, D.C.

34. Rossi PH, Lipsey MW, Freeman HE (2003) Evaluation: a system-atic approach, 7th edn. Sage, Thousand Oaks, CA

35. Pion GM, Cordray DS (2008) The Burroughs Welcome FundCareer Awards in the Biomedical Sciences: challenges to andprospects for estimating the causal effects of career developmentprograms. Eval Health Prof 31:335–369

36. Fuhrmann CN, Halme DG, O’Sullivan PS, Lindstaedt B (2011)Improving graduate education to support a branching careerpipeline: recommendations based on a survey of doctoralstudents in the basic biomedical sciences CBE-Life SciencesEducation 10:239–249

37. Sauermann, H., and M. Roach. Science PhD career preferences:levels, changes, and advisor encouragement. PLoSOne 7 (5):e36307.

26 J Canc Educ (2013) 28:18–26