EVALUABILITY ASSESSMENT – THE CRITICAL FIRST

STEP IN PLANNING AN EVALUATION THAT YOU MAY BE

OVERLOOKING

National Institutes of HealthProgram Planning & Evaluation Special Interest Group

December 9, 2015

The Madrillon Group Inc.

Evaluability Assessment: What Is It?

2

“Evaluability Assessment is a pre-evaluation activity designed to maximize the chances that any subsequent evaluation of programs, practices, or policies will result in useful information.”

Joseph S. Wholey, Evaluation: Promise and Performance, Washington DC, The Urban Institute 1979

Why Did OBSSR Choose a Feasibility

Study?

• OBSSR had 7 Summer Research Training Institutes (SRTIs) it

wanted to evaluate

• The SRTIs covered multiple initiatives/areas of research, which

were all enclosed under a single umbrella of training

• SRTI Characteristics included:

• Clear boundaries and a clear set of goals and objectives

• Recruited different subjects

• Some have distinct and different outcomes

• But again they all fall under the umbrella of “training”

• OBSSR was not certain an Outcome Evaluation was practical

given the nature of each institute

3

Questions OBSSR Asked

• Some of our questions included:

• Has enough time passed for the program to have had an

impact?

• Is there a baseline of questions that cuts across each of the

institutes?

• Can we harmonize data from each of the institutes?

• Is it possible to evaluate stable initiatives vs. newly emerging

fields?

• Can we group them in some ways, but still allow for

differentiation?

• Where will the data come from?

4

Benefits of a Feasibility Study

• Relatively inexpensive way to determine whether it’s practicalto evaluate the various institutes

• Allowed time to design a better Outcome Study:

• Asked what kinds of outcomes were appropriate?

• Identified stakeholders to see what they want to know

• Allowed more time for stakeholder buy-in

• Sussed out what the IC/Office wants to know

• Determined data collection methods, such as surveys,interviews, etc.

• Design Types

• For example: Is it possible or feasible to do some kind ofcomparison group in order to better understand thecounterfactual of institute training contribution

5

THE TRANS-NIH SUMMER RESEARCH TRAINING

INSTITUTE PROGRAM: AN OVERVIEW

6

Why offer summer research training institutes?

• By the late 1990s, OBSSR and other NIH staff had recognized the

need to strengthen the application of behavioral and social science

perspectives in NIH biomedical research:

• Quality of many behavioral and social sciences research (BSSR) grant

applications was lower than desired (e.g., priority scores)

• Many early-stage research investigators lacked adequate graduate

training in BSSR theories and methods

• Addressing training deficiencies not enough—also needed to build and

grow communities of research investigators

• By mid-2000s, new research fields and approaches emerged for which

there was little or no graduate training

7

Between 2001 and 2011, OBSSR and other NIH

Institutes initiated seven training institutes:

• Randomized Clinical Trials Involving Behavioral Interventions (2001)

• Advanced Training Institute on Health Behavior Theory (2004)

• NIH Summer Institute on Social and Behavioral Intervention Research

(2004)

• Summer Institute in Applied Research in Child and Adolescent

Development (2007)

• Institute on Systems Science and Health (2009)

• mHealth Training Institute (2011)

• Summer Training Institute on Dissemination and Implementation

Research in Health (2011)

8

Program design

• All seven training institutes were based on principles from adult

learning theory.

• Four aspects of the training institutes are viewed as critical

ingredients, including:

• Mentoring

• Networking among trainees and faculty

• Individual or group projects

• Residential immersive setting (5-9 day training)

9

Trainee selection

• The seven training institutes targeted trainees by career stage.

• Training institutes in established fields—new & early stage

investigators

• Training institutes for emerging fields—include middle/late

stage investigators and researchers outside health

• Trainees were a highly selected group of individuals who were

thought to be likely to be successful even without the training.

10

PHASE I FEASIBILITY STUDY

October 2013 – July 2014

11

Design of the Phase I Feasibility Study

• Evaluation Feasibility studies answer three questions:

1. Is an evaluation of a program feasible and warranted at this time?

2. What type of evaluation approach is suitable for this program at its currentlevel of maturity?

3. What evaluation design, key variables, data sources, and data collectionapproaches are most appropriate for this study, given constraints on time,budget and the quality and availability of data?

• Methods for the Phase I Summer Research Training InstituteFeasibility Study:• Literature review

• Stakeholder interviews

• Review of existing data on the training institutes

12

Feasibility Study findings address three areas

• Past evaluations of summer research training institutes at NIH

• Federal stakeholders’ assessments of desired outcomes and

evaluation designs

• Importance of the Kirkpatrick Four-Levels® Model as a

conceptual framework

13

Past evaluations of individual NIH summer

research training institutes

• Review of 21 reports of evaluations of 12 NIH summer research traininginstitutes conducted between 1997 and 2013

• Evaluation designs included the following:• Single group (trainees only) one-time follow-up survey at 6-12 months after training (12 of

21 evaluations, 57%)

• Satisfaction questionnaire at conclusion of training (5 of 21 evaluations, 24%)

• One-time follow-up survey of trainees versus comparison group (unselected applicants orcomparable non-applicants) (4 evaluations, 19%)

• Most commonly assessed outcomes included new NIH grants (90% ofevaluations), new research publications (81%), and use of training contentin teaching (33%)

• Outcomes examined were consistent with those described by theKirkpatrick Four-Levels® Model

• Past evaluations focused on what was achieved, but not how or why.

14

Interviews with NIH stakeholders and training institute

faculty members

Interviewees identified immediate, intermediate and long range outcomes:

• Immediate outcomes: increased self-efficacy to conduct research, increasedunderstanding and ability to apply training institute content;

• Intermediate outcomes: increased research productivity (NIH grants andpublications), increased collaboration with other trainees and training faculty,formation and/or participation in multidisciplinary research teams;

• Long-range outcomes: increased career progression

Interview respondents were sharply divided in their assessment of whether a comparison group was necessary to demonstrate the value of the training institutes.

Apart from a prospective experimental design, no retrospective comparative design was endorsed by a majority of interview respondents.

15

Findings from the literature review and stakeholder interviews

underscored the importance of the Kirkpatrick framework:

• The Kirkpatrick Four-Levels® Model is the dominant evaluation model usedin training evaluation today.

• The Kirkpatrick model emphasizes that training is one factor among manythat contribute to observed outcomes, and the purpose of trainingevaluation is to demonstrate that training contributes to (rather thancauses) improved performance.

• The types of outcomes identified by federal stakeholders and facultyassociated with each training institute identified outcomes that map into thefour levels of the Kirkpatrick model.

• A major contributing factor addressed by Kirkpatrick but not reflected inpast evaluations of NIH summer research training institutes is the role ofworkplace supports in facilitating (or in their absence, hindering) improvedperformance.

16

17

The Kirkpatrick Model as a Planning and Evaluation

Approach

Level 4: RESULTS

The degree to which targeted

outcomes occur, as a result of the learning event(s) and subsequent

reinforcement.

Level 3: BEHAVIOR

The degree to which participants

apply what they learned during training, when they are back on the

job.

Level 2: LEARNING

The degree to which participants

acquire the intended knowledge, skills, and attitudes based on their

participation in the learning event.

Level 1:

REACTION

The degree to which participants react

favorably to the learning event. DATA

COLLECTION

PROGRAM & EVALUATION

PLANNING

Theory of Change for Training Institutes

• Trainees perceive their training as relevant and useful, and plan to

use it in their research.

• Trainees have learned the new knowledge and skills taught in the

training institute, and made new professional contacts.

• Trainees return to their home research institutions which provide

them with promised supports for their research activities.

• Trainees use the new knowledge, skills, and contacts gained through

training to produce field-relevant grant applications, publications,

and teaching activities.

• Success with field-relevant research grants, publications, and

teaching activities leads to career advancement (promotions,

awards, etc.).

• Successful trainees increase the stock of NIH’s research portfolios.

18

How did the Phase I Feasibility Study Maximize the Chances that the Phase II Outcome Evaluation Will Provide Useful Information?

Advantages of EvaluabilityAssessment

Contributions of Phase I Study to Phase II Outcome Study

Determines whether a full-scale evaluation is feasible

Identified important sources of variability in program history that were incorporated into Phase II design

Identifies stakeholder perspectives on program goals, questions, etc.

Identified range of outcomes that have not been included in previous NIH training institute evaluations

Ensures that full-scale evaluation questions are relevant to stakeholders

Identified the specific questions various stakeholders wanted to address

Provides preliminary information on level of program implementation

Identified two distinct groups among the seven training institutes (Established vs. Emerging Sciences)

Improves and strengthens design of full-scale evaluation

Showed that a comparative design was feasible (trainees vs. unselected applicants)

Can be completed at relatively low cost and within a brief time frame

Completed in 9 months

19

Key Design Elements for an Evaluation Feasibility Study

Feasibility Study Tasks Specific ActivitiesDefine the program’s boundaries and history

• Review program documents• Review evaluation literature

Elicit stakeholders’ perspectives

• Identify perceptions about program• Identify and prioritize evaluation questions

Construct a conceptual framework

• Program logic (inputs, activities, outputs,outcomes)

• Theory of change (how the program produceschange)

Explore feasible evaluation designs, measures, and data sources

• What types of evaluation designs are feasible andappropriate?

• What measures of outputs and outcomes areappropriate?

• What existing data sources can be used, and whatnew data collection is necessary?

Make recommendations • Is it feasible to evaluate this program at this time?• What evaluation questions will address

stakeholders’ interests and concerns?• What types of measures will be most credible to

management?

20

SRTI PHASE II OUTCOME STUDY

September 2014—May 2016

21