Evaluation 101: A Workshop Overview, Concepts and Application
Today:
Review some definitions Talk about practical concepts for
AETC evaluation Discuss a useful framework Apply these concepts and the
framework to your programs Identify other evaluation resources
Definition: Rossi, Freeman and Lipsey “Program evaluation is the use of
social research procedures to systematically investigate the effectiveness of … programs.”
Concepts: Rossi et al’s 5 program domains Program evaluation typically involves
assessment of one or more of the five program domains:
1. The need for the program 2. Design of the program 3. Program implementation and service
delivery 4. Program impact or outcomes 5. Program efficiency
Concepts: Chlemisky’s Perspectives
Accountability: Information collected for decision makers (emphasis: outcomes)
Developmental: Information collected to improve performance (emphasis: process measures)
Knowledge: Information collected in the interest of generating understanding and explanation (emphasis on processes and outcomes).
Scriven and context
Evaluation has two arms: 1. Data gathering 2. Contextualizing results
Some definitions…
Process evaluation: Addresses questions about how well the
program is functioning Is useful for diagnosing outcome Is critical in quality improvement
Some definitions…
Key questions in process evaluation: Who is served? What activities or services are provided? Where is the program held? When and how long?
Some definitions…
Outcome evaluation: Gauges the extent to which a program
produces the intended improvements it addresses Addresses effectiveness, goal attainment
and unintended outcomes Is critical in quality improvement
Some definitions…
Key questions in outcome evaluation: To what degree did the desired change(s)
occur? Outcomes can be initial, intermediate
or longer-term Outcomes can be measured at the
patient-, provider-, organization or system level.
Some definitions…
Impact is sometimes used synonymous to “outcome.” Impact is perhaps better defined as a
longer-term outcome. For clinical training programs, impacts may be improved patient outcomes.
Some definitions…
Indicators or measures are the observable and measurable data that are used to track a program’s progress in achieving its goals. Monitoring (program or outcome
monitoring, for example) refers to on-going measurement activity
Some definitions…
CQI/QM: HAB’s definition of quality: The degree
to which a health or social service meets or exceeds established professional standards and user expectations
Some tools for planning…
United Way’s Outcomes Manual HRSA’s Quality Improvement
Handbook W.K. Kellogg Foundation Evaluation
Handbook
Some methods…
Identify some Quantitative Methods Identify some Qualitative Methods Which is best, qual or quant?
A framework for AETC evaluation: Kirkpatrick Measure Reaction Measure Learning Measure Behavior Measure Results
Identify some ways to measure each
in an AETC training setting
Application to Your Program:
Identify Program Goals For each goal: Identify Process Objectives Identify Outcome Objectives
For each objective: Identify Indicators Identify Data Source Plan Data Collection Plan Data Analysis
Exercise:
Goal 1:
Objectives: Indicator Data Source
Data Collection
Data Analysis
Process Objective 1:
Outcome Objective 1:
Recommended Reading
Definitely: Kirkpatrick Cruz the American Journal Evaluation (the
Journal of the American Evaluators’ Association)
If you have time: Chelimsky Rossi
If you want a good read: Hunt
Handouts
Table 1.1 from Chapter 1 in Chelimsky: The Coming Transformations in Evaluations
Exhibit 2-P An Australian Teams’ Ten Step Approach to Program Evaluation (p. 75 in Rossi et al)
Chapters 1-3 in Kirkpatrick American Evaluators’ Association Guiding
Principles for Evaluators Isenberg’s CQI Program Handout