View
219
Download
2
Tags:
Embed Size (px)
Citation preview
Evaluation in the Field: Evaluation in the Field:
Putting Concepts into Putting Concepts into ActionAction
Janet Myers, PhD MPHJanet Myers, PhD MPHRichard Vezina, MPHRichard Vezina, MPHCAPS HIV Prevention ConferenceCAPS HIV Prevention ConferenceApril 21, 2006April 21, 2006
Overview
Warm-up Where does Evaluation fit? Approaches to Evaluation Examples Q&A: Evaluating Your Programs
Warm-Up
Who here is …
Evaluator?
Service Provider?
Administrator?
4 Questions:
What are the benefits of evaluating your programs?
What are the challenges to evaluating your programs?
What needs (besides $) do you have in order to plan/conduct evaluation?
What resources do you have/use for evaluation?
Where does Evaluation fit?
Mission
Goals
Objectives
Activities
Outcomes
Impact
Program & Process EvaluationIMPLEMENTATION
Evaluation
Program&
EvaluationPLANNING
Outcome
Mission
Provides the vision
How this work makes a difference in the world
Broadest scope
Goals (Ross & Mico, 1980; McKenzie & Smeltzer 2001)
“A future event toward which a committed endeavor is directed”
Simple & concise
2 basic components
“Who will be affected”
“What will change as a result of the program”
Objectives
Specific steps that contribute to a goal. Often several objectives per goal.
Good objectives are SMART:
SS – specific
MM – measurable
AA – attainable
RR – realistic
TT – time-bound
Good Objectives Show… (McKenzie & Smeltzer
2001)
What will change: Outcome that will be achieved
When will it change: Conditions under which the outcomes will be observed
How much change: Criteria for deciding whether the outcomes has been achieved
Who will change: Target population
Activities
Internal: administrative, etc. External: the services you provide
to clients Based on your goals/objectives
Outcomes Changes that occur in people being served
by your program
Attribution: To the best extent possible, show that change is a result of your program (but note…causality is difficult)
Standards are typically different for evaluation (vs. research)
To assess, you need at least 2 time points (pre- and post-) and/or a comparison group
Impact
The scope of the program’s effects, the duration of its outcomes and the extent of its influence on the broader context (for example, HIV incidence)
Attribution: Can be more challenging to show causality, because looking for more diffuse effects
Usually broad and long-term
Typically not in the scope of program evaluation
Approaches to Evaluation
Why do we evaluate?
To determine if objectives are being met
To improve quality of the program
To decide how to change content
To identify the effects of the program
Process vs. Outcome Evaluation Process
Demographics (Who’s being trained?) Reaction to content (“Smile Sheets”) Service units delivered
Outcome Changes in knowledge/attitudes/beliefs Changes in behavior Impact on patients/clients
Process Evaluation can help us…
Create better learning environment
Improve presentation skills
Show accountability
Reflect the target populations
Track service units
Outcome Evaluation can help us…
Show the program’s effects
Allow for comparisons over time
Provide specific guide points for improving programs
Show accountability
Planning Your Evaluation (1)
Figure out your questions: What will this be used for?
Consider your Resources Staffing Time Materials $$$
Choose Methods Quantitative: Surveys, pre/post tests, etc. Qualitative: Interviews, focus groups, etc.
Planning Your Evaluation (2)
Select Indicator(s) Relevant Measurable Improvable
Instrument/Tool Development Don’t reinvent the wheel!
Analysis: Get answers to your questions
Reporting: Formal & Informal
Examples