Upload
melvin-wells
View
221
Download
2
Embed Size (px)
DESCRIPTION
Evaluability Assessment n Is a program ready to be evaluated? –Description of program –Goals and objectives –Organization ready? Identify decision-makers –Political and social factors –Model of how program works –Are resources to conduct an evaluation available? n What part of program will be evaluated?
Citation preview
Readings Text: Riddick & Russell
– Ch1 stakeholders – p10
– Ch 2 an evaluation system– Proposal p25-36
– Ch 4 – Lit Review Coursepack
– GAO report Ch 1 & 2, pp 177-185– MSU Sports pp 256-260
Evaluation is a process
Evaluability assessment Evaluation research proposal Review of the proposal Conduct evaluation Report results Implementation of findings
Evaluability Assessment
Is a program ready to be evaluated?– Description of program– Goals and objectives– Organization ready? Identify decision-makers– Political and social factors– Model of how program works– Are resources to conduct an evaluation available?
What part of program will be evaluated?
Steps for Evaluability Study
Program description – review documents Identify targets, objectives, inputs, outputs Interview key personnel Scout program Develop program model Get agreement to proceed/cooperation
INPUTS Process Outputs
Labor, Time
Capital, $$$
Land/Facilities
Use measures
“Benefits to users”
$$$
Effects
Impacts
A Program Model
Evaluation criteria
Effort – qnty & qlty of inputs Performance- qnty & qlty of outputs Adequacy – meet needs? Efficiency – costs/benefits Process – how & why pgm works? Equity – who benefits, who pays?
Purposes of Proposal
Communicate with Client Demonstrate your grasp of problem Plan the study in advance, so others can
evaluate the study approach– will it work?– have you overlooked something?– will results be useful to client?– Can we afford it?
Proposal Format1. Problem Statement - define program to be evaluated/problem to be studied,
users & uses of results. Justify importance of the problem/study.
2. Objectives : Concise listing . In evaluation studies, the objectives usually focus on the key elements of program to be evaluated & the evaluation criteria. These are the study objectives NOT the program objectives.
3. Background/Literature Review - place for more extensive history/structure of program. Focus on aspects most relevant to proposed evaluation. Discuss previous studies or the relevant methods.
4. Methods - details on procedures for achieving objectives - data gathering and analysis, population, sampling, measures, etc. Who will do what to whom, when, where, how and why?
5. Attachments - budget, timeline, measurement instruments, etc.
NOTE: Most “programs” must be narrowed to specific components to be evaluated. Think of a “Program of studies” rather than a single evaluation study. The proposal should define this specific study & how it fits into a broader program of studies.
Sample Evaluation Objectives
1. Estimate benefits and costs of program
2. Estimate economic impacts of program on local community (social, environmental, fiscal).
3. Determine effects of program on target population.
4. Describe users and non-users of program
5. Assess community recreation needs, preferences
6. Determine market/financial feasibility of program
7. Evaluate adequacy or performance of program
Describe a sample or population
Identify/test relationships between variables in a population: statistical cause-effect
Quantify the relationship
Average income of MI Snowmobilers in 1998 is $45K
SB with higher incomes spend more moneyAfter safety program, SB’s have fewer accidents
SB spend per day = $25 + .4 * Income
Typical Research Objectives
Variables : any characteristic that varies across individuals in a population (i.e. takes on different values for different individuals).
Dependent variable is the one you are trying to predict or explain, usually the focus of your study
Independent variables are the ones that help explain the dependent variable.
In Program evaluation, the outcomes are generally the dependent variables and characteristics of program or target populations are independent. In a cause effect relationship, cause is independent & effect dependent.
Variable Terminology
MSU Sports Programs
What is program(s)? Inputs, outputs, processStakeholders Which piece to evaluateEvaluation criteriaMethods to use
Methods Choices Overall Approach/Design
– Qualitative or Quantitative– Primary or secondary data– Survey, experiment, case study, etc.
Who to study - population, sample– individuals, market segments, populations
What to study - concepts, measures– behavior, knowledge, attitudes
Cost vs Benefit of Study
Qualitative vs Quantitative Approaches
QualitativeFocus GroupIn-Depth InterviewCase StudyParticipant observationSecondary data analysis
QuantitativeSurveysExperimentsStructured observationSecondary data analysis
Qualitative vs Quantitative
Purpose
Perspective
Procedures
QuantitativeGen’l LawsTest HypothesesPredict behavior
Outsider-Objective
Structured formal measures probability samples statistical analysis
QualitativeUnique/Individual caseUnderstandingMeanings/Intentions
Insider-Subjective
Unstructured open ended measures judgement samples interpretation of data
Primary or Secondary Data
Secondary data are data that were collected for some purpose other than your study, e.g. government records, internal documents, previous surveys
Choice between Primary /Secondary Data– Costs (time, money, personnel)– Relevance, accuracy, adequacy of data
How ....WhereGathered
Household On-Site Laboratory
PersonalInterview
Surveys Surveys,Field Expmts
Focus Groups
Telephone/Computer
Surveys ComputerInterviews
ComputerInterviews
Self-Admin.Quest.
Surveys,Field Expmts
Experiments
Observation& Traces
NA ObservableCharacteristics
ObservableCharacteristics
SecondarySources
NA InternalRecords
NA
Research Designs/Data Collection Approaches
Survey vs ExperimentSurvey - measure things as they are,
snapshot of population at one point in time, generally refers to questionnaires
(telephone, self-administered, personal interview)
Experiment - manipulate at least one variable (treatment) to evaluate response, to study cause-effect relationships
(field and lab experiments)
General Guidelines on when to use different approaches
1. Describing a population - surveys2. Describing users/visitors - on-site
survey3. Describing non-users, potential users or
general population - household survey4. Describing observable characteristics of
visitors - on-site observation5. Measuring impacts, cause-effect
relationships - experiments
Guidelines (cont)
6. Anytime suitable secondary data exists - secondary data
7. Short, simple household studies - phone8. Captive audience or very interested
population - self-administered survey9. Testing new ideas - experimentation or
focus groups10. In-depth study - in-depth personal
interviews, focus groups, case studies