Upload
guest916929
View
350
Download
0
Tags:
Embed Size (px)
Citation preview
Planning Campaigns with Evaluation in mind
By Jorge Restrepo M.G.AEureka Facts LLC
Design• Clearly define the intended outcomes of a program and
ensure that clear measures can be obtained. The key challenges in program evaluation stem from unclear goals or promised outcomes that go beyond the scope of a grant proposal.
• Educational, language and cultural barriers are necessary considerations in good evaluation design.
• Know what data you need that the respondent will not answer, and develop alternative methods of collection or inference.
Design
• The time crunch – this most often impacts design and instrument testing. Carefully consider timelines for data collection- especially when programs require pre-post measures. Consider other methods to make maximum use of the time required for program deployment.
• Pre-post evaluation are complex so consider their use before promising a specific method in a campaign proposal.
• Evaluate outcome not just activity– changes in awareness, knowledge, behavior, are also good indicators of the extent of a campaign’s outcome.
Methods - data collection
Tips for in-person data collection• Protocols and interviewer training are key• Supervision and redundant verifications are
important • Dress-codes and interviewer profile• Compensation/ Investment• Controls• Timing• Response rate
Methods - data collection
Mail surveys• Lists – sample frame• Who is your respondent?• Sponsor, package, timing, incentive• Respondent burden• Education/language• Coding visibly, invisible coding• Flexibility• Timing• Response rate - examine non-response closely
data collectionOnline
• Who is your respondent? - Your sample frame is everything
• Spam filters are a bigger concern than non-response
• Web links create disproportionate sampling• Make web links and pop-ups as unobtrusive
as possible• Respondent fatigue• CANSPAM • COPA
data collectionTelephone
• Who is your respondent? - Your sample frame is everything
• RDD sampling – often no longer representative• Combine methods • No land line households• Reach• Incentives• Cooperation• Do not call registry
Challenges in Analysis• Most distributions – especially in public education
efforts are not normal so common centrality statistics (mean, median) are not very meaningful.
• People often articulate categories (i.e. well, not well) better than continuum scales.
Analysis• Think outside the box – examine
findings spatially
Stars represent respondents to a campaign
Analysis• Think outside the box –
examine findings spatially
Flat findings examined by other means revealed strong differences by county
Simplify constructs• Build your assessment on existing frameworks
and models, but create high level aggregates as well
Mean Levels of Knowledge and Action
Source: Random survey
State
SCNCGAFL
Me
an
8
6
4
2
0
Know ledge
Action
This evaluation measures knowledge and action
High and Low Impact Groups• A powerful tool is the use of classification trees
to identify pockets of a population with high or low impact.
Compare - Always
Mirrored instruments are especially valuable in examining outcomes
Parents can move their child to a different school when their school has a failing grade.
0
20
40
60
80
100
Jan
Feb.
Mar
.Apr
ilM
ayJu
ne July
Aug.
Sept.
Oct. Nov.Dec.
Month
Random survey
Callback survey
Response cards
Segment for Success
Outcomes are rarely evenly spread out throughout a population
Common cross-tabbing analysis
• Ethnicity
• Hispanic Origin
• Gender
• Age
• Education
• Urban locale
• Generational cohort
• Socioeconomic status
• Life stage, career stage
• Primary language
• Geography
Who we are
Eureka Facts brings you the smart marketing Information you need to
discover opportunities
maximize results
identify issues
make decisions
reduce market risks
Smart Evaluation
www.EurekaFacts.com
EurekaFacts LLC451 Hungerford Drive
Suite 515 Rockville MD 20850
(301) 610-0590