Embed Size (px)
Citation preview
1/22/2015
1
Using sound evaluation practices to support your project’s success
Evaluation for VISTAs
By Sara McGarraughImprove Group Research Analyst
Let’s get to know each other
Overview
Purpose and benefits of evaluation
Evaluation tools that you can use
Designing an evaluation
1/22/2015
2
“In reality, (nonprofit) performance is all about
translating caring, believing, and compassion into results.”
Letts, Ryan & Grossman High Performance Nonprofit Organizations
1999
What is evaluation?
What is evaluation?
Assessing strengths and weaknesses …to improve
effectiveness
“
”
1/22/2015
3
Demonstrate results
*Fake data is used to create this dashboard
Reflect
Create a learning community
1/22/2015
4
Influence others
Engage stakeholders
Plan for sustainability
1/22/2015
5
What are the benefits of evaluation?
Knowledge and reflection
Information to share with stakeholders
Good evaluation practices are also good program management practices
Barriers to Evaluation
Humans resist change
We want to get along
Evaluation might suggest we are doing something wrong or would need to change
Time and cost
The Role of Evaluation
Formative evaluation: How is the program delivered and is it meeting expectations?
1/22/2015
6
The Role of Evaluation
Summative evaluation: Did the program meet its goals?
The evaluation process Define
Purpose
Develop questions
Prepare methods
Measure
Analyze
Interpret
Improve
PUTTING EVALUATION TO PRACTICE
1/22/2015
7
Start up: Define Purpose
Who are the key stakeholders of the evaluation? What you hope to learn as part of the evaluationPrioritize competing interests
Design & Planning: Refine goals
What are your program’s aspirations (your goals)?Goals should be SMART
Specific
Measureable
Attainable
Realistic
Time-bound
S
M
A
R
T
Design & Planning: Define project outcomes
1/22/2015
8
PRACTICE: REFINING GOALS AND OUTCOMES
Logic models: a basic evaluation tool
A Logic Model can describe how
your program’s goals and
activities lead to results and how
to measure them
How are they useful?
Help get everyone on the same pageEncourage investment and buy-inFacilitate organizationProvide clear and concrete guidelinesServe as a roadmap throughout evaluation processUseful for grant proposals
1/22/2015
9
What does it show?
GOALS ACTIVITIES OUTPUTS
INTENDED OUTCOMES
DefinitionsActivities:
What your program does
Outputs: Countable products showing an activity occurred (evidence)
Outcomes: Benefit received from your programKnowledge, behavior, condition changes
Example
GOAL:For all students to be at or above
their grade reading level
ACTIVITIES:-Tutoring
-Theater
-Book club
OUTPUT: # of tutoring sessions
# of books/articles read
INTENDED OUTCOMES:-Students are more confident in reading
-Students think reading is fun
TOOLS:
-Survey-Check-in form
1/22/2015
10
PRACTICE: LOGIC MODEL DEVELOPMENT
BREAK
DESIGNING DATA COLLECTION TOOLS
1/22/2015
11
Design measures and tools
What type of evidence will demonstrate outcomes?Design evaluation tools
Traditional data-gathering strategies
Surveys
Tests
Focus Groups
Interviews
Administrative data
Consider creative ways to gather data
1/22/2015
12
Matching evaluation tools to your work
Use available resources
Use knowledge of staff
Honor the wisdom of all of your stakeholders
Supported by infrastructure
PRACTICE: DATA COLLECTION TOOL DESIGN
Say what? If the question is a scale…
“How valuable was your training session with TORCH?”
a) Very valuableb) Somewhat valuablec) Not valuable
1/22/2015
13
65% of volunteers in their training program found their involvement with TORCH to be very valuable – OR –
Over half of volunteers found their involvement with TORCH to be very valuable. – OR –
80% of all respondents found their involvement with Sample Program to be somewhat or very valuable.
Then, you can report outcome statements like:
Say what? If the question is open‐ended…
“What was the most valuable part of volunteering for TORCH?”
Respondents frequently reported community engagement, connecting with youth, and learning about issues in the community as the most valuable part of volunteering with TORCH.
Then, you can report outcome statements like:
1/22/2015
14
Say what? If the question is a retrospective pre‐test…
“Please rate your opinion of the importance of volunteerism before AND after participating with the United Way.”
Before participation After participationVery
importantSomewhat important
Not important
Very important
Somewhat important
Not important
“85% of respondents rated volunteerism as very important after participating with United Way compared to only 42% before participating.”
Then, you can report outcome statements like:
Types of questions
What are the pros and cons of these question types?
1/22/2015
15
Photo by Heather McQuaid
Focus on simple and effective
Use parameters to make responding easier
Photo by Oli Shaw
Avoid double-barreled questions, using jargon, or vague questions
Photo by Nate Bolt
1/22/2015
16
PRACTICE: SURVEY OR INTERVIEW QUESTION DESIGN
Implementation: Data gathering
Smile!Practice in advanceHave a contact person for questionsRespect time & privacyCreate instructions
Survey administration
TimingResponse rates
1/22/2015
17
Reflection: Share lessons and refine practices
Meet with the program teamPrepare report(s)Distill key findingsProvide recommendations/ suggestions/areas for consideration
Evaluator’s Toolkit
A logic model or a questions and sources gridA working knowledge of different methods
A network of people who can help you
Create an evaluation plan that…
Addresses stakeholders’ needsIs validContributes to your organizationCan be implemented and takes into account available resources
1/22/2015
18
Questions or Comments?Sara McGarraugh
Continue the conversation
Or find us on Facebook and LinkedInThank you!
