Using sound evaluation practices to support your project’s success

Evaluation for VISTAs

By Sara McGarraughImprove Group Research Analyst

Let’s get to know each other

Overview

Purpose and benefits of evaluationEvaluation tools that you can useDesigning an evaluation

“In reality, (nonprofit) performance is all about

translating caring, believing, and compassion into results.”

Letts, Ryan & Grossman High Performance Nonprofit Organizations

1999

What is evaluation?

What is evaluation?

Assessing strengths and weaknesses …to improve

effectiveness“

”

Why Evaluation?

Demonstrate results

*Fake data is used to create this dashboard

Reflect

Create a learning community

Influence others

Engage stakeholders

Plan for sustainability

What are the benefits of evaluation?

Knowledge and reflectionInformation to share with stakeholdersGood evaluation practices are also good program management practices

Barriers to Evaluation

Humans resist changeWe want to get alongEvaluation might suggest we are doing something wrong or would need to changeTime and cost

The Role of Evaluation Formative evaluation: How is the program delivered and is it meeting expectations?

The Role of Evaluation

Summative evaluation: Did the program meet its goals?

The evaluation process

How are project and evaluation phases similar?

PUTTING EVALUATION TO PRACTICE

Start up: Define PurposeWho are the key stakeholders of the evaluation? What you hope to learn as part of the evaluationPrioritize competing interests

Design & Planning: Refine goals

What are your program’s aspirations (your goals)?Goals should be SMART

Specific

Measureable

Attainable

Realistic

Time-bound

SMA

RT

Design & Planning: Define project outcomes

What is the

CHANGE you wish

to see?

PRACTICE: REFINING GOALS AND OUTCOMES

Logic models: a basic evaluation tool

A Logic Model can describe how your program’s goals and

activities lead to results and how to measure them

How are they useful?

Help get everyone on the same pageEncourage investment and buy-inFacilitate organizationProvide clear and concrete guidelinesServe as a roadmap throughout evaluation processUseful for grant proposals

What does it show?

GOALS ACTIVITIES

OUTPUTS

INTENDED OUTCOME

S

MEASURES

DefinitionsActivities:

What your program doesOutputs:

Countable products showing an activity occurred (evidence)

Outcomes: Benefit received from your programKnowledge, behavior, condition changes

Example

GOAL:For all students

to be at or above their

grade reading level

ACTIVITIES:-Tutoring-Theater-Book club

OUTPUT: # of tutoring sessions# of books/articles read

INTENDED OUTCOMES:-Students are more confident in reading-Students think reading is fun

MEASURES:-Survey

-Check-in form

An alternative to a logic model

PRACTICE: LOGIC MODEL DEVELOPMENT

BREAK

DESIGNING DATA COLLECTION TOOLS

Is it a tool, instrument, or protocol?

Design measures and toolsWhat type of evidence will demonstrate outcomes?Design evaluation tools

Traditional data-gathering strategies

SurveysTests

Focus GroupsInterviewsAdministrative data

Consider creative ways to gather data

Matching evaluation tools to your work

Use available resourcesUse knowledge of staffHonor the wisdom of all of your stakeholders Supported by infrastructure

PRACTICE: DATA COLLECTION TOOL DESIGN

Say what? If the question is a scale…

“How valuable was your involvement with Sample Program?”

a) Very valuableb) Somewhat valuablec) Not valuable

65% of all respondents found their involvement with Sample Program to be very valuable – OR – Over half of respondents found their involvement with Sample Org to be very valuable.

80% of all respondents found their involvement with Sample Program to be somewhat or very valuable.

Then, you can report outcome statements like:

Say what? If the question is open-ended…

“What was the most valuable part of volunteering for Sample Program?”

Respondents frequently reported community engagement, connecting with participants, and learning about issues in the community as the most valuable part of volunteering with Sample Program.

Then, you can report outcome statements like:

Say what? If the question is a retrospective pre-test…

“Please rate your opinion of the importance of volunteerism before AND after participating with Sample Program.”

Before participation After participationVery

importantSomewhat important

Not important

Very important

Somewhat important

Not important

“85% of respondents rated volunteerism as very important after participating with Sample Organization compared to only 42% before participating.”

Then, you can report outcome statements like:

Types of questions

What are the pros and cons of these question types?

Photo by Heather McQuaid

Focus on simple and effective

Use parameters to make responding easier

Photo by Oli Shaw

Avoid double-barreled questions, using jargon, or vague questions

Photo by Nate Bolt

PRACTICE: SURVEY OR INTERVIEW QUESTION DESIGN

Implementation: Data gathering

Smile!Practice in advanceHave a contact person for questionsRespect time & privacyCreate instructions

Survey administration

TimingResponse rates

Reflection: Share lessons and refine practices

Meet with the program teamPrepare report(s)Distill key findingsProvide recommendations/ suggestions/areas for consideration

Evaluator’s ToolkitA logic model or a questions and sources gridA working knowledge of different methods

A network of people who can help you

Create an evaluation plan that…

Addresses stakeholders’ needsIs validContributes to your organizationCan be implemented and takes into account available resources

Questions or Comments?Sara McGarraugh

Continue the conversation

saras@theimprovegroup.com www.theimprovegroup.com

Or find us on Facebook and LinkedIn Thank you!