Embed Size (px)
Citation preview
Evaluating Program Success
Cherie McCrawBorn to Read InstituteNovember 2001
Plan for Evaluation
“Evaluate with Passion and Purpose”
Decide what is appropriate Decide what is feasible Decide what is adequate Don’t reinvent the wheel
Traditional Project Measurement
Inputs Activities Outputs
Resources
- Funding
- Staff
- Volunteers
- Equipment
Services
- Training
- Tutoring
- Mentoring
- Installing
Products
- # of classes
- # of students
- # of books
- # participants
We don’t just look at “countable”things anymore.
We look at the outcome(s) of ourprograms…
how are people changed? how are people benefited?
Outcomes Measurement
Inputs Activities Outputs Outcomes
Resources
- Funding
- Staff
- Volunteers
- Equipment
Services
- Training
- Tutoring
- Mentoring
- Installing
Products
- # of classes
- # of students
- # of books
- # participants
Change inClient
- Knowledge
- Skills
- Behavior
- Condition
Pressure to Know More From:
– Funders - local, state and federal– Public - press, community
“How do we know the dollars make a difference?”
From:– Boards– Management– Staff
“How do we best use our scarce resources?”
Why Change?
The numbers don’t answer “So what?” Outcomes give us structure Create a strong case for funds in a
competitive market Establish ourselves as players for
community-wide outcomes “Just because we always have” …is not
a good reason to do something today
Outcomes Can:
Provide focus Measure results Take 3-5 years to implement
Outcomes Can’t:
Tell you whether your program caused the outcome
Indicate why the level of outcome was achieved
Suggest actions to take to improve the outcome
Choosing Which Outcomes to Measure
Use the KISS method! Look at the data you already collect Choose outcomes that are clear and
meaningful to the public For similar projects or grants- identify
common outcome(s)
Outcome Measurement
Resources– money– staff– volunteers– equipment &
supplies
Constraints– laws– regulations– funders’
requirements
Inputs Activities Outputs Outcomes
Criteria for Inputs
Identify a resource used for the project Identify a single element Are concise, clear, and lingo-free Are quantified if possible
Examples of BTR Inputs
Easy books Children’s librarian Partners: (ID each) Even Start families Deposit collections 5 trained volunteers 2 library computers Printed materials
from hospitals
51 Laubach Literacy Action trained & certified volunteers
Library facilities Matching funds Grant funds Office supplies 1 County vehicle BTR program
materials
Outcomes Measurement
Services– training– education– counseling– mentoring
Inputs Activities Outputs Outcomes
Criteria for Activities
Are verbs Are focused on the client not on the
operations of the program Show what the program does Identify a single action
Examples of BTR Activities
Provide weekly toddler and infant storytimes in all library locations
Put packets together Present parenting
workshops Conduct lapsits Provide bags for
parents Train volunteers
Enhance collections with age appropriate books
Distribute packets through home visits
Enroll low income new parents/teens and their children
Provide referral service to the adult literacy tutoring program
Examples of BTR Activities
Develop brochure Conduct BTR publicity
campaign: radio, billboards, and posters
Develop a BTR Family Literacy web page with links
Conduct phone interviews, personal interviews and focus groups
Host meetings with partners
Implement a summer reading program
Create BTR newsletter and mailing list
Issue library cards to program participants
Outcomes Measurement
Products– classes taught– counseling sessions
conducted– educational
materials distributed– hours of service
delivered– participants served
Inputs Activities Outputs Outcomes
Criteria for Outputs
Measure “how many” for the activities Often are simply the activities
quantified-50 computer classes taught Measure a single activity Do not measure a change for a client
(that would be an outcome)
Examples of BTR Outputs
120 pregnant new moms served
8 lapsits at Health Department
12 Expectant Mothers programs, 150 people attend
5,000 packets 50 new books added 2 training sessions for
project partners
# attending lapsits # of parent programs
given # of bags distributed # of volunteers # web page hits # of posters,
billboards, radio PSAs # parents/children
attending storytimes
Outcomes Measurement
Benefits for People– new knowledge– increased skills– changed attitudes or
values– modified behavior– improved condition– altered status
Inputs Activities Outputs Outcomes
Criteria for Outcomes
Are client-focused Measure a single meaningful change in
the client Are within the scope of the project Identify who achieves the outcome Are concise, clear, and lingo-free Are objective Are specific
Examples of BTR Outcomes
Parents or caregivers read to their children
Parents set a goal for their own or their child’s education
BTR staff gain insight into the information needs of the clients
Community organizations cooperate to provide services to break inter-generational cycle of illiteracy
Parents or caregivers use the library
We know what we want to achieve with our programs…
BUT...
How do we measure our success AND demonstrate to funders, boards and commissionsthat we were successful?
Your Evaluation Plan!
Outcomes Measurement:Evaluation Plan
Indicators Sources/Methods
Data you collect to measure indicators of success will let you know you have achieved your outcome(s) and to whatextent you have achieved them.
Inputs Activities Outputs Outcomes
Criteria for Indicators Show how well a program is doing on an
outcome Show that the outcomes have been achieved-
“Number of parents who read daily to their children”
Are stated as a number and/or a percentage Each outcome has at least one indicator Measure at the individual, not group, level
Setting Targets
Most programs cannot establish targets in the first year of collecting data
Possible sources for targets:– Program baseline– National statistics– Best practices research
Examples of BTR Indicators
40% program participants come to the library.
50% increase in time parents or caregivers read to their children.
150 library cards issued.
# of times library card is used per month.
# of positive responses to questionnaires.
# of parents sharing books and reading with their children.
Examples of BTR Indicators # & % of teen
program participants who use the library.
# & % of parents reading to their child on a regular basis.
# & % of parents attending workshops.
# & % of parents setting educational goals.
# & % of partners who say the partnership helped extend their services.
# of certificates given at end of program.
Criteria for Sources “Who” provides the data There should be at least one source for each indicator Possible sources include:
– Clients– Family members– Project staff– Existing records– Volunteer observers
Criteria for Methods
“How” the outcome will be measured Provide specific data to measure an
indicator Possible methods include:
– Review of project records– Questionnaire or survey– Interview– Rating by a trained observer
Examples of BTR Data Sources and Methods
Library staff/Count of coupons for diaper bags given at programs and redeemed at the library.
Parent interviews by library and Healthy Start staff.
Automated circulation system/track of usage of coded library cards.
Program administrator/Attendance records.
Questionnaires distributed by home visitors.
Examples of BTR Data Sources and Methods
Reading diaries checked by Library staff.
Project manager/Student information forms.
Project manager/agreement forms signed by students and tutors
Project manager/Focus group with program participants.
Library staff/Telephone survey asked of BTR participants.
Tutors/Pre- and post-tests.
Plan Before Collecting Data
Train data collectors with the instruments they will be using or giving out.
Develop procedure for accurately recording the results.
Setting the scene:– Signage– Collection boxes– Return envelopes
Uses of Outcome Findings
Internal– Provide direction for staff– Identify training needs– Improve programs– Support planning– Guide budgets and justify resource
allocations– Suggest outcome targets
Uses of Outcome Findings
External– Recruit talented staff and volunteers– Promote the program to potential
participants and referral sources– Identify partners for collaboration– Enhance the program’s public image– Retain and increase funding
The most compelling argumentyou make will occur when you
convey the passion you have for your project!
