10Program Evaluation

Training Busy StaffTraining Busy Staff to Succeed with Volunteers: to Succeed with Volunteers:

The 55-Minute Training SeriesThe 55-Minute Training Series

Copyright © 2007Betty Stallings

Energize, Inc.

Volunteer Program Evaluation

© 2007, Betty Stallings - Energize, Inc.

S-2

Purposes of this Session►Identify the steps needed to establish a

volunteer program evaluation.►Explore 4 types of volunteer program

evaluation. ►Learn to identify significant

stakeholders who need results of the evaluation communicated to them.


S-3

Learning Objectives►Discuss the purpose and values of carrying

out a volunteer program evaluation.► Identify key steps and questions to be

answered prior to designing a volunteer program evaluation.

►Understand the difference in purpose and value of 4 types of volunteer program evaluation techniques.

► Identify key stakeholders who must be engaged in the creation/approval of the evaluation process and given the results derived from it.


S-4

Definition►Volunteer program evaluation is a

process to determine the strengths, weaknesses, and impact of your volunteer program through use of appropriate instruments or techniques to gather this information.


S-5

Warm-up Exercise►What are some advantages to

evaluating the success of the volunteer program in your organization, unit, department, or for yourselves as individuals?


S-6

Advantages to Evaluation► Can be used to set goals for future volunteer

involvement.► Creates buy-in from all who are involved in any

aspect of the volunteer program,► Can educate staff and volunteers about the

elements of a successful volunteer program.► If given to appropriate stakeholders, may

stimulate/support needed changes.►May anticipate future concerns that can be

worked on to avert significant future problems.


S-7

Advantages to Evaluation ► Builds professionalism into the volunteer program.► Can help diffuse negative feelings. ► Can analyze cost/benefits of the volunteer

program.► Great tool for a new staff person.► Gathers various viewpoints from those involved in

the delivery of the volunteer program.► Gains a profile of the current program: the big

picture.► Can be the basis for a staff retreat to enhance the

program.


S-8

4 Key Concepts of Volunteer Program Evaluation

1. Assessment begins with clearly determining what information you want, the best process to obtain that information, and which stakeholders might be interested in the results.

2. The most common methods of gathering data for evaluating volunteer programs are: Evaluating Output Statistics Exploring Customer Satisfaction Comparison to Outside Standards


S-9

4 Key Concepts of Volunteer Program Evaluation

3. The ultimate information to derive from Volunteer Program Evaluation is the impact the volunteer program is having on the mission of the organization.

4. After data and impact of volunteer activity is measured, effectively communicate these to significant stakeholders.


S-10

Key Concept 1

Assessment begins with: ►clearly determining what information

you want, ►the best process to obtain that

information, and ►which stakeholders might be

interested in the results.


S-11

Preparation for Designing Volunteer Program Evaluation

For what purpose is the evaluation being done? What do you want to be able to decide as a result of the evaluation?

Who are the audiences that will want this information?

Do we have the support of those who will be able to carry out the suggestions arising from the evaluation?


S-12

Preparation for Designing Volunteer Program Evaluation

What kinds of information are needed to make decision and from whom do we need to gather that information?

What methods can be used to access that information?

What roles will be taken and by whom to: Develop/pilot test/approve the questions. Carry out the tabulation/analysis/interpretation of the

results. Present the material to appropriate audiences.


S-13

Discussion Question

►Have you ever carried out an evaluation and left out one of these 6 steps? What impact was there on the effectiveness/value of the evaluation?


S-14

Additional Issues to Consider► How confidentiality will be handled.► Do you have the necessary resources to carry out an

evaluation?►Who will lead the process?► How you obtain information will be impacted by the size

and culture of your organization.► Always involve a task force to carry out the evaluation.►Make certain that those who are in a position to carry out

the recommendations are in support of the evaluation.


S-15

Key Concept 2

►The most commonly used methods of gathering data for evaluating volunteer programs are: Evaluating Output Statistics Exploring Customer Satisfaction Comparison to Outside Standards


S-16

Evaluating Output Statistics► This method measures quantitative

information about a volunteer program: Numbers of volunteers, hours, clients served,

etc. Dollar value of donated time Numbers of volunteers serving in different

capacities Specifics about volunteers, such as: gender,

age, ethnicity, education, etc. Percentage of volunteers who also gave

financial donations


S-17

Methods of Computing the Value of Volunteer Hours► Minimum Wage

Lowest equivalency► Imputed Wage System

Department of Labor figure calculated annually in the US by Independent Sector

► Equivalent Wage System Calculating cost of equivalent paid job

classifications, plus fringe benefits.


S-18

Output Statistics – Potential Benefits►Change in numbers or kinds of

volunteers in your organization.►Information for volunteer insurance

purposes.►Volunteers who are giving financial

support to the organization.


S-19

Output Statistics – Potential Shortcomings► Numbers do not deal with impact or

quality of the work donated. ► There are many other quantity markers

beyond “hours served” that would be useful such as: Sources of new volunteers Average length of service Numbers of clients assisted Range of jobs performed by volunteers


S-20

Exploring Customer Satisfaction► Asking for feedback from your “customers”

Paid staff Volunteers Executive management Clients (if appropriate) Funders Community

► Qualitative information elicited by: Surveys Interviews Focus groups


S-21

Customer Satisfaction – Potential Benefits►Satisfaction from customers greatly

impacts the success of the volunteer program.

►Determines major weaknesses and strengths in the program from perspective of customers.

►Gains buy-in from staff for volunteer program

►Great form of volunteer appreciation.


S-22

Customer Satisfaction – Potential Shortcomings

►Answers not accurate if there are organizational trust issues.

►Must prepare those taking the survey regarding reasons for survey and what you will do with the results/ recommendations.


S-23

Activity►Look over the examples of customer

surveys in Handout #3. ►What are the benefits and

shortcomings of each example?►Which questions would you keep or

discard for your organization and what others would you add? Why?


S-24

Comparison to Outside Standards►Comparison to objective management

standards created by an outside body or authority.

►Examples of such standards: The Volunteer Management Audit (Energize, Inc.) The Canadian Code for Volunteer Programs

(Volunteer Canada) Points of Light Paradigm Project and Volunteer

Program Evaluation


S-25

Comparison to Standards – Potential Benefits

►Education of excellence in standards of volunteer management.

►Identification of areas of weakness that may be greatly impacting your program.

►Gives you information that will help you develop goals for improving the program.


S-26

Comparison to Standards – Potential Shortcomings

►Takes dedication/time.►Not helpful if no action occurs after you

identify areas needing improvement.►May be challenging to compare your

program to a set of generic standards.


S-27

Key Concept 3

►The ultimate information to derive from Volunteer Program Evaluation is the impact that the volunteer program is having on the mission of the organization.


S-28

Impact or Outcome-Based Evaluation►Focuses on results – not just activities.►Question asked is: What has changed

in the lives of individuals, families, organizations, or the community as a result of this program?

►Keep asking: “So what?”


S-29

Impact or Outcome-Based Evaluation►An outcome-based evaluation discovers:

What target groups will change? What about the target will change and in what direction

will it change? What will happen to create that change? How will the change will be identified and tracked? How will the change will be measured? How will we know that our service is what impacted the

change? How long must the change remain in effect before being

considered successful?


S-30

Challenges of Impact Evaluation► Some attempts and research tools to

demonstrate the impact of volunteers.► None, to date, are without result flaws and all

are very demanding in terms of time.►We can’t strictly evaluate the exact impact of a

volunteer, but in we can measure discrete planned change in the recipients of volunteer service in

Knowledge gain Values altered Behavioral adjustment

Alteration in social or economic status

Attitude change Improved skills Client reaching set

goals Mood, stress

reduction


S-31

Example►The mission of the mentoring program

is to keep high-risk students out of gangs and involved in further education.

►To describe the impact of the mentoring volunteers, begin to describe the program. For each statement, ask: So what? To whom might this matter? Is this really evidence of something important?


S-32

Example. continued► We had 90 volunteers working on the mentor project for

3 months. (So what?)► The volunteers were mentoring high risk students at the

continuation high school. (So what?)► Most students reported liking their mentor. (So what?)► Most students began to turn in their homework more

regularly. (So what?)► The students began to develop more pride in their work

and began to get passing grades. (So what?)► As a result of receiving better grades, many were able

to transition back into regular high school. (So what?)


S-33

Example, continued► 59% of high risk students being mentored for a minimum

of 3 months by a caring adult tested higher on standardized tests, got after-school jobs that they had kept for 2 months, and were expressing interest in additional education to prepare them for the work world. (So what?)

► It has been statistically shown that students who are able to keep an after school job and are motivated to get additional training/education for future employment have a 70% less chance of breaking the law and becoming dependent on drugs and involved with street gangs. OH, WOW, GREAT!!! So that is the impact! I

would love to be involved and/or support that program!


S-34

Activity►Find a partner.►One person explains the impact of some

volunteer work on the mission of the organization.

►The other person keeps asking, “So what?” until both of you are satisfied that you understand the impact of the volunteer work being done.


S-35

Key Concept 4

►After data and impact of volunteer activity is measured, effectively communicate these to significant stakeholders.


S-36

Communicating to Stakeholders

►Who needs the information? Volunteers Paid Staff Executive Team or Executive Director Board Clients Funders The Community


S-37

Activity►What is your experience with volunteer

program evaluations?

►How might you establish an evaluation system for whatever involvement you have with the volunteer program?

►To whom would you give the results of that evaluation to help improve the program in your area and/or in the larger organization?


S-38

Summary► Answer key questions before designing your

evaluation.► Select the appropriate method(s) to access the

information needed: Statistical Outputs Customer Satisfaction Standards-based evaluation Impact/Outcome Evaluation

►Share Information with key stakeholders. Summary of key findings Complete report on results and recommendations

Documents

10Program Evaluation