RethinkingHomelessness

Their Future Depends on it!

Program Evaluation

Homeless Education State Coordinators Meeting

February 3, 2009

Key Bridge Marriott

Arlington, VA

Program Evaluation

“Program Evaluation” and

“Performance Evaluation”

are comprehensive

terms encompassing two

critical components:

Project Design Project Evaluation

Project Design - Outline

Review performance evaluation purposes and processes

Create sample outcomes and activities with criteria and measures

Assess/create your outcome measures with SMART processes

Performance Evaluation

Performance measurement:

a process that

systematically evaluates whether your efforts are making an

impact/change on/to the clients you are serving or the problem you are targeting.

Why should programs be interested in performance measurement?

We are all in the business of helping people; we need to…

• understand whether current activities are working to achieve intended results.

We are all in the business of helping people; we need to…

• understand whether current activities are working to achieve intended results.

• drive program improvement and share information on effective practices with others.

Why should programs be interested in performance measurement?

Why should programs be interested in performance measurement?

We are all in the business of helping people; we need to…

• understand whether current activities are working to achieve intended results.

• drive program improvement and share information on effective practices with others.

• acknowledge that high-performing programs receive credibility/funding especially in competitive processes.

Creating SMART Objectives With Measurable Outcomes

S pecific

M easurable

A ctivity-oriented

R ealistic/feasible

T ime-oriented

Building Blocks of Performance Measurement

Inputs

Activities

Outputs

Outcomes

Building Blocks of Performance Measurement

Inputs

include resources dedicated to, or consumed by the program—

e.g., money, staff and staff time, volunteers and volunteer time, facilities, equipment and supplies.

Building Blocks of Performance Measurement

Activities

are what the program does with the inputs to fulfill its mission–

e.g., providing school supplies or staff training, conducting a tutoring or summer program.

Building Blocks of Performance Measurement

Outputs

are the direct products of program activities, usually represented in terms of the volume of work accomplished—

e.g., number of students served and the number of staff trained—

and are often confused for outcomes.

Building Blocks of Performance MeasurementOutcomes

are benefits or changes among clients during or after participating in program activities. Outcomes relate to measurable change in student knowledge, behavior, skills, conditions, or other attributes. Outcomes impact the target you are addressing, which for us in generally the kids we serve.

Building Blocks of Performance Measurement

Outcomes (continued)

How does the activity that is engaged in impact on the target?

Performance Measurement Process

ActivitiesTrainingTutoring

Summer programField trips

School supplies

Outputs# staff trained on MV

# of students in tutoring# of students in summer

program# of students receiving school

supplies

Outcomes30% more identified40% increase TAKS 25% incrs reading

retention50%increased

attendance

How do we document

our efforts?

What did our efforts achieve?

Inputs

ProgramsInfrastructure

StaffPartnersSupplies

Should we adjust how we spend

our resources?

Should we add or

change use of resources

to expand our impact?

Outputs vs. Outcomes

An outcome is:An output is: Focused on what the

student will gain/how will change

Focused on what the program will do (activity) to achieve the outcome. A way to measure the

student-level impact with clear targets and methods for measuring change.

A way to quantify the frequency and intensity of the activity. Reasonably attributable

(a result of) to an output or outputs

Specific to the activity described

Meaningful and attainable.

Feasible and attainable.

Be mindful to distinguishbetween outputs and outcomes.

If outcomes show the program works, then outputs are needed to understand how to replicate results

Outcome, Output or Neither?

School supplies distributed to 150 campuses

90% of homeless students pass the state assessments

75% of all LEAs’ staff will be MV trained

40% of homeless students improve reading by one grade level

Homeless student attendance improves by 50%

Examples Answers

Output

Outcome

Output

Outcome

Outcome

Linking Outputs to Outcomes

Outputs Outcomes

75% of homeless students who attend after school tutoring will increase reading by one grade level

Homeless students’ attendance will increase by 50% statewide

100 % of students in homeless situations are enrolled immediately

95% of homeless students in LEAs attend after school tutoring

100 percent of homeless students receive school supplies and uniforms

LEA staff statewide staff are MV trained

Collecting the Necessary Data

Can you collect thedata you need

in order to proveyour outcome?

What data will you need to collect?

How will you collect it?

What system will you put in place to collect the data?

What criteria will tell you that you are on the right track?

When do you need to begin collecting data?

Collecting the Necessary Data

Budget Support and Alignment

Do large categories and line items support activities?

How will you know if your money is spent in the most effective way? How will you stay on top of anticipated expenses? What will you do to build a relationship with your business office? Are there business office practices that make it difficult for you to work from current figures, or know what invoices are paid?

Project Design: Summary

1. Need (with baseline reference)

2. Objective(s)

3. Anticipated Successful Target Outcome(s)

4. Timeline(s)

5. Inputs

6. Activities

7. Anticipated Outputs

Project Design: Summary

See sample schedule for

converting this project

design into a planning and

implementation document.

Project Evaluation: Summary

Objective(s)

Actual Outcome compared to Anticipated Target Outcome

Activity(ies)

Criteria used to measure activities

Project Evaluation: Summary

Actual Outputs compared toAnticipated Outputs

Analyze Status (mid-year and end of the year)

What worked? What didn’t?

Changes

What will you change to moreclosely align anticipatedoutcomes to actual outcomes –mid year or for next year?

Project Design and Evaluation: Developing M-V Outcomes

Objective:

What do you hope to accomplish?

Outcome (measurable):

How will you know you’ve accomplished it? What will tell you that you have been successful?

Project Design and Evaluation: Developing M-V Outcomes

Activities:

What activities will you put in place to accomplish your outcome?

Criteria:

What criteria will you look at to see if the activities are being completed?

Project Design and Evaluation: Developing M-V Outcomes

Measure/Output:

What will be the measure (output) of each criteria/activity to tell you that you that you are on track to meet your outcome?

Project Design and Evaluation: Developing M-V Outcomes

See sample Project Design

and Project Evaluation

handout developing a

statewide M-V outcome for

Florida.

Project Design and Evaluation: Group Activities

See sample M-V outcome

activity sheets, or develop

outcomes selected by the

group.

Special Thanks To:

Brooke Spellman &

Michelle Abbenante

For sharing their knowledge and expertise

regarding program evaluation, and for

granting their permission for to use/revise

much of their presentation delivered at the

NAEH Conference on Ending Family

Homelessness - February 8, 2008

Nothing will work if YOU don’t work!