4c6ba937b52c03bcf0_l4m6b3559

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

1/37

Measuring Success UsingPerformance Measurement

NAEH Conference on Ending Family Homelessness

Michelle Abbenante

Brooke Spellman

February 8, 2008

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

2/37

2008 NAEH Family Conference - Outcomes Workshop 2

What is Performance Measurement?

Performance measurement is a process

that systematical ly evaluateswhether your

effor tsare making an impacton the clients

you are serving or the problem you are

targeting.

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

3/37


Multiple Levels of Performance Measurement

Local Service

Provider

1. Program Level Program Funding Report,such as HUD APR

CoC

2. CoC/System LevelTen-Year Plan Report Card

or CoC Application

3. State LevelState-wide Report or

Performance Measurement

Tool

4. National LevelNAEH Assessment of

National Progress or

Federal GRPA and PART

Reviews

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

4/37


Why Should Programs be Interested inPerformance Measurement?

We are all in the business of helping people,which means we need to

understand whether current activities are working

to achieve intended results.

drive program improvement and share information

on effective practices with others.

acknowledge that high-performing programs are

more likely to receive funding through competitive

funding processes.

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

5/37


Building Blocks of Performance Measurement

Inputs include resources dedicated to, or consumed by, the

programe.g., money, staff and staff time, volunteers andvolunteer time, facilities, equipment and supplies.

Activities are what the program does with the inputs to fulfill its

mission, such as providing shelter, feeding the homeless, or

providing job training.

Outputs are the direct products of program activities. They

usually are presented in terms of the volume of work

accomplishede.g., number of participants served and the

number of service engagements.

Outcomes are benefits or changes among clients during or after

participating in program activities. Outcomes may relate to

change in client knowledge, attitudes, values, skills, behaviors,

conditions, or other attributes.

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

6/37


Performance Measurement Process

ActivitiesOutreach

Shelters

Case Management

Rent Subsidies &

Services

Outputs# Clients Served by Program

Service Linkages

New PSH Units/Subsidies

Vacancy Statistics

Outcomes

30% exited to PH

40% increased income

25% reduction in CH

25% shorter LOS

< recidivism

How do

we

document

our

efforts?

What did

our efforts

achieve?

Inputs

$ (CoC and Other)

Programs

Infrastructure

Staff

Should weadjust how

we spend our

resources?

Should we

add or change

use of

resources to

expand our

impact?

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

7/372008 NAEH Family Conference - Outcomes Workshop 7

Outputs vs. Outcomes

Whereas, an outcome is: An output is: Focused on what the

participant will gain from

the program.

Focused on what the

program will do to

achieve the outcome. A way to measure the

client-level impact with

clear targets and methods

for measuring change.

A way to quantify the

frequency and intensity

of the activity.

Attributable (a result of)to that program.

Specific to the activitydescribed for the program.

Meaningful and

attainable. Feasible and attainable.

Be mindful to distinguish between outputs and

outcomes.

If outcomes show the program works outputs are

needed to understand how to replicate results

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559


Group Exercise:Outcome, Output, or Neither?

150 clients received prevention counseling andone-time financial assistance.

90% of persons will obtain employment bycompletion of program.

75% of program staff will be trained in crisismanagement techniques.

Met 40% (50) of Permanent Supportive

Housing goal.

65% of clients with chronic medical conditionwill improve physical health

Examples Answers

Output

Outcome

Activity

Output

Outcome

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559


Achieving Your Outcomes

Short-term outcomes: What change will the client

experience within a month of his/her involvement in

the program? How will you measure this?

Intermediate outcomes: What change will the client

experience within a yearof being involved in the

program? How will you measure this? Long-term outcomes: What is the long-term (e.g., 3-

year) impact of the program on clients? Has it been

sustained? How will it be measured?

Achieving your outcomes can be a progression.

Monitoring this progression requires data that are

collected at different intervals:

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559


Example: Employment Programs Impact OverTime

93% of participants (97% of

people who completed thejob training class) will show

improved job skills


those who complete the job

training class) will obtain full-

time employment


those who get a job) will

retain their jobs for > 12

months.

~95 participants will completejob training classes.

All of those who complete the

training classes (~95 clients)

will be referred to jobs and

receive job placement

counseling.

All of those who get a job

(~52 clients) will receive

weekly check-up calls and job

counseling, as needed.

100 people expected to participate

in the program annually

Short-term

Long-term

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559


Framework for Converting Program Goals intoOutcomes

Who is the base

for measuring

results?

Step 1

What do you

hope to achieve

with this

population?

Step 2

Within the base, how

many persons

achieved it?

Step 3

Who is the base population for

measuring results?

Within the base, how many

persons achieved it?

Step 4

Outcome (%)

1. How do I convert program goals into measurableoutcomes?

2. What do I need to calculate the outcomes?

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559


Converting Program Goals into Outcomes:Example

Program Goal: Supporting participants in stable housing at least 6

months

Base= Persons who

have been enrolled >

6 mo or have exited

(n=40 people)

Step 1

Goal: remain

housed > 6 mo

Step 2

20 people are still in

stable housing (6+mo)

or exited after being in

housing for 6+ mo

Step 3

40

20

Step 4

50% remain in

stable housing

at least 6 months

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559


Look Out for Ambiguous Concepts

For example, what do we mean by

Developing and measuring performance outcomesoften invites ambiguous concepts into the process.

obtaining stable housing? obtaining employment?

increasing income?

accessingservices?

becoming more self-sufficient?

Which data elements and responses will count?

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559


HMIS Data Elements Are the Building Blocks ofPerformance Measurement

Universal Data Elements: Program-Specific Data Elements:

Name Income & Sources *

Social Security Number Non-Cash Benefits *

Date of Birth Physical Disability

Ethnicity & Race Developmental Disability

Gender

HIV/AIDS

Veteran Status Mental Health

Disabling Condition Substance Abuse

Residency Prior to Entry Domestic Violence

Zip Code of Last Permanent Add Services Received

Entry Date Destination

Exit Date Reasons for Leaving

Person, Program, & HH ID

* These data elements are collected at entry and exit.

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559


Wherever Possible Use HMIS to Define YourConcepts

Emergency shelter Apartment or house that you own

Based on the Destination HMIS data element, we candefine stable housing (narrowly) using the following

response categories:

Transitional housing Permanent housing

Substance abuse facility or detoxcenter

Hospital (non-psychiatric)

Jail, prison or juvenile detentioncenter

Room, apartment, or house thatyou rent

Dont Know

Staying/living with family

Staying/living with friends

Hotel or motel voucher paid forwithout ES voucher

Place not meant for humanhabitation

Other

Foster care home or group home

Refused

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559


Group Exercise

Employment Program

The goals of the program are to help unemployed clients obtain

employment and help employed clients get better jobs. During the

past year, the program served 6 (unduplicated) persons:

Client

ID

Entry Date Exit Date Employment

Entry

Employment Exit

1 1/31/07 9/15/07 Unemployed Employed

2 3/15/07 6/28/07 Unemployed Unemployed

3 7/11/07 -- Unemployed --

4 7/7/07 9/18/07 Employed SameEmployment

5 8/2/06 5/12/07 Employed Higher Paying Job


7/27/2019 4c6ba937b52c03bcf0_l4m6b3559


Define the Base Population for Each Goal

Client

ID

Entry Date Exit Date Employment Entry Employment Exit


2 3/15/07 6/28/07 Unemployed Unemployed

3 7/11/07 -- Unemployed --

4 7/7/07 9/18/07 Employed Same Employment

5 8/2/06 5/12/07 Employed Higher Paying Job


Goal 1: Achieve employment at exitGoal 2: Obtain better employment at exitIs everyone part of the target population?E.g., Do you expect to calculate an outcome for everyone?

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559


Calculate the Outcome for Goal 1

All unemployed

persons at entry who

exited (N= 3)

Step 1

Achieve

employment

Step 2

2 persons achieved

employment

Step 3

3

2

Step 4

67% achieved

employment

Program Goal 1: Obtain Employment at Exit

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559


Calculate the Outcome for Goal 2

Persons who were

employed at entry

and exited (N= 2)

Step 1

Improve

employment

Step 2

1 person increased

earnings

Step 3

2

1

Step 4

50% gained better

employment

Program Goal 2: Improved Employment at Exit

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559


Exercise 2 and the Performance MeasurementProcess

Inputs

Money: $250,000

Staff: 4 FTEs

1 Facility

ActivitiesJob Training Classes

Interview Assistance

Job Placement

Services

Outputs6 enrolled in

weekly services

6 employment

assessments

Referred to av. 4

jobs each

Outcomes

67% achieved empl.

50% improved empl.

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

21/37


Using Outcomes to Inform Future ProgramOperations

Step 1

Review

outcomes

with programmanagers

Develop

action steps

and timelines

Step 2

Implement

action steps

Step 3

Regular

monitoring

Step 4

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

22/37


Step 1: Reviewing Outcomes with Managers

Program director and managers should review

outcomes collaboratively to understand what the

outcomes are suggesting.

Break down the outcomes to understand theunderlying forces:

Whats Going On?

What are we doing right? What activities contributed to our

ability to meet/exceed our benchmarks?

Where do we need to improve? What activities fell short ofproducing the desired outcomes?

What else might be contributing to our outcomes? How can

we influence or mitigate these external forces to further our

positive outcomes?

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

23/37


Step 2: Developing Action Steps and Timelines

Reinforcing the Good and Adjusting the Bad

Outcomes that were achieved/exceeded: Continue tosupport the activities that led to our positiveperformance.

Outcomes that were not achieved: Allocate our inputsdifferently to support different types/levels ofactivities.

Set target dates for reviewing all outcomese.g., 3-month intervals.

Collaborate with other service providers to controlthe external impacts on the program.

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

24/37


Step 3: Implementing the Action Steps

Getting Buy-In Through Information Sharing

You cant implement what you dont understand:program directors, managers and front-line staff must

understand the reasons for making changes in

program operations.

Information sharing promotes the idea that we are allin this together.

Information sharing is fluid: program directors,managers and front-line staff can learn from one

another; its not a one-way (top-down) process.

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

25/37


Step 4: Regular Monitoring

Its Easier to Adjust Program Operations

Incrementally than Wholesale

Monitor your progress by generating yourperformance outcomes at different periods of time

e.g., 3-month intervals.

Adjust your approach as needed, but usuallyincrementally.

Important to acknowledge that clients needs mayshift, and thus program goals and approach may also

need to shift.

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

26/37


Comparing Program Results

You can compare results from one program to another tosee which programs are working best and which areworking least well

With limited dollars, you want to fund the programs that aremost effective.

You can use program results to identify best practiceprograms and those that need TA

You can use results from multiple programs to help set acommunity expectation or standard of performance

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

27/37


Case Study: How Washington, D.C. UsesProgram Results

Outputs/Efficiency Measures

Clients Served

Chronically Homeless Served

Occupancy: the rate at which program was used

Interim Outcome Measures Permanent Housing

Positive client destinations at exit (TH programs)

Retain clients for 6+ months (PSH programs)

Income: the amount of income or sources obtained

Self-Sufficiency: change in substance use, education, mental illness oremployment

Measures apply differently to each program type and aresupplemented with qualitative data for ranking purposes

W hi t DC

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

28/37


Provider Performance Measures

ClientsServed

ChronicallyHomeless

OccupancyRate

HousingDestinations

Income Length ofStay

SelfSufficiency*

OutreachPrograms Severe

Weather &Low Barrier

Shelters

TemporaryShelter

TransitionalPrograms PermanentSupportiveHousing

Supportive

ServicePrograms

*Self Sufficiency Temporary, Transitional and Permanent Supportive Housing Programs required to

submit a Self Sufficiency Indicator will have to choose from Substance Abuse, Education, Mental Illness

Assistance or Employment.

Washington, DC

FY 07 DHS Performance Measures

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

29/37


Apples to Apples: Risk Adjustment

Problem: Comparing program results canencourage programs to cream to ensure strongresults

Solution: Risk adjustment allows you to account for

differences in client populations when comparingresults across programs

Results can be adjusted on the basis of

Client characteristics, such as demographics,family size, disability

Client history, such as past eviction, criminalbackground,

Client functionality or level ofengagement/commitment to change

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

30/37


Risk Adjustment Requires Expertise

To adjust for client differences, programs need to collect consistent

data on clients to use during analysis of program results

Agree on these standards beforehand

Develop an analysis plan for how you intend to adjust for client

differences

Engage a researcher to help develop the plan

Even if you dont formally adjust results, acknowledge that differentprograms may have different outcome expectations based ondifferences in clients targeted and/or served

Si lifi d Ill t ti f Di ti Cli t

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

31/37


Simplified Illustration of Dissecting ClientOutcomes on Increased Earned Income

By establishing targets, programs can be compared against CoC

expectations in the future to determine if program performance is

higher or lower than expected.

Program A

61% (61 clients)

All Clients (n=100)

10%

(1 of 10 persons)

Disabled Clients

67%

(60 of 90 persons)

Non-disabled Clients

23% (70 clients)

All Clients (n=300)

12%

(30 of 250 persons)

Disabled Clients

80%

(40 of 50 persons)


Program B

12%

(31 of 260 persons)

Disabled Clients

71%

(100 of 140 persons)


CoC OutcomeResults By

Population

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

32/37


System Performance Measurement

Are your actions achieving your intended goals at thesystem level?

Does the system work?

If yes, what makes it work?

If no, what part doesnt work, and how do you fix it to makeit work?

Note that you may have system goals that only relate tocertain types of clients or parts of the system (e.g.,different goals for severely disabled persons)

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

33/37


Sample Impact Measures

Incidence of homelessness - Is homelessness declining?

Incidence of street or CH - Is street or chronic homelessnessdeclining?

Length of stay in system, across all homeless programs - Do people

stay homeless for shorter periods of time?

Prevention Are fewer people experiencing homelessness for thefirst-time?

Rates of Recidivism Are repeat occurrences of homelessnessavoided or declining?

Cross-tabulate results by core characteristics to understand if/how

results vary for different subpopulations

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

34/37


Steps to Calculate System Length of Stay

De-duplicate clients

across programs

Step 1

Consolidate sequential

stays into single

episode (gaps < 30

days = same episode)

Step 4

Create table with all

Entry/Exit Dates by

Client

Step 2

Calculate LOS for

each Stay

Step 3

Calculate mean (168

days), low (81 days),

high (309 days)

Step 5

Client

ID

Prog

ID

1 A

1 B

2 A

3 C

Entry Exit

Date Date

5/8/06 5/30/07

6/1/07 9/01/07

3/1/07 5/21/07

2/1/06 12/7/06

LOS

22

92

81

309

114

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

35/37


Some notes of caution

There is more to performance measurement than

conducting the analysis Educate, train, obtain buy-in

Be careful about how you interpret and use the

data1. Jump in, but dont be careless in how you use the results

2. Look at the results within the context of all the outputs, interimmeasures and impact measures to validate the interpretationthats being made

3. Vet the results before publicly releasing anything4. Appropriately caveat the limitations of the data and analysis

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

36/37


Summary of System Performance Measurement

ActivitiesAdjust type andintensity of

activities based on

outcomes; track if

improves

OutputsDocument the level of

effort provided

Use to ensure activities

delivered efficiently

OutcomesInterim Outcomes

signal client success;

Impact outcomes

track progress to

goals

Inputs

Use program-level &

system-wide results

to adjust use of

resources

7/27/2019 4c6ba937b52c03bcf0_l4m6b3559

37/37

Questions?

Contact us for more information or assistance:

Michelle Abbenante, [email protected] Spellman, [email protected]
mailto:[email protected]:[email protected]:[email protected]:[email protected]

Documents

4c6ba937b52c03bcf0_l4m6b3559