HMIS Project Evaluation September 13-14, 2005 St. Louis, Missouri Sponsored by the U.S. Department of Housing and Urban Development Oscar Gutierrez, Change

HMIS Project Evaluation

September 13-14, 2005St. Louis, Missouri

Sponsored by the U.S. Department of Housing and Urban Development

Oscar Gutierrez, Change and Strategy Solutions

Roger Blake, University of Massachusetts Boston

Rob Anthony, Lake County Department of Planning, Building, & Development

HMIS Project Evaluation Framework



Oscar Gutierrez, Change and Strategy SolutionsRoger Blake, University of Massachusetts Boston

September 13-14, 2005 St. Louis, MissouriSponsored by the U.S. Department of Housing and Urban Development

3

Contents

• Challenges: Why is Evaluation Necessary?• What is an HMIS Project Evaluation?• Objectives of the Evaluation• Types of Evaluation• Evaluation Strategy• Who Should Conduct the Evaluation?


4

Challenges

• How do we explain the project’s progress when there are few tangible results to guide the evaluation?

• How do we explain in non-technical terms the progression of a technical initiative.

• How do we explain that HMIS implementation projects are much more than acquiring equipment, software and training?


5

What is HMIS Project Evaluation?

• Possible Evaluation Categories in HMIS• Program Evaluation – e.g., how effective the community

is operating

• System Evaluation – e.g. ,determining which HMIS should be selected

• Project Evaluation – • How are we doing with our HMIS project

– Are we on track?– Will we get the benefits we are supposed to?


6

A Common Myth About Evaluation

• When one first thinks of evaluation, the common assertion is that hard, numeric figures are the primary and sometimes only reliable source of assessment.

• Why is it that in HMIS implementations this assertion seldom holds true?


7

“Evaluate”

• “To calculate or set down the numerical value of; express numerically”

• “To examine and judge carefully”


8

Definition

• HMIS Project Evaluation is:• A mechanism for keeping the implementation process in

check.

• A mechanism for assigning a measure of accountability with respect to the HMIS initiative.

• A mechanism for identifying areas where improvements or corrective action may be needed.


9

HMIS Project Evaluation

HMIS Project Evaluations may be performed in light of:

• Compliance• Analysis and Documentation• Monitoring• Improvement


10

HMIS

An HMIS depends on more than hardware and software alone:

Successful HMIS

Technology

Culture

Processes

People

HardwareSoftwareDatabasesNetworking

Defined structures

DirectionSupport from leadership

Measured results

Collaboration

Training

Motivation


11

Types of Evaluation

While evaluations of information systems tend to be either

• Outcome-Based (Analytical)OR

• Process-Based (Interpretive)


12

Outcome-Based Project Evaluation

• Define a measurable outcome, how to measure it, and when it will be measured

• Compare measured outcome to expectations


13

Process-Based Project Evaluation

• Uses an in-depth examination of the project• Considers the views of the various groups

involved in the HMIS implementation and use• Documents the issues, changes and major

decisions that may affect the HMIS’s project progression

• Considers project objectives and procedures• Asks “how and why” • May still use numbers


14

Effective HMIS Project Evaluation needs to be a hybrid of each

• Hybrid approach• Process • Outcome

• Context should determine the approach• Project status• Goals• Barriers• Participants• Stakeholders


15

Evaluation is Contingent on the Project’s Context

• HMIS has multiple objectivesJustifying and documenting AnalysisImproving services

• Organization with a focus on people, not technology

• Multiple and sometime contrasting views of whether and how to use HMIS


16

Evaluation Strategy

• Define baseline criteria against which outcomes and goals will be measured

• Select the methods by which data will be collected

• Determine a time horizon covered in the evaluation

• Develop/apply the protocols for data collection

• Define a point in time to conduct the evaluation

• Analyze and interpret the information

• Document results

• Develop specific recommendations


17

Defining Baseline Criteria

• There are three pieces here:• What are we working with?• What is the criteria?• How can we or are we going to explain/document progress?

• What are we working with, what is the starting point?• It is first necessary to document the framework under which

the HMIS project is being implemented.• This can be determined based on estimated totals and goals

for the project– Clients served– Participating agencies– HMIS users– HMIS levels of usage– Training program


18

Defining Baseline Criteria

• Identifying the criteria and expected outcomes by which progress can be determined. • Examples:

• Number of expected client records• Number of training sessions• Number of referrals made• Number of users trained

• Defining how will progress be documented.• Straightforward with outcome-based measures• With process-based aspects one can use:

• The project tasks • A deployment framework


19

Selecting Evaluation Methods

• No method is going to be perfect; but the method must be realistic

• Common evaluation methods• Service Monitoring• Usage Monitoring• Satisfaction Surveys• Qualitative Methods


20

Defining Evaluation Timeline

• Identifying the time periods for which data will be extracted or collected for the purpose of evaluation.

• Examples:• 3-year program with yearly evaluation• Ongoing yearly evaluation• One-time evaluation


21

Defining Protocols for Evaluation

• How will data be collected? • Who will be responsible for producing data sets? And in

what format?• What existing data sets or reports can be used?• What HMIS project implementation documentation should

be used?• What surveys should be applied?

• Consumer• Agency administrator• HMIS user


22

The Actual Evaluation

• Define a set time for an actual “visit” that will include:

• A review of progress• Demonstrations• Hand-off of documentation and data set or reports to the

evaluation team


23

Analysis and Interpretation

• The evaluation team conducts the analysis of results by:

• Objectively reporting on the numbers• Commenting on how the numbers fit/match the set

baseline criteria• Interpreting how progress has been made


24

Documentation

• Preparing a written report for public release.


25

Specific Recommendations from the Evaluation

• The evaluation can only be useful if it generates specific actions or strategies for improvement.

• The evaluation must include specific recommendations for action.


26

Who Should Conduct the Evaluation?

• Internal. Project management.• External. A project evaluator.• Appointed. A local evaluation team.

Evaluating Project IMPACTLake County, Illinois



Rob Anthony Lake County Department of Planning, Building, & Development

Lake County, Illinois


28

Lake CountyDemographics

• Population: 644,356• Median Household

Income: $66,973• Estimated that 387

people meet HUD’s definition of homeless at any given point-in-time

• 38% of homeless are persons in families

• 24% are chronically homeless

• 30% are veterans

HMIS Participation Individual’s

BedsFamily Beds

Emergency Shelter 67% 47%

Transitional Housing 37% 100%

Permanent Supportive

Housing100% 100%


29

ADVANCED SEARCH

Agency Name Search:

[Copyrights, disclaimers, email link to webmaster, other footer info appears here; logos above are links to sites]

What Is the Helping Hands Community INFO System?

How Does This Site Work?

Where Else Can I Find This Information?

About Us

BVM Olenti Helping Hands Provider Version

Food Services

Shelter Services

Entertainment

Child Care

Family Services

Elder Care

Disabled Services

Medical Services

Financial Services

Job Postings

Welcometo the BVM Olenti

Helping HandsCOMMUNITY INFOSystemfor Lake County, IL

Click on any of the category buttons above to initiate a subjectsearch, type in an agency name in the Agency Name Search box to look up a specific agency, or click on Advanced Search for other forms of searching.

VOLUNTEER / DONATIONS

The Provider Version is brought to you by the IMPACT project sponsored by:

Project IMPACT

• $560,000 Technology Opportunity Program (TOP) Grant

• HMIS• I&R Kiosks / Learning

Centers• Desktop I&R services• Reports Module


30

Homeless Management Information System

• 15 Programs• 75 Users• HMIS participants

include drop-in centers, emergency shelters, transitional housing, permanent housing, representative payee services, legal counseling, & housing placement services


31

Information & Referral Kiosks

• 9 public access kiosks located in the Lake County Health Department clinics & the local IL Department of Human Services office

• Searchable database of health & human services (in English & Spanish), jobs, entertainment, and dining as well as three interactive educational modules


32

Information & Referral Website

ADVANCED SEARCH

Agency Name Search:

[Copyrights, disclaimers, email link to webmaster, other footer info appears here; logos above are links to sites]

What Is the Helping Hands Community INFO System?

How Does This Site Work?

Where Else Can I Find This Information?

About Us

BVM Olenti Helping Hands Provider Version

Food Services

Shelter Services

Entertainment

Child Care

Family Services

Elder Care

Disabled Services

Medical Services

Financial Services

Job Postings

Welcometo the BVM Olenti

Helping HandsCOMMUNITY INFOSystemfor Lake County, IL

Click on any of the category buttons above to initiate a subjectsearch, type in an agency name in the Agency Name Search box to look up a specific agency, or click on Advanced Search for other forms of searching.

VOLUNTEER / DONATIONS

The Provider Version is brought to you by the IMPACT project sponsored by:


33

Report Module


34

Why Conduct An Evaluation?

Initial Reasons• External evaluation required as

part of the TOP grant• Determine whether IMPACT had

the desired impact• Provide objective data to justify

ongoing funding• Develop recommendations for

implementation• Determine whether the project

can and should be replicated elsewhere

Secondary Reasons• External evaluators brought a

high level of expertise, credibility, & experience

• Ability to understand how the project changed (or didn’t change) practices & protocols

• Forced the implementation team to address issues that we may have put off and assigned accountability for tasks

• Positive feedback encouraged the implementation team

• Helped objectively determine whether the team was on track


35

Evaluation Scheme

Barriers

Tools

Technology Applications

Processes

• Internet• Kiosk• Database Technology

• I&R Kiosk• I&R Desktop• I&R Learning• Case Management• Language Translation

• Individual Level• Agency Level• Government/ Funding Agency Level

• Increased Access to Services• Improved Planning and Resource Allocation• Increased Use of Aggregate Data

• Increased Number of Beneficiaries• Faster/Better Linkages to Services• Increased Access to Health Education• Increased Interagency Service Coordination• Improved Service Outcomes• Increased Social Capital• Enhanced Community Planning

Outcomes

• Infrastructure Development• Usage• Planning & Coordination• Data Aggregation• Analytical Capabilities• Project Development

Goals


36

HMIS Baseline Process

• Intake – in person or phone interviews, paper intake forms, computer

• I&R – staff compile their own lists, County directory updated every 2-3 years

• Referrals – fax, phone, or mail

• Outreach – street outreach & presentations

• Eligibility Assessment – paper forms, applications, and telephone

• Information Sharing – signed releases that are faxed or sent in mail

• Reporting – tabulating paper files or data pulled from internal computer files


37

Sample Focus Group Questions

1. Do you use the HMIS? Does your Agency? How is it being used? Please describe the processes supported with ServicePoint.

2. Does the use of the HMIS simplify or complicate reporting requirements in the short term, long term? In what way does it simplify or complicate matters?

3. Are clients formally communicated of how the HMIS is utilized? Where information does or does not go? What has been the response of the clients to the fact that their information is recorded on HMIS?

4. Does the presence of the HMIS affect a case manager’s ability to work with a client?

5. Does HMIS help you locate services more quickly or less quickly than before? How so?

6. Does the technology result in a different quality of services for the client in your estimation?

7. Do you think the technology helps or will help clients to better access transitional and permanent housing? Why do you say that?

8. How much training time did you receive? How much would you like?


38

HMIS Projected Outcomes

Number of records (consumers) in HMIS

• Year 1: None (Implementation)

• Year 2: 150 records

• Year 3: 1,000 records

Supported Services

• Year 1: None (Implementation)

• Year 2: 3-5 programs conducting CM, I&R, reporting, and partial community data

• Year 3: 10 programs conducting CM, I&R, reporting, referral tracking, interagency CM, & full community data


39

Comparison of Projected Outcomes vs. Actual Results

10001265

0

500

1000

1500

ProjectedRecords

Actual Records

Clients Served (Year 3)


40

Community Planning Baseline Process

• Needs Assessment – facilitated by CoC

• Data Gathering, Analysis, & Aggregation – PIT counts, CoC meetings, surveys, focus groups, & hearings

• Resource Allocation – CoC recommendations to the Community Development Commission

• Evaluation & Monitoring – Planning Department monitors grants mostly relying on manually recorded data that cannot be aggregated electronically

• Service Coordination – primarily done through CoC and Homeless Coalition Meetings

• Participation – approximately 10 active agencies


41

Reasons for Homelessness

Domestic Violence

9%

Eviction

9%

Mental Health

12%

Loss of Income

9%

Alcohol Abuse

10%

Disagreement

5%

Substance Abuse

33%

Other

13%


42

Consumer Baseline Satisfaction

• Lack of knowledge regarding the range of services available and how they can be accessed

• Confusion regarding program eligibility requirements

• Insufficient availability of key services

• Embarrassment, language and literacy barriers

• Lengthy and circuitous referrals

• Long delays in accessing requested services

• Lack of service coordination

• Repetitious intake procedures when going from agency to agency


43

Instrument

Period Media

Date

Agency Administrator Survey

October 31st Electronic or paper October 31st

Agency Staff Survey


Client Survey: I&R


Client Survey: HMIS October 31st Electronic or paper October 31st

Case Management Reports Quarterly Electronic October 5th, January 5th,April 5th, July 5th

Learning Center Kiosk usage reports

Quarterly Electronic October 5th, January 5th,April 5th, July 5th

Healthy Touch application usage reports


I&R: Provider version usage reports


I&R: Web Version usage reports


Documentation, Quarterly Progress Report


Documentation, Copies of agreements, policies, standard,procs.

Every six months Electronic

January 5th,July 5th

System documentation (e.g. specs, design, modification rpts)

Every six months Electronic January 5th,July 5th

Schedule of Data Collection


44

Examples of Evaluation Team Recommendations

• Focus on developing broad-based participation to better realize the benefits of the case management application

• Create two levels of minimal data requirements

• Create a custom screen for minimal data requirements

• Hold more regular user groups

• Develop a support mechanism for HMIS

• Develop a set of standard operating procedures for participating agencies

• Develop policies regarding the release of data sets

• Further utilize paper version of HMIS intake form

• Examine contextual issues


45

Final Evaluation

Outcome/Process Goals Met• Improved access to

services (decreased barriers at individual level)

• Integrated information and referral with HMIS

• Unduplicated community level data on homelessness

• Effective infrastructure development process

• Successful interface between HMIS & other Management Information Systems

Outcome/Process Goals Not Met

• Improved efficiencies in service delivery

• Comprehensive use of CM across the network of participating agencies

• Data usage for policy making


46

Policies, Reports, and Forms

Annual evaluation reports available at:www.mccormack.umb.edu/csp/csp_hmis.jsp#LakeCounty

Project IMPACT policies, agreements, & forms available at:www.co.lake.il.us/planning/community/project_impact_forms.asp

Contact information:

Rob AnthonyLake County Department of Planning, Building, & Development18 North County StreetWaukegan, IL [email protected]

http://www.mccormack.umb.edu/csp/csp_hmis.jsp#LakeCounty

http://www.co.lake.il.us/planning/community/project_impact_forms.asp

mailto:[email protected]