DRAFT Evaluation Plan for Hostels to Homes (HTH ... to...DRAFT Evaluation Plan for Hostels to Homes Table of Contents 1.0 Overview 1 1.1 The Hostels to Homes: Emergency Hostel Redirection

DRAFT Evaluation Plan for Hostels to Homes (HTH): Emergency Hostels Redirect Pilot Initiative

October 27, 2006

Developed in collaboration with MCSS Policy Research and Analysis Branch by:

Centre for Research and Education in Human Services 73 King Street West, Suite 300 Kitchener, ON N2G 1A7

DRAFT Evaluation Plan for Hostels to Homes

Table of Contents 1.0 Overview ..........................................................................................................................1

1.1 The Hostels to Homes: Emergency Hostel Redirection Pilot Initiative .............................. 1 1.2 The Evaluation Plan........................................................................................................... 1

2.0 Components of the Evaluation Framework ..................................................................3

2.1 Overview of Framework Components ................................................................................ 3 2.2 Introduction to Logic Model Template................................................................................. 4 2.3 Introduction to Evaluation Questions Table ........................................................................ 5 2.4 Introduction to Data Gathering Framework......................................................................... 6 2.5 Introduction to Methods Framework ................................................................................... 7

3.0 Implementation of Evaluation Plan ...............................................................................8

3.1 Local and Provincial Evaluation Roles................................................................................ 8 3.2 Methods Summary.............................................................................................................. 8 3.3 Research Modules............................................................................................................ 10 Research Modules Table........................................................................................................ 11

4.0 Appendices....................................................................................................................15

Appendix A: Logic Model Template ........................................................................................ 15 Appendix B: Evaluation Questions Table ............................................................................... 16 Appendix C: Data Gathering Framework................................................................................ 19 Appendix D: Methods Framework .......................................................................................... 34

Table of Contents


DRAFT Evaluation Plan for Hostels to Homes (HTH)

1.0 Overview 1.1 The Hostels to Homes: Emergency Hostel Redirection Pilot Initiative Through the “Hostels to Homes: Emergency Hostel Redirection Pilot Initiative”, the Ministry of Community and Social Services (MCSS) is providing funding for pilot projects in six municipalities in order to:

• stabilize the use of emergency hostel services and return its service delivery to its original intent: short term and infrequent use; and

• provide municipalities (as service system managers) with the flexibility to use/reallocate existing cost-shared emergency per diem funding to create innovative cross-sectoral (housing, health and social) supports/strategies to adequately and appropriately house and support homeless persons.

MCSS is funding the six pilots for 18 months beginning October 2006, though the sites are expected to begin a process of phase-out of pilot funding by 12 months. MCSS expects that each pilot site will achieve one or more of the following outcomes:

1) ensure emergency hostel residents have access and attachment to affordable, stable, transitional or permanent accommodation

2) show savings to both the municipality and the province by transitioning people out of emergency hostels and moving them into sustainable affordable accommodation

3) reduce the length of stay/usage of emergency hostel beds by chronic bed users within the timeframe of the pilot

4) enhance the coordination and integration in the provision of housing and other support services

1.2 The Evaluation Plan In order to determine whether the pilot initiative has successfully met its objectives, the ministry will conduct an independent, ongoing evaluation of the pilot. The results of the evaluation will be used to inform MCSS decisions about whether the pilot should be expanded to include additional sites, and about the merit of developing a wider policy concerning the use of funding for emergency hostel services and integrated social service delivery models. The purposes of the evaluation plan are

• to provide a framework for gathering and analyzing information that describes the implementation of the 6 pilot projects participating in this initiative (process evaluation), and

• to provide a framework for evaluating the success of the overall pilot and each pilot site in achieving the ministry’s and the CMSM’s stated objectives (outcome evaluation).

The evaluation plan is intended as a tool both for evaluations at each site, and for the coordination of evaluation focus for roll-up across the six sites. The plan is intended as a detailed but flexible framework for evaluation, based on the following principles for implementation:

Consultation with Advisory Committee: The evaluation framework was developed by evaluation consultants and MCSS provincial staff in consultation with the Advisory Committee.

Overview

1


The Advisory Committee will continue to monitor the evaluation over the course of the pilot to ensure its relevance and usefulness to local sites who will be asked to participate in it and, possibly, to implement evaluation recommendations. Adaptability to local context and needs: The evaluation framework is intended to be adapted to the needs and contexts of each pilot sites through consultation with each site. Consistency across sites: While allowing for some variation at the local level, the evaluation framework is also intended to ensure consistency across the pilot sites in order to make possible an analysis of outcome for the overall provincial pilot. Integration of different levels of questions: The evaluation framework integrates evaluation questions that can be answered for each site with provincial-level questions that can only be answered through an analysis of data from all sites.

Overview

2


2.0 Components of the Evaluation Framework 2.1 Overview of Framework Components The evaluation framework is composed of a number of interrelated documents that, used together, can guide a Hostels to Homes evaluation through the sequence of building a local logic model and generating approaches to measurement, design, data collection and analysis. There are four main documents:

1. The Hostels to Homes Logic Model Template

2. A list of Key Evaluation Questions (and a menu of useful sub-questions)

3. The Data Gathering Framework

4. The Methods Framework These documents roughly correspond to the steps an evaluation would follow in building a comprehensive plan. The flowchart below outlines the central documents that make up the framework, their basic function, and how they interrelate and build on one another. More details on each component are described in subsequent sections. Basic Function Document How the documents interrelate

Framework

The Logic Model Template describes major program components, planning steps, example activities, outputs and short- and long-term outcomes for H2H projects

Logic Model

Template

Specifying what you want to know about how the program is working and its impact

Evaluation Questions

Table

Generating strategies on how to go about answering your questions

Data Gathering

Summarizing your data collection methods for each source of information

Methods Framework

The Logic Model Template is organized into four sections. Each section has different types of evaluation questions associated with it. These question groups become the main elements of the Evaluation Questions Table

Groups of Evaluation Questions are listed in reference to planning and design, project management, implementation and outcomes, etc.

Groups of Evaluation Questions become the main elements of the Data Gathering Framework

The Data Gathering Framework provides indicator and data needs, data sources, methods and comments, in relation to each evaluation question

Data sources (people, documents, etc.) from the data gathering framework become the main elements in the Methods Framework

Understanding what the program looks like and how it works

The Methods Framework summarizes what data is needed, how it will be obtained and when, for each data source.

Framework Components 3


2.2 Introduction to Logic Model Template The Logic Model Template (Appendix A) serves as the core resource that will help structure and guide local Hostel to Homes evaluations. The template achieves this central function because it captures, in visual form, the main components that make up any Hostel to Homes program and its evaluation. The logic model template has at least 3 main purposes:

1. To describe in general terms the overall Hostel to Homes initiative 2. To guide the creation of Hostel to Homes logic models at each pilot site 3. To provide a basis for planning the various components of local process and outcome evaluations

The template, which runs in sequence from top to bottom, is composed of nine sections. They are as follows: Provincial process objective: A general statement of what the Hostel to Homes Pilot Initiative does. This is essentially the mandate of the Hostel to Homes provincial project and will be consistent across all participating local sites. Program components: These are the main activity components of the Hostel to Homes project. Program Components are general in nature – broad categories that capture the range of things that are typically done at local sites. These components are

• Community Service Coordination: Includes activities that attempt to create service partnerships, and other systems level capacity building

• Intake, Screening, & Planning: Includes activities aimed at establishing criteria for eligibility, and program and support planning with participants

• Individual Case Management & Support: Includes activities to support participants in an variety of services (e.g., housing, employment, counselling, etc.)

All local Hostels to Homes projects engage in activities that fall within these three broad components. Inputs: Inputs are program resources. Inputs help answer questions about how the project is designed and implemented. Examples could include “# of staff”, “# of staff hours”, “program space”, “funding dollars”, etc. Planning and design steps: Each program site needs to some preparatory work in order for direct service activities to be properly implemented. For example, staff training (a planning step) may need to happen before staff can provide appropriate counselling (a program activity). The template provides examples of possible planning and design steps. Pilot sites would list their own planning and design steps in their local-level logic models, by drawing on, modifying, or adding to these examples. Implementation of activities: This section of the model lists examples of activities that describe programs being delivered at each local pilot site. These activities are consistent with the overall program components but are more specifically worded to capture what is happening at the local pilot site level. For example, the activity “Screen pilots for eligibility in the program” is an example activity that falls under the component “Intake, Screening, and Planning” (and, as can be seen, this activity example is listed beneath the relevant component in the template). As with planning and design steps, the template merely lists possible examples. Local sites will establish the specific wording of their own activities when creating their own models. Outputs: Outputs are measures of activities that have been delivered and help answer questions about program process. For example, if an activity is to provide counselling sessions, an output would be the number of counselling sessions that were actually provided. Other examples include attendance at meetings, number of referrals, number clients receiving life skills supports, etc. Short-term outcomes: Short-term outcomes represent the first meaningful and important changes (benefits) experienced by program participants that follow directly from program activities. The short-term outcomes provided in the template are examples that follow, hypothetically, from the range of activities occurring in

Framework Components

4


Hostels to Homes pilots (e.g., “Improved life skills). Each site must develop and refine their own short-term outcomes that best reflect the presumed impact of the project activities. Long-term outcomes: Long-term outcomes represent the longer-term and more durable changes experienced by program participants and which follow from the attainment of short-term outcomes. The long-term outcome objectives listed in the model are those articulated by MCSS in the Hostels to Homes Pilot Guidelines, and should be shared amongst the Hostels to Homes pilot sites. Goals: Goals represent the long-term vision and mission of the Hostels to Homes project – to stabilize hostel service use so it is used infrequently and temporarily, and to provide overall savings to municipalities and the province. In constructing site evaluations, each pilot site can begin by drawing upon the content of the Template to produce their own logic models. Each site-level logic model, will be able to visually trace the life of program using these sections. Evaluation Question Groupings: Along the right-hand side of the template are 4 groups of questions. These question groups are listed alongside corresponding elements of the program:

Group A: Questions about the provincial-level process objective (design and context of the pilot initiative). These questions help evaluate the creation and context of the initiative as a whole and at a system-policy level

Group B: Questions about the design and planning phase (e.g. project rationale, design, & preparation). These types of questions help evaluate the preparation and development phases of the program (Inputs help in this regard).

Group C: Questions about implementation of activities (what happened? to, with, and for whom? when?). These questions help evaluate whether or not activities were delivered or implemented as planned and if not why not. These questions are strongly associated with “process evaluation”

Group D: Questions about program outcomes and links to process (what was the impact? why and how did that impact occur?). These questions help evaluate the short- and long-term impacts of the project, and understand why they occurred or did not occur.

These groups of evaluation questions help link the logic model, which visually describes the program and the evaluation, to actual evaluation methodology. The first document the Logic Model Template links to is the “Evaluation Questions Table”. 2.3 Introduction to Evaluation Questions Table The Evaluation Questions Table (Appendix B) lists the evaluation questions by the groups listed in the Logic Model Template. Under each question group is a list of several “key evaluation questions”. In turn, under each key evaluation question is a menu of possible sub-questions that evaluators can use to structure their site-level evaluation. The following reprises the main content of the Evaluation Questions Table. Group A: Provincial-level questions about design/context of pilot initiative.

Key questions: • What is the background history and context of the Hostels to Homes initiative?

Group B: Questions about planning and design.

Key questions: • What were the main tasks and considerations of project planning? • What management and administrative structures, processes, and issues need to be

considered? • What are the governance structures of the project?


5


• What personnel issues need to be considered to properly plan the project? • What is the funding model that is being used for the project(s)?

Group C: Questions about program implementation.

Key questions: • What is the rationale and method for participant recruitment? • How were program activities implemented and delivered? • What changes were made to the implementation plans, and why?

Group D: Questions about program outcomes.

Key questions: • To what extent did the pilot contribute to the achievement of intended short- and long-term

participant outcomes? • What are other additional participant outcomes were observed? • To what extent were participants satisfied with different aspects of the project? • To what extent were there impacts on the service system (efficiency, partnerships,

integration)? • What were the costs and cost-savings of the project?

The Evaluation Questions Table also appends a series of “Overarching Research Questions for Each Pilot and Across Pilots”. These questions are broader in scope and typically require synthesis of multiple data types obtained from answering an array of questions above. The number of questions and sub-questions in this document is extensive. These key questions are central and can be answered by gathering a wide variety of data. Sub-questions have been formulated to stimulate strategic data gathering and to give pilots some ideas of how to proceed. This list is also not exhaustive. Individual pilots may wish to pose additional questions that suit their own context. 2.4 Introduction to Data Gathering Framework The Evaluation Questions Table is specifically linked to the third document within this framework, the Data Gathering Framework (Appendix C). The Data Gathering Framework is another table that takes the evaluation questions and organizes them in reference to suggested evaluation methods. At this stage, the framework becomes more concrete because it links evaluation questions to specific strategies to begin answering them. For each question the table contains the following columns of information: Indicators/data needs: These are the “data pieces” necessary to answer an associated question. Common examples are inputs (e.g., # of staff hours that went into an activity), outputs (# of people served), responses from people (e.g., scale ratings, other survey data, interview content, etc), or archive information (e.g., policy statements). This is the “raw information” that needs to be interpreted to answer the evaluation question. Data sources: This column lists the data sources that need to be pursued to gather the indicator/data needs. Data sources are the people that provide the information (e.g., staff, participants, management, the broader community, etc.) and written documents and other sources (e.g., project proposals, written review of best practices). Methods: The “Methods” column provides more specific information on how, exactly, the indicator/data needs will be systematically gathered from the data sources. For example, the method to gather information on best practices (the indicator/data need) requires a document review of relevant books and articles (the data sources). Or, the method to gather self-reports of satisfaction (the indicator/data need) from participants (the data source) requires administering a brief survey of scale items.


6


Comments: The last column allows space for comments on the preceding content. For example, it could provide information on how and where to gather information, potential barriers, or other design issues (e.g. potentially difficulty in scheduling interviews). In summary, this table serves the function of pulling together all the main questions that need answering in the evaluation and pairing them with concrete information plans – what information is needed, from what source, and how that information will be acquired. 2.5 Introduction to Methods Framework The Methods Framework (Appendix D) adds an additional table to the evaluation framework that helps to clarify and organize the plan in reference to common data sources. Data sources are listed down the left hand column and are paired with the following columns:

• Indicators/data needs (What data needs to be gathered?) • When will this data need to be collected? • Methods (How will this data be collected?) • Comments

The Methods Framework allows the reader to more readily plan data collection in reference to each data source and, furthermore, to become more specific about the methods that would be used in each case (such as specifying when data collection should happen).


7


3.0 Implementation of Evaluation Plan 3.1 Local and Provincial Evaluation Roles Conducting the evaluation across the six pilot sites will require a balance of central coordination with local variability to facilitate across-site roll-up while allowing for responsiveness and relevance to local contexts. Some evaluation activities will take place at the local level, while others may be conducted centrally across sites. All evaluation activities will be managed by a provincial evaluation team, with targeted responsibilities for site evaluations. Provincial Evaluation Responsibilities

Development and Tool Design: Review of relevant research literature •

•

•

•

•

•

• •

• •

• •

•

• •

Development and implementation of a standardized quarterly reporting form for use by pilot sites (see attached list of indicators for this tool) Development of protocols for participant intake/exit, surveys, key informant interviews, and focus groups (see attached indicators) Coordinating site evaluation plans and program logic models to ensure consistency

Provincial data gathering

Interviews with MCSS staff and members of the Hostels to Homes pilot site selection committee (at baseline) Interviews with pilot site managers (at baseline and at project completion)

Completion of provincial evaluation report

Analysis of all relevant data, including economic data Completion of provincial evaluation report, presentation of findings, and incorporation of feedback from Advisory Committee

Ongoing support to pilot sites regarding tracking & evaluation issues

Facilitate the development of a program logic model for the pilot project Adapt provincial data collection tool protocols to reflect local conditions and issues through consultation with pilot sites Design additional local data collection tools if needed Assist staff in completion of quarterly report

Local data gathering

Interviews with pilot site steering committees (if applicable), partners, and other local service providers at various points throughout the pilot

Completion of pilot site evaluation reports

Analysis of all relevant data, including economic data Completion of local reports, presentation of findings, and incorporation of community feedback

3.2 Methods Summary Answering the Evaluation Questions will require a mix of qualitative and quantitative methods. The following are methods that are included in the evaluation plan. The actual mix of methods used will depend on the budget and scope of the evaluation, the priority given to longer-term outcome measurement, local-level logistics, and the source of the data being gathered.

Implementation 8


The methods themselves will all be familiar to the researcher(s) recruited to conduct the evaluation:

• Document review • Literature review • Local logic model process and

measurement review • Reporting and standardized tools

development

• Participant intake, exit, and follow-up interviews

• Focus Groups • Key Informant Interviews • On-line survey

Analysis of the Evaluation Questions suggests a range of data sources that should be integrated into the research design. The following lists these data sources and summarizes the related methods: Project documents document review (Hostels to Homes guidelines, pilot proposals, pilot

reports, etc.)

Research literature literature review (impact of housing stability on costs in other social services and programs; economic model for estimating savings generated by changes in service use)

Economist economic analyses and modelling MCSS provincial staff key informant interview or focus group

All professionals involved in the pilot site (pilot site manager, pilot site steering committee members, pilot site partners)

logic model process and measurement review follow-up data gathering:

o focus groups o individual interviews with key informants o on-line survey

Pilot site manager (project manager or coordinator for the pilot site)

project reporting guidelines/template key informant interview near beginning of pilot (implementation data) key informant interview at pilot end (complex implementation and

needs data, qualitative outcome data)

Project participants intake and exit interviews1 (CMSM administered) in-depth qualitative interviews follow-up interviews

Pilot site steering committee (where applicable)

focus group in-depth interviews with key informants on-line survey (quantitative implementation and outcome data)

Pilot site partners

focus group in-depth interviews with key informants on-line survey (quantitative implementation and outcome data)

Other Key Informants (where applicable)

in-depth interviews with key informants who do not fit the categories above

1 “Exit” interviews are to be conducted when the participant leaves the pilot, whether at the end of the pilot or sooner

Implementation 9


3.3 Research Modules The Research Modules Table (below) presents a range of evaluation methods that could be employed for the Houses to Homes evaluation. The assumption underlying the table is that most of the methods proposed will be used in the evaluation, but that some will not. The mix of methods to use depends on factors such as evaluation budget, the extent of site evaluation appropriate to MCSS priorities, and the overall scope of evaluation that MCSS would like to conduct. The purpose of the Research Modules Table is to assist MCSS in decision-making about the evaluation by providing an overview of potential methods, along with their implications for evaluator time and ability to answer the Evaluation Questions. Time estimates provided in the table are estimates only and are intended as guidelines for MCSS deliberations on the scope of evaluation desired, and for the assessment of bids from evaluation firms. Time estimates for each of the research modules MCSS decides to implement can be expected to vary to some degree among evaluation teams bidding on the contract.

Research Modules 10


Research Modules Table 11

Research Modules Table

Method Tasks Time

estimate: Comments senior junior

Document & tracking data review

Review and code OW and shelter tracking data, and data from project documents (H2H guidelines, pilot proposals, pilot reports, etc.) 1 3

Required: essential for answering questions from group A, B, C and D

Follow-up at 12 months post pilot

Gather and review/code OW and shelter tracking data 3

Strongly recommended: Improves understanding of longer-term impact of pilots on basic indicators such as average length of stay in hostels (Group D)

Literature Review

Review literature related to impact (including cost-savings) of housing stability on social services and programs, and economic models for estimating savings generated by changes in service use 1 2

Required: Needed to answer cost-savings questions from Group D questions about outcomes.

Orientation and evaluation support for pilot sites

Ongoing dialogue between provincial evaluation teams and pilot sites; one half-day orientation meeting 3 6 Required

Logic model process (1 per site)

Draft logic model, stakeholder consultation, finalize logic model 6 9 Required

Review and consolidate program logic models

Roll up logic models at provincial level 3 3 Required

Local data gathering plan Local stakeholder consultation on research questions and indicators

6.5 9

Recommended: Ensures local relevance of indicators, provides opportunity to integrate locally-useful research questions, and builds buy-in for evaluation process

Standardized tools (not including survey instruments)

Participant intake/exit protocols

Design of standard participant intake/exit interview items to be integrated into intake/exit procedures administered by CMSM staff in all sites; consultation with pilot sites 3 6

Strongly recommended: enables consistent gathering of pre/post test qualitative & quantitative data for answering Group C & D research questions (program implementation and outcomes). Assumes implementation of intake/exit interview by local site staff; consultation is important for ensuring local relevance and feasibility

Project reporting guideline/template

Devise reporting template in consultation with pilot sites

3 6

Required: Consultation with local sites is recommended as a means of ensuring relevance and feasibility of reporting to local site

Design interview protocol Organize and conduct interview

Focus group or key informant interview with H2H pilot selection committee at beginning of pilot Code and analyze data 0.5 1

Required for answering Group A provincial-level questions

Pilot site manager Design interview protocol 0.5 Organize and conduct interview 1 3

Implementation interviews with local staff leads at beginning of pilot (1 per site) Code and analyze data 3

In-depth reporting or interviews required. Interviews strongly preferred to reporting for data gathering. Conduct near the beginning of evaluation process to answer Group B Planning & Design questions not answered in the project proposal


Method Tasks Time


Design interview protocol 0.5 Organize and conduct interviews 1 3

Follow-up implementation interviews at pilot midpoint (1 per site)

Code and analyze data 3

Recommended: Conduct midway through the pilot period to answer Group C questions about program implementation, implementation challenges, and planning and design changes made (or required) to respond to those challenges

Design interview protocol 0.5

Organize and conduct interviews 1 3

Outcomes interviews with pilot site managers at pilot completion (1 per site)

Code and analyze data 3

In-depth reporting or interviews required. Interviews strongly preferred to reporting for data gathering. Conduct at end of pilot period to answer Group C questions about program implementation, and Group D questions about program outcomes; some of this data could be gathered through project-end reporting. Interviewing ensure completeness of data and minimizes reporting burden.

Design interview protocol 0.5 Follow-up interviews at 12 mos. post pilot Conduct and analyze interviews 3

Strongly recommended: Improves understanding of longer-term impact of pilots (Group D)

Participants Intake/exit interviews Analyze data gathered by CMSM staff 3 8 Required

Develop sampling criteria/strategy 1.5 Design interview protocol 1 Organize and conduct interview 6

In-depth interviews with sample at pilot midpoint (estimate for 5 interviews per site) Code and analyze data

3.5

6

Recommended: Conduct midway through the pilot period to answer Group C questions about program implementation, implementation challenges, and planning and design changes implemented (or required) to respond to those challenges and achieve outcomes

Develop sampling criteria/strategy 1.5 Design interview protocol 1 Organize and conduct interview 6

In-depth interviews with sample at pilot completion (estimate for 5 interviews per site) Code and analyze data

3.5

6

Strongly recommended: Enables gathering of in-depth qualitative data for answering questions about implementation (Group C), participant stories, and the conditions/participant characteristics surrounding achievement (or not) of program outcome objectives (Group D).

Follow-up interviews at 12 mos. after exit from pilot (estimate for 5 interviews per site)

Organize, conduct interview, and analyze (using post-test items from intake/exit protocol) 2

Recommended: Enables better understanding of longer-term impact of pilot on individual housing journey (Group D) 12

Pilot site steering committee (where applicable) Design interview protocol .5 2

Organize and facilitate focus groups 3 6 Focus group with site steering committee at pilot midpoint (1 per site where applicable) Code and analyze data 3 6

Recommended : Conduct midway through the pilot period to answer Group C questions about program implementation, implementation challenges, and planning and design changes implemented (or required) to respond to challenges and achieve outcomes

Design interview protocol .5 2

Organize and facilitate focus groups 3 6 Focus group with site steering committee at pilot end (1 per site where applicable) Code and analyze data 3 6

Required: the most efficient means of gathering perspectives from governance-level stakeholders relevant to questions in Groups C & D about program implementation and outcomes



Method Tasks Time


Design interview protocol

.5 1

Organize and conduct interview .5 3

In-depth interviews with key informants at pilot midpoint (estimate based on 2 per site)

Code and analyze data .5 2

Recommended where implementation challenges have been identified and further info is required for addressing them. Helps to answer Group C questions about program implementation and implementation challenges, and builds consensus and buy-in for making necessary planning and design changes to respond to challenges and achieve outcomes


.5 1 Organize and conduct interview .5 3

In-depth interviews with key informants at pilot completion (estimate bon 2 per site)

ased


Recommended for gathering of more specific, nuanced, in-depth data from key stakeholders (without the influence of others) for more thorough understanding of context and challenges involved in implementing the project and achieving outcomes (Group C & D questions)

Design and pilot survey 2 5 Implement survey/trouble shoot 3.5

On-line survey at pilot completion

Analyze data 0.5 3

Strongly recommended for efficient and consistent gathering of quantitative (and limited qualitative) data across all sites, and enables quantification of Group C & D questions (implementation and outcomes)

Follow-up at 12 months post pilot See "pilot sites follow-up" below

Strongly recommended. Project partners

Design interview protocol .5 3 Organize and facilitate focus groups 3 6 Focus group at pilot

midpoint (1 per site) Code and analyze data 3 6

Required: Efficient means of gathering qualitative data from those who are directly involved in implementing the pilot (Groups C & D questions)

Design interview protocol .5 3

Organize and facilitate focus groups 3 6 Focus group at pilot

completion (1 per site)

Code and analyze data 3 6

Recommended: Conduct midway through the pilot period to answer Group C questions about program implementation, implementation challenges; generates insights and builds consensus and buy-in for making planning and design changes required for achieving outcomes


.5 1


In-depth interviews with key informants at pilot midpoint (estimate based on 2 per site)


Recommended where implementation challenges have been identified, and further info is required for addressing them. Helps to answer Group C questions about program implementation and implementation challenges, and builds consensus and buy-in for making necessary planning and design changes to respond to challenges and achieve outcomes


.5 1


In-depth interviews with key informants at pilot completion (estimate based on 2 per site) Code and analyze data .5 2

Recommended for more specific, nuanced, in-depth data from key stakeholders (without the influence of others) for more thorough understanding of context & challenges in implementing the pilot and achieving outcomes (Group C & D questions)



Method Tasks Time


Design and pilot survey 2 5 Implement survey/trouble shoot 3.5

On-line survey at pilot completion

Analyze data 0.5 3

Strongly recommended: Enables efficient and consistent gathering of quantitative (and limited qualitative) data across all sites, and enables quantification of Group C & D questions (implementation and outcomes)

Follow-up at 12 months post pilot See "pilot sites follow-up" below

Strongly recommended. Other key informants (where applicable)

Design interview protocol 0.5 0.5

Organize and conduct interview 0.5 2

In-depth interviews with key informants at pilot completion (estimate for 2 per site)

Code and analyze data 0.5 2

Recommended where relevant key informants exist. May not be relevant in all sites. Allows for gathering of specific, nuanced, in-depth data from key informants external to the project who nonetheless have valuable perspective to contribute (e.g., service providers, community leaders, researchers, etc.). Could contribute more thorough understanding of context and challenges involved in implementing the project and achieving outcomes (Group C & D questions)

Design focus group protocol 0.5 1 Pilot sites follow-up focus group (all relevant professionals) at one year post pilot (1 per site) Organize and conduct focus group 3 6

Strongly recommended: Enables retrospective understanding of longer-term impact of pilot from a range of professionals involved in the pilot site; intended to include both direct project partners and, where appropriate, a sample of key community service providers (Group D)

Reporting

Evaluation Report: local Synthesize data and daft evaluation report 6 18

Strongly recommended: Individual evaluation report for each site.

Evaluation Report: provincial Roll-up site data, and integrate individual evaluation reports into overall pilot evaluation 3 5

Required: Evaluation report on overall pilot initiative, with some site-specific findings.

Follow-up report at one year post pilot (one report for all 6 sites)

Synthesize data from all follow-up data gathering activities 3 5

Strongly recommended: Improves understanding of longer-term impact of the pilot initiative (Group D)

TOTAL Days 68 28212 0



4.0 Appendices Appendix A: Logic Model Template (following page)

Logic Model Template 15

Plan

ning

and

de

sign

ste

ps(e

xam

ples

)In

puts

Prog

ram

co

mpo

nent

s(e

xam

ples

)

Impl

emen

tatio

n of

act

iviti

es

(exa

mpl

es)

Out

put

(exa

mpl

es)

Shor

t-ter

m

outc

omes

(exa

mpl

es)

Long

-term

ou

tcom

es

Community Service Coordination

Intake, Screening, &

PlanningIndividual case management & support

• Establish selection criteria and screening process

• Create individual screening and planning tools

• Improved clarity of service system roles

• Reduced system gaps & service barriers

• Establish service protocols, referral processes• Review local housing options• Develop support model• Create planning tools for independent living

Relevant to all program activities: Determine staff & partners’ roles, governance structures, required resources & materials, project component timelines, budget allocations, etc.

• Environmental scan of community service partners, service gaps

• Local needs assessment • Develop coordination plan

Inputs could include program funding, # of staff, hours, program space, products necessary for activities, service protocols, best practice resources, community information & demographics, etc.

• Establish project advisory committee

• Create project service & referral structure with partners

• Screen clients for eligibility for program

• Develop transition plan with clients

• Provide life skills training• Provide employment supports • Monitor and support clients during pilot• Develop housing plans and supports with each client• Develop independence plans with clients• Monitor clients post pilot

• Meetings of project advisory committee

• Service agreements created

• screenings & eligible clients• transition plans completed

• referrals to community services

• clients with independence plans

• monitored clients post intervention

• Strengthened participation of clients in search for more stable, affordable housing

• Improved life-skills• Improved

employability (skills, resources)

• Improved access to services and support for transition to suitable housing.

• client confidence in being able to live independently

Enhanced coordination and integration in the provision of housing

and other support services

• Reduction in the length of sty/usage of emergency hostel beds bychronic bed users

• Increased access to affordable, stable housing• Reduced incidences of re-entry into the hostel system

Overall savings to the municipality and the province

Stabilize the use of emergency hostel services and return its service delivery to its original intent: short term and infrequent use

(Group B)Questions about the planning and design steps (e.g. project rationale, design, & preparation)

Inputs will help answer certain questions about design and implementation

(Group C)Questions about implementation of activities (what happened? to, with, and for whom? When?)

Outputs will help answer certain questions about program process

(Group D)Questions about program outcomes (what was the impact? Why and how did that impact occur?)

• clients with life skills supports• clients with employment supports• clients with housing plans

(Group A)Questions about the provincial-level process objective (design/context of the overall pilot)

Evaluation Question

Groupings

• Provide municipalities with the flexibility to create innovative cross-sectoral (housing, health and social) supports/strategies to adequately and appropriately house and support homeless persons

Prov

inci

al

proc

ess

obje

ctiv

e

Goa

lsLogic Model Template: Hostels to Homes Evaluation Framework


Appendix B: Evaluation Questions Table Group A: Questions about the provincial-level process objective (design/context of pilot initiative) Project context: Provincial • What are the provincial goals/outcomes for the project • What is the funding model • What were the rationales for developing the pilots Group B: Questions about planning and design (e.g., project rationale & design, management, administration, preparation, etc.) Project Planning • What size is the homeless population within each pilot community? • What is the number of emergency hostel spaces within each pilot community? • How many emergency hostel spaces are targeted for the pilots? • What are annual expenditures on emergency hostels? • How was the project concept developed?

Who are the local partners for each pilot Who was involved in the development of the project concept? What was the approval process

for the project design at the local level? • On what knowledge is the project design based?

What best practices informed the project design? Did a needs assessment inform the project design? What were the findings of the needs

assessment? What situation analyses informed the project design?

Project design • What was the proposed design that was intended for implementation?

(number of spaces, administrative structure, staffing, criteria for participants, delivery model? Project management and administration • Who are the stakeholders playing a role in the project, and what role(s) did they play? • Is there clarity and consensus among key stakeholders on the roles they play? • How is the project administered? • What is the administrative structure of the project?

What administrative processes (e.g., planning, organizing, data gathering, reporting requirements, budgeting, processing clients, etc.) are in place for the pilot?

• Were the administrative processes/structures created for the pilot or did they already exist? • How efficient were the administrative processes? (data gathering and reporting requirements,

planning, etc.)? • How could the administrative model be improved?

What challenges did staff find in administering the pilot and how were the challenges addressed? Are there outstanding challenges?

Project governance • What structures and processes are used to govern the project?

Which, if any, stakeholders are involved in project governance? What are the accountability and reporting expectations?

What was the approval process for implementing the pilot • How effective were the governance structure and processes in supporting the project? • How could this governance model be improved?

What are the barriers to its effectiveness? Personnel • What personnel were involved in the implementation of the pilot?

Evaluation Questions Table 16


What positions do they hold? What training (if any) is provided for pilot staff? • Was staffing sufficient and appropriate? Project funding • What is the local funding model? • How did the funding of the pilot align with funding for existing business processes? Group C: Questions about implementation of activities Participant recruitment and retention What criteria and process were used to select individuals/households for the pilot project? •

• •

• • •

•

•

How appropriate were the participant selection/recruitment criteria and processes when implemented? Were the recruitment criteria appropriate? Was the process for recruiting participants efficient

and carried out in a timely manner? How could the criteria and processes for recruitment be improved? What were the characteristics of the individuals/households included in the pilot (e.g., family/individual, employment status/history, vulnerabilities [MH, DD, etc.], housing history [# and length of stays in hostel], etc)”? How long did participants remain in the pilot? What was the attrition rate? What were the causes of attrition (if any)? What proportion of participants remained in the pilot throughout the length of the pilot?

Why did participants drop out of the project? What were the characteristics of the participants that remained/dropped out of the pilot? For participants who succeeded in accessing transitional/community housing, how long did it take from intake into the pilot?

Implementation of program activities • What interventions were available for individuals/households (housing settings, social supports,

financial assistance, etc.)? What type of case management supports were implemented? What type of , if any, “continuity planning” was put in place following the phasing out period?

• To what extent were the program activities tailored to the characteristics/ needs of individuals/households?

• What activities occurred as expected; what activities did not occur as planned? Was there a phasing out of supports as planned? What factors caused changes in the implementation of the planned activities?

• Was the time-line for the pilot projects sufficient for the implementation of the planned activities? • What was the uptake of individuals for each intervention? • How well did the funding model of the pilot work?

What were the new opportunities provided by the model? What, if any, were funding constraints? Would there have been a more direct/efficient/appropriate means to fund the support services?

What short-term costs were unanticipated? Were there any other necessary costs that were considered out of scope? Who incurred them?

Group D: Questions about program outcomes Participant Outcomes (for all short-term and long-term outcomes) • To what extent has the pilot project contributed to the achievement of the intended outcome? • What practices/interventions or participant characteristics contribute to the achievement of this outcome? • What practices/interventions, participant characteristics, or other factors prevented or limited the

achievement of this outcome? • What additional supports (if any) are needed to improve outcomes?



Additional Participant Outcomes • What if any were the unintended outcomes of the project for participants? • How did the participants’ quality of life (e.g., participation in community, participation with family,

friends, etc.) change with their participation in the pilot? What factors were associated with any changes? Where applicable to short-term outcome objectives • How did the participant’s employment status change with the implementation of the pilot? Were the

participant’s barriers to employment addressed as part of the pilot? • Did the pilot facilitate the participant’s use of benefits offered by the OW program? • Did the participants know what they're supposed to accomplish while being in the pilot? Did they

"buy into" the program's goals and intended outcomes? Participant satisfaction • How satisfied were participants with the services and support offered? • What did the pilot participants like and dislike about their participation in the pilot project? Did the

participants receive the supports that matched their needs? What supports would participants have wanted that they did not receive?

Service system • To what extent has the pilot project led to a more coordinated and integrated provision of housing and

other support services? What type of collaboration was there among service providers for the pilot project?

• How well did collaboration work? What factors contributed to strong service collaboration? What were the barriers?

• To what extent did the implementation of the project impact the use of community services? • What linkages does the pilot have to other municipal, federal or provincial programs/initiatives? Costs and Cost-Savings • What is the breakdown of the pilot costs and the timing/outlay of those costs? • To what degree did the pilot generate savings to MCSS, the province overall, and the municipality? • Within the timeframe of the pilot project, what costs and savings to the Ontario Works Program could

be attributed to pilot? • What costs and savings could be attributed to other provincial and provincial/municipal programs

(e.g., ODSP, Corrections, police, land ambulance, health, etc.) Overarching Research Questions for Each Pilot and Across Pilots • What was the impact of the pilot initiative on the emergency hostel population across the 6 pilot sites?

(e.g., # of users, frequency of repeat and chronic users, use of support services, length of occupancy) • To what extent did the implementation of the pilot reduce the average length of stay in emergency hostels? • How did the participant’s housing history change with the implementation of the pilot? • Did individuals/households have a more stable housing history upon participating in the pilot (short

term and long term? • What practices/interventions contributed to the achievement of housing retention/employment? (type

of supports provided, participant’s barriers addressed through supports, sufficiency (i.e., intensity, duration) of supports for expected outcomes and profiles of participants)

• Was there a sufficient range and availability of interventions to assist the individuals/households in accessing and maintaining their accommodation?

• For those participants that were unsuccessful in retaining housing, what factors contributed to the episode of homelessness? What were the participant’s barriers to housing?

• Is there a net cost benefit of the pilot within MCSS? To the province overall? Municipality? • Is the intervention financially sustainable in the long-term? • Was the time-line for the pilot projects sufficient for the implementation of the planned activities? • What are the implications of the learnings from the pilots for future funding approaches?


DRAFT Evaluation Plan for Hostels to Homes Appendix C: Data Gathering Framework

Objectives Evaluation questions Indicators/data needs Data sources Methods Comments

Group A: Questions about provincial-level process objective (design/context of overall pilot initiative)

Project Context: Provincial • What are the provincial

goals/outcomes for the project?

provincial goals/outcomes

• What is the funding model? description of funding model

• Hostels to Homes Guidelines

• Document review

• What were the rationale for the development and financing of each pilot project?

rationale behind funding decisions

• MCSS • Document review or K.I. Interview

No new data required Methods depends on MCSS process and the availability of documentation from selection process

Group B: Questions about project planning and design (e.g., project rationale & design, management, administration, preparation, etc.)

• What is the size of the homeless population within each pilot community?

approximate # of homeless in pilot municipality

• How many hostel spaces exist within each pilot community?

# of hostel spaces in pilot municipality

• What are the annual expenditures on emergency hostels in each pilot community?

annual expenditures on emergency hostels in each pilot community

• Project proposal

• Pilot site manager

• Document review

• K.I. Interview or project survey/reporting form

Project Planning

• On what knowledge is the project design based?

summary of best practices that informed the project design

summary of key evidence from needs assessments and/or situational analyses that informed the project design



• Document review

• K.I. Interview or project survey/reporting form

Data Gathering Framework 19

DRAFT Evaluation Plan for Hostels to Homes Objectives Evaluation questions Indicators/data needs Data sources Methods Comments

• How was the project concept developed?

stakeholders involved in the development of the project concept

local partners for each of the projects

list of main steps in the planning process, and time planning process took

description of the approval process for the project design at the local level



• Document review

• KI Interview, or survey/project reporting form

description of the delivery model was proposed for each pilot

• Funding proposal

description of planned program activities and time-line for implementation



• Pilot site partners

Logic model consultation: • Document

review • Stakeholder

consultation

Project design • What was the proposed design that was intended for implementation?

# of hostel spaces are targeted in each pilots

description of criteria and process planned for the selection and recruitment of participants



• Document review


KI or survey of project lead proponents only if data is not already available in proposal documents

Project management and administration • Who are the stakeholders

playing a role in the project, and what role(s) did they play?

the stakeholders, the organizations or communities they represent, the positions they hold

description of the roles played in the planning, implementation, and/or governance of the project



• Document review




Is there clarity and consensus among key stakeholders on the roles they play?

Stakeholder understandings of roles

• Pilot site steering committee

• Pilot staff

• Focus groups and/or survey

How is the project administered?

description or chart of the administrative structure of the project

description of administrative processes in place for the pilot (planning, organizing, data gathering, reporting requirements, budgeting, processing participants, etc.)



Were these administrative/ processes/structures created for the pilot or did they already exist?

response of project lead proponent




• Document review

What administrative linkages does the pilot have to other municipal, federal or provincial programs/initiatives?

description of administrative linkages with other municipal, federal and provincial programs/initiatives



How efficient were the administrative processes? (data gathering and reporting requirements, planning, etc.)

stakeholder qualitative report or quantitative rating of efficiency

• How could the administrative model be improved?

stakeholder perceptions and descriptions of inefficiencies and particular efficiencies.

stakeholder perceptions of challenge faced by staff find in administering the pilot project, and descriptions of how these challenges were addressed

• Project staff • Implementati

on partners • Pilot site

steering committee

• Focus group and/or survey

Could get quantitative measure of perceived efficiency (and other data) by survey Qualitative methods will provide richer data on question of how admin model could be improved

•

•

•

•

•



Project Governance What structures and processes are used to govern the project?

stakeholders involved in project governance o description of

committee or other body responsible for monitoring and governing the project

description of accountability and reporting expectations at each pilot site



How effective were the governance structure and processes in supporting the project?

stakeholder qualitative report or quantitative rating of effectiveness


• Project staff

• Survey and/or focus groups

• How could this governance model be improved?

stakeholder perspectives on strengths and limitations of governance model

stakeholder ideas for improvements


• Project staff

• Focus group

Could get quantitative measure of perceived efficiency (and other data) by survey Qualitative methods will provide richer data on question of how governance could be improved

What personnel were involved in the implementation of the pilot?

# of personnel involved positions they hold qualifications/experience of

personnel training provided (if any)



Personnel

• Was staffing sufficient and appropriate?

stakeholder perspectives on staffing

• Project staff • Implementati

on partners • Pilot site

steering committee

• Focus group or K.I. interviews

Project Funding • What was the local funding model?

description of funding model used at each pilot site



•

•

•



• How did the funding of the pilot align with funding for existing business processes?

description of relationship of pilot to other funding mechanisms and business processes


• Project financial report

Group C: Questions about implementation of activities Participant recruitment and retention

• What criteria and process were used to select individuals/households for the pilot project?

description of criteria and processes used for participant selection

How appropriate were the participant selection/recruitment criteria and processes when implemented?

description and explanation of deviations from planned processes and criteria (if any)

comparison of actual participants with planned criteria



How could the criteria and processes for recruitment be improved?

stakeholder perspectives on strengths and short-comings of criteria and processes used

• Implementation partners



What are the characteristics of the individuals/households included in the pilot?

Dimensions for each participants: family/individual, employment status/history, vulnerabilities (MH, DD, etc.), sources of income, housing history (number and length of stays in hostel), etc.

• Participants • Intake form/interview

Standardized intake tool (or at minimum provision of standardized set of items for intake at all sites)

How long did participants remain in the pilot?

average and minimum/max participation periods for each pilot site



• What was the attrition rate? # and proportion of participants who dropped out of the program



•

•

•

•



• What were the causes of attrition (if any)?

characteristics of participants who dropped out

participant self-reported reasons for dropping out of program

project staff perspectives on reasons for drop out


• Implementation tracking data

Implementation of Program Activities • Were the activities of the

pilot implemented as planned?

# and characteristics of participants

description of program implementation (activities and time-line, including program supports and phase-out plans)

description and explanation of activities that were not implemented as expected and those that were not (activities and time-line, including program supports and phase-out plans)



• Was the time-line for the pilot projects sufficient for the implementation of the planned activities?

implementation data from cell above

stakeholder perspectives


• Pilot implementation partners


• Focus group

• To what extent were program activities tailored to the characteristics/needs of individuals/households?

description of individualized activities including case management supports

description of “continuity planning” in place for after the phasing out period

attrition and participation rate (below)



Gathering direct perspectives on this question might generate additional insights, but conclusions can be drawn from implementation and outcomes data. Additional stakeholder perspectives can be expected to be biased toward wanting a longer period of pilot project funding.



Participant satisfaction participant satisfaction with

services and supports provided • participants • Exit interview • How satisfied were

participants with the services and support offered? participant reports of likes and

dislikes about their participation in the pilot project

participants assessment of appropriateness of supports received to their needs

participants descriptions of additional supports needed

• participants • Exit interview and/or interviews with sample

What was the uptake of individuals for each intervention?

# of participants size of targeted population



• What is the breakdown of the pilot costs and the timing/outlay of those costs?

revenue and expense figures • Pilot site manager

• Project financial report

description unanticipated short-term costs

description of necessary costs considered outside the scope of the pilot funding, and who incurred them


• Project financial and narrative report

Project funding • How well did the funding model for the pilot work?

description of new administrative or programmatic opportunities the funding model provided for

description of administrative or programmatic constraints of the funding model

stakeholder perspectives on more direct / efficient / appropriate means to fund support services established by pilots

• Implementation partners



•



Group D: Questions about program outcomes Participant Outcomes

To what extent has the pilot project contributed to the increased access to affordable, stable housing?

To what extent are targeted individuals/households now placed in more stable residential settings?

• # and % of participants placed in affordable housing

• evidence of progress toward relevant short-term outcome objectives

Project staff

Project tracking data Long-term Objective #1: Increased access to affordable, stable housing

To what extent are individuals/households able to remain housed in the short-term and the long-term?

• # and % of participants placed in affordable housing

• Site staff • participants

• OW tracking data??

• Survey

Requires an ability to track participants beyond the pilot period (e.g., after one year from pilot end).

What practices/interventions or participant characteristics contribute to the achievement of this outcome?

• characteristics of participants successfully housed

• description of supports provided to participants successfully housed

• participant reports of most beneficial supports provided

• project staff • participants

• Implementation data

• Survey or focus groups

• Exit interviews or in-depth interview with sample of participants

• For participants who succeeded in accessing transitional/community housing, how long did it take from intake into the pilot?

• length of time from intake to entry into transitional/community housing

• project staff • OW data

tracking system

• Project tracking data


This question would be more meaningfully answered with post-pilot follow-up data

• What practices/interventions, participant characteristics, or other factors prevented or limited the achievement of this outcome?

• characteristics of participants who are unsuccessful in retaining housing

• service provider and participant descriptions of barriers to success

• service provider and

• project staff • participants

• Implementation data

• Survey or focus groups

• Exit interviews or in-depth interview with

•



participant self-report of reasons for episodes of homelessness (where applicable)

sample of participants

Analysis: outcome data + data on barriers to success • service provider

focus group or survey

• How well did the support provided address the participant’s barriers to achieving this outcome?

• service provider and participant reports of barriers

• service providers

• participants • participant pre-

post survey (intake and exit interview)

• service provider assessments of additional supports needed

• project staff • focus group or survey

• What additional supports (if any) are needed to improve outcomes? • participant assessments of

additional supports needed • participants • exit interview or

in-depth interview with sample of participants

Example Example Example Example

To what extent has the pilot project improved the services and support available to participants in finding suitable housing?

• How satisfied are participants with the supports provided?

• establishment and execution of individualized housing plan

• description of types of services and support provided

Project staff (case managers)

• case management tracking data

Project staff should be consulted on this item to develop consensus and buy-in for appropriate on-going data gathering

Short-term Objective: Example Improved access to services and support for transition to suitable housing

• service provider and participant assessments of improvements

• Participants • Other key

informants

• participant pre-post survey (intake and exit interviews)

• KI interviews with service providers



Key Indicator • Did the pilot facilitate the participant’s use of benefits offered by the OW program?

• Changes in OW benefits accessed by participants

Participants • participant pre-post survey (intake, exit and follow-up interviews)

Example • participant self-reported

understanding of objectives of pilot

• participant satisfaction with services and supports provided by the pilot

Example • participants

Example • Exit interview

Short-term Objective: Example Strengthened participation of clients in search for more stable, affordable housing

Example To what extent has the pilot strengthened participation of clients in search for more stable, affordable housing? • Did they "buy into" the

program's goals and intended outcomes?

• project staff assessment of participant engagement in program activities

Project staff • case management tracking data

Short-term Objective Evaluation questions Indicators Data source Measures/tools Add row for each of the short term outcome objectives that most directly relate to the long-term outcome objective Long-term Objective #2: Reduced incidence of re-entry into the hostel system

To what extent has the pilot reduced the number of repeat/chronic residents/# of repeat visits? To what extent were

individuals/households able to remain housed in the short-term and the long-term?

• # and % of participants who remain chronic shelter users after completion of pilot

• average # of nights participants spend in emergency shelters

• OW records • participants


• Participant pre-post survey (intake, exit and follow-up interviews)

Implies measurement at different intervals and therefore the ability to track beyond the pilot period.

Example Example Example Example Short-term Objective: Example Improved life-skills

To what extent were individuals able to improve or gain additional life skills to facilitate their transition from hostels to homes • What kinds of life skills

have participants acquired?

• self reports of participants • project staff assessment of

change in skills • description of skills taught

• Participants • Project staff • Project report

• Participant survey (exit interview)

• Tracking forms • Document

review

Consultation with implementers of life skills activities will be useful in ensuring meaningful and practical data tracking, and for generating buy-in for data tracking



• What other skills or supports do participants need?

Short-term Objective #2

Evaluation questions Indicators Data source Tools/tools

Add row for each of the short term outcome objectives that most directly relate to the long-term outcome objective Long-term Objective #3: A reduction in the length of stay/usage of emergency hostel beds by chronic bed users

To what extent did the pilot reduce the length of stay and usage of hostel beds by chronic hostel users? • To what extent were shelter

stays reduced in the short-term and the long-term?

• # of chronic bed users; proportion of total hostel population

• average length of stay among shelter users

• average frequency and length of stay among chronic bed users

• average frequency and length of stay among chronic bed users

Project staff Pre-post (and follow-up) shelter stats

Implies measurement at different intervals and therefore the ability to track beyond the pilot period.

Example To what extent were clients able to improve their employability and find meaningful employment • What is the effect of the

intervention on employment status of the individual/household?

Example • # and types of training

programs/resources provided • # and % of participants whose

employment status improved

Short-term Objective: Example Improved employability (skills, resources) Key Indicator

•

•

What barriers remain to participants’ full employment? Which program activities and supports helped the most?

• self-reports of participants and observations of service providers

Example • OW tracking

system • Program

tracking form • Participants

Example • OW tracking data • Document

review • Participant pre-

post survey (intake, exit and follow-up interviews)

Short-term Objective Evaluation questions Indicators Data source Tools/tools

Add row for each of the short term outcome objectives that most directly relate to the long-term outcome objective



What if any were the unintended consequences of the project for participants as a result of the intervention?

Other Outcomes for Participants (local objectives or unintended outcomes)

• How did the participants’ quality of life change with their participation in the pilot and what factors were associated with any changes?

• stakeholder observations of additional participant outcomes

• participant self-report of

additional outcomes

• project partners

• participants

• Focus group • Exit interview or

in-depth qualitative interview with sample

Service system outcomes



• summary of collaborative process/activities facilitated by pilot

Service providers • Focus group • service providers reports of

changes in service coordination (seamlessness, cross-sectoral collaboration, etc.)

Service providers focus group or survey

• participant pre-post reports of satisfaction with services

Intake and exit interview

• changes in the time it takes for clients to move through the system and in the # of agency contacts required

participants

To what extent has the pilot project led to a more coordinated and integrated provision of housing and other support services? How well did collaboration

work?

See indicators related to short-term objectives

Long-term Objective #4: Enhanced coordination and integration in the provision of housing and other support services

• What factors contributed to strong service collaboration? What were the barriers?

• Service provider observations Service providers and participants

Focus group or interview (e.g., participant exit interview)

Short-term Objective: Example Reduced system gaps

To what extent has the pilot project led to a reduction in system gaps and service barriers?

• changes made in system of services to reduce barriers

• changes made in system of services to address gaps


Project reporting form

A participant survey would seek to get information such as: What factors

•



• changes in participant satisfaction with accessibility of services

• changes in participant reports of unmet needs

Participants Participant pre-postsurvey (intake, exit and follow-up interviews)

• change in service provider report of system gaps and service barriers

Service providers Focus group and KI interviews

Short-term Objective Evaluation questions Indicators Data source Tools/tools

Add row for each of the short term outcome objectives that most directly relate to the long-term outcome objective Costs and cost-savings

• savings to OW See below See below • reduction in use of other

services by pilot participants See below

See below

To what degree did the pilot generate savings to MCSS, the province overall, and the municipality?

• research evidence that housing stability generates cost-savings in other social services and programs

• research literature

Literature review

• estimate of savings generated by changes in service use

• research literature

• engagement of economist

economic estimate model

Long-term Objective #5: Overall savings to the municipality and the province

•

•

•

Within the timeframe of the initiative, what costs/savings were there to the Ontario Works Program as a result of this pilot? Comparison of

• breakdown of costs and time/outlay of those costs of pilot; # of participant nights not spent in hostels shelter per diem amount

• Pilot site

manager

• Project reporting

template • Key informant

interviews

• • What costs/savings have been created for other provincial and

changes in participant self-reports of frequency and use of provincial and municipal services

participants Participant pre-post survey (intake, exit and follow-up interviews)

Making a credible estimate of the full savings to the province would require a significant input of time and money. Economic models for estimate cost-savings generated by increased housing stability may exist in literature, but until further research is done, whether any existing models can be effectively adapted to the H2H pilot is difficult to determine. Evidence can quite easily be gathered on changes in participants’ use of community and social services, and literature is available



service provider reports of changes in service use

Community, health and social service providers

Focus group provincial/municipal programs (e.g., ODSP, Corrections, police, land ambulance, health, etc.) • estimate of savings generated

by changes in service use • literature

review • engagement

of economist

economic estimate model

that analyzes the economic impact of stable housing and the absence thereof.

Overarching Research Questions for Each Pilot and Across Pilots • • What was the impact of the pilot initiative on the

emergency hostel population across the 6 pilot sites? change in # of shelter users, # and % of shelter users who are chronic users, frequency of use by chronic users


Shelter tracking data (pre and post)

To what extent did the implementation of a pilot reduce the average length of stay in emergency hostels?

change in average length of stay among shelter users

Pilot site manager

Shelter tracking data (pre, post & follow-up)

How did the participant’s housing history change with the implementation of the pilot?

qualitative summary of housing history pre-pilot and post pilot

participants • Intake and exit interview

• Key informant interview at follow-up

Did individuals/households have a more stable housing history upon participating in the pilot (short term and long term?

change in use of emergency shelters

Project staff Shelter tracking data (pre, post and follow-up)

What practices/interventions contributed to the achievement of housing retention/employment? (type of supports provided, participant’s barriers addressed through supports, sufficiency (i.e., intensity, duration) of supports for expected outcomes and profiles of participants)

Synthesis of implementation and outcome findings from all sites

Was there a sufficient range and availability of interventions to assist the individuals/households in accessing and maintaining their accommodation?


For those participants who were unsuccessful in retaining housing, what factors contributed to the episode of homelessness? What were the participant’s barriers to housing?

Synthesis of outcome and needs findings from all sites Synthesis of needs and barriers findings from all sites

•

• • •

• • •

• •

•

•

•



Is there a net cost benefit of the pilot within MCSS? To the province overall? Municipality?

Synthesis of costs savings findings from all sites

Is the intervention financially sustainable in the long-term?

• Synthesis of design and implementation findings from all sites • Cost-savings findings

Was the time-line for the pilot projects sufficient for the implementation of the planned activities?


What are the implications of the learnings from the pilots for future funding approaches?

Analysis of all data

•

•

•

•



Appendix D: Methods Framework

Data Source Indicators/data needs (What data needs to be gathered?) When? Methods (How?) Comments

Hostels to Homes Guidelines

provincial goals/outcomes description of funding model

Already available

• Document review

MCSS provincial staff

(pilot proposal review/selection staff or committee)

rationale behind funding decisions At beginning of evaluation process

• Key Informant Interview (unless documentation is available)

Research literature research evidence that housing stability generates cost-savings in other social services and programs

economic model for estimating savings generated by changes in service use

At beginning of evaluation process

• Literature review

• Engagement of economist

Pilot Site Manager

(the person responsible for reporting and other administrative functions)

approximate # of homeless in pilot municipality # and % of total population who are chronic bed users size of target population average length of stay in shelter average frequency and length of stay among chronic bed users average length of hostel stay among pilot participants annual expenditures on emergency hostels in each pilot community

Baseline Pre-post and follow-up

• Project proposal and/or reporting docs/project end survey

Logic model process will be useful in drawing out and describing the model of delivery and the planning/design steps involved

Pilot Site Manager

# and characteristics of participants participation and attrition participation periods (how long participants stay involved) characteristics of participants who dropped out participants placed in affordable housing average length of time from intake to entry into transitional/community

housing average # of nights participants spend in emergency shelters

Requires tracking through pilot period (and beyond)

• Local tracking system

• Exit interview • Follow-up

interview / survey

Methods Framework 34



# and % of participants who remain chronic shelter users after completion of pilot

Pilot Site Manager

# of hostel spaces in pilot municipality # of hostel spaces targeted by municipality criteria and processes planned for selection and recruitment of

participants

the local partners for each of the projects summary of best practices that informed the project design

o summary of key evidence from needs assessments and/or situational analyses that informed the project design

stakeholders involved in the development of the project concept

local partners for each of the projects list of main steps in the planning process, and time planning process took description of the approval process for the project design at the local level description of the delivery model was proposed for each pilot description of planned program activities and time-line for

implementation

description or chart of the administrative structure of the project description of administrative processes in place for the pilot (planning,

organizing, data gathering, reporting requirements, budgeting, processing participants, etc.)

were these administrative/governance processes/structures created for the pilot or did they already exist?

description of administrative linkages with other municipal, federal and provincial programs/initiatives

description of funding model used at each pilot site description of relationship of pilot to other funding mechanisms and

business processes shelter per diem amount

At beginning of evaluation process

• Project proposal or reporting docs

• interviews

Much of this data can be captured through a project-end reporting framework. Other data is more efficiently and meaningfully gathered using qualitative methods. Choice of methods depends on the type of data, and on level of reporting expected of pilot sites. Key informant interviews shift the burden of data gathering onto the evaluator. Answering all evaluation questions is best accomplished with a mix of qualitative and quantitative methods.




Pilot site manager

the stakeholders, the organizations or communities they represent, the positions they hold

description of the roles played in the planning, implementation, and/or governance of the project

stakeholders involved in project governance o description of committee or other body responsible for monitoring

and governing the project description of accountability and reporting expectations at each pilot site qualitative report or quantitative rating of effectiveness of governance model perspective on strengths and limitations of governance model ideas for improvements

qualitative report or quantitative rating of efficiency of administrative model

perceptions and descriptions of inefficiencies and particular efficiencies of administrative model

perceptions of challenge faced by staff find in administering the pilot project, and descriptions of how these challenges were addressed

# of personnel involved positions they hold qualifications/experience of personnel training provided (if any) perspectives on sufficiency and appropriateness of staffing

revenue and expense figures description unanticipated short-term costs description of necessary costs considered outside the scope of the pilot

funding, and who incurred them description of new administrative or programmatic opportunities the

funding model provided for description of administrative or programmatic constraints of funding model perspectives on more direct/efficient/appropriate means to fund support

services established by pilots

End of pilot period

• Project reporting docs

• Interviews or project end survey




Pilot Site Manager

description of criteria and processes used for participant selection description and explanation of deviations from planned processes and

criteria (if any) comparison of actual participants with planned criteria perspectives on strengths and short-comings of criteria and processes

used

# and characteristics of participants description of program implementation (activities and time-line,

including program supports and phase-out plans) o description and explanation of activities that were not implemented

as expected and those that were not (activities and time-line, including program supports and phase-out plans)

perspectives on sufficiency of the time-line for implementation of planned activities

description of individualized activities including case management supports

description of “continuity planning” put in place for following the phasing out period

attrition and participation rate average and minimum/max participation periods for each pilot site # and proportion of participants who dropped out of the program characteristics of participants who dropped out What activities occurred as expected; what activities were not

o What factors caused the implementation of the planned activities? What type of case management supports were implemented? What, if any, “continuity planning” was put in place following the

phasing out period? Was there a phasing out of supports as planned?

# and % of participants placed in affordable housing # and % of participants who remain chronic shelter users after

completion of pilot o average # of nights participants spend in emergency shelters

End of pilot period

• Project reporting docs

• Interviews or project end survey




observations of additional participant outcomes achieved breakdown of costs and time/outlay of those costs of pilot # of participant nights not spent in hostels description of additional supports provided average length of hostel stay among pilot participants

summary of collaborative process/activities facilitated by pilot assessment of changes in service coordination (seamlessness, cross-

sectoral collaboration, etc.) changes made in system of services to address gaps

Pilot site steering committee (or other local body governing the pilot project)

stakeholder understandings of roles qualitative report or quantitative rating of effectiveness of governance model perspective on strengths and limitations of governance model ideas for improvements

qualitative report or quantitative rating of efficiency of admin. model perceptions & descriptions of inefficiencies and particular efficiencies

o stakeholder perceptions of challenge faced by staff find in administering the pilot project, and descriptions of how these challenges were addressed

perspectives on sufficiency and appropriateness of staffing

perspectives on strengths and short-comings of criteria and processes used



description of administrative or programmatic constraints of the funding model o perspectives on more direct/efficient/appropriate means to fund

support services established by pilots

At pilot end • Focus group or stakeholder interviews

• On-line survey

On-line survey would be an efficient means of answering questions in a quantified way (e.g., on a scale of 1-5, how efficient was the administrative model used?), but qualitative methods would be better for capturing insights and ideas about what made the model efficient or not.




Project partners (agencies directly involved in the implementation of the pilot)

qualitative report or quantitative rating of effectiveness of governance model perspective on strengths and limitations of governance model ideas for improvements

qualitative report or quantitative rating of efficiency of administrative model perceptions and descriptions of inefficiencies and particular efficiencies

o stakeholder perceptions of challenge faced by staff find in administering the pilot project, and descriptions of how these challenges were addressed

perspectives on sufficiency and appropriateness of staffing


description of administrative or programmatic constraints of the funding model o perspectives on more direct/efficient/appropriate means to fund

support services established by pilots

perspectives on strengths and short-comings of criteria and processes used for selection/recruitment of participants


project staff perspectives on reasons for attrition (if any)

characteristics of participants successfully housed description of supports provided to participants successfully housed

characteristics of participants who are unsuccessful in retaining housing descriptions of barriers to achievement of outcomes perspectives on reasons for episodes of homelessness (where applicable) assessments of additional supports needed for achievement of outcomes observations of additional participant outcomes

assessments/descriptions of changes in service coordination (seamlessness, cross-sectoral collaboration, etc.)

At pilot end • Focus group (or K.I. interviews)

• On-line survey




perspectives on contributing factors and barriers to service collaboration

Participants dimensions for each participants: family/individual, employment status/history, vulnerabilities (MH, DD, etc.), sources of income, housing history (number and length of stays in hostel), etc.

use of emergency shelters (frequency and duration of stay)

pre-post reports of satisfaction with services changes in the time it takes for clients to move through the system and in

the # of agency contacts required changes in participant satisfaction with accessibility of services changes in participant reports of unmet needs changes in participant self-reports of frequency and use of provincial and

municipal services qualitative summary of housing history pre-pilot and post pilot

Pre-Post: at baseline, at pilot end, at follow-up

• Intake/exit survey/interviews

• Follow-up survey/interview

Participants participant satisfaction with services and supports provided self-report of additional outcomes participant reports of likes and dislikes about their participation in the

pilot project participants assessment of appropriateness of supports received to their needs

o participants descriptions of additional supports needed self-reported reasons for dropping out of program (where relevant) participant reports of most beneficial supports provided descriptions of barriers to achievement of outcomes self-report reasons for episodes of homelessness (where applicable) assessments of additional supports needed for achievement of outcomes

At pilot end • Exit interview • In-depth

interviews with sample of participants




Other Key Informants

(Service providers not directly involved in the pilot but whose services are affected by it; e.g. community mental health services, emergency services, emergency food/shelter providers, etc.)

reports of barriers to achievement of program outcomes

assessments/descriptions of changes in service coordination (seamlessness, cross-sectoral collaboration, etc.)

service provider report of change in system gaps and service barriers service provider observations of contributing factors and barriers to

strong service collaboration service provider qualitative observations of value added of pilot’s service

coordination model

service provider reports of changes in service use

End of pilot (retrospective)

• Focus Group • Survey

provider survey


Centre for Research & Education in Human Services

73 King Street West, Suite 300 Kitchener, Ontario N2H 4Z9

www.crehs.on.ca

Contact: Robert Case; [email protected]; 519-741-1318 X234

http://www.crehs.on.ca/

mailto:[email protected]

Documents

DRAFT Evaluation Plan for Hostels to Homes (HTH ... to...DRAFT Evaluation Plan for Hostels to Homes Table of Contents 1.0 Overview 1 1.1 The Hostels to Homes: Emergency Hostel Redirection