Implementation Evaluation of Canada’s At Home/Chez Soi Housing First Program Geoffrey Nelson, National Research Team, At Home/Chez Soi Project, Dept. of Psychology, Wilfrid Laurier University, Waterloo, Ontario, Canada Tim Aubry, National Research Team, At Home/Chez Soi Project, School of Psychology, University of Ottawa, Ottawa, Ontario, Canada 9th European Research Conference – Homelessness in Times of Crisis, FEANTSA, Warsaw, Friday, September 19, 2014

/ 2

Overview of Presentation

• Background and Description of the At Home/Chez Soi Study • Fidelity Findings • Qualitative Implementation Findings

/ 3

At Home/Chez Soi Project Description

In 2008 the Federal government allocated $110 million

At Home/Chez Soi is:

The largest study of its kind in the world (>2100 participants)

Pathways Housing First model

Implemented in 5 Canadian cities: Vancouver, Winnipeg, Toronto, Montreal, Moncton

Funding – 85% services / 15% research

A randomized controlled trial

/ 4

Housing First

• Housing is immediate and permanent • Recovery & choice emphasized • Housing is primarily scattered site, private market • Supports include guaranteed rent subsidies – clients pay 30% • Additional services (>1 visit) are voluntary • No readiness/maintenance conditions • Tenancy rights • Draws upon Pathways to Housing program model • Two service delivery approaches (ACT and ICM) for two different populations (High and Moderate Need)

/ 5

/ 6

Vancouver

•Outcomes of individuals with serious substance use issues • 3rd arm – a congregate setting

/ 7

• Focus on Aboriginal homeless population • 3rd arm included traditional healing approaches (e.g., medicine wheel)

Winnipeg

/ 8

Toronto

• 3rd arm – ethno-culturally appropriate based on anti-racist/anti-oppression model • Also emphasis on primary care/chronic physical illnesses

/ 9

• Intervention included test of a vocational approach – individualized placement and support (IPS)

Montreal

/ 10

Moncton

• Blended intervention - “flexible ACT” (FACT) model • Focus on smaller cities and rural areas

/ 11

Implementation of Housing First

• National and local training events • Technical assistance visits • Communities of practice • Quality Assurance Team conducts fidelity assessments

/ 12

What is the Purpose of the Mixed Methods Fidelity and Implementation Evaluation?

Quantitative fidelity assessment • Looks at degree of implementation of critical ingredients

• Rates programs numerically using scale

• Provides feedback for quality improvement

Qualitative process and implementation evaluations uncovers: • Strengths

• Challenges/Trouble spots

/ 13

Can Housing First Be Adapted to Local Context and Culture?

Yes, but must be careful that the adaptation does not deviate from Housing First elements/principles • Toronto’s Anti-racism, Anti-oppression ICM program

• Winnipeg’s Aboriginal-focused medicine wheel program

But “no,” when the adaptation goes against the basic principles of Housing First, because it enters a “zone of drastic mutation” (Hall & Loucks, 1978)

Housing

First

Fidelity

Adaptation

/ 15

Why is Housing First Fidelity Important?

Fidelity matters because it is related to outcomes. The greater the fidelity, the better the outcomes of Housing First

• Gilmer et al. (2014) • Davidson et al. (2014)

/ 16

Relationship of Fidelity with Program Processes and Participant Outcomes

• Higher fidelity in the in the 12 programs (5 ACT, 7 ICM) was related to greater direct and indirect service time and more contacts with participants (corr. = .55 to .60)

• Higher fidelity was associated with greater housing stability (% of time in housing) (OR = 1.11) and larger improvement in quality of life (d = .10) and community functioning (d = .11) over the course of the study

/ 17

Housing First Fidelity Domains

Housing and recovery-oriented approach – 37 item scale with items falling into 5 domains; each item completed for each program

• Housing choice and structure • Separation of housing and services • Treatment philosophy (recovery orientation) • Service array • Program structure

/ 18

Sample Housing First Fidelity Items

Housing choice and structure domain

• Housing choice • Housing availability • Permanent housing • Affordable housing • Integrated housing • Privacy

/ 19

Housing Availability Item – Anchors

All items are rated on a 4-point scale, with 4 indicating high fidelity, and 1 indicating low fidelity

• Extent to which program helps participants move quickly into units of

their choosing.

1. Less than 55% of program participants move into a unit of their choosing within 6 weeks of having a housing subsidy or receiving a voucher

2. 55-69% of program participants move into a unit of their choosing within 6 weeks of having a housing subsidy or receiving a voucher

3. 70-84% of program participants move into a unit of their choosing within 6 weeks of having a housing subsidy or receiving a voucher

4. 85% of program participants move into a unit of their choosing within 6 weeks of having a housing subsidy or receiving a voucher

/ 20

Fidelity Assessment External quality assurance team

Data Collected Time 1 – 9-13 Months of Program Operation

Time 2 – 26-31 Months of Program Operation

Staff Interviews 84 89

Consumer Focus Groups 10 11

Chart Reviews 100 102

Number of Programs

10 10

Number of Sites 5 5

/ 21

Summary of Fidelity Rating Findings

• 10 programs rated on 37 fidelity items in 5 domains • Overall, strong fidelity to the Housing first model • Also, improvement noted from early to later

implementation (71% program ingredients rated above 3 on a 4-point scale at early implementation and 78% of program ingredients rated above 3 at later implementation)

• Average fidelity rating across all items is 3.47 at early implementation and 3.62 at later implementation

• Variation at individual program level and by service delivery type (e.g., more challenges for ICM regarding service array domain)

• Both fidelity and adaptation are important

/ 22

Qualitative Implementation Evaluation • Aim: to better understand the reasons for implementation

fidelity strengths and challenges/trouble spots in the At Home/ Chez Soi project • Qualitative implementation evaluations were conducted in years one

and three of the project

• Primary questions: 1. What are implementation strengths?

2. What are implementation challenges/trouble spots?

/ 23

Qualitative Implementation Evaluation Site qualitative research teams and national qualitative research team

Data Collected Time 1 – 9-13 Months of Program Operation

Time 2 – 26-31 Months of Program Operation

Key Informant Interviews

72 36

Focus Groups 35(211) 17(99)

Number of Programs 10 10

Number of Sites

5 5

/ 24

Methods • Sampling and sample • Purposeful sampling

• 283 participants total at T1 and 135 total at T2

• Data collection • Interviews in English or French

• At participants’ workplaces or site offices

• Interviews audio recorded and transcribed verbatim

• Data analysis • Thematic analysis for site reports and cross-site report

/ 25

Results – Q1: Implementation Strengths • Partnerships and collaboration between services teams, and

with participants, community organizations, and professionals were important strengths • One respondent at the Vancouver site stated, “I’ve never worked in such a

supportive and collaborative place. The team is so committed to what it

does, and I think it’s reflected in our communication and the changes

we’ve seen in our clients.”

/ 26

Results – Q1: Implementation Strengths

• Despite low vacancy rates and lack of affordable housing in every city, all sites reported success with specific aspects of housing and rehousing participants • Across sites, this was achieved primarily by 1) creatively matching

participants with housing units; 2) learning from past experiences; 3)

achieving stability; 4) planning preventative strategies to avoid future

rehousing incidents; and 5) forming collaborative relationships with

landlords and property management companies

• “Compared to other landlords, those in our project get a great deal of

support… We underestimated how important that support is. It’s like an

extra set of hands helping them do their job.” (Vancouver team member)

/ 27

Results – Q1: Implementation Strengths

• Presence of dedicated specialists and professionals on housing and service teams • For example, the Moncton site

reported that the addition of a home

economist, psychiatric consultant, and

vocational specialist to their ACT team

enabled program delivery and support

• Toronto noted that their service

provision success could be attributed

to the high level of clinical expertise

from team leaders and psychiatrists

/ 28

Results – Q2: Implementation Challenges/Trouble Spots

• Staffing • Substantial staff turn-over across sites due to heavy workloads, pressure

to meet project timelines, and concerns about job security

• This is a common problem in these types of programs and not something

unique or unusual about the At Home/ Chez Soi project

• Scattered-site housing model • Vancouver, Winnipeg, Montreal, and Moncton faced challenges related to

scattered-site being the sole model of housing for the project, particularly

for people with more complex physical and mental health challenges

• Winnipeg suggested congregate-type housing as an alternative;

Vancouver introduced the idea of “step-up, step-down” options; and

Moncton opened a transitional apartment building

/ 29

Results – Q2: Implementation Challenges/Trouble Spots

• Participant feelings of isolation and loneliness • Isolation is a broader social issue which is more

acute for this population

• Sites attributed these difficult experiences to

participants’ changing social networks, leaving

friends behind, and vast lifestyle and living

arrangement changes

• “…the person that is addicted lives a life of

loneliness… You can’t be around your friends, so

the drug is your best friend… An agency has to

make sure that you can be there to complement

that… until they have been able and have had time

to build new friends.” (Moncton key informant)

/ 30

Results – Q2: Implementation Challenges/Trouble Spots

• Some sites had problems integrating peer support into the program • Some difficulty identifying and training potential peer specialists

• Moncton and Winnipeg sites had concerns about appropriately defining

and clarifying the roles that peer support workers should play on service

teams

• Documentation • Sites noted the need to more clearly document housing and service

teams’ work in their case notes

• Certain implementation challenges and low implementation fidelity for

certain criteria could be attributed to inconsistent record and chart-

keeping by team members

/ 31

Results – Q2: Implementation Challenges/Trouble Spots

• Most sites identified some trouble spots regarding housing between the second and third years of implementation • Challenges around finding good quality, affordable, vacant housing units in areas in

which participants desired to live

• Housing restrictions related to smoke-free and pet-free policies (Vancouver)

• In Toronto, issues of tenancy management, relationships between housing teams

and participants, dealing with unit damage, and rehousing participants emerged as

trouble-spots in housing

• “So sometimes, the only interaction we had with clients was when there was

chaos, when their was a real problem that was going on… I mean, the landlords

are calling us and complaining… but we have no idea who the person is so that

was a disadvantage to me that’s going forward.” (Toronto team member)

/ 32

Results – Q2: Implementation Challenges/Trouble Spots

• Vocational and educational goal supports • The Vancouver site reported that initially engaging

and/ or re-engaging participants in work, volunteer,

and educational activities had become difficult

• Vocational specialists often had to tend to

emergency crises rather than develop more

proactive, long-term goals with participants

• Toronto team members found limited variety and

availability of educational and employment

opportunities within the community

/ 33

Conclusions • Findings provide information about identifying and

maintaining strengths, as well as identifying and potentially preventing challenges or trouble spots when implementing complex community-based housing interventions