6
Engaging stakeholders in the planning of a collaborative multi-agency evaluation: The HousingPlus Collaborative Communities Project John Sylvestre a, *, J. Bradley Cousins b , Purnima Sundar c , Tim Aubry d , Val Hinsperger e a School of Psychology and Center for Research on Educational and Community Services - University of Ottawa, 125 University Pvt., Room 416A, Ottawa, ON, K1N 6N5 Canada b Faculty of Education and Centre for Education on Educational and Community Services - University of Ottawa, 145 Jean-Jacques Lussier Street, Lamoureux Hall, Room 229, Ottawa, ON, K1N 6N5 Canada c The Provincial Centre of Excellence for Child and Youth Mental Health at the Children’s Hospital of Eastern Ontario (CHEO), 401 Smyth Road, Ottawa, ON, K1H 8L1 Canada d School of Psychology and Center for Research on Educational and Community Services - University of Ottawa, 30-32 Stewart, Room 106, Ottawa, ON, K1N 6N5 Canada e Emily Murphy Non-Profit Housing Corp, 2676 Innes Road, Gloucester, ON, K1B 1A2 Canada Cousins and colleagues differentiate collaborative evaluation in terms of interests or justifications for engaging stakeholders in evaluation and dimensions of form or process of evaluation (Cousins & Whitmore, 1998; Weaver & Cousins, 2004). Whereas these frameworks help us to distinguish between different approaches to collaborative evaluation (practical participatory, transformative, empowerment) they tell us little about the process of non- evaluator stakeholder 1 engagement. On reflection, we would suggest that there are five critical issues worthy of consideration in the context of thinking about, planning, or otherwise stimulating stakeholder engagement in collaborative evaluation. They are: (1) the context of the evaluation; (2) the evaluation approach being used; (3) the diverse nature and interests of the stakeholders associated with the program or entity being evaluated and the actual evaluation; (4) the justifications for engaging stakeholders in the evaluation; (5) the roles for stakeholders who collaborate in the evaluation. In this article, we use each of these five issues as a basis for describing and analyzing a collaborative multi-agency and multi- stakeholder evaluation, the HousingPlus Collaborative Commu- nities (HPCC) project. Although this evaluation is still in its early development, this time period can be crucial for successfully engaging relevant stakeholder groups. Therefore, after briefly describing the case, we discuss the strategies used to engage stakeholders in planning the evaluation. Table 1 includes a summary of this case example presented according to the five issues we mentioned above, and the steps undertaken to engage stakeholders. In the HPCC project, the evaluators are considered stakeholders along with other groups. As a result, our discussion includes the steps undertaken by various stakeholder groups (including evaluators and non-evaluators) in engaging each other in the evaluation process. We conclude with lessons learned from the evaluation planning process and the strategies required to continue to engage stakeholders. Studies in Educational Evaluation 34 (2008) 212–217 ARTICLE INFO Keywords: Collaboration Supportive housing Implementation evaluation Utilization ABSTRACT This article describes the planning of a multi-agency and multi-stakeholder evaluation of the implementation of supportive housing. The Ottawa HousingPlus Collaborative Communities project is a collaboration between university-based evaluators and members of a local network of supportive housing providers. The objectives of the initiative are (1) to develop tools and methods for evaluating the implementation of supportive housing by network members, (2) to evaluate program implementation, and (3) to use evaluation findings to assist network members to collaborate in finding solutions to shared implementation challenges. This article examines the early planning of the project. It describes the process used to engage stakeholders in the conception and planning of the project, the challenges and successes in the early planning efforts, and the issues that will need to be addressed over the course of the 3-year project term. These challenges and successes of engaging stakeholders are discussed in the context of adapting educational concepts to the evaluation of social services and in engaging multiple agencies, at multiple levels, in a collaborative evaluation. ß 2008 Elsevier Ltd. All rights reserved. * Corresponding author. E-mail addresses: [email protected] (J. Sylvestre), [email protected] (J.B. Cousins), [email protected] (P. Sundar), [email protected] (T. Aubry), [email protected] (V. Hinsperger). 1 The term ‘‘non-evaluator stakeholder’’ is meant to imply persons with an interest in the program or entity being evaluated who are not evaluators. In this paper, from this point forward, we use the term stakeholder to convey non-evaluator stakeholder, it being understood that researchers/evaluators are stakeholders as well. Contents lists available at ScienceDirect Studies in Educational Evaluation journal homepage: www.elsevier.com/stueduc 0191-491X/$ – see front matter ß 2008 Elsevier Ltd. All rights reserved. doi:10.1016/j.stueduc.2008.10.005

Engaging stakeholders in the planning of a collaborative multi-agency evaluation: The HousingPlus Collaborative Communities Project

Embed Size (px)

Citation preview

Studies in Educational Evaluation 34 (2008) 212–217

Engaging stakeholders in the planning of a collaborative multi-agency evaluation:The HousingPlus Collaborative Communities Project

John Sylvestre a,*, J. Bradley Cousins b, Purnima Sundar c, Tim Aubry d, Val Hinsperger e

a School of Psychology and Center for Research on Educational and Community Services - University of Ottawa, 125 University Pvt., Room 416A, Ottawa, ON, K1N 6N5 Canadab Faculty of Education and Centre for Education on Educational and Community Services - University of Ottawa, 145 Jean-Jacques Lussier Street, Lamoureux Hall, Room 229, Ottawa,

ON, K1N 6N5 Canadac The Provincial Centre of Excellence for Child and Youth Mental Health at the Children’s Hospital of Eastern Ontario (CHEO), 401 Smyth Road, Ottawa, ON, K1H 8L1 Canadad School of Psychology and Center for Research on Educational and Community Services - University of Ottawa, 30-32 Stewart, Room 106, Ottawa, ON, K1N 6N5 Canadae Emily Murphy Non-Profit Housing Corp, 2676 Innes Road, Gloucester, ON, K1B 1A2 Canada

A R T I C L E I N F O

Keywords:

Collaboration

Supportive housing

Implementation evaluation

Utilization

A B S T R A C T

This article describes the planning of a multi-agency and multi-stakeholder evaluation of the

implementation of supportive housing. The Ottawa HousingPlus Collaborative Communities project is

a collaboration between university-based evaluators and members of a local network of supportive

housing providers. The objectives of the initiative are (1) to develop tools and methods for evaluating the

implementation of supportive housing by network members, (2) to evaluate program implementation,

and (3) to use evaluation findings to assist network members to collaborate in finding solutions to shared

implementation challenges. This article examines the early planning of the project. It describes the

process used to engage stakeholders in the conception and planning of the project, the challenges and

successes in the early planning efforts, and the issues that will need to be addressed over the course of the

3-year project term. These challenges and successes of engaging stakeholders are discussed in the context

of adapting educational concepts to the evaluation of social services and in engaging multiple agencies, at

multiple levels, in a collaborative evaluation.

� 2008 Elsevier Ltd. All rights reserved.

Contents lists available at ScienceDirect

Studies in Educational Evaluation

journal homepage: www.e lsev ier .com/stueduc

Cousins and colleagues differentiate collaborative evaluation interms of interests or justifications for engaging stakeholders inevaluation and dimensions of form or process of evaluation (Cousins& Whitmore, 1998; Weaver & Cousins, 2004). Whereas theseframeworks help us to distinguish between different approaches tocollaborative evaluation (practical participatory, transformative,empowerment) they tell us little about the process of non-evaluator stakeholder1 engagement. On reflection, we wouldsuggest that there are five critical issues worthy of consideration inthe context of thinking about, planning, or otherwise stimulatingstakeholder engagement in collaborative evaluation. They are: (1)the context of the evaluation; (2) the evaluation approach being

* Corresponding author.

E-mail addresses: [email protected] (J. Sylvestre), [email protected]

(J.B. Cousins), [email protected] (P. Sundar), [email protected] (T. Aubry),

[email protected] (V. Hinsperger).1 The term ‘‘non-evaluator stakeholder’’ is meant to imply persons with an

interest in the program or entity being evaluated who are not evaluators. In this

paper, from this point forward, we use the term stakeholder to convey non-evaluator

stakeholder, it being understood that researchers/evaluators are stakeholders as

well.

0191-491X/$ – see front matter � 2008 Elsevier Ltd. All rights reserved.

doi:10.1016/j.stueduc.2008.10.005

used; (3) the diverse nature and interests of the stakeholdersassociated with the program or entity being evaluated and theactual evaluation; (4) the justifications for engaging stakeholdersin the evaluation; (5) the roles for stakeholders who collaborate inthe evaluation.

In this article, we use each of these five issues as a basis fordescribing and analyzing a collaborative multi-agency and multi-stakeholder evaluation, the HousingPlus Collaborative Commu-nities (HPCC) project. Although this evaluation is still in its earlydevelopment, this time period can be crucial for successfullyengaging relevant stakeholder groups. Therefore, after brieflydescribing the case, we discuss the strategies used to engagestakeholders in planning the evaluation. Table 1 includes asummary of this case example presented according to the fiveissues we mentioned above, and the steps undertaken to engagestakeholders. In the HPCC project, the evaluators are consideredstakeholders along with other groups. As a result, our discussionincludes the steps undertaken by various stakeholder groups(including evaluators and non-evaluators) in engaging each otherin the evaluation process. We conclude with lessons learned fromthe evaluation planning process and the strategies required tocontinue to engage stakeholders.

Table 1Issues to consider when engaging stakeholders in the process of evaluation.

Issues HPCC case example Engaging stakeholders in evaluation planning

Context of and drivers for the evaluation Limited funding for supportive housing Network contacted evaluators to launch discussions

Competing housing models Mutual process of engagement between network and

evaluators to link network needs and evaluator

interests and expertiseChallenges in describing programs and their unique

benefitsOpen and collaborative process of proposal

development over an extended period of timeEvaluators’ interest in benchmarking supportive

housing, and expertise in implementation evaluation

methods

Evaluation approach and methods Using implementation evaluation to support

collaborative inquiry into optimal implementation

of supportive housing, actual implementation of

supportive housing, and strategies for improving

program implementation.

Open and collaborative process of evaluation planning

Adapting educational concepts and evaluation methods

for use in this context, with these stakeholders

Stakeholder diversity and interests Member agencies of the HousingPlus network Memorandum of understanding to identify participating

agenciesStaff members of agencies (executive directors/senior

managers, front-line staff) Determining agency capacity to contribute to the

evaluationTenantsAddressing barriers to participationEvaluatorsIdentifying unique opportunities for stakeholders to

contribute to funding, planning and launching evaluationFunders

Identifying ways of engaging funders and other partnersOther partners

Justification for engaging stakeholders For evaluation Open and collaborative process of proposal development

and evaluation planningImproved evaluation findingsTransparency and accountability of evaluation teamIncreased evaluation utilizationCreation of structures for participation of various

stakeholders

For stakeholders

Development of strategies for ongoing engagement of

agencies, staff, tenants, and other stakeholders

Improved collaboration, strengthened network

Improved professional identities (staff)

Improved learning about the program (staff)

Improved leadership and communication skills (tenants)

Enriched understanding of collaboration, developing

facilitation skills (evaluators)Increased benefits of housing and support to tenants

Improved achievement of program outcomes

Roles for stakeholders Members of Steering Committee Collaborative Community

(executive directors/senior managers, evaluators)

Distinguishing between roles for stakeholders with

different expertise, in the development of evaluation

tools and methodsMembers of Collaborative Community (staff members,

tenants, evaluators) Creation of structures for participation of various

stakeholdersEvaluation coordination/administration, group facilitation,

communication, evaluation technical expertise (evaluators) Creation of roles for various stakeholders

Funders Recruitment of additional stakeholders in evaluation

Consumers of products/knowledge (funders, partners) Network seeks additional project funding

Information session for staff and tenants

Project launch to engage other stakeholders

J. Sylvestre et al. / Studies in Educational Evaluation 34 (2008) 212–217 213

Issue analysis

Context of the evaluation

Social housing in Canada has witnessed decreasing funding overthe past 20 years (Drummond, Burleton, & Manning, 2004;Hulchanski, 2002) leaving meager resources for social housingagencies to properly maintain housing, offer competitive wages tostaff, and to invest in staff and organizational development.Supportive housing is a particular form of social housing that offershousing and support to enable people to live more independently.The housing is typically congregate settings (such as largeconverted homes), and clustered apartments (in which all or alarge number of units in a building are dedicated to an identifiedpopulation). It also offers professional support that is linked, inwhole or in part, to residency in the dwelling and that focuses onrehabilitation, group decision making, life skills, and conflictresolution (Parkinson, Nelson, & Horgan, 1999).

The supportive housing model has been challenged by theemergence of a second approach known as supported housing.Supported housing emphasizes independent apartments scatteredthroughout a city with individualized and portable support (i.e.,support that is not attached to the dwelling, but that follows the

individual when he or she changes housing) (Carling, 1993).Supported housing has garnered increased research support(Gulcur, Stefancic, Shinn, Tsemberis, & Fischer, 2003; Tsemberis,Gulcur, & Nakae, 2004). In contrast, the collectivist philosophy ofsupportive housing has not yet been clearly articulated, nor haveits particular benefits for tenants been as rigorously documented.There is a need to better define the nature of supportive housing asa social program to ensure that it is optimally implemented, as wellas to support better quality research to determine its benefits fortenants.

The HPCC project is a collaboration among university-basedevaluators and HousingPlus, a network of supportive housingagencies in Ottawa, Ontario. The member agencies support peoplewho have difficulties finding and keeping housing includingwomen and children fleeing violence, people with drug or alcoholissues, people living with HIV/AIDS, young single parent families,and people with serious mental illness. HousingPlus has existed forover a decade in Ottawa. Agency representatives at the networkinclude agency executive directors or senior managers, andnetwork leadership is shared by participating members. Whereasmembership is open to all supportive housing providers in the city,about 10 agencies have been more consistent participants in thenetwork.

J. Sylvestre et al. / Studies in Educational Evaluation 34 (2008) 212–217214

In recent years, network members have identified the need formore research on its programs, as well as a desire for morecollaborative network efforts. Since 2005 HousingPlus has workedwith university-based evaluators to identify opportunities to worktogether to meet the needs of the network. HPCC was jointlydeveloped by HousingPlus and the university-based evaluators tosupport the HousingPlus network to better understand, to bettercommunicate about, and to improve the delivery of its programs.As such, the goals, form and process of HPCC are a reflection of theneeds, interests, and expertise of these two groups.

The evaluation approach and methods: implementation evaluation as

a foundation for collaborative practice innovation development

HPCC employs implementation evaluation to describe andunderstand how supportive housing programs are currentlyimplemented in Ottawa, and as a basis for developing innovationsto improve program delivery. The approach adopted in this workbuilds on concepts familiar to educators. First, it builds onprofessional learning communities (PLCs), which are groups ofeducators who collaborate to create a vision for change ineducational settings and who work towards shared solutions thatare then implemented to observe their effects (DuFour & Eaker,1998; Hord, 1997; Westheimer, 2008). HPCC will function as avenue within which participating HousingPlus agencies cancompare how they deliver their programs and learn from eachother how to improve program delivery. HPCC differs from PLCs inthe nature of the collaborative learning it envisions. Whereas PLCsare often collaborations of teachers within a particular school,HPCC is a collaboration involving individuals from a number oforganizations who participate in the HousingPlus network. Inaddition, HPCC also seeks to engage a broader range ofstakeholders in the collaborative learning, including the executivedirectors and senior managers who have historically participatedin the HousingPlus network and front-line staff members and thetenants of the housing programs.

HPCC also builds upon multi-dimensional innovation profiles

developed by Leithwood and Montgomery (1987) to evaluate theimplementation of educational curriculum. Cousins and colleaguesadapted innovation profiles for the study of intensive casemanagement in community mental health (Cousins, Aubry, SmithFowler, & Smith, 2004). The resulting key component profiles (KCP)approach involves developing a multi-dimensional description ofprogram implementation in key domains in the delivery of aparticular program. For each domain, implementation objectivesare specified along with scales that describe levels of currentprogram implementation ranging from low implementation to fullimplementation. The KCP approach enables programs to specify acurrent level of implementation as well as to identify the specificways for programs to achieve higher levels of program imple-mentation.

The KCP approach offers a number of advantages to the HPCCevaluation. First, the detailed description of program implementa-tion will be valuable for describing an intervention as complex assupportive housing, as well helping the various stakeholders tocome to agreement on how specifically the programs should beoptimally implemented. Second, by specifying levels of imple-mentation and by describing what higher levels of implementationshould look like, the tool will also support the various stakeholdersin identifying strategies for improving program implementation inthose areas where there are currently implementation challenges.

HPCC will support a comparison of program implementationacross participating agencies by specifying common programcomponents and by elaborating the different levels of implemen-tation for each of these components. The evaluators and the

HousingPlus network will collaborate in developing a KCP tool forevaluating the implementation of supportive housing, and willthen use the tool to evaluate current program implementationacross the participating agencies. Evaluation findings will then beused by stakeholders to collaborate on identifying and addressingshared implementation challenges.

The project is divided into four phases. Phase I is a recentlycompleted planning process that lasted almost 2 years. It includedthe initial engagement of stakeholders, project conception, thesearch for funding, and planning for project implementation. PhaseII involves collaborative efforts to (1) develop a common set ofvalues to guide decisions concerning optimal practice in thedelivery of supportive housing, (2) identify and describe the criticalpractices in the delivery of supportive housing programs, and (3)develop a KCP tool to evaluate the current implementation ofsupportive housing by participating agencies. Phase III will involvethe use of the evaluation tool to evaluate current implementationof supportive housing by network agencies. Phase IV will involveuse of evaluation findings to develop solutions to sharedimplementation challenges and support agencies in adoptingthese innovations.

Program and evaluation stakeholder diversity and interests

There are a number of stakeholder groups associated withsupportive housing in Ottawa, that bring different experiences andexpertise to the project, as well as having different incentives andmotivations for collaborating. The stakeholders include theagencies with interests in supportive housing, various staffmembers within these agencies, and tenants living in a supportivehousing setting. The agencies’ contributions to this work includethe investment of staff time and resources in the project. Individualstaff members (including executive directors, senior managers,and front-line staff) contribute in terms of their time and expertise.The findings from this work will likely have direct implications forhow agencies and staff perform their work. The agencies and theirstaffs are also motivated to demonstrate to funders the value ofthese housing programs, and to show their capacity to engage incollaboration to address systemic challenges. The tenants, as theintended beneficiaries, have unique perspectives on the goals ofthe programs and how they can be optimally delivered. Theirincentives for participating lie in the promise of improvedprograms, as well as in the desire to contribute to programs fromwhich they have personally benefited.

Other stakeholders include program funders and programpartners. The participating agencies receive funding from a varietyof federal, provincial and municipal funders, each of whom areinterested in seeing the program optimally implemented in a waythat reflects increased systemic integration. Other partners, includ-ing organizations offering comparable programs may be invested inlearning how they may adapt findings from this project in their ownwork. The university-based evaluators who intend to publish toolsand findings from the project are also stakeholders. These evaluatorshave dedicated a considerable amount of time and resources to theproject and, in line with university community incentives, aremotivated to produce publishable findings and to contribute tocollaboration theory in the community/social service sector.Additionally, they are keen to contribute in direct ways to fielddevelopment within the community of practice through collabora-tion with community partners.

Justification for stakeholder engagement in the evaluation

Following from a consideration of the diversity of interests ofthe various stakeholders in the work of the HPCC, are the

J. Sylvestre et al. / Studies in Educational Evaluation 34 (2008) 212–217 215

justifications for engaging them in this evaluation. A considerationof both the diversity of interests among stakeholders and thejustifications for engaging them in evaluation can be useful forsubsequent considerations of the roles that stakeholders may playin the evaluation. Some of the justifications can be found in thebenefits that these stakeholders will gain from the evaluation,either from the results of the evaluation or as a direct or indirectoutcome from the evaluation process or their participation in theevaluation, otherwise known as process use (Patton, 1997; Cousins,2007). Process use implies the development of knowledge andskills as a result of proximity to evaluation and quite independentof the findings of the evaluation.

The approach that this evaluation has taken is aligned withpractical participatory evaluation, as opposed to a more transfor-mative approach guided by principles of social justice (Cousins &Whitmore, 1998). Practical participatory evaluation represents apragmatic, problem-solving orientation to collaborative evaluationwhere the chief concern is the creation of meaningful evaluationknowledge that will be useful in supporting program decisionmaking. It is distinguished from transformative participatoryevaluation, which is typically normative in form and intent andguided by principles of social justice such as the amelioration ofsocial inequity (Cousins & Whitmore, 1998).

Within the framework of practical participatory evaluation thereare a number of benefits that might be expected from engagingmultiple agencies at multiple levels. First, multiple agencies areengaged in order to increase the shared pool of experience andexpertise in the description of optimal program implementation andin the development of strategies for improving practice. Multipleagency perspectives can aid in the development of evaluationmethods and tools that are valid in so far as they capture thecomplexity of the program from several distinct but complementaryperspectives. Second, engagement at each of these levels canfacilitate the actual evaluation of program implementation bygauging multiple perspectives on program successes and challenges.Third, this form of engagement can promote the use of evaluationfindings, as well as stimulating broader organizational change.Because individuals at multiple levels of an organization haveparticipated, the evaluation findings will be likely to have greaterconsiderable relevance to a large constituency that is invested in theevaluation and committed to acting on its findings.

The engagement of multiple agencies at multiple levels is alsoexpected to have benefits for the agencies and participantsthemselves. In the education sector, PLCs have been credited withovercoming the ‘‘egg carton’’ atmosphere of schools in whichteachers tend to focus on what transpires within their ownclassrooms rather than on the goals they share with teachers in theclassrooms that surround them (Westheimer, 2008). Supportivehousing organizations encounter the same challenges associatedwith work demands that are compounded by chronic under-funding of these programs. This initiative can provide a foundationfor further collaborations between agencies to address othershared challenges. Such a benefit would reflect a collective form ofprocess use that might enable future program and/or organiza-tional problem solving.

For staff members, there is the potential benefit that isassociated with enhancing their professional work identities byinteracting with other staff members and contributing toevidence-based discussions of program improvement. Suchphenomena have been observed with PLCs of teachers (DuFour& Eaker, 1998; Westheimer, 2008). The potential for learning aboutthe program (context, logic, implementation challenges, effects),would be enhanced in addition to corollary knowledge and skillbuilding. In this way we would expect both use of findings andprocess use to occur. Tenants may benefit from opportunities to

give back by helping to improve programs from which they havebenefited. These experiences may help to develop communicationand leadership skills that could have an empowering effectgeneralizing to other areas of their lives, another instance ofindividual-level process use.

Roles for stakeholders in the evaluation

In this evaluation, the evaluators have proposed adapting a PLCapproach to engage multiple supportive housing agencies atvarious levels—senior management, front-line staff, and tenants.First, the evaluators proposed a steering committee comprised ofexecutive directors or senior managers of the participatingagencies. The steering committee has participated in the planningof the project. It will oversee the implementation of the project, aswell as contribute the development of evaluation tools andmethods. Steering committee members will promote the projectwithin their respective organizations and will support theevaluation across and within their own agencies.

Second, in adapting a PLC approach, the evaluators proposedbroadening participation to include the tenants of the housingprograms. These individuals are dependent on the programs, andvulnerable to the changes that are made to them. They have uniqueperspectives on what works and what does not work within theprogram context. In this project, the evaluators and the steeringcommittee agreed to have tenants of the housing and front-linestaff members form a single learning group, distinct from the groupincluding executive directors and senior managers group. Thefront-line staff members and tenants will develop and discussvalues and principles that should underlie the implementation ofthese programs, and will document their perceptions of whatconstitutes good practices in supportive housing. These discus-sions will help to identify the key components of the delivery ofsupportive housing for the KCP tool.

The steering committee decided in favor of a separate group sothat discussions would be free-flowing and uninhibited by thepresence of organizational leaders. Furthermore, both tenants andfront-line staff members have an intimate knowledge of theprogram through their day-to-day experiences with it. Mostnotably, most of their experiences are with each other. Therefore,each group can shed light on issues arising from their interactionswith each other. The inclusion of these two perspectives in a singlegroup also provides an opportunity to identify, discuss, and wherenecessary, resolve differences in perspectives of staff and tenantson challenges and opportunities for improving the implementationof supportive housing.

The evaluator stakeholders have assumed all project manage-ment and administrative responsibilities. Theyplay a facilitation rolefor both the steering committee and staff/tenant group, and serve asthe conduit for information moving between the two groups. Theywill also ensure that the project aligns with professional standards ofevaluation practice such as the Program Evaluation Standards (JointCommittee for Educational Evaluation, 2007).

Engaging stakeholders in planning HPCC

The HPCC project has recently completed its planning stage and isnow being launched. Although these are early stages, the process ofinvolving stakeholders has beenongoingforalmost2 years.Thistimehas been critical for engaging stakeholders, and ensuring theirinvolvement and their commitment to the project. In the followingparagraphs we describe the strategies used to engage stakeholders inthis work. In Table 1, we link these strategies to the issues instakeholder engagement thatwere identifiedin the previous section.

J. Sylvestre et al. / Studies in Educational Evaluation 34 (2008) 212–217216

Mutual stakeholder engagement

Planning for this project stretched over 2 years. The initialinteractions between the evaluators and other stakeholdersreflected a process of mutual engagement. Recognizing the needto become involved in research and evaluation, but with a limitedability to proceed on its own, the HousingPlus network contactedthe university-based research and evaluation center with whichfour authors of this paper are affiliated. Over a number of months,the first author met with the network members to discuss ways oflinking network research needs and interests to the interests andexpertise of the researcher. The network initially expressed adesire to describe its programs to its funders and partners, as wellas to collect stories of tenants testifying to the value of supportivehousing. The researcher was interested in building on participatoryresearch previously conducted with supportive housing providersin another city focused on developing benchmarks for supportivehousing (Sylvestre, Ollenberg, & Trainor, 2007). This process ofengagement led to a reciprocal accommodation of each group’sinterests, and ultimately a project that was a joint product.

Funding search and proposal development

Funding for this evaluation was sought from an externalacademic source, creating some potential risks for the collabora-tive nature of the project. Although numerous proposal drafts werecirculated and discussed at regular meetings of the evaluationteam and the HousingPlus network, the proposal was primarilywritten by the evaluation team in an academic voice, emphasizingthe theoretical and technical aspects of research. This proposalmay not have reflected all the interests and objectives of theagency stakeholders. Because of the nature of the granting councilthat funds university-based research, the proposal was alsosubmitted with the academic evaluators serving as the investiga-tors and managing the project’s financial resources.

An open process for developing the project work plan andbudget helped to mitigate these challenges. This process helped todetermine the extent to which agencies could participate withoutplacing too much strain on them, and to clarify the meaning ofparticular unfamiliar educational and evaluation terms. A chal-lenge at this stage was determining which network memberswould commit to the project. Whereas the participation of someagencies was consistent throughout the planning process, otherstook part less reliably. In the end, network members developed andcirculated a draft memorandum of understanding to clarifyexpectations for agencies participating in the project, whichhelped to identify those agencies that were committed toparticipating in the project.

During this phase, network members took the initiative toactivate other stakeholders to secure additional funds. Althoughthe proposal was funded, the budget was cut substantially by theexternal academic funder. This provided an opportunity for thenetwork to submit a small request for funding to a municipalfunder to cover, in part, agency costs from participating in theevaluation. The process helped to solidify the collaborative natureof the project by providing a role for the network in solicitingfunding. The network members indicated that this was also aunique investment by the city in their participation in evaluation,and credited the university-network partnership and the academicfunding as keys to increasing their status and credibility.

Project planning

Once funding was received, a planning process was launched torefine the work plan. Network members formed a project steering

committee and became actively involved in planning all aspects ofthe evaluation. The evaluation team facilitated the project steeringcommittee meetings. The evaluators maintained transparency andaccountability in the role of facilitator by tracking progress using aproject-planning chart. This chart listed all areas that requiredplanning decisions, the decisions that were made, and responsi-bilities and timelines for acting on the decisions. The evaluatorsdiscussed with the steering committee their research interests inthe collaborative process of the project, academic writingprospects, and opportunities for including other stakeholders inthis work if they were interested. In the case of this article theevaluators circulated the call for papers and the article proposal,and invited interested individuals to contribute to the article. Anumber provided feedback on a draft, with one providingsubstantive contributions leading to co-authorship. The involve-ment of steering committee members extended to the develop-ment and review of a submission to the university’s ethics reviewboard. This provided an additional opportunity to clarify respon-sibilities of the evaluators towards participants, and to clarify theanticipated research and evaluation activities.

Key issues discussed at this stage included strategies forengaging staff and tenants in the project and ensuring that theprocess was meaningful for them. Network members raised issuesrelated to engaging the various vulnerable populations housed bythe agencies in the evaluation. These issues included tenantconfidentiality, literacy, voluntary participation, and the skillsrequired for effective participation. These discussions led toguidelines for stakeholder recruitment in the evaluation asmembers of a staff/tenant collaborative community. In addition,an information event was jointly held by representatives of thenetwork and the evaluation team with prospective staff and tenantparticipants in the project to inform them of the project and thenature of their participation should they elect to become involved.

During this phase, steering committee members took the leadin planning a launch event that paired the launch of the evaluationwith the launch of the HPCC network’s new website and marketingefforts. The event was conceived as an effort to begin to engageother stakeholders (e.g., municipal funders, partners, members ofboards of directors) in the project and to increase their awarenessof the network’s goals and activities. The event was also attendedby staff and tenants who had expressed an interest in participatingin the project. Ultimately the launch ran successfully with a goodturnout and local media coverage.

Challenges for the future

There are a number of challenges that HPCC will have toconfront as the network and the evaluators continue to work toengage the organizations, staff, tenants and other stakeholders inthe project. First, how do we best engage a larger number of staffand tenants who do not directly participate in the staff/tenantgroup? Greater participation by members of these organizationswould potentially aid in understanding implementation chal-lenges and implementing improvements. Initial discussions withthe steering committee have centered on site visits by theevaluators to the organizations to present the project and solicitwider participation. Other strategies will involve working withboth the steering committee and the staff/tenant group to promotethe work in the agencies and to gain input from other stakeholders.

The efforts to create collaboration among multiple organiza-tions at multiple levels means that there may be challenges inmanaging discrepancies in power, particularly when there aredivergent perspectives across types of participants (i.e., seniormanagement, staff, tenants). Issues may also arise due to thediversity in the participants in the study. The participating

J. Sylvestre et al. / Studies in Educational Evaluation 34 (2008) 212–217 217

agencies serve different populations. This necessarily means thatthere some differences in the programs they offer, and perhapseven in the underlying core values. These differences haveimplications for the development of a common vision forsupportive housing and a common set of optimal programpractices.

Another challenge is a practical one. Program implementationwill be evaluated across eight different programs. This will requirethat evaluators become familiar with and collect data in each ofthese program settings, which may be challenging given the rangeof populations served and the varied nature of programs offered. Itis expected, however, that using the KCP tool will help address thischallenge by identifying program components that were endorsedas common to all participating programs.

Other challenges to be addressed relate to the inclusion of thepartner agencies that support HousingPlus network members indelivering their programs, by, for example, filling propertymanagement, property maintenance, or professional supportfunctions. The evaluators and network members will have toaddress the extent to which these partners participate as sources ofdata, users of evaluation findings, or full evaluation partners.Questions have also arisen from potential stakeholder agencieswho have not been part of the process as to where and how canthey fit in. This issue, like the one before, represents the challengeof managing the demands of the evaluation so that it does notbecome unwieldy. It reflects the need to work towards creating acollaborative group that ensures active participation and sig-nificant relationship-building while also ensuring that it issufficiently varied to include the skills needed to accomplish thegroup’s tasks.

Conclusion

Although in its early phases, this work has shown the value ofengaging stakeholders from the beginning of the process of anevaluation, and working towards mutual understanding of allstakeholders, including the evaluators. This involves not onlyunderstanding the contributions of direct participants conductingthe evaluation, but also appreciating the need to engagestakeholders who are not direct participants in doing theevaluation. A key strategy employed in this planning work hasbeen an open and transparent process in which the evaluatorsremain accountable to other stakeholders in facilitating theevaluation planning.

In this paper we have outlined five factors we considerimportant to consider when engaging stakeholders in evaluation:(1) the context of the evaluation; (2) the evaluation approach beingused; (3) the diverse nature and interests of the stakeholders; (4)the justifications for engaging stakeholders in the evaluation; (5)the roles for stakeholders who collaborate in the evaluation.Although there are certainly more challenges to be experienced inthis work, careful attention to these five factors will, in our view,

help to ensure the employment of stakeholder engagementstrategies that are relevant, effective, and meaningful to allparticipants throughout the process.

Acknowledgements

This work was funded by grants from the Social Sciences andHumanities Research Council and the City of Ottawa. We thankHousingPlus agencies for their support of this work, and MatthewManion, Katie Bendell, and Jaclynne Smith for their contributionsto this project.

References

Carling, P. (1993). Housing supports for persons with mental illness: Emergingapproaches to research and practice. Hospital and Community Psychiatry, 44,439–449.

Cousins, J. B. (Ed.). (2007). Process use in theory, research and practice. New directions forevaluation. No. 116. San Francisco: Jossey Bass.

Cousins, J. B., Aubry, T., Smith, M., & Smith Fowler, H. (2004). Using key componentprofiles for the evaluation of program implementation in intensive mental healthcase management. Evaluation and Program Planning, 27, 1–23.

Cousins, J. B., & Whitmore, E. (1998). Framing participatory evaluation. In E. Whitmore(Ed.), Understanding and practicing participatory evaluation. New directions in eva-luation, No. 80 (pp. 3–23). San Francisco: Jossey Bass.

Drummond, D., Burleton, D., & Manning, G. (2004). Affordable housing in Canada: Insearch of a new paradigm. In J. D. Hulchanksi & M. Shapcott (Eds.), Finding room:Policy options for a Canadian rental housing strategy. Toronto, ON: CUCS Press.

DuFour, R., & Eaker, R. (1998). Professional learning communities at work: Best practicesfor enhancing student achievement. Bloomington, IN: National Educational Service.

Gulcur, L., Stefancic, A., Shinn, B., Tsemberis, S., & Fischer, S. (2003). Housing, hospi-talization, and cost outcomes for individuals with psychiatric disabilities partici-pating in housing first and continuum of care programs. Journal of Community &Applied Social Psychology, 13(2), 171–186.

Hord, S. M. (2004). Professional learning communities: An overview. In Hord, S. M.(Ed.). Learning together, leading together: Changing schools through professionallearning communities. (pp. 5–14). New York: Teacher Cllege Press.

Hord, S. M. (1997). Professional learning communities: Communities of continuous inquiryand improvement. Austin, TX: Southwest Educational Development Laboratory.Retrieved September 18, 2006, from http://www.sedl.org/pubs/change34/plc-cha34.pdf.

Hulchanski, J. D. (2002). Housing policy for tomorrow’s cities. Ottawa, ON: CanadianPolicy Research Networks.

Joint Committee for Educational Evaluation Standards (2007, October). The programevaluation standards (3rd. ed.) DRAFT #3. Thousand Oaks, CA.

Leithwood, K. A., & Montgomery, D. J. (1987). Improving classroom practice usinginnovation profiles. Toronto: Ontario Institute for Studies in Education Press.

Parkinson, S., Nelson, G., & Horgan, S. (1999). From housing to homes: A review of theliterature on housing approaches for psychiatric consumer/survivors. CanadianJournal of Community Mental Health, 18, 145–164.

Patton, M. Q. (1997). Utilization-focused evaluation: A new century text (3rd ed.).Thousand Oaks, CA: Sage.

Sylvestre, J., Ollenberg, M. D., & Trainor, J. (2007). A participatory benchmarkingstrategy for describing and improving supportive housing. Psychiatric Rehabilita-tion Journal, 31(2), 115–124.

Tsemberis, S., Gulcur, L., & Nakae, M. (2004). Housing first, consumer choice, and harmreduction for homeless individuals with a dual diagnosis. American Journal of PublicHealth, 94, 651–656.

Weaver, L., & Cousins, J. B. (2004). Unpacking the participatory process. Journal ofMultidisciplinary Evaluation, 1, 19–40.

Westheimer, J. (2008). Learning among colleagues: Teacher community and the sharedenterprise of education. In M. Cochran-Smith, S. Feiman-Nemser, & J. McIntyre(Eds.). Handbook of research on teacher education (pp. 756–783). Reston, VA:Association of Teacher Educators, Lanham, MD: Rowman & Littlefield.