Upload
lamkhanh
View
213
Download
0
Embed Size (px)
Citation preview
1
Mosaic Counselling and Family Services
Evaluation Implementation Grant (EIG - 1167)
Program: Families and Schools Together (F&ST)
September, 2010
2
Executive Summary
The Families and Schools Together (F&ST) program was created in 1988 to prevent school
failure, substance abuse, violence and delinquency in “at-risk children” from stressed, isolated
and low-income families by addressing the needs and providing support to families and
promoting community development1 (McDonald, Billingham, Conrad, Morgan & Payton, 1997, p.
141) and has been offered by Mosaic Counselling and Family Services (Mosaic) for nine years.
For the past several years, F&ST has used a pre-post non-experimental design with two raters
(parent and teacher). The standardized evaluation instruments have included several
quantitative methodologies such as questionnaires and an open-ended questionnaire providing
the family an opportunity to explain how the program impacted their lives. Parents and teachers
completed the questionnaires before the program began and after it ended. The measures were
paired to determine the level of change2. It was discovered that many people, including F&ST
staff as well as school personnel did not feel that the information gathered using these data
collection methods was meaningful to them.
The current evaluation involved using a variety of research methods including quantitative
surveys as well as qualitative open-ended questionnaires, interviews, observation and an
innovative technique called Photovoice. Participants involved in the evaluation activities
included F&ST families, parent partners, school personnel, F&ST staff and volunteers, and
personnel from Public Health. Thirty-three interviews were conducted with individuals across all
capacities/roles of the program and eleven individuals participated in Photovoice.
The findings suggest that the F&ST program has many short-term benefits to those involved
and has been shown to accomplish several of the short term goals including connecting families
to their school, allowing them to meet new people within their community, and is enjoyed by
those who participate including F&ST staff, F&ST volunteers, Public Health personnel and
school personnel as well.
1 McDonald, Billingham, Conrad, Morgan & Payton, 1997 (p. 141) 2 Family Service Canada, n.d.
3
However, according to a description of the program components and roles of staff and
volunteers3, there are disconnects between the documentation and actual practice and
distinctions between actual and desired long term outcomes. It may be that in order to achieve
the longer term outcomes of the program and therefore accomplish all of the goals of the
program, the program may have to adhere more closely to the original model4.
There are many aspects of the program that are experienced as valuable and successful by all
of the participants, including time to connect with family, meet new people, connecting with
school and community, and linking to Mosaic Counselling Services. Further, some of the
successful aspects of the program could be built on in order to provide even better services to
clients.
Using a variety of evaluation techniques, new information and insight has been gathered
regarding the successes, challenges as well as short and long term outcome of the program.
Further, the new evaluation process was more enjoyable for participants than earlier methods
used. Of particular success was Photovoice which all families who were involved in enjoyed.
Photovoice was felt to be fun by parents and children and offered people an opportunity to
demonstrate through images what happens in F&ST. Asking questions about the lives and
experiences of families was less intrusive when families could select the pictures and
experiences that they wanted to talk about.
The experiences and learnings generated from the current evaluation are important to share
with the agency as well as others and there are several recommendations for how to amend the
program as well as prepare for and conduct evaluation in the future. The experience of working
with the Centre has been positive for a multitude of reasons.
From an agency perspective, the in-depth nature of this evaluation project is a first. The
qualitative evaluation methods including participant observation and interviews offered a close
look at the inner workings of a Mosaic program. The results are recommendations for aspects of
the program to retain and aspects to change, based on extensive study and review of the
literature.
3 McDonald, Billingham, Conrad, Morgan & Payton, 1997 4 Ibid
4
Table of Contents
EExxeeccuuttiivvee SSuummmmaarryy ................................................................................................................... 2
IInnttrroodduuccttiioonn ................................................................................................................................ 6
History and Purpose of F&ST .............................................................................................. 6
Overview of the Program and Activities ............................................................................. 6
Some Research on F&ST ..................................................................................................... 8
Past F&ST Evaluation Processes at Mosaic .....................................................................10
Purpose of the Current Evaluation ....................................................................................11
MMeetthhooddoollooggyy .............................................................................................................................11
Methods Used in the Current Evaluation ...........................................................................11
Methodology Limitations ....................................................................................................17
RReessuullttss ......................................................................................................................................18
Strengths of the Program ...................................................................................................18
Challenges of the Program .................................................................................................20
Observational Findings ......................................................................................................25
Previous Evaluation of the Program ..................................................................................26
New Evaluation Process .....................................................................................................26
Other Information ................................................................................................................27
DDiissccuussssiioonn aanndd LLeessssoonnss LLeeaarrnneedd..............................................................................................28
Sharing Information ............................................................................................................28
Learning ...............................................................................................................................29
The Evaluation Experience .................................................................................................29
CCoonncclluussiioonnss aanndd RReeccoommmmeennddaattiioonnss//NNeexxtt SStteeppss .......................................................................32
Recommendations for Others ............................................................................................32
Recommendations for the Organization ............................................................................33
KKnnoowwlleeddggee EExxcchhaannggee ...............................................................................................................36
5
Works Cited……………………………………………………………..……………………………....38 AAppppeennddiicceess ...............................................................................................................................39
Appendix A: F&ST Logic Model and Evaluation Framework ................................................40
Appendix B: Informed Consent for F&ST Families and Parent Partners ..............................43
Appendix C: Interview Consent Form for F&ST Staff and Volunteers, School Personnel,
Public Health Personnel and Mosaic F&ST Staff ...................................................................44
Appendix D: Interview Schedules ........................................................................................45
Appendix E: Recruitment Flyer for Potential Photovoice Participants ..................................49
Appendix F: Informed Consents for Photovoice Participants................................................50
Appendix G: Information and PowerPoint Introduction to Photovoice for Families ...............51
Appendix H: Photovoice Focus Group Recruitment Flyer ...................................................58
Appendix I: Photovoice Focus Group Informed Consent Form ............................................59
Appendix J: One Page Summary for Poster Conference at CHEO ......................................60
6
Introduction History and Purpose of F&ST Lynn McDonald created the F&ST program in 1988 to prevent school failure, substance abuse,
violence and delinquency in “at-risk children” from stressed, isolated and low-income families by
addressing the needs and providing support to families and promoting community development
(McDonald, Billingham, Conrad, Morgan & Payton, 1997, p. 141). McDonald recognized that
traditional mental health services including individual or family therapy often failed to reach ‘at-
risk’ clients and also failed to address the overall impact of one’s environmental context. For
example, factors such as unemployment, violence or alienated communities affect the well-
being of children and particularly their ability to thrive in school. Further, these environmental
factors are linked with later substance abuse, violence, delinquency and school failure. The
F&ST program, then, was created to help build families and neighbourhoods, with a goal of
improving individual and social functioning (McDonald et al., 1997).
Social isolation, low income, unemployment, unsafe neighbourhoods and other environmental
factors can be detrimental to parents as they may come to believe that they are unfit to be
parents and undermine their confidence in their capacity to effectively parent their children.
Connecting them with neighbours within their community allows for these parents to create
natural relationships which they can rely on as a source of support rather than relying on
therapeutic assistance. Community connection in addition to stronger familial connections
creates feelings of empowerment for parents and also works to increase child functioning
(McDonald et al., 1997). According to McDonald (1997) “F&ST is not simply a children and
families program that builds on protective factors, but is a community-building strategy as well”
(p.146).
Overview of the Program and Activities
According to the literature, F&ST is designed to be a two-year program consisting of an eight-
week F&ST cycle and a two year F&STWORKS cycle held immediately afterward (Family
Service Canada, n.d.; McDonald, 1997).
Partner Selection Before a F&ST program can be run in any school, a F&ST team including at least four partners
– “a parent from the local community, two community agency professional representatives (one
from a mental health agency and one from a substance-abuse program), and one professional
7
from the local elementary school” (p. 145) – must be formed. All members of this team must
commit to being involved in the program for the full two years (McDonald, et al., 1997).
Training F&ST teams must receive training on how to run the F&ST program (Family Service Canada,
n.d.). F&ST trainers are also responsible for visiting two or three times during the course of the
eight week program. After the program has ended, the trainer is required to meet with F&ST
staff to reflect on their experiences of the program as well as determine what worked well as
well as what did not work as well in that particular community. Trainers also seek feedback from
program participants about the program and work with them to concretize the needs, issues and
agendas to be addressed in F&STWORKS (Family Service Canada, n.d.).
Participant Recruitment Families are primarily recruited to F&ST through their children: ‘at-risk’ children are identified by
their teachers while in elementary school. Once potential child participants are identified, a
“parent-professional team” which usually consists of F&ST graduates formally invites the family
of that child to participate in the program (p. 141). Inviting parents to participate offers them an
opportunity to make a decision/choice and works to increase their level of self-confidence
(McDonald, et al., 1997).
Program Sessions F&ST sessions are identical in their agenda/schedule with each activity carefully structured and
planned based on empirical research: the F&ST program is founded on the family stress theory
as well as “systematically integrates exemplary practices from both community organizing and
clinical casework and uses positive engagement of families and community to help at-risk
children succeed at school, at home, and in their community” (McDonald, 1992 in McDonald, et
al., 1997, p. 141). Sessions are led by F&ST staff members but to assist them, there are several
additional F&ST team members who take part including a parent graduate of the program, a
member of the school, a social worker and a substance abuse counsellor (McDonald, et al.,
1997, p. 144). F&ST staff members are made available to recruited F&ST families to contact
with questions or concerns during the week between F&ST sessions (McDonald, et al., 1997). A
more detailed description of each of the activities engaged in through F&ST and the theory
behind each is referred to in the F&ST Program Workbook (Family Service Canada, n.d.).
8
F&STWORKS After families graduate from the F&ST program, they are invited to attend F&STWORKS
(meetings held monthly for two years to help sustain the skills and knowledge learned in F&ST).
F&STWORKS is designed to assist parents become more independent as well as offer support
to one another. By the time parents begin F&STWORKS, they should already feel more
connected their child’s school, feel connected with community resources in addition to other
families within their community, and feel more confident in their parenting abilities.
F&STWORKS is initially to be run by F&ST staff and a “parent advisory council of graduates”
(McDonald, et al., 1997, p. 145). During F&STWORKS, families become more self-sufficient in
terms of scheduling meetings and determining the content and topics of the meetings which are
held in neighbourhood clusters. Families increasingly set the agenda of the F&STWORKS
meetings while the parent advisory council monitors the budget and meeting location
(McDonald, et al., 1997).
The eight-week portion of the F&ST program becomes a “bridge for moving new families into
the F&STWORKS program” and “the initial F&ST experience builds interdependency among
participants thus increasing the likelihood that they will join the F&STWORKS program”
(McDonald, et al., 1997, p. 145). The purpose of F&STWORKS is to create a “local association
of parents” so that parents who were not previously involved in their communities or within their
child’s school become more actively involved and feel able to “take more risks” because they
feel supported by others in their new social network (McDonald, et al., 1997, p. 145). As more
and more families graduate from F&ST and join the F&STWORKS portion of the program, the
local community network grows and strengthens. F&STWORKS then, builds families’ capacity
to become leaders within their own community (McDonald, et al., 1997).
Some Research on F&ST The majority of research done to evaluate F&ST and its outcomes has been conducted by the
creator of F&ST, Lynn McDonald. She is well published on the outcomes of the program
including long-term outcomes and has compiled findings across program sites. Also, in
partnership with other authors, she has researched the impacts of F&ST on various cultural
communities including Latino communities (2006), Aboriginal communities (2003) and on
teenaged single mothers (2008) in the United States. Findings suggest that the program is
beneficial for various cultural communities and the research has also allowed cultural
considerations for the program to be integrated. While there is also literature on several of the
9
sites in Canada that have implemented the F&ST program (Annual Program Report, 2006-2007;
The Family Centre Annual Report, 2007-2008; The Vanier Institute of the Family, 2006), it is
difficult to locate and there is not a lot available.
Based on the literature that is available, it is evident that the F&ST program has long-lasting
outcomes and indeed benefits the families that are involved in the program. For example, in
Manitoba, the F&ST program has been running since 1997 and program staff have reported the
outcomes of the program on families, including: increased family cohesion, reduced social
isolation, and increased connections to the school. Further, participants of the Manitoba F&ST
felt that they had become included in greater support systems and had increased involvement in
their child’s school (i.e. as volunteers, on parent council, etc.) (The Family Centre, 2007).
An annual report from Winnipeg (Families and Schools Together, 2007) outlined similar findings
and also dedicated an entire section to the developments, activities and outcomes of the
F&STWORKS portion of the program specifically. For example, in this F&STWORKS program,
the F&STWORKS (F&ST graduates) parents were supported in: developing a voluntary Parent
Advisory Council for F&ST consisting of parent partners from numerous communities offering
F&ST; developing a manual which includes “guidelines for organizing a meeting, managing the
financial affairs of a volunteer group and a list of possible activities for the monthly
F&STWORKS meetings” (p. 1, App. C); developing job descriptions and assigning responsibility
for planning and implementing F&STWORKS at various sites; having F&STWORKS
coordinators problem solve with issues and concerns at F&STWORKS get-togethers, and;
having F&STWORKS coordinators attend training in the Parents Assist Learning and Schooling
workshop series developed by the Canadian Home and School Foundation to learn how to
“Parent with Confidence”. In some instances, where city-wide events have taken place for
families in F&STWORKS, over 300 parents and children have attended.
An executive summary including the Canadian National Evaluation of Outcomes 1996-2004 put
together by Family Service Canada (2004) reported that data from 187 program cycles across
Canada showed the F&ST program provides immediate positive outcomes. These outcomes
include improved behaviour of children, improved emotionality of children, increased familial
cohesion, reduced social isolation of parents, and increased parental involvement in their child’s
school.
10
An article by McDonald and Sayger (1998) investigated the longer term impacts of the F&ST
program on single mothers, looking at outcomes at six months and two years after program
cessation. The authors found that the program indeed had long-term positive impacts and,
particularly for parents, the program provided social supports and empowered them to become
involved, in some instances, in leadership roles (i.e. Parent Advisory Council for F&ST). This
finding is important because “disenfranchised, poor, single mothers” were the program
participants involved (McDonald & Sayger, 1998, p. 83).
In 2009, several outcomes of children and families in an experimental study were investigated
(Kratochwill, McDonald, Levin, Scalia & Coover, 2009). Children with behavioural issues from
kindergarten to grade three were assigned to either the F&ST program or continuing with school
special education and assessed immediately and at one year following program cessation. The
findings indicated that the F&ST program participants faired better in terms of family
adaptability, and improved behaviour of children. Further, only one child involved in F&ST
required special education services in the future. The authors suggested that the program
results in some positive short-term outcomes for children.
The research on the F&ST program reveals that the outcomes are positive and both short- and
long-term outcomes are evident.
Past F&ST Evaluation Processes at Mosaic For the past several years, F&ST has used a pre-post non-experimental design with two raters
(parent and teacher). The standardized evaluation instruments have included: the Behavioural
and Emotional Rating Scale (BERS); the Family Adaptability and Cohesion Evaluation Scales
(FACES III); the Isolation Subscale of the Parenting Stress Inventory; the Witte Parent Survey,
and; the Families and Schools Together Program Evaluation by the Family, an open-ended
questionnaire providing the family an opportunity to explain how the program impacted their
lives. Parents and teachers completed the BERS, FACES III, and a survey on parent
involvement before the program began and after it ended. The measures were paired to
determine the level of change (Family Service Canada, n.d.). F&ST staff collected the data, and
in the past, a consultant analyzed and reported the data.
11
Purpose of the Current Evaluation With the help of the Centre of Excellence, Mosaic has had increased resources to implement a
more thorough and meaningful evaluation of the F&ST program. The challenges with the past
evaluation approach for the F&ST program were twofold: the approach was felt to be
inadequate in reflecting the nature of the program (collaborative and relationship-based), and
the findings were not reported in ways that could be easily understood by program staff
(intended users). Up until recently, a consultant would analyze and report the results in a short
report that included each school’s results. The results were shared in a PowerPoint slide during
F&ST training sessions, but little attention was generally paid to school reports. The quantitative
outcome data that was produced, while positive, was not found to be particularly useful to staff.
Recently, a teacher mentioned to a F&ST staff member that the surveys she completed did not
allow her to capture the stories of success she has witnessed in the program. The current
project sought to improve the approach to evaluation in the F&ST program and increase staff
and participant interest in being involved in evaluation activities. With the resources from the
Centre of Excellence, Mosaic re-engaged F&ST staff, partners and participants in implementing
evaluation and broadly sharing the results with staff, participants, partners, community, funders
and potential funders. Involving F&ST participants was an important part of connecting goals of
the program with evaluation.
Methodology Methods Used in the Current Evaluation The evaluation of F&ST for the Fall of 2009 incorporated several new methodologies not
originally included as part of the evaluation process including interviews and Photovoice, each
of which will be discussed in more detail.
A research design was developed in September 2009 and data collection took place from
October 2009 until May 2010. Both “methodological” and “investigator” triangulation were used
(Patton, 1980, p. 108): several methodologies were used to collect data and several
researchers were involved in collecting data (observation data, interview and Photovoice data)
respectively. A logic model for the F&ST program and an evaluation framework is included in
Appendix A. The evaluation methodology was designed to answer the research question “does
the F&ST program result in the changes it was designed to?” as well as a process evaluation
12
question, several evaluation questions directed at measuring short-term outcomes and several
evaluation questions directed at measuring long-term outcomes (see Appendix A).
.
Data was collected by several researchers (4) in the Research and Development Department at
Mosaic. Interview data was collected using purposeful sampling, the intent of which was to
gather a “breadth” (rather than a “depth”) of knowledge (Patton, 1980, p. 98) Researchers were
interested in gathering a breadth of knowledge to ensure that participants involved in the
evaluation represented all roles within the F&ST program (i.e. F&ST staff at Mosaic, F&ST staff
and volunteers, parent partners, Public Health personnel, teachers and other school personnel,
and past F&ST families) and as such, evaluation participants were purposefully selected from
each of roles (rather than hearing a lot of information from individuals from only a few program
roles). Another purpose for gathering a breadth of knowledge rather than a depth of knowledge
in this evaluation was to help determine what information should be gleaned by using a more in-
depth knowledge collection strategy in future evaluations of the program.
Researchers were responsible for analyzing and reporting interim and final data. It is important
to note that all participants that were invited to be interviewed accepted – none declined.
Participant Observation In the Fall of 2009, one researcher received the two-day training necessary to become a
certified F&ST volunteer which helped to develop an understanding of what the F&ST program
sessions included and the roles of all F&ST personnel. The two-day training assisted in this data
collection as increased knowledge was generated prior to observation regarding program
design, how it was to be implemented and the roles of the persons involved in the program.
Detailed information about how the program should be implemented as well as how effective
program implementation was taught in training was shared with other researchers and
afterward, several members of the research team observed the program running at several sites
throughout the Fall of 2009. The objective of this data collection strategy was to purposefully
sample each activity of the program to observe program participants and staff behavior. Greater
emphasis was placed, however, on “tracking” F&ST staff (Mosaic staff, program staff and
volunteers) with the purpose of following staff “around during their usual daily routines and
watching their activities and the other people they interact with” (Berg, 2007, p. 196) in order to
learn how closely the program matched in practice with training. With the research question
13
and evaluation questions in mind, program participants were observed to see if changes took
place over the course of the program (i.e. more interaction between family members, parents
providing instructions to children, etc.) and staff were observed to determine how closely the
program matched the intended model (i.e. staff whispering instructions to parents, etc.). In
addition, several researchers, with participant permission, took photos and video clips during the
program as a form of visual data. This was done while attending several of the actual program
sessions and photographing the activities engaged in by program participants and staff in order
to share this information through a different form of media (i.e. pictures and videos). This data
collection method allowed the researchers not only to collect photographic information on the
program, but also to observe numerous sessions of the F&ST program in addition to a
F&STWORKS session running in actual practice. Further, this data, in conjunction with the data
collected through interviews and the training offered for F&ST staff, gave a more accurate
understanding of what the program should look like in practice.
With regard to the F&ST program sessions, several researchers observed numerous F&ST
sessions in the Fall of 2009 in all four schools offering the program as well as one
F&STWORKS session. In addition, over 200 photographs and videos were collected from the
program sessions.
Data collected through observation were analyzed qualitatively to compare observational data
between researchers and findings were used to augment information gathered through interview
and Photovoice techniques.
Interviews Semi-structured interviews were conducted with a variety of persons involved in the program
including participants whose graduation from the F&ST program spanned several years, F&ST
staff and volunteers, school personnel, Public Health personnel, F&ST parent partners, and
F&ST staff employed at Mosaic. Thirty-three interviews (not including Photovoice participants)
were conducted by researchers in the Research and Development Department at Mosaic.
Interviews were conducted one-on-one via telephone or face-to-face and took approximately a
half hour to complete (see Appendix C and D for Informed Consent documents and Appendix B
for interview questions). Interview questions were reviewed prior to implementation by all
members of the Research and Development Department as well as several members of the
Mosaic F&ST staff.
14
With regard to participant recruitment, in some instances, such as with past F&ST families,
there were many potential families to invite to participate making it impossible to recruit them all
so both purposeful random sampling and convenience sampling were used for these
participants. Names and contact information was gathered for most families that had
participated across the years of the program and contact was attempted for many (purposeful
random sampling). Only the families who could still be reached through the contact information
provided were contacted for an interview (convenience sampling). The researchers wanted to
learn about the short-term impacts of the program as well as the impacts longitudinally which
would involve talking to recent graduates of the program as well as those who had graduated
over the years, but only the families that could still be reached using the contact information that
F&ST staff had were invited to participate. However, in some program capacities/roles, all
individuals (i.e. such as all F&ST staff employed at Mosaic) were interviewed because there
were fewer individuals in that role and it was possible to interview them all.
Data was analyzed using qualitative research techniques (coding) to search for and highlight
themes across evaluation participants. “Individual variation” was sought in how individuals
involved in the F&ST program, in various capacities/roles, felt about the program including its
successes and areas for improvement, and how well the program meets its intended goals
(Patton, 1980, p. 99). Interview participants in many roles (i.e. F&ST families, school personnel,
Public Health personnel, Mosaic and F&ST staff) spanned the numerous years that the program
has been operating and participants represented the various schools involved in the program
(represented several different sites).
Photovoice Data for the F&ST program was also collected using a technique called Photovoice. Photovoice
is a data collection technique that includes offering a camera to a number of participants and
asking them to capture a personal or community experience or issue through pictures (Berg,
2007; Wang & Burris, 1997). Photovoice is most often used to address community issues where
individuals relay messages about their community’s successes or failures on a particular issue
to persons such as government officials through images coupled with a story about the picture
(Berg, 2007; Wang & Burris, 1997). According to Wang and Burris (1997), Photvoice “entrusts
cameras to the hands of people to enable them to act as recorders, and potential catalysts for
15
change, in their own communities” (p. 369). To our knowledge, Photovoice has never been used
to evaluate a program making this a unique way to capture program feedback.
F&ST families participating in the program in the Fall of 2009 from Wilson Public School,
Howard Robertson Public School and Cederbrae Public School were invited to use Photovoice
to capture their experiences of the program. Sixteen families agreed to participate in
Photovoice: six from Wilson, six from Cederbrae and four from Howard Robertson. One
participant from Cederbrae ended up not participating due to irreversible damage done to the
camera provided. Eleven of the 15 families who were involved in Photovoice were interviewed –
it was not possible to connect with the remaining four families. All participating families were
given a digital camera and were asked to share their pictures and offer their stories of their
experiences in the program. F&ST families involved in the Photovoice data collection technique
were also asked the same interview questions as all other F&ST families interviewed during the
evaluation in addition to questions about their experience of participating in Photovoice as a
research methodology.
Photovoice participants were recruited by both Mosaic F&ST staff and researchers from Mosaic:
Mosaic F&ST staff first introduced the Photovoice method to the F&ST families at all three
schools and then introduced the researchers who offered more specific information. Interested
families were invited by the researchers to attend an information session about Photovoice that
would be held at a location near their respective schools (see Appendix F for a recruitment
flyer). Thirteen interested families attended two information sessions about Photovoice and
agreed to participate (see Appendix G for the Informed Consent form for Photovoice). Three
additional participants were recruited separately because they were unable to attend the
information session but were offered the same information as those who did attend (i.e.
informational handouts and informed consent document) (see Appendix H for information
slides).
After the program finished, families were invited to attend a focus group to share their pictures
and “accounts” of the photographs including why the pictures “were taken, what the image
means to the individual, and what they intended the photograph to depict” (Berg, 2007, p. 234).
This technique offers participants the opportunity to share information that they, and not the
researcher, have prioritized (Berg, 2007). Mosaic F&ST staff assisted researchers in arranging
16
the final focus groups and interviews with families (where they shared their pictures and stories)
(see Appendix I for the focus group recruitment flyer).
Six families attended two group gatherings for families involved in Photovoice in their respective
community (i.e. all families from Wilson school met at a location near the school) – families
shared their photographs and stories individually with a researcher from the Research and
Development Department and not in a group setting for purposes of confidentiality.
Approximately eight children were also involved in sharing their family’s pictures and stories and
commenting on their perspectives of the F&ST program (answering interview questions). One-
on-one interviews were conducted with the remaining five families (see Appendix J for the
Informed Consent document for Photovoice focus groups/interviews and Appendix B for
Photovoice and interview questions). Children were invited to attend the focus groups or
interviews. There was a plan to offer these children the option of a book to draw in to capture
their stories and experiences of F&ST while their families shared and discussed pictures.
However, these books were never used because all of the children in attendance at the focus
groups participated with their families in sharing pictures and stories.
Photovoice data was analyzed using participatory action research techniques. Families were
provided a camera to collect data by taking photographs of what they believed would indicate
successes and areas for improvement for the program. Families selected the photographs that
they wanted to share and discuss with researchers at Mosaic. Photovoice also allowed an entire
family to be involved in the data collection and analysis process – all family members could take
pictures of what they felt were important aspects of the program and share the pictures that they
wanted to. There were several instances in which a whole family was involved in data collection
as well as with sharing the pictures with researchers. The stories shared by families were
captured like an interview: semi-structured questions were asked including the same questions
as for other F&ST families, but additional questions were also posed that were directly related to
Photovoice to gather information on how families felt about this particular data collection
technique. This data was analyzed using qualitative analysis (coding) to search for common
themes across Photovoice participants, and compared to information provided across all other
F&ST families interviewed. According to Berg (2007) it is often the case that the Photovoice
technique is used in conjunction with other research techniques such as photographs taken by a
researcher, and also in this case, used in addition to interviews.
17
Surveys Some, although not all, of the standardized instruments originally used to evaluate the F&ST
program were also included in the current evaluation. Some of the survey instruments have not
been used for the last several years due to poor response rates from participants (i.e. a lack of
understanding about the meaning of the questions particularly with participants whose first
language was not English, difficulty with getting participants to complete the surveys, etc.). The
instruments included in the current evaluation were the Behavioural and Emotional Rating Scale
(BERS) filled out by teachers before and after the program through which the measures were to
be paired to determine the level of change; the Family Adaptability and Cohesion Evaluation
Scales (FACES III) (also pre-post test comparison) filled out by families, and; the Families and
Schools Together Program Evaluation by the Family which is an open-ended questionnaire
affording the family an opportunity to explain how the program impacted their lives. The
measures were to be paired to determine the level of change. F&ST staff collected the data to
be analyzed. Copies of the surveys have not been included in this report for copyright reasons.
Methodology Limitations The purpose of a triangulated data collection strategy including interviews, observational
techniques and cumulating all quantitative data was to answer the research question “does the
F&ST program result in the changes it was designed to?” according to the aforementioned
description of the program, program goals, and intended outcomes. It was also used to make
the data collection and findings more interesting, meaningful and useful for all participants
involved in the evaluation, particularly F&ST families, F&ST staff and volunteers and F&ST staff
employed at Mosaic.
One limitation of the current evaluation methodology would be the purposeful sampling
technique used. Specifically, it was of interest to gather a breadth of knowledge (to hear from
many individuals with various perspectives of the program - from staff to participants to others in
the community) but due to the large number of roles that the program incorporates, there were
only a few individuals represented in each of the capacities/roles. For example, only three
Public Health staff members were interviewed. If, for example, more individuals were
interviewed but fewer program roles were represented, more information would be known for
the roles that were targeted. In contrast, however, the wealth of knowledge gained from
interviewing individuals from all program roles provided information not previously known and
provided a greater insight into the program – its activities and its outcomes. Indeed, according to
18
Patton (1980), breadth and depth are both important and neither is better, it is simply a choice of
which fits better and in this evaluation, seeking a breadth of information was considered more
important for reasons previously discussed.
Another limitation was the use of convenience samples to connect with past program
participants - participants that could easily be connected with were contacted for an interview.
This meant that more participants were interviewed that had recently graduated the program
than those that had graduated several years ago and thus, more short-term findings were
investigated than longer-term findings. This limitation, however, points to a key finding, which
will be discussed in greater detail to follow, that program participants who had graduated
several years ago may have been easier to contact had they still been involved in
F&STWORKS, the continuation of the program after the end of the initial F&ST programming.
Results
The findings outlined here are the major themes that have been discovered in the data that
were noticed across all participants involved in the evaluation (i.e. from all roles in the program).
The findings were analyzed using qualitative methods and a triangulation of various data
collection techniques (interviews, Photovoice, observation). These findings are organized into
the strengths of the program (successes of the program and areas of the program that are
beneficial to families) and challenges of the program (areas for improvement and aspects of the
program where short- and/or long-term outcomes are not evident).
Strengths of the Program These are the areas of the program that were found to be beneficial to families in the program,
demonstrated an outcome intended by the program, and were found to be successful through
interview and Photovoice data.
Family Time Many participants discussed the benefits of having an opportunity for families to come together
for an evening. One of the positive aspects of the program is to help families learn how to better
connect, communicate, and support each other as well as provide an opportunity for them to
spend time as a family.
19
“for some of these families who might be single parent families or experiencing some
frustration with their children at home, it’s a night of fun where they can kind of forget
about their problems”
Meeting New People and Building Community
The program allowed people to familiarize themselves with their community and meet new
people in their neighbourhoods including new Canadians. The program also provides an
opportunity for families to connect with others: children could meet other children and parents in
particular could discuss parenting concerns with other parents and offer each other support in
an inclusive manner.
“a lot of parents who are new to the area, or who are new to the school, get to know
other families…it creates a support network in our area”
Connection to the School
The program increased the level of connection between parents, their children and the school.
The connection to the school and to school personnel allows children and parents to become
familiar with the school and school personnel and feel more comfortable in the school system.
“I’ve learned the value of making connections with the families that are coming new to
your school and hooking them in as early as possible to maintain their involvement in the
school”
Involvement in the Program F&ST staff and volunteers stated that they enjoy giving parents the opportunity to have a fun,
family night, being able to support them in modeling good communications and parenting to
their children and enjoy developing relationships with the families involved in the program.
Teachers also indicated that they enjoyed this as well as getting to know children outside of the
classroom setting.
Inclusivity Several participants mentioned that the program offered a chance for all individuals, including
those with financial difficulties, and single parents, to participate. Further, the program provided
space for newcomers to Canada to practice and improve their English skills.
Supporting Enhanced Parenting Skills
20
The program offered support to parents by helping them enhance their parenting skills and
assisting them in learning how to manage their child’s behaviour.
“Parents are seeking permission to tell their children what to do and to guide them -
they’re thinking that that isn’t their job, and they see that they are the one telling their
child what to do through this program then it really kind of turns things around in a
positive way – the parents have more confidence with telling their kids what they need to
do and have that that leadership role in their families which is something that some of
them lack”
Growth in the Program Changes were observed by school personnel, F&ST staff and volunteers and Public Health
personnel in families and individual family members over the course of the program. In
particular, increased communication was seen amongst family members and toward others
such as teachers and F&ST staff.
“the children or the parents who come at the beginning who don’t talk to you, barely talk
to you, don’t make eye contact, are guarded through the activities and by the end of the
eight weeks, they call you by your first name, they come and hug you when they get
there…you can sense the enjoyment”
Counselling Services Several participants discussed the importance of the program for providing support much like
the type of support that might be offered through group counselling.
“ we are not in the business of rescuing children from their parents...we are in the
business of strengthening parents. And I always thought that this program should have
been a model for all our children’s group that we do at the agency” Respondents across all components/roles of the program stated that the program was positive
and helpful to families involved. The program is also positive for those involved in administering
it such as for F&ST staff members and volunteers, school personnel and Public Health
personnel for a multitude of reasons.
Challenges of the Program While it is evident that the program offers numerous benefits for families involved, respondents
did speak to the areas that could be improved. These findings also indicate that short- and/or
long-term outcomes are not evident in all aspects of the program.
21
F&STWORKS
Overall, among F&ST families, there was a lack of knowledge of, or engagement in, the
F&STWORKS part of the program, originally intended by McDonald et al. (1997) to be the main
focus of the F&ST program in building community engagement, strengthening family systems
and thus decreasing problematic behaviour in children. Further, very few interviewees and
Photovoice participants discussed F&STWORKS suggesting that this is not a core component
of the program. Indeed, several interviewees stated that there is little turnout for or little
knowledge of F&STWORKS which indicates a lack of focus on this aspect of the program.
“F&STWORKS is…where we need to focus our attention. We don’t get great turn out.
The best turn out last month where we had 6 families come. That was huge for us,
That’s full.”
“if I were to say something that could be improved upon, how they do
F&STWORKS…they don’t do things that are interesting”
“I’ve never heard of that [F&STWORKS]”
F&ST staff are trained on F&STWORKS (i.e. the purpose of F&STWORKS, the theory behind it,
etc.) but in practice, this portion of the program is not very strongly promoted and very few
families become involved which may be one reason why some long-term outcomes are not
evident.
Time Commitment Many participants mentioned that the time commitment that is required, particularly of F&ST
staff, volunteers and school personnel was challenging. Many found it challenging to commit to
being available for the full eight or ten weeks of the program.
“the time commitment may involve in teacher involvement…ten weeks, even eight weeks
[is long]”
In contrast, however, many of the families who had participated in F&ST stated that the program
was too short. For example, numerous families felt that at the completion of the program, they
were not prepared for it to end.
“I think if it was a little bit longer, we would have probably established much deeper
connections right? With different families”
These findings indicate that there may be insufficient emphasis being placed on F&STWORKS.
Involvement in F&STWORKS allows families to continue to learn and practice many aspects of
22
the program and provides ongoing networks of support following the end of the initial 8-week
F&ST sessions. However, with an increased emphasis on this second stage of the program,
F&ST and Mosaic staff, volunteers and teachers would actually be required to offer more of their
time to the program (approximately two years) which could be seen as positive or negative in
terms of time commitment/ could increase their concerns regarding the challenges involved in
the time commitment required.
Program Structure One very important aspect of the program is to provide support to parents so that they can
enhance their parenting skills. This is done through tangible supports offered but also through
F&ST staff, volunteers, teachers and other F&ST team members relinquishing their typical
“roles” and providing instructions throughout each session (i.e. activity changes, behavioural
feedback, etc.) quietly to the parents so that all instructions given to children come directly from
their parent (McDonald, et al., 1997). F&ST team members are to whisper instructions to
parents rather than offer them directly to children (Family Service Canada, n.d.). Feedback from
F&ST team members indicated that it is difficult for many to forget their normal roles, particularly
F&ST staff and teachers, and to engage in the required activities of the program. Some
interviewees suggested that because of this challenge, the structure of the program had in
some cases been compromised. For example, if staff, volunteers, teachers or other persons
involved could not detach themselves from their typical roles (whether their role is being a
teacher, a counselor, etc.) and assist parents in actual parenting, the standards of the program
as outlined in the F&ST Program Workbook (Family Service Canada, n.d.) were not being
followed and the goals of the program not sufficiently being met (i.e. to help parents learn and
practice parenting skills).
“I think it’s something people forget and it’s hard for professionals like teachers to get
their head around…whispering in a parent’s ear to get them to do it [parent]…the intent
is to…empower the parent to let them know ‘we have faith in your parenting
capacity’…[this is] something that does get dropped”
Several interviewees also described the importance of remaining true to the program structure
and how the format is necessary to achieve the intended outcomes of the program.
“this is where I feel it would be important to keep in contact with Lynn McDonald and,
you know, her original design was so based on choosing activities that were fun filled,
strength based, results oriented...she had a set of principles that were in many ways
23
common sense principles – but she was able to take her principles and to see what has
worked in practice”
It was also suggested that the role of the team leader should include more structured
observation to ensure that activities are being followed.
“to strengthen the role of the team leader…to mentor and observe…how we’re doing as
team members”
Participant Recruitment Several challenges were discussed with regard to recruiting participants. A particular challenge
mentioned by several teachers was recruiting participants into the Fall portion of the program.
This was particularly difficult both because parents were not familiar with teachers, making trust
and program buy-in challenging, and because teachers were not familiar with children, making
referrals based on behaviour challenging. “it’s a struggle to get parents because our program is right at the beginning [of the
school year]”
Several school personnel, F&ST staff and Mosaic F&ST staff discussed ‘repeat’ F&ST families
and the difficulty in negotiating how often to let participants return to the program versus recruit
new families. Some families were allowed to repeat the program because they did not fully
understand what was being taught. “some of them [families] need to be here more than once to really grasp what is being
taught”
“some families are repeats which is nice…just because I think it gets talked up in the
family…I think our parents enjoy that aspect of it”
Having families repeat the program may be problematic because it prevents new families from
joining. Addressing this issue may be one solution to the suggestion that the program should be
offered to more people.
Program Outcomes Many interviewees discussed the shorter-term impacts of the program including meeting new
people (for children and parents), getting to know the school, and being involved in activities as
a family. It was suggested by individuals from different program roles, however, that the
program may not have led to much change, especially over time.
24
“I feel like I’ve never been like, “wow, I really see that all of that time and effort has made
a difference for that kid”
“Has it [the program] improved my life? Has it improved our family? I mean it did at the
time. Has there been like a long lasting improvement? No. I don’t think it’s made any
significant long lasting impact. It was something that was positive at the time”
Families who had been through the program also failed to demonstrate lasting changes. Also,
parent-child interactions were not greatly impacted because improvement noted may in fact be
due to families becoming more familiar with the program activities themselves. “as families got better at doing the program so that they got better doing charades, or
um, you know whatever. I think they, started going through the motions. Sometimes it
was just uh, there wasn’t a lot of thought put through, and they could just kind of go
through the motions to get to the next point. I think we needed to be more diligent about
encouraging mom and dad to set the tone to take the time to play the charade or to do
some drawing”
The idea that families may only be improving as a result of becoming familiar with activities
coupled with the finding that there is little improvement over time suggests that the program may
not be achieving all short- or long-term goals. It may also mean that children and families
recruited to the program might be functioning fairly well as a family prior to entering the program
and are not in need of much support.
Purpose of the Program All interviewees were asked to describe how they perceived F&ST and the purpose of the
program. Most (from all roles) described it as an opportunity for families to spend a night
together and join each other for a free meal. Others included that it was an opportunity for
families to get to know others and their schools. Some also described it as an opportunity to
build children’s skills.
“I don’t make it [family time] a formal thing or organized at a particular point in the
evening [since the program]…I guess we’ve kind of semi-kept up with it…I like the idea
of having to go out”
These findings do not, however, incorporate the longer term impacts of the program, focused on
during F&STWORKS which are improving the behaviour and success of children through
strengthening families, building greater social networks, getting to know resources, and
strengthening the community (McDonald, et al., 1997). This, along with the findings that suggest
25
that many respondents have little engagement in, or knowledge of F&STWORKS, strongly
indicates that emphasis is placed on the eight-week portion of the program and that as such,
many long-term outcomes are not achieved.
Staying Connected to New People
While the program allows children and parents to meet new people within the program, previous
F&ST families stated that they did not spend much time with other parents once the program
ended. “we don’t call or anything but when you see them around the school yard and stuff you’ll
stop and chat”
This finding indicates that while parents and children do have an opportunity to connect with
others and some do remain connected, many do not develop long lasting relationships - one of
the goals of the F&STWORKS portion of the program (McDonald, et al., 1997). Indeed, one of
goals of F&STWORKS is to empower numerous parents within a given community to become
leaders in that community and work as a collective to improve their neighbourhood (Family
Service Canada, n.d.; McDonald, et al., 1997).
Observational Findings The observational findings corroborate several of the suggestions made by participants.
F&ST Sessions
It was observed at F&ST sessions by several researchers that several of the intended goals of
the program were being achieved during the program. For example, families spent time together
eating a meal and participating in various program activities and appeared to enjoy their
evenings at the program. As the program progressed, it was observed that parents began to
engage with other parents and children spent time playing with other children.
It was also observed, however, that the structure of the program was not closely adhered to.
There appeared to be gaps and inconsistencies in parental support during the session. For
example, transitions in the program (i.e. activity changes) were offered by staff rather than
whispered to parents who would then offer them to their children. Further, behaviours
considered inappropriate (i.e. children running around freely and not engaging in the program
activities, etc.) were not often reprimanded and when they were, were corrected by staff rather
than by parents with staff assistance. These findings were mirrored in the information offered
through interview data. In addition, a lot of “standing around” was noticed at all program sites by
26
F&ST team members suggesting that there may be alternative ways to use human resources
that would better achieve program goals or that there may have been too many human
resources at each program site. These observations were similar at the one F&STWORKS
session that was observed.
Previous Evaluation of the Program
Evaluation was both criticized and praised by interviewees in all program roles. Several
interviewees stated that they understood the importance of having an evaluation, but many
specifically disliked the original pre- and post-test format of evaluation.
“the paperwork…I mean more specifically… all the questionnaires, and as someone
with a research background, I totally totally understand the importance of evaluative
programs to see if there’s a change from point A to point B but I think the problem lies in
the fact that the…the questionnaires are quite long and I think there’s three or four of
them and it’s quite onerous on families and but also even very literate people don’t like
filling out forms, it’s much more difficult for families who are new to Canada…it’s difficult
to get families to do it”
Filling out questionnaires was found to be difficult, particularly for teachers because they found it
challenging to determine how to rate the children based on the questions available in the
survey. Particularly when the F&ST program ran in the Fall, teachers were still unfamiliar with
children and therefore unfamiliar with their behaviour. It was felt by most respondents, however,
that evaluation was important in order to glean insight into what aspects of the program worked
well and what improvements could be made.
New Evaluation Process
It was felt by all researchers based on direct feedback and the amount of information gleaned
about the program, that incorporating interviewing and Photovoice techniques in particular as
well as observational data collection strategies into the evaluation structure was beneficial. New
and more accurate data was gathered through more participatory evaluation models. For
example, only by interviewing participants could it be learned that teachers struggled with filling
out pre- and post questionnaires about students that they hardly knew.
Participants who were involved in Photovoice found this way of collecting information to be fun
for them and their families. It promoted buy-in to the evaluation process and captured stories
27
that may not have been captured otherwise. It also gave participants a chance to take pictures
of their families which they not have taken otherwise.
“I give it to [a friend] to take a picture from us in my own camera because I would like for
memorizing…for memory for also when my kids growing, I will tell them what we have
done and where we go and I will show them the picture that I took”
In fact, some Photovoice participants wished that the camera would have been given out earlier
in the program to capture more stories.
“it was a good idea but I think, I just hope you would get the camera earlier…you miss
opportunities”
Photovoice was also felt by the researchers involved in the project to be a good way to collect
data from participants. Having pictures shared by families worked to develop rapport quickly and
revealed more information about the lives of participants. Further, participants were in control of
what pictures they wanted to take and share and thus empowered families to tell their stories.
Other Information As mentioned, the purpose of this evaluation was to answer the research question “does the
F&ST program result in the changes it was designed to?” as well as several evaluation
questions. As can be seen based on the findings, while families do enjoy the program (i.e. find it
a fun evening with their family) and often feel better connected to the school and their
community, it is not clear that the program does result in many of the short- or long-term
changes that it was designed to for a variety of reasons including program infidelity. For
example, families indicated that they enjoy the program and this answers the process question
about the level of satisfaction of program participants. There is not clear evidence that the
program results in increased support for families, increased connectedness to the school or
community, increased ability to handle life’s challenges, increased ability for parents to act as
primary prevention agents for their children nor that it improves success in school for children or
improves their behavior in school. While some of these outcomes are evident immediately
following the program (i.e. increased connectedness to the school), these do not always remain
after a long period of time.
It is important to note that one data collection technique originally included in the data collection
strategy, an accumulation and analysis of quantitative collected over the course of the F&ST
program, was not executed. This was because the collection of data that took place during the
current evaluation suggested that the data collected in the years prior may not be a good
28
indication of the successes or areas of improvement for the program. For example, part of the
prior evaluation included collecting data from teachers but it was discovered through the current
evaluation that in many instances, teachers do not know their students well and thus, do not
have an accurate understanding of how their behavior would have changed over the course of
the F&ST program in order to judge program success or failure. As such, consolidating data that
may not be entirely accurate did not seem desirable. Instead, more interviews were conducted
and analyzed. While the purpose of the data analysis was to seek “individual variation” amongst
evaluation participants with respect to the roles in the program (i.e. variation among teachers,
among families, etc.), it was noticed that there were similarities in what was said across groups
in that several themes were heard across several types of program roles, across several
program years and across several program sites. For example, many participants, regardless of
their role in the program, what year they were involved and what site they were at, felt that the
program was simply a fun night out for the family which suggests a misconception about the
purpose of the program. Indeed, according to Patton (1980), this type of sampling (i.e. selecting
3-4 individuals that represent each program role) “increases confidence in common patterns
that cut across different programs” (p. 105) or in this instance, program roles, sites, and years
offered. Overall, more information was gathered this year about the program than any other
year that is likely more accurate and presents a greater picture of the program.
Discussion & Lessons Learned Sharing Information At the beginning of the evaluation, an evaluation advisory committee was formed. This
committee included Mosaic F&ST staff, members of the Research and Development
Department at Mosaic, and other community stakeholders involved in the operation of the F&ST
program. This committee was involved in assisting the researchers develop an evaluation
strategy and provided input and feedback to the researchers regarding the anticipated strengths
and challenges of the new evaluation strategy, any amendments necessary, and provided
feedback on the research tools (i.e. thoughts and feedback on Photovoice, the interview
questions, which questionnaires to administer, etc.). In addition, Mosaic F&ST staff were asked
to provide ongoing input into the research design and research tools and were actively involved
in participant recruitment for the evaluation. For example, Mosaic F&ST staff suggested that
since rapport had already been built between themselves and F&ST families, that they should
introduce to families Photovoice, the opportunity to be involved in Photovoice, assist
29
researchers in providing families with more specific details about the technique as well as help
to arrange the final interviews with families (where they shared their pictures and stories).
Learning One learning that occurred during the current evaluation was regarding the data collection
technique Photovoice. Specifically, researchers and Mosaic F&ST staff members discussed and
agreed on the difficulty they experienced in ensuring that the goal of the technique was
achieved: to take photographs of the experience of the program including areas of success and
areas for improvement. For example, Photovoice participants were reminded several times
throughout the course of the program by researchers and Mosaic F&ST staff to take images that
met the goals of this data collection technique but all families but one took pictures of more
things outside of the program than of the program itself (i.e. photographs that had nothing to do
with the program). While the purpose of Photovoice as a participatory technique is to allow its
participants to control the data they collect through photographs, it was evident that too much
freedom caused participants to veer away from the goal of the technique. Ultimately, each
family did have pictures that fulfilled the purpose of this technique but many more of the
photographs did not. This learning, however, was felt by researchers and Mosaic F&ST staff to
be very important to how the data collection technique would be promoted and monitored in the
future for program evaluation. For instance, a disposable rather than a digital camera would be
offered to Photovoice participants to limit the number of photographs a family could take in the
hope that most pictures would be related to the program. In addition, greater emphasis would be
placed on the purpose of Photovoice and on capturing what worked well about the program and
what could be amended rather than the “experience” of the program. It is important to note,
however, that overall, Photovoice was felt by the participants and researchers to be an
innovative and positive way to collect data as well as involve individuals in data collection and
with a few amendments, it would be very successful as a program evaluation technique.
The Evaluation Experience Prior to the commencement of the evaluation, there was a consensus between the Research
and Development Department and the agency’s Executive Director that research and evaluation
were of value and importance to the agency for learning about successes and areas for
improvement for all practices in the agency. Indeed, research and evaluation had become an
important component of all applications developed and submitted for funding by the agency.
There was also an assumption that agency staff fully understood and embraced this integration
30
of research into the practice of the agency. This evaluation project provided an opportunity for a
reassessment of the degree to which a research culture had been sufficiently embedded at
Mosaic and the steps required to further develop staff understanding and comfort with
integrating research knowledge into their work. While there is agreement regarding the value of
evaluation, it became clear to the Research and Development Department that there are
differing perspectives amongst staff of where, when and how much research and evaluation are
needed to support good practice. This evaluation prompted an in-depth discussion of research
and evaluation within the F&ST program and across departments, and in particular, the ways in
which the evaluation findings should be assessed and integrated in program delivery. In
addition, this evaluation prompted dialogue amongst staff about research validity and rigor –
staff sought clarification about what is involved in rigorous research and evaluation and so
demonstrated their curiosity and confusion about the process. This dialogue was productive and
will require an ongoing commitment to mutual learning as there remain various, often competing
opinions amongst agency staff about what constitutes rigorous research. Through this
evaluation process, therefore, it became apparent that a more clear and in-depth understanding
of what constitutes research in general, as well as rigorous research, is required for agency
staff. Indeed, if agency staff are unfamiliar with what research is, and what is involved in
conducting research, this leaves room for uneasiness in accepting presented findings and a lack
of clarity in how to learn from them.
This evaluation resulted in a debate amongst various staff members with regard to the
components of the program and the intended outcomes of the F&ST model. Specifically, the
evaluation provided an opportunity for staff to re-visit and re-consider program components
including the purpose of each component and whether each was achieving its intended
outcome in addition to whether the sum of all components were helping families achieve short
and longer-term outcomes. This evaluation project led to the mobilization of program staff
around the value of the program and each of its components but also provided an opportunity
for each staff member to reflect on how the program, as it is offered in actual practice, deviates
from the original design. Staff members were provided the chance to consider how significant
the deviation is and whether and how the deviation benefits program participants. The
mobilization of staff around the value of the program and its components, however, created
some challenges for researchers involved in the project to convey both program successes and
challenges equally. Indeed, given the passionate support and confidence in the program’s
effectiveness that many staff feel, there was some reluctance on the part of some to accept
31
findings that pointed to areas for improvement within the program. This had led to a degree of
mistrust in the evaluation findings on the part of some program staff that may require additional
dialogue to ensure that this does not create a barrier for making the required improvements to
the program to ensure it achieves its intended short- and long-term goals.
This evaluation highlights the strong need for individuals in leadership roles, including agency
Executive Directors and others in a leadership capacity, to understand and believe in the value
of research to build a capacity for, and a culture of, evaluation within their agency. Further, it is
important for those in leadership roles to have some understanding of what is involved in
conducting rigorous research and evaluation in order to continue to promote its importance and
relevance to staff. Indeed, it is not enough to assume agency staff appreciate or understand
research – frequent and consistent messaging and organizational learning about research and
evaluation would be required to increase interest, and ease distrust in evaluation.
This evaluation provided evidence that there must, within an agency, be a partnership between
agency leaders, staff and researchers to ensure an effective translation of findings into practice.
All agency staff should have an understanding of research, and more importantly, an
appreciation for research and evaluation in order to welcome all findings (successes and
challenges). Moreover, it is not enough to assume there is a culture of evaluation and that
agency staff understand research and evaluation (what research is, what it means to conduct
research, and how to use findings). Further, change can only occur when agency leaders and
staff are ready for, and interested in it. What became evident through this project is that in order
for this evaluation to lead to change, there must first be a culture of evaluation, or at least an
appreciation for evaluation in place within an agency.
The experience of working with the Centre was very positive. Researchers at Mosaic felt
supported throughout the process so that at any time, questions could be asked and answers
were immediately provided. Researchers also felt that information was constantly available
through the Centre’s website, trainings and through staff. Regular conversations with Centre
staff allowed for the researchers to keep connected with the Centre and provide updates about
progress made, ideas about the process of the evaluation and initial findings. The Centre also
allowed space for researchers to voice concerns about the evaluation and discuss any changes
made during the process. The most beneficial aspect of the Centre’s approach to planning for
an evaluation and reporting on the process, progress made and findings was the constant
32
encouragement to discuss the aspects of the evaluation that were challenging, required
amendment or that did not go as planned. This was particularly helpful because it acknowledges
that even when evaluation is planned in detail it does not always go as expected. The Centre
perceives that this is not a failure, but rather a chance for the agency itself and others to learn
about what could be done to remedy the problem next time. The Centre also allowed for many
agency staff to be involved in the evaluation process by creating space for all agency staff to
participate in online training as well as in the onsite visit at the beginning of the project. This was
beneficial because it offered staff an opportunity to learn about evaluation and research and be
involved in the evaluation process from the outset.
Conclusion & Recommendations/ Next Steps Recommendations for Others The initial recommendation flowing from the current experience of evaluation would be to
sufficiently assess the capacity of the agency to engage in and understand research and
evaluation as well as respond to and integrate findings prior to commencing an evaluative
project. This would include determining the actual level of interest amongst staff members in
evaluation, the level of comfort staff have with conducting an evaluation and how receptive staff
members would be to both positive and negative findings as a result of evaluation. For example,
how open would staff be to amending a particular part of their program or agency if it was
discovered that improvements were required? Further, it is important to determine the level of
knowledge that staff have about research and evaluation and provide ongoing opportunities for
learning so that with greater knowledge, fear or distrust are eased, and interest and buy-in are
increased.
An additional recommendation would be to design and implement various types of evaluative
strategies that may be unique and innovative. For example, in the current evaluation,
Photovoice was used as a data collection technique and while the success of this technique
was not entirely clear beforehand, it was determined that this is a very successful technique
according to F&ST program participants that could be used again to further evaluate this
program, or other programs at Mosaic. Using different techniques allows researchers and
evaluators to discover new, innovative and effective ways to evaluate programs, learn new and
valuable information, and gather greater knowledge and understanding of program participants.
33
Recommendations for the Organization A suggestion that could be offered, particularly to F&ST staff and Mosaic F&ST staff, is to
recognize that while participants involved in the evaluation offered information on what aspects
of the program need to be amended, agency staff are in a unique position to make these
amendments. Many people, including participants from all roles of the program, spoke about
how appreciative they were of the kindness, dedication and positivity of the staff and would
therefore likely be open to any amendments staff made to the program. The evaluation process
simply pointed to aspects of the program in need of improvement and while it is ultimately the
responsibility of the agency to ensure that these amendments are made, F&ST staff and Mosaic
F&ST staff are in a position to promote buy-in for change and growth within the program among
program participants (families) and program partners (i.e. school personnel, Public Health
personnel, etc.).
Another recommendation would be to consider the findings from the evaluation to determine
one of three options, each of which will be discussed in greater detail to follow: 1) amend the
practices in the program so that they follow the original program design and thus remedy issues
of fidelity, 2) re-vision/re-structure the program such that only the components of the program
with demonstrated success remain and re-name the program, or 3) cease offering the program.
Remedy Fidelity Issues
As discussed above, based on observational data, previous knowledge of the program acquired
through the standardized F&ST training and feedback from interview participants, the program
is not being implemented entirely as outlined in the F&ST Workbook, by McDonald et al. (1997)
or by other agencies running the program (Family Service Canada, 2007; Kratochwill et al.,
2009; McDonald et al., 1997; McDonald & Sayger, 1998; The Family Centre, 2008; The Vanier
Institute of the Family, 2006). It is important, according to Melde, Esbensen and Tusinski (2006),
to provide consider information pertaining to which program components contributed to
“success” and “failure” (p. 717). It is also crucial to include a full discussion of what proper
implementation constitutes in order to achieve the intended outcomes of a program. This is also
important to successfully replicate a program elsewhere (Melde, et al. 2006). Indeed, while
McDonald (1997) and the F&ST Program Workbook (Family Service Canada, n.d.) clearly state
that modifications to the program are acceptable to fit with the needs of the community, these
changes should not fundamentally alter the nature of the program and therefore, the core
components (i.e. activities, program duration and staff roles) should remain constant. It can be
34
argued that to date, the program has been altered too greatly to achieve the desired outcomes
and to be compared with other F&ST programs in the country. For example, the F&STWORKS
component of the program deviates more than any other area of the program from the original
design and as such, offers the fewest outcomes for participants. This is evident in the lack of
information that interviewees and Photovoice participants offer about either the successes or
challenges of this program component. It also explains the comments suggesting that this
particular aspect of the program requires amending made by few interviewees. If the core
components of F&STWORKS more closely matched the original design, the outcomes could
then be compared to other programs offered across the country.
Program Components to Keep for a New Program The findings suggest that there are numerous aspects of F&ST that respondents enjoyed and
felt were beneficial for families in the program. One option, then, is to re-structure the program
in order to keep offering the successful components of the program while removing the
remaining components and re-name the program. The following are the successful components
of the program:
Family Time - Allowing families to come together as a unit Meeting New People - Creating opportunities for people to connect with others Connection to the School - Increasing the level of connection between parents, their
children and the school Involvement in the Program - Providing an opportunity to staff and volunteers to be
role models for appropriate parenting and communication for children
Building Community - Families who are new to an area or to the country can
familiarize themselves with their neighbourhood and meet new people Inclusivity - All individuals, particularly those who living on low-income, are able to
participate and be offered support Parenting Support – There is potential to provide support to parents and assists them
in learning how to manage their child’s behaviour and increase the confidence of parents to
“parent” their children Counselling Services - Demonstrating the importance of counselling and offering
support much like counselling would but in a fun manner and within the client’s community
Cease Offering the Program
35
It was evident through this evaluation that there were numerous successes and positive aspects
of the program. It can be argued, however, that it was not clear that many short- or long-term
goals were being achieved and it may be appropriate to reconsider whether this particular
program is the most effective way to achieve the desired outcomes of the program.
Continue to Implement the New Evaluation Strategy Another recommendation would be with regard to the new evaluation strategy created to
evaluate the program and to utilize this, or a similar strategy in the future. A great quantity of
information was generated from including interviews, observational strategies and Photovoice
into the program evaluation. Interviewees from several program roles (i.e. school personnel,
parents, F&ST staff, etc.) discussed the difficulties they experienced in filling out pre- and post-
questionnaires but felt that evaluation is important. Photovoice in particular was well liked by
participants and researchers felt that interviews, Photovoice and observational strategies
provided information and stories about F&ST that would not have otherwise been captured.
Strengthen Evaluation Capacity The final recommendation would be to strengthen the capacity for evaluation within the agency
before evaluating any other programs. Further, a recommendation for individuals in leadership
positions, specifically, would be to persistently work to ease distrust of research amongst staff in
the future. This is important so that future suggestions based on evaluation project findings are
more positively received by staff members, that staff members will actively seek to be involved
in research processes and that both positive findings and suggestions for improvement are
perceived as equally important and desired among staff. Further, greater capacity for evaluation
and the perceived importance of research will, in time, result in staff members actively pursuing
areas of the agency that could be evaluated in the hope of future improvement.
The Centre The Centre provided more than sufficient access to assistance and education regarding
evaluation. Training was easily accessible both in terms of seeking information (i.e. on the
internet, etc.) but also in terms of having it provided in a way that could be easily understood by
those less familiar with evaluation. Further, Centre staff were always accessible if further
assistance was required. The greatest success of the Centre was to offer the opportunity and
support, specifically in this project, for the research methodology to include a data collection
technique not often used to determine how successful it would be for gathering information.
36
Knowledge Exchange Knowledge sharing activities originally included providing information to agency staff, volunteers
and partners as well as F&ST staff, volunteers, partners and program participants verbally (i.e.
staff meetings, face-to-face conversations, large scale presentation) and/or through writing such
as reports and the agency website. Knowledge exchange was also to take place with the F&ST
head office in order to share research findings.
Information about the evaluation process and findings were shared in a variety of ways
throughout the course of the project. An interim report of findings was developed in April of 2010
to share interim findings with the Executive Director of Mosaic as well as Mosaic F&ST staff.
The interim report included interview and Photovoice data collected as well as observational
data collected by April. It also included, based on interim findings, successes of the program as
well as recommendations for program amendments so that services for the children involved in
the program and their families could be improved and both short- and long-term goals could be
more effectively achieved. The Executive Director and Mosaic F&ST staff also had an
opportunity to discuss with the researchers their thoughts and feedback about the interim report,
findings and recommendations.
In addition, a large-scale event was held for Photovoice participants and their families in June of
2010. Specifically, the images that F&ST families that participated in Photovoice selected (one
image for each family) were enlarged and framed along with the “story” (a quote) that they
offered to describe the importance of the photograph and how it related to their experience of
the F&ST program. The event was offered on a Friday evening and Photovoice participants as
well as other F&ST families, Mosaic and F&ST staff, F&ST school personnel and several other
community members were invited to come and view the images. Information about Photovoice
as a data collection technique was offered to those in attendance and a slideshow of images
taken at the F&ST program by the researchers and Mosaic F&ST staff played during the event.
In addition, the display was made available in the front lobby of Mosaic for several weeks
afterward to offer agency clients and partners an opportunity to learn about the F&ST program,
Photovoice as a data collection strategy and of the research and evaluation being conducted by
the Research and Development Department at Mosaic. Children and their families involved in
Photovoice attended the event and were offered an opportunity to review and showcase the
picture selected by their family. This event, then, was an opportunity to empower children and
families by sharing broadly their experiences in the program and their ability as a family to take
37
creative, powerful images. Families were also offered their framed photographs and stories to
keep as a token of appreciation for their involvement.
Several researchers also attended an event entitled “Central West Regional Conference 2009-
2010: Meeting changing needs with emerging knowledge” in March 2010 at CHEO to share
their experiences of and some of the information gathered through the Photovoice technique.
The researchers developed a poster as well as a one page handout with information about
Photovoice, initial findings and contact information (see Appendix K for the one page summary).
This knowledge sharing event was of particular importance because it offered researchers an
opportunity to share with others the effectiveness of using Photovoice as a participatory action
research technique to engage participants, and particularly children, in collecting and reviewing
data. Photovoice as a research technique in this instance not only allowed children to be
involved in the process of collecting data (via taking photographs) but also allowed them to
share their experiences of a particular program with their own families and with researchers.
The process created opportunities for children and their families to connect with one another to
negotiate what photographs to take and in some cases, to discuss how these images reflected
what they, as a family, believed to be important information to share with others about the
program. The process also allowed several families to come together as a unit to select one
picture to be shared with others through the large scale event in June. Photovoice is a unique
way to capture the experiences, knowledge and perspectives of children and families.
As can be seen, several knowledge activities have already taken place during the course of the
evaluation. Some of the planned knowledge exchange activities originally outlined for the
evaluation, however, such as with agency partners, volunteers, F&ST staff or participants, are
still in development given the response of staff toward the findings. As aforementioned, this
evaluation has resulted in productive conversation about the future and direction of the program
as well as led to new learning about the program and about evaluation for the agency. It has
also halted knowledge exchange activities of program success or needed improvements with
other stakeholders or partners due to the reluctance of agency staff and leaders to share or
implement the recommended amendments. One planned knowledge exchange activity for the
current evaluation findings, however includes sharing the current report with agency leaders,
agency staff and the Board of Directors. After this knowledge exchange activity, it will be
decided amongst these individuals how to proceed with further knowledge sharing and with
what amendments, if any, are to be made to improve the program.
38
Works Cited Berg, B.L. (2007). Qualitative research methods for the social sciences (6th Ed.). Boston: Allyn &
Bacon. Family Service Canada. (n.d.). F&ST Program Workbook. Family Service Canada. (2004). Families and Schools Together Canada Canadian National
Evaluation of Outcomes 1996-2004 Executive Summary. Ottawa: no author. Family Service Canada. (2007). Family Service Canada Annual Report 2006-2007. Ottawa: no
author. Families and Schools Together. (2007). Families and Schools Together Canada Annual Report
2006-2007. No author. Kratochwill, T.R., McDonald, L., Levin, J.R., Scalia, P.A. & Coover, G. (2009). Families and
schools together: An experimental study of multi-family support groups for children at risk. Journal of School Psychology, 47, 245-265.
Melde, C., Esbensen, F. & Tusinski, K. (2006). Addressing program fidelity using onsite
observations and program provider descriptions of program delivery. Evaluation Review, 30 (6), 714-740.
McDonald, L., Billingham, S., Conrad, T., Morgan, A., O, N., & Payton, E. (1997). Families and
schools together (F&ST): Integrating community development with clinical strategies. Families in Society: Journal of Contemporary Human Services, 140-155.
McDonald, L. & Sayger, T.V. (1998). Impact of a family and school based prevention program
on protective factors for high risk youth. Drugs & Society, 12 (1), 61-85. Patton, M.Q. (1980). Qualitative evaluation methods. Beverly Hills, California: Sage Publications
Inc. The Family Centre. (2008). Building inclusive and respectful communities 2007-2008 Annual
Report. Winnipeg: no author. The Vanier Institute of the Family. (2006). Building emotional intelligence: Darwin reconsidered.
Retrieved April 3, 2010 from http://www.vifamily.ca/library/cft/eq.html. Wang, C. C. & Burris, M. A. (1997). Photovoice: Concept, methodology, and use for
participatory needs assessment. Health Education and Behavior, 24(3), 369-387.
39
APPENDICES
40
APPENDIX A – F&ST Logic Model and Evaluation Framework
41
42
43
APPENDIX B – Informed Consent for F&ST Families and Parent Partners
44
APPENDIX C – Interview Consent Form for F&ST Staff and Volunteers, School Personnel, Public Health Personnel and Mosaic F&ST Staff
45
APPENDIX D – Interview Schedules
INTERVIEW SCHEDULES Interviews – Current Families
1) If you were to describe F&ST to another person who didn’t know about it, how would you describe it?
2) Can you tell me a story about a positive experience you had at F&ST? 3) Can you tell me a story about a negative experience you had at F&ST? 4) What do you like best about the F&ST program in general? 5) If you could change one thing about the F&ST program in general, what would it be?
6) What have you learned from the program? Probe: What information have you taken away from the program? 7) How has the program helped you?
8) How has the program helped your family? Probe: What has the program done for you and your family? 9) Do you have any stories to share that are particularly meaningful? 10) How do you feel about this interview format (versus the survey)?
Staff/Volunteers
1) If you were to describe F&ST to another person who didn’t know about it, how would you describe it?
Probe: What would you want other people to know about the program? 2) How long have you been involved in F&ST? 3) Can you tell me a story about a positive experience you have had with F&ST? 4) Can you tell me a story about a negative experience you have had with F&ST? 5) What did you enjoy most about helping to run F&ST? 6) What did you enjoy least about helping to run F&ST? 7) What are the strengths of the program in general? 8) If you could change one thing about the program in general, what would it be? 9) How well do you feel that the things taught in training are happening as they are supposed to? 10) How has the program contributed to your agency? Probe: How does the program align with your agency mission or goals? 11) How has the program contributed to the lives of the families involved? Probe: What has the program done for the children and their families involved? 12) What have you personally learned from the program? Probe: What information have you taken away from the program?
46
13) Do you have any success stories to share? 14) How do you feel with this interview format?
Parent Partners
1) If you were to describe F&ST to another person who didn’t know about it, how would you describe it?
Probe: What would you want other people to know about the program? 2) How long have you been involved in F&ST? 3) Can you tell me a story about a positive experience you have had with F&ST? 4) Can you tell me a story about a negative experience you have had with F&ST? 5) What did you enjoy most about helping to run F&ST? 6) What did you enjoy least about helping to run F&ST? 7) What are the strengths of the program in general? 8) If you could change one thing about the program in general, what would it be? 9) How well do you feel that the things taught in training are happening as they are supposed to? 10) How has the program contributed to the lives of the families involved? Probe: What has the program done for the children and their families involved? 11) What have you personally learned from the program? Probe: What information have you taken away from the program? 12) Do you have any success stories to share? 13) How do you feel with this interview format (versus the survey format)?
Teachers/School Personnel
1) If you were to describe F&ST to another person who didn’t know about it, how would you describe it?
Probe: What would you want other people to know about the program? 2) How long have you been involved in F&ST? 3) Can you tell me a story about a positive experience you have had with F&ST? 4) Can you tell me a story about a negative experience you have had with F&ST? 5) What did you enjoy most about helping to run F&ST? 6) What did you enjoy least about helping to run F&ST? 7) What are the strengths of the program in general? 8) If you could change one thing about the program in general, what would it be? 9) How well do you feel that the things taught in training are happening as they are supposed to? 10) How has the program contributed to the lives of the families involved?
47
Probe: What has the program done for your student(s) and their family(ies)? 11) How has the program contributed to your school? Probe: How does the program align with your school’s mission or goals? 12) What have you personally learned from the program? Probe: What information have you taken away from the program? 13) Do you have any success stories to share? 14) How do you feel with this interview format (versus the survey format)?
Public Health
1) If you were to describe F&ST to another person who didn’t know about it, how would you describe it?
Probe: What would you want other people to know about the program? 2) How long have you been involved in F&ST? 3) Can you tell me a story about a positive experience you have had with F&ST? 4) Can you tell me a story about a negative experience you have had with F&ST? 5) What did you enjoy most about helping to run F&ST? 6) What did you enjoy least about helping to run F&ST? 7) What are the strengths of the program in general? 8) If you could change one thing about the program in general, what would it be? 9) How well do you feel that the things taught in training are happening as they are supposed to? 10) How has the program contributed to the lives of the families involved? Probe: What has the program done for your student(s) and their family(ies)? 11) How has the program contributed to your agency? Probe: How does the program align with your agency mission or goals? 12) What have you personally learned from the program? Probe: What information have you taken away from the program? 13) Do you have any success stories to share? 14) How do you feel with this interview format?
Past Participants
1) If you were to describe F&ST to another person who didn’t know about it, how would you describe it?
Probe: What would you want other people to know about the program?
48
2) Can you tell me a story about a positive experience you had at F&ST? 3) Can you tell me a story about a negative experience you had at F&ST? 4) What are the strengths of the program in general? 5) If you could change one thing about the program in general, what would it be? 6) 7) How has the program contributed to your family’s life? Probe: What has the program done for you and your family? 8) Have there been any lasting changes in your family since the program? 9) Have you kept in contact with any families in the program? 10) 11) What have you personally learned from the program? Probe: What information have you taken away from the program? 12) Do you have any stories to share that are particularly meaningful? 13) How do you feel with this interview format (versus the survey format?)
Photovoice Instructions: Please take pictures that represent important and meaningful discoveries made during the program and the ways that the program has impacted you and your family. Please take pictures that show: Anything new that you or your family have learned through the program Anything changes that have taken place in your family as a result of the program Anything you and your family feel is special to you about the program Anything you and your family would want others to know about the program
1) Where was this picture taken? 2) Why did you take this picture? 3) What was happening in this situation? Probe: what would you want people to know about this situation/picture? Probe: How does this picture connect to the F&ST program? 4) Was this situation positive or negative? Why? 5) How did you feel about participating in Photovoice? 6) How would you feel about participating in Photovoice again? 7) What did you enjoy most about participating in Photovoice? 8) What would you change about Photovoice?
Please also ask interview questions for current F&ST families
49
Appendix E – Recruitment Flyer for Potential Photovoice Participants
50
APPENDIX F – Informed Consents for Photovoice Participants
51
APPENDIX G – Information and PowerPoint Introduction to Photovoice for Families
52
53
54
55
56
57
58
APPENDIX H – Photovoice Focus Group Recruitment Flyer
59
APPENDIX I – Photovoice Focus Group Informed Consent Form
60
APPENDIX J – One Page Summary for Poster Conference at CHEO
61