uOttawa.cauOttawa.ca

A Formative Evaluation of Two Crown Ward Education Championship Teams in Ontario

Kelly Weegar, A. J. Hickey, Samantha Shewchuk, Mariama Mary Fall, & Robert J. Flynn

crecs.uOttawa.caCentre for Research on Educational and Community Services (CRECS)

crecs.uOttawa.ca

Presentation Outline

1. Overview of the CWECT program in Ontario2. Evaluation purpose, scope, and questions3. Relevant literature4. Evaluation participants and methods5. Evaluation findings and recommendations6. Conclusions and next steps

crecs.uOttawa.ca

What is a Crown Ward Education Championship Team (CWECT)?• Supported by a tri-ministry working group• “The goal is to enable better education, training and

employment opportunities for all youth with Crown wardship status by:– Developing a local community-based partnership model– Identifying youth with Crown wardship status within the

community– Creating a sustainable and responsive support system for

these youth to help them make the transitions from one level of education to the next”

crecs.uOttawa.ca

Other (e.g., other youth organizations)Regional Ministry representation (MCYS/MED/MAESD)Youth Employment Ontario service providersLocal post-secondary institutions (colleges and

universities)Local child welfare organizationsLocal school boards (English & French)

crecs.uOttawa.ca

History of the CWECT Program

Figure 1. Spread of CWECTs in Ontario, 2008-2012.

crecs.uOttawa.ca

Purpose & Scope of the Evaluation• Given the relative newness of the CWECT program in

Ontario, together with the lack of previous evaluations, we chose to conduct a formative evaluationa. To find out how the CWECT program was working

“on the ground”, in the opinions of its stakeholders• To discover how clearly and with how much consensus

the stakeholders perceived the intended purpose of the two programs, how useful they saw program activities, and how helpful they felt the programs had been

b. To ultimately prepare for an outcome evaluation

crecs.uOttawa.ca

Purpose & Scope of the Evaluation• Local level (two individual CWECTs)

– Purpose: to assess how the program appears to be functioning and to suggest how it might be improved

– Scope: Young people in care (more than current Crown Wards)

• Provincial level– Purpose: to make a contribution to the functioning of

the CWECT initiative province wide (i.e., to enhance the relevance and quality of the program )

7

crecs.uOttawa.ca

Evaluation Questions Addressed

1. What are the needs of the stakeholders (i.e., the local Crown Wards/youth in care, caregivers, and community organizational partners)?

2. Does the CWECT program meet the needs of its stakeholders?

3. How was the CWECT program being implemented?4. Does the CWECT program appear to be working, in the

opinion of the stakeholders?

crecs.uOttawa.ca

Relevant Literature• At present, there are no evidence-based interventions

for increasing post-secondary access and retention for youth in foster care.

• However, a number of educational advocacy programs, like CWECTs, do aim to support the educational goals of at-risk youth. These programs have produced preliminary findings that suggest positive impacts, including:– Fewer school absences or changes in schools– Improved high school graduation rates– Successful resolution of education-related challenges

crecs.uOttawa.ca

Relevant Literature• Post-secondary education graduates with bachelor’s

degrees or college diplomas have higher incomes over the short-term and long-term than non-graduates.

• Many young people in care experience lower levels of academic achievement than youths in the general population, including lower rates of graduation or entry to and completion of post-secondary education.

• According to estimates from the OACAS Gateway to Success surveys of educational attainment, only about 46% of young people in care in Ontario currently graduate from high school within the usual four or five-year period.

crecs.uOttawa.ca

Relevant Literature• Without early mastery of basic academic skills in

reading and math, young people in care are at increased risk in the shorter and longer terms of:– Lower educational success than is warranted by their

academic potential.– Higher levels of school dropout, involvement in criminal

activity, homelessness, and mental health difficulties. • The need for effective help is urgent, in the preschool,

primary school, and secondary school years, for successful transitions to post-secondary opportunities for children and youth in out-of-home care.

crecs.uOttawa.ca

Evaluation Participants• Two CWECTs, representing four child welfare agencies:

1. CWECT-HS-FLA• Highland Shores Children’s Aid Society • Family and Children’s Services of Frontenac,

Lennox & Addington 2. CWECT-LLG-R

• Family and Children’s Services of Lanark, Leeds, & Grenville

• Family and Children’s Services of Renfrew County

crecs.uOttawa.ca

Evaluation Participants

Participants Collection Method(s)CWECT team members (N = 23) * • Phone interviews

• Observations Caregivers (N = 103) • QuestionnairesYouth in care (N = 65) • Questionnaires

• Phone interviews• Focus group

Community partners (N = 12) • Questionnaires

• Four participant groups, 203 participants total

* Organizations represented: children’s aid societies, employment services, school boards, colleges, universities, counselling services, Ministry of Education.

crecs.uOttawa.ca

Evaluation Methods• Mixed-methods approach

– Document analysis (e.g., meeting minutes, business plans)– Literature reviews– Interviews (phone and in-person)– Questionnaires (online and paper)– Focus groups– Observation

• Evaluation team met monthly to discuss progress• Evaluation team consulted with other CWECT teams

(those not part of the current evaluation) to better understand the CWECT initiative

crecs.uOttawa.ca

Results & Recommendations• Logic model• Findings and recommendations for each individual

CWECT and overall, by each evaluation question:1. What are the needs of the stakeholders?2. Does the CWECT program meet the needs of its

stakeholders?3. How was the CWECT program being implemented?4. Does the CWECT program appear to be working, in the

opinion of the stakeholders?

crecs.uOttawa.ca

Logic Model

• One of the main objectives of formative evaluation is often to establish the program theory or framework– Connects programs activities to the

intended outcomes for all stakeholders served by the program

crecs.uOttawa.ca

Logic Model• During the development of logic models, a critical

insight was the expansion of understanding about the clients served by the CWECT initiative – Although the program documentation for the CWECT

initiative typically identifies the program clients as Crown Wards, this evaluation and the logic models have made it explicit that the CWECT initiative also intends to serve important needs youth in care more broadly, as well as caregivers and diverse community partners.

crecs.uOttawa.ca

Question 1. What are the needs of the stakeholders?Youth in Care• According to the online questionnaires completed by the

youth for one of the CWECTs, all planned to pursue some kind of post-secondary education or training:– 17 or 70.8% aimed to go to college– 5 or 20.8% indicated wanting to pursue university– 5 or 20.8% planned to pursue an apprenticeship– 1 youth (4.2%) planned to pursue military training

• For this same sample, the majority of youth also rated the importance of post-secondary education highly– On a scale from 1 (not important) to 10 (very important),

the average rating was 7.83 out of 10

crecs.uOttawa.ca

Question 1. What are the needs of the stakeholders?Youth in Care• Nevertheless, it was consistently reported that the

primary need for young people in care is increased awareness and access to information and resources.– “…help researching different colleges, help with

budgeting… help me develop more skills that are related to what program I want to go into.”

– “Everything is in place, but youth don’t always know about them. More direction is needed to help navigate the information online.”

• Many CWECT team members also commented that positive, supportive, and consistent relationships are vital to the the educational success of youth in care

crecs.uOttawa.ca

Question 1. What are the needs of the stakeholders?• Related to this, both CWECTs have consistently devoted

time and resources to allow their youth in care to explore possibilities and to hear about others’ experience with post-secondary planning and transitioning to independent living, particularly in the face of adversity. For example:– College tours– University tours– Experiential learning events– Motivational speakers– Career information sessions

crecs.uOttawa.ca

Question 1. What are the needs of the stakeholders?Caregivers• Similarly, caregivers from both teams stated that they

needed more information to be able to guide their youth in care effectively.– “I need to be educated on what the necessary steps are

and how to go about them (as things have changed since I went). For example, how to apply for college, residence, OSAP, etc. Also, there are so many grants available for our youths that learning about them and how to apply would be helpful.”

• CWECT members also stated that caregivers need more information about their important role in and influence on the educational success of youth in care.

crecs.uOttawa.ca

Question 1. What are the needs of the stakeholders?• Information from interviews with CWECT members and

program documentation confirmed that attempts have been made by both teams to share this kind of information with caregivers (e.g., evening information sessions, including guest speakers).

• However, both teams commented that it is a challenge to get caregivers to attend knowledge-sharing events, and many were not sure why. – Some team members speculated that geographical factors

(e.g., large, rural areas mean more time is needed for travel) remain one of the biggest challenges

crecs.uOttawa.ca

Question 1. What are the needs of the stakeholders?Community partners• Largely based on results from the CWECT team member

interviews, the biggest need for community partners appears to be related to awareness and lacking information. – Several CWECT members noted that community partners

tend to lack awareness about the needs of youth in care beyond academics (e.g., mental health needs), and could benefit from learning more about how positive school experiences and supportive relationships can significantly influence to their success.

crecs.uOttawa.ca

Question 1. What are the needs of the stakeholders?Recommendation: We recommend that young people in care receive more one-to-one support to help them navigate the educational and employment process.

Recommendation: CWECTs should continue to expand their services of information and support to all three stakeholder groups—young people in care, and the caregivers and community partner organizations who are essential collaborators in helping the young people reach their educational and career goals.

– E.g., information-sharing among caregivers and community partners through a variety of methods (e.g., newsletters, information sessions, or webinars).

crecs.uOttawa.ca

Question 1. What are the needs of the stakeholders?Recommendation: To help target the needs of all clients served by the CWECT, a central website is needed to serve the informational needs of all local CWECTs and their various stakeholders. This web site would alleviate the need for local CWECTs to try to “reinvent the wheel” (which, in any event, exceeds their financial and operational capacities). The central website would be updated regularly with items about new opportunities related to the education and living situation of young people in care in Ontario. (For an example of what such a website might look like, see http://www.equalfutures.org/about/ ).

crecs.uOttawa.ca

Question 2. Does the CWECT program meet the needs of its stakeholders?• All members agreed that the purpose is about helping

youth in care improve educational and employment outcomes, and the belief that they can be successful– “I think we fundamentally do share all the same beliefs, I

think we do believe in the importance of keeping the needs of youth in care at the forefront, and I think that we're all committed to doing the best we can and continuing to get better in serving those kids within the structures that we have—within whatever system we come from. That’s fundamentally why I believe that we do what we do.”

• At this time, it appears that some needs of some youth are being met (e.g., information sharing and experiential learning events are taking place)

crecs.uOttawa.ca

Question 2. Does the CWECT program meet the needs of its stakeholders?• Many members from both teams noted that there is a

need to boost involvement of caregivers. Thus, the needs of caregivers are likely not being met.

• Due to difficulties reaching community partners, we are unable to assess whether their needs are being met.

crecs.uOttawa.ca

Question 2. Does the CWECT program meet the needs of its stakeholders?• But, are there adequate resources for CWECTs to

be able to carry out program activities? – Almost all CWECT team members noted lacking guidance

from the three Ministries in terms of planning effective ways to engage youth, caregivers and community partners.

– Furthermore, several team members added that having a full-time child welfare representative chair the team as a side job was a limitation of the current CWECT model (this was not a reflection of the current chairs, but rather a structural issue – i.e., each CWECT needs a full-time paid coordinator).

crecs.uOttawa.ca

Question 2. Does the CWECT program meet the needs of its stakeholders?• Recommendation: Work with local CWECTs to make it

possible to deploy a full-time CWECT coordinator in each program, as the present model of part-time coordinators simply does not provide enough time for program planning and development and taking care of the many needs of all the stakeholders (the young people in care, caregivers, and staff of community-partner organizations). – Ideally, this individual would be familiar with all involved

systems, including child welfare and the various employment and school systems (primary, secondary, college/university/apprenticeship programs) in the region.

crecs.uOttawa.ca

Question 2. Does the CWECT program meet the needs of its stakeholders?• Recommendation: The three sponsoring Ontario

Ministries need to provide increased and consistent support to the individual CWECTs. For instance, the three Ministries could:– Sponsor provincial or regional workshops to help local

CWECTs improve their capacity to plan, implement, evaluate, and improve their services.

– Compile a database of CWECT success stories, for local programs to use, and place this database on the central website for easy access.

crecs.uOttawa.ca

Question 3. How is the CWECT program being implemented?• Both CWECTs have devoted many resources to activities

for the youth, and attempts have been made to increase knowledge and nurture relationships with caregivers and community partners.

• However, implementation of program activities has been largely affected by when they are receiving their funds; i.e., annual funds have typically been received halfway through the fiscal year, which has led to a sense of urgency to plan and implement activities.

crecs.uOttawa.ca

• Recommendation: At minimum, funds for the individual CWECTs should be delivered promptly at the beginning of each fiscal year. Improvements in program implementation are also likely if the teams are able to receive funds for multiple years at once. For instance, if the CWECTs were allocated funds on a 3-5 year basis, this would allow the teams to set and plan for longer-term goals.

Question 3. How is the CWECT program being implemented?

crecs.uOttawa.ca

Question 4. Does the CWECT program appear to be working?• CWECT members all said they were hoping they were

making a difference for youth in care and felt as though they were, but had no real data/evidence to confirm this.

• However, the biggest impact seems to be in the area of fostering partnerships between child welfare and education (e.g., supporting targeted conversations, increasing the accountability for both to adhere to a plan to support Crown Wards).

crecs.uOttawa.ca

Question 4. Does the CWECT program appear to be working?– “We are getting much closer to talking the same

language now in terms of the educational needs of these kids.”

– “I think we’re having an impact because we continue to work together, and we continue to gain more community partners. I think we gain a better understanding the longer we work together. The more we understand the perspective of others will help us in supporting kids.”

crecs.uOttawa.ca

Conclusions & Next Steps• Consider funding a pilot outcome evaluation of one or

more local CWECT programs as a needed follow-up to the present formative evaluation.

• CWECT committee members (and the three sponsoring Ontario Ministries) should acquaint themselves, caregivers, and community partners (including schools) with the wealth of accessible online information about effective, evidence-based methods of intervening to help young people in care to improve their educational outcomes (e.g., one-to-one tutoring). Such information is provided in chapter 4 of our evaluation report.

crecs.uOttawa.ca

Acknowledgements

• Thank you especially to:– Highland Shores-Frontenac, Lennox &

Addington CWECT– Lanark, Leeds, & Grenville-Renfrew CWECT

crecs.uOttawa.ca

Contact information and to receive ane-copy copy of our final evaluation report

Kelly Weegar• Email: kweeg065@uottawa.ca

Robert Flynn• Email: rflynn@uottawa.ca

Centre for Research on Educational and Community Services (CRECS)University of Ottawa136 Jean Jacques LussierVanier Hall, Room 5002Ottawa, Ontario, Canada K1N 6N5