Strategies for communicating evaluation findings · Strategies for communicating evaluation findings ... Zarinpoush, Von Sychowski, & Sperling, 2007. Planning for knowledge exchange

Strategies for communicating

evaluation findingsMay 7, 2013

Housekeeping notes

• This webinar will be recorded, and will be available on the Centre’s website as an educational resource.

• The slides have been sent to participants.• Log in with a phone whenever possible for optimal

audio quality.• We have staff online to assist with any technical

difficulties.• There will be a short evaluation survey for all webinar

attendees at the completion of the webinar.

Housekeeping notes

• This webinar will be followed by a question and answer period, however questions are encouraged throughout the presentation.

• Questions can be submitted electronically or verbally. Specifics around this process will be clarified at the end of the webinar .

Addressing questions:

Presenters

Lindsey ThomsonProgram Associate

Ontario Centre of Excellence for Child and Youth Mental Health

Trevor BosseQuality Assurance Associate

Child and Community Resources

What is the Centre?

Vision

Optimal mental health and well‐being for children and youth.

Mission

We bring people and knowledge together to strengthen the quality and effectiveness of mental health services for children, youth, and their families and caregivers.

5

What is the Centre?The plan outlines three key strategic priorities:

Goal 1Foster a culture of organizational learning to support agencies in using evidence to improve client outcomes

Goal 2Build and develop collaborative partnerships to sustain capacity within mental health services

Goal 3Be a true learning organization and lead by example

6

The Learning Journey

Overview

• To talk about the importance of communicating evaluation findings

• To suggest ways of tailoring communication to different stakeholders

• To have an agency share its knowledge exchange experiences

• To share ideas on innovative knowledge exchange approaches

• To share information on helpful resources

Goals of the webinar:

What is knowledge exchange?

The Centre’s definition of knowledge exchange

“At the Centre, we see knowledge exchange as the continuous and purposeful flow of evidence and expertise that fosters mutual learning and leads to stronger mental health care for Ontario children, youth and families. By gathering, sharing and using the best available information, we work to amplify the impact of current knowledge, build meaningful connections, and strengthen our collective ability to respond to evolving challenges in the field”.

Various terms for knowledge exchange

Knowledge translation

Knowledge mobilization

Knowledge transfer

Implementation science

Outreach and interactions

Zarinpoush, Von Sychowski, & Sperling, 2007

Planning for knowledge exchange

Why communicate evaluation findings?

• To share information about the evaluation process and its findings

• To facilitate understanding of the evaluation among different audiences

• To support decision‐making about the program and/or the evaluation

Torres, R.T., Preskill, H., & Piontek, M.E. (2005). Evaluation strategies for communicating and reporting: Enhancing learning in organizations (2nd ed.). Thousand Oaks, CA: Sage Publications.

Preparing a knowledge exchange plan

• Planning for knowledge exchange should occur while planning the evaluation project

• The plan should include at a minimum:o a listing of the possible audienceso information on the best ways of communicating the evaluation findings to each audience

Preparing a knowledge exchange plan

Knowledge exchange questions

What are our goals in sharing evaluation findings?

Who are the stakeholders?

What are the key messages of the evaluation?

Who needs to know what?

How can each audience best be reached?

What are their information needs?

How can I best present this information to enhance understanding and use?

How will you share sensitive or negative results?

Identifying audiencesPossible audiences are those who:

• sponsored or commissioned the evaluation.• will make decisions based on the evaluation results.• plan, develop or deliver the program being evaluated.

• participated in the evaluation and provided information.• participate in the program.• fund the program.

• are interested in the program (e.g. advocacy groups).• have a right to the information (e.g. legislators, taxpayers).

Adapted from: Torres, R.T., Preskill, H., & Piontek, M.E. (2005). Evaluation strategies for communicating and reporting: Enhancing learning in organizations (2nd ed.). Thousand Oaks, CA: Sage Publications.

Tailoring the communication to the audience

The communication strategy will depend on audience characteristics such as:• accessibility (e.g. geographic location)

• reading ability

• familiarity with (or interest in) the program

• role in decision‐making

• familiarity with evaluation

Adapted from: Torres, R.T., Preskill, H., & Piontek, M.E. (2005). Evaluation strategies for communicating and reporting: 2nd edition. Thousand Oaks, CA: Sage Publications, Inc.

An example ‐ presentationAudience Message Delivery method Messenger Evaluation

What are the general demographics of the audience?What may be their expectations? How can I reach them, let them know about the presentation?

What problems or questions should be addressed?What are the concluding remarks and actionable messages?

Using visual aids, short examples, or anecdotes helps to engage the audience. Posing and answering questions are techniques to encourage a two‐way flow of information and prevent a “knowledge dump.” The pace should be adjusted to the knowledge of the participants.

What are the required knowledge and skills for this presentation and who has them? Credibility and affiliations?

What are the realistic, achievable objectives of the presentation? What are the benefits to the users/receivers? What specific knowledge or message can they take back to their organizations and share?

Reporting your findings

Reporting findings – evidence tables

Evaluation question Evidence Conclusion Recommendation

Have children aged 6‐12 demonstrated increased social competence?

Self report – Statistically significant change on the SSRS and CGI scales

Parent ratings –Statistically significant change on the SSRS and CGI scales

Observation – Positive change on the BOSS

Qualitative interviews – campers, parents and staff

Children and youth leave the program with increased social competence

Seeing as this program meets a specific need for this population of children and youth with learning disabilities, funding and support for this program should continue.

Example: Evaluation of a therapeutic summer residential program

Adapted from: Evaluation Implementation Grant 889 ‐ Integra

Sharing Evaluation Results

Celebrating with Stakeholders

Trevor Bosse | May 7, 2013

What we want to achieve…


Our Valued Stakeholders




Funders:• Ensure program is meeting the needs of

clients• Program accountability• Resource allocation



Clients:• Why do we evaluate?• At a glance, how successful are we at

achieving what we’ve set out to do?• How are you improving your services?



Staff:• How did we do?• Ok, so what’s next?



Governing Bodies (Board of Directors):• How did we do?• How are we improving?

Sharing Results

1. Comprehensive Final Report2. One‐pager for external stakeholders3. One‐pager for Board of Directors4. Crafty ways of sharing results5. Presenting Results to Teams







www.communitysolutions.ca


www.communitysolutions.ca


Present Results to Teams



The little things, the BIG things & everything in‐between!!!


2012 Knowledge Exchange

• Offered to all CCR staff in June 2012• Increasing the capacity of staff surrounding

program evaluation and research, and how they contribute to evidence‐informed practice and organizational learning





What we found…• 99% of attendees (n=109) agreed or strongly agreed that they were satisfied with the workshop

• 100% agreed or strongly agreed that the format of the session was effective for their learning

Some examples of knowledge exchange strategies

Innovative knowledge exchange strategies

• Poster

• Video

• Infographic

• Photovoice

• Board game

• Drama/theatre

• Storytelling/graphic novel

• World Café

Acknowledgements: Work at Adventure Central is part of CYFAR Project # 2003-41520-01560. This research was supported by the Susan Barkman Research and Evaluation Award, National Association of Extension 4-H Agents.

Theresa M. Ferrari, Nate Arnett, & Kristi Lekies – Ohio State University Extension

Youths’ Perspective on Their Long-Term Participation in an Urban 4-H Youth Development Program: Results of a Qualitative Study

I’ve done

“things you

would never

imagine you

would do…It

has just opened

new doors.

“There is a lot of team building ….before we even became group

leaders and actually worked here, we were still doing team building activities in our groups when we were younger. That helped us to adapt to other

people, like problem solving skills and being able to communicate.”

“There is a lot of team building ….before we even became group

leaders and actually worked here, we were still doing team building activities in our groups when we were younger. That helped us to adapt to other

people, like problem solving skills and being able to communicate.”

“I like it a lot here. Not only is it

educational, but it is fun here as well.”

“I like it a lot here. Not only is it

educational, but it is fun here as well.”

Participants

•16 participants between 11 & 16 years old

•Participated for at least 3 years (range from 3 to 7 years)

Purpose of the Study

Explore the process and outcomes of long-term participation in a youth development program.Research Questions

1. How has participation been helpful?

2. What skills & attitudes have youth acquired?

3. What opportunities have been afforded to them?

4. What program features have captured their interest & engaged them in sustained participation?

•Youth obtain developmental benefits from consistent participation in well-run, quality youth development programs.

•Frequency, duration, and breadth of participation relate to more positive outcomes.

•Joining and persisting in out-of-school activities is a dynamic process.

•Beginning to understand features that contribute to engagement.

Program DescriptionResearch BaseResearch Base

Methodology

•Partnership between OSU Extension 4-H Youth Development & Five Rivers MetroParks in Dayton, OH

•Opened in October 2000

•Hub of out-of-school activities 4 days a week

•Youth aged 5 to 18

•Computer lab and homework assistance

•Programs focus on science, nature, literacy, & healthy living, workforce skills, leadership, & cultural literacy

•Diverse array of full-time staff, part-time staff, & volunteers

Implications & Recommendations

ResultsResults

Data Collection & Analysis

•Four focus groups

•Qualitative analysis for themes

Youth Cited Three Key Themes Related to their Long-Term Participation:

•Multiple OpportunitiesFrom an early age, youth had new and challenging opportunities to demonstrate leadership, have a “real” work experience, travel, go outdoors and engage with nature, and get homework help.

•Having Fun and LearningBeyond keeping them out of trouble, youth recognized that they were learning and having fun while learning skills.

•BelongingMultiple references to “home” and “family” were used to describe the program environment to include staff and peers.

•Use of a model of positive youth development programming connecting program features, developmental needs, and outcomes

• Increasing opportunities for youth as they get older

•Connecting youth with other leadership opportunities

•Potential model for other urban 4-H and youth development programs to implement

Video

Present video:http://www.youtube.com/watch?v=YJSMa7AA3cU

Infographic

Source: http://infographiclist.com/2012/05/20/foster-youth-infographic/

Photovoice

Source: “Capturing the unique strengths and challenges of thepedestrian environment in Fort Frances, ON” –Safe KidsCanada and local partners

Board game

Source: http://www.youth.society.uvic.ca/node/297

Drama/Theatre

Present part of video: http://youtu.be/8oyQsXitWzM

Storytelling/Graphic novel

Source: http://www.youth.society.uvic.ca/sites/default/files/images/PAGE%204-1.jpg

World Café

Present video: http://www.youtube.com/watch?v=vRM1I90SOUw&feature=endscreen&NR=1

Summary• Knowledge exchange should be planned in the early stages of an evaluation project, and should happen at various points throughout the project.

• Communication strategies should be tailored to the audience.

• A range of communication strategies should be considered.

• Learning from others’ knowledge exchange experiences can be helpful.

We’d like to hear from you…

Any questions?

Questions?

To submit questions electronically, use the Question box located in the control panel.To submit questions verbally, use the ‘Raised Hand’ icon also located in the control panel.

Upcoming Webinars

To infinity and beyond: Sustaining evaluationMelissa Jennings & Surbhi Bhanot‐Malhotra

May 21 at 1 p.m.

Final report preparation and writingLindsey Thomson & Jennifer Puddicombe

June 4 at 1 p.m.

ContactFor more information on this webinar or topic

Lindsey ThomsonProgram Associate

Ontario Centre of Excellence for Child and Youth Mental HealthPhone: 613‐737‐2297 ext. 3426

[email protected]

Trevor BosseQuality Assurance Associate

Child and Community ResourcesPhone:(705) 222‐5000 ext. 2818

[email protected]

Documents

Strategies for communicating evaluation findings · Strategies for communicating evaluation findings ... Zarinpoush, Von Sychowski, & Sperling, 2007. Planning for knowledge exchange