Embed Size (px)
Citation preview
Strategies for Strategies for communicating communicating
evaluation findingsevaluation findings
Housekeeping Notes
This webinar will be recorded, and will be available on the Centre’s website as an educational resourceThe slides have been sent to participantsLog in with a phone whenever possible for optimal audio qualityWe have staff online to assist with any technical difficulties
Housekeeping Notes
§ This webinar will be followed by a question and answer period, however questions are encouraged throughout the presentation
§ Questions can be submitted electronically or verbally. Specifics around this process will be clarified at the end of the webinar.
Addressing Questions:
ABOUT THE CENTRE
What is the Centre?
VisionOptimal mental health and well-being for children and youth.
MissionWe bring people and knowledge together to strengthen the quality and effectiveness of mental health services for children, youth, and their families and caregivers.
5
What is the Centre?The plan outlines three key strategic priorities:
Goal 1Foster a culture of organizational learning to support agencies in using evidence to improve client outcomes
Goal 2Build and develop collaborative partnerships to sustain capacity within mental health services
Goal 3Be a true learning organization and lead by example
6
Overview
§ Gain an understanding of the importance of communicating findings
§ Understand how to gear your report to your audience
§ Hear about some creative ways of reporting findings
§ Gain some useful examples and resources for future reference
What we hope you will take away today:
Presenters
Ilana SmythResearch Associate
Ontario Centre of Excellence for Child and Youth Mental Health
Tracey MacLaurinKE Project Officer
Ontario Centre of Excellence for Child and Youth Mental Health
Presenters
Melissa Legree, M.ADS (ABA)Behaviour Consultant, Lake Ridge Community
Support Services (LRCSS)
Yona Lunsky, Ph.D., C.Psych.Clinical Research Manager, Dual Diagnosis
Program, Centre for Addiction and Mental Health (CAMH)
Section Head, Dual Diagnosis Program, Centre for Addiction and Mental Health (CAMH)
Purpose of CAMH/LRSS Knowledge Exchange Event
§ Sharing information• About two social skills programs for children and
youth with Asperger syndrome or high functioning autism spectrum disorders
• On the program evaluation process and our results
• On our lessons learned§ Building a community of practice
• Opportunity for different parts of province to hear and share information about what they’re doing for kids with ASD and their families
WHAT IS KNOWLEDGE EXCHANGE (KE)?
Some definitions…
“The exchange of knowledge refers to the interaction between the knowledge user and the researcher, resulting in mutual learning.”
- CIHR
The two-way dialogue and exchange of information between those who generate and those who receive and use knowledge, which is also operational throughout the project or research study. These activities serve to facilitate the use of knowledge in practice.
– Barwick et al. 2005
“The collaborative problem- solving between researchers and decision makers that happens through linkage and exchange. Effective knowledge exchange involves interaction between knowledge users and researchers and results in mutual learning through the process of planning, producing, disseminating, and applying existing or new research in decision-making.”
-CHSRF
Various terms for Knowledge Exchange
§ Knowledge translation
§ Knowledge mobilization
§ Knowledge transfer
§ Implementation science
Key components of a KE Strategy
Goal• Improved CYMH outcomes
Identify the KE need• Speak with other agencies, experts in the field, an “opinion leader” to identify need• Conduct focus groups, needs assessments• Lead by example/mutual learning
Key components of a KE Strategy con’t
Determine levels of impact • People – type of service provider & service-user• System (cross-sectoral) – mental health, health, education• Organization
Identify facilitators & barriers to KE efforts• Leadership buy-in• Change readiness• Awareness• Collaborative partnership
Key components of a KE Strategy con’tStrategies to support KE
• Capacity building• Monitoring, adapting & mutual learning• Tailor to audience capacity & need• Marketing, branding• Implementation/mobilization - mode of delivery
Outcomes related to KE efforts• Foster organizational learning• Develop/support a culture of EIP • Evaluating outcomes• Sustaining knowledge• Creating community of practice• Linking agencies for mentorship
Why communicate evaluation findings?
§ To share information
§ To increase understanding
§ To support decision making
§ Engage stakeholders in evaluation
Torres, R.T., Preskill, H., & Piontek, M.E. (2005). Evaluation strategies for communicating and reporting: 2nd edition. Thousand Oaks, CA: Sage Publications, Inc.
Considering your audience
Who is your audience?
Type of Audience
Primary Audience
Secondary Audience
Level of Involvement
Directly involved Not directly involved
Role in Project
May fund program May apply findings to their work
Link to Evaluation
May be responsible for evaluation
May make decisions based on evaluation
Gearing the findings to your audience
§ How familiar is the audience with the program?
§ What is the best way to reach your audience?
§ What experience do they have in comprehending evaluation findings?
§ How comfortable are they reading program logic models or evaluation frameworks?
Torres, R.T., Preskill, H., & Piontek, M.E. (2005). Evaluation strategies for communicating and reporting: 2nd edition. Thousand Oaks, CA: Sage Publications, Inc.
Reporting your findings
Reporting findings
Evidence TablesExample: Evaluation of a therapeutic summer residential program
Evaluation Question Evidence of Success
Have children aged 6-12 demonstrated increased social competence?
Self report – Statistically significant change on the SSRS and CGI scales
Parent ratings –Statistically significant change on the SSRS and CGI scales
Observation – Positive change on the BOSS
Qualitative interviews – campers, parents and staff
Adapted from Evaluation Implementation Grant 889 - Integra
Reporting findings
Evaluation Question Evidence Conclusion Recommendation
Have children aged 6-12 demonstrated increased social competence?
Self report – Statistically significant change on the SSRS and CGI scales
Parent ratings – Statistically significant change on the SSRS and CGI scales
Observation – Positive change on the BOSS
Qualitative interviews – campers, parents and staff
Children and youth leave the program with increased social competence
Seeing as this program meets a specific need for this population of children and youth with learning disabilities, funding and support for this program should continue.
Evidence Tables
Approaches to KE
Dissemination: a one-way movement of information from producer/sender to potential user/receivers.
Presentation: a form of communication with an audience.
Training Session: transferring knowledge and collecting feedback from trainees by testing their knowledge at the end of the session.
Conference/symposium/seminar/webinar/forum: initially organized to bring a group of people together for various purposes such as discussion, presentation or networking.
Workshop: a setting that is planned for interaction, sharing information, and active learning.
Meeting/brainstorming/roundtable/networking: designed to help people share information and ideas in a structured environment. These activities can generate new ideas, find solutions to a problem, build relationships, and create opportunities for innovations.
Outreach and interactions
Zarinpoush, Von Sychowski, & Sperling, 2007
Some examples of KE Strategies
An example - presentationAudience Message Delivery Method Messenger Evaluation
What are the general demographics of the audience? What may be their expectations? How can I reach them, let them know about the presentation?
What problems or questions should be addressed? What are the concluding remarks and actionable messages?
Using visual aids, short examples, or anecdotes helps to engage the audience. Posing and answering questions are techniques to encourage a two-way flow of information and prevent a “knowledge dump.” The pace should be adjusted to the knowledge of the participants.
What are the required knowledge and skills for this presentation and who has them? Credibility and affiliations?
What are the realistic, achievable objectives of the presentation? What are the benefits to the users/receivers ? What specific knowledge or message can they take back to their organizations and share?
Our example – webinar on communicating findings
Audience Message Delivery Method Messenger Evaluation
Advertise on website; send out email invitation; audience is mostly grant recipients looking to share their findings from an evaluation; they are hoping to learn best methods for sharing their findings
Should address why it’s important to share findings; tailoring their message is important; give some examples of successful methods and questions to think about.
Asking an agency partner to talk about their experiences as real life examples; preparing a slide deck with examples, tables and diagrams; posing questions to the audience ahead of time and during presentation; encouraging discussion throughout.
Presenters should have first hand knowledge of how to communicate findings, and/or familiarity with knowledge exchange activities.
Participants should take away some examples of how to communicate findings and how to tailor their findings to a specific audience.
CAMH/LRSS example – video conferenceAudience Message Delivery Method Messenger Evaluation
Advertise on website; send out email invitation; use list-serves, also ask Ministry about their contacts so they could circulate.Audience is mostly clinicians interested in running groups or doing an evaluation.
Social skill or other interventions for families with ASD can be done.Interventions can be evaluated!Agencies can learn and improve services from evaluations.
Bringing together two projects to engage more people, show overlap and unique issues.Show video that makes what we did (intervention) more alive, concrete.Had times in session to allow for discussion (but tricky with videoconference and so many sites).
Some presenters had more experience than others with doing video-conferences.
We all had experience presenting and we had some credibility in the field.
Audience took away two examples of how to run and evaluate groups for youth with ASD and their families.Audience learned about available resources for conducting their own evaluations.
Summary
§ Communicating your findings and engaging in knowledge exchange activities is important at all stages of your evaluation project.
§ Communicating your findings should be tailored to your audience.
§ Look for creative and unique ways to communicate your findings.
§ Look at what has been done – what has worked, and what hasn’t worked.
We’d like to hear from you…
Any Questions?
Questions?
To submit questions electronically, use the Question box located in your control panelTo submit questions verbally, use the ‘Raised Hand’ icon also located in the control panel
Further ResourcesKnowledge Translation Planning Template
Barwick, Melanie (2008) Hospital for Sick Children, TorontoAvailable at http://www.melaniebarwick.com/training.phpSupplement to the Knowledge Translation Research Plan TemplateAvailable at http://www.sickkidsfoundation.com/grants/ktmethod/ktsupplement.html
Ontario Centre of Excellence for Child and Youth Mental HealthA Tool Kit on Knowledge Exchange: Doing More With What You Know
Evaluation Strategies for Communicating and Reporting 2nd Ed.: Enhancing learning in organizations. (2005) Sage Publications. Torres, R.T., Preskill, H. & Piontek, M.E.
Zarinpoush, F, Von Sychowski, S, & Sperling, J. Department of Canadian Heritage, Imagine Canada. (2007). Effective knowledge transfer & exchange for nonprofit organizations: a framework. Toronto: Government of Canada.
ContactFor more information on this webinar or topic, please contact:
Ilana SmythResearch Associate
Ontario Centre of Excellence for Child and Youth Mental HealthPhone: 613-737-7600 ext. 3325
Tracey MacLaurinKE Project Officer
Ontario Centre of Excellence for Child and Youth Mental HealthPhone: 613-737-7600 ext. 3732
ContactFor more information on this webinar or topic, please contact:
Melissa LegreeBehaviour Consultant
Lake Ridge Community Support ServicesPhone:(905)666-9688, ext. 230
Yona LunskyClinical Research Manager, Dual Diagnosis Program
Centre for Addiction and Mental Health (CAMH)Phone:(416) 535-8501 x7813
