Research and

Evaluation Data:A Chat with the Mentoring

Research Experts

2011 National Conference • Dallas, Texas • June 14 - 16

Panelists

David DuBois, Ph.D., Professor, Institute for Health Research and Policy,

University of Illinois at Chicago

Timothy Cavell, Ph.D., Professor and Director of Clinical Training, Department of Psychology, University of Arkansas

Michael Karcher, Ph.D., Professor of Education and Human Development, University of Texas, San Antonio.

Findings of Recent Meta-Analysis (DuBois, Portillo, Rhodes, Silverthorn, & Valentine, in press)—Not for external distribution or citation at this point without permission

Analyzed results from 73 evaluations conducted 1999- 2010The “Good News”o Positive program impacts on youth in multiple domains: behavioral, social,

emotional, and academic (including school attendance, grades, academic achievement test scores)

o Individual programs often have made in-roads in two or more outcome domains (e.g., social and academic)

o Two-dimensional benefits – preventing declines in youth outcomes that might otherwise occur and promoting improvements

o Benefits generalize across key dimensions such as age of youth, format (1-to-1 vs. group), and age of mentor (older peers vs. adults)

o Magnitude of effects generally within range of those found for related types of child and youth interventions

David DuBois

2011 National Conference • Dallas, Texas

Type of outcome Current Other meta-analyses

Attitudinal/Motivational 0.19 0.23r, 0.25b

Social/Relational 0.17 0.15a, 0.17i, 0.24r, 0.29b, 0.39g

Psychological/Emotional 0.15 0.10a, 0.17p, 0.19d, 0.24r, 0.37b

Conduct problems 0.21 0.02j, 0.07k, 0.14h, 0.15s, 0.21a, 0.21e, 0.22r, 0.30b, 0.30c, 0.41l

Academic/School 0.21 0.11a, 0.23n, 0.27r

School attendance 0.19 0.14b

Grades 0.24 0.22b

Achievement test scores 0.18 0.11a, 0.20b, 0.24f, 0.30c

Physical health 0.06 0.08m, 0.17t, 0.29q, 0.41o

Comparison of Mean Post-Treatment Effect Sizes for Mentoring Programs in the Current Meta-Analysis to Effect Sizes Reported in Other Meta-Analyses of School- and

Community-Based Interventions for Children and Adolescents

The “Challenging News”o No evidence of improved effectiveness over prior generation of

programs o Too few studies to evaluate impacts on several key outcomes

(e.g., school drop-out, juvenile offending)o Same largely true for longer-term, “follow-up” effects

The “Informative News”o Several program practices associated with greater effectiveness,

including: Targeting “at risk” youth (exception: populations high on both

individual and environmental risk) Utilizing mentors with educational/occupational backgrounds

that are a good fit with program goals Matching youth and mentors based on similarity of interests Supporting mentors in adopting teaching and advocacy roles

David DuBois

2011 National Conference • Dallas, Texas

Bottom-Line Assessmento Value in continued investment in youth mentoring as an

intervention strategy within the policy arenao Strongest argument can be made for utilization of mentoring when

interest is in promoting outcomes across multiple areas of a young person’s development

o Policy recommendations to maximize ROI1) Ensure adherence to core practices (e.g., mentor

screening and training) essential to program quality2) Facilitate research-informed development of program

innovations3) Foster stronger practitioner-researcher collaboration in

design, implementation, evaluation, and ongoing refinement of programs

David DuBois

2011 National Conference • Dallas, Texas

Michael KarcherTEAM Framework: Understanding Relationship Activities

Suggests why we should train mentors to consider:

Focus: How relational vs. directive are the mentoring interactions (activities, discussions)?

Authorship—How collaborative the conversation or activity decision is? Who authors their story?

Purpose: Do interactions serve serious, future- oriented, adult or playful, youth-oriented

goals?From Karcher, M.J. & Nakkula, M.J (2010). Youth mentoring with a balanced focus, a shared purpose, and collaborative interactions. In “Play, talk, learn: Promising Practices in Youth Mentoring,” Jossey-Bass.

Michael KarcherTEAM Framework: Understanding Relationship Activities

What should Baloo and Bagheera doto get Mowgli where he needs to be?

Arrival in the Man Village (Success: mentor-mentee relationship quality)

Baloo-type(Relational

Conversations

Bagheera (goal-directedconversations)

casual conversation

talk about family

talk about friends

listening & learning

talk about school

discuss attendance

discuss behavior

talk about the future

Copyright 2009 Michael J Karcher (Note: Baloo, Bagheera, and Mowgli are trademarked by the Walt Disney Corporation)

Karcher, M. J., Herrera, C., & Hansen, K., (2010).“I dunno, what do you wanna do?”: Testing a framework to guide mentor training and activity selection. New Directions in Youth Development, 126.

Michael Karcher

Michael Karcher

What do we know about bullying• Prevalence

– 30-40% of youth involved in bullying– Peaks in middle school – # of victims drops, but they are more visible

• Bullying can be physical, verbal, or relational

• Victims = unpopular and friendless children

Timothy Cavell

Peers Seldom Intervene

Anti-bullying Interventions• School-wide programs can reduce overall # of

children being bullied

• But success requires whole school buy-in and effects tend to fade with time

• We need programs for chronically bullied children

Timothy Cavell

School-Based Mentoring:An Indirect Way to Help

• Why indirect help?• Many bullied children…

– Reluctant to ask for help or resist help– Doubt whether adults can or will help– Believe adult help will make things worse

Timothy Cavell

Lunch Buddy Mentoring: One type of SBM• Mentors visit twice/week for 30 minutes during lunch period• College student mentors who sit with mentee and peers

Recent Pilot Study• Method

– Lunch Buddy children (n = 12)– 2 matched control groups

• Same Controls (n = 12) – same school• Different Controls (n = 12) – different school

• Results– LB children significantly less bullied after 1 semester (peer reports)– Significantly less than Different Controls (but not Same Controls)– Parents & teachers: high satisfaction with LB mentoring– No evidence of harm to bullied children (child, parent, teacher

ratings)

Elledge, Cavell, Ogle, & Newgent (2010). Journal of Primary Prevention, 31, 171-187.

Timothy Cavell

What One Lunch Buddy Mentor Said:

There were kids who would ask why I sat with him. I’d say he was my friend and it was pretty cool to sit with him. They looked shocked, but then they began to sit next to us on every visit. It was amazing how their little attitudes towards my mentee changed by me saying I was his friend and I like sitting with him.

Timothy Cavell

Questions for Panelists1. Why is fidelity to a program model so important?

2. Why is our Outcome Evaluation System critical in communicating effectiveness?

3. What are some simple things agencies can do with their AIM data to help improve practice and demonstrate value to stakeholders?

4. What do you feel is the most important change mentoring programs can make to their CBM programs to improve youth impact?