Upload
allan-hodge
View
217
Download
2
Tags:
Embed Size (px)
Citation preview
Lessons learned from the Evaluation of Baltimore Expanded School Mental Health:
Challenges and Solutions
Georgetown University National Technical Assistance Center Webinar SeriesOctober 21, 2010
Making the Case for Evaluation of School Mental Health Programming: Strengthening Partnerships for Positive Outcomes
Bruno J. Anthony, Ph.D.
Georgetown University Center for Child and Human Development
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
Collaborators• Baltimore City Public Schools
– Louise Fink– Ike Dibor – Shawn Hales
• Baltimore Mental Health Systems– Jane Paplinger– David Jones– Denise Wheatley-Rowe
• Georgetown Center for Child and Human Development– Bruno Anthony– Joyce Sebian– Courtney Holland– Clare Dunne
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
Overview BESMH• Baltimore ESMH a recognized state and national leader for more than 20
years.
• ESMH services have complemented and supplemented the services provided by Baltimore City Public Schools (social workers, psychologists, and guidance counselors)
• Currently, prevention and treatment services provided in 106 schools (approximately half of the schools in Baltimore City) through 12 provider agencies and 106 clinicians.
• Intended to address barriers to learning to enable students to make better use of educational programs in their schools and to provide an alternative to mental health services within structure of Special Education System
• Funding: Braided funding from school system, Department of Human Resources, Baltimore Substance Abuse System, Baltimore Mental Health System.
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
Evaluation of BESMH• Purpose: Assess the impact of a comprehensive ESMH program
– Test assumptions that ESMH is beneficial in terms of academic outcomes
– Inform programming
– Inform ongoing evaluation and sustainability efforts.
• Funding– Planning: Internal funding from BESMH budget
– Implementation: State of Maryland Mental Health Transformation State Incentive Grant (TSIG)
Demand for academic accountability heightens the need for studies examining the specific impact of ESMH
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
Challenges and Approaches: Developing a Meaningful and Feasible Plan
Key Informant Input– Focus Groups (administrators, program directors,
clinicians/providers, caregivers, youth)– Consultation (advisory committee, monthly program directors
meetings, leaders in the field)– School system partners (DREAA, Student Support Services, leaders)
Goals– Identifying important barriers and challenges – Identifying the appropriate and relevant outcomes – Incorporating content expertise in understanding ESMH in
Baltimore– How to use the evaluation findings to improve the ESMH program
and for advocacy.
A participatory approach
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
Challenges and Approaches: Examining Outcomes at Individual Level
. Employed the “zipper” for linking separate confidential data sets that
contain personal identifying information while preserving anonymity.– Safe analyst to link and process data who is not part of either the
research team, the agency with overall administrative oversight over the ESMH program or the school system.
– Providers send service use data set with ID numbers sent to the “safe analyst,” who has no access to student names.
– School system sends demographic and academic/school functioning with ID numbers to safe analyst.
– Safe analyst uses ID numbers to merge the two data sets then encrypts ID numbers
Risk of connecting private school and ESMH data
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
Challenges and Approaches: Capturing Spectrum of ESMH Services
Electronic data collection– Developed in a participatory process
• Individual: For each date of contact, the type(s) of services and, at the end of each month, the focus of treatment and the type of the interventions.
• Group: attendance in prevention and treatment groups.• Universal: four types of prevention activities.
– Training: direct contact, manual and web-based
Service logs were not capturing the range of promotion, prevention and treatment work carried out by clinicians.
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
Extent and Type of Services
ESMH Clinicians touch an enormous number of students, caregivers and school staff, extending the range of clinical service that could not happen in more traditional settings.Clinicians had direct contact with more 6254 students -1/5th
of students attending the 106 schools that support ESMHA large part of the clinicians work involves indirect,
collateral services
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
Percentage of Students by Grade
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
Percentage of Service Types
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
Percent of Direct and Indirect Services
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
Service Focus
• Clinicians focus on issues that disrupt a student’s engagement with school
• Clinicians for the most part report using approaches with an evidence base
• Family/caregiver engagement is challenging
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
Percentage of Focus Areas Coded
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
Percentage of Intervention Types Coded
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
Challenges and Approaches: Comparison Groups
Quasi-experimental strategies to support causal influences of ESMH.
Approaches:– Contact Groups
– Matched Comparison Groups
– Dosage Groups
As is often the case with on-going, community-based programs, a full-fledged experimental (random-assignment) design was not practical, feasible or ethical.
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
E-E: Students receiving ESMH services in an ESMH schoolNoE-E: Students NOT receiving ESMH services in an ESMH schoolNoE-NoE: Students NOT receiving ESMH services in a school without ESMH
Distribution of Students in Different Dosage Groups
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
Matched Comparison Group Formation• Propensity scores
– Probability of a student in a school with an ESMH clinician receiving ESMH services, given a set of background variables for that individual.
– Determined background risk factors that predicted participation in ESMH using Binary Logistic Regression (2007-08 attendance, suspensions, MSA Reading and Math scale scores).
– Children receiving ESMH would have high propensity scores based on the set of background variables . Students with similar background characteristics but not receiving ESMH services should also have high propensity scores.
– Propensity score of each ESMH student matched with a student drawn randomly from those students not enrolled in ESMH to form Comparison Group.
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
Preliminary Overall Results
A greater percentage of students attending a school with ESMH services and accessing these services showed– Improvement over the course of the year in Math and
English benchmarks.
– Decreases in suspensions
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
Challenges and Approaches: Outcome Measurement Framework
Integrate data needs and system pressures of different stakeholders.– Use available data records, outcome measurements already in place,– Stakeholders recognized the imperative of demonstrating
effectiveness in the school setting with selected mediating/moderating contextual variables.
– The importance of looking at school outcomes (e.g., academic success, promotion, absences and suspensions) to ensure sustainability
Devising an outcome measurement framework that satisfies different stakeholders without overburdening clinicians
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
Evaluation Contributions to Sustaining ESMH
Building evaluation and research model – participatory process Building infrastructure for ongoing evaluation and research– Service Trends
• Impact on programming decisions• Impact on financial models• Training
– Leveraging other initiatives. Utilizing evaluation data to make the case for ESMH program– Case for Ancillary and Supportive Services
• Teacher Consultation • Care Coordination• Family contacts
– Case for School-related Benefits • Improvement in academic measures• Positive effects on school attendance and behavior
© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY© 2010 NATIONAL TECHNICAL ASSISTANCE CENTER FOR CHILDREN’S MENTAL HEALTH, GEORGETOWN UNIVERSITY
ContactingContacting UsUs
BESMH Evaluation Team:
Bruno Anthony, Ph.D.Professor and Director of Research and Evaluation,
Email: [email protected]
Joyce Sebian MS. Ed.Senior Policy Associate,
Email: [email protected]
Georgetown University Center for Child and Human Development
Department of Pediatrics3300 Whitehaven Street, NW, #3300
Washington, DC 20007
27