Outcomes Assessment in School Outcomes Assessment in School Mental HealthMental Health
Webinar and dialogue sponsored by the
Quality and Evidence-Based Practice (QEBP)
Practice Group and the IDEA Partnership
March 16, 2011
Today’s webinar Today’s webinar • Overview of QEBP practice group
• Highlights of QEBP practice group discussions at 2010 CSMH conference (Albuquerque, NM)
• Evaluation of school mental health (SMH) programs in Minnesota Public Schools (Dr. Mark Sander)
• Psychosocial, Placement and Cost Assessment in Maryland’s SMH programs for youth in special education (Dr. Sharon Stephan)
• QEBP updates from the CSMH (Dr. Nicole Evangelista)
• Call for Papers on SMH outcomes assessment - special issue of Advances in School Mental Health Promotion(Dr. Michael Kelly)
QEBP Practice Group Mission and Priorities
• Mission: To provide resources and promote sharing of information across individuals and groups interested in improving the quality of school mental health (SMH) programs and services.
• Priority areas include: • improving dissemination and sharing of evidence-based
practices in SMH; • bridging the research-practice and practice-research gaps
in the field; • and, understanding and promoting the use of the best
student-, program-, and school-level evaluation strategies.
QEBP practice group discussions at 2010 CSMH conference (Albuquerque, NM)
• Split into two groups (research and front line) to answer 3 questions:1) What are our most pressing concerns related to
Quality and Evidence-based Practice?
2) What is one specific action sep for our group (research or front line) this coming year?
3) What is one specific action step you would like to see from the “other” group this coming year?
Mark Sander, Psy.D., LP
Hennepin County/Minneapolis Public [email protected]
612-668-5489
School Mental Health: Local Evaluation and Research Lessons
Learned
School Mental Health Programs Across MN
Minneapolis Public Schools established in 2005 in five schools
Gradually expanded to 15 schools
County wide – 10 agencies, 14 school districts, over 70 schools
State wide – 21 grantees, serving 63 counties, about 200 school districts, and 550 schools
Collaboratively Building a Evaluation Model
What is “must have” data? o Mental Health Outcome (SDQ and CASII)o Educational Outcome (Suspension and Attendance)o Access and Engagement
Who informs it?o Stakeholder “interviews”o Going back and talking to stakeholderso Survey of Principal, Assistant Principals and Social
Workers (later years)
Data Informed Sustainability Planning
How do you do it?o We went low tech for the first 4 yearso Now we have developed on-line database
How do you fund it?o Used SS/HS funding to develop database and work with
statistician on analysis
How do you use it?o Developed preliminary evaluation modelo Referral and service trends o Impact on programming decisiono Impact on financial model and cash flow for agencies
Utilizing the Data to Inform and Sustain
Referral and Service Trendso Impact on programming decisiono Impact on financial model and cash flow for agencies
Case for Ancillary and Supportive Serviceso Teacher Consultation o Care Coordinationo Training for educational staff; classroom
presentations
Case for educational Benefit o Reduction in Suspensions o Reduction in Suspensions for students with 3 or moreo Reclaimed instructional time (not analyzed yet)
Referral Data SY06
ESMH Ref er ral SY06
31
23
37
56
1719
23
35
12 11
1
0
10
20
30
40
50
60
J ul 05 Aug 05 Sept 05 Oct 05 Nov 05 Dec 05 J an 06 Feb 06 Mar 06 Apr 06 May 06 J un 06
Ser ies1
MPS Student Demographics SY06-SY09
1017 Students 61% male and 39% female General Ed: 55% and 45% Special Ed (SPED 15.9%)Ethnicity
NATIVE AMERICAN – 8.3% (4.5%) AFRICAN AMERICAN – 48.9% (39.6%) ASIAN – 2.1% (9.0%) HISPANIC – 28.1% (17.1%) WHITE – 12.6% (29.8%) NATIVE HAWAIIAN PAC ISL – 0%
ELL – 31.2 % (23.2%)
District wide percentages are in parenthesis
Evaluation of MPS ESMHP
Access and sustained engagement in treatment 85 % of students seen once face to face; 70% within 10
days; 65% 1st time receiving services Average 17 visits per school year
Improved mental health functioning Parents and teachers report decreases in the emotional
and behavioral problems Improved school functioning
Decrease in school suspensions for students receiving mental health treatment
Principals and school social workers reported reduced office referrals and student suspensions
Impact on Suspensions
SY07 N=298 Seen 4x and had 1 suspension SY06 (n=82); only 27.5% had suspension in SY06
Impact on suspensions: SY06 to SY07 – N=82 50% reduced 1-6 32% stayed the same
Multiply Suspended SY06-07
SY06-SY07 Multiply Suspended (N=19) SPED=64% ELL=37% Male=75% African American=63%
2 suspensions or less=78.9%
52.6% to 1 or less
Moved to Zero=37%
Multiply Suspended SY06-SY07 (N=19)
0
1
2
3
4
5
6
7
8
9
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Individual Students
Nu
mb
er o
f S
usp
ensi
on
s
SY06
SY07
Change in Mean Suspensions: SY06-SY07
Impact on Suspensions: 156 Tx and 133 Comparison
0.92
0.54
0.83
1.80
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
Baseline After Treatment
Aver
Days S
usp
en
ded
Full Tx
Full Comp
MPS – SY07 Change Mean Suspension: Full/African American
Impact on Suspensions: Full: 156 Tx, 133 Com; AA: 83 Tx, 76 Comp
0.92
0.54
0.83
1.80
1.30
0.61
1.131.17
0.00
0.20
0.40
0.60
0.80
1.00
1.20
1.40
1.60
1.80
2.00
Baseline After Treatment
Full Tx
Full Comp
African Amer Tx
African Amer Comp
Change in Attendance: SY06 to SY07 (N=35, N=20)
Total Sample of Treatment kids = 159 40 (25%) had less than
90% Attendance in SY06 (5 kids missing data in SY07; sample 35 students)
Total Sample of 133 comparison 20 (15%) of them had
lower than 90% attendance in SY06
Treatment Comparison
N % N %
35 100 20 100
22 62.9% 7 35%
Psychosocial, Placement and Cost Outcomes in Maryland’s SMH Programs for Youth in
Special Education
Sharon H. Stephan, Ph.D.
University of Maryland School of Medicine
Center for School Mental Health
Acknowledgements• Maryland State Department of Education
• Dr. Carol Ann Heath• Dr. Jodi King
• Prince George’s County Public Schools• Dr. Pamela Downing-Hosten• Dr. Florence Foreman
• Baltimore City Public Schools• Ms. Kim Lewis
• Center for School Mental Health• Drs. Dana Cunningham, Eric Slade, Mark Weist, Nancy
Lever• Christianna Andrews, Kerri Chambers, Emily Sidway
Background• Nationally, there has been an upward trend in students
identified with emotional disturbances (ED) requiring intensive services to access educational standards.
• This trend is evident in Maryland with the Maryland State Department of Education listing ED as a high growth category increasing approximately 75% over the last ten years.
• In the five years prior to initiating the Prince George’s County SMH Initiative, the use of non-public placements in Maryland had increased by 20%, with some districts their use by over 30%.
Current Issues in Nonpublic Placements• Non-public placements cost local school districts over
40K per student
• Many districts do not have mental health services that are devoted to students identified as disabled
• There are multiple indicators that the use of non-public placements are racially disproportionate
• The outcomes associated with these placements are inconsistent
Rationale for Implementation
• Initiation of PGSMHI (2006)• Collaboration between MSDE, state
university, and local school system• Concern over high rate of referrals to
non-public schools• Concern over racial disproportionality in
placement in non-public schools• High costs of non-publics• Relative lack of school based services
Rationale for Implementation
• Initiation of BSMHI (2009)• Collaboration between MSDE, state
university, and local school system• Concern over high rate of referrals to
non-public schools• High costs of non-publics• Documented success of PGSMHI
Target PopulationTarget Population• Students in special education who are at Students in special education who are at
risk of entering non-public settings due to risk of entering non-public settings due to an increase in behavioral and/or emotional an increase in behavioral and/or emotional problemsproblems
• Students in non-public settings who are Students in non-public settings who are prepared to return to their home schoolprepared to return to their home school
Program Model
• Therapists
• Case managers
• Psychiatric consultation
• Family Support Partners (BSMHI)
Limited Local Capacity for Data Analysis
• The data on highly restrictive placements was not organized to answer questions and guide interventions
• Breaking this barrier:• Identified risk factors that contributed to restrictive
placements - Age, Grade, Compliance Issues, Suspensions, Disability, Gender
• Analyzed national, state, local and individual data sources
• Targeted efforts on suspected risk factors• Collected extensive data on clients
Breaking Down Barriers
• Financing• Total costs of Non-public and alternative programs are
hard to determine• Non-public placements involve financing from two
sources- the state and local level• Completed low and high estimates of cost savings for
both the state and local district• Moved thinking from “GRANT” thinking to “DIVERSION”
thinking.
Economic Evaluation
• Determined costs associated with PGSMHI based on 3 years of data and 77 students• Personnel• Trainings• Equipment and supplies
• Determined average cost of nonpublic placements• Education costs• Transportation costs
Economic Evaluation• Moved from dichotomous thinking ….stay or go, Emotionally
Impaired or not, etc.• to every day counts = continuous variable success
for each student
• Question changed to how many days was the student maintained in the public setting?
• Unit of analysis is the number of days prevented in non-public (not whether the student was placed)
• Cost determined by the number of days placement was delayed
or averted times cost differential per day
CostsAggregate Per student
per dayPer student per year
Total Nonpublic Costs
3,964,928 163 39,038
Total PGSMHI Costs
732,487 30 7,212
Total Savings
3,232,440 133 31,826
Note: Data is based on a program year of 240 days
Cost Savings
• $133 per student/day averted• For students receiving intense
services and are considered at imminent risk for non-public placement
• These savings are not expected for students who have emotional conditions but are not at risk for non-public placements
• These savings are not expected for “usual care” mental health services provided in a school
Limitations of Evaluation
• Did not assess indirect costs of adverse consequences for school staff and students of retaining students in public schools
• No comparison sample of students in nonpublic setting
• Assume that students would have been quickly placed in a nonpublic setting without intervention
Student ObservationsStudent Observations
0
5
10
15
20
25
30
35
Off Task TchrPrompt
Neg Resp Attn
Obs 1Obs2
All SymptomsAll Symptoms
0
0.5
1
1.5
2
2.5
3
3.5
Youth Parent Teacher
Intake Time 2
Youth Intake N = 59Parent Intake N = 47
BSMHI Placement Outcomes
• 93% of the students enrolled in the BSMHI were maintained in their current placement.
• The number of students referred to non public placement at primary school decreased 50% from the 2008-2009 school year to the 2009-2010 school year.
BSMHI Cost Outcomes
• Students were diverted a total of 24.3 years from more restrictive placements. With an estimated additional cost of $31,915* per student per year for a nonpublic placement, approximately $775,534 was saved the first year of this project.
$43,000 Nonpublic Placement for 1 Year- $11,085 Public School Cost for 1 Year $31,915 Add’l Cost for Nonpublic Placementx 24.3 Placement Years Deferred$775,534.50 Cost Savings
Center for School Mental Health
• Objective 1.2 Continue and strengthen ongoing CSMH efforts to conduct comprehensive analyses of established, data driven and impactful local and state SMH programs, with data-driven findings and case examples shared in yearly summary reports (12/11, 12/12, 12/13).
• Objective 1.3 Enhance CSMH efforts to assist state and local program leaders to identify and measure targeted student and school-level outcomes and to incorporate data-based decision making into practice via a quarterly SMH outcomes/data management newsletter, annual webinar (5/11, 5/12, 5/13), and a regularly maintained compendium of online resources.
• Objective 2.7 Narrow the research-practice divide and improve communication and knowledge sharing, and facilitate partnerships between researchers and other SMH stakeholders via the development and convening of a SMH Research-Practice Network and encouraging increased participation of researchers in the annual conference and National COP.
Call for Papers!• Special Issue on School-Based Mental Health Practice, Data-
Driven School Mental Health Practice: (How) Is It Possible?
Advances in School Mental Health Promotion
This issue will seek to better understand how data-driven and research-based educational interventions are being manifested in school mental health delivery.
For more information, please contact the Special Issue Guest Editor Michael S. Kelly at [email protected].
The deadline for manuscript submission is June 15, 2011. Submissions should be sent to: [email protected]