I Am SAM: The Self- Assessment of Multi-Tiered Systems of Support€¦ · •Kevin Stockslager...

This product was developed by Florida’s Positive Behavior Support Project through the University of South Florida, Louis de la

Parte Florida Mental Health Institute funded by the State of Florida, Department of

Education, Bureau of Exceptional Education and Student Services, through federal assistance under the Individuals with

Disabilities Education Act (IDEA), Part B.

I Am SAM: The Self-Assessment of Multi-Tiered

Systems of Support

Karen Elfner Childs, M.A.Natalie Romer, Ph.D.

Jose Castillo, Ph.D.Kevin Stockslager, Ph.D.Amber Brundage, Ph.D.

I Am SAM: The Self-Assessment of

Multi-Tiered Systems of Support

Welcome

and

Introductions

No, not SAM I AM

Advance Organizer

• Our Vision of MTSS

• Self-Assessment of MTSS (SAM)– SAM

Development Procedures

– Components of the SAM

– SAM Psychometric Properties

• SAM and PBIS

• Questions and Comments

OUR VISION OF MTSS

What is MTSS?

• MTSS=Multi-Tiered Systems of Support– Evidence-based instruction delivered to students

in varying intensities (multiple tiers) based on student need

– Utilization of data-based problem-solving to integrate academic and social-emotional instruction and intervention

– Use of data to make educational decisions

Broader than PBIS

• Both social-emotional &academic throughout

• The collaborative vision of the PS/RtI and the PBIS:MTSS Projects is to:

– Enhance the capacity of all Florida school districts to successfully implement and sustain a multi-tiered system of student supports with fidelity in every school

Why This Instrument?

• Our Inter-Project vision: to enhance capacity of districts to support MTSS with fidelity in schools

• Current tools not adequate for assessing all components of MTSS

• Desire for an instrument to guide action planning towards improved implementation

SAM DEVELOPMENT

SAM Development Steps

1. Literature review2. Construct/theory development3. Item generation4. Expert review panel5. Cognitive interviewing6. Initial pilot (Fall 2013)7. National pilot (Spring 2015)

• “Gold standard” survey development procedures recommended by DeVellis (2012)

Initial Item Development

• Review of SAPSI and additional school-level measures of RtI/MTSS/PBIS

• Review of literature on RtI/MTSS/PBIS implementation, systems change, educational reform initiatives

• Feedback from Inter-Project Leadership Team and Project staff members

• Development of 3-point scoring rubric

Expert Review Panel

• 11 district-, state-, and national-level experts on RtI/MTSS and PBS implementation

• Feedback on item relevancy and clarity/conciseness

• 80% criterion used to identify quality items

• Items not meeting criteria revised by the evaluation team

e

Cognitive Interviews

• 6 cognitive interviews were conducted

• Provided feedback on difficult terms or jargon

• Items identified as problematic were revised by the evaluation team

• Common feedback from interviewees included:– Some items were too wordy

– Defining terms like “staff,” “stakeholders,” “implementation fidelity,” and “parent involvement”

– One person cannot have all the information needed to rate every item

Content Domains

• Leadership

• Building the Capacity/Infrastructure for Implementation

• Communication and Collaboration

• Data-Based Problem Solving

• Three-Tiered Instructional/Intervention Model

• Data-Evaluation

Leadership

• Administrators and school leadership teams are critical to implementing MTSS in schools

• Critical roles for leadership teams:– Continuous professional development on implementing MTSS– Strategically plan for MTSS implementation– Model a problem-solving process for school improvement

• Critical roles for administrators:– Communicating a mission and vision to school staff– Providing resources for planning as well as implementing

instruction and intervention– Ensuring staff have data necessary to engage in problem-

solving.

Building Capacity/Infrastructure for MTSS Implementation

• Critical components:– Ongoing professional development and coaching– Emphasis on data-based problem solving– Implementing a multi-tiered instruction and

intervention model– Scheduling that allows staff to engage in planning and

implementing instruction and intervention– Processes and procedures for engaging in data- based

problem-solving

Communication and Collaboration

• The following increase the likelihood that practices will be implemented and sustained – Consensus among key stakeholders

– Feedback to implementers to support continuous improvement

– Involving stakeholders in planning

– Infrastructure to communicate and collaborate with families and community partners

Data-Based Problem Solving

• Data-based problem solving at a school-, classroom-, and student-level is a critical to MTSS implementation

• Includes problem solving for student outcomes across content areas, grade levels and tiers, and to address barriers to school wide implementation of MTSS

• Four-step problem solving evaluated in this instrument includes, – 1) defining the goals and objectives to be attained– 2) identifying possible reasons why the desired goals are not being

attained– 3) developing and implementing evidence-based strategies to

attain goals– 4) evaluating the effectiveness of the plan

Three-Tiered Instructional/Intervention Model

• The three-tiered instructional/intervention model is another critical component of MTSS implementation

• In a typical model, the three tiers include:– Tier 1: instruction that all students receive– Tier 2: additional, supplemental instruction or intervention

provided to students not meeting benchmarks– Tier 3: intensive, small-group or individual interventions

provided to students exhibiting significant barriers to learning the skills required for school success

• Within an MTSS model, it is critical to consider both academic and social-emotional/behavioral instruction and interventions when examining this domain

Data/Evaluation

• Importance of data-based problem solving in MTSS establishes a need for data and evaluation system

• School staff need to understand and have access to data sources that address four purposes of assessment (i.e., screening, diagnostic assessment, formative evaluation, and summative evaluation)

• Efficient & useful data system, procedures and decision making protocols for assessing and using data

• Data on the fidelity of MTSS implementation for leadership to examine status and make systemic changes for improving implementation

Sample Items (Scoring Rubric 0-1-2-3)

Preliminary Pilot - Fall 2013

• 2 states, 7 districts, 155 schools• Reviewed descriptive statistics and qualitative

feedback• Revised survey (e.g., item clarity, errors, etc.)

and added 5 items based on feedback– Scheduling (2)– Disaggregating data across groups– Allocation of resources– Monitoring of data sources

2015 National Pilot: Sample

• 8 states, 15 districts, 436 schools

• School type– 269 elementary

– 75 middle

– 69 high

– 23 other (Alternative, Combined, etc.)

• Recruitment

SAM Domain Averages

Confirmatory Factor Analysis

• Strong Factor Structure (6-factor model)– 6-factor conceptual CFA based on 6 SAM domains– Chi-square test of fit = 1734.06 (p <.0001)– Root Mean Square Error of Approximation (RMSEA) = .059

(criteria < .08)– CFI = .96 (criteria > .95)

• High Internal Consistency Reliability .79-.91

SAM and Behavior Outcomes

• SAM (overall score) significantly correlated with BOQ: r(188)= .31, p<.001

• SAM (overall score) significantly correlated with Out of School Suspension (OSS) days r(243)= -.14, p<.05

SAM Feasibility Items

• Average time to complete: 1-1.5 hours

• Will SAM help your school implement?– Avg. score 3.55 (1: Not at all-5: Very)

• Will your school use the SAM again?– Avg. score 3.57 (1: Not at all-5: Very)

1 - Not at all Very -5

23

4

SAM – SWPBIS Fidelity Tools Crosswalk• All BoQ & BAT items were related with SAM items. • 7 SAM items were not on BoQ or BAT items.

– Infrastructure for family involvement (as opposed to practices), academic practices, procedures for databased decision making, and coaching.

• PBIS measures focused on components of a data system, but not problem solving itself

• Coaching and the use of data to problem solve and make decisions were not emphasized in the PBIS tools despite being considered critical feature of implementation.

More on Use of the SAM with SWPBIS

• Blended District or School Leadership Teams

• PBIS Coaches’ expertise in systems implementation

• Build connections through common features

• Other?

District Example: Broward County

Diversity, Prevention & Intervention

CLIMATE Grant

Data Collected Focus of ToolSAM MTSS with FidelityBoQ MTSS with FidelityCHAMPs 7 UP classroom management skillsHERO Usage Reinforcement of positive

behaviorsBASIS Documentation of strategies &

ODRsODRs Misbehavior frequencyInternal Suspensions out of seat timeExternal Suspensions out of seat timeReferrals to PROMISE possible arrest misbehaviorsAnnual Climate Survey Climate, safety

▪ Self-Assessment of Multi-Tiered System of Supports (SAM)

▪ SAM School-level Inventory

▪ Critical Component Checklist (CCC) Implementation Fidelity

▪ Observation Checklist Implementation of Fidelity

▪ Beliefs and Perceptions Skills Survey

▪ Needs Assessment

▪ Pre/Post Test Surveys

Quality Assurance & Evaluation Measure

31

32

33

Potential Data Sources

• Self-Assessment of MTSS (SAM)

• Tiers I/II and Tier III CCCs

• Tier I/II Observation Checklist, Problem-Solving Team Meeting Checklist

• PBIS Implementation data– Benchmarks of Quality (BOQ)

– PBS Implementation Checklist (PIC)

– Benchmarks for Advanced Tiers (BAT)

• Walkthrough data, lesson plans

• Tier 2 and 3 intervention documentation worksheets

• Others?

Data Dissemination

• Identify key stakeholders (e.g., District Leadership Teams, School Leadership Teams, instructional staff)

• Share data quickly and frequently• Use Questions to guide discussions• Graph data to more easily identify trends• Compare different types of MTSS fidelity data if

possible (self-report, permanent product reviews, observations)

• Inform professional development planning

Interested in Using the SAM?

• User’s Manual in Development

• Article on Validation in Development

• Email us!

Connect with Us!

Florida’s Problem-Solving/Response to Intervention Project

– http://www.floridarti.usf.edu/

– Email: rti@usf.edu

– Facebook: flpsrti

– Twitter: @flpsrti

Florida Positive Behavioral Interventions & Support Project

– http://flpbs.fmhi.usf.edu/

– Email: flpbs@fmhi.usf.edu

– Facebook: www.facebook.com/flpbs

– FLPBIS on Twitter: www.twitter.com@flpbs

– YouTube: https://www.youtube.com/user/flpbs137

Contact Information

• Karen Childs– kchilds2@usf.edu

• Natalie Romer– romer@usf.edu

• Kevin Stockslager– kstocksl@usf.edu

• Jose Castillo– jmcastil@usf.edu

• Amber Brundage– abrundage@mail.usf.edu

Including strands specifically designed for our Mental Health,

Juvenile Justice, and Family/Community Partners!

PBIS: Systems for Enhancing

Climate & CultureDonald Stephens Convention CenterRosemont, IL

Donald E. Stephens Convention Center

Rosemont, Illinois

This two-day forum for school, state, district and regional Leadership Teams and other professionals has been designed to increase the effectiveness of PBIS implementation.

Sessions are organized by strands that support initial through advanced implementation in elementary, middle, and high schools as well as juvenile justice facilities:

PBIS FoundationsClassroom Applications

Tier 2 Systems & Practices Tier 3 Systems & Practices

Aligning SystemsJuvenile Justice

Mental Health IntegrationEquity

Applied EvaluationSpecial Topics

Visit the Upcoming Events page at www.pbis.org for more

information

2016 National PBIS Leadership Forum