Embed Size (px)
Citation preview
DAN KOONCE
TECHNICAL ASSISTANCE COORDINATOR
IL PBIS NETWORK
BEN DITKOWSKY
DIRECTOR OF ASSESSMENT, INTERVENTION AND RESEARCH
LINCOLNWOOD SCHOOL DISTRICT 74
Designing a School Recognition Process: Triangulating the Data Using Evaluation Tools
Session Goals
� Provide a process for examining discipline data using tools that lead to effective method of recognizing schools.
� Provide data-based information to guide schools in improving their implementation efforts.
� Provide the opportunity for skill development to critically review discipline data using case studies
National Academic Recognition
� Coleman Report (1966)
� Interrelationship between school characteristics and student achievement
� Three decades of research yielded mixed results
� Blue Ribbon Schools Program (1982) by Secretary of Education Terrell Bell
Local Academic Recognition
� Illinois State Board of Education (ISBE) recognizes high poverty schools
� Student performance on the state achievement tests indicates closing of achievement gap
� Process is complex and controversial since each state adopts own methodology for determining accountability
Stimulus Response
� Reinforcement principles
� Severe and disruptive behavior is a barrier for maintaining and sustaining social competence and academic gains.
� > 4,000 schools are implementing PBS.
B. F. SkinnerI. Pavlov
Universal Recognition for Behavior
� Recent review of the literature indicates no empirical study recognizing public schools for improving behavioral outcomes of students school-wide
� (using keywords: behavior, schools, recognition)
� Implications: Schools are being held accountable for variables impacting the academic achievement of students (suspensions, out of district placements)
� Several states (i.e., Florida, Illinois, Maryland, North Carolina) adopted recognition process for school implementing SWPBS
Components of Recognition (Illinois)
� Phases of Implementation
� Team Implementation Checklist
� Benchmarks of Quality / School Evaluation Tool
� Self-Assessment Survey
� IL PBIS School Profile Tool
Phases of Implementation (POI)
� 66-item instrument (2010) version broken down into three phases at each tier (U/S/T)
� Team self-assessment tracking schools’ progress through the PBIS implementation process at all three tiers (U/S/T)
� Bi-annually (optional update in March each year)
� All items checked across each phase and tier
Progress Over Time: Phases of Implementation
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
YEAR 1 YEAR 2 YEAR 3
Time
Perc
ent Im
ple
mente
d
Universal (Phase 1)
(Phase 2)
(Phase 3)
Team Implementation Checklist (TIC)
� 17-item instrument
� Team self-assessment of progress toward implementation and implementation activities for next quarter
� Completed quarterly
� Criterion is 80% or higher at Universal or Tier 1
School-wide Evaluation Tool (SET)
� 28-item instrument involving review of school documents, examination of physical spaces, interviews with students and staff
� External evaluation of how well school personnel are implementing practices and systems associated with PBIS
� Completed annually
� Criterion is summary score of 80% and “expectations taught”subscale score of 80%
Schoolwide Evaluation Tool
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Year 1 Year 2 Year 3 Year 4
Percentage Implemented
Teaching
Total
Legend
Self – Assessment Survey
Schoolwide
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
In
Place
Partial Not
Non-classroom
In Place Partial Not
Classroom
In Place Partial Not
Individual Student
Systems
In Place Partial Not
� Formally known as the EBS Survey
� Instrument completed by the entire faculty and staff in the school
� Assesses staff and faculty perceived level of PBIS implementation in four content areas
� Completed annually
IL School Profile Tool
� Provides a framework for school teams to document the interventions (across all three level of the behavior continuum) completed throughout the
school year.
� Share progress and success to school staff, district administration, school board, families, and
community.
� Opportunity to share progress and success with the
IL PBIS Network
� Opportunity to receive technical assistance from IL PBIS Network (technical assistance coordinator)
Tier 1/Universal Intervention Example
Type of PBIS Tier
1/Universal InterventionDates of
Implementation
Brief Description of
Intervention
% of Staff
Involved
Impact of Intervention
data: Include pre-intervention data (data that indicated need)
and post-intervention data (data that indicated effectiveness)
Population (pick one)
���� School-wide (all kids)
���� Selected Population (i.e.
grade level, etc.)
Start Date:
___08/28/08____End Date:
___08/30/08____OR
���� Ongoing
Description:
Teach behavioral
expectations, ending with
a school-wide
celebration.
Length of intervention (daily, # of
hours/minutes): 2 days
���� 90 – 100 %
���� 80 – 90%
���� 70 – 80%
���� 60 – 70%
���� < 50%
Pre-Intervention data (please
check data source and provide
specific information)
���� # ODRs: _60 ODRs Sept.
2007________
���� # Tardies/days absent:
_____________
���� Grades (academic indicator):
_________________________
���� Other:
___________________________
Setting (pick one)
���� All settings
���� Specific setting
Post-Intervention data (please
use same data source identified
above)
���� # ODRs: _12 ODRs Sept.
2008________
���� # Tardies/days absent:
_____________
���� Grades (academic indicator):
_________________________
���� Other:
___________________________
Tier 1/Universal Name of School: Lincoln ElementaryIntervention Name: PBIS Kick-off School District Name and #: Snowy Trail 57
How Much Universal Data is Enough?
� When we collect data, how are we using it?
� Do you see any redundancy in collection of universal data?
� What are the most critical pieces you would use for recognition or decision-making?
Streamline the process
“Weed the garden“ (Doug Reeves )
� “… entities must not be multiplied beyond what is necessary”William of Occam (1284-1347)
� A problem should be stated in its basic and simplest terms.
� the simplest theory that fits the facts of a problem is the one that should be selected
What else does the process need?
� We have a lot of information, but…
� Do we have the right information to document fidelity of implementation of PBIS?
Does the recognition process sufficiently link process and product?
3-Tiered System of Support
Necessary Conversations (Teams)
CICO
SAIG
Group w. individualfeature
Complex
FBA/BIP
Problem Solving
Team
Tertiary
Systems Team
Brief
FBA/
BIP
Brief
FBA/BIP
WRAP
Secondary
Systems Team
Plans SW & Class-wide supports
Uses Process data; determines overall
intervention effectiveness
Standing team; uses FBA/BIP process for one youth at a time
Uses Process data; determines overall
intervention effectiveness
Universal
Team
Universal
Support
Case Study - A Non Example
� Fidelity of universal implementation declining
� Implementing a version of a model being trained
� Administrator has individual student orientation for school-wide support
� School-wide data are not triangulated to implementation across all levels
� Administrator has not attended any trainings
� Administrator not active member of district leadership team
Phases of Implementation - Universal(Middle School A)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
YEAR 1 YEAR 2 YEAR 3
Time
Percent Implemented
Universal (Phase 1)
(Phase 2)
(Phase 3)
Phases of Implementation - Secondary (Middle School A)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
YEAR 1 YEAR 2 YEAR 3
Time
Percent Implemented
Secondary (Phase 1)
(Phase 2)
(Phase 3)
Phases of Implementation – Tertiary(Middle School A)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
YEAR 1 YEAR 2 YEAR 3
Percent Im
plemented
Time
Tertiary (Phase 1)
(Phase 2)
(Phase 3)
School-wide Evaluation Tool(Middle School A)
90%
80%
70%
50%
91%97%
81%77%
0%
20%
40%
60%
80%
100%
Year 1 Year 2 Year 3 Year 4
Teaching Total
Team Implementation Checklist(Middle School A)
88%
94%
94%
0% 20% 40% 60% 80% 100%
2008-09
2007-08
2006-07
Tim
e
Percent Fully Implemented
Middle School ATriangle Data (Major ODRs)
Mean Percentage of Students by Major ODRs 2006-07 (Middle School A)
Non-Example
Mean Percentage of Students by Major ODRs 2007-08 (Middle School A)
Mean Percentage of Students by Major ODRs 2008-09 (Middle School A)
Non-Example
Out of School Suspensions (Middle School A)
124 124132 131
112
148
0
40
80
120
160
Year 1 Year 2 Year 3
One occasion Two or more occasions
(N = 869) (N = 922) (N = 927)
Access to General Education(Middle School A)
46.3 40.944.1
49.2 49.2 49.1
0
20
40
60
80
100
Year 1 Year 2 Year 3
Percent
School A State Avg.
Downward trend in the percentage of students with IEPs participating
in the general education environment for 80% or more of the day.
Case Study - An Example of Progress
� Fidelity of universal implementation reaching sustainability
� Implementing a version of the model trained
� Administrator has good understanding of full continuum of supports
� School-wide data are triangulated across all levels
� Administrator active participant in training with team members
Phases of Implementation - Universal(Middle School B)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
YEAR 1 YEAR 2
Time
Percent Implemented
Universal (Phase 1)
(Phase 2)
(Phase 3)
Phases of Implementation - Secondary (Middle School B)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
YEAR 1 YEAR 2
Time
Percent Implemented
Secondary (Phase 1)
(Phase 2)
(Phase 3)
Phases of Implementation – Tertiary(Middle School B)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
YEAR 1 YEAR 2
Time
Percent Implemented
Tertiary (Phase 1)
(Phase 2)
(Phase 3)
School-wide Evaluation Tool(Middle School B)
90% 90%100%93% 98% 100%
0%
20%
40%
60%
80%
100%
Year 1 Year 2 Year 3
Teaching Total
Team Implementation Checklist(Middle School B)
88%
76%
71%
0% 20% 40% 60% 80% 100%
2008-09
2007-08
2006-07
Tim
e
Percent Fully Implemented
Out of School Suspensions (Middle School B)
(N = 663) (N = 650) (N = 651)
Access to General Education(Middle School B)
34.8 29.3 37.0
49.2 49.2 49.1
0
20
40
60
80
100
Year 1 Year 2 Year 3
Percent
School B State Avg.
Upward trend in the percentage of students with IEPs participating
in the general education environment for 80% or more of the day.
Triangle Data (Major ODRs)(Middle School B)
Decreases in OSS by Student Ethnicity
Addressing Disproportionality
� What steps do the take?
� Conversations for awareness.
� Professional development activities.
� Developing a goal for progress.
Check list for Individual
Student Systems (CISS)
Check list for Individual
Student Systems (CISS)Individual Student Systems
Evaluation Tool (ISSET)
Individual Student Systems
Evaluation Tool (ISSET)
� Self-assessment of the implementation status of secondary and
tertiary behavior support systems
� Based on factors drawn
from ISSET (three parts)
� Criterion 80%
� Measures the implementation status of programs for
individual students
� Organized in three parts:
� Foundations, Targeted or Secondary, and Intensive
� Criterion 80%
Consider Assessing Secondary and Tertiary Systems
CISS & ISSET (Middle School A)
Non-Example
CISS & ISSET (Middle School B)
Progressing
Recommendations
� Reflect on the importance and utility of external evaluation
� Checklist for Individual Student Systems
� Individual Student Systems Evaluation Tool
� Take time to examine the relationship between process and product (i.e., Student outcome data)
� What is happening with the total number of referrals?
� Examine triangle data
� What trends emerge?
� Is it reflective of implementation process that move from
exploration and adoption to program installation of secondary and or tertiary supports?
Recommendations
� Include out of school suspension data
� Analyze single and multiple incidents and looking for decreasing trend
� Rates of disproportionality (calculated against state average percentage) should be decreasing
� Consider examining LRE (least restrictive environment) data
� What is happening with students who have been identified for
intervention with regards to their referrals?
What do you earn for recognition?
� Recognition by state (visitation by government officials)
� Focused technical assistance.
� Alternative training options.
� Consideration of being identified as a model site.
� Consideration for grant supported demonstration project.
Do we have the resources to improve?
Questions
