E10 Koonce and Ditkowsky - · PDF filedan koonce technical assistance coordinator il pbis...

DAN KOONCE

TECHNICAL ASSISTANCE COORDINATOR

IL PBIS NETWORK

BEN DITKOWSKY

DIRECTOR OF ASSESSMENT, INTERVENTION AND RESEARCH

LINCOLNWOOD SCHOOL DISTRICT 74

Designing a School Recognition Process: Triangulating the Data Using Evaluation Tools

Session Goals

� Provide a process for examining discipline data using tools that lead to effective method of recognizing schools.

� Provide data-based information to guide schools in improving their implementation efforts.

� Provide the opportunity for skill development to critically review discipline data using case studies

National Academic Recognition

� Coleman Report (1966)

� Interrelationship between school characteristics and student achievement

� Three decades of research yielded mixed results

� Blue Ribbon Schools Program (1982) by Secretary of Education Terrell Bell

Local Academic Recognition

� Illinois State Board of Education (ISBE) recognizes high poverty schools

� Student performance on the state achievement tests indicates closing of achievement gap

� Process is complex and controversial since each state adopts own methodology for determining accountability

Stimulus Response

� Reinforcement principles

� Severe and disruptive behavior is a barrier for maintaining and sustaining social competence and academic gains.

� > 4,000 schools are implementing PBS.

B. F. SkinnerI. Pavlov

Universal Recognition for Behavior

� Recent review of the literature indicates no empirical study recognizing public schools for improving behavioral outcomes of students school-wide

� (using keywords: behavior, schools, recognition)

� Implications: Schools are being held accountable for variables impacting the academic achievement of students (suspensions, out of district placements)

� Several states (i.e., Florida, Illinois, Maryland, North Carolina) adopted recognition process for school implementing SWPBS

Components of Recognition (Illinois)

� Phases of Implementation

� Team Implementation Checklist

� Benchmarks of Quality / School Evaluation Tool

� Self-Assessment Survey

� IL PBIS School Profile Tool

Phases of Implementation (POI)

� 66-item instrument (2010) version broken down into three phases at each tier (U/S/T)

� Team self-assessment tracking schools’ progress through the PBIS implementation process at all three tiers (U/S/T)

� Bi-annually (optional update in March each year)

� All items checked across each phase and tier

Progress Over Time: Phases of Implementation

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

YEAR 1 YEAR 2 YEAR 3

Time

Perc

ent Im

ple

mente

d

Universal (Phase 1)

(Phase 2)

(Phase 3)

Team Implementation Checklist (TIC)

� 17-item instrument

� Team self-assessment of progress toward implementation and implementation activities for next quarter

� Completed quarterly

� Criterion is 80% or higher at Universal or Tier 1

School-wide Evaluation Tool (SET)

� 28-item instrument involving review of school documents, examination of physical spaces, interviews with students and staff

� External evaluation of how well school personnel are implementing practices and systems associated with PBIS

� Completed annually

� Criterion is summary score of 80% and “expectations taught”subscale score of 80%

Schoolwide Evaluation Tool

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Year 1 Year 2 Year 3 Year 4

Percentage Implemented

Teaching

Total

Legend

Self – Assessment Survey

Schoolwide

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

In

Place

Partial Not

Non-classroom

In Place Partial Not

Classroom

In Place Partial Not

Individual Student

Systems

In Place Partial Not

� Formally known as the EBS Survey

� Instrument completed by the entire faculty and staff in the school

� Assesses staff and faculty perceived level of PBIS implementation in four content areas

� Completed annually

IL School Profile Tool

� Provides a framework for school teams to document the interventions (across all three level of the behavior continuum) completed throughout the

school year.

� Share progress and success to school staff, district administration, school board, families, and

community.

� Opportunity to share progress and success with the

IL PBIS Network

� Opportunity to receive technical assistance from IL PBIS Network (technical assistance coordinator)

Tier 1/Universal Intervention Example

Type of PBIS Tier

1/Universal InterventionDates of

Implementation

Brief Description of

Intervention

% of Staff

Involved

Impact of Intervention

data: Include pre-intervention data (data that indicated need)

and post-intervention data (data that indicated effectiveness)

Population (pick one)

���� School-wide (all kids)

���� Selected Population (i.e.

grade level, etc.)

Start Date:

___08/28/08____End Date:

___08/30/08____OR

���� Ongoing

Description:

Teach behavioral

expectations, ending with

a school-wide

celebration.

Length of intervention (daily, # of

hours/minutes): 2 days

���� 90 – 100 %

���� 80 – 90%

���� 70 – 80%

���� 60 – 70%

���� < 50%

Pre-Intervention data (please

check data source and provide

specific information)

���� # ODRs: _60 ODRs Sept.

2007________

���� # Tardies/days absent:

_____________

���� Grades (academic indicator):

_________________________

���� Other:

___________________________

Setting (pick one)

���� All settings

���� Specific setting

Post-Intervention data (please

use same data source identified

above)

���� # ODRs: _12 ODRs Sept.

2008________

���� # Tardies/days absent:

_____________

���� Grades (academic indicator):

_________________________

���� Other:

___________________________

Tier 1/Universal Name of School: Lincoln ElementaryIntervention Name: PBIS Kick-off School District Name and #: Snowy Trail 57

How Much Universal Data is Enough?

� When we collect data, how are we using it?

� Do you see any redundancy in collection of universal data?

� What are the most critical pieces you would use for recognition or decision-making?

Streamline the process

“Weed the garden“ (Doug Reeves )

� “… entities must not be multiplied beyond what is necessary”William of Occam (1284-1347)

� A problem should be stated in its basic and simplest terms.

� the simplest theory that fits the facts of a problem is the one that should be selected

What else does the process need?

� We have a lot of information, but…

� Do we have the right information to document fidelity of implementation of PBIS?

Does the recognition process sufficiently link process and product?

3-Tiered System of Support

Necessary Conversations (Teams)

CICO

SAIG

Group w. individualfeature

Complex

FBA/BIP

Problem Solving

Team

Tertiary

Systems Team

Brief

FBA/

BIP

Brief

FBA/BIP

WRAP

Secondary

Systems Team

Plans SW & Class-wide supports

Uses Process data; determines overall

intervention effectiveness

Standing team; uses FBA/BIP process for one youth at a time

Uses Process data; determines overall

intervention effectiveness

Universal

Team

Universal

Support

Case Study - A Non Example

� Fidelity of universal implementation declining

� Implementing a version of a model being trained

� Administrator has individual student orientation for school-wide support

� School-wide data are not triangulated to implementation across all levels

� Administrator has not attended any trainings

� Administrator not active member of district leadership team

Phases of Implementation - Universal(Middle School A)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

YEAR 1 YEAR 2 YEAR 3

Time

Percent Implemented

Universal (Phase 1)

(Phase 2)

(Phase 3)

Phases of Implementation - Secondary (Middle School A)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

YEAR 1 YEAR 2 YEAR 3

Time

Percent Implemented

Secondary (Phase 1)

(Phase 2)

(Phase 3)

Phases of Implementation – Tertiary(Middle School A)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

YEAR 1 YEAR 2 YEAR 3

Percent Im

plemented

Time

Tertiary (Phase 1)

(Phase 2)

(Phase 3)

School-wide Evaluation Tool(Middle School A)

90%

80%

70%

50%

91%97%

81%77%

0%

20%

40%

60%

80%

100%

Year 1 Year 2 Year 3 Year 4

Teaching Total

Team Implementation Checklist(Middle School A)

88%

94%

94%

0% 20% 40% 60% 80% 100%

2008-09

2007-08

2006-07

Tim

e

Percent Fully Implemented

Middle School ATriangle Data (Major ODRs)

Mean Percentage of Students by Major ODRs 2006-07 (Middle School A)

Non-Example

Mean Percentage of Students by Major ODRs 2007-08 (Middle School A)

Mean Percentage of Students by Major ODRs 2008-09 (Middle School A)

Non-Example

Out of School Suspensions (Middle School A)

124 124132 131

112

148

0

40

80

120

160

Year 1 Year 2 Year 3

One occasion Two or more occasions

(N = 869) (N = 922) (N = 927)

Access to General Education(Middle School A)

46.3 40.944.1

49.2 49.2 49.1

0

20

40

60

80

100

Year 1 Year 2 Year 3

Percent

School A State Avg.

Downward trend in the percentage of students with IEPs participating

in the general education environment for 80% or more of the day.

Case Study - An Example of Progress

� Fidelity of universal implementation reaching sustainability

� Implementing a version of the model trained

� Administrator has good understanding of full continuum of supports

� School-wide data are triangulated across all levels

� Administrator active participant in training with team members

Phases of Implementation - Universal(Middle School B)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

YEAR 1 YEAR 2

Time

Percent Implemented

Universal (Phase 1)

(Phase 2)

(Phase 3)

Phases of Implementation - Secondary (Middle School B)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

YEAR 1 YEAR 2

Time

Percent Implemented

Secondary (Phase 1)

(Phase 2)

(Phase 3)

Phases of Implementation – Tertiary(Middle School B)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

YEAR 1 YEAR 2

Time

Percent Implemented

Tertiary (Phase 1)

(Phase 2)

(Phase 3)

School-wide Evaluation Tool(Middle School B)

90% 90%100%93% 98% 100%

0%

20%

40%

60%

80%

100%

Year 1 Year 2 Year 3

Teaching Total

Team Implementation Checklist(Middle School B)

88%

76%

71%

0% 20% 40% 60% 80% 100%

2008-09

2007-08

2006-07

Tim

e

Percent Fully Implemented

Out of School Suspensions (Middle School B)

(N = 663) (N = 650) (N = 651)

Access to General Education(Middle School B)

34.8 29.3 37.0

49.2 49.2 49.1

0

20

40

60

80

100

Year 1 Year 2 Year 3

Percent

School B State Avg.

Upward trend in the percentage of students with IEPs participating

in the general education environment for 80% or more of the day.

Triangle Data (Major ODRs)(Middle School B)

Decreases in OSS by Student Ethnicity

Addressing Disproportionality

� What steps do the take?

� Conversations for awareness.

� Professional development activities.

� Developing a goal for progress.

Check list for Individual

Student Systems (CISS)

Check list for Individual

Student Systems (CISS)Individual Student Systems

Evaluation Tool (ISSET)

Individual Student Systems

Evaluation Tool (ISSET)

� Self-assessment of the implementation status of secondary and

tertiary behavior support systems

� Based on factors drawn

from ISSET (three parts)

� Criterion 80%

� Measures the implementation status of programs for

individual students

� Organized in three parts:

� Foundations, Targeted or Secondary, and Intensive

� Criterion 80%

Consider Assessing Secondary and Tertiary Systems

CISS & ISSET (Middle School A)

Non-Example

CISS & ISSET (Middle School B)

Progressing

Recommendations

� Reflect on the importance and utility of external evaluation

� Checklist for Individual Student Systems

� Individual Student Systems Evaluation Tool

� Take time to examine the relationship between process and product (i.e., Student outcome data)

� What is happening with the total number of referrals?

� Examine triangle data

� What trends emerge?

� Is it reflective of implementation process that move from

exploration and adoption to program installation of secondary and or tertiary supports?

Recommendations

� Include out of school suspension data

� Analyze single and multiple incidents and looking for decreasing trend

� Rates of disproportionality (calculated against state average percentage) should be decreasing

� Consider examining LRE (least restrictive environment) data

� What is happening with students who have been identified for

intervention with regards to their referrals?

What do you earn for recognition?

� Recognition by state (visitation by government officials)

� Focused technical assistance.

� Alternative training options.

� Consideration of being identified as a model site.

� Consideration for grant supported demonstration project.

Do we have the resources to improve?

Questions