Evaluation Tools, On-Line Systems, and Data-Based Decision Making

Version 3.0, Rev. 12.2.11 This is a presentation of the Illinois PBIS Network. All rights reserved.

Adapted from the Illinois PBIS Network

AgendaEvaluation Tools:

Team Implementation Checklist (TIC) Benchmarks of Quality (BoQ) Self Assessment Survey (SAS)

PBIS Assessment @ www.pbisapps.org

Outcomes of Session

Introduction of Tier 1 Evaluation Tools Understand purpose of Tools Understand process of Tool completion Practice using the Tools Know where to complete surveys and

generate reports on www.pbisapps.org

Tier 1 Evaluation Tools…

Team Implementation Checklist (TIC) Benchmarks of Quality (BoQ) Self-Assessment Survey (SAS)

Effort data (Are we working the plan?) Team Implementation Checklist (TIC) Self-Assessment Survey (SAS)

Outcome data (Is it having an effect?) “Big 5” Graphs Triangle %

Fidelity data (Are we following the plan?) School Evaluation Tool (SET) Benchmarks of Quality (BoQ)

Purpose of PBIS Evaluation Tools

Getting Started• Self Assessment Survey (SAS) • Team Implementation Checklist (TIC)

Getting to Implementation Fidelity• Team Implementation Checklist (TIC)• Benchmarks of Quality (BoQ)

Sustaining Improvement• SAS, BoQ, SET

PBIS Assessment Account

Team Implementation Checklist 3.1 Benchmarks of Quality Self Assessment Survey

Your PBIS Assessment AccountYour PBIS Technical Assistance Coordinator orExternal Coach will:

Give each school a unique account number Provide written directions on how to complete the

surveys Set open/close time periods convenient for school(s)

to complete the survey When a survey period has been closed the survey

results and graphs are generated automatically Closed surveys can be reopened You can take a survey more than once

A Brief Overview of PBIS Assessment:www.pbisassessment.org

School Account and Login

Select “Surveys”Access your open survey by clicking on “Launch”

PBIS Assessment Reports

Three reports are available for the Tools:

Team Implementation Checklist (TIC)

Team Implementation Checklist (TIC)

Self-assessment tool for monitoring the implementation of school-wide PBS

Team self-assessment in six core areas• Establish commitment• Establish and maintain team• Self-assessment• Establish school-wide expectations• Establish information system• Build capacity for function-based support

Team Implementation Checklist (TIC)

WHO: Completed by the Universal PBIS Team

WHAT: Guides the development, implementation, monitoring and revision process for building a positive school culture

WHERE: At your school during a Universal PBIS Team meeting

WHEN: 3x per year

HOW: One person, usually the Internal PBIS Coach, enters the data online at www.pbisapps.org

Online View of Team ChecklistEnter date the survey was completed

Enter the TIC Data

TIC Total Score Report

TIC Subscale Report

TIC Individual Items Report

TIC Activity In a Small Group…

Review the results of the TIC Total Score, Subscale & Individual Item Reports

What do the survey data results say about this Universal team’s functioning?

What next steps would you recommend this team include in their Action Plan?

Why?

School-Wide Evaluation ToolSET

The SET is a research-validated instrument that is designed to assess and evaluate the critical

features of school-wide effective behavior support across an academic school year.

School-Wide Evaluation Tool (SET)

To determine the extent to which schools are already using School-wide Positive Behavior Support (PBIS)

To determine if training and technical

assistance efforts result in change when using PBIS

To determine if use of PBIS procedures is related to valued change in the safety, social culture, and violent behavior in schools

The Set was Designed To:

Assess features that are in place

Determine annual goals

Evaluate on-going efforts

Design and revise procedures

Compare year to year efforts in the area of PBIS

Information Gathered Can Be Used To:

Expectations defined Behavioral expectations

taught

Acknowledgement procedures

Correction procedures

Monitoring and evaluation

Management

District-level support

SET Evaluates Questions Across Seven Featured Areas:

SWIS

What are our “major” and “minor” behavior referral forms indicating?

What data options are available when a student needs additional social interventions by a teacher in the classroom (minor) or when other school personnel need to me informed (major), how data gets collected.

SWIS – Major & MinorOther district based system

Report Options using SWIS: Average Referrals Per Day Per Month Referrals By Problem Behavior Referrals By Location By Student Referrals By Time

How are we doing?Look for trendsPlan time to re-teach

Are there target behaviors?Behavior of the week orStaff training/development

By Problem Behavior

Look for hot spotsReview the rules for the locationsReview the lessons for teaching the rules

By Location

By Time

By Student

How are we doing?Look for trendsPlan time to re-teach

Do we have a solid green zone?

80%?

Self Assessment Survey(SAS)

Self Assessment Survey (SAS)

SAS assesses the extent to which PBIS practices and systems (essential components) are in place within a school• School-wide• Non-classroom• Classroom• Individual Student

SAS provides entire staff (certified and non-certified) perspective

Self-Assessment Survey (SAS) WHO: Completed by all teachers, staff and

administrators in your school building

WHAT: Examines the status and need for improvement of four behavior support systems (school-wide, classroom, non classroom, individual students)

WHERE: At your school

WHEN: Annually, usually at the beginning or towards the end of each school year

HOW: All school staff take the survey online at PBIS Assessment: www.pbisapps.org

Online View of Self-Assessment Survey (SAS)

Online View of Self-Assessment Survey (SAS)

• School-wide • Non-classroom

• Classroom

• Individual

SAS Total Score Report

SAS Subscale Report

SAS Individual Items Report

Self-Assessment Survey Activitywith a Small Group…

Use the SAS Individual Items Report as a reference.

Discuss how you might approach problem solving with your team if the staff identified an item or items as HIGH for Not In Place but LOW for Improvement Priority.

Data-Based Decision Making

Data-Based Decision-Making

SWIS or SWIS Like Systems

Developing Precise Statements Solution Development

• Prevention, Teaching, Reward, Extinction, Corrective Consequence, and Data Collection

Action Plan for Results

Using Data for On-Going Problem-Solving

Start with the perceptions not the data Use data in “decision layers”

• Is there a problem? (overall rate of ODR)• Localize the problem

(location, problem behavior, students, time of day)

• Get specific Don’t drown in the data It’s “OK” to be doing well Be efficient

Six Things to Avoid

Define a solution before defining the problem Build solutions from broadly defined, or fuzzy

problem statements Failure to use data to confirm/define problem Agree on a solution without building a plan for

how to implement or evaluate the solution Agree on a solution but never assess if the

solution was implemented Serial problem solving without decisions

Uses of Data

Review current status and identify problems early.• Use data on a regular basis (every two weeks) to

monitor key indicators, and identify problems before they become difficult.

Refine a problem statement to a level of precision that will allow functional solutions.

Use data to test possible solutions.

Use data to assess if solutions are working• If many students are making the same mistake it typically

is the system that needs to change not the students.

Benchmarks of Quality(BoQ)

Benchmarks of Quality (BoQ)

The Benchmarks of Quality measures the degree to which a school is implementing the universal level of school-wide positive behavior support.

The results provide a mechanism for school teams to identify strengths and weakness based on fidelity criteria for use in establishing future action plans.

Benchmarks of Quality (BoQ)

WHO: The PBIS Universal Team

WHAT: Lists 10 critical elements of quality for Universal School-wide PBIS implementation. This tool identifies strengths and opportunities for change and growth

WHERE: At your school

WHEN: The spring of each school year Mar/Apr/May

HOW: Online directions at www.pbisillinois.org • PoI Narrated Instructions (PPT 2003 & 2007)

Technical Assistance Coordinator Enters BoQ Online

BoQ Total Score Report

BoQ Subscale Report

BoQ Individual Items Report

BoQ Activity In a Small Group…

Review the results of the BoQ Subscale and BoQ Individual Item Reports.

What action items would you recommend this team consider for improving their Faculty Commitment and Discipline Procedures?

CollectCollect and Useand Use

DataData

Review Status and

Identify Problems

Develop andRefine

Hypotheses

Discuss andSelect

Solutions

Develop andImplementAction Plan

Evaluate andRevise

Action Plan

Problem Solving Foundations

Team Initiated Problem Solving (TIPS) Model

Identifying Problems/Issues What data to monitor

• ODR per day per month• OSS, ISS, Attendance, Teacher report• Team Checklist/ SET (are we doing what we planned to do?)

What question to answer• Do we have a problem?

What questions to ask of Level, Trend, Peaks• How do our data compare with last year?• How do our data compare with national/regional norms?• How do our data compare with our preferred/expected

status? If a problem is identified, then ask

• What are the data we need to make a good decision?

Using Data to Refine Problem Statement

The statement of a problem is important for team-based problem solving.

• Everyone must be working on the same problem with the same assumptions.

Problems often are framed in a “Primary” form, that creates concern, but that is not useful for problem-solving.

• Frame primary problems based on initial review of data• Use more detailed review of data to build “Solvable

Problem Statements.”

Precise Problem Statements(What are the data we need for a decision?)

Precise problem statements include information about the Big Five questions:• What is problem, and how often is it

happening• Where is it happening• Who is engaged in the behavior• When the problem is most likely• Why the problem is sustaining

Primary vs. Precision Statements

Primary Statements• Too many referrals• September has more

suspensions than last year

• Gang behavior is increasing

• The cafeteria is out of control

• Student disrespect is out of control

Precision Statements• There are more ODRs

for aggression on the playground than last year. These are most likely to occur during first recess, with a large number of students, and the aggression is related to getting access to the new playground equipment.

Primary vs. Precision Statements

Primary Statements• Too many referrals• September has more

suspensions than last year

• Gang behavior is increasing

• The cafeteria is out of control

• Student disrespect is out of control

Precision Statements• There are more ODRs

for aggression on the playground than last year. These are most likely to occur during first recess, with a large number of students, and the aggression is related to getting access to the new playground equipment.

Precise or Primary Statement?

Children are using inappropriate language with a high frequency in the presence of both adults and other children. This is creating a sense of disrespect and incivility in the school.

James D. is hitting others in the cafeteria during lunch, and his hitting is maintained by peer attention.

Precise or Primary Statement?

ODRs during December are higher than in any other month.

Minor disrespect and disruption are increasing over time, and are most likely during the last 15 minutes of our block periods when students are engaged in independent seat work. This pattern is most common in 7th and 8th grades, involves many students, and appears to be maintained by escape from work (but may also be maintained by peer attention… we are not sure).

Precise or Primary Statement?

Three 5th grade boys are name calling and touching girls inappropriately during recess in an apparent attempt to obtain attention.

Boys are engaging in sexual harassment.