FloridaRtIf

FloridaRtI.usf.edu

A collaborative project between the Florida Department of Education and the University of South Florida

Assessing Response to Intervention

Advance Organizer

Team Activity - Review Day Four Intervention Plan

Why Monitor Progress?

Graphing Conventions

Goal Setting

Interpreting Graphs

Decision Making

Review of Problem-Solving Steps

Team Activity-Review Day 4 Intervention Plan

1. How did the intervention plan your team wrote on Day 4 differ from intervention plans typically written at your school? How was the plan similar?

2. For which components of the intervention plan did your team provide the most descriptive and specific details?

3. For which components of the intervention plan could your team have provided additional detail to clarify what needed to happen for implementation to occur?

4. How has the writing of intervention plans changed since your team attended the Day 4 training? How have the plans remained the same?

Beliefs Survey

Your project ID is:

• Last 4 digits of SS#

• Last 2 digits of year of birth

Response to Interventionin Context

Identify

the Problem

Monitor

ProgressAnalyze

the ProblemImplement

Intervention

Evaluate

Intervention

Effectiveness

Timeline

Select/DesignIntervention

• Unless we monitor progress, we cannot determine the rate at which the gap is closing.

• Continuous feedback improves instructional planning (formative assessment)

• Allows earlier decisions about what to do-Increase time, decrease time, revisit problem-solving, etc.

• Provides measure of intervention effects-Getting better, staying the same, or getting worse

• We don’t know the effectiveness of an intervention until we implement it and monitor progress

Why Monitor Progress?

Randy Allison, 2004

Why Monitor Progress?• Provides clear idea of expectations of performance over

time.

• Student outcomes improve when performance is assessed regularly

• Allows visual comparison to a standard

• Data collection provides an objective data base for decision-making

• Know if learning is being enabled, and if so, under what conditions

• Continuous feedback on performance enhances motivation for many

Randy Allison, 2004

Outcome of Monitoring: Diagnose Conditions that Enable Learning

• Identify conditions under which student desired learning and behavior is accelerated and you have determined what enables learning.

• If conditions are configured like X, Y, or Z, then…what benefits result for the student?

Hi

LoTime

X Y Z

... . .. . . .... .......

. .......

Randy Allison, 2004

Why Not Just UsePre - Post Test Comparisons?

• We must measure and demonstrate more than improvement. We must show the rate necessary to attain benchmarks within a time frame.

• May be unreliable because of small amount of data collected

• May be significant time lag between pre-test and post-test

• Not sensitive to small changes in direction of performance in a timely manner

• More difficult to analyze patterns of performanceRandy Allison, 2004

Basics of CBM - Scientifically Based Progress Monitoring

• Research Efforts Led By Stanley Deno, Beginning in 1971 with Federal Funding in 1978 to Provide Viable Progress Monitoring toward IEP Goals

• Almost 30 Years of Continuous Research

• Mid 1990s Witnessed Move to Standard, High Quality Assessment Materials

• A Number of Members of the CBM “Family” including DIBELS

M. Shinn

Why CBM…or any Other Measure?

• Quick and inexpensive

• Linked to instruction and curriculum

• Frequently repeatable

• Sensitive to small increments of growth

• Reliable and valid

• Can be used for multiple purposes and to answer different questions

Jenkins, Deno, and Mirkin

Academic Measures -CBM-

Reading Fluency Words read correctly

Reading Comprehension

Correct words inserted when the seventh word is left out

Spelling Correct words spelled and correct letter sequences

Written Expression

Words written or correct word sequences written

Math Computation

Correct digits written from math computation problems

Math Applications

Correct digits written from math application problems

From M.Shinn

Behavioral Measures

• Office discipline reports

• Behavioral incidents

• Suspension/Detention records

• Observations

• Self-assessments

• Surveys

• Attendance data

• Teacher checklists

• Screening instruments

• Rating scales

Sam

Digits correct per minute

7

6

5

4

3

2

1

12

11

10

9

8

12

1 2 3 4 5 6 7 8Days

Why use graphs?

• Teachers are able to make sound decisions about the instruction being delivered to students based upon data, not guesswork

• Parents are kept well informed about their child’s progress with specific information about how their child is responding to instruction. Parents may assist in making suggestions for instructional adjustments.

• Students know what is expected of them. They receive specific feedback about their performance along the way rather than only at the end of the marking period. Goal setting and progress monitoring are some of the most effective strategies to improve academic engaged time.

General Outcomes /Specific Skills

• The general outcome expectations for students are comprised of many requisite subskills. Both general outcomes and specific skills may be measured.

• What subskills would be necessary for:

Long division?

Fluent reading?

Compliance with teacher direction?

Graph Components

Skill

equalincrements

Time - equal increments

Instructional Change Line

Goal

Intervention(Group or Individual)

Baseline

Aim Line

Trend Line

Skill

Time2 wks 2 wks

Keep increments consistent!

Skill

Time2 wks 2 wks


In both conditions, the rate of skill acquisition is 1/week.

Dissimilar x-axis increments give the impression that learning rate is increased during second time interval

Skill

Time2 wks


Similar x-axis increments reveal true progress over time

2 wks

Goal Setting

• Set goals which are ambitious, but reasonable -- rate required to reach goal is 25-50% above typical student rate

• Goal has two components Level of performance desired

Time within which that level will be attained

Goal Setting

• Measure difference between desired and current performance

• Divide by number of weeks

• Compare to standard to determine reasonable & ambitious growth rate (an increase of 25-50% of typical rate)

Goal SettingDesired - Current

Number of Weeks

110 wcm - 60 wcm

20 weeks

50 wcm

20 weeks

2.5 wcm/week2.0 wcm/week is typical

Compare to rate for typical peers: in 25-50% range?

Goal Setting

To determine an increase of 25-50%-

For lower end of range - multiply typical rate X 1.25For upper end of range - multiply typical rate X 1.50

Examples:

If typical rate = 2.00 words correct per minute/week Ambitious range would be 2.5 - 3.0 words correct per minute/week

If typical rate = 3.00 digits correct per minute/weekAmbitious range would be 3.75 - 4.50 digits correct per minute/week

Goal Setting

If:

Goal is 3.5 words correct per minute/week &2.0 words correct per minute/week is typical

Is this in ambitious range?

Goal is .75 digits correct per minute/week &.5 digits correct per minute/week is typical

Is this in ambitious range?

2.0 letter sounds correct per minute/week is typical

What is ambitious range?

ExampleGrowth Rates

• Realistic growth rates (words/ week)

• Grade 1 2.00 words



• Grade 4 .90 words



• Ambitious growth rates (words/week)







Fuchs, Fuchs, 1993

Attend to:

• Level • Slope / Rate

• Variability

Decision Rules: What is a “Good” Response to Intervention?

• Positive Response

Gap is closing

Can extrapolate point at which target student(s) will “come in range” of target--even if this is long range

• Questionable Response

Rate at which gap is widening slows considerably, but gap is still widening

Gap stops widening but closure does not occur

• Poor Response

Gap continues to widen with no change in rate.

Performance

Fall

Positive Response to Intervention

Expected Performance

Observed Performance

Winter Spring

Performance

Time

Positive Response to Intervention

Expected Trajectory

Observed Trajectory



Gap is closing





• Poor Response


Performance

Fall

Questionable Response to Intervention



Winter Spring

Performance

Time

Questionable Response to Intervention

Expected Trajectory

Observed Trajectory



Gap is closing





• Poor Response


Performance

Fall

Poor Response to Intervention



Winter Spring

Performance

Time

Poor Response to Intervention

Expected Trajectory

Observed Trajectory

Performance

Time

Response to Intervention

Expected Trajectory

Observed Trajectory

Positive

Questionable

Poor

DecisionsWhat to do if RtI is:

• Positive

• Continue intervention with current goal

• Continue intervention with goal increased

• Fade intervention to determine if student(s) have acquired functional independence.


• Questionable Was intervention implemented as intended?

• If no - employ strategies to increase implementation integrity

• If yes -

Increase intensity of current intervention for a short period of time and assess impact. If rate improves, continue. If rate does not improve, return to problem solving.


• Poor Was intervention implemented as intended?

• If no - employ strategies in increase implementation integrity

• If yes -

Is intervention aligned with the verified hypothesis? (Intervention Design)

Are there other hypotheses to consider? (Problem Analysis)

Was the problem identified correctly? (Problem Identification)

Intervention Integrity Decisions

Evidence based intervention linked to verified hypothesis planned

Evidence based intervention implemented

Student Outcomes (SO)

Assessed

Treatment Integrity (TI) Assessed

Data-based Decisions

Continue Intervention

Implement strategies to promote treatment integrity

Modify/change Intervention

+SO +TI

-SO +TI

-SO -TI

From Lisa Hagermoser Sanetti, 2008 NASP Convention

Response to Interventionin Context

Identify

the Problem

Monitor

ProgressAnalyze

the ProblemImplement

Intervention

Evaluate

Intervention

Effectiveness

Timeline

Select/DesignIntervention

Progress Monitoring Resources

• Interventioncentral.org

• Studentprogress.org

• Dibels.uoregon.edu

www.interventioncentral.com

www.studentprogress.org

dibels.uoregon.edu

School Level Data Review Worksheet

Your project ID is:



• Read the case study

• Answer six questions using data provided

Review

Consensus

Infrastructure

Implementation

Review

The steps of problem solving as an integrated, fluid, self-

correcting process.

Step 1 - What’s the Problem?

In order to identify a problem, you’ve got to start with three pieces of data-

• Benchmark level of performance

• Student level of performance

• Peer level of performance

Is this an individual student problem or a larger systemic problem?

Decision Making Rubricfor use with

School-Wide Screening

adapted from:

Heartland AEA 11, Improving Children’s Educational Results

Are over 20% of students

struggling?

Are between 5% and 20% of Are 5% or fewer

and developgroup

intervention

Examine instruction,

curriculum, and environment for

needed adaptations

Develop small group

intervention

Go to problem definition

Go to intervention evaluation

studentsstruggling?

studentsstruggling?

Step 1 - What’s the Problem?

0

10

20

30

40

50

60

70

80

90

100

Sept Oct Nov Dec Jan Feb

% Compliance

35%

Benchmark75%

= Peer Group = Aim Line

BASELINE

The problem is occurring because ________________.

If ____________ would occur, the problem would be reduced.

Goal: The development of hypotheses about probable causes for the identified problem.

Assessments are then conducted to gather information to determine which are most / least likely

Prediction statement:

Step 2- Why is it occurring?

Assessment:How Do We Confirm Hypothesis?

DOMAINSR

ReviewI

InterviewO

ObserveT

Test

IInstruction

CCurriculum

EEnvironment

LLearner

Step 3- What are we going to do about it?

• Effective teaching strategies consider both what to teach and how to teach it.

• Making good decisions will increase student progress.

• It is critical that the instruction be matched to the problem.

Howell & Nolet, 2000

Step 4- Is it working?

Goal

ClassroomIntervention I

Making instructional decisions based on the review and analysis of student data

Progress monitoring always includes graphing

Progress Monitoring

ClassroomIntervention 2

Traditionalvs.


Intervention

Intervention

Consider ESETraditional

Intervention

Intervention

Intervention Consider ESE

Ifnecessary


RegularEducation

MonitorProgress

MonitorProgress

What is the Problem?Why is itoccurring?Is it working?What are we going to do about it?

Tiers of Service Delivery

I

II

III

Problem Identification

Problem Analysis

Intervention Design

Responseto

Intervention

What is RtI?

RTI is the practice of (1) providing high-quality instruction/intervention matched to student needs and (2) using learning rate over time and level of performance to (3) make important educational decisions to guide instruction.

National Association of State Directors of Special Education, 2005

Perceptions of Skills Survey

Your project ID is:



Final Thoughts

Problem Solving &


Training Evaluation

No project ID needed

Thanks!

Resources

http://www.oswego.edu/~mcdougal/web_site_4_11_2005/index.html (Graphing made easy)

http://www.jimwrightonline.com/php/chartdog_2_0/chartdog.php (Chart Dog : on-line graphing tool)

http://www.studentprogress.org/

http://www.progressmonitoring.org/

Resources

• Armstrong, J., & Anthes, K. (2001). How data can help: Putting information to work to raise student achievement. American School Board Journal, 188, 38-41

• Coburn, C., & Talbert, J. (2006). Conceptions of evidence use in school districts: Mapping the terrain. American Journal of Education, 112, 469-495.

• Kerr, K., Marsh, J., Ikemoto, G., Darilek, H., & Barney, H. (2006). Strategies to promote data use for instructional improvement: Actions, outcomes, and lessons from three urban districts. American Journal of Education, 112, 496-520.

• Young, V. (2006). Teachers’ use of data: Loose coupling, agenda setting, and team norms. American Journal of Education, 112, 521-548.

Documents

FloridaRtIf