Tom Jenkins, Ed.D. Educational Consultation Services, LLC Wilmington, NC

The Formative Evaluation Component of Your MTIM

Data Based Instructional Decision Making from the Building Level to the Individual Student

Level

Tom Jenkins, Ed.D.

Educational Consultation Services, LLCWilmington, NC

Formative Evaluation

• Collecting data to make determinations about the effectiveness of your MTIM implementation

Collecting data to make determinations about the effectiveness of instruction with students – and everything in between

Managing Complex Change= Change

Confusion=

=

=

=

=

+ + + +Vision Skills Incentives ResourcesAction Plan

+ + + +Skills Incentives Resources Action Plan

+ + + +Vision Incentives ResourcesAction Plan

+ + + +Vision Skills ResourcesAction Plan

+ + + +Vision Skills IncentivesAction Plan

+ + + +Vision Skills Incentives Resources

Adapted from Knoster, T.

Anxiety

Resistance

False Starts

Frustration

5Big Ideas

ciCore Instruction

siSupplemental

Instruction

iiiIntensive

Instruction

Efforts lack focus and priority. There is not a

focus on important priority skills for

improvement.+ +3

Assessments(Screening, diagnostic,

progress)+ + =

BIBig Ideas

iiInitial Instruction

Overall low achievement. Student

learning problems across all subgroup

areas.+ +3


progress)+ + =

Lack of direction to know what needs to be

improved, which students need

intervention, and whether or not

interventions have been effective.

+ +3


progress)+ + =

siStrategic

Instruction

Lack of resources due to attempts to provide intensive interventions

for those students whose needs could be met through strategic

interventions.+ +

3Assessments

(Screening, diagnostic, progress)

+ + =

iiiIntensive

Instruction/Intervention

Gap increases between average and "at risk students" Continued low performance for some subgroups.+ +

3Assessments


+ + =

IDMSuccess+ +

3Assessments


+ + =

ciCore Instruction

ciCore Instruction

ciCore Instruction

ciCore Instruction

BIBig Ideas

BIBig Ideas

BIBig Ideas

BIBig Ideas

siSupplemental

Instruction

siSupplemental

Instruction

siSupplemental

Instruction

siSupplemental

Instruction

iiiIntensive

Instruction

iiiIntensive

Instruction

iiiIntensive

Instruction

iiiIntensive

Instruction

Analyzing Your MTIM Implementation

• Building consensus is necessary in three stages– pre-implementation– implementation– maintenance


• Pre-implementation tasks– Identify the purpose of

implementation and the critical data

• whose performance will improve as a result of implementation?

• what aspect of performance will improve?

• how will you know?

• take some time to think about this


• Pre-implementation tasks continued– Identify the mechanisms that will cause

student performance to change• enhanced delivery of Core• screening• intervening• progress monitoring


• Pre-implementation tasks continued– Present plan to stakeholders

• allow for stakeholder input• create forms of communication for stakeholder

groups• don’t forget your school board• disagreement is normal and should not be

overlooked


• Implementation tasks– Consensus is most

easily gained through success!

– implement where leadership is strong

– assess treatment integrity

– start small– provide support– communicate success

to stakeholders


• Maintenance tasks– Build RTI/TIM into the school culture

• set up team meetings that are on the calendar• present success stories and current status data at

staff meetings


• Maintenance tasks continued– Training is required

each and every year• assessment tool

training• intervention training• new staff• survey your staff

activity


• Maintenance tasks continued– maintain communication with and present

results to stakeholders– show connection between implementation

and state accountability measures


• Things that reduce consensus and support for RTI implementation– lack of identifiable team or individual who is in

charge– lack of focus on what RTI is about– asking classroom teachers to do everything– not communicating to critical stakeholders– not identifying how or when students move

between tiers

Data For Each Tier - Where Do They Come From?

• Tier 1: Universal Screening, accountability assessments, grades, classroom assessments, referral patterns, discipline referrals

• Tier 2: Universal Screening - Group Level Diagnostics (maybe), systematic progress monitoring, formative assessment large-scale assessment data and classroom assessment

• Tier 3: Universal Screenings, Individual Diagnostics, intensive and systematic progress monitoring, formative assessment, other informal assessments

Building MTIM Infrastructure

• Formative evaluation process

• Informed by data• Highly involved school-based leadership

team (SBLT)• School-based MTIM coach

– Provide Technical Assistance– Interpretation and Use of Data– Facilitates regular data meetings for building

and grade levels

Formative Evaluation Component of Infrastructure

• What we need:– Screening system for

identifying students at risk– Diagnostic assessment

tools for identifying specific needs of students identified by screening

– Systematic, explicit, research based instructional strategies – differentiated instruction

– Progress monitoring plan– Evaluation of whether

instruction is effective

Data For Each Tier – Where Do They Come From?

Addl.Diagnostic

Assessment

InstructionResults

Monitoring

IndividualDiagnostic

IndividualizedIntensive

weekly

Screen All Students

Monthly

Screening

Bench-Mark

Assessment

AnnualTesting

Behavior Academics

None ContinueWithCore

Instruction with differentiation

GradesClassroom

AssessmentsYearly Assessments

StandardProtocol

SmallGroupInterventionBy skill

2 times/month

Step 1Step 2 Step 3 Step 4

Supplemental

1-5%

5-10%

80-90%

Core

Intensive

Tier I - Core/Benchmark

• Universal Screening – Academics: Screen all students, begin in

kindergarten; 3 times per year with appropriate early literacy and math measures

– More intense instruction and monitoring within classroom for students below cut scores

– See worksheet

Cut Score Worksheet• Step One: Put all student scores on

the university screening measure on a histogram type chart.

• Step Two: Calculate typical Growth Rate of specific skills. Three formulas can be used here.

• EOYBM – BOYBM / 36 weeks = GR• Or• EOYBM – MOYBM / 18 weeks = GR• Or• MOYBM – BOYBM / 18 weeks = GR• Step Three: Determine the Targeted

Growth Rate for students. Two formulas can be used here depending on the desired amount of ambitiousness.

GR * 1.5 = TGR

• OrGR * 2.0 = TGR

Cut Score Worksheet• Step Four: Calculate the Growth Goal for

the instructional period. • TGR * NWI (18 or 36) = GG• Step Five: Calculate the Cut Score for

determination of level of instruction. Two formulas can be used here depending on the length of the instructional period used in step four.

MOYBM – GG = CS

• OrEOYBM – GG = CS

• Step Six: Using the Cut Score place a line of demarcation on the histogram created in step one. Any students above the Cut Score should obtain the GG via Core instruction. Any students below the Cut Score may need Strategic instruction to obtain the TGR and GG. Students in need of Intensive instruction should be identified using progress monitoring data during Strategic instruction implementation. Progress monitoring within all three tiers allows for students movement between the tiers during the instructional period.

Cut Score Worksheet Activity

• Knowing that your MOYBM is 40 and your BOYBM is 20 what would be the cut score using a accelerator of 1.5?

• BOYBM = 20• MOYBM = 40• 18 weeks of

instruction/intervention• Accelerator of 1.5

Cut Score Worksheet Activity

• Step Two: MOYBM – BOYBM / 18 = GR40 – 20 / 18 = 1.11

• Step Three: GR * 1.5 = TGR1.11 * 1.5 = 1.67

• Step Four: TGR * NWI = GG1.67 * 18 = 29.97

• Step Five: MOYBM – GG = CS40 – 29.97 = 10.03

• Step Six: All students above the score of 10.03 should be able to meet the expected Growth Goal via Core instruction. All students below the score of 10.03 would probably be initially placed in Strategic instruction to obtain the Targeted Growth Rate and necessary Growth Goal.

05

101520253035

1 3 5 7 9 11

Series1

Universal Screening ResultsFidelity Check: Are you doing the right

thing?

• Assess success of instructional program– Percent of students at or above benchmarks– If necessary, examine curriculum, instruction,

or both

• Identify students below benchmarks– Interventions within general education

classroom– Assess progress and consider need for more

intensive interventions

Decision Rules: What is a “Good” Response to Intervention?

• Positive Response

– Gap is closing

– Can extrapolate point at which target student(s) will “come in range” of target--even if this is long range

– Level of “risk” lowers over time

• Questionable Response

– Rate at which gap is widening slows considerably, but gap is still widening

– Gap stops widening but closure does not occur

• Poor Response

– Gap continues to widen with no change in rate.

Performance

Time

Positive Response to Intervention

Expected Trajectory

Observed Trajectory

Decision Rules: Linking RtI to Intervention Decisions

• Positive

• Continue intervention with current goal

• Continue intervention with goal increased

• Fade intervention to determine if student(s) have acquired functional independence.

Decision Rules: What is a “Questionable” Response to Intervention?


– Gap is closing





– Level of “risk” remains the same over time

• Poor Response


Performance

Time

Questionable Response to Intervention

Expected Trajectory

Observed Trajectory


• Questionable

– Was intervention implemented as intended?

• If no - employ strategies to increase implementation integrity

• If yes -

– Increase intensity of current intervention for a short period of time and assess impact. If rate improves, continue. If rate does not improve, return to problem solving.

Decision Rules: What is a “Poor” Response to Intervention?


– Gap is closing





• Poor Response


– Level of “risk” worsens over time

Performance

Time

Poor Response to Intervention

Expected Trajectory

Observed Trajectory


• Poor

– Was intervention implemented as intended?

• If no - employ strategies in increase implementation integrity

• If yes -

– Is intervention aligned with the verified hypothesis? (Intervention Design)

– Are there other hypotheses to consider? (Problem Analysis)

– Was the problem identified correctly? (Problem Identification)

Building Your MTIM

Thoughts, fleshing out your pyramid, and action plan activity

• Is your district/school ready for a universal screening process? Consider……– Do you have a person with the ability to be a

data coach?– Who would be your school leadership

team/data analysis team?– Current screening assessment tools– Staff ability to analyze data and make

instructional decisions– Do you have consensus?

Data For Each Tier – Where Do They Come From?

Addl.Diagnostic

Assessment

InstructionResults

Monitoring

IndividualDiagnostic

IndividualizedIntensive

weekly

Screen All Students

ODRsMonthly

Bx Screening

Bench-Mark

Assessment

AnnualTesting

Behavior Academics

None ContinueWithCore

Instruction with differentiation

GradesClassroom

AssessmentsYearly Assessments

StandardProtocol

SmallGroupInterventionBy skill

2 times/month

Step 1Step 2 Step 3 Step 4

Supplemental

1-5%

5-10%

80-90%

Core

Intensive

Need for CBM Type Assessments

• All reading probes scored corrects per minute


• Math computations are scored by correct digits per minute


• Correct Sequences for written expression

• Two words form a sequence, word and punctuation form a sequence.

• Most words and punctuation are used twice

• Three minutes to brainstorm, write, and edit

Tier II Strategic and Tier III Intensive Progress Monitoring

• Progress monitoring is essential for four reasons– There is no guarantee that interventions will be successful, thus

the intervention must be “tested” to evaluate effectiveness– Increased emphasis of specific outcomes for students, data

base must be generated to guide intervention decision making– Pre/post testing has be shown to be unreliable (small amount of

data) and provides too little data to allow for instructional decision making – progress monitoring allows for evaluation of level of performance and rate of learning

– Research has shown that progress monitoring is associated with improved educational outcomes

Research has shown that it works!

• If we use research validated reading practices, monitor students’ progress and make changes to instruction based on what we find, between 95 and 100 percent of children can become proficient readers.

• Torgesen, 2000, Learning Disabilities Research and Practice.

• Individual Differences in Response to Early Intervention in Reading: The Lingering Problem of Treatment Resisters


• Essential components that must be in place for successful progress monitoring within each tier– A well-defined behavior– Identification of student’s current level of performance

(baseline)– Instruction/Intervention– Goal– A measurement strategy– Graph– Decision-making plan


• Target behaviors that are observable and measurable

• Skill specific assessments

• Focus on enabling skills– Skills that are prerequisite skills for more

complex skills– Deficiencies in enabling skills often adversely

affects performance on global assessments


• Enabling skills for reading– Phonemic awareness– Alphabetic understanding– Fluency– Sight words– Comprehension


• Enabling skills for math– Number sense– Facts– Computation– Applications– Problem solving

• Enabling skills for written expression– Mechanics– Expression


• Enabling skills for behavior– Social skills– Work completion– Compliance– Problem solving skills


• Goal setting…………….• Standard against which progress can be compared• Allows for aimline to be established• Possible goals

– Level of behavior that is expected – several ways to establish this

– Most frequently used is accelerated growth rates for academics and percent of time expectations for behavior


• To identify an accelerated growth rate– Take the growth rate that is calculated and

multiply it by 1.5 to obtain a slightly ambitious growth rate or

– Take the growth rate that is calculated and multiply it by 2 to obtain a more ambitious growth rate

– Then multiply by the number of weeks of intervention


• For behavior research indicates that a 75% level of performance can be used for non threatening behaviors

• For behaviors that are threatening or dangerous a 100% level of performance should be used


• Goal calculation activity– Using the data below what would Nicole’s goal be, in

each area, for an intensive intervention plan that was implemented for nine weeks using a 1.5 accelerator?

Skill Nicole’s Baseline

District 3rd Grade BOYBM

District 3rd Grade MOYBM

Sight Word Phrase Fluency

53 81 91

Oral Reading Fluency

83 131 157


• Graph provides a visual representation of a large amount of data

• A visual representation of student’s acquisition of skills and allows for easier analysis of progress

• Paper and pencil or electronic based graphing

Charting Activity

• Activity– Plot the baseline data

• Monday – 14• Tuesday – 10• Wednesday – 6• Thursday – 5• Friday – 10

– Plot the median and signify with a heavy dark line

Charting Activity

• Plot the goal at the end of eight weeks– 40

• Indicate the aimline

Charting Activity

• Plot the first two weeks progress monitoring data– Week One

• Tuesday – 16• Thursday – 14

– Week Two• Monday – 10• Wednesday – 18• Friday – 16

• Make an informed decision regarding the effectiveness of the intervention

Charting Activity

• Plot progress monitoring data for next two weeks– Week Three

• Tuesday – 14• Thursday - 10

– Week Four• Monday – 15• Wednesday – 16• Friday – 16


Charting Activity

• Phase two of the intervention– Plot the progress monitoring of the next two weeks– Week Five


– Week Six• Monday – 26• Wednesday – 24• Friday – 26


Charting Activity

• Phase two of the intervention– Plot the progress monitoring of the next two weeks– Week Seven


– Week Eight• Monday – 32• Wednesday – 36• Friday – 38


• Final decision?

Charting Activity Continued

• Trend lines• Line that you draw through a series of data points

that represents the student’s actual rate of progress

• If trend line slope is flatter than aimline slope, then adjust intervention

• If trend line slope is steeper than aimline slope, then adjust goal

• If slopes are the same, make no change


• Trend line procedures– Draw vertical line in middle of graph, half data

points on one side, half on the other – if odd number of data points, put line through middle point

– Draw a vertical line in first half of data to separate data half and half

– Draw a vertical line in second half of data to separate data half and half


• Trend line procedures– Draw a horizontal line through median of first

half of data – form an intersection with vertical line

– Draw a horizontal line through median of second half of data – form an intersection with vertical line

– Connect the two intersections– Make your decision, is it the same decision?

Formative Evaluation Case StudySkill Prior To

Tier 3 Intervention

Nicole’s Baselines

District 3rd Grade Fall

Mean

Proficiency WBP

Phoneme ID 32,32,30 41 28 22

Blend ID 3,4,6 24 11 5

Sight Word ID 50,53,56 81 68 58

Words In Passages

80,83,84 131 105 80

Addition 0,0,5 13 9 7

Subtraction 2,1,2 5 1 1

Multiplication 7,5,4 11 4 2

Math Concepts

3,2,1 7 4 2

Formative Evaluation Case StudySkill District 3rd

Grade Winter Mean

Proficiency WBP

Phoneme ID 54 43 36

Blend ID 32 16 9

Sight Word ID 91 77 68

Words In Passages

157 114 94

Addition 14 10 8

Subtraction 6 3 2

Multiplication 21 15 11

Math Concepts

10 6 5

Formative Evaluation Case Study

• Nicole– Progress monitoring data for addition

• Week one 8,3,13• Week two 13,9,8• Week three 11,3,11• Week four 13,12,12


• Nicole – Progress monitoring data for blends

• Week one 7,8,6• Week two 6,5,5• Week three 4,6,4• Week four 4,4,2• Week five 6,6,7• Week six 5,8,10


• Nicole – Progress monitoring data for sight words

• Week one 61,63• Week two 64,49• Week three 51,53• Week four 50,52• Week five 56,48• Week six 44,50


• Nicole– Based on your progress monitoring data,

complete the Analysis of Intervention Plan• Identify median score of last three progress

monitoring data points for each skill• Compare current level of performance to

previously established goals• Make a decision about the results of the

intervention plan and the next step• Is this a student that needs to be considered for

specially designed instruction?

0

20

40

60

80

100BenchmarkGoal

Egbert

Weeks

Wo

rds

Co

rre

ct

Per

Min

ute

Student in January of First Grade

0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20

Benchmark=24

Baseline=11

0

20

40

60

80

100 Line 1

Line 2

Weeks

Wo

rds

Co

rre

ct

Per

Min

ute

Determine Goal: Class=1.5 wd growth per week; Student Goal: 2.25 wd growth per week

0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20

Bchm=24

Baseline=11

Benchmark Line

Student goal line

0

20

40

60

80

100Bench

Goal

Student

Weeks

Wo

rds

Co

rre

ct

Per

Min

ute

Monitor Students’s Progress Relative to Goal

0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20

Bchm=24

Baseline=11

Benchmark Line

Student goal line

0

20

40

60

80

100Bench

Goal

student

Weeks

Wo

rds

Co

rre

ct

Per

Min

ute

Formative Evaluation: Change Intervention

0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20

Bchm24

Baseline=11

Benchmark

Student goal line

ChangeIntervention

0

20

40

60

80

100Bench

Goal

student

Weeks

Wo

rds

Co

rre

ct

Per

Min

ute

Continue Intervention and Monitor Progress

0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20

Bnch=24

Baseline=11

Benchmark

Student goal line

ChangeIntervention

0

20

40

60

80

100Class

Bench

studentLine 4

Weeks

Wo

rds

Co

rre

ct

Per

Min

ute

Continue Intervention and Monitor Progress

0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20

Bchmrk=24

Baseline=11

Benchmark

Student goal line

ChangeIntervention

Fade Tier ?

0

20

40

60

80

100Class

Goal

Weeks

Wo

rds

Co

rre

ct

Per

Min

ute

Determine Goal: Class=1.5 wd growth per week; Student Goal: 2.25 wd growth per week

0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20

Bnchmrk=24

Baseline=11

Benchmark Line

Student goal line

0

20

40

60

80

100ClassGoalstudent

Weeks

Wo

rds

Co

rre

ct

Per

Min

ute

Monitor Student Progress Relative to Goal

0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20

Bchmrk=24

Baseline=11

Benchmark line

Student goal line

0

20

40

60

80

100ClassGoalstudent

Weeks

Wo

rds

Co

rre

ct

Per

Min

ute

Change Student Intervention

0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20

Bchmk=24

Baseline=11

Benchmark line

Student goal line

ChangeIntervention

0

20

40

60

80

100ClassGoalstudent

Weeks

Wo

rds

Co

rre

ct

Per

Min

ute

Implement Revised Intervention and Continue to Monitor Progress

0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20

Benchmark Line

Student goal line

ChangeIntervention

0

20

40

60

80

100ClassGoalstudent

Weeks

Wo

rds

Co

rre

ct

Per

Min

ute

Implement Second Intervention Revision

0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20

Benchmark

student goal line

ChangeIntervention

0

20

40

60

80

100ClassGoalstudent

Weeks

Wo

rds

Co

rre

ct

Per

Min

ute

Gap Not Closing: Consider Eligibility and More Intensive Interventions

0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20

Benchmark

ChangeIntervention

ClassWCM=54

StudentWCM=32


• If the decision is to adjust an intervention, small changes or refinements are recommended before major changes

• However, changes should be substantial enough that it has a possibility to result in improved student performance

• If making an adjustment, do not make two at the same time. It may result in the team being unable to determine what caused increased student performance

• Consider intensity vs methodology

Building Your MTIM

Thoughts, fleshing out your pyramid, and action plan activity

• Is your district/school ready for implementation of a multi-tiered assessment model? Consider……– Current diagnostic and formative assessment

tools– Staff ability to analyze data and make

instructional decisions– Do you have consensus?– Where are the gaps in your pyramid?

Questions?

• Dr. Tom Jenkins, Director• Educational Consultation Services, LLC• Wilmington, NC• (910) 367-7209• [email protected]• www.educationalconsultationservices.com

mailto:[email protected]

Thanks

• Dan Reschly

• Dave Tilly

• George Batsche

• Ed Shapiro

• Tracy Hall

Documents

Tom Jenkins, Ed.D. Educational Consultation Services, LLC Wilmington, NC