MN RtI Center1

RTI Measurement Overview: Measurement Concepts for RTI

Decision MakingA module for pre-service and in-service

professional development

MN RTI CenterAuthor: Lisa H. Stewart, PhD

Minnesota State University Moorhead

www.scred.k12.mn.us click on RTI Center

MN RtI Center2

MN RTI Center Training Modules

This module was developed with funding from the MN legislature It is part of a series of modules available from the MN RTI Center

for use in preservice and inservice training:

Module Title Authors

1. RTI Overview Kim Gibbons & Lisa Stewart

2. Measurement and RTI Overview Lisa Stewart

3. Curriculum Based Measurement and RTI Lisa Stewart

4. Universal Screening (Benchmarking): (Two parts)

What, Why and How

Using Screening Data

Lisa Stewart

5. Progress Monitoring: (Two parts)

What, Why and How

Using Progress Monitoring Data

Lisa Stewart & Adam Christ

6. Evidence-Based Practices Ann Casey

7. Problem Solving in RTI Kerry Bollman

8. Differentiated Instruction Peggy Ballard

9. Tiered Service Delivery and Instruction Wendy Robinson

10. Leadership and RTI Jane Thompson & Ann Casey

11. Family involvement and RTI Amy Reschly

12. Five Areas of Reading Kerry Bollman

13. Schoolwide Organization Kim Gibbons

MN RtI Center3

Overview

Purpose(s) of assessment Characteristics of effective measurement for RTI Critical features of measurement and RTI in the

areas of screening, progress monitoring, and diagnostic instructional planning

CBM/GOMs as a frequently used RTI measurement tool

Multiple sources of information and convergence

MN RtI Center4

Why Learn About Measurement?

“In God we trust…

All others must have data.”Dr. Stan Deno

MN RtI CenterDRFT May 27, 2009 5

Adapted from Logan City School District, 2002

Curriculum and Instruction Assessment

School Wide Organization &

Problem Solving Systems

(Teams, Process, etc)

Assessment: One of the Key Components in RTI

MN RtI Center

Measurement and Assessment

Schools have to make many choices about measurement tools and the process of gathering information used to make decisions (assessment)

We need different measurement tools for different purposes

MN RtI Center

Some Purposes of Assessment

Screening

Diagnostic - instructional planning

Monitoring student progress (formative)

Evaluation (summative)

MN RtI CenterDRAFT May 27, 2009 8

Screening

Standardized measures given to all students to: Help identify students at-risk in a PROACTIVE way Give feedback to the system about how students progress

throughout the year at a gross (e.g., 3x per year) level If students are on track in the fall are they still on track in the

winter? What is happening with students who started the year below

target, are they catching up? Give feedback to the system about changes from year to year

Is our new reading curriculum having the impact we were expecting?

MN RtI Center

Diagnosis/Instructional Planning

Measures given to understand a student’s skill level (strengths and weaknesses) help guide: Instructional grouping Where to place the student in the curriculum &

curricular materials What skills are missing or weak and may need to be

retaught or practiced and the level of support and explicitness needed

Development or selection of curriculum and targeted interventions

MN RtI Center

Monitoring Student Progress (Formative)

Informally this happens all the time and helps teachers adjust their teaching on the spot

More formalized progress monitoring involves standardized measures, tied to important educational outcomes, and given frequently (e.g. weekly) to: Prompt you to change what you are doing with a student if it is not

working (formative assessment) so you are effective and efficient with your time and instruction

Make decisions about instructional goals, materials, levels, and groups Aid in communication with parents Document progress for special education students as required for

periodic and annual reviews

MN RtI Center

Evaluation (Summative)

Measures used to provide a snapshot or summary of student skill at one particular point in time, often at the end of the instructional year or unit E.g. state high stakes tests

"When the cook tastes the soup, that’s formative; when the guests taste the soup, that’s summative."

MN RtI Center

One Test Can Serve More Than One Purpose

To the extent a test does more than one thing well, it is a more efficient use of student time and school resources

Example 1: Reading CBM measures of Oral Reading Fluency can be used for screening and progress monitoring

Example 2: the NWEA (MAP) test may be used for screening and instructional planning

MN RtI Center

Activity

On Measurement Overview Purposes of Assessment Worksheet

Make a list of all the tests you have learned about or have seen used in the school setting (or are currently in use in your school)

Try to decide what purpose(s) each test served

MN RtI Center

Assessment Tools and Purpose(s)

Name of Test Purpose(s)(Screening, Instructional Planning, Progress Monitoring, Program Eval.)

MN RtI Center

Buyer Beware

Although it is good if a test can serve more than one purpose, just because a test manual or advertisement SAYS it is useful for multiple purposes, doesn’t mean the test actually IS useful for multiple purposes

Example: Many tests designed for diagnostic purposes or for summative evaluation state they are also useful for progress monitoring, but are too time consuming, too costly, too unreliable, or too insensitive to changes in student skills to be of practical use for progress monitoring

MN RtI Center16

Establishing a Measurement System

A core feature of RTI is identifying a measurement system Screen large numbers of students

Identify students in need of additional intervention

Monitor students of concern more frequently 1 to 4x per month

Typically weekly

Diagnostic testing used for instructional planning to help target interventions as needed

17

valid

reliable

simple

quick

inexpensive

easily understood

can be given often

sensitive to growth over short periods of time

Characteristics of An Effective Measurement System for RTI

Credit: K Gibbons, M Shinn

MN RtI Center

Technical Characteristics of Measurement Tools Reliability- the consistency of the measure

If tested again right away or by a different person or with an alternate equivalent form of the test, the score should be similar

Allows us to have confidence in the score and use the score to generalize what we see today to other times and situations If a student knows how to decode simple words on a sheet of

paper at 8am this morning, we would expect him to be able to decode similar simple words at noon… and the next day…

MN RtI Center

Why is Reliability so Important?

Assume you have a test that decides whether or not you need to take (and pay for) a remedial math class in college that does not count toward graduation. The test average score is 50 points. The test has a “cut off” score of 35, so students who

score below 35 have to take the remedial class.

MN RtI Center

Why is Reliability so Important? (Cont’d)

If the test is reliable, and you get a score of 30, if you take another version of the test or take the test again a week later (without major studying or changing what you know!) you would likely get a score very close to 30….

If the test is not reliable, and you get a score of 30…You might be able to take the test again or take another version of the test and get a score of 40…or a score of 20! If the test is unreliable we can’t have much faith in the score

and it becomes difficult to use the test to make decisions!

MN RtI Center

Validity But what if the test IS reliable and you get a score of 30 but

your math skills are much better than the score implies? What if you get a score of 30 but you don’t really need a remedial math class?

Then the test has an issue with VALIDITY- A test is valid only if the interpretation of the test scores are supported A common definition of validity is that “the test measures what it says

it measures” Another definition is that a test is valid if it helps you make better

decisions or leads to better outcomes than if you had never given the test

MN RtI Center

Types of Validity

There are many ways to try to demonstrate validity:

Content validity

Criterion related validity: concurrent and predictive

Treatment Validity

Construct Validity

MN RtI Center

Types of Validity (Cont’d)

Content validity The test content is reasonable

Criterion related validity: two types Concurrent- the scores from this test are similar to

scores from other tests that measure the same/similar thing

Predictive- the test scores from this test do a pretty good job of letting us know what score a student will get on another test in the future

MN RtI Center

Types of Validity (Cont’d)

Treatment Validity

If you use this test to decide about some treatment or intervention or instructional approach….

Do you make better decisions? Do you have better goals? Planning? Student

engagement? Most importantly: Are the outcomes for your students

better?

MN RtI Center

Types of Validity (Cont’d)

Construct Validity Does the test measure the theoretical trait or

characteristic? E.g. If the theory says children need to have a base of

solid decoding skills before they will be fast and fluent readers of new text, do the scores on the reading test of decoding and fluency support that?

All other ways to try to document validity are in some way also addressing construct validity (content, criterion, treatment, etc.)

MN RtI Center

The NOT Validity Kind of Validity

Face validity is NOT really validity Positive: It “looks” good

Just because a test looks good or you (or your colleague) like to give it does not mean it gives you good information or is the best test to use

Negative: I just don’t like it Just because a test isn’t set up exactly how you like it does not

mean it does NOT give you good information

Look for EVIDENCE of reliability and validity, don’t rely on your reaction, or the reactions and testimonials of colleagues, alone.

MN RtI Center

Reliability and Validity

Just because a test is reliable does not mean it is valid It may reliably give you an inaccurate score!

If a test is not reliable, it cannot be valid No test or test score is perfectly reliable We use test scores to help make a variety of

decisions-- some “low stakes” and some “high stakes” decisions…. So how reliable is “reliable enough”? It depends ….

MN RtI Center

Measuring Reliability and Validity

Typically reliability and validity evidence involves comparing the test to itself or to other tests or outcomes

The statistic used to sum up that comparison is often a correlation ( r )

Correlations vary from r = 0.0 to 1.0 The closer a correlation is to 1.0 the “stronger” the

relationship or the better you can predict one score or outcome if you know the other one

MN RtI Center

How Reliable is Reliable Enough?

For important INDIVIDUAL decisions? r = .90

For SCREENING decisions? r = .80 Salvia & Yselldyke, 2006

“Reliability is like money, as long as you have it, it’s not a problem, but if you don’t, it’s a BIG problem!” ~ Fred Kurlinger

MN RtI Center

How Valid is Valid Enough?

Ranges Interpretation

.00-.20.00-.20 Little/no validityLittle/no validity

.21-.40.21-.40 Below average Below average validityvalidity

.41-.55.41-.55 Average validityAverage validity

.56-.80.56-.80 Above average Above average validityvalidity

.80-.99.80-.99 Exceptional validityExceptional validity

Source: Webb, MW, 1983 journal of reading, 26(5) 414-424

MN RtI Center

Looking at Validity With a Purpose in Mind

Predictive Validity is really important if you are using the test as a screening tool to predict which students are at risk or not at risk of reading difficulty

Treatment validity is really important if you are using the test in an effort to lead to some sort of improved outcome

MN RtI Center

Validity isn’t Just About the Test

Validity has to do with the test use and interpretation, so even a “valid” test can be used for the wrong reasons or misinterpreted or misused

Example 1: A test score for an ELL student should reflect the student’s skills, not her ability to understand the directions and what is being asked

Example 2 on next slide

MN RtI Center

Validity isn’t Just About the Test (Cont’d) Example 2: Letter Naming Fluency (LNF)

LNF involves giving a student a page of randomized upper and lower case letters and having the student name as many letters as they can in one minute.

As a test of early literacy, LNF has good reliability and concurrent and predictive validity, especially predictive validity

However, it can be easily MISUSED— If interpreted correctly, LNF can identify students at risk for early reading difficulty

and get those students into well-rounded early literacy instruction well suited to them,

BUT, if it is interpreted to mean that a student low in LNF needs to just have a lot of instructional time spent only learning letter names (often taking time away from high quality well-rounded early literacy instruction) it can actually have a negative impact.

MN RtI Center

Test Utility

Is it easy to use, time efficient, and cheap? Even if a test is reliable and valid, if it is too difficult

to use, too time consuming, or too expensive it just won’t get used If a reliable and valid progress monitoring tool took 30

minutes per child and you wanted to monitor 10 students in your class every week, would you use it?

However, if a test is easy and short and cheap… but isn’t reliable or valid… it’s still a waste of time, no matter how short!

MN RtI Center

Test Utility (Cont’d)

Is it sensitive enough for the decisions you want to make? Can it detect the differences between groups of

kids or within an individual that you need to help you make a decision? If a progress monitoring tool can only show gains of 1 point

per month, is it sensitive enough to help give you timely feedback on the student’s response to your instruction?

MN RtI Center

Activity On “Characteristics of Assessment Tools for RTI”

Worksheet Make a list of tests you have learned about or have seen used in the

school setting (or are currently in use in your school) Can use all or some of the tools from the Purposes of

Assessment Worksheet for your list Is the test reliable and valid FOR THE PURPOSE IT IS BEING

USED? Is it quick and simple? Is it inexpensive? Can it be given often (has alternate forms, etc)? Is it sensitive?

MN RtI Center

Characteristics of Assessment Tools for RTI

Name of tool

Reliable Valid Quick & simple

Cheap Can be given often

Sensitive to growth over short time

MN RtI Center

Some Help in Looking for Evidence

Measurement tools are reviewed at the following sites: www.rti4success.org www.studentprogress.org

These sites only review tests submitted, if it is not on the list it doesn’t mean it is bad, just that it wasn’t reviewed

Be sure you know the purpose of assessment (screening, progress monitoring, etc) to best interpret the information

MN RtI Center39

Critical Features of Measurement and RTI

Screening

Progress Monitoring

Diagnostic Instructional Planning

MN RtI Center

Measurement and RTI: Screening

Reliability coefficients of at least r =.80. Higher is better, especially for screening specificity.

Well documented predictive validity Evidence the criterion (cut score) being used is

reasonable and creates not too many false positives (students identified as at risk who aren’t) or false negatives (students who are at risk who aren’t identified as such)

Brief, easy to use, affordable, and results/reports are accessible almost immediately

MN RtI Center

Measurement and RTI: Progress Monitoring

Reliability coefficients of r=.90+ Because you are looking at multiple data points

over time, it is possible to use a test with a lower reliability (e.g. .80-.90), but wait until you have several data points and use the combined data to increase confidence in your decisions

Well documented treatment validity!

MN RtI Center

Msrmnt & RTI: Progress Monitoring (Cont’d)

Test and scores are very sensitive to increases or decreases in student skills over time Evidence of what slope of progress (how much growth

in a day, week or a month) is typical under what conditions can greatly increase your ability to make decisions

VERY brief, easy to use, affordable, alternate forms, and results/reports are accessible immediately

MN RtI Center

Measurement and RTI: Diagnostic Assessment for Instructional Planning

Reliability coefficients of r =.80+ ASSUMING you are open to changing the instruction (formative assessment) if your planning didn’t work out as you thought it might

Aligned with research on the development and teaching of reading

Well documented treatment validity, utility for instructional planning!

Time and cost efficient but specific enough to be useful for designing effective interventions

Linked to standards and curriculum scope and sequence

MN RtI Center

Msrmnt & RTI: Diagnostic Assessment for Instructional Planning (Cont’d)

Many instructional planning tools have limited information on reliability and validity—Look for tools that do have data.

If creating your own tests, use best practices in test construction.

Overall be sure you are doing standardized frequent progress monitoring and looking at student engaged time as other sources of information to ensure instruction is well planned.

MN RtI Center45

RTI, General Outcome Measures and Curriculum Based Measurement

Many schools use Curriculum Based Measurement (CBM) general outcome measures for screening and progress monitoring You don’t “have to” use CBM, but many schools do

Most common CBM tool in Grades 1- 8 is Oral Reading Fluency (ORF) Measure of reading rate (# of words correct per minute on a grade

level passage) and a strong indicator of overall reading skill, including comprehension

Early Literacy Measures are also available such as Nonsense Word Fluency (NWF), Phoneme Segmentation Fluency (PSF), Letter Name Fluency (LNF) and Letter Sound Fluency (LSF)

MN RtI Center46

Why GOMs/CBM? Typically meet the criteria needed for RTI screening and

progress monitoring Reliable, valid, specific, sensitive, practical Also, some utility for instructional planning (e.g., grouping)

They are INDICATORS of whether there might be a problem, not diagnostic! Like taking your temperature or sticking a toothpick into a cake Oral reading fluency is a great INDICATOR of reading decoding,

fluency and reading comprehension Fluency based because automaticity helps discriminate between

students at different points of learning a skil

MN RtI CenterDRAFT May 27, 2009 47

GOM…CBM… DIBELS… AIMSweb…

MN RtI Center48

CBM Oral Reading Fluency

Give 3 grade-level passages using standardized administration and scoring; use median (middle) score

3-second rule (tell the student the word & point to next word) Discontinue rule (after 0 correct in first row, if <10 correct on 1st

passage do not give other passages)

Errors Not Errors

Hesitation for >3 seconds

Incorrect pronunciation for context

Omitted Words

Words out of order

Repeated Sounds

Self-Corrects

Skipped Row

Insertions

Dialect/Articulation

MN RtI Center

Fluency and Comprehension

A good measures of overall reading proficiency is reading fluency because of its strong correlation to measures of comprehension.

The purpose of reading is comprehension

MN RtI Center

The Importance of Multiple Sources of Information

No ONE test is going to serve all purposes or give you all the information you need.

Use MULTIPLE sources of data to make the best decisions Screening, progress monitoring, diagnostic, and evaluative

data from multiple sources and/or across time Teacher observation and more formal observations Other pieces of relevant information such as behavior,

attendance, health, the curriculum and instructional environment, etc.

Look for CONVERGENCE of data- places where several sources of data point to the same decision or conclusion

MN RtI Center51

Articles Available with this Module

Shoemaker, J. (2006). Reliability and Validity Stats “crib sheet” from Heartland AEA (Iowa) Traditional and Modern Concepts of Validity. ERIC/AE

Digest Also see articles specific to particular uses of

measurement in benchmark and progress monitoring modules

MN RtI Center

Recommended Resources

American Psychological Association, American Educational Research Association, & National Council on Measurement in Education. (1985). Standards for educational and psychological testing. Washington, DC: American Psychological Association.

Educational Measurement Text, e.g. texts by Hogan, Marzano, or Salvia & Ysseldyke, or a good Educational Psychology text that covers reliability, validity and utility of measurement

MN RtI Center

Web Resource on Measurement

Heartland (Iowa) website link with powerpoints on common myths and confusions about assessment http://www.aea11.k12.ia.us/assessment/myt

hbuster.html

MN RtI Center54

RTI Related Resources National Center on RTI

http://www.rti4success.org/

RTI Action Network – links for Assessment and Universal Screening http://www.rtinetwork.org

MN RTI Center http://www.scred.k12.mn.us/ and click on link

National Center on Student Progress Monitoring http://www.studentprogress.org/

Research Institute on Progress Monitoring http://progressmonitoring.net/

MN RtI CenterDRAFT May 27, 2009 55

RTI Related Resources (Cont’d)

National Association of School Psychologists www.nasponline.org

National Association of State Directors of Special Education (NADSE) www.nasdse.org

Council of Administrators of Special Education www.casecec.org

Office of Special Education Programs (OSEP) toolkit and RTI materials http://www.osepideasthatwork.org/toolkit/ta_responsiveness_int

ervention.asp

MN RtI CenterDRAFT May 27, 2009 56

Quiz

1. A purpose of assessment is what? A.) Screening B.) Diagnostic C.) Progress Monitoring D.) Evaluation E.) All of the above

2. True or False? A test is useful for multiple purposes as long as its manual or advertisement says it is.

MN RtI Center

Quiz

3. The consistency of the measure is called its what? A.) Validity B.) Reliability C.) Criterion D.) Sensitivity

4. If the test measures the construct it says it measures it has? A.) Validity B.) Reliability C.) Criterion D.) Sensitivity

MN RtI Center

Quiz

True or False for each statement?

5.) Even if a test is not valid, it can still be reliable.

6.) Even if a test is not reliable, it can still be valid.

7.) Validity is not just about the test—it has to do with the test use and interpretation, so even a valid test can be used for the wrong reasons, misinterpreted, or misused.

MN RtI Center

The End

Note: The MN RTI Center does not endorse any particular product. Examples used are for instructional purposes only.

Special Thanks: Thank you to Dr. Ann Casey, director of the MN RTI Center, for

her leadership Thank you to Aimee Hochstein, Kristen Bouwman, and Nathan

Rowe, Minnesota State University Moorhead graduate students, for editing, writing quizzes, and enhancing the quality of these training materials