Interpreting Scores - Home - Region 10 Website

7/21/2015

1

Interpreting ScoresPresented by:Kay Torchiana

Evaluation and Child Find ConsultantRegion 10

TEA Copyright Slide© 2015 Texas Education Agency

Copyright © Notice The materials are copyrighted © and trademarked ™ as the property of the Texas Education Agency (TEA) and may not be reproduced without the express written permission of TEA, except under the following conditions:

1) Texas public school districts, charter schools, and Education Service Centers may reproduce and use copies of the Materials and Related Materials for the districts’ and schools’ educational use without obtaining permission from TEA.

2) Residents of the state of Texas may reproduce and use copies of the Materials and Related Materials for individual personal use only without obtaining written permission of TEA.

3) Any portion reproduced must be reproduced in its entirety and remain unedited, unaltered, and unchanged in any way.

4) No monetary charge can be made for the reproduced materials or any document containing them; however, a reasonable charge to cover only the cost of reproduction and distribution may be charged.

Private entities or persons located in Texas that are not Texas public school districts, Texas Education Service Centers, or Texas charter schools or any entity, whether public or private, educational or non‐educational, located outside the state of Texas MUST obtain written approval from TEA and will be required to enter into a license agreement that may involve the payment of a licensing fee or a royalty.

For information, contact:

Office of Intellectual Property Texas Education Agency,

Room 2‐186 1701 N. Congress Ave.

Austin, TX 78701‐1494

phone: 512‐463‐9270 or 512‐463‐9713

e‐mail: [email protected]

© 2015 Texas Education Agency

2

7/21/2015

2

Acknowledgements

• Special thanks to Barbara Wendling and the 2008 Statewide Leadership Network


3

Community Considerations

•Limit technology distractions.

•Help your colleagues.•Participate throughout the session.•What is learned here, leaves here.


4

7/21/2015

3

Today’s Objectives

Participants will understand

• qualitative and quantitative data;

• how data relates to dyslexia assessment; and

• how the two data types may be used to explain a student’s strengths and deficits to inform a Section 504 committee.

“Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.” –Albert Einstein


5

Thinking About Scores


6

7/21/2015

4


7

= Measurable

= Observablebut not

measurable7 ounces glass container

frothy look cinnamon sprinkles

serving temperature 140 degrees

smells strong cup 3½ inches tall© 2015 Texas

Education Agency8

7/21/2015

5

Level 1 Qualitative, informal, error analysis Useful for instructional planning/identification

Observations Useful for behavioral observations

Level 2 Level of development Age equivalent

Level of instruction Grade equivalent

Level 3 Level of quality proficiency Relative Proficiency Index, CALP

Easy-to-difficult range Developmental/Instructional zone

Level 4 Relative standing in group Standard scores

Rank order Percentile ranks

Levels of Interpretive Information

© 2015 Texas Education

Agency

9

• Observe and analyze behaviors

• Validate interpretation of individual's test performance

• Analyze task demands and response processes

• Infer processing strengths and weaknesses

• Analyze errors© 2015 Texas

Education Agency10

7/21/2015

6

Determine the task and response demands

Examiner directions: Point to each word and say, “What word is this?”

Item scoring: Credit is given if the examinee says the word as a connected whole. No penalty for earlier attempts to sound out the word as long as final attempt is a smooth pronunciation.

on was it web

coming brother point eleven

Examinee Page:


11

Response demands

• The response requires reading the word orally. • There is no penalty for time or self-corrections. • The word must be said altogether as a whole

word, not in parts.

Task demands

The task requires reading real words. It does not require knowing meaning.


12

7/21/2015

7

1. no for on, if for it, saw for was

2. sometime for sometimes,brighten for brightened

3. /bruver/ for brother

4. /git/ for get

5. home for house

6. / t / / t / / rrrr / / tr / ă / / ŭ / / k / for truck

Analyze these errors from an identification perspective:


13

• Had hesitant, long delays between words

• Did not say words quickly and automatically

• Tried to sound words out

• Made errors that were typically real words

• Stated “Reading is hard.”

Observations made during testing


14

7/21/2015

8

What are the implications you can derive from all of this information?

(task and response demands, error analysis, and

observations)

Take a moment to list them now.


15

Instructional implications

• Orthography?

• Rules for long/short vowels?

• Knowledge of phonemes/graphemes?

• Oral vocabulary OK?

• Possible dyslexia program?

• Extended time to complete assigned readings?

• Fewer practice items?

• Scaffolding? © 2015 Texas Education Agency

16

7/21/2015

9

Rawscores

Age-gradeequivalents


17

Level 2: Age or Grade Equivalents

• Based on raw score

• Not affected by choice of age or grade norms

• Reflects age or grade level in norming at which average score is the same as the examinee’s raw score

• Abbreviated AE or GE

• Written with hyphen or period (10-4, 6.8)


18

7/21/2015

10

Level 2: Age or Grade Equivalents

Sample descriptions

• On the phonemic awareness task, 12-year-old Lisa scored similarly to an average 6-year-old.

• The number of items Tom, a 7th grader, answered correctly on the decoding task is comparable to the average student in early grade 4.


19

Level 3: Proficiency

Types of Level 3 scores: W scores, RPI, instructional or developmental ranges

• Indicates the quality of performance

•Helps monitor progress

• Indicates the range of development or instruction (independent to frustration)


20

7/21/2015

11

• Every item in the test is located at a point on the ruler.

• Every person’s performance is located at a point on this same ruler.

• The mean for each norm group is located at a point on this ruler.

Envision that an equal interval ruler underlies the test.

Measurement Rules

© 2015 Texas Education Agency 21

This allows us to determine the individual’s functional or developmental range.

Age 10-0Grade 5.0

Examinee, 5.0

We can see where on the ruler each examinee’s performance is located and how far it is from the average performance for his or her age or grade.


22

7/21/2015

12

RPIs are expressed as a fraction with the denominator fixed at 90. The numerator indicates the examinee’s proficiency on that task and can range from 0-100.

90/90: Examinee has average proficiency on task

• Provides a translation of a normative score into task-performance terms

• Compares person’s proficiency to average age or classmates

• Predicts level of success on similar tasks• Shows actual distance from average

Relative Proficiency Index (RPI)


RPI Ranges

RPI Ease of performance

95/90‐100/90 Easy to extremely easy

82/90 ‐ 94/90 Manageable

67/90 – 81/90 Difficult

24/90 – 66/90 Very difficult

4/90 – 23/90 Extremely difficult

0/90 – 3/90 Impossible© 2015 Texas Education Agency 24

7/21/2015

13

Example write‐up

Sam’s RPI of 21/90 on the Phoneme/Grapheme cluster

indicates that on similar tasks, in which the average

fourth‐grade student would demonstrate 90%

proficiency, Sam would demonstrate 21% proficiency.

Sam’s knowledge of phoneme‐grapheme

correspondence and spelling patterns is very limited.


2.2

6.6

3.6

2.8

6.6

2.7

5.7

Oral Language

Total ACH

Broad Reading

Broad Math

Broad Written Language

Basic Reading Skills

Math Calculation Skills

Jody’s grade placement = 5.0

3.8 11.1

2.8 4.6

2.4 3.3

5.0 8.8

1.9 3.7

1.9 2.5

4.2 7.8

94/90 70

67/90 22

24/90 8

96/90 84

51/90 8

5/90 4

93/90 68

RPI PR

K.0 1.0 2.0 3.0 4.0 6.0 8.0 10.0 12.0 16.0

K.5 1.5 2.5 3.5 5.0 7.0 9.0 11.0 14.0 18.0

What is the purpose of

the instructional

range?


7/21/2015

14

Level 3: Proficiency

Sample descriptions

• Karen will find decoding tasks easy at a beginning 3rd-grade level but difficult at a mid-4th-grade level.


27

Write descriptions for the following scores:Level 3: Proficiency and Instructional Range

Juan

• Eighth grader

• RPI=45/90 on written expression

Lena

• Fifth grader

• Instructional range on reading comprehension is 2.5 to 3.8


28

7/21/2015

15

Level 4: Peer Comparisons/Relative Standing

• Compares examinee to age peers

• Standard scores (equal interval)• Describe performance relative to the average

performance of the comparison group

• Examples: M=100, SD=15 or M=10, SD=3

• Percentile ranks (not equal interval)• Describe performance as relative standing in the

comparison group on a scale of 1 to 99

• Indicate the percentage of comparison group who had scores the same as or lower © 2015 Texas

Education Agency29

Criterion‐referenced

• Determines whether student has mastered certain skills or concepts

• Compares student to preset standard

Norm‐referenced

• Ranks student in relation to others

• Compares test‐taker to performance results of statistically selected group of same age or grade


30

7/21/2015

16

NormativeAverage Range

85-115

Normative Weakness

<85

Normative Strength

>115

The Bell Curve


More with the Curve


7/21/2015

17

Low averageBelow average

9 to 2480 to 89

Average25 to 7590 to 110

Very superiorVery high

98 to >99.9131 and above

Descriptive LabelsPercentile Rank Standard Scores

121 to 130 Superior High

92 to 97

76 to 91111 to 120 High averageAbove average

LowBorderlineWell below average

3 to 870 to 79

Very lowExtremely low

< 0.1 to 269 and below

85-115 is “average” on some tests.

NOTES:

Different tests use different ranges and labels.


33

Confidence Intervals

• Increases “confidence” that the examinee’s true score falls within an identified range.

• Uses the standard error of measure (SEM) around the obtained standard score to create the range


Commonly available confidence levels are 68%, 90%, and 95%. The higher the confidence level, the wider the band.

34

7/21/2015

18

• score +/-1 SEM = 68% level of confidence

• score +/-2 SEMs = 95% level of confidence

If the obtained score is 74 and the SEM is +/-3, then the range will be

71-77 at the 68% level (+/-1 SEM) +/-368-80 at the 95% level (+/-2 SEM) +/-6

Confidence Interval Examples


35

Confidence Interval at 68%


36

7/21/2015

19

Confidence Interval at 95%


37

Level 4: Peer Comparison/Relative Standing

Sample descriptions

• Only 2% of Betsy’s age mates scored higher than she did on rapid word reading (PR=98).

• Less than 1% of grade mates scored as low or lower than Bret on spelling (PR=.5).

• Compared to other 6th graders, Jesse’s performance in reading was in the low-average- to-average range (SS=88-96).


38

7/21/2015

20

Level 4: Peer Comparison/Relative Standing

Write descriptions for the following scores:


• Fourth grader

• SS=142 in word attack

Manuel

• Second grader

• SS=66‐74 (68% confidence) in word reading

Lacy

• Ninth grader

• PR=25 in phonological processing

Josh

39

How Scores Work Together


Results from Word Attack

Second grader (2.9)

College senior (16.9)

SS 75 75

PR 5 5

GE 1.1 6.3

RPI 10/90 68/90

40

7/21/2015

21

25th1st 75th 99th

reflect relative standing.Standard scores and percentiles

Sprint analogy: All racers finish close together.

Norms: A narrow distribution (Individuals did not vary too much on the task.)


A Racing Analogy

41

25th1st 75th 99th

Notice that the percentiles don’t changesince relative standing remains the same.

Marathon analogy: Racers are spread out at the finish.

Norms: A wider distribution (Individuals vary widely on the task)© 2015 Texas Education Agency 42

7/21/2015

22

Average level of performance or proficiency

50th

1st 25th 75th 99th

1st 25th 75th 99th

Actual distance from average


“How far from average is a horse’s performance?”

80/90 87/90 92/90 99/90

RPI

1st 25th 75th 99th

PR

Average


7/21/2015

23

1st 25th 75th 99th

Relative standing has not changed.

RPI

58/90 72/90 92/9099/90

Absolute distance from average has changed.


1. The student has a standard score of 80 in reading comprehension.

2. The student finds reading comprehension tasks easy at the beginning third-grade level and difficult at the end fourth-grade level.

3. On grade-level tasks, this student has limited proficiency in reading comprehension. He will have 3% success when average grade mates have 90% success (RPI=3/90).

Which is most helpful for instructional planning?

4. Four percent of grade mates scored this low or lower in reading comprehension.

5. In reading comprehension, this 6th-grade student had the same number correct as the average student in grade 3.5.


46

7/21/2015

24

Example: Using All Scores (5th grader)

Norm-referenced information• Word attack, SS=90• Word reading, SS= 91

Criterion-referenced information• Word attack, RPI = 74/90• Word reading, RPI=61/90• Oral reading fluency, 50 wcpm

(138 is benchmark)

Developmental/Instructional information• Word attack, Instructional zone: 2.5 to 4.9 • Word reading, Instructional zone: 2.9 to 4.3


47

Educational and family history

Performance in other achievement areas

Phonemic awareness, language, abilities

What Else Do We Need to Know?

Qualitative information examples• Makes guesses based on visual characteristics• Lacks word knowledge• Knows beginning sounds• Does not apply phonological analysis • Has no apparent strategies for comprehension


48

7/21/2015

25

Tricky Score Issues


49

Declining Standard Score Issue

In Grade 2.0, a student obtained an SS of 106 on a test.

How would you explain this decline in the standard score?

Has the student made any progress? How would you determine this? © 2015 Texas

Education Agency

In Grade 4.0, the same student obtained an SS of 89 on that same test.

50

7/21/2015

26

“Tests do not think for themselves, nor do they directly

communicate with patients. Like a stethoscope, a blood

pressure gauge, or an MRI scan, a psychological test is a

dumb tool, and the worth of the tool cannot be separated

from the sophistication of the clinician who draws

inferences from it and then communicates with patients

and professionals”

Meyer et al. (2001, February). Psychological testing and psychological assessment. American Psychologist. © 2015 Texas

Education Agency

Last Thoughts

51

Documents

Interpreting Scores - Home - Region 10 Website