A Causal Rasch Model for Understanding Comprehension in the Context of Reader-Text-Task

1

A Causal Rasch Model for Understanding Comprehension in the Context of Reader-

Text-TaskAERA/NCME April 13-17, 2012

Vancouver, Canada

A. Jackson StennerDonald S. Burdick

Mark H. Stone

2

Reading is a process in which information from the text and the

knowledge possessed by the reader act together to

produce meaning as measured by a particular task.

Anderson, R.C., Hiebert, E.H., Scott, J.A., & Wilkinson, I.A.G. (1985) Becoming a nation of readers: The report of the Commission on Reading Urbana, IL: University of Illinois

3

Data Structure for Testing the Lexile Theory

a b c d e f g h i j k l m … N123456789

10111213…N

Reader-text-task transaction produces an outcome which can be viewed as reading comprehension and/or text comprehensibleness usefully presented as a percent.

Texts (Native Task)

Read

ers

Empirical Text Complexity Measures

New Task Types

Read

er A

bilit

y M

easu

res

4

A Causal Rasch Model

= Reader Ability

Text Complexity

Comprehension

-Conceptual

Statistical

RawScore

=i

e (RA – TC - TD)i

1 + e (RA – TC i - TD)

RA = Reading AbilityTC = Text ComplexityTD = Task Difficulty

Task Difficulty

-

5

200L

1700L 200L

1700L

-300L 300L

Reader AbilityDial

Text ComplexityDial Task Difficulty

Dial

0

ComprehensionDisplay

72%

The Measurement Trade-off Property

6

Imagine a world where assessment items are generated

and scored in real-time.

7

Theoretical versus Empirical Text Complexity for

719 Articles*

Reliability = 0.997

SEM = 12.8L

r = 0.968

r” = 0.969

R2” = 0.938

RMSE” = 89.6L

* Inclusion criteria: 50 encounters and 1,000 items

Mean Theoretical = 884.4L (356.2)

Mean Empirical = 884.4L (355.0)

8

Artifactual Sources of Variance in Empirical Text complexity Measures1. Random measurement error2. Sampling error3. Range restriction4. Systematic error in empirical complexity measures5. Wrong function form (not linear)6. Variation in empirical text complexities across

estimation algorithms

We have estimated that the first three of these artifactual sources of variance account for no more than 4% of the total variance in the system – leaving 2% still unexplained. Sources 4-6 may account for this remaining 2%.

9

May 2007 – April 2011347 Encounters138,695 Words3,342 Items983 Minutes

Student 15287th GradeMaleHispanicPaid Lunch

Text Demands forCollege and Career

1200

1000

1400

1600

May 2016(12th Grade)

10

Conclusions1. There is only a very small proportion of

variation in empirical text complexity left unexplained.

2. None of the hypotheses about genre (expository vs. narrative), coherence, cohesion, grade dependence, gender dependence, second language dependence have been supported.

3. It is possible that the small amount of unexplained variance is due to artifacts in the estimation of empirical text complexity.

11

A. Jackson Stenner Chairman & CEO, MetaMetrics

University of North Carolina, Chapel Hilljstenner@Lexile.com

Contact Info: