Montréal, CSSE / CERA 2010 1.Raîche, Béland, Magis, Blais and Brochu

Modelling atypical students response patterns using multidimensional parametric models

Gilles Raîche, UQAMSébastien Béland, UQAM

David Magis, Université de LiègeJean-Guy Blais, Université de Montréal

Pierre Brochu, CMEC

Large-Scale Assessments: Policy, Research and PracticeCSSE / CERA

Montréal, 2010

Montréal, CSSE / CERA 2010 2.Raîche, Béland, Magis, Blais and Brochu

• Introduction and Objectives• Unidimensional IRT Models• IRT Person Parameters Models

– Person response Curve– Multidimensional Item Response Models– Estimation– An R Package: irtProb

• Examples• Other Considerations• References and contacts

SUMMARY

Montréal, CSSE / CERA 2010 3.Raîche, Béland, Magis, Blais and Brochu

• Presentation

• IRT Models of Interest– Unidimensional latent proficiency– Dichotomous response– Monotonic– Logistic Probability Distribution

INTRODUCTION

Montréal, CSSE / CERA 2010 4.Raîche, Béland, Magis, Blais and Brochu

• Simulation of Inappropriate Response Patterns

• Person Misfit Detection Indices• Distributional Properties of Person Misfit

Indices• Adjusted Proficiency Level Estimation in

Presence of Person Misfit

OBJECTIVES

Montréal, CSSE / CERA 2010 5.Raîche, Béland, Magis, Blais and Brochu

UNIDIMENSIONAL IRT MODELS

3 Parameter Logistic (3PL) (Birnbaum, 1968)

4 Parameters Logistic (4PL) (McDonald, 1967)

where if ai is considered as a standard deviation

)(1)(),,,|1(

iji baii

iiiijijij ecccabXPP

)(1)1(),,,|1(

iji bai

iiiijijij ecccabXPP

2

1

i

is

a

Montréal, CSSE / CERA 2010 6.Raîche, Béland, Magis, Blais and Brochu

PERSON RESPONSE CURVE

(Trabin and Weiss, 1983)-4 -2 0 2 4

0.0

0.2

0.4

0.6

0.8

1.0

-- Difficulty (b) +

Pro

babi

lity

0-200

3-100

000.30

0300.2

Montréal, CSSE / CERA 2010 7.Raîche, Béland, Magis, Blais and Brochu

MULTIDIMENSIONAL ITEM RESPONSE MODELS• Personal Variance (σ2) (Ferrando, 2004; Thurstone, 1927)

• Personal Inattention (δ)

• • Personal Pseudo-Guessing (χ) (Strandmark and Linn, 1987)

2

)(

1

),,,,,|1(i

ij

s

biji

ijjiiiiij

e

cdcdcbsxP

22

)(

1

),,,,,|1(ji

ij

s

bii

ijjiiiiij

e

cdcdcbsxP

2

)(

1

),,,,,|1(i

ij

s

bjii

jijjiiiiij

e

cdcdcbsxP

Montréal, CSSE / CERA 2010 8.Raîche, Béland, Magis, Blais and Brochu

MULTIDIMENSIONAL ITEM RESPONSE MODELS• Higher Order Models

22

)(

1

),,,,,,|1() and ,(ji

ij

s

bjii

ijjjiiiiij

e

cdcdcbsxP

22

)(

1

)(),,,,,,|1() and ,(

ji

ij

s

biji

ijjjiiiiij

e

cdcdcbsxP

22

)(

1

)(),,,,,,,|1() and ,,(

ji

ij

s

bjiji

ijjjjiiiiij

e

cdcdcbsxP

Montréal, CSSE / CERA 2010 9.Raîche, Béland, Magis, Blais and Brochu

• Package: irtProb• MAP Estimators• A Priori Probability Distribution

– σ : U(0,4) – θ: U(-4,4)– X: U(0,1)– δ : U(0,1)

ESTIMATION OF SUBJECT PARAMETERS

Montréal, CSSE / CERA 2010 10.Raîche, Béland, Magis, Blais and Brochu

• Available on R Cran Site• Functionnalities

– Estimation of Person Parameters (MAP)– Likelihood Curves– Person Characteristic Curves– Probability, Density and Random Functions– Simulation of Response Patterns– Classical <-> IRT Item Parameters– Model Selection

A R PACKAGE: irtProb

Montréal, CSSE / CERA 2010 11.Raîche, Béland, Magis, Blais and Brochu

EXAMPLES – 01 (X)Table 1. Person Parameter estimation from 100 simulated

subjects attempting to augment their estimated proficiency level to a 40 items test [person parameter (Standard Error)]

Pseudo-GuessingModel1 0.00 0.10 0.20 0.30 0.40

1 θ = -2 -2.18 (0.62)

-1.28 (0.69)

-0.41 (0.69)

0.03 (0.83)

1.00 (0.72)

2 θ = -2C

-2.18 -2.18 (0.62)(0.62)0.00 0.00

(0.01)(0.01)

-1.75 -1.75 (0.69)(0.69) 0.06 0.06 (0.07)(0.07)

-1.55 -1.55 (0.70)(0.70) 0.16 0.16 (0.10)(0.10)

-1.96 -1.96 (0.88)(0.88) 0.26 0.26 (0.12)(0.12)

-1.64 -1.64 (1.18)(1.18) 0.35 0.35 (0.13)(0.13)

3 θ = -2CSD

-1.98 (0.64)0.01

(0.02)0.07

(0.23)0.03

(0.08)

-1.61 -1.61 (0.74)(0.74) 0.07 0.07 (0.08)(0.08)0.17 0.17

(0.44)(0.44)0.03 0.03

(0.07)(0.07)

-1.46 -1.46 (0.79)(0.79) 0.17 0.17 (0.11)(0.11)0.23 0.23

(0.51)(0.51)0.03 0.03

(0.06)(0.06)

-1.62 -1.62 (1.10)(1.10) 0.24 0.24 (0.14)(0.14)0.44 0.44

(0.92)(0.92)0.03 0.03

(0.07)(0.07)

-1.22 -1.22 (1.59)(1.59) 0.33 0.33 (0.17)(0.17)0.43 0.43

(0.87)(0.87)0.02 0.02

(0.05)(0.05)

1 σ = 0, δ = 0, b = -5 to 5, c = 0, d = 0, 40 items, 100 simulated sujects Model 1: θ only Model 2: θ and Pseudo-Guessing Model 3 σ, θ, Pseudo-Guessing and δ

Montréal, CSSE / CERA 2010 12.Raîche, Béland, Magis, Blais and Brochu

EXAMPLES – 01 (X)

(θ=-2, X=0.2) (θ=-2, X=0.4)

-4

-2

0

2

4

0.0

0.2

0.4

0.6

0.8

1.0

0e+00

2e-11

4e-11

6e-11

8e-11

P(X)

-4

-2

0

2

4

0.0

0.2

0.4

0.6

0.8

1.0

0.0e+00

5.0e-11

1.0e-10

1.5e-10

2.0e-10

2.5e-10

P(X)

Montréal, CSSE / CERA 2010 13.Raîche, Béland, Magis, Blais and Brochu

EXAMPLES – 02 (X)Table 2. Person Parameter estimation from 100 simulated

subjects attempting to augment their estimated proficiency level to a 40 items test [person parameter (Standard Error)]

Pseudo-GuessingModel1 0.00 0.10 0.20 0.30 0.40

1 θ = 2 2.19 (0.55) 2.52 (0.61) 3.02 (0.58) 3.31 (0.62) 3.46 (0.56)

2 θ = 2C

2.09 (0.61)0.02 (0.06)

2.37 (0.60)2.37 (0.60)0.04 (0.10)

2.52 (0.85)0.12 (0.18)

2.74 (0.88)0.16 (0.21)

2.51 (1.01)0.27 (0.23)

3 θ = 2SCD

1.95 (0.55)0.11 (0.27)0.02 (0.05)0.01 (0.02)

2.11 (0.61)2.11 (0.61)0.17 (0.34)0.17 (0.34)0.08 (0.13)0.08 (0.13)0.00 (0.01)0.00 (0.01)

2.10 (0.93)2.10 (0.93)0.12 (0.32)0.12 (0.32)0.20 (0.19)0.20 (0.19)0.00 (0.01)0.00 (0.01)

2.42 (0.90)2.42 (0.90)0.12 (0.40)0.12 (0.40)0.22 (0.21)0.22 (0.21)0.00 (0.01)0.00 (0.01)

2.02 (1.08)2.02 (1.08)0.09 (0.30)0.09 (0.30)0.35 (0.23)0.35 (0.23)0.00 (0.01)0.00 (0.01)1 σ = 0, δ = 0, b = -5 to 5, c = 0, d = 0, 40 items, 100 simulated subjects

Model 1: θ only Model 2: θ and Pseudo-Guessing Model 3 σ, θ, Pseudo-Guessing and δ

Montréal, CSSE / CERA 2010 14.Raîche, Béland, Magis, Blais and Brochu

EXAMPLES – 02 (X)

(θ=2, X=0.2) (θ=2, X=0.4)

-4

-2

0

2

4

0.0

0.2

0.4

0.6

0.8

1.0

0e+00

2e-05

4e-05

6e-05

8e-05

P(X)

-4

-2

0

2

4

0.0

0.2

0.4

0.6

0.8

1.0

0.0e+00

5.0e-06

1.0e-05

1.5e-05

2.0e-05

2.5e-05

3.0e-05

P(X)

Montréal, CSSE / CERA 2010 15.Raîche, Béland, Magis, Blais and Brochu

EXAMPLES – 03 (σ)Table 3. Person Parameter estimation from 100 simulated

subjects with fluctuating proficiency level to a 40 items test [person parameter (Standard Error)]

FluctuationModel1 0.00 0.50 1.00 2.00 4.00

1 θ = -2 -2.14 (0.59) -2.07 (0.61) -2.00 (0.62) -1.71 (0.70) -1.01 (0.86)

2 θ = -2S

-2.06 (0.58) 0.33 (0.42)

-2.03 (0.59) 0.51 (0.57)

-2.05 (0.67) 0.90 (0.68)

-2.00 (0.84) 1.87 (0.90)

-1.63 (1.28) 3.45 (0.75)

3 θ = -2SCD

-1.92 (0.66) 0.11 (0.24)0.01 (0.01)0.04 (0.09)

-1.75 (0.79) 0.15 (0.29)0.01 (0.02)0.06 (0.11)

-1.54 (0.83) 0.28 (0.45)0.01 (0.03)0.11 (0.14)

-0.97 (1.42) 0.71 (0.84)0.03 (0.05)0.18 (0.18)

-0.68 (2.37) 0.79 (1.24)0.15 (0.14)0.23 (0.21)

1 X = 0, δ = 0, b = -5 to 5, c = 0, d = 0, 40 items, 100 simulated subjects Model 1: θ only Model 2: θ and σ Model 3 σ, θ, Pseudo-Guessing and δ

Montréal, CSSE / CERA 2010 16.Raîche, Béland, Magis, Blais and Brochu

EXAMPLES – 03 (σ)

(θ=-2, σ=1) (θ=2, σ=4)

-4

-2

0

2

4

0

1

2

3

4

2.0e-12

4.0e-12

6.0e-12

8.0e-12

1.0e-11

1.2e-11

P(X)

-4

-2

0

2

4

0

1

2

3

4

2e-06

4e-06

6e-06

8e-06

P(X)

Montréal, CSSE / CERA 2010 17.Raîche, Béland, Magis, Blais and Brochu

OTHER CONSIDERATIONS

• Multidimensional EAP Estimation Very Computer Intensive

• Warm Weighted Likelihood Estimator

• Item Parameters Estimation

• Confidence Interval For The Additionnal Person Parameters

• Other Person Fit Indices: Pseudo-Guessing and Inattention

Montréal, CSSE / CERA 2010 18.Raîche, Béland, Magis, Blais and Brochu

Barton, M. A. and Lord, F. M. (1981). An upper asymptote for the three-parameter logistic item-response model. Research bullelin 81-20. Princeton, NJ: Educational Testing Service.

Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord and M. Novick (Eds): Statistical theories of mental test scores. New York, NJ: Addison-Wesley.

Ferrando, P. J. (2004). Person reliability in personality measurement: an item response theory analysis. Applied Psychological Measurement, 28(2), 126-140.

Hulin, C. L., Drasgow, F., and Parsons, C. K. (1983). Item response theory. Homewood, IL: Irwin.

Levine, M. V., and Drasgow, F. (1983). Appropriateness measurement: validating studies and variable ability models. In D. J. Weiss (Ed.): New horizons in testing. New York, NJ: Academic Press.

Magis, D. (2007). Enhanced estimation methods in IRT. In D. Magis (Ed.): Influence, information and item response theory in discrete data analysis. Doctoral dissertation, Liège, Belgium: University de Liège.

REFERENCES / 1

Montréal, CSSE / CERA 2010 19.Raîche, Béland, Magis, Blais and Brochu

McDonald, R. P. (1967). Nonlinear factor analysis. Psyhometric Monographs, 15.

Raîche, G., and Blais, J.-G. (2003). Efficacité du dépistage des étudiants et des étudiants qui cherchent à obtenir un résultat faible au test de classement en anglais, langue seconde, au collégial. In J.-G. Blais, and G. Raîche (Ed.): Regards sur la modélisation de la mesure en en éducation et en sciences sociales. Ste-Foy, QC: Presses de l’Université Laval.

Strandmark, N. L. and Linn, R. L. (1987). A generalized logistic item response model parameterizing test score inappropriateness. Applied Psychological Measurement, 11(4), 355-370.

Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34, 273-286.

Trabin, T. E., and Weiss, D. J. (1983). The person response curve : fit of individuals to item response theory models. In D. J. Weiss (Ed.): New horizons in testing. New York, NJ: Academic Press.

REFERENCES / 2

Montréal, CSSE / CERA 2010 20.Raîche, Béland, Magis, Blais and Brochu

• Gilles Raîche– http://camri.uqam.ca

• Sébastien Béland– sebastien.beland.1@hotmail.com

• David Magis– david.magis@psy.kuleuven.be

• Jean-Guy Blais– http://www.griemetic.ca

• Pierre Brochu– p.broche@cmec.ca

CONTACTS