Embed Size (px)
Citation preview
Investigating item quality using fit and other indicatorsRobert CoeRasch User Group, Durham, 18 March 2016
@ProfCoe#RUD2016
∂
Quality
1. Utility for measurementa) Fit with measurement model (Rasch)b) Alignment with intended construct interpretations
2. Utility for learninga) Alignment with intended learning aimsb) Value of diagnostic information
i. For teachersii. For students
c) Reinforcement, retrieval
2
∂
Model fit
§ INFIT/OUTFIT§ Discrimination
– IRT parameter/index– Item-measure correlation– 27% rule (Kelley, 1939)
§ H -coeff of homogeneity (Loevinger, 1948; Mokken, 1971; Mokken & Lewis, 1982)
§ Other fit statistics?
3
∂
Problems with INFIT/OUTFITKarabatsos (2000) JAppMeas
§ ‘Residual fit statistics’ are confounded: parameters are estimated from data; fit stats test fit between data and parameters …
§ Interpretation of INFIT/OUTFIT is sample dependent
§ They do a poor job of identifying misfit
4
∂
5
∂
∂
7
ItemStatisticsNumberofresponses 7,685Maximumscore 1Meanscoreonitem 0.12
Itemdifficulty(Raschmeasure) 1.98INFIT(meansq) 0.98OUTFIT(meansq) 3.80IRTDiscriminationparameter 0.92
Item-measurecorrelation(actual) 0.42Item-measurecorrelation(expected) 0.46
Percentmatchmodel(observed) 91Percentmatchmodel(expected) 90
∂
3. b) Infit and outfit indicate model fit?
infit 1.07, outfit 1.15 infit 1.04, outfit 1.08 infit 1.06, outfit 1.27
∂
WINSTEPS category probability curves can be misleading
3.002.001.000.00-1.00-2.00-3.00
1.20
1.00
0.80
0.60
0.40
0.20
0.00
-0.20
Logistic probability of correct response v Person ability
Smoothed local average proportion correct v Person ability
Equal density distribution plot v Person ability
Infit = 1.31
∂
10
∂
11
∂
12
∂
13
∂
14
∂
Enlargement 8→12, so ?→18
15
12
14
Missing
∂
Algebra
16
∂
Question: ALG04EEMultn+5by4
ItemStatisticsNumberofresponses 7,841Maximumscore 1Meanscoreonitem 0.11
Itemdifficulty(Raschmeasure) 1.94INFIT(meansq) 0.92OUTFIT(meansq) 0.6682IRTDiscriminationparameter 1.0816
Item-measurecorrelation(actual) 0.4338Item-measurecorrelation(expected) 0.3969
Percentmatchmodel(observed) 90.664Percentmatchmodel(expected) 90.128
17
∂
18
Outfitforpersonsforwhothe itemis:0<p<0.05 0.05<p<0.2 0.2<p<0.8 0.8<p<0.95 0.95<p<1
thresholdVeryhard Hard About right Easy Veryeasy
1 Outfit 0.42 0.84 1.05 0.79 0.03N 3997 2512 1288 39 5
“Multiply n+5 by 4”
∂
19
4n+20
4(n+5)
missing
4n+5
n+20
5n+5
“Multiply n+5 by 4”
∂
Question: ALG04AAAdd4to8
ItemStatisticsNumberof responses 7,841Maximumscore 1Meanscoreonitem 0.74
Itemdifficulty (Raschmeasure) -2.86INFIT(meansq) 1.2693OUTFIT(meansq) 1.627IRTDiscriminationparameter 0.6577
Item-measurecorrelation(actual) 0.4636Item-measurecorrelation(expected) 0.5731
Percentmatchmodel(observed) 78.117Percentmatchmodel(expected) 83.017
20
∂
21
“Add 4 to 8”
∂
22
12missing
8+4
“Add 4 to 8”
∂
Question: ALG04FFMult3nby4
ItemStatisticsNumberofresponses 7,841Maximumscore 1Meanscoreonitem 0.34
Itemdifficulty(Raschmeasure) -0.11INFIT(meansq) 1.5233OUTFIT(meansq) 1.8801IRTDiscriminationparameter 0.1397
Item-measurecorrelation(actual) 0.3335Item-measurecorrelation(expected) 0.5512
Percentmatchmodel(observed) 65.338Percentmatchmodel(expected) 78.106
23
∂
24
“Multiply 3n by 4”
∂
25
“Multiply 3n by 4”
missing
12n
7n
4x(3n)
