AN ALGORITHM FOR TESTING UNIDIMENSIONALITY AND CLUSTERING ITEMS IN RASCH MEASUREMENT

Rudolf Debelak & Martin Arendasy

Outline

1. Aims of this study2. PCA and Parallel analysis based on

tetrachoric correlations3. The proposed algorithm

1. Procedures2. Statistical test

4. Simulation study5. Empirical study6. Discussion

Aims of Study

Clustering items: exploratory approach to identify items scales with strict criterion

Testing unidimensionality: confirmatory approach to test a unidimensional item set whether yielding a single cluster

Literature Review

Commonly used procedures to test unidimensionality: PCA and EFA Applying to binary data -> based on

tetrachoric correlations Correct number of components/factors ->

parallel analysis Cluster analysis

Cluster analysis (from Wikipedia)

Cluster analysis or clustering is the task of assigning a set of objects into groups (called clusters) so that the objects in the same cluster are more similar (in some sense or another) to each other than to those in other clusters.

Hierarchical Cluster Analysis is based on the core idea of objects being more related to nearby objects than to objects farther away.

Measure of similarity (distance)

The Basic Structure of the Procedure

Test Item Triplets

Expand Item Set

O3

A3

On

Function f

Maximum

NOT Maximum

An+1

On+1

Function f

NOT Maximum

Maximum

Ok

p less than 0.5

Assessing the Model Fit

“a function f” is a global fit statistics in this study

The test can be used to evaluate whether the set of items, as a whole, fits the model. ( Suarez-Falcon & Glas, 2003) First-order statistics (R1): violation of the

property of monotone increasing and parallel item characteristic curves

Second-order statistics (R2): violation of the assumptions of unidimensionality and local independence

R1C Statistics

R1C can be regarded as being asymptotically chi-square distribution with (k – 1)(k – 2) degrees of freedom.

Simulation Study

Aim: whether able to detect and reconstruct subsets of items that fit the Rasch model.

Two subsets of items which fit the Rasch model

Six variables were manipulated (next slide)

10,000 replications were carried out with eRm package which employed the CML estimation method.

Variables

1. The distribution of the item parameters (normal, uniform)

2. The standard deviations of the item and person parameters

3. The size of the person sample (250, 500, 1000)

4. The size of the item set (10, 30, 50)

5. The correlation between the person parameters (0.0, 0.5)

Type Person

Item

A 1.0 0.5

B 1.5 0.5

C 2.0 1.5

D 2.5 1.5

Data Analysis

The proposed algorithm The PCA based on tetrachoric

correlations with parallel analysis (95th percentile eigenvalues)

Results (Proposed Method)

Results (PCA with Parallel Analysis)

Sample Size in PCA

Sample size small than 250 (test length = 10 items) would result in large numbers of indefinite matrices of tetrachoric correlations, making the application of PCA impossible. (Parry & McArdle, 1991; Weng & Cheng, 2005)

Empirical Study

The Basic Intelligence Functions (IBF; Blum et al., 2005)

Subtests and Items: verbal intelligence functions (2; 12+15), numerical intelligence functions (2), long-term memory (1; 15), visualization (1; 13).

Between 281 and 284 persons

Data Analysis

Using Raschcon for scale-construct with the proposed algorithm.

Andersen likelihood ratios (eRm), fit statistics and PCA on residuals (Winsteps) were calculated.

PCA and parallel analysis of tetrachoric correlations were performed.

all subtests were analyzed separately

Results

Proposed algorithm: all subtests identical to the respective subtest; fit to Rasch model.

Andersen tests: fit at .01 level; 3 out of 4 subtests unfit at .05 level.

Mean square in/outfit: all ranged [1.33, 0.65]

PCA on residuals: long-term memory(2.0), others (<1.4)

PCA: long-term memory (2 components)

Discussion

A new algorithm was presented and compared with another method PCA of tetrachoric correlations.

R1c statistics: when sample size is large and correlation between latent traits was low, and better in small scales, large variances of item and person parameters.

More preferable than PCA of tetrachoric when sample size is small and scales are large.

Further Studies

1. Systematic comparison with PCA of tetrachoric

2. Involve more tests for model assumptions

3. Compare test statistics for the fit of Rasch model

4. Conduct other IRT models

For the conditions of higher correlation and small sample size, is it possible to find a cut-point (correction) to improve the use of this method?