Item response modeling of paired comparison and ranking data

paired comparison and ranking data

• paired comparison– n(n-1)/2 pairs

• Ranking– n!– A special case of paired comparison when no

intransitive pattern• Thurstone’s model (1927)– Utility (property of item)

Ranking

• Ypair = 1 if ti – tk > 0• Design matrix:• Utility distribution:– Case 3: – Case 5:

• Separate person parameter out of random error:– Note they think of loading parameter as item

attributes (e.g., male actors versus female athletes)

Pairwise comparison

• Add intransitive error:• Then

• Identification:– Origin and scale: N(0,I) for η– Rotation: -------------------------– Additions due to pairwise design:• Fix loading parameters of a statement to 0.• Fix one mean parameter of a statement to 0.• Fix one unique variance of a statement to 1.

• Identification:– At least n=5, 6, 7 for m = 1, 2, 3 (number of

dimension). Why?– If not, require more constraints! What it is?• Constrain All the covariance matrix (I have tried this!)

Thurstonian IRT model• Recall• Do substitution

• Take n=3 as example:

Parameter estimation

• MML may be infeasible because ICCs are conditionally dependent for Thurstone IRT model.

• Limited information method is applicable by using Mplus.

• But d.f. should be modified for ranking data:

Item characteristic function

• Recall Y* =

• Re-expressed as

• Note

Latent trait estimation, information functions, and reliability estimation

• Locally independence is violated!• MAP• Information function

• Reliability – 1.– 2.

•

Simulation studies

• To estimate• Sample size: 200, 500, 1000 • Item size: 6, 12• Equal or unequal variance:

•

Results

Results

How "close" are the observed values to those which would be expected under the fitted

model?

MAP

Vocational interest (pairwise comparison)

• Unrestricted thresholds: p=.046, RMSEA=.016• Equal w: p=.000, RMSEA=.025• Constrained thresholds: p=.000, RMSEA=.025• Reliability: .62 (theoretical); .43 (empirical) due to shrunken MAP

RealisticInvestigativeArtisticConventionalSocialEnterprising

Vocational interest (pairwise comparison)

Work motivation (ranking data)

• Chi-square fit index: p=.000, RMSEA=.062

Work motivation (ranking data)

• Reliability: .74 (theoretical); .76 (empirical)

Discussion

• Locally dependence– Using MCMC

• Discrimination: positively vs negatively worded• Multiple traits• Forced-choice design