Multidimensional Item Response Theory Modelswith General and Specific Latent Traits

for Ordinal Data

Irene Martelli

Tutor: Prof. Stefania Mignani

Bologna, 15 Maggio 2014

Outline

I Aim and structure of the thesis

I Item Response Theory: a brief introduction

I Unidimensional Graded Response Model

I Multidimensional Graded Response Models

I Model implementation using OpenBUGS

I Simulation study and results

I Application to real data

I Conclusions

Aim and structure of the thesis

I Aim of the thesis: to propose a Markov chain Monte Carloestimation of the multidimensional additive item response theorymodel for ordinal data, where the latent traits are allowed tocorrelate.

I Structure of the thesis:

Ch.1 - An introduction to Item Response Theory (IRT)

Ch.2 - Multidimensional IRT (MIRT) models: a review

Ch.3 - Bayesian estimation of MIRT models

Ch.4 - MIRT graded response models with complex structures

Ch.5 - Simulation study

Ch.6 - Application to real data

Ch.7 - Conclusions

Item Response Theory: Aims and Fields of Application

I IRT has the final aim to measure abilities and attitudes ofindividuals through the responses on a number of test items.

I By using IRT methods we wish to determine the position ofthe individual along some latent dimension. This latent traitrepresents the unobservable characteristic of the individuals.

I The latent trait is often called ability, because of the intenseuse of IRT in educational and psychological fields. Newapplications in medicine.

I New interesting application: extension of IRT models toSocial Sciences (well-being, satisfaction, ...)

I Where the available information is typically collected byquestionnaires with multiple ordinal items (i.e. Likert scales)

IRT: the concept of model and estimation methods

Prob( response | latent variable(s))Item withcategorical responses

ExploratoryConfirmatoryApproach:

Multidimensional modelsUnidimensional models

dimensions:of latentNumber

ThreeTwoOne

item characteristics:describing theNumber of parameters

Polytomous responsesBinary responses

of the data:Structure

LogitProbitresponse and the examinee’s ability:Probability model used to link the

(MCMC techniques)

approachBayesian

methodsLikelihoodMaximum

MethodsEstimation

Different models: literature review

MCMC estimation of IRT Models

Different models: literature review

MCMC estimation of IRT Models

Samejima unidimensional model for graded responses (1)

I The unidimensional graded response model (Samejima, 1969) is usedfor data collected on n subjects who have responded to each of pordinal items.

I The response Yij of the i-th individual to the j-th item can take valuesin the set {1, . . . ,Kj}.

I With πijk we denote the probability that the i-th subject will select thek-th category on item j.

I In order to construct the probability πijk we first need to consider thecumulative probabilities Pijk that are expressed as a function of thepredictor ηijk, which depends on θi and ξjk: ηijk = f(θi, ξjk)

I We focus on:

i. Normal ogive models: Pijk = Φ(ηijk)⇒ Φ−1(Pijk) = ηijk

ii. ξjk = (αj , κjk), where αj and κjk are the discrimination andthreshold parameters, respectively.

Samejima unidimensional model for graded responses (2)

Each item has Kj − 1

thresholds κj1, . . . , κj,Kj−1 that

satisfy the order constraint

κj1 < · · · < κj,Kj−1.

Pijk = P (Yij ≤ k|θi, αj , κjk) = Φ(κjk − αjθi) =

∫ κjk−αjθi

−∞φ(z)dz

I The probability πijk that the i-th subject will select the k-th categoryon item j can thus be written as:

πij1 = Pij1πijk = Pijk − Pi,j,k−1 for k = 2, . . . ,Kj − 1πijKj = 1− Pi,j,Kj−1

Towards multidimensional models

I Advantages of unidimensional models:

i. Fairly simple mathematical forms;

ii. Numerous examples of applications;

iii. Evidence of robustness to violations of assumptions.

I However, the actual interactions between persons and testitems are not simple as implied by unidimensional models:

i. Examinees are likely to bring more than a single ability to bearwhen responding to a particular test item;

ii. The problems posed by test items are likely to requirenumerous skills and abilities to determine a correct solution.

I There is a need for more complex IRT models that moreaccurately reflect the complexity of the interactions betweenexaminees and test items ⇒ Multidimensional IRT models.

MIRT Models: underlying latent structures

Additive structureHierarchical structuresMultiunidimensional structure

Item j

I Let consider:

I n individuals;I a test consisting of p ordinal items, divided into m subtests;I the existence of m latent abilities θi = (θ1i, . . . , θmi)

′.

I We focus on the following models:

I Multiunidimensional Graded Response Model, where items in eachsubtest characterize a single ability.

I Additive Graded Response Model, where each item measures ageneral and a specific ability directly.

The multiunidimensional GRM

Pvijk = P (Yij ≤ k|θvi, αvj , κjk)

= Φ(κjk − αvjθvi) =

∫ κjk−αvjθvi

−∞

1√2πe−t

2/2dt

I αvj and κjk are item parameters representing the itemdiscrimination and the threshold between categories k and k + 1;

I θvi represents the i-th examinee ability in the v-th ability dimension;

I Probability πvijk that the i-th examinee will select the k-th categoryon item j in subtest v:

πvij1 = Pvij1

πvijk = Pvijk − Pv,i,j,k−1 for k = 2, . . . ,Kj − 1

πvijKj = 1− Pv,i,j,Kj−1

The additive GRM

Pvijk = P (Yij ≤ k|θ0i, θvi, α0vj , αvj , κjk)

= Φ(κjk − α0vjθ0i − αvjθvi) =

∫ κjk−α0vjθ0i−αvjθvi

−∞

1√2πe−t

2/2dt

I For each item j of the subtest v: α0vj , αvj and κjk;

I θ0i represents the i-th overall ability and θvi represent the specificabilities (with v = 1, . . . ,m);

I The probability πvijk that the i-th examinee will select the k-thcategory on item j in subtest v is obtained as in themultiunidimensional GRM;

I Both general and specific abilities are involved in determining theresponse probability by following a compensatory approach.

Identification issues

I In general, Bayesian item response models can be identified(Fox, 2010):

I By imposing restrictions on the hyperparameters.

I Via a scale transformation in estimation procedure.

I Then, for identification purposes, a multivariate normal priordistribution with a fixed correlation structure is assumed forabilities:

θi = (θ0i, θ1i, . . . , θmi)′ , θi ∼ Nm+1(0,Σ), i = 1, . . . , n

with Σ covariance matrix with diagonal elements being 1and off-diagonals elements being the ability correlations.

Additive GRM implementation using OpenBUGS (1)

I Categorical distribution is assumed for responses:

Yij |• ∼ dcat(πvij1, . . . , πvijKj ) for j = 1, . . . , p

i = 1, . . . , n

P (Yij = k|•) = π[k=1]vij1 · π

[k=2]vij2 . . . π

[k=Kj ]vijKj

I Model specification according to additive GRM:

Pvijk = Φ(κjk − α0vjθ0i − αvjθvi) v = 1, . . . ,m

PvijKj = 1 j = 1, . . . , p

k = 1, . . . ,Kj − 1

πvij1 = Pvij1

πvijk = Pvijk − Pv,i,j,k−1 for k = 2, . . . ,Kj

Additive GRM implementation using OpenBUGS (2)

I Normal prior distributions are assumed for item parameters:

αvj ∼ N(0, 1)(αvj > 0) v = 1, . . . ,m

α0vj ∼ N(0, 1)(α0vj > 0) j = 1, . . . , p

κ∗jk ∼ N(0, 1) k = 1, . . . ,Kj − 1

{κjk, . . . κj,Kj−1} = ranked{κ∗j1, . . . , κ∗j,Kj−1}1

I Multivariate normal prior distribution is assumed for abilities:

θi = (θ0i, θ1i, . . . , θmi)′ , θi ∼ Nm+1(0,Σ), i = 1, . . . , n

with Σ covariance matrix with diagonal elements being 1and off-diagonals elements being the ability correlations.

1In order to satisfy the order constraint on thresholds.

Simulation study (1)

I The aim is to evaluate the parameter recovery of themultiunidimensional and the additive graded response modelsunder several conditions.

I We consider the bidimensional case m = 2.

I Ability correlations structures:

0.3 0.2 1 0.4 1 0.2 1 0.4 0.3

=B 0 0 1 0 1 0 1 0 0

=AAdditive:0.4 1 1 0.4=B0 1

1 0=AMultiunidimensional:

I General simulation conditions:

I 30,000 iterations (15,000 burn-in) to ensure stationarity;

I Two chains (to compute the R Gelman and Rubin diagnosticstatistic for convergence);

I Sample sizes: n = 500 and n = 1000;

I 10 replications for each simulation.

Simulation study (2)

I Simulation conditions:

Additive model (Blocks 1 and 2)Multiunidimensional model (Block 1)

I For each of the 16th scenarios: median RMSE and median absolutebias for each set of item parameters, estimated ability correlations.

I Main results:

I Item parameters and ability correlations are well reproduced.

I Higher biases are noticed when sample sizes are smaller (as expected).

I Stimulating observation: relation between number of test items (p) andnumber of categories (K).

Simulation study results: Additive model (1)

Simulation study results: Additive model (2)

Simulation study results: estimated ability correlations

Perception of tourism impact on the quality of life (1)

Ravenna

Forlì-CesenaRimini

Rep. diSan Marino

Aim: to investigate Romagna and San Marino

residents’ perception and attitudes toward the

tourism industry.

Perception of benefitsPerception of costsResidents Tourism

Industry

I Data analyzed: subset of results of a research on the subjectivewell-being, conducted by the University of Bologna (in the end of 2010).

“Criticity levels”Thresholds (item) parameters

towards tourism”“General attitudetourism benefits”

“Perception of

tourism costs”“Perception of

the tourism industry to distinct aspect of

on a set of items referredRespondents’ opinionsObserved variables

Perception of tourism impact on the quality of life (2)

I Sample size: n = 794.

I Answers on a 7-point scale, from “strongly disagree” to “strongly agree”.

I We focus (among the others) on items of the questionnaire concerningresidents’ evaluations about costs and benefits of the tourism industry:

Items on benefits

Subtest 1

I Personal information (age, gender, nationality, residence and occupation)were also collected.

Tourism impact: results for the multiunidimensional GRM

I Public services (B3), job opportunities (B4) and cultural activities(B5) are the most informative items on the perception of the tourismadvantages (high discrimination parameters).

I Traffic (C4) and pollution (C5) are the most informative items on theperception of the tourism costs (high discrimination parameters).

I The main and immediate advantages of tourism are identified in theeconomic support (B1) (low thresholds’ parameters related to thehighest categories).

I The cost of life (C1) can be regarded as a marginal negative aspect oftourism (low thresholds’ parameters related to the highest categories).

I Estimated correlation between the latent traits: r12 = −0.37, theperception of a high economic advantage of tourism is associated with anegative perception of tourism costs.

Tourism impact: results for the additive GRM

I The general trait is estimated on the basis of the perception of eitherbenefits and costs but conditionally on the specific effects of the twotraits.

I Specific discrimination parameters and thresholds’ parameters: resultsare similar to the multiunidimensional model.

I Public services (B3), cultural activities (B5), traffic (C4) andpollution (C5) principally influence the general residents’ attitudetowards tourism (high general discrimination parameters).

I Estimated correlation between the latent traits: r01 = 0.03,r02 = 0.18 and r12 = −0.62, the correlation between the benefitlatent trait and the attitude is very low, and slightly higher is theestimated correlation between the cost latent trait and the generalattitude.

I The additive model (DIC= 8945) fits the data better than themultiunidimensional model (DIC=10950).

Heterogeneity in residents’ perceptions and conclusions

Normalized mean perceptionand attitude scores

Conclusions and future works

I Fields of application characterized by ordinal manifest variables: needingto avoid the loss of information due to data transformation processes.

I Actual interactions between respondents and test items are complex:multidimensional latent structures with correlated traits.

I GRMs proposed are complex: MCMC estimation procedure, within aBayesian approach (overtake the problem of analytically intractableexpressions).

I Quite easy implementation: attractive to non-statistical specialists.

I Limitations about the application study: choice of the prior distributions,sample size, number of item categories, test and subtests lengths areimportant issues that have to be considered.

I Future works: to increase the number of latent dimensions, to extend thework to different latent structure, to introduce the presence of covariatesin model specification.

Main references I

I Albert J.H. (1992): Bayesian Estimation of Normal Ogive Item ResponseCurves Using Gibbs Sampling, Journal of Educational Statistics, Vol. 17,N.3, pp. 251-269.

I Albert J.H. and Chib S. (1993): Bayesian Analysis of Binary andPolychotomous Response Data, Journal of the American StatisticalAssociation, Vol. 88, N. 422, pp. 669-679.

I Beguin A., Glas C.A.W. (2001): MCMC Estimation and Some Model-FitAnalysis of Multidimensional IRT Models, Psychometrika, Vol. 66,pp. 541-562.

I Edwards M.C. (2010): A Markov Chain Monte Carlo Approach toConfirmatory Item Factor Analysis, Psychometrika, Vol. 75, N. 3,pp. 474-497.

I Fox J.P. (2005): Multilevel IRT Using Dichotomous and PolytomousResponse Data, British Journal of Mathematical and StatisticalPsychology, Vol. 58, pp. 145-172.

Main references II

I Fox J.P. (2010): Bayesian Item Response Modeling: Theory andApplications, Springer, New York.

I Patz R.J., Junker B.W. (1999a): A Straightforward Approach to MarkovChain Monte Carlo Methods for Item Response Models, Journal ofEducational and Behavioral Statistics, Vol. 24, N. 2, pp. 146-178.

I Patz R.J., Junker B.W. (1999b): Applications and Extensions of MCMCin IRT: Multiple Item Types, Missing Data, and Rated Responses, Journalof Educational and Behavioral Statistics, Vol. 24, N. 4, pp. 342-366.

I Samejima F. (1969): Estimation of Latent Ability Using a ResponsePattern of Graded Scores, (Psychometric Monograph No. 17).Richmond, VA: Psychometric Society.

I Sheng Y. (2008): A MATLAB Package for Markov Chain Monte Carlo witha Multi-Unidimensional IRT Model, Journal of Statistical Software, Vol.28, Issue 10.

Main references III

I Sheng Y. (2010): Bayesian Estimation of MIRT Models with General andSpecific Latent Traits in MATLAB, Journal of Statistical Software, Vol. 34,Issue 3.

I Sheng Y., Headrick (2012): A Gibbs Sampler for the MultidimensionalItem Response Model, ISRN Applied Mathematics, Vol. 2012.

I Sheng Y., Wikle C. K. (2007): Comparing Multiunidimensional andUnidimensional Item Response Theory Models, Educational andPsychological Measurement, Vol. 67, pp. 899-919.

I Sheng Y., Wikle C. K. (2009): Bayesian IRT Models IncorporatingGeneral and Specific Abilities, Behaviormetrika, Vol. 36, pp. 27-48.

Thank you for your attention!