Introduction to psychometrics

Christophe Lalanne

Fall 2011

OutlineWhat is psychometrics?

Key concepts

Measurement and instrument

Latent variable models

Items and scales

Measurement models

Reliability and validity

“It is rather surprising that systematic studies of human abilities were not under-taken until the second half of the last century. . .An accurate method was availablefor measuring the circumference of the earth 2,000 years before the first systematicmeasures of human ability were developed.” —Nunnally and Bernstein, 1994

What is psychometrics?Psychometrics has evolved as a subfield of psychology to become thescience of measurement of unobservable individual characteristics. Itencompasses

• the construction and analysis of measurement instruments;

• the development of theoretical approaches to measurement.

It is sometimes opposed to clinimetrics which encourages the use ofclinical expertise instead of formal statistical models, or even biomet-rics (de Vet et al., 2003).

Psychometrics

Educometrics Sociometrics

IRTFA

SEM

HLM

MDS 3Mode

CA

LogLin

Foometrics“If Foo is a science then Foo

often has both an area Foometricsand an area Mathematical Foo.

Mathematical Foo appliesmathematical modeling tothe Foo subject area, while

Foometrics develops and studiesdata analysis techniques for

empirical data collected in Foo.Each of the social and behaviouralsciences has a form of Foometrics,

although they may not all use a namein this family.” (de Leeuw, 2006)

Some landmarksG. T. Fechner(1801–1887)

W. Wundt(1832–1920)

E. H. Weber(1795–1878)

S. S. Stevens(1906–1973)

G. von Békésy(1899–1972)

R. Likert(1903–1981)

L. L. Thurstone(1887–1955)

F. Galton(1822–1911)

J. McKeen Cattell(1860–1944)

C. Spearman(1863–1945)

L. L. Thurstone(1887–1955)

K. Pearson(1857–1936)

G. Rasch(1901–1980)

F. M. Lord(1912–2000)

A. Jensen(1923–)

L. J. Cronbach(1916–2001)

Key conceptsWe, human beings, like to make either relative or absolute judgments.Although it is often easy to tell whether a given object is taller thananother object, it is harder to tell whether someone is more proficientin a given discipline than another person. Likewise, it is difficultto assess groupwise opinions by asking a random set of questions.Assessing one’s preference for a set of objects is also a difficult task.Why?

Mostly because the preceding points call for a process that allows torate and scale the above object of measurement. Moreover, judg-ments elicited this way will rather lead to subjective measures, unlikemost physical measurements.

never − almost never ?≡ always − almost alwaysUltimately, we may want to build a measurement tool that shares thesame characteristics as the ones shown below, or at least one thatallows us to locate and compare individuals’ responses or objects’properties using some metrical system.

Measurement and instrumentAn instrument is used to relate, or ‘map’, something observed inthe real world to some other thing measured as part of a theory.The former is called a manifest variable, while the latter is usuallysubsumed under the term latent variable or factor.Measurement is best understood as the task of assigning number tocategories (Stevens, 1946), in order for the researcher to “provide areasonable and consistent way to summarize the responses that peoplemake to express their achievements, attitudes, or personal points ofview through instruments such as attitude scales, achievement tests,questionnaires, surveys, and psychological scales.” (Wilson, 2005)

Latent variable modelsA good overview of the use of LVMs in biomedical research is providedby Rabe-Hesketh and Skrondal, 2008.For a more general framework, we can consider how latent and man-ifest variables relate each other: (Bartholomew and Knott, 2011)

Manifest variables

Metrical Categorical

Latent variablesMetrical Factor analysis Latent trait analysis

Categorical Latent profile analysis Latent class analysis

The four building blocks (Wilson, 2005)• A Construct map features a coherent and substantive definition for

the content of the construct which is composed of an underlyingcontinuum (for ordering respondents and/or items responses).

• Items design deals with the standardized construction of itemsthat are supposed to stimulate responses, assimilable to observa-tions about the construct.

• The Outcome space is the set of well-defined categories, finite andexhaustive, ordered, context-specific, and research-based.

• A Measurement model is needed in order to relate the scoredoutcomes from the items design and the outcome space back tothe construct that was the original inspiration of the items.

Self-reported measures What is an item?An item constitutes the fundamental unit of measurement, and it canbe considered as a criteried way to measure a psychological constructotherwise unobservable (e.g. reading fluency, specific cognitive ability,quality of life, personality traits).

Example: Below are nine items from the MOS 36-item short form,scored 1 (‘All of the Time’) to 6 (‘None of the Time’) points.

From items to scaleAggregating items together leads to a scale, which is often confoundedwith the dimension it is assumed to reflect. Provided it only mea-sures a single construct (unidimensionality), we can assign scores topatterns of observed responses (e.g., summated-scale score) to inferone’s location on the underlying latent trait, but more generally rankitems or individuals on this scale.Two desired properties of a scale are unidimensionality and localindependence. That is,

• The instrument should measure a single construct which accountsfor one’s response to a specific item given his location on the latenttrait (and not other factors);

• If the latent variable is maintained at a given level, all the itemresponses should be independent.

Quoting Falissard, 2006, “A set of items is unidimensional if there ex-ists a variable (often called a latent variable, as this variable may notbe observed) which ‘explains’ all the correlations observed betweenthe items”, but conceiving a single construct as underlying a broadconcept (e.g., depression) might be misleading, see e.g. (Borsboom,2006). Factor analytic methods can be used to study the dimen-sionality of an instrument, and provide much better information thansingle-number summary which are often inappropriately used (e.g.,Cronbach’s alpha). Local independence is required in many models,including IRT or LCA models.

Examples of multidimensional instruments• The HADS includes two scales (2 × 7 items, 21 points max.) for

exploring anxiety and depression-related symptoms (Zigmond andSnaith, 1983 and Herrmann, 1997).

• The NEOPI-R is a personality inventory (240 or 60 items) rely-ing on the five-factor model (extraversion, agreeableness, consci-entiousness, neuroticism, and openness; 6 facets) (McCrae andJohn, 1992 and McCrae and Costa, 2004).

• The MOS-HIV for HIV/AIDS includes 35 items assessing: generalhealth perceptions, pain, physical functioning, role functioning,social functioning, energy/fatigue, mental health, health distress,cognitive function, and quality of life (Wu et al., 1997a, 1997b).

A list of existing HRQL instruments is available on http://www.proqolid.org/.

Two common measurement modelsThe classical test theory assumes that there exists a true score, de-fined as the expected response of an individual following repeatedindependent admistration of the same test (error-free measure). Butwe only have access to

observed score = true score + some error

Although it has long been considered as a principled method in mea-surement, it suffers from different flaws, well reviewed in (Borsboom,2006). Connected conceptual issues are reliability of test scores anditem analysis. Here, a respondent having a ‘higher amount of thelatent trait’ will have a higher score, usually computed as the sum ofhis responses.

The CTT approach includes few but simple mathematical formula-tion, with straightforward estimation of model parameters; it alsoyields scoring rules of practical interest.

IRT models on the contrary rely on a mathematical model wherebythe probability of endorsing an item depends on both item and personlocation (parameters) on the latent trait. For every item in a scale, aset of properties (item parameters) are estimated. The item slope ordiscrimination parameter describes how well the item performs in thescale in terms of the strength of the relationship between the itemand the scale. The item difficulty or threshold parameter(s) identifiesthe location along the construct’s latent continuum where the itembest discriminates among individuals.

A well-known example is the Rasch model which deals with binaryresponses and assumes that all items are equally discriminative. IRTmodels are more flexible and do not assume an fixed axiomaticallyfixed relationship between the observed and true score. They allowfor the joint scaling of person and item parameters when mapping therelationship between latent traits and responses to test items.The use of IRT model is particularly interesting when the objectiveis to develop bank of calibrated items (Reeve et al., 2007), to delivertargeted and adaptive testing measures (Rebollo et al., 2010), orto study measurement invariance across groups of patients (Teresi,2006).

Another illustrationThe following example is taken from Partchev, 2004.Let us consider an item like

What is the area of a circle having a radius of 3 cm?

1. 9.00 cm2

2. 18.85 cm2

3. 28.27 cm2

The first of the three options is possibly the most naive one, thesecond is wrong but implies more advanced knowledge (the area isconfused with the circumference), and the third is the correct one.

A probabilistic account of item responses

COSMIN Taxonomy (Mokkink et al., 2010)

Interpretability

Content validity

facevalidity

Criterion validityb

Construct validity

Structural validity

Hypotheses testing

Cross-cultural validity

Validity

Responsiveness

Responsiveness

Internal consistency

Measurementerrora

Reliabilitya

Reliability

a (test-retest, inter-rater, intra-rater)b (concurrent validity, predictive validity)

The many facets of validity“ Validation of instruments is the process of determining whether there aregrounds for believing that the instrument measures what it is intended tomeasure, and that it is useful for its intended purpose. (Fayers and Machin,2000)”

Let’s consider the following definitions: (Falissard, 2008)

• Content validity reflects the adequacy of the domains or dimen-sions spanned by the items;

• Criterion validity demonstrates that scales have empirical associa-tion with external criteria, such as gold standards or other instru-ments purpoted to measure equivalent concepts;

• Construct validity relates each inter-items and item-scale relation-ships from a theoretical point of view.

Convergent and discriminant validities are both subsumed in the gen-eral and theoretical concept of construct validity. A critical appraisalof the concept of validity can be found in (Zumbo, 2007 and Bors-boom et al., 2004, 2009).A questionnaire is valid to the extent test scores are interpretatedin relation to the construct that the questionnaire purpots to assess.An alternative (and more elegant) definition of construct validity isthat it “is a property of measurement instruments that codes whetherthese instruments are sensitive to variation in a targeted attribute”(Borsboom et al., 2009).

Some caveats about clinical validityIn clinical setting, the validity of a medical diagnosis requires a clearaetiology. However, most of functional diagnosis of mental health-related disorders (e.g. personality disorder, schizophrenia) are definedin a circular fashion: The diagnosis is made on the basis of symptomsand the symptoms are accounted for by the diagnosis.Likewise, although most psychiatrists will agree on what depression is,demonstrating categorically that clinical depression differs from dys-phoria or everyday unhappiness is nearly impossible (Pilgrim, 2009).Finally, the problems of valid and reliable case identification in psy-chiatric epidemiology remain unsolved because no physical cause aregenerally associated to the disease under consideration.

Different types of reliabilityThe sources of scores variability may be very different, e.g. (Dunn,2000)

• repeated assessment of several subjects by the same rater or clin-ician;

• alternative assessment of a given subject by different raters;• alternative administration of the same questionnaire or a parallel

form;• use of different subscales of a single questionnaire to infer one’s

performance;

Reliability analysis aims at quantifying these variations, in order toprovide an idea of the error of measurement.

Reliability vs. significanceA note of caution: One must distinguish between reliability and sig-nificance: (Thompson, 2003)

• ‘statistical’ significance evaluates the probability or likelihood ofthe sample results, with reference to a reference population wherethe null is exactly true; ‘practical’ and ‘clinical’ significance areclosely related concepts underlying the extent to which sampleresults diverge from the null (as measured by effect size statistics)or the way treated patients may not be distinguished from controlor normal ones.

• However, ‘statistical’, ‘practical’, and ‘clinical’ significance allstand on the assumption that scores are meaningful and reliableindicators of individuals’ performance. . .

Construction of a questionnaire

content validity construct validityconcourant validity

scores reliability

responsiveness

test construction test administration follow-up!eld testing

face validity

large-scale testing

phase II phase III phase IV

correspondence with RCT

validation

Bibliography1 Nunnally, J. and Bernstein, I. (1994). Psychometric Theory . 3rd edition McGraw-Hill.2 de Vet, H., Terwee, C. and Bouter, L. (2003). Clinimetrics and psychometrics: two sides of the

same coin. Journal of Clinical Epidemiology, 56, 1146–1147. DOI: 10.1016/j.jclinepi.2003.08.010.3 de Leeuw, J. (2006). R in Psychometrics and Psychometrics in R. In The R User Conference,

UseR! 2009. R Foundation for Statistical Computing.4 Stevens, S. (1946). On the theory of scales of measurement. Science, 103, 677–680.5 Wilson, M. (2005). Constructing measures. An item response modeling approach. Taylor &

Francis Group.6 Rabe-Hesketh, S. and Skrondal, A. (2008). Classical latent variable models for medical research.

Statistical methods in medical research.7 Bartholomew, D. and Knott, M. (2011). Latent Variable Models and Factor Analysis: A Unified

Approach. 3rd edition Wiley.8 Falissard, B. (2006). The unidimensionality of a psychiatric scale: a statistical point of view.

International Journal of Methods in Psychiatric Research, 8(3), 162–167. DOI: 10.1002/mpr.66.9 Borsboom, D. (2006). The attack of the psychometricians. Psychometrika, 71(3), 425–440.

DOI: 10.1007/s11336-006-1447-6.10 Zigmond, A. and Snaith, R. (1983). The hospital anxiety and depression scale. Acta Psychiatry

Scandinavia, 67(6), 361–370.

11 Herrmann, C. (1997). International experiences with the hospital anxiety and depression scale–areview of validation data and clinical results. Journal of Psychosomatic Research, 42(1), 17–41.

12 McCrae, R. and John, O. (1992). An introduction to the five-factor model and its applications.Journal of Personality, 60(2), 175–215.

13 McCrae, R. and Costa, P. (2004). A contemplated revision of the neo five-factor inventory.Personality and Individual Difference, 36(3), 587-596.

14 Wu, A., Hays, R., Kelly, S., Malitz, F. and Bozzette, S. (1997a). Applications of the medicaloutcomes study health-related quality of life measures in hiv/aids. Quality of Life Research,6(6), 531-554.

15 Wu, A., Revicki, D., Jacobson, D. and Malitz, F. (1997b). Evidence for reliability, validity andusefulness of the medical outcomes study hiv health survey (mos-hiv). Quality of Life Research,6(6), 481–493.

16 Reeve, B., Hays, R., Bjorner, J., Cook, K. and Crane, P. et al. (2007). Psychometric evaluationand calibration of health-related quality of life item banks. Medical Care, 45(5), S22–S31.PMID: 17443115.

17 Rebollo, P., Castejón, I., Cuervo, J., Villa, G. and García-Cueto, E. et al. (2010). Validation ofa computer-adaptive test to evaluate generic health-related quality of life. Health and Qualityof Life Outcomes, 8, 147. PMID: 21129169.

18 Teresi, J. (2006). Different approaches to differential item functioning in health applications:Advantages, disadvantages and some neglected topics. Medical Care, 44, S152–S170.

19 Partchev, I. (2004). A visual guide to item response theory. Friedrich-Schiller-UniversitätJena. URL: website.

20 Mokkink, L., Terwee, C., Patrick, D., Alonso, J. and Stratford, P. et al. (2010). The COSMINstudy reached international consensus on taxonomy, terminology, and definitions of measure-ment properties for health-related patient-reported outcomes. Journal of Clinical Epidemiology,63, 737–745. PMID: 20494804.

21 Fayers, P. and Machin, D. (2000). Quality of life. The assessment, analysis and interpretationof patient-reported outcomes. Wiley.

22 Falissard, B. (2008). Mesurer la subjectivitÃľ en santÃľ. Perspective mÃľthodologique etstatistique. Masson.

23 Zumbo, B. (2007). Validity: Foundational issues and statistical methodology. In Rao, C. andSinharay, S., editors, Handbook of Statistics, Vol. 26: Psychometrics, pages 45-79. ElsevierScience B.V.: The Netherlands.

24 Borsboom, D., Mellenbergh, G. and Heerden, J. v. (2004). The concept of validity. Psycho-logical Review, 111, 1061–1071.

25 Borsboom, D., Cramer, A., Kievit, R., Scholten, A. and Franic, S. (2009). The end of con-struct validity. In Lissitz, R., editor, The concept of validity: Revisions, new directions, andapplications. Information Age Publishers.

26 Pilgrim, D. (2009). Key Concepts in Mental Health. 2nd edition Sage.27 Dunn, G. (2000). Statistics in Psychiatry . Hodder Arnold.28 Thompson, B., editor (2003). Score Reliability. Contemporary Thinking on Reliability issues.

Sage Publications.

Made with ConTEXt version 2011.05.18 18:04 (pdfTEX version 140),

psychometrics.tex, 285c54e on 2011/10/14