Validity: Introduction

Reliability and ValidityReliability

Low High

Validity Low

High

•

••

•

•

•

•

• •

••

•

•••

•••

•• •• ••

Types of Error in Repeated Measurements

True value

¦ ¦ ¦ ¦ ¦ ¦ ¦ ¦ ¦ ¦ ¦ ¦ ¦ ¦ ¦

¦ ¦ ¦ ¦ ¦ ¦ ¦ ¦ ¦

The Validation Sequence

“Does it measure what it’s supposed to measure?”

1. First, define what you want to measure (conceptual basis)

2. Select indicators or items that represent that topic (content validity; a sampling issue)

3. Check that items are clear, comprehensible and relevant (face validity, “sensibility”).

4. This produces a pool of items ready for item analysis stage, which involves administering the test and analyzing responses.

Validation Sequence (2): Internal structure

Item analysis refers to a series of checks on the performance of each item. Some fall under the heading of reliability, some validity. Faulty items are discarded or replaced. Analyses include:

• Item distributions & missing values: an item that does not vary cannot measure anything

• Correlations among items, maybe with factor analysis

• Item response theory (IRT) analyses

Validation Sequence (3):External associations

• Criterion validation, if a “gold standard” exists. Sensitivity & specificity are normal statistics.

• Correlational evidence, leading to construct validation, where there is no single, clear gold standard. Correlations are the normal statistic.

• Correlations often divided into convergent and discriminant coefficients, according to hypothesized associations

These analyses tend to use the entire test administered to selected samples; inadequate performance leads back to basic

design

Validation Sequence (4):Group Discrimination

Once you show that the test correlates with other measures as intended, its actual performance is evaluated in rating groups of respondents. Analyses generally use representative samples

• “Known groups” (can it distinguish well from sick; similar to criterion validity)

• Sensitivity to change over time (relevant to an evaluative measure); responsiveness.

• Do scores show ceiling or floor effects?

Conclusion• Validation is rarely complete. Many instruments

continue to be checked for validity 20 years after their invention. Times change, phrasing makes old items obsolete, etc.

• It is long and expensive. Basic test development and validation may take 3 - 5 years: it’s not a thesis project.

• Remember: validity is about the interpretation of scores. It is a relative concept: a test is not valid or invalid, but only valid or not for a particular application.

• The viagara principle: a test intended for one purpose may prove good for an unanticipated application.