What does exam validity really mean?

Andrew MartinPurdue Pesticide Programs

What we’re going to doTake a brief look at the evolution of validity theory from 1950 to the present.

Review modern validity theory.Develop an analogy between test validation and pesticide enforcement.

Examine modern validity theory and its practical implications for pesticide applicator certification tests.

That’s your opinion. Everybody knows that. Our

expert says…!Sources1.:

Cureton, E. (1951)Cronbach, L. (1971)Messick, S. (1989)Shepard, L. (1993) Kane, M. (2004)AERA, APA, & NCME (1966, 1985, & 1999)

1. Full citations available upon request

Validity theory ca. 1950

(does this sound familiar?)It [validity] indicates how

well the test serves the purpose for which it is used.

Cureton (1951)

Validity defined ca. 1950

The validity of a test is an estimate of the correlation between raw test scores and the “true” (that is, perfectly reliable) criterion scores.

Cureton (1951)

Problems with the criterion model

The criterion model raises two questions:What counts as a good criterion measure?

How can the criterion measure be validated?

Validity theory ca. 1960

Validity information indicates the degree to which the test is capable of achieving certain aims…we may distinguish three of the numerous aims of testing:Content validityCriterion-related validityConstruct validity.

APA, AERA, & NCME (1966)

How was the trinitarian model supposed to work?

Validation of an instrument calls for an integration of many types of evidence. The varieties of investigation are not alternatives any one of which would be adequate.The investigations supplement one another.

Cronbach, 1971

Problems with the trinitarian model

The trinitarian model encouraged test developers to cherry-pick validity evidence.

Validity theory todayValidity is a unified concept…

Messick, 1989

It is a judgment of the extent to which theory and evidence support the way test scores are interpreted and used.

See also APA, AERA, & NCME, 1985 & 1999

Are there problems with a unified

approach to validity?[Current validity standards] do not help answer the question “How much evidence is enough?”

Shepard, 1993

Validity and validation

Theory is supposed to guide practice.

Modern validity theory assumes a guiding role when it is coupled with a conception of validation as a practical argument.

Steps in an argument-based approach to

validation1. Specify the the proposed interpretation and use of the test scores.

2. Assemble available evidence to support the plausibility of test score interpretation and use.

3. Evaluate doubtful assumptions in detail.

4. Repeat steps 2 and 3 until the argument is plausible (or rejected)

Kane, 2004

How is a validation argument like an

enforcement case?The test developer is supposed to:gather appropriate evidence, organize it in a compelling fashion,

and make a sound (plausible) case for how test scores will get interpreted and used.

What does this mean for me, the test developer (or how much evidence is enough)?

An argument-based approach to validation offers answers to two vexing questions:1. How do I get started?2. Where do I focus my resources?