Technical Issues Two concerns Validity Reliability
Slide 3
Data Collection Quiz 1 Answer the five questions on Quiz 1.Quiz
1
Slide 4
Data Collection Quiz 1 Answers Score you paper using the
following key 1. A 2. B 3. A 4. B 5. B
Slide 5
Data Collection Quiz 1 How well did you do? Should I use this
score as a part of your grade? Does this score indicate your level
as a graduate student? What we have here is a serious lack of
communication! Most students object strongly to using this score as
a part of their grade because it isnt fair. Most students object
strongly to being labeled bright or challenged on the basis of
their grade. Their reasoning is the test isnt fair it doesnt cover
material relevant to this course. Welcome to the technical world of
instrumentation.
Slide 6
Technical Issues Validity extent to which interpretations made
from a test score are appropriate Characteristics The most
important technical characteristic Situation specific Does not
refer to the instrument but to the interpretations of scores on the
instrument Best thought of in terms of degree
Slide 7
Technical Issues Validity (continued) Four types Content to
what extent does the test measure what it is supposed to measure
Item validity Sampling validity Determined by expert judgment
Slide 8
Technical Issues Validity (continued) Construct the extent to
which a test measures the construct it represents Underlying
difficulty defining constructs Estimated in many ways
Criterion-related Predictive to what extent does the test predict a
future performance Concurrent - to what extent does the test
predict a performance measured at the same time Estimated by
correlations between two tests
Slide 9
Technical Issues Validity (continued) Consequential to what
extent are the consequences that occur from the test harmful
Estimated by empirical and expert judgment Factors affecting
validity Unclear test directions Confusing and ambiguous test items
Vocabulary that is too difficult for test takers
Slide 10
Technical Issues Factors affecting validity (continued) Overly
difficult and complex sentence structure Inconsistent and
subjective scoring Untaught items Failure to follow standardized
administration procedures Cheating by the participants or someone
teaching to the test items
Slide 11
Technical Issues Reliability the degree to which a test
consistently measures whatever it is measuring Characteristics
Expressed as a coefficient ranging from 0 to 1 A necessary but not
sufficient characteristic of a test
Slide 12
Technical Issues Test reliability Stability consistency over
time with the same instrument Test retest Estimated by a
correlation between the two administrations of the same test
Equivalence consistency with two parallel tests administered at the
same time Parallel forms Estimated by a correlation between the
parallel tests
Slide 13
Technical Issues Test reliability (continued) Internal
consistency artificially splitting the test into halves Several
coefficients split halves, KR 20, KR 21, Cronbach alpha All
coefficients provide estimates ranging from 0 to 1