Upload
nush1590
View
216
Download
0
Embed Size (px)
Citation preview
8/13/2019 Psy 407 Validity
1/43
8/13/2019 Psy 407 Validity
2/43
Determining Validity
There are several ways to measure validity. The mostcommonly addressed include:
- Face Validity
- Construct & Content Validity
- Convergent & Divergent Validity
- Predictive Validity
- Discriminant Validity
8/13/2019 Psy 407 Validity
3/43
Validity
Refers to measuring what we intend to measure.
If math and vocabulary truly represent intelligence then
a math and vocabulary test might be said to have high
validity when used as a measure of intelligence.
8/13/2019 Psy 407 Validity
4/43
Face Validity Is the extent to which it is self-evident that a scale is measuring
what it is suppose to measure.For Example - you might look at a measure of math ability, read through
the questions, and decide that yep, it seems like this is a good measure of
math ability
It would clearly be weak evidence because it is essentially a
subjective judgment call. Just because it is weak evidence doesn't
mean that it is wrong. We need to rely on our subjective judgment
throughout the research process.
For example- suppose you were taking an instrument reportedly
measuring your attractiveness, but the questions were asking you to
identify the correctly spelled word in each list. Not much of a link
between the claim of what it is supposed to do and what it actuallydoes.
8/13/2019 Psy 407 Validity
5/43
8/13/2019 Psy 407 Validity
6/43
Face Validity (cont.)
In all cases, face validity is not based on empirical research
evidence.
When used alone, face validity provides very weak support for the
overall validity of a scale.
8/13/2019 Psy 407 Validity
7/43
Face Validity (cont.)
Possible Advantage of face validity... If the respondent trusts the test and believes it is
measuring what it should be measuring, they may
provide more useful, thoughtful, and accurate answers. E.g., if a nursing test seems to measure constructs not
associated with nursing, the respondent may become
frustrated and answer poorly or without much thought
8/13/2019 Psy 407 Validity
8/43
Possible Disadvantage of face validity...
If the respondent knows what information weare looking for, they might try to bend &
shape their answers to what they think wewant.
i.e., fake good or fake bad
8/13/2019 Psy 407 Validity
9/43
Content Validity
Does the test contain items from the desired content
domain?
Based on assessment by experts in that content domain.
Is especially important when a test is designed to have
low face validity.
Is generally simpler for tests of ability than for
psychological constructs to demonstrate content
validity
.
8/13/2019 Psy 407 Validity
10/43
For Example- Easier for math experts to agreeon an item for an algebra test than it is forpsych experts to agree whether or not an item
should be placed in a personality measure
8/13/2019 Psy 407 Validity
11/43
Content Validity (cont.)
Basic Procedure for Assessing Content Validity:
1. Describe the content domain
2. Determine the areas of the content domain that aremeasured by each test item
For Example - In developing a nursing licensure exam, experts on
the field of nursing would identify the information and issues
required to be an effective nurse and then choose (or rate) items
that represent those areas of information and skills.
8/13/2019 Psy 407 Validity
12/43
Content Validity (cont.)
Lawshe (1975) proposed that each rater should respond to
the following question for each item in content validity:
Is the skill or knowledge measured by this item
1. Essential
2. Useful but not essential
3. Not necessary
We need to have a high proportion of essential
responses from the raters
8/13/2019 Psy 407 Validity
13/43
Construct Validity
Construct Validity basically refers to the general validity
of the measurement tool.
Does the instrument measure the construct that it is
intended to measure?
There is no statistical test that will provide an absolute
measure for construct validity. Therefore, construct
validity is never proven, it can only be supported.
8/13/2019 Psy 407 Validity
14/43
A) Assess how well the test predicts the relevantconstruct
Assume that we have developed theJohnson
Inventory of Aggression (JAS) A person who scores high on the JAS should
be more likely to engage in aggressive behaviorin any situation relative to those who score lowon the JAS
8/13/2019 Psy 407 Validity
15/43
B) Compare the new measure to an existing, valid measure (i .e, the
JAS should correlate highly with an existing valid measure of
aggression) .
Although a valid measure will exist but a new scale is being
created that will have some advantage over the older measure.
Advantages of new measure A) more consistent with current theory B) shorter and more accurate
-
8/13/2019 Psy 407 Validity
16/43
Sometimes existing valid measures dontexist. That is often why the new scale isbeing created in the first place.
8/13/2019 Psy 407 Validity
17/43
8/13/2019 Psy 407 Validity
18/43
Tyson (1997) contends that there is a clearracial divide between Blacks and Whites intheir perceptions of the prevalence of
discrimination in America Crocker (1999) found that when compared to
White Americans, Black Americans (across allsocioeconomic levels) were much more likely
to believe in conspiracy theories that suggestthat the government engaged in organizedefforts to harm Blacks
8/13/2019 Psy 407 Validity
19/43
Monteith & Spicer (2000)showed thatBlack negative attitudes towards
Whites generally reflected reactionsto perceived racism (eg., Theyalways have negative thoughts about
Blacks)
8/13/2019 Psy 407 Validity
20/43
A) Media and experiences
8/13/2019 Psy 407 Validity
21/43
83% of Blacks have experienced racism in their day-to-daylife and 68% experiencing racism during healthcare.(Peters, 2006)
70% of Blacks reported being treated unfairly by strangers,34% reported unfair treatment by helping professions, and31% were called derogatory names connected to race.(Pieterse & Carter, 2007).
67% of Black participants felt that their life would havebeen different over the past year if they had notexperienced these racist events
8/13/2019 Psy 407 Validity
22/43
However, despite popular opinions,Blacks are not a monolithic group!
There is variation among Blacks regarding their
racism expectations involving Whites
8/13/2019 Psy 407 Validity
23/43
In his book Country of Strangers,Shiplersuggests that Blacks in America range fromthose who tend to see racism in everyencounter with a white to those who trymightily not to see it all (p. 448).
Yet there was no direct measureof variation in Black beliefs inWhite racism
To address the limitations in the racism studies
8/13/2019 Psy 407 Validity
24/43
To address the limitations in the racism studiesliterature,Johnson & Lecci (2003) developedthe Johnson-Lecci Scale of Black antiWhite bias
This scale was based on responses and lifeexperiences of approximately 450 Black collegestudents
The scale had 20 items and four subscales
A major subscale was the ingroupstigmatization and discrimination expectation(I.e., the expectation that the typical White personwill discriminate against Blacks)
8/13/2019 Psy 407 Validity
25/43
***All questions were answered on a four point scale (1-strongly disagree, 4-strongly agree)
I believe that most whites really do support the ideas andthoughts of racist political groups
I believe that most whites really believe that blacks aregenetically inferior.
I believe that most whites would discriminate against blacks, ifthey could get away with it
I believe that most of the negative actions of whites towards
blacks are due to racist feelings I believe that most whites would harm blacks if they could get
away with it.
I believe that most whites think that they are superior to blacks.
8/13/2019 Psy 407 Validity
26/43
However, scales are useless if they dontpredict!
8/13/2019 Psy 407 Validity
27/43
The JLS has been shown to predict:
A) the probability of perceiving racism in anambiguously racist scenario (Johnson & Lecci, 2003) Restaurant example, airline choice
B) the number of a Black persons White friends
(Johnson & Lecci, 2003) C) the probability that a Black person will confront a
racist person (Johnson et al., 2006)
D) racial preferences regarding a mental health
counselor (Ferguson et al., 2008) E) the probability that of prosocial responses
towards a White person-in-need (Johnson et al.,2008)
F) bias against lighter skinned Blacks who were inneed ohnson et al., under review
8/13/2019 Psy 407 Validity
28/43
What would be a comparable measure inPacific Island/Fiji culture?
What might be some of the items on the scale?
8/13/2019 Psy 407 Validity
29/43
Discriminant Validity The statistical assessment of Construct Validity
Does the instrument show the right pattern of interrelationships
with other instruments.
Discriminant Validity has two parts:
Convergent Validity
Divergent Validity
8/13/2019 Psy 407 Validity
30/43
Convergent & Divergent Validity
Convergent Validity: the extent to which the scale
correlates with measures of the same or related concepts.
e.g.,
A new scale to measure Assertiveness should
correlate with existing measures of Assertiveness, andwith existing measures of related concepts like
Independence.
8/13/2019 Psy 407 Validity
31/43
Divergent Validity: the extent to which does notcorrelate with measures of unrelated or distinctconcepts.
e.g., An assertiveness scale shouldnot correlate with measures of aggressiveness.
8/13/2019 Psy 407 Validity
32/43
This type of validity measures the relationship between thepredictor and the criterion, and the accuracy with which the
predictor is able to predict performance on the criterion.
What is the actual association between a test to predictjob
performance and actualjob performance
If the test has high criterion validity, then it should be close
Criterion-Related Validity (cont.)
8/13/2019 Psy 407 Validity
33/43
HOW THIS TYPE IS ESTABLISHED:Criterion-related validity can be either concurrent or
predictive. The distinguishing factor is the time when criterion
and predictor data are collected.
Concurrent- criterion data are collected before or at the same
time that the predictor is administered.
Predictive- criterion data are collected after the predictor is
administered.
Criterion-Related Validity (cont.)
8/13/2019 Psy 407 Validity
34/43
Concurrent Validity This type of validity indicates the correlation between the
predictor and criterion when data on both were collected ataround the same time.
Is used to determine a persons current status.
For ExampleIf a person seems depressed, they should score fairly
high on a depression inventory given the same day
8/13/2019 Psy 407 Validity
35/43
Predictive Validity This type of validity also indicates the correlation between
the predictor (X) and the criterion (Y). However, criteriondata are collected after predictor data are obtained. In other
words, this method determines the degree, that X can
accurately predict Y
For Example - giving high school juniors the ACT test for
admission to a university.
The test is the predictor and first semester grades in college are
the criterion. If the correlation is large, this means the ACT
is useful for predicting future grades.
8/13/2019 Psy 407 Validity
36/43
Predictive Validity (cont.)
The extent to which scores on the scale are related to,and predictive of, some future outcome that is of
practical utility.
e.g., If higher scores on the SAT are positively correlated with
higher G.P.A.s and visa versa, then the SAT is said to
have predictive validity.
The Predictive Validity of the SAT is mildly supported by
the relation of that scale with performance in graduate
school.
8/13/2019 Psy 407 Validity
37/43
Predictive Validity
A predictive validity study consists of two basic steps:
1.Obtain test scores from a group of respondents, but do not
use the test in making a decision.
2.At some later time, obtain a performance measure for those
respondents, and correlate these measures with test scores to
obtain predictive validity.
A Test Must Be Valid -
8/13/2019 Psy 407 Validity
38/43
When evaluating test to real-life predictions, even very
modest correlations of r= .02 or .03 can be of
considerable importance. For example, the impact of
chemotherapy on breast cancer survival is r= .03.
In selection, hiring, and counseling contexts, current
interpretations suggest that correlations as low as r= .02
or .03 are meaningful, with many psychological (andmedical test) assessments and real life criteria falling in
the r = .10 to .30 level, and a few rising beyond that level.
A Test Must Be Valid -
Predictive Validity
8/13/2019 Psy 407 Validity
39/43
Resilience-----an ability to recover from oradjust easily to misfortuneor change
Sh ld i t lli b di ti f th
http://www.merriam-webster.com/dictionary/misfortunehttp://www.merriam-webster.com/dictionary/misfortune8/13/2019 Psy 407 Validity
40/43
Should intelligence be predictive of theprobability of joining a cult?
More importance should a measure of
resilience be predictive of (i.e., correlate with)joining a cult?
8/13/2019 Psy 407 Validity
41/43
FACTORS THAT INFLUENCE VALIDITY
Inadequate sample
Items that do not function as intended
Improper arrangement/unclear directionsToo few items for interpretation
Improper test administration
Scoring that is subjective
8/13/2019 Psy 407 Validity
42/43
RELATIONSHIP BETWEEN
RELIABILITY AND VALIDITY
Reliability means
nothing when theproblem is Validity.
Reliability caps
validity
8/13/2019 Psy 407 Validity
43/43
Reliability and validity are two different standards used to gage the
usefulness of a test. Though different, they work together.
It would not be beneficial to design a test with good reliability that
did not measure what it was intended to measure. The inverse,
accurately measuring what you desire to measure with a test that isso flawed that results are not reproducible, is impossible.
Reliability is a necessary requirement for validity. This means that
you have to have good reliability in order to have validity. Reliability
actually puts a cap or limit on validity, and if a test is not reliable, it
can not be valid.
- from the MSCEIT Manual
Relationship Between Reliability & Validity