Week 2 Internal and External Validity. Internal Validity: Evaluating Your Experiment from the Inside Internal Validity The concept of internal validity

Week 2Week 2 Internal and External Internal and External

Validity Validity

Internal Validity: Evaluating Internal Validity: Evaluating Your Experiment from the Your Experiment from the InsideInside• Internal Validity

• The concept of internal validity revolves around the question of whether your IV actually caused any change that you observe in your DV.

Internal Validity: Evaluating Internal Validity: Evaluating Your Experiment from the Your Experiment from the InsideInside• Internal Validity

• The concept of internal validity revolves around the question of whether your IV actually caused any change that you observe in your DV.

• If you use adequate control techniques, your experiment should be free from confounding and you can, indeed, conclude that your IV caused the change in your DV.

Threats to Internal Threats to Internal ValidityValidity• History

• History refers to events that occur between the DV measurements in a repeated measures design.

Threats to Internal Threats to Internal ValidityValidity• History• Maturation

• Maturation refers to changes in participants that occur over time during an experiment.

Threats to Internal Threats to Internal ValidityValidity• History• Maturation

• Maturation refers to changes in participants that occur over time during an experiment.

• These changes could include actual physical maturation or tiredness, boredom, hunger, and so on.

Threats to Internal Threats to Internal ValidityValidity• History• Maturation• Testing

• Testing is a threat to internal validity that occurs because measuring the DV causes a change in the DV.

Threats to Internal Threats to Internal ValidityValidity• History• Maturation• Testing

• Testing is a threat to internal validity that occurs because measuring the DV causes a change in the DV.

• Campbell (1957) noted that if you take the same test more than once, scores on the second test may vary systematically from the first scores simply because you took the test a second time.

Threats to Internal Threats to Internal ValidityValidity• History• Maturation• Testing• Practice Effect

• A practice effect is a beneficial effect on a DV measurement caused by previous experience with the DV.

Threats to Internal Threats to Internal ValidityValidity• History• Maturation• Testing• Practice Effect• Reactive Measures

• Reactive measures are DV measurements that actually change the DV being measured.

Threats to Internal Threats to Internal ValidityValidity• History• Maturation• Testing• Practice Effect• Reactive Measures

• Reactive measures are DV measurements that actually change the DV being measured.

• Many attitude questionnaires are reactive measures. If we ask you a number of questions about how you feel about people of different racial groups, or about women’s rights, or about the president’s job performance, you can probably figure out that your attitude is being measured.

Threats to Internal Threats to Internal ValidityValidity• History• Maturation• Testing• Practice Effect• Reactive Measures• Instrumentation (Instrument Decay)

• Instrumentation is a threat to internal validity that occurs if the equipment or human measuring the DV changes the measuring criterion over time.

Threats to Internal Threats to Internal ValidityValidity• History• Maturation• Testing• Practice Effect• Reactive Measures• Instrumentation (Instrument Decay)• Statistical Regression

• Statistical regression occurs when low scorers improve or high scorers fall on a second administration of a test due solely to statistical reasons.

Threats to Internal Threats to Internal ValidityValidity• History• Maturation• Testing• Practice Effect• Reactive Measures• Instrumentation (Instrument Decay)• Statistical Regression• Selection

• If we choose participants in such a way that our groups are not equal before the experiment, we cannot be certain that our IV caused any difference we observe after the experiment.

Threats to Internal Threats to Internal ValidityValidity• History• Maturation• Testing• Practice Effect• Reactive Measures• Instrumentation (Instrument Decay)• Statistical Regression• Selection• Mortality

• Mortality can occur if experimental participants from different groups drop out of the experiment at different rates.

Threats to Internal Threats to Internal ValidityValidity• History• Maturation• Testing• Practice Effect• Reactive Measures• Instrumentation (Instrument Decay)• Statistical Regression• Selection• Mortality• Interactions with Selection

• Interactions with selection can occur when the groups we have selected show differences on another variable (i.e., maturation, history, or instrumentation) that vary systematically by groups.

Threats to Internal Threats to Internal ValidityValidity• History• Maturation• Testing• Practice Effect• Reactive Measures• Instrumentation (Instrument Decay)• Statistical Regression• Selection• Mortality• Interactions with Selection• Diffusion or Imitation of Treatments

• Diffusion or imitation of treatments can occur if participants in one treatment group become familiar with the treatment of another group and copy that treatment.

Protecting Internal Protecting Internal ValidityValidity• Two Approaches

• You can (and should) implement various control procedures.

Protecting Internal Protecting Internal ValidityValidity• Two Approaches

• You can (and should) implement the various control procedures.

• Use a standard procedure• Experimenters used standard procedures called

experimental designs to help ensure internal validity.

Protecting Internal Protecting Internal ValidityValidity• How Important is Internal Validity?

Protecting Internal Protecting Internal ValidityValidity• How Important is Internal Validity?

• It is the most important property of any experiment.• If you do not concern yourself with the internal validity

of your experiment, you are wasting your time.

External Validity: External Validity: Generalizing Your Generalizing Your Experiment to the OutsideExperiment to the Outside• The second type of evaluation that you must make of

your experiment involves external validity.

External Validity: External Validity: Generalizing Your Generalizing Your Experiment to the OutsideExperiment to the Outside• The second type of evaluation that you must make of

your experiment involves external validity. • When you consider external validity, you are asking a

question about generalization.

External Validity: External Validity: Generalizing Your Generalizing Your Experiment to the OutsideExperiment to the Outside• External Validity

• A type of evaluation of your experiment that asks whether your experimental results apply to populations and situations that are different from those of your experiment.

External Validity: External Validity: Generalizing Your Generalizing Your Experiment to the OutsideExperiment to the Outside• External Validity

• A type of evaluation of your experiment that asks whether your experimental results apply to populations and situations that are different from those of your experiment.

• Generalization• Applying the results from an experiment to a different

situation or population.• In essence, we would like to take our results beyond the

narrow confines of our specific experiment.

• Generalization is an important aspect for any science.

External Validity: External Validity: Generalizing Your Generalizing Your Experiment to the OutsideExperiment to the Outside• There are three customary types of generalization in

which we are interested:• Population generalization• Environmental generalization• Temporal generalization


which we are interested:• Population generalization

• Applying the results from an experiment to a group of participants that is different and more encompassing than those used in the original experiment.

• Environmental generalization• Temporal generalization


which we are interested:• Population generalization• Environmental generalization

• Applying the results from an experiment to a situation or environment that differs from that of the original experiment.

• Temporal generalization


which we are interested:• Population generalization• Environmental generalization• Temporal generalization

• Applying the results from an experiment to a time that is different from that when the original experiment was conducted.

Threats to External Threats to External Validity (Based on Validity (Based on Methods)Methods)• Here is a summary of factors relating to external

validity from Campbell and Stanley (1966):


validity from Campbell and Stanley (1966):• Interaction of Testing and Treatment

• A threat to external validity that occurs when a pretest sensitizes participants to the treatment yet to come.

• Occurs for the pretest-posttest control group design• Because of a pretest, your participants’ reaction to the

treatment will be different.


validity from Campbell and Stanley (1966):• Interaction of Testing and Treatment• Interaction of Selection and Treatment

• A threat to external validity that can occur when a treatment effect is found only for a specific sample of participants.

• Occurs when the effects that you demonstrate hold true only for the particular groups that you selected for your experiment.

• Treatment interaction becomes greater as it becomes more difficult to find participants for your experiment (Campbell & Stanley, 1966).


validity from Campbell and Stanley (1966):• Interaction of Testing and Treatment• Interaction of Selection and Treatment• Reactive Arrangements

• A threat to external validity caused by an experimental situation that alters participants’ behavior, regardless of the IV involved.

• We cannot be sure that the behaviors we observe in the experiment will generalize outside that setting because the artificial conditions of the experiment do not exist in the real world.


validity from Campbell and Stanley (1966):• Interaction of Testing and Treatment• Interaction of Selection and Treatment• Reactive Arrangements• Demand Characteristics

• Features from the experiment that inadvertently lead participants to respond in a particular manner.

• Demand characteristics make generalizations difficult because it is not clear from a set of research findings whether the participants are responding to an experiment’s IV, its demand characteristics, or both.


validity from Campbell and Stanley (1966):• Interaction of Testing and Treatment• Interaction of Selection and Treatment• Reactive Arrangements• Demand Characteristics• Multiple-Treatment Interference

• A threat to external validity that occurs when a set of findings results only when participants experience multiple treatments in the same experiment (repeated measures designs).

Threats to External Threats to External Validity (Based on Our Validity (Based on Our Participants)Participants)• The Infamous White Rat

Threats to External Threats to External Validity (Based on Our Validity (Based on Our Participants)Participants)• The Infamous White Rat

• If you are interested in the behavior of subhumans, generalizing from rats (and pigeons) to all other animals may be a stretch.

• If you are interested in generalizing from animal to human behavior, there are certainly closer approximations to humans (and pigeons) in the animal kingdom.

Threats to External Threats to External Validity (Based on Our Validity (Based on Our Participants)Participants)• The Infamous White Rat• The Ubiquitous College Student


• Researchers who want to conduct human research turn to a ready, convenient source of human participants – students in introductory courses (a technique referred to as convenience sampling).


• Researchers who want to conduct human research turn to a ready, convenient source of human participants – students in introductory courses (a technique referred to as convenience sampling).

• Convenience Sampling• A researcher’s sampling of participants based on ease

of locating the participants; often does not involve true random selection.

Threats to External Threats to External Validity (Based on Our Validity (Based on Our Participants)Participants)• The Infamous White Rat• The Ubiquitous College Student• The “Opposite” or “Weaker” or “Inferior” or “Second”

Sex


Sex• All four of these derogatory labels have been applied to

women at various points in time.


Sex• All four of these derogatory labels have been applied to

women at various points in time.• The supposed inferiority of women has carried over into

some psychological theories.• Freud’s theories• Erikson’s theory of psychosocial crises (“Eight Stages of

Man”)


Sex• Carol Tavris’s (1992) thesis is that “despite women’s gains

in many fields in the last twenty years, the fundamental belief in the normalcy of men, and the corresponding abnormality of women, has remained virtually untouched” (p. 17)


Sex• Even the Rats and Students Were White


Sex• Even the Rats and Students Were White

• Just as history has failed to record the accomplishments of many women throughout time, it has largely ignored the accomplishments of African Americans and other minority groups.

• When we conduct research and make generalizations, we should be cautious that we do not exclude minority groups from our considerations.


Sex• Even the Rats and Students Were White• Even the Rats, Students, Women, and Minorities Were

American

Threats to External Threats to External Validity (Based on Our Validity (Based on Our Participants)Participants)• The Infamous White Rat• The Ubiquitous College Student• The “Opposite” or “Weaker” or

“Inferior” or “Second” Sex• Even the Rats and Students Were

White• Even the Rats, Students, Women,

and Minorities Were American

Threats to External Threats to External Validity (Based on Our Validity (Based on Our Participants)Participants)• The Infamous White Rat• The Ubiquitous College Student• The “Opposite” or “Weaker” or “Inferior” or

“Second” Sex• Even the Rats and Students Were White• Even the Rats, Students, Women, and

Minorities Were American• In the mid-1960’s, social researchers started taking

culture and ethnicity more seriously. The field of cross-cultural ______(fill in the blank—nursing, etc.) has evolved from those changes that began in the 1960s.

Threats to External Threats to External Validity (Based on Our Validity (Based on Our Participants)Participants)• Cross-Cultural Research

• A branch of research whose goal is to determine the universality of research results.

• Ethnocentricity• Other cultures are viewed as an extension of one’s own.

The Devil’s Advocate: The Devil’s Advocate: Is External Validity Always Is External Validity Always Necessary?Necessary?

• Mook (1983) pointed out four alternative goals of research that do not stress external validity:

The Devil’s Advocate: The Devil’s Advocate: Is External Validity Always Is External Validity Always Necessary?Necessary?• Mook (1983) pointed out four alternative goals of

research that do not stress external validity:• We may merely want to find out if something can happen

(not whether it actually happens).



(not whether it actually happens).• We may be predicting from the real world to the lab –

seeing a phenomenon in the real world, we think it will operate in a certain manner in the lab.



(not whether it actually happens).• We may be predicting from the real world to the lab –

seeing a phenomenon in the real world, we think it will operate in a certain manner in the lab.

• If we can demonstrate that a phenomenon occurs in a lab’s unnatural setting, the validity of the phenomenon may actually be strengthened.


• Mook (1983) pointed out four alternative goals of research that do not stress external validity:• We may merely want to find out if something can happen (not

whether it actually happens).• We may be predicting from the real world to the lab – seeing a

phenomenon in the real world, we think it will operate in a certain manner in the lab.

• If we can demonstrate that a phenomenon occurs in a lab’s unnatural setting, the validity of the phenomenon may actually be strengthened.

• We may study phenomena in the lab that don’t even have a real-world analogy.

The Devil’s Advocate: The Devil’s Advocate: Is External Validity Always Is External Validity Always Necessary?Necessary?• Replication

• An additional scientific study that is conducted in exactly the same manner as the original research project.

• When we replicate an experimental finding, we are able to place more confidence in that result.


• Replication• An additional scientific study that is conducted in exactly

the same manner as the original research project.• When we replicate an experimental finding, we are able to

place more confidence in that result.• Replication With Extension

• An experiment that seeks to confirm (replicate) a previous finding but does so in a different setting or with different participants or under different conditions.

ReliabilityReliability

• Reliability • Refers to the extent that the test or inventory is

consistent in its evaluation of the same individuals over repeated administrations.

• The greater the similarity between scores produced by the same individuals on repeated administrations, the greater the reliability of the test or inventory.

ProceduresProcedures

• Reliability (assessment)• Test-retest procedure


• Reliability (assessment)• Test-retest procedure

• The test is simply given a second time and scores from the two tests are compared; the greater the similarity, the higher the reliability


• Reliability (assessment)• Test-retest procedure• Split-Half technique


• Reliability (assessment)• Test-retest procedure• Split-Half technique

• Involves dividing a test or inventory into two halves or subtests and then administering them to the same individuals on different occasions or administering the entire test and then splitting it into two halves.

Documents

Week 2 Internal and External Validity. Internal Validity: Evaluating Your Experiment from the Inside Internal Validity The concept of internal validity