INTERNAL VALIDITY AND BASIC RESEARCH DESIGN. Internal Validity the approximate truth about inferences regarding cause-effect or causal relationships

INTERNAL VALIDITY AND BASIC RESEARCH DESIGN

Internal Validity

the approximate truth about inferences regarding cause-effect or causal relationships

only relevant in studies that try to establish a causal relationship Or wherever the language of the study infers a causal

relationship

Internal Validity

internal validity is a “zero-generalizability concern”

Establishing Cause and Effect

3 Criteria for a causal relationship to exist: temporal precedence covariation of cause and effect no plausible alternative explanations

Example: Does drinking beer make you happy?


1. temporal precedence beer comes before happiness

2. covariation of cause and effect if beer then happy, if no beer then not happy the more beer then the more happy


3. no plausible alternative explanations

“Never drink alone”…social causes?

To drink, you need to have cash…it’s an economic difference?

Drinkers experience smoke too…it’s all about the ciggies?

When you drink you go to the loo more often…something to do with bladder swelling/emptying?a

re t

hese

pla

usi

ble

?

Identifying Possible Threats

Possible studies testing whether beer makes you happy?

give beer

measure happiness

XX OOgive beer

measure happiness

XX OO

measure happiness

OO

KNR 164

KNR 164

give beer

measure happiness

XX OO

measure happiness

OOKNR 164 (8am)

OOOOKNR 164 (9am)

Internal Validity Threats

single-group threats

multiple-group threats

social interaction threats

Single Group Threats

history threat some event (or series of events) occurring during the

study that affects the outcome maturation threat

observed effect due to normal maturation or development of subjects


testing threat (pre-post only) taking the pre-test affects performance on post-test

instrumentation threat (pre-post only) change in tests (observations, measures) is

responsible for change in outcome


mortality threat non-trivial or non-random dropout affects the results

regression threat it is not the treatment causing the effect, but rather

the posttest scores are simply moving back toward the population mean (up or down)

Solving Single Group Threats

Add a control group the only difference between the control group and the

treatment group should be the presence or absence of the treatment

sometimes this means you need multiple control groups

Multiple Group Threats

all are selection bias threats the critical question is: “were the groups equal (on

the measure of choice) at the start of the study?”






selection-history threat some event (or series of events) occurring between

the pre- and post-test affects the groups differently selection-maturation threat

there are differential rates of normal development between the pre-test and post-test for the groups


selection-testing threat the effect of taking the pre-test varies between groups

selection-instrumentation threat changes in the tests across time vary between the

groups


selection-mortality threat differential non-random dropout occurs between the

groups, so that post-test scores differ selection-regression threat

differential rates of regression to the mean cause post-test scores to differ across groups.

Solving Multiple Group Threats

randomly assign to groups random assignment ensures no systematic difference

between groups (and therefore decreases the possibility that any observed affect is due to selection bias)

there have to be a reasonable number of participants in each group for the random assignment to work can’t just randomly assign two people to two groups and

expect them to be equal!





Social Interaction Threats

diffusion or imitation of treatment control group learns about treatment somehow and

does it on their own control group becomes more like treatment group

compensatory rivalry control group knows about treatment and becomes

competitive with the experimental group, which affects how they respond on the posttest

control group becomes more like treatment group


resentful demoralization basically the opposite of compensatory rivalry upon finding out about the treatment the control

group members become discouraged, angry, etc. probably perform worse on post-test than they would

have compensatory equalization of treatment

the researcher somehow affects the outcome by treating the control group favorably in some way


experimenter bias the belief in the treatment in some way causes the

experimenter to behave differently around the two groups, thus turning the findings into a self-fulfilling prophecy

Reducing Social Interaction Threats

1. Do the researchers claim in any way that the relationships they uncovered in their study are casual in nature (e.g., the intervention caused changes in the dependent variable, the treatment lead to group differences in the dependent variable)?

2. If so, do you think that the causal inference made by the researchers is reasonable or justifiable given the design of the study? [This is the key Internal Validity question]

The following questions should help answer Question #2. 1. Consider the type of design (e.g. are there multiple groups, how were subjects

assigned to groups, is there a pre and post-test).2. Does their design provide evidence of temporal precedence between the

cause and effect (i.e. did the cause happen before the effect)?3. Does their design provide evidence of covariation between the cause and

effect (i.e. is there an effect when the cause is present, but not when it is not)?4. Are there plausible alternative explanations for the relationship between the

cause and effect? If so, what are they (use the potential single group, multiple group, and or social interaction threats to internal validity as a guide)?

1. Do the researchers claim in any way that the relationships they uncovered in their study are casual in nature (e.g., the intervention caused changes in the dependent variable, the treatment lead to group differences in the dependent variable)?

2. If so, do you think that the causal inference made by the researchers is reasonable or justifiable given the design of the study? [This is the key Internal Validity question]

The following questions should help answer Question #2. 1. Consider the type of design (e.g. are there multiple groups, how were subjects

assigned to groups, is there a pre and post-test).2. Does their design provide evidence of temporal precedence between the

cause and effect (i.e. did the cause happen before the effect)?3. Does their design provide evidence of covariation between the cause and

effect (i.e. is there an effect when the cause is present, but not when it is not)?4. Are there plausible alternative explanations for the relationship between the

cause and effect? If so, what are they (use the potential single group, multiple group, and or social interaction threats to internal validity as a guide)?

Guiding Questions for Critiquing the Internal Validity of Research

Practice

identify potential internal validity issues (using the threats) for each of the following studies

1.The purpose of the study was to examine the relationship between exercise participation and happiness. A total of 300 students served as subjects, all of whom were recruited from a personal fitness class at a small liberal arts school in the mid-western portion of the United States. Students were led through a 30 minute traditional step aerobics class during one of their regularly scheduled class periods. At the end of the class, the students completed the 5-item Happiness subscale from the General Mood Inventory. Results indicated that the students were extremely happy, thus supporting the positive effect of exercise on happiness.

1.The purpose of the study was to examine the relationship between exercise participation and happiness. A total of 300 students served as subjects, all of whom were recruited from a personal fitness class at a small liberal arts school in the mid-western portion of the United States. Students were led through a 30 minute traditional step aerobics class during one of their regularly scheduled class periods. At the end of the class, the students completed the 5-item Happiness subscale from the General Mood Inventory. Results indicated that the students were extremely happy, thus supporting the positive effect of exercise on happiness.

2. A teacher wanted to determine whether allowing students to choose the topics covered in introduction to exercise science class would translate into better understanding of the subject. At the beginning of the semester students in both sections of KNR 164 were given a 100-item exam assessing general knowledge about exercise science. During the course of the semester, students who signed up for Section 1 of the class (MWF 8am) were given a choice by the teacher about which topics they would like to cover during the class period. Students who signed up for Section 2 of the class (MWF 6pm) followed a standard curriculum. At the end of the semester, students in both sections were given different versions of a 200-item exam covering key concepts of exercise science. Results indicated that the students in the experimental class (i.e., those who were allowed to choose the topics) scored significantly higher on the end of the semester exam, suggesting that the type of teaching style affected learning.

3. The purpose of the study was to test whether a mental imagery training package resulted in better athletic performance. Freshman football players from ISU were randomly assigned to a control and experimental group. During a private session at the beginning of the season the players assigned to the experimental group were given 2 hours of instruction on using mental imagery, and were encouraged to use the technique for 10 minutes before each practice and game during the season. The players assigned to the control group also had a private meeting at the beginning of the season where they watched the movie “Remember the Titans”. At the end of the season, the amount of playing time and the head coach’s rating of overall performance for each athlete were compared to see if the imagery training was effective. Athletes who received the imagery training were found to perform better than those who did not receive the training.

3. The purpose of the study was to test whether a mental imagery training package resulted in better athletic performance. Freshman football players from ISU were randomly assigned to a control and experimental group. During a private session at the beginning of the season the players assigned to the experimental group were given 2 hours of instruction on using mental imagery, and were encouraged to use the technique for 10 minutes before each practice and game during the season. The players assigned to the control group also had a private meeting at the beginning of the season where they watched the movie “Remember the Titans”. At the end of the season, the amount of playing time and the head coach’s rating of overall performance for each athlete were compared to see if the imagery training was effective. Athletes who received the imagery training were found to perform better than those who did not receive the training.

Documents

INTERNAL VALIDITY AND BASIC RESEARCH DESIGN. Internal Validity the approximate truth about inferences regarding cause-effect or causal relationships