Embed Size (px)
Citation preview
Factorial Designs: Programmatic Research, RH: Testing & Causal
Inference
• Applications of Factorial designs in Programmatic Research
• Research Hypotheses for Factorial Designs• Variable Role Explication in Factorial Designs &
Causal Interpretation
Using Factorial Designs in Programmatic Research I
Adding a 2nd IV
Perhaps the most common application of factorial designs it so look at the separate (main) and combined (interaction) effects of two IVs
Often our research starts with a simple RH: that requires only a simple 2-group BG research design.
Tx1 Control
Keep in mind that to run this study, we made sure that none of the participants had any other treatments !
At some point we are likely use Factorial designs to ask ourselves about how a 2nd Tx/IV also relates to the DV
Factorial Designs – Separate (Main) and combined (interaction) effects of two treatments
Tx1 Control
Tx2
Control
Gets neither Tx1 not Tx2
Gets both Tx1 & Tx2Gets Tx2 but not Tx1
Gets Tx1 but not Tx2
Using Factorial Designs in Programmatic Research II
“Correcting” Bivariate Studies
40 40
Tx1 Tx2
40 40
Male FemaleOur well sampled, carefully measured, properly analyzed study showed …
… nothing !
Our well sampled, randomly assigned, manipulated, controlled, carefully measured, properly analyzed study showed …
… nothing !
Looks like neither TX nor Gender is related to the DV !!!
Tx1 Tx2
However, when we analyzed the same data including both variables as IVs …
60
20 60
20Male
Female
40 40
40
40
There are treatment effects both for Males & Females – the marginal Tx means are an “aggregation error”
So, instead of the “neither variable matters” bivariate results, the multivariate result shows that both variables are conditionally related to the DV -- they interact !!!!!
There are Sex effects both for those in Tx1 & those in Tx2 – the marginal Sex means are an “aggregation error”
Using Factorial Designs in Programmatic Research IIIGeneralization across Populations, Settings & Tasks
Often our research starts with a simple RH: that requires only a simple 2-group BG research design.
Computer Lecture
Keep in mind that to run this study, we had to make some choices/selections:
For example:population College Students setting Lecture settingstim/task teach Psychology
When we’ve found and replicated an effect, making certain selections, it is important to check whether changing those selections changes the results.
60 40
If there is an interaction – if the results “depend upon” the population, task/stimulus, setting, etc – we need to know that, so we can apply the “correct version” of the study to our theory or practice
If there is no interaction – if the results “don’t depend upon” the population, task/stimulus, setting, etc – we need to know that, so we can apply the results of the study to our theory or practice, confident in their generalizability
Computer Lecture
At some point we are likely use BG Factorial designs to ask ourselves how well the results will generalize to:
• other populations – college vs. high school
Tx Control
Col
HS
• other settings – lecture vs. laboratory
Tx Control
Lecture
On-line
• other tasks/stimuli – psyc vs. philosophy
Tx Control
Psyc
Phil
Tx Control
Col
HS Tx Control
Lecture
On-line
Tx Control
Psyc
Phil
Notice that each factorial design includes a replication of the earlier design, which used the TX instructional methods to :• teach Psychology• to College Students• in a Lecture setting
Each factorial design also provides a test of the generalizability of the original findings:• w/ Philosophy vs. Psychology• to High School vs. College Students• in an On-line vs. Lecture setting
Tx Control60 40
60 40
60 40
60 40
?? ???? ??
?? ??
?? ??
Using Factorial Designs in Programmatic Research IV
Do effects “depend upon” length of treatment ?
As before, often our research starts with a simple RH: that requires only a simple 2-group BG research design.
Tx1 Tx2
Time Course Investigations
In order to run this study we had to select ONE treatment duration (say 16 weeks):• we assign participants to each condition• begin treatment of the Tx groups• treat for 16 weeks and then measured the DV
20 20
Using this simple BG design we can “not notice” some important things. A MG Factorial can help explore the time course of the Tx effects.
Tx1
Tx2
Tx1 Tx2
Short Medium
20
20 40
40
By using a MG design, with different lengths of Tx as the 2nd IV, we might find different patterns of data that we would give very different interpretations
20 20
Tx1
Tx2
Short Medium
20
20 20
40 Tx1
Tx2
Short Medium
20
20 60
40 Tx1
Tx2
Short Medium
20
20 40
0
Using Factorial Designs in Programmatic Research V
Evaluating Initial Equivalence when Random assignment is not possible
As before, often our research starts with a simple RH: that requires only a simple 2-group BG research design.
Tx1 Tx2
Initial Equivalence Investigations
In order to causally interpret the results of this study, we’d have to have initial equivalence• but we can’t always RA & manipulate the IV• So what can we do to help interpret the post-treatment differences of the two treatments?• Answer – compare the groups before treatment too!
Tx1
Tx2
Pre Post
20
20 20
40
By using a MG design, we can compare the groups pre-treatment and use that information to better evaluate post-treatment group differences (but can’t really infer cause). For which of these would you be more comfortable conclusing that Tx1> Tx2 ??
Tx1
Tx2
Pre Post
40
20 20
40
Tx1
Tx2
Pre Post
60
20 20
40Tx1
Tx2
Pre Post
30
20 40
60
As good as it gets!
Nah – Post dif = pre dif !
Nah – Tx1 lowered scoreMaybe – more in-
crease by Tx1
Replication & Generalization in Factorial Designs
Most factorial designs are an “expansion” or an extension of an earlier, simpler design, often by adding a second IV that “makes a variable out of an earlier constant”. This second IV may related to the population, setting or task/stimulus involved.
Study #1 – Graphical software Study #2
Mean failures PC = 5.7, std = 2.1
Mean failures Mac = 3.6, std = 2.1
PC Mac
Graphical
Computing
5.9
3.1
4.5
3.6
3.8
3.7
What gives us the most direct replication? The main effect of PC vs. Mac or one of the SEs of PC vs. Mac?
Did Study #2 replicate Study #1?
Identifying the “replication” in a factorial design
Replication & Generalization in Factorial Designs, cont…
Most factorial designs are an “expansion” or an extension of an earlier, simpler design, often by adding a second IV that “makes a variable out of an earlier constant”. This second IV may related to the population, setting or task/stimulus involved.
Study #1 – Mix of Networked & Study #2
Stand-alone computers
Mean failures PC = 5.7, std = 2.1
Mean failures Mac = 3.6, std = 2.1
PC Mac
Networked
Stand-alone
8.9
3.1
6.0
1.6
5.8
3.7
What gives us the most direct replication? The main effect of PC vs. Mac or one of the SEs of PC vs. Mac?
Did Study #2 replicate Study #1?
Identifying the “replication” in a factorial design
RH: for Factorial Designs
Research hypotheses for factorial designs may include
• RH: for main effects
• involve the effects of one IV, while ignoring the other IV
• tested by comparing the appropriate marginal means
• RH: for interactions
• usually expressed as differences between hypothesizedresults for a set of simple effects
• tested by comparing the results of the appropriate set of simple effects
• That’s the hard part -- determining which set of simple effects gives the most direct test of the interaction RH:
#1 Sometimes the Interaction RH: is explicitly stated
• when that happens, one set of SEs will provide a direct test of the RH: (the other won’t)
This is most directly tested by inspecting the simple effect of paper vs. computer presentation for easy tasks, and comparing it to the simple effect of paper vs. computer for hard tasks.
Here’s an example:
Easy tasks will be performed equally well using paper or computer presentation, however, hard tasks will be performed better using computer presentation than paper.
Presentation Comp PaperTask Diff.
Easy
Hard
=
>
Your Turn...
Young boys will rate playing with an electronic toy higher than playing with a puzzle, whereas young girls will have no difference in ratings given to the two types of toys.
Type of Toy Elec. PuzzleGender
Boys
Girls =
>
Judges will rate confessions as more convincing than do Lawyers, however, Lawyers will rate witnesses as more convincing than do Judges.
Type of Evidence Confession WitnessRater
Judge
Lawyer
<>
SE Toy @ Gender
SE Rater @ Evidence
#2 Sometimes the set of SEs to use is “inferred” ...Often one of the IVs in the study was used in previous research, and the other is “new”.
• In this case, we will usually examine the simple effect of the “old” variable, at each level of the “new” variable
•this approach gives us a clear picture of the replication and generalization of the “old” IV’s effect.
e.g., Previously I demonstrated that computer presentations lead to better learning of statistical designs than does using a conventional lecture. I would like to know if the same is true for teaching writing.
Let’s take this “apart” to determine which set of SEs to use to examine the pattern of the interaction...
Previously I demonstrated that computer presentations lead to better learning of statistical designs than does using a conventional lecture. I would like to know if the same is true for teaching writing.
Here’s the design and result of the earlier study about learning stats.
Type of Instruction Comp Lecture
>
Here’s the design of the study being planned.
Type of Instruction Comp LectureTopic
Stats
Writing
What cells are a replication of the earlier study ?
So, which set of SEs will allow us to check if we got the replication, and then go on to see of we get the same results with the new topic ?
Yep, SE of Type of Instruction, for each Topic ...
Your turn ..
I have previously demonstrated that rats learn Y-mazes faster than do hamsters. I wonder if the same is true for radial mazes ?
I’ve discovered that Psyc majors learn statistics & Ethics about equally well. My next research project will also look at how well Sociology majors learn these topics.
Type of Rodent Rat Hamster>
Major Psyc Soc
=Maze
Y
Radial ?
>
Type of Rodent Rat Hamster
Topic
Stats
Ethics ?
=
Topic Stats Ethics
SE Rodent @ Maze
SE Topic @ Major
#3 Sometimes the RH: about the interaction and one about the main effects are “combined”
• this is particularly likely when the expected interaction pattern is of the > vs. > type (the most common pattern)
Here’s an example…
Group therapy tends to work better than individual therapy, although this effect is larger for patients with social anxiety than with agoraphobia.
Type of Therapy Group Indiv.Anxiety
Social
Agora.>
>
Main effect RH:
>Int. RH:
So, we would examine the interaction by looking at the SEs of Type of Therapy for each type of Anxiety.
Young girls have better verbal skills than motor skills, however the difference gets smaller with age (DV = skill score)
Type of Skill Verbal MotorAge
4 yrs
9 yrs >
>
Confession is considered more convincing than eyewitness testimony. This preference is stronger for jurors than judges.
(DV = convincingness rating)
Type of Evidence
Confession
Witness
Rater
Judge Jurors
>>
Your Turn…
>
>
SE Skill @ Age
SE Evidence @ Rater
Explicating Variable Roles
Before a study After the study
IV IV
DV DVPotential Control Variables Confounds -- initial equiv. of subj vars
-- ongoing equiv of proc vars
Confound Variables -- subject var confounds -- procedural var confounds
Explicating Design Variables
What I want you to be able to do is to tell the specific “function” of any variable in any study you read -- even if that variable is not mentioned in the description of the method & procedure !
We’ll start by reviewing basic elements of variables and functions
Subject variables and procedural variables
• subject variables are things the value of which participants “bring with them” when they arrive at the study
• age, gender, personality characteristics, prior history, etc.
• Procedural variables are thing the value of which are “provided” or “created” by the researcher during the study
About Potential Confounding VariablesLike IVs, potential confounds are causal variables
• they are variables that we think (fear) could have a causal influence on a subject’s DV score
• if equivalent (on the average) across IV conditions, then they are “control variables “ and contribute to the casualinterpretability of the results
• if nonequivalent (on the average) across IV conditions, they are “confounds” that introduce alternative explanations ofwhy the mean DV scores differed across the IV conditions
Candidates for Confounding Variables
• variables that researchers in your area have attempted to control (recognized confounds)
• variables know to be causal influences upon your DV (previously effective IVs) that are not the IV in your study
Explicating the “role” of variables in research designs
any given variable must be …• a manipulated variable or a subject variable• a DV or an IV or a control variable or a confound• a control variable has either been...
balanced (usually by RA or matching) or held constant or eliminated
• a confounding variable is either a problem with initial equiv. (subject variable) or ongoing equivalence (procedural variable)
Remember:
• all subject variables are controlled by RA (of individuals)
• all subject variables are confounds in QE or NG designs (except for any that were used in post hoc matching)
• with a priori matching - all subject variables are controlled with post hoc matching -- only matching variable(s) is controlled
manipulated subject independent dependent confound controlled
"constant" eliminated balanced matched random assignment
Always pick ONE of these two !!!
Always pick ONE of these four !!!
If you say the variable is a CONTROL variable, always pick one of these three types of control !!!
If you say the variable was controlled by BALANCING, be sure to tell which balancing technique was used
If you say the variable was a CONFOUND, tell if confound of initial or ongoing equivalence
Now for the hard part…
You have to do the variable role explication 3 (count ‘em 3) times Once for each main effect and once for the interaction effect!
For this to make more sense, let’s look at how you decide whether or not you can make a causal interpretation of an effect in a factorial design…
Here’s the rule most folks carry in their heads (good rule!)…
Start by assessing the causal interpretability of each main effect
In order to causally interpret an interaction, you must be able to casually interpret BOTH main effects.
Study of Age and Gender no casually interpretable effects (main effects nor interaction)
Study of Age and Type of Toy (RA + Manip)
only casually interpretable effect would be the main effect of Type of Toy (not the main effect of Age, nor the interaction).
Study Type of Toy (RA + Manip) and Playing Situation (RA + manip)
all effects are causally interpreted (both main effects and the interaction).
Some examples…
Here’s the real rule…
You can causally the interaction when you can causally interpret the differences between causally interpretable simple effects!
Presentation Comp PaperTask Diff.
Easy
Hard
=
>
1st - I can RA, manip & control Presentation – so I can causally interpret each of the simple effects of Presentation.
2nd – I can RA, manip & control Task Difficulty – so I can causally interpret the difference between the causally interpretable simple effects of Presentation for Easy & Hard tasks!
So I CAN causally interpret the interaction!!
Example #2
You can causally the interaction when you can causally interpret the differences between causally interpretable simple effects!
Presentation Comp PaperGender
Male
Female
=
>
1st - I can RA, manip & control Presentation – so I can causally interpret each of the simple effects of Presentation.
2nd – I can’t RA, manip & control Gender – so I can’t causally interpret the difference between the causally interpretable simple effects of Presentation for Males & Females!
So I CAN’T causally interpret the interaction!!
Example #3
You can causally the interaction when you can causally interpret the differences between causally interpretable simple effects!
Presentation Comp PaperGender
Male
Female
=>
2nd - I can RA, manip & control Presentation – so I can causally interpret the difference between the Gender simple effects – even though I can’t causally interpret either of them!
1st – I can’t RA, manip & control Gender – so I can’t causally interpret the difference between Males & Females during either presentation!
So I CAN’T causally interpret the interaction!!
How does this apply to variable role explication??
Presentation Comp PaperGender
Male
Female
=>
Take “motivation”…
It is a subject variable. It is not the DV or an IV.It is not reasonably a constant.It is not a matching variable.
There was RA to Presentation so, motivation (and all other subject variables) is presumably controlled by balancing between computer and paper (for both males & females)
There was not RA to Gender so motivation (and all other subject variables) is presumably a confound (initial eq) when comparing computer and paper (for both males & females)
Motivation would also be a confound for the interaction !!
