Upload
annis-sullivan
View
234
Download
4
Tags:
Embed Size (px)
Citation preview
Experimental design
Experimental design
Outlines • What is an experiment?
• Controlling the experiment
• Design categories (Experimental, Quasi-
experimental and Ex Post Facto)
• Comparison
• Meta analysis
• Examples
Experimental design
Experiment Ordered investigation that attempts to prove or disapprove a hypothesis
Questions
Hypothesis
Experiment
Analysis
Conclusion
Scientific Method Components
Experimental design
Experimental design
• So .. The hypothesized relationship isIndependent variable -> dependent variable
Experimental design
• An independent variable is one that the researcher studies as a possible cause of something else.
• Dependent variable is a variable that is potentially influenced by the independent variable.
• Manipulation: Researcher manipulates the independent variable ( treatment – intervention)
Experimental design
• So .. The hypothesized relationship isIndependent variable -> dependent variable
Experimental design
• So .. The hypothesized relationship isIndependent variable -> dependent variable Certain Drug impact on cancer
Experimental design
• A researchers can convincingly identify cause-and-effect relationships by experimental design
• Many possible factors that might cause or influence a particular condition or phenomena.
• The researcher attempts to control for all influential factors except the study factor.
Controlling for confounding variables
• 1- keep something constant• 2- Include a control group• 3- Randomly assign people to group• 4- Assess equivalence before the treatment with
one or more pretests .• 5- Expose participants to all experimental
conditions• 6- statistically control for confounding variables.
• Regarding cause-effect or causal relationships, internal Validity is the approximate truth about inferences.
• It is relevant in studies that try to establish a causal relationship.
• The key question in internal validity is whether observed changes can be attributed to your program or intervention (i.e., the cause) and not to other possible causes.
• Did the treatment cause the outcome to occur? Or were there other confounding factors that caused the outcome?
Internal Validity
• Are the findings unique to just participants we studied or could they apply to other groups?
• Refers to the extent to which the results of an experiment can be generalized across populations, time and settings.
External Validity
• Imagine that we are studying the effects of an education program in mathematics for first grade students on a measure of math performance such as a standardized math achievement test.
• In the post-only design, we would give the first graders the program and then give a math achievement posttest.
• Consider what would happen if you observe a certain level of posttest math achievement or a change or gain from pretest to posttest.
• You want to conclude that the outcome is due to your math program. How could you be wrong?
Single Group Threats
• History Threat: It's not your math program that caused the outcome, it's something else, some historical event that occurred.
• Maturation Threat: The children would have had the exact same outcome even if they had never had your special math training program. All you are doing is measuring normal maturation or growth in understanding that occurs as part of growing up.
• Testing Threat: This threat only occurs in the pre-post design. What if taking the pretest made some of the children more aware of that kind of math problem -- it "primed" them for the program so that when you began the math training they were ready for it in a way that they wouldn't have been without the pretest.
Single Group Threats
• Instrumentation Threat: Like the testing threat, this one only operates in the pretest-posttest situation. What if the change from pretest to posttest is due not to your math program but rather to a change in the test that was used?
• Mortality Threat: It means that people are "dying" with respect to your study.
• Regression Threat: Also known as a "regression artifact" or "regression to the mean" is a statistical phenomenon that occurs whenever you have a nonrandom sample from a population and two measures that are imperfectly correlated.
Single Group Threads
Design Categories
• Pre-experimental designs
• True Experimental designs
• Quasi-experimental designs
• Ex post facto designs
• Factorial designs
Design Categories
• Some are true experimental designs as such they allow us to identify cause-and-effect
• Some give alternative explanations of an observed change.
Design Categories
• All of the designs have one thing in common : clearly identify independent and dependent variable.
Design Categories
We illustrate the designs using tables that have this general format "Table"
Group 1
Group 2
Group Time ->
Pre- Experimental Design
• The cells have one of four notations:• Tx: Indicates that a treatment is presented • Obs: Indicates that observation is made • : Indicates that nothing occurs during a
particular time period. • Exp: Indicates a previous experience
Pre- Experimental Design: One-shot Experimental Case Study
The most primitive type of experiment that might be termed "research“
• The design has low internal validity • 1- the characteristics or the behavior observed after
the treatment existed before the treatment as well.• 2- influenced by other factors • 3- a single measurement or observation doesn't
guarantee that situation has change or not.
Group 1 Tx Obs
Pre- Experimental Design: One-shot Experimental Case Study
• Example .. • The design will be something like this :
Exposure to cold + Damp ground (TX)-> Child has a cold (Obs)
• One-shot experimental case study is simple to carry out, its results are meaningless.
Pre- Experimental Design: One- Group Pretest-posttest Design
• We know that a change has taken a place. But we have not ruled out other possible explanation for the change.
Group 1 Obs Tx Obs
Pre- Experimental Design: Static Group Comparison
• Involves both an experimental group and a control group.
• No attempt is made to check wither they are similar or not before the treatment so no way to know if the treatment actually causes any differences between the two groups.
Group 1 Tx Obs
Group 2 Obs
• If random assignment is used, we call the design a randomized experiment or true
experiment. • If random assignment is not used, then we have to ask a second question: Does
the design use either multiple groups or multiple waves of measurement? • If the answer is yes, we would label it a quasi-experimental design. If no, we
would call it a non-experimental design.
Types of Design
• True experimental design is regarded as the most accurate form of experimental research, in that it tries to prove or disprove a hypothesis mathematically, with statistical analysis.
• For an experiment to be classed as a true experimental
design, it must fit all of the following criteria:– The sample groups must be assigned randomly.– There must be a viable control group.– Only one variable can be manipulated and tested. The tested subjects
must be randomly assigned to either control or experimental groups.
True Experimental Design
True Experimental Design: Pretest-Posttest Control Group Design
• The pretest-posttest equivalent groups design provides for both a control group and a measure of change but also adds a pretest.
• It is important that the two groups be treated in a similar manner.
Random Assignment
Group 1 Obs Tx Obs
Group 2 Obs Obs
• The Solomon Four-Group Design is designed to deal with a potential testing thread.
• This design has four groups.• Two of the groups receive the treatment and two do not.
True Experimental Design: The Solomon Four-Group Design
Random Assignment
Group 1 Obs Tx Obs
Group 2 Obs - Obs
Group 3 - Tx Obs
Group 4 - - Obs
• Sometimes, randomness is either impossible or impractical. In those situations use quasi-experimental design.
• However, has features that can eliminate many threats to internal validity.
• Used frequently in evaluation because: – Often randomization is impossible or difficult– Ethical/legal prohibitions against randomization– No viable control group available – Inadequate resources to conduct randomization
Quasi-Experimental Designs
• To show that two groups are equivalent with respect to the dependent variable prior to the treatment, thus eliminating initial group differences as an explanation for post-treatment differences.
• Differs from experimental designs because test and control groups are not totally equivalent; equivalence on the pretest ensures equivalence only for variables that have specifically been measured.
Quasi-Experimental Designs: Nonrandomized Control Group Pretest-Posttest Design
Group 1 Obs Tx Obs
Group 2 Obs - Obs
• Threats to validity:– Partly controls for history threat(external event would
affect both groups, provided groups, provided groups are similar), maturation, instrument threats and instrumentation threats.
– However, even if groups are statistically very similar, if the intervention is given to volunteers they may behave differently than control group(due to self-selection).
– Regression threat.
Quasi-Experimental Designs: Nonrandomized Control Group Pretest-Posttest Design
• To show that, for a single group change occurs during a lengthy period only after the treatment has been administered.
• Provides a stronger alternative to “One group pretest-posttest design; external validity can be increased by repeating the experiment in different places under different condition.
Quasi-Experimental Designs: Simple time-series experiment
Group 1 Obs Obs Tx Obs Obs
• Threats to validity:– Extension of pretest-posttest design but reduces
maturation, testing, regression threats– No control group so no selection threat.– History threat is controlled partially.– if measurement changes around time of program
instrumentation threat may be present
Quasi-Experimental Designs: Simple time-series experiment
• Bolstering the internal validity of the preceding design with the addition of a control group.
• Involves conducting parallel series of observations for experiment and control groups.
Quasi-Experimental Designs: Control group, time-series design
Group 1 Obs Obs Tx Obs ObsGroup 2 Obs Obs - Obs Obs
• Showing , in a single group or individual, that a treatment consistently leads to a particular effect.
• Is an on-again, off-again design in which the experimental treatment is sometimes present, sometimes absent.
Quasi-Experimental Designs: Reversal, time-series design
Group 1 Tx Obs - Obs Tx Obs
• Showing , in a single group or individual, that different treatments have different effects.
• Involves sequentially administrating different treatments at different times and comparing their effects against the possible consequent of non-treatment.
Quasi-Experimental Designs: Alternating treatments design
Group1 Txa Obs - Obs Txb Obs
• Showing, the effect of a treatment by initiating at different times for different groups or individuals, or perhaps in different setting for a single individual.
• Involves tracking two or more groups or individuals over time, or tracking a single individual in two or more settings, for a lengthy period of time, as well as initiating the treatment at different times for different groups, individuals, or settings.
Quasi-Experimental Designs: Multiple-baseline design
Group1 - Obs Tx Obs Tx Obs
Group2 - Obs - Obs Tx Obs
• Showing , the effect of a treatment by initiating it at different times for
• Involves sequentially administrating different treatments at different times and comparing their effects against the possible consequent of non-treatment.
Quasi-Experimental Designs: Alternating treatments design
Group1 Txa Obs - Obs Txb Obs
Ex Post Facto Designs
• Sometimes we can’t manipulate some variables
– Impossible, e.g.: Characteristics
– Unethical, e.g.: Virus
• Ex post facto = after the fact
– Already happened in the past
• Possible effect of an experience/condition that
occurred in the past
• May show difference but not conclusive
Ex Post Facto: Simple Design
Group 1 Exp Obs
Group 2 - Obs
Group Time ->
• 2+ independent variables
• Simultaneously or sequential
Factorial Designs
Time ->
Factorial Design:Two-factor experimental design
Random Assignme
nt
Group 1 Tx1 Tx2 Obs
Group 2 Tx1 - Obs
Group 3 - Tx2 Obs
Group 4 - - Obs
Group Time ->
Factorial Design:Two-factor experimental design
Random Assignme
nt
Group 1 Tx1 Tx2 Obs
Group 2 Tx1 - Obs
Group 3 - Tx2 Obs
Group 4 - - Obs
Group Time ->
Factorial Design:Two-factor experimental design
Random Assignme
nt
Group 1 Tx1 Tx2 Obs
Group 2 Tx1 - Obs
Group 3 - Tx2 Obs
Group 4 - - Obs
Group Time ->
Factorial Design:Two-factor experimental design
Random Assignme
nt
Group 1 Tx1 Tx2 Obs
Group 2 Tx1 - Obs
Group 3 - Tx2 Obs
Group 4 - - Obs
Group Time ->
Factorial Design:Two-factor experimental design
Random Assignme
nt
Group 1 Tx1 Tx2 Obs
Group 2 Tx1 - Obs
Group 3 - Tx2 Obs
Group 4 - - Obs
Group Time ->
Factorial Design:Two-factor experimental design
Random Assignme
nt
Group 1 Tx1 Tx2 Obs
Group 2 Tx1 - Obs
Group 3 - Tx2 Obs
Group 4 - - Obs
Group Time ->
Factorial Design:Two-factor experimental design
Random Assignme
nt
Group 1 Tx1 Tx2 Obs
Group 2 Tx1 - Obs
Group 3 - Tx2 Obs
Group 4 - - Obs
Group Time ->
Factorial Design:Two-factor experimental design
Random Assignme
nt
Group 1 Tx1 Tx2 Obs
Group 2 Tx1 - Obs
Group 3 - Tx2 Obs
Group 4 - - Obs
Group Time ->
Factorial Design:Two-factor experimental design
• Example: Clustering using K-means
– Tx1: A different method to choose initial centroids
– Tx2: A different formula to calculate distance
Factorial Design: Combined experimental and ex post facto design
Group1 Expa
Random Assignme
nt
Group 1a Txa Obs
Group 1b Txb Obs
Group2 Expb
Random Assignme
nt
Group 2a Txa Obs
Group 2b Txb Obs
Group Time ->
Factorial Design: Combined experimental and ex post facto design
• Example: Clustering using K-means
– Expa, Expa: 2 different kinds of datasets
– Txa, Txb: 2 different formulas to calculate distance
Comparison
Meta-analyses
• Replication
• Quantitative analysis of the analyses
– Primarily statistical
Examples & Discussion