l 1

Chapter 11

Experimental and Quasi-experimental Designs

Dr. Bill Bauer

l 2

Key Ideas

• History of experimental resign• Key characteristics of experimental

design• Types of experimental design• Steps in conducting experimental

research• Criteria for evaluating experimental

research

l 3

A Brief History of Experimental Designs

• Schuyler 1903: control groups• McCall 1916: randomly selected

groups• McCall 1925: book on experiments• Fisher 1936: statistical methods

book

l 4

A Brief History of Experimental Designs• Campbell and Stanley 1963: 15 types

of experimental designs evaluated in terms of threats to validity

• Cook and Campbell 1979: four types of validity

• 1980 and beyond: computer enhancements to experimental design2

l 5

Key Characteristics of Experimental Designs

• Participants selected and assigned to groups• control• experimental

• An intervention is applied to one or more groups

• Outcomes are measured at the end of the experiment

l 6

Key Characteristics of Experimental Designs• Procedures are designed that

address potential threats to validity• Internal• External• Construct• Statistical Conclusion

• Statistical comparisons of different groups are conducted

l 7

Selecting Participants and Assigning Them to Treatments• Decide on the experimental unit of

analysis to be treated• individual• group or groups• organization

• Randomly assign individuals to groups control for extraneous characteristics that might influence the outcome

l 8

Selecting Participants and Assigning Them to Treatments• Control for extraneous factors

• random assignment (equating groups)

• pretest/posttest• covariates• matching participants• selecting homogenous samples• using blocking variables

l 9

IndependentVariable

IndependentVariable: Typeof Instruction

DependentVariable:Rates of Smoking

Controlling for CovariatesNo Covariates

DependentVariable

Covariate Introduced

Covariate:Parents Who

SmokeVarianceRemoved

Variance

l 10

Matching Process Based on Gender

ExperimentalGroup

ControlGroup

JohnJimJamesJoshJacksonJaneJohannaJulieJeanJeb

l 11

Applying an Intervention or Treatment

• Identify a treatment variable• identify the conditions or levels of

the variable• Manipulate the treatment

conditions

l 12

The Experimental Manipulation of a Treatment Group

Independent Variables1. Age (can’t manipulate)2. Gender (can’t manipulate)3. Types of Instruction (can

manipulate)a. Lecture (control)b. Lecture + Hazard Instruction (Comparison)c. Lecture + Hazard Instruction + slides of damaged lungs (experiment)

Dependent Variable

Frequency of Smoking

l 13

Threats to Statistical Conclusion Validity

• Low statistical power due to low sample size

• Violation of assumptions of statistical tests

• Use of unreliable measures

l 14

Threats to Internal Validity

• History• Maturation• Regression• Selection• Mortality• Interactions with selection

l 15

Threats to Internal Validity

• Diffusion of treatments• Compensatory equalization• Compensation rivalry• Resentful demoralization• Testing• Instrumentation

l 16

Threats to Construct Validity

• Lack of good operational definitions

• Apprehensiveness by participants• Participants “guessing” what the

researcher hopes to find

l 17

Threats to External Validity

• Interaction of selection and treatment

• Interaction of setting and treatment

• interaction of history and treatment

l 18

Treatment Comparisons in an ExperimentPhase 1: Relationship Picture

Error Correction Treatment+

Spelling Accuracy

Phase 2: Timeline Picture Test 1 Test 2 Test 36 Weeks 6 Weeks 6 Weeks6 Weeks 6 Weeks 6 Weeks

Class A: Regular Spelling Practice (Control)Class B: Reduced word list (Comparison)

Phase 3: Statistical ComparisonsClass A Class B Class C F-value

Test 1

Test 2

Test 3

10.3(3.6)

10.7 (3.3)

11.1(3.3)

10.8(4.3)

10.6(3.8)

10.3(3.6)

9.9(3.9)

13.9(4.2)

13.1(3.8)

0.27

4.90*

3.31*

* p<.05

Error Correction Treatment 6 Weeks 6 Weeks 6 Weeks

Class C: Error Correction (Experimental)

l 19

Types of Experimental Designs: Between Groups

• True Experiments• Pre- and posttest• Posttest Only

• Quasi Experiments• Pre- and posttest• Posttest Only

• Factorial Designs

l 20

Types of Experimental Designs: Within-Group or Individual • Time series experiments

• interrupted• uninterrupted

• Repeated measures experiments• Single-subject experiments

• A/B design• Multiple baseline design• Alternating treatments

l 21

Means and Main Effects of Eight groups in Factorial Design Depression

Mean rateof

smoking

Low

Mean rateof

smoking

Mean rateof

smoking

Mean rateof

smoking

Mean rateof

smoking

Mean rateof

smoking

Health lecture

Type of Instruction

Standard lecture

Main Effectsof Type ofInstruction

Main Effects ofDepression

Medium High

l 22

Graphs Showing Main and Interaction Effects

High

LowLow Medium High

High

LowLow Medium High

High

LowLow Medium High

(a) No interaction Effects (Parallel)

Extent of Smoking

Extent of Smoking

Extent of Smoking

(b) Interaction Effects (Crossed)

(c) Interaction Effects (Not Parallel)

(a) Interaction Effects (Crossed)

Standard lecture

Health lecture

Standard lecture

Health lecture

Standard lecture

Health lecture

l 23

Steps in Conducting Experimental Research

• Decide if an experiment addresses the research problem

• Form hypotheses to test cause-effect relationships

• Select an experimental treatment and introduce it

• Identify study participants

l 24

Steps in Conducting Experimental Research

• Choose a type of experimental design

• Conduct the experiment• Organize and analyze the data• Develop an experimental research

report

l 25

Criteria for Evaluating Experimental Research• Does the experiment have a powerful

intervention?• Does it employ few treatment groups

(e.g. only two)?• Will participant profit from the

intervention?• Is there a systematic way the

researcher derived the number of participants per group?

l 26

Criteria for Evaluating Experimental Research• Were there an adequate number of

participants used in the study?• Were valid, reliable, and sensitive

measures or observations used?• Did the study control for extraneous

factors?• Did the researcher control for

threats to internal validity?

l 27

Applying What you Have Learned: An Experimental StudyReview the article and look for the following:

• The research problem and use of quantitative research

• Use of the literature• The purpose statement and research hypothesis• Types and procedures of data collection• Types and procedures of data analysis and

interpretation• The overall report structure