Chapter 3 Experiments, Quasi-Experiments, and Field Observations Winston Jackson and Norine Verberg Methods: Doing Social Research, 4e

Chapter 3Experiments, Quasi-Experiments, and Field Observations

Winston Jackson and Norine Verberg

Methods: Doing Social Research, 4e

© 2007 Pearson Education Canada 3-2

The Rationale of the Experiment

John Stuart Mill — Method of Difference — the experiment is the key tool for discerning causal relations

A well-designed experiment should provide clear evidence of a cause-effect relation Indicate whether or not the treatment variable

(e.g., studying with the radio on) will bring about a change in some dependent measure (grade performance), other things being equal


Rationale (continued)

Internal validity: The extent to which the researcher can demonstrate that the treatment variable is having an impact on the dependent variable, and that other sources have been controlled

External validity: The extent to which the researcher can extrapolate the study findings to other groups in general


Key Elements in Experimental Designs

A. Dependent variable – the effect in a cause-effect relationship

B. Independent variable – the variable the researcher manipulates to determine whether and how it will change the dependent variable the cause in a cause-effect relationship


Key Elements (cont’d)

Kinds of independent variables:i. Treatment Variables – variable studied

ii. Control Variables – major influences intentionally controlled for in the experiment

iii. Confounding Variables – variables that can unintentionally obscure or enhance results

iv. Random Variables – vary without control, but are taken into account in study design (e.g., randomization)


Key Elements (cont’d)

C. Levels – often two or three levels 2 x 2: two levels of the treatment variable

and two levels in a control variable


Pseudo-Experimental Designs

Have limited scientific merit Also called pre-experimental designs

Share some elements of classic experiment, however, they do not permit clear causal inferences

Two types: Same group: pretest/post-test design Exposed/comparison group design


A. Same Group: Pretest/Post-Test


Threats to Internal Validity

i. History – concurrent events

ii. Maturation – changes in the individual subject

iii. Testing – possible of response bias

iv. Instrument Decay – unreliable measurement

v. Statistical Regression – extreme scores


B. Exposed/Comparison Group

Measures are taken at only one point in time.

Problem: groups may not have been similar initially.

The result may, or may not, be due to the treatment variable.


More Threats to Internal Validity

vi. Selection – Subjects selecting themselves into the study

vii. Mortality – Subjects selecting themselves out of the study


Classic Experimental Designs

Two types: Between-Subjects Design Within-Subject Design

Both types of design allow a researcher to demonstrate causal inference


A. Between-Subjects Design


Between-Subjects Design (cont’d)

Involves a control and an experimental group The experimental group is exposed to

treatment intervention The control group is exposed to neutral

treatment


Key to Experimental Design

Construct treatment and control groups to be as similar as possible before the experiment begins. This is done by: Randomization – each subject has an equal

chance of being assigned to either group (provides control over both known [control] and unknown [random] factors)

Precision matching – matching subjects between groups

Combination of the above two methods


Key to Experimental Design (cont’d)

Blocking – Group subjects according to some controlled variable before randomly assigning them to a group

Baseline stability – Taking measures of the variable prior to introducing treatment


Analyzing the Data

TABLE 3.2 PERCENT WANTING TO ATTEND UNIVERSITY

BY EXPOSURE AND NON-EXPOSURE TO CD-ROM

PERCENTAGE WANTING TO ATTEND UNIVERSITY

GROUP TIME 1 TIME 2 DIFFERENCE

Treatment 57.0 73.0 73 – 57 = 16

Control 55.0 61.0 61 – 55 = 6

Estimated impact of CD-ROM: 10


Demonstrating a Causal Relation

1. Changes in treatment variable occur prior to changes in the dependent variable

2. The treatment and dependent variables are associated: as the treatment variable goes up, the dependent varies systematically

3. Nothing but the treatment variable has influenced the dependent variable


Ruling out Confounding Effects

Ensure that context is the same

Balance the background characteristics

Neutralize confounding (sources of spuriousness) variables

Deal with random variables


B. Within-Subject Designs

In the between-subjects design, the control for known and unknown factors is achieved through randomization

In the within-subject design, the control for known and unknown factors is achieved by exposing a subject to the different treatments Since the subject is the same person,

background characteristics, attitudes, and intelligence are all perfectly controlled

Also called control by constancy


Within-Subject Design (cont’d)

Subjects are exposed to the various treatments Subjects’ own scores when exposed to different

treatments are compared Importance of having a baseline measure and returning

to the original condition The within-subject ABBA design:

A – measure dependent variable under original condition B – measure dependent variable under treatment

condition B – continue treatment condition and measure dependent

variable A – measure dependent variable after returning to

original condition


Hawthorne Effect

Refers to any variability in the dependent variable that is not the direct result of variations in the treatment variable

Hypothesis: worker productivity would increase as lighting intensity was increased When lighting increased, productivity increased HOWEVER, when lighting was later decreased,

productivity did not decrease. WHY? Interpretation: something other than treatment

variable influenced workers – perhaps they worked faster because they knew were being observed


Quasi-Experimental Designs

Approximation of experimental design: done in situations where it is not possible to: use random assignment control the nature or timing of the treatment

Example: Henry & Ginzberg: Racial Discrimination in

Employment (See Box 3.4, text pp. 75-77.)


Racial Discrimination in Employment

Two job applicants matched with respect to age, sex, education, physical appearance (dress), and personality were sent to apply for the same advertised job. Only difference: one was White, one was Black

Results Both offered job 5.0% White offered job 13.4% Black offered job 4.5% Neither offered job77.1%


Field Experiments

Researcher intervenes in a natural settings Direct observations, “real” behaviour

Researcher intervention Greeting stranger

Proxemics: norms surrounding personal space and the conditions under which such space will or will not be violated

Examples: cutting-through behaviour, greeting behaviour, helping behaviour


Naturalistic Observational Studies

Observe and record behaviour that occurs in a natural setting with those being observed unaware that they are being studied Do not attempt to alter social environment No intervention, simply record behaviour

Tally sheets are designed, then used to record the behaviour Andrew Harrell’s Grocery Cart Safety study


Samples of Student Research Projects

Dressing for winter Parking violations Gender and smoking Professor/student

participation: gender Seat belt compliance Speeding Antigonish Buying healthy food

ABM behaviour Termination of

conversations Drinking patterns Smoking behaviour in

teens Stop sign Tipping


Steps in Doing Study

1. Restrict observations

2. Review of literature

3. Develop hypotheses

4. Define terms

5. Develop a tally sheet (See Figure 3.5, p. 90)

6. Transfer data to master table (see Figure 3.6, p. 90)

7. Creating tables (Tables 3.4, 3.5, 3.6, p. 91)

8. Writing the report


Field and Observational Studies: An Assessment

Weak on generalizations Strong on validity (real behaviour) Making causal inferences a challenge Multivariate a problem Probing strong with participant observation,

in-depth interviews, and focus groups Probing weak with naturalistic observational


Advantages and Disadvantages of Experimental Designs Advantages:

Ease of making clear causal inferences

Disadvantages: Low external validity: poor on generalization to

a larger population Concerns about the artificiality of lab Poor on probing, poor on multivariate Experiments cannot study all topics

Documents

Chapter 3 Experiments, Quasi-Experiments, and Field Observations Winston Jackson and Norine Verberg Methods: Doing Social Research, 4e