EXPERIMENTS AND EXPERIMENTAL DESIGN

ETHICAL ISSUES IN RESEARCH

Logic of causality

If X, then Y• andIf not X, then not Y

Or, colloquially:If the stimulus is given, then the outcome occurs• andIf the stimulus is not given, then the outcome does not

occur

Basic requirements of a true experiment:

- At least 2 groups: one experimental & one control group- Random assignment of subjects to groups. Every case has:

(1) the same chance of being selected (i.e. equal probability)(2) the chance of selection does not change (i.e. constant probability)

- Manipulation of the IV.- Measurement of the DV.- Except for the experimental manipulation, the 2 groups =

treated exactly alike

Results

- controlling for pre-existing subject differences; differences

are to be neutralized/distributed approximately evenly among the groups random assignment of subjects to treatment & control groups

- making sure that the events occurring within each experimental group are exactly the same, except for the manipulated independent variable controlling for extraneous factors & experiences during the experiment

Randomization for groups

Tests

• Use tests of statistical significance to determine the likelihood that the results occurred by chance rather than being caused by the treatment variable (i.e. significant differences btw. the groups).

• A result is statistically significant at the 0.05 level: it means that only 5% of the time (5 times in 100 if we were to repeat the experiment) would differences btw. the experimental & control groups occur by chance when the experimental variable actually has no effect.

Tests

• Differences between the mean values for T and C

• The differences between slopes for T and C

Mean values for T and C

Slope

Staging of an experiment

1. Introduction: Instructions and Cover Story

2. Manipulation of the IV

3. Measurement of the DV (verbal reports; behavioral observation)

4. Debriefing (explain purpose of experiment; explore subjects’ reactions; alleviate subjects’ anxiety about their performance/ participation)

Validity

• Internal Validity: the approximate truth about inferences regarding the tested causal relationships in the experimental setting

• External Validity: the degree to which the conclusions in the

study would hold for other persons in other places & at other times (i.e. how generalizable are the results of the study)

Issues to consider

• Pre-testing of experiments • Laboratory experiments vs. experiments in natural settings

(field experiments)

• Trade-off btw. increase in external validity & control over extraneous variables & manipulation of the IV

• Subjects’ reaction to the social meaning of the situation:

often a problem for laboratory experiments

Types of experiments

• Pre-test – post-test control group design• Post-test-only control group design• The Solomon four group design• Factorial design• Quasi-experimental design

– Experimental Design in Survey

Pre-test – post-test control group design

Tt1 X Tt2

R Ct1 -- Ct2

R = random assignment of subjects to groupsT = Treatment group, C = Control groupt1 = first measurement, before experimental treatment

(pretest), t2 = 2nd measurement, after experimental treatment (post-test)

Post-test-only control group design

X Tt1

R -- Ct1

Many variations on this experimental design:

See www.socialresearchmethods.net - various experimental designs.

The Solomon four group design

Tat1 X Tat2

Cat1 -- Cat2

R X Tbt1

-- Cbt1

Outcome I

Outcome II

Source: www.socialresearchmethods.net

Factorial design

The Main Effects and Interaction Effects

• A main effect is an outcome that is a consistent difference between levels of a factor.

• Because of the way we combine levels in factorial design, it also enables us to examine the interaction effects that exist between factors. An interaction effect exists when differences on one factor depend on the level of another factor.

Explanation of effects, I

Explanation of effects, II

Explanation of effects, III

Explanation of effects, IV

Explanation of effects, V

Quasi-experimental design (QED)

QED takes an experimental approach without having full experimental control; lacks some feature of true experiments, generally randomization

• - Separate sample pretest-posttest design (both groups receive the treatment, but one is tested before the treatment, and the other afterward)

• - None-equivalent control group design; control group but without random assignment;

• - Interrupted time series-design: series of observations before and after the treatment;

• - Multiple time series design: extends the one above by adding a series of observations on a non-equivalent control group.

Experimental Design in Surveys

• Differently worded questions: As part of a larger survey: direct a question with one wording to a randomly selected sub-sample of respondents; and a differently worded question to the remaining respondents.

• Vignettes: Short, detailed and concrete descriptions of situations that contain references to the factors that are thought to be important in decision making. To test experimentally which factors affect the subject’s decision making, a number of versions of the vignettes are developed by systematically varying the details of the situation. The different versions are then randomly assigned to respondents.

• Constructing different sets of questionnaires

ETHICAL ISSUES IN RESEARCH

Basic question:

For what to to whoe the researchers in the social sciences are resposible?

Human subjests

Peers

Society

Ethics of treatment of human subjects

• - Ethical considerations preclude withholding treatment from a control group: education, correction, clinical psychology, medicine frequently the case.

• Institutional review boards assess the ethics of the

proposed research (including on risk vs. benefits). Overall guiding principle: potential benefits of research should be weighed against potential costs (risks).

General rules:

• - Researcher should not expose participants to substantial risk of physical or psychological harm, unless the benefits of participation exceed the risks, and the subject knowingly chooses to participate;

• - Subjects should be informed that their participation is voluntary throughout the entire research; should be told about any aspects of the research that may influence their willingness to participate;

• - If deception is necessary- appropriate debriefing as soon as possible after their participation;

• - Confidentiality and anonymity: researchers should use all possible means to protect the confidentiality of information participants provide (identity).

Ethics of data collection and analysis

• - Honesty and accuracy in collection & analysis of data, and reporting of results

• - Scientific misconduct, if identified, kills researcher’s career

Ethics of responsibility to society

• Awareness of potential use and misuse of scientific results by others