Experiments:Validity, Reliability and Other Design Considerations

At least three different meanings:

▪ Controlled Studies▪ Control in an

Experiment▪ Controls used in a

study or analysis

2

Assigning people to groups vs. observing people who ‘assign’ themselves

Example of pitfalls in experimental assignment:

▪ Portacaval Shunt Example

Examples of key pitfalls of observational studies:

▪ Cervical cancer and circumcision study

▪ Alcohol consumption and lung cancer

3

Measurement Concerns Construct Validity

Design Concerns Internal Validity External Validity Ecological Validity

4

“Face” validity deals with subjective judgement of appropriate operationalization

“Content” validity is a more direct check against relevant content domain for the given construct.

5

How do we know that our independent variable is reflecting the intended causal construct and nothing else?

6

Internal Validity deals with questions about whether changes in the dependent variable were caused by the treatment.

7

Effect

? ?

? ?

History ▪ additional I.V. that occurs

between pre-test and post-test

Maturation▪ Subjects simply get older and

change during experiment

Testing▪ Subjects “get used” to being

tested

Regression to the Mean▪ Issue with studies of extremes on

some variable8

Demand Characteristics▪ Anything in the experiment that could guide subjects to expected outcome

Experimenter Expectancy▪ Researcher behavior that guides subjects to expected outcome (self-fulfilling prophecy)

9

Evaluation Apprehension

“Hawthorne Effect” Temporary improvement based on

observation

Solutions Double-blind experiments Experiments in natural setting (i.e.,

subjects do not know they are in an experiment)

Cover stories Hidden measurements

10

Naïve experimenter Those conducting study are not aware of theory or

hypotheses in the experiment Blind

Researcher is unaware of the experiment condition that he/she is administering

Standardization Experimenter follows a script, and only the script

“Canned” Experimenter Audio/Video/Print material gives instructions

11

Selection Bias▪ Issue with non-random selection of subjects

Mortality▪ Departure of subjects in the experiment

Diffusion, Sharing of Treatments▪ Control group unexpectedly obtains treatment

Other ‘social’ threats?▪ Compensatory rivalry, resentful demoralization, etc.

12

Setting

Population

History

13

External Validity– How far does the given experiment generalize to similar groups, individuals, etc?

14

15

Internal Validity

External & Ecological Validity

Balance is important between the types of validity, but internal validity is usually (if not always) the more important factor.

Some experiments can be conducted in a real-world setting while maintaining random assignment and manipulation of treatments

16

Milliman (1986) Study of music tempo and restaurant customer behavior

Cheshire and Antin (2008) Study of Incentives and Contributions of Information in an Online Setting

1998 Total Solar Eclipse: testing temperature of sun’s corona

17

Pro’s Gives researcher tight control over independent factors Allows researcher to test key relationships with as few

confounding factors as possible Allows for direct causal testing

Con’s Often very small N; enough for statistical purposes but

not ideal for generalizability Sometimes give up large amounts of external validity in

favor of construct validity and direct causal analysis Require a large amount of planning, training, and time–

sometimes to test relationship between only 2 factors!

18

Cost and Effort Is the effort worth it to test the concepts you are

interested in?Manipulation and Control

Will you actually be able to manipulate the key concept(s)?

Importance of GeneralizabilityAre you testing theory, or trying to establish a

real-world test?

19