Types of Studies

Comparing observational studies

and experiments …

Comparing Two (or More!) Populations

Population #1 Population #2

How do the populations differ?

Do their average “whatevers” differ?

Do the percentages having a “whatever” differ?

The strength of the conclusion that can be made depends on the type of the study conducted.

Types of Studies

• Experiments

• Observational studies

Experiment

• An “experiment” deliberately imposes some “treatments” on individuals to see if the treatments have an effect on some “outcome” or measurement.

• The different treatments define the populations being compared.

Example of an experiment

Ascorbic Acid Placebo

Do the average number of colds differ?

Do their average lengths of colds differ?

Outcomes:

Population #1 Population #2

And another example ...

Cool shoes Not so cool shoes

Do the average number of “dates” differ?

Do the percentage of “successful dates” differ?

Outcomes:

Male population #1 Male population #2

Do “shoes really make the man”?

Who gets which treatment?

• To conduct a good experiment, “treatment assignments” must be “random.”

• “Random” means everybody has an equal chance of getting a treatment.

• “Haphazard” does not mean random!

Who is in control?

• Every experiment should have a “control group.”

• People in control group are treated exactly the same way as the other people in the experiment, except they do not get the “active treatment.”

• A “placebo group” is a special kind of control group.

How many suckers?

• You cannot conduct a good experiment on just a few people.

• The number of suckers (people) necessary is determined by power considerations.

• Ask a statistician.

Close your eyes!

• To conduct a good experiment, individuals should be “blinded” to the treatment assignments.

• An experiment is “double-blinded,” if neither the researcher nor the individuals know who received what.

Can this be an experiment?

Males Females

Do the average number of telephone minutes per day differ?

Do the percentage on steroids differ?

Outcomes:

Population #1 Population #2

Observational study

• An “observational study” observes the natural characteristics of individuals to see if the characteristics have an effect on some outcome or measurement.

• The natural characteristics define the populations being compared.

• Observational studies may be necessary for obvious reasons or for ethical reasons.

Example of an observational study

Townies Gownies

Do the average number of parking tickets differ?

Do the percentage of towed vehicles differ?

Outcomes:

Population #1 Population #2

And another example …

Drive and talk Don’t drive and talk

Does the risk of having an accident differ?

Outcome:

Population #1 Population #2

Advantages of experiments

• Randomization should make the two populations similar, on average, with respect to everything except the treatment.

• So if outcomes are different for the two populations, can conclude that it is the treatment that caused it.

Disadvantages of observational studies

• The populations may differ in many ways besides the natural characteristics of interest.

• Cannot conclude that the primary characteristic of interest is causing the difference in the outcome.

• One of the other characteristics may be causing the difference.

So, some guidelines ...

• Whenever possible, conduct a randomized, blinded, controlled experiment.

• If you cannot conduct an experiment for obvious or ethical reasons, then perform an observational study.

• But if you conduct an observational study, be careful of how strong you state your conclusion.