Research Methods

Research

• In your “Testing Claims” project you will be investigating research studies that back up a claim.

• You use the C.R.A.P. test to evaluate sources, but what do you do if you find a research study from what appears to be a reliable source, but may have other problems.

• How do you analyze the problems that may exist in a research study?

Strength of Evidence

Weak Strong

Sample size (1 pt)Small sample size Large sample size

Sample selection (1 pt)Non-random sample Random sample

Confounds (1 pt)Significant Confounds Insignificant Confounds

Researcher Bias (1 pt)Researcher Bias Present Researcher Bias Absent

• Based on results from a recent experiment, Dr. Lite claims that she has found a way to prevent 95% of hair loss in men and women (aged 25-80), with no adverse side effects. You are interested in investing in Dr. Lite’s product, but first you need to determine the quality of her study. You would like to make sure that Dr. Lite’s research findings would replicate if the study was repeated with a different sample. You know that research findings from studies with rigorous methods are more likely to replicate, so you decide to ask Dr. Lite a few questions.

Example:

Which of the following questions about the study’s research methodology would you ask Dr. Lite first?

A. Did the participants think the treatment was easy to use?

B. How did you select your sample?

C. Did participants report hair regrowth?

D. Was there a significant difference between treatment and control groups?

Clicker Question 1Clicker Question 1

After learning that Dr. Lite randomly selected participants (50% female, aged 25-80) from the population of interest, what would you ask Dr. Lite next?

A. How many participants were in your sample?B. Was this a convenience sample?C. Were subjects paid to participate?D. Did family members notice a change in the

participants’ hair following the treatments?

Clicker Question 2Clicker Question 2

Dr. Lite informs you that there were approximately 1000 participants in the study. Half of the participants were in the experimental group (used the treatment) and half of the participants were in the control group (used a placebo treatment). What would you want to ask Dr. Lite next?

A. Was there a significant difference in hair loss between the treatment and control group?

B. How many weeks did the treatment effects last?C. How were subjects assigned to conditions?D. Were there any confounds in this study?

Clicker Question 3Clicker Question 3

You learn that the 1000 participants were randomly assigned to the experimental and control groups.

A. Were the investigators and participants blind to the study conditions?

B. How much does the treatment cost?

C. Does the treatment effectiveness change depending on initial hair thickness?

D. Who funded this study?

Clicker Question 4Clicker Question 4

Dr. Lite informs you that the researchers and participants were blind to the conditions of the study (i.e., double blind). What might you ask next?

A. How many researchers helped with this study?B. How many weeks did the treatment effects last?C. Was there a significant difference in hair loss

between the treatment and control group?D. Were the procedures for the experimental and

control group the same, with the exception of the actual treatment?

Clicker Question 5Clicker Question 5

After learning that Dr. Lite used strict experimental control, and that the conditions for each group were exactly the same, how would you rate the strength of the evidence provided by this study’s findings?

1. very weak 2. moderately weak3. moderately strong4. very strong

Clicker Question 6Clicker Question 6

Weak Strong

Small sample size Large sample size

Non-random sample Random sample

Significant Confounds Insignificant Confounds

Researcher Bias Present Researcher Bias Absent

Your Project:

• Use one of the references from your project with the scale below to evaluate the strength of the evidence reported in the study.

• For each of the 4 categories, explain why you thought the study earned it’s +1 point rating.

• Then provide an overall rating out of 4.Weak

StrongSmall sample size Large sample size

Non-random sample Random sample

Significant Confounds Insignificant Confounds

Researcher Bias Present Researcher Bias Absent