Experimental Design making causal inferences Richard Lambert, Ph.D

Experimental DesignExperimental Designmaking causal making causal

inferencesinferences

Richard Lambert, Ph.D. Richard Lambert, Ph.D.

The Gold StandardThe Gold Standard

What are the differences What are the differences between these types of between these types of studies?studies?

True experimentsTrue experimentsObservational studiesObservational studiesSurveysSurveys

Observational StudiesObservational Studies Intact groups, not randomly assignedIntact groups, not randomly assigned

If there is a treatment, it may not be If there is a treatment, it may not be under the control of the researcherunder the control of the researcher

The goal is often just to look at The goal is often just to look at relationships between variables relationships between variables

Correlation not causalityCorrelation not causality

SurveysSurveysPopulation is definedPopulation is defined

Sample is selected from the population Sample is selected from the population so that is can be representative of the so that is can be representative of the population – SRS, Stratified RSpopulation – SRS, Stratified RS

The focus is on estimation of The focus is on estimation of parameters, not the effects of a parameters, not the effects of a treatmenttreatment

SurveysSurveys

Compare and contrast SRS and Stratified Compare and contrast SRS and Stratified RSRS

Both give everyone in the population an Both give everyone in the population an equal chance of selection AND give every equal chance of selection AND give every possible sample of size n an equal possible sample of size n an equal probability of selectionprobability of selection

For Stratified RS, sampling is done within For Stratified RS, sampling is done within homogeneous stratahomogeneous strata

SurveysSurveys For Stratified RS, we divide the For Stratified RS, we divide the

population into homogeneous population into homogeneous subgroups according to some variable subgroups according to some variable we expect to be related to the survey we expect to be related to the survey questionsquestions

We then sample within strataWe then sample within strata

Compare and contrast Stratified SRS Compare and contrast Stratified SRS and and BlockingBlocking

True ExperimentsTrue Experiments

The Gold Standard for Causal The Gold Standard for Causal InferenceInference

Random assignment to treatment Random assignment to treatment and control conditionsand control conditions

A treatment is imposed on the A treatment is imposed on the subjects, the treatment is under subjects, the treatment is under the control of the researcherthe control of the researcher

Treatment is ImposedTreatment is Imposed

Presence or Absence of Presence or Absence of TreatmentTreatment

Dosage Level Controlled By Dosage Level Controlled By ResearcherResearcher

Multiple Types of Treatment Multiple Types of Treatment ConditionsConditions

True ExperimentsTrue Experiments

The Gold Standard for Causal The Gold Standard for Causal InferenceInference

Why is it so hard to do in Why is it so hard to do in education?education?

Is it the only “way of knowing” in Is it the only “way of knowing” in educational research?educational research?

Causal and EffectCausal and Effect

The IV precedes the DV in timeThe IV precedes the DV in time

The IV and DV are relatedThe IV and DV are related

There are no plausible There are no plausible additional variables that could additional variables that could reasonably explain the reasonably explain the relationshiprelationship

Causal and EffectCausal and Effect

When we say the IV and DV are When we say the IV and DV are related, think of a dose – response related, think of a dose – response relationship in a drug study.relationship in a drug study.

The amount of treatment The amount of treatment exposure is related to a exposure is related to a corresponding amount of some corresponding amount of some effect.effect.

The Essential The Essential CharacteristicsCharacteristicsRandom SelectionRandom Selection

Random AssignmentRandom Assignment

Treatment is ImposedTreatment is Imposed

Control ConditionControl Condition

Additional Factors to Additional Factors to ConsiderConsider Placebo or Comparison Group(s)Placebo or Comparison Group(s)

Multiple Measurements Over Multiple Measurements Over TimeTime

Control Over Confounding Control Over Confounding Variables Variables

Controlling ConfoundsControlling Confounds

The Experimental The Experimental EnvironmentEnvironment

Admissibility criteriaAdmissibility criteria

Blocking Blocking

BlockingBlocking

Creates homogeneous subsetsCreates homogeneous subsets

Builds potential confounds into the Builds potential confounds into the design design

Reduces error termReduces error term

Makes a more sensitive and Makes a more sensitive and therefore therefore powerful experimentpowerful experiment

BlockingBlocking

When using blocking, be sure to:When using blocking, be sure to:

Divide the sample into Divide the sample into homogeneous subsetshomogeneous subsets

Randomly assign subjects to Randomly assign subjects to treatments treatments within blockswithin blocks

Experimental ValidityExperimental Validity

Internal ValidityInternal Validity

External ValidityExternal Validity

Construct ValidityConstruct Validity

Statistical Conclusion ValidityStatistical Conclusion Validity

Released QuestionsReleased Questions

Work on 1999 Free Work on 1999 Free Response #3 with your Response #3 with your partnerpartner

Released Questions – Released Questions – 1999 #31999 #3

3. The dentists in a dental clinic would like to determine if there is a difference 3. The dentists in a dental clinic would like to determine if there is a difference between the number of new cavities in people who eat an apple a day and in between the number of new cavities in people who eat an apple a day and in people who eat less than one apple a week. They are going to conduct a study people who eat less than one apple a week. They are going to conduct a study with 50 people in each group.with 50 people in each group.

Fifty clinic patients who report that they routinely eat an apple a day and 50 clinic Fifty clinic patients who report that they routinely eat an apple a day and 50 clinic patients who report that they eat less than one apple a week will be identified. patients who report that they eat less than one apple a week will be identified. The dentists will examine the patients and their records to determine the The dentists will examine the patients and their records to determine the number of new cavities the patients have had over the past two years. They will number of new cavities the patients have had over the past two years. They will then compare the number of new cavities in the two groups.then compare the number of new cavities in the two groups.

Why is this an observational study and not an experiment?Why is this an observational study and not an experiment?

Explain the concept of confounding in the context of this study. Include an example Explain the concept of confounding in the context of this study. Include an example of a possible confounding variable.of a possible confounding variable.

If the mean number of new cavities for those who ate an apple a day was statistically If the mean number of new cavities for those who ate an apple a day was statistically smaller than the mean number of new cavities for those who ate less than one smaller than the mean number of new cavities for those who ate less than one apple a week, could one conclude that the lower number of new cavities can be apple a week, could one conclude that the lower number of new cavities can be attributed to eating an apple a day? Explain.attributed to eating an apple a day? Explain.

The Essential The Essential CharacteristicsCharacteristicsRandom SelectionRandom Selection

Random AssignmentRandom Assignment

Treatment is imposedTreatment is imposed

Control ConditionControl Condition

Confounding VariablesConfounding VariablesHave to be related to the DVHave to be related to the DV

Are mixed up with, or inseparable Are mixed up with, or inseparable from Group membershipfrom Group membership

Create non-equivalence of treatment Create non-equivalence of treatment groupsgroups

Disrupt your ability to conclude that Disrupt your ability to conclude that the treatment and only the the treatment and only the treatment caused the outcomestreatment caused the outcomes



2003 Free Response #4

2003 #42003 #4

Rubric for Part ARubric for Part A

Identify a plausible example of a problemIdentify a plausible example of a problem– ““Because a deadline has been moved back...”Because a deadline has been moved back...”

Relate the identified problem to the change in Relate the identified problem to the change in stress levelstress level– ““...the stress levels of those working in the ...the stress levels of those working in the

department have been lowered...”department have been lowered...”

State that the problem effects can not be State that the problem effects can not be distinguished from the treatment effectsdistinguished from the treatment effects– ““...which could be mistakenly attributed to the ...which could be mistakenly attributed to the

treatment.”treatment.”


Give a reason for the necessity of Give a reason for the necessity of random assignment. random assignment.

State that randomization is relied upon State that randomization is relied upon to create comparable groups. to create comparable groups.

State that randomization helps reduce State that randomization helps reduce the influence of potential confounding the influence of potential confounding variables.variables.


““Without random assignment of volunteers Without random assignment of volunteers to the two programs, it is possible that the to the two programs, it is possible that the two treatment groups could differ in some two treatment groups could differ in some way that affects the outcome of the way that affects the outcome of the experiment. Randomization “evens out” experiment. Randomization “evens out” the possible effects of potentially the possible effects of potentially confounding variables.”confounding variables.”

Rubric for Part BRubric for Part B Indicate that a control group does provide Indicate that a control group does provide

additional informationadditional information

Explain that the control group allows the Explain that the control group allows the company to determine if either or both company to determine if either or both treatments are effective in reducing stresstreatments are effective in reducing stress

Explain that the control group provides a Explain that the control group provides a baseline for comparison, an indication of baseline for comparison, an indication of what might have happened anyway, even what might have happened anyway, even without the treatmentwithout the treatment

Rubric for Part BRubric for Part B

““Without the control group, the company Without the control group, the company could compare the two treatments, but would could compare the two treatments, but would not be able to say whether the observed not be able to say whether the observed reduction in stress was attributable to reduction in stress was attributable to participation in the programs. For example, a participation in the programs. For example, a change in the work environment during this change in the work environment during this period might have reduced the stress level of period might have reduced the stress level of all employees. The addition of a control all employees. The addition of a control group would enable the company to assess group would enable the company to assess the magnitude of the mean reduction the magnitude of the mean reduction attributable to each treatment, as opposed to attributable to each treatment, as opposed to just determining if the two programs differ.”just determining if the two programs differ.”

Rubric for Part CRubric for Part C

Indicate that one cannot generalize, Indicate that one cannot generalize, and give a plausible reason, such and give a plausible reason, such as...as...

The participants were volunteers The participants were volunteers and volunteers my not be and volunteers my not be representative of the populationrepresentative of the population

The participants were not randomly The participants were not randomly selected from the populationselected from the population

Rubric for Part CRubric for Part C

““No it is not, for this experiment we No it is not, for this experiment we took volunteers but the problem took volunteers but the problem with it is that the people who with it is that the people who volunteered are very likely the ones volunteered are very likely the ones who needed the stress reduction the who needed the stress reduction the most...Therefore, it is not reasonable most...Therefore, it is not reasonable to generalize because most likely to generalize because most likely the people who volunteered are not the people who volunteered are not representative of the population.”representative of the population.”

Common Student ErrorsCommon Student Errors

Did not understand the difference Did not understand the difference between random allocation of subjects between random allocation of subjects and random sampling.and random sampling.

Often used the word "confounding" in Often used the word "confounding" in part (a), but did not explain how the part (a), but did not explain how the treatment results were mixed up with treatment results were mixed up with some other variable.some other variable.

Common Student ErrorsCommon Student Errors

Seemed to think that a larger sample Seemed to think that a larger sample size would fix any problem in the size would fix any problem in the experiment, rather than recognizing experiment, rather than recognizing that the major problem of the that the major problem of the experiment was that there was no experiment was that there was no random sampling of employees. random sampling of employees.

Incorrectly stated that random Incorrectly stated that random allocation "eliminates" bias. allocation "eliminates" bias.



2001 Free Response #4

2001 #42001 #4


Blocking Scheme A is preferableBlocking Scheme A is preferable

Creates homogeneous blocks with Creates homogeneous blocks with respect to forest exposurerespect to forest exposure

Plots will have similar forest Plots will have similar forest exposureexposure

Rubric for Part BRubric for Part B

Randomization within blocks Randomization within blocks should reduce bias due to the should reduce bias due to the influence of confounding influence of confounding variablesvariables

(fertility of soil, moisture, etc.)(fertility of soil, moisture, etc.) on the productivity of the on the productivity of the

trees.trees.

2001 FR #4 Rubric

Extension QuestionsExtension Questions

How would you randomize How would you randomize trees within blocks?trees within blocks?

What other confounding What other confounding variables might impact the variables might impact the results?results?

Documents

Experimental Design making causal inferences Richard Lambert, Ph.D