Experimental DesignExperimental Designmaking causal making causal

inferencesinferences

Causal and EffectCausal and Effect

The IV precedes the DV in The IV precedes the DV in timetime

The IV and DV are correlatedThe IV and DV are correlatedThere are no plausible There are no plausible

additional variables that additional variables that could reasonably explain the could reasonably explain the correlationcorrelation

The Gold StandardThe Gold StandardWhat are the differences between these What are the differences between these

types of studies?types of studies?

True experimentsTrue experimentsQuasi experimentsQuasi experimentsObservational studiesObservational studiesSurveysSurveys

Why is it so hard to do in education?Why is it so hard to do in education?

The Gold StandardThe Gold Standard

True experiments versus True experiments versus observational studies, or observational studies, or quasi-experimentsquasi-experiments

The Randomized Clinical The Randomized Clinical Trial modelTrial model

Why is it so hard to do in Why is it so hard to do in education?education?

The Gold StandardThe Gold Standard

True experimentsTrue experiments– Random assignment to treatment Random assignment to treatment

and control conditionsand control conditions– The Gold Standard for causal The Gold Standard for causal

inferencesinferences– The treatment is under the The treatment is under the

control of the researchercontrol of the researcher

The Gold StandardThe Gold Standard

Quasi experimentsQuasi experiments– The treatment is under the The treatment is under the

control of the researchercontrol of the researcher– As many of the characteristics of As many of the characteristics of

a true experiment as possiblea true experiment as possible– Intact groups, random Intact groups, random

assignment is not possibleassignment is not possible

The Gold StandardThe Gold Standard

Observational studiesObservational studies– Intact groups, not randomly Intact groups, not randomly

assignedassigned– If there is a treatment, it may not If there is a treatment, it may not

be under the control of the be under the control of the researcherresearcher

– Correlations between variables, Correlations between variables, not causality not causality

The Gold StandardThe Gold Standard

SurveysSurveys– The focus is on estimation of The focus is on estimation of

parameters, not the effects of a parameters, not the effects of a treatmenttreatment

– Snap shot in timeSnap shot in time– Self-report, subjective perceptions of Self-report, subjective perceptions of

respondents, not objective or externalrespondents, not objective or external– Not designed to provide hard outcomesNot designed to provide hard outcomes

The Essential The Essential CharacteristicsCharacteristicsRandom SelectionRandom Selection

Random AssignmentRandom Assignment

Manipulation of the IVManipulation of the IV

Control ConditionControl Condition

Additional Factors to Additional Factors to ConsiderConsider Placebo or Comparison Group(s)Placebo or Comparison Group(s)

    Multiple Measurements Over Multiple Measurements Over TimeTime

  Control Over Confounding Control Over Confounding Variables Variables 

Controlling ConfoundsControlling Confounds

The experimental The experimental environmentenvironment

  Admissibility criteriaAdmissibility criteria

  Blocking Blocking 

BlockingBlocking

Creates homogeneous subsetsCreates homogeneous subsets

  Builds potential confounds into the Builds potential confounds into the design design

  Reduces error termReduces error term

Makes a more sensitive and Makes a more sensitive and therefore powerful experimenttherefore powerful experiment

Additional Design FeaturesAdditional Design Features

Measurement of Compliance Measurement of Compliance or Implementation Effectsor Implementation Effects

BlindingBlindingMeasurement of OutcomesMeasurement of OutcomesIntegrity of Data Collection, Integrity of Data Collection,

Entry, and Reporting Entry, and Reporting ProceduresProcedures

Ethical ConsiderationsEthical Considerations

Informed consentInformed consent PrivacyPrivacy Do no harmDo no harm Denying servicesDenying services InducementsInducements Dual rolesDual roles

2003 #42003 #4

Rubric for Part ARubric for Part A

Identify a plausible example of a problemIdentify a plausible example of a problem– ““Because a deadline has been moved back...”Because a deadline has been moved back...”

Relate the identified problem to the change in Relate the identified problem to the change in stress levelstress level– ““...the stress levels of those working in the ...the stress levels of those working in the

department have been lowered...”department have been lowered...”

State that the problem effects can not be State that the problem effects can not be distinguished from the treatment effectsdistinguished from the treatment effects– ““...which could be mistakenly attributed to the ...which could be mistakenly attributed to the

treatment.”treatment.”

Rubric for Part ARubric for Part A

Give a reason for the necessity of Give a reason for the necessity of random assignment. random assignment.

State that randomization is relied upon State that randomization is relied upon to create comparable groups. to create comparable groups.

State that randomization helps reduce State that randomization helps reduce the influence of potential confounding the influence of potential confounding variables.variables.

Rubric for Part ARubric for Part A

““Without random assignment of volunteers Without random assignment of volunteers to the two programs, it is possible that the to the two programs, it is possible that the two treatment groups could differ in some two treatment groups could differ in some way that affects the outcome of the way that affects the outcome of the experiment. Randomization “evens out” experiment. Randomization “evens out” the possible effects of potentially the possible effects of potentially confounding variables.”confounding variables.”

Rubric for Part BRubric for Part B Indicate that a control group does provide Indicate that a control group does provide

additional informationadditional information

Explain that the control group allows the Explain that the control group allows the company to determine if either or both company to determine if either or both treatments are effective in reducing stresstreatments are effective in reducing stress

Explain that the control group provides a Explain that the control group provides a baseline for comparison, an indication of baseline for comparison, an indication of what might have happened anyway, even what might have happened anyway, even without the treatmentwithout the treatment

Rubric for Part BRubric for Part B

““Without the control group, the company Without the control group, the company could compare the two treatments, but would could compare the two treatments, but would not be able to say whether the observed not be able to say whether the observed reduction in stress was attributable to reduction in stress was attributable to participation in the programs. For example, a participation in the programs. For example, a change in the work environment during this change in the work environment during this period might have reduced the stress level of period might have reduced the stress level of all employees. The addition of a control all employees. The addition of a control group would enable the company to assess group would enable the company to assess the magnitude of the mean reduction the magnitude of the mean reduction attributable to each treatment, as opposed to attributable to each treatment, as opposed to just determining if the two programs differ.”just determining if the two programs differ.”

Rubric for Part CRubric for Part C

Indicate that one cannot generalize, Indicate that one cannot generalize, and give a plausible reason, such and give a plausible reason, such as...as...

The participants were volunteers The participants were volunteers and volunteers my not be and volunteers my not be representative of the populationrepresentative of the population

The participants were not randomly The participants were not randomly selected from the populationselected from the population

Rubric for Part CRubric for Part C

““No it is not, for this experiment we No it is not, for this experiment we took volunteers but the problem took volunteers but the problem with it is that the people who with it is that the people who volunteered are very likely the ones volunteered are very likely the ones who needed the stress reduction the who needed the stress reduction the most...Therefore, it is not reasonable most...Therefore, it is not reasonable to generalize because most likely to generalize because most likely the people who volunteered are not the people who volunteered are not representative of the population.”representative of the population.”

Common Student ErrorsCommon Student Errors

Did not understand the difference Did not understand the difference between random allocation of subjects between random allocation of subjects and random sampling.and random sampling.

Often used the word "confounding" in Often used the word "confounding" in part (a), but did not explain how the part (a), but did not explain how the treatment results were mixed up with treatment results were mixed up with some other variable.some other variable.

Common Student ErrorsCommon Student Errors

Seemed to think that a larger sample Seemed to think that a larger sample size would fix any problem in the size would fix any problem in the experiment, rather than recognizing experiment, rather than recognizing that the major problem of the that the major problem of the experiment was that there was no experiment was that there was no random sampling of employees. random sampling of employees.

Incorrectly stated that random Incorrectly stated that random allocation "eliminates" bias. allocation "eliminates" bias.

2001 #42001 #4

Rubric for Part ARubric for Part A

Blocking Scheme A is preferableBlocking Scheme A is preferable

Creates homogeneous blocks with Creates homogeneous blocks with respect to forest exposurerespect to forest exposure

Plots will have similar forest Plots will have similar forest exposureexposure

Rubric for Part BRubric for Part B

Randomization within blocks Randomization within blocks should reduce bias due to the should reduce bias due to the influence of confounding influence of confounding variablesvariables

(fertility of soil, moisture, etc.)(fertility of soil, moisture, etc.) on the productivity of the on the productivity of the

trees.trees.

2001 FR #4 Rubric

Extension QuestionsExtension Questions

How would you randomize How would you randomize trees within blocks?trees within blocks?

What other confounding What other confounding variables might impact the variables might impact the results?results?

Extension QuestionsExtension Questions

How does this concept of How does this concept of blocking apply to educational blocking apply to educational evaluation?evaluation?

StudentsStudents ClassroomsClassrooms SchoolsSchools