Copyright © Allyn & Bacon (2007) Controls to Reduce Threats to Validity Graziano and Raulin Research Methods: Chapter 9 This multimedia product and its

Copyright © Allyn & Bacon (2007)Copyright © Allyn & Bacon (2007)

Controls to ReduceControls to ReduceThreats to ValidityThreats to Validity

Graziano and RaulinGraziano and RaulinResearch Methods: Chapter 9Research Methods: Chapter 9This multimedia product and its contents are protected under copyright law. The following are This multimedia product and its contents are protected under copyright law. The following are prohibited by law: (1) Any public performance or display, including transmission of any image prohibited by law: (1) Any public performance or display, including transmission of any image over a network; (2) Preparation of any derivative work, including the extraction, in whole or in over a network; (2) Preparation of any derivative work, including the extraction, in whole or in part, of any images; (3) Any rental, lease, or lending of the program.part, of any images; (3) Any rental, lease, or lending of the program.


Threats to ValidityThreats to Validity

Covered in Chapter 8Covered in Chapter 8 Validity can be threatened in many Validity can be threatened in many

waysways– Presence of confounding variablesPresence of confounding variables– Unrepresentative samplesUnrepresentative samples– Inappropriate statistical tests or violations Inappropriate statistical tests or violations

of statistics assumptionsof statistics assumptions– Subject and experimenter effectsSubject and experimenter effects

All these threats can be controlledAll these threats can be controlled


Control ProceduresControl Procedures

General control procedures General control procedures (applicable to virtually all research)(applicable to virtually all research)

Control over subject and Control over subject and experimenter effectsexperimenter effects

Control through the selection and Control through the selection and assignment of participantsassignment of participants

Control through experimental designControl through experimental design


General Control General Control ProceduresProcedures Preparation of the settingPreparation of the setting

– Free of distractions that might interfereFree of distractions that might interfere– A natural setting increases external A natural setting increases external

validityvalidity Response MeasurementResponse Measurement

– Use reliable and valid measuresUse reliable and valid measures ReplicationReplication

– Demonstrates that findings are consistent Demonstrates that findings are consistent and robustand robust


Types of ReplicationTypes of Replication

Exact ReplicationExact Replication– Repeating a study using identical Repeating a study using identical

procedures to the originalprocedures to the original Systematic ReplicationSystematic Replication

– Using a theoretical or procedural Using a theoretical or procedural changechange

Conceptual ReplicationConceptual Replication– Varying the operational definitions of Varying the operational definitions of

the variables to get new research the variables to get new research hypotheseshypotheses


Subject andSubject andExperimenter EffectsExperimenter Effects Blind proceduresBlind procedures

– Best control for expectancy effectsBest control for expectancy effects– Single-blindSingle-blind: The experimenter : The experimenter

does not know what condition the does not know what condition the participant is inparticipant is in

– Double-blindDouble-blind: Neither the : Neither the experimenter nor the participant experimenter nor the participant knows what condition the participant knows what condition the participant is inis in


Subject andSubject andExperimenter EffectsExperimenter Effects AutomationAutomation

– Reduces contact between participants and Reduces contact between participants and the experimenterthe experimenter

– Gives the experimenter less opportunity to Gives the experimenter less opportunity to affect participantsaffect participants

Using objective measuresUsing objective measures– Objective measure require less judgmentObjective measure require less judgment– Provides less opportunity for subtle Provides less opportunity for subtle

experimenter biases to affect the dataexperimenter biases to affect the data


Subject andSubject andExperimenter EffectsExperimenter Effects Multiple observersMultiple observers

– Reduces bias because it challenges Reduces bias because it challenges observers to be as objective as possibleobservers to be as objective as possible

– Can measure amount of observer Can measure amount of observer agreement (percent agreement or Kappa)agreement (percent agreement or Kappa)

Using deceptionUsing deception– Hides purpose of the study from Hides purpose of the study from

participantsparticipants– Balanced placebo design is a good Balanced placebo design is a good

exampleexample


Balanced Placebo Balanced Placebo DesignDesign Separates the pharmacological Separates the pharmacological

effects from the expectancy effects from the expectancy effects of alcoholeffects of alcohol

A two-factor design A two-factor design – Factor 1 is whether the person Factor 1 is whether the person

drinks alcoholdrinks alcohol– Factor 2 is whether the person Factor 2 is whether the person

thinks he or she is drinking alcoholthinks he or she is drinking alcohol


Balanced Placebo Balanced Placebo DesignDesignThis design crosses theThis design crosses theconsumption of alcoholconsumption of alcoholwith the belief that with the belief that alcoholalcoholis being consumedis being consumed

People LedPeople Ledto Believeto Believe

DrinkingDrinkingAlcoholAlcohol

Not Not DrinkingDrinkingAlcoholAlcohol

ActualActualSituatioSituatio

nn

DrinkingDrinkingAlcoholAlcohol

Not Not DrinkingDrinkingAlcoholAlcohol


Participant SelectionParticipant Selection

Can generalize only if your sample is Can generalize only if your sample is representativerepresentative

Populations and samplesPopulations and samples– General populationGeneral population: all potential : all potential

participantsparticipants– Target populationTarget population: those participants you : those participants you

are interested inare interested in– Accessible populationAccessible population: portion of target : portion of target

population that is available to the researcherpopulation that is available to the researcher– SampleSample: drawn from the accessible : drawn from the accessible

populationpopulation


Populations and Populations and SamplingSampling This figure shows This figure shows

the relationship the relationship between the between the various various populationspopulations– General PopulationGeneral Population– Target PopulationTarget Population– Accessible Accessible

PopulationPopulation– SampleSample


Sampling ProceduresSampling Procedures

Random samplingRandom sampling– Every participant has an equal chance of being Every participant has an equal chance of being

sampledsampled

Stratified random samplingStratified random sampling– Random sampling within strata (subgroups)Random sampling within strata (subgroups)

Ad hoc samplesAd hoc samples– Random sample from accessible populationRandom sample from accessible population– Must generalize cautiouslyMust generalize cautiously– Should describe sample to help define limits of Should describe sample to help define limits of

generalizationgeneralization


Participant Participant AssignmentAssignment Critical to internal validityCritical to internal validity Assignment ProceduresAssignment Procedures

– Free random assignmentFree random assignment Random assignment of participants to groupsRandom assignment of participants to groups

– Matched random assignmentMatched random assignment Random assignment of participants in matched sets Random assignment of participants in matched sets

to groupsto groups

– Other matching proceduresOther matching procedures e.g., match groups on key characteristicse.g., match groups on key characteristics


Matched Random Matched Random AssignmentAssignment Match on relevant variables Match on relevant variables

– Variables likely to affect the dependent measureVariables likely to affect the dependent measure– Variables that show the largest variability in the Variables that show the largest variability in the

populationpopulation

ProceduresProcedures– Match in sets on the relevant variableMatch in sets on the relevant variable

Set size is the number of groups in the studySet size is the number of groups in the study– Randomly assign participants from the set, one to Randomly assign participants from the set, one to

each groupeach group– Keep track of matching data for the statistical Keep track of matching data for the statistical

analysisanalysis


Experimental DesignExperimental Design

Main focus of Chapters 10 through Main focus of Chapters 10 through 1313

Experimental design maximizes Experimental design maximizes validity validity – Need to also include the other control Need to also include the other control

procedures covered in this chapterprocedures covered in this chapter

Key elements of experimentsKey elements of experiments– One or more control groupsOne or more control groups– Random assignment of participants to groupsRandom assignment of participants to groups


SummarySummary

Most threats to validity can be minimized Most threats to validity can be minimized with proper use of control procedureswith proper use of control procedures

Broad classes of control proceduresBroad classes of control procedures– General control proceduresGeneral control procedures– Control over subject and experimenter effectsControl over subject and experimenter effects– Control through participant selection and Control through participant selection and

assignmentassignment– Control through specific experimental designControl through specific experimental design

Documents

Copyright © Allyn & Bacon (2007) Controls to Reduce Threats to Validity Graziano and Raulin Research Methods: Chapter 9 This multimedia product and its