Embed Size (px)
Citation preview
1
Experimental Design
Part IRichard S. Balkin, Ph. D, LPC-S, NCC
Balkin, R. S., 2008
Overview
Experimental design is the blueprint forquantitative research and serves as thefoundation of what makes quantitativeresearch valid.
Too often, consumers of research mayfocus on the rationale for a study and thediscussion of the results. However, withoutproper methods, a quantitative study hasno meaning.
Balkin, R. S., 2008
Foundations of Quantitative Research
Instead of merely focusing on the statistical test, the endresult so to speak, a quantitative study is based on astrong foundation of a generalizable sample, reliable andvalid instrumentation, and appropriate research design
Participant Selection and Assignment
Instrumentation
ExperimentalDesign
StatisticalTest
2
Balkin, R. S., 2008
Consider the following scenario
A school counselor wishes to improve upon a programfor students with low academic achievement. Currently,the school counselor leads a psychoeducational groupon study skills. Based upon the responses from theparticipants, the school counselor believes that studentswith academic problems are having difficulty in areasoutside of study skills. The school counselor decides toimplement a group counseling program in addition to thestudy skills program and wishes to investigate whethergroup counseling would have a greater effect inincreasing academic achievement among theparticipating students.
Balkin, R. S., 2008
Consider the following scenario
A researcher needs to consider how a programor intervention is deemed effective. For example,if the participants obtaining group counselingand study skills interventions are performingbetter, can the school counselor conclude it isdue to the program? Progress could be due toeither intervention, the combined intervention, oranother variable not measured. The merepassage of time could be a reason forimprovement.
Balkin, R. S., 2008
Defining Variables
In an experimental study there are twotypes of variables:
Independent variable (I will abbreviate this asthe IV)
Dependent variable (I will abbreviate this as theDV)
3
Balkin, R. S., 2008
Independent variable
The independent variable in classicalexperimental design is the variable that is beingmanipulated.
In the previous scenario, participants could berandomly assigned to (a) study skills only, (b)group counseling and study skills, and (c) nointervention.
Participation in group counseling and studyskills, study skills only, and no intervention arethe various levels of the independent variable.
Balkin, R. S., 2008
Dependent variable
The dependent variable is the variable thatis being measured
In the previous scenario, participants couldbe measured on change in grade pointaverage (gpa).
Balkin, R. S., 2008
Experimental Validity
Experimental validity refers to the process in whichresults are generalizable because the factors that havebeen tested or manipulated (the independent variable)truly effect a change in an outcome (dependentvariable), and the results of the study can be applied tosettings outside of the experimental setting (Best &Kahn, 2006).
In order to determine the effectiveness of a program orintervention, appropriate methods for sampling (i.e.random sampling), measuring the outcome, andestablishing the validity of the findings are necessary.
4
Balkin, R. S., 2008
Experimental Validity
For example, Kelly, Halford, and Young (2000) studiedthe effects of a short-term intervention related to alcoholabuse. The researchers assigned 16 participants to atreatment group and 16 participants to begin theintervention after one month (control group). After theinitial month, the researchers reported that the treatmentgroup had made statistically significant progress inreducing alcohol and depressive symptoms. The controlgroup did not show any progress with respect to alcoholabuse but did show a statistically significant reduction indepressive symptoms.
Can the researchers conclude that the intervention wasmore successful in treating depressive symptoms?
Balkin, R. S., 2008
Experimental Validity
NO
The researchers cannot conclude that the interventionwas more successful in treating depressive symptoms.Depressive symptoms appeared to diminish whether ornot an intervention occurred. In order to determinewhether a program or intervention is effective,experimental conditions are required. These conditionsmay include the implementation of treatment and controlgroups, random assignment, and measures to insureexperimental validity.
Balkin, R. S., 2008
Random assignment
Random assignment refers to the equallikelihood that a participant will be assigned to atreatment, control, or comparison group.
Random assignment is similar to randomsampling. In random sampling, participants froman accessible population had an equal chance ofbeing selected for a study.
In random assignment, each participant has anequal chance of being selected for a particularintervention or no intervention at all.
5
Balkin, R. S., 2008
Random assignment
In the group counseling/study skillsexample, the school counselor wouldwant to randomly assign participantsto group counseling and study skillsand study skills only. The schoolcounselor may opt to also include acontrol group (no services) but thisshould be handled carefully as thereare ethical issues in denying ordelaying services.
Balkin, R. S., 2008
Why random assignment is important
Random assignment helps ensure experimentalvalidity by providing a measure in equalizinggroups.
Random assignment protects against selectionbias; no group is predisposed to a treatment orintervention.
Random assignment also protects the influenceof confounding variables, variables andattributes that are not being measured but mayinfluence the results of a study (Gall et al.,2006).
Balkin, R. S., 2008
Why random assignment is important
By using random assignment, aconfounding variable is theoreticallydispersed equally across all groups.
“Random assignment is meant to makecontrol and experimental groupsequivalent” (Vogt, 2007, p. 96).
6
Balkin, R. S., 2008
Three Types of Design
There are three major types of experimentaldesign:
(a) pre-experimental in which there is no randomassignment and no comparison group
(b) quasi-experimental in which a comparisongroup is used but no random assignment
(c) true experimental in which both randomassignment and a comparison group is utilized.
Balkin, R. S., 2008
Pre-experimental designs
Pre-experimental designs are easy toimplement but the findings may not bereflective of what is being measuredbecause of the absence of a comparisongroup.
The meaningfulness of a study utilizing apre-experimental design should always bequestioned as the conclusions cannot begeneralized.
Balkin, R. S., 2008
Pre-experimental designs
In a pre-experimental design, participants wouldbe placed in both study skills and groupcounseling. The school counselor wouldevaluate whether or not academic performanceimproved as evidenced by changes in gradepoint average, standardized test scores, etc.
If academic progress was demonstrated by theparticipants, could the school counselorconclude the intervention was successful?
7
Balkin, R. S., 2008
Pre-experimental designs
No.
Without any comparison group, theresearcher does not know if groupcounseling, study skills, or simply thepassage of time results in change of theDV (such as gpa).
Balkin, R. S., 2008
Quasi-experimental designs
Quasi-experimental designs are verycommon in social science researchbecause of the use of non-manipulatedfactors, which prevent random assignmentof the independent variable.
Special attention to ensuring groupequivalence is pertinent.
Balkin, R. S., 2008
Quasi-experimental designs
In a quasi-experimental design, participants would beassigned to study skills and group counseling and studyskills only, but the assignment would not be random.
For example, the school counselor could use intactgroups based on their class schedule. Before the schoolcounselor evaluates whether or not academicperformance improved as evidenced by changes ingrade point average, standardized test scores, etc.,statistical tests should be run to make sure that neithergroups has significantly higher or lower test scores orgrade point average, as this could bias the results.
8
Balkin, R. S., 2008
Quasi-experimental designs:Group Equivalence
Group equivalency refers to each group beingsimilar prior to a manipulation of an IV
For example, if one group of students with anaverage gpa of 1.65 was assigned to a studyskills group and another group of students withan average gpa of 2.50 was assigned to studyskills and group counseling would we haveequivalent groups?
Balkin, R. S., 2008
Quasi-experimental designs:Group Equivalence No.
And if the study skills and group counselingparticipants had greater increase in gpa, wewould not know if it was due to the interventionor the fact that the participants were alreadyperforming better
When group equivalency is demonstrated,results can be generalized as the effect of theindependent variable can be evaluated.
Balkin, R. S., 2008
True experimental designs
True experimental designs are morecomplicated to implement but the resultsare generalized more easilyRandom assignment promotes group
equivalencyThe effect of the independent variable can be
ascertained more easily due to the presence ofa comparison group.
9
Balkin, R. S., 2008
True experimental designs
In a true experimental design, participants wouldbe randomly assigned to study skills and groupcounseling and study skills only. Thus, allparticipants would have an equal chance atbeing selected for study skills and groupcounseling or study skills only.
The school counselor would evaluate whether ornot academic performance improved asevidenced by changes in grade point average,standardized test scores, etc.
Balkin, R. S., 2008
Threats to experimental validity
Before examining models of experimentaldesign, it is important to understand whythese models exist
—to substantiate experimental validity.
Balkin, R. S., 2008
Threats to experimental validity
There are two types of experimentalvalidity.Internal validity is the extent to which the
independent variable(s) truly effect the changein the dependent variable.
External validity is the extent to which the studycan be generalized to other settings andpopulations (Best & Kahn, 2006).
10
Balkin, R. S., 2008
Threats Internal Experimental Validity
Threats to internal validity are due to extraneousvariables
Extraneous variables are any variables that arenot controlled for that can affect the outcome.
In social sciences, it is impossible to control forevery conceivable extraneous variable.
However, true experimental designs canminimize the effects of extraneous variables.
Balkin, R. S., 2008
Threats Internal Experimental Validity
To understand the effect of an extraneous variable,consider the group counseling example at the beginningof the lecture. Even if the groups were randomlyassigned and participants in study skills and groupcounseling had greater change in academic performancethan participants receiving study skills only, could theschool counselor be certain that the change in academicperformance was due to the added group counselingintervention?
Campbell and Stanley (1966) identified nine factors thatcould threaten internal validity. Most of the threats listedin this section can be controlled for through randomassignment.
Balkin, R. S., 2008
Threats Internal Experimental Validity
Maturation History Testing Instrumentation Statistical regression
Selection bias Interaction of
selection andmaturation
Mortality Experimenter bias
Review pages 172-175 in your textbook. You will be responsible for understanding these concepts. Please email me if you have questions.
11
Balkin, R. S., 2008
Maturation
Maturation refers to the change in theparticipants over time. Participants maychange simply due to time passing duringthe study.
Other factors that may involve maturationare changes in emotional, intellectual,and/or physical functioning or fatigue fromparticipation in the study (LaFountain &Bartos, 2002).
Balkin, R. S., 2008
Maturation
Remember the alcohol abuse/depressionstudy on slide 10?
In the Kelly et al. (2000) study, participantsin the control group exhibited a statisticallysignificant decrease in depression despitenot having any intervention.
This would be known as a maturationeffect.
Balkin, R. S., 2008
History
Unplanned events that occur during thestudy can have an effect on the outcome.
Participants may experience an eventoutside of the experimental setting thatinfluences the outcome of the experiment(LaFountain & Bartos, 2002).
12
Balkin, R. S., 2008
History
For example, a researcher is doing a study onhigh school athletes and moral development.During the study, the football team throws aparty to celebrate a great victory. The schoolprincipal receives a call the next day that severalfootball players were intoxicated and a femalestudent was sexually assaulted. Such an eventwas beyond the control of the researcher.However, if the event had the same effect onboth treatment and control groups, the effect ofthe event may be equalized. Otherwise, resultsare confounded due to the unplanned event(Best & Kahn, 2006).
Balkin, R. S., 2008
Testing
This type of threat is common in designsthat utilize a pretest.
The actual pretest may affect futureperformance on a posttest.
Participants may be more knowledgeableabout how to answer so that progressappears more evident.
Balkin, R. S., 2008
Testing
For example, using practice tests to prepare for a standardizedexam may improve scores because the participant is more familiarwith the question format, as opposed to being more knowledgeableabout the material.
However, if both the experimental and control groups are affectedsimilarly, then the effect of the pretest is controlled.
When the same test is administered repeatedly, or a series ofdifferent tests, instruments, or observations are utilized, then thetesting effect is sometimes referred to as a sequencing effect.
The sequence of the administrations may affect the validity of theexperiment.
The sequence of the treatments might be responsible for the changein the dependent variable rather than the treatments themselves.
13
Balkin, R. S., 2008
Instrumentation
Instruments need to measure a constructaccurately and consistently.
Instruments that lack evidence of reliabilityand validity are more likely to lead toerroneous results.
Balkin, R. S., 2008
Instrumentation
In the group counseling example, consider theimplications if the measure of academic performancewas change in grade point average. Poor academicperformance for students in remedial courses may bequite different from poor academic performance fromstudents in more accelerated coursework.
Thus, grade point average may not be a good measureof change in academic performance because each groupmember has a different set of courses.
Problems with instrumentation may also develop whenrating systems are used. Raters may have differentstandards or be influenced by other variables unrelatedto the study.
Balkin, R. S., 2008
Instrumentation
For example, Riniolo, Johnson, Sherman, &Misso (2006) examined the influence of physicalattractiveness on student evaluations forprofessors and found that instructors perceivedas physically attractive by students had strongerevaluations than professor who were notperceived as physically attractive. Variations instudent perceptions of physical attractivenessaffected professor ratings, despite the fact thatraters are not asked to consider physicalattractiveness in the evaluation of an instructor.
14
Balkin, R. S., 2008
Statistical regression
Statistical regression refers to baseline (very lowscores) and ceiling effects (very high scores).
This problem is often seen in studies where arepeated measure is used.
For example, in the group counseling example,students with very low academic performancewill likely score higher simply because repeatedmeasures over time tend to move toward themean.
Students who score very low scores are morelikely to score higher on follow-up evaluations.
Balkin, R. S., 2008
Statistical regression
Consumers of research should utilize results cautiouslywhen the participants are selected based on very low orhigh scores (Best & Kahn, 2006).
The key to preventing, while never totally eliminating,statistical regression is to have variability within thesample (LaFountain & Bartos, 2002).
Remember that baseline and ceiling effects occur whenscores are at an extreme.
If participants tend to score at one extreme or the other,groups should be divided and separate analyses mayneed to be conducted with participants at either extreme.
Balkin, R. S., 2008
Selection bias
Selection bias is a common problem inpre-experimental and quasi-experimentaldesigns in which random assignment doesnot occur or intact groups are utilized.
In such cases the experimental andcontrol groups may not start at the samelevel or with similar scores orcharacteristics.
(See slides on Group Equivalence: 22-23)
15
Balkin, R. S., 2008
Selection bias
Consider what might happen in the group counseling example if theparticipants receiving study skills only had an average grade point of1.8 and participants receiving study skills and group counseling hadan average grade point of 1.2. The group with the higher grade pointaverage may fair better because they already have more skills.Even in cases where random assignment is used there is noguarantee that the groups are equal.
Researchers can assess the influence of selection bias in a study byutilizing pretests to ensure group equivalence at the beginning of astudy.
Researchers can also employ matching groups, which entails usinga pretest and matching participants with equivalent scores.Participants are randomly assigned to separate groups in order toassure equivalent groups (LaFountain & Bartos, 2002).
Balkin, R. S., 2008
Interaction of selection and maturation
This type of threat may occur due todifferent attributes between the variousgroups.
For example, if students in the study skillsgroup were mostly in third grade andstudents in the study skills and groupcounseling group were mostly in fifthgrade, the results may be erroneous if theage of the students is not controlled.
Balkin, R. S., 2008
Interaction of selection and maturation
Interaction with selection is a common problemin social science research.
Demographic variables such as sex, ethnicity,and socioeconomic status are often studied, butsuch studies may lack random assignmentbecause you cannot randomly assignparticipants to a variables such as sex—it isalready predetermined.
In other words, simply because groupdifferences exist does not mean that thedifferences are due to belonging to a specificgroup.
16
Balkin, R. S., 2008
Mortality
Participants in research studies may notcomplete a study.
As a matter of fact, it is unethical to forceor influence participants to complete thestudy if they wish to withdraw.
If dropouts are occurring in one groupmore than another group, the researchermay need to consider whether the groupsare still equivalent.
Balkin, R. S., 2008
Mortality
If a large number of students in the groupcounseling and study skills group dropped outbut the study skills only group remained intact,the equivalency of the groups may have beencompromised.
Attrition in long-term studies is expected. Whenattrition occurs in longitudinal research,comparisons with the initial sample may becompromised, especially if sample size is small.The remaining participants may not berepresentative of the original sample(LaFountain & Bartos, 2002).
Balkin, R. S., 2008
Experimenter bias
Experimenter bias occurs when theexperimenter predisposes participants to aparticular treatment.
Experimenter bias is likely to occur whenrandom assignment is not utilized.
17
Balkin, R. S., 2008
Experimenter bias
If a school counselor, for instance,believed in the value of group counselingand did not use random assignment, theschool counselor may select participantsthat he/she believes are good candidatesfor group counseling, thereby affecting theoutcome of the study.
Balkin, R. S., 2008
Experimenter bias
Experimenter bias is difficult to eliminate insocial science research, especially whenparticipants are selected to receive anintervention based on need.
It is difficult to ascertain the effectiveness of anintervention, such as individual counseling, whenparticipants who receive counseling likelyvolunteered for it.
Naturally, the intervention is likely to have animpact because the participant wants theintervention to work.
Balkin, R. S., 2008
Threats to external validity
Threats to external validity are related tothe artificiality of the experimentalcondition.
Campbell and Stanley (1966) identifiedfive factors that could threaten externalvalidity.
18
Balkin, R. S., 2008
Threats to external validity
Interference of priortreatment
Artificial experimentalsetting
Interaction effect oftesting and treatment
Interaction ofselection andtreatment
Interaction oftreatmentimplementation
Review pages 175-176 in your textbook. You will be responsible for understanding these concepts. Please email me if you have questions.
Balkin, R. S., 2008
Threats to external validity—Consider thisscenario: Clients who are hospitalized in a residential
setting may make considerable therapeuticprogress to the extent that the client appearsready to discharge. However, upon leaving theresidential program, client may regress back totheir previous high risk behavior. Despite makingprogress in the residential setting, thetherapeutic progress may not have translatedwell to the realities of the real world and weremore a result of being in a structured setting.Thus, the therapeutic progress did notgeneralize to other settings.
Balkin, R. S., 2008
Interference of prior treatment
Participants who have a prior history withthe treatment condition could affect theoutcome of the study, particularly if such ahistory is not equally dispersed throughoutthe treatment and control groups.
19
Balkin, R. S., 2008
Interference of prior treatment
Using the previous above, clients whohave a history of receiving counselingservices and are placed in an institutionalsetting may have an understanding ofwhat they need to say in order to appearhealthier.
The client therefore is discharged, but theprogress was very superficial.
Balkin, R. S., 2008
Artificial experimental setting
Change may occur as a result of thesetting of the research.
Balkin, R. S., 2008
Artificial experimental setting
For example, participants who receivestudy skills and group counseling mayexpress desire to make changes in studyhabits because such disclosure ispositively reinforced by the group.However, upon leaving group and thecounseling setting, the participant may findit difficult to change study habits due tobeing outside of the experimental settingwhere support may be much less.
20
Balkin, R. S., 2008
Interaction effect of testing and treatment
This is similar to the testing effect inthreats to internal validity, in which thepractice of taking a pre-test affects lateradministrations.
In an interaction of testing and treatment,the pretest may affect the treatment andcontrol groups differently.
Balkin, R. S., 2008
Interaction effect of testing and treatment
For example, a researcher wants to study the effects ofyoga on memory. The experimental group receives amemory test, two weeks of yoga, and then anothermemory test. The control group receives a memory test,no treatment, and then another memory test after threeweeks. Even if the treatment group scored higher on thesecond memory test, it is important to note that thehigher memory scores could be due to the participants inthe treatment group attending more to the method of thepretest, rather than the higher scores resulting from yogapractice.
Random assignment is a preventative measure for thisthreat.
Balkin, R. S., 2008
Interaction of selection and treatment
Research in the social sciences is complex. Randomsampling and random assignment is difficult to obtain.Social scientists depend upon cooperation from externalgroups, such as schools, agencies, hospitals, etc. Thus,participants are often obtained by utilizing intact groups.
Sampling is often convenient in social science research For that reason, selection and treatment can
compromise experimental validity. The researcher needsto be able to demonstrate that the participants in thestudy are truly representative of the target population.
21
Balkin, R. S., 2008
Interaction of selection and treatment
Reporting demographic characteristicsinforms readers about the generalizabilityof the study to other populations. Whenstudies are conducted on populations withspecific characteristics, then the study isgeneralizable to individuals with thosecharacteristics.
Balkin, R. S., 2008
Interaction of treatment implementation
When developing interventions acrossgroups, it may be difficult for a singleresearcher to deliver the same interventionin the same manner.
Balkin, R. S., 2008
Interaction of treatment implementation
For example, a school counselor may wish touse a colleague to provide study skillspsychoeducation to one group, while the schoolcounselor provides study skills psychoeducationand group counseling to another group.
Even if the study skills are taught using thesame materials, the presentation of theinformation may be qualitatively different.
Researchers need to have procedures in placeto verify that a treatment or intervention wasconducted properly (Best & Kahn, 2006).
