11
RESEARCH DESIGN
22
– WhatWhat--What was studied?--What was studied?– What aboutWhat about--What aspects of --What aspects of the subject were studied? the subject were studied? – What forWhat for--What is/was the--What is/was the significance of the study?significance of the study?
– What did prior lit./research say?What did prior lit./research say?
– What was doneWhat was done--How was the--How was the study conducted?study conducted?
– What was found?What was found?– So what?So what?– What now?What now?
1. Introduction, Research Problems/ Objectives, & Justification
2. Literature Review
3. Methodology (Research sample, data collection, measurement, data analysis)
4. Results & Discussion 5. Implications6. Conclusions and
Recommendations for Future Research
PROCESS OF DESIGNING AND CONDUCTING A PROCESS OF DESIGNING AND CONDUCTING A RESEARCH PROJECT:RESEARCH PROJECT:
33
RESEARCH DESIGNRESEARCH DESIGN
RESEARCH DESIGNRESEARCH DESIGN refers to the plan, structure, and strategy of research-- refers to the plan, structure, and strategy of research--the the blueprintblueprint that will that will guide the research processguide the research process. .
Developing ResearchHypotheses
Intriguing Observation,Intellectual Curiosity
Defining ResearchProblem & Objectives
Testing Hypo.:Data Analysis &Interpretation
Sampling Design
Refinement of theory(Inductive Reasoning)
Data Coding,And
Editing
Developing OperationalDefinitions for
Research Variables
Building the Theoretical Framework and the
Research Model
Data Collection
More Careful Studyingof the Phenomenon
THE PROCESS OF
EMPIRICAL RESEARCH
44
RESEARCH DESIGNRESEARCH DESIGN
CONCLUSION VALIDITYCONCLUSION VALIDITY refers to the extent of researcher’s refers to the extent of researcher’s ability to ability to draw accurate conclusionsdraw accurate conclusions from the research. That from the research. That is, the degree of a study’s: is, the degree of a study’s:
a)a) Internal ValidityInternal Validity——correctness of conclusionscorrectness of conclusions regarding the regarding the relationships among variablerelationships among variables examined s examined
Whether the research findings accurately reflect how the research variables are Whether the research findings accurately reflect how the research variables are really connected to each other.really connected to each other.
b)b) External ValidityExternal Validity – –Generalizability of the Generalizability of the findings to the findings to the intended/appropriate population/settinintended/appropriate population/settingg
Whether Whether appropriate subjectsappropriate subjects were selected for conducting the study were selected for conducting the study
RESEARCH DESIGNRESEARCH DESIGN:: The The blueprint/roadmapblueprint/roadmap that will guide the that will guide the research. research. The test for the The test for the quality of a study’s research design quality of a study’s research design is theis the study’s study’s conclusion validityconclusion validity..
55
RESEARCH DESIGNRESEARCH DESIGN
• Variance of the INDEPENDENT & DEPENDENT Variance of the INDEPENDENT & DEPENDENT variables (variables (Systematic VarianceSystematic Variance))
• Variability of potential NUISANCE/EXTRANEOUS/ Variability of potential NUISANCE/EXTRANEOUS/ CONFOUNDING variables (CONFOUNDING variables (Confounding VarianceConfounding Variance))
• Variance attributable to ERROR IN MEASUREMENT Variance attributable to ERROR IN MEASUREMENT ((Error VarianceError Variance).).
How?How?
How do you achieveHow do you achieve internal and external validity internal and external validity (i.e., (i.e., conclusion validity)?conclusion validity)?
By effectively By effectively controlling 3 types of controlling 3 types of variancesvariances::
66
Effective Research Design Effective Research Design
– MAXMAXimize Systematic Varianceimize Systematic Variance– MINMINimize Error Varianceimize Error Variance– CONCONtrol Variance of Nuisance/Extraneous/ trol Variance of Nuisance/Extraneous/
Exogenous/Confounding variablesExogenous/Confounding variables
Guiding principleGuiding principle for effective control of for effective control of variances (and, thus, effective research variances (and, thus, effective research design) is: design) is:
The The MAXMINCONMAXMINCON Principle Principle
77
Effective Research Design Effective Research Design
IN EXPERIMENTS?IN EXPERIMENTS? (where the researcher actually (where the researcher actually manipulates the independentmanipulates the independent
variable and measures its impact on the dependent variable):variable and measures its impact on the dependent variable):– Proper manipulation of experimental conditionsProper manipulation of experimental conditions
to ensure high variability in indep. var.to ensure high variability in indep. var.
IN NON-EXPERIMENTAL STUDIES?IN NON-EXPERIMENTAL STUDIES? (where independent and dependent variables are measured (where independent and dependent variables are measured
simultaneously and the relationship between them are simultaneously and the relationship between them are examined):examined):– Appropriate subject selectionAppropriate subject selection (selecting subjects (selecting subjects
that are sufficiently different with respect to the that are sufficiently different with respect to the study’s main var.)--avoid study’s main var.)--avoid Range RestrictionRange Restriction
MAXMAXimizing Systematic Variance: imizing Systematic Variance:
Widening the range of valuesWidening the range of values of research variables. of research variables.
88
Effective Research DesignEffective Research Design
Sources of error variance:Sources of error variance:– Poorly designed measurement instrumentsPoorly designed measurement instruments
((instrumentation errorinstrumentation error))
– Error emanating from study subjects (e.g., Error emanating from study subjects (e.g., response errorresponse error))
– Contextual factorsContextual factors that reduce a sound/accurate that reduce a sound/accurate measurement instrument’s capacity to measure measurement instrument’s capacity to measure accurately.accurately.
How to Minimize Error Variance?How to Minimize Error Variance?– Increase Increase validityvalidity and and reliabilityreliability of of
measurement instruments.measurement instruments.– Measure variables under as Measure variables under as ideal ideal
conditionsconditions as possible. as possible.
MINMINimizing Error Variance imizing Error Variance (measurement error):(measurement error): MinimizingMinimizing the part of variability in scores that is the part of variability in scores that is caused by caused by error in measurementerror in measurement..
1. Historical data on pollution and longevity
2. Relationship between likelihood of hearing problems and high blood pressure
3. Recent stat. show in-vitro kids are 5 times more likely to develop eye tumors (Culprit: in-vitro fathers’ older age)
4. Significantly more armed store robberies during the cold winter days. 99
Effective Research DesignEffective Research Design
May or may not be of primary interest to the researcher,May or may not be of primary interest to the researcher, But, can produce But, can produce undesirable variationundesirable variation in the study's in the study's
dependent variable, and cause dependent variable, and cause misleadingmisleading or or weirdweird results results Thus, if not controlled, Thus, if not controlled, can contaminate/distort the true can contaminate/distort the true
relationship(s)relationship(s) between the independent and dependent between the independent and dependent variable(s) of interestvariable(s) of interest
• i.e., confounding var. can result in a i.e., confounding var. can result in a spuriousspurious-- as opposed to -- as opposed to substantivesubstantive--correlation --correlation between IV and DVbetween IV and DV. . Example?Example?
Hearing Blood Problem Pressure
CONCONtrolling Variance of Confounding/Nuisance Variables:trolling Variance of Confounding/Nuisance Variables:
FIRST, what are FIRST, what are Nuisance/ConfoundingNuisance/Confounding Variables? Variables?
Age
1010
Effective Research DesignEffective Research Design
– Conducting the experiment in a Conducting the experiment in a controlled environment controlled environment (e.g., (e.g., laboratory), where we can laboratory), where we can hold valueshold values of potential confounding of potential confounding variables variables constantconstant..
– Subject selectionSubject selection (e.g., matching subjects in experiments) (e.g., matching subjects in experiments)– Random assignmentRandom assignment of subjects (variations of confounding variables of subjects (variations of confounding variables
are evenly distributed between the experimental and control groups)are evenly distributed between the experimental and control groups)
In Survey Research:In Survey Research:– Sample selectionSample selection (e.g., including only subjects with appropriate (e.g., including only subjects with appropriate
characteristics—using characteristics—using malemale college graduatescollege graduates as subjects will control as subjects will control for potential confounding effects of gender and education)for potential confounding effects of gender and education)
– Statistical ControlStatistical Control---anticipating, measuring, and -anticipating, measuring, and statistically statistically controlling for confounding variables’ effectscontrolling for confounding variables’ effects (i.e., hold them (i.e., hold them statistically constantstatistically constant, or statistically removing their effects)., or statistically removing their effects).
HOW TO CONTROL FOR CONFOUNDING/HOW TO CONTROL FOR CONFOUNDING/ NUISANCE VARIABLES? NUISANCE VARIABLES? In Experimental Settings In Experimental Settings (e.g., Fertilizer Amount Rate of Plant Growth)(e.g., Fertilizer Amount Rate of Plant Growth) ::
Some Potential Confounding Variables?Some Potential Confounding Variables?
1111
Effective Research DesignEffective Research Design
Adequate (full range of) variability in values of Adequate (full range of) variability in values of research variables,research variables,
Precise and accurate measurement,Precise and accurate measurement, Identifying and controlling the effects of Identifying and controlling the effects of
confounding variables, andconfounding variables, and Appropriate subject selectionAppropriate subject selection
RECAP:RECAP:Effective research design is a function of ?Effective research design is a function of ?
1212
BASIC DESIGNSBASIC DESIGNS
Experimental Designs:Experimental Designs:– True Experimental StudiesTrue Experimental Studies– Pre-experimental StudiesPre-experimental Studies– Quasi-Experimental StudiesQuasi-Experimental Studies
Non-Experimental Designs:Non-Experimental Designs:– Expost Facto/Correlational StudiesExpost Facto/Correlational Studies
SPECIFIC TYPES OF RESEARCH DESIGN
BASIC RESEARCH DESIGNS:
1313
EXPERIMENTAL DESIGNSEXPERIMENTAL DESIGNS
RESULT:RESULT: Significant Improvement from O1 to O2 Significant Improvement from O1 to O2 (i.e., sig. O2 - O1 difference)(i.e., sig. O2 - O1 difference)
QUESTION:QUESTION: Did X (the drug) cause the Did X (the drug) cause the improvement?improvement?
One of the simplest experimental designs is the ONE GROUP PRETEST-POSTTEST DESIGN--EXAMPLE?
One way to examine Efficacy of a Drug:
O1 X O2 Measure DRUG Measure
Patients’ Condition Experimental Patients’ Condition
(Pretest) Condition/ (Posttest) intervention
1414
EXPERIMENTAL DESIGNSEXPERIMENTAL DESIGNS
– Have Have only shownonly shown “X” is a SUFFICIENT“X” is a SUFFICIENT condition condition for the change for the change “Y”“Y” (i.e., presence of X is (i.e., presence of X is associated with a change in Y).associated with a change in Y).
But, But, is “X” also a NECESSARYis “X” also a NECESSARY condition for condition for Y?Y?
– How do you verify the latter?How do you verify the latter? By showing that the change would not have By showing that the change would not have
happened in the absence of X—using ahappened in the absence of X—using a CONTROL GROUP.CONTROL GROUP.
David Hume would have been tempted to say “YES.” He was a positivist and wanted to infer causality basedon high correlations between events.
But such an inference could be seriously flawed.
Why?
David Hume, 18th Century Scottish
Philosopher
1515
EXPERIMENTAL DESIGNSEXPERIMENTAL DESIGNS
CONTROL GROUPCONTROL GROUP simulates absence of X simulates absence of X– Origin of using Control Groups (A tale from Origin of using Control Groups (A tale from ancient Egypt)ancient Egypt)
Pretest Post-Test Pretest Post-Test Control GroupControl Group Design-- Design--Suppose random Suppose random assignment (assignment (RR) was used to control confounding variables:) was used to control confounding variables:
R R Exp. Group O1E X O2EExp. Group O1E X O2ERR Ctrl Group O1C O2C Ctrl Group O1C O2C
RESULT:RESULT: O2E > O1E & O2C Not> O1C O2E > O1E & O2C Not> O1CQUESTION:QUESTION: Did X cause the improvement in Exp. Did X cause the improvement in Exp.
Group?Group?
1616
EXPERIMENTAL DESIGNSEXPERIMENTAL DESIGNS
– Need Need proper form of controlproper form of control—e.g., —e.g., PlaceboPlacebo..
RR Exp. Group O1E X O2E Exp. Group O1E X O2E
RR Ctrl Group O1C Placebo O2C Ctrl Group O1C Placebo O2C QUESTION:QUESTION: Can we now conclude X caused the improvement Can we now conclude X caused the improvement
in Exp. Group?in Exp. Group?
NOT NECESSARILY! Why not?
• Power of suggestibility (The Hawthorne Effect)CONCLUSION?
• Maybe, but be aware of the Experimenter Effect (it tends to prejudice the results especially in medical research).• SOLUTION: Double Blind Experiments (neither the subjects nor the experimenter knows who is getting the placebo/drug).
1717
EXPERIMENTAL DESIGNSEXPERIMENTAL DESIGNS
Experimental studies Experimental studies need to need to controlcontrol for for potentialpotentialconfounding factorsconfounding factors that may threaten internal validity that may threaten internal validityof the experiment:of the experiment:
– Hawthorne Effect Hawthorne Effect is only one potential confounding factor is only one potential confounding factor in experimental studies. in experimental studies.
Other such factors areOther such factors are::– History?History?
Biasing events that occur between pretest and post-testBiasing events that occur between pretest and post-test
– Maturation?Maturation? Physical/biological/psychological changes in the subjects Physical/biological/psychological changes in the subjects
– Testing?Testing? Exposure to pretest influences scores on post-test Exposure to pretest influences scores on post-test
– Instrumentation?Instrumentation? Flaws in measurement instrument/procedureFlaws in measurement instrument/procedure
1818
EXPERIMENTAL DESIGNSEXPERIMENTAL DESIGNS
Experimental studiesExperimental studies need to need to controlcontrol for for potentialpotentialconfounding factorsconfounding factors that may threaten internal validity that may threaten internal validityof the experiment (Continued):of the experiment (Continued):
– Selection?Selection? Subjects in experimental & control groups different from the startSubjects in experimental & control groups different from the start
– Statistical Regression (regression toward the mean)?Statistical Regression (regression toward the mean)? Subjects selected based on extreme pretest valuesSubjects selected based on extreme pretest values Discovered by Discovered by Francis GaltonFrancis Galton in 1877 in 1877
– Experimental Mortality?Experimental Mortality? Differential drop-out of subjects from experimental and control groups Differential drop-out of subjects from experimental and control groups
during the studyduring the study– EtcEtc..
Experimental designs mostly used in Experimental designs mostly used in natural and physical natural and physical sciences sciences..
Generally, higher internal validity, lower externalGenerally, higher internal validity, lower external validity validity
1919
2020
2121
CORRELATIONAL DESIGNSCORRELATIONAL DESIGNS
The design of choice in The design of choice in social sciencessocial sciences since the since the phenomenonphenomenon under study is usually under study is usually not reproducible in a laboratory settingnot reproducible in a laboratory setting
Researcher has Researcher has little or no controllittle or no control over study’s indep., dep. over study’s indep., dep. and the and the numerous potential numerous potential confounding variablesconfounding variables,,
Often the researcher Often the researcher concomitantly measuresconcomitantly measures all the study all the study variables (e.g., independent, dependant, etc.),variables (e.g., independent, dependant, etc.),
Then examines the following types of relationships:Then examines the following types of relationships:– correlations among variablescorrelations among variables or or– differences among groupsdifferences among groups,,
Inability to controlInability to control for effects of for effects of confounding variables confounding variables makes makes causal inferencescausal inferences regarding relationships among variables regarding relationships among variables more difficultmore difficult and, thus: and, thus:
Generally, higher external validity, lower internal validityGenerally, higher external validity, lower internal validity
NON-EXPERIMENTAL/CORRELATIONAL DESIGNSNON-EXPERIMENTAL/CORRELATIONAL DESIGNS
2222
CORRELATIONAL DESIGNSCORRELATIONAL DESIGNS
NOT NECESSARILY! EXAMPLES:NOT NECESSARILY! EXAMPLES:– Water Fluoridation and AIDSWater Fluoridation and AIDS
((San Francisco ChronicleSan Francisco Chronicle, Sep. 6, 1984), Sep. 6, 1984)– Armed store robberies and cold weatherArmed store robberies and cold weather– Longevity and PollutionLongevity and Pollution– In-vitro birth and likelihood of developing eye tumorsIn-vitro birth and likelihood of developing eye tumors– Hearing problem and blood pressure Hearing problem and blood pressure
What can a significant correlation mean then?What can a significant correlation mean then?
Non-experimental designs rely on correlational evidence.
QUESTION: Does a significant correlation between two variables in a non-experimental study necessarily represent a causal relationship between those variables?
2323
CORRELATIONAL STUDIESCORRELATIONAL STUDIES
a.a. Both variables are effects of a common cause (or Both variables are effects of a common cause (or bothbothcorrelated with a third variable), i.e., spurious correlationcorrelated with a third variable), i.e., spurious correlation(e.g., air pollution and life expectancy, hearing problem & (e.g., air pollution and life expectancy, hearing problem & blood pressure, country’s annual ice cream sales and blood pressure, country’s annual ice cream sales and frequency of hospital admissions for heat stroke)frequency of hospital admissions for heat stroke)
b.b. Both var. Both var. alternative indicatorsalternative indicators of same concept of same concept(e.g., Church attend. & Freq. of Praying--religiosity).(e.g., Church attend. & Freq. of Praying--religiosity).
c.c. Both parts of Both parts of a common "system" or "complex;" tend to a common "system" or "complex;" tend to come as a packagecome as a package(e.g., martini drinking and opera attendance--life style)(e.g., martini drinking and opera attendance--life style)
d.d. Fortuitous--Fortuitous--Coincidental correlationCoincidental correlation, no logical relationship, no logical relationship(e.g., Outcome of super bowl games and movement of stock (e.g., Outcome of super bowl games and movement of stock market)market)
AT LEAST FOUR OTHER POSSIBLE INTERPRETATIONS/REASONSFOR CORRELATIONS BETWEEN TWO VARIABLES:
2424
CORRELATIONAL STUDIESCORRELATIONAL STUDIES
Covariation RuleCovariation Rule (X and Y must be (X and Y must be correlated)--Necessary but not sufficient condition.correlated)--Necessary but not sufficient condition.
Temporal Precedence RuleTemporal Precedence Rule (If X is the cause, Y (If X is the cause, Y should not occur until after X).should not occur until after X).
Internal Validity RuleInternal Validity Rule (Alternative plausible (Alternative plausible explanations of Y and X-Y relationships should be explanations of Y and X-Y relationships should be ruled out (i.e., eliminate other possible causes).ruled out (i.e., eliminate other possible causes).– In practice, this means exercising caution by In practice, this means exercising caution by
identifying potential identifying potential confounding variables and confounding variables and controlling for their effectscontrolling for their effects).).
WHEN IS IT SAFER TO INFER CAUSALLINKAGES FROM STRONG CORRELATIONS?
John Stuart Mill’s Rules for Inferring Causal Links:John Stuart Mill
1806-1873
2525
Questions or Comments
?