Reading and

Evaluating Research

Method

MethodEssential question to ask about the Method:• “Is the operationalization of the hypothesis valid?

Sections:

• Section 1 – Design• Section 2 – Participants• Section 3 – Materials• Section 4 – Procedures

Overarching Concepts:• Construct Validity• External Validity• Internal Validity• Confounds

Concept:Concept: Construct Validity

• The heart of research design is translating your theoretical variable (constructs) into operational variables (materials and procedures). Construct validity is the degree to which you accurately operationalize your construct. There are two ways in which to evaluate Construct Validity…– Absolute– Relative

Concept:Concept: Construct Validity

• Questions to ask about CV:– Quality/quantitative of CV– Content validity– Face validity– Poor or too weak?– Engaged so has an effect?– Understandable?– Interpreted as intended?– Interpreted same way by each person?

Concept:Concept: External Validity

• The heart of research design is also generalizing the findings beyond the specific study. External validity is the degree to which your measures/manipulations generalize beyond your specific study to other people and situations. There are two ways in which to evaluate External Validity…– Absolute– Relative

Concept:Concept: External Validity

• Questions to ask about CV:– Generalize to different settings, times, real-

world contexts?– Participants chosen for particular reason?– Participants unique or distinct in some

way?– What were the characteristics of the

sample?– How were the participants sampled?– Were people excluded, why?– What was sample size?

Concept:Concept: Internal Validity

• Internal validity is about whether the study proved a cause-and-effect relationship by showing that your intended constructs caused the outcome. Internal validity only applies to experimental research, not other types of methodology like surveys, observational research, etc. – Subset of Construct Validity

(not accurately operationalizing your construct prevents you from proving cause-and-effect)

– Subset of External Validity (not using randomly representative participants prevents you from proving cause-and-effect)

Concept:Concept: Internal Validity

• Questions to ask about Internal Validity:– Did manipulation of IV occur before DV– Were the conditions different before the

study began?– Were participants randomly assigned?– Did the study have an appropriate

comparison point and/or control condition?– Should there be multiple comparison

groups to rule out confounds?– Does the study require a manipulation

check? If so, does it accurately measure the intended construct?

Concept:Concept: Confounds

• “Confounds” is the generic name for any factor that could interfere with the intended purpose of the study. Confounds can be due to the situational factors, the experimenter(s), the participant(s), the measures, and/or the manipulations– If confounds exist, must state concretely

how the confound would bias the results. Simply having a confound does not necessarily reduce the validity of the research.

– If confounding variables were controlled for in the study, what the control appropriate?

Concept:Concept: Confounds

• Questions to ask about Confounds:– Confounds due to experimenters (s)

• Are there demand characteristics?• Were the researchers trained?• Was content coding appropriate?• Were the researchers blind to hypo?

– Confounds due to participant(s)• Was there reactance/compliance?• Did participants interact with each other?• Are there social desirability concerns?• Was there response fatigue?• What was drop out / toss out rate, and why?

Concept:Concept: Confounds

• Questions to ask about Confounds:– Confounds due to M&M(s)

• Are there order effects?• Problems with instrumentation tools?• What is match between construct and M&M?

– Confounds due to situational factor(s)• Was the study in a controlled environment?• If uncontrolled, did the researchers keep a

record of distraction to control for those factors statistically?

• Are there historical confounds?• Did testing occur at different times, sites?