Upload
lucas-harvey
View
216
Download
1
Embed Size (px)
Citation preview
Example critique.Complete abstract critiques...
A comment on grades...
Where does your grade for this class come from?
From the syllabus...
Pretty simple
A comment on grades...
Where does your grade for this class come from?
From the syllabus...
Pretty simple
A comment on grades...
Where does your grade for this class come from?
From the syllabus...
Pretty simple
A comment on grades...
Where does your grade for this class come from?
From the syllabus...
Ok, harder
A comment on grades...
Where does your grade for this class come from?
From the syllabus...
Ok, harder
But let’s think about this...say you got A’s in the first three. That’s an average of 95% (say) in 50% of the
grade. What do you have to get in the remainder to still get an “A”? (For those of you who are arithmetically
challenged, the answer is anything greater than 84.9%)
A comment on grades...
Where does your grade for this class come from?
From the syllabus...
Ok, harder
But let’s think about this...say you got A’s in the first three. That’s an average of 95% (say) in 50% of the
grade. What do you have to get in the remainder to still get an “A”? (For those of you who are arithmetically
challenged, the answer is anything greater than 84.9%)
So the question
should be, how hard is
that?
A comment on grades...
Getting a middle B on the exam and the group projects: First, the exam is on blackboard.
Therefore, it is open book Second, the group projects are group
projects – you get to help each other Third, and most importantly, in terms of
getting a high grade, what do you actually have to do (how will they be graded?)▪ Listen up...
The full abstract...
A professor was curious about the impact of student diet, especially sugar and caffeine, on test performance. She invited students from her classes to participate in a study before their mid-term exams. Forty students accepted the invitation to participate. The professor randomly assigned them to her experimental groups. The students in one group were told not to change their food and drink consumption during the week before their exams but to record everything they ate and drank in a diary. The students in the second group were also asked to record their nutritional intake in the same kind of diary, but were also given unlimited free coffee at the campus coffee shop. The students in the third group received both free coffee and free donuts at the coffee shop, and again were asked to record everything they ate and drank in a diary. The professor planned to use the average of their mid-term grades in her classes as the dependent variable.
a) the independent variable Diet (3 levels: normal vs caffeine vs
sugar and caffeine) b) the dependent variable
Test performance c) the study design using design
notationR X1 OR X2 OR X3 O
d) a critique of the external validity
What is the relationship being studied?▪ A professor was curious about the impact of
student diet, especially sugar and caffeine, on test performance▪ This will provide the starting and finishing
point for all critiques (the next would too, usually, but as you’ll see...)
What are the study findings, if any?▪ None – no data collected yet.
e) a critique of the external validity
What is the sampling strategy? Well, she is a professor, and “She invited
students from her classes to participate in a study before their mid-term exams. Forty students accepted the invitation to participate.”
SO:1. It is non-probabilistic2. It is a sample of convenience3. It is a volunteer study (not all will have volunteered)
e) a critique of the external validity To what population/time/setting is the study
being generalized/targeted (either implied or actual)? (Look for author wording confirming the attempted generalization) A professor was curious about the impact of
student diet, especially sugar and caffeine, on test performance▪ The only qualifiers are that she is interested in
students’ diets, and she is interested in its effects on test performance only.
▪ So there is something about population (students) and something about setting/timing (effects on test performance – so being applied to when they are taking a test)▪ The test performance will be taken care of by construct validity
(it’s one of the constructs in the study, so doesn’t need to be considered here)...so we’ll restrict ourselves to the population issue here.
▪ Also all recorded their food consumption in a diary (part of study settings)
e) a critique of the external validity
What does the sampling strategy and the actual time and settings of the study imply about the merits of any generalization found in (b) above? (Is there a good match, or do you see some mismatches?) Ok, now we get to the crux of it. Does allowing volunteers
from her classes in some way provide a biased sample for the study, such that these students do not adequately represent other students? Or in other settings?▪ Clearly there are potential problems here. It is very likely that this
sample has systematic differences from the general population due to ▪ All participants are from one college, one department, one instructor’s classes▪ All participants are volunteers▪ All recorded their food intake (general population does not, explicitly anyway)
e) a critique of the external validity
To what extent do you think there is a problem of generalization? In other words, do you think the relationship under investigation (see 1.B.), or the results of the study (see 1.C.) would have potential to change given any of the issues raised in 2.C? To answer this question, we have to consider:
▪ What moderator variables might have been unwittingly introduced by the sampling strategy
▪ Whether these moderators could interact with the study variables
So what moderators do we have?▪ SES? Education level?▪ Interest in research? Responsibility?▪ Overall awareness of diet?
Based on one college etc.
Based on volunteerism.
Based on diaries.
e) a critique of the external validity
But ultimately the goal is to provide a rationale as to whether your moderators would in fact interact with the study findings (have the potential to change them) SES? Education level?
▪ Could this change the results of a study looking at diet and performance?▪ (e.g. if these students were more than typically capable, would they be
less susceptible to anxiety effects and the concomitant hypertension that caffeine might bring about?)
Interest in research? Responsibility?▪ Could this change the results of a study looking at diet and performance?▪ (e.g. if people are more interested in research, are they more likely to be
the kind of disciplined student whose preparation for the test is diligent enough to allay crises of confidence? Or are they likely to be precisely the people who have such crises of confidence, and therefore be more likely to participate...and be more sensitive to the effects of caffeine/sugar dosing)
Should be considered but a definitive answer is not needed for an “A”
e) a critique of the construct validity What are the constructs in the study intended to measure?
State the idea(s) behind the dependent measures State the idea(s) behind the independent measures
Test Performance: Likely, a capacity to produce competent academic work in a
stressful situation. Type of work/test? Duration?
Diet (especially sugar and caffeine): Probably getting at energy highs and lows, and memory and
cognition type effects, and the supposed effects of S & C Sugar and caffeine...what are the supposed effects? When do
they exert their influence? Long term? Short term? Dose? Likely you’d have further questions about these 2 from a
“full paper”
e) a critique of the construct validity
How is each construct in the study operationalized? State how dependent measures are
operationalized State how independent measures are
operationalized Dependent measure:
Average of mid-term grades Independent measure:
Free coffee, doughnuts (depending on group assignment) at campus coffee shop.
e) a critique of the construct validity What are the strengths and/or weaknesses of the
observed operationalizations? Consider the types of validity: face, content, etc. Consider the laundry list of threats. Do so for both constructs!
Mid-term grades representing test performance Has face validity...looks right. Details? What kind of test was the
mid-term? This was only one kind of test...how hard was it? Multi-choice? Essay? Timed? Untimed? Level of class (es)? Need more details!
Diet and free coffee/doughnuts Did they actually partake? Did all of them? On a consistent basis?
What we really expect to see are elevated levels of caffeine and or sugar in the diet (for the treatment groups). Is this going to achieve it?
No control of dose, so potentially very weak operationalization.
e) a critique of the construct validity Is the construct validity issue as described in
previous slide likely to alter the results of the study?
Midterm grades: Could. What does the effect depend on? Exam stress and
it’s response? If so, an easy test might mean a null response, a difficult test an exaggerated one.
Diet: Could: If the resultant dosing was too low on average
across either group, then null results might come out. Might also get great variability within groups (this also lessens the chance of getting a result).
Note: no group with just doughnuts. So no way to examine the effect of sugar alone. Incomplete design.Should be considered, but a
definitive answer is not needed for an “A”
e) a critique of the internal validity What is the study’s design?
R X1 OR X2 OR X3 O
e) a critique of the internal validity Does the study’s design establish
temporal precedence? Yes.
e) a critique of the internal validity Does the study’s design establish
covariation of cause and effect? Yes, potentially. (Did they drink and eat?)▪ In this case the uncertainty over dosage (a
construct threat) also impacts internal validity, because it makes it less certain that we will gather good, clean evidence of covariation of cause and effect.
e) a critique of the internal validity
To what extent does the study’s design control for alternate interpretations of the causal relationship? Does it control for: Single group threats?▪ yes
Multiple group threats?▪ Yes (random assignment – and 40 seems an adequate
number). Social interaction threats?▪ Nope.
e) a critique of the internal validity
Using your response to A-D, to what extent is the assertion that the relationship under investigation is causal a reasonable one? While groups were probably equal at the start
of the experiment, the fact that they all attended the same classes suggests they may have become aware of the alternate conditions of the experiment.
The nature of the conditions was such that numerous social issues could have arisen.
e) a critique of the internal validity
In the event of observed weak internal validity, can you suggest an alternative causal interpretation? Can the social issues change the results of the
experiment?▪ Possibly. ▪ Frustration and feeling cheated could conceivably
result in people in the control group over-compensating by trying harder, or feeling aggrieved and getting extra stressed and doing poorly. ▪ Note the different pattern of results these possibilities would result
in.
Should be considered, but a definitive answer is not needed for an “A”