Advanced Study Design February 19, 2010. Today’s Class Last Week’s Probing Question Advanced Study Design Assignments

  • View
    213

  • Download
    0

Embed Size (px)

Text of Advanced Study Design February 19, 2010. Today’s Class Last Week’s Probing Question Advanced...

  • Slide 1
  • Advanced Study Design February 19, 2010
  • Slide 2
  • Todays Class Last Weeks Probing Question Advanced Study Design Assignments
  • Slide 3
  • Probing Question Lets say you wanted to do a large-scale research study on boredom Under what conditions would it be preferable to use Questionnaire items Experience sampling method Quantitative field observations
  • Slide 4
  • Todays Class Last Weeks Probing Question Advanced Study Design Assignments
  • Slide 5
  • Today Validity Validity Threats Stratification Counterbalancing and Cross-over Designs Regression-Discontinuity Designs
  • Slide 6
  • Validity Useful jargon...
  • Slide 7
  • Validity (Trochim & Donnelly, 2007) Conclusion validity Internal validity Construct validity External validity Ecological validity
  • Slide 8
  • Conclusion Validity The degree to which conclusions you reach about relationships in your data are justified
  • Slide 9
  • Internal Validity Assuming that there is a relationship in the study, can you justifiably infer that the relationship is causal?
  • Slide 10
  • Construct Validity The degree to which inferences can legitimately be made from the operationalizations in your study, to the theoretical constructs on which those operationalizations were based
  • Slide 11
  • External Validity Do your results generalize to other people, procedures, places, and times?
  • Slide 12
  • Ecological Validity What is the degree to which the methods, materials, and settings of the study are relevant to natural/legitimate settings?
  • Slide 13
  • Ecological vs. External Validity Ecological validity not about *generalization* to real-life situations about the whether the "methods, materials and settings" are similar (or identical) to real life. Ecological validity is about real-world *relevance* External validity is about generalizability
  • Slide 14
  • Examples? High External Validity, Low Ecological Validity Low External Validity, High Ecological Validity
  • Slide 15
  • High External Validity, Low Ecological Validity Lab studies of seductive details effect Instruction that does not include interesting but ultimately irrelevant details leads to better learning, for students of variety of ages performed in lab settings at 2 universities with children of different socio-economic status (SES) & race
  • Slide 16
  • Low External Validity, High Ecological Validity A classroom study, with real students, involving legitimate educational tasks, presented in exactly the way a teacher would present them
  • Slide 17
  • Low External Validity, High Ecological Validity A classroom study, with real students, involving legitimate educational tasks, presented in exactly the way a teacher would present them With 1 student in each condition
  • Slide 18
  • Lets consider a few examples Vote on which type of validity is violated (any of the five, could be multiple, could even be none) Explain your reasoning
  • Slide 19
  • Which type of validity is violated? Students who read bug messages perform more poorly on post-test So bug messages hurt learning! You have chosen a categorical variable for the X axis; however, scatterplot graphs can only contain numerical variables.
  • Slide 20
  • Which type of validity is violated? I have proven that students learn more Calculus from my Calculus tutoring system Here is my test, used both pre and post How well do you know Calculus? 1 2 3 4 5 Not well Very well
  • Slide 21
  • Which type of validity is violated? My new tutoring system is much better than the previous tutoring system!
  • Slide 22
  • Which type of validity is violated? My new tutoring system is much better than the previous tutoring system!
  • Slide 23
  • Which type of validity is violated? I conducted a study comparing my new tutoring system to a previous one Students who completed the whole tutoring system performed significantly better on post-test in the experimental condition than control condition
  • Slide 24
  • Which type of validity is violated? I conducted a study comparing my new tutoring system to a previous one Students who completed the whole tutoring system performed significantly better on post-test in the experimental condition than control condition Oops did I mention only 3% of students completed the whole tutoring system in the control condition?
  • Slide 25
  • Which type of validity is violated? Now that I have tested my new learning environment that responds to off-task behavior by giving it to single students in the guidance counselors office after school, we can be confident it will work in all school settings
  • Slide 26
  • Which type of validity is violated? Now that I have tested my new learning environment with a set of 10 8 th graders in Tuktoyaktuk (Northwestern Territory of Canada), all bilingual English-Inuvialuit, with fathers who work in the mine nearby, we can be confident it will work for all students
  • Slide 27
  • Which type of validity is violated? Now that I have tested my new learning environment with a set of 120 8 th graders in a predominantly middle-class Caucasian suburb of Worcester, we can be confident it will work for all students
  • Slide 28
  • Some Popular Threats to Internal Validity
  • Slide 29
  • Maturation Threat Something happens between pre-test and post-test, aside from your intervention, that impacts student change E.g. the same thing would have happened whether or not you ran your study
  • Slide 30
  • Maturation Threat Something happens between pre-test and post-test, aside from your intervention, that impacts student change E.g. the same thing would have happened whether or not you ran your study Any horror stories from your research?
  • Slide 31
  • Maturation Threat Something happens between pre-test and post-test, aside from your intervention, that impacts student change E.g. the same thing would have happened whether or not you ran your study One teacher taught the same material in class during the same week as the study
  • Slide 32
  • Mortality Threat Common in urban classrooms
  • Slide 33
  • Mortality Threat Large numbers of participants systematically drop out of the study Any horror stories from your research?
  • Slide 34
  • Mortality Threat Large numbers of participants systematically drop out Example: I ran a study with homeschool students; response rates were different between conditions
  • Slide 35
  • Regression to the Mean If you choose a group based on pre-test performance The most frequent gamers The students who scored in the bottom 10% on the pre-test Some of them were in that group by chance And can be expected to do better on the post- test
  • Slide 36
  • Diffusion of Treatment You assign kids to different conditions, but they see each others screens (or talk in the hallway, etc.) You assign classes randomly to condition within-teacher, but teachers learn strategies from the better condition and use that knowledge in the other condition
  • Slide 37
  • Diffusion of Treatment You assign kids to different conditions, but they see each others screens (or talk in the hallway, etc.) You assign classes randomly to condition within- teacher, but teachers learn strategies from the better condition and use that knowledge in the other condition A major study comparing curricula in Baltimore was called into question because teachers took teaching strategies from the experimental condition to the control condition
  • Slide 38
  • Compensatory rivalry/ resentful demoralization Students in condition A learn about condition B, which is obviously better Resentful demoralization its no fair they got the better software, lets just quit Compensatory rivalry we can beat them, even if they got the better software
  • Slide 39
  • Compensatory rivalry/ resentful demoralization Students in condition A learn about condition B, which is obviously better Resentful demoralization its no fair they got the better software, lets just quit More common for students Compensatory rivalry we can beat them, even if they got the better software More common for teachers
  • Slide 40
  • Confounding You changed multiple things in your intervention (often inadvertantly), and its not clear which change had the impact Some examples?
  • Slide 41
  • Confounding Your meta-cognitive intervention takes longer to go through Better learning, or just more time-on-task?
  • Slide 42
  • Comments? Questions?
  • Slide 43
  • Stratification
  • Slide 44
  • Pure random sampling Lets say you have an intervention that you want to test in 4 groups: urban, wealthy suburban, working-class suburban, and rural students You have access to students in Worcester, Auburn, Ashburnham, and Cambridge If you just randomly sample in your population, you are going to get a lot more people from Worc