1

Exploring Quasi-Experiments Lab 5: May 9, 2008

Guthrie, J.T., Wigfield, A., & VonSecker, C. (2000). Effects of integrated instruction on motivation and strategy use in reading. Journal of Educational Psychology, 92, 331-341.

2

Purpose of Research

• Causal relationship– To identify whether the

Concept-Oriented Reading instruction (CORI) intervention produces greater intrinsic motivation and strategy-use than traditional instruction

3

Variables of interest• The Independent Variables (causes)

– Instructional intervention:• CORI intervention• traditional comparison group

• The Dependent Variables (outcomes) – Intrinsic motivation (operationalized: curiosity,

involvement, and preference for challenge)– Extrinsic motivation (operationalized:

recognition and competition)– Strategy use (operationalized: self-report of

cognitive strategies)

• The Covariates– Past achievement (operationalized:

standardized reading achievement scores—CBST/MAT)

4

Participant assignment

• Non-random assignment into intervention and comparison groups (i.e., into classrooms)

• Classrooms assigned based on comparable “subjective matching” of teachers, students, and school settings

5

Cook and Campbell’s UTOS

• Units: 3rd and 5th grade low-achieving students.

• Treatment: Concept-oriented reading instruction (CORI)

• Observations: intrinsic and extrinsic motivation, and strategy use (also past achievement).

• Setting: three different mid-Atlantic grade schools.

6

Threats to Internal ValiditySpecific to Quasi-Experiments• History

– Some event related to the outcome could occur to one group but not the other

• Maturation– Groups may differ in rate of change on the

outcomes prior to treatment

• Instrumentation– Measurement might change from pre to post

test in only one group

• Statistical Regression– Variable used to determine group selection may

be unreliable or unstable

7

Threats to Internal ValiditySpecific to Quasi-experiments• History: A celebrity may visit the schools to

discuss the importance of reading; CORI students may be more susceptible to the message and therefore show greater gains in motivation (due to the celebrity, not the intervention).

• Maturation: Children in the traditional classrooms could be losing motivation at a faster rate than those in the CORI classrooms prior to treatment.

• Instrumentation: Likely not an issue given the lack of pretest design.

• Statistical Regression: The choosing of similar classrooms based on teacher, student, and school make-up could have been based on inaccurate, unreliable, or temporarily skewed subjective judgments.

8

AdditionalThreats to Internal Validity• Lack of pretest makes it difficult to say whether

differences are due to CORI or whether the differences existed at the onset of the research

• Attrition: 11% for grade 3 and 17% for grade 5 due to moving.

• Resentful Demoralization: teachers or students could have shown less motivation knowing that they were not getting the treatment (p.334/ p. 47)

• Compensatory Rivalry: comparison teachers may have tried harder because of the study (mentioned on p. 334 as the John Henry effect).

• Third variables: Teacher expectations could have produced results

• Authors did enhance their design by including extrinsic motivation (p. 332) as a nonequivalent dependent variable (p. 74).

9

Threats to Construct Validity• Diffusion of treatment: Traditional teachers

frequently visited the CORI classrooms and adopted some texts used in the CORI condition. – The authors did collect video, interview, and

questionnaire data regarding the use of the treatment integrity in the CORI classrooms.

• Low internal consistency: Several internal consistencies were low; because reliability puts a lid on validity, this is cause for concern.

• Mono-operation bias: Motivation is captured using only aspects of the Motivation for Reading Questionnaire (MRQ) when other methods may be possible. (e.g., Teacher-reports on individual students).– The authors did address more than one aspect

of motivation (intrinsic and extrinsic)

10

Threats to External Validity

• Schools are chosen based on need, therefore results may not generalize to less needy students/schools.

• Results may not generalize to other grades

• Results may not generalize to interventions that are less intensive.

• Results may not generalize to areas outside of the mid-Atlantic.

• Results may not generalize to participants from dissimilar schools.

11

Threats to Statistical Conclusion Validity

• Violated statistical assumptions: Group administration of treatment violates assumption of independent observations (p. 49).– Addressed by analyzing

the data using HLM

12

Improving the research• Proximal Similarity: The traditional instruction

comparison group could have maintained integrity and differentiation from CORI instruction group. The measures could have more closely approximated intrinsic motivation.– Researchers did a good job employing similar

settings and units to whom they wished to generalize.

– Researchers did a good job ensuring that CORI instruction was being implemented in the CORI classrooms

• Heterogeneous Irrelevancies: Triangulating measures of motivation could have improved construct validity.– The authors did use two different Units and three

different Settings.• Causal explanation: Rule out third variables and

include pretest.