Comments on “The Role of Large Scale Assessments in ...€¦ · Unpacking the CIPO model also requires accounting for the fact that these processes are, by deﬁnition, mediators,

Comments on“The Role of Large Scale Assessments in Research on Educational

Effectiveness and School Development”by Eckhard Klieme, Ph.D.

David KaplanDepartment of Educational Psychology

The General Theme

❖The General Theme❖ Important Features ofLSAs❖ Issues of CausalInference using ILSAs

❖The CounterfactualDefinition of Causation❖Mackie and the INUSCondition for Causation❖Woodward and theManipulability Theory ofCausation

❖ Invariance❖What does this have todo with ILSAs?❖What aboutmethodology?

❖Conclusion

ILSA 2011 2 / 18

● This paper addresses a large number of important issues that go wellbeyond Eckhard’s presentation:

1. The paper presents a very nice and succinct history of ILSAs.

2. The paper raises important issues around problems of causalinference in the context of ILSAs.

3. The paper presents some empirical findings that address whatcan be learned from ILSAs and how ILSAs can be improved foreducational effectiveness studies.

● In the time I have, I would like to focus on issues raised in this paperregarding causal inference in ILSAs.

Important Features of ILSAs




❖Conclusion

ILSA 2011 3 / 18

● Among the many uses of ILSAs that Eckhard articulates, an importantuse is that ILSAs can establish a knowledge base regardingeducational effectiveness.

● We come to understand educational effectiveness by unpacking theinputs, processes and outputs of the educational system.

● But inputs, processes, and outputs exist, as Eckhard notes, withincontexts.

● This leads to the CIPO model which we find underlies our work onPISA of which Eckhard and others has made substantial contributions.

● Unpacking the CIPO model also requires accounting for the fact thatthese processes are, by definition, mediators, and hence structuralspecifications are required.

● The CIPO model, as Eckhard points out, is also multilevel, andtherefore multilevel specifications are required.

● Finally, the CIPO model implies an underlying dynamic, and this needsto be considered as well.

Issues of Causal Inference using ILSAs




❖Conclusion

ILSA 2011 4 / 18

● Eckhard then goes on to raise issues regarding the limitations of ILSAsfor effectiveness studies, and for drawing causal inferences inparticular.

● Naturally all data collection and design paradigms have limitations (e.g.generalizability of RCTs), and those that are raised by Eckhard areimportant to note.

● However, I am not as pessimistic about using ILSAs for causalinference as Eckhard seems to be, and I would like to attend to certainissues that I think might have a solution along the lines of theenhancements to ILSAs raised by Eckhard.

● Of course, there is always the chance that I am a naive optimist.

● But first...my biases.




❖Conclusion

ILSA 2011 5 / 18

● My orientation is toward a structural approach to causal inference, but Iam a pluralist in the sense of the philosopher Nancy Cartwright(assuming I understood her).

● I believe that there are many ways to get at how interventions mightlead to change.

● I also believe that the limitations of design constrain how we mightassess causal effects. Sometimes RCTs are needed, sometimes ILSAsare available and should be exploited.

● I also believe that a distinction can be made between studying “theeffects of causes” versus “the causes of effects” (Holland, 1986).

● Whereas the focus in education effectiveness (at least in the US) hasbeen on the former via RCTs, I believe that ILSAs are very well suitedfor the latter.

● Despite my pluralism, I accept that the potential outcomes frameworkof Neyman and Rubin is the strongest conceptual framework for causalinference.




❖Conclusion

ILSA 2011 6 / 18

● To fully realize the utility of ILSAs for examining the causes of effects,we need to worry about some issues that Eckard raises, but not others.

● Focusing on PISA, Eckhard understandably worries that ILSAs can’t beused for causal inference because prior achievement is not obtained.

✦ However, as PISA is a “yield study” and not an achievement test, itis unclear what prior yield would mean in this context. Of course,longitudinal data would be ideal, but then yield would indicate aschool or classroom effect, changing, perhaps, the meaning thegoals of PISA.

● Eckhard is also concerned that ANCOVA or propensity score balancingmight not be successful because prior achievement variable is notavailable.

✦ However, as nice as having prior achievement would be, we know(e.g. Steiner, et al., 2010) that we can achieve very good balancewith a judicious choice of covariates, reliably measured and relevantto the selection mechanism. QEGs need to consider this carefully.




❖Conclusion

ILSA 2011 7 / 18

● It is this judicious choice of covariates that Eckhard raises as the mostimportant limitation of ILSAs – and I agree.

● Regardless of whether one desires to use ILSAs to study a particularquasi-experimental effect or (but not in opposition to) specifying a largemultilevel structural model, the choice of covariates and how they aremeasured is of utmost importance.

● The issue of how the inputs and processes are measured allows one toconsider the counterfactual/potential outcomes theory of causalinference more seriously in the ILSA context.

● My view, which I have shared with Eckhard, examines how ILSAs canbe used within the counterfactual/potential outcomes framework, rootedin Mackie’s (1980) idea of the INUS condition, Woodward’s (2003)ideas of manipulability, and Hoover’s (e.g. 2001) structural econometricperspective.

The Counterfactual Definition of Causation




❖Conclusion

ILSA 2011 8 / 18

● To keep things simple, what does it mean to say “x causes y”?

● Based on Hume’s (second) definition of causality, the counterfactualdefinition of causation is

✦ “If x occurs then y occurs, and y would not have occurred if x hadnot”

● This is the operating definition underlying “potential outcomes”literature (e.g. Neyman, 1923, Rubin, 1974). ”...if x had not” is thecounterfactual.

● Under “potential outcomes”, we must imagine the possibility of anindividual causal effect

δi = y1i − y0

i (1)

where δi is the treatment effect for individual i under the treatmentcondition and the control condition.

Mackie and the INUS Condition for Causation




❖Conclusion

ILSA 2011 9 / 18

● This definition is too strong, because y can occur if x had not.

● We need to understand the context under which “x occurred”

● Let A, B, C..., etc, be a list of factors that lead to some effect wheneversome conjunction of the factors occurs. For example, take threeconditions under which reading development can take place: incomingphonemic awareness; teacher competency; parental reading activities.

● A conjunction of events may be ABC or DEF or JKL, etc.

● This allows for the possibility that ABC might be a cause or DEF mightbe a cause, etc.

● Assume the set is finite. Then factors (ABC or DEF or JKL) is acondition that is both necessary and sufficient for the effect to occur.

● Each specific conjunction, such as ABC, is sufficient but not necessaryfor the effect.

Mackie (cont’d)




❖Conclusion

ILSA 2011 10 / 18

For example:

● ABC is a “minimal sufficient” condition insofar as none of its conjunctsare redundant.

● AB is not sufficient for the effect.

● A itself is neither a necessary nor sufficient condition for the effect.

● However, the single factor A, is related to the effect in an importantfashion - viz.

✦ “[I]t is an insufficient but non-redundant part of an unnecessary butsufficient condition: it will be convenient to call this... an inuscondition.” (pg. 62)

● If that is confusing, consider calling it a Necessary Element of aSufficient Set.

Mackie (cont’d)




❖Conclusion

ILSA 2011 11 / 18

● But what about remaining relevant causes?

● According to Mackie, these remaining causes are relegated to thecausal field which includes what we recognize as the disturbance term.

● Hoover argues that the notion of a causal field has to be expanded forMackie’s ideas to be relevant to indeterministic problems.

● If parameters of a causal question are truly constant or mostly stable,then they can be relegated to the causal field without too much worry.Prior cycles of PISA, say, would help determine this.

“The causal field is a background of standing conditionsand, within the boundaries of validity claimed for the causalrelation, must be invariant to exercises of controlling theconsequent by means of the particular causal relation (INUScondition) of interest” (Hoover, 2001, pg. 222)




❖Conclusion

ILSA 2011 12 / 18

● The INUS condition should be an attractive idea to those of us workingon the development and application of ILSAs to support causalinferences because it focuses our attention on some aspect of thecausal problem without having to be concerned directly with knowingevery minimal sufficient subset of the full cause of the event.

● In fact, I would argue that a fruitful collaboration of content area expertgroups (REG, MEG, SEG) along with the QEG, with focused attentionon instantiating INUS causal variables, could lead to ILSAs being moreproductively used for educational effectiveness studies.

Woodward and the Manipulability Theory of Causation




❖Conclusion

ILSA 2011 13 / 18

● In many situations, the goal is to change x – i.e. intervene on x – tochange y .

● In practice, we may want to engage in model-based simulationsinvolving a hypothetical change in x as a precursor to an experiment orto gain insights into potential problems when scaling up an intervention.

● Intervention on x requires that x be measured so as to support acounterfactual proposition.

● Recently, a manipulability theory of causation was put forth byWoodward (2003) as an attempt to provide a firmer foundation forcausal explanation.

● For Woodward a causal explanation is an explanation that providesinformation for purposes of manipulation and control.

● An important aspect of Woodward’s theory involves clarifying theproblem of intervention and invariance.

Invariance




❖Conclusion

ILSA 2011 14 / 18

● Intervention relates to the notion of an ideal experimental manipulationof a variable, x , with the goal of determining if this ideal manipulationchanges y , and that no change in y would have occurred without thechange in x .

● For Woodward, a necessary and sufficient condition for the descriptionof a causal relationship is that it be invariant under some appropriateset of interventions.

● If a purported causal relationship is not invariant under a set ofinterventions, it is not, in fact, a causal relationship – and indeed mightonly describe a correlation.

● Statistically, invariance concerns the extent to which the parameters ofthe conditional distribution do not change when there are changes inthe parameters of the marginal distribution.

● This leads to the problem exogeneity.




❖Conclusion

ILSA 2011 15 / 18

● How might we conceive of this invariance and exogeneity problem inthe context of ILSAs?

● Over cycles of PISA (for example), we can examine statisticalrelationships among variables that have been repeatedly measured,say for trend.

● Examining these relationships with an eye toward major country-levelpolicy shifts, we could then examine if these relationships haveremained somewhat stable.

● Given the careful nature of PISA sampling (particular because of itsneed to preserve trend), any clear change in relationships of interestmight signal super-exogenous changes in policy that has led toproblems of parameter invariance.

● Careful linkages with system level data provided by the OECD couldgive insights into the nature of these changes.

What does this have to do with ILSAs?




❖Conclusion

ILSA 2011 16 / 18

● In my view ILSAs provide the data collection framework to measurevariables that can be used to help define the causal field in whichinterventions take place. This is the C in the CIPO model that Eckharddiscusses.

● ILSAs can be enhanced to ask specific questions regarding the inputsand processes of schooling. Here, Eckhard and I are in completeagreement and Eckhard’s paper does a nice job of pointing outpossible enhancements.

● However, these questions should be measured in such a way as tosupport a hypothetical manipulability question of the sort “what if thingswere different”?

● I’m less concerned about outputs. ILSAs such as PISA deal with themvery well.

What about methodology?




❖Conclusion

ILSA 2011 17 / 18

● For an analytic framework to support these ideas, I believe thatmultilevel structural modeling within a Bayesian perspective is mostuseful. Why?

1. Multilevel SEM (Muthen & Satorra, 1995; Kaplan & Kreisman,2000) can address mediating and moderating effects within amultilevel framework (ala Levin).

2. The Bayesian perspective of multilevel SEM (Kaplan & Depaoli, inpress) allows one to incorporate prior information into modelestimation via careful elicitation (e.g. from prior cycles of PISA;consultation with PISA expert groups, etc.).

3. Bayesian multilevel SEM directs us away from some of theinternal inconsistencies of null hypothesis testing and towardmodel selection based on predictive capacity.

4. Bayesian multilevel SEM (and conventional SEM for that matter)can allow for model-based simulations of counterfactuals withinvarying context “settings”. This can direct us toward evolutionaryknowledge development within the CIPO framework of schooling.

Conclusion




❖Conclusion

ILSA 2011 18 / 18

● Although I might not share Eckhard’s pessimism regarding the use ofILSAs for causal inference, his paper gives us a great deal to thinkabout, and is well worth reading.

● It is extremely important to consider the enhancements he suggests.But I also believe we have the methodologies and philosophicalunderpinnings in place to engage in bold modeling efforts.

● The issues raised in his paper need to be taken very seriously if we areto move forward in developing ILSAs to support an evolutionaryknowledge base of research on educational effectiveness and schooldevelopment in an international context.

Documents

Comments on “The Role of Large Scale Assessments in ...€¦ · Unpacking the CIPO model also requires accounting for the fact that these processes are, by deﬁnition, mediators,