LTRC 2007 Messick Address Slide 1 June 9, 2007 Toward a Test Theory for the Interactionalist Era Robert J. Mislevy University of Maryland Samuel J. Messick

LTRC 2007 Messick Address Slide 1June 9, 2007

Toward a Test Theory for the Interactionalist Era

Robert J. MislevyUniversity of Maryland

Samuel J. Messick Memorial LecturePresented at the Language Testing Research Colloquium, Barcelona, Spain, June 9, 2007.


Chi, Feltovich, & Glaser, 1981


Key Ideas

Generative patterns from the domain organized experts’ perceptions, understanding, and actions

Experts assembled pieces from patterns to fit particulars of context and purpose» F=MA: Conventional v. situated meaning

Critical role of narrative layer» Integrates principles & equations with context » Locus for understanding, planning, & action


Why Am I Telling You This?

Situative / sociocultural / interactionist perspective is frontier in all domains

Analogues between model-based reasoning and using language

Test theory is model-based reasoning» Interactionalist perspective on test theory

» Bottleneck the narrative layer


Outline

Messick on assessment arguments Interactionalist perspective

» Re language, comprehension, cultural meaning

A narrative space / metaphor for assessment in this light» Attention to senses and roles of context

Implications for building and using measurement models


The Assessment Argument What complex of knowledge, skills, or other

attributes should be assessed? What behaviors or performances should

reveal those constructs? What tasks or situations should elicit those

behaviors? Messick (1994) “The interplay of evidence and consequences in

the validation of performance assessments”


Perspectives on the L2 Construct

What is foregrounded? Trait

» Focus on underlying abilities of individuals that are called upon in a wide variety of situations.

Behaviorist» Focus on context, from external point of view—

success of action in specified classes of situations.

Interactionalist …


An Interactionalist Perspective

Concern with language used in specific discursive practices rather than on language ability independent of context.

Focus on the co-construction of discursive practices by all participants ...

A set of general interactional resources that participants draw upon in specific ways in order to co-construct a discursive practice.

(Young, 2000, p. 5)


An Interactionalist Perspective

successful interaction presupposes not only a shared knowledge of the world, …

but also the construction of a shared internal context … that is built through the collaborative efforts of the interactional partners.

Kramsch ( 1986, p. 367)


Challenges for Assessment Amending the construct of individual ability

to accommodate [how] language use in a communicative event reflects dynamic discourse, which is co-constructed among participants; and …

reconciling [the notion that language ability is local] with the need for assessments to yield scores to generalize across contextual boundaries.

Chalhoub-Deville (2003, p. 373)


A Narrative Frame

Themes from, e.g., cognitive psychology, literacy, neuroscience, anthropology: » Connectionist metaphor, Associative memory

Situated cognition & information processing» Construction-Integration (CI) theory of

comprehension (Kintsch and others)

Individual Sociocultural perspectives» A cognitive theory of cultural meaning

(Strauss & Quinn, 1997)


A Cognitive Theory of Cultural Meaning “Interactional Resources”

External: knowledge “out there”, patterns that exist in use by groups, in tools & processes» Cultural models: What ‘being sick’ means,

restaurant script, Newton’s laws, complaints» Linguistic: Grammar, conventions, frames

Interactional: enable the co-construction of new shared meanings

Internal: patterns in individuals attuned to shared external patterns


Inheritance from Schema Theory

Knowledge as organized patterns, at many levels…

Assembled to understand & to create particular

situations in the world

Developed, strengthened, modified by use Associations of all kinds, including applicability,

affordances, procedures, strategies, affect

» “The user’s knowledge of the language rules is interlocked

with his knowledge of when, where, and with whom to use

them” (Ellis, 1985)


B Inside BInside AContext

A

A la Kintsch: “Conventional” meaning, or propositional content of text / speech…

A la Kintsch: “Conventional” meaning, or propositional content of text / speech…

and all aspects of context…and all aspects of context…



A

The C in CI theory, Construction:Activation of both relevant and irrelevant …Linguistic models, e.g.

•Conventions, Rhetorical frames Cultural models, e.g.,

•Equilibrium, Human motivation

The C in CI theory, Construction:Activation of both relevant and irrelevant …Linguistic models, e.g.

•Conventions, Rhetorical frames Cultural models, e.g.,

•Equilibrium, Human motivation

•If B hasn’t developed a given pattern in past experience, it can’t be activated (although it may get constructed in the interaction).•Relevant patterns from LTM may be activated in some contexts but not others (e.g., physics models; question formation (Tarone)).

•If B hasn’t developed a given pattern in past experience, it can’t be activated (although it may get constructed in the interaction).•Relevant patterns from LTM may be activated in some contexts but not others (e.g., physics models; question formation (Tarone)).



A

The I in CI theory, Integration:•Resulting synthesis of reinforced activated cultural / linguistic / situational patterns:•the Situation model•Akin to ‘stories’ in Larkin’s physics study•Situation model is the understanding

The I in CI theory, Integration:•Resulting synthesis of reinforced activated cultural / linguistic / situational patterns:•the Situation model•Akin to ‘stories’ in Larkin’s physics study•Situation model is the understanding



A

Situation model is also the basis of planning and action.

Situation model is also the basis of planning and action.



Context

Context

Context

A


A B Inside BInside AContext

Context

Context

Context

Ideally, participants activate cultural &

linguistic models that are compatible in relevant ways…

Ideally, participants activate cultural &

linguistic models that are compatible in relevant ways…



Context

Context

Context

To lead to co-constructed meaning / sufficiently (?)

shared understanding

To lead to co-constructed meaning / sufficiently (?)

shared understanding

Kramsch’s "shared internal context"

Kramsch’s "shared internal context"



Context

Context

Context

Preceeding overheads highlight the importance of a common narrative space for thinking about assessment:A = LinguistB = Psychometrician

Preceeding overheads highlight the importance of a common narrative space for thinking about assessment:A = LinguistB = Psychometrician

“shared internal context” re nature & use of knowledge would help ground compatible views of assessment purpose, design, analysis, and use for the job at hand.

“shared internal context” re nature & use of knowledge would help ground compatible views of assessment purpose, design, analysis, and use for the job at hand.


Senses and roles of “context”A B Inside BInside A

Context

Context

Context

Context

Can distinguish external and internal aspects of context (e.g., Douglas, 2000)

Can distinguish external and internal aspects of context (e.g., Douglas, 2000)

Some external aspects of context are public & objective, e.g.,

•Setting•Physical attributes•Directives

Some external aspects of context are public & objective, e.g.,

•Setting•Physical attributes•Directives

Target language use (TLU) featuresAssessment task features(Bachman & Palmer)

Target language use (TLU) featuresAssessment task features(Bachman & Palmer)



Context

Context

Context

Some aspects of context can be interpreted by an external observer through the lens of targeted cultural/linguistic models. e.g., apology situation. In assessment, we can often arrange for this to be the case.

Some aspects of context can be interpreted by an external observer through the lens of targeted cultural/linguistic models. e.g., apology situation. In assessment, we can often arrange for this to be the case.

The question at issue in assessment is whether the examinee activates targeted compatible l/c models, then constructs and acts accordingly through a corresponding situation model.

The question at issue in assessment is whether the examinee activates targeted compatible l/c models, then constructs and acts accordingly through a corresponding situation model. Note the need to activate many other l/c

models in order to construct a situation model, plan, and carry out action.•Many places to slip, but others to compensate.•“Alternative explanations” in assessment argument.

Note the need to activate many other l/c models in order to construct a situation model, plan, and carry out action.•Many places to slip, but others to compensate.•“Alternative explanations” in assessment argument.



Context

Context

Context

Some aspects of context can be interpreted by an external observer through the lens of targeted cultural/linguistic models. e.g., apology situation. In assessment, we can sometimes arrange for this to be the case; sometimes watch for it to happen.

Some aspects of context can be interpreted by an external observer through the lens of targeted cultural/linguistic models. e.g., apology situation. In assessment, we can sometimes arrange for this to be the case; sometimes watch for it to happen.

As assessment designers, we use these situations that call upon targeted linguistic/ cultural models to determine what examinee actions would signal recognition, comprehension, action through them.

As assessment designers, we use these situations that call upon targeted linguistic/ cultural models to determine what examinee actions would signal recognition, comprehension, action through them.This sense of context plays a key role in

•Evaluation of performance, hence •Observable variables that go into a measurement model.

This sense of context plays a key role in•Evaluation of performance, hence •Observable variables that go into a measurement model.


What can we say about individuals?

Use of interactional resources in appropriate contexts in appropriate ways; i.e.,

Attunement to targeted cultural/linguistic patterns: Recognize markers of externally-viewed patterns? Construct internal meanings in their light? Act in ways appropriate to targeted c/l models in the

assessment contexts? What are the ranges and circumstances of activation?

(variation of performance across contexts)


Implications for measurement models

Basic form: Probability of aspects of performance Xij given

parameters for person i and situation j

(all could be vector-valued)

jiij ,XProb

• Way too simple • No explicit connection with CI comprehension model, interaction processes, etc.• Apparent separation of person and situation characteristics

These are indeed properties of the conventional meaning of the measurement model and parameters.

These are indeed properties of the conventional meaning of the measurement model and parameters.


An Interactionalist Perspective: Instantiation in a Context

Xs result from particular persons calling upon resources in particular contexts (or not, or how)

Mechanically s simply accumulate info across situations

Our chosing situations and what to observe drives their situated meaning.

Situated meaning of s are tendencies toward these actions in these situations that call for certain interactional resources, via l/c models.



Inference to criterion contexts (TLU) depends on analysis of what l/c models are called upon in assessment use argument…

What is similar, what is different, re the resources task & criterion situations call for?

To what degree does activation and success in task context correspond to activation and success in criterion context?

(e.g., Bachman, Chalhoub-Deville, Douglas, Chapelle)



What demands do we minimize via task design, so needn’t model?

What resources do we already know examinees can draw upon, so tasks can require them but we needn’t model?» “Hidden” but essential to meaning» Occupational English Test (McNamara)» Analogous to ‘focus on forms’ learning



How to model inconsistent performance? If “unmotivated,” it’s noise; via probability model Promising direction: Model individual’s degree or

pattern in variation in terms of context features If “motivated”: Model in terms of s

» Divide & Conquer: Multiple unidimensional tests (OET)

» Exploratory multidimensional: Discover patterns in data. » Controlled: Structured multidimensional models

(e.g., Embretson, Adams & Wilson, von Davier)


Structured Multidimensional Models

Still way too simple, but …» purposeful modeling of motivated variation in

complex tasks when persons differ in targeted ways» exploit what is known about examinees but not

modeled

Controlled mixes of demand features» E.g., in OET-like situations, wrt medical knowledge,

complexity of stimulus language, complexity of language to be produced.

“Throwing the data over the wall” won’t work


Structured Multidimensional Models

Probabilities modeled in terms of task features:» Which dimensions are relevant for which observables

from which tasks? (Robinson’s “difficulty” features)» Task parameters modeled in terms of Robinson’s

“complexity” features.

Hence a priori structure of patterns to interpret » Can organize s in terms of traits or context features

Coordinated task design and measurement model» Create tasks within task models


How much can testing gain from modern cognitive psychology?

So long as testing is viewed as something that takes place in a few hours, out of the context of instruction, and for the purpose of predicting a vaguely stated criterion,

then the gains to be made are minimal. Buzz Hunt (1986)

Conclusion


I have argued that we need to capitalize on [method effects] by designing tests for specific populations -- tests that contain instructions, content , genre, and language directed toward that population.

The goal is to produce tests … that would provide information interpretable as evidence of communicative competence in context.

Douglas (1998)

Conclusion


Conclusion

Interactionalist view of test theory… for assembling, analyzing, and interpreting

assessments, for arguments in interactionalist view of language Methods and exemplars needed,

but more pressing need is narrative frame … » To connect view of language proficiency with the

machinery of test theory, » Toward modeling purposeful variations in a coherent

design space.

Documents

LTRC 2007 Messick Address Slide 1 June 9, 2007 Toward a Test Theory for the Interactionalist Era Robert J. Mislevy University of Maryland Samuel J. Messick