If Language is a Complex Adaptive System, What is Language Assessment?

(1)

If Language is a Complex Adaptive System, What is Language Assessment?

Presented at “Language as a Complex Adaptive System”, an invited conference celebrating the 60th Anniversary of Language Learning, at the University of Michigan, Ann Arbor, MI, November 7-9, 2008. The first author was supported by a grant from the Spencer Foundation.

Robert J. MislevyUniversity of Maryland

Chengbin YinCenter for Applied Linguistics

(2)

Key Ideas Assessment as evidentiary argument, not

simply as measurement. Arguments constructed around …

» View of the nature of proficiency.» Situations and ways people acquire it and

use it. Relevant work taking place in language

testing from an interactionist perspective. Reconceiving measurement models

(3)

The Assessment Argument(Messick, 1994)

What complex of knowledge, skills, or other attributes should be assessed?

What behaviors or performances should reveal those constructs?

What tasks or situations should elicit those behaviors?

We’ll look at a more technical representation in a little while.

(4)

Snow & Lohman, 1989

Summary test scores, and factors based on them, have often been though of as “signs” indicating the presence of underlying, latent traits. [] …

An alternative interpretation of test scores as samples of cognitive processes and contents … is equally justifiable and could be theoretically more useful.

(5)

LaCAS and Assessment Arguments

Interactionalist perspective in language testing:

» Communicative competence

» Contextual features of tasks

» Language tests for specific purposes

(6)

An Interactionalist Perspective (Young, 2000, 2008)

… language used in specific discursive practices rather than … language ability independent of context.

Focus on the co-construction of discursive practices by all participants ...

A set of general interactional resources that participants draw upon in specific ways in order to co-construct a discursive practice.

(7)

An Interactionalist Perspective (Young, 2000, 2008)

Relationship between participants’

employment of interactional resources and

the context in which they are employed.

Varying with the practice and the

participants…

(8)

Challenges for Assessment (Chalhoub-Deville, 2003)

Amending the construct of individual ability to accommodate [how] language use in a communicative event reflects dynamic discourse, which is co-constructed among participants; and …

reconciling [the notion that language ability is local] with the need for assessments to yield scores to generalize across contextual boundaries.

(9)

Sociocognitive Foundations

Themes from, e.g., cognitive psychology, literacy, neuroscience, anthropology: » Connectionist metaphor, associative memory

Situated cognition & information processing» Construction-Integration (CI) theory of

comprehension (Kintsch and others)

Individual Sociocultural perspectives» A cognitive theory of cultural meaning

(Strauss & Quinn, 1997)

(10)

A Cognitive Theory of Cultural Meaning “Interactional Resources”

Extrapersonal:» Cultural models: What ‘being sick’ means,

restaurant script, Newton’s laws, complaints» Linguistic: Grammar, conventions, constructions

Intrapersonal: » Patterns from experience at many levels» Schemas / frames / understandings / assumptions

Interplay Situated understandings » Access to, and ways of interacting with, shared

structures in order to accomplish goals

The user’s knowledge of the language

rules is interlocked with his knowledge

of when, where, and with whom to use

them. (R. Ellis, 1985)

(11)

B Inside BInside A A

observablenot observable not observable

(12)

B Inside BInside AContext

A

A la Kintsch: Propositional content of text / speech…

A la Kintsch: Propositional content of text / speech…

and internal and external aspects of context …

and internal and external aspects of context …

(13)


A

The C in CI theory, Construction:Activation of both relevant and irrelevantbits of cultural models, experiences, e.g.,

•Restaurant script, Human motivation Guided in part by linguistic models, e.g.

•Conventions, constructions, rhetorical framesContent of utteranceHistory with interlocutorConversation thus far

The C in CI theory, Construction:Activation of both relevant and irrelevantbits of cultural models, experiences, e.g.,

•Restaurant script, Human motivation Guided in part by linguistic models, e.g.

•Conventions, constructions, rhetorical framesContent of utteranceHistory with interlocutorConversation thus far

E.g., tasks in Occupational English Test (OET; McNamara, 1996) call upon patterns re language, but also genre, medical knowledge, use of information in clinical settings.

E.g., tasks in Occupational English Test (OET; McNamara, 1996) call upon patterns re language, but also genre, medical knowledge, use of information in clinical settings.

•If a pattern hasn’t been developed in past experience, it can’t be activated (although it may get constructed in the interaction).•A relevant pattern from LTM may be activated in some contexts but not others (e.g., physics models; question formation (Tarone)).

•If a pattern hasn’t been developed in past experience, it can’t be activated (although it may get constructed in the interaction).•A relevant pattern from LTM may be activated in some contexts but not others (e.g., physics models; question formation (Tarone)).

(14)


A

The I in CI theory, Integration:•Situation model: synthesis of coherent / reinforced activated cultural / linguistic / situational patterns •Situation model is basis of understanding

The I in CI theory, Integration:•Situation model: synthesis of coherent / reinforced activated cultural / linguistic / situational patterns •Situation model is basis of understanding

(15)


A

Situation model is also the basis of planning and action.

Situation model is also the basis of planning and action.

(16)


Context

Context

Context

A

Previous situation models are input to subsequent situation models.

Previous situation models are input to subsequent situation models.

(17)


Context

Context

Context

A

Ideally, activation of relevant and

compatible cultural & linguistic models…

Ideally, activation of relevant and

compatible cultural & linguistic models…

(18)


Context

Context

Context

A

to lead to (sufficiently) shared understanding;

i.e., co-constructed meaning.

to lead to (sufficiently) shared understanding;

i.e., co-constructed meaning.

Kramsch’s "shared internal context "

Kramsch’s "shared internal context "

(19)

Comments about context…A B Inside BInside A

Context

Context

Context

Context

Can distinguish external and internal aspects of context (e.g., Douglas, 2000)

Can distinguish external and internal aspects of context (e.g., Douglas, 2000)

External / pubic aspects of context, e.g.,

•Setting•Physical attributes

External / pubic aspects of context, e.g.,

•Setting•Physical attributesRe assessment,

Target language use (TLU) featuresTask features

Re assessment,Target language use (TLU) featuresTask features

(20)

A B Inside BInside AContext

Context

Context

Context

Aspects of cultural/linguistic/interaction context as interpreted by an external observer.

Aspects of cultural/linguistic/interaction context as interpreted by an external observer.

Used to determine what actions signal recognition, comprehension, action through targeted cultural /linguistic models.

Used to determine what actions signal recognition, comprehension, action through targeted cultural /linguistic models.

As such, in assessment, plays role in•Evaluation, hence •Observable variables

As such, in assessment, plays role in•Evaluation, hence •Observable variables

(21)

What can we say about individuals?

Use of resources in appropriate contexts in appropriate ways; i.e.,

Attunement to cultural/linguistic patterns: Recognize markers of externally-viewed patterns? Construct internal meanings in their light? Act in ways appropriate to targeted cultural/linguistic

models? What is the range and circumstances of activation?

(variation of performance across contexts)

(22)

The Assessment Argument(Messick, 1994)

What complex of knowledge, skills, or other attributes should be assessed?

What behaviors or performances should reveal those constructs?

What tasks or situations should elicit those behaviors?

(23)

Toulmin’s Argument

Claim

Backing

unless

sinceWarrant

Alternativeexplanationso

Data

Structure

(24)

Student acting inassessment situation

on account of

Backing concerning assessment situation

Alternative explanations

unless

Warrantconcerning assessment

since

Warrant concerning evaluation since

Warrant concerning task design since

Other information concerning student vis a vis

assessment situation

so

Claim about student

Data concerning student

performance

Data concerning task situation

(25)


on account of



unless


since





so

Claim about student


performance


In interactive task,•performance flows in time, •performance changes situation, •may or may not be series of task and observable variables

In interactive task,•performance flows in time, •performance changes situation, •may or may not be series of task and observable variables

(26)


on account of



unless


since





so

Claim about student


performance


Concerns features of (possibly evolving) context as seen from the view of the assessor – in particular, those seen as relevant to targets of inference.

Concerns features of (possibly evolving) context as seen from the view of the assessor – in particular, those seen as relevant to targets of inference.

(27)


on account of



unless


since





so

Claim about student


performance


Evaluation of performance concerns context features indirectly: clues that suggest attunement to features of cultural / linguistic models of interest. (did examinee recognize, comprehend, act accordingly?)

Evaluation of performance concerns context features indirectly: clues that suggest attunement to features of cultural / linguistic models of interest. (did examinee recognize, comprehend, act accordingly?)

(28)


on account of



unless


since





so

Claim about student


performance


“Hidden” aspects of context—not in test theory model but essential to argument: What attunements to cultural / linguistic models can be presumed among examinees, to condition inference re targeted l/c models?

“Hidden” aspects of context—not in test theory model but essential to argument: What attunements to cultural / linguistic models can be presumed among examinees, to condition inference re targeted l/c models?

Fundamental to situated meaning of student variables in measurement models;Both critical and implicit.

Fundamental to situated meaning of student variables in measurement models;Both critical and implicit.

(29)


on account of



unless


since





so

Claim about student


performance


Assessment context always has its own features that activate some cultural / linguistic models and suppress others in different ways for different examinees.(i.e., “method effects”)

Assessment context always has its own features that activate some cultural / linguistic models and suppress others in different ways for different examinees.(i.e., “method effects”)

Important for …•Alternative explanations•Variable performance

Important for …•Alternative explanations•Variable performance

(30)


on account of



unless


since





so

Claim about student


performance


Design Argument

(31)

Claim about student in use situation

Other information concerning student vis a

vis use situation

Warrant concerning use situation since

on account of


unless

Design Argument

Use Argument

Data concerning use situation


on account of



unless


since





so

Claim about student


performance


Backing concerning use situation

(Bachman)

(32)



vis use situation


on account of


unless

Design Argument

Use Argument



on account of



unless


since





so

Claim about student


performance



(33)



vis use situation


on account of


unless

Design Argument

Use Argument



on account of



unless


since





so

Claim about student


performance



(34)



vis use situation


on account of


unless

Design Argument

Use Argument



on account of



unless


since





so

Claim about student


performance



(35)



vis use situation


on account of


unless

Design Argument

Use Argument



on account of



unless


since





so

Claim about student


performance



Shared backing for test and use arguments grounds warrant for presumed appropriate activation in TLU context.

Shared backing for test and use arguments grounds warrant for presumed appropriate activation in TLU context.

What features do tasks and TLUs share?•Implicit in trait arguments •Explicit in interactionalist arguments

What features do tasks and TLUs share?•Implicit in trait arguments •Explicit in interactionalist arguments

This is the essence of warrant for claim in use argument.

This is the essence of warrant for claim in use argument.

(36)



vis use situation


on account of


unless

Design Argument

Use Argument



on account of



unless


since





so

Claim about student


performance



What features do tasks and TLUs not have in common?

What features do tasks and TLUs not have in common?

Questions of validity / generalizability:•TLU features that call for other cultural / linguistic models that weren’t in task and may or may not be in examinee’s resources.•Target models not activated in LTM in TLU context.

Questions of validity / generalizability:•TLU features that call for other cultural / linguistic models that weren’t in task and may or may not be in examinee’s resources.•Target models not activated in LTM in TLU context.

Knowing about target examinees and TLUs is key to strong inferences (Douglas, 1998)

Knowing about target examinees and TLUs is key to strong inferences (Douglas, 1998)

(37)

Implications for measurement models

Basic form: Probability of aspects of performance Xij given

parameters for person i and situation j

(all could be vector-valued)

jiij ,XProb

• Way too simple • No explicit connection with CI comprehension model, interaction processes, etc.• Apparent separation of person and situation characteristics

These are indeed properties of the conventional meaning of the measurement model and parameters.

These are indeed properties of the conventional meaning of the measurement model and parameters.

(38)

An Interactionalist Perspective: Instantiation in a Context

Xs result from particular persons calling upon resources in particular contexts (or not, or how)

Mechanically s simply accumulate info across situations

Our choosing situations and what to observe drives the situated meaning of s.

Situated meaning of s are tendencies toward these actions in these situations that call for certain interactional resources, via l/c models.

(39)

How to model inconsistent performance?

Traditional: Model as “noise” / unreliability Promising direction: Model individual’s degree or

pattern in variation in terms of context features If “motivated”: Model in terms of s

» Divide & Conquer: Multiple unidimensional tests (OET)

» Exploratory multidimensional models

» Controlled : Structured multidimensional models

» Critical importance of what else you know

Structured multidimensional

(40)

How much can testing gain from modern cognitive psychology?

So long as testing is viewed as something that takes place in a few hours, out of the context of instruction, and for the purpose of predicting a vaguely stated criterion,

then the gains to be made are minimal. Buzz Hunt (1986)

Conclusion

(41)

I have argued that we need to capitalize on [method effects] by designing tests for specific populations -- tests that contain instructions, content , genre, and language directed toward that population.

The goal is to produce tests … that would provide information interpretable as evidence of communicative competence in context.

Douglas (1998)

Conclusion

(42)

Conclusion

Interactionalist view of test theory… for arguments in interactionalist view of language to assemble, analyze, & interpret assessments in light

of context and purpose.

Methods and exemplars needed, but pressing need is

interpretive frame … » To connect view of language proficiency with the

machinery of test theory, » Toward modeling students’ (inter)actions in purposeful

variations in task contexts.

(43)

Thank you!

Documents

If Language is a Complex Adaptive System, What is Language Assessment?