If Language is a Complex Adaptive System, What is Language Assessment?

Presented at “Language as a Complex Adaptive System”, an invited conference celebrating the 60th Anniversary of Language Learning, at the University of Michigan, Ann Arbor, MI, November 7-9, 2008. The first author was supported by a grant from the Spencer Foundation.

Robert J. MislevyUniversity of Maryland

Chengbin YinCenter for Applied Linguistics

Key Ideas Assessment as evidentiary argument, not

simply as measurement. Arguments constructed around …

» View of the nature of proficiency.» Situations and ways people acquire it and

use it. Relevant work taking place in language

testing from an interactionist perspective. Reconceiving measurement models

The Assessment Argument(Messick, 1994)

What complex of knowledge, skills, or other attributes should be assessed?

What behaviors or performances should reveal those constructs?

What tasks or situations should elicit those behaviors?

We’ll look at a more technical representation in a little while.

Snow & Lohman, 1989

Summary test scores, and factors based on them, have often been though of as “signs” indicating the presence of underlying, latent traits. [] …

An alternative interpretation of test scores as samples of cognitive processes and contents … is equally justifiable and could be theoretically more useful.

LaCAS and Assessment Arguments

Interactionalist perspective in language testing:

» Communicative competence

» Contextual features of tasks

» Language tests for specific purposes

An Interactionalist Perspective (Young, 2000, 2008)

… language used in specific discursive practices rather than … language ability independent of context.

Focus on the co-construction of discursive practices by all participants ...

A set of general interactional resources that participants draw upon in specific ways in order to co-construct a discursive practice.

An Interactionalist Perspective (Young, 2000, 2008)

Relationship between participants’

employment of interactional resources and

the context in which they are employed.

Varying with the practice and the

participants…

Challenges for Assessment (Chalhoub-Deville, 2003)

Amending the construct of individual ability to accommodate [how] language use in a communicative event reflects dynamic discourse, which is co-constructed among participants; and …

reconciling [the notion that language ability is local] with the need for assessments to yield scores to generalize across contextual boundaries.

Sociocognitive Foundations

Themes from, e.g., cognitive psychology, literacy, neuroscience, anthropology: » Connectionist metaphor, associative memory

Situated cognition & information processing» Construction-Integration (CI) theory of

comprehension (Kintsch and others)

Individual Sociocultural perspectives» A cognitive theory of cultural meaning

(Strauss & Quinn, 1997)

A Cognitive Theory of Cultural Meaning “Interactional Resources”

Extrapersonal:» Cultural models: What ‘being sick’ means,

restaurant script, Newton’s laws, complaints» Linguistic: Grammar, conventions, constructions

Intrapersonal: » Patterns from experience at many levels» Schemas / frames / understandings / assumptions

Interplay Situated understandings » Access to, and ways of interacting with, shared

structures in order to accomplish goals

The user’s knowledge of the language

rules is interlocked with his knowledge

of when, where, and with whom to use

them. (R. Ellis, 1985)

B Inside BInside A A

observablenot observable not observable

B Inside BInside AContext

A la Kintsch: Propositional content of text / speech…

and internal and external aspects of context …

The C in CI theory, Construction:Activation of both relevant and irrelevantbits of cultural models, experiences, e.g.,

•Restaurant script, Human motivation Guided in part by linguistic models, e.g.

•Conventions, constructions, rhetorical framesContent of utteranceHistory with interlocutorConversation thus far

The C in CI theory, Construction:Activation of both relevant and irrelevantbits of cultural models, experiences, e.g.,

•Restaurant script, Human motivation Guided in part by linguistic models, e.g.

•Conventions, constructions, rhetorical framesContent of utteranceHistory with interlocutorConversation thus far

E.g., tasks in Occupational English Test (OET; McNamara, 1996) call upon patterns re language, but also genre, medical knowledge, use of information in clinical settings.

•If a pattern hasn’t been developed in past experience, it can’t be activated (although it may get constructed in the interaction).•A relevant pattern from LTM may be activated in some contexts but not others (e.g., physics models; question formation (Tarone)).

The I in CI theory, Integration:•Situation model: synthesis of coherent / reinforced activated cultural / linguistic / situational patterns •Situation model is basis of understanding

Situation model is also the basis of planning and action.

Context

Previous situation models are input to subsequent situation models.

Context

Ideally, activation of relevant and

compatible cultural & linguistic models…

Ideally, activation of relevant and

compatible cultural & linguistic models…

Context

to lead to (sufficiently) shared understanding;

i.e., co-constructed meaning.

to lead to (sufficiently) shared understanding;

i.e., co-constructed meaning.

Kramsch’s "shared internal context "

Comments about context…A B Inside BInside A

Context

Can distinguish external and internal aspects of context (e.g., Douglas, 2000)

External / pubic aspects of context, e.g.,

•Setting•Physical attributes

External / pubic aspects of context, e.g.,

•Setting•Physical attributesRe assessment,

Target language use (TLU) featuresTask features

Re assessment,Target language use (TLU) featuresTask features

A B Inside BInside AContext

Context

Aspects of cultural/linguistic/interaction context as interpreted by an external observer.

Used to determine what actions signal recognition, comprehension, action through targeted cultural /linguistic models.

As such, in assessment, plays role in•Evaluation, hence •Observable variables

What can we say about individuals?

Use of resources in appropriate contexts in appropriate ways; i.e.,

Attunement to cultural/linguistic patterns: Recognize markers of externally-viewed patterns? Construct internal meanings in their light? Act in ways appropriate to targeted cultural/linguistic

models? What is the range and circumstances of activation?

(variation of performance across contexts)

The Assessment Argument(Messick, 1994)

What complex of knowledge, skills, or other attributes should be assessed?

What behaviors or performances should reveal those constructs?

What tasks or situations should elicit those behaviors?

Toulmin’s Argument

Backing

unless

sinceWarrant

Alternativeexplanationso

Structure

Student acting inassessment situation

on account of

Backing concerning assessment situation

Alternative explanations

unless

Warrantconcerning assessment

Warrant concerning evaluation since

Warrant concerning task design since

Other information concerning student vis a vis

assessment situation

Claim about student

Data concerning student

performance

Data concerning task situation

on account of

unless

Claim about student

performance

In interactive task,•performance flows in time, •performance changes situation, •may or may not be series of task and observable variables

on account of

unless

Claim about student

performance

Concerns features of (possibly evolving) context as seen from the view of the assessor – in particular, those seen as relevant to targets of inference.

on account of

unless

Claim about student

performance

Evaluation of performance concerns context features indirectly: clues that suggest attunement to features of cultural / linguistic models of interest. (did examinee recognize, comprehend, act accordingly?)

on account of

unless

Claim about student

performance

“Hidden” aspects of context—not in test theory model but essential to argument: What attunements to cultural / linguistic models can be presumed among examinees, to condition inference re targeted l/c models?

Fundamental to situated meaning of student variables in measurement models;Both critical and implicit.

on account of

unless

Claim about student

performance

Assessment context always has its own features that activate some cultural / linguistic models and suppress others in different ways for different examinees.(i.e., “method effects”)

Important for …•Alternative explanations•Variable performance

on account of

unless

Claim about student

performance

Design Argument

Claim about student in use situation

Other information concerning student vis a

vis use situation

Warrant concerning use situation since

on account of

unless

Design Argument

Use Argument

Data concerning use situation

on account of

unless

Claim about student

performance

Backing concerning use situation

(Bachman)

vis use situation

on account of

unless

Design Argument

Use Argument

on account of

unless

Claim about student

performance

vis use situation

on account of

unless

Design Argument

Use Argument

on account of

unless

Claim about student

performance

vis use situation

on account of

unless

Design Argument

Use Argument

on account of

unless

Claim about student

performance

vis use situation

on account of

unless

Design Argument

Use Argument

on account of

unless

Claim about student

performance

Shared backing for test and use arguments grounds warrant for presumed appropriate activation in TLU context.

What features do tasks and TLUs share?•Implicit in trait arguments •Explicit in interactionalist arguments

This is the essence of warrant for claim in use argument.

vis use situation

on account of

unless

Design Argument

Use Argument

on account of

unless

Claim about student

performance

What features do tasks and TLUs not have in common?

Questions of validity / generalizability:•TLU features that call for other cultural / linguistic models that weren’t in task and may or may not be in examinee’s resources.•Target models not activated in LTM in TLU context.

Knowing about target examinees and TLUs is key to strong inferences (Douglas, 1998)

Implications for measurement models

Basic form: Probability of aspects of performance Xij given

parameters for person i and situation j

(all could be vector-valued)

jiij ,XProb

• Way too simple • No explicit connection with CI comprehension model, interaction processes, etc.• Apparent separation of person and situation characteristics

These are indeed properties of the conventional meaning of the measurement model and parameters.

An Interactionalist Perspective: Instantiation in a Context

Xs result from particular persons calling upon resources in particular contexts (or not, or how)

Mechanically s simply accumulate info across situations

Our choosing situations and what to observe drives the situated meaning of s.

Situated meaning of s are tendencies toward these actions in these situations that call for certain interactional resources, via l/c models.

How to model inconsistent performance?

Traditional: Model as “noise” / unreliability Promising direction: Model individual’s degree or

pattern in variation in terms of context features If “motivated”: Model in terms of s

» Divide & Conquer: Multiple unidimensional tests (OET)

» Exploratory multidimensional models

» Controlled : Structured multidimensional models

» Critical importance of what else you know

Structured multidimensional

How much can testing gain from modern cognitive psychology?

So long as testing is viewed as something that takes place in a few hours, out of the context of instruction, and for the purpose of predicting a vaguely stated criterion,

then the gains to be made are minimal. Buzz Hunt (1986)

Conclusion

I have argued that we need to capitalize on [method effects] by designing tests for specific populations -- tests that contain instructions, content , genre, and language directed toward that population.

The goal is to produce tests … that would provide information interpretable as evidence of communicative competence in context.

Douglas (1998)

Conclusion

Interactionalist view of test theory… for arguments in interactionalist view of language to assemble, analyze, & interpret assessments in light

of context and purpose.

Methods and exemplars needed, but pressing need is

interpretive frame … » To connect view of language proficiency with the

machinery of test theory, » Toward modeling students’ (inter)actions in purposeful

variations in task contexts.

Thank you!

If Language is a Complex Adaptive System, What is Language Assessment?

Documents

Running title: Complex adaptive systems

Phoenix: Complex Adaptive System of Systems (CASoS ... · Complex Adaptive Systems of Systems, or CASoS, are vastly complex ecological, sociological, ... Climate and Infrastructure

Promoting the Use of Complex Innovative Designs in …€¢ Overview of adaptive clinical trial designs • Considerations for adaptive designs (including complex adaptive designs)

Complex Adaptive Systems and International Security Analysis

Architectural Applications of Complex Adaptive Systemspapers.cumincad.org/data/works/att/acadia03_042.content.09646.pdf · tn02-050: ARCHITECTURAL APPLICATIONS OF COMPLEX ADAPTIVE

Ants Complex Adaptive Systems, CS 591 Ants Complex Adaptive Systems, CS 591 April 16, 2008 Prof. Melanie Moses

Modeling molecular motion: complex adaptive thermodynamic

Complex Adaptive Systems Conceptualizing the Problem · Complex Adaptive Systems ... problems that require intervention – Development – Security – Economic stability – Climate

RESEARCH Open Access Complex adaptive systems …

COMPLEX ADAPTIVE SYSTEMS J. Stephen Lansing

Distributed Networked Forces Complex Adaptive Environments Distributed Networked Forces operating in Complex Adaptive Environments CDR Gregory Glaros,

Complex Adaptive Systems in Health

Complex Adaptive Systems, Evolutionism, and Ecology within

Complex Adaptive Systems

Lansing Complex Adaptive Systems

Complex Valued Nonlinear Adaptive Filters

Healthcare teams as complex adaptive systems

A Complex Adaptive Systems Model of Organization … Complex Adaptive Systems Model of ... A Complex Adaptive Systems Model of Organization Change ... Contingency theory claims that

Fres 1010: Complex Adaptive Systems

Complex Adaptive IT EA