Robert J. Mislevy University of Maryland National Center for Research on Evaluation, Standards, and...

Robert J. Mislevy

University of MarylandNational Center for Research on Evaluation,Standards, and Student Testing (CRESST)

NCME San Diego, CAApril 15, 2004

Some Observations on Cognitive Psychology and Educational Assessment

Outline of the talk

Themes from cog psych

How cog psych informs what we assess and how we might assess it (esp. school & work)

How cog psych helps us understand and organize what we do in assessment

Themes

Capabilities & limitations

Reasoning in terms of patterns

Psychological perspectives

Acquiring expertise

Forms of knowledge representation

Capabilities & limitations

Ways we are the same / differing / unique

Experiential & reflective cognition

• Optical illusions / cognitive illusions

• Limited working memory & attention

• Can think about our thinking (metacognition)

• Benefit from procedures, methods, tools

An Optical Illusion (http://www.optillusions.com)

Reasoning in terms of patterns

Perception combines input from environment and patterns from experience

• Chi, Feltovich, Glaser example

Narratives / schemas / scripts / mental models

• This is how we make sense of the world

• Some “wired in”

• Some learned informally and experientially

• Some through instruction and conscious effort

What is this a picture of? (http://www.optillusions.com)

Reasoning in terms of patterns

Simultaneous use of patterns at many levels

Perception / Meaning /Action

Key role of interacting with situation

• Inquiry cycle / model-based reasoning

• Interactive tasks (construction, simulation)

• Even in static tasks, focus on perception / explanation / action

Reasoning in terms of patternsAssessment as Evidentiary Argument

What complex of knowledge, skills, or other attributes should be assessed … ?

What behaviors or performances should reveal those constructs [broadly construed]?

What tasks or situations should elicit those behaviors?

(Messick, 1994)

Psychological perspectives

Trait/Differential (Spearman, Carroll)

• Origin of machinery of psychometrics

Behaviorist (e.g., CRTs of 1970s)

Developmental (Piaget)

Information-processing (Newell & Simon)

Sociocultural/situative (Vygotsky, Lave)(Greeno, Pearson, and Schoenfeld (1996)

Psychological perspectives

A perspective shapes…

• what you pay attention to;

• what entities and relationships you use in explanations;

• what you see as problems and solutions.

A perspective both enables and constrains thinking.

Psychological perspectives

For assessment, perspective shapes…

• Inferences you target – patterns that shape students’ actions (Meaning & Action)

• What you look for in what students say, do, or make (Perception)

• What are the features of the situation that evoke the evidence you need.

A perspective both enables and constrains what you can learn from an assessment.

What will Jimmie’s path be if he steps off the merry-go-round right now?

Psychological perspectives

Hydrive

• Info-processing + sociocultural

AP Studio Art

• Sociocultural; interpretational

• Note interpretation of variables in model

Task-based language assessment

• All perspectives relevant

• Target language use (Bachman & Palmer)

• What to stress, how to design situations

Acquiring expertise

Expertise as overcoming human cognitive processing limitations

• Patterns for perceiving, understanding, acting (incl. sociocultural)

• Use of knowledge representations

• Automating processes to varying degrees

• Metacognitive skills

Acquiring expertise

Examples in assessment

• Katz, re NCARB simulations as example for “design under constraint” (assessment is another such domain!)

• Embretson as example for differential perspective measurement

• Marshall & Derry as example for assessment design based on schemata

• Stevens re ordered pairs of actions

Forms of knowledge representation

Symbol sets & manipulation

Forms of knowledge representation (KRs)

• Maps, diagrams, object models, flow charts

Central to expertise

Mediated cognition

Distributed cognition

Nexus between info-processing & sociocultural perspectives

Forms of knowledge representation

Some forms of knowledge representation for design and using assessments:

Measurement models & representations

Argument structures

Evidence-centered design structures

Design patterns, templates, object models

IMS/QTI standards

Three basic models that embody theassessment argument

Forms of knowledge representation

Forms of knowledge representation

Measurement models: Multivariate models for different aspects of

knowledge / skill / propensities (MRCMLM)

Integration of statistical inference with task design (Tatsuoka, Embretson): Cognitive diagnosis, mixed strategies, multilevel models

Conditional dependence (re interaction)

Re-interpretation of variables (propensities to act in situations w x features; rater models)

Example: HYDRIVE

Student-model variables in HYDRIVE

• Motivated by cognitive task analysis

• Scope shaped by purpose

• Grain-size determined by instructional options

A Bayes net fragment

Overall Proficiency

Procedural Knowledge

PowerSystem

SystemKnowledge

Strategic Knowledge

Use ofGauges

SpaceSplitting

Electrical Tests

SerialElimination

Landing GearKnowledge

Canopy Knowledge

ElectronicsKnowledge

HydraulicsKnowledge

Mechanical Knowledge

HYDRIVE, continued

A Bayes Net Measurement Model, docked with Student Model

Canopy Situation--No split possible

Canopy Situation--No split possible

Use ofGauges

SerialElimination

Canopy Knowledge

HydraulicsKnowledge

Mechanical Knowledge

Library of Measurement Model fragments

Conclusion

Assessment is a particular kind of narrative:

• An evidentiary argument about aspects of what students know and can do, based on a handful of particular things that have said, done, or made.

Assessment integrates perceiving, understanding, and acting.

Assessment forms both enable and constrain thinking about students.

Conclusion

Cognitive psychology helps us understand what to make inferences about, what we need to see, what situations can provide us with clues.

• Conceiving targets of assessment

• Explicating and improving the design and use of assessments