Situation Models and Embodied Language Processes

Franz Schmalhofer

University of Osnabrück / Germany

1) Memory and Situation Models

2) Computational Modeling of Inferences

3) What Memory and Language are for

4) Neural Correlates

5) Integration of Behavioral Experiments and Neural Correlates (ERP; fMRI) by Formal Models

Text comprehension

• Mary heard the ice-cream van coming.

• She remembered the pocket money.

• She rushed into the house.

Kintsch, Welsch, Schmalhofer & Zimny (1990)

How the strengths of the different memory representations can be empirically determined

Original/Explicit:

Mary heard the ice cream van coming.

Paraphrase:

Mary noticed the ice cream van coming.

Inference:Mary picked up her pocket money.

False/Incorrect:

Mary was 50 years of age.

Propositional Representations

Example: The propositional representation of the sentence George loves Sally

[LOVES(GEORGE,SALLY)]

Many cognitive science theories assume that knowledge and/or the meaning of sentences is represented by propositions, semantic nets and the like (e.g. Anderson, 1976; Kintsch, 1974; Collins & Quillian, 1969; Schank 1975, Schank & Abelson, 1978)

Compare to: Two word sentences of children during language learning;Protolanguage (Bickerton, 1981, 1995)

The difference between pictures and perceptions

Situation models are formed by perceptual symbols

Comprehension

Comprehension includes a large range of topics in cognitive psychology:

• pattern recognition,

• knowledge representations,

• Working memory,

• Recognition and recall,

• learning, problem solving and decision making

Kintsch, W. (1998). Comprehension as a Paradigm for Cognition

The Construction-Integration Model (Kintsch 1992)

Comprehension: a two phase process

Construction:Constructing mental units and interconnecting them in a network

Integration:Integration of constructed units via a context sensitive process

The Construction-Integration Model (Kintsch 1992)

Text Comprehension

Up to the 1980`s language comprehension was mostly viewed as the representation of the meaning of the text itself (focus on propositional representations)

Now language is viewed as a set of processing instructions (Zwaan, 2004) on how to construct a mental representation of the described situation (mental model or situation model) (Johnson-Laird, 1983; van Dijk & Kintsch, 1983)

Situation Models as Event Indices (Zwaan and Radvansky, 1998):

The Event-indexing model of Zwaan & Radvansky (1998) suggests that readers monitor five indexes (aspects) of the evolving situation model at the time when they read stories:

- Protagonist- Temporality- Causality- Spatiality- Intentionality

Or more generally: space, time, causes, agents, intentions;In other words: everything that is relevant for planning

actions and predicting future perceptions

How do people acquire knowledge from different materials?

• Text– general, in a natural language– relatively short– sentence may describe a single

attribute of a concept

• The function first requires one argument. The argument of the function first must be a list. The function first returns the first element of the argument.

• Text of LISP function description

• Examples– specific,

– possibly a large set of examples required

– exemplification of many attribute instances

• (FIRST `(A B)) A(FIRST `((A) B)) (A)(FIRST `(A (B C))) A(FIRST `(A)) A

Informational Equivalence of different learning materials

The Function First

Properties of

A unifying model (KIWi-Model)Schmalhofer (1998)

related domain knowledge

situation model

Sensory encodingtext repres.

common sense

direct experiencetext

Subject group

Diff

eren

ce o

f r

egre

ssiv

e e

ye-m

ovem

ents

0

0,2

0,4

0,6

0,8

1

1,2

1,4

1,6

Freshmen Non-

Programmers

Programmers LISP Group

Regressive eyemovements during reading

Reading times for studying examples

new partially new redundant negative

-5

0

5

10

novices (ex. before text)

comp. users (ex. before text)

novices (text before ex.)

comp. users (text before ex.)

type of examples

readin

g t

ime r

esid

ual in

seconds

new partially new redundant negative

-5

0

5

10

novices

computer users

type of example

readin

g t

ime r

esid

ual in

seconds

Experiment 8 (examples only)

151050

-2

0

2

4

6

8 situation model

mean total processing time [in sec.]

151050

-2

0

2

4

6

8 text representation

mean total processing time [in sec.]3020100

-2

0

2

4

6

8 template base

mean total processing time [in sec.]

Text novices

Text computer users

Example novices

Example computer users

•text novices

•text experts

•example novices

•example experts

Memory retrieval after learningfrom text or examples

Correct Responses in Example Verification Task as a Function of Different Amounts of Text

1210864200,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1,0

Processing Time (sec)

Rel

ativ

e Fr

eque

ncy

of C

orre

ct R

espo

nses

Complete TextPartial TextNo TextBaseline Control

Summary of the tested predictions of KIWi-Model

• KA is a goal-driven process, consisting of construction and integration phases

• Text and examples can be equated for informational contents

• The material-related representations are constructed by general heuristics, the situation model depends on domain knowledge

• Experts construct and use deep knowledge (situation model), novices rely on material-related representations

• Integrative KA (perception, language and memory) instead of the dominance of one source

Research on

Text Comprehension(learning from text)

• memory issues

• encoding processes

• retrieval

• representational issues

Concept Formation(learning from examples)

• search processes

• hypothesis formation

• concept identification

and

Two alternative ways of acquiring knowledge

“memory paradigm“ ”Problem solving paradigm“

Comparison to Other Cognitive Models

Cases Unclassified Examples

Classified Examples

Examples Examples and Theoriy

GOAL (Input)

Knowledge

Skill

Goal (Output)

Systems: CHEF JUDGE

DA-PRODIGY PUPS SME

COBWEB CL

BP INDUCE ID3 VS

ACT-R/KnCo BACON SOAR

CASCADEEBG GENESIS

TR

Analogy

Solutions

Hypothesis Space

Search Explanation

HeuristicsDomain Knowledge

Rules Schema(revised)

Theory

Domain Knowledge

Heuristics

Case-based Search-based Comprehension-based

Theory

Importance of Unified Theories

• Consistent and consensual theories drive cumulative progress

• The chronology of research produces a need for readjustment of the mappings between theoretical constructs and empirical data

– Natural Texts versus Textoids– Changes in Task Types (Priming, Reading time, Memory)– Changes in Available Methods (lateralized presentation,fMRI,

ERP)

• Possibility that some controversies can be resolved by synthesis into a unified theory

– For Example: Predictive and Bridging Inferences

Types of inferences (Graesser et al., 1994)

• “Mary heard the ice-cream van coming.She remembered the pocket money.She rushed into the house.”

• What types of inferences are there and when are they drawn?

• Referential

• Case structure assignment(role: e.g. agent)

• Causal antecendent

• Superordinate goal

• Thematic

• Character emotional reaction

• Causal consequence

• Bridging

• Predictive

Landscape of Inferences (from Graesser, 2003; HC)

TYPES OF INFERENCES1. Referential2. Case structure role assignment3. Instantiation of a noun category4. Superordinate goal5. Superordinate goal or action6. Instrument

7. Causal antecedent

8. Causal consequence

9. Character emotional reaction10. Emotion of reader11. State12. Themes13. Author‘s intent

Causal consequence: The inference is on a forecasted causal chain, including physical events, psychological events, and new goals, plans, and actions of agents.

Causal antecedent: The inference is on a causal chain that bridges the current explicit action, event, or state to the previous passage context.

Construction and persistence of predictive and bridging inferences

(e.g. McKoon & Ratcliff, 1986; Potts et al., 1988; Keefe & McDaniel (1993)

The director and the cameraman were preparing to shoot closeups of the actress on the edge of the roof of the 14th story building

explicit

predictive inferencing

cycle 1 cycle 2 cycle 3

bridging inferencing

when suddenly the actress fell and was pronounced dead

+The director was talking to the cameraman and did not see what happened

+

when suddenly the actress fell

+

The director was talking to the cameraman and did not see what happened

-

Her orphaned daughters sued the director and the studio for negligence.

+

Overarching theoretical assumptions

• Kintsch‘s (1998) C-I theory– Multi-level representation:

surface-level, propositional text representation and situation model

– Processing cycles with construction-integration phases

• Enhancing assumptions– Situation models are built from

perceptual symbols (Zwaan et al. 2001); they often build a visuo-spatial representation (Fincher-Kiefer, 2002)

• Instead of a nominal distinction of inference types, like predictive, bridging, causal etc. inference,a functional description of cognitive processes

• Similar to object constancy in visual perception, a situation constancy is postulated in the formation of situation models

• Inferencing achieves this situation constancy, i.e. inferencing as a pattern completion process

The 2nd processing cycle for the explicit and predictive conditions

Model predictions of the inference encoding scores for Keefe & McDaniel data

MODEL

DATA

Input Cycle 2 3 2 3

______________________________________________________

Explicit 33 32 30* 33*

Predictive Inference 24 6 35* 4

Bridging Inference 22 22*

______________________________________________________

Evaluation of experimental predictions

• Experiment: 2 (instructions) x 4 (text) mixed factorial design

– 1) situation condition: elaborate on the context described in the passage

– 2) concentrate on the precise wording of the sentences

• Textmaterials from McKoon & Ratcliff (1986), Potts et al.

(1988), ….

• Latencies in word pronunciation task

• Sentence recognition task – Pr(yes=old) as dependent measure

Reaction time in ms in Pronunciation Task (to inference targets)

Predictive Bridging Explicit Control

Situation 569 556 571 614

Word 595 576 550 609  

Text- and situation focused reading (3-rd processing cycle)

Model Data

Reading Focus text situation text situation

__________________________________________________________________

Condition

Explicit 23 63 59* 43*

Predictive inference 5 61 14 45*

Bridging inference 21 41 33* 58*

_________________________________________________________________

Sentence Recognition Task

Explicit probe

• The cameraman was preparing to shoot closeups.

Inference probe

• The actress was pronounced dead.

Elaboration probe

• The actress died from her injuries.

Inconsistent probe

• The actress lived a long life.

Pr („explicitly mentioned“) in sentence recognition task

Text Types Predictive Bridging Explicit Control _________________________________________________________________________________Reading focus text situation text situation text situation text situation_____________________________________________________________________________ Explicit .72 .82 .75 .94 .77 .86 .51 .52 Inference .22 .29 .27 .36 .93 .87 .15 .11 Elaboration .24 .33 .26 .42 .23 .38 .15 .16 Inconsistent .11 .12 .14 .05 .11 .04 .12 .11

Strength of situational representation for the critical consequence as d‘-values (from elaboration and inconsistent statements)

Reading Focus on

Predictive

Text

Bridging

Text

Explicit

Text

Situation 0.80 1.48 1.39

Text O.54 0.48 0.53

Griesel, Friese & Schmalhofer (2003)

Verification

seconds

1197531

rel.

freq

uenc

y of

"ye

s"-r

espo

nses

1,0

,9

,8

,7

,6

,5

,4

,3

,2

,1

0,0

explicit

paraphrase

bridging

predictive

control

Recognition

seconds

1197531

rel.

freq

uenc

y of

"ye

s"-r

espo

nses

,8

,7

,6

,5

,4

,3

,2

,1

0,0

explicit

paraphrase

bridging

predictive

control

Modeling predictive and bridging inferences in comparison to explicit statements

• Differences in interconnectivity are the key:– High interconnectivity at situation level (predictive) compared to– High interconnectivity at the propositional level (explicit)

Schmalhofer, McDaniel, and Keefe (2002)

• Focus on situation and time course may even keep predictive inferences activated in a later processing cycle (this was also the model prediction)

McDaniel, Schmalhofer, and Keefe (2001)

Application to Beeman, Bowden and Gernsbacher (2000) data

• Differential contribution of LH an RH in inference generationLH fine semantic coding

Strong activation of small semantic fieldsRH coarse semantic coding

Weak activation of large semantic fields

• Mapping to CI modelLH is verbatim and propositional RH is situation

• Both hemis process in parallel with sharing at critical times

• Activation of hemis assessed at predictive and bridging inference points

Experiment of Beeman et al. (2000)

Bob took his daughter Karen out of school for the day so she could enjoy a very historic event that would take place that morning. The shuttle sat on the ground in the distance, (1)

waiting for the signal to be given (2).

predictiveLH -RH +

After a huge roar (3).

and a bright flash, the shuttle disappeared into space (4).

Leaving clouds of smoke in the wake (5), and the audience cheered.

predictiveLH -RH+

bridging LH +RH +

Beeman, Bowden and Gernsbacher (2000)

surfa c e p ro p o sitiona l situa tiona l

Relative frequency of “yes”-responses in the verification task for left visual and right visual field presentations (Griesel et al., 2003)

LVF/RH RVF/LH

explicit .92 .93

bridging .89 .90

predictive .78 .79

control .20 .17

Mean latencies in ms of the “yes”-responses in the verification task

for left visual and right visual field presentations

Mean latencies

RVF/LHLVF/RH

ms

1100

1000

900

800

explicit

bridging

predictive

Summary

• Experimentation for differentiation; Theorizing for integration;

• Theories of text comprehension can be instantiated to simulate data from multiple experiments in detail– Systematic relation of dependent and independent variables

to the different conceptual entities in models

• Integration of existing data and theories is exciting, especially in view of ERP and new brain imaging data, related to inferencing