Bilateral semantic processing: Inferences in language ...cseweb.ucsd.edu/~gary/cs200/f06/slides/Jung-Beeman-cogsci200-11-20-2006.pdfBilateral semantic processing: Inferences in language,

Bilateral semantic processing:

Inferences in language, insight in problem solving

Mark Jung-Beeman

Northwestern UniversityDepartment of Psychology

Neuroscience Institute

Cognitive Brain Mapping Group

\



Northwestern University Drexel UniversityZoe Clancy John Kounios

Jason Haberman (UCDavis) Debbie Green

Sandra Virtue (Depaul U) Jennifer Frymiare (U Wisc)

Stella Arambel (deceased) Jessica Fleck

Dianne Patterson Richard Greenblatt

Todd Parrish

Paul Reber Bar-Ilan UniversityTerri Swan Miriam Faust

Karuna Subramaniam Nira Mashal

Ed Bowden

Research sponsored by NIDCD/NIH

OUTLINE:

• Drawing inferences from stories -- bilateral comprehension

• Three bilateral component semantic processes (to start)

• Insight -- bilateral, parallel processing during problem solving



Bilateral Activation, Integration, and

Selection model of semantic processing

• Semantic activation - “Wernicke’s area”

– Bottom-up lexical-semantic activation: index of semantic

representations (pMTG)

• Semantic integration - anterior Sup. Temp. Gyrus

– Compute semantic overlap - detect or generate (aSTG)

• Semantic selection - Inf. Frontal Gyrus

– Select among competing activated concepts (IFG)

OUTLINE:






Problems with view that language is purely a LH function

• General anatomical symmetry

• RH damaged patients - some language problems

• Recovery from aphasia, hemispherectomy, callosotomy

• Neuroimaging - always some RH signal, some tasks RH>LH

• Some tasks lvf-RH better than rvf-LH

Natural language, stories, discourse

• Higher level semantic processing (plus all lower levels)

As language input more complex (and natural):

• More anterior temporal lobes • More bilateral processing

Brain bases of comprehensionof natural language

Causal bridging (coherence) inferences

“Before going to the wedding, John was sitting around in his

jeans, so he went to his bedroom to find some clothes.”

Brain bases of cognitive processes whenpeople draw inferences from stories



jeans, so he went to his bedroom to find some clothes.

He came out wearing his tuxedo, which had belonged to John's

father, but looked like new.”




jeans, so he went to his bedroom to find some clothes.

He came out wearing his tuxedo, which had belonged to John's

father, but looked like new.”

CHANGECHANGE


We know people make such causal inferences

We know a lot about other types of inferences that people

make - types of text, motivation, knowledge, capacity

We still don’t know much about component processes

that support this seemingly complex behavior


RHD patients have difficulty drawing inferences

• Answer questions about inferable events less accurately than

control subjects; intact on explicitly stated facts(Brownell et al., 1986; Beeman, 1993)

• Do not show inference-related priming; control subjects do(Beeman, 1993)

RH semantic processing and inferences

Proposed component processes of

inference generation

• 1) Activation / integration (detect overlap)

• 2) Selection

• 3) Incorporation / integration (map overlap)

• Hemispheric cooperation

• RH activates information that may support inferences.

Weak activation not reach consciousness.

Time course of inference related semantic activation

in both hemispheres during story comprehension.

“Before going to the wedding,, John was sitting around in

his jeans,1 so he went to his bedroom to find some

clothes.22 After a few minutes,tes,3 3 he came out wearing

his tuxedo,44 which had belonged to John's father55, but

was still fashionable and looked like new.”

- CHANGE- CHANGE

• (1) and (2): Predictive inference.

• (3): Transition.

• (4): Coherence or bridging inference.

• (5): Resolved and incorporated.

Right visual field

Left Hemisphere Right Hemisphere

Left visual field

“Before going to the wedding, John was sitting around in his jeans,1 so he

went to his bedroom to find some clothes.2 After a few minutes,3 he came out

wearing his tuxedo,4 which had belonged to John's father5, but was still

fashionable and looked like new.” Brain and Language, 2000

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5

Test point

rvf-LH

lvf-RHPriming:

Inference

faster than

Unrel

Asymmetric dynamic semantic fields:relatively coarser coding in RH; better selection in LH

footfoot

CUT

TOES

RULER

Right HemisphereLeft Hemisphere

Small but strongly activated;

Focused on dominant or

contextually relevant concepts

- easy to select, interpret, output

Large but weakly activated;

Diffuse, including secondary

and less relevant concepts

- hard to select, output

foot

painglassglass pain

foot

RH coarse semantic coding:Increased likelihood of semantic overlap

for distant semantic relations










RH Middle & superior temporal gyrus

involved in computing semantic integration

• Deriving theme from paragraphs (St. George et al.)

• Generating best ending (Kirchner et al.)

• Generating inferences? - moderately related sentence pairs

(Mason & Just)

• Metaphoric over literal sentences (Bottini et al.)

• Detecting temporal/emotional inconsistency (Ferstl)

• Generating insight solutions (Jung-Beeman et al; Kounios et al)

Brain activity when people draw inferences

on-line, as indexed by fMRI

Three ways to contrast inference versus no-inference conditions:

- Text: infernce versus no-inference; strong vs. weak constraint

- Individual differences: high versus low Working Memory

- Behavioral measures: recall of inferences

General Results:

Bilateral activity in pMTG; aSTG; IFG

- modulated by constraint, WM, time

Brain activity when people draw inferences

on-line, as indexed by fMRI

Inference:

… John was going to a wedding, but he had been sitting aroundthe house in his jeans, so he went to his bedroom to find some

clothes. Soon he came out wearing his tuxedo, * …

Explicit:

…went to his bedroom to change his clothes. Soon he came out

wearing his tuxedo ,* …

- High baseline, ongoing stories; small input difference

Semantic integration at moment of implied events:

Predominantly RH aSTG

L R Post Ant L R

Semantic integration at event point:

Bilateral anterior Superior Temporal Gyrus

L R

Lower (ns) threshold, selected for LH STG

Semantic activation and integration at coherence break (“tuxedo”):

Predominantly LH STG

Semantic selection:

High versus low working memory

High WM (reading span) subs show stronger, earlierevidence of semantic selection of inferences

(St. George et al; many behavioral)

• Completion requires selection, incorporation

Semantic selection:

Inferior frontal gyrus

Selecting some concepts over competitors

• Usually IFG in LH (Thompson-Schill et al; Barch; Friston)

Some instances, RH IFG

• (Seger 2000; Friederici et al., 2000; Jung-Beeman et al.)

Semantic selection:

Inferior frontal gyrus

Selecting some concepts over competitors

• Usually IFG in LH (Thompson-Schill et al; Barch; Friston)

Some instances, RH IFG

• Unusual verb generation (cake -> “decorate”) (Seger 2000)

• Repair grammatical errors (Friederici et al., 2000)

• Utilize unintended meaning of ambiguous words in sentence– (Jung-Beeman et al.)

Semantic selection: fMRI signal in IFG (LH > RH) at

coherence break in High WM subs only

(Fig: High WM > Low WM)

Replication and extension:

Working memory and predictability

Unpredictable inferences: LH activation, IFG, pSTG

• searching for connections

Predictable inferences: Bilateral activation, IFG, pSTG

• building on connections

Higher WM (n=13) > lower WM (n=13):


• facile comprehension

RH

pSTG

Successful integration versus continued activation:

STG in High vs. Low WM subs at coherence break,

Predictable inferences

RH

IFG

High WM subs

show bilateral

(stronger in RH)

Low WM show

LH only

p<.001







Higher WM (n=13) > lower WM (n=13):

RH activation, pSTG, IFG, and a little aSTG



Successful integration versus continued activation:

STG in High vs. Low WM subs at coherence break,

Predictable inferences

RH

aSTG

High WM subs

show bilateral

(stronger in RH)

Low WM show

LH only, no aSTG

p<.005







Higher WM (n=13) > lower WM (n=13): RH activation, pSTG, IFG, and alittle aSTG



Conclusions about inferences

• Semantic integration builds up as story hints thatsome event might occur: anterior STG; RH (?)

• At coherence break: integration and activation(STG), especially in LH

• completing the inference requires selection (IFG)

• RH contributes to facile inferencing/comprehension,not just kick in when demands are high

Current projects, Future directions

• Shift semantic distance for integration --> shift hemiasymmetry

• Closely tie to behavioral markers of inference activation,selection, incorporation

– Recall of inferences !

– Priming of inferences

• Successful integration versus effort of difficult integration

– Incorporation (recall study)

Recalled inferences

• If inferences recalled, must have been incorporated

• Working Memory correlates with

– total recall

– Recall of inferences

– NOT with recall of episodes w/o inferences

• Contrast fMRI signal of recalled infs versus recall episode,no infs

L R Post Ant L R

Inferences recalled versus

Episode recalled, inf not recalled

L R R R

p<.005 , positive only

Bilateral pMTG, stronger in RH RH aSTS, bilat IFG

So what?

Knowing where processing occurs informs and

constrains what and how it occurs

OUTLINE:















Why does the RH code more coarsely?

Asymmetries in neural microcircuitry

Given topographic mapping of brain,

broader input/output fields => coarser semantic coding

footfoot

CUT

TOES

RULER

Right HemisphereLeft Hemisphere

Small but strongly activated;

Focused on dominant or

contextually relevant concepts

Large but weakly activated;

Diffuse, including secondary

and less relevant concepts

foot

painglassglass pain

foot

RH coarse semantic coding:Increased likelihood of semantic overlap

for distant semantic relations

Why a separate area for semantic integration?

• Could form associations in “activation” area

BUT

• Higher level relations, correlated co-occurrence, indirect

• Ability to extract, attend to, & manipulate relations

– Analogous to individual areas within vision (e.g., motion)

Why anterior STS/STG for semantic integration?

• Again, neural architecture

L R Post Ant L R

Patchy organization and multisensory integration(Beauchamp 2004)

Why anterior STS/STG for semantic integration?

• Again, neural architecture

• More anterior = longer intrinsic conxns, better to

integrate across patches

• RH = longer than LH

Important clarifications

• Not an “inference area”

– Semantic integration - participates in many functions

– Not specific to categories of inferences - varies with demand

• Tight comparison not reveal whole network

– Just areas that differ when storied imply versus explicitly state events

• RH and LH cooperate

OUTLINE:



• Insight -- bilateral, parallel processing in problem solving



Most problems solved with mix of analytic and insight

processing

• Distinct computations, distributed across hemispheres, allows

two approaches to proceed simultaneously (partially

interactive)

• Hemispheric components, task shielding/switching

Brain bases of insight during problem solving:Aha! and antecedents

Archimedes and the crown

King’s crown - gold, or silver

Archimedes knew gold and silver differed in density

Archimedes knew weight, but couldn’t geometrically measure to

obtain volume (and compute density)


Why has story persisted so long?


Why has story persisted so long?

• Resonates with our own experiences of

solving insight problems solving problems with

insight


• Solvers reach impasse (dead-end) - couldn’t measure

• Must reinterpret some aspect of problem

– Volume by water displacement

• Unconscious processing important

– If not thinking of crown, how recognize importance of water?

• Solution accompanied by “Eureka!”

Insight component processes?

Insight solutions associated with

• Switching to new strategy or associations (“restructuring”)

• Semantic integration -- solvers see connections thatpreviously eluded them

– Right hemisphere?

Solving problems with insight

Characteristics of both “insight problems” and solving

processes similar to characteristics of discourse and

comprehension processes for which the Right Hemisphere

(RH) seems to make contributions

•Drawing inferences, understanding the gist

•Getting jokes, metaphors, connotations

•2ndary word meanings

Solving problems with insight

• Solvers reach impasse (dead-end)

• Must reinterpret some aspect of problem

• Unconscious processing important

• Solution accompanied by “Aha!”

Short insight problems:

RAT Compound Remote Associate Problems Bowden & Jung Beeman, 1998

Remote Associates Test: The RAT (Mednick, 1962)

child

scanlamesame strike

tennis

RAT Compound Remote Associate Problems Bowden & Jung Beeman, 1998

child

scanlamesame strike

tennis

Aha! experience

• Solution appears sudden and obvious

• As soon as you think of solution, you “justknow” it works for all three words

– Comes as a whole, not part by part

• (vs strategic, step-by-step testing, etc)

Event-related fMRI design

• Insight solutions versus noninsight solutions

• Very “tight” comparison

– Not reveal whole network of problem solving

– Highlights just components that are uniquely engaged (or

at least emphasized) for insight solutions

L R Post Ant L R

Insight effect in RH anterior Superior Temporal Gyrus:

FMRI signal for insight > noninsight solutions.

L coronal R axial sagittal

p < .005, cluster > 500 mm3

-0.10

0.00

0.10

0.20

0.30

0.40

-2 2 4 6 8 10

-0.10

0.00

0.10

0.20

0.30

0.40

-2 2 4 6 8 10

Perc

ent sig

nal change

Percent

Signal

change

Time (sec)

RH aSTG: Singal change across the active region

Signal change for insight Insight effect

and noninsight solutions (Ins - non)

Signal change for insight and noninsight solutions, in aSTG

across hemispheres

-0.05

0.00

0.05

0.10

0.15

0.20

0.25

LH RHHemisphere

Percen

t sig

nal

ch

an

ge

Insight

Non

“Best” cluster

within each

hemisphere!!

Parallel study with 128 channel EEG

•Temporal specificity

•Processing specificity - frequencies

Gamma band insight effects

Insight solving conclusions

Insight solutions associated with increased activity in RH aSTG

• Binding and conscious accessibility (gamma) over RH aSTG

• Preceded by visual gating (alpha) - RH temp/ occipital areas




– - Lexical or semantic integration


– - Sensory gating indicates cognitive control?

Replication plus… more areasNew data set: improved N, scanner, protocol

RH aSTG (distant semantic integration)

• Anterior Cingulate (monitoring response competition, switching)

• Posterior Cingulate - same?

• Hippocampus/parahippocampal gyri - memory, reorgnzn?

L R Post Ant L R

Insight effect in RH Superior Temporal Gyrus:

FMRI signal for insight > noninsight solutions.


p < .001, cluster > 1000 mm3 ant and post STG

NONinsight effect in LH Inf. Frontal Gyrus:

FMRI signal for NONinsight > insight solutions.

sagittal

p < .005, cluster > 1000 mm3

LH IFG - dominant semantic retrieval

or selection

- turns on at problem onset

- off at solution, esp’y Insight

RH IFG - unusual retrieval / selection

- off at problem onset

- on at solution (I>NI, ns)

General vs specific mechanisms -

Visual Aha!

•

L R Post Ant L R

Visual Aha! effect in RH anterior Mid Temporal Gyrus:

FMRI signal for insight > noninsight recognition


p < .01, cluster > 500 mm3

L R Post Ant L R

Visual Aha! effect in RH anterior Mid Temporal Gyrus:



p < .01, cluster > 500 mm3

L R Post Ant L R

Visual Aha! effect in RH Angular Gyrus:



p < .01, cluster > 500 mm3 Also: RH Sup Frontal Gyrus

L R Post Ant L R

Visual Aha! effect in Bilateral M. Occipital Gyri:

FMRI signal for NONinsight > insight recognition


p < .005, cluster > 500 mm3

Visual Aha! conclusions

• NOT just for verbal problems

• Similarities - shared mechanisms (not “insight”, but…)

– Insight: top-down, cognitive control, integration

– RH -- unconscious, weak but mutually constraining, integration

– Recognition comes as a whole, not part by part

– Noninsight: bottom-up

• Some differences - Angular Gyrus somewhat surprising


Visual Aha!

•








– - Lexical or semantic integration


– - Sensory gating indicates cognitive control?


Insight solutions associated with

• Semantic integration -- solvers see connections thatpreviously eluded them

• When “the light goes on…”










Insight preparation

Do different mental states influence how you solve problems?

• Brain activity during a “rest period” (fMRI) or at a“Ready?” prompt (EEG), prior to getting a problem

• Problems solved with insight versus without insight

Preparation for Insight

• Is there a general form of preparation for insight

that begins before a problem is presented?

• We examined neural activity during the 2 sec

immediately before each problem was presented.

• Compared neural activity preceding problems

solved with insight to activity preceding problems

solved without insight.

8-9 Hz 9-10 Hz

-3.25

+3.25

I - T

N - T

t

Conclusions

• Two forms of preparation.

– Noninsight: Increased visual attention to displayed

problem.

– Insight: Mobilization and control of cognitive resources;

activation of temporal lobe semantic regions;

suppression of irrelevant thoughts.

Summary

• Insight is different from ordinary problem solving.

• Insight involves a sudden, discrete, awareness of thesolution to a problem.

• Insight involves different neural structures andmechanisms.

• Insight is the result of a special form of preparationinvolving cognitive regulation by medial frontal region.

Is insight really sudden?

Part II: Antecedents of insight

Positive mood facilitates insight and creative problem solving

(Isen et al.)

Insight and mood

Positive mood associated with increased creativity

– Better access to more distant associations

– Increased cognitive flexibility

• Anxiety associated with decreased creativity

– narrower focus of attention

Positive mood and insight

• Mood

– Positive mood enhances (anxiety impedes):

» Total solution rate

» % solved with insight

» Insight-like preparatory activity in ACC

Positive mood modulates prep

activity in ACC

Insight >NonPrep activity

Pos Aff>Neg in prep activ

Convergence


Visual Aha!

•

Thank you!

Documents

Bilateral semantic processing: Inferences in language ...cseweb.ucsd.edu/~gary/cs200/f06/slides/Jung-Beeman-cogsci200-11-20-2006.pdfBilateral semantic processing: Inferences in language,