Brain, Psycholinguistics, & Cognitive Science

Outline

• How does psycholinguistics fit within the umbrella of cognitive science?

• What do we know about language and the brain?

Cognitive Science

Cognitive Computer Cognitive Psychology Linguistics Philosophy Science/AI Neuroscience

Psycholinguistics Natural Language Processing Neurolinguistics

Inter-relationships

What do cognitive psychologists do?

Construct theories that describe how mental processing works in real time:– Within some domain, what are the mental

representations and how are they manipulated?– How does information flow through the mind? – How is processing impacted by memory

constraints, stimulus quality, mode of input, tasks, etc?

Levels of processingSerial vs. parallel processing

Top-down/bottom-upAutomatic vs. strategic processing

What do (formal theoretical) linguists do?

• Construct formal theories of our linguistic knowledge– Sets of rules/principles/assumptions

for generating utterances

• Criteria for a good theory– The rules generate all and only

grammatical outputs (intuitions)– In the simplest way

Derivational Theory of Complexity

• Miller’s clause-processing model of syntactic processing, motivated by Chomsky’s Transformational Grammar.

• Example of taking a linguistic theory and trying to implement it directly as a psychological theory of processing.

Transformational Grammar

• Two levels of syntactic structure, related by transformational rules; accounts for similarity in meaning across different strings.– Surface structure ((John) (picked __ (the box) up)).

– Transformations Particle Movement

– Deep Structure ((John) (picked up (the box))).

• Phrase structure rules and lexical insertion rules used to construct DS

Diff SS, Same DS

Transformations can move, delete, or add words. Complex SS’s require a sequence of transformations– Wh-movement & Subj/Aux Inversion

SS: Which book did Jim buy __?

SS Jim did buy a book.

DS: Jim did buy a/which book

– PassivizationSS: The beer was drunk by Jim.

SS: Jim drank the beer.

DS: Jim drank the beer.

DTC

1.Determine the surface structure of the sentence

2.Reverse the transformations one by one

3.Recover the deep structure

4.Map DS to meaning

Processing Unit is entire Sentence

• Wh-movement & Subj/Aux InversionSS: Which book did Jim buy __?

SS: Jim did buy a book.

DS: Jim did buy a/which book

• PassivizationSS: The beer was drunk by Jim.

SS: Jim drunk the beer.

DS: Jim drunk the beer.

How would you test the DTC?

• Early evidence seemed to support it.

• Later evidence was problematic

• No longer a viable theory of sentence comprehension.

Memory & Language

• Semantic Memory: LTM storehouse of conceptual knowledge

–What is a cow? What is truth?

• Lexicon: LTM storehouse of knowledge about words

–Lexical Entry: For each word/morpheme,Spelling, pronunciation, syntactic category,

pointer to semantic memory

• Episodic Memory: LTM storehouse of our experiences

Spreading Activation Model (Collins & Loftus, 1975)

• Length codes typicality

• Link codes type of relation

• Some redundancy

is a

is

uses

Commmon Current Assumptions about Semantic Memory & the Lexicon

• Semantic Memory is a network of concepts, organized by semantic similarity

• Lexicon is a network of words, organized by phonological similarity

• Interconnections link meanings to words

Cognitive Neuroscience

• Cognitive psychologists tend to talk about the architecture of the mind in terms of functionality– E.g., what is the input to word recognition? What is the output?

• Ultimately, the mental operations described by cognitive psychology occur in the brain

• In some cases, neuroscience can inform cognitive psychology– E.g., we may be able to learn about how words/concepts are

represented by investigating activation patterns in response to different classes of words: action verbs activate motor cortex; perception verbs activate visual cortex

Brain Anatomy (& Language)

– Is language localized in the brain?– Is language lateralized?

Frontallobe

Temporallobe

Parietal lobe

Occipitallobe

Cerebellum

LH lobes

Some Lg-relevant areas

Video Clip (15 min)

Brain Story: First among Equals– The first segment has been comparing human

and chimp abilities to plan. They conclude that chimps can make and execute plans, but not as far into the future as humans.

– We’ll watch a segment on LG• Aphasia patient

• Mapping the brain prior to surgery

What did you learn from the clip?

• What does it mean to have aphasia?

• Do aphasics recover language function? How?

• What is the current view on the role of Broca’s and Wernicke’s areas?

Schiff et al. (2005)

• Do minimally conscious patients process speech?

Schiff, N. D. et al. Neurology 2005;64:514-523

Figure 1. Functional maps obtained during listening to narrativesYellow= forward;Blue = backwardRed = both

Averaged healthy data

Schiff, N. D. et al. Neurology 2005;64:514-523

Figure 2. Volumes of activation during the passive listening tasks. The 2 patients are in blue and red; Averaged healthy activation in black.

Schiff et al.

– paper was published shortly after the Terry Schiavo media/political frenzy.

– How do we decide whether a minimally conscious patient is experiencing a life worth living?

• Is language comprehension relevant?

• Why don’t we care as much about how their brain responds to smells?

Speech & Spoken Word Recognition

Outline

• Why is speech perception difficult for computers to do?– Problem of Invariance

• How do humans do it so easily?– Bottom-up information (acoustic signal)– Top-down information (higher level context)

Automatic Speech Recognition Follies (David Pogue, NYTimes, 8/15/02)

What I saidbookmark itMotorollamodem porta procedureand then stick it in the

mailmovie clipsI might addInscrutablethe right or left

What was transcribedbook marketmotor roll amode importupper seizureand dense thicket in the

mailmove eclipseI my dadin screw doublethe writer left

Same sequences of phonemes form different words

Different phonemes, at underlying level

The Problem of Invariance

• Individual phonemes do not have invariant acoustic cues.• There’s a lot of variability in the acoustic signal!• Variance in the acoustic signal has many sources:

– Coarticulation– Differences among speakers– Differences within speakers: yelling/whispering, phone/in-person,

etc.

If the bottom-up acoustic signal doesn’t provide consistent cues, how do we recognize phonemes/words?

Reading a spectrogram

Can you see invariant cues associated with /i/ ?

formants

If we can’t/don’t rely solely on bottom-up input, how do we

recognize speech?

• Perceive speech as (intended?) articulatory gestures, not as acoustic signal?– McGurk Effect– Sine Wave Speech: There are no essential acoustic

properties that enable speech perception. Rather second-order changes in frequency and amplitude over time are responsible. sine wave speech demos

• Use top-down information (word and sentence context) to complement bottom-up information– If so, when and how?

Is Speech Special?

Modularity Thesis (Fodor, 1983)– The mind is not a unified whole. In addition to

central processes, there are specialized input-output modules

• Central: decision making• Input: color perception, voice recognition• Output: throwing, touch-typing, articulation

– Modules are fast, informationally-encapsulated, mandatory, exhibit characteristic breakdowns, and have shallow outputs.

– Speech perception may be handled by a specialized input module

The Essence of Motor Theory

• Speech perception is grounded in our knowledge of speech production. We recognize phonemes by covertly re-creating the articulatory gestures. (Lieberman et al., 1967)

• Consistent with philosophy that performance & perception are inextricably linked.

• Assume innate, encapsulated phonetic module

Spoken Word Recognition

• Overcoming the problem of invariance in speech perception– Motor Theory– Top-Down Feedback: Word to Phoneme

• TRACE

• Cohort theory of Spoken Word Recognition

Evidence for Top-Down influence on speech perception

• Phoneme Restoration Effect (Warren, 1970)

• Lexical bias in categorical perception task, e.g. dype vs. type (Clifton & Connine, 1987)

TRACE (McClelland & Elman, 1986)

• Interactive connectionist model

• Nodes in network represent phonetic features, segments, & words

• Feature nodes activated by consistent input

• Activation spreads up through network & back down again

• Predicts top-down effects

Example: initial phoneme in “pick” is ambiguous betwn /b/ & /p/.

(Lexical Bias)

Is word recognition Automatic & Modular?

Automatic Processes– Fast– Do not require attention– Feed-forward (can’t be guided, controlled,

or stopped midstream)– Not subject to top-down feedback

(informational encapsulation)

Stroop Effect

Name font color

RED

GREEN

BLUE

YELLOW

GREEN

What happens if you have to name word?

Differences between spoken and written word recognition

• For relatively short words, letters in a written word are processed in parallel– Eye movement data– Word superiority effect– Letter-Search Task

• Spoken word unfolds across time– Can recognize some words before they are

completely pronounced.

Research on the Lexicon (Outline)

• How are lexical entries accessed? (Word Recognition) What is the input?– Speaking (Ashcraft)– Reading– Listening

• How is lexical ambiguity resolved?