Language and Perception Ling 411 – 17. Perception: Starting view Perception is a bottom-up...

Language and Perception

Ling 411 – 17

Perception: Starting view

Perception is a bottom-up process• From primary perceptual area upwards

E.g. primary auditory, for auditory perception Takes place in a single perceptual area

• E.g. auditory cortex for auditory perception Works by processing input to the sense organ

• E.g. auditory input for auditory perception

Each of these points is wrong!

The McGurk Effect

Acoustic syllable [ba] presented to subjects with visual presentation of articulatory gestures for [ga] Subjects typically heard [da] or [ga] “Evidence has accumulated that visual speech modifies

activity in the auditory cortex, even in the primary auditory cortex.”

Mikko Sams (2006)

http://www.youtube.com/watch?v=aFPtc8BVdJk

How does it work? (1)Visual input(2)Top-down processing

Perception – Refining a simple-minded view

1. It is not confined to a single perceptual modality• The McGurk effect

Auditory perception affected by visual input • Conceptual structure affects auditory perception

The influence of context on speech perception

Perception – Refining a simple-minded view

1. Not confined to a single perceptual modality• The McGurk effect

Visual input affects auditory perception• Conceptual structure affects auditory perception

2. Not just bottom-up• Top-down processing fills in unsensed details

3. Not even confined to posterior cortex• Can also use motor neurons

Experiment: left hand or right hand? Mirror neurons

Top-down processing in perception

TCUP

MADE OF GLASS

CERAMICSHORT

HAS HANDLE

Properties

Conceptual and perceptual information

Node for CUP in conceptual area for drinking vessels

Visual properties are in occipital and lower temporal areas

Bidirectional processing and inference

T

CUP

MADE OF GLASS

CERAMICSHORT

HANDLE

These connections are bidirectional

Pertinent neuroanatomical findings: Bidirectional Processing

An established fact of neuroanatomy:• A connection from point A to point B in

the cortex is generally accompanied by a connection from point B to point A

Separate fibers (axons): (1) A to B, (2) B to A

In short, cortico-cortical connections are generally bidirectional

Bidirectional processing: reciprocal links

excitatory

inhibitory

Bidirectional processing and inference

T

CUP

SHORT

HANDLE

Thought process: 1. The cardinal concept node is activated by a subset of its property nodes 2. Feed-backward processing activates other property nodes

Consequence: We “apprehend” properties that are not actually present in the sensory input

Bidirectional processing and inference

T

CUP

MADE OF GLASS

CERAMICSHORT

HANDLE

These connections are bidirectional

Separate fibers for the two directions; shown as one line in the notation

Cortical Structure and Inference:Perceiving things that are not in the input

T

Category

Properties

A

B F

E

Consequence: If A and B, then E and F

C D

Examples

Looks like a duck• Probably quacks

Ceramic, cup-shaped, handle• Probably holds coffee (without

breaking) Dark clouds, thunder

• It’s going to rain ATM

• Probably has money

Perception depends mainly on cortical structures already present before sensory input

“Perception is hallucination constrained by sensory data”

Shepherd

A terminological problem

We need to distinguish• Perception narrowly conceived

The basic process of recognition Single perceptual modality Bottom-up processing No motor involvement

• Perception broadly conceived Two different terms needed

• Recognition (a.k.a. ‘microperception’) Bottom-up process in a single perceptual modality

• Perception (the broad conception) (a.k.a. ‘macroperception’)

“Micro-perception” and “macro-perception”

Microperception and macroperception Microperception

• A.k.a. recognition• The local process of integrating features• Performed in one perceptual modality• Bottom-up

Macroperception• The overall process of perception• Uses multiple modalities• Uses top-down processing

Perception – Refining a simple-minded view

1. Not just bottom-up• Top-down processing fills in unsensed details

2. Not confined to a single perceptual modality• The McGurk effect

Visual input affects auditory perception• Conceptual structure affects auditory perception

3. Not even confined to posterior cortex• Can also use motor neurons

Experiment: left hand or right hand? Mirror neurons

Left hand or right hand?

Left hand or right hand?

Left hand or right hand?

Left hand or right hand?

Left or right hand?

Imaging experiment Subjects were shown pictures of one hand Asked to identify: left or right Functional imaging showed increased CBF in

hand area of motor cortex

Peter Fox, ca. 2000

Motor structures in perception

The left-hand vs. right-hand experiment ‘Mirror neurons’ in motor cortex Articulation as aid to phonological perception Articulation in reading Motor activity in listening to music Watching an athletic event

Mirror Neurons

NY Times: “One mystery remains: What makes them so smart?” (Jan. 10, 2006)

Answer: They are not smart in themselves• Their apparent smartness is a result of

their position: at top of a hierarchy• Compare:

The general of an army The head of a business

Similarly, high-level conceptual nodes• The “grandmother node”

Mirror Neurons

What makes mirror neurons appear to be special?• Ans.: They receive input from visual perception• The superior longitudinal fasciculus

Connects visual perception to motor areas How can a motor neuron receive perceptual input?

• Motor neurons are supposed to operate top-down• Answer: bidirectional processing

They also receive perceptual information• Bottom-up processing

Superior Longitudinal Fasciculus

From O. D. Creutzfeldt, Cortex Cerebri (1995)

Are some neurons “smarter” than others?

Claim: A grandmother node would have to be very smart • Identifies very complex object• Even in many varieties

Alternative: the head of a hierarchy• It is the hierarchy as a whole that has

those ‘smarts’• Similarly, mirror neurons

They get visual input since they are connected to visual areas

• Superior longitudinal fasciculus

Implications of hierarchical organization

Nodes at a high level in a hierarchy may give the appearance of being very “smart”

This appearance is a consequence of their position — at top of hierarchy

As the top node in a hierarchy, a node has the processing power of the whole hierarchy• Grandmother nodes• Mirror neurons• Compare:

The general of an army The head of a business organization

Multi-Modal Perception

1. Perception is not just bottom-up• Top-down processing fills in unsensed details

2. It is not confined to a single perceptual modality• The McGurk effect

Visual input affects auditory perception• Conceptual structure affects auditory perception

3. It is not even confined to posterior cortex• Can also use motor neurons

Motor activation in speech perception Mirror neurons

Perceptual structures in motor production

Perceptual structure is used in two ways1. Planning (e.g. visualizing while painting)2. Monitoring

Examples• Phonological recognition in speech production

Cf. Wernicke’s aphasia• Painting• Musical production• Baseball, soccer, tennis, etc.

The Influence of language on non-linguistic perception

As we have seen, non-speech input affects phonological perception

It is also the case that language affects non-linguistic perception• E.g., visual perception

Language and (Visual) Perception

Phonological Phonological Object Production Perception Categories

VisionLanguage

N.B.: These connections are bidirectional

Recent experiments of Kay et al.

Experiments at UC Berkeley• Color perception: do differences in color naming

across languages influence color perception? Main finding:

• Lateralized influence of language on perception• Response time faster for between-category

discrimination – especially for RVF presentation• A left hemisphere (RVF) phenomenon

green blue

Perception: A simple-minded view, revisited

Perception is a bottom-up process• From primary perceptual area upwards

E.g. primary auditory, for auditory perception Takes place in a single perceptual area

• E.g. auditory cortex for auditory perception Works by processing input to the sense organ

• E.g. auditory input for auditory perception

Each of these points in wrong

Another hypothesis of Whorf

Grammatical categories of a language influence the thinking of people who speak the language

Can we explain this too in terms of brain structure?

Mechanisms of operation

1. Entrenchment • Strengthening of connections

through repeated activation An automatic brain process Important in learning

2. Reverberation of activation 3. Priming4. Language as a major means of learning

conceptual and perceptual distinctions

Entrenchment and thinking: a mechanism

Connections become stronger with use• (entrenchment)

Grammatical categories make speakers constantly heed selected phenomena

Connections for phenomena which speakers must constantly heed.. • Will be repeatedly traversed• Therefore will get progressively stronger

Example: Grammatical gender

Does talking about inanimate objects as if they were masculine or feminine actually lead people to think of inanimate objects as having a gender?

Could the grammatical genders assigned to objects by a language influence people’s mental representation of objects?

Boroditsky (2003)

Plausibility of the possibility

Children learning to speak a language with grammatical gender may suppose that gender indicates a meaningful distinction between types of objects

Other grammatical distinctions do reflect actual perceptual differences: singular:plural

Children learning a language with gender

“For all they know, the grammatical genders assigned by their language are the true universal genders of objects.”

Boroditsky et al, 2003

Experiment: Gender and Associations(Boroditsky et al. 2002)

Subjects: speakers of Spanish or German• All were fluent also in English• English used as language of experiment

Task: Write down the 1st 3 adjectives that come to mind to describe each object• All the (24) objects have opposite gender

in German and Spanish Raters of adjectives: Native English speakers

Examples:

Key (masc in German, fem in Spanish)

• Adjectives used by German speakers: Hard, heavy, jagged, metal, serrated, useful

• Adjectives used by Spanish speakers: Golden, intricate, little, lovely, shiny, tiny

Bridge (fem in German, masc in spanish)

• Adjectives used by German speakers: Beautiful, elegant, fragile, peaceful, pretty

• Adjectives used by Spanish speakers: Big, dangerous, long, strong, sturdy,

towering

Results of the Experiment(Boroditsky et al. 2002)

Raters of adjectives were native English speakers

Result: Adjectives were rated as masculine or feminine in agreement with the gender in subject’s native language

end