PDP: Motivation, basic approach

Cognitive psychology

or

“How the Mind Works”

Information processing

Perception / sensation Action

Transformations Mental representations

Key questions:

• What are the mental representations?• What are the transformations and how do they work?• Where do these come from?

Answers circa 1979

• Mental representations are data structures (symbols bound together in relational structures)

• “Transformation” processes are rules, like the instructions in a computer program.

• The rules and starting representations are largely innate• Learning is a process of building up new structured

representations from primitives and deriving (and storing) new rules.

Why were (are?) these ideas so appealing?

EG modular, serial processing in word recognition…

Their application is highly intuitive, logical, maybe even obvious!

Explain generalization / generativity, central to human cognition!

EG: Today I will frump the car, yesterday I….______________ the car.

EG: Colorless green ideas sleep furiously!

Can build mechanistic, implemented models of behavior…

Explains modular cognitive impairments!

Pure alexia: Word representations gone!

Prosopagnosia: Face reps gone!

Category-specific impairment: Animals gone!

Broca’s aphasia: Phonological word forms gone!

And so on…

So why isn’t this a class on symbolic cognitive modeling?

W O R K

EG: Today I will meep the car, yesterday I….______________ the car.

Learning and development…

Cognitive impairments not so modular…

Pure alexia: Can’t see high spatial frequencies.

Prosopagnosia: Abnormal visual word recognition.

Category-specific impairment: Can still name half the animals.

Broca’s aphasia: Can still produce single concrete nouns, can’t do grammar.

And so on…

But also Had a lot of Other Problems…

Some issues with modular, serial, symbolic approaches• Constraint satisfaction, context sensitivity• Quasi regularity• Learning and developmental change• Graceful degradation

Rumelhart

• Brains are different than serial digital computers.• Brain:

• Neurons are like little information processors• They are slow and intrinsically noisy…• …but there are many of them and they behave in parallel, not in serial.• It turns out that noisy, parallel computing systems are naturally suited to

some of the kinds of behavioral tasks that challenged symbolic theories of the time.

In other words…

• Rather than thinking of the mind as some kind of unconstrained “universal computer,” maybe we should pay attention to the kinds of computations a noisy, parallel, brain-like system can naturally conduct.

• Paying attention to the implementation might offer some clues about / constraints on theories about how the mind works.

• Added bonus: Such theories might offer a bridge between cognition and neuroscience!

Golgi Stain

Cell body

Dendrites(receive signals)

Axon(transmits signals)

Axon Terminal

-

-

--

--

+

+

+

+

+

++ +

++

+ + + +

- - - - -----

+

--

- -

--

+

+

+

+

+

+

Membrane potential at axon hillock

DepolarizedHyperpolarized

p(Fi

ring

a sp

ike)

Input: Depends on activation of sendingneurons and efficacy of synapses

Output: Train of action potentials at a particular rate

Weight: Effect on downstream neuron

Six elements of connectionist models:

1. A set of units2. A weight matrix3. An input function4. A transfer function5. A model environment6. A learning rule

Six elements of connectionist models:

1. A set of units• Each unit is like a population of cells with a similar

receptive field.• Think of all units in a model as a single vector, with

each unit corresponding to one element of the vector.

• At any point in time, each unit has an activation state analogous to the mean firing activity of the population of neurons.

• These activation states are stored in an activation vector, with each element corresponding to one unit.

Input

Output

Hidden [1 0 .51 .52 .45]

Input

Output

Hidden

Bias

[1 0 .51 .52 .45 1]

Six elements of connectionist models:

2. A weight matrix• Each unit sends and receives a weighted connection

to/from some other subset of units.• These weights are analogous to synapses: they are

the means by which one units transmits information about its activation state to another unit.

• Weights are stored in a weight matrix

Input

Output

Hidden

Bias

[1 0 .51 .52 .45 1]

Sending

Receiving

Six elements of connectionist models:

3. An input function• For any given receiving unit, there needs to be some

way of determining how to combine weights and sending activations to determine the unit’s net input

• This is almost always the dot product (ie weighted sum) of the sending activations and the weights.

Input

Output

Hidden

Bias

[1 0 ?? ?? ?? 1]

-.2

-1.8

0.5 0.2

1.1 -2.2

.1 .2 .3 .4 .2

Sending

Receiving

Six elements of connectionist models:

Input

Output

Hidden

Bias

[1 0 ?? ?? ?? 1]

-.2

-1.8

0.5 0.22

1.1 -2.2

.1 .2 .3 .4 .2

Sending

Receiving

Six elements of connectionist models:

5. A model environment• All the models do is compute activation states over units, given

the preceding elements and some partial input.• The model environment specifies how events in the world are

encoded in unit activation states, typically across a subset of units.

• It consists of vectors that describe the input activations corresponding to different events, and sometimes the “target” activations that the network should generate for each input.

In1 In2 Out

0 0 0

0 1 1

1 0 1

1 1 0

X-OR function

Input

Output

Hidden

Bias

Input1 Input2 Hidden1 Hidden2 OutputInput1Input2Hidden1Hidden2Outputbias

Input1 Input2 Hidden1 Hidden2 OutputInput1Input2Hidden1Hidden2Outputbias

• Note that the model environment is always theoretically important!

• It amounts to a theoretical statement about the nature of the information available to the system from perception and action or prior cognitive processing.

• Many models sink or swim on the basis of their assumptions about the nature of inputs / outputs.

Six elements of connectionist models:

6. A learning rule• Only specified for models that learn (obviously)• Specifies how the values stored in the weight matrix

should change as the network processes patterns• Many different varieties that we will see:

• Hebbian• Error-correcting (e.g. backpropagation)• Competitive / self-organizing• Reinforcement-based

In1 In2 Out

0 0 0

0 1 1

1 0 1

1 1 0

X-OR function

Input

Output

Hidden

Bias

Input1 Input2 Hidden1 Hidden2 OutputInput1Input2Hidden1Hidden2Outputbias

Input1 Input2 Hidden1 Hidden2 OutputInput1Input2Hidden1Hidden2Outputbias

Central challenge

• Given just these elements, can we build little systems—models—that help us to understand human cognitive abilities, and their neural underpinnings?

One early example…