MEMORY

observations and conjectures

last time

• consistent neurological anatomy– six -layer neocortex– columnar organization– dedicated areas (by genetics)

• accepts some observation– Mountcastle's common algorithm– Felleman and Van Essen's "hierarchical" structure

• facts– all perception is by spikes through nerves– spike intensity in repetitions– specificity in patterns

spatial-temporal patterns

• vision :– saccades about three times per second– move from fixation point to fixation point– not so much a photo, temporal changes– without saccades no quick recognition

• hearing:– pitches are heard at different places– high pitches at stiff base

low pitches at flexible end

• sensing– object not recognized

by touch– moving is essential

brain versus computer

• neurons are slow (5ms) compared to transistors• parallel is not an argument• object recognition in .5s that is 100 synapses in sequence

example: ask someone whether there is a cat in a picture

or: carry 100 heavy stones through the desert

• what is that difference?

the brain does not compute the recognition, it remembers stored memories as invariant representations

only conclusion can be :the human brain works fundamentally differentby laminar computing using cortical algorithms

solve the problem of catching a ball• computer needs millions of steps

– calculate the flight of the ball– joints of robot are adjusted in concert– process repeated

as better trajectory data is collected

• human brain– recalls appropriate memory upon seeing the ball– recalls a temporal sequence of muscle commands– makes adjustments

to accommodate the particulars of the moment• neocortical memory vs. computer memory

– the neocortex stores sequences of patterns.

– the neocortex recalls patterns auto-associatively.

– the neocortex develops invariant forms

– the neocortex is organized in a hierarchy.

storing sequences• memories are stored in a sequential fashion and

can only be recalled in the same sequence

– for example: say the alphabet in reverse order

– or: tell a story

– or: imagine (eyes closed) your home

– or: recognize a song (without pitch notation)

• people walk through a temporal sequence of doing– one pattern (open a door)– evokes another pattern (go through the door)– which evokes the next pattern (go down the hall)

• low level sensory memories

– memory for the tactile texture of gravel is based on pattern sequences across the pressure and vibration sensing neurons in your skin

sequential processing: brains v computers

• neocortex memory– all memories are stored in the synaptic connections

between neurons.– as you recall something, one set of neurons becomes

active, which then leads to another set of neurons becoming active, and so on.

– although the human brain stores an incredibly large number of memories, we can only remember a few at any time and can only do so in a sequence of associations.

• computer memory– does not normally store sequences of patterns.– it can be made to do so using various software tricks.– but that will cost time.

auto-associative recall

• an auto-associative memory system is one that can recall complete patterns when given only partial or distorted inputs

• this recall system works for both spatial and temporal patterns.– spatial - a child’s foot sticking out from behind draperies– temporal - smell of a madeleine cookie

• we are constantly completing patterns … it is an ubiquitous and fundamental feature of how memories are stored in the cortex

• during each waking moment, each functional region is essentially waiting for familiar pattern fragments to come in

• inputs to the brain auto-associatively link to themselves, filling in the present, and auto-associatively link to what normally follows next

• the chain of memories is called thought, and although its path is not indeterministic, we are not fully in control of it either

• truly random thoughts do not exist

invariant representations

• our brain does not remember or recall things with complete fidelity … because the brain remembers the important relationships in the world, independent of the details.

• a computer’s memory is designed to store information exactly as it is presented.

– artificial auto-associative memory systems fail to recognize patterns if they are moved, rotated, rescaled, or changed in other ways.

• when you see a friend’s face:– the pattern of activity of neurons in the visual input area

(V1) of your cortex is different for each view of her face.– the activity of cells in your face recognition area

– several steps higher than v1 in the cortical hierarchy –we find stability.

• the stability of cell firing is an invariant representation.

• everyday examples– a friend’s face (dimensions,proportions,colors,..)– recognizing a melody (pitch-invariant storage)– holding a book (robust againt displacement, folding,..)– finding sunglasses in a glove compartment

(just tactile clues, with different parts of the skin)– putting a key in your car’s ignition (not like robot)– writing your signature (invariant sequence of strokes)

• one of the biggest mysteries in science– Plato attempted to explain our ability to know a

perfect circle with his theory of forms

– stripped from all metaphysics he was right• prebirth superreality with perfect FORMS

• everyday forms are “reduced” to some FORMe.g. the concept of a circle

the “generic” dog

• not necessarily the same for everyone (unknown)

invariant representations

so far ……• revise the concept of understanding• plasticity of the brain

– Mountcastle’s observation– rewiring the brain

so far ……• revise the concept of understanding• plasticity of the brain

– Mountcastle’s observation– rewiring the brain (ferret experiment, braille reading, etc)

• presence of specialized regions (genetically determined)– right parietal lobe: perception on the left side– Broca’s area: grammar, not vocabulary– fusiform gyrus: recognition of faces

• everything is in cortex neurons (105/mm2) and synapses (potentially 3x1013, 3x1010 realized)

• spatio-temporal patterns for senses• memory

– sequences of patterns– auto-associative recall– invariant representations– hierarchy

what is understanding? • notice the difference

– a new object in a familiar environment– a shuffled sequence from a cd– another version of a well-known work– surprises in a daily routine

• indicate low-level sensory predictions– listening to somebody speaking– blind spots

• not completely new:– Mackay [1956]: intelligent machines based on

“match what is received”– Llinas [2001]: “the capacity to predict future outcome

- critical to successful and save movement –is most likely the ultimate and most commonof all brain functions” (of the neocortex)

– Pearl: “belief propagation” modelled by bayesian networks

• IQ test

with or without the neocortex

• reptiles– well developed senses and ‘old brain’– complex behavior: swim, run, hide, hunt, ambush, sun, nest and mate– cannot predict (not escape from a maze it has seen before)

• rats– old brain plus cortex– can escape from a maze– survive with a damage motor cortex

• dolphins– 3-layer cortex– yet, great navigation and recognition abilities

• chimpanzees– much smaller cortex, mainly the posterior part

• humans– old brain plus cortex which much larger anterior part (half)– heavily connection to motor part, adding the future– paralyzed when motor cortex is damaged

what is intelligence

• memory-prediction framework– store sequences of patterns– retrieve by similarities to past patterns (auto-association)– memories stored in invariant forms

• the chinese room does not understand chinese– it cannot predict the next character– it cannot predict the next story element

• turing test– behavior does not prove intelligence– prediction is not included in the test

1) shared representations lead to generalization and efficiency

2) hierarchy matches spatial and temporal hierarchy of causes in world

3) belief propagation techniques ensure all nodes quickly reach mutually compatible beliefs

4) affords mechanism for attention

why does hierarchy make a difference?

80% woof

20% meow

70% pig image

30% cat image

90% cat

• information flows up and down• collective activity on a bundle of

fibers is a pattern• by the time get to the top layer,

cells fire whenever an object is present

• network of feedback connections (more than forward!)

• prediction requires a comparison between what is happening and what you expect to happen

invariant representations

IT

V4

V2

V1

invariant representations

� happens up and down each sense, up to association areas which are between senses

visionauditiontouchspatiallyspecific

spatiallyinvariant

invariant representations

visionauditiontouchchanging

fastly

changingslowly

� happens up and down each sense, up to association areas which are between senses

invariant representations

visionauditiontouchfeatures,

details

“objects”

� happens up and down each sense, up to association areas which are between senses

invariant representations

• transformation from specific to invariant occurs in all sensory areas of cortex

– pressure � sounds � words � phrases

• the neural activity corresponding to the mental perception of objects

– lasts longer than the individual input patterns.– higher in cortex, fewer changes over time

integrating the senses

visionauditiontouch

• something I hear can lead to a prediction of what I see• surprised if it is not what you expect

• all sensory and association areas act as one: a multibranched hierarchy

• all predictions are learned by experience (it’s all just a little bit of history repeating…)

• the motor cortex behaves like sensory system: downward flow in motor = commands

• all one sense: seeing, hearing, touching, and acting

are profoundly intertwined

integrating the senses

invariant representations

visionauditiontouch

� happens up and down each sense, up to association areas which are between senses

the size of “lower” areas

a new look at hierarchical levels

• why should IT be the only region with invariant representations?

• V1 should be seen as made up of smaller subregions

• the role of each higher region is to memorize patterns of the lower regions

� think of left V1 and right V1 as separate sensory streams that get united higher up, just like senses on a broader scale

� now each layer of cortex forms invariant representations of the input from lower areas

visionauditiontouch

visionauditiontouch

what does the neocortex look like?

• layer 1: mainly axons inside layer!• layer2,3: packed with

pyramidal cells• layer 4: star-shaped cells• layer 5: regular pyramidal cells,

but also bigger ones.• layer 6: often specialized cells• arranged in columns

with many connection

sequences (or sets) of sequences

• unfolding of sequences (preamble)• the same amount of detail in the feedback applies to each level• the exception to this is:

if the lower regions of cortex fail to predict what patterns they are seeing, they consider this an error and pass the error up the hierarchy-until something recognizes the pattern

• the brain must classify patterns• the cortex is flexible in its pattern classifications• difference between your brain and a machine:

it recognizes sequences of patterns that correspond to the world, as opposed to matching objects with prototypes. the sequences are the only reality you know.

a model of the world

• nested or hierarchical structure: notes � intervals � phrases � melodies � songs

• sub-objects: line segments � shapes � noses � faces � person

• only exist in one moment in time, but hierarchy allows you to extrapolate the permanent details

• sequences! follow each other in time, if not always order. (facial recognition example)

• “predictability is the very definition of reality: if a region of cortex finds it can reliably and predictably move among these input patterns using a series of physical motions and can predict them accurately as they unfold in time, the brain interprets these as having a causal relationship.”

• brain stores sequences of sequences[Hawkins]

cortexworld

people

cars

buildings

words

songs

ideas

senses

patterns

a model of the world“causes” “beliefs”

our intelligent

machine

cause1 0.22

cause2 0.07

cause3 0.00

cause4 0.63

cause5 0.00

cause6 0.08

what should the intelligent machine do?

1 discover causes in the world

2 infer novel input

3 predict future

4 direct motor behavior

representations

of causes

sensory data

beliefseach node:discovers causes

(of its input)

passes beliefs up

passes predictions down

each node:stores common sequences

changing sensory data forms stable beliefs at top

stable beliefs at top form changing sensory predictions

hierarchy of "memory nodes"

1) why does hierarchy make a difference?

2) how does each node discover and infer causes?

3) how does time come into the proces?

1) shared representations lead to generalization and efficiency

2) hierarchy matches spatial and temporal hierarchy of causes in world

3) belief propagation techniques ensure all nodes quickly reach mutually compatible beliefs

4) affords mechanism for attention

why does hierarchy make a difference?

80% woof

20% meow

70% pig image

30% cat image

90% cat

how does each node discover causes?1) learn common spatial patterns

2) learn common sequences of spatial patternsad 1:things that happen at the same time are likely to have a common cause

common patterns:

remember

uncommon patterns:

ignore

ad 2:things that happen consecutively might have a common cause

common sequence:assign to cause

time

common sequence:assign to cause

uncommon sequence:ignore

4 pixels

Level 1

Level 2

Level 3

Is there a prototype? Simple HTM vision system (32x32 pixel)

causes

sensory data

representation

of causes

intelligent machinethe world

1 discover causes in the world2. infer novel input3. predict future4. direct motor behavior

each node: discovers causes (of its input), passes beliefs up, predictions down

each node: stores common sequences, changing sensory data forms stable beliefs at topstable beliefs at top form changing sensory predictions

1) shared representations lead to generalization and efficiency

2) hierarchy matches spatial and temporal hierarchy of causes in world

3) belief propagation techniques ensure all nodes quickly reach mutually compatible beliefs

4) affords mechanism for attention

benefits of "hierarchical" structure