Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary Itti: CS564 - Brain Theory and Artificial Intelligence University of Southern

Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary

Itti: CS564 - Brain Theory and Artificial Intelligence

University of Southern California

Lecture 28. Overview & Summary

Reading Assignment:

TMB2 Section 8.3

Supplementary reading: Article on Consciousness in HBTNN


You said “brain” theory??

First step: let’s get oriented!




Major Functional Areas

Primary motor: voluntary movementPrimary somatosensory: tactile, pain, pressure, position, temp., mvt.Motor association: coordination of complex movementsSensory association: processing of multisensorial informationPrefrontal: planning, emotion, judgementSpeech center (Broca’s area): speech production and articulationWernicke’s area: comprehen-

sion of speechAuditory: hearingAuditory association: complex

auditory processingVisual: low-level visionVisual association: higher-level

vision


Neurons and Synapses


http://www.radiology.wisc.edu/Med_Students/neuroradiology/fmri/








Limbic System

Cortex “inside” the brain.Involved in emotions, sexual behavior, memory, etc(not very well known)


Major Functional Areas


Some general brain principles

Cortex is layered

Retinotopy

Columnar organization

Feedforward/feedback


Layered Organization of Cortex

Cortex is 1 to 5mm-thick, folded at the surface of the brain (grey matter), and organized as 6 superimposed layers.

Layer names:1: Molecular layer2: External granular layer3: External pyramidal layer4: internal granular layer5: Internal pyramidal layer6: Fusiform layer

Basic layer functions:Layers 1/2: connectivityLayer 4: InputLayers 3/5: Pyramidal cell bodiesLayers 5/6: Output


Retinotopy

Many visual areas are organized as retinotopic maps: locations next to each other in the outside world are represented by neurons close to each other in cortex.Although the topology is thus preserved, the mapping typically is highly non-linear (yielding large deformations in representation).

Stimulus shown on screen… and corresponding activity in cortex!


Columnar Organization

Very general principle in cortex: neurons processing similar “things” are grouped together in small patches, or “columns,” or cortex.

In primary visual cortex… as in higher (object recognition) visual areas…

and in many, non-visual, areas as well (e.g., auditory, motor, sensory, etc).



Felleman & Van Essen, 1991

Interconnect


Neurons???

Abstracting from biological neurons to neuron models


The "basic" biological neuron

The soma and dendrites act as the input surface; the axon carries the outputs.

The tips of the branches of the axon form synapses upon other neurons or upon effectors (though synapses may occur along the branches of an axon as well as the ends). The arrows indicate the direction of "typical" information flow from inputs to outputs.

Dendrites Soma Axon with branches and

synaptic terminals


Transmenbrane Ionic Transport

Ion channels act as gates that allow or block the flow of specific ions into and out of the cell.


Action Potential and Ion Channels

Initial depolarization due to opening sodium (Na+) channels

Repolarization due to opening potassium (K+) channelsHyperpolarization happens because K+ channels stay open longer than Na+ channels (and longer than necessary to exactly come back to resting potential).


A McCulloch-Pitts neuron operates on a discrete time-scale, t = 0,1,2,3, ... with time tick equal to one refractory period

At each time step, an input or output is

on or off — 1 or 0, respectively.

Each connection or synapse from the output of one neuron to the input of another, has an attached weight.

Warren McCulloch and Walter Pitts (1943)

x (t)1

x (t)n

x (t)2

y(t+1)

w1

2

n

w

w

axon


From Logical Neurons to Finite Automata

AND

1

1

1.5

NOT

-10

OR

1

1

0.5

Brains, Machines, and Mathematics, 2nd Edition, 1987

X Y

Boolean Net

X

Y Q

Finite Automaton


Leaky Integrator Neuron

The simplest "realistic" neuron model is a continuous time model based on using

the firing rate (e.g., the number of spikes traversing the axon in the most recent 20 msec.)

as a continuously varying measure of the cell's activity

The state of the neuron is described by a single variable, the membrane potential.

The firing rate is approximated by a sigmoid, function of membrane potential.


Leaky Integrator Model

= - m(t) + h

has solution m(t) = e-t/ m(0) + (1 - e-t/)h

h for time constant > 0.

We now add synaptic inputs to get the

Leaky Integrator Model:

= - m(t) + i wi Xi(t) + h

where Xi(t) is the firing rate at the ith input.

Excitatory input (wi > 0) will increase

Inhibitory input (wi < 0) will have the opposite effect.

m(t)

m(t)

m(t)


Models of what?

We need data to constrain the models

Empirical data comes from various experimental techniques:

PhysiologyPsychophysicsVarious imagingEtc.


Electrode setup

- drill hole in cranium under anesthesia- install and seal “recording chamber”- allow animal to wake up and heal- because there are no pain receptors in brain, electrodes can then

be inserted & moved in chamberwith no discomfort to animal.


Receptive field


Example: yes/no task

Example of contrast discrimination using yes/no paradigm.

- subject fixates cross.- subject initiates trial by pressing space bar.- stimulus appears at random location, or may not appear at all.

- subject presses “1” for “stimulus present” or “2” for “stimulus absent.”

- if subject keeps giving correct answers, experimenter decreases contrast of stimulus (so that it becomes harder to see).

time

+


Staircase procedure

Staircase procedure is a method for adjusting stimulus to each observer such as to find the observer’s threshold. Stimulus is parametrized, and parameter(s) are adjusted during experiment depending on responses.

Typically:- start with a stimulus that is very easy to see.

- 4 consecutive correct answers make stimulus more difficult to see by a fixed amount.- 2 consecutive incorrect answers make stimulus easier to see by a fixed amount.


Example SPECT images


Reconstruction using coincidence


Example PET image


BOLD contrast

Bo Bo

with deoxygenated blood

with oxygenatedblood

The magnetic properties of blood change withthe amount of oxygenation

resulting in small signal changes


Vascular System

arteries arterioles(<0.1mm)

capillaries venules(<0.1mm)

veins


Oxygen consumpsion

O 2

O 2

O 2

O 2

O 2

O 2

O 2

O 2

O 2

O 2

O 2

O 2

O 2

O 2

O 2

O 2

The exclusive source of metabolic energy of the brain is glycolysis:

C6H12O6 + 6 O2 6 H2O + 6 CO2


BOLD Contrast

stimulation

neuronal activation

metabolic changes

hemodynamic changes

local susceptibility changes

MR-signal changes

data processing

functional image

signal detection


Example of Blocked paradigm

Gandhi et al., 1999


First BOLD-effect experiment

Kwong and colleagues at Mass. General Hospital (Boston).

Stimulus: flashing light.


Summary 2


Case study: Vision

Vision is the most widely studied brain function

Our goals:

- analyze fundamental issues- Understand basic algorithms that may address those issues

- Look at computer implementations- Look at evidence for biological implementations- Look at neural network implementations


Eye Anatomy


Visual Pathways


Retinal Sampling


Origin of Center-Surround

Neurons at every location receive inhibition from neurons at neighboring locations.


Origin of Orientation Selectivity

Feedforward model of Hubel & Wiesel: V1 cells receive inputs from LGN cells arranged along a given orientation.


Oriented RFs

Gabor function:product of a grating anda Gaussian.

Feedforward model:equivalent to convolvinginput image by sets ofGabor filters.


Cortical Hypercolumn

A hypercolumnrepresents one visuallocation, but manyvisual attributes.

Basic processing “module” in V1.

“Blobs”: discontinuitiesin the columnar structure.Patches of neurons concernedmainly with color vision.



From neurons to mind

A good conceptual intermediary between patterns of neural activity and mental events is provided by the schema theory


From Schemas to Schema Assemblages

The Famous

Duck-Rabbit


Bringing in Context

For Further Reading:TMB2:Section 5.2 for the VISIONS system for schema-based interpretation of visual scenes.HBTNN: Visual Schemas in Object Recognition and Scene Analysis


A First “Useful” Network

Example of fully-engineered neural net that performsUseful computation: the Didday max-selector

Issues:

- how can we design a network that performs a given task

- How can we analyze non-linear networks


Winner-take-all Networks

Goal: given an array of inputs, enhance the strongest (or strongest few) and suppress the others

No clear strong input yieldsglobal suppression

Strongest input is enhancedand suppresses other inputs


Didday’s Model

= copy of input

= inhibitory inter-neurons

= receives excitationfrom foodness layerand inhibition fromS-cells

retinotopicinput


NN & Physics

Perceptrons = layered networks, weights tuned to learnA given input/output mapping

Winner-take-all = specific recurrent architecture for specific purpose

Now: Hopfield nets = view neurons as physical entities and analyze network using methods inspired from statistical physics


Hopfield Networks

A Hopfield net (Hopfield 1982) is a net of such units subject to the asynchronous rule for updating one neuron at a time:

"Pick a unit i at random.

If wij sj i, turn it on.

Otherwise turn it off."

Moreover, Hopfield assumes symmetric weights:

wij = wji


“Energy” of a Neural Network

Hopfield defined the “energy”:

E = - ½ ij sisjwij + i sii

If we pick unit i and the firing rule (previous slide) does not change its si, it will not change E.


si: 0 to 1 transition

If si initially equals 0, and wijsj i

then si goes from 0 to 1 with all other sj constant,

and the "energy gap", or change in E, is given by

E = - ½ j (wijsj + wjisj) + i

= - ( j wijsj - i) (by symmetry)

0.


si: 1 to 0 transition

If si initially equals 1, and wijsj < i

then si goes from 1 to 0 with all other sj constant

The "energy gap," or change in E, is given, for symmetric wij, by:

E = j wijsj - i < 0

On every updating we have E 0


Minimizing Energy

On every updating we have E 0

Hence the dynamics of the net tends to move E toward a minimum.

We stress that there may be different such states — they are local minima. Global minimization is not guaranteed.

B

C

A

Basin of

Attraction for C

D

E


Attractors

1. The state vector comes to rest, i.e. the unit activations stop changing. This is called a fixed point. For given input data, the region of initial states which settles into a fixed point is called its basin of attraction.

2. The state vector settles into a periodic motion, called a limit cycle.

For all recurrent networks of interest (i.e., neural networks comprised of leaky integrator neurons, and containing loops), given initial state and fixed input, there are just three possibilities for the asymptotic state:


Strange attractors

3. Strange attractors describe such complex paths through the state space that, although the system is deterministic, a path which approaches the strange attractor gives every appearance of being random.

Two copies of the system which initially have nearly identical states will grow more and more dissimilar as time passes.

Such a trajectory has become the accepted mathematical model of chaos,and is used to describe a number of physical phenomena such as the onset of turbulence in weather.


The traveling salesman problem 1

There are n cities, with a road of length lij joining city i to city j.

The salesman wishes to find a way to visit the cities that is optimal in two ways: each city is visited only once,

and the total route is as short as possible.

This is an NP-Complete problem: the only known algorithms (so far) to solve it have exponential complexity.


Associative Memories

http://www.shef.ac.uk/psychology/gurney/notes/l5/l5.html

Idea: store:

So that we can recover it if presented with corrupted data such as:

http://www.shef.ac.uk/psychology/gurney/notes/l5/l5.html


Associative memory with Hopfield nets

Setup a Hopfield net such that local minima correspondto the stored patterns.

Issues:-because of weight symmetry, anti-patterns (binary reverse) are stored as well as the original patterns (also spurious local minima are created when many patterns are stored)

-if one tries to store more than about 0.14*(number of neurons) patterns, the network exhibits unstable behavior

- works well only if patterns are uncorrelated


Learning

All this is nice, but finding the synaptic weights that achieve a given computation is hard (e.g., as shown in the TSP example or the Didday example).

Could we learn those weights instead?


Simple vs. General Perceptrons

The associator units are not interconnected, and so the simple perceptron has no short-term memory.

If cross-connections are present between units, the perceptron is called cross-coupled - it may then have multiple layers, and loops back from an “earlier” to a “later” layer.


Linear Separability

A linear function of the form

f(x) = w1x1+ w2x2+ ... wdxd+wd+1 (wd+1 = - )

is a two-category pattern classifier.

f(x) = 0 w1x1+ w2x2+ ... wdxd+wd+1 =

gives a hyperplane as the decision surface

Training involves adjusting the coefficients (w1,w2,...,wd,wd+1) so that the decision surface produces an acceptable separation of the two classes.

Two categories are linearly

separable patterns if in fact

an acceptable setting of such

linear weights exists.

A

A

A

A

AA

AA

A

AA

A

A

A

A

A

AAA

AA

BB

BB

BB

B

BBB

B

B

B

B

B

BB

B

B

B

B

B

BB

Bx1

x2

f(x) 0

f(x) < 0


Classic Models for Adaptive Networks

The two classic learning schemes for McCulloch-Pitts

formal neurons i wixi

Hebbian Learning (The Organization of Behaviour 1949)

— strengthen a synapse whose activity coincides with the firing of the postsynaptic neuron

[cf. Hebbian Synaptic Plasticity, Comparative and Developmental Aspects (HBTNN)]

The Perceptron (Rosenblatt 1962)

— strengthen an active synapse if the efferent neuron fails to fire when it should have fired;

— weaken an active synapse if the efferent neuron fires when it should not have fired.


Hebb’s Rule

where synapse wij connects a presynaptic neuron with firing rate xj to a postsynaptic neuron with firing rate yi.

Peter Milner noted the saturation problem

von der Malsburg 1973 (modeling the development of oriented edge detectors in cat visual cortex [Hubel-Wiesel: simple cells])

augmented Hebb-type synapses with:

- a normalization rule to stop all synapses "saturating"

wi = Constant

- lateral inhibition to stop the first "experience" from "taking over" all "learning circuits:” it prevents nearby cells from acquiring the same pattern thus enabling the set of neurons to "span the feature space"

xj yi

The simplest formalization of Hebb’s rule is to increase wij by: wij = k yi xj (1)


Perceptron Learning Rule

The best known perceptron learning rule - strengthens an active synapse if the efferent neuron fails to

fire when it should have fired, and - weakens an active synapse if the neuron fires when it should not have:

wij = k (Yi - yi) xj (2)

As before, synapse wij connects a neuron with firing rate xj to a neuron with firing rate yi, but now

Yi is the "correct" output supplied by the "teacher."

The rule changes the response to xj in the right direction: If the output is correct, Yi = yi and there is no change, wij = 0. If the output is too small, then Yi - yi > 0, and the change in wij will add wij xj = k (Yi - yi) xj xj > 0 to the output unit's response to (x1, . . ., xd).

If the output is too large, wij will decrease the output unit's response.


Backpropagation: a method for training a loop-free network which has three types of unit:

input units;hidden units carrying an internal representation;output units.

Back-Propagation


Example: face recognition

Here using the 2-stage approach:


Non-Associative and Associative Reinforcement Learning

Non-associative reinforcement learning, the only input to the learning system is the reinforcement signal

Objective: find the optimal actionAssociative reinforcement learning, the learning system also receives information about the process and maybe more.

Objective: learn an associative mapping that produces the optimal action on any trial as a function of the stimulus pattern present on that trial.

[Basically B, but with new labels]


Self-Organizing Feature Maps

Localized competition & cooperation yield emergent global mapping

o o o o o o o o o o o o o

o o o o o o o o o o o o o


Capabilities and Limitations of Layered Networks

To approximate a set of functions of the inputs byA layered network with continuous-valued units andSigmoidal activation function…

Cybenko, 1988: … at most two hidden layers are necessary, with arbitrary accuracy attainable by adding more hidden units.

Cybenko, 1989: one hidden layer is enough to approximate any continuous function.

Intuition of proof: decompose function to be approximated into a sum of localized “bumps.” The bumps can be constructed with two hidden layers.

Similar in spirit to Fourier decomposition. Bumps = radial basis functions.


Optimal Network Architectures

How can we determine the number of hidden units?

- genetic algorithms: evaluate variations of the network, using a metric that combines its performance and its complexity. Then apply various mutations to the network (change number of hidden units) until the best one is found.

- Pruning and weight decay:- apply weight decay (remember reinforcement

learning) during training- eliminate connections with weight below threshold- re-train

- How about eliminating units? For example, eliminate units with total synaptic input weight smaller than threshold.


Large Network Example

Example of network with many cooperating brain areas:Dominey & Arbib

Issues:

- how to use empirical data to design overall architecture?

- How to implement?- How to test?

PP

SC

vm

sm

ms

FOn

FEF

vm

sm

vs

CD

sm

vs

SNR

vs

ms

qv

TH

qv

LGN

VisCx

wta

Brainstem Saccade Generator

eye movement

RetinaVisInput

delay

FEFvsFEFms

FEFvsFEFms

swit

chPPctr

delay

FOn

Filling in the Schemas: Neural Network Models Based on Monkey NeurophysiologyPeter Dominey & Michael Arbib: Cerebral Cortex, 2:153-175

Develop hypotheses on Neural Networks that yield an equivalent functionality: mapping schemas (functions) to the cooperative cooperation of sets of brain regions (structures)

FEF

CAUDATESC

SNr

SG

VisCx

PP

MD


Low-Level Processing

Remember: Vision as a change in representation.

At the low-level, such change can be done by fairly streamlined mathematical transforms:

- Fourier transform- Wavelet transform

these transforms yield a simpler but more organized image of the input.

Additional organization is obtained through multiscale representations.


Laplacian Edge Detection

Edges are defined as zero-crossings of the second derivative (Laplacian if more than one-dimensional) of the signal.

This is very sensitive to image noise; thus typically we first blur the image to reduce noise. We then use a Laplacian-of-Gaussian filter to extract edges.

Smoothed signal First derivative (gradient)




Illusory Contours

Some mechanism is responsible for our illusory perception of contours where there are none…


Long-range Excitation


Modeling long-range connections


Depth & Stereo

C

A B

D

L R

OA OA

-qmax

-qmax +qmax

+qmax

qRqL

qLB, qLD

q0 q0

qLA, qLC qRB, qRC

qRA, qRD


Correspondence problem

Segment & recognize objects in each eye separately first,

then establish correspondence?

No! (at least not only): Julesz’ random-dot stereograms


Regularization


Higher Visual Function

Examine components of mid/high-level vision:

AttentionObject recognitionGistAction recognition

Scene understandingMemory & consciousness


Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and SummaryItti & Koch, Nat Rev Neurosci, Mar. 2001


Bre

fczynski &

De

Yo

e, Na

ture

Ne

uroscie

nce 1

999



Tre

ue &

Mar

tinez

-Tru

jillo

, N

atur

e 19

99




Challenges of Object Recognition

The binding problem: binding different features (color, orientation, etc) to yield a unitary percept. (see next slide)

Bottom-up vs. top-down processing: how much is assumed top-down vs. extractedfrom the image?

Perception vs. recognition vs. categorization: seeing an object vs. seeing is as something. Matching views of known objects to memory vs. matching a novel object to object categories in memory.

Viewpoint invariance: a major issue is to recognize objects irrespectively of the viewpoint from which we see them.




Fusiform Face Area in Humans


Eye Movements

1) Free examination

2) estimate material circumstances of family

3) give ages of the people

4) surmise what family hasbeen doing before arrivalof “unexpected visitor”

5) remember clothes worn bythe people

6) remember position of peopleand objects

7) estimate how long the “unexpectedvisitor” has been away from family




Several Problems…

with the “progressive visual buffer hypothesis:”

Change blindness:

Attention seems to be required for us to perceive change in images, while these could be easily detected in a visual buffer!

Amount of memory required is huge!

Interpretation of buffer contents by high-level vision is very difficult if buffer contains very detailed representations (Tsotsos, 1990)!


The World as an Outside Memory

Kevin O’Regan, early 90s:

why build a detailed internal representation of the world?

too complex…not enough memory…

… and useless?

The world is the memory. Attention and the eyes are a look-up tool!


The “Attention Hypothesis”

Rensink, 2000

No “integrative buffer”

Early processing extracts information up to “proto-object” complexity in massively parallel manner

Attention is necessary to bind the different proto-objects into complete objects, as well as to bind object and location

Once attention leaves an object, the binding “dissolves.” Not a problem, it can be formed again whenever needed, by shifting attention back to the object.

Only a rather sketchy “virtual representation” is kept in memory, and attention/eye movements are used to gather details as needed


Gist of a Scene

Biederman, 1981:

from very brief exposure to a scene (120ms or less), we can already extract a lot of information about its global structure, its category (indoors, outdoors, etc) and some of its components.

“riding the first spike:” 120ms is the time it takes the first spike to travel from the retina to IT!

Thorpe, van Rullen:

very fast classification (down to 27ms exposure, no mask), e.g., for tasks such as “was there an animal in the scene?”




One lesson…

From 50+ years of research…

Solving vision in general is impossible!

But solving purposive vision can be done. Example: vision for action.


Grip Selectivity in a Single AIP Cell

A cell that is selective for side opposition (Sakata)


FARS (Fagg-Arbib-Rizzolatti-Sakata) Model Overview

AIP

F5

dorsal/ventral streams

Task Constraints (F6)

Working Memory (46)

Instruction Stimuli (F2)

Task Constraints (F6)Working Memory (46?)Instruction Stimuli (F2)

AIPDorsalStream:Affordances

IT

VentralStream:Recognition

Ways to grab this “thing”

“It’s a mug”PFC

AIP extracts the set of affordances for an attended object.These affordances highlight the features of the object relevant to physical interaction with it.


Visual

Cortex

ParietalCortex

InferotemporalCortex

How (dorsal)

What (ventral)

reach programming

grasp programming

AT and DF: "How" versus "What"

“What” versus “How”: AT: Goodale and Milner: object parameters for grasp (How) but not for saying or pantomimingDF: Jeannerod et al.: saying and pantomiming (What) but no “How” except for familiar objects with specific sizes.

Lesson: Even schemas that seem to be normally under conscious control can in fact proceed without our being conscious of their activity.


Documents

Itti: CS564 - Brain Theory and Artificial Intelligence. Overview and Summary Itti: CS564 - Brain Theory and Artificial Intelligence University of Southern