Computational and Theoretical Neuroscience

Computational and Theoretical Neuroscience

Krishnamoorthy V. Iyer

Department of Electrical Engineering, IIT Bombay

August 22, 2011

Krishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 1 / 169

Outline

1 IntroductionBrain as a Computer(Human) Brain: Neuroscience Basics 101Outline of Lectures 1 and 2Computational and Theoretical Neuroscience

2 Computer Vision and (Visual) Computational NeuroscienceComputer Vision and Visual Neuroscience BasicsPrimary Visual Cortex and Low-Level Psychophysical Visual PhenomenaOrganization of Part IIBasics of Retina and V1 Physiology and ArchitectureVisual Field Representation in V1 - the Complex Log MapComputation in Primary Visual Cortex

How V1 may do stereopsis

3 Single Neuron ComputationSingle Neuron BasicsSingle neuron as a ”computer“


Introduction

The (Human) Brain


Introduction

Remember this is a presentation, NOT a paper Keep the numberof references to a minimum Not more than ten references overe 2talks, average of 5 per talk Sejnowski and Churchland, Schwartz,Schwartz and Yeshurun, David Marr, Laurence Abbott and PeterDayan, William Bialek Spikes, one or two papers of Bialek andco-workers Maybe one or two websites This also reduces theamount of work


Introduction Brain as a Computer

Theme of this lecture: Brain as a “Computer”

Question: Is the brain a “computer“ in the sense that a Pentium or aMacintosh is? Obviously not!

Calling the brain a ”computer“ is a metaphor ....

Like describing an electron as ”both” a “particle“ and a “wave“ - themetaphor refers to the use of the mathematical techniques ofwave equations as well as properties associated with particles, such asposition, for instance



What the brain as a ”computer“ metaphor refers to

The brain as an information processor,

How neural architecture and dynamics represents information (neuralcode) and processes information (neural computation)

Mathematical description i.e. building mathematical models ofneural architecture, dynamics, development, and neuralrepresentations and computation



What is a computational explanation of a physical event?

Refers to the information content of the physical signals

How the information is used to accomplish a task

Not all physical events have information content: Theirdescription in terms of causation and the laws of physics suffice togive an ”understanding“ of the phenomenon

Some do: When we type numbers into a calculator and receive ananswer



What is a computational explanation of a physical event?

Refers to the information content of the physical signals

How the information is used to accomplish a task

Not all physical events have information content: Theirdescription in terms of causation and the laws of physics suffice togive an ”understanding“ of the phenomenon

Some do: When we type numbers into a calculator and receive ananswer

These require an explanation that describes the computation, and notmerely one at the level of dynamics

Better ... an explanation that maps between the two levels [?],


Introduction (Human) Brain: Neuroscience Basics 101

The Human Brain: Major Regions

Before studying the brain as a ”computer“, we need to know somethingabout the brain

Much of the brain is divided into two hemispheres, a left and a right



Connecting link is a bundle of nerves: corpus callosum



The (Human) Brain: Major Regions ContdCortex and Thalamus

(Cerebral) Cortex:

In popular parlance, the cerebrum “is” the brain.Cortex is Latin for “bark” or “outer rind”Seat of all higher mental processes:

Sensory Information ProcessingMovementLanguage (Speech and Comprehension)Reason and Empathy

Hugely developed in mammals.

Thalamus: Cortex’s

Mini-Me: 1-1 correspondence with cortical regionsGateway: 90 percent of information to cortex (except smell) routedthrough thalamus



Major Regions ContdHippocampus

Hippocampus: (Episodic) Memory: Where were you and whatwere you doing when the World Trade Center was destroyed?

Long-Term Memory: Autobiographical details

Not crucial for short-term memory

Patient HM

Patient Clive Wearing

“Fifty First Dates” starring Drew Barrymore

“Ghajini” starring Aamir Khan



Major Regions ContdHippocampus

Hippocampus: (Episodic) Memory: Where were you and whatwere you doing when the World Trade Center was destroyed?

Long-Term Memory: Autobiographical details

Not crucial for short-term memory

Patient HM

Patient Clive Wearing

“Fifty First Dates” starring Drew Barrymore

“Ghajini” starring Aamir Khan

Does not deal with muscle memory.For ex: Riding a bicyle, dancing, playing an instrument





Other sub-cortical structures

1 Hypothalamus: Basic biological needs/drives:2 Basal Ganglia: Motivation aka “The Will”

affected in Parkinson’s disease

3 Cerebellum: Fine motor control: Playing the violin (but learninghow to play involves the motor cortex)

4 Amygdala: Experience of fear: Famous case of a woman patientwho literally did not experience fear as a consequence of damage tothe amygdala (Damasio)



Focus of this talk: (Primary Visual) Cortex

Historically, neuroscientists have considered two kinds of divisions ofcortical regions

1 Anatomical:

2 Functional:



CortexAnatomical Divisions

Note the colors are purely for visual appeal!

The cortical surface is a dull grey (hence, “grey matter”)



CortexAnatomical Divisions

Anatomical divisions include:

1 Occipital: (almost exclusively) Vision

2 Temporal: Vision, (also Hearing i.e. Audition)

3 Parietal: Body Sense (somatosensory) and multisensory modalityintegration (including Vision)

4 Frontal: Reason and Empathy (Psychopaths are suspected to haveproblems with neural circuitry in this region).



Functional Divisions



CortexFunctional Divisions

Functional divisions include:

1 Visual Cortex: Vision - more than 50% of the cortex in humans andprimates devoted exclusively to vision

2 Auditory Cortex: Hearing3 Somatosensory: Body sense and touch

Phantom Limb phenomenon

4 Motor Cortex: (Learning to) Dance, (Learning to) Play anInstrument



Cortex and Thalamus Relation

Mini-Me

Gate-Keeper

area marked in red is the thalamus



Recurring Theme

Do anatomical divisions correspond to functional divisions?



Brain as a “Computer”Some Misconceptions

The architecture of the brain has very little in common with thevon Neumann architecture

The circuitry consists of multiple (and overlapping) spatial scales oforganization

Likewise, the dynamics consists of multiple (and overlapping)temporal scales of activity



Brain as a “Computer”Some Misconceptions Contd

No CPU, and hence no homunculus

The frontal cortex may be said to come closest to the notion of a CPUNo “homunculus”:

Hence the naive theories of visual perception as a TV in the brain mustbe discardedRepresentation of information in the brain becomes a fundamentalproblemThe left hemifield of vision projects to the right cerebral cortex, andvice-versa the right hemifield of vision

The hippocampus comes closest to notion of a hard drive

So what does neural architecture look like?

What about neural dynamics?



Levels of Organizationcorrespond closely to Spatial Scales

1 (Large) Molecules: Ion Channels (Proteins) 10−4 micrometer

2 Synapses: Scale: 1 micrometer

3 Single Neuron: Scale: 100 micrometers (Neuron is the primary braincell involved in signaling)

4 Networks (aka Microcircuitry) Scale: 1 millimeter

5 Maps: Scale: 1cm Histologically well-defined areas within a brainstructure: Ex: Striate Cortex

6 Systems: Scale: Major anatomical regions: Ex: Cortex, Hippocampus

7 Central Nervous System (CNS) 1 meter8 Focus on levels 3, 4 and 5:

Map level ((Primary) Visual) Cortex, level 5Single Neuron, level 3Networks of neurons, level 4



Other kinds of spatial organization

1 Laminar i.e. layered structure:

All regions of cortex, except the region devoted to smell, have 6 layers.There is cross-connectivity b/w the layers as well as within the layers

2 Topographic organization and Self-similarity in the map fromretina to cortex: concatenated complex log map



Other kinds of spatial organization

1 Laminar i.e. layered structure:

All regions of cortex, except the region devoted to smell, have 6 layers.There is cross-connectivity b/w the layers as well as within the layers

2 Topographic organization and Self-similarity in the map fromretina to cortex: concatenated complex log map

3 Columnar organization



Temporal scales

Events of interest have time scales ranging from:

1 10−3 seconds Ex: generation of an action potential

2 10 years Ex: developmental changes associated with the life of theorganism

3 and many time scales in-between


Introduction Outline of Lectures 1 and 2

Outline of this lecture

1 Part I: What is computational and theoretical neuroscience?

How computer scientists can contribute to computational andtheoretical neuroscience?

2 Part II: (Primary) Visual Cortex: Hero: Eric SchwartzVisual Cortex is the part of the brain devoted to visual informationprocessing Attention: CS475, CS675, CS663


Introduction Outline of Lectures 1 and 2

Outline of the next lecture

1 Part III: Single Neuron: Star: William Bialek

A principle in much of Bialek’s work is that neural systems (andbiological systems more generally) may have reached optimumperformance limits under the constraints of evolutionary history, andnoise and energy consumption limits Attention: CS435 and CS709

2 Part IV: Networks of Neurons: Laurence Abbott, Nancy Kopell

3 Part V: Other neuroscientists whose work I would encourage you toexplore

4 Part VI: Conclusion and Discussion: CS students interested inNeuroscience


Introduction Computational and Theoretical Neuroscience

Computational and Theoretical neuroscience: What it isNOT

1 “Branch” of Biology, but overlaps with it2 “Branch” of Neurology: Study of diseases of the brain: properly a

branch of medicine.

Sridevi Vedula Sarma, PhD, EECS, MIT, LIDS, now at Johns HopkinsNandini Chatterjee-Singh, PhD, Physics, University of Pune, now atNBRC, Gurgaon

3 “Branch” of Neurosurgery But has an important supporting role toplay



Computational and Theoretical neuroscience: What it isNOT Continued

1 Artificial Neural Networks Some examples: McCulloch-Pittsneuron, Rosenblatt’s Perceptron, Hopfield networks, Boltzmannmachines, backpropagation

2 Artificial Intelligence Goals somewhat similar: How to performcomputational tasks such as object recognition following a”rules-based” approach

3 But in CTNS the emphasis in is on figuring out how the braindoes it



Potential issue with the goals of computationalneuroscience

1 Maybe trying to figure out how the brain does a task is like trying tofigure out how birds fly w/o knowing the principles of aerodynamics?

2 Solution: C and T neuroscientists often maintain close connectionswith these fields



Computational and Theoretical Neuroscience: What it is

The brain as an information processor, taking into account

biological imperatives and history: purpose, development, evolutionaryhistoryphysical constraints: energy consumption, noise in sensors and networkelements, processing speed requirements

Interesting possibility: Evolutionary pressures may have caused neuralsystems to have achieved some kinds of optimizations. (WilliamBialek). (At least local, if not global, optima in the landscape ofevolutionary possibilities - my take).

As calculus is integral to Physics, and algorithms to ComputerScience, so the mathematical techniques of computational andtheoretical neuroscience are an integral part of the subject.



Scientists contributing to Computational and TheoreticalNeuroscience include

1 Physicists: Both theorists and experimentalists2 Engineers:

Electrical Engineers, especially signal processing, communication,control, computing, and VLSI, fields of EE that deal with informationprocessingNote that EE types in other areas and Chemical Engineers andMechanical Engineers also can play a role, with some retraining

3 Computer Scientists:

4 Mathematicians: Mathematicians interested in studying the brainare almost by definition applied mathematicians

5 A small but growing number of scientists from traditionallynon-mathematical scientific disciplines such as biology,biochemistry, psychology, and medicine who are willing and able tostudy mathematical techniques from Physics, Engineering, and CS



How CS types can contribute to computational andtheoretical neuroscience

Computer and Machine Vision meets Biological Vision: (ArtificialIntelligence meets (Visual) Neuroscience)

Ex: Eric Schwartz’s group

What are the algorithms the brain uses to do visual processing?

Machine Learning: Analysis of large data sets. Curse ofdimensionality

Theoretical Computer Science: Computational Learning Theory,Computational Complexity, Vapnik-Chervonenkis (VC) Dimension

Attention: CS435, CS475, CS675, CS663, CS709



Brief note about the term “Computational Neuroscience”

Eric Schwartz introduced the term ”ComputationalNeuroscience“

Eric Schwartz describes his areas of interest as ComputationalNeuroscience and Computer Vision



Brain as a “Computer”: Two themesRepresentation of information and Transformation of Information aka Computation

1 Theme I: Representation of Information:1 Map level: Mathematical techniques used: Complex analysis and more

specifically, numerical conformal mapping and computationalgeometry

2 Single neuron level: Mathematical techniques used: Information theory,machine learning, and possibly theory of computation

2 Theme II: Transformation of Information: Computation1 Single Neuron Level: Theory of computation,

Statistical/Computational Learning Theory2 Network Level: same as above, also graph theory and the theory of

network flows


Computer Vision and (Visual) Computational Neuroscience Computer Vision and Visual Neuroscience Basics

(Computer/Machine) VisionBackground for Part II

The study of computer/machine vision has traditionally been divided into:

“Low-level“ Vision: Estimating the scene from the image

Image: Raw pixel values






Image: Raw pixel valuesScene: Interpreting the image to obtain some (comparatively basic)information Example: edge detection

”High-level“ Vision:

Object recognition and classification






Image: Raw pixel valuesScene: Interpreting the image to obtain some (comparatively basic)information Example: edge detection

”High-level“ Vision:

Object recognition and classificationVisual cognition and volition: extraction of shape properties andspatial relations while performing tasks such as manipulating objectsand planning movements



Lecture Theme: How the Brain does VisionVisual Cortex and Visual Field Representation

Thalamus (marked LGN) plays gatekeeper’s role

Information to the cortex (marked occipital poles) is routed viathalamus (LGN)



Visual Field Representation

Each visual hemifield projects onto nasal (’nose’) hemiretina of thesame side eye and to the temporal (’temples’) hemiretina on theopposite side

Left visual hemifield projects onto

nasal hemiretina of the left eyetemporal hemiretina of the right eye

Take home message: Each visual hemifield is represented byneurons on opposite hemisphere visual cortex



Cortical Hemisphere Input

Each cortical hemisphere gets input from:

one hemifieldboth eyes

Important to the functional (computational) role of ocular dominancecolumns (see below)



Visual Field Representation

We will come back to the important issue of representation of (visual)information at a later stage

Prerequisites:

Receptive FieldsRetinotopyCortical magnification

But before we study these matters, we try to obtain a broad overviewof vision as performed by the brain



Vision and Visual Cortical Processing

”Low-level“ vision - ”Early“ stages of visual cortex

”High-level“ vision - ”Later“ stages of visual cortex

Distinction not sharp (large number of ”top-down“ and”bottom-up“ connections in the brain).

”top-down“ refers to connections from ”higher“ to ”lower“ corticalareas of from cortex to subcortical structures. Ex: V2 to V1, cortex tothalamus”bottom-up” refers to connections from “lower“ to ”higher“ areas Ex:thalamus to V1, V1 to V2



Lecture Theme: How the Brain does VisionVisual Cortex and Visual Information Processing

Low-level Vision: Primary Visual Cortex





Edge detection: What is an edge in an image?Contour detectionIllusory contours





Edge detection: What is an edge in an image?Contour detectionIllusory contoursMotion detection/estimation

High-level Vision:

Object recognition and classification: Inferotemporal (IT) Cortex(”what“ aka ”perception“ stream of vision)Spatial perception, navigation and attention: (Posterior) ParietalCortex (”where“ or ”how“ aka ”action“ stream of vision)



”What“ and ”Where“or ”How” Streamsaka “Perception” and “Action” streams

Purple: “What” stream, processed in inferotemporal (IT) cortex.Object recognition

Green: “Where” or “How” stream, processed in posterior parietalcortex. Spatial Perception and navigation

The classification is controversial and has been called into question


Computer Vision and (Visual) Computational NeurosciencePrimary Visual Cortex and Low-Level Psychophysical Visual

Phenomena

Primary Visual Cortex (V1) and Low-Level Vision

Image Scene

edge detection pixel values edgecontour detection pixel values contourillusory contours pixel values illusory contourshape from shading pixel val-

ues(luminanceinformation)

shape

figure-ground segmentation pixel values assigning contour to one of twoabutting regions

stereopsis one imageframe fromeach of tworetinae

est. rel. dist. to objects basedon lateral displacements b/wsuperimposed images

motion estimation two succ.image frames

extract motion info

super-resolution low-freq.image

estimated is a high-freq. image



Phenomena

Task VisualCortex Area

Cell Type

edge detection V1 Simplecontour detection V1 Complexillusory contours V2shape from shading pixel values

(luminanceinformation)

shape

figure-ground segmentation pixel values assigning contour to one of twoabutting regions

stereopsis V1 Monocular and Binocular /Hypercolumn

motion estimation V1 Complex



Phenomena

Kanizsa TriangleIllusory Contours

Believed to take place due to neurons in V2



Phenomena

Task Visual CortexArea

Structures

stereopsis V1 Monocular+Binocular Cells /Hypercolumn

Image representation V1 complex log map

Evidence from psychophysics, imaging studies and neurophysiology

Theoretical and modeling studies indicate that these processes take place inthe primary visual cortex aka V1 and V2


Computer Vision and (Visual) Computational Neuroscience Organization of Part II

Organization of Part IIArchitecture of V1

Neuronal Tuning and Receptive Fields

Orientation Columns

Ocular Dominance Columns

Hypercolumn

Ice-Cube Model

”Pinwheels“

Topographic map (Complex log map) from retina to V1


Computer Vision and (Visual) Computational Neuroscience Organization of Part II

Organization of Part II Contd

Visual information representation:

Retinotopy and Topographic OrganizationComplex Log Mapping between retina and visual cortexApplication to Machine Vision: Space-Variant Vision

Psychophysics: How does the brain perceive depth?

StereopsisPanum’s AreaHypercolumn’s role in Stereopsis

Psychophysics: How does the brain perceive edges?

What is an edge in an image?Different kinds of edges in an imageEdge-detection algorithmsHow the brain processes edges


Computer Vision and (Visual) Computational Neuroscience Basics of Retina and V1 Physiology and Architecture

Neuronal Tuning I

Neuronal tuning curve: Graph of the average firing rate of theneuron as a function of stimulus parameters relevant to that classof neurons

A typical tuning curve (top) looks like a half-wave rectified cosinefunction



Neuronal Tuning II

Interpretation: Stimuli that cause the neuron to fire at the highestpossible rate are considered to be most important

If neurons encode information in their average firing rate, then this islikely to be true

However, this is open to question (as we will see in the next lecture)



Receptive Fields (RFs) in Visual Processing

Visual receptive fields refer to the region of (visual) space in which anappropriate stimulus will cause the neuron to respond

Appropriate varies b/w regions: from retina to V1 to V2 to V4 and IT

We consider RFs in retina and V1

In ”higher“ visual cortical regions,

RF sizes increaseappropriate stimuli become more complexIn IT cortex, ∃ cells that respond when a person recognizes faceCalled face-recognition cells



Receptive Fields in Retina

RFs in retina come in two types

On−SurroundOff−Center On−Center

Off−Surround

Retinal Cells Receptive Fields

The spatial structure of retinal (ganglion) cell RFs is well captured bya difference-of-Gaussians model



Receptive Fields in Retina Contd

Why two different kinds? Won’t a single kind suffice?

The answer lies in energy efficiency

The brain is an extremely energy-hungry organ

brains typically weigh about 2% of body weightedconsume about 20% total body oxygenutlise about 25% body glucose

To signal swings in both directions about zero contrast, thespontaneous firing rate must be high.

This would cause energy consumption to rise



Receptive Fields in V1Orientation Selectivity

Hubel and Wiesel discovered nature of RFs in V1

1981 Nobel Prize in Physiology or Medicine

Retinal cells respond to light or dark ”spots”

Individual V1 Cells are tuned to “edges” of a particular orientation.



Simple Cells RFs

0 deg

90 deg

180 deg

360 degSimple cell RF orientations othersnot shown

0, 90, 180, 360 deg

Only cells with orientation preferences of 0, 90, 180, 360 degrees havebeen shown for the sake of clarity

Cells with intermediate orientation preferences have not been shown



Simple Cells Contd

Every individual cell has an orientation preference (neuronaltuning)

Population as a whole has cells whose preference ranges from 0 to360 degrees

distinct excitatory and inhibitory regions

linear summation (superposition) property

regions are anatgonistic - if the light is diffuse, then the cell does notrespond

Different simple cells respond to different orientations

Receptive field sizes are much smaller than Complex Cells (see below)

May act as edge detectors at different scales



Complex Cells

Similar to simple cells: also have orientation preferences

Complex cells have larger receptive fields, so some spatialinvariance

May act as movement detectors

Also complex cells receive inputs from many simple cells, may act todetect contours in an image, enabling figure-groundsegmentation at early stages of visual processing

I encourage you to explore the work of:

Stephen Grossberg and collaboratorsZhaoping Li

Hypercomplex Cells

Using Gabor type filters to model the RF of simple cells by



Take Home Message: (Visual) Neuroscience inspiringComputer Vision

∃ cells in V1 that are tuned to distinct orientations

The population as a whole has cells responsive to every orientationfrom 0 to 360 degree

For each orientation, ∃ cells with different RF sizes

These may detect edges of the same orienation but at different scales

This inspired Jones and Malik multi-orientation multi-scale(MOMS) filters used to do stereo (more on this later)



Other computational theories that have been proposed forthe role of simple and complex cells

(Visual) Texture analysis: Process by which visual system definesregions that differ in the statistical properties of spatial structure

Matt or gloss finishWavy or straight or curly hair

Structure from Shading: how information about the curvature ofsurfaces can be extracted from changes in luminance due to depthstructure in the image



Orientation Columns

Cells tuned to a particular orientation are arranged to form ofcolumns, called orientation columns

Cells tuned to nearby orientations are arranged next to each other

∃ cells responsive to all orientations



Ocular Dominance ColumnsRole in Stereopsis

Some cells in V1 respond mainly to i/p from left eye,

Others to i/p from the right eye

A few driven by both: so-called binocular cells

These cells did not seem to be tuned for binocular disparityMany of these studies use somewhat ad hoc methods of classification,so terms such as monocular and binocular must be treated with caution

Monocular cells arranged to form columns called ocular dominancecolumns

Known to play a role in binocular vision and stereopsis.



Relationship b/w Ocular Dominance and OrientationColumnsHypercolumn

Ocular dominance and orientation columns run almost orthogonal toeach other

Two adjacent ocular dominance columns (each containing cellsresponsive to all orientations) form a hypercolumn

Hypercolumn def as “a unit containing full set of values for any givenset of RF parameters”.



HypercolumnTheme: Relationship b/w Anatomical and Functional Modules

In human V1, a hypercolumn is about 2mm, forms a basic anatomicalmodule

Yeshurun and Schwartz show that it corresponds to a functional i.e.psychophysically measurable module

a biologically plausible algorithm instantiated on a hypercolumn thatsolves stereopsisconsistent with psychophysical data



Ice-Cube Model of V1

L

R

ICE−CUBE MODEL OF V1

1 hypercolumn

repeated many times

L and R are the ocular dominance columns due to the left and righteyes resp.

the orientation columns are seen to run perpendicular to the oculardominance columns



Short digression: Pinwheels in V1

Applying topological reasoning, Schwartz and co-workers showed thatthe ice-cube model must be modified

∃ singularities in the map of orientations: orientation vortices aka“pinwheels”

Introduce a figure shoring pinwheels of orientation selectivity



Lecture 2 begins here



Story so far ...

Ice-Cube model

Qualitatively, each cortical hemisphere receives input from

the opposite hemifield onlyboth eyes

Next Step: Quantifying and mathematizing this description of visualfield representation in the visual cortex


Computer Vision and (Visual) Computational Neuroscience Visual Field Representation in V1 - the Complex Log Map

Visual field representation in V1Retinotopy and topographic organization

Retinotopy: Spatial organization of neural responses to visual stimuli

Retinotopic maps: Special case of topographic organization

Topographic organization: Projection of a sensory surface ontostructures in the central nervous system (CNS).Examples:

Somatosensory Map: Body surface (skin) to the somatosensory cortexRetinotopic Map: Retina to V1

Neighboring points in the sensor (skin or eye) activate (usually)neighboring regions in the (somatosensory or visual) cortex



Digression: Topographic organization of touch:Somatosensory map

Cortical hemisphere surfaceBody regional surfaces mapped as aboveNote the large amount of space devoted to hands, face, lips, andtongueSkilled pianist/violinist - increased region to fingersSkilled dancer - increased region to legsPhantom limb and/or Neural plasticity



non-Uniform sampling and non-uniform representation

Intuitively obvious: Lips (as all of us are presumably aware) and fingertips (as Braille readers know) are more sensitive surfaces than (say)the back.

Demonstrated by experimental studies.

But why?

Reason: More sensors per unit area of the fingers than the back

Consequently, non-Uniform:

Sampling at the sensor surfaceRepresentation in the cortex



Eric Schwartz

Eric Schwartz introduced the term ”Computational Neuroscience“

Eric Schwartz and co-workers’ mathematical models of corticalneuroanatomy one of the most successful quantitative models inNeuroscience

His ouevre demonstrates an intense preoccupation withstructure-function (computation) relationships in Neuroscience

In other words, ’’how the brain does a task“



Topographic organization in visionComplex log map from retina to V1

Retinotopic map: Mathematical transformation is a complexlogarithmic map

Mathematical description of retinotopic map was discovered by EricL. Schwartz



Complex log map from retina to V1 Part II

w = log(z + a), where

w represents position on cortical surfacez is a complex variable representing azimuthal angle, θ andeccentricity, ǫ on retinaz = ǫe iθ

Fε

F fixation point

θ

a is an experimentally obtained parameter

measure of foveal sizediffers from species to species



Cortical magnification factor

(Magnitude of) derivative of w w.r.t z called (cortical)magnification factor

|dwdz

| = | 1z+a

|

Mathematically:

Centre (fovea): z << a, so | dwdz| ≈ constant, (map approx linear)

Periphery: z >> a, so δwδz

≈ 1ǫ, (map approx logarithmic)



Physical meaning and biological significance of the corticalmagnification factor

Physical meaning:

As eccentricity ǫ increases, magnification decreasesFoveal magnification (and thus sampling) greater than peripheryConsequently, foveal representation (much) greater than peripheryrepresentation in cortex

Biological significance:No. of neurons ’responsible’ for processing a stimulus of a given size,as a function of location in visual field decreases in the peripheralregions



Complex log mapLimitations and Extensions

Excellent fit for foveal region across many species

Not a good fit for peripheral retina

More sophisticated models (numerical conformal mapping, dipolemap, wedge-dipole map), have been developed by Schwartz andco-workers for

peripheral retinal regionsother visual cortical regions (V2, V3)



Visual field representation in cortex’: Practical implicationsTheme: Neuroscience’s impact on Computer Vision and Robotics I

non-Uniform sampling (Somatosensory map also shows non-uniformsampling and representation)

Small portion of visual field sampled at high resolutionCoarser sampling at periphery

non-Uniform representation

Consequence: Far more processing resources i.e. neurons in cortex aredevoted to objects at the center of gaze



Theme: Neuroscience’s impact on Computer Vision andRobotics II

Eric Schwartz pointed out that a correspondence could be made b/w:Cortical RF size variation Multiple Resolution

Surface of cortex ”Horizontal“Single Point in visual field ”Vertical“

∃ cells in hypercolumns that respond to the same RF location, but havedifferent RF sizes

These have inspired ”multiscale” or ”pyramid“ approaches to computervision



Theme: Vision Neuroscience’s impact on Computer Visionand Robotics

Summary:

non-Uniform sampling and representationand multi-scale resolution aka ”pyramids“ have

inspired space-variant computer vision architectures, savings incomputational requirements, and thence to economy



Information representation and Information Transformationaka Computation

Complex log map: Information representation (at the level of maps)

Now: Information transformation aka computation in the visual cortex



Short digression: Nature versus Nurture debate

Nature versus nurture debate:

Respective roles played by the environment versus innate developmentrules (genetics and the womb/egg)

Many animals, especially reptiles, are born with astonishing survivalskills

Newborn mammals are largely helpless and usually spend theirchildhood learning

Humans have a very long period of childhood even compared withother mammals

Learning period typically lasts until mid-adolescence at least



Nature versus Nurture IIRepresentational Learning

Retinotopic mapping: Information representation in V1

Question: Are these representations

learned (due to nurture i.e. environmentally determined) orinnate (due to nature i.e. developmentally determined)?

Topographic mappings ”largely“ ”innate“.

Learning does play a role here, ∵ if one eye is damaged around birth,the OcuDom columns due to that eye will not form

For learning about (non-innate) neural representations, a goodstarting point is the classic textbook:Title: ”Theoretical Neuroscience‘‘Subtitle: Computational and Mathematical Modeling of NeuralSystemsAuthors: Peter Dayan and Laurence F. Abbott


Computer Vision and (Visual) Computational Neuroscience Computation in Primary Visual Cortex

CS213 in your Brain!!!

Attention: CS213 Data Structures and Algorithms

CS students: Symbiotic relationship b/w data structures andalgorithms

Good DSs enable elegant algorithmic solutions to a desiredcomputation

Historically significant example follows



A historically significant Data Structure

Arithmetic w/ Hindu vs. Roman numerals

Data Structure used by Hindus involved

place-value notationbase-10 representationthe zero as placeholder

DS enabled simplified algorithms to do arithmetic operations such asaddition (+), subtraction (-), multiplication (*) and division (/)



What function/computation could the ocular dominancecolumns serve?

Slide title an example of a broader question regarding the visualcortex as a whole

Comp.Sc

Brain

info. representation DS OcuDom columns inV1

info. transformation (computation) algorithm windowed cepstralfilter (suggested)

cepstral filter:



Why is the world perceived as 3-D?Why do we not perceive the world as 2 2-D images?

Images projected onto the retinas are 2-D (i.e. flat, lacking”thickness”)

But we do not (visually) perceive the world as two nearly identical(but slightly laterally displaced) and flat images

So where does the perception of ”depth“ come from?



3-D movies and 3-D glasses

3-D movies

Chota ChetanAvatar

3-D glasses fuse the two images into one to give depth perception

In day-to-day life, the brain automatically fuses the 2 2-D imageson the retina, so we perceive the world as one 3-D

How does the brain do it?



Depth Perception

1 Depth perception refers to the perception of “solidity“

2 Brain uses many different cues to perceive depth

3 Cues may be monocular (single eye) or binocular (both eyes)4 Monocular Cue: Geometric information such as occlusion used to

compute depth

Occlusion refers to the hiding of one surface by another due to opacityof the intervening surface.Provides info re relative depth

5 Here we consider only one depth clue, a binocular clue calledstereopsis

6 Stereopsis is closely connected with binocular disparity



Stereopsis

1 “Stereo“ ”solid“ or ”three-dimensional“

2 ”Opsis“ View or Sight

3 Stereopsis means ”three-dimensional vision”

4 Slightly misnamed phenomenon

5 ∵ Stereopsis refers to one of many ways humans perceive depth

6 Stereopsis: Perception of depth arising from the slightly differentprojections of the world to form two slightly different retinalimages

7 Only images themselves are used to perceive depth (see Random dotstereograms)

8 Other cues - such as occlusion - not used



Binocular Disparity

Difference in image location of an object as seen by left and right eyes

Arises due to the separation b/w the two eyes

Left Eye Right Eye

CP F

F

P

C

C

PF

C closer points, crossed disparity

P point of fixation, no disparity

F further points, uncrossed disparity

Binoc. disp. used to extract depth info from the two two-dimensionalretinal images in stereopsis.



Computing disparity

Retinal pixels do not come with “labels“ indicating correspondingpixels in the images

∴ binocular disparity b/w images must be computed from the imagesthemselves - first step in any stereo algorithm

Algorithm will have to figure out corresponding points b/w theimages.

Then disparity can be easily computed



Binocular disparity, ”filling-in“, and stereopsis

Input Data (aka “Image“); Pair of “almost” “identical” images, onelaterally displaced from the other.

Intermediate step: Compute disparity i.e. lateral displacementb/w the images

Output: Stereopsis or a perception of depth

Final Step: following computation of disparity, ”fill in“ smoothsurfaces.

We will consider an algorithm for signaling disparity that the brainmay perform.

We do not discuss the step of ”filling in“



Why do we not perceive the world as 2 2-D images?First (incomplete) solution

Because we focus our eyes on a point.

Shortcoming of this answer: circle (sphere) of fixation (horopter)



Horopter

The two eyes, and the point of fixation determine a circle called thehoropter

(Recall from school geometry that 3 points determine a circle)



Horopter: Significance

Points on horopter cast images at exactly corresponding pts inthe two retinae

Can be fused into one image w/o difficulty when the optical pathsfrom the eyes cross in V1

Points to the front and back cause images on the retinae that arelaterally displaced

Consequence: Except for a sphere of infinitesimal thickness, (almost)all points in visual field should give rise to double images.

Yet we do not perceive the world that way! Why?

Answer: Cortex performs a computation that

fuses the two images into one perceptin doing so, creates an illusion of depth



Where does fusion take place?

Fusion can occur only if the optical info from two eyes’ imagesconverge.

In the retina and (visual) thalamus, the paths are separate

Convergence happens in area V1



What comes next .....

Human Stereo: Perceptual Aspects

Stereopsis: Formal statement of the computational problem(w/o ref to how the brain does stereo vision)

Random dot stereograms and their significance

Overview of suggested algorithms (w/o ref to how the brain doesstereo vision)

How the brain (V1) does stereopsis



Perceptual Aspects of Human Stereo Vison IPanum’s Fusion Area

Region around horopter where disparate images can be perceptuallyfused into a single objectPsychophysical data s.t. area increases (linearly) with eccentricity, ǫ



Perceptual Aspects of Human Stereo Vision IIRobustness of Stereo Perception

Human stereo vision is robust to many kinds of imagemanipulations:

image degradations (including Gaussian blur, random noise, changesin image intensity),rotations (upto 6 degrees),and differential expansions (upto 15%)

A complete psychophysical theory of stereoscopic depth perceptionshould explain these perceptual data re Panum’s area and robustness



Stereopsis: The Computational Problem

Stereopsis includes

Matching corresponding image points in the two eyes (so-calledCorrespondence Problem)measuring their disparity,from this info. recovering the 3-D structure of the objects seen by theviewer (stage of “filling-in”)

Human stereopsis is also robust to minor image manipulatons of oneor the other image, as seen above



StereopsisThe Correspondence Problem

Matching corresponding points could be done either

after recognizing objects - easy and unambiguous computationbefore recognizing objects - using only disparity information

Physiological evidence indicates that binocular cells lie in V1 (muchbefore object recognition takes place in IT cortex)



Random dot stereograms aka Magic-Eye stereograms

Magic Eye stereogram: p. 215 Vision: From Photons toPerception



Random dot stereogramsSignificance

Showed brain can compute stereoscopic depth w/o:

Objects

Perspective

Cues available to either eye alone i.e. monocular cues such asocclusion



Radha Krishna image: Not a pure random dot stereogram

Same Principle

We can only form the percept after fusing the two images



Stereopsis: The Computational ProblemSuggested Algorithms

Before studying how the brain does stereopsis (we don’t really know)...

We overview various proposed computational/algorithmic solutionsand their limitations from a:

a computational perspectivein terms of biological implementability i.e. whether the brain couldactually use the proposed solution



Stereo algorithmsCategories

1 Pixel-based schemes

2 Edge-based schemes

3 Area-based schemes

4 Filtering algorithms



Stereo algorithmsCategories II

Szeliski and Zabih have proposed another classification:

1 Correlation-style stereo algorithms:Example of an (unconventional) correlation-based algorithm that isbiologically plausible

2 Global methods

Use well-chosen energy minimization to obtain a depth map thatminimizes some energy functionMinimization can be done by means of simulated annealing, graphcuts, or mean field methodsOne global method that does not use energy minimization is due toZitnick and Kanade



Stereo AlgorithmsCategories III

Stereo algorithms can also be classified based on whether algorithmused is:

1 sequential (uses iterations or relaxation, such as simulated annealing):Difficulty: Speed w/ which humans do stereo too fast for iterationschemes for computing stereo

2 parallel

We will consider a windowed (noncanonical) correlation-basedone-shot parallel algorithm called the windowed cepstrum due toYeshurun and Schwartz



Algorithms for Stereo Vision IPixel-Based Schemes

Algorithms for computing stereo can be categorised into:

Pixel-based algorithms:Example: 1st Marr-Poggio algorithm (1979)

Match individual pixels in left and right images.Computational difficulty:

Correspondence problem: which features in one (retinal) imagecorrespond to which features in the otherNOT robust to minor changes other than simple shifting (unlikehuman stereo vision)

Biologically implausible -

Only in retina are RFs “pixel-like‘‘.But stereopsis requires convergence of info. from both eyes, cannottake place in the retina, ∵ images separateOTOH, in V1, edge detection would have already taken place.∴ unclear why the brain would use pixels to do stereo



Algorithms for StereoEdge-Based Schemes

Edge-Based Algorithms: Marr and Poggio also presented anedge-based algorithm for stereo.

Algo attempts to match edges in the two images rather than pixelsComputational Issues:

Correspondence problem not as bad, ∵ (usually) far fewer edgesthan pixels in an image.OTOH, algo requires pre-processing to detect edges (itself anon-trivial problem)Limitation: Resulting depth information sparse, ∴ available at only afew locations.Further step needed to interpolate depth across surfaces in a scene

Biologically Plausibility:

More biologically plausible ∵ it’s known that binocular processingbegins in V1 after optical info. flow via simple cells (which may act asedge detectors)But not robust to the kinds of image manipulations discussed above



Algorithms for Stereo VisionArea and Filtering-Based Schemes

Area-based algorithms:

Correlation b/w brightness patterns in local neighborhood of a pixel w/brightness patterns in the local nhd. of the other image.Simplest is correlation

Filtering Algorithms: Jones and Malik (1992)

Algo matches local regions around the point.Uses biologically inspired linear spatial filters that differ in theirorientation and size tuning, called multi-oriented multi-scale(MOMS) filters.Inspiration from biology: ∃ cells in a hypercolumn whose RFs arecentered on the same retinal position, each cell tuned to differentorientation and scale



Jones and Malik’s motivation:

Some good algorithm for computing stereo matchingsBiological inspiration was the starting point, not the criterion forjudging soundness

Lecture theme: Algorithmic strategy that takes biological constraintsinto account - ”how does the brain do it‘‘

A windowed (unconventional) correlation-based scheme developed byYeshurun and Schwartz

Cross-Correlation (not the usual) used is called the cepstrum



How V1 does stereopsis

Yeshurun and Schwartz, in a series of papers, indicated:

Possible functional (computational) role for the ocular dominancecolummns by showing how:

OcuDom columns enable a data structure allowing a

One-shot (i.e. no iterations required) algorithm called cepstral filter to

Provide a robust solution to stereo matching that is

Consistent with psychophysical data on stereopsis



Data StructureOcular Dominance Columns

Visual (hemi)-field representation:Right cortical hemisphere gets input from left visual hemifield fromboth eyes (and vice-versa)

Interlaced image:

A B

A C B D

C D

interlaced image

Cut the two images into stripsPlace corresponding strips in alternate positions

In cortex, OcuDom columns w/ left and right eye image strips fromleft hemifield alternating



Power Spectrum or Power Spectral Density (PSD)

Power spectrum or Power Spectral Density (PSD) of a signal: Thesignal power as a function of frequency

Measures power content of component frequencies of the signal

Fourier transform of the autocorrelation function



Algorithm supported by the data structureCepstrum

Cepstrum is the power spectrum of the log of the power spectrum

Windowed cepstrum can be applied to the binocular interlaced image

Computational Issues:

Easy to compute windowed spectrum given above DSNeed to estimate power spectral density and apply a log

Biological plausibility: Cortical neurons can:

Act as medium-bandwidth power spectral filtersPerform logarithmic computation and multiplication



Cepstrum calculation I

Suppose the left and right eye images are identical i.e. no disparity

Suppose the width of a column is D

Image s(x , y) and the identical image repeated and abutted, viz.,s(x − D, y)

Interlaced image, composed of a single column pair is given byf (x , y) = s(x , y) ⊗ {δ(x , y) + δ(x − D, y)}

where ⊗ represents convolution



Cepstrum calculation II

Fourier transform of interlaced image is:F (u, v) = S(u, v) · {1 + e−iπDu}

Log is: logF (u, v) = logS(u, v) + log(1 + e−iπDu)

Fourier transform of the 2nd term on RHS has a prominent peak atdisparity D (can be shown)

Peak position can be recovered by a peak detection algorithm

Spatial position in the cepstrum is a direct measure of”disparity“ of left and right images



Requirements for a good theory of human stereopsis

Cepstrum robust to image manipulations that human stereo is robustto

We now turn to how Panum’s area change with eccentricity accordingto psychophysical data is explained by this algorithm



Perceptual data re Panum’s area explained by the abovealgorithm

DS used for computation of stereopsis: OcuDom columns

Combined width of a pair of OcuDom columns (i.e. hypercolumn) is2 mm, almost constant through V1

Recall: ∆w∆z

≈ 1z+a

,

In words, ratio of change in cortical position to visual angle subtendedis prop. to magnification factor



Perceptual data re Panum’s area explained by the abovealgorithm II

∴ ∆z ≈ ∆w1

z+a

If ∆w hypercolumn width, then ∆z is visual angle subtended

∴ angle subtended linearly proportional to inverse magnification factor

Magnification factor 1z+a

inversely proportional to eccentricity ǫ



Perceptual data re Panum’s area explained by the abovealgorithm III

∴ angle subtended by hypercolumn scales linearly w/ eccentricity ǫ

Known from psychophysical experiments: Panum’s area (range offusion) also scales linearly w/ eccentricity

Data consistent w/ the hypercolumn as a functional(computational) module to compute stereo fusion

We conclude that range of fusion extends over a single hypercolumn,regardless of position in the visual field



Edges

What are edges? Answer: Discontinuous changes in image luminance



Lecture Theme: Algorithms the brain employs to do edgedetectionRF scatter enhances boundary contour representation in V1

Give a brief summary of Bomberger and Schwartz’s paper



Eric L. Schwartz’s question

What are the computational functions of the visual cortex?

To Computer Science students of Computer Vision interested in howthe brain does vision, Eric Schwartz’s web-site:

http://cns.bu.edu/∼eric/

Plus a beautiful display of the complex log map!



Potpourri: Suggested theoretical principles for thecomputational functions of cortexPoints to ponder, and names to Google

Bayesian Inference: David Mumford

Liquid State Machines: Wolfgang Maass, in collaboration with HenryMarkram

No. of ”top-down“ connections approx. ten times number of”bottom-up“ connections. What is the computational role played bythese? After all, facial recognition takes place in time too short fortop-down connections to play a role.



Potpourri II: Suggested theoretical principles for thecomputational functions of cortexPoints to ponder, and names to Google

Schwartz and collaborators have taken one approach to therelationship between architecture and function, a continuum approach

Laurence Abbott and Peter Dayan have attempted to describenetwork connectivity and dynamics, which treats the neurons as”discrete“ units

“What and ”where“ pathways

Laminar Computation: Stephen Grossberg

Terrence Sejnowski



Eric SchwartzNeuroscience’s Enrico Fermi

Recurring theme in his work: Biological plausibility of suggestedalgorithms for a computational task(s)

IOW, how the brain does it

Eric Schwartz: Department of Cognitive and Neural Systems, BostonUniversity

Professor of Cognitive and Neural SystemsProfessor of Electrical and Computer EngineeringProfessor of Anatomy and Neurobiology,PhD Experimental High Energy Physics, Columbia

His work demonstrates how a top-quality experimental science anddata analysis can suggest mathematically formulated theoretical work

How an all-round scientist (he has theoretical insights andexperimental and modeling work of the highest quality to his credit)can be at home with all of them



Eric Schwartz

Experimental (wet-biology) work in Neurophysiology

Mathematical techniques from following subjects used in his work:

Computer Vision (Algorithms for space-variant vision)Image ProcessingSignal Processing (Application of cepstrum)Complex Analysis (Numerical conformal mapping)Graph Theory (Graph partitioning)

Non-trivial contributions to machine vision and robotics

Built a self-navigated robot



Eric Lee Schwartz’s webpage

Home Page: http://cns.bu.edu/∼eric/

Wikipedia entry: Eric L Schwartz


Single Neuron Computation Single Neuron Basics

Part III: Single Neuron Computation

How a single neuron looks to a:

Biologist - Basic terminology

Electrical Engineer or a Computer Scientist -

McCulloch-Pitts neuronRosenblatt’s PerceptronSpiking Neural Networks

Biophysicist: Hodgkin-Huxley model - describes biophysicalmechanisms of action potential generation

Theoretical Bio(logical) Physicist: William Bialek and co-workers

One of the world’s top experts in single neuron computation isBartlett Mel - I encourage you to explore his work



Single Neuron: Biology basics I

Cell body (blue) - contains most of the cell’s volume, mass andnucleusAxon hillock: o/p ”terminal“ of the neuronAxon (or o/p ”wire“ of the neuron) marked RedDendrites: neuron’s i/p terminals (blue tendril-like structures on cellbody)



Single NeuronBasic Terminology

Soma aka Cell body: ”The” neuron.

Membrane Potential: Difference in voltage between the neuron’sinterior and exterior

Dendrites: Input terminals: How a neuron receives information fromother neurons

Axon Hillock: Output terminal: Where the output is generated

Axon: Neuron messages other neurons via this “wire”



Single NeuronBasic Terminology

Action Potential aka Spike:Role: A neuron’s way of signaling other neuronsMechanism: Biophysical mechanisms cause a positive feedback processto cause the membrane potential to rise dramatically at the axonhillock, and this voltage change is transmitted down the axon.



Single Neuron: Biology basics IIPyramidal Cell

Primary excitatory neurons in the cortex, hippocampus, amygdalaSo-called because soma or cell body is ”triangular“ or ”pyramid“shape

Stylised figure aboveKrishnamoorthy V. Iyer (Department of Electrical Engineering, IIT Bombay)Computational and Theoretical Neuroscience August 22, 2011 136 / 169


Single Neuron: Modeled as an Artificial Neuron (circa1950s and 1960s)

Threhold Logic Unit, developed by McCulloch and Pitts, and calledthe McCulloch-Pitts neuron

Based on neurophysiology of the 1950so/p = φ(Σm

j=0wjxj)



Single Neuron: Modeled as an Artificial Neuron (circa1950s and 1960s) Contd

Perceptron, developed by Rosenblatt

Linear Binary classifierFeedforward neural network.Linear weighted summation (i.e. projection) followed by anonlinearity such as thresholding.Similar to McCulloch-PittsRosenblatt also developed a learning algorithmLearning algo does not terminate if i/p linearly inseparable

Perceptron cannot learn XOR (Not linearly separable)

Multilayer perceptrons overcome many of these obstacles



Single Neuron: Modeled as an Artificial Neuron: circa2010sSpiking Neural Networks aka SNNs

Current generation of artificial neural network models attempt to moreclosely model the behaviour of a single neuron or a small microcircuit

Biological neurons may encode information in the timing of theiroutput (pulse code), and not just their average rate (as we will seebelow)

Motivation: To study information encoded in spike timing andspatiotemporal codes

See Wikipedia article on Spiking neural network



Single Neuron: Modeled as an Artificial Neuron II: circa2010sSpiking Neural Networks

Textbook: Title: Spiking Neuron ModelsSubtitle: Single Neurons, Networks, PlasticityAuthors: Wulfram Gerstner, Werner KistlerTextbook available at his website, and also on the Wikipedia site

Liquid State Machines have the universal function approximationproperty

LSMs developed by :

Wolfgang Maass (theoretical computer scientist to Neuroscience)collaborator Henry Markram (biochemistry (?) switched toNeuroscience, director of the Blue Brain project)



Single Neuron Spiking: The first quantitatively successfulmodel in Neuroscience

developed by physiologists-cum-biophysicists Hodgkin and Huxley

Nobel Prize for Physiology or Medicine 1963

nonlinear ordinary differential equations describing

ionic mechanisms underlying the generation of an action potential



Hodgkin-Huxley model of the “firing“ of a Squid Axon

Squids: Marine life-form, largest known invertebrates.

Axon is huge 1 millimeter in diameter - ∴ good to experimental work

nonlinear ODEs analytically intractable - ∴ numerical methods used

Model has been successfully extended to describe biophysics ofspiking behaviour in other animals and other kinds of neurons



Why study spiking neur(on)al networks?Ans: Behaviors associated w/ SNNs

Single neuron:

Adaptation to constant stimuliRefractory period after a spike generation’Bursting’ behaviour

Neuronal populations:

ChaosSynchronization



Spiking neuronal networks underlie physiologicalphenomena

Circadian rhythm, of which the sleep-wake cycle is one example

Epilepsy

Many kinds of learning. Ex: Hippocampal theta rhythm

Central Pattern Generator (CPG): a neuronal n/w producing rhythmicpatterned output without sensory feedbackInvolved in locomotion and respiration, among others


Single Neuron Computation Single neuron as a ”computer“

Beyond Hodgkin-Huxley: The single neuron as a”computer“

William Bialek’s question: What is the computation performedby a single neuron?

Can we find a general mapping between the description of a neuron:

as a dynamical system (Hodgkin-Huxley model mathematicaldescription in terms of nonlinear ODEs)and in terms of the computational functions of the neuron(mathematical description in terms of information theory, theory ofcomputation, computational learning theory)?

Coding: Representation of Information

Computation: Transformation b/w different representations



Holy Grail of Neuroscience: The Neural Code

Schwartz and collaborators work discussed above refers to therepresentation of information at the level of maps

In the case of coding by a single neuron, or by a small population ofneurons, both

spatial scaletemporal scale

are much smaller than the case with cortical mapping (i.e.topographic organization)

The Neural Code refers to these dynamically changing firingpatterns



Neural CodingEncoding and Decoding

Neural coding can be studied from two points of view:

Encoding: Map from stimulus to response

Decoding: Reverse map from response to stimulus



Neural CodingEncoding

1 Encoding: Map from stimulus to response

Neuronal tuning curve and the receptive fields were two of the earliestmodels of neural encodingNatural viewpoint for the earliest experimenters, who tried to figure outwhat stimuli would maximize neuronal firingBuild models that predict the neural response as a function of stimulusBut ...

Neural responses are stochastic and show high variabilitySame stimulus may elicit different responses on different trials

∴ encoding models try to capture average properties of neuralresponses, such as the average firing rate - the tuning curve being anexample

2 High variability of responses to stimuli raises an interesting question:



Neural CodingEncoding Cont’d

How is the neuronal response to be mathematically characterised?

First-the stimulus mathematical characterisation

The best discussion (by far) of these matters is in the textbook:Title: Spikes: Exploring the Neural CodeAuthors: Fred Rieke, David Warland, Rob de Ruyter van Steveninck,William Bialek



Characterizing the stimulus

1 The stimulus can be considered to be a random variable drawn from acollection of possible stimuli

2 The particular stimulus is a realization

3 Throwing a die:

4 Random Variable :outcome ofexperiment,

5 Ensemble is the collection of possible outcomes 1, 2, 3, 4, 5, 6,

6 Realization is the face that turns up after the throwing



1 Since both stimulus and response are random variables, with a(probabilistic) causation linking them, the joint distributionP(Stimulus, Response) is well-defined

2 Also, due to neuronal variability, the response is a probabilisticfunction of the stimulus

3 This can be modeled by a conditional probability distribution. Thus:P(Response|Stimulus)

4 By Bayes’s rule, we haveP(Stimulus|Response) = P(Stimulus)∗P(Response|Stimulus)

P(Response)



Neural CodingDecoding

Decoding: Reverse map, from response to stimulus

Goal try to reconstruct some aspects of the stimulus

The “internal“ ”observer’s“ point of view



Need for mathematizing the questions of neural coding

Using Bayes’s Rule, we can characeterize the neural code by:

either listing the rules for translating stimuli into spikes -encoding

or by listing the rules for translating spikes into stimuli -decoding



Rate Codes, Temporal Codes, Population Codes

Early workers posited that neurons signaled/messaged/encoded infoby avg. firing rates

Rationale: When a neuron receives a stimulus it’s tuned to, it sendsout a stream of spikes

Otherwise it stays (almost) silent



Problem w/ Rate Code IdeaSimplest Experimental Objection

Known that facial recognition takes place within 150 milliseconds

Face recognition takes place in inferotemporal cortex, and requiresthat information be routed through at least 6 stages:

Retina(Visual) ThalamusArea V1Area V2Area V3Infero-Temporal Cortex



Problem w/ Rate Code Idea IISimplest Experimental Objection

∃ a time lag of 10 milliseconds between any two stages. Leaves about150 - 50 = 100 milliseconds for within stage processing over 5 stages

With 5 stages, this means an average of 20 milliseconds per stage

To compute an average firing rate in 20 milliseconds, assuming atleast 2 spikes are needed, would require that neurons in visual cortexspike at rates of 100 per second.

No neurons are known that fire at this rate



The Neural CodeTemporal Codes aka How neurons represent information in their firing pattern(s)

Individual neurons represent info in timing of o/p pulses, and not justtime-averaged rate (spike-count code)

Familiar to communication engineers in the form of pulse positionmodulation, where pulse time encodes info

Introduce a figure here showing PPM

A neural population can encode info. in relative timing of spikes

See Wikipedia article on phase-of-firing code: a temporal codecombining spike-count code w/ a time reference based on slowoscillations



The Neural CodeTemporal Codes aka How neurons represent information in their firing patterns

Spatiotemporal population activity patterns can support codes ofimmense complexity

Complex codes necessitate high computational power

Some questions arise: Can we quantify the computational power of aneuron/neuronal network?

Computational/Statistical Learning TheoryVapnik-Chervonenkis dimension



Computation: Neurons as feature detectors

Notion of neuron as a feature selector goes back to the receptive field

Images: Vectors in a high dimensional space,

Pixel intensities: coordinates

Receptive fields: Neuron as sensitive to one direction or projection inthis high dimensional space, and performs a correlation

Linear summation followed by thresholding to determine the firingprobability recalls the perceptron

Complex cells receive i/p from many simple cells,

Complex cells perform projections followed by nonlinear summation

and thresholding



Reverse correlation aka Spike-Triggered Average (STA)

Average stimulus preceding a spike

Provides an estimate of the neuron’s linear receptive field

Limitation of technique:i/p-o/p function of a neuron is inherently nonlinear:

adaptation of neural firing rates to a constant inputrefractory periods during which a neuron will not fire, no matter whatthe input

Assumption: No. of stimulus dim to which the neuron responds is 1, orin the worst case, a very small number



Estimating subspace dimensionality from data

Experimental support for the idea of characterizing the neuronalresponse as a reduction of dimensionality:

Studies of motion-sensitive neuron in the fly visual system

I encourage you to visit William Bialek’s web-page at Princeton



Motion estimation... in the fly visual system

Bialek and co-workers have shown that the fly’s estimate is the bestpossible given noise in its motion sensitive cell

”Best possible” as computed by applying optimal estimation theoryprinciples



Charaterizing the computation performed by a neuron

Three steps:

Estimate the number of relevant stimulus dimensions, and hope it issmall

The “curse of dimensionality” should be familiar to students ofmachine learning (CS 419, CS625, CS725)

Identify a set of filters that project onto the relevant subspace

Characterize the nonlinear function (Perceptron: Nonlinear function isthresholding)



Optimization as an overarching theoretical principle

Bialek and co-workers have discovered for optimal performance in:

photon counting in visionoptimal coding in spike trains

Neural systems seem to have reached the performance limits imposedby contraints on neural circuitry such as noise and energyconsumption



Part V: Networks of Neurons

I have chosen to tread lightly over the level of modeling inneuroscience that is most familiar to engineers and CS types, namely,the network level

One can study the connectivity of neurons in networks, whilede-emphasizing the role of biophyiscal mechanisms at the singleneuron scale, and also ignoring the map level scale of 1 cm



Neuronal Network Modeling as a tool to study networks ofneurons

NEURON

Liquid State Machines: A concept pioneered by Wolfgang Maass andHenry Markram



Pawan Sinha, CS switched to Neuroscience, currently at MIT,Cambridge, USA

In India, Arun Sripati at the Indian Institute for Science, Center forNeuroscience, alumnus of IIT Bombay, dept. of Electrical Engineering,PhD, Johns Hopkins, Electrical Engineering and Neuroscience



Textbooks and Websites

This list reflects my own tastes and interests

1 Theoretical Neuroscience by Peter Dayan and Laurence Abbott

2 Spikes: Reading the Neural Code by Fred Rieke, Rob de Ruyter vanSteveninck, David Warland and William Bialek

3 The Computational Brain by Terrence Sejnowksi and PatriciaChurchland

4 Vision: From Photons to Phenomenology by Stephen Palmer

5 Computational Neuroscience by Eric Schwartz (Not a textbook.Collection of original research papers)

6 Eye, Brain and Cortex by David Hubelhttp://hubel.med.harvard.edu/book/bcontex.htm



Acknowledgements

Thanks are due to:

Prof. Pushpak Bhattacharyya, for giving me this opportunity toaddress CS students of IIT Bombay

Teaching Assistants: Janardhan and Munish Minia and Arindam


Documents

Computational and Theoretical Neuroscience