Neurobiological Models of Visual Attentiontsotsos/Homepage of John K_files/Models... · 2000. 10. 20. · Theories/Models The number of models that address ... Müller (1873) re-inforcement

11

Neurobiological Models of VisualAttention

Neurobiological Models of VisualAttention

John K. TsotsosJohn K. Tsotsos

Dept. of Computer ScienceDept. of Computer Science

andand

Centre for Vision ResearchCentre for Vision Research

York UniversityYork University

©J.K.Tsotsos ©J.K.Tsotsos 22

Müller (1873) Exner (1894) Wundt (1902) Pillsbury (1908)Müller (1873) Exner (1894) Wundt (1902) Pillsbury (1908)Broadbent 1958 (Early Selection)Broadbent 1958 (Early Selection)Deutsch, Deutsch & Norman 1963/68 (Late Selection)Deutsch, Deutsch & Norman 1963/68 (Late Selection)Treisman 1964Treisman 1964Milner Milner 1974 1974 **Grossberg 1976+ (Adaptive Resonance Theory) Grossberg 1976+ (Adaptive Resonance Theory) **Treisman & Gelade 1980 (Feature Integration Theory)Treisman & Gelade 1980 (Feature Integration Theory)von der Malsburg 1981+ (Correlation Theory) von der Malsburg 1981+ (Correlation Theory) **Crick 1984 Crick 1984 **Koch and Ullman 1985Koch and Ullman 1985Anderson and Van Essen 1987 (Shifter Circuits) Anderson and Van Essen 1987 (Shifter Circuits) **Sandon 1989 Sandon 1989 ‡‡Wolfe et al. 1989+ (Guided Search 1.0, 2.0. 3.0)Wolfe et al. 1989+ (Guided Search 1.0, 2.0. 3.0)Phaf, Van der Heijden, Hudson 1990 (SLAM)Phaf, Van der Heijden, Hudson 1990 (SLAM)Tsotsos et al. 1990+ (Selective Tuning) Tsotsos et al. 1990+ (Selective Tuning) * * ‡‡Mozer 1991 (MORSEL)Mozer 1991 (MORSEL)Ahmad 1991 (VISIT) Ahmad 1991 (VISIT) **Olshausen, Anderson & Van Essen 1993 Olshausen, Anderson & Van Essen 1993 * * ‡‡Niebur, Koch et al. 1993+ Niebur, Koch et al. 1993+ **Desimone & Duncan 1995 (Biased Competition) Desimone & Duncan 1995 (Biased Competition) **Postma 1995 (SCANPostma 1995 (SCAN) ) ** ‡‡Schneider 1995 (VAM) Schneider 1995 (VAM) **LaBerge LaBerge 1995 1995 **Itti & Koch 1998 Itti & Koch 1998 ‡‡Cave et al. 1999 (FeatureGate)Cave et al. 1999 (FeatureGate)

Theories/ModelsTheories/ModelsTheories/Models

The number of models that addressThe number of models that addressthe neurobiology of visual attentionthe neurobiology of visual attentionis small (is small (** in the list). The number in the list). The numberthat have real computational teststhat have real computational testson actual images is even smaller (on actual images is even smaller (‡‡in the list). However, many relevantin the list). However, many relevantideas have appeared inideas have appeared inpsychological models.psychological models.

A selected historical perspective onA selected historical perspective onthe ideas important to the modellingthe ideas important to the modellingtask appears in the following slides.task appears in the following slides.

22


Models of visual attention need to include solutions to or exhibit observedModels of visual attention need to include solutions to or exhibit observedneurobiological/psychophysical performance for:neurobiological/psychophysical performance for:

FF computational complexity of visual processes computational complexity of visual processes

FF information routing through the processing hierarchy information routing through the processing hierarchy

FF attentional control attentional control

FF time course of attentive modulation time course of attentive modulation

FF single cell attentive modulation single cell attentive modulation

FF attentive modulation in (apparently) all visual areas attentive modulation in (apparently) all visual areas

FF suppressive surround effects suppressive surround effects

FF serial/”parallel” visual search performance serial/”parallel” visual search performance

FF binding of features to objects binding of features to objects

IssuesIssuesIssues


Format of OverviewFormat of Overview

Not all models are included, only those that have historicalNot all models are included, only those that have historical importance or that claim neuro-psycho relevance importance or that claim neuro-psycho relevance

Due to space and time limits, each model is described only with:Due to space and time limits, each model is described only with:1. key 1. key referencesreferences2. 2. key ideaskey ideas3. neurobiological relationship (where possible)3. neurobiological relationship (where possible)

( ( has supporting evidence has supporting evidence XX does not have supporting evidence does not have supporting evidence

?? open question) open question)

Note that this can only be regarded as a partial review!Note that this can only be regarded as a partial review!

33


The nature of the attentional influence has beenThe nature of the attentional influence has been debated for over a century: debated for over a century:

Müller (1873)Müller (1873) re-inforcementre-inforcement Exner (1894)Exner (1894) inhibition and re-inforcementinhibition and re-inforcement Wundt (1902)Wundt (1902) inhibitioninhibition

Pillsbury (1908)Pillsbury (1908) ‘attention is in dis-array’‘attention is in dis-array’

Inhibition or Enhancement?Inhibition or Enhancement?Inhibition or Enhancement?

©J.K.Tsotsos ©J.K.Tsotsos 66Broadbent 1958 Early Selection ModelBroadbent 1958 Early Selection ModelBroadbent, D. (1958). Broadbent, D. (1958). Perception and communicationPerception and communication, Pergamon Press, NY., Pergamon Press, NY.

Key ideas:Key ideas:- short term store acts to extend duration of stimulus- short term store acts to extend duration of stimulus- stimuli could be partitioned into channels (modalities)- stimuli could be partitioned into channels (modalities)- selective filter selects among channels- selective filter selects among channels- limited capacity channel processes selected channel- limited capacity channel processes selected channel X (X (split-span experiments of Deutsch & Deutsch )split-span experiments of Deutsch & Deutsch )

44

©J.K.Tsotsos ©J.K.Tsotsos 77Deutsch/Norman Model 1968 Late Selection ModelDeutsch/Norman Model 1968 Late Selection ModelDeutsch, J., Deutsch, D. (1963). Attention: Some theoretical considerations, Deutsch, J., Deutsch, D. (1963). Attention: Some theoretical considerations, Psych. Review 70Psych. Review 70, 80-90., 80-90.Norman, D. (1968). Toward a theory of memory and attention, Norman, D. (1968). Toward a theory of memory and attention, Psych. Review 75Psych. Review 75, 522-536., 522-536.

Late Selection Model

Summary of early ideas on attention

Key ideas:Key ideas:- all information is recognized before it receives- all information is recognized before it receives the attention of a limited capacity processor the attention of a limited capacity processor X (X (shadowing with target word tapping, Treisman)shadowing with target word tapping, Treisman)- recognition can occur in parallel- recognition can occur in parallel- stimulus relevance determines what is attended- stimulus relevance determines what is attended

©J.K.Tsotsos ©J.K.Tsotsos 88Treisman 1964Treisman, A. (1964). the effect of irrelevant material on the efficiency of selectivelistening, American J. Psychology 77 533-546.

Treisman 1964Treisman 1964Treisman, A. (1964). the effect of irrelevant material on the efficiency of selectiveTreisman, A. (1964). the effect of irrelevant material on the efficiency of selectivelistening, listening, American J. Psychology 77American J. Psychology 77 533-546. 533-546.

Key ideas:Key ideas:- filter attenuates (is not binary) unattended signals casing them to be incompletely analyzed- filter attenuates (is not binary) unattended signals casing them to be incompletely analyzed- filter can operate at different levels - signal or meaning - so attention is hierarchical - filter can operate at different levels - signal or meaning - so attention is hierarchical

(Kastner et al. 1998)(Kastner et al. 1998)

55

©J.K.Tsotsos ©J.K.Tsotsos 99Milner 1974Milner, P. (1974). A model for visual shape recognition, Psych. Rev. 81, 521-535.Milner 1974Milner 1974Milner, P. (1974). A model for visual shape recognition, Milner, P. (1974). A model for visual shape recognition, Psych. Rev. 81Psych. Rev. 81, 521-535., 521-535.

Key ideas:Key ideas:- unity of a figure at the neuronal level defined by synchronized firing - unity of a figure at the neuronal level defined by synchronized firing activity activity ??- attention acts in two ways: - attention acts in two ways:

to select relevant figure from among othersto select relevant figure from among others

to activate the feedback pathways to activate the feedback pathways (Felleman & Van Essen 1991)(Felleman & Van Essen 1991)fromfrom the cell the cell assembly to the early visual cortex forassembly to the early visual cortex for precise precise localizationlocalization

- feedback pathways communicate attentional instructions- feedback pathways communicate attentional instructions ??

©J.K.Tsotsos ©J.K.Tsotsos 1010Grossberg 1976+ Adaptive Resonance TheoryGrossberg, S., Carpenter, G., et al. (1998). The what-and-where filter: a spatial mapping neural network for object recognition and image understanding, Computer Vision and Image Understanding 69(1): 1-22.Grossberg, S. (1998). How does the cerebral cortex work? Learning, attention and grouping by the laminar circuits of visual cortex,Technical Report CAS/CNS-97-023

Grossberg 1976+ Adaptive Resonance TheoryGrossberg 1976+ Adaptive Resonance TheoryGrossberg, S., Carpenter, G., et al. (1998). The what-and-where filter: a spatial mappingGrossberg, S., Carpenter, G., et al. (1998). The what-and-where filter: a spatial mapping neural network for object recognition and image understanding, neural network for object recognition and image understanding, Computer Vision andComputer Vision and Image Understanding 69(1): Image Understanding 69(1): 1-22. 1-22.Grossberg, S. (1998). How does the cerebral cortex work? Learning, attention and grouping byGrossberg, S. (1998). How does the cerebral cortex work? Learning, attention and grouping by the laminar circuits of visual cortex,Technical Report CAS/CNS-97-023 the laminar circuits of visual cortex,Technical Report CAS/CNS-97-023

Key ideas:Key ideas:- ART algorithms are clustering algorithms that obey the following:- ART algorithms are clustering algorithms that obey the following:

bottom-up activation can drive a cell if strong enoughbottom-up activation can drive a cell if strong enoughtop-down priming can modulate a celltop-down priming can modulate a cellcell becomes active if it receives large enough top-down and bottom-up activationcell becomes active if it receives large enough top-down and bottom-up activationtop-down activation, even small, can negate bottom-up activation top-down activation, even small, can negate bottom-up activation feedback leads to resonance and convergencefeedback leads to resonance and convergence

- top-down attentional mechanisms should occur in every cortical area where learning - top-down attentional mechanisms should occur in every cortical area where learning

can occurcan occur ??- specific circuitry for interactions (slide 12) - specific circuitry for interactions (slide 12) ??

66


ART rule may be realized by a top-ART rule may be realized by a top-down on-center off-surround networkdown on-center off-surround network


Top-down, bottom-up andTop-down, bottom-up andhorizontal interactions in LGN, V1horizontal interactions in LGN, V1and V2 based on ART Rule.and V2 based on ART Rule.

Green - Green - preattentivepreattentive excitatory excitatorymechanismsmechanisms

Red - inhibitory mechanismsRed - inhibitory mechanisms

Blue - top-down attentional Blue - top-down attentional mechanismsmechanisms

77


Treisman & Gelade 1980 Feature Integration TheoryTreisman, A., Gelade, G. (1980). A feature integration theory of attention, Cognitive Psychology 12: 97-136.

Treisman & Gelade 1980 Feature Integration TheoryTreisman & Gelade 1980 Feature Integration TheoryTreisman, A., Gelade, G. (1980). A feature integration theory of attention, Treisman, A., Gelade, G. (1980). A feature integration theory of attention, CognitiveCognitive Psychology 12: Psychology 12: 97-136. 97-136.

Key ideas:Key ideas:- master map of locations- master map of locations

- attentional spotlight - attentional spotlight XX

Predictions: Predictions: - popout in visual search without attention - popout in visual search without attention

- conjunction search requires attention- conjunction search requires attention XX X (Wolfe, Nakayama) X (Wolfe, Nakayama)

©J.K.Tsotsos ©J.K.Tsotsos 1414von der Malsburg 1981 Correlation Brain Theoryvon der Malsburg, C. (1981). The correlation theory of brain function, Internal Rpt. 81-2, Dept. of Neurobiology, Max-Planck-Institute for Biophysical Chemistry, Gottingen, Germany.

von der Malsburg 1981von der Malsburg 1981 Correlation Brain Theory Correlation Brain Theoryvon der Malsburg, C. (1981). The correlation theory of brain function, Internal Rpt. 81-2,von der Malsburg, C. (1981). The correlation theory of brain function, Internal Rpt. 81-2, Dept. of Neurobiology, Max-Planck-Institute for Biophysical Chemistry, Gottingen, Germany. Dept. of Neurobiology, Max-Planck-Institute for Biophysical Chemistry, Gottingen, Germany.

Key ideas:Key ideas:- synaptic modulation - synapses switch between conducting and non-conducting states - synaptic modulation - synapses switch between conducting and non-conducting states - modulation governed by correlations in temporal structure of signals- modulation governed by correlations in temporal structure of signals- momentarily useless connections are deactivated and interference between different - momentarily useless connections are deactivated and interference between different memory traces are reduced and memory capacity increased memory traces are reduced and memory capacity increased

- - dynamic modulationdynamic modulation ( (Moran & DesimoneMoran & Desimone 19851985) )

- brain does not contain complex feature detector cells- brain does not contain complex feature detector cells XX (e.g., face cells) (e.g., face cells)

- timing correlations signal objects- timing correlations signal objects ??

88

©J.K.Tsotsos ©J.K.Tsotsos 1515Crick 1984Crick, F. (1984). Function of the thalamic reticular complex: The searchlight hypothesis, Proc. Natl. Acad. Sci. USA 81, 4586-4590.

Crick 1984Crick 1984Crick, F. (1984). Function of the thalamic reticular complex: The searchlight hypothesis,Crick, F. (1984). Function of the thalamic reticular complex: The searchlight hypothesis, Proc. Natl. Acad. Sci. USA 81Proc. Natl. Acad. Sci. USA 81, 4586-4590., 4586-4590.

Key ideas:Key ideas:- Treisman’s searchlight is controlled by the- Treisman’s searchlight is controlled by the

reticular complex of the thalamus reticular complex of the thalamus XX- searchlight is expressed by rapid bursts of firing- searchlight is expressed by rapid bursts of firing from subsets of thalamic neurons from subsets of thalamic neurons- conjunctions are mediated by rapidly modifiable- conjunctions are mediated by rapidly modifiable

synapses (Malsburg synapses) by these bursts synapses (Malsburg synapses) by these bursts ??- activation of Malsburg synapses produces transient- activation of Malsburg synapses produces transient

cell assemblies connecting neurons at different levels cell assemblies connecting neurons at different levels ??

©J.K.Tsotsos ©J.K.Tsotsos 1616Koch and Ullman 1985Koch, C., Ullman, S. (1985). Shifts in selective visual attention: Towards the underlying neural circuitry, Human Neurobiology 4, 219-227.

Koch and Ullman 1985Koch and Ullman 1985Koch, C., Ullman, S. (1985). Shifts in selective visual attention: Towards the underlyingKoch, C., Ullman, S. (1985). Shifts in selective visual attention: Towards the underlying neural circuitry, neural circuitry, Human Neurobiology 4Human Neurobiology 4, 219-227., 219-227.

Key ideas:Key ideas:• saliency map (Treisman’s map) • saliency map (Treisman’s map) ??• winner-take-all competition • winner-take-all competition

(Findlay 199, Lee et al. 1999)(Findlay 199, Lee et al. 1999) - WTA selects items to route to central - WTA selects items to route to central representation representation XX

• inhibition of return for shifts • inhibition of return for shifts ??• time to move attention requires time • time to move attention requires time logarithmic in distance between stimuli logarithmic in distance between stimuli X (Krose & Julesz 1989) X (Krose & Julesz 1989)

• no single cell modulations • no single cell modulations XX

99

©J.K.Tsotsos ©J.K.Tsotsos 1717Anderson and Van Essen 1987 Shifter CircuitsAnderson, C., Van Essen, D. (1987). Shifter Circuits: a computational strategy for dynamic aspects of visual processing, Proc. Natl. Academy Sci. USA 84: 6297-6301.

Anderson and Van Essen 1987 Shifter CircuitsAnderson and Van Essen 1987 Shifter CircuitsAnderson, C., Van Essen, D. (1987). Shifter Circuits: a computational strategy for dynamicAnderson, C., Van Essen, D. (1987). Shifter Circuits: a computational strategy for dynamic aspects of visual processing, aspects of visual processing, Proc. Natl. Academy Sci. USA 84Proc. Natl. Academy Sci. USA 84: 6297-6301.: 6297-6301.

Key ideas:Key ideas:• information routing is accomplished by simple shifting circuits starting in the LGN and • information routing is accomplished by simple shifting circuits starting in the LGN and

input layers of primate visual area V1. input layers of primate visual area V1. XX• realignment is based on the preservation of spatial relationships • realignment is based on the preservation of spatial relationships XX• stages linked by diverging excitatory inputs. • stages linked by diverging excitatory inputs. • direction of shift by inhibitory neurons that selectively suppress sets of ascending inputs. • direction of shift by inhibitory neurons that selectively suppress sets of ascending inputs. • stages are grouped into small and large scale shifts. • stages are grouped into small and large scale shifts.

• control comes from pulvinar • control comes from pulvinar ??

©J.K.Tsotsos ©J.K.Tsotsos 1818Wolfe 1989+ Guided SearchWolfe, J., Cave, K., Franzel, S. (1989). Guided search: An alternative to the feature integration model for visual search, J. Exp. Psychology: Human Perception and Performance 15, 419-433.Wolfe, J. (1994). Guided search 2.0: a revised model of visual search, Psychonomic Bulletin and Review, 1(2):202-238.Wolfe, J., Gancarz, G. (1996). Guided Search 3.0: A Model of Visual Search Catches Up With Jay Enoch 40 Years Later, in V. Lakshminarayanan (Ed.), Basic and Clinical Applications Vision Science, Dordrecht, Netherlands: Kluwer Academic. p189-192.

Wolfe 1989+ Guided SearchWolfe 1989+ Guided SearchWolfe, J., Cave, K., Franzel, S. (1989). Guided search: An alternative to the feature integrationWolfe, J., Cave, K., Franzel, S. (1989). Guided search: An alternative to the feature integration model for visual search, model for visual search, J. Exp. Psychology: Human Perception and Performance 15J. Exp. Psychology: Human Perception and Performance 15, 419-433., 419-433.Wolfe, J. (1994). Guided search 2.0: a revised model of visual search, Wolfe, J. (1994). Guided search 2.0: a revised model of visual search, Psychonomic Bulletin andPsychonomic Bulletin and Review, 1(2): Review, 1(2):202-238.202-238.Wolfe, J., Gancarz, G. (1996). Guided Search 3.0: A Model of Visual Search Catches UpWolfe, J., Gancarz, G. (1996). Guided Search 3.0: A Model of Visual Search Catches Up With Jay Enoch 40 Years Later With Jay Enoch 40 Years Later, , in V. Lakshminarayanan (Ed.), in V. Lakshminarayanan (Ed.), Basic and Clinical ApplicationsBasic and Clinical Applications Vision ScienceVision Science, Dordrecht, Netherlands: Kluwer Academic. p189-192., Dordrecht, Netherlands: Kluwer Academic. p189-192.

Key ideas:Key ideas:- attentional deployment of limited - attentional deployment of limited resources is guided by output of resources is guided by output of earlier parallel processes earlier parallel processes

- activation map - activation map ??

1010


Guided Search 3.0Guided Search 3.0

©J.K.Tsotsos ©J.K.Tsotsos 2020Sandon 1990Sandon, P. (1990). Simulating visual attention, J. Cognitive Neuroscience 2:213-231.Sandon 1990Sandon 1990Sandon, P. (1990). Simulating visual attention, Sandon, P. (1990). Simulating visual attention, J. Cognitive Neuroscience 2J. Cognitive Neuroscience 2:213-231.:213-231.

Key ideas:Key ideas:- first real implementation of Koch & Ullman model- first real implementation of Koch & Ullman model - first real implementation of - first real implementation of

any attention modelany attention model- hierarchical, multiscale (pyramid) connectionist network- hierarchical, multiscale (pyramid) connectionist network- translation-invariant object recognition- translation-invariant object recognition- bottom-up feature guidance- bottom-up feature guidance- Koch and Ullman WTA scheme- Koch and Ullman WTA scheme- no neurobiological predictions- no neurobiological predictions

1111

©J.K.Tsotsos ©J.K.Tsotsos 2121Phaf, Van der Heijden, Hudson 1990 SLAMPhaf, R., Van der Heijden, A., Hudson, P. (1990). SLAM: A connectionist model for attention in visual selection tasks, Cognitive Psychology 22, 273 - 341.

Phaf, Van der Heijden, Hudson 1990 SLAMPhaf, Van der Heijden, Hudson 1990 SLAMPhaf, R., Van der Heijden, A., Hudson, P. (1990). SLAM: A connectionist model for attentionPhaf, R., Van der Heijden, A., Hudson, P. (1990). SLAM: A connectionist model for attention in visual selection tasks, in visual selection tasks, Cognitive Psychology 22Cognitive Psychology 22, 273 - 341., 273 - 341.

Key ideas: - based on McClelland & Rumelhart 1981 model for visual word recognition - adds response selection and evaluation - inhibitory competition to reduce distractor interference - attended items enhanced

©J.K.Tsotsos ©J.K.Tsotsos 2222Tsotsos 1990+ Selective Tuning Model

Tsotsos, J.K., Analyzing Vision at the Complexity Level, Behavioral and Brain Sciences 13-3, p423 - 445, 1990.Tsotsos, J.K. (1993). An Inhibitory Beam for Attentional Selection, in Spatial Vision in Humans and Robots, ed. by L. Harris and M. Jenkin, p313 - 331, Cambridge University Press.Tsotsos, J.K., Culhane, S., Wai, W., Lai, Y., Davis, N., Nuflo, F. (1995). Modeling visual attention via selective tuning, Artificial Intelligence 78(1-2),p 507 - 547.Tsotsos, J.K. (1995). Towards a Computational Model of Visual Attention, in Early Vision and Beyond, ed. by T. Papathomas, C, Chubb, A. Gorea, E. Kowler, MIT Press/Bradford Books, p207 - 218.Tsotsos, J.K., Culhane, S., Cutzu, F., From Theoretical Foundations to a Hierarchical Circuit for Selective Attention, Visual Attention and Cortical Circuits, ed. by J. Braun, C. Koch & J. Davis, MIT Press (in press).

Tsotsos 1990+ Selective Tuning ModelTsotsos 1990+ Selective Tuning Model

TsotsosTsotsos, J.K., Analyzing Vision at the Complexity Level, , J.K., Analyzing Vision at the Complexity Level, Behavioral and Brain SciencesBehavioral and Brain Sciences 13-3 13-3, p423 - 445, 1990., p423 - 445, 1990.Tsotsos, J.K. (1993). An Inhibitory Beam for Attentional Selection, in Tsotsos, J.K. (1993). An Inhibitory Beam for Attentional Selection, in Spatial Vision in HumansSpatial Vision in Humans and Robots and Robots, ed. by L. Harris and M. Jenkin, p313 - 331, Cambridge University Press., ed. by L. Harris and M. Jenkin, p313 - 331, Cambridge University Press.Tsotsos, J.K., Culhane, S., Wai, W., Lai, Y., Davis, N., Nuflo, F. (1995). Modeling visual attentionTsotsos, J.K., Culhane, S., Wai, W., Lai, Y., Davis, N., Nuflo, F. (1995). Modeling visual attention via selective tuning, via selective tuning, Artificial IntelligenceArtificial Intelligence 78(1-2),78(1-2),p 507 - 547.p 507 - 547.Tsotsos, J.K. (1995). Towards a Computational Model of Visual Attention, in Tsotsos, J.K. (1995). Towards a Computational Model of Visual Attention, in Early Vision andEarly Vision and BeyondBeyond, ed. by T. Papathomas, C, Chubb, A. Gorea, E. Kowler, MIT Press/Bradford Books,, ed. by T. Papathomas, C, Chubb, A. Gorea, E. Kowler, MIT Press/Bradford Books, p207 - 218. p207 - 218.Tsotsos, J.K., Culhane, S., Cutzu, F., From Theoretical Foundations to a Hierarchical Circuit forTsotsos, J.K., Culhane, S., Cutzu, F., From Theoretical Foundations to a Hierarchical Circuit for Selective Attention, Selective Attention, Visual Attention and Cortical CircuitsVisual Attention and Cortical Circuits, ed. by J. Braun, C. Koch & J. Davis,, ed. by J. Braun, C. Koch & J. Davis, MIT Press (in press). MIT Press (in press).

1212


neuron ‘sees’ thisneuron ‘sees’ thisreceptive fieldreceptive field

subject ‘attends’subject ‘attends’to single itemto single item

Key ideas:Key ideas:- attention modulates neurons to earliest levels; wherever there is a many-to-one- attention modulates neurons to earliest levels; wherever there is a many-to-one mapping mapping - signal interference controlled by surround inhibition- signal interference controlled by surround inhibition throughout processing network throughout processing network - task knowledge biases computations throughout processing network- task knowledge biases computations throughout processing network- inhibition of connections not units- inhibition of connections not units Hernandez-Peon, Scherrer, Jouvet (1956)Hernandez-Peon, Scherrer, Jouvet (1956)- attentional control is local, distributed and internal- attentional control is local, distributed and internal- competition is based on WTA (different form than previous models)- competition is based on WTA (different form than previous models)- pyramid representation with reciprocal convergence and divergence- pyramid representation with reciprocal convergence and divergence Salin &Bullier(1995)Salin &Bullier(1995) attentional

spotlight

effectivereceptive fieldof selected unitin unattended case

layers of inputabstraction hierarchy

inhibitoryattentional beam

"pass" zone

"inhibit" zone

The basic idea The basic idea (BBS 1990)(BBS 1990)


processingpyramid

inhibited pathways

passpathways

unit of interestat top

input

Caputo & Guerra 1998Caputo & Guerra 1998Bahcall & Kowler 1999Bahcall & Kowler 1999Vanduffel, Tootell, Orban 2000Vanduffel, Tootell, Orban 2000Smith et al. 2000Smith et al. 2000

Kastner, De Weerd, Kastner, De Weerd, Desimone, Ungerleider, Desimone, Ungerleider, 19981998

1313

©J.K.Tsotsos ©J.K.Tsotsos 2525Ahmad 1991 VISITAhmad 1991 VISITAhmad, S. (1992). VISIT: a neural model of covert visual attention, in Ahmad, S. (1992). VISIT: a neural model of covert visual attention, in Advances in NeuralAdvances in Neural Information Processing SystemsInformation Processing Systems, edited by J.E. Moody, et al., 4:420-427, San Mateo, CA:, edited by J.E. Moody, et al., 4:420-427, San Mateo, CA: Morgan Kaufmann. Morgan Kaufmann.

Key ideas:Key ideas:- complexity is linear in number of pixels- complexity is linear in number of pixels- bottom-up connectionist model- bottom-up connectionist model- can compute spatial relations as well as model visual search- can compute spatial relations as well as model visual search- gated feature maps that inhibit unattended features- gated feature maps that inhibit unattended features- peaks in weighted feature saliency map locate focus of attention- peaks in weighted feature saliency map locate focus of attention

Predicting the attentional roles forPredicting the attentional roles forseveral cortical areasseveral cortical areas

Architecture for visual searchArchitecture for visual search


Mozer 1991 MORSELMozer, M.C. (1991). The perception of multiple objects, MIT Press, Cambridge, MAMozer 1991 MORSELMozer 1991 MORSELMozer, M.C. (1991). Mozer, M.C. (1991). The perception of multiple objectsThe perception of multiple objects, MIT Press, Cambridge, MA, MIT Press, Cambridge, MA

Key ideas:Key ideas:- connectionist model of spatial attention and object recognition- connectionist model of spatial attention and object recognition- BLIRNET builds location invariant representations of letters and words- BLIRNET builds location invariant representations of letters and words- BLIRNET includes a pull-out net and an attentional mechanism to limit processing- BLIRNET includes a pull-out net and an attentional mechanism to limit processing- pull-out net uses semantic and lexical knowledge to select best interpretation- pull-out net uses semantic and lexical knowledge to select best interpretation- attention selects location - guided bottom-up by locations of stimuli and by top-down task - attention selects location - guided bottom-up by locations of stimuli and by top-down task bias (as in controlling temporal order in reading) bias (as in controlling temporal order in reading)- attention gates input to BLRNET- attention gates input to BLRNET- does not inhibit, just - does not inhibit, just transmitstransmits activation with lower probability activation with lower probability- attention uses neural net optimization search- attention uses neural net optimization search

1414



Olshausen, Anderson & Van Essen 1993Olshausen, B., et al. (1993). A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information, J. of Neuroscience, 13(1):4700-4719.

Olshausen, Anderson & Van Essen 1993Olshausen, Anderson & Van Essen 1993Olshausen, B., et al. (1993). A neurobiological model of visual attention and invariant patternOlshausen, B., et al. (1993). A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information, recognition based on dynamic routing of information, J. of Neuroscience, 13(1):J. of Neuroscience, 13(1):4700-4719.4700-4719.

Key ideas:Key ideas:- implementation of shifter circuits- implementation of shifter circuits

- forms position and scale invariant representations at the output layer - forms position and scale invariant representations at the output layer XX- control neurons, originating in the pulvinar, dynamically modify synaptic weights - control neurons, originating in the pulvinar, dynamically modify synaptic weights

of intracortical connections to achieve routing of intracortical connections to achieve routing ??- the topography of the selected portion of the visual field is preserved - the topography of the selected portion of the visual field is preserved

- uses Koch & Ullman mechanism (luminance saliency only) for selection - uses Koch & Ullman mechanism (luminance saliency only) for selection XX- associative recognition at output layer- associative recognition at output layer

1515


OlshausenOlshausen seeks to achieve seeks to achievetranslation-rotation invariant translation-rotation invariant recognitionrecognition

only attended itemreaches outputlayer

©J.K.Tsotsos ©J.K.Tsotsos 3030Niebur, Koch et al. 1993+Niebur, E., Koch, C., Rosin, C. (1993). An oscillation-based model for the neural basis of attention, Vision Research 33, 2789-2802. Niebur, E., Koch, C. (1994). A model for the neuronal implementation of selective visual attention based on temporal correlation among neurons, J. Comput. Neuroscience 1(1), 141- 158.Usher, M., Niebur, E. (1996). Modeling the temporal dynamic of IT neurons in visual search: A mechanism for top-down selective attention, J. Cognitive Neuroscience 8:4, 311-327.

Niebur, Koch et al. 1993+Niebur, Koch et al. 1993+Niebur, E., Koch, C., Rosin, C. (1993). An oscillation-based model for the neural basis ofNiebur, E., Koch, C., Rosin, C. (1993). An oscillation-based model for the neural basis of attention, attention, Vision Research 33Vision Research 33, 2789-2802. , 2789-2802. Niebur, E., Koch, C. (1994). A model for the neuronal implementation of selective visualNiebur, E., Koch, C. (1994). A model for the neuronal implementation of selective visual attention based on temporal correlation among neurons, attention based on temporal correlation among neurons, J. Comput. Neuroscience 1(1),J. Comput. Neuroscience 1(1), 141 141-- 158 158..Usher, M., Niebur, E. (1996). Modeling the temporal dynamic of IT neurons in visual search: AUsher, M., Niebur, E. (1996). Modeling the temporal dynamic of IT neurons in visual search: A mechanism for top-down selective attention, mechanism for top-down selective attention, J. Cognitive Neuroscience 8:4J. Cognitive Neuroscience 8:4, 311-327, 311-327..

Key ideas:Key ideas:- selection by the Koch & Ullman mechanism- selection by the Koch & Ullman mechanism- attentional modulation is added at V1 and affects only the temporal structure of - attentional modulation is added at V1 and affects only the temporal structure of the spike trains of V1 neurons but not their mean firing rate. the spike trains of V1 neurons but not their mean firing rate. - the existence of frequency-selective inhibitory interneurons are assumed in V4- the existence of frequency-selective inhibitory interneurons are assumed in V4- selective attention activates competition within a microcolumn of neurons in V4. - selective attention activates competition within a microcolumn of neurons in V4. - outputs of V1 neurons are tagged, their postsynaptic targets in V4 will win in the - outputs of V1 neurons are tagged, their postsynaptic targets in V4 will win in the V4 level competition. V4 level competition. - no attentional effects on firing rates in V1, only in V4 or higher areas. - no attentional effects on firing rates in V1, only in V4 or higher areas. - refinement in 2nd paper proposes firing coincidences among V2 neurons is sufficient.- refinement in 2nd paper proposes firing coincidences among V2 neurons is sufficient.

- - no attentional effects on firing rates in V1, only in V4 or higher areas. no attentional effects on firing rates in V1, only in V4 or higher areas. XX - temporal synchrony/coincidence- temporal synchrony/coincidence ??

1616

©J.K.Tsotsos ©J.K.Tsotsos 3131Postma 1994 SCANPostma 1994 SCANPostma, E., et al. (1997). SCAN: a scalable model of attentional selection, Postma, E., et al. (1997). SCAN: a scalable model of attentional selection, Neural NetworksNeural Networks 10(6): 10(6): 993-1015.993-1015.

Key ideas:Key ideas:- hierarchical network of gating lattices (pyramid)- hierarchical network of gating lattices (pyramid)- bottom-up WTA - leads to an attentional pathway- bottom-up WTA - leads to an attentional pathway


classifierclassifier(follows ART)(follows ART)

controlcontrolnetworknetwork

gatinggatingnetworknetwork

thresholdthreshold

datadatapartpart

1717

©J.K.Tsotsos ©J.K.Tsotsos 3333Desimone & Duncan 1995 Biased CompetitionDesimone, R., Duncan, J. (1995). Neural mechanisms of selective visual attention, Annual Reviews of Neuroscience 18, 193-222.

Desimone & Duncan 1995 Biased CompetitionDesimone & Duncan 1995 Biased CompetitionDesimone, R., Duncan, J. (1995). Neural mechanisms of selective visual attention, Desimone, R., Duncan, J. (1995). Neural mechanisms of selective visual attention, AnnualAnnual Reviews of Neuroscience 18 Reviews of Neuroscience 18, 193-222., 193-222.

Key ideas:Key ideas:- all stimuli in visual field participate in a competition; interactions due to different objects - all stimuli in visual field participate in a competition; interactions due to different objects activating the same neurons are mutually suppressive activating the same neurons are mutually suppressive- strength of competitive interactions depends inversely on distance between stimuli- strength of competitive interactions depends inversely on distance between stimuli- bias to favour one stimulus in a cluttered field can arise through many mechanisms- bias to favour one stimulus in a cluttered field can arise through many mechanisms- feedback bias is not only spatial, but can be for a feature- feedback bias is not only spatial, but can be for a feature- main source of top-down bias is working memory, prefrontal cortex- main source of top-down bias is working memory, prefrontal cortex- result is the suppression of the neuronal representations of behaviourally irrelevant stimuli - result is the suppression of the neuronal representations of behaviourally irrelevant stimuli in extrastriate cortex in extrastriate cortex

©J.K.Tsotsos ©J.K.Tsotsos 3434Schneider 1995 VAMSchneider, W. X. (1995). VAM: neuro-cognitive model for visual attention control of segmentation, object recognition, and space-based motor action, Visual Cognition 2, 331-375.

Schneider 1995 VAMSchneider 1995 VAMSchneider, W. X. (1995). VAM: neuro-cognitive model for visual attention control ofSchneider, W. X. (1995). VAM: neuro-cognitive model for visual attention control of segmentation, object recognition, and space-based motor action, segmentation, object recognition, and space-based motor action, Visual Cognition 2Visual Cognition 2, 331-375., 331-375.

Key ideas:Key ideas:- selection-for-object-recognition and selection-for-space-based-motor-action- selection-for-object-recognition and selection-for-space-based-motor-action- follows von der Malsburg model- follows von der Malsburg model- what-based attention identifies locations that share features with the target- what-based attention identifies locations that share features with the target- where-based attention locates differences among local stimulus elements- where-based attention locates differences among local stimulus elements- inhibition of return- inhibition of return

(1) (1) The color representationThe color representationof the object is globallyof the object is globallysegmented by this top-downsegmented by this top-downsignalsignal

(2) (2) feedforward flow of anfeedforward flow of anattentional signal from V1 toattentional signal from V1 tothe type-levelthe type-level

attentionselection in V1

1818


(1) (1) The color representation ofThe color representation ofthe object is globallythe object is globallysegmented by this top-downsegmented by this top-downsignalsignal

(2) (2) feedforward flow of anfeedforward flow of anattentional signal from V1 toattentional signal from V1 tothe type-levelthe type-level

(3) (3) where-based attentionalwhere-based attentionalcontrol selects a regioncontrol selects a region

(4) (4) attentional signal to theattentional signal to thetype-level modulestype-level modules

what and where based endogenous attentional control

©J.K.Tsotsos ©J.K.Tsotsos 3636LaBerge 1995 Triangular Circuit ModelLaBerge, D. (1995). Attentional processing: The brain's art of mindfulness. Cambridge, MA: Harvard University Press.

LaBerge 1995LaBerge 1995 Triangular Circuit ModelTriangular Circuit ModelLaBerge, D. (1995). LaBerge, D. (1995). Attentional processing: The brain's art of mindfulnessAttentional processing: The brain's art of mindfulness. Cambridge,. Cambridge, MA: Harvard University Press. MA: Harvard University Press.

Key ideas:Key ideas:- attention requires 3 simultaneous activities: expression, enhancement, control- attention requires 3 simultaneous activities: expression, enhancement, control- expression - clusters of neurons in posterior and anterior cortex- expression - clusters of neurons in posterior and anterior cortex- enhancement - thalamic nuclei excitatory neurons activate neurons in cortical columns- enhancement - thalamic nuclei excitatory neurons activate neurons in cortical columns- control clusters of neurons in frontal cortex- control clusters of neurons in frontal cortex

1919

©J.K.Tsotsos ©J.K.Tsotsos 3737Itti 1998Itti, L., Koch, C., Niebur, E. (1998). A model for saliency-based visual attention for rapid scene analysis, IEEE Trans. Pattern Analysis and Machine Intelligence 20, 1254-1259.

Itti 1998Itti 1998Itti, L., Koch, C., Niebur, E. (1998). A model for saliency-based visual attention for rapidItti, L., Koch, C., Niebur, E. (1998). A model for saliency-based visual attention for rapid scene analysis, scene analysis, IEEE Trans. Pattern Analysis and Machine Intelligence 20IEEE Trans. Pattern Analysis and Machine Intelligence 20, 1254-1259., 1254-1259.

Key ideas:Key ideas:- a newer implementation of Koch and Ullman’s scheme- a newer implementation of Koch and Ullman’s scheme- fast and parallel pre-attentive extraction of visual features across 50 spatial maps - fast and parallel pre-attentive extraction of visual features across 50 spatial maps (for orientation, intensity and color, at six spatial scales) (for orientation, intensity and color, at six spatial scales)- features are computed using linear filtering and center-surround structures - features are computed using linear filtering and center-surround structures - these features form a saliency map- these features form a saliency map- Winner-Take-All neural network to select the most conspicuous image location- Winner-Take-All neural network to select the most conspicuous image location- inhibition-of-return mechanism to generate attentional shifts - inhibition-of-return mechanism to generate attentional shifts - saliency map topographically encodes for the local conspicuity in the visual scene, - saliency map topographically encodes for the local conspicuity in the visual scene, and controls where the focus of attention is currently deployed and controls where the focus of attention is currently deployed


2020


Cave 1999 Feature GateCave. K. (1999). The FeatureGate model of visual selection, Psychological Res. 62: 182-194.Cave 1999 Feature GateCave 1999 Feature GateCave. K. (1999). The FeatureGate model of visual selection, Cave. K. (1999). The FeatureGate model of visual selection, Psychological Res. 62Psychological Res. 62: 182-194.: 182-194.

Key ideas:Key ideas:- hierarchy of spatial maps encode features- hierarchy of spatial maps encode features- inhibition of distractor locations during an attentive task- inhibition of distractor locations during an attentive task- flow of information is governed by a set of gates that control competition and prevent- flow of information is governed by a set of gates that control competition and prevent interference interference- inhibition is applied at several levels of hierarchy to inhibit distractor locations- inhibition is applied at several levels of hierarchy to inhibit distractor locations- selection based on local differences in a bottom-up WTA (with top-down biases)- selection based on local differences in a bottom-up WTA (with top-down biases)- inhibition of return- inhibition of return

©J.K.Tsotsos ©J.K.Tsotsos 4040Hierarchy of MapsHierarchy of Maps

top-down : chooseslocationsthat share features with thetarget

2121


levels of spatial map hierarchylevels of spatial map hierarchy

attentional gatesattentional gates

width indicates signal strengthwidth indicates signal strength


2222


ConclusionsConclusions

Several ideas have endured:Several ideas have endured:

FF Winner-Take-All for selection (competition) Winner-Take-All for selection (competition)

FF Hierarchies Hierarchies

FF Inhibition of return to force serial search Inhibition of return to force serial search

FF Some kind of ‘gating’ Some kind of ‘gating’ processprocess

FF Inhibitory surrounds Inhibitory surrounds

FF However, modelling seems to be still in its early days However, modelling seems to be still in its early days

FF Progress will depend on whether modellers and Progress will depend on whether modellers and experimenters can work togetherexperimenters can work together

Documents

Neurobiological Models of Visual Attentiontsotsos/Homepage of John K_files/Models... · 2000. 10. 20. · Theories/Models The number of models that address ... Müller (1873) re-inforcement