Embed Size (px)
Citation preview
Image Stabilization by Bayesian Dynamics
Yoram BurakSloan-Swartz annual meeting, July 2009
What does neural activity represent?
In Bayesian models: probabilities
Direction of motion: single, static variable
Accumulated evidence in area LIPShadlen and Newsome (2001)
What does neural activity represent?
In Bayesian models: probabilities
Direction of motion: single, static variable
What about multi-dimensional, dynamic quantities?
Accumulated evidence in area LIPShadlen and Newsome (2001)
Foveal vision and fixational drift
Foveal vision and fixational drift
By XaqPitkow
- between micro-saccades -~20 receptive fields
Image from: X. Pitkow
- between spikes (100 Hz) -~2-4 receptive fields !
Fixational drift is large in the fovea:
cone separation: 0.5 arcmin
Foveal vision and fixational drift
By XaqPitkow
- between micro-saccades -~20 receptive fields
Image from: X. Pitkow
- between spikes (100 Hz) -~2-4 receptive fields !
Downstream areas require knowledge
of trajectory to interpret spikes
Fixational drift is large in the fovea:
cone separation: 0.5 arcmin
Joint decoding of image and position
Bayesian:
Discrimination task: vs. X. Pitkow et al, Plos Biology (2007)
N x 2 probabilities
# positions
Bayesian:
Discrimination task: vs. X. Pitkow et al, Plos Biology (2007)
N x 2 probabilities
Unconstrained image 30 x 30 binarypixels
# positions
N x 2900 probabilities
Joint decoding of image and position
Bayesian:
Discrimination task: vs. X. Pitkow et al, Plos Biology (2007)
N x 2 probabilities
Unconstrained image 30 x 30 binarypixels
# positions
N x 2900 probabilities
Can the brain apply a Bayesian approach to this problem?
Joint decoding of image and position
Can the brain apply a Bayesian approach to this problem?
Decoding strategy
Performance in parameter space
What are the biological implications?
Can the brain apply a Bayesian approach to this problem?
Decoding strategy
Performance in parameter space
What are the biological implications?
Decoding strategy
Discards information about correlations
Factorized representation:
Decoding strategy
Discards information about correlations
minimizeDKL
Factorized representation:
Exact if trajectory is known.
evidence, diffusion
Update dynamics:
Decoding strategy
Discards information about correlations
minimizeDKL
Factorized representation:
Exact if trajectory is known.
evidence, diffusion
evidence - Poisson spiking (rate λ1 for on pixels, λ0 for off)diffusion - Random walk (diffusion coefficient D)
Retinal encoding model:
Update dynamics:
Decoding strategy
Discards information about correlations
Neural Implementation - Two populations: where , what
For 30 x 30 pixels: N × 2900 → N + 900
quantities.
Factorized representation:
Update rulesUpdate of what neurons:
multiplicative gating
Ganglion cells
What
Where
nonlinearity
Update rulesUpdate of what neurons:
Update of where neurons:
multiplicative gating
Ganglion cells
What
Where
Where
What
multiplicative gating
Ganglion cells
+ diffusion
nonlinearity
Demo
image
retina
m x m binary pixels
2d diffusion (D)
Poisson spikes:100 Hz (on), 10 Hz (off)
Decoder
Demo
Decoding strategy
Performance in parameter space
What are the biological implications?
Can the brain apply a Bayesian approach to this problem?
Performance
D D
Con
verg
en
ce t
ime [
s]
acc
ura
cy
Performance degrades with larger D (and smaller λ)
Performance
D D
Con
verg
en
ce t
ime [
s]
Faster and more accurate for larger images
m = 5, 10, 30, 50, 100
acc
ura
cy
Demo
Performance
D D
Con
verg
en
ce t
ime [
s]
Faster and more accurate for larger images
acc
ura
cy
m = 5, 10, 30, 50, 100
Performance
D D
Con
verg
en
ce t
ime [
s]
Faster and more accurate for larger images
acc
ura
cy
m = 5, 10, 30, 50, 100
Performance
D D
Con
verg
en
ce t
ime [
s]
Faster and more accurate for larger images
acc
ura
cy
m = 5, 10, 30, 50, 100
Performance
D/m D/m
Con
verg
en
ce t
ime [
s]
acc
ura
cy
scales with linear image size m
m x m pixels
Performance
D/m D/m
Con
verg
en
ce t
ime [
s]
acc
ura
cy
scales with linear image size m
Analytical scaling:
D*
m x m pixels
Performance
Performance improves with image size.
Success for images 10 x 10 or larger
Prediction for psychophysics:
Degradation in high acuity tasks when visual scene
contains little background detail.
Temporal response of Ganglion cells
Common view: fixational motion important to activate cells, due to biphasic response
f(t)
t
Temporal response makes decoding much more difficult.
50 ms
Need history
Non-Markovian:
Temporal response of Ganglion cells
Approach: Choose decoder that is Bayes optimal if the trajectory is known.
What
Ganglion
“filteredtrajectory”
Where
history dependent decoder / naive decoder
Converg
ence
tim
e [
s]
acc
ura
cy
D D
Temporal response of Ganglion cells
Is fixational motion beneficial?
Known trajectory , perfect inhibitory balanceC
onverg
ence
tim
e [
s]
D
Optimal D - order of magnitude smaller than biological value
Can the brain apply a Bayesian approach to this problem?
Decoding strategy
Performance in parameter space
What are the biological implications?
Network architecture
Each ganglion cell innervates multiple what & where cells(spread: ~10 arcmin)
WhereWhat
Ganglion
Reciprocal, multiplicative gating
Activity:
What neuronsSlow dynamics, evidence accumulation
Where neuronsFewer. Highly dynamic activityTonic, sparse in retinal stabilization conditions.
Activity:
What neuronsSlow dynamics, evidence accumulation
Where neuronsFewer. Highly dynamic activityTonic, sparse in retinal stabilization conditions.
Where in the brain?
Monocular
LGN?
V1?
If so, suggests LGN or V1
Modulatory inputs to relay cells (gating?)
Lateral connectivity in where network, Increase in number of neurons.
SummaryStrategy for stabilization of foveal visionFactorized Bayesian approach to multi-dimensional inference
SummaryStrategy for stabilization of foveal vision
Explicit representation of stabilized image“What” and “where” populations
Factorized Bayesian approach to multi-dimensional inference
SummaryStrategy for stabilization of foveal vision
Explicit representation of stabilized image“What” and “where” populations
Good performance at 1 arcmin resolutionProblem is easier for large images, for coarser reconstruction
Factorized Bayesian approach to multi-dimensional inference
SummaryStrategy for stabilization of foveal vision
Explicit representation of stabilized image“What” and “where” populations
Good performance at 1 arcmin resolutionProblem is easier for large images, for coarser reconstruction
Factorized Bayesian approach to multi-dimensional inference
Network architecture:Many-to-one inputs from retina, multiplicative gating (what/where)
Uri Rokni
Haim Sompolinsky Markus Meister
Special thanks - the Swartz foundation
Acknowledgments
