View
215
Download
0
Category
Tags:
Preview:
Citation preview
Stochastic Pooling Networks:On the interaction between redundancy,noise and lossy compression in biological neurons.
Mark D. McDonnellUniversity of South Australia
Pierre-Olivier AmblardCNRS, Grenoble, France
Nigel G. StocksUniversity of Warwick, UK
6 July 2008
What is a Stochastic Pooling Network?
What do SPNs Model?
Emergent properties of SPNs
This talk is about understanding how biological neurons efficiently compress stimuli during coding.
Real signals are analog but nearly everything in modern electronics is digital. Why is this?
X
XCEO
R1
RN
R2
N
ii RR
1
x
x
xN
1
2
N
1 2
What mechanisms do biological sensory systems use to compress information during transduction?
There is lossy, and lossless compression… and maybe something in between: “loss-least!”
Information theory underpins fast and accurate communication. But it is also about selecting what information is worth storing and communicating.
A “Recipe” for Stochastic Pooling Networks
Ingredients• 1 information source• N > 1 independent sensors (compressive types)• N > 1 random noise sources
PreparationStep 1: For each sensor, measure the information
source mixed with one of the noise sources, and then compresses its measurement.
Step 2: Ensure the whole network produces a single measurement by pooling each sensor’s.
SPNs can model many sensor network or source coding scenarios
• Digital Beamforming Arrays– Sonar, radar, MIMO communications
• Digital signal processing– Noise reduction via coherent averaging after digitization in
ADCs.• Distributed/Decentralized sensor networks
– CEO problem– Multiaccess Communication Channels– Power constrained wireless sensor networks
• Biological neurons– Representation of analog stimuli by rate coding action
potentials– Quantal release of chemical neurotransmitters at synaptic
junctions– Maybe some ion channels?
SPNs can model many sensor network or source coding scenarios
• Reliability schemes in nano-electronics– Averaging and redundancy to overcome
parameter variations and noise [Ferran Martorell, Spain]
• Social networks– Subjective voting on a continuous variable
• Quantum optical communication using polarization detection of single photons?
• Coupled multistable dynamic systems?• Reconfigurable chaotic logic gates?
The neurons that code sounds immediately after transduction can be modelled as an SPN
Cochlea Inner Hair Cell
Auditory NerveSound
Ear
Information
The neurons that code sounds immediately after transduction can be modelled as an SPN
Innerhair cell
Basilarmembrane
Brain
Outerhair cell
Slide courtesy of Prof Tony Burkett, Uni. Melbourne
Pooling must occur “naturally,” without external intervention, e.g. adding, or superposition.
We will not have a pooling network otherwise!
We assume combining of the ingredients is left to physics: POOLING
g1 (.)
g2 (.)
gN(.)
1
2
N
x
y1 {0,1,..,M}
y2 {0,1,..,M}
yN {0,1,..,M}
P y = h(y1, ..,yN)
|y| << M
This is a surprising emergent property of SPNs
Pooling loses no (or negligible) information!
Assume the information source, x, is random.The mutual information, I(x,y) loosely measures how well, on average, the SPN output, y, provides a good estimate for x.
McDonnell, Stocks et. al., Fluct. Noise Lett. 5, L457-L468, 2005.
McDonnell, “Applying stochastic signal quantization theory to the robust digitization of noisy analog signals", Book chapter in Springer Verlag Complexity Series, In Press, 2008.
SPNs digitize (quantize) their input
SPNs reduce noise via coherent averaging… but not in a linear way!
• For small noise, performance is limited by compression: I(x,y) < log(1+MN)
• For large noise, performance is limited by “averaging”: I(x,y) < 0.5log(1+N SNR)
N=1,M=511
N=511,M=1
0.5log2(1+NSNR)
There are a number of differences in the final SNR achieved by linear analog averaging vs SPNs
Output SNR Analog SPN
Dependency on source distribution
no yes
Decoding
dependency
N/A yes
Scaling with input SNR
linear -linear for small SNRs,
-nonlinear for large SNRs
Scaling with N
(no. of times averaged)
proportional - Approx. proportional for small SNRs
-no effect for large SNRs
Scaling with M (bits) N/A -increases for small SNRs
-no effect for large SNRs
McDonnell, “Signal Estimation Via Averaging of Coarsely Quantised Signals,” Proc IEEE Information, Decision and Control, Adelaide, Australia, pp 100-105, 2005.
Probability of Error: binary detection.
[Zozor, Amblard and Duchene, Fluct. Noise Lett. 7, L39-L60, 2007]
Simulation of cochlear implant coding.
[Stocks et. al., Proc SPIE, 2007.]
Multiplicative Noise at the input to network nodes.[Nikitin, Stocks and Morse, Phys. Rev. E, 2007]
SSR measured by mutual information foradditive random noise.[Stocks, Phys. Rev. Lett., 2000]
Suprathreshold Stochastic Resonance in SPNs
*Also studies by others, e.g. Rousseau & Chapeau-Blondeau (2003), Hoch et al (2003), Martorell (2005)…
McDonnell, Stocks et. al., “Optimal information transmission in nonlinear arrays through suprathreshold stochastic resonance,” Phys. Lett. A 352, pp. 183-189, 2006.
Optimizing the nodes: more noise means more nodes that are identical
We have observed similar effects for optimized networks of Poisson neuron models, and binary detection networks.
There are many other surprising emergent properties
• Very noisy SPNs behave like analogue Gaussian channels [McDonnell, IEEE Aus. Comm Theory Wkshp, 2008]
• Very large SPNs behave like multiplicative noise channels [McDonnell and Stocks, Proc. SPIE. 2007]
• Optimal reconstruction depends only on the noise distribution and the number of sensors [McDonnell, Stocks and Abbott, Phys. Rev. E 75, Art. No. 061105.]
• Optimizing the noise distribution is like optimizing a neuron’s stimulus-response curve.
• Negative correlation provides improved MI.• The optimal stimulus is actually discrete!
Unsolved problems on SPNs in biology
• Do biological senses really use SPN principles?
• What mechanisms does the brain use to compress/reduce information
• Will the controlled use of random noise in cochlear implants improve the hearing of patients?
If healthy auditory neurons act like SPNs then bionic ears should stimulate them randomly!
Electrode arrayElectrode array
11stst turn turnofof
inner earinner earAuditoryAuditory
nervenervefibresfibres
Image courtesy of Cochlear Ltd, 2008
If healthy auditory neurons act like SPNs then bionic ears should stimulate them randomly!
© Australasian Science, 2008
“Stochastic beamforming coding strategy”,Morse, Holmes, Shulgin, Nikitin and Stocks, 2007
There are many unsolvedtheoretical problems on SPNs
• Can the clustering of nodes be predicted mathematically?
• What further complexities can be added to SPNs without changing the basic ingredients?
Recommended