On Natural Scenes Analysis, Sparsity and Coding Efficiency

On Natural Scenes Analysis, Sparsity

and Coding Efficiency

Redwood Center forTheoretical NeuroscienceUniversity of California, Berkeley

Mind, Brain & ComputationStanford University

Vivienne Ming

Adapted by J. McClelland for PDP class, March 1, 2013

Rev jlm 3/5/2010Natural Scenes AnalysisVivienne Ming, Ph.D.

Two Proposals Natural Scene Analysis

Neural/cognitive computation can only be fully understood in “naturalistic” contexts

Efficient (Sparse) Coding TheoryNeural computation should follow

information theoretic principles


Classical Physiology



+



+


Reverse Correlation

Jones and Palmer (1987)


Limits of Classical Physiology

Assumes units (neurons) are linear so known nonlinearities are "added on" to the models

Contrast sensitivity “Non-classical receptive fields” Two-tone inhibition ETC.

Assumes that units operate independently activity of one cell doesn't depend on the activity of others i.e., characterizing cell-by-cell equivalent to characterizing the whole

population of evolution and development, drifting gratings

and white noise are very "unnatural“ Is it possible that our sensory systems are functionally adapted to the

statistics of “natural” (evolutionarily relevant) signals? Would this adaptation affect our characterization of cells?


Response to Natural MovieClassical Receptive

Field Response

Response in“Context”


Limits of Classical Physiology

Assumes units (neurons) are linear so known nonlinearities are "added on" to the models

Contrast sensitivity “Non-classical receptive fields” Two-tone inhibition ETC.

Assumes that units operate independently activity of one cell doesn't depend on the activity of others i.e., characterizing cell-by-cell equivalent to characterizing the whole

population

Finally, in terms of evolution and development, drifting gratings and white noise seem very "unnatural“ Is it possible that our sensory systems are functionally adapted to the

statistics of “natural” (evolutionarily relevant) signals? Would this adaptation affect our characterization of cells? How can we test this?


Efficient Coding Theory Barlow (1961); Attneave (1954)

Natural images are redundantStatistical dependencies amongst pixel

values in space and time An efficient visual system should

reduce redundancyRemoving statistical dependencies


Information TheoryShannon (1949)

Optimally efficient codes reflect the statistics of target signals


Natural Scenes Analysis:First-Order StatisticsN

aïve

Mod

els


Natural Scenes Analysis:First-Order Statistics

Histogram Equalization

Intensity Histogram


Natural Scenes Analysis:Second-Order Statistics






Spatial CorrelationsCompare intensityat this pixel

To the intensityat this neighbor


Spatial Correlations


The Ubiquitous .

Flat (White) PowerSpectrum

f1


Example: synthetic 1/f signals


Natural Scenes Analysis:Principal Components Analysis

PCA Rotation Whitening

Information theory saysthis is an ideal code.

No redundancy


PCA vs. Center Surround


Natural Scenes Analysis:Higher-Order Statistics

PCA Rotation Whitening

Principle dimensions of variation don’t align with data’s intrinsic structure


Natural Scenes Analysis:Higher-Order Statistics

Need a more powerful learning algorithmIndependent Component Analysis (ICA)


Which are the independent components in the scene

below?


M

mmm tstx

1

)()(

+_______

+=


The Modelx = s + n

Overcomplete: #(s) >> #(x) Factorial: p(s) = i p(si) Sparse: p(si) = exp(g(si))

Where g(.) is some non-Gaussian distribution e.g., Laplacian: g(s) = −|s| e.g., Cauchy: g(s) = −log(2 + s2)

The noise is assumed to be additive Gaussian n ~ N(0, 2I)

Goal: find dictionary of functions, , such that coefficients, s, are as sparse and statistically independent as possible

Information Theorydemands sparseness


Learning log likelihood L() = <log p(x|)> Learning rule:

Basically the delta rule:

D = (x − s)sT

Impose constraint to encourage the variances of each s to be approximately equal to prevent trivial solutions

Usually whiten the inputs before learning Forces network to find structure beyond second-order Increases stability

DL


Sparsity


?


Efficient Auditory CodingSmith & Lewicki (2006)

Extend Olshausen (2002) to deal with time-varying signalse.g., sounds or movies

Train the network on “Natural” soundsEnvironmental TransientsEnvironmental AmbientsAnimal Vocalizations



Cat ANF Revcor Filters


Efficient Kernels


Population Coding


Population Coding


Population Coding


Population Coding


Population Coding


Speech


Speech


Speech


Efficient Coding Literature Empirical

Weliky, Fiser, Hunt & Wagner (2003) Vinje & Gallant (2002) DeWeese, Wehr & Zador (2003) Laurent (2002) Theunissen (2003)

Theoretical Field (1987) van Hateren (1992) Simoncelli & Olshausen (2001) Olshausen & Field (1996) Bell & Sejnowski (1997) Hyvarinen & Hoyer (2000) Smith & Lewicki (2006) Doi & Lewicki (2006)


Hierarchical Structure? Can we identify interesting structure in the

world by looking at higher order statistics of the activations of the linear features discovered by the first-order model? Karklin and Lewicki (2005) looked for patterns

at the level of the variances of the linear features.

Karklin and Lewicki (2009) looked for patterns at the level of the covariances of the linear features.


Looking at Hierarchical Structure










Generalizing the standard ICA model


Generalizing the standard ICA model

Instead of:

we now have units u and v such that


Independent density components


Karklin & Lewicki (2009)

The model tries to find the values of the yj’s that lead to a combined covariance matrixC that matches the covariance of the data represented by activities across first-level filters.

The learning process involves a search for vectors bk and weights wjk that allow themodel to fit the data while keeping the yj’s sparse and independent.





Responses of Cell to Gratings



Efficient Coding SummaryStatistic Computatio

nAlgorithmExample

Biological Example Reference

1st-orderContrast

gain control

Histogram equalizatio

n

Retina or H1

adaptationFairhall et al.

(2001)

2nd-order Whitening PCARetinal/

Thalamic coding

Atick (1992)

Higher-order

Sparse Coding

ICA / Sparsenet V1 coding Olshausen &

Field (1996)

Time-varying

Shift-invariance

Efficient Spike

CodingCochlear coding Smith & Lew

icki 2006

Hierarchical

Conditional Independenc

eHierarchical coding ?

Karklin & Lewicki ’05,’09

http://www.nature.com/nature/journal/v412/n6849/abs/412787a0.html



http://www.iop.org/EJ/article/0954-898X/3/2/009/ne920209.pdf?request-id=4GOMewru3BG973_W2wi7Kg



http://cns.upf.edu/jclub/SmithLewickiNatureFeb06.pdf

http://cns.upf.edu/jclub/SmithLewickiNatureFeb06.pdf

Documents

On Natural Scenes Analysis, Sparsity and Coding Efficiency