Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Entropy and Shannon information
For a random variable X with distribution p(x), the entropy is H[X] = - Sx p(x) log2p(x)
Information is defined as I[X] = - log2p(x)
Entropy and Shannon information
Typically, “information” = mutual information: how much knowing the value of one random variable r (the response) reduces uncertainty about another random variable s (the stimulus). Variability in response is due both to different stimuli and to noise. How much response variability is “useful”, i.e. can represent different messages, depends on the noise. Noise can be specific to a given stimulus.
Mutual information
Information quantifies how independent r and s are: I(S;R) = DKL [P(R,S), P(R)P(S)]
I(S;R) = H[R] – Ss P(s) H[R|s] .
Alternatively:
Mutual information
Need to know the conditional distribution P(s|r) or P(r|s). Take a particular stimulus s=s0 and repeat many times to obtain P(r|s0). Compute variability due to noise: noise entropy
Mutual information is the difference between the total response entropy and the mean noise entropy: I(S;R) = H[R] – Ss P(s) H[R|s)] .
Mutual information
Information is symmetric in r and s
Extremes: 1. response is unrelated to stimulus: p[r|s] = ?, MI = ? 2. response is perfectly predicted by stimulus: p[r|s] = ?
Mutual information
r+ encodes stimulus +, r- encodes stimulus -
Simple example
but with a probability of error: P(r+|+) = 1- p P(r-|-) = 1- p What is the response entropy H[r]? What is the noise entropy?
Entropy Information
Entropy and Shannon information
H[r] = -p+ log p+ – (1-p+)log(1-p+)
H[r|s] = -p log p – (1-p)log(1-p)
When p+ = ½,
Noise limits information
Channel capacity
A communication channel SR is defined by P(R|S)
I(S;R) = Ss,r P(s) P(r|s) log[ P(r|s)/P(r) ]
The channel capacity gives an upper bound on transmission through the channel:
C(R|S) = sup I(S;R)
Source coding theorem
Perfect decodability through the channel:
T
If the entropy of T is less than the channel capacity, then T’ can be perfectly decoded to recover T.
S R T’ encode transmit decode
Data processing inequality
Transform S by some function F(S):
R
The transformed variable F(S) cannot contain more information about R than S.
S F(S) encode transmit
How can one compute the entropy and information of spike trains? Entropy:
Strong et al., 1997; Panzeri et al.
Discretize the spike train into binary words w with letter size Dt, length T. This takes into account correlations between spikes on timescales TDt. Compute pi = p(wi), then the naïve entropy is
Calculating information in spike trains
Many information calculations are limited by sampling: hard to determine P(w) and P(w|s) Systematic bias from undersampling. Correction for finite size effects:
Strong et al., 1997
Calculating information in spike trains
Information : difference between the variability driven by stimuli and that due to noise. Take a stimulus sequence s and repeat many times. For each time in the repeated stimulus, get a set of words P(w|s(t)). Average over s average over time: Hnoise = < H[P(w|si)] >i. Choose length of repeated sequence long enough to sample the noise entropy adequately. Finally, do as a function of word length T and extrapolate to infinite T.
Reinagel and Reid, ‘00
Calculating information in spike trains
Fly H1:
obtain information rate of
~80 bits/sec or 1-2 bits/spike.
Calculating information in spike trains
Another example: temporal coding in the LGN (Reinagel and Reid ‘00)
Calculating information in the LGN
Apply the same procedure:
collect word distributions
for a random, then repeated stimulus.
Calculating information in the LGN
Use this to quantify how
precise the code is,
and over what timescales
correlations are important.
Information in the LGN
How much information does a single spike convey about the stimulus?
Key idea: the information that a spike gives about the stimulus is the reduction
in entropy between the distribution of spike times not knowing the stimulus,
and the distribution of times knowing the stimulus.
The response to an (arbitrary) stimulus sequence s is r(t).
Without knowing that the stimulus was s, the probability of observing a spike
in a given bin is proportional to , the mean rate, and the size of the bin.
Consider a bin Dt small enough that it can only contain a single spike. Then in
the bin at time t,
Information in single spikes
Now compute the entropy difference: ,
Assuming , and using
In terms of information per spike (divide by ):
Note substitution of a time average for an average over the r ensemble.
prior
conditional
Information in single spikes
Given
note that: • It doesn’t depend explicitly on the stimulus • The rate r does not have to mean rate of spikes; rate of any event. • Information is limited by spike precision, which blurs r(t), and the mean spike rate.
Compute as a function of Dt:
Undersampled for small bins
Information in single spikes
Adaptation and coding efficiency
1. Huge dynamic range: variations over many orders of magnitude
Natural stimuli
1. Huge dynamic range: variations over many orders of magnitude
2. Power law scaling: highly nonGaussian
Natural stimuli
Natural stimuli
1. Huge dynamic range: variations over many orders of magnitude
2. Power law scaling: highly nonGaussian
Natural stimuli
1. Huge dynamic range: variations over many orders of magnitude
2. Power law scaling: highly nonGaussian
In order to encode stimuli effectively, an encoder should match its outputs to the statistical distribution of the inputs
Shape of the I/O function should be determined by the distribution of natural inputs Optimizes information between output and input
Efficient coding
Laughlin, ‘81
Fly visual system
Contrast varies hugely in time. Should a neural system optimize over evolutionary time or locally?
Variation in time
For fly neuron H1, determine the input/output relations throughout the stimulus presentation
A. Fairhall, G. Lewen, R. R. de Ruyter and W. Bialek (2001)
Time-varying stimulus representation
Extracellular in vivo recordings of responses to whisker motion in rat S1 barrel cortex in the anesthetized rat
M. Maravall et al., (2007)
Barrel cortex
r (s
pik
es/s
)
r (s
pik
es/s
)
R. Mease, A. Fairhall and W. Moody, J. Neurosci.
Single cortical neurons
Using information to evaluate coding
As one changes the characteristics of s(t), changes can occur both in the feature and in the decision function
Barlow ’50s, Laughlin ‘81, Shapley et al, ‘70s, Atick ‘91, Brenner ‘00
Adaptive representation of information
Barlow ’50s, Laughlin ‘81, Shapley et al, ‘70s, Atick ‘91, Brenner ‘00
Feature adaptation
The information in any given event can be computed as:
Define the synergy, the information gained from the joint symbol:
or equivalently,
Negative synergy is called redundancy.
Synergy and redundancy
Brenner et al., ’00.
In the identified neuron H1, compute information in a spike pair, separated
by an interval dt:
Multi-spike patterns