week 4 - Efficient Coding II Flashcards
How does decorrelating neurons/pixels reduce redundancy between neurons/pixels?
How does this contribute to the efficient coding hypothesis?
-decorrelation removes the redundant information between the two neurons/pixels
-redundant info is what one neuron already know about the other neuron or overlapping info. this is therefore predictable info. remove redundancy = efficient coding
Why is it not a good efficient coding hypothesis if neighbouring pixels have similar brightness/ are correlated?
we can predict what neuron 1 is going to do from neuron 1 = lots of redundant information
What are the two steps of whitening? what does this look like on a Gaussian linear model graph for each step?
- decorrelate data (making it a vertical line instead of diagonal on graph)
- scale axes to equalize range (make data points one big circle)
How does whitening comply to the ECH?
reduces redundancy by decorrelating data
reduce redundancy = good ECH
What are the three MATHEMATICAL steps in the process of whitening?
- Eigen-decomposition of the covariance matrix
- Rotate data
- Scale data
Does a Fourier DECOMPOSITION comply to ECH? why?
no because there is negative CORRELATION between y =power x = frequency of change (of pixels)
therefore there is redundant information between pixels = not good ECH
What does the linear Gaussian model capture in natural images?
captures pair-wise pixel correlations
What image is produced from applying basic whitening functions? Does it look natural/look like anything occurring in nature??
checkboard receptive fields which don’t look like natural receptive fields
What is the issue with just adding whitening to the linear Gaussian model?
What can you do to fix this?
-whitening = large circle of data on graph
issue= you can make more changes to the model like rotating the circle
-add another constraint and localise the circle in space = localised whitening
What does the image look like when you add LOCALIZED whitening basis functions to the model?
it look like receptive fields in neurons lower down in neural circuit (circle in a bigger circle)
Additionally, what can you do to improve the model once you have added localised whitening basis functions?
-filter out noise and also make energy efficient
What does the image look like when natural images have second-order redundancy removed?
What has happened to the correlation between neighbouring pixels in this image?
- you can still see the structure however the image looks washed out because edges and contrast are missing
-correlation between neighbouring pixels = 0 (correlation/redundant info has been removed)
What does a perfectly decorrelated image look like?
like swirls in a tree stump
What does applying second-order whitening to a Gaussian model look like on a graph?
For second-order whitening, how do you find the FINAL rotation?
-like an x in a big circle data points
-find the direction in the data that are the least Gaussian (skinny pointy peak curve = non-Gaussian)
Why, after localised basis whitening do we find directions in the data that are the ‘least’ Gaussian? (2nd order model)
so we can recover our independent components
Why can’t independent components be Gaussian?
majority of independent components are non-Gaussian -> if they were all Gaussian, we wouldn’t be able to separate them properly because a Gaussian distribution is symmetric and lacks unique structure
Are the independent components non-Gaussian or Gaussian?
What is the central limit theorem (CLT) in terms of Gaussianity?
-independent components are non-Gaussian
-when we mix multiple non-Gaussian signals (independent components), their combination becomes more Gaussian
Why do we want to recover our independent components?
How can we achieve this?
-the independent components are non-Gaussian however, due to CLT, when mixed they have become Gaussian. so we want to recover independent components
-find directions in the data where the output is least Gaussian
What are the three ways we can recover the independent components?
- Maximize a measure of non-Gaussianity (kurtosis)
- Pick non-Gaussian distribution as prior over inputs
- Minimize mutual information between outputs
How is independent component analysis ICA explained with an example of the eyes?
eyes detect signals and mix them in brain. brain then also de-mixes them to find original sources
What does the image look like when you add ICA filters?
like receptive fields in primary visual cortex V1 which are localised and orientation specific
they are Gabor-like
The response properties of retinal ganglion / thalamic visual neurons can be (mostly) explained by ________ ?
decorrelation
Is redundant information and mutual information the same thing?
Why?
-mutual info is how well the system can predict the input from the output
redundant info is information you can predict in input/output
-no because we want to maximise mutual info and minimise redundant info
