Recurrent Networks & Associative Memory Flashcards
How does Pennartz broadly describe the advantages of the Hopfield model as a model of neural networks?
computationally tractable, yet very insightful
Performatively what are the advantages of the Hopfield model? (5)
*Capable of handling noisy, fuzzy and/or incomplete information (“pattern completion”)
*Robust and fault tolerant; “graceful degradation” (when network loses cells)
*High learning capacity and flexibility; adapt to new environmental input without the need to be re-programmed
*Massive parallelism; many simultaneous operations
*High speed of computation (# logical operations/sec.)
How does the architecture of a recurrent neural network differ to that of a CNN? What is it often used to model?
Often used for long-term associative memory or working memory
Recurrent neural network: all (or most) neurons project to each other:
* learning capacity distributed across recurrent (“associative”) connections
* connectivity can be “full” or partial / sparse
What is a prototypical case of these recurrent auto-associative memory networks?
Hopfield model of (auto)-associative memory
What is a requirement of the Hopfield model
Store a set of patterns in such a way that when presented with a new pattern P, the network responds by producing whichever one of the stored patterns most closely resembles P
What are the chief features of a Hopfield model?
- fully connected, recurrent network (but no autapses; self activation)
- can do asynchronous parallel processing
- content-addressable memory
How would an image look in the Hopfield network
Black and white images in which each picture pixel
is a neuron (black=inactive, white=active; all connected)
How are these Hopfield networks related to Alzheimers?
Take away synapses and neurons –> Still able to recognise pattern –> related to Alzheimer, only a huge loss leads to visible changes or decline
What do the train and test stages look like for a Hopfield network?
Neural networks are first trained on input patterns,
and then tested on a cognitive problem
Describe the training phase
Training phase of auto-associative memory task:
* Input pattern is presented; tuned neurons are activated
* Distributed pattern of activity leads to distributed weight changes (Δwi,j)
* Each new trial (iteration) leads to an input pattern being ‘imprinted’ which is implemented by updating of synaptic weights (~LTP, LTD)
What does the test phase consist of?
Present test pattern: does network complete/retrieve the pattern?
What could this imprinting be compared to in ecology?
imprinting in newborn animals:
Grouse chick imprints mother features
=> Chick recognises mother even when occluded/noisy/partially complete
(Use: memory & Gestalt grouping principles)
In neurobiological networks, do neurons code “pixels”?
No, rather: features, present in a receptive field
Small RF, V1 simple cells; simple stimuli
Large RF, e.g. inferotemporal cortex; more complex stimuli, size invariance
What activation function does the Hopfield network use?
ai=1 if Σwijaj > Threshold value (usually 0) (bi)
ai=0 if Σwijaj < Threshold value
What learning rule does the Hopfield network utilise? Describe what we want to do conceptually
Learning rule / information storage algorithm:
We want to “imprint” a pattern p in the network, basically
according to Hebb’s rule:
* case 1: pre- and postsynaptic activity correlated (both ai= +1)
=> set their connecting synapse to a value of +1 (also: if both ai=0, aj= 0)
- case 2: pre- and postsynaptic cell anti-correlated
(only one neuron with a= +1, the other 0)
=> set their connecting synapse to a value of -1
How do we compute this rule mathematically?
Δwij = (2ai-1)(2aj-1):
Δwij = (2(1)-1)(2(1)-1) = 1
Δwij = (2(0)-1)(2(0)-1) = 1
Δwij = (2(1)-1)(2(0)-1) = -1
Next: store a set of N patterns (states) in the network:
Δwij = Σ(N_p=1) (2ap_i -1)(2ap_j-1)
What happens if you store too many patterns relative to the number of neurons?
If you store too many patterns relative to the amount of neurons that you have you have a cancellation effect
Is all of this physiologically plausible?
Firstly separating learning from testing is a lil artificial and being fully connected is not super realistic
Secondly this model works in terms of LTP; both neurons excite each other and strengthen synapse. This doesn’t really make sense with LTD however, as activation is required for LTD so a=0 could not be the cause (?):
Change of the weight between pre to post neuron = modification of postsynaptic activity (+1)
Two neurons have activity of +1 = 1 product of equation is = 1
Two neurons have activity of 0 = 1 –> LTP
One neuron a = 1 and other neuron a = 0 –> -1 –> LTD
How does the network evolve from one state to the next?
- We pick out two neurons, which were anti-correlated during “imprinting”
- This implies that their reciprocal weights should be negative
- Recall phase: during the “test” phase, no further weight changes occur
Describe a “tense situation” and how it is resolved in a recurrent network. Does this translate to biology?
“Tense situation”: activity states do not conform to pre-set weights. If two neurons both had an activation of 0 previously and therefore had a negative weight, however now down have an activation of one. We rely on existing weights that have been established during learning. One neuron will
win out over the other neuron (due to noise). Now the activity state is anticorrelated, according to the pre-set weights:
a ”relaxed situation”; Evolving to the attractor state
Biology e.g. Once one of the two neurons inhibits the other, the second can’t inhibit the first one anymore –> one neuron will win out over the other
