8 - Recurrent Neural Networks Flashcards
What is the problem with using a time as space model?
-expensive
-how far back in time should you go?
-semantically not correct
How to deal with problems in time as space model?
allow activity to reverberate through the network by using RECURRENT CONNECTIONS -> dynamic network activity
What analogy can you use to explain reverberating activity?
throwing stone in puddle and wave of puddle can tell you where input(stone) came from
In Elman Networks specifically, what is the purpose of feedback weights?
connect the hidden layer to the context layer
Elman Network - What sort of activity is seen between hidden layer and context layer?
reverberating activity
Why cant a perceptron (single rate neuron model) perform an XOR function?
output is not linearly separable
What can solve the XOR problem?
with added layers of neurons !!
Are there any feedback connections in the feed forward model ? (outputs which feedback into itself)
noooooooooooooo (called feedforward rate networks for a reason)
What is the added additional layer in the Elman Network compared to a Feedforward Network?
context layer
What tool does learning sequence information use to predict the sequential XOR INPUT?
uses backpropagation
When the error is reduced does the ability of the network to predict the next letter/number in the sequence increase or decrease?
increases
Word with similar semantic have similar or different pattern of activity? (with predicting letter and level or error)
similar semantics and similar patterns
What does adding recurrent connections between layers (reverberation activity) allow?
-introduces a new kind of ‘memory’ capacity which enables networks to represent temporal structure in the sequences of training patterns
Why can you read a text which has letters omitted in the words?
What sort of memory is this called?
What network is this memory seen in?
because there is a pattern in language which the brain has stored (Elman)
-associative memory
-Hopfield Network
What is cell assembly in memory?
What is a cell assembly able to do?
-a selection of neurons connected with increased synaptic weights
-this increase in synaptic weights of connections between neurons allows you to store an item in memory
How do cell assemblies contribute to associate memory? (Hebbian)
-items stored via creation of cell assembly
-associated items can be recalled due to activation of cell assembly (this is due to the increase in synaptic weights between neurons in Hebbian Network)
What learning is this: neurons that fire together, wire together?
What does this mean?
Hebbian Learning: neurons that are active together results in strengthened connections between them
In ferromagnets, is the direction of the atom’s spin independent on the neighbouring atom’s spin?
no they are not independent - they can have strong or weak interactions which influence eachother
If a ferromagnet atom gets excited and spins so to point in the opposite direction, what happens afterwards if you apply the Hebbian dynamics eqn.?
How does this relate to Hebbian Learning/Network?
-atom flips back to original orientation
-an example of recalling memory is when the atom flips back as the original orientation is the ‘stored’ memory
What is the definition of a Hopfield Network?
a recurrent network with symmetric weights and no self-connections
In the definition of a Hopfield Network, what does it mean when the weights are symmetric?
the connection from neuron j to i is equal to the connection between i to j
In a Hopfield Network, what is form is the output?
What is the output in a Hopfield Network?
-either active +1 or inactive -1
-output OF EACH UNIT is weighted sum of the inputs pushed through a step function to generate +1 or -1
Do individual neurons interact with themselves in a Hopfield network?
non there are no self-connections
How are memories stored as in a Hopfield Network?
stored as low-energy states of the dynamics
Finish the sentence:
When an axon of cell j repeatedly or persistently takes part in firing cell i,
-What networks is this rule seen in?
then j’s efficiency as one of the cells firing i is increased
-it is the basis of Hebbian Learning but it is also used in Hopfield Network
What does yi/yj represent in Hopfield Network?
“ xi/xj
“ wij
-the activity of the network
-the pattern we want to store
-the weights
What is the dynamics equation for the Hopfield Network?
What does the 𝑗≠𝑖 mean in this eqn.?
𝑦𝑖 (𝑡+1)=𝑠𝑖𝑔𝑛(∑(𝑗≠𝑖) 𝑤𝑖𝑗 𝑦𝑗 (𝑡) )
weighted sum of all neurons connections apart from connecting to itself so 𝑗≠𝑖
What is the weight between two neurons in Hopfield Network equal to?
Energy: 𝐸𝑖𝑗 = −𝑤𝑖𝑗 𝑦𝑖 𝑦𝑗
Why do we set the weights wij in a Hopfield Network?
-to store a pattern x (memorisation)
How do we store and recall a pattern in a Hopfield Network? (two steps)
-set the weights to wij=xixj to store a pattern 𝒙 (memorisation)
-start the activity at 𝒚 that is similar to 𝒙, the dynamics moves 𝒚 toward 𝒙 (recall)
How do you set the weights in memorisation/pattern storage in a Hopfield Network?
1) Start with 𝑤_𝑖𝑗=0
2) To memorise pattern 𝒙^𝑝, update weights: 𝑤𝑖𝑗←𝑤𝑖𝑗+𝑥𝑖^𝑝 𝑥𝑗^𝑝
3) Repeat step (2) for every other pattern
In a Hopfield Network, what does it mean when connections enforce correlation?
How do you recall a pattern in a Hopfield Network? (with associative memory eg. you see an incomplete letter/pattern M which is similar/associated with the stored memory of M (xp)
-start with network activity at y(1)
-For each neuron 𝑖 update activity by 𝑦𝑖(𝑡+1)=𝑠𝑖𝑔𝑛(∑(𝑗≠i) 𝑤𝑖𝑗𝑦𝑗 (𝑡)) until 𝑦𝑖 (𝑡+1)=𝑦𝑖 (𝑡) for every 𝑖
-This “stable” activity will match the pattern 𝑥 that 𝑦(1) is most similar to.
Hopfield Network: Why are memories stored as patterns which are random/abstract concepts in the brain? (instead of as images)
to avoid interference as images would be too similar
What are attractor states in Hopfield Networks?
How are the involved in recall?
What are the similar to?
-where the pattern is stored
-during recall the system is push towards the attractor state to get original stored pattern
-fixed points
What do Hopfield Dynamics do to energy?
What is the eqn. for this?
-minimises energy as you go toward attractor states (in a well, like a fixed point)
𝐸𝑖𝑗 = − 𝑤𝑖𝑗 𝑦𝑖 𝑦𝑗
When do spurious attractors occur in Hopfield Networks?
What are they?
when a network is trying to store too many memories, spurious attractors will appear
-attractors which are very hard to reach compared to desired attractors (have shallow energy wells)
What can Hopfield Networks be trained to do?
trained to store memories as fixed-point attractor states
If the neuron is in a learned pattern state what is the energy equal to?
“ not is a learned pattern state?
Why is it this way around?
E = -1
E = +1
-because the energy always is minimised when moving in the direction of the attractor state
What type of learning do attractors in Hopfield Networks employ?
Hebbian Learning
How does Elman introduce networks that introduce time?
by introducing recurrent connections from the hidden layer to the context layer
How are memories stored in a Hopfield Network?
As fixed-point attractor states
How are neurons connected in a Hopfield network?
Neurons are connected to all other neurons in the network but not themselves, and these connections are symmetric.
What is the output of a Hopfield Network defined as? ( include what it generates)
output of each unit (yi) = is a weighted sum of the inputs pushed through a step function to generate +1 or -1
In the Hopfield model, how are memories stored?
random patterns (M) of binary weights corresponding to a stored concept
-not pictures
In terms of attractor states, what is the difference between memorisation and recall in the Hopfield Model?
Memorisation is creating the attractor states, recall is reaching an attractor state
In the Hopfield Network Model, how are memories recalled?
What do x and y represent?
-Getting y to iterate until it matches the pattern x correctly
-y is the process of recall (activity of the network)
x is the stored pattern in the brain/memory (the correct pattern that y is trying to reach)
Hopfield model, in memory recall, when is a an attractor state (fixed point of model) reached?
when the pattern of y activity is the same as x (after all the iterations)
