7.20 & 7.21 - Recurrent Neural Networks Flashcards
When is a neural network recurrent?
When neurons in the network can form loops (1 sends to 2 and 2 sends to 1).
A single artificial model neuron can also be recurrently connected to itself
What are Hopfield Networks?
Recurrent neural networks where the weights are symmetric and the activity of units is either on or off.
Can a Random Recurrent Neural Network have spontaneous dynamics?
Yes.
How do we train read-outs from attractors in Echo State Networks?
Via linear regression of the output weights
What is a dynamical repository?
It is a network that can have many different attractors that can be accessed via multiple parameterizations.
What kinds of tasks are typical for an RNN?
In general RNN’s are competent in tasks where the output has history dependence.
Sequence Learning (e.g. word sequences), Sequence Generation, Sequence Classification (e.g., Language Parsing), Sensorimotor Transformations (for robot control).
What are the content-addressable memories (CAM)?
Memories that can be retrieved based on incomplete/partial stimuli.
Thus, in the case of CAM memories, your mother’s phone number is not retrieved by providing the address of the memory location in the storage, but rather, memory uses the input information (name of your mother) to retrieve the number. This is an example of an auto-associative CAM.
How is the term ‘energy’ used in Hopfield networks?
Hopfield networks have a scalar value associated with each neuron of the network that resembles the notion of energy. The sum of these individual scalars gives the “energy” of the network.
If we update the network weights to learn a pattern, this value will either remain the same or decrease, hence justifying the name “energy.”
What is the function of a Hopfield network?
A Hopfield network retrieves patterns stored as fixed-point attractors. It is a model of associative memory.
It is a ‘content addressable memory’.
To take an example of hand-written digit recognition, a Hopfield network would reproduce a digit stored onto the network (instead of classifying it).
What are the assumptions underlying Hopfield networks?
- The activation is either +1 or -1
- All of the connections are reciprocal
What are the possible states of a neuron in a Hopfiled network?
It is a binary neuron (Mc Culloch-Pitts)
(0 or 1) or sometimes (-1 1)
How are Hopfield networks trained?
Via Hebbian Learning: neurons that are co-active during a pattern are more strongly connected.
Why do recurrent neural networks produce spontaneous dynamics?
Because of recurrent connections previous outputs become inputs again, and so on in a sequence.
What is an Echo-State Network?
A Recurrent Neural Network with Random weights, where the output is trained to produce desired time-series.
What are time-series?
In general, a sequence of numbers representing a certain variable acquired through time.
In the case of neural network activity, a sequence of activations values.
