Recurrent Neural Networks Flashcards
What is a recurrent neural network (RNN)?
An RNN is a class of neural network where a ‘loop’ within the network allows data to persist, or be ‘saved within’ itself, allowing that data to be used to evaluate the next input in the sequence.
At any time t, the cell has an input x(t) and an output y(t). Part of y(t) - notated by h(t) - is fed back into the cell for use in evaluating x(t + 1).
What are recurrent neural networks (RNNs) used for?
They are extremely good at dealing with temporal data - i.e. sports results - as it can remember previous evaluations.
What is the process of padding sequences?
Padding sequences is used to ensure all sequences have the same length, which helps when dealing with NNs that expect fixed-size inputs.
e.g. [1,2], [3,4,5] -> [1,2,0], [3,4,5]
What is the process of word embedding?
Word embedding transforms the indices into dense vectors where each dimension encodes information about how the word relates to others in the given context.
i.e. “Hope to see you soon!” -> [1,2,3,4,5,6]
“Nice to see you again!” -> [7,2,3,4,8,6]
What is the main problem with N-grams in word prediction?
Only prior local context, by the Markov assumption, affects the next word.
How can we use an RNN to perform word prediction?
Tokenize the text into words or subwords. Then, create input-output sequences where the input is a sequence, and the output is the next word in that sequence.
Use a set of RNN layers to capture sequential dependencies. The more RNN layers used, the wider the local context.
What is the main issue with RNNs?
RNNs are susceptible to the vanishing gradient problem, as for long sequences, the gradients of weights can reach zero, making it impossible to learn.
What is Long Short-Term Memory (LSTM)?
LSTM networks are a class of RNN that uses special units called ‘memory cells’ that can maintain information in memory for long periods of time using ‘gates’.
What is the input gate of an LSTM network?
Controls the intake of new information.
What is the forget gate of an LSTM network?
Determines what part of the cell state to be updated.
What is the output gate of an LSTM network?
Determines what part of the cell state to output.
