RNN Flashcards
What are the use cases from a RNN? Name and give application examples
One to one:
- single input and single output
Application: Standart Neural Network (Vanilla)
One to many:
- Single input, many outputs
Application: image Captioning
Many to one:
- Many inputs, single output
Application: sentiment classification, action recognition/classification
Many to many:
- many inputs, many outputs
- Application: machine translation
Many to many:
-Many inputs, many outputs
-Application: video classification
Explain the functionality of a simple Vanilla RNN.
Simple Vanilla RNN processes sequential data, updating hidden state “ht” using input “Xt” and previous hidden state “ht-1.” Calculates output “Yt” from “ht” through an activation function. Repeated at each time step to capture temporal dependencies.
What the problem of a simple RNN that occurs when processing long inputs?
The vanishing/explode gradient problem. That occurs because of the repeated multiplication of gradients during the backpropagation, causing the gradients to be really small (vanish the gradient) , in case the weights are less than one, or really large (explode the gradient) , in case the weight of the RNN are larger than one.
About the problem with long inputs in a RNN. How is this problem fixed using LSTMs or GRU?
because incorporates gates mechanisms, that regulate the flow of information. This gates allows to retain and update information, based on how significant the information is. That enables to a better long term memory and prevent the issues of vanishing or exploding gradients.
Name two methods to convert text into a compact representation/an embedding.
Word Embeddings: Word2vec and Skip-gram techniques (train to oredict neighborhood words)
Sentence Embeddings: Skip Thoughts Technique (train to predict neighborhood “sentences”
What is word Embedding? What are the advantages over one-hot encodings ?
Word embedding is a technique to represent words as dense vectors in a continuous space. The advantage compared to one hot encode is that it can work with large vocabularies and capture semantic and contextual relationships between the words
Why are RNNs useful for image captioning?
RNNs are useful for image captioning because they can deal with variable amounts of inputs and outputs, capture the dependencies and similarities between words and context, and generate coherent and contextual captions for the image description
What gates/models does the LSTMs have? Draw the components and their connections I the figure bellow. How many parameters does a LSTM contain for a given number H of hidden units?
LSTM (Long Short-Term Memory) has three main gates: the input gate (i), the forget gate (f), and the output gate (o). Additionally, it has a cell state (Ct) that regulates information flow.
Number of Parameters = 4𝐻 × 2𝐻
*𝐻: Hidden States
What is an RNN and how isthe hidden activation computed?
An RNN (Recurrent Neural Network) is a type of neural network designed for sequential data. The hidden activation is computed by combining the current input with the previous hidden activation using a weight matrix and an activation function
What is a skip-gram-model and how does it learn a word embedding? Draw and explain the network structure during training
A skip-gram model is a type of word embedding. It learns word embeddings by predicting context words from a target word in a large corpus, capturing word relationships and meaning in a dense vector space.
What gates/modules does the GRU have? Please draw the components and their connection into the figure below. How many params does a GRU contain for a given number H of hiddenunits?
The GRU (Gated Recurrent Unit) has two main gates: the update gate (z) and the reset gate (r).
The number of parameters is 3𝐻 × 2𝐻
