RNN

Question 1

When are RNN used?

Answer 1

For sequential data like text, speech, video, time series data.

RNNs have the features of memory i.e., they can remember the past inputs. That is why they work great on the sequential data.

Question 2

Why RNN over ANN?

Answer 2

• Length of input may vary for sequential data, whereas in case of ANN, length of input is fixed.
• ANN will be computationally expensive as for sequential data like text, the number of words may be too large.
• For sequential data, the size of test input may be higher than the training input.
• ANN will ignore the semantic meaning of the text data.

Question 3

Zero Padding in RNN

Answer 3

Pad the sequences to the maximum length with zeros

Question 4

Type of RNNs

Answer 4

1. Many to one - Sentiment Analysis
2. One to many - Image to Captioning
3. Many to many - Synchronous - NER, POS
   Asynchronous - Machine translation, Text summarization, Chatbot, Q/A, Speech to text

Question 5

How does input in RNN looks like?

Answer 5

Input = (Timesteps, No. of features)

Question 6

Why RNN is called so (Recurrent)?

Answer 6

This is because we are using the same hidden network multiple time (re-occurring)

Question 7

Problem with RNNs

Answer 7

1. Long term dependencies (Vanishing gradient problem)
   As the time steps increases, the partial derivative term in weight updation consist of many terms (in range [-1,1]- due to tanh), and that is why the overall value becomes very less.
2. Unstable training (Exploding gradient problem) - Similar case as above, except Relu is used as activation.
   Can be prevented by gradient clipping, learning rate