LSTM

Question 1

Define LSTM

Answer 1

Long Short Term Memory

It is a more complex form of Recurrent unit, instead of just adding the effects of input and the past, LSTM have gating units that can turn these effects on or off based on the input.

These gates have their own parameters that are trained during backpropagation.

Question 2

What are the three different gates?

Answer 2

1. Forget Gate - Removes from cell state
2. Input Gate - Adds to cell state
3. Output Gate - Calculate hidden state

Forget gate and input gate together updates cell state.

Question 3

What is the hyper parameter in LSTM?

Answer 3

No. of node of Neural network present in each gate.

Dimension of Cell state, hidden state, and all the state of gates are same and is equal to no. of node of neural network.

However, dimension of input may be different.

Question 4

Why ft*ct-1 is called as forget step?

Answer 4

Since, we are using sigmoid [0,1] in output layer of ft, therefore the final output (ft*ct-1) is controlled and vary between [0,ct-1]

Question 5

How is ct (cell state) calculated?

Answer 5

ct = ftct-1 + itcŧ
Element wise addition

Question 6

How is ht (hidden state) calculated?

Answer 6

ht = ot * tanh(ct)

Question 7

What is GRU and why was it needed?

Answer 7

Gated Recurrent Unit is a simpler version of LSTM with fewer gates and less computation.
It was needed because of complex LSTM with higher number of parameters increasing its complexity.

GRU has two gates: -
* Reset gate
* Update gate

Question 8

What is the difference between LSTM and GRU?

Answer 8

LSTM: Three gates - forget, input, output
GRU: Two gates - reset, update

LSTM: Two states - cell & hidden
GRU: One state - Hidden

LSTM has more parameter than GRU.

LSTM are computationally expensive due to extra gate and cell state

GRU is preferred for simpler task.
LSTM is preferred for complex task.

Question 9

When to use deep RNNs?

Answer 9

Stacking multiple RNNs on top of one another.

1. For advanced tasks like speech recognition, machine translation
2. When we have large dataset
3. If we have enough computational power

Deep RNN maintains hierarchical structure; initial RNN ensures word level dependency, as we go deep, sentence, paragraph level dependency are captured.

Question 10

What is BiRNN/BiLSTM/BiGRU

Answer 10

Bi-Directional
Used when context of a word depends on future words.
Ex - I love amazon, website; I love amazon, river.

Question 11

Application BiRNN/BiLSTM/BiGRU

Answer 11

NER, POS tagging, Machine Translation, Sentiment Analysis, Time Series Forecasting

Question 12

Disadvantage of BiRNN/BiLSTM/BiGRU

Answer 12

1. Overfitting (due to increased complexity)
2. Latency issues in real-time speech recognition