ANN Lecture 10 - LSTMs

Question 1

Motivation of LSTMs

Answer 1

• The vanilla RNN can only capture short term dependencies but isnot able to capture longer dependencies.
• If you unfold it, you will get a deep network, where it is hard to backpropagate the errors for those long term dependencies.
• LSTMs is able to capture long term dependencies

Question 2

LSTM - Cell State

Answer 2

• The cell state is the major enhancement of the model, representing the “long-term memory”
• The cell state receives only two linear operations, which makes it easy for information to flow through
• Two gates regulate which information is added or removed from the cell state

Question 3

LTSM - Gates

Answer 3

• Gates consist of a fully-connected feed-forward layer with logistic activation function + a pointwise multiplication operation
• Sigmoid activation is between 0 and 1:
  
  • Activation close to 0: block the information
  • Activation close to 1: let the information pass
• LSTM has 3 gates:
  
  • Forget gate (cell state)
  • Input gate (cell state)
  • Output gate

Question 4

LTSM - Forget Gate

Answer 4

• Forget gate takes as input the input and the previous
  hidden state
• It regulates, which information of the cell state is removed
• Sigmoid activation
• Parameters: Weights and Bias

Question 5

LTSM - New Candidate for Cell State

Answer 5

• Create a new candidate for the cell state update from
  input and previous hidden state
• TanH activation
• Parameters: Weights and Bias

Question 6

LTSM - Input Gate

Answer 6

• Input gate takes as input the input and the previous
  hidden state
• Input gate filters the candidate before the cell
  state is updated
• Sigmoid activation
• Parameters: Weights and Bias

Question 7

LTSM - Update of Cell State

Answer 7

• Multiply cell state with forget gate to remove information
• Add filtered new candidate to cell state to add new
  information

Question 8

LTSM - New Candidate for Hidden State

Answer 8

• Create a candidate for hidden state from the cell
  state
• TanH activation

Question 9

LTSM - Output Gate

Answer 9

• Output gate takes as input the input and the previous hidden state
• Output gate regulates which information of the new
  candidate for the hidden state can pass
• Sigmoid activation
• Parameters: Weights and Bias

Question 10

LTSM - Update of the Hidden State

Answer 10

• Update the hidden state by filtering the candidate for the hidden state
• Multiply output gate with the new candidate for hidden state