ANN Lecture 8 - Recurrent Neural Networks

Question 1

What is the motivation for a recurrent neural network?

Answer 1

• A feed forward neural network only process and return single entities or objects
• The length of data sequences can vary
• Models need some kind of internal memory to process the whole sequence in context

Question 2

Examples of sequential data

Answer 2

• audio data
• text data
• temporal data e.g. stock market
• video data
• …

Question 3

Recurrent neural network

Answer 3

A ANN that allows feed-back connections

Question 4

Feed-back connection

Answer 4

Connection connecting a layer to itself or even to earlier layers.

Question 5

Vanila RNN

Answer 5

• RNN processes sequential data
• input, hidden state and output can be high dimensional.
• weights and biases are used for each input of the sequence
• for each input the model produces an output
• in each iteration the hidden state is fed into itself (memory)
• there are biases at the hidden and the output layer

Question 6

Vanila RNN Algorithm

Answer 6

• Initialize weights
• Initialize bias with zeros
• Initialize hidden state H0 with zeros

newHiddenState = 
activationFunction(
Input * WeightsXH +
old HiddenState * WeightsHH +
BiasH)

output = outputFunction(
newHiddenState * WeightsHY + BiasY)

Question 7

Backpropagation through time

Answer 7

The overall loss is the mean over all the losses at each timestep.

Question 8

Truncated backpropagation through time - Motivation and Explanation

Answer 8

Motivation:

• If the the Input sequence is very long, unfolding the RNN results in a very deep neural network
• If backpropagating through a very deep network gradients in early layers tend to either vanish or explode

TBPTT:

• Cutting the original sequence of length N into (N-n) n-long sub sequences, with own target sub sequences.
• First sub sequence gets a hidden state initialized with zeros fed in
• Next sub sequence gets the first hidden state of previous sub sequence fed in