Handout #8 - Recurrent Neural Networks

Question 1

Explain what can be used to train a RNN

Answer 1

Back-Propogation Through Time (BPTT)

1. Unroll the network to expand it into a standard feedforward network and then apply back-propogation as per usual.

Question 1

What’s the problem with BPTT

Answer 1

The unrolled network can grow very large and might be hard to fit into the GPU memory.

Process is seq. -> can’t be parallelised.

Question 2

What’s the problem with the Simple RNN layer

Answer 2

RNN can grow very deep -> gradient descent can vanish (or explode) very quickly.

Question 3

What time of data is RNN’s used for?

Answer 3

It’s used on sequential data -> any data with tie series (e.g. audio signal, stock market, machine translation)

Question 4

Is an RNN a feedforward network?

Answer 4

Not, it’s cyclic

Question 5

Explain why LSTM is useful

Answer 5

Deals with the exploding and vanishing gradient problem (when unrolling the network).

LSTM has three gates; forget gate, input gate and output gate.

1. Forget gate; forget irrelevant information
2. Input gate; add/update new information
3. Output gate; pass updated information

Question 6

Explain GRU

Answer 6

A simpler alternative to LSTM -> faster to train.

Instead of linear combination (w1u1 + w2u2), the gating mechanism is based on a multiplication of both inputs.

Question 7

What is the critical issue with RNN

Answer 7

They aren’t suitable for transfer learning.

Can’t do stuff in parallel.