Simple recurrent networks

Question 1

What are the gates of the Long-Short Term Memory (LSTM) networks?

Answer 1

input, output, forget

Question 2

How can Simple Recurrent Networks be trained?

Answer 2

trained with the standard back propagation algorithm, because the temporal structure can be “ unrolled ”
into space.

Weights are adjusted with the sum of
the changes at each level (timestep) :

Question 3

The difference between Elman and Jordan networks

Answer 3

Jordan nets use output representation as context / memory + Elman nets use internal representations as context / memory, so they are more powerful

Question 4

What does the feedback loop do?

Answer 4

provides the network with a memory (copy) of the last output state, which is stored in «context» neurons

Question 5

The self-feedback loop:

Answer 5

allows to keep trace of previous past states