RNN

Question 1

For what kind of problems could we use many to one RNN?

Answer 1

Sentence classification, speech recognition (sound waves to one word), video classification

Question 2

For what kind of problems could we use one to many RNN?

Answer 2

Image captioning, music composition, natural text generation.

Question 3

For what kind of problems could we use many to many RNN?

Answer 3

Video classification on frame level, speech enhancment, continues emotion prediction

Question 4

What makes RNN different from FW networks?

Answer 4

It has feedback.

Question 5

What is the formula for updating hidden state and calculating output at a timestep in a simple RNN cell?

Answer 5

h_t = tanh( W_{hh} * h_{t-1} + W_{xh} * x_t 
y_t = W_{hy} h_t

Question 6

What is direct feedback?

Answer 6

The hidden state is used in the same cell at the next timestep

Question 7

What is indirect feedback?

Answer 7

The hidden state is connected to a previous cell at the next timestep

Question 8

What is lateral feedback?

Answer 8

A cell is connected to a cell in the same layer.

Question 9

How does lateral feedback often affect the output of a layer?

Answer 9

Cells strenghten themself, while weakening others, the strongest cell becomes active.

Question 10

What is a RNN with symetrical connections to all other cells called?

Answer 10

A Hopfield network

Question 11

What is the main challenge of using deep RNN’s

Answer 11

Gradient vansihing/ explosion. Batch normalization and dropout layers help.

Question 12

What is a biderectional RNN?

Answer 12

Cells see inputs both from the past and the future.

Question 13

What is a LSTM (Long short-term memory cell)

Answer 13

A long short term memory cell has a seperate path for cell state ensuring better gradient propagation?

Question 14

What is the forget gate in a LSTM cell?

Answer 14

Controls how much of the previous cell state to remember.

Question 15

What is the input gate in a LSTM cell?

Answer 15

Controlls how much to write to a cell

Question 16

What is the ouput gate in a LSTM cell?

Answer 16

Controlls how much to output from the cell

Question 17

How are the values for the different gates calculated in a LSTM cell?

Answer 17

i = sigmoid( W_i * [h_{t-1}, x_t])
o = sigmoid( W_o * [h_{t-1}, x_t])
f = sigmoid( W_f * [h_{t-1}, x_t])
g = tanh( W_g * [h_{t-1}, x_t])

c_t = c_{t-e}*f + i*g
h_t = o * tanh(c_t)

Question 18

How do the weight matricies in RNN differ for different timesteps?

Answer 18

They are the same for all timesteps.

Question 19

Why is the gradient flow improved in LSTM?

Answer 19

No matrix multiplication of cell state.

Question 20

What are peephole connections in LSTM?

Answer 20

c_{t-1} is connected to the forget, input and output gates

Question 21

What is a GRU ( Gated reccurent unit)?

Answer 21

Compared to LSTM, cell state is eliminated and uses only two gates, reset and update. The hidden state path still avoids matrix multipliation to allow efficient matrix multiplication.

Question 22

What is the main advantage of GRU compared to LSTM?

Answer 22

GRU gates have fewer parameters and often perform comparable to LSTM.

Question 23

What is pooling over time?

Answer 23

The output can be a average, max, sum… over time of the output of the individual cells. (For … to one problems)

Question 24

How can we solve problems with a high input dimension at each timestep and going over several timesteps?

Answer 24

CNN + RNN

Question 25

How can the vanishing/exploding gradient in a basic RNN be leviated?

Answer 25

Exploding: Clipping gradient
Vanishing: Changing to LSTM (Or Gru)

Question 26

What is BPTT(Back propagation trough time)?

Answer 26

In BPTT we run the forward phase for the entire sequence (over time) without updating the weights. We then calculate the loss of all outputs of the RNN and let the gradient propagate back to eariler states (trough time).

Question 27

What is sequence representation learning with RNNs?

Answer 27

Use a RNN “encoder” to get a single output form a sequence, and a RNN decoder to turn the ouput into a sequence. Can also use “attention” a combination of the ouputs from each cell in the encoder that is used as additional input to each cell in the decoder.

Question 28

What is the moativation behind CTC ( Connectionist Temporal Classification)?

Answer 28

If we want to translate audio into e.g. text with a normal RNN we need to