NLP 2

Question 1

What is RNN?

Answer 1

neural network with a loop is called a Recurrent Neural Network (RNN)

Question 2

3 layers of RNN

Answer 2

Input (embedding for vocab size to hidden layer), hidden (fully connected), output (to predict the target token)

Question 3

Why BPTT is needed?

Answer 3

RNNs process sequential data by maintaining a hidden state that is updated at each time step. However, training an RNN is challenging because the network’s loss depends not just on the current input but also on all the previous inputs due to the recurrence relationship. This sequential dependency means that the network’s weights must be updated based on errors accumulated over multiple time steps, not just a single layer.

Question 4

What is BPTT?

Answer 4

Backpropagation Through Time

Question 5

Concept of multilayer RNN and why it’s used

Answer 5

outputs of a first RNN are used as the input for a second RNN; While the model is very deep in principle, each predicted token only depends on one linear layer

Question 6

Why Long Short-Term Memory is needed?

Answer 6

Helps to separately learn (1) information required to predict the next token and (2) contextual information learned throughout the already seen token. (saves gender for the next words)

Question 7

How LSTM works?

Answer 7

Second hidden state: “short term memory” / “cell state”

Question 8

Four main networks in LSTM

Answer 8

Forget gate, input gate (gender), cell gate (female), output gate

Question 9

What is dropout?

Answer 9

During each training iteration,
randomly deactivate neurons with a probability p
– To compensate, all activations are multiplied
by 1/(1 – p) during training

Question 10

Drawbacks of dropout?

Answer 10

Less input used to calculate the output

Question 11

Other regularisation techniques

Answer 11

Weight decay, Activation regularisation (AR), Temporal activation regularisation (TAR)

Question 12

Weight tying?

Answer 12

Mapping from input to hidden and hidden to output have the same weights. Used in AWD-LSTM

Question 13

In an RNN, the embedding of a token does not depend on its position in the sequence.

Answer 13

True. The token embedding is determined by a lookup table or an embedding matrix, which assigns a fixed vector representation to each token based on the vocabulary.

Question 14

Which factors influence the number of parameters in RNN for next-token-prediction?

Answer 14

Size of the embedding, Number of tokens in the vocab

Question 15

Techniques used in AWD-LSTM?

Answer 15

Activation regularization (AR), dropout

Question 16

What is AWD-LSTM?

Answer 16

Average Weight-Dropped Long Short-Term Memory. It introduces specific modifications and techniques to standard LSTMs to make them more effective for training on sequential data.

Question 17

Explain on an intuitive level how an LSTM works.

Answer 17

The LSTM uses a long-short term memory to store meta or contextual information in its cell state that is used together with the short term or hidden state to make predictions.

Question 18

How many times does the cell state flow trough a neural network?

Answer 18

The cell state never flows trough any neural network.

Question 19

Is it certain, that both the hidden and cell state contribute to the prediction?

Answer 19

Technically it is not certain, as the hidden state flows trough a sigmoid function before being merged for the prediction which has the ability to make it a tensor of all 0. However it is super unlikely for this to happen.

Question 20

Is it possible to have a multilayer LSTM architecture?

Answer 20

Yes it is possible to chain LSTM models on top of each other just as is for RNN’s.

Question 21

Explain - in non technical terms - the difference between the sigmoid and tanh function?

Answer 21

The sigmoid function chooses IF a value is used, the tanh function scales it.

Question 22

How does the LSTM achieve regularization compared to an RNN?

Answer 22

After any network in the LSTM either a sigmoid or tanh function is used to normalize the results.