DEEP LEARNING FOR NLP

Question 1

What is Deep Learning DL?

Answer 1

subset of machine learning that involves neural networks with multiple layers
The solution system is a neural network
Builds end-to-end systems, which take raw objects as the input (no initial feature extraction)
eg raw image = input is pixel values

DL vs NN: DL emphasises networks with higher number of layers

Question 2

NLP tasks: Sequence distribution

Answer 2

Model probability distribution of a sequence
p( x1 | x2,…xn) or p(x1…xn)
text generation / completion
eg modelling a chatbot

Question 3

NLP tasks: Sequence Classification

Answer 3

to learn a representation vector of a sequence and use it to classify the sequence
f(x1 -> x2 -> ..-> xk) = class
(sequence of input only 1 class output)
eg sentiment analysis, spam filtering

Question 4

NLP tasks: Sequence labeling

Answer 4

Learn a representation vector for each state (element) in a sequence and use it to predict the class label for each state
f(x1 -> x2 -> ..-. xk) = class1 -> class2 ..-> class k
(sequence of input, many class outputs)
eg POS tagging, named entity recognition

Question 5

NLP tasks: seq2seq learning

Answer 5

To encode information in an input sequence (seq) and decode it to generate an output sequence (2seq)
f(x1 -> x2 ->..xk) = y1 -> y2 ->..yk
eg language translation, question answering

Question 6

What is sentiment analysis

Answer 6

“I liked the film a lot” = positive class
(Uses sequence classification)

Question 7

What is Vanilla RNN

Answer 7

The simplest RNN design
When we chose f, we use a single perception (standard neuron operation to compute the hidden representation vector)

Question 8

What is the issue with Vanilla RNN

Answer 8

Can result in vanishing gradient problems, which negatively affects the training

Question 9

What is the Vanishing Gradient

Answer 9

In deep learning, most training is gradient-descent based
The gradient information is what is used to update the neural network
The term (Wh)^k-i in the gradient equation is problematic
k is the state of interest
i is a previous state
Numbers in the gradient matrix can become very small for long-distance past states, consequently they do not contribute to learning the correct weights
Vanishing gradients causes dependency loss between current and long-distance past states
Meaning it is biased to information in recent past states
“The write of the books …”
books is biased to “are” (plural) but correct answer is “is”

Question 10

How do we fix Vanishing gradient

Answer 10

Challenging
Requires new cell designs -> LSTM cells or Gated Recurrent Units
modify the function used

Question 11

What is Information Bottleneck

Answer 11

All the information in the encoder is accumulated in the final hk and sent to start the decoder
We assume hk is good enough to hold all this information - dangerous

Question 12

What is the Attention RNN

Answer 12

Used to solve the information bottleneck problem
Concern with the loss of information between states in the encoder
Automatically searches for parts of a source sequence that are relevant to the target prediction
Selectively builds direct connections between each state in the decoder and states in the encoder
Autoregressive structure

Question 13

What can we say about the softmax function

Answer 13

monotonically increasing
maps all numbers to positive and then only between 0 and 1 proportionally
results are good to be used as weights in attention RNN

Question 14

What is the benefit of Attention RNN

Answer 14

Improve model performance
Solve bottleneck information problem
help vanishing gradient problem
provide interpretability

Question 15

What is the motivation of multi head attention

Answer 15

to make the model more complex

Question 16

What is the difference between RNN vs Attention

Answer 16

RNN connects each state with the previous state
Only takes current and previous into account

Attention automatically identifies the relevant past states
Does not care about order - only about similarity

Question 17

What is a Transformer

Answer 17

the state-of-the-art neural network architecture for NLP, used in almost all recent language models
Concerned with attention only “Attention Is All You Need”

Question 18

Transformers: What is positional encoding

Answer 18

First step of transformer encoding
motivation : Injects order information to the model
Adds the order of each state i to the input vector
For an even ordered state (eg 4) we use sine
for odd we use cosine

Question 19

Transformers: What is Encoder Multi-head Attention

Answer 19

After positional encoding, we store the output vectors g1, g2, g3 as rows of a matrix G
Pass this G as query, key and value (self-attention) into the encoder multi-head attention

Question 20

Transformers: Add and Layer Normalisation

Answer 20

Motivation: prevent information loss/change caused by previous attention layer (assume there is loss)
So we add the output of the i-2 and the i-1 layers into i (current) layer and normalise them
(Instead of just taking i-1 output)
This helps stabilise the training process

Question 21

Transformers: What is the whole encoder

Answer 21

1) positional encoding
2) multi head attention
3) add and layer normalisation
4) fully connected NN
5) add and layer normalisation

These 5 steps are one building block of the decide which can be repeated N times
The fully connected NN uses a hidden layer with ReLU
The output vector for each state in the input sequence has the same dimensions

Question 22

Transformers: What is Decoder Multi-head Attention

Answer 22

Same structure as the encoder attention except in decoding we do not know the subsequent states (we MASK them = assume they don’t exist)
We simply remove the future states from the query, key and value matrices (just use the previous ones)

Question 23

Transformers: What is Encoder-decoder Attention

Answer 23

Comes after the decoder attention layer
Offers us the opportunity to inject the information from the encoder into the decoder

The encoder output is used as the value and key input into the encoder-decoder attention
The query is from the decoder output

Question 24

Transformers: What is the whole decoder

Answer 24

1) Masked decoder attention
2) Add and layer normalisation
3) encoder-decoder attention
4) add and layer normalisation
5) fully connected feedforward NN
6) sent to the prediction layer

This is one building block of the decoder, can be repeated N times

Question 25

What building blocks do Transformers involve?

Answer 25

• Encoder
• Decoder
  with:
• multi-head attentions
• fully-connected feedforward neural networks
• add&Norm operation
• positional encoding