C5W4(Transformer Network)

Question 1

Bottleneck of plain RNN(GRU, LSTM)

Answer 1

Sequential way of processing input data (one by one)

Question 2

What is Self-Attention?

Answer 2

Compute Attention-based representation for each word.
A(q, K, V) - W1query, W2key, W3*value

Question 3

What is MultiHead attention?

Answer 3

Apply Self-Attention several time(#h times. H - hyperparameter), each time with unique weights.