Handout #10 - Attention Mechanism and Transformer Model

Question 1

What’s the problem with RNN?

Answer 1

1. Sequential in nature -> can’t parallelise
2. context computed from past only
3. no explicit distinction between short and long range dependencies
4. training is tricky

Question 1

Why can’t CNN and DNN be used in text processing?

Answer 1

The dependencies between different words aren’t between the current and previous context sample

-> the verb is not always the next word after the subject.

Question 2

What’s cross-attention?

Answer 2

CA allows you to work with multiple modailities (e.g. audio, video, images, text)

-> works because it doesn’t depend to the position of the keys/values and can deal with any synchronisation issues.

Question 3

Give the layman’s definition of the Transformer

Answer 3

It’s an encoder/decoder network that is solely based on sequences of Attention Layer blocks