Transformers: The attention Mechanism From Scratch

Question 1

Step 1

Answer 1

encoder representations of four different words

Defining or obtaining the word embeddings

word_1 = array([1, 0, 0])
word_2 = array([0, 1, 0])
word_3 = array([1, 1, 0])
word_4 = array([0, 0, 1])

Question 2

Step 2

Answer 2

…
# generating the weight matrices
random.seed(42) # to allow us to reproduce the same attention values
W_Q = random.randint(3, size=(3, 3))
W_K = random.randint(3, size=(3, 3))
W_V = random.randint(3, size=(3, 3))