Week 3 - Transformer Models

Question 1

What is an encoder decoder networks

Answer 1

Also known as sequence-to-sequence networks
Generate arbitrary length output sequences which are contextually appropriate

Question 2

What are the 3 componenets of encoder decoder networks

Answer 2

Encoder
Context vector (a function of h1:n from encoder)
Decoder

Question 3

What does an encoder do in encoder decoder networks

Answer 3

Takes an input sequence x1:n and generates a corresponding sequence of contextualised representations h1:n
(captures the information and dependencies between tokens)

Question 4

What does a decoder do in encoder decoder networks

Answer 4

Takes c (context vector) as input and generates a sequence of hidden states h1:m from which an output sequence y1:m can be generated

Question 5

How can encoders and decoders be realised

Answer 5

By sequence based architectures
eg RNN

Question 6

what is the purpose of <s></s>

Answer 6

Marks the end of the source text

Question 7

In the decoder for translation, what is each word generation conditioned on

Answer 7

The previous hidden state and the embedding for the last word generated

Question 8

How does the encoder work for machine translation (during inference time)

Answer 8

takes the source text
Encoder network generates the hidden states
Uses the input embedding token as well as the output from the previous hidden state (ht-1)
The output of the source is essentially ignored

Question 9

How does the decoder work for machine translation (during inference time)

Answer 9

The decoder takes the last hidden state of the encoder and the first token (which will be <s>) to predict the first translated word
(repeats)</s>

Question 10

What do h with superscripts e and d represent in transformer diagrams

Answer 10

hidden states in the encoder/decoder

Question 11

How is the context vector formed and used in machine translation

Answer 11

Encoder generates h^e n = c
Decoder uses c as its initial hidden state c = h^d 0

Question 12

Which states in the decoder is c available to

Answer 12

It is available to all states
otherwise “context weakens”
c is added as a parameter in the computation of each h^d

Question 13

What is context

Answer 13

All information from the source text

Question 14

How does rnn work for machine translation during training

Answer 14

in the same way the source text is processed in the encoder
In the decoder, the separator token and last hidden state are taken
sent through softmax to create a probability distribution
Then the loss is calculated using CE loss between each generated token and the correct token (gold answers)
average the loss across all states -> adjust weight accordingly

Question 15

what is the difference between the inference and training stages in rnn

Answer 15

training uses teacher forcing
i.e. if an incorrect token is output, it will not be used as input for the succeeding states
Instead correct token will be used

Question 16

What is the bottleneck

Answer 16

final state h^e n represents all information on the source text
may not sufficiently represent all necessary information
decoder relies heavily on c

Question 17

What is the purpose of the attention mechanism

Answer 17

Addresses the bottleneck
allows the decoder to obtain information from all hidden states in the encoder

Question 18

how is an attention ‘weighted’

Answer 18

Weights determine which parts of source text are relevant for current decoder token
Allows focusing on specific portions of input sequence during decoding.

Question 19

What is different about c in attention mechansim

Answer 19

c is dynamically derived
Varies based on decoder token being produced

Question 20

What is each hidden state h^d i in the attention conditioned on

Answer 20

Prior hidden state
Previous token
Dynamically generated context vector ci

Question 21

What are transformers

Answer 21

deep learning model that does sequence processing without the need for recurrent connections

Question 22

What is the shortcoming of sequence based architectures (i.e. RNNs)

Answer 22

Cannot perform calculations in parallel

Question 23

What is a causal(left to right) transformer

Answer 23

For each input, the network has access to all inputs up to and inc current one
No access to input information beyond current
computation of each item is done independently and hence parallelised
makes use of self-attention: information in contexts can be accessed without recurrent connections

Question 24

What is the causal model as a LM

Answer 24

main difference is there are no reccurent connections, replaced with a transformer block
For each input word, calculate self attention between that word and every other word preceding it
means we can parallelise

Question 25

What are bidirectional transformer encoders

Answer 25

Allow the self-attention mechanism to range over the entire input
not autoregressive
parallelised
map sequences of embeddings to sequences of output embeddings which are contextualised using information from the entire input sequence
contextualization done via self attention

Question 26

What are 3 types of LMs

Answer 26

• transformer based
• RNNs
• n-grams

Question 27

Encoder-decoder networks encompass which models

Answer 27

transformer-based (standard transformers, BERT, GPT)
RNNs (standard RNNs, LSTMs, GRUs),
CNNs

(attention mechanism can be added to all of these but is not an encoder-decoder network itself)

Question 28

what is inference

Answer 28

testing

Question 29

does the causal model for LM use teacher forcing

Answer 29

yes
if the model predicts the wrong word, the truth label word is used as input