Hoorcollege 11 pretrained language models

Question 1

ELMO architecture

Answer 1

• 2 layer, bidirectional
• word vector: weighted sum of hidden states
• aH0 + bH1 + cH2 = F(H0, H1, H2)

Question 2

GPT (generative pre-trained transformers) architecture

Answer 2

• Transformer architecture, undirectional (“decoder”)
• Each token gets a vector for token level tasks
• Whole sequence also get’s a vector

Question 3

BERT

Answer 3

• Bidirectional
• Has for each token a bidirectionally contextualized representation at each layer
• For this it uses either the trick of replacing 15% of tokens by [MASK] or Next Sentence
  Prediction

Question 4

NSP (Next sentence predicting)

Answer 4

• uses CLS tokens to separate sentences and sentence into vector as input to NSP
  classifier to predict if the second sentence would be a logical next sentence or not
• C of [cls] -> classify sequences
• sequences useful for: sentiment analysis, NLI, paraphrasing
• token vectors useful for: POS tagging, NER, WSD
• different layers are useful in different tasks

Question 5

Ways of using pretrained models

Answer 5

• Freeze: use the embeddings ars they are
• Fine-tuning: adjust the model’s parameters while training the classifier

Question 6

Transformer

Answer 6

MultiHeadAttention -> addition -> LayerNorm -> FFN -> addition not of data ->
LayerNorm

Question 7

Generation as sampling

Answer 7

• greedy
• Beam search
• Random sampling with y =
  SoftMax(u)
  Random sampling: use top-k sampling, top-p sampling, temperature sampling

Scaling laws, but not only size matters

Question 8

Winograd schema resolution

Answer 8

for “he” having a P to be either one or another person replace
“he” by name and calculate P