XLM

Question 1

XLM - What is an autoregressive model?

Answer 1

the output variable depends linearly on its own previous values and on a stochastic term (an imperfectly predictable term)

Question 2

XLM - How is autoregressive model being translated to NLP?

Answer 2

CLM - causal language modelling - predicting the next work based on the previous word.

Question 3

XLM - What is MLM?

Answer 3

Masked language modelling - predicting masked words in sentences.

Question 4

XLM - What is TLM?

Answer 4

Translation language modelling - The input is 2 sentences, that are translation to each other, one in each language.
Words are masked and the model is trying to predict them using the context from either language.

Question 5

XLM - what is their new unsupervised method for learning cross-lingual representations?

Answer 5

Pretrain the encoder and decoder separately on each language using MLM or CLM, merge them and train using back-translation interations.

Question 6

XLM - What is a key for unsupervised MT

Answer 6

Pretraining

Question 7

XLM - what is the method of back-translation taken from Lample 2018?

Answer 7

Create 2 models source->target and a target->source translations.
Create the target sentence of a source sentence, then train the target->source model. Do vice versa.
Work iteratively until saturation.