Large Language Models

Question 1

How many params does chinchilla have

Answer 1

70b

Question 2

Training Compute-Optimal Large Language Models” by Hoffmann et al Mar 2022: key conclusion

Answer 2

scaling the number of training tokens (that is, the amount of text data the model is fed) is as important as scaling model size.
Given a fixed compute budget, researchers should allocate it in similar proportions to increase model size and number of training tokens to reach the compute-optimal model (measured by minimal training loss). “For every doubling of model size the number of training tokens should also be doubled.

Question 3

Chinchilla scores on MMLU

Answer 3

67.6%

Question 4

PaLM was released when in relation to chinchilla?

Answer 4

A week later (I’m guessing like early April late march 2022)

Question 5

PaLM # parameters

Answer 5

540B

Question 6

Chinchilla is made by who?

Answer 6

Deepmind

Question 7

PaLM is made by who?

Answer 7

Google

Question 8

2022 estimated PaLM training cost

Answer 8

$23.1m

Question 9

What is diffusion LM?

Answer 9

A non auto regressive language model based on continuous diffusions

Question 10

MMLU stands for

Answer 10

Multi task language understanding