Andrej Karpathy Flashcards
1
Q
What system does GPT use to tokenize?
A
tiktoken
2
Q
How many tokens does GPT have in its vocabulary?
A
50,000 (approx)
3
Q
What is the length of text that is sampled called?
A
Block Size
4
Q
What’s the simplest neural network for text patterns and generation?
A
Bigram Language Model
5
Q
The method of back-propagation for soft-max outputs is…
A
Cross-entropy
6
Q
A
