Lecture 3

Question 1

What are the building blocks of AI?

Answer 1

addition and multiplication

Question 2

There are two sides to the “is AI intelligent” debate, what are these two sides?

Answer 2

It’s just math (so no) vs. look at the performance (so yes)

aka how ai works vs what it can do

Question 3

What is the Xor problem?

Answer 3

Very simple models cannot learn the logical function of either this or that

Question 4

What is the solution to the Xor problem?

Answer 4

more complicated neural networks

Question 5

What is a neural network?

Answer 5

A neural network is really just stacked logistic regression (and intermediate steps)

Look at figure

Question 6

Why do neural networks work better than logistic regression?

Answer 6

because they can do multiple things at once, intermediate steps

(i.e. use genre as a determinant as the example in the lecture aka consider other factors that have an effect)

Question 7

What is the trade-off of using neural networks vs. logistic regression?

Answer 7

Neural networks need large amounts of data to work

Question 8

The intermediate steps that are a part of neural networks, do they have to be put in?

Answer 8

No the model predicts these intermediate predictors themself, based on prior data

Question 9

What is the universal approximation theorem?

Answer 9

Any function that looks like predictor <> outcome can be captured with neural networks

Question 10

Does an AI like chatGPT work on this neural network model?

Answer 10

No, but it is one of the building blocks

Question 11

How do language models work (simply)?

Answer 11

They predict the next word (dependent on the data they trained with)

Question 12

Why do neural networks not work for language models?

Answer 12

They disregard the order of the words

Question 13

What is the solution for neural networks not working for language models?

Answer 13

Transformer models

Question 14

What are transformer models used for?

Answer 14

Language based models, but nowadays most AI have this as their basis (non-language included)

Question 15

there are two sides to the transformer models, why?

Answer 15

The left side is to transform the input words into numbers and the right side is to put the numbers back into words

Question 16

What are skip-connections?

Answer 16

Basically the number strings (words) are copied and pasted throughout the model, as to not forget good info that was early in the model

Basically you combine the early simple thing with the later complicated thing to get the best result

Question 17

The skip-connections are the add-norm part of the model, what do the add and norm individually do tho?

Answer 17

The add part is really the skip-connections, so the simple + complicated

The norm part is to make sure it is not blown out of proportion aka you only take a small part of the early and not the whole

Question 18

What is the positionwise FFN?

Answer 18

The “basic” neural networks introduced in the lecture (stacked logistic regression)

Question 19

What are embeddings?

Answer 19

Embeddings are numerical representations of word meanings based on it’s associated w/ other words

Question 20

How do embeddings work?

Answer 20

Basically basic NN with singular word input with multiple outputs of the words’ associaton with other words in the language (based on previous learning)

Question 21

[0.4, 0.4, 0] is a simplistic version of an embedding string, what do these numbers mean?

Answer 21

Probabilities of the words’ association with other words

Question 22

What is positional encoding?

Answer 22

Words get new set of strings that = position in the sentence

Question 23

How does positional encoding work?

Answer 23

Wavelines on graph, look at which position the word should be and look at that same word on the graph, then assign associated string w/ it

Question 24

Between which numbers is positional encoding always?

Answer 24

-1 and 1

Question 25

How do positional encoding and embedding combine?

Answer 25

adding together - [1, 0.3] and [-0.2, 0.8] become [0.8, 1.1]

Question 26

What is multi head attention?

Answer 26

Enriching the numbers with the context of other words in the sentence (i.e. prize or price depend on context)

Question 27

How does multi head attention work? short answer

Answer 27

Check the word for it’s association with others in this specific sentence and put these into an equation

Question 28

How does multi head attention work? long answer

Answer 28

Previous string is taken and 3 new strings are made from it (query, key and value). You take the query from a word and the key from the word you want to check it’s association with (this goes into equation as). The value string you have left is what you put into the equation as the word meaning

Example: wins = correlation * “wins value” + cor * “prize v” + cor * “home v”

Question 29

The transformer model is usually called a base model, when in reality a chat model is usually used, what is the difference?

Answer 29

The chat model has an “end token” that makes sure the input is seen as a whole statement so the chat generates new text

Question 30

What is an evaluation type that models often depend on?

Answer 30

Human evaluation