lecture 7

Question 1

GPT-3

Answer 1

deep language model
 Question answering, text summarization, classification, etc.

 Controversial when announced
o Might have been too dangerous for the public bc it could be used as manipulation campaigns

 Core idea:
Instructing the model with prompts
 i.e., conditioning its probability function
= what to predict as a next word

Question 2

transformer models

Answer 2

o inside the transformer model there are stacked encoders and decoders

o Have an attention mechanism (and a neural network)

o seq2seq

Question 3

seq2seq

Answer 3

you take a sequence as an input and you get a sequence as an output

Question 4

attention mechanism

Answer 4

 Tells the model to pay attention to more than just the immediate context of a word. (programma verliest de draad niet)

• since language isn’t structured like this. Relationships between concepts can be quite wide apart. The attention mechanism helps with this

Question 5

Autoregressive text generation

Answer 5

predicts the next word

Question 6

Before deep language models

Answer 6

o Human language explained with interpretable models that combine symbolic elements

o Rule based operations

Question 7

GPT-3 approach (deep learning)

Answer 7

o Learning from data in the wild
 This is the game changer in language models

o No instruction/supervision, no syntax, no rules.

Question 8

temperature

Answer 8

How much risk you take

o 0 = deterministic (always gives the same)
 Chooses the most likely answer

o 1 = always gives a different answer

Question 9

moral chain of thought

Answer 9

1. Check rule violation
2. Reflect on purpose of the rule
3. Consider utility loss and gain

 You need to give GPT-3 instructions to follow certain steps, so that it ‘thinks’ about the problem step-by-step like humans

 If you use default models without using this chain of thought, they can’t make the same decisions that models make

Question 10

Linda problem

Answer 10

Conjunction fallacy

o GPT-3 is making the same (wrong) mistakes that humans make on the Linda problem

 Not a stochastic parrot and could pass as a subject

Question 11

GPT-3 personality

Answer 11

Resembles female participants
 i.e., High in honesty-humility

Low on emotionality

Question 12

GPT-3 values

Answer 12

shows theoretically consistent patterns
 High on universalism, benevolence, self direction, and stimulation
 Low on security, conformity, achievement, and power

 i.e., pattern of consistency is the same as in humans
* however, with increased temperature pushes the model to the extremes, goes from female to male, and becomes younger.

Question 13

cognitive reflection test

Answer 13

elicits intuitive answers that are incorrect
o for this reason, GPT-3 makes the same mistakes that humans make

 this is because it predicts what the most likely answer is. If people are more likely to make mistakes on these questions, so will GPT-3.

Question 14

Illusory truth effect

Answer 14

repetition of information increases subjective truth judgements.
o Regardless of veracity, plausibility, or knowledge of the information

 This also happens with GPT-3 when repeatedly exposed to information.
o Increased exposure increases its truth-rating
o This way, you can intervene with the system’s dynamics

Question 15

emergent abilities

Answer 15

An ability is emergent if it is not present in smaller language models, but is present in larger language models
o When quantitative changes in the system results in qualitative changes

 Developing abilities that you didn’t have before bc of phase transitions
 Sudden understanding of a problem when training examples increase

 Quant: model becomes bigger, human becomes older
 Qual: abilities

Question 16

core principles GPT-3 and humans

Answer 16

1. Continuous context-dependent next-word prediction before word onset
2. Matching pre-onset predictions to the incoming word to induce post-onset surprise
3. Representing words using contextual embeddings
   a. 3 words that have different meanings in different contexts are interpreted differently

Question 17

Gpt-3 struggles on tasks that

Answer 17

o Assess social commonsense and emotional intelligence in social interactions

o Measure its ability to understand other people’s mental states and realities in short stories

because GPT-3 has no ToM