LLM Modeling

Question 1

What model did we use in the class

Answer 1

Flan-T5

Question 2

RNN

Answer 2

recurrent neural networks (PREVIOUS GEN) à only to word before it

Question 3

LLM

Answer 3

large language models = all words to each other; and weight of attention/influence between the words

Question 4

Tokenize

Answer 4

Convert each word into numbers (which is store in a vector)

Question 5

Self Attention

Answer 5

analyzes the relationships between the tokens

Question 6

Encoder

Answer 6

inputs prompts with contextual understand and outputs vector

Question 7

Decoder

Answer 7

accepts inputs token and generators token

Question 8

Sequence to Sequence

Answer 8

encoder to decoder model; Translation, text summarization, answering questions, is a sequence to sequence task (T5, BART)

Question 9

Decoder only mode

Answer 9

good at generating text (GPT)

Question 10

Zero Shot Inference

Answer 10

pass no grading/sentiment (type of In-context learning (ICL))

Question 11

One shot Inference

Answer 11

pass one grading/sentiment (type of In-context learning (ICL))

Question 12

Few shot inference

Answer 12

pass few grading/sentiment (type of In-context learning (ICL))

Question 13

Greedy

Answer 13

always take the most probably word, which will have the outputs be the same over and over again

Question 14

Big data

Answer 14

When LMM is too big for a single GPU

Question 15

DDP

Answer 15

Distributed Data Parallel

Question 16

Fully Shredded Data Parallel (FSDP)

Answer 16

BIGGER SCALE reduce memory by distributing/sharding model parameter across GPUs

Question 17

Three main variables for scale

Answer 17

1) Constraints (GPU, time, cost) 2)Data set Size (number of token ) 3) Model Size (number of parameters)

Question 18

Chinchilla

Answer 18

very large models may be over parameterized and under trained à thus less parameters but feed it more data verse bigger and bigger

Question 19

Fine Tuning an existing model

Answer 19

the output is a new model

Question 20

Specific Use Case Training

Answer 20

500 examples (via prompt completion pairs) –> resets all parameters –> could lead to catastrophic forgetting (but that may not matter for a single type of use implementation) –> Very CPU intensive

Question 21

Multiple Use Case Training

Answer 21

1000’s of examples across multiple tasks –> resets all parameters –> less likely to have catastrophic forgetting since its trained across multiple tasks

Question 22

PEFT (Parameter Efficient Fine Tuning)

Answer 22

small number of trainable parameters; rest are frozen –> MUCH MORE EFFICIENT

Question 23

Reparametrize model weight (LoRA)

Answer 23

i. Freezes most of original weights
Inject matrixes to update model; updates the non frozen

Question 24

Additive

Answer 24

add training layer or parameter to model –> KEEP THE ENTIRE EXISTING MODEL

Question 25

PROMT ENGINEERING

Answer 25

one shot inference etc

Question 26

PROMPT TUNING

Answer 26

Prompt Tuning: It fine-tune the LLM but with the structured data, which is consisted of some contents like “instruction”, “response” etc. - Fine Tuning: It fine-tune the LLM with unstructured data like raw text.

Question 27

FLAN

Answer 27

Fine- Tuned Language Net are the specific instruction used to perform fine tuning

Flan - T5; FLAM-PALM is the tuning of the t% and PALM model

Question 28

Rogue

Answer 28

used for text summarization, compares to 1 or more reference summaries

Question 29

Recall

Answer 29

I very long response can have a recall of 100% but be too wordy

Question 30

Precision

Answer 30

how many extra words are there in the output?

Question 31

F1

Answer 31

A ratio of recall and precision

Question 32

BLEU

Answer 32

used for text translation; compares to human generated translation

Question 33

RLHF

Answer 33

Reinforcement Learning from Human Feedback - Tuning a model to be helpful, honest, harmless (Three HHH)

Have humans tag how ‘good’ a response by comparing 3 options (how helpful? Or how harmful? Or how honest?)

Question 34

PPO

Answer 34

Proximal Policy Optimization - a popular algorithm who helps solve reinforcement learning problems. Makes updates within a very small region (proximal) to LLM via many iterations to bad handle HHH

Question 35

Reward model

Answer 35

supervised learning taking your human prompts and ‘rewarding’ the human tagged responses [comparing classes hate vs not hate probability)

Question 36

Reward Hacking

Answer 36

• where a model tries to optimize it’s scores by making answers that are long and wordy

Avoid reward hacking by comparing to Reference Model via KL Divergence Shift Penalty

Question 37

Constitutional AI

Answer 37

allows you to scale Reinforcement Learning without human intervention. Constitutional AI (CAI) is similar to RLHF except instead of human feedback, it learns through AI feedback.

Question 38

LLM Optimization Techniques

Answer 38

Distillation - train a smaller student model from a larger teacher model
Post Training Quantization (PTQ) - reduce precision of model weights (aka from 32 bits to 8 bits)
Pruning - remove model weights with values close to or equal to 0 (in theory makes sense to reduce , but in actuality there may not be many weights are Zero or close to 0

Question 39

3 types of issues of models

Answer 39

1) Out of date
2) Bad at math (can’t do calculation)
Hallucinations and guessing the answers it doesn’t know

Question 40

How to mitigate issues with models

Answer 40

RAG-retrieval augmented generating; get the details directly from a DB/API then pass I to the model

Chain of Thought –> Provide hints of how to break the problem into smaller parts [good for simple problems]

Program Aided Language Models (PAL)- integrate with python to do the math and model to call python

Question 41

Responsible AI - how do mitigate:

Answer 41

Toxicity [curating training data, train guardrail models, diverse group of human annotators], Hallucination [educate/add disclaimers]
Intellectual Property [not easy, machine ‘unlearning’, filtering blocking]`

Question 42

Existing metrics to measure hallucination

Answer 42

1) Rogue –> compare to an expected results
2) Ask Chat GPT to grade
3) Probability checks

Question 43

A new way to score LLM Hallucinations (Gailio)

Answer 43

ChainPull

Pass results to chain of thought’s model which includes a score and a logic path

Question 45

What does GPT stand for?

Answer 45

Generative Pre-trained Transformers, commonly known as GPT, are a family of neural network models that uses the transformer architecture and is a key advancement in artificial intelligence (AI) powering generative AI applications such as ChatGPT.