N-gram Modeling

Question 1

Probability of a Sentence

Answer 1

Multiplying together the probability of each word occurring after the previous word AND its context.

Question 2

Count-based Models

Answer 2

To get probability of word, using corpus, count up frequency of word occurrence (can include given previous word) and divide by total number of words.

Question 3

Count-based Model Formula

Answer 3

P(wi∣wi−n+1 ,…,wi−1)=
C(wi−n+1,…,wi−1,wi)/ C(wi−n+1,…,wi−1)

Question 4

Probability of a Sentence Formula

Answer 4

P(S)=P(w1)⋅P(w2∣w1)⋅P(w3∣w2)⋅…⋅P(wn∣wn−1)

Question 5

Perplexity

Answer 5

Inverse probability of the sentence normalized by the number of words in the sentence.
HIGHER means less confident, found more possible words
LOWER means more confident, found less possible words

Question 6

Ways to Evaluate Models

Answer 6

1. Log-likelihood
2. Per-word Log Likelihood
3. Per-word (cross) Entropy *
4. Perplexity ***

Question 7

Ways to Sample Text with LMs

Answer 7

1. Greedy decoding
2. Beam search
3. Nucleus sampling
4. Top-k sampling

Question 8

Greedy Decoding

Answer 8

Choose next word with highest probability of occurring after previous word.
Problem: Most probable next word doesn’t lead to most probable sentence.

Question 9

Beam Search

Answer 9

Maintain several paths for most probably sentences. 1. Choose two most probable words
2. Calculate total probability of sentence
3. Choose two most probable words for each sentence
4. Eliminate 2 least likely sentences
5. Repeat
Problem: too many similar sequences, too close to optimal

Question 10

Nucleus Sampling

Answer 10

Take top p% of distribution, sample within that

Question 11

Top-k Sampling

Answer 11

Take top k most likely words, sample from those