Smoothing

Question 1

What does smoothing achieve?

Answer 1

reduce the effects of over-fitting and avoid zero probabilities. It assigns more probability mass to less likely events, taking some from the most likely events

Question 2

What does add-one smoothing do?

Answer 2

For all possible n-grams: using maximum-likelihood estimation, add one to the count of occurrences and add the vocabulary size to the denominator. For example, the ML of P(w|v) is: P(w|v) = count(w,v)/count(v) Add one smoothing: P(w|v) = (count(w,v) + 1 )/count(v) + v

Question 3

1) Why do we add v to the denominator?
2) Show why adding v works for the first question.

Answer 3

To make sure the n-gram probability is a valid probability and thus sums to 1.

Question 4

The problems with add-one smoothing.

Answer 4

1. It takes too much weight from larger probability masses.
2. The ML estimates for each probability are quite accurate so we do not want to change these too much.

Question 5

What is add α-smoothing

Answer 5

add a constant α<1 to each count on the numerator and multiply the “v” by α as well.

P(w|v) = C(v,w) + α/C(v) + αv

Question 6

What methodology can be used to optimise α for α-smoothing?

Answer 6

validation testing.

Split the dataset into a training set, validation set and test set. Train models with various values for the parameter α and then report their result on the validation set. The selected value for α of the model with the best score on the validation set is used on the test set to report the overall performance of the model.

Question 7

Advantage and disadvantage of using n-grams with large n

Answer 7

high-order N-grams are sensitive to more context, but have sparse counts

Question 8

Advantage and disadvantage of using n-grams with low n

Answer 8

low-order N-grams consider only very limited context, but have robust counts for maximum-likelihood estimation.

Question 9

What is interpolation?

Answer 9

Composing a higher order n-gram of lower order n-grams in a mixture model. So for example:

P_I(w₃|w₁,w₂) = λ₁P₁(w₃) + λ₂P₂(w₃|w₂) + λ₃P₃(w₃|w₁,w₂)

Where Σ_iλ_i = 1

Question 10

What is a “mixture model”?

Answer 10

Any weighted combination of distributions

Question 11

What is any weighted combination of distributions?

Answer 11

A mixture model

Question 12

What is Kneser-Ney smoothing? How is it done?

Answer 12

A smoothing technique for n-grams.

Replace the raw counts of each vector with the count of their distinct histories.

This is given by N₁₊(•w_i) = |{w_i-1:c(w_i-1,w_i) > 0}|

So instead of

P_ML(w) = Count(w_i)/Σ_iCount(w_i)

We have

P_KN(w_i) = N₁₊(•w)/ Σ_iN₁₊(•w_i)

Question 13

What are using smoothing techniques equivalent to?

Answer 13

Bayesian estimation.

Using alpha-smoothing is the same as using a Dirichlet prior

Using Kneser-Ney smoothing is the same as using a Pitman-Yor prior

Question 14

What would we use if we wanted to construct a model that captures semantic similarities between words/sentences?

Answer 14

Word embeddings (also called distributed word representations)

Question 15

An advantage of an n-gram LMs to a distributed LMs

Answer 15

They are much quicker to compute

Question 16

What problems do interpolation and back-off solve?

Answer 16

Previous smoothing operations assign equal probability to all unseen events.

Intuition: We have information about potential unseen sequences because we may create lower-order n-grams.

Question 17

Pro and cons of lower vs higher-order n-grams

Answer 17

1. Higher-order n-grams are sensitive to more context, however they are less robust and have sparser counts.
2. Lower-order n-grams are more robust and have larger counts but are less sensitive to context.

Question 18

Differene between backoff and interpolation

Answer 18

There are two ways to use this N-gram “hierarchy”, backoff and interpolation.

In backoff, if we have non-zero trigram counts, we rely solely on the trigram counts. We only “back off” to a lower order N-gram if we have zero evidence for a higher-order N-gram.

By contrast, in interpolation, we always mix the probability estimates from all the N-gram estimators, i.e., we do a weighted interpolation of trigram, bigram, and unigram counts.