Language Modelling

Question 1

What is a language model?

Answer 1

It is a model that assigns probabilities to sequences of worsd

Question 2

What is the most basic of language models?

Answer 2

The n-gram model, which assigns probabilities to sentences and sequences of words

Question 3

What can n-gram models be used to?

Answer 3

Estimate the probability of the last word of an n-gram given the previous words, and assign probabilities to entire sequences

Question 4

Where can language models be used?

Answer 4

Speech Recognition
Spelling correction or grammatical error correction
Machine translation
Augmentative and alternative communication systems

Question 5

What does a unigram assume?

Answer 5

That the appearance of words are independent of each other
P(w1, w2, w3, w4) = P(w1) * P(w2) * P(w3) * P(w4)
It assumes that the previous words have no influence on the next word

Question 6

What do n-gram models inform us?

Answer 6

The probability of the next word in the text is dependent on the previous n-1 words in the text.
If we have a word w, and some history h, we want to find out of the times that h occurred, how many times was it followed by w.

Question 7

In general, how do n-gram probabilities work?

Answer 7

P(w1) = P(w1)
P(w1,w2) = P(w2 | w1) * P(w1)
P(w1, …, wn) = P(wn | w1, …, wn-1) * P(w1, …, wn-1)

Question 8

How does the bigram model approximate the probability of the next word?

Answer 8

We can use the probability of the next word given the previous word:
P(wn | wn-1)

Question 9

What assumption is made with the n-gram model?

Answer 9

The Markov Assumption - the probability of a word depends only on the previous word, this can be generalised to trigrams and so on

Question 10

What does MLE stand for?

Answer 10

Maximum Likelihood Estimation

Question 11

What does MLE do?

Answer 11

It is the process of choosing the right set of bigram parameters to make our model correctly predict (maximise the likelihood of) the nth word in the the text

Question 12

How do we obtain the MLE for the parameters of an n-gram model?

Answer 12

We observe the n-gram counts from a representative corpus
Normalise them (dividing by a total count) to lie between 0 and 1

Question 13

What format is used when computing language model probabilities?

Answer 13

We use the log format so we can add the probabilities instead of multiplying them

Question 14

What are some n-gram problems?

Answer 14

Even if we have a large corpus, only a tiny minority of possible n-grams exist in any corpus. The probability of a word appearing given the previous word is 0 for most sequences due to the sparsity of the matrix.

Question 14

What are some n-gram problems?

Answer 14

Even if we have a large corpus, only a tiny minority of possible n-grams exist in any corpus. The probability of a word appearing given the previous word is 0 for most sequences due to the sparsity of the matrix.

Question 15

What do we use to keep a language model from assigning a zero probability to unseen n-grams?

Answer 15

A number of smoothing/discounting methods.

Question 16

What does Laplace smoothing do?

Answer 16

It adds 1 to all the bigram counts, before we normalise them into probabilities, meaning that all counts that were 0 are now 1, 1s become 2s, and so one.

Question 17

What is another name for Laplace smoothing?

Answer 17

Add-one smoothing

Question 18

How do we normalise the count when we do Laplace smoothing?

Answer 18

We add one to the observed bigram, and divide by the number of appearances of N, plus V, where V is the the total number of words in the vocabulary

Question 19

What can we do if instead of changing the numerator and denominator in Laplace smoothing?

Answer 19

We can add one to the count, multiplied by N and all divided by N + V

Question 20

What are some problems with Laplace smoothing?

Answer 20

The extra V observations added to normalisation are problematic as too much probability mass has been moved to the zero-occurrence cases

Question 21

What is k-smoothing?

Answer 21

It is a way of moving a bit less probability mass from the seen to unseen events.

Question 22

What is the equation for k-smoothing?

Answer 22

Question 23

Does k-smoothing solve the problem of imbalanced probability counts?

Answer 23

No - it can still be problematic

Question 24

What are some alternatives to smoothing?

Answer 24

Backoff and interpolation

Question 25

What is backoff?

Answer 25

It is where if we do not have data about higher order n-grams, we can fall back to information about lower order n-grams
For example, if we don’t have trigrams, we use bigrams, if we don’t have bigrams, we use trigrams

Question 26

What does backoff allow?

Answer 26

It allows for generalisation for contexts that the model has not been trained on

Question 27

Explain how backoff is different to interpolation

Answer 27

Backoff only uses lower order n-gram if we have zero evidence of higher order n-grams
Interpolation always mixes the probability from all the n-gram estimators, weighing and combining the counts

Question 28

How does simple linear interpolation work?

Answer 28

We combine different order n-grams by linearly interpolating the models, we add weights to the trigram, bigram and unigram, where the weights sum to one to get the probabilities

Question 29

What is interpolated Kneser-Ney smoothing?

Answer 29

It makes use of two approaches - an absolute discounting factor of 0.75 and the probability of a unigram being a novel continuation of another word in a bigram.
An example is a frequent word, e.g. Britain, will have a low continuation probability if it only appears in a single context of Great Britain

Question 30

What is an even better way than interpolated Kneser-Ney and Laplace smoothing to solve the discounting problem?

Answer 30

Stupid backoff - although it only works for very large models

Question 31

What is stupid backoff?

Answer 31

Stupid backoff is where you multiply the lower order estimate probability by a constant factor (0.4) where the count is 0

Question 32

What is class-based N-gram model?

Answer 32

It is where probabilities of words are based on the clases of the previous words. E.g. you cluster Shanghai with other city names and the estimate based on the likelihood of matching phrases from the entire cluster

Question 33

What are skip-grams?

Answer 33

Skip grams is where we have a value k, and we take words that are not necessarily adjacent, but separated by a distance k.
In the text: “the rain in spain falls mainly on the plain”, the 1-skip-2grams are: all the bigrams, and the non adjacent pairs, such as the in, rain spain, in falls, spain mainly…

Question 34

What are skip-grams good for?

Answer 34

They capture word contexts with less sparsity than strict n-grams - useful in Word2Vec