Statistical Language Modelling

Question 1

what is NLP

Answer 1

Natural Language Processing builds systems that uses computational techniques to model and process natural languae in an automated way.

Question 2

what is word-level processing

Answer 2

Before doing any text processing, we need to prepare out input data into sentences, then words, then tokens

Question 3

What are the three ways of predicting the probability of a sequence

Answer 3

1. Spellchecking
2. Grammatical error correction
3. Autocomplete/suggestions

Question 4

what are the 4 n-grams

Answer 4

Unigrams
Bigrams
Trigrams
Quadrigrams

Question 5

what is smoothing

Answer 5

techniques to ensure a low probability for unseen combinations without compromising the overall statistics of the training set

Question 6

what are the 3 types of smoothing

Answer 6

1. Laplace smoothing
2. add-k smoothing
3. Kneser-Ney smoothing

Question 7

what is laplace smoothing

Answer 7

adds one to all counts.

Question 8

what is add-k smoothing

Answer 8

rather than adding 1 to all counts, we can generalize to arbitrary k (typically between 0 and 1).

Question 9

what n-gram is better?

Answer 9

• For higher n we capture more context and so we can make better predictions
• but for higher n, we also need more data and inevitably it will be sparse

Question 10

How can we maximize probabilities

Answer 10

Given a tarin set and a test set, we want the model to maximize the probability of the test set. For bigarms this means we want to maximize:
p(w1w2…wn)=P(wi|wi-1)