Probability and Language Modelling

Question 1

How can you generally obtain better language models? (3)

Answer 1

More training data: This is limited by resources/money/availability

Better models: This is limited by either one’s own knowledge or one’s resources (hardware for example)

Better estimation method: This is limited by either one’s own knowledge or one’s resources (hardware for example)

Question 2

What is a language model?

Answer 2

A model that assigns a probability to the occurrence of sentences.

Question 3

How many parameters does a mixture of two Gaussians have?

Answer 3

5, 2 for the means, 2 for the variances and 1 for the probability of assigning a value to one of the groups (one of the Gaussians)

Question 4

What is a model’s structure (or form)? What is a model’s parameters?

Answer 4

Structure: the form of the equations,usually determined by knowledge about the problem. Parameters: Specific values set to variables in the model’s formulas that affect its behaviour/shape

Question 5

Relative Frequency Estimation formula

Answer 5

P(x) = count(x)/N where N is the total number of items in the dataset.

Question 6

What is the likelihood?

Answer 6

The probabiltiy of observing a given set of data from a distribution with parameters θ L = P(x|θ) = L(θ|x)

Question 7

Why is trying to use maximum-likelihood estimation to produce probabilities for each item in a large dataset difficult?

Answer 7

Each probability will be close to 0 since the normalisation term will be so large. This creates a sparse data problem.

Question 8

What is a “bag-of-words” model?

Answer 8

A unigram model.

Question 9

What assumption is necessary for the validity of an n-gram model?

Answer 9

The markov assumption up to n “the future word only depends on n-1 previous words”

Question 10

What is the noisy-channel framework?

Answer 10

A framework that assumes that some input Y is affected by some noise/errorful encoding P(X|Y) and produces an output X. So given some output X we want to find a Y that is most like the original input. argmax_y(P(Y|X))

Question 11

How does the entropy of a non-uniform distribution over N outcomes compare to that of a uniform distribution?

Answer 11

Any non-uniform distribution over N outcomes has lower entropy than the corresponding uniform distribution

Question 12

Disadvantages of Language Models (3)

Answer 12

1) No way to share “information” about similar words. e.g.: The probabilities for “She bought a car” and “She bought a bicycle” would have to be calculated from scratch despite thier obvious similarities.

(Solved by class-based language models - model classes of words instead of the words themselves and calculate probabilities of classes instead of words)

2) No way to condition on context with intervening words. e.g. Dr. Gertrude Smith vs Dr. Jane Smith. You would have to use a unigram to calculate “Dr. Gertrude”, “Dr Jane” , “Gertrude Smith” and “Jane Smith”.

(Skip-Grams are a solution to this)

3) Can not handle long-distance dependencies since we limit the context to n-1 words.

Question 13

What’s a parameter?

Answer 13

A variable in the equation that specifies a distribution - it is often determine by the observed data in probability estimation.

Question 14

Given that X is an output sequence and Y is an input sequence.

In language modelling,

1) how is the language model defined?
2) how is the noise model defined?

3) what do we want to obtain

Answer 14

Since there is noise added to the input sequence we know that P(X) ≠ P(Y), the noise model is defined as P(X|Y)

The language model is estimated as P(Y)

We want to know the most likely input sequence so we want to find P(Y|X)

We want to find Y that maximises P(Y|X)

Question 15

For a particular input y and output x, how do we find P(y|x)?

Answer 15

Using Bayes rule, we want to find the value of y that maximises P(y|x) = P(x|y)P(y)

Question 16

Does the noise-model P(x|y) depend on the task?

Does the language model P(y) depend on the task?

Answer 16

The noise model depends on the task given

The language model is task independent.

Question 17

What is the noise-model trained on?
What is the languge models trained on?

Answer 17

Corpus data for both

Question 18

If we can train P(X|Y), why can’t we just train P(Y|X)? (2)

Answer 18

1. We can often do so
2. Training P(X|Y) and P(Y|X) requires input/output pairs which are often limited

i.e. a spellchecking system needs correcly spelt words and incorectly spelt words,

a translation system needs the translation of words or sentence

both of these need annotations and for someone to do the annotation.

• Sometimes we have this data, sometimes we don’t!

Language models can be trained on unannotated data - raw text is easy to get unlike annotated data

Question 19

Give a con and pro of a uni-gram language model

Answer 19

1. Gives the same probabiltiy to sentences that contain the same words regardless of the order.
2. Good for finding the general “aboutness” of an entire document since it will produce the most likely words from that document, you can guess roughly what the document is about.

Question 20

What tasks are bag-of-words (uni-gram) language models good for?

Answer 20

Tasks like lexical disambiguation, text-classification and information retrieval

Question 21

What is the independence assumption in a tri-gram and bi-gram model?

Answer 21

tri-gram: P(wi|wi-1,wi-2), the probability of a word wi is only dependent on the previous two words

bi-gram: P(wi|wi-1), the probability of a word wi is only dependent on the previous word

Question 22

How do we fix the problem of indicating whether a word makes sense at the beginning or end of a sentence?

Answer 22

Use end-of-sentence and start-of-sentence tags - and