LANGUAGE MODELS AND REPRESENTATION

Question 1

What is a model

Answer 1

an abstract representation of something,
often in a computational form
– e.g. tossing a coin

Question 2

What is the BoW representation

Answer 2

The simplest model
Unordered bag of words including repetitions
Disregards grammar and word order but keeps track of the frequency of each word
Can also have bag of terms, bag of tokens, back of stems, etc

Question 3

Enhancing the BoW representation

Answer 3

Ranking words by using weights
skipping stop words

Question 4

What is Zipf’s law

Answer 4

The frequency of any word in a given collection is inversely proportional to its rank in the frequency table
The most freq word appears twice as much as the second, thrice as much as the third etc

Question 5

What is Luhn’s Hypothesis

Answer 5

The words exceeding the upper cutoff are considered to be common
Those below the lower cut-off are rare, and therefore not considered to be contributing significantly to the content of the article

Question 6

What is vector representation

Answer 6

A way to implement the BoW model
Each document is represented as a vector
d = [w1,w2,w3,…]
Where w1 is the weight of term 1

This is a very high-dimensional representation
Millions of dimension
Very sparse vectors - most entries are zero
0 if a word does not exist, 1 if it does (if not weighted)

Question 7

What does it mean to represent a term as an incidence

Answer 7

1 if word exists in doc,0 if it does not

Question 8

What is document frequency (dft)

Answer 8

The number of documents that contain term t
Inverse measure of the ‘informativeness’ of t

Question 9

What is the inverse document frequency

Answer 9

idft = log10 ( N / dft )
– High for terms that appear in few documents
– Low for terms that appear in many documents

N= total occurrences of t across all data

Question 10

Why do we use log in idf

Answer 10

log scaling is used to dampen the effect of very high term frequencies

Question 11

What is tf*idf weighting

Answer 11

Most common method for weighting
tf.idf = (1 + log10tf) x log10 (N / dft)

Note: still a very sparse model

Question 12

When does tf.idf increase

Answer 12

With the number of occurrences in a document(tf)
with the rarity of the term in the collection(idf)

Question 13

Why do we adapt tf in the tf*idf equation

Answer 13

1+ is a smoothing factor to prevent a 0 where a term does not exist
log10 log scaling is used to dampen the effect of very high term frequencies

Question 14

Unigrams/bigrams/trigrams

Answer 14

uni: this, is, a, sentence
bi: this is, is a, a sentence
tri; this is a, is a sentence

Question 15

character unigram/ bigram

Answer 15

t, h, i, s
th, hi, is

Question 16

What is a probabilistic language model

Answer 16

a function that assigns a probability over a piece of text so that ‘natural’ pieces have a larger probability
measures the probability of natural language utterances, giving higher scores to those that are more common

Question 17

What is a language model for words that appear independently

Answer 17

P(S) = p(w1) x p(w2) x …x p(wn)

Question 18

Simplest unigram language model

Answer 18

p(w) = #(w) / Sumt#(t)
Where #w is the count of occurrences of the word w
and
Sumt#(t) is the total number of words in the training data

Question 19

How to deal with OOV in the language model

Answer 19

OOV = 0 is too harsh because then we will never be able to model something unseen
+1 smoothing

Question 20

Why is the main issue of unigram language modelling

Answer 20

complete loss of word order - different ordering of sentences have the same probability

Question 21

Using the Chain rule

Answer 21

S = the cat in the hat
P(S) = p(the) · p(cat|the) · p(in|the cat) ·
p(the|the cat in) · p(hat|the cat in the)

Markov assumption - take only the one (or two) previous words
P(S) = p(the) · p(cat|the) · p(in|the cat) ·
p(the|cat in) · p(hat|in the)

Question 22

Chain rule: unigram v bigram

Answer 22

Unigram
P(S) = p(the) · p(cat)· p(in) · p(the) · p(hat)

Bigram
P(S) = p(the) · p(cat|the) · p(in|cat) ·
p(the|in) · p(hat|the)

Trigram
P(S) = p(the) · p(cat|the) · p(in|the cat) ·
p(the|cat in) · p(hat|in the)

Question 23

Count-based estimation for bigrams

Answer 23

Probability that a word (w1) appears after another word (w2) divided by the total occurrences of w2

<s> I ...
<s> To ...
<s> I ...

P ( I | <s>) = 2/3
P(To |<s>) = 1/3
</s></s></s></s></s>

Question 24

Why are n-grams insufficient language models

Answer 24

because language has long-distance dependencies

“The computer which I had just put into the machine room on the fifth floor crashed.”

But often we can get away with N-gram models

Question 25

What is an evaluation metric

Answer 25

tells us how well our model does on the test set

Question 26

What is extrinsic evaluation of LMs

Answer 26

Best evaluation
Run Models A and B on a task
compare their accuracy

time consuming

Question 27

What is intrinsic evaluation of LMs

Answer 27

Dont apply the model to a task
just assess internal qualities
like giving best probabilities for a test set

but it is often a bad approximation unless the test and training data is very similar

Question 28

The overfitting of Ngrams

Answer 28

• N-grams only work well for word prediction if the test corpus looks like the training corpus
• We need to generalise and avoid overfitting

Question 29

Maximum likelihood estimates

Answer 29

If we see the word ‘computer’ 30 times in a 10,000 word corpus
MLE = 0.003
This may be a bad estimate for other corpus

Question 30

Laplace smoothing

Answer 30

Using the +1 smoothing

Question 31

How do we train for unknown words

Answer 31

Create word token <UNK>
at training we change unknown words to <UNK>
Use <UNK> probabilities for any word not in training
data</UNK></UNK></UNK>