Topic 2: N-gram modeling

Question 1

N-gram

Answer 1

N-gram is a N-token sequence of words. example of bi-gram, tri-gram??

Question 2

N-gram model

Answer 2

language model is a prediction model. predicting words form previous N-1 words.
statistical model
Language Model

Question 3

application of n-gram model

Answer 3

give with example

1. spelling correction
2. speech recognition
3. augmentative communication
4. machine translation

Question 4

Simple n-grams

Answer 4

probability of the word,w given history P( w | h )

Question 5

Relative frequency counts

Answer 5

example…

Question 6

Corpus based estimation

Answer 6

example…

Question 7

Easier Estimation

Answer 7

this utilizes chain rule of probability. example
P(its water was so transparent) 
= P(its) * 
P(water|its)*
P(so|its water was)*
P(transparent|its water was so)|...

Question 8

Intuition of n-gram model

Answer 8

approximate the history by just the last few words instead of computing probability of the entire word history.

Question 9

Markov assumption

Answer 9

N-gram model comes with independent assumption that probability of some future unit can be predicted without looking too far into the past.

Question 10

Exercise bi-gram model with maximum likelihood estimation
I am Sam
Sam I am
I do not like green eggs and ham
Calculate bigram probabilities from this corpus.
P(I|) = 2/3 = 0.67 P(Sam|) = P(am|I) =
P(|Sam) = P(Sam|am) = P(do|I) =

Answer 10

P(I|)=2/3=0.67
P(Sam|)=1/3=0.33
P(am|I)=2/3=0.67 
P(|Sam)=1/2=0.5
P(Sam|am)=1/2=0.5
P(do|I)=1/3=0.33

Question 11

Relative Frequency

Answer 11

obtained by dividing frequency of a sequence by frequency of a prefix

Question 12

Bi-gram count exercise

Answer 12

refer to slides. have unigram count..have bigram table
calculated the probability by
value in bigram table divided by unigram word count

Question 13

what knowledge can be captured by N-gram probabilities.

Answer 13

1. world knowledge
2. syntax
3. discourse

Question 14

Evaluating language models. what are the 2 types of evaluation

Answer 14

extrinsic evaluation - embed language model in an application and measure how it improves, measure end to end performance, expensive
intrinsic evaluation - training and test set. measure quality of model, independent of application

Question 15

Training and testing paradigm

Answer 15

• evaluating different architectures
• development, training and test set
  split 80:20

Question 16

Intuition of perplexity

Answer 16

how to predict next word, a better model is the one predicts the word that’s actually occurs

Question 17

Perplexity as evaluation metric

Answer 17

best language model is the one best predicts an unseen test set.
Perplexity is the inverse probability of the test set, normalized by the number of word.
perplexity calculation….

Question 18

Generalization

Answer 18

the statistical model is highly dependent on the training corpus
it’s pretty useless if the training sets and test sets are from different genre..
business meetings vs movie?

Question 19

challenges in language model

Answer 19

dynamically adapt to different genres

Question 20

Zeros

Answer 20

one kind of generalization: zeros

test set have data occurrences that does not occur in the training set.

Question 21

Incorrect estimation

Answer 21

this is the problem of zeros
underestimated probability that all sorts of words will occur.
if the probability of any word in test set is 0. entire probability of test set is 0.
perplexity can’t be computed

Question 22

Smoothing (Laplace)

Answer 22

introduced to overcome zero problem
how it’s being done?
add 1 to all the word counts denominator is also adjusted with extra V observation

Question 23

Laplace smoothing exercise

Answer 23

probability with added count
bi gram table
vocab number