Natural Language Processing

Question 1

What is term frequency?

Answer 1

How often does this term appear in the document (prominence)

Question 2

What is inverse document frequency?

Answer 2

How infrequently does the term appear across docuemnts

Question 3

What is Bayes Theorem?

Answer 3

P( A | B) = P(B |A) P(A)/P(B)

Question 4

What is laplace smoothing?

Answer 4

Add one smoothing simply adds a constant to each count. This accounts for overfitting.

Question 5

What is information theory?

Answer 5

Study of transmission, storage and retrieval of digital information.

Question 6

What is entropy?

Answer 6

The average uncertainty of a random variable:
H(p) = H(X) = -weighted_sum(p(x)log2p(x))

Question 7

What is joint entropy?

Answer 7

Specify 2 variables, information required for both
chgeck slides for formula

Question 8

What is conditional entropy?

Answer 8

Civen one variable, how much info to specify to the other.
Check slides for formula

Question 9

What is the chain rule?

Answer 9

H(X, Y) = H(x) + H(Y|X)

Question 10

What is mutual information?

Answer 10

H(X) + H(Y|X) = H(Y) + H(X|Y)
therefore: H(X) - H(X | Y) = H(Y) - H(Y |X)

Question 11

What is the noisy channel model?

Answer 11

First used in speech recognition, is used to reconstruct a message from an input channel.

Question 12

What is a statistical language model?

Question 13

What are the advantages and disadvantages of Neural networks?

Answer 13

Advantages:
- Unlimited input length
- Model size is independent of input size
- History dependent
- Model parameters shared across time steps

Disadvantages:
- Long delays are a problem
- Can’t see the future

Question 14

What is the statistical language model?

Answer 14

mi = argmax p(zh|mi)p(mi)
Translation models have been estimated by aligned corpora, and this makes it hard to estimate p(zh|mi).
p(zh| mi) - is the translation model
p(mi) - is the language model

Question 15

What are some facts about chatGPT?

Answer 15

Trained 93% english, 7% other.
By June 2020: 175 x 10^9 parameters defined f

Question 16

What is the attention model?

Answer 16

This model allows the RNN to pay attention to specific parts of the input that is considered as being important, which improves the performance of the resulting model in practise.
Check slide for formula

Question 17

What is attention weight?

Answer 17

The amount of attention that the output should pay to the activation .
Check slide for formula