Entropy

Question 1

What is the entropy of a random variable X?

Answer 1

H(X) = -ΣxP(x)log2(P(x))

Question 2

Entropy of a uniform distribution of N outcomes

Answer 2

log2(N)

Question 3

What has more entropy? A non-unfiorm distribution over N outcomes or a uniform distribution over N outcpmes?

Answer 3

Any non-uniform distribution over N outcomes has lower entropy than the corresponding uniform distribution

Question 4

What is cross-entropy and what does cross-entropy model/measure?

Answer 4

(-1/n)log(P_{M}(W)
How well a model M predicts the data

Question 5

What is perplexity and what does it measure?

Answer 5

2^{cross-entropy}, it models the same thing as cross-entropy, how well a probability distribution or probability model predicts a sample.

Better models q of the unknown distribution p will tend to assign higher probabilities q(xi) to the test events. Thus, they have lower perplexity: they are less surprised by the test sample.