Week 7 - Information theory

Question 1

Entropy

Answer 1

H(X) = -E[P(X) * log P(X)]

Question 2

Odds

Answer 2

P(X) / (1 - P(X))

Question 3

Logits

Answer 3

Log(Odds)

Question 4

P(B | A)

Answer 4

P(A n B) / P(A)

Question 5

Joint Entropy

Answer 5

H(X, Y) = - Ei [ Ej [ P(xi, yi) log(P(xj, yj))]]

Question 6

Conditional Entropy

Answer 6

H(X|Y) = - Ei [ Ej [ P(xi, yi) log(P(yj | xj))]]

Question 7

H(X,Y) - H(X)

Answer 7

H(Y|X)

Question 8

Gini Index

Answer 8

Ig(p) = 1 - E(i=1 to J)[(pi)^2] where pi is the fraction of items labeled with class i in the dataset and J is the number of classes

Question 9

IG(Y,X)

Answer 9

H(Y) - H(Y|X)

Question 10

I(X;Y)

Answer 10

H(X) - H(X|Y)

Question 11

Mutual Information

Answer 11

Measures how much knowing one variable reduces the uncertainty about the other