Chapter 2_2 Information Theory

Question 1

information theory

Answer 1

Information theory deals with uncertainty and the transfer or storage of quantified information in the form of bits.

Question 2

entropy

Answer 2

the quantified uncertainty in predicting the value of a random variable

how does one quantitatively measure the randomness of a random variable?

Roughly, the entropy of a random variable X, H(X), is a measure of expected number of bits needed to represent the outcome of an event x~X

H(X) = - ∑p(x) log p(x)

Question 3

H(X) = 0

H(X) = 1

Answer 3

the outcome is known

there is complete randomness in that all events are equally likely to happen.

the amount of randomness varies from 0 to 1

Question 4

H(Y) < H(Z)

Answer 4

a random variable Y always takes a value of 1

a random variable Z is equally likely to take a value of 1, 2, or 3. So, in this case, H(Y) < H(Z), since the outcome of Y is much easier to predict than the outcome of Z

Question 5

conditional entropy

Answer 5

Intuitively, if X is completely determined by Y, then H(X | Y) = 0 since once we know Y, there would be no uncertainty in X,
whereas if X and Y are independent, then H(X | Y) would be the same as the original entropy of X, i.e., H(X | Y) = H(X) since knowing Y does not help at all in resolving the uncertainty of X.

Question 6

mutual information

Answer 6

Another useful concept is mutual information defined on two random variables, I(X; Y), which is defined as the reduction of entropy of X due to knowledge about another random variable Y, i.e.,