Week 6

Question 1

What is Information Theory?

Answer 1

It answers “how to quantify information?”

Question 2

How is information quantified in Information Theory?

Answer 2

In terms of probability distributions and “uncertainty” (surprise), disregarding semantics.

Question 3

What are the three axioms on which Shannon proposed Self-Information?

Answer 3

• An event with probability 100% is perfectly unsurprising, yielding no information.
• The smaller the probability, the greater the surprise/information of an event.
• If two independent events are measured separately, total info amount is the sum of the individual self-informations.

Question 4

What is the equation for Self-Information?

Answer 4

Given random variable X:

lX(x) = -logb[PX(x)] = logb 1/PX(x)

with different bases b.

Question 5

What are the different bases possible for Self-Information?

Answer 5

2: bits
e: nats (natural units)
10: dits

Question 6

What does Entropy quantify?

Answer 6

The uncertainty in a random variable X.

Question 7

What is Joint Entropy?

Answer 7

A measure of uncertainty associated with a set of variables.

Question 8

What is Conditional Entropy?

Answer 8

Quantifies uncertainty of the outcome of a random variable Y given the outcome of another random variable X.

Question 9

What is KL Divergence?

Answer 9

Relative entropy - quantifies the distance betwen two probability distributions.

Question 10

What is Mutual Information?

Answer 10

Measures how much knowing one variable reduces the uncertainty about the other.