Lecture 12: Information Theory

Question 1

What is information?

Answer 1

Information is surprise.

Eg.. if someone tells you something unexpected you have gained information.

Question 2

Information Content. Explain.

Answer 2

information content is usually expressed as the number of bits needed to communicate a message.

Question 3

Information + Probabilities

Answer 3

the amount of information associated with an even (e), occurring is defined by an “probability of occurrence”
I(e) = information of an event
I(p) = information of a probability

Question 4

Three properties.

Answer 4

1. an event cannot have negative information
2. more similar events carry less info
3. more events provide more information

Question 5

What is Entropy?

Answer 5

Is a source of stream of symbols from a finite set of symbols called the alphabet.

Question 6

What is a Variable length code:

Answer 6

a mapping of strings of source symbols to strings, called “Code words”

1) binary
2) ASCII
3) morse

Question 7

What are several desirable code properties?

Answer 7

1. uniquely decodable - only one corresponding string

2, instantenous - only able to decode after the whole word is attained

Question 8

Codes as Trees. Explain

Answer 8

1. A tree is a graph where node = root
2. each node is connected to a parent node (unique)
   and also to a child node
3. A leaf term called for no child nodes
   4.the edge connecting nodes are labelled with a different symbol in coding alphabet