Information Theory

Question 1

Information Theory

Answer 1

the scientific study of the quantification, storage, transmission, and processing of digital information

Question 2

Entropy

Answer 2

the amount of uncertainty in the value of a random variable or the outcome of a random process

Question 3

Information

Answer 3

the resolution of uncertainty

Question 4

Bit

Answer 4

binary digit (1 or 0) that is the most basic unit of knowledge

*1 = on/yes, 0 = off/no
*any communication signal can be quantified into a yes/no question

Question 5

Noise

Answer 5

anything that stands between the message and its destination and can cause distortion of the signal

Question 6

Channel Capacity

Answer 6

the maximum rate of information that can be sent over a channel

*if you increase the rate of transmission, channel capacity can be exceeded

Question 7

Symbol Rate

Answer 7

the number of symbol transfers per second

Question 8

Fundamental Theorem for a Noiseless Channel

Answer 8

1. The amount of scatter in a symbol set determines the minimum channel capacity required, under conditions of maximum efficient coding
2. The amount of scatter in a symbol set determines the minimum efficiency of coding required to transmit a signal drawn from that symbol set over a channel, with a given maximum signal capacity

Question 9

Scatter

Answer 9

how many different symbols there are in the set

e.g. binary code has a scatter of 2, the alphabet has a scatter of 26

Question 10

According the the fundamental theorem for a noiseless channel, how is it possible to create a code that enables nearly error-free transmission

Answer 10

R (transmission rate) < C (channel capacity)

Question 11

Maximum Efficiency

Answer 11

using the least number of bits to transmit information without losing any of it

Question 12

What does higher entropy indicate? Lower entropy?

Answer 12

higher entropy = more uncertainty
lower entropy = more predictability

Question 13

According the the Fundamental Theorem for a discrete channel with noise, inaccuracy can be brought close to zero so long as what two conditions are met?

Answer 13

1. The noise entropy in the channel must be less than the message entropy
   key terms: noise entropy and message entropy
2. The sum of the source entropy + noise entropy in the channel cannot exceed the channel capacity
   key terms: source entropy, noise entropy, channel capacity

Question 14

Noise Entropy

Answer 14

the amount of uncertainty introduced by the noise in the communication channel

*not a measure of the noise itself, but the uncertainty caused by the noise

Question 15

Message Entropy

Answer 15

the amount of information or complexity in the actual message being sent

Question 16

Source Entropy

Answer 16

the amount of uncertainty in the message being sent

Question 17

How does a discrete channel transmit information?

Answer 17

it transmits information in distinct, separate units or symbols