Information Theory in Neuroscience

Question 1

What is the aim of information theory ?

Answer 1

It helps to understand how to efficiently encore, transmit and decode data.

Question 2

Define Enthropy

Answer 2

Measure of uncertainty or unpredictability in a set of possible outcomes

Question 3

Define Information

Answer 3

The reduction of uncertainty: When a message is received, it provides information it reduces the possible unknowns.

Question 4

Define “Compression”

Answer 4

Technique to represent information with fewer bits by eliminating redundancy.

Question 5

Define “Channel capacity”

Answer 5

The maximal amount of information that can be transmitted over a communication channel without error.

Question 6

Define “Error correction”

Answer 6

Methods to detect and correct errors in data transmission or storage.

Question 7

In which unity is the information calculated ?

Answer 7

In bits.

Question 8

How much information is contained in a head or tail experience ?

Answer 8

1 bits (50% chance of tail, 50% chances of head, 0.5 + 0.5 = 1)

Question 9

Define what “bits” refers to in information theory

Answer 9

The average nbr of yes/no question required to ascertain the value of a variable.

Question 10

Site advantages of information theory (5)

Answer 10

-Model Free (Not necessary to hypothesize a specific structure to the interactions between variables in a data set to use information theory).
-You can use any mixture of data type
-Detects linear and non linear interactions.
- Multivariate (Possess several metrics design to quantify the behavior of systels with any number of variable).
- Results in bits (facilitates straightforward comparison)

Question 11

What can information theory tell you ?

Answer 11

Quantify uncertainty of one or more variables as well as the influence of one or more variables on one or more other variables.

Question 12

What is the main limit of information theory analysis ?

Answer 12

Can’t produce a model that describres how a system works. It can only be used to restrict the space of possible models.

Question 13

Define what is a probability distribution

Answer 13

A distribution that describes the likelihood of certain outcomes of a random variable or group of variable.

Question 14

How is noted the probability distribution?

Answer 14

p(A)

Question 15

Is a p(A) discrete or continuous ?

Answer 15

Can be both.

Question 16

In a discrete probability distribution, what must be the value of the sum of the possible states and the integral of possible values ??

Answer 16

1

Question 17

With what type of distribution can we describe a system of more than one variable?

Answer 17

A Joint probability distribution

p(c1, c2) = p(c1)p(c2)

e.g : two independent coins

Question 18

What is a joint probability distribution ?

Answer 18

Describe a system with more than one variable.

p(c1, c2) = p(c1)p(c2)

Question 19

What is a maginal probability distribution ?

Answer 19

Represents the likelihood for the outcomes of a subset of variables in the joint distribution.

it describes the probabilities of one or more variables while ignoring (or marginalizing over) the others.

Question 20

How is calculated the marginal probability distribution ?

Answer 20

Summing across certain variables in a joint distribution.

p(c1) = ∑ p(c1, c2)
(c2)

Question 21

If we calculate the probability distribution of a system with two magically linked (dependent) variables, will the distribution will be uniform ?

Answer 21

No

Question 22

What is a conditional probability distribution ?

Answer 22

It is another way to represent probabilities in system full of multiple variable. It describes the likelihood to obtain outcomes of certain variables assuming that other variables are known.

Question 23

How is noted the conditional probability?

Answer 23

It is noted as:
p(A|B) (“the probability of A given B”).

Question 24

What is Data Binning ?

Answer 24

Binning multiple observations of some variables across time or trials). It is a pre processing technique used to reduce the effect of minor observation error.

E.g : measuring voltage we could bin datas in these three categories:
>1, <-1 or ]-1, 1[.

Question 25

How is estimated the probability of a state ?

Answer 25

Total nbr of observation of that state divided by the total number of observations for all states. s : State N(s) : Nbr of experimental observations Nobs: Total nbr of experimental observations p(s) = N(s) / Nobs

Question 26

What guideline can we follow to know how many observations we perfom to adequately sample the space of possible joint states ?

Answer 26

- >10 observations per possible state is ideal.

Question 27

What is meant by the assumption of stationarity ?

Answer 27

The assumption that each observation contributes to a picture of the same probability distribution.

Question 28

What is discretization

Answer 28

Converting continuous data into discrete data.

Question 29

What are the two main binning procedures ?

Answer 29

Uniform width and uniform count.

Question 30

What is the principle of uniform width binning ?

Answer 30

Dividing the total range of the data into Nbins nbr of equal-WIDTH bins.

Question 31

What is the principle of uniform count binning ?

Answer 31

Divinding the total range of the data into Nbis equal-COUNT bins : probability of falling into one bin is the same as for others.

Question 32

What is parameter fishing

Answer 32

Testing parameters until statistically different. Leads to false positive and misleading conclusions. It is a form of p hacking: manipulation of analysis to get desirable fishing.

Question 33

What is a null model ?

Answer 33

Default model used in scq exp to represent what you would expect to happen if nothing special is going on. Used as comparison.

Question 34

What are Kernel-based or binless strategies

Answer 34

Other methods for handling continuous values.

Question 35

How is noted enthropy ?

Answer 35

H(x)

Question 36

What is H(X)

Answer 36

enthropy

Question 37

Does the uniform count binning procedure minimize or maximize enthropy ?

Answer 37

It maximizes the enthropy

Question 38

How many bits is a Byte ?

Answer 38

8

Question 39

What is joint enthropy ? how is it noted ?

Answer 39

H(X, Y), enthorpy for sys with more than one variable.

Question 40

What is H(X, Y)

Answer 40

Joint enthropy

Question 41

What is conditional enthropy ? how is it noted ?

Answer 41

Quantifies the average uncertainty in a variable given the stateof another variable. H(X | Y)

Question 42

Express what is joint enthropy in fct of enthropy and enconditional enthropy.

Answer 42

H(X, Y) = H(X) + H(Y | X)

Question 43

Explain what is this H(X) = H(X|Y) + I(X;Y)

Answer 43

The enthropy of event X is the sum of the conditional entropy (uncertainly of a variable given the knowledge of another variable) and of the information provided by Y about X (= I(X;Y))

Question 44

What is the measure of I(X;Y) and what is it

Answer 44

bits Information by Y about X.

Question 45

what is p(x, y) if the two variables are independant ?

Answer 45

p(x, y) = p(x)*px(y)