Bayesian Inference & Decision Theory

Question 1

What is decision theory?

Answer 1

The practice of making decisions under uncertainty

Question 2

What are the different perspectives on distribution parameters between Frequentist Statistics and Bayesian Statistics?

Answer 2

In Frequentist Statistics, parameter theta is considered to be fixed at some point estimate. In Bayesian Statistics, theta is considered to be something that can vary based on some distribution (i.e. posterior probabilities)

Question 3

What is the equation of P(A|B)?

Answer 3

[P(A and B)]/P(B)

or

[P(B|A).PA]/[P(B|A).P(A) + P(B|not A).P(not A)] (for 2 possible events)
*the denominator is the same as P(B)

Question 4

What is the equation for P(A and B)?

Answer 4

P(A|B).P(B)

Question 5

What is the equation for the expected value of a pmf?

Answer 5

summation(x.p(x)) through all outcomes

Question 6

What is the objective of Bayesian Inference?

Answer 6

To logically and coherently update state probabilities as new evidence becomes available

Question 7

Is it always logical to maximise profit or minimise loss?

Answer 7

No. We have to consider the scale of the downside. Sometimes risk-tolerance makes us less perfectly rational

Question 8

What is EVPI?

Answer 8

The expected value of perfect information difference between the best expected outcome with the prior information and the best possible outcome perfect information

Question 9

What is predictive probability?

Answer 9

The overall probability of observing some X in the data

Question 10

What is a posterior probability?

Answer 10

A probability representing theta equalling some value after factoring additional information

Question 11

What is the Excel function for weighted summation?

Answer 11

= SUMPRODUCT(number cells, weight cells)

Question 12

What is the equation for m(x(k))? (I.e. the predictive probability of x(k)

Answer 12

summation through states of [P(x(k) | theta).P(theta)]

Question 13

What does the Latin term “a posteriori” mean?

Answer 13

Dependent on empirical evidence or experience

Question 14

What does the Latin term “a priori” mean?

Answer 14

Independent of experience

Question 15

What is EVSI?

Answer 15

The expected value of sample information is the difference between the best outcome with prior probabilities and the best outcome with posterior.

Question 16

What is the difference between subjective Bayesian Inference and more traditional objective Bayesian Inference?

Answer 16

More objective Bayesian Inference uses objective statistical inference to determine the distributions of observations conditional on the underlying states (i.e. P(X|theta))

Question 17

What are the conditions for a sample from binomial sampling to follow a binomial distribution?

Answer 17

• The population must be so large that the sample does not disturb the population proportions
  or
• We sample with replacement

Question 18

What is binomial sampling?

Answer 18

When a number of observations are sampled and grouped into one of two levels (traditionally success or failure)

Question 19

What is a subjective probability?

Answer 19

A probability indicating the current assessment of how likely it is that the tree value for theta is some value. Normally shown as pi(theta=n)

Question 20

What is the Excel function for calculating binomial probabilities?

Answer 20

=BINOMDIST

Question 21

What would happen to our posterior distribution for theta if we observe an outcome of x=5 in our binomial sample of 100?

Answer 21

The posterior distribution of theta would be centered approximately on theta=0.05 (i.e. the proportion of success in the sample [5/100])

Question 22

What do we do in our setting up of the prior distribution when we want to consider values over some continuous interval as opposed to some set of discrete values?

Answer 22

We assign a probability density function to theta

Question 23

How do we set up our prior distribution of theta if we want to consider all continuous values between some interval with equal likelihood a priori?

Answer 23

We use a uniform distribution for theta (i.e. pi(theta) = 1/[a+b] for a < theta < b)

Question 24

What is the posterior distribution of theta when using a uniform prior and binomial sampling?

Answer 24

pi(theta|x) = k(theta^x)[(1-theta)^(n-x)]

• this is actually a beta distribution where k would be gamma(n+2)/[(gamma(x+1).gamma(n-x+1))]
• where k is a constant that comes about from canceling terms in joint/predictive and scaling to ensure that the resulting distribution integrates to 1

Question 25

Assume we use a beta distribution to model the prior distribution for theta, what is the posterior distribution of theta?

prior = k.[(theta^(a-1)).(1-theta)^(b-1)]

Answer 25

pi(theta|x) = k.[(theta^(a+x-1)).(1-theta)^(b+n-x-1)]

Question 26

What is the expected value of a theta which follows a beta distribution?

Answer 26

E[theta] = a/(a+b)

Question 27

What is the expected value of 1-theta from a beta distribution?

Answer 27

E[1-theta] = b/(a+b)

Question 28

What is the variance of a theta which follows a beta distribution?

Answer 28

var[theta] = ab/[(a+b)²(a+b+1)]

Question 29

How do we calculate what the variance of a theta posterior distribution should be based on the prior expectations?

Answer 29

We use the approximate number of standard deviations in the given range to give the quantity for a single deviation. We then square that and make it equal var(theta)

Question 30

Is the uniform distribution a special case of the beta distribution? If so, what are its parameters?

Answer 30

Yes. The uniform distribution is a beta distribution with alpha=beta=1.

Question 31

What can we say about the centrality of the posterior distribution if we use a relatively informative prior? (like a uniform distribution)

Answer 31

The posterior distribution will be more-or-less centered around the sample means

Question 32

What are “iid” observations?

Answer 32

Observations that are independent and identically distributed

Question 33

What is a likelihood function?

Answer 33

A likelihood function is kind of like the reverse of a probability density function. Instead of taking parameters and giving the probability of x given those parameters (i.e. f(x|theta)), the likelihood function takes the data (x) and returns the likelihood that theta equals some value given the data.

L(theta|x) = product over all n of f(x|theta)

Question 34

Why is the requirement of a prior distribution seen as both a strength and a weakness of the Bayesian approach to inference?

Answer 34

• We sometimes do have real and meaningful prior information which we can then exploit (especially if the results of the analysis are to be used as a basis for decision making)
• Some argue that prior information introduces an element of subjectivity which may conflict with a desire for objectivity in the data analysis.

Question 35

What is the probability density function for theta as using a likelihood function?

Answer 35

pi(theta) = [L(theta|x1,x2,…xn).pi(theta)]/[m(x1,x2,…xn)]

• where m is the predictive probability density function given by the integral over -inf to int of the numerator d(theta).
• The role of the denominator is basically to scale/standardize the posterior distribution so that it integrates to 1.
• Because of this, we often write the posterior in the form of simply k.L(theta|x1,x2,…xn) and say the posterior distribution of theta is directly proportional to L(theta|x1,x2,…xn)

Question 36

What is an advantage of using a normalization constant?

Answer 36

Any factors in L(theta|x1,x2…xn) or pi(theta) which do not depend on theta can be absorbed into the normalization constant and effectively ignored.

Question 37

What are the 3 main areas of statistical inference that we might use the posterior distribution of theta for?

Answer 37

• Hypothesis Testing
• Interval Estimation
• Point Estimation

Question 38

What is a type-I error?

Answer 38

The rejection of H0 when it is in fact true

Question 39

What is a type-II error?

Answer 39

The acceptance of H0 when it is in fact false

Question 40

Given a null hypothesis of theta being some value (theta0) with probability pi0, what are the odds of theta0 being the correct value?

Answer 40

It will be the ratios of the probabilities of theta being theta0 and theta not being theta0.

= [L(theta0|data).pi0]/[L(theta1|data).(1-pi0)]

This represent the product of [the a priori odds] and a likelihood ratio also known as a Bayes Factor.

Question 41

Assume that the cost of a type-I error is CI and that the cost of a type-II error is CII, what is the decision which minimizes expected cost?

Answer 41

Reject H0 if CI.pi(theta0|data) <= CII.pi(theta1|data)

Question 42

How can we approximate values for L and U such that the probability of theta belonging to the interval [L,U] is equal to 100(1-alpha)%? A.k.a Interval Estimation or calculating a credibility interval

Answer 42

We set the integral of the posterior distribution for theta equal to 1-alpha with the bounds L and U then numerically approximate.

Question 43

What is the Bayesian estimate?

Answer 43

The expected value of the posterior distribution. Under certain circumstances, other measures such at the mode or the median will be used as the estimator

Question 44

What is the likelihood function of lambda for a sample from the Poisson distribution?

Answer 44

L(lambda | data) is directly proportional to [lambda^(sum of xi over 1 to n)].[e^-n.lambda]

• We know for sure that lambda is strictly positive so the prior distribution should be restricted to positive values only

Question 45

Assume a prior distribution of theta to be a gamma distribution with parameters alpha and phi. What is the posterior distribution of this prior

Answer 45

Gamma distribution with parameters (alpha+sum of xis) and (phi + n)

Question 46

What is the posterior expectation of of lambda given a gamma posterior distribution?

Answer 46

a/b

*If you expand this, it equates to a weighted average of the prior and sample estimates

Question 47

DO NORMAL SAMPLING

Answer 47

K