Week 5

Question 1

MAP (abbreviation)

Answer 1

Maximum A Posteriori method

Question 2

What does the MAP method output?

Answer 2

It finds only a single value of a parameter that maximizes the posterior, so it forgets the uncertainty in the posterior.

Question 3

What is the Laplace approximation?

Answer 3

When you don’t have a formula for the posterior, the Laplace approximation finds a different distribution that tries to approximate the posterior.

Question 4

In what ways should the approximation agree with the posterior in Laplace approximation?

Answer 4

The maximum likelihood, w^ needs to be the same, so this fixes the choice of the mean for the approximation. Plus the area around w^ needs to behave the same in both functions.

Question 5

What property does the multivariate Gaussian of a Laplace approximation need to have in order to satisfy agreement of the area around w^?

Answer 5

It needs to have the same Hessian of w^ as the actual posterior.

Question 6

What two things of the posterior do you need in order to construct the laplace approximation for a particular posterior?

Answer 6

The point of maximization and the 2nd derivatives.

Question 7

T/F:
The gamma distribution is a continuous distribution

Answer 7

True

Question 8

Over what is the density of the gamma distribution defined?

Answer 8

It is continuous, so on positive real numbers.

Question 9

Over what is a Gaussian distribution defined?

Answer 9

Over positive and negative reals.

Question 10

Over what is the beta distribution defined?

Answer 10

between 0 and 1.

Question 11

In Bayesian statistics, what is the distribution called that you need to have when you want to compute the probability that a specific new datapoint has a certain value?

Answer 11

The Bayesian predictive distribution.

Question 12

What happens in Monte Carlo sampling?

Answer 12

Draw many random independent samples of w, compute P(t.new|w,…) for each w and take the average of these.

Question 13

How can you write the expected value of a discrete random variable?

Answer 13

as a summation of all possible values with their probabilities.

Question 14

What is the output of the Metropolis-Hastings algorithm?

Answer 14

a sequence of random samples, w1, …, wNs, with Ns= the number that we determine beforehand.

Question 15

What requirement of samples do you drop when using the MH algorithm?

Answer 15

The requirement that samples are independent.

Question 16

What does the MH algorithm do?

Answer 16

It explores the space of w’s and if it finds a region with large density of w’s it picks more before it continues to another region.

Question 17

Describe the MH algorithm in pseudo-code:

Answer 17

choose w1
for s=2,3,…,Ns
{ propose new sample ~ws
compute acceptance ratio r
if probability is min(1,r)
{ accept proposal, so ws= ~ws}
else
{ reject proposal, so ws= ws-1}
}

Question 18

How do you pick w1 in the MH algorithm?

Answer 18

By sampling from the prior.

Question 19

is MH an optimization algorithm?

Answer 19

no

Question 20

What is the acceptance ratio a product of?

Answer 20

The posterior ratio and the proposal ratio.

Question 21

How is the posterior ratio calculated in the MH algorithm?

Answer 21

The posterior of the proposed point is divided by the posterior of the previous point.

Question 22

what is the goal of the proposal ratio?

Answer 22

it compensates for tendencies of the proposal distribution. If the proposal density has a tendency to go to the upper right, the ratio should not always accept upper right points.

Question 23

Why is the proposal ratio for a multivariate gaussian always equal to 1?

Answer 23

Stepping in either direction is equally likely

Question 24

What is ||x||?

Answer 24

The Euclidian norm of x

Question 25

What is a Euclidian norm?

Answer 25

The length of the vector when measured from (0,0).

Question 26

For MH with a symmetric proposal distribution, the acceptance ratio depends on…

Answer 26

the posterior density at the proposed point and the posterior density at the previous point.

Question 27

Why can the MAP estimate be considered a step up from the maximum likelihood solution?

Answer 27

it incorporates the prior.

Question 28

Burn-in

Answer 28

The first x samples in the MH algorithm, called burn-in because it may take a while to find regions where the posterior is large.