Lecture 9 - Bayesian inference: approximation & sampling

Question 1

What is Bayesian inference?

Answer 1

It involves updating the probability of a hypothesis as more evidence becomes available, using Bayes’ theorem.

Question 2

What is the Laplace approximation?

Answer 2

It approximates a complicated posterior distribution with a simpler multivariate Gaussian 𝑁(𝜇,Σ).

Question 3

What are the key parameters in the Laplace approximation?

Answer 3

1. μ is the mode of the posterior (maximum of 𝑔(𝑤)).
2. Σ=−𝐻−1 , where 𝐻 is the Hessian matrix at w^\hat{}.

Question 4

How is the mean 𝜇 of the Gaussian chosen in the Laplace approximation?

Answer 4

The mean 𝜇 is set to the maximum of the posterior distribution (𝑤^ ).

Question 5

How is Σ determined in the Laplace approximation?

Answer 5

Σ =−𝐻−1, where 𝐻 is the Hessian matrix (second derivative of the log-posterior) at 𝑤^

Question 6

How does the Laplace approximation perform with the Gamma distribution?

Answer 6

It approximates well near the mode 𝑦^ but diverges significantly further away (Page 4, visual example).

Question 7

What is Monte Carlo sampling?

Answer 7

A technique to estimate integrals or expectations by averaging values from random samples

Question 8

How do you estimate the Bayesian predictive distribution?

Answer 8

By sampling 𝑤 from 𝑁(𝜇,Σ) and averaging 𝑃(𝑡new∣𝑤)

Question 9

What is the Metropolis-Hastings algorithm?

Answer 9

It generates samples from the posterior by proposing and accepting/rejecting steps based on an acceptance ratio.

Mnemonic: “Propose, compare, accept (or stay).”

Question 10

How is the acceptance ratio calculated in Metropolis-Hastings?

Answer 10

It combines the posterior ratio and the proposal ratio:

Question 11

What does the Metropolis-Hastings algorithm achieve?

Answer 11

It generates samples from a posterior distribution even when it cannot be computed analytically

Question 12

What are common challenges in sampling?

Answer 12

High-dimensional spaces require many burn-in samples.

Risk of exploring only local maxima.

High rejection rates without careful proposal densities.

Question 13

What is the “burn-in” period in sampling?

Answer 13

The initial samples discarded because they may not represent the posterior accurately.

Question 14

What are the main steps of the Metropolis-Hastings algorithm?

Answer 14

1. Propose a new sample 𝑤~𝑠 based on the previous sample 𝑤𝑠−1
2. Compute acceptance ratio 𝑟.
3. Accept 𝑤~𝑠 with probability
   min(𝑟,1); otherwise, keep 𝑤𝑠−1

Question 15

What are some challenges with the Metropolis-Hastings algorithm?

Answer 15

Requires discarding “burn-in” samples.

Risk of exploring only local maxima.

May reject most proposals if the proposal distribution is poorly chosen.