Variational Inference

Question 1

The goal of Inference is to learn about latent (unknown) variables trough the posterior. Why is a analytic solution usually not an option?

Answer 1

The marginal integral is usually intractable.

Question 2

Name some options for posterior inference

Answer 2

MCMC sampling
Laplace approximation
Expectation propagation
Variational inference

Question 3

What is the main advantage of variational inference?

Answer 3

It is the most scalable method currently known.

Question 4

What is the main idea behind variational inference?

Answer 4

Approximate the true posterior by defining a family of approximate distrubution q_v and optimizing the variational parameters v.

Question 5

What is the KL (Kullback Leibler) divergence?

Answer 5

KL(p(x)||q(x)) =
integ p(x) log(p(x)/q(x)) dx =
E[log(p(x)/q(x))]

Question 6

What is differential entropy?

Answer 6

H[q(x)]= -Eq[log q(x)]

Question 7

How is the KL divergence often used in variational inference?

Answer 7

Use a KL(q(z) ||p(z|x)) as objective function to minimize

Question 8

What does Jensens inequality state, and for what function do we often use this inequality?

Answer 8

For concave functions f:
f(E[x]) >= E[f(x)]

This is often used for logarithmes,
log E[x] >= E[log(x)]

Question 9

What can we do instead of minimizing KL(q(z) || p(z|x))?

Answer 9

We can maximise ELBO (Evidence Lower Bound):
Eq[log p(x|z)] - KL(q(z) ||p(z)) = 
Eq[log p(x,z)] + H[log q(z|v)] = 
Eq[log p(x,z)] - Eq[log q(z|v)] = 
Eq[log p(x,z) / q(z|v)]

Question 10

What is a mean field approximation?

Answer 10

In a mean field approximation q(z|x) is fully factorized, meaning q(z|x) = prod q(z_i). The resulting distribution is with global parameters beta, and local paramters z_i is given by:
q(b, z|v) = q(B | lambda) prod (z_n | phi_n) with
v = [lambda, phi_1, phi_2…., phi_n]

Question 11

In the mean field approximation the q-factors don’t depend directly on the data. How is the family of q’s connected to the data?

Answer 11

Trought the maximization of ELBO.

Question 12

What is the algorithm for mean field approximation?

Answer 12

1. Initialize parameters randomly
2. Update local variational paramters
3. Update global variational paramters
4. Repeat.

Question 13

What are the limitations of mean field approximation?

Answer 13

Generally mean field tend to be too compact, need a better class for approximation

Question 14

Classical mean field approximation has to evaluate all datapoints to update parameters, making it unscaleable to large datasets. How can we leaviate this problem?

Answer 14

Use stochastic variational inference, updating the parameters with a stochastic subset of the data.

Question 15

How do we maximize the ELBO?

Answer 15

Set q*(z_i) = exp(E{-i}[p(x,z)])

Question 16

What is a important criterion for using mean field approximation with stochastic (noisy) gradients?

Answer 16

The gradients should be unbiased

Question 17

What is a problem with maximizing the ELBO?

Answer 17

ELBO is non-convex, we only get local minima.

Question 18

What is the natural gradient?

Answer 18

Uses one datapoint to compute the gradient of distrubutions with respect to the parameters v, for example the gradient of KL divergence with respect to v.

Question 19

What are some advantages of the natural gradinet?

Answer 19

• Invariant to parametrization, for example variance vs precission.
• Uses only one datapoint.

Question 20

How do we update the global parameter using noisy gradients?

Answer 20

Use a running mean so:

lambda_t

Question 21

What happens if we try to use the cavi to do Bayesian logistic regression?

Answer 21

We get a mean that can’t be calculated in closed form.

Question 22

Why can’t we use monte carlo approximation to calculate the intractable mean when doing inference on logistic regression?

Answer 22

Can’t push gradients trough monte carlo sampling. (We can optimize a lower bound instead, but the lower bound is model spesific).

Question 23

Why do we have to swap the order of the integraton (for ELBO expecations) and derivation in BBVI ( Black box VI)

Answer 23

The integrals are intractable for non-conjugate models, which makes gradient computation difficult (impossible?).

Question 24

What is the idae behind score function gradients?

Answer 24

Switch the order of integration and differentation, then simplify the expectation computations

Question 25

How can we practically calculate the score function gradient?

Answer 25

MC estimate

Question 26

What do we need to calculate the score funciton gradients?

Answer 26

1. Sampling from q
2. Evaluate score function gradient_v {log q(z|v)}
3. Evaluate log q(z|v) and p(x,z)

Question 27

What is a problem with the score function gradients?

Answer 27

As we use MC sampling they are noisy. This can be leviated by for example control variates.

Question 28

If f: X -> Y so f(x) = y, how are integ{xdx} and integ{ydy} related?

Answer 28

integ{ydy} = det(dF/dx) integ{xdx}. The “area” is multiplied by the determinand of the Jacobian.

Question 29

What are the properties of pathwise gradient compared to score function gradient?

Answer 29

Lower variance, but more restricted model classes (differentiable models and z = t(e,v) )

Question 30

What is the score function ELBO gradient and the pathwise ELBO gradient

Answer 30

Score:
Eq[g(z, v) * dv log q(z|v)]
Pathwise:
Ep[dz g(z,v) * dv t(e,v)]

Question 31

What is the idea behind behind amortized inference?

Answer 31

Learn a mapping, f(x_n, theta) = phi_n, from datapoints to local parameters. This means:

1. We do not need to learn any local parameters
2. No more independent update of local and
3. New theta can be found using SGD

Question 32

What is the idea of atuoregressive distributions?

Answer 32

we make z_i dependend on all former z_j with j < i.

Question 33

What is the idea of normalizing flows?

Answer 33

We aply k invertible transformations to q(z|v)