Gaussian Processes

Question 1

What is a Gaussian Process?

Answer 1

A generalization of the multivariate Gaussian distrubution to infintly any variables. Formally, a Gaussian process is a collection of random variables, any finit number of which is Gaussian distributed.

Question 2

What is the formula for posterior mean in a gaussian process?

Answer 2

m_post() = m() + k(, X)(k(X, X) + sigma^2I)^(-1)(y-m(X))

Question 3

What is the formula for posterior cov in a gaussian process?

Answer 3

k_post(, ) = k(, ) - k(, X)(k(X, X) + sigma^2I)^(-1)k(X, *)

Question 4

How can we create new covariance function?

Answer 4

If k1, k2 are covariance funcitons and u(x) is a transform of the input space, then

1) k1 + k2
2) k1*k2
3) k1(u(x), u(x’))

Are also covariance functions

Question 5

Name some parameters of the GP.

Answer 5

Parameters of the mean and covariance function and the noise variance.

Question 6

How can we choose hyperparameters?

Answer 6

Maximize the marginal liklehood with f integrated out (Called Maximum liklehood type II).

Question 7

What three local optima do we often get when minimizing the marginal liklehood, especially with little data?

Answer 7

1. High noise variance, long length scale (almost linear)
2. Medimum noise, medium length scale
3. Low noise, low length scale (Higly non-linear)

Question 8

What do we need to fully specify a Gaussian process?

Answer 8

Mean and covariance function

Question 9

Why don’t we use maximum liklehood or MAP to find hyperparameters?

Answer 9

This will lead to overfitting as it is possible to set f(X) = y and letting the noise go to 0. The marginal likelihood does not fit function values, but integrates them out so overfitting can’t happen in the same way.

Question 10

What properties does a covariance (kernel) function have?

Answer 10

Symetric and postive semi-definit.

Question 11

How do Gaussian processes scale in the training points with respect to training, prediction andd memory requirement?

Answer 11

O(N^3), O(N^2), O(ND + N^2)