Test

Question 2

What is a Gaussian Process (GP)?

Answer 2

An (infinite) set of random variables indexed by some set X

For each x ∈ X, there’s a random variable f such that for all A ⊆ X, A = {x1,…,xm}, f_A ~ N(μ_A, K_A).

Question 3

What does the likelihood function P(data | f) represent?

Answer 3

The probability of observing the data given the function f.

Question 4

What is the posterior distribution P(f | data)?

Answer 4

The probability of the function f given the observed data.

Question 5

What are the components of a Gaussian Process?

Answer 5

• Mean function (µ)
• Covariance (kernel) function (k)

Question 6

What is the formula for the predictive distribution in Gaussian Processes?

Answer 6

f | x1,…,xm, y1,…,ym = GP(f; μ0, k0)

Question 7

What is the closed form formula for prediction in Gaussian Processes?

Answer 7

μ* = μ(x) + k,A K^-1(yA - μA)
k = k(x, x’) - k,A K^-1 kA,

Question 8

What is the purpose of optimizing kernel parameters in Gaussian Processes?

Answer 8

To improve predictive performance.

Question 9

What is one method for optimizing hyperparameters in Gaussian Processes?

Answer 9

Cross-validation on predictive performance.

Question 10

What is the Bayesian perspective on optimizing hyperparameters?

Answer 10

Maximize the marginal likelihood of the data.

Question 11

What does maximizing marginal likelihood help with in Gaussian Processes?

Answer 11

It helps guard against overfitting.

Question 12

What is an Empirical Bayes method?

Answer 12

Estimating a prior distribution from data by maximizing marginal likelihood.

Question 13

What computational cost is associated with prediction using Gaussian Processes?

Answer 13

Θ(n^3) due to solving linear systems.

Question 14

What are some basic approaches for accelerating Gaussian Process computations?

Answer 14

• Exploiting parallelism (GPU computations)
• Local GP methods
• Kernel function approximations
• Inducing point methods

Question 15

True or False: The posterior covariance k’ in Gaussian Processes depends on the observed data yA.

Answer 15

False.

Question 16

Fill in the blank: The covariance function in Gaussian Processes is also known as the ______.

Answer 16

[kernel function]

Question 17

What is the effect of kernel parameters in Gaussian Processes?

Answer 17

They influence the shape and smoothness of the function being modeled.

Question 18

What is the significance of the mean function in a Gaussian Process?

Answer 18

It represents the expected value of the function at each point.

Question 19

What do inducing point methods in Gaussian Processes do?

Answer 19

They reduce the computational complexity by approximating the full GP model.

Question 20

What is a common kernel function used in Gaussian Processes?

Answer 20

Squared exponential (Gaussian/RBF) kernel.

Question 21

What is the relationship between Gaussian Processes and Bayesian linear regression in terms of computational complexity?

Answer 21

GP requires Θ(n^3), while Bayesian linear regression requires Θ(nd^2).

Question 22

What is the computational cost of Bayesian linear regression?

Answer 22

𝑂(𝑛 𝑚^2 + 𝑚^3) instead of 𝑂(𝑛^3)

This refers to the cost of using a low-dimensional feature map to approximate the true kernel function.

Question 23

What are the basic approaches for accelerating Gaussian Processes (GP)?

Answer 23

• Exploiting parallelism (GPU computations)
• Local GP methods
• Kernel function approximations (RFFs, QFFs, …)
• Inducing point methods (SoR, FITC, VFE etc.)

Question 24

True or False: Fast GP methods do not address the cubic scaling in n.

Answer 24

True

Fast GP methods yield substantial speedup but still face cubic scaling issues.

Question 25

What is a shift-invariant kernel?

Answer 25

A kernel 𝑘(𝑥, 𝑥’) is called shift-invariant if 𝑘(𝑥, 𝑥’) = 𝑘(𝑥 − 𝑥’)

Question 26

What is the Fourier transform of a shift-invariant kernel?

Answer 26

𝑘(𝑥 − 𝑥’) = K(𝑝(𝜔)𝑒^{-𝜎𝑥’})

This relates the kernel to its frequency representation.

Question 27

What is the key idea behind Random Fourier Features?

Answer 27

Interpret kernel as expectation: 𝑘(𝑥 - 𝑥’) = E[cos(𝜔𝑥 + 𝑏)cos(𝜔𝑥’)]

Question 28

What is the performance theorem for Random Fourier Features?

Answer 28

For compact subset M with diameter diam(M), the probability bound holds: Pr[sup |𝑧(𝑥) − 𝑘(𝑥, 𝑥’)| ≥ 𝜖] ≤ 2d exp(-diam(M)^2 / (𝜖^2 * (d + 2)))

Question 29

What is the primary function of inducing point methods?

Answer 29

Summarize data via function values of f at a set of m inducing points.

Question 30

Fill in the blank: A kernel 𝑘(𝑥, 𝑥’) for 𝑥, 𝑥’ ∈ ℝ+ is called ______ if 𝑘(𝑥, 𝑥’) = 𝑘(𝑥 − 𝑥’).

Answer 30

shift-invariant

Question 31

What are examples of stationary kernels?

Answer 31

• Gaussian
• Exponential
• Cauchy

Question 32

True or False: The Gaussian kernel has the Fourier transform that is the standard Gaussian distribution in d dimensions.

Answer 32

True

Question 33

What do local GP methods exploit?

Answer 33

Covariance functions that decay with distance of points.

Question 34

What is the outcome of applying Bayesian linear regression with explicit feature maps?

Answer 34

It approximates Gaussian Processes.

Question 35

What is the computational cost of Gaussian Processes inference?

Answer 35

Requires solving linear systems.

Question 36

What modern GP libraries implement parallel GP inference?

Answer 36

• GPflow
• GPyTorch

Question 37

What is the main idea behind kernel function approximation?

Answer 37

Construct explicit low-dimensional feature map that approximates the true kernel function.

Question 38

What is the relationship between inducing points and data summarization?

Answer 38

Inducing points help to summarize data by function values at a set of inducing points.

Question 39

What does the theorem by Bochner state regarding shift-invariant kernels?

Answer 39

A shift-invariant kernel is positive definite if and only if p(𝜔) is nonnegative.

Question 40

What does SoR stand for in the context of Gaussian processes?

Answer 40

Subset of Regressors

Question 41

What does the SoR approximation replace in the Gaussian process training conditional?

Answer 41

𝑝 𝒇 𝒖 = 𝑁𝑲𝒇,𝒖𝑲’𝒖,𝟏𝒖𝒖, 𝑲𝒇,𝒇 − 𝑸𝒇,𝒇

Question 42

What is the resulting model from the SoR approximation?

Answer 42

A degenerate GP with covariance function

Question 43

What does FITC stand for in Gaussian processes?

Answer 43

Fully independent training conditional

Question 44

What does the FITC approximation replace in the Gaussian process training conditional?

Answer 44

𝑝 𝒇 𝒖 = 𝑁𝑲𝒇,𝒖𝑲’𝒖,𝟏𝒖𝒖, 𝑲𝒇,𝒇 − 𝑸𝒇,𝒇

Question 45

What is the computational cost for inducing point methods SoR and FITC dominated by?

Answer 45

The cost of inverting 𝐾𝒖,𝒖

Question 46

What is the computational cost’s relationship to the number of inducing points and data points?

Answer 46

Cubic in the number of inducing points, linear in the number of data points

Question 47

What are some methods for picking inducing points?

Answer 47

• Chosen randomly
• Chosen greedily according to some criterion (e.g., variance)
• Equally spaced in the domain
• Random points
• Deterministic grid

Question 48

How can inducing points be optimized?

Answer 48

By treating 𝒖 as hyperparameters and maximizing marginal likelihood of the data

Question 49

What must be ensured about the inducing points 𝒖?

Answer 49

They must be representative of the data and where predictions are made

Question 50

What is the relationship between Gaussian processes and Bayesian Linear Regression?

Answer 50

Gaussian processes = kernelized Bayesian Linear Regression

Question 51

What can be computed in closed form with Gaussian processes?

Answer 51

Marginals / conditionals

Question 52

How are hyperparameters optimized in Gaussian processes?

Answer 52

By maximizing the marginal likelihood

Question 53

What exists for fast approximations to exact Gaussian process inference?

Answer 53

Various fast approximations

Question 54

Which chapters of ‘Gaussian Processes for ML’ by Rasmussen & Williams should be read?

Answer 54

• Chapter 2: 2.1.1-2.3
• Chapter 4: up to 4.2

Question 55

Which paper provides a unifying view of sparse approximate Gaussian process regression?

Answer 55

Quiñonero-Candela & Rasmussen: ‘A Unifying View of Sparse Approximate Gaussian Process Regression’, JMLR 2005

Question 56

Which paper discusses random features for large-scale kernel machines?

Answer 56

Rahimi & Recht: ‘Random Features for Large-Scale Kernel Machines’, NeurIPS 2007