Regression

Question 1

What is the basic model for linear regression?

Answer 1

Y = f(X) + ε, where f is a linear function modeling E[Y|X], and ε is a noise term.

Question 2

In Bayesian framework, how are parameters typically estimated?

Answer 2

Using the posterior distribution, often with the maximum a posteriori (MAP) estimate.

Question 3

What is the ordinary least squares (OLS) estimate?

Answer 3

θ̂ = arg min_θ Σ(y_i - f(x_i))^2, which minimizes the squared error between predictions and observations.

Question 4

How is the OLS solution calculated when X^T X has full rank?

Answer 4

θ̂ = (X^T X)^-1 X^T y

Question 5

What is ridge regression and how does it differ from OLS?

Answer 5

Ridge regression adds a penalty term: θ̂(λ) = arg min_θ ||Xθ - y||^2 + λ||θ||^2, where λ is the regularization strength.

Question 6

What is a kernel function?

Answer 6

A function κ(x_i, x_j) = φ(x_i)^T φ(x_j), where φ is a feature map.

Question 7

What is the “kernel trick”?

Answer 7

The ability to compute κ(x_i, x_j) without explicitly computing φ(x).

Question 8

Name three example kernel functions

Answer 8

Linear kernel, polynomial kernel, and radial basis function (RBF) kernel.

Question 9

What are hyperparameters in kernel regression?

Answer 9

Parameters of the kernel function and the regularization strength λ (if ridge kernel regression)

Question 10

What is a feature map in Kernel regression?

Answer 10

A feature map in Kernel regression transforms the input data into a higher-dimensional space to make it easier to find linear relationships.

Question 11

What is the basic idea behind random features?

Answer 11

The basic idea behind random features in kernel regression is to approximate the kernel function using a finite set of random projections to reduce computational complexity.

Question 12

What is the main limitation of kernel methods for large datasets?

Answer 12

The kernel matrix grows quadratically with the number of samples.

Question 13

What is kernel regression suited for the best?

Answer 13

High-dimensional data points in moderate datasets.

Question 14

What is the difference between parameters and hyperparameters in a model?

Answer 14

Parameters control the likelihood function, while hyperparameters parametrize the prior distribution in a Bayesian setting.

Question 15

What is the Bayesian interpretation of the ridge regression penalty?

Answer 15

The penalty λ ||θ||^2 can be interpreted as a Gaussian prior.

Question 16

In what case does ridge regression converge to the minimum ℓ2-norm OLS solution?

Answer 16

As λ approaches 0.

Question 17

What is polynomial regression?

Answer 17

A form of regression where Y = θ_1 + θ_2X + θ_3X^2 + θ_4X^3 + … + ε

Question 18

What is the general form of regression in feature space?

Answer 18

Y = φ(X)θ + ε, where φ is a feature map.

Question 19

What is the kernel matrix K?

Answer 19

K = φ(X)φ(X)^T, or K_ij = κ(x_i, x_j)

Question 20

How is prediction made in kernel regression?

Answer 20

ŷ = Σ_i κ(x_i, x_new)η̂_i, where x_i are training points and η̂ are estimated parameters. It requires the full training set.

Question 21

What is the main advantage of random feature approximation?

Answer 21

: It allows kernel methods to be applied to large datasets by reducing computational complexity.

Question 22

In what case should one consider using linear regression in feature space instead of kernel regression?

Answer 22

When there’s a small amount of features and the data is sparse in feature space.

Question 23

if p > n, what happens to OLS estimate?

Answer 23

OLS estminate has infinitely many solutions, we take the theta with minimal length (minimal L2 norm solution, Lagrange problem)

Question 24

What is Moore-Penrose pseudo inverse?

Answer 24

The Moore-Penrose pseudo inverse is a generalization of the matrix inverse that can be applied to non-square or singular matrices to solve linear least squares problems.

Question 25

Benefits of feature maps?

Answer 25

Allows linear models to capture non-linear relationships
Can significantly improve model performance on complex datasets
Keeps the simplicity and interpretability of linear models

Question 26

Drawbacks of feature maps

Answer 26

Can lead to overfitting if too many features are created
May increase computational complexity
Requires careful selection of appropriate feature maps