Week 4: Dimension Reduction

Question 1

What does projection refer to in dimension reduction techniques?

Answer 1

The fact that we project high-dimensional data onto lower dimensions

Question 2

What are the two main reasons for dimension reduction?

Answer 2

1) To make parameter estimation easier (parameters increase w/ dimensions), 2) to make visualization of data easier.

Question 3

What is the learning task in PCA?

Answer 3

To choose how many dimensions we want to project into (D) and then pick a projection vector, w_d, for each.

Question 4

How can dimension reduction prevent overfitting?

Answer 4

It reduces the number of model parameters and thus reduces the risk of overfitting.

Question 5

How do we notate the new predictors (where M = new dimensions &laquo_space;p = original predictors) in dimension reduction?

Answer 5

Z1, Z2,…,ZM

Question 6

How do we select M, the number of new predictors?

Answer 6

Via cross-validation.

Question 7

What does scale-invariant mean?

Question 8

What does it imply for the method that PCR and PLS are not scale-invariant?

Answer 8

That we have to standardize each original input xj to have mean 0 and variance 1.

Question 9

When don’t we have to standardize the predictors X before creating the new, constructed predictors z?

Answer 9

If all x:s are measured in the same units.

Question 10

What are the dimensions of the X matrix of the p original variables?

Answer 10

N x p (for each x_i we have p values corresponding to that x value for each parameter)

Question 11

What can we decompose each x into?

Answer 11

A direction vector (e_i) and coefficients (x_1i,..x_pi).

Question 12

Where will the new observed values of a 2-dim reduction (into 1-dim) when projecting onto the direction e_1?

Answer 12

On the x-axis only.

Question 13

What does it mean in terms of information when projecting on e_1, and what can we do about this?

Answer 13

That we loose all information about x_i2. Instead, we can find a new direction v_1 that is neither of the two original dimensions (Y and X-axis).

Question 14

How can we write the standardized inputs, Z11,…Z1N (1-dim) in terms of matrices of the original variable and new direction vector?

Answer 14

Z_1 = Xv_1

Question 15

How does PLS and PCR respectively choose the direction vector v_1 (and hence Z_1)?

Answer 15

PCR: choose v_1 that minimize distance to X,

PLS: choose v_1 that explains y well.

Question 16

Which constraint do we need to consider when choosing an arbitrary direction vector, v_1?

Answer 16

v_1 ^T v_1 = 1

Question 17

How can we divide the minimzation problem (and what is is?) of the PCR into two different optimization problems? Why a maximization?

Answer 17

We can split it into first a minimization problem using the x:s, minus a maximization problem.

The mazimization part follows from the fact that we want to find v_1 such that the variance of the projections of x_i onto the direction vector is maximized - to preserve information.

Question 18

What does it mean for v_1 to minimize squared loss, in terms of the new variable?

Answer 18

For the sample variation of the new variable, Z_i1 (1-dim), to be maximized.

Question 19

Explain the PLS algorithm.

Question 20

Out-of-sample

Question 21

How is PCR and ridge regression similiar?

Answer 21

Both operate via the principal components of the input matrix.

Question 22

When is a square matrix orthogonal?

Answer 22

Sq. matrix A is orthogonal if A^T = A^(-1) or if AA^T = A^T A = I.

Question 23

What is the OOS R-sq. and how can it be interpreted?

Answer 23

Intepretead as the fraction of out-of-sample variation in output variable explained by our predicted model.