Dimensionality reduction

Question 1

Data preprocessing for PCA

Answer 1

Mean normalization and feature scaling

Question 2

Mean normalization

Answer 2

Calculate the mean of a data set and subtract it from each data point

Question 3

Feature scaling

Answer 3

Scale features to have a comparable range of values. (ie. divide by std. dev or range)

Question 4

2D PCA vs. linear regression

Answer 4

1) linear regression tries to predict y based on x; PCA treats all features equally 2) linear regression minimizes squared vertical distance; PCA minimizes projected distance from line

Question 5

Steps in PCA

Answer 5

1) Compute covariance matrix (Σ)
2) Compute Eigenvectors of Σ. First k columns of U provide our new basis vectors

3)

Question 6

Covariance matrix

Answer 6

Question 7

Compute Eigenvectors of Σ

(Octave code)

Answer 7

[U,S,V] = svd(Sigma);

Question 8

Compute Σ

(Octave code)

Answer 8

Sigma = (1/m) * X’ * X

Question 9

Define new reduced basis vectors

(Octave code)

Answer 9

Ureduce = U(:,1:k);

Question 10

Convert n-dim data x to k-dim data z

(Octave code)

Answer 10

z = Ureduce’ * x

Question 11

% of Variance Retained

Answer 11

Avg squared projection error / Total variation in the data

Question 12

Determine k dim to retain

Answer 12

1) Run PCA
2) Pick smallest k, st. [99%] of variance retained

Question 13

How to avoid overfitting

Answer 13

Use regularization - not PCA

Question 14

When to use PCA

Answer 14

Always try to run process on raw data first. Use PCA only if there is a good reason

Question 15

Good uses of PCA

Answer 15

Compression: Reduce memory / disk space needed; speed up learning

Visualization: Reduce to 2/3 dimensions