PCA (Brainscape)

Question 1

Can PCA be used to reduce the dimensionality of a highly nonlinear dataset?

Answer 1

PCA can be used to significantly reduce the dimensionality of most datasets, even if they are highly nonlinear, because it can at least get rid of useless dimensions.

Question 2

Suppose you perform PCA on a 1000 dimensional dataset, setting the explained variance ratio to 95%. How many dimensions will the resulting dataset have?

Answer 2

Let look at the two extreme:

First, suppose the dataset is composed of points that are almost perfectly aligned. In this case, PCA can reduce the dataset down to just 1 dimension while still preserving 95% of the variance.

Second suppose a dataset with perfectly random point, it will lower the feature to 95% of 1000 which is 950.

Question 3

In what cases would you use vanilla PCA, Incremental PCA, or kernel PCA?

Answer 3

1) Regular PCA is the default, but it works only if the dataset fits in memory.
2) incremental PCA is useful for large datasets that do not fit in memory, but it is slower.
3) Kernal PCA is usefull for nonlinear datasets.

Question 4

What is the main idea of principal component analysis (PCA)?

Answer 4

Principal component analysis (PCA) is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. This transformation is defined in such a way that the first principal component has the largest possible variance (that is, accounts for as much of the variability in the data as possible), and each succeeding component in turn has the highest variance possible under the constraint that it is orthogonal to the preceding components.

Question 5

When using PCA what are the principal components?

Answer 5

PCA identifies the axis that accounts for the largest amount of variance in the training set. This axis would be our 1th principal component (PC). The algorithm then proceeds to find the second axis, orthogonal to the first one, with the highest variance. That would be our 2th PC. The algorithm then proceeds to the third one and so on.

Question 6

How do you find the principal components (PC) when using PCA?

Answer 6

We need to apply a standard matrix factorization technique called singular value decomposition (SVD) that can decompose the training set matrix X into the matrix multiplication of three matrices:

X = UΣVᵀ

Where V contains the unit vectors that define all the principal components that we are looking for.

Question 7

(REMAIN MISTAKE)When using PCA, once you used the singular value decomposition:

X = UΣVᵀ

How do you project down the data to d dimensions?

Answer 7

X-ₚᵣₒⱼ = XWd

Where Xd-proj is the reduced dataset of dimensionality d, X is the original dataset ,and Wd is defined as the matrix containing the first d columns of V. In other words, Wd contain the orthogonal vectors that induce the lowest reduction in the variance.

note in sklearn pca.explained_variance_ratio tell you the amount of variance contain in each orthogonal axis.