Lecture 5 - Dimensionality Reduction - Principal Component Analysis, Linear Discrimination Analysis, Singular Value Decomposition

Question 1

What is meant by “Degrees of freedom”?

Answer 1

Degrees of Freedom refers to the maximum number of logically independent values, which are values that have the freedom to vary, in the data sample

Question 2

What is dimensionality reduction?

Answer 2

Dimensionality reduction is the process of deriving a set of degrees of freedom which can be used to reproduce most of the variability of a data set

Question 3

What is the goal of Dimensionality Reduction? And in broad terms how does it work?

Answer 3

Goal: To reduce dimensions by removing redundant and dependent features

How: By transforming features from higher dimensional space to a lower dimensional space

Question 4

What are the different methods that can help us reduce dimensions?

Answer 4

Unsupervised where no need for labelling classes of data:

• Independent Component Analysis (ICA)
• Non-negative Matrix Factorization (NMF)
• Principal Component Analysis (PCA)
  
  • Ideal for visualization and noise removal

Supervised where class labels are considered:

• Mixture Discriminant Analysis (MDA)
• Linear Discrimanant Analysis (LDA)
  
  • Ideal for biometrics, Bioinformatics and chemistry

Question 5

What is Principal Component Analysis (PCA)?

Answer 5

PCA is

• A popular technique for dimensionality reduction.
• A “classical” approach that only characterize linear sub-spaces in data

Involves a dataset with observations on numerical variables

• An exploratory data analysis tool
• A simple, non-parametric method of extracting relevant information from data sets

Question 6

How does PCA reduce dimensions?

Answer 6

PCA reduces dimensions by exposing underlying information in data sets

• An unsupervised approach
• Aims to explain most of the variability in data with a smaller number of variables
• Identifies axis that accounts for the largest amount of variance in the training set

Question 7

You should not use PCA if the data is…

Answer 7

showing some non-linearity

Question 8

What are the three different types of PCA?

Answer 8

1. Randomized PCA quickly finds an approximation of the first d principal components.
   
   • Issue: Whole training set need to fit in memory
2. Incremental PCA (IPCA) splits the traning set into mini-batches and feed an IPCA algorithm one mini-batch at a time
3. Kernel PCA helps perform complex nonlinear projections for dimensionality reduction

Question 9

How can we calculate the PCA?

Answer 9

Primary PCA calculation steps:

• Calculate covariance matrix
• Calculate ordered eigenvalues and eigenvectors of the matrix
• Compute principal components

Question 10

How do we calculate the Principal Components?

Answer 10

Overall PC calculation process:

• For each PC:
  
  • PCA finds a zero-centered unit vector pointing in the direction of PC.
  • Direction of unit vectors returned by PCA is not stable
• If you perturb training set slightly and run PCA again
  
  • Unit vectors may point in opposite direction as original vectors
    
    • Still, they will lie on same axes

(Don’t know how important it is to remember this)

Question 11

What are the key characteristics of Linear Discriminant Analysis(LDA)?

Answer 11

Linear Discriminant analysis

• Works as a pre-processing step
• Is a supervised technique

Question 12

What are the different types of LDA?

Answer 12

Types to deal with classes: Class-dependent and class-independent

Class-dependent LDA: One separate lower dimensional space is calculated for each class to project its data on it

Class-independent LDA: Each class will be considered as a separate class against other classes.
- There is just one lower dimensional space for all classes to project their data on it

Question 13

What are the steps of calculating LDA?

Answer 13

Goal: Project original data matrix onto a lower dimensional space.

Step 1: Between-class variance/matrix:
    - Calculate separability between different classes (i.e. the distance between the means of different classes).

Step 2: Within-class variance/matrix
    - Calculate distance between the mean and the samples of each class

Step 3: Construct lower dimensional space
    - By maximizing between-class variance and minimizing within-class variance

Question 14

What are the issues with LDA?

Answer 14

Issues:
Small Sample Problem (SSP): Fails to find lower dimensional space
- If dimensions > number of samples
- Here within-class matrix becomes singular

Linearity problem: Cannot discriminate between classes
- If different classes are non-linearly separable

Question 15

What are the differences in how LDA works vs PCA?

Answer 15

PCA detects the directions of maximal variance

LDA finds subspace that maximizes class separability

Question 16

What is Singular Value Decomposition (SVD)?

Answer 16

SVD is a method for transforming correlated variables into a set of uncorrelated variables
- To better expose various relationships among original data items

SVD is a method for identifying and ordering dimensions along which data points exhibit most variations

SVD can also be seen as a method for data reduction

Question 17

What are the basic steps of SVD?

Answer 17

Consider a high dimensional, highly variable set of data points

Reduce it to a lower dimensional space that exposes substructure of original data (more clearly)

Orders it from most variation to least

Question 18

True or False: SVD is fast even when number of features grows

Answer 18

False. SVD approach can get very slow when number of features grows

Question 19

True or False: SVD is fast even when number of samples grows

Answer 19

True. SVD can handle large training sets efficiently, provided they can fit in memory

Question 20

True or False: Training Linear regression model for large number of features is faster using Gradient Descent than using SVD

Answer 20

True

Question 21

What is variability in data?

Answer 21

Variability (or dispersion) is the extent to which a distribution is stretched or squeezed. Has multiple components: variance, standard deviation, interquartile range, and (I’ve also seen) range

Question 22

How does a PCA plot get plotted?

Answer 22

A PCA plot converts the correlations (or lack thereof) among all of the features into a 2D graph (or more dimensions, it depends)

Observations that are highly correlated cluster together

Question 23

How does PCA plot lines?

Answer 23

PCA finds the best fitting line by maximizing the sum of the squared distances from the projected points to the origin.

Question 24

How many PC should you use?

Answer 24

In a general n(observations) x p(variables) data matrix X, there are up to min(n-1, p) PCs

But there is no fixed method you should use

I think you should use the number of PCs that you consider adequate to contain the most variability of data

Question 25

How do you define PC 1 and PC 2

Answer 25

PC 1 the linear combination of features that has the highest variance

PC 2 is the linear combination of data sets that has the second-highest variance (it is not correlated to PC 2) and it is orthogonal (perpendicular on PC 1)