Feature Selection & Dimensionality Reduction

Question 1

What are the main aims for applying Feature Selection and Dimensionality Reduction techniques?

Answer 1

• Reduce the impact caused by Curse of Dimensionality
• Remove redundant features to improve performance
• Increase computational efficiency
• Reduce cost in new data acquisition

Question 2

What factors should be considered when using Feature Selection or Dimensionality Reduction methods?

Answer 2

• The target dimension i.e. what you wish to reduce down to
• Interpretability (Yes - use Feature Selection, No - Use either method)
• Feature Correlations/dependency
• Feature reliability and repeatability
• Methods (different methods result in different features being used)

Question 3

What are the three popular Feature Selection methods?

Answer 3

Wrapper Method - Searches for optimal feature subset that maximises the decision-making performance.

Embedded Methods - Integrates Feature Selection into the Model Learning process

Filter-based Methods - Selection is based on feature relationships and statistics, rather than performance

Question 3

What are some examples of Wrapper Methods, in regards to Feature Selection?

Answer 3

Recursive Feature Elimination
Sequential Feature Selection

Question 4

What are some examples of Embedded Methods, in regards to Feature Selection?

Answer 4

Ridge (ElasticNet)
LASSO
Random Forest (feature ranking)

Question 5

What are some examples of Filter-based Methods, in regards to Feature Selection?

Answer 5

Univariate (ANOVA)
Chi Square
Correlation/Variance

Question 6

How does the Forward Feature Selection method work?

Answer 6

Starts with an empty set of features (X), and adds features one by one. The goal is to identify a subset of features that maximises the model’s performance based on a chosen evaluation metric, such as Accuracy, F1 Score, Mean Squared Error, etc…

Question 7

What is the step-by-step breakdown of Forward Feature Selection?

Answer 7

Start with an Empty Feature Set
Then, for each feature:
- If it increases the evaluation metric beyond the previously defined evaluation metric, add it to the list of features and update the evaluation value.
- If it doesn’t, then ignore it
After iterating through all the features, then return the newly defined subset of features as the features to be used

Question 8

What is the difference between Forward Feature Selection and Recursive Feature Elimination?

Answer 8

Forward Feature Selection starts with an empty set of features, and adds them one by one
Recursive Feature Elimination starts with a full set of features, and removes them one by one if they add no value.

Question 9

What is the LASSO method?

Answer 9

LASSO is an embedded method, and it is a regularisation technique used in regression analysis to enhance model performance. It introduces a penalty to the loss function to prevent overfitting and perform feature selection.

Question 10

How does LASSO work?

Answer 10

Lasso regression works by adding an L1 regularisation term to reduce the number of effective features in the feature space. This penalty encourages sparsity in the coefficient vector, causing some coefficients to shrink to 0, which removes certain features from the model.

Question 11

What does the regularisation parameter in Lasso control?

Answer 11

The regularisation parameter, also known as gamma, controls the degree of regularisation, balancing model accuracy and feature selection. It is the integral part of LASSO, which is used to identify key features whilst also building a model.

Question 12

What can happen if you set the regularisation parameter too high/at a higher value in LASSO?

Answer 12

A higher regularisation parameter will make some of the weights of X in the equation become 0, which will reduce the number of dimensions by removing features that are less relevant/important.

Question 13

What does a Chi-Square do?

Answer 13

A Chi-Square tests the independence of predictor and outcome events.
It is suitable for categorical features in categorical outcomes.

Question 14

What does a T-test do?

Answer 14

T-test compares the statistical difference of two groups (binary class), and is used for continuous features.
It checks how the mean of two groups are different from one another.

Question 15

What does ANOVA do?

Answer 15

ANOVA uses variance to test the relationship between categorical predictors and continuous outcome response e.g. gender, age group to predict exam mark

Question 16

What does a correlation test do?

Answer 16

A correlation test works for predictors and outcomes that are both continuous

Question 17

What is the null hypothesis for the Chi-Square test?

Answer 17

Null hypothesis - The two categorical data are independent

Question 18

What is the null hypothesis for the T-test?

Answer 18

The mean of the two groups are the same

Question 19

What is the null hypothesis for ANOVA?

Answer 19

The variance across categories are equal.

Question 20

What is the null hypothesis for Correlation?

Answer 20

The two groups of data are not correlated.

Question 21

How do you disprove a null hypothesis?

Answer 21

You calculate the P-value, and if it is below the threshold e.g. 0.01, then it can be used to reject the null hypothesis.

Question 22

What are the three popular Dimensionality Reduction Methods?

Answer 22

Principal Component Analysis (PCA)
Linear Discriminant Analysis (LDA)
Manifold Learning (non-linear)

Question 23

How does PCA work?

Answer 23

It works by transforming the original data into a set of orthogonal features (uncorrelated) called Principal Components, which capture the most significant information from the original features.

Question 24

What is the step-by-step process for PCA?

Answer 24

1. Standardise the data using the formula for standardisation (X_(standardised) = (X - mean)/standard deviation)
2. Find the covariance matrix of the dataset, which describes how the features vary with respect to each other.
3. Compute the Eigenvalues and Eigenvectors of the covariance matrix.
4. Sort the Eigenvectors in descending order, where the first in the order represents the first principal component
5. Select the top k amount of Principal Components
6. Project the data onto the new Principal Components

Question 25

Why should you use PCA in Machine Learning?

Answer 25

• It reduces the amount of dimensions in the dataset
• Noise Reduction, by keeping the largest eigenvalues for the principal components
• Feature Independence, whereby the new features that are produced by PCA are uncorrelated, which can be beneficial for algorithms.

Question 26

What are some limitations of PCA?

Answer 26

PCA assumes that the data is linearly correlated, so it may not perform well for data with complex, non-linear relationships

The new components produced by PCA may not be easy to interpret

PCA is sensitive to the scaling of data, so it’s crucial to standardise the data used when features have different scales.

Question 27

How does LDA (Linear Discriminant Analysis) work?

Answer 27

It aims to find a linear combination of features that best separates different classes in the data by focusing on maximising the separation between different classes

Question 28

What is the step-by-step process for performing LDA?

Answer 28

1. Compute the mean vectors for each class, and then compute the overall mean vector across all classes
2. Compute the Between-Class Scatter Matrix
3. Compute the Within-Class Scatter Matrix
4. Compute the Fisher Criterion, which measures the ratio of between-class variance to within-class variance.
5. Solve for the projection matrix

Question 29

What is the main difference between PCA and LDA?

Answer 29

PCA is supervised, whereas LDA is unsupervised.

Question 30

How does Manifold Learning work?

Answer 30

Manifold learning aims to learn the latent representation of the original data in lower dimensions.

Question 31

What is an example method of Manifold Learning?

Answer 31

PCA is (technically) a linear manifold learning method, but that doesn’t work for all cases.