Dimension Reduction

Question 1

Reasons for dimension reduction

Answer 1

1. Computational Cost
2. Financial Cost
3. Interpretability

Question 2

Explain the Filter strategy for feature selection

Answer 2

Its a pre-processing step that ranks and filters features independently of the choice of classifier. It assigns a score to the different feature subsets using an evaluation function of choice

Question 3

What is a good strategy for selecting the top features from a Filter

Answer 3

evaluate classifier performance using feature subsets of increasing size

Question 4

Disadvantages of Filters

Answer 4

1. No model bias: doesn’t account for feature suitability across models
2. No feature dependencies

Question 5

Explain the Wrapper strategy for feature selection

Answer 5

The classifier is “wrapped” in the feature selection mechanism. Feature subsets are evaluated directly based on their performance when used with that specific classifier

Question 6

Advantages of Wrappers

Answer 6

1. Accounts for bias
2. considers features in context (feature dependencies)

Question 7

List the types of search used in feature subset search

Answer 7

• exhaustive
• exponential (heuristics)
• sequential (add/remove one f at a time)

Question 8

Describe the steps of Forward Sequential search

Answer 8

1. Start with an empty subset
2. find the most informative feature and add it to the subset
3. Repeat until there us no improvement by adding features

Question 9

Describe the steps of Backward Elimination

Answer 9

1. Start with the complete set of features
2. remove the least informative feature
3. repeat until there is no improvement by dropping features

Question 10

Compare Forward Sequential Search (FSS) to Backward Elimination (BE)

Answer 10

• FSS requires less running time if completed early
• BE tends to find better models, can find subsets with interacting features, but tends to be slower

Question 11

Disadvantages of Wrappers

Answer 11

• Computational cost
• risk of overfitting

Question 12

What is the general idea of projection methods

Answer 12

They are used in feature transformation to map the original d-dimensional space to a new (k-d)-dimensional space, with the minimum loss of information

Question 13

What is Principal Component Analysis (PCA)

Answer 13

an unsupervised projection method which aims to keep as much of the variance in the data as possible

Question 14

What are principal components in PCA

Answer 14

new dimensions constructed (from eigenvectors) as linear combinations of the original features, which are uncorrelated with one another.
The first PC accounts for the most variability in the data, and so on…

Question 15

Give context as to what eigenvectors and eigenvalues are

Answer 15

Given a matrix X, an eigenvector of the matrix is a non-zero vector v that satisfies the equation: Xv = λv, where λ is the eigenvalue.

Question 16

What is eigendecomposition

Answer 16

the factorisation of a matrix into its eigenvalues and eigenvectors

Question 17

Eigenvectors in the context of PCA

Answer 17

• orthogonal matrices which have direction
• each eigenvalue represents the variance in its direction

Question 18

What is a Covariance Matrix in PCA

Answer 18

Symmetric matrix which measures the tendency for features to vary in the same direction

Question 19

Give the steps of the PCA Algorithm (given dataset matrix X with n examples)

Answer 19

1. Calculate the mean of the columns of X
2. Subtract the column means from each row of X, to create the centred matrix Y
3. Calculate the Covariance matrix C
4. Calculate the eigenvectors of C
5. The PCs are given by the eigenvalues
6. Select PCs as a new reduced representation of X