Chapter 6- Feature Selection

Question 1

what is the aim of feature selection?

Answer 1

automatically identify meaningful smaller subsets of feature variables

Question 2

why do different types of models have different best feature sets?

Answer 2

different models draw different types of boundaries and allow different degrees of flexibility when you change their parameters

Question 3

for d features, how many possible feature sets are there?

Answer 3

2^d

Question 4

what is combinatorial optimisation?

Answer 4

finding the right point in a binary search space

Question 5

give the steps of wrapper method for feature selection

Answer 5

start with an initial guess for a good set of features

train and test a model (maybe cross val)

if your test error is deemed good enough, stop

otherwise, choose a new set of features and go to line 2

Question 6

name some wrapper methods

Answer 6

greedy search
genetic algorithm
simulated annealing
branch and bound

Question 7

what is forward selection?

Answer 7

add features greedily and sequentially. Find which of the remaining ones improves our model the most and add it permanently to our set

Question 8

what is backward elimination?

Answer 8

sequentially evaluate removing features and discard the one that damages performance the least.

Question 9

what is stepwise, or floating selection?

Answer 9

wrapper method that combines forward and backward selection

two steps forward and one step back

Question 10

what are filter methods?

Answer 10

find out how useful a feature is without training any models

Question 11

Describe the pearsons correlation coefficient equation in words

Answer 11

covariance of the two variables divided by the product of their standard deviations

Question 12

pearsons correlation coefficient, r = ?

Answer 12

sum (x-xmean)(y-ymean) / square root of(sum:(x-xmean^2) x sum:(y-ymean^2) )

Question 13

how do we rank features?

Answer 13

In order of the absolute value of the correlation coefficient

Question 14

what type of correlation does pearsons measure?

Answer 14

linear

Question 15

what is entropy?

Answer 15

the reduction in uncertainty

Question 16

what is information gain?

Answer 16

it quantifies the reduction in uncertainty (entropy) of adding a new feature

Question 17

information gain I(X;Y) = ?

Answer 17

H(Y) - H(Y|X)

Question 18

another name for information gain is?

Answer 18

mutual information

Question 19

when we are measuring the information gain of each feature, I(X,Y), we are really calculating? (hint: relation to Y)

Answer 19

the mutual information between the feature and the target label

Question 20

what is the major advantage of information gain?

Answer 20

detects nonlinearities

Question 21

what is the major disadvantage of information gain?

Answer 21

may choose redundant features, including very similar features that may be detrimental to the learning algorithm

Question 22

if i use a wrapper method, with N samples and d features and use a greedy search, how many models will I have to create

Answer 22

(d)(d+1) / 2

Question 23

why do we perform feature selection (3)

Answer 23

logistical - there may be too much data to process

interpretability - we may collect more data than is useful

overfitting - inclusion of too many features could mean we overfit

Question 24

pros (3) and cons (2) of forward and backward selection (wrapper method)

Answer 24

• Impact of a feature on the ML classifier is explicit
• Each subset of features, we know exactly how well the model performs.
• Better than exhaustive search.
• No guarantee of best solution
• Need to train and evaluate lots of models

Question 25

what two types of filter are there

Answer 25

univariate

multivariate

Question 26

what is a univariate filter

Answer 26

Evaluate each feature independently

Question 27

what is a multivariate filter

Answer 27

Evaluate features in context of others

Question 28

what filtering metric can we use when the data is divided into 2 classes, rather than a regression

Answer 28

fisher score

Question 29

give the equation for fisher score, F=

Answer 29

v1 + v2

Question 30

in words, the fisher score puts ….. over …

Answer 30

between class scatter (m1-m2)^2
over
within class scatter (v1 + v2)

Question 31

what is the main disadvantage of fisher score

Answer 31

only works on single features

Question 32

J(X) is the mutual information of X, what are the possibilities of what X can be?

Answer 32

a feature

the joint probability of two or more features

Question 33

what are embedded methods for feature selection

Answer 33

In Embedded Methods, the feature selection algorithm is integrated as part of the learning algorithm. Embedded methods combine the qualities of filter and wrapper methods.

Question 34

which wrapper method combines forward and backward selection in two steps forward and one step back

Answer 34

stepwise / floating selection