9-featureselection

Question 1

What are the two main ways to do feature selection?

Answer 1

Wrapper methods and filtering

Question 2

What are wrapper methods?

Answer 2

Wrapper methods are a feature selection method focussed on choosing a subset of attributes that give the best performance on the development data

Question 3

What are the advantages of wrapper methods?

Answer 3

Build feature set with optimal performance on development data

Question 4

What are the disadvantages of wrapper methods?

Answer 4

They take a long time

Question 5

What are more practical wrapper methods?

Answer 5

Greedy wrapper method
Ablation wrapper method

Question 6

What is the greedy wrapper approach?

Answer 6

Train and evaluate model on each single attribute. Choose best attribute. Then train by combining best(s) attributes with each other attribute. Choose best combination. End when accuracy is not increased

Question 7

What are the disadvantages of greedy wrapper approach?

Answer 7

Still takes n^2/2 time, and converges usually to a suboptimal outcome

Question 8

What is the ablation approach?

Answer 8

Start with the entire feature set. Remove each attribute and assess on the remaining set. Stop when performance significantly degrades

Question 9

What are the advantages of ablation method?

Answer 9

Quickly remove irrelevant attributes

Question 9

What is pointwise mutual information?

Answer 9

PMI(A,C) = log2(P(A,C)/P(A)P(C)). We want to find values with high PMI

Question 9

What are the disadvantages of ablation method?

Answer 9

Still takes O(m^2). Assumes features are independent.

Question 10

What are feature filtering methods?

Answer 10

Methods that evaluate the goodness of each feature, by finding features that better predict the class

Question 10

What makes a single feature good?

Answer 10

Well correlated with class, reverse correlated with class, well correlated with not class

Question 11

What is mutual information?

Answer 11

The weighted average of all PMI.

P(a, c)PMI(a, c) + P( ̄a, c)PMI( ̄a, c)+
P(a, ̄c)PMI(a, ̄c) + P( ̄a, ̄c)PMI( ̄a, ̄c

Question 12

What are alternatives to mutual information?

Answer 12

Chi-square

Question 13

What is the principle of Chi-square?

Answer 13

Compare the observed value O(w) to the expected value E(w)

Question 14

How do we conduct feature selection on nominal attributes?

Answer 14

Either treat attributes as multiple binary attributes or modify mutual information definition

Question 15

How do we conduct feature selection on continuous attributes?

Answer 15

Estimate probability using Gaussian distribution

Question 16

How do we conduct feature selection on ordinal attributes?

Answer 16

Treat as binary, treat as continuous or treat as nominal

Question 17

What are the disadvantages of mutual information?

Answer 17

It is biased towards rare, uninformative features

Question 18

What is an unsupervised feature selection method used for text documents?

Answer 18

Term frequency, inverse document frequency (TF-IDF). This helps us find words that are relevant to a document in a given document collection.

Question 19

What are examples of models that do feature selection inherently?

Answer 19

Decision trees
Regression models with regularisation
Neural networks