Classification

Question 1

What is a feature space

Answer 1

a coordinate space used to represent the input examples for a given problem, with one coordinate for each descriptive feature

Question 2

Eager Learning Classification Strategy

Answer 2

• classifier build a full model during an initial training phase, to use later when new query examples arrive
• more offline setup work, less work at run-time
• generalise before seeing the query example

Question 3

Lazy Learning Classification Strategy

Answer 3

• Classifier keeps all the training examples for later use.
• Little work is done offline, wait for new query examples.
• Focus on the local space around the examples

Question 4

What learning strategy does KNN Classifier use and how?

Answer 4

Lazy. K-NN identifies the k most similar previous examples from the training set for which a label has already been assigned, using some distance function

Question 5

How does Weighted kNN differ from the regular model

Answer 5

Weighted voting, closer neighbours get higher votes.

Question 6

Is there a “best” distance measure

Answer 6

No, the choice of distance measure is highly problem-dependent

Question 7

What is the difference between a local distance function (LDF) and a global distance function (GDF)

Answer 7

LDFs measure the distance between two examples based on a single feature, where as GDFs are based on the combination of the local distances across all features

Question 8

Define the overlap function (measuring distance)

Answer 8

Returns 0 if the two values for a feature are equal and 1 otherwise

Question 9

Define Hamming Distance (measuring distance)

Answer 9

GDF which is the sum of the overlap differences across all features

Question 10

Define Absolute Difference (measuring distance)

Answer 10

Absolute value of the difference between values for a feature or several features

Question 11

Define Absolute Difference for ordinal features (measuring distance)

Answer 11

calculate the absolute value of the difference between the two positions in the ordered list of possible values

Question 12

Define Euclidean Distance,and give the formula

Answer 12

• “Straight line” distance between two points in a feature space
• calculated as the square root of the sum of squared differences for each feature f, representing a pair of examples.

ED(p, q) = SQR(SUM_f (q_f - p_f)^2)

Question 13

What are Heterogeneous (Diverse) Distance Functions

Answer 13

GDF created from different local distance functions, using an appropriate function for each feature

Question 14

Min-max normalization formula

Answer 14

z_i = x_i - min(x) / max(x) - min(x)

Question 15

Standard Normalisation Formula

Answer 15

z_i = x_i - μ / σ

Question 16

Advantages of kNN

Answer 16

• little training time
• interpretability and explainability
• transparency

Question 17

Disadvantages of kNN

Answer 17

• need to carefully customise the distance function
• query time depends on the complexity of the function

Question 18

Decision Tree Algorithm

Answer 18

1. All training examples in root node
2. Examples are split by one feature into child nodes
3. The process repeats for each child node
4. Continue until all leaf nodes contain the same class

Question 19

Entropy formula

Answer 19

H(X) = - SUM_x p(x) log p(x)

Question 20

What does Information Gain measure and give the formula

Answer 20

measures the reduction in entropy when a feature is used to split a set into subsets

IG for feature A that splits a set of examples S into {S₁,…S_m}:
IG(S, A) = (original entropy) - (entropy after split)
IG(S, A) = H(S) - SUM(i=1, m) |S_i|/ |S| H(S_i)

Question 21

Steps of computing IG for each feature in a dataset

Answer 21

1. calculate overall dataset entropy
2. calculate entropy for each feature
3. calculate IG for each feature

Question 22

Why do ensembles work?

Answer 22

When the average probability of an individual being correct is > 50%, the chance of the ensemble of them reaching the correct decision increases as more members are added.
This holds true only if the diversity in ensemble continues to grow as well.

Question 23

What is the key idea of Bagging

Answer 23

Train classifiers on different subsets of the training data

Question 24

What is Bootstrap aggregation, what classifiers does it work better for

Answer 24

Bagging technique which randomly samples with replacement
Works better for “unstable” classifiers, e.g. DT, NN

Question 25

What is the key idea of Random Subspacing, and what does it encourage?

Answer 25

Train n base classifiers, each on a different subset of features.
Encourages diversity in the ensemble

Question 26

When would you choose weighted voting for your ensembles

Answer 26

When the individual classifiers do not give equal performance, we should give more influence to the better classifiers

Question 27

Discuss the Accuracy vs Diversity Trade-off in ensemble classification

Answer 27

An ideal ensemble is one that consists of highly accurate members which at the same time disagree, therefore we face a trade-off between diversity and accuracy when constructing an ensemble of classifiers

Question 28

What is the key idea in Boosting

Answer 28

Train a sequence of classifiers sequentially, so that later classifiers are trained to better predict class labels that earlier ones performed poorly on

Question 29

Give the basic approach to Boosting

Answer 29

1. Assign an equal weight to all training examples
2. Get a random sample from the training examples based on the weights.
3. Train a classifier on the sample
4. Increase weights for misclassified examples, decrease weight for correctly classified examples
5. Output final model based on all classifiers (e.g. majority voting model)

Question 30

Explain the bias-variance trade off

Answer 30

Bias is how close the classifier’s predictions are from the correct values and Variance is the error from sensitivity to small changes in the training set. There is often a tradeoff between minimising the two

Question 31

Discuss how ensemble generation methods affect the bias-variance tradeoff

Answer 31

• Bagging can often reduce the variance part of error
• Boosting can often reduce variance and bias, because it focuses on misclassified examples
• Boosting can sometimes increase error, as it is susceptible to noise, which can lead to overfitting

Question 32

Which classifiers generally suffer from overfitting and which classifiers generally suffer underfitting?

Answer 32

• low bias but high variance classifiers tend to overfit
• high bias but low variance classifiers tend to underfit