The machine learning process

Question 1

What is feature engineering?

Answer 1

The process of crafting features from existing columns using domain knowledge and intuition.

Question 2

Give an example of where feature engineering might improve a machine learning model

Answer 2

E.g. prediction of life satisfaction using GDP. Intuitively, there are diminishing returns. The relationship is more likely to be logarithmic: i.e. a doubling of wealth leads to a set increase in happiness. So use log(GDP).

Question 3

Define polynomial regression.

Answer 3

The use of polynomial features: e.g. x^2, x^3 etc. Draws from Taylors theorem: any analytic function has a polynomial series expansion.

Question 4

What is linear truncation and how is it implemented?

Answer 4

Introduce a kink point where the trend in the data seems to change, fit a linear regression with two weights: one for the data before the kink and one after.

Question 5

How can you deal with multiple categorial variables?

Answer 5

One-hot encoding: a binary classifier for each category. I.e. IsDog, IsCat and IsMouse

Question 6

Explain feature selection

Answer 6

The process of selecting a good subset of features to use in a model.

Question 7

How might you manually select features?

Answer 7

If features are standardized, OLS coefficients might indicate how important each feature is.
We could look for features that are correlated with the outcome.

Question 8

Explain best subset selection.

Answer 8

Find the subset of features such that the generalisation error is minimised.

Question 9

What are the steps of the forward subset selection algorithm?

Answer 9

Start with a constant model, for each additional feature, only select the one which results in the greatest risk reduction, and store that model. Then there are a sequence of models to choose from, select the one with the lowest generalisation error.

Question 10

In machine learning it is common to split your data into three, what are each of these subsets called and what are they used for?

Answer 10

Training: this set is used to train the model by minimising the Risk given the labels.

Validation: this set is unseen in the training stage, but is used to tune hyperparameters (e.g. number of features used).

Test: this set is used to measure model metrics such as accuracy, loss etc.

Question 11

What is a hyperparameter?

Answer 11

A hyperparameter is a model parameter that is independent of the data. For example: learning rate, tree depth etc. It should be tuned for optimal generalisation performance.

Question 12

Explain the idea of k-cross validation

Answer 12

Useful for hyperparameter tuning when the data set is small. Divide data into partitions of similar size. Train the model on all data not in that partition, then evaluate the model on the partition, repeating for each partition. Report statistics for the metrics over the partitions.

Question 13

Why is it important to consider the sampling for splitting the data?

Answer 13

Training data should be representative of the population. Sampling might be necessary to avoid over/under representation among classes.

Question 14

What is stratified sampling?

Answer 14

Data is grouped into classes, and points are sampled from each class and recombined into training and test sets.

Question 15

How can this be implemented on multiple columns?

Answer 15

Make more columns with combinations of the features, then stratify on that.

Question 16

How can it be implemented on a continuous variable?

Answer 16

The data can be grouped by quantiles and stratified.

Question 17

What is the R-squared and for what type of model can it be used as an evaluation metric?

Answer 17

1 - (residual sum of squares over total sum of squares). This can be used on regression models.

Question 18

What are the disadvantages to using average log loss and accuracy score to evaluate classification models?

Answer 18

These errors do not illuminate how the model is misclassifying, which could be important information depending on the business problem.

Question 19

What does the confusion matrix show?

Answer 19

The total true negatives, false negatives, false positives and true positives

Question 20

Which parts of the confusion matrix might be more important in spam classification? What about cancer screening?

Answer 20

In spam classification, we care more about minimising FP than FN because missing a non-spam email is worse than seeing a spam in your inbox.

Cancer screening: we minimise FN over FP because being sent for further tests without actually having cancer much preferable to missing a sign of cancer.

Question 21

What are the true positive/true negative rates?

Answer 21

TPR = TP / (All positives) = TP / (TP + FN)
TNR = TN / (All negatives) = TN / (TN + FP)

Question 22

For classification prediction functions, what is the cutoff hyperparameter?

Answer 22

For binary classification, a prediction is a probability of membership to each class. The cut-off is the probability above which is assigned a positive result.

Question 23

What is an ROC curve?

Answer 23

This is a curve in TPR-TNR space, parameterised by the cut off, for a given classification model.

Question 24

How does the ROC change with scaling of scores/probabilities

Answer 24

The curve remains unchanged: its parameterisation is different but it occupies the same region of TPR-TNR space.

Question 25

What does the area under the ROC curve represent?

Answer 25

This measures the extent to which, with the ideal choice of cut-off, you can correctly separate classes.

Question 26

A model with an ROC AUC of 0.5 is better or worse than one with an AUC of 0.1?

Answer 26

An ROC of 1 means that there is a cut-off that perfectly separates the classes, but 0.5 is the AUC for a random classification, so really, 0.1 is better because you could flip classes for a better accuracy.

Question 27

What is a calibration curve?

Answer 27

It represents how well calibrated a model is to the actual distribution of classes

Question 28

What does a well-calibrated classification prediction function look like?

Answer 28

a line with gradient 1

Question 29

Explain the aim behind regularisation. Which trade-off does it hope to optimise?

Answer 29

Regularisation aims to reduce overfitting by simplifying the models and reducing the hypothesis space.

Question 30

What is a complexity measure?

Answer 30

A complexity measure is a positive valued functional which quantifies how complex the function is.

Question 31

What is Ivanov regularisation?

Answer 31

The minimisation of the Risk functional subject to the constraint that the complexity function is below some threshold.

Question 32

What is Tykhonov regularisation?

Answer 32

Minimisation of a modified risk functional containing a penalty term which is related to the complexity measure.

Question 33

What is the ridge regression objective function?

Answer 33

The usual regression risk functional, plus an L2-norm term

Question 34

What is the reason for this objective function?

Answer 34

By minimising this risk functional, the trade off between training set accuracy and model complexity is balanced, since a smaller L2-norm means the model is less sensitive to inputs.

Question 35

What must we do to the features before performing ridge regression?

Answer 35

We must standardize them.

Question 36

What is the difference between ridge and lasso regression?

Answer 36

Lasso is the same, but uses an L1-norm.

Question 37

What are three pros to using lasso regression over ridge regression?

Answer 37

Lasso gives sparse solutions: good for model interpretability and maintenance. Also quicker to compute.

Question 38

What is one pro to using ridge regression over lasso regression?

Answer 38

Ridge gives less sparse, meaning the model is less dependent on a few features.