Chapter 2: End-to-End Machine Learning Project

Question 1

In the machine learning project checklist, what is the first step? (Hint: Look at…)

Answer 1

…the big picture.

Question 2

In the machine learning project checklist, what is the second step? (Hint: Get…)

Answer 2

.. the data.

Question 3

In the machine learning project checklist, what is the third step? (Hint: Explore and visualize…)

Answer 3

…the data to gain insights.

Question 4

In the machine learning project checklist, what is the fourth step? (Hint: Prepare the…)

Answer 4

…data for machine learning algorithms.

Question 5

In the machine learning project checklist, what is the fifth step? (Hint: Select a…)

Answer 5

…model and train it.

Question 6

In the machine learning project checklist, what is the sixth step? (Hint: Fine…)

Answer 6

… tune your model.

Question 7

In the machine learning project checklist, what is the seventh step? (Hint: Present…)

Answer 7

… your solution.

Question 8

In the machine learning project checklist, what is the eighth step? (Hint: Launch…)

Answer 8

… monitor and maintain your system.

Question 9

What are the considerations to take when deciding between batch or online learning?

Answer 9

• Is there a continuous flow of data coming in to the system?
  No = Batch / Yes = Online.
• Will the model need to adjust to data changing rapidly?
  No = Batch / Yes = Online.
• Is the data small enough to fit in to memory?
  No = Batch / Yes = Online.

Question 10

What is a suitable performance metric for a linear regression?

Answer 10

Root Mean Squared error, this gives an idea of how much error the system typically makes in its predictions, with a higher weight given to larger errors due to the error being squared.

Question 11

What are the advantages and disadvantages of mean absolute error (MAE)?

Answer 11

Advantages:
- Robust to outliers.
- Same units as the output variable.

Disadvantages:
- Graph of MAE is not differentiable, using it as a loss function requires optimizers like Gradient Descent.

Question 12

What are the advantages and disadvantages of root mean squared error (RMSE)?

Answer 12

Advantages:
- Same units as the output variable.

Disadvantages:
- Not robust to outliers as is bias to larger error values.

Question 13

When is RMSE preferred to MAE?

Answer 13

When errors follow a Gaussian distribution and outliers are exponentially rare.

Question 14

What is an example of checking the assumptions on a machine learning project?

Answer 14

An assumption on a machine learning project could be that the output of the model is going to be used as a numerical value i.e. when predicting price it will be a $ value rather than a category e.g. high price/medium price/low price.

Checking this assumption is very important as it guides the solution architecture i.e. $ value would require a regression model, category a classification model.

Question 15

When you start to explore the data in a machine learning project, what characteristics of each attribute should you look at?

Answer 15

1. Name.
2. Data Type (categorical, int/float, bounded/unbounded, structured/unstructured).
3. % null values.
4. Noisiness/type of noise. (Stochastic, outliers/round errors).
5. Usefullness for the task.
6. Distribution (Gaussian/Uniform/Logarithmic).

Question 16

Before performing any Exploratory Data Analysis for a machine learning task, what should you do?

Answer 16

Split the data in to full train/test/dry run (84%/15%/1%). Splitting the data and hiding the test set prevents data snooping. Creating a dry run set is used for testing code pipelines.

Question 17

When creating a train/test split, why is using a hash of an identifier preferable over a random split?

Answer 17

When using a random split, each time the script is run the data gets randomly shuffled again and eventually all of the data will be seen by the data scientist and ML algorithm, potentially leading to data snooping.

Using a hash of a unique ID will create a stable dataset split.

Question 18

When exploring the data through visualisations, how should you treat geographical data such as longitude and latitude?

Answer 18

• Visualise it using a scatterplot or a mapping library such as plotly.
• Overlay different attributes, using colour or density, from the dataset and inspect for any patterns that appear.

Question 19

What is useful to compute between pairs of continuous attributes?

Answer 19

The correlation matrix or the scatter matrix.

Question 20

Why should we be cautious when looking at the correlation between continuous features?

Answer 20

• It only measures linear correlations and would completely miss non-linear relationships.
• A perfect correlation would be given to the same variable but in different units.

Question 21

How can experimenting with attribute combinations be helpful when exploring the data? Can you give an example?

Answer 21

Some attributes in a dataset may provide more information value when combined. An example of this is when including tenure and promotions in a risk model for attrition, dividing the number of promotions by tenure gives an indication of promotion per year of tenure which can be more informative than the raw number of promotion’s.

Question 22

When cleaning the data when preparing it for machine learning, what strategies can be used for dealing with missing data?

Answer 22

• Remove the rows with missing data, if the number of rows affected is low.
• Remove the attribute with missing data, if the number of rows affected is high.
• Impute the missing values using an appropriate statistic (mean/median) or zero.
• Impute the missing value using another machine learning model, with the missing values as the target. Examples of these are KNNImputer, which uses the k-nearest neigbors algorithm and the IterativeImputer, which trains a regression model per feature.

Question 23

When dealing with missing values, what is it important to consider?

Answer 23

It is important to consider if the values missing are random or systematic.

For example, in an unbalanced classification task, there may be 1% of records missing for an attribute, but those records are all for instances in the target class. Therefore dropping the rows would not be suitable.

Question 24

How can you process categorical variables in to numbers for machine learning?

Answer 24

• Ordinal encoding: This replaces each category with a number between 0 and the number of categories in the attribute. This works best for categories with a logical order i.e. rating scale of “bad”, “average”, “good”, and “excellent”.
• One hot encoding: This creates a new binary attribute for each category e.g. one attribute equal to 1 when the attribute is a certain value and zero otherwise.

Question 25

When dealing with categorical attributes which contain a large number of categories, why is using one hot encoding not optimal? What are alternative methods?

Answer 25

One hot encoding creates k-1 new features for k categories, if k is large this will result in a large number of new features, leading to slow training and degraded performance.

Alternative method include:
- Converting the categories to a useful numerical feature related to the category.
- Create a learnable, low-dimensional vector called an embedding.

Question 26

When preparing data for machine learning, why do we scale numerical features?

Answer 26

Machine learning algorithms don’t perform well when the input numerical attributes have very different scales, most models will bias towards larger scale features. Feature scaling transforms all the features to the same scale.

Question 26

What are two common feature scaling methods?

Answer 26

• MinMax scaling (normalisation): Values are shifted and rescaled so that they end up ranging from 0 to 1 by subtracting the min value and dividing by max - min.
• Standardization: The mean of the feature is subtracted from each value and the result is divided by the standard deviation.

Question 27

What is a drawback of using MinMax scaling?

Answer 27

MinMax scaling is affected by outliers. For example, if a feature had values 0 - 15 and value had been inputted by mistake to 100, MinMax would scaled the 100 down to 1 and crush all of the other values down to 0 - 0.15.

Question 28

What does a feature with a heavy tail refer to?

Answer 28

When values far from the mean are not exponentially rare.

Question 29

What will happen if you scale a distribution with a heavy tail? Are there any steps you should take before scaling a heavy tailed distribution?

Answer 29

MinMax and Standardization will squash most values in the distribution in to a very small range, which is not optimal for ML models.

Before scaling the feature, you should transform it to shrink the heavy tail and ideally make the distribution symmetrical.

Question 30

What are common transformations to shrink a heavy tailed distribution?

Answer 30

• Square root transformation (positive features, tail to the right).
• Logarithmic transformation (long heavy tails, power law distribution).
• Bucketizing the feature and replacing the feature value with the index of the bucket it belongs to i.e. replace each value with its percentile.

Question 31

What transformation can be applied to multimodal distributions?

Answer 31

Bucketize the feature and use the bucket IDs as a category and one hot encode.

This approach allows regression models to more easily learn different rules for different ranges of feature value.

Question 32

What does RBF refer to?

Answer 32

Radial Basis Function, which is a similarity measure that depends on the distance between an input value and fixed point.

The most common is a Gaussian RBF, which decays exponentially as the input value moves away from a fixed point.

Question 33

How can an RBF be used in feature engineering?

Answer 33

If you have multimodal data, creating an RBF with the fixed point being the value of one of those modes, if the fixed point is correlated with the target value, the new feature will help.

Question 34

Should transformation and scaling be applied to the target variable?

Answer 34

Yes, when suitable e.g. a heavy tailed target variable can be log transformed. However, the regression would be predicting log of the target variable and the output will have to be inversely transformed back to its business related output.

Question 35

What does sklearn.utils.validation contain?

Answer 35

The sklearn.utils.validation package contains several functions we can use to validate data inputs.

Question 36

When using a model performance measure such as RMSE, how should you put it in to business context?

Answer 36

Think about the error term in the context of the range of the value that is trying to be predicted. Also consider what it would mean in a practical sense if your model made this magnitude of error in a prediction.

For example, when trying to predict a house price, when the house prices range from £120k - £265k and a model has a RMSE of £65k, this would not be acceptable.

Question 37

When selecting and training a model, what is a short checking to follow?

Answer 37

• Select a model, starting with the simplest (e.g. linear regression).
• Train on the dataset and evaluate using k-fold cross validation.
• Check the evaluation score, inspecting the mean/std yielded by the cross validation.
• Judge the score and repeat previous steps with different varieties of models.
• Shortlist two - five of the best performing models for further fine tuning.

Question 38

What are the two different hyperparameters can be explored in hyperparameter tuning?

Answer 38

Model hyper-parameters: These are external configuration variables of the model and can be thought of like “settings” on the model. Examples of model hyper-parameters are the number of branches in a decision tree or the regularisation term in a regression.

Feature engineering as a hyper-parameter: Preprocessing features can be treated as a hyper-parameter, for example discretising vs not-discretising a continuous variable or using two different imputation methods can be compared.

Question 39

What are two methods of automated hyperparameter tuning?

Answer 39

Grid search: This method takes a dictionary of hyper-paramters and a range of values, evaluates the model performance using all combinations of these values and returns the set of the optimal hyper-parameter values.

Random search: This method selects a random value for each hyperparameter at each iteration. These values are chosen from a list of possible values or a probability distribution.

Question 40

When would one use a random search over a grid search?

Answer 40

When the hyperparameter search space is very large.

Question 41

How does Random Search CV work?

Answer 41

For a given number of iterations, for example 100, RandomSearchCV will explore 100 random values for each of these hyper-parameters.

Question 42

What are the benefits of Random Search vs Grid Search?

Answer 42

• Random Search time is determined by the number of iterations specified by the data scientist, where as GridSearch is determined by the number of combinations of variables.

Question 43

What are Ensemble Methods?

Answer 43

Ensemble methods are a method of fine tuning where the best performing models are combined to produce better performance than the best individual model. This is optimised when models in the ensemble make different types of errors.

Question 44

When analysing the best models, what are checks that should be performed?

Answer 44

Feature importance:
- Check the feature importance scores.
- Drop less useful features.
- Perform sanity checks on the features to see if they make sense and are not inadvertently causing target leakage.

Prediction Errors:
- Check where the system is making errors.
- Try to understand why it makes them.
- Try to fix the problem (adding extra features, cleaning up outliers etc).

Systematic performance concerns:
- Check to see if there are any subsets of categories in the dataset the model systematically performs badly on.
- If it does, it is not ready for deployment until this is resolved or predictions for this category should be not be used.

Question 45

After fine tuning the model to a point where it performs sufficiently well on the validation set, what is the next step?

Answer 45

Evaluate the model on the completely unseen test set.

Question 46

When evaluating the performance on the test set, what more information would you want in addition to a point estimate of the generalisation error?

Answer 46

An idea of the precision of the point estimate of the generalisation error, which can be obtained by computing a 95% confidence interval using stats.t.interval.