ML System Design

Question 1

Elements of ML System Design

Answer 1

Clarify functional requirements (business objective)
Performance requirements
Frame as an ML Problem (inputs and outputs)
What data do we have access to for training
Feature engineering
Choose a model
Prediction Pipeline
Training pipeline
Offline & Online Metrics

Question 2

Questions to ask

Answer 2

• How much training data do we have access to?
• What is the state of the data (already in the form of features, event data, log data)?
• What’s most important? accuracy or response time?
• Hardware constraints?
• Time constraints?
• Model retraining (spam vs recommendation systems)

Question 3

confusion matrix

Answer 3

• summary of prediction results from a classification model
• predicted values on y axis
• actual values on x axis

Question 4

false positive rate

Answer 4

false positive / total negatives
FP / (FP + TN)

Question 5

Imbalanced dataset

Answer 5

A classification data set with skewed class proportions (far more positives than negatives or vice versa)

Question 6

Difference between precision and FPR

Answer 6

Precision measures the probability that a sample classified as positive is actually positive while the FPR measures the ratio of false positives to total negatives

Precision is a better metric for datasets with a large number of negative samples

Question 7

Training set

Answer 7

Examples used for learning to fit the parameters of the model

Question 8

Validation set

Answer 8

Set of examples used to tune model parameters. For example, the number of layers of a neural network or batch size

Question 9

Test set

Answer 9

Used to assess the performance of a fully trained model

Question 10

sensitivity

Answer 10

measures the model’s ability to predict true positives in each available category

Question 11

specificity

Answer 11

measures the model’s ability to predict true negatives for each available category

Question 12

System design: data

Answer 12

Identify target variables
implicit (putting an item in your shopping cart) vs explicit (buying an item)

Question 13

Example features

Answer 13

user-location
user-age
aggregate features like user-candidate total likes

Question 14

What to do about missing data and outliers

Answer 14

If the dataset is large enough, you can drop them
If you can’t afford to drop any data, you can impute feature values by replacing them with a default (typically the mean, median, or mode)

Question 15

sample bias

Answer 15

Happens when the collected data doesn’t accurately represent the environment the program is expected to run into.

e.g. training facial recognition in daytime lighting conditions only

Question 16

exclusion bias

Answer 16

Happens as a result of excluding some feature(s) from our dataset usually under the umbrella of cleaning our data.

Use feature importance tools. Don’t guess.

Question 17

Measurement bias

Answer 17

Systematic value distortion happens when there’s an issue with the device used to observe or measure.

Question 18

Prejudice bias

Answer 18

Happens as a result of cultural influences or stereotypes.

Images of nurses or wedding dresses

Question 19

Ranking model (recommendation systems)

Answer 19

Estimates the probability a video will be watched

Question 20

Feature Scaling Techniques

Answer 20

• Normalization (Min/Max Scaling)
• Standardization (Z-score normalization)
• Log scaling
• Discretization (Bucketing)
• Encoding categorical features (integer encoding, one-hot encoding, embedding learning)

Question 21

Normalization

Answer 21

• Min/max scaling
• Values are mapped from 0 to 1
• Normalization does not change the distribution of a feature

Question 22

Standardization (Z-Score Normalization)

Answer 22

• Changing the distribution of a feature to have 0 mean and a std dev of 1

Question 23

Log Scaling

Answer 23

• Mitigates skew

Question 24

Discretization

Answer 24

• Bucketing
• The process of converting a continuous feature into a categorical feature
• Age -> bucket (reduces a discrete feature to a number of categories)

Question 25

Techniques for Encoding Categorical Features

Answer 25

• Integer encoding
• One-hot encoding
• Embedding learning

Question 26

When is integer encoding a good choice?

Answer 26

When there is an ordinal relationship between the categories. Integer encoding works well for excellent = 5 stars, Good = 4 stars

Question 27

One hot encoding

Answer 27

A new binary feature is created for each unique value
If your feature has 3 colors, convert that to three booleans (isRed, isGreen, isBlue).

Question 28

Embedding Learning

Answer 28

• Map a categorical feature into an N-dimensional vector
• Useful when the number of unique values of a feature is very large
• Learning an N-dimensional vector for each unique value the categorical feature may take. The resulting vectors are smaller in size than the one-hot encoded versions would be.

Question 29

Harmonic mean

Answer 29

More heavily weighted to the smallest numbers which mitigates the impact of large outliers.