Educative Machine Learning System Design - ML Primer

Question 1

What should you expect in a machine learning interview?

Answer 1

Most major companies expect Machine Learning engineers to have solid engineering foundations and hands-on Machine Learning experiences. The candidates go through a similar method of problem solving (Leetcode style), system design, knowledge of machine learning and machine learning system design.

Question 2

What’s the standard development cycle of machine learning?

Answer 2

It includes data collection, problem formulation, model creation, implementation of models, and enhancement of models.

Question 3

What are the 6 basic steps to approach Machine Learning System Design?

Answer 3

Question 4

What do you do during the “problem statement” step to approach Machine Learning System Design?

Answer 4

Asking questions is crucial to filling in any gaps and agreeing on goals.

It’s important to state the correct problems. It is the candidates job to understand the intention of the design and why it is being optimized. It’s important to make the right assumptions and discuss them explicitly with interviewers.

If we are clear on the problem statement of designing a Feed Ranking system, we can then start talking about relevant metrics like user agreements.

Question 5

What do you do during the “identify metrics” step to approach Machine Learning System Design?

Answer 5

During the development phase, we need to quickly test model performance using offline metrics. You can start with the popular metrics like logloss and AUC for binary classification, or RMSE and MAPE for forecast.

Question 6

What do you do during the “identify requirements” step to approach Machine Learning System Design? Which two requirements do you need to identify?

Answer 6

• Training requirements
• Inference requirements

Training Requirements

There are many components required to train a model from end to end. These components include the data collection, feature engineering, feature selection, and loss function. For example, if we want to design a YouTube video recommendations model, it’s natural that the user doesn’t watch a lot of recommended videos. Because of this, we have a lot of negative examples. The question is asked:

How do we train models to handle an imbalance class?

Once we deploy models in production, we will have feedback in real time.

How do we monitor and make sure models don’t go stale?

Inference requirements

Once models are deployed, we want to run inference with low latency (<100ms) and scale our system to serve millions of users.

How do we design inference components to provide high availability and low latency?

Question 7

What do you do during the “identify requirements” step to approach Machine Learning System Design? Which two requirements do you need to identify?

Answer 7

• Training requirements
• Inference requirements

Training Requirements

There are many components required to train a model from end to end. These components include the data collection, feature engineering, feature selection, and loss function. For example, if we want to design a YouTube video recommendations model, it’s natural that the user doesn’t watch a lot of recommended videos. Because of this, we have a lot of negative examples. The question is asked:

How do we train models to handle an imbalance class?

Once we deploy models in production, we will have feedback in real time.

How do we monitor and make sure models don’t go stale?

Inference requirements

Once models are deployed, we want to run inference with low latency (<100ms) and scale our system to serve millions of users.

How do we design inference components to provide high availability and low latency?

Question 8

What do you do during the “train and evaluate model” step to approach Machine Learning System Design?

Answer 8

3 components:

• feature engineering
• feature selection
• models

For example, in Rental Search Ranking, we will discuss if we should use ListingID as embedding features. In Estimate Food Delivery Time, we will discuss how to handle the latitude and longitude features efficiently.

Question 9

What do you do during the “train and evaluate model” step to approach Machine Learning System Design?

Answer 9

3 components:

• feature engineering
• feature selection
• models

For example, in Rental Search Ranking, we will discuss if we should use ListingID as embedding features. In Estimate Food Delivery Time, we will discuss how to handle the latitude and longitude features efficiently.

Question 10

What do you do during the “design high level system” step to approach Machine Learning System Design?

Answer 10

Goal: identify a minimal, viable design to demonstrate a working system.

In this stage, we need to think about the system components and how data flows through each of them. We need to explain why we decided to have these components and what their roles are.

• For example, when designing Video Recommendation systems, we would need two separate components: the Video Candidate Generation Service and the Ranking Model Service.

Question 11

What do you do during the “scale the design” step to approach Machine Learning System Design?

Answer 11

In this stage, it’s crucial to understand system bottlenecks and how to address these bottlenecks.

You can start by identifying:

• Which components are likely to be overloaded?
• How can we scale the overloaded components?
• Is the system good enough to serve millions of users?
• How we would handle some components becoming unavailable, etc.

Question 12

Where in the 6 step process to approach machine learning system design is the “problem statement” step located?

Answer 12

The “problem statement” step is the 1st step of the process.

It is to be done before the “identify metrics” step.

Question 13

Where in the 6-step process to approach machine learning system design is the “identify metrics” step located?

Answer 13

It is the 2nd step of the 6 step process of the approach to machine learning system design.

It is to be done after the “problem statement” step and before the “identify requirements”.

Question 14

Where in the 6-step process to approach machine learning system design is the “identify requirements” step located?

Answer 14

It is the 3rd step of the 6-step process to approach machine learning system design interview.

It is to be done after the “identify metrics” step and before the “train and evaluate models” step.

Question 15

Where in the 6-step process to approach machine learning system design is the “train and evaluate models” step located?

Answer 15

It is the 4th step of the process.

It is to be done after the “identify requirements” step and before the “design high level requirements” step.

Question 16

Where in the 6-step process to approach machine learning system design is the “design high level system” step located?

Answer 16

It is the 5th step in the process.

It is to be done after the “train and evaluate models” step and before the “scale the design” step.

Question 17

Where in the 6-step process to approach machine learning system design is the “scale the design” step located?

Answer 17

It is the last and final step of the process.

It is to be done after the “design high level system” step.

Question 18

What is one hot encoding?

Answer 18

It converts categorical variables into a one-hot numeric array.

One hot encoding is a very common technique in feature engineering. One hot encoding is very popular when you have to deal with categorical features that have medium cardinality.

Question 19

What are common problems with one-hot encoding?

Answer 19

• Expansive computation and high memory consumption are major problems with one hot encoding. High numbers of values will create high-dimensional feature vectors. For example, if there are one million unique values in a column, it will produce feature vectors that have a dimensionality of one million.
• One hot encoding is not suitable for Natural Language Processing tasks. Microsoft Word’s dictionary is usually large, and we can’t use one hot encoding to represent each word as the vector is too big to store in memory.

Question 20

What are some best practices for one-hot encoding?

Answer 20

• Depending on the application, some levels/categories that are not important, can be grouped together in the “Other” class.
• Make sure that the pipeline can handle unseen data in the test set.

Question 21

At which step of the machine learning system design do you discuss feature selection and feature engineering?

Answer 21

You discuss it during the “identify requirements” step as well as the “train and evaluate models” step.

Question 22

What is feature hashing?

Answer 22

Feature hashing, called the hashing trick, converts text data or categorical attributes with high cardinalities into a feature vector of arbitrary dimensionality.

Question 23

What are the benefits of feature hashing?

Answer 23

Feature hashing is very useful for features that have high cardinality with hundreds and thousands of unique values. Hashing trick is a way to reduce the increase in dimension and memory by allowing multiple values to be present/encoded as the same value.

Question 24

What are some problems associated with feature hashing?

Answer 24

• One problem with hashing is collision. If the hash size is too small, more collisions will happen and negatively affect model performance. On the other hand, the larger the hash size, the more it will consume memory.
• Collisions also affect model performance. With high collisions, a model won’t be able to differentiate coefficients between feature values. For example, the coefficient for “User login/ User logout” might end up the same, which makes no sense

Question 25

What is a crossed feature?

Answer 25

Crossed features, or conjunction, between two categorical variables of cardinality c1 and c2 is just another categorical variable of cardinality c1×c2. If c1 and c2 are large, the conjunction feature has high cardinality, and the use of the hashing trick is even more critical in this case. Crossed feature is usually used with a hashing trick to reduce high dimensions.

As an example, suppose we have Uber pick-up data with latitude and longitude stored in the database, and we want to predict demand at a certain location. If we just use the feature latitude for learning, the model might learn that a city block at a particular latitude is more likely to have a higher demand than others. This is similar for the feature longitude. However, a feature cross of longitude by latitude would represent a well-defined city block. Consequently, the model will learn more accurately

Question 26

What is embedding?

Answer 26

Feature embedding is an emerging technique that aims to transform features from the original space into a new space to support effective machine learning. The purpose of embedding is to capture semantic meaning of features; for example, similar features will be close to each other in the embedding vector space.

Question 27

How to generate/learn embedding vector?

Answer 27

For popular deep learning frameworks like TensorFlow, you need to define the dimension of embedding and network architecture. Once defined, the network can learn embedding automatically.

Question 28

For numeric features, what can be done?

Answer 28

Normalization and Standardization

For numeric features, normalization can be done to make the mean equal 0, and the values be in the range [-1, 1]. There are some cases where we want to normalize data between the range [0, 1].

If features distribution resembles a normal distribution, then we can apply a standardized transformation.

Question 29

What is are problems associated with normalization?

Answer 29

In practice, normalization can cause an issue as the values of min and max are usually outliers. One possible solution is “clipping”, where we choose a “reasonable” value for min and max.

Question 30

How do you handle imbalance class distribution?

Answer 30

• Use class weights in loss function
• Use naive resampling
• Use synthetic resampling

Question 31

Training Pipeline

What kind of format should you store data in to enable high throughput at low cost?

Answer 31

One common solution is to store data in a column-oriented format like Parquet or ORC. These data formats enable high throughput for ML and analytics use cases. In other use cases, the tfrecord data format is widely used in the TensorFlow ecosystem.

Question 32

Training Pipeline

Why do we need data partitioning?

Answer 32

Parquet and ORC files usually get partitioned by time for efficiency as we can avoid scanning through the whole dataset.

In this example, we partition data by year then by month. In practice, most common services on AWS, RedShift, and Athena support Parquet and ORC. In comparison to other formats like csv, Parquet can speed up the query times to be 30x faster, save 99% of the cost, and reduce the data that is scanned by 99%.

Partition training data in Parquet format

Question 33

Training Pipeline: Choose the right loss function

What are common metrics for forecast problems?

Answer 33

Most common metrics are the Mean Absolute Percentage Error (MAPE) and the Symmetric Absolute Percentage Error (SMAPE).

Question 34

Training Pipeline: Choose the right loss function

What’s the most popular loss function for binary classification?

Answer 34

Cross-entropy

Question 35

Training Pipeline: Retraining requirements

What’s a common common design pattern for retraining?

Answer 35

A common design pattern is to use a scheduler to retrain models on a regular basis, usually many times per day.

Question 36

Training Pipeline: Retraining requirements

What do you need to balance between when designing a retraining system?

Answer 36

Machine learning engineers need to make the training pipeline run fast and scale well with big data. When you design such a system, you need to balance between model complexity and training time.

Question 37

Inference

What’s a common pattern when designing an inference system?

Answer 37

During inference, one common pattern is to split workloads onto multiple inference servers. We use similar architecture in Load Balancers. It is also sometimes called an Aggregator Service.

Dispatcher diagram

Question 38

Inference

How does the aggregator service for inferencing work?

Answer 38

1. Clients (upstream process) send requests to the Aggregator Service. If the workload is too high, the Aggregator Service splits the workload and sends it to workers in the Worker pool. Aggregator Service can pick workers through one of the following ways:
   
   1. Work Load
   2. Round Robin
   3. Request Parameter
2. Wait for response from workers.
3. Forward response to client.

Question 39

What is the non-stationary problem in inferencing?

Answer 39

In an online setting, data is always changing. Therefore, the data distribution shift is common. So, keeping the models fresh is crucial to achieving sustained performance. Based on how frequently the model performance degrades, we can then decide how often models need to update/retrain. One common algorithm that can be used is the Bayesian Logistic Regression.

Question 40

What is Thompson Sampling?

Answer 40

In an Ad Click prediction use case, it’s beneficial to allow some exploration when recommending new ads. However, if there are too few ad conversions, it can reduce company revenue. This is a well-known exploration-exploitation trade-off. One common technique is Thompson Sampling where at a time, t, we need to decide which action to take based on the reward.

Question 41

What kind of metrics are used in offline environments?

Answer 41

During offline training and evaluating, we use metrics like logloss, MAE, and R2 to measure the goodness of fit. Once the model shows improvement, the next step would be to move to the staging/sandbox environment to test for a small percentage of real traffic.

Question 42

Online metrics

How do we evaluation models in online environments?

Answer 42

A/B Testing

This diagram shows one way to allocate traffic to different models in production. In reality, there will be few a dozen models, each getting real traffic to serve online requests. This is one way to verify whether or not a model actually generates lift in the production environment.

Allocate traffic for multiple models in production