Systems Design

Question 1

What is ML systems design?

Answer 1

It takes a systems approach to MLOps, which means that the ML system is considered holistically to ensure that all the components (business requirements, the data stack, infrastructure, deployment, monitoring) and their stakeholders can work together to satisfy the specified objectives and requirements

Question 2

What are the four general requirements for ML systems?

Answer 2

• Reliability
• Scalability
• Maintainability
• Adaptability

Question 3

Why is reliability a requirement for ML systems?

Answer 3

The system should continue to perform the correct function at the desired level of performance even in the face of adversity (hardware or software faults, human error). Traditional software systems yield an error, but ML systems can fail silently

Question 4

Why is scalability a requirement for ML systems?

Answer 4

ML systems can grow in multiple ways: complexity (more parameters), traffic volume (more predictions per given time), model count (more use cases). These are examples of resource scaling, but handling growth also includes artifact management

Question 5

Why is maintainability a requirement for ML systems?

Answer 5

Structuring workloads and set up infrastructure such that all contributors can work using tools they want is important. Code should be documented. Code, data, and artifacts should be versioned. Models should be sufficiently reproducible.

Question 6

Why is adaptability a requirement for ML systems?

Answer 6

To adapt to shifting data distributions and business requirements, the system should have some capacity for both discovering aspects for performance improvement and allowing updates without service interruption

Question 7

What are some examples of nonprobability sampling?

Answer 7

• Convenience sampling
• Snowball sampling
• Judgment sampling
• Quota sampling

Question 8

What is convenience sampling?

Answer 8

A nonprobability sampling method where samples of data are selected based on their availability. This sampling method is popular because it’s convenient

Question 9

What is snowball sampling?

Answer 9

A nonprobability sampling method where future samples are selected based on existing samples. For example, to scrape legitimate Twitter accounts, you start with a small number of accounts, then you scrape all the accounts they follow, and so on

Question 10

What is judgment sampling?

Answer 10

A nonprobability sampling method where the experts decide what samples to include

Question 11

What is quota sampling?

Answer 11

A nonprobability sampling method where samples are selected based on quotas for certain slices of data without any randomization. For example, when the same number of samples are selected per age group for a survey, regardless of the actual age distribution

Question 12

What are some examples of random sampling?

Answer 12

• Simple random sampling
• Stratified sampling
• Weighted sampling
• Reservoir sampling
• Importance sampling

Question 13

What is simple random sampling?

Answer 13

In this form of random sampling, all samples in the population are given equal probabilities of being selected

Question 14

What is a drawback of simple random sampling?

Answer 14

Rare categories of data might not appear in your selection

Question 15

What is stratified sampling?

Answer 15

A random sampling method where the population is divided into groups that are relevant and sample from each group separately. For example, to sample 1% of data that has classes A and B, 1% can be sampled from each class separately. This way, both classes will be included in the selected, no matter how rare class A or B is

Question 16

What is a drawback of stratified sampling?

Answer 16

It is not always possible to divide all samples into groups. For instance, when a sample belongs to multiple groups, as in the case of multilabel tasks

Question 17

What is weighted sampling?

Answer 17

A random sampling method where each sample is given a weight, which determines the probability of it being selected

Question 18

What is the advantage of using weighted sampling?

Answer 18

This method leverages domain expertise. For example, if a certain subpopulation of data, such as more recent data, is more valuable to the model and needs to have a higher chance of being selected, a higher weight can be given to this subpopulation. If the data comes from a different distribution compared to the true data, samples that are underrepresented in the data can be given a higher weight

Question 19

What is reservoir sampling?

Answer 19

A random sampling method that is especially useful when dealing with streaming data. Reservoir sampling ensures that every sample has an equal probability of being selected and when the selection algorithm is stopped at any time, the samples are selected with the correct probability

Question 20

How does the reservoir sampling algorithm work?

Answer 20

The algorithm involves a reservoir, which can be an array and consists of three steps:

1. Put the first k elements into the reservoir
2. For each incoming nth element, generate a random number i such that 1 ≤ i ≤ n
3. If 1 ≤ i ≤ k: replace the ith element in the reservoir with the nth element. Else, do nothing

Question 21

What is importance sampling?

Answer 21

A random sampling method that allows sampling from a distribution when there is only access to another distribution. For example, if distribution P(x) is expensive or infeasible to sample from and Q(x) is easy to sample from, a sample x is selected from Q(x) and then weighed by P(x)/Q(x). Q(x) is called the proposal distribution or the importance distribution. Q(x) can be any distribution as long as Q(x) > 0 whenever P(x) ≠ 0

Question 22

What is label ambiguity or label multiplicity?

Answer 22

When there are multiple conflicting labels for a data instance

Question 23

What is an example of label multiplicity?

Answer 23

When a company uses multiple sources and relies on multiple annotators who have different levels of expertise

Question 24

What is weak supervision?

Answer 24

A labeling technique where labeling functions (LFs) are used to label samples

Question 25

What are labeling functions in weak supervision?

Answer 25

A function that encodes heuristics. They can be developed with subject matter expertise. Examples of LFs include keyword heuristic (if this is mentioned then do that), regular expression, database lookup, outputs of other models

Question 26

What is semi-supervision?

Answer 26

A labeling technique that leverages structural assumptions to generate new labels based on a small set of initial labels

Question 27

What are examples of semi-supervision?

Answer 27

1. self-training: training a model on an existing set of labeled data and using this model to make predictions for unlabelled samples. Assuming that predictions with high raw probability scores are correct, the labels predicted with high probability are added to the training set and train a new model on this expanded training set
2. Assuming that data samples that share similar characteristics share the same labels (e.g. unlabelled hashtags in a tweet with a labeled hashtag)
3. Perturbation (e.g. small adjustments to labeled images to create new training instances)

Question 28

What is transfer learning?

Answer 28

Refers to the family of methods where a model developed for a task is reused as the starting point for a model on a second task

Question 29

What is active learning?

Answer 29

A method for improving the efficiency of data labels. Rather than randomly labeling data samples, the samples that are most helpful to the model according to some metrics or heuristics are labeled. For example, only labeling the examples that the model is the least certain about (e.g. choosing the samples with the lowest probability for the predicted class)

Question 30

What are some ways to handle class imbalance?

Answer 30

• Using the right evaluation metrics (accuracy, precision, recall, f1 score)
• Resampling (undersampling, oversampling)
• Cost-sensitive learning
• Class-balanced cost
• Focal loss

Question 31

What is cost-sensitive learning?

Answer 31

A method to handle class imbalance in which the loss function is modified to take into account different costs depending on the class. The problem with this loss function is that the cost matrix must be manually defined, which is different for different tasks

Question 32

What is class-balanced loss?

Answer 32

A method to handle class imbalance by punishing the model for making wrong predictions on minority classes

Question 33

What is focal loss?

Answer 33

A method to handle class imbalance where the loss is adjusted so that if a sample has a lower probability of being right, it will have a higher weight