Educative Grokking the ML Interview

Question 1

How a candidate should approach machine learning system design questions?

Answer 1

1. Setting up the problem
2. Understanding the scale and latency requirements
3. Defining metrics
4. Architecture discussions
5. Offline model building & execution
6. Online model execution & evaluation
7. Iterative model improvement

Question 2

How do you close the gap between your understanding of the question and the interviewer’s expectations from your answer?

Answer 2

Ask questions

You will be able to narrow down your problem space, chalk out the requirements of the system, and finally arrive at a precise machine learning problem statement.

Question 3

What should you do after arriving at a precise machine learning problem statement?

Answer 3

Discuss about performance and capacity considerations of the system

Latency requirements and scale of the data

Question 4

Why is defining metrics important?

Answer 4

Metrics will help you to see if your system is performing well.

Question 5

What are some metrics used for offline testing?

Answer 5

You may have generic metrics; for example, if you are performing binary classification, you will use AUC, log loss, precision, recall, and F1-score. In other cases, you might have to come up with specific metrics for a certain problem. For instance, for the search ranking problem, you would use NDCG as a metric.

Question 6

When is offline metrics used?

Answer 6

You will use offline metrics to quickly test the models’ performance during the development phase.

Question 7

When is online metrics used?

Answer 7

Once you have selected the best performing models offline, you will use online metrics to test them in the production environment. The decision to deploy the newly created model depends on its performance in an online test.

Question 8

Which type of metrics do you may need to come up with online metrics?

Answer 8

While coming up with online metrics, you may need both component-wise and end-to-end metrics.

Consider that you are making a search ranking model to display relevant results for search queries. You may use a component-wise metric such as NDCG to measure the performance of your model online. However, you also need to look at how the system (search engine) is performing with your new model plugged in, for which you can use end-to-end metrics. A commonly used end-to-end metric for this scenario is the users’ engagement and retention rate.

Question 9

When should the architecture discussion come up?

Answer 9

The architecture discussion comes up after defining metrics, which comes after setting up the problem and understanding requirements.

Question 10

What do you need to think about when figuring out the architecture of the system?

Answer 10

You need to think about the components of the system and how the data will flow through those components.

Question 11

Which phase/step in the ML system setup helps in chalking out the architecture?

Answer 11

The requirements gathered during problem setup help you in chalking out the architecture.

Question 12

What are the different ways one can generate training data?

Answer 12

• Human labeled data
• Data collection through a user’s interaction with the pre-existing system

Question 13

What should our goal be when working on machine learning-based system?

Answer 13

As we work on a machine learning-based system, our goal is generally to improve our metrics (engagement rate, etc.) while ensuring that we meet the capacity and performance requirements.

Question 14

When does major performance and capacity discussions come up again after defining the requirements?

Answer 14

Major performance and capacity discussions come in during the following two phases of building a machine learning system:

1. Training time: How much training data and capacity is needed to build our predictor?
2. Evaluation time: What are the Service level agreement (SLA) that we have to meet while serving the model and capacity needs?

Question 15

What are the three different types of complexities that ML algorithms have?

Answer 15

• Training complexity
• Evaluation complexity
• Sample complexity

Question 16

What is the training complexity of a ML algorithm?

Answer 16

The training complexity of a machine learning algorithm is the time taken by it to train the model for a given task.

Question 17

What is the evaluation complexity of a ML algorithm?

Answer 17

The evaluation complexity of a machine learning algorithm is the time taken by it to evaluate the input at testing time.

Question 18

What is the sample complexity complexity of a ML algorithm?

Answer 18

The sample complexity of a machine learning algorithm is the total number of training samples required to learn a target function successfully.

Question 19

What the 3 components that ML systems consist of?

Answer 19

A machine-learning system consists of three main components. They are the training algorithm (e.g., neural network, decision trees, etc.), training data, and features. The training data is of paramount importance.

Question 20

What are the hypotheses that was fomulated for the A/B test?

Answer 20

Null hypotheses and alternative hypotheses

Question 21

What is the null hypothesis?

Answer 21

H0 is when the design change will not have an effect on variation. If we fail to reject the null hypothesis, we should not launch the new feature.

Question 22

What is the alternative hypothesis?

Answer 22

H1 is alternate to the null hypothesis whereby the design change will have an effect on the variation. If the null hypothesis is rejected, then we accept the alternative hypothesis and we should launch the new feature. Simply put, the variation will go in production.

Question 23

What is P-value?

Answer 23

P-value is used to help determine the statistical significance of the results. In interpreting the p-value of a significance test, a significance level (alpha) must be specified.

The significance level is a boundary for specifying a statistically significant finding when interpreting the p-value. A commonly used value for the significance level is 5% written as 0.05.

Question 24

What is a back test?

Answer 24

Backtesting is a term used in modeling to refer to testing a predictive model on historical data

Question 25

Why is transfer learning useful?

Answer 25

It is because the model doesn’t have to learn from scratch and can achieve higher accuracy in less time as compared to models that don’t use transfer learning.