Machine Learning Systems

Question 1

What are the three main components of a ML system that we need to decide on?

Answer 1

1. Training algorithm
2. Training data
3. Features

Question 2

What two methods are there to collect training data?

Answer 2

1. Online: Users’ preexisting interactions with the system, e.g., impressions for a search engine
2. Offline: Human annotators, which are expensive, must be QA’d, etc.

Question 3

What kind of human annotation is there?

Answer 3

1. MTurk/cheap
2. Specialized/hired annotators
3. Public datasets

Question 4

What are the two most common ML system SLA metrics?

Answer 4

1. Performance

2. Capacity

Question 5

What is the ML system performance metric?

Answer 5

Time to get back results, usually measured in ms for 99% of queries

Question 6

What is the ML system capacity metric?

Answer 6

Load the system can handle, usually measured in queries per second (QPS)

Question 7

What are the three types of ML system complexity? (Think what decisions make it slow.)

Answer 7

1. Training: time to train the model
2. Evaluation/Inference: time to evaluate input at testing/deployment time
3. Sampling: total number of samples required to learn the target

Question 8

What bad thing happens when you switch from a popularity-based recommendation model to a user-specific one?

Answer 8

Bias in recommendations leads to user-specific recommendations being shown lower in the list, leading to a self-reinforcing cycle of only ever showing popularity-based recommendations.

Question 9

What is a SERP?

Answer 9

Search engine results page

Question 10

What are the names for the two types of pages in an A/B test?

Answer 10

1. Control

2. Variation

Question 11

What are the five (+optional sixth) steps in designing an A/B test?

Answer 11

1. Come up with a hypothesis
2. Design an experiment to measure the hypothesis using a NHST
3. Use a power test to determine the number of samples you need to form a conclusion
4. Run the experiment
5. Check for a statistically significant result via NHST
6. (optional) Consider backtesting

Question 12

What is backtesting in the context of an A/B test?

Answer 12

After the A/B test is complete and the null hypothesis rejected, swapping the control and variations and confirming that a negative effect exists.

Question 13

What is an embedding?

Answer 13

A low-dimensional version of a high-dimensional feature (speech, image, text) such that it captures the important semantic information.

Question 14

How are embeddings used?

Answer 14

They allow us to train machine learning models on dense representations of data that captures the dimensions we care about.

Question 15

What are two ways of generating text embeddings?

Answer 15

1. word2vec

2. Context-based embeddings

Question 16

What are two models for word2vec text embeddings?

Answer 16

1. Continuous bag of words (CBOW)

2. Skipgrams

Question 17

How does continuous bag of words (CBOW) work?

Answer 17

It’s an LSTM model where, given a sequence of words of time $t_(-2)$, $t_(-1)$, $t_1$, $t_2$, all one-hot encoded, it predicts $t$.

Question 18

How do skipgrams work?

Answer 18

It’s an LSTM model where, given a one-hot encoded word at $t$, it predicts $t_(-2)$, $t_(-1)$, $t_1$, and $t_2$.

Question 19

Why do we need context-based text embeddings?

Answer 19

Word2vec strips out context-based information. For example, the word “apple” can refer to either Apple the company or an apple the fruit.

Question 20

What are two models for context-based text embeddings?

Answer 20

1. Embeddings from Language Models (ELMo)

2. Bidirectional Encoders Representations from Transformers (BERT)

Question 21

How does ELMo work?

Answer 21

Uses a bi-directional LSTM to capture words that appear before and after the current word.

Question 22

How does BERT work?

Answer 22

Uses an attention mechanism to see words in context, and only uses the ones that help with predictions.

Question 23

What are two models for generating visual embeddings?

Answer 23

1. Auto-encoders

2. Visual supervised learning tasks

Question 24

How do auto-encoders generate visual embeddings?

Answer 24

They generate low-dimensional representations of images with another path that re-creates the higher dimensional representation from that lower one. The model is optimized for recreating the original image as closely as possible. The path after the low-dimensional representation is then cut out and used as a feature vector.

Question 25

What kind of supervision do most text-based embedding models use?

Answer 25

Self-supervised learning

Question 26

How do visual supervised learning models generate visual embeddings?

Answer 26

The layer immediately before the softmax output layer is very low dimensionality and can be employed as a dense feature vector.

Question 27

How do you generate embeddings for pairs of entities, like users and videos on YouTube?

Answer 27

Build a two-tower neural net that generates embeddings for both entities in the same embedding space, and then produces random pairwise comparisons where it optimizes loss for similar entities to show up together and disjoint entities to show up apart.

Question 28

How do you do transfer learning with text data?

Answer 28

You can use an embedding generator as a feature extractor for text which you then go on to pass into another model like a spam classifier.

Question 29

What are the five steps in the initial building of a ML model?

Answer 29

1. Identify the business problem and what you’re trying to accomplish
2. Decide on one or more algorithms and techniques you’ll use to solve it
3. Train the model on the given data and features
4. Evaluate performance using identified metrics and iterate using hyperparameter searches
5. Deploy the model as soon as it performs equal to or better than the current model

Question 30

What are three issues that can lead to initial online ML model deployments not performing as well as offline?

Answer 30

1. Online/offline feature distribution drift
2. Online/offline feature extraction disparity
3. Under/overfitting