ML Models

Question 1

Cons of Logistic Regression

Answer 1

• Non-linear problems can’t be solved with LR (only produces a linear decision boundary).
• Can’t capture feature interactions when the value of one feature influences the value of another (between user, ad, and publisher for instance)

Question 2

Pros of Logistic Regression

Answer 2

Pros:
- Easy to implement
- Easy to train
- Fast inference
- Interpretable
- Often useful as a baseline model

Question 3

Logistic Regression

Answer 3

• Models the probability of a binary outcome using a weighted linear combination of features.
• Work well when the data is linearly separable
• Not a good choice for ad click prediction

Question 4

Gradient-boosted decision trees

Answer 4

Pros
- Interpretable and easy to understand
- Can be used for feature selection (importance) and feature extraction

Cons
- Inefficient for continual learning. Not designed to be fine-tuned with new data. Usually need to retrain the model from scratch.

Question 5

Two-tower neural network

Answer 5

• Generate user embeddings for user features
• Generate ad embeddings for ad features
• The similarity between the user and ad embeddings is used to calculate relevance

Question 6

Challenges of ad click prediction

Answer 6

• Feature space is large and sparse. most features are filled with zeros.
• Difficult to capture pairwise interactions
• Continuous retraining

Question 7

Deep & Cross Network

Answer 7

• Can replace manual feature cross method
• Deep network: Learns complex generalizable features using DNN arch
• Cross network: Automatically captures feature interactions and learns good feature crosses

Question 8

Factorization machines

Answer 8

• Efficiently captures pairwise interactions between features
• Improves logistic regression
• Useful for ad click prediction

Question 9

How do factorization machines work?

Answer 9

Learns an embedding vector for each feature. The interaction between two features is the dot product of their embeddings

Question 10

Support Vector Machines

Answer 10

Kind of like logistic regression in multi-dimensional space

• Find a shape in n-dim space that classifies data points

Question 11

What is Learn To Rank?

Answer 11

• Supervised machine learning to solve ranking problems
• Given a query and a list of items, determine the optimal ordering of the items from most relevant to least relevant

Question 12

What are the types of Learn to Rank?

Answer 12

• Pointwise
• Pairwise
• Listwise

Question 13

Point-wise Learn to Rank

Answer 13

• The score of each item is predicted independently of the other items
• The final ranking is achieved by sorting the predicted relevance scores

Question 14

Pair-wise Learn to Rank

Answer 14

• Given a query and two items, predicts which item is more relevant to the query
• Examples: RankNet, LambdaRank, LambdaMART

Question 15

List-wise Learn to Rank

Answer 15

• Given a query and a list of items, predict the optimal ordering of an entire list
• Examples: SoftRank, ListNet, and AdaRank

Question 16

Tradeoffs for Learn to Rank Approaches?

Answer 16

• Pairwise and listwise approaches produce more accurate results but they are more difficult to implement and train

Question 17

Pros of Decision Trees

Answer 17

• Fast training
• Fast inference
• Minimal data prep (doesn’t require normalization or scaling) since the algorithm doesn’t depend on the distributions of input features
• Interpretable

Question 18

Cons of Decision Trees

Answer 18

• Over-fitting. Decision trees and very sensitive to small variations in data. A small change in input may lead to different outcomes at serving time. Small changes in training data can produce a different tree structure.
• Too sensitive. Predictions are less reliable. Naive decision trees are rarely used in practice.

Question 19

Techniques to reduce the sensitivity of decision trees

Answer 19

Bagging
Boosting

Question 20

Bagging (Decision Trees)

Answer 20

• Ensemble learning method that trains a set of ML models in parallel on multiple subsets of training data
• Predictions of all these trained models are combined to make a final prediction
• Reduces the model sensitivity

Question 21

Random Forest

Answer 21

• Builds multiple decision trees in parallel during training
• A voting mechanism is used to combine the predictions to make a final prediction
• Example of bagging a decision tree

Question 22

Advantages of Bagging

Answer 22

-Reduces the effect of over-fitting (high variance)
- Doesn’t increase training time very much because the decision trees can be trained in parallel
- Does not add much latency at the inference time because decision trees can process the input in parallel

Question 23

Disadvantages of Bagging

Answer 23

• Not helpful when the model faces under-fitting (high bias)
• Need boosting for that

Question 24

Linear Regression vs Logistic Regression

Answer 24

• Linear regression is used to estimate the dependent variable in case of a change in independent variables. For example, predict the price of houses.
• Logistic regression is used to calculate the probability of an event. For example, classify if tissue is benign or malignant.

Question 25

Boosting

Answer 25

• Ensemble learning technique to improve the performance of weak learners by combining predictions.
• The final model is a weighted combination of the weak learners

Question 26

Bias

Answer 26

Stubbornness of the algorithm when confronted with new data

Question 27

Deep Factorization Machines

Answer 27

• Combines the strengths of a NN and FM
• The DNN captures higher order features
• The FM captures low-level pairwise interactions