ML Training 2

Question 1

What is Precision?

Answer 1

From all the test examples that were assigned a label, how many actually were supposed to be categorized with that label.

TP/(TP+FP)

Question 2

What is Recall?

Answer 2

From all the test examples that should have had the label assigned, how many were actually assigned the label.

TP/(TP+FN)

Question 3

What are other ways to evaluate an AutoML model?

Answer 3

Precision, Recall, Confusion Matrix (see diagonal line), use Precision-Recall curve to decide score threshold (possible to assign to labels individually).

Question 4

Difference between Colab Enterprise vs Vertex AI Notebook

Answer 4

Colab Enterprise: A collaborative, managed notebook environment with the security and compliance capabilities of Google Cloud. Choose this if your project’s priorities are to collaborate with others and to avoid spending time managing infrastructure.

Vertex AI Workbench: A Jupyter notebook-based environment provided through virtual machine (VM) instances with features that support the entire data science workflow. Choose this if your project’s priorities are control and customizability.

Question 5

What platforms or features does Vertex AI Workbench support?

Answer 5

Importing conda environments, access data from Cloud Storage or BigQuery, automated notebook runs and idle shutdown, custom containers, third party credentials, monitoring instance, full control over infrastructure (VM instance).

Question 6

How do you overcome imbalanced datasets?

Answer 6

Downsample the majority class examples and upweight the downsampled examples to reduce prediction bias. Experiment with this rebalancing ratio, just like a hyperparameter. The batch size should be several times greater than the imbalance ratio (>=5).

Question 7

What is prediction bias?

Answer 7

A value indicating how far apart the average of predictions is from the average of labels in the dataset.

Question 8

What is selection bias?

Answer 8

Errors in conclusions drawn from sampled data due to a selection process that generates systematic differences between samples observed in the data and those not observed.

Includes Coverage bias, sampling bias, non-response/participation bias.

Question 9

What is coverage bias?

Answer 9

The population represented in the dataset doesn’t match the population that the machine learning model is making predictions about.

Question 10

What is sampling bias?

Answer 10

Data is not collected randomly from the target group.

Question 11

What is non-response/participation bias?

Answer 11

Users from certain groups opt-out of surveys at different rates than users from other groups.

Question 12

What is collaborative filtering model?

Answer 12

Collaborative filtering is a recommendation technique that filters and predicts items a user might like based on the reactions and preferences of similar users.

The fundamental premise is that people who agreed in their evaluation of certain items are likely to agree again in the future.

Question 13

What are the three main approaches to building recommendation systems on Google Cloud?

Answer 13

The three approaches are Matrix Factorization in BigQuery Machine Learning (BQML), Recommendations AI, and Two-Tower built-in algorithm.

Question 14

What is required to train a matrix factorization model on BigQuery?

Answer 14

A table with three input columns: user(s), item(s), and a feedback variable (implicit or explicit, such as ratings).

Question 15

What are the main benefits of Matrix Factorization on BigQuery?

Answer 15

The benefits include minimal ML expertise required (uses SQL), simple data input requirements, and ability to discover new user interests through collaborative filtering.

Question 16

What are the limitations of Matrix Factorization?

Answer 16

The limitations include inability to handle large feature sets (cannot handle more than 2 dimensions (user vs items), difficulty with incorporating new items into the matrix (cannot be continuously updated) and requirements for sufficient feedback data due to sparse input matrix.

Question 17

How does one test recommendation systems?

Answer 17

Set up A/B experiments.

Question 18

What is Recommendations AI?

Answer 18

It’s a fully managed service that deploys scalable recommendation systems using state-of-the-art deep learning techniques, including two-tower encoders.

Using ML, it solves the limitations of Matrix factorization.

Question 19

How often are Recommendations AI models updated?

Answer 19

Models are automatically retrained daily and tuned quarterly to capture changes in customer behavior, product assortment, pricing, and promotions.

Question 20

How do Recommendations AI achieve low serving latency?

Answer 20

It utilizes a scalable approximate nearest neighbors (ANN) service for efficient item retrieval during inference, resulting in low latency.

Question 21

What feature ensures data consistency in Recommendations AI?

Answer 21

It employs a scalable feature store that maintains consistency between online and offline tasks, preventing data leakage and training-serving skew issues.

Question 22

What makes the deployment process reliable in Recommendations AI?

Answer 22

It uses a robust CI/CD routine that validates models before deployment and ensures zero-downtime transitions to production.

Question 23

What is the main purpose of Two-Tower encoders?

Answer 23

They surface the most relevant items for users by encoding both candidate and query data into the same embedding space.

Question 24

What are the key benefits of the Two-Tower approach?

Answer 24

Benefits include greater control over model training, ability to handle various feature types (text, images), and better handling of cold-start cases.

Question 25

What are the requirements for implementing Two-Tower encoders?

Answer 25

You need training data combining query/user data with candidate/item data, including matched pairs, and an input schema describing the combined training data.

Question 26

What other services can be used together with Two-Tower encoders?

Answer 26

Vertex AI Matching Engine and ScANN provide a high-scale and low-latency Approximate Nearest Neighbor (ANN) service so you can more easily identify similar embeddings.

Hyperparameter tuning service such as Vizier can help you identify the optimal hyperparameters. Hardware Accelerators such as GPUs or TPUs.

Question 27

Who should consider Two-Tower encoders?

Answer 27

It’s best for users seeking greater control and flexibility, who have strong technical ML expertise and can work in a managed notebook environment. Note that you cannot do real-time personalization with this.

Question 28

Who should use Matrix Factorization?

Answer 28

It’s best for users with simplified datasets looking to quickly develop a baseline recommendation system.

Question 29

Who is Recommendations AI best suited for?

Answer 29

It’s ideal for teams lacking technical experience with production recommendation systems or those wanting to allocate resources to other priorities.

Question 30

With new data, do you need a new CT pipeline?

Answer 30

No, the previously deployed CT pipeline is executed. No new pipelines or components are deployed; only a new prediction service or newly trained model is served at the end of the pipeline.

Question 31

What’s the function of a CI/CD pipeline in ML training?

Answer 31

To deploy new implementations of ML pipelines and components quickly. If given new implementation, a successful CI/CD pipeline deploys a new ML CT pipeline.