Machine Learning in the Enterprise

Question 1

How do you call a process of developing a pipeline for model (re)training?

Answer 1

Model operationalization

Question 2

What are the different types of inference?

Answer 2

Online prediction - API for real-time prediction
Streaming prediction - near real-time event-based predictions
Batch prediction - offline prediction in batches
Embedded prediction - prediction on different devices like mobile phone

Question 3

What is a Data Catalog?

Answer 3

Data Catalog is a data management service that can add additional metadata to the data coming from various sources like BigQuery, Cloud Storage, Dataplex, etc. It marks the data and make it discoverable and understandable to everyone.

Question 4

What is Dataplex?

Answer 4

Enables you to centrally manage your data coming from various sources like data lakes, data warehouses and data marts. It consists of Lakes, Zones, and assets. You can control who has access to which data by logically organizing them into Zones/Lakes, making it easily discoverable, data lineage, automatic data quality checks, and automatic meta-data extraction.

Question 5

What is Analytics Hub?

Answer 5

It is a central location where you can publish your data and subscribe to data from other publishers, easily ingest and use it in your project. Publishers pay for the storage of the data while subscribers pay for the analytics workload.

Question 6

What are the different data preprocessing options on GCP?

Answer 6

BigQuery - recommended for structured data, performing transformations on the data and storing it in a “clean” dataset
Dataflow - recommended for unstructured data
Dataproc - for customers who already have their data preprocessing pipeline in Spark and Hadoop
Tensorflow Extended - if it is used for model pipelines

Question 7

What is Dataprep?

Answer 7

UI on top of Dataflow. It has some nice graphs, quality checks, statistics, easy to use transformations on data… The most important feature is recipes that chain together different types of transformations (predefined or your own).

Question 8

What are some optimal and maximal values for batch size?

Answer 8

40-100 as an optimal value and 500 as a maximum

Question 9

Is it better to start with a small or larger batch size?

Answer 9

Smaller

Question 10

What is model parallelism?

Answer 10

When your model is too big to fit on one device, you have to split it it per layers or cluster nodes where portions of deep neural network would be split between nodes.

Question 11

How can you run your Python training application in a custom job, what options do you have for running your code?

Answer 11

You use pre-build container provided by Google for various frameworks or use a custom image.

Question 12

How to work with large datasets that can not fit into memory during training?

Answer 12

Approaches could be to introduce streaming or load data into batches.

Question 13

What is the required thing to do when the custom job is completed with model training?

Answer 13

You need to store the model in Cloud Storage, after training is completed, the VM is shut-down and if you don’t save the model for later usage it will be deleted.

Question 14

Can you use model artifacts stored in GCS directly in Vertex AI for prediction?

Answer 14

No, you have to first store it to Vertex AI Model Registry

Question 15

What are the different ways of packaging your code for custom training?

Answer 15

• Store your code in a single python file and use it in a combination with pre-built container (good for prototyping)
• Package your code in Python source distribution with a pre-built container
• Create a custom Docker image and store in Artifact Registry

Question 16

What is Cloud Storage FUSE?

Answer 16

It is a tool that enables you to mount GCS buckets to your code and access files/folders directly without downloading them. They are available under “/gcs/” root folder and it can be treated like a file system.

Question 17

What are the different ways of loading data for custom training?

Answer 17

• Load data from Cloud Storage using Cloud Storage FUSE
• Mount a NFS share
• Use managed datasets

Question 18

If you need to access other Google resources from a custom training job, what kind of authentication is the recommended way to access them?

Answer 18

The recommended way is to use ADC (Authentication Default Credentials). Vertex AI automatically configures Custom Code Service Agent with predefined permissions but if you need a different set of permissions you can create a custom service account.

Question 19

What do you need to do to read files from GCS in Vertex AI custom jobs?

Answer 19

Nothing, they are automatically accessible on “/gcs/” path as GCS FUSE is automatically integrated.

Question 20

What will happen if a VM is shutdown/restarted during model training? How to approach this problem?

Answer 20

If VM restarts during model training, training progress will be lost and when it starts up again it will start the training from beginning. To resolve this, rule of the thumb is if your training lasts more than 4 hours make sure to:
1. Store training intermediate steps to GCS
2. When training starts, first check if training progress already exists

Question 21

How can you create a custom container image for training?

Answer 21

You can use autopackaging feature that automatically builds a Docker image, push it to Container Registry and creates CustomJob resource based on container image in a single command. This is not working for TrainingPipeline and HyperparameterTuningJob. Another option would be to manually create a Docker file.

Question 22

What is the difference between Custom Job and Training Pipeline?

Answer 22

CustomJob is a single execution of your custom training code while Training Pipeline orchestrates multiple Custom Job, hyperparameter tuning jobs, storing model to Model Artifact, etc.

Question 23

What type of search algorithms is Vertex AI Vizier supporting?

Answer 23

Grid - search each combination of hyperparameters
Random - random combination of params
Bayesian - decide which params to choose based on results of previous iterations (default)

Question 24

What is the baseline for skew and drift in model monitoring?

Answer 24

Skew - statistical distribution of feature values in training data
Drift - statistical distribution of feature values in the recent past

Question 25

What kind of models is Vertex AI model monitoring supporting?

Answer 25

AutoML tabular and custom tabular models

Question 26

What are the things you can monitor OOTB in Vertex AI Monitoring?

Answer 26

Skew and drift detection for prediction requests and feature attributions.

Question 27

What are the necessary steps to enable monitoring?

Answer 27

1. Upload the model to Vertex AI Endpoint
2. Configure a model monitoring specification
3. Upload model monitoring specification to Vertex AI Endpoint
4. Upload or automatic generation of input schema for parsing
5. For feature skew, upload the training data for automatic generation of the feature distribution
6. For feature attributions, upload the corresponding Vertex AI Explainability configuration

Question 28

Explain in detail how model monitoring works in GCP.

Answer 28

When model monitoring is enabled on Vertex AI Endpoint and input schema is configured prediction logs are stored in the Big Query table that will be used for skew and drift detection.

You have to configure monitoring intervals (how often you will check for skew and drift) and provide a sample rate (how many prediction requests should be logged and used for monitoring - a value between 0 and 1 where 0.5 means that you randomly take 50% of incoming prediction requests).

For skew detection, you have to upload a training data sample against which drift will be evaluated (no need for AutoML).

For drift detection of feature attributions, you have to provide a Vertex AI Explainability configuration (if not AutoML)

When skew or drift is detected based on a certain threshold that you configure, an alert is sent to the configured email(s).

Question 29

What is the purpose of input schema in model monitoring?

Answer 29

For the model to be able to parse prediction features there has to be an input schema. For Auto ML it will be created automatically, for custom models it will try to create it based on the first 1000 prediction requests, or you have to upload it manually. It works best with standard key: value pairs.

Question 30

What type of data/models is model monitoring supporting?

Answer 30

Only Auto ML tabular and tabular custom-trained models

Question 31

Should you monitor both training-serving skew and prediction drift at the same time?

Answer 31

It is possible, but you should prioritize. If training data is available to you, you should monitor training-serving skew. If not available, you should monitor prediction drift.

Question 32

Which type of features are supported in model monitoring?

Answer 32

Numerical and categorical features - monitors changes in the distribution of features

Question 33

How are feature distributions calculated in model monitoring?

Answer 33

Categorical by percentage/number of occurrences and numerical are first organized into bins and then the same as categorical.

Question 34

What happens if you don’t specify a target column in your training data in model monitoring?

Answer 34

It will take the last feature in the training data.

Question 35

What happens if you don’t provide input schema from model monitoring?

Answer 35

It will try to automatically detect schema from first 1000 prediction requests but it will stay in pending state until it is created.

Question 36

Is it always recommended to use Vertex AI Pipelines if you need to orchestrate a few steps in ML workflow?

Answer 36

If you are using Tensorflow which processes terabytes of structured and unstructured data it is recommended to use TFX.

Question 37

Explain the process of Vertex AI Pipeline creation

Answer 37

1. Define components (functions) and their input and output that represent pipeline steps
2. Chain them together in a pipeline and define the output of which component is the input of which component
3. Compile pipeline which results in the creation of a .json or .yaml file that represents the pipeline
4. Run the pipeline by referencing that file

Question 38

What options are available to create components in Vertex AI Pipelines?

Answer 38

You can create a custom component or use prebuilt Google components.

Question 39

What is the benefit of using Vertex AI Pipelines compared to Kubeflow pipelines?

Answer 39

Because it is a managed service, you don’t have to maintain a Kubernetes cluster.

Question 40

What kind of data is Vertex Explainable AI supporting?

Answer 40

Tabular and image data

Question 41

Is it possible to have lower latency online predictions if your application resides within the local network?

Answer 41

Yes, you can use a private endpoint with VPC where the communication will not leave the private network (go to the internet).

Question 42

What are the options to pass the data for batch prediction?

Answer 42

You can use the data from the data lake/warehouse or from the Vertex AI Feature Store.