12. Model Monitoring, Tracking, and Auditing Metadata

Question 1

What are two types of drifts?

Answer 1

Concept drift
Data drift

Question 2

What is concept drift?

Answer 2

The relationship between input variables and predicted variables change.

Question 3

How do you prevent model deterioration?

Answer 3

Model monitoring, i.e., monitor input data and evaluate the model with the same metrics during the training phase.

Question 3

What is data drift?

Answer 3

Input data change, e.g., statistical distribution, schema, feature definition

Question 4

What can Vertex AI model monitoring monitor?

Answer 4

Training-serving skew: Feature distribution differences between production and training.
Prediction drift: Input’s statistical distribution changes in production over time.

Question 5

What are the two types of data that can be monitored?

Answer 5

Categorical features and numerical values

Question 5

How do you calculate baselines?

Answer 5

Baseline for skew detection: The statistical distribution of the feature’s values in the training data
Baseline for drift detection: The statistical distribution of the feature’s values in the production data (recent past)
Distribution calculations for categorical and numerical features (bin): The count or percentage of instances of each possible value.

Question 6

How does Vertex AI monitor drift and skew?

Answer 6

It compares the baselines and the equivalent latest values in the production.
Categorical features: L-infinity distance
Numerical features: Jensen-Shannon divergence
Vertex AI takes as an anomaly if the distance score hits a pre-defined threshold.

Question 6

What are the factors for effective monitoring?

Answer 6

Sampling rate
Monitoring frequency
Alerting thresholds
Number of models in an endpoint

Question 7

How do you monitor input schemas?

Answer 7

The input values are part of the payload of the prediction requests. You can specify a schema when you configure model monitoring.

Question 7

What are the two types of schema?

Answer 7

Automatic schema: Model monitoring will analyze and detect the schema.
Custom schema: User specified in Open API format

Question 7

What are the three types of schema formats?

Answer 7

Object: key/value pairs
Array: array-like format
String: csv-string

Question 7

What is the reason for logging?

Answer 7

Monitor input trends
Auditing

Question 8

What are the models supported by Vertex AI logging?

Answer 8

AutoML models (tabular & image)
Custom-trained models
Logging can be enabled during model deployment or endpoint creation

Question 8

What are the three types of logging?

Answer 8

Container Logging: stdout and stderr for debugging
Access Logging: Time stamp and latency for each request to Cloud Logging
Request-Response Logging: sample of the online prediction requests and responses.

Question 9

Where can you change log settings?

Answer 9

Create an endpoint
deploy a model to an endpoint

Question 10

What is parameters, artifacts and metrics of an ML experiment called?

Answer 10

Metadata

Question 11

What are the limitations for model monitoring and logging?

Answer 11

Both services use BigQuery table
Model monitoring is enabled, you can’t enable request-response logging.
Request-response logging is enabled first, then model monitoring. The request-response logging can’t be modified.

Question 12

What are the uses of metadata?

Answer 12

Detect model degradation
Compare different sets of hyperparameters
Track the lineage of the ML artifacts
Rerun an ML workflow with the same parameters
Track downstream usage of artifacts for audit purposes

Hints: Whales Dive, Humpbacks Love Abyss.

Question 13

What is Metadata store?

Answer 13

It is the top-level container for all the metadata resources, i.e., one by organization

Question 14

How do you manage ML Metadataschema?

Answer 14

Most common types of resources stored have predefined schemas called system schemas.

Question 14

What are Metadata resources?

Answer 14

Artifacts: Pieces of data created by or consumed by an ML workflow, e.g., datasets, models, input files, training logs, metrics
Context: A group of artifacts and executions that can be queried for identifying the best model.
Execution: A step in a ML workflow.
Events: An event connects artifacts and executions.
Metadataschema: Specifies the schema to be used by the particular types of data like artifact or execution.

Hints: Apples Can Entertain Every Monkey.

Question 15

What are the functions of Vertex AI Experiments?

Answer 15

Track the steps for an experiment run (preprocessing, embedding, training)
Track input like algorithms, hyperparameters, dataset, etc.
Track output of these steps like models, metrics, checkpoints, etc

Question 15

What are automatically generated when you use Vertex AI Pipelines?

Answer 15

The model metadata and artifacts are automatically stored in the metadata store for lineage tracking, e.g., dataset summary, model evaluation metrics, metadata on certain executions.

Question 16

How do you use Vertex AI Debugging?

Answer 16

Install interactive Bash shell in the training container
Run the custom training
Make sure the user having right permissions
Enable enableWebAccess API
You can use interactive shell to do the followings:
Check permission
Visualize Python execution with profiling tools
Analyze performance of training node using Perf
Check CPU or GPU usage