MLOps Question Bank

Question 1

How many row-level predictions are saved each day?

Answer 1

Up to 2.4M. We save the first 100k scored records per hour

Question 2

How does DataRobot calculate data drift?

Answer 2

The data drift plot defaults to population stability index (PSI) scores. You can also get KL divergence, Hellinger divergence, Jensen–Shannon divergence, and Histogram intersection scores through the Python client

Question 3

What types of features do we calculate data drift for?

Answer 3

The Feature Drift vs. Feature Importance chart monitors the 10 most impactful numerical or categorical features in your data. This chart excludes any text, percentage, or currency features, which means that you can have less than 10 features plotted

Question 4

How do you change the cutoff for accuracy drift?

Answer 4

You can change the cutoff in the Monitoring tab of the Settings section inside a deployment.

Question 5

How does the AssociationID work? What happens when you upload an actual with a duplicate AssociationID?

Answer 5

The actuals payload must contain the column names associationId and actualValue. Use the optional column wasActedOn to indicate if the prediction was acted on in a way that could have affected the actual outcome. If you submit multiple actuals with the same association ID value, either in the same or a subsequent request, DataRobot uses the latest actuals value

Question 6

At what granularity can we see drift and service health? Hourly, daily, weekly, monthly?

Answer 6

At what granularity can we see drift and service health? Hourly, daily, weekly, monthly?

Question 7

What roles are associated with deployments? How are they different than roles associated with projects?

Answer 7

User/Admin/Consumer? Need to look up.

Question 8

Do we support any language?

Answer 8

We support models built in most languages. Two notable exceptions in 5.2 are SAS and DataRobot models (however, you can use Agents to monitor codegen).

Question 9

Do we support any modeling type (i.e. regression, classification, multi-class)?

Answer 9

We currently only support regression and binary classification. We plan to relex these limitations in future releases.

Question 10

What is the maximum number of environments we can support?

Answer 10

10

Question 11

Can you adjust the threshold for drift?

Answer 11

Yes - MLOps Agent.The MLOps agent can report predictions data and metrics for all external deployments, whether fully connected, intermittently connected, or completely disconnected from MLOps. If you have deployments running in isolated environments and disconnected from the network, for example, you can provide their data to the agent, and then view and manage them from MLOps.

Question 12

What is the lowest resolution that you can display Data Drift and Accuracy?

Answer 12

Daily

Question 13

Do we offer tracking of externally deployed models

Answer 13

Yes - MLOps Agent.The MLOps agent can report predictions data and metrics for all external deployments, whether fully connected, intermittently connected, or completely disconnected from MLOps. If you have deployments running in isolated environments and disconnected from the network, for example, you can provide their data to the agent, and then view and manage them from MLOps.

Question 14

What are the levels of Role Based Access Control (or User Management) provided by MLOps?

Answer 14

Deployment Admin, Owner, User, Consumer

Question 15

How do you change the prediction threshold in MLOps and how does this differ in MMM for AutoML/AutoTS

Answer 15

You currently cannot change the prediction threshold in AutoML / AutoTS for DataRobot models, but you can change it for Custom Models inside MLOps

Question 16

What access does the Deployment Admin user offer users?

Answer 16

A deployment administrator role, assigned by the system administrator, has User role permissions for all existing and newly created deployments within their organization; the deployment admin is also able to approve new deployments facilitating governnance

Question 17

Does MLOps offer governance capabilities? Explain.

Answer 17

Role Based Access Control (User Management), Deployment Admin, Workflow Approvals, Materiality Score (Importantance), Real Time Monitoring

Question 18

What is the alert notification structure for MLOps, what are you able to customize?

Answer 18

Email. And you can customize when alerts are scheduled, who receives them, as well at thresholds for accruacy

Question 19

How does DataRobot calculate accuracy drift?

Answer 19

How does DataRobot calculate accuracy drift?

Question 20

What are some components of service health that MLOps will monitor?

Answer 20

Number of predictions,number of requests, execution time, response time, predictions over certain time, data error, system error, consumers, cache hit rate

Question 21

What is Association ID and how is it used by MLOps?

Answer 21

Association ID is unique identifier for each prediction request. It can be optional or mandatiroy field passed with a prediction request. Once actuals are available they are sent back to DR along with association ID so that DR can tie back the actuals and predictions

Question 22

What are some reasons for Red/Failing model health (service, data, or accuracy)?

Answer 22

At least one 5xx error, At least one higher-importance attribute’s distribution has shifted since the model was deployed, Accuracy has severely declined since the model was deployed

Question 23

Does MLOps support external deployments?

Answer 23

Two components: 1. MLOps Agent 2. Adding External Deployment. You can use the deployment management tool to analyze historical predictions and continuously assess model quality based on prediction output. Note that while not all the tools of model management are available to externally imported datasets, the inventory, data drift, and accuracy tabs provide an excellent starting point.

Question 24

What data from training is used as a baseline to calculate Model Health?

Answer 24

uses the holdout data distribution. Where model is trained on holdout it goes back to the same BP on non holdout data and uses that for baseline. For Custom Models, baseline is not currently provided but it is on the roadmap.

Question 25

What types of model classes does MMM support? What about MLOps

Answer 25

MMM: Binary Classification, Regression, Limited TS. MLOps: Binary Classificationn, Regression

Question 26

Does MLOps support time aware projects?

Answer 26

No TS, but does support OTV partitioning

Question 27

How many features do we currently support drift tracking for? What variable types are supported?

Answer 27

The Feature Details chart provides a histogram that compares the distribution of a selected feature in the training data to the distribution of that feature in the scoring data. From the results you can easily identify details of changed values (which values became more or less frequent), helping to assess the severity of the problem as well as its causes and resolutions. The chart plots up to the top 25 values based on Feature Impact from your training data, as well as Missing and Other counts. The Missing count contains all records with missing feature values (that is, NaN as the value of one of the features). The Other count, which applies to categorical features, includes all features other than the 25 most frequent values.

Question 28

What metric is used to calculate feature important for MLOps?

Answer 28

For DataRobot model deployments, DataRobot uses the Feature Impact score. You must calculate the Feature Impact score for the model before the chart is available. For external models, DataRobot uses the Importance score it calculated when ingesting the data (ACE). This is the same calculation run during EDA2 in the standard model building process.

Question 29

In which quadrant of the Data Drift plot would you see Red/Failing features?

Answer 29

They should explain the four quadrant interpretation of the Data Drift chart. High importance feature(s) are experiencing high drift. It would be advised that users investigate these immediately.

Question 30

What is the Feature Details plot? How should it be framed to users?

Answer 30

The Feature Details chart provides a histogram that compares the distribution of a selected feature in the training data to the distribution of that feature in the scoring data. From the results you can easily identify details of changed values (which values became more or less frequent), helping to assess the severity of the problem as well as its causes and resolutions.

Question 31

How does MLOps handle new categorical levels from your scoring data?

Answer 31

For categorical features, there is an additional New level count. It only displays after you make predictions with a scoring file that has a new value for a feature which the training set did not.

Question 32

What are some of the restrictions around model replacement?

Answer 32

DR -> DR, Custom -> Custom

Question 33

How does DataRobot calculate a baseline for drift?

Answer 33

There are two key components to Model Health: Accuracy and Drift. For DataRobot AutoML & AutoTS models, Model Monitoring uses holdout data for Data Drift AND Accuracy. If a model is trained into its holdout then it goes back to the same BP on non holdout data and uses that for its baseline. For Custom Models, Data Drift is created from that models Training Data. Accuracy baseline is not currently available for Custom Models, it is on the roadmap.

Question 34

What happens when you click on test custom model

Answer 34

Background process checks if it can spin off the model using the environment and also sees if the model returns predictions on the test file uploaded by you

Question 35

What are the mandatory fields for uploading custom models

Answer 35

Name, tar file, target column, target type, if classification what is positive and negative target values

Question 36

In tracking agents what type of models are supported

Answer 36

Currently python and java. R support is coming soon

Question 37

How is feature importance calculated for custom models

Answer 37

ACE methodology on uploaded training data

Question 38

My custom model treats zip_code as a categorical, but it appears as a numeric in the data drift tab. Why?

Answer 38

It's possible that the DataRobot project created in the background detected zip_code as a numeric field

Question 39

My custom model treats purpose as a categorical, but it's not appearing the data drift tab. Why?

Answer 39

My custom model treats purpose as a categorical, but it's not appearing the data drif tab. Why?

Question 40

What are the different ways you can upload actuals back

Answer 40

Use a file or table and use API call to upload. Or you can use AI catalog to upload it from the settings tab

Question 41

What happens if you upload two different actuals for same association ID

Answer 41

The backed data will be overwritten with the latest record. The chart however will reflect the first record. Chart does not get refreshed. This will change in future

Question 42

Is there a limit on number of records/predictions DR will use for dirft tracking

Answer 42

Per hour 100k and overall 1 million per day

Question 43

What is the "prediction was acted on" flag meant to indicate?

Answer 43

??

Question 44

What metrics are available for Data Drift

Answer 44

PSI for UI, and PSI, KL divergence, Hellinger divergence, Jensen–Shannon divergence and Histogram intersection using API

Question 45

Product distinctions between Model Monitoring and Management (MMM) included with the AutoML and MLOps product?

Answer 45

MMM: deploy, monitor, and manage DataRobt model models, Deployment Admin, Security and User Management, Autoscaling Prediction Servers. MLOps Exclusive: deploy, monitor, and manage DR models + custom models + remote models + DR Scoring Code + Agents. Transfer models to a seperate scoring env, approval workflow and compliance