MLOps

Question 1

What is MLOps?

Answer 1

Set of practices that aims to deploy and maintain ML models in production reliably and efficiently

Question 2

What is continuous integration?

Answer 2

Frequent merging of several small changes into a main branch. Automatically testing each change when you commit or merge them, and automatically kicking off a build.

Question 3

What is continuous delivery?

Answer 3

A practice that works in conjunction with continuous integration to automate the process of preparing and releasing code changes to production

Question 4

What is continuous deployment?

Answer 4

A practice that ensures code changes are continuously released into production

Question 5

What is the difference between continuous delivery and continuous deployment?

Answer 5

In continuous delivery, the team takes control of deploying new releases to end-users. Continuous deployment is a special case of continuous delivery where the team must ensure the builds passed all tests and automatically deploy them without human intervention.

Question 6

What is MLflow?

Answer 6

A platform for managing the end-to-end machine learning lifecycle. It is designed to make it easier for data scientists and machine learning engineers to track, compare, and reproduce machine learning experiments, and to deploy models into production

Question 7

What are the main components of MLflow?

Answer 7

• Experiments
• Runs
• Parameters
• Code
• Artifacts
• Model Registry

Question 8

What are experiments in MLflow?

Answer 8

A collection of machine learning runs, each of which represents a single execution of a machine learning model. Experiments can be organized into groups, allowing users to compare and contrast different runs

Question 9

What is a run in MLflow?

Answer 9

A single execution of a machine learning model. It includes a record of the parameters, code, and results of the model, as well as any artifacts that were generated during the run

Question 10

What are parameters in MLflow?

Answer 10

The settings and configurations that are used to control the behavior of a machine learning model. Parameters are stored as key-value pairs and are associated with a particular run

Question 11

How is the code treated in MLflow?

Answer 11

It is stored as a versioned artifact, allowing users to reproduce a particular run by executing the same code that was used in the original run

Question 12

What is an artifact in MLflow?

Answer 12

Files or directories that are generated during a machine learning run, such as model files or output files. Artifacts are stored as versioned artifacts, allowing users to retrieve and compare the artifacts from different runs

Question 13

What is the model registry in MLflow?

Answer 13

A centralized repository for storing and managing machine learning models. It allows users to track the history of a model, including the runs that were used to train and evaluate it, and to deploy the model to production

Question 14

What is a feature store?

Answer 14

A feature store pulls data from various sources and transforms the data to features required by the model

Question 15

What is a metadata store?

Answer 15

It is used to store and manage metadata about machine learning models, such as the training data that was used to build the model, the algorithms that were used, and the performance of the model. It can also be used to store metadata about other machine learning assets, such as feature data, pipelines, and experiments. In MLflow this is called the Tracking Server

Question 16

What is model monitoring?

Answer 16

The practice of monitoring the performance and behavior of machine learning models in production. Model monitoring typically involves collecting and analyzing data about the performance and behavior of machine learning models, such as their accuracy, precision, and recall

Question 17

Why is model monitoring important in machine learning?

Answer 17

• Model drift
• Model bias
• Model accountability

Question 18

What is model drift in model monitoring?

Answer 18

Degradation or change in performance when models are exposed to different data or environments

Question 19

What is model bias in model monitoring?

Answer 19

Models can be biased if they are trained on data that is not representative of the population they are intended to serve

Question 20

What is model accountability in model monitoring?

Answer 20

Organizations may be held accountable for the performance and behavior of their machine learning models. Model monitoring can help organizations to demonstrate the performance and behavior of their models, ensuring that they are transparent and accountable

Question 21

What is the difference between data drift and concept drift in model monitoring?

Answer 21

Data drift happens when the input data distribution or feature space changes (e.g. a service launched in a new country, expected features becoming NaNs). Model drift happens when the same inputs expect different outputs (e.g. when people searched for Wuhan pre-covid, they expected very different things from what they do now)