Developing Machine Learning Solutions

Question 1

What is the ML development lifecycle?

Answer 1

Machine learning (ML) lifecycle refers to the end-to-end process of developing, deploying, and maintaining machine learning models.

The end-to-end machine learning lifecycle process includes the following phases:

• Business goal identification
  ML problem framing
• Data processing (data collection, data preprocessing, and feature engineering)
• Model development (training, tuning, and evaluation)
• Model deployment (inference and prediction)
• Model monitoring
• Model retraining

Question 2

What is Amazon SageMaker?

Answer 2

Amazon SageMaker is a fully managed ML service. In a single unified visual interface, you can perform the following tasks:

• Collect and prepare data.
• Build and train machine learning models.
• Deploy the models and monitor the performance of their predictions.

Question 3

What are some ways to use Amazon SageMaker?

Answer 3

The following are ways to use SageMaker to build your ML model:

• Pre-trained models require the least effort and are models ready to deploy or to fine-tune and deploy using SageMaker JumpStart.
• Built-in models available in SageMaker require more effort and scale if the dataset is large and significant resources are needed to train and deploy the model.
• If there is no built-in solution that works, try to develop one that uses pre-made images for machine learning and deep learning frameworks for supported frameworks such as scikit-learn, TensorFlow, PyTorch, MXNet, or Chainer.
• You can build your own custom Docker image that is configured to install the necessary packages or software

Question 4

What is bias in ML?

Answer 4

A bullseye is a nice analogy because, generally speaking, the center of the bullseye is where you aim your darts. The center of the bullseye in this situation is the label or target—it predicts the value of your model—and each dot is a result that your model produced during training.

Think about bias as the gap between your predicted value and the actual value, whereas variance describes how dispersed your predicted values are.

In ML, the ideal algorithm has low bias and can accurately model the true relationship. The ideal algorithm also has low variability, by producing consistent predictions across different datasets.

Question 5

What is variance in ML?

Answer 5

Think about bias as the gap between your predicted value and the actual value, whereas variance describes how dispersed your predicted values are. In ML, the ideal algorithm has low bias and can accurately model the true relationship, and it has low variability by producing consistent predictions across different datasets

Question 6

What are the metrics used to evaluate classification?

Answer 6

• Accuracy
• Precision
• Recall
• F1
• AUC-ROC

Question 7

What are the metrics used to evaluate regression?

Answer 7

• Mean squared error
• R squared

Question 8

What are the elements of a confusion matrix?

Answer 8

True positive (TP)

If the actual label or class is “cat,” which is identified as “P” for positive in the confusion matrix, and the predicted label or class is also “cat,” then you have a true positive result. This is a good outcome for your model.

True Negative (TN)

Similarly, if you have an actual label of “not cat,” which is identified as “N” for negative in the confusion matrix, and the predicted label or class is also “not cat,” then you have a true negative. This is also a good outcome for your model. In both cases, your model predicted the correct outcome when using the testing data.

False positive (FP)

This is less than ideal and is when the actual class is negative, so “not cat,” but the predicted class is positive, so “cat.” This is called a false positive because the prediction is positive but incorrect.

False negative (FN)

This is also less than ideal. A false negative occurs when the actual class is positive, so “cat,” but the predicted class is negative, so “not cat.”

Question 9

What is accuracy in ML?

Answer 9

Accuracy

Formula for accuracy : tp + tn divided by tp + tn + fp + fn
Calculation for a model’s accuracy

To calculate the model’s accuracy, also known as its score, add up the correct predictions and then divide that number by the total number of predictions.

Question 10

What is precision in ML?

Answer 10

Precision

Precision removes the negative predictions from the picture. Precision is the proportion of positive predictions that are actually correct. You can calculate it by taking the true positive count and dividing it by the total number of positives.

Formula for precision: tp divided by tp + fp
Calculation for precision

When the cost of false positives are high in your particular business situation, precision can be a good metric. Think about a classification model that identifies emails as spam or not. In this case, you do not want your model labeling a legitimate email as spam and preventing your users from seeing that email.

Question 11

What is recall in ML?

Answer 11

Recall

In addition to precision, there is also recall (or sensitivity). In recall, you are looking at the proportion of correct sets that are identified as positive. Recall is calculated by dividing the true positive count by the sum of the true positives and false negatives. By looking at that ratio, you get an idea of how good the algorithm is at detecting, for example, cats.

Formula for precision: tp divided by tp + fn
Calculation for recall

Think about a model that needs to predict whether a patient has a terminal illness or not. In this case, using precision as your evaluation metric does not account for the false negatives in your model. It is extremely important and vital to the success of the model that it not give false negative results. A false negative would be not identifying a patient as having a terminal illness when the patient actually does have a terminal illness. In this situation, recall is a better metric to use.

Question 12

What is AUC-ROC in ML?

Answer 12

AUC-ROC

Area under the curve-receiver operator curve (AUC-ROC) is another evaluation metric. ROC is a probability curve, and AUC represents the degree or measure of separability.

AUC-ROC uses sensitivity (true positive rate) and specificity (false positive rate)

In general, AUC-ROC can show what the curve for true positive compared to false positive looks like at various thresholds. That means that when you calculate the AUC-ROC curve, you plot multiple confusion matrices at different thresholds and compare them to one another to find out the threshold you need for your business use case.

Question 13

What is mean squared error?

Answer 13

Mean squared error

The general purpose of mean squared error (MSE) is the same as the classification metrics. You determine the prediction from the model and compare the difference between the prediction and the actual outcome.

Calculation for mean squared error

More specifically, you take the difference between the prediction and actual value, square that difference, and then sum up all the squared differences for all the observations.

The smaller the MSE, the better the model’s predictive accuracy.

Question 14

What is R squared?

Answer 14

R squared

R squared is another commonly used metric with linear regression problems. R squared explains the fraction of variance accounted for by the model. It’s like a percentage, reporting a number from 0 to 1. When R squared is close to 1, it usually indicates that a lot of the variance in the data can be explained by the model itself.

MSE focuses on the average squared error of the model’s predictions to provide a measure of model performance. R squared provides a measure of the model’s goodness of fit to the data. Both are important but provide different perspectives.

Question 15

What is model deployment?

Answer 15

Model deployment is the integration of the model and its resources into a production environment so that it can be used to create predictions.

Question 16

What are the model deployment types and how would you describe them?

Answer 16

Self-hosted API

In a self-hosted API approach, you deploy and host your ML models on your own infrastructure, either on premises or in the cloud (using virtual machines or containers). This approach involves setting up and managing the necessary infrastructure, such as web servers, load balancers, and databases, to serve your ML models as APIs.

Managed API

Managed API services are cloud-based services that provide a fully managed environment for deploying and hosting your ML models as APIs. SageMaker is an example. These services abstract away the underlying infrastructure management so you can focus on building and deploying your models.

Question 17

What are the advantages/disadvantages of managed vs. self-hosted apis?

Answer 17

Advantages of self-hosted APIs include greater control over the infrastructure, potential cost savings (depending on usage), and the ability to customize the deployment environment. However, this approach requires more operational overhead and responsibility for managing and maintaining the infrastructure.

The choice between a managed API service or a self-hosted API for ML deployment depends on factors such as the specific requirements of your use case, the level of control and customization needed, the available resources and expertise, and cost considerations.

Question 18

What is Amazon SageMaker?

Answer 18

SageMaker is a fully managed ML service. With SageMaker, data scientists and developers can quickly and confidently build, train, and deploy ML models into a production-ready, hosted environment. Within a few steps, you can deploy a model into a secure and scalable environment.

Question 19

What are the advantages of Amazon SageMaker?

Answer 19

Sagemaker provides the following:

• Deployment with one click or a single API call
• Automatic scaling
• Model hosting services
• HTTPS endpoints that can host multiple models

Question 20

How can you deploy models using Amazon SageMaker?

Answer 20

Real Time

Real-time inference is ideal for inference workloads where you have real-time, interactive, and low latency requirements.

Batch Transformations

Use batch transform when you need to get inferences from large datasets and don’t need a persistent endpoint. You can also use it when you need to preprocess datasets to remove noise or bias that interferes with training or inference from your dataset.

Asynchronous

SageMaker asynchronous inference is a capability in SageMaker that queues incoming requests and processes them asynchronously. This option is ideal for requests with large payload sizes (up to 1GB), long processing times (up to one hour), and near real-time latency requirements.

Serverless

On-demand serverless inference is ideal for workloads that have idle periods between traffic spurts and can tolerate cold starts. It is a purpose-built inference option that you can use to deploy and scale ML models without configuring or managing any of the underlying infrastructure.

Question 21

What is MLOps?

Answer 21

MLOps refers to the practice of operationalizing and streamlining the end-to-end machine learning lifecycle from model development and deployment to monitoring and maintenance. It helps ensure that models are not just developed but also deployed, monitored, and retrained systematically and repeatedly.

Question 22

How do we use MLOps?

Answer 22

Applications that expose trained models might have different hosting requirements and strategies than standard applications. Trained models are sensitive to changes in data; therefore, a model-based application that works well when first implemented might not perform as well days, weeks, or months after being implemented. To account for these differences, you need different processes and procedures for applications that are based in managing ML.

MLOps accounts for the unique aspects of artificial intelligence and machine learning (AI/ML) projects in project management, continuous integration and delivery (CI/CD), and quality assurance. With it, you can improve delivery time, reduce defects, and make data science more productive.

Question 23

What are the goals of MLOps?

Answer 23

A goal of MLOps is to get ML workloads into production and keep them operating. To meet this goal, MLOps adopts many DevOps principles and practices for the development, training, deployment, monitoring, and retraining of machine learning models. The aim is to use MLOps to do the following:

• Increase the pace of the model development lifecycle through automation.
• Improve quality metrics through testing and monitoring.
• Promote a culture of collaboration between data scientists, data engineers, software engineers, and IT operations.
• Provide transparency, explainability, audibility, and security of the models by using model governance.

Question 24

What are the benefits of MLOps?

Answer 24

Adopting MLOps practices gives you faster time-to-market for ML projects by delivering the following benefits:

Productivity

By providing self-service environments with access to curated datasets, data engineers and data scientists can move faster and waste less time with missing or invalid data.

Reliability

By incorporating CI/CD practices, developers can deploy quickly with increased quality and consistency.

Repeatability

By automating all the steps in the machine learning development lifecycle, you can ensure a repeatable process, including how the model is trained, evaluated, versioned, and deployed.

Auditability

By versioning all inputs and outputs, from data science experiments to source data to trained models, you can demonstrate exactly how the model was built and where it was deployed.

Data and Model quality

With MLOps, you can enforce policies that guard against model bias and track changes to data statistical properties and model quality over time.

Question 25

What are the key principles of MLOps?

Answer 25

Version control

For reproducibility, machine learning workflows must track changes to assets like data, code, and models. It can be rolled back to previous versions when needed. Overall, version control and code review provide reproducible, trustworthy machine learning.

Automation

For repeatability, consistency, and scalability, you can automate the various stages in the machine learning pipeline. This includes the data ingestion, pre-processing, model training, and validation and deployment stages. Automated testing helps you discover problems early for fast error fixes and learnings.

CI/CD

Through automation, you can continuously test and deploy assets in the following ways:

• Continuous integration extends the validation and testing of code to data and models in the pipeline.
• Continuous delivery automatically deploys the newly trained model or model prediction service.
• Continuous training automatically retrains ML models for redeployment.
• Continuous monitoring uses data monitoring and model monitoring of metrics related to business.

Model governance

Good governance of machine learning systems requires close collaboration between data scientists, engineers, and business stakeholders. Clear documentation, effective communication channels, and feedback mechanisms help align everyone and improve models over time. It is also crucial to protect sensitive data, secure access, and meet compliance rules. A structured process for reviewing, validating, and approving models before deployment checks for fairness, bias, and ethics. Governance manages all aspects of systems for efficiency.

Question 26

What is the ML/MLOps lifecycle?

Answer 26

Managing code, data, and models throughout the ML lifecycle requires the following touchpoints:

• Processing code in data preparation
• Training data and training code in model building
• Candidate models, test, and validation data in model evaluation
• Metadata during model selection
• Deployment-ready models and inference code during deployment
• Production code, models, and data for monitoring

Question 27

Provide and example of the MLOps automation process?

Answer 27

1. Data Preparation
2. Model Build
3. Model Evaluation
4. Model Selection
5. Deployment
6. Monitoring

• The model building pipeline creates new models upon inititation, for example when new data become available.
• When the model building pipeline completes, you can implement quality control measures at the model registration step. The quality control step can be either manual (human in the loop) or automated. If a model meets baseline performance metrics, it can be registered with a model registry.
• You can use the registry to approve or reject model versions. The model approval action can act as an initiation to start the deployment pipeline.
• The deployment pipeline is most similar to traditional CI/CD systems. This pipeline includes steps such as the following:
  • Source
  • Build
  • Deployment to staging environment
  • Testing
  • Promotion to production environment
• As soon as the model is in production, you should get feedback from the live system. For ML solutions, monitor the hosting infrastructure, data quality, and model performance.