Developing Machine Learning Solutions

Question 1

ML Lifecycle

Answer 1

Business goal identification
ML problem framing

Data processing (data collection, data preprocessing, and feature engineering)

Model development (training, tuning, and evaluation)

Model deployment (inference and prediction)

Model monitoring

Model retraining

Question 2

Feature engineering is the process of creating, transforming, extracting, and selecting variables from data.

Answer 2

the process of creating, transforming, extracting, and selecting variables from data.

Question 3

Model Development

Answer 3

Initially, upon training, the model will not yield the expected results. Therefore, developers will do additional feature engineering and tune the model’s hyperparameters before retraining.

Question 4

Amazon SageMaker Data Wrangler is a

Answer 4

low-code no-code (LCNC) tool. It provides an end-to-end solution to import, prepare, transform, featurize, and analyze data by using a web interface. Customers can add their own Python scripts and transformations to customize workflows.

Question 5

For more advanced users and data preparation at scale,

Answer 5

Amazon SageMaker Studio Classic comes with built-in integration of Amazon EMR and AWS Glue interactive sessions to handle large-scale interactive data preparation and machine learning workflows within your SageMaker Studio Classic notebook.

Finally, by using the SageMaker Processing API

Question 6

Amazon SageMaker Feature Store helps data scientists, machine learning engineers, and general practitioners to

Answer 6

create, share, and manage features for ML development.

Features stored in the store can be retrieved and enriched before being served to the ML models for inference

Question 7

Customers aiming at a LCNC option can use Amazon SageMaker Canvas.

Answer 7

With SageMaker Canvas, they can use machine learning to generate predictions without needing to write any code.

Question 8

Amazon SageMaker JumpStart provides

Answer 8

pretrained, open source models that customers can use for a wide range of problem types.

Question 9

Customers can use Amazon SageMaker Experiments to

Answer 9

experiment with multiple combinations of data, algorithms, and parameters, all while observing the impact of incremental changes on model accuracy.

Question 10

Amazon SageMaker Automatic Model Tuning

Answer 10

Hyperparameter tuning is a way to find the best version of your models. does that by running many jobs with different hyperparameters in combination and measuring each of them by a metric that you choose.

Question 11

Amazon SageMaker Model Monitor, customers can

Answer 11

observe the quality of SageMaker ML models in production. They can set up continuous monitoring or on-schedule monitoring. SageMaker Model Monitor helps maintain model quality by detecting violations of user-defined thresholds for data quality, model quality, bias drift, and feature attribution drift.

Question 12

SageMaker JumpStart provides

Answer 12

pretrained open source models for a range of problem types to help you get started with machine learning. models are ready to deploy or to fine-tune

Question 13

AutoML is available in SageMaker Canvas. It

Answer 13

simplifies ML development by automating the process of building and deploying machine learning models.

Question 14

Built-in models available in SageMaker require more

Answer 14

effort and scale if the dataset is large and significant resources are needed to train and deploy the model

Question 15

If there is no built-in solution that works, try to develop one that uses

Answer 15

pre-made images for machine learning and deep learning frameworks for supported frameworks such as scikit-learn, TensorFlow, PyTorch, MXNet, or Chainer.

Question 16

You can build your own custom Docker image that is configured to

Answer 16

install the necessary packages or software.

Question 17

Think about bias as the gap between your predicted value and the actual value, whereas variance describes how dispersed your predicted values are.

Question 18

Classification Metrics

Answer 18

Accuracy
Precision
Recall
F1
AUC-ROC

Question 19

Regression Metrics

Answer 19

Mean squared error
R squared

Question 20

Accuracy

Answer 20

To calculate the model’s accuracy, also known as its score, add up the correct predictions and then divide that number by the total number of predictions.

Question 21

Problem with Accuracty

Answer 21

Although accuracy is a widely used metric for classification problems, it has limitations. This metric is less effective when there are a lot of true negative cases in your dataset. This is why two other metrics are often used in these situations: precision and recall.

Question 22

Precision

Answer 22

Precision removes the negative predictions from the picture. Precision is the proportion of positive predictions that are actually correct. You can calculate it by taking the true positive count and dividing it by the total number of positives.

Question 23

When the cost of false positives are high in your particular business situation, this can be a good metric

Answer 23

precision can be a good metric. Think about a classification model that identifies emails as spam or not. In this case, you do not want your model labeling a legitimate email as spam and preventing your users from seeing that email.

Question 24

Recall

Answer 24

recall (or sensitivity). In recall, you are looking at the proportion of correct sets that are identified as positive. Recall is calculated by dividing the true positive count by the sum of the true positives and false negatives. By looking at that ratio, you get an idea of how good the algorithm is at detecting, for example, cats.

Question 25

WHen to use recall

Answer 25

Think about a model that needs to predict whether a patient has a terminal illness or not. In this case, using precision as your evaluation metric does not account for the false negatives in your model. It is extremely important and vital to the success of the model that it not give false negative results. A false negative would be not identifying a patient as having a terminal illness when the patient actually does have a terminal illness. In this situation, recall is a better metric to use

Question 26

Area under the curve

Answer 26

In general, AUC-ROC can show what the curve for true positive compared to false positive looks like at various thresholds.

Question 27

Regression metric Mean squared error

Answer 27

The general purpose of mean squared error (MSE) is the same as the classification metrics. You determine the prediction from the model and compare the difference between the prediction and the actual outcome. The smaller the MSE, the better the model’s predictive accuracy.

Question 28

Regression metric r squared

Answer 28

R squared explains the fraction of variance accounted for by the model. It’s like a percentage, reporting a number from 0 to 1. When R squared is close to 1, it usually indicates that a lot of the variance in the data can be explained by the model itself.

Question 29

Two model deployment options

Answer 29

Self hosted API
Managed API (SageMaker is an example)

Question 30

SageMaker Deployment provides

Answer 30

Sagemaker provides the following:

Deployment with one click or a single API call
Automatic scaling
Model hosting services
HTTPS endpoints that can host multiple models

Question 31

SageMaker Asyncronous Deploy

Answer 31

SageMaker asynchronous inference is a capability in SageMaker that queues incoming requests and processes them asynchronously. This option is ideal for requests with large payload sizes (up to 1GB), long processing times (up to one hour), and near real-time latency requirements

Question 32

SageMaker Serverless Deploy

Answer 32

On-demand serverless inference is ideal for workloads that have idle periods between traffic spurts and can tolerate cold starts. It is a purpose-built inference option that you can use to deploy and scale ML models without configuring or managing any of the underlying infrastructure.

Question 33

Four Key principals of MLOps

Answer 33

Version Control
Automation
CI/CD
Model Governance

Question 34

MLOps: SageMaker training jobs

Answer 34

SageMaker provides a training job feature to train models using built-in algorithms or custom algorithms.

Question 35

MLOps: SageMaker Experiments

Answer 35

Use SageMaker Experiments to experiment with multiple combinations of data, algorithms, and parameters, all while observing the impact of incremental changes on model accuracy.

Question 36

MLOps: SageMaker Processing Job

Answer 36

SageMaker Processing refers to the capabilities to run data pre-processing and post-processing, feature engineering, and model evaluation tasks on the SageMaker fully managed infrastructure.

Question 37

MLOps: SageMaker Model Registry

Answer 37

With SageMaker Model Registry you can catalog models, manage model versions, manage the approval status of a model, or deploy models to production.

Question 38

bias

Answer 38

(the gap between your predicted value and the actual value)

Question 39

\variance

Answer 39

(how dispersed your predicted values are)