Machine Learning with Viya® 3.4® Lesson 6: Model Assessment and Deployment

Question 1

How would you add a challenger model to a pipeline comparison in Model Studio?

Answer 1

Select the model in its pipeline.

Question 2

Which assessment measure should be used to determine the champion model for predicting an interval target?

Answer 2

Average Square Error

Question 3

How would you identify which model has the best classification accuracy using C-statistic values?

Answer 3

The model with highest C-statistic value has the best performance.

Question 4

Which dataset partition will be used to select the champion model when using the default settings in Model Studio?

Answer 4

Validation

Question 5

What type of data is used during champion-challenger testing to compare the performance of the currently deployed model and a challenger model during the model deployment phase?

Answer 5

Champion-challenger testing compares performance on historic data during model deployment.

Question 6

What are the primary considerations for choosing an appropriate model selection statistic?

Answer 6

• business needs
• ​the prediction type
• the measurement level

Question 7

The confusion matrix is the foundation of which assessment plot?

Answer 7

the ROC chart

Question 8

A confusion matrix helps you classify which type of target?

Answer 8

Binary

Question 9

Which validation method would you recommend for a small dataset?

Answer 9

Cross-validation

Question 10

A cumulative lift chart shows that a machine learning model has a lift of 2.6 at a depth of 10%. What does this mean?

Answer 10

For the top 10% of cases, the machine learning model captures 2.6 times more primary outcome cases than a random model.

Question 11

What model fit statistics are recommended for a decision prediction?

Answer 11

accuracy or misclassification

KS

Question 12

What are two commonly used performance statistics for estimate predictions?

Answer 12

Schwarz’s Bayesian Criterion (SBC)

Weighted Square Error?

Question 13

What assessment measure would you use to assess the probability of a customer responding to a targeted ad campaign?

Answer 13

the Gains chart

Question 14

What is another term for gains chart?

Answer 14

Cumulative Percentile Hits chart

Question 15

What is another term for a Cumulative Lift chart?

Answer 15

a Gains chart

Question 17

What is a cumulative lift chart?

Answer 17

A lift chart indicates how well the model did as compared to no model. The lift is the ratio between the result predicted by the model and the result using no model.

cumulative lift plotted as a percent on the vertical axis

Question 18

What’s the calculation for the lift for a given percentile when evaluating a Cumulative Captured Response (Gains) Chart?

Answer 18

Divide the Model Response Rate by the Random Response Rate:

Lift = (P,M) = CPH(P,M) / P

where P is a given percentile

Question 19

What are two things you can investigate in an ICE plot?

Answer 19

Subgroups and interactions among model variables.

Question 20

What do level differences in an ICE plot suggest?

Answer 20

group effects

Question 21

What does an intersecting slope in an ICE plot indicate?

Answer 21

Interactions between the plot variable and one or more additional model variables

Question 22

What are two things to look for in an ICE plot?

Answer 22

1. intersecting slopes
2. level differences

Question 23

Which machine learning model is the easiest to interpret?

Answer 23

Decision trees are highly interpretable because they are based on English rules, which are rules that use Boolean logic.

Question 24

Which dataset partition assists in comparing possible models?

Answer 24

Validation

Question 25

Which dataset partition generates the possible models?

Answer 25

Training data

Question 26

Which dataset partition should be used to assess how the final model generalizes to new data?

Answer 26

Test data

Question 27

What does a PD plot show you?

Answer 27

How model inputs affect the model’s performance

Question 28

What does publishing a model do?

Answer 28

Publishing a model is used to makes score available in CAS, Teradata or Hadoop.

Question 29

You want to predict the rankings of a target variable and have built multiple models. Which selection statistic should be used to compare your models?

Answer 29

a ROC Index or Gini Coefficient

Question 30

Which model fit statistics are commonly used for ranking predictions?

Answer 30

ROC Index

Gini Coefficient

Question 31

How do you register a model in Model Studio?

Answer 31

Select Register Models from the Project Pipeline menu on the Pipeline Comparison tab.

Question 32

What does registering a model do?

Answer 32

Registering the model in Model Studio makes the model available in Model Manager.

Question 33

Which of the following statements is true about an ROC chart?

a. The selection value of each point is displayed on the chart.
b. Each point on the chart corresponds to a specific fraction of the sorted data.
c. True positives are on the x axis.
d. For a perfect model, the ROC curve is a straight line from the bottom left corner to the top right corner of the plot.

Answer 33

In an ROC chart, each point corresponds to a specific fraction of the sorted data.

Question 34

What kind of model can you import using a Score Code Import node?

Answer 34

A Score Code import node can only import either an ASTORE model or a single DATA step file model.

Question 35

What must you do before you can score and manage a model in Model Studio?

Answer 35

Register the model in Model Studio.

Question 36

In model comparison, the best model has the highest value of which measure?

Answer 36

Sensitivity

Question 37

What is the true positive rate referred to as?

Answer 37

Sensitivity

Question 38

What is sensitivity?

Answer 38

The true positive rate

Question 39

Calculate Sensitivity:

Answer 39

divide the true positive decisions by the total number of known primary cases

Specificity = TruePositive / TruePositive + FalsePositive

Question 40

What is the true negative rate referred to as?

Answer 40

Specificity

Question 41

Calculate Specificity:

Answer 41

divide the true negative decisions by the total number of known secondary cases

Specificity = TrueNegative / TrueNegative + FalsePositive