7 - Tools for Machine Learning

Question 1

What was the initial proportion of incorrect decisions in the prior authorization process?

Answer 1

50 percent

Question 2

What is the current proportion of incorrect decisions after model improvement?

Answer 2

30 percent

Question 3

In an ideal world, what percentage of prior authorization decisions does Kamala hope to achieve?

Answer 3

0 percent

Question 4

What approach does David prefer when building models in data science?

Answer 4

Start with a simple model

Question 5

What is a residual in predictive modeling?

Answer 5

The difference between the true value of the outcome and what the model predicted

Question 6

What was the average absolute difference between the predicted and actual health care expenditure in the initial model?

Answer 6

$12,000

Question 7

What features did the linear regression model initially include?

Answer 7

• Demographic data
• Medical claims
• Prior authorization report

Question 8

What were the top features selected by the linear regression model?

Answer 8

• Prior history of back pain
• Prior medication usage
• Age

Question 9

How did the first team improve the model’s performance?

Answer 9

By examining residuals and adding more job-related features

Question 10

What was the average residual after adding new features related to patients’ jobs?

Answer 10

$9,000

Question 11

What is the purpose of interaction terms in feature engineering?

Answer 11

To represent the relationship between two main terms

Question 12

What transformation techniques did the second team apply to improve model fitting?

Answer 12

• Square transformation
• Square root transformation
• Log transformation

Question 13

What is a general rule of thumb when including polynomial terms in a regression model?

Answer 13

Always include the lower-order terms

Question 14

What strategy did the third team use to address the issue of unrepresentative training data?

Answer 14

Weighted regression

Question 15

What are outliers in the context of health care expenditure?

Answer 15

Patients who incur significantly higher health care spending than the majority

Question 16

What is the effect of using least squares linear regression on outliers?

Answer 16

It is affected by outliers more than by non-outliers

Question 17

What was the goal of the hackathon organized by David?

Answer 17

To improve the performance of the model predicting health care expenditure

Question 18

What did the team find regarding the residuals of patients with different job types?

Answer 18

Higher residuals for patients in manual labor jobs

Question 19

Fill in the blank: A _______ is a special term used to represent the relationship between two main effect variables.

Answer 19

interaction term

Question 20

True or False: The model’s performance improved after adding features about patients’ jobs.

Answer 20

True

Question 21

What are outliers in the context of health care expenditure?

Answer 21

Outliers are patients who incur significantly higher health care expenditures compared to the majority, such as those with hundreds of thousands of dollars in back pain-related spending over 3 years.

Question 22

Why is least squares linear regression problematic when dealing with outliers?

Answer 22

Least squares is affected by outliers more than by nonoutliers, which can skew predictions and result in worse predictions for nonoutliers.

Question 23

Explain the concept of squaring residuals in least squares regression.

Answer 23

Squaring residuals means that a residual of 4 is considered four times as bad as a residual of 2, disproportionately affecting the model’s predictions.

Question 24

What is least absolute deviation?

Answer 24

Least absolute deviation is a method that uses the absolute value of residuals instead of squaring them, making it less sensitive to outliers.

Question 25

What is quantile regression?

Answer 25

Quantile regression focuses on identifying specific quantiles, such as the median, which are robust to outliers.

Question 26

How does mean wealth differ from median wealth when outliers are present?

Answer 26

Mean wealth can be significantly affected by outliers, while median wealth remains unchanged, providing a more stable measure.

Question 27

What is K-nearest neighbor (K-NN) regression?

Answer 27

K-NN regression predicts outcomes based on the K most similar observations, using local information instead of the entire data set.

Question 28

How is the parameter K defined in K-NN regression?

Answer 28

K represents the number of observations used to make a prediction, such as K = 20 meaning the model looks at the 20 most similar data points.

Question 29

What is the common approach to define similarity in K-NN?

Answer 29

The most common approach is to calculate the Euclidean distance between data points.

Question 30

What are the advantages of K-NN regression?

Answer 30

K-NN can handle nonlinear relationships and does not assume linearity, making it flexible for various types of data.

Question 31

What is Naive Bayes?

Answer 31

Naive Bayes is a probabilistic machine learning model used for classification problems, based on Bayes' theorem.

Question 32

What assumption does Naive Bayes make about features?

Answer 32

Naive Bayes assumes that each feature is independent of all other features in the data set.

Question 33

What is the primary output of a Naive Bayes model?

Answer 33

Naive Bayes predicts the probability of a binary outcome, such as whether health expenditure will exceed $20,000.

Question 34

What is the training process for Naive Bayes?

Answer 34

The training process involves estimating prior probabilities of each class and conditional probabilities of each feature given the class.

Question 35

What is a Classification and Regression Tree (CART)?

Answer 35

CART modeling divides the population into smaller subpopulations and makes predictions on those smaller groups.

Question 36

How does the CART algorithm work?

Answer 36

CART starts with the entire population and repeatedly searches for the best variable and value to split the population until stopping rules are met.

Question 37

What are the advantages of CART modeling?

Answer 37

* No assumptions required about features * Can predict various types of variables * Less sensitive to outliers * Can handle missing data * Easily interpretable

Question 38

What are the limitations of CART modeling?

Answer 38

CART models can be sensitive to small changes in data, do not produce coefficients like regression models, and require careful evaluation of variable importance.

Question 39

What is a random forest?

Answer 39

A random forest is a collection of multiple CART models built from random samples of the data.

Question 40

What is the significance of randomness in random forests?

Answer 40

Randomness is applied in creating random samples of data and in selecting input variables at specific nodes.

Question 41

What is the goal of boosting, bagging, and ensembling?

Answer 41

These techniques combine multiple machine learning models to improve predictive performance.

Question 42

What are ensembling methods?

Answer 42

Techniques for combining the predictions of several models into a single, more accurate prediction. ## Footnote Ensembling methods can improve the robustness of predictions.

Question 43

What is model stacking?

Answer 43

A method where different models are created and their predictions are used as input variables to a new model for final predictions. ## Footnote The first models are known as level 1 models, and the final model is called the level 2 model.

Question 44

What is a level 1 model?

Answer 44

The initial models created in model stacking that serve as inputs to the level 2 model.

Question 45

What is the function of a level 2 model in stacking?

Answer 45

To combine predictions from level 1 models to produce a final result.

Question 46

What is bagging?

Answer 46

A special case of ensembling called bootstrap aggregation that involves resampling data sets from the original data set. ## Footnote Bagging can improve prediction accuracy and help avoid overfitting.

Question 47

What does bootstrapping refer to?

Answer 47

Making several data sets from the original data set by resampling observations.

Question 48

What is gradient boosted machine learning (GBML)?

Answer 48

A technique where models learn by giving misclassified observations more weight in the next training iteration.

Question 49

What is the purpose of boosting in machine learning?

Answer 49

To improve predictions by focusing on difficult-to-predict observations while maintaining performance on easier ones.

Question 50

Name two commonly used boosting algorithms.

Answer 50

* Adaboost * Arcboost

Question 51

What models were included in the ensemble described by David?

Answer 51

* Linear regression model * K-NN model * Regression tree model

Question 52

What was the average residual achieved by the stacked ensemble?

Answer 52

$5,000

Question 53

True or False: The best individual model had an average residual of $8,500 before ensembling.

Answer 53

True

Question 54

Fill in the blank: The process of combining multiple models to improve prediction accuracy is called _______.

Answer 54

ensembling

Question 55

How does ensembling improve model performance?

Answer 55

By leveraging the strengths of different models to produce a more accurate and robust prediction.

Question 56

What analogy did David use to explain the benefits of ensembling?

Answer 56

A trivia game show partnership where two individuals complement each other's knowledge.

Question 57

What should be examined when analyzing the residuals of a model?

Answer 57

Patterns in the residuals

Question 58

What is a potential benefit of using metalearning methods like boosting, bagging, and ensembling?

Answer 58

Improved model performance.

Question 59

What is a common challenge when using K-nearest neighbor approaches?

Answer 59

Determining whether it is superior to parametric regression approaches in specific scenarios.

Question 60

What type of terms might be included in modeling based on data knowledge?

Answer 60

* Polynomial terms * Interaction terms