6 - Building Your First Model

Question 1

What is prior authorization?

Answer 1

The process in which an insurance company requires a physician to get clearance for reimbursement before providing a service or procedure.

Prior authorization is used to ensure reimbursement for medically necessary services and to control costs.

Question 2

Why do insurers use prior authorizations?

Answer 2

To ensure reimbursement for medically necessary services and to control costs by adding friction to the system for physicians and patients.

Question 3

What is a significant issue with current prior authorization processes?

Answer 3

There are too many false positives and false negatives regarding surgery approvals, leading to higher costs and complications for patients.

Patients who need surgery may be denied, while those who don’t need it may be approved.

Question 4

What role does data science play in improving prior authorization decisions?

Answer 4

Data science can help empower decision-makers with data to make better choices regarding surgery approvals and denials.

Question 5

What is the challenge faced by reviewers in the prior authorization process?

Answer 5

Reviewers have difficulty predicting whether a patient would benefit from nonsurgical treatment options.

Question 6

What is the main goal of the model that David and the team want to build?

Answer 6

To accurately determine which patients would benefit from nonsurgical treatment options for back pain.

Question 7

What is the difference between prediction and explanation in data modeling?

Answer 7

Prediction focuses on accurately forecasting outcomes, while explanation seeks to understand the variables leading to those outcomes.

Question 8

What is meant by defining the outcome in a predictive model?

Answer 8

The outcome is the quantity that the model aims to predict, such as a patient’s chance of success on a nonsurgical treatment plan.

Question 9

How did the team propose to measure success for nonsurgical treatment?

Answer 9

By predicting health care utilization related to back pain, such as medications and physical therapy, rather than just recovery from pain.

Question 10

What potential bias did Jenna raise regarding health care utilization predictions?

Answer 10

Patients may have lower utilization due to not seeking care, which does not necessarily mean their back pain is well managed.

Question 11

What is feature engineering?

Answer 11

The process of creating variables to feed into a predictive model.

Question 12

What type of data will the model use?

Answer 12

Tabular data that can be organized in spreadsheet format.

Question 13

What temporal restriction is important when building the model?

Answer 13

The model should use only data collected before the prior authorization to ensure access to the relevant data during model application.

Question 14

What are catastrophic events mentioned as potential outcomes?

Answer 14

Hospitalizations or deaths from complications, which are significant for both patient welfare and cost considerations.

Question 15

What time frame did Jenna suggest for measuring health care expenditure?

Answer 15

Three years after the prior authorization request for surgery.

Question 16

What is the significance of defining specific outcomes for the model?

Answer 16

Specific outcomes help focus the model on what is actually being measured, improving its accuracy and relevance.

Question 17

True or False: The model can use any type of data for predictions.

Answer 17

False.

The model requires data to be in a tabular format, which excludes unstructured text data.

Question 18

What is the primary model outcome discussed in the meeting?

Answer 18

Health care utilization related to back pain.

Question 19

What is the potential risk of using health care expenditure as an outcome measure?

Answer 19

It may not accurately reflect a patient’s condition if they do not seek care, leading to misleading conclusions.

Question 20

What did David suggest about the prediction window for health care utilization?

Answer 20

To try a range of prediction windows from one month to one year to see how model accuracy changes.

Question 21

What was Kamala’s concern about older patients in the model?

Answer 21

Older patients may die soon after submitting a request, potentially skewing health care expenditure predictions.

Question 22

What is feature engineering?

Answer 22

The process by which we create variables to feed into our model from other data sources.

Question 23

How can clinician’s notes be utilized in modeling?

Answer 23

Feature engineering techniques from natural language processing can convert free text into a table of numbers.

Question 24

What is optical character recognition used for?

Answer 24

To extract numerical and text data from scanned PDFs, turning them into machine-readable format.

Question 25

What must be considered before using scanned PDF data in modeling?

Answer 25

Exhausting all existing tabular data before spending time on scanned PDFs.

Question 26

What is an example of creating a feature from claims data?

Answer 26

Counting the total number of times a patient was hospitalized in the year before the prior authorization submission.

Question 27

What are some metrics that can be created from existing claims data?

Answer 27

* Total number of hospital visits * Total number of medications prescribed * Average number of claims per month * Rate of change in prescription drug use

Question 28

Is there a specific number of features needed for a model?

Answer 28

There’s no set rule; the goal is to find the most highly predictive features.

Question 29

What is the role of the model in feature selection?

Answer 29

To identify statistical relationships between variables and find the strongest predictors.

Question 30

True or False: A model can become confused if there are too many features.

Answer 30

True.

Question 31

What is feature selection?

Answer 31

The process of identifying which features to include in your model.

Question 32

What can happen if irrelevant variables are included in a model?

Answer 32

It can add more noise, resulting in worse predictions.

Question 33

What is a common filtering method for variable selection?

Answer 33

Selecting variables that have a high correlation with the outcome variable.

Question 34

What are wrappers in feature selection?

Answer 34

Methods like forward selection, backward selection, and stepwise selection that examine how useful each feature is.

Question 35

What does forward selection do?

Answer 35

Adds the most important variable first and continues adding until no significant variables are left.

Question 36

What is an embedded approach in feature selection?

Answer 36

A method that automatically includes variable selection as part of the model’s optimization algorithm.

Question 37

What is the LASSO algorithm?

Answer 37

A variation of regression that minimizes the sum of the squares of the residuals plus the sum of the absolute values of the regression coefficients.

Question 38

What is the main goal of model training?

Answer 38

To feed the model data and allow it to learn patterns.

Question 39

What does model testing involve?

Answer 39

Showing previously unseen data to the model and evaluating its predictions.

Question 40

How does a model identify statistical patterns?

Answer 40

Using statistical associations and correlations from the training data.

Question 41

What is the concept of fitting a trend line in modeling?

Answer 41

Finding the optimal values of parameters that best fit the data points.

Question 42

What is the equation of a line used in linear prediction models?

Answer 42

Y = mX + b, where m is the slope and b is the y-intercept.

Question 43

What happens if the model encounters a data point that is very different from the training data?

Answer 43

It may perform poorly due to lack of representative training data.

Question 44

What happens if the training data is not representative of real-life situations?

Answer 44

The model may perform poorly in unexpected scenarios ## Footnote For example, if a model is trained only on pictures of cats and dogs, it may struggle with pictures of hippos.

Question 45

How can we ensure our model is robust?

Answer 45

By making our training data as representative as possible and filtering out dissimilar data points ## Footnote This helps the model learn from a diverse array of cases.

Question 46

What is the trade-off when deciding how much data to use for training versus testing?

Answer 46

Using more data for training improves performance on the training set but complicates performance evaluation on the test set ## Footnote A common rule of thumb is to use 20 to 40 percent for testing.

Question 47

How does the size of the data set affect the amount reserved for testing?

Answer 47

In huge data sets, just 1 percent may be sufficient as a representative test set ## Footnote This allows for effective evaluation without needing a large portion of the data.

Question 48

What should be considered when handling new patient data?

Answer 48

The model's ability to generalize to new populations that may differ from the training population ## Footnote For example, coal miners may have different health care utilization patterns than the general population.

Question 49

Can we retrain the model if it performs poorly on testing?

Answer 49

Yes, but tweaking the model after testing can lead to overfitting ## Footnote This is similar to studying for a practice test and not performing well on the actual test.

Question 50

What is overfitting?

Answer 50

Optimizing the model to perform well on test data at the expense of generalizing to other data sets ## Footnote This can lead to poor performance when applied to new, unseen data.

Question 51

What is the benefit of using cross-validation?

Answer 51

It reduces the likelihood of overfitting by testing the model on multiple validation sets ## Footnote This involves splitting data into folds and validating across them.

Question 52

What are the three main levers to improve model performance?

Answer 52

* Changing the data used by the model * Changing the type of model * Tuning hyperparameters ## Footnote These adjustments can significantly affect prediction accuracy.

Question 53

What is a hyperparameter?

Answer 53

A setting knob on a model that can be tuned to optimize performance ## Footnote For example, the time spent studying a page in a textbook can be viewed as a hyperparameter.

Question 54

How do we decide when the model is performing well enough?

Answer 54

It depends on how the model will be used and the required accuracy for practical applications ## Footnote This is often determined by stakeholder needs.

Question 55

What is the mean absolute error?

Answer 55

It is the average absolute difference between predicted and actual expenditures ## Footnote This metric helps quantify model performance in a comprehensible way.

Question 56

What is a limitation of mean absolute error?

Answer 56

It treats large and small errors equally ## Footnote This means that a $200 error is penalized the same as a $100 error, potentially masking larger discrepancies.

Question 57

What is mean absolute error?

Answer 57

The mean absolute error is the average of the absolute values of the differences between predicted and actual values ## Footnote It provides a single number indicating how much the model is off, on average.

Question 58

What is a limitation of mean absolute error?

Answer 58

Large errors and small errors are treated the same, meaning an error of $200 is penalized only twice as much as an error of $100.

Question 59

What is mean squared error?

Answer 59

Mean squared error is the square of the difference between predicted and actual values, penalizing larger discrepancies more than smaller ones.

Question 60

How does squared error compare to absolute error?

Answer 60

If the error is $100, the squared error is $10,000; if the error is $200, the squared error is $40,000.

Question 61

What is the Brier score?

Answer 61

The Brier score is a mean squared error used to compare predicted probabilities to actual binary outcomes (0 or 1).

Question 62

What does a lower Brier score indicate?

Answer 62

A lower Brier score indicates better model performance.

Question 63

What does it mean for a model to be well calibrated?

Answer 63

A well-calibrated model's predicted probabilities behave like true probabilities.

Question 64

How can you check the calibration of predicted probabilities?

Answer 64

By comparing predicted probabilities to the actual frequency of outcomes within a group of data points.

Question 65

What is thresholding in the context of binary outcome models?

Answer 65

Thresholding involves picking a probability threshold and designating predicted probabilities above it as 'yes' and below as 'no.'

Question 66

What common threshold do model builders often use?

Answer 66

Many model builders arbitrarily pick a threshold of 0.5.

Question 67

What assumption does a 0.5 threshold make?

Answer 67

It assumes that false negatives are equally undesirable as false positives.

Question 68

Why might a threshold of 0.5 be inappropriate in certain models?

Answer 68

In scenarios where false positives are worse than false negatives, a threshold closer to 0 may be more suitable.

Question 69

Who should determine the costs of false positives and false negatives?

Answer 69

The person using the model should communicate the relative costs of false positives and false negatives.

Question 70

What are the four important numbers for measuring classifier performance?

Answer 70

True positives, true negatives, false positives, false negatives.

Question 71

What is sensitivity in model performance metrics?

Answer 71

Sensitivity, or true positive rate, is the percentage of all positives the model identifies.

Question 72

What is specificity in model performance metrics?

Answer 72

Specificity, or true negative rate, is the percentage of all negatives the model identifies.

Question 73

What is positive predictive value?

Answer 73

The probability that a predicted positive is a true positive, calculated as true positives divided by true positives plus false positives.

Question 74

What is negative predictive value?

Answer 74

The probability that a predicted negative is a true negative, calculated as true negatives divided by true negatives plus false negatives.

Question 75

What key questions should be asked when building prediction models?

Answer 75

Questions regarding population, outcome variable, feature selection, training, and model performance should be considered.

Question 76

What should be considered in feature selection?

Answer 76

Most predictive features, steps taken for selection, importance techniques, and feature engineering techniques.

Question 77

What strategies can be implemented to avoid overfitting?

Answer 77

Regularization techniques, cross-validation, or early stopping.

Question 78

What performance metrics should be used for model evaluation?

Answer 78

Metrics appropriate for the use case, including sensitivity, specificity, and predictive values.

Question 79

What is the importance of model evaluation techniques?

Answer 79

They assess the performance of trained models and validate generalizability.