Understanding Predictive Modeling

Question 1

What is oversampling?

Answer 1

disproportionately over-represents the event cases (for example, an equal number of events and non-events)

Question 2

What is a use case for oversampling?

Answer 2

This is typically done when the original data set is very large and the ratio of events to non-events is very small.

Question 3

Name four common data-related challenges

Answer 3

observational data
mixed measurement scales
high dimensionality
rare target events

Question 4

What is the curse of dimensionality?

Answer 4

High dimensionality limits your ability to explore and model the relationships among the variables.

Question 5

What are two issues associated with the curse of dimensionality?

Answer 5

The number of variables may affect computational performance more than the number of cases

Including more variables makes the values more spread out, making it difficult to identify relationships in the data

Question 6

What is a biased sample?

Answer 6

a sample that is produced by oversampling

Question 7

What is meant by observational data?

Answer 7

Data gathered for some purpose other than data analysis, i.e. operational data

Question 8

What are some common problems found in operational data?

Answer 8

errors
missing data
redundant variables

Question 9

Name three analytical challenges:

Answer 9

nonlinearities
interactions
model selection

Question 10

What is the process of choosing the model with the highest predictive value accuracy?

Answer 10

Model selection

Question 11

What does it mean to overfit the data?

Answer 11

Using an overly complex model that is too sensitive to peculiarities in the sample data set that cannot generalize well to new data

Question 12

What does it mean to underfit the data?

Answer 12

Underfitting the data occurs when the model is too simple and systematically misses the true features in the data.

Question 13

When is joint sampling appropriate?

Answer 13

When the target event is not rare

Question 14

What is joint sampling?

Answer 14

Joint sampling selects a representative sample of the data by randomly selecting input-target pairs.

Question 15

Which sampling method would you choose when your event is not rare?

Answer 15

Joint sampling

Question 16

Which sampling method would you choose when your event is rare?

Answer 16

Separate sampling (which is used to oversample the data)

Question 17

What sampling method creates a target-based sample is created by drawing samples separately based on the target outcome – that is, whether it is a non-event or an event?

Answer 17

Separate sampling

Question 18

What is the optimism principle?

Answer 18

The optimism principle states that when you assess the accuracy of a predictive model on the same data that was used to fit the model, you tend to get better assessment statistics than when you assess the model on other data.

Question 19

What does a large differences between the performance on the training and test data sets usually indicate?

Answer 19

overfitting / optimism bias

Question 20

What should you be concerned with as underlying model becomes more flexible and the data less plentiful?

Answer 20

overfitting

Question 21

Name two activities that increase the risk of overfitting?

Answer 21

variable selection methods and supervised input preparation (such as collapsing levels of nominal variables based on associations with the target)

Question 22

What is optimism bias?

Answer 22

Overfitting the data causing large differences between the performance on the training and test data sets

Question 23

What is assessing the performance of a model on new data that was not used to fit the original model referred to as?

Answer 23

honest assessment

Question 24

What are some methods of honest assessment?

Answer 24

Splitting the data, k-fold validation, and bootstrapping

Question 25

What is the simplest way to do an honest assessment of how well your model generalizes to new data?

Answer 25

Split your data into two data sets: a training data set and a validation data set

Question 26

How does splitting the data allow you to avoid optimism bias?

Answer 26

Because you are assessing your model on different cases than you used to fit the model, you avoid the optimism bias and get a valid assessment measurement.

Question 27

Which dataset is used to assess and compare models?

Answer 27

the test dataset

Question 28

Which dataset is used to fit the model?

Answer 28

the training dataset

Question 29

Which dataset is used for comparing, selecting, and tuning models?

Answer 29

the validation dataset

Question 30

Which dataset is used for the final assessment of a model?

Answer 30

the test dataset

Question 31

How much of the data is typically used as validation dataset?

Answer 31

one-fourth to one-half of the data

Question 32

Which sampling method will ensure that he training and validation sets have an equal percentage of events?

Answer 32

stratified random sampling

Question 33

What are strata?

Answer 33

non-overlapping groups

Question 34

Which data set is used to calculate the parameter estimates?

Answer 34

the training dataset

Question 35

What does adding the OUTALL option on a PROC SURVEYSELECT do?

Answer 35

The OUTALL option returns the initial data set, augmented by a flag to indicate selection in the sample

Question 36

How can you produce the same split of the data on each run of PROC SURVEYSELECT

Answer 36

Set a value greater than 0 using the SEED= option

Question 37

What PROC can be used to verify the stratification in a sample?

Answer 37

PROC FREQ