Section 1: General Model Building Steps

Question 1

What is Descriptive Analytics?

Answer 1

Focus: What happened in the past?
Aim: to “describe” or interpret observed trends by identifying relationships between variables

Question 2

What is Predictive Analytics?

Answer 2

Focus: What will happen in the future?
Aim: to make accurate “predictions”

Question 3

What is Prescriptive Analytics?

Answer 3

Focus: The impacts of different “prescribed” decisions (assumptions)
Aim: to answer the “what if?” and “what is the best course of action?” questions

Question 4

Why do we need relevant data?

Answer 4

need to ensure that the data is unbiased (i.e., representative of the environment where the model will operate)

Question 5

What is Random Sampling?

Answer 5

“randomly” draw observations from the underlying population without replacement. each record is equally likely to be sampled

Question 6

What is Stratified Sampling?

Answer 6

Divide the underlying population into a number of non-overlapping “strata” non-randomly, then randomly sample a set number of observations from each stratum (this helps you get a more representative sample)

Question 7

What is a special case of Stratified Sampling?

Answer 7

Systematic sampling: draw observations according to a set pattern; no random mechanism controlling which observations are sampled

Question 8

What is Granularity?

Answer 8

referees to how precisely a variable is measured (i.e., level of detail for the information contained by the variable)

Question 9

What are examples of data quality issues? (name at least 3)

Answer 9

1. Reasonableness (ex. variables such as age, time, and income are non-negative)
2. Consistency (ex. same measurement unit for numeric variables, same coding scheme for categorical variables)
3. Sufficient Documentation (ex. clear description of each variable)
4. Personally identifiable info (PII)
5. Variables with legal/ethical concerns
6. Target Leakage (on other flashcard)

Question 10

what is Target Leakage?

Answer 10

when predictors in a model “leak” information about the target variable that would not be available when the model is deployed in practice

Question 11

Univariate exploration tools (Numeric and categorical

Answer 11

Numeric - mean, median, variance, min/max. visuals: histograms, boxplots

categorical - class, frequencies. visuals: bar charts

Question 12

Bivariate exploration tools (numericXnumeric, numericXcategorical, categoricalXcategorical)

Answer 12

NumericXNumeric - correlations. visuals: scatterplots

NumericXCategorical - mean/median of numeric variable split by categorical variable. visuals: split boxplots, histograms

CategoricalXCategorical - 2 way frequency table. visuals: bar charts

Question 13

What are the three common data issues for numeric variables?

Answer 13

1. Highly correlated predictors
2. Skewness (esp. right skewness due to outliers
3. Should they be converted to a factor?

Question 14

Highly correlated predictors (problems)

Answer 14

a) difficult to separate out the individual effects of different predictors on the target variable
b) For GLMs, coefficients become widely varying in sign and magnitude, difficult to interpret

Question 15

Highly correlated predictors (solutions)

Answer 15

a) drop one of the strongly correlated predictors
b) Use PCS to compress the correlated predictors into a few PCs

Question 16

Skewness (problems)

Answer 16

Extreme values:
a) exert a disproportionate effect on model fit
b) distort visualizations (e.g., axes expanded inordinately to take care of outliers

Question 17

Skewness (solutions)

Answer 17

a) transformations to reduce right skewness (Log, Square root)
b) options to handle outliers (Remove, Keep, Modify, Using robust model forms

Question 18

A common issue for categorical predictors

Answer 18

Sparse levels (reduce the robustness of models and may cause overfitting)

Question 19

What is an interaction?

Answer 19

relationship between a predictor and the target variable depends on the value/level of another predictor

Question 20

Regression problems are used when a target is __________.

Answer 20

numeric (quantitative)

Question 21

Classification problems are used when a target is __________.

Answer 21

categorial (qualitative)

Question 22

metrics on a training set measures

Answer 22

the goodness of fit of to the training data

Question 23

metrics on the test set measures

Answer 23

the prediction performance on new, unseen data

Question 24

What does a loss function do?

Answer 24

captures the discrepancy between the actual and predicted values for each observation of the target variable

Question 25

examples of loss functions

Answer 25

a) Square Loss (most common for numeric targets)
b) Absolute loss
c) Zero-one loss (mostly for categorical variables)

Question 26

Confusion matrices

Answer 26

table showing prediction versus reference (actual) counts

Question 27

Accuracy

Answer 27

proportion of correctly classified obs

Question 28

classification error rate

Answer 28

proportion of misclassified obs.

Question 29

Sensitivity

Answer 29

proportion of +ve obs. correctly classified as +ve

Question 30

Specificity

Answer 30

proportion of -ve obs. correctly classified as -ve

Question 31

Precision

Answer 31

proportion of +ve predictions truly belonging to +ve class

Question 32

accuracy weighted average relation

Answer 32

accuracy = n_/n (specificity) + n+/n (sensitivity)

Question 33

common uses of CV

Answer 33

a) Model assessment (able to preform without a test set)
b) Hyperparameter tuning

Question 34

Hyperparameters

Answer 34

parameters with values supplied in advance; not optimized by the model fitting algorithm

Question 35

considerations when selecting the best model

Answer 35

a) prediction performance
b) interpretability
c) ease of implementation

Question 36

Unbalanced data for binary targets (problems)

Answer 36

a) classifier implicitly places more weight on the majority class and tries to fit those observations well, but the minority class may be the +ve class
b) a high accuracy can be deceptive

Question 37

Unbalanced data for binary targets (solution)

Answer 37

a) Undersampling - keep all obs. from minority class but draw fewer obs. from majority class (con: you have less data)
b) Oversampling - keep all obs. from majority class, but draw more observations from minority class (con: more data, computational burden)

Question 38

Effects of undersampling and oversampling on model results

Answer 38

+ve class becomes more prevalent in the balanced data –> predicted probabilities for +ve class will increase –> for a fixed cutoff, sensitivity increases but specificity decrease

Question 39

what is overfitting?

Answer 39

Model is trying too hard to capture not only the signal, but also the noise specific to the training data

Question 40

What indicates overfitting?

Answer 40

Small training error, but large test error

Question 41

What problems come from overfitting?

Answer 41

An overfitted model fits training data well, but does no generalize well to new, unseen data (poor predictions)

Question 42

relationship between complexity, bias, variance, training error and test error.

Answer 42

as complexity increases, variance increases, bias decreases, training error decreases, and the test error has a U-shape

Question 43

mathematical definition of bias

Answer 43

difference between the expected value of the predication and the true value

Question 44

mathematical definition of variance

Answer 44

amount of variability of the prediction

Question 45

significance of Bias in PA

Answer 45

part of the test error caused by the model not being flexible enough to capture the signal (underfitting)

Question 46

significance of Variance in PA

Answer 46

part of the test error caused by the model being too complex (overfitting)

Question 47

dimensionality applicability

Answer 47

specific to categorical variables

Question 48

granularity applicability

Answer 48

applies to both numeric and categorical variables

Question 49

dimensionality comparability

Answer 49

two categorical variables can always be ordered by dimension

Question 50

granularity compatibility

Answer 50

not always possible to order two variables by granularity

Question 51

what is the aim of model validation?

Answer 51

to check that the selected model has no obvious deficiencies and the model assumptions are largely satisfied

Question 52

for a “nice” GLM, the deviance residuals should:

Answer 52

1) (purely random) have no systematic patterns
2) (homoscedasticity) have approximately constant variance upon standardization
3) (normality) be approximately normal (for most target distributions)

Question 53

what are the two validation methods based on the test set?

Answer 53

1) predicted vs actual values of the target - the two sets of values should be close (can check this quantitatively or graphically)
2) benchmark model - show that the recommended model outpreforms a benchmark model, if one exists (e.g., intercept-only GLM, purely random classifier), on the test set

Question 54

Next steps after model validation

Answer 54

1) Adjust the business problem - changes in external factors may cause initial assumptions to shift, so we modify business problem to incorporate the new conditions
2) consult with the subject matter experts - seek validation of model results from external subject matter experts
3) gather additional data - enlarge training data with new obs. and/or variables, and retain the model to improve robustness
4) apply new types of models
5) refine existing models
6) field test proposed model