Section 2: Specific Types of Models

Question 1

Independence Assumption for LMs/GLMs

Answer 1

Given the predictor values, the observations of the target variable are independent (same for both LMs/GLMs)

Question 2

Target Distribution assumptions for LMs and GLMs

Answer 2

LMs: Given the predictor values, the target variable follows a normal distribution
GLMs: Given the predictor values, the target distribution is a member of the linear exponential family

Question 3

Mean assumptions for LMs and GLMs

Answer 3

LMs: the target mean directly equals the linear predictor (mu = B0 + B1X1+ … + BpXp)
GLMs: A function (“link”) of the target mean equals the linear predictor ( g(mu) = n)

Question 4

Variance assumptions for LMs and GLMs

Answer 4

LM: constant, regardless of the predictor values
GLM: varies with mu and the predictor values

Question 5

what is a target distribution?

Answer 5

A distribution in the linear exponential family; choose one that aligns with the characteristics of the target

Question 6

Important considerations when choosing a link function

Answer 6

1) ensure the predictions match the range of values of the target mean
2) ensure ease of interpretation (log link)
3) canonical links make convergence more likely

Question 7

Common distributions

Answer 7

Normal, Binomial, Poisson, Gamma, inverse gaussian, tweedie

Question 8

Normal distribution variable type and common link

Answer 8

real-valued with a bell-shaped dist.

identity link

Question 9

Binomial variable type and common link

Answer 9

Binary (0/1)

logit link

Question 10

Poisson variable type and common link

Answer 10

Count (>=0, integers)

Log link

Question 11

Gamma, inverse gaussian variable type and common link

Answer 11

positive, continuous with right skew

log link

Question 12

tweedie variable type and common link

Answer 12

> = 0, continuous with a large mass at zero

log link

Question 13

methods for handling non-monotonic relations

Answer 13

GLMs, in their basic form, assume that numeric predictors have a monotonic relationship with the target variable

1) polynomial regression
2) binning
3) piecewise linear functions

Question 14

polynomial regression

Answer 14

add polynomial terms to the model equation

pros: can take care of more complex relationships between the target variable and predictors. the more polynomial terms included, the more flexible the fit

cons: a) coefficients become harder to interpret (all polynomial terms move together) b) usually no clear choice in highest power; can be tuned by CV

Question 15

Binning

Answer 15

“bin” the numeric variable and convert it into a categorical variable with levels defined as non-overlapping intervals over the range of the original variable

pros: no definite order among the coefficients of the dummy variables corresponding to different bins -> target mean can vary highly irregularly over the bins

cons: a) usually no clear choice of the no. of bins and the associated boundaries b) results in a loss of information (exact values of the numeric predictor gone)

Question 16

adding piecewise linear functions

Answer 16

add features of the form (X-c)+

pros: a simple way to allow the relationship between a numeric variable and the target mean to vary over different intervals

cons: usually no clear choice of the break points

Question 17

Handling categorical predictions - binarization

Answer 17

how it works: a categorical predictor becomes a collection of dummy (binary) variables indicating one and only one level and the dummy variables serve as predictors in model equation

Question 18

baseline level

Answer 18

the level at which all dummy variables equal 0

R’s default: the alpha-numerically first level
Good practice: reset it to the most common level

Question 19

interactions

Answer 19

need to “manually” include interaction terms of the product form XiXj, where the coefficient of Xi will vary with the value of Xj

Question 20

interpretation of coefficients

Answer 20

coefficient estimates capture the effect (magnitude + direction) of features on the target mean

Question 21

p-value statistical significance

Answer 21

the smaller the p-value, the more significant the feature

Question 22

Offset: form of target variable and how they affect the mean/var of the target

Answer 22

form: aggregate (e.g., total number of claims in a group of similar policyholders)

affect: target mean is directly proportional to exposure

Question 23

Weights: form of target variable and how they affect the mean/var of the target

Answer 23

form: average (e.g., average number of claims in a group of similar policyholders)

affect: variance is inversely related to exposure. observations with a larger exposure will play a more important role in model fitting

Question 24

stepwise selection

Answer 24

sequentially add/drop features, one at a time, until there is no improvement in the selection criterion

Question 25

Forward selection

Answer 25

start with intercept-only model, add variables until no improvement in model

tends to produce a simpler model

Question 26

backward selection

Answer 26

starts with full model, drop variables until no improvement

Question 27

selection criteria based on penalized likelihood

Answer 27

idea: prevent overfitting by requiring an included/retained feature to improve model fit by at least a specified amount

common choices are AIC and BIC

Question 28

AIC

Answer 28

AIC = -2l + 2(p+1)

penalty per parameter = 2

Question 29

BIC

Answer 29

BIC = -2l + ln(n)*(p+1)

penalty per parameter = ln(n)

Question 30

AIC vs BIC

Answer 30

for both, the lower the value, the better

BIC is more conservative and results in simpler models

Question 31

Manual Binarization

Answer 31

convert factor variables to dummy variables manually before running stepwise selection

pros: to be able to add/drop individual factor levels that are statistically significant/insignificant with regards to the baseline

cons: more steps in stepAIC() procedure; possibly non-intuitive results (e.g., only a few levels of a factor are retained)

Question 32

regularization

Answer 32

idea: reduce overfitting by shrinking the size of the coefficient estimates, especially those of non-predictive features

Question 33

how does regularization work?

Answer 33

to optimize training log-likelihood (equivalently, training deviance) adjusted by a penalty term that reflects the size of the coefficients, i.e., to minimize
deviance + regularization penalty
the formulation serves to strike a balance between goodness of fit and model complexity

Question 34

common forms of penalty term

Answer 34

1) Lasso - some coef. may be zero
2) ridge regression - none reduced to zero
3) elastic net - some coef. may be zero

Question 35

Two hyperparamters

Answer 35

1) Lambda: regularization (a.k.a. shrinkage) parameter
2) alpha: mixing parameter

Question 36

lambda

Answer 36

a) controls the amount of regularization (bigger lambda, more shrinkage, less complexity, squared bias increases and variance decreases)
b) feature selection property: for elastic nets with alpha > 0 (lasso in particular) some coefficient estimates become exactly zero when lambda is large enough
c) typically tuned by CV: choose lambda with the smallest CV error

Question 37

Alpha

Answer 37

a) controls the mix between ridge (alpha=0) and lasso (alpha=1)
b) provided that lambda is large enough, increasing alpha from 0 to 1 makes more coefficient estimates zero
c) cannot be tuned by cv.glmnet(); need to tune manually

Question 38

Single decision Trees: basics

Answer 38

idea: divide the feature space into a set of non-overlapping regions containing relatively homogeneous observations w.r.t target

deliverable: a set of classification rules based on the values/levels of predictors and represented in the form of a “tree”

predictions: observations in the same terminal node share the same predicted mean (for numeric targets) or same predicted class (for categorical targets)

Question 39

recursive binary splitting

Answer 39

Greedy: at each step, adopt the split that leads to the greatest reduction in impurity at that point, instead of looking ahead and selecting a split that results in a better future step

top-down: start from the “top” of the tree, go “down” and sequentially partition the feature space in a series of splits

Question 40

node impurity measure properties

Answer 40

a) the smaller, the purer the observations in the node
b) gini index and entropy are similar numerically
c) gini index and entropy are more sensitive to node impurity than classification error rate

Question 41

minbucket

Answer 41

minimum bucket size

min # of obs. in a terminal node

effect: higher, tree less complex

Question 42

cp

Answer 42

complexity parameter

min improvement required for a split to be made (not 100% right…)

effect: higher, tree less complex

Question 43

maxdepth

Answer 43

maximum depth

no. of edges from root node to furthest node

effect: higher, tree more complex

Question 44

What can be used to tune cp?

Answer 44

CV within rpart()

Question 45

what can be used to tune maxdepth and minbucket?

Answer 45

must be tuned by trial and error

Question 46

What can you comment on when interpreting trees?

Answer 46

1) number of tree splits
2) split sequence, e.g., start with X1, further split the larger bucket by X2
3) which are the most important predictors (usually those in the early splits)?
4) which terminal nodes have the most observations? any sparse nodes?
5) any prominent interactions?
6) (classification trees) combinations leading to the positive event

Question 47

Cost-complexity pruning rationale

Answer 47

to reduce tree complexity by pruning branches from the bottom that do no improve goodness of fit by a sufficient amount -> prevent overfitting and ease interpretation

Question 48

What is the process of cost-complexity pruning?

Answer 48

step 1) grow a large tree T0
step 2) minimize the penalized objective function = relative training error + Cpx|T| ,(tree complexity)

training error = {RSS, for regression
# misclassification for classification}

Question 49

what is the relationship between cp and the complexity of a tree?

Answer 49

as cp increases the tree is less complex (smaller)

Question 50

what is the alternative to cost-complexity pruning?

Answer 50

One-standard-error (1-SE) rule

how: select a simpler and more interpretable tree with comparable prediction performance

Question 51

does transforming the target variable affect GLMs or Trees?

Answer 51

GLMs: Yes, the transformations alter the values of the predictors and target variable that go into the likelihood function

Trees: Yes, the transformations can alter the calculations of node impurity measures, e.g., RSS, that define the tree splits

Question 52

does transforming the predictors affect GLMs or Trees?

Answer 52

GLMs: Yes, same reasoning target variable

Trees: Yes, unless the transformations are monotonic, e.g., log (monotonic transformations will not change the way tree splits are made.)

Question 53

random forests

Answer 53

(variance reduction) combine the results of multiple trees fitted to different bootstrapped training samples in parallel -> reduce variance of overall predictions

(randomization) take a random sample of predictors as candidates for each split -> reduce correlation between base trees -> further reduce variance of overall predictions

Question 54

key parameters of random forests

Answer 54

1) mtry: # of features sampled as candidates at each split

2) ntree: # of trees to be grown

Question 55

mtry general info

Answer 55

a) lower mtry -> greater variance reduction
b) common choice: sqrt(p) (classification) or p/3 (regression)
c) typically tuned by CV

Question 56

ntree general info

Answer 56

a) higher ntree, more variance reductions
b) often overfitting does not arise even if set to a large number
c) set to a relatively small value to save run time

Question 57

Boosting

Answer 57

in each iteration, fit a tree to the residuals of the preceding tree and subtract a scaled-down version of the current tree’s predictions from the residuals to form the new residuals

each tree focuses on obs.’s the previous tree predicted poorly

Question 58

Key parameters of boosting

Answer 58

1) eta: learning rate (or shrinkage) parameter
2) nrounds: max # of rounds in the tree construction process

Question 59

eta general info

Answer 59

a) effects of eta: higher eta -> algorithm converges faster but is more prone to overfitting

b) rule of thumb: set to a relatively small value

Question 60

nrounds general info

Answer 60

a) effects of nrounds: higher nrounds -> algorithm learns better but is more prone to overfitting

b) rule of thumb: set to a relatively large value

Question 61

fitting process for random forest vs boosting

Answer 61

random forest: in parallel

boosting: in series (sequential)

Question 62

focus for random forest vs boosting

Answer 62

random forest: variance

boosting: bias

Question 63

overfitting for random forest vs boosting

Answer 63

random forest: less vulnerable

boosting: more vulnerable

Question 64

hyperparameter for random forest vs boosting

Answer 64

random forest: less sensitive

boosting: more sensitive

Question 65

what are the two interpretation tools for ensemble trees?

Answer 65

variance importance plots and partial dependence plots

Question 66

Variable importance plots

Answer 66

definition of importance scores: the total drop in node impurity (RSS for regression trees and Gini index for classification trees) due to splits over a given predictor, averaged over all base trees

use: to identify important variables (those with a large score)

limitation: unclear how important variables affect the target

Question 67

partial dependence plots

Answer 67

definition of partial dependence: model predictions obtained after averaging the values/levels of variables not of interest

use: Plot PD(X1) against various X1 to show the marginal effect of X1 on the target variable

limitations: a) assume predictor of interest is independent of other predictors
b) some predictions may be based on practically unreasonable combinations of predictors values

Question 68

Pros of GLMS

Answer 68

1) (target distribution) GLMs excel in accommodating a wide variety of distributions for the target variable
2) (interpretability) the model equation clearly shows how the target mean depends on the features; coefficients = interpretable measure of directional effects of features
3) (implementation) simple to implement

Question 69

Cons of GLMS

Answer 69

1) (complex relationships) unable to capture non-monotonic (e.g., polynomial) or non-additive relationships (e.g., interaction), unless additional features are manually incorporated
2) (interpretability) for some link functions (e.g., inverse link), the coefficients may be difficult to interpret

Question 70

Pros of regularized GLMs

Answer 70

1) (categorical predictors) via the use of model matrices, binarization of categorical variables is done automatically and each factor level is treated as a separate feature to be removed
2) (Tuning) an elastic net can be tuned by CV using the same criterion (e.g., MSE, accuracy) ultimately used to judge the model against unseen test data
3) (variable selection) for elastic nets with alpha > 0, variable selection can be done by making lambda large enough

Question 71

Cons of regularized GLMs

Answer 71

1) (categorical predictors) possible to see some non-intuitive or nonsensical results when only a handful of the levels of a categorical predictor are selected
2) (target distribution) limited/restricted model forms allowed by glmnet() (WEAK POINT!)
3) (interpretability) coefficient estimates are more difficult to interpret when variables are standardized (WEAK POINT!)

Question 72

pros of single trees

Answer 72

1) (interpretability) if there are not too many buckets, trees are easy to interpret because of the if/then nature of the classification rules and their graphical representation
2) (complex relationships) trees excel in handling non-monotonic and non-additive relationships without the need to insert extra features manually
3) (categorical variables) categorical predictors are automatically handled by separating their levels into two groups without the need for binarization
4) (variable selection) variables are automatically selected as part of the model-building process. Variables that do not appear in the tree are filtered out and the most important variables show up at the top of the tree

Question 73

cons of single trees

Answer 73

1) (overfitting) strongly dependent on training data (prone to overfitting) -> predictions unstable with a high variance -> lower user confidence
2) (numeric variables) usually need to split based on a numeric predictor repeatedly to capture its effect effectively -> tree becomes large, difficult to interpret
3) (categorical variables) tend to favor categorical predictors with a large no. of levels

Question 74

pros of ensemble trees

Answer 74

1) much more robust and predictive than base trees by combining the results of multiple trees

Question 75

cons of ensemble trees

Answer 75

1) Opaque (“black box”), difficult to interpret
2) computationally prohibitive to implement