Predictive Analytics

Question 1

Advantages of Converting Numeric Variables to Factor Variables

Answer 1

1. Provides added flexibility to model
2. When treated as numeric variables, there is an implicit assumption that there is a monotonic relationship between these variables and the target variable in GLMS
3. In decision trees, splits have to respect the order of the variable’s values
4. Potential improvement in predictive accuracy due to the flexibility of capturing the effects of the variable across different values of their range

Question 2

Disadvantages of Converting Numeric Variables to Factor Variables

Answer 2

1. When converted, these variables have to be represented by a greater number of dummy variables in a GLM, which inflates the dimension of the data and possibly dilutes predictive power
2. In a decision tree, the number of splits increases, which increases the computing burden and diminishes interpretability

Question 3

Check to Support Conversion of Numeric Variable to Factor Variable

Answer 3

Examine the mean of the target variable split by different values of the integer variable. If the mean does not vary in a monotonic fashion, this supports the conversion.

Question 4

Define offset

Answer 4

In a predictive model, an offset is a variable that serves to account for the different exposure periods of different observations and therefore yield more accurate predictions.

In a GLM, an offset is a predictor whose regression coefficient is known to be 1 a priori

Question 5

Introducing Training / Test Sets

Answer 5

Prior to fitting any models, I split the data into a training set (70% of the observations) and a test set (30% of the observations) using stratified sampling. To check that the two sets are representative, I note that the means of the target variable in the two data sets are comparable. The predictive power of the models we will study will be performed on the training data, while the predictive performance of these models will be performed on the test data.

Question 6

Define minbucket

Answer 6

The minimum number of observations in any terminal node of the tree. The higher the value, the smaller the number of splits and less complex the tree.

Question 7

Define cp

Answer 7

The minimum improvement (with respect to R-squared) needed in order to make a split. The higher the value, the less complex the tree.

Question 8

Define maxdepth

Answer 8

The maximum number of branches from the tree’s root note to the furthest terminal node. The higher the value, the more complex the tree.

Question 9

Describe Cost-Complexity Pruning

Answer 9

Technique that performs cross-validation to evaluate the predictive performance of a tree. The algorithm divides the training data in 10 (default for xval parameter) folds , trains the tree on all but one fold, and then computes the R-squared on the held-out fold. This is performed for all values of the cp paramter greater than the defined parameter to determine which cp value yields the lowest error. The tree is then pruned using this cp value. Splits that do not fulfill the impurity reduction threshold are removed.

Question 10

One-Standard-Error Rule

Answer 10

Chooses the smallest tree whose cross validation error is within one standard error of the minimum cross-validation error. Generally results in more simple trees.

Question 11

Define ntree

Answer 11

The number of trees to be grown in a random forest. It is generally a good idea to set ntree to a large number to increase the variance reduction and ensure that each observation and each feature is represented at least once in the random forest. However, too large a value fo ntree may lead to excess run time.

Question 12

Define mtry

Answer 12

Every time a split is made in a random forest, a random sample of features is taken and considered as split candidates. The number of candidates is defined by the mtry parameter. Making mtry too large can reduce the correlation between the predictions of different trees. Making mtry too small can impose severe restrictions on the tree growing process.

Question 13

What is a random forest?

Answer 13

Ensemble method that relies on bagging to produce a large number of bootstrapped training samples over which trees are constructed in parallel. The results of the different trees are combined , which reduces the variance of predictions and prevents overfitting.

Question 14

What is overdispersion?

Answer 14

Overdispersion refers to the situation when the variance of the target variable is greater than the mean.

Question 15

Benefits of Using Log Link

Answer 15

1. Ensures the model predictions are non-negative
2. Makes the model easy to interpret.
3. Also, it is the canonical link for the Poisson distribution and thus facilitates the convergence of the model fitting algorithm.

Question 16

Pros of Using Binarization with Stepwise Selection

Answer 16

Allows the selection procedure to drop individual factor levels if they are statistically insignificant with respect to the base level (instead of retaining or dropping the factor variables in their entirety

Question 17

Cons to Using Binarization with Stepwise Selection

Answer 17

May cause procedure to take significantly more time to complete.

Resulting model may also be hard to interpret.

Question 18

Components of Stepwise Selection

Answer 18

Selection Criterion (AIC, BIC) and Selection Process (Forward, Backward)

Question 19

AIC

Answer 19

Performance metric used to rank competing models (similar to BIC). Defined as -2l + 2p, where l is the loglikelihood of the model on the training set and p represents the number of parameters. Balances the goodness of fit of a model to the training data. Lower AIC, better fit model.

Question 20

BIC

Answer 20

Performance metric used to rank competing models (similar to AIC). Defined as -2l + ln(n)p, where l is the loglikelihood of the model on the training set, n is the number of observations, and p represents the number of parameters. Balances the goodness of fit of a model to the training data. Lower BIC, better fit model.

Question 21

AIC vs BIC

Answer 21

Both metrics demand that for the inclusion of an additional feature to improve the performance of the model, the feature must increase the loglikelihood by at least a certain amount (the penalty amount).

In general, the penalty is typically greater for BIC, so the BIC is more stringent for complex models and is a more conservative approach.

Question 22

Forward Selection

Answer 22

Opposite of backward selection. Starts with the simplest model (model with only the intercept and no features) and progressively adds the feature that results in the greatest improvement of the model until no features can be added to improve the model.

Question 23

Backward Selection

Answer 23

Opposite of forward selection. Starts with the most complex model (model with all features) and progressively removes the feature that causes, in its absence, the greatest improvement in the model according to a certain criterion.

Question 24

Forward vs Backward Selection

Answer 24

Forward selection is more likely to result in a simpler model relative to backward selection given forward selection starts with a model with no features.

Question 25

How to Interpret Coefficients of Log Link

Answer 25

Exponentiate the coefficients and subtract 1. This results in the algebraic changes in the target variable

Question 26

What is Regularized Regression?

Answer 26

Alternative approach to reducing the complexity of a GLM and preventing overfitting. It works by adding a penalty term that reflects the size of the coefficients to the function to optimize (loglikelihood). This shrinks the magnitude of the estimated coefficients of features with limited predictive power towards zero. In some cases, the coefficients are forced exactly to zero, thus removing those features from the model.

Question 27

Regularization vs Stepwise Selection

Answer 27

1. Binarization of factor variables is done automatically in regularization, and each factor level is treated as a separate feature to remove 2. Cross-validation can be used to optimize the regularization parameter (lambda) such that the RMSE is minimized 3. Coefficient estimates are more difficult to interpret in regularization because the variables are standardized 4. The glmnet package only allows a restricted set of model forms

Question 28

Supervised Learning Methods

Answer 28

Target variable guides the analysis. Main Methods: GLMs, Decision Trees

Question 29

Unsupervised Learning Methods

Answer 29

Target variable is absent; interested in extracting relationships between variables in the data (lend themselves to high-dimensional datasets) Main Methods: Principal Components Analysis, Cluster Analysis

Question 30

Regression vs Classification

Answer 30

Regression has numeric target variable, classification has categorical target variable

Question 31

Training vs Test Set

Answer 31

Training Set is the data used to develop the predictive model (typically the largest portion of the data) Test Set is the data used the evaluate the predictive performance when applied to data the model has not seen before

Question 32

What is Cross-Validation?

Answer 32

Alternative to training/test split when data set is small Splits data into k equal folds; each fold is used once as the validation set, while the remaining folds are used as the training set Predictive model is fit k times and predictions are combine

Question 33

Common Performance Metrics Used in Regression Problems

Answer 33

Root Mean Squared Error (RMSE): aggregate of all prediction erros in the test set

Question 34

Common Performance Metric Used in Classification Problems

Answer 34

Classification Error Rate: proportion of observations in the test set that are incorrectly classified

Question 35

Define Bias

Answer 35

Difference between expected value and true value More complex, lower bias

Question 36

Define Variance

Answer 36

Amount by which expected value would change using a different training set More complex, higher variance

Question 37

What is Irreducible Error?

Answer 37

The variance of the noise. Cannot be reduced no matter how good the predictive model is

Question 38

Variables vs Features

Answer 38

Variables are raw recorded measurements from the original dataset without any transformations Features are derivatives of raw variables

Question 39

Feature Generation

Answer 39

Process of developing new features based on existing variables in the data Seeks to enhance the flexibility of the model and lower the bias of the predictions at the expense of an increase in variance Plays more prominent role in GLMs compared to Decision Trees

Question 40

Feature Selection (Removal)

Answer 40

Opposite of feature generation; process of dropping features with limited predictive power Important concept in GLMs and Decision Trees

Question 41

Feature Selection Methods

Answer 41

Forward/Backward Selection

Question 42

Other Commonly Used Performance Metrics

Answer 42

RMSE (most common) Loglikelihood R-Squared (goodness of fit measure) Chi-square

Question 43

What is Binning?

Answer 43

Creating a categorical predictor whose levels are defined as non-overlapping intervals of the original variable

Question 44

What is Binarization?

Answer 44

Feature generation method used for categorical variables, which turns a categorical variable into a collection of binary variables Baseline level is automatically defaulted by alphabetical order Good idea to make baseline level the level with most observations

Question 45

Interactions Between Continuous and Categorical Predictors

Answer 45

Need to multiply continuous variable by each of the binary variables created from the categorical variable To assess the extent of the interaction graphically, use a scatterplot

Question 46

Interactions Between Two Categorical Predictors

Answer 46

Need to multiply each of the binary variables created from both categorical variables To assess the extent of the interaction graphically, use a box plot

Question 47

Regularization

Answer 47

Alternative to forward/backward selection that shrinks the magnitude of coefficients of features with limited predictive importance towards zero Goal is to simplify model and avoid overfitting

Question 48

Ridge Method

Answer 48

Type of regularization in which the penalty is the sum of squares of the slope coefficients

Question 49

Lasso Method

Answer 49

Type of regularization in which the penalty is the sum of absolute values of the slope coefficients

Question 50

Elastic Net Regression Method

Answer 50

Type of regularization in which the penalty captures both lasso and ridge

Question 51

Regularization Parameter (Lambda)

Answer 51

When lambda = 0, the coefficient estimates are equal to the ordinary least squares estimates As lambda increases, the effect of regularization increases

Question 52

Ridge vs Lasso

Answer 52

The lasso method forces coefficients to exactly zero, whereas the ridge method reduces them but not ever to zero (retains all features) Lasso tends to produce simpler models with fewer features

Question 53

Advantages of Regularization

Answer 53

Computationally more efficient than stepwise selection algorithms

Question 54

Disadvantages of Regularization

Answer 54

May not produce the most interpretable model (especially for Ridge) During model fitting, all numeric coefficients are standardized, which makes the interpretation of their estimates less intuitive Cannot accommodate all distributions for GLMs

Question 55

GLMs vs Linear Models

Answer 55

GLMs offer considerably more flexibility: 1. The target variable can be any member of the exponential family of distributions 2. GLMs have the ability to analyze situations in which the effects of the predictors on the target mean are more complex than merely additive in nature

Question 56

Target Distribution for Continuous, Positive Data

Answer 56

Gamma and Inverse Gaussian most appropriately capture the skewness of the target variable Ex. claim amounts, income, amount of insurance coverage

Question 57

Target Distribution for Binary Data

Answer 57

Binomial (occurrence or non-occurrence of event)

Question 58

Target Distribution for Count Data

Answer 58

Poisson most natural candidate Drawback: requires mean to equal variance, otherwise overdispersion may exist

Question 59

Link Function for Positive Target Mean

Answer 59

Log link is most natural candidate, as it ensures predictions are always positive Also easily interpretable

Question 60

Link Function for Target Mean Between 0 and 1

Answer 60

Logit link good candidate, as it ensures predictions are always between 0 and 1 Also easily interpretable due to connection to the log link function

Question 61

What is a Canonical Link Function?

Answer 61

Link function that simplifies the estimation procedure Should not always be used; need to consider interpretability

Question 62

Canonical Link Function for Normal Distribution

Answer 62

Identity

Question 63

Canonical Link Function for Binomial Distribution

Answer 63

Logit

Question 64

Canonical Link Function for Poisson Distribution

Answer 64

Natural Log

Question 65

Canonical Link Function for Gamma Distribution

Answer 65

Inverse

Question 66

Canonical Link Function for Inverse Gaussian Distribution

Answer 66

Squared Inverse

Question 67

Weights

Answer 67

Used in GLMs to assign a higher emphasis to observations that are averaged across more subjects of similar characteristics Variance of each observation is inversely related to the group size Does not affect the mean of the target variable

Question 68

Offsets

Answer 68

Additional predictor used to account for different means of different observations Important for offset to be on same scale as the linear predictor Group size is positive related to the mean of the target variable Do not affect the variance of the target variable

Question 69

Method for Estimating Parameter Coefficients in Linear Models

Answer 69

Least Squares Regression

Question 70

Method for Estimating Parameter Coefficients in GLMs

Answer 70

Maximum Likelihood Estimation

Question 71

What is Deviance?

Answer 71

Goodness-of-fit measure for GLMs (compared to R-squared in linear models) Measures the extent to which the GLM departs from the most elaborate (or saturated) model Lower the deviance, closer the GLM is to a perfect fit

Question 72

What are Deviance Residuals?

Answer 72

Measure of Deviance used in GLMs Used over raw residuals because deviance residuals are normally distributed

Question 73

Q-Q Plot

Answer 73

Displays the standardized deviance residuals against the standard normal quartiles

Question 74

Confusion Matrix

Answer 74

Tabular display of how predictions of a binary classifier line up with the observed classes

Question 75

Classification Error Rate

Answer 75

(False Negatives + False Positives) / n

Question 76

Sensitivity

Answer 76

(True Positives) / (True Negatives + False Negatives) Higher sensitivity and specificity, better classifier

Question 77

Specificity

Answer 77

(True Negatives) / (True Negatives + False Positives) Higher sensitivity and specificity, better classifier

Question 78

Receiver Operator Characteristic Curve (ROC)

Answer 78

Graphical tool that displays sensitivity vs specificity Classifier with perfect predictive performance rises quickly to top left corner

Question 79

AUC

Answer 79

Area under the curve of a ROC plot Measure of the predictive performance of the model (want AUC to be closest to 1)

Question 80

Regression Trees

Answer 80

Have a quantitative target variable and use the average of the target variable in that group as the predicted variable

Question 81

Classification Trees

Answer 81

Have qualitative target variables and use the most common class (mode) of the target variable in that group as the predicted class

Question 82

Node

Answer 82

Point on the tree that corresponds to a subset of the data

Question 83

Root Node

Answer 83

Node at the top of the tree representing the full data set

Question 84

Terminal Node

Answer 84

Also referred to as leaf. The nodes at the bottom of the tree that cannot be split any further

Question 85

Binary Tree

Answer 85

Each node only has two children

Question 86

Depth

Answer 86

The number of branches from the root node to the furthest terminal node

Question 87

Measure of Impurity in Regression Trees

Answer 87

Residual sum of squares

Question 88

Measure of Impurity in Classifcation Trees

Answer 88

Most common is classification error. Other measures of node impurity are entropy and gini. Choice of impurity measure does not have a significant impact on the performance of the tree

Question 89

Pruning

Answer 89

Process to control the complexity of the tree structure Similar to stepwise selection in GLMs Start at the bottom of the tree and remove splits that do not meet a specified impurity reduction

Question 90

Advantages of Decision Trees

Answer 90

1. Generally easier to interpret compared to GLMs 2. Excel in handling nonlinear relationships and do not require transformations 3. Good at automatically recognizing interactions between variables 4. Do not require binarization for categorical variables 5. Variables are automatically selected with the most important variables appearing at the top of the tree 6. Much less susceptible to model mis-specification than GLMs 7. Can easily be modified to deal with missing data

Question 91

Disadvantages of Decision Trees

Answer 91

1. More prone to overfitting relative to GLMs and produce unstable predictions with more variance (small change in training data can lead to big changes in the fitted tree) 2. Favor categorical features with many levels over those with few levels 3. Lack of model diagnosis

Question 92

Random Forests

Answer 92

Ensemble method that generates multiple bootstrapped samples of the training set and fits base tree models to each bootstrapped sample Results from all base trees are combined to form an overall prediction Randomization is performed at each step Improves model variance and predictive performance

Question 93

Advantages of Random Forests

Answer 93

1. Much more robust than single trees | 2. More precise predictions with lower variance

Question 94

Disadvantages of Random Forests

Answer 94

1. Not easily interpretable | 2. Takes considerably more computational power

Question 95

Boosting

Answer 95

Ensemble method that builds a sequence of interdependent trees using information from previously grown trees Each iteration builds on the residuals of the prior tree Improves model bias

Question 96

Advantages of Boosting

Answer 96

1. Improves predictive accuracy

Question 97

Disadvantages of Boosting

Answer 97

1. More vulnerable to overfitting 2. Significant computational cost 3. Not easily interpretable

Question 98

Principal Components Analysis

Answer 98

Advanced technique that transforms a large number of (possibly correlated) variables to a smaller, more manageable, set of representative variables that capture much of the information in the full data set Resulting variables are referred to as principal components and are a linear combination of the existing variables Particularly useful for feature generation To perform PCA on categorical variables, they must first be binarized

Question 99

Scree Plot

Answer 99

Provides simple visual inspection method for determining number of principal components to use Depicts the PVE (proportion of variance explained) of each PC

Question 100

Elbow of Scree Plot

Answer 100

Point at which the PVE drops off significantly PCs beyond the elbow have a very small PVE and can be dropped

Question 101

Cluster Analysis

Answer 101

Partitions heterogeneous observations into a set of distinct homogeneous groups (clusters) Observations within each cluster have similar characteristics Two main methods: k-means clustering, hierarchical clustering

Question 102

K-Means Clustering

Answer 102

Assigns each observation into one of k clusters, with k being defined beforehand Algorithm automatically searches for the best configuration of the k clusters Clusters are chosen such that the variance within each cluster is small while the variance among different clusters is large Advisable to run the algorithm many times with different initial cluster assignments and then choose the algorithm with the lowest within-cluster variance Features must be standardized before clustering

Question 103

Elbow Method

Answer 103

Plot of the ratio of between-cluster variation to total variation against k Used to determine k - see when proportion of variance explained plateaus

Question 104

Hierarchical Clustering

Answer 104

Clustering method that does not require the choice of k at the start Uses a dendrogram, which is a tree-based visualization of the hierarchy of clusters Consists of a series of fusions (mergers) of clusters

Question 105

Complete Linkage

Answer 105

Measure of intercluster dissimilarity based on the maximal pairwise distance between observations in two clusters

Question 106

Single Linkage

Answer 106

Measure of intercluster dissimilarity based on the minimal pairwise distance between observations in two clusters

Question 107

Average Linkage

Answer 107

Measure of intercluster dissimilarity based on the average pairwise distance between observations in two clusters

Question 108

Dendrogram

Answer 108

Tree-based visualization of the hierarchy of clusters Clusters towards bottom of the dendrogram are similar to one another, while clusters towards the top are far apart

Question 109

K-Means vs Hierarchical Clustering

Answer 109

1. K-Means clustering requires standardization 2. Number of clusters are pre-defined in k-means clustering 3. Hierarchical clustering uses nested clusters

Question 110

nstart parameter

Answer 110

Used in cluster analysis, the nstart parameter controls the number of random selections of initial cluster centers that are used by the kmeans algorithm. Improves the chances of finding a better local optimum. Recommended 20-50

Question 111

Define interaction

Answer 111

An interaction exists if the effect of one variable on the target variable changes with the value or level of another variable.

Question 112

eta

Answer 112

Learning rate parameter in boosting methods. Scalar multiple between 0 and 1. Predictions of the current tree are scaled by the eta parameter and added to the overall model. Lower eta's "slow" the learning and generally result in more accurate models.

Question 113

nrounds

Answer 113

Maximum number of boosting rounds. Often a large number 100-200 to ensure a sufficient number of trees are grown.