Chapter 6

Question 1

What is Multiple Linear Regression?

Answer 1

When a regression model has more than one input

Used to fit a linear relationship between a quantitative dependent variable Y

Question 2

Linear Regression

Answer 2

most commonly used predictive modelling technique, “best fit line”

Question 3

Input, Target variables

Answer 3

must be numeric

Question 4

Best fit

Answer 4

Minimizes the sum of squares if the vertical distance from the data points of the line.

Question 5

Goodness of Fit

Answer 5

Difference between the predicted and actual values, called residuals

Question 6

R squared

Answer 6

0 to 1 value - measures the proportion of variance in the target that is explained by the input.

Question 7

Y

Answer 7

Dependent Variable, aka outcome or response variable

Question 8

X

Answer 8

Predictors, aka independent or input variables, regressors, covariates

Question 9

B

Answer 9

B: Coefficients

Question 10

E

Answer 10

The noise or unexplained part

Question 11

The data are used to estimate

Answer 11

the coefficients and the variability of the noise

Question 12

Objectives of fitting a model related to a quantitative outcome

Answer 12

Understanding the relationship between factors (focus of classical stats)
Predicting the outcome of new cases (focus of Data Mining)

Question 13

Explanatory vs Predictive Modeling

Answer 13

The choice of model is closely tied to which is the goal

Both use a dataset to fit a model (i.e. estimate coefficients)

However, there several differences between the two:
Explanatory fits data closely - Predictive predicts new cases accurately
Explanatory uses entire data set - Predictive splits into partitions
Performance measures:
Explanatory: How well data fits model
Predictive: Predictive accuracy

Question 14

Explanatory Modeling

Answer 14

Goal: Explain relationship between predictors (explanatory variables) and target

Familiar use of regression in data analysis

Model Goal: Fit the data well and understand the contribution of explanatory variables to the model

“goodness-of-fit”: R2, residual analysis, p-values

Question 15

Predictive Modeling

Answer 15

Goal: predict target values in other data where we have predictor values, but not target values
Classic data mining context
Model Goal: Optimize predictive accuracy
Train model on training data
Assess performance on validation (hold-out) data
Explaining role of predictors is not primary purpose (but useful)

Question 16

*You cannot include all the binary dummies

Answer 16

*You cannot include all the binary dummies; in regression this will cause a multicollinearity error.
Other data mining methods can use all the dummies.

Question 17

Selecting Subsets of Predictors

Answer 17

Goal: Find parsimonious model (the simplest model that performs sufficiently well)
More robust
Higher predictive accuracy

We will assess predictive accuracy on validation data
Exhaustive Search = “best subset”
Partial Search Algorithms
Forward
Backward
Stepwise

Question 18

Partial Search Algorithms

Answer 18

Forward
Backward
Stepwise

Question 19

Exhaustive Search = Best Subset

Answer 19

All possible subsets of predictors assessed (single, pairs, triplets, etc.)
Computationally intensive, not feasible for big data
Judge by “adjusted R2”
Adjusted R2 for the models with 1 predictor, 2 predictors, 3 predictors, etc. (exhaustive search method)

Adjusted R2 rises until you hit 7-8 predictors, then stabilizes, so choose model with 7 predictors, according to the adj R2 criterion

Question 20

Forward Selection

Answer 20

Start with no predictors
Add them one by one (add the one with largest contribution)
Stop when the addition is not statistically significant

Question 21

Backward Elimination

Answer 21

Start with all predictors
Successively eliminate least useful predictors one by one
Stop when all remaining predictors have statistically significant contribution

Question 22

p value

Answer 22

if it is less than 0.05 then it is less likely to occur by chance

Question 23

Stepwise

Answer 23

Like Forward Selection

Except at each step, also consider dropping non-significant predictors