Regression Trees

Question 1

What are the differences between regression, LASSO, and machine learning?

Answer 1

Regression: analyst defines variables, functional form, other models, and selects best method
LASSO: analyst defines variables, functional form, analyst defines broadest model, and algorithm selects best model
Machine Learning: analyst defines a set of variables, not functional form, algorithm defines possible model and the best model. No formula, no coefficients

Question 2

What is a regression tree?

Answer 2

A model that produces predicted values of y for observations with particular valeus of the x variables

Question 3

What algorithm do regression trees use?

Answer 3

CART: Classification and Regression Trees

Question 4

Define top node and the cutoff point.

Answer 4

Top node: the starting point of the sample
Cutoff point: the value at which the split happens for a given variable x

Question 5

When do we stop branching?

Answer 5

By using a stopping rule. This can be as simple as stopping after a certain number of levels, minimum number of observations, and the minimum amount of fit improvement (complex parameter)

Question 6

What are terminal nodes?

Answer 6

The set of nodes after the algorithm stops

Question 7

How do we pick the best possible model with regression trees?

Answer 7

Find the predictor AND the cutoff that yields the smallest RMSE

Question 8

What are the differences between OLS and CART?

Answer 8

The output for OLS are coefficient estimates and CART are a corresponding x values and average y values

Question 9

How can we improve the CART method?

Answer 9

1. Early stopping rule (short-sighted: a split now may not be as important as a split later)
2. Prune a tree

Question 10

How do we prune a tree?

Answer 10

1. Grow a tree, with a small stopping rule
2. Prune it afterwards, then let it grow again

Question 11

Define cost complexity pruning.

Answer 11

At each level, delete one node, the one that brings the least improvement to the model. And stop when we cannot imporve the model fit by deleting any of the nodes

Question 12

Define variable importance.

Answer 12

Measures how much fit improve when a particular x variable is used for splitting, summed across all splits in which that variable occurs

ie: measures how important a variable is in improving the prediction

Question 13

T/F: We want low variable importance in a model.

Answer 13

False: want high variable importance to determine if a variable should be included in the model

Question 14

T/F: Variable importance for a regression tree is a post prediction diagnostic tool, and therefore should be performed on the holdout set.

Answer 14

True

Question 15

Why is a regression tree a regression?

Answer 15

It calculates the average y as a function of x

Question 16

Why is the regression tree a non-parameric regression?

Answer 16

There are no parameters (intercept, slope coefficents) that describe how average y depends on the values of x.

Question 17

What is a top down, greedy approach?

Answer 17

Top down: begins at the top and trickles its way down
Greedy: makes the best split at a particular step and does not look ahead for better splits

Question 18

What can’t we do when the CART algorithm performs automatic pattern detection?

Answer 18

Cannot make mistakes by omitting something that would be important

Question 19

What are the disadvantages of using automatic pattern detection?

Answer 19

1. Might lead to overfitting, even after pruning
2. The algorithm is sensitive to extreme values and data errors
3. variable importance still does not tell us how important variable x is associated with y, in what functional form, or through what interactions (black box)

Question 20

Explain how a regression tree works.

Answer 20

First a regression tree looks at all your x variables and chooses which x, and which value to split the observations into two groups based on the lowest RMSE. Then, it repeats this process over and over again. It will look at the next nodes, and determine which x variable and what value to split to return the lowest RMSE. This will continue until it reaches a stopping rule or there is only 1 observation in each bin.