Decision Trees

Question 1

True or false: decision trees are a non-parametric alternative to regression

Answer 1

True

Question 2

How do decision trees work?

Answer 2

They split predictors into regions and then assign the average value of the region in the regression setting and the most common value in the classification setting.

Question 3

What algorithm is used to grow the trees and how does it work?

Answer 3

Recursive Binary Splitting
It selects a binary split that minimizes the MSE. The algorithm is greedy - it only optimizes the current split. The algorithm continues until the number of observations in a region is below a specified number.

Question 4

Why do we prune the tree?

Answer 4

The resulting tree from recursibe binary splitting is probably too big: more splits means more flexibility, lower biais and higher variance.

Question 5

True or false: there is no optimal number of splits that minimizes MSE?

Answer 5

True

Question 6

What are the two pruning methods?

Answer 6

1. Cost complexity pruning

2. Weakest link pruning

Question 7

What is the tuning parameter (alpha)?

Answer 7

The cost of a tree per terminal nods.

Question 8

Wow is the tuning parameter selected?

Answer 8

Cross-validation

Question 9

In a classification tree, what measure is used instead of the MSE as a number to minimize?

Answer 9

The classification error rate

Question 10

For tree growing, why can’t we use the classification error rate and what can we use instead?

Answer 10

Not sensitive enough

Gini index or Cross-entropy

Question 11

True or false: the Gini index is the variance of observations?

Answer 11

True

Question 12

In a classification tree, what measures can be used for:

1. Pruning the tree
2. Splitting the tree

Answer 12

1. Classification error rate

2. Gini index or cross-entropy

Question 13

What are the advantages of decision trees over linear regression?

Answer 13

1. Easier to explain
2. Closer to the way human decisions are made
3. Tree can be graphed, making it easier to interpret
4. Easier to handle categorical predictors (linear regression requires dummy variables)

Question 14

What are the decision tree’s shortcomings?

Answer 14

1. Do not predict as well as linear regression

2. Not robust (small change in the input data can have a big effect on trees)

Question 15

What methods can be used to adress the decision tree’s shortcomings?

Answer 15

Bagging, random forest and boosting

Question 16

What is the effect of bagging, random forest and boosting on the variance of decision trees?

Answer 16

They lower the variance

Question 17

Explain the bagging method.

Answer 17

Bagging is a form of bootstraping.

1. Select B bootstrap samples from the n observations.
2. Construct B trees
3. Average the results of each trees

Question 18

What is a bootstrap sample?

Answer 18

Simulating a bootstrap sample of size n means drawing n items from the initial sample with replacement.

Question 19

What is the effect of bagging on a simple tree’s variance?

Answer 19

Divides it by B (# of bootstrap saples)

Question 20

Is there a danger of over fitting by making B too large in bagging?

Answer 20

No.

Question 21

What is out-of-bag (OOB) validation?

Answer 21

For n sufficently large, about 1/3 of the initial observations won’t be used in the bootstrap samples. For each tree, the test MSE can be computed using the OOB part of the sample. This eliminates the need for cross-validation.

Question 22

Explain the random forest method

Answer 22

1. Specify a positive integer ‘m’

2. At each split, m predictors are selected randomly and those are the only predictors that are considered for splitting

Question 23

Is there a danger of over fitting by making B too large in random forest?

Answer 23

No.

Question 24

Why would we use random forest over bagging?

Answer 24

Bagged trees may be correlated. Selectng m predictors has the effect of decorrelating the trees.

Question 25

True or false : if m = k, random forest is reduced to bagging?

Answer 25

True.

Question 26

Is there a danger of over fitting by making B too large in boosting?

Answer 26

Yes.