Decision Trees

Question 1

What are internal nodes?

Answer 1

Points along the tree where splits occur.

Question 2

What are terminal nodes/leaves?

Answer 2

Represent the partitions of the predictor space.

Question 3

What makes a decision tree a “stump”?

Answer 3

It only has one internal node (i.e. one split).

Question 4

Name four advantages to trees:

Answer 4

1. Easy to interpret and explain.
2. Can be presented visually.
3. Manage categorical variables without the need for dummy variables.
4. Mimic human decision-making.

Question 5

Name two disadvantages to trees:

Answer 5

1. Not robust (i.e. you are not going to get the same results every time you go to create a tree from the same data)
2. Not as accurate as other statistical methods, such as supervised learning methods.

Question 6

What makes recursive binary splitting a top-down and greedy approach?

Answer 6

It is top-down because we are starting from all observations and split from there. It is greedy because it only considers the best split at that point in time and does not consider future splits.

Question 7

What is a drawback to recursive binary splitting and how can we resolve this issue?

Answer 7

Recursive binary splitting can create a tree that is too leafy/complex, i.e. has high variance. We can cut down on variance by pruning the leaves of the tree via cost-complexity pruning, which is tweaked by the tuning parameter, alpha.

Question 8

What is the algorithm to selecting the best subtree based on alpha in cost-complexity pruning?

Answer 8

1. Construct a large tree with g terminal nodes using recursive binary splitting.
2. Obtain a sequence of best subtrees, as a function of alpha, using cost-complexity pruning.
3. Choose alpha by applying k-fold cross validation. Select the alpha that results in the lowest cross validation error.
4. The best subtree is the subtree created in step 2 with the selected alpha value from step 3.
   *Note that if alpha=0 is the original large tree with g terminal nodes.

Question 9

What is bootstrapping?

Answer 9

It is random sampling WITH replacement, so an observation can appear in a bootstrap sample more than once.

Question 10

What is node purity? What can we say about the node depending on its value?

Answer 10

Pure nodes are those that have one class (i.e. all observations all belong in one class). Value increases as node becomes more impure (i.e. observations do not all belong in the same class).

Question 11

Does the number of clusters in a dendrogram increase or decrease as you go up?

Answer 11

Decreases…At the bottom of the dendrogram, each observation is its own cluster. As you go up and observations start to cluster together, the number of clusters decreases.

Question 12

T/F: For a given number of clusters, hierarchical clustering can sometimes yield less accurate results than K-means clustering.

Answer 12

True. K-means does a fresh analysis for each value of K while for hierarchical clustering, reduction in the number of clusters is tied to clusters already made. This can miss cases where the clusters are not nested.

Question 13

If p=total number of features and m=number of features selected at each split in a random forest, what is the typical choice of m?

Answer 13

sqrt(p) or p/3.

Question 14

For a random forest, let p
be the total number of features and m
be the number of features selected at each split. What is the probability a split will not consider the strongest predictor?

Answer 14

(p-m)/p

Question 15

How do bagging/random forest differ from boosting in terms of variance/bias tradeoff?

Answer 15

Bagging reduces variance, whereas boosting reduces bias (i.e. increases variance).