Wronged Questions: Decision Trees

Question 1

T/F: Small shrinkage parameter requires more iterations because it has a slower learning rate

Answer 1

True

Question 2

T/F: Boosting can lead to overfitting if you have a high number of iterations

Answer 2

True

Question 3

As K increases, flexibility (increases/decreases).

Answer 3

Decreases

Question 4

Classification rate, gini index, and entropy are (inappropriate/appropriate) when pruning a tree.

Answer 4

Appropriate

Question 5

T/F: Decision trees are easier to interpret than linear models.

Answer 5

True

Question 6

T/F: Decision trees are more robust than linear models.

Answer 6

False. Decision trees are generally less robust than linear models; they can produce significantly different outcomes with small changes in the input data.

Question 7

T/F: Decision trees handle qualitative predictors more easily than linear models.

Answer 7

True. Decision trees naturally handle qualitative (categorical) predictors without the need for preprocessing steps such as creating dummy variables, which are often required in linear models.

Question 8

T/F: In boosting, the number of terminal nodes in each tree is independent of the number of splits.

Answer 8

False. In boosting, the number of terminal nodes in each tree is directly related to the number of splits. The number of terminal nodes (leaves) in a tree is one more than the number of splits.

Question 9

T/F: Boosting does not allow for the adjustment of model complexity through the parameter.

Answer 9

False. Parameter d is specifically used to adjust the complexity of the model in boosting.

Question 10

T/F: Boosting considers only a random subset of predictors at each node in every tree.

Answer 10

False. In classical boosting algorithms, all available predictors are considered at each split, not a random subset.

Question 11

T/F: A smaller value of d in boosting necessitates a larger number of trees to adequately model the data.

Answer 11

True. A smaller implies simpler base learners (trees), which individually capture less of the data’s complexity. Therefore, more trees are needed to aggregate enough information to model the data effectively.

Question 12

T/F: Like bagging, boosting is a general approach that can be applied to many statistical learning methods for regression or classification.

Answer 12

True

Question 13

T/F: Each tree is fit on a modified version of the bootstrapped samples for boosting.

Answer 13

False. Boosting does not involve bootstrap sampling; instead each tree is fit on a modified version of the original data set.

Question 14

T/F: Unlike fitting a single large decision tree to the data, which amounts to fitting the data hard and potentially overfitting, the boosting approach instead learns slowly.

Answer 14

True

Question 15

T/F: In boosting, unlike in bagging, the construction of each tree depends strongly on the trees that have already been grown.

Answer 15

True

Question 16

T/F: Like bagging, boosting involves combining a large number of decision trees.

Answer 16

True

Question 17

T/F: Unlike bagging and random forests, boosting can overfit if B is too large.

Answer 17

True

Question 18

T/F: Cross-validation is used to select B.

Answer 18

True

Question 19

T/F: A very small value of the shrinkage parameter can require using a very large number of trees to achieve good performance.

Answer 19

True

Question 20

T/F: An interaction depth of zero often works well and the boosted ensemble is fitting an additive model.

Answer 20

False. An interaction depth of one often works well and the boosted ensemble is fitting an additive model.

Question 21

T/F: In boosting, because the growth of a particular tree takes into account the other trees that have already been grown, smaller trees are typically sufficient.

Answer 21

True

Question 22

T/F: Individual trees in a random forest are left unpruned, contributing to the ensemble’s variance reduction despite their own overfitting.

Answer 22

True. In a random forest, individual trees are typically grown to their full depth without pruning, which might make them prone to overfitting. However, when these overfitted trees are aggregated, the ensemble model achieves a significant reduction in variance.

Question 23

T/F: The combination of results from unpruned trees in a random forest leads to a reduction in the overall variance of the model.

Answer 23

True. Emphasizing the ensemble effect where the aggregation of multiple unpruned, overfitted trees results in a model with reduced overall variance, leveraging the strength of the ensemble to balance out individual tree overfitting.

Question 24

T/F: Increasing m leads to a higher degree of decorrelation between the trees, where m is the number of predictors chosen as split candidates at each split.

Answer 24

False. As a larger value of m tends to increase the correlation between trees. The parameter m is the number of predictors chosen as split candidates at each split.

Question 25

T/F: Random forests can effectively handle both regression and classification problems.

Answer 25

True. It acknowledges the versatility of random forests in addressing different types of prediction problems.

Question 26

T/F: A variable importance plot is a useful tool for identifying which predictors are most influential in a random forest model.

Answer 26

True. Variable importance plots are indeed utilized to identify the most important predictors in a random forest, offering insights into how different variables contribute to the predictive power of the model.

Question 27

T/F: Bagging and random forests make use of bootstrapped samples in their algorithms, but boosting does not.

Answer 27

True. Bagging and random forests employ bootstrapped samples to build multiple decision trees, enhancing model accuracy and robustness by aggregating their predictions.

Conversely, boosting sequentially constructs trees, each focusing on correcting errors from previous ones, without utilizing bootstrapped samples to improve performance.

Question 28

T/F: Boosting can overfit if the number of iterations is set too high, unlike bagging or random forests.

Answer 28

True. Boosting’s performance can be sensitive to the number of iterations, leading to potential overfitting.

Question 29

T/F: The optimal number of iterations in boosting is often determined through cross-validation.

Answer 29

True. Cross-validation is commonly used to select the optimal B in boosting to balance bias and variance.

Question 30

T/F: For bagging and random forests, the choice of B is less critical to avoiding overfitting compared to boosting.

Answer 30

True. Bagging and random forests are generally robust against overfitting due to their aggregation methods, making the specific choice of B less critical.

Question 31

T/F: Pruning is a common strategy in both bagging and boosting to prevent overfitting.

Answer 31

False. Pruning is not used in either bagging or boosting as a strategy to prevent overfitting.

Question 32

T/F: Bagging significantly reduces variance by averaging multiple predictions.

Answer 32

True. One of the primary advantages of bagging is its ability to reduce the variance of complex models, like deep/large decision trees, by averaging the predictions of multiple bootstrapped models, which tends to make the ensemble prediction more robust than any single model.

Question 33

T/F: Each bagged tree uses approximately one-third of observations from the original training set.

Answer 33

False. In bagging, each tree, on average, makes use of around two-thirds of the observations due to the nature of bootstrap sampling, where some observations are repeated, and others are left out.

Question 34

T/F: Bagging is exclusively effective for decision trees and cannot be applied to other statistical learning methods.

Answer 34

False. Bagging is a general-purpose procedure that can be applied to many types of statistical learning methods, not just decision trees, although it is particularly beneficial for models that exhibit high variance.

Question 35

T/F: On average, (p-m)/p of the splits will not even consider the strong predictor.

Answer 35

True

Question 36

T/F: The main difference between bagging and random forests is the choice of predictor subset size.

Answer 36

True

Question 37

T/F: If a random forest is built using m = p^1/2, then this amounts to bagging.

Answer 37

False. If a random forest is built using m = p, then this amounts to bagging.

Question 38

T/F: Using a small value of m in building a random forest will typically be helpful when we have a large number of correlated predictors.

Answer 38

True

Question 39

T/F: Random forests will not overfit if we increase B, so in practice we use a value of B sufficiently large for the error rate to have settled down.

Answer 39

True

Question 40

T/F: An alpha value of zero results in the largest, unpruned tree.

Answer 40

True. An a value of zero implies no penalty on the tree’s complexity, and thus the tree grows to its largest size without any pruning. This results in the most complex tree possible.

Question 41

T/F: Increasing alpha leads to a decrease in the variance of the model.

Answer 41

True. As the tree becomes simpler with higher a
values, its variance decreases due to less model flexibility.

Question 42

T/F: Increasing alpha leads to a decrease in the squared bias of the fitted tree.

Answer 42

False. Increasing alpha leads to an increase in the squared bias of the fitted tree. This is because increasing the value of a in cost complexity pruning penalizes the addition of splits to the tree, resulting in a simpler tree (lower variance).

A simpler tree is less flexible in fitting the data, which leads to an increase in the squared bias as the model becomes increasingly unable to capture the underlying patterns in the data.

Question 43

Three non-parametric statistical learning methods

Answer 43

KNN, Decision Trees, Bagging/Random Forest/Boosting

Question 44

T/F: To build each tree using random forests, a bootstrapped sample of n observations is used, and for each split within the tree, a new random selection of m predictors is made.

Answer 44

True. In random forests, each tree is built from a bootstrapped sample of the original dataset, containing n observations. At each split in the construction of a tree within a random forest, a random subset of m predictors is selected from all available predictors.

Question 45

T/F: Out-of-bag estimation can be used to estimate the test error for random forests.

Answer 45

True. Out-of-bag estimation, which uses each observation’s predictions from trees where that observation was not in the bootstrap sample, provides an estimate of the test error without needing a separate test set.

Question 46

T/F: Random forests reduce bias through the averaging of multiple decorrelated trees.

Answer 46

False. It is incorrect because the main benefit of random forests is variance reduction, not necessarily bias reduction. Random forests reduce variance by averaging the results of multiple decorrelated trees. While this ensemble method is effective at addressing overfitting and reducing variance, the reduction in bias is not guaranteed.

Question 47

T/F: Pruning always decreases both training and test error rates.

Answer 47

False. Pruning does not always decrease both error rates.

Question 48

T/F: Pruning reduces the training error rate but may increase the test error rate.

Answer 48

False. Pruning typically increases the training error rate due to less perfect fitting to the training data, the impact on the test error rate can be mixed.

Question 49

T/F: Pruning increases the training error rate but has a predictable effect on reducing the test error rate.

Answer 49

False. There is usually not a predictable effect on the test error rate.

Question 50

T/F: Pruning tends to increase the training error rate, but its effect on the test error rate can vary.

Answer 50

True. Pruning a classification tree involves removing splits that provide less generalizable decision-making ability, leading to a simpler model. While this process makes the tree less flexible and typically increases the training error rate due to less perfect fitting to the training data, the impact on the test error rate can be mixed.

Question 51

T/F: Pruning a classification tree has no impact on the training error rate but tends to decrease the test error rate.

Answer 51

False. Pruning a classification tree increases the training error rate. We still do it to prevent overfitting.

Question 52

T/F: Bagging increases the interpretability of a model.

Answer 52

False. Bagging does not make the model more interpretable; in fact, it makes it less interpretable.

Question 53

T/F: Out-of-bag (OOB) error estimation requires each observation to be predicted by all trees in the ensemble.

Answer 53

False. OOB error estimation uses only the trees for which the specific observation was not in the bootstrap sample.

Question 54

T/F: A very high number of bootstrap samples will inevitably lead to overfitting in bagged models.

Answer 54

False. Increasing the number of trees does not lead to overfitting due to the aggregation of predictions.

Question 55

T/F: Bagging cannot improve prediction accuracy in classification settings.

Answer 55

False. Bagging is indeed useful for improving prediction accuracy in classification settings.

Question 56

T/F: For a sufficiently large number of bootstrap samples, out-of-bag error is virtually equivalent to leave-one-out-cross-validation error.

Answer 56

True. For a sufficiently large number of trees, out-of-bag error is virtually equivalent to leave-one-out cross-validation error. OOB error estimation, which uses each tree’s predictions on observations not included in its bootstrap sample, provides a reliable estimate of the model’s performance that becomes increasingly accurate as the number of bootstrap samples increases.

Question 57

T/F: Each bootstrapped dataset in bagging contains a completely different set of observations from the original dataset.

Answer 57

False. Each bootstrapped dataset likely contains repeated observations due to sampling with replacement, and not all observations will be different.

Question 58

T/F: First step in building a regression tree: Use recursive binary splitting to grow a large tree on the training data, stopping only when each terminal node has fewer than some minimum number of observations.

Answer 58

True

Question 59

T/F: Second step in building a regression tree, apply cost complexity pruning to the large tree in order to obtain a sequence of best subtrees, as a function of a nonnegative tuning parameter.

Answer 59

True

Question 60

T/F: Third step in building a regression tree, utilize k-fold cross-validation to select the tuning parameter α by evaluating the mean squared prediction error on the data in the left-out kth fold, as a function of k.

Answer 60

False. utilize k-fold cross-validation to select the tuning parameter α by evaluating the mean squared prediction error on the data in the left-out kth fold, as a function of α.

Question 61

T/F: 4th step in building a regression tree, average the test errors from the k-fold validation for each value of the tuning parameter alpha, and pick alpha that minimizes the average error.

Answer 61

True

Question 62

T/F: The final step is to return to the full data set and obtain the subtree corresponding to alpha that minimizes the average error.

Answer 62

True

Question 63

Three examples of greedy algorithms

Answer 63

Recursive binary splitting, forward stepwise selection, backward stepwise selection

Question 64

T/F: Single decision tree models generally have higher variance than random forest models.

Answer 64

True. Random forest models average the result of each individual tree to obtain a single prediction for each observation. The action of averaging reduces the variance.

Question 65

T/F: Random forests provide an improvement over bagging because predictions from the trees in a random forest are less correlated than those in bagged trees.

Answer 65

True. Random forests consider a random subset of predictors at each split to decorrelate trees.

Question 66

T/F: Classification error is sufficiently sensitive for tree-growing.

Answer 66

False. It’s sufficiently sensitive for pruning but not tree growing :(

Question 67

T/F: The Gini index is a measure of total variance across the K classes.

Answer 67

True

Question 68

T/F: The Gini index takes on a small value only if all of the Pmk’s are all near zero.

Answer 68

False. The Gini index takes on a small value if all of the
Pmk’s are all near zero or near one. For this reason the Gini index is referred to as a measure of node purity—a small value indicates that a node contains predominantly observations from a single class.

Question 69

T/F: The entropy will take on a value near zero only if the Pmk’s are all near zero.

Answer 69

False. The entropy takes on a small value if all of the
Pmk’s are all near zero or near one. For this reason the Gini index is referred to as a measure of node purity—a small value indicates that a node contains predominantly observations from a single class.

Question 70

T/F: Gini index and the entropy are quite different numerically.

Answer 70

False. Gini index and the entropy are quite similar numerically.

Question 71

T/F: Bagging is a procedure used to reduce bias of a statistical learning method.

Answer 71

False. Bagging is a procedure used to reduce variance of a statistical learning method.

Question 72

T/F: Pruning is a technique applied in the construction of bagging models.

Answer 72

False. Pruning is not typically used in bagging, random forests, or boosting as part of their standard methodologies.

Question 73

T/F: Random forests utilize pruning to reduce the complexity of individual trees.

Answer 73

False.

Question 74

T/F: Boosting models are pruned to prevent overfitting.

Answer 74

False. The complexity control is achieved through parameters like the number of trees (B), the learning rate, and the depth of the trees (d) rather than pruning.

Question 75

T/F: The out-of-bag error is a valid estimate of the test error for a bagged model.

Answer 75

True.

Question 76

T/F: In bagging, the bagged trees are grown deep and then pruned.

Answer 76

False. In bagging, the bagged trees are grown deep and not pruned. The idea is to produce trees that each has a high variance but a low bias. Averaging the trees reduces the variance.

Question 77

T/F: Random forests are better than bagging at thoroughly exploring the model space.

Answer 77

True. In bagging, because each bagged tree is trained independently, the tree may converge to one of the suboptimal solutions (local optima) that are not the best overall solution for the entire ensemble.

Question 78

T/F: In random forests, fitting a very large number of trees will not lead to overfitting.

Answer 78

True. It is also true for bagging. After a sufficient number of trees, the test error will settle down and flatline.

Question 79

T/F: Boosting generally requires growing smaller trees than random forests do.

Answer 79

True. This is because, in boosting, the growth of a particular tree takes into account the other trees that have been grown, unlike in random forests where the trees are grown independently of each other.

Question 80

T/F: Cost complexity pruning is a way to select a small set of subtrees for consideration, also known as weakest link pruning.

Answer 80

True. Weakest link pruning is a method used to select a limited number of subtrees from a larger set of possibilities. This approach focuses on identifying and pruning the weakest links or least important branches in the decision tree, resulting in a simpler and more interpretable model.

Question 81

T/F: Rather than considering every possible subtree, this method (decision trees?) considers a sequence of trees indexed by a nonnegative tuning parameter α.

Answer 81

True. Instead of exhaustively considering every potential subtree, weakest link pruning involves evaluating a sequence of trees indexed by a nonnegative tuning parameter α. This parameter controls the complexity of the tree, allowing for the exploration of different tree sizes and structures.

Question 82

T/F: As the tuning parameter α value increases, there is a price to pay for having a tree with many terminal nodes, so the error sum of squares plus the number of terminal nodes will be minimized for a smaller subtree.

Answer 82

False. As the tuning parameter increases, there is a price to pay for having a tree with many terminal nodes, and so the RSS plus the number of terminal nodes times the tuning parameter α will tend to be minimized for a smaller subtree.

Question 83

T/F: One difference between boosting and random forest is that in boosting, smaller trees are typically not sufficient.

Answer 83

False. In boosting, because the growth of a particular tree takes into account the other trees that have already been grown, smaller trees are typically sufficient.

Question 84

T/F: Often d=1 works well in boosting.

Answer 84

True. Often d=1 works well, in which case each tree is a stump, consisting of a single split. In this case, the boosted ensemble is fitting an additive model, since each term involves only a single variable.

Question 85

T/F: In a bagged model, an average of about one-third of the observations are used to train each bagged tree.

Answer 85

False. In a bagged model, an average of about 2/3 of the observations are used to train each bagged tree.

Question 86

T/F: For random forests, the more predictors are considered at each split, the greater the decorrelation benefit.

Answer 86

False. For random forests, the fewer predictors are considered at each split, the greater the decorrelation benefit.

Question 87

T/F: If the training dataset has a strong predictor and a few moderately strong predictors, a random forest model offers greater variance reduction than a bagged model.

Answer 87

True. This is due to the random forest model only considering a random subset of predictors at each split, preventing a strong predictor from dominating the top split and producing similar bagged trees that result in limited variance reduction.

Question 88

T/F: In a boosted model, the shrinkage parameter determines the number of splits in each tree.

Answer 88

False. In a boosted model, the interaction depth determines the number of splits in each tree.

Question 89

T/F: In a boosted model, each tree is independent from the previous tree.

Answer 89

False. In a boosted model, the trees are grown sequentially, with each tree fitted to the residuals of the previous tree, which creates dependency between the trees.

Question 90

T/F: The classification error rate is a measure directly related to the most commonly occurring class in a node.

Answer 90

True

Question 91

T/F: Node purity increases are always accompanied by decreases in the classification error rate after a split.

Answer 91

False

Question 92

T/F: Boosting reduces bias

Answer 92

True

Question 93

T/F: To apply bagging to regression trees, we simply construct B regression trees using a single bootstrapped training set, and average the resulting predictions.

Answer 93

False. To apply bagging to regression trees, we simply construct B regression trees using α multiple bootstrapped training sets, and average the resulting predictions.

Question 94

T/F: Regression trees are grown deep, and are not pruned for bagging.

Answer 94

True

Question 95

T/F: Each individual bagged tree has high variance, but low bias.

Answer 95

True

Question 96

T/F: Averaging these B (bagged) trees reduces the variance.

Answer 96

True

Question 97

T/F: Bagging has been demonstrated to give impressive improvements in accuracy by combining together hundreds or even thousands of trees into a single procedure.

Answer 97

True

Question 98

T/F: To obtain a single prediction for the i-th observation, we average the predicted responses obtained using each of the trees where the observation is OOB if regression is the goal or take a majority vote if classification is the goal.

Answer 98

True

Question 99

T/F: An OOB prediction can be obtained for each of the entire set of n observations, from which the overall OOB MSE for a regression problem or classification error for a classification problem can be computed.

Answer 99

True

Question 100

T/F: Recursive binary splitting divides the predictor space into non-overlapping regions.

Answer 100

True. Recursive binary splitting creates a hierarchical partitioning of the predictor space, where each split further subdivides the predictor space into smaller regions.

Question 101

T/F: Recursive binary splitting can make non-orthogonal splits or splits that are not aligned with the axes of the predictor space.

Answer 101

False. Recursive binary splitting can only make orthogonal splits or splits that are aligned with the axes of the predictor space.

Question 102

T/F: Setting a strict stopping criterion to build a small tree is preferable to building a large tree and pruning it back.

Answer 102

False. Building a large tree and pruning it back is preferable to setting a strict stopping criterion to build a small tree.

Question 103

T/F: Cost complexity pruning produces a series of subtrees, as a function of the tuning parameter α, that may or may not be nested.

Answer 103

False. Cost complexity pruning produces a sequence of nested subtrees.

Question 104

T/F: It is not possible for cross-validation to select a stump as the best subtree.

Answer 104

False. If cross-validation chooses a tuning parameter value that corresponds to a stump, then the stump is the best subtree.

Question 105

In a classification tree, a split at the bottom of the tree creates two terminal nodes that have the same predicted value.

T/F: Both terminal nodes are pure.

Answer 105

False. In order for a split to produce two pure terminal nodes with the same predicted value, the node must be pure before the split, in which case the split would not happen as there would be no node purity improvement to be made.

Question 106

In a classification tree, a split at the bottom of the tree creates two terminal nodes that have the same predicted value.

T/F: The split could increase confidence in the predicted value.

Answer 106

True. The increased node purity that results from this split increases our confidence in the predicted value, especially if a test observation belongs to the purer of the two terminal nodes.

Question 107

In a classification tree, a split at the bottom of the tree creates two terminal nodes that have the same predicted value.

T/F: The split decreases the classification error rate.

Answer 107

False. It does not decrease the classification error rate.

Question 108

In a classification tree, a split at the bottom of the tree creates two terminal nodes that have the same predicted value.

T/F: The split does not decrease the Gini index.

Answer 108

False. The split does decrease the Gini index.

Question 109

In a classification tree, a split at the bottom of the tree creates two terminal nodes that have the same predicted value.

T/F: The split does not decrease the entropy.

Answer 109

False. The split does decrease the entropy.

Question 110

T/F: Decision trees generally have better predictive accuracy compared to other statistical methods.

Answer 110

False. They generally have worse predictive accuracy compared to other statistical methods.

Question 111

T/F: Decision trees provide accurate in-sample testing results, but exhibit poor performance in out-of-sample testing.

Answer 111

True. Decision trees are prone to overfitting, especially when the trees are grown deep. In cases like this, they fit the training data well, but do not fit the test data well.

Question 112

T/F: It is difficult to measure variable importance in decision trees.

Answer 112

False. Identifying important variables in decision trees is straightforward, as these variables typically appear at the top of the tree.

Question 113

T/F: Pruning a tree increases its residual sum of squares.

Answer 113

True

Question 114

T/F: When using cost complexity pruning, the penalty decreases as the number of terminal nodes in the subtree increases.

Answer 114

False. The penalty increases as the number of terminal nodes in the subtree increases.

Question 115

T/F: A decision tree considers all predictors X1, … Xp, and all possible values of the cutpoint for each of the predictors, and then chooses the predictor and cutpoint such that the resulting tree has the lowest residual sum of squares.

Answer 115

True

Question 116

T/F: The difference between random forests and bagging is that decorrelated trees are utilized in random forests.

Answer 116

True. When building trees in bagging, a full set of predictors is considered at each split. However, only a subset of predictors was considered at each split in random forests. This makes trees less correlated and the average of the resulting trees more reliable.

Question 117

T/F: The best hyperparameters are usually possible to determine beforehand.

Answer 117

False.