Wronged Questions: Decision Trees Flashcards
T/F: Small shrinkage parameter requires more iterations because it has a slower learning rate
True
T/F: Boosting can lead to overfitting if you have a high number of iterations
True
As K increases, flexibility (increases/decreases).
Decreases
Classification rate, gini index, and entropy are (inappropriate/appropriate) when pruning a tree.
Appropriate
T/F: Decision trees are easier to interpret than linear models.
True
T/F: Decision trees are more robust than linear models.
False. Decision trees are generally less robust than linear models; they can produce significantly different outcomes with small changes in the input data.
T/F: Decision trees handle qualitative predictors more easily than linear models.
True. Decision trees naturally handle qualitative (categorical) predictors without the need for preprocessing steps such as creating dummy variables, which are often required in linear models.
T/F: In boosting, the number of terminal nodes in each tree is independent of the number of splits.
False. In boosting, the number of terminal nodes in each tree is directly related to the number of splits. The number of terminal nodes (leaves) in a tree is one more than the number of splits.
T/F: Boosting does not allow for the adjustment of model complexity through the parameter.
False. Parameter d is specifically used to adjust the complexity of the model in boosting.
T/F: Boosting considers only a random subset of predictors at each node in every tree.
False. In classical boosting algorithms, all available predictors are considered at each split, not a random subset.
T/F: A smaller value of d in boosting necessitates a larger number of trees to adequately model the data.
True. A smaller implies simpler base learners (trees), which individually capture less of the data’s complexity. Therefore, more trees are needed to aggregate enough information to model the data effectively.
T/F: Like bagging, boosting is a general approach that can be applied to many statistical learning methods for regression or classification.
True
T/F: Each tree is fit on a modified version of the bootstrapped samples for boosting.
False. Boosting does not involve bootstrap sampling; instead each tree is fit on a modified version of the original data set.
T/F: Unlike fitting a single large decision tree to the data, which amounts to fitting the data hard and potentially overfitting, the boosting approach instead learns slowly.
True
T/F: In boosting, unlike in bagging, the construction of each tree depends strongly on the trees that have already been grown.
True
T/F: Like bagging, boosting involves combining a large number of decision trees.
True
T/F: Unlike bagging and random forests, boosting can overfit if B is too large.
True
T/F: Cross-validation is used to select B.
True
T/F: A very small value of the shrinkage parameter can require using a very large number of trees to achieve good performance.
True
T/F: An interaction depth of zero often works well and the boosted ensemble is fitting an additive model.
False. An interaction depth of one often works well and the boosted ensemble is fitting an additive model.
T/F: In boosting, because the growth of a particular tree takes into account the other trees that have already been grown, smaller trees are typically sufficient.
True
T/F: Individual trees in a random forest are left unpruned, contributing to the ensemble’s variance reduction despite their own overfitting.
True. In a random forest, individual trees are typically grown to their full depth without pruning, which might make them prone to overfitting. However, when these overfitted trees are aggregated, the ensemble model achieves a significant reduction in variance.
T/F: The combination of results from unpruned trees in a random forest leads to a reduction in the overall variance of the model.
True. Emphasizing the ensemble effect where the aggregation of multiple unpruned, overfitted trees results in a model with reduced overall variance, leveraging the strength of the ensemble to balance out individual tree overfitting.
T/F: Increasing m leads to a higher degree of decorrelation between the trees, where m is the number of predictors chosen as split candidates at each split.
False. As a larger value of m tends to increase the correlation between trees. The parameter m is the number of predictors chosen as split candidates at each split.
T/F: Random forests can effectively handle both regression and classification problems.
True. It acknowledges the versatility of random forests in addressing different types of prediction problems.
T/F: A variable importance plot is a useful tool for identifying which predictors are most influential in a random forest model.
True. Variable importance plots are indeed utilized to identify the most important predictors in a random forest, offering insights into how different variables contribute to the predictive power of the model.
T/F: Bagging and random forests make use of bootstrapped samples in their algorithms, but boosting does not.
True. Bagging and random forests employ bootstrapped samples to build multiple decision trees, enhancing model accuracy and robustness by aggregating their predictions.
Conversely, boosting sequentially constructs trees, each focusing on correcting errors from previous ones, without utilizing bootstrapped samples to improve performance.
T/F: Boosting can overfit if the number of iterations is set too high, unlike bagging or random forests.
True. Boosting’s performance can be sensitive to the number of iterations, leading to potential overfitting.
T/F: The optimal number of iterations in boosting is often determined through cross-validation.
True. Cross-validation is commonly used to select the optimal B in boosting to balance bias and variance.
T/F: For bagging and random forests, the choice of B is less critical to avoiding overfitting compared to boosting.
True. Bagging and random forests are generally robust against overfitting due to their aggregation methods, making the specific choice of B less critical.
T/F: Pruning is a common strategy in both bagging and boosting to prevent overfitting.
False. Pruning is not used in either bagging or boosting as a strategy to prevent overfitting.
T/F: Bagging significantly reduces variance by averaging multiple predictions.
True. One of the primary advantages of bagging is its ability to reduce the variance of complex models, like deep/large decision trees, by averaging the predictions of multiple bootstrapped models, which tends to make the ensemble prediction more robust than any single model.
T/F: Each bagged tree uses approximately one-third of observations from the original training set.
False. In bagging, each tree, on average, makes use of around two-thirds of the observations due to the nature of bootstrap sampling, where some observations are repeated, and others are left out.
T/F: Bagging is exclusively effective for decision trees and cannot be applied to other statistical learning methods.
False. Bagging is a general-purpose procedure that can be applied to many types of statistical learning methods, not just decision trees, although it is particularly beneficial for models that exhibit high variance.
T/F: On average, (p-m)/p of the splits will not even consider the strong predictor.
True
T/F: The main difference between bagging and random forests is the choice of predictor subset size.
True
T/F: If a random forest is built using m = p^1/2, then this amounts to bagging.
False. If a random forest is built using m = p, then this amounts to bagging.
T/F: Using a small value of m in building a random forest will typically be helpful when we have a large number of correlated predictors.
True
T/F: Random forests will not overfit if we increase B, so in practice we use a value of B sufficiently large for the error rate to have settled down.
True
T/F: An alpha value of zero results in the largest, unpruned tree.
True. An a value of zero implies no penalty on the tree’s complexity, and thus the tree grows to its largest size without any pruning. This results in the most complex tree possible.
T/F: Increasing alpha leads to a decrease in the variance of the model.
True. As the tree becomes simpler with higher a
values, its variance decreases due to less model flexibility.
T/F: Increasing alpha leads to a decrease in the squared bias of the fitted tree.
False. Increasing alpha leads to an increase in the squared bias of the fitted tree. This is because increasing the value of a in cost complexity pruning penalizes the addition of splits to the tree, resulting in a simpler tree (lower variance).
A simpler tree is less flexible in fitting the data, which leads to an increase in the squared bias as the model becomes increasingly unable to capture the underlying patterns in the data.
Three non-parametric statistical learning methods
KNN, Decision Trees, Bagging/Random Forest/Boosting
T/F: To build each tree using random forests, a bootstrapped sample of n observations is used, and for each split within the tree, a new random selection of m predictors is made.
True. In random forests, each tree is built from a bootstrapped sample of the original dataset, containing n observations. At each split in the construction of a tree within a random forest, a random subset of m predictors is selected from all available predictors.
T/F: Out-of-bag estimation can be used to estimate the test error for random forests.
True. Out-of-bag estimation, which uses each observation’s predictions from trees where that observation was not in the bootstrap sample, provides an estimate of the test error without needing a separate test set.
T/F: Random forests reduce bias through the averaging of multiple decorrelated trees.
False. It is incorrect because the main benefit of random forests is variance reduction, not necessarily bias reduction. Random forests reduce variance by averaging the results of multiple decorrelated trees. While this ensemble method is effective at addressing overfitting and reducing variance, the reduction in bias is not guaranteed.
T/F: Pruning always decreases both training and test error rates.
False. Pruning does not always decrease both error rates.
T/F: Pruning reduces the training error rate but may increase the test error rate.
False. Pruning typically increases the training error rate due to less perfect fitting to the training data, the impact on the test error rate can be mixed.
T/F: Pruning increases the training error rate but has a predictable effect on reducing the test error rate.
False. There is usually not a predictable effect on the test error rate.
T/F: Pruning tends to increase the training error rate, but its effect on the test error rate can vary.
True. Pruning a classification tree involves removing splits that provide less generalizable decision-making ability, leading to a simpler model. While this process makes the tree less flexible and typically increases the training error rate due to less perfect fitting to the training data, the impact on the test error rate can be mixed.
T/F: Pruning a classification tree has no impact on the training error rate but tends to decrease the test error rate.
False. Pruning a classification tree increases the training error rate. We still do it to prevent overfitting.
T/F: Bagging increases the interpretability of a model.
False. Bagging does not make the model more interpretable; in fact, it makes it less interpretable.
T/F: Out-of-bag (OOB) error estimation requires each observation to be predicted by all trees in the ensemble.
False. OOB error estimation uses only the trees for which the specific observation was not in the bootstrap sample.
T/F: A very high number of bootstrap samples will inevitably lead to overfitting in bagged models.
False. Increasing the number of trees does not lead to overfitting due to the aggregation of predictions.
T/F: Bagging cannot improve prediction accuracy in classification settings.
False. Bagging is indeed useful for improving prediction accuracy in classification settings.
T/F: For a sufficiently large number of bootstrap samples, out-of-bag error is virtually equivalent to leave-one-out-cross-validation error.
True. For a sufficiently large number of trees, out-of-bag error is virtually equivalent to leave-one-out cross-validation error. OOB error estimation, which uses each tree’s predictions on observations not included in its bootstrap sample, provides a reliable estimate of the model’s performance that becomes increasingly accurate as the number of bootstrap samples increases.
T/F: Each bootstrapped dataset in bagging contains a completely different set of observations from the original dataset.
False. Each bootstrapped dataset likely contains repeated observations due to sampling with replacement, and not all observations will be different.
T/F: First step in building a regression tree: Use recursive binary splitting to grow a large tree on the training data, stopping only when each terminal node has fewer than some minimum number of observations.
True
T/F: Second step in building a regression tree, apply cost complexity pruning to the large tree in order to obtain a sequence of best subtrees, as a function of a nonnegative tuning parameter.
True
T/F: Third step in building a regression tree, utilize k-fold cross-validation to select the tuning parameter α by evaluating the mean squared prediction error on the data in the left-out kth fold, as a function of k.
False. utilize k-fold cross-validation to select the tuning parameter α by evaluating the mean squared prediction error on the data in the left-out kth fold, as a function of α.
T/F: 4th step in building a regression tree, average the test errors from the k-fold validation for each value of the tuning parameter alpha, and pick alpha that minimizes the average error.
True
T/F: The final step is to return to the full data set and obtain the subtree corresponding to alpha that minimizes the average error.
True
Three examples of greedy algorithms
Recursive binary splitting, forward stepwise selection, backward stepwise selection
T/F: Single decision tree models generally have higher variance than random forest models.
True. Random forest models average the result of each individual tree to obtain a single prediction for each observation. The action of averaging reduces the variance.
T/F: Random forests provide an improvement over bagging because predictions from the trees in a random forest are less correlated than those in bagged trees.
True. Random forests consider a random subset of predictors at each split to decorrelate trees.
T/F: Classification error is sufficiently sensitive for tree-growing.
False. It’s sufficiently sensitive for pruning but not tree growing :(
T/F: The Gini index is a measure of total variance across the K classes.
True
T/F: The Gini index takes on a small value only if all of the Pmk’s are all near zero.
False. The Gini index takes on a small value if all of the
Pmk’s are all near zero or near one. For this reason the Gini index is referred to as a measure of node purity—a small value indicates that a node contains predominantly observations from a single class.
T/F: The entropy will take on a value near zero only if the Pmk’s are all near zero.
False. The entropy takes on a small value if all of the
Pmk’s are all near zero or near one. For this reason the Gini index is referred to as a measure of node purity—a small value indicates that a node contains predominantly observations from a single class.
T/F: Gini index and the entropy are quite different numerically.
False. Gini index and the entropy are quite similar numerically.
T/F: Bagging is a procedure used to reduce bias of a statistical learning method.
False. Bagging is a procedure used to reduce variance of a statistical learning method.
T/F: Pruning is a technique applied in the construction of bagging models.
False. Pruning is not typically used in bagging, random forests, or boosting as part of their standard methodologies.
T/F: Random forests utilize pruning to reduce the complexity of individual trees.
False.
T/F: Boosting models are pruned to prevent overfitting.
False. The complexity control is achieved through parameters like the number of trees (B), the learning rate, and the depth of the trees (d) rather than pruning.
T/F: The out-of-bag error is a valid estimate of the test error for a bagged model.
True.
T/F: In bagging, the bagged trees are grown deep and then pruned.
False. In bagging, the bagged trees are grown deep and not pruned. The idea is to produce trees that each has a high variance but a low bias. Averaging the trees reduces the variance.
T/F: Random forests are better than bagging at thoroughly exploring the model space.
True. In bagging, because each bagged tree is trained independently, the tree may converge to one of the suboptimal solutions (local optima) that are not the best overall solution for the entire ensemble.
T/F: In random forests, fitting a very large number of trees will not lead to overfitting.
True. It is also true for bagging. After a sufficient number of trees, the test error will settle down and flatline.
T/F: Boosting generally requires growing smaller trees than random forests do.
True. This is because, in boosting, the growth of a particular tree takes into account the other trees that have been grown, unlike in random forests where the trees are grown independently of each other.
T/F: Cost complexity pruning is a way to select a small set of subtrees for consideration, also known as weakest link pruning.
True. Weakest link pruning is a method used to select a limited number of subtrees from a larger set of possibilities. This approach focuses on identifying and pruning the weakest links or least important branches in the decision tree, resulting in a simpler and more interpretable model.
T/F: Rather than considering every possible subtree, this method (decision trees?) considers a sequence of trees indexed by a nonnegative tuning parameter α.
True. Instead of exhaustively considering every potential subtree, weakest link pruning involves evaluating a sequence of trees indexed by a nonnegative tuning parameter α. This parameter controls the complexity of the tree, allowing for the exploration of different tree sizes and structures.
T/F: As the tuning parameter α value increases, there is a price to pay for having a tree with many terminal nodes, so the error sum of squares plus the number of terminal nodes will be minimized for a smaller subtree.
False. As the tuning parameter increases, there is a price to pay for having a tree with many terminal nodes, and so the RSS plus the number of terminal nodes times the tuning parameter α will tend to be minimized for a smaller subtree.
T/F: One difference between boosting and random forest is that in boosting, smaller trees are typically not sufficient.
False. In boosting, because the growth of a particular tree takes into account the other trees that have already been grown, smaller trees are typically sufficient.
T/F: Often d=1 works well in boosting.
True. Often d=1 works well, in which case each tree is a stump, consisting of a single split. In this case, the boosted ensemble is fitting an additive model, since each term involves only a single variable.
T/F: In a bagged model, an average of about one-third of the observations are used to train each bagged tree.
False. In a bagged model, an average of about 2/3 of the observations are used to train each bagged tree.
T/F: For random forests, the more predictors are considered at each split, the greater the decorrelation benefit.
False. For random forests, the fewer predictors are considered at each split, the greater the decorrelation benefit.
T/F: If the training dataset has a strong predictor and a few moderately strong predictors, a random forest model offers greater variance reduction than a bagged model.
True. This is due to the random forest model only considering a random subset of predictors at each split, preventing a strong predictor from dominating the top split and producing similar bagged trees that result in limited variance reduction.
T/F: In a boosted model, the shrinkage parameter determines the number of splits in each tree.
False. In a boosted model, the interaction depth determines the number of splits in each tree.
T/F: In a boosted model, each tree is independent from the previous tree.
False. In a boosted model, the trees are grown sequentially, with each tree fitted to the residuals of the previous tree, which creates dependency between the trees.
T/F: The classification error rate is a measure directly related to the most commonly occurring class in a node.
True
T/F: Node purity increases are always accompanied by decreases in the classification error rate after a split.
False
T/F: Boosting reduces bias
True
T/F: To apply bagging to regression trees, we simply construct B regression trees using a single bootstrapped training set, and average the resulting predictions.
False. To apply bagging to regression trees, we simply construct B regression trees using α multiple bootstrapped training sets, and average the resulting predictions.
T/F: Regression trees are grown deep, and are not pruned for bagging.
True
T/F: Each individual bagged tree has high variance, but low bias.
True
T/F: Averaging these B (bagged) trees reduces the variance.
True
T/F: Bagging has been demonstrated to give impressive improvements in accuracy by combining together hundreds or even thousands of trees into a single procedure.
True
T/F: To obtain a single prediction for the i-th observation, we average the predicted responses obtained using each of the trees where the observation is OOB if regression is the goal or take a majority vote if classification is the goal.
True
T/F: An OOB prediction can be obtained for each of the entire set of n observations, from which the overall OOB MSE for a regression problem or classification error for a classification problem can be computed.
True
T/F: Recursive binary splitting divides the predictor space into non-overlapping regions.
True. Recursive binary splitting creates a hierarchical partitioning of the predictor space, where each split further subdivides the predictor space into smaller regions.
T/F: Recursive binary splitting can make non-orthogonal splits or splits that are not aligned with the axes of the predictor space.
False. Recursive binary splitting can only make orthogonal splits or splits that are aligned with the axes of the predictor space.
T/F: Setting a strict stopping criterion to build a small tree is preferable to building a large tree and pruning it back.
False. Building a large tree and pruning it back is preferable to setting a strict stopping criterion to build a small tree.
T/F: Cost complexity pruning produces a series of subtrees, as a function of the tuning parameter α, that may or may not be nested.
False. Cost complexity pruning produces a sequence of nested subtrees.
T/F: It is not possible for cross-validation to select a stump as the best subtree.
False. If cross-validation chooses a tuning parameter value that corresponds to a stump, then the stump is the best subtree.
In a classification tree, a split at the bottom of the tree creates two terminal nodes that have the same predicted value.
T/F: Both terminal nodes are pure.
False. In order for a split to produce two pure terminal nodes with the same predicted value, the node must be pure before the split, in which case the split would not happen as there would be no node purity improvement to be made.
In a classification tree, a split at the bottom of the tree creates two terminal nodes that have the same predicted value.
T/F: The split could increase confidence in the predicted value.
True. The increased node purity that results from this split increases our confidence in the predicted value, especially if a test observation belongs to the purer of the two terminal nodes.
In a classification tree, a split at the bottom of the tree creates two terminal nodes that have the same predicted value.
T/F: The split decreases the classification error rate.
False. It does not decrease the classification error rate.
In a classification tree, a split at the bottom of the tree creates two terminal nodes that have the same predicted value.
T/F: The split does not decrease the Gini index.
False. The split does decrease the Gini index.
In a classification tree, a split at the bottom of the tree creates two terminal nodes that have the same predicted value.
T/F: The split does not decrease the entropy.
False. The split does decrease the entropy.
T/F: Decision trees generally have better predictive accuracy compared to other statistical methods.
False. They generally have worse predictive accuracy compared to other statistical methods.
T/F: Decision trees provide accurate in-sample testing results, but exhibit poor performance in out-of-sample testing.
True. Decision trees are prone to overfitting, especially when the trees are grown deep. In cases like this, they fit the training data well, but do not fit the test data well.
T/F: It is difficult to measure variable importance in decision trees.
False. Identifying important variables in decision trees is straightforward, as these variables typically appear at the top of the tree.
T/F: Pruning a tree increases its residual sum of squares.
True
T/F: When using cost complexity pruning, the penalty decreases as the number of terminal nodes in the subtree increases.
False. The penalty increases as the number of terminal nodes in the subtree increases.
T/F: A decision tree considers all predictors X1, … Xp, and all possible values of the cutpoint for each of the predictors, and then chooses the predictor and cutpoint such that the resulting tree has the lowest residual sum of squares.
True
T/F: The difference between random forests and bagging is that decorrelated trees are utilized in random forests.
True. When building trees in bagging, a full set of predictors is considered at each split. However, only a subset of predictors was considered at each split in random forests. This makes trees less correlated and the average of the resulting trees more reliable.
T/F: The best hyperparameters are usually possible to determine beforehand.
False.
