Chapter 15 Improve Performance with Ensembles

Question 1

EXTERNAL Q: WHY DOESN’T GRADIENT BOOST USE SAMPLE WITH REPLACEMENT?

Answer 1

in default mode, in python, gradient boost subsample parameter is equal to 1, so by default it doesn’t use sampling with replacement. But we can change it to a number (0,1] to have subsamples in this case, based on the link, sample with replacement is done, this reduces variance and increase bias

Question 2

WHAT IS THE DIFFERENCE BETWEEN STOCHASTIC GRADIENT BOOST AND RANDOM FOREST IN SAMPLING

Answer 2

Stochastic gradient descent (as used by XGBoost) adds randomness by sampling observations and features in each stage, which is similar to the random forest algorithm except the sampling is done without replacement

Question 3

WHAT ARE THE THREE MOST POPULAR METHODS FOR COMBINING THE PREDICTIONS FROM DIFFERENT MODELS? P100

Answer 3

ˆ Bagging. Building multiple models (typically of the same type) from different subsamples of the training dataset.
ˆ Boosting. Building multiple models (typically of the same type) each of which learns to fix the prediction errors of a prior model in the sequence of models.
ˆ Voting. Building multiple models (typically of differing types) and simple statistics (like calculating the mean) are used to combine predictions.

Question 4

WHAT IS ANOTHER NAME FOR BAGGING? P101

Answer 4

Bootstrap Aggregation

Question 5

WHAT IS ANOTHER NAME FOR BAGGING? P101

Answer 5

Bootstrap Aggregation

Question 6

HOW IS THE BAGGING PROCESS DONE? P101

Answer 6

By taking multiple samples from your training dataset (with replacement) and training a model for each sample. The final output prediction is averaged across the predictions of all of the sub-models

Question 7

WITH WHICH TYPE OF ALGORITHM DOES BAGGING PERFORM THE BEST? P101

Answer 7

High variance

Question 8

WHAT IS THE SIMILARITY AND THE DIFFERENCE BETWEEN RANDOM FOREST AND BAGGED DECISION TREES? P102

Answer 8

SIMILARITY:
Random Forests is an extension of bagged decision trees.
Samples of the training dataset are taken with replacement.
DIFFERENCE: The trees in random forest are constructed in a way that reduces the correlation between individual classifiers, Specifically, rather than greedily choosing the best split point in the construction of each tree, only a random subset of features are considered for each split

Question 9

HOW DOES BOOSTING ALGORITHMS WORK? P103

Answer 9

Boosting ensemble algorithms create a sequence of models that attempt to correct the mistakes of the models before them in the sequence.

Question 10

HOW DO BOOSTING ALGORITHMS WORK? P103

Answer 10

Boosting ensemble algorithms create a sequence of models that attempt to correct the mistakes of the models before them in the sequence.

Question 11

WHAT ARE THE TWO MOST COMMON BOOSTING ENSEMBLE ML ALGORITHMS? P103

Answer 11

ˆ AdaBoost.

ˆ Stochastic Gradient Boosting (also called Gradient Boosting Machines)

Question 12

HOW DOES ADABOOST WORK? P103

Answer 12

It generally works by weighting instances in the dataset by how easy or difficult they are to classify, allowing the algorithm to pay more or less attention to them in the construction of subsequent models.

Question 13

WHAT IS THE DIFFERENCE BETWEEN STACKING AND VOTING ENSEMBLE MODELS? P105

Answer 13

The predictions of the sub-models can be weighted, but specifying the weights for classifiers manually or even heuristically is difficult. More advanced methods can learn how to best weight the predictions from sub-models, but this is called stacking (stacked aggregation). In stacking, a weight is dedicated to each model, in Voting the results are combined using a simple statistics like mean.