19-Ensembe learning

Question 1

What is ensemble learning?

Answer 1

Ensemble learning constructs a set of base classifiers from a given set of training data and aggregates the outputs into a single meta classifier

Question 2

What are the approaches for ensemble learning?

Answer 2

Instance manipulation
Feature manipulation
Class label manipulation
Algorithm manipulation

Question 3

What is instance manipulation?

Answer 3

Generate multiple training datasets through sampling and train a base classifier over each dataset

Question 4

What is feature manipulation in the context of ensemble learning?

Answer 4

Generate multiple training datasets through different feature subsets and train a base classifier over each dataset

Question 5

What is class label manipulation?

Answer 5

Generate multiple training datasets by manipulating the class labels in a reversible manner

Question 6

What is algorithm manipulation?

Answer 6

Semi randomly tweak internal parameters within an algorithm to generate multiple base classifiers over a given dataset

Question 7

What is the intuition behind ensemble learning?

Answer 7

1. Combination of lots of weak classifiers can be at least as good as 1 strong classifier
2. A combination of strong classifiers is at least as good as the best of the classifiers

Question 8

What is the relationship between base and ensemble classifiers error if they’re independent?

Answer 8

Logit functions

Question 9

What is the relationship between base and ensemble classifiers error if they’re identical?

Answer 9

Linear

Question 10

What is stacking?

Answer 10

Use different algorithms to train multiple base classifiers
Use base classifiers to generate predictions on unseen samples

Question 11

What are the pros of stacking?

Answer 11

Mathematically simple
Able to combine heterogeneous classifiers
Generally results in as good or better results than the best of the base classifiers

Question 12

What are the cons of stacking?

Answer 12

Computationally expensive

Question 13

What is bagging?

Answer 13

Bagging is used to reduce variance.

Create multiple training datasets for training multiple classifiers based on the same algorithm and average the predictions

Question 14

How do we generate datasets?

Answer 14

Randomly sample the original dataset (N instances) N times, with replacement. Any individual instance is absent with probability (1-1/N)^N

Question 15

What is the benefit of bagging?

Answer 15

Possibility to parallelise computation of individual base classifiers
Highly effective over noisy datasets
Produces the best results on unstable models that have high variance and low bias

Question 16

What is a random forest?

Answer 16

Ensemble of random trees. Random trees are decision trees where only some attributes are considered at a node

Question 17

What are the benefits of random forest?

Answer 17

Parallelisable and robust to overfitting

Question 18

What are the cons of random forest?

Answer 18

Sacrifices interpretability

Question 19

What is boosting?

Answer 19

Iteratively change the weights of training instances to train next base classifier and combine the base classifiers via weighted voting

Question 20

What is adaboost?

Answer 20

A boosting algorithm

Question 21

What’s the difference between how boosting and bagging build models?

Answer 21

Bagging builds models in parallel. Boosting builds models sequentially

Question 22

What’s the difference between how boosting and bagging sample data?

Answer 22

Bagging resamples data points with replacements. Boosting uses iterative sampling. It reweights data points by modifying their distribution

Question 23

What’s the difference between how boosting and bagging weight models?

Answer 23

Bagging has the same weight for each model. Boosting has different weights

Question 24

What’s the difference between how boosting and bagging impact evaluation?

Answer 24

Bagging reduces variance. Boosting reduces bias

Question 25

What’s the difference between how boosting and bagging fit data?

Answer 25

Bagging is not prone to overfitting. Boosting is prone to overfitting