Ensemble

Question 1

How do ensemble methods work?

Answer 1

They work by combining predictions from several estimators built with a given learning algorithm in order to improve generalization.

Question 2

What kind of methods are used?

Answer 2

• averaging - reduce variance of ‘strong’ estimators

- boosting - reduce bias of ‘weak’ estimators

Question 3

How does averaging work?

Answer 3

It works by building several estimators independently and averaging their predictions to reduce the variance.

Question 4

How does boosting work?

Answer 4

It works by building sequentially several estimators such that the combined estimator has a reduced bias.

Question 5

What is ‘Pasting’ - averaging methods?

Answer 5

Algorithm that uses random subsets of samples drawn randomly from the dataset for its independent estimators

Question 6

What is ‘Bagging’?

Answer 6

For averaging methods this means that the samples are drawn with replacement.

Question 7

What is ‘Random Subspaces’ - averaging methods?

Answer 7

Algorithm that uses random subsets of samples drawn as random subsets of features.

Question 8

What is ‘Random Patches’ - averaging methods?

Answer 8

Algorithm that uses random subsets of samples drawn as random subsets of both data and features.

Question 9

RandomForests and Extra-Trees - averaging methods

Answer 9

They are combine-and-perturb methods based on constructing randomized decision trees and then averaging their prediction results.

Question 10

Differences between RandomForests and ExtraTrees

Answer 10

During tree construction:

• in RFs the node split is picked based on the best split among a random subset of features
• in ETs the splits are drawn at random for each candidate feature with the best one being picked for the node split

Question 11

What is bias?

Answer 11

It is the error from erroneous assumptions in the learning algorithm.
High bias can cause an algorithm to miss the relevant relations between features and target outputs (underfitting).

Question 12

What is variance?

Answer 12

It is error from sensitivity to small fluctuations in the training set.
High variance can cause overfitting: modeling the random noise in the training data, rather than the intended outputs.

Question 13

What is a decision tree?

Answer 13

It is a model for predicting a dependent variable Y using an independent var X by checking a collection of splits..

Question 14

What is a decision tree split?

Answer 14

A split is a condition or query on a single independent variable that is either true or false.
Splits are arranged as a tree with 2 child nodes: left for true condition, right for false condition

Question 15

Boosting intuition

Answer 15

minimizes bias by using estimators with low variance and high bias (i.e. shallow decision trees)

Question 16

Bagging intuition

Answer 16

minimizes variance by using estimators low bias and high variance (i.e. full decision trees)

Question 17

What is Gini impurity?

Answer 17

• a measure of how often a randomly chosen element from a set will be classified incorrectly
• the classification is done randomly according to the distribution of the labels in the set

Question 18

What is information gain?

Answer 18

• a measure given by the difference between the entropy of, for example, the target variable and the entropy of the target variable conditioned by a regressor
  IG(T, rgr) = H(T) - H(T | rgr)

Question 19

What is ‘reduced error’ pruning?

Answer 19

• method of reducing overfitting in decision trees
• ‘bottom-up’ algo
• starting at the leaves, each node is replaced with the most popular class. If prediction accuracy is improved, the change is kept.