Chapter 10- Ensemble Models

Question 1

what is the principle of committee decision?

Answer 1

individual predictions combined appropriately should have better overall accuracy on average than an individual committee member

Question 2

what three methods can be used to combine decisions?

Answer 2

averaging, voting, probablistic

Question 3

what is the wisdom of crowds?

Answer 3

different versions make errors in different ways. combining them cancels out the errors of the individuals

Question 4

what is bagging?

Answer 4

generate different training sets, by sampling with replacement to build a committee of models

Question 5

is bagging a sequential or parallel method?

Answer 5

parallel

Question 6

what are random forests?

Answer 6

an extension of bagging applied to decision trees.

Question 7

what are the two methods of generating randomisation for random forests?

Answer 7

bootstrap, as in bagging

random selection of features are each split point

Question 8

why do we force a random selection of feature at each split point in random forests?

Answer 8

to ensure it doesnt choose the best feature at every split.

Question 9

what does the lack of pruning of random forest trees ensure

Answer 9

that the models aren’t too simple and therefore too similar to each other.

Question 10

is adaboost parallel or sequential?

Answer 10

sequential

Question 11

describe the adaboost algorithm

Answer 11

a new model trained at each round. at the end of each round the mis-classified examples are identified and have their emphasis increased

Question 12

what is the key difference between bagging and boosting?

Answer 12

at each round, bagging has a uniform distribution while boosting adapts a non uniform distribution.

bagging is parallel, boosting is sequential.

Question 13

what is the main boosting algorithm?

Answer 13

adaboost

Question 14

what is the update of the distribution function in adaboost after each round?

Answer 14

examples from the erroneous portion make up 50% of the next training

Question 15

give the distribution update scheme for adaboost, i.e. what do we multiply Dj(i) by

Answer 15

1 / 2ej if the classification was incorrect

1 / 2(1-ej) if the classification was correct

Question 16

Generally, an ensemble method works better if the individual models have…

Answer 16

Less correlation among predictions

Question 17

Boosting is a family of algorithms, where

Answer 17

Each base model has a dataset drawn from a weighted distributed, proportional to the errors made by the previous model

Question 18

Bagging is an ensemble algorithm, where …. is used to ensure predictors are …

Answer 18

Bootstrapping is used to ensure predictors are less correlated in their errors

Question 19

where does the name bagging come from

Answer 19

bootstrap aggregating

Question 20

list common ML models in order or stability (most to least)

Answer 20

svm
knn
logistic regression/perceptrons
neural networks 
decision trees

Question 21

what kinds of models are completely unaffected by bootstrapping

Answer 21

stable