12. Ensemble Method*

Question 1

decision trees

Answer 1

while easy to implement, train, use and interpret, it is inaccurate

Question 2

core idea of ensemble method

Answer 2

when weak models are carefully trained and combined, they should be able to produce more accurate and robust prediction results

Question 3

what are the ensemble methods

Answer 3

1. bagging (bootstrap aggregation)
   - create diverse sample data with replacement
   - train model with subset of data and aggregate -> learn B weak learners
   - aggregate predictions from all learners
   - achieve good results by reducing variance
2. boosting
   - training a sequence of weak models, each compensating the weaknesses of its predecessors
   - correct predicting error
   - achieve good results by decreasing the bias
3. stacking
   - train multiple models from the same training dataset and use another model to aggregate prediction

Question 4

random forest

Answer 4

improve variance deduction of bagging by reducing the correlation between base models without increasing the variance

p * sigma^2 (1 - p) * sigma^2 / n

how does random forest work..?

Question 5

adaboost**

Answer 5

put more weight on difficult instances and less on those already learned

1. set weights of all samples
2. bootstrap k samples based on weights
3. fit leaner XX on sample data
4. evaluate weak learner G on dataset and calculate loss
5. compute impacting factor of alpha
6. update sample weights based on e
7. update G
8. repeat till terminate

Question 6

gradient boost**

Answer 6

learn the errors (pseudo residuals) made from predecessors

Question 7

what are the advantages to ensemble

Answer 7

1. able to learn complex problems
2. generally high accuracy

Question 8

what are the disadvantages to ensemble

Answer 8

1. might not fit well in simple problems
2. lack of interpretability
3. computationally expensive