Ensemble Learning: Boosting Flashcards
What are the key differences between Random Forest and Boosting in terms of tree size and voting?
Random Forest: Uses random trees of typical size, where each tree usually has an equal vote in the aggregated decision.
Boosting: Uses stumps (one root, two leaves) where different stumps have different significance.
How do Random Forest and Boosting differ in terms of tree independence and order?
Each tree is made independently. Order is important, the error of one stump influences the next stump
What is the first rule of AdaBoost?
Combine stumps (i.e., weak learners)
What is the second rule of AdaBoost?
Some stumps influence the classification more than others.
What is the third rule of AdaBoost?
Each stump is built based on the error in the previous stumps
What is the error rate of boosting if the base classifier error rate is less than 50%?
If the base classifier (stump) produces a classifier with an error rate < 50%, boosting can give a ~0% error rate classifier.
Describe the basic framework of boosting.
1.Obtain the first classifier f1.
2.Find the other function f2 to help f1.
3.Ensure f2 is complementary to f1.
4.Obtain the second classifier f2.
5.Continue this process, and finally, combine all classifiers/
How is re-weighting of training data done in boosting?
-The weight of correctly classified samples is updated as π’(t+1) = π’(t) Γ π or as π’(t+1) = π’(t) Γ π^βΞ±.
-The weight of incorrectly classified samples is updated as π’(t+1) = π’(t) / π or as π’(t+1) = π’(t) Γ π^Ξ±.
-Where d = (1-error)/error
What is the general formula for re-weighting training data?
The general formula is π’(t+1) = π’(t) Γ π^βΕ·π(π₯)Ξ±, where Ε· and f(x) are Β±1 for binary classification.
-Ε· = f(x): Weight is updated by π^βΞ±.
-Ε· β f(x): Weight is updated by π^Ξ±
How are the classifiers aggregated in boosting?
Uniform weight: Simply add classifiers βf.
Non-uniform weight: Use weighted classifiers
When is Boosting preferred over Bagging, and vice versa?
Useful when subtle features matter (e.g., Face Recognition) and high precision is needed.
*Bagging: Useful when subtle reviews donβt matter (e.g., Sentiment Analysis from Customer Reviews) and there is high variance due to diversity.
How does model complexity differ between Bagging and Boosting?
Bagging uses complex models like deep decision trees, whereas boosting uses simple models such as stumps
How does model independence differ between Bagging and Boosting?
Bagging trains models independently, whereas boosting builds models sequentially.
How does model aggregation differ between Bagging and Boosting?
Bagging uses equal weighting or averaging, while boosting uses non-uniform weights depending on error.
What does each focus on?
Bagging is used to reduce variance, while boosting focuses on reducing bias.
