ML learn

Question 1

List xgboost benefits

Answer 1

L1l2 reg prevents overfitting on high dimensional space
Missing values handling
Cross-validation
Allow early stopping
Option to look at the learning graph to choose different checkpoint
Multiple cpu optimised
Goes to deep trees -> allows more optimised trees for inference
Allow multiple objective functions
Easy interface (python)

Question 2

XGBoost init

Answer 2

Data and a differentiable loss function

Question 3

XGBoost step 1

Answer 3

Init the model with a constant value : F0=argmin (gamma) SUM i=1..n (yi, gamma)

Question 4

XGBoost step 2 headline

Answer 4

m=1 to M

Question 5

XGBoost step 2 a

Answer 5

compute r_im = [diff(L(yi, F(xi))/diff(F(xi))] when F(xi)=F_m-1(xi) for i=1..n

Question 6

XGBoost 2 b

Answer 6

Fit a regression tree to the $r_{im}$ values and create terminal regions $R_{jm}$, for $j = 1, \dots, J_m$

Question 7

XGBoost 2 c

Answer 7

For $j = 1, \dots, J_m$, compute
[
\gamma_{jm} = \arg\min_\gamma \sum_{x_i \in R_{jm}} L(y_i, F_{m-1}(x_i) + \gamma)
]

Question 8

XGBoost 2 d

Answer 8

Update
[
F_m(x) = F_{m-1}(x) + \nu \sum_{j=1}^{J_m} \gamma_{jm} \mathbb{I}(x \in R_{jm})
]

Question 9

XGBoost 3

Answer 9

Output Fm(x)

Question 10

XGBoost start - explain in words what is the initial prediction

Answer 10

The value that minimises the loss function over the observations

Question 11

XGBoost - explain in words how to refine the prediction over the previous prediciton

Answer 11

Add a tree that minimises the errors of the previous predictions

Question 12

XGBoost - explain in words how to add the new tree prediction that minimises the errors of the previous prediction

Answer 12

Scale the tree with the learning rate