In the lecture about XGboost, it was explained that XGBoost has a builtin regularization technique to improve performance. How will regularization be applied to boosted trees where we do not have any learnable weights like neural nets? What will be regularized to improve performance?
1 Like
Hello @Mohamed_Javeed,
XGBoost is not neural network so it does not have learnable weights like neural network. To discuss regularization in XGBoost, we can forget about neural network, and start with one thing that a fully grown tree will always match the training data 100% well. This is obviously overfitting because training data has noise and fitting a model well to noise is not desirable.
Therefore, to “regularize” a decision-tree based model, we can limit it from being fully grown by, for example, setting the max_depth parameter which limits the size of a tree. On the other hand, we might also hide a certain part of the data from the training algorithm so that a tree can’t be fully grown to match the whole set of data, such as by setting the “subsample” parameter to only show a certain percent of the samples to the algorithm.
This official note talks about preventing overfitting in XGBoost, and there are indeed more parameters (than what are mentioned here or in the note) that can contribute to regularizing a decision-tree based model. You will need to read the list of parameters yourself and study it.
Raymond
2 Likes
Hello sir. Thank you for your great explanation 
1 Like
You’re welcome @Mohamed_Javeed!