Hi everyone,
While building a decision tree, we might need to stop splitting the tree if the information gain is no longer significant, or if we reach some pre-defined limit of the depth. This clearly save the time of processing on attributes that are not that important.
However, I think that it also helps prevent something like overfitting. The intuition is that, if we split all until the end (that is, there is no sample or all samples are in the same class), the tree might be too specific corresponding to the training dataset, which is quite the same to the term overfitting in linear regression, that is the tree does not generalize well on real world dataset. Therefore, it is not always good to split all the way down to the leaves.
Do I understand it correctly?
Thank you.
Yes, I think you understand it correctly.
this thread might be relevant for you since it also touches upon decision trees:
https://community.deeplearning.ai/t/can-decision-tree-algo-used-for-regression/268415/8
Best regards
Christian
@francesco4203:I believe in general you described the concept well, but I am not sure what you mean concerning linear regression:
In general, a linear regression model is rather a quite simple model, e.g. it can work even with one weight and one bias if you only use one feature. Therefore, it’s usually not associated with overfitting.
Of course you have a point if someone would add lots(!!) of features in a linear regression model that go way beyond tha capacity of the data. But just to understand your statement correctly: May I ask what you associate here with overfitting or what you are referring to?
Thanks!
Best regards
Christian
@francesco4203, you have very well explained how a decision tree overfits to the training data.
Cheers!
Raymond