Regularization: Intuition and Conservation of influence

[Christian_Simonis]

Hi there,

I think the best way intuitive-wise is to take a look at your loss function and the components it consists of. Ask yourself how large is lambda, so how „important“ is regularization in comparison with your performance goal:

Overfitting: L2 regularization  |  Machine Learning  | ...

The result afterwards is the output of your optimization problem on this very loss function. Independent of this: feature ranking might be a nice tool for you if you are interested in evaluating the importance of features: Permutation Importance vs Random Forest Feature Importance (MDI) — scikit-learn 1.6.1 documentation

This thread might be worth a look, too:

Do you see regularization the way I do?

Thanks for your post. The purpose of regularization is to reduce the model complexity by penalising it and so reduce overfitting. So it’s about reducing the model dependency of many parameters, e.g. by: driving weights exactly to zero (L1 regularization) or driving weights close to zero (L2 regularization) Also dropout is a useful technique to tackle overfitting. I like these explanation here, too. Feel free to take a look: -Regularization for Simplicity: L₂ Regularization  |  Machine Lea…

Best
Christian