# Doubt on Regularization **URL:** \ **Category:** Supervised ML: Regression and Classification\ **Tags:** week-module-3\ **Created:** [June 3, 2024, 11:42am UTC](https://community.deeplearning.ai/t/doubt-on-regularization/640216 "2024-06-03T11:42:44Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![om\_pandey07](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/om_pandey07/32/88779_2.png) [@om\_pandey07](https://community.deeplearning.ai/u/om_pandey07)\ **Post date:** [June 3, 2024, 11:42am UTC](https://community.deeplearning.ai/t/doubt-on-regularization/640216/1 "2024-06-03T11:42:44Z")
Struggling to get intuition of the regularization term. Like how adding the L2 regularization term decreases the weight of useless features and why it doesn’t make significant impact (or doesn’t decreases) other features’ weight? Thanks! ---
**Author:** ![Nevermnd](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/nevermnd/32/126021_2.png) [@Nevermnd](https://community.deeplearning.ai/u/Nevermnd)\ **Post date:** [June 3, 2024, 12:12pm UTC](https://community.deeplearning.ai/t/doubt-on-regularization/640216/2 "2024-06-03T12:12:55Z")
@om_pandey07 I wouldn’t say so much that the features regularization mostly effects (applies the highest penalty to) are ‘useless’. Rather the goal is to avoid overfitting the model and increase generalization. You are trying to reduce noise and the effect of extreme outliers in the model. I mean in the simplest case, imagine you have a data set of 10 points. Eight of these points are highly linearly correlated (so imagine you can drawing a line through it)-- but two of these points are way off. Obviously you don’t want them to have _too_ much influence on your model, so those weights are decreased. However, this happens not to just those features-- but _all_ the features in your model. If you did not do so you would be essentially changing your dataset (and now answering a different problem). The key thing is as these other points are more inline with the rest of the data, their weight decrease is much less. ---
**Author:** ![om\_pandey07](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/om_pandey07/32/88779_2.png) [@om\_pandey07](https://community.deeplearning.ai/u/om_pandey07)\ **Post date:** [June 3, 2024, 12:29pm UTC](https://community.deeplearning.ai/t/doubt-on-regularization/640216/3 "2024-06-03T12:29:48Z")
I see…so how does adding that regularization term helps us decreasing the weight and how does it “knows” what features weight it has to reduce more. I mean, like , if a feature is relevant, it’s weight would be generally be high, thus adding that term will increase the loss, so won’t the model will decrease the weight of it also? ---
**Author:** ![Nevermnd](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/nevermnd/32/126021_2.png) [@Nevermnd](https://community.deeplearning.ai/u/Nevermnd)\ **Post date:** [June 3, 2024, 12:37pm UTC](https://community.deeplearning.ai/t/doubt-on-regularization/640216/4 "2024-06-03T12:37:56Z")
Well the ultimate effect can also be a bit different whether we are talking about L1 or L2 regularization. Are you asking about in the case of L2 specifically? ---
**Author:** ![om\_pandey07](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/om_pandey07/32/88779_2.png) [@om\_pandey07](https://community.deeplearning.ai/u/om_pandey07)\ **Post date:** [June 3, 2024, 12:46pm UTC](https://community.deeplearning.ai/t/doubt-on-regularization/640216/5 "2024-06-03T12:46:20Z")
Yes for L2 ---
**Author:** ![Nevermnd](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/nevermnd/32/126021_2.png) [@Nevermnd](https://community.deeplearning.ai/u/Nevermnd)\ **Post date:** [June 3, 2024, 1:04pm UTC](https://community.deeplearning.ai/t/doubt-on-regularization/640216/6 "2024-06-03T13:04:26Z")
Well a big part of this is you have to recall the regularization is tagged on to our cost function, and we are running through this with forward and back prop many times. It ‘knows’ which terms are relevant because increasing their weight will decrease our loss. However, since the regularization is a (literally) ‘+’ added on parameter, we end up with a trade off between whether the increase on a given weight term provides provides a net gain-- or net loss in cost versus the size of that particular terms addition to the regularization parameter (since L2 is a squared sum). Regularization is going to _increase_ cost, loss, one way or another, but it also forces the model to adjust the weights in such away that the overall benefit of increasing the weight on some nodes, while decreasing others, provides an end net gain. If cost _increases_ for that node, the weights are forced down. The opposite occurs of vice versa for _decreases_. ---
**Author:** ![om\_pandey07](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/om_pandey07/32/88779_2.png) [@om\_pandey07](https://community.deeplearning.ai/u/om_pandey07)\ **Post date:** [June 3, 2024, 2:30pm UTC](https://community.deeplearning.ai/t/doubt-on-regularization/640216/7 "2024-06-03T14:30:50Z")
That clearly helps! Thank you so much ---
**Author:** ![paulinpaloalto](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/paulinpaloalto/32/187_2.png) [@paulinpaloalto](https://community.deeplearning.ai/u/paulinpaloalto)\ **Post date:** [June 3, 2024, 2:39pm UTC](https://community.deeplearning.ai/t/doubt-on-regularization/640216/8 "2024-06-03T14:39:26Z")
> [@om\_pandey07](#): > > I mean, like , if a feature is relevant, it’s weight would be generally be high, thus adding that term will increase the loss, so won’t the model will decrease the weight of it also? Yes, the effect of the L2 regularization term will be more pronounced on weights with larger magnitudes. If a given weight is already lower in absolute value, then it will not be affected as much. As Anthony mentioned, the intent is to prevent particular individual features from having too much influence on the results on the theory that this situation represents overfitting. Of course there is no guarantee that this (meaning L2 regularization) will work in every case and it’s also very much a matter of correctly tuning the \lambda value. It’s a battle between the base cost function and the regularization term. If you make \lambda a relatively large value, then the L2 term dominates and in the limit you drive all the weights to zero and the model becomes useless. It’s been a while since I watched these lectures, but I’m pretty sure I remember Prof Ng making that point in the lectures. To succeed here you need to experiment to determine just the “Goldilocks” amount of the L2 suppression of the weights to give a general solution. ---
**Author:** ![om\_pandey07](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/om_pandey07/32/88779_2.png) [@om\_pandey07](https://community.deeplearning.ai/u/om_pandey07)\ **Post date:** [June 3, 2024, 4:50pm UTC](https://community.deeplearning.ai/t/doubt-on-regularization/640216/9 "2024-06-03T16:50:54Z")
Thanks for the explaination :))