How to chose the right value for the regularization parameter?

Sure! For example, let’s say our model assumption is y=wx+b , and before scaling, the cost function with L2 regularization is

J = \frac{1}{2m}\sum_{i=1}^{m}{(wx^{(i)}+b-y^{(i)})^2}+\frac{\lambda }{m}w^2

Now we want to rescale x by {x^{(i)}}' = \frac{x^{(i)} - x_{\text{mean}}}{x_{\text{std}}}, and by the following derivation we have the relationship between w and w', and b and b', and we just put them all into J to get

J = \frac{1}{2m}\sum_{i=1}^{m}{(w'{x^{(i)}}'+b'-y^{(i)})^2}+\frac{1}{m}\frac{\lambda}{x_{\text{std}}^2}{w'}^2

Now this J(w',b') is what we are optimizing for given the rescaled dataset x'


1 Like