Derivative of regularization term

Hello @pritamdodeja

Lets look at what happens mathematically to the regularization term \frac{\lambda}{2m}\sum_{j=1}^{n}w_j^2 when we are finding the derivative of cost function w.rt each weight variable, w_j → Lets take the case of w_1.

\frac {\partial {J(\vec{w},b)}}{\partial w_1} = \frac {\partial}{\partial w_1}(.....+ \frac{\lambda}{2m}\sum_{j=1}^{n}w_j^2)

= \frac {\partial}{\partial w_1}(.....+ \frac{\lambda}{2m}(w_1^2 + w_2^2 + ... + w_n^2)) → Expanding the summation

= .....+ \frac{\lambda}{2m}(2* w_1) → because \frac {\partial}{\partial w_1}(w_2^2 + ... + w_n^2) = 0 since they are all constants w.rt w_1

Thus, \frac {\partial {J(\vec{w},b)}}{\partial w_1}= .....+ \frac{\lambda}{m}w_1 → And this is how the summation for the regularization component vanished

And in the general case, \frac {\partial {J(\vec{w},b)}}{\partial w_j} = .....+ \frac{\lambda}{m}w_j

Hope its clear now.

6 Likes