Why we need to add regularization lambda function into the cost function since we already did the regularization in the gradient descent

Course Q&A Machine Learning Specialization Supervised ML: Regression and Classification
1

week-3
Like in the gradient descent we are already doing this, isn’t the w we get is the smaller one? After each literation?

image
image855×384 79.9 KB

Why we are still need to add this into the cost function? Someone explain it in a simple way in math plz…
image

1 Like
2

The regularized part of the cost function is to encourage the learned weights to be slightly smaller. This helps prevent overfitting.

The part circled in red in your first image is the portion of the gradients that account for the regularization term in the cost function.

3

But like the idea of gradient descent is to find the best fit “w” and “b”, so after we apply regularization in the gradient descent, in the end isn’t we already get the regularized “w”? Then why don’t we just use the regularized “w” in the cost function without lambda?
image

1 Like
4

The formula you posted is not regularized. It’s extremely likely to cause overfitting of the training set.

5

I think, the gradient descent “regularization” is derived from the cost function regularization. So to apply regularization to your algorithm, (to avoid overfitting) you have to add the regularization term to both the cost function and the gradient descent function. (the gradient descent function has components of the derivative of the cost function anyways. so they are related!). I hope this helps!

1 Like
6

It all starts with the cost function.
That should include a regularization term.
Then since the gradients are the partial derivatives of the cost function, this gives you the expression for the regularized gradients.

1 Like