Relation between Lambda(Regularization Parameter) and Weight?

What is the relation between Lambda(Regularization Parameter) and Weight? Is there any formula between them?

Hi, @Deependra.

Lambda is a hyperparameter that multiplies the regularization term, which is a function of the weights, to adjust its impact. More precisely, increasing lambda strengthens the effect of regularization.

You have all the formulas in this lecture :slight_smile:

1 Like

Hello @nramon,
I’ve just rewatched the lecture. I still didn’t see how exactly lambda is related to the weights. Professor Ng stated that increasing the value of lambda makes the weights smaller so I assume they are inversely related. Is there an explicit formula that shows how the two are related? Or is the regularization term fixed to a certain value such that when you increase the value of lambda, the weights get smaller?
Thank you so much in advance. Looking forward to hearing from you.

Regards,
Nay

Hi, @Nay.

This is the expression that relates them:

But remember that you’re trying to minimize J.That is, find the values w^{[1]}, b^{[1]}, ..., w^{[L]}, b^{[L]} that make it as small as possible. All else being equal, increasing \lambda makes the penalty term larger, so once \lambda is fixed, how can you make this term smaller? :slight_smile: