Can we regularize only specific parameters

Course Q&A Machine Learning Specialization Supervised ML: Regression and Classification
1

HI
Regullarization is used to reduce overfitting. But if we apply it to all parameters then wont it reduce the accuracy of the model. Can it be used to specify only few parameters?

2

Can you explain why it would reduce the model’s accuracy on training set?

Can you explain why it would reduce the model’s accuracy on cv set?

It’s helpful to know how you came to this statement.

I have never seen people did this, and I am not sure what benefit it can bring us. However, theoretically speaking, you can only regularize selected weights, but given that you implement such functionality yourself.

Raymond

3

It will reduce it on the training set because the weights will not be able to take their overfitted value which they would have without regularization.

Can you explain what cv set is ?

4

Exactly. Since accuracy drop on training set is a natural consequence of regularization, we don’t call it a bad thing. However, when is it a bad thing? When it reduces accuracy on a CV set. CV set is just another dataset that is never used for training. We preserve it for evaluation purpose. If a trained model can do well on the CV set, we call it a good model. If a trained model can ONLY do well on the training set but not on the CV set, it could be diagnosed with overfitting.

Course 2 spends Week 3 on overfitting, underfitting, and how to diagnose them with a CV set. CV set will also be introduced in that week. So let’s wait until that week and see how the lectures are going to explain those ideas to you.

Cheers,
Raymond

5

Hi everyone

let me add:
If we take this exemplary architecture:

1534×957 96 KB

Source

So I have seen it several times in this kind of architectures, that e.g.:

  • dropout as regularization technique is applied to the dense resp. fully connected layer to reduce overfitting there,
  • but conv layers are not regulized further since the derived features were sufficiently robust, see also these sources 1 and 2.

So also selective regularization for a group of parsmeters is also a discussed approach in literature.

Hope that helps, @kanishk_rastogi!

Best regards
Christian