# Decreasing Regularization

**URL:** <https://community.deeplearning.ai/t/decreasing-regularization/57113>\
**Category:** Structuring Machine Learning Projects\
**Tags:** coursera-platform\
**Created:** [October 30, 2021, 12:42pm UTC](https://community.deeplearning.ai/t/decreasing-regularization/57113 "2021-10-30T12:42:50Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![heeseong\_kim](https://avatars.discourse-cdn.com/v4/letter/h/439d5e/32.png) [@heeseong\_kim](https://community.deeplearning.ai/u/heeseong_kim)\
**Post date:** [October 30, 2021, 12:42pm UTC](https://community.deeplearning.ai/t/decreasing-regularization/57113/1 "2021-10-30T12:42:50Z")

</div>

In C3W1 Bird Recognition problem number 10,

 ![image](https://global.discourse-cdn.com/dlai/original/2X/5/5cd150df8416ecae3f6e5b7d8a551bbcd3c12ea2.png)

The answers of the question are (1) Train a bigger model (2) Try decreasing regularization.  
I understand the answer (1). But I do not understand how decreasing regularization can help in this situation.

---

<div class="post-metadata">

**Author:** ![jonaslalin](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/jonaslalin/32/63_2.png) [@jonaslalin](https://community.deeplearning.ai/u/jonaslalin)\
**Post date:** [November 1, 2021, 7:25pm UTC](https://community.deeplearning.ai/t/decreasing-regularization/57113/2 "2021-11-01T19:25:56Z")

</div>

Because we are currently underfitting the training data, by using too much regularization, which has resulted in a model that is too simple:

 ![image](https://global.discourse-cdn.com/dlai/original/2X/5/5ee3323543ec6cb295387d4db07efe09d98a3843.jpeg)

A model that exhibits small variance and high bias will underfit the target, while a model with high variance and little bias will overfit the target.

---

<div class="post-metadata">

**Author:** ![heeseong\_kim](https://avatars.discourse-cdn.com/v4/letter/h/439d5e/32.png) [@heeseong\_kim](https://community.deeplearning.ai/u/heeseong_kim)\
**Post date:** [November 2, 2021, 12:27am UTC](https://community.deeplearning.ai/t/decreasing-regularization/57113/3 "2021-11-02T00:27:56Z")

</div>

Thank you for your answer.  
I have one more question.

In the course, I learned as follows:  
If we are “underfitting” the training data → larger network, train longer  
If we are “overfitting” the training data → regularization, more data

So, if we are “overfitting” the training data, can we use “smaller” network and train “shorter”? Will this be appropriate as applying more regularization and getting more data?

---

<div class="post-metadata">

**Author:** ![jonaslalin](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/jonaslalin/32/63_2.png) [@jonaslalin](https://community.deeplearning.ai/u/jonaslalin)\
**Post date:** [November 2, 2021, 10:12am UTC](https://community.deeplearning.ai/t/decreasing-regularization/57113/4 "2021-11-02T10:12:26Z")

</div>

Using a smaller network and regularizing the network is similar in the sense that regularization results in fewer weights doing the heavy lifting, which is the same as using a smaller network where all those weights are participating in the heavy lifting.

During hyperparameter tuning, the first thing to make sure is that you train for the right number of epochs, i.e., before you start overfitting because you train too long. Hence, you always use early stopping in some way or the other.  
Next, you have to check how you do compared to your validation set. If you overfit, you add more data if possible, otherwise you start regularizing, by penalizing weights, adding dropout, decreasing network size or any other method you can think of.

Goodfellow writes

> in practical deep learning scenarios, we almost always do find—that the best fitting model (in the sense of minimizing generalization error) is a large model that has been regularized appropriately…