# Week 1, Understanding Dropout

**URL:** <https://community.deeplearning.ai/t/week-1-understanding-dropout/164174>\
**Category:** Improving Deep Neural Networks: Hyperparameter tun\
**Tags:** coursera-platform\
**Created:** [July 18, 2022, 9:13am UTC](https://community.deeplearning.ai/t/week-1-understanding-dropout/164174 "2022-07-18T09:13:13Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Lina\_Hourieh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/lina_hourieh/32/27879_2.png) [@Lina\_Hourieh](https://community.deeplearning.ai/u/Lina_Hourieh)\
**Post date:** [July 18, 2022, 9:13am UTC](https://community.deeplearning.ai/t/week-1-understanding-dropout/164174/1 "2022-07-18T09:13:13Z")

</div>

In the video, Andrew mentioned: ‘’ dropout can formally be shown to be an adaptive form of L2 regularization, but the **L2 penalty on different ways are different depending on the size of the activation is being multiplied into that way**.‘’  
Can you please illustrate that??

---

<div class="post-metadata">

**Author:** ![Elemento](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/elemento/32/854_2.png) [@Elemento](https://community.deeplearning.ai/u/Elemento)\
**Post date:** [July 18, 2022, 9:28am UTC](https://community.deeplearning.ai/t/week-1-understanding-dropout/164174/2 "2022-07-18T09:28:03Z")

</div>

Hey @Lina_Hourieh,  
Welcome to the community. I am a little confused as to what you mean by

> [@Lina\_Hourieh](#):
>
> L2 penalty on different ways are different depending on the size of the activation is being multiplied into that way

Just to be on the same page, L2 penalty simply adds the sum of the squares of the weights, scaled by the regularization parameter \lambda to the cost function and divided by the batch size 2 \* m (_where 2 is just a mathematical trick to make the differentiation easier_), which eventually results in minimising the values of the weight parameters, and in-turn, reduces over-fitting.

Now, when Prof Andrew mentions the statement

> [@Lina\_Hourieh](#):
>
> dropout can formally be shown to be an adaptive form of L2 regularization

He is hinting towards the fact that “in the presence of dropout”, the weights are distributed, and most of the neurons are assigned some importance (_although it may be little, but still some_), as opposed to the case of “absence of dropout”, in which, higher significance could be given to some neurons, i.e., the network might rely on some neurons more heavily by assigning larger weights to them. So, essentially, dropout is trying to break down larger weight values into smaller values.

For instance, consider the “absence of dropout”, let’s say we have a large weight assigned to a single neuron `9`. Now, when we implement the dropout, it re-distributes this weight into say `2, 3, 4`. Now, although 9 = 2 + 3 + 4, but 9^2 = 81 \> 2^2 + 3^2 + 4^2 = 29. So, you see how dropout redistributed the weights to decrease the sum of the squares of the weights, which is the exact thing that L2 penalty does. Let me know if this helps.

Regards,  
Elemento