# L2 Regularization With Mini-batch GD

**URL:** <https://community.deeplearning.ai/t/l2-regularization-with-mini-batch-gd/18395>\
**Category:** Improving Deep Neural Networks: Hyperparameter tun\
**Tags:** coursera-platform\
**Created:** [July 1, 2021, 9:37am UTC](https://community.deeplearning.ai/t/l2-regularization-with-mini-batch-gd/18395 "2021-07-01T09:37:30Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![MalayAgr](https://avatars.discourse-cdn.com/v4/letter/m/51bf81/32.png) [@MalayAgr](https://community.deeplearning.ai/u/MalayAgr)\
**Post date:** [July 1, 2021, 9:37am UTC](https://community.deeplearning.ai/t/l2-regularization-with-mini-batch-gd/18395/1 "2021-07-01T09:37:30Z")

</div>

In L2 regularization, the new cost is computed as:

`cost = J without regularization + (lambda / 2m) * (Frobenius norm of weights)`

Here, the ‘m’ refers to the size of the entire training set. But, in mini-batch GD, we train on smaller batches of data at a time. My question is that, similar to how the ‘m’ in “J without regularization” changes to the size of the batch, does the ‘m’ in the regularization term also change to the size of the batch?

My intuition tells me that it does change to the batch size but I’d like some expert eyes on this.

Additionally, consider SGD with momentum The weight update is as follows:

`W = W - lr * velocity`, where `velocity = beta * velocity + (1 - beta) * grad(W)`.

With L2 regularization, `grad(W)` can be written as:

`grad(W) = grad(W) without regularization + lambda * W`

My question is that, while updating velocity, should we only use “grad(W) without regularization” or should we use the above, new definition of `grad(W)`?

That is, should the weights be updated as:

`W = W - lr * velocity - lr * W`, where `velocity` is computed using the old, no-regularization definition of `grad(W)`

or just:

`W = W - lr * velocity,` where `velocity` is now computed on the new definition of `grad(W)`.

Thanks!

---

<div class="post-metadata">

**Author:** ![nramon](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/nramon/32/129097_2.png) [@nramon](https://community.deeplearning.ai/u/nramon)\
**Post date:** [July 1, 2021, 4:18pm UTC](https://community.deeplearning.ai/t/l2-regularization-with-mini-batch-gd/18395/2 "2021-07-01T16:18:40Z")

</div>

Hi, @MalayAgr.

It’s the same `m`, the size of the mini-batch. [Here](https://stats.stackexchange.com/questions/287920/regularisation-why-multiply-by-1-2m)’s an interesting discussion about this scaling factor.

When you add momentum, W is still calculated as ![image](https://global.discourse-cdn.com/dlai/original/2X/3/31b36eef3446136cfb8aca427b13b60dec07a091.png). But V\_{dw} depends on dw, which now has an [additional term](https://community.deeplearning.ai/t/week-1-back-propagation-with-l2-regularization/16505/2): ![image](https://global.discourse-cdn.com/dlai/original/2X/8/85c9f71b88c309af742b99cf099d9418645cfd26.png)

Hope that helped 🙂