# Why do you add epsilon in the denominator of the final updating step of Adam optimization?

**URL:** <https://community.deeplearning.ai/t/why-do-you-add-epsilon-in-the-denominator-of-the-final-updating-step-of-adam-optimization/21708>\
**Category:** Improving Deep Neural Networks: Hyperparameter tun\
**Tags:** coursera-platform\
**Created:** [July 18, 2021, 1:45am UTC](https://community.deeplearning.ai/t/why-do-you-add-epsilon-in-the-denominator-of-the-final-updating-step-of-adam-optimization/21708 "2021-07-18T01:45:06Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jeffreywang](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/jeffreywang/32/3849_2.png) [@jeffreywang](https://community.deeplearning.ai/u/jeffreywang)\
**Post date:** [July 18, 2021, 1:45am UTC](https://community.deeplearning.ai/t/why-do-you-add-epsilon-in-the-denominator-of-the-final-updating-step-of-adam-optimization/21708/1 "2021-07-18T01:45:06Z")

</div>

I might have missed something here… but why is there a +epsilon in the denominator of the final step of Adam optimization?

As far as I can tell, this term was not included in the RMSprop lesson.

![Screen Shot 2021-07-17 at 6.44.55 PM](https://global.discourse-cdn.com/dlai/original/2X/5/550ccc9b746a632f4b0ab1752ff968dcad37f825.png)

---

<div class="post-metadata">

**Author:** ![paulinpaloalto](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/paulinpaloalto/32/187_2.png) [@paulinpaloalto](https://community.deeplearning.ai/u/paulinpaloalto)\
**Post date:** [July 18, 2021, 5:17am UTC](https://community.deeplearning.ai/t/why-do-you-add-epsilon-in-the-denominator-of-the-final-updating-step-of-adam-optimization/21708/2 "2021-07-18T05:17:12Z")

</div>

Prof Ng explained that in the lectures. It is to avoid “divide by zero”.

---

<div class="post-metadata">

**Author:** ![jeffreywang](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/jeffreywang/32/3849_2.png) [@jeffreywang](https://community.deeplearning.ai/u/jeffreywang)\
**Post date:** [July 18, 2021, 5:18am UTC](https://community.deeplearning.ai/t/why-do-you-add-epsilon-in-the-denominator-of-the-final-updating-step-of-adam-optimization/21708/3 "2021-07-18T05:18:20Z")

</div>

Looks like I missed it. My apologies and thank you for the prompt response! (I suppose it’s obvious now why epsilon isn’t a _major_ hyperparameter, haha)

---

<div class="post-metadata">

**Author:** ![Sahil\_Singh1](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/sahil_singh1/32/43024_2.png) [@Sahil\_Singh1](https://community.deeplearning.ai/u/Sahil_Singh1)\
**Post date:** [April 23, 2023, 6:16am UTC](https://community.deeplearning.ai/t/why-do-you-add-epsilon-in-the-denominator-of-the-final-updating-step-of-adam-optimization/21708/4 "2023-04-23T06:16:21Z")

</div>

It is there in RMSProp too, though there is slight difference. In RMSProp it is within the sqaure root in the denminator, while in Adam is it outside it.

> [@Course 2 week 2 question on the equation for Adam](https://community.deeplearning.ai/t/course-2-week-2-question-on-the-equation-for-adam/16183/4):
>
> Hi Nramon, With reference to the lecture notes, Adam puts epsilon outside of the square root, while RMSProp puts it inside the root. Is there a specific reason for the different location? Thanks!

 ![image](https://global.discourse-cdn.com/dlai/original/3X/2/9/291786e4495c71cd2e25ef4dfb8c7164bc48df64.png)