Why do you add epsilon in the denominator of the final updating step of Adam optimization?

[jeffreywang]

I might have missed something here… but why is there a +epsilon in the denominator of the final step of Adam optimization?

As far as I can tell, this term was not included in the RMSprop lesson.

[paulinpaloalto]

Prof Ng explained that in the lectures. It is to avoid “divide by zero”.

[jeffreywang]

Looks like I missed it. My apologies and thank you for the prompt response! (I suppose it’s obvious now why epsilon isn’t a major hyperparameter, haha)

[Sahil_Singh1]

It is there in RMSProp too, though there is slight difference. In RMSProp it is within the sqaure root in the denminator, while in Adam is it outside it.

Course 2 week 2 question on the equation for Adam

Hi Nramon, With reference to the lecture notes, Adam puts epsilon outside of the square root, while RMSProp puts it inside the root. Is there a specific reason for the different location? Thanks!