# Optimization algorithms

**URL:** <https://community.deeplearning.ai/t/optimization-algorithms/305154>\
**Category:** Improving Deep Neural Networks: Hyperparameter tun\
**Tags:** coursera-platform\
**Created:** [April 8, 2023, 2:03am UTC](https://community.deeplearning.ai/t/optimization-algorithms/305154 "2023-04-08T02:03:59Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ibrahim\_Mustafa](https://avatars.discourse-cdn.com/v4/letter/i/7ea924/32.png) [@Ibrahim\_Mustafa](https://community.deeplearning.ai/u/Ibrahim_Mustafa)\
**Post date:** [April 8, 2023, 2:03am UTC](https://community.deeplearning.ai/t/optimization-algorithms/305154/1 "2023-04-08T02:03:59Z")

</div>

i reaaly did’t understand what is the diffrence between momentum and RMSprob  
it seams to me that both do the same thing

---

<div class="post-metadata">

**Author:** ![Elemento](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/elemento/32/854_2.png) [@Elemento](https://community.deeplearning.ai/u/Elemento)\
**Post date:** [April 8, 2023, 6:35am UTC](https://community.deeplearning.ai/t/optimization-algorithms/305154/2 "2023-04-08T06:35:01Z")

</div>

Hey @Ibrahim_Mustafa,  
All the optimization algorithms have the same goal, i.e., to converge to the minimum value as fast as possible. The algorithm which will work best in your case, will depend on your application. As to the theoretical differences, I am sure Prof Andrew does a great job in explaining those, still let me highlight the key distinction.

While “Gradient Descent with Momentum” computes the moving average of the **gradients** , “RMSProp” on the other hand computes the moving average of the **gradients squared** , which the algorithms use further to update the weights and biases. I hope this resolves your query.

Cheers,  
Elemento

---

<div class="post-metadata">

**Author:** ![Christian\_Simonis](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/christian_simonis/32/142583_2.png) [@Christian\_Simonis](https://community.deeplearning.ai/u/Christian_Simonis)\
**Post date:** [April 8, 2023, 10:18am UTC](https://community.deeplearning.ai/t/optimization-algorithms/305154/3 "2023-04-08T10:18:40Z")

</div>

Hi @Ibrahim_Mustafa,

in addition to @Elemento‘s great reply, check out this exemplary viz:

[![](https://global.discourse-cdn.com/dlai/original/3X/9/6/965bf974f378c9eee2ddc326a26c3ca6bb469567.gif) ](https://global.discourse-cdn.com/dlai/original/3X/9/6/965bf974f378c9eee2ddc326a26c3ca6bb469567.gif)  
[Source](https://community.deeplearning.ai/t/how-different-initialization-of-centroids-of-k-means-results-in-drastic-different-clusters-they-all-share-common-cost-function/223900/4)

RMSprop applies an adaption based on **squared gradients** , see also the [Keras Doc](https://keras.io/api/optimizers/rmsprop/):

- Maintain a moving (discounted) average of the square of gradients
- Divide the gradient by the root of this average

Also momentum can by implemented using a moving average, but over the past **gradients** directly, see also this [source](https://towardsdatascience.com/gradient-descent-with-momentum-59420f626c8f).  
Momentum can be imagined like „memorising“ the inertia to not get stuck in local minima but make it hopefully to the global optimum in theory. (In practice often you do not need to get to the global optimum as long as the model performance is robust and **sufficient** …)

If you want to see how RMSprop and momentum can be combined, check out this thread: [Adam Optimization Question - #2 by Christian\_Simonis](https://community.deeplearning.ai/t/adam-optimization-question/248565/2)

In summary also a bit with respect to the usual goals:

- momentum accelerates your search in the direction of the global minimum by „using the inertia“ to make it over local minima
- RMSProp is sort of preventing to search in the direction of oscillations since it punishes outliers stronger due to squaring gradients
- ADAM combines the heuristics of both Momentum and RMSProp as pointed out in this nice article:  
Source: [Intro to optimization in deep learning: Momentum, RMSProp and Adam](https://blog.paperspace.com/intro-to-optimization-momentum-rmsprop-adam/amp/), see also this thread: [Why not always use Adam optimizer - #2 by Christian\_Simonis](https://community.deeplearning.ai/t/why-not-always-use-adam-optimizer/245823/2)

Hope that helps!

Best regards  
Christian