# Optimization algorithms **URL:** \ **Category:** Improving Deep Neural Networks: Hyperparameter tun\ **Tags:** coursera-platform\ **Created:** [April 8, 2023, 2:03am UTC](https://community.deeplearning.ai/t/optimization-algorithms/305154 "2023-04-08T02:03:59Z")\ **Posts on this page:** 1\ **Showing post:** 3
**Author:** ![Christian\_Simonis](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/christian_simonis/32/142583_2.png) [@Christian\_Simonis](https://community.deeplearning.ai/u/Christian_Simonis)\ **Post date:** [April 8, 2023, 10:18am UTC](https://community.deeplearning.ai/t/optimization-algorithms/305154/3 "2023-04-08T10:18:40Z")
Hi @Ibrahim_Mustafa, in addition to @Elemento‘s great reply, check out this exemplary viz: [![](https://global.discourse-cdn.com/dlai/original/3X/9/6/965bf974f378c9eee2ddc326a26c3ca6bb469567.gif) ](https://global.discourse-cdn.com/dlai/original/3X/9/6/965bf974f378c9eee2ddc326a26c3ca6bb469567.gif) [Source](https://community.deeplearning.ai/t/how-different-initialization-of-centroids-of-k-means-results-in-drastic-different-clusters-they-all-share-common-cost-function/223900/4) RMSprop applies an adaption based on **squared gradients** , see also the [Keras Doc](https://keras.io/api/optimizers/rmsprop/): - Maintain a moving (discounted) average of the square of gradients - Divide the gradient by the root of this average Also momentum can by implemented using a moving average, but over the past **gradients** directly, see also this [source](https://towardsdatascience.com/gradient-descent-with-momentum-59420f626c8f). Momentum can be imagined like „memorising“ the inertia to not get stuck in local minima but make it hopefully to the global optimum in theory. (In practice often you do not need to get to the global optimum as long as the model performance is robust and **sufficient** …) If you want to see how RMSprop and momentum can be combined, check out this thread: [Adam Optimization Question - #2 by Christian\_Simonis](https://community.deeplearning.ai/t/adam-optimization-question/248565/2) In summary also a bit with respect to the usual goals: - momentum accelerates your search in the direction of the global minimum by „using the inertia“ to make it over local minima - RMSProp is sort of preventing to search in the direction of oscillations since it punishes outliers stronger due to squaring gradients - ADAM combines the heuristics of both Momentum and RMSProp as pointed out in this nice article: Source: [Intro to optimization in deep learning: Momentum, RMSProp and Adam](https://blog.paperspace.com/intro-to-optimization-momentum-rmsprop-adam/amp/), see also this thread: [Why not always use Adam optimizer - #2 by Christian\_Simonis](https://community.deeplearning.ai/t/why-not-always-use-adam-optimizer/245823/2) Hope that helps! Best regards Christian --- _[View the full topic](https://community.deeplearning.ai/t/optimization-algorithms/305154)._