# Adam Optimization Question

**URL:** <https://community.deeplearning.ai/t/adam-optimization-question/248565>\
**Category:** Improving Deep Neural Networks: Hyperparameter tun\
**Tags:** coursera-platform\
**Created:** [December 28, 2022, 7:30pm UTC](https://community.deeplearning.ai/t/adam-optimization-question/248565 "2022-12-28T19:30:43Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![JuanVT](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/juanvt/32/35646_2.png) [@JuanVT](https://community.deeplearning.ai/u/JuanVT)\
**Post date:** [December 28, 2022, 7:30pm UTC](https://community.deeplearning.ai/t/adam-optimization-question/248565/1 "2022-12-28T19:30:43Z")

</div>

This is a general question about the Adam Optimization Algorithm. Why do we implement bias correction and we don’t in Grad Descent with momentum and RMSprop? Is it because of the major complexity? Or just to get rid of any inconsistencies?

---

<div class="post-metadata">

**Author:** ![Christian\_Simonis](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/christian_simonis/32/142583_2.png) [@Christian\_Simonis](https://community.deeplearning.ai/u/Christian_Simonis)\
**Post date:** [December 28, 2022, 8:53pm UTC](https://community.deeplearning.ai/t/adam-optimization-question/248565/2 "2022-12-28T20:53:02Z")

</div>

Hi there,

Adam uses the calculation of an exponentially filtered moving average, combining RMSProp and Momentum. At initialisation, usually aggregate gradient is set to 1, meaning they tend to be biased to zero since the averaging parameters are 1. This bias is the reason for the correction, see also this [source](https://optimization.cbe.cornell.edu/index.php?title=Adam).

So the purpose of bias correction in exponential filtering is to improve the smoothening of early values, see also this plot (green line: with bias correction): [https://napsterinblue.github.io/notes/stats/techniques/ewma/ewma\_39\_0.png](https://napsterinblue.github.io/notes/stats/techniques/ewma/ewma_39_0.png)

[Source](https://napsterinblue.github.io/notes/stats/techniques/ewma/)

In classic momentum or RMSProp we do not have this bias correction, but I agree with you: depending on the initial assumptions of the exponentially weighted moving average, it would be possible in general and it might make sense, see also:

```auto
    
    # Momentum
    v_dW = beta1 * v_dW + (1 - beta1) dW
    v_db = beta1 * v_db + (1 - beta1) db
    
    v_dW_corrected = v_dw / (1 - beta1 ** t)
    v_db_corrected = v_db / (1 - beta1 ** t)
    
    # RMSprop    
    s_dW = beta * v_dW + (1 - beta2) (dW ** 2)
    s_db = beta * v_db + (1 - beta2) (db ** 2)
    
    s_dW_corrected = s_dw / (1 - beta2 ** t)
    s_db_corrected = s_db / (1 - beta2 ** t)
   
    # Combine
    W = W - alpha * (v_dW_corrected / (sqrt(s_dW_corrected) + epsilon))
    b = b - alpha * (v_db_corrected / (sqrt(s_db_corrected) + epsilon))

```

Source: [Momentum, RMSprop, and Adam Optimization for Gradient Descent](https://napsterinblue.github.io/notes/machine_learning/neural_nets/adam_opt/)

---

<div class="post-metadata">

**Author:** ![Christian\_Simonis](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/christian_simonis/32/142583_2.png) [@Christian\_Simonis](https://community.deeplearning.ai/u/Christian_Simonis)\
**Post date:** [December 28, 2022, 8:56pm UTC](https://community.deeplearning.ai/t/adam-optimization-question/248565/3 "2022-12-28T20:56:07Z")

</div>

The Adam bias correction can be interpreted as a modification to the learning rate, see also this [paper](https://arxiv.org/pdf/2110.10828.pdf) where an alternative is proposed and discussed:

> The originally stated goal of the bias-correction factor was at least partially to reduce the initial learning rate in early steps, before the moving averages had been well initialized (Kingma and Ba, 2017; Mann, 2019).

This thread is relevant for your question. Feel free to take a look: [Why not always use Adam optimizer - #4 by Christian\_Simonis](https://community.deeplearning.ai/t/why-not-always-use-adam-optimizer/245823/4)

Best  
Christian