Learning rate decay vs RMSprop

[Christian_Simonis]

Hi @am003e

Upfront: Adam uses the calculation of an exponentially filtered moving average, combining RMSProp and Momentum. So it’s not like a classic LRD is used, but rather a dynamic (adaptive learning rate) approach
Therefore, this thread cshould be interesting to you:

Adam Optimization Question

Hi there, Adam uses the calculation of an exponentially filtered moving average, combining RMSProp and Momentum. At initialisation, usually aggregate gradient is set to 1, meaning they tend to be biased to zero since the averaging parameters are 1. This bias is the reason for the correction, see also this source. So the purpose of bias correction in exponential filtering is to improve the smoothening of early values, see also this plot (green line: with bias correction): https://napsterinblue.…

@am003e

In practice, how powerful is RMSprop compared with LRD? More generally have researchers sought to combine RMSprop and LRD into a combined, optimized alpha update rule?

I find your thoughts interesting… Do you have a certain cost functions in mind where you see this method particularly useful also compared to other gradient-based optimizers.

Please let me know what you think​

Best regards
Christian