V and S stand for in equations of optimisation algorithms?

Course Q&A Deep Learning Specialization Improving Deep Neural Networks: Hyperparameter tun
,
1

image
image1524×794 85.1 KB

Referring this image,
Just curious, is there any specific reason for the use of variable names of V and S in these equations (I think they are coming up in other optimisation algorithms like RMSprop and Gradient descent with momentum). Do they stand for something particular?

2

I think as far as I remember now, they stand for velocity and speed which is an anlogy of particle movements in physics!

2 Likes
3

Hi @computerandgyein

  • V stands for Velocity (an exponentially decaying moving average of past gradients)

  • S stands for Squared Gradient (accumulates the square of gradients to adjust the learning rate for each parameter individually)

Hope it helps!

5 Likes