Gradient descent with Momentum - Week 2

[Sao-Mai]

#Week2
#Grandient descent with Momentum

Hi,
I have a question regarding gradient descent with momentum, please.

In the video, it says that in some research papers, instead of writing the equation on the left, they write the equation on the right: V_dw = B*v_dw + dW. We have this equation because we do *1/(1-B) on the equation on the left and we find the equation on the right.
I do not understand why we don’t have : 1/(1-B)*V_dw = 1/(1-B)*v_dw + dW, please ?

Secondly, our V_dw is being scale by 1/(1-B). So, when we performing gradient decent update, we need to change the alpha by 1/(1-B).
I do not understand because if we use this equation above: V_dw = Bv_dw + dW. and we change alpha by 1/(1-B) - for this last term: dW we already do 1/(1-B) (as in the first image) but do it 1*/(1-B) a second time with the alpha…

Can someone explain me the intuition, please ?
I hope I was clear in my explanation
Thank you,
Kind regards,
Sao Mai

[TMosh]

Is this thread related to your question?

Momentum Formula

Here is the formula for momentum gradient descent in the week assignment which passed the test: on itiration of l from 1 to L+1 v[“dW”+str(l)]=betav[“dW”+str(l)]+(1-beta)grads[“dW”+str(l)] v[“db”+str(l)]=betav[“db”+str(l)]+(1-beta)grads[“db”+str(l)] parameters[“W”+str(l)]=parameters[“W”+str(l)]-learning_ratev[“dW”+str(l)] parameters[“b”+str(l)]=parameters[“b”+str(l)]-learning_ratev[“db”+str(l)] And all the vdW1, vdW2… are all set to zero initially. what I find by itirating l is that v doe…

[Sao-Mai]

Hi,
Thank you very much for your reply !
I fact, not really.
My question was: The formula we learnt in the videos for the gradient decent with Momentum is: v_dw = Betav_dw + (1-Beta)dW.
However, in some literature we often see: v_dw = Betav_dw + dW. With the (1-Beta) ommited. I wanted to know, please how we end up with this formula : v_dw = Betav_dw + dW ?
Secondly, by using this formula v_dw = Betav_dw + dW, how do we need to update our parameter W, please ? (instead of W = W - av_dw, what should we use ?)

I would like to understand the mechanisms behind, please.
Thank you very much
Sao Mai

[TMosh]

@Sao-Mai

how we end up with this formula : v_dw = Betav_dw + dW

Sorry, I do not know.

[Sao-Mai]

No worries! Thank you so much for you reply

[saifkhanengr]

@Sao-Mai

I wanted to know, please how we end up with this formula : v_dw = Betav_dw + dW ?

We are not deriving the formula on the right-hand side (v_{dw} = \beta v_{dw} + dW) from the formula written on the left-hand side (v_{dw} = \beta v_{dw} + (1 - \beta) dW). These are two different formulas for momentum to control vertical movement and as Prof. Andrew said, “both of these will work just fine”, though he also mentioned some limitations of the right-hand side formula.

@Sao-Mai

Secondly, by using this formula v_dw = Betav_dw + dW, how do we need to update our parameter W, please ? (instead of W = W - av_dw, what should we use ?)

Its the same as for other methods: W = W - \alpha v_{dw} and b = b - \alpha v_{db}

[Sao-Mai]

I see we are not deriving the formula on the right-hand side from the formula written on the left-hand side.
Thank you so much for your explaination, everything is clear