Hi, I just finished the programming assignment of week 2: optimization method.
In section 7, there is this formula for exponential decay
lr = (1 / (1 + decay_rate * epoch_num)) * lr0
which is quite different from the lecture. There is another formula that prof. Andrew called exponential decay, but not this one if I am not mistaken. And it’s also different from the implementation shown on the TensorFlow website
here:
Can someone clarify for me, please? or It can be arranged this way?
Decay can be arranged in many ways, as long as it serves the purpose of reducing it with iterations. There are and should be many implementations around also depending on NN model behaviour and application.
Right. Also note that the “order of operations” in the code you show looks a little, how shall we say, “suspect” 
…
Do you mean the one on the TensorFlow site??
I meant the one you showed in your post. Suppose I write the following line of python:
x = 1. / 1. + 2.
What do you think the value of x will be? If you are expecting .333333, you are in for a surprise.
Here’s maybe an even more interesting example, but it’s not related to the code you show:
m = 5.
y = 1. / 2. * m
What do you think the value of y will be?
Okay, I got you. It was my bad
It should be
lr = lr0 / (1 + decay_rate * epoch_num)
something like this, right?
Yes, that’s correct.