C3W3 Lab RL why setting an adam optimizer while only updating gradients?

Course Q&A Machine Learning Specialization Unsupervised Learning, Recommenders, Reinforcement
1

Hello,

I managed to get “C3_W3_A1_Assignment.ipynb” to work locally.

I read the lesson again and the code. And I wonder why setting a specific adam optimizer while only updating gradients afterward?
Any optimizer would not work as well as far as we don’t run any optimization with it in the whole code?
Or maybe I missed something hidden in the other files.
Or something I missunderstood?

Best regards,

Francis

2

Hey @Francis60,

The Adam optimizer did its part in the agent_learn function when we called its apply_gradient method. If we didn’t have this line of code, no weight update would have happened to the Q network.

Cheers,
Raymond

3

I’m not sure whether I fully detailed my question. :thinking:

In the code I don’t see any of those:

model.compile(
model.fit(

to explicitly call the model adjustment.

I tried with a different optimizer (basic gradient descent):

optimizer = SGD(learning_rate=ALPHA)
# optimizer = Adam(learning_rate=ALPHA)

SGD : solved in 913 episodes! Total Runtime: 20.61 min
Adam: solved in 596 episodes! Total Runtime: 14.17 min
(I have a 16-gig cpu)

So what I understand is that this:

optimizer.apply_gradients(zip(gradients, q_network.trainable_variables))

is actually calling the optimizer and then the model performs all Adam refinements each time it runs and is not only strictly applying the computed gradients.

Is it?

1 Like
4

I asked ChatGPT 3.5, it answers this:

  • If model.compile() or model.fit() are not executed, the model is not trained, and therefore, the optimization process, including adaptation, does not occur. The optimizer’s parameters remain unchanged until the model is compiled and trained.

And also that:

  • Applying Gradients : After computing the gradients, you call the optimizer’s apply_gradients() method, passing it the gradients along with the corresponding model parameters. The optimizer then applies these gradients to update the model’s parameters according to its optimization algorithm.
  • when optimizer.apply_gradients() is called, the optimizer refines not only the learning rates but also any other parameters it maintains, such as momentums.

So ‘apply_gradients()’ does not only apply gradients straight away but using optimizer full capabilities and keeps a track of previous steps. :ok_hand:

Where is the ‘resolved’ button ?

1 Like
5

Hello @Francis60,

Yes. adam.apply_gradient does apply the gradients in the manner of Adam. We utilize our optimizer this way when we have a customized training process - getting new (s, a, r, s) tuples and soft-updating Target Q Network. These customized processes are not in the standard training procedure initiated by model.compile and model.fit.

There should be a “Solution” button but somehow this and some of the other buttons just don’t show up in some of the threads… It’s quite mysterious to me too :thinking:

Cheers,
Raymond

2 Likes
6

I recommend you not use a chat tool for programming advice.

7

Okay @TMosh , and what about coursera coach (Beta) ?

Cheers,
Francis

8

I’m not aware of that tool.

If you are enrolled in a programming course (like MLS), you should be very careful to do your own work. That’s covered in the Code of Conduct.

9

This is the tool Coursera has been providing for the past few months.

Of course, I do my own work.

Regards,

10

That’s good to know.

11

e.g.:

  • Explain this topic in simple terms
  • Give me practice questions
  • Give me a summary
  • Give me real-life examples