Hi there
I have some issues with setting the learning rate for my model to predict time series data. I have added some plots as reference.
Problem (A): In the plot A as attached here(learning rate=1e-6), loss is higher and I have to reduce the prediction loss. If I tune the rate, loss result also changes again and get the desired result. But I am not sure how to find the best learning rate from the plot A of the training result.
Problem (B): I got the prediction loss with learning rate 5e-6(as shown in plot B), it is a randomly choose number and it worked. Nevertheless, it is not so clear how I can get 5e-6 from the curve B. Please help me to clear my doubts.
lrs = 1e-8 * (10 ** (np.arange(100) / 20))
plt.figure(figsize=(10, 6))
plt.grid(True)
plt.semilogx(lrs, history.history[...])
plt.tick_params(....')
plt.axis([1e-8, 1e-3, 0, 300])
Question A: We can see that the curve has always been very good. At this time, it can be set to any number (between 1e-8 and 1e-3), but we know that the larger the learning rate, the faster the convergence speed, but the less difficult it is Convergence to the local optimal solution, the smaller the learning rate, the slower the convergence speed, but it is easy to converge to the local optimal solution, so we choose the learning rate as a compromise.
Question B: We can see that the learning rate is very effective before 1e-5, after 1e-5, the loss fluctuates significantly, so we will choose before 1e-5.
To sum up, the principle of choosing the learning rate is to make the convergence speed as fast as possible while it is easy to converge to the local optimal solution.
Hey @SAHJAD_HOSSAIN ,
Sorry for the late reply.I think my fellow mentor @SteveSun has explained quiet well the answers of both of your questions. In case,you have some more queries please feel free to reach us out again.
Happy Coding:))
Thanks and Regards,
Mayank Ghogale