Why is there not a mathematical formula for calculating the best learning rate for gradient descent? Is there a way to get close to the optimal learning rate by using statistics and/or calculus? I’m looking for a better way than guess and check.
Perhaps look into the Adam algorithm.
Fixed rate GD is not used very much because it is computationally inefficient. But it is a good first introduction.
The main reason is that gradient descent is an iterative optimization method. It only has local information (the gradient at the current point), not a global view of the entire loss landscape. Because of that, there is no universal equation that can tell us the best learning rate for every problem.
You can think of it as a blind person trying to find the deepest valley. Each step is based on what they can sense locally, then they re-evaluate and take the next step. That’s why these methods are fundamentally iterative and involve some trial and adjustment.
Adaptive optimizers like Adam, RMSprop, and others help by automatically adjusting the step sizes during training, reducing—but not eliminating—the need for learning-rate tuning.