Why is there not a mathematical formula for calculating the best learning rate for gradient descent? Is there a way to get close to the optimal learning rate by using statistics and/or calculus? I’m looking for a better way than guess and check.
Perhaps look into the Adam algorithm.
Fixed rate GD is not used very much because it is computationally inefficient. But it is a good first introduction.