Hi @tbhaxor,
optimizers have a long history and plenty of applications and historical numerical use cases, such as finite element simulations.
Note that there are also gradient-free optimizers that are often used for hyperparameter optimization, see also this thread: Question about optimizers - #2 by Christian_Simonis
To answer your question: Even in gradient-based optimizers there are differences and sometimes also fine nuances where you have benefits for different optimization problems. There are certainly popular optimizers in ML, but they have different strengths and weaknesses and they might be even designed to a dedicated hardware, serving different complexity levels, see also: Why not always use Adam optimizer - #4 by Christian_Simonis
Optimizers are also an active field of research and I guess in next years there will be further improvements.
Hope this answers your question, @tbhaxor.
Best regards
Christian