My take on this is that unfortunately there is not the only „silver bullet“ to solve all optimization problems. My experience is that it depends on the data and also the problem that you are solving, which then results in a different cost function where different optimizers have different strength and weaknesses.
Here you can find a nice overview of some popular optimizers: An updated overview of recent gradient descent algorithms – John Chen – ML at Rice University
I like to think of optimizers as tools with different complexity levels and features that help you to solve your business problem sufficiently, see also this thread.
Best regards
Christian