The cost surfaces are in very high dimensions and are incredibly complex. You can “go off a cliff” at any point in the training. It might be worth trying a different optimization algorithm that uses more sophisticated controls on learning rate and the like. Here’s a thread which talks a bit more about non-convexity and also links to a paper by Yann LeCun’s group that is worth a look for insight about whether reasonable solutions exist and can be found or not.
The other point here is that there is some randomness in TF/Keras training, meaning that you don’t get the same results every time, even if you try to set random seeds and control the randomness. I’ve been meaning to research the TF docs more to see if this is controllable, but haven’t gotten around to it yet.