Cost function shape in neural network

[Juan_Olano]

@KiraDiShira ,

There is a previous post that discusses this very same topic in more depth. I recommend that you read it as it will shed lots of light to your question:

Cost function stuck at local minima

It is an important question, but the answer has lots of layers to it. For the simple case of Logistic Regression, the cost function is actually convex, so it has a single global minimum and no local minima. Once we graduate to real Neural Networks, though, that is no longer true. The cost surfaces are not convex and there can be lots of local optima. One high level point to make is that convergence (even to a local minimum) is never guaranteed: if you pick a learning rate that is too high, you…

Juan