None of the networks here are convex. You need to use the cost curves like the ones Prof Ng is explaining here to judge when you are no longer making progress or are perhaps diverging rather than converging. With the complex solution surfaces here, there is never any guarantee of smooth monotonic convergence: you may need to adjust the various hyperparameters like learning rate, number of iterations or even to adjust the architecture of your network or apply regularization in order to get things to work.
Also note that there is never any guarantee that a given solution is not a local minimum either, but it turns out that is not a big problem in general. Prof Ng makes this comment in a couple of places in the lectures, but doesn’t go into the details. It turns out that the math is pretty deep here, but here’s a thread that points to a well known paper from Yann LeCun’s group on this question of whether there are good (achievable) solutions for this kind of optimization problem.