The squared error cost function (SECF) in logistic regression gives a non-convex graph, which leads to multiple local minima. Can I get proof of this non-convexity?
I got an article on the non-convexity of the SECF of logistic regression by googling, and the screenshot is below. Here, one training example is considered, and the red graph is for y = 1, and the blue graph is for y = 0.
But still, this proof couldn’t show the existence of multiple local minima.
Mathematically, a convex function has a positive 2nd partial derivative for all values.
The math proof for the logistic cost function involves a couple of pages of calculus. This course doesn’t cover it.
Yes, as Tom says, Prof Ng has set everything up here so that no calculus is required to take these courses, but that means we have to take his word for things in a lot of cases.
But if you have the math background, there is a lot of info available out there. E.g. here’s a paper from Yann LeCun’s group that shows some interesting results relative to local minima and the fact that they aren’t generally a serious problem.
And here is a Quora thread which is relevant, which is also linked from other forum posts here.
And here’s a thread locally that shows a graph of what the MSE cost surface looks like for Logistic Regression.
So just doing a little searching of the forums also turns up some answers.
