In this example, Logistic Regression separated space into blue and red regions. The accuracy of the model is 47%. However, if we move the line so that all dots are in the blue region and leave 0 dots at the red one, the accuracy will be 50%. Why the model is less accurate than it can be?
Really interesting and perceptive question! You have probably heard it stated that the cost function for Logistic Regression is actually convex, so there is a unique global minimum for the cost. So as long as we don’t make incorrect choices for the learning rate, we should be able to converge to something pretty close to the unique solution that minimizes the cost function. But the key point to realize is that the cost function is taking the average over the individual errors on each sample. That is what it is minimizing for. What you have proposed is effectively an artificial “sledge hammer” solution: since I know that I have exactly the same number of red points as blue points in the input, I can get 50% accuracy by setting the prediction function to be a constant:
Predicting \hat{y} = 0 for all inputs will give me 50% accuracy. Or always predicting \hat{y} = 1 gives the same result.
It is analogous to always betting “heads” on a coin flip. Statistically you’ll be right 50% of the time. But if you compared the cost (the actual J value) of either of those solutions it would be significantly worse than the cost of the actual 47% accurate solution it solves for. So I guess you could view this as an artificial case in which you can make the output a constant and it just happens to get a better result than you can with Logistic Regression, which is only capable of doing “linear separation”.
Taking a higher level view, the point is that neither of those solutions is really worth anything, so we need a better approach to the problem. That was the whole point of including this Logistic Regression example to motivate why Neural Networks (even simple ones with a single hidden layer) can do a better job at this type of problem.
1 Like