In the final assignment of Week 3, we are using two exam scores(X1 and X2) to predict if students will get selected or not.
The final decision boundary looks like this:
My question is, why didn’t we get a curved decision boundary from Logistic Regression. A curve would have been a better fit than a linear line.
Is there any way to improve the results so that the decision boundary curves?
The line boundary being straight (with respect to the input features) is a characteristic of the logisitic regression. This characteristic originiates from the fact that the model takes each feature as is:
y = \sigma(-(w_1x_1+w_2x_2+b))
Note that z = w_1x_1+w_2x_2+b is a straight line with respect to the features x_1 and x_2, and when z > 0, \sigma(-z) > 0.5 and the binary prediction is 1.
If you want a boundary being curved with respect to x_1 and x_2, you can do feature engineering and creates new, polynomial, input features such as x_1^2 and x_2^2. Now the following model will produce a circular boundary with respect to x_1 and x_2 as long as the weights are all non-zero:
y = \sigma(-(w_1x_1^2 + w_2x_2^2+b))
(Let me know if you have questions about the following statement:)
Note that, although it is circular with respect to x_1 and x_2, it is straight with respect to x_1^2 and x_2^2.
Knowing that engineering polynomial features can give you a cruved boundary, the remaining question is what polynomial features you will need in order to bend the boundary the way you want.
It would be a better fit, but we should note that it would be better for the training data. Fitting a perfect boundary to training data is not very difficult, but our objective is to train a model that predicts well on non-training data, because this is how we determine if a model has a good generalization power to future, currently unknown data.
In the model development stage, if we fit the model perfectly with the training data, but poorly on the non-training data (which is called a cv data), this is a well-known problem called overfitting. You will find videos about this in Course 2. We want to avoid overfitting problem so that the trained model performs excellent with the non-training data, but not primarily the training data.
Cheers,
Raymond