Hello,
I understand linear regression, but I don’t understand why the cost function uses the squared error.
Can someone explain the intuition behind it?
Thank you.
Using the square of the error (the difference between the true value and the predicted value) gives better mathematical behavior for this type of application. One easy intuitive way to see this is to consider how it behaves if the error is < 1 versus > 1: when you square a number less than one, it actually gets smaller, right? But if you square a number > 1, it gets much larger. So using the squared error has the effect of more severely punishing errors that are relatively large, versus ones that are relatively small.
Another more sophisticated way to see this same point is to look at the derivative of the cost function. The derivative of squared error is linear. This is particularly relevant because we will be using the gradients of the error to learn the best solution here, so with the squared error, the “correction” that we get from the gradient (the multidimensional derivative) of the error is proportional to the size of the error. The gradients are used in Gradient Descent to push the parameters in the direction of a higher quality solution. If we used the absolute value of the error, then the derivative is either 1 or -1 depending on whether the prediction is greater or less than the label. So the gradients in that case aren’t as good at pushing the parameters in the direction of a better solution. Because of that difference in behavior, Gradient Descent works better in this application with the squared error function.
As you progress through Machine Learning, you may encounter cases in which they do use the absolute value of the error rather than the squared error, but those are more sophisticated problems than the Linear Regression we are starting with here.
Squaring the error is the default for linear regression due to three main reasons:
- Smooth Convergence: The derivative of squared error is linear, meaning the gradient shrinks as the prediction gets closer to the target. This acts as a natural brake for gradient descent, allowing it to settle smoothly at the minimum. With absolute error, the derivative is constant (1 or -1), which causes the model to overshoot and oscillate around the minimum.
- Handling Large Errors: Squaring penalizes large errors far more severely than small ones. An error of 10 yields a cost of 100, while an error of 1 yields a cost of 1. This forces the model to focus on correcting major deviations.
- Mathematical/Statistical Alignment: Minimizing squared error is mathematically equivalent to maximizing the likelihood of the data (Maximum Likelihood Estimation) under the assumption that the noise in the target variable is normally (Gaussian) distributed.
If you want the model to ignore outliers, you can use absolute error (L1 loss). However, for most linear regression tasks, squared error (L2 loss) is preferred because it optimizes much better.