Difference between regularization and normalization

At the end of " Why does Batch Norm work?" lecture, in W3 of DLS C2 Andrew said:
“don’t turn to batch norm as a regularization; Use it as a way to normalize your hidden units activations and therefore speed up learning”

Can you please clarify what’s the difference between the mentioned regularization and normalization?

Hello @SRezaS ,
Thanks a lot for asking this question. I will do my best to clarify the differences between regularization and normalizaton.

In machine learning, regularization and normalization are two different concepts that serve distinct purposes.

Normalization is a data preprocessing technique that adjusts the values of features to a common scale, typically between 0 and 1, without distorting the differences in the range of values. This is done to ensure that all features contribute equally to the model, especially when the features have different scales or units. Normalization can help improve the convergence of the learning algorithm and the overall performance of the model.

On the other hand, regularization is a technique used to prevent overfitting in a model by adding a penalty term to the loss function. Overfitting occurs when a model learns the training data too well, including the noise, and performs poorly on new, unseen data. Regularization helps the model generalize better by preventing it from becoming too complex. There are two common types of regularization: L1-norm (Lasso) and L2-norm (Ridge Regression). Both of these methods add a penalty term to the loss function, which encourages the model to use simpler fitting functions and reduces the magnitude of the model parameters.

In summary, normalization is a data preprocessing technique that adjusts the scale of feature values, while regularization is a method used to prevent overfitting by adding a penalty term to the loss function. Both techniques can help improve the performance of a machine learning model, but they serve different purposes and are applied at different stages of the modeling process.

I hope I was able to give a good reply to your question and have explained regularization and normalization correctly. Please feel free to reply with a followup question if you have additional questions about my reply.
Regards,
Can Koz

2 Likes