Loss function for softmax regression when a = 0

[francesco4203]

Hi everyone,
In the loss function for softmax regression, if ai approaches 0, -log(ai) approaches -infinity, then how can a model handle this value?
Thanks.

[rmwkwok]

Hello @francesco4203,

First, this thing → cannot be equal to zero. The best you can get is for it to tend to zero as z_1 tends to negative infinity.

However, in a computer, it is possible because it is not infinitely precise, and in that case, one usual trick is to add a very small number to make this → log ( 1e-7 + a) such that even if a_i numerically becomes 0, the 1e-7 will take care of that. Below is a relevant post on how Tensorflow uses a small number called epsilon to handle that.

C1_W3_Logistic_Regression_Potential problem

Hi @VladimirFokow, I think you do make a point here about the np.inf problem, and I think you might also be interested in how popular ML packages take care of that while keeping the loss formula the way it is most well-known today which is -y\log(f)-(1-y)\log(1-f). In tensorflow, if you trace through the source code from this beginning, you will end up seeing these lines: output = tf.clip_by_value(output, epsilon_, 1. - epsilon_) return -tf.reduce_sum(target * tf.math.log(output), axis) T…

Cheers,
Raymond

[francesco4203]

Clear explanation!
Thank you.