Loss function for softmax regression when a = 0

Hi everyone,
In the loss function for softmax regression, if ai approaches 0, -log(ai) approaches -infinity, then how can a model handle this value?

Hello @francesco4203,

First, this thing → image cannot be equal to zero. The best you can get is for it to tend to zero as z_1 tends to negative infinity.

However, in a computer, it is possible because it is not infinitely precise, and in that case, one usual trick is to add a very small number to make this → log ( 1e-7 + a) such that even if a_i numerically becomes 0, the 1e-7 will take care of that. Below is a relevant post on how Tensorflow uses a small number called epsilon to handle that.


1 Like

Clear explanation!
Thank you.