# What exactly does the improved implementation of softmax video mean?

**URL:** <https://community.deeplearning.ai/t/what-exactly-does-the-improved-implementation-of-softmax-video-mean/171001>\
**Category:** Advanced Learning Algorithms\
**Tags:** week-module-2\
**Created:** [July 27, 2022, 4:49am UTC](https://community.deeplearning.ai/t/what-exactly-does-the-improved-implementation-of-softmax-video-mean/171001 "2022-07-27T04:49:19Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nhat\_Minh](https://avatars.discourse-cdn.com/v4/letter/n/e36b37/32.png) [@Nhat\_Minh](https://community.deeplearning.ai/u/Nhat_Minh)\
**Post date:** [July 27, 2022, 4:49am UTC](https://community.deeplearning.ai/t/what-exactly-does-the-improved-implementation-of-softmax-video-mean/171001/1 "2022-07-27T04:49:19Z")

</div>

I don’t understand. Why we change the activation output layer into ‘linear’, and add from\_logits in loss function: BinaryCrossentropy(from\_logits = True)

Help meee  
Thanks so much

---

<div class="post-metadata">

**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\
**Post date:** [July 27, 2022, 8:14am UTC](https://community.deeplearning.ai/t/what-exactly-does-the-improved-implementation-of-softmax-video-mean/171001/2 "2022-07-27T08:14:47Z")

</div>

Hello @Nhat_Minh, welcome to our community! 🎉

 ![Screenshot from 2022-07-27 16-01-35](https://global.discourse-cdn.com/dlai/original/3X/0/e/0ec29c6ed947251287ab1a209dd1e22c7b3f6045.png)

The motivation behind the improvement is, tensorflow works more accurately with the 3rd equation on the left then with the 2nd equation on the left. We can see that in the 3rd equation, the term `a` never shows up, which means that by adopting the 3rd equation, tensorflow does not need to calculate `a`. This is good because calculating `a` out can generate numerical inaccuracy which is not favourable.

To make tensorflow works without calculating `a` out, we need to change the activation in the output layer from `sigmoid` to `linear`, because having `sigmoid` there is the reason for tensorflow to compute `a` out. Using `linear` actually means that we do not need any activation function. Therefore, changing from `sigmoid` to `linear` means that we are changing from passing `a = g(z)` into the loss to passing `z` into the loss.

Moreover, we need to let Tensorflow knows we are passing `z` into the loss instead of `a`, because tensorflow cannot detect the change itself. And we notify tensorflow by adding `from_logits=True` there. Also, logit is the name for `z`.

Now, with these two code changes, we enjoy a more accurate process of model training.

Cheers,  
Raymond

---

<div class="post-metadata">

**Author:** ![Nhat\_Minh](https://avatars.discourse-cdn.com/v4/letter/n/e36b37/32.png) [@Nhat\_Minh](https://community.deeplearning.ai/u/Nhat_Minh)\
**Post date:** [July 28, 2022, 7:38am UTC](https://community.deeplearning.ai/t/what-exactly-does-the-improved-implementation-of-softmax-video-mean/171001/3 "2022-07-28T07:38:36Z")

</div>

Thanks so much, @rmwkwok. I understand much more. But I want to know more. Help me if you have time.

You say: we passing z into the loss instead of a. But actually, we pass 1/(1 + e^-z). I am trying to googling binary\_crossentropy() function in details to understand more, putting much time, effort I haven’t searched for it.

---

<div class="post-metadata">

**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\
**Post date:** [July 28, 2022, 7:44am UTC](https://community.deeplearning.ai/t/what-exactly-does-the-improved-implementation-of-softmax-video-mean/171001/4 "2022-07-28T07:44:38Z")

</div>

Hello @Nhat_Minh,

Without `from_logit`, the function’s form takes the look of the second equation, which accepts an `a`. With `from_logit`, the function’s form takes the look of the third equation, which accepts only a `z`. Depending on whether `from_logit` is True or False, Tensorflow uses different functions.

Cheers,  
Raymond

---

<div class="post-metadata">

**Author:** ![Maxim\_Kupfer](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/maxim_kupfer/32/24931_2.png) [@Maxim\_Kupfer](https://community.deeplearning.ai/u/Maxim_Kupfer)\
**Post date:** [August 23, 2022, 11:34pm UTC](https://community.deeplearning.ai/t/what-exactly-does-the-improved-implementation-of-softmax-video-mean/171001/5 "2022-08-23T23:34:49Z")

</div>

I have follow up to this. What I’m missing is that I’m still unsure how we are skipping a calculation. Don’t we still have to compute the logistic or softmax values at some point?

---

<div class="post-metadata">

**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\
**Post date:** [August 24, 2022, 12:18am UTC](https://community.deeplearning.ai/t/what-exactly-does-the-improved-implementation-of-softmax-video-mean/171001/6 "2022-08-24T00:18:43Z")

</div>

Hello @Maxim_Kupfer, thank you for the question!

The log loss value will be calculated but not the sigmoid nor the softmax value. I am going to show you that and please feel free to check my calculation if you would like to. I will use the case of a binary outcome for simplicity, but the core idea is identical.

Given  
p = \frac{1}{1+\exp{(-z)}}  
Loss = -y\log{p} - (1-y)\log{(1-p)}

Here z is the logit value, and without calculating the sigmoid to get the corresponding value of p, we have the freedom to substitute p into Loss to make simplification that improves numerical stability:

Loss = -y\log{(\frac{1}{1+\exp{(-z)}})} - (1-y)\log{(1-\frac{1}{1+\exp{(-z)}})}   
= -y\log{(\frac{1}{1+\exp{(-z)}})} - (1-y)\log{(\frac{\exp{(-z)}}{1+\exp{(-z)}})}   
= \log(1+\exp{(-z)})- zy +z

If you calculate Loss in this manner, you will never have calculated p out, agree? Now there is one more trick to improve stability, which is to consider the case where z \< 0 and z \ge 0 separately, because in the former case the \exp{(-z)} term can yield a very large value that overflows any floating point data type. This is what we are going to do when z \<0:

\log(1+\exp{(-z)})- zy +z = \log{(1+\exp{(z)}))} -zy

which will never yield any exponentially _ **large** _ numbers because .\exp{(z)} \approx 0 when z\<\<0

In summary, now we have, from our original Loss function,

Loss^{\ge 0} = \log(1+\exp{(-z)}) -zy + z  
Loss^{\< 0} =\log{(1+\exp{(z)}))} -zy

that will never produce any exponentially large number.

Raymond

---

<div class="post-metadata">

**Author:** ![Maxim\_Kupfer](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/maxim_kupfer/32/24931_2.png) [@Maxim\_Kupfer](https://community.deeplearning.ai/u/Maxim_Kupfer)\
**Post date:** [August 24, 2022, 5:38pm UTC](https://community.deeplearning.ai/t/what-exactly-does-the-improved-implementation-of-softmax-video-mean/171001/7 "2022-08-24T17:38:43Z")

</div>

Thanks for the reply @rmwkwok.

> If you calculate Loss in this manner, you will never have calculated p out, agree?

That’s where I’m not understanding since aren’t you are still doing a calculation in the end with the only difference being that the terms were rearranged? Is it just that this new calculation is simplified?

> log(1+exp(−z))−zy+z=log(1+exp(z)))−zy

My algebra is a bit rough. Could you elaborate on how you simplified in the above equation?

---

<div class="post-metadata">

**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\
**Post date:** [August 24, 2022, 10:49pm UTC](https://community.deeplearning.ai/t/what-exactly-does-the-improved-implementation-of-softmax-video-mean/171001/8 "2022-08-24T22:49:56Z")

</div>

No problem!

> [@Maxim\_Kupfer](#):
>
> aren’t you are still doing a calculation in the end with the only difference being that the terms were rearranged?

The key difference is whether we explicitly calculated p or not.  
Let’s say y=0 and z=-305.8, if we do

p = \frac{1}{1+\exp{(-z)}}  
Loss = -y\log{p} - (1-y)\log{(1-p)}

we will have to first calculate p and for this purpose, we have to evaluate \exp{(+305.8)} = 6.415825835×10^{132}.

However, if we do

Loss^{\ge 0} = \log(1+\exp{(-z)}) -zy + z  
Loss^{\< 0} =\log{(1+\exp{(z)}))} -zy

we will not have to calculate p nor evaluate \exp{(+305.8)}, only \exp{(-305.8)}=0 which won’t overflow.

> [@Maxim\_Kupfer](#):
>
> My algebra is a bit rough. Could you elaborate on how you simplified in the above equation?

\log(1+\exp{(-z)}) = \log(\exp{(-z)}\times(1+\exp{(z)})) = -z + \log{(1+\exp{(z)})}

Cheers,  
Raymond

---

<div class="post-metadata">

**Author:** ![futurejj](https://avatars.discourse-cdn.com/v4/letter/f/a4c791/32.png) [@futurejj](https://community.deeplearning.ai/u/futurejj)\
**Post date:** [August 18, 2023, 3:45am UTC](https://community.deeplearning.ai/t/what-exactly-does-the-improved-implementation-of-softmax-video-mean/171001/9 "2023-08-18T03:45:35Z")

</div>

I have one query, shouldn’t using logits be the default behaviour of the softmax output in Tensorflow? As in, if using softmax leads to numerical inaccuracies, then in which scenario is it preferred over the logits way?

---

<div class="post-metadata">

**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\
**Post date:** [August 18, 2023, 4:14am UTC](https://community.deeplearning.ai/t/what-exactly-does-the-improved-implementation-of-softmax-video-mean/171001/10 "2023-08-18T04:14:01Z")

</div>

I cannot think of a case where logits is not preferred.

Tensorflow is a flexible framework that allows you to go the logits way or not. There is no default to that particular setting.

However, logits is my default.

Cheers,  
Raymond