https://www.coursera.org/learn/advanced-learning-algorithms/lecture/Tyil1/improved-implementation-of-softmax

[Vedbhai]

I cannot understand is the difference created by changing the model. compile() line.
Also I am unclear as to why directly substituting the value of the function will lead to a more accurate answer.

Please help
Regards

[rmwkwok]

Hello, @Vedbhai,

@Vedbhai

is the difference created by changing the model. compile() line

Two changes. ↓

@Vedbhai

why directly substituting the value of the function will lead to a more accurate answer.

e^x and e^{-x} are easily evaluated to be very large or small numbers. For example, when x = 10, e^{10} > 22000 and e^{-10} < 0.0001. While these numbers are not “very” large/small, but what if x=100 or even larger? The problem is, very large number can cause overflow, and very small number will be zeroed out.

Accuracy is increased because we can mathematically simplify the following equation into something that avoid the very large number problem and lessen the very small number problem.

The following thread shows the simplification. Since these courses are designed to be not Maths intensive, so it would be fine to skip the simplification.

Why is`from_logits=True` plus `activation=linear` more stable?

From C2 W2 “Improved implementation of softmax”, we know that, for a binary classification problem, the following approach A is more stable than the approach B: Approach Output layer’s activation Loss Function A “linear” tf.keras.losses.BinaryCrossentropy(from_logits=True) B “sigmoid” tf.keras.losses.BinaryCrossentropy(from_logits=False) This post will show the maths reason, and begin with the following slide: [image] The lecture replaces the middle equation (Approach B) with th…

Cheers,
Raymond