In the above image we can see that for the given true_labels and predicted_logits, the cross entropy loss is 1.0508.
Above is the formula for cross entropy loss, if we apply this to the given “true_labels” and “predicted_logits” we get:
(-1 / 4) * [ln(0.1) + ln(0.7) + ln(0.4) + ln(0.4)]
(-1 / 4) * [-2.302 -0.356 -0.916 -0.916]
= 1.1225
But as we can see that the code expects the values as 1.0508.
I even checked the tensorflow documentation for tf.keras.losses.SparseCategoricalCrossentropy() and the formula i mentioned i what they used, because for the below image loss is:
(-1/2) * [ln(0.95) + ln(0.1)]
(-1/2) * [-0.0512 -2.302]
=1.177, which is what is the output in the below image.
Can you tell me the reason for this discrepancy?
This discrepancy is merely due to two different values of the from_logits argument in the tf.keras.losses.SparseCategoricalCrossentropy function.
I assume that in the assignment you used from_logits=True, since you got the expected output correctly.
This is indeed required in the assignment, as the instructions state:
But the default in tf.keras.losses.SparseCategoricalCrossentropy() is from_logits=False:
So if you use the default arguments with the values provided in assignment you get:
Which is close enough to 1.1225 you calculated above. I assume the difference in the 4th decimal is due to some rounding.
Similarly for the other set of values you provided:
Regarding the difference of from_logits=False & from_logits=True:
- from_logits=False is for use with probabilities that have already undergone a softmax activation function
- from_logits=True is for use with raw logits - before applying a softmax activation function
Given the above, I would say that the use of predicted_logits = [[0.1,0.6,0.3] , [0.2,0.7,0.1], [0.1, 0.5,0.4], [0.4,0.4,0.2]] in the testing snippet of the assignment could be confusing with regards to the use of from_logits=True: the list of predicted_logits is clearly probabilities, they are all positive and sum up to 1. Yet the masked_loss function itself was written for use with raw logits as the instructions state.
Best
2 Likes
