My code for Exercise 5 - Emojify_V2 passes all unit test cases, and even the automatic grader gave it 100%. However, while training the Emojify_v2 model, I sometimes get “loss: nan” and the model does not actually train. I checked the training dataset and it does not have null values (it has valid sentence indices). I have also restarted the kernel multiple times. Any explanation on why we get loss: nan? thanks
Hey @Mo_Mosaad_H,
Welcome to the community. If I am not wrong, the notebook uses 50 epochs to train the model and it seems like you have used the same as well (as per your code shown). Now, you might have changed the batch size, which could result in extremely large or extremely small values of loss, which resulted in loss = nan, and based on your output, it seems like that the loss is too large, since it is nan from the first epoch itself. Also, since the first iteration through the network gives loss = nan, hence, the back-propagation also fails, since it is not possible to differentiate something that is nan, and hence, the loss doesn’t decrease.
If the epochs = 50 only, then try changing the batch-size and see if the issue resolves. The trick is to set the initial value of these parameters so that the loss value remains within the constraints of Python variables, i.e., neither under-flows nor over-flows. And once they are set to good enough values, you have a finite loss value, which can be back-propagated through, and after some CO_2 emissions, yous loss will eventually come down and your accuracy will go up.
Thanks for your response.
I restarted the notebook. The problem still happens but randomly and I cannot really understand why it is happening. I ran the training with the provided parameters and it finished. But the accuracy was about 80%, less than the expected one between 90-100%. I ran it again, I got the “loss : nan” at later epochs, as shown below. Then, I ran it for a third time and the loss: nan appeared from the first epoch.
Hey @Mo_Mosaad_H,
That’s indeed strange. The code that you have shown now indicates that your loss became nan at the 41st epoch only, and was working fine up to that. Can you please DM your notebook as an attachment to me? I can only tell anything after taking a look at the code once, and after trying it out myself.
Hey @Ludovic_Legrand,
Welcome to the community. In this assignment, you are supposed to use the “Softmax” activation function, as you can read in the markdown of “Exercise 2 - model”. Mohamed used the ReLU activation function in place of the Softmax activation function, which resulted in this error.