My code for Emoji_v3a passes all of the internal tests and shows high accuracy and low loss values when training the Emojify_V2 model but consistently gets a test accuracy value around 0.5. I carefully went through my code to compare my outputs against the expected outputs but couldn’t find any problems so I submitted the assignment, hoping the grader would identify which section of my code was incorrect, but the grader gave me 100/100 and said all functions are correct. Should I really expect the test accuracy to be between 80% and 95% and if so, where should I look to see what’s wrong?
I get 80% on the test set when I run my noteboook.
What loss and accuracy do you get from cell with the model.fit() results?
Please post the values from the last few iterations.
Those are the training set results, they look fine.
I’m not sure why you’d have that much lower accuracy on the test set.
I’ll think about it for a bit, and reply again if I have any ideas.
1 Like
I got a 0.625 test accuracy, but training accuracy also looks totally fine.
Same issue here, test accuracy between 65% and 75%:
2/2 [==============================] - 0s 3ms/step - loss: 1.4789 - accuracy: 0.6607
Test accuracy = 0.6607142686843872
Train accuracy gets up to 100%.
The model passes both the test and the autograder, so I can assume it’s correct. I did set the embedding layer as non-trainable.
Any ideas on what could be wrong?
Found the issue - the test dataset CSV file has been mishandled in the past, and the strings contain \t (TAB character) at the end. The training set does not suffer from this issue.
I used .split(’ ') to separate the words, which meant the last word of the sentence had a \t stuck to it and was not in the dictionary. The solution is to always use .split() without argument, which strips whitespace by default.
With correctly split words, I’m getting 82% accuracy, which is within the expected range, but surprisingly, still below the much simpler average vector model.
4 Likes
I come here for the same problem. I got 78% for test accuracy. Maybe this is because overfitting due to the use of embeddings??
