I am getting poor predictions when testing with my sentences:
You are good
You are not good
bad
not bad
What could be the reason for this, and how can it be improved?
- Better model
- Larger training set
- Better/larger pre-trained Embedding layer
Hey @Milan_Adamovic,
The training set only consists of 127 examples, so definitely, a larger training set can help to increase the performance of your model. As to a better model, I guess that depends on your improved dataset. If you add more examples following the same constraints but different words perhaps, then I guess the same model could give you better results as well, since the model seems to have enough complexity for the training examples similar to the current ones. But yes, a better model is indeed another way to go.
Here’s something from the assignment regarding the above point:
- If the training set were larger, the LSTM model would be much better than the Emojify-V1 model at understanding more complex sentences.
As to a better/larger pre-trained embedding layer, you will find that the Word2Vec map used in the assignment has 400K words. So, that should be pretty good in my opinion. So, I guess a larger embedding layer won’t help you that much in this scenario as compared to the other 2 approaches, but yes, in general, you can try getting higher dimensional embeddings trained on a larger corpus, and they could help you as well. I hope this helps.
Cheers,
Elemento
1 Like
The same problem I have and want to improve. I will implement what you’ve suggested @Elemento
I will get back here once I complete it.
Hi @Elemento I am back with a problem.
I have a large dataset (200000, 50). 200000 tweets and 50 different emojis. I did preprocess what was suggested in the assignment and everything is fine but the problem is in the training model. I am getting constant accuracy in training at each epoch. you can see the below screenshot. I investigated this problem and found the following cause:
- maLen = 179. However, most of the length of sentences are in the range of 20-40. 179 length adding too many zeros that dwindle performance
- I need to work on parameter tuning like learning rate, number of nodes in each layer, optimizer etc.
However, I also want you to look into this issue and please explain me why I am getting this constant accuracy. Any help will appreciate
I look forward to hearing from you
LET ME KNOW HOW COULD I SHARE MY NOTEBOOK WITH YOU. I AM WORKING ON KAGGLE NOTEBOOK
Hey @Shawn_Frost,
I guess I would be able to tell you better after playing with the code myself a bit. For instance, you can try reducing the maximum length based on the distribution of the length of sentences, but whether this will give you a considerable boost in the accuracy or not, that I can only tell after taking a look at the distribution.
Also, as you can see in your training, the loss is decreasing very slowly. This could indicate slower learning, and hence, you might try increasing the learning rate, but once again, how much it will help you, I can only tell after playing with the code.
As to sharing, if your notebook doesn’t contain any solution code from any of the assignments, feel free to make it public and share the link to your notebook here. However, if it does, here is my Kaggle profile, just provide me with the view access, and I will be able to take a look at your code. I hope this helps.
Cheers,
Elemento
Hi @Elemento
Thank you. Let’s solve this problem together I solved some problems, that I mentioned above. However, one problem is model is getting overfitting. Validation accuracy is not increasing it is stuck after 0.31. I made several changes like increasing LSTM layers, trying different optimizers, and changing learning_rate but not get good results. I shared with you as a collaborator. You may have gotten a notification.
https://www.kaggle.com/code/codenigma/emojify-v2/edit
Hey @Shawn_Frost,
Sure, let me take a look at it.
Cheers,
Elemento
Hey @Shawn_Frost,
I took a look at your notebook. In fact, I tried doing some manipulations myself, which you can check out here. The different versions of this notebook contain the different manipulations that I tried doing, but none of them worked.
The one key thing that I noticed is that in the Emojify assignment, neither of validation accuracy and validation loss has been logged, i.e., the model is trained on the entire dataset, and when that happens, accuracy on samples taken from the same dataset will naturally be good. Now, in the Emojify assignment, this makes sense as well, since the number of examples are only 127, but in your case, this won’t make much of a sense.
Still, what if we try this, i.e., training on the entire dataset. Have you tried that? If not, can you try that and share the results. At least, that will tell us a hint about the model’s competency. Or otherwise, please give me some time, I have 3 assignment deadlines and a paper deadline, and I have to complete as much as possible before I go back to my home this week. I will try my best to revert back as soon as possible.
Cheers,
Elemento
Thank you so much for your advice. Yeah, I see there is a problem. I will try to train on the whole dataset. Let you know.
I ran up to 50 epochs but didn’t work.
Test accuracy here = 0.269565224647522. Is there any problem with embedding or LSTM layer?
See this:
After splitting with all the white characters(default setting of .split), my acc bumped to 83%.
Thank you for your suggestion @Zhihan_Zhang. Sure, I will implement that. I was a bit busy with my VISA process. I will let you know in case I get a problem.