Hello guys, I am a beginner in AI. I just started Andrew Ng’s deeplearning and neural network specialisation for the purpose of my internship topic, but I came to know that this course was made well before transformers were invented. So now I am planning to finich it first and then start another transformers only course. So do you guys have any recommendations for a transformers course? Thanks a lot.
You’re on the right track with the basics. Stick around, though; we have some transformative updates coming to the platform soon.
Hi guys,
Actually, I’m curious about an approach to use Transformers model for time-series data?
Would it bring an advantage to use this model instead of using a LSTM model?
Could this model calculate data for future time periods, as is done with the LSTM model?
The original version of DLS (Deep Learning Specialization) was published in 2017 and early 2018 (for course 5, which is the one that covers Transformers). The famous Attention Is All You Need paper from Google was published in June 2017 and the first version of DLS Course 5 (Sequence Models) was published in February of 2018. I’m pretty sure they introduced Attention in the first version of Course 5. All the DLS courses were updated in 2021 to convert from TensorFlow v1 to v2 (with Eager Mode). At that time they beefed up the material in DLS Course 5 (Sequence Models) Attention Models material to include a section on Transformers.
The Natural Language Processing Specialization (NLP) Course 4 is dedicated to Attention Models and covers Transformers as well.
@Ardinc your focus is Transformer related future time prediction but related to what factor?
please don’t tell me future prediction 
Another great course related transformer is RAG architecture course by @Zain_Hassan. He really has explained each detail of transformer and various techniques used to retrieve information.
Regards
Dr. Deepti
Hello @Ardinc,
While there is a practical limit due to vanishing gradient, LSTM can theoretically take a time-series input as long as you wish, but Transformer has a hard limit on the length of its context window. If we don’t care about that, then Transformer can make prediction like LSTM. However, it would be more rewarding to switch to Transformer if we deal with a dataset large enough for Transformer to learn the structure of time and each sample’s meaningful context window is too long for LSTM to capture well, otherwise, it could be better that we leverage LSTM’s recursive nature (process time steps one after another) to save the burden for model to learn temporal order and train a good model with limited amount of data.
Cheers,
Raymond
Hello @rmwkwok ,
Thank you very much for your detailed and helpful response.
I think I will stick with the LSTM structure, which I can handle better, and which is more suitable for battery level predictions in my zener estimation project.
Thanks a lot and best regards,
Ardinc
You are welcome, @Ardinc. Small model can still shine. Good luck!
Raymond