In the first Colab notebook where we are first introduced to using huggingface and pipeline, why does BERT fail at answering the question on which characters were created between 1939 and 1941 given the context has the answer?
Language models have trouble understanding “1939” and “1941”. For us humans this seems to be trivial but for Language Models it’s a problem (they are excellent at grammar, sentence construction (since they are trained on predicting next word) but they struggle at math, reasoning etc.)
Cheers
Thank you! That is an interesting piece of information. I am wondering, then, what else BERT is surprisingly not good at. If you know a blog post about that, let me know. Thanks!
No problem! 
I would hesitate to recommend a “blog post” because there is so many biases (and deepness of knowledge) on this topic that to know who is correct is practically impossible. There are ways to cherry pick examples to show where language models have problems and then say either that they are just a matter of time (days maybe ) or it will never be solved at all - and everything in between.
For example, on the one hand Nvidia might suggest that all you need is more GPU’s, on the other hand some researchers might suggest that “computation” alone is not enough. And then pick shortcomings and examples from that.
I would recommend, though, to read a A Primer in BERTology paper which might be a good starting point. (It’s 2020 Novemeber paper, so in terms of DL it’s almost two hundred years old now ). This paper points to papers like Do NLP Models Know Numbers? which addresses one of the problems.
If you are interested more in examples and possible implications.
I would also suggest recent interesting takes related to this topic (for me) by Dr. Walid Saba and Prof. Pedro Domingos.
Converselly if you find something interesting on this topic, please share with us
Cheers
P.S. later I found this blog post by Gary Marcus relevant to this topic.
Thank you so much for all the material and feedback! Really appreciate it! If I find something interesting, I will post it. Thanks!
1 Like