Question 10 in Module 2, Graded Quiz

Course Q&A Transformers in Practice
, ,
1

The Question 10 in Quiz of Module 2 is “Without positional encodings, how would a transformer process the phrase ‘The mouse sat on the elephant’ compared to ‘The elephant sat on the mouse’?”
Out of the given options, the correct answer is “It would treat them as identical, since both contain the same set of tokens.”. The explanation given is “The core self-attention mechanism is permutation-invariant. Positional encodings are required to inject sequence order.”
Other options are clearly wrong, however I have not completely understood the explanation for this answer.

Causal masking is not a type of positional encoding. It simply means not showing future to the model. So, even if positional encoding is not used, causal masking should still be used. Due to this reason, in first sentence, ‘mouse’ will attend only to [the]. In second sentence, ‘mouse’ will attend to [the, elephant, san, on, the].
Hence, both the sentences would not be treated as identical.

Please let me know if I am missing anything. Thanks!

2

hi @CharviVitthal

Great question.

Without positional encoding, a Transformer views a sentence as an unordered bag of words

Basically transformer will lack the ability to see the sequential difference between the two phrases even if you are talking about casual mask, remember casual mask prevents tokens from looking at future tokens, but it does not inject position information. So when the two phrases transformer process, it would have identical attention matrices. The self-attention mechanism relies entirely on position-agnostic dot-products. Basically the context will confuse the model :zany_face: without positional encoding.

Regards
Dr. Deepti