Transformer_Positional Encoding


I think the word Jane pos is equal to 0 , instead of equals to 1.

In the slide, <SOS> i.e. Start Of Token is at index 0. So, that moves the token Jane to index 1. Boils down to how you want to encode data.

Hello CDM,

Mentor Balaji is correct.

But we can always take another possibility like ‘first word of the sentence’ is Jane in the sequence, giving it the positional encoding 1 for its start position. It is believed that in certain implementations, positional encoding is assigned 1 instead of 0.