Why do we need positional encoding instead of representing word position with index values like 0, 1, 2, 3…?

Hello @evilyoda Word Embeddings is a technique where individual words are represented as real-valued vectors in a lower-dimensional space and captures inter-word semantics. Each word is represented by a real-valued vector with tens or hundreds of dimensions, i.e. every word is given a one hot encod…

Week 4 Positional Encoding

Course Q&A Deep Learning Specialization Sequence Models

balaji.ambresh April 9, 2024, 9:27am 2

The result of embedding + positional encoding would become large in this case.
Scale of inputs is important for faster convergence.

Topic		Replies	Views
Positional encoding in transformer networks (W4) - why adding as opposed to concatenating? Sequence Models	3	550	November 16, 2022
Help! I still don't understand how transformer works! Sequence Models	3	473	August 4, 2023
Position encoding: Time series Sequence Models	2	566	May 2, 2022
Doesn't positional encoding create noise in embedding(features) of word? NLP with Attention Models week-2	1	564	September 24, 2022
Can positional encoding be meaningfully generalized to non-integer position? Sequence Models	2	774	July 18, 2021

Week 4 Positional Encoding

Related topics