[week 4] Transformer Network - get_angles

To compute position encoding vectors, in the assignment:

  • We initialize a matrix of all possible values for the sine and cosine functions: N = get_angels(...).

    The matrix N has dimensionality pos x i. Where pos is a position (or time step) and i refers to a different dimension of the position encoding (e.g. i = {1,2,...,n}).

  • Later we use values in the matrix N to calculate actual values for each position encoding passing values from N as arguments to sine and cosine functions.

    Each row of the matrix N provides values required to calculate a position encoding for the position pos.

    The value of the position encoding vector at index i correspond to a point on the sinusoid curve for odd indices i in {1,3,5,...} and a point on the matched cosine curve for even indices i in {2,4,6,...}.

Screen Shot 2021-05-23 at 15.44.02

@shubhamchhetri I hope this makes things clear.