An improvement suggestion in C5_W4_A1_Transformer_Subclass_v1

The current code that visualizes pos_encoding is poor because dimension (512) is bigger than the figure resolution (432) in X direction. It shows some artificial vertical bands, and it took me a while to figure out that they are not a property of the positional encoding, but they come from the sampling effect in visualization :slight_smile: :

The fix is to reduce the dimension from 512 to 256, which is strictly less than 432 to avoid the sampling effect so that the artificial vertical bands are gone:

Please feel free to use the following code for your convenience:

# This line tells the resolution of the plt.
# The max position and dimension should be less than the resolution to mitigate the anti-aliasing effect
print(np.array(plt.rcParams.get('figure.figsize')) * plt.rcParams.get('figure.dpi'))

pos_encoding = positional_encoding(50, 256)

print (pos_encoding.shape)

plt.pcolormesh(pos_encoding[0], cmap='RdBu')
plt.xlim((0, 256))

Thanks for offering that code.

Forgot to add a comment: The narrow blue and white vertical bands on the right is a result of the alternating β€œsin” and β€œcos” functions (for x=2i and x=2i+1) as defined in the formula.