Course 5 Week 4 Transformers Lab Exercise 3

Hello,

In Exercise 3 of the Transformers Lab , I see that q has shape (3,4) and k has shape (4,4). So, their multiplication gives shape (3,4).

But shouldn’t q and k have the same dimension as there is one vector q and one vector k for every word i? This is mentioned in the second video of week 5.

Thank you!

q and k have the same dimension indeed, which is 4. The sequence lengths of q and k needn’t be the same (q is 3, k is 4 in the case.)